# Exploring the Evidence-Based Beliefs and Behaviors of LLM-Based Programming
Assistants

Chris Brown  Virginia TechBlacksburgVAUSA<EMAIL_ADDRESS>and  Jason Cusati
Virginia TechBlacksburgVAUSA<EMAIL_ADDRESS>
###### Abstract.

Recent innovations in artificial intelligence (AI), primarily powered by large
language models (LLMs), have transformed how programmers develop and maintain
software—leading to new frontiers in software engineering (SE). The advanced
capabilities of LLM-based programming assistants to support software
development tasks have led to a rise in the adoption of LLMs in SE. However,
little is known about the evidenced-based practices, tools and processes
verified by research findings, supported and adopted by AI programming
assistants. To this end, our work conducts a preliminary evaluation exploring
the beliefs and behaviors of LLM used to support software development tasks.
We investigate 17 evidence-based claims posited by empirical SE research
across five LLM-based programming assistants. Our findings show that LLM-based
programming assistants have ambiguous beliefs regarding research claims, lack
credible evidence to support responses, and are incapable of adopting
practices demonstrated by empirical SE research to support development tasks.
Based on our results, we provide implications for practitioners adopting LLM-
based programming assistants in development contexts and shed light on future
research directions to enhance the reliability and trustworthiness of
LLMs—aiming to increase awareness and adoption of evidence-based SE research
findings in practice.

##  1\. Introduction

Recent technological advancements in machine learning have transformed how
software is constructed and maintained. For example, artificial intelligence
(AI)-powered programming assistants, such as ChatGPT (gpt, [n. d.]), GitHub
Copilot (cop, [n. d.]), and JetBrains AI (jet, [n. d.]), leverage large
language models (LLMs) trained on code to support the completion of software
engineering (SE) tasks. Prior work shows these AI-powered programming
assistants are able to support a wide variety of software development tasks,
including designing software (Ahmad et al., 2023), fixing programming bugs
(Sobania et al., 2023), and generating tests (Rao et al., 2023; Yuan et al.,
2023). Software engineers have quickly adopted LLMs into development
workflows. For instance, the JetBrains Developer survey results show 84% of
developers are familiar with generative AI tools (JetBrains, 2023). Moreover,
as of February 2023, GitHub reports approximately half of developers’ code
across programming languages is assisted by
Copilot.111https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai-
model-and-new-capabilities/

Despite these innovations, the quality of software systems primarily relies on
the efforts and behavior of human programmers to innovate, design, implement,
test, and maintain the various components of that software. For instance,
White et al. conclude that “significant human involvement and expertise is
currently necessary to leverage LLMs effectively for automating common
software engineering task” (White et al., 2023, p. 13). Further, while AI-
based programming assistants have shown promise to effectively automate
development tasks, prior work highlights the lack of credibility in generated
output—suggesting LLMs such as ChatGPT can provide insecure (Khoury et al.,
2023) and buggy (Liu et al., 2024) code.

To support software and its engineering, decades of empirical SE research has
investigated tools and processes to understand and enhance software
development. Yet, despite scientific evidence supporting development
practices—software engineers largely ignore purported research findings in
practice. For example, empirical SE research shows using static analysis
tools, or systems that automatically examine code to detect errors without
running the program, is beneficial for enhancing code quality (Ayewah and
Pugh, 2010), preventing errors (Bessey et al., 2010), decreasing debugging
time (Layman et al., 2007), and reducing developer effort (Singh et al.,
2017). However, studies also show software engineers often avoid static
analysis tools (Johnson et al., 2013), sparingly fix reported bugs (Marcilio
et al., 2019), and ignore notifications from automated code analysis systems
(Imtiaz et al., 2019). Findings produced by scientific evidence aim to better
decision-making processes in software development and maintenenace (Kitchenham
et al., 2004). Yet, little is known about the perceptions (“beliefs”) and
capabilities (“behaviors”) of LLMs regarding empirical SE research claims.

Motivating Example. Consider the LLM-based programming assistant
Devin222https://devin.ai/—recently announced as “the first AI software
engineer”. Devin is a novel AI system that can work with developers to
generate code, fix bugs, add new features, and deploy projects in its own
development environment (Wu, 2024). In three demonstration videos for Devin,
the AI software engineer uses print statements to debug errors in code,333See:
https://www.youtube.com/watch?v=fjHtjT7GO1c (0:52–1:10);
https://www.youtube.com/watch?v=TiXAzn2_Xck (2:12–2:25); and
https://www.youtube.com/watch?v=ReE2dFJn_uY&t=50s (0:40–0:49) despite being
equipped with its own coding editor and common development tools. While print-
statement debugging is widely adopted by programmers in practice (Beller et
al., 2018; Ko et al., 2023), it is discouraged by research in favor of
automated debugging techniques (Lieberman, 1997; Ko and Myers, 2004). As
humans and AI increasingly work together to generate code for software-based
systems, understanding LLM perceptions of evidence-based findings in
development tasks is crucial. Prior work has explored the beliefs and
behaviors of human software engineers (Devanbu et al., 2016). We extend this
by observing the evidence-based beliefs and behaviors of LLM-based programming
assistants. Our preliminary evaluation aims to answer the following research
questions (RQs):

  * RQ1:

What do LLM-based programming assistants believe about empirical SE research
claims

  * RQ2:

How do LLM-based programming assistants obtain evidence to support their
beliefs?

  * RQ3:

Can LLM-based programming assistants adopt behaviors that align with empirical
SE research claims?

Exploring these questions is crucial as software influences a growing extent
of modern society, and LLMs are increasingly used to support software (Ozkaya,
2023). To investigate our RQs, we conduct an exploratory study prompting AI-
powered systems to compare LLM-generated beliefs and behaviors regarding
empirically verified SE research findings based on prior work (Devanbu et al.,
2016). Our results show LLM-based programming assistants mostly agree with
these claims, but provide ambiguous answers and lack credible sources of
evidence. Further, these systems are incapable of adopting beneficial SE
practices to support development work. Based on these findings, we provide
implications for developers utilizing LLM-driven systems to complete
development tasks. We also offer future research directions to enhance
development workflows and leverage LLMs to improve the adoption of evidence-
based practices in software development.

##  2\. Background

This section defines terminology for this paper (Section 2.1) and outlines the
AI tools (Section 2.2) used in our evaluation.

###  2.1. Terminology

####  2.1.1. LLM-Based Programming Assistants

Large language models are deep learning models trained on large amounts of
data, such as books, articles, websites, and source code (Ozkaya, 2023). More
recently, LLMs have shown promise in supporting a wide variety of software
development tasks (White et al., 2023), and are being increasingly used by
software engineers in development contexts (Russo, 2024). Further, developers
collaborating with LLMs has been shown to have a positive impact on software
development activities (Hou et al., 2023). In this paper, we define LLM-based
programming assistants as tools powered by LLMs that have capabilities to
support assistance in programming and software development tasks. For this
project, we explore the capabilities of a variety of LLM-based programming
assistants (see Section 2.2) with regard to empirical claims supported by
software engineering research.

####  2.1.2. Empirical SE Claims

We define empirical SE claims as evidence-based findings derived through the
collection and analysis of data using empirical methods that regard factors
that affect software development. Researchers have consistently offered
insights to guide systematic efforts to support data collection and analysis
in SE contexts (Kitchenham et al., 2002; Storey et al., 2020). For our
exploratory study, we derived empirical SE claims from prior work exploring
the beliefs of empirical SE findings from the perspective of human software
engineers (Devanbu et al., 2016). In addition, we also added an additional
claim related to the use of debugging tools based on our motivating example of
Devin using print-statement debugging to find and fix coding errors. The list
of claims is presented in Table 1. From these claims, we observed the beliefs
and behaviors of LLM-based programming assistants.

####  2.1.3. LLM-Generated Belief

We define an LLM-generated belief as an assumption made in natural language
responses from systems powered by generative AI with regard to empirically
verified research-based concepts. While there are numerous distinct uses of
the concept of ‘belief’ in AI across many philosophical and cognitive science
domains (Hadley, 1991), we take the syntactically defined logic-based “belief
as ‘what one could rapidly discover”’ approach, presented by Hadley as
follows:

> Agent X believes [sentence] S if and only if S is in X’s belief base, or X
> could prove S, using the belief base and a logic L, within time T belief
> base. (Hadley, 1991, p. 66)

This definition incorporates the capabilities of modern LLMs—which are trained
on information provided in a large belief base, but have the ability to
generate new knowledge in a fixed amount of time—as opposed to other
approaches (i.e., belief as “degrees of confidence”). Further, this definition
of belief regarding AI can impact human reasoning (Hadley, 1991).

####  2.1.4. LLM-Generated Behavior

We define an LLM-generated behavior as the capability of generative AI-based
systems to adopt or neglect beneficial practices related to empirical SE
claims. Prior work suggests LLMs exhibit social behavior during interactions
with humans (Leng and Yuan, 2023). However, most LLM-based assistants are
black box systems, based on opaque or hidden information and algorithms (von
Eschenbach, 2021). In particular for SE-related tasks, the methods by which
output is generated is unknown—leading to major consequences such as insecure
(Khoury et al., 2023) or incorrect (Liu et al., 2024) code. In this work, we
aim to understand if LLM-based programming assistants have capability to
behave in accordance with empirically supported research practices. To observe
this, we ask the systems to provide insight on their ability to adopt
evidence-based SE tools and processes when generating and analyzing provided
code to support software development tasks.

###  2.2. LLM-Based Programming Assistants

Our goal is to understand the beliefs and behaviors of LLMs related to
empirical SE research claims. To accomplish this, we selected a variety of
LLM-based programming assistants—interactive chat-based systems powered by
generative AI models that are capable of supporting software development
tasks. The following LLM-driven programming assistants were leveraged for our
evaluation:

ChatGPT: ChatGPT (gpt, [n. d.]) is a popular AI chatbot by OpenAI. Extensive
research demonstrates the capabilities and effects of ChatGPT on software
development, showing this platform is able to support tasks related to
requirements analysis (White et al., 2023), software architecture and design
(Ahmad et al., 2023), unit testing (Guilherme and Vincenzi, 2023), debugging,
refactoring (DePalma et al., 2024; AlOmar et al., 2024), code understanding
(Nam et al., 2024), question answering (Da Silva et al., 2024), and more. For
this project, we leveraged ChatGPT 3.5 through its web interface.

GitHub Copilot: GitHub Copilot (cop, [n. d.]) is a programming assistant
powered by OpenAI’s GPT-4 model. This system has the ability to solve simple
programming problems (Wermelinger, 2023) and increase productivity for
developers (Imai, 2022). Research has explored how programmers use Copilot in
software development processes (Bird et al., 2023). Further, studies at GitHub
show Copilot increases the productivity, happiness, and satisfaction of
developers (Kalliamvakou, 2022). GitHub Copilot recently introduced a chat
interface that allows users to ask questions and receive LLM-generated answers
within a supported integrated development environment
(IDE).444https://docs.github.com/en/copilot/github-copilot-chat/using-github-
copilot-chat-in-your-ide We accessed the GitHub Copilot Chat using Visual
Studio Code555https://code.visualstudio.com/ (Version 1.87.1).

Gemini: Google Bard, now known as Gemini (gem, [n. d.]), is an online system
powered by Google AI trained on information from Google services. Prior work
by Belzner et al. utilized Gemini in a case study to conduct SE requirements
engineering, design, and testing tasks in a comparison with GPT (Belzner et
al., 2023). Moreover, Gemini Code Assist is a programming assistant based on
Gemini that can support software development within coding editors and Google
Cloud.666https://cloud.google.com/gemini/docs/codeassist/overview For this
evaluation, we utilized the Basic version of Gemini 1.

Blackbox AI: Blackbox (bla, [n. d.]) is an AI programming assistant that
claims to be the “best AI model for code”.777https://www.blackbox.ai/ There is
limited research investigating the capabilities of Blackbox AI in SE contexts,
however this system is trained on an AI model specifically for code
generation—intended to save time in coding tasks by translating natural
language into functional programs and acting as a go-to coding companion
(Stork, 2024). Blackbox AI offers a Blackbox Code Chat feature, which can be
utilized as a VS Code plugin or in an Internet browser, to answer coding
questions and assist users while writing code. For our study, we use Blackbox
AI Code Chat version 2.2.10 accessed online.

Claude: Claude (cla, [n. d.]) is an AI assistant powered by Anthropic, a
company focused on building reliable, interpretable, and safe AI
systems.888https://www.anthropic.com/ While there is no peer-reviewed work
exploring the capabilities of this system for automating SE tasks, to our
knowledge, Claude claims to be able to support code generation—from “creating
websites” to “debugging complex code
bases”.999https://www.anthropic.com/claude Online news outlets and blogs also
report Claude outperforms existing AI assistants and is effective for
generating code, understanding various programming languages, explaining code
functionality, and supporting debugging tasks (i.e., (Ahmed, 2024),
(Wilkinson, 9233), etc.). We used the basic version of Claude 3 Haiku for this
work, which has been shown to have over 75% capabilities for undergraduate-
level knowledge and code
generation.101010https://www.anthropic.com/news/claude-3-haiku

##  3\. Methodology

To answer our research questions, we analyzed AI-driven programming assistants
to observe their “beliefs” and “behaviors” regarding various claims related to
software development practices—inspecting LLM-generated output against
empirically validated research findings.

###  3.1. Data Collection

To observe the beliefs and behaviors of LLM-based programming assistants, we
prompted each AI tool presented in Section 2.2. Prompting is a key feature of
conversational LLMs where users submit natural language-based queries, or
prompts, and receive generated responses in a natural language format
(Zamfirescu-Pereira et al., 2023). In software development contexts, prompts
can be used to create LLM-generated responses to support a variety of software
development tasks (White et al., 2023). To gain insight into LLM-based
programming assistants’ beliefs and behaviors regarding empirical SE research
claims, we used zero-shot prompting to prompt each system once without
providing examples or additional clarification (DAIR.AI, 2024). All LLM-based
programming assistant data was collected between April and June 2023. The
complete list of prompts and LLM-generated responses for this research are
available in an online repository (Brown and Cusati, 2024).

Table 1. Empirical Software Engineering Research Claims

Claim
---
C1: Code quality (defect occurrence) depends on which programming language is
used (Ray et al., 2014).
C2: Fixing defects is riskier (more likely to cause future defects) than
adding new features (Shihab et al., 2012; Mockus and Weiss, 2000).
C3: Geographically distributed teams produce code whose quality (defect
occurrence) is just as good as teams that are not geographically distributed
(Bird et al., 2009; Kocaguneli et al., 2013).
C4: When it comes to producing code with fewer defects specific experience in
the project matters more than overall general experience in programming
(Rahman and Devanbu, 2011).
C5: Well commented code has fewer defects (Tenny, 1988).★
C6: Code written in a language with static typing (e.g., C#) tends to have
fewer bugs than code written in a language with dynamic typing (e.g., Python)
(Ray et al., 2014; Hanenberg, 2010).
C7: Stronger code ownership (i.e, fewer people owning a module or file) leads
to better software quality (Hanenberg, 2010; Bird et al., 2011; Weyuker et
al., 2008).
C8: Merge commits are buggier than other commits (Devanbu et al., 2016; Shen
et al., 2023).
C9: Components with more unit tests have fewer customer-found defects (Janzen,
2005).★
C10: More experienced programmers produce code with fewer defects (Rahman and
Devanbu, 2011; Mockus and Weiss, 2000).
C11: More defects are found in more complex code (Kemerer, 1995).
C12: Factors affecting code quality (defect occurrence) vary from project to
project (Rahman et al., 2012; Zimmermann et al., 2009).
C13: Using asserts improves code quality (reduces defect occurrence) (Baudry
et al., 2001; Aniche et al., 2013).★
C14: The use of static analysis tools improves end user quality (fewer defects
are found by users) (Thung et al., 2012; Zheng et al., 2006).★
C15: Coding standards help improve software quality (Boogerd and Moonen,
2009).★
C16: Code reviews improve software quality (reduces defect occurrence) (Porter
et al., 1997).★
C17*: Debugging tools improve software quality (reduce defect occurrence) more
efficiently than print-statement debugging.★

  * •

Evidence-based SE claims derived from (Devanbu et al., 2016), C17 is unique to
this research. $\bigstar$ denotes claims also evaluated as a behavior.

####  3.1.1. RQ1: Collecting LLM-Generated Beliefs

We observed natural language responses to prompts regarding empirical SE
findings from the LLM-based programming assistants used in our evaluation to
determine their beliefs. To prompt each AI programming assistant, the claims
(see Table 1) were transformed into questions to ask each AI programming
assistant platform to provide a response for each item.For example, “Code
quality (defect occurrence) depends on which programming language is used” was
input as “Does code quality (defect occurrence) depend on which programming
language is used?” (C1). We initially prompted AI systems to provide a 5-point
Likert response from one (Strongly Disagree) to five (Strongly Agree) for each
claim–however, after several attempts of using this approach we observed the
AI programming assistants would generate arguments for each Likert rating
without selecting a level of agreement. Thus, we transitioned to prompting
with yes or no questions to observe LLM-generated beliefs. This approach still
provided ambiguous responses, which we incorporate into our analysis (see
Section 3.2.1).

####  3.1.2. RQ2: Deriving Evidence for LLM-Generated Beliefs

As software developers are adept to blindly accept recommendations from LLMs
(Kammerath, 2024), examining the evidence for the beliefs of LLM-based
programming assistants is crucial to assess generated output for SE
activities. To examine how LLMs evince their beliefs, we analyzed the
responses for LLM-generated beliefs and prompted AI assistants to understand
how they came up with responses to claims about beliefs. Following each prompt
to derive LLM-generated beliefs on the empirical research claims studied (see
Section 3.1.1 and Table 1), we prompted “How did you arrive at your answer?”
to gain further insight into how the LLMs explored in this study form the
basis of their responses to questions regarding empirical SE research claims.

####  3.1.3. RQ3: Collecting LLM-generated Behaviors

To understand the capabilities of LLM-based programming assistants with regard
to empirically supported behaviors, we prompted systems to understand whether
they adopt practices based on SE research claims in their code generation and
analysis processes. In particular, we used C5 (commenting code), C9 (unit
testing), C13 (asserts), C14 (static analysis tools), C15 (coding standards),
C16 (code reviews), C17a (debugging tools), and C17b (print-statement
debugging). This consisted of asking the agents if they adopt certain
behaviors when generating code or performing tasks on provided code. Examples
include “Can you use static analysis tools to improve generated code?” (C14)
and “Can you use debugging tools to debug provided code?” (C17). In certain
instances, we only modified the prompt to specify the behavior was directed
toward the LLM when responses were not applicable (e.g., for C14, changing
“Can you use static analysis tools to improve generated code?” to “Can ChatGPT
use static analysis tools to improve generated code?”). These responses were
analyzed to understand if LLMs can adopt evidence-based practices while
supporting software development tasks.

###  3.2. Data Analysis

To analyze the collected LLM-generated beliefs, evidence for beliefs, and
behaviors, we use a mixed methods approach.

####  3.2.1. RQ1: Beliefs

To analyze the overall empirical beliefs of LLM-based programming assistants,
an initial inspection was performed on generated responses from each LLM
observed in our study. The goal of this process was to produce a high-level
categorization of “Yes” (Y), “No” (N), “Ambiguous” (A), or “Not Applicable”
(N/A)—indicating whether the LLM agreed with, disagreed with, or provided a
noncommittal (i.e., did not make a clear stance by integrating both yes and no
into responses) response. For example, a yes categorization for C14 with
respect to belief signifies the text output from an LLM agreed that static
analysis tools improve code quality. Sample categorized responses are provided
in Appendix A. We measured ambiguous responses since we do not re-prompt LLMs
to obtain a definitive answer, maintaining a consistent zero-shot approach for
LLMs in our evaluation. For our initial evaluation, the authors performed a
closed coding of the 85 responses to categorize LLM-generated
beliefs—individually rating responses based on the above criteria, then
meeting to discuss respective ratings and resolve disagreements. We use
Cohen’s Kappa ($\kappa$) to measure inter-rater agreement, observing “fair
agreement” ($\kappa=0.398$) (Landis and Koch, 1977). Most disagreements arose
coding A with Y ($n=12/26$) and N ($n=10/26$) responses, e.g., as LLM replies
often began positively but ended vaguely.

To further investigate the evidence-based beliefs of LLMs, we observed several
quantitative metrics. We calculate the rate of correct responses (✓%) for
beliefs (belief = Y) to determine if LLM-based programming assistant beliefs
align with empirical SE findings. We measure the rate of correctness across
generative AI systems to investigate the beliefs of each LLM and across each
SE research claim to observe which findings are most upheld by LLMs. We also
calculate the variance for each empirical SE claim observed in our study to
determine which beliefs and behaviors are the most controversial among the
evaluated LLM-based programming assistants. To calculate the variance, the
initial categorizations were coded to numerical values, reclassifying yes to
1, noncommittal to 2, and no to 3 (Kaplan, 2020, Chapter 3.2). Then, we
calculate the variance of responses ($\sigma^{2}$) using the formula shown in
Equation 1.

####  3.2.2. RQ2: Evidence

To answer RQ2, we inspected the original response from LLMs based on given
prompts to gain insight on generated statements regarding beliefs. We were
particularly interested in whether or not LLMs provided evidence to support
their responses. To measure this, we examined the LLM-generated belief
responses to examine the text for evidence—or resources and content to support
their answers. The first author scanned responses for links to and clear
mentions of sources of evidence. For links, this involved observing the
content of the provided source and the goals of the hosting website. Based on
this analysis, the evidence was then categorized into one the following types:
(1) research papers published in academic venues; (2) opinionated blogs by
individuals or groups; (3) posts shared on social media; (4) training
resources such as tutorials, documentation, and materials for learning; (5)
posts on Q&A platforms; (6) tech reports summarizing non-academic and
unpublished studies from industry; and (7) LLM-generated code examples.

Further, we further prompted LLM-based programming assistants to understand
how they arrived at their responses to questions regarding empirical SE
research claims (see Section 3.1.2). The follow-up questions allowed us to
gain insight into the evidence used to support responses for each LLM-
generated belief from a prompted response. We leveraged an open coding
thematic analysis approach to inspect the responses provided by LLMs (Braun
and Clarke, 2012). The responses to the 17 follow-up questions focusing on
evidence for each research claim were examined by the two authors, with each
independently evaluating the output to investigate sources of evidence for
responses. The concepts used to label the data were derived directly from the
content of responses. After developing initial codes, the researchers
segmented and grouped responses into categories based on common patterns and
recurring themes. We observed four themes described as sources of evidence for
LLMs: research, training data, it’s own reasoning, and previous conversations
in the chat. We report the evidence from original and prompted responses in
Section 4.2 with our results.

####  3.2.3. RQ3: Behaviors

To analyze the behavioral capabilities of LLM-based programming assistants, we
used a similar closed coding approach described in Section 3.2.1. We
categorized responses regarding behavior as Y, N, A, or N/A to indicate
whether or not LLMs are able to adopt practices supported by empirical SE
research. For instance, a no categorization for C14 with respect to behavior
indicates an LLM cannot use static analysis tools to examine generated code.
Example categorizations are presented in Table LABEL:tab:gioia in Appendix A.
The authors performed a closed coding on the 40 responses regarding the LLM-
generated behaviors, individually rating responses before resolving
disagreements. We observed “substantial agreement” ($\kappa=0.780$) between
the raters’ categorizations (Landis and Koch, 1977).

We also conducted quantitative analyses to understand the capabilities of LLM-
based programming assistants. We calculated the rate of correctness (✓%) to
indicate LLMs are able to adopt the observed behaviors (behavior = Y). We
report the LLM able to adopt the most evidence-based behaviors, and the
behavior that is most capable of being adopted by LLMs. For behavior, we
exclude C17b concerning print-statement debugging. While we are interested in
if LLMs are capable of using print statements to support debugging efforts,
this is not considered best practice and was removed from the measurement of
correct behavior. Further, we observe the rate inconsistency ($I$) for
empirical SE research claims between the beliefs of LLM-based programming
assistants. That is, instances where the output from LLMs supports empirical
SE research claims (belief = Y), but the model itself reports not adopting the
corresponding practice when automating SE-related tasks (behavior = N).
Finally, we calculate the variance of responses ($\sigma^{2}$) by translating
the behavioral categories (Y, N, and NC) to numbers as described in Section
3.2.1 and using Equation 1.

(1) $\sigma^{2}=\frac{1}{n}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}$ Eq. 1. where $n$
represents the total number of samples, $x_{i}$ corresponds with each value in
the dataset, and $\bar{x}$ is the mean of all values in the dataset.

###  3.3. Threats to Validity

There are several threats to the validity of our research findings.

##### Threats to External Validity:

Our study is based on only five AI-based programming assistants. We attempted
to mitigate this threat by analyzing systems based on generative AI models
trained on varying data and offered by different organizations (i.e., OpenAI,
Google, Blackbox AI, and Anthropic). We also utilize different access methods,
obtaining LLM-generated output through IDE plugins and online chat interfaces.
In addition, we examined a limited number ($n=17$) of empirical SE research
claims. The claims we analyzed, except for C17, were motivated by prior work
exploring the software engineer beliefs of empirically verified research
(Devanbu et al., 2016). To enhance the generalizability of this project,
future work can explore using additional LLM-driven systems (i.e., Devin (Wu,
2024) or watsonx (wat, [n. d.])), incorporating advanced versions of the
models used in this evaluation (i.e., GPT-4o), and investigating beliefs and
behaviors for more claims supported by findings from the empirical SE research
community.

##### Threats to Construct Validity:

Our constructs of an “LLM-generated belief” and “LLM programming assistant
behaviors” are flawed, as AI systems in and of themselves cannot believe and
behave—as their output is based on the data the models are trained on. While
our methods for collecting the beliefs and behaviors of LLMs have yet to be
supported by experimental results, we use these terms for “technical
convenience, and not as an analysis of ‘real belief”’ (Hadley, 1991, p. 66).
In particular for behavior, since we do not have knowledge of the inner
workings of LLM-based programming assistants, our analysis is solely focused
on the self-reported behaviors derived through analyzing natural language
output from the systems. Further, to analyze behaviors we only analyze the
output from LLMs in responses to prompts. In addition, further inspection and
more in-depth knowledge of specific models is needed to understand whether
LLM-based programming assistants do actually adopt these practices in support
of development tasks. For RQ2, the types of evidence categories for the
original responses have opportunities for overlap (i.e., Q&A posts on social
media, blogs summarizing research papers, etc.). We inspected the evidence
content and goals of hosting website, when applicable, to assign unique
categories to each resource.

##### Threats to Internal Validity:

Our data collection and analysis methods for LLM-generated beliefs, evidence,
and behaviors have limitations. LLM-based chatbots, such as ChatGPT and
Gemini, can perform differently in different chat sessions and have biased
results based on previous discussions (Morris, 2024). Similarly, studies show
re-prompting LLMs with the same prompt can generate different responses
(Ouyang et al., 2023). In this preliminary evaluation, we use zero-shot
prompting for each LLM—observing initial responses by prompting once without
follow-up. This can limit our results, as pre-trained models may not be
specifically trained for our task (Ramlochan, 2024). Additional techniques
such as one- or few-shot prompting (Kadam and Vaidya, 2020), Chain-of-Thoughts
prompting (Wei et al., 2022), other finetuning of instructions (Wei et al.,
2021), and changing temperature values (Renze and Guven, 2024) could impact
our results. LLMs also have the ability to hallucinate, or confidently provide
false information in responses (Hu et al., 2023). For instance, we observed
Copilot note “As an AI, I don’t execute or run code”—when in fact, it can
(cop, [n. d.]).

The manual qualitative analysis used to categorize the beliefs, evidence, and
behaviors of LLMs can incorporate bias. Converting categorical data to
numerical values for variable calculations can limit our results (Kaplan,
2020). One researcher categorizing types of sources for evidence can also be
biased. We also only report if evidence exists, but do not assess the quality
of evidential sources. The closed coding analyses inhibit our evaluation, as
open coding thematic analysis approach could provide more insight on the LLM-
generated beliefs and behaviors. Our analysis approach was used due to the
exploratory and preliminary nature of this work, and future work is needed to
further analyze the beliefs and behaviors of LLMs regarding empirical SE
research claims. Further, our collected and analyzed data are available in a
public repository to support the verifiability and replicability of our work
(Brown and Cusati, 2024).

##  4\. Results

This section presents our findings on the evidence-based beliefs, evidence,
and behaviors of five LLM-based programming assistants observed in our
methodology to answer our research questions.

###  4.1. RQ1: LLM-Generated Beliefs

Our results regarding the beliefs of LLM-based programming assistants are
presented in Table 2. The majority of responses ($n=55$, 64.7%) supported the
evidence-based claims explored in our study. We observed a limited number
($n=9$, 10.6%) of negative responses disagreeing with research claims and
noted 22 ambiguous responses (25.9%). We provide further insights based on the
most positive, negative, and controversial beliefs as well as the accuracy of
LLMs.

##### Most Supported Beliefs.

All of the LLM-based programming assistants agreed that evidence-based
practices such as unit testing (C9), assert statements (C13), static analysis
tools (C14), coding standards (C15), and code reviews (C16) enhance code
quality and benefit software projects. In addition, each LLM agreed with the
claims that more complex code leads to more defects (C11) and quality varies
based on project context (C12). These indicate evidence-based claims supported
by empirical SE research findings that LLMs agree are valuable for enhancing
software development in practice.

##### Least Supported Beliefs.

Several claims supported by empirical SE research had no positive claims. For
instance, all LLMs provided ambiguous responses for the claims that statically
typed languages have fewer bugs (C6) and stronger code ownership leads to
higher quality software (C8). In addition, no LLM agreed that merge commits
are buggier than other types of commits (C8). Moreover, we observed C8 and C2,
the claim that fixing bugs is riskier than adding new features, received the
most negative responses from LLM-generated output ($n=3$). Thus, despite
research evidence in support of these claims, they are not promoted by LLMs
commonly used for software development tasks.

##### Most Controversial Belief

We define the most controversial belief as the research claim with the highest
variance, indicating the highest disagreement between the LLM-based
programming assistants evaluated in our study. Our results show the claim that
fixing defects is more likely to cause defects than adding new features (C2)
was the most controversial ($\sigma^{2}=0.96$), incurring three negative
responses (ChatGPT, Copilot, and Gemini) and two positive responses (Blackbox
AI, Claude). The negative responses directly opposed this claim—stating, for
instance, “in general, fixing defects is often perceived as less risky than
adding new features” (ChatGPT) and “Fixing defects is generally considered
less risky because you’re dealing with existing code…Adding new features
inherently has more risk due to the introduction of entirely new code and its
interaction with the existing system” (Gemini). However, research suggests the
opposite trend (Shihab et al., 2012; Mockus and Weiss, 2000). For instance,
prior work suggests bug fixing changes introduced on Fridays in source code
for Mozilla and Eclipse had a higher chance of introducing new bugs (Śliwerski
et al., 2005). On the other hand, positive responses provided reasons for bug
fixing being riskier, such as unintended consequences in defect repair (i.e.,
“Defects are often interconnected, and fixing one defect can have a ripple
effect, causing other defects to emerge. This can lead to a cascade of fixes,
each with its own set of risks”, Blackbox AI), and arguments for adding new
features being less risky (i.e., “ adding new features typically involves
working with a more isolated and well-defined scope, which can be easier to
design, implement, and test thoroughly”, Claude).

##### Most Accurate LLMs

Overall, we found Blackbox AI and Claude were the most accurate LLMs with
regard to the empirical SE research findings evaluated in our study—providing
positive responses for 13 of the 17 claims (76%). Both models also did not
respond negatively to any evidence-based claims, while providing ambiguous
responses to four. The models were in agreement on all claims except two.
Claude was the only LLM to clearly state well-commented code can reduce defect
occurrence (C5). Blackbox AI responded positively to experienced programmers
produce code with fewer defects (C10), but Claude was ambiguous. To understand
the rationale for these responses, more information is needed on the training
data and AI models powering these LLM-based programming assistants.

##### Least Accurate LLMs

Alternatively, we observed ChatGPT and GitHub Copilot were the least accurate
LLMs observed in our study, responding positively to only nine claims (53%).
Copilot was the most incorrect model, responding negatively to the most
evidence-based claims ($n=4$, 23.5%), disagreeing that code quality depends on
programming language (C1), fixing defects is more likely to cause bugs than
adding new features (C2), specific project experience decreases bugs compared
to general programming experience (C4), and merge commits are buggier than
other types of commits (C8). ChatGPT also responded negatively to C2, C4, and
C8, but provided a positive response to C1. However, we found ChatGPT, along
with Gemini, were the most noncommittal LLMs in responses ($n=5$, 29.4%).
These models consistently provided ambiguous responses by sharing both reasons
to agree with and disagree with empirical research claims, not taking a clear
stance in either direction. The inaccuracies of ChatGPT and Copilot could
point to a limitation of the underlying model, as both are powered by OpenAI’s
GPT.

Table 2. LLM-Generated Belief (RQ1) Results Belief | Summary | ChatGPT | GitHub Copilot | Gemini | Blackbox AI | Claude | ✓% | $\sigma^{2}$
---|---|---|---|---|---|---|---|---
C1 | Quality depends on language | Y | N | Y | Y | Y | 0.8 | 0.64
C2 | Bug fixing is riskier than new features | N | N | N | Y | Y | 0.4 | 0.96
C3 | Distributed teams $\approx$ non-distributed | A | A | Y | Y | Y | 0.6 | 0.24
C4 | Specific experience ¿ general experience | N | N | A | Y | Y | 0.4 | 0.80
C5 | Well commented code reduces defects | A | A | A | A | Y | 0.2 | 0.16
C6 | Static typed languages have fewer bugs | A | A | A | A | A | 0.0 | 0.00
C7 | Stronger code ownership = better quality | A | A | A | A | A | 0.0 | 0.00
C8 | Merge commits buggier than others | N | N | N | A | A | 0.0 | 0.24
C9 | More unit testing = fewer defects | Y | Y | Y | Y | Y | 1.0 | 0.00
C10 | Experienced programmers = fewer defects | A | Y | Y | Y | A | 0.6 | 0.24
C11 | More complex code = more defects | Y | Y | Y | Y | Y | 1.0 | 0.00
C12 | Quality varies between projects | Y | Y | Y | Y | Y | 1.0 | 0.00
C13 | Assert statements improve code quality | Y | Y | Y | Y | Y | 1.0 | 0.00
C14 | Static analysis improves end user quality | Y | Y | Y | Y | Y | 1.0 | 0.00
C15 | Coding standards improve quality | Y | Y | Y | Y | Y | 1.0 | 0.00
C16 | Code reviews improve quality | Y | Y | Y | Y | Y | 1.0 | 0.00
C17 | Debugging tools improve quality | Y | Y | A | Y | Y | 0.8 | 0.16
✓% | [LLM correctness] | 0.53 | 0.53 | 0.59 | 0.76 | 0.76 | — | —
$\sigma^{2}$ | [LLM variance] | 0.58 | 0.68 | 0.48 | 0.18 | 0.18 | — | —

Finding 1: LLM-based programming assistants agree with most evidence-based
claims, but a large number of responses are ambiguous and noncommittal.

###  4.2. RQ2: LLM-Generated Belief Evidence

To observe the evidence provided to support the beliefs of LLM-based
programming assistants, we inspected the original responses for LLM-generated
beliefs and prompted responses asking how LLMs arrived at their beliefs.

##### Original Responses

All of the original responses from LLM-based programming assistants regarding
their beliefs for the claims observed in this study provided a rationale for
their response. However, the majority of responses were unsubstantiated,
lacking clear supportive evidence ($n=64$, 75.2%). We only observed 21
responses with sources of evidence across three LLMs: Claude ($n=2$), Gemini
($n=2$), and Blackbox AI ($n=17$). For Claude and Gemini, we found generic
references—such as “research has shown that the programming language used can
impact code quality” (C1, Claude) and “studies on this topic have shown mixed
results” (C6, Gemini), without including specific references or details.
Additionally, the response from Gemini for C3 included “Research suggests that
geographically distributed teams can achieve high code quality, but it may
take longer to achieve due to the communication challenges [1]” without
providing a citation for “[1]”. Blackbox AI included direct links to sources
in every response to the research claims. We compiled a list of 162 sources of
evidence provided by Blackbox AI, averaging 10 per response. Blackbox AI
referenced a variety of sources (see Figure 1). Most of the links to evidence
provided ($n=45$, 28.4%) were from blog posts related to various software
development topics. We also noticed a substantial number of links ($n=37$,
22.8%) to relevant posts on Q&A websites such as
Quora111111https://www.quora.com/ ($n=14$), Software Engineering Stack
Exchange121212https://softwareengineering.stackexchange.com/ ($n=13$), and
StackOverflow131313https://stackoverflow.com ($n=9$).

We observed 31 (19.1%) mentions of academic research papers, representing
evidence-based support for queries on SE topics. However, upon further
inspection we observed multiple issues with the provided research evidence.
For instance, there were several references ($n=4$) to the original paper we
base our SE research claims on (Devanbu et al., 2016). Further, we found
duplicated references linking to multiple sources for the same paper. For
example, the paper “On the Impact of Programming Languages on Code Quality”
(Berger et al., 2019) was correctly used as evidence for C1, however Blackbox
AI provided four links to this exact same paper through
HackerNews,141414https://news.ycombinator.com/item?id=20104156
arxiv,151515https://arxiv.org/pdf/1901.10220, one of the authors’ personal
websites,161616http://janvitek.org/pubs/toplas19.pdf, and another unknown
website.171717https://d1b10bmlvqabco.cloudfront.net/attach/k04f77aiv6j1mi/hzn6sgvaeos5dr/k66vt4qdsnvr/2019.pdf
We also observed instances where Blackbox AI summarized specific research
studies (i.e., “A study by Microsoft Research found that the bug density of C#
code was similar to that of Python code” for C6) without providing a relevant
link to the specific paper presenting this study. We also observed one broken
link. Thus, while the ability to provide specific and credible evidence in
Blackbox AI is commendable, future efforts are needed to consistently and
accurately provide evidence supporting responses in LLMs.

$0$$10$$20$$30$$40$$50$Tech ReportCode ExampleQ&AResearch PaperTraining
ResourceSocial MediaBlog$3$$3$$37$$31$$26$$16$$46$

Figure 1. Types of Evidence (RQ2) Provided in LLM-Generated Belief Responses Table 3. Types of Evidence (RQ2) Provided in LLM Responses to “How did you arrive at your answer?” | Research | Reasoning | Training Data | Prior Response
---|---|---|---|---
ChatGPT | 13 | 4 | 16 | 0
Copilot | 0 | 0 | 17 | 0
Gemini | 16 | 15 | 0 | 1
Blackbox AI | 17 | 16 | 0 | 0
Claude | 5 | 14 | 15 | 0
Total: | 51 | 49 | 48 | 1

##### Prompted Response

We also prompted LLMs to understand how they evidenced belief responses. Our
analysis revealed four high-level categories: research ($n=51$), reasoning
($n=49$), training data ($n=48$), and prior response ($n=1$). There results
are presented in Table 3. We observed most LLMs reported using research to
inform their responses. Blackbox AI reported using research to support all 17
research claims. ChatGPT and Gemini also frequently reported using research to
inform responses. For instance, when describing how it arrived to its response
for C1, ChatGPT replied “My response is informed by research and insights
gathered from software engineering literature…Studies and articles often
discuss how language choice impacts software quality, citing examples and
empirical data” without specific citations. For how it derived its answer to
C3, Gemini incorrectly reports “I included a reference to a relevant study to
support the claim that distributed teams can achieve high code quality”, and
added “I did not directly include any URLs in the response, but if you’d like
to see some of the specific resources I used, I can provide some search
queries to help you find them yourself”—indicating the model may not be
capable of providing specific evidence itself. In another instance (C6),
Gemini noted “I reached this answer by considering several factors and
refining my previous response”, basing its answer on a previous
conversation—but the context of the prior response was unclear.

Upon further inspection, we found 36 of the 55 positive LLM-generated
responses to the evidence-based SE claims (65.5%) reportedly used research for
form the answer. In addition, we observed four out of nine negative responses
(44%) and 10 out of 22 ambiguous responses (45.5%) mentioned using research in
responses, indicating LLM-generated responses based on research tended to be
more correct. However, vague mentions of research as well as LLMs using their
own reasoning about software development in practice (i.e., “I recognized the
widespread adoption of coding standards in the software industry, which
suggests that they are seen as an effective way to improve software
quality”—Blackbox AI) and training data (i.e., “I generate responses based on
patterns and information in the data I was trained on”—Copilot) to support
responses indicate a lack of verified evidence in generated output.

Finding 2: Most LLM-based programming assistants lack evidence-based responses
to inquiries about SE claims. Only Blackbox AI provided clear evidence,
predominantly from blogs, Q&A posts, and research papers. Further, responses
that reported using research to derive their answers across LLMs were more
accurate.

###  4.3. RQ3: LLM-Generated Behaviors

Our findings on the behaviors of LLM-based programming assistants are
presented in Table 4. Overall, we found LLMs do not inherently adopt evidence-
based practices ($n=25$, 71.4%), only receiving eight positive responses
(22.9%). Google Gemini was the only system to provide ambiguous responses for
behaviors ($n=2$, 5.7%), claiming it did not know whether the specific
behavior could be adopted or not. For example, in response to C14 Gemini
replied “There’s a good chance Gemini leverages static analysis tools in some
way to improve the quality of generated code for the end user, but we can’t
say for sure without official confirmation from Google”. We provide further
observations on the most and least adopted practices, as well as the LLMs most
capable and incapable of adopting the studied research-based claims.

##### Most Adopted Behavior

We found most LLMs observed in our study can apply coding standards to
generated code. For example, GitHub Copilot responded “I follow a set of
coding standards and best practices to generate high-quality code”. Output
from ChatGPT noted specific coding standards, stating “I follow language-
specific coding standards and guidelines, such as Python’s PEP 8, JavaScript’s
ESLint rules, or Java’s coding conventions, to ensure consistency and
compatibility with the ecosystem”. Claude was the only LLM not able to use
coding standards. Prior work emphasizes the value of coding standards, noting
their ability to reduce errors and improve software quality (Boogerd and
Moonen, 2009; Popić et al., 2018).

##### Least Adopted Behaviors

No LLMs were intrinsically able to adopt code reviews and debugging tools to
improve AI-generated code quality. Code review processes usually involve human
input, as noted in LLM responses stating, “Code reviews are a human-centric
process where developers examine and discuss code for functionality,
efficiency, and adherence to best practices” (Gemini) and “as an AI, I do not
actually have the capability to generate code, conduct code reviews, or
directly participate in software development activities myself” (Claude).
Regarding debugging tools, there were varied responses provided across the
LLMs. Gemini reported its lack of development environment and limited code
execution prevent it from using debuggers “in the same way a programmer
would”. Claude also reported “As an AI assistant without access to an actual
programming environment, I cannot execute, inspect or step through code using
debugging utilities”. Thus, future work is needed to examine how these
processes can be incorporated into LLM-generated output for software
development tasks.

##### Most Capable LLM

We found ChatGPT was the most capable LLM-based programming assistant,
reporting the ability to adopt three out of the seven (42.9%) observed
evidence-based behaviors: commenting code (C5), using assert statements (C13),
and adhering to coding standards (C15). The system also reported being able to
use print-statements for debugging purposes (C17b), which was not considered
an evidence-based practice in this work. Regarding the behavioral prompt for
C5, ChatGPT replied “Yes, I can add code comments to reduce defects in
generated code…By incorporating comments effectively into generated code, I
can help improve its quality, maintainability, and understandability”.
However, as we only inspected output from LLMs to assess their behavior,
further work is needed to investigate if LLM-based programming assistants, in
particular ChatGPT, do in fact adopt these practices while generating code for
users.

##### Least Capable LLM

We observed Claude was not able to adopt any of the evidence-based behaviors
observed in our study. Every response in Claude’s output included a similar
version of the templated response: “As an AI assistant, I do not have the
ability to generate, write, or manage any code directly” and “My role is
limited to providing information to you based on my training data”. Similar
responses occurred across other LLMs, noting an inability to adopt particular
behaviors because they are an AI—however, Claude was the only LLM to always
reply in this manner. As such, Claude was also the most inconsistent regarding
discrepancies between beliefs and behaviors ($n=6$).

Table 4. LLM-Generated Behavior (RQ3) Results

Behavior | Summary | ChatGPT | GitHub Copilot | Gemini | Blackbox AI | Claude | ✓% | $\sigma^{2}$
---|---|---|---|---|---|---|---|---
C5 | Commenting code | Y | Y | N | N |  N† | 0.4 | 0.96
C9 | Unit testing | A |  N† | A |  N† | A | 0.0 | 0.24
C13 | Assert statements | Y |  N† |  N† |  N† |  N† | 0.2 | 0.64
C14 | Static analysis |  N† | A | A | Y |  N† | 0.2 | 0.56
C15 | Coding standards | Y | Y | Y | Y |  N† | 0.8 | 0.64
C16 | Code reviews |  N† |  N† |  N† |  N† |  N† | 0.0 | 0.0
C17a | Debugging tools |  N† |  N† | N |  N† |  N† | 0.0 | 0.0
C17b* | Print-statement debugging* | Y | N | N | N | N | 0.2 | 0.64
✓% | [LLM correctness] | 0.43 | 0.29 | 0.14 | 0.29 | 0.0 | — | —
$\sigma^{2}$ | [LLM variance] | 0.86 | 0.78 | 0.53 | 0.82 | 0.12 | — | —

  * •

$\dagger$ indicates a belief-behavior inconsistency, C17b is not included in
LLM behavior calculations (see Section 3.2.3)

Finding 3: LLM-based programming assistants can adhere to coding standards,
but are unable to adopt most evidence-based practices when generating code for
software development tasks.

##  5\. Implications

AI programming assistants have shown to be effective for generating code,
however our results show that LLM-based programming assistants lack beliefs
and behaviors in accordance with empirical SE research. We found most LLMs
have ambiguous beliefs about research findings, lack evidence to support their
claims, and are unable to adopt evidence-based behaviors to support software
development in practice. Based on these findings, we provide implications for
software developers adopting LLM-based programming assistants in their
development workflows.

##### LLMs have mostly positive perceptions of evidence-based practices, but
can be noncommittal.

We observed the majority of responses from LLMs agreed with the SE research
claims in our study (see Table 2). Over half of the LLM-generated beliefs were
positive, with unanimous agreement with the findings more complex code results
in more defects (C11), factors affecting code quality vary between projects
(C12), static analysis tools improve end user quality (C14), coding standards
improve software quality (C15), and code reviews improve software quality
(C16). However, developers utilizing these systems for SE tasks should note
they can be noncommittal on important development concepts. For instance, two
LLM-based programming assistants provided ambiguous responses on the claim
that well-commented code has fewer defects (C5), stating “this correlation is
not absolute” (ChatGPT) and “components with more unit tests do not
necessarily have fewer customer-found defects” (Blackbox AI). Thus additional
measures should be taken to mitigate ambiguous responses from LLMs. For
example, prior work suggests vague responses from ChatGPT can be overcome by
refining prompts, increased understanding of the model’s reasoning, providing
feedback and examples, and using external tools or plugins to enhance
communication with ChatGPT (jen, 2023).

##### LLM-generated responses lack credible evidence

Moreover, for claims that LLM-based programming assistants do agree with, the
large majority do not provide evidence to support their beliefs. We found most
LLMs claim to use research, but do not provide specific research findings
which they base their responses on. Further, LLMs frequently use their own
reasoning and training data to inform responses (see Table 3), but these are
often unknown to users and can lead to unsubstantiated and misleading
responses (jen, 2023; bla, [n. d.]). For instance, every prompted response
from GitHub Copilot indicated its responses are generated based on the data it
was trained on. Only one LLM-based programming assistant (Blackbox AI)
consistently provided evidence to support claims made by the generative AI
model—but, these were largely from non-scientific sources (see Figure 1).
While the evidential support for responses in Blackbox AI is a valuable
feature, we observed several complications including duplicated references,
missing citations, and broken links. The evidence provided by Blackbox AI and
Claude could contribute to their accuracy in LLM-generated beliefs. However,
the discovered issues and lack of evidence for other LLMs assistants should
caution practitioners using their output to support development tasks and
motivate future work to enhance evidenced-based responses from generative AI
in SE contexts.

##### LLMs can adhere to coding standards

We found most LLM-based programming assistants observed in our study can apply
coding standards (C15) to generated code. This aligns with prior work
suggesting AI is effective for teaching elegant coding styles to students
(Woodrow et al., 2024). For instance, responses included “By incorporating
coding standards into the generation process, I strive to produce high-quality
code that meets industry best practices and standards, ultimately improving
the overall software quality and developer experience” (ChatGPT), “As an AI
developed by OpenAI, I follow a set of coding standards and best practices to
generate high-quality code” (Copilot), “By being influenced by coding
standards, Gemini can generate code that is more consistent, easier to
understand, and potentially less error-prone” (Gemini), and “Yes, Blackbox AI
uses coding standards to improve the software quality of generated
code…Blackbox AI’s use of coding standards is essential in improving the
software quality of generated code” (Blackbox AI). In addition, the LLMs we
observed also unanimously agreed that using coding standards is a beneficial
practice for improving software quality. Thus, while we observed inconsistent
adoption of behaviors across the LLMs observed in our study, our findings show
they can be useful for formatting generated code to adhere to specific coding
standards.

##### LLMs cannot adopt evidence-based behaviors, but can be used to support
them

LLM-based programming assistants are increasingly used in software development
activities (Russo, 2024). However, our findings indicate these systems are not
able to adopt many practices supported by empirical SE research—even when they
perceive these practices as beneficial to software development (see Table 4).
While these systems cannot adopt evidence-based practices, many generative AI
models reported being able to assist with useful practices supported by
empirical SE research. For example, ChatGPT does not use static analysis tools
to assess generated code (C14), but can “help you understand how they work and
how you can use them to improve the quality of generated code”. Similarly,
multiple LLMs reported they cannot add comments to generated code (C5), but
stated, “If you’d like to explore ways to improve the readability of your
generated code, I can provide some tips on best practices for code structure
and naming conventions” (Gemini) and “While Blackbox AI can generate code, it
does not directly add code comments to reduce defects in the generated code.
However, Blackbox AI can provide suggestions for comments based on the code’s
functionality and structure” (Blackbox AI). Similarly, while LLMs cannot
actively conduct code reviews and use debugging tools, all five reported
having the ability to assist developers in completing code review and
debugging tasks in some capacity. Thus, LLM-based programming assistants are
capable of providing insight and resources to help developers improve
evidence-based practices.

##### LLMs cannot adopt evidence-based behaviors, but can be instructed to do
so

In our approach, we used zero-shot prompting with no examples or
clarifications provided to understand the beliefs and behaviors of LLM-based
programming assistants. However, the output of some tools reports
incorporating specific instructions and details into input, or finetuning
instructions, can enhance the abilities of LLMs to incorporate empirically-
supported practices. For example, in response to several behaviors (C5, C9,
and C13), GitHub Copilot responded that it does not inherently apply these
practices to generated code, yet added “However, I can certainly [include
comments, help create unit tests, include asserts] if instructed to do so”.
Similarly, Gemini replied “While Gemini itself might not use asserts directly,
it can generate code that facilitates their use during the development
process” (C13) and ChatGPT noted “If you have specific coding standards or
guidelines you’d like me to adhere to, feel free to specify them, and I’ll
ensure that the generated code complies with them” (C15). Thus, providing
specific instructions for LLMs to incorporate useful behaviors in prompts can
improve the behavior of these systems, enhancing the credibility of the
generated output.

##### LLM-generated beliefs are similar to human beliefs

The evidence-based claims used for our study were derived from previous work
exploring the beliefs of human software engineers at Microsoft (Devanbu et
al., 2016). We do not directly compare human and LLM beliefs due to
differences in the rating scales used (Y/N/A vs. 5-point scale). However, at a
glance we observe similar trends in the beliefs of LLMs and humans. For
instance, the four least controversial beliefs for humans—code reviews (C16),
coding standards (C15), static analysis tools (C14), and assert statements
(C13)—were all unanimously agreed on by the LLMs observed in our study.
Likewise, four of the claims with the most disparity in disagreement between
humans—code quality depends on language (C1), fixing defects is riskier than
adding new features (C2), and specific project experience matters more than
general programming experience (C4)—also were the most varied among the LLM-
based programming assistants. Thus, this motivates the need for future
research to enhance evidence-based software development beliefs and practices
for LLMs and human software engineers.

##  6\. Research Directions

LLMs have transformed software development and will continue to innovate the
development and maintenance of software as adoption increases among developers
(Russo, 2024) and generative AI is integrated into a growing number of
applications to support SE including development environments (i.e., JetBrains
AI (jet, [n. d.]) and GitHub Copilot (cop, [n. d.])), online coding-focused
question-answering platforms (i.e., Overflow AI (soa, 2023) within Stack
Overflow181818https://stackoverflow.com/), SE education and courses (Kirova et
al., 2024), and developer communication platforms (i.e., Slack AI (sla, 2024)
within Slack191919https://slack.com/). The results from our preliminary
evaluation show LLMs do not adhere to evidence-based SE practices. Based on
these findings, we provide avenues for future research to increase the
adoption of evidence-based practices in LLM-based workflows and enhance the
beliefs and behaviors of software developers interacting with LLMs to complete
SE tasks.

##### Evidence-Based LLMs

Our results show that LLMs lack credible evidence to back their claims. To
mitigate this, future work can explore enhancing LLMs on evidence-based
practices. LLM-based chatbots have been implemented to provide responses
backed by scientific evidence to natural language questions in other domains,
such as medicine (e.g., Med-Palm (med, [n. d.])), law (e.g., Rocket Matter
(roc, [n. d.])), writing (e.g., (jen, [n. d.])), and science (e.g.,
MirrorThink (mir, [n. d.])). For example, Med-PaLM 2 has been used to power a
live question-answering system for medical inquiries about liver cancer (Qian
et al., 2024). Specialized LLMs have also been developed for technical fields,
such as cryptocurrency (e.g., ChainGPT (cha, [n. d.])) and software
development (e.g., Devin (Wu, 2024)). To promote evidence-based SE, future
LLM-based programming assistants could incorporate empirical SE research
findings in their training data to provide responses based on scholarly
publications. In addition, training data could incorporate code bases shown to
be high-quality and developed using evidence-based tools and practices. This
could increase the reliability of training data, ideally leading to improved
output generated for models. For instance, providing evidence-based responses
to software developers’ queries about practical SE topics.

##### Evidence-Based Practices in LLM Development Workflows

In our study, we observed that most LLM-based programming assistants do not
adopt development practices supported by empirical SE research and conclude
developers should use LLMs in conjunction with evidence-based practices.
Future work can investigate techniques to enhance this collaboration. For
instance, a layered workflow architecture can be used to apply evidence-based
practices on LLM-generated code before the output is presented to users.
Organizing a software system into multiple layers is one of the earliest and
most popular design architectures (Savolainen and Myllarniemi, 2009). In this
context, layered workflow architectures incorporate multiple levels or stages
of execution for a workflow, or predefined set of work steps, to help
organize, control, and execute activities (Kim and Ellis, 2001). Prior work
has proposed using multiple layers to improve other aspects of software
development, such as CI/CD process quality measurements (Lai et al., 2022) and
security with AI/ML in DevSecOps pipelines (Camacho, 2024). A potential
example for integrating this concept in an LLM-based workflow could be as
follows:

  1. (1)

A programmer prompts an LLM-based programming assistant to provide code for a
programming task;

  2. (2)

the LLM generates code based on the input prompt;

  3. (3)

the output is redirected to a different model or system which applies
empirically supported SE practices on the generated code (e.g., running
FindBugs (Ayewah et al., 2008) on LLM-generated Java code); and

  4. (4)

the generated code and resulting static analysis report is then provided to
the programmer to inspect and use in their project. Or preferentially, the
automated program repair capabilities of LLMs (Xia et al., 2023) can be
leveraged to automatically fix reported bugs before presenting the generated
code snippet to users.

Recent tools, such as AutoGen (aut, [n. d.]), provide multi-agent
collaborations for customized workflows. As usage of LLMs increases in
software development, future work can explore the development of tools and
infrastructure to augment current LLM-based development workflows by
incorporating research tools and evidence-based practices on generated code.
Studies can also further evaluate the effects of a comparable approach on the
quality and maintainability of the larger code base and the behavior of
software developers.

##### LLMs for Translating Research Evidence

Empirical SE research aims to support effective and efficient software
development. However, the translation of these findings into practice is
particularly challenging. One issue is that research output—typically provided
through academic papers—is inaccessible to practitioners (Wilson et al.,
2024). For instance, journal and conference papers may be financially
burdensome or complicated to access; and once an article is obtained, the
research itself can be difficult to comprehend (Dunlop, 2019). Thus, future
work can explore using LLMs to automatically summarize and distribute research
findings to practitioners to mitigate this challenge. Prior work suggests LLMs
are effective for summarizing natural language text (Jin et al., 2024) and
code (Ahmed and Devanbu, 2022). In our evaluation, our findings show zero-shot
prompting of LLMs regarding evidence-based practices often provides
noncommittal and unsupported responses. However, we also observed examples
where LLMs were able to cite evidence and succinctly summarize research
output. For instance, we observed the following responses from Blackbox AI
summarizing research to support its belief about experienced programmers
producing fewer defects (C10):

> “A study by IBM found that experienced developers were 2.5 times more
> productive and had 50% fewer defects than less experienced developers.
> A study by Microsoft found that developers with 5-10 years of experience had
> a 30% lower defect rate than those with 0-5 years of experience.
> A study by the IEEE found that experienced developers were 3.5 times more
> likely to write defect-free code than less experienced developers.”

Horvitz et al. suggest one of the top ten priorities for the future of AI is
to power resources to bridge the academia-industry gap (Horvitz et al., [n.
d.]). Given this, we suggest future work explore the potential for LLM-
generated summaries and the role they can play in supporting the behavior of
software practitioners. Summarizing research papers into accessible content
can enhance developers’ understanding of evidence-based without needing to
read and understand full conference or journal papers. Prior work has explored
human summaries of research findings, finding this approach ineffective
(Wilson et al., 2024). However, leveraging LLMs can mitigate the challenges
with human-written summaries, including automatically generating summaries,
customizing summaries to specific users, and incorporating content within
development workflows.

##  7\. Related Work

We discuss existing literature exploring evidence-based SE beliefs and
behaviors, capabilities and limitations of LLMs in SE, and integrating LLMs
into empirical SE research contexts as related work.

###  7.1. Beliefs and Behaviors in SE

Empirical SE is is a subset of SE research that leverages empirical methods,
based on observed and measured phenomena derived from experience as opposed to
theory (Cahoy et al., 2024), in software development contexts (Shull et al.,
2007). Many studies demonstrate behaviors supported empirical research
findings are not adopted in practice—despite scientific evidence supporting
the advantages of these practices (i.e., static analysis (Johnson et al.,
2013), automated debugging (Parnin and Orso, 2011), refactoring (Pinto and
Kamei, 2013), and security (Witschey et al., 2015) tool usage). Kitchenham et
al. suggest software can benefit from evidence-based software engineering and
offer steps to promote evidence-based practices in SE similar to the medical
field (Kitchenham et al., 2004). SE researchers have further investigated this
research-practice gap (Ivanov et al., 2017) to contextualize issues preventing
the adoption of evidence-based SE behaviors in industry. Lo et al. show that
most developers find SE research as irrelevant and not essential, noting
issues with unnecessary tools, lack of actionable and generalizable insights,
and costs related to adoption (Lo et al., 2015). Prior work also suggests the
majority of SE research is not applicable in real-world software development
environments (Winters, 2024) and require better understanding and enhanced
research contexts (Murphy-Hill et al., 2010) to increase their impact in
practice.

In the research most closely related to ours, prior work has explored the
beliefs of software practitioners regarding empirical SE claims. Devanbu and
colleagues highlight varied beliefs among human programmers when assessing the
empirical SE claims used in this study (Devanbu et al., 2016). Similarly,
Smith and colleagues surveyed software practitioners to understand if SE
beliefs vary based on personality (Smith et al., 2016). Shrikanth et al. found
no clear support of programmers’ beliefs with regard to research find
regarding software quality and productivity (Shrikanth et al., 2021). Finally,
Passos et al. show that strong beliefs held by developers based on personal
experiences impact software development practices (Passos et al., 2011). Our
study extends this line of work by investigating the beliefs and behaviors of
LLM-based programming assistants—which are increasingly used in conjunction
with human programmers in software development contexts (JetBrains, 2023), and
have been shown to impact the behavior and decision-making of software
engineers (Russo, 2024; Vereschak et al., 2021)—regarding empirical SE
research findings shown to be beneficial for completing development tasks.

###  7.2. LLMs in SE

There is an emerging research interest in the affects of LLMs and AI-driven
tools in software development. For instance, these innovations are receiving
increased interest in the SE research community. Showing effective
capabilities for code generation (Xu et al., 2022), LLMs have now seen
expanded usage in software development beyond suggesting code to programmers
(Khojah et al., 2024). Zheng et al. outlined 123 research papers exploring
LLMs across SE tasks, including code generation, vulnerability detection, code
evaluation, and Q&A interactions (Zheng et al., 2023). LLMs have been shown to
be able to support SE tasks related to various facets of the software
development lifecycle, such as code quality, refactoring, requirements, and
design (White et al., 2023), in addition to other development tasks, such as
automatically fixing GitHub issues (Yang et al., 2024).

Research has also explored limitations of LLMs in software development. For
instance, Wang et al. suggest developers find ChatGPT excels at performing
simple and general-purpose coding tasks, but lacks the ability to support
other development activities (Wang et al., 2024). Borji et al. broadly
categorized failures of ChatGPT, including limitations in generating code
(Borji, 2023). Khoury et al. show that, despite being aware of potential
vulnerabilities and being prompted with specific instructions, ChatGPT still
produced code vulnerable to specific attacks (Khoury et al., 2023). Imai
posits that, while Copilot is effective for increasing programming
productivity (i.e., increased lines of code added), it contributes to lower
quality of code (i.e., increased lines of code subsequently deleted) (Imai,
2022). Spiess et al. demonstrate machine learning models in SE can be wrong,
and enhance these systems applying calibration methods to LLM-based
programming assistants (Spiess et al., 2024). Further, Liang and colleagues
explored the usability of LLM-based programming assistants, characterizing
issues preventing programmers from adopting AI tools and suggested code (Liang
et al., 2024b). Our study uncovers deficiencies in LLM-based development
workflows regarding their empirical SE research beliefs, sources of evidence,
and evidence-based practices.

###  7.3. LLMs in Research

Prior work has also investigated the capabilities of LLMs in scientific
contexts. For instance, Auer and colleagues show ChatGPT struggles to
correctly answer research-based questions from question answering benchmarks
from varying domains—including computer science (Auer et al., 2023).
Researchers have explored the capabilities of LLMs for generating data for
human-computer interaction research through replicating human behavior in
computational tasks (Wu et al., 2023; Tavast et al., 2022; Hämäläinen et al.,
2023). However, Atari et al. show that responses from LLMs primarily reflect
Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies and
values, failing to incorporate cross-cultural diversity (Atari et al., 2023).
Additional LLM-based systems (i.e., MirrorThink (mir, [n. d.]) and Research
Kick (kic, [n. d.])), leverage GPT-4 to provide responses to scientific
questions posed by researchers. Similar approaches have also been implemented
in tools to support question-answering and beyond in legal (roc, [n. d.];
Sydorenko, 2023) and medical (med, [n. d.]; Clusmann et al., 2023) domains.

LLMs have further been applied to empirical SE research contexts. For
instance, Liang et al. explored the extent to which GPT-4 can replicate
empirical software engineering research findings—noting the generative AI
model’s ability to provide correct assumptions but failure to apply this
knowledge (Liang et al., 2024a). Furthermore, Storey and colleagues also
introduce MyResearchPlaybook, a platform powered by GPT to support empirical
SE research in the midst of disruptive innovations—such as generative AI in
software development (Storey et al., 2024). Prior work has also explored using
LLMs to provide evidence-based responses to practical SE-related questions
through ResearchBot, a GPT-powered chatbot (Farzanehpour et al., 2024). This
project extends this work to understand how LLM-based programming assistants
perceive findings supported by empirical SE research and motivate future
research to enhance the evidence-based beliefs and behaviors of LLMs.

##  8\. Conclusion

Software engineering is a complex activity that relies on practitioners to
develop, maintain, and innovate software-based systems. To support software
development, empirical research has offered a variety of tools and practices
supported by scientific evidence to enhance the development and maintenance of
software. More recently, large language models (LLMs) have transformed the
development and maintenance of software—receiving rapid attention and adoption
for their ability to automatically support a wide variety of SE-related tasks.
However, little is known about these AI-driven systems with regard to
evidence-based practices purported by empirical software engineering research.
This project analyzes the beliefs and behaviors of popular LLM-based
programming assistants regarding empirically validated research claims. Our
findings show that LLM-generated beliefs are ambiguous, not providing a clear
stance on the benefits or detriment of evidence-based practices. Moreover, we
show LLM-based programming assistants are unable to adopt useful practices
supported by research into code generation and analysis tasks in software
development. In light of these findings, this work provides implications for
software engineers using LLM-based programming assistants in software
development contexts and provides future research directions to enhance the
beliefs and behaviors of evidence-based practices in software engineering.

## References

  * (1)
  * aut ([n. d.]) [n. d.].  AutoGen.  https://www.microsoft.com/en-us/research/project/autogen/.
  * bla ([n. d.]) [n. d.].  Blackbox AI.  https://www.blackbox.ai/.
  * cha ([n. d.]) [n. d.].  ChaintGPT.  https://app.chaingpt.org/.
  * gpt ([n. d.]) [n. d.].  ChatGPT.  https://chat.openai.com.
  * cla ([n. d.]) [n. d.].  Claude.  https://claude.ai/chat/.
  * gem ([n. d.]) [n. d.].  Gemini.  https://gemini.google.com.
  * cop ([n. d.]) [n. d.].  GitHub Copilot.  https://github.com/features/copilot.
  * jen ([n. d.]) [n. d.].  Jenni AI.  https://jenni.ai/.
  * jet ([n. d.]) [n. d.].  JetBrains AI.  https://www.jetbrains.com/ai/.
  * med ([n. d.]) [n. d.].  Med-PaLM.  https://sites.research.google/med-palm/.
  * mir ([n. d.]) [n. d.].  MirrorThink.  https://mirrorthink.ai/.
  * kic ([n. d.]) [n. d.].  Research Kick.  https://www.researchkick.com/.
  * roc ([n. d.]) [n. d.].  Rocket Matter.  https://www.rocketmatter.com/chatgpt/.
  * wat ([n. d.]) [n. d.].  watsonx.  https://www.ibm.com/products/watsonx-code-assistant.
  * soa (2023) 2023\.  Announcing OverflowAI.  _Stack Overflow Blog_ (2023).  https://stackoverflow.blog/2023/07/27/announcing-overflowai/.
  * jen (2023) 2023\.  Understanding ChatGPT’s Vagueness: Why It Happens & How to Overcome It.  _Jenni AI_ (2023).  https://jenni.ai/chat-gpt/vague-answers-errors.
  * sla (2024) 2024\.  Slack AI has arrived.  _Slack Blog_ (2024).  https://slack.com/blog/news/slack-ai-has-arrived.
  * Ahmad et al. (2023) Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023.  Towards human-bot collaborative software architecting with chatgpt. In _Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering_. 279–285.
  * Ahmed (2024) Firas Ahmed. 2024.  How to Use Claude AI – Introduction to Claude AI + Code Example.  _freeCodeCamp_ (2024).  https://www.freecodecamp.org/news/introduction-to-claude-ai/.
  * Ahmed and Devanbu (2022) Toufique Ahmed and Premkumar Devanbu. 2022.  Few-shot training llms for project-specific code-summarization. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–5.
  * AlOmar et al. (2024) Eman Abdullah AlOmar, Anushkrishna Venkatakrishnan, Mohamed Wiem Mkaouer, Christian D Newman, and Ali Ouni. 2024.  How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations.  _arXiv preprint arXiv:2402.06013_ (2024).
  * Aniche et al. (2013) Mauricio Finavaro Aniche, Gustavo Ansaldi Oliva, and Marco Aurélio Gerosa. 2013.  What do the asserts in a unit test tell us about code quality? a study on open source and industrial projects. In _2013 17th European Conference on Software Maintenance and Reengineering_. IEEE, 111–120.
  * Atari et al. (2023) Mohammad Atari, Mona J Xue, Peter S Park, Damián E Blasi, and Joseph Henrich. 2023.  Which Humans?  https://doi.org/10.31234/osf.io/5b26t
  * Auer et al. (2023) Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. 2023\.  The sciqa scientific question answering benchmark for scholarly knowledge.  _Scientific Reports_ 13, 1 (2023), 7240.
  * Ayewah and Pugh (2010) Nathaniel Ayewah and William Pugh. 2010.  The google findbugs fixit. In _Proceedings of the 19th international symposium on Software testing and analysis_. ACM, 241–252.
  * Ayewah et al. (2008) Nathaniel Ayewah, William Pugh, David Hovemeyer, J David Morgenthaler, and John Penix. 2008.  Using static analysis to find bugs.  _IEEE software_ 25, 5 (2008), 22–29.
  * Baudry et al. (2001) Benoit Baudry, Yves Le Traon, and J-M Jézéquel. 2001.  Robustness and diagnosability of OO systems designed by contracts. In _Proceedings Seventh International Software Metrics Symposium_. IEEE, 272–284.
  * Beller et al. (2018) Moritz Beller, Niels Spruit, Diomidis Spinellis, and Andy Zaidman. 2018.  On the dichotomy of debugging behavior among programmers. In _Proceedings of the 40th International Conference on Software Engineering_. 572–583.
  * Belzner et al. (2023) Lenz Belzner, Thomas Gabor, and Martin Wirsing. 2023.  Large language model assisted software engineering: prospects, challenges, and a case study. In _International Conference on Bridging the Gap between AI and Reality_. Springer, 355–374.
  * Berger et al. (2019) Emery D. Berger, Celeste Hollenbeck, Petr Maj, Olga Vitek, and Jan Vitek. 2019.  On the Impact of Programming Languages on Code Quality: A Reproduction Study.  _ACM Trans. Program. Lang. Syst._ 41, 4, Article 21 (oct 2019), 24 pages.  https://doi.org/10.1145/3340571
  * Bessey et al. (2010) Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010.  A few billion lines of code later: using static analysis to find bugs in the real world.  _Commun. ACM_ 53, 2 (2010), 66–75.
  * Bird et al. (2023) Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023.  Taking flight with copilot.  _Commun. ACM_ 66, 6 (2023), 56–62.
  * Bird et al. (2009) Christian Bird, Nachiappan Nagappan, Premkumar Devanbu, Harald Gall, and Brendan Murphy. 2009.  Does distributed development affect software quality? an empirical case study of windows vista.  _Commun. ACM_ 52, 8 (2009), 85–93.
  * Bird et al. (2011) Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011.  Don’t touch my code! Examining the effects of ownership on software quality. In _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_. 4–14.
  * Boogerd and Moonen (2009) Cathal Boogerd and Leon Moonen. 2009.  Evaluating the relation between coding standard violations and faultswithin and across software versions. In _2009 6th IEEE International Working Conference on Mining Software Repositories_. IEEE, 41–50.
  * Borji (2023) Ali Borji. 2023.  A categorical archive of chatgpt failures.  _arXiv preprint arXiv:2302.03494_ (2023).
  * Braun and Clarke (2012) Virginia Braun and Victoria Clarke. 2012.  _Thematic analysis._ American Psychological Association.
  * Brown and Cusati (2024) Chris Brown and Jason Cusati. 2024.  Supplemental Materials for “Exploring Evidence-Based Beliefs and Behaviors of LLM-Based Programming Assistants”.  https://github.com/code-world-no-blanket/LLM-beliefs.
  * Cahoy et al. (2024) Ellysa Cahoy, Stephen Woods, and Bernadette Lear. 2024.  Empirical Research in the Social Sciences and Education.  _Penn State University Libraries_ (2024).
  * Camacho (2024) Nicolas Guzman Camacho. 2024.  Unlocking the Potential of AI/ML in DevSecOps: Effective Strategies and Optimal Practices.  _Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023_ 3, 1 (2024), 106–115.
  * Clusmann et al. (2023) Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. 2023\.  The future landscape of large language models in medicine.  _Communications medicine_ 3, 1 (2023), 141.
  * Da Silva et al. (2024) Leuson Da Silva, Jordan Samhi, and Foutse Khomh. 2024.  ChatGPT vs LLaMA: Impact, Reliability, and Challenges in Stack Overflow Discussions.  _arXiv preprint arXiv:2402.08801_ (2024).
  * DAIR.AI (2024) DAIR.AI. 2024.  Zero-Shot Prompting.  _Prompt Engineering Guide_ (2024).  https://www.promptingguide.ai/techniques/zeroshot.
  * DePalma et al. (2024) Kayla DePalma, Izabel Miminoshvili, Chiara Henselder, Kate Moss, and Eman Abdullah AlOmar. 2024.  Exploring ChatGPT’s code refactoring capabilities: An empirical study.  _Expert Systems with Applications_ 249 (2024), 123602.
  * Devanbu et al. (2016) Prem Devanbu, Thomas Zimmermann, and Christian Bird. 2016.  Belief & evidence in empirical software engineering. In _Proceedings of the 38th international conference on software engineering_. 108–119.
  * Dunlop (2019) Patrick Dunlop. 2019.  How to access academic research when you’re not affiliated with a university.  _Linkedin Pulse_ (2019).  https://www.linkedin.com/pulse/how-access-academic-research-when-youre-affiliated-patrick-dunlop/.
  * Farzanehpour et al. (2024) Sahar Farzanehpour, Swetha Rajeev, Huayu Liang, Ritvik Prabhu, and Chris Brown. 2024.  ResearchBot: Bridging the Gap between Academic Research and Practical Programming Communities.  arXiv:2407.02643 [cs.SE] https://arxiv.org/abs/2407.02643
  * Guilherme and Vincenzi (2023) Vitor Guilherme and Auri Vincenzi. 2023.  An initial investigation of ChatGPT unit test generation capability. In _Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing_. 15–24.
  * Hadley (1991) Robert F Hadley. 1991.  The many uses of ‘belief’in AI.  _Minds and Machines_ 1 (1991), 55–73.
  * Hämäläinen et al. (2023) Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023.  Evaluating large language models in generating synthetic hci research data: a case study. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Hanenberg (2010) Stefan Hanenberg. 2010.  An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In _Proceedings of the ACM international conference on Object oriented programming systems languages and applications_. 22–35.
  * Horvitz et al. ([n. d.]) Eric Horvitz, Vincent Conitzer, Sheila McIlraith, and Peter Stone. [n. d.].  Now, Later, and Lasting: 10 Priorities for AI Research, Policy, and Practice.  _Commun. ACM_ ([n. d.]).
  * Hou et al. (2023) Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023.  Large language models for software engineering: A systematic literature review.  _arXiv preprint arXiv:2308.10620_ (2023).
  * Hu et al. (2023) Rui Hu, Junhao Zhong, Minjie Ding, Zeyu Ma, and Mingang Chen. 2023.  Evaluation of Hallucination and Robustness for Large Language Models. In _2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)_. IEEE, 374–382.
  * Imai (2022) Saki Imai. 2022.  Is github copilot a substitute for human pair-programming? an empirical study. In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_. 319–321.
  * Imtiaz et al. (2019) Nasif Imtiaz, Akond Rahman, Effat Farhana, and Laurie Williams. 2019.  Challenges with responding to static analysis tool alerts. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 245–249.
  * Ivanov et al. (2017) Vladimir Ivanov, Alan Rogers, Giancarlo Succi, Jooyong Yi, and Vasilii Zorin. 2017.  What do software engineers care about? gaps between research and practice. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. 890–895.
  * Janzen (2005) David S Janzen. 2005.  Software architecture improvement through test-driven development. In _Companion to the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications_. 240–241.
  * JetBrains (2023) JetBrains. 2023.  The State of Developer Ecosystem 2023.  _Developer Ecosystem_ (2023).
  * Jin et al. (2024) Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. 2024.  A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods.  _arXiv preprint arXiv:2403.02901_ (2024).
  * Johnson et al. (2013) Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013.  Why Don’t Software Developers Use Static Analysis Tools to Find Bugs?. 2. Conference. In _Proceedings of the 2013 International Conference on Software Engineering (ICSE)_ (San Francisco, CA, USA) _(ICSE ’13)_. IEEE Press, Piscataway, NJ, USA, 672–681.  https://doi.org/10.1109/ICSE.2013.6606613
  * Kadam and Vaidya (2020) Suvarna Kadam and Vinay Vaidya. 2020.  Review and analysis of zero, one and few shot learning approaches. In _Intelligent Systems Design and Applications: 18th International Conference on Intelligent Systems Design and Applications (ISDA 2018) held in Vellore, India, December 6-8, 2018, Volume 1_. Springer, 100–112.
  * Kalliamvakou (2022) Eirini Kalliamvakou. 2022.  Research: quantifying GitHub Copilot’s impact on developer productivity and happiness.  _GitHub Blog_ (2022).  https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/.
  * Kammerath (2024) Jan Kammerath. 2024.  The AI Dilemma: When Large Language Model Training Reaches A Dead End.  _Medium_ (2024).  https://medium.com/@jankammerath/the-ai-dilemma-when-large-language-model-training-reaches-a-dead-end-e2cf1de4a2ad.
  * Kaplan (2020) Daniel Kaplan. 2020.  A Compact Guide to Classical Inference.  (2020).  https://dtkaplan.github.io/CompactInference/.
  * Kemerer (1995) Chris F Kemerer. 1995.  Software complexity and software maintenance: A survey of empirical research.  _Annals of Software Engineering_ 1 (1995), 1–22.
  * Khojah et al. (2024) Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024.  Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.  _arXiv preprint arXiv:2404.14901_ (2024).
  * Khoury et al. (2023) Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023.  How secure is code generated by chatgpt?. In _2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2445–2451.
  * Kim and Ellis (2001) Kwang-Hoon Kim and Clarence A Ellis. 2001.  Workflow performance and scalability analysis using the layered queuing modeling methodology. In _Proceedings of the 2001 ACM International Conference on Supporting Group Work_. 135–143.
  * Kirova et al. (2024) Vassilka D Kirova, Cyril S Ku, Joseph R Laracy, and Thomas J Marlowe. 2024.  Software engineering education must adapt and evolve for an llm environment. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_. 666–672.
  * Kitchenham et al. (2004) Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004.  Evidence-based software engineering. In _Proceedings. 26th International Conference on Software Engineering_. IEEE, 273–281.
  * Kitchenham et al. (2002) Barbara A Kitchenham, Shari Lawrence Pfleeger, Lesley M Pickard, Peter W Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002.  Preliminary guidelines for empirical research in software engineering.  _IEEE Transactions on software engineering_ 28, 8 (2002), 721–734.
  * Ko and Myers (2004) Amy J Ko and Brad A Myers. 2004.  Designing the whyline: a debugging interface for asking questions about program behavior. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 151–158.
  * Ko et al. (2023) Minhyuk Ko, Dibyendu Brinto Bose, Hemayet Ahmed Chowdhury, Mohammed Seyam, and Chris Brown. 2023.  Exploring the Barriers and Factors that Influence Debugger Usage for Students. 2. Conference. In _Visual Languages and Human-Centric Computing_.
  * Kocaguneli et al. (2013) Ekrem Kocaguneli, Thomas Zimmermann, Christian Bird, Nachiappan Nagappan, and Tim Menzies. 2013.  Distributed development considered harmful?. In _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 882–890.
  * Lai et al. (2022) Sen-Tarng Lai, Heru Susanto, and Fang-Yie Leu. 2022.  Combining Pipeline Quality with Automation to CI/CD Process for Improving Quality and Efficiency of Software Maintenance. In _Innovative Mobile and Internet Services in Ubiquitous Computing: Proceedings of the 15th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS-2021)_. Springer, 362–372.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ (1977), 159–174.
  * Layman et al. (2007) Lucas Layman, Laurie Williams, and Robert St Amant. 2007.  Toward reducing fault fix time: Understanding developer behavior for the design of automated fault detection tools. In _Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on_. IEEE, 176–185.
  * Leng and Yuan (2023) Yan Leng and Yuan Yuan. 2023.  Do LLM Agents Exhibit Social Behavior?  _arXiv preprint arXiv:2312.15198_ (2023).
  * Liang et al. (2024a) Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. 2024a.  Can GPT-4 Replicate Empirical Software Engineering Research?  arXiv:2310.01727 [cs.SE]
  * Liang et al. (2024b) Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024b.  A large-scale survey on the usability of ai programming assistants: Successes and challenges. In _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_. 1–13.
  * Lieberman (1997) Henry Lieberman. 1997.  The debugging scandal and what to do about it.  _Commun. ACM_ 40, 4 (1997), 26–30.
  * Liu et al. (2024) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Lo et al. (2015) David Lo, Nachiappan Nagappan, and Thomas Zimmermann. 2015.  How practitioners perceive the relevance of software engineering research. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_. 415–425.
  * Marcilio et al. (2019) Diego Marcilio, Rodrigo Bonifácio, Eduardo Monteiro, Edna Canedo, Welder Luz, and Gustavo Pinto. 2019.  Are static analysis violations really fixed? a closer look at realistic usage of sonarqube. In _2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)_. IEEE, 209–219.
  * Mockus and Weiss (2000) Audris Mockus and David M Weiss. 2000.  Predicting risk of software changes.  _Bell Labs Technical Journal_ 5, 2 (2000), 169–180.
  * Morris (2024) Chris Morris. 2024.  ChatGPT and Google’s Gemini will now remember your past conversations.  _Fast Company_ (2024).  https://www.fastcompany.com/91029395/chatgpt-google-gemini-remember-past-conversations.
  * Murphy-Hill et al. (2010) Emerson Murphy-Hill, Gail C Murphy, and William G Griswold. 2010.  Understanding context: creating a lasting impact in experimental software engineering research. In _Proceedings of the FSE/SDP workshop on Future of software engineering research_. 255–258.
  * Nam et al. (2024) Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024.  Using an llm to help with code understanding. In _Proceedings of the IEEE/ACM 46th International Conference on Software Engineering_. 1–13.
  * Ouyang et al. (2023) Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023.  LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation.  _arXiv preprint arXiv:2308.02828_ (2023).
  * Ozkaya (2023) Ipek Ozkaya. 2023.  Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications.  , 4-8 pages.  https://doi.org/10.1109/MS.2023.3248401
  * Parnin and Orso (2011) Chris Parnin and Alessandro Orso. 2011.  Are automated debugging techniques actually helping programmers?. In _Proceedings of the 2011 international symposium on software testing and analysis_. 199–209.
  * Passos et al. (2011) Carol Passos, Ana Paula Braun, Daniela S Cruzes, and Manoel Mendonca. 2011.  Analyzing the impact of beliefs in software project practices. In _2011 International Symposium on Empirical Software Engineering and Measurement_. IEEE, 444–452.
  * Pinto and Kamei (2013) Gustavo H Pinto and Fernando Kamei. 2013.  What programmers say about refactoring tools? an empirical investigation of stack overflow. In _Proceedings of the 2013 ACM workshop on Workshop on refactoring tools_. 33–36.
  * Popić et al. (2018) Srdan Popić, Gordana Velikić, Hlavač Jaroslav, Zvjezdan Spasić, and Marko Vulić. 2018.  The benefits of the coding standards enforcement and it’s influence on the developers’ coding behaviour: A case study on two small projects. In _2018 26th Telecommunications Forum (TELFOR)_. IEEE, 420–425.
  * Porter et al. (1997) Adam A Porter, Harvey P Siy, Carol A Toman, and Lawrence G Votta. 1997.  An experiment to assess the cost-benefits of code inspections in large scale software development.  _IEEE transactions on software engineering_ 23, 6 (1997), 329–346.
  * Qian et al. (2024) Jili Qian, Zhengyu Jin, Quan Zhang, Guoqing Cai, and Beichang Liu. 2024.  A Liver Cancer Question-Answering System Based on Next-Generation Intelligence and the Large Model Med-PaLM 2.  _International Journal of Computer Science and Information Technology_ 2, 1 (2024), 28–35.
  * Rahman and Devanbu (2011) Foyzur Rahman and Premkumar Devanbu. 2011.  Ownership, experience and defects: a fine-grained study of authorship. In _Proceedings of the 33rd International Conference on Software Engineering_. 491–500.
  * Rahman et al. (2012) Foyzur Rahman, Daryl Posnett, and Premkumar Devanbu. 2012.  Recalling the” imprecision” of cross-project defect prediction. In _Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering_. 1–11.
  * Ramlochan (2024) Sunil Ramlochan. 2024.  0-Shot vs Few-Shot vs Partial-Shot Examples in Language Model Learning.  _Prompt Engineering and AI Institute_ (2024).  https://promptengineering.org/0-shot-vs-few-shot-vs-partial-shot-examples-in-language-model-learning/.
  * Rao et al. (2023) Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J Hellendoorn. 2023.  CAT-LM training language models on aligned code and tests. In _2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 409–420.
  * Ray et al. (2014) Baishakhi Ray, Daryl Posnett, Vladimir Filkov, and Premkumar Devanbu. 2014.  A large scale study of programming languages and code quality in github. In _Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering_. 155–165.
  * Renze and Guven (2024) Matthew Renze and Erhan Guven. 2024.  The effect of sampling temperature on problem solving in large language models.  _arXiv preprint arXiv:2402.05201_ (2024).
  * Russo (2024) Daniel Russo. 2024.  Navigating the complexity of generative ai adoption in software engineering.  _ACM Transactions on Software Engineering and Methodology_ (2024).
  * Savolainen and Myllarniemi (2009) Juha Savolainen and Varvana Myllarniemi. 2009.  Layered architecture revisited—Comparison of research and practice. In _2009 Joint Working IEEE/IFIP Conference on Software Architecture & European Conference on Software Architecture_. IEEE, 317–320.
  * Shen et al. (2023) Bowen Shen, Muhammad Ali Gulzar, Fei He, and Na Meng. 2023.  A characterization study of merge conflicts in Java projects.  _ACM Transactions on Software Engineering and Methodology_ 32, 2 (2023), 1–28.
  * Shihab et al. (2012) Emad Shihab, Ahmed E Hassan, Bram Adams, and Zhen Ming Jiang. 2012.  An industrial study on the risk of software changes. In _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_. 1–11.
  * Shrikanth et al. (2021) NC Shrikanth, William Nichols, Fahmid Morshed Fahid, and Tim Menzies. 2021.  Assessing practitioner beliefs about software engineering.  _Empirical Software Engineering_ 26, 4 (2021), 73.
  * Shull et al. (2007) Forrest Shull, Janice Singer, and Dag IK Sjøberg. 2007.  _Guide to advanced empirical software engineering_.  Springer.
  * Singh et al. (2017) Devarshi Singh, Varun Ramachandra Sekar, Kathryn T Stolee, and Brittany Johnson. 2017.  Evaluating how static analysis tools can reduce code review effort. In _2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 101–105.
  * Śliwerski et al. (2005) Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller. 2005.  When do changes induce fixes?  _ACM sigsoft software engineering notes_ 30, 4 (2005), 1–5.
  * Smith et al. (2016) Edward K Smith, Christian Bird, and Thomas Zimmermann. 2016.  Beliefs, practices, and personalities of software engineers: a survey in a large software company. In _Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering_. 15–18.
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An analysis of the automatic bug fixing performance of chatgpt. In _2023 IEEE/ACM International Workshop on Automated Program Repair (APR)_. IEEE, 23–30.
  * Spiess et al. (2024) Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2024.  Quality and Trust in LLM-generated Code.  _arXiv preprint arXiv:2402.02047_ (2024).
  * Storey et al. (2020) Margaret-Anne Storey, Neil A Ernst, Courtney Williams, and Eirini Kalliamvakou. 2020.  The who, what, how of software engineering research: a socio-technical framework.  _Empirical Software Engineering_ 25 (2020), 4097–4129.
  * Storey et al. (2024) Margaret-Anne Storey, Daniel Russo, Nicole Novielli, Takashi Kobayashi, and Dong Wang. 2024.  A Disruptive Research Playbook for Studying Disruptive Innovations.  arXiv:2402.13329 [cs.SE]
  * Stork (2024) Stork. 2024.  Harness the Power of BlackboxAI to Elevate Your Coding Game.  _Stork Tech_ (2024).  https://www.stork.ai/ai-tools/blackbox.
  * Sydorenko (2023) Pavlo Sydorenko. 2023.  Top 5 applications of Large Language Models (LLMs) in legal practice.  _medium_ (2023).  https://medium.com/jurdep/top-5-applications-of-large-language-models-llms-in-legal-practice-d29cde9c38ef.
  * Tavast et al. (2022) Mikke Tavast, Anton Kunnari, and Perttu Hämäläinen. 2022.  Language models can generate human-like self-reports of emotion. In _27th International Conference on Intelligent User Interfaces_. 69–72.
  * Tenny (1988) Ted Tenny. 1988.  Program readability: Procedures versus comments.  _IEEE Transactions on Software Engineering_ 14, 9 (1988), 1271–1279.
  * Thung et al. (2012) Ferdian Thung, Lucia, David Lo, Lingxiao Jiang, Foyzur Rahman, and Premkumar T Devanbu. 2012.  To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In _Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering_. 50–59.
  * Vereschak et al. (2021) Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux. 2021.  How to evaluate trust in AI-assisted decision making? A survey of empirical methodologies.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (2021), 1–39.
  * von Eschenbach (2021) Warren J von Eschenbach. 2021.  Transparency and the black box problem: Why we do not trust AI.  _Philosophy & Technology_ 34, 4 (2021), 1607–1622.
  * Wang et al. (2024) Wei Wang, Huilong Ning, Gaowei Zhang, Libo Liu, and Yi Wang. 2024.  Rocks Coding, Not Development–A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks.  _arXiv preprint arXiv:2402.05650_ (2024).
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ (2021).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ 35 (2022), 24824–24837.
  * Wermelinger (2023) Michel Wermelinger. 2023.  Using github copilot to solve simple programming problems. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_. 172–178.
  * Weyuker et al. (2008) Elaine J Weyuker, Thomas J Ostrand, and Robert M Bell. 2008.  Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models.  _Empirical Software Engineering_ 13 (2008), 539–559.
  * White et al. (2023) Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. 2023.  Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design.  _arXiv preprint arXiv:2303.07839_ (2023).
  * Wilkinson (9233) Lindsey Wilkinson. 2024, note = https://www.ciodive.com/news/anthropic-claude-3-opus-sonnet-haiku/709233/.  Anthropic rolls out Claude 3, says it outperforms generative AI rivals.  _CIO Dive_ (2024, note = https://www.ciodive.com/news/anthropic-claude-3-opus-sonnet-haiku/709233/).
  * Wilson et al. (2024) Greg Wilson, Jorge Aranda, Michael Hoye, and Brittany Johnson. 2024.  It Will Never Work in Theory.  _IEEE Software_ (2024).
  * Winters (2024) Titus Winters. 2024.  Thoughts on applicability.  _Journal of Systems and Software_ (2024), 112086.
  * Witschey et al. (2015) Jim Witschey, Olga Zielinska, Allaire Welk, Emerson Murphy-Hill, Chris Mayhorn, and Thomas Zimmermann. 2015.  Quantifying developers’ adoption of security tools. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_. 260–271.
  * Woodrow et al. (2024) Juliette Woodrow, Ali Malik, and Chris Piech. 2024.  Ai teaches the art of elegant coding: Timely, fair, and helpful style feedback in a global course. In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_. 1442–1448.
  * Wu (2024) Scott Wu. 2024.  Introducing Devin, the first AI software engineer.  _Cognition Labs Blog_ (2024).  https://www.cognition-labs.com/introducing-devin.
  * Wu et al. (2023) Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, et al. 2023\.  Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms.  _arXiv preprint arXiv:2307.10168_ (2023).
  * Xia et al. (2023) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023.  Automated program repair in the era of large pre-trained language models. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1482–1494.
  * Xu et al. (2022) Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A systematic evaluation of large language models of code. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_. 1–10.
  * Yang et al. (2024) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024.  SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.  arXiv:2405.15793 [cs.SE]
  * Yuan et al. (2023) Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023.  No more manual tests? evaluating and improving chatgpt for unit test generation.  _arXiv preprint arXiv:2305.04207_ (2023).
  * Zamfirescu-Pereira et al. (2023) JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023.  Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–21.
  * Zheng et al. (2006) Jiang Zheng, Laurie Williams, Nachiappan Nagappan, Will Snipes, John P Hudepohl, and Mladen A Vouk. 2006.  On the value of static analysis for fault detection in software.  _IEEE transactions on software engineering_ 32, 4 (2006), 240–253.
  * Zheng et al. (2023) Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. 2023.  Towards an understanding of large language models in software engineering tasks.  _arXiv preprint arXiv:2308.11396_ (2023).
  * Zimmermann et al. (2009) Thomas Zimmermann, Nachiappan Nagappan, Harald Gall, Emanuel Giger, and Brendan Murphy. 2009.  Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In _Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering_. 91–100.

##  Appendix A Data Analysis Example

A sample of the data analyzed for this work is provided below. The complete
dataset and analysis are included in our supplemental materials (Brown and
Cusati, 2024).

Table 5. Qualitative Analysis Example for LLM Beliefs (RQ1) and Behaviors (RQ3) Regarding Commented Code (C5) | Beliefs | Behaviors
---|---|---
LLM |  Response | Category |  Response | Category
ChatGPT |  Well-commented code can contribute to reducing defects indirectly, but the presence of comments alone does not directly correlate with fewer defects. Here’s a nuanced perspective on how comments impact code quality and defect occurrence: 1\. **Understanding and Clarity**: Comments can improve code understanding by providing explanations of complex algorithms, design decisions, or business rules. When developers can easily comprehend the code’s intent and logic, they are less likely to introduce defects due to misunderstandings or misinterpretations. 2\. **Maintenance and Updates**: Well-commented code aids in maintaining and updating software. Comments that describe the rationale behind certain implementations or potential pitfalls can guide developers in making modifications without inadvertently introducing defects. 3\. **Documentation**: Comments serve as documentation for future developers who may need to work on the codebase. Clear documentation reduces the likelihood of errors during maintenance or refactoring tasks by ensuring that developers understand the context and purpose of existing code. 4\. **Code Reviews**: During code reviews, comments can highlight areas that require attention or improvements. This can lead to better quality code as issues are identified and addressed early in the development process. However, it’s essential to note a few considerations: \- **Code Quality vs. Comments**: Comments are supplementary to the code itself. Well-commented code does not inherently mean better code quality if the underlying logic or design is flawed. \- **Maintainability**: While comments can aid in understanding, overly verbose or misleading comments can sometimes detract from code clarity. Comments should be concise, accurate, and focused on explaining the why rather than the what. \- **Dependency on Context**: The effectiveness of comments in reducing defects depends on the context and the quality of comments. Meaningful comments that provide insights into complex or critical areas of the codebase are more likely to contribute positively to defect prevention. In summary, while well-commented code can support defect prevention by enhancing understanding, aiding maintenance, and guiding developers, it is not a direct or sole factor in reducing defects. The overall quality of the code, adherence to best practices, and thorough testing remain fundamental in ensuring software reliability and minimizing defects. | A |  Yes, I can add code comments to reduce defects in generated code. Comments play a crucial role in improving code readability, maintainability, and overall quality. Here’s how I can use comments effectively in generated code: 1\. **Explanation of Intent**: I can include comments to explain the purpose and intent of specific sections of code, especially if the logic is complex or non-obvious. This helps other developers understand the code more easily and reduces the likelihood of misinterpretation. 2\. **Documentation of Assumptions**: Comments can document any assumptions or constraints that the code relies on. By making these assumptions explicit, developers are less likely to inadvertently violate them, reducing the risk of defects. 3\. **Clarification of Edge Cases**: I can add comments to highlight edge cases or special conditions that need to be considered. This ensures that developers are aware of these scenarios and can handle them appropriately, reducing the likelihood of bugs related to unexpected behavior. 4\. **TODOs and FIXMEs**: I can include TODO or FIXME comments to indicate areas of code that need attention or improvement. This helps ensure that issues are not overlooked and encourages developers to revisit and address them in future iterations. 5\. **Function and Method Documentation**: I can generate docstrings for functions and methods to provide documentation on their inputs, outputs, and behavior. This helps developers understand how to use the code correctly and can also facilitate automated documentation generation tools. 6\. **Change History**: Comments can document changes made to the code over time, including the reasons for the changes and any considerations that were taken into account. This helps maintain a record of the code’s evolution and can aid in troubleshooting and debugging. By incorporating comments effectively into generated code, I can help improve its quality, maintainability, and understandability. If you have specific preferences for how comments should be used in the generated code or if there are particular areas where you’d like additional comments, feel free to let me know, and I can tailor the comments accordingly. | Y
GitHub Copilot |  Well-commented code doesn’t inherently have fewer defects, as comments don’t affect the execution of the code. However, good commenting practices can indirectly contribute to reducing defects in several ways: Understanding: Comments can help developers understand what the code is supposed to do, which can prevent misunderstandings that lead to defects. Maintenance: Comments can make it easier for developers to maintain and modify the code, reducing the likelihood of introducing defects during these processes. Review: During code reviews, comments can help reviewers understand the code faster and more accurately, increasing the chances of catching potential defects. However, it’s important to note that comments should be used judiciously. Over-commenting can make the code cluttered and harder to read, and outdated or incorrect comments can be misleading. Good code should be self-explanatory as much as possible, with comments reserved for explaining why the code is doing something, not how. | A |  As an AI, I don’t inherently add comments to the code I generate. However, I can certainly include comments if instructed to do so. Comments can be useful for explaining complex sections of code, outlining the purpose of a function, or providing context for future developers. While comments don’t directly reduce defects in the code, they can help developers understand the code better, which can lead to fewer errors in the future. | Y
Gemini |  While well-commented code doesn’t directly guarantee fewer defects, it can strongly influence defect prevention and reduction in a few ways: **Improved Readability:** * Clear comments explain the purpose of code sections, algorithms used, and non-obvious logic. This makes it easier for the original developer and others to understand the code’s intent and functionality. Better understanding can lead to identifying potential issues before they become defects. **Enhanced Maintainability:** * Comments act as documentation within the code itself. When modifications are needed in the future, good comments help developers understand the existing code’s behavior and make changes with less risk of introducing new errors. This reduces the chance of bugs being introduced during maintenance or updates. **Improved Collaboration:** * Clear comments can act as a communication bridge between developers. They can explain design decisions, assumptions made, and potential edge cases. This fosters better collaboration and helps identify potential problems during code reviews or while working on different parts of the codebase. **Thinking Process Documentation:** * The act of writing comments often forces the developer to think more deeply about the code’s logic and potential issues. This self-documentation process can help catch errors during the initial development phase. However, it’s important to consider some caveats: * **Poorly Written Comments:** Comments that are vague, outdated, or incorrect can be misleading and do more harm than good. Comments should be clear, concise, and accurate. * **Over-Commenting:** Excessive commenting can clutter the code and make it harder to read. Comments should focus on explaining complex logic or non-obvious parts, not simply restating the code itself. In conclusion, well-commented code is a strong indicator of good coding practices and promotes better code understanding, maintainability, and collaboration. These factors can significantly contribute to fewer defects in the long run. However, the quality and focus of the comments themselves are crucial for them to be truly beneficial. | A |  No, Gemini itself does not add comments to the code it generates. The decision of adding comments rests with the developer or the system that uses the generated code. Here’s why Gemini doesn’t add comments: * **Focus on functionality:** Gemini’s primary function is to generate code that fulfills a specific task as instructed. Adding comments isn’t part of its core functionality. * **Potential for Misinterpretation:** As discussed earlier, comments can become outdated if the code they describe changes. This could lead to misunderstandings for developers who rely on them. While Gemini doesn’t add comments, well-written code can improve understanding and potentially reduce defects. If you’d like to explore ways to improve the readability of your generated code, I can provide some tips on best practices for code structure and naming conventions. | N
Blackbox AI |  Well-commented code is more likely to have fewer defects, but it’s not a direct cause-and-effect relationship. Here’s why: Why well-commented code tends to have fewer defects: Clearer understanding: Well-commented code helps developers understand the code’s intent, logic, and assumptions. This clearer understanding reduces the likelihood of misinterpretation, which can lead to defects. Improved maintainability: Commented code is easier to maintain, as developers can quickly understand the code’s purpose and make changes without introducing unintended defects. Better debugging: Comments can aid in debugging by providing context and helping developers identify issues more quickly. This reduces the time spent debugging, which can lead to fewer defects. Code reviews: Well-commented code is more likely to receive thorough code reviews, which can catch defects and improve overall code quality. Developer discipline: Writing well-commented code requires discipline and attention to detail, which can translate to other aspects of coding, such as writing more robust and defect-free code. Why well-commented code doesn’t guarantee fewer defects: Comments can be misleading: Comments can be outdated, incorrect, or misleading, which can lead to defects if developers rely on them too heavily. Over-commenting: Excessive commenting can make the code harder to read and maintain, potentially introducing defects. Comments don’t replace testing: Comments don’t replace thorough testing and validation. Even well-commented code can contain defects if not properly tested. Developer expertise: A developer’s expertise and attention to detail are more critical factors in producing defect-free code than the presence of comments alone. Best practices for commenting code: Use clear and concise comments: Comments should be easy to understand and provide relevant information. Keep comments up-to-date: Comments should be updated when the code changes to ensure they remain accurate. Use comments to explain intent: Comments should explain the code’s intent, assumptions, and logic, rather than just repeating the code. Don’t over-comment: Comments should be used judiciously, as excessive commenting can make the code harder to read. In summary, while well-commented code is more likely to have fewer defects, it’s not a guarantee. A combination of good coding practices, thorough testing, and attention to detail is still necessary to produce high-quality, defect-free code. Sources: • https://statmodeling.stat.columbia.edu/2024/02/07/when-all-else-fails-add-a-code-comment/ • https://ardalis.com/when-to-comment-your-code/ • https://bytedev.medium.com/code-comment-anti-patterns-and-why-the-comment-you-just-wrote-is-probably-not-needed-919a92cf6758 • https://dev.to/codemouse92/to-comment-or-not-to-comment-3f7h • https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/ • https://developers.slashdot.org/story/02/05/19/2216233/what-is-well-commented-code • https://softwareengineering.stackexchange.com/questions/336430/coding-standard-for-clarity-comment-every-line-of-code • https://stackoverflow.com/questions/3236751/when-are-comments-too-much-and-when-are-they-not-enough • https://www.quora.com/Does-commented-code-affect-performance • https://www.reddit.com/r/webdev/comments/17mv7ln/good_code_does_not_need_comments_is_taken_way_too/ | A |  While Blackbox AI can generate code, it does not directly add code comments to reduce defects in the generated code. However, Blackbox AI can provide suggestions for comments based on the code’s functionality and structure. These comments can help improve the code’s readability and maintainability, making it easier for developers to understand and modify the code. Additionally, Blackbox AI can use various techniques to improve the quality of the generated code, such as following coding standards, using code reviews, and employing debugging tools. These practices can indirectly contribute to reducing defects in the generated code, as they promote better coding practices and help catch errors early in the development process. In summary, Blackbox AI can provide suggestions for code comments and use various techniques to improve the quality of generated code, which can indirectly help reduce defects in the code. Sources: • https://news.ycombinator.com/item?id=33632610 • https://www.linkedin.com/pulse/blackbox-ai-supercharging-your-coding-workflow-swarup-mukharjee-5gqbe/ • https://www.analyticsvidhya.com/blog/2024/02/how-blackbox-ai-can-solve-all-your-coding-problems/ • https://www.youtube.com/watch?v=dLBEk49licI | N