# Investigating and Designing for Trust in AI-powered Code Generation Tools

Ruotong Wang<EMAIL_ADDRESS>University of
WashingtonSeattleWashingtonUSA98195 ,  Ruijia Cheng<EMAIL_ADDRESS>University
of WashingtonSeattleWashingtonUSA98195 ,  Denae Ford<EMAIL_ADDRESS>Microsoft ResearchRedmondWashingtonUSA98052 and  Thomas Zimmermann
<EMAIL_ADDRESS>Microsoft ResearchRedmondWashingtonUSA98052

(2018)

###### Abstract.

As AI-powered code generation tools such as GitHub Copilot become popular, it
is crucial to understand software developers’ trust in AI tools—a key factor
for tool adoption and responsible usage. However, we know little about how
developers build trust with AI, nor do we understand how to design the
interface of generative AI systems to facilitate their appropriate levels of
trust. In this paper, we describe findings from a two-stage qualitative
investigation. We first interviewed 17 developers to contextualize their
notions of trust and understand their challenges in building appropriate trust
in AI code generation tools. We surfaced three main challenges—including
building appropriate expectations, configuring AI tools, and validating AI
suggestions. To address these challenges, we conducted a design probe study in
the second stage to explore design concepts that support developers’ trust-
building process by 1) communicating AI performance to help users set proper
expectations, 2) allowing users to configure AI by setting and adjusting
preferences, and 3) offering indicators of model mechanism to support
evaluation of AI suggestions. We gathered developers’ feedback on how these
design concepts can help them build appropriate trust in AI-powered code
generation tools, as well as potential risks in design. These findings inform
our proposed design recommendations on how to design for trust in AI-powered
code generation tools.

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Human-centered computing Empirical studies in HCI††ccs: Human-centered
computing HCI design and evaluation methods††ccs: Software and its
engineering††ccs: Computing methodologies Artificial intelligence

##  1\. Introduction

Emerging generative AI applications began supporting human tasks in various
domains, ranging from creative work such as writing and drawing to complex
information work such as software engineering. In the domain of software
engineering, AI-powered code generation tools such as Github Copilot (Git,
2023), Tabnine (AIA, 2023), Kite (Kit, 2022) and Amazon CodeWhisperer (MLp,
2023) have quickly emerged and gained popularity in programmer communities
(Dohmke, 2022). Designed to be developers’ “AI pair programmer” (Git, 2023;
MLp, 2023) that enables a new way of programming assistance (Sarkar et al.,
2022; Barke et al., 2022), these tools can generate multiple lines of code in
real-time based on a prompt within the Integrated Development Environment
(IDE) (Sarkar et al., 2022).

While researchers and users are excited about AI-powered code generation
tools, these tools also introduce new design challenges in creating proper
user experiences. One significant challenge involves helping users build the
_right_ trust with the AI. Software developers’ trust in programming support
tools has long been studied as a crucial design requirement for such tools, as
it serves as a key prerequisite for the safety of resulting software products
(Lipner, 2004; Hasselbring and Reussner, 2006; Durumeric et al., 2014).
However, with little explicit consideration of users’ trust in the current
design of AI-powered code generation tools, developers can find it challenging
to form an accurate mental model of what AI tools can do or not (Sarkar et
al., 2022), determine in which tasks they should trust AI, or determine the
quality of specific AI suggestions (Vaithilingam et al., 2022; Barke et al.,
2022); thus becoming vulnerable to either over and under trust AI.

Therefore, we argue that the design of AI-powered code generation tools should
empower users to develop _appropriate_ trust in the tool by considering users’
trust through interface design. Indeed, the trustworthiness of technology is
not inherent in an AI system, but is based on how users interpret the
information communicated by the AI system (Liao and Sundar, 2022a) and can
vary by situation and context (Zhang et al., 2022). Therefore, the interface
between the developer and AI should communicate the appropriate levels of
trustworthiness to users. Yet, while emerging work has begun investigating the
general usability of AI assistant tools in software engineering or in broader
domains (Vaithilingam et al., 2022; Barke et al., 2022; Sarkar et al., 2022;
Ziegler et al., 2022; Bird et al., 2023; Weisz et al., 2021), we still know
little about how designers should consider trust when designing system
interfaces for AI-powered code generation tools.

In this paper, we present results from a two-stage qualitative study. Given
that trust is subjective and situational in nature (Zhang et al., 2022), we
started by understanding developers’ notions of trust in the particular
context of using AI code generation tools. In stage 1, we conducted interviews
with 17 developers who have various levels of experience in using AI-powered
code generation tools in real-life scenarios. We analyzed the results from
stage 1 to answer the questions of what contributes to developers’ trust in
AI-powered code generation tools among developers (RQ1) and what challenges
developers face in forming appropriate trust (RQ2). We then explored how we
can design system interfaces to support developers in building appropriate
trust (RQ3) in stage 2 of the study. Specifically, we developed three groups
of visual design concepts and conducted design probe sessions with 12
participants to explore the design space. Our findings contribute to the
literature of the FAccT community with (1) an in-depth empirical understanding
of user trust in AI-powered code generation tools and the challenges that
users face in forming appropriate trust, and (2) a series of user-evaluated
design concepts to help users build appropriate trust in generative AI tools.

##  2\. Related Work

###  2.1. Generative AI in software engineering: AI-powered code generation
tools

The recent development of generative AI models (Brown et al., 2020) unleashes
new possibilities for AI-generated content. Generative AI models produce
textual or image artifacts in response to human prompts, resulting in the
potential of supporting more creative human tasks such as writing (Gero and
Chilton, 2019b, a; Cheng et al., 2022a; Clark et al., 2018), drawing (Oh et
al., 2018; Fan et al., 2019), game design (Guzdial et al., 2019), music
creation (Louie et al., 2020) and software engineering (Barke et al., 2022;
Ziegler et al., 2022; Bird et al., 2023; Weisz et al., 2022).

Building on prior work exploring the emerging interaction paradigms between
users and generative AI models mostly in creative tasks, we study user
interactions with generative AI tools in the context of software engineering.
Software engineering is a type of complex and high-demanding information work
that often involves high cognitive load and stress (Sarkar et al., 2022; Ernst
and Bavota, 2022; Helgesson et al., 2019; Gonçales et al., 2019), and
therefore demands high-quality support. As a means to provide support for this
complex work, commercial AI-powered code generation tools such as GitHub
Copilot (Git, 2023), Tabnine (AIA, 2023), Kite (Kit, 2022), and Amazon Code
Whisperer (MLp, 2023) have emerged and become a novel service to expert and
novice code creators alike. These tools provide AI services powered by large
language models trained on code data (Git, 2023; Sobania et al., 2022) and can
suggest code based on user prompts and project context (Brown et al., 2020).
For instance, powered by the OpenAI Codex model and trained on source code
repositories on Github (Ope, 2021), Copilot functions as an extension in
multiple code editors that can generate code suggestions as ghost text at the
user’s cursor location. When using such AI tools, users can write comments in
natural language and prompt the AI to generate code that they can accept,
reject, make edits, and choose from various candidates. The AI can also
complete users’ in-progress code within a single line or by completing the
function. Compared to traditional code completion tools based on defined rules
and existing repositories, AI-powered code generation tools produce longer and
more contextually relevant code snippets by synthesizing new code that might
not exist in any code base (Bird et al., 2023).

As AI code generation tools introduce a brand new interaction paradigm between
developers and AI, user needs and experiences have not been fully studied yet.
Early empirical investigations started to explore developers’ mental models
when interacting with such AI tools. Some found that developers would spend
more time reading and reviewing code than writing code when using AI code
generation tools (Bird et al., 2023; Vaithilingam et al., 2022), but often
faced challenges in evaluating the quality of AI-generated code (Vaithilingam
et al., 2022; Barke et al., 2022). Others suggest that without a proper
understanding of its capacity, developers can be limited in their interaction
with AI, resulting in ineffective collaboration (Sarkar et al., 2022; Barke et
al., 2022; Perry et al., 2022). All these studies have pointed to the need for
better user interfaces to enhance human interaction with generative AI systems
such as AI-powered code generation tools (Guzdial et al., 2019; Lee et al.,
2022) and a further understanding of how to design user interfaces to support
users’ needs and experiences.

###  2.2. User trust in AI-powered code generation tools

In particular, we focus on understanding developers’ trust in AI code
generation tools. Trust is often considered one of the key factors that affect
the interaction between people and AI (Commission, 2019; Das and Rad, 2020;
Liao and Sundar, 2022b). On one hand, the lack of trust can prevent users from
adopting AI tools in their workflow, even when the performance of the system
is superior (Boubin et al., 2017; O’Connor et al., 2019). On the other hand,
blind trust in AI, especially in high-stake tasks such as software
engineering, can result in overlooking mistakes or risks produced by AI
(Pearce et al., 2022; Perry et al., 2022). While user trust has long been
considered crucial in the design of traditional programming support tools,
such as compilers and version control systems (Xiao et al., 2014; Witschey et
al., 2015) to ensure software safety (Lipner, 2004; Hasselbring and Reussner,
2006), developers’ trust in AI code generation tools needs even more nuanced
and careful consideration (Pearce et al., 2022; Dakhel et al., 2022; Bird et
al., 2023) due to the uncertainty introduced by AI. For example, since the
mechanism of generative AI is more opaque and the outputs are more difficult
to anticipate than traditional developer support tools, developers must
establish an appropriate level of trust with these AI tools and be cautious
about the potential risks (Bird et al., 2023).

One of the goals of our study is to describe what contributes to developers’
trust in AI-powered code generation tools. Most commonly, trust in AI is
defined as the user’s judgment or expectations about how the AI system can
help when the user is in a situation of uncertainty or vulnerability (Liao and
Sundar, 2022a; Lee and See, 2004; Vereschak et al., 2021a). That is, when a
user trusts an AI system, they display positive expectations while accepting
the perceived risks of undesirable outcomes (Vereschak et al., 2021a).
Further, trust in AI is subjective and situational and should be studied as an
attitude. Indeed, Mayer et al. (1995) described trust as “an affective
construct that can vary depending on the context and experiences of a person
rather than simply being a rational or an objective reality.” User trust can
also grow or change over time as users interact with AI systems repeatedly
(Holliday et al., 2016) and can be calibrated regarding the context and stake
of the task (Zhang et al., 2020). Building on these works, in our study, we
study developers’ trust by focusing on understanding developers’ positive
expectations and perceived risks associated with AI code generation tools.
Unlike prior research studying users’ experience with AI code generation tools
mostly in controlled lab settings (Barke et al., 2022; Vaithilingam et al.,
2022; Bird et al., 2023), we conducted retrospective interviews with
developers who have experience using AI code generation tools in real-life
scenarios. In our research setting, developers face real consequences if AI
produces undesirable outcomes, which could help us uncover the interplay of
positive expectations and perceived risks.

###  2.3. Designing user interface to support appropriate trust in AI systems

Researchers have studied how to design AI systems to support user trust. While
system performance has been shown to correlate with user trust (Fogg and
Tseng, 1999), it has also been highlighted that _interface_ between the user
and AI matters. Liao and Sundar (2022a) proposed the notion of trust
affordances and argued that designers of AI systems should design user
interfaces and experiences to help _communicate_ the internal trustworthy
characteristics of AI, upon which users make judgments about whether and how
to trust it. Similarly, many design guidelines for human-AI interaction have
also listed user trust as one of the key design goals when designing the user
experience of AI systems (Amershi et al., 2019; Commission, 2019; Cheng et
al., 2022a).

One core principle is to design for calibrated trust or warranted trust, that
is, to use interface design to help users determine how much trust they could
have in AI in different contexts and scenarios (Liao and Sundar, 2022a; Jacovi
et al., 2021; Buçinca et al., 2020; Lee and See, 2004; Zhang et al., 2020;
Passi and Vorvoreanu, 2022). Specifically, some emphasize the value of
transparency (Drozdal et al., 2020) and predictability (Daronnat et al.,
2021), which helps a user build proper expectations when interacting with AI
(Jacovi et al., 2021). Others highlight the need for supporting explainability
and interpretability of model mechanism (Sun et al., 2022; Liao et al., 2020;
Lipton, 2018), such as the confidence score of model output (Zhang et al.,
2020; Weisz et al., 2021), to help users understand the model and make
informed judgments. These mechanisms gave users means and metrics to better
assess the performance of AI, which helps users build appropriate trust with
the system (Jacovi et al., 2021). However, the effectiveness of these UI
features has not been shown to be persistent across contexts or scenarios. For
example, Agarwal et al. found that model confidence scores could be misleading
for users’ perception of the quality of model output (Agarwal et al., 2021).

Notably, these prior works studying the effect of UI features on users’ trust
mostly focus on classification or prediction tasks, while not much effort has
been put into supporting appropriate user trust in the recent development of
generative AI. Yet, designing for appropriate user trust in generative AI
introduces new challenges and opportunities, such as richer and more complex
input and output space (Sun et al., 2022) and more flexible roles in human-AI
collaboration (Guzdial et al., 2019).

In summary, considering developers’ trust in the design of AI-powered code
generation tools is important. However, the current literature falls short in
understanding how to design system interfaces to support developers in
building appropriate trust. Specifically, while prior work studying AI code
generation tools has investigated their performance and usability (Bird et
al., 2023; Barke et al., 2022; Ziegler et al., 2022; Vaithilingam et al.,
2022), little is known about developers’ perception of trust and the
challenges in building trust. At the same time, while prior work on designing
user interfaces in AI systems outlined potential principles that could
contribute to users’ trust, most studies focused on classification or
prediction systems (Vereschak et al., 2021b) and it is unclear how the
principles generalize to generative AI tools, given its novel interaction
paradigm (Guzdial et al., 2019). In this work, we investigate users’ trust in
generative AI tools in the context of AI code generation tools, strive to
understand how developers’ interaction with AI systems shapes their trust, and
learn how to design tools to better support appropriate trust.

##  3\. Methods

Our study consists of two stages: an interview study followed by a design
probe study. We first conducted interviews to understand what contributes to
developers’ trust in AI-powered code generation tools among developers and
what challenges developers face in forming appropriate trust (RQ1 and RQ2).
Then in the second stage, we conducted a design probe study to understand the
design concepts that can facilitate the trust-building process (RQ3).

###  3.1. Stage 1: Retrospective Interview Study

To gain an understanding of what contributes to developers’ trust in AI code
generation tools, together with what challenges they face in forming
appropriate trust, we conducted retrospective interviews with 17 developers
who have different levels of experience with a variety of such tools.

Study Procedure and Data Analysis A week prior to the interviews, we asked
participants to collect significant moments—i.e., moments where participants
were either particularly satisfied, disappointed, or surprised by their usage
of any AI-powered code generation tools. We asked participants to share the
descriptions and screenshots of those moments with us via an instant messaging
tool during the week. These records of significant moments provided us with
realistic contexts to understand developers’ trust of AI tools and can help
participants recall the nuances of their experience in the interviews. A
similar approach of retrospective interviews and critical incident sampling
has been applied to study patients’ trust during medical visits (Yañez-
Gallardo and Valenzuela-Suazo, 2012; Wendt et al., 2004) and interpersonal
trust in business negotiations (Münscher and Kühlmann, 2015).

During the 60-min retrospective interview sessions, we started with
participants’ experience using AI code generation tools by asking them to walk
through the significant moments and speak about their general experience with
those AI tools. We specifically probed for factors that help them form trust
or erode their trust in AI, as well as what appropriate trust meant to them.
All interviews were video and audio recorded and later transcribed. Our
analysis of the interview data followed the procedure of inductive thematic
analysis (Braun et al., 2019). We generated themes on factors that contribute
to developers’ trust and the challenges they had in the process, and
iteratively refined and grouped the themes through group discussions among all
authors.

Participants We recruited participants within our institute through
advertisements in internal group chat channels and mailing lists. Recruiting
participants internally allows participants to share significant moments
without privacy concerns. During recruitment, we screened for participants’
demographic profiles and experience using AI-powered code generation tools. We
were able to select 17 participants to obtain a balance mix of experience
using AI-powered code generation tools, years of programming, and type of
programming (e.g., front-end, back-end, data science). Detailed profiles of
our participants are presented in Supplementary material (Table 1).

###  3.2. Stage 2: Design Probe Study

We then conducted a design probe study to explore the design space of
supporting appropriate trust in AI-powered code generation tools. We first
developed three groups of design concepts with visual representations, and use
the visuals to elicit users’ feedback in 12 design probe sessions. Notably, we
do not aim to settle on or evaluate the effectiveness of any specific
design—rather, we want to understand how UI design concepts could facilitate
appropriate user trust through nuanced qualitative exploration. A similar
approach has been used to explore interface designs for AI-assisted decision-
making systems in child welfare (Kawakami et al., 2022) and clinical diagnosis
(Yang et al., 2023).

Developing design concepts and visual stimuli Based on the findings of the
interviews, the research team brainstormed potential design requirements and
interface features that can address user needs. Then we translated the design
concepts into low-fidelity visual stimuli made in Microsoft Powerpoint slides.
These visual stimuli served as probes to elicit users’ reaction, feedback, and
new ideas in our design probe session. We deliberately presented our designs
in low-fidelity to encourage participants to give feedback on the design
concepts, instead of focusing on the feasibility and discoverability of
features. Specifically, we used Github Copilot as an example of AI code
generation tools and Visual Studio Code as an example of an Integrated
Development Environment (IDE) in our prototypes. We also included code
snippets from toy examples to allow users to imagine how they might use these
designs in their realistic programming tasks. The specific design concepts and
visual stimuli are introduced in §4.3.

Procedure, participants, and data analysis We conducted one-to-one 60-min
design probe sessions with 12 participants. At the beginning of each session,
participants briefly reflected on their experience with AI code generation
tools and their trust in AI. We then showed the three sets of stimuli in the
same order to the participants. We explained each design and asked for
participant feedback and reaction, including questions on how they might use
the features and in what scenarios, and if and how the features contribute to
trust. We also encouraged participants to brainstorm new features. We
recruited participants by posting a recruitment message on social media and
internal developer forums. Among the responses we received, we sampled for
participants’ diversity in terms of their programming experience and frequency
of using AI code generation tools. Detailed profiles of our participants are
presented in Supplementary material (Table 2). Similar to Study 1, all
sessions were video and audio recorded and later transcribed. The data
analysis followed the procedure of thematic analysis. In the analysis, we
focused on analyzing ways that interface features could be helpful or not
helpful for users to build appropriate trust with the AI, as well as potential
risks and requirements in the design.

##  4\. Results

###  4.1. RQ 1: What contributes to developers’ trust in AI-powered code
generation tools?

Developers’ expectations of the capability of and potential risks introduced
by AI-powered code generation tools form the basis of their trust. In our
interviews, we observed a variety of positive expectations and perceived risks
that developers formed about AI tools.

Expectation of AI tools’ capability One positive expectation developers have
about AI code generation tools is the boost to their productivity. For
example, P12 expects the AI to “save a lot of time”. Others expect AI tools to
help improve their code quality by serving as an idea sounding board. The AI’s
ability to scaffold learning is another key factor in building trust. For
example, P15 trusts Copilot because “it would prompt me for ways to do it that
maybe I wouldn’t have thought of myself”, which helped them pick up a new
programming language more quickly. Besides understanding what AI code
generation tools can help, developers also want to learn “how well” can the
tools help. Echoing previous design guidelines (Amershi et al., 2019), we
observed that learning AI’s limits can help developers set proper expectations
and avoid disappointment when AI underperforms. Specifically, developers want
to “build intuitions on what Copilot is suited for verses not suited for.” For
example, many developers shared that AI tools are great at generating
boilerplate code, but they would not expect AI tools to output perfect
solutions for tasks with complex logic, such as implementing a web server from
scratch. Based on the understanding of the suitable use cases of using the
tools, developers’ trust in AI tools is situational to specific programming
tasks. P13 explained that they “trust Copilot for small snippets of
predictable code…but I don’t distrust Copilot because it doesn’t do what it
doesn’t do.” Though developers do not expect AI tools to perform well in every
situation, they are more cautious with AI tools in high-stake and high-impact
scenarios, such as tasks that involve core business logic. P13 is reluctant to
use Copilot when working on codebases that could “impact millions of customers
and millions of dollars potentially”. Developers became much less tolerant of
low-quality code on high-stake tasks and would rather write the code
themselves rather than trust AI.

Recognition of potential risks of using AI tools Developers also anticipated
risks with AI tools and sought ways to mitigate anticipated risks to build
trust. One risk is that AI’s poor suggestions may require extra work for
developers to fix. Recognizing that AI suggestions are not perfect, developers
preferred suggestions that “lay the foundation very well” (P10) and were easy
to build on. Another common concern among developers is that AI would erode
their learning abilities and replace their work over time. For example, after
accepting high-quality suggestions from Copilot for a while, P8 started to
worry about losing their “muscles.” (P8) Many developers appreciated signals
that AI would prioritize their leanings, instead of simply outputting
suggestions. For example, P9 wanted to have a “sensei” version of Copilot,
which is more “endearing” and will “invest in you by suggesting what you could
learn” based on their previous experiences. Last but not least, developers
also wanted to understand the security and privacy implications of using AI
tools to decide how much to trust them. For instance, P7 sought information on
code ownership and P9 was concerned about their programming process being
“listened to” and wanted clarification on the intention and “boundaries” of
the AI. Similarly, P16 wanted “an end-to-end transparent diagram with what’s
exactly going on” so that they know “exactly what’s being tested to not make
sure that this code is copyrighted.” This was a valid concern that many
developers shared.

###  4.2. RQ2: What challenges do developers have in building appropriate
trust?

####  4.2.1. Challenges in building appropriate expectations.

While many official demos or marketing materials focus on “what” and “how
well” AI can work, developers preferred to build expectations of the
capability and limits of AI tools through empirical evidence. Developers often
wanted to see an AI tool demonstrating its values in practice and “feel the
benefits of it” (P17) in order to build up positive expectations. Some
developers built up intuitions about Copilot by using AI in their routine
programming tasks. For example, P13 shared that “Once you’ve seen it 10 times,
I’m pretty sure Copilot will do this thing the 11th time.” Others learn about
it via intentional experimentation. For instance, P6 “played around” and tried
to see “if I could break it’’ when first started using Copilot, so that they
can “know where its limits are”, which helped “set my expectations on how to
use it.” However, developers still face challenges in building reasonable
expectations of AI.

Tedious learning process. First, it takes time and effort for developers to
form reasonable expectations of the capabilities and limitations of AI via
empirical evidence. For example, P13 shared that “You have to give it the
benefit of the doubt for a while until it makes a little more sense to you as
a tool…You just have to ignore those things until your expectation lines up
more with Copilot’s capabilities.” (P13) The time commitment required to build
proper expectations with the AI tool took their time away from doing the
actual work. For example, P8 struggled to balance doing actual work themselves
and testing Copilot’s capabilities: “I can’t do 100 percent of my work because
one part [of me] wants to code, one part [of me] want to see how smart it is.”
In addition, it required effort for developers to update their expectations of
the AI once formed. Developers found it difficult to regain trust after an
undesirable encounter with AI. For example, P5 commented that: “It takes three
good recommendations to build trust versus one bad recommendation to lose
trust.” In another example, P16 shared that their colleague once tried out
Copilot but did not get “the results that he was expecting” and stopped using
it after the first few tries. These examples suggest that the lack of support
and incentives of trial-and-error often leaves developers with a limited view
of AI tools’ capabilities.

Limited use scenarios. Developers are often limited by personal experience and
unable to gain a comprehensive understanding of AI tools in broader scenarios.
P15 pointed out that given “everyone’s writing code in different
circumstances”, it is hard to know “exactly how well [Copilot] is going to
perform in your unique situation until you try it.” For example, although P3
had many successes with Copilot when writing “proof-of-concept type of the
projects,” they are unsure about how much to trust Copilot in an “actual
production setting.” In another example, P11 trusted Copilot for back-end
programming tasks in C++ or C#, but was hesitant to use Copilot for other
front-end programming tasks since they did not know how well Copilot’s
capabilities generalize in the new setting.

Biased preexisting knowledge. Further, developers tended to project their
expectations of traditional programming support tools onto new AI tools,
resulting in confusion and frustration. For example, many have previous
experience with traditional, widely used auto-completion tools, such as
IntelliSense, that complete a single line of code using standard code directly
pulled from documentations. AI tools, on the other hand, generate new code
that is diverse, and flexible, but often needs review and edits. Developers
who expect AI tools to complete lines with 100% correct suggestions find the
chunks of AI-generated code jarring and as a result, have hard time
understanding its benefits. For example, P14 recalled their initial experience
with Copilot where they got frustrated because it did not give them “the right
solution” or in fact the same solution as IntelliSense. This bias from their
previous experience prevented them from strategically using Copilot, which P14
described as involving “reading the code that it provides, and then trying to
understand the logic and compare it with my own and figure out if I want to
adopt its code or not.”

####  4.2.2. Challenges in controlling AI

To trust AI tools, developers needed to be able to control for what task, at
what timing, and in what format AI suggestions should come in. For this
purpose, developers formed strategies to cooperate with AI, delegating AI with
tasks that they trust AI to do, and making adjustments so that AI can show its
full potential. For example, P2 typically “lays out the overall direction” of
a complex programming task themselves and only allows the AI to complete
small, mundane tasks. P11’s trust in Copilot improved when they found that
writing comments helped Copilot generate high-quality code: “the more time
I’ve spent using it, the more I’ve realized, I can do this and it’ll make
better suggestions.” When prompting Copilot in their comments or in in-
progress code, P6 often “tell(s) Copilot a little more” about their specific
use cases so that the AI can produce code in higher quality. In addition,
developers also prefer to customize the length and timing of AI suggestions
that fit their existing workflow. For example, P14 turns AI suggestions on
“during the implementation stage” and off “during the tweaking stage” to avoid
distraction. We observed two main challenges for users to control AI to make
it more trustworthy using existing system affordances.

Lack of guidance to harness AI output. First, developers often are uncertain
about how to harness AI tools to produce desirable outputs. For example, many
find themselves wasting time trying to fine-tune prompts for Copilot to
produce perfect results, but later realize that it is more efficient to let AI
generate code and make edits to it. Related to this, developers are often
uncertain about how much information should be included in their prompts or in
in-progress code for AI to produce desired outputs. Many, like P1, ended up
writing the code themselves instead of waiting for AI suggestions when
“there’s no information anywhere about what I’m going to do or what variable I
want to access, what manipulation I want to do it…there’s no reason to expect
Copilot will read my mind and figure out what I want to do now.” In addition,
control strategies such as dividing tasks or engineering prompts can result in
tedious trial and error and time cost. It can be challenging for developers to
decide whether it is worth their time to control the AI at all: “should I
trust it to give it time to get something in return? Is it worthwhile my
time?” (P8) Developers wished to have more guidance and feedback on strategies
to harness AI effectively in their workflow.

Difficulty in controlling the timing of suggestion. Second, developers
struggled to predict and control the timing that AI suggestions appear. Many
were frustrated that they did not get enough suggestions when they desired
AI’s help; whereas other times they found AI suggestions to be intrusive,
getting in the way of their programming flow. For example, P13 often found
AI’s suggestions to create unnecessary “visual clutter” on the screen when
they already knew what they wanted to write. P7 even felt that they had to
“fight with Copilot” to let the suggestion go away. Several others wanted the
AI tools to wait to provide suggestions until they set up the context in
comments or in code. For example, P11 did not want “Copilot to jump the gun
and suggest before I finish fully defining the method” because it would likely
lead to “suggestions that are way off the mark.” The lack of effective ways to
control the input from AI often left the impression of the AI being aggressive
and obtrusive, which can erode user trust. As P8 said, Copilot is “wanting to
sit in my seat…It started as a co-pilot, but now it’s the pilot and I’m
becoming the co-pilot.”

####  4.2.3. Challenges in evaluating suggestion quality

Developers always wanted to maintain the final say on AI suggestion to catch
any problematic output in time: “[I] would very carefully look at what it was
suggesting before I hit enter tab because I had to be certain.” (P7)
Strategies to review and evaluate the quality of AI-generated code suggestions
included “logically going through the problem,” (P11) to “validate by testing
it,” (P13) and to run the code through formal tests (P1). Developers also
wanted to look into the black box to understand the AI model’s generation
process to better evaluate the model’s output, such as how confident the model
is with its suggestions and how the model is trained. For example, P11 wants
to know why the AI “‘think(s) somehow this is correct” so that they can decide
how to trust it. They wish a feature that allowed them to view the confidence
of a particular suggestion: ‘If I know where it came from, I know it’s
perspective. ” The current affordances of AI code generation tools pose a
number of challenges to users in evaluating AI suggestions.

Lack of debugging support. Validating the code generated by AI can be tedious
and frustrating. Multiple reasons contribute to the difficulty of debugging AI
generated code. The AI generated code snippet can be long and span across
multiple lines, making it hard to spot nuanced mistakes. For instance, it took
P9 half an hour to discover an additional bracket generated by AI. Sometimes,
AI-generated code can also involve complex logic that is hard to understand in
the first place. As a result, developers like P13 prefer to not use AI tools
and wrote the code themselves when the code involves complex logic, because
there’s no information on the quality of the suggestions. It’s even harder to
validate AI-written code that uses unfamiliar language or libraries. For
example, P1 once had to spend extra effort researching a method used in
suggestions they were not familiar with in order to debug: “I had to look back
at documentation, and it was using fields that were deprecated or right
nonsense fields that just created on its own.”

Cognitive load of reviewing. Additionally, the frequent need to validate AI’s
code suggestions can interrupt the developers’ flow of programming. Many
developers described the process of constantly switching between writing and
reviewing code to be “mentally draining” (P7) and “creates more work” (P8).
Given the high cognitive cost of task switching, some developers even consider
stopping using AI tools to avoid distraction. For example, P9 shared that
“[Copilot] might derail my mind. Sometimes it’s easier to focus my brainpower
on solving the problem itself rather than trying to figure out what really is
Copilot is trying to tell me if that makes any sense.”

###  4.3. RQ3: How can we support developers in building appropriate trust via
interface design?

Based on findings from study 1, we developed three groups of design concepts
for AI code generation tools and solicit feedback and new design requirements
from the design probe sessions.

####  4.3.1. Communicating AI performance via usage statistics

Developers’ expectations of AI tools form the basis of their trust. However,
developers’ current process of learning AI’s capability which relies on
intuitions accumulated through personal experiences can be tedious, limited,
and biased, as we showed in Section 4.2.1. We explored two design concepts
that help developers form accurate and comprehensive expectations of AI
performance via explicit usage statistics. Our design shows users’ overall
usage stats (Figure 1(a)) and usage stats broken down by files (Figure 1(b))
in a feedback analytics report after users have (hypothetically) used Copilot
for a certain period of time. Overall usage statistics include data such as
total hours of usage and average acceptance rate, while situational usage
statistics include data such as most accepted categories of suggestions and
comparison between user acceptance rates and Copilot’s confidence.

(a) Overall usage stats

(b) Situational usage stats

Figure 1. Visual stimulus for design concepts that communicate AI performance
via usage statistics.

Participants found that both usage analytics provide explicit information on
Copilot’s performance, including both values and limitations, helping them
form appropriate expectations on its capacities and limitations. For example,
D8 thought that “the aggregated measures of how I’ve used Copilot over time
helps me form an image of my relationship with Copilot, which helps me
evaluate Copilot’s performance and form informed goals.” D4 appreciated the
ability to see how many suggestions have been made, which helps them access
how much Copilot has helped them.

D10 suggests that seeing Copilot demonstrate success in statistics will boost
their trust in it: “‘if I can see a quantifiable number of how much Copilot
increases my productivity or saved me time, I’m more prone to depend on it
more.” Participants also find the analytics to be helpful for learning the
limits of Copilot. When D7 saw the analytics of a number of edits they’ve made
to Copilot’s suggestions, they are reminded “the hassle working with it” and
that it requires them to “learn or to adjust in order to make it work.”
Similarly, D8 wished to see breakdowns of Copilot’s performance based on
functional concepts, so that they can better “navigate that space with which
topics is Copilot the best at.”

At the same time, it can be challenging for users to interpret the statistics
shown on the dashboard. For example, D9 worried that: “I also need to analyze
the correlation and causation, the statistical numbers. I think it’s just put
into many works to developers.” Therefore, in addition to the numbers, users
prefer more actionable insights. D2 wished that the dashboard can not only
tells users “how users used things,” but also “how to use something,” by
including some actionable tips how to use Copilot in unobtrusive ways. In
addition, participants were concerned about potential privacy issues,
especially for workplace surveillance when tracking telemetry data. For
example, D12 worried that organizations will use the tracked data to evaluate
employees: “It’s scary if this feature can be used by those companies in
evaluating how good the programmers are. They will just fire those programmers
with bad coding skills.”

####  4.3.2. Allowing users to control AI by setting and adjusting preferences

Our results in Section 4.2.2 show that developers want more agency in
controlling what and how AI tools help them. However, the current system only
allows users to control AI’s behavior via natural language prompts, which can
be ineffective. We generated two design concepts to explore the potential of
graphical user interfaces in controlling AI’s behavior: The control panel
(Figure 2(a)) enables users to set explicit intentions and scoped goals at the
project initialization. Users can specify specific benefits they expect to
gain from using Copilot in the programming sessions (e.g., to help them speed
up by serving as a prototyping tool or to help them learn as a programming
tutor). Users can also further customize settings by adjusting the
configuration on the right side of the control panel. We included options such
as suggestion scope, maximum line of suggestion, the timing of suggestion, and
type of validation (e.g., only suggest solutions that pass security checks).
The context slider (Figure 2(b)) enables users to control AI further during
the programming sessions. Users can drag the control bar next to each file
name or the code snippets to manually select the context that they would like
to include as part of the prompts for code generation.

(a) Control panel at the project initialization

(b) Sliders that enable users to select contexts for code generation

Context slider

Figure 2. Visual stimulus for design concepts that allow users to control AI
by setting and adjusting preferences.

Participants found both the control panel in project initialization and the
context control sliders during programming sessions helpful with their sense
of agency when using Copilot. The context sliders offer “more tools to guide
Copilot to the right answer” that allows developers to: “teach the model what
to do for me when I need it.” (D1) The control panel, on the other hand,
allows developers to customize how much and what kind of help they get from
Copilot at project initialization, which makes the AI more predictable and
controllable. For example, D10 once worried that Copilot might introduce
unnoticed security bugs, but the option in the control panel for users to
customize the type of suggestions could allow them to “only get suggestions
that have been scanned for any security vulnerabilities.” Indeed, control
mechanisms allowed users to customize a more reliable and helpful version of
Copilot, as D7 described, “if the performance is not reliable anymore or if
there’s suggestions that I don’t need, I would turn those off those function
just for precision and clarity.” Trust was fostered in the process since users
felt they had control over what and how the AI will make suggestions: “the
ability to set a boundary [for Copilot] and have it respect that boundary is
the core of building trust. If it can work in that boundary, then you trust it
more, and you can give it more permission.” (D4)

Interestingly, although we did not explicitly design the control settings for
the purpose of informing users’ expectations, developers thought that the
control mechanisms allowed them to develop more concrete expectations of what
Copilot can and cannot do. For example, D5 thought that seeing all
possibilities to control is almost like interactive documentation for copilot
functionalities that shows the full capacity of Copilot. D12 thought that the
control panel is especially helpful in project initialization because it
allows them to have “concrete expectations of what is going to happen,” such
as “how many lines of code there will be in suggestions.” Others imagined
experimenting with functionalities using the controls to understand the
strengths and limitations of Copilot in more targeted ways. For example, D7
imagined themselves to “turn off everything and see what each function does
and see which functions are more helpful,” which allowed them to have “the
full scope of what Copilot does.”

At the same time, developers expressed concern that “too much” control could
be a burden for users. For example, D5 expressed doubts about the usefulness
of the context slider due to its high interaction cost: “if I start spending a
bunch of time managing what context it has that the utility starts dropping
because I’m investing more time than am I getting anything more out of it.”
The choice of what type of controls to grant users and how to foreshadow their
impact on AI behaviors also needs careful considerations. A few developers
were confused about some of the current designs, and hoped to see more
examples in action and “visual cues for what this looks in the editor” (D5) or
“examples like how the code will be different, like turning it on and off”
(D7), on top of the textual description of the control mechanisms. D10 also
thought the presets could be helpful, “especially for someone who has no idea
about all these customization settings.

####  4.3.3. Offering indicators of model mechanism to support evaluation

Evaluation of the quality of individual suggestions forms the basis of
developers’ trust in AI-powered code generation tools. However, developers
find it challenging to evaluate suggestion quality using the current interface
due to the lack of quality indicators, as we discussed in Section 4.2.3. To
address this challenge, we explored three design concepts to help developers
reason about the quality of suggestions. The File-level familiarity
explanation (Figure 5(a)) communicates to users the model’s familiarity with
the context. For example, if the model has not seen input in the specific
programming language or using a particular library, the familiarity indicator
might turn yellow or red to indicate that the model is not familiar with the
context provided in the file. The Solution-level confidence explanation
(Figure 5(b)) indicates the model’s aggregated confidence of the solution both
in the editing window in the left pane and Copilot’s suggestion window in the
right pane. The Token level confidence/uncertainty explanation (Figure 5(b))
highlights tokens in the solution where the model is less confident.

(a) File-level familiarity explanation

File-level familiarity explanation

(b) Solution-level and token-level confidence explanations

Solution-level and token-level confidence explanations

Figure 3. Visual stimulus for design concepts that offer additional indicators
of model mechanism to support evaluation of AI suggestions.

Participants found that the explanations helps them with assessing and
validating the quality of code suggestions. For example, D2 thought that
Copilot’s file-level confidence indicates the helpfulness of Copilot in
nuanced and accurate ways: “If I know the Copilot is not very familiar with
this code, I am not going to have high expectations that the code the Copilot
produces will be accurate.” Similarly, D8 talked through how the transparency
signals could help them determine how much they should rely on Copilot: “low
familiarity can be a sign of vulnerability for the machine. If I know
[Copilot] is not good at it. I will be more vigilant, careful when I’m writing
the code myself or incorporating it… [the transparency signals] help me know
how much I should be relying on it.” These signals also help prompt for
subsequent user actions, telling users where in the AI suggestions they should
pay most attention. For instance, D5 would use the highlights of low-
confidence to guide their validation process and “target where I’m reviewing
the logic and say, yeah, it wasn’t super confident about these parts, so I
should look more closely at what it did there.”

At the same time, developers indicated that any explanation signals, including
the confidence scores, should be carefully designed to enrich user
understanding and scaffold judgement. For example, D5 thought that the
solution-level confidence indicators were not very helpful because: “Even that
20 percent, maybe I have to tweak five lines, it’s still a win.” D7 also
expressed a similar reluctance to fully rely on the numeric metrics: “I will
pick that a solution even though the confidence score is a little bit lower
than the others because it meets my needs better. Human judgment actually
knows that that is a better solution.” Indeed, many developers also reported
challenges in interpreting the context of model confidence numbers—the same
numeric score could communicate different information for different developers
in a variety of scenarios. Lastly, direct indication of model confidence also
introduced potential bias in users’ trust judgments, as users may be “more
likely to accept without critically thinking about a suggestion” or “reject a
valid solution or a valid suggestion based on low familiarity, even though
it’s a perfectly valid solution that is ultimately productive.” (D6)

##  5\. Discussion

###  5.1. Trust in generative AI tools

Previous research highlighted the importance of context in understanding
users’ trust in AI systems (Liao and Sundar, 2022a; Jacovi et al., 2021; Zhang
et al., 2022). However, there has been a lack of research that specifically
focuses on the context of AI-powered code generation tools. Our work addresses
this gap by providing a detailed account of developers’ trust in AI code
generation tools by synthesizing results from retrospective interviews with
developers who have used such tools in real-life scenarios. Echoing prior work
indicating that trust is situational and multidimensional (Hoffman, 2017;
Jacovi et al., 2021; Lyon et al., 2015), we categorized three layers of
developers’ trust in AI code generation tools, namely global trust (“overall
trust in the AI tool”), situational trust (“trust in the AI tool in a
particular situation”), and local trust (“trust in specific AI suggestion”).

Developers form a global trust understanding based on the overall assessment
of the benefits and risks of using AI code generation tools. This affects
their usage of AI tools at all and their goals when using them. For example,
some developers find AI code generation tools helpful in increasing
productivity, while other developers use them to improve code quality, as
learning tools, or as idea sounding boards. This finding aligns with prior
work on generative AI tools in creative tasks, where users form drastically
different expectations for the AI tools, ranging from a friend, collaborator,
student, and manager (Guzdial et al., 2019). Developers also form situational
trust in AI code generation tools, given that the performance of generative AI
models can vary by context. Building on prior work that has also shown that
trust is situational (Zhang et al., 2022), we found that factors such as the
complexity and stake of tasks often affect situational trust, leading
developers to use AI tools in selective scenarios. For example, we found that
many developers trust AI tools for mundane tasks such as writing unit tests or
generating boilerplate code but not for tasks that involve complex business
logic. Lastly, developers reevaluated their local trust for each AI
suggestion. Many developers in the interviews would validate suggestions every
time before accepting them, even if they trust the AI tools’ capability
overall. The local assessments of trust ultimately determined whether
developers would accept the suggestion from AI code generation tools or not.

In our study, we found that the design of AI system interfaces needs to
support all three layers of trust. We started exploring the different
affordances that can support developers in building each layer of trust. For
example, a feedback analytic dashboard that demonstrates the value of AI tools
(Figure 1(a) and 1(b)) can help establish global and situational trust while
explanations of specific suggestions (Figure 5(b) and 5(a)) can help
developers make local trust judgments. Our study also indicates that these
different layers of trust are distinct but also interconnected. For example,
each local experience of success and failure can accumulate to inform
situational and global trust. Developers’ global and situational trust in AI
can also affect the users’ strategy in making local trust judgments. This
interconnectedness was also translated into the designs. For example, our
design of confidence scores for individual suggestions (Figure 5(b)), while
designed to support local trust, also supports users building intuition for
situational trust overtime. In another example, our design of the control
dashboard (Figure 2(a)), whose usage is often informed by developers’
situational trust may hinder developers from building a comprehensive
understanding of AI tools’ capability if the user sticks with a particular
setup, which might affect global trust over time.

###  5.2. Design for appropriate user trust in AI code generation tools

Our findings solidify the responsibility of toolsmiths to offer users
opportunities to build and adjust their trust in AI tools. Responding to the
call for supporting _calibrated_ and warranted trust in AI (Liao and Sundar,
2022a; Jacovi et al., 2021; Buçinca et al., 2020; Lee and See, 2004; Zhang et
al., 2020; Passi and Vorvoreanu, 2022), our study further outlines three
aspects of the design space that are particularly important to help users
build appropriate trust in generative AI tools: (1) communicating AI
performance to support users form reasonable expectations on AI; (2) affording
users to set and adjust preferences to control the content and timing of AI
suggestions; and (3) offering indicators of model mechanism to support
evaluation of AI suggestions. Our findings also echo many design principles
that support human interaction with AI classification and prediction systems.
For example, similar to what general guidelines of human-AI collaboration
suggest (Amershi et al., 2019; Passi and Vorvoreanu, 2022), we show that
system design should support developers in anticipating and previewing AI
output and maintaining agency when using AI-powered code generation tools.

We highlight some important factors that future designers of generative AI
systems should consider when following our design recommendations. First, when
helping users to form reasonable expectations, toolsmiths should pay special
attention to supporting users in updating their expectations. In particular,
we need to design not only for building and expanding trust but also for
shrinking trust. On the one hand, as AI-code generation tools evolve and
become more capable, developers deserve the opportunity to be prompted to
update their trust perception accordingly. On the other hand, during this
evolution of tools, there will inevitably be times when the tools to do get
things wrong, causing trust to be broken (and rightfully so). In these
scenarios, designers of AI tools should be intentional about advising users
and repairing that trust afterward. In addition, future systems should also
find ways to help developers see a range of different use cases and how AI
performs in different scenarios, perhaps through crowd-sourced galleries as
shown in Cheng et al. (2022b), so that users could further extend their
understanding of the capacities and limitations of AI and update their trust.

Second, while introducing control options for users to set and adjust
preferences to AI, designers should be careful and intentional about the type
of control that they are granting users, as well as how to explain and help
users preview the outcome of different control options. In the design probe
study, we demonstrate examples of control options that allow users to
customize the timing, characteristics, and local context of AI suggestions.
Although the users reacted positively to the design probes, they also pointed
to the challenges of communicating and explaining the control options. Future
systems can explore how to introduce the control options more clearly. For
example, an interactive onboarding session could potentially address the issue
by demonstrating to users the effect of control options in action. In
addition, more controls come with more responsibilities. Designers of
generative AI systems need to be cautious about overburdening users with
decisions that they are not confident in making, and perhaps customize and
offer meta-support of control settings based on users’ background and
expertise.

Third, when introducing indicators of model mechanisms to support user
evaluation of AI suggestions, designers should strive to communicate the
indicators in ways that are both digestible and not overly decisive. In our
design probes, we explored the possibility of using model confidence scores as
indicators of code quality. While developers find the indicators useful for
evaluating solutions and guiding their actions, they also find them hard to
interpret and can potentially introduce biases. Future systems could provide
support for users to better understand the nature of confidence scores or
other indicators of model mechanisms, explaining how they are generated, what
they mean, and how they should be interpreted and used.

###  5.3. Limitations and future work

In this study, we investigated developers’ trust in AI-powered code generation
tools via qualitative interviews. Future research can build on our qualitative
investigation by implementing and evaluating interactive prototypes in
controlled experiments to better quantify the effects of interface design on
users’ trust. In addition, our sample was not balanced in gender due to the
limitation of sampling via social media and mailing lists. Further, since our
final sample is a voluntary sample, it is likely to have self-selection bias,
such as bias toward developers who are more interested in AI tools. Lastly,
our data in stage 1 were collected at a single company. Although we encouraged
participants to also discuss their experience outside of the work in stage 1
and intentionally sampled outside of the organization in stage 2, there may be
additional needs that we miss because of the specific organizational setting.

##  6\. Conclusion

AI code generation tools introduce new possibilities but also risks in the
context of programming. In this paper, we present a two-stage qualitative
study to understand developers’ trust in AI code generation tools. In stage 1,
we interviewed 17 developers to contextualize developers’ trust and identify
three main challenges with the current system design in helping developers
build appropriate trust. In stage 2, we explored three groups of design
concepts to address the above challenges and evaluated them with users.
Overall, the study demonstrates the importance of designing system features
that can help developers build appropriate trust with AI code generation
tools.

## References

  * (1)
  * Ope (2021) 2021\.  OpenAI Codex.  https://openai.com/blog/openai-codex/
  * Kit (2022) 2022\.  Kite is saying farewell.  https://www.kite.com/blog/product/kite-is-saying-farewell/ Section: product.
  * AIA (2023) 2023\.  AI Assistant for software developers | Tabnine.  https://www.tabnine.com/
  * Git (2023) 2023\.  GitHub Copilot · Your AI pair programmer.  https://github.com/features/copilot
  * MLp (2023) 2023\.  ML-powered coding companion – Amazon CodeWhisperer – Amazon Web Services.  https://aws.amazon.com/codewhisperer/
  * Agarwal et al. (2021) Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, Michael Muller, John Richards, Steven Ross, and Justin D. Weisz. 2021\.  Quality Estimation & Interpretability for Code Translation.  https://doi.org/10.48550/arXiv.2012.07581 arXiv:2012.07581 [cs].
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Barke et al. (2022) Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022\.  Grounded Copilot: How Programmers Interact with Code-Generating Models.  https://doi.org/10.48550/arXiv.2206.15000 arXiv:2206.15000 [cs].
  * Bird et al. (2023) Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023\.  Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools.  _Queue_ 20, 6 (Jan. 2023), Pages 10:35–Pages 10:57.  https://doi.org/10.1145/3582083
  * Boubin et al. (2017) Jayson G Boubin, Christina F Rusnock, and Jason M Bindewald. 2017.  Quantifying compliance and reliance trust behaviors to influence trust in human-automation teams. In _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ , Vol. 61. SAGE Publications Sage CA: Los Angeles, CA, 750–754.
  * Braun et al. (2019) Virginia Braun, Victoria Clarke, Nikki Hayfield, and Gareth Terry. 2019.  _Thematic Analysis_.  Springer Singapore, Singapore, 843–860.  https://doi.org/10.1007/978-981-10-5251-4_103
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Buçinca et al. (2020) Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020.  Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In _Proceedings of the 25th international conference on intelligent user interfaces_. 454–464.
  * Cheng et al. (2022a) Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel Tetreault, and Alejandro Jaimes-Larrarte. 2022a.  Mapping the Design Space of Human-AI Interaction in Text Summarization. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 431–455.  https://doi.org/10.18653/v1/2022.naacl-main.33
  * Cheng et al. (2022b) Ruijia Cheng, Ruotong Wang, Thomas Zimmermann, and Denae Ford. 2022b.  ”It would work for me too”: How Online Communities Shape Software Developers’ Trust in AI-Powered Code Generation Tools.  https://doi.org/10.48550/ARXIV.2212.03491
  * Clark et al. (2018) Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018\.  Creative writing with a machine in the loop: Case studies on slogans and stories. In _23rd International Conference on Intelligent User Interfaces_. 329–340.
  * Commission (2019) European Commission. 2019\.  Building Trust in Human-Centric Artificial Intelligence.  Retrieved September 1, 2022 from https://digital-strategy.ec.europa.eu/en/library/communication-building-trust-human-centric-artificial-intelli.
  * Dakhel et al. (2022) Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, and Jiang. 2022\.  GitHub Copilot AI pair programmer: Asset or Liability?  https://doi.org/10.48550/ARXIV.2206.15331
  * Daronnat et al. (2021) Sylvain Daronnat, Leif Azzopardi, Martin Halvey, and Mateusz Dubiel. 2021.  Inferring Trust From Users’ Behaviours; Agents’ Predictability Positively Affects Trust, Task Performance and Cognitive Load in Human-Agent Real-Time Collaboration.  _Frontiers in Robotics and AI_ 8 (2021), 194.  https://doi.org/10.3389/frobt.2021.642201
  * Das and Rad (2020) Arun Das and Paul Rad. 2020\.  Opportunities and challenges in explainable artificial intelligence (xai): A survey.  _arXiv preprint arXiv:2006.11371_ (2020).
  * Dohmke (2022) Thomas Dohmke. 2022\.  GitHub Copilot is generally available to all developers.  https://github.blog/2022-06-21-github-copilot-is-generally-available-to-all-developers/
  * Drozdal et al. (2020) Jaimie Drozdal, Justin Weisz, Dakuo Wang, Gaurav Dass, Bingsheng Yao, Changruo Zhao, Michael Muller, Lin Ju, and Hui Su. 2020.  Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 297–307.  https://doi.org/10.1145/3377325.3377501
  * Durumeric et al. (2014) Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicolas Weaver, David Adrian, Vern Paxson, Michael Bailey, and J. Alex Halderman. 2014.  The Matter of Heartbleed. In _Proceedings of the 2014 Conference on Internet Measurement Conference_ (Vancouver, BC, Canada) _(IMC ’14)_. Association for Computing Machinery, New York, NY, USA, 475–488.  https://doi.org/10.1145/2663716.2663755
  * Ernst and Bavota (2022) Neil A. Ernst and Gabriele Bavota. 2022.  AI-Driven Development Is Here: Should You Worry?  _IEEE Software_ 39, 2 (2022), 106–110.  https://doi.org/10.1109/MS.2021.3133805
  * Fan et al. (2019) Judith E. Fan, Monica Dinculescu, and David Ha. 2019\.  collabdraw: An Environment for Collaborative Sketching with an Artificial Agent. In _Proceedings of the 2019 on Creativity and Cognition_ _(C &C ’19)_. Association for Computing Machinery, New York, NY, USA, 556–561.  https://doi.org/10.1145/3325480.3326578
  * Fogg and Tseng (1999) B. J. Fogg and Hsiang Tseng. 1999.  The elements of computer credibility. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_ _(CHI ’99)_. Association for Computing Machinery, New York, NY, USA, 80–87.  https://doi.org/10.1145/302979.303001
  * Gero and Chilton (2019a) Katy Ilonka Gero and Lydia B Chilton. 2019a.  How a Stylistic, Machine-Generated Thesaurus Impacts a Writer’s Process.  In _Proceedings of the 2019 on Creativity and Cognition_. 597–603.
  * Gero and Chilton (2019b) Katy Ilonka Gero and Lydia B. Chilton. 2019b.  Metaphoria: An Algorithmic Companion for Metaphor Creation. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300526
  * Gonçales et al. (2019) Lucian Gonçales, Kleinner Farias, Bruno da Silva, and Jonathan Fessler. 2019.  Measuring the Cognitive Load of Software Developers: A Systematic Mapping Study. In _2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)_. 42–52.  https://doi.org/10.1109/ICPC.2019.00018
  * Guzdial et al. (2019) Matthew Guzdial, Nicholas Liao, Jonathan Chen, Shao-Yu Chen, Shukan Shah, Vishwa Shah, Joshua Reno, Gillian Smith, and Mark O. Riedl. 2019.  Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, Glasgow Scotland Uk, 1–13.  https://doi.org/10.1145/3290605.3300854
  * Hasselbring and Reussner (2006) W. Hasselbring and R. Reussner. 2006.  Toward trustworthy software systems.  _Computer_ 39, 4 (2006), 91–92.  https://doi.org/10.1109/MC.2006.142
  * Helgesson et al. (2019) Daniel Helgesson, Emelie Engström, Per Runeson, and Elizabeth Bjarnason. 2019.  Cognitive Load Drivers in Large Scale Software Development. In _2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE)_. 91–94.  https://doi.org/10.1109/CHASE.2019.00030
  * Hoffman (2017) Robert R Hoffman. 2017\.  A taxonomy of emergent trusting in the human–machine relationship.  _Cognitive Systems Engineering_ (2017), 137–164.
  * Holliday et al. (2016) Daniel Holliday, Stephanie Wilson, and Simone Stumpf. 2016\.  User trust in intelligent systems: A journey over time. In _Proceedings of the 21st international conference on intelligent user interfaces_. 164–168.
  * Jacovi et al. (2021) Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021.  Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 624–635.  https://doi.org/10.1145/3442188.3445923
  * Kawakami et al. (2022) Anna Kawakami, Venkatesh Sivaraman, Logan Stapleton, Hao-Fei Cheng, Adam Perer, Zhiwei Steven Wu, Haiyi Zhu, and Kenneth Holstein. 2022\.  “Why Do I Care What’s Similar?” Probing Challenges in AI-Assisted Child Welfare Decision-Making through Worker-AI Interface Design Concepts. In _Designing Interactive Systems Conference_. 454–470.
  * Lee and See (2004) John D. Lee and Katrina A. See. 2004.  Trust in Automation: Designing for Appropriate Reliance.  _Human Factors_ 46, 1 (March 2004), 50–80.  https://doi.org/10.1518/hfes.46.1.50_30392 Publisher: SAGE Publications Inc.
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Liao and Sundar (2022a) Q.Vera Liao and S. Shyam Sundar. 2022a.  Designing for Responsible Trust in AI Systems: A Communication Perspective. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 1257–1268.  https://doi.org/10.1145/3531146.3533182
  * Liao et al. (2020) Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020\.  Questioning the AI: informing design practices for explainable AI user experiences. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Liao and Sundar (2022b) Q Vera Liao and S Shyam Sundar. 2022b.  Designing for Responsible Trust in AI Systems: A Communication Perspective.  _arXiv preprint arXiv:2204.13828_ (2022).
  * Lipner (2004) S. Lipner. 2004\.  The trustworthy computing security development lifecycle. In _20th Annual Computer Security Applications Conference_. 2–13.  https://doi.org/10.1109/CSAC.2004.41
  * Lipton (2018) Zachary C Lipton. 2018\.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ 16, 3 (2018), 31–57.
  * Louie et al. (2020) Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020.  Novice-AI music co-creation via AI-steering tools for deep generative models. In _Proceedings of the 2020 CHI conference on human factors in computing systems_. 1–13.
  * Lyon et al. (2015) Fergus Lyon, Guido Mšllering, and Mark NK Saunders. 2015\.  _Handbook of research methods on trust_.  Edward Elgar Publishing.
  * Mayer et al. (1995) Roger C. Mayer, James H. Davis, and F. David Schoorman. 1995\.  An Integrative Model Of Organizational Trust.  _Academy of Management Review_ 20, 3 (July 1995), 709–734.  https://doi.org/10.5465/amr.1995.9508080335 Publisher: Academy of Management.
  * Münscher and Kühlmann (2015) Robert Münscher and Torsten M Kühlmann. 2015.  Using critical incident technique in trust research.  In _Handbook of research methods on trust_. Edward Elgar Publishing, 210–222.
  * Oh et al. (2018) Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018\.  I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3173574.3174223
  * O’Connor et al. (2019) Annette M O’Connor, Guy Tsafnat, James Thomas, Paul Glasziou, Stephen B Gilbert, and Brian Hutton. 2019\.  A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?  _Systematic reviews_ 8, 1 (2019), 1–8.
  * Passi and Vorvoreanu (2022) Samir Passi and Mihaela Vorvoreanu. 2022.  Overreliance on AI: Literature review.  (2022).
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022\.  Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In _2022 IEEE Symposium on Security and Privacy (SP)_. 754–768.  https://doi.org/10.1109/SP46214.2022.9833571
  * Perry et al. (2022) Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2022.  Do Users Write More Insecure Code with AI Assistants?  https://doi.org/10.48550/ARXIV.2211.03622
  * Sarkar et al. (2022) Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022.  What is it like to program with artificial intelligence?  https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs].
  * Sobania et al. (2022) Dominik Sobania, Dirk Schweim, and Franz Rothlauf. 2022\.  A comprehensive survey on program synthesis with evolutionary algorithms.  _IEEE Transactions on Evolutionary Computation_ (2022).
  * Sun et al. (2022) Jiao Sun, Q Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D Weisz. 2022.  Investigating Explainability of Generative AI for Code through Scenario-based Design. In _27th International Conference on Intelligent User Interfaces_. 212–228.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, 1–7.  https://doi.org/10.1145/3491101.3519665
  * Vereschak et al. (2021a) Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux. 2021a.  How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (Oct. 2021), 327:1–327:39.  https://doi.org/10.1145/3476068
  * Vereschak et al. (2021b) Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux. 2021b.  How to evaluate trust in AI-assisted decision making? A survey of empirical methodologies.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (2021), 1–39.
  * Weisz et al. (2021) Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021.  Perfection not required? Human-AI partnerships in code translation. In _26th International Conference on Intelligent User Interfaces_. 402–412.
  * Weisz et al. (2022) Justin D Weisz, Michael Muller, Steven I Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. 2022.  Better together? an evaluation of ai-supported code translation. In _27th International Conference on Intelligent User Interfaces_. 369–391.
  * Wendt et al. (2004) Eva Wendt, Bengt Fridlund, and Evy Lidell. 2004\.  Trust and confirmation in a gynecologic examination situation: a critical incident technique analysis.  _Acta obstetricia et gynecologica Scandinavica_ 83, 12 (2004), 1208–1215.
  * Witschey et al. (2015) Jim Witschey, Olga Zielinska, Allaire Welk, Emerson Murphy-Hill, Chris Mayhorn, and Thomas Zimmermann. 2015.  Quantifying Developers’ Adoption of Security Tools. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. Association for Computing Machinery, New York, NY, USA, 260–271.  https://doi.org/10.1145/2786805.2786816
  * Xiao et al. (2014) Shundan Xiao, Jim Witschey, and Emerson Murphy-Hill. 2014\.  Social Influences on Secure Development Tool Adoption: Why Security Tools Spread. In _Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing_ (Baltimore, Maryland, USA) _(CSCW ’14)_. Association for Computing Machinery, New York, NY, USA, 1095–1106.  https://doi.org/10.1145/2531602.2531722
  * Yañez-Gallardo and Valenzuela-Suazo (2012) Rodrigo Yañez-Gallardo and Sandra Valenzuela-Suazo. 2012.  Critical incidents of trust erosion in leadership of head nurses.  _Revista Latino-Americana de Enfermagem_ 20 (2012), 143–150.
  * Yang et al. (2023) Qian Yang, Yuexing Hao, Kexin Quan, Stephen Yang, Yiran Zhao, Volodymyr Kuleshov, and Fei Wang. 2023.  Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems.  https://doi.org/10.1145/3544548.3581393
  * Zhang et al. (2020) Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020\.  Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 295–305.  https://doi.org/10.1145/3351095.3372852
  * Zhang et al. (2022) Yixuan Zhang, Nurul Suhaimi, Nutchanon Yongsatianchot, Joseph D Gaggiano, Miso Kim, Shivani A Patel, Yifan Sun, Stacy Marsella, Jacqueline Griffin, and Andrea G Parker. 2022.  Shifting Trust: Examining How Trust and Distrust Emerge, Transform, and Collapse in COVID-19 Information Seeking. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, 1–21.  https://doi.org/10.1145/3491102.3501889
  * Ziegler et al. (2022) Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022.  Productivity Assessment of Neural Code Completion. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ (San Diego, CA, USA) _(MAPS 2022)_. Association for Computing Machinery, New York, NY, USA, 21–29.  https://doi.org/10.1145/3520312.3534864

##  Appendix A Participant information

Table 1. The participants of Study 1. The Column _Exp_ indicates the years of
programming experience. The _Job Title_ was self reported by participants.

ID |  Tool (Frequency) |  Gender | Race |  Age | Education | Job Title |  Exp
---|---|---|---|---|---|---|---
P1 |  GitHub Copilot (Daily) |  Male | White |  25-34 | Bachelor degree | Researcher |  7
P2 |  GitHub Copilot (Weekly) |  Male | Asian |  18-24 | Bachelor degree | Program Manager |  5
P3 |  GitHub Copilot (Monthly) |  Male | White |  45-54 | Bachelor degree | Software Engineer |  25
P4 |  GitHub Copilot (Monthly), Tabnine (Yearly) |  Male | White |  25-34 | Bachelor degree | Software Engineer |  12
P5 |  GitHub Copilot (Daily), Tabnine (Daily) |  Male | Asian Indian |  25-34 | Ongoing Masters degree | Software engineer |  9
P6 |  GitHub Copilot (Daily) |  Female | White |  25-34 | Bachelor degree | Software Engineer |  4
P7 |  GitHub Copilot (Monthly) |  Male | White |  25-34 | PhD degree | Software Engineer |  22
P8 |  GitHub Copilot (Weekly) |  Male | Middle Eastern |  35-44 | Bachelor degree | Security Engineer |  20
P9 |  GitHub Copilot (Weekly) |  Male | Asian |  25-34 | Master degree | Software Engineer |  8
P10 |  GitHub Copilot (Daily) |  Male | Black or African American |  25-34 | High school diploma | Software Engineer |  8
P11 |  GitHub Copilot (Daily) |  Male | White |  25-34 | Bachelor degree | Software Engineer |  9
P12 |  GitHub Copilot (Daily) |  Male | White |  35-44 | Master degree | Software Engineer |  21
P13 |  GitHub Copilot (Daily) |  Male | White |  18-24 | Bachelor degree | Software engineer |  6
P14 |  GitHub Copilot (Never) |  Female | Asian |  25-34 | Bachelor degree | Software engineer |  8
P15 |  GitHub Copilot (Daily) |  Male | Hispanic or Latino |  18-24 | High school diploma | Software engineer intern |  3
P16 |  GitHub Copilot (Daily) |  Male | White |  18-24 | High school diploma | Software engineer intern |  2
P17 |  GitHub Copilot (Never) |  Male | Asian |  25-34 | Bachelor degree | Software engineer |  5

Table 2. The participants of Study 2. The Column _Exp_ indicates the years of
programming experience.

ID | Copilot | Gender | Race | Age | Education | Exp
---|---|---|---|---|---|---
D1 | I use the tool regularly | Female | White | 35-44 | PhD degree | 4
D2 | I use the tool regularly | Male | White | 55-64 | Bachelor degree | 45
D3 | I’ve tried the tool but no longer using it | Male | Asian | 35-44 | Bachelor degree | 5
D4 | I use the tool regularly | Male | Black or African American | 25-34 | Bachelor degree | 6
D5 | I use the tool regularly | Male | White | 25-34 | Bachelor degree | 15
D6 | I recently started using the tool | Male | White | 25-34 | Master degree | 7
D7 | I recently started using the tool | Female | Asian | 25-34 | Master degree | 6
D8 | I use the tool regularly | Female | Asian | 18-24 | Bachelor degree | 3
D9 | I use the tool regularly | Male | Asian | 25-34 | Master degree | 6
D10 | I’ve tried the tool but no longer using it | Male | Middle Eastern | 18-24 | High school diploma | 6
D11 | I use the tool regularly | Male | Black or African American | 35-44 | Master degree | 11
D12 | I use the tool regularly | Male | Asian | 25-34 | Master degree | 7

##  Appendix B Study material for stage 1

###  B.1. Example interview questions

The retrospective interviews were semi-structured, so the questions below only
represent a general structure of the interviews. In the actual interviews, we
followed up with participants whenever they mentioned topics relevant to their
understanding of trust and the challenges they have in building appropriate
trust.

  * •

Could you tell us a bit more about the kind of programming project or tasks
that you work on? What kind of development activities (e.g, front-end) are you
typically involved in?

  * •

What experience do you have with AI-powered code generation tools?

  * •

How do you trust the AI tool?

  * •

Can you walk me through the significant moments you collected?

    * –

What was your task?

    * –

How did you interact with the tool? [Feel free to share screen]

    * –

How do these interactions affect your trust in the tool? Why?

  * •

Now think about your general experience interacting with the tool. How would
you define trust?

  * •

Where do you think the trust come from?

  * •

What tasks do you trust/distrust the tool to do? Why?

  * •

Were there moments where you trusted the AI tool but later realized that you
shouldn’t?

  * •

Were there moments where you didn’t trust the AI tool but later realized that
you should?

  * •

How has your perception of trust in the tool changed over time?

  * •

How would you want to improve the design of AI-powered code generation tool so
that you can trust it more appropriately?

###  B.2. Message sent to participants to collect significant moments

> Hi [Participant Name], Thank you for signing up for the experience in AI-
> powered code generation tools research study. We would like to invite you
> for an interview to learn more about your experience. To prepare for the
> interview, we would like to invite you to collect significant moments in
> your experience using AI-powered code generation tools (e.g., Copilot) in
> the next few days. Our goal is for you to collect these significant moments,
> so that you can reflect on your experience more concretely in the interview.
> Specifically, please aim to share 1 to 3 significant moments each day. Some
> examples of significant moments are when you are appreciative of, frustrated
> by, or hesitant/uncertain to use the AI-powered code generation tool (e.g.,
> copilot). For each time you share, you can use one or two sentences to
> describe the instance, take a screenshot or share a snippet of code. You can
> share these in our chat directly. We will also send you a quick reminder
> message everyday morning during the week. In the case that you do not use
> AI-powered code generation tools during the day, it would also be helpful to
> share a quick update in the chat (e.g., did not use AI tools today). We will
> schedule an interview with you after you successfully complete the
> preparation phase (collect several significant moments).

##  Appendix C Study material for stage 2

###  C.1. Example design probe questions

We begin the interview by briefing participants that:

  * •

We are evaluating the prototype, not you, so feel free to comment on anything.

  * •

Do not worry about the technical implementation of the designs. The purpose of
the session is to get feedback on the concept of designs, instead of the
feasibility of the designs.

  * •

Do not worry about usability (e.g., layout, color, style) of the design

  * •

Feel free to think aloud as you look at the design prototypes

  * •

The code snippets are only placeholders. Try to imagine how you will use the
design in your daily workflow.

Next, we ask the following questions to understand participants’ understanding
of trust.

  * •

What experience do you have with AI-powered code generation tools, such as
copilot?

  * •

How do you trust the AI tool?

  * •

How do you define trust?

  * •

Are there challenges in knowing what to expect from the AI tool?

  * •

Are there challenges in integrating the AI tool into your workflow?

  * •

How would you want to design the interaction with the AI tool differently so
that you can better judge when to trust the AI tool or not?

Present and give brief explanations of the mockups to participants one by one.
For each mockup, ask the following questions:

  * •

What do you think of this design?

  * •

How might you use this feature in your daily coding task?

  * •

Thinking about your overall experience interacting with the tool, to what
extent do you think it will help you better judge when to trust copilot or
not?

  * •

Which of all the mockups is the most helpful in helping you judge when to
trust copilot or not?

  * •

What other features do you like to add to this prototype?

  * •

What other features do you like to remove or change to this prototype?

###  C.2. Design concepts shown in the study

Below are the design prototypes that we showed to the study participants.

(a) Control panel

(b) Context slider

Context slider

Figure 4. Group 1: Ways to control AI

(a) File-level familiarity explanation

File-level familiarity explanation

(b) Solution-level and token-level confidence explanations

Solution-level and token-level confidence explanations

Figure 5. Group 2: Quality indicators of AI suggestions

Figure 6. Group 3: Feedback analytics dashboard