regular

#  “A Tale of Two Prompts”: Investigating the Impact of Guidance and
Interaction Strategies for LLM Use on Learner Performance and Perception

Harsh Kumar<EMAIL_ADDRESS>0000-0003-2878-3986 University of Toronto40
St. George StreetTorontoOntarioCanadaM5S 2E4 ,  Ilya Musabirov
<EMAIL_ADDRESS>University of TorontoCanada ,  Mohi Reza
<EMAIL_ADDRESS>University of TorontoCanada ,  Jiakai Shi
<EMAIL_ADDRESS>University of TorontoCanada ,  Anastasia Kuzminykh
<EMAIL_ADDRESS>University of TorontoCanada ,  Joseph Jay
Williams<EMAIL_ADDRESS>University of TorontoCanada and  Michael
Liut<EMAIL_ADDRESS>0000-0003-2965-5302 University of Toronto
MississaugaCanada

###### Abstract.

Personalized chatbot-based teaching assistants can be crucial in addressing
increasing classroom sizes, especially where direct teacher presence is
limited. Large language models (LLMs) offer a promising avenue, with
increasing research exploring their educational utility. However, the
challenge lies not only in establishing the efficacy of LLMs but also in
discerning the nuances of interaction between learners and these models, which
impact learners’ engagement and results. We conducted a formative study in an
undergraduate computer science classroom (N=145) and a controlled experiment
on Prolific (N=356) to explore the impact of four pedagogically informed
guidance strategies and the interaction between student approaches and LLM
responses. Direct LLM answers marginally improved performance, while refining
student solutions fostered trust. Our findings suggest a nuanced relationship
between the guidance provided and LLM’s role in either answering or refining
student input. Based on our findings, we provide design recommendations for
optimizing learner-LLM interactions.

large language models, tutoring systems, human-ai collaboration, artificial
intelligence in education

††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs: Social and professional topics
Computing education††ccs: Human-centered computing Empirical studies in
interaction design††ccs: Human-centered computing Interaction techniques

##  1\. Introduction

In the contemporary educational landscape, the value of personalized attention
in enhancing student learning outcomes is well recognized, especially when
considering the 2 Sigma problem by Benjamin Bloom (Bloom, 1984), where Bloom
found that students who were tutored one-to-one using mastery learning
techniques performed two standard deviations (or “2 sigma”) better than
students who learned via conventional instructional methods. However, the
continuous expansion of classrooms poses challenges: With a growing demand for
tailored teaching approaches, there is a notable shortage of human teaching
assistants to provide individualized attention, leading to a critical
bottleneck in achieving optimal educational impacts.

To address this issue, previous research on Intelligent Tutoring Systems (ITS)
and Intelligent Teaching Assistants (ITA) has pioneered adaptive learning
experiences and personalized instructional feedback (Weitekamp et al., 2020;
Anderson et al., 1995). Furthermore, the rapid emergence of large language
models (LLMs) in recent years has created even more promising opportunities
for technological support of students’ personalized attention needs. (Zhou et
al., 2022; Sejnowski, 2023). These models, with their ability to comprehend
and generate human-like text, have the potential to reshape the landscape of
personalized education. Consequently, researchers started exploring the
potential advantages of the development of LLM-based agent tutors as potential
surrogates for human teaching assistants (Rahman and Watanobe, 2023; Dergaa et
al., 2023; Baidoo-Anu and Owusu Ansah, 2023).

The dialogic nature of LLM-based agents, their ability to adapt to
individualized learning patterns, and their ease of integration into diverse
educational contexts make them a potential boon for educational frameworks
(Farrokhnia et al., 2023). While learning environments vary between subjects
and educational levels, the versatility of LLMs allows their integration into
these specialized contexts, providing a unified platform for varied
educational needs. These LLM-based systems can support different interaction
strategies and, to effectively adapt them to pedagogical needs, we need to
understand what strategies should be implemented in learning support systems.

Although there is a considerable volume of existing research on LLM
applications, a noticeable gap remains when applied to education. The question
is not only about the feasibility of these LLM-based tutors, but also about
the ambiguity surrounding the effects of different interaction strategies on
students. In particular, the diversity of pedagogical approaches to how a
teacher should interact with a student poses a question of choosing
appropriate interaction strategies when designing LLM-based tutors. For
instance, how could these strategies influence students’ reception of the
assistance, self-assessment, and their subsequent performance? In education,
the pedagogical and interaction effects are inherently interconnected, but
discerning their nuanced impacts is essential for the effective design and
implementation of LLM tutors.

To gain a deeper understanding of student interactions with LLMs in
educational contexts, in this paper, we distinguish student-agent interaction
into three key elements: the Guidance Received, which refers to the directions
or suggestions students obtain for using LLMs; the Learner Approach,
indicating whether learners first consult the LLM or attempt problem-solving
independently; and the LLMs’ Response, emphasizing the importance of dialogue
quality with LLM. These components are intertwined, with shifts in one
potentially influencing the others. We then empirically investigated these
influences through two user studies. In both studies, we explored four
variations of the first element, using four distinct guidance strategies:
list-based suggestions, example-based instruction, metacognitive questioning
(making learners think about their use of LLMs before actual use) and a
problem-solving-then-refinement approach (making learners solve a problem
first and then refine it with LLMs). These four guidance strategies manifested
in two distinct student approaches (second element): directly consulting the
LLM or relying on self-first problem-solving. Finally, we manipulated the
third element – LLMs’ Response – through pre-interaction manipulations (Brown
et al., 2020; Kumar et al., 2022)). These combinations allowed us to explore
the intertwined relationship between the guidance provided to learners and the
initial prompts given to the LLM in this tale of two prompts.

While there is previous research on student-AI interaction, LLMs seem to
provide different levels of interaction experience and convincingness, even in
the presence of errors. Thus, studying the multidimensional picture of student
perceptions and their dynamics and analyzing them in a formative experimental
setup, combining preliminary quantitative measures as well as extensive
qualitative feedback is needed at this initial stage. Our formative study in
an undergraduate computer science classroom ($N=145$) tasked students to solve
an assignment while receiving help from an LLM-based chatbot. We explored the
impact of our four guidance strategies, and the resulting two approaches, on
the student’s performance (in the assignment, as well as in the final exam 2
weeks after the assignment), system perception (helpfulness, confidence in
LLM’s response, tolerance of mistakes, willingness to interact again), and
their self-assessment (change in their self-efficacy, confidence in their
answers). As we discuss further, from a statistical standpoint, this setting
is traditional in its inherent limitations of sample size by course
enrollment, student dropout, and selective engagement. It allows us to use it
only to gather formative data for different conditions. In addition, exposing
students to experimental stimuli also allows for gathering instances of
qualitative feedback specific to different stimuli.

Informed by findings from formative deployment, we conducted a controlled
study on Prolific (Palan and Schitter, 2018) in which crowdworkers (N=356)
were asked to solve math problems at the elementary and high school level with
the help of an LLM chatbot. Besides manipulating the different forms of
guidance given to students, the online controlled setting gave us affordances
to compare a pre-prompted LLM with an unprompted LLM. Through a 2 (LLM
Chatbot: Prompted vs. Unprompted) x 5 (Guidance strategy: G1-List of
Suggestions vs. G2-Example-based Instruction vs. G3-Metacognitive Questioning
vs. G4-Solve then refine with LLM vs. No Instruction), we explored the impact
of interaction strategies on the learners’ performance and their confidence
(in LLM’s responses, ability to use LLMs, and their Math-solving ability)
across the different stages of interaction (before receiving guidance,
immediately after receiving guidance, and after task completion). We did not
find statistically significant differences in Learner + AI task performance
across different Guidance Types and LLM Types (prompted vs unprompted), we
observe interesting patterns in confidence in the ability to use LLMs to solve
math problems and trust in LLM responses over different stages of interaction.
Guidance shortly boosts confidence in the ability to use LLM, but both
measures significantly drop after learners’ attempt to solve real tasks with
LLM use. This might suggest a partially negative interaction experience on
solving tasks, as well as the need for extra learning support for LLM use in
the particular context.

We use these experiments to highlight ways of operationalizing and measuring
the variables that are important to pay attention to, as there are many
concurrent processes and conflicting trade-offs. Our experiments and analysis
can be valuable for anyone who wants to investigate the use of LLMs to enable
learning. In summary, our contributions are as follows.

  * •

Pedagogically Informed Design of Guidance Strategies and Approaches: We
proposed a pedagogically informed framework detailing guidance strategies and
approaches, laying the foundation for optimizing student-LLM interactions.
This could serve as a blueprint for educators and developers to use LLMs in
instructional contexts. [Refer to Section 3].

  * •

Empirical Insights from Classroom Interactions with LLMs: Through our
formative evaluation in a classroom, we share findings from authentic student-
LLM interactions. This first-hand exploration provides insights into the real-
world implementation of LLMs in educational settings. [Refer to Section 4].

  * •

Insights from Online Controlled Experiment with Crowdworkers: Capitalizing on
the flexibility of an online environment, we conducted a controlled experiment
with crowdworkers on Prolific. This in-depth examination allowed us to probe
deeper into the intricacies of Learner-LLM interactions. [Refer to Section 5].

  * •

Design Recommendations for Future Learner-LLM Implementations: Through our
empirical data and analysis, we offer targeted recommendations for various
stakeholders, including instructors, instructional designers, and HCI
researchers [Refer to Section 6].

Our work can inform the design of systems for learner-LLM interactions,
ensuring the effective and empathetic use of LLMs in pedagogical contexts. In
the following section (Section 2), we start by describing the related work
done in incorporating LLMs in classroom, and highlight the need to explore
different LLM interaction strategies. In Section 3, we describe the design
space of learner-LLM interactions, and propose interaction designs that are
evaluated in Sections 4 and 5. We conclude with a discussion of key findings
and operationalize our findings in a set of design considerations.

##  2\. Related Work

###  2.1. Integrating LLMs in Education

Conversational AI, described as a ”game-changer for science,” (Van Dis et al.,
2023), is making waves in educational circles (Dwivedi et al., 2023; Denny et
al., 2023c). As Large Language Models (LLMs) inch closer to human-level
language capabilities (Zhou et al., 2022), educators and researchers alike are
both thrilled (Rahman and Watanobe, 2023; Dergaa et al., 2023; Baidoo-Anu and
Owusu Ansah, 2023) and concerned (Rahman and Watanobe, 2023; Dergaa et al.,
2023) about their integration into diverse learning environments. Notably,
tools like ChatGPT (cha, [n. d.]), Bard (bar, [n. d.]), BingChat (bin, [n.
d.]), and KhanMigo (kha, [n. d.]) collectively serve hundreds of thousands of
students, signaling the significant footprint of such technologies (Baidoo-Anu
and Owusu Ansah, 2023). Educators in various disciplines including computer
science (Jacques, 2023), math (Pardos and Bhandari, 2023), language learning
(Kohnke et al., 2023), and even medicine (Lee, 2023) are investigating ways to
safely incorporate chat-based LLMs into their curricula.

Many institutions now advise instructors to clarify their stance on generative
AI in course syllabi—whether they prohibit, limit, or endorse its use (Haque
et al., 2022; Jalil et al., 2023). At this pivotal juncture, while such tools
can elevate or impede learning, there’s a consensus on the need to harness
AI’s benefits optimally (Deng and Lin, 2022; Qureshi, 2023; Dwivedi et al.,
2023; Adiguzel et al., 2023; Sallam, 2023). It has become paramount for
educators to comprehend and leverage these advantages (Baidoo-Anu and Owusu
Ansah, 2023; Sok and Heng, 2023). This paper contributes to our understanding
of how to guide students toward the effective use of conversational AI
systems. Our approach centers on studies involving chat-based prototype
interfaces based on LLM, emblematic of the conversational AI tools currently
in use by students across numerous educational settings.

###  2.2. Prompt Engineering is Deceptively Difficult

The response from LLM-based chatbots is primarily steered by text-based
prompts (Wu et al., 2022; Zhou et al., 2022). However, recent research has
indicated that crafting effective prompts is challenging (Zamfirescu-Pereira
et al., 2023), particularly for those without a deep understanding of AI
(Abdellatif et al., 2020; Zamfirescu-Pereira et al., 2023; Fiannaca et al.,
2023). The conversation-like interface of these chatbots mimics human
interaction (Smestad and Volden, 2019), potentially misleading students into
thinking that prompt-writing is straightforward as talking to humans
(Hofstadter, 1995). This gap between perceived simplicity and the actual
complexity of effective prompt-writing can breed overconfidence. Yet, when the
chatbot fails to respond as expected, students’ trust in the AI system (Denny
et al., 2023a) can wane, leading to frustration (Pérez et al., 2020). In
response to these challenges, informal communities like the
r/aipromptprogramming and r/ChatGPT subreddits have emerged to share best
practices for prompt-writing. Simultaneously, researchers are starting to
establish formal guidelines (Denny et al., 2023b; Liu and Chilton, 2022; Ekin,
2023), but given the nascent nature of this field, much is left to be
explored. To enhance student success with LLMs, there is a need to convert
these guidelines into a more digestible format, thereby improving LLM literacy
(Zamfirescu-Pereira et al., 2023; Yang, 2023). This paper delves into the
creation of these student-centric guidelines, rendering them into clear
instructions for LLM use. More specifically, we evaluate how various guidance
strategies—including direct instruction, example-based teaching, metacognitive
questioning, and worked examples—affect students’ performance, self-
confidence, trust in the chatbot, and their perceptions of its efficacy.

###  2.3. Different Forms of Guidance

In prior pedagogical literature, various forms of instructional guidance for
students have been explored (Bonawitz et al., 2011). Among these, direct
instruction (White, 1988), example-based instruction (Wittwer and Renkl,
2010), and metacognitive strategies (Livingston, 2003; Şen, 2009)—such as
posing reflective questions (Ellis et al., 2014; Alt and Raichel, 2020)—have
gained significant traction among educators. This paper narrows its focus to
these techniques, owing to their documented effectiveness in prior research
(Gersten et al., 1986; Adams and Engelmann, 1996; Zendler and Klein, 2018;
Anderson et al., 2007; Van Gog and Rummel, 2010; Palincsar, 1986; Bourner,
2003). However, a gap persists in understanding their specific applicability
to LLMs, especially considering the unique challenges associated with crafting
effective prompts for these systems, as elucidated in the previous section.
Further complicating the landscape is the dual-edged nature of LLMs: while
they stand as powerful tools for problem-solving and learning (Kasneci et al.,
2023; Qureshi, 2023), there’s potential for students to misuse them (Kooli,
2023), sidestepping the learning journey and directly seeking solutions. Such
shortcuts not only stifle genuine learning but also risk propagating
misinformation, especially if an LLM, hallucinates and provides incorrect
answers (Najafali et al., 2023). A proposed remedy, endorsed by some in the
community, involves implementing guardrails on LLM responses (Liffiton et al.,
2023; Goyal et al., 2023; Chen et al., 2023a). These guardrails, though
beneficial in curbing potentially harmful responses, have proven challenging
to enforce effectively, with users finding ways—or ”jailbreaks”—to coax
unintended outputs from LLMs (Wei et al., 2023; Li et al., 2023; Liu et al.,
2023). Beyond just guardrails, enhancing students’ proficiency in prompt-
writing is vital, as is fostering a culture of LLM usage that truly bolsters
learning. We champion this multi-pronged strategy in our paper, and provide
insight into how traditional guidance methods apply to student use of LLMs
systems in a CS course.

###  2.4. Learners’ Decision-Making in Tool Usage

The choice of when to consult external tools during problem solving isn’t
exclusive to the realm of LLMs. In broader education research, students’
decisions to access resources - be it textbooks, online forums, peers, or
tutors - often comes down to a balance between self-efficacy, perceived task
difficulty, and the accessibility of the tool.

Self-Efficacy and Task Difficulty. Research has shown that learners with high
self-efficacy and (over-) confidence can influence learning behavior (Moores
and Chang, 2009; Vancouver et al., 2002). High self-efficacy students might
first attempt to tackle problems independently. However, if they perceive the
task as excessively challenging or outside of their competence, they are more
likely to consult external sources earlier in their problem-solving process.

Availability and Perceived Utility of the Tool. The ease of accessing a tool
and the learner’s belief in its utility can impact when and how often they use
it. For instance, the popularity of online forums in CS courses can be
attributed to the instantaneous, community-validated feedback they provide
(Glassman et al., 2015).

Balancing Independence and Reliance. The idea of balancing self-reliance with
tool reliance fits well in the framework of self-regulated learning(Karabenick
and Dembo, 2011). In many settings, students are trained to strike a balance,
ensuring that they understand the basic concepts, while not hesitant to seek
help when needed.

In the context of LLMs, the aforementioned considerations take on a nuanced
significance. The availability and perceived efficacy of LLMs might encourage
more immediate reliance, potentially overshadowing traditional problem-solving
approaches. Our paper’s exploration of different student approaches offers
insights into these nuanced behaviors in the specific context of LLM
interactions.

##  3\. Design Considerations for Learner Interaction with LLMs

As language models continue to find their way into educational settings,
understanding how students interact with LLM-based tools becomes imperative.
In our exploration of learner interactions with LLMs, we identified three
critical factors influencing these engagements:

  1. (1)

Guidance Received: The directions or suggestions that students receive can
shape their entire interaction experience. The form and content of this
guidance can determine how students approach and utilize LLMs to their
advantage (Section 3.1).

  2. (2)

Student Approach: Students’ initial inclination, whether to consult the LLM
immediately or to first wrestle with the problem on their own, can have
implications for their learning process. The strategies they adopt can be
influenced by the guidance they have received and can significantly vary in
efficacy (Section 3.2).

  3. (3)

LLMs’ Response: While guidance and approach set the stage, the actual dialogue
with the LLM remains paramount. The quality, clarity, and relevance of the
LLMs’ responses can make or break the learning experience (Section 3.3).

Each of these components does not exist in isolation but intertwines and
influences the others. It is a dynamic relationship where modifying one aspect
might lead to shifts in the others. Grasping these intricate relationships is
key for educators, designers, and policymakers who want to harness the full
potential of LLMs in the learning environment.

###  3.1. Guidance Strategies for LLM Engagement

In Section 2.3, we discussed how our guidance strategies drew insights from
previous work in the pedagogical literature (Bonawitz et al., 2011; Gersten et
al., 1986; Adams and Engelmann, 1996; Zendler and Klein, 2018; Anderson et
al., 2007; Van Gog and Rummel, 2010; Palincsar, 1986; Bourner, 2003). In this
section, we revisit some of those insights and describe the specifics of our
design and the rationale behind each guidance strategy. Figure 1 shows the
format of each guidance strategy used in our study.

G1: List of Suggestions. We devised a set of suggestions for LLM chatbot use,
as depicted in Figure 1A. Given the limited existing research on LLM guidance
for students, our suggestions drew from prominent online sources (Pangu, 2023;
Miquelino, [n. d.]; Mollick, 2023b). Recognizing the growing classroom trend
of prescribed LLM instructions (eth, 2023; Mollick, 2023a), G1 serves as a
representative model for evaluating its efficacy against other guidance
strategies.

G2: Example-based instruction. Leveraging examples is an intuitive approach to
tool introduction (Lee, 2004). However, the choice of examples plays a crucial
role. For our classroom deployment, we took examples of interactions from
previous interview studies with students [REDACTED]. These examples, validated
by an instructor and a teaching assistant, showcased interactions where
students found the LLM beneficial. We showed participants two examples of
interactions. Participants were presented with two such exemplary
interactions, as illustrated in Figure 1B.

G3: Metacognitive questioning (making learners think about their use of LLMs
before actual use). Framing the initial question effectively is crucial when
seeking help, more so with LLM chat tools where the initial query shapes
subsequent responses (Daniel, 2017; Zamfirescu-Pereira et al., 2023). To
enable this skill, participants were presented with an example problem and
prompted to draft their opening question to the LLM chatbot. Subsequently,
they reflected on the potential benefits of their chosen approach (Figure 1C).

G4: Solve, then refine with LLM. Worked examples, step-by-step illustrations
of the process required to complete a task, are established effective learning
tools (Sweller, 2006). In this multistep strategy, we first ask the students
to solve a problem and provide the solution. They then engaged with the LLM
chatbot to refine their initial answers (Figure 1D).

Figure 1. Different forms of guidance strategies.

###  3.2. Approaches to Leverage LLMs in Problem Solving

The manner in which students engage with LLM-based tools is often influenced
by the guidance or instructions they receive. The sequence—whether students
consult the LLM first or try to solve problems on their own—has implications
for both their learning process and the effectiveness of the LLM as a tool.
This builds on research in ”when” is it better to provide hints for learning
(Razzaq and Heffernan, 2010).

LLM-First Approach (LFA): Students begin by seeking input from the LLM. This
approach can assist those who need a foundation or direction before delving
into problem-solving, ensuring they are on the right track from the onset.
However, it may also promote dependency on the LLM, potentially limiting
independent problem-solving capabilities.

Self-First Approach (SFA): Students first attempt to solve the problem
independently and then consult the LLM for refinements. This method promotes
independent thinking and can boost confidence in one’s problem-solving
abilities. The LLM acts as a secondary check, reinforcing correct
methodologies or suggesting improvements. Conversely, there might be a risk
that students get stuck or frustrated if their initial attempts are
misdirected.

Guidance strategies, as detailed in Section 3.1, influence these approaches.
For example, G4 explicitly steers students towards the SFA, while G1-3 offer
more flexibility in how students choose to engage with the LLM.

###  3.3. Response from Large Language Models

The response generated from proprietary models is perhaps the most difficult
to control in our study, due to ever-evolving (behind-the-scene) updates to
the models (Chen et al., 2023b). However, the response of a Prompted LLM
definitely differs from an Unprompted LLM. A few-shot-prompted LLM can achieve
better performance in the task at hand (helping students solve problems) than
an uninitiated one (Brown et al., 2020).

To explore this design space, we conducted two studies - one in a computer
science classroom teaching students about database-related concepts and a
second in an online controlled setting with crowdworkers solving mathematics
problems. Figures 2 and 3 show a high-level overview of both study designs.
They are described in detail in the following sections.

##  4\. Study-1: A formative field study in a CS classroom

This research study was approved by the local ethics board. The data used in
this deployment was obtained from a third-year Introduction to Databases
course (DB) at a large research-focused North American university in Winter
2023 for Computer Science (CS) students. This course spanned 12 weeks,
employing a flipped classroom setup (students were required to complete
preparation work: video modules, technical questions, and self-reflections,
prior to attending lecture in-person using a learning management system). DB
had one course instructor, eleven teaching assistants, and was an upper-year
elective course for computer science students. The majority of students were
computer science majors, however, some are computer science minors. Although
natural classroom limitations impose constraints on achievable statistical
power and, as a result, the complexity of questions we can reliably answer
with quantitative methods, we used this deployment as a formative one,
focusing on both acquiring statistical evidence where it is reliable and
exploratory analysis without statistical conclusions to inform larger-scale
deployments (both in the virtual lab (Study 2, see Section 5) and in the field
(future studies)).

We designed a randomized factorial experiment, for the guidance given to the
student, with 2 (G1-List of Suggestions: present vs. absent) x 2 (G2-Example-
based instruction: present vs. absent) x 2 (G3-Metacognitive questioning:
present vs. absent) x 2 (G4-Solve then refine: present vs. absent) between
subjects. The presence of G4 encouraged a Self-First Approach (SFA), while the
absence provided more flexibility (LLM-First Approach – LFA) (refer to Section
3). At the beginning of the assignment, students were primed to use LLM-based
support tool with the following:

> “You will now get some instructions related to the use and access of the
> dialogue-based support system. The system uses a fine-tuned large language
> model. Large language models (LLMs) are artificial intelligence systems that
> use deep learning to analyze vast amounts of text and generate responses
> that are contextually relevant.”

After seeing this text, students were randomly assigned to one of the forms of
instruction composed of factors described in Section 3. After receiving
guidance and access to the LLM chatbot, students attempted to solve multiple-
choice problems (related to databases) while receiving support from LLMs.
Besides accuracy in the assignment, we collected pre- and post-measures
regarding perception of LLMs (confidence in LLMs’ responses, helpfulness,
willingness to interact again and error tolerance) and perception of self
(confidence in their answers, self-confidence for the given topic of
assignment). To measure long-term learning, we measured their performance on
the final exam on the same topic, comprising isomorphic questions to
assignment problems.

Figure 2. Schematic of the formative study in a CS classroom. 2 (List of
Suggestions: present vs. absent) x 2 (Example-based Instruction: present vs.
absent) x 2 (Metacognitive-questioning based Instruction: present vs. absent)
x 2 (Solve, then refine with LLM: present vs. absent) between subjects.

###  4.1. LLM-based chat support tool

Each student was given access to a chatbot tutor, based on GPT-3 (described in
Section A.1.1), to help them solve assignment problems. The prompt to the
model was designed after careful iterations between the researchers of this
study, the instructor, and the teaching assistant. The prompt given to the
model was the following:

”The following is a conversation with a database instructor. The instructor
helps the human solve assignment problems related to database. The instructor
never explicitly gives the solution. The instructor also never writes the SQL
query. The instructor would only provide brainstorms to possible solutions
without providing any SQL statements. This rule should be enforced in the
entirety of the conversation. Following are the DDL files to create table
given as part of the assignment.”

###  4.2. Participants

There were 218 students enrolled in DB, of which a total of 145 students (67%)
completed the assignment. When students were asked about their familiarity
with Large Language Models (LLMs), 51.03% reported occasional use, 16.55% were
regular users, 15.86% had tried them once, and 16.55% had never used them
before. On a scale of 1 to 7, the students’ average initial self-efficacy for
the topic stood at 4.10 (SD = 1.38).

###  4.3. Results

####  4.3.1. Students’ attitudes towards LLMs.

Students rated their average initial trust in LLMs at 3.58 out of 7 (SD =
1.54). Previous experience of LLM use is connected with the initial trust in
LLM ($\chi^{2}=18.186$, $df=6$, $p<0.01$), with students having regular
experience with LLMs reporting more trust in them. The overall difference in
trust towards LLMs (before the experiment) and on LLM chatbots (after the
experiment) across balanced experimental conditions shows a significant growth
trend (from $\text{Median}_{before}=4$ to $\text{Median}_{after}=5$ on 7-point
Likert scale, _V_(Wilcoxon) = 1421.00, $p<0.001$)).

Category | Measure | Range | G1 | G2 | G3 | G4
---|---|---|---|---|---|---
|  |  | Absent | Present | Absent | Present | Absent | Present | Absent | Present
Performance | Homework Score | 0 - 100 | $58.69(\pm 2.09)$ | $59.46(\pm 1.85)$ | $57.52(\pm 1.81)$ | $60.62(\pm 2.10)$ | $57.63(\pm 1.87)$ | $60.47(\pm 2.05)$ | $61.38(\pm 1.83)$ | $56.87(\pm 2.06)$
| Exam Score | 0 - 100 | $56.20(\pm 1.52)$ | $54.80(\pm 1.70)$ | $53.61(\pm 1.74)$ | $57.33(\pm 1.45)$ | $55.49(\pm 1.79)$ | $55.47(\pm 1.44)$ | $55.00(\pm 1.43)$ | $55.95(\pm 1.78)$
| Score Diff | (-100) - 100 | $-2.49(\pm 2.34)$ | $-4.66(\pm 2.23)$ | $-3.91(\pm 2.14)$ | $-3.29(\pm 2.42)$ | $-2.14(\pm 2.19)$ | $-5.00(\pm 2.37)$ | $-6.38(\pm 1.97)$ | $-0.92(\pm 2.51)$
| Perception
---
of System
Helpfulness | 1 - 7 | $4.51(\pm 0.22)$ | $4.82(\pm 0.16)$ | $4.56(\pm 0.19)$ | $4.78(\pm 0.19)$ | $4.70(\pm 0.19)$ | $4.64(\pm 0.20)$ | $4.54(\pm 0.19)$ | $4.80(\pm 0.20)$
|  | Helpfulness for
---
Other Topics
1 - 7 | $5.17(\pm 0.21)$ | $5.28(\pm 0.16)$ | $5.17(\pm 0.18)$ | $5.29(\pm 0.19)$ | $5.32(\pm 0.18)$ | $5.14(\pm 0.19)$ | $5.00(\pm 0.19)$ | $5.45(\pm 0.18)$
|  | Tolerance for
---
Mistakes
1 - 7 | $4.32(\pm 0.22)$ | $4.18(\pm 0.20)$ | $4.29(\pm 0.21)$ | $4.21(\pm 0.22)$ | $4.62(\pm 0.20)$ | $3.89(\pm 0.22)$ | $4.24(\pm 0.20)$ | $4.26(\pm 0.22)$
|  | Willingness to
---
Interact Again
1 - 7 | $5.25(\pm 0.22)$ | $5.34(\pm 0.18)$ | $5.22(\pm 0.21)$ | $5.37(\pm 0.19)$ | $5.54(\pm 0.19)$ | $5.07(\pm 0.20)$ | $5.13(\pm 0.20)$ | $5.46(\pm 0.20)$
|  | Change in
---
Confidence in
the LLM’s
Ability to Help
(-14) - 14 | $1.08(\pm 0.2)$ | $0.77(\pm 0.22)$ | $1.07(\pm 0.22)$ | $0.78(\pm 0.20)$ | $0.70(\pm 0.21)$ | $1.14(\pm 0.20)$ | $0.80(\pm 0.19)$ | $1.04(\pm 0.23)$
| Perception
---
of Self
| Change in
---
Self-confidence
(-14) - 14 | $-0.04(\pm 0.16)$ | $0.49(\pm 0.20)$ | $0.06(\pm 0.17)$ | $0.40(\pm 0.19)$ | $0.41(\pm 0.17)$ | $0.05(\pm 0.19)$ | $0.21(\pm 0.18)$ | $0.24(\pm 0.18)$
|  | Confidence on
---
Their Answers
1 - 7 | $3.86(\pm 0.19)$ | $3.78(\pm 0.16)$ | $3.78(\pm 0.18)$ | $3.86(\pm 0.16)$ | $3.80(\pm 0.17)$ | $3.84(\pm 0.18)$ | $4.01(\pm 0.16)$ | $3.64(\pm 0.18)$

Table 1. Summary of results for Study-1’s classroom deployment.

####  4.3.2. Performance measures

For Homework Score, we see a contrast between G4: Solve-Then-Refine (focusing
on getting them to solve a problem first, then go to the LLM with questions),
vs. G1, G2, G3 (which focused on using the LLMs while solving problems). The
data suggest that G4-Solve-Then-Refine (encouraging self-first approach) may
not have helped homework performance as much compared to the other forms of
guidance (see Table 1).

####  4.3.3. Perception of system

On average, we observe a small increase in ”confidence in the ability of LLM
to help” across different forms of guidance strategies and use approaches. All
conditions report moderately high ratings for ”Helpfulness” for the given
topic. However, they report it to be even higher on average for ”Helpfulness
for other topics”, with the presence of G4-solve-then-refine standing out for
being particularly helpful in general. The data suggest that the presence of
G3-Metacognitive-Questioning reduced tolerance for mistakes and willingness to
interact again.

####  4.3.4. Perception of self

There is negligible change in the student’s self-confidence (their confidence
on the given topic) and this is consistent across different conditions. The
students’ confidence in their answers is similar across different conditions,
however, G4-Solve-then-refine (or encouraging self-first approach) stands out
- the absence of G4 in the overall guidance resulted in relatively higher
confidence in their answers.

The findings from our formative deployment established the potential for
differences between different guidance strategies (particularly G4) and the
resulting approach (self-first approach) to influence different measures
related to performance and confidence. We further evaluate the effect of
different interaction strategies in the following section. One key takeaway
from the formative deployment was to introduce intermediate measures
immediately after giving guidance to the learner.

##  5\. Study-2: Solving Math Problems in an Online Controlled Setting

To further explore the design space of the LLM-learner interaction, we
designed a 2 (LLM Chatbot: Prompted vs. Unprompted) x 5 (Guidance strategy:
G1-List of Suggestions vs. G2-Example-based Instruction vs. G3-Metacognitive
Questioning vs. G4-Solve then refine with LLM vs. No Instruction) (Figure 3)
randomized controlled trial. Upon consenting to participate in the experiment,
participants were asked to report their confidence in their ability to solve
math problems, how frequently they used LLMs, and their confidence in using
and trusting the responses generated by LLMs. The participants were then
randomized to access a chatbot, either with or without a preprompt (detailed
in Section 5.3). They received one of four guidance strategies (explained in
Section 3) or a basic description of how LLMs work (no instruction). After
solving a common example problem with the LLM chatbot, they rated their
confidence in using and trusting LLMs. Subsequently, they solved 4 math
problems (randomly selected from a pool of 8 problems) with support from the
LLM chatbot (Section 5.2). Finally, they reported their final confidence in
their ability to solve math problems, their confidence in using and trusting
LLM responses. The experiment flow just described is summarized and visualized
in Figure 3.

Figure 3. Schematic of the experiment design for Study 2 with crowdworkers. 2
(LLM Chatbot: Prompted vs. Unprompted) x 5 (Guidance strategy: G1-List of
Suggestions vs. G2-Example-based Instruction vs. G3-Metacognitive Questioning
vs. G4-Solve then refine with LLM vs. No Instruction) between subjects for the
type of chatbot and guidance received.

###  5.1. Participants

We recruited 356 participants, fluent in English, from Prolific (Palan and
Schitter, 2018). Previous work has shown that crowdworkers can be involved in
user studies (Kittur et al., 2008), and that responses from crowd-sourcing
platforms can be of similar quality to those from traditional methods, such as
samples from undergraduate students and the community (Follmer et al., 2017).
Crowdworkers have been shown to be reliable proxies for online learners (Davis
et al., 2018). Table 2 describes the participants.

Category |  Results
---|---
Initial self-confidence in solving math problems |  1 (1.40%), 2 (3.93%), 3 (8.15%), 4 (18.26%), 5 (28.93%), 6 (22.75%), 7 (16.57%)
where 1 is “not confident at all” and 7 is “extremely confident”
Prior use of LLMs |  Never (23.88%), Once (20.51%), Occasionally (34.55%), Regularly (21.07%)
Age |  18-25 (43.26%), 25-34 (41.57%), 35-44 (10.96%), 45-54 (2.81%), 55-64 (1.12%), 65+ (0.28%)
Gender |  Male (47.75%), Female (49.44%), Non-binary / third gender (1.97%), Prefer not to say (0.84%)
Education |  Some high school or less (2.25%), High School graduate (10.96%), Some college credit with no degree (23.31%), Associate degree (4.49%), Bachelor’s degree (41.85%), Graduate or Professional degree (14.89%), Prefer not to say (2.25%)
Total Participants = 356
Table 2. Demographic information of the participants recruited for Study-2 on
Prolific.

###  5.2. Task: Answering multiple choice math questions

Each participant tackled two elementary-level math questions and two high-
school-level math questions. The sequence of these tasks was randomized for
each individual. From the MMLU benchmark dataset (Hendrycks et al., 2021a, b),
we randomly chose 4 questions each from the elementary and high school math
levels. The participants were then tasked with solving two questions (randomly
sampled) from each level. Besides providing their answer to the problem, they
were also asked to report how confident they were in their answer (Figure 4).

Figure 4. Example task from Study 2. Participants provide a solution and
indicate their confidence in the answer.

###  5.3. Steering LLM Responses with Pre-Prompts: Prompted vs. Unprompted LLM
Chatbot

In a controlled setting, our aim was to explore the differences between
prompted and unprompted interactions with LLMs in a chatbot format. The
distinction is crucial for several reasons:

  * •

Real-world relevance: The unprompted version mirrors common LLM chatbots such
as ChatGPT, representing the typical interaction users might have with
publicly available LLMs.

  * •

Educational context: A prompted LLM, on the other hand, could resemble a
scenario in which instructors guide the model to provide specific types of
response. This distinction was especially relevant as our classroom experiment
(Section 4) could not employ unprompted LLMs due to ethical considerations.

Our design probe was a GPT-3-based chat interface (see Section A.1.1),
representative of other chat-based LLM tutors. Below, we detail the two
distinct variants of LLM chatbots used in our study:

####  5.3.1. GPT-3 with Pre-Prompt

This variant was introduced to explore whether guiding the model with specific
pre-prompts can offer advantages over generic out-of-the-box LLMs. We used the
following pre-prompt for GPT-3:

”You are a professional K12 math teacher helping students answer math
questions.
Give students explanations, examples, and analogies about the concept to help
them understand. You should guide students in an open-ended way. Make the
answer as precise and succinct as possible.
You should help them in a way that helps them (1) learn the concept, (2) have
confidence in their understanding, and (3) have confidence in your ability to
help them. Before answering, reflect on how your answer will help you achieve
goals (1), (2), and (3). Update your answer based on this reflection.”

####  5.3.2. GPT-3 without Any Pre-Prompt

This unprompted version served as a control, allowing us to contrast its
performance and responses with the prompted version.

###  5.4. Results

####  5.4.1. Performance (Learner + AI performance in the task)

The performance (average score in the task) was not significantly different
between different types of guidance ($\chi^{2}_{\text{Kruskal-
Wallis}}(4)=0.13$, $n=356$, $p=1$) or whether the LLM was prompted or
unprompted ($W_{\text{Mann-Whitney}}=15025$, $n=356$, $p=0.37$).

Exploratory analysis of subgroups (see Figure 5) suggest that guidance
strategies G2 (examples) and G3 (Metacognitive Questioning) appeared to
improve performance when paired with unprompted LLM. This suggests that these
guidance approaches may be effective in counterbalancing the absence of a pre-
prompt given to the LLM. Specifically, the use of examples (G2) with
unprompted LLM achieved an average score of $68.13\%\pm 2.68\%$, compared to
$58.09\%\pm 4.32\%$ obtained with prompted LLM. A similar trend was observed
for G3 (Metacognitive Questioning) across the two LLM conditions, calling for
further investigation of the potential advantages of these guidance strategies
when paired with an unprompted LLM.

Figure 5. Performance (operationalized as accuracy in the 4 questions) across
the different conditions of the experiment.

####  5.4.2. Confidence in ability to use LLMs to solve math problems over
different stages of interaction (Figure 6)

The confidence is measured in three stages in the study in response to the
prompt “I feel confident in my ability to use Large Language Models (LLMs) to
assist me in solving math.” General confidence is moderately high, with the
median being 4 out of 7.

On average, across conditions, we observed significant differences in self-
reported confidence in use of LLMs ($\chi^{2}_{\text{Friedman}}(2)=72.82$,
$n_{\text{pairs}}=356$, $p<0.001$), with growth ($p<0.001$) between initial
confidence ($\text{Mean}=4.51$, $\text{Me}=5$, $\text{SD}=1.49$) and
immediately after getting guidance ($\text{Mean}=5.19$, $\text{Me}=6$,
$\text{SD}=1.66$), and a significant drop ($p<0.001$) after use
($\text{Mean}=4.57$, $\text{Me}=5$, $\text{SD}=1.81$). The first part of this
pattern suggests that people after receiving guidance, increase their
confidence (as opposed to judging interaction as too complex), but after
acquiring more experience in solving real tasks, their confidence slightly
decreases. The reversion of the initial confidence level after the task is
interesting, as it suggests that while guidance may temporarily boost
confidence, the practical challenges of applying knowledge during problem-
solving can temper this heightened confidence. Further exploratory analysis
suggests that this pattern occurs for all the different forms of guidance, and
even when there isn’t guidance being provided, for the Prompted and the
Unprompted.

Within the different conditions, these general trends are observed with
varying degrees of intensity. For instance, the condition where the LLM is
Prompted with G4: Solve, then Refine with LLM guidance shows the most
pronounced retention of confidence after the problem-solving task. On the
other hand, the Unprompted LLM condition with G2: Example-based Instruction
shows the least initial boost in confidence post-guidance. These observations
suggest potential nuanced interplays between LLM prompting and the guidance
type in influencing learner confidence.

Figure 6. This grid visualizes the participants’ average confidence in their
ability to use LLMs across different experimental conditions. Each cell
represents a combination of LLM prompting (either ”Prompted” or ”Unprompted”)
and a specific guidance type. The gray dotted line in each cell represents the
overall average confidence across all conditions, providing a point of
reference. The overarching trend showcases varying levels of confidence shifts
across different conditions.

####  5.4.3. Trust in LLM’s responses over different stages of interaction
(Figure 7)

On average, across conditions, we observed significant differences in reported
trust in LLM responses ($\chi^{2}_{\text{Friedman}}(2)=80.31$,
$n_{\text{pairs}}=356$, $p<0.001$).

While the growth between initial trust ($\text{Mean}=4.36$, $\text{Me}=4$,
$\text{SD}=1.36$), immediately after getting guidance ($\text{Mean}=4.35$,
$\text{Me}=5$, $\text{SD}=1.89$) is not significant, there is a significant
decline in final post-experience trust to LLM measure ($\text{Mean}=3.49$,
$\text{Me}=3$, $\text{SD}=1.80$) both compared to initial ($p<0.001$) and
after instructions ($p<0.001$) measurements. This sharper decline might
suggest that participants experience untrustworthy advice from LLM.

Further exploratory analysis within the different conditions generally
confirms this observed trend. For example, in conditions where the LLM is
Unprompted with No Instruction guidance, the decline in confidence after the
task is least pronounced. However, there are exceptions. For instance, the
Unprompted LLM condition with G2: Example-based Instruction shows the most
significant decline in confidence immediately post-guidance, suggesting that
certain guidance types might not instill trust in the LLM’s responses.

Figure 7. This grid illustrates the average confidence of participants in
trusting LLM responses across different experimental conditions. Each cell
showcases a combination of LLM prompting and guidance type. The gray dotted
line in each cell represents the overall average confidence across all
conditions, providing a point of reference. The general trend indicates a
decline in trust across most conditions, with a few exceptions.

####  5.4.4. Changes in self-confidence in solving math problems before and
after the problem-solving task with LLM

Our analysis reveals that participants’ confidence in their math-solving
abilities remains relatively stable throughout the interaction (see Figure 8.
The average confidence before receiving any guidance is $5.04$
($\text{SD}=1.41$). After they complete the problem-solving task, this
confidence is $5.16$ ($\text{SD}=1.62$). The observed difference between these
two stages is not statistically significant with $p=0.079$.

When exploring the individual experimental conditions, this stability in
confidence is largely consistent. For instance, under the condition where the
LLM is Unprompted and the guidance provided is G3: Metacognitive Questioning,
we observe a slight increase in confidence. On the other hand, the Unprompted
LLM condition combined with G2: Example-based Instruction guidance shows a
minor decrease in post-task confidence. However, it’s worth noting that these
variations might not necessarily imply a significant impact on the
participants’ self-assessment of their math-solving abilities.

Figure 8. The grid displays the average confidence of participants in their
math-solving abilities across distinct experimental conditions. Each cell
represents a combination of LLM prompting and a specific guidance type. The
gray dotted line in each cell represents the overall average confidence across
all conditions, providing a point of reference. The confidence levels across
conditions appear to be stable with minor fluctuations.

####  5.4.5. Confidence in answer vs actual performance in task

We explored the relationship between learners’ confidence in their answers and
their actual performance across different experimental conditions. Using a
Kernel Density Estimation (KDE) combined with jitter plots, we visualized the
distribution of learners’ confidence against their performance (see Figure 9).
A notable observation was the red zone, representing areas where the learners’
confidence exceeded their actual performance. Across the experimental
conditions, we found varying percentages of participants falling into this
overconfidence zone. Comparing these percentages across conditions, we did not
find significant differences (($\chi^{2}=13.63$, $p=0.136$).

Further exploratory analysis drives our attention to ”Unprompted LLM” with
”G1: Suggestions” where approximately 23.53% of the participants were in this
potentially overconfidence zone. As researchers continue to look at the
dynamics of learner confidence and performance on a larger scale, such
observations could offer valuable insight into designing better learning
environments.

Figure 9. The grid visualizes the relationship between learners’ confidence in
their answers and their performance in a problem-solving task involving
collaboration with an AI. The red-shaded area specifically represents zones
where participants’ confidence exceeds the median while their performance is
below the median for that particular condition, indicating overconfidence.
Percentages in each cell indicate the proportion of participants exhibiting
potential overconfidence for the respective condition.

##  6\. Discussion

Through the two studies, we explored the design space of providing guidance to
learners for using LLMs, the resulting approaches from these guidance
strategies and the prompts given to the LLM. We revisit the key findings and
contextualize them with existing literature and real-world implications.

###  6.1. Key Findings

####  6.1.1. Immediate performance and long-term learning

Our formative study highlights a potential design tension in LLM research: the
balance between providing immediate assistance and ensuring long-term skill
development for students. This fits into the long line of work studying the
effects of automated hints on learner performance and learning (Marwan et al.,
2019; Corbett and Anderson, 2001; Fossati et al., 2015; Roy Choudhury et al.,
2016; Rivers, 2017). An interesting observation in the formative study was the
contrast between G4: Solve-Then-Refine (focusing on getting them to solve a
problem first, then go to the LLM with questions), vs G1, G2, G3 (which
focused on using the LLMs while solving problems). There is suggestive
evidence that the presence of G4-Solve-Then-Refine (encouraging self-first
approach) may not have helped homework performance as much compared to the
other forms of guidance (refer to Table 1). However, the performance in the
final exam without any LLM support (considered as a measure of learning) was
similar between the different forms of guidance and learner approaches. Future
studies can further investigate the long-term learning impact of different
learner approaches (for example, trying first and then getting help from LLM
vs. getting help directly without trying). For Study-2, we did not find any
significant differences between the guidance strategies or chatbot types on
average, in terms of performance on the math problem-solving task with LLM
(see Figure 5). However, exploratory analysis looking at groups of Prompted
vs. Unprompted LLM suggested that G2 (examples) and G3 (metacognitive
questioning) improved performance when paired with unprompted LLM. This
suggests that the right student guidance may offset the lack of a pre-prompt
to the LLM. This finding is crucial when students independently use publicly
available LLMs like ChatGPT. Instructors may guide this independent use by
showing proper examples or encouraging deeper thinking about LLM use.

####  6.1.2. Learner confidence in using LLMs and trusting LLM responses
across different stages of problem-solving task

Our formative study suggested a small increase in the confidence of the
students in the ability of LLM to help, after solving the assignment problems
with LLM. However, there were no significant differences between the different
guidance strategies and approaches of LLM use. In Study-2, when we looked at
participants’ confidence in using LLMs, we observed an interesting pattern.
After receiving the guidance, the confidence of the people initially
increased. However, as they gained more hands-on experience solving tasks,
their confidence dipped slightly. This poses questions for further research:
Is this confidence drop a moment of realization about the challenges of
seeking help, be it from LLMs or others? Might this realization push them to
invest more effort in mastering the skill? Does the dip indicate a need for
better guidance and support to ensure that users continue to take advantage of
these tools for learning and assistance? Their trust in LLM’s responses showed
a similar trend. Although the growth between initial trust and immediately
after receiving guidance is not significant, there is a significant decline in
final post-task self-reported confidence in trusting LLM’s responses. There
could be complementary mechanisms behind this: People are either getting
incorrect or misleading responses, or the responses are just not very helpful.
Further studies could investigate the qualitative aspects of learners’
experiences to answer these questions.

####  6.1.3. Learners’ self-assessment during the problem-solving task

Across both studies, we did not find significant differences between the pre-
task and post-task measures of learners’ confidence in their ability to solve
problems for the given topic with LLM assistance. Our findings contrast with
some existing work in education that states mastery experiences (i.e., success
in tasks) elevate self-confidence (Bandura, 1982). This could suggest that the
perceived role of LLM in problem-solving might mediate the relationship
between task success and self-confidence. This holds significance for the
education research - mere problem solving with advanced tools does not
translate to increased confidence. This underscores the need for complementary
strategies in educational settings - such as reflection, discussion, or
additional feedback mechanisms - to foster self-belief along with skill
acquisition.

In Study-2, participants reported moderately high confidence in their answers.
But more important than just the average confidence is the calibration -
whether their confidence is aligned with their actual accuracy. Approximately
23.53% of participants in the ”Unprompted LLM” with ”G1: List of Suggestions”
condition reported higher than median confidence in their answers, but
achieved a lower than median score in the math problems. This misalignment
echoes the broader educational concern of calibration in self-assessment - it
is not just about how confident learners feel, but how that confidence matches
up to their real-world performance. For educators and technology developers,
this underscores the importance of creating tools that not only assist in
problem-solving but also foster accurate self-reflection and awareness in
learners. Future research might delve deeper into strategies to better
calibrate learners’ self-perceptions when utilizing LLM-based tools.

####  6.1.4. Other subjective measures of helpfulness, tolerance for mistakes,
willingness to interact again with the LLM chatbot (formative study)

Our formative study in the classroom allowed us to gather subjective ratings
for different usability measures for LLM tutors. The presence of
G3-Metacognitive-Questioning asked students to reflect on the questions they
wanted to ask before attempting the problem, analogous to what they might as a
teacher. One could hypothesize having clearer questions to ask could improve
their interaction with the system as they have a clear sense of what to ask,
to get useful information. However, the data suggested this might not be the
case, or even the opposite might happen, as the presence of G3-Metacognitive-
Questioning reduced tolerance for mistakes and willingness to interact again.
This might happen by learners gaining unrealistic expectations about how LLMs
could help, or by them having overly specific questions, whereas an LLM might
be more beneficial for other reasons. For example, giving them space to self-
explain or think through in a discussion might be helpful (Williams et al.,
2016a, b). All conditions report moderately high ratings for ”Helpfulness” for
the given topic. However, they report it to be even higher on average for
”Helpfulness for other topics”, with the presence of G4-solve-then-refine
standing out as being particularly helpful in general.

###  6.2. Design Considerations for Learner-LLM Interactions

Our investigations underscore that interactions with LLMs in educational
settings do not conform to a one-size-fits-all approach. This has dual
implications: LLMs have the flexibility to support a variety of strategies,
but they also require careful oversight to ensure that the most effective
strategy is applied in a given context. Although our study delved into
specific interaction techniques, future research should explore and evaluate
other potential strategies. Based on our findings, we highlight some design
considerations for instruction designers, HCI researchers, and developers.

##### Balancing immediate help with long-term learning:

In Study-1, even though G4-Solve-Then-Refine might not have boosted homework
performance, it showed similar learning outcomes in final exams compared to
other strategies. Designers should weigh the benefits of immediate LLM
assistance against long-term skill building.

##### Calibration of confidence and expectations:

We saw in Study-2, a large percentage of participants in the ”Unprompted LLM”
with ”G1: List of Suggestions” condition overestimated their performance. When
building LLM-based learning support tools, special considerations must be made
to promote self-awareness and self-assessment to help learners align their
confidence with their abilities. Moreover, the observed decline in trust in
LLM’s responses after problem-solving task highlights the need to set clear
expectations. Demonstrations or guidelines on effectively using LLMs can
prevent over-reliance or unrealistic expectations.

##### Reinforcing mastery experiences:

Both studies highlighted that simply solving problems with LLM did not improve
the self-confidence of learners. Integrating reflection, feedback, or
discussion mechanisms can help improve self-belief.

##### Tailored interactions:

In our formative study, the presence of G3 (metacognitive questioning) altered
learners’ perceptions of the LLM. In Study-2, G2 (examples) and G3
(metacognitive questioning) enhanced outcomes with the unprompted LLM. Such
findings highlight the importance of adapting LLM interfaces to cater to
diverse learning contexts.

###  6.3. Future Work & Limitations

There is potential for further explorations of the design space of learner-LLM
interactions. We could not experiment with pre-prompts in the classroom
deployment due to ethical and sample size constraints, and the controlled
online setting did not have a measure for long-term learning. Future studies
could explore the effect of changing LLM pre-prompts on the longer-term
learning outcomes of the students. Moreover, there is rich literature on how
to guide people to use tools (especially in the context of learning) (Gersten
et al., 1986; Adams and Engelmann, 1996; Zendler and Klein, 2018; Anderson et
al., 2007; Van Gog and Rummel, 2010; Palincsar, 1986; Bourner, 2003). Future
research can build on the guidance strategies proposed in this paper and
explore the use of other designs to guide students, as well as to encourage
different approaches. Furthermore, while designing learner-LLM interactions,
it is crucial to ensure the right balance of learners’ confidence in their
answers while using LLMs, with actual accuracy of their answers. High
confidence with low accuracy could signal over-reliance and overconfidence.
Future research could look at designing interactions that ensure an optimal
balance of confidence and accuracy.

There are limitations to the current work. In our formative study, the
constraints of small to medium-sized classroom field studies inherently limit
the scope of broad statistical generalizations. Additionally, observations
from our specific classroom setting of an introduction to database course may
not be generalized to other classroom settings and audiences. For example,
students with non-computer science backgrounds might interact with LLMs
differently and differ in LLM-related trust and confidence. More studies are
needed to see whether results from the online controlled setting generalize to
learning tasks in other contexts and populations of online learners, who may
differ from crowdworkers in their goals and motivation. A larger sample
confirmatory study might help to reliably evaluate multiple interactions
between student characteristics, experimental variables, achievement, and
perception outcomes, which were described as exploratory in the current work.
Additionally, our findings could benefit further from a rigorous qualitative
analysis of conversations between learners and LLM chatbots, which can help
explain the mechanisms behind some of the effects observed in our studies.

##  7\. Conclusion

Through this work, we highlight complexities of designing learner-LLM
interactions. We explore the design space of this interaction with four
guidance strategies resulting in two different approaches of use, with a
prompted vs. unprompted LLM, in a classroom and a controlled online setting.
Even with our significant discoveries and rich data, we merely scratch the
surface upon the complexities of these interactions with LLMs. This paves the
way for more in-depth investigations and broadens the HCI community’s avenues
for further research. For Learner-LLM interactions, the task for HCI
researchers and designers is twofold: understanding the effective
implementation of these models in learning contexts and discerning the optimal
interaction paradigms to ensure not just information dissemination, but
genuine student understanding and engagement. Educators should be prepared to
guide learners effectively by utilizing examples or templates to positively
steer LLM interactions. Incorporating these insights is imperative for both
educators and educational technology developers to collaboratively shape the
next generation of LLM tools, ensuring they are both intuitive and
pedagogically effective for diverse learners.

###### Acknowledgements.

We thank Xinyuan (Ben) Wang for his help with data analysis. We thank the
members of the Intelligent Adaptive Interventions Lab for their feedback on
the project. We also thank Marko Choi and Hammad Sheikh for their help in
developing the chat interface, which allowed us to collect data at scale. We
acknowledge the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC) (#RGPIN-2019-06968) and the Office of Naval Research
(ONR) (#N00014-21-1-2576) grants awarded to Dr. Williams. We also acknowledge
an award from DARPA for AI Tools for Adult Learning awarded to TutorGen for
QuickTA.

## References

  * (1)
  * bin ([n. d.]) [n. d.].  Bing Chat.  https://www.microsoft.com/en-us/edge/features/bing-chat?form=MT00D8.  Accessed: 2023-08-18.
  * cha ([n. d.]) [n. d.].  ChatGPT.  https://openai.com/chatgpt.  Accessed: 2023-08-18.
  * bar ([n. d.]) [n. d.].  An important next step on our AI journey.  https://blog.google/technology/ai/bard-google-ai-search-updates/.  Accessed: 2023-08-18.
  * kha ([n. d.]) [n. d.].  World-class AI for education.  https://www.khanacademy.org/khan-labs#khanmigo.  Accessed: 2023-08-18.
  * eth (2023) 2023\.  Practical AI for Instructors and Students Part 1: Introduction to AI for Teachers and Students.  https://www.youtube.com/watch?v=t9gmyvf7JYo&list=PLwRdpYzPkkn302_rL5RrXvQE8j0jLP02j&index=2
  * Abdellatif et al. (2020) Ahmad Abdellatif, Diego Costa, Khaled Badran, Rabe Abdalkareem, and Emad Shihab. 2020.  Challenges in chatbot development: A study of stack overflow posts. In _Proceedings of the 17th international conference on mining software repositories_. 174–185.
  * Adams and Engelmann (1996) Gary L Adams and Siegfried Engelmann. 1996.  _Research on Direct Instruction: 25 years beyond DISTAR._ ERIC.
  * Adiguzel et al. (2023) Tufan Adiguzel, Mehmet Haldun Kaya, and Fatih Kürşat Cansu. 2023.  Revolutionizing education with AI: Exploring the transformative potential of ChatGPT.  _Contemporary Educational Technology_ 15, 3 (2023), ep429.
  * Alt and Raichel (2020) Dorit Alt and Nirit Raichel. 2020.  Reflective journaling and metacognitive awareness: Insights from a longitudinal study in higher education.  _Reflective Practice_ 21, 2 (2020), 145–158.
  * Anderson et al. (1995) John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995.  Cognitive tutors: Lessons learned.  _The journal of the learning sciences_ 4, 2 (1995), 167–207.
  * Anderson et al. (2007) Richard Anderson, Ruth Anderson, Katie M Davis, Natalie Linnell, Craig Prince, and Valentin Razmov. 2007.  Supporting active learning and example based instruction with classroom technology.  _Acm Sigcse Bulletin_ 39, 1 (2007), 69–73.
  * Baidoo-Anu and Owusu Ansah (2023) David Baidoo-Anu and Leticia Owusu Ansah. 2023.  Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning.  _Available at SSRN 4337484_ (2023).
  * Bandura (1982) Albert Bandura. 1982.  Self-efficacy mechanism in human agency.  _American psychologist_ 37, 2 (1982), 122.
  * Bloom (1984) Benjamin S Bloom. 1984.  The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring.  _Educational researcher_ 13, 6 (1984), 4–16.
  * Bonawitz et al. (2011) Elizabeth Bonawitz, Patrick Shafto, Hyowon Gweon, Noah D Goodman, Elizabeth Spelke, and Laura Schulz. 2011.  The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery.  _Cognition_ 120, 3 (2011), 322–330.
  * Bourner (2003) Tom Bourner. 2003.  Assessing reflective learning.  _Education+ training_ 45, 5 (2003), 267–272.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2023b) Lingjiao Chen, Matei Zaharia, and James Zou. 2023b.  How is ChatGPT’s behavior changing over time?  _arXiv preprint arXiv:2307.09009_ (2023).
  * Chen et al. (2023a) Zheng Chen, Ziyan Jiang, and Fan Yang. 2023a.  Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding.  _arXiv preprint arXiv:2305.14449_ (2023).
  * Corbett and Anderson (2001) Albert T Corbett and John R Anderson. 2001.  Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 245–252.
  * Daniel (2017) Kahneman Daniel. 2017.  _Thinking, fast and slow_.
  * Davis et al. (2018) Dan Davis, Claudia Hauff, and Geert-Jan Houben. 2018.  Evaluating Crowdworkers as a Proxy for Online Learners in Video-Based Learning Contexts.  _Proc. ACM Hum.-Comput. Interact._ 2, CSCW, Article 42 (nov 2018), 16 pages.  https://doi.org/10.1145/3274311
  * Deng and Lin (2022) Jianyang Deng and Yijia Lin. 2022.  The benefits and challenges of ChatGPT: An overview.  _Frontiers in Computing and Intelligent Systems_ 2, 2 (2022), 81–83.
  * Denny et al. (2023a) Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. 2023a.  Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources.  _arXiv preprint arXiv:2306.10509_ (2023).
  * Denny et al. (2023b) Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A Becker, and Brent N Reeves. 2023b.  Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators.  _arXiv preprint arXiv:2307.16364_ (2023).
  * Denny et al. (2023c) Paul Denny, James Prather, Brett A Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023c.  Computing Education in the Era of Generative AI.  _arXiv preprint arXiv:2306.02608_ (2023).
  * Dergaa et al. (2023) Ismail Dergaa, Karim Chamari, Piotr Zmijewski, and Helmi Ben Saad. 2023.  From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing.  _Biology of Sport_ 40, 2 (2023), 615–622.
  * Dwivedi et al. (2023) Yogesh K. Dwivedi, Nir Kshetri, Laurie Hughes, Emma Louise Slade, Anand Jeyaraj, Arpan Kumar Kar, Abdullah M. Baabdullah, Alex Koohang, Vishnupriya Raghavan, Manju Ahuja, Hanaa Albanna, Mousa Ahmad Albashrawi, Adil S. Al-Busaidi, Janarthanan Balakrishnan, Yves Barlette, Sriparna Basu, Indranil Bose, Laurence Brooks, Dimitrios Buhalis, Lemuria Carter, Soumyadeb Chowdhury, Tom Crick, Scott W. Cunningham, Gareth H. Davies, Robert M. Davison, Rahul Dé, Denis Dennehy, Yanqing Duan, Rameshwar Dubey, Rohita Dwivedi, John S. Edwards, Carlos Flavián, Robin Gauld, Varun Grover, Mei-Chih Hu, Marijn Janssen, Paul Jones, Iris Junglas, Sangeeta Khorana, Sascha Kraus, Kai R. Larsen, Paul Latreille, Sven Laumer, F. Tegwen Malik, Abbas Mardani, Marcello Mariani, Sunil Mithas, Emmanuel Mogaji, Jeretta Horn Nord, Siobhan O’Connor, Fevzi Okumus, Margherita Pagani, Neeraj Pandey, Savvas Papagiannidis, Ilias O. Pappas, Nishith Pathak, Jan Pries-Heje, Ramakrishnan Raman, Nripendra P. Rana, Sven-Volker Rehm, Samuel Ribeiro-Navarrete, Alexander Richter, Frantz Rowe, Suprateek Sarker, Bernd Carsten Stahl, Manoj Kumar Tiwari, Wil van der Aalst, Viswanath Venkatesh, Giampaolo Viglia, Michael Wade, Paul Walton, Jochen Wirtz, and Ryan Wright. 2023.  Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy.  _International Journal of Information Management_ 71 (2023), 102642.  https://doi.org/10.1016/j.ijinfomgt.2023.102642
  * Ekin (2023) Sabit Ekin. 2023.  Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices.  (2023).
  * Ellis et al. (2014) Arthur K Ellis, David W Denton, and John B Bond. 2014.  An analysis of research on metacognitive teaching strategies.  _Procedia-Social and Behavioral Sciences_ 116 (2014), 4015–4024.
  * Farrokhnia et al. (2023) Mohammadreza Farrokhnia, Seyyed Kazem Banihashem, Omid Noroozi, and Arjen Wals. 2023.  A SWOT analysis of ChatGPT: Implications for educational practice and research.  _Innovations in Education and Teaching International_ (2023), 1–15.
  * Fiannaca et al. (2023) Alexander J Fiannaca, Chinmay Kulkarni, Carrie J Cai, and Michael Terry. 2023.  Programming without a Programming Language: Challenges and Opportunities for Designing Developer Tools for Prompt Programming. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Follmer et al. (2017) D Jake Follmer, Rayne A Sperling, and Hoi K Suen. 2017.  The role of MTurk in education research: Advantages, issues, and future directions.  _Educational Researcher_ 46, 6 (2017), 329–334.
  * Fossati et al. (2015) Davide Fossati, Barbara Di Eugenio, STELLAN Ohlsson, Christopher Brown, and Lin Chen. 2015.  Data driven automatic feedback generation in the iList intelligent tutoring system.  _Technology, Instruction, Cognition and Learning_ 10, 1 (2015), 5–26.
  * Gersten et al. (1986) Russell Gersten, John Woodward, and Craig Darch. 1986.  Direct instruction: A research-based approach to curriculum design and teaching.  _Exceptional Children_ 53, 1 (1986), 17–31.
  * Glassman et al. (2015) Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller. 2015.  OverCode: Visualizing variation in student solutions to programming problems at scale.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 22, 2 (2015), 1–35.
  * Goyal et al. (2023) Abhay Goyal, Muhammad Siddique, Nimay Parekh, Zach Schwitzky, Clara Broekaert, Connor Michelotti, Allie Wong, Lam Yin Cheung, Robin O Hanlon, Munmun De Choudhury, et al. 2023\.  ChatGPT and Bard Responses to Polarizing Questions.  _arXiv preprint arXiv:2307.12402_ (2023).
  * Haque et al. (2022) Mubin Ul Haque, Isuru Dharmadasa, Zarrin Tasnim Sworna, Roshan Namal Rajapakse, and Hussain Ahmad. 2022.  ” I think this is the most disruptive technology”: Exploring Sentiments of ChatGPT Early Adopters using Twitter Data.  _arXiv preprint arXiv:2212.05856_ (2022).
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a.  Aligning AI With Shared Human Values.  _Proceedings of the International Conference on Learning Representations (ICLR)_ (2021).
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring Massive Multitask Language Understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ (2021).
  * Hofstadter (1995) Douglas R Hofstadter. 1995.  _Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought._ Basic books.
  * Jacques (2023) Lorraine Jacques. 2023.  Teaching CS-101 at the Dawn of ChatGPT.  _ACM Inroads_ 14, 2 (2023), 40–46.
  * Jalil et al. (2023) Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023.  Chatgpt and software testing education: Promises & perils. In _2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE, 4130–4137.
  * Karabenick and Dembo (2011) Stuart A. Karabenick and Myron H. Dembo. 2011.  Understanding and facilitating self‐regulated help seeking.  _New Directions for Teaching and Learning_ 2011, 126 (June 2011), 33–43.  https://doi.org/10.1002/tl.442
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023\.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.
  * Kittur et al. (2008) Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008.  Crowdsourcing user studies with Mechanical Turk. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 453–456.
  * Kohnke et al. (2023) Lucas Kohnke, Benjamin Luke Moorhouse, and Di Zou. 2023.  ChatGPT for language teaching and learning.  _RELC Journal_ (2023), 00336882231162868.
  * Kooli (2023) Chokri Kooli. 2023.  Chatbots in education and research: A critical examination of ethical implications and solutions.  _Sustainability_ 15, 7 (2023), 5614.
  * Kumar et al. (2022) Harsh Kumar, Kunzhi Yu, Andrew Chung, Jiakai Shi, and Joseph Jay Williams. 2022.  Exploring the potential of chatbots to provide mental well-being support for computer science students. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2_. 1339–1339.
  * Lee (2023) Hyunsu Lee. 2023.  The rise of ChatGPT: Exploring its potential in medical education.  _Anatomical Sciences Education_ (2023).
  * Lee (2004) Yo-An Lee. 2004.  The work of examples in classroom instruction.  _Linguistics and education_ 15, 1-2 (2004), 99–120.
  * Li et al. (2023) Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023.  Multi-step jailbreaking privacy attacks on chatgpt.  _arXiv preprint arXiv:2304.05197_ (2023).
  * Liffiton et al. (2023) Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023.  CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes.  _arXiv preprint arXiv:2308.06921_ (2023).
  * Liu and Chilton (2022) Vivian Liu and Lydia B Chilton. 2022.  Design guidelines for prompt engineering text-to-image generative models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–23.
  * Liu et al. (2023) Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023.  Prompt Injection attack against LLM-integrated Applications.  _arXiv preprint arXiv:2306.05499_ (2023).
  * Livingston (2003) Jennifer A Livingston. 2003.  Metacognition: An Overview.  (2003).
  * Marwan et al. (2019) Samiha Marwan, Joseph Jay Williams, and Thomas Price. 2019.  An evaluation of the impact of automated programming hints on performance and learning. In _Proceedings of the 2019 ACM Conference on International Computing Education Research_. 61–70.
  * Miquelino ([n. d.]) Ricardo dos Santos Miquelino. [n. d.].  Better results with CHATGPT? – it’s all about asking the right questions.  https://www.linkedin.com/pulse/better-results-chatgpt-its-all-asking-right-questions-ricardo/
  * Mollick (2023a) Ethan Mollick. 2023a.  https://www.oneusefulthing.org/p/all-my-classes-suddenly-became-ai
  * Mollick (2023b) Ethan Mollick. 2023b.  Assigning AI: Seven ways of using AI in class.  https://www.oneusefulthing.org/p/assigning-ai-seven-ways-of-using
  * Moores and Chang (2009) Trevor T. Moores and Jerry Cha-Jan Chang. 2009.  Self-efficacy, overconfidence, and the negative effect on subsequent performance: A field study.  _Information & Management_ 46, 2 (March 2009), 69–76.  https://doi.org/10.1016/j.im.2008.11.006
  * Najafali et al. (2023) Daniel Najafali, Justin M Camacho, Erik Reiche, Logan Galbraith, Shane D Morrison, and Amir H Dorafshar. 2023.  Truth or lies? The pitfalls and limitations of ChatGPT in systematic review creation.  _Aesthetic Surgery Journal_ (2023), sjad093.
  * Palan and Schitter (2018) Stefan Palan and Christian Schitter. 2018.  Prolific. ac—A subject pool for online experiments.  _Journal of Behavioral and Experimental Finance_ 17 (2018), 22–27.
  * Palincsar (1986) Annemarie Sullivan Palincsar. 1986.  Metacognitive strategy instruction.  _Exceptional children_ 53, 2 (1986), 118–124.
  * Pangu (2023) Eileen Pangu. 2023.  How to use large language models (LLM) in your own domains.  https://towardsdatascience.com/how-to-use-large-language-models-llm-in-your-own-domains-b4dff2d08464
  * Pardos and Bhandari (2023) Zachary A Pardos and Shreya Bhandari. 2023.  Learning gain differences between ChatGPT and human tutor generated algebra hints.  _arXiv preprint arXiv:2302.06871_ (2023).
  * Pérez et al. (2020) José Quiroga Pérez, Thanasis Daradoumis, and Joan Manuel Marquès Puig. 2020.  Rediscovering the use of chatbots in education: A systematic literature review.  _Computer Applications in Engineering Education_ 28, 6 (2020), 1549–1565.
  * Qureshi (2023) Basit Qureshi. 2023.  Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges.  _arXiv preprint arXiv:2304.11214_ (2023).
  * Rahman and Watanobe (2023) Md Mostafizer Rahman and Yutaka Watanobe. 2023.  ChatGPT for education and research: Opportunities, threats, and strategies.  _Applied Sciences_ 13, 9 (2023), 5783.
  * Razzaq and Heffernan (2010) Leena Razzaq and Neil T Heffernan. 2010.  Hints: is it better to give or wait to be asked?. In _Intelligent Tutoring Systems: 10th International Conference, ITS 2010, Pittsburgh, PA, USA, June 14-18, 2010, Proceedings, Part I 10_. Springer, 349–358.
  * Rivers (2017) Kelly Rivers. 2017.  _Automated data-driven hint generation for learning programming_.  Ph. D. Dissertation. Carnegie Mellon University.
  * Roy Choudhury et al. (2016) Rohan Roy Choudhury, Hezheng Yin, and Armando Fox. 2016.  Scale-driven automatic hint generation for coding style. In _Intelligent Tutoring Systems: 13th International Conference, ITS 2016, Zagreb, Croatia, June 7-10, 2016. Proceedings 13_. Springer, 122–132.
  * Sallam (2023) Malik Sallam. 2023.  ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In _Healthcare_ , Vol. 11. MDPI, 887.
  * Sejnowski (2023) Terrence J Sejnowski. 2023.  Large language models and the reverse turing test.  _Neural computation_ 35, 3 (2023), 309–342.
  * Şen (2009) H Şenay Şen. 2009.  The relationsip between the use of metacognitive strategies and reading comprehension.  _Procedia-Social and Behavioral Sciences_ 1, 1 (2009), 2301–2305.
  * Smestad and Volden (2019) Tuva Lunde Smestad and Frode Volden. 2019.  Chatbot personalities matters: improving the user experience of chatbot interfaces. In _Internet Science: INSCI 2018 International Workshops, St. Petersburg, Russia, October 24–26, 2018, Revised Selected Papers 5_. Springer, 170–181.
  * Sok and Heng (2023) Sarin Sok and Kimkong Heng. 2023.  ChatGPT for education and research: A review of benefits and risks.  _Available at SSRN 4378735_ (2023).
  * Sweller (2006) John Sweller. 2006.  The worked example effect and human cognition.  _Learning and instruction_ (2006).
  * Van Dis et al. (2023) Eva AM Van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. 2023.  ChatGPT: five priorities for research.  _Nature_ 614, 7947 (2023), 224–226.
  * Van Gog and Rummel (2010) Tamara Van Gog and Nikol Rummel. 2010.  Example-based learning: Integrating cognitive and social-cognitive research perspectives.  _Educational psychology review_ 22 (2010), 155–174.
  * Vancouver et al. (2002) Jeffrey B. Vancouver, Charles M. Thompson, E. Casey Tischner, and Dan J. Putka. 2002.  Two studies examining the negative effect of self-efficacy on performance.  _Journal of Applied Psychology_ 87, 3 (2002), 506–516.  https://doi.org/10.1037/0021-9010.87.3.506
  * Wei et al. (2023) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023.  Jailbroken: How Does LLM Safety Training Fail?  _arXiv preprint arXiv:2307.02483_ (2023).
  * Weitekamp et al. (2020) Daniel Weitekamp, Erik Harpstead, and Ken R. Koedinger. 2020.  An Interaction Design for Machine Teaching to Develop AI Tutors. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3313831.3376226
  * White (1988) William AT White. 1988.  A meta-analysis of the effects of direct instruction in special education.  _Education and Treatment of children_ (1988), 364–374.
  * Williams et al. (2016a) Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016a.  Axis: Generating explanations at scale with learnersourcing and machine learning. In _Proceedings of the Third (2016) ACM Conference on Learning@ Scale_. 379–388.
  * Williams et al. (2016b) Joseph Jay Williams, Tania Lombrozo, Anne Hsu, Bernd Huber, and Juho Kim. 2016b.  Revising learner misconceptions without feedback: Prompting for reflection on anomalies. In _Proceedings of the 2016 CHI conference on human factors in computing systems_. 470–474.
  * Wittwer and Renkl (2010) Jörg Wittwer and Alexander Renkl. 2010.  How effective are instructional explanations in example-based learning? A meta-analytic review.  _Educational Psychology Review_ 22 (2010), 393–409.
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In _Proceedings of the 2022 CHI conference on human factors in computing systems_. 1–22.
  * Yang (2023) Hong Yang. 2023.  How I use ChatGPT responsibly in my teaching.  _Nature_ (2023).
  * Zamfirescu-Pereira et al. (2023) JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023.  Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–21.
  * Zendler and Klein (2018) Andreas Zendler and K Klein. 2018.  The effect of direct instruction and web quest on learning outcome in computer science education.  _Education and Information Technologies_ 23 (2018), 2765–2782.
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_ (2022).

##  Appendix A Research Methods

###  A.1. Formative Study

####  A.1.1. LLM Model Specification

  * •

model version: text-davinci-003

  * •

number of parameters: 175B

  * •

date of use: April 2023

Configuration Settings:

  * •

temperature: 0

  * •

max tokens: 300

  * •

top-p: 1

  * •

frequency penalty: 0

  * •

presence penalty: 0.6

Prompt Design:

> “The following is a conversation with a database instructor. The instructor
> helps the human solve assignment problems related to database. The
> instructor never explicitly gives the solution. The instructor also never
> writes the SQL query. The instructor would only provide brainstorms to
> possible solutions without providing any SQL statements. This rule should be
> enforced in the entirety of the conversation. Following are the ddl files to
> create table given as part of the assignment.”

Interaction Environment:

  * •

environment: A platform with a conversation interface hosted on university
servers. The platform directly calls the OpenAI’s API for generated contents.

###  A.2. Online Controlled Study

####  A.2.1. LLM Model Specification

  * •

model version: text-davinci-003

  * •

number of parameters: 175B

  * •

date of use: September 2023

Configuration Settings:

  * •

temperature: 0

  * •

max tokens: 300

  * •

top-p: 1

  * •

frequency penalty: 0

  * •

presence penalty: 0.6

Prompt Design:

  * •

prompted LLM:

> “You are a professional K12 math teacher helping students answer math
> questions.
>
> Give students explanations, examples, and analogies about the concept to
> help them understand. You should guide students in an open-ended way. Make
> the answer as precise and succinct as possible.
>
> You should help them in a way that helps them (1) learn the concept, (2)
> have confidence in their understanding, and (3) have confidence in your
> ability to help them. Before answering, reflect on how your answer will help
> you achieve goals (1) (2) (3). Update your answer based on this reflection.”

  * •

unprompted LLM:

> “You are a helpful assistant.”

Interaction Environment:

  * •

environment: A platform with a conversation interface hosted by Microsoft
Azure service. The platform directly calls the OpenAI’s API for generated
contents.