0 Research application/design study Yihan Liu, Zhen Wen, Luoxuan Weng, Ollie
Woodman, Yi Yang, and Wei Chen are with the State Key Lab of CAD&CG, Zhejiang
University. E-mail: {lyh1024 $|$ wenzhen $|$ lukeweng $|$ orw $|$ yang-yi $|$
<EMAIL_ADDRESS>Wei Chen is the corresponding author.

# SPROUT: Authoring Programming Tutorials with Interactive Visualization of
Large Language Model Generation Process

Yihan Liu  Zhen Wen  Luoxuan Weng  Ollie Woodman  Yi Yang  and Wei Chen

###### Abstract

The rapid development of large language models (LLMs), such as ChatGPT, has
revolutionized the efficiency of creating programming tutorials. LLMs can be
instructed with text prompts to generate comprehensive text descriptions of
code snippets. However, the lack of transparency in the end-to-end generation
process has hindered the understanding of model behavior and limited user
control over the generated results. To tackle this challenge, we introduce a
novel approach that breaks down the programming tutorial creation task into
actionable steps. By employing the tree-of-thought method, LLMs engage in an
exploratory process to generate diverse and faithful programming tutorials. We
then present , an authoring tool equipped with a series of interactive
visualizations that empower users to have greater control and understanding of
the programming tutorial creation process. A formal user study demonstrated
the effectiveness of , showing that our tool assists users to actively
participate in the programming tutorial creation process, leading to more
reliable and customizable results. By providing users with greater control and
understanding, enhances the user experience and improves the overall quality
of programming tutorial. A free copy of this paper and all supplemental
materials are available at
https://osf.io/uez2t/?view_only=5102e958802341daa414707646428f86.

###### keywords:

Large language model, programming tutorial, authoring tool, interactive
visualizations

provides a series of interactive visualizations of the programming tutorial
authoring process with large language models. The outline view (A) shows the
intermediate generation process in the form of a tree graph, which is also
equipped with interactions for various editing operations. The branch view (B)
enables users to switch between different branches to explore alternative
thought paths. The node space view (C) showcases the content alternatives for
further customization. Each of them is mapped to a paragraph in the tutorial,
describing a code segment of the source code provided by users (D).

Introduction

Programming tutorials are used to teach learners coding skills and how to
solve programming problems. Numerous programmers disseminate their programming
expertise through authoring various tutorials[18], such as blog articles,
online videos, programming games, etc. As the demand to learn programming has
expanded, there has been a rise in the number of tutorial authors (_e.g._ ,
developers[15], computer science educators[11], data scientists[39], etc.),
and the research on improving authoring experiences and workflow of
programming tutorials.

The recent prevalence of Large Language Models (LLMs) has revolutionized the
conventional process of authoring articles [8, 37, 2]. Similarly, programming
tutorial authors can provide LLMs with code snippets and text instructions to
generate tutorial articles. The prominent ability of LLMs significantly
diminishes the effort for authors to complete programming tutorials.

However, the text generated by LLMs is not always perfect or error-free[14].
Sometimes the exhaustive LLM-generated content is also not aligned with users’
expectations. These problems call for further refinement on the auto-generated
content. Owing to the inherent non-transparency of the generation process of
LLMs, users lose access to the intermediate steps of the creation, leading to
limited control and customizability when authoring articles[37]. Additionally,
to achieve ideal modifications often necessitates users’ continuous
interactions with LLMs via text-based conversation interfaces, which can not
adequately support the nuanced and complex requirements of users seeking to
fine-tune tutorials or address specific preferences. It is even more time-
consuming when dealing with programming tutorials, as it requires precise
verification for the correctness and consistency.

The aforementioned limitations pose challenges in the process of authoring
programming tutorials using LLMs. These challenges have an impact on the
overall quality and reliability of LLM-generated programming tutorials. Some
previous works delve in training LLMs to generate code-related content[4, 43]
or proposing advanced prompting methods to promote the generation quality of
LLMs[3, 40]. However, to tackle with programming tutorial task, these
approaches are not out-of-box solutions for generating faithful content
accurately under human’s interventions. Several interactive systems have also
been developed to enhance the efficiency of interactions with LLMs. While
proving useful, these systems primarily concentrate on specific needs, such as
creative writing or data-driven article authoring[46, 37]. Unfortunately,
these approaches are not suitable for the specific context of programming
tutorial authoring, which necessitate accurate and precise information
organized in a more coherent way aligning with the source code. Therefore,
novel approaches need to be developed that account for the need to generate
customizable tutorial and guarantee its faithfulness and consistency with the
underlying source code.

One key point for addressing these issues is to decompose the tutorial
authoring tasks into a step-by-step exploratory process towards final document
accomplishment. This approach enables users to attain a lucid comprehension
and control over the intermediate steps throughout the entire creation
process. We focus on utilizing interactive visualizations to provide insights
into the reasoning paths of model as they progress through the tutorial
authoring tasks. Moreover, this approach allows for flexible and precise human
interventions. For the sake of better understanding the underlying challenges,
we conducted a formative study with 6 experienced tutorial authors to have a
grasp of the writing process, the requirements, and the challenges these
authors encounter when transitioning from their conventional workflow to the
authoring workflow with LLMs. Participants reported that they can not
effectively engage in the creation process using typical conversation
interfaces, and the overhead for comprehending and verifying the tutorial also
created hurdles.

Consequent upon the the findings of formative study, we designed 111 is
derived from“Step-by-step programming tutorial authoring tool”, with the
metaphorical representation of the thoughts generating process of Large
Language Models, symbolizing the system’s goal to enable users a more
controllable experience by utilizing the interactive visualizations of this
process. (Step-by-step programming tutorial authoring tool), an interactive
system that breaks down the tutorial creation task into actionable steps.
These steps are accompanied by interactive visualizations that empower users
to have a greater control and understanding over the exploratory process,
ending with more accurate and reliable results. adopts tree-like prompting
strategies to allow LLMs to generate diverse and faithful tutorial content
more reasonably, which serves as a entry point for subsequent user
interventions to make accurate adjustment on tutorial in different aspects. A
series of purpose-specific interactions are also provided on the basis of the
real-time visualizations of the LLM generation process, aiding users in
crafting the tutorial from multiple levels. To facilitate an intuitive
understanding of the generated content, leverages the inherent connections
between the source code and tutorial, which are distilled by LLMs and
demonstrated with visual representations to users, enabling the explicit
comprehension of the creation process and model’s behavior. To evaluate the
effectiveness of , we conducted a technical evaluation and a user study. The
technical evaluation demonstrated that achieved accurate text-code connections
and generated high-quality programming tutorials. The user study revealed that
effectively assisted users in generating, modifying, and understanding
programming tutorials with LLMs, ultimately leading to more reliable and
customized results.

In summary, this paper makes the following contributions:

  * •

Understanding of the challenges inherent in the process of authoring
programming tutorials with LLMs.

  * •

, an interactive system that utilizes novel prompting strategies and
interactive visualizations to facilitate the step-by-step generation of
programming tutorials with LLMs.

  * •

A technical evaluation demonstrating the effectiveness of our prompting
methods and a user evaluation confirming the usability of and its support for
the programming tutorial authoring workflow.

##  1 Related Work

###  1.1 Programming Tutorial Authoring Tools

Tutorials are a prevalent medium through which developers disseminate coding
and programming knowledge. Typically, a tutorial consists of source code,
textual explanations, code examples, and multimedia components such as images
and videos[13]. Crafting a high-quality tutorial necessitates consideration of
various perspectives[18, 38, 27]. Even experienced authors require significant
time to structure and present the tutorial content. Therefore, numerous tools
have been designed to improve the tutorial authoring process. In-editor
plugins like JTutor[20] and JTourBus[28] facilitate the construction of Java
tutorials within code editors. Some studies leverage interaction provenance
collected from operating systems[26] or web-based editors[32] to capture the
programming process and produce tutorials. For example, VT-Revolution[1]
offers interactive video tutorial creation based on captured user actions,
while Chat.codes[29] promotes collaborative code annotations and explanations.
Other efforts support users in drafting code examples by propagating changes
to multiple code versions[9] or extracting concise, executable code using a
mixed-initiative strategy[12].

Among the many forms of tutorials, the step-by-step structure is particularly
accessible for novice authors due to its straightforward nature. Also,
describing code as _“a tour through the source code”_[28] proves beneficial
for beginners in programming. Therefore, our primary emphasis is on the
authoring of textual content in step-by-step programming tutorials, where LLMs
exhibit optimal performance. We investigate the pain points in the authoring
process and leverage LLMs to assist users in efficiently generating, modifying
and understanding tutorial content, which furthers the line of research on
programming tutorial authoring.

###  1.2 LLMs for Document Generation

The remarkable generative capabilities of LLMs have spurred the proliferation
of LLM-based applications across various domains in writing scenarios. In the
domain of code-related content generation, several studies investigate the
potential of LLMs in crafting code explanations by comparing LLM-generated
content with that sourced from students[21] or previous approaches[17], and
validate its accuracy and comprehensibility. Some endeavors have integrated
these auto-generated code explanations into educational scenarios. For
instance, MacNeil et al. incorporate LLM-generated code explanations within a
web software development e-book[22]. Other efforts utilize LLMs to generate a
myriad of CS learning materials including programming assignments and
multifaceted code explanations[23, 24]. Additionally, Sarsa et al. explore
LLMs’ abilities in creating programming exercises and code explanations, with
evaluations indicating that the generated content is novel, sensible and ready
to use in certain cases[34]. Nonetheless, previous works mostly focus on the
quality of the generated content, assessing the thoroughness and accuracy of
code explanations. For effective presentation to readers, such content often
necessitates additional refinements, either through manual modifications or
carefully-designed prompts. NLP researchers are actively examining diverse
prompt strategies and frameworks to optimize LLMs’ performance[45, 44] or
mitigate _hallucinations_[7, 35], seeking to harness their full operational
potential. Our work builds on top of these endeavors by employing advanced
prompt strategies to more effectively incorporate LLMs into the programming
tutorial authoring workflow. This enables users to obtain reliable and high-
quality tutorials that meet their expectations.

###  1.3 Visual Interfaces for LLMs

Despite the impressive capabilities of LLMs, traditional text-based
conversational interfaces present challenges for complex tasks. Therefore, a
variety of visual interfaces have been developed to enhance user interactions
with LLMs. For instance, researchers have designed interactive interfaces for
visual programming prompt chains to lower the barrier for non-AI-experts[41,
42]. In terms of diverse output representations, Graphologue[16] translates
LLM responses into graphical diagrams, while Sensecape[36] constructs a multi-
level abstraction for information exploration and sensemaking. Both of them
facilitate the information-seeking process. Other works focus on designing
interfaces for specific writing tasks, such as email writing[10], story
creation[46], argumentative writing[47], scientific writing[8] and
journalistic angle ideation[33]. These interfaces ensure users achieve ideal
results through iterative interactions. Notably, DataTales[37] introduces more
flexible and intuitive interactions for authoring data-driven narratives,
while TaleBrush[6] utilizes line sketching interactions with GPT-based models
to control and understand a protagonist’s fortune in co-created stories. More
recently, Kim et al. develop a framework for object-oriented interaction with
LLMs, which can generalize to a wider range of writing tasks[19].

Motivated by these prior studies, our work aims to facilitate the process of
authoring programming tutorials with LLMs. Through a set of interactive
visualizations, our approach empowers users to actively participate in the
generation and refinement of tutorials while maintaining flexibility and
precision in their control over the process. Additionally, we extract and
present visual representations of the connections between source code and the
corresponding generated results, facilitating users’ explicit understanding of
the tutorial content.

##  2 Formative Study

In order to understand how authors create tutorials and figure out problems in
typical workflow with LLMs, we conducted a formative interview study, from
which we conclude the challenges authors may encounter. Based on the findings,
we then derive three design requirements for better programming tutorial
authoring process with LLMs.

###  2.1 Participants and Procedure

Six experienced tutorial authors were interviewed (age from 23 to 28). Four of
them are experienced programmers (E1-4) and two are educators. One of the two
educators teaches computer science (E5), while the other instructs competitive
programming (E6). All of them have more than three years of experience
authoring programming tutorials. In addition, they all have used ChatGPT in
recent months. Before the interview, we collected twenty text-based
programming tutorials from various sources, including Stack Overflow,
Wikipedia, GeeksforGeeks, etc., as well as those auto-generated by LLMs. In
our formal interviews, we first inquired about the authors’ writing process
and commonly-used tools for creating tutorials, then presented them with all
of the collected tutorials and solicited their opinions on the advantages and
downsides of each. Afterwards, participants were asked to create programming
tutorials utilizing ChatGPT and document editors. Finally, we collected
feedback on their authoring experience, focusing on how they utilized LLMs in
their workflow and what made it challenging to create high-quality tutorials
in this process. The interviews were conducted through online meetings and
lasted 45 to 60 minutes.

###  2.2 Findings

All participants tried to author programming tutorials with LLMs. They
provided the source code, retrieved the resulting document, and iteratively
modified it by seeking ChatGPT’s assistance for polishing. We found that they
usually managed to generate different versions with multiple categories of
content to enrich the tutorial. However, during the tutorial authoring process
with LLMs, participants commonly encountered challenges that impacted
production efficiency.

Losing control of the document when generating lengthy content. Participants
often started document authoring with a code snippet, and ended with a
complete article generated by ChatGPT based on the given code. However, most
of them desired for greater flexibility tailored to diverse generation
scenarios. For example, E1 hoped that ChatGPT can “provide a semi-finished
result, allowing him to gradually understand the code and finish the tutorial
step by step, while ChatGPT tended to generate a work that “seems perfect but
lengthy to read” (E1). While in some cases they had intention about next step,
they only needed LLMs to “solidify the ideas” (E3). E5 tried to let ChatGPT
only explain the important parts of the code, but “the lack of context leads
to unsatisfactory results”, forcing him to put effort in modifying the prompt.
Such experiences underscore the need for a more flexible document generation
workflow with ChatGPT in writing scenarios.

Lack of flexible interactions for document modification. As auto-generated
content seldom fully met participants’ expectations, they had to iteratively
modify the tutorial content during the exploration. Some of them sought to
simplify this process using ChatGPT, but only to find it “inconvenient and
confusing” (E6). E6 reported that combining two paragraphs to shorten the
document necessitated a meticulously crafted prompt, otherwise ChatGPT’s
response was merely “a non-sensible mess” (E6). Additionally, participants
mentioned concerns over the tediousness of constantly “switching back and
forth between the code, document, and ChatGPT” (E4), crafting proper prompts,
and ensuring that the newly added content seamlessly integrated into the
existing context.

Significant overhead for understanding and verifying the tutorial. The linear
conversation interface of ChatGPT, where user inputs and LLM outputs appear
sequentially, becomes problematic, particularly in multi-step interactions
with iterative refinement. Participants highlighted the lack of visual
linkages between code and text, complicating their efforts to quickly
understand the tutorial’s structure and find references to verify them. They
usually needed to “browse the whole document” (E5) to discern which code
segment each paragraph is describing, as ChatGPT omitted code references. It
was not uncommon for initial drafts by the LLM to focus on unimportant code
parts, which “just wastes my efforts” (E2). The situation was exacerbated
during modifications, as responses often excluded the original content and
code, forcing participants to “painstakingly match code snippets to their text
because of their separate presentations” (E3). Moreover, the independently
showcased nested code often lost its proper indentation, pushing participants
to trace back to its context. This caused “additional mental overload to
recognize the execution scope” (E2). E2 further noted the need to manually
validate content by “examining the whole document”. Collectively, participants
felt that these unpredictabilities “somewhat negated the intended efficiency
gains of LLMs” (E6).

###  2.3 Design Requirements

To tackle the problems concluded above, we aim to implement an interactive
system to improve the authoring process with LLMs and obtain desirable
programming tutorials. The Design Requirements can be summarized as follows:

DR1. Generating diverse and faithful content. In programming tutorial
authoring, users tend to utilize LLMs to generate a wide range of content,
including variations in structure, level of detail, and other aspects, in
order to inspire their writing ideas. Simultaneously, they require the
generated content aligns precisely with the intended functionality and logic.
Therefore, the system should generate diverse content while maintaining its
faithfulness to the original source code.

DR2. Supporting precise and in-context modification. Users often engage in
iterative interactions with LLMs through text-based conversations to refine
the generated tutorial content according to their preferences. However, even
slight modifications in the input prompts can result in significant and
unforeseen changes in the entire generated output. To maintain the coherence
and accuracy of the final output, the system should support users in making
precise modifications while preserving the contextual integrity of the
generated content.

DR3. Maintaining explicit connection between inputs and outputs. Users usually
need to carefully read the source code and outputs with the purpose of
understanding and verifying the LLM-generated content. However, in linear
conversation interfaces, this task becomes time-consuming and challenging.
Additionally, the lack of transparency and traceability between user inputs
and model outputs can result in inadequate trust and confidence towards the
final tutorial. Therefore, in this system, the tutorial content should be
augmented with explicit visual representations of connections between the
generated content and its corresponding code snippets, which should be
extracted and maintained by the system. This allows users to easily comprehend
and verify the tutorial and the reasoning behind the it.

DR4. Providing intuitive and flexible interactions. Utilizing natural language
as the channel to communicate with LLMs poses challenges for users to clearly
convey their requirements and ideal output. In order to acquire qualified and
satisfactory tutorials, users have to craft them continually, which calls for
cumbersome editing operations and iterations on prompts for generation and
modifications on the draft from ChatGPT. Therefore, to circumvent these
issues, the system should be synthesized with intuitive and flexible
interactions to eliminate users’ efforts for manual adjustment on content or
revision on prompts; thus enable them to concentrate on the tutorial more.

Figure 1: The framework of ’s prompt strategies.

##  3 Prompting for Programming Tutorial Authoring

A primary goal of our research is to make the LLM generation process more
transparent and controllable. To achieve this, we adopt the tree-of-thought
(ToT) methodology[44] to prompt models. It enables models to take intermediate
steps towards completing the programming tutorial. By adopting the ToT
methodology, we allow models to explore multiple choices during generation
process. This promotes a more interactive and iterative approach to tutorial
authoring, empowering users to have greater control over the generated
content. In this section, we present our proposed prompt design that combines
the ToT methodology with the specific requirements of programming tutorial
authoring. Figure 1 illustrates the framework of our prompt strategies. Our
prompt strategies are developed and tested on `gpt-3.5-turbo` version of
ChatGPT[30], one of the most popular LLMs. Please refer to supplemental
materials for original prompts.

###  3.1 Generation of Thoughts

We envision the LLM as an agent that takes on the role of making decisions on
how to plan and execute steps to accomplish programming tutorials. The agent
is instructed to think in a step-by-step manner to generate faithful content,
and expand thoughts in a tree-like structure to produce diverse tutorial
content (DR1).

Dividing task into actions. We break down the programming tutorial creation
task into discrete and actionable steps. This allows the agent to consider
these steps as fundamental units of a plan. Through the formative study
(Section 2), we have identified 5 key components of tutorial content,
including the title, background, code explanation, notification, and summary.
Consequently, we define a list of actions that correspond to writing different
content within the tutorial, e.g., `‘write title’` or `‘write background’`.
The action list and the descriptions of actions are provided to the agent
before solving tasks.

Thinking step by step. To guide the decision-making process leading to
faithful results, we instruct the agent to think step-by-step following the
ReAct (Reason and Act) paradigm[45], which requires the agent to solve the
task in a specific order: observation, thought, action. At each step, the
agent first presents the observation derived from the source code and the
current content of tutorial. Subsequently, it generates the thought of the
appropriate next action to be taken with a reasonable justification. The agent
then takes the action and continues to plan for the next step. This process
continues iteratively until the tutorial creation process is completed. This
systematic approach allows the agent to make informed decisions at each stage
of the tutorial creation process, resulting in faithful and reliable outcomes.

Expanding tree of thoughts. We integrate the tree-of-thought methodology in
the decision-making process to enable the agent to generate multiple potential
thoughts for consideration, leading to diverse content in the generated
tutorial. For example, after explaining a specific code snippets, the agent
may generate thoughts such as writing a notification highlighting common
mistakes related to the code or directly providing a summary to conclude the
tutorial. Consequently, the agent votes on the generated thoughts and make a
decision on the next action to take. To provide an exploratory space for
authors, we maintain the tree of thoughts as a memory space for the agent.
Users are allowed to navigate through the tree of thoughts to explore multiple
potential results of generation.

###  3.2 Manipulation of Model Actions

The above prompt strategies enables a completely automatic process to create
tutorial without human intervention. We propose to perform in-context
interventions by inserting instructions in the gap of each step to manipulate
the model actions. This capability enables authors to make contextual
adjustments to the tutorial materials, resulting in more tailored and
customized programming tutorials (DR2).

In particular, we design prompt strategies to support authors in the
generation, organization, and refinement process:

  * •

Generation: “ _The next step should be to write for $\langle$ code
$\rangle$_”. This prompt is used to enable users to define a specific range of
code to explain. It instructs the agent to thought closely associated with the
specific code fragments when generating new content.

  * •

Organization: “ _Explain $\langle$ code $\rangle$ in the next multiple steps_”
or “ _Explain $\langle$ code $\rangle$ in one paragraph_”. These prompts are
used to adjust the structure of the tutorial. They assist users in organizing
the tutorial by separating the explanation of important code into multiple
steps or summarizing less important code in a concise manner.

  * •

Refinement: “ _Explain $\langle$ code $\rangle$ in the style of $\langle$ style $\rangle$_”, “ _Refine the last step $\langle$ ‘shorter’ | ‘longer’ $\rangle$_”. These prompts are used to refine the writing style or level of detail in a paragraph, which allows users to modify the tutorial content in a more granular manner.

These prompt strategies empower authors to iteratively refine and customize
the tutorial content according to their preferences and needs.

###  3.3 Extraction of Text-Code Connections

The relationship between the LLM-generated tutorial and its underlying code
are not always clarified. The extraction of text-code connections strategy
focuses on identifying and establishing links between relevant textual
explanations and corresponding code snippets (DR3).

We instruct the agent to present each code-related text accompanied with its
connection to the code. The agent are required to provide three parameters
when writing textual descriptions for a specific piece of code: step number,
explanation, and code. We then link the reference code with the source code
using string matching. This strategy is seamlessly integrated into the
generation process, and also ensures that textual explanations are closely
associated with the relevant code, enhancing the overall comprehensibility and
effectiveness of the tutorial materials. The tree of thoughts annotated with
extracted connections helps authors structure their thoughts and logically
sequence the tutorial materials. It assists authors in creating and organizing
tutorial content effectively, aiding authors in maintaining coherence and
clarity throughout the tutorial.

Figure 2: The system interface of . The Code View (A) displays the source
code. The Tutorial View (B) presents the LLM-generated tutorial content.
Between them is the Chain View (C), showing the node chain of current
paragraphs selected by users. The Outline (D), Branches (E), and Node Space
(F) Views provide interactive visualizations for multi-level modifications
such as elaborating, adjusting structure, and polishing content.

##  4 SPROUT

We develop _SPROUT_ , a prototype system that supports efficient and flexible
step-by-step programming tutorial authoring with LLMs according to interactive
visualizations of the intermediate generation process. In this section, we
first present an overview of the system, introducing the interface design and
visual representations in Section 4.1. Next, we introduce a series of
interactions that are designed for enhancing user experience in the process of
tutorial authoring (Section 4.2-Section 4.5).

###  4.1 Overview

Our system consists of 6 views as demonstrated in Fig. 2. The code view (Fig.
2 A) allows users to enter the source code they intend to describe about. The
tutorial view (Fig. 2 B) displays the text content generated by LLM with a
block-based structure, where a block presents a paragraph. Between the code
view and tutorial view lies the chain view (Fig. 2 C), which serves as a
connector, assisting users to navigate between code and tutorial. Within this
view, each node * is a simplified visual representation of the discrete output
units generated during the step-by-step creation process. Each node displays
information about a specific paragraph and the corresponding code it
describes. The chain * is concatenated by these nodes, which collectively
represent the entire tutorial.

The outline view (Fig. 2 D) visualizes the intermediate generation steps of
LLM in the form of a tree graph *. It consists of nodes * and branches *,
illustrating the hierarchical structure of the LLM’s thought process. The
branch view (Fig. 2 E) enables users to switch between different branches * to
explore alternative thought paths and make adjustments to the structure of
generated tutorial. The node space view (Fig. 2 F) offers alternative content
options for a selected node. Users have the flexibility to choose from these
alternatives and customize them in multiple dimensions.

###  4.2 Content Generation from Code Snippets

supports two interactions that enable the generation of diverse and user-
customized content from the code (DR1, DR4).

Agent generation. Sometimes users have limited knowledge about the code they
want to describe or feel hesitated about the next steps to take. In this
situation, they can make the LLM-based agent shoulder the responsibility to
sketch the framework. In the tutorial view (Fig. 2 B), users can click on the
Generate option to start the generation. then automatically generates the
tutorial content step by step, following the tree-of-thought methodology. To
ensure users’ awareness and control over the tutorial generation process,
updates the tutorial content and renders the tree graph in real-time.
Meanwhile, users can use the Pause option to halt the generation process and
review the current tutorial content, ensuring it aligns with their
expectations, especially if the auto-generated content significantly deviates
from what users anticipated.

User-defined generation. In some cases, users have a clear intention regarding
the code snippet they want to write about next. However, they may find it
challenging to effectively communicate their ideas in written form to inform
the LLMs. saves users’ arduous work to write and test various prompts to get a
satisfactory response. Instead, they can simply brush their target code
snippet and click the floating button to effortlessly add new content to the
current tutorial (Fig. 2 A1). Supported by prompting strategies we proposed
(Section 3), generates paragraphs related to the code with consideration of
the coherence to the surrounding contexts. And the node corresponding to the
newly generated content is appended to the end of the current chain. This
ensures that the generated content is seamlessly integrated and follows a
logical progression within the existing tutorial.

###  4.3 Tutorial Modification through Tree Graph

enables users to utilize the tree graph as the primary controller for flexible
and intuitive modification of the tutorial content through manipulating the
nodes within the tree (DR2, DR4). This feature aims to alleviate the need for
users to directly prompt the LLMs and reduces their burden. For instance,
users can consolidate multiple paragraphs to reduce their quantity, emphasize
important code snippets by elaborating on them, and conveniently remove
unnecessary parts from the tree graph for easier content management.

Condensing through node grouping. Sometimes LLMs provide overly exhaustive
content for users, resulting in unclear key points in the tutorial and
overwhelming readers. Users have to manually identify paragraphs that are less
important or too familiar to the majority of readers and condense the content
together in the typical workflow. thus supports node grouping in the outline
view to simplify this process. For example, when a user finds out that the
content of two paragraphs can be integrated into one, they can select them in
the tree graph and navigate to the Group option to make the LLM achieve a
seamless content combination (Fig. 3). The selected nodes will all be replaced
by the newly-integrated node in a new branch rather than modified in the
original branch, ensuring access to their old version.

Figure 3: Select two nodes (A) and group them into one (B).

Elaborating through node splitting. In addition to condensing, elaboration is
another essential task for maintaining a balanced quantity throughout the
entire tutorial. When users explore the tree graph and discover that a key
concept is mentioned but lacks sufficient explanation, they can select the
corresponding nodes and navigate to the Split option. prompts the LLM to
generate additional content, which is then presented in new paragraphs.
Simultaneously, creates a copy of the branch where the selected nodes are
located, and the selected nodes are split into newly-generated nodes (Fig. 2
D1).

Deleting through node trimming. Since showcases complete thinking paths of the
agent when authoring the tutorial, the tree graph may accumulate redundant
nodes as users iterate on the tutorial, making it difficult to navigate and
manage effectively. Therefore, offers a node trimming feature that allows
users to delete specific nodes in the tree graph that contain the undesired
content. When users pinpoint some redundant or unnecessary paragraphs, they
can use the Trim option to delete the corresponding nodes, and their children
nodes will be removed from the tree at the same time (Fig. 2 D1).

###  4.4 Context Switch across Branches

To facilitate a more diverse (DR1) and flexible (DR4) organization of the
tutorial’s content, provides two interactions to assemble paragraphs in
various contexts generated by LLMs.

####  4.4.1 Quick Assembling

supports swift switching between different branches. When users are not
satisfied with the current version of the tutorial and desire to explore
additional possibilities, they can select an arbitrary node in the graph as
the tail of the chain. As a result, the selected node and its ancestor nodes
will be automatically assembled together and replace the original chain (Fig.
4). By selecting an arbitrary node as the starting point for exploration,
users can delve into new directions and uncover untapped potential in the
tutorial’s structure and content.

Figure 4: Users can achieve quick assembling nodes by selecting any node in
the tree graph as the tail of the chain. (A) and (B) show different
combinations and their corresponding document structures.

####  4.4.2 Step-by-Step Assembling

Instead of quickly assembling a chain of nodes, also provides a deliberate
method to manually decide on each step from any selected node in the graph as
the entry point. The branch view (Fig. 2 E) is designed to demonstrate the
details of multiple reasoning paths from the selected parent node, which
encompasses two aspects:

  * •

Choices: When the agent generates several candidate thoughts for the next
step, we recommend the best choice through a voting process. At each step, the
agent casts its vote for the most promising thoughts. then presents the top 3
choices, visually representing the voting results through the line width that
connects the nodes. This visual encoding serves as a reference for evaluating
the reliability of the diverse thinking processes employed by the agent.

  * •

Reasons: instructs the agent to provide explicit reasons for its choices.
Leveraging our prompt strategies, the agent mostly bases its reasoning on the
logical connection between code snippets or the coherence of textual
paragraphs related to nodes. presents the textual reason of each choice on the
links between nodes, enabling users to make more informed and discerning
decisions.

When a candidate node is chosen, the branch view transitions to a deeper
level, allowing users to delve further into exploring the possibilities for
the next paragraph. This process can be repeated iteratively until users
finalize the new framework of the tutorial. Moreover, users retain the
capability to navigate back to previous nodes, allowing them to backtrack and
revise their decisions whenever necessary. This iterative and flexible
approach empowers users to progressively explore and refine their
understanding of the tutorial content.

###  4.5 Detail Refinement in Node Space

After confirming the structure of the tutorial, users often need to further
refine the textual content to align it with their expectations (DR2). They may
not satisfied with the text content pertaining to a particular intent, such as
an explanation for specific code fragments, and try to refine it. To
streamline this process, implements two-part interactions, providing users
with a systematic approach to refine the content and achieve the desired
outcome (DR4).

####  4.5.1 Exploring Alternatives

offers users the ability to chose alternative representations for a specific
intent. LLM-generated paragraphs are visualized as scattered nodes in the node
space view (Fig. 2 F1). We utilize OpenAI’s text embedding model[31] to
measure the semantic similarity of generated paragraphs and projected the
embeddings onto a 2D plane using the UMAP algorithm[25]. As a result,
paragraphs with similar meanings are positioned closer to each other in this
2D space, which facilitates the quick retrieval of semantically relevant
alternatives.

The node alternatives that share the same intent as the user-selected node are
highlighted, with color distinguishing their origin, aligning with the color
encoding in the outline view (Fig. 2 D). Nodes are considered to have the same
intent if they are generated through the same action type and are associated
with the same code fragments. For example, two nodes may both describe the
first line of code but with distinct representations. This view enables users
to filter and select content that aligns with their specific intent, serving
as a foundation for subsequent steps.

####  4.5.2 Rewriting Details

According to the feedback from the formative study, we conclude several
aspects that users primarily focus on when they intend to refine the generated
tutorial content. then allows users to customize the content from the
following aspects:

  * •

Writing styles. Users may have their preferences for the style of the content.
provides several options for users to make LLMs rephrase the content in an
designated style.

  * •

Level of details. Users might be troubled by the unpredictable initial
quantity of LLM-generated content, while an excellent tutorial should be
reasonably detailed. Therefore, allows users to adjust the level of details in
the paragraph while preserving the originally conveyed information.

As shown in Fig. 5, when users find the background is too brief and not
attracting enough, they can adjust the level of details and choose a polishing
style in the configuration panel, then navigate to the Confirm option to
prompt the LLM to rephrase the content accordingly.

Figure 5: In , users can polish the content and adjust the levels of detail of
any paragraph explaining code (A) by setting options in configuration panel
(B), then retrieve polished paragraph (C) from LLM.

###  4.6 Visualization for Tutorial Understanding

To strike a balance between clearly separating two kinds of media resources
(i.e., code and text) and offering visual elements of their connections (DR3),
designs visual representations for users to understand what the text block is
about and where it comes from.

Brief Block. Although the chain view provides the structure of a tutorial
explicitly, it is still arduous work for users to grasp the main idea of each
paragraph. Therefore, prompts LLMs to brief each paragraph in few words to
enable users to quickly comprehend the key concept it intends to convey. To
prevent occupying too much space and hinder access to other parts, these
briefed summaries will be revealed only when users focus on certain nodes by
clicking on them in the chain view (Fig. 2 C).

Elements Matching. To reduce the switch of users’ attention between the code
view and the tutorial content, presents a set of visual cues for element
matching between the source code and the tutorial document. We extract the
connection between the code snippet and paragraph, then present the
information in the node. Furthermore, provides visual links of the connection
between code snippet and text blocks when users focus on certain node or text
block; thus, they can locate the text and code quickly. During the tutorial
generation stage, the focused node will be automatically updated as the latest
generated node, enabling users to follow the LLM’s generation process and
absorb the information in a more moderate way.

##  5 Technical Evaluation

To validate the effectiveness of our prompting technique, we conducted a
technical evaluation focusing on (1) the accuracy of the extracted text-code
connections and (2) the quality of the generated tutorials.

###  5.1 Experiment Settings

Dataset. We collected 10 code snippets with diverse programming languages,
lengths, and structures, covering different categories. These snippets were
fed into our system to automatically generate programming tutorials. Then, we
manually labeled the code segments that each paragraph references,
establishing a ground truth for evaluating connection extraction accuracy.

Procedure. We examined the text-code connections extracted by to calculate
their accuracy. Subsequent analyses were conducted on failure cases and the
contributing factors to these errors. For quality evaluation, we utilized
GPT-4 to rate the tutorials, a method previously employed in prior work[5].
Additionally, we compared the tutorial scores from our system with those from
conventional workflows using single-sentence prompts.

###  5.2 Results

The dataset comprised 10 programming tutorials, encompassing 138 paragraphs
and the corresponding text-code connections. Also, we obtained another 10
programming tutorials using single-sentence prompts for comparison. Please
refer to supplemental materials for detailed results.

Metrics. The extracted text-code connections exhibited an overall accuracy of
86.2%, comprising 119 correct connections and 19 erroneous ones. Additionally,
tutorials produced by received an average quality rating of 8.6 on a 10-point
scale, exceeding the 8.35 score obtained by tutorials generated through
single-sentence prompts.

Error Analysis. Of the 19 erroneous connections identified, we categorized
them into three types: (1) _No Code_ (5/19), (2) _Incorrect Code Range_
(12/19), and (3) _Incorrect Code Content_ (2/19). For the first error type,
the LLM failed to provide any code for the associated paragraph. For the
second error type, the LLM provided an inappropriate code range mismatched
with the corresponding paragraph. Notably, the majority (10/12) presented the
entire code range excessively, whereas only two instances demonstrated
incomplete or unrelated connections. For the third error type, the LLM
provided fictitious code content misaligned with the original code snippet. We
discern these errors as manifestations of LLMs’ _hallucination_ , particularly
evident when employing complex prompting strategies like Tree-of-Thoughts,
coupled with a high temperature which might lead to unexpected results.

Summary. Despite the few failure cases, the results indicate the reliability
and accuracy of our prompting technique in generating programming tutorials
and extracting text-code connections. This efficiently facilitates users in
obtaining faithful and high-quality content, laying a robust foundation for
further refinements.

##  6 User Study

To evaluate the effectiveness of our system in facilitating the programming
tutorial authoring process, we conducted a user evaluation to collect feedback
on (1) the features of and (2) its support for programming tutorial authoring
compared to a baseline interface.

###  6.1 Methodology

####  6.1.1 Participants

We recruited 12 participants (P1-P12) for our experiment, consisting of 5
females and 7 males, aged 22-28, from a local university. P1-6 were
experienced tutorial authors who had also participated in the formative study
(E1-6). The remaining participants were students with experience in code
documentation, but none had experience in authoring programming tutorials. All
participants had more than 4 years of programming experience and had read
programming tutorials in the past. Ten of them were familiar with ChatGPT and
regularly utilized it for writing tasks (_e.g._ , ideation, drafting,
polishing), while two had heard of it but seldom incorporated it into daily
work. Participants attended our experiment through online meetings, used our
system deployed online, and the experiment process was recorded.

####  6.1.2 Experiment Condition

The experiment was conducted under two conditions:

  * •

ChatGPT (_baseline_). The baseline interface offered an environment comprising
a document editor, a code viewer, and ChatGPT. Users could freely arrange
these elements within the interface.

  * •

(_ours_). The system integrated a code viewer and document editor, enriched
with a series of interactive visualizations of the creation process with LLMs.

Each participant was tasked with creating tutorials using both of the
aforementioned authoring tools. The order of tool usage was counterbalanced
among participants: half began with ChatGPT, while the other half started with
.

####  6.1.3 Procedure

The experiment consists of the following phases:

Introduction and Training. First, the purpose of the experiment was presented.
Participants then signed a consent form and filled out a pre-study
questionnaire regarding their background and experience with tutorial
authoring. After that, they were introduced to ’s features, followed by a
demonstration of crafting a complete algorithm tutorial using these features.
We gave the participants adequate time to familiarize themselves with the
system and encouraged them to reproduce the tutorial independently. All
participants were free to ask any clarifying questions.

Targeted Task. For this task, participants were provided with a piece of code
for reference. Their objective was to author a tutorial based on a given
framework, which involved two sub-tasks: modifying the content of at least one
paragraph and describing the main idea of each paragraph. Participants
underwent two trials in this phase, creating tutorials in each trial using two
different tools – _baseline_ and _ours_ – respectively.

Open-ended Task. For this task, only the problem’s background and the source
code were provided. Participants were encouraged to create a step-by-step
programming tutorial freely using . They were expected to produce at least
three tutorial versions and select one as the final result. We ensured
participants had enough time to validate the generated content and prepare
their drafts. There were no time constraints on the writing process during
trials, allowing them to create tutorials to the best of their abilities.

Post-study Interview. Upon completion of all trials, we conducted a semi-
structured interview with each participant. They were first invited to fill
out a questionnaire with 5-point Likert-scale questions, regarding five
features in , the general usability, and the support for programming tutorial
authoring with our system and the baseline. Then, we inquired their thoughts
on how could help their actual workflow, the advantages and limitations of the
current system design, as well as their suggestions for potential
improvements.

Figure 6: User feedback regarding the features in , measured on a 5-point
Likert-scale.

###  6.2 Results Analysis

####  6.2.1 Features

We evaluated five features of through 5-point Likert-scale questionnaires in
the post-study interview. Figure 6 illustrates the detailed ratings.

Participants chose different generation methods based on their needs. They
responded favorably to the feature of generating content based on selected
code, finding it useful when they “already have an initial framework in mind”
(P6) for authoring content related to familiar code. The free generation
method was also praised since it could “provide diverse perspectives as a
starting point when at lost” (P3) and allowed them to “pick up a best one
among different versions” (P1).

Participants utilized visual connections to quickly grasp the tutorial
content. The visual representations of connections were favored by all
participants, as it alleviated the effort to “maintain the connections in
mind” (P5). In traditional chat-based interfaces, even if referred code is
provided, participants sometimes still needed to “scroll back to check the
position of the code snippet within the complete code” (P11) and “the
indentation of the code is missing” (P2).

Participants switched between thoughts smoothly and sensibly. According to
some participants, utilizing the branch view with thoughts is like “building
text blocks under the guidance of LLMs” (P3). Moreover, the quick assembling
in Outline by selecting leaf nodes enables “swift comparison” (P9) between
documents based on different thoughts.

Participants employed the tree graph to modify the tutorial. Most participants
praised the graph-level modifications such as splitting node, describing them
as “novel and convenient interactions within a document” (P7) that would be
“frequently used” (P10). For example, P6 split the node which described the
main loop in Dijkstra algorithm, because he found the content was too brief.
Similarly, P8 utilized “Trim” to delete the content describing the engineering
process, which “can be omitted in a tutorial”.

Participants achieved easy detail refinement in node space. When focusing on
specific content from LLMs, the node space serves as “a good container which
collects all the content for me” (P4). also enables purpose-specific content
refinement. For instance, P9 easily got a concise paragraph in , whereas with
ChatGPT, he had to test prompts iteratively. However, P7 expressed the
dissatisfaction about this feature because their demands were not covered
currently, such introducing metaphors in the tutorial.

Figure 7: The results of the questionnaire regarding the authoring support of
our system and the baseline.

####  6.2.2 Tutorial Authoring Support Analysis

At the end of the study, we asked participants to rate the two authoring tools
across seven dimensions. The results are displayed in Fig. 7. We then
collected their feedback on the tools’ usability and their effectiveness in
supporting real-world workflow. We also investigated how participants combined
the features of to support their individual tutorial creation processes.

In general, participants found more effective in assisting them to generate,
modify, and understand tutorials compared to the baseline, which also led to
more confidence in the quality of output produced using . Although did not
receive scores as high as the baseline in terms of “easy to learn” due to
learning barriers of visualizations and interactions, participants reported
that it became easy to use once familiarizing with it. Furthermore, they
expressed willingness to use it again.

enables flexible tutorial generation for enhanced engagement. Based on
participants’ feedback, the generation methods in allow them to conveniently
tailor LLM-generated content to match their expectations, offering different
degrees of autonomy. In contrast, it is challenging with ChatGPT to maintain
such control. For example, P4 combined two methods for more flexible
generation. He began with the free generation as the “initial execution for
inspirations”, paused the process, and then searched for “a reasonably
detailed content” as a starting point. After spotting a favorable structure,
he chose to finish the tutorial by generating content based on the selected
code, ensuring the outcome aligned with his expectations.

provides easier modifications on tutorials hierarchically. Participants
reported that the interactions provided in enabled easy tutorial modifications
across various aspects, eliminating the need for “repeatedly writing prompts”
(P11). Some participants concentrated on the narrative order of the tutorial,
while others tended to manipulate the style or detail levels of certain
paragraphs. Often, participants employed a combination of these methods. For
example, P3 first decided the tutorial structure according to LLM
recommendations, then grouped some nodes to shorten the tutorial, and finally
refined the text of an unsatisfactory paragraph. In contrast, when working
with ChatGPT, participants had to interact in multiple rounds, as the prompts
were “ambiguous to convey their demands” (P7).

enhances tutorial comprehension and reliability during the authoring process.
Participants reflected that they could produce tutorials with more confidence
and reliability on the outcomes throughout the workflow of . During the
generation process, the connections are updated synchronously, which “offers
clear visual cues” (P1) about the content of each paragraph. P12 also pointed
out that when utilizing ChatGPT to modify content, the revised content often
appeared separately from the original tutorial and code, leaving him uncertain
about its correctness. Moreover, we found that most participants would browse
the entire tutorial to guarantee the key points were covered. In , the visual
connections can support users to “easily verify the coverage of the provided
code” (P7), thus speeding up the examination process.

##  7 Discussion and Future Work

In this section, we reflect on our research, mainly discussing the
implications, limitations, and future work.

###  7.1 Implications

Incorporating LLMs into programming tutorial authoring workflow. The
development of large language models opens vast possibilities to facilitate
the programming tutorial authoring process through reducing manual effort on
writing and editing. While our work provides a step-stone towards programming
tutorial authoring with the assistance of LLMs, future work could investigate
on further enriching tutorials with more multifaceted content, such as
execution examples, flow charts, or figures illustrating code execution
processes, which calls for utilizing multi-modal approaches and specific
interactions to combine versatile media into current workflow. We believe that
our work can be a solid foundation for future researches on authoring
programming tutorials. By incorporating a wider array of materials, we can
ultimately create content with higher quality, granting readers a richer
understanding of the programming concepts presented.

Step-by-step exploratory process offers a more controlled authoring
experience. Numerous studies have delved into advancing strategies that prompt
LLMs to think step by step[44, 45, 40] for higher quality content. Beyond
this, we see potential in using this decomposed process as an entry point to
grant users broader access to the intermediate steps of the authoring
workflow. This empowers them to achieve anticipated outcomes, transforming a
typically passive experience into an active one. The evaluation results
indicated that the interactive, segmented representations of the creation
process and generated tutorial content in offer more precise generation and
allow for multi-level adjustments to modules. Such an approach addresses the
often-encountered challenge of accurately conveying intentions in traditional
chat-based interfaces.

Maintaining connections between provided inputs and LLM-generated outputs
ensures transparency and traceability. We identify and visualize latent
connections between the source code and the generated tutorial, as we believe
it can assist users to comprehend and verify the reasoning behind LLMs’
creation process. Informed by our evaluation results, users utilized these
presented connections to examine the outcomes after generating or modifying
procedures, easing the burden of spending extra cognitive efforts to recognize
and memorize them. Moreover, consistently maintaining these connections
significantly improves user confidence in the final results crafted
collaboratively by users and LLMs. This is essential in materials-backed
authoring scenarios where consistency and correctness are vital.

Striking a balance on the degree of interaction integrity in interface design.
A key finding is that user preferences for interacting with LLMs vary by
individuals. For example, depending on the specific operations they intend to
perform, some individuals preferred utilizing integrated interactions as a
means to effortlessly obtain accurate outcomes, while others expressed
reservations regarding the comprehensiveness of these approaches in covering
all possible scenarios. There exists a conflict between the extensiveness of
features coverage and the precision of the results after execution when
designing interactions. Thus, our research could inspire future studies to
more systematically explore the interface design paradigm for LLMs, ranging
from completely-free form to structured module-based interactions.

###  7.2 Limitations and Future Work

Although our work has shown promising results, there are several limitations
and opportunities for future research.

Scalability. Although our system effectively supports tutorial authoring
process in most cases, it may face limitations in adequately covering the
context when dealing with lengthy source code and tutorials, primarily due to
token limitations. Nevertheless, these limitations can be addressed with
advancements in LLM capabilities or by utilizing memory summarization
techniques.

Potentiality. While our study focuses on modifying content within a single
chain of thought, the current design framework can support more varied
tutorial revision operations. For example, future work could explore
facilitating users in merging thoughts from two distinct paths, ensuring
context preservation. This enhancement, rooted in our current prompting
strategies, would also necessitate broadening the scope of graph manipulations
to enable more flexible interaction.

Generalizability. Although our study focuses on utilizing interactive
visualizations for programming tutorial authoring, the methodologies employed
herein can potentially be extended to other creation scenarios requiring
consistency with reference materials, such as fact-based analysis documents.
The underlying idea involves designing purpose-specific and scenario-oriented
prompting methods to enable appropriate exploratory workflows. Further
exploration into multi-modal approaches can benefit tasks involving non-
textual elements, like data visualization. Based on the nature of source
materials and expected outcomes, visual representations beyond tree graphs
should be explored to enhance comprehension and manipulation during the
creation process. For example, future researches can investigate how to reify
the creation process with LLMs when handling non-linear structured data.

Limitations. While the current prompting methods are sufficiently reliable and
accurate for the authoring workflow, the extraction accuracy can be further
enhanced. Future work could focus on utilizing correction prompts to examine
the extracted connections. Additionally, the integration of add-on modules to
identify connections could be investigated to guarantee a higher accuracy.

##  8 Conclusion

This work presents , a programming authoring tool with LLMs which breaks down
the programming tutorial authoring task into actionable steps and adopts novel
prompting strategies to generate high-quality and diverse tutorial content.
With a series of interactive visualizations of the LLM generation process,
users are able to effectively engage in an exploratory process to generate,
modify, and understand the generated tutorial. A user study with 12
participants demonstrates that users can effectively utilize to author
programming tutorials with LLMs in a transparent and traceable process,
leading to more controllable and reliable results.

## References

  * [1] L. Bao, Z. Xing, X. Xia, and D. Lo.  Vt-revolution: Interactive programming video tutorial authoring and watching system.  IEEE Trans. Software Eng., 45(8):823–838, 2019.
  * [2] S. Biswas.  Chatgpt and the future of medical writing, 2023.
  * [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.  Evaluating large language models trained on code.  CoRR, abs/2107.03374, 2021.
  * [5] C.-H. Chiang and H.-y. Lee.  Can large language models be an alternative to human evaluations?  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631. Association for Computational Linguistics, Toronto, Canada, July 2023. doi: 10 . 18653/v1/2023 . acl-long . 870
  * [6] J. J. Y. Chung, W. Kim, K. M. Yoo, H. Lee, E. Adar, and M. Chang.  Talebrush: Sketching stories with generative pretrained language models.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–19, 2022.
  * [7] S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston.  Chain-of-verification reduces hallucination in large language models, 2023.
  * [8] K. I. Gero, V. Liu, and L. Chilton.  Sparks: Inspiration for science writing using language models.  In Designing interactive systems conference, pp. 1002–1019, 2022.
  * [9] S. Ginosar, L. F. D. Pombo, M. Agrawala, and B. Hartmann.  Authoring multi-stage code examples with editable code histories.  In UIST, pp. 485–494. ACM, 2013.
  * [10] S. M. Goodman, E. Buehler, P. Clary, A. Coenen, A. Donsbach, T. N. Horne, M. Lahav, R. MacDonald, R. B. Michaels, A. Narayanan, et al.  Lampost: Design and evaluation of an ai-assisted email writing prototype for adults with dyslexia.  In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–18, 2022.
  * [11] S. Hamouda, S. H. Edwards, H. G. Elmongui, J. V. Ernst, and C. A. Shaffer.  Recurtutor: An interactive tutorial for learning recursion.  ACM Trans. Comput. Educ., 19(1):1:1–1:25, 2019.
  * [12] A. Head, E. L. Glassman, B. Hartmann, and M. A. Hearst.  Interactive extraction of examples from existing code.  In CHI, p. 85. ACM, 2018.
  * [13] A. Head, J. Jiang, J. Smith, M. A. Hearst, and B. Hartmann.  Composing flexibly-organized step-by-step tutorials from linked source code, snippets, and outputs.  In CHI, pp. 1–12. ACM, 2020.
  * [14] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung.  Survey of hallucination in natural language generation.  ACM Comput. Surv., 55(12):248:1–248:38, 2023.
  * [15] H. Jiang, J. Zhang, Z. Ren, and T. Zhang.  An unsupervised approach for discovering relevant tutorial fragments for apis.  In ICSE, pp. 38–48. IEEE / ACM, 2017.
  * [16] P. Jiang, J. Rayan, S. P. Dow, and H. Xia.  Graphologue: Exploring large language model responses with interactive diagrams.  arXiv preprint arXiv:2305.11473, 2023.
  * [17] J. Y. Khan and G. Uddin.  Automatic code documentation generation using gpt-3.  In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–6, 2022.
  * [18] A. S. Kim and A. J. Ko.  A pedagogical analysis of online coding tutorials.  In SIGCSE, pp. 321–326. ACM, 2017.
  * [19] T. S. Kim, Y. Lee, M. Chang, and J. Kim.  Cells, generators, and lenses: Design framework for object-oriented interaction with large language models.  In The 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), October 29-November 1, 2023, San Francisco, CA, USA, UIST ’23. Association for Computing Machinery, New York, NY, USA, 2023. doi: 979-8-4007-0132-0/23/10
  * [20] C. Kojouharov, A. Solodovnik, and G. Naumovich.  Jtutor: an eclipse plug-in suite for creation and replay of code-based tutorials.  In eTX, pp. 27–31. ACM, 2004.
  * [21] J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas.  Comparing code explanations created by students and large language models.  arXiv preprint arXiv:2304.03938, 2023.
  * [22] S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen.  Experiences from using code explanations generated by large language models in a web software development e-book.  In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 931–937, 2023.
  * [23] S. MacNeil, A. Tran, J. Leinonen, P. Denny, J. Kim, A. Hellas, S. Bernstein, and S. Sarsa.  Automatically generating cs learning materials with large language models.  arXiv preprint arXiv:2212.05113, 2022.
  * [24] S. MacNeil, A. Tran, D. Mogil, S. Bernstein, E. Ross, and Z. Huang.  Generating diverse code explanations using the gpt-3 large language model.  In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2, pp. 37–39, 2022.
  * [25] L. McInnes, J. Healy, and J. Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [26] A. Mysore and P. J. Guo.  Torta: Generating mixed-media GUI and command-line app tutorials using operating-system-wide activity tracing.  In UIST, pp. 703–714. ACM, 2017.
  * [27] S. M. Nasehi, J. Sillito, F. Maurer, and C. Burns.  What makes a good code example?: A study of programming q&a in stackoverflow.  In ICSM, pp. 25–34. IEEE Computer Society, 2012.
  * [28] C. Oezbek and L. Prechelt.  Jtourbus: Simplifying program understanding by documentation that provides tours through the source code.  In ICSM, pp. 64–73. IEEE Computer Society, 2007.
  * [29] S. Oney, C. Brooks, and P. Resnick.  Creating guided code explanations with chat.codes.  Proc. ACM Hum. Comput. Interact., 2(CSCW):131:1–131:20, 2018.
  * [30] OpenAI.  Introducing ChatGPT.  https://openai.com/blog/chatgpt.  [Online; accessed 2023-04-30].
  * [31] OpenAI.  Introducing text and code embeddings.  https://openai.com/blog/introducing-text-and-code-embeddings.  [Online; accessed 2023-10-01].
  * [32] E. L. Ouh, B. K. S. Gan, and D. Lo.  ITSS: interactive web-based authoring and playback integrated environment for programming tutorials.  In ICSE (SEET), pp. 158–164. IEEE, 2022.
  * [33] S. Petridis, N. Diakopoulos, K. Crowston, M. Hansen, K. Henderson, S. Jastrzebski, J. V. Nickerson, and L. B. Chilton.  Anglekindling: Supporting journalistic angle ideation with large language models.  In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–16, 2023.
  * [34] S. Sarsa, P. Denny, A. Hellas, and J. Leinonen.  Automatic generation of programming exercises and code explanations using large language models.  In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1, pp. 27–43, 2022.
  * [35] R. Sennrich, J. Vamvas, and A. Mohammadshahi.  Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding, 2023.
  * [36] S. Suh, B. Min, S. Palani, and H. Xia.  Sensecape: Enabling multilevel exploration and sensemaking with large language models.  arXiv preprint arXiv:2305.11483, 2023.
  * [37] N. Sultanum and A. Srinivasan.  Datatales: Investigating the use of large language models for authoring data-driven articles.  arXiv preprint arXiv:2308.04076, 2023.
  * [38] R. Tiarks and W. Maalej.  How does a typical tutorial for mobile development look like?  In MSR, pp. 272–281. ACM, 2014.
  * [39] A. Y. Wang, D. Wang, J. Drozdal, M. J. Muller, S. Park, J. D. Weisz, X. Liu, L. Wu, and C. Dugan.  Documentation matters: Human-centered AI system to assist data science code documentation in computational notebooks.  ACM Trans. Comput. Hum. Interact., 29(2):17:1–17:33, 2022.
  * [40] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In NeurIPS, 2022.
  * [41] T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai.  Promptchainer: Chaining large language model prompts through visual programming.  In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–10, 2022.
  * [42] T. Wu, M. Terry, and C. J. Cai.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts.  In Proceedings of the 2022 CHI conference on human factors in computing systems, pp. 1–22, 2022.
  * [43] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn.  A systematic evaluation of large language models of code.  In MAPS@PLDI, pp. 1–10. ACM, 2022.
  * [44] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  CoRR, abs/2305.10601, 2023.
  * [45] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao.  React: Synergizing reasoning and acting in language models.  In ICLR. OpenReview.net, 2023.
  * [46] A. Yuan, A. Coenen, E. Reif, and D. Ippolito.  Wordcraft: story writing with large language models.  In 27th International Conference on Intelligent User Interfaces, pp. 841–852, 2022.
  * [47] Z. Zhang, J. Gao, R. S. Dhaliwal, and T. J.-J. Li.  Visar: A human-ai argumentative writing assistant with visual programming and rapid draft prototyping.  ArXiv, abs/2304.07810, 2023.