# Not Just Novelty: A Longitudinal Study on Utility and Customization of AI
Workflows

Tao Long<EMAIL_ADDRESS>Columbia UniversityNew York CityNew YorkUSA ,
Katy Ilonka Gero<EMAIL_ADDRESS>Harvard
UniversityCambridgeMassachusettsUSA and  Lydia B. Chilton
<EMAIL_ADDRESS>Columbia UniversityNew York CityNew YorkUSA

(2024)

###### Abstract.

Generative AI brings novel and impressive abilities to help people in everyday
tasks. There are many AI workflows that solve real and complex problems by
chaining AI outputs together with human interaction. Although there is an
undeniable lure of AI, it’s uncertain how useful generative AI workflows are
after the novelty wears off. Additionally, tools built with generative AI have
the potential to be personalized and adapted quickly and easily, but do users
take advantage of the potential to customize? We conducted a three-week
longitudinal study with 12 users to understand the familiarization and
customization of generative AI tools for science communication. Our study
revealed that the familiarization phase lasts for 4.3 sessions, where users
explore the capabilities of the workflow and which aspects they find useful.
After familiarization, the perceived utility of the system is rated higher
than before, indicating that the perceived utility of AI is not just a novelty
effect. The increase in benefits mainly comes from end-users’ ability to
customize prompts, and thus appropriate the system to their own needs. This
points to a future where generative AI systems can allow us to design for
appropriation.

workflow, longitudinal user experience, customization, technology
appropriation, ownership, familiarization, mental model, AI chains,
scaffolding, science communication, novelty, generative AI, LLMs

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Empirical
studies in HCI††ccs: Computing methodologies Natural language generation††ccs:
Human-centered computing Human computer interaction (HCI)††ccs: Social and
professional topics Computer science education††ccs: Human-centered computing
User studies

##  1\. Introduction

Generative AI has impressive abilities to generate text, images, and code
based on simple prompts. Building on this, a powerful technique to solve more
complex problems is to break problems into AI workflows that chain together
prompts (Wu et al., 2022b). Often, the steps of the chain involve interfaces
with human interaction to check the results and steer progress in productive
directions. This can enable more complex processes such as following divergent
and convergent thinking (Wang et al., 2023; Jiang et al., 2023), the human-
centered design process (Liu et al., 2022; Petridis et al., 2023a; Wang et
al., [n. d.]), the process model of writing (Mirowski et al., 2023; Gero et
al., 2022; Zhang et al., 2023), and simulating thoughts, behaviors and actions
of a group of people interacting (Park et al., 2023). Studies of AI workflows
tend to show that people find them useful. They save time (Long et al., 2023)
and mental effort (Petridis et al., 2023a; Mirowski et al., 2023), and they
can produce more output in a fixed time (Liu et al., 2022).

However, most of these systems test their utility in a single lab session. The
presence of AI in a workflow brings great excitement, expectations, and hype.
But could this just be an effect of the novelty of including AI in a workflow?
For workflows that provide scaffolding, it is natural to assume that
scaffolding might not be needed after the user has internalized the workflow.
Workflows are often criticized for pushing users towards similar outputs and
constraining diversity and creativity. Moreover, workflows can be brittle —
although they may work for one set of well-defined tasks, they might not
generalize to more complex examples that users encounter in the wild. Is the
success of AI workflows a product of the novelty effect, or is there a lasting
benefit?

The novelty effect is a common effect seen in technology adoption (Rodrigues
et al., [n. d.]; Sadeghi et al., 2022; Shin et al., [n. d.]) where the desire
to use a new technology is initially high but then diminishes over time. Some
technologies overcome the novelty effect, and users found more useful after
familiarization (Jeno et al., 2019; Rodrigues et al., [n. d.]), but some do
not (Sadeghi et al., 2022; Shin et al., [n. d.]). Even when there is a novelty
effect, the familiarization effect can actually increase engagement over time
(Rodrigues et al., [n. d.]). Technology that becomes familiar can become
internalized and thus used with less effort, leading to greater rewards. Long-
term studies on education technologies indicate that technologies that cater
to users’ needs and feelings of autonomy can overcome the initial novelty
effect (Jeno et al., 2019). Currently, it is unclear whether the appeal of AI
workflows stems from a novelty effect or if there is a lasting benefit past
the workflow familiarization.

We conducted a three-week longitudinal study with 12 users to understand the
familiarization and customization of generative AI tools for science
communication. We study a 7-step human-AI workflow to help STEM experts write
tweets that motivate a topic in their field to the general public (Long et
al., 2023). The science communication task is called a Tweetorial hook (Gero
et al., 2021), which is challenging because it requires STEM experts to think
about examples or experiences with the topic the public might relate to. For
example, the web programming topic of AJAX is very technical, but users have
all experienced it on social media when new posts load as they scroll without
users explicitly requesting them. A previous study (Long et al., 2023) shows
that in a single session, the large language model (LLM)-based workflow
reduced the mental demand for writing a Tweetorial hook compared to writing
without it (simply using web search). We recreated this workflow and added two
simple customization features. First, we display the LLM prompt used at every
stage and allow users to edit it to customize the responses and the workflow.
Second, we add their hooks to the prompts as training examples so the system
can be personalized based on examples. Then, we study a three-week deployment
of the system, over which participants use the system in 10 different sessions
to write ten different hooks. We identify if and when familiarization occurs.
To understand whether there is a novelty effect, we use surveys, observations,
and interviews to assess the system’s usefulness before and after
familiarization.

Our study revealed that:

  * •

The familiarization phase lasts an average of 4.27 sessions, during which the
system is already perceived as useful (5.47/7).

  * •

The system is rated 12.1% more useful after familiarization, indicating that
the novelty does not wear off (from 5.47/7 to 6.32/7, p-value ¡ 0.005).

  * •

This change in the system’s usefulness can mainly be attributed to the
increased usefulness of the prompt editing feature (an 11.4% improvement, from
5.07/7 to 5.87/7, p-value ¡ 0.005). The other features show no change in their
usefulness scores throughout the study.

  * •

Users edit prompts for three major reasons: to keep generations relevant, to
guide narrative directions and styles, and to make the workflow more
efficient. Chaining users’ previous hooks to the prompt as training examples
shows no significant benefits.

  * •

After familiarization, user’s mental models of the workflow did not change,
but their perceptions of ownership increased. There were nuanced ways that
users experienced ownership after multiple experiences with the system, but
there is a clear relationship between involvement and ownership.

Though these results stem from one AI workflow over three weeks, it is
evidence that generative AI workflows have the potential to escape the novelty
effect. More importantly, the improvements in the system’s perceived
usefulness are due mostly to prompt editing. This is consistent with
appropriation theory in HCI (Dourish, [n. d.]; Dix, 2007), which shows how
users can make their system their own and adapt it to purposes the designers
had not intended (and potentially add back into the system). For generative AI
workflows, natural language prompts also simplify the customization process,
prompting a future of systems and workflows that facilitate user customization
and appropriation of tools, as well as a continuous co-design process between
system developers and users.

##  2\. Related Works

###  2.1. Generative AI-Based Workflows

Generative AI models hold huge potential for supporting various tasks with
their domain knowledge and contextual understanding capabilities (OpenAI, [n.
d.]; Bommasani et al., [n. d.]). For example, users can use LLMs for simple
tasks like answering math questions and complex ones such as reviewing and
editing academic papers or even writing screenplay scripts. However, for those
harder ones, LLMs can hardly solve and write with one prompt and one linear
interaction (Wu et al., 2022b; Grunde-McLaughlin et al., [n. d.]).
Specifically, LLMs face challenges in producing high-quality outputs when LLMs
have limited knowledge about the task (Eloundou et al., 2023) or multi-step
reasoning is required (Wei et al., [n. d.]). Also, due to the limitations of
the underlying generative models, LLM outputs often contain hallucination,
repetitiveness, and vague language (Holtzman et al., 2020; Ippolito et al.,
2019; Bommasani et al., [n. d.]).

AI workflows help solve complex real-life problems and mitigate LLM
shortcomings. As the foundation structure of workflows, AI chains (Wu et al.,
2022b), or known as LLM chaining, break down large complex tasks into
manageable steps and link LLM steps together for effective problem-solving. In
this process, the outputs of previous steps serve as the inputs for the
following LLM calls, aggregating gains at each step to reach the final
artifact (Grunde-McLaughlin et al., [n. d.]; Wu et al., 2022a, b). This multi-
step problem-solving process with LLMs helps tackle the complex tasks that
LLMs are not well-trained for or able to do in one shot. Also, it provides
user control for users to find errors, fix errors, and improve generations
according to their preferences (Grunde-McLaughlin et al., [n. d.]; Long et
al., 2023; Wu et al., [n. d.]).

AI workflows successfully complement users with guided support in many
traditionally cognitively demanding tasks. For example, various workflows
demonstrate their strong system performances or satisfactory user experiences
to support writing (Wang et al., [n. d.]; Long et al., 2023; Mirowski et al.,
2023; Li et al., [n. d.]; Schick et al., [n. d.]; Yang et al., [n. d.];
Petridis et al., [n. d.]; Shen et al., 2023), brainstorming (Wang et al.,
2023; Petridis et al., 2023a; Bursztyn et al., [n. d.]; Kim et al., [n. d.];
Wu et al., [n. d.]; Zhang et al., 2023; Choi et al., [n. d.]; Seo et al., [n.
d.]; Suh et al., 2023), visual design (Liu et al., 2022, 2023; Jiang et al.,
2023; Petridis et al., 2023b; Kim et al., 2022; Tseng et al., [n. d.]; Son et
al., [n. d.]; Feng et al., [n. d.]; Xiao et al., [n. d.]; Huang et al., [n.
d.]) and question answering and sensemaking (Eisenstein et al., [n. d.]; Xie
et al., 2023; Du et al., [n. d.]; Kazemitabaar et al., [n. d.]; Arawjo et al.,
[n. d.]; Gao et al., [n. d.]; Suh et al., [n. d.]; Zhou et al., [n. d.]; Li et
al., 2023).

However, most of these studies are conducted solely within the framework of
controlled user studies lasting typically 30 to 60 minutes. This approach
might only capture users’ initial reactions of satisfaction or dissatisfaction
with the system rather than providing more comprehensive experiences
(Greenberg and Buxton, 2008; Blaynee et al., 2016; Sporka et al., 2011; Jain
and Balakrishnan, 2012). Thus, a gap exists in understanding how AI workflows
perform and whether they continue to aid users well after repeated usage.

###  2.2. Novelty Effect

The novelty effect is a common effect seen in technology adoption (Lai, 2017;
Koch et al., 2018; Davis et al., 2013), where the desire to use new technology
is initially high, but then diminishes over time. It is related to humans’
innate desire for novelty and our psychological stress response: individuals
exhibit a stronger stress response the first time they face a challenging or
threatening experience (Rutten and Geerts, 2020; Hekkert et al., [n. d.]).
With the stress, users are more likely to focus, embrace learning through
trials and errors with a higher tolerance, and perceive higher performance
(Rutten et al., 2021; Rutten and Geerts, 2020; Kosch et al., 2023; Nielsen,
[n. d.]). However, novelty effects will eventually diminish as users become
more familiar with the technology, with the duration varying depending on the
technology (Koch et al., 2018; Lai, 2017).

After the novelty wears off, some technologies fail to overcome this effect,
wherein users find them less useful after becoming familiar with them,
ultimately leading to dissatisfaction and abandonment (Sadeghi et al., 2022;
Shin et al., [n. d.]; Voit et al., 2020; Long and Chilton, [n. d.]). However,
many technologies successfully navigate this challenge, as users continue to
find them useful over time (Jeno et al., 2019; Rodrigues et al., [n. d.]).
Also, even though the novelty effect plays a role, familiarizing these
technologies can increase users’ engagement and perceived usefulness over time
(Rodrigues et al., [n. d.]). Technology that becomes familiar can become
internalized and thus used with less effort, leading to greater rewards.
Additionally, long-term studies on education technologies indicate that tools
that cater to users’ needs and feelings of autonomy can overcome the initial
novelty effect (Jeno et al., 2019). This is aligned with self-determination
theory (Deci and Ryan, 2000) in educational tools and technology: when the
needs for autonomy, competence, and relatedness are satisfied, positive
outcomes follow (Baumeister and Leary, 1995; Ryan and Deci, 2017; MacLean et
al., 1990).

Generative AI workflows also draw concern regarding the strong novelty effect
due to their seemingly dazzling performance (Mukherjee and Chang, 2023). There
exists a gap in examining whether the benefits of AI tools and workflows hold
up and what happens after the novelty wears off for these workflows.

###  2.3. Technology Customization and Appropriation

Users customize and appropriate the technology over usage and time. Many long-
term studies (Kjærup et al., 2021; Zhou et al., 2023; Shin et al., 2023;
Lindley et al., 2017; Levordashka et al., 2023; Hasan et al., 2023; Han et
al., 2023; Benke et al., 2022) and technology domestication theory (Berker et
al., 2005) mapped user’s journey of adopting and customizing these
technologies. Beginning with the learning and comprehending the system
affordances, especially the essential features, users navigate the learning
phase (Davis et al., 2013; Qiu et al., 2023). Once they better understand the
technology and the supported task, users engage in customization to shape the
technology for better alignment with their own preference (Han et al., 2023;
Belin and Prié, 2012; Moran, [n. d.]). Then, users start to appropriate the
technology by modifying it and applying it for new use cases that the original
designers had not anticipated (Dix, 2007; Carroll et al., 2001; Dourish, [n.
d.]; Muller et al., 2016; Krischkowsky et al., [n. d.]). Along with the
increased familiarization and insights gained over usages, users gradually
develop their own perceptions, mental models, and opinions towards the system
(Ross et al., 2023; Han et al., 2023; Gero et al., 2023; Kjærup et al., 2021).
Technologies that allow customization and appropriation can elevate user
satisfaction and performance and meet more user needs (Dix, 2007; Proença et
al., 2021; Feng et al., [n. d.]; Kim et al., 2023). Both customization and
appropriation offer users a better sense of control and ownership over the
system and its generations (Dix, 2007; Marathe and Sundar, 2011).

A 2007 paper (Dix, 2007) offers guidelines to design systems that support
customization and appropriation, outlining three key principles. First, the
system should ”allow interpretation”: enable users to assign their
interpretation to system elements instead of fixing them. Second, it needs to
”provide visibility”: make the system’s functionality transparent for users to
understand the effects of their actions. Third, it needs to ”support not
control” and offer ”plugability and configuration”: design for efficiency and
flexibility, allowing the system to automate and users to combine parts of the
system according to their needs.

For generative AI workflows, natural language prompts largely simplify the
customization process. Given the flexibility to alter the system through word-
level changes, users can easily customize workflows by tweaking prompts or
incorporating examples from their previous work. It broadens the access to
customization for non-experts (Zamfirescu-Pereira et al., 2023; Liu and
Chilton, 2022), in contrast to traditional methods requiring developers and
designers to engage directly with users for system updates. While this form of
customization holds the potential to enhance the utility of AI systems over
time, it remains uncertain whether it can overcome the novelty effect and how
users will actually engage in customization or appropriation.

##  3\. Background on Tweetorial Hooks

###  3.1. Science Communication and Tweetorials

Communicating science effectively and engagingly is key for the public to
understand and navigate our rapidly changing world. Historically, the realm of
science communication was largely confined within the boundaries of scholarly
papers (Bucchi and Trench, 2021). These papers, designed primarily for a niche
audience of fellow scientists and scholars, often assumed background knowledge
and overlooked connecting concepts to relatable daily experiences for
motivations and contexts (Williams et al., 2022). However, as technology
advances and social media spreads, instead of academic papers, there’s been a
marked shift in how people access scientific information online (Williams et
al., 2022). More and more people are starting to learn science from their
fragmented time and interesting ways as they stay on social media more, and
the contents there are more digestible and engaging to fit the pace.

Starting in late 2017, Tweetorials have become a popular science communication
medium for STEM experts to reach the general audience (Symplur, 2019). A 2021
study analyzed popular Tweetorials to identify the science communication
principles that made them effective (Gero et al., 2021). They found that
Tweetorials are generally short (threads of 8 - 15 tweets) and use informal,
narrative-driven, and personal language.

As shown in Figure 1, the key feature of a Tweetorial is the first tweet,
which is called the ‘hook’ to motivate the topic and draw readers in. The hook
has an intriguing question: “Are your friends more popular than you?” It
provides a relatable idea: “If you check out your Twitter friends, you might
find that on average they have more friends than you.” This introduces the
topic of the Friendship Paradox without jargon. And the details make it really
personal as something the reader may have encountered in their own life. Then,
the rest of the Tweetorial is the “body” that explains the concept, where the
hook can pique the reader’s interest in continuing to read about this real-
world experience.

[]A Tweetorial about the Friendship Paradox, a technical concept in social
networks, annotated for its structure. It comes with a hook and later body
sections. The hook is highlighted in green: ”Are your friends more popular
than you? If you check out your Twitter friends, you might find out that, on
average, they have more friends than you! Don’t worry—this is the Friendship
Paradox… (1/n)”

Figure 1. An example Tweetorial about the Friendship Paradox, a technical
concept in social networks, annotated for its structure and adherence to the
requirements (See section 3.2). It comes with a hook and later body sections.

However, studies on Tweetorial writing have shown that writing hooks are a key
challenge for STEM experts (Gero et al., 2021). They are trained to write
about their work in a formal tone for other experts, and it is difficult to go
against that training. Another difficulty is finding examples and layman’s
terms to motivate the topic for general audiences. Also, they feel
uncomfortable using subjective and informal language and avoid personal
details, even though 80% of the Tweetorials have them. In an exploratory study
using LLMs to support Tweetorial writers, one of the major use cases was
ideating concrete examples for the hook (Gero et al., 2022). This indicates
there is potential to use LLMs to help write Tweetorial hooks.

###  3.2. What Makes a Good Tweetorial Hook?

Previous work (Gero et al., 2021) has analyzed Tweetorial hooks through the
lens of science communication principles and they found that most hooks had
three important characteristics, which were subsequently formulated into
requirements for assessing an effective hook by (Long et al., 2023):

  * •

Requirement 1: Jargon-Free. The hook should contain no jargon so the general
audience can understand it easily, with the possible exception of naming the
topic to intrigue readers.

  * •

Requirement 2: Specific and Relatable Example(s). The hook should include an
analogy, a current event, a popular misconception, or a common experience.

  * •

Requirement 3: Sparks Curiosity. The hook should include an intriguing
question that sparks readers’ curiosity and drives readers to continue
reading.

Figure 2 is an example of Tweetorial hooks for Language Models, a computer
science topic, exhibiting these properties:

[]A Tweetorial hook about the Language Model, a technical concept in computer
science: ”My son relies on his Alexa to help with his math homework every
single night. While I am concerned about his learning, I am interested in how
Alexa understands what he is saying? Is it the same way that humans understand
language? What is the difference? A thread on how language models help with
this:”

Figure 2. An example Tweetorial hook about language models, a technical
concept in computer science, annotated for its adherence to the requirements.
It comes with a hook and later body sections. In this hook, the relatable
experience is talking to a smart device, like Alexa. The specific details
include parents’ concerns about the kids’ learning and Alexa helping with math
homework. The concerns about his son spark curiosity: “how Alexa understand
what he is saying?”

As shown, the relatable and specific content is an experience with the
technology. Although this doesn’t work for all topics, it tends to work well
for computer science topics because those concepts are usually the
technologies we all rely on and can be comparatively smoothly contextualized
into real-life use cases. Thus, the existing workflow listed below focuses on
supporting this because it is an effective technique well suited to an
important domain to explain to the public.

###  3.3. Existing AI Workflow for Tweetorial Hooks

A 2023 study (Long et al., 2023) presents a human-AI workflow that scaffolds
the process of writing a Tweetorial hook. As shown in Figure 3, the workflow
has seven steps and uses AI chains to help with the generations among them.
The user first inputs their topic (ex. language models), and the system
generates five everyday examples, and the user selects or edits one (ex.
“autocomplete feature in gmail”). Based on the topic and example, the system
generates five common experiences, and the user picks one (ex. “helps in
finding the next words and writing emails faster”). Next, the system generates
five user scenarios where people might have this experience (ex. “rushing to
email the professor about a homework extension”). The user can again select or
edit one. Next, the system generates a personal anecdote (ex. “I was sending a
last-minute email to my professor asking for extended deadlines…”). Then,
based on it, it continued to contextualize and generate a more specific
anecdote about the user scenario, which includes more details to make users
feel relatable and personal (ex. “I was sick… so I’m asking for an extension
on submitting my English Literature thesis..”). Lastly, the system
consolidates details from all the previous steps to generate example hooks,
allowing users to either use them directly or start from them in the final
step to write a hook.

This workflow was then validated by a user study (n=10) comparing the hook-
writing experience with and without the system (Long et al., 2023). The
workflow was shown to statistically and significantly reduce mental demand,
effort, and frustration while improving performance. However, this study was
conducted in a single 60-minute session, during which users wrote six hooks
(three with the system and three without). The novelty of the system might
have influenced their perception. Therefore, we conducted a study over three
weeks with ten usages to determine if effects and perceptions change after the
novelty wears off.

[]An illustration of the scaffolding workflow by making science more and more
contextualized: First, the user inputs their topic (ex. language models), and
the system generates 5 everyday examples and the user selects or edits one
(ex. “autocomplete feature in gmail”). Based on the topic and example, the
system generates 5 common experiences and the user picks one (ex. “helps in
finding the next words and writing emails faster”). Next, the system generates
5 user scenarios where people might have this experience (ex. “rushing to
email the professor about a homework extension”). The user can again select or
edit one. Next, the system generates a personal anecdote (ex. “I was sending a
last-minute email to my professor asking for extended deadlines…”). Then,
based on it, it continued to contextualize and generate a more specific
anecdote about the user scenario, which includes more details to make users
feel relatable and personal (ex. “I was sick… so I’m asking for an extension
on submitting my English Literature thesis..”). Lastly, the system
consolidates details from all the previous steps to generate example hooks,
allowing users to either use them directly or start from them in the final
step to write a hook.

Figure 3. An illustration of the existing AI workflow for crafting Tweetorial
hooks, following (Long et al., 2023). For each step inside the workflow, users
can regenerate, modify, or accept the workflow suggestions before going to the
next steps.

##  4\. System

[]A screenshot illustrates the first two steps of the study system, showcasing
user interactions. In this example, the user creates a Tweetorial about
”language models” using the system. They input their topic and review the
generated prompt (A). If satisfied, they press ”START” to initiate the
generation process. The tweet topic is automatically stored in Data Table (B).
Next, the user examines the Quick Hook Generation but decides not to proceed
due to a lack of relatability and interest in the ”personal vocab” focus.
Moving on to the 5 everyday examples (C), the user finds the autocorrecting
typos relevant and bookmarks one. This bookmarked generation is copied to the
Notes area (D) for future use. The user explores the ”Predictive Testing While
Writing Messages” option, finding it interesting but wanting to add their
creative writing experiences. They press the generated result, which is copied
to input bot [everyday_example] in Step 2 (E). The user makes edits to the
result and continues with the following steps.

Figure 4. A screenshot illustrates the first two steps of the study system,
showcasing user interactions. In this example, the user creates a Tweetorial
about ”language models” using the system. They input their topic and review
the generated prompt (A). If satisfied, they press ”START” to initiate the
generation process. The tweet topic is automatically stored in Data Table (B).
Next, the user examines the Quick Hook Generation but decides not to proceed
due to a lack of relatability and interest in the ”personal vocab” focus.
Moving on to the 5 everyday examples (C), the user finds the autocorrecting
typos relevant and bookmarks one. This bookmarked generation is copied to the
Notes area (D) for future use. The user explores the ”Predictive Testing While
Writing Messages” option, finding it interesting but wanting to add their
creative writing experiences. They press the generated result, which is copied
to input bot [everyday_example] in Step 2 (E). The user makes edits to the
result and continues with the following steps. Note: A full system walkthrough
can be found in the Appendix A

###  4.1. System Design

As shown in Figure 4, we designed a web application to understand users’ long-
term experience with AI workflows. Building on the existing workflow laid out
in section 3.3, we first used the latest GPT-4 model for timeliness and made
several simple user interface changes to enhance visual information design and
usability. We designed more salient sections and intuitive clickable buttons
for all system-generated results, ensuring users can effortlessly copy these
results. Also, we introduced ”Quick Hook Generation” so users see an example
hook based on their input topic in Step 1.

More importantly, to facilitate user customization throughout usages, we
display the system’s prompts with variable placeholders, thus empowering users
to understand, edit, and customize their system, and promoting user control
and autonomy. And, starting from the third week, users are offered chances to
feed their previously written hooks back into the system as training examples.
For data collection and management, we added a bookmark feature, storing
users’ favorite generation in a floating notepad before regeneration. We also
save and visualize all their step inputs in a data table. Users can reference
IDs, such as [tweet_topic] or [everyday_example], for prompt editing.

###  4.2. System Implementation and Deployment

We developed an interactive web application prototype utilizing HTML, Python,
JavaScript, Flask, AJAX, and the GPT-4 API. This prototype underwent testing
with three pilot users over 10 consecutive days. The evaluation covered
diverse research topics, bug reporting, UI responsiveness checks, and other
technical troubleshooting. Subsequently, we transitioned the system to a
cloud-hosted platform, deploying the code repository into 13 distinct web
applications. Also, all user interactions and system generations are auto-
logged, stored both on the backend and the user’s browser, and will be
automatically downloaded as a JSON file post-session. This ensures users can
revisit their work, enhancing their system comprehension. We believe these
minor adjustments retain the benefits of the original workflow while
augmenting the user experience for longitudinal interactions. This overall
architecture ensured that users had personalized access links always
accessible, while also enabling us to independently log every interaction on
the backend and later customize the system by adding their own hook examples.

##  5\. Study

To better understand novelty effects, familiarization and customization in
generative AI workflow, this study investigates the following research
questions (RQs):

  * •

RQ 1: Familiarization. How long is the familiarization phase? How do the
usefulness and performance of the system change before and after
familiarization?

  * •

RQ 2: Prompt Editing. How and why do users customize prompts?

  * •

RQ 3: Mental Model and Ownership. How do users’ mental models, sense of
ownership, and involvement with the system evolve over time?

###  5.1. Participants

We recruited PhD-level researchers in computer science (CS) who were
interested in communicating their research to the public on social media.
Because we wanted domain experts, we targeted senior PhD students, but anyone
with at least two years of research expertise was deemed expert enough. We
focused on CS as a discipline because the original workflow was shown to work
for CS topics. Other fields like microbiology or chemistry were not as easy to
motivate with an everyday experience. Other workflows could be designed for
those fields, but to test the long-term effects of a workflow, we focused on a
single domain where it has already been shown to work.

#### Recruitment

We advertised to PhD students in a CS department through department-wide
emails and Slack communications. The advert described that the study would
last three weeks, would require about six hours of time in total, and that
participants would be compensated each week - for a total of $150 if they
completed all three weeks. All recruitment materials and processes were
approved by the university IRB.

#### Demographics

As listed in Table LABEL:tab:demographics, we had 12 PhD-level students in CS
to complete the three-week study. Originally, we recruited 13 participants,
but P5 dropped out after day one due to personal circumstances. Out of the
final 12 participants, the average age was 27.5, with six men and six women.
On average, they had five years of research experience. Their expertise spans
various CS research

Table 1. Overview of participant demographics and research background Participant ID | Gender | Age | Research Area | PhD Year | Research Experience | SciComm Frequency
---|---|---|---|---|---|---
P1 | Male | 39 | Security and Reliability of Systems | 5th Year | 5 Years | Weekly
P2 | Male | 25 | Theoretical Computer Science | 4th Year | 6 Years | Weekly
P3 | Male | 26 | Software Systems | 2nd Year | 4 Years | Monthly
P4 | Female | 24 | Natural Language Processing | 1st Year | 4 Years | Weekly
P5 (Dropped) | - | - | - | - | - | -
P6 | Female | 28 | Speech Processing | 2nd Year | 5 Years | Weekly
P7 | Male | 26 | System for Machine Learning | 3rd Year | 5 Years | Daily
P8 | Female | 24 | Natural Language Processing | 2nd Year | 2 Years | Daily
P9 | Male | 23 | Natural Language Processing | 1st Year | 5 Years | Weekly
P10 | Female | 26 | Natural Language Processing | 4th Year | 7 Years | Daily
P11 | Female | 34 | Human-Computer Interaction | N/A | 3 Years | Weekly
P12 | Male | 24 | Virtual Reality | 1st Year | 2 Years | Monthly
P13 | Female | 31 | Machine Learning Theory | 4th Year | 10 Years | Daily

areas, including theory, systems, security, machine learning, natural language
processing, and human-computer interaction. They have varying levels of
experience with large language models: six out of 12 users use it weekly, four
use it monthly, one uses it daily, and one has never used it. All users had
frequent science communication experience, mostly at conferences. Only five
participants mentioned that they tweeted about scientific topics or papers.

###  5.2. Procedure

As shown in Figure 5, all 12 participants wrote 10 hooks in 10 separate
sessions over the course of three weeks. Each session lasted around 20
minutes, followed by a 10-minute survey. In the first session, users are
introduced to the system and study and write a hook with the system with the
study administrator present to observe the process and answer questions about
the task, the system, and the study. In the remaining nine sessions, the
following pattern is repeated three times: two solo system usages (without the
administrator present), then a system usage with the administrator present to
observe and conduct a 30-minute semi-structured interview.

[]An illustration of the Timeline of study activities over three weeks. In the
first session, users are introduced to the system and study and write a hook
with the system with the study administrator present to observe the process
and answer questions about the task, the system, and the study. In the
remaining nine sessions, the following pattern is repeated three times: two
solo system usages (without the administrator present), then a system usage
with the administrator present to observe and conduct a semi-structured
interview.

Figure 5. Timeline of study activities over three weeks. Each week, users need
to use the system three times to write Tweetorial hooks, followed by a semi-
structured interview conducted at the week’s end.

###  5.3. Data Collection and Analysis

#### System Log

We logged all 120 system usage sessions to record all interaction details,
including the user’s inputs and prompts for each step, the system’s generation
results, the system outputs they bookmarked and proceeded with, and the
timestamps associated with each activity. During each system usage (either
personal use or with the study administrator), we logged every interaction on
the Python server backend in a separate .log file. After the user concluded
the usage session, a JSON file with the same information was automatically
downloaded to the user’s laptop, allowing users to review their experiences if
needed and ensuring transparency in data saving. Then, they need to submit the
file for the session survey to verify their usage.

To analyze the log data, the first author used Python to analyze the user’s
session interactions, including duration spent on the system and each step, a
list of steps taken, skipped, or revisited by users, and the data and prompts
utilized for each step’s generation. Then, to understand how users make prompt
edits, the first author used a collaborative affinity diagramming tool to
group users’ new prompts into subgroups and groups based on their editing
objectives with an inductive approach. The final (sub)groups and their
examples were iteratively refined and discussed, with feedback sought from the
third author.

#### Survey

We received 118 surveys for the 120 system usage sessions111 Each user was
supposed to submit 10 surveys, resulting in a total of 120 surveys. However,
P10 and P13 missed their survey submission for their first usage sessions. The
study administrator later followed up with them about their experiences during
week one interviews, and both users shared they didn’t notice any significant
difference between their first and second usage. to understand users’ task
experience, perceptions of the system’s usefulness, and performance. After
each system usage session, users were asked to complete a Google Form survey.
After submitting their participant ID and their session JSON files, users
needed to fill out three sections: Section A focused on the hook writing task
experience alone and included questions adapted from the NASA Task Load Index
(NASA TLX) (Hart and Staveland, 1988). Section B primarily focused on their
system experience, probing participants’ perceptions regarding the system’s
usefulness, mental model, and expectations for future uses. Lastly, Section C
had users self-assess the quality and effectiveness of their hook, followed up
with a question on the level of ownership they felt over it.

To analyze the survey data, the first author first conducted exploratory data
analysis using Python to see the relationship between the variables based on
ground theory (Strauss and Corbin, 1994) to uncover relationships among
session experiences, system performance, user perceptions, and end output
evaluations over time and novelty. Then, the first author conducted
statistical testing, including t-tests and chi-square tests, to test
statistical significance.

#### Interview

We conducted 36 interviews with users to understand their degree of
familiarity with the system and other experiences and thoughts. Before each
interview, users first engaged in a system usage session with the study
administrator, during which they were encouraged to think aloud, express their
feelings, explain their rationales behind specific choices, and detail their
overall experience with the system. Following this, we conducted semi-
structured interviews to understand their experiences from the past week,
covering any noticeable shifts in their experiences, as well as their
expectations for the upcoming week regarding workflow and experiences. We
audio-recorded and transcribed the interview sessions using Zoom’s auto-
transcription feature after receiving the user’s consent each time.

To analyze the interview data, the first author performed qualitative coding
on the interview transcripts, which were then verified by the second and third
authors. The qualitative coding follows the inductive thematic approach
(Fereday and Muir-Cochrane, 2006) to discern patterns and shifts in user
experiences, building an affinity diagram based on the codes. The first author
then iterated on the codes and (sub)themes to answer research questions,
reporting to the other two authors more than five times for refinement.

##  6\. Results

Table 2. Number of sessions until the familiarization phase concludes. Participant ID | Familiarization concluded by # sessions
---|---
P1 | 5
P2 | Didn’t conclude familiarization
P3 | 3
P4 | 6
P6 | 4
P7 | 4
P8 | 4
P9 | 4
P10 | 5
P11 | 4
P12 | 5
P13 | 3
Average | 4.27
Median | 4

#### Initial User Experience

Based on the initial survey responses and insights gathered from week one
interviews, all 12 users had positive experiences with the system during their
first system usage session. Their initial interactions with the system were
described as enjoyable and relaxing (P8), expressive (P1, P9, P11), and useful
for traditionally challenging tasks (P2, P3, P4, P8, P12, P13). Notably, the
ratings for the system were already high: the average rating for the writing
task experience was 5.72 out of 7, while the perceived system usefulness
scored 5.82 out of 7. These comparatively high starting ratings aligned well
with findings from earlier research on workflow and the novelty effect, as
users might find such automated systems new and ”magical” only in the early
phase. Thus, the following section addresses the following questions: Were
these positive reactions primarily driven by a novelty effect? And more
importantly, would users’ perceptions of the system’s usefulness persist over
time?

## RQ 1: Familiarization

How long is the familiarization phase (RQ 1.1)? How do the usefulness and
performance of the system change before and after familiarization (RQ 1.2 and
RQ 1.3)?

### RQ 1.1 How long is the familiarization phase?

Summary: On average, the familiarization ends after 4.27 sessions. Before
familiarization, users were exploring to gain a better understanding of the
system. After familiarization, users understood the workflow, were able to
anticipate the workflow outputs, and finalized their own versions of the
workflow. These changes influenced users’ perceptions of the system’s
usefulness and performance.

To understand when users became familiar with the system, we asked them to
reflect on their experience and the degree to which they are familiar with the
system during our week two and week three interviews (after sessions 7 and
10). During the two interviews, we first asked, ”Do you feel you fully
understand the workflow now?” Then, if the user shared that they had already
mastered the system, we followed up with the question, “At which session do
you think you fully understood the systems?” On average, users identified
their familiarization phase ending at 4.27 sessions (median: 4 sessions). This
equates to 1.42 hours of system usage, and it was achieved in the first half
of week 2 of the study (Table 2 shows the session number for each user needed
to end familiarization).

We asked the users to characterize their experiences and interactions before
familiarization. All 12 users shared that they were primarily exploring and
attempting to understand the system. Four users (P1, P3, P7, and P11)
described that they were primarily ”going with the workflow” rather than
trying to edit or critique it. Similarly, three users (P4, P9, and P13) mainly
chose from the options provided with little editing. P8 expressed it as
follows:

> > ”I feel like [the familiarization] ends early this week, [so after] the
> fourth usage. I just remembered what each step is and… understood it in the
> way, not from the definition of the step, but also from the generation
> results of the step. So that kind of helped me to climb through the learning
> curve…”
>>

>> — P8, Week 2 Interview

After familiarization, users expressed three key changes in their experiences.
First, users mentioned they understood each step more concretely and how the
steps build towards a final hook. Second, users were able to anticipate system
generations. They could more quickly decide which steps to skip and which
steps to spend more time on based on their importance to the outcome and the
abilities of the LLM. Third, 10 users felt they ”finalized” their own version
of the workflow. They identified their preferred ways of using the system and
the prompt edits and strategies that work best for their respective topics. P3
expressed it as follows:

> > “[Now I] have my own fixed routine to generate hook… For me, the workflow
> starts to converge… the output is satisfying for me, so I kinda stop
> exploring a new way to improve the result. I have a fixed workflow… so my
> workflow stops changing.”
>>

>> — P3, Week 3 Interview

One of the 12 users, P2, did not reach familiarization within the ten sessions
of the study. When reflecting on why the system didn’t work, they said that
their topics were in an area of theoretical computer science, which lacked
obvious everyday use cases that the system was designed to support. Then, they
thought analogies were a more natural fit for motivating his theoretical
topics. Thus, they tried adapting the workflows to generate analogies instead
of relying on the use case mechanism and thought that was promising. We return
to this challenge in the Discussion section 7.2.

[]A bar chat showing system usefulness changes before and after
familiarization. Many bars are inside the graphs for Note Section, Prompt
Editing, Step1, Step2, Step3, Step4, Step5, Step6, Step7, and Overall System
Usefulness. Then, the average scores across 12 Users for Pre- and Post-
Familiarization show only Note Section, Prompt Editing, and Overall System
Usefulness, which are statistically significant.”

Figure 6. System Usefulness (average scores across 12 users before and after
familiarization, ** denotes statistical significance at the p-value ¡ 0.005
level, * denotes statistical significance at the p-value ¡ 0.05 level)

### RQ 1.2 System usefulness before and after familiarization222Based on each
user’s familiarization phase cutoffs, we categorized 120 user sessions into
two phases: ’Before Familiarization’ with 57 sessions and ’After
Familiarization’ with 63 sessions for our log analysis and survey results. We
put all ten sessions of P2 under the ‘Before Familiarization’ tag, as P2
indicated that they did not reach familiarization within the entire study.

Summary: On average, the overall system usefulness improved by 12.1% after
familiarization. The only features that show an increase in usefulness are the
prompt editing feature and the notes & bookmark feature, which serves as a
support for prompt editing. In contrast, the usefulness of all the individual
steps within the workflow (Step 1 to 7) remained unchanged.

Based on the session survey results shown in Figure 6, users reported a 12.1%
(0.85/7) improvement in overall system usefulness before and after
familiarization (5.47/7 to 6.32/7, p-value ¡ 0.005). This indicates that there
is no novelty effect — over time, workflow utility increases rather than
decreases.

When analyzing where the benefits come from, we analyzed the survey on the
usefulness of specific system features. On average, there is no change in the
perceived usefulness of the individual system steps (Step 1, Step 2,… , Step
7). By week 3, most users still felt every step was “useful and meaningful
(P1),” mostly by providing an easy ”direction to start (P6)”. At the
conclusion of the study, P12 continues to find all the steps and the system
useful:

> > ”Even in Steps 1, 2, 3, I’m finding that I need to regenerate a lot to get
> what I want. It’s still much easier than coming up with everything on my
> own. […] I don’t think there’s a single step that I don’t find useful. Even
> Step 7 is useful because it gives me like a word counter for tweets.”
>>

>> — P12, Week 3 Interview

Instead, the system usefulness data showed only two features with
statistically higher perceived usefulness: prompt editing with an 11.4%
improvement (5.07/7 to 5.87/7, p-value ¡ 0.005) and notes & bookmarks with a
12.3% improvement (3.44/7 to 4.3/7, p-value ¡ 0.05). Both of these features
are essential for customizing the workflow. Through prompt editing, users can
tailor the system to their style and topic. The bookmarks feature allows
people to ”heart” their favorite generation, enabling them to explore multiple
generations before choosing the best one. As users generate customized prompts
and generate more, bookmarks become increasingly important. Thus, it seems
that two customization-related features are mostly responsible for the
increase in overall system usefulness. For example, P13 noted this association
in their week three interview:

> > ”I now better know how to deal with the prompt [editing] and the [bookmark
> & save feature] box on the left side… It was useful… After learning how I
> can edit the prompt, yes, [the system gets] more useful.” — P13, Week 3
> Interview

### RQ 1.3 Performance before and after familiarization22footnotemark: 2

[]NASA Task Load Index (average scores across 12 users before and after
familiarization, ** denotes statistical significance at the p-value ¡ 0.005
level)

Figure 7. NASA Task Load Index (average scores across 12 users before and
after familiarization, ** denotes statistical significance at the p-value ¡
0.005 level)

[]Hook evaluation based on three requirements (average scores across 12 users
before and after familiarization, ** denotes statistical significance at the
p-value ¡ 0.005 level)

Figure 8. Hook evaluation based on three requirements (average scores across
12 users before and after familiarization, ** denotes statistical significance
at the p-value ¡ 0.005 level)

Summary: Users reported a 14.9% improvement in their task performance and a
7.1% improvement in time efficiency. From their self-assessments, users
believe they produced better hooks, with more specific and relatable examples
(Requirement 2) and better at sparking curiosity (Requirement 3).

From the NASA TLX scores reported from the surveys, users noticed a 14.9%
improvement in performance (5.15/7 to 6.19, p-value ¡ 0.005) and a 7.1%
improvement in time efficiency for completing the task (2.89/7 to 2.32/7,
p-value ¡ 0.005) after familiarization. However, as shown in Figure 8, the
task remained similarly mentally demanding, effort-demanding, and frustration-
inducing over the two stages, with changes of less than 0.2. Paired t-tests
further confirmed that these marginal differences were not statistically
significant. This is consistent with RQ 1.1 that people explore less after
familiarization, thus saving them time but not necessarily further reducing
mental load. For example, P6 shared in the week three that they noticed an
improvement in temporal demand:

> > ”I used the system much faster this week… For the first week, I tried to
> understand what [each step] means… for the last week, I already knew what it
> means… so I chose the [generation result] option quicker than at the
> beginning.”
>>

>> — P6, Week 3 Interview

Based on users’ self-assessments of their final hooks, they first reported
producing better hooks after the familiarization phase. In Figure 8, users
shared a 9% improvement for Requirement 2: Specific and Relatable Example(s)
(from 5.80/7 to 6.43/7, p-value ¡ 0.005) and an 8% improvement for Requirement
3: Sparks Curiosity (from 5.69/7 to 6.25/7, p-value ¡ 0.005). Also, users
noted a 17.3% reduction in the frequency of hooks containing factual
inaccuracies, biases, or potential risks. Before familiarization, 68.4% of
hooks were labeled as accurate and unbiased (39 out of the 57 hooks). After
familiarization, this increased to 85.7% (54 out of 63 hooks).

This again argues that familiarization with the AI workflow was not just a
novelty effect but provided genuine value to users.

## RQ 2: Prompt Editing

### How and why do users customize prompts?

Summary: Users customized 12.4% of the available prompts (89 of 720). These
edits addressed three main problems with the AI workflow: going off-topic,
undesired narrative and style, and inefficient workflow. In week 3, we added
users’ own hooks into the prompt as training examples, but there was no
significant increase in the utility of the system. Nine users favored the new
feature, while three reported no benefits.

### Prompt Editing Themes & Strategies

Across 120 sessions, all users made a collective total of 89 prompt edits. One
author coded the 89 edited prompts using the inductive thematic analysis and
reported to another author for feedback and iterations on codes. Table
LABEL:tab:strategies shows the six prompt editing strategies that the users
used to solve three workflow problems:

#### Problem #1: Off-Topic

The system occasionally failed to maintain a clear topic definition or
emphasis over generations. To rectify this, users either reiterated the topic
for relevance or added definitions and more details:

1a. Reiterate Topic (22 edits): When the system veered off the technical topic
during a lengthy generation sequence, users reinforced the theme. They either
reiterated their Tweetorial topic or inserted the reference ID, like
[tweet_topic], into the prompt to keep generations relevant.

1a. Clarify Definition (5 edits): Occasionally, users perceived the system as
lacking depth or contemporary knowledge about specific topics. To address
this, they either embedded a direct definition or sourced definitions from
online platforms like Wikipedia.

#### Problem #2: Undesired Narrative and Style

The system’s outputs sometimes didn’t align with users’ desired narrative or
stylistic preferences. To personalize and guide the generations, users either
add preferred examples as training to guide generations or provide longer
instructions:

2a. Guide Narrative (25 edits): The system sometimes missed users’ preferred
narratives or specific nuances they wanted to highlight. Users then reinforced
their desired narrative direction by adding specifics or personal experiences.

2b. Guide Style (18 edits): To better mirror their unique writing styles,
users made adjustments. Whether they wanted casual language, content tailored
for a teenage audience, or the inclusion of emojis, users fine-tuned the
prompt to achieve the desired output.

#### Problem #3: Inefficient Workflow

The system sometimes failed to generate users’ anticipated outcomes
efficiently after familiarization. To utilize preferred, shorter version of
the workflow or explore more options at once, they either sidestepped some
middle steps or requested the system to diversify generations:

3a. Increase Variations (14 edits): Users observed that the original system
prompt often generated a restricted number of examples. To expand their
options, they modified the prompt to produce between 10 and 20 example outputs
for broader brainstorming.

3b. Skip Steps (5 edits): When users identified valuable ideas early on, they
sometimes wished to advance without adhering to every step. To streamline
their process, they altered the system’s reference ID to reflect the results
of the previous step.

Table 3. Overview of the 6 prompt editing strategies and their examples. Problems |  Editing Strategies |  Example
(Note: Highlighted text indicates user additions or deletions to the original
prompt.)
---|---|---
Problem 1:
Off-Topic |  Reiterate Topic (22 edits) |  Please list 5 user scenarios someone encounters with [common_experience] and [tweet_topic] and augmented reality. Each scenario should be under 15 words, from a first-person perspective, related to a general audience, and not require technical knowledge. (P12, Step 3: Generate User Scenarios)
Clarify Definition (5 edits) |  Please list 5 everyday examples of [tweet_topic]. [tweet topic] is defined as using computational methods, often natural language processing, to analyze social science such as literature. Each example should be under 10 words, related to a general audience, and not require technical knowledge. (P4, Step 1: Generate Everyday Examples)
Problem 2:
Undesired
Narrative
and Style |  Guide Narrative (25 edits) |  Here is a personal anecdote about the topic, [tweet_topic], for the experience of [common_experience]: [specific_anecdote]. Based on this anecdote, please write 3 versions of the opening tweet to introduce the concept of [tweet_topic]. The tweet should have no jargon. It should contain a specific and relatable example. It should have a driving question to spark curiosity and make readers want to read more. It should emphasize giving everyone an equal number of candies in broad categories, like chocolates, caramels, and that some candies can be in multiple categories. (P2, Step 6: Generate Example Hooks)
Guide Style (18 edits) |  Please list 5 common experiences with using [everyday_example]. Each experience should be under 10 words, related to a teenage audience, and not require technical knowledge. (P4, Step 2: Generate Common Experiences)
Problem 3:
Inefficient
Workflow |  Increase Variation (14 edits) |  Please list 5 20 user scenarios someone encounters with [common_experience]. Each scenario should be under 15 words, from a first-person perspective, related to a general audience, and not require technical knowledge. (P9, Step 3: Generate User Scenarios)
Skip Step (5 edits) |  Please write 3 short personal anecdotes about the topic, [tweet_topic], for the user scenario of [user_scenario]. Each anecdote should be under 30 words, from a first-person perspective, related to a general audience, and not require technical knowledge. (*Note: The user skipped three steps from Step #1 to #4, thus using [tweet_topic] for this step.) (P7, Step 4: Generate Personal Anecdotes)

### Augmenting Prompts with Users’ Hooks

Starting from the 8th session (day one of week three), we introduced a new
feature: the Learn-My-Style option inside ‘Step 6: Generate Example Hooks’.
This step allows the user to see two versions of the prompt: one is the
original generate-example-hook prompt, and the other combines the original
prompt with their own hook examples taken from their previous seven sessions.
Users have the option to select one of the prompts to proceed, and they can
also edit their own hook examples within the prompt if they wish (See the
updated interface in Appendix B).

In theory, this Learn-My-Style prompt could enhance the system’s generations
by better adhering to users’ writing styles. It also has the potential to
provide more context about the topics or expertise areas users usually select
from.

However, the session surveys revealed no statistically significant difference
in users’ perceived system usefulness and performance before and after we
implemented the new feature. Specifically, between week two (sessions 5, 6, 7)
and week three (sessions 8, 9, 10), there was no statistically significant
difference in perceived usefulness for the overall system (6.03/7 - 6.06/7,
p-value = 0.87) and for Step 6 (5.83/7 - 6.08/7, p-value = 0.47).
Additionally, there was no statistically significant difference in performance
(5.91/7 - 6.03/7, p-value = 0.40).

During the final interview, nine users expressed positive opinions on this
feature, while three others showed either indifference or did not like the
addition. All nine users who favored the new prompt compared the generated
hooks with and without adding their own hook training data to understand how
it works. Among them, four noted that their unique style was reflected in the
generated hooks. For instance, P4 stated their preference for humor and darker
themes, commenting that the hooks generated by the new prompt resonated well
with their taste. P7 found the new prompt successfully captured their habit of
ending tweets with a question and avoiding direct mention of topic names. P10
mentioned they like using personal pronouns, like ”you,” ”she,” ”my,” etc., to
make hooks more relatable, and they saw this trait also from using the Learn-
My-Style prompt.

Among the users less enthusiastic about this feature, P6 expressed hesitancy,
feeling they needed more time for familiarization. P13 preferred using the
original prompt, finding its simplicity and current results satisfactory. P9
offered a unique perspective: they saw the system as a tool to explore styles
beyond their own. They believed the original system could offer a broader
creative spectrum than their own style while incorporating their previous
generations inside the prompt might constrain creativity and limit variety:

> > ”I [already] had some biases in selecting hooks… [When I used] the
> original hook system, it might have just like more creative [and] expressive
> results. But then, I also tried to use the Learn-My-Style thing, and I was
> like, yeah, the creativity does get hindered. ”
>>

>> — P9, Week 3 Interview

In conclusion, prompt editing becomes more useful over time. After users learn
how and what they can add to prompts to enhance them, they report finding it
significantly more valuable. Users were divided on whether adding their hooks
into the prompt as an example was beneficial. Even for those who found it
useful, it addressed surface-level aspects like style rather than deeper
structural issues like constraining the generations or skipping steps. Thus,
AI generation systems would likely benefit users in the long term by allowing
end users to edit prompts, enabling them to make structural changes to the
system.

## RQ 3: Mental Models and Ownership

### How have users’ mental models, ownership, and involvement changed over
time?

Summary: Users’ mental models remain fairly consistent over time. However,
users exhibit different levels of involvement and ownership throughout
longitudinal interactions. Eight of 12 users thought that more involvement
with the system led to increased feelings of ownership.

### Mental Models

We analyzed the survey question on how users would describe the system
throughout the 120 sessions. They primarily viewed the system as a tool (90.7%
of the sessions, 107 times) and content generator (84.7% of the sessions, 100
times), with less frequent perceptions like co-pilot (44.1% of the sessions,
52 times), reference guide (38.1% of the sessions, 45 times), and ghostwriter
(28.8% of the sessions, 34 times).

Before and after familiarization, there is no significant change in the user’s
mental model of the system (See Table 4). After familiarization, there was a
small increase in references to the system as a ”content generator” and
”collaborator”, with increases of 11.70% and 11.45% respectively. Descriptions
of the system as a ”tool” decreased by 7.27%. However, these shifts were
subtle and not statistically significant. This indicates that users maintain a
consistent perception and mental model of the system before and after becoming
familiar with it. This suggests that when the novelty wears off, users don’t
usually process new mental models for such a customizable system.

Table 4. Users’ mental model of the system before and after familiarization (percentage of the session where users described the system by its role) Description of the system | Before fam. | After fam. | Change
---|---|---|---
Tool | 92.98% | 85.71% | -7.27%
Content Generator | 77.19% | 88.89% | 11.70%
Co-pilot | 40.35% | 46.03% | 5.68%
Reference Guide | 38.60% | 36.51% | -2.09%
Collaborator | 29.82% | 41.27% | 11.45%

### Ownership and Involvement

After every interaction with the system, we asked users their degree of
ownership over the hook333Survey question: Please rate the following statement
from 1 - 7 (Strongly Disagree - Strongly Agree): ”In this round, I feel a
strong sense of ownership over the final hook I wrote, reflecting my personal
style, effort, and creativity.”. On average, users perceive a 13.3% increase
in ownership over hooks after familiarization (p-value ¡ 0.005). Before
familiarization, users had an average ownership perception of 4.28/7, showing
their attitude towards ownership was neutral. After familiarization, their
ownership perception increased to 5.21/7, showing they were between somewhat
agree and agree that they felt strong ownership over their hooks. One reason
the user P11 gave was that, over time, they put more effort into using the
system and that their involvement contributed to increased feelings of
ownership — from simple efforts like more carefully reading and selecting LLM
outputs to making more edits to both the text and the prompts.

> > ”Now thinking back, I would say I feel my ownership is increased… One
> [reason] is that I actually read through every sentence and every word to
> make sure I feel good about it… So just by reading through all of these
> [system generations and…] choosing the one that I agree with make me feel
> like I own it… Over the 3 weeks, I started to rewrite more of the hooks, and
> that actually makes me feel even stronger with my ownership.”
>>

>> — P11, Week 3 Interview

To investigate the relationship between involvement and ownership, we looked
at our interview data. During our week three interviews, we asked the users,
”How did your involvement within the sessions change after familiarization?”
and ”Do you find your level of involvement influences your sense of
ownership?” From the interviews, there were four types of answers stemming
from different experiences with the system:

#### 1\. More involved, more ownership: ”My hard work pays off” (4 users)

P6, P10, P11, and P12 shared that they became more involved in the process
after familiarization. All of them mentioned that after gaining a better
understanding of the system, they felt confident and comfortable enough to
modify the prompt, thus dedicating more involvement and effort to the session.
Specifically, P6, P10, and P12 iterated on their prompt edits multiple times
to find a favorable output aligned with their interests. P11 and P12 focused
on reading every result closely to select the best option. Additionally, P10,
P11, and P12 frequently edited the outputs and conducted fact-checks through
web searches to verify the information’s accuracy. Thus, throughout this
process, they all shared an increased sense of connection and ownership to the
generated artifacts.

> > ”After making more edits and being more engaged in the interaction, I feel
> a stronger connection to the tone and effort.”
>>

>> — P10, Week 3 Interview

#### 2\. Less involved, but maintain ownership. ”I still lead the system” (2
users)

P7 and P8 found themselves less involved over time but still maintained high
ownership. They found the system reliable and capable enough to assist with
the task after observing its commendable performance over a span of three
weeks. Thus, they started to allow the system to take on more work and made
fewer but more concise edits. They shared that they still ”led the system”
(P8) and their guidance over the system still makes a huge impact on the final
artifact.

> > ”I feel like, even if I did no revision at all, I’d still feel a high
> ownership because I did all the selections [and] I did guide the system
> through… if I have specific revisions in the words or if I have a revision
> in the prompts, I may feel more ownership over it versus if I did nothing…
> but still it’s overall pretty high ownership generally.”
>>

>> — P8, Week 3 Interview

#### 3\. Less involved, less ownership. ”I am now just a reviewer” (4 users)

P1, P3, P4, and P9 found themselves less involved with a drop in perceived
ownership. Similar to P7 and P8, these four users also found the system
reliable and capable. Moreover, P3 and P4 noted that they trusted the system
more over time. Consequently, they started only providing high-level feedback
to the system and reviewing its generation. For example, P3 expressed that,
over usage, they increasingly allowed the system to handle more parts of the
tasks. P3 mentioned they now acted more as reviewers:

> > ”I know the system’s capability better… I can trust its ability to
> generate some everyday examples and a final hook… I have become more reliant
> on the system… I become more like a human reviewer to help guide the GPT to
> generate a better hook… [so now] I think my ownership dropped a lot…”
>>

>> — P3, Week 3 Interview

#### 4\. No changes in involvement/ownership over time. ”Topic dictates
effort” (2 users)

P2 and P13 didn’t report changes in their involvement or perception of
ownership. They both found their experience was predominantly influenced by
how well the LLM was able to support their chosen topic. P2 mentioned that the
involvement increased only when they found the system failed to support their
hook topics. P13 mentioned that they cannot fully trust the LLMs and have to
intervene along the way. But they both agreed that the more edits they made,
the more ownership they felt, but it wasn’t a function of time. P13 compared
their system experiences to mentoring undergraduate interns:

> > ”It’s like an undergrad intern… I had to teach [the system about my
> topic]. I couldn’t fully trust GPT… [It’s like] working with my undergrad,
> and I tell him what to code […but not] how to code… Then he comes back to
> me, and we go over [the code to see…] if there’s anything that needs more
> rigorous thinking… what’s wrong… and how should we deal with it.”  — P13,
> Week 3 Interview

In conclusion, although average feelings of ownership increased, there were
nuanced ways that users experienced ownership after multiple experiences with
the system. There does seem to be a relationship between effort and ownership.
Based on the current state of LLM technology, many users find it necessary to
edit text and prompts at every stage, but it is also a function of the topic -
some topics are easier for LLMs to get right.

##  7\. Discussion

###  7.1. Does the novelty wear off?

This paper presented a three-week study of an LLM-based workflow to support
science writing. Though generative AI has impressive abilities at first
glance, it’s unclear whether that is due to the initial impression of a new
technology or whether the workflow has genuine utility of the system. We found
that after users had used the system for an average of 4.27 sessions, they had
become familiarized. After familiarization, they no longer explored its novel
capabilities but focused solely on performing the task. They understood the
workflow better, had a better ability to anticipate system generation, and
performed better in writing hooks. Surprisingly, we found the average system
utility actually increased by 12.1% after familiarization (5.47/7 to 6.32/7,
p-value ¡ 0.005), indicating that not only that the novelty doesn’t wear off,
but the workflow even becomes more useful. Why doesn’t the novelty wear off?

#### Reason #1: The workflow is supporting a task with cognitively challenging
steps.

One possible reason the workflow maintains its usefulness over time is that
the underlying task targeted by the AI workflow has steps with high cognitive
load. Thus, even with practice and continued use of the system, the support
workflow still provides genuine assistance in reducing cognitive load and
helps users achieve better outcomes with less effort.

According to the cognitive process theory of writing (Flower and Hayes, 1981),
two mentally demanding steps in the writing process are: ideation and
translation (drafting complex thoughts into a linear narrative). For the AI
workflow we studied in this paper, Steps 1, 2, and 3 aid in ideation—coming up
with a specific and relatable use case that will be the vehicle for hooking
non-technical audiences. These three steps help overcome fixation in human
ideation. Many workflows, especially generative AI-based ones, have shown that
brainstorming support increases the number, diversity, and quality of ideas
(Wang et al., 2023; Liu et al., 2022; Petridis et al., 2023a; Zhang et al.,
2023; Singh et al., 2022).

After users have ideated a use case, Steps 4, 5, 6, and 7 help with
translating it into prose that is jargon-free, relatable, and sparks
curiosity. Translation is difficult because it forces users to synthesize many
pieces of information into one coherent, linear text. Often, the shorter the
text, the harder it is to write, but generative AI can help produce examples
of linear prose that satisfy some or all of these design goals. Thus, it can
be an excellent starting point or simply motivation to overcome the mental
activation energy needed to start drafting. Even when LLMs make mistakes,
producing example text to work from helps users get over the cognitive barrier
of translation. For our workflow, users consistently find all steps maintain
their high usefulness over sessions, as the hook writing task is still
challenging and necessitates support for ideation and translation.

We expect that AI workflows that prove to be useful in a single session will
likely maintain their usefulness over time if they assist with fundamentally
challenging tasks. This includes cognitive challenges like brainstorming,
sensemaking challenges like clustering, socially challenging tasks like
imagining a reader’s perspectives, and time-consuming tasks like formatting
text. Moreover, the utility of AI workflows can increase over time as users
become more efficient at using it, better at applying it to their domain, and
can easily customize it to their needs.

#### Reason #2: The workflow breaks down a hard problem into steps.

Another possible reason the workflow maintains its usefulness over time is
that the scaffolding and chaining techniques into steps genuinely enhance
collaboration between users and AI. Generative AI, by itself, makes a lot of
mistakes. In their first few generations, these models can produce some
seemingly miraculous results. However, over time, they may still produce
staggeringly poor results. For example, the system generates a “quick hook” in
Step 1 so users can directly take them as their final hook and conclude the
writing session if they want. However, users rarely take it: out of the 120
sessions, users only used that first generated hook 13 times. Most often, it
was either factually incorrect or poorly constructed. Thus, many users
expressed willingness to invest more effort in utilizing the entire workflow,
deeming it worthwhile to achieve a better answer. This is because workflows,
which utilize scaffolding and AI chains, break down tasks into fundamental
pieces and allow humans to check AI at every step, leading to better outcomes.
Task decomposition has been shown to enable effective human-human
collaboration (Little et al., 2010; Bernstein et al., 2010; Retelny et al.,
2014), so it is fitting that task decomposition enables better human-AI
collaboration, even in the long term. In addition to the error prevention and
recovery provided by the scaffolding, generative AI workflows can potentially
provide more surprising results at each step, piquing users’ interest.
Moreover, with more steps, there are additional opportunities to customize the
system, offering greater control over the process.

###  7.2. Customization and Appropriation in Generative AI Workflow

In this longitudinal study, we saw a significant increase in system utility
due to a simple customization feature of exposing the prompt box so users
could edit it. We believe this is a simple and powerful technical that all AI
workflows should explore to help users mold the system to their needs.
Moreover, AI poses unique opportunities for end-user customization within a
system. Traditional workflows are often brittle, constraining, and difficult
for end users to edit. However, with natural language editing, users can
customize workflows in their own words.

Beyond customization, there is the potential for users to appropriate a
workflow for tasks it was not originally designed for. In the seminal paper on
Designing for Appropriation (Dix, 2007), Alan Dix summarizes 7 guidelines for
building systems that “plan for the expected”. In the age of generative AI,
many of these principles have become trivial to apply to systems. Systems can
“provide visibility” by exposing prompts; they can “support not control” by
allowing editing. The very nature of prompting with natural language “allows
for interpretation (Don’t make everything in the system or product have a
fixed meaning, but include elements where users can add their own meanings).”
If the system is logging user’s prompt edits, it becomes possible to “learn
from users’ appropriations” at an astonishing speed. At an extreme, it
presents an opportunity for a continuous co-design process by sharing
successful appropriations with the user community.

In our interviews, users saw the potential to appropriate the system. For
example, P2 initially encountered difficulties when they found that their
theoretical topics didn’t align well with the existing workflow, which
primarily focused on everyday use cases. Thus, starting from session 4, they
began to appropriate the system by using analogies rather than use cases to
make theoretical topics relatable. After some prompt engineering, the metaphor
strategy started to work. This could be a valuable approach for other
theoretical topics that lack relatable use cases. Moreover, towards the end of
the study, seven users shared expectations to appropriate the workflow to
support other science communication tasks, like writing and revising their own
research papers (P1, P8, P9, P10, P12, P13), drafting an elevator pitch about
their research (P10), or making learning materials for themselves (P11). These
are all excellent directions for this workflow to go in, and we are excited to
whether users can take it there on their own.

##  8\. Limitations and Future Work

Our study focused on a specific AI workflow utilizing a particular generative
model. While we believe that this choice provides insights that can be
generalized to other workflows, as the use of AI chains and scaffolding to
solve cognitive challenges is common across workflows, it is essential to
explore how utility and customization change a broader range of AI workflows.
For example, the familiarization period for our system occurs after
approximately 4.27 sessions, equivalent to around 1.42 hours spread across a
week. However, the adequacy of this duration remains open to interpretation
and warrants contextualization relative to different tasks, workflow,
underlying models, and user communities. Moreover, the current landscape of AI
is evolving rapidly. For instance, many prompt edits arise from the chosen
topic’s compatibility with the knowledge base and contextual understanding
capabilities of the models. Consequently, as these models continue to advance,
the results we obtained may undergo significant changes, necessitating
different levels of human intervention and workflow customization.

Our study encompassed three weeks (ten usage sessions) and 11 of 12 users
reported they already reached familiarization as they fully understood the
technology. However, it might be possible for after there are more nuances
beyond a month to be observed. Also, despite our efforts to understand users’
experiences through observed system usage logs, session surveys, and weekly
semi-structured interviews, capturing the full user journey is challenging,
and some nuances might have been missed.

Our participant pool was limited to Computer Science PhD students, a
demographic proficient with technology. Even though they possess differing
levels of experience with using generative AI – ranging from daily to never –
their relatively high technology/AI literacy and openness to accepting new
technology might differ from that of the general population. This could still
introduce potential bias. Additionally, users were not required to post their
hooks publicly, which could have influenced their interaction with the system.
If the stakes were higher, their critique of the system and session
experiences might have been more stringent, or they might have invested more
effort. Future studies might consider diversifying the user base or altering
the stakes to observe any shifts in results.

##  9\. Broader Impact

Our study presents various exciting impacts for both the general audience and
academia: First, our study focuses on the task of utilizing emerging
technologies to empower scientific communication. Science communication is an
important public affair that bridges the broader audience to understand
technological advancements and combat misinformation. Second, our study is
among the first to examine the novelty effects and long-term utility of
generative AI-based workflows. Thus, the study results shed light on insights
for the future development of AI models and workflows based on them.
Additionally, the study contributes a solid methodological foundation for
future empirical studies, particularly long-term investigations, on generative
AI and its workflows. Third, many of the paper insights align well with
existing design theories, such as technology domestication theory and
technology acceptance model, thereby offering new design guidelines for
designing longitudinal interactions in the era of generative AI and AI-powered
creativity support tools. Fourth, our paper highlights the idea of technology
appropriation, which allows and encourages users to customize and adapt
systems for new use cases. We initiate a new discussion about using generative
AI as design material to promote user-centric co-design for system
development.

However, we also acknowledge that our study primarily reports positive user
interaction with AI workflows over time, lacking emphasis on the potential
risks and harms associated. Our user group might have already encountered
these risks and harms in various contexts before the study. This familiarity
can breed trust and contribute to higher tolerance levels, potentially leading
to the oversight or disregard of associated harms observed in the study.

##  10\. Conclusion

To understand the novelty and long-term utility of AI workflows, we conducted
a three-week longitudinal study with 12 users to understand the customization
of generative AI tools for science communication. Our study revealed that the
familiarization phase lasts for 4.27 sessions, which equates to 1.42 hours or
one and a half weeks of usage. Before familiarization, users are mainly
exploring the capabilities of the workflow and which aspects they find useful.
After familiarization, the perceived utility of the system is rated 12.1%
higher than before, indicating that the perceived utility of AI is not just a
novelty effect. The increase in benefits mainly comes from end-users’ ability
to customize prompts, and thus appropriate the system to their own needs,
ranging from addressing system problems to adapting for new use cases. Though
the mental models remain over time, users begin to change their level of
involvement while collaborating with the AI workflow, thus leading to a shift
of their perception of ownership. This points to a future in which generative
AI systems and workflows can facilitate user customization and appropriation,
as well as a continuous co-design process between system developers and users.

## References

  * (1)
  * Arawjo et al. ([n. d.]) Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. [n. d.].  ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing.  https://doi.org/10.48550/arXiv.2309.09128
  * Baumeister and Leary (1995) Roy F Baumeister and Mark R Leary. 1995.  The need to belong: desire for interpersonal attachments as a fundamental human motivation.  _Psychological bulletin_ 117, 3 (1995), 497.
  * Belin and Prié (2012) Amaury Belin and Yannick Prié. 2012.  DIAM: Towards a model for describing appropriation processes through the evolution of digital artifacts. In _Proceedings of the designing interactive systems conference_. 645–654.
  * Benke et al. (2022) Ivo Benke, Maren Schneider, Xuanhui Liu, and Alexander Maedche. 2022.  TeamSpiritous - A Retrospective Emotional Competence Development System for Video-Meetings.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW2 (Nov 2022), 1–28.  https://doi.org/10.1145/3555117
  * Berker et al. (2005) Thomas Berker, Maren Hartmann, and Yves Punie. 2005.  _Domestication of media and technology_.  McGraw-Hill Education (UK).
  * Bernstein et al. (2010) Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010.  Soylent: a word processor with a crowd inside. In _Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology_ (New York, New York, USA) _(UIST ’10)_. Association for Computing Machinery, New York, NY, USA, 313–322.  https://doi.org/10.1145/1866029.1866078
  * Blaynee et al. (2016) Jessica Blaynee, David Kreps, Maria Kutar, and Marie Griffiths. 2016.  Collaborative HCI and UX: longitudinal diary studies as a means of uncovering barriers to digital adoption. In _Proceedings of the 30th International BCS Human Computer Interaction Conference 30_. 1–6.
  * Bommasani et al. ([n. d.]) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. [n. d.].  On the Opportunities and Risks of Foundation Models.  arXiv:2108.07258 [cs] http://arxiv.org/abs/2108.07258
  * Bucchi and Trench (2021) Massimiano Bucchi and Brian Trench. 2021.  _Routledge Handbook of Public Communication of Science and Technology_.  Routledge.  Google-Books-ID: Em4ZEAAAQBAJ.
  * Bursztyn et al. ([n. d.]) Victor Bursztyn, David Demeter, Doug Downey, and Larry Birnbaum. [n. d.].  Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models. In _Findings of the Association for Computational Linguistics: EMNLP 2022_ (Abu Dhabi, United Arab Emirates, 2022-12), Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 1676–1686.  https://doi.org/10.18653/v1/2022.findings-emnlp.121
  * Carroll et al. (2001) Jennie Carroll, Steve Howard, Frank Vetere, Jane Peck, and John Murphy. 2001.  Identity, Power And Fragmentation in Cyberspace: Technology Appropriation by Young People.  (01 2001).
  * Choi et al. ([n. d.]) DaEun Choi, Sumin Hong, Jeongeon Park, John Joon Young Chung, and Juho Kim. [n. d.].  CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AI.  arXiv:2312.11949 [cs] http://arxiv.org/abs/2312.11949
  * Davis et al. (2013) Nicholas Davis, Holger Winnemöller, Mira Dontcheva, and Ellen Yi-Luen Do. 2013.  Toward a cognitive theory of creativity support. In _Proceedings of the 9th ACM Conference on Creativity & Cognition_ (Sydney, Australia) _(C &C ’13)_. Association for Computing Machinery, New York, NY, USA, 13–22.  https://doi.org/10.1145/2466627.2466655
  * Deci and Ryan (2000) Edward L Deci and Richard M Ryan. 2000.  The” what” and” why” of goal pursuits: Human needs and the self-determination of behavior.  _Psychological inquiry_ 11, 4 (2000), 227–268.
  * Dix (2007) Alan Dix. 2007.  Designing for appropriation. In _Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI…but Not as We Know It - Volume 2_ (University of Lancaster, United Kingdom) _(BCS-HCI ’07)_. BCS Learning & Development Ltd., Swindon, GBR, 27–30.
  * Dourish ([n. d.]) Paul Dourish. [n. d.].  The Appropriation of Interactive Technologies: Some Lessons from Placeless Documents.  12, 4 ([n. d.]), 465–490.  https://doi.org/10.1023/A:1026149119426
  * Du et al. ([n. d.]) Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. [n. d.].  Improving Factuality and Reasoning in Language Models through Multiagent Debate.  arXiv:2305.14325 [cs] http://arxiv.org/abs/2305.14325
  * Eisenstein et al. ([n. d.]) Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. [n. d.].  Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model.  arXiv:2210.02498 [cs] http://arxiv.org/abs/2210.02498
  * Eloundou et al. (2023) Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023.  GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models.  arXiv:2303.10130 (Aug 2023).  http://arxiv.org/abs/2303.10130 arXiv:2303.10130 [cs, econ, q-fin].
  * Feng et al. ([n. d.]) K. J. Kevin Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy X. Zhang, and David W. McDonald. [n. d.].  Canvil: Designerly Adaptation for LLM-Powered User Experiences.  arXiv:2401.09051 [cs] http://arxiv.org/abs/2401.09051
  * Fereday and Muir-Cochrane (2006) Jennifer Fereday and Eimear Muir-Cochrane. 2006.  Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development.  _International journal of qualitative methods_ 5, 1 (2006), 80–92.
  * Flower and Hayes (1981) Linda Flower and John R. Hayes. 1981.  A Cognitive Process Theory of Writing.  _College Composition and Communication_ 32, 4 (1981), 365–387.  https://eric.ed.gov/?id=EJ256235
  * Gao et al. ([n. d.]) Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. [n. d.].  CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models.  arXiv:2304.07366 [cs] http://arxiv.org/abs/2304.07366
  * Gero et al. (2022) Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022.  Sparks: Inspiration for Science Writing Using Language Models. In _Designing Interactive Systems Conference_ (Virtual Event, Australia) _(DIS ’22)_. Association for Computing Machinery, New York, NY, USA, 1002–1019.  https://doi.org/10.1145/3532106.3533533
  * Gero et al. (2021) Katy Ilonka Gero, Vivian Liu, Sarah Huang, Jennifer Lee, and Lydia B. Chilton. 2021.  What Makes Tweetorials Tick: How Experts Communicate Complex Topics on Twitter.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW2, Article 422 (oct 2021), 26 pages.  https://doi.org/10.1145/3479566
  * Gero et al. (2023) Katy Ilonka Gero, Tao Long, and Lydia B Chilton. 2023.  Social Dynamics of AI Support in Creative Writing. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 245, 15 pages.  https://doi.org/10.1145/3544548.3580782
  * Greenberg and Buxton (2008) Saul Greenberg and Bill Buxton. 2008.  Usability evaluation considered harmful (some of the time). In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Florence, Italy) _(CHI ’08)_. Association for Computing Machinery, New York, NY, USA, 111–120.  https://doi.org/10.1145/1357054.1357074
  * Grunde-McLaughlin et al. ([n. d.]) Madeleine Grunde-McLaughlin, Michelle S. Lam, Ranjay Krishna, Daniel S. Weld, and Jeffrey Heer. [n. d.].  Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows.  arXiv:2312.11681 [cs] http://arxiv.org/abs/2312.11681
  * Han et al. (2023) Guhyun Han, Jaehun Jung, Young-Ho Kim, and Jinwook Seo. 2023.  DataHalo: A Customizable Notification Visualization System for Personalized and Longitudinal Interactions. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–21.  https://doi.org/10.1145/3544548.3580828
  * Hart and Staveland (1988) Sandra G Hart and Lowell E Staveland. 1988.  Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.  _Human mental workload_ 1, 3 (1988), 139–183.
  * Hasan et al. (2023) Md Tanim Hasan, Shaila Zaman, Amanveer Wesley, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2023.  Sympathetic Activation in Deadlines of Deskbound Research - A Study in the Wild. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–8.  https://doi.org/10.1145/3544549.3585585
  * Hekkert et al. ([n. d.]) Paul Hekkert, Dirk Snelders, and Piet C. W. Van Wieringen. [n. d.].  ‘Most advanced, yet acceptable’: Typicality and novelty as joint predictors of aesthetic preference in industrial design.  94, 1 ([n. d.]), 111–124.  https://doi.org/10.1348/000712603762842147 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1348/000712603762842147.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The Curious Case of Neural Text Degeneration.  _arXiv:1904.09751 [cs]_ (Feb. 2020).  http://arxiv.org/abs/1904.09751
  * Huang et al. ([n. d.]) Qirui Huang, Min Lu, Joel Lanir, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. [n. d.].  GraphiMind: LLM-centric Interface for Information Graphics Design.  arXiv:2401.13245 [cs] http://arxiv.org/abs/2401.13245
  * Ippolito et al. (2019) Daphne Ippolito, Reno Kriz, Maria Kustikova, João Sedoc, and Chris Callison-Burch. 2019.  Comparison of Diverse Decoding Methods from Conditional Language Models.  _arXiv:1906.06362 [cs]_ (June 2019).  http://arxiv.org/abs/1906.06362 arXiv: 1906.06362.
  * Jain and Balakrishnan (2012) Mohit Jain and Ravin Balakrishnan. 2012.  User learning and performance with bezel menus. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 2221–2230.
  * Jeno et al. (2019) Lucas M. Jeno, Vigdis Vandvik, Sigrunn Eliassen, and John-Arvid Grytnes. 2019.  Testing the novelty effect of an m-learning tool on internalization and achievement: A Self-Determination Theory approach.  _Computers & Education_ 128 (2019), 398–413.  https://doi.org/10.1016/j.compedu.2018.10.008
  * Jiang et al. (2023) Peiling Jiang, Jude Rayan, Steven P. Dow, and Haijun Xia. 2023.  Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 3, 20 pages.  https://doi.org/10.1145/3586183.3606737
  * Kazemitabaar et al. ([n. d.]) Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, and Tovi Grossman. [n. d.].  CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs.  arXiv:2401.11314 [cs] http://arxiv.org/abs/2401.11314
  * Kim et al. ([n. d.]) Jeongyeon Kim, Sangho Suh, Lydia B Chilton, and Haijun Xia. [n. d.].  Metaphorian: Leveraging Large Language Models to Support Extended Metaphor Creation for Science Writing. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh PA USA, 2023-07-10). ACM, 115–135.  https://doi.org/10.1145/3563657.3595996
  * Kim et al. (2022) Tae Soo Kim, DaEun Choi, Yoonseo Choi, and Juho Kim. 2022.  Stylette: Styling the Web with Natural Language. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 5, 17 pages.  https://doi.org/10.1145/3491102.3501931
  * Kim et al. (2023) Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. 2023.  Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 4, 18 pages.  https://doi.org/10.1145/3586183.3606833
  * Kjærup et al. (2021) Maria Kjærup, Mikael B. Skov, Peter Axel Nielsen, Jesper Kjeldskov, Jens Gerken, and Harald Reiterer. 2021.  _Longitudinal Studies in HCI Research: A Review of CHI Publications From 1982–2019_.  Springer International Publishing, Cham, 11–39.  https://doi.org/10.1007/978-3-030-67322-2_2
  * Koch et al. (2018) Michael Koch, Kai von Luck, Jan Schwarzer, and Susanne Draheim. 2018.  The novelty effect in large display deployments–Experiences and lessons-learned for evaluating prototypes. In _Proceedings of 16th European conference on computer-supported cooperative work-exploratory papers_. European Society for Socially Embedded Technologies (EUSSET).
  * Kosch et al. (2023) Thomas Kosch, Robin Welsch, Lewis Chuang, and Albrecht Schmidt. 2023.  The Placebo Effect of Artificial Intelligence in Human–Computer Interaction.  _ACM Trans. Comput.-Hum. Interact._ 29, 6, Article 56 (jan 2023), 32 pages.  https://doi.org/10.1145/3529225
  * Krischkowsky et al. ([n. d.]) Alina Krischkowsky, Manfred Tscheligi, Katja Neureiter, Michael Muller, Anna Maria Polli, and Nervo Verdezoto. [n. d.].  Experiences of Technology Appropriation: Unanticipated Users, Usage, Circumstances, and Design.  ([n. d.]).
  * Lai (2017) Poey Chin Lai. 2017.  The literature review of technology adoption models and theories for the novelty technology.  _JISTEM-Journal of Information Systems and Technology Management_ 14 (2017), 21–38.
  * Levordashka et al. (2023) Ana Levordashka, Jamie Eastman, Eleni Anna Skoulikari, Anca Salagean, Darren Cosker, and Danaë Stanton Fraser. 2023.  An Exploration of Theatre Rehearsals in Social Virtual Reality. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–7.  https://doi.org/10.1145/3544549.3585685
  * Li et al. ([n. d.]) Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. [n. d.].  Teach LLMs to Personalize – An Approach inspired by Writing Education.  arXiv:2308.07968 [cs] http://arxiv.org/abs/2308.07968
  * Li et al. (2023) Grace Li, Tao Long, and Lydia B Chilton. 2023.  Eliciting Topic Hierarchies from Large Language Models.  _arXiv preprint arXiv:2310.19275_ arXiv:2310.19275 (2023).  http://arxiv.org/abs/2310.19275
  * Lindley et al. (2017) Joseph Lindley, Paul Coulton, and Miriam Sturdee. 2017.  Implications for Adoption. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 265–277.  https://doi.org/10.1145/3025453.3025742
  * Little et al. (2010) Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010.  TurKit: human computation algorithms on mechanical turk. In _Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology_ (New York, New York, USA) _(UIST ’10)_. Association for Computing Machinery, New York, NY, USA, 57–66.  https://doi.org/10.1145/1866029.1866040
  * Liu and Chilton (2022) Vivian Liu and Lydia B Chilton. 2022.  Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 384, 23 pages.  https://doi.org/10.1145/3491102.3501825
  * Liu et al. (2023) Vivian Liu, Tao Long, Nathan Raw, and Lydia Chilton. 2023.  Generative Disco: Text-to-Video Generation for Music Visualization.  arXiv:2304.08551 (Apr 2023).  https://doi.org/10.48550/arXiv.2304.08551 arXiv:2304.08551 [cs].
  * Liu et al. (2022) Vivian Liu, Han Qiao, and Lydia Chilton. 2022.  Opal: Multimodal Image Generation for News Illustration. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 73, 17 pages.  https://doi.org/10.1145/3526113.3545621
  * Long and Chilton ([n. d.]) Tao Long and Lydia B. Chilton. [n. d.].  Challenges and Opportunities for the Design of Smart Speakers.  https://doi.org/10.48550/arXiv.2306.05741 arXiv:2306.05741 [cs]
  * Long et al. (2023) Tao Long, Dorothy Zhang, Grace Li, Batool Taraif, Samia Menon, Kynnedy Simone Smith, Sitong Wang, Katy Ilonka Gero, and Lydia B. Chilton. 2023.  Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Media. In _Proceedings of the 14th Conference on Computational Creativity_ _(ICCC ’23)_. Association for Computational Creativity.
  * MacLean et al. (1990) Allan MacLean, Kathleen Carter, Lennart Lövstrand, and Thomas Moran. 1990.  User-tailorable systems: pressing the issues with buttons. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Seattle, Washington, USA) _(CHI ’90)_. Association for Computing Machinery, New York, NY, USA, 175–182.  https://doi.org/10.1145/97243.97271
  * Marathe and Sundar (2011) Sampada Marathe and S. Shyam Sundar. 2011.  What drives customization? control or identity?. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Vancouver, BC, Canada) _(CHI ’11)_. Association for Computing Machinery, New York, NY, USA, 781–790.  https://doi.org/10.1145/1978942.1979056
  * Mirowski et al. (2023) Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023.  Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 355, 34 pages.  https://doi.org/10.1145/3544548.3581225
  * Moran ([n. d.]) Thomas P Moran. [n. d.].  Everyday adaptive design. In _Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques_ (London England, 2002-06-25). ACM, 13–14.  https://doi.org/10.1145/778712.778715
  * Mukherjee and Chang (2023) Anirban Mukherjee and Hannah Chang. 2023.  Managing the Creative Frontier of Generative AI: The Novelty-Usefulness Tradeoff.  _California Management Review_ (2023).
  * Muller et al. (2016) Michael Muller, Katja Neureiter, Nervo Verdezoto, Alina Krischkowsky, Anna Maria Al Zubaidi-Polli, and Manfred Tscheligi. 2016.  Collaborative Appropriation: How Couples, Teams, Groups and Communities Adapt and Adopt Technologies. In _Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion_ (San Francisco, California, USA) _(CSCW ’16 Companion)_. Association for Computing Machinery, New York, NY, USA, 473–480.  https://doi.org/10.1145/2818052.2855508
  * Nielsen ([n. d.]) Peter Axel Nielsen. [n. d.].  Longitudinal Studies in Information Systems.  In _Advances in Longitudinal HCI Research_ , Evangelos Karapanos, Jens Gerken, Jesper Kjeldskov, and Mikael B. Skov (Eds.). Springer International Publishing, 41–56.  https://doi.org/10.1007/978-3-030-67322-2_3
  * OpenAI ([n. d.]) OpenAI. [n. d.].  GPT-4 Technical Report.  https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs]
  * Park et al. (2023) Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023.  Generative Agents: Interactive Simulacra of Human Behavior. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 2, 22 pages.  https://doi.org/10.1145/3586183.3606763
  * Petridis et al. (2023a) Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Stan Jastrzebski, Keren Henderson, Jeffrey V. Nickerson, and Lydia B. Chilton. 2023a.  AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_.
  * Petridis et al. (2023b) Savvas Petridis, Michael Terry, and Carrie Jun Cai. 2023b.  PromptInfuser: Bringing User Interface Mock-Ups to Life with Large Language Models. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI EA ’23)_. Association for Computing Machinery, New York, NY, USA, Article 237, 6 pages.  https://doi.org/10.1145/3544549.3585628
  * Petridis et al. ([n. d.]) Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J. Cai, and Michael Terry. [n. d.].  ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles.  arXiv:2310.15428 [cs] http://arxiv.org/abs/2310.15428
  * Proença et al. (2021) Mailson de Queiroz Proença, Vivian Genaro Motti, Kamila Rios da Hora Rodrigues, and Vânia Paula de Almeida Neris. 2021.  Coping with Diversity - A System for End-users to Customize Web User Interfaces.  _Proc. ACM Hum.-Comput. Interact._ 5, EICS, Article 201 (may 2021), 27 pages.  https://doi.org/10.1145/3457151
  * Qiu et al. (2023) Huilian Sophie Qiu, Anna Lieb, Jennifer Chou, Megan Carneal, Jasmine Mok, Emily Amspoker, Bogdan Vasilescu, and Laura Dabbish. 2023.  Climate Coach: A Dashboard for Open-Source Maintainers to Overview Community Dynamics. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–18.  https://doi.org/10.1145/3544548.3581317
  * Retelny et al. (2014) Daniela Retelny, Sebastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014.  Expert crowdsourcing with flash teams. In _Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST ’14_. ACM, New York, NY, USA, 75–85.  https://doi.org/10.1145/2642918.2647409
  * Rodrigues et al. ([n. d.]) Luiz Rodrigues, Filipe D. Pereira, Armando M. Toda, Paula T. Palomino, Marcela Pessoa, Leandro Silva Galvão Carvalho, David Fernandes, Elaine H. T. Oliveira, Alexandra I. Cristea, and Seiji Isotani. [n. d.].  Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study.  19, 1 ([n. d.]).  https://doi.org/10.1186/s41239-021-00314-6 Publisher: SpringerOpen.
  * Ross et al. (2023) Steven I. Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D. Weisz. 2023.  The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 491–514.  https://doi.org/10.1145/3581641.3584037
  * Rutten et al. (2021) Isa Rutten, Lawrence Van den Bogaert, and David Geerts. 2021.  From Initial Encounter With Mid-Air Haptic Feedback to Repeated Use: The Role of the Novelty Effect in User Experience.  _IEEE Transactions on Haptics_ 14, 3 (2021), 591–602.  https://doi.org/10.1109/TOH.2020.3043658
  * Rutten and Geerts (2020) Isa Rutten and David Geerts. 2020.  Better Because It’s New: The Impact of Perceived Novelty on the Added Value of Mid-Air Haptic Feedback. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376668
  * Ryan and Deci (2017) Richard M Ryan and Edward L Deci. 2017.  _Self-determination theory: Basic psychological needs in motivation, development, and wellness_.  Guilford publications.
  * Sadeghi et al. (2022) Soheil Sadeghi, Somit Gupta, Stefan Gramatovici, Jiannan Lu, Hao Ai, and Ruhan Zhang. 2022.  Novelty and Primacy: A Long-Term Estimator for Online Experiments.  _Technometrics_ 64, 4 (2022), 524–534.  https://doi.org/10.1080/00401706.2022.2124309 arXiv:https://doi.org/10.1080/00401706.2022.2124309
  * Schick et al. ([n. d.]) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. [n. d.].  PEER: A Collaborative Language Model.  arXiv:2208.11663 [cs] http://arxiv.org/abs/2208.11663
  * Seo et al. ([n. d.]) Woosuk Seo, Chanmo Yang, and Young-Ho Kim. [n. d.].  ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events.  arXiv:2309.12244 [cs] http://arxiv.org/abs/2309.12244
  * Shen et al. (2023) Hua Shen, Chieh-Yang Huang, Tongshuang Wu, and Ting-Hao Kenneth Huang. 2023.  ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing. In _Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing_ (Minneapolis, MN, USA) _(CSCW ’23 Companion)_. Association for Computing Machinery, New York, NY, USA, 384–387.  https://doi.org/10.1145/3584931.3607492
  * Shin et al. ([n. d.]) Grace Shin, Yuanyuan Feng, Mohammad Hossein Jarrahi, and Nicci Gafinowitz. [n. d.].  Beyond novelty effect: a mixed-methods exploration into the motivation for long-term activity tracker use.  2, 1 ([n. d.]), 62–72.  https://doi.org/10.1093/jamiaopen/ooy048 _eprint: https://academic.oup.com/jamiaopen/article-pdf/2/1/62/32298485/ooy048.pdf.
  * Shin et al. (2023) Sungbok Shin, Sanghyun Hong, and Niklas Elmqvist. 2023.  Perceptual Pat: A Virtual Human Visual System for Iterative Visualization Design. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–17.  https://doi.org/10.1145/3544548.3580974
  * Singh et al. (2022) Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L. Glassman. 2022.  Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence.  _ACM Trans. Comput.-Hum. Interact._ (feb 2022).  https://doi.org/10.1145/3511599 Just Accepted.
  * Son et al. ([n. d.]) Kihoon Son, DaEun Choi, Tae Soo Kim, Young-Ho Kim, and Juho Kim. [n. d.].  GenQuery: Supporting Expressive Visual Search with Generative Models.  https://doi.org/10.48550/arXiv.2310.01287 arXiv:2310.01287 [cs]
  * Sporka et al. (2011) Adam J Sporka, Torsten Felzer, Sri H Kurniawan, Ondřej Poláček, Paul Haiduk, and I Scott MacKenzie. 2011.  Chanti: Predictive text entry using non-verbal vocal input. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 2463–2472.
  * Strauss and Corbin (1994) Anselm Strauss and Juliet Corbin. 1994.  Grounded theory methodology: An overview.  (1994).
  * Suh et al. (2023) Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2023.  Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation.  _arXiv preprint arXiv:2310.12953_ (2023).
  * Suh et al. ([n. d.]) Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. [n. d.].  Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (2023-10-29). 1–18.  https://doi.org/10.1145/3586183.3606756 arXiv:2305.11483 [cs]
  * Symplur (2019) Symplur. 2019.  Tweetorials — From Early Beginnings to Huge Growth and Beyond.  https://www.symplur.com/blog/tweetorials-from-early-beginnings-to-huge-growth-and-beyond/
  * Tseng et al. ([n. d.]) Tiffany Tseng, Ruijia Cheng, and Jeffrey Nichols. [n. d.].  Keyframer: Empowering Animation Design using Large Language Models.  arXiv:2402.06071 [cs] http://arxiv.org/abs/2402.06071
  * Voit et al. (2020) Alexandra Voit, Jasmin Niess, Caroline Eckerth, Maike Ernst, Henrike Weingärtner, and Paweł W. Woźniak. 2020.  ‘It’s not a romantic relationship’: Stories of Adoption and Abandonment of Smart Speakers at Home. In _Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia_ (Essen, Germany) _(MUM ’20)_. Association for Computing Machinery, New York, NY, USA, 71–82.  https://doi.org/10.1145/3428361.3428469
  * Wang et al. ([n. d.]) Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V. Nickerson, and Lydia B. Chilton. [n. d.].  ReelFramer: Human-AI Co-Creation for News-to-Video Translation.  https://doi.org/10.48550/arXiv.2304.09653 arXiv:2304.09653 [cs]
  * Wang et al. (2023) Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B Chilton. 2023.  PopBlends: Strategies for conceptual blending with large language models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Wei et al. ([n. d.]) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [n. d.].  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 ([n. d.]).  https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]
  * Williams et al. (2022) Spencer Williams, Ridley Jones, Katharina Reinecke, and Gary Hsieh. 2022.  An HCI Research Agenda for Online Science Communication.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW2 (Nov 2022), 1–22.  https://doi.org/10.1145/3555591
  * Wu et al. (2022a) Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022a.  PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, Article 359, 10 pages.  https://doi.org/10.1145/3491101.3519729
  * Wu et al. (2022b) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022b.  AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages.  https://doi.org/10.1145/3491102.3517582
  * Wu et al. ([n. d.]) Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, Jenny T. Liang, Ryan Liu, Ihita Mandal, Jeremiah Milbauer, Xiaolin Ni, Namrata Padmanabhan, Subhashini Ramkumar, Alexis Sudjianto, Jordan Taylor, Ying-Jui Tseng, Patricia Vaidos, Zhijin Wu, Wei Wu, and Chenyang Yang. [n. d.].  LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs.  arXiv:2307.10168 [cs] http://arxiv.org/abs/2307.10168
  * Xiao et al. ([n. d.]) Shishi Xiao, Liangwei Wang, Xiaojuan Ma, and Wei Zeng. [n. d.].  TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation.  https://doi.org/10.1145/3613904.3642185 arXiv:2401.11094 [cs]
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, MingSung Kan, Junxian He, and Qizhe Xie. 2023.  Self-Evaluation Guided Beam Search for Reasoning.  https://api.semanticscholar.org/CorpusID:258426922
  * Yang et al. ([n. d.]) Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. [n. d.].  Re3: Generating Longer Stories With Recursive Reprompting and Revision. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ (Abu Dhabi, United Arab Emirates, 2022-12), Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 4393–4479.  https://doi.org/10.18653/v1/2022.emnlp-main.296
  * Zamfirescu-Pereira et al. (2023) J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023.  Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages.  https://doi.org/10.1145/3544548.3581388
  * Zhang et al. (2023) Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023.  VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 5, 30 pages.  https://doi.org/10.1145/3586183.3606800
  * Zhou et al. (2023) Ran Zhou, Zachary Schwemler, Akshay Baweja, Harpreet Sareen, Casey Lee Hunt, and Daniel Leithinger. 2023.  TactorBots: A Haptic Design Toolkit for Out-of-lab Exploration of Emotional Robotic Touch. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–19.  https://doi.org/10.1145/3544548.3580799
  * Zhou et al. ([n. d.]) Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du. [n. d.].  InstructPipe: Building Visual Programming Pipelines with Human Instructions.  arXiv:2312.09672 [cs] http://arxiv.org/abs/2312.09672

##  Appendix A Complete System Walkthrough

##  Appendix B Week 3 System Updates: Step 6 Learn-My-Style Prompt