# AI Alignment in the Design of Interactive AI: Specification Alignment,
Process Alignment, and Evaluation Support

Michael Terry<EMAIL_ADDRESS>Google Research ,  Chinmay Kulkarni
Emory University ,  Martin Wattenberg  Harvard University and Google Research
,  Lucas Dixon  Google Research and  Meredith Ringel Morris  Google DeepMind

(2023)

###### Abstract.

AI alignment considers the overall problem of ensuring an AI produces desired
outcomes, without undesirable side effects. While often considered from the
perspectives of safety and human values, AI alignment can also be considered
in the context of designing and evaluating interfaces for interactive AI
systems, where users interactively align an AI to their goals. To this end,
this paper maps concepts from AI alignment onto a basic, three step
interaction cycle, defined as 1) user input, 2) system processing, and 3)
system output. This mapping yields a corresponding set of alignment objectives
for interactive AI: 1) specification alignment: ensuring the user can
efficiently and reliably communicate objectives to the AI, and verify they
were correctly interpreted, 2) process alignment: providing the ability to
verify and optionally control the AI’s execution process, and 3) evaluation
support: ensuring the user can verify and understand the AI’s output. We also
introduce the concepts of a surrogate process, defined as a simplified,
separately derived, but controllable representation of the AI’s actual
process; and the notion of a Process Gulf, an extension of Norman’s Gulfs of
Execution and Evaluation (Norman, 2002) that highlights how differences
between human and AI processes can lead to challenges in AI control. To
illustrate the value of this framework, we describe commercial and research
systems along each of the three alignment objectives, and show how interfaces
that provide interactive alignment mechanisms can lead to qualitatively
different and improved user experiences. We conclude by highlighting promising
areas of future work at the intersection of AI alignment, HCI research, and
the design of interactive AI.

††journalyear: 2023

##  1\. Introduction

Recent years have witnessed notable advances in the ability for AI to solve a
wide range of increasingly sophisticated and challenging problems. For
example, GPT-4 (OpenAI, 2023), a large language model (LLM), is capable of
passing exams, writing compelling new poems, synthesizing code, rewriting
text, constructing mathematical proofs, and solving problems involving spatial
reasoning (among many other tasks) (Bubeck et al., 2023). Other generative AI
systems have similarly demonstrated significant advances in the ability for
people to easily produce images (Saharia et al., 2022; Ramesh et al., 2022),
audio (Agostinelli et al., 2023), and videos (Ho et al., 2022), all from
natural language descriptions alone. Because of their ability to be applied to
a wide range of problems, these new models are sometimes referred to as
foundation models (Bommasani et al., 2022).

The newly realized capabilities of modern-day, general-purpose AI have
heightened the relevance of existing discussions on AI alignment. AI alignment
considers the overall problem of how to ensure an AI produces the intended
outcomes (as determined by its creator and/or user), without additional,
undesirable side effects (e.g., by not performing operations that could
negatively affect individuals, groups, or even society at large)111A classic
(though extreme) example of an AI alignment problem is illustrated by the
“paperclip maximizer” thought experiment (Bostrom, 2003), in which a
hypothetical AI told to produce as many paperclips as possible might do so
through means that diverge from typical human task interpretations and value
systems, such as killing all humans so that their atoms can be converted into
additional paperclips. (Leike et al., 2018; Christiano, [n. d.]; Kenton et
al., 2021; Kirchner et al., 2022; Christian, 2020).

While the AI alignment literature sometimes considers scenarios in which AI is
far more capable than present-day AI (Bostrom, 2014; Russell et al., 2016),
the need for mechanisms that improve human-AI alignment is already evident in
current human-AI systems (Yang et al., 2020). For example, users often report
challenges in knowing how to effectively “prompt” LLMs and other generative AI
systems to produce the desired outcomes (e.g., what input to provide a
generative image model to produce the desired image) (Zamfirescu-Pereira et
al., 2023b, a; Jiang et al., 2022b; Chang et al., 2023). The AI alignment
literature describes this as the specification problem, or the problem of AI
reliably and correctly interpreting a user’s intent (Leike et al., 2017;
Ortega et al., [n. d.]). Given the existence of this issue in current
interactive AI systems, there is value in considering issues such as
specification alignment not only in terms of the AI’s underlying design and
implementation, but also in terms of the design and implementation of the
interfaces mediating the moment-to-moment interactions between the user and
the AI. More specifically, how can we design interactive human-AI systems that
support users in efficiently and reliably aligning AI on the desired outcomes
and the means of achieving those outcomes? And how can we help users verify
that the desired outcomes were actually achieved by the AI? Focusing on
interactive AI systems, we call this overall problem the interactive alignment
problem, and scope the paper by considering this problem in the context of
single user, single agent AI systems (e.g., a chatbot or a system that can
interpret and act on natural language input).

In the spirit of making progress on the problem of designing and evaluating
alignable, interactive systems, we join related threads in AI alignment and
HCI research using the common ground of a basic three step interaction cycle,
defined by 1) the user providing input to the system, 2) the system processing
that input, and 3) the user assessing the system’s output with respect to
their goals (see Figure 1A). These three fundamental steps underpin any
interaction cycle with a computational system, whether it is with a
traditional, non-AI system (Figure 1A) or an AI (Figure 1B).

Given this foundation, we map concepts of AI alignment onto the interaction
cycle to highlight three specific AI alignment objectives one can consider in
the design and evaluation of interactive AI (see Table 1 and Figure 1):

Table 1. Dimensions of AI alignment in the design and evaluation of interactive AI systems. Term |  Definition |  Example
---|---|---
Specification alignment |  Provide the ability for the user to efficiently and reliably communicate desired outcomes to the AI, and ensure they can verify and make necessary corrections to the AI’s interpretation of the intended outcome(s). |  Given the input “Write a TODO app,” a code synthesis AI replies that it will write a basic TODO app in JavaScript and HTML, with the ability to add and delete TODO items.
Specification alignment: Outcome specification |  A description of the desired outcome the AI should produce. |  Create a functional TODO app in HTML and JavaScript.
Specification alignment: Specification constraints |  Additional constraints and desiderata for the final outcome. |  Make the code understandable to someone who just started learning how to program.
Process alignment |  Provide mechanisms that enable the user to align the AI on the process by giving users the ability to directly define, modify, and/or influence the AI’s process. |  A code synthesis AI describes which libraries and coding styles it will use in writing the TODO app, and provides the ability for the user to modify these choices.
Process alignment: Means alignment |  Enable the user to align the AI on the means by which it should produce its output. Means alignment can include concerns such as safety, cost, and ethics, as well as user interaction needs (e.g., maintaining a sense of agency and control in creating the output). |  The user specifies that the AI should not replicate code in its training data without proper attribution.
Process alignment: Control alignment |  A refinement of process alignment with the goal of providing human-optimized controls and data representations that enable the user to modify specific steps of the AI’s process. |  The AI provides a pseudocode description of the code it will write, and allows the user to edit the pseudocode to alter the actual code produced (as in Liu et al.’s system (Liu et al., 2023)).
Process alignment: Surrogate process |  A simplified, but controllable, representation of the AI’s actual process. |  Liu et al.’s pseudocode strategy described above (Liu et al., 2023) is a surrogate process for the actual code synthesis process.
Evaluation support |  Ensure the user can validate that the AI’s output meets their objectives. |  The user can determine whether the code actually creates a functional TODO app.
Evaluation alignment: Verification support |  Ensure the user can verify the correctness or suitability of the AI’s output. |  Synthesized code is run in a sandbox to aid in verification.
Evaluation support: Comprehension support |  Ensure the user can fully comprehend (i.e., understand) the AI’s output. |  A code synthesis AI comments its code.

Figure 1. Basic models of interaction. A: In interacting with a traditional
non-AI system, the user chooses an operation to perform and provides input to
the system to perform that operation (1). The system performs the operation
(2), then provides the output to the user, which they assess (3) with respect
to their goals. B: When interacting with an AI, the user communicates their
desired outcome to the AI (1), the AI interprets the goal and performs
operations to achieve that goal (2), and (3) the output is sent to the user.
C: The same human-AI interaction cycle with AI alignment concepts mapped onto
the three steps: (1) Specification alignment mechanisms provide means for the
user to align the AI on the specific task to perform. (2) Process alignment
mechanisms enable the user to modify how the task is performed, potentially
offering the user direct control over specific steps. (3) Evaluation support
mechanisms help the user assess and understand the output.

Figures showing a) users submitting input to a computer and receiving output,
b) users submitting a goal to an AI, it processing the input, and providing
output, and c) users submitting a goal to an AI, then interacting with the AI
to get alignment on the specification, process, and evaluation (assessment) of
the end result.

  1. (1)

Specification alignment: Provide the ability for the user to efficiently and
reliably communicate desired outcomes to the AI, and ensure they can verify
and make necessary corrections to the AI’s interpretation of the intended
outcome(s). Specification alignment can be further considered in terms of:

    1. (a)

Outcome specification: A description of the desired outcome the AI should
produce.

    2. (b)

Specification constraints: Additional constraints and desiderata for the final
outcome (e.g., make the synthesized code understandable to a novice).

  2. (2)

Process alignment: Provide the ability for the user to align the AI on the
process by which it produces its result, including the ability for users to
directly define, modify, and/or influence the AI’s process. Process alignment
can further be faceted into:

    1. (a)

Means alignment: Enabling the user to align the AI on the means by which it
should produce its output. Means alignment can include concerns such as
safety, cost, and ethics, as well as user interaction needs (e.g., maintaining
a sense of agency and control in creating the output).

    2. (b)

Control alignment: Providing human-optimized controls and data representations
that enable the user to control specific steps of the AI’s process.

  3. (3)

Evaluation support: Ensure the user can validate that the AI’s output meets
their objectives. Evaluation support can be further divided into:

    1. (a)

Verification support: Ensuring the user can verify the correctness or
suitability of the AI’s output.

    2. (b)

Comprehension support: Ensuring the user can fully comprehend (i.e.,
understand) the AI’s output.

These concepts have been discussed in varying degrees in past AI alignment and
HCI literature (e.g., the notion of a “specification” is discussed in both the
AI alignment (Leike et al., 2017; Ortega et al., [n. d.]) and HCI literature
(Nielsen, 2023)); this paper brings this related work together by utilizing a
common vocabulary and grounding context (the basic interaction cycle).

To demonstrate the value of this interaction-focused framing of AI alignment,
we analyze two sets of preexisting interactive AI systems: one set of systems
designed to support image generation, the other, code synthesis. In examining
these example systems, we show how the framework provides a vocabulary for
describing the types of human-AI alignment mechanisms each system provides,
and observe how the presence of specific types of interactive alignment
mechanisms can lead to qualitatively different and improved user experiences.
We also show how the framework suggests additional interactive capabilities
that could be useful to include in the system designs. These examples thus
illustrate both the descriptive and prescriptive value of the framework.

The rest of the paper contextualizes this work in the domains of AI alignment
and HCI research, then provides a detailed description of the user-centered AI
alignment concepts described above. This detailed description also introduces
the concept of a surrogate process (a simplified, separately derived, but
controllable representation of the AI’s actual process; see Figure 2), and the
notion of a Process Gulf (Figure 3), an extension of Norman’s concepts of the
Gulfs of Execution and Evaluation (Norman, 2002) that highlights the gulf that
can arise between a person and an AI due to the qualitatively different ways
in which each entity produces an output (e.g., a diffusion model produces
images in a qualitatively different way than a person). When the user needs to
more directly control the AI’s process, they must explicitly bridge this
Process Gulf, which can be challenging if the AI’s process and operations
differ substantially from the user’s. We illustrate the value of this overall
framework by applying it to the analysis of example systems, and conclude with
a discussion and implications for future research in interactive alignment.

##  2\. Related Work

This section ties together related work in human-computer and human-AI
interaction, AI alignment, human-AI design guidelines, and mixed-initiative
interfaces.

###  2.1. A Basic Model of Human-Computer Interaction

Human-computer interaction can be modeled as a basic three step process
consisting of 1) the user providing input to the system, 2) the system acting
on that input, and 3) the user assessing the system’s output with respect to
the user’s goals and potential next actions (Figure 1A). We use this basic
three step model as a foundation for considering alignment needs when AI is
introduced into an interactive system.

Artificial intelligence (AI) can be defined in a number of ways, but one
common approach is to define AI as an agent that can perceive its environment
and take actions in pursuit of goals (Russell and Norvig, 2009; Horvitz,
1999). At present, some of the most successful AI implementations make use of
generative machine learning models (e.g., (Leike et al., [n. d.]; Bommasani et
al., 2022; Agostinelli et al., 2023; Ramesh et al., 2022; Saharia et al.,
2022; Ho et al., 2022)), with large language models (LLMs) used in the
creation of general-purpose AI agents, such as ChatGPT (OpenAI, [n. d.]) and
Bard (Google, [n. d.]).

When AI is introduced into an interactive system, it can change the basic
interaction cycle in fundamental ways (Yang et al., 2020; Amershi et al.,
2019). In a traditional interaction cycle, the user provides input intended to
invoke a specific operation. With interactive, agent-like AI systems, the
user’s initial input can instead be an articulation of the desired outcome
(i.e., their goal, such as an an image to generate; see Figure 1B) (Nielsen,
2023). In the second step of the interaction cycle, the system processes the
user’s input, with the key difference for AI systems being that the AI
determines what operations to perform to achieve the goal (Nielsen, 2023). In
the final step, the system produces output, which the user assesses with
respect to their goals. In this last step, AI-based systems can introduce some
additional user needs. For example, the amount of AI output can be voluminous
(e.g., as observed in music generation (Louie et al., 2020) and other
generative AI systems (Buschek et al., 2021)), and it may require verification
of its correctness and/or completeness respect to user goals. Advanced AI
systems in particular may produce output that is beyond the abilities of an
end-user to deeply understand (Bostrom, 2014; Morris, 2023).

###  2.2. AI Alignment

As mentioned in the Introduction, AI alignment seeks to ensure an AI produces
intended outcomes without undesirable side effects (Leike et al., 2018;
Christiano, [n. d.]; Kenton et al., 2021; Kirchner et al., 2022; Christian,
2020; Irving et al., 2018). This overall goal can be considered from a number
of different perspectives. For example, Gabriel (Gabriel, 2020) observes that
one could attempt to align an AI with “instructions, intentions, revealed
preferences, ideal preferences, interests and values.” Issues of safety and
harms (Irving et al., 2018; Russell et al., 2016), including potential long-
term existential risks (Bostrom, 2014), are often included in discussions of
AI alignment. AI alignment can also be considered with respect to different
stakeholders, including system creators, the users, and other individuals
potentially affected by the AI. In this paper, we specifically focus on the
problem of AI alignment from the perspective of interface needs for users
directly interacting with an AI to accomplish a specific goal. Given this
scoping, we limit our coverage of otherwise critical topics such as AI safety
(Russell et al., 2016), specification gaming (Krakovna et al., 2020), and
malicious and/or deceptive AI (Bostrom, 2014).

From this foundation, we next consider AI alignment in the context of the
basic three step interaction cycle described above.

###  2.3. Specification

A number of definitions of AI alignment focus on ensuring the AI executes
according to the user’s or designer’s intent (e.g., (Leike et al., 2018;
Yudkowsky, 2001; Kenton et al., 2021; Leike et al., [n. d.]; Ortega et al.,
[n. d.])). For an AI to be able to act on a user’s intent, that intent must be
communicated to and/or inferred by the AI. When done explicitly, communicating
a goal can be described as providing a specification (Leike et al., 2017;
Ortega et al., [n. d.]; Nielsen, 2023). Modern AI enables people to explicitly
communicate a specification through mechanisms such as natural language
prompts (Leike et al., [n. d.]; Bommasani et al., 2022; Agostinelli et al.,
2023; Ramesh et al., 2022; Saharia et al., 2022; Ho et al., 2022; GitHub, [n.
d.]) and sketches (AI, [n. d.]). Some systems infer goals from context, such
as code synthesis assistants like CoPilot (GitHub, [n. d.]), or research
systems like Horvitz’s LookOut (Horvitz, 1999).

Fully and accurately describing the intended outcome can be challenging for a
user (Leike et al., 2018; Irving et al., 2018). A user may also not be aware
of how their specification could be incorrectly interpreted, or the potential,
unwanted side effects that could arise (Bostrom, 2014; Russell and Norvig,
2009). Additionally, user input may be ambiguous, under-specified, or even
incorrect. Collectively, these challenges in providing a complete, correct
specification are referred to as the specification problem and motivate the
need for mechanisms that support users in aligning an AI on their specific
goals (i.e., specification alignment) (Leike et al., 2018; Ortega et al., [n.
d.]).

Recent advances in large language models (LLMs) have helped to partially
address the specification alignment problem by enabling people to directly
state their intended goals using natural language examples (i.e., few-shot
examples (Brown et al., 2020)) and/or instructions (OpenAI, 2023; Ouyang et
al., 2022). Notably, greater specification alignment seems to be achieved with
user-centered model tuning such as instruction tuning (Ouyang et al., 2022),
which explicitly incorporates human feedback in model training. The ability
for users to specify intent using natural language leads to an interaction
paradigm Jakob Nielsen describes as “intent-based outcome specification,”
where the AI, not the person, determines how to achieve that outcome (Nielsen,
2023).

Despite advances in interpreting natural language intent, recent research
suggests that prompting AI to produce desired outcomes can still be
challenging. For example, studies of prompting (Zamfirescu-Pereira et al.,
2023b; Jiang et al., 2022a) and chatbot design (Zamfirescu-Pereira et al.,
2023a) describe challenges users face in identifying effective strategies for
prompting a model. These persistent issues suggest the value in further
research on specification alignment, both in terms of the underlying AI and
the capabilities provided in the user interface.

###  2.4. Process Control

A common finding in human-AI research is the desire for users to more directly
control AI, in domains ranging from AI medical assistants (Cai et al., 2019)
to music generation (Louie et al., 2020). Numerous tools and mechanisms have
been developed to offer users greater control over AI, including those that
enable users to directly influence search mechanisms (Cai et al., 2019), music
generation (Louie et al., 2020), recommendations (Sanner et al., 2023), and
image generation (Chung and Adar, 2023). Providing AI control requires linking
low-level AI processes with high-level, user-centered controls, which is
typically a non-trivial process.

The AI safety and alignment literature also emphasizes the need and value of
providing human control of AI systems after they have begun operating, in case
the system design is discovered to be imperfect or potentially unsafe during
operation (Russell et al., 2016). This has been described as the “control
problem” (Bostrom, 2014).

###  2.5. Evaluation

As AI becomes increasingly capable in terms of the difficulty and complexity
of the problems it can solve, a significant challenge is evaluating its
outputs (Leike et al., 2018; Kim, 2022; Irving et al., 2018). The problem of
evaluation can be further divided into two problems: 1) verifying the AI’s
output correctly and completely fulfills the user’s intent, and 2) actually
understanding the AI’s output (Kim, 2022; Irving et al., 2018). In the AI
alignment literature, human evaluation has also been considered from the point
of view of ensuring users can effectively evaluate the AI’s output as part of
the training process (i.e., to provide feedback to improve a reward function
used to train the model) (Leike et al., 2018). In this work, we focus on forms
of evaluation of interest to end-users using AI to accomplish a goal.

For some problems, verification can be straightforward. For example, it may be
easy to assess whether an image produced by a generative AI meets the user’s
needs. However, in other cases, verification may not be possible or require
extensive work. For example, code synthesis is now a common task for
generative text models, but the correctness of arbitrary code is a well-known
example in the class of undecidable problems (Turing, 1936). Even when the
output of an AI can be proven to be correct, it can still be extremely
difficult to do so. For example, despite four years of work by a panel of 12
referees appointed by the Annals of Mathematics, the panel concluded that they
could not certify the correctness of all of the computer’s calculations when
assessing a machine-assisted proof of the Kepler conjecture (Hales, 2005).

As mentioned above, one challenge in evaluating output is that an AI may
produce a significant amount of output in a short period of time (Louie et
al., 2020). However, an AI may also produce a relatively small amount of
output that nonetheless requires extensive investigation to understand. For
example, AlphaGo’s Move 37 presents a compelling example of an AI producing a
relatively small output that was not initially understood: Go experts
initially thought it was a mistake. However, after detailed analysis, it was
found to be an excellent move (Menick, 2016). As AI advances continue, it is
expected that more and more such situations will arise, where the AI achieves
the desired outcome, yet understanding why (and if) its output is correct will
be challenging (Kim, 2022).

Table 2. Highly relevant principles of mixed initiative interfaces and human-
AI guidelines from Horvitz (Horvitz, 1999) and Amershi et al. (Amershi et al.,
2019), respectively. Horvitz’s principles are marked with a P, and Amershi et
al.’s guidelines are marked with a G. Related Principles of Mixed Initiative
Interfaces (P) and Human-AI Guidelines (G)
---
P5. Employ dialog to resolve key uncertainties.
P6. Allow efficient direct invocation and termination.
P9. Provide mechanisms for efficient agent-user collaboration to refine
results.
G1. Make clear what the system can do.
G2. Make clear how well the system can do what it can do.
G7. Support efficient invocation.
G8. Support efficient dismissal.
G9. Support efficient correction.
G11. Make clear why the system did what it did.

###  2.6. Human-AI Guidelines and Principles of Mixed Initiative Interfaces

In considering the design of human-AI interactions, both Horvitz’s Principles
of Mixed Initiative Interfaces (Horvitz, 1999) and Amershi et al.’s Guidelines
for Human-AI Interactions (Amershi et al., 2019) are highly relevant.
Horvitz’s principles derive from his work producing intelligent systems that
can 1) infer a user’s intent and 2) automatically operate on their behalf
(e.g., a system that can automatically extract a meeting invite from an email
and add it to a calendar). Notably, his principles consider challenges both in
the design of the underlying AI and the user interface to the AI (e.g., how to
design interactions given uncertainty in inference). Amershi et al.’s work
builds on many of these earlier principles and offers a set of 18 design
guidelines.

In the context of this work, Horvitz’s principles 5, 6, and 9, and Amershi et
al.’s guidelines G1-G2, G7-G9, and G11 are perhaps most relevant (see Table 2
for a summary of these principles and guidelines). Principle 5 argues for the
value of employing dialog with the user to resolve uncertainty in a user’s
intentions, and guidelines G1-G2 advocate for providing an indication of what
an AI can do, and how well it can do it. These guidelines all have relevance
to the goal of attaining specification alignment.

Principles 6 and 9, and Guidelines G7-G9 argue for the value in providing
users control in starting and terminating agent processes, and in refining or
correcting results provided by the AI. These concepts link to the idea of
process alignment. This subset of the principles and guidelines also
underscore the importance of considering which party (the human or the AI) has
control over which processes, and whether it is possible to transfer control
between parties.

Guideline G11 suggests the value in providing an indication of why the system
“did what it did.” This guideline could be interpreted as a suggestion to
provide interpretable or explainable AI capabilities, but could also be viewed
as a suggestion that the AI provide a rationale for its output. In the
framework described in this paper, the notion of evaluation support maps most
closely to this latter interpretation (i.e., offering rationale for the output
produced).

Finally, both Horvitz and Amershi et al.’s principles and guidelines argue for
systems that can continue to learn by observing. Designing for continual
learning seems especially important in the context of AI alignment, and is a
topic we return to in the later Discussion section.

##  3\. Dimensions of AI Alignment in Interactive Systems

In this section, we elaborate on the concepts of specification alignment,
process alignment, and evaluation support. In considering these dimensions of
AI alignment, we constrain our scope to users of interactive AI systems,
though we recognize that there are additional stakeholders one could consider,
including system designers or even society at large. As previously mentioned,
we also largely consider single user scenarios in which they interact with a
single AI.

Implicit in the notion of achieving alignment is the idea that it is an
interactive process between the user and the AI. Thus, there is an assumption
of interactive mechanisms for users to increase alignment. A joint challenge
for the AI and HCI communities is to develop effective, reliable, and usable
mechanisms for interactive alignment.

###  3.1. Specification Alignment

We define specification alignment as the process of 1) providing (or
inferring) a specification of the user’s goal to the AI, 2) confirming
(validating) that the AI’s interpretation of the desired outcome matches the
user’s, free of undesirable side effects, and 3) making necessary refinements
until sufficient alignment has been reached.

In attaining specification alignment, we further distinguish between an
outcome specification, or a description of the ultimate goal of the user
(e.g., to produce an image, to synthesize code, etc.), and specification
constraints and desiderata, which describe additional constraints for the
outcome produced.

Importantly, specification alignment only considers how well the AI interprets
the user’s objectives; how well it actually performs the task is a separate
issue. For example, an AI system that can generate photorealistic images may
still have low specification alignment if it omits requested details.
Conversely, an image synthesis AI may include everything requested by the
user, but produce an image lacking the desired quality.

####  3.1.1. When Specification Alignment Can Occur

Specification alignment can be achieved prospectively (by providing an
interpretation of the user’s request before any actions are taken), in tandem
with actions being taken (i.e., a specification is provided while the AI
performs actions), and retrospectively (a specification of what was done is
provided after actions were taken). Providing a specification in parallel
with, or after, taking actions may seem counter to the intent of specification
alignment, but taking actions before achieving specification alignment can
lead to efficiencies when the AI correctly interprets the user’s intent, or
when the AI’s output is close enough that editing the final result is at least
as efficient as editing the specification (e.g., as can happen in code
synthesis (Liu et al., 2023)). However, there are also obvious circumstances
where taking action before specification verification occurs should not
happen. For example, if the actions an AI can take are costly, have safety
implications, or can’t be undone, the interaction should be designed so the
user can verify the AI’s plan of action before it proceeds.

####  3.1.2. Specification Refinement and Clarification

It is possible that users provide under-specified and/or ambiguous
specifications to the system. For example, a user may request a code synthesis
AI to “write a TODO app,” a specification with many details omitted (e.g.,
target platform, desired features, visual design details, etc.). Thus,
specification alignment mechanisms should not only include ways to refine the
AI’s interpretation of the goal, but also include mechanisms for the AI to
request additional information or clarifications from the user (what Horvitz
describes as a “dialog” with the user when there is uncertainty (Horvitz,
1999)). Recent experiments with LLM-based agents provide examples of this
concept, with agents requesting clarifications from users before executing on
the user’s request (Osika, 2023).

####  3.1.3. Specification Alignment for Specialized (Narrow) AI

For narrow-purpose AI (i.e., AI intended to perform only one task), the
problem specification is fixed (e.g., an AI that detects and grades prostate
cancer has a fixed task specification). However, even in these cases, notions
of specification alignment can still be useful, as they suggest ways to design
the interaction to remind the user about what problem the AI is, and is not,
solving.

###  3.2. Process Alignment

Process alignment refers to the ability for the user to directly define,
modify, and/or influence the process by which the AI produces the intended
outcome. More concretely, process alignment makes the AI’s underlying process
accessible to the user to view and/or control. As an example, the PromptPaint
system enables users to control specific stages of a text-to-image AI’s
generation process (e.g., by changing the prompt the AI uses at specific steps
of the image generation process) (Chung and Adar, 2023).

We further divide process alignment into means alignment and control
alignment, where means alignment considers whether the means by which the AI
achieves the intended outcome are as desired (e.g., the means are safe, cost-
effective, aligned with human values (Russell et al., 2016), and aligned with
user needs or desires, such as a desire to maintain a sense of agency in the
process (Louie et al., 2020)), and control alignment considers whether the
system provides useful and usable user-centered control of the AI’s processes
(e.g., can the user directly control any individual operation the AI
performs).

While the goal of control alignment is to grant users the ability to more
directly control the AI, control alignment can also help address the needs of
specification alignment: One potential way to achieve specification alignment
is to present the actual actions the AI will perform. However, specification
alignment and control alignment are still distinct concepts, even if they’re
solved using a common mechanism: specification alignment defines the desired
outcome; how that is actually achieved (i.e., the process) can vary.

Figure 2. Surrogate processes. A surrogate process is an alternative way of
producing the same result as the AI. To create a surrogate process, the AI
produces output as normal (A). The original input and the AI’s output is then
sent to a system to reverse-engineer a process for producing the same output
(B). The resulting process is a surrogate process (C)—an alternative way for
producing the same output. The intent of a surrogate process is to increase
the ease with which the user can understand and control the process of
producing the given outcome, in lieu of reliable and effective mechanisms for
controlling the AI’s original process. In the ideal, the output produced by
the surrogate process is identical to that produced by the AI, but in reality,
there may be some loss in fidelity when creating a surrogate process to
approximate what the AI did.

Figure shows input provided to an AI to produce an output, with the input and
output sent to a process to reverse-engineer a new series of operations to
produce the same output from the given input.

####  3.2.1. Surrogate Processes, and When Process Alignment Can Occur

Like specification alignment, process alignment may be achieved before,
during, or after the AI has produced an output. While it may seem
counterintuitive to align on process after the AI has already acted, there are
several reasons this can be useful.

One motivation for aligning on process after an action has already been
performed is that developing usable mechanisms to reliably control an AI may
be challenging to implement. In these cases, it may be possible to enhance
control alignment by reverse-engineering a simplified, but controllable,
representation of the AI’s actual process (see Figure 2). We call this
reverse-engineered process a surrogate process222Our use of the term
“surrogate process” derives from the concept of a surrogate model, which
refers to a simplified version of a complex model. For example, techniques
such as LIME (Ribeiro et al., 2016) create simplified versions of more complex
models to assist in model interpretability.. Liu et al.’s system (Liu et al.,
2023) for synthesizing code provides an example of this strategy: After the AI
generates code from a natural language request, their system reverse-engineers
a pseudocode representation of the synthesized code that the user can edit.
Once the user has edited the pseudocode, the actual code can be regenerated.
In this case, the pseudocode is a surrogate for the actual code: The
pseudocode provide an accessible interface for understanding and controlling
the actual code synthesis process. In short, a surrogate process synthesizes a
new process to recreate the AI’s original end result, where that new process
employs representations and operations that are easily understood and
controllable by users.

Another motivation for post-output process alignment comes from cases where
the user does not have enough information to align on the means up front. As
an example, consider a generative AI platform that also indicates the data
sources that were most influential in producing an output (e.g., for reasons
of copyright, attribution, etc.). A user may wish to draw upon some sources
and avoid others, but they may not have enough knowledge a priori to indicate
which sources should be referenced and which should be avoided. In cases like
these, means alignment may only be possible after the first run of the
process. The branch of interpretability research focused on training data
attribution already provides some tools to help address needs like these (Koh
and Liang, 2020; Pruthi et al., 2020).

Figure 3. Process Gulf. How a person achieves an outcome can be quite
different from how an AI achieves the same outcome, leading to a Process Gulf.
Differences in process can make it challenging for users to control an AI’s
process, as users need to manipulate operations and representations different
from the ones they normally use.

Figure shows input provided to both a person and an AI, with them processing
that input in very different ways. The differences between their processes is
labeled the Process Gulf.

####  3.2.2. The Process Gulf

One challenge in achieving process alignment is that there can be qualitative
differences between how a user achieves an outcome and how an AI achieves the
same outcome. We call this the Process Gulf (Figure 3), an extension of the
concepts of the Gulfs of Execution and Evaluation introduced by Norman333The
Gulfs of Execution and Evaluation, respectively, describe the gulfs a user
must bridge to understand 1) how to use a user interface and 2) how to
evaluate its current state (Norman, 2002). (Norman, 2002). As an example of a
Process Gulf, a diffusion model for image generation incrementally transforms
an image of pure, statistical noise into a coherent image (Saharia et al.,
2022), an image creation process foreign to most people. When the user needs
to modify or more directly control the AI’s process, they must explicitly
bridge this Process Gulf, which can be challenging if the AI’s underlying
operations and data representations differ significantly from those the user
is familiar with.

As mentioned, the Process Gulf can be exacerbated by a lack of understanding
of the low-level operations an AI employs to produce an outcome. For example,
the area of mechanistic interpretability attempts to understand how an AI-
trained algorithm solves a problem. In this research area, it has been
observed that even a seemingly simple problem, such as explaining how a one
layer neural network learns to do modular arithmetic, requires significant
effort to understand (Pearce et al., 2023). Fortunately, because AI systems
can be understood at many levels of abstraction, understanding low-level
implementation details (e.g., the exact algorithm learned by an AI) is often
not needed to provide some form of meaningful control to end-users.
Additionally, strategies such as surrogate processes (described above) can
help bridge the Process Gulf.

###  3.3. Evaluation Support

Evaluation support recognizes the user’s need to assess the output of the AI.
For example, a code synthesis AI may translate the request “write a TODO app”
into dozens or hundreds of lines of code, which the user must assess with
respect to their goals. A system that provides evaluation support offers
capabilities to help users efficiently and effectively assess the system’s
output.

The depth of assessment required can vary, so we further differentiate between
two levels of evaluation support: verification support, which focuses on the
ability for users to verify that the output meets their objectives, and
comprehension support, which focuses on the ability to understand the output.
This latter requirement is a much stronger requirement than verification, but
an important one to aspire to for systems that rival or exceed users’
capabilities or expertise.

In contrast to specification and process alignment, evaluation support is not
concerned with aligning the AI and its actions, but a recognition that
successful completion of an interaction requires evaluating the AI’s output
with respect to the user’s goals (i.e., did the AI actually achieve the
intended outcome). As such, a human-AI interface aiming for a high degree of
alignment will benefit from capabilities that support evaluation.

Seen in the broader context of achieving a specific goal, evaluation support
can also viewed as providing assistance in understanding the state of the
problem after the AI performs some work: what does the overall solution look
like, what is needed next, etc. This focus on understanding the output of the
AI, as opposed to how the AI actually works internally, distinguishes this
concept from work in AI interpretability, which seeks to explain how the model
produced a particular outcome.

As an example of evaluation alignment, consider again the example of
synthesizing code for a TODO app. To aid evaluation, the system may:

  1. (1)

Comment the code.

  2. (2)

Create a natural language summary of the code produced.

  3. (3)

Create an architectural diagram of the code.

  4. (4)

Execute the code, showing the result of code execution.

Other explanations of the code are obviously possible, but the crucial idea
here is to help the user understand and navigate the output of the AI.

##  4\. Examples

In this section, we review a set of existing interactive AI systems to
demonstrate how user-centered dimensions of AI alignment can be used to both
describe interfaces and suggest opportunities for further design exploration
and/or research. In each system, it is useful to note how the alignment
mechanisms provide an enhanced user experience compared to a baseline
condition that lacks these mechanisms.

While these example systems employ AI that is less capable than forms of AI
often contemplated in AI alignment and safety research, they are nonetheless
representative of the types of AI systems many people now interact with on a
daily basis. Studying AI alignment in the context of these existing, real-
world systems (commercial and research) provides valuable insights into how to
holistically design for alignment now (i.e., at the level of both the AI and
the user interface); we expect lessons learned from these existing systems
will also inform the design of future systems employing far more capable AI.

###  4.1. Image Generation: Midjourney and PromptPaint

In this section, we describe the AI alignment mechanisms for two image
generation systems. Table 3 summarizes the alignment mechanisms for both
systems.

Table 3. Examples of AI alignment mechanisms for Midjourney and PromptPaint (Chung and Adar, 2023). Alignment Dimension |  Midjourney |  PromptPaint
---|---|---
Specification alignment mechanisms |  The “variations” command enables users to 1) choose an output image most similar to their desired output, and 2) request more images like that image. The “shorten” command highlights the most influential parts of the prompt, providing some assistance in revising the prompt to achieve the desired output. |  Users can blend prompts together using a paint palette metaphor, enabling them to specify the desired outcome.
Process alignment mechanisms |  Parameters like “chaos” provide some amount of process control. |  Users can change which prompt is in effect at different points in the image generation process.
Evaluation support mechanisms |  Images can be quickly scanned by users, but one could imagine providing additional evaluation mechanisms. For example, a visualization could display links between objects described in the text prompt and their locations in the generated image to help users verify everything was included in the image. |  As with Midjourney, no mechanisms seem to be available to specifically assist with evaluation.

####  4.1.1. Midjourney

Midjourney is a commercial system for producing images using text-to-image
models (Midjourney, [n. d.]). Its primary interface is text-based: Users enter
a prompt describing the intended image and multiple images are produced. Users
can influence the image generation process through parameters, such as
“chaos,” which affects the diversity of outputs generated. Once a set of
images has been generated, a user can choose one of the images and invoke an
operation to produce variations of that image. The documentation indicates
that this operation “maintains the general style and composition of the
selected image.”

To help learn how to craft an effective prompt, users can upload an image and
use the “describe” command to request four separate text-based prompts that
describe the image. There is also a “shorten” command to highlight which parts
of the prompt were most influential in producing the intended outcome.

Within the alignment dimensions introduced above, the text prompt constitutes
the specification to the system. In this interface, specification alignment is
largely supported through post hoc operations such as the “variations”
operation and the “shorten” command: The “variations” operation provides a
coarse means for users to refine their specification by requesting more images
like a chosen image. The “shorten” command doesn’t provide a means to directly
change a specification, but it does provide information that may be helpful in
revising the prompt (i.e., the specification).

For process alignment, parameters like “chaos” enable a degree of process
control, but these mechanisms are not interactive and no support is provided
to understand how a particular value will impact the generated images. There
do not appear to be any mechanisms to support means alignment, but one could
imagine specifying constraints such as generating images that draw from
particular facets of the training data, such as female artists.

In terms of evaluation support, images can be quickly reviewed by people,
lessening the need for explicit mechanisms to support evaluation. However, one
could imagine mechanisms that show links between the text inputs and the
corresponding regions of the image to help people verify that objects
specified in the prompt are also present in the rendered image.

####  4.1.2. PromptPaint

PromptPaint is a research system that introduces a number of mechanisms to
grant the user greater control over text-to-image AI (Chung and Adar, 2023).
Like Midjourney, the user provides a text-based prompt to generate an image.
However, the user can also specify multiple prompts and 1) “blend” or
interpolate between the prompts using a paint palette metaphor, or 2) request
that the system swap input prompts at different points of image generation
(for example, prompt A could be used for the first third of the image
generation process, and prompt B used for the remainder of the image
generation process). These abilities provide forms of both specification
alignment and control alignment. The ability to blend prompts using a paint
palette metaphor provides a form of specification alignment, while the ability
to swap prompts during image generation offers a form of control alignment.

Like Midjourney, no explicit support seems to be available to assist with
evaluation.

###  4.2. Coding: CoPilot and Liu et al.’s Spreadsheet Code Synthesis

We consider AI alignment mechanisms for two code synthesis systems, both of
which are summarized in Table 4.

Table 4. Examples of AI alignment mechanisms for CoPilot and Liu et al.’s spreadsheet system (Liu et al., 2023). Alignment Dimension |  CoPilot |  Liu et al.’s spreadsheet code synthesis
---|---|---
Specification alignment mechanisms |  Users can write specifications using comments, documentation, and/or code. Some support for specification alignment is available by virtue of the ability to cycle between different code suggestions. |  The system generates pseudocode representing the underlying code generated. This pseudocode represents the system’s interpretation of the user’s request, and thus acting as a specification. Users can edit this pseudocode to change the synthesized code, thus increasing specification alignment.
Process alignment mechanisms |  The system offers an option to block suggestions matching public code, providing some degree of control over its code generation process (means alignment). |  The generated pseudocode not only acts as a specification, but a representation of the process the system will follow to achieve the user’s stated outcome. Because edits to the pseudocode alter the generated code, editing of the pseudocode assists in attaining process alignment.
Evaluation support mechanisms |  Evaluation is aided if the system generates comments for the code it produces, but at present, there does not seem to be a way for the user to control this aspect of the code generation process. |  The pseudocode provides an alternative representation of the underlying code that is ideally easier to understand (and thus evaluate) than the actual code.

####  4.2.1. CoPilot

CoPilot is a commercial product that synthesizes code suggestions from
surrounding comments and code (GitHub, [n. d.]). Code suggestions are
automatically generated as users interact with the editor, and users can cycle
through code suggestions to choose the suggestion closest to their needs.

In this system, the process of providing a specification can be considered to
be both explicit and implicit (i.e., inferred). Users can provide an explicit
specification by writing comments, documentation, and/or code that describe
the desired code. Implicit specifications are inferred by the model
automatically generating suggestions as the user writes and edits code.

CoPilot offers the ability to block suggestions matching public code, which
provides some degree of control over its code generation process (process
alignment, specifically, means alignment). A basic form of evaluation support
can be achieved if the system comments the code that it synthesizes.

####  4.2.2. Liu et al.’s Spreadsheet Code Synthesis

Liu et al.’s natural language code synthesis system transforms natural
language requests into equivalent spreadsheet operations (Liu et al., 2023).
However, in contrast to systems like CoPilot, their system does not expose the
generated code to the user. Instead, it generates code, then deterministically
transforms that code back into pseudocode consisting of a series of natural
language steps. Users can directly edit this pseudocode then resubmit the
pseudocode to regenerate the actual code.

The automatically generated pseudocode can be seen as helping the user attain
both specification and control alignment: The pseudocode reflects the system’s
interpretation of the user’s request, but also the process it will follow to
achieve the user’s stated goal. This strategy of echoing back an
interpretation of the user’s input is similar in spirit to Midjourney’s
“describe” and “shorten” commands; in both systems, these echoed
specifications can help users learn effective strategies for prompting the
models. This reverse-engineering of the AI’s process also represents a
surrogate process—a more accessible representation of the AI’s actual,
underlying process. Finally, because the pesudocode helps users understand the
model’s output, it also provides evaluation support.

In a user study conducted by Liu et al. (Liu et al., 2023), participants found
the pseudocode to be useful to understand what the actual code was doing, and
also helpful for debugging. These findings suggest the value of process
alignment mechanisms in the design of interactive human-AI systems.

##  5\. Discussion

In this section, we discuss implications of user-centered perspectives on AI
alignment, along with promising areas for future work.

###  5.1. Appropriate Levels of Alignment: Helping, Not Hindering Task
Progression

The notion of achieving AI alignment suggests a process that needs to be
completed before any action can be taken. However, in human-AI interactions,
it can be cumbersome or impractical to reach alignment on everything before an
action is taken if the costs of performing a (potentially incorrect) action
are negligible. For example, for code synthesis, rather than requiring a
detailed specification of what code to produce, it may be more efficient to
simply generate the code and make refinements to 1) the initial request, 2)
the resulting code, or 3) some representation of the generated code (e.g.,
pseudocode). CoPilot employs this strategy, with an interface design that
allows the user to dismiss or ignore irrelevant or unneeded suggestions.

With these points in mind, the alignment objectives described in this paper
should not be interpreted as an argument for reaching complete alignment
before the AI takes any action. As mentioned above, this may not be practical
or provide a positive user experience. Instead, what these issues highlight is
a design challenge for creating appropriately aligned AI systems: How can we
design interfaces that enable users to reach appropriate levels of alignment
with the AI, without negatively affecting the user experience (and without the
potential for negative side effects)? CoPilot and Liu et al.’s code synthesis
system both provide examples of system designs that strike a balance between
moving forward and providing facilities to make corrections as necessary (Liu
et al., 2023).

###  5.2. The Challenge of Supporting Evaluation

As AI capabilities increase, the need for users to be able to verify and
understand the AI’s output also increases. This need has been the focus of
significant research in the AI alignment community in the context of training
aligned AI. For example, Irving et al. consider how self-debate could be used
to train AI that produces true, useful information (Irving et al., 2018). With
the similar goal of creating aligned AI, Leike et al. consider how to evaluate
AI in problem domains where it is difficult for people to evaluate the
correctness of a solution (e.g., because of complexity, lack of expertise,
etc.) (Leike et al., 2018). Their proposed strategy of “recursive reward
modeling” includes the notion of training agents that help people evaluate
individual facets of the AI’s output, rather than asking people to evaluate
the entire output all at once (e.g., asking a person to evaluate summaries of
character arcs in a novel, as opposed to asking them to evaluate the entire
novel). To improve the quality of model outputs when performing chain-of-
thought reasoning (Wei et al., 2023), Lightman et al. showed that process
supervision (i.e., improving the model’s reasoning process) can lead to better
output (Lightman et al., 2023).

While this prior work is largely concerned with optimizing AI performance and
alignment during AI training, the methods developed have direct applicability
to the design of mechanisms intended to support evaluation of AI output in
interactive AI systems. For example, Irving et al.’s self-debate strategy and
Leike et al.’s recursive reward modeling will both yield AI that produce
outputs that are easier for people to evaluate when interactively using AI.
Similarly, an improved and more reliable chain-of-thought reasoning process
can be exposed to the user to help them evaluate the outcome by seeing how it
was derived, step-by-step.

This prior work provides a useful foundation and inspiration for evaluation
support mechanisms, but there are significant opportunities for additional
research into how to support people in reliably and efficiently verifying and
understanding AI outputs.

###  5.3. Designing for Continual Learning and Machine Teaching

To date, most AI alignment interfaces have been manually designed, with
largely fixed feature sets. To produce useful and usable AI alignment
mechanisms, there are clear opportunities to design systems for continual
learning, as has been noted in both the HCI (e.g., (Horvitz, 1999; Amershi et
al., 2019)) and alignment literature (e.g., (Leike et al., 2018)). Many real-
world systems are already designed to continually learn and improve (such as
recommender systems), but tools and methods that further lower the barrier to
designing and implementing systems that can learn from experience will help
accelerate work in this space. In particular, AI systems that can continually
experiment with their user interface designs (e.g., through reinforcement
learning strategies (Sutton and Barto, 2018) and dynamic generation of user
interfaces or user interface components (Jiang et al., 2023)) enable the AI to
continually improve its alignment mechanisms.

As an example of how continually learning systems could be valuable, Liu et
al.’s system for spreadsheets generates pseudocode that represents the AI’s
synthesized code (Liu et al., 2023). In their implementation, they manually
wrote the functionality to generate the pseudocode, which required making
decisions about the style and content of the generated pseudocode. However,
one could conceive of a system that automatically tests these different
variations to discover the most useful and usable formats for the user. For
example, the system could test different coding styles, whether to include
comments, how detailed to make generated comments, etc. Extending these ideas
to the interface design itself, one could conceive of the system experimenting
with different ways of representing the synthesized code to the user to
increase understanding. For example, the system could test whether to show
pseudocode, the actual code, or both; or it could test showing animations of
the code executing step-by-step, showing intermediate results to aid
evaluation. Systems that can not only test alternative interface designs, but
also generate the candidates to test, seem like they could be particularly
useful in this context.

Despite the potential value of continually evolving AI interfaces, research in
the adaptive user interfaces community also highlights the challenges in
creating adaptive users interfaces that do not incur significant costs to
users as the interface designs change over time (e.g., due to learning costs,
frustration, etc.) (Akiki et al., 2014; Bunt et al., 2004). As such, creating
AI systems that can continually learn how to best interact with users, without
introducing significant overhead to the learning process, remains an open
research problem.

In addition to the strategy of designing for continual learning, significant
research has also been conducted in the area of interactive machine learning
and machine teaching, where users explicitly and interactively define the
desired behavior (Amershi et al., 2014; Fails and Olsen, 2003). These threads
of research could be similarly leveraged to introduce mechanisms for users to
interactively teach the AI about their specific needs and preferences in AI
alignment processes.

###  5.4. Human-Optimized Interfaces for AI Alignment

In considering the design of interactive AI systems, it can be helpful to
distinguish between AI-native and human-optimized operations and
representations. AI-native representations and operations include things like
word embeddings (Mikolov et al., 2013), diffusion maps for image models
(Saharia et al., 2022), and low-level operations such as matrix and vector
manipulations. Human-optimized formats and operations focus on the problem of
providing representations and processes tuned to the perceptual, motor, and
cognitive capabilities of users. Returning to the notion of interfaces that
can continually improve over time, any such improvements will be a result of a
tight coupling between the interface and the perceptual, motor, and cognitive
capabilities of the users.

The HCI and machine learning communities have both made contributions to this
problem of providing human-optimized formats and controls for users. As
examples in the HCI community, Cai et al. developed a technique called
“refine-by-concept” that enables users to refine image search results by
indicating that search results should have more or less of a given high-level
concept (e.g., in medical image search, more or less of a specific biological
concept, such as “fused glands”) (Cai et al., 2019). The PromptPaint system in
the Examples section above employs the metaphor of a painter’s palette to make
the concept of mixing prompts accessible (Chung and Adar, 2023). In the
machine learning community, instruction tuning of LLMs provides an example of
optimizing AI to provide an interface better matched to human needs and
capabilities (Wei et al., 2022).

While it can be useful to draw a distinction between AI-native or human-
optimized control mechanisms, it should also be noted that AI-native
mechanisms and data formats can also lead to novel and usable interface
mechanisms. For example, Cai et al.’s “refine-by-concept” technique and
PromptPaint’s ability to blend prompts both leverage low-level, learned
representations of images (image embeddings) (Cai et al., 2019; Chung and
Adar, 2023)).

As the examples cited above indicate, there is already notable work examining
the problem of developing human-optimized representations and tools for
interacting with AI. However, there continue to be significant opportunities
for further research into human-optimized representations and processes to
enhance human-AI interaction and AI alignment, with approaches that
continually improve to match innate perceptual, motor, and cognitive
capabilities being particularly promising.

###  5.5. AI Affordances

A common struggle with modern generative AI is understanding what it can
actually do—what its capabilities are (Jiang et al., 2022b; Zamfirescu-Pereira
et al., 2023b; Yang et al., 2020; Lee et al., 2022). In traditional interface
design, interface affordances can help communicate the system’s capabilities
(Norman, 2002). However, the significantly broader capabilities of modern
generative AI, and particularly language-driven AI, complicates the process of
offering affordances. For example, it is not feasible to create a
comprehensive menu system for all of the capabilities of an LLM. At present, a
common approach to this problem is to provide guides, such as an LLM prompting
guide (e.g., (Shieh, [n. d.])). A system may also provide a set of optional
parameters, such as Midjourney’s “style” parameter, that help guide users to
specific outcomes.

Another strategy currently being applied is to instill the system with a sense
of what it can or cannot do, either explicitly or through learning. With this
information, the AI can help guide the user during the specification alignment
process by providing suggestions of what it can do, or by indicating what it
cannot do. This approach is widely used by researchers and developers of
chatbots, often to manage potential tensions between the end users’ task
specification and the system designers’ specification of intended use. For
instance, chatbots are tuned to provide users with descriptions for what the
AI cannot do, and suggest different behaviors that might be helpful to the
user instead (Bai et al., 2022). As an example, when prompted to write an
essay, a chatbot might respond with something like: “I can’t write a college
admissions essay for you, but if you provide me with the text of your own
essay, I can proofread it for grammar and make suggestions.”

With advances in parameter efficient tuning methods, such as LoRA (Hu et al.,
2021), the ability to design these types of system affordances into the
specification alignment process is becoming more widely available, cheaper,
and easier to do. For example, these methods have already been shown effective
at reducing the work and scale of data needed to make new safety classifiers
for LLMs (Mozes et al., 2023), which are often used to filter the outputs of
chatbots to stop users from being able to use them in unintended ways.

Revisiting the theme of continual learning and improvement, there are similar
opportunities for designing AI that can learn the boundaries of what it can
and cannot do from interactions with users (as opposed to requiring the AI’s
designers to embed all this knowledge from the start). As a concrete example,
it was quickly discovered that early LLMs had significant limitations in
performing arbitrary calculations (Brown et al., 2020). A system that learns
through user interactions could conceivably learn limitations of its own
capabilities from its failures, and warn users in future interactions.

In general, the problem of understanding and controlling the affordances for
AI is still an open problem, but an important problem to address in providing
user-centered AI alignment mechanisms.

###  5.6. Alignment with Multiple Parties

Our discussions of achieving alignment have assumed a 1:1 dynamic with a
single user interacting with a single AI. However, it is also useful to
consider alignment for interactions that include multiple entities, as this
can significantly alter interaction requirements (e.g., see Suh et al.’s study
examining how collaborations can be affected when an AI is introduced into a
music creation task involving two people (Suh et al., 2021)). In these
contexts, the same alignment goals still exist, with the added dimension of
attaining alignment across all parties involved. This is an area worthy of
additional research, especially as AI becomes part of more and more
activities.

###  5.7. Centering on the User and Their Goals

This paper has identified common areas of research between the AI alignment
and HCI communities, and argued for the value in considering alignment in the
context of users’ moment-to-moment interactions with the AI. In discussions of
AI alignment, AI is often a focal point (its design, its implementation, its
training, etc.). However, in the interactive AI systems considered in this
paper, AI alignment also explicitly involves a user who has a goal to achieve
with an AI. In this context, the interface’s design is obviously a critical
component in the alignment process. With this in mind, the alignment
dimensions enumerated in this paper provide a reminder to ask the question of
how to design AI and its user interface to help users solve problems and
achieve goals of interest to them.

##  6\. Conclusion

This paper introduces a basic framework and vocabulary for considering the
problem of AI alignment in the context of interactive AI systems and their
user interface designs. Specifically, we map concepts of AI alignment onto a
basic three step interaction cycle to highlight the concepts of specification
alignment, process alignment, and evaluation support. Specification alignment
can be further considered in terms of the outcome specification and
specification constraints and desiderata. Process alignment can be faceted
into the concepts of means alignment (how an outcome is produced) and control
alignment (the ability to modify any part of the AI’s process). Evaluation
support can be additionally considered from the perspectives of verification
and comprehension, with the latter being a much stronger requirement to meet.

These dimensions are both descriptive and prescriptive: They provide a
vocabulary for describing mechanisms intended to increase alignment with
interactive AI, and they suggest areas worthy of further research (e.g.,
mechanisms to help people assess and understand the AI’s output). These
dimensions also provide a common foundation for related communities to join
together to improve the design of aligned and interactively alignable human-AI
systems that are useful, usable, and enjoyable.

We also introduce the notion of a Process Gulf, which describes the gulf that
can arise between a user and the AI due to differences in how they produce an
outcome. When the user needs to modify or more directly control the AI’s
process, they must explicitly bridge this Process Gulf, which can be
challenging if the AI’s underlying operations and data representations
significantly differ from those the user is familiar with.

A surrogate process can help bridge the Process Gulf and make AI more
controllable. A surrogate process is a separately derived, but more easily
controllable representation of the AI’s actual process for producing a given
outcome. Surrogate processes enable users to retrospectively modify the
process when the AI’s output does not fully meet their needs.

Finally, we identify a number of areas worthy of future research, including
interactive alignment mechanisms that can continually improve to match the
perceptual, motor, and cognitive needs of users; interactive mechanisms to
better support evaluation of AI outputs; and the need for additional research
into interactive alignment for multi-user, interactive AI systems.

We conclude by observing that there is considerable research and discourse in
the AI alignment community, and growing interest in the concept of alignment
in the HCI community. There are clear opportunities to further distill ideas
in AI alignment into concrete implementations in interactive AI systems.

###### Acknowledgements.

We thank Iason Gabriel, Geoffrey Irving, Michael Liu, Carrie Cai, and Adam
Pearce for helpful discussions and feedback. We also thank Aaron Donsbach for
producing our figures.

## References

  * (1)
  * Agostinelli et al. (2023) Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023\.  MusicLM: Generating Music From Text.  arXiv:2301.11325 [cs.SD]
  * AI ([n. d.]) Stability AI. [n. d.].  Stable Doodle.  https://clipdrop.co/stable-doodle.  Accessed: 2023-08-31.
  * Akiki et al. (2014) Pierre A. Akiki, Arosha K. Bandara, and Yijun Yu. 2014.  Adaptive Model-Driven User Interface Development Systems.  _ACM Comput. Surv._ 47, 1, Article 9 (may 2014), 33 pages.  https://doi.org/10.1145/2597999
  * Amershi et al. (2014) Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014.  Power to the People: The Role of Humans in Interactive Machine Learning.  _AI Magazine_ 35, 4 (Dec. 2014), 105–120.  https://doi.org/10.1609/aimag.v35i4.2513
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022.  Constitutional AI: Harmlessness from AI Feedback.  arXiv:2212.08073 [cs.CL]
  * Bommasani et al. (2022) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022.  On the Opportunities and Risks of Foundation Models.  arXiv:2108.07258 [cs.LG]
  * Bostrom (2003) Nick Bostrom. 2003\.  Ethical Issues in Advanced Artificial Intelligence.  (01 2003).
  * Bostrom (2014) Nick Bostrom. 2014\.  _Superintelligence: Paths, Dangers, Strategies_ (1st ed.).  Oxford University Press, Inc., USA.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023\.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  arXiv:2303.12712 [cs.CL]
  * Bunt et al. (2004) Andrea Bunt, Cristina Conati, and Joanna McGrenere. 2004\.  What Role Can Adaptive Support Play in an Adaptable System?. In _Proceedings of the 9th International Conference on Intelligent User Interfaces_ (Funchal, Madeira, Portugal) _(IUI ’04)_. Association for Computing Machinery, New York, NY, USA, 117–124.  https://doi.org/10.1145/964442.964465
  * Buschek et al. (2021) Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021\.  Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems.  arXiv:2104.00358 [cs.HC]
  * Cai et al. (2019) Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019.  Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3290605.3300234
  * Chang et al. (2023) Minsuk Chang, Stefania Druga, Alexander J. Fiannaca, Pedro Vergani, Chinmay Kulkarni, Carrie J Cai, and Michael Terry. 2023.  The Prompt Artists. In _Proceedings of the 15th Conference on Creativity and Cognition_ (Virtual Event, USA) _(C &C ’23)_. Association for Computing Machinery, New York, NY, USA, 75–87.  https://doi.org/10.1145/3591196.3593515
  * Christian (2020) B. Christian. 2020\.  _The Alignment Problem: Machine Learning and Human Values_.  WW Norton.  https://books.google.com/books?id=VmJIzQEACAAJ
  * Christiano ([n. d.]) Paul Christiano. [n. d.].  Clarifying “AI Alignment”.  https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment.  Accessed: 2023-08-23.
  * Chung and Adar (2023) John Joon Young Chung and Eytan Adar. 2023.  PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions. In _Proceedings of UIST 2023_. Association for Computing Machinery, New York, NY, USA, 17 pages.  https://doi.org/10.1145/3586183.3606777
  * Fails and Olsen (2003) Jerry Alan Fails and Dan R. Olsen. 2003.  Interactive Machine Learning. In _Proceedings of the 8th International Conference on Intelligent User Interfaces_ (Miami, Florida, USA) _(IUI ’03)_. Association for Computing Machinery, New York, NY, USA, 39–45.  https://doi.org/10.1145/604045.604056
  * Gabriel (2020) Iason Gabriel. 2020\.  Artificial Intelligence, Values, and Alignment.  _Minds and Machines_ 30, 3 (sep 2020), 411–437.  https://doi.org/10.1007/s11023-020-09539-2
  * GitHub ([n. d.]) GitHub. [n. d.].  GitHub CoPilot.  https://github.com/features/copilot/.  Accessed: 2023-08-31.
  * Google ([n. d.]) Google. [n. d.].  Bard.  https://bard.corp.google.com/.  Accessed: 2023-09-26.
  * Hales (2005) TC Hales. 2005.  A proof of the Kepler conjecture.  _Annals of Mathematics_ 162 (11 2005), 1065–1185.
  * Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. 2022.  Imagen Video: High Definition Video Generation with Diffusion Models.  arXiv:2210.02303 [cs.CV]
  * Horvitz (1999) Eric Horvitz. 1999\.  Principles of Mixed-Initiative User Interfaces. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Pittsburgh, Pennsylvania, USA) _(CHI ’99)_. Association for Computing Machinery, New York, NY, USA, 159–166.  https://doi.org/10.1145/302979.303030
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021\.  LoRA: Low-Rank Adaptation of Large Language Models.  arXiv:2106.09685 [cs.CL]
  * Irving et al. (2018) Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018\.  AI safety via debate.  arXiv:1805.00899 [stat.ML]
  * Jiang et al. (2022a) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022a.  PromptMaker: Prompt-Based Prototyping with Large Language Models. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages.  https://doi.org/10.1145/3491101.3503564
  * Jiang et al. (2022b) Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022b.  Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 386, 19 pages.  https://doi.org/10.1145/3491102.3501870
  * Jiang et al. (2023) Yue Jiang, Yuwen Lu, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, and Wolfgang Stuerzlinger. 2023.  The Future of Computational Approaches for Understanding and Adapting User Interfaces. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI EA ’23)_. Association for Computing Machinery, New York, NY, USA, Article 367, 5 pages.  https://doi.org/10.1145/3544549.3573805
  * Kenton et al. (2021) Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021\.  Alignment of Language Agents.  arXiv:2103.14659 [cs.AI]
  * Kim (2022) Been Kim. 2022.  Beyond interpretability: developing a language to shape our relationships with AI.  https://medium.com/@beenkim/beyond-interpretability-4bf03bbd9394.  Accessed: 2023-08-23.
  * Kirchner et al. (2022) Jan H. Kirchner, Logan Smith, Jacques Thibodeau, Kyle McDonell, and Laria Reynolds. 2022.  Researching Alignment Research: Unsupervised Analysis.  arXiv:2206.02841 [cs.CY]
  * Koh and Liang (2020) Pang Wei Koh and Percy Liang. 2020.  Understanding Black-box Predictions via Influence Functions.  arXiv:1703.04730 [stat.ML]
  * Krakovna et al. (2020) Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. 2020\.  Specification gaming: the flip side of AI ingenuity.  https://deepmindsafetyresearch.medium.com/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4.  Accessed: 2023-08-29.
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 388, 19 pages.  https://doi.org/10.1145/3491102.3502030
  * Leike et al. (2018) Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018\.  Scalable agent alignment via reward modeling: a research direction.  arXiv:1811.07871 [cs.LG]
  * Leike et al. (2017) Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017\.  AI Safety Gridworlds.  arXiv:1711.09883 [cs.LG]
  * Leike et al. ([n. d.]) Jan Leike, John Schulman, and Jeffrey Wu. [n. d.].  Our approach to alignment research.  https://https://openai.com/blog/our-approach-to-alignment-research.  Accessed: 2023-08-23.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023\.  Let’s Verify Step by Step.  arXiv:2305.20050 [cs.LG]
  * Liu et al. (2023) Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023\.  “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 598, 31 pages.  https://doi.org/10.1145/3544548.3580817
  * Louie et al. (2020) Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J. Cai. 2020.  Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376739
  * Menick (2016) John Menick. 2016\.  Move 37: Artificial Intelligence, Randomness, and Creativity.  https://www.johnmenick.com/writing/move-37-alpha-go-deep-mind.html.  Accessed: 2023-08-23.
  * Midjourney ([n. d.]) Midjourney. [n. d.].  Midjourney.  https://www.midjourney.com/.  Accessed: 2023-08-31.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality. In _Advances in Neural Information Processing Systems_ , C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (Eds.), Vol. 26. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf
  * Morris (2023) Meredith Ringel Morris. 2023\.  Scientists’ Perspectives on the Potential for Generative AI in their Fields.  arXiv:2304.01420 [cs.CY]
  * Mozes et al. (2023) Maximilian Mozes, Jessica Hoffmann, Katrin Tomanek, Muhamed Kouate, Nithum Thain, Ann Yuan, Tolga Bolukbasi, and Lucas Dixon. 2023\.  Towards Agile Text Classifiers for Everyone.  arXiv:2302.06541 [cs.CL]
  * Nielsen (2023) Jakob Nielsen. 2023\.  AI: First New UI Paradigm in 60 Years”.  https://www.nngroup.com/articles/ai-paradigm/.  Accessed: 2023-08-23.
  * Norman (2002) Donald A. Norman. 2002\.  _The Design of Everyday Things_.  Basic Books, Inc., USA.
  * OpenAI ([n. d.]) OpenAI. [n. d.].  ChatGPT.  https://chat.openai.com/.  Accessed: 2023-09-26.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Ortega et al. ([n. d.]) Pedro A. Ortega, Vishal Maini, and the DeepMind safety team. [n. d.].  Building safe artificial intelligence: specification, robustness, and assurance.  https://deepmindsafetyresearch.medium.com/building-safe-artificial-intelligence-52f5f75058f1.  Accessed: 2023-08-23.
  * Osika (2023) Anton Osika. 2023\.  GPT Engineer.  https://github.com/AntonOsika/gpt-engineer.  Accessed: 2023-08-25.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022\.  Training language models to follow instructions with human feedback.  arXiv:2203.02155 [cs.CL]
  * Pearce et al. (2023) Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg, and Lucas Dixon. 2023.  Do Machine Learning Models Memorize or Generalize?  https://pair.withgoogle.com/explorables/grokking/
  * Pruthi et al. (2020) Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. 2020.  Estimating Training Data Influence by Tracing Gradient Descent.  arXiv:2002.08484 [cs.LG]
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  arXiv:2204.06125 [cs.CV]
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Francisco, California, USA) _(KDD ’16)_. Association for Computing Machinery, New York, NY, USA, 1135–1144.  https://doi.org/10.1145/2939672.2939778
  * Russell et al. (2016) Stuart Russell, Daniel Dewey, and Max Tegmark. 2016\.  Research Priorities for Robust and Beneficial Artificial Intelligence.  arXiv:1602.03506 [cs.AI]
  * Russell and Norvig (2009) Stuart Russell and Peter Norvig. 2009.  _Artificial Intelligence: A Modern Approach_ (3rd ed.).  Prentice Hall Press, USA.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  arXiv:2205.11487 [cs.CV]
  * Sanner et al. (2023) Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, and Lucas Dixon. 2023.  Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences.  arXiv:2307.14225 [cs.IR]
  * Shieh ([n. d.]) Jessica Shieh. [n. d.].  Best practices for prompt engineering with OpenAI API.  https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api.  Accessed: 2023-08-31.
  * Suh et al. (2021) Minhyang (Mia) Suh, Emily Youngblom, Michael Terry, and Carrie J Cai. 2021.  AI as Social Glue: Uncovering the Roles of Deep Generative AI during Social Music Composition. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 582, 11 pages.  https://doi.org/10.1145/3411764.3445219
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA.
  * Turing (1936) Alan M. Turing. 1936\.  On Computable Numbers, with an Application to the Entscheidungsproblem.  _Proceedings of the London Mathematical Society_ 2, 42 (1936), 230–265.  http://www.cs.helsinki.fi/u/gionis/cc05/OnComputableNumbers.pdf
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned Language Models are Zero-Shot Learners. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=gEZrGCozdqR
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 [cs.CL]
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376301
  * Yudkowsky (2001) Eliezer Yudkowsky. 2001\.  _Coherent Extrapolated Volition_.  The Singularity Institute.
  * Zamfirescu-Pereira et al. (2023a) J.D. Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023a.  Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 2206–2220.  https://doi.org/10.1145/3563657.3596138
  * Zamfirescu-Pereira et al. (2023b) J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023b.  Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages.  https://doi.org/10.1145/3544548.3581388