# Evaluating Human-Language Model Interaction

Mina Lee⋆ Megha Srivastava Amelia Hardy John Thickstun
Esin Durmus Ashwin Paranjape Ines Gerard-Ursin§ Xiang Lisa Li
Faisal Ladhak Frieda Rong Rose E. Wang Minae Kwon
Joon Sung Park Hancheng Cao Tony Lee
Rishi Bommasani Michael Bernstein Percy Liang⋆

Stanford University §Imperial College London

###### Abstract

Many real-world applications of language models (LMs), such as code
autocomplete and writing assistance, involve human-LM _interaction_. However,
the main LM benchmarks are _non-interactive_ in that a system produces output
without human involvement. To evaluate human-LM interaction, we develop a new
framework, Human-AI Language-based Interaction Evaluation (HALIE), that
expands non-interactive evaluation along three dimensions, capturing (i) the
interactive process, not only the final output; (ii) the first-person
subjective experience, not just a third-party assessment; and (iii) notions of
preference beyond quality. We then design five tasks ranging from goal-
oriented to open-ended to capture different forms of interaction. On four
state-of-the-art LMs (three variants of OpenAI’s GPT-3 and AI21’s J1-Jumbo),
we find that non-interactive performance does not always result in better
human-LM interaction and that first-person and third-party metrics can
diverge, suggesting the importance of examining the nuances of human-LM
interaction.

##  1 Introduction

⋆Direct correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>full contributions in Appendix A.

Language models (LMs) have rapidly advanced, demonstrating unprecedented
capabilities for generation and striking generality for tackling a wide range
of tasks (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022).
However, these models are at present primarily evaluated _non-interactively_ :
given an input text, a model generates a completion with the focus solely on
the quality of the completion. Almost all benchmarks, even those with a
diverse range of tasks such as BIG-Bench (Srivastava et al., 2022), GEM
(Gehrmann et al., 2021), and HELM (Liang et al., 2022), encode this non-
interactive view.111 There are specific tasks such as dialogue that are
inherently interactive (Paranjape et al., 2020; Thoppilan et al., 2022b;
Shuster et al., 2022), but we are interested in the general capabilities of
LMs.

Figure 1:  Dimensions in human-LM interaction evaluation. We propose a
framework, HALIE, that expands on non-interactive evaluation along three
dimensions: (i) we capture the full _process_ in addition to the final
_output_ (targets); (ii) we capture the _first-person_ subjective experience
of users interacting with the LM in addition to the perspective of a _third-
party_ (perspectives), and (iii) we consider notions of _preference_ beyond
_quality_ (criteria). These dimensions interact to define the full space of
evaluation metrics (i.e., a 2x2x2 cube  ), which goes beyond standard, non-
interactive evaluation (i.e., gray cell).  Figure 2:  Five tasks and human-LM
interaction in the context of accomplishing the tasks. We study tasks ranging
from goal-oriented to open-ended and cover the common usages of LMs reported
by Ouyang et al. (2022). When designing human-LM interaction for these tasks,
we make use of various interaction styles, such as strict turn-taking (e.g.,
social dialogue) and iterative query reformulation (e.g., question answering).

Our goal is to evaluate human-LM interaction instead of just model
completions. LMs are already deployed in _interactive_ settings, where humans
work with them to brainstorm ideas (e.g., Jasper, Copy.ai), paraphrase
sentences (e.g., Wordtune, QuillBot), reformulate queries (Nogueira and Cho,
2017), autocomplete sentences (Chen et al., 2019), write code (e.g., Coilot,
TabNine), and so forth.222 This interactive use of LMs aligns with our
ultimate goal of _augmenting_ human capabilities rather than _automating_ them
(i.e., intelligent augmentation), reflecting views by Engelbart (1962);
Skagestad (1993, 1996); Shneiderman and Maes (1997); Horvitz (1999); Hassani
et al. (2020); Brynjolfsson (2022); Shneiderman (2022).  We anticipate that
the adoption rate will continue to accelerate as LMs become more capable and
novel use cases are discovered (Bommasani et al., 2021, §2.5).333 For
instance, over 300 applications were built within a year of the release of
GPT-3 (OpenAI, 2021), and ChatGPT (Schulman et al., 2022) amassed 1 million
users in five days after launching (New York Times, 2022).  But as a
community, if we explicitly or implicitly optimize for non-interactive
performance, will this improve interactive performance?

We develop an evaluation framework, Human-AI Language-based Interaction
Evaluation (HALIE) that puts interaction at the center. First, we make
explicit that end users interact with a _system_ (as opposed to a raw LM),
which includes a user interface (UI), the system logic which constructs
prompts and invokes the LM. The distinction between LM and system is important
because, unlike non-interactive evaluation, interactive evaluation is
fundamentally a function of the user, the LM, and the system, which acts like
the glue. HALIE formally defines the system logic with states and actions. A
_state_ of the system is textual contents in the UI (e.g., dialogue history
and the current text in a user input box), and an _action_ is taken by a user
(e.g., clicking the “send” button or typing into a text box) through the UI,
which subsequently triggers the system to update the state. We define an
_interaction trace_ to be the resulting sequence of state-action pairs.

To evaluate human-LM interaction, HALIE defines metrics on interaction traces,
which are organized along the following three dimensions (Figure 1)—targets,
perspectives, and criteria: (1) _targets_ include more than just the final
output, and cover the entire interaction process (e.g., user queries and
edits); (2) _perspectives_ are not limited to third parties, but the users who
interact with LMs to capture first-person experiences; and (3) _criteria_
include not only quality (e.g., accuracy), but also preference (i.e.,
attitudinal measures of humans such as enjoyment). In contrast, non-
interactive benchmarking devises automatic metrics or third-party human
annotations (perspectives) to solely evaluate the quality (criteria) of
outputs (targets).

We design five tasks ranging from goal-oriented to open-ended (Figure 2) to
capture a variety of different interactions, and construct an interactive
system for each task by defining states and actions. With these systems, We
evaluated four state-of-the-art LMs: three variants of OpenAI’s GPT-3 (Brown
et al., 2020; Ouyang et al., 2022)—TextDavinci (text-davinci-001), TextBabbage
(text-babbage-001), and Davinci (davinci)—and AI21’s Jurassic-1 (Lieber et
al., 2021)—Jumbo (j1-jumbo). We choose these models to study the impact of
model size (TextDavinci vs. TextBabbage), instruction tuning444 In this paper,
instruction tuning refers to supervised fine-tuning LMs on human-written
demonstrations and highly rated model completions (OpenAI, 2022).
(TextDavinci vs. Davinci), and implementation details regarding training data,
architecture, and other factors (Davinci vs. Jumbo). While the impact of these
differences on non-interactive benchmark performance has been studied quite
extensively (Brown et al., 2020; Ouyang et al., 2022; Liang et al., 2022), we
want to study their effect in interactive settings.

From the 1015 interaction traces we collected, we observe that better non-
interactive performance does not always lead to better human-LM interaction
(e.g., Section 3.2). Furthermore, sometimes a subset of quality metrics might
overpower other metrics when it comes to user preference (e.g., Section 3.1),
or helpfulness perceived by users could exceed actual performance improvements
(e.g., Section 3.3). We also find that a model that achieves the best
performance based on third-party perspectives might be the worst based on
first-person perspectives (e.g., Section 3.4), and visa versa (e.g., Section
3.5). We release our interaction traces, their replay links, and system
interfaces at https://github.com/minggg/halie.

##  2 Framework

In this section, we introduce our framework, HALIE, for evaluating human-LM
interaction. Concretely, we describe tasks and system construction for
studying human-LM interaction, as well as interaction traces that we use to
represent the interaction process. Lastly, we propose dimensions and metrics
for evaluating human-LM interaction.

###  2.1 Solving tasks interactively

We study human-LM interaction in the context of _tasks_ , which provide a
structured context for facilitating and evaluating these interactions. Figure
2 shows five tasks we study in this paper: social dialogue, question
answering, crossword puzzles, text summarization, and metaphor generation (see
Section 3 for a detailed description of each task). The tasks span the
spectrum from goal-oriented (e.g., question answering) to open-ended tasks
(e.g., metaphor generation). They also have high coverage on the common usages
of LMs reported by Ouyang et al. (2022) (e.g., generation, open QA, and
brainstorming), showing that the selection captures how users currently
interact with LMs.

###  2.2 Constructing an interactive system

For each task, we build a system that facilitates human-LM interaction in the
context of the task. There are many ways to design the system based on the
same LM: Users could query an LM and edit the output (i.e., post-editing),
reformulate their queries based on the output (i.e., query reformulation), or
query an LM and select one of the outputs (i.e., suggestion selection). These
design decisions can have a significant impact on human-LM interaction (Clark
et al., 2018; Buschek et al., 2021; Ippolito et al., 2022).

##### Language models.

We consider an LM to be a black box that takes a text _prompt_ and decoding
parameters as input and stochastically returns a set of text _completions_
(QueryLM in Algorithm 1).

$s_{0}\leftarrow\textnormal{task-specific contents}$

for  _t = 1, 2, …_ do

User takes an action $a_{t}$

if  _$a_{t}\textnormal{ finishes the interaction}$_ then

break

else

System updates the state $s_{t+1}\leftarrow$ Transition(_$s_{t}$ , $a_{t}$_)

end for

return $[(s_{1},a_{1}),(s_{2},a_{2}),...]$

Function  _Transition(_$s_{t}$ , $a_{t}$_)_:

if  _$a_{t}\textnormal{ requires querying an LM}$_ then

$p$ = CreatePrompt ($s_{t}$)

Fetch completions $c=\textnormal{{{QueryLM}}}(p)$

return ShowCompletions ($s_{t}$, $c$)

else if  _$a_{t}\textnormal{ fills out a survey}$_ then

return $s_{t}$ with results of survey

else

// (e.g., $a_{t}$ edits user input)

return $s_{t}$ updated with $a_{t}$

end if

Algorithm 1 Generate an interaction trace

##### System logic.

Given an LM, we define a system logic to be a set of states, potential user
actions, and a transition function that specifies how states get updated,
borrowing the terminology and intuitions from Markov decision processes. A
_state_ $s_{t}$ captures everything the system needs to know about the current
interaction at time $t$. This is primarily the visual state—the text contents
of each text field in the graphical interface (e.g., dialogue history and the
current text in user input), but also includes any hidden dependencies, such
as the history of previous dialogues. Given a state, a user can take one of a
set of _actions_. For example, the user may type to modify user input or click
a button.

Given a state $s_{t}$ and an action $a_{t}$, the system produces the updated
state $s_{t+1}$ through a _transition function_ (Transition in Algorithm 1).
When the action is typing to modify user input, the update is a
straightforward rendering of textual changes. When an action requires querying
an LM, the system constructs a prompt (CreatePrompt in Algorithm 1), queries
the underlying LM with the prompt (QueryLM in Algorithm 1), and updates the
state with the completions from the LM (ShowCompletions in Algorithm 1). For
example, in a dialogue system (Figure 3), a user may write “Thanks!” in user
input and click the “send” button, which will trigger the system to construct
a prompt (e.g., concatenating current dialogue history and user utterance),
fetch a completion from the LM, and show it to the user.

Dimensions | Tasks
---|---
|  |  | Social | Question | Crossword | Text | Metaphor
Targets | Perspectives | Criteria | dialogue | answering | puzzles | summarization | generation
Process | First-person | Preference | Reuse | Ease | Enjoyment |  | Enjoyment
Process | First-person | Quality |  | Helpfulness | Helpfulness | Improvement | Helpfulness
Process | Third-party | Preference |  |  | Queries |  |
Process | Third-party | Quality |  | Queries |  | Edit distance | Queries
Output | First-person | Preference | Interestingness |  |  |  | Satisfaction
Output | First-person | Quality | Specificity | Fluency | Fluency | Consistency | Helpfulness
Output | Third-party | Preference |  |  |  |  | Interestingness
Output | Third-party | Quality |  | Accuracy | Accuracy | Consistency | Aptness
Table 1:  We define a set of metrics for evaluating human-LM interaction
across 5 tasks (see Appendix D for the full list); each metric can be
characterized along three dimensions (targets, perspectives, and criteria).
Note that some metrics, such as the number of _queries_ from users, can be
viewed as proxies for different quality (e.g., efficiency) or preference
(e.g., enjoyment) metrics depending on the task.

##### Interaction traces.

We define a sequence of state-action pairs as an _interaction trace_ , similar
to Lee et al. (2022). Algorithm 1 shows the process of generating an
interaction trace as a result of human-LM interaction.

##### Design considerations.

We highlight important design considerations that can directly affect the
dynamics of human-LM interaction. First, CreatePrompt determines how much
control users have over prompts. For example, CreatePrompt can simply take the
content of user input, or enforce a pre-defined prompt regardless of the user
input. It can also decide how much each query depends on past interactions, by
providing previous interactions as part of the prompt. Second, decoding
parameters in QueryLM influence the quality of completions (Holtzman et al.,
2020), thus changing the dynamics of human-LM interaction (Lee et al., 2022).
In our experiment, we choose a set of decoding parameters for each task based
on initial pilots and use it for all models. Third, ShowCompletions controls
how model completions are shown to users. For instance, how many completions
are shown at a time, and how intrusive are they? These decisions can have a
trade-off between efficiency and ideation (Buschek et al., 2021), or lead to
distraction or cognitive load (Clark et al., 2018).

###  2.3 Evaluating human-LM interaction

To evaluate interaction traces, we expand on non-interactive evaluation along
three dimensions and define metrics characterized by those dimensions.

##### Dimensions.

Figure 1 shows three dimensions in human-LM interaction evaluation. The first
dimension is _targets_ (i.e., what to evaluate). Standard evaluation only
considers the final _output_ as a target, which is often simply a model
completion $c$, but can be a result of more complicated processes (e.g.,
sample multiple completions and use heuristics to construct the final output).
In contrast, we consider the entire interaction _process_ , which is
represented as an interaction trace $[(s_{1},a_{1}),(s_{2},a_{2}),...]$ in
HALIE.

The second dimension is _perspectives_ (i.e., whose perspective is reflected).
Non-interactive evaluation solely depends on a _third-party_ perspective;
automatic metrics can be computed on an interaction trace without human
intervention; likewise, human evaluation is done by third parties who did not
interact with LMs in any way. In the interactive setting, however, evaluation
should reflect the _first-person_ perspective of the user who actually takes
actions $a_{t}$ to interact with an LM through a system.

The last dimension is _criteria_ (i.e., how to evaluate). Standard evaluation
focuses on _quality_ , which tend to be objective and include metrics like
accuracy and fluency (defined for outputs) or helpfulness and edit distance
(defined for processes). On the other hand, interactive evaluation adds
attitudinal measures of a human, which we define as _preference_. These
measures tend to be subjective and often captured by metrics like enjoyment
and satisfaction. Note that preference criteria apply to both first-person and
third-party perspectives. One example of preference metrics with third-party
perspectives is creative story writing: some readers may not like certain
kinds of stories (low preference), although they think that they are good
(high quality). Also, sometimes metrics can reflect both preference and
quality when human preference is aligned with the quality metrics of interest.

##### Metrics.

For each task, we define a set of metrics that cover the space defined by the
three dimensions. Some metrics are a function of interaction traces (e.g.,
number of queries users made), whereas some metrics only depend on outputs
(e.g., accuracy in solving a crossword puzzle). Some metrics are defined based
on survey responses from users (e.g., satisfaction over the final output) and
third-party evaluators (e.g., consistency of a summary given a document).

Table 1 shows the mapping from the space to the metrics in the five tasks (see
Table 21 in Appendix D for the full list of metrics). Note that non-
interactive evaluation only covers one combination of the dimensions (i.e.,
outputs for targets, third-party for perspectives, and quality for criteria)
corresponding to one gray cell in Figure 1. In contrast, our evaluation
considers all combinations, corresponding to  in Figure 1.

##  3 Experiments

In this section, we introduce five tasks, construct systems for them, have
human users interact with LMs using the systems, and evaluate human-LM
interaction. For LMs, we choose four state-of-the-art LMs: three variants of
OpenAI’s GPT-3 (Brown et al., 2020; Ouyang et al., 2022)—TextDavinci (text-
davinci-001), TextBabbage (text-babbage-001), and Davinci (davinci)—and AI21’s
Jurassic-1 (Lieber et al., 2021)—Jumbo (j1-jumbo). This allows us to study the
effect of model size (TextDavinci vs. TextBabbage), instruction tuning
(TextDavinci vs. Davinci), and implementation details regarding training data,
architecture, and others (Davinci vs. Jumbo).

###  3.1 Social dialogue

Figure 3:  [Social dialogue] A dialogue system’s _state_ consists of a scenario, dialogue history, and user input. When users take an _action_ (e.g., clicking the “send” button), the system updates the state (e.g., updating the dialogue history with a model completion).  Model | Fluency | Sensibleness | Specificity | Humanness | Interestingness | Inclination | Reuse
---|---|---|---|---|---|---|---
| (/100%) $\uparrow$ | (/5) $\uparrow$
TextDavinci | 93 $\pm$ 1.0 |  94 $\pm$ 1.0** | 83 $\pm$ 1.6* | 37 $\pm$ 2.0 | 36 $\pm$ 2.0 | 91 $\pm$ 1.2 |  4.09 $\pm$ .14**
TextBabbage | 90 $\pm$ 1.4 |  84 $\pm$ 1.7* |  81 $\pm$ 1.8* | 29 $\pm$ 2.1 | 30 $\pm$ 2.1 | 88 $\pm$ 1.5 | 3.35 $\pm$ .16*
Davinci | 92 $\pm$ 1.3 | 89 $\pm$ 1.4 |  92 $\pm$ 1.3** | 24 $\pm$ 2.0 | 27 $\pm$ 2.0 | 91 $\pm$ 1.3 | 3.80 $\pm$ .17
Jumbo | 89 $\pm$ 1.3 | 86 $\pm$ 1.5 | 84 $\pm$ 1.5 | 24 $\pm$ 1.8 | 32 $\pm$ 2.0 | 87 $\pm$ 1.4 |  3.21 $\pm$ .20*
Table 2:  [Social dialogue] Users perceived TextDavinci to have the best
_fluency_ , _sensibleness_ , _humanness_ , _interestingness_ , and _quality_ ,
but they expressed the similar _inclination_ to continue interacting with
Davinci whose responses were most _specific_ to what users had said. For the
first six metrics, the numbers indicate the percentages of system responses
under each metric (0–100%). The numbers for _reuse_ indicate the ratings of
each model after completing a dialogue (1–5). The means, standard errors, and
statistical significance666 Throughout the paper, results are denoted by * if
models had a significant effect relative to TextDavinci, * if significant
relative to TextBabbage, * if significant relative to Davinci, and * if
significant relative to Jumbo at the $p=0.05$ level using a Tukey-Kramer test.
are shown in the table.

Dialogue is a natural and flexible way to engage in communication, which makes
it an intuitive and popular mode of interaction for LMs. In this section, we
evaluate human-LM interaction in the context of open-ended dialogue about
social situations (henceforth social dialogue). We defer related work and
experimental details to Appendix B.1 for space.

##### Task.

We consider the task where given a social _scenario_ , users converse with a
system (or chatbot) about the scenario. Users initiate the conversation and
take turns with the chatbot to carry out the conversation until users choose
to finish it. We randomly select ten scenarios from the EmpatheticDialogues
(Rashkin et al., 2019) and CommonsenseDialogues datasets (Zhou et al., 2021).

##### System logic.

Figure 3 shows a dialogue system _state_ consisting of scenario, dialogue
history, and user input and possible _actions_ : pressing a key to modify user
input, clicking the “send” button to submit their input, and finishing the
dialogue. When users click the “send” button, the system creates a prompt with
five dialogue examples for in-context learning and the current dialogue
history, invokes an LM with the prompt, fetches a completion, and shows the
completion in the dialogue history in the interface (ShowCompletions). See
Appendix B.1.2 for details on CreatePrompt and QueryLM.

##### User study procedure.

We recruited a total of 189 crowd workers (or users) on Amazon Mechanical
Turk. For each of the scenarios and models, we had three users each produce a
dialogue. To ensure each dialogue is long enough, the interface allowed users
to finish the conversation only after taking more than ten turns or exceeding
250 words in total. At the end of the conversation, users completed a survey
about their experience.

##### Survey questions.

We ask users to evaluate the _interestingness_ , _sensibleness_ (i.e., whether
a chatbot response makes sense), and _specificity_ (i.e., whether a chatbot
response is specific to what a user had said) of a chatbot, following the
survey questions proposed by Thoppilan et al. (2022a). We also ask users to
evaluate the _fluency_ of the chatbot and their _inclination_ to continue
talking with the chatbot, as proposed by Smith et al. (2022a). We ask dataset-
specific questions regarding empathy (EmpatheticDialogues) and commonsense
understanding (CommonsenseDialogues), which are labeled as _humanness_
collectively. Finally, after a dialogue, we ask if users are willing to talk
to the chatbot again using a 5-point Likert scale: _reuse_. See Appendix B.1.3
for the full list of questions.

##### Results.

For social dialogue, we specifically care about models’ ability to show an
emotional connection to users, by being empathetic and having commonsense
about social situations (_humanness_), in addition to being _fluent_ and
_sensible_ , and _specific_ to what they have said. Independently, users may
have their own subjective views on whether system responses are interesting
and make them want to continue interacting with the system, which are measured
by preference metrics _interestingness_ and _inclination_ , respectively.

#####  1 Instruction tuning improves performance on most quality metrics, but
not specificity.

Table 5 shows that the instruction tuned TextDavinci scored highest on most
metrics, including _fluency_ , _sensibleness_ , and _humanness_. In
particular, users evaluated TextDavinci to have considerably higher
_humanness_ than Davinci. And the instruction tuned TextBabbage model, despite
its much smaller size, was reported to have higher humanness than Davinci or
Jumbo.

However, instruction tuning decreased the _specificity_ of model completions
as shown in the significant drop in performance of TextDavinci compared to
Davinci (Table 5). From this observation, we hypothesize that general
instruction-tuning makes models better aligned with most metrics at the cost
of being more generic (see Appendix B.1.4 for the examples of model
completions), although previous work suggests that dialogue-specific fine-
tuning improves all metrics including specificity Thoppilan et al. (2022b).

#####  2 Users may prefer to interact with a more specific LM.

Although users evaluated TextDavinci to be the best LM according to most
metrics, they expressed similar _inclination_ to continue interacting with
Davinci during dialogue. Given that Davinci was perceived to be worse than
TextDavinci on all metrics except _specificity_ and _inclination_ , we
hypothesize that users expressed an inclination to interact with Davinci
_because_ its responses were the most specific to what users had said. We
recommend further study of the relationship between specificity and
inclination to interact with an LM, and in particular, whether specificity
affects _reuse_ of the LM once the initial novelty of this interactive
experience has faded.

###  3.2 Question answering

Model | Accuracy | Time | Queries | Ease | Fluency | Helpfulness
---|---|---|---|---|---|---
| (/100%) $\uparrow$ | (min) $\downarrow$ | (#) $\downarrow$ | (/5) $\uparrow$
TextDavinci | 69 $\pm$ 2.2 | 1.36 $\pm$ .13 |  1.78 $\pm$ .06** | 4.53 $\pm$ .08 |  4.35 $\pm$ .07*** |  4.60 $\pm$ .07***
TextBabbage | 52 $\pm$ 2.8 | 1.77 $\pm$ .33 | 2.57 $\pm$ .13 * | 4.09 $\pm$ .12 | 3.84 $\pm$ .12*** | 3.84 $\pm$ .12***
Davinci | 48 $\pm$ 2.7 | 2.09 $\pm$ .14 |  2.66 $\pm$ .12 * | 3.73 $\pm$ .13 | 3.22 $\pm$ .11** | 3.52 $\pm$ .13***
Jumbo | 54 $\pm$ 2.9 | 1.67 $\pm$ .09 | 2.32 $\pm$ .11 | 3.87 $\pm$ .14 |  3.17 $\pm$ .11** |  3.26 $\pm$ .14***
Table 3:  [Question answering] Performance averaged across all questions
conditioning on the use of AI assistance. Users assisted by TextDavinci
achieved the highest _accuracy_ while requiring the least effort (_queries_ ,
and _ease_) and being perceived to be the most _fluent_ and _helpful_. The
numbers indicate means and standard errors, and the markers denote statistical
significance,6 conditioning on the use of AI assistance; when the assistance
was provided, users queried the system 86% of the time.  Figure 4:  [Question
answering] The system’s _state_ consists of a multiple-choice question, user
input, and system output. When users take an _action_ (e.g., clicking the
“generate” button), the system updates the state (e.g., updating the system
output with a model completion).

Question answering is a canonical task in NLP. We consider an _interactive_
version of question answering where a user is asked to answer a question given
access to an LM that they can query. In practice, users tend to query
information systems repeatedly, refining and reformulating their queries in
response to the system’s output (Huang and Efthimiadis, 2009a; Jansen et al.,
2009; Jiang et al., 2013). For related work and details of experiments, see
Appendix B.2.

##### Task.

Given a sequence of multiple-choice questions (or _quiz_), users try to select
the correct answer with advice from an LM. As an example, consider this
multiple-choice question:

> Before Nixon resigned how many believed he should be removed from office?
>  A. 79% B. 98 % C. 33 % D. 57%

Users can query the system multiple times with free-form text inputs; they
might start with the question verbatim, re-phrase the question to increase
specificity (e.g., “Did 79 percent of people believe Nixon should have been
removed from office?”), or find a different way to query, perhaps bringing in
their own prior knowledge (e.g., “Nixon approval rating after watergate”).

We use questions from the Measuring Massive Multitask Language Understanding
(MMLU) dataset Hendrycks et al. (2021). Concretely, we first chose five
diverse subjects from the dataset (Global facts, Nutrition, US foreign policy,
College chemistry, and Miscellany), and selected $6$ questions from each
subject to construct a pool of $30$ questions. We constructed quizzes by
randomly selecting ten questions from the pool and adding one attention check
question in the middle.

##### System logic.

Figure 4 shows a _state_ of our QA system consisting of a multiple-choice
question, user input, and system output. Users can take the following
_actions_ : pressing a key to modify user input, clicking the “generate”
button to query the system, selecting one of the multiple choices, clicking
the “next” button to submit their answer, and finishing the quiz. When users
click the “generate” button, the system creates a prompt with the user input,
invokes an LM with the prompt, fetches a completion, and shows the completion
in the system output box in the interface (ShowCompletions). See Appendix
B.3.1 for the exact implementation of CreatePrompt and QueryLM.

##### User study procedure.

We recruited 342 crowd workers (users) on Amazon Mechanical Turk. Each user
answered half of the questions with assistance from a single LM (treatment)
and answered the other half without assistance (control). We instructed users
not to use search engines to answer questions, and alerted users (“Please do
not switch tabs or open a new window during your participation.”) when we
detected the tab switching. At the end of each quiz, users completed a survey
about their experience.

##### Survey questions.

We asked users for first-person evaluation of _fluency_ , _helpfulness_ , and
_ease of interaction_ of the system on a 5-point Likert scale. Also, we asked
users to describe why they found the system helpful or unhelpful, provide
adjectives to describe the system, and explain whether their interaction
changed over the course of answering questions. See Appendix B.3.2 for the
full list of questions.

##### Results.

For the goal-oriented QA task, third-party evaluation is characterized by
_accuracy_ (i.e., how many questions users get correct answers for) and
_efficiency_ (i.e., how much effort users put in to find an answer) (Dibia et
al., 2022). Because the bulk of a user’s time in interactive tasks is spent
reacting to system outputs, we count the number of _queries_ needed to answer
each question as a proxy measurement for efficiency.

#####  1 Users with LM assistance generally outperform the user or LM alone,
but not always.

Figure 9 shows human+LM accuracy, zero-shot LM accuracy,777 For fully-
automated models, we report deterministic results with temperature $=0$.  and
human-LM interactive accuracy broken down by question category. Users assisted
by TextDavinci outperformed users alone in all categories. Users with LM
assistance generally outperformed an LM alone, with the notable exception of
the US Foreign Policy category, where both TextDavinci and Davinci achieved
better accuracy as zero-shot QA systems than as interactive assistants. For
the Global Facts and Miscellany categories, users with LM assistance
significantly outperformed zero-shot LMs. In these contexts, users are able to
figure out how to query the LM and when to trust it.

#####  2 Non-interactive performance does not always lead to better human-LM
interaction.

Further analysis of Figure 9 shows that for the Nutrition category,
TextBabbage had the worst accuracy as a zero-shot QA system, but achieved the
best performance as an interactive LM assistant (tied with TextDavinci). For
US Foreign Policy, Jumbo performed worst as a zero-shot QA system and Davinci
performed best, but as an LM assistant Jumbo outperformed Davinci. This
supports the broader point that non-interactive performance is an imperfect
proxy for evaluating human-LM interaction.

Figure 5:  [Question answering] Per-question accuracy broken down by category.
With TextDavinci, human-LM interaction results in improved accuracy for Global
Facts, College Chemistry, and Miscellany, but not Nutrition and US foreign
policy. The shaded bars indicate the performance of models as zero-shot, non-
interactive QA systems when given a question text verbatim,999The zero-shot,
non-interactive QA systems’ accuracy can be found at
https://nlp.stanford.edu/helm/interactive_qa_mmlu/?suite=interactive_qa_mmlu&group=mmlu
and gray bars indicate user performance without AI assistance. The results are
averaged across questions for which users were provided with AI assistance.

#####  3 Instruction tuning improves accuracy and efficiency for human-LM
interaction.

Table 13 shows that TextDavinci was the most efficient and accurate tool (1.78
queries/question with 69% accuracy). In contrast, Davinci was least efficient,
requiring an average of 1.5x as many queries to answer a question with only
48% accuracy. Despite its smaller size, TextBabbage performed better than
Davinci on most metrics, demonstrating the effectiveness of instruction tuning
for this task. Instruction tuned models were also perceived most favorably in
first-party survey evaluation.

###  3.3 Crossword puzzles

Figure 6:  [Crossword puzzles] The system’s _state_ consists of a crossword
puzzle, selected clue (can be none), user letters entered in the puzzle,
dialogue history, and user input. When users take an _action_ (e.g., pressing
the enter key after writing user input), the system updates the state (e.g.,
updating the dialogue history with a model completion).

Crossword puzzles have been studied as a challenging task for AI systems
(Ginsberg, 2014; Littman et al., 2002; Wallace et al., 2022b; Rozner et al.,
2021). Unlike multiple-choice QA, solving a crossword puzzle is a _generative_
task requiring open-ended responses to clues. The crossword puzzle task also
provides additional structure, whereby a user can check whether a candidate
answer satisfies the lexical constraints of the puzzle. Finally, clues are
often not straightforward (e.g. “Christmas in Chamonix”), and a user might
need to reformulate the query to find the desired information.

Model | Accuracy (letter) | Accuracy (clue) | Fluency | Helpfulness | Ease | Enjoyment
---|---|---|---|---|---|---
| (/100%) $\uparrow$ | (/5) $\uparrow$
TextDavinci |  63 $\pm$ 2.9* |  53 $\pm$ 3.4* |  3.65 $\pm$ .10** |  3.14 $\pm$ .12*** |  4.35 $\pm$ .10** |  2.91 $\pm$ .20***
TextBabbage |  47 $\pm$ 3.3* |  38 $\pm$ 3.5* | 3.14 $\pm$ .13** | 2.27 $\pm$ .14* | 3.78 $\pm$ .15** | 2.19 $\pm$ .22**
Davinci | 55 $\pm$ 3.5 | 46 $\pm$ 3.6 |  2.26 $\pm$ .11** |  1.92 $\pm$ .10* | 3.32 $\pm$ .14** | 1.92 $\pm$ .17**
Jumbo | 56 $\pm$ 2.8 | 45 $\pm$ 3.1 | 2.30 $\pm$ .10** | 2.20 $\pm$ .10* |  3.08 $\pm$ .15** |  1.66 $\pm$ .18*
Table 4:  [Crossword puzzles] Users assisted by TextDavinci found their model
more _fluent_ , _helpful_ , and _easy_ and _enjoyable_ to interact with
compared to other models, and in general provided more accurate solutions
across all puzzles. However, while users with Davinci and Jumbo performed
worst on the self-reported survey metrics, users with TextBabbage had the
worst _accuracy_ , suggesting a disconnect between first-person preference and
automated quality metrics. The numbers indicate means and standard errors, and
the markers denote statistical significance.6

Figure 7:  [Crossword puzzles] Although across all puzzles, users assisted by
TextDavinci were significantly more likely to report the model as helpful
(left), they did not always provide significantly more accurate puzzle
solutions (right). In fact, for the NYT-1 puzzle, users assisted by
TextDavinci were significantly less likely to provide accurate solutions,
despite being more likely to find the model helpful.

##### Task.

A crossword puzzle requires solving a series of word clues (e.g., “Oscar the
Grouch’s home”) that correspond to either rows (“Across” clues) or columns
(“Down” clues) of white squares on a rectangular grid. As users solve one clue
and places letters in squares, they reveal partial information (e.g., shared
letters) that can affect their strategy for solving future clues. In our
setting, users try to complete an entire crossword puzzle with the ability to
query an LM-based “AI Teammate” via a chat-based interface. Users may message
the AI Teammate with free-form text, where each individual message is used as
the full prompt to the underlying LM.

##### System logic.

Figure 6 shows a _state_ of our crossword system, which we adapt from _Down
for a Cross_ , an open-source software for the multi-player crossword puzzles.
Users can take the following _actions_ : pressing a key to modify user input,
pressing the “enter” key to submit input, selecting a square in the puzzle,
entering a letter into a square, selecting a clue from the list, and finishing
the session. While attempting to solve different crossword clues
(ShowCompletions), users (with the chat ID: You) are able to repeatedly
interact with an LM (with the chat ID: AI) via zero-shot prompts in a
dialogue-based chat interface. Although we display the chat history to aid
players in remembering past information, only the most recent prompt from the
player is sent to the LM. The same LM is fixed throughout the course of the
task, which ends either after 30 minutes or when all clues are correctly
solved. See Appendix B.4.2 for details on CreatePrompt and QueryLM.

##### User study procedure.

We recruited 350 workers (users) on Amazon Mechanical Turk, split across each
of the four language models and five puzzles (Love, Sit, Nyt-1, Nyt-2, and
Elect). Each user was provided a link to the interface, and asked to engage
with the puzzle (e.g., solve clues, interact with the LM) for at least 30
minutes. Afterwards, each user completed a survey about their experience
interacting with the AI Teammate. Further details including instructions and
information about compensation can be found in the Appendix B.4.3.

##### Survey questions.

After playing with provided crossword puzzles, users are then asked to rank
different qualities of the AI assistant on a 5-point Likert scale, including
_fluency_ , _helpfulness_ , _enjoyment_ , and _ease of the interaction_ with
the AI Teammate. Users are additionally asked to describe why they found the
teammate helpful or unhelpful, what adjectives best describe the assistant,
and whether their interaction changed over the course of answering questions.
See Appendix B.4.4 for the full list of the questions.

##### Results.

Solving a crossword puzzle is simultaneously an open-ended goal-oriented task
and a form of entertainment—therefore, third-party evaluation should consider
both accuracy and engagement of the human-LM interaction. Because some users
may enjoy solving crossword puzzles independently, we count the number of
queries used over the course of the task as a proxy measurement for user
preference to engage with a given LM.

#####  1 Helpfulness perceived by users exceeds accuracy improvements.

We measure the perceived _helpfulness_ of a model based on a post-task survey
question with a 5-point Likert scale. Figure 7 shows that users significantly
preferred TextDavinci over other models with respect to helpfulness for all
puzzles. Interestingly, this perception of helpfulness is not necessarily
reflected in the overall interaction _accuracy_ : as shown in Figure 7,
assistance from TextDavinci led to statistically significantly higher
crossword letter accuracy only for the SIT crossword puzzle; for the NYT-1
puzzle, users actually performed worse with both instruction tuned models
(TextDavinci and TextBabbage). One hypothesis for this behavior is that these
models are capable of providing confident and fluent misinformation, which can
be incorrectly perceived to be helpful. Furthermore, the variance in model
performance between puzzles strongly suggests that different puzzles’
distributions over clue types may require different capabilities from LMs —in
Table 10, we examine performance across 6 clue category types, and find that
LMs, and in particular TextDavinci, were most useful for clues about factual
knowledge, but struggle with wordplay and phrases.

#####  2 Short prompts exacerbate misinformation and toxicity.

We observed significant amounts of misinformation across all models.
Misinformation was particularly pernicious using TextBabbage, which achieved
relatively positive responses in user-reported survey responses but the lowest
objective accuracy for puzzle solutions provided by users (Table 4). We
include a large set of examples of misinformation from TextBabbage and
TextDavinci in Appendix B.4.7. While we believe misinformation played a big
role in the difference between user perceived and actual performance on the
task, the overall effect of misinformation might be more nuanced: the
structure of crossword puzzles provides constraints that may help users more
quickly identify incorrect model suggestions.

Additionally, we observed instances of toxic content in generated crossword
outputs. Explicit responses often occurred when users, perhaps influenced by
the style of crossword clues, provided short keyword prompts: e.g., “japanese
car”, “nick wilde”, or “synonym of delayed.” Short, ungrammatical prompts may
more closely match toxic patterns in the training data (e.g. pornographic
video captions or social media comments). Other safety concerns we observed
included generating personal phone numbers and YouTube accounts belonging to
non-celebrities, as well as discussing more sensitive topics, which we
describe further in Appendix Section B.4.8. Longer prompts that setup the
appropriate context might mitigate toxic outputs, but its effectiveness merits
further study.

#####  3 Users demonstrate diverse engagement behavior.

The relatively long period of interaction between users and the AI Teammate in
the crossword puzzle task provides an opportunity to closely study engagement,
which we measure in two ways. First, we ask users in the post-task survey to
answer “How enjoyable was it to interact with the AI Teammate?”, and show in
Table 4 that users found TextDavinci significantly more enjoyable to use (for
some puzzles, TextBabbage was also rated highly), suggesting that users tended
to enjoy interacting with LMs that were most helpful for the task. However, in
Figure 14 in the Appendix, we take a closer look at engagement across the
entire 30 minutes task length, and observe a diverse set of user behaviors:
while some users decided to stop querying the AI Teammate after receiving
incorrect responses, others chose to solve clues independently at the start
and only query later when stuck. A long-tail of users, mostly interacting with
TextDavinci and TextBabbage, had significant sustained interaction throughout
the course of the puzzle, often experimenting with re-phrasing prompts to
elicit more helpful responses. Finally, unlike with the use of AI solvers that
seek to solve a crossword puzzle perfectly (Wallace et al., 2022a), some users
found the challenge of figuring out how to best use the AI Teammate itself
entertaining:

> Davinci Player: This is an enjoyable task that may not be as fun if the AI
> would give you all the answers!

###  3.4 Text summarization

Text summarization is a long-standing problem in NLP (Luhn, 1958; Mani, 1999;
Spärck Jones, 1999; Nenkova and McKeown, 2012). Notably, it has been studied
in interactive settings for multi-document summarization, where the users
interact with the system to query for additional information to expand upon an
initial summary (Avinesh et al., 2018; Shapira et al., 2021, 2022). In
contrast, we focus on human-LM interaction for single-document summarization,
where users interact with the system to correct LM-generated summaries. In
addition, as users correct summaries for a sequence of documents, we provide
previous human-edited summaries as examples to the system to improve future
summaries via in-context learning. For more related work and details of
experiments, see Appendix B.5.

##### Task.

We consider the task, where given a document and a model-generated summary,
users edit the summary to be consistent, relevant, and coherent. We randomly
select 964 documents from XSum dataset Narayan et al. (2018) and construct 20
summarization _sessions_ by randomly choosing ten documents per session
without replacement.

Figure 8:  [Text summarization] The system’s _state_ consists of a document,
user summary history, user input, and survey questions. When users take an
_action_ (e.g., responding to survey questions and clicking the “next”
button), the system updates the state (e.g., showing the next document and
model-generated summary).

##### System logic.

Figure 8 shows the system state and actions. A summarization system _state_
consists of a document, user summary history (not visible to users), user
input, and survey questions. Users can take the following _actions_ : pressing
a key to modify a model-generated summary, clicking the “next” button,
responding to survey questions, and finishing the session.

One notable difference in the summarization system’s state is that
CreatePrompt generates a prompt by concatenating all the previous document and
user-edited summary pairs in the user’s summary history as part of the prompt.
This enables us to study whether an LM can learn from user examples and
improve over time, as well as how users behavior change as the models learn or
fail to learn from their examples. See Appendix B.5.2 for details on
CreatePrompt and QueryLM.

##### User study procedure.

We recruited $39$ crowd workers (or users) on Amazon Mechanical Turk. For each
model, we collected $20$ summarization sessions ($80$ in total), while
allowing the same users to participate multiple times. Upon receiving a model-
generated summary for each document, users were asked to edit the summary to
be consistent, relevant, and coherent. To encourage users to pay attention to
the three quality criteria when editing, we asked users to evaluate the
consistency, relevance, and coherence of the original (model-generated)
summary and edited summary before and after editing. At the end of a
summarization session, users completed a survey about their overall experience
interacting with the system.

##### Survey questions.

We ask per-summary and per-session questions to the users. Summary-level
questions ask _consistency_ , _relevance_ , and _coherence_ of the original
and edited summaries, following Fabbri et al. (2021). Session-level questions
evaluated users’ overall perceptions of the summarization system with respect
to its _helpfulness_ and _improvement over time_. See Appendix B.5.2 for the
full list of questions.

##### Third-party human evaluation.

We also asked third-party evaluators to evaluate the consistency, relevance,
and coherence of a subset of the summaries generated by LMs. To this end, we
randomly sampled $100$ documents from our user study and recruited $18$
workers (third-party evaluators) on Amazon Mechanical Turk to assess the
summaries written for the documents. Each summary was assessed by $3$
different evaluators.

Model | Original | Edited | Edit distance $\downarrow$ | Revision $\downarrow$ | Helpfulness $\uparrow$ | Improvement $\uparrow$
---|---|---|---|---|---|---
| (word) | (/5)
TextDavinci | 17.70 $\pm$ .47 | 25.08 $\pm$ .89 |  12.38 $\pm$ .89* | 3.00 $\pm$ .19 |  4.20 $\pm$ .19* | 2.60 $\pm$ .29
TextBabbage | 20.11 $\pm$ .43 | 32.61 $\pm$ .85 |  16.34 $\pm$ .91* | 3.40 $\pm$ .17 | 3.40 $\pm$ .28 | 2.15 $\pm$ .21
Davinci | 16.96 $\pm$ .38 | 25.05 $\pm$ .89 | 14.19 $\pm$ .89 | 3.25 $\pm$ .20 | 3.80 $\pm$ .22 | 2.10 $\pm$ .20
Jumbo | 14.75 $\pm$ .34 | 25.33 $\pm$ .82 | 15.12 $\pm$ .83 | 3.30 $\pm$ .21 |  3.30 $\pm$ .25* | 2.40 $\pm$ .28
Table 5:  [Text summarization] Users edited summaries generated by TextBabbage the most, and TextDavinci the least. The survey responses for the perceived amount of revision (_revision_) accurately reflect the actual edit distance. Overall, users perceived TextDavinci to be most helpful (_helpfulness_) and better improves over time (_improvement_) compared to the other models. The first three metrics refer to the length of model-generated summaries (_original_), human-edited summaries (_edited_), and the Levenshtein distance between them (_edit distance_). The numbers indicate means and standard errors, and the markers denote statistical significance.6 Model | Consistent | Relevant | Coherent
---|---|---|---
| (/100%) $\uparrow$ | (/5) $\uparrow$
TextDavinci | 65 $\pm$ 4* | 4.07 $\pm$ .08* |  4.70 $\pm$ .04*
TextBabbage |  89 $\pm$ 3*** |  4.15 $\pm$ .06** |  4.51 $\pm$ .06*
Davinci | 57 $\pm$ 4* |  3.70 $\pm$ .09** | 4.53 $\pm$ .05
Jumbo |  56 $\pm$ 4* | 3.80 $\pm$ .07* | 4.60 $\pm$ .04
Table 6:  [Text summarization] Third-party evaluation of model-generated
summaries. According to third-party evaluation, TextBabbage performed the best
with respect to _consistency_ and _relevance_ , while TextDavinci was rated
highest for _coherence_. The numbers indicate means and standard errors, and
the markers denote statistical significance.6

##### Results.

In text summarization, we ask third-party evaluators to evaluate the quality
of summaries with respect to their _consistency_ (i.e., all the information in
the summary is inferred from the document), _relevancy_ (i.e., the summary
includes only important information and no excess information), and
_coherence_ (i.e., the summary organizes the relevant information in a well-
structured manner). For first-person perspectives, we compute _edit distance_
from interaction traces, and consider survey responses for _helpfulness_ and
_improvement_ (i.e., the improvement of AI assistance as more summaries are
edited).

#####  1 There is a discrepancy between metrics based on third-party and
first-person perspectives.

Table 5 shows that users found summaries generated by TextDavinci to be the
most helpful starting point for editing, receiving the smallest _edit
distance_ and _revision_ score along with the highest _helpfulness_ score.
However, from third-party evaluation, original summaries from TextBabbage were
rated as the most _consistent_ and _relevant_ among the four models (Table 6).
According to the _density_ metric (i.e., the average length of the extractive
fragment to which each word in the summary belongs) Grusky et al. (2018),
summaries generated by TextBabbage are much more extractive (16.11) than those
by other models (4.19 for TextDavinci, 4.86 for Davinci, and 4.09 for Jumbo).
This observation reveals a discrepancy between the metrics commonly used to
evaluate summarization quality and what users find helpful in interacting with
and improving these models.

Figure 9:  [Text summarization] We observe that users edited less with
TextDavinci and more with TextBabbage over time, suggesting that TextDavinci
is better at in-context learning the user’s preferences for summarization. The
plot shows the rolling average of edit distance between original and edited
summaries over ten summaries (1st through 10th summary) across all
summarization sessions, with the window size of 2.

#####  2 Users save time and effort with TextDavinci, suggesting it is better
at in-context learning.

We are interested in how users behavior change over time as LMs succeed or
fail to learn from previous examples. To that end, we observed how the _edit
distance_ between the original and edited summaries change as a summarization
session progresses. Figure 9 shows the change of edit distance across ten
summaries in a sessions. We observe that users had to edit less with
TextDavinci over time, whereas they had to edit more with TextBabbage. The
survey response also indicated that users perceived TextDavinci to be best at
improving over time (_improvement_ in Table 5). From this observation, we
hypothesize that TextDavinci is better at in-context learning the user’s
preferences for summarization.

###  3.5 Metaphor generation

Metaphors are used throughout poetry, journalism, and science education to
communicate complex or abstract ideas (Mio, 1997; Lakoff and Turner, 2009;
Niebert et al., 2012). Creating metaphors requires divergent, lateral thinking
(Glucksberg and McGlone, 2001; Gero and Chilton, 2018). To help with ideation,
prior work designed metaphor generation tools where users could query multiple
times to get suggestions and showed that these suggestions enabled people to
write metaphors that they might never have thought of (Gero and Chilton, 2019;
Chakrabarty et al., 2022). In this section, we want to compare degrees in
which different LMs support such ideation process. We defer related work and
experimental details to Appendix B.6 for space.

##### Task.

Given a seed metaphor, the task is to write metaphorical sentences that evoke
the given metaphor in a limited amount of time. For example, for the metaphor
“Time is money,” we might generate the following metaphorical sentences:

> How do you spend your time?
> That flat tire cost me an hour.
> I’ve invested a lot of time in him.

From Lakoff and Johnson (2008), we select $3$ metaphors and their
corresponding sentences as in-context learning examples, and choose $4$
thematically diverse metaphors for evaluation. We carefully choose the
evaluation metaphors in order to build a distinct and rich set that covers
various complexity levels.

Figure 10:  [Metaphor generation] The system’s _state_ consists of a seed
metaphor, user sentence history, user input, and system suggestions. When
users take an _action_ (e.g., clicking the “get suggestions” button), the
system updates the state (e.g., showing five model completions as
suggestions).

##### System logic.

Figure 10 shows our metaphor system _state_ consisting of a seed metaphor,
user sentence history, user input, and system suggestions. Users can take one
of the following _actions_ : pressing a key to modify user input, clicking the
“get suggestions” button, selecting a suggestion from the list of suggestions
or dismiss all of them, and finishing the session. When users click the “get
suggestions” button, the system creates a prompt with three in-context
examples and the current user input, invokes an LM in the state, fetches five
outputs with the prompt, and shows the outputs in the popup box in the
interface (ShowCompletions). See Appendix B.6.2 for the exact implementation
of CreatePrompt and QueryLM.

##### User study procedure.

We recruited 32 workers (users) on Amazon Mechanical Turk, and each of them
could work on all four seed metaphors if desired. For each seed metaphor, we
randomly assigned one of the four LMs. In each session, each user was given 10
minutes to come up with as many metaphorical sentences as they could using the
system. In the instructions, we asked users to write sentences that are apt,
specific, and imageable along with examples, following the evaluation criteria
in Gero and Chilton (2019). At the end of the session, users completed a
survey about their experience.

##### Survey questions.

In the survey, we asked about users’ perceptions concerning _fluency_ ,
_helpfulness_ , _ease_ of interaction, _enjoyment_ , _satisfaction_ ,
_ownership_ , and willingness to _reuse_ the system on a 5-point Likert scale,
similar to Lee et al. (2022). See Appendix B.6.3 for the full list of
questions.

##### Third-party evaluation.

We also asked third-party evaluators to evaluate whether written metaphorical
sentences are apt, specific, and imageable, following Gero and Chilton (2019).
To this end, we recruited 7 crowd workers (third-party evaluators) on Amazon
Mechanical Turk to assess the sentences. Each sentence was assessed by 2
different evaluators.

Model | Time | Queries | Acceptance | Edit distance
---|---|---|---|---
| (min) $\downarrow$ | (#) $\downarrow$ | (/100%) $\uparrow$ | (word) $\downarrow$
TextDavinci | 0.74 $\pm$ .07 | 0.92 $\pm$ .07 |  51 $\pm$ 4.4 ** | 4.79 $\pm$ .52
TextBabbage | 0.73 $\pm$ .04 | 0.97 $\pm$ .09 | 56 $\pm$ 3.6* | 6.43 $\pm$ .73
Davinci | 0.60 $\pm$ .05 | 0.77 $\pm$ .06 |  71 $\pm$ 4.1** | 4.83 $\pm$ .60
Jumbo | 0.75 $\pm$ .06 | 1.03 $\pm$ .11 | 68 $\pm$ 4.3 * | 5.59 $\pm$ .54
Table 7:  [Metaphor generation] User effort in writing a metaphorical
sentence. Users spent the least effort writing metaphorical sentences with
Davinci, taking least _time_ and making the fewest _queries_ among the four
models. The same model achieved the highest _acceptance_ rate. However,
conditioning on the acceptance of suggestions, users edited the least amount
(_edit distance_) when working with suggestions from TextDavinci. The numbers
indicate means and standard errors (for writing one metaphorical sentence),
and the markers denote statistical significance.6

##### Results.

In metaphor generation, we measure user effort in writing with _time_ and the
number of _queries_ per sentence. Then, we look at user survey responses on
_helpfulness_ , _satisfaction_ , _ease_ of interaction, and willingness to
_reuse_ the system. Lastly, we evaluate the quality of metaphorical sentences
(based on how _apt_ , _specific_ , and _imageable_ the sentences are) through
third-party evaluation.

#####  1 Users are more willing to reuse models that require less effort.

Table 7 shows that users needed the least effort when working with Davinci,
given that they took the least _time_ and made the fewest _queries_. Despite
the fewest number of queries, and therefore fewest suggestions, the acceptance
rate for suggestions was the highest for Davinci and the final metaphorical
sentences were rated most highly by third-party evaluators (Table 9. From
these observations, we hypothesize that users could query a few times and
quickly find acceptable sentences generated by Davinci. According to survey
responses, users also perceived the same model as the _easiest_ to interact
with and were most willing to _reuse_ the system when the model was used.

#####  2 More satisfying user experiences may not correlate with the quality
of outputs.

On the other hand, users found TextDavinci to be most _helpful_ and
_satisfactory_ according to the survey responses (Table 8). However, third-
party evaluators considered sentences written with the same model to be the
worst among the four models (Table 9). Although it is hard to draw strong
conclusions due to the relatively small size of users and third-party
evaluators, one possible explanation is that users’ value judgment for
satisfaction and reuse may depend on different factors (e.g., helpfulness and
ease, respectively) based on the context.

Model | Helpfulness | Satisfaction | Ease | Reuse
---|---|---|---|---
| (/5) $\uparrow$
TextDavinci | 4.21 $\pm$ .18 | 4.42 $\pm$ .14 | 3.68 $\pm$ .22 | 4.42 $\pm$ .18
TextBabbage | 3.64 $\pm$ .21 | 4.14 $\pm$ .20 | 3.82 $\pm$ .23 | 4.39 $\pm$ .17
Davinci | 4.17 $\pm$ .23 | 4.33 $\pm$ .20 | 3.94 $\pm$ .25 | 4.61 $\pm$ .14
Jumbo | 4.13 $\pm$ .24 | 4.40 $\pm$ .13 | 3.87 $\pm$ .24 | 4.47 $\pm$ .19
Table 8:  [Metaphor generation] User survey responses. Overall, users perceived TextDavinci to be the most _helpful_ and _satisfied_ with the results from working with the model. However, users perceived Davinci to be the _easiest_ to work with and were willing to _reuse_ the model the most. Overall, perceived _helpfulness_ of models and user _satisfaction_ had a strong positive correlation ($r=0.95$), while _ease_ of interaction and users’ willingness to _reuse_ the system were also positively correlated ($r=0.74$). The numbers indicate means and standard errors. For these outcomes, no results were found to have statistical significance using a Tukey-Kramer test.  Model | Apt | Specific | Imageable | Overall
---|---|---|---|---
| (/100%) $\uparrow$
TextDavinci | 75 $\pm$ 4.0 | 78 $\pm$ 4.6 | 75 $\pm$ 4.6 | 78 $\pm$ 3.4
TextBabbage | 75 $\pm$ 4.0 | 79 $\pm$ 3.7 | 70 $\pm$ 5.0 | 78 $\pm$ 3.0
Davinci | 90 $\pm$ 3.3 | 90 $\pm$ 3.3 | 83 $\pm$ 5.3 | 88 $\pm$ 3.0
Jumbo | 77 $\pm$ 5.5 | 83 $\pm$ 5.3 | 72 $\pm$ 5.7 | 84 $\pm$ 3.9
Table 9:  [Metaphor generation] Third-party evaluation on the quality of
metaphorical sentences. The numbers indicate means and standard errors. These
metrics were not found to exhibit any statistically significant differences
using a Tukey-Kramer test at the $p=0.05$ level. Conditioning on the use

##  4 Related work

We first review the evaluation of LM-based language generation systems while
distinguishing the evaluation of model _completions_ and _interaction traces_.
Then, we briefly recount specialized interactive systems other than LMs.

##### Evaluation of model completions.

Evaluation has a long history in NLP (see Spärck Jones and Galliers, 1995;
Liberman, 2010). Traditionally, evaluations for generative systems have
centered on specific tasks: for example, WMT for machine translation (WMT,
2006), TIPSTER SUMMAC for text summarization (Mani et al., 1999), Dialogue
System Technology Challenges (DSTC) (Gunasekara et al., 2020) and Alexa Prize
Challenges (Hakkani-Tür, 2021; Gottardi et al., 2022) for dialogue systems.
More recently, efforts like GEM (Gehrmann et al., 2021) and GEMv2 (Gehrmann et
al., 2022) have consolidated and standardized practices for natural language
generation across many tasks.

For LMs, we have seen an analogous trend: LMs were initially evaluated for the
probabilities they assigned to unseen corpora like the Penn Treebank (Marcus
et al., 1993), One Billion Word Benchmark (Chelba et al., 2013), and
WikiText-103 (Merity et al., 2017). However, as LMs have functioned as
foundation models (Bommasani et al., 2021) underpinning myriad downstream
systems, evaluation practices have shifted to consolidated evaluations for
many downstream tasks (Wang et al., 2019b, a; Gao et al., 2021; Srivastava et
al., 2022; Liang et al., 2022). We emphasize that across all the benchmarks
used to evaluate generation systems and/or LMs, the prevailing norm has been
_non-interactive_ evaluation with few notable exceptions (e.g., dialogue
benchmarks).

##### Evaluation of interaction traces.

While human-LM interaction has been studied within NLP in specific domains,
such as dialogue (Hu et al., 2021; Thoppilan et al., 2022a; Smith et al.,
2022a) and story generation (Akoury et al., 2020; Clark and Smith, 2021),
these works only consider some aspects of user interaction. Namely, while they
target the interaction process (e.g., dialogue history), they often foreground
quality and third-party evaluations, with less consideration of preference and
first-person experiences. In contrast, our evaluations cover a broader range
of evaluation dimensions by working with richer user interactions (e.g.,
users’ keystrokes to type in prompts, button clicks for querying systems, the
appearance of a popup box for showing completions to users, and users’ cursor
movements to edit after getting completions) along with timestamps in
_interaction traces_. In HCI, interaction traces have been used to reason
about the helpfulness of completions (Roemmele and Gordon, 2018; Clark et al.,
2018), the time that humans take to work with given completions (Buschek et
al., 2021), and the language, ideation, and collaboration capabilities of the
systems (Lee et al., 2022). Our goal is to study the process of human-LM
interaction through interaction traces and compare interactive performance to
non-interactive performance.

##### Interactive systems.

Human interaction with user-facing intelligent systems has been explored in
many disparate communities (see Bommasani et al., 2021, §2.5). Examples
include work in information retrieval and search engines (Salton, 1970; Belkin
et al., 1982; Kuhlthau, 1991; Ingwersen, 1992; Marchionini, 2006; Micarelli et
al., 2007; White and Roth, 2009; Kelly, 2009; Croft, 2019), recommender
systems (Goldberg et al., 1992; Konstan, 2004), voice assistants (Kamm, 1994;
Cohen et al., 2004; Harris, 2004), human-robot interaction and assistive
technologies (Hadfield-Menell et al., 2016; Mataric, 2017; Xie et al., 2020;
Jeon et al., 2020), and broadly as a means for user creativity and expression
(Fede et al., 2022; Lee et al., 2022). Relative to these works, we study how
the unique aspects of LMs mediate user interaction with language generation
systems.

When interaction has been consider in prior work, many works understand
interaction to be a feedback signal for model improvement. Examples include
work in the active learning (Settles, 2012) and language learning (Wang et
al., 2016) communities, with ter Hoeve et al. (2021) specifically proposing
interactive language modeling to improve learning. In contrast to these
learning-from-interaction settings, while we do consider the role of in-
context learning in text summarization (Section 3.4), our focus is on the
broader user experience with LMs (Yang et al., 2019).

##  5 Discussion

Benchmarking LMs in interactive settings introduces new and unique challenges.
In this section, we share the challenges we encountered while designing both
our tasks and systems as well as working with users. Given these experiences,
we also discuss potential solutions and paths forward.

###  5.1 Low latency matters

Latency significantly influences human-LM interaction and how humans perceive
models. Guidelines in HCI research recommend that interactive systems respond
within 0.1 seconds to appear instantaneous and within 1 second to avoid
interrupting the user’s flow of thought (Miller, 1968; Card et al., 1991;
Nielsen, 1994). When we conducted our user study, some models were simply too
slow (e.g., 50x slower than Davinci), which significantly influenced user
perception and eventually caused us to exclude these models. Meeting these
latency standards can be a challenge, potentially exacerbated by the growing
scale of existing LMs. With that said, the observed latency is largely
determined by the factors beyond model scale alone (e.g., allocated compute
resources, query optimization, API support) as observed by Liang et al. (2022,
§4.9). Overall, we underscore that latency is crucial to positive user
experience for deploying human-LM interaction in the real world.

###  5.2 Complexity of interactive study design

User study design requires researchers to account for individual difference:
we empirically observe significant heterogeneity in how users qualitatively
experience and interact with LMs. Due to these individual effects, especially
when the number of recruited users is not especially large, it is generally
desirable for each user to interact with most/all models. Consequently, this
would requires the set of models themselves to be decided upon in advance.
This is in contrast to many non-interactive settings, where the user-level
heterogeneity may be more negligible. If users interact with multiple models,
sequential effects need to be controlled for (e.g., through the randomization
of model order across users). Otherwise, results may be subject to undesired
confounding from the _novelty effect_ (i.e., initial performance improvement
due to the increased interest in the new technology) and _user adaptation_
(i.e., performance improvement over time due to the increased familiarity of
the technology). If possible, we recommend recruiting a large number of
diverse users to allow for more flexibility in selecting which models to
evaluate and to alleviate concerns of sequential effects (e.g., by having each
user only interact with one model).

###  5.3 Potential impact on users

##### Harms.

LMs are prone to generating toxic, biased, or otherwise undesirable text. When
users are exposed to this text via interaction, this can cause psychological
harm. We observe that toxic content is elicited by seemingly innocuous
prompts, even for instruction-tuned models designed to discourage this
behavior. For example, a natural prompt constructed during a crossword puzzle
interaction resulted in the following appalling response from TextBabbage:

> User: What is a young pigeon called?
> System: A young pigeon is called a n****.

We emphasize that in this setting the user’s prompts were benign, a departure
from prior work that specifically designs prompts to elicit unsafe behavior
Ganguli et al. (2022); Perez et al. (2022).

Responsible deployment of interactive human-LM systems requires a harm
mitigation strategy (Kirk et al., 2022). In the context of benchmarking,
striking a balance between extensive filtering and evaluation of model
performance remains an open problem; for the studies presented in this paper,
we filtered completions based on a keyword block list.

##### Accommodation.

One important aspect of human-AI interaction to account for is user
accommodation, or the ability of users to adapt their behavior as they learn
more about the strengths and weaknesses of a system and the underlying model.
For QA and crossword puzzles, we asked users to answer the survey question
“Did the way you chose to interact with the AI Teammate change over time? If
so, how?,” and report example responses in the Appendix. Oftentimes, user
accommodation reflected underlying properties of the models—for example, in
QA, prompts phrased as questions yield successful outputs mainly from
TextBabbage and TextDavinci, so users assisted with Davinci or Jumbo were more
likely to switch to declarative, “fill-in-the-blank” style prompting
strategies. Additionally, for some scenarios, we studied user accommodation
quantitatively by either measuring how the edit distance or query strategies
of user prompts changed over time. We found that task framing can also affect
accommodation—while for QA, users engaged in more reformulation over time and
became less inclined to copy the provided question verbatim, for crossword
puzzles, users became more likely to copy clue text over time, perhaps due to
later relying on the LM to understand unfamiliar clue texts. Because of these
results, we believe future work should closely study more long-term
interactive scenarios, where the time it takes a user to develop familiarity
with a tool is outweighed by the overall interaction period.

##### Lasting impact.

In our work, the emphasis is on the short-term interaction of users with LMs:
we evaluate this localized experience. However, human-LM interaction, and
human-AI interaction more broadly, can have more lasting and longitudinal
impacts on users. In the context of LMs specifically, we expect interactions
with LMs may influence human writing practices, opinions and beliefs, broader
perception of language technology and AI systems, and potentially many other
aspects of human experience that are mediated by language. That is, we expect
these effects could persist to environments where LMs are not present.
Existing evidence includes the works of Jakesch et al. (2022), which finds
interacting with an opinionated language model influences written opinions and
reported attitudes, and Wenker (2022), which finds that machine agency affects
the content users write. Further, Bommasani et al. (2022) explore whether
reliance on the same LMs could lead to homogenization in outcomes, which could
include homogenization and standardization of writing styles. More established
evidence for other language technologies, such as search engines, shows users’
knowledge, beliefs, and attitudes can be significantly altered by sustained
interaction with these technologies (Allam et al., 2014). Overall, especially
given the rapid deployment of LMs, including a variety of different
interfaces, we recommend that future work should actively monitor how these
interactions come to affect human language practices (e.g., writing, reading,
listening, speaking), culture, well-being, and broader society.

## Acknowledgements

We thank Eric Horvitz for the valuable feedback on the paper, Lama Ahmad and
Miles Brundage from OpenAI, and Opher Lieber, Barak Lenz, Dan Padnos and Yoav
Shoham from AI21 Labs, for their support and for providing credits to evaluate
their models. We thank the Stanford Center for Research on Foundation Models
(CRFM) and the Institute for Human-Centered Artificial Intelligence (HAI) for
support throughout this project.

## References

  * Akoury et al. (2020) Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020.  STORIUM: A dataset and evaluation platform for machine-in-the-loop story generation.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Allam et al. (2014) Ahmed Allam, Peter Johannes Schulz, Kent Nakamoto, et al. 2014.  The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: two experiments manipulating google output.  _Journal of medical internet research_ , 16(4):e2642.
  * Ammanabrolu et al. (2019) Prithviraj Ammanabrolu, William Broniec, Alex Mueller, Jeremy Paul, and Mark Riedl. 2019.  Toward automated quest generation in text-adventure games.  In _Proceedings of the 4th Workshop on Computational Creativity in Language Generation_.
  * Ashktorab et al. (2020) Zahra Ashktorab, Q. Vera Liao, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2020.  Human-AI collaboration in a cooperative game setting: Measuring social perception and outcomes.  In _Proceedings of ACM Human-Computer Interaction (CSCW)_.
  * Avinesh et al. (2018) PVS Avinesh, Carsten Binnig, Benjamin Hättasch, Christian M Meyer, and Orkan Özyurt. 2018.  Sherlock: A system for interactive summarization of large text collections.  _Proc. VLDB Endow._ , 11(12):1902–1905.
  * Bansal et al. (2021a) Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S. Weld. 2021a.  Is the most accurate AI the best teammate? optimizing AI for teamwork.  In _Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Bansal et al. (2021b) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021b.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance.  In _Conference on Human Factors in Computing Systems (CHI)_ , pages 1–16.
  * Bartolo et al. (2020) Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020.  Beat the ai: Investigating adversarial human annotation for reading comprehension.  _Transactions of the Association for Computational Linguistics (TACL)_ , 8:662–678.
  * Belkin et al. (1982) Nicholas J. Belkin, Robert N. Oddy, and Helen M. Brooks. 1982.  Ask for information retrieval: Part i. background and theory.  _J. Documentation_ , 38:61–71.
  * Bland and Altman (1995) J Martin Bland and Douglas G Altman. 1995.  Multiple significance tests: the bonferroni method.  _BMJ_ , 310(6973):170.
  * Böhm et al. (2019) Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019.  Better rewards yield better summaries: Learning to summarise without references.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3110–3120, Hong Kong, China. Association for Computational Linguistics.
  * Bommasani et al. (2022) Rishi Bommasani, Katie Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. 2022\.  Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Bowman et al. (2022) Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. 2022.  Measuring progress on scalable oversight for large language models.  _arXiv preprint arXiv:2211.03540_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Brynjolfsson (2022) Erik Brynjolfsson. 2022.  The turing trap: The promise & peril of human-like artificial intelligence.  _Daedalus_ , 151.
  * Buschek et al. (2021) Daniel Buschek, Martin Zürn, and Malin Eiband. 2021.  _The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers_. Association for Computing Machinery, New York, NY, USA.
  * Campos and Shern (2022) Jon Ander Campos and Jun Shern. 2022.  Training language models with language feedback.  In _ACL Workshop on Learning with Natural Language Supervision. 2022._
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6633–6649.
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018.  Faithful to the original: Fact-aware neural abstractive summarization.  AAAI Press.
  * Card et al. (1991) Stuart K Card, George G Robertson, and Jock D Mackinlay. 1991.  The information visualizer, an information workspace.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022.  Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing.
  * Chakrabarty et al. (2021) Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021.  MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding.  _arXiv:2103.06779 [cs]_.  ArXiv: 2103.06779.
  * Chattopadhyay et al. (2017) Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, and Devi Parikh. 2017.  Evaluating visual conversational agents via cooperative human-AI games.  In _AAAI Conference on Human Computation & Crowdsourcing_.
  * Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, T. Brants, Phillip Todd Koehn, and Tony Robinson. 2013.  One billion word benchmark for measuring progress in statistical language modeling.  In _Interspeech_.
  * Chen et al. (2019) Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. 2019.  Gmail smart compose: Real-time assisted writing.  In _International Conference on Knowledge Discovery and Data Mining (KDD)_.
  * Chen et al. (2021) Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021.  Improving faithfulness in abstractive summarization with contrast candidate generation and selection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5935–5941.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, A. Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, B. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, M. Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, S. Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, D. Luan, Hyeontaek Lim, Barret Zoph, A. Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, T. S. Pillai, Marie Pellat, Aitor Lewkowycz, E. Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, K. Meier-Hellstern, D. Eck, J. Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling language modeling with pathways.  _arXiv_.
  * Clark et al. (2018) Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018.  Creative writing with a machine in the loop: Case studies on slogans and stories.  In _23rd International Conference on Intelligent User Interfaces_.
  * Clark and Smith (2021) Elizabeth Clark and Noah A. Smith. 2021.  Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models.  In _Association for Computational Linguistics (ACL)_.
  * Clark et al. (2019) Leigh Clark, Phillip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, Joao Cabral, Cosmin Munteanu, and Benjamin Cowan. 2019.  The State of Speech in HCI: Trends, Themes and Challenges.  _Interacting with Computers_ , 31.
  * Cohen et al. (2004) Michael Cohen, James P. Giangola, and Jennifer Balogh. 2004.  _Voice User Interface Design_.  Addison-Wesley.
  * Croft (2019) W. Bruce Croft. 2019.  The importance of interaction for information retrieval.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 1–2, New York, NY, USA. Association for Computing Machinery.
  * Deriu et al. (2021) Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021.  Survey on evaluation methods for dialogue systems.  _Artificial Intelligence Review_ , 54.
  * Deriu et al. (2020 2020) Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020 2020.  Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems.  In _Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP)_.
  * Dibia et al. (2022) Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2022.  Aligning offline metrics and human judgments of value of ai-pair programmers.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Engelbart (1962) Douglas C Engelbart. 1962.  Augmenting human intellect: A conceptual framework.  Technical report, SRI Summary Report AFOSR-3223.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating Summarization Evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Fede et al. (2022) Giulia Di Fede, Davide Rocchesso, Steven P. Dow, and Salvatore Andolina. 2022.  The idea machine: LLM-based expansion, rewriting, combination, and suggestion of ideas.  In _ACM_.
  * Frans (2021) Kevin Frans. 2021.  AI charades: Language models as interactive game environments.  In _IEEE Conference on Games (CoG)_ , pages 1–2.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yushi Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom B. Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark. 2022.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _ArXiv_ , abs/2209.07858.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.  A framework for few-shot language model evaluation.  _Version v0. 0.1. Sept_.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 96–120, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna V. Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qinqin Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja vStajner, Sébastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Yi Xu, Yisi Sang, Yixin Liu, and Yufang Hou. 2022.  Gemv2: Multilingual nlg benchmarking in a single line of code.  _ArXiv_ , abs/2206.11249.
  * Gero and Chilton (2018) Katy Gero and Lydia Chilton. 2018.  Challenges in finding metaphorical connections.  In _Proceedings of the Workshop on Figurative Language Processing_.
  * Gero et al. (2020) Katy Ilonka Gero, Zahra Ashktorab, Casey Dugan, Qian Pan, James Johnson, Wener Geyer, Maria Ruiz, Sarah Miller, David R Millen, Murray Campbell, Sadhana Kumaravel, and Wei Zhang. 2020.  Mental models of AI agents in a cooperative game setting.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Gero and Chilton (2019) Katy Ilonka Gero and Lydia B Chilton. 2019.  Metaphoria: An algorithmic companion for metaphor creation.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Ginsberg (2014) Matthew L. Ginsberg. 2014.  Dr.fill: Crosswords and an implemented solver for singly weighted csps.  In _Journal Of Artificial Intelligence Research_ , pages 851–886.
  * Glass (1999) James Glass. 1999.  Challenges for spoken dialogue systems.  In _In Proceedings of the 1999 IEEE ASRU Workshop_.
  * Glucksberg and McGlone (2001) Sam Glucksberg and Matthew S McGlone. 2001.  _Understanding figurative language: From metaphor to idioms_.  Oxford University Press on Demand.
  * Goldberg et al. (1992) David Goldberg, David Nichols, Brian M Oki, and Douglas Terry. 1992.  Using collaborative filtering to weave an information tapestry.  _Communications of the ACM_ , 35.
  * Gottardi et al. (2022) Anna Gottardi, Osman Ipek, Giuseppe Castellucci, Shui Hu, Lavina Vaz, Yao Lu, Anju Khatri, Anjali Chadha, Desheng Zhang, Sattvik Sahai, Prerna Dwivedi, Hangjie Shi, Lucy Hu, Andy Huang, Luke Dai, Bofei Yang, Varun Somani, Pankaj Rajan, Ron Rezac, Michael Johnston, Savanna Stiff, Leslie Ball, David Carmel, Yang Liu, Dilek Hakkani-Tür, Oleg Rokhlenko, Kate Bland, Eugene Agichtein, Reza Ghanadan, and Yoelle Maarek. 2022.  Alexa, let’s work together: Introducing the first alexa prize taskbot challenge on conversational task assistance.  In _Alexa Prize TaskBot Challenge Proceedings_.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gunasekara et al. (2020) R. Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D’Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, Dilek Hakkani-Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Liden, Kaili Huang, Shahin Shayandeh, Runze Liang, Baolin Peng, Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David R. Traum, Maxine Eskénazi, Ahmad Beirami, Eunjoon Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, and Rajen Subba. 2020\.  Overview of the ninth dialog system technology challenge: DSTC9.  _CoRR_ , abs/2011.06486.
  * Hadfield-Menell et al. (2016) Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016.  Cooperative inverse reinforcement learning.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Hakkani-Tür (2021) Dilek Hakkani-Tür. 2021.  Alexa prize socialbot grand challenge year iv.  In _Alexa Prize SocialBot Grand Challenge 4 Proceedings_.
  * Harris (2004) R.A. Harris. 2004.  _Voice Interaction Design: Crafting the New Conversational Speech Systems_.  Morgan Kaufmann series in interactive technologies. Elsevier Science.
  * Hassani et al. (2020) Hossein Hassani, Emmanuel Sirimal Silva, Stephane Unger, Maedeh TajMazinani, and Stephen Mac Feely. 2020.  Artificial intelligence (ai) or intelligence augmentation (ia): what is the future?  _Ai_ , 1.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations (ICLR)_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations (ICLR)_.
  * Horvitz (1999) Eric Horvitz. 1999.  Principles of mixed-initiative user interfaces.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Hu et al. (2021) Shui Hu, Yang Liu, Anna Gottardi, Behnam Hedayatnia, Anju Khatri, Anjali Chadha, Qinlang Chen, Pankaj Rajan, Ali Binici, Varun Somani, Yao Lu, Prerna Dwivedi, Lucy Hu, Hangjie Shi, Sattvik Sahai, Mihail Eric, Karthik Gopalakrishnan, Seokhwan Kim, Spandana Gella, Alexandros Papangelis, Patrick Lange, Di Jin, Nicole Chartier, Mahdi Namazifar, Aishwarya Padmakumar, Sarik Ghazarian, Shereen Oraby, Anjali Narayan-Chen, Yuheng Du, Lauren Stubell, Savanna Stiff, Kate Bland, Arindam Mandal, Reza Ghanadan, and Dilek Hakkani-Tür. 2021.  Further advances in open domain dialog systems in the fourth alexa prize socialbot grand challenge.  In _Alexa Prize SocialBot Grand Challenge 4 Proceedings_.
  * Hua and Raley (2020) Minh Hua and Rita Raley. 2020.  Playing with unicorns: AI dungeon and citizen NLP.  In _Digital Humanities Quarterly_.
  * Huang and Efthimiadis (2009a) Jeff Huang and Efthimis N Efthimiadis. 2009a.  Analyzing and evaluating query reformulation strategies in web search logs.  In _Proceedings of the 18th ACM conference on Information and knowledge management_ , pages 77–86.
  * Huang and Efthimiadis (2009b) Jeff Huang and Efthimis N. Efthimiadis. 2009b.  Analyzing and evaluating query reformulation strategies in web search logs.  In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_.
  * Huang et al. (2020) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020.  Challenges in building intelligent open-domain dialog systems.  _ACM Transactions on Information Systems (TOIS)_ , 38.
  * Ingwersen (1992) Peter Ingwersen. 1992.  _Information retrieval interaction_ , volume 246.
  * Ippolito et al. (2022) Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022.  Creative writing with an ai-powered writing assistant: Perspectives from professional writers.  _arXiv preprint arXiv:2211.05030_.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?  In _Association for Computational Linguistics (ACL)_.
  * Jakesch et al. (2022) Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2022\.  Interacting with opinionated language models changes users’ views.
  * Jansen et al. (2009) Bernard J Jansen, Danielle L Booth, and Amanda Spink. 2009.  Patterns of query reformulation during web searching.  _Journal of the american society for information science and technology_ , 60(7):1358–1371.
  * Jeon et al. (2020) Hong Jun Jeon, Dylan P. Losey, and Dorsa Sadigh. 2020.  Shared autonomy with learned latent actions.  _ArXiv_ , abs/2005.03210.
  * Ji et al. (2022) Tianbo Ji, Yvette Graham, Gareth Jones, Chenyang Lyu, and Qun Liu. 2022.  Achieving reliable human assessment of open-domain dialogue systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6416–6437, Dublin, Ireland. Association for Computational Linguistics.
  * Jiang et al. (2013) Jiepu Jiang, Wei Jeng, and Daqing He. 2013.  How do users respond to voice input errors? lexical and phonetic query reformulation in voice search.  In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_ , pages 143–152.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Association for Computational Linguistics (ACL)_.
  * Kamm (1994) Candace Kamm. 1994.  _User Interfaces for Voice Applications_ , page 422–442. National Academy Press, USA.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori B Hashimoto. 2020.  Improved natural language generation via loss truncation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 718–731.
  * Kelly (2009) Diane Kelly. 2009.  Methods for evaluating interactive information retrieval systems with users.  _Foundations and Trends® in Information Retrieval_ , 3.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _North American Association for Computational Linguistics (NAACL)_ , pages 4110–4124.
  * Kirk et al. (2022) Hannah Rose Kirk, Abeba Birhane, Bertie Vidgen, and Leon Derczynski. 2022.  Handling and presenting harmful text in nlp research.
  * Konstan (2004) Joseph A Konstan. 2004.  Introduction to recommender systems: Algorithms and evaluation.  _ACM Transactions on Information Systems (TOIS)_ , 22.
  * Kuhlthau (1991) Carol C. Kuhlthau. 1991.  Inside the search process: Information seeking from the user’s perspective.  _Journal of the American Society for Information Science_ , 42(5):361–371.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  In _Association for Computational Linguistics (ACL)_.
  * Ladhak et al. (2022) Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown. 2022.  Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1410–1421, Dublin, Ireland. Association for Computational Linguistics.
  * Lakkaraju et al. (2019) Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Lescovec. 2019.  Faithful and customizable explanations of black box models.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_.
  * Lakoff and Johnson (2008) George Lakoff and Mark Johnson. 2008.  _Metaphors we live by_.  University of Chicago press.
  * Lakoff and Turner (2009) George Lakoff and Mark Turner. 2009.  _More than cool reason: A field guide to poetic metaphor_.  University of Chicago press.
  * Langevin et al. (2021) Raina Langevin, Ross J Lordon, Thi Avrahami, Benjamin R Cowan, Tad Hirsch, and Gary Hsieh. 2021.  Heuristic evaluation of conversational agents.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Lau and Horvitz (1999) Tessa Lau and Eric Horvitz. 1999.  Patterns of search: Analyzing and modeling web query refinement.  In _User Modeling_ , pages 119–128.
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  CoAuthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities.  In _Conference on Human Factors in Computing Systems (CHI)_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Liberman (2010) Mark Liberman. 2010.  Obituary: Fred jelinek.  _Comput. Linguist._ , 36(4):595–599.
  * Lieber et al. (2021) Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021.  Jurassic-1: Technical details and evaluation.  _White Paper, AI21 Labs_.
  * Littman et al. (2002) Michael L. Littman, Greg A. Keim, and Noam Shazeer. 2002.  A probabilistic approach to solving crossword puzzles.  In _Artificial Intelligence_ , pages 23–55.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Liu et al. (2013) Jian Liu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2013.  How do users grow up along with search engines?: A study of long-term users’ behavior.  In _Proceedings of the 22nd ACM Conference on Information and Knowledge Management_.
  * Luhn (1958) Henry P. Luhn. 1958.  The automatic creation of literature abstracts.  _IBM Journal of Research and Development_ , 2:159–165.
  * Mani (1999) Inderjeet Mani. 1999.  _Advances in Automatic Text Summarization_.  MIT Press, Cambridge, MA, USA.
  * Mani et al. (1999) Inderjeet Mani, Gary Klein, Lynette Hirschman, Therese Firmin, David House, and Beth Sundheim. 1999.  The TIPSTER SUMMAC text summarization evaluation.  In _European Association for Computational Linguistics (EACL)_.
  * Marchionini (2006) Gary Marchionini. 2006.  Exploratory search: From finding to understanding.  _Commun. ACM_ , 49(4):41–46.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Mataric (2017) Maja J Mataric. 2017.  Socially assistive robotics: Human augmentation versus automation.  _Science_ , 2.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_.
  * Micarelli et al. (2007) Alessandro Micarelli, Fabio Gasparetti, Filippo Sciarrone, and Susan Gauch. 2007\.  Personalized search on the world wide web.  In _The adaptive web_ , pages 195–230. Springer.
  * Miller (1968) Robert B Miller. 1968.  Response time in man-computer conversational transactions.  In _Proceedings of the December 9-11, 1968, fall joint computer conference, part I_.
  * Mio (1997) Jeffery Scott Mio. 1997.  Metaphor and politics.  _Metaphor and symbol_ , 12.
  * Mittal et al. (2022) Anirudh Mittal, Yufei Tian, and Nanyun Peng. 2022.  AmbiPun: Generating Humorous Puns with Ambiguous Context.  ArXiv:2205.01825 [cs].
  * Nan et al. (2021a) Feng Nan, Cicero dos Santos, Henghui Zhu, Patrick Ng, Kathleen Mckeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O Arnold, and Bing Xiang. 2021a.  Improving factual consistency of abstractive summarization via question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6881–6894.
  * Nan et al. (2021b) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen Mckeown, and Bing Xiang. 2021b.  Entity-level factual consistency of abstractive text summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Nenkova and McKeown (2012) Ani Nenkova and Kathleen McKeown. 2012.  A survey of text summarization techniques.  In _Mining text data_ , pages 43–76. Springer.
  * New York Times (2022) New York Times. 2022.  The brilliance and weirdness of chatgpt.  https://www.nytimes.com/2022/12/05/technology/chatgpt-ai-twitter.html.  Accessed: 2022-12-07.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  In _Workshop on Cognitive Computing at NIPS_.
  * Niebert et al. (2012) Kai Niebert, Sabine Marsch, and David F Treagust. 2012.  Understanding needs embodiment: A theory-guided reanalysis of the role of metaphors and analogies in understanding science.  _Science_ , 96.
  * Nielsen (1994) Jakob Nielsen. 1994.  _Usability engineering_.  Morgan Kaufmann.
  * Nogueira and Cho (2017) Rodrigo Nogueira and Kyunghyun Cho. 2017.  Task-oriented query reformulation with reinforcement learning.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * OpenAI (2021) OpenAI. 2021.  Gpt-3 powers the next generation of apps.  https://openai.com/blog/gpt-3-apps/.
  * OpenAI (2022) OpenAI. 2022.  Model index for researchers.  https://beta.openai.com/docs/model-index-for-researchers.  Accessed: 2022-12-16.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, J. Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, P. Welinder, P. Christiano, J. Leike, and Ryan J. Lowe. 2022.  Training language models to follow instructions with human feedback.  _arXiv_.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Pang and Kumar (2011) Bo Pang and Ravi Kumar. 2011.  Search in the lost sense of “query”: Question formulation in web search queries and its temporal changes.  In _Association for Computational Linguistics (ACL)_.
  * Paranjape et al. (2020) Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020.  Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations.  _arXiv preprint arXiv:2008.12348_.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_.
  * Peyrard (2019) Maxime Peyrard. 2019.  A simple theoretical model of importance for summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1059–1073, Florence, Italy. Association for Computational Linguistics.
  * R Core Team (2020) R Core Team. 2020.  _R: A Language and Environment for Statistical Computing_.  R Foundation for Statistical Computing, Vienna, Austria.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling language models: Methods, analysis & insignts from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: a new benchmark and dataset.  In _Association for Computational Linguistics (ACL)_.
  * Roemmele and Gordon (2018) Melissa Roemmele and Andrew S. Gordon. 2018.  Automated assistance for creative writing with an rnn language model.  _Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion_.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Association for Computational Linguistics (ACL)_.
  * Rozner et al. (2021) Joshua Rozner, Christopher Potts, and Kyle Mahowald. 2021.  Decrypting cryptic crosswords: Semantically complex wordplay.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Salton (1970) G. Salton. 1970.  Evaluation problems in interactive information retrieval.  _Information Storage and Retrieval_ , 6(1):29–44.
  * Schulman et al. (2022) John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak, Valerie Balcom, David Schnurr, Tomer Kaftan, Chris Hallacy, Nicholas Turley, Noah Deutsch, and Vik Goel. 2022.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt/.
  * Settles (2012) Burr Settles. 2012.  Active learning.  _Synthesis lectures on artificial intelligence and machine learning_ , 6.
  * Shapira et al. (2022) Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Ido Dagan, and Yael Amsterdamer. 2022\.  Interactive query-assisted summarization via deep reinforcement learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2551–2568, Seattle, United States. Association for Computational Linguistics.
  * Shapira et al. (2021) Ori Shapira, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2021.  Extending multi-document summarization evaluation to the interactive setting.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 657–677, Online. Association for Computational Linguistics.
  * Shneiderman (2022) B. Shneiderman. 2022.  _Human-Centered AI_.  OUP Oxford.
  * Shneiderman and Maes (1997) Ben Shneiderman and Pattie Maes. 1997.  Direct manipulation vs. interface agents.  _ACM Interactions_.
  * Shuster et al. (2022) Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022.  Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage.
  * Skagestad (1993) Peter Skagestad. 1993.  Thinking with machines: Intelligence augmentation, evolutionary epistemology, and semiotic.  _Journal of Social and Evolutionary Systems_ , 16.
  * Skagestad (1996) Peter Skagestad. 1996.  The mind’s machines: The turing machine, the memex, and the personal computer.  _Semiotica_ , 111.
  * Smith et al. (2022a) Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022a.  Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents.  In _Proceedings of the 4th Workshop on NLP for Conversational AI_.
  * Smith et al. (2022b) Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022b.  Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents.  In _Proceedings of the 4th Workshop on NLP for Conversational AI_ , pages 77–97, Dublin, Ireland. Association for Computational Linguistics.
  * Spärck Jones (1999) Karen Spärck Jones. 1999.  Automatic summarizing: factors and directions.  _Advances in automatic text summarization_ , pages 1–12.
  * Spärck Jones and Galliers (1995) Karen Spärck Jones and Julia R. Galliers. 1995.  _Evaluating Natural Language Processing Systems: An Analysis and Review_.  Number 1083 in Lecture Notes in Computer Science. Springer Verlag, Berlin.
  * Spink and Ozmultu (2002) Amanda Spink and H. Cenk Ozmultu. 2002.  Characteristics of question format web queries: an exploratory study.  In _Information Processing & Management_.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C’esar Ferri Ram’irez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz’alez, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fatemeh Siar, Fernando Mart’inez-Plumed, Francesca Happ’e, François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-L’opez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik-Soon Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco’n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg, Jos Rozen, José Hernández-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng’ Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col’on, Luke Metz, Lutfi Kerem cSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, M’aty’as Schubert, Medina Baitemirova, Melissa Arnaud, Melvin Andrew McElrath, Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pi-Bei Hwang, P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ram’on Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Sameh Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, T. N. Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler O. Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay V. Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, W Vossen, Xiang Ren, Xiaoyu F Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song, Yasaman Bahri, Ye Ji Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yushi Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _ArXiv_ , abs/2206.04615.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020.  Learning to summarize from human feedback.  _CoRR_ , abs/2009.01325.
  * Stowe et al. (2021a) Kevin Stowe, Nils Beck, and Iryna Gurevych. 2021a.  Exploring Metaphoric Paraphrase Generation.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 323–336, Online. Association for Computational Linguistics.
  * Stowe et al. (2021b) Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. 2021b.  Metaphor Generation with Conceptual Mappings.  _arXiv:2106.01228 [cs]_.  ArXiv: 2106.01228.
  * Teevan et al. (2007) Jaime Teevan, Eytan Adar, Rosie Jones, and Michael A. S. Potts. 2007.  Information re-retrieval: Repeat queries in yahoo’s logs.  In _Annual Conference of the Association for Computing Machinery Special Interest Group in Information Retrieval_.
  * ter Hoeve et al. (2021) Maartje ter Hoeve, Evgeny Kharitonov, Dieuwke Hupkes, and Emmanuel Dupoux. 2021\.  Towards interactive language modeling.  _ArXiv_ , abs/2112.11911.
  * Thomas et al. (2020) Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2020.  Expressions of style in information seeking conversation with an agent.  In _ACM Special Interest Group on Information Retreival (SIGIR)_.
  * Thoppilan et al. (2022a) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022a.  LaMDA: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Thoppilan et al. (2022b) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Díaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022b.  Lamda: Language models for dialog applications.  _ArXiv_ , abs/2201.08239.
  * Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktaschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019\.  Learning to speak and act in a fantasy text adventure game.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272.
  * Wallace et al. (2022a) Eric Wallace, Nicholas Tomlin, Albert Xu, Kevin Yang, Eshaan Pathak, Matthew Ginsberg, and Dan Klein. 2022a.  Automated crossword solving.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3073–3085, Dublin, Ireland. Association for Computational Linguistics.