# The Illusion of Artificial Inclusion

William Agnew  0000-0002-1362-554X Carnegie Mellon UniversityPittsburghPAUSA ,
A. Stevie Bergman  0000-0002-4331-1357 Google DeepMindNew YorkNYUSA ,
Jennifer Chien  0009-0009-8768-1761 University of California San DiegoSan
DiegoCAUSA ,  Mark Díaz  0000-0003-0167-9839 Google ResearchNew YorkNYUSA ,
Seliem El-Sayed  0000-0003-4819-1136 Google DeepMindLondonUK ,  Jaylen Pittman
0000-0002-5176-5922 Stanford UniversityStanfordCAUSA ,  Shakir Mohamed
0000-0002-1184-5776 Google DeepMindLondonUK and  Kevin R. McKee
0000-0002-4412-1686 Google DeepMindLondonUK

(2024)

###### Abstract.

Human participants play a central role in the development of modern artificial
intelligence (AI) technology, in psychological science, and in user research.
Recent advances in generative AI have attracted growing interest to the
possibility of replacing human participants in these domains with AI
surrogates. We survey several such “substitution proposals” to better
understand the arguments for and against substituting human participants with
modern generative AI. Our scoping review indicates that the recent wave of
these proposals is motivated by goals such as reducing the costs of research
and development work and increasing the diversity of collected data. However,
these proposals ignore and ultimately conflict with foundational values of
work with human participants: representation, inclusion, and understanding.
This paper critically examines the principles and goals underlying human
participation to help chart out paths for future work that truly centers and
empowers participants.

Human participants, language models, generative AI, participation, user
research, AI development, representation, inclusion, understanding

††copyright: rightsretained††journalyear: 2024††doi:
10.1145/3613904.3642703††conference: Proceedings of the CHI Conference on
Human Factors in Computing Systems; May 11–16, 2024; Honolulu, HI,
USA††booktitle: Proceedings of the CHI Conference on Human Factors in
Computing Systems (CHI ’24), May 11–16, 2024, Honolulu, HI, USA††price:
15.00††isbn: 979-8-4007-0330-0/24/05††ccs: Human-centered computing Human
computer interaction (HCI)††ccs: Computing methodologies Natural language
generation††ccs: Applied computing Psychology††ccs: Applied computing
Sociology††ccs: Applied computing Economics

##  1\. Introduction

Participation is a foundational element of the social-behavioral sciences and
in the design of new technology. In psychology, user research, human-computer
interaction (HCI), and other related fields, research participants offer a
window into human cognition and decision making. In the development of new
technologies, human participants ground the design process in real-life needs,
perspectives, and experiences.

The past year has seen a growing number of proposals recommending the
replacement of human participants in technology development and scientific
research with large language models (LLMs), a new class of artificial
intelligence (AI) systems. These proposals include at least thirteen technical
reports and peer-reviewed research articles, jointly sharing over one thousand
citations at the time of this manuscript’s submission (Aher et al., 2023;
Argyle et al., 2023; Bai et al., 2022; Byun et al., 2023; Chiang and Lee,
2023; Dillion et al., 2023; Gerosa et al., 2023; Gilardi et al., 2023;
Hämäläinen et al., 2023; Heyman and Heyman, 2023; Horton, 2023; Park et al.,
2022; Wang et al., 2021), in addition to several commercial products
(OpinioAI, [n. d.]; Synthetic Users, [n. d.]; User Persona, [n. d.]).

The momentum building behind these proposals echoes broader social reactions
to modern AI systems: LLMs have shocked and fascinated both AI researchers and
the general public with their ability to produce fluent, human-like text in a
range of domains (Roose, 2023; Thompson, 2022; Tiku, 2022). In this paper, we
seek to understand and contextualize the arguments for replacing human
participants with LLMs and other generative AI systems. Our thematic content
analysis suggests that these proposals are motivated by aims such as reducing
the costs of scientific research, protecting participants from potential
harms, and augmenting diversity in data collection. However, a critical
examination of the social structures and epistemic frameworks surrounding
human participation reveals fundamental conflicts between these substitution
proposals and the values underpinning research and development. In particular,
substitution undermines three values: the representation of participants’
interests; participants’ inclusion and empowerment in the development process;
and the understanding that researchers otherwise develop through
intersubjective engagement with participants. AI-generated substitutes cannot
meaningfully support technological development and scientific research in the
absence of a deeper reckoning with their relationship to representation,
inclusion, and understanding. While current proposals undermine these values,
there may be space for scientific and engineering communities to re-imagine
ways for generative AI to support human participation. Overall, our analysis
underscores the need for researchers and developers to take action at every
stage of their work to center and empower participants and the communities to
which they belong.

##  2\. Background

###  2.1. Historic applications of artificial intelligence in social-
behavioral research

Advances in artificial intelligence closely intertwine with the histories of
psychology and user research (Grudin, 2009, 2022; Newell, 1982). As early as
the 1980s, scientists envisioned a key role for AI in supporting psychology
research (e.g., (Schmidt et al., 1978; Van Cuilenburg et al., 1988)), as well
as enabling technological development and advancing user experience (e.g.,
(Belkin et al., 1987; Bing, 1986)). Over the following decades, the increasing
availability of machine learning—AI techniques that solve tasks by adapting to
incoming data, in contrast to early rule-based approaches—opened up new
possibilities for AI applications in user research and in neighboring social-
behavioral fields. Early machine learning algorithms were relatively
rudimentary, but nonetheless enabled the automation of various tasks in the
research process. For example, new AI tools allowed scientists to quickly
analyze sentiment in open-ended text data (Pang et al., 2002) and to make
fine-grained predictions of user preferences (Heckerman et al., 2000).

Today, applications of AI pervade throughout user research and psychology.
Many occurrences likely seem pedestrian, including common statistical
techniques such as regression, factor analysis, and classification. The
research community has also pushed a number of boundaries with new, creative
applications of AI. For example, researchers have leveraged machine learning
to automate the development of user personas (Zhang et al., 2016) and
accelerate design iteration (Buschek et al., 2020). In psychology, AI methods
have helped to scale the coverage of traditional experimental tools (Nicolas
et al., 2021) and explore novel solutions to persistent behavioral puzzles
(McKee et al., 2023). Overall, recent innovations expand the focus of AI
applications from predominantly supporting analysis to aiding the full
discovery process (Stige et al., 2023).

###  2.2. The emergence and capabilities of large language models

Recent breakthroughs in AI research, most notably the emergence of large
language models (LLMs; e.g., (Brown et al., 2020; Chowdhery et al., 2022;
Devlin et al., 2019)), have invigorated new interest in real-world
applications of AI systems. At their core, LLMs are natural language
processing (NLP) systems: a user can prompt an LLM with text, and the LLM will
output text in response. Developing an AI system capable of proficiently
conversing with humans has been a longstanding aspiration of AI researchers
(e.g., (Turing, 1950)), and NLP systems have existed for decades, in forms
including chatbots, text completion, and search engines. However, LLMs
represent a substantial breakthrough in language capabilities for AI systems.
LLMs can generate text that is fluent, grammatically correct, and surprisingly
human-like, in response to a dizzying range of inputs (e.g., (OpenAI, 2023)).
This flexibility enables them to perform remarkably well in applications
including summarizing books (Anthropic, 2023a), generating computer code
(Schulman, 2022), and conversing with users for some specific tasks (Thoppilan
et al., 2022), though serious problems persist (e.g., (Ji et al., 2023)). Some
contemporary models can process and generate data in multiple formats,
including not just text, but also images and audio (Alayrac et al., 2022).
These multimodal models are often grouped with language-only models in the
broader category of “generative AI” (Stokel-Walker and Van Noorden, 2023).

The capabilities of LLMs and generative AI have fascinated researchers,
policymakers, and members of the public (Novak, 2023; Roose, 2023; Tiku, 2022;
Vincent, 2022). LLMs have also courted controversy, particularly over claims
that they “understand” natural language (e.g., (Manning, 2022; y Arcas, 2022))
and exhibit signs of general intelligence (e.g., (Bubeck et al., 2023)).
Critics argue that such claims of understanding and general intelligence
generate overconfidence in model abilities, which in turn could deter efforts
at oversight and encourage unsafe or inappropriate applications (Bender et
al., 2021; Shah and Bender, 2022; Weidinger et al., 2022). Regardless of these
grander debates, in their capability to produce fluent and human-like text for
various tasks in English and other highly datafied languages, LLMs represent a
large advance over prior approaches to NLP.

###  2.3. Human participants in the context of AI development and social-
behavioral research

Humans interact with the AI development lifecycle at multiple points (National
Institute of Standards and Technology, 2023), including by driving the
development process itself (as scientists, engineers, and other development
team members); contributing data to train models (Hutchinson et al., 2021;
Paullada et al., 2021; Rolf et al., 2021; Sambasivan, 2021; Sambasivan et al.,
2021); evaluating models as annotators, auditors, and testers (Bergman and
Diab, 2022; McKee, 2023; Partnership on AI, [n. d.]); and reviewing products
and outcomes as system stakeholders (Fazelpour and De-Arteaga, 2022).
Similarly, in psychology, user research, and other social-behavioral fields,
humans play key roles across multiple stages of the research process,
including in ideation, hypothesis generation, and study design (Coan, 1979;
Jamieson et al., 2023); data provision and collection (Danziger, 1994); and
the interpretation of study results (Ziman, 1991).

Across both of these domains, “human participants” are the class of people who
provide information and feedback to development work and research projects,
typically as distinguished from the developers and researchers
themselves.111This distinction is stronger today than at various points in the
past. Notably, early efforts in psychology and supervised machine learning
often blurred the roles of researcher and participant. Before the advent of
online participant platforms, for instance, NLP research teams frequently took
responsibility for annotating training datasets themselves (Shmueli et al.,
2021). Similarly, in the earliest years of psychology research, scientists
often served the roles of both investigator and participant (Walsh, 2014). In
AI development, they are the individuals “whose judgment and intelligence are
widely employed […] to train and validate models” (Partnership on AI, [n.
d.]), as well as members of communities who review outcomes as project
stakeholders (e.g., (Zytko et al., 2022)). In the behavioral and social
sciences, human participants are people who provide information and data to
researchers, particularly in systematic investigations intended to produce
generalizable insights and knowledge (U.S. Department of Health and Human
Services, 2018). Traditionally, research participants interact with scientists
through channels including laboratory studies and experiments, focus groups,
interviews, and surveys. For a detailed historical examination of the role of
human participants in social and behavioral research, see (Danziger, 1994).

##  3\. Reviewing substitution proposals and their stated motivations

Enthusiasm for the capabilities of LLMs has motivated a wave of real-world
applications of AI, including systems designed to assist scientists and
engineers. For instance, several recent models offer support to research
scientists and AI developers through article summarization (Anthropic, 2023b)
and code generation (Chen et al., 2021). A nascent class of these
proposals—emerging across academic papers, technical reports, and startup
products—suggests using LLMs to take over the roles of human participants in
research and development. The defining feature of these “substitution
proposals” is the idea that LLMs can simulate or replicate the cognition,
decision making, and attitudes of human participants in order to fulfill a
role that participants normally play in a project.222Notably, this scope
excludes social simulacra projects that study LLMs as social or intelligent
agents in themselves (cf. (Levy, 1992)), rather than as a replacement for
human participants in research and development. For instance, Park et al.
(2023) investigate the emergent social dynamics between LLM-based agents
rather than testing any predictions about human behavior or replacing inputs
to development. For example, to empirically test the possibility of
substitution in HCI, Hämäläinen et al. (2023) collected materials from a
previous user study with human participants and reformulated them as
appropriate inputs for an LLM. The authors passed these inputs to an LLM,
parsed the model’s outputs as synthetic user responses, and then analyzed and
interpreted those responses as they would from human participants.

We conducted a scoping review to better understand the arguments motivating AI
substitution, sampling and evaluating a set of prominent papers and sources
that advance substitution proposals. Our goal was to survey the different
arguments offered for substitution by identifying the motivations set out in
the text of the proposals themselves. To conduct this review, we collected an
initial purposive sample of proposals, then leveraged snowball sampling to
expand to a final set for analysis (Atkinson and Flint, 2001). Our initial
sample included eight proposals (Byun et al., 2023; Dillion et al., 2023;
Gilardi et al., 2023; Horton, 2023; OpinioAI, [n. d.]; Synthetic Users, [n.
d.]; User Persona, [n. d.]; Wang et al., 2021), each of which gained
visibility after publication in disciplinary venues that we monitor (e.g.,
Proceedings of the National Academy of Sciences and Trends in Cognitive
Sciences) or through discussion on social media platforms (e.g., (@infoxiao,
2023; @schock, 2023)). We subsequently employed snowball sampling to expand
our survey, iteratively identifying additional proposals cited by the sources
in our sample (backward connections) and referring to sources in our sample
(forward connections). We focused the review on proposals with a demonstrated
interest in testing or pursuing AI substitution, excluding sources that
conceptually discuss substitution, but that do not empirically explore the
idea. Our final sample comprised sixteen sources, including thirteen technical
reports and research articles, and three commercial products (Table 1).

Proposal | Motivation(s) discussed | Position | Application domain | Venue
---|---|---|---|---
Aher et al. (2023) |  | Increase speed,
---
reduce costs,
augment diversity,
protect participants
In favor | Behavioral science |  | Int. Conf. Mach.
---
Learn. (ICML)
Argyle et al. (2023) | Augment diversity | In favor | Survey research | Political Anal.
Bai et al. (2022) |  | Increase speed,
---
protect participants
In favor | AI development | arXiv*
Byun et al. (2023) |  | Increase speed,
---
augment diversity,
protect participants
Mixed |  | Human-computer
---
interaction
| Conf. Hum. Factors
---
Comput. Syst. (CHI)
Chiang and Lee (2023) |  | Increase speed,
---
reduce costs,
protect participants
In favor | AI development |  | Annu. Meet. Assoc.
---
Comput. Linguist. (ACL)
Dillion et al. (2023) | Increase speed | Mixed | Psychological science | Trends Cogn. Sci.
Gerosa et al. (2023) |  | Increase speed,
---
augment diversity,
protect participants
Mixed | Software engineering | arXiv*
Gilardi et al. (2023) | Reduce costs | In favor | AI development | Proc. Natl. Acad. Sci.
Hämäläinen et al. (2023) |  | Increase speed,
---
reduce costs,
augment diversity
Mixed |  | Human-computer
---
interaction
| Conf. Hum. Factors
---
Comput. Syst. (CHI)
Heyman and Heyman (2023) | Increase speed | Critical | Psychological science | Behav. Res. Methods
Horton (2023) |  | Increase speed,
---
reduce costs
In favor | Experimental economics | Natl. Bur. Econ. Res.*
OpinioAI ([n. d.]) |  | Increase speed,
---
reduce costs,
augment diversity
In favor | Marketing research | Product website
Park et al. (2022) |  | Increase speed,
---
protect participants
Mixed |  | Human-computer
---
interaction
| Symp. User Interface
---
Softw. Technol. (UIST)
Synthetic Users ([n. d.]) |  | Increase speed,
---
reduce costs,
augment diversity
In favor | User research | Product website
User Persona ([n. d.]) |  | Increase speed,
---
augment diversity
In favor | Marketing research | Product website
Wang et al. (2021) |  | Increase speed,
---
reduce costs
In favor | AI development |  | Empir. Methods Nat.
---
Lang. Process. (EMNLP)
Table 1. A summary of the surveyed proposals to substitute human participants
in research and development with large language models (LLMs) and generative
artificial intelligence (AI). The “Motivation(s) discussed” column lists the
potential goals of substitution explicitly discussed by each source.
“Position” reflects whether the source overall voices support for simulating
human participants, criticism for it, or a combination of optimism and
concern. “Application domain” describes the field or area in which the source
considers simulating participation. For papers and technical reports, “Venue”
provides the ISO 4 abbreviation for the conference, journal, or repository
where the proposal was shared, with common abbreviations in parentheses;
asterisks indicate technical reports. For products, “Venue” simply states
“product website”.

These substitution proposals focus on human participation in AI development
(Chiang and Lee, 2023; Gilardi et al., 2023) and in a range of scientific
fields including user research (Hämäläinen et al., 2023; Byun et al., 2023;
Synthetic Users, [n. d.]), social and cognitive psychology (Dillion et al.,
2023), and survey research (Argyle et al., 2023). While we do not claim that
this survey represents an exhaustive review, the diversity of the surveyed
sources helps provide a broad perspective on the idea of substituting
generative AI for human participation. If substitution proposals continue to
emerge over time, future research could test the representativeness of our
survey themes through a systematic literature review.

###  3.1. Evaluating manifest motivations for substitution

We take an iterative, inductive, and semantic approach to code the themes in
this sample of substitution proposals. Our protocol aims to identify themes
through a process of repeated review and refinement, without imposing pre-
existing concepts or categories (iterative and inductive (Braun and Clarke,
2006)), focusing on the explicit contents of the text (semantic; also known as
“manifest content” (Berelson, 1952; Potter and Levine-Donnerstein, 1999)). We
independently read the proposals, noting any quality concerns that might
affect the depth or rigor of our analysis. We separately coded passages that
communicate explicit motivations for substitution, then jointly discussed and
organized these text data into themes. We returned to the text of the sources
during these discussions to ensure alignment on our final thematic codes. We
also discussed and coded the overall position that each source adopted on AI
substitution: overall in favor (offering limited or no reservations about
substitution), overall critical (indicating limited or no endorsement of
substitution), or mixed (sharing meaningful support alongside meaningful
critique or concern).

Table 1 summarizes our overall findings. As a preliminary step for our
analysis, we found the sources in our review to be generally clear and
coherent. Our sample included, for example, ten research articles that
indicated undergoing and passing academic peer review. They, along with the
three non-peer reviewed technical reports and the three product websites, were
well-written and easy to follow, communicating their arguments effectively.
Most sources articulated a position somewhere between guarded excitement
(“Practically speaking, LLMs may be most useful as participants when studying
specific topics, when using specific tasks, at specific research stages, and
when simulating specific samples” (Dillion et al., 2023)) and unbridled
optimism (“These language models can be used prior to or in the absence of
human data” (Argyle et al., 2023)). We coded 10 sources as overall in favor of
substitution, five with a mixture of support and concern, and only one overall
critical of the idea.

Every article communicated at least one potential motivation for simulating
participants with LLMs. Across the entire sample, our thematic analysis
identified four recurring motivations for replacing human participants with
generative AI.

####  3.1.1. “Substitution increases the speed and scale of research and
development”

Fourteen proposals—nearly every source in our sample—discuss the possibility
of increasing the speed and scale of research or development as a motivation
for replacing participants with LLMs. For instance, Wang et al. (2021) cite
the “time-consuming and labor-intensive” nature of work with human
participants as crucial context for preferring to employ state-of-the-art
models, since they “can label data non-stoppingly at a much faster speed than
human labelers”. Chiang and Lee (2023) similarly describe simulated
participants as “cheaper and faster than human evaluation”, and Dillion et al.
(2023) admire how “LLMs can rapidly answer hundreds of questions without
fatigue”. Bai et al. (2022) argue generative AI can “scale supervision” and
“reduce iteration time by obviating the need to collect new human feedback
labels when altering the [feedback] objective”. User Persona ([n. d.]) touts
the capability of synthesized participants “to unlock more time for high-value
work”, and Horton (2023) describes “the advantages in terms of speed” as
“enormous”. Pointing out that the majority of researchers who work with human
participants are “time-limited”, Synthetic Users ([n. d.]) offers pithy
advice: “simulate to accelerate”.

The proposals frequently conditioned these speed and scale arguments on claims
about the accuracy of LLMs. Immediately after identifying “high speed” as a
potential benefit of substitution, Hämäläinen et al. (2023) muse that “data
quality” might prove to be a concomitant drawback. Gerosa et al. (2023) state
that “if [software engineers] could simulate […] human behaviors with
sufficient accuracy, the potential for scaling research efforts could be
unprecedented”. Finally, Aher et al. (2023) propose that future “increase[s]
in accuracy” with LLMs can help alleviate “considerations regarding scale” in
behavioral research.

####  3.1.2. “Substitution lowers costs and reduces reliance on human labor”

Reducing cost was another prominent argument, referenced in eight of the
proposals. Gilardi et al. (2023) estimate that the cost of LLM surrogates is
on the order of “thirty times cheaper than MTurk”, a standard online platform
for recruiting human participants. Likewise, Hämäläinen et al. (2023) laud the
possibility of “low cost and high speed of LLM data generation” in the context
of user research. Four sources connect this point to a related motivation:
reducing the need for manual labor. As argued by Chiang and Lee (2023),
substitution alleviates development costs by reducing the need for humans in
data collection. Aher et al. (2023) similarly note the intertwined benefits of
limiting monetary costs and reducing reliance on effort by human participants.
Interestingly, these considerations appear to primarily emerge in proposals
for AI development, HCI, and commercial products.

Mirroring the discussion of speed and scale advantages, arguments about cost
efficiency were frequently paired with assertions concerning the accuracy of
AI substitution. In the context of reviewing the high “algorithmic fidelity”
of LLMs replacing human participants, Argyle et al. (2023) explain that LLMs
should require “fewer resources than a parallel data collection with human
respondents”. Similarly, Wang et al. (2021) argue that AI surrogates “can
significantly reduce labeling cost while achieving the same performance with
human-labeled data”. Hämäläinen et al. (2023) pose this as a “crucial
question” for user researchers: “when can low cost and latency offset issues
with quality?”

####  3.1.3. “Substitution augments demographic diversity in datasets”

Eight different proposals mention that certain communities or demographic
traits are of particular interest for simulation with generative AI. Aher et
al. (2023) explain that substitution offers “a simple way to simulate gender
and racial diversity”. Argyle et al. (2023) contend that LLMs could generate
“patterns of attitudes and ideas” held by “many groups (women, men, White
people, people of color, millennials, baby boomers, etc.)”, and—along with
Gerosa et al. (2023)—suggest that simulated participants can help explore
intersectionality. Synthetic Users ([n. d.]), OpinioAI ([n. d.]), and User
Persona ([n. d.]) similarly mention synthesizing user data reflecting
different genders, ethnicities, income levels, geographic locations,
languages, and political views. Byun et al. (2023) state simply that
generative AI can be used to simulate the participation of “vulnerable
populations”. Several proposals go as far as arguing that an open-ended range
of communities may be simulated, limited only by the researcher or developer’s
imagination in writing a textual prompt for the LLM (Synthetic Users, [n. d.];
OpinioAI, [n. d.]).

####  3.1.4. “Substitution protects participants from harms”

Six proposals argue that substitutions with generative AI can protect
participants from potential harm. Byun et al. (2023) summarize this idea by
suggesting that simulated participants may “prove invaluable in scenarios
where human involvement would be impractical, unethical, or unsafe”.
Similarly, Aher et al. (2023) discuss the use of substitution to explore
research questions that would be “unethical to run on human subjects”,
offering “experiments on what to say to a person who is suicidal” as an
example in which LLMs could shoulder the burdens from high-risk research
questions. Bai et al. (2022) argue that, in AI development work, the use of
LLMs reduces the need for human participants “to engage in the rather unsavory
work of trying to trick AI systems into generating harmful content”.

###  3.2. Contextualizing the motivations reported by proposals

To ensure a balanced evaluation of these proposals and seriously engage with
the idea of substitution, we next seek to place these motivations in context.
How much merit do these specific concerns carry, given the contemporary state
of scientific research and AI development?

Current funding dynamics in research and development offer validation for the
first two motivations frequently cited by substitution proposals—reducing
costs and increasing the efficiency of data collection. Typically, LLM users
pay based on the amount of text they feed into the model and the amount of
text it produces (e.g., (OpenAI, 2023)). Based on standard rates, LLMs can
produce data at fractions of the cost of recruiting and working with human
participants. Meanwhile, financial resources are a critical constraint on
social-behavioral science (Almeida, 2023). In HCI, for example, the need for
funds to recruit and work with human participants steers a large proportion of
the field into industry partnerships (Critical Platform Studies Group et al.,
2019). Financing plays a similarly central role in the AI field. Large
technology companies enjoy a great deal of influence over AI development,
given the generous resources at their disposal and the smaller pools available
to academic laboratories and other organizations (Birhane et al., 2022b;
Luitse and Denkena, 2021; Martínez-Plumed et al., 2021). Relative
affordability would thus make AI surrogates especially helpful for scientists
and developers working outside of corporate laboratories.

Public dialogues around AI and science offer validation for the other two
stated goals that we find manifest in substitution proposals: protecting
participants from harm and augmenting demographic diversity in datasets. In
traditional ethics frameworks, the principles of non-maleficence (often
summarized as “do no harm”) and justice (which seeks to ensure a fair and
equitable distribution of risks and benefits from research) are paramount
(American Psychological Association, 2016; National Commission for the
Protection of Human Subjects of Biomedical and Behavioral Research, 1978).
These principles emerged in response to historical incidents where scientists
abused their power over participants (Brandt, 1978; National Commission for
the Protection of Human Subjects of Biomedical and Behavioral Research, 1978).
The scientists, policymakers, and ethicists who developed these frameworks
intended the concept of justice in particular to address the vulnerability of
marginalized communities in the research process. With the advent of big data,
interpretations of justice now also focus on ensuring that marginalized groups
are (safely) reflected in research datasets and outcomes (Metcalf and
Crawford, 2016). Non-maleficence and justice are similarly two of the foremost
concerns of policymakers and ethicists over AI development (Jobin et al.,
2019). Thus, arguments to use substitution to protect participants and to
improve demographic diversity in data collection echo widely shared ethical
concerns over research and development.

##  4\. Evaluating challenges to the replacement of human participants

Of course, the role of a critical scholar is not to simply accept prevailing
narratives at face value, but rather to interrogate them. We next turn a
critical gaze to the manifest motivations offered by substitution proposals.
To rigorously analyze these arguments, we first consider the technical
properties and limitations of current language models. We then explore the
social structure and epistemic foundations of research and development to help
deconstruct the claims, promises, and potential pitfalls embedded within these
proposals.

###  4.1. Practical obstacles to the replacement of human participants

The substitution proposals themselves raise our first practical issue with
substitution: modern language models are not yet ready to simulate human
cognition and decision making. Despite their general fluency with language,
LLMs routinely make mistakes across all domains to which they are applied
(e.g., (Anthropic, 2023b; OpenAI, 2023)). Their strong tendency to
“hallucinate” false information is a particular problem, especially because
they do not indicate when they produce or cite false information (Ji et al.,
2023). In early 2023, for instance, a U.S. lawyer asked an LLM to act as a
legal assistant and subsequently submitted a court filing riddled with
fabricated legal references—partially because he believed the model’s
statements assuring him that the cases were real (Weiser, 2023). Indeed,
multiple proposals in our survey acknowledged the risk that hallucination
introduces when simulating participants with an LLM (Argyle et al., 2023; Byun
et al., 2023; Chiang and Lee, 2023; Dillion et al., 2023; Heyman and Heyman,
2023; Horton, 2023). This high risk of generating ungrounded responses
undermines the cost and efficiency arguments for substitution, given their
assumption of high accuracy levels.

A second technical obstacle is “value lock-in” (Weidinger et al., 2022). The
social and cultural conditions of a time period constrain and influence the
ways in which people express attitudes, beliefs, and behavior (Cooper and
Denner, 1998). Conceptually, LLMs develop their representations of human
behavior and decision making from the norms and attitudes present in human
writing (e.g., implicitly learning from the attitudes expressed on internet
websites and in books). However, an LLM is typically trained just once. When
the underlying attitudes and conventions shift, an LLM does not update itself
to encode new norms: the responses it generates will still reflect attitudes
from the time when it was trained. An LLM trained on data before 2018, for
instance, would not predict the shift in privacy concerns among social media
users following March of that year (González-Pizarro et al., 2022; Jones,
2018). Much like the constraints imposed by hallucination, value lock-in
challenges the accuracy of substitution, and therefore weakens the arguments
focused on efficiency and financial benefits.

Third, modern LLMs struggle to model the wide range of opinions held across
human communities, especially minority perspectives. Existing language models
most readily simulate opinions from liberal, highly educated, and high-income
earning individuals, and struggle to generate opinions reflective of
demographic groups including individuals who are older individuals, religious,
or widowed (Santurkar et al., 2023). As Crockett and Messeri (2023) point out,
the training data used for LLMs echoes and reinforces the focus of psychology
and HCI research on western, educated, industrialized, rich, and democratic
people (Henrich et al., 2010; Linxen et al., 2021). Indeed, current LLMs
exhibit substantial cultural bias (Atari et al., 2023; Durmus et al., 2023;
Naous et al., 2023) and routinely produce stereotyped images of minority
individuals (Abid et al., 2021; Ousidhoum et al., 2021). These myopias
severely limit the suggestion that substitution can augment research and
development work by simulating perspectives from marginalized communities.

Finally, user and psychology research rely on a variety of non-linguistic
indicators to study and understand human cognition and behavior. The
measurement of reaction time, facial expressions, and even pupil dilation have
been integral in the history of psychology research (Ekman, 1993; Hess and
Polt, 1964; Robinson, 2001). While it is easy to see how the text produced by
an LLM can simulate participants’ text responses or multiple-choice
selections, language is not a natural analog for many behavioral measures.
Though it may accelerate the speed of data collection and alleviate funding
issues, substitution also restricts the types of questions available for
exploration, and thus on the possible paths that research can take.

###  4.2. Intrinsic challenges to the replacement of human participants

These four practical obstacles impose substantial constraints on the sorts of
research questions and insights that LLM substitutes can support. Nonetheless,
they are not inherent flaws in generative modeling. Future iterations and
improvements to models could discover novel solutions to address
hallucinations, the preclusion of minority perspectives, and value lock-in.
Even the problem of non-linguistic measures may admit an eventual solution
through clever uses of language (e.g., (Yao et al., 2022)) or multimodal
generative AI (e.g., (Ho et al., 2022)). Technical progress may alleviate
these concerns, and perhaps overcome them entirely.333Indeed, modern AI
research has frequently defied expectations, sometimes on remarkable
timescales. For example, see the survey conducted by Grace et al. (2018),
collecting predictions of important milestones in AI development from AI
scientists. Multiple breakthroughs handily subverted their corresponding
expert-predicted timelines. In addition, while these obstacles undermine three
of the four motivations revealed in our review, they do not erode the argument
for protecting participants from risk. A researcher interested in
experimenting with AI surrogates might consider such protections worth the
costs.

However, practical issues are only part of the picture. We next critically
appraise the broader context of human participation. This exploration reveals
tensions that are considerably more fundamental in nature. In particular,
substitution proposals conflict with values intrinsic to AI development work
and research science: namely, representation, inclusion, and understanding.

####  4.2.1. The values of representation and inclusion

First, replacing human participants with LLMs inherently conflicts with
representation and inclusion, two principles core to the participatory
development of AI technology.

Discussing the values of participation, representation, and inclusion can be
difficult, given both the ambiguous definitions of the underlying concepts and
the way that researchers and developers often use the terms interchangeably
(Birhane et al., 2022a; Chasalow and Levy, 2021; Cornwall, 2008). More
importantly, invocations of all three concepts have been criticized for
connoting the involvement or presence of people without any substantive
changes over the status quo (Barton and Tan, 2020; Bergman et al., 2023;
Birhane et al., 2022a; Chasalow and Levy, 2021; Cornwall, 2008; Quick and
Feldman, 2011). To paraphrase Cornwall (2008), being involved in a process is
not equivalent to having any voice or any power over it. Participation in
particular is often defined shallowly, with “participant” reflecting an
incidental sort of role, resulting simply from being part of the environment
or process that leads to a technological product (Groves et al., 2023; Jing et
al., 2023). Whether intentional or not, shallow uses of the “participatory”
label produce a veneer of representation and inclusion, despite the lack of
deeper engagement with participating individuals and communities.

Here we focus on “participation” not in these shallow forms, but as the
concept was originally conceived in public planning and development. In a
foundational treatise, Arnstein (1969) articulated a vision of participation
that is “responsive to” and reflective of the “views, aspirations, and needs”
of those participating. For Arnstein, authentic participation must involve
“the redistribution of power that enables the [have-nots], presently excluded
from the political and economic processes, to be deliberately included in the
future”. This description reveals two intertwined goals for participatory
processes: first, to reflect and respond to the interests of those
participating; and second, to redistribute power to those participating. We
refer to these attributes as representation and inclusion, respectively, based
both on Arnstein’s arguments444Arnstein’s treatise is foundational in
participatory design, but did not introduce these ideas for the first time.
The intellectual lineage of the ideas of responsiveness and power traces back
several decades before Arnstein’s piece, emerging across multiple schools of
thought in the history of planning (Lane, 2005). and on established frameworks
from political science and organizational theory. The goal of representation
involves an active process of understanding and “making present” the needs,
interests, and perspectives of those being represented.555In particular,
Pitkin (1967) describes representation as ”acting in the interest of the
represented, in a manner responsive to them”. Arnesen and Peters (2018) offer
an eloquent elaboration of this idea, defining “substantive representation” as
the situation where the “‘making present’ of [a] group’s ideas and the ‘acting
for’ that group is done by people who understand what it means to be part of
this group”. In contrast, the goal of inclusion emphasizes participants’
agency and exercise of power.666Concerning inclusion, Acemoğlu and Robinson
(2016) argue that inclusive institutions distribute power broadly, empowering
a wide range of people to participate in their activities. Similarly, Dahl
(2008) recognizes inclusiveness as a fundamental dimension of democracy,
defining it as the participation of all members of society in the exercise of
power. In organization theory, Quick and Feldman (2011) muse that “inclusion
practices” must remake rather than recreate existing power structures in order
to truly invite communities to “co-produc[e] processes, policies, and
programs”. Synthesizing these aims, participatory work should draw in
participants to collaboratively set the agenda, empower them with influence
over the development process, and guarantee them avenues of recourse,
reparation, and resistance.

Corroborating this conceptual framework, prominent AI institutions frequently
advocate for participatory approaches to development that explicitly tie
participation to the goals of representation and inclusion. Zaremba et al.
(2023), for instance, argue that future AI systems should “be shaped to be as
inclusive as possible” through democratic and representative processes. Other
organizations echo similar plans, including one advocating for “democratic
participation” to develop principles to govern the behavior of future AI
systems (Roose and Newton, 2023). Glaese et al. (2022) likewise appeal for
“participatory input from a diverse array of users and affected groups” for
the development of more advanced systems. These appeals signify that the
participation of individuals and communities is not just about improving data
collection, but also about representation and empowerment. Turning these ideas
back to substitution proposals: simulating human participants with LLMs may
support more efficient data collection, but does it embody these participatory
goals?

Unfortunately, substitution proposals only support representation in a very
weak sense. At some point in their training, the models process linguistic
data from humans, and development teams subsequently use these models to stand
in for the perspectives and opinions that helped produce those data. But how
well do the modeling outputs reflect people’s perspectives, and how responsive
is the substitution process to changes in those views? The current class of
substitution proposals do not specify any possible approaches to re-align a
wayward LLM, and previous attempts to manually steer models observed limited
improvements to representativeness (Santurkar et al., 2023). Without
mechanisms to address these questions, substitution is not truly
representative—it does not “make present” people’s experiences.

As for inclusion: if the sharing of power is core to inclusiveness, then the
current class of substitution proposals are inherently exclusionary.
Participation in the development process, even through data enrichment,
provides people with critical capabilities and affordances. These include the
right to withhold data, the right to opt out, the right to express discontent,
and the right to resist, to name a few (Agnew et al., 2023). Substitution
proposals shift several of these powers to AI surrogates, thereby denying
participants agency over mechanisms of feedback and change—and implicitly
ceding control to the developers. Some rights are lost altogether. When faced
with an unethical situation, for instance, LLMs will not report the issue via
regulatory authorities, media organizations, or other institutions that could
bring the situation to light.

These exclusionary dynamics are especially important given that several
proposals recommend simulating racial minorities, women, and other
marginalized communities to help address issues with data diversity (Argyle et
al., 2023; Byun et al., 2023). These communities have long, painful histories
of exclusion and exploitation in scientific and technological development
(National Commission for the Protection of Human Subjects of Biomedical and
Behavioral Research, 1978; Scharff et al., 2010). To ignore this historical
context is to risk conflating technological change with social progress,
devoid of any “moral and political standards” (Marx, 2010). Indeed, scholars
in the humanities argue that visions of replacing human labor with AI
perpetuate troubling social and political legacies. The types of labor
considered “appropriate” for automation are typically those already devalued
and exploited under capitalism (Rhee, 2018). As a consequence, the idea that
new technology will render human labor obsolete echoes colonial fantasies of
an ideal servant class, working invisibly without need or desire (Atanasoski
and Vora, 2019). Ironically, the very conceit of simulating marginalized
communities as a solution to exclusion reinforces the devaluation of their
labor and lived experiences.

Ultimately, replacing participants denies them opportunities to directly
combat this devaluation, to influence the goals of the development process,
and to help adjudicate the balance between their rights and the risks that
they experience. It may be the case that the individuals and communities
involved in development work wish to relinquish their participatory role.
Rather than entirely turning to technological solutions, a truly
representative and inclusive process would explicitly ask that question of
them.

####  4.2.2. The value of understanding

The second intrinsic challenge to substitution proposals concerns the
foundations of user research and psychological science. Replacing human
participants with AI disrupts the intersubjectivity between researcher and
participant, undermining the goal of understanding.

A core aim of research science is the production of generalizable insights and
knowledge (cf. (Mook, 1983)). Indeed, in the U.S., government regulation
highlights the goal of “develop[ing] or contribut[ing] to generalizable
knowledge” as a definitional characteristic of research (U.S. Department of
Health and Human Services, 2018). Scientific culture often idealizes research
as an objective process that leverages precise measurement and logical
interpretation to uncover general insights about the natural world (Reiss and
Sprenger, 2014). From this perspective, psychological fields observe and
measure external indicators like behavior to better understand internal
processes like cognition and decision making.

In reality, the basis of psychological research and insight is not objective
measurement, but intersubjective corroboration (Mascolo and Kallio, 2020).
Consider a user experience (UX) researcher in their laboratory, conducting a
study on user preferences over several possible website designs. Each user in
the laboratory’s study room sits in front of a computer and sees two designs
in sequence. They then respond to a question asking for their relative
preference between the two on a scale from one to seven. In an objectivist
account, the user’s preferences are a construct that exists in the real world,
but which the researcher cannot directly access. By measuring an indicator
that is externally observable and quantifiable, the researcher can estimate
the user’s preferences (and then test various hypotheses over that construct).

But how does the researcher know the meaning of the external indicator in the
first place? If internal states and constructs are truly private, why does the
researcher believe that acting in a certain way (e.g., interacting with the
mouse and selecting one of the numbers on the scale) maps to some specific
internal experience (e.g., a preference for one design over the other)? The
answer is that the researcher does not deduce from external measurements
alone. Rather, they combine those “objective” indicators with their own
internal experience to draw an inference about the participant. The idea of
intersubjectivity can seem abstruse on first encounter, but it can be
summarized simply as the researcher’s assumption that if they were to take
another person’s position, they would see or experience the situation as that
person does. In short, intersubjectivity is “the possibility of trading
places” (Duranti, 2010).

Sometimes, it is more or less credible for two individuals to “trade places”.
Variations in this possibility—that is, in the degree of
intersubjectivity—constrain and shape a scientist’s ability to conduct
research.777Indeed, it is for this reason that philosophers of science
sometimes distinguish the social sciences as different in kind from the
natural and physical sciences (Mayntz, 2005). Unlike psychologists and other
social scientists, natural and physical scientists have no intersubjective
relationship with the objects of their study. A physicist cannot trade places
with a particle, and a geologist cannot trade places with a rock. These
scientists cannot (and need not) corroborate their knowledge claims with
intersubjectivity. While all scientists seek to develop knowledge of the
world, the study of humans demands a special kind of knowledge: Verstehen, or
a “participatory” type of understanding (Weber et al., 1947). For instance, an
HCI researcher traveling and encountering a foreign culture for the first time
would find it very difficult to draw sound inferences about user behavior from
its inhabitants without first spending more time studying local norms and ways
of living (e.g., (Bourges-Waldegg and Scrivener, 1998)). Situations with
limited intersubjectivity require researchers to continuously question and
challenge their own assumptions.

To return to the topic of AI substitution, let us revisit our hypothetical UX
laboratory. Our intrepid researcher remains in the laboratory late one evening
to analyze their data. As they diligently work, an alien appears in the study
room in front of the computer. It moves its arms—or what might be its arms—and
in doing so, nudges the mouse, pressing down on its buttons several times.
These actions cause the computer to flip through both websites before
registering a two on the researcher’s seven-point preference scale. Can our
researcher thank the alien, not just for the fame of first contact, but also
for the free data provided to their study? Should they add a tally to the
count of participants who prefer the first design? Perhaps the alien does
prefer the first design, much like the human participant who visited earlier.
However, the crucial intersubjective assumption is missing. The researcher has
no actual assurance that internal preferences drove the alien’s action, or
indeed that the alien possesses something resembling human preferences. Until
our researcher gains a much deeper understanding of this alien, attempts to
transfer any observations to human cognition are made on shaky grounds.

Language models, of course, are not alien—they are trained on human
data.888Though as demonstrated by the recently popularity of the “shoggoth”
meme (@hlntnr, 2023; @jacyanthis, 2023; @TetraspaceWest, 2022), many members
of the AI community are skeptical of the human-likeness of LLMs. Perhaps they
share some form of an intersubjective relationship with research scientists.
As in other situations with limited intersubjectivity, research with AI
surrogates might then support careful insight into human cognition and
decision making. For instance, when prompted to choose from a list of possible
consumer options, LLMs demonstrate a fairly reliable preference for the first
option (e.g., (Kirshner, 2024)). A researcher might infer from this model
behavior—and from the general way that LLMs are trained on data from
humans—that the average consumer is more likely to be swayed by the initial
option presented in a user interface (cf. (Mantonakis et al., 2009)).
Ultimately, though, the credibility of trading places with language models and
experiencing a situation as they do seems quite limited (Bender and Koller,
2020; Shanahan, 2022). And importantly, any intersubjective relationship that
a human researcher shares with a language model is different in kind from the
relationship between a human researcher and a human participant. Behavioral
scientists generally maintain a good sense of the limits of intersubjectivity
with their models. For example, researchers are keenly aware of the limited
external validity of rule-based models (e.g., (Jackson et al., 2017)).
However, LLMs present a distinctly powerful risk of anthropomorphism, even for
researchers (Weidinger et al., 2022). Their language proficiency can challenge
and distract from the precept that “a map is not the territory”. Overall, the
limited intersubjectivity that researchers share with generative AI systems
constrains the possibility of substituting AI for human participants in
research. Improvements in efficiency and acceleration do not actually benefit
research if substitution ultimately produces unsound conclusions.

####  4.2.3. Intrinsic challenges across research and development

Thus far, we have primarily discussed the goals of representation and
inclusion in the context of AI development, and the aim of understanding in
the context of research science. However, these issues cross over these
boundaries, emerging across both domains.

Representation and inclusion are also core to user research, psychology, and
other social and behavioral fields. Participatory HCI (Vines et al., 2013),
computer-supported cooperative work (Ciolfi et al., 2023), participatory
action research (Fals-Borda and Rahman, 1991), and critical psychology (Walsh,
2014) argue that scientists should avoid viewing participants primarily as
sources of data, whose only role is to accept certain inputs and subsequently
produce outputs of a desired form. Rather, they argue, researchers have
ethical obligations to partner and share power with their participants. Here
too, substitution proposals undermine the processes meant to make present
participants’ interests and erode participants’ ability to influence the
overall undertaking.

Similarly, intersubjectivity plays an essential role in modern AI
development—particularly in its reliance on data annotation and enrichment. As
Lambert and Ustalov (2023) explain, “The core challenge in data labeling is to
make the annotators understand the task the same way you do”. At the design
stage of an annotation study, in order to construct appropriate instructions,
interfaces, and questions, developers must have an appreciation for their
annotators’ perspectives and experiences. Similarly, at the interpretation
stage, developers must understand their annotators’ perspective to make sense
of their results and identify any issues that could interfere with their
ability to interpret their annotations. Data annotation and enrichment are
predicated on the possibility of “trading places”.

####  4.2.4. Summary

Overall, the values of representation, inclusion, and understanding pose
considerable challenges to substitution proposals. These challenges are
different in kind from the initial obstacles discussed in Section 4.1.
Conflicts with representation, inclusion, and understanding cannot be
alleviated with better training or improved model performance alone: they are
intrinsic to the models and to the proposed approach to replacement itself.

##  5\. Conclusion and paths forward

Large language models (LLMs) and other generative artificial intelligence (AI)
systems will likely change the way that people approach many professional
tasks. Research science and technology development are no exception. Recently,
a set of curious proposals has emerged, recommending the substitution of LLMs
for human participants in AI development and in research contexts ranging from
human-computer interaction to opinion polling. These proposals carry several
potential merits: increasing the speed of research and development; reducing
financial costs; mitigating participants’ exposure to potential harms; and
augmenting demographic diversity. However, three of these merits are
constrained by technical shortcomings of current AI models. More pressingly,
replacement faces two intrinsic challenges from the broader social and
epistemological structures that support human participation. First, while
participatory development aims to “make present” participants’ perspectives
and share power with them, substitution proposals have the opposite
effect—impeding representation and inclusion. Second, substitution proposals
remove the intersubjective basis of psychological and user research, limiting
the inferences that psychologists and researchers can draw from studies and
experiments with LLMs. These undercut all four stated motivations for
substitution, and are thus considerable challenges to any proposal for the
replacement of human participants in AI development and psychology research.

A deeper reckoning with the values of representation, inclusion, and
understanding is needed to identify changes that could mitigate these
intrinsic challenges. A first step toward representative and inclusive
versions of substitution will be to involve and empower participants in the
agenda-setting process. After formally setting the agenda, projects can
maintain an ongoing role for participants, allowing them to monitor and
supervise whether their substitutes are making present their perspectives and
concerns. This practice of continuous engagement should strive not only to
align the development process with those participating, but also to allow
participants to exercise power over the development process: provide feedback,
express discontent, and seek recourse. To make substitution compatible with
the goal of understanding, user researchers and psychologists will need to
confront a potentially uncomfortable idea: that scientists are not immune to
the temptation to anthropomorphize language models. One potential approach to
mitigating the risk of anthropomorphism would be to establish a clear
framework for evaluating the results of substitution experiments and assessing
whether they generate any valid insights. Such a framework could help
researchers remain mindful of the limits on “trading places” with language
models. Overall, by establishing these structures and practices, we may start
to build generative AI systems that support the values fundamental to research
and development, moving toward authentic—rather than artificial—inclusion.

###### Acknowledgements.

We thank William Isaac, Piotr Mirowski, our anonymous reviewers, and our
anonymous area chair for insightful feedback on our manuscript.

## Author Contributions

W.A., J.P., and K.R.M. proposed the research idea; K.R.M. supervised the
project; W.A. and K.R.M. curated and annotated sources, with assistance from
A.S.B., J.C., M.D., S.E., J.P., and S.M.; W.A., A.S.B., J.C., M.D., S.E.,
J.P., S.M., and K.R.M. interpreted results; and W.A., A.S.B., J.C., M.D.,
S.E., J.P., S.M., and K.R.M. wrote the paper.

## References

  * (1)
  * Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou. 2021\.  Persistent anti-Muslim bias in large language models. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 298–306.
  * Acemoğlu and Robinson (2016) Daron Acemoğlu and James A. Robinson. 2016.  Paths to inclusive political institutions.  _Economic History of Warfare and State Formation_ (2016), 3–50.
  * Agnew et al. (2023) William Agnew, Kevin R. McKee, Iason Gabriel, Jackie Kay, William Isaac, A. Stevie Bergman, Seliem El-Sayed, and Shakir Mohamed. 2023\.  Technologies of resistance to AI. In _Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–13.
  * Aher et al. (2023) Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023\.  Using large language models to simulate multiple humans and replicate human subject studies. In _Proceedings of the 40th International Conference on Machine Learning_. Proceedings of Machine Learning Research, 337–371.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: A visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ 35, 23716–23736.
  * Almeida (2023) Jorge Almeida. 2023\.  Underfunding basic psychological science because of the primacy of the here and now: A scientific conundrum.  _Perspectives on Psychological Science_ 18, 2 (2023), 527–530.
  * American Psychological Association (2016) American Psychological Association. 2016\.  Ethical principles of psychologists and code of conduct.  https://web.archive.org/web/20230909061239/https://www.apa.org/ethics/code/.  Accessed: 2023-09-14.
  * Anthropic (2023a) Anthropic. 2023a.  Introducing 100K Context Windows.  https://web.archive.org/web/20230903233613/https://www.anthropic.com/index/100k-context-windows.  Accessed: 2023-09-14.
  * Anthropic (2023b) Anthropic. 2023b.  Model Card and Evaluations for Claude Models.  https://web.archive.org/web/20230811185640/https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf.  Accessed: 2023-09-14.
  * Argyle et al. (2023) Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ 31, 3 (2023), 337–351.
  * Arnesen and Peters (2018) Sveinung Arnesen and Yvette Peters. 2018.  The legitimacy of representation: How descriptive, formal, and responsiveness representation affect the acceptability of political decisions.  _Comparative Political Studies_ 51, 7 (2018), 868–899.
  * Arnstein (1969) Sherry R. Arnstein. 1969\.  A ladder of citizen participation.  _Journal of the American Institute of Planners_ 35, 4 (1969), 216–224.
  * Atanasoski and Vora (2019) Neda Atanasoski and Kalindi Vora. 2019.  _Surrogate humanity: Race, robots, and the politics of technological futures_.  Duke University Press.
  * Atari et al. (2023) Mohammad Atari, Mona J. Xue, Peter S. Park, Damián Blasi, and Joseph Henrich. 2023.  Which humans?  _PsyArXiv_ (2023).
  * Atkinson and Flint (2001) Rowland Atkinson and John Flint. 2001.  Accessing hidden and hard-to-reach populations: Snowball research strategies.  _Social Research Update_ 33, 1 (2001), 1–4.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022\.  Constitutional AI: Harmlessness from AI feedback.  _arXiv preprint arXiv:2212.08073_ (2022).
  * Barton and Tan (2020) Angela Calabrese Barton and Edna Tan. 2020.  Beyond equity as inclusion: A framework of “rightful presence” for guiding justice-oriented studies in teaching and learning.  _Educational Researcher_ 49, 6 (2020), 433–440.
  * Belkin et al. (1987) Nicholas J. Belkin, Helen M. Brooks, and Penny J. Daniels. 1987\.  Knowledge elicitation using discourse analysis.  _International Journal of Man-Machine Studies_ 27, 2 (1987), 127–144.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 610–623.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 5185–5198.
  * Berelson (1952) Bernard Berelson. 1952\.  _Content analysis in communication research_.  Free Press.
  * Bergman and Diab (2022) A. Stevie Bergman and Mona Diab. 2022.  Towards responsible natural language annotation for the arieties of Arabic. In _Findings of the Association for Computational Linguistics: ACL 2022_. 364–371.
  * Bergman et al. (2023) A. Stevie Bergman, Lisa Anne Hendricks, Maribeth Rauh, Boxi Wu, William Agnew, Markus Kunesch, Isabella Duan, Iason Gabriel, and William Isaac. 2023\.  Representation in AI evaluations. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 519–533.
  * Bing (1986) Jon Bing. 1986.  The text retrieval system as a conversion partner.  _International Review of Law, Computers & Technology_ 2, 1 (1986), 25–39.
  * Birhane et al. (2022a) Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Díaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022a.  Power to the people? Opportunities and challenges for participatory AI. In _Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–8.
  * Birhane et al. (2022b) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022b.  The values encoded in machine learning research. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 173–184.
  * Bourges-Waldegg and Scrivener (1998) Paula Bourges-Waldegg and Stephen A. R. Scrivener. 1998.  Meaning, the central issue in cross-cultural HCI design.  _Interacting with Computers_ 9, 3 (1998), 287–309.
  * Brandt (1978) Allan M. Brandt. 1978\.  Racism and research: The case of the Tuskegee Syphilis Study.  _Hastings Center Report_ (1978), 21–29.
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative Research in Psychology_ 3, 2 (2006), 77–101.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ 33, 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with GPT-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Buschek et al. (2020) Daniel Buschek, Charlotte Anlauff, and Florian Lachner. 2020\.  Paper2Wire: A case study of user-centred development of machine learning tools for UX designers.  In _Proceedings of Mensch und Computer 2020_. 33–41.
  * Byun et al. (2023) Courtni Byun, Piper Vasicek, and Kevin Seppi. 2023\.  Dispensing with humans in human-computer interaction research. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–26.
  * Chasalow and Levy (2021) Kyla Chasalow and Karen Levy. 2021.  Representativeness in statistics, politics, and machine learning. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 77–89.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations?. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 15607–15631.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  PaLM: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Ciolfi et al. (2023) Luigina Ciolfi, Myriam Lewkowicz, and Kjeld Schmidt. 2023\.  Computer-supported cooperative work.  In _Handbook of Human-Computer Interaction_. Springer, 1–26.
  * Coan (1979) Richard W. Coan. 1979\.  _Psychologists: Personal and theoretical pathways_.  Irvington Publishers.
  * Cooper and Denner (1998) Catherine R. Cooper and Jill Denner. 1998.  Theories linking culture and psychology: Universal and community-specific processes.  _Annual Review of Psychology_ 49, 1 (1998), 559–584.
  * Cornwall (2008) Andrea Cornwall. 2008\.  Unpacking “participation”: Models, meanings and practices.  _Community Development Journal_ 43, 3 (2008), 269–283.
  * Critical Platform Studies Group et al. (2019) Critical Platform Studies Group, Lilly Irani, Niloufar Salehi, Joyojeet Pal, Andrés Monroy-Hernández, Elizabeth Churchill, and Sneha Narayan. 2019.  Patron or poison? Industry funding of HCI research. In _Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing_. 111–115.
  * Crockett and Messeri (2023) Molly Crockett and Lisa Messeri. 2023.  Should large language models replace human participants?  _PsyArXiv_ (2023).
  * Dahl (2008) Robert A. Dahl. 2008\.  _Polyarchy: Participation and opposition_.  Yale University Press.
  * Danziger (1994) Kurt Danziger. 1994\.  _Constructing the subject: Historical origins of psychological research_.  Cambridge University Press.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.
  * Dillion et al. (2023) Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023\.  Can AI language models replace human participants?  _Trends in Cognitive Sciences_ (2023).
  * Duranti (2010) Alessandro Duranti. 2010\.  Husserl, intersubjectivity and anthropology.  _Anthropological Theory_ 10, 1-2 (2010), 16–35.
  * Durmus et al. (2023) Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023\.  Towards measuring the representation of subjective global opinions in language models.  _arXiv preprint arXiv:2306.16388_ (2023).
  * Ekman (1993) Paul Ekman. 1993\.  Facial expression and emotion.  _American Psychologist_ 48, 4 (1993), 384.
  * Fals-Borda and Rahman (1991) Orlando Fals-Borda and Muhammad Anisur Rahman. 1991.  _Action and knowledge: Breaking the monopoly with participatory action-research_.  Intermediate Technology Publications.
  * Fazelpour and De-Arteaga (2022) Sina Fazelpour and Maria De-Arteaga. 2022.  Diversity in sociotechnical machine learning systems.  _Big Data & Society_ 9, 1 (2022), 20539517221082027\.
  * Gerosa et al. (2023) Marco A. Gerosa, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. 2023.  Can AI serve as a substitute for human subjects in software engineering research?  _arXiv preprint arXiv:2311.11081_ (2023).
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023\.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ 120, 30 (2023), e2305016120.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022\.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_ (2022).
  * González-Pizarro et al. (2022) Felipe González-Pizarro, Andrea Figueroa, Claudia López, and Cecilia Aragon. 2022.  Regional differences in information privacy concerns After the Facebook-Cambridge Analytica data scandal.  _Computer Supported Cooperative Work_ 31, 1 (2022), 33–77.
  * Grace et al. (2018) Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. 2018.  When will AI exceed human performance? Evidence from AI experts.  _Journal of Artificial Intelligence Research_ 62 (2018), 729–754.
  * Groves et al. (2023) Lara Groves, Aidan Peppin, Andrew Strait, and Jenny Brennan. 2023.  Going public: The role of public participation approaches in commercial AI labs. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 1162–1173.
  * Grudin (2009) Jonathan Grudin. 2009\.  AI and HCI: Two fields divided by a common focus.  _AI Magazine_ 30, 4 (2009), 48–48.
  * Grudin (2022) Jonathan Grudin. 2022\.  _From tool to partner: The evolution of human-computer interaction_.  Springer Nature.
  * Hämäläinen et al. (2023) Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023\.  Evaluating large language models in generating synthetic HCI research data: a case study. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Heckerman et al. (2000) David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, and Carl Kadie. 2000.  Dependency networks for inference, collaborative filtering, and data visualization.  _Journal of Machine Learning Research_ 1, Oct (2000), 49–75.
  * Henrich et al. (2010) Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010\.  Most people are not WEIRD.  _Nature_ 466, 7302 (2010), 29–29.
  * Hess and Polt (1964) Eckhard H. Hess and James M. Polt. 1964.  Pupil size in relation to mental activity during simple problem-solving.  _Science_ 143, 3611 (1964), 1190–1192.
  * Heyman and Heyman (2023) Tom Heyman and Geert Heyman. 2023.  The impact of ChatGPT on human data collection: A case study involving typicality norming data.  _Behavior Research Methods_ (2023), 1–8.
  * @hlntnr (2023) @hlntnr. 2023\.  If you spend much time on AI twitter, you might have seen this tentacle monster hanging around. But what is it, and what does it have to do with ChatGPT? It’s kind of a long story. But it’s worth it! It even ends with cake. THREAD.  https://web.archive.org/web/20230306124925/https://twitter.com/hlntnr/status/1632030583462285312.
  * Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, et al. 2022\.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ (2022).
  * Horton (2023) John J. Horton. 2023\.  _Large language models as simulated economic agents: What can we learn from homo silicus?_ Technical Report. National Bureau of Economic Research.
  * Hutchinson et al. (2021) Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021\.  Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 560–575.
  * @infoxiao (2023) @infoxiao. 2023\.  A trend I see at #CHI23 is HCI researchers beginning to use LLMs to create synthetic user studies. E.g., https://dl.acm.org/doi/10.1145/3544548.3580688. While it is convenient, I do worry about the potential representational harms posed by such trend – experiences of minorities being erased.  https://web.archive.org/web/20240113081958/https://nitter.net/infoxiao/status/1651160544307544064.
  * Jackson et al. (2017) Joshua Conrad Jackson, David Rand, Kevin Lewis, Michael I. Norton, and Kurt Gray. 2017\.  Agent-based modeling: A guide for social psychologists.  _Social Psychological and Personality Science_ 8, 4 (2017), 387–395.
  * @jacyanthis (2023) @jacyanthis. 2023\.  Here’s the latest iteration of the “techniques like RLHF are just putting superficial smiley faces on an opaque, alien, Shoggoth intelligence that we have no real control over” AI safety meme. Artist: Anna Husfeldt, CC-BY SA 3.0.  https://web.archive.org/web/20230306110137/https://twitter.com/jacyanthis/status/1631291175381475331.
  * Jamieson et al. (2023) Michelle K. Jamieson, Gisela H. Govaart, and Madeleine Pownall. 2023.  Reflexivity in quantitative research: A rationale and beginner’s guide.  _Social and Personality Psychology Compass_ (2023), e12735.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _Comput. Surveys_ 55, 12 (2023), 1–38.
  * Jing et al. (2023) Felicia S. Jing, Sara E. Berger, and Juana Catalina Becerra Sandoval. 2023\.  Towards labor transparency in situated computational systems impact research. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 1026–1037.
  * Jobin et al. (2019) Anna Jobin, Marcello Ienca, and Effy Vayena. 2019\.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (2019), 389–399.
  * Jones (2018) J. Jones. 2018.  Facebook users’ privacy concerns up since 2011.  The Gallup Organization.
  * Kirshner (2024) Samuel N. Kirshner. 2024\.  GPT and CLT: The impact of ChatGPT’s level of abstraction on consumer recommendations.  _Journal of Retailing and Consumer Services_ 76 (2024), 103580.
  * Lambert and Ustalov (2023) Nathan Lambert and Dmitry Ustalov. 2023.  Reinforcement learning from human feedback: A tutorial at ICML 2023.  https://doi.org/10.5281/zenodo.8186168.
  * Lane (2005) Marcus B. Lane. 2005\.  Public participation in planning: An intellectual history.  _Australian Geographer_ 36, 3 (2005), 283–299.
  * Levy (1992) Steven Levy. 1992\.  _Artificial life: The quest for a new creation_.  Random House Inc.
  * Linxen et al. (2021) Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021.  How weird is CHI?. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Luitse and Denkena (2021) Dieuwertje Luitse and Wiebke Denkena. 2021.  The great transformer: Examining the role of large language models in the political economy of AI.  _Big Data & Society_ 8, 2 (2021), 20539517211047734\.
  * Manning (2022) Christopher D. Manning. 2022\.  Human language understanding & reasoning.  _Daedalus_ 151, 2 (2022), 127–138.
  * Mantonakis et al. (2009) Antonia Mantonakis, Pauline Rodero, Isabelle Lesschaeve, and Reid Hastie. 2009.  Order in choice: Effects of serial position on preferences.  _Psychological Science_ 20, 11 (2009), 1309–1312.
  * Martínez-Plumed et al. (2021) Fernando Martínez-Plumed, Pablo Barredo, Sean O. Heigeartaigh, and Jose Hernandez-Orallo. 2021\.  Research community dynamics behind popular AI benchmarks.  _Nature Machine Intelligence_ 3, 7 (2021), 581–589.
  * Marx (2010) Leo Marx. 2010.  Technology: The emergence of a hazardous concept.  _Technology and Culture_ 51, 3 (2010), 561–577.
  * Mascolo and Kallio (2020) Michael F. Mascolo and Eeva Kallio. 2020.  The phenomenology of between: An intersubjective epistemology for psychological science.  _Journal of Constructivist Psychology_ 33, 1 (2020), 1–28.
  * Mayntz (2005) Renate Mayntz. 2005\.  Forschungsmethoden und erkenntnispotential: Natur-und sozialwissenschaften im vergleich.  Max-Planck-Institut für Gesellschaftsforschung.
  * McKee (2023) Kevin R. McKee. 2023\.  Human participants in AI research: Ethics and transparency in practice.  _arXiv preprint arXiv:2311.01254_ (2023).
  * McKee et al. (2023) Kevin R. McKee, Andrea Tacchetti, Michiel A. Bakker, Jan Balaguer, Lucy Campbell-Gillingham, Richard Everett, and Matthew Botvinick. 2023.  Scaffolding cooperation in human groups with deep reinforcement learning.  _Nature Human Behaviour_ 7, 10 (2023), 1787–1796.
  * Metcalf and Crawford (2016) Jacob Metcalf and Kate Crawford. 2016.  Where are human subjects in big data research? The emerging ethics divide.  _Big Data & Society_ 3, 1 (2016), 2053951716650211\.
  * Mook (1983) Douglas G. Mook. 1983\.  In defense of external invalidity.  _American Psychologist_ 38, 4 (1983), 379.
  * Naous et al. (2023) Tarek Naous, Michael J. Ryan, and Wei Xu. 2023.  Having beer after prayer? Measuring cultural bias in large language models.  _arXiv preprint arXiv:2305.14456_ (2023).
  * National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1978) National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. 1978\.  _The Belmont report: Ethical principles and guidelines for the protection of human subjects of research_.  United States Government Printing Office.
  * National Institute of Standards and Technology (2023) National Institute of Standards and Technology. 2023.  Artificial Intelligence Risk Management Framework.  https://web.archive.org/web/20230914024011/https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf.  Accessed: 2023-09-14.
  * Newell (1982) Allen Newell. 1982\.  Intellectual issues in the history of artificial intelligence.  _Artificial Intelligence: Critical Concepts_ (1982), 25–70.
  * Nicolas et al. (2021) Gandalf Nicolas, Xuechunzi Bai, and Susan T. Fiske. 2021\.  Comprehensive stereotype content dictionaries using a semi-automated method.  _European Journal of Social Psychology_ 51, 1 (2021), 178–196.
  * Novak (2023) Matt Novak. 2023\.  That viral image of Pope Francis wearing a white puffer coat is totally fake.  _Forbes_ (2023).  https://web.archive.org/web/20230914181410/https://www.forbes.com/sites/mattnovak/2023/03/26/that-viral-image-of-pope-francis-wearing-a-white-puffer-coat-is-totally-fake/
  * OpenAI (2023) OpenAI. 2023.  GPT-4 System Card.  https://web.archive.org/web/20230914132031/https://cdn.openai.com/papers/gpt-4-system-card.pdf.  Accessed: 2023-09-14.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * OpenAI (2023) OpenAI. 2023.  Pricing.  https://web.archive.org/web/20230914164619/https://openai.com/pricing.
  * OpinioAI ([n. d.]) OpinioAI. [n. d.].  OpinioAI.  https://web.archive.org/web/20230914170909/https://opinio.ai/.  Accessed: 2023-09-14.
  * Ousidhoum et al. (2021) Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021.  Probing toxic content in large pre-trained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4262–4274.
  * Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002\.  Thumbs up? Sentiment classification using machine mearning techniques. In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)_. 79–86.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  _arXiv preprint arXiv:2304.03442_ (2023).
  * Park et al. (2022) Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022.  Social simulacra: Creating populated prototypes for social computing systems. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_. 1–18.
  * Partnership on AI ([n. d.]) Partnership on AI. [n. d.].  Responsible sourcing of data enrichment services.  https://web.archive.org/web/20230722043958/https://partnershiponai.org/responsible-sourcing-considerations/.  Accessed: 2023-09-14.
  * Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021\.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  _Patterns_ 2, 11 (2021).
  * Pitkin (1967) Hanna Pitkin. 1967\.  _The concept of representation_.  University of California Press.
  * Potter and Levine-Donnerstein (1999) W. James Potter and Deborah Levine-Donnerstein. 1999\.  Rethinking validity and reliability in content analysis.  _Journal of Applied Communication Research_ (1999).
  * Quick and Feldman (2011) Kathryn S. Quick and Martha S. Feldman. 2011.  Distinguishing participation and inclusion.  _Journal of Planning Education and Research_ 31, 3 (2011), 272–290.
  * Reiss and Sprenger (2014) Julian Reiss and Jan Sprenger. 2014.  Scientific objectivity.  In _The Stanford Encyclopedia of Philosophy_. Zalta, Ed.
  * Rhee (2018) Jennifer Rhee. 2018\.  _The robotic imaginary: The human and the price of dehumanized labor_.  University of Minnesota Press.
  * Robinson (2001) David K. Robinson. 2001\.  Reaction-time experiments in Wundt’s institute and beyond.  In _Wilhelm Wundt in History: The Making of a Scientific Psychology_. Springer, 161–204.
  * Rolf et al. (2021) Esther Rolf, Theodora T. Worledge, Benjamin Recht, and Michael Jordan. 2021.  Representation matters: Assessing the importance of subgroup allocations in training data. In _Proceedings of the 38th International Conference on Machine Learning_. Proceedings of Machine Learning Research, 9040–9051.
  * Roose (2023) Kevin Roose. 2023\.  A conversation with Bing’s chatbot left me deeply unsettled.  _The New York Times_ (2023).  https://web.archive.org/web/20230914044532/https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html
  * Roose and Newton (2023) Kevin Roose and Casey Newton. 2023.  Dario Amodei, C.E.O. of Anthropic, on the paradoxes of A.I. safety and Netflix’s “Deep Fake Love”.  https://web.archive.org/web/20230908235314/https://www.nytimes.com/2023/07/21/podcasts/dario-amodei-ceo-of-anthropic-on-the-paradoxes-of-ai-safety-and-netflixs-deep-fake-love.html.
  * Sambasivan (2021) Nithya Sambasivan. 2021\.  Seeing like a dataset from the global south.  _Interactions_ 28, 4 (2021), 76–78.
  * Sambasivan et al. (2021) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021\.  “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023\.  Whose opinions do language models reflect?  _arXiv preprint arXiv:2303.17548_ (2023).
  * Scharff et al. (2010) Darcell P. Scharff, Katherine J. Mathews, Pamela Jackson, Jonathan Hoffsuemmer, Emeobong Martin, and Dorothy Edwards. 2010.  More than Tuskegee: Understanding mistrust about research participation.  _Journal of Health Care for the Poor and Underserved_ 21, 3 (2010), 879.
  * Schmidt et al. (1978) Charles F. Schmidt, Natesa S. Sridharan, and John L. Goodson. 1978.  The plan recognition problem: An intersection of psychology and artificial intelligence.  _Artificial Intelligence_ 11, 1-2 (1978), 45–83.
  * @schock (2023) @schock. 2023\.  Now class, can anyone tell me why this might be a bad idea?  https://web.archive.org/web/20230516201648/https://twitter.com/schock/status/1643392611560878086.
  * Schulman (2022) John Schulman. 2022\.  ChatGPT: Optimizing Language Models for Dialogue.  https://web.archive.org/web/20221130180912/https://openai.com/blog/chatgpt/.  Accessed: 2023-09-14.
  * Shah and Bender (2022) Chirag Shah and Emily M. Bender. 2022.  Situating search. In _Proceedings of the 2022 Conference on Human Information Interaction and Retrieval_. 221–232.
  * Shanahan (2022) Murray Shanahan. 2022\.  Talking about large language models.  _arXiv preprint arXiv:2212.03551_ (2022).
  * Shmueli et al. (2021) Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021\.  Beyond fair pay: Ethical implications of NLP crowdsourcing. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 3758–3769.
  * Stige et al. (2023) Åsne Stige, Efpraxia D. Zamani, Patrick Mikalef, and Yuzhen Zhu. 2023.  Artificial intelligence (AI) for user experience (UX) design: A systematic literature review and future research agenda.  _Information Technology & People_ (2023).
  * Stokel-Walker and Van Noorden (2023) Chris Stokel-Walker and Richard Van Noorden. 2023.  What ChatGPT and generative AI mean for science.  _Nature_ 614, 7947 (2023), 214–216.
  * Synthetic Users ([n. d.]) Synthetic Users. [n. d.].  Synthetic Users.  https://web.archive.org/web/20230913012207/https://www.syntheticusers.com/.  Accessed: 2023-09-14.
  * @TetraspaceWest (2022) @TetraspaceWest. 2022\.  [Reply to “Humans can’t accept the truth about GPT-3, so they modified GPT-3 to be understandable”].  https://web.archive.org/web/20230814053841/https://twitter.com/tetraspacewest/status/1608966939929636864.
  * Thompson (2022) Derek Thompson. 2022\.  Breakthroughs of the year.  _The Atlantic_ (2022).  https://web.archive.org/web/20230421060355/https://www.theatlantic.com/newsletters/archive/2022/12/technology-medicine-law-ai-10-breakthroughs-2022/672390/
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022\.  LaMDA: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ (2022).
  * Tiku (2022) Nitasha Tiku. 2022\.  The Google engineer who thinks the company’s AI has come to life.  _The Washington Post_ (2022).  https://web.archive.org/web/20230826183724/https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/
  * Turing (1950) Alan M. Turing. 1950\.  Computing machinery and intelligence.  _Mind_ LIX, 236 (1950), 433–460.
  * U.S. Department of Health and Human Services (2018) U.S. Department of Health and Human Services. 2018.  Code of Federal Regulations: Title 45, Part 46.  https://web.archive.org/web/20230710202634/https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/revised-common-rule-regulatory-text/index.html.  Accessed: 2023-09-14.
  * User Persona ([n. d.]) User Persona. [n. d.].  User Persona.  https://web.archive.org/web/20230722124412/https://userpersona.dev/.  Accessed: 2023-09-14.
  * Van Cuilenburg et al. (1988) Jan J. Van Cuilenburg, Jan Kleinnijenhuis, and Jan A. De Ridder. 1988\.  Artificial intelligence and content analysis: Problems of and strategies for computer text analysis.  _Quality and Quantity_ 22, 1 (1988), 65–97.
  * Vincent (2022) James Vincent. 2022\.  All these images were generated by Google’s latest text-to-image AI.  _The Verge_ (2022).  https://web.archive.org/web/20230725102931/https://www.theverge.com/2022/5/24/23139297/google-imagen-text-to-image-ai-system-examples-paper
  * Vines et al. (2013) John Vines, Rachel Clarke, Peter Wright, John McCarthy, and Patrick Olivier. 2013.  Configuring participation: On how we involve people in design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 429–438.
  * Walsh (2014) Richard T. G. Walsh. 2014\.  _Researcher-participant relationship_.  645–672.
  * Wang et al. (2021) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021.  Want to reduce labeling cost? GPT-3 can help. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. 4195–4205.
  * Weber et al. (1947) Max Weber, A. M. Henderson, and Talcott Parsons. 1947\.  _The theory of social and economic organization_.  Free Press.
  * Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022\.  Taxonomy of risks posed by language models. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 214–229.
  * Weiser (2023) Benjamin Weiser. 2023\.  Here’s what happens when your lawyer uses ChatGPT.  _The New York Times_ (2023).  https://web.archive.org/web/20230913041650/https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html
  * y Arcas (2022) Blaise Agüera y Arcas. 2022\.  Do large language models understand us?  _Daedalus_ 151, 2 (2022), 183–197.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2022.  ReAct: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
  * Zaremba et al. (2023) Wojciech Zaremba, Arka Dhar, Lama Ahmad, Tyna Eloundou, Shibani Santurkar, Sandhini Agarwal, and Jade Leung. 2023.  Democratic inputs to AI.  https://web.archive.org/web/20230525194339/https://openai.com/blog/democratic-inputs-to-ai.  Accessed: 2023-09-14.
  * Zhang et al. (2016) Xiang Zhang, Hans-Frederick Brown, and Anil Shankar. 2016.  Data-driven personas: Constructing archetypal users with clickstreams and user telemetry. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 5350–5359.
  * Ziman (1991) John Ziman. 1991\.  Public understanding of science.  _Science, Technology, & Human Values_ 16, 1 (1991), 99–105.
  * Zytko et al. (2022) Douglas Zytko, Pamela J. Wisniewski, Shion Guha, Eric P. S. Baumer, and Min Kyung Lee. 2022\.  Participatory design of AI systems: Opportunities and challenges across diverse users, relationships, and application domains. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–4.