# Human Participants in AI Research: Ethics and Transparency in Practice

Kevin R. McKee  0000-0002-4412-1686<EMAIL_ADDRESS>0000-0002-4412-1686
Google DeepMind8 Handyside StreetLondonUnited KingdomN1C 4DJ

(2023)

###### Abstract.

In recent years, research involving human participants has been critical to
advances in artificial intelligence (AI) and machine learning (ML),
particularly in the areas of conversational, human-compatible, and cooperative
AI. For example, around 12% and 6% of publications at recent AAAI and NeurIPS
conferences indicate the collection of original human data, respectively. Yet
AI and ML researchers lack guidelines for ethical, transparent research
practices with human participants. Fewer than one out of every four of these
AAAI and NeurIPS papers provide details of ethical review, the collection of
informed consent, or participant compensation. This paper aims to bridge this
gap by exploring normative similarities and differences between AI research
and related fields that involve human participants. Though psychology, human-
computer interaction, and other adjacent fields offer historic lessons and
helpful insights, AI research raises several specific concerns—namely,
participatory design, crowdsourced dataset development, and an expansive role
of corporations—that necessitate a contextual ethics framework. To address
these concerns, this paper outlines a set of guidelines for ethical and
transparent practice with human participants in AI and ML research. These
guidelines can be found in Section 4 on pp. 4–4.3.

artificial intelligence, human participants, research ethics

††copyright: none††ccs: Human-centered computing††ccs: Applied computing Law,
social and behavioral sciences

##  1\. Introduction

> “An overarching and inspiring challenge […] is to build machines that can
> cooperate and collaborate seamlessly with humans and can make decisions that
> are aligned with fluid and complex human values and preferences.” –Littman
> et al. (2021)

> “Development of increasingly sophisticated AI capabilities must go hand-in-
> hand with increasingly sophisticated human-machine interaction.” –Johnson
> and Vera (2019)

Proposed applications of artificial intelligence (AI) frequently invoke images
of collaboration and cooperation between humans and artificial systems.
Current research endeavors imagine AI and machine learning (ML) systems as
assistants, decision aids, and augmentative tools (Littman et al., 2021). Such
proposals increasingly highlight the importance of compatibility and
interactivity with humans (Brynjolfsson, 2022; Dafoe et al., 2020; Irving and
Askell, 2019; Johnson and Vera, 2019; McKee et al., 2023).

The goal of developing human-compatible AI presses the need for research with
human participants, aimed at understanding and validating the interactive
capabilities of AI. In this class of research, humans—and their interactions
with artificial agents—are a primary focal point of behavioral or
psychological interest. Similar to experiments in psychology and human-
computer interaction (HCI), the aim of these studies is to draw general
inferences or make generalizable claims, often beyond the sample at hand.
Human-participation projects are regularly published at AI-specific
conferences such as Neural Information Processing Systems (NeurIPS; (Carroll
et al., 2019; Du et al., 2020)), the International Conference on Machine
Learning (ICML; e.g., (Saha et al., 2020)), the International Conference on
Learning Representations (ICLR; e.g., (Shih et al., 2020)), the annual
conference for the Association for the Advancement of Artificial Intelligence
(AAAI; e.g., (Freedman et al., 2018)), and the annual conference for
Autonomous Agents and Multi-Agent Systems (AAMAS; e.g., (McKee et al., 2022)).
Participants play an additional critical role in both training and evaluating
models at the current frontier of AI research (e.g., (Glaese et al., 2022;
Ouyang et al., 2022; Geng et al., 2023)).

This research area is rapidly growing: across 2021, 2022, and 2023, roughly
12% of the papers at AAAI and 6% of the papers at NeurIPS involved the
collection of human data (Table 1, second column). In other fields,
independent ethical review, the collection of informed consent, and
participant compensation are all considered indispensible for the responsible
conduct of human-participant research. In contrast, fewer than one out of
every three of these AAAI and NeurIPS papers reported details of these
practices (Table 1; see also Appendix A). Guidance for responsible and ethical
research practice with human participants from psychology, HCI, and other
adjacent fields have evidently not reached AI researchers.

This paper aims to bridge this gap by providing guidelines to AI researchers
to support ethical and transparent work with human participants. Neighboring
disciplines—in particular, human-computer interaction, psychology, and the
field of research ethics—offer key lessons to be adapted within the particular
historical and normative context of AI research. Drawing on these insights,
this paper proposes four ethical principles to guide AI research with human
participants (pp. 3–4) and outlines a set of concrete guidelines for ethical
and responsible conduct of AI research with human participants, tailored
specifically for AI researchers who may not have had exposure to such guidance
in the past (pp. 4–4.3).

Publication | Total no. of papers | No. of papers | No. of papers | No. of papers
---|---|---|---|---
involving participants | reporting indep. review | reporting consent | reporting comp.
NeurIPS (2021–2022) | 304 / 5005 (0.06) | 163 / 304 (0.54) | 133 / 304 (0.44) | 102 / 304 (0.34)
AAAI (2021–2023) | 534 / 4551 (0.12) | 53 / 534 (0.10) | 14 / 534 (0.03) | 104 / 534 (0.19)
Total | 814 / 9556 (0.09) | 221 / 814 (0.27) | 159 / 814 (0.20) | 273 / 814 (0.33)
Table 1. Snapshot of research on human participants at two major artificial
intelligence (AI) research conferences from 2021 to 2023. Cells report number
of papers, with the number in parentheses indicating the proportion out of the
total number of papers (second column) or out of the number of papers
involving research participants (third through fifth columns). See Appendix A
for methodology and further results.

##  2\. Contextual concerns: Why AI research needs its own guidelines

The quest for ethical guidelines need not start from scratch in AI research.
Historical milestones in psychology, HCI, biomedical research, and other
adjacent fields provide a number of lessons for the AI community. Chief among
them are the Belmont Report and the Common Rule, two historical and regulatory
artifacts from the United States that have influenced ethical practices around
the world (e.g., (National Health and Medical Research Council, 1999; Holm,
2020; McDonald, 2008); see also (Capron, 2008) and Appendix B). The Belmont
Report and the Common Rule offer a particularly helpful starting point for AI
research, including with the challenge of defining a scope for research
participation (see Appendix C).

Drawing inspiration from the Common Rule and experimental psychology, we can
define human participants as living individuals who provide or contribute
information to researchers, particularly in systematic investigations
“designed to develop or contribute to generalizable knowledge” (45 C.F.R. §
46.102). While research participation is essential for the progress of
scientific research, it is not an especially dignifying pursuit for
participants themselves. Following a series of historic lapses in responsible
research conduct, biomedical research, experimental psychology, and other
human-centered fields developed explicit guidance for conducting studies with
human participants (e.g., (American Psychological Association, 2017; Oates et
al., 2021); see also Appendix B). Frameworks from these nearby human-centered
fields offer a helpful point of embarkation for AI research.

Nonetheless, AI research exists as a distinct scientific discipline.
Disciplinary boundaries have traditionally kept AI, psychology, and HCI
research relatively separate (Grudin, 2009; Yang et al., 2020; Kogan et al.,
2020; Urban and Gates, 2021). Even as new developments—particularly the
growing interest in conversational AI systems—soften those boundaries, AI
scientists face challenges that psychologists and other behavioral scientists
typically do not. Three normative and historic trends in the AI
field—participatory design, crowdsourced dataset development, and the role of
corporations—complicate the issues involved in work with human participants,
and create a need for guidelines tailored to AI research.

The particular definition of human participation considered here—inheriting
from the Common Rule and from experimental psychology—is distinct from
participatory design, a school of thought running through the history of
technology development. Participatory design refers to research approaches
that directly “involv[e] affected populations in shaping the goals of the
overall system” (Kulynych et al., 2020) (see also (Birhane et al., 2022a;
Simonsen and Robertson, 2013)). The origins of participatory design trace back
to mid-20th century movements advocating for greater community involvement in
public decision making (Schuler and Namioka, 1993). Early Scandinavian
experiments in participatory design are particularly well known; these
experiments incorporated union workers in the technological design process,
reflecting a deep commitment to democratization (Gregory, 2003).
“Participation” in this sense is a common calling card in contemporary AI
research (Birhane et al., 2022a; Sloane et al., 2022). For instance, many of
the organizations responsible for the development of large language models
argue that democratic participation must be a core component of future AI
systems (Glaese et al., 2022; Roose and Newton, 2023; Zaremba et al., 2023).
The prominence of such appeals within the field places an onus on researchers
to contend with the relationship between participatory design and their own
work with human participants.

Broadly, participatory design places no particular emphasis on creating
generalizable knowledge. It instead underscores the importance of remaking
power imbalances (Kulynych et al., 2020). Critiquing the misuse of
“participation” language in machine learning research, Sloane et al. (2022)
argue that participatory design can manifest in several modes, each varying in
its extent of power redistribution—including “participation as consultation”
and “participation as justice.” Arnstein (1969) exemplified this concern with
power, arguing that “participation without redistribution of power is an empty
and frustrating process for the powerless.” In contrast with this focus,
redistribution is inherent neither to the Common Rule nor to most experimental
psychology. Participants in the sense of the Common Rule often do not qualify
as participants in the framework of participatory design, given the latter’s
emphasis on co-design and co-direction. Nonetheless, several research
subfields demonstrate that these definitions of participation can be
compatible. Participatory action research (Fals-Borda and Rahman, 1991) and
participatory HCI (Vines et al., 2013), for example, advocate for community
involvement in setting experimental research agendas, effectively marrying
participatory design with human-participant research. Overall, given its
historic and contemporary intertwining with participatory design, AI research
must grapple with the role of communities in human-participant work.

Crowdsourced dataset development, a ubiquitous activity in AI and ML research,
similarly challenges the boundaries of traditional “participant” definitions.
Supervised learning algorithms learn to predict outputs through exposure to
labeled training data (i.e., example input-output pairs). In early periods of
the field, in-house, expert annotators generated the annotations for training
data. These experts were oftentimes the researchers themselves, though
occasionally were students and professional annotators (Snow et al., 2008).
The emergence of online crowdsourcing transformed the annotation process,
improving the speed, scalability, cost, and convenience of dataset
development. AI and ML researchers now crowdsource most labor for data
labeling, annotation, and production (see (Gray and Suri, 2019)). The
recruitment of human annotators appears to be growing rapidly at the frontier
of AI research, given the amount of data needed to fine-tune large language
models (e.g., (Glaese et al., 2022; Ouyang et al., 2022)). A current debate
revolves around whether such crowdsourced workers qualify as research
participants—“a gray area that is open to interpretation,” as noted by Shmueli
et al. (2021). For example, the Association for Computational Linguistics
indicates that “human annotators (e.g., crowdworkers)” are distinct from
“human subjects” in its “responsible research” checklist (Association for
Computational Linguistics, nd). In contrast, Sloane et al. (2022) argue that
such unrecognized contributions to the production and improvement of ML can be
considered “participation as work,” and thus impart particular ethical
responsibilities upon researchers making use of crowdsourcing. The inclusion
of annotators as research participants would have profound practical
implications: it would afford them protections and oversight that could help
protect their well-being and reduce unnecessary harms. Given recent incidents
and the field’s expansive reliance on crowdsourced data (e.g., (Hao and
Seetharaman, 2023; Gray and Suri, 2019)), these protections present a strong
argument to include crowdworkers in the framework of human participation for
AI research. Nonetheless, this ultimately remains a decision for the AI
community to make.

A final concern for AI research revolves around the relationship between
participants and researchers. The fields of psychology and HCI have long
contended with the imbalanced power dynamics built into the researcher-
participant relationship. Most researchers view the sole responsibility of the
participant as responding to the research study as they have been directed.
The participant’s main affordance is to interact with the study materials, or
sometimes with the research team: few other levers are available to
them.111Indeed, this relative lack of autonomy often emerges in the language
used to refer to participants in scientific writing (e.g., “subjects” of a
research study, rather than “participants”) and even in regulatory text (e.g.,
the Common Rule’s passive description of a participant as “an individual about
whom an investigator […] obtains information”; see Appendix D). In contrast,
the researcher controls most functions outside of data production, including
the provision of tools for recourse, compensation for research participation,
and the dissemination of research benefits. Because of this imbalance,
researchers can readily cause harms when attention to ethical practice lapses,
as demonstrated by a long history of oversights (Brandt, 1978; Gillespie,
1989; Flick, 2016).

In AI work, financial and compute resources have proved crucial for recent
progress (Ahmed and Wahed, 2020; Sutton, 2019). These resource requirements
have made technology companies disproportionately influential in AI research,
given the particularly large research budgets and caches of compute that they
can bring to bear on algorithmic development (Birhane et al., 2022b; Luitse
and Denkena, 2021; Martínez-Plumed et al., 2021). The involvement of companies
introduces commercial interests into the research process, complicating the
researcher-participant relationship. Commercial interests create potential
conflicts with the responsibilities that researchers hold toward participants:
research teams may face situations where particular tasks or methods can help
further industry goals, but are not in the best interests of participants (cf.
(Morin et al., 2002)). Separately, the resources held by technology companies
amplify the power imbalance between participants and researchers. Participants
in AI research are likely to encounter especially large disparities in
financial, legal, and public-relations resources and knowledge between
themselves and teams administering research. These imbalances may potentially
limit participants’ agency and access to modes of recourse, especially
relative to individuals participating in research in other
fields.222Recognition of both these commercial interests and these power
imbalances has led in part to the AI community’s particularly strong emphasis
on assessing the “societal impact” of its research (e.g., (Prunkl et al.,
2021; Nanayakkara et al., 2021)).

Overall, these idiosyncratic challenges create concerns for AI and ML
researchers that are not typically shared by scientists. Psychology, HCI, and
related fields can offer initial insights and a shared vocabulary for ethical
research practice, but as a distinct scientific discipline, AI research
necessitates its own frameworks and guidelines.

##  3\. Ethical principles for AI research with human participants

Research ethics is both iterative and contextual. The Belmont Report offered a
foundation for the Common Rule, which in turn formed the basis of modern
ethics frameworks for experimental psychologists (American Psychological
Association, 2017) (see also (Musschenga, 2005) and Appendix B). AI research
can similarly iterate by integrating the normative concerns of participatory
design, crowdsourced dataset development, and the role of corporations into
traditional research ethics frameworks. Through this process, we can consider
four ethical principles—autonomy, beneficence, justice, and accountability—as
a starting point for ethical AI research with human participants. The first
three principles emerge directly from the Belmont Report and the Common Rule,
though contextual concerns give them a new focus. The role of corporations in
AI research elevates the final principle, accountability, as another focal
point for ethical practice with human participants. These four principles
provide justification for the practical guidelines presented in Section 4.
Together, these principles and guidelines form an extended scaffolding for
ethical and responsible practice with human participants.

###  3.1. Autonomy

> “The principle of respect for persons thus divides into two separate moral
> requirements: the requirement to acknowledge autonomy and the requirement to
> protect those with diminished autonomy.” (National Commission for the
> Protection of Human Subjects of Biomedical and Behavioral Research, 1978)

Within the Common Rule framework, the “respect for persons” principle reflects
the idea that researchers should preserve and protect participants’ autonomy.
It places an emphasis on the voluntary nature of participation, motivating
practices such as the upfront discussion of risks and benefits of
participation and the necessity of informed consent for participants.

In the context of AI research, it may be clearer to refer to this concept as
the principle of autonomy. AI ethics frameworks frequently identify autonomy
as a key value for the governance of AI systems, asserting that deployed
systems should promote (rather than restrict) freedom and self-determination
(Floridi and Cowls, 2022; Jobin et al., 2019). Research with human
participants is rarely explicitly discussed within these ethics frameworks.
Nonetheless, researchers and developers should extend the principle of
autonomy to interactions with human research participants. That is, people
should be able to choose how and whether to engage with AI systems, both in
the real world and in the laboratory. As an example of this principle in
practice, scientists should transparently communicate with participants about
their involvement in AI research and development. After all, many individuals
may not wish to support the development of new AI systems. AI research has not
always met this bar (Gibney, 2017).

###  3.2. Beneficence

> “Two general rules have been formulated as complementary expressions of
> [beneficence] in this sense: (1) do not harm and (2) maximize possible
> benefits and minimize possible harms.” (National Commission for the
> Protection of Human Subjects of Biomedical and Behavioral Research, 1978)

The principle of beneficence represents two priorities: first, preserving
participant well-being, and second, maximally advancing the interests of the
participants. It is sometimes distilled to a simple arithmetic rule: do the
benefits resulting from the research outweigh the risks?

In AI research, a growing number of initiatives advocate for an analogous
calculus for “social good” or “social benefit” (Tomasev et al., 2020; Cowls et
al., 2021). In the context of human-participants research, such a calculus
entails limiting the exposure of each participant to risks and potential
harms. Work on algorithmic fairness, for instance, imposes difficult tradeoffs
between group equity and individual privacy. Developing AI systems that treat
marginalized groups fairly requires collecting data from vulnerable members of
those groups, introducing direct risks to the privacy and safety of those
individuals (Tomasev et al., 2021). The prevalence of crowdsourced dataset
development for state-of-the-art systems raises a number of similar questions.
When is it ethical to recruit participants to train models that will replace
them (Brynjolfsson, 2022)? What sort of protections should developers offer
when tasking individual participants with reviewing disturbing or toxic
content when the outcome of research is an AI system intended for use by a
broader population (Hao and Seetharaman, 2023)?

###  3.3. Justice

> “Who ought to receive the benefits of research and bear its burdens? This is
> a question of justice, in the sense of ‘fairness in distribution’ or ‘what
> is deserved.”’ (National Commission for the Protection of Human Subjects of
> Biomedical and Behavioral Research, 1978)

Within the Common Rule framework, the principle of justice codifies the idea
that both the burden and the benefits of research should be equitably shared
among potential participants. Its inclusion in the Belmont Report and the
Common Rule reflects scientists’ and ethicists’ concern over historic
injustices against marginalized communities, as well as the possibility that
future research will continue such exploitation. The justice principle is
especially relevant for research that by design involves participants from
vulnerable or marginalized populations.

Within AI research, who will provide the labor and cost necessary to cultivate
progress, and who will reap the benefits? In the tradition of participatory
design, promoting justice requires understanding the structures of power under
which research unfolds. For instance, recent research advances have vastly
improved the capabilities of contemporary language models; one potential
application of these models is to facilitate efforts to teach and revitalize
endangered languages. However, many indigenous communities still keenly feel
the scars and trauma from their historic experiences with colonialism. As a
result, industry-affiliated researchers working on endangered languages are
likely to meet skepticism from indigenous communities, who fear that any data
collected will be used to help exploit their members (Lewis et al., 2020).
Such cases call for thoughtful attention to the experiences of marginalized
groups, in order to avoid perpetuating further injustice.

###  3.4. Accountability

> “To be accountable for [research], researchers must be able to describe in a
> way sufficient for the social situation at hand how any perceived [research]
> problems are anomalous, correctable, or in fact not problematic at all—they
> must be ‘answerable’ for their [research].”333Mayernik (2017) offers the
> original quote in the context of discussing accountability for data.
> Nonetheless, it eloquently extends to capture the principle of
> accountability in research. (Mayernik, 2017)

The principle of accountability reflects the responsibility of investigators
to explain research processes and outcomes to research participants, to fellow
researchers, to academic and public institutions, and to community
stakeholders. Such explanations should include discussion of the validity,
benefits, and harms of research. In this respect, the principle of
accountability also encompasses an openness to scrutiny concerning potential
issues with research processes and outcomes. As Makel and Plucker (2017)
explains, “at its foundational level, the heart of science is that its methods
allow for others to believe its results.” Klein et al. (2018) similarly argue
that the cumulative and self-corrective properties of science operate best
“when the scientific community is able to access and examine the key products
of research.”

Those involved in psychology and HCI research have issued repeated calls for
accountability, transparency, and social responsibility in scientists’
relationship with the public (Bazelon, 1982; DeLeon, 1988; Grimpe et al.,
2014). Nonetheless, the Belmont framework does not make accountability focal
as a principle. The AI community, in contrast, widely recognizes that
accountability is a lodestone value for algorithmic deployment (Diakopoulos,
2016; Floridi and Cowls, 2022; Jobin et al., 2019). Accountability is
similarly essential for AI work with human participants, on account of the
central involvement of corporations and competitive interests. “Ghost work”
exemplifies the exploitative practices and harms that can proliferate when
researchers obscure the individuals that provide data for algorithmic
development and evaluation (Gray and Suri, 2019).

##  4\. Principles in practice: Guidelines for AI research with human
participants

Autonomy, beneficence, justice, and accountability provide a rough direction
of travel for the ethical developer. To best improve the odds of upholding
ethical conduct, AI researchers also need navigational tools: some method of
translating principles into practice. In scientific fields, guidelines are one
common way of doing so. This section draws from the four principles, in
addition to sources from psychological and behavioral research (e.g.,
(American Psychological Association, 2017; Oates et al., 2021)), and presents
a set of guidelines tailored for AI research with human participants. This
guidance can be separated into three groups: guidelines for before the study,
during the study, and after the study. To emphasize the theoretical motivation
for this guidance, each guideline identifies its corresponding principle or
principles in italics.

###  4.1. Before the study: Independent ethics review

At the beginning of the research process, the research team takes several
important steps that lay the conceptual foundation for ethical interactions
with participants. All four principles motivate this ethics groundwork.

Undergo independent ethical review. Researchers should subject their proposed
research studies to independent ethical review before starting any data
collection or processing. Primarily, this entails an assessment by a group
with a mixture of scientific experts, community representatives, and at least
some members knowledgeable of research ethics. Group members should neither be
affiliated with the research (e.g., not a member of the research team) nor
possess conflicting interests with regard to research outcomes (e.g.,
economically benefit from any outcomes). Many countries designate official
bodies to conduct independent ethical review for research with human
participants, such as institutional review boards (IRBs) in the United States
and research ethics committees in the United Kingdom and the World Health
Organization framework. In the United States, many argue that IRBs have become
overly bureaucratized, focusing more on procedural concerns than deeper
ethical questions (Heimer and Petty, 2010; White, 2007). The purpose of
independent review is not to subject data collection to excessive bureaucratic
work and yellow tape, but to think earnestly about whether a proposal advances
ethical research principles. For example, various organizations have developed
a new class of analogous ethics bodies outside the IRB framework, including
the Ethics and Society Review board at Stanford University (Bernstein et al.,
2021), the Ethics Review Program at Microsoft Research (Microsoft, nd), and
the Institutional Review Committee at Google DeepMind (Google DeepMind, ndb).
Independent ethical review is imperative for AI researchers undertaking work
with human participants, supporting each of the four principles: autonomy,
beneficence, justice, and accountability.

Solicit peer feedback. To supplement formal review processes, researchers
should consider seeking out informal feedback on their research proposals from
their peers. Peers are often well positioned to evaluate and offer comment on
proposed studies, both with their disciplinary expertise and from any ethical
concerns they have. Relative to reviews by institutional committees, peer
feedback may more effectively assess “technical” aspects of research, such as
the importance of a research question or the validity of a proposed study
design. In addition, institutional committees may not always be aware of the
behavior of state-of-the-art models. Inappropriate design choices may
undermine results and inferences, reducing the benefits of research (as per
the principle of beneficence). Given their on-the-ground perspective, peer
feedback on ethical considerations may also serve as a useful supplement to
formal review. This also offers peers themselves practice with ethical design.
Ethical deliberation need not be restricted to institutional review, after
all: researchers should have the chance to deliberate on potential dilemmas
with their colleagues (Hilppö et al., 2019). In a decentralized manner, these
peer channels can help encourage the development of an ethical community among
AI researchers. As with the guideline of undergoing formal ethical review,
this guideline pertains to each of the four principles: autonomy, beneficence,
justice, and accountability.

Strongly justify any use of deception. Experiments have long made use of
deception; for nearly just as long, psychologists and other social scientists
have debated the acceptability of deception as an experimental practice
(Herrera, 1997; Ortmann and Hertwig, 2002). These questions are especially
critical for AI research. Should researchers alert or disclose to participants
that they are interacting with AI systems? More broadly, should researchers
inform participants of the sorts of systems they may be helping to develop?
Concealing the involvement or goals of AI research from participants
inherently conflicts with the principles of autonomy and beneficence.
Researchers should have a cogent justification for deceiving participants,
such as investigating the effects of “dishonest anthropomorphism” (Leong and
Selinger, 2019). In studies where deception is justified, researchers should
debrief participants as soon as data collection is complete, informing them of
the deception and providing corrected information.

###  4.2. During the study: Ethical conduct and engagement with participants

While conducting the study, the researcher should seek not just to carry out
their planned research protocol, but also to affirm participant dignity. The
following guidelines are tailored to online research platforms, given the
current prevalence of online study administration. Nonetheless, they can also
be readily adapted to in-person studies or other modes of research. Overall,
this set of guidelines is motivated by the principles of autonomy and
beneficence.

Collect (genuinely) informed consent. The concept of informed consent captures
two interwoven concerns. First, researchers should provide any individuals
they recruit with as much study information as possible. Second, recruited
individuals should only participate after processing this information and
providing voluntary, decisionally capacitated consent. The consent process
should communicate information including the general research topic, the
general activities involved, risks and benefits of participation, compensation
details, participants’ right to withdraw, and contact information in case
participants have any questions or concerns. To minimize the difficulty
participants have processing this information, investigators should write
forms in easily comprehensible language (Perrault and Nazione, 2016; Perrault
and Keating, 2018). Taking the step of collecting informed consent supports
the principle of autonomy.

Limit risks. Research participation should entail minimal risk of harm for
participants. Researchers should not just consider physical or psychological
risks. Potential risks from research can also be social, legal, or economic in
nature. The ideal standard is no-greater-than-minimal (“everyday”) risk. As
defined by the Common Rule, no-greater-than-minimal risk means that “the
probability and magnitude of harm or discomfort anticipated in the research
are not greater in and of themselves than those encountered in daily life or
during the performance of routine physical or psychological examinations or
tests” (45 C.F.R. § 46.102). Researchers should be able to provide a strong
justification of any higher level of risk with specific, concrete benefits or
needs, as per the principle of beneficence. Human-AI interaction work merits
particular attention to risks. For example, when chatting with human
participants, conversational models may produce toxic or harmful language or
provide professional advice against a participant’s best interests, despite an
initial appearance of helpfulness and harmlessness (see also (Weidinger et
al., 2022)). Limiting such risks underpins the principle of beneficence.

Support dignity. Research participation is not a particularly empowering
experience. Researchers should especially strive to support participants’
dignity. Participants provide researchers the privilege and honor of access to
their thoughts and perspectives. Studies can provide opportunities for
participants to voice their opinions, thoughts, and reactions to the study.
Often, this is as simple as including an open-ended debriefing question at the
end of the study, asking: “Do you have any other thoughts you’d like to share
with us?” Despite the frustrations that participants experience during
studies, many also feel pride and derive personal meaning from their
contributions ((Fowler et al., 2022; Gupta et al., 2014); cf. (Allan et al.,
2019)).444Researchers genuinely interested in understanding the experience and
perspectives of research participants can read posts on open forums where
participants discuss their experiences and ask each other for advice (e.g.,
https://www.reddit.com/r/ProlificAc/). These posts offer a peek into the ups
and downs of research work, including the range of joys and frustrations that
participants feel on a daily basis. An especially important way for
researchers to support these positive experiences is to inform participants
about the purpose of studies to which they contribute (e.g., disclosing the
intended uses of any system their data will help to develop). Even seemingly
small moments can support dignity, such as taking inclusive approaches to
asking participants about their gender identity when collecting demographic
data (e.g., (Fraser, 2018)). These efforts enshrine the principles of
autonomy, beneficence, and justice.

Maintain channels for clear and timely communication with participants. Online
data collection rarely goes as planned. Server connectivity issues, unforeseen
bugs in user interfaces, misconfigured study conditions, unclear
instructions—a long list of problems can disrupt online studies. Participants
consistently cite lack of communication channels and lack of researcher
responsiveness among their top frustrations with online research (Berg, 2015;
Fowler et al., 2022). In contrast, researchers in on-site laboratories rarely
leave participants unsupervised. Typically, a team member remains on hand to
answer questions and troubleshoot any technical difficulties. The same
standards should apply to online data collection: researchers hold the
responsibility to maintain convenient channels (and remain personally present)
for clear and timely communication with participants. Providing such channels
reinforces the principles of beneficence and accountability.

Ensure fair compensation. Low compensation is a substantial problem among
online recruitment platforms (Semuels, 2018). In one study, researchers found
that the median online participant on a popular research platform earned only
$2 each hour (Hara et al., 2018). On these platforms, a participant’s
compensation is distributed across time spent completing studies, as well as
time spent searching for studies and reading instructions. Researchers can
adopt several different models for compensation, including payment per
completed task or payment per hour (Partnership on AI, nd). Given the
possibility of misestimating the time a task takes to complete, an hourly rate
offers a stable, predictable financial structure for both participants and
research teams. Researchers should strongly consider using living wage
calculations (Global Living Wage Coalition, nd) as a standard for determining
study compensation. Researchers can also consider an adaptive system,
supplementing compensation for individual participants who are particularly
engaged or perform especially well on a task (potentially up to a specified
upper bound). During recruitment, researchers should clearly communicate
compensation details, including any conditions for study compensation and any
lower or upper limits for studies with flexible compensation. During the
study, researchers should monitor the duration of each session, offering
bonuses for any “overtime” participation. Overall, fair compensation promotes
the principle of beneficence.

Ensure ability to withdraw without consequence. Participants should be able to
freely withdraw from a study, even after initially providing their consent for
participation. Withdrawal should not carry any adverse consequences for
participants; researchers should still compensate participants for the time
they spent on these partial sessions (e.g., by pro-rating compensation).
Participants may also request to withdraw their research data from the study.
Consent forms should explain as clearly as possible the point at which
investigators can no longer identify and withdraw a particular participant’s
data. This guideline reinforces the principles of autonomy and beneficence.

Avoid coercion and undue influence. Coercion and undue influence are two
interconnected concepts in research ethics. Coercion refers to implicit or
explicit threats to cause harm to an individual or to withhold a benefit from
them in order to change their behavior (e.g., coercing them to complete a
research study). It may be a concern when inviting employees of a company to
participate in research studies, in particular when the investigators work at
the same company (Resnik, 2016). This tension is especially relevant for AI
research, given the number of industry researchers in the field. Undue
influence manifests when researchers offer excessive amounts of compensation
when recruiting participants, typically for studies that may conflict with
participants’ interests and well-being. Thus, researchers should be especially
careful when working on research involving higher levels of risk (e.g., bias
or toxicity work with language models). This guideline supports the principles
of autonomy, beneficence, and justice.

###  4.3. After the study: Transparent reporting

This final subset of guidelines identifies information that researchers should
publicly report about their work, framed as questions that can be answered in
the main text or appendices of publications. Existing sources in AI research
outline several (but not all) of these details (Association for Computational
Linguistics, nd; Neural Information Processing Systems, nd). Given their focus
on encouraging the transparent and public dissemination of information, each
of these guidelines directly supports the principle of accountability.

Report details of independent ethical review. Did the protocol receive
independent ethical review before data collection began? What was the outcome
of that review? What is the name of the body that undertook the assessment? If
that body assigned the reviewed protocol an identifier, researchers should
provide that identifier. If any peers provided feedback on the ethics of the
research, this information should also be included.

Report collection of informed consent. Did participants provide informed
consent before data collection began? Did any issues arise during the
provision of study information or the collection of informed consent? Were
participants informed of the purpose of the study, including the connection to
AI research and the involvement of any AI systems?

Report recruitment source. Where and how were participants recruited? Popular
sources include dedicated online participant pools, such as Prolific (Pe’er et
al., 2021) and Mechanical Turk (Buhrmester et al., 2011). Given recent
concerns about financial exploitation of online participants (Semuels, 2018;
Hara et al., 2018), some recruitment platforms—including Prolific—incorporate
guarantees for fair compensation rates. Studies also frequently recruit from
student populations at academic institutions or from online communities (e.g.,
Reddit fora; see (Bartneck et al., 2015; Shatz, 2017)). Broader populations
may serve as the recruitment source for projects leveraging panel methods.

Report sampling method. What approach was used to sample participants? In
combination with recruitment source, the sampling method of a study helps
determine study generalizability. Certain sampling methods, including
representative panel approaches, strengthen the generalizability of study
results. Others, including convenience or snowball sampling, can reduce costs
and facilitate recruitment at the cost of generalizability.

Report study duration. How long did participants take to complete the study,
on average?

Report compensation details. What was the average compensation provided to
participants? When appropriate, include details about base compensation,
average bonus, and minimum hourly compensation.

Report procedure and methods. What steps did the study follow? Which materials
did the study use (e.g., tasks or questions for participants)? These details
closely relate to experimental replicability. Researchers should provide
materials where possible (e.g., screenshots of participant interface) or refer
to other sources that allow other researchers to re-implement the protocol.

##  5\. Discussion

AI research with human participants currently falls short of the ethical
standards expected of behavioral research. To help address this gap, this
paper explores both normative concerns in AI research and historical tragedies
in the treatment of research participants to construct a guiding framework for
ethical AI research with human participants. This framework identifies four
key principles (autonomy, beneficence, justice, and accountability) and a set
of practical guidelines (outlined in Section 4) to help AI researchers
navigate their work amidst the context of participatory design, dataset
crowdsourcing, and the major role played by companies in research and
development efforts.

The principles and guidelines described here are neither exhaustive nor
finalized. They provide no formal guarantee that a research study provides a
dignifying experience for participants, but should help foster ethical,
responsible interactions between participants and researchers. Additions,
modifications, and deletions will likely prove necessary over the long run.
For instance, given the influence of participatory design on AI research, co-
partnership—in the sense of community participation in research agenda-
setting—may prove a promising addition. Do AI researchers have a
responsibility to partner with research participants when identifying ideal
outcomes and benefits of research? A co-partnership principle could expand
beyond the focus on protections encoded by existing principles and help
advance the difficult work of aligning AI systems with human values (Gabriel,
2020).

Overall, the framework presented here is a starting point for AI researchers
to responsibly conduct research with human participants. Such research efforts
are likely to increase in scope and pace as contemporary AI continues to
advance and expand. This ethical framework is likewise a conversation in
progress—more discussion will follow.

###### Acknowledgements.

I am indebted to Mark Díaz, Seliem El-Sayed, Sara McLoughlin Figel, Julia
Haas, Will Hawkins, Piotr Mirowski, Patrick Pilarski, and Laura Weidinger for
their insightful comments and feedback on this manuscript, and to Manuel
Kroiss and Markus Kunesch for technical support on the paper annotation
process.

## References

  * (1)
  * Ahmed and Wahed (2020) Nur Ahmed and Muntasir Wahed. 2020.  The de-democratization of AI: Deep learning and the compute divide in artificial intelligence research.  _arXiv preprint arXiv:2010.15581_ (2020).
  * Allan et al. (2019) Blake A. Allan, Cassondra Batz-Barbarich, Haley M. Sterling, and Louis Tay. 2019.  Outcomes of meaningful work: A meta-analysis.  _Journal of Management Studies_ 56, 3 (2019), 500–528.
  * American Psychological Association (2017) American Psychological Association. 2017\.  Ethical principles of psychologists and code of conduct.  https://www.apa.org/ethics/code.  Accessed: 2023-08-15.
  * American Psychological Association (2019) American Psychological Association. 2019\.  Participation in research.  https://apastyle.apa.org/style-grammar-guidelines/bias-free-language/research-participation.  Accessed: 2023-08-15.
  * Arnstein (1969) Sherry R. Arnstein. 1969\.  A ladder of citizen participation.  _Journal of the American Institute of Planners_ 35, 4 (1969), 216–224.
  * Association for Computational Linguistics (nd) Association for Computational Linguistics. n.d.  Responsible NLP research checklist.  https://aclrollingreview.org/Responsible_NLP_Research_Checklist___Fillable_form.pdf.  Accessed: 2023-08-15.
  * Bartneck et al. (2015) Christoph Bartneck, Andreas Duenser, Elena Moltchanova, and Karolina Zawieska. 2015.  Comparing the similarity of responses received from studies in Amazon’s Mechanical Turk to studies conducted online and with direct recruitment.  _PLOS One_ 10, 4 (2015), e0121595.
  * Bazelon (1982) David L. Bazelon. 1982\.  Veils, values, and social responsibility.  _American Psychologist_ 37, 2 (1982), 115.
  * Berg (2015) Janine Berg. 2015\.  Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers.  _Comparative Labor Law & Policy Journal_ 37 (2015), 543.
  * Bernstein et al. (2021) Michael S. Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz, and Charla Waeiss. 2021.  ESR: Ethics and Society Review of Artificial Intelligence Research.  _arXiv preprint arXiv:2106.11521_ (2021).
  * Bibace et al. (2009) Roger Bibace, Joshua W. Clegg, and Jaan Valsiner. 2009\.  What is in a name? Understanding the implications of participant terminology.  _Integrative Psychological and Behavioral Science_ 43, 1 (2009), 67–77.
  * Birhane et al. (2022a) Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022a.  Power to the people? Opportunities and challenges for participatory AI.  _Equity and Access in Algorithms, Mechanisms, and Optimization_ (2022), 1–8.
  * Birhane et al. (2022b) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022b.  The values encoded in machine learning research. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 173–184.
  * Brandt (1978) Allan M. Brandt. 1978\.  Racism and research: The case of the Tuskegee Syphilis Study.  _Hastings Center Report_ (1978), 21–29.
  * Brannigan et al. (2015) Augustine Brannigan, Ian Nicholson, and Frances Cherry. 2015\.  Introduction to the special issue: Unplugging the Milgram machine.  _Theory & Psychology_ 25, 5 (2015), 551–563.
  * Brynjolfsson (2022) Erik Brynjolfsson. 2022\.  The Turing trap: The promise & peril of human-like artificial intelligence.  _Daedalus_ 151, 2 (2022), 272–287.
  * Buhrmester et al. (2011) Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling. 2011\.  Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality data?  _Perspectives on Psychological Science_ 6, 1 (2011), 3–5.
  * Campbell and Stanley (1963) Donald T. Campbell and Julian C. Stanley. 1963.  _Experimental and quasi-experimental designs for research_.  Rand McNally & Company.
  * Capron (2008) Alexander M. Capron. 2008\.  _Legal and regulatory standards of informed consent in research_.  Oxford University Press.
  * Carroll et al. (2019) Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019.  On the utility of learning about humans for human-AI coordination.  _Advances in Neural Information Processing Systems_ 32 (2019), 5174–5185.
  * Cowls et al. (2021) Josh Cowls, Andreas Tsamados, Mariarosaria Taddeo, and Luciano Floridi. 2021.  A definition, benchmark and database of AI for social good initiatives.  _Nature Machine Intelligence_ 3, 2 (2021), 111–115.
  * Dafoe et al. (2020) Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. 2020\.  Open problems in Cooperative AI.  _arXiv preprint arXiv:2012.08630_ (2020).
  * DeLeon (1988) Patrick H. DeLeon. 1988\.  Public policy and public service: Our professional duty.  _American Psychologist_ 43, 4 (1988), 309.
  * Diakopoulos (2016) Nicholas Diakopoulos. 2016\.  Accountability in algorithmic decision making.  _Commun. ACM_ 59, 2 (2016), 56–62.
  * Du et al. (2020) Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. 2020\.  AvE: Assistance via empowerment.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Evans (2010) James A. Evans. 2010\.  Industry collaboration, scientific sharing, and the dissemination of knowledge.  _Social Studies of Science_ 40, 5 (2010), 757–791.
  * Fals-Borda and Rahman (1991) Orlando Fals-Borda and Muhammad Anisur Rahman. 1991.  Action and knowledge: Breaking the monopoly with participatory action-research.  (1991).
  * Fisher (1935) Ronald A. Fisher. 1935\.  _The design of experiments_.  Oliver & Boyd, Edinburgh & London.
  * Flick (2016) Catherine Flick. 2016\.  Informed consent and the Facebook emotional manipulation study.  _Research Ethics_ 12, 1 (2016), 14–28.
  * Floridi and Cowls (2022) Luciano Floridi and Josh Cowls. 2022.  A unified framework of five principles for AI in society.  _Machine learning and the city: Applications in architecture and urban design_ (2022), 535–545.
  * Fontana and Frey (2000) Andrea Fontana and James H. Frey. 2000.  _The interview: From structured questions to negotiated text_. Vol. 2.  645–672.
  * Fowler et al. (2022) Craig Fowler, Jian Jiao, and Margaret Pitts. 2022.  Frustration and ennui among Amazon MTurk workers.  _Behavior Research Methods_ (2022), 1–17.
  * Fraser (2018) Gloria Fraser. 2018\.  Evaluating inclusive gender identity measures for use in quantitative psychological research.  _Psychology & Sexuality_ 9, 4 (2018), 343–357.
  * Freedman et al. (2018) Rachel Freedman, Jana Schaich Borg, Walter Sinnott-Armstrong, John P. Dickerson, and Vincent Conitzer. 2018.  Adapting a kidney exchange algorithm to align with human values.  _Proceedings of the 32nd AAAI Conference on Artificial Intelligence_ (2018).
  * Gabriel (2020) Iason Gabriel. 2020\.  Artificial intelligence, values, and alignment.  _Minds and Machines_ 30, 3 (2020), 411–437.
  * Gamble (1993) Vanessa Northington Gamble. 1993\.  A legacy of distrust: African Americans and medical research.  _American Journal of Preventive Medicine_ 9, 6 (1993), 35–38.
  * Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.  Koala: A dialogue model for academic research.  https://bair.berkeley.edu/blog/2023/04/03/koala/.  Accessed: 2023-08-15.
  * Gero et al. (2020) Katy Ilonka Gero, Zahra Ashktorab, Casey Dugan, Qian Pan, James Johnson, Werner Geyer, Maria Ruiz, Sarah Miller, David R. Millen, Murray Campbell, et al. 2020\.  Mental models of AI agents in a cooperative game setting.  (2020), 1–12.
  * Gibney (2017) Elizabeth Gibney. 2017\.  Google secretly tested AI bot.  _Nature_ 541, 7636 (2017), 142.
  * Gillespie (1989) Richard Gillespie. 1989\.  Research on human subjects: an historical overview.  _Bioethics News_ 8, 2 (1989), s4–s15.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022\.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_ (2022).
  * Global Living Wage Coalition (nd) Global Living Wage Coalition. n.d.  Living Wages Resource Library.  https://www.globallivingwage.org/resource-library/?fwp_resource_type=livingwage.  Accessed: 2023-08-15.
  * Google DeepMind (nda) Google DeepMind. n.da.  About – Google DeepMind.  https://deepmind.google/about/.  Accessed: 2023-11-01.
  * Google DeepMind (ndb) Google DeepMind. n.db.  Human rights policy.  https://www.deepmind.com/human-rights-policy.  Accessed: 2023-08-15.
  * Gray and Suri (2019) Mary L. Gray and Siddharth Suri. 2019.  _Ghost work: How to stop Silicon Valley from building a new global underclass_.  Eamon Dolan Books.
  * Gregory (2003) Judith Gregory. 2003\.  Scandinavian approaches to participatory design.  _International Journal of Engineering Education_ 19, 1 (2003), 62–74.
  * Grimpe et al. (2014) Barbara Grimpe, Mark Hartswood, and Marina Jirotka. 2014\.  Towards a closer dialogue between policy and practice: Responsible design in HCI.  _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (2014), 2965–2974.
  * Grudin (2009) Jonathan Grudin. 2009\.  AI and HCI: Two fields divided by a common focus.  _AI Magazine_ 30, 4 (2009), 48–48.
  * Gupta et al. (2014) Neha Gupta, Andy Crabtree, Tom Rodden, David Martin, and Jacki O’Neill. 2014.  Understanding Indian crowdworkers. In _Back to the Future of Organizational Work: Crowdsourcing and Digital Work Marketplaces Workshop @ CSCW 2014_.
  * Hao and Seetharaman (2023) Karen Hao and Deepa Seetharaman. 2023.  Cleaning up ChatGPT takes heavy toll on human workers.  _The Wall Street Journal_ (2023).
  * Hara et al. (2018) Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018.  A data-driven analysis of workers’ earnings on Amazon Mechanical Turk.  _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (2018), 1–14.
  * Hawkins and Mittelstadt (2023) Will Hawkins and Brent Mittelstadt. 2023.  The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 261–270.
  * Heimer and Petty (2010) Carol A. Heimer and JuLeigh Petty. 2010.  Bureaucratic ethics: IRBs and the legal regulation of human subjects research.  _Annual Review of Law and Social Science_ 6 (2010), 601–626.
  * Herrera (1997) C. D. Herrera. 1997\.  A historical interpretation of deceptive experiments in American psychology.  _History of the Human Sciences_ 10, 1 (1997), 23–36.
  * Hilppö et al. (2019) Jaakko Hilppö, Niklas Alexander Chimirri, and Antti Rajala. 2019.  Theorizing research ethics for the study of psychological phenomena from within relational everyday life.  _Human Arenas_ 2 (2019), 405–415.
  * Holm (2020) Søren Holm. 2020\.  Belmont in Europe: A mostly indirect influence.  _Perspectives in Biology and Medicine_ 63, 2 (2020), 262–276.
  * Irving and Askell (2019) Geoffrey Irving and Amanda Askell. 2019.  AI safety needs social scientists.  _Distill_ 4, 2 (2019), e14.
  * Jackman and Kanerva (2015) Molly Jackman and Lauri Kanerva. 2015.  Evolving the IRB: Building robust review for industry research.  _Washington and Lee Law Review Online_ 72 (2015), 442.
  * Jobin et al. (2019) Anna Jobin, Marcello Ienca, and Effy Vayena. 2019\.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (2019), 389–399.
  * Johnson and Vera (2019) Matthew Johnson and Alonso Vera. 2019.  No AI is an island: The case for teaming intelligence.  _AI Magazine_ 40, 1 (2019), 16–28.
  * Klein et al. (2018) Olivier Klein, Tom E. Hardwicke, Frederik Aust, Johannes Breuer, Henrik Danielsson, Alicia Hofelich Mohr, Hans IJzerman, Gustav Nilsonne, Wolf Vanpaemel, Michael C. Frank, et al. 2018\.  A practical guide for transparency in psychological science.  _Collabra: Psychology_ 4, 1 (2018).
  * Kogan et al. (2020) Marina Kogan, Aaron Halfaker, Shion Guha, Cecilia Aragon, Michael Muller, and Stuart Geiger. 2020\.  Mapping out human-centered data science: Methods, approaches, and best practices.  _Companion of the 2020 ACM International Conference on Supporting Group Work_ (2020), 151–156.
  * Kramer et al. (2014) Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014.  Experimental evidence of massive-scale emotional contagion through social networks.  _Proceedings of the National Academy of Sciences_ 111, 24 (2014), 8788–8790.
  * Kulynych et al. (2020) Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Angela Zhou, and Richard Zemel. 2020\.  Participatory approaches to machine learning.  International Conference on Machine Learning Workshop.
  * Leong and Selinger (2019) Brenda Leong and Evan Selinger. 2019.  Robot eyes wide shut: Understanding dishonest anthropomorphism.  _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (2019), 299–308.
  * Lewis et al. (2020) Jason Edward Lewis, Angie Abdilla, Noelani Arista, Kaipulaumakaniolono Baker, Scott Benesiinaabandan, Michelle Brown, Melanie Cheung, Meredith Coleman, Ashley Cordes, Joel Davison, et al. 2020\.  _Indigenous protocol and artificial intelligence position paper_.  Technical Report. Aboriginal Territories in Cyberspace.
  * Littman et al. (2021) Michael L. Littman, Ifeoma Ajunwa, Guy Berger, Craig Boutilier, Morgan Currie, Finale Doshi-Velez, Gillian Hadfield, Michael C. Horowitz, Charles Isbell, Hiroaki Kitano, et al. 2021\.  Gathering strength, gathering storms: The one hundred year study on artificial intelligence (AI100) 2021 study panel report.
  * Luitse and Denkena (2021) Dieuwertje Luitse and Wiebke Denkena. 2021.  The great transformer: Examining the role of large language models in the political economy of AI.  _Big Data & Society_ 8, 2 (2021), 20539517211047734\.
  * Makel and Plucker (2017) Matthew C. Makel and Jonathan A. Plucker. 2017.  _Toward a more perfect psychology: Improving trust, accuracy, and transparency in research_.  American Psychological Association.
  * Martínez-Plumed et al. (2021) Fernando Martínez-Plumed, Pablo Barredo, Sean O. Heigeartaigh, and Jose Hernandez-Orallo. 2021\.  Research community dynamics behind popular AI benchmarks.  _Nature Machine Intelligence_ 3, 7 (2021), 581–589.
  * Mayernik (2017) Matthew S. Mayernik. 2017\.  Open data: Accountability and transparency.  _Big Data & Society_ 4, 2 (2017), 2053951717718853\.
  * McDonald (2008) Michael McDonald. 2008\.  From code to policy statement: Creating Canadian policy for ethical research involving humans.  _Health Law Review_ 17 (2008), 12–25.
  * McKee et al. (2022) Kevin R. McKee, Xuechunzi Bai, and Susan T. Fiske. 2022\.  Warmth and competence in human-agent cooperation.  _Proceedings of the 21st International Conference on Autonomous Agents and MultiAgent Systems_ (2022).
  * McKee et al. (2023) Kevin R. McKee, Xuechunzi Bai, and Susan T. Fiske. 2023\.  Humans perceive warmth and competence in artificial intelligence.  _iScience_ 26 (2023), 107256.  Issue 8.
  * Meyer (2020) Michelle N. Meyer. 2020\.  There oughta be a law: When does (n’t) the US Common Rule apply?  _Journal of Law, Medicine & Ethics_ 48, S1 (2020), 60–73.
  * Microsoft (nd) Microsoft. n.d.  Microsoft Research Ethics Review Program & IRB.  https://www.microsoft.com/en-us/research/microsoft-research-ethics-review-program-irb/.  Accessed: 2023-08-15.
  * Milgram (1974) Stanley Milgram. 1974\.  _Obedience to authority_.
  * Mook (1983) Douglas G. Mook. 1983\.  In defense of external invalidity.  _American Psychologist_ 38, 4 (1983), 379.
  * Morin et al. (2002) Karine Morin, Herbert Rakatansky, Frank A. Riddick Jr, Leonard J. Morse, John M. O’Bannon III, Michael S. Goldrich, Priscilla Ray, Matthew Weiss, Robert M. Sade, and Monique A. Spillman. 2002.  Managing conflicts of interest in the conduct of clinical trials.  _Journal of the American Medical Association_ 287, 1 (2002), 78–84.
  * Morrison (1994) Toni Morrison. 1994\.  Nobel lecture 1993.  _World Literature Today_ 68, 1 (1994), 4–8.
  * Musschenga (2005) Albert W. Musschenga. 2005\.  Empirical ethics, context-sensitivity, and contextualism.  _The Journal of Medicine and Philosophy_ 30, 5 (2005), 467–490.
  * Nanayakkara et al. (2021) Priyanka Nanayakkara, Jessica Hullman, and Nicholas Diakopoulos. 2021.  Unpacking the expressed consequences of AI research in broader impact statements.  _AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society_ (2021).
  * National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1978) National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. 1978\.  _The Belmont report: Ethical principles and guidelines for the protection of human subjects of research_.  United States Government Printing Office.
  * National Health and Medical Research Council (1999) National Health and Medical Research Council. 1999.  National statement on ethical conduct in research involving humans.  https://trove.nla.gov.au/version/45395935.  Accessed: 2023-08-15.
  * Neural Information Processing Systems (nd) Neural Information Processing Systems. n.d.  NeurIPS 2021 Paper Checklist Guidelines.  https://neurips.cc/Conferences/2021/PaperInformation/PaperChecklistview.org/Responsible_NLP_Research_Checklist___Fillable_form.pdf.  Accessed: 2023-08-15.
  * Nicholson (2011) Ian Nicholson. 2011\.  “Torture at Yale”: Experimental subjects, laboratory torment and the “rehabilitation” of Milgram’s “Obedience to Authority”.  _Theory & Psychology_ 21, 6 (2011), 737–761.
  * Nuremberg Military Tribunals (1947) Nuremberg Military Tribunals. 1947\.  The Nuremberg code.
  * Oates et al. (2021) John Oates, David Carpenter, Martin Fisher, Simon Goodson, Beth Hannah, Richard Kwiatowski, Kisane Prutton, Dawn Reeves, and Tony Wainwright. 2021.  _BPS code of human research ethics_.  Technical Report. British Psychological Society.
  * OpenAI (nd) OpenAI. n.d.  OpenAI Charter.  https://openai.com/charter.  Accessed: 2023-11-01.
  * Ortmann and Hertwig (2002) Andreas Ortmann and Ralph Hertwig. 2002.  The costs of deception: Evidence from psychology.  _Experimental Economics_ 5, 2 (2002), 111–131.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Parker and Tritter (2006) Andrew Parker and Jonathan Tritter. 2006.  Focus group method and methodology: Current practice and recent debate.  _International Journal of Research & Method in Education_ 29, 1 (2006), 23–37.
  * Partnership on AI (nd) Partnership on AI. n.d.  Responsible sourcing of data enrichment services.  https://partnershiponai.org/wp-content/uploads/2021/08/PAI-Responsible-Sourcing-of-Data-Enrichment-Services.pdf.  Accessed: 2023-08-15.
  * Paulhus et al. (2007) Delroy L Paulhus, Simine Vazire, et al. 2007\.  _The self-report method_. Vol. 1.  Guilford;, 224–239.
  * PDFium authors (2023) PDFium authors. 2023\.  PDFium.  https://pdfium.googlesource.com/pdfium/+/refs/heads/chromium/5961/README.md.  Accessed: 2023-08-15.
  * Pe’er et al. (2021) Eyal Pe’er, David Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. 2021.  Data quality of platforms and panels for online behavioral research.  _Behavior Research Methods_ (2021), 1–20.
  * Perrault and Keating (2018) Evan K. Perrault and David M. Keating. 2018.  Seeking ways to inform the uninformed: Improving the informed consent process in online social science research.  _Journal of Empirical Research on Human Research Ethics_ 13, 1 (2018), 50–60.
  * Perrault and Nazione (2016) Evan K. Perrault and Samantha A. Nazione. 2016.  Informed consent–uninformed participants: Shortcomings of online social science consent forms and recommendations for improvement.  _Journal of Empirical Research on Human Research Ethics_ 11, 3 (2016), 274–280.
  * Prunkl et al. (2021) Carina E. A. Prunkl, Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe. 2021.  Institutionalizing ethics in AI through broader impact requirements.  _Nature Machine Intelligence_ 3, 2 (2021), 104–110.
  * Resnik (2016) David B. Resnik. 2016\.  Employees as research participants: ethical and policy issues.  _IRB: Ethics & Human Research_ 38, 4 (2016), 11–16.
  * Roose and Newton (2023) Kevin Roose and Casey Newton. 2023.  Dario Amodei, C.E.O. of Anthropic, on the paradoxes of A.I. safety and Netflix’s “Deep Fake Love”.  https://www.nytimes.com/2023/07/21/podcasts/dario-amodei-ceo-of-anthropic-on-the-paradoxes-of-ai-safety-and-netflixs-deep-fake-love.html.
  * Russell (2011) Nestar John Charles Russell. 2011\.  Milgram’s obedience to authority experiments: Origins and early evolution.  _British Journal of Social Psychology_ 50, 1 (2011), 140–162.
  * Saha et al. (2020) Debjani Saha, Candice Schumann, Duncan Mcelfresh, John Dickerson, Michelle Mazurek, and Michael Tschantz. 2020.  Measuring non-expert comprehension of machine learning fairness metrics.  _International Conference on Machine Learning_ (2020), 8377–8387.
  * Saylor Academy (2012) Saylor Academy. 2012\.  _Principles of social psychology_.  Saylor Academy.
  * Schuler and Namioka (1993) Douglas Schuler and Aki Namioka. 1993.  _Participatory design: Principles and practices_.  Lawrence Erlbaum Associates, Inc.
  * Semuels (2018) Alana Semuels. 2018\.  The internet is enabling a new kind of poorly paid hell.  _The Atlantic_ 23 (2018).
  * Shatz (2017) Itamar Shatz. 2017\.  Fast, free, and targeted: Reddit as a source for recruiting participants online.  _Social Science Computer Review_ 35, 4 (2017), 537–549.
  * Shaw (2016) David Shaw. 2016\.  Facebook’s flawed emotion experiment: Antisocial research on social network users.  _Research Ethics_ 12, 1 (2016), 29–34.
  * Shih et al. (2020) Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, and Dorsa Sadigh. 2020.  On the critical role of conventions in adaptive human-AI collaboration.  _International Conference on Learning Representations_ (2020).
  * Shmueli et al. (2021) Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021\.  Beyond fair pay: Ethical implications of NLP crowdsourcing.  _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ (2021), 3758–3769.
  * Simonsen and Robertson (2013) Jesper Simonsen and Toni Robertson. 2013.  _Handbook of participatory design_. Vol. 711.  Routledge.
  * Sloane et al. (2022) Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2022.  Participation is not a design fix for machine learning.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–6.
  * Smith and Freeman (2021) Jason Smith and Jason Freeman. 2021.  Effects of deep neural networks on the perceived creative autonomy of a generative musical system.  _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ 17, 1 (2021), 91–98.
  * Snow et al. (2008) Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y. Ng. 2008.  Cheap and fast–but is it good? Evaluating non-expert annotations for natural language tasks.  _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ (2008), 254–263.
  * Strouse et al. (2021) DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, and Richard Everett. 2021.  Collaborating with humans without human data.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Sutton (2019) Richard Sutton. 2019\.  The bitter lesson.  _Incomplete Ideas_ 13, 1 (2019).
  * Tomasev et al. (2020) Nenad Tomasev, Julien Cornebise, Frank Hutter, Shakir Mohamed, Angela Picciariello, Bec Connelly, Danielle C. M. Belgrave, Daphne Ezer, Fanny Cachat van der Haert, Frank Mugisha, et al. 2020\.  AI for social good: Unlocking the opportunity for positive impact.  _Nature Communications_ 11, 1 (2020), 2468.
  * Tomasev et al. (2021) Nenad Tomasev, Kevin R. McKee, Jackie Kay, and Shakir Mohamed. 2021.  Fairness for unobserved characteristics: Insights from technological impacts on queer communities.  _AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society_ (2021).
  * Urban and Gates (2021) Christopher J. Urban and Kathleen M. Gates. 2021.  Deep learning: A primer for psychologists.  _Psychological Methods_ (2021).
  * Vines et al. (2013) John Vines, Rachel Clarke, Peter Wright, John McCarthy, and Patrick Olivier. 2013.  Configuring participation: On how we involve people in design.  _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (2013), 429–438.
  * Visser et al. (2000) Penny S. Visser, Jon A. Krosnick, and Paul J. Lavrakas. 2000\.  _Survey research_.  Cambridge University Press.
  * Vollmann and Winau (1996) Jochen Vollmann and Rolf Winau. 1996.  Informed consent in human experimentation before the Nuremberg code.  _British Medical Journal_ 313, 7070 (1996), 1445–1447.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022\.  Taxonomy of risks posed by language models. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 214–229.
  * White (2007) Ronald F. White. 2007\.  Institutional review board mission creep: The Common Rule, social science, and the nanny state.  _The Independent Review_ 11, 4 (2007), 547–564.
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  Re-examining whether, why, and how human-AI interaction is uniquely difficult to design.  _Proceedings of the 2020 CHI conference on Human Factors in Computing Systems_ (2020), 1–13.
  * Zaremba et al. (2023) Wojciech Zaremba, Arka Dhar, Lama Ahmad, Tyna Eloundou, Shibani Santurkar, Sandhini Agarwal, and Jade Leung. 2023.  Democratic inputs to AI.  https://web.archive.org/web/20230525194339/https://openai.com/blog/democratic-inputs-to-ai.  Accessed: 2023-09-14.

Figure 1. The automated annotation process applied to each paper. Boxes with
rounded corners depict steps that do not include the instruction-tuned large
language model (LLM). Boxes with sharp corners represent steps that produce
annotations by applying custom prompts to the LLM. See Section A.1 for the
exact text used in prompts.

##  Appendix A Evaluating current practices for human-participants research

Evaluating scientific publications at popular AI research venues offers an
understanding of current practices in work with human participants. The
present annotation analysis examined proceedings from the 2021, 2022, and 2023
AAAI conferences and the 2021 and 2022 NeurIPS conferences. The number of
research papers in each of these proceedings is quite high (e.g., 1319 and
2671 papers at the 2022 AAAI and NeurIPS conferences, respectively). The
papers are long-form (typically 6–8 pages of main text), with no guarantees
that the involvement of human participants will be mentioned in the abstract.

Given these difficulties, Hawkins and Mittelstadt (2023) applied an automated
keyword analysis to annotate practices around human data collection at NeurIPS
and other AI research venues. A keyword-based approach can efficiently process
and annotate large number of papers, but is relatively inflexible: the initial
selection of keywords may lead to large gaps in coverage (i.e., false
negatives).

To attempt to iterate on this approach, the current evaluation automates the
annotation process with an instruction-tuned large language model (LLM) and a
chained annotation process (cf. (Wei et al., 2022)). The evaluation pipeline
iterated through each paper in the proceedings for the specified conferences
(Figure 1).

For each paper, the annotation process first extracted text from the paper
using PDFium (PDFium authors, 2023). The next step partitioned the text into
sections of 1000 characters (roughly 250 tokens for the LLM). Each section
then passed through a conditional chain of prompts to check for any
indications of human participation. The assessments of each section were
combined and fed back into the LLM for an overall evaluation for the paper. If
this overall evaluation indicated that the paper involved human participants,
the annotation process subsequently applied a secondary set of prompts to
assess the reporting of independent ethical review, collection of informed
consent, and participant compensation.

This evaluation processed 9556 papers from the 2021, 2022, and 2023 AAAI and
2021 and 2022 NeurIPS conferences, with topline results reported in Table 1.
Spot checks of the model explanations identified and removed a small number of
false positives before the final tabulation.

In addition to the results reported in Table 1, this investigation produced
several new insights. Overall, the automated annotation process estimates a
rate of human data collection that is somewhat higher than appraised by
Hawkins and Mittelstadt (2023). This increased estimate may stem from several
different causes. First, the evaluation revealed several terms indicative of
human participation beyond those identified by Hawkins and Mittelstadt (2023),
including “volunteer” and “user study”. Second, the process of extracting text
from conference papers is error-prone, even when focusing on just the original
set of key terms. The LLM was generally robust to these errors, whereas
keyword matching can be disrupted when the text-extraction process adds an
incorrect character or misplaces a space. Third, though abstracts often
communicated whether research included human participants, a substantial
number of papers indicated the involvement of human participants through only
a paragraph (or sometimes just a single sentence) in the methods or results
sections. (This third pattern does not explain the difference in assessments
between keyword search and language model-based annotation, but is nonetheless
relevant for future paper reviews.)

Overall, this exploratory evaluation indicates that the recruitment of human
participants for AI research is more prevalent than previously estimated, and
confirms that ethical practices are not widespread in AI and ML research,
despite the community’s proximity to the fields of psychology and HCI.

###  A.1. Prompts

As summarized in Figure 1, the annotation process applied an LLM and a series
of prompts to evaluate the involvement of human participants in research
papers. The following subsections lay out the prompts used in this process,
with ellipses indicating a formatted input (e.g., text from the paper or from
previous annotations). Because LLMs can generate complex text in response to
these prompts, this process produces a number of auxiliary outputs in the form
of the model explanations. These explanations can facilitate the process of
quality review.

### “Involves participants?” prompt

Question: The following text is a section of a research paper. Does this
section indicate that the research included data collection from human
participants, annotators, user studies, etc.? This is often, though not
always, revealed through key terms like “participant”, “annotator”,
“labeller”, “user study”, or recruitment sources like “Mechanical Turk”.
Research involving “subjective evaluation” or “perceived” metrics may also
rely on human participants to produce these measurements.

Text: “…”

Reminder: Start your answer with “this section does not involve participants,
annotators, etc.”, or with “this section involves” followed by an explanation
of exactly how you know the section involves human participants in less than
20 words. Ignore reference sections.

### “Involves new data?” prompt

Question: The following text is a section of a research paper. Does this
section indicate that the research included original data collection from
human participants, annotators, user studies, etc., or just the re-use of
human data collected from other sources? Only say that this section includes
original data collection if you are absolutely sure the data is original
rather than re-used: do not include instances where other papers or citations
collect the human data, or where the section merely mentions a new dataset or
new annotations.

Text: “…”

Previous notes: “…”

Reminder: Start your answer with “this section describes the re-use of data”,
or with “this section describes” followed by an explanation of exactly how you
know the section involves original data collection in less than 20 words.

### “Not just human-related topic?” prompt

Question: The following text is a section of a research paper. Does this
section indicate that the research included the active involvement of human
participants in data collection, or does it just discuss human-related topics?

Text: “…”

Previous notes: “…”

Reminder: Start your answer with “this section just discusses human-related
topics”, or with “this section describes” followed by an explanation of
exactly how you know the section involves the active involvement of human
participants in less than 20 words.

### “Draft summary” prompt

Question: The following notes describe a section of a research paper. Please
combine the notes to answer the question: Does this section indicate that the
research included data collection from human participants, annotators, user
studies, etc.? Make sure to preserve details about whether the data collected
is original or re-used, as well as whether the section actively involves human
participants, annotators, user studies, etc. or just discusses human-related
topics.

First note: “…”

Second note: “…”

Third note: “…”

Reminder: Do not make up new information.

### “Finalize summary” prompt

Question: Given the following text, is this statement accurate or inaccurate?
Make sure to preserve details about whether the data collected is original or
re-used, as well as whether the section actively involves human participants,
annotators, user studies, etc., or just discusses human-related topics.

Text: “…”

Statement: “…”

Reminder: Start your answer with “Yes, this section describes” and then
explain in less than 20 words, or state “this section does not involve
participants, annotators, etc.”

### “Overall evaluation” prompt

Question: The following statements are all judgments of sections from a
research paper. Each statement assesses a different section of the paper,
indicating whether the paper includes work with or new data collection from
human participants, annotators, labelers, user studies, etc. Based on these
statements, do you think the paper actually includes new data collection from
human participants, annotators, labelers, user studies, etc.? Ignore the re-
use of human data collected from other sources and sections that just discuss
human-related topics.

Statements: […]

Options: [“yes”, “no”]

Reminder: Ignore datasets that do not involve human participants, annotators,
etc., even if they are new datasets. Answer only “yes” or “no”, with no
additional text or explanation.

##  Appendix B Placing research ethics for human participants in historical
context

What historical moments contributed to the particular ethical concerns
highlighted by the modern research ethics movement? The events described in
this appendix raised new, provocative concerns for scientists, in some cases
prompting the formalization of ethical frameworks to reorient power relations
between researchers and participants. These provocations—around informed
consent, participant well-being, and related values—foreshadow many of the
principles and guidelines outlined in this paper.

###  B.1. Early incidents in biomedical research

Two major incidents in the 20th century jump-started the modern research
ethics movement: Nazi medical experiments during the Holocaust and the US
Public Health Service Syphilis Study at Tuskegee. Revelations concerning these
biomedical research projects raised new provocations around participant well-
being and informed consent.

During World War Two, Nazi physicians forced large numbers of prisoners into
medical experiments (Gillespie, 1989). The procedures involved in the
experiments were dangerous, abusive, and painful—causing extraordinary rates
of death, disability, and disfiguration among the many victims. After the war
ended, US military tribunals judged the physicians involved in the experiments
at the war crime trials held in Nuremberg. The so-called doctors’ trial
resulted in prison and death sentences for the majority of the physicians,
though some were acquitted. The tribunals subsequently produced what is now
known as the Nuremberg Code: a set of 10 points concerning ethical research
conduct (Nuremberg Military Tribunals, 1947). Given the atrocities committed
in the Nazi medical experiments, the Code argued for multiple standards
protecting participant well-being, including the “avoidance of unnecessary
pain and suffering” and the “belief that […] experimentation will not end in
death or disability.” The Nuremberg Code also enshrined a strong conviction
that “the voluntary consent of the human subject is absolutely essential.”
Though the concept of informed consent predates World War Two, the Nuremberg
trials catalyzed the modern stance that informed consent should be the primary
basis for ethical research with human participants (Vollmann and Winau, 1996).

Figure 2. The letter from the Center for Disease Control ending the US Public
Health Service study at Tuskegee. Source: US National Archives.

In 1932, the US Public Health Service recruited several hundred Black American
men from disadvantaged communities in Tuskegee for biomedical research on
syphilis (Brandt, 1978). Officials promised the men free medical care for
their involvement in the study, but did not inform them of their syphilis
diagnoses. The study lasted four decades; throughout this time, the study
researchers did not provide treatment to any of the infected men, even at the
point penicillin was established as an effective (and readily available)
treatment. The study ended in 1972 (see Figure 2) after a social worker leaked
details of the affair to the press (Gamble, 1993). As with the experiments
conducted during World War Two, revelation of the treatment of the men at
Tuskegee prompted public outcry. Given the denial of available treatment to
patients, the US Public Health Service study re-emphasized the importance of
prioritizing participant well-being and the avoidance of unnecessary risks.
The investigators’ exploitation of a disadvantaged community based on race was
particularly troubling. As Brandt (1978) lambasted, the study “revealed more
about the pathology of racism than the pathology of syphilis.” In 1974, as a
result of the ethical lapses reflected in the investigators’ actions, the US
Congress established the National Commission for the Protection of Human
Subjects. Building on the framework laid by the Nuremberg Code, the Committee
went on to issue a number of reports and recommendations to establish stronger
standards for research ethics with human participants, most notably including
the Belmont Report.

###  B.2. Mid-century dilemmas in psychology research

Following the revelations within the Nuremberg trials, social science found
itself profoundly affected by the atrocities committed in the Holocaust. Many
scientists took up new research programs to better understand how such events
could come to pass. Stanley Milgram, an American psychologist, was one such
researcher. Milgram ruminated on the widespread belief that “associated the
occurrence of the Holocaust with blind obedience to authority” and stereotyped
Germans as particularly conformist or obedient (Russell, 2011). To evaluate
this theory empirically, Milgram devised an experiment to measure an
individual’s inclination to obey authority. In his experiment, an investigator
told participants that they had been randomly assigned a “teacher” role and
needed to administer questions to another participant, the “learner.” Every
time the learner answered a question incorrectly, the learner was to give them
an electric shock of gradually increasing voltage. Labels on the shock machine
described the shocks, starting with “Slight shock” and ending with “XXX” (see
Figure 3). In actuality, the learner was an actor hired by Milgram and
received no actual shocks. Nonetheless, his reactions were convincing. The
investigator would respond to any participant inquiry or resistance by
stating: “I’m responsible for anything that happens to [the learner]. Continue
please.” Against his expectations, Milgram found that the majority of his
participants would proceed all the way to administering the maximum shock.
After conducting a series of experimental variations, Milgram published his
results in three journal papers and most notably in his 1974 book Obedience to
Authority (Milgram, 1974).

Figure 3. The “shock machine” used in Milgram’s experiments. Source: Saylor
Academy (2012).

Though initially venerated in psychology, recent perspectives raise a number
of questions about the ethics of Milgram’s experiments. The experiments
purposefully led participants to believe that they were causing pain and
suffering. As Milgram’s own notes admit, participants “indicated that it
caused them discomfort to watch the victim in agony” (Russell, 2011). Others
“recalled the anxiety and distress they felt over an extended period at having
to live with the knowledge that they may have killed someone” (Brannigan et
al., 2015). How justifiable was it for the study to cause this level of
distress in participants? Milgram’s approach to deception compounds the issue.
Milgram repeatedly insisted that he debriefed all participants in the
experiments. However, subsequent archival review of the study’s records
clearly demonstrate that Milgram did not debrief all of his participants and
inform them that they had not harmed the actor, at least in part to ensure
knowledge of the deception did not bias subsequent participants (Nicholson,
2011).555This conduct is particularly surprising given Milgram’s disavowal of
deception during his graduate studies, writing that “deception for any
purposes [in psychology] is unethical and tends to weaken the fabric of
confidence so desirable in human relations” (Brannigan et al., 2015).

###  B.3. New questions around industry and oversight in the 21st century

In 2014, a team of researchers from Facebook and Cornell University published
an experimental paper entitled “Experimental evidence of massive-scale
emotional contagion through social networks” in the Proceedings of the
National Academy of Sciences (Kramer et al., 2014). In the paper, the
researchers described controlled interventions to shift the emotional valence
of Facebook users’ newsfeeds. The study aimed to understand whether emotional
contagion occurs in online social networks. Soon after publication, the paper
prompted considerable public criticism. Attention primarily coalesced around
whether the users involved in the study had provided informed consent. The
research team held that users’ agreement to Facebook’s data use policy
constituted informed consent, as one term in the policy referred to the use of
user information “for internal operations, including troubleshooting, data
analysis, testing, research and service improvement” (Shaw, 2016). Critics
argued that this approach diverged from ethical standards in academic
research, which emphasizes transparent and comprehensible communication around
study participation (Flick, 2016). Overall, the discussion drew attention to
tensions at the intersection of psychology, research oversight, and industry
versus academic standards for research ethics.

###  B.4. Regulatory artifacts and current trends

Together, these historical moments produced a series of advisory and
regulatory artifacts that continue to steer modern research ethics. The
Belmont Report (Figure 4) and the Common Rule, both developed in the United
States, exert a particularly strong influence on current thought and
practices—in the United States and beyond.666See also Holm (2020), who argues
that the Belmont Report exerted a clear but indirect influence on the
development of research ethics in Europe, but notes that European behavioral
researchers were largely able to resist the formal establishment of regulatory
frameworks, diverging from American behavioral scientists. The framework
introduced in the Belmont Report acted as the basis of the 1991 Federal Policy
for the Protection of Human Subjects, more commonly known as the “Common
Rule.” The three core ethical principles identified by the Belmont
Report—respect for persons, beneficence, and justice—continue to organize
federal regulation, persisting through the 2018 revision of the Common Rule to
address new technologies and concerns. Formally, the Common Rule pertains only
to research funded by the United States government. In practice, virtually all
US academic institutions apply the Common Rule to research projects with human
participants, irrespective of funding source. Similar principles and guidance
appears in codes of ethical conduct in many other countries (Shmueli et al.,
2021).

Figure 4. The cover of the Belmont Report. Source: The Internet Archive.

##  Appendix C Defining the scope of research participation in AI research

Humans enter many points of the process of designing, developing, and
deploying AI systems. Research participants represent a particular subset of
these entry points—that is, a particular subset of humans involved in research
studies. Within the United States, the Department of Health and Human Services
(HHS) drives standards and norms for research involving human participants,
largely as laid out in the Federal Policy for the Protection of Human Subjects
(the “Common Rule”). These standards exert broad influence in the global
setting; national regulations in many other countries are modeled after the
Common Rule (Capron, 2008). Understanding the scope of AI research with human
participants, then, can begin with the definitions offered by the Common Rule.

The Common Rule defines the scope of “research” as “a systematic
investigation, including research development, testing, and evaluation,
designed to develop or contribute to generalizable knowledge” (45 C.F.R. §
46.102). Based on this definition, the Common Rule specifies that a “human
subject” is:

> “a living individual about whom an investigator (whether professional or
> student) conducting research: (i) Obtains information or biospecimens
> through intervention or interaction with the individual, and uses, studies,
> or analyzes the information or biospecimens; or (ii) Obtains, uses, studies,
> analyzes, or generates identifiable private information or identifiable
> biospecimens.” (45 C.F.R. § 46.102)

As described in Section 2, this paper adopts an adapted version of the Common
Rule definition to scope human participation for AI research. Primarily, the
adaptation shifts away from “biospecimens” and focuses on the general category
of “information”:

> Participants are living individuals who provide or contribute information to
> researchers, particularly in systematic investigations designed to develop
> or contribute to generalizable knowledge.

Teams at technology companies undertake a large proportion of contemporary AI
research (Birhane et al., 2022b; Luitse and Denkena, 2021; Martínez-Plumed et
al., 2021). The scoping of research to investigations intended to “contribute
to generalizable knowledge” is sometimes used to excuse industry researchers
from research ethics frameworks (Jackman and Kanerva, 2015; Meyer, 2020),
since they do not always intend technology and knowledge production for
scientific dissemination (Evans, 2010). This “non-generalizable” defense
typically casts industry projects as aiming to improve products (e.g., A/B
testing) rather than for general knowledge, thus exempting them from research
ethics frameworks. Two general patterns challenge this exclusion. First, there
is growing recognition among ethicists that “generalization” is a limited
diagnostic test to identify research. That is, generalization to broader
reference groups is not the only objective in psychology and behavioral
research. For example, research may also strive to generate a deeper
understanding of the processes underlying a phenomenon or to test the
sufficiency of certain conditions in bringing about an effect (Mook, 1983).
Second, many of the technology companies and commercial organizations involved
in AI research espouse goals that are more general than simply improving
products, such as creating safe AI systems that benefit humanity (Google
DeepMind, nda; OpenAI, nd). In this sense, research conducted with human
participants at these companies indeed aims to develop or contribute to
generalizable knowledge. Thus, in several respects, the AI work undertaken in
industry merits scrutiny within research-ethics frameworks.

On a qualitative level, research typically follows several procedural and
methodological patterns. A common characteristic of research with human
participants is submission and publication at peer-reviewed, scholarly venues,
though releases in pre-prints and technical blogs are increasingly common. AI
studies involving human participants also tend to follow established research
designs, incorporating “traditional” psychology and human-computer interaction
methods.

Many projects conduct behavioral, cognitive, or psychological experiments
(Campbell and Stanley, 1963; Fisher, 1935). The hallmark of experimental
studies is the controlled manipulation of a variable of interest—the
independent variable. Experiments with human participants expose participants
to particular values of the independent variable and measure changes in one or
more dependent variables. Researchers design the experiment to minimize the
influence all other environmental variables, typically by holding each at a
constant value across all participants. For example, Strouse et al. (2021)
recruited a sample of $N=114$ online participants for an experiment on human-
agent collaboration. Their experiment randomly paired participants with
various agents to test the hypothesis that certain training algorithms
supported better human-agent collaboration.

Questionnaires (Paulhus et al., 2007) and surveys (Visser et al., 2000) are
also common. Questionnaires often feature in observational studies, though
they can also be integrated within experimental studies. For example, Smith
and Freeman (2021) investigated perceptions of autonomy and creativity in the
context of generative music systems with a sample of $N=5$. They selected
their participants from a pool of Georgia Tech Music Technology graduate
students. The primary task involved participants rehearsing a musical
performance with neural network-based generative music systems. Participants
answered questionnaires both before and after the music-performance task,
allowing the researchers to understand participants’ background with machine
learning concepts, as well as their reactions to the particular generative
systems in the study.

AI research with human participants may also draw on techniques such as semi-
structured interviews (Fontana and Frey, 2000) and focus groups (Parker and
Tritter, 2006). Gero et al. (2020) provide an example in this space,
conducting semi-structured interviews to understand people’s mental models of
AI agents. In one study, they recruited $N=11$ participants from within a
technology company. Participants first played a game with a cooperative AI
agent, then completed interviews that explored their perceptions and
understanding of the agent.

##  Appendix D A note on terminology

Scientists have long debated how to refer to individuals participating in
research (Bibace et al., 2009), including the choice of “subjects” or
“participants.” US federal regulation, including the Common Rule, adopts the
term “subjects” (45 C.F.R. § 46.102). As a result, “human subjects”
terminology is frequently seen in areas governed or influenced by the Common
Rule—including the official guidance offered at many academic institutions.

Some scholars and stakeholders argue that “subjects” terminology carries
negative implications concerning individual agency and autonomy. Using
“participants” is thought to more directly recognize that people should
participate in—not be subjected to—research. For example, the British
Psychological Society advocates for the use of “participant” over “subject”,
as the former better “serves to acknowledge the autonomy and agency of the
individual in contributing to the research” (Oates et al., 2021). The American
Psychological Association maintains a slightly more circumspect position, but
advises the use of language that acknowledges the “contributions and agency”
of research participants (American Psychological Association, 2019). These
tensions emerge in other linguistic norms, such as the passive language with
which the Common Rule discusses participants, defining them as “individuals
about whom an investigator […] obtains information” (rather than “individuals
who contribute information” or “who produce information then collected by
investigators”, for instance). This approach reflects an atavistic connection
to historical perspectives in psychology: both Wilhelm Wundt and Franz
Brentano viewed their study participants as social equals who actively
provided researchers “the privilege of access into their innermost feelings
and thoughts” (Bibace et al., 2009).

Recognizing that the language of research itself can reinforce harmful power
structures (Morrison, 1994), this paper adopts “participant” terminology
except in cases where directly quoting a source’s use of “subject” or
“subjects.”