# A Systematic Literature Review of Human-Centered, Ethical, and Responsible
AI

Mohammad Tahaei  0000-0001-9666-2663 Nokia Bell Labs21 J.J. Thomson AvenueCB3
0FACambridgeUK<EMAIL_ADDRESS>,  Marios Constantinides
0000-0003-1454-0641 Nokia Bell LabsCambridgeUK marios.constantinides@nokia-
bell-labs.com ,  Daniele Quercia  0000-0001-9461-5804 Nokia Bell
LabsCambridgeUK<EMAIL_ADDRESS>and  Michael Muller
0000-0001-7860-163X IBM Research AICambridgeMAUSA<EMAIL_ADDRESS>
(2023)

###### Abstract.

As Artificial Intelligence (AI) continues to advance rapidly, it becomes
increasingly important to consider AI’s ethical and societal implications. In
this paper, we present a bottom-up mapping of the current state of research at
the intersection of Human-Centered AI, Ethical, and Responsible AI (HCER-AI)
by thematically reviewing and analyzing 164 research papers from leading
conferences in ethical, social, and human factors of AI: _AIES_ , _CHI_ ,
_CSCW_ , and _FAccT_. The ongoing research in HCER-AI places emphasis on
governance, fairness, and explainability. These conferences, however,
concentrate on specific themes rather than encompassing all aspects. While
AIES has fewer papers on HCER-AI, it emphasizes governance and rarely
publishes papers about privacy, security, and human flourishing. FAccT
publishes more on governance and lacks papers on privacy, security, and human
flourishing. CHI and CSCW, as more established conferences, have a broader
research portfolio. We find that the current emphasis on governance and
fairness in AI research may not adequately address the potential unforeseen
and unknown implications of AI. Therefore, we recommend that future research
should expand its scope and diversify resources to prepare for these potential
consequences. This could involve exploring additional areas such as privacy,
security, human flourishing, and explainability.

human-centered AI, ethical AI, responsible AI, systematic literature review

††copyright: rightsretained††booktitle: ACM Comput. Surv.††journalyear:
2023††ccs: Social and professional topics††ccs: Human-centered computing††ccs:
Theory of computation††ccs: Information systems††ccs: Software and its
engineering††ccs: Security and privacy

##  1\. Introduction

Artificial Intelligence (AI) is a rapidly growing field with tremendous
potential to transform our lives on an unprecedented scale through a new
technological revolution (Kissinger et al., 2021). It can advance the global
economy and contribute to human flourishing (Laker, 2022; Lee, 2022). However,
like every significant technological revolution, ethical and societal risks
such as invasion of privacy or identity theft in social media (Geoffrey A.
Fowler, 2021; Associated Press, 2021) are taking center stage. Risks
associated with AI are no exception. If left unattended, AI can negatively
impact specific populations, perpetuate historical injustices, or, even worse,
amplify them (Baeza-Yates, 2018; Leonardo Nicoletti and Dina Bass, 2023).
These risks include misclassifying people of color (Buolamwini and Gebru,
2018; Leonardo Nicoletti and Dina Bass, 2023); exacerbating economic
disadvantage by denying bank loans through digital redlining (Hertzberg et
al., 2010; Kizilaslan and Lookman, 2017); reducing medical care for
economically disadvantaged people based on their prior medical spending
(Arthur et al., 2008; Jones et al., 2020; Wilkinson et al., 2020); denying
bail to Queer, Trans, Black, Indigenous, or People of Color (QTBIPOC)(Silva
and Kenney, 2018); imposing longer prison sentences on QTBIPOC people(Lyn,
2020); and removing children from QTBIPOC families into foster care (Leckning
et al., 2021; Saxena et al., 2020). In addition to the considerable attention
given to biases within data sources, recent work examines human “wrangling”
activities necessary to make so-called “raw” data fit-for-purpose. However,
these changes to datasets and models during AI development introduce biases as
a consequence (Muller and Strohmayer, 2022; Pine and Liboiron, 2015; Feinberg,
2017; Passi and Jackson, 2017; Mentis et al., 2016). These examples remind us
that people build AI systems and that social biases and prejudices may be
included—intentionally or inadvertently (Aragon et al., 2022; Muller and
Strohmayer, 2022).

The scientific community has recently made attempts to address these pressing
issues. Many research conferences, journals, and groups have begun to argue
for responsible AI development, bringing to the table issues including but not
limited to fairness, explainability, and privacy in AI (Google, 2022;
Microsoft, 2022; PwC, 2022; Nokia Bell Labs, 2022; Barredo Arrieta et al.,
2020; The Organisation for Economic Co-operation and Development (2019), OECD;
National Institute of Standards and Technology, 2023), and centering AI around
humans (Aragon et al., 2022; Chancellor, 2023; Ehsan et al., 2021a). These
topics are discussed in research conferences with a long history of advocating
for human-centered design, such as the Conference on Human Factors in
Computing Systems (CHI) and the Conference on Computer-Supported Cooperative
Work and Social Computing (CSCW), and in newer conferences like the Conference
on AI, Ethics, and Society (AIES), and the Conference on Fairness,
Accountability, and Transparency (FAccT), which were established to cover
topics related to the ethics of AI. In response to the large and growing
interest in AI, there have been several survey and review papers mapping out
the AI research landscape (Capel and Brereton, 2023; Wong et al., 2023b), such
as those surveying responsible development of AI in healthcare (Siala and
Wang, 2022), biases and fairness in AI (Sun et al., 2019; Pessach and Shmueli,
2022; Orphanou et al., 2022), and explainable AI (Abdul et al., 2018; Bertrand
et al., 2022; Barredo Arrieta et al., 2020; Angelov et al., 2021). However, a
comprehensive overview at the intersection of _human-centered AI_ , _ethical
AI_ , and _responsible AI_ is lacking. The intersection of Human-Centered,
Ethical, and Responsible AI (HCER-AI) is significant because it presents a
complex challenge of addressing multiple aspects simultaneously (see Section 3
for how we define these terms). Each aspect—human-centeredness, ethical
considerations, and responsible AI—cannot be achieved in isolation; they are
interdependent. This interdependence underscores the crucial need to
thoroughly understand and explore this intersection to create AI systems that
are comprehensive and aligned with societal values. We aim to fill this gap by
surveying the current state of research in HCER-AI in four leading research
conferences on these topics. We formulated four Research Questions (RQs):

  1. RQ1:

What is the state of research in HCER-AI in the four research conferences that
cover topics related to HCER-AI?

  2. RQ2:

What research methods are used in HCER-AI studies?

  3. RQ3:

What are the research gaps in HCER-AI?

To further assess the current state of research in HCER-AI and compare it with
potential future implementations, as indicated by patents, we conclude with
the following question:

  1. RQ4:

What is the landscape of patent applications in HCER-AI?

To answer our RQs, we reviewed and thematically analyzed 164 research papers
from the proceedings of AIES, CHI, CSCW, and FAccT related to HCER-AI (Section
3). We found that the landscape of HCER-AI covers six primary themes:
_governance_ , _fairness_ , _explainability_ , _human flourishing_ , _privacy_
, and _security_ , with a heavy emphasis on the first three themes (Section
5). These results echo prior empirical studies with different groups of
people, highlighting a gap between public and AI research concerns (Jakesch et
al., 2022). For instance, while there is a critical mass of research on
fairness, average users may care more about their safety and privacy (Jakesch
et al., 2022). We also found that conferences have distinct areas of focus
within HCER-AI. For example, AIES and FAccT publish more papers on governance
and fairness, while CHI and CSCW cover a broader range of topics, including
explainability and human flourishing. Our results suggest that future research
should take a broader view of AI and diversify resources beyond governance and
fairness to prepare for AI’s unexpected and unknown ramifications.

As part of our posteriori analysis, we used AI-assisted summarization to
answer our RQs using the abstracts of the papers as input to ChatGPT 4.0,
following the recent emerging literature on using AI for qualitative analysis
(Byun et al., 2023; Abram et al., 2020). We compared its findings with our
manual analysis (Section 8). Although ChatGPT provided an overview of the
dataset, it is important to highlight that its classification was not
specifically tailored to the research areas currently under investigation.
Instead, it provided a broad classification intended for general public
understanding, rather than specialized research purposes. In the future, AI-
powered research tools like ChatGPT could be a helpful supplementary resource
for gaining new insights. Further examination is needed to determine how AI
can be effectively utilized in literature reviews.

Materials concerning our classification and analysis will be available on our
project’s website, which can be found at social-dynamics.net/RAI-Review.

##  2\. Related Work

To situate our review within the broader human-centered, ethical, and
responsible AI literature, we next discuss previous surveys that focused on
human-centered AI, specific concerns with AI such as fairness and
explainability, and surveys dedicated to specific domains of AI.

Developing AI involves socio-cultural and technical factors, with scholars
arguing for developing human-centered AI in the past few years (Aragon et al.,
2022; Shneiderman, 2022; Bingley et al., 2023; Ehsan et al., 2021b). In
particular, there is an emphasis on the challenges of AI integration into
socio-technical processes to preserve human autonomy and control, as well as
the impacts of AI deployment and applications on society, organizations, and
individuals (Boyarskaya et al., 2020; Whittlestone et al., 2019). Scholars
also argue that understanding socio-technical and environmental factors can
surface why and how an AI may become human-centered (Shneiderman, 2020; Ehsan
et al., 2021a; Liao and Varshney, 2021; Muller et al., 2022).

Despite its recent surge, bias in AI has been a longstanding topic of interest
among academic circles (Caton and Haas, 2020). Several surveys discussed data
and model biases across domains, including social data biases that stem from
user-generated context and behavioral traces (Olteanu et al., 2019), gender
bias in natural language processing (Sun et al., 2019), and fairness metrics
with potential mitigation strategies (Caton and Haas, 2020; Pessach and
Shmueli, 2022; Orphanou et al., 2022). Others investigated emerging research
trends in responsible AI, such as fair adversarial learning, fair word
embeddings, fair recommender systems, and fair visual description (Pessach and
Shmueli, 2022).

As AI comes with the promise of advancing our economy and augmenting our
lives, it becomes increasingly essential for people to understand AI’s
implications and remain in control (Barredo Arrieta et al., 2020). To this
end, much of the previous literature has been dedicated to discussing
explainability in AI. For example, a survey of 289 core papers on explanations
and explainable systems and more than 12,000 citing papers represents the
diverse research landscape of AI, such as algorithmic accountability,
interpretable machine learning, context awareness, cognitive psychology, and
software learnability (Abdul et al., 2018). More recently, Angelov et al.
(2021) provided an overview of AI explainability techniques considering recent
advancements in machine learning and deep learning. Methods included featured-
oriented ones such as SHapley Additive exPlanation (SHAP) (Lundberg and Lee,
2017), and surrogate models such as Local interpretable model-agnostic
explanations (LIME) (Dieber and Kirrane, 2020). Another line of work reviewed
specific domains (e.g., recommender systems, social networks, and healthcare).
These reviews cover notions of fairness specifically tailored for recommender
systems (Li et al., 2022), machine learning fairness in the domain of public
and population health (Mhasawade et al., 2021), and algorithmic fairness in
computational medicine (Xu et al., 2022).

The closest survey to ours is Capel and Brereton (2023)’s survey, published at
CHI 2023, where the authors focus on the “human-centered AI” as the primary
search term. The landscape mapping resulted in four major themes: explainable
and interpretable AI, human-centered approaches for designing and evaluating
AI, human-AI teaming, and ethical AI. Our survey extends this survey beyond
human-centered AI by adding two keywords (ethical AI and responsible AI).
Adding these keywords has significance because it presents the complex
challenge of addressing multiple aspects simultaneously. Each aspect—human-
centeredness, ethical considerations, and responsible AI—cannot be achieved in
isolation; they are interdependent. This interdependence underscores the
crucial need to thoroughly comprehend and explore this intersection in order
to create AI systems that are comprehensive and aligned with societal values.
Our work also shows the differences and similarities between the research
communities, which are not covered in the prior work.

While previous surveys mapped out parts of research in human-centered AI,
ethical AI, and responsible AI, these mappings do not fully cover the
intersection of the three, underscoring the need to explore it systematically.
We also provide a lens into different research practices across the research
communities as a new angle to the prior surveys. Therefore, this review aims
to fill this gap and contribute to the ongoing discourse about HCER-AI.

##  3\. Method

To answer our RQs (§1), we surveyed the HCER-AI literature in AIES, CHI, CSCW,
and FAccT following the PRISMA 2020 statement for conducting systematic
literature reviews (Page et al., 2021). Figure 1 shows the flowchart of our
method.

Figure 1. An overview of our research method. We started with 228 records from
the ACM Digital Library. After assessing quality and eligibility based on our
inclusion criteria, we included and analyzed 164 research papers.

###  3.1. Positionality Statement

Understanding researcher positionality is essential for demystifying our lens
on data collection and analysis (Frluckaj et al., 2022; Havens et al., 2020).
We situate this paper in two Western countries in the 21st century, writing as
authors primarily working as academic and industry researchers. We identify as
males from the Middle East, Southern Europe, and North America with diverse
ethnic and religious backgrounds. Our shared research backgrounds include
human-centered computing, privacy, security, software engineering, AI, social
and ubiquitous computing, urbanism, and conducting systematic literature
reviews. While we aimed at a bottom-up literature analysis, our results are
constructs of our expertise and understanding of the landscape, as well as our
positionality and cultural background. At least two authors were involved in
all research steps to diversify the biases that attend any humans’ judgments
in analyzing the papers and reporting the results.

###  3.2. Eligibility and Inclusion Criteria

For this systematic review and our scoping, papers qualified for the analysis
if their contributions pertained to three main areas: _human-centered AI_ ,
_responsible AI_ , and _ethical AI_. By searching for keywords, we relied on
the paper authors’ use of terms related to our scoping. Our interpretation of
AI is based on the definition of the National Institute of Standards and
Technology (NIST), published in 2023: “as an engineered or machine-based
system that can, for a given set of objectives, generate outputs such as
predictions, recommendations, or decisions influencing real or virtual
environments. AI systems are designed to operate with varying levels of
autonomy.”(National Institute of Standards and Technology, 2023) In the next
section (Section 3.3), we describe our data collection method.

###  3.3. Data Collection

We decided to base our review on four research conferences: two well-
established venues with a wide net for human-computer interaction research and
two recently established venues with a narrow emphasis on AI and its
implications (excerpts taken from conferences’ websites):

  * •

AIES (established in 2018): “our goal is to encourage talented scholars in
these and related fields to submit their best work related to morality, law,
policy, psychology, the other social sciences, and AI.”

  * •

CHI (established in 1982): “annually brings together researchers and
practitioners from all over the world and from diverse cultures, backgrounds,
and positionalities, who have as an overarching goal to make the world a
better place with interactive digital technologies.”

  * •

CSCW (established in 1986): “is the premier international venue for research
in the design and use of technologies that affect groups, organizations,
communities, and networks.”

  * •

FAccT (established in 2019; previously known as FAT*): “a computer science
conference with a cross-disciplinary focus that brings together researchers
and practitioners interested in fairness, accountability, and transparency in
socio-technical systems.”

We collected 228 records from the ACM Digital Library from these four
conferences in May 2023. We queried for ‘‘human-centered AI’’ OR ‘‘human-
centered artificial intelligence’’ OR ‘‘ethical artificial intelligence’’ OR
‘‘ethical AI’’ OR ‘‘responsible AI’’ OR ‘‘responsible artificial
intelligence’’ within anywhere in the proceedings of AIES, CHI, CSCW, and
FAccT, including only full research papers using ACM’s filters (excluding
other materials such as abstracts, panels, and tutorials). We chose these
keywords because human-centered AI, ethical AI, responsible AI are widely used
in academia and industry; examples include prior literature (Capel and
Brereton, 2023; Shneiderman, 2022) and guidelines produced by large technology
companies (e.g., Google, Microsoft, and IBM use responsible AI to address
ethical concerns and risks with AI (Google, 2022; Microsoft, 2022; PwC, 2022;
Nokia Bell Labs, 2022; Barredo Arrieta et al., 2020; The Organisation for
Economic Co-operation and Development (2019), OECD; Tahaei et al., 2023)). We
define human-centered AI as a paradigm that advocates for including people’s
requirements and needs in designing, developing, and deploying AI systems.
Ethical AI and responsible AI refer to paradigms that emphasize the inclusion
of ethical values in the design, development, and deployment of AI, as well as
the responsibility to consider the risks and consequences on individuals and
societies. Our review aims to explore the intersection of these research
areas. We explore the topics related to the impact of AI to inform the
building of AI systems that prioritize the needs and requirements of people
and society while mitigating potential harms through a human-centered
approach.

The above process resulted in 190 full research papers that two authors read
to assess their relevance and eligibility (Section 3.2). They examined papers
discussing human-centered AI, ethical AI, and responsible AI. They excluded
papers that mention the keywords in the references or acknowledgments without
discussing them in the main text and papers that did not cover our topics of
interest (human-centered AI, ethical AI, and responsible AI). After discussing
and resolving disagreements, 26 of these papers were excluded. Therefore, our
review and the rest of this paper are based on a final set of 164 research
papers from AIES (n=30), CHI (n=68), CSCW (n=34), and FAccT (n=32).

We were also interested in discovering patents related to our keywords.
Patents can show the practical applications of an invention and how it
propagates in the industry and products (Cao et al., 2023). HCI research, in
particular, significantly impacts patents compared to other computer science
research areas (Cao et al., 2023). Therefore, we ran the same query we
executed on ACM on the United States Patent and Trademark Office search tool
(United States Patent and Trademark Office, 2023) and retrieved 67 results.
One author manually reviewed these patents and removed 30 because they were
unrelated and only appeared in the results due to a keyword (e.g., patents
with “responsible for AI” were also included). This process left us with 37
patents.

###  3.4. Data Analysis

Following a thematic analysis method (Braun and Clarke, 2006; Lazar et al.,
2017a), the first two authors split the papers between themselves. After
discussions, they decided to open code each paper with the research methods
used, primary contributions, human aspects, responsible aspects, and sample
description (any information about the population or dataset of the paper).
They used the paper authors’ words to code each paper (also known as “in vivo
coding” (Lazar et al., 2017b)). When in doubt, they flagged a paper for
further discussion (n=27). In a joint session, they used thematic analysis to
create themes from the codes and resolved disagreements through discussions.
They did not use a set of predefined themes but came up with the results using
a bottom-up approach. In our analysis, a paper may appear in multiple
codes—codes are not mutually exclusive. Sections 4 and 5 are derived from this
analysis. Although our findings are primarily based on this analysis and
review, as authors actively engaged in this field, we have also utilized other
papers and online resources to enhance our discussion beyond the reviewed
papers, especially when addressing future directions. For a comprehensive list
of the reviewed papers, refer to Table LABEL:tab:all-papers in the Appendix.

In a joint session, two authors read the title and abstract of the remaining
37 patents and built an affinity diagram. During this process, they removed
another 9 patents because they were unrelated and did not discuss our topics
of interest as their primary claims. The final set of patents totaled 28.
Section 6 is based on the findings of this analysis.

###  3.5. Limitations

We acknowledge that our literature search may not be comprehensive and
exhaustive. However, covering AIES, CHI, CSCW, and FAccT as prominent venues
for research in AI, Human-Computer Interaction (HCI), ethical, and responsible
AI gave us insights into the state of the art of HCER-AI. Future research may
also build on our work and expand it to other academic and industry
literature. We also used keywords instead of reviewing a list of papers and
deciding which paper to include or exclude based on our judgment (a typical
approach in running systematic literature reviews, for example, Holländer et
al. (2021) and Bergram et al. (2022)). Through this approach, we collected
papers related to HCER-AI from the authors’ perspective rather than using our
own judgment of whether a paper is related to HCER-AI. Future research may
take a representative sample of papers from research publications, conduct a
bottom-up analysis, and compare results with ours.

##  4\. Results: Research Methods of HCER-AI

In the 164 papers reviewed, 85 were qualitative (e.g., user studies for system
design and evaluation, interviews, and workshops), 25 were quantitative (e.g.,
survey and log analysis), 21 were theoretical (e.g., essays and framework
proposal), 20 were mixed-methods (a combination of qualitative and
quantitative), 8 were reviews (e.g., review of literature, policies, and
guidelines), and 5 were a combination of a review and theory (Figure 2). CHI
and CSCW are heavily focused on qualitative studies (as expected since they
are the premier venues for human-centered studies); however, they lack papers
derived from theoretical reviews. AIES and FAccT cover a broader range of
research methods with a preference for theoretical papers.

Figure 2. Count of HCER-AI papers per research method (n=164).

Papers did not have a consistent way of reporting demographics. The typical
demographics were gender, job title, age, and prior experience in a particular
topic of interest for the research (Table 1). Out of the papers that reported
the location of research or participants, the primary focus was on North
America (n=47), and many did not report demographics (n=28) (Table 2).

Table 1. Count of HCER-AI papers per demographic value (n=103, includes papers with some information about the participants). Prior experience is in a particular topic of interest for the research. Reported demographics with at least 3 papers in the respective demographic. Demographic | Count of HCER-AI papers
---|---
Gender | 59 (57%)
Age | 48 (47%)
Job title | 43 (42%)
Prior experience | 21 (20%)
Education | 16 (16%)
Ethnicity | 10 (10%)
Industry type | 6 (0.6%)
Organization type | 3 (0.3%)
Income | 3 (0.3%)
Table 2. Count of HCER-AI papers per continent (n=164). Not applicable includes theory and review papers. Continent | Count of HCER-AI papers
---|---
North America | 47 (29%)
Asia | 18 (11%)
Europe | 15 (9%)
Africa | 3 (2%)
Oceania | 1 (0%)
Not applicable | 69 (42%)
Not reported | 28 (17%)

#### Recommendations for Future Research

Future researchers should consider reporting participant demographics to help
with replicability and to understand how generalizable the results are. As
discussed, there is a lack of consistency when reporting demographics. The
community’s own findings emphasize that building an AI that respects its users
is highly dependent on the target population (see Section 5.2 for details).
Therefore, we see a need to provide a framework for reporting demographics for
future researchers if they want to make their research comparable and
replicable, which echoes Sambasivan et al. (2021b)’s finding of: “ _the need
for incentivizing data excellence in which academic papers should evolve to
offer data documentation, provenance, and ethics as mandatory disclosure._ ”
(Sambasivan et al., 2021b)

Furthermore, considering conferences that often suffer from an excessive focus
on Western societies (Linxen et al., 2021; van Berkel et al., 2023; Yfantidou
et al., 2023; Septiandri et al., 2023), publishing this information will help
with research transparency.111For example, based on the data from CHI papers
published between 2016 to 2020, “ _73% of CHI study findings are based on
Western participant samples, representing less than 12% of the world’s
population._ ”(Linxen et al., 2021) It may also create a push for extending
the research to other populations. Especially with the current trend and
growth in HCER-AI research, more people will be influenced by the research
done in this community. Thus, additional care is required to make the research
inclusive and ensure that everyone is not lumped into the most powerful group,
for instance, by contextualizing the findings. To achieve this, one approach
could be by encouraging one of the several proposed methods for reporting
datasets, models, and participant demographics (e.g., Data Cards (Pushkarna et
al., 2022), Model Cards (Crisan et al., 2022), Datasheets for Datasets (Gebru
et al., 2021), FactSheets (Richards et al., 2019), and Data Statements (Bender
and Friedman, 2018)). The outcomes of these methods could serve as input for a
visualization tool to support easy and quick probing of transparency and
inclusivity.

We also want to highlight the invaluable contribution of a group of
researchers that shed light on less studied populations, such as India in this
case. Five papers in our set had a sample from India with shared authors.
Therefore, the community is interested in accepting papers from less studied
countries in the Global South. We encourage more researchers to study
marginalized populations.

##  5\. Results: Research Themes in HCER-AI

There has been an exponential growth in HCER-AI research in the past few years
(Figure 3). Based on our thematic analysis of 164 papers in this area, these
efforts fall into six main research themes (Figure 4): governance (n=94),
fairness (n=71), explainability (n=41), human flourishing (n=23), privacy
(n=19), and security (n=10). All conferences had at least one paper in each
research theme focused on governance and fairness. While AIES and FAccT are
smaller, with a narrow focus on AI and its ethical considerations, CHI and
CSCW have a slightly more diverse research portfolio, possibly because they
are larger conferences and attract a broader audience (human-computer
interaction broadly defined).

Figure 3. Count of HCER-AI papers per year. There is an exponential growth in
HCER-AI research within the past few years. Figure 4. Count of HCER-AI papers
per research theme. The top three most popular research themes among the
reviewed papers are governance, fairness, and explainability. Themes are not
mutually exclusive.

We created a pairwise similarity matrix for the pairs of research themes.
Based on Figure 5, we observed that privacy and security, as well as fairness
and governance, are the most common pairs (the closer the number gets to 1,
the more likely the pair appears together in a paper). On the other hand, many
pairs, such as governance with security, privacy, and human flourishing, did
not appear together in our reviewed papers. For example, papers on governance
often had a more general view of AI, except for some overlapping research with
fairness and explainability. Across the four conferences, security had the
least number of published papers. One possible explanation for this is the
well-established nature of security as a field, particularly within computer
science. Prominent conferences covering human factors, such as USENIX Security
and IEEE S&P, may attract papers at the intersection of security and
responsible AI. Future research may explore similar areas in privacy and
security conferences with human subjects.

Figure 5. Similarities between the research themes (n=164). The closer the
number gets to 1, the more likely the pair appears together in a paper. For
example, (privacy and security) and (fairness and governance) are often
researched together.

In the following we discuss each theme, with the number of resources in each
theme indicated next to the theme title (#N).

###  5.1. Governance (#94)

All conferences actively publish papers related to AI governance. Aside from
general guidelines and critical views on AI governance, papers here fall into
two sub-themes: guidelines about how AI should work, be governed, and
regulated (Muller and Strohmayer, 2022; Langer et al., 2022; Maas, 2018;
Erdélyi and Erdélyi, 2020; Whittlestone et al., 2019; Schiff et al., 2020;
Henriksen et al., 2021; Costanza-Chock et al., 2022; Ha, 2022; Weidinger et
al., 2022; Stapleton et al., 2022; Knowles and Richards, 2021; Abdalla and
Abdalla, 2021; Pushkarna et al., 2022; Smith et al., 2022; Crisan et al.,
2022; Sloane and Zakrzewski, 2022; Ashurst et al., 2022); and tools for
auditing AI (Henriksen et al., 2021; Krafft et al., 2021; Kroll, 2021; Ramesh
et al., 2022; Black et al., 2022; Costanza-Chock et al., 2022; Knowles and
Richards, 2021).

####  5.1.1. AI Guidelines

The three main stakeholders in the AI ecosystem are individuals (e.g., users,
developers, and AI experts), organizations (e.g., ACM, companies, and
independent institutes), and national and international bodies (e.g., UNICEF
and regulators) (Deshpande and Sharp, 2022; Whittlestone et al., 2019; Wang et
al., 2023).222Despite the increasing interest in more collectivist views
(McNaney et al., 2018; Vitos et al., 2017; Vigil-Hayes et al., 2017; Peters et
al., 2013; Reinecke et al., 2013), these considerations were not present in
our AI-centric search results. Several organizations and communities have
already started to respond to this urgent need by sharing responsible AI
guidelines (Varanasi and Goyal, 2023; Wang et al., 2023). These guidelines are
typically used not only for addressing AI’s design challenges but also for
education, cross-functional communication, and developing internal resources,
as illustrated in a study with Google’s People + AI Guidebook (Yildirim et
al., 2023). From a theoretical standpoint, AI can raise several tensions
between ethical values. For instance, improving service quality may require
additional data collection from users (Whittlestone et al., 2019). These
trade-offs are further confirmed by people who work for or co-founded a
startup where AI entrepreneurs face a dilemma between academic integrity and
potentially overrated marketing campaigns (Winecoff and Watkins, 2022).
Addressing these trade-offs requires understanding AI’s ethical challenges
(Schiff et al., 2020; Eicher et al., 2018), developing principles for
precautionary policy making (Ha, 2022; Terzis, 2020), educating students about
AI at schools (Lin and Van Brummelen, 2021), and building practical
recommendations for the safe and trustworthy deployment of AI (Maas, 2018;
Eicher et al., 2018; Baughan et al., 2023; Burgess et al., 2023). Previous
actions and strikes by employees in other industries represent one direction
that may support promoting ethical values in the AI industry, but they require
more commitment from employees than just open letters and petitions (Boag et
al., 2022). As existing legal liability systems fall short in assigning
responsibility when potentially harmful conduct occurs from using AI, the need
to establish AI schemes that ensure responsible development of AI is further
highlighted (Erdélyi and Erdélyi, 2020; Bietti, 2020). Considering individual
experiences, empirically understanding needs, and breaking traditional
abstract views of society to enable research communities to be self-reflective
and mindful of the previously mentioned trade-off in AI are directions that
we, as researchers, need to take to move toward ethics in AI (Siapka, 2022;
Washington and Kuo, 2020; Stark and Hoey, 2021; Rismani et al., 2023; Shahid
and Vashistha, 2023).

Funding sources often drive academic research. AI research is no exception;
hiring graduate students and researchers requires faculty and universities to
attract funding to recruit individuals for research. The AI industry’s
indirect influence on the governance of AI may come from these fundings that
shape academic research (Abdalla and Abdalla, 2021; Young et al., 2022). In a
sample of 149 faculties from four prestigious North American universities,
about half of those with known funding sources received money from large
technology companies like Google and Microsoft. This type of influence
resonates with the tobacco industry’s influence when academic research was
exploited to shift the public’s negative opinion toward the benefits and
positives of tobacco (Abdalla and Abdalla, 2021). The arbitrary reporting of
limitations in AI research exacerbates such concerns. The machine learning
research community lacks a single, agreed-upon definition of limitations and
does not have a standardized process for disclosing and discussing
limitations, in addition to often being non-inclusive (Chi et al., 2021) and
lacking model-work documentation (Crisan et al., 2022) or data-work
documentation (Sambasivan et al., 2021b; Muller and Strohmayer, 2022). On a
positive note, recommendations to fill these gaps exist. REAL ML, for
instance, provides a set of guided activities to help machine learning
researchers recognize, explore, and articulate the limitations of their
research (Smith et al., 2022). Alternatively, frameworks have been proposed to
disentangle different components of ethical research in machine learning to
allow AI researchers, practitioners, and regulators to systematically analyze
existing cultural understandings, histories, and social practices of ethical
AI (Sloane and Zakrzewski, 2022; Ashurst et al., 2022), guiding the creation
of human-AI teams (Flathmann et al., 2021).

In addition to the body of research on more organized and private AI companies
and research institutes, research on open-source communities shows that the
nature of open-source and unrestricted use of code propagates within the AI
open-source community as well (Widder et al., 2022). Working on open-source AI
projects creates a sense of neutrality and inevitability toward the technology
and its consequences, while there can be serious ramifications to these
projects, such as use cases for deep fakes. Similar approaches such as
documentation (Miceli et al., 2022) and nudging developers into responsible
development are suggested to alleviate the harm (Widder et al., 2022).
However, the community’s adoption of these guidelines and frameworks is still
unknown.

####  5.1.2. Tools for AI Auditing and Research

This category of papers aims to aid experts, activists, and independent bodies
in improving the quality and impact of auditing AI (Costanza-Chock et al.,
2022; Lam et al., 2023) (especially in the age of large language models (Zhou
et al., 2023)), to understand accountability as a responsibility put on the
shoulders of AI engineers (Henriksen et al., 2021), to discuss the right of
contestability (Lyons et al., 2021; Yurrita et al., 2022), and to engage with
theories from moral philosophy (Nashed et al., 2021). At the initial stages of
a project, design cards (Elsayed-Ali et al., 2023) can help bring knowledge in
areas such as creative inspiration, human insights, problem definition, and
team building (Hsieh et al., 2023). Other tools include computational
notebooks (Ayobi et al., 2023) to help practitioners explore machine learning
models, workflows that encourage designers to explore model behavior and
failure patterns early in the design process (Moore et al., 2023), data probes
to help practitioners surface the well-being and positionalities that shape
their work strategies (Lee et al., 2023a), and exhibitions to teach critical
thinking (Lee et al., 2023a). Another idea for creating a foundation for audit
is to apply the tracing and debugging mechanism from software engineering
(Kroll, 2021; Balayn et al., 2023). Version control systems keep track of
changes in code repositories, and similar ideas could be applied in auditing
AI, which requires structure, logs, and wide adoption (Kroll, 2021). Besides
technical documentation about how things work and their respective risks
(Pushkarna et al., 2022; Weidinger et al., 2022), many users (average users
without technical knowledge and interest in how systems work) are interested
in what the system does, how it can affect them, and whether it is
fundamentally reasonable to replace the traditional mechanism with AI (Knowles
and Richards, 2021; Stapleton et al., 2022).

Only by comprehending the foundations of algorithmic decision-making can we
establish legitimate control over algorithmic processes (Burrell et al.,
2019). Recently, scholars have begun to discuss the right to contestability,
which entails the ability to contest algorithmic decisions (Lyons et al.,
2021; Alfrink et al., 2023; Yurrita et al., 2022). For example, a study
conducted through a series of participatory design workshops found that by
designing for contestability, users can actively shape and exert influence
over algorithmic decision-making (Vaccaro et al., 2021). Another study showed
that, in high-stakes AI, physicians’ trust in AI was less dependent on their
general acceptance of AI but more on their contestable experiences with AI
(Verma et al., 2023). Another large body of work aims to engage AI
practitioners with ethical theories. For example, by expanding and defending
the Ethical Gravity Thesis, scholars proposed a framework that allows one to
situate ethical problems at the appropriate level of abstraction, which can be
used to target appropriate interventions (Kasirzadeh and Klein, 2021). Another
study provided a mathematical framework to model how much learning is required
for an intelligent agent to behave morally with negligible error (Shaw et al.,
2018), while another developed a computational model for building moral
autonomous vehicles by learning and generalizing from human moral judgments
(Kim et al., 2018).

#### Recommendations for Future Research

Future research could study the specifics of governing other research themes
in depth. Most of the current governance research is general except for
fairness; therefore, studying the governance of AI with respect to privacy,
security, human flourishing, and explainability is needed (Figure 5).
Furthermore, providing usable documentation and communicating AI’s abilities
and implications to the public to create trust in AI is necessary and needs
research beyond simple explanations that may cause over- or under-trust issues
(Liao and Sundar, 2022). One suggestion is to involve users and communities in
helping situate the requirements within the target individuals and groups
(Ramesh et al., 2022; Sambasivan et al., 2021a). The rise of “user-driven
audits” on social media such as Twitter (i.e., users hypothesizing, collecting
data, amplifying, contextualizing, and escalating AI issues and possible
harms) can provide a window into people’s expectations and provide a flow of
integration of AI by the communities (Wang et al., 2019b).

The industry’s impact on academic integrity is a known issue. In privacy, for
example, similar influences have been observed, discussed, and documented
(Waldman, 2021). As pointed out in the research methods of HCER-AI (§4),
transparency in data collection, reporting data characteristics, and funding
resources with an on-demand visualization tool to provide an overview of these
aspects could help. While participatory design approaches have often been used
for user-originated or worker-originated critiques of technologies, it is not
yet clear whether and how they can be applied to complex algorithmic systems
(Delgado et al., 2022; Neuhauser and Kreps, 2011; Zytko et al., 2022).
Designers of design cards also need to further extend the scope of these tools
to evaluation and later stages of AI projects beyond the current design cards
that focus on ideation and initial stages.

###  5.2. Fairness (#71)

Scholars and practitioners argue that fairness is contextual, cultural, and
individually dependent, among many other dimensions; one size does not fit all
(Docherty and Biega, 2022; Kapania et al., 2022; Wang et al., 2022c; Cheng et
al., 2021; Lee and Rich, 2021; Sambasivan et al., 2021a; Kasirzadeh and Smart,
2021; Richardson et al., 2021; Ashktorab et al., 2023; Lewicki et al., 2023;
Sharma et al., 2021; Aka et al., 2021; Cruz Cortés and Ghosh, 2020; Yang et
al., 2022; Deng et al., 2023). Therefore, when designing experiments,
contextual considerations should be considered to reduce errors in research
design and results (Kapania et al., 2022; Niforatos et al., 2020; Ashktorab et
al., 2023). For example, conclusions about individuals’ perceptions or
expectations of fairness may not be replicable when the country, age, or
gender group changes (Kapania et al., 2022; Wang et al., 2022c; Simons et al.,
2021). Some biases are inherent in practitioners’ (Ashktorab et al., 2023) or
data annotators’ (Kapania et al., 2023) backgrounds, and some are also present
in the inevitable changes to datasets and models during AI development (Muller
and Strohmayer, 2022; Muller et al., 2021). For example, during the data
annotation and curation process, when raw data is prepared for the final
model, disagreement between annotators may get lost, resulting in the final
ground truth being based on the agreement biased towards a certain annotator
or group (Muller and Strohmayer, 2022; Muller et al., 2021) and questioning
its credibility (Chen and Sundar, 2023). This is why scholars proposed new AI
data curation approaches grounded in feminist epistemology and informed by
critical theories of race and feminist principles (Leavy et al., 2021). While
the field of algorithmic fairness has primarily explored the notion of
fairness as _treating individuals alike_ or arguing for _social inclusion_
(Huang and Liem, 2022), there has been a recent debate on the concept of
_vertical equity_ —appropriately accounting for relevant differences across
individuals—which has also been a central fairness component in many public
policy settings (Black et al., 2022). The empirical notion of fairness from
non-experts also shows that people may accept an AI making inferences about
portraits when the use case is advertising but do not find it permissible for
AI to make inferences about an individual based on their gender, age, or race,
demonstrating the complexity of fairness (Engelmann et al., 2022).

Terminology (e.g., artificial intelligence, algorithm, and robot) also impacts
individuals’ perceptions of fair AI and how they trust the system’s decisions.
Therefore, when researchers conduct research with human participants and use
various terms to refer to an automated decision-making system, participants
may have a mixed understanding and expectation of trust and fairness from the
system when the terms addressing the studied system change (Langer et al.,
2022). Another example that can impact participants’ perceptions of an AI’s
fairness is people’s lived experiences. When a marginalized group has low
trust in a human decision-maker for historical reasons (e.g., not trusting a
medical doctor’s decision), people have similar expectations of fairness from
an AI (e.g., (Lee and Rich, 2021; Nakao et al., 2022; Sambasivan et al.,
2021a)).

Conversely, if an individual has a positive historical experience with the
human decision-maker (e.g., trusting a medical doctor’s decision), the
individual may place higher trust in the AI’s decisions as well (Lee and Rich,
2021). While in other situations, people often consider AI more capable than
humans but less morally trustworthy (Tolmeijer et al., 2022). Overall, human
agents are often held responsible in decision-making (Tolmeijer et al., 2022;
Lima et al., 2021). To alleviate the gap between designers and users of AI,
designers could bring users and diverse viewpoints into the design stages of
AI to create a shared understanding of fairness between different stakeholders
(Yildirim et al., 2022; Subramonyam et al., 2022; Park et al., 2022; Raz et
al., 2021; To et al., 2021; Choi et al., 2023; Burrell et al., 2019). To
further consider biases in the deployment stages, humans can be integrated
into the loop with AI (Cheng et al., 2022; Reitmaier et al., 2022; Raz et al.,
2021), resulting in human-algorithm collaboration (Donahue et al., 2022; Zheng
et al., 2023). For example, in deciding whether a child was maltreated, when
AI decisions were combined with a human decision-maker, the number of racially
biased decisions was reduced compared to when AI was the only decision-maker.
Hence, the human in the loop may improve decisions by taking an overarching
assessment and filling AI’s limitations in decision-making (Cheng et al.,
2022).

#### Recommendations for Future Research

Future research could study fairness from multiple perspectives (e.g.,
culture, gender, ethnicity, age) and compare expectations or understandings in
various groups (Lee and Rich, 2021; Nakao et al., 2022; Sambasivan et al.,
2021a). Multiple studies show the value of diversity in AI development and
research teams to uncover potential biases and gaps, a practice that academics
and industry need to adopt (Septiandri et al., 2023; Ashktorab et al., 2023).
Furthermore, the studied toolkits provide metrics to evaluate fairness,
primarily for AI developers (or experts). Researching how to present these
metrics to a non-expert user who is impacted by the system could be a
direction to pursue: how to present fairness to the public with minimal
knowledge about fairness (e.g., is _statistical parity_ the most
understandable way of presenting fairness to a senior adult or a child about
the fairness of an AI?). These tools can be helpful for non-expert users to
understand AI decisions and make informed decisions, as typically,
interactions between people and machines range between two extremes: humans
either tend to under-rely on an algorithm by ignoring its recommendations
(algorithmic aversion) or over-rely on it by blindly accepting any
recommendation (automation bias) (De-Arteaga et al., 2020; He et al., 2023).
Additionally, some scholars have argued that an AI can somehow decide
“objectively,” (Houser, 2019). In contrast, others have argued that humans
insert their biases into the data they wrangle and the models they build
(Muller and Strohmayer, 2022). The former “objectivity” argument could lead to
expectations of better treatment by an AI rather than a person. The latter
“bias” argument could lead to expectations of equally bad or worse treatment
by an AI. While toolkits for audit are focused on fairness, we believe there
is space to explore practices and build toolkits for audit and accountability
in other aspects like privacy and security. Another aspect to consider is that
accountability could provide explainability over decisions but make people
feel surveilled (Ehsan et al., 2021a); therefore, finding a balance between
accountability, privacy, and fairness is a challenge that requires further
exploration.

###  5.3. Explainability (#41)

If an AI operates as an opaque box,333Where a decision is generated without
explanation or interpretation, preventing users from understanding why and how
the decision was made users may have trouble trusting the outputs.
Consequently, a recent movement in AI seeks to make these decisions
comprehensible to garner users’ trust (Liao et al., 2020; Bertrand et al.,
2022; Kim et al., 2023; Yuan et al., 2023), and explores ways to measure their
impact (Cabitza et al., 2023). In human resource management, decisions made by
AI can be further elucidated to gain employees’ trust, helping them understand
the rationale behind a decision with clear evaluation criteria (which may
instill a sense of fairness) (Park et al., 2021, 2022). However, explanations
may not always absolve the AI of blame (Lima et al., 2023). Often, trust
issues depend on the personal relevance of the explanations and what is at
stake (Yuan et al., 2023), or on the lack of tools assisting AI practitioners
in generating meaningful explanations (Liao et al., 2023). Paradoxically,
explainability can sometimes conflict with privacy and security (Hall et al.,
2022; Park et al., 2022, 2021). We further discuss this trade-off in Sections
5.5 and 5.6.

However, explanations need to be human-centered, usable, and contextual (Liao
et al., 2020; Bansal et al., 2021; Ehsan et al., 2021a; Long et al., 2022;
Zhang et al., 2022; Raz et al., 2021; Lee et al., 2021; Toreini et al., 2020)
and may even need to be augmented with more than just simple text (Yang et
al., 2023). Otherwise, they could create a false sense of control or be used
as a scapegoat for responsibility (Lima et al., 2022). For example, when
explanations were employed in a human-AI team, they did not help improve the
accuracy of the final decision (Bansal et al., 2021). However, when combined
with an opaque box model that elucidated what the system does, users’
confusion was reduced (Bell et al., 2022). In either case, not all
explanations are beneficial. To enhance explanations, AI developers could
profit from posing _how_ and _why not_ questions to consider users’
expectations and needs (Liao et al., 2020), and could also utilize
participatory design approaches (Lee et al., 2021). Similar to asking the
“how” questions, an alternative framework, Sensible AI (Kaur et al., 2022)—for
interpretability was proposed, grounded in Weick’s sensemaking theory (Weick,
1995). From organizational studies, sensemaking describes the individual,
environmental, social, and organizational context affecting human
understanding. Similarly, Sensible AI translates these properties into the
human-machine context. Another way to improve explanations is to add graphics
and control sliders to help users understand an AI’s decision, making them
feel more in control (Viswanathan et al., 2022). Explanations can also help
improve AI literacy for public and educational purposes, potentially leading
to broader adoption and understanding of AI implications and harms (e.g.,
teaching kids about feature selection in machine learning using visual
explanations)(Long et al., 2022). Another study examined the role of cognition
in understanding explanations (Buçinca et al., 2021), proposing three
cognitive forcing interventions to compel people to engage more thoughtfully
with the AI-generated explanations.

#### Recommendations for Future Research

Future research is still needed to improve the explainability of AI’s inner
workings, decisions, and outputs, as there is some criticism about the value
of explanations like SHAP for humans (Kumar et al., 2020). Toolkits for
communicating datasets also need to track changes over time, from raw data to
the final dataset used in the model, and changes that occur post-deployment,
to mitigate the forgettance of data work—as Muller and Strohmayer (2022) put
it: “ _Forgettance in data science is when each action tends to push previous
actions into the infrastructure, where the action itself and its consequence
are easily forgotten._ ”(Muller and Strohmayer, 2022) One way of documenting
datasets is through a standardized process, such as using datasheets (Gebru et
al., 2021), nutrition labels (Chmielinski et al., 2022), or data statements
(Bender and Friedman, 2018). Furthermore, taking a cue from computer hardware
and industry safety standards, datasheets for datasets should include dataset
provenance, key characteristics, relevant regulations, test results, and also
more subjective yet noteworthy information (e.g., the potential bias of a
dataset (Gebru et al., 2021)).

###  5.4. Human Flourishing (#23)

Sustainable growth and the well-being of humans (also known as human
flourishing) using or being affected by AI were discussed in several studies
(Wang et al., 2022c; Docherty and Biega, 2022; Wang et al., 2022b; Steiger et
al., 2021; Beede et al., 2020; Morrison et al., 2021; Lee et al., 2021). For
example, if an employer wants to increase productivity using AI from a
sustainable perspective, employees’ expectations should be considered
alongside implementing positive changes, such as monetary compensation (Park
et al., 2022). AI can also facilitate real-time communication for users of
augmentative and alternative communication (Valencia et al., 2023). AI’s
impact on children has been deemed crucial due to its potential for long-term
effects. Instances of using robots for education at early stages of child
development can shape children’s perceptions of robots. Moreover,
entertainment systems that collect children’s data may affect their entire
life if service providers do not respect their privacy and keep their data
indefinitely (Wang et al., 2022c).444Adults are also susceptible to persuasion
by, for example, “cute” – but lethal – robots (Kemper and Kolain, 2022;
Vaisman, 2021; Young, 2021). In HCI literature, safety drivers working with
Autonomous Vehicles (AV) often comprise an under-explored population impacted
by AI. A study found that, as front-line workers, safety drivers are forced to
shoulder risks accumulated from the upstream AV industry while confronting
restricted self-development prospects when working for AV development (Chu et
al., 2023). Similarly, socio-environmental factors must also be considered for
the successful implementation of AI in real-world settings (e.g., a blind
person navigating social interactions or the country of post-deployment)
(Beede et al., 2020; Morrison et al., 2021). For example, offering
explanations and options for employers to customize an AI-based scheduling
system resulted in a smoother transition to AI and less resistance from users
(Lee et al., 2021).

A few papers focused on the labor behind AI (Wang et al., 2022b; Steiger et
al., 2021; Klinova and Korinek, 2021). AI creates new markets and
opportunities which bring wealth to those who will be more in demand (e.g., AI
developers) and shrinks the wealth of those who will be less in demand (e.g.,
customer service agents replaced by an AI chatbot) (Klinova and Korinek,
2021). However, new AI markets may not always be accommodating and lucrative
for all. As part of the data creation process, annotators often work long
hours with low wages, especially when cost-effective methods of annotating
large datasets are in demand, leading AI companies to hire annotators from
third-party annotation companies in the Global South for poor wages. In these
companies, some annotators start working with the expectation that annotation
work will open up opportunities for high-paying jobs in the future and serve
as a stepping stone to becoming an engineer. However, this is often a fallacy,
and these annotators rarely progress in their careers (Wang et al., 2022b).
Similarly, content moderators are sometimes exposed to inappropriate or
disturbing content, causing severe stress and trauma. Some of these
individuals were not fully aware of the risks associated with the job when
they signed up for it due to a lack of transparency in the job description
(Steiger et al., 2021).

Either of these jobs (annotation or content moderation) is often perceived as
a “dirty” job and receives minimal attention from companies (Steiger et al.,
2021; Wang et al., 2022b; Sambasivan et al., 2021b), or as “patchwork,” that
is, the human labor that occurs in the space between what AI purports to do
and what it actually accomplishes (Fox et al., 2023). Nevertheless, their
contributions to the AI economy are invaluable555“ _Without the work and labor
poured into the data annotation process, ML efforts are no more than
sandcastles._ ” (Wang et al., 2022b) and foundational (by generating datasets
and keeping AI safe from inappropriate content that others do not have to deal
with). The following four recommendations can serve as a starting point to
help these individuals manage their jobs’ challenges (Gray and Suri, 2019;
Moreschi et al., 2020; Steiger et al., 2021): (1) providing a detailed job
description outlining its associated risks could give candidates an
understanding of whether the job is appropriate for them, (2) limiting the
number of contents they view or annotate per day or week, (3) creating a
supportive community to help with stress relief, and (4) companies should
consider offering employee benefits, including physical and mental health
benefits, to these ”ghost” workers.

#### Recommendations for Future Research

Future research could study the long-term effects of AI on people, the people
who directly use AI (e.g., a person using a fitness tracker with an AI coach
(Garcia and Cifor, 2019)), those who are indirectly affected by AI (e.g., a
person receiving an AI-assisted loan decision from a credit institute
(Hertzberg et al., 2010; Kizilaslan and Lookman, 2017), or the people who may
lose their jobs or fear that they may lose their job because of AI (Josie Cox,
2023)), and the people involved in AI’s development and deployment (e.g.,
annotators and developers). A longer view of the impacts of AI algorithms
could include neighborhoods affected by over-policing through AI technologies
(Gee, 2021). For instance, as AI in workplaces becomes a technological
intervention, which may have different impacts on different groups within that
workplace (Park et al., 2021, 2022), multiple workplace scenarios can be
analyzed through value-sensitive design (Muller and Weisz, 2022). Docherty and
Biega (2022) argue that the tech industry’s focus on user engagement and time
spent on the platform is a reductionist view of human well-being. When
measuring well-being, an individual’s specific needs and characteristics
should be considered. Building on this idea, future researchers could study
the well-being of people impacted by AI beyond limiting engagement time and
think about human flourishing (i.e., “ _the ability to live a good life_ ”
(Health Equity & Policy Lab, 2022)) from a positive computing viewpoint (Calvo
and Peters, 2014). Furthermore, although not discussed in the reviewed papers,
we see a need to study AI’s environmental effects on natural resources from a
nature-centered perspective. For instance, a tool could be designed to remind
users about the energy consumption of their AI tools or to provide information
about the energy consumption of AI models (including all stages, such as
design, development, and deployment) with an easy-to-understand label, like
the Energy Star.

###  5.5. Privacy (#19)

Several papers have touched on the privacy ramifications of AI (Mlynar et al.,
2022; Lee and Rich, 2021; Zhang et al., 2022; Ehsan et al., 2021a; Park et
al., 2022, 2021; Yildirim et al., 2022; Subramonyam et al., 2022; Wang et al.,
2022c; Hall et al., 2022). For example, the use of human resource management
tools in companies, which bear privacy implications, can make employees
uncomfortable. These systems may be as simple as controlling a gate and
storing that information for performance measures. However, collecting more
data to improve the model’s accuracy can induce a sense of surveillance
(Greiffenhagen et al., 2023; Constantinides and Quercia, 2022). Nevertheless,
from the viewpoint of those deploying the system, such data collection may be
seen as a means of providing a fair assessment of the employee’s performance
(Park et al., 2022, 2021). Such systems may create conceptual mismatches
between normative (or cultural) and legal expectations regarding the use of
personal data (Nielsen, 2021). Designers and engineers acknowledge this
tension between privacy and collecting more data or sharing datasets between
different groups or teams to explore new opportunities. In fact, there is
evidence that while people support the sharing of their data to improve
technology, they also express concerns over commercial use, associated
metadata, and the lack of transparency about the impact of their data
(Kamikubo et al., 2023). Nonetheless, when a company is a large organization
(especially when client data is involved), sharing data is difficult because
data ownership can be vague and challenging (Yildirim et al., 2022;
Subramonyam et al., 2022). In contrast, smaller companies grapple with several
tensions, for example, the trade-off between privacy and ubiquity, resource
management and performance optimization, or access and monopolization (Hopkins
and Booth, 2021).

A few papers discuss the trade-off between privacy and explainability (Zhang
et al., 2022; Ehsan et al., 2021a; Hall et al., 2022). Providing additional
information about the model for explainability may compromise privacy or vice
versa. For instance, in a study with people’s WiFi data, participants were
initially private and sensitive about sharing their WiFi data. However,
showing visualizations about their data usage made them feel more comfortable
sharing their data because it fostered a sense of trust (Hall et al., 2022).
On the other hand, privacy can become a barrier to providing explanations in
image classification systems. For example, when images are obfuscated for
privacy reasons, providing explanations for the classification may reveal the
identities of people in the images (Zhang et al., 2022). Other scholars have
focused on how privacy legislation is discussed among different sets of
relationships (e.g., between companies and investors (Wong et al., 2023a))
through the lens of law and ethics (Benthall and Goldenfein, 2021). For
example, a study found that startups with data-sharing partnerships with high-
technology firms or prior experience with privacy regulations are more prone
to adopting ethical AI principles (Bessen et al., 2022). Furthermore, they are
more inclined to take costly actions, such as eliminating training data or
rejecting business opportunities, to adhere to their ethical AI policies.

#### Recommendations for Future Research

Future research may explore the trade-off between privacy (and/or security)
and explainability, as there is a thin line between what _can_ be monitored
and what _should_ be monitored (Constantinides and Quercia, 2022). One
perspective is to look at this trade-off from the lens of historian and
philosopher Yuval Noah Harari, who posits that the digital platforms, whether
powered by AI or not, need to adhere to three basic rules to protect humans
from “ _digital dictatorships_ ” (Yuval Noah Harari, 2021). These rules are
(1) reasonable use cases for data collection, meaning that any data being
collected should be used to assist people rather than manipulate, control, or
harm them (e.g., providing Indigenous data sovereignty over culturally-
sensitive data (Kukutai et al., 2020; Marley, 2019; Tsosie, 2019; Walter and
Suina, 2019)), (2) surveillance should be bidirectional, implying that if an
entity (e.g., an organization or a government) increases surveillance of
individuals, accountability on the entity’s side should correspondingly
increase, and (3) the elimination of data monopolies, which arise from the
concentration of data in a single entity. Additionally, the General Data
Protection Regulation (GDPR) (The European Parliament and the Council of the
European Union, 2018) in Europe argues for limiting the storage time only to
legitimate purposes and doing regular reviews to delete unneeded data. Human-
centered approaches and design methods can be employed to explore solutions
for these problems through committees that oversee data collection and,
ideally, ensure that data is in the hands of the individuals.

###  5.6. Security (#10)

AI security aims to ensure safety and reduce harm to individuals (Wang et al.,
2022c; Toreini et al., 2020). It involves enhancing a system’s ability to
resist external threats by testing its resilience against vulnerabilities and
cyber-attacks, all while safeguarding the integrity and confidentiality of
personal data (Fjeld et al., 2020). However, security may conflict with
explainability. For instance, in the case of human resource management
systems, disclosing information about models for explainability purposes could
pose a security risk. Such openness could harm the organization’s reputation
and provide attack opportunities if all model details are publicly available
(Park et al., 2022, 2021). Furthermore, using AI in security-sensitive
decision-making can be questionable. If AI decisions yield many false
positives, reliance on them can create unwanted stress, as the operator in
charge of making the final decision might feel compelled to act on them. This
can be particularly stressful for junior staff who may feel obliged to report
every incident (Ehsan et al., 2021a).

#### Recommendations for Future Research

Future research may benefit from studying the security aspects of AI from a
human-centered perspective. While there is a specific security and privacy
track in CHI that publishes papers about the human factor of security and
privacy in various technologies such as smartphones, apps, and browsers, and
investigates the role of software developers in secure development, we believe
the community can benefit from a dedicated focus on what it means to provide
human-centered security in AI. This should consider the trade-off between
explainability and security, which echoes the classic trade-off between
usability and security (Cranor and Garfinkel, 2005; Tahaei and Vaniea, 2019).
Creating usable security systems is often challenging. In response to this
challenge, a dedicated venue, the Symposium on Usable Privacy and Security
(SOUPS), was established in 2005. Implementing more complicated security (or
privacy) mechanisms can burden users, deter them from using the system, clash
with their mental models, and lead to insecure actions. Similarly, providing
more explanations for AI outcomes may cause users to overtrust the system.
Therefore, finding the right balance between the two calls for further
research.

##  6\. Results: Patents of HCER-AI

The current state of patents in HCER-AI is encapsulated by the following
themes: explainability (n=13), fairness (n=11), governance (n=6), human
flourishing (n=4), privacy (n=4), and security (n=2). The themes in patents
and research papers somewhat follow each other, except for governance being
the major theme in research, which is expected because research venues cover
theoretical, review, and discourse papers, while patents are more on the
industry application level (Table 3). Also, given that the first academic
paper on the subject was published in 2018, it is unsurprising to see a small
set of patents in this area. HCI research takes 10.5 years to appear in a
patent application, and this lag is increasing over time (Cao et al., 2023).
The exponential rise in research papers in HCER-AI presents an opportunity for
early-adopter industries to work on related patents.

Table 3. Count of HCER-AI papers (n=164) and patents (n=28) in each theme. | Governance | Fairness | Explainability | Human flourishing | Privacy | Security
---|---|---|---|---|---|---
Research papers | 94 (57%) | 71 (43%) | 41 (25%) | 23 (14%) | 19 (12%) | 10 (6%)
Patents | 6 (21%) | 11 (39%) | 13 (46%) | 4 (14%) | 4 (14%) | 2 (7%)

##  7\. Mapping the HCER-AI Landscape With the NIST AI Risk Management
Framework

We also mapped our constructed HCER-AI research themes onto the U.S. National
Institute of Standards and Technology (NIST) Artificial Intelligence Risk
Management Framework, published in 2023. We chose NIST because it is a recent
framework from a renowned organization for developing frameworks and
standards. Alternatives include the Principled Artificial Intelligence from
the Berkman Klein Center (Fjeld et al., 2020), which aligns with the NIST
framework. Such mapping provides insights into the different perspectives on
the landscape from the viewpoints of a standardization body versus academic
research. This approach helps highlight areas needing further attention,
potentially directing more funding and focus toward those areas to balance the
research portfolio.

  * •

Governance (n=94): maps with NIST’s “Govern” function: “cross-cutting function
that is infused throughout AI risk management and enables the other functions
of the process,” “Accountable and Transparent” risk: “accountability
presupposes transparency. Transparency reflects the extent to which
information about an AI system and its outputs is available to individuals
interacting with such a system – regardless of whether they are even aware
that they are doing so,” and “Valid and Reliable” risk: “Validity and
reliability for deployed AI systems are often assessed by ongoing testing or
monitoring that confirms a system is performing as intended.”

  * •

Fairness (n=71): maps with NIST’s “Fair – with Harmful Bias Managed” risk:
“includes concerns for equality and equity by addressing issues such as
harmful bias and discrimination.”

  * •

Explainability (n=41): maps with NIST’s “Explainable and Interpretable” risk:
“refers to a representation of the mechanisms underlying AI systems’
operation, whereas interpretability refers to the meaning of AI systems’
output in the context of their designed functional purposes.”

  * •

Human flourishing (n=23): AI should support “happiness and life satisfaction,
meaning and purpose, character and virtue, and close social relationships.”
(Willen et al., 2022) This theme also maps loosely with the NIST’s “Safe”
risk: “AI systems should not under defined conditions, lead to a state in
which human life, health, property, or the environment is endangered.”

  * •

Privacy (n=19): maps with NIST’s “Privacy-Enhanced” risk: “refers generally to
the norms and practices that help to safeguard human autonomy, identity, and
dignity.”

  * •

Security (n=10): maps with NIST’s “Secure and Resilient” risk: “AI systems, as
well as the ecosystems in which they are deployed, may be resilient if they
can withstand unexpected adverse events or changes in their environment or
use.”

##  8\. Posteriori Analysis Using ChatGPT 4.0

Motivated by recent work on using AI for qualitative analysis (Byun et al.,
2023; Abram et al., 2020; Tahaei et al., 2020), we explored the possibilities
of using ChatGPT in a similar analysis without manual analysis based only on
the abstracts (because of input limitations of AI models). We used all the
abstracts of the 164 papers as input for ChatGPT to evaluate its ability to
produce a similar report to ours. The primary tasks were finding research
themes, methods, and recommendations (our RQs). Noting that, we used ChatGPT
4.0, like any other text clustering and classification tool, to probe for
topics we might have missed during our research. This process was a complete
posthoc procedure after human authors wrote and finalized all the paper
sections except this one. All of our prompts and all the content, besides the
main headlines and topics, produced by ChatGPT are intentionally left in
Appendix B to avoid confusion between our analysis and findings vs. AI-
generated analysis and findings.

We first asked for research themes from the abstract following the thematic
analysis proposed by Braun and Clarke (2006) (our approach). ChatGPT came up
with the following four research themes: ethical implications of AI, practical
applications of AI, understanding and documenting data, and user engagement.
We further asked for the research methods used in the abstracts with a caveat
that not all papers mention their methods in their abstracts. The answer was
more nuanced than the research themes: interviews, participatory design, co-
design workshops, prototyping, observation, critical discourse analysis,
iterative co-design activities, characterization of collective actions, and
case studies. We followed the exploration with research recommendations and
future research directions based on the thematic analysis. The following
recommendations were generated: user-centric design, greater transparency in
AI, AI in marginalized communities, collective action in tech industry, real-
world AI evaluations, AI in disaster risk management, effective AI
communication, AI ethics integration in education, work-integrated learning
for AI, and AI governance practices. We did the same analysis for research
themes of patents; the resulting themes were: AI-driven decision making and
predictions, transparency, interpretability, and explainability in AI, and
machine learning model performance & enhancement.

The research themes produced by ChatGPT (ethical implications of AI, practical
applications of AI, understanding and documenting data, and user engagement)
could help get a general understanding of the dataset but were not enough to
get an in-depth grasp of the data. Our research themes (governance, fairness,
explainability, human flourishing, privacy, and security) are also a construct
of our positionality (see Section 3.1). Despite the differences in both lists,
AI governance and AI’s ethical implications are visible. We also recognize
that the classification generated by AI does not constitute a compelling
taxonomy, with some themes containing others without being of the same level
of importance. Possibly, because the AI-generated themes may be meant for
general consumption and do not directly correlate with specific research areas
of our analysis. Overall, we speculate that AI-assisted research tools like
ChatGPT may, in the future, act as a quick secondary point to help researchers
with unseen perspectives—considering that it took us less than two hours to
produce these results compared to months of work for the rest of the paper.
Future research may explore how researchers can better leverage AI to deliver
valuable literature reviews by conducting more experiments, comparisons with
human-generated analysis, and prompt engineering.

##  9\. Conclusion

We collected 228 records related to HCER-AI from four conference proceedings:
AIES, CHI, CSCW, and FAccT, as well as 67 patent applications. We then
selected and thematically analyzed 164 research papers and 28 patents from
these records. We found that the HCER-AI landscape emphasizes governance,
fairness, and explainability. The research community needs to put effort into
HCER-AI beyond fairness and create usable tools for non-expert users to audit
AI and think about privacy and security from a human-centered viewpoint. While
these topics are part of the broader conversation in responsible AI, there is
still a need for future research to bring them into practice.

###### Acknowledgements.

We thank Ali Septiandri, Edyta Bogucka, Ke Zhou, Lucy Havens, Sanja
Scepanovic, and Vera Liao, for their constructive feedback that helped improve
the paper.

## References

  * (1)
  * Abdalla and Abdalla (2021) Mohamed Abdalla and Moustafa Abdalla. 2021.  The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462563
  * Abdul et al. (2018) Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankanhalli. 2018.  Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ _(CHI ’18)_. ACM.  https://doi.org/10.1145/3173574.3174156
  * Abram et al. (2020) Marissa D. Abram, Karen T. Mancini, and R. David Parker. 2020\.  Methods to Integrate Natural Language Processing Into Qualitative Research.  _International Journal of Qualitative Methods_ (2020).  https://doi.org/10.1177/1609406920984608
  * Aka et al. (2021) Osman Aka, Ken Burke, Alex Bauerle, Christina Greer, and Margaret Mitchell. 2021.  Measuring Model Biases in the Absence of Ground Truth. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462557
  * Alfrink et al. (2023) Kars Alfrink, Ianus Keller, Neelke Doorn, and Gerd Kortuem. 2023.  Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580984
  * Angelov et al. (2021) Plamen P. Angelov, Eduardo A. Soares, Richard Jiang, Nicholas I. Arnold, and Peter M. Atkinson. 2021.  Explainable artificial intelligence: an analytical review.  _WIREs Data Mining and Knowledge Discovery_ (2021).  https://doi.org/10.1002/widm.1424
  * Aragon et al. (2022) Cecilia Aragon, Shion Guha, Marina Kogan, Michael Muller, and Gina Neff. 2022.  _Human-centered data science: An introduction_.  MIT Press.  https://mitpress.mit.edu/9780262543217/human-centered-data-science/
  * Arthur et al. (2008) Melanie Arthur, Jerris R Hedges, Craig D Newgard, Brain S Diggs, and Richard J Mullins. 2008.  Racial disparities in mortality among adults hospitalized after injury.  _Medical care_ (2008).  https://doi.org/10.1097/MLR.0b013e31815b9d8e
  * Ashktorab et al. (2023) Zahra Ashktorab, Benjamin Hoover, Mayank Agarwal, Casey Dugan, Werner Geyer, Hao Bang Yang, and Mikhail Yurochkin. 2023.  Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581227
  * Ashurst et al. (2022) Carolyn Ashurst, Solon Barocas, Rosie Campbell, and Deborah Raji. 2022.  Disentangling the Components of Ethical Research in Machine Learning. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533781
  * Associated Press (2021) Associated Press. 2021\.  _Judge approves $650m settlement of privacy lawsuit against Facebook_.  Guardian News & Media Limited.  Retrieved January 2023 from https://www.theguardian.com/technology/2021/feb/27/facebook-illinois-privacy-lawsuit-settlement
  * Ayobi et al. (2023) Amid Ayobi, Jacob Hughes, Christopher J Duckworth, Jakub J Dylag, Sam James, Paul Marshall, Matthew Guy, Anitha Kumaran, Adriane Chapman, Michael Boniface, and Aisling Ann O’Kane. 2023.  Computational Notebooks as Co-Design Tools: Engaging Young Adults Living with Diabetes, Family Carers, and Clinicians with Machine Learning Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581424
  * Baeza-Yates (2018) Ricardo Baeza-Yates. 2018\.  Bias on the Web.  _Commun. ACM_ (May 2018).  https://doi.org/10.1145/3209581
  * Balayn et al. (2023) Agathe Balayn, Natasa Rikalo, Jie Yang, and Alessandro Bozzon. 2023.  Faulty or Ready? Handling Failures in Deep-Learning Computer Vision Models until Deployment: A Study of Practices, Challenges, and Needs. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581555
  * Banovic et al. (2023) Nikola Banovic, Zhuoran Yang, Aditya Ramesh, and Alice Liu. 2023\.  Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579460
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021\.  Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445717
  * Barredo Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ (2020).  https://doi.org/10.1016/j.inffus.2019.12.012
  * Baughan et al. (2023) Amanda Baughan, Xuezhi Wang, Ariel Liu, Allison Mercurio, Jilin Chen, and Xiao Ma. 2023\.  A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581152
  * Bawa et al. (2020) Anshul Bawa, Pranav Khadpe, Pratik Joshi, Kalika Bali, and Monojit Choudhury. 2020.  Do Multilingual Users Prefer Chat-Bots That Code-Mix? Let’s Nudge and Find Out!  _Proc. ACM Hum.-Comput. Interact._ (May 2020).  https://doi.org/10.1145/3392846
  * Beede et al. (2020) Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis. 2020\.  A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM.  https://doi.org/10.1145/3313831.3376718
  * Bell et al. (2022) Andrew Bell, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich. 2022.  It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533090
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science.  _Transactions of the Association for Computational Linguistics_ (2018).  https://doi.org/10.1162/tacl_a_00041
  * Benthall and Goldenfein (2021) Sebastian Benthall and Jake Goldenfein. 2021.  Artificial Intelligence and the Purpose of Social Systems. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462526
  * Bergram et al. (2022) Kristoffer Bergram, Marija Djokovic, Valéry Bezençon, and Adrian Holzer. 2022.  The Digital Landscape of Nudging: A Systematic Literature Review of Empirical Research on Digital Nudges. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517638
  * Bertrand et al. (2022) Astrid Bertrand, Rafik Belloum, James R. Eagan, and Winston Maxwell. 2022.  How Cognitive Biases Affect XAI-Assisted Decision-Making: A Systematic Review. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’22)_. ACM.  https://doi.org/10.1145/3514094.3534164
  * Bessen et al. (2022) James Bessen, Stephen Michael Impink, and Robert Seamans. 2022.  The Cost of Ethical AI Development for AI Startups. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’22)_. ACM.  https://doi.org/10.1145/3514094.3534195
  * Bietti (2020) Elettra Bietti. 2020\.  From Ethics Washing to Ethics Bashing: A View on Tech Ethics from within Moral Philosophy. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. ACM.  https://doi.org/10.1145/3351095.3372860
  * Bingley et al. (2023) William J. Bingley, Caitlin Curtis, Steven Lockey, Alina Bialkowski, Nicole Gillespie, S. Alexander Haslam, Ryan K.L. Ko, Niklas Steffens, Janet Wiles, and Peter Worthy. 2023.  Where is the human in human-centered AI? Insights from developer priorities and user experiences.  _Computers in Human Behavior_ (2023).  https://doi.org/10.1016/j.chb.2022.107617
  * Black et al. (2022) Emily Black, Hadi Elzayn, Alexandra Chouldechova, Jacob Goldin, and Daniel Ho. 2022.  Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533204
  * Boag et al. (2022) William Boag, Harini Suresh, Bianca Lepe, and Catherine D’Ignazio. 2022.  Tech Worker Organizing for Power and Accountability. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533111
  * Boyarskaya et al. (2020) Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford. 2020\.  Overcoming Failures of Imagination in AI Infused System Development and Deployment. In _In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020_.  https://www.microsoft.com/en-us/research/publication/overcoming-failures-of-imagination-in-ai-infused-system-development-and-deployment/
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative Research in Psychology_ (2006).  https://doi.org/10.1191/1478088706qp063oa
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research)_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR.  https://proceedings.mlr.press/v81/buolamwini18a.html
  * Burgess et al. (2023) Eleanor R. Burgess, Ivana Jankovic, Melissa Austin, Nancy Cai, Adela Kapuundefinedcińska, Suzanne Currie, J. Marc Overhage, Erika S Poole, and Jofish Kaye. 2023.  Healthcare AI Treatment Decision Support: Design Principles to Enhance Clinician Adoption and Trust. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581251
  * Burrell et al. (2019) Jenna Burrell, Zoe Kahn, Anne Jonas, and Daniel Griffin. 2019\.  When Users Control the Algorithms: Values Expressed in Practices on Twitter.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2019).  https://doi.org/10.1145/3359240
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021.  To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making.  _Proc. ACM Hum.-Comput. Interact._ (April 2021).  https://doi.org/10.1145/3449287
  * Byun et al. (2023) Courtni Byun, Piper Vasicek, and Kevin Seppi. 2023\.  Dispensing with Humans in Human-Computer Interaction Research. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’23)_. ACM, Article 413, 26 pages.  https://doi.org/10.1145/3544549.3582749
  * Cabitza et al. (2023) Federico Cabitza, Andrea Campagner, Riccardo Angius, Chiara Natali, and Carlo Reverberi. 2023\.  AI Shall Have No Dominion: On How to Measure Technology Dominance in AI-Supported Human Decision-Making. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581095
  * Cabrera et al. (2021) Ángel Alexander Cabrera, Abraham J. Druck, Jason I. Hong, and Adam Perer. 2021.  Discovering and Validating AI Errors With Crowdsourced Failure Reports.  _Proc. ACM Hum.-Comput. Interact._ (Oct. 2021).  https://doi.org/10.1145/3479569
  * Calvo and Peters (2014) Rafael A Calvo and Dorian Peters. 2014.  _Positive Computing: Technology for Wellbeing and Human Potential_.  MIT press.  https://mitpress.mit.edu/9780262533706/positive-computing/
  * Cao et al. (2023) Hancheng Cao, Yujie Lu, Yuting Deng, Daniel Mcfarland, and Michael S. Bernstein. 2023.  Breaking Out of the Ivory Tower: A Large-Scale Analysis of Patent Citations to HCI Research. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581108
  * Capel and Brereton (2023) Tara Capel and Margot Brereton. 2023.  What is Human-Centered about Human-Centered AI? A Map of the Research Landscape. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM, Article 359\.  https://doi.org/10.1145/3544548.3580959
  * Caton and Haas (2020) Simon Caton and Christian Haas. 2020.  Fairness in Machine Learning: A Survey.  https://doi.org/10.48550/ARXIV.2010.04053
  * Chancellor (2023) Stevie Chancellor. 2023\.  Toward Practices for Human-Centered Machine Learning.  _Commun. ACM_ (Feb. 2023).  https://doi.org/10.1145/3530987
  * Chancellor et al. (2019) Stevie Chancellor, Eric P. S. Baumer, and Munmun De Choudhury. 2019\.  Who is the ”Human” in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2019).  https://doi.org/10.1145/3359249
  * Chen and Sundar (2023) Cheng Chen and S. Shyam Sundar. 2023.  Is This AI Trained on Credible Data? The Effects of Labeling Quality and Performance Bias on User Trust. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580805
  * Cheng et al. (2022) Hao-Fei Cheng, Logan Stapleton, Anna Kawakami, Venkatesh Sivaraman, Yanghuidi Cheng, Diana Qing, Adam Perer, Kenneth Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022\.  How Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3501831
  * Cheng et al. (2021) Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021\.  Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445308
  * Chi et al. (2021) Nicole Chi, Emma Lurie, and Deirdre K. Mulligan. 2021\.  Reconfiguring Diversity and Inclusion for AI Ethics. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462622
  * Chmielinski et al. (2022) Kasia S. Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. 2022.  The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence.  https://doi.org/10.48550/ARXIV.2201.03954
  * Choi et al. (2023) Yoonseo Choi, Eun Jeong Kang, Min Kyung Lee, and Juho Kim. 2023\.  Creator-Friendly Algorithms: Behaviors, Challenges, and Design Opportunities in Algorithmic Platforms. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581386
  * Chu et al. (2023) Mengdi Chu, Keyu Zong, Xin Shu, Jiangtao Gong, Zhicong Lu, Kaimin Guo, Xinyi Dai, and Guyue Zhou. 2023\.  Work with AI and Work for AI: Autonomous Vehicle Safety Drivers’ Lived Experiences. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581564
  * Constantinides and Quercia (2022) Marios Constantinides and Daniele Quercia. 2022.  Good Intentions, Bad Inventions: How Employees Judge Pervasive Technologies in the Workplace.  _IEEE Pervasive Computing_ (2022).  https://doi.org/10.1109/MPRV.2022.3217408
  * Costanza-Chock et al. (2022) Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. 2022.  Who Audits the Auditors? Recommendations from a Field Scan of the Algorithmic Auditing Ecosystem. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533213
  * Cranor and Garfinkel (2005) Lorrie Faith Cranor and Simson Garfinkel. 2005.  _Security and Usability: Designing Secure Systems That People Can Use_.  O’Reilly Media, Inc.  https://www.oreilly.com/library/view/security-and-usability/0596008279/
  * Crisan et al. (2022) Anamaria Crisan, Margaret Drouhard, Jesse Vig, and Nazneen Rajani. 2022.  Interactive Model Cards: A Human-Centered Approach to Model Documentation. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533108
  * Cruz Cortés and Ghosh (2020) Efrén Cruz Cortés and Debashis Ghosh. 2020.  An Invitation to System-Wide Algorithmic Fairness. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’20)_. ACM.  https://doi.org/10.1145/3375627.3375860
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020\.  A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM.  https://doi.org/10.1145/3313831.3376638
  * Delgado et al. (2022) Fernando Delgado, Solon Barocas, and Karen Levy. 2022\.  An Uncommon Task: Participatory Design in Legal AI.  _Proc. ACM Hum.-Comput. Interact._ , Article 51 (April 2022).  https://doi.org/10.1145/3512898
  * Deng et al. (2023) Wesley Hanwen Deng, Boyuan Guo, Alicia Devrio, Hong Shen, Motahhare Eslami, and Kenneth Holstein. 2023.  Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581026
  * Deshpande and Sharp (2022) Advait Deshpande and Helen Sharp. 2022.  Responsible AI Systems: Who Are the Stakeholders?. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’22)_. ACM.  https://doi.org/10.1145/3514094.3534187
  * Dieber and Kirrane (2020) Jürgen Dieber and Sabrina Kirrane. 2020.  Why model why? Assessing the strengths and limitations of LIME.  https://doi.org/10.48550/ARXIV.2012.00093
  * Docherty and Biega (2022) Niall Docherty and Asia J. Biega. 2022.  (Re)Politicizing Digital Well-Being: Beyond User Engagements. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3501857
  * Donahue et al. (2022) Kate Donahue, Alexandra Chouldechova, and Krishnaram Kenthapadi. 2022\.  Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533221
  * Ehsan et al. (2021a) Upol Ehsan, Q. Vera Liao, Michael Muller, Mark O. Riedl, and Justin D. Weisz. 2021a.  Expanding Explainability: Towards Social Transparency in AI Systems. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445188
  * Ehsan et al. (2023) Upol Ehsan, Koustuv Saha, Munmun De Choudhury, and Mark O. Riedl. 2023.  Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579467
  * Ehsan et al. (2021b) Upol Ehsan, Philipp Wintersberger, Q. Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O. Riedl. 2021b.  Operationalizing Human-Centered Perspectives in Explainable AI. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’21)_. ACM.  https://doi.org/10.1145/3411763.3441342
  * Eicher et al. (2018) Bobbie Eicher, Lalith Polepeddi, and Ashok Goel. 2018\.  Jill Watson Doesn’t Care If You’re Pregnant: Grounding AI Ethics in Empirical Studies. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’18)_. ACM.  https://doi.org/10.1145/3278721.3278760
  * Elsayed-Ali et al. (2023) Salma Elsayed-Ali, Sara E Berger, Vagner Figueredo De Santana, and Juana Catalina Becerra Sandoval. 2023\.  Responsible & Inclusive Cards: An Online Card Tool to Promote Critical Reflection in Technology Industry Work Practices. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580771
  * Engelmann et al. (2022) Severin Engelmann, Chiara Ullstein, Orestis Papakyriakopoulos, and Jens Grossklags. 2022.  What People Think AI Should Infer From Faces. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533080
  * Erdélyi and Erdélyi (2020) Olivia J. Erdélyi and Gábor Erdélyi. 2020.  The AI Liability Puzzle and a Fund-Based Work-Around. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’20)_. ACM.  https://doi.org/10.1145/3375627.3375806
  * Feinberg (2017) Melanie Feinberg. 2017\.  A Design Perspective on Data. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ _(CHI ’17)_. ACM.  https://doi.org/10.1145/3025453.3025837
  * Feuston and Brubaker (2021) Jessica L. Feuston and Jed R. Brubaker. 2021.  Putting Tools in Their Place: The Role of Time and Perspective in Human-AI Collaboration for Qualitative Analysis.  _Proc. ACM Hum.-Comput. Interact._ (Oct. 2021).  https://doi.org/10.1145/3479856
  * Fjeld et al. (2020) Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Srikumar. 2020.  Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI.  _Berkman Klein Center Research Publication_ (2020).  https://doi.org/10.2139/ssrn.3518482
  * Flathmann et al. (2021) Christopher Flathmann, Beau G. Schelble, Rui Zhang, and Nathan J. McNeese. 2021.  Modeling and Guiding the Creation of Ethical Human-AI Teams. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462573
  * Fox et al. (2023) Sarah E. Fox, Samantha Shorey, Esther Y. Kang, Dominique Montiel Valle, and Estefania Rodriguez. 2023\.  Patchwork: The Hidden, Human Labor of AI Integration within Essential Work.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579514
  * Frluckaj et al. (2022) Hana Frluckaj, Laura Dabbish, David Gray Widder, Huilian Sophie Qiu, and James D. Herbsleb. 2022.  Gender and Participation in Open Source Software Development.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555190
  * Garcia and Cifor (2019) Patricia Garcia and Marika Cifor. 2019.  Expanding Our Reflexive Toolbox: Collaborative Possibilities for Examining Socio-Technical Systems Using Duoethnography.  _Proc. ACM Hum.-Comput. Interact._ , Article 190 (Nov. 2019).  https://doi.org/10.1145/3359292
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021\.  Datasheets for Datasets.  _Commun. ACM_ (Nov. 2021).  https://doi.org/10.1145/3458723
  * Gee (2021) Harvey Gee. 2021\.  Reducing Gun Violence with ShotSpotter Gunshot Detection Technology and Community-Based Plans: What Works?  https://scholarsbank.uoregon.edu/xmlui/handle/1794/27170
  * Geoffrey A. Fowler (2021) Geoffrey A. Fowler. 2021\.  _There’s no escape from Facebook, even if you don’t use it_.  The Washington Post.  Retrieved January 2023 from https://www.washingtonpost.com/technology/2021/08/29/facebook-privacy-monopoly/
  * Google (2022) Google. 2022.  _Responsible AI practices_.  Retrieved February 2023 from https://ai.google/responsibilities/responsible-ai-practices/
  * Gray and Suri (2019) Mary L Gray and Siddharth Suri. 2019.  _Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass_.  Eamon Dolan Books.  https://ghostwork.info/
  * Greiffenhagen et al. (2023) Christian Greiffenhagen, Xinzhi Xu, and Stuart Reeves. 2023\.  The Work to Make Facial Recognition Work.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579531
  * Gu et al. (2021) Hongyan Gu, Jingbin Huang, Lauren Hung, and Xiang ’Anthony’ Chen. 2021.  Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists.  _Proc. ACM Hum.-Comput. Interact._ (April 2021).  https://doi.org/10.1145/3449084
  * Ha (2022) You Jeen Ha. 2022\.  South Korean Public Value Coproduction Towards‘AI for Humanity’: A Synergy of Sociocultural Norms and Multistakeholder Deliberation in Bridging the Design and Implementation of National AI Ethics Guidelines. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533091
  * Hall et al. (2022) Kaely Hall, Dong Whi Yoo, Wenrui Zhang, Mehrab Bin Morshed, Vedant Das Swain, Gregory D. Abowd, Munmun De Choudhury, Alex Endert, John Stasko, and Jennifer G Kim. 2022\.  Supporting the Contact Tracing Process with WiFi Location Data: Opportunities and Challenges. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517703
  * Havens et al. (2020) Lucy Havens, Melissa Terras, Benjamin Bach, and Beatrice Alex. 2020.  Situated Data, Situated Systems: A Methodology to Engage with Power Relations in Natural Language Processing Research. In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_. Association for Computational Linguistics.  https://aclanthology.org/2020.gebnlp-1.10
  * He et al. (2023) Gaole He, Lucie Kuiper, and Ujwal Gadiraju. 2023.  Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581025
  * Health Equity & Policy Lab (2022) Health Equity & Policy Lab. 2022\.  _Human Flourishing_.  University of Pennsylvania.  Retrieved January 2023 from https://www.healthequityandpolicylab.com/human-flourishing
  * Heger et al. (2022) Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, and Jennifer Wortman Vaughan. 2022\.  Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555760
  * Henriksen et al. (2021) Anne Henriksen, Simon Enni, and Anja Bechmann. 2021\.  Situated Accountability: Ethical Principles, Certification Standards, and Explanation Methods in Applied AI. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462564
  * Hertzberg et al. (2010) Andrew Hertzberg, Jose Maria Liberti, and Daniel Paravisini. 2010.  Information and incentives inside the firm: Evidence from loan officer rotation.  _The Journal of Finance_ (2010).  https://doi.org/10.1111/j.1540-6261.2010.01553.x
  * Holländer et al. (2021) Kai Holländer, Mark Colley, Enrico Rukzio, and Andreas Butz. 2021.  A Taxonomy of Vulnerable Road Users for HCI Based On A Systematic Literature Review. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445480
  * Holstein et al. (2023) Kenneth Holstein, Maria De-Arteaga, Lakshmi Tumati, and Yanghuidi Cheng. 2023.  Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservables.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579628
  * Hopkins and Booth (2021) Aspen Hopkins and Serena Booth. 2021.  Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462527
  * Houser (2019) Kimberly A Houser. 2019\.  Can AI solve the diversity problem in the tech industry: Mitigating noise and bias in employment decision-making.  _Stan. Tech. L. Rev._ (2019).  https://ssrn.com/abstract=3344751
  * Hsieh et al. (2023) Gary Hsieh, Brett A. Halperin, Evan Schmitz, Yen Nee Chew, and Yuan-Chi Tseng. 2023.  What is in the Cards: Exploring Uses, Patterns, and Trends in Design Cards. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580712
  * Huang and Liem (2022) Han-Yin Huang and Cynthia C. S. Liem. 2022.  Social Inclusion in Curated Contexts: Insights from Museum Practices. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533095
  * Jakesch et al. (2022) Maurice Jakesch, Zana Buçinca, Saleema Amershi, and Alexandra Olteanu. 2022.  How Different Groups Prioritize Ethical Values for Responsible AI. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533097
  * Jia et al. (2022) Chenyan Jia, Alexander Boltz, Angie Zhang, Anqing Chen, and Min Kyung Lee. 2022.  Understanding Effects of Algorithmic vs. Community Label on Perceived Accuracy of Hyper-Partisan Misinformation.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555096
  * Jones et al. (2020) Bernadette Jones, Paula Toko King, Gabrielle Baker, and Tristram Ingham. 2020.  COVID-19, intersectionality, and health equity for indigenous peoples with lived experience of disability.  _American Indian Culture and Research Journal_ (2020).  https://doi.org/10.17953/aicrj.44.2.jones
  * Josie Cox (2023) Josie Cox. 2023\.  _AI anxiety: The workers who fear losing their jobs to artificial intelligence_.  Retrieved June 2023 from https://www.bbc.com/worklife/article/20230418-ai-anxiety-artificial-intelligence-replace-jobs
  * Kamikubo et al. (2023) Rie Kamikubo, Kyungjun Lee, and Hernisa Kacorri. 2023\.  Contributing to Accessibility Datasets: Reflections on Sharing Study Data by Blind People. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581337
  * Kang et al. (2022) Jimoon Kang, June Seop Yoon, and Byungjoo Lee. 2022\.  How AI-Based Training Affected the Performance of Professional Go Players. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517540
  * Kapania et al. (2022) Shivani Kapania, Oliver Siy, Gabe Clapper, Azhagu Meena SP, and Nithya Sambasivan. 2022.  ”Because AI is 100% Right and Safe”: User Attitudes and Sources of AI Authority in India. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517533
  * Kapania et al. (2023) Shivani Kapania, Alex S Taylor, and Ding Wang. 2023.  A Hunt for the Snark: Annotator Diversity in Data Practices. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580645
  * Kasirzadeh and Klein (2021) Atoosa Kasirzadeh and Colin Klein. 2021.  The Ethical Gravity Thesis: Marrian Levels and the Persistence of Bias in Automated Decision-Making Systems. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462606
  * Kasirzadeh and Smart (2021) Atoosa Kasirzadeh and Andrew Smart. 2021.  The Use and Misuse of Counterfactuals in Ethical Machine Learning. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. ACM.  https://doi.org/10.1145/3442188.3445886
  * Kaur et al. (2022) Harmanpreet Kaur, Eytan Adar, Eric Gilbert, and Cliff Lampe. 2022\.  Sensible AI: Re-Imagining Interpretability and Explainability Using Sensemaking Theory. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533135
  * Kemper and Kolain (2022) Carolin Kemper and Michael Kolain. 2022.  K9 Police Robots-Strolling Drones, RoboDogs, or Lethal Weapons?. In _WeRobot2022 conference_.  https://doi.org/10.2139/ssrn.4201692
  * Kim et al. (2023) Minji Kim, Kyungjin Lee, Rajesh Balan, and Youngki Lee. 2023\.  Bubbleu: Exploring Augmented Reality Game Design with Uncertain AI-Based Interaction. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581270
  * Kim et al. (2018) Richard Kim, Max Kleiman-Weiner, Andrés Abeliuk, Edmond Awad, Sohan Dsouza, Joshua B. Tenenbaum, and Iyad Rahwan. 2018.  A Computational Model of Commonsense Moral Decision Making. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’18)_. ACM.  https://doi.org/10.1145/3278721.3278770
  * Kissinger et al. (2021) Henry Kissinger, Eric Schmidt, and Daniel P Huttenlocher. 2021\.  _The Age of AI: And Our Human Future_.  John Murray London.
  * Kizilaslan and Lookman (2017) Atay Kizilaslan and Aziz A Lookman. 2017.  Can Economically Intuitive Factors Improve Ability of Proprietary Algorithms to Predict Defaults of Peer-to-Peer Loans?  _Available at SSRN 2987613_ (2017).  https://doi.org/10.2139/ssrn.2987613
  * Klinova and Korinek (2021) Katya Klinova and Anton Korinek. 2021.  AI and Shared Prosperity. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462619
  * Knowles and Richards (2021) Bran Knowles and John T. Richards. 2021.  The Sanction of Authority: Promoting Public Trust in AI. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. ACM.  https://doi.org/10.1145/3442188.3445890
  * Krafft et al. (2021) P. M. Krafft, Meg Young, Michael Katell, Jennifer E. Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, Corinne Bintz, Daniella Raz, Pa Ousman Jobe, Franziska Putz, Brian Robick, and Bissan Barghouti. 2021\.  An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. ACM.  https://doi.org/10.1145/3442188.3445938
  * Kroll (2021) Joshua A. Kroll. 2021\.  Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. ACM.  https://doi.org/10.1145/3442188.3445937
  * Kukutai et al. (2020) Tahu Kukutai, Stephanie Russo Carroll, and Maggie Walter. 2020.  Indigenous data sovereignty.  (2020).
  * Kumar et al. (2020) I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle A. Friedler. 2020.  Problems with Shapley-Value-Based Explanations as Feature Importance Measures. In _Proceedings of the 37th International Conference on Machine Learning_ _(ICML’20)_. JMLR.org.  https://doi.org/10.48550/arXiv.2002.11097
  * Laker (2022) Benjamin Laker. 2022\.  _Artificial Intelligence Can Help Leaders Drive Global Economy Forward In 2022_.  Forbes.  Retrieved November 2022 from https://www.forbes.com/sites/benjaminlaker/2022/01/19/artificial-intelligence-can-help-leaders-drive-global-economy-forward-in-2022/
  * Lam et al. (2022) Michelle S. Lam, Mitchell L. Gordon, Danaë Metaxa, Jeffrey T. Hancock, James A. Landay, and Michael S. Bernstein. 2022\.  End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555625
  * Lam et al. (2023) Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang, James A. Landay, and Michael S. Bernstein. 2023.  Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581290
  * Langer et al. (2022) Markus Langer, Tim Hunsicker, Tina Feldkamp, Cornelius J. König, and Nina Grgić-Hlača. 2022\.  “Look! It’s a Computer Program! It’s an Algorithm! It’s AI!”: Does Terminology Affect Human Perceptions and Evaluations of Algorithmic Decision-Making Systems?. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517527
  * Lazar et al. (2017a) Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017a.  Chapter 11 - Analyzing qualitative data.  In _Research Methods in Human Computer Interaction_ (second edition ed.), Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser (Eds.). Morgan Kaufmann.  https://doi.org/10.1016/B978-0-12-805390-4.00011-X
  * Lazar et al. (2017b) Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017b.  Chapter 8 - Interviews and focus groups.  In _Research Methods in Human Computer Interaction_ (second edition ed.), Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser (Eds.). Morgan Kaufmann.  https://doi.org/10.1016/B978-0-12-805390-4.00008-X
  * Leavy et al. (2021) Susan Leavy, Eugenia Siapera, and Barry O’Sullivan. 2021\.  Ethical Data Curation for AI: An Approach Based on Feminist Epistemology and Critical Theories of Race. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462598
  * Leckning et al. (2021) Bernard Leckning, Vincent YF He, John R Condon, Tanja Hirvonen, Helen Milroy, and Steven Guthridge. 2021.  Patterns of child protection service involvement by Aboriginal children associated with a higher risk of self-harm in adolescence: A retrospective population cohort study using linked administrative data.  _Child Abuse & Neglect_ (2021).  https://doi.org/10.1016/j.chiabu.2021.104931
  * Lee (2022) Kai-Fu Lee. 2022\.  _AI and the human future: Net positive_.  Economics.  Retrieved November 2022 from https://impact.economist.com/projects/innovation-matters/blogs/ai-and-the-human-future-net-positive/
  * Lee et al. (2019) Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D. Procaccia. 2019.  WeBuildAI: Participatory Framework for Algorithmic Governance.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2019).  https://doi.org/10.1145/3359283
  * Lee et al. (2021) Min Kyung Lee, Ishan Nigam, Angie Zhang, Joel Afriyie, Zhizhen Qin, and Sicun Gao. 2021\.  Participatory Algorithmic Management: Elicitation Methods for Worker Well-Being Models. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’21)_. ACM.  https://doi.org/10.1145/3461702.3462628
  * Lee and Rich (2021) Min Kyung Lee and Katherine Rich. 2021.  Who Is Included in Human Perceptions of AI?: Trust and Perceived Fairness around Healthcare AI and Cultural Mistrust. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM, Article 138.  https://doi.org/10.1145/3411764.3445570
  * Lee et al. (2023b) Patrick Yung Kang Lee, Ning F. Ma, Ig-Jae Kim, and Dongwook Yoon. 2023b.  Speculating on Risks of AI Clones to Selfhood and Relationships: Doppelganger-Phobia, Identity Fragmentation, and Living Memories.  _Proc. ACM Hum.-Comput. Interact._ (April 2023).  https://doi.org/10.1145/3579524
  * Lee et al. (2023a) Sunok Lee, Dasom Choi, Minha Lee, Jonghak Choi, and Sangsu Lee. 2023a.  Fostering Youth’s Critical Thinking Competency About AI through Exhibition. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581159
  * Leonardo Nicoletti and Dina Bass (2023) Leonardo Nicoletti and Dina Bass. 2023\.  _HUMANS ARE BIASED. GENERATIVE AI IS EVEN WORSE_.  Retrieved June 2023 from https://www.bloomberg.com/graphics/2023-generative-ai-bias/
  * Lewicki et al. (2023) Kornel Lewicki, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2023.  Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service”. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581463
  * Li et al. (2023) Rena Li, Sara Kingsley, Chelsea Fan, Proteeti Sinha, Nora Wai, Jaimie Lee, Hong Shen, Motahhare Eslami, and Jason Hong. 2023.  Participation and Division of Labor in User-Driven Algorithm Audits: How Do Everyday Users Work Together to Surface Algorithmic Harms?. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3582074
  * Li et al. (2022) Yunqi Li, Hanxiong Chen, Shuyuan Xu, Yingqiang Ge, Juntao Tan, Shuchang Liu, and Yongfeng Zhang. 2022.  Fairness in Recommendation: A Survey.  https://doi.org/10.48550/ARXIV.2205.13619
  * Liao and Sundar (2022) Q.Vera Liao and S. Shyam Sundar. 2022.  Designing for Responsible Trust in AI Systems: A Communication Perspective. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3533182
  * Liao et al. (2020) Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020\.  Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM.  https://doi.org/10.1145/3313831.3376590
  * Liao et al. (2023) Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, and Jennifer Wortman Vaughan. 2023.  Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580652
  * Liao and Varshney (2021) Q. Vera Liao and Kush R. Varshney. 2021.  Human-Centered Explainable AI (XAI): From Algorithms to User Experiences.  (2021).  https://doi.org/10.48550/ARXIV.2110.10790
  * Lima et al. (2021) Gabriel Lima, Nina Grgić-Hlača, and Meeyoung Cha. 2021\.  Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445260
  * Lima et al. (2023) Gabriel Lima, Nina Grgić-Hlača, and Meeyoung Cha. 2023\.  Blaming Humans and Machines: What Shapes People’s Reactions to Algorithmic Harm. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3580953
  * Lima et al. (2022) Gabriel Lima, Nina Grgić-Hlača, Jin Keun Jeong, and Meeyoung Cha. 2022.  The Conflict Between Explainable and Accountable Decision-Making Algorithms. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3534628
  * Lin and Van Brummelen (2021) Phoebe Lin and Jessica Van Brummelen. 2021.  Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445377
  * Linxen et al. (2021) Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021.  How WEIRD is CHI?. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445488
  * Long et al. (2021) Duri Long, Takeria Blunt, and Brian Magerko. 2021.  Co-Designing AI Literacy Exhibits for Informal Learning Spaces.  _Proc. ACM Hum.-Comput. Interact._ (Oct. 2021).  https://doi.org/10.1145/3476034
  * Long et al. (2022) Duri Long, Anthony Teachey, and Brian Magerko. 2022\.  Family Learning Talk in AI Literacy Learning Activities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3502091
  * Lundberg and Lee (2017) Scott M. Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ _(NIPS’17)_. Curran Associates Inc.  https://dl.acm.org/doi/pdf/10.5555/3295222.3295230
  * Lyn (2020) Alexandra Lyn. 2020\.  Risky Business: Artificial Intelligence and Risk Assessments in Sentencing and Bail Procedures in the United States.  _Available at SSRN 3831441_ (2020).  https://doi.org/10.2139/ssrn.3831441
  * Lyons et al. (2021) Henrietta Lyons, Eduardo Velloso, and Tim Miller. 2021\.  Conceptualising Contestability: Perspectives on Contesting Algorithmic Decisions.  _Proc. ACM Hum.-Comput. Interact._ (April 2021).  https://doi.org/10.1145/3449180
  * Maas (2018) Matthijs M. Maas. 2018\.  Regulating for ’Normal AI Accidents’: Operational Lessons for the Responsible Governance of Artificial Intelligence Deployment. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’18)_. ACM.  https://doi.org/10.1145/3278721.3278766
  * Marley (2019) Tennille L Marley. 2019\.  Indigenous data sovereignty: University institutional review board policies and guidelines and research with American Indian and Alaska Native communities.  _American Behavioral Scientist_ (2019).
  * McDonald and Pan (2020) Nora McDonald and Shimei Pan. 2020.  Intersectional AI: A Study of How Information Science Students Think about Ethics and Their Impact.  _Proc. ACM Hum.-Comput. Interact._ (Oct. 2020).  https://doi.org/10.1145/3415218
  * McNaney et al. (2018) Roisin McNaney, John Vines, Andy Dow, Harry Robinson, Heather Robinson, Kate McDonald, Leslie Brown, Peter Santer, Don Murray, Janice Murray, David Green, and Peter Wright. 2018\.  Enabling the Participation of People with Parkinson’s and Their Caregivers in Co-Inquiry around Collectivist Health Technologies. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ _(CHI ’18)_. ACM.  https://doi.org/10.1145/3173574.3174065
  * Mentis et al. (2016) Helena M. Mentis, Ahmed Rahim, and Pierre Theodore. 2016\.  Crafting the Image in Surgical Telemedicine. In _Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing_ _(CSCW ’16)_. ACM.  https://doi.org/10.1145/2818048.2819978
  * Mhasawade et al. (2021) Vishwali Mhasawade, Yuan Zhao, and Rumi Chunara. 2021\.  Machine learning and algorithmic fairness in public and population health.  _Nature Machine Intelligence_ (2021).  https://doi.org/10.1038/s42256-021-00373-4
  * Miceli et al. (2022) Milagros Miceli, Tianling Yang, Adriana Alvarado Garcia, Julian Posada, Sonja Mei Wang, Marc Pohl, and Alex Hanna. 2022.  Documenting Data Production Processes: A Participatory Approach for Data Work.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555623
  * Microsoft (2022) Microsoft. 2022\.  _Responsible AI_.  Retrieved February 2023 from https://www.microsoft.com/en-us/ai/responsible-ai
  * Mlynar et al. (2022) Jakub Mlynar, Farzaneh Bahrami, André Ourednik, Nico Mutzner, Himanshu Verma, and Hamed Alavi. 2022\.  AI beyond Deus Ex Machina – Reimagining Intelligence in Future Cities with Urban Experts. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517502
  * Moitra et al. (2022) Aparna Moitra, Dennis Wagenaar, Manveer Kalirai, Syed Ishtiaque Ahmed, and Robert Soden. 2022\.  AI and Disaster Risk: A Practitioner Perspective.  _Proc. ACM Hum.-Comput. Interact._ (Nov. 2022).  https://doi.org/10.1145/3555163
  * Moore et al. (2023) Steven Moore, Q. Vera Liao, and Hariharan Subramonyam. 2023\.  FAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. ACM.  https://doi.org/10.1145/3544548.3581242
  * Moreschi et al. (2020) Bruno Moreschi, Gabriel Pereira, and Fabio G Cozman. 2020\.  The Brazilian Workers in Amazon Mechanical Turk: dreams and realities of ghost workers.  _Contracampo_ (2020).  https://doi.org/10.22409/contracampo.v39i1.38252
  * Morrison et al. (2021) Cecily Morrison, Edward Cutrell, Martin Grayson, Anja Thieme, Alex Taylor, Geert Roumen, Camilla Longden, Sebastian Tschiatschek, Rita Faia Marques, and Abigail Sellen. 2021\.  Social Sensemaking with AI: Designing an Open-Ended AI Experience with a Blind Child. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445290
  * Muller et al. (2022) Michael Muller, Plamen Agelov, Shion Guha, Marina Kogan, Gina Neff, Nuria Oliver, Manuel Gomez Rodriguez, and Adrian Weller. 2022.  _NeurIPS 2021 Workshop Proposal: Human Centered AI_.  Retrieved March 2023 from https://sites.google.com/view/hcai-human-centered-ai-neurips/home
  * Muller and Strohmayer (2022) Michael Muller and Angelika Strohmayer. 2022.  Forgetting Practices in the Data Sciences. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491102.3517644