# “A cold, technical decision-maker”: Can AI provide explainability,
negotiability, and humanity?

Allison Woodruff  Google<EMAIL_ADDRESS>,  Yasmin Asare Anderson  Ipsos
<EMAIL_ADDRESS>,  Katherine Jameson Armstrong  Ipsos
<EMAIL_ADDRESS>,  Marina Gkiza  Ipsos<EMAIL_ADDRESS>,  Jay Jennings  PixelInsight<EMAIL_ADDRESS>,  Christopher Moessner
Ipsos<EMAIL_ADDRESS>,  Fernanda Viégas  Google
<EMAIL_ADDRESS>,  Martin Wattenberg  Google<EMAIL_ADDRESS>,
Lynette Webb  Google<EMAIL_ADDRESS>,  Fabian Wrede  Google
<EMAIL_ADDRESS>and  Patrick Gage Kelley  Google<EMAIL_ADDRESS>
(2020)

###### Abstract.

Algorithmic systems are increasingly deployed to make decisions in many areas
of people’s lives. The shift from human to algorithmic decision-making has
been accompanied by concern about potentially opaque decisions that are not
aligned with social values, as well as proposed remedies such as
explainability. We present results of a qualitative study of algorithmic
decision-making, comprised of five workshops conducted with a total of 60
participants in Finland, Germany, the United Kingdom, and the United States.
We invited participants to reason about decision-making qualities such as
explainability and accuracy in a variety of domains. Participants viewed AI as
a decision-maker that follows rigid criteria and performs mechanical tasks
well, but is largely incapable of subjective or morally complex judgments. We
discuss participants’ consideration of humanity in decision-making, and
introduce the concept of ‘negotiability’, the ability to go beyond formal
criteria and work flexibly around the system.

accountability, algorithmic decision-making, artificial intelligence,
explainability, interpretability

††copyright: acmcopyright††journalyear: 2020††doi: ††ccs: Social and
professional topics Computing / technology policy††ccs: Human-centered
computing Human computer interaction (HCI)††ccs: Computing methodologies
Artificial intelligence

##  1\. Introduction

Motivated by expectations of improved efficiency, efficacy, consistency, and
objectivity, algorithmic systems are increasingly deployed to make decisions
in many areas of people’s lives, from finance, employment, medicine and
criminal justice to advertising, media recommendations and more (Citron, 2007;
Edwards and Veale, 2017; Pasquale, 2015; Selbst and Barocas, 2018). This shift
from individual or organizational human decision-making to algorithmic
decision-making has been accompanied by concern that important decisions are
being made in new ways that are often mysterious, and that current oversight
does not adequately ensure social values such as fairness, privacy, and safety
(Citron and Pasquale, 2014; Kroll et al., 2016; Pasquale, 2015; Selbst and
Barocas, 2018).

A variety of approaches have been proposed in technical, social science,
legal, and regulatory communities to address this issue and provide
accountability for algorithmic systems. However, the best path forward is as
yet unclear. For example, increased transparency or explainability are often
proposed as mechanisms to provide accountability, and frequently feature in
regulatory and tech ethics requirements or discussions (Fjeld et al., 2020;
Goodman and Flaxman, 2017; on Artificial Intelligence, 2019; Kaminski, 2019b;
Wachter et al., 2017). While these have significant potential upside, concerns
have been raised that such mechanisms are unlikely to be sufficiently
effective or satisfactory, due to fundamental challenges in producing
meaningful human explanations of the operation of complex machines (Ananny and
Crawford, 2016; Edwards and Veale, 2017; Kroll et al., 2016; Sendak et al.,
2020). Further, in some contexts practical constraints may require that
transparency or explainability be traded off against qualities such as a
system’s accuracy or its ability to resist adversarial attacks (Burrell, 2016;
Kroll et al., 2016; London, 2019).

Limited understanding of public attitudes towards algorithmic decision-making
makes it difficult to include public needs and desires in balancing these
complex tradeoffs. To inform policy and technical approaches to algorithmic
decision-making, we present results from a qualitative study comprised of five
workshops conducted with a total of 60 participants in Finland, Germany, the
United Kingdom, and the United States. Our contributions are as follows:

  * •

We present results from a qualitative study of algorithmic decision-making
conducted in four countries

  * •

We identify key desirable qualities of decision-makers and discuss how AI is
perceived as able or not able to provide them

  * •

We discuss how algorithmic decision-making is intended to regularize
institutional processes and make them more efficient, but in so doing is
perceived as reducing flexibility, compassion, and other qualities desired by
recipients of a decision

  * •

We consider how our findings inform potential design of algorithmic decision-
making systems, as well as allocation of responsibilities between AI and
humans in human-in-the-loop approaches

In the remainder of the paper, we review relevant background, describe our
methodology, present and discuss our findings, and conclude.

##  2\. Background

###  2.1. Algorithmic Decision-Making Systems, Artificial Intelligence, and
Machine Learning

In recent decades, physical and digital systems across a range of sectors have
increasingly transitioned to automation for a range of functions including
decision-making,111In this article we use the term algorithmic decision-making
systems, following (Diakopoulos, 2016; Kaminski, 2019a) and others. The term
automated decision-making is often used synonymously, commonly describing
decision-making functions that are carried out in part or in full by a device
or machine (Parasuraman et al., 2000). in hopes of improving efficiency and
performance, and regularizing decisions by removing human and procedural
variability (Eubanks, 2017; Immonen et al., 2018; Ostrom et al., 2002). The
significant impact, risks, and challenges of this transition have been an
object of substantial study, raising concerns related to fairness, privacy,
safety, and more as algorithmic systems increasingly shape information people
are exposed to as well as influence decisions about employment, finances, and
other opportunities (Barocas and Selbst, 2016; boyd and Crawford, 2012; Citron
and Pasquale, 2014; Cottom, 2015; Eubanks, 2017; Munoz et al., 2016; Fourcade
and Healy, 2013; Friedman and Nissenbaum, 1996; Commission, 2016; Gillespie,
2014; Introna and Nissenbaum, 2000; Lustig et al., 2016; O’Neil, 2016;
Pasquale, 2015; Sandvig et al., 2015).

Artificial intelligence (AI) underlies many algorithmic decision-making
systems. Machine learning is a particular type of artificial intelligence that
has the ability to learn and improve over time, and it is an increasingly
common choice for many applications including decision-making. Accordingly, in
our study we focused primarily although not exclusively on algorithmic
decision-making systems that use different types of AI, including machine
learning.

###  2.2. Accountability, Transparency, Explainability, and Interpretability

In search of mechanisms to detect and address social ills that may result from
the use of algorithmic decision-making systems, and building on the
expectation that insight into a system’s production, operation, and effects
can enable accountability, many regulators, civil society groups, and scholars
have called for a cluster of related ideas encompassing transparency,
explainability, and interpretability.222These notions are strongly related and
these terms are often used interchangeably although some scholars and
disciplines also distinguish their meanings in various ways (Biran and Cotton,
2017; Gilpin et al., 2018; Kluttz et al., 2020; Lipton, 2016; Miller, 2019;
Rader et al., 2018). For our purposes in this article, we define transparency
as broad visibility into the construction, behavior, and processes of a
system; explainability as the ability to provide human understandable reasons
for individual decisions or a set of decisions made by an algorithmic system;
and interpretability as the ability of a given model to explain how it
calculates specific decisions. Consideration of these approaches dates to the
1970s with the recognition of the need to explain the decisions of expert
systems, with a resurgence of this topic in recent years as contemporary
machine learning models have become both pervasive and increasingly complex
(Abdul et al., 2018; Biran and Cotton, 2017; Gilpin et al., 2018; Miller,
2019; Selbst and Barocas, 2018). Due to its often impenetrable and opaque
nature, machine learning, particularly deep learning, has generated concern
and attracted heavy critique as it can be difficult to interrogate its
reasoning and remediate potential injustices or errors (Burrell, 2016;
Gillespie, 2014). Interpretability is therefore a topic of substantial debate
at the highest levels of machine learning expertise (Caruana et al., 2017;
Rudin, 2019), with many arguing there is an inherent tension between a
system’s accuracy and its interpretability, but others arguing it is possible
to finesse this technical challenge with methods such as creating human-
understandable models that are proxies for more complex models, intentionally
designing models that are more readily interpretable, or conducting “outside
the box” analyses of system design and behavior (Breiman et al., 2001; Gilpin
et al., 2018; Gupta et al., 2016; Lipton, 2016; Ribeiro et al., 2016; Selbst
and Barocas, 2018). A number of scholars further argue that making systems
more understandable is at odds with other values such as protecting systems
from adversarial attacks or seamless design (Burrell, 2016; Eslami et al.,
2019; Kroll et al., 2016), and even when there is not an inherent tradeoff, it
can distract from more important priorities such as empirical (non-
explanatory) verification of system performance (Ananny and Crawford, 2016;
Edwards and Veale, 2017; Lipton, 2016; London, 2019; Sendak et al., 2020).
More fundamentally, there is a growing recognition that transparency,
explainability, and interpretability are often intended as a means to an end,
for example laying the groundwork to challenge a decision or to ensure
fairness, yet they provide inadequate support for such underlying objectives
(Kluttz et al., 2020; Rader et al., 2018; Woodruff, 2019). As a result, a
number of approaches have been articulated in recent years that hew more
closely to direct needs of users and society. Counterfactuals specify
conditions under which a different decision would have been made (Karimi et
al., 2020; Russell, 2019; Wachter et al., 2018). Contestability provides the
ability to challenge decisions, and allows domain experts to collaborate with
and correct predictive algorithms (Hirsch et al., 2017; Kluttz et al., 2020;
Vaccaro et al., 2019). Recourse considers the ability to correct faulty or
incomplete data, and also identify clear actions an individual can take in the
future to get a different outcome (Ustun et al., 2019; Venkatasubramanian and
Alfano, 2020).

###  2.3. Algorithmic Literacy and Explanation Design

Beyond the challenges described above, provision of good explanations is
further complicated by the fact that end users often have fundamental
questions or misconceptions about the operation of algorithmic systems
(Bucher, 2017; Eslami et al., 2015; Rader and Gray, 2015; Ur et al., 2012;
Warshaw et al., 2016), and the construction of useful explanations remains
elusive (Nunes and Jannach, 2017; Rader et al., 2018). These challenges
notwithstanding, a number of studies have shown that explanations have
potential to increase users’ trust and satisfaction (Biran and Cotton, 2017;
Nunes and Jannach, 2017), and they remain an object of great interest with a
number of efforts pursuing best practices for their design (Balog and
Radlinski, 2020; Cai et al., 2019; Eslami et al., 2016; Herman, 2017; Miller,
2019; Nunes and Jannach, 2017; Poursabzi-Sangdeh et al., 2018; Wang et al.,
2019).

###  2.4. Public Perception of AI and Algorithmic Decision-Making

A number of studies have explored public perception of AI, for example, survey
research (Northstar, 2017; Capital, 2019; Cave et al., 2019; Edelman, 2019;
Ipsos, 2019; Kelley et al., 2020; Mozilla, 2019; University and Gallup, 2018;
West, 2018a; Zhang and Dafoe, 2019), sentiment analysis (Fast and Horvitz,
2017; Garvey and Maskal, 2019), and narrative analysis (Cave et al., 2018),
most often in Western, English-speaking contexts. AI is viewed as likely to
have significant impact on the future and overall expectation is often
positive. At the same time, AI is neither interpreted as exclusively
beneficial nor exclusively disadvantageous, and public response is often
ambivalent (Northstar, 2017; Capital, 2019; Kelley et al., 2020). Specific
concerns have been expressed regarding social issues, such as privacy, job
loss, AI benefiting the wealthy and harming the poor, and AI increasing social
isolation and reducing human capabilities (Edelman, 2019; Kelley et al.,
2020). Resonating with this interest in the impact of AI on social issues,
some public perception research has focused specifically on its use in
decision-making, e.g. a survey in the United Kingdom found that people were
not very familiar with the use of automated decision systems, and indicated
low levels of support and high levels of opposition to them (Balaram et al.,
2018).

###  2.5. Workshop as Method

In taking up a workshop format, we draw on traditions within and just beyond
HCI. This includes programs of participatory action research, participatory
design, and living labs (DiSalvo et al., 2008, 2011; Jungk and Müllert, 1987;
Rosner et al., 2016). Within the context of HCI and design research, workshop
approaches often seek to invite members of the public to engage with practices
of design while exploring values and beliefs around technology with each
other, positing alternative techniques and outcomes. We also draw inspiration
from Citizens’ Juries, a form of deliberative democracy dating from the 1970s
that brings together individuals from different backgrounds to spend several
days hearing expert evidence and considering a public policy question (Armour,
1995; Balaram et al., 2018; Office, 2019; Center, 2004; Oswald, 2019). Two
deliberative democracy studies run in the United Kingdom that used Citizens’
Juries methodology are closest in nature to our own work. One explored under
what circumstances participants would find the use of automated decision-
making appropriate, finding they valued accountability and ethical development
processes (Singh, 2019). Another focused on how participants prioritized
accuracy versus explainability for AI decision-making, reporting that context
mattered and accuracy outweighed explainability in healthcare but not other
non-medical situations presented (Oswald, 2019; Singh, 2019). We build on,
extend, and broaden this previous work by exploring complementary research
objectives; exploring a wider range of domains, tradeoffs, and types of
transparency; and expanding the scope to three additional countries beyond the
United Kingdom.

##  3\. Methodology

In order to better understand perception of algorithmic decision-making
systems, we partnered with our co-authors from [_anonymized market research
firm_] to conduct five workshops with 60 participants in major cities in
Finland, Germany, the United Kingdom, and the United States. These countries
were selected due to European and United States regulatory interest in
accountability of AI systems (on Artificial Intelligence, 2019; Kaminski,
2019b) and pre-existing literature on perceptions of AI or algorithmic
decision-making that we could build on and triangulate with, for example
(Balaram et al., 2018; Cave et al., 2019; Office, 2019; Kelley et al., 2020;
Zhang and Dafoe, 2019). Partnering with an international market research firm
ensured professional quality recruiting and moderation across multiple
countries, languages, and cultural contexts. Our research objectives were to
learn more about the following:

  * •

What qualities do members of the public want in algorithmic decision-making
systems?

  * •

To what extent do members of the public value or prioritize explainability in
particular, for what reasons?

  * •

What qualities do members of the public believe an AI-based algorithmic
decision-making system has, or is capable of? In what circumstances do they
see its use as beneficial?

###  3.1. Participants

We recruited 12 participants for each of five workshops (two in Germany and
one each in the other countries) through professional recruiting firms in each
country. In each region, participants were recruited to represent a diverse
spread across the following: age, gender, income, life stage, socioeconomic
status, educational level, digital literacy, and exposure to AI. They were
recruited to represent a mix of ethnicity representative of the region.
Participants were compensated for their time, at or above the living wage for
their area.

###  3.2. Workshops

Each group participated in a 3-hour workshop at third party facilities in each
country during November and December 2019. Two moderators were in the
discussion rooms with the 12 participants, and participants were aware that
several researchers and staff were in observation rooms or viewing remotely.
All moderators were from a professional qualitative research background and
were fluent in the local language. In non-English speaking countries (Finland
and Germany), simultaneous translators were present in the observation rooms
and their translations were broadcast to remote viewers and captured on
recordings so that the researchers and staff observing the sessions could hear
the discussion and pass questions and input to the moderators in real-time.
Both moderators and simultaneous translators received specific training in the
objectives and content of the workshops before the sessions. Refreshments were
available throughout the sessions. Workshops followed this agenda (times are
approximate):

  * •

Introductions and warm-up activity sharing experiences with human decision-
making – _10 minutes_

  * •

Presentation about AI and machine learning, their use in decision-making,
tradeoffs, and Q/A – _60 minutes_

  * •

First scenario (in two breakout groups, with six participants each) – _40
minutes_

  * •

Break – _10 minutes_

  * •

Second scenario (in same breakout groups as the first scenario) – _30 minutes_

  * •

Reconvene and summarize – _30 minutes_

The scenarios described hypothetical situations in which organizations were
considering implementing algorithmic decision-making and were choosing among
three systems with different characteristics (see Tables 3 and 4 in Appendix
A). For each scenario, we began by sharing a handout describing it and asking
participants to take a few minutes to read and consider it. We then invited
participants to share their initial reactions and facilitated a group
discussion of the scenario. Midway through the discussion, we asked
participants to fill out a handout and then vote in group discussion on which
system they preferred.333While most of the procedure was consistent across the
different countries, the voting process was somewhat variable as the study
evolved, for example, in some cases participants voted both midway and at the
end of the discussion. As a result, we do not report voting data here,
although it was consistent with the qualitative findings we report. The
scenarios we discussed represented a wide range of domains and we encouraged
discussion of other domains, so the discussion often branched out to other
areas in which algorithmic decision-making might be used. After completing
both scenarios, all participants joined back together and we concluded the
workshop with a broad group discussion where participants reflected on ideas
that had emerged throughout the session and synthesized and summarized their
opinions. If participants requested, they were informed of the sponsor of the
research at the end of the session. During the workshops, we took care to
encourage collaborative interpretation, problem-solving, and discussion among
participants, and to make space for all participants to share their ideas and
opinions.

After each session, moderators, researchers, and other team members held a
debriefing session to discuss early observations and themes, as well as
potential procedural refinements for future sessions such as timing
adjustments. Detailed notes were taken on observations and themes for each
country as the study progressed. All sessions were video- and audio-recorded
and transcribed in English. For the United Kingdom and the United States, the
transcripts were then compared against the original recordings for accuracy,
and for Finnish and German the transcripts were reviewed and compared against
the simultaneous translations for accuracy.

###  3.3. Material Development and Translation

We developed materials designed to explore our research objectives, drawing
heavily for inspiration on Citizens’ Juries work previously conducted in the
United Kingdom (Balaram et al., 2018; Office, 2019; Oswald, 2019; Singh,
2019). This included a slide deck and accompanying script to provide
participants relevant background on AI and machine learning; descriptions of
rule-based systems, conventional machine learning, and deep learning; the use
of AI and machine learning in decision-making; and tradeoffs between accuracy
and various forms of transparency. In some sessions we also showed a short
introductory video about machine learning available on the internet. We also
developed four hypothetical scenarios chosen to represent decision-making in
topical and diverse contexts: stroke diagnosis, job applicant screening,
credit scoring, and law enforcement, as shown in Table 3 (Appendix A). For
each scenario, participants were presented with three different systems which
varied in a number of ways, similar to (Office, 2019; Oswald, 2019), as shown
in Table 4 (Appendix A). The primary differences centered on accuracy versus a
form of transparency (explainability, appeal, or oversight), which we
emphasized due to the centrality of this discussion in current policy and
academic discussions. In some cases the magnitude and nature of the tradeoff
were exaggerated in order to prompt useful discussion, rather than strictly
adhering to specific current performance levels, again similar to (Office,
2019; Oswald, 2019). Finally, we developed a moderator discussion guide to use
with these materials. The materials were developed in English, and after they
were complete, our market research partner translated materials into Finnish
and German.

###  3.4. Analysis

In our analysis, we used a general inductive approach (Thomas, 2006), which
relies on detailed readings of raw data to derive themes relevant to
evaluation objectives. In our case, the primary evaluation objectives were
those listed at the beginning of this section, chosen to inform technical and
policy approaches to algorithmic decision-making. After the workshops were
over, one member of our research team reviewed all transcripts and prepared
participant precis and breakout group precis summarizing the main points for
each individual and breakout group, and then prepared a spreadsheet
summarizing the key ideas raised by each participant and breakout group. This
was work done in active consultation with another member of the research team
and built on the collective debriefs conducted by the broader team after each
workshop. Throughout, we continued to identify emergent themes (Beyer and
Holtzblatt, 1997), iteratively revising and refining categories. In keeping
with the general inductive approach, our analytic process yielded a small
number of summary categories. We summarize those most relevant to our
evaluation objectives in Section 4.

###  3.5. Limitations

We note several limitations of our methodology that should be considered when
interpreting this work. First, it carries with it the standard issues
attendant with qualitative methodologies and group interviews. While it yields
rich insight, one must be cautious in generalizing beyond those studied or
drawing conclusions across countries (Armour, 1995). Second, we asked
participants to consider hypothetical scenarios that most did not perceive as
common in their daily lives at the current time. They may have had different
reactions if they had been speaking to commonplace experiences, rather than
extrapolating from more limited experiences in their current lives. Third, we
studied a small number of scenarios, often with high stakes import. Different
scenarios might yield varying results or further refine the insights we share
here. Fourth, while we were heavily inspired by the Citizens’ Juries approach
(Armour, 1995; Center, 2004), our workshops did not follow all aspects of this
approach, most notably in that they were significantly shorter and more
lightweight engagements (three hours in our case, versus several days for
Citizens’ Juries) with less instructional content, no expert witnesses, and
less discussion. In this way, while our data represents participants who were
provided substantial information, our data does not represent what members of
the public might think with more extensive intervention. Fifth, the discussion
allowed open-ended conversation but also at some point required participants
to express a clear choice for each scenario based largely on the tradeoff
between explainability and accuracy. While Citizens’ Juries typically require
jurors to make such a clear choice which prompts useful insight and discussion
(Oswald, 2019), it also focuses some of the discussion on particular issues.
To mitigate this, both in moderation and analysis, we took a wide view beyond
these specific choices. Sixth, while members of the research team and/or
market research partner team have experience conducting research in all
markets studied, most reside in the United States or the United Kingdom. We
have worked to minimize the risk of misinterpretation by collaboration and
discussion with in-country colleagues but recognize that our interpretations
may lack context or nuance that would have been more readily available to
local residents.

##  4\. Findings

In this section, we describe the main findings that emerged from our analysis.
We begin by describing how participants perceived algorithmic decision-making
systems as offering efficiency and performance advantages in some
circumstances but potentially making substantial errors in others, as well
posing a potential threat to dignity and autonomy. We discuss the perceived
opportunity for explanations and human intervention to ameliorate some of
these concerns. We then summarize the desirable qualities of decision-makers,
whether AI or human, and close this section by discussing situational factors
that influence tradeoffs among those qualities and influence the allocation of
responsibility to algorithmic decision-making systems versus humans.

###  4.1. AI as an emotionless, mechanical decision-maker

Participants recognized strong advantages of algorithmic decision-making in
some circumstances, such as speed, scale, or improved accuracy. Participants
perceived AI as performing objective, mechanical tasks well, and characterized
it as thinking in black-and-white terms and following rigid criteria.
Correspondingly, they often viewed AI as incapable of subjective or morally
complex judgments.444Our discussions with participants focused on AI-based
algorithmic decision-making systems, with a broad range of functionality from
basic rule-based systems to advanced deep learning systems. Accordingly, when
participants speak of AI they typically refer to it in the context of an AI-
based algorithmic decision-making system.

  * “We are talking about 3 systems that have no gut instinct algorithm or anything similar, but which simply make decisions based on clear structures.”  _–Germany A08_ 555For ease of reading, we have followed editing conventions consistent with applied social science research practices as described in (Corden and Sainsbury, 2006). Specifically, we edited quotes to remove content such as filler words and false starts, and in some cases we re-punctuated. We use ellipses to indicate substantial omissions. We identify each participant by country name plus a numerical identifier from 01 to 12 for each workshop. For Germany, we distinguish between the two workshops by prepending A or B to the numerical identifier.

  * _Germany A11_ : “This is all theory, and life is everything in between! Artificial Intelligence has black or white. I mean, when it comes to the rules. And that’s where we also, as people, can make different decisions based on the situation.”
_Germany A07_ : “We’ve heard about a system which, in the personnel field, can
do the work for us and decide within two or three minutes whether the
applicant is good or not… But then there’s a certain ‘gut feeling’ involved,
there’s not just black or white or grey, but also something in between, and
you have to look at whether somebody as a whole actually fits, and the system
can’t do that, because it only looks at ‘studies’ yes or no, ‘foreign
languages’ yes or no, but there are always some things in between that a
computer, from my perspective, can’t evaluate.”

Beyond perceiving AI as well-suited to making straightforward decisions,
participants sometimes saw AI as appropriate for complex technical tasks, such
as autopilot for airplanes or cars, with some variation in preferred tradeoffs
between safety (accuracy) versus explanation.

  * “I would imagine it could apply to areas of autonomous driving or flying or similar. Complex things that are technically-based where I, as a user, would like to have the highest amount of safety [and would not require explanation].”  _–Germany A09_

  * “I wanted to add something regarding trust. For me, it depends on how difficult the task is… When I know the task is quite simple and it has received much data, then I think, I don’t care how it does it, but okay, it has done it. But when it is a more complex task, then I would rather like to know how it works. Otherwise I wouldn’t feel too safe.”  _–Germany B05_

By contrast, AI was seen as poorly suited to tasks that dealt with non-
specific factors, such as complex moral judgments or assessments of intangible
factors, consistent with an argument made in legal scholarship that policies
that require the exercise of human discretion cannot be automated (Citron,
2007). Participants in a study in the United Kingdom believed that machines
operate with logic and rules that cannot form the basis of ethical decisions
(MORI, 2017); here we develop this argument further and provide evidence of it
in other countries.

  * “[AI] doesn’t allow for those things that can’t be defined by specifics.”  _–United Kingdom 12_

  * “In these kinds of humane decision-making things, where more complicated, abstract things are weighed… justice, fairness and such… something much higher than just crunching numbers, for that artificial intelligence is not suitable.”  _–Finland 11_

  * “I’d say it depends on the role ethics play in the situation. So, when it comes to hospital decisions, we very likely all agree that we want to save lives, everybody should be as healthy as possible and therefore, if technology can help us with that, we might be happy to take it. Whereas with other questions, where we are not even sure how we should act as humans, questions where we don’t even agree on how much power the police should actually have, in those cases AI might be a tool that some reject as a matter of principle. And these are no longer yes or no questions, they are far more complicated because they are ethical, legal and philosophical. And when even we as humans can’t even agree on that, there’s no point in bringing AI into it.”  _–Germany B04_

Many participants emphasized that AI does not take into account soft factors
(qualities that are intangible or difficult to specify), leading to strong
skepticism that AI could make good or fair decisions for tasks where such
factors are important, consistent with (Singh, 2019).

  * “If I was an applicant myself, then in my opinion, it would be unfair for me if only my knowledge was evaluated, and not that ‘that’s a really good person, matches this team exactly, it’s exactly who I’m looking for.’ Can it do that, the artificial intelligence, can it know what type of people I like and with whom I like to work? … I don’t believe an artificial intelligence can know whether I’m suitable for some job or not.”  _–Finland 04_

  * _Germany B11_ : “[Even with a hip replacement], there are still certain things to weigh… there are still soft factors involved and a human assessment of whether this is still justified for someone.”
…
_Germany B01_ : “Even if the computer says you need a new hip, it always
depends on how much a person is suffering. The individual physical strain, how
the body works, something does not fit together, whether you got an
inflammation. This is individually different. It always depends on the
individual level of suffering and the future intelligence cannot decide this
at all.”
…
_Germany B07_ : “Many also managed to overcome cancer based on their personal
willpower. A machine won’t be able to tell how strong a human’s will is.”

AI was sometimes characterized as cold and emotionless, consistent with
(Balaram et al., 2018; Kelley et al., 2020), and lacking human qualities such
as ‘gut instinct.’

  * _Finland 04_ : “I’m thinking about whether it becomes so-called too intelligent, that it becomes a cold, technical decision-maker… Then certain issues, where there should be the humane aspect, can it then do such things?”
…
_Finland 06_ : “I was reminded of the old saying: ‘Good servant but poor
master.”’

###  4.2. Mysterious Dictates

Concerns have long been raised in both scholarly and fictional literature that
algorithmic decision-making and automated systems can undermine human autonomy
if powerful institutions issue consequential decisions that cannot be
understood or appealed (Cave et al., 2019; Edwards and Veale, 2017; Kaminski,
2019a; West, 2018b; Selbst and Barocas, 2018). Autonomy is considered
essential to human wellbeing, making the notion of such unfettered power both
frightening and dehumanizing (Cave et al., 2018; Eubanks, 2017; Kelley et al.,
2020; Kluttz et al., 2020; Selbst and Barocas, 2018; Venkatasubramanian and
Alfano, 2020). Participants often brought up these issues. For example, the
notion that deep learning systems learn on their own, developing their own
unseen rules and assumptions, was sometimes experienced as disturbing, eerie
or even scary.

  * “Deep learning concerns me because it seems totally computer led and based, and there’s no human intervention. Although they all have ultimately some human behind it… But I think the deep-learning route is just making up its own set of rules and assumptions.”  _–United Kingdom 12_

Participants associated this autonomy and opacity with a loss of human
influence and control. Sometimes this was framed as a transfer of power from
humans to AI, or even as humans being “at the mercy of the machines.” In some
cases these concerns resonated with common media tropes about AI taking over
society.

  * “It is difficult, if there is something starting to develop itself, and you can’t influence it anymore, at least only in a limited way.”  _–Germany B09_

  * “I also got a scary feeling from [deep learning], the feeling that we’ve given power to somewhere else.”  _–Finland 07_

  * _Germany B02_ : “[System C is] not transparent. It can’t give a reason.”
_Germany B11_ : “As they don’t know it themselves. Nobody knows why the
computer is making a certain decision because it is learning by itself,
somewhere deep inside.”
_Germany B07_ : “And we are supposed to trust them. [laughter]”
_Germany B02_ : “… there is no argument against the decision.”
_Germany B10_ : “So you elevate the machine to be equal to God.”

Some participants formulated the use of deep learning in algorithmic decision-
making as overreliance on machines, expressing concern about blindly accepting
decisions made by machines. Some were also concerned about humans subjecting
themselves to the dictates of algorithmic decision-making systems, saying for
example that a deep learning credit scoring system that did not provide
explanations was “too authoritarian.”

  * “You don’t want somebody to just dictate everything for you. You want to have the ability to be like… ‘Wait, that’s not really right.”’  _–United States 07_

  * “I’m so anti-relying on machines and this would force me to say, ‘… because you, the machine tell me this is 95% accurate, I have to automatically accept it.’ … ‘Well, F you, sorry, I don’t agree with it.”’  _–United States 12_

Such concerns were exacerbated by fear or anger at the thought of ending up in
the group about whom AI made an inaccurate decision, and a sense that it was
“random” chance whether one ended up in that group. Participants sometimes
compared subjecting oneself to decisions from such a system to gambling at
poker or dice. A few participants also worried that it would be difficult to
escape from an inaccurate decision once it had been made, as decisions might
be shared across institutions, leaving them essentially powerless and without
recourse.

  * “If I’m one of that 5%, then it’s really shitty that I’m accused of something I really haven’t done.”  _–Finland 01_

  * “Transparency [is more important than accuracy]. Because 95% means that there are still five percent which are wrong. And if I am part of the 5% then it is bad.”  _–Germany A03_

  * “It only sucks if you are the five percent. That’s the issue… for most it is good, but for those for whom it does not work, it is shit… Oh man, that’s like playing poker. [laughter] That is so much like playing poker. Even with the best hand, you can always lose.”  _–Germany B11_

  * _Germany A09_ : “I think this would be bullshit [to not get an explanation regarding a decision, for example, the reason your application for a loan has been rejected], because you cannot escape from this hamster’s wheel. No matter what you do, you cannot escape it. You are simply unlucky. And you cannot learn from it. You cannot get behind the reason et cetera, you cannot do anything.”
 _Germany A05_ : “You cannot prepare better for the next job application
because you don’t know why they did not accept your previous application.”
 _Germany A07_ : “And if everybody works with the same AI system, it would not
matter whether I write a hundred or a thousand applications. The AI system
would always say, ‘You are out.’ … Nowadays, the situation is different. I can
go to [three different institutions] and they might apply different criteria…
With System C I have to hope that I don’t belong to the 5% group.”
 _Germany A09_ : “Yes… There will always be winners or losers, however, once
you are a loser, you will always be a loser.”

###  4.3. What do explanations offer?

Explanations are a potential mechanism to ameliorate the concerns raised in
the previous sections, and participants portrayed explanations as having a
number of useful or desirable functions as summarized in Table 1. Explanations
were seen as providing transparency, allowing people to detect errors (e.g.
incorrect data leading to a poor credit score), and laying the groundwork for
appeals to change decisions. Explanations were also seen as a mechanism for
oversight and accountability, for example, a way to detect and expose
discrimination in job application screening decisions.

Purposes of Explanations |  Notable Mentions in Other Work
---|---
Enable individual challenge |  Legal (Edwards and Veale, 2017; Selbst and Barocas, 2018; Kaminski, 2019b, a; Citron and Pasquale, 2014; Wachter et al., 2018);
|  Deliberative Democracy (Office, 2019);
|  Industry Roundtable (Office, 2019); Technical (Tintarev and Masthoff, 2007)
Inform personal behavior |  Legal (Selbst and Barocas, 2018; Wachter et al., 2018); Deliberative Democracy (Office, 2019; Oswald, 2019);
|  Industry Roundtable (Office, 2019)
Enable accountability & oversight |  Legal (Selbst and Barocas, 2018; Kaminski, 2019a; Citron and Pasquale, 2014); Deliberative Democracy (Office, 2019)
Be respectful |  Legal (Edwards and Veale, 2017; Selbst and Barocas, 2018; Kaminski, 2019a)
Build user trust & satisfaction |  Technical (Biran and Cotton, 2017; Herman, 2017; Tintarev and Masthoff, 2007)
Advance human knowledge |  –
I’m the kind of person who needs explanations |  –
Reasons explanations may be seen as unnecessary |
May not understand explanation |  Deliberative Democracy (Office, 2019)
Bad time to receive explanation |  Deliberative Democracy (Office, 2019)
Trust system (e.g. system accuracy has been verified) |  Ethics (London, 2019); Technical (Doshi-Velez and Kim, 2017)
Don’t require an explanation from humans in that situation |  Deliberative Democracy (Oswald, 2019)
Reason for decision is obvious |  –
Already know reason for decision |  –
Table 1. In the lefthand column we first list purposes participants discussed
in our study and then reasons why explanations were seen by our participants
as unnecessary, as described in Section 4.3. In the righthand column we draw
connections with notable mentions of these ideas in legal, technical and other
literatures. Previous work has been largely conceptual, with rare mentions in
user research. Our work provides new and supporting evidence that these ideas
resonate with members of the public.

  * _United States 02_ : “You need the explanation to make the appeal.”
_United States 01_ : “First. That’s the foundation which you base your appeal
on.”
…
_United States 05_ : “The explanation… that’s the most important component of
it. If you get the explanation, you might realize that there is no reason for
you to appeal or you might not need to appeal.”

Beyond corrective measures and transparency, explanations were highlighted as
informing personal future behavior or actions, for example, understanding how
to modify one’s future financial decisions in order to increase a credit
score.

  * _United States 12_ : “What’s bothering me about deep learning is that there’s no way of explaining how it reaches its decision. That is bothering me… because, when push comes to shove, I want to know, ‘How’d you reach the decision? What was the basis? What was the factors?’ I need to know.”
_United States 06_ : “It’s going to be a common problem that somebody has a
low credit score, and if there’s no way to tell them how to improve, that’s a
little frustrating.”

  * “Sometimes, there’s not a need for an appeal, but there’s a need for improvement. So if you have help on what you could do to improve, then that’s the most important.”  _–United States 05_

  * “If you had a system where they could pick out specific bits of the CV… what it would allow people to do is to go away and go, ‘Oh right, okay. I just haven’t got this job because I haven’t got the criteria. This is the specific things that I need to work on.”’  _–United Kingdom 08_

  * “What can I do myself to better my situation, so the computer will smile upon me?”  _–United States 01_

A few participants also highlighted that explanations can advance human
knowledge, for example, expressing concern medical progress might be slowed in
the stroke diagnosis scenario if decision-making was given over to deep
learning and explanations were not available.

  * “Doctors can perhaps even learn from it, that this has made the decision based on these grounds. Doctors might learn something, whether it’s right or wrong.”  _–Finland 08_

  * “[Deep learning means] man does not learn anything anymore… the knowledge is transferred into a system, where you cannot get it out.”  _–Germany B11_

  * “For me, the limit is, where it is better than a doctor. If I suppose that if normal doctors have a failure diagnosis rate of 25%, then I have a higher chance to survive if a system with the 85% probability executes the examination. The only thing that would bother me is, if we don’t get any explanations and only have the system, that the knowledge about the disease and the illness will get lost.”  _–Germany 05_

Others spoke of explanations as building good will or demonstrating respect,
consistent with discussion in the legal literature that explanations serve a
dignitary and humane purpose (Edwards and Veale, 2017; Kaminski, 2019a; Selbst
and Barocas, 2018). Some participants also wanted explanations due to broad
personal preference.

  * “My issue with deep learning is that I’ve always been the type of person – I’m not going to just take something for what it is. I need to know. I’m 100% okay with whatever result it is, but just take me through the process of how we got to that.”  _–United States 07_

  * “I’m a person that’s very analytical, so I need to be able to understand, to see what it is. Let me see it, let me read it, let me understand it, let me digest it.”  _–United States 02_

However, some participants also argued that there were situations in which
explanations were simply not useful or necessary, or would be substantially
less important than other qualities such as accuracy or appeal, as shown in
Table 1. Some participants even challenged others who said they prioritized
explanations over accuracy in given scenarios, arguing that those explanations
would not be helpful to them.

  * “If I know that it works well in 19 out of 20 cases, then I don’t necessarily need to know how exactly it works.”  _–Germany A12_

  * “At the end of the day, there are things that no matter how much someone explains to me, I’m not going to understand regardless, because I know sometimes I have really intelligent friends who try to explain things to me. I’m not going to get it… I’m not going to understand everything, so if it’s just going to tell me the score and then I still have the opportunity to appeal it and discuss it with someone, I’m going to go with C, because it’s the most accurate.”  _–United States 07_

Participants often viewed straightforward decisions as not requiring
explanation or human review.

  * “It’s situational. You run a red light, it recognizes your plate. You should get a ticket for that. You don’t need [an explanation]… the factors are undisputable, right?”  _–United States 08_

  * “It depends on the example. If we look, for example, at a case of parking in a wrong spot and the fact that the owner of a vehicle needs to pay a fine for parking in a prohibited zone, and furthermore assume that the photo that has been taken is 100 percent accurate, I won’t need a [human] judge. It is quite clear that my vehicle was parked in that spot.”  _–Germany A03_

  * “Maybe in areas of sports. For example, when a decision has to be made whether the person was in front or behind a line. In this case, I would rely on a technique that has a high hit-ratio [and not need additional explanation].”  _–Germany A01_

In Table 1 we summarize the ideas described in this subsection, i.e. the
purposes participants thought explanations could serve as well as the reasons
they sometimes saw them as unnecessary. Legal, ethical, and technical
scholarship has explored some of these ideas conceptually, but to our
knowledge, at best limited evidence has been provided that they resonate with
members of the public. Here we draw together these disparate ideas from
multiple disciplines, and provide new and supporting evidence that these ideas
have a meaningful role in public discourse and policy.

###  4.4. Participants felt some decisions require a human touch

In previous sections we focused on how participants perceived algorithmic
decision-makers, and how explanations might address potential issues with
them. We now turn our attention to how participants perceived human decision-
makers as a potential alternative or remedy to deficits in algorithmic
decision-making. Participants frequently brought up humans and emphasized that
humans have important decision-making capabilities that algorithmic decision-
making systems do not. Relative to algorithmic decision-making systems, humans
were seen (in some or all situations) as more trustworthy, compassionate and
capable of treating people well, capable of complex judgment and assessing
intangible factors, and engaging in negotiation. Decision-making tasks
involving such capabilities were seen as best left to humans. Further,
participants viewed humans as having a critical role in overseeing and
intervening in algorithmic decision-making systems, in order to prevent the
loss of autonomy discussed in Section 4.2.

Participants often brought up the importance of ‘the human touch’ or ‘the
human element’ in decision-making. Human knowledge, collective or individual,
was often seen as more trustworthy than machine knowledge or artificial
intelligence. Some participants observed that System A ‘had the most human
element’ because it was based on human thinking and the knowledge of human
professionals, and preferred it for that reason.

  * “I somehow wish for a human for this kind of thing [stroke diagnosis].”  _–Finland 03_

  * “I am worried about moving to a future where there’s literally no human element. I really am.”  _–United States 09_

  * “I think there’s that element of us as humans, just not being able to trust anything that’s not us.”  _–United Kingdom 08_

As mentioned in Section 4.1, AI was characterized as lacking human
capabilities such as emotion, empathy, or compassion. Relatedly, the lack of
human involvement in a decision was sometimes characterized as devaluing the
recipient. Human participation was viewed as adding emotional capacity and
respect to the decision-making process.

  * “That’s the human factor, someone who explains it empathically, not just the machine, which says ‘You’re bad’.”  _–Finland 10_

  * “I mean, the robot can’t be merely a robot. It has to sort of adopt some human features of empathy and compassion.”  _–United Kingdom 09_

As also discussed in Section 4.1, AI was typically seen as lacking the
capacity to make complex moral judgments or assess soft factors. By contrast,
humans were viewed as strong at these tasks. Therefore, participants felt
decision-making responsibilities involving these more subjective judgments
should be given to humans rather than AI.

Human involvement was called out as particularly important for negotiation.
Some participants highlighted the importance of going beyond formal decision-
making criteria, and having flexibility to work around the system. This moved
beyond assessment of intangibles like soft factors, to actions like
considering extenuating circumstances, granting leniency for catastrophic
events in people’s lives, ‘giving people a chance,’ or taking into account
personal trust.

  * “I prefer a human-based one because I can say, ‘Look, I’ve got some of the criteria that you’ve asked for, but the others, I’m willing to learn. I’m a fast learner. Hire me, for a month, and if I haven’t learnt it, then obviously get someone else in. But I’m prepared to learn. And I’ll try my best.”’  _–United Kingdom 02_

  * “I work in social housing, and there’s some questions that are asked all the time. And rather than having someone answer the same question all the time [chuckling] … do you know what I mean? ‘How do I pay my rent?’ Those basic questions. A chatbot can do that. But if it’s, ‘I can’t pay my rent because …’ ‘I can pay universal credit’ or something else, for me, a human will always have to do that.”  _–United Kingdom 12_

  * “I mean, if you went in the Old West, into a bank and you’re sitting down with the manager and they say, ‘Well, I trust you, I know your family, let me give you a loan.’ That’s the human element. There is no human element in the credit system.”  _–United States 08_

Humans were also seen as more variable than algorithmic decision-making
systems, with approaches to multiple humans offering another avenue to get a
favorable decision. Algorithmic decision-making systems were seen as
consistent within institutions, and perhaps even across institutions.

  * _United States 01_ : “The human ability to make judgment calls. We are not all the same, with the same parameters. There are times within the human experience –”
_United States 02_ : “Things happen.”
_United States 01_ : “Things happen. I mentioned, your kid falls out of a
tree, you all of a sudden have huge medical bills, which changes your whole
ability to handle other finances in your life and this is why I think appeal
is important. And here’s another thing, it depends on who you get on the
phone.”
…
_United States 02_ : “That’s true, because I’ve gotten people on the phone
when I spoke to my credit card company – like I’ll speak to somebody at 11:00
and I may speak to somebody at 1:00 –”
_United States 01_ : “You get two different –”
_United States 02_ : “You get two different answers. Two different
temperaments.”

Finally, humans were seen as having an essential role in decision-making
systems. Human oversight was viewed as necessary to keep control over machines
and ensure they were behaving in a socially responsible manner, for example,
by periodically auditing the system. Further, it was seen as advantageous to
integrate humans in particular circumstances or at moments when algorithmic
decision-making might be flawed, for example to use a human instead or have a
human review output from the algorithm to make a final decision. However,
other research points to challenges in successfully allocating
responsibilities between humans and AI, as we discuss further in Section 5.4.

###  4.5. Comparison of Human and AI Decision-Making

| Perceived to do well
---|---
| Humans | AI
Speed | No | Yes
Scale | No | Yes
Consistency | No | Yes
Objectivity | No | Mixed
Accuracy | Mixed | Mixed
Provide Explanations | Mixed | Mixed
Provide Opportunity to Challenge | Mixed | Mixed
Provide Opportunity to Negotiate | Yes | No
Provide Human Touch | Yes | No
Make Subjective Judgments | Yes | No
Make Straightforward Decisions | Yes | Yes
Table 2. Desirable qualities of decision-makers.

In Table 2 we summarize the desirable qualities of good decision-makers,
whether human or AI, that were raised during the workshops.666While some of
the items on the list were presumably influenced by information provided
during the workshops, workshop discussion was open-ended. Participants
sometimes disagreed with or challenged the information provided, and often
raised new issues that had not been discussed. A follow-up study, either
qualitative or quantitative, could be performed to determine the extent to
which participants generate these concepts spontaneously. See also the related
discussion of advantages and disadvantages of automated decision-making in
(Singh, 2019) which mentions a subset of these issues. We note that while
consistency is often positioned as an advantage of automation, as discussed
above, some participants saw downsides in removing human variability and
variability across institutions, as that variability gave them more ‘chances’
to get a favorable decision.

Although there is some debate on this topic, the predominant position appears
to be that all these qualities cannot be simultaneously achieved in entirety,
leading to the need to prioritize. We now discuss the relative value of these
qualities to participants when considering: (1) which factors were most valued
for an algorithmic decision-making system, in a given situation; and (2) which
factors would prompt more or less human versus AI involvement, in a given
situation. Considering tradeoffs between different algorithmic decision-making
systems, qualities such as accuracy, speed, scale, explanation, and challenge
were seen as critically important for at least one of the scenarios presented,
and often more. For example, participants often emphasized accuracy and speed
in the stroke diagnosis scenario.

  * “You go for accuracy when your life is at stake.”  _–United States 03_

  * “In this kind of stroke, the faster the correct decision comes, the better it is… There’s no time for nitpicking.”  _–Finland 11_

  * _Finland 02_ : “If I were the patient, I would be interested especially in accuracy. If we think about the 25% probability of getting a wrong diagnosis, I’m more interested in whether the diagnosis is correct, instead of the explanation of the diagnosis, in this case…”
_Finland 08_ : “Yes. It doesn’t give much consolation to get a good
explanation if you’re already in a casket.”

A small number of participants called out a potential downside of accuracy,
that decisions could be harder to overturn if the algorithmic decision-making
system were known to be highly accurate. Therefore, they expressed a
preference for lower accuracy in some situations because they thought it would
be easier to successfully appeal if the system were less trusted, resonating
with the concept of ambiguity as a design resource (Aoki and Woodruff, 2005;
Gaver et al., 2003).

  * “One of the reasons I chose A is because I thought that if it was going to be that inaccurate – only 75% – it wouldn’t be trusted as much, like it would maybe be used to speed up the investigations or something, but it wouldn’t be used as evidence.”  _–United States 10_

Participants sometimes preferred a system that balances competing priorities.

  * “[System B is] a decent compromise. I’m willing to give up some accuracy to get more communication, more of a two-way street of dialogue, versus the machine saying, ‘This is my final roll in, lots of luck.’ … So, it’s a good middle ground.”  _–United States 01_

  * “I am the kind of person who takes the middle road. There is still room for a little bit of freedom. You can influence the system and you can trace the process. It is not the extreme system of the three of them.”  _–Germany A11_

  * “[System B is] the middle path. Fairly precise, but gives the possibility of challenging it.”  _–Finland 01_

Situational factors influenced not only preference among different algorithmic
decision-making systems (e.g. one that was more accurate but provided less
explanation, or vice versa), but also perception of whether a human must be
involved in the decision-making process, for example, providing oversight over
an algorithmic system and making the final decision, or whether an algorithmic
decision-making system was seen as appropriate to implement at all versus
relying solely on a human decision-maker.

  * “It depends so much on the situation. There are situations where [AI] could be and surely also is more precise than a human.”  _–Finland 08_

  * “I think that it really depends on the topic. If we are talking about face recognition and stroke, you are talking about health and your civil rights and whether you’ll be detained or not, so it’s about your freedom. Those are two absolutely extreme things concerning you and I would never give any AI the full and 100% authority to make a decision.”  _–Germany B11_

  * “I’ve experienced something like that in my family a year ago. A life-saving surgery had to happen, and parts of it were done by a robot, which couldn’t be done by humans. And my dad wouldn’t be alive if that wouldn’t have happened… so I thought, okay there isn’t any doubting anymore, you just try it, because it is the only thing that can help. And then, that is the thing you cling to. In that moment, completely other things matter. When you know there is no other way, then you do it. It is maybe also in relation to what the initial circumstances are, are there still many options, or aren’t there any left? And then it is maybe the only possibility. And in this case, I believe it is good that it exists, and that the development got so far, that they are more accurate and more unerring and can do things that humans aren’t able to.”  _–Germany B09_

Overall, considering tradeoffs such as accuracy versus explainability, a
number of situational factors were important. Some of the most influential
factors were: whether the decision was mechanical versus subjective; the
severity of the consequences; whether it was the recipient’s only chance at a
decision (e.g. a medical decision or legal trial might offer only one
meaningful opportunity at a decision, whereas an individual could apply
multiple times for a loan); whether the recipient could take meaningful action
based on having information; or the perspective of different parties (for
example, for an algorithmic decision-making system that screens job
applications, an applicant might prioritize explanations but a hiring manager
might prioritize accuracy).

  * “It varies between scenarios… it just depends on the situation.”  _–United States 07_

  * “This is obviously different from the first one [job application screening], where this [credit scoring] is very black and white. You either did or you did not pay on time your mortgage bill, your credit card, blah, blah, blah. And so initially you think, this is so accurate you go with C. However, this is so much more important than a job interview, because if you get an inaccurate credit rejection and you can’t find out why, that’s really concerning. Because that… will affect your life.”  _–United Kingdom 11_

  * “Facial recognition affects us on a much more personal level.”  _–Germany A11_

  * “The effects of the inaccuracy are more damaging [for law enforcement] than with credit. I mean, a wrong facial recognition could change a life.”  _–United States 05_

  * “If you are all of a sudden part of the 25% and you have been categorised in the wrong way and then you are up for a crime… It goes much deeper than a decision about whether an application is accepted or not.”  _–Germany A02_

  * “System A becomes more relevant in terms of the consequences that result from it. If there is a threat to go to jail, it is more important for me that the system is transparent and that I can prove it was not me. System A might be less relevant in cases where I apply for a job and am declined, because I can apply to other companies.”  _–Germany A06_

  * “I would like to know how it functions, if it makes decisions this big, as medical ones or legal ones, in this country. It’s important that there’s access to see how it functions.”  _–Finland 11_

  * “From the company’s point of view, it would be the accuracy. But from someone applying, it would be the explanation [murmurs of agreement].”  _–United Kingdom 10_

##  5\. Discussion

Our findings suggest productive directions for design, technical and research
approaches to algorithmic decision-making.

###  5.1. Design choices related to explainability should account for
situational variation

Our research demonstrated that tradeoffs change across scenarios, for example,
in some cases explanation, appeal, or other forms of transparency were a
higher priority than accuracy but in other cases the reverse was true. Our
research therefore suggests that compromising other qualities of an
algorithmic decision-making system in order to provide an explanation is not
always the right choice. We also highlight that while severity was sometimes a
factor, it was often not the dominant consideration and was not sufficient to
predict whether participants felt an explanation should be required. These
findings reinforce and extend previous arguments that broad requirements to
provide explanations are unlikely to meet the full spectrum of public needs
(Balaram et al., 2018; Office, 2019; Oswald, 2019; Singh, 2019). Our research
also underscores that explanation per se is not necessarily the end goal
(Kluttz et al., 2020; Rader et al., 2018; Woodruff, 2019), but rather there
are goals underlying requests for explanation that can often be met via other
mechanisms that provide accountability, contestability, appeal, advancement of
human knowledge, or other qualities.

###  5.2. AI’s (perceived) incapacity for humanity and subjectivity

AI was seen as lacking human qualities, such as the ability to show empathy,
or the ability to make subjective decisions that involved complex moral
judgments or intangible qualities. These perceptions are likely accurate for
most of the algorithmic decision-making systems people have interacted with
and that are in production today. However, future systems may be designed with
more of these qualities as goals. Further, even for today’s systems,
perception of AI as highly inflexible may not be entirely accurate.
Communication and education about AI might be valuable towards influencing the
public’s perception of its qualities.

###  5.3. Negotiability in decision-making systems

Our findings surfaced a new concept that is on the periphery of related
capabilities in the literature such as contestability and recourse (Hirsch et
al., 2017; Kluttz et al., 2020; Ustun et al., 2019; Vaccaro et al., 2019;
Venkatasubramanian and Alfano, 2020). Discussion of these capabilities
generally focuses on a decision-making framework, for example, examining how
to challenge decisions that have not been made correctly within that
framework, how experts may engage with the system to modify the framework, or
how individuals can receive guidance on how they can change their behavior to
meet the criteria of the framework. We propose negotiability, the ability of a
decision-making system to allow individuals to request that a decision be
modified not because it is based on inaccurate data and not because the
decision is incorrect, but rather because they would like to ask the decision-
maker to go beyond the established decision-making framework and criteria, to
take into account factors outside of the accepted decision-making process,
such as extenuating circumstances or personal relationships. In existing
encounters with institutional decision-making, participants had sometimes been
able to navigate human variability, empathy, and inconsistency to get a
favorable result, and they felt that this opportunity would be lost with a
non-human decision-maker. This connected with their beliefs that algorithmic
decision-making systems were dispassionate, authoritarian, inflexible, and
consistent, and would not be susceptible to appeals for compassion or special
treatment. Such negotiability is not without challenges and drawbacks.
Institutional movement towards automation and bureaucracy in many ways seeks
to eliminate exactly such inconsistency and attendant concerns such as
unfairness or bias. Even with these limitations, negotiability is a feature
that some respondents appear to value in existing approaches, and it is worth
considering further whether or how this need might be met (whether by
algorithmic systems or human review) while also preserving other important
qualities.

###  5.4. Further research on human-in-the-loop

Participants saw strong advantages to AI in many cases, and participant
interest in instilling humanity in decision-making processes raises
interesting questions about the role of humans-in-the-loop (De-Arteaga et al.,
2020; Fjeld et al., 2020). Participants were highly interested in the roles
humans and AI would play at different moments in the decision-making process,
and their opinions regarding whether the use of AI was beneficial for a given
use case often rested on detailed specifics of roles played by humans versus
AI in the overall decision-making process. Further, well-executed human-in-
the-loop strategies have the potential to ameliorate concerns about autonomy.
Given that participants perceived that AI and humans had many complementary
strengths, it would be productive to further explore how to effectively
allocate responsibility and tasks between them in algorithmic decision-making
systems. At the same time, other research has shown that handoffs between
humans and AI are complex and difficult to design well (Elish, 2019; Green and
Chen, 2019). We believe this to be an underexplored area and highlight this as
a valuable topic for future research.

##  6\. Conclusion

We reported on a qualitative study of perceptions of algorithmic decision-
making, conducted in Finland, Germany, the United Kingdom, and the United
States. Participants shared thoughtful, nuanced insights on the qualities of
human and AI decision-makers. Participants saw strong advantages to AI in many
circumstances, but in some cases also felt that at certain moments humans
should assume important responsibilities in algorithmic decision-making
systems. We found high situational variance in the prioritization of different
qualities, for example, explainability and appeal were prioritized in certain
situations but not others. Further, algorithmic decision-making has been
heralded as offering many benefits in terms of efficiency and consistency.
While participants saw some of these qualities as desirable, they saw others
as undesirable, and associated them with a loss of humanity, judgment, and
flexibility. Future efforts can explore whether these qualities are inherently
in tension with a shift towards algorithmic decision-making, and whether AI
can effectively support perceived humanity and negotiability.

## References

  * (1)
  * Abdul et al. (2018) Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankanhalli. 2018.  Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–18.  https://doi.org/10.1145/3173574.3174156
  * Ananny and Crawford (2016) Mike Ananny and Kate Crawford. 2016.  Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability.  _New Media & Society_ (2016).  https://doi.org/10.1177/1461444816676645
  * Aoki and Woodruff (2005) Paul M. Aoki and Allison Woodruff. 2005.  Making space for stories: ambiguity in the design of personal communication systems. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2005)_. 181–190.  https://doi.org/10.1145/1054972.1054998
  * Armour (1995) Audrey Armour. 1995\.  The citizens’ jury model of public participation: a critical evaluation.  In _Fairness and Competence in Citizen Participation_. Springer, 175–187.
  * Balaram et al. (2018) Brhmie Balaram, Tony Greenham, and Jasmine Leonard. 2018\.  Artificial Intelligence: Real Public Engagement.  www.thersa.org/discover/publications-and-articles/reports/artificial-intelligence-real-public-engagement
  * Balog and Radlinski (2020) Krisztian Balog and Filip Radlinski. 2020.  Measuring Recommendation Explanation Quality: The Conflicting Goals of Explanations. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 329–338.  https://doi.org/10.1145/3397271.3401032
  * Barocas and Selbst (2016) Solon Barocas and Andrew D. Selbst. 2016.  Big Data’s Disparate Impact.  _California Law Review_ 104 (2016), 671.
  * Beyer and Holtzblatt (1997) Hugh Beyer and Karen Holtzblatt. 1997.  _Contextual Design: Defining Customer-centered Systems_.  Elsevier.
  * Biran and Cotton (2017) Or Biran and Courtenay Cotton. 2017.  Explanation and justification in machine learning: A survey. In _IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)_ , Vol. 8. 8–13.
  * boyd and Crawford (2012) danah boyd and Kate Crawford. 2012.  Critical Questions For Big Data.  _Information Communication & Society_ 15, 5 (2012), 662–679.  https://doi.org/10.1080/1369118X.2012.678878
  * Breiman et al. (2001) Leo Breiman et al. 2001\.  Statistical modeling: The two cultures (with comments and a rejoinder by the author).  _Statist. Sci._ 16, 3 (2001), 199–231.
  * Bucher (2017) Taina Bucher. 2017\.  The algorithmic imaginary: exploring the ordinary affects of Facebook algorithms.  _Information, Communication & Society_ 20, 1 (2017), 30–44.
  * Burrell (2016) Jenna Burrell. 2016\.  How the machine ‘thinks’: Understanding opacity in machine learning algorithms.  _Big Data & Society_ 3, 1 (2016).  https://doi.org/10.1177/2053951715622512
  * Cai et al. (2019) Carrie Jun Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019.  “Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. In _Proceedings of the 2019 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’19)_.  https://doi.org/10.1145/3359206
  * Capital (2019) Blumberg Capital. 2019\.  Artificial Intelligence in 2019: Getting Past the Adoption Tipping Point.
  * Caruana et al. (2017) Rich Caruana, Patrice Simard, Kilian Weinberger, and Yann LeCun. 2017.  The Great AI Debate – Position: Interpretability is necessary for machine learning, for and against.  The 31st Conference on Neural Information Processing Systems (NIPS 2017).
  * Cave et al. (2019) Stephen Cave, Kate Coughlan, and Kanta Dihal. 2019\.  ”Scary Robots”: Examining Public Responses to AI. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES 2019)_. 331–337.  https://doi.org/10.1145/3306618.3314232
  * Cave et al. (2018) Stephen Cave, Claire Craig, Kanta Sarasvati Dihal, Sarah Dillon, Jessica Montgomery, Beth Singler, and Lindsay Taylor. 2018\.  Portrayals and perceptions of AI and why they matter.  (2018).
  * Center (2004) The Jefferson Center. 2004\.  The Citizens’ Jury Handbook.
  * Citron (2007) Danielle Keats Citron. 2007\.  Technological Due Process.  _Washington University Law Review_ 85 (2007), 1249–1313.
  * Citron and Pasquale (2014) Danielle Keats Citron and Frank Pasquale. 2014.  The Scored Society: Due Process for Automated Predictions.  _Washington Law Review_ 89 (2014), 1.
  * Commission (2016) Federal Trade Commission. 2016\.  Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues.
  * Corden and Sainsbury (2006) Anne Corden and Roy Sainsbury. 2006.  _Using verbatim quotations in reporting qualitative social research: researchers’ views_.  University of York.
  * Cottom (2015) Tressie McMillan Cottom. 2015\.  Credit Scores, Life Chances, and Algorithms.  https://tressiemc.com/uncategorized/credit-scores-life-chances-and-algorithms/
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020\.  A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_.  https://doi.org/10.1145/3313831.3376638
  * Diakopoulos (2016) Nicholas Diakopoulos. 2016\.  Accountability in algorithmic decision making.  _Commun. ACM_ 59, 2 (2016), 56–62.  https://doi.org/10.1145/2844110
  * DiSalvo et al. (2011) Carl DiSalvo, Thomas Lodato, Laura Fries, Beth Schechter, and Thomas Barnwell. 2011.  The collective articulation of issues as design practice.  _CoDesign_ 7, 3-4 (2011), 185–197.  https://doi.org/10.1080/15710882.2011.630475
  * DiSalvo et al. (2008) Carl DiSalvo, Illah Nourbakhsh, David Holstius, Ayça Akin, and Marti Louw. 2008.  The Neighborhood Networks Project: A case study of critical engagement and creative expression through participatory design. In _Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (PDC’08)_. 41–50.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ (2017).
  * Edelman (2019) Edelman. 2019.  2019 Edelman AI Survey.
  * Edwards and Veale (2017) Lilian Edwards and Michael Veale. 2017.  Slave to the algorithm? Why a right to an explanation is probably not the remedy you are looking for.  _Duke Law and Technology Review_ 16 (2017).
  * Elish (2019) Madeleine Clare Elish. 2019\.  Moral crumple zones: Cautionary tales in human-robot interaction.  _Engaging Science, Technology, and Society_ 5 (2019), 40–60.
  * Eslami et al. (2016) Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016.  First I “like” it, then I hide it: Folk Theories of Social Feeds. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 2371–2382.  https://doi.org/10.1145/2858036.2858494
  * Eslami et al. (2015) Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015\.  ”I always assumed that I wasn’t really that close to [her]”: Reasoning about Invisible Algorithms in News Feeds. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, 153–162.  https://doi.org/10.1145/2702123.2702556
  * Eslami et al. (2019) Motahhare Eslami, Kristen Vaccaro, Min Kyung Lee, Amit Elazari Bar On, Eric Gilbert, and Karrie Karahalios. 2019.  User attitudes towards algorithmic opacity and transparency in online reviewing platforms. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_.  https://doi.org/10.1145/3290605.3300724
  * Eubanks (2017) Virginia Eubanks. 2017\.  _Automating inequality: How high-tech tools profile, police, and punish the poor_.  St. Martin’s Press.
  * Fast and Horvitz (2017) Ethan Fast and Eric Horvitz. 2017.  Long-term trends in the public perception of artificial intelligence. In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Fjeld et al. (2020) Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Srikumar. 2020.  Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI.  _Berkman Klein Center Research Publication_ 2020-1 (2020).
  * Fourcade and Healy (2013) Marion Fourcade and Kieran Healy. 2013.  Classification situations: Life-chances in the neoliberal era.  _Accounting, Organizations and Society_ 38, 8 (2013), 559–572.
  * Friedman and Nissenbaum (1996) Batya Friedman and Helen Nissenbaum. 1996.  Bias in computer systems.  _ACM Transactions on Information Systems (TOIS)_ 14, 3 (1996), 330–347.  https://doi.org/10.1145/230538.230561
  * Garvey and Maskal (2019) Colin Garvey and Chandler Maskal. 2019.  Sentiment Analysis of the News Media on Artificial Intelligence Does Not Support Claims of Negative Bias Against Artificial Intelligence.  _OMICS: a Journal of Integrative Biology_ (2019).
  * Gaver et al. (2003) William W. Gaver, Jacob Beaver, and Steve Benford. 2003\.  Ambiguity as a resource for design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 233–240.  https://doi.org/10.1145/642611.642653
  * Gillespie (2014) Tarleton Gillespie. 2014\.  _The relevance of algorithms_.  MIT Press.
  * Gilpin et al. (2018) Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018\.  Explaining explanations: An overview of interpretability of machine learning. In _2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)_. IEEE, 80–89.
  * Goodman and Flaxman (2017) Bryce Goodman and Seth Flaxman. 2017.  European Union regulations on algorithmic decision-making and a “right to explanation”.  _AI magazine_ 38, 3 (2017), 50–57.
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  The Principles and Limits of Algorithm-in-the-Loop Decision Making.  3, CSCW (2019).  https://doi.org/10.1145/3359152
  * Gupta et al. (2016) Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, and Alexander Van Esbroeck. 2016.  Monotonic calibrated interpolated look-up tables.  _The Journal of Machine Learning Research_ 17, 109 (2016), 1–47.
  * Herman (2017) Bernease Herman. 2017\.  The promise and peril of human evaluation for model interpretability. In _Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017)_.
  * Hirsch et al. (2017) Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E. Imel, and David C. Atkins. 2017\.  Designing contestability: Interaction design, machine learning, and mental health. In _Proceedings of the 2017 Conference on Designing Interactive Systems_. 95–99.  https://doi.org/10.1145/3064663.3064703
  * Immonen et al. (2018) Mika Immonen, Sanna Sintonen, and Jouni Koivuniemi. 2018\.  The value of human interaction in service channels.  _Computers in Human Behavior_ 78 (2018), 316–325.
  * Introna and Nissenbaum (2000) Lucas D. Introna and Helen Nissenbaum. 2000.  Shaping the Web: Why the politics of search engines matters.  _The Information Society_ 16, 3 (2000), 169–185.  https://doi.org/10.1080/01972240050133634
  * Ipsos (2019) Ipsos. 2019.  Widespread concern about artificial intelligence.  www.ipsos.com/en/widespread-concern-about-artificial-intelligence/
  * Jungk and Müllert (1987) Robert Jungk and Norbert Müllert. 1987.  _Future Workshops: How to create desirable futures_.  Institute for Social Inventions.
  * Kaminski (2019a) Margot E. Kaminski. 2019a.  Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability.  _Southern California Law Review_ 92, 6 (2019).
  * Kaminski (2019b) Margot E. Kaminski. 2019b.  The right to explanation, explained.  _Berkeley Technology Law Journal_ 34, 1 (2019).
  * Karimi et al. (2020) Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. 2020.  Model-Agnostic Counterfactual Explanations for Consequential Decisions. In _Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS)_.
  * Kelley et al. (2020) Patrick Gage Kelley, Yongwei Yang, Courtney Heldreth, Christopher Moessner, Aaron Sedley, Andreas Kramm, David Newman, and Allison Woodruff. 2020.  “Happy and Assured that life will be easy 10years from now.”: Perceptions of Artificial Intelligence in 8 Countries.  _arXiv preprint arXiv:2001.00081_ (2020).
  * Kluttz et al. (2020) Daniel Kluttz, Nitin Kohli, and Deirdre K. Mulligan. 2020\.  Shaping Our Tools: Contestability as a Means to Promote Responsible Algorithmic Decision Making in the Professions. In _After the Digital Tornado: Networks, Algorithms, Humanity_ , Kevin Werbach (Ed.). Cambridge University Press.
  * Kroll et al. (2016) Joshua A. Kroll, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2016.  Accountable algorithms.  _University of Pennsylvania Law Review_ 165 (2016).
  * Lipton (2016) Zachary C. Lipton. 2016\.  The mythos of model interpretability. In _2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016)_.
  * London (2019) Alex John London. 2019\.  Artificial intelligence and black-box medical decisions: accuracy versus explainability.  _Hastings Center Report_ 49, 1 (2019), 15–21.
  * Lustig et al. (2016) Caitlin Lustig, Katie Pine, Bonnie Nardi, Lilly Irani, Min Kyung Lee, Dawn Nafus, and Christian Sandvig. 2016.  Algorithmic authority: the ethics, politics, and economics of algorithms that interpret, decide, and manage. In _Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems_. 1057–1062.  https://doi.org/10.1145/2851581.2886426
  * Miller (2019) Tim Miller. 2019\.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ 267 (2019), 1–38.
  * MORI (2017) Ipsos MORI. 2017\.  Public views of Machine Learning: Findings from public research and engagement conducted on behalf of the Royal Society.  https://royalsociety.org/-/media/policy/projects/machine-learning/publications/public-views-of-machine-learning-ipsos-mori.pdf
  * Mozilla (2019) Mozilla. 2019.  We Asked People Around the World How They Feel About Artificial Intelligence. Here’s What We Learned.  https://foundation.mozilla.org/en/blog/we-asked-people-around-the-world-how-they-feel-about-artificial-intelligence-heres-what-we-learned/
  * Munoz et al. (2016) Cecilia Munoz, Megan Smith, and D.J. Patil. 2016.  _Big data: A report on algorithmic systems, opportunity, and civil rights_.  Executive Office of the President.
  * Northstar (2017) ARM — Northstar. 2017\.  AI Today, AI Tomorrow. Awareness and Anticipation of AI: A Global Perspective.  www.arm.com/solutions/artificial-intelligence/survey/
  * Nunes and Jannach (2017) Ingrid Nunes and Dietmar Jannach. 2017.  A systematic review and taxonomy of explanations in decision support and recommender systems.  _User Modeling and User-Adapted Interaction_ 27, 3-5 (2017), 393–444.
  * Office (2019) Information Commissioner’s Office. 2019\.  Project ExplAIn Interim Report.  https://ico.org.uk/about-the-ico/research-and-reports/project-explain-interim-report/
  * on Artificial Intelligence (2019) High-Level Expert Group on Artificial Intelligence. 2019.  Ethics Guidelines for Trustworthy AI.
  * O’Neil (2016) Cathy O’Neil. 2016\.  _Weapons of math destruction: How big data increases inequality and threatens democracy_.  Broadway Books.
  * Ostrom et al. (2002) Amy L. Ostrom, Mary J. Bitner, and Matthew L. Meuter. 2002\.  Self-service technologies.  _e-Service: New Directions in Theory and Practice, ME Sharpe, NY_ (2002), 45–64.
  * Oswald (2019) Malcolm Oswald. 2019\.  Artificial Intelligence (AI) & Explainability Citizens’ Juries Report.  https://assets.mhs.manchester.ac.uk/gmpstrc/C4-AI-citizens-juries-report.pdf
  * Parasuraman et al. (2000) Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. 2000.  A model for types and levels of human interaction with automation.  _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ 30, 3 (2000), 286–297.
  * Pasquale (2015) Frank Pasquale. 2015\.  _The Black Box Society: The Secret Algorithms that Control Money and Information_.  Harvard University Press.
  * Poursabzi-Sangdeh et al. (2018) Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018.  Manipulating and measuring model interpretability.  _arXiv preprint arXiv:1802.07810_ (2018).
  * Rader et al. (2018) Emilee Rader, Kelley Cotter, and Janghee Cho. 2018\.  Explanations as Mechanisms for Supporting Algorithmic Transparency. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_.  https://doi.org/10.1145/3173574.3173677
  * Rader and Gray (2015) Emilee Rader and Rebecca Gray. 2015.  Understanding User Beliefs about Algorithmic Curation in the Facebook News Feed. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. 173–182.  https://doi.org/10.1145/2702123.2702174
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ‘Why should I trust you?’: Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 1135–1144.
  * Rosner et al. (2016) Daniela K. Rosner, Saba Kawas, Wenqi Li, Nicole Tilly, and Yi-Chen Sung. 2016.  Out of time, out of place: Reflections on design workshops as a research method. In _Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16)_. 1131–1141.  https://doi.org/10.1145/2818048.2820021
  * Rudin (2019) Cynthia Rudin. 2019\.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1, 5 (2019), 206–215.
  * Russell (2019) Chris Russell. 2019\.  Efficient Search for Diverse Coherent Explanations. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ _(FAT* ’19)_. 20–28.  https://doi.org/10.1145/3287560.3287569
  * Sandvig et al. (2015) Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2015.  Can an Algorithm be Unethical?. In _65th Annual Meeting of the International Communication Association_.
  * Selbst and Barocas (2018) Andrew D. Selbst and Solon Barocas. 2018.  The intuitive appeal of explainable machines.  _Fordham Law Review_ 87 (2018), 1085–1139.
  * Sendak et al. (2020) Mark Sendak, Madeleine Clare Elish, Michael Gao, Joseph Futoma, William Ratliff, Marshall Nichols, Armando Bedoya, Suresh Balu, and Cara O’Brien. 2020\.  ‘The human body is a black box’: supporting clinical decision-making with deep learning. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 99–109.  https://doi.org/10.1145/3351095.3372827
  * Singh (2019) Asheem Singh. 2019\.  Democratising decisions about technology: A toolkit.  (2019).  https://www.thersa.org/discover/publications-and-articles/reports/democratising-decisions-technology-toolkit
  * Thomas (2006) David R. Thomas. 2006\.  A general inductive approach for analyzing qualitative evaluation data.  _American Journal of Evaluation_ 27, 2 (2006), 237–246.
  * Tintarev and Masthoff (2007) Nava Tintarev and Judith Masthoff. 2007.  A survey of explanations in recommender systems. In _Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop_. 801–810.
  * University and Gallup (2018) Northeastern University and Gallup. 2018.  Optimism and Anxiety: Views on the Impact of Artificial Intelligence and Higher Education’s Response.
  * Ur et al. (2012) Blase Ur, Pedro Giovanni Leon, Lorrie Faith Cranor, Richard Shay, and Yang Wang. 2012.  Smart, useful, scary, creepy: perceptions of online behavioral advertising. In _Proceedings of the Eighth Symposium on Usable Privacy and Security (SOUPS) 2012_.  https://doi.org/10.1145/2335356.2335362
  * Ustun et al. (2019) Berk Ustun, Alexander Spangher, and Yang Liu. 2019\.  Actionable Recourse in Linear Classification. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 10–19.  https://doi.org/10.1145/3287560.3287566
  * Vaccaro et al. (2019) Kristen Vaccaro, Karrie Karahalios, Deirdre K. Mulligan, Daniel Kluttz, and Tad Hirsch. 2019\.  Contestability in Algorithmic Systems. In _Conference Companion Publication of the 2019 Conference on Computer Supported Cooperative Work and Social Computing_. 523–527.  https://doi.org/10.1145/3311957.3359435
  * Venkatasubramanian and Alfano (2020) Suresh Venkatasubramanian and Mark Alfano. 2020.  The Philosophical Basis of Algorithmic Recourse. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 284–293.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017\.  Why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation.  _International Data Privacy Law_ 7, 2 (2017), 76–99.
  * Wachter et al. (2018) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018\.  Counterfactual explanations without opening the black box: Automated decisions and the GDPR.  _Harvard Journal of Law and Technology_ 31 (2018).
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019\.  Designing theory-driven user-centric explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_.  https://doi.org/10.1145/3290605.3300831
  * Warshaw et al. (2016) Jeffrey Warshaw, Nina Taft, and Allison Woodruff. 2016\.  Intuitions, Analytics, and Killing Ants: Inference Literacy of High School-educated Adults in the US. In _Proceedings of the Twelfth Symposium on Usable Privacy and Security (SOUPS) 2016_. 271–285.
  * West (2018a) Darrell M. West. 2018a.  Brookings survey finds divided views on artificial intelligence for warfare, but support rises if adversaries are developing it.  _Brookings_ (August 2018).
  * West (2018b) Sarah Myers West. 2018b.  Censored, suspended, shadowbanned: User interpretations of content moderation on social media platforms.  _New Media & Society_ 20, 11 (2018), 4366–4383.  https://doi.org/10.1177/1461444818773059
  * Woodruff (2019) Allison Woodruff. 2019\.  10 things you should know about algorithmic fairness.  _Interactions_ 26, 4 (2019), 47–51.
  * Zhang and Dafoe (2019) Baobao Zhang and Allan Dafoe. 2019.  Artificial intelligence: American attitudes and trends.  _Available at SSRN 3312874_ (2019).

##  Appendix A Scenario Methodology

Scenario |  Description
---|---
Stroke Diagnosis |  Rapid and accurate diagnosis of stroke greatly increases chances of survival and recovery of the patient. This is highly specialised work which ideally should be done by neuroradiologists with many years of training and experience. However, these experts are not available in every hospital, 24 hours a day, 7 days a week, and in practice diagnosis is often done by nonspecialist emergency medicine doctors. As diagnostic data are accumulated from previous stroke patients, AI systems could provide stroke diagnosis that is fast, and always available in hospitals.
|  Draws heavily from the NIHR Greater Manchester Patient Safety Translational
Research Centre and the Information Commissioner’s Office (ICO) Citizens’
Juries work http://www.bit.ly/GMPSTRCCitizensJuries
Job Applicant Screening |  A large company in the private sector receives a high volume of applications, and plans to use an AI system to screen job applications and make shortlisting decisions. This will allow the company to screen every application and process applications more quickly. This system may also more objectively make decisions that would otherwise be prone to human errors or judgment. On the other hand, depending on how the system is trained, it may learn to reinforce existing patterns of hiring.
|  Draws heavily on the ICO and RSA (Royal Society for the encouragement of
Arts, Manufactures and Commerce) Citizens’ Juries work (Balaram et al., 2018)
Credit Scoring |  A new credit scoring company has decided to enter the market, and provide a new credit score. They would like their scores to be more accurate, and are considering using AI to assign people scores in the hopes of better predicting whether they will repay money. Using AI will allow for many more features to be input and also pick out a more accurate way to combine features to predict a person’s likelihood of repaying. AI advocates and experts think this will allow companies to lend more money to people who are likely to pay it back, but who might have been excluded in the traditional system.
|  New, but fits within the generic ICO format
Law Enforcement |  Imagine that local law enforcement in your city is under-resourced, sometimes causing it to fail to prevent crimes or leave crimes unsolved. The city government decides to procure a facial recognition system to make law enforcement more efficient. The system will allow law enforcement to identify people or verify their identities, for example, to find missing persons, match surveillance footage from a crime scene against a database of faces, or verify people’s identities when they are stopped by the police.
|  New, but fits within the generic ICO format
Table 3. Excerpts from each of the four hypothetical scenarios. |  System A |  System B |  System C
---|---|---|---
|  Rule-Based |  Conventional Machine Learning |  Deep Learning
_Stroke Diagnosis_ |  |
Accuracy |  75% (nonspecialist doctor’s level) |  85% (expert doctor’s level) |  95% (beyond human level)
Transparency |  Full explanation of all the specific reasons that led to diagnosis |  Partial explanation describing features that played a role in diagnosis |  No explanation
_Law Enforcement_ |  |
Accuracy |  75% (accurate in good lighting and conditions) |  85% (highly accurate) |  95% (most accurate AI technology)
Transparency |  Provides an ability for people who have had their faces matched to appeal (request human review prior to formal decision-making), plus monthly reports for regular external oversight |  Provides an ability for people who have had their faces matched to appeal (request human review prior to formal decision-making), but no process for regular external oversight |  No process to appeal (request human review prior to formal decision-making), no process for regular external oversight
Table 4. Descriptions of the three systems participants could choose among for
the Stroke Diagnosis and Law Enforcement scenarios. For all scenarios,
participants were offered three systems with 75%, 85%, and 95% accuracy, where
the most accurate system provided the least transparency. In all scenarios
except Law Enforcement, transparency was accomplished through explanations;
for Law Enforcement we offered transparency options focused on appeal and
oversight.