# In Search of Verifiability: Explanations Rarely Enable Complementary
Performance in AI-Advised Decision Making

Raymond Fok<EMAIL_ADDRESS>University of WashingtonSeattleWAUSA and
Daniel S. Weld<EMAIL_ADDRESS>Allen Institute for AI &
University of WashingtonSeattleWAUSA

(2022)

###### Abstract.

The current literature on AI-advised decision making—involving explainable AI
systems advising human decision makers—presents a series of inconclusive and
confounding results. To synthesize these findings, we propose a simple theory
that elucidates the frequent failure of AI explanations to engender
appropriate reliance and complementary decision making performance. In
contrast to other common desiderata, e.g., interpretability or spelling out
the AI’s reasoning process, we argue explanations are only useful to the
extent that they allow a human decision maker to verify the correctness of the
AI’s prediction. Prior studies find in many decision making contexts AI
explanations do not facilitate such verification. Moreover, most tasks
fundamentally do not allow easy verification, regardless of explanation
method, limiting the potential benefit of any type of explanation. We also
compare the objective of complementary performance with that of appropriate
reliance, decomposing the latter into the notions of outcome-graded and
strategy-graded reliance.

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††booktitle:
Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018,
Woodstock, NY††copyright: none

##  1\. Introduction

Figure 1. Researchers suggest that AI explanations could aid numerous human-AI
processes, including decision making, model development, knowledge discovery,
and model audit. In this paper, we focus solely on understanding whether
explanations are helpful in the context of AI-advised decision making. We
claim AI explanations cannot foster appropriate reliance and engender
complementary performance in decision making, except in the rare instances in
which they efficiently verify the AI’s recommendation.

Paper |  Decision Making Task |  Explanations |  Enables Verification |  Complementary Performance
---|---|---|---|---
Biran and McKeown (2017) | Stock price prediction | Rationale | ✗ | ✗
Green and Chen (2019) | Pretrial detention | Feature importance | ✗ | ✗
Green and Chen (2019) | Financial lending | Feature importance | ✗ | ✗
Weerts et al. (2019) | Income prediction | Feature importance | ✗ | ✗
Lai and Tan (2019) | Deceptive review | Feature importance (text highlights) | ✗ | ✗
Lai et al. (2020) | Deceptive review | Feature importance (text highlights) | ✗ | ✗
Buçinca et al. (2020) | Food fat content prediction | Feature importance | ✗ | ✗
Carton et al. (2020) | Toxicity detection | Feature importance (text highlights) | ✗ | ✗
Zhang et al. (2020) | Income prediction | Feature importance | ✗ | ✗
Liu et al. (2021) | Recidivism prediction | Feature importance | ✗ | ✗
Alufaisan et al. (2021) | Income prediction | Feature importance | ✗ | ✗
Alufaisan et al. (2021) | Recidivism prediction | Feature importance | ✗ | ✗
Wang and Yin (2021) | Recidivism prediction | Feature importance | ✗ | ✗
Wang and Yin (2021) | Forest cover prediction | Feature importance | ✗ | ✗
Nourani et al. (2021) | Kitchen policy verification | Feature importance | ✗ | ✗
Poursabzi-Sangdeh et al. (2021) | Apartment price prediction | Feature importance | ✗ | ✗
Jesus et al. (2021) | Fraud detection | Feature importance (text highlights) | ✗ | ✗
Nguyen et al. (2021) | Image classification | Attribution maps | ✗ | ✗
Kim et al. (2022) | Image classification | Attribution maps & Prototype-based | ✗ | ✗
Parrish et al. (2022) | Long document MCQA | Rationale | ✗ | ✗
Sivaraman et al. (2023) | Sepsis treatment | Feature importance | ✗ | ✗
Bansal et al. (2021) | Sentiment classification | Feature importance (text highlights) | ✗ | $\star$
Bansal et al. (2021) | Logical reasoning | Rationale | ✗ | $\star$
Taesiri et al. (2022) | Image classification | Nearest neighbors | ✗ | $\star$
Feng and Boyd-Graber (2019) | Quizbowl trivia QA | Rationale | ✓ | ✓
González et al. (2021) | Wikipedia ODQA | Rationale | ✓ | ✓
Vasconcelos et al. (2023) | Maze solving | Path highlights | ✓ | ✓
Lee et al. (2022) | Multitask MCQA | Rationale (interactive) | ✓ | ✓
Bowman et al. (2022) | Long form MCQA | Rationale (interactive) | ✓ | ✓
Table 1. Recent studies investigating the effect of AI explanations on task
performance in AI-advised decision making. Common explanation types include
descriptions of feature importance (over both tabular and textual features)
and natural language rationales. Most of these explanations describe the AI’s
decision-making process rather than help a human decision maker verify the
AI’s recommendation. But only the latter engender complementary performance
(✓), where team performance exceeds that of the human or AI alone. Sometimes
explanations which do not enable verification may appear to yield
complementary performance ($\star$); however, in these cases, it is unclear
whether explanations actually offer utility as providing only AI
recommendations and confidence scores (without the explanations) also resulted
in complementary performance.

Recent years have seen an explosion of work on explainable AI (XAI), but there
have been mixed results on whether explanations actually help humans who are
making decisions with AI support. In this decision making context, the role of
explanations is to foster appropriate reliance by helping the human understand
whether or not the AI’s advice should be trusted. Appropriate reliance is
desired in order to achieve complementary performance, where the human-AI team
performs better than either the human or AI alone (Bansal et al., 2021). But
here we see a confusing montage of results: not only do most papers find
explanations don’t induce complementary performance more than baseline
methods, such as displaying AI accuracy or confidence, but these papers
suggest explanations can in fact increase over-reliance, where the human
trusts the AI even when it errs. The inconclusive nature of these results
raises a huge question for the field of XAI: when are explanations useful?

We focus solely on the process of AI-advised decision making, defined as the
following: given an instance of a decision making task, an AI makes a
recommendation, and drawing on features of the task, the AI’s recommendation,
and possibly an explanation for the AI’s recommendation, a human decision
maker arrives at a final decision (Figure 1). There are many other possible
uses for AI explanations (Tan, 2022; Kim et al., 2023b), including model
debugging and auditing, e.g., to help the human understand whether the AI’s
reasoning will generalize, but our arguments pertain only to decision making.

In this paper, we present a perspective we believe explains the seemingly
mixed empirical results found throughout the XAI literature. Furthermore, our
proposal is consistent with the way human groups reach consensus on
“intellective” tasks (Laughlin and Ellis, 1986). We argue explanations
provided by an AI model are helpful in decision making (engender complementary
performance (Bansal et al., 2021)) to the extent they allow a decision maker
to verify the AI’s recommendation. While this theory may appear self-evident,

  1. (1)

Most work on XAI has focused instead on creating inherently interpretable
models or generating faithful post-hoc explanations of the AI’s reasoning
process.

  2. (2)

Most of these explanations do not support such verification.

Explanations which faithfully expose the AI’s reasoning process may well be
useful for debugging the AI or predicting its ability to generalize, but it
does not seem to help human decision makers make judgements on individual task
instances. Indeed, most human-subject studies have shown that explanations
fail to produce complementary performance in decision making; the sole
exceptions are explanations that support answer verification (Table 1).

The rest of this paper is structured as follows. The next section surveys the
conflicting results from prior studies on XAI utility. Section 3 details the
decision making context, which is our focus in this paper. Section 4 presents
our core argument — explanations must facilitate verification in order to
engender complementary performance. Section 5 discusses the concept of
appropriate reliance, arguing this term has become overloaded, leading to
confusion; we propose two alternative terms: outcome-graded reliance and
strategy-graded reliance to tease these apart. Section 6 places our conjecture
in a broader context, and Section 7 concludes.

##  2\. Background

Figure 2. Examples of AI-advised decision making tasks. The top two rows
present prototypical tasks found in the XAI literature: deceptive review
detection, with textual features, and recidivism prediction, with tabular
features. Both tasks often utilize feature importance explanations, which
studies suggest rarely engender complementary performance or foster
appropriate reliance. The bottom two rows show examples of a maze solving and
open-domain question answering task, for which studies have found AI
explanations can engender complementary performance.

Providing interpretability for AI models, for instance through explanations,
has been one way researchers have attempted to facilitate more informed and
accurate decision making (Bilgic and Mooney, 2005). One set of studies have
found AI explanations can improve human-AI performance over human decision
making alone and over human-AI teams with AI recommendations without
explanations (Eiband et al., 2019; Lai et al., 2020; Buçinca et al., 2020,
2021; Biran and McKeown, 2017; Horne et al., 2019; Lai and Tan, 2019).
However, none of these studies found explanations could improve human-AI
performance beyond the original capabilities of the AI model. The studies also
involved collaborations in which the AI model significantly outperformed the
human decision maker. It is therefore unclear whether explanations truly
improved the human-AI decision making process, or if explanations convinced
humans to blindly trust the AI’s recommendations. In this type of scenario —
where the AI performs at a superhuman level — should we simply allow the AI to
operate independently? It is arguably naive to believe the human is performing
any meaningful oversight, when the effect of explanations is inducing blind
trust.

In response, researchers have investigated how explanations might support
decision makers in developing appropriate reliance on AI assistance, rather
than improving performance through blind trust. While appropriate reliance is
not well-defined in the literature (as explained in §5), one common definition
states that appropriate reliance is achieved when decision makers agree with
the AI when its recommendation is correct, and disagree with the AI when its
recommendation is incorrect. Unfortunately, achieving this notion of
appropriate reliance through explanations has been elusive (Bansal et al.,
2021; Chandrasekaran et al., 2018; Kiani et al., 2020; Alqaraawi et al., 2020;
Jacobs et al., 2021; Carton et al., 2020; Liu et al., 2021; Nourani et al.,
2021; Poursabzi-Sangdeh et al., 2021; Weerts et al., 2019; Zhang et al., 2020;
Jesus et al., 2021; Parrish et al., 2022; Green and Chen, 2019; Sivaraman et
al., 2023). Fundamentally, it can be hard for people to know how much to trust
recommendations (Ghai et al., 2021; Jacovi and Goldberg, 2021; Sivaraman et
al., 2023; Buçinca et al., 2020). And though AI explanations were hypothesized
to make the AI’s decision making process more interpretable and in turn
support appropriate reliance, most experiments have found little evidence for
such support. Explanations have not only often failed to foster appropriate
reliance, many studies have found explanations can actually negatively impact
appropriate reliance (Buçinca et al., 2021, 2020; Gajos and Mamykina, 2022;
Sivaraman et al., 2023; Kim et al., 2023a).

For example, the mere presence of explanations can increase a decision maker’s
trust in the AI’s recommendations (Buçinca et al., 2021; Wang et al., 2019; Yu
et al., 2019; Zhang et al., 2020; Kim et al., 2022) (even irrespective of
explanation quality (Eiband et al., 2019)), revealing fallacies in human
decision-making — most notably, confirmation bias. If shown to human decision
makers before they have a chance to formulate a decision themselves,
explanations can further introduce an anchoring bias (Ghai et al., 2021;
Nourani et al., 2021; Bansal et al., 2021; Bowman et al., 2022; Buçinca et
al., 2021; Wang et al., 2019). Explanations may be effective in increasing
decision makers’ perceptions of an AI’s usefulness and decision making
confidence (Sivaraman et al., 2023; Wang and Yin, 2021; Tschandl et al., 2020;
Alam and Mueller, 2021), but these characteristics do not necessarily
translate to improved decision making performance.

Another set of studies have found explanations fail to improve decision making
performance (Green and Chen, 2019; Weerts et al., 2019; Zhang et al., 2020;
Alufaisan et al., 2021; Bansal et al., 2021; Carton et al., 2020; Liu et al.,
2021; Poursabzi-Sangdeh et al., 2021; Jesus et al., 2021; Parrish et al.,
2022; Kim et al., 2023a). One hypothesis for these negative results suggests
decision makers do not actually cognitively engage with AI explanations
(Chromik et al., 2021; Eiband et al., 2019; Ghai et al., 2021; Bunt et al.,
2012), resulting in diminished utility of an explanation regardless of its
type or quality. Another hypothesis for these negative results lies in the
potentially low fidelity of post-hoc explanations for black-box models. Some
argue these post-hoc explanations can provide inaccurate representations of
the original model, and as a result, decision makers are burdened with
determining the trustworthiness of not only the AI’s recommendation but also
its explanation (Rudin, 2019; Ghassemi et al., 2021; Mittelstadt et al.,
2019). However, this supposed interpretability gap between intrinsically
interpretable models and post-hoc interpretability has shown no significant
impact on decision making performance (Poursabzi-Sangdeh et al., 2021; Bell et
al., 2022).

A third set of studies suggests explanations can sometimes yield complementary
performance (Bansal et al., 2021), defined as when the decision making
performance of the human-AI team exceeds the capabilities of either the human
or AI alone (Feng and Boyd-Graber, 2019; Vasconcelos et al., 2023; González et
al., 2021; Lee et al., 2022; Bowman et al., 2022). Unlike the first set of
studies where overall human-AI performance lagged behind AI performance, these
studies found explanations influenced decision makers into appropriately
relying on the AI recommendations.

All in all, the current literature on XAI-advised decision making presents a
smattering of intriguing empirical results. Some studies have found
explanations cannot foster appropriate reliance and fail to engender
complementary performance, while others have found explanations can achieve
complementary performance. Rather than viewing these results as conflicting,
we aim to provide a unifying theory that explains under what circumstances
these explanations can in fact improve decision making performance, if at all.

##  3\. AI-Advised Decision Making

We restrict our analysis of AI explanation utility to the established paradigm
of AI-advised decision making (Bansal et al., 2019b), defined as follows:

  1. (1)

Given an individual instance of a task, an AI makes a recommendation, and
possibly provides an explanation.

  2. (2)

A human decision maker makes the final decision, drawing on features of the
task, an AI’s recommendation, and (if available) its explanation (Figure 2).

Moreover, we are concerned with the effect of explanations on decision making
performance via objective measures of efficacy such as accuracy, error rate,
or speed of decision-making (Bansal et al., 2019a).

Beyond simply improving the performance of a human-AI team, we are interested
in complementary performance, where the human-AI system performs better (for
example, its accuracy is higher or decisions are made more cheaply) than
either the human or AI separately. While complementarity is not necessary for
AI assistance to be deemed “useful,” for many researchers it is the intuitive
motivation for coupling a human and AI together. Furthermore, if the
consequence of using a human-AI team lowers performance (for whatever metric
is deemed most important), we should have clear understanding of this effect.

Given the focus on complementary performance, we do not attempt to
characterize the utility (or lack thereof) of explanations for other possible
objectives in AI-advised decision making. These include task-centered metrics
such as subjective assessments of efficacy (e.g., self-rated accuracy,
decision confidence) or satisfaction (e.g., cognitive load, helpfulness), and
AI-centered metrics such as trust and reliance on the AI, perceptions of AI
fairness, and understanding of the AI.

Explanations are often characterized based on the scope of information they
convey about an AI model. For decision making, the existing literature largely
studies the influence of local explanations, which provide information about
individual AI recommendations. These include visualizations of model
uncertainty, feature importance, rule-based explanations, example-based
explanations, counterfactual explanations, or natural language rationale
explanations. In contrast, global explanations provide holistic insight into
the entirety of an AI model, for instance by visualizing or detailing a
model’s complete architecture. Faithfulness is another dimension used to
characterize explanations. A faithful explanation is one that accurately
represents the reasoning process behind a model’s prediction. Some
intrinsically interpretable models, such as GA2Ms (Caruana et al., 2015), are
favored because it is (relatively) easy for a human to understand the model’s
behavior. Other explanations are generated by post-hoc analysis, e.g. LIME
(Ribeiro et al., 2016) or SHAP (Lundberg et al., 2018). While these methods
produce local explanations for otherwise inscrutable models, they are by
definition approximations of a model’s true reasoning process, and hence raise
concerns of faithfulness.

In this paper, we present a theory of verification agnostic to these
particular characterizations of explanations (local vs. global, intrinsically
interpretable vs approximate post-hoc). We argue explanation faithfulness is
largely irrelevant; the main issue that affects decision making performance is
whether the explanation helps the human verify the proposed solution.

Figure 3. In decision making contexts where the AI outperforms the human,
improved team performance can often be achieved by providing an AI
explanation, as explanations tend to cause humans to trust the AI regardless
of its correctness. Unfortunately, when the AI errs more than the human, these
explanations can reduce performance due to over-reliance. However,
explanations that support quick verification may still produce complementary
performance by allowing the human to recognize situations where they have made
a mistake—even if the AI is less accurate on average than the human.

##  4\. Verifiability

Figure 4. Tasks in which AI explanations have been shown to provide
verification of an AI recommendation (e.g., maze solving) resemble NP-complete
decision problems. Complementary performance can be achieved in these cases
because the AI does the time-consuming problem solving and the human can
quickly verify the correctness of a solution. Of course, this verification
process is task specific. Sometimes fast verification is only possible for
some answer types; for example, an illegal maze path does not preclude the
existence of a legal path. On average, however, these explanations speed human
verification (Vasconcelos et al., 2023).

We argue that explanations can enable complementary performance in AI-advised
decision making to the extent they allow a decision maker to verify an AI’s
recommendation. Here, we refer to verification of a candidate answer as the
process of determining its correctness. In the psychology literature, this
notion is also called demonstrability and has been associated with the ability
of one individual in a group to convince other group members to switch to a
correct answer on ‘intellective’ problems such as math puzzles (Laughlin and
Ellis, 1986).

Note that some problems may not have a verifiable answer, and many AI
explanations fundamentally cannot satisfy this desideratum and thus do not
effectively support complementary decision making. For instance, one of the
most common types of explanations is local feature importance, e.g., model
coefficients of interpretable models or approximate post-hoc explanations.
These types of explanations which describe a decision-making process can
either align with human intuitions, potentially resulting in confirmation
bias, or contradict human intuitions, in which case a decision maker may often
find it more convenient to explain away any differences. In the best case, the
AI’s explanation reveals a salient misalignment between the AI’s decision
making process and its expected behavior, allowing a decision maker to
disregard the AI’s recommendation. While feature importance explanations may
provide some indication of how much each feature influenced the AI’s decision,
they typically do not allow a decision maker to verify the AI’s
recommendation.

###  4.1. Useful Explanations Allow Verification

To see the importance of verification, consider the task of AI-advised maze
solving (similar to the study in (Vasconcelos et al., 2023))—the goal is to
determine whether a valid path exists between a specified entrance and exit in
the maze. The AI recommends a binary decision (i.e., yes or no), and explains
its decision with a highlighted path through the maze to the exit. Without the
explanation, a human cannot verify the AI’s recommendation short of solving
the maze themselves. Thus the human must adopt a policy of blind trust or
disregard the AI’s advice—neither of which produce complementary performance.
Given the path, however, the decision maker is able to easily verify the AI’s
recommendation, filtering mistakes, and only expending effort to solve mazes
where the AI errs. As shown in Figure 3, this can yield complementary
performance even when the AI is overall less accurate than the human.

Verification given AI explanations in decision making closely resembles
verification of certificates or solutions within canonical NP-complete
computational decision problems (Figure 4). While finding a solution to an NP-
complete problem likely takes exponential time, verifying the certificate can
be done with polynomial computation. In both contexts, while finding a
solution is challenging (e.g., requiring an exhaustive search), checking the
correctness of an explanation enables efficient verification of the answer.

Unfortunately, our maze example is unusual — explanations in the tasks most
commonly considered in XAI studies (e.g., recidivism prediction, medical
applications, sentiment analysis, or deceptive review detection) do not enable
verification. Explanations for these tasks typically include providing model
coefficients or key features, which serve to elucidate the model’s decision
making process rather than enable verification of a proposed decision.

For example, for a sentiment analysis task an AI model may explain itself by
highlighting spans of text which were most influential to its decision (Figure
5 (right)). This may appear reassuring, but can lead to a false sense of trust
when the AI makes mistakes. Indeed, user studies have shown that these
explanations aren’t any more helpful than seeing the AI’s internal confidence
(Bansal et al., 2021). A rare exception where an AI explanation consisting of
an extracted span of text can lead to complementary performance is factoid
question answering from trusted sources (illustrated in Figure 5 (left)); such
a span, even in isolation, does allow verification of answer correctness.

###  4.2. A Spectrum of Verification

Whether an explanation affords verification of a recommendation is usually not
an all or nothing proposition, but rather a point along a spectrum.
Characteristics of the human decision maker, the decision making task, and the
AI explanation may influence where an explanation lies along this spectrum —
hence, determining its utility. On one extreme are contexts that afford no
verification whatsoever, such as problems where the actual outcomes are
independent from the problem features, or where all errors made by the AI
assistance are random rather than systematic. The other end of the spectrum
includes intractable but easily verifiable problems (e.g., the maze solving
task, NP-complete decision problems). But even the maze task affords only one-
sided verifiability: an explanation depicting an illegal path doesn’t imply
that no solution exists.

Explanations do not need to provide complete verification of an AI
recommendation. Rather, by sufficiently lowering the cost of verification,
explanations can provide enough utility to improve team performance
(Vasconcelos et al., 2023). Consider a variant of the maze solving task where
the AI provides an explanation which highlights 90% of the path to the exit
rather than the complete path. We argue this explanation may reduce the
human’s cognitive cost of verifying the AI enough to yield complementary
performance.

Figure 5. The utility of an explanation is not a simple function of
explanation type, as these two examples illustrate. Both use an explanation
style that highlight key spans of text, but their ability to induce
complementary performance is very different. While these explanations can be
useful for factoid QA from trusted sources, they are less successful for
sentiment analysis (Bansal et al., 2021). The difference lies in whether the
human can reliably verify the AI’s answer by solely inspecting the highlighted
text.

Another variation of the maze problem likely leads to a different result.
Suppose the question is not just path existence, but optimality. Specifically,
consider the following variation on the maze task: given a maze with a start
and three alternative exits, which of the exits can be reached with the
shortest valid path from the start? The AI recommends a single exit and
explains by drawing a path from the start to its specified exit. We conjecture
the following hypotheses: 1) this explanation will not lead to complementary
performance for this task, since it doesn’t significantly reduce the cost of
verifying the AI’s recommendation. However, 2) if the explanation consisted of
the shortest valid paths to all three exits, then this would yield
complementary performance.

With feature-based explanations on real-world decision-making tasks, positive
verification is rarely possible. Instead, explanations may be better suited to
convey a fatal flaw in the AI’s decision making process. For example, if an AI
trained to classify lung cancer from CT images justifies its diagnosis by
highlighting an artifact outside the lung, that should raise concern (Caruana
et al., 2015). Some studies have shown that explanations can reveal model
error to yield complementary decision making performance (in particular, when
the revealed errors are sufficiently egregious to indicate strong signals of
model unreliability) (Taesiri et al., 2022; Chen et al., 2023b). However,
across most decision making contexts, we believe there are two reasons why
such explanations frequently fail to produce complementary performance:

  1. (1)

Plausible but wrong: The AI may make a bad decision for a credible reason
(e.g., Figure 5 right). Hence, explanations which clearly reveal model error
to the human are rare.

  2. (2)

Right for the wrong reason: The AI’s recommendation may still be right, even
when its explanation is wrong. In this case, the human may incorrectly reject
the correct answer. However, even if the human escapes that pitfall, they
still need to solve the problem from scratch, and no time is saved.

Many real-world decision making tasks, such as those involving inferences over
human action, can further exhibit aleatoric uncertainty, arising from the
inherent stochastic nature of the dependency between the observed instance
features and the actual decision outcome. In these cases, a correct decision
is not entirely determined by features evident to the human or the AI model,
but is also influenced by irreducible uncertainty within latent factors (e.g.,
environmental, social, or societal). For example, one might be able to
correlate a criminal’s probability for recidivism with factors such as age or
gender, but there is no perfect way to predict the future. As a result, the
utility of verifying a recommendation becomes limited by the extent of
stochasticity present in a task.

In summary, the more easily an explanation enables verification of prediction
correctness, the more likely it is to produce complementary performance. Note
that both the semantic content of the explanation and its presentation (i.e.,
the user interface which presents the explanation) affect the ease of human
verification. However, other factors may also contribute, such as a decision
maker’s desire to engage with explanations within cognitively effortful tasks
(i.e., Need For Cognition (Buçinca et al., 2021)) and the task’s inherent
cognitive load.

###  4.3. Verification and Human Intuitions

Human verification of an AI recommendation (with or without an explanation) is
only possible if a decision maker has sufficient task-specific knowledge to
characterize solutions. For instance, verification of an explanation in the
maze solving task requires implicit assumptions about the human’s
understanding of the task (e.g., what constitutes a valid path through a
maze).

Chen et al. (2023a) present a similar framework to ours, articulating how
human intuitions can interact with AI explanations to improve decision making.
They describe various examples of task-specific human intuitions; for
instance, in prostate cancer diagnosis, intuitions might refer to knowledge of
the location of the prostate in a medical image and the association between
darkness and tumor. In sentiment analysis, intuitions might refer to an
understanding of language and its influence on polarity. Through their
theoretical framework, they suggest human intuitions can lead to complementary
performance in one of two ways: revealing signs of model error and supporting
the discovery of novel knowledge.

In line with this theory, the belief that AI explanations could sufficiently
interact with task-specific human intuitions to reveal model errors and thus
enable complementary performance was a key motivation for providing AI
explanations in AI-advised decision making. In practice, however, this type of
explanation does not appear to produce complementary performance on most
tasks.

##  5\. What is Appropriate Reliance?

Figure 6. We propose a clarification of two notions of reliance behavior
commonly conflated in the literature. Outcome-graded reliance is appropriate
if the human decision maker accepts an AI recommendation when it is correct
and rejects it otherwise; note that this definition is conditioned on the
post-hoc correctness of the AI. We argue this definition is problematic given
its outcome-dependent and nondeterministic nature. In contrast, strategy-
graded reliance is behavior that defers to the AI when its expected
performance is superior to the human’s expected performance.

Previous sections of this paper focus on complementary performance — an ideal
team that performs better than the human or AI alone. However, many
researchers strive instead to design XAI systems that induce appropriate
reliance.

Unfortunately, “appropriate reliance” is not well-defined within the XAI
literature. One common characterization suggests reliance is appropriate if
the human accepts an AI recommendation when the AI is correct, and rejects the
recommendation when the AI is incorrect (Vasconcelos et al., 2023; Bansal et
al., 2021; Buçinca et al., 2021; Wang and Yin, 2021; Yang et al., 2020;
Schemmer et al., 2023). Reliance is therefore inappropriate when the human
accepts the recommendation when the AI is incorrect (over-reliance), and when
the human rejects the recommendation when the AI is correct (under-reliance).
This characterization captures a notion of outcome-graded reliance (see Figure
6 left), and is evaluable given actual decision outcomes. Unfortunately, we
suggest outcome-graded reliance is inadequate to measure the perceived
improvements in decision making performance. There are two major problems with
the outcome-graded definition:

  * •

Post-hoc: It is impossible to know if one’s reliance is ‘appropriate’ until
after seeing the final result. Was it ‘inappropriate’ to see the best doctor
in the world, if one happens to get an unlucky outcome?

  * •

Nondeterministic: Consider two identical examples (e.g., bank loan
applications) whose input features are exactly the same, and suppose the AI
provides the same recommendation in both cases (e.g., unlikely to default).
But suppose that one borrower ends up defaulting, while the other repays the
loan. The definition says it is ‘appropriate’ to rely on the AI’s advice on
one case but not on the other — even though they are indistinguishable!

Instead, consider an alternative definition, strategy-graded reliance, where
reliance is appropriate if the human accepts an AI recommendation when the AI
is expected to outperform the human, and rejects otherwise (see Figure 6
right). Unlike outcome-graded reliance, strategy-graded reliance is neither
post-hoc nor nondeterministic; it considers the appropriateness of reliance
given the expected relative performance of the human and the AI. The optimal
strategy is to rely on the party most likely to have the correct answer. A key
question here is “Upon what information is that expectation computed?” There
are several possibilities.

  * •

Past performance: If past experience shows the AI is more likely to be correct
than the human, it might be appropriate to defer to the AI even without
information about this particular decision instance. Note this policy cannot
produce complementary performance.

  * •

Previous characteristics \+ instance features: Conditioning on the current
instance (i.e., specific details of the task at hand) can lead to
complementary performance. For instance, a camera’s automatic exposure mode is
a convenient form of automation that is usually accurate. Yet it can err in
unusual situations, such as snowy landscapes, so experienced photographers
often prefer to override the automation when they recognize these scenarios.
When automated, this type of conditioning resembles a human-AI delegation
workflow.

  * •

Previous characteristics \+ the AI’s recommendation: Conditioning on the AI’s
recommendation allows the human to adopt a policy of the form “I know the AI
is conservative and very unlikely to err with a false positive, so I will
accept positive recommendations and only scrutinize instances when the AI
offers a negative recommendation.”

  * •

Previous characteristics \+ the AI’s explanation: In this paper, we have
argued this condition rarely improves upon the previous strategy, and only
when the explanation supports verification.

In contrast to complementary performance, which refers to the team’s measured
performance, both notions of reliance define an attribute of the human’s
behavior relative to the AI. Furthermore, while a policy of strategy-guided
reliance will hopefully lead to complementary performance, it’s not guaranteed
to do so. In particular, if the human’s estimates of their relative accuracy
(compared to the AI) are poor, team performance may drop.

We believe the strategy-graded definition of reliance is the better objective.
To illustrate the shortcomings of outcome-graded reliance, consider a decision
making task in which the human is historically 60% accurate, while the AI is
99.999% accurate. On any given instance of the task, if the human is uncertain
of the answer, is it appropriate to rely on the AI’s recommendation?
Intuitively, the answer seems a clear ‘yes’. But if the AI is later found
incorrect, the outcome-graded definition says “Inappropriate,” while the
strategy-graded definition matches intuition and says “Appropriate.”

Outcome-graded reliance is similar to complementary performance in that both
qualities can only be measured post hoc. However, there are subtle differences
between these notions, beyond the fact that one measures a pattern of human
behavior and the other the performance of a human-AI team. For example,
imagine a AI-assisted image classification task with 1,000 potential classes.
Suppose two individuals, Avery and Blake, are 80% and 50% accurate at the task
alone, respectively, and the AI is 10% accurate. Luckily, the AI outputs
verifiable explanations, so both Avery and Blake can perfectly tell when the
AI is correct. Both follow a policy of accepting the AI’s recommendation when
it is correct, and otherwise solving the problem themselves. Both their
strategies lead to very different expected team performance: 82% for Avery and
55% for Blake.111Expected accuracy can be calculated as $E[acc_{Avery}]$ =
$0.1+0.8\times(1-0.1)=0.82$ for Avery and $E[acc_{Blake}]$ =
$0.1+0.5\times(1-0.1)=0.55$ for Blake. Yet the definition suggests both have
(near) perfect outcome-graded reliance because their policy dictates following
the AI when it is correct and not when it is incorrect. Moreover, in most
scenarios in which the human and AI both err, the reliance is technically
‘appropriate’ as the human is likely to reject the AI recommendation and
select a different, albeit incorrect answer.222 There is a small, non-zero
probability of each individual trivially selecting the same incorrect answer
as the AI. With many possible answers (e.g., 1,000 classes) and assuming that
errors are uniformly distributed between the 999 possible incorrect answers,
the probability of the human and AI both erring and then selecting the same
incorrect answer approaches zero as the answer space grows (e.g., Avery:
$((1-0.1)\times\frac{1}{999})\times((1-0.8)\times\frac{1}{999})<1e^{-6}$).

Given the limitations of the outcome-graded definition of appropriate
reliance, we suggest researchers focus on the strategy-graded, or eschew the
term ‘appropriate reliance’ altogether. We argue overall performance is a
better objective when evaluating a human-AI team architecture — whether it
involves explanations or not. Furthermore, complementary performance is the
holy grail.

##  6\. Discussion

We have argued that verifiability of AI explanations is crucial for realizing
complementary performance in AI-advised decision making. In this section, we
consider the connection between verifiability and faithfulness (§6.1),
alternative paradigms of human-AI collaboration which might yield
complementary performance (§6.2), and the aims of interpretability beyond
complementary performance and decision making support (§6.3). Finally, we
contextualize the verifiability of AI explanations for human-AI decision
making among established research on demonstrability and social combination
processes in collective group (human-human) decision making (§6.4).

###  6.1. Faithfulness vs. Verifiability

Some may argue that faithfulness is an intrinsic requirement of explanations.
If an explanation does not accurately reflect the underlying prediction or
decision making process, it feels deceptive. Certainly, faithful explanations
may help a decision maker build a better model of an AI’s capabilities over
time. And of course, faithfulness is often a desirable property for
explanations. We argue, however, that for the context of a single isolated
decision, faithfulness is not as important as verifiability.

Consider an unfaithful explanation that facilitates verification. This is
hard, but not impossible. Indeed, Kahneman (2011) argues people often solve
problems using fast “System 1” reasoning and then justify their answers with
slower “System 2” reasoning. Since the System 2 rationalization fails to match
how the answer was discovered, one might call it unfaithful. But if it affords
verification then it will suffice to aid the decision maker. On the other
hand, consider faithful explanations that do not facilitate verification. Some
may be too costly to verify. For instance, showing the numerical weights of a
large linear model may provide insight into how a decision was formed, but is
infeasible to use for absolute verification. Other explanations may be
faithful to the underlying model but fail to capture the entire picture (e.g.,
Figure 5 (right)), or are stymied by uncertainty in the decision making task
(e.g., §4.2).

###  6.2. Paradigms of XAI-Advised Decision Making

A typical paradigm of AI-advised decision making, and one that we focus our
analysis on, involves an AI that provides a recommendation and possibly an
explanation. However, other paradigms are also possible. For example, in lieu
of recommending any singular decision, an evaluative XAI might instead
generate and present evidence to support or refute several of the options,
forcing the human to remain active in forming the final decision. Such
hypothesis-driven support may help mitigate fixation, anchoring and associated
automation biases by assisting decision makers to better consider the trade-
offs between multiple options.

Evaluative XAI is an example of cognitive forcing, a class of strategies that
has been proposed to improve the efficacy of AI explanations for decision
making by introducing interventions which elicit greater decision maker
engagement with AI explanations and facilitate deeper thinking about
individual task instances (Buçinca et al., 2021; Miller, 2023; Gajos and
Mamykina, 2022). One associated drawback of cognitive forcing is the increased
cognitive load on the decision maker, increasing accuracy at the expense of
decision making speed, human effort, or human satisfaction. By reducing the
costs of engagement, verifiable explanations may improve the effectiveness of
these strategies.

Achieving verifiability in practical applications may also involve moving
beyond the conventional single-turn, static AI explanations, and toward
interactive explanations that can be generated on-demand and contextualized to
individual decision makers. Perspectives from social science theory on
explanations tend to support interactive explanations (Miller, 2019; Lombrozo,
2006; Mittelstadt et al., 2019), and XAI researchers have similarly encouraged
interactivity in AI explanations (Bansal et al., 2021; Adadi and Berrada,
2018). However, interactive explanations come at a cost—they often require
more time and effort to use effectively (Cheng et al., 2019), and as a result
can further reinforce detrimental cognitive biases (Liu et al., 2021).

###  6.3. Beyond Complementary Performance

While complementary performance may represent a holy grail for human-AI teams,
explanations can also serve other desiderata. In high-risk settings or where
regulatory compliance mandates the presence of human judgment, and yet the AI
outperforms humans, it may be desirable for the human-AI team to achieve
performance comparable to the AI alone; explanations can help in this regard,
since they tend to make the audience trust the recommendation (Bansal et al.,
2021). Moreover, in some scenarios, objective measures of performance such as
accuracy may be of second importance to task efficiency, cognitive load, or
decision fairness and transparency.

Rather than supporting decision making over individual instances, AI
explanations may be better equipped at facilitating holistic AI processes such
as model development and debugging, regulatory audit, and knowledge discovery
(Figure 1). To improve models throughout development and after release,
explanations have been used to uncover spurious patterns in learned models
(Caruana et al., 2015) and enable interactive debugging processes (Zylberajch
et al., 2021). Explanations also offer intelligibility of AI models desirable
for ensuring compliance with organizational policies, legal imperatives, and
government regulation. For instance, one provision in the European Union’s
General Data Protection Regulation (GDPR) describes an individual’s “right to
explanation,” particularly in automated decision-making or profiling scenarios
that may significantly influence the individual, e.g., legally or financially.
Finally, explanations may help decision makers develop new knowledge about a
decision making task (Das and Chernova, 2020; Chen et al., 2023a). If an AI
model significantly outperforms human decision makers, and its explanations
can effectively convey learned signals or patterns in the task features, then
humans may be able to replicate such behavior to improve their own decision
making on future instances.

###  6.4. Verifiability in Collective Decision Making

AI-advised decision making may be beneficially viewed as an extension of
existing social and cognitive psychology research, which has long studied the
decision-making processes of human groups. Many of these prior studies
together corroborate a generalized theory that groups collectively perform
better than the average (and many times, best) individual on many types of
problem-solving tasks (Laughlin and Ellis, 1986; Laughlin and Adamopoulos,
1980; Hill, 1982; Laughlin and Johnson, 1966; Laughlin et al., 2006; Carey and
Laughlin, 2012; Stasser and Dietz-Uhler, 2001; Steiner, 1972; Hackman and
Morris, 1975; Davis, 1992). One relevant line of work postulates a
relationship between task type and how human groups would arrive at a
collective decision (Laughlin and Ellis, 1986; Laughlin and Adamopoulos, 1980;
Hill, 1982). These studies suggest that the relative superiority of group
decision making performance over independent individuals can be explained by
the demonstrability of a solution for any given task. Initially coined by
Laughlin and Adamopoulos (1980), demonstrability requires four conditions: (1)
Group consensus on a verbal (e.g., vocabulary, syntax) or mathematical system
(e.g., primitives, axioms, permissible operations like algebra); (2)
Sufficient information for a solution within the system; (3) Group members who
cannot solve the problem must have sufficient knowledge to recognize and
accept a proposed solution; (4) The correct team member must have sufficient
ability, motivation, and time to demonstrate the correct solution to other
members.

Demonstrability is therefore contingent on the information system and the
extent to which the task type affords recognition of correct solutions.
Laughlin and Adamopoulos (1980) define a continuum of tasks that exhibit
greater or lesser demonstrability, anchored by intellective tasks (those with
a demonstrably correct answer with a specific verbal or mathematical
conceptual system, similar to tasks with verifiable explanations) and
judgmental tasks (those involving evaluative, behavioral, or aesthetic
judgments for which there does not exist a demonstrably correct, or
verifiable, answer). The criterion for success is different between the tasks:
agreement about the correct answer in the former, and achievement of consensus
on a collective decision for the latter.

Laughlin and Ellis (1986) showed that human groups, which were working to
solve intellective tasks, only required a single group member to have the
correct answer, as long as the solution was demonstrable (could be explained
in a verifiable manner). Examples of these tasks include puzzles (e.g.,
“cannibals and missionaries,” Tower of Hanoi, or Sudoku) or mathematical
problems. Note that these domains are very similar to the cases we addressed
earlier (e.g., maze solving, puzzles, or factual QA), which are also
intellective and where AI explanations have been shown to yield complementary
team performance. In contrast, tasks such as recidivism prediction and loan
application evaluation, are examples of judgmental tasks, where no
demonstrable (i.e., verifiable) solution may exist.

Despite their parallels, current human-AI teams also differ from human groups
in classical collective intelligence:

#### Unequal distribution of responsibility and power

In human-AI teams, humans typically retain agency and are responsible for
making the final decision. Moreover, in contrast to the social psychology
literature, which often studies collective reasoning processes with larger
groups of humans, the human-AI teams considered by the XAI literature
typically consist of just one human and one AI member. Future XAI work might
consider if there are parallel results to psychologists’ observations that the
number of human group members required to arrive at a quality collective
response is inversely proportional to the demonstrability afforded by the
intellective tasks (Laughlin and Ellis, 1986).

#### Rich interaction modalities benefit collective intelligence

Numerous psychology studies have shown that human explanations are
interactive, a type of dialog (Miller, 2019). Yet most existing XAI systems
are static and single-turn. Moreover, current interaction paradigms are
unilateral; explanations are provided by AI models to the human, but rarely
are AI systems able to incorporate or adjust to a human explanation. Recent
interactive explanations have begun to revise this narrative (Weld and Bansal,
2019), with abundant opportunities for future work.

##  7\. Conclusion

There are many reasons to develop AI systems that can explain themselves, but
it has become apparent that different uses require different types of
explanations. It is therefore important to ensure a given type of explanation
is well-aligned with its use. In this paper, we focused specifically on the
utility of AI explanations for instance-level human-AI decision making. We
proposed that explanations can engender complementary performance in decision
making to the extent the explanation allows a human decision maker to verify
correctness of the AI’s recommendation. Unfortunately, most kinds of
explanations—notably those which elucidate the AI’s reasoning process—do not
facilitate verification and hence are not helpful for decision making. We
further teased apart two notions of appropriate reliance, outcome-graded and
strategy-graded, advocating use of the strategy-graded formulation, which
sidesteps the problem of nondeterminism and aligns with intuitions of
appropriateness. In the pursuit of complementary team performance, we should
reevaluate the extent to which the AI explanations we often integrate into our
decision making systems facilitate verification.

###### Acknowledgements.

This research was supported by ONR grant N00014-21-1-2707 and the Allen
Institute for Artificial Intelligence (AI2). We thank Gagan Bansal, Tongshuang
Wu, and the members of the Lab for Human-AI Interaction at the University of
Washington for formative discussions. Chacha Chen, Hal Daumé III, Madeleine
Grunde-McLaughin, Rao Kambhampati, Sunnie Kim, Raymond Mooney, Giang Nguyen,
Marissa Radensky, Mark Riedl, Suzanna Sia, and Chenhao Tan provided helpful
comments on the manuscript. We also thank the anonymous reviewers for their
insightful feedback.

## References

  * (1)
  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI).  _IEEE Access_ 6 (2018), 52138–52160.
  * Alam and Mueller (2021) Lamia Alam and Shane Mueller. 2021.  Examining the Effect of Explanation on Satisfaction and Trust in AI Diagnostic Systems.  _BMC medical informatics and decision making_ 21, 1 (2021), 178\.
  * Alqaraawi et al. (2020) Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020\.  Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 275–285.
  * Alufaisan et al. (2021) Yasmeen Alufaisan, Laura Marusich, Jonathan Bakdash, Yan Zhou, and Murat Kantarcioglu. 2021.  Does Explainable Artificial Intelligence Improve Human Decision-Making?  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 8 (05 2021), 6618–6626.
  * Bansal et al. (2019a) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019a.  Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ 7, 1 (Oct. 2019), 2–11.
  * Bansal et al. (2019b) Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, and Eric Horvitz. 2019b.  Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 01 (July 2019), 2429–2437.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021\.  Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages.
  * Bell et al. (2022) Andrew Bell, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich. 2022.  It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 248–266.
  * Bilgic and Mooney (2005) Mustafa Bilgic and Raymond J. Mooney. 2005.  Explaining Recommendations: Satisfaction vs. Promotion. In _Proceedings of Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research at the 2005 International Conference on Intelligent User Interfaces_ _(IUI ’05)_. Association for Computing Machinery, San Diego, CA, 1–6.
  * Biran and McKeown (2017) Or Biran and Kathleen McKeown. 2017.  Human-Centric Justification of Machine Learning Predictions. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence_. International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia, 1461–1467.
  * Bowman et al. (2022) Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan. 2022\.  Measuring Progress on Scalable Oversight for Large Language Models.
  * Buçinca et al. (2020) Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020.  Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 454–464.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021.  To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 188 (April 2021), 21 pages.
  * Bunt et al. (2012) Andrea Bunt, Matthew Lount, and Catherine Lauzon. 2012\.  Are Explanations Always Important? A Study of Deployed, Low-Cost Intelligent Interactive Systems. In _Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces_ (Lisbon, Portugal) _(IUI ’12)_. Association for Computing Machinery, New York, NY, USA, 169–178.
  * Carey and Laughlin (2012) Harold R. Carey and Patrick R. Laughlin. 2012.  Groups perform better than the best individuals on letters-to-numbers problems: Effects of induced strategies.  _Group Processes & Intergroup Relations_ 15, 2 (2012), 231–242.
  * Carton et al. (2020) Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2020.  Feature-Based Explanations Don’t Help People Detect Misclassifications of Online Toxicity.  _Proceedings of the International AAAI Conference on Web and Social Media_ 14 (May 2020), 95–106.
  * Caruana et al. (2015) Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015\.  Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-Day Readmission. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Sydney, NSW, Australia) _(KDD ’15)_. Association for Computing Machinery, New York, NY, USA, 1721–1730.
  * Chandrasekaran et al. (2018) Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018.  Do Explanations Make VQA Models More Predictable to a Human?. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 1036–1042.
  * Chen et al. (2023a) Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. 2023a.  Machine Explanations and Human Understanding. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 1\.
  * Chen et al. (2023b) Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal. 2023b.  Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations.  _Proc. ACM Hum.-Comput. Interact._ 7, CSCW2, Article 370 (Oct. 2023), 32 pages.
  * Cheng et al. (2019) Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F. Maxwell Harper, and Haiyi Zhu. 2019.  Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, Glasgow Scotland Uk, 1–12.
  * Chromik et al. (2021) Michael Chromik, Malin Eiband, Felicitas Buchner, Adrian Krüger, and Andreas Butz. 2021\.  I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI. In _26th International Conference on Intelligent User Interfaces_. ACM, College Station TX USA, 307–317.
  * Das and Chernova (2020) Devleena Das and Sonia Chernova. 2020.  Leveraging Rationales to Improve Human Task Performance. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 510–518.
  * Davis (1992) James H Davis. 1992\.  Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples 1950–1990.  _Organizational Behavior and Human Decision Processes_ 52, 1 (1992), 3–38.
  * Eiband et al. (2019) Malin Eiband, Daniel Buschek, Alexander Kremer, and Heinrich Hussmann. 2019.  The Impact of Placebic Explanations on Trust in Intelligent Systems. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI EA ’19)_. Association for Computing Machinery, New York, NY, USA, 1–6.
  * Feng and Boyd-Graber (2019) Shi Feng and Jordan Boyd-Graber. 2019.  What Can AI Do for Me? Evaluating Machine Learning Interpretations in Cooperative Play. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ (Marina del Ray, California) _(IUI ’19)_. Association for Computing Machinery, New York, NY, USA, 229–239.
  * Gajos and Mamykina (2022) Krzysztof Z. Gajos and Lena Mamykina. 2022.  Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning. In _27th International Conference on Intelligent User Interfaces_ (Helsinki, Finland) _(IUI ’22)_. Association for Computing Machinery, New York, NY, USA, 794–806.
  * Ghai et al. (2021) Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021.  Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers.  _Proc. ACM Hum.-Comput. Interact._ 4, CSCW3, Article 235 (Jan. 2021), 28 pages.
  * Ghassemi et al. (2021) Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L. Beam. 2021\.  The False Hope of Current Approaches to Explainable Artificial Intelligence in Health Care.  _The Lancet Digital Health_ 3, 11 (Nov. 2021), e745–e750.
  * González et al. (2021) Ana Valeria González, Gagan Bansal, Angela Fan, Yashar Mehdad, Robin Jia, and Srinivasan Iyer. 2021.  Do Explanations Help Users Detect Errors in Open-Domain QA? An Evaluation of Spoken vs. Visual Explanations. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 1103–1116.
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  The Principles and Limits of Algorithm-in-the-Loop Decision Making.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (Nov. 2019), 1–24.
  * Hackman and Morris (1975) J Richard Hackman and Charles G Morris. 1975.  Group tasks, group interaction process, and group performance effectiveness: A review and proposed integration.  _Advances in experimental social psychology_ 8 (1975), 45–99.
  * Hill (1982) Gayle W Hill. 1982\.  Group versus individual performance: Are N+ 1 heads better than one?  _Psychological bulletin_ 91, 3 (1982), 517\.
  * Horne et al. (2019) Benjamin D. Horne, Dorit Nevo, John O’Donovan, Jin-Hee Cho, and Sibel Adalı. 2019.  Rating Reliability and Bias in News Articles: Does AI Assistance Help Everyone?  _Proceedings of the International AAAI Conference on Web and Social Media_ 13, 01 (July 2019), 247–256.
  * Jacobs et al. (2021) Maia Jacobs, Melanie F. Pradier, Thomas H. McCoy, Roy H. Perlis, Finale Doshi-Velez, and Krzysztof Z. Gajos. 2021.  How Machine-Learning Recommendations Influence clinician Treatment Selections: The Example of Antidepressant Selection.  _Translational Psychiatry_ 11, 1 (Feb. 2021), 108.
  * Jacovi and Goldberg (2021) Alon Jacovi and Yoav Goldberg. 2021.  Aligning Faithful Interpretations with their Social Attribution.  _Transactions of the Association for Computational Linguistics_ 9 (March 2021), 294–310.
  * Jesus et al. (2021) Sérgio Jesus, Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, and João Gama. 2021.  How Can I Choose An Explainer? An Application-grounded Evaluation of Post-hoc Explanations. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 805–815.
  * Kahneman (2011) Daniel Kahneman. 2011\.  _Thinking, fast and slow_.  Farrar, Straus and Giroux, New York.
  * Kiani et al. (2020) Amirhossein Kiani, Bora Uyumazturk, Pranav Rajpurkar, Alex Wang, Rebecca Gao, Erik Jones, Yifan Yu, Curtis P. Langlotz, Robyn L. Ball, Thomas J. Montine, Brock A. Martin, Gerald J. Berry, Michael G. Ozawa, Florette K. Hazard, Ryanne A. Brown, Simon B. Chen, Mona Wood, Libby S. Allard, Lourdes Ylagan, Andrew Y. Ng, and Jeanne Shen. 2020.  Impact of a Deep Learning Assistant on the Histopathologic Classification of Liver Cancer.  _npj Digital Medicine_ 3, 1 (Dec. 2020), 23\.
  * Kim et al. (2023a) Joon Sik Kim, Valerie Chen, Danish Pruthi, Nihar B. Shah, and Ameet Talwalkar. 2023a.  Assisting Human Decisions in Document Matching.  arXiv:2302.08450
  * Kim et al. (2022) Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. 2022.  HIVE: Evaluating the Human Interpretability of Visual Explanations. In _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII_ (Tel Aviv, Israel). Springer-Verlag, Berlin, Heidelberg, 280–298.
  * Kim et al. (2023b) Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023b.  ”Help Me Help the AI”: Understanding How Explainability Can Support Human-AI Interaction. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 250, 17 pages.
  * Lai et al. (2020) Vivian Lai, Han Liu, and Chenhao Tan. 2020.  ”Why is ’Chicago’ Deceptive?” Towards Building Model-Driven Tutorials for Humans. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.
  * Lai and Tan (2019) Vivian Lai and Chenhao Tan. 2019.  On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Atlanta, GA, USA) _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 29–38.
  * Laughlin and Adamopoulos (1980) Patrick R Laughlin and John Adamopoulos. 1980.  Social combination processes and individual learning for six-person cooperative groups on an intellective task.  _Journal of Personality and Social Psychology_ 38, 6 (1980), 941.
  * Laughlin and Ellis (1986) Patrick R Laughlin and Alan L Ellis. 1986.  Demonstrability and social combination processes on mathematical intellective tasks.  _Journal of Experimental Social Psychology_ 22, 3 (1986), 177–189.
  * Laughlin et al. (2006) Patrick R Laughlin, Erin C Hatch, Jonathan S Silver, and Lee Boh. 2006.  Groups perform better than the best individuals on letters-to-numbers problems: effects of group size.  _Journal of Personality and social Psychology_ 90, 4 (2006), 644.
  * Laughlin and Johnson (1966) Patrick R Laughlin and Homer H Johnson. 1966.  Group and individual performance on a complementary task as a function of initial ability level.  _Journal of Experimental Social Psychology_ 2, 4 (1966), 407–414.
  * Lee et al. (2022) Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E. Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael Bernstein, and Percy Liang. 2022\.  Evaluating Human-Language Model Interaction.
  * Liu et al. (2021) Han Liu, Vivian Lai, and Chenhao Tan. 2021.  Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (Oct. 2021), 1–45.
  * Lombrozo (2006) Tania Lombrozo. 2006\.  The Structure and Function of Explanations.  _Trends in Cognitive Sciences_ 10, 10 (2006), 464–470.
  * Lundberg et al. (2018) Scott M. Lundberg, Bala Nair, Monica S. Vavilala, Mayumi Horibe, Michael J. Eisses, Trevor Adams, David E. Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, and Su-In Lee. 2018.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  _Nature biomedical engineering_ 2, 10 (Oct. 2018), 749–760.
  * Miller (2019) Tim Miller. 2019\.  Explanation in Artificial Intelligence: Insights From the Social Sciences.  _Artificial Intelligence_ 267 (2019), 1–38.
  * Miller (2023) Tim Miller. 2023\.  Explainable AI is Dead, Long Live Explainable AI! Hypothesis-Driven Decision Support Using Evaluative AI. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 333–342.
  * Mittelstadt et al. (2019) Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019\.  Explaining Explanations in AI. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Atlanta, GA, USA) _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 279–288.
  * Nguyen et al. (2021) Giang Nguyen, Daeyoung Kim, and Anh Nguyen. 2021.  The effectiveness of feature attribution methods and its correlation with automatic evaluation scores.  _Advances in Neural Information Processing Systems_ 34 (2021), 26422–26436.
  * Nourani et al. (2021) Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021.  Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In _26th International Conference on Intelligent User Interfaces_. ACM, College Station TX USA, 340–350.
  * Parrish et al. (2022) Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel Bowman. 2022.  Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions. In _Proceedings of the First Workshop on Learning with Natural Language Supervision_. Association for Computational Linguistics, Dublin, Ireland, 17–28.
  * Poursabzi-Sangdeh et al. (2021) Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021.  Manipulating and Measuring Model Interpretability. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 237, 52 pages.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Francisco, California, USA) _(KDD ’16)_. Association for Computing Machinery, New York, NY, USA, 1135–1144.
  * Rudin (2019) Cynthia Rudin. 2019\.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1, 5 (2019), 206–215.
  * Schemmer et al. (2023) Max Schemmer, Niklas Kuehl, Carina Benz, Andrea Bartos, and Gerhard Satzger. 2023.  Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 410–422.
  * Sivaraman et al. (2023) Venkatesh Sivaraman, Leigh A Bukowski, Joel Levin, Jeremy M. Kahn, and Adam Perer. 2023\.  Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 754, 18 pages.
  * Stasser and Dietz-Uhler (2001) Garold Stasser and Beth Dietz-Uhler. 2001.  _Collective Choice, Judgment, and Problem Solving_.  John Wiley & Sons, Ltd, Chapter 2, 31–55.
  * Steiner (1972) Ivan Dale Steiner. 1972\.  _Group process and productivity_.  Academic press, New York, NY, USA.
  * Taesiri et al. (2022) Mohammad Reza Taesiri, Giang Nguyen, and Anh Nguyen. 2022\.  Visual correspondence-based explanations improve AI robustness and human-AI team accuracy.  _Advances in Neural Information Processing Systems_ 35 (2022), 34287–34301.
  * Tan (2022) Chenhao Tan. 2022\.  On the Diversity and Limits of Human Explanations. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 2173–2188.
  * Tschandl et al. (2020) Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al. 2020\.  Human–computer collaboration for skin cancer recognition.  _Nature Medicine_ 26, 8 (2020), 1229–1234.
  * Vasconcelos et al. (2023) Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. 2023.  Explanations Can Reduce Overreliance on AI Systems During Decision-Making.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (2023), 1–38.
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019\.  Designing Theory-Driven User-Centric Explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.
  * Wang and Yin (2021) Xinru Wang and Ming Yin. 2021.  Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making. In _26th International Conference on Intelligent User Interfaces_. ACM, College Station TX USA, 318–328.
  * Weerts et al. (2019) Hilde J. P. Weerts, Werner van Ipenburg, and Mykola Pechenizkiy. 2019\.  A Human-Grounded Evaluation of SHAP for Alert Processing.  arXiv:1907.03324
  * Weld and Bansal (2019) Daniel S. Weld and Gagan Bansal. 2019.  The challenge of crafting intelligible intelligence.  _Commun. ACM_ 62 (June 2019), 70–79.
  * Yang et al. (2020) Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L. Arendt. 2020.  How Do Visual Explanations Foster End Users’ Appropriate Trust in Machine Learning?. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 189–201.
  * Yu et al. (2019) Kun Yu, Shlomo Berkovsky, Ronnie Taib, Jianlong Zhou, and Fang Chen. 2019.  Do I Trust My Machine Teammate? An Investigation from Perception to Decision. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ (Marina del Ray, California) _(IUI ’19)_. Association for Computing Machinery, New York, NY, USA, 460–468.
  * Zhang et al. (2020) Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020\.  Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 295–305.
  * Zylberajch et al. (2021) Hugo Zylberajch, Piyawat Lertvittayakumjorn, and Francesca Toni. 2021\.  HILDIF: Interactive Debugging of NLI Models Using Influence Functions. In _Proceedings of the First Workshop on Interactive Learning for Natural Language Processing_. Association for Computational Linguistics, Online, 1–6.