# A Statistical Framework for Measuring AI Reliance

Ziyang Guo<EMAIL_ADDRESS>0009-0004-4200-6774 Northwestern
Univerisity633 Clark StreetEvanstonIllinoisUSA60208 ,  Yifan wu  Northwestern
Univerisity633 Clark StreetEvanstonIllinoisUSA60208<EMAIL_ADDRESS>,  Jason Hartline<EMAIL_ADDRESS>Northwestern Univerisity633 Clark
StreetEvanstonIllinoisUSA60208 and  Jessica Hullman<EMAIL_ADDRESS>Northwestern Univerisity633 Clark StreetEvanstonIllinoisUSA60208

(2024; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Humans frequently make decisions with the aid of artificially intelligent (AI)
systems. A common pattern is for the AI to recommend an action to the human
who retains control over the final decision. Researchers have identified
ensuring that a human has appropriate reliance on an AI as a critical
component of achieving complementary performance. We argue that the current
definition of appropriate reliance used in such research lacks formal
statistical grounding and can lead to contradictions. We propose a formal
definition of reliance, based on statistical decision theory, which separates
the concepts of reliance as the probability the decision-maker follows the
AI’s prediction from challenges a human may face in differentiating the
signals and forming accurate beliefs about the situation. Our definition gives
rise to a framework that can be used to guide the design and interpretation of
studies on human-AI complementarity and reliance. Using recent AI-advised
decision making studies from literature, we demonstrate how our framework can
be used to separate the loss due to mis-reliance from the loss due to not
accurately differentiating the signals. We evaluate these losses by comparing
to a baseline and a benchmark for complementary performance defined by the
expected payoff achieved by a rational decision-maker facing the same decision
task as the behavioral decision-makers.

Machine learning, reliance, decision making, rational decision-maker

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Human-centered computing HCI theory, concepts and models††ccs: Computing
methodologies Machine learning

##  1\. Introduction

AI-advised decision making, in which a human decision-maker has access to the
recommendation of an artificial intelligence (AI system) and can choose
whether or not to follow it, is often preferred as a means of retaining human
control (BWZ+, 21) in deploying predictive models. The motivation behind this
approach is complementary performance; i.e., the human-AI team can outperform
the AI or the human alone. However, many studies have shown that human-AI
teams under-perform the AI alone in tasks where the AI’s accuracy is higher
than humans (BWZ+, 21; BLGG, 20; BSO, 15; GC, 19; JPMJ+, 21; LT, 19; VW, 19;
Kon, 01). One solution to this problem is to identify ways to ensure that the
human, as the final decision-maker, has appropriate reliance on AI.
Appropriate reliance is typically defined as submitting the AI prediction when
it is correct and not submitting it when it is not correct.

We argue that this definition of reliance lacks formal statistical grounding,
leading to contradictions. For example, situations in which a human-AI team
outperforms the human alone but underperforms the AI alone suggest that the
human underrelies on the AI (BWZ+, 21). However, when researchers apply the
above definition of appropriate reliance to their experimental results, they
discover that the primary source of performance loss stems from the humans
accepting the AI’s inaccurate predictions (BSO, 15; JPMJ+, 21; LT, 19),
considered over-reliance by the conventional definition.

Implicit in discussions of complementarity are assumptions of a human with
some internal model of the data-generating process and an AI with its own
model. Studying reliance implies that the human consults the AI prediction,
infers the probability that its decision is correct, then decides whether it
is worth following its prediction. Problems arise because defining appropriate
reliance as submitting the AI’s prediction when it is correct and rejecting it
when it is not confounds two challenges a human may face in an AI-advised
decision-making: that of forming correct beliefs about the probability that
the AI is correct, and that of making the optimal decision about whether to
follow the AI conditional on one’s beliefs. Without a definition that allows
separation of different sources of performance loss, it is unclear what can be
done to improve the performance of the human-AI team. For example, if the
human has inaccurate beliefs about the probability that the AI is correct,
this might stem from a lack of information about the relationship prior
probability that the AI is correct (potentially fixable by providing the AI’s
accuracy on held-out data (YWVW, 19)), or from their failure to arrive at an
accurate estimate of the AI’s probability of being correct (potentially
fixable via cognitive forcing functions (BMG, 21; FCL, 21) or better
explanations (BWZ+, 21)). If the human correctly perceives the accuracy of the
AI model, but uses the wrong decision rule to decide when to follow its
prediction, then the human may not understand the utility of different
outcomes of the task (e.g., a differential cost of using the AI’s prediction
versus generating their own), or the researcher studying real-world human-AI
teams may have misidentified the utility function.

Another issue with the conventional definition of appropriate reliance is that
it is a binary measure. Consequently, researchers cannot distinguish whether
the human decision-maker mistakenly used (or did not use) the AI’s prediction
in a situation where (A) the probability that relying on their own judgment
would have been correct is similar to the probability that the AI was correct
versus (B) very different. Intuitively, over-reliance is a bigger concern in B
than in case A. We argue that the concept of reliance should be characterized
within a continuous payoff space to allow for more fine-grained assessment.

We propose a formal definition of AI reliance. Following previous work on
generating benchmarks for studies of information displays (WGM+, 23), our
approach is grounded in statistical decision theory. Our definition separates
the concepts of a reliance level (the probability that the human decision-
maker goes with the AI prediction) from the belief updating that a rational
decision-maker is expected to do upon viewing an instance and associated AI
prediction. The framework we provide defines a benchmark for complementary
performance representing the maximum attainable performance with the
corporation of AI and human and a baseline for complementary performance
representing the maximum performance without any corporation. We apply the
framework to three well-regarded AI-advised decision making experiments from
literature (BWZ+, 21; LT, 19; FCL, 21). In all three cases, we show 1) that
examining the results against the baseline and benchmark for complementary
performance better reveals the the limits of human behavioral performance and
2) specific sources of behavioral loss that help explain the experiment
results but were not accounted for by the original interpretations of the
results.

##  2\. Formulating Assumptions for Studying Reliance

In AI-advised decision-making scenarios (BNK+, 19; WYAL, 19), the human makes
a decision about a set of instances with the assistance of an AI prediction.
In formulating our definition of reliance below, we make several assumptions
about this scenario:

  1. (1)

The human makes their own prediction about each instance prior to seeing the
AI prediction for that instance.

  2. (2)

The human consults the AI prediction prior to making their decision.

There are two benefits to making these assumptions for AI-advised decision-
making experiments. First, the assumptions ensure that participants neither
anchor solely on the AI predictions (completely neglecting to consider their
own predictions) nor that they neglect to consult the AI prediction at all
(BMG, 21; FCL, 21). It is difficult to conceive of reliance in such cases.
Second, and most importantly, by assuming we have access to the human’s own
prediction prior to their interaction with the AI prediction, we can compare
the results of experiments we run to a benchmark of complementary performance,
which is attained by optimally combining the information contained in the
human’s predictions with that contained in the AI’s predictions, and a
baseline of using either the AI or human only.

##  3\. Definition of Reliance

We define appropriate reliance, over-reliance, and under-reliance on AI
predictions in AI-advised decision making. Our framework conceives of three
roles in the decision problem: a human predictor, an AI predictor, and a
decision-maker. The two predictors provide informational input to the
decision-maker in the form of recommendations. The decision-maker chooses
which predictor to follow on a decision task.

To formalize a decision task requires five key elements (Table 1): payoff-
related states on which the decision is evaluated, a data genertaing model
that generates the states and signals that inform about the state, the action,
the information (i.e. signal) given to the decision-maker, and a scoring rule
assessing the choice of action under the payoff-related state.

| The original decision task | The derived binary-adoption decision task
---|---|---
Payoff-related state |  • $\theta=\text{Ground truth }y\in Y$ |  • $\hat{\theta}=(y,y^{H},y^{AI})$
|  |  • Ground truth $y\in Y$
|  |  • Human prediction $y^{H}\in Y$
|  |  • AI prediction $y^{AI}\in Y$
Data generating model |  • Feature values $x$ from feature space $X$
|  • $(x,y)\sim\pi(X\times Y)$
|  • Human prediction $y^{H}$ and AI prediction $y^{AI}$:
|  $(x,y^{H})\sim\pi^{H}(X\times Y)$
|  $(x,y^{AI})\sim\pi^{AI}(X\times Y)$
|  • Explanation $e(y^{AI})$
Action (choice) | $a\in Y$ | $\hat{a}\in\\{0=\text{human},1=\text{AI}\\}$
Signal | $v=\\{x,y^{H},y^{AI},e(y^{AI})\\}$
Scoring rule (payoff) | $S(a,\theta)$ |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{H},y)$ if $\hat{a}=\text{human}$
|  |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{AI},y)$ if $\hat{a}=\text{AI}$
Table 1. Notation for original decision task and derived binary-adoption
decision task in our framework.

We define the reliance level of an decision-maker on the AI as the overall
probability that she chooses the AI prediction, conditional on the decision
maker facing different predictions from the human and the AI. The definition
targets a conditional probability, because the reliance level cannot be
defined when the human makes the same prediction as the AI.

###### Definition 0 (Reliance).

The reliance level $\gamma$ of any decision-maker on the AI is defined as the
conditional probability $\gamma=\Pr[a=y^{AI}|y^{AI}\neq y^{H}]$ that the
decision-maker chooses the AI prediction, conditional on the AI prediction
$y^{AI}$ being different from the human prediction $y^{H}$.

###  3.1. Rational Decision-Maker

We define the rational decision-maker in a binary-adoption decision task
(Table 1) derived from the original one. This derived decision task limits the
rational decision-maker to making a final decision by selecting between the
human prediction and the AI prediction. We define the rational benchmark
representing the expected performance of a rational Bayesian decision-maker
who perfectly perceives the provided information in the signal and chooses the
optimal action between choosing the human and the AI prediction under the
scoring rule for each decision task. The rational benchmark is the maximum
payoff that can be expected from a behavioral decision-maker, i.e., the
benchmark for complementary performance. Following the framework proposed by
Wu et al. (WGM+, 23), we also define a baseline for expected performance using
this rational Bayesian decision-maker. The rational baseline is the maximum
payoff that can be expected from the behavioral decision-maker when they must
choose between always going with either the AI or the human predictor, i.e.,
they do not consult the individual signals in making their decisions. The
rational baseline represents the minimum threshold for achieving complementary
performance, i.e., the baseline for complementary performance. Using the
rational benchmark and the rational baseline, we define the value of rational
complementation, representing the expected improvement in payoff to a rational
decision-maker that the joint human+AI setting provides over the better of
either the AI or the human alone.

These three values construct a space of payoffs within which behavioral
participants’ performance can be quantified and compared. The rational
benchmark also describes the appropriate reliance level, which maximizes the
expected payoff. Throughout the paper, we use superscript $r$ to denote
notation for the rational decision-maker. For example, $a^{r}$ is the action
taken by the rational decision maker, and $\gamma^{r}$ the rational decision-
maker’s reliance level.

  * •

Rational Baseline

The rational baseline is the expected performance of the rational decision-
maker without access to the signal on a randomly chosen decision task from the
experiment. Without access to the signal, the rational decision-maker can only
make decisions with prior beliefs based on her knowledge of the data-
generating model and decision task. This is the better of the two scores
achieved by the human alone and the AI alone.

(1) $\textsc{R}_{\varnothing}=\max_{\hat{a}}\text{\bf
E}_{\pi(\hat{\theta})}\\![{\hat{S}(\hat{a},\hat{\theta})}]=\max_{\hat{a}}\text{\bf
E}_{\pi(\hat{\theta})}\\![{S(y{}^{\hat{a}},y)}].$

  * •

Rational Benchmark

The rational benchmark is the expected performance of the rational decision-
maker with the signal on a randomly chosen decision task from the experiment.
Let $a^{r}(v)$ be the action taken by the rational decision-maker given signal
$v$. She chooses $a^{r}$ to maximize her expected utility with
$\pi(\hat{\theta}|v)$, the distribution of the payoff-related state
conditioned on the signal $v$:

(2) $a^{r}(v)=\operatorname*{argmax}_{\hat{a}}\text{\bf
E}_{\pi(\hat{\theta}|v)}\\![{\hat{S}(\hat{a},\hat{\theta})}].$

Hence, the rational benchmark is the rational decision-maker’s expected score
with the utility maximizing action:

(3) $\textsc{R}=\max_{a^{r}(\cdot)}\text{\bf
E}_{\pi(v,\hat{\theta})}\\![{\hat{S}(a^{r}(v),\hat{\theta})}].$

The rational benchmark upperbounds the expected performance of any behavioral
decision-maker in the experiment.

  * •

Value of rational complementation

The value of rational complementation is the increase in payoff over the
rational baseline when the rational decision-maker sees the signal.

(4) $\Delta=\textsc{R}-\textsc{R}_{\varnothing}.$

The value of rational complementation provides a scale for comparing expected
performance in terms of the “lift” we see from having access to the
information in the signals. In the context of AI-advised decision making, it
also represents the maximum improvement of performance we can expect from a
complementation of the human and the AI conditioned on the information
structure of the signals. If we treat $\Delta$ as a comparative unit by
normalizing all scores within the range where the baseline
$\textsc{R}_{\varnothing}$ is $0$ and the benchmark R is $1$, we get a sense
of the proportion of possible score increase that different settings provide.
For example, we could compare expected human performances
$\textsc{B}_{\alpha}$ and $\textsc{B}_{\beta}$ under two conditions $\alpha$
and $\beta$ (e.g., $\alpha$ explanation and $\beta$ explanation) by
calculating $(\textsc{B}_{\alpha}-\textsc{B}_{\beta})/\Delta$.

Given the definitions above, we can define the appropriate reliance level as
the reliance level of the rational decision-maker, conditional on the human
prediction being different from the AI prediction, $y^{H}\neq y^{AI}$. Note
that the appropriate reliance level maximizes the expected score of the
decision.

###### Definition 0.

The appropriate reliance level $\gamma^{r}$ is the rational decision-maker’s
reliance level on the AI, $\gamma^{r}=\Pr[a^{r}=1|y^{AI}\neq y^{H}]$.

###  3.2. Behavioral Decision-Maker

The behavioral decision-maker who completes the decision task takes action
$a^{b}$, and is evaluated by their expected performance on the task. We view
the behavioral action as a random variable correlated with the signal, and
hence also with the ground truth. Denote the joint distribution as
$\pi(v,a^{b},\theta)$.

  * •

Behavioral Performance

(5) $\textsc{B}=\text{\bf E}_{\pi(v,a^{b},\theta)}\\![{S(a^{b},\theta)}].$

We define behavioral under-reliance and over-reliance by comparing behavioral
reliance level $\gamma^{b}$ to the appropriate reliance level $\gamma^{r}$.

###### Definition 0.

When $\gamma^{b}<\gamma^{r}$, the behavioral decision-maker under-relies on
the AI.

###### Definition 0.

When $\gamma^{b}>\gamma^{r}$, the behavioral decision-maker over-relies on the
AI.

In addition to the reliance level, we analyze the difference between the
behavioral decision-maker’s expected score and the rational decision-maker’s
expected score to measure decision quality. To understand why we analyze the
difference in score versus in the action space, consider the extreme case
where the human predictor and the AI predictor are both uninformative about
the ground truth. Adopting either the AI prediction or the human prediction
would achieve an equally bad expected payoff, such that any reliance level
between $0\%$ and $100\%$ would perform similarly. Simply evaluating the
reliance level by comparing to the best reliance level ignores the close
payoffs achieved by all reliance levels and leads to misleading conclusions.

We separate the behavioral decision-maker’s loss in score into two sources:
loss from mis-reliance, and what we term discrimination loss, referring to the
loss from not accurately distinguishing when the AI predictor has better
expected payoff than the human predictor or vice versa. To separate these
sources of loss, we define another benchmark representing the expected score
of a rational decision-maker who is constrained to a specific reliance level.

  * •

Mis-Reliant Rational benchmark

The expected score of a mis-reliant rational decision-maker with reliance
level $\gamma$:

(6) $\displaystyle\textsc{R}^{\text{m}}(\gamma)=\max_{a^{r}(\cdot)}\quad$
$\displaystyle\text{\bf
E}_{\pi(\hat{v},\hat{\theta})}\\![{\hat{S}(a^{r}(\cdot),\hat{\theta})}]$ s.t.
$\displaystyle\Pr[a^{r}=1|y^{AI}\neq y^{H}]=\gamma$

Hence, the mis-reliant rational benchmark $\textsc{R}^{\text{m}}$ represents
the best score an decision-maker with a given reliance level $\gamma$ could
attain had they perfectly perceived the probability that the AI is correct
relative to the probability that the human is correct on every decision task.
By constraining a rational decision-maker to the same reliance level $\gamma$
as each corresponding behavioral decision-maker, we can get a rational
decision-maker who simulates the decision rule of the behavioral decision-
maker but optimally perceives the signal and arrives at the Bayesian posterior
beliefs on each instance. By comparing the expected score of these rational
decision-makers and behavioral decision-makers, we can distinguish between the
following sources of loss:

  * •

Reliance loss, the loss from over- or under-relying on the AI, defined as
$(\textsc{R}-\textsc{R}^{\text{m}})/\Delta$. We measure reliance loss in
payoff space rather than assessing the deviation from the optimal reliance
level. The latter treats all errors identically, whereas using payoff space
accounts for how big an error is in terms of lost payoff.

  * •

Discrimination loss, the loss from not accurately differentiating the
instances where the AI is better than the human from the ones where the human
is better than the AI, defined as $(\textsc{R}^{\text{m}}-\textsc{B})/\Delta$.
Since $\textsc{R}^{\text{m}}$ and B have the same reliance level and accept
the same percentage of AI predictions, the difference in the decisions of
$\textsc{R}^{\text{m}}$ and the decisions of B lies entirely in accepting the
AI predictions at different instances. $\textsc{R}^{\text{m}}$ always accepts
the top $x\%$ AI predictions ranked by performance advantage over human
predictions, but B may not.

In other words, we decompose the difference between the best attainable
performance in the study (R) and the observed behavior of study participants
(B) into two parts. We show an example of the quantities, R,
$\textsc{R}^{\text{m}}$, B, and $\textsc{R}_{\varnothing}$, from our framework
in Figure 1. Figure 1 illustrates how the behavioral performance B and mis-
reliant rational benchmark $\textsc{R}^{\text{m}}$ are bounded. B must be
equal to or lower than the rational benchmark R. If B is higher than the
rational baseline $\textsc{R}_{\varnothing}$ (i.e., the better performance of
either AI predictions or human predictions alone), we say B fulfills the
requirement of complementary performance. $\textsc{R}^{\text{m}}$ must fall
between B and R.

Figure 1. An example of the composition of the quantities defined in our
framework. $\textsc{R}_{\varnothing}$ and R can be calculated using knowledge
of the experiment design, which in our framework includes the human
predictions and the AI predictions in addition to the components of the
decision problem (Table 1). $\textsc{R}^{\text{m}}$ and B can be calculated
given observed data on the human decision-maker’s decisions in an AI-assisted
scenario.

##  4\. Applying the Framework to AI Reliance Studies

We discuss how to apply the framework to AI reliance studies using an example.

Experiment design and data collection. The first step in applying the
framework is to formulate the experiment design as a decision problem by
defining the ground truth state, data-generating model, action space, signal,
and scoring rule. Imagine we run an experiment studying AI-advised recidivism
decisions with $200$ humans, where each completes 20 trials. In each trial
they view a profile of the defendent, and must predict whether the defendent
will be re-arrested. The participants are assisted with an AI model that is
deterministic and calibrated on the ground truth. We equally divide the $200$
participants into two groups, randomly assigning $100$ to one explanation
condition and the other $100$ to a different explanation condition. All
participants do the $20$ predictions by themselves before they see any AI
predictions, then make final decisions on the same $20$ instances with the AI
assistance. For every correct decision on the second batch of trials (final
decisions made after revising AI predictions), the participant receives
$\$0.5$ as incentivization. The decision task and its derived binary-adoption
decision task for the rational decision-maker are formalized in Table 2 in
Appendix B. When the experiment is complete, we have collected $4000$ decision
observations in total. Each observation includes information about the profile
of the defendent, the outcome of whether the defendent is re-arrested, the
human prediction on the first batch of trials, the AI prediction, the
explanation of the AI prediction, and the final decision on the second batch
of trials.

Rational baseline $\textsc{R}_{\varnothing}$. Recall that the rational
baseline represents the expected performance of the rational decision-maker
without access to the signal on the derived binary-adoption decision task from
the experiment. Hence, the best action is the better of always following the
AI and always following the human prediction. We estimate the rational
baseline by identifying the best-response to the empirical distribution of
states in the experiment. Our empirical distribution is simulated with $4000$
observations. This calculation is illustrated in Algorithm 1 in Appendix A.

(Approximating) Rational benchmark R. To calculate the rational benchmark we
must identify the best response to each signal. When the signal space has
finite size, we can calculate the rational benchmark by simulating the best
response to each signal on the empirical distribution of the experiment
observations. However, for a large number of decision tasks in the literature
(including, e.g., the demonstrations in Section 5), the signal space has near
infinite size (e.g., it involves text documents) such that each experimental
observation might involve a different unique signal. Thus, the identified best
response action may overfit to the data relative to the true expected score of
the rational decision-maker on a randomly chosen decision task from the
experiment. We approximate the rational benchmark by designing an upperbound
and a lowerbound.

  * •

Upperbound: Overfitting to the empirical distribution. We calculate the
rational benchmark on the empirical joint distribution
$\tilde{\pi}(\hat{\theta},v)$ over the payoff-relevant state $\hat{\theta}$
and the signal $v$, treating the empirical distribution as the true data
generating model. Algorithm 2 in Appendix A calculates this empirical
distribution.

To see why this is an upperbound and why we call it overfitting, consider the
case where the signal space is continuous. Each entry in the experiment data
has a distinct signal. Without repetition, it is impossible to approximate the
true distribution of the payoff-relevant state $\hat{\theta}$ conditioning on
each signal $v$. Treating the empirical distribution as the true data
generating model, there is no randomness in the payoff-relevant state given
the rational decision-maker’s knowledge. Hence, the rational decision-maker
always takes the best action given the empirical distribution, but this action
may not necessarily match the best action under the true data generating
model.

  * •

Lowerbound: Learning the best response on the optimally discretized empirical
distribution to avoid overfitting. Assuming continuity on the joint
distribution $\tilde{\pi}(\hat{\theta},v)$ over the payoff-relevant state
$\hat{\theta}$ and the signal $v$, we approximate the rational benchmark by
coarsening the signal space into finite discrete signals
$\tilde{v}_{1},\tilde{v}_{2},\dots,\tilde{v}_{k}$, and calculating the best
response on the empirical distribution over the discretized space
$\\{\tilde{v}_{i}\\}_{i}$. An example using the $k$-means algorithm to
discretize the signals is shown in Algorithm 3 in Appendix A.

To see why this is an lowerbound on the rational benchmark, first note that
the rational decision-maker with the true data generating model can always
perform the same discretization as the algorithm on the signal space, and such
discretization to the signal can only decrease the expected performance. It
remains to make sure the discretization is not too fine, such that the
estimate on the empirical distribution is close to the rational decision-
maker’s expected payoff on the discretized signal (i.e. the estimate does not
overfit to the data points from the experiment). We ensure this by performing
cross-validation on the estimated average payoff. We randomly split the
experiment data into a clustering set and a test set. Intuitively, increasing
the number of clusters $k$ leads to an expected payoff closer to the rational
benchmark, but a higher gap between the estimated payoff on the clustering set
and the test set (a.k.a. the generalization error). We select $k$ to balance
the increase in expected payoff and the generalization error.

The calculation of the rational benchmark hence takes an empirical
distribution as input. For a finite signal space, the rational benchmark is
calculated on the empirical distribution. For an infinite signal space, the
upperbound is calculated on the empirical distribution, while the lowerbound
is calculated on the discretized empirical distribution.

Regardless of which bound we are calculating, given an empirical distribution
(e.g, the $4000$ observations), we simulate the rational decision-maker’s
decision. For each observation, the rational decision-maker receives a signal
(raw signal or discretized signal) and calculates the posterior distribution
of states given the signal, denoted as $\pi(\hat{\theta}|v)$. We calculate the
posterior distribution of states by Bayes rule:
$\pi(\hat{\theta}|v)=\frac{\pi(\hat{\theta},v)}{\pi(v)}$. We pick the action
with higher expected payoff under the posterior distribution as the decision
of the rational benchmark on the current observation. We repeat this process
for all observations and then take the expectation on all the rational
benchmarks we get. When we want to compare the rational benchmark across
different conditions, e.g., different explanations, we can take the
conditional expectation on the variable of interest. This calculation is
illustrated in Algorithm 4 in Appendix A.

Behavioral performance B. The expected performance of a behavioral decision-
maker who makes the final decision after viewing the human prediction and the
AI prediction is estimated on the joint behavior of the behavioral decision-
makers in the experiment, denoted as $\pi(v,\theta,a^{b})$. We can use the
$4000$ observations to represent the joint behavior of the behavioral
decision-makers or we can estimate it using a model trained on the $4000$
observations to predict the behavioral decision-makers’ decisions111When we
estimate the joint behavior by a model, how good the estimates of behavioral
performance are will depend on how well the model predicts the behavioral
data.. Whenever the goal of the study is to compare the performance of human
participants across different conditions, e.g., different explanations, the
expected behavioral decision-maker performance should be conditioned on the
manipulated variables of interest. This calculation is illustrated in
Algorithm 5 in Appendix A.

(Approximating) Mis-reliant rational benchmark $\textsc{R}^{\text{m}}$. The
mis-reliant rational benchmark is the expected score of a rational decision-
maker with the same behavioral reliance level as the human participant. To
calculate this, we simulate the rational decision-maker completing the same
set of trials as the behavioral decision-makers do but additionally constrain
the rational decision-maker’s reliance level to be the same as the reliance
level produced by the behavioral decision-makers $\pi(v,\theta,a^{b})$. In our
example experiment, each behavioral decision-maker completes $20$ trials. We
compute their reliance levels across these $20$ trials,
$\gamma^{b}=\Pr[a^{b}=y^{AI}|y^{AI}\neq y^{H}]$. As the rational decision-
maker traverses the $4000$ observations, like behavioral participants she
should engage in $20$ consecutive trials for each set. Suppose that the
signals that the rational decision-maker receives in the $20$ consecutive
trials are $v_{1},\ldots,v_{20}$. For each signal $v_{i}$, the rational
decision-maker knows the payoff of the human prediciton and the AI prediction
under the posterior distribution of states, i.e., $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{AI},y)}]$ and $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{H},y)}]$. Then, the rational decision-
maker ranks the signals in decreasing order of $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{AI},y)}]-\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{H},y)}]$ and accepts the AI prediction
from the first signal in the sorted list, up to a $\gamma^{b}$ fraction of
$20$ signals. After the rational decision-maker takes an action for $4000$
observation, we take the expectation over all observations (or conditionally
on the manipulated variable of interest depending on the study design). This
calculation is illustrated in Algorithm 6 in Appendix A.

Note that estimation of the mis-reliant rational benchmark faces the same risk
of overfitting as the rational benchmark. When the signal space is infinite,
we approximate the mis-reliant rational benchmark the same way that we do the
rational benchmark by calculating the upper- and lower-bound.

Quantifying uncertainty. All the quantities calculated by the above algorithms
are point estimates of the expectation of the rational baseline, the rational
benchmark, the behavioral performance, and the mis-reliant rational benchmark.
To get a robust estimate, we bootstrap to compute the expectation. For each
iteration in bootstrapping, we sample from the $4000$ observations, and run
the four algorithms on the ratio of the sample. The estimations of the
expected payoff generated through iterations quantify the uncertainty. This
calculation is illustrated in Algorithm 7 in Appendix A.

##  5\. Demonstration

We apply our framework to three AI-advised decision making experiments (BWZ+,
21; LT, 19; FCL, 21). 222We use the upper bound (overfit) method to
approximate the rational benchmarks and the mis-reliant rational benchmark,
i.e., estimating the empirical distribution using the observations of signals
and payoff-relevant state and treating the empirical distribution as the true
data generating model. We confirmed our conclusions from this approach using
the approximation of the rational benchmark with discretized signals in
Appendix C. We reanalyze the reliance levels of behavioral decision-makers
within the payoff space by comparing to the rational baseline and the rational
benchmark. We also identify the discrimination loss.333See our supplementary
materials for complete analysis with full code and original data:
https://osf.io/2cbxf/?view_only=d4ac47806ea84b159fda76fd566941cc

###  5.1. On Human Predictions with Explanations and Predictions of Machine
Learning Models (LT, 19)

Lai and Tan (LT, 19) compare different approaches to integrate an AI in the
task of detecting deception in hotel reviews.

####  5.1.1. Experiment design.

Following (OCCH, 11), participants are asked to look up a hotel review and
then make a decision on whether the review is genuine or deceptive. Lai and
Tan (LT, 19) proposed seven conditions with different levels of AI assistance
along a hypothesized spectrum from full human agency to full automation: no
information from the AI, only example-based explanation, only highlight-
feature explanation, only heatmap explanation, only predicted label, predicted
label with random heatmap explanation, predicted label with example-based
explanation, predicted label with heatmap explanation, and predicted label
with accuracy. Since the reliance problem we study is defined only for the
scenario where the AI prediction is provided to the human decision maker, we
analyze only the five conditions including AI information. The decision task
is summarised in Table 4 in Appendix B.

####  5.1.2. Analysis

Figure 2. Expected payoffs of benchmarks, baselines, and observed performance
in Lai and Tan (LT, 19). Figure 3. Plots demonstrating how the rational agent
arrives at the appropriate reliance level by maximizing her payoff in the
decision-making problem defined by Lai and Tan (LT, 19), including A) quantile
plot (y-axis: $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{AI},y)}]-\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{H},y)}]$ ranked in descending order;
x-axis: the cummulative probability (quantile) of signal $v_{i}$) and B) 50%
and 95% intervals on behavioral decision-makers’ reliance levels.

The conclusions drawn by Lai and Tan (LT, 19) include: AI-advised decisions
were better when the AI system interfered more with the human decision-maker’s
process, and trust in the AI prediction increased with more AI-based
information. Trust was evaluated by the rate at which the AI predictions were
accepted. Their results ranking the AI-based conditions by both performance
and trust is (from worst to best) were: no predicted label < only predicted
label < predicted label with random heatmap explanation < predicted label with
example-based explanation < predicted label with heatmap explanation <
predicted label with accuracy. Using our approach, we examine the ranking of
behavioral performance within the scale created by the rational baseline and
rational benchmark. Instead of evaluating reliance as rate of acceptance of AI
predictions, we evaluate the reliance level of the behavioral decision-makers
in payoff space.

Extending the author’s original conclusions, we find that the rational
baseline dominates almost all other quantities in our framework except the
rational benchmark, including the behavioral performance and the mis-reliant
rational benchmark across all explanation conditions, as shown in Figure 2
(the rational baseline and the rational benchmark). Additionally, the rational
benchmark only improves marginally over the rational baseline, i.e., the
rational decision-maker does not gain much from access to human predictions,
as shown in Figure 2A (the rational benchmark and the rational baseline).
Consequently, it is hard to expect behavioral decision-makers to achieve
complementary performance. These findings suggest that the experimental design
was poorly suited for studying complementary performance, because the AI
consistently outperforms the human.

Using our approach, we extend the authors’ results by observing that different
explanation conditions result in different levels of discrimination loss and
reliance loss. For example, the condition with heatmap explanations and the
condition directly providing model accuracy show similar reliance loss (Figure
2C) but the discrimination loss in the latter is smaller than the former. This
suggests why showing accuracy can help the behavioral decision-makers achieve
higher performance than heatmap explanations: the accuracy information helps
the behavioral decision-makers better differentiate instances where the AI
predictor outperforms the human predictor from those where the human predictor
outperforms the AI predictor, presumably because it provides information on
the joint distribution of the AI prediction and the ground truth that is
absent from the heatmap explanations.

###  5.2. Does the Whole Exceed its Parts? (BWZ+, 21)

Bansal et al. (BWZ+, 21) use an online crowdsourced experiment to investigate
the effects of explanations on the degree of complementary performance
achieved by AI-advised humans. In contrast to prior studies like (LT, 19),
Bansal et al. (BWZ+, 21) controlled the AI’s accuracy to be comparable to the
humans’, to avoid the AI being obviously better than human performance on the
task.

####  5.2.1. Experiment design

The experiment compares human-AI team decisions across four approaches to
explaining AI predictions: no explanation, explanation for the most confident
AI prediction, explanations for the top-2 most confident AI predictions, and
adaptively showing explanations for the top-1 or top-2 most confident AI
predictions, randomly assigned between subjects. The participants are tasked
with using the AI prediction and its explanation for two tasks: sentiment
classification and LSAT (multiple-choice questions where one of four choices
is the correct answer). Because the manipulation of interest (explanation
types) and conclusions drawn about the complementary performance of the human-
AI teams across different explanation types are the same between the two
tasks, we analyze only the results of the LSAT task. The decision task is
summarised in Table 3 in Appendix B.

####  5.2.2. Analysis

Figure 4. Expected payoffs of benchmarks, baselines, and observed performance
in Bansal et al. (BWZ+, 21).  Figure 5. Plots demonstrating how the rational
agent arrives at the appropriate reliance level by maximizing her payoff in
the decision-making problem defined by Bansal et al. (BWZ+, 21), including A)
quantile plot (y-axis: $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{AI},y)}]-\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{H},y)}]$ ranked in descending order;
x-axis: the cummulative probability (quantile) of signal $v_{i}$) and B) 50%
and 95% intervals on behavioral decision-makers’ reliance levels.

Bansal et al. (BWZ+, 21) drew several conclusions from their results: AI-
advised decision making achieved complementary performance (i.e., a higher
payoff than expected of the human or AI alone), and presenting explanations to
the human-AI team led to no observable performance improvements using null
hypothesis significance testing (NHST) with $\alpha=0.05$. The authors
speculated that the reason they did not observe improvement from explanations
is because people over-relied on the AI when explanations are provided. This
is supported by evidence that providing explanations increased decision
performance when the AI was correct and decrease it when the AI was incorrect.
We use our framework to evaluate this conclusion. Specifically, we compare the
observed behavioral payoffs to the rational baseline and rational benchmark,
and evaluate the reliance level of participants in payoff space by comparing
the behavioral payoffs to the mis-reliant rational benchmark. Our results are
shown in Figure 4.

Extending the authors’ original conclusions, we find that despite the
behavioral decision-makers achieving complementary performance, there is still
considerable room for improvement, shown as the distance between the
behavioral performance and the rational benchmark (Figure 4A and B). The
behavioral payoff surpasses the rational baseline, as shown in all rows
representing different explanation conditions in Figure 4. This comparison
leads to the authors’ conclusion that complementary performance is observed in
every condition. However, comparing to the rational benchmark, the behavioral
decision-makers only improve a small proportion over the rational baseline
(Figure 4). Our analysis more clearly demonstrates the remaining need to
identify ways to bridge the remaining substantial gap.

Applying NHST as in the original study, we corroborate the authors’ conclusion
that there are no significant improvements for explanation conditions over the
no explanation condition. Using our approach we confirm there are not
significant reductions in either discrimination loss or reliance loss. For
example, in Figure 4 (behavioral performance and mis-reliant rational
benchmark), the behavioral decision-makers in the no explanation and the
adaptive explanation condition achieve similar performance; the same is true
of the Explain-Top-1 and Explain-Top-2 conditions.

Further extending the original conclusions, we find that despite the over-
reliance shown by the original paper, poor reliance itself is not the main
source of loss. While the behavioral decision-makers’ reliance levels across
all conditions are higher than the optimal reliance level in expectation
represented by the rational benchmark, our analysis suggests that
miscalibrated reliance of the behavioral decision-makers does not lead to
substantial loss in payoff. As shown in Figure 4C, the mis-reliant rational
benchmarks across all conditions are very close to the rational benchmark,
such that reliance loss is very minor compared to the total behavioral losses.

Instead our approach shows that the behavioral decision-makers have
substantially lower performance compared to the rational benchmark due to
large discrimination loss (i.e., accepting the AI predictions for the wrong
instances), as shown in Figure 4D. Combined with the evidence that the
behavioral decision-makers have low reliance loss, this could suggest that the
explanations be designed specifically to help users distinguish the intances
where the AI is expected to succeed from those where the AI is expected to
fail, instead of aiming to calibrate the human’s overall trust in the AI’s
accuracy or adjusting the human’s decision rule. For example, explanations
could give information on the joint distribution of AI prediction and the
ground truth, i.e., $\pi(y^{AI},y)$ rather than focusing on describing only
the decision rule of AI, e.g., as in LIME (RSG, 16) or SHAP (LL, 17).

###  5.3. The Impact of Algorithmic Risk Assessments on Human Predictions and
its Analysis via Crowdsourcing Studies (FCL, 21)

Fogliato et al. (FCL, 21) conduct an online crowdsourcing experiment where
participants face the task of assessing a defendant’s risk of re-arrest after
viewing the defendant’s profile. The experiment investigates the research
questions of whether anchoring effects impact participants’ predictions and
whether the evaluation of participants’ decisions depends on the types of
predictions (probablity or binary decision), both of which can be modeled as
decision tasks in our framework.

####  5.3.1. Experiment Design

The experiment compares AI-assisted human predictions under two different
conditions: anchoring and non-anchoring. Participants assigned to the
anchoring condition see the question presented together with the AI’s
prediction, while under the non-anchoring condition, participants are asked to
predict the risk before seeing AI prediction and then to revise their
assessment after having AI prediction. In each question, participants are
shown the profile of a defendant, including demographics, current charge, and
criminal history. Participants are asked to report: 1) the probability of the
defendant being re-arrested from $[0,100\%]$, and 2) a binary choice of
whether the defendant will be re-arrested within a given duration or not. The
decision tasks for probabilty and binary decision are summarised in Table 5 in
Appendix B.

####  5.3.2. Analysis

Figure 6. Expected payoffs of benchmarks, baselines, and observed performance
in Fogliato et al. (FCL, 21) Figure 7. Plots demonstrating how the rational
agent arrives at the appropriate reliance level by maximizing her payoff in
the decision-making problem defined by Fogliato et al. (FCL, 21), including A)
quantile plot (y-axis: $\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{AI},y)}]-\text{\bf
E}_{\pi(\hat{\theta}|v_{i})}\\![{S(y^{H},y)}]$ ranked in descending order;
x-axis: the cummulative probability (quantile) of signal $v_{i}$) and B) 50%
and 95% intervals on behavioral decision-makers’ reliance levels.

Fogliato et al. (FCL, 21) report that 1) the probability of re-arrest reported
by the participants did not uniformly map to their binary decision, such that
behavioral predictive performance and reliance level must be considered
separately, and 2) no clear differences between participants’ accuracy, false
positive rate, false negative rate, positive predicted values, or AUC were
found between the anchoring and no anchoring condition. Our analysis of their
results is shown in Figure 6 for the binary decision task and the
probabilistic decision task.

Corroborating with the authors’ conclusion, by putting both tasks on the same
payoff scale, we find that people are better at the probability task than the
decision task. First, we observe that the behavioral decision-makers doing the
probability task can achieve higher performance than those doing the binary
decision task overall. For example, the behavioral performance for the
probability task is much higher than the behavioral performance for the binary
decision task (Figure 6). Second, the behavioral performance is higher than
the performance of the human only baseline in the probabilistic task while
they perform similarly in the decision task, as shown in Figure 6B. These
results corroborate the conclusion by Fogliato et al. (FCL, 21) that there is
no determinstic decision rule that describes how the participants’ probability
estimates map to their binary decisions.

We also find that the rational baselines and the rational benchmarks differ
for each task between the anchoring and the no anchoring conditions,
suggesting a need to reconsider Fogliato et al. (FCL, 21)’s conclusion about
the similarity between anchoring and no anchoring. As shown in Figure 6A, the
rational baseline in the anchoring condition is slightly higher than in the
non-achoring condition. This implies that just comparing the absolute
performance of the behavioral decision-makers can mislead. Despite the
behavioral performance being similar across the conditions in terms of
absolute values, the behavioral decision-makers have better relative
performance in the non-anchoring condition than the anchoring condition when
compared to the rational baseline and the rational benchmark.

Similarly, contradicting the authors’ conclusion, we find that the behavioral
decision-makers’ reliance is closer to the appropriate reliance under the non-
anchoring condition than the anchoring condition in both tasks. As shown in
Figure 6C, the reliance loss
($\frac{\textsc{R}-\textsc{R}^{\text{m}}}{\textsc{R}-\textsc{R}_{\varnothing}}$)
is lower for the no anchoring condition, while the discrimination loss
($\frac{\textsc{R}^{\text{m}}-\textsc{B}}{\textsc{R}-\textsc{R}_{\varnothing}}$)
is slightly higher. This suggests that letting the behavioral decision-makers
make a decision by themselves first (a.k.a., the non-anchoring effects) can
improve their reliance, but not necessarily help them distinguish between the
signals where the AI prediction is expected to outperform the human prediction
and the signals where the human prediction is expected to outperform the AI
prediction.

##  6\. Discussion

We contribute a formal definition of reliance and corresponding framework for
interpreting losses in behavioral decision-makers’ decision-making performance
within the baseline and benchmark for complementary performance. The first
source of loss concerns the difference in the rate at which the behavioral
decision-maker relies on the AI relative to the appropriate level of reliance
defined by the decision problem, calculated in payoff space. The second source
of loss concerns the difference in score between a behavioral decision-maker
and the best score a rational decision-maker who relies on the AI at the same
rate as the behavioral decision-maker but who perfectly perceives the
probability that the AI’s answer is correct could achieve. By contributing
clear comparison points in the form of performance benchmarks to the design
and interpretation of studies of human reliance on AI, our work enables
researchers to identify the upper-bound of complementary performance and how
far the human-AI team is from this optimal attainable performance.

Closest to the motivation of our work, Fok and Weld (FW, 23) motivate the need
for a notion of “strategy-graded reliance,” where appropriate reliance is
determined from the relative expected performance of the human and the AI,
over “outcome-graded reliance” based on the human’s acceptance of AI advice
conditioned on its post-hoc correctness. Several other prior works propose
studying reliance using conditional probability (e.g., (SKB+, 23; YHSA, 20;
WY, 21)) to separate cases where the human prediction is better than the AI
prediction from cases where the AI prediction is better than the human
prediction. We unambigously define strategy-guided reliance and show how to
calculate optimal reliance and disentangle sources of behavioral loss.

Our framework enables evaluating reliance in payoff space, in contrast to
prior work which has evaluated reliance in action space only (BWZ+, 21; SKB+,
23; YHSA, 20). Studying reliance only in the action space still neglects
sensitivity in the payoff, such as the magnitude of improvement that the human
prediction provides over the AI prediction or vice versa. Defining a
measurement of reliance in payoff space also enables the calculation of a
benchmark to compare with, which we show in our demonstrations to be highly
valuable for learning from a reliance evaluation.

Decoupling sources of behavioral loss in human AI-advised decisions is
essential for designing and interpreting AI-advised decision-making
experiments, because different types of losses imply different design
improvements to address the corresponding weaknesses (KLL+, 18). In recent
years, numerous papers (BWZ+, 21; YWVW, 19; FBG, 19; FCL, 21; BMG, 21; CJT+,
22; HB, 20; CLWVB, 23; WY, 21; Cho, 17; DF, 18; BSO, 15; YBC+, 16; AW, 19; HB,
15; HNO+, 19; ZLB, 20; JKGG, 18; LLT, 21; BLGG, 20; BWZ+, 21) have employed
user studies to investigate how various factors contribute to enhancing the
complementary performance of human-AI teams. Without a well-grounded notion of
reliance, such studies have limited ability to identify possible solutions. If
reliance loss is larger, designers need to increase or decrease reliance of
the behavioral decision-makers on average. For example, stating the model’s
accuracy on hold out data allows users to calibrate their behavior to a
prediction of what proportion of AI predictions are likely to unreliable. If
discrimination loss is larger, it means that users misinterpret the instance
where AI performs badly as the instance where AI performs well. Designers
should focus on providing users better information on the relationship between
the ground truth and the AI predictions, in order to avoid poor information
reversing the order of perception. For example, improving the explanations to
highlight where the AI predictions probably fail or providing the AI’s
confidence score on this prediction can help users rank the instances by
expected difference between AI performance and human performance.

Importantly, our framework hypothesizes two distinct roles in the decision-
making process to separate human’s predictions without AI assistance from the
the process by which the human makes the final decision with access to their
own predictions and the AI predictions. This setup allows researchers to
better interpret experiments and design the decision process they study;
however, the generalizability of our framework to alternative study set-ups
still holds. Our framework can be applied to situations where the human is
both making a prediction and deciding whether to follow AI prediction, i.e.,
where the human predictor and decision-maker are the same person. However,
when the human does the tasks without constraints, they might ignore the AI
prediction and just submit their own prediction or anchor on the AI without
thinking to make the decision by themselves. Both of these two cases cause
inaccurate measurement of reliance, since AI predictions and human predictions
are not consulted in human’s decision rule. For experiments employing a single
human role in the decision task, efforts should be made to align with the
assumptions of our framework to facilitate the interpretation of experimental
results.

###  6.1. Limitations

We formalize the AI-advised decision-making problem into a binary choice of
whether to adopt a human prediction or an AI prediction. However, this may not
be suitable for every real world case. For example, when the prediction space
is continuous (e.g., regression), the human decision-maker is likely to make a
decision that is different from the human prediction or the AI prediction.
Future work could extend our definition to continuous prediction spaces.

We only identify two losses affecting human decision-makers, though more fine-
grained losses may exist in AI-advised decision-making and be worth analyzing.
For example, discrimination loss can be caused by two possible reasons:
misidentifying the probability that the AI is correct or misidentifying the
probability that the human is correct. Improving the former implies better
conveying the AI’s accuracy, while improving the latter implies giving
information on the human’s average performance on the task. More fine-grained
behavioral losses can increase learning from experimental results and imply
more targeted improvement of designs. Future work can seek to identify and
separate such additional behavioral losses and explore possible design choices
to address them.

##  7\. Acknowledgement

We would like to thank the authors of Lai and Tan (LT, 19), Bansal et al.
(BWZ+, 21), and Fogliato et al. (FCL, 21), who provided their data for
demonstration in this paper.

## References

  * AW [19] Maryam Ashoori and Justin D Weisz.  In ai we trust? factors that influence trustworthiness of ai-infused decision-making processes.  arXiv preprint arXiv:1912.02675, 2019.
  * BLGG [20] Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman.  Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems.  In Proceedings of the 25th international conference on intelligent user interfaces, pages 454–464, 2020.
  * BMG [21] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos.  To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–21, 2021.
  * BNK+ [19] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz.  Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2429–2437, 2019.
  * BSO [15] Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan.  The role of explanations on trust and reliance in clinical decision support systems.  In 2015 international conference on healthcare informatics, pages 160–169. IEEE, 2015.
  * BWZ+ [21] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2021.
  * Cho [17] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5(2):153–163, 2017.
  * CJT+ [22] Valerie Chen, Nari Johnson, Nicholay Topin, Gregory Plumb, and Ameet Talwalkar.  Use-case-grounded simulations for explanation evaluation.  Advances in Neural Information Processing Systems, 35:1764–1775, 2022.
  * CLWVB [23] Valerie Chen, Q Vera Liao, Jennifer Wortman Vaughan, and Gagan Bansal.  Understanding the role of human intuition on reliance in human-ai decision-making with explanations.  Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–32, 2023.
  * DF [18] Julia Dressel and Hany Farid.  The accuracy, fairness, and limits of predicting recidivism.  Science advances, 4(1):eaao5580, 2018.
  * FBG [19] Shi Feng and Jordan Boyd-Graber.  What can ai do for me? evaluating machine learning interpretations in cooperative play.  In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 229–239, 2019.
  * FCL [21] Riccardo Fogliato, Alexandra Chouldechova, and Zachary Lipton.  The impact of algorithmic risk assessments on human predictions and its analysis via crowdsourcing studies.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–24, 2021.
  * FW [23] Raymond Fok and Daniel S Weld.  In search of verifiability: Explanations rarely enable complementary performance in ai-advised decision making.  arXiv preprint arXiv:2305.07722, 2023.
  * GC [19] Ben Green and Yiling Chen.  The principles and limits of algorithm-in-the-loop decision making.  Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–24, 2019.
  * HB [15] Kevin Anthony Hoff and Masooda Bashir.  Trust in automation: Integrating empirical evidence on factors that influence trust.  Human factors, 57(3):407–434, 2015.
  * HB [20] Peter Hase and Mohit Bansal.  Evaluating explainable ai: Which algorithmic explanations help users predict model behavior?  arXiv preprint arXiv:2005.01831, 2020.
  * HNO+ [19] Benjamin D Horne, Dorit Nevo, John O’Donovan, Jin-Hee Cho, and Sibel Adalı.  Rating reliability and bias in news articles: Does ai assistance help everyone?  In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 247–256, 2019.
  * JKGG [18] Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta.  To trust or not to trust a classifier.  Advances in neural information processing systems, 31, 2018.
  * JPMJ+ [21] Maia Jacobs, Melanie F Pradier, Thomas H McCoy Jr, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos.  How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection.  Translational psychiatry, 11(1):108, 2021.
  * KLL+ [18] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan.  Human decisions and machine predictions.  The quarterly journal of economics, 133(1):237–293, 2018.
  * Kon [01] Igor Kononenko.  Machine learning for medical diagnosis: history, state of the art and perspective.  Artificial Intelligence in medicine, 23(1):89–109, 2001.
  * LL [17] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
  * LLT [21] Han Liu, Vivian Lai, and Chenhao Tan.  Understanding the effect of out-of-distribution examples and interactive explanations on human-ai decision making.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–45, 2021.
  * LT [19] Vivian Lai and Chenhao Tan.  On human predictions with explanations and predictions of machine learning models: A case study on deception detection.  In Proceedings of the conference on fairness, accountability, and transparency, pages 29–38, 2019.
  * OCCH [11] Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock.  Finding deceptive opinion spam by any stretch of the imagination.  arXiv preprint arXiv:1107.4557, 2011.
  * RSG [16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * SKB+ [23] Max Schemmer, Niklas Kuehl, Carina Benz, Andrea Bartos, and Gerhard Satzger.  Appropriate reliance on ai advice: Conceptualization and the effect of explanations.  In Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 410–422, 2023.
  * VW [19] Michelle Vaccaro and Jim Waldo.  The effects of mixing machine learning and human judgment.  Communications of the ACM, 62(11):104–110, 2019.
  * WGM+ [23] Yifan Wu, Ziyang Guo, Michails Mamakos, Jason Hartline, and Jessica Hullman.  The rational agent benchmark for data visualization.  arXiv preprint arXiv:2304.03432, 2023.
  * WY [21] Xinru Wang and Ming Yin.  Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making.  In 26th international conference on intelligent user interfaces, pages 318–328, 2021.
  * WYAL [19] Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim.  Designing theory-driven user-centric explainable ai.  In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–15, 2019.
  * YBC+ [16] Kun Yu, Shlomo Berkovsky, Dan Conway, Ronnie Taib, Jianlong Zhou, and Fang Chen.  Trust and reliance based on system accuracy.  In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, pages 223–227, 2016.
  * YHSA [20] Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt.  How do visual explanations foster end users’ appropriate trust in machine learning?  In Proceedings of the 25th international conference on intelligent user interfaces, pages 189–201, 2020.
  * YWVW [19] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach.  Understanding the effect of accuracy on trust in machine learning models.  In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1–12, 2019.
  * ZLB [20] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy.  Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.  In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 295–305, 2020.

##  Appendix A The algorithms for calculations in the framework

This appendix includes all the algorithms in the form of pseudocode for all
the calculations we introduce in Section 4.

Input: the experimental data $D$ with each row representing one experimental
trial, and the scoring rule for the derived binary-adoption decision task
$\hat{S}$

Output: the rational baseline $\textsc{R}_{\varnothing}$

$payoff\leftarrow 0$;

for  _$action\leftarrow\\{0,1\\}$ ($action=0$ follow human, $1$ follow AI)_ do

for  _$row\in D$_ do

$\hat{\theta}\leftarrow\text{the state realized in }row$;

$payoff\leftarrow payoff+\hat{S}(action,\hat{\theta})$;

end for

$payoff_{action}\leftarrow payoff/\text{the number of rows in }D$;

end for

$\textsc{R}_{\varnothing}=\max\\{payoff_{0},payoff_{1}\\}$;

Algorithm 1 Rational baseline

Input: the experimental data $D$ with each row representing one experimental
trial, the space of derived binary-adoption states $\hat{\Theta}$, and the
space of signals $V$

Output: the empirical distribution $\tilde{\pi}(\hat{\theta},v)$

$\tilde{\pi}(\hat{\theta},v)\leftarrow
0\,{\mathds{1}}_{|\hat{\Theta}|}{\mathds{1}}_{|V|}^{\top}$;

/* Initializing a matrix with all $0$. */

for  _$row_{i}\in D$_ do

$\hat{\theta}_{i}\leftarrow\text{the state realized in }row_{i}$;

$v_{i}\leftarrow\text{the signal realized in }row_{i}$;

$\tilde{\pi}(\hat{\theta}_{i},v_{i})\leftarrow\tilde{\pi}(\hat{\theta}_{i},v_{i})+1$;

end for

$\tilde{\pi}(\hat{\theta},v)\leftarrow\tilde{\pi}(\hat{\theta},v)/|\tilde{\pi}(\hat{\theta},v)|$;

/* Normalizing to get the joint distribution. */

Algorithm 2 Calculating the empirical distribution

Input: the experimental data $D$ with each row representing one experimental
trial, the total number of clusters $K$, the space of derived binary-adoption
states $\hat{\Theta}$, and the space of signals $V$

Output: the empirical distribution $\tilde{\pi}(\hat{\theta},\tilde{v})$ on
the optimally discretized space

$\tilde{\pi}(\hat{\theta},\tilde{v})\leftarrow
0\,{\mathds{1}}_{|\hat{\Theta}|}{\mathds{1}}_{K}^{\top}$;

/* Initializing a matrix with all $0$. */

$\\{v_{i}\\}\leftarrow\text{all signals realized in }D$;

$kmeans\leftarrow initialize\\_kmeans(\\{v_{i}\\},K)$;

/* Training the K-means model. */

for  _$row_{i}\in D$_ do

$\hat{\theta}_{i}\leftarrow\text{the state realized in }row_{i}$;

$v_{i}\leftarrow\text{the signal realized in }row_{i}$;

$\tilde{v}_{i}\leftarrow kmeans(v_{i})$;

$\tilde{\pi}(\hat{\theta}_{i},\tilde{v}_{i})\leftarrow\tilde{\pi}(\hat{\theta}_{i},\tilde{v}_{i})+1$;

end for

$\tilde{\pi}(\hat{\theta},\tilde{v}_{i})\leftarrow\tilde{\pi}(\hat{\theta},\tilde{v}_{i})/|\tilde{\pi}(\hat{\theta},\tilde{v}_{i})|$;

/* Normalizing to get the joint distribution. */

Algorithm 3 Discretizing signals using the cluster generated by K-means

Input: the experimental data $D$ with each row representing one experimental
trial, the joint distribution between states and signals
$\pi(\hat{\theta},v)$, and the scoring rule for the derived binary-adoption
decision task $\hat{S}$

Output: the rational benchmark R

$payoff\leftarrow 0$;

for  _$row\in D$_ do

$v\leftarrow\text{the signal realized in }row$;

$\pi(\hat{\theta}|v)=\pi(\hat{\theta},v)/\pi(v)$;

/* the posterior distribution of the binary-adoption state */

$action\leftarrow\operatorname*{argmax}_{\hat{a}\sim\\{\text{human},\text{AI}\\}}{E_{\hat{\theta}\sim\pi(\hat{\theta}|v)}(\hat{S}(\hat{a},\theta))}$;

/* the action made on the posterior distribution */

$\hat{\theta}\leftarrow\text{the state realized in }row$;

$payoff\leftarrow payoff+\hat{S}(action,\hat{\theta})$;

end for

$\textsc{R}\leftarrow payoff/\text{the number of row in }D$;

Algorithm 4 Rational benchmark

Input: the experimental data $D$ with each row representing one experimental
trial, the joint behavior $\pi(v,\theta,a^{b})$, and the scoring rule $S$

Output: the behavioral performance B

$payoff\leftarrow 0$;

for  _$row\in D$_ do

$v\leftarrow\text{the signal realized in }row$;

$\theta\leftarrow\text{the state realized in }row$;

$action\leftarrow\text{action drawn from
}\pi(a^{b}|\theta,v)=\pi(v,\theta,a^{b})/\pi(\theta,v)$;

$payoff\leftarrow payoff+S(action,\theta))$;

end for

$\textsc{B}\leftarrow payoff/\text{the number of row in }D$;

Algorithm 5 Behavioral performance

Input: the experimental data $D$ with each row representing one experimental
trial, the joint distribution between states and signals
$\pi(\hat{\theta},v)$, the scoring rule for the original decision task $S$,
and the scoring rule for the derived binary-adoption decision task $\hat{S}$

Output: the mis-reliant rational benchmark $\textsc{R}^{\text{m}}$

$P\leftarrow\\{P_{1},\ldots,P_{M}\\}$;

/* The sets of trials finished by each participant; $M$ participants in total.
*/

for  _$i\in\\{1,\ldots,M\\}$_ do

$P_{i}\leftarrow filter(D,participant\\_id==i)$;

end for

$payoff\leftarrow 0$;

for  _$P_{i}\in P$_ do

$\text{Sort }P_{i}\text{ in decreasing order of }\text{\bf
E}_{\pi(\hat{\theta}|v)}\\![{S(y^{AI},y)}]-\text{\bf
E}_{\pi(\hat{\theta}|v)}\\![{S(y^{H},y)}]$;

$\\{v_{j}\\}\leftarrow\\{\text{the signal realized in }row_{j}\\}_{row_{j}\in
P_{i}}$;

$\\{\theta_{j}\\}\leftarrow\\{\text{the state realized in
}row_{j}\\}_{row_{j}\in P_{i}}$;

$\\{a^{b}_{j}\\}\leftarrow\\{\text{action drawn from
}\pi(a^{b}|\theta_{j},v_{j})\\}_{row_{j}\in P_{i}}$;

$\gamma^{b}\leftarrow\sum_{row_{j}\in
P_{i}}{\mathds{1}[a^{b}_{j}=y^{AI}_{j}\&y^{AI}_{j}\neq y^{H}_{j}]}$;

$N\leftarrow\text{the number of rows in }P_{i}$;

$\\{a^{r}_{j}\\}\leftarrow\\{AI\\}_{j\in\\{1,\ldots,\gamma^{b}\\}}\cup\\{human\\}_{j\in\\{\gamma^{b}+1,\ldots,N\\}}$;

$\\{\hat{\theta}_{j}\\}\leftarrow\\{\text{the binary-adoption state realized
in }row_{j}\\}_{row_{j}\in P_{i}}$;

$payoff\leftarrow payoff+\sum_{j\in[N]}S(a^{r}_{j},\hat{\theta}_{j}))$;

end for

$\textsc{R}^{\text{m}}\leftarrow payoff/\text{the number of rows in }D$;

Algorithm 6 Mis-reliant rational benchmark

Input: the experimental data $D$ with each row representing one experimental
trial, total number of iterations $T$, the sample size $k$, prior distribution
of the binary-adoption state $\pi(\hat{\theta})$, the joint distribution
between states and signals $\pi(\hat{\theta},v)$, the joint behavior
$\pi(v,\theta,a^{b})$, the scoring rule $S$, and the scoring rule for derived
binary-adoption decision task $\hat{S}$

Output: the distribution of the rational baseline
$\\{\textsc{R}_{\varnothing}{}_{i}\\}_{i\in[T]}$, the rational benchmark
$\\{\textsc{R}_{i}\\}_{i\in[T]}$, the behavioral performance
$\\{\textsc{B}_{i}\\}_{i\in[T]}$, and the mis-reliant rational benchmark
$\\{\textsc{R}^{\text{m}}_{i}\\}_{i\in[T]}$

for  _$i\in[T]$_ do

$\tilde{D}\leftarrow sample(D,k)$;

$\textsc{R}_{\varnothing}{}_{i}\leftarrow\text{Rational
baseline}(\tilde{D},\pi(\hat{\theta}),\hat{S})$;

$\textsc{R}_{i}\leftarrow\text{Rational
benchmark}(\tilde{D},\pi(\hat{\theta},v),\hat{S})$;

$\textsc{B}_{i}\leftarrow\text{Behavioral
performance}(\tilde{D},\pi(v,\hat{\theta},a^{b}),S)$;

$\textsc{R}^{\text{m}}_{i}\leftarrow\text{Mis-reliant rational
baseline}(\tilde{D},\pi(\hat{\theta},v),S,\hat{S})$;

end for

Algorithm 7 Quantifying uncertainty

##  Appendix B Formalized decision tasks

| The original decision task | The derived binary-adoption decision task
---|---|---
Payoff-related state |  • $\theta=\text{Ground truth }y\in\\{0,1\\}$ |  • $\hat{\theta}=(y,y^{H},y^{AI})$
|  Be re-arrested or not |  • Ground truth $y\in\\{0,1\\}$
|  |  • Human prediction $y^{H}\in\\{0,1\\}$
|  |  • AI prediction $y^{AI}\in\\{0,1\\}$
Data generating model |  • A profile $x$ of a defendent who is randomly drawn from the defendent population
|  • Ground truth $y$ drawn from a distribution conditioned on $x$.
|  • The human prediction $y^{H}$ is produced by the decision rule of the
human predictor,
|  represented by the joint behavioral $\pi(y^{H},x,y)$
|  • AI prediction $y^{AI}$ for the profile $x$
|  • The explanation $e(y^{AI})$
Action (choice) |  $a\in\\{0,1\\}$ Be re-arrested or not | $\hat{a}\in\\{0=\text{human},1=\text{AI}\\}$
Signal | $v=\\{x,y^{H},y^{AI},e(y^{AI})\\}$
Scoring rule (payoff) | $S(a,\theta)=0.5\times\mathds{1}[a=\theta]$ |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{H},y)$ if $\hat{a}=\text{human}$
|  |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{AI},y)$ if $\hat{a}=\text{AI}$
Table 2. Example of original and derived binary-adoption decision task in hypothetical recidivism experiment Payoff-related state |  • $\theta=\text{Correct answer }y\in\\{A,B,C,D\\}$ |  • $\hat{\theta}=(y,y^{H},y^{AI})$
---|---|---
|  |  • Ground truth $y\in\\{A,B,C,D\\}$
|  |  • Human prediction $y^{H}\in\\{A,B,C,D\\}$
|  |  • AI prediction $y^{AI}\in\\{A,B,C,D\\}$
Data generating model |  • Question $x$ drawn from the scope of LSAT questions
|  • Correct answer $y$ for $x$
|  • AI prediction $y^{AI}$ for $x$
|  • Human prediction $y^{H}$:
|  $y^{H}\sim\pi(x,y^{H})/\pi(x)$
|  • Explanation $e(y^{AI})$
Action (choice) | $a\in\\{A,B,C,D\\}$ | $a\in\\{0=\text{human},1=\text{AI}\\}$
Signal | $v=\\{x,y^{H},y^{AI},e(y^{AI})\\}$
Scoring rule (payoff) | $S(a,\theta)=\mathds{1}[a=\theta]$ |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{H},y)$ if $\hat{a}=\text{human}$
|  |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{AI},y)$ if $\hat{a}=\text{AI}$
Table 3. BWZ+ [6] decicion task under our framework. | The original decision task | The derived binary-adoption decision task
---|---|---
Payoff-related state |  • $\theta=\text{Ground truth }y\in\\{0,1\\}$ |  • $\hat{\theta}=(y,y^{H},y^{AI})$
|  Deceptive or genuine |  • Ground truth $y\in\\{0,1\\}$
|  |  • Human prediction $y^{H}\in\\{0,1\\}$
|  |  • AI prediction $y^{AI}\in\\{0,1\\}$
Data generating model |  • Ground truth $y\sim\text{Bernoulli}(0.5)$, indicating whether the review
| is written by a person who has been going to the hotel or not.
|  • Review text $x$ generated by the person
|  $x\sim\pi(x,y)/\pi(y)$
|  • Human prediction $y^{H}$
|  $y^{H}\sim\pi(x,y^{H})/\pi(x)$
|  • AI prediction $y^{AI}$ for $x$
|  • Explanation $e(y^{AI})$
Action (choice) |  $a\in\\{0,1\\}$ Deceptive or genuine | $\hat{a}\in\\{0=\text{human},1=\text{AI}\\}$
Signal | $v=\\{x,y^{H}(x),y^{AI}(x),e(y^{AI})\\}$
Scoring rule (payoff) | $S(a,\theta)=\mathds{1}[a=\theta]$ |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{H},y)$ if $\hat{a}=\text{human}$
|  |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{AI},y)$ if $\hat{a}=\text{AI}$
Table 4. LT [24] decision task under our framework. | The binary decision task | The probabilistic decision task | The binary-adoption decision task
---|---|---|---
Payoff-related state |  • $\theta=\text{Ground truth }y\in\\{0,1\\}$ Re-arrest or not |  • $\hat{\theta}=(y,y^{H},y^{AI})$
|  |  • Ground truth $y\in\\{0,1\\}$
|  |  • Human prediction $y^{H}\in\\{0,1\\}$
|  |  • AI prediction $y^{AI}\in\\{0,1\\}$
Data generating model |  • A defendent $p$ is randomly drawn from the defendent population.
|  • The profile $x$ for $p$
|  • Ground truth $y$ for $p$
|  • Human prediction $y^{H}$
|  $y^{H}\sim\pi(x,y^{H})/\pi(x)$
|  • AI prediction $y^{AI}$ for $x$
|  • AI’s confidence score $e(y^{AI})$
Action (choice) |  $a\in\\{0,1\\}$ Re-arrest or not |  $a\in[0,1]$ Probability of re-arrest | $\hat{a}\in\\{0=\text{human},1=\text{AI}\\}$
Signal | $v=\\{x,y^{H}(x),y^{AI}(x),e(y^{AI})\\}$
Scoring rule (payoff) | $S(a,\theta)=1-(a-\theta)^{2}$ |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{H},y)$ if $\hat{a}=\text{human}$
|  |  $\hat{S}(\hat{a},\hat{\theta})=S(y^{AI},y)$ if $\hat{a}=\text{AI}$
Table 5. FCL [12] decision task under our framework.

##  Appendix C The results of demonstrations using discretized signal
approximation

This appendix includes our additional results for demonstrations in Section 5,
where we use the discretized signals to approximate the rational benchmark and
the mis-reliant rational benchmark. We subsequently re-check the conclusions
we get in Section 5 with the results shown in this appendix. All the
conclusions analyzed under the results of approximation using discretized
signals corroborate with the conclusions we get in Section 5.

###  C.1. Does the Whole Exceed its Parts? [6]

Figure 8. Estimated payoffs of the experiment data in BWZ+ [6].

First, the results also show considerable room for improvement to achieve to
the rational benchmark, as shown in Figure 8A and B. Second, no significant
improvement by displaying explanations is evidenced in the results. As shown
by Figure 8, the behavioral performance and themis-reliant rational benchmark
perform similarly across the explanation conditions and the no explantion
condition. Third, the reliance loss is modest to the behavioral loss, while
the discrimination loss is the main source of loss, as shown in Figure 8C and
D.

###  C.2. On Human Predictions with Explanations and Predictions of Machine
Learning Models [24]

Figure 9. Estimated payoffs of the experiment data in LT [24].

First, similarly to what we get in Section 5, the rational baseline dominates
all other quantities defined by our framework except the rational benchmark,
leading to the conclusion about the failure of complementary performance in
the decision task. Second, the rational benchmark only shows marginal
improvement over the rational baseline, as shown in Figure 8A. Third, the
explanations can improve the behavioral performance and the reliance, as shown
in Figure 8C. Finally, we observed the same pattern of reliance loss and
discrimination loss in the results, e.g., Figure 8D.

###  C.3. The Impact of Algorithmic Risk Assessments on Human Predictions and
its Analysis via Crowdsourcing Studies [12]

Figure 10. Estimated payoffs of the experiment data in FCL [12].

First, we also find the quantities under our framework act differently between
the probabilistic decision task the the binary decision task. For example, the
behavioral performance exceeds the performance of human predictions in the
probabilistic decision task while acts the same in the binary decision task
(Figure 8B). Second, the rational baseline and the rational benchmark have
different values on the anchoring effect condition and the non-anchoring
effect condition, as shown in Figure 8A. Finally, the anchoring effect
condition can improve the reliance loss over the non-anchoring effect
condition, as shown in Figure 8C.