# On the Quest for Effectiveness in Human Oversight:
Interdisciplinary Perspectives

Sarah Sterz  Dependable Systems and Software, Saarland UniversitySaarland
Informatics Campus, SaarbrückenGermany<EMAIL_ADDRESS>,  Kevin
Baum  Neuro-Mechanistic Modeling, German Research Center for Artificial
Intelligence (DFKI)SaarbrückenGermany<EMAIL_ADDRESS>,  Sebastian Biewer
Dependable Systems and Software, Saarland UniversitySaarland Informatics
Campus, SaarbrückenGermany<EMAIL_ADDRESS>,  Holger Hermanns
Dependable Systems and Software, Saarland UniversitySaarland Informatics
Campus, SaarbrückenGermany<EMAIL_ADDRESS>,  Anne Lauber-Rönsberg
IRGET, TU DresdenDresdenGermany<EMAIL_ADDRESS>,  Philip
Meinel  IRGET, TU DresdenDresdenGermany<EMAIL_ADDRESS>and
Markus Langer  Department of Psychology, University of FreiburgFreiburgGermany
<EMAIL_ADDRESS>
###### Abstract.

Human oversight is currently discussed as a potential safeguard to counter
some of the negative aspects of high-risk AI applications. This prompts a
critical examination of the role and conditions necessary for what is
prominently termed _effective_ or _meaningful_ human oversight of these
systems. This paper investigates effective human oversight by synthesizing
insights from psychological, legal, philosophical, and technical domains.
Based on the claim that the main objective of human oversight is risk
mitigation, we propose a viable understanding of effectiveness in human
oversight: for human oversight to be effective, the human overseer has to have
(a) sufficient causal power with regards to the system and its effects, (b)
suitable epistemic access to relevant aspects of the situation, (c) self-
control over their own actions, and (d) fitting intentions for their role.
Furthermore, we argue that this is equivalent to saying that a human overseer
is effective if and only if they are morally responsible and have fitting
intentions. Against this backdrop, we suggest facilitators and inhibitors of
effectiveness in human oversight when striving for practical applicability. We
discuss factors in three domains, namely, the technical design of the system,
individual factors of human overseers, and the environmental circumstances in
which the overseer operates. Finally, this paper scrutinizes the upcoming AI
Act of the European Union – in particular Article 14 on Human Oversight – as
an exemplary regulatory framework in which we study the practicality of our
understanding of effective human oversight. By analyzing the provisions and
implications of the European AI Act proposal, we pinpoint in how far that
proposal aligns with our analyses regarding effective human oversight as well
as how it might get enriched by our conceptual understanding of effectiveness
in human oversight.

Human Oversight and AI Governance and Risk Mitigation and AI Ethics and
Psychology and Law and Responsible AI

We want to thank Sven Hetmank and Franz Lehr for their contributions to
previous work that this paper is based on.

##  1\. Introduction

Effective or meaningful111Henceforth, we focus exclusively on the term
“effective” and omit the term “meaningful”. Given their synonymous use in the
context of human oversight, any statements regarding effectiveness also extend
to meaningfulness. Furthermore we only are interested in _human_ overseers, so
we occasionally drop the first word in “human oversight” for the sake of
simpler wording. human oversight is a notorious centerpiece of ethical
guidelines (Jobin et al., 2019) and global legislation (European Parliament,
2024; Green, 2022) intended to govern the deployment of AI-based systems,
particularly in high-risk contexts. The regulatory significance of human
involvement in automated decision-making is unmistakable, as evidenced by a
range of legislation (Green, 2022; Enqvist, 2023), including the overarching
principles embedded in the General Data Protection Regulation of the EU
(European Union, 2016) and more targeted legislation, such as the European AI
Act (European Parliament, 2024), which mandates human oversight for high-risk
AI applications. Research has made strides in defining and assigning
responsibility for human oversight (Enqvist, 2023), or placing it within a
broader risk management framework (Green, 2022). Human oversight has also been
subject to criticism, with concerns being raised about the efficacy of
oversight in practice, and its potential role as a mere vehicle to legitimize
imperfect AI systems (Green, 2022; Zerilli et al., 2019).

Effective oversight has been discussed by many (Green, 2022; Enqvist, 2023;
Laux, 2023; Methnani et al., 2021), however, a question that is crucial for
this discussion has not been sufficiently addressed: _When is human oversight
effective?_. The pressing need for conceptual clarity in this regard is
evident, particularly given that the European Unions AI Act is to be
implemented in 2024. Nevertheless, the definition and conditions for achieving
_effective_ human involvement remain underexplored, highlighting a critical
gap in the current state of art of AI governance.

The following core contributions are made by this paper, and as such provide a
structural frame for it:

  * •

We suggest a viable understanding of effectiveness in the context of human
oversight based on the idea that the central objective of oversight is risk-
mitigation. Namely, we propose that an overseer is effective if and only if
they have (a) sufficient causal power with regards to the system and its
effects, (b) suitable epistemic access to relevant aspects of the situation,
(c) self-control over their own actions, and (d) fitting intentions for their
role. We point out that (a)–(c) are jointly sufficient for moral
responsibility.

So, phrased boldly, we could say that

Effectiveness $=$ Moral Responsibility $+$ Fitting Intentions.

  * •

We exemplify facilitators and inhibitors of effectiveness in human oversight
in three categories: the technical design of the system, the individual
factors of the human overseer, and the environmental circumstances in which
the overseer operates.

  * •

As a litmus test, we look at the details of the European AI Act, relative to
our proposed conceptualization. We identify possible benefits and shortcomings
of the AI Act. In particular, we argue that our conceptualization provides
both, a more general _and_ a more practically useful conceptualization of
effective human oversight compared to the respective stipulations in the AI
Act.

##  2\. Related Work

Researchers have spelled out who will be in charge of (ensuring) human
oversight and have analyzed how human oversight fits into a more global risk
management concept (as, e.g., in the AI Act (European Parliament, 2024))
together with other approaches on accurate and robust systems, data-privacy
adhering systems, and transparent systems (Enqvist, 2023; Laux, 2023). The
concept of human oversight, however, also faces substantial criticism, as
empirical evidence suggests that humans may be ineffective in supervising AI-
based systems (Green, 2022). This has prompted a call for a more institutional
oversight approach, emphasizing the need for empirical evidence regarding
humans’ capability to oversee AI systems prior to enforcing the oversight
(Green, 2022). This section will explore literature on effective oversight
from different disciplines in more detail.

##### Effective human oversight in legislation and guidelines

Oversight is an essential aspect of various ethical guidelines and legislation
on the use of AI-based systems in high-risk contexts. After the analysis of 41
policy documents around the globe, Green (Green, 2022) observes a call for
limiting solely automated decisions (e.g., Article 22 of the European General
Data Protection Regulation (European Union, 2016)), and an emphasis on the
need for human discretion in AI-based decision-making (e.g., Canadian
Directive on Automated Decision-Making (Government of Canada, 2023)). The
requirement for effective oversight is particularly highlighted in the
European AI Act. At the time of writing, a final version of the AI Act was
approved by the EU Parliament in March 2024 (European Parliament, 2024) but
has not yet been formally adopted. However, we will already use this final
version, but will refer to earlier versions when relevant.

##### Human-centric AI

The focus on human oversight seems to stem from the background of ensuring a
human-centric development and deployment of AI-based systems. One of the most
prominent concerns regarding the wide-range deployment of AI-based systems
seems to be “that technological development and rationalized efficiency will
take place at the cost of human agency and safety or rights” (Enqvist, 2023).
An obvious way to counter such a techno-centric deployment of AI would be a
human-centric development and deployment of AI-based systems. As Enqvist
(Enqvist, 2023) notes: “Human centricity, as it has come to be (broadly)
understood, does not only reflect that human needs are to be met by new
technologies, but also incorporates the aim to safeguard individual rights and
increase human well-being.” Indeed, in computer science a popular area of
research is concerned with _human-centered AI_ (HCAI) which investigates
efficient collaboration between humans and AI systems (Bansal et al., 2019;
Buçinca et al., 2021; Ehsan et al., 2022; Lai et al., 2023; Schoeffer et al.,
2023; Tahaei et al., 2023).

##### Objectives of effective human oversight

One of the foundational assumptions for the effectiveness of oversight
measures is that human oversight should not be an end in itself (Laux, 2023),
but that the involvement of humans in contexts where AI-based systems affect
high-risk decision-making can make things better in some regard. For example,
one hope is that humans may be better than AI-based systems at incorporating
ethical considerations and social norms into decision-making contexts
(Enqvist, 2023). Another hope is that humans might be better at judging single
or unusual cases, for instance, in decisions that would affect individual
human beings (Longoni et al., 2019). Additionally, associating the term
“effective” with human oversight shows the hope that a human can effectively
contribute to a joint decision situation together with an AI-based system,
thus enabling decisions better than those of either the human or the system
alone (Bansal et al., 2021; Buçinca et al., 2021; Schoeffer et al., 2023).
Other hopes include increased safety (Koulu, 2020), improved accuracy (Jones,
2017), or better trustworthiness and more actual trust (Methnani et al.,
2021). The most important objective of human oversight recently seems to be
the mitigation of risks to fundamental rights, both directly (as one reading
of the AI Act (European Parliament, 2024) suggests), or indirectly via, e.g.,
the goal to improve safety or the avoidance of ethically and socially
undesirable outcomes (Enqvist, 2023).

##### Criticism of human oversight

However, these foundational assumptions have received substantial criticism
calling into question the usefulness of human oversight (Green, 2022; Zerilli
et al., 2019). Critics (in particular (Green, 2022)) point to a lack of
empirical evidence showing that we, as of now, can reliably integrate human
and system abilities in a way that leads to better decisions compared to
decisions that each single entity would produce in isolation (Bartlett and
McCarley, 2020; Bansal et al., 2021; Rieger and Manzey, 2022). For instance,
there seem to be substantial challenges with respect to achieving adequate
trust in AI-based systems (Bansal et al., 2019; Lee and See, 2004; Hoff and
Bashir, 2015; Zhang et al., 2020). Humans may overly rely on the outputs
produced by automated systems, leading to situations where they do not detect
erroneous or unfair outputs (Parasuraman and Manzey, 2010). In other
situations, humans may have too little trust in AI-based systems or too high
confidence in their own abilities which leads to people overriding actually
accurate or fair system outputs (Green and Chen, 2019). Especially in high
reliability contexts, automation bias (cf. Section 4.2) is a likely-to-observe
phenomenon where people tend to use the outcome of an automated decision
entity “as a heuristic replacement for vigilant information seeking and
processing” ((Mosier et al., 1996) p. 205). This makes it particularly
challenging for people to detect and adequately address system failures or
erroneous outputs. Whereas research has identified several avenues to improve
human abilities to oversee systems (e.g., training of human overseers (Bahner
et al., 2008) augmenting system outputs with confidence scores (Bahner et al.,
2008; Bansal et al., 2019)), the positive effects associated with these
avenues seem to be inconsistent (Green, 2022). The criticism regarding human
oversight could be expressed in an even more foundational way: The driving
momentum behind many AI-based systems is the intention of producing decisions
and actions that are, in some regard, better than what human experts could
achieve (Green, 2022; Zerilli et al., 2019). Ironically, the very human beings
whose imperfections are meant to be overcome by such an AI-based system end up
in the role of overseers of the system (Zerilli et al., 2019). This is, for
instance, a typical situation in certain areas of AI-based medical diagnostics
(Haggenmüller et al., 2021; Nishida et al., 2022).

##### Positive effects of human oversight

However, it is also possible to draw more positive conclusions from the
literature. There are relevant examples where human operators successfully
identified errors made by automated systems (McBride et al., 2013; De-Arteaga
et al., 2020). In addition, research has enhanced our understanding of factors
that affect human abilities to detect errors made by automated systems (De-
Arteaga et al., 2020; McBride et al., 2013; Lai et al., 2023; Schoeffer et
al., 2023). Furthermore, ongoing research contributes to successfully
unveiling design options to improve human-system collaboration (Lai et al.,
2023; Vasconcelos et al., 2023). For example, recent work proposes to combine
the strengths of machine and human decision making by using so-called
_conformal predictions_ for multi-label classifiers. There, the machine
proposes a set of outputs (instead of a single output), which is guaranteed to
contain the correct output with a probability of at least some pre-defined
value (Straitouri et al., 2023). If this set contains only a single output,
confidence is considered high enough to let the decision happen fully
autonomously. If instead it contains more than one output, the choice is
delegated to a human. It was shown that for certain tasks, the human-system
decision accuracy indeed is higher when the human choice is restricted to the
outputs that are in the proposed set (Straitouri and Rodriguez, 2023).

Overall, research reports mixed results regarding human oversight. However,
the fact that effective human oversight is a popular requirement in
legislation emphasizes the necessity of obtaining clarity on the very concept
of effective human oversight. This clarity is crucial to better devise
powerful laws and guidelines and for ensuring the successful and safe
implementation of AI in high-risk contexts. Thus, in this paper we aim to
derive high-level conditions for effectiveness in human oversight.

##  3\. What is Effective Human Oversight?

We understand human oversight to be the supervision of a system by at least
one natural person, typically with the authority to influence its operations
or effects.222While this understanding of human oversight will serve as our
working definition, it is not the only possible candidate (cf. Section 6). For
further analysis of the concept of human oversight, also see (Laux, 2023).
This influence operates at various levels throughout the system’s lifecycle,
including intervention during execution, reversal of faulty decisions, or
adjustments of parameters to improve results (Methnani et al., 2021). Although
the overseer may be assigned additional responsibilities, such as monitoring
system performance or making engineering decisions, these tasks are not
themselves part of human oversight. Even when having fixated a meaning for
“human oversight”, it is not clear what “effective” means in this regard.

###  3.1. An objective-first approach to effectiveness of human oversight

We want oversight to be useful. Therefore, it makes sense to define the
effectiveness of human oversight in terms of how well it achieves its
objectives. As we have discussed above, risk mitigation can be taken to be one
of the main objectives of human oversight. This is the objective we will be
focusing on. Additionally, effective human oversight not only mitigates risks
but arguably may also yield other positive effects, such as improved
incorporation of ethical considerations (Enqvist, 2023), or enhanced judgment
in cases impacting individual well-being (Longoni et al., 2019).

It is important to note, though, that the understanding of effectiveness we
are about to suggest is not the only viable one. While offering a structured
framework, it is just one among potentially numerous conceptualizations. One
way to change our conceptualization of effectiveness in human oversight would
be to put objectives other than risk mitigation into focus, such as improved
accuracy, trustworthiness, actual trust, human autonomy, accountability and
responsibility, or liability. Discussing them is beyond the scope of this
paper, however, _prima facie_ , there appears to be no inherent conflict
between our proposal and these objectives. Furthermore, we believe that our
conceptualization will be helpful beyond risk mitigation, since it will also
be able to address other objectives of human oversight, including (moral)
responsibility as we will demonstrate in Section 3.2.2.

In line with the proposition that the main objective of effective human
oversight is to mitigate risk, the main question therefore is: _When does
human oversight facilitate the mitigation of risks?_ We propose that this is
the case if the human overseer meets the following four key conditions: causal
power, epistemic access, self-control, and fitting intentions. We believe that
this makes intuitive sense: Usually, risks can be reliably mitigated by a
human only if there is an action available to them that averts the risk
(_causal power_), they can actually perform that action (_self-control_), they
know that the risk is imminent and which action could mitigate it (_epistemic
access_), and they also want to mitigate it (_fitting intentions_). This
intuition can be put more precisely, as is done in the remainder of this
subsection. Additionally, Figure 1 displays the four conditions of
effectiveness.

Figure 1. The four conditions of effectiveness in human oversight and their
relation to moral responsibility. The person in the diagram depicts the human
overseer and the graph a schematic depiction of their current decision
situation: the node on the left is their current position, the edges are the
actions that they could perform, and the nodes on the right are the
consequences of their actions.

####  3.1.1. Causal power

To be effective, the human must be able to make a relevant change. More
precisely:

###### Definition 3.1 (Causal power).

_The agent has the power to establish a sufficient causal connection to the
relevant aspect of the world._

Applied to the context of human oversight, this means that _the human overseer
has the power to establish a causal connection to parts of the system or its
effects, especially the risk_. For instance, this may include the ability to
influence the system, including its components, operation, or outputs. Which
kind of causal power is appropriate and necessary depends on the context at
hand. For example, a factory robot assisting a human in assembly might need a
stop button or a manual control mechanism. For an AI tool that assesses
university applications and makes recommendations for the admission committee
it might be enough if the human overseer can overwrite or disregard the
system’s output and then forward possible issues to the system provider.

In any case, it is not enough if the human overseer is a mere observer of the
system. Without proper means to interfere with it or at least with its
effects, they could only watch the manifestation of a risk without being able
to mitigate it. Also, it is not sufficient if the human only has ‘pseudo-
options’, i.e., if they can only _seemingly_ influence the system but not
really make a relevant difference.333Pseudo-options, though, are not to be
confused with merely unattractive but real options, such as pushing a stop-
button for fun and thereby halting an entire production line for no reason.
Instances of such scenarios are malfunctioning stop buttons, or environments
in which the overseer’s actions are by default not considered.

####  3.1.2. Epistemic access

To be effective, the human overseer must know what to do and when to do it.
More precisely:

###### Definition 3.2 (Epistemic access).

_The agent has sufficient knowledge of their decision situation (also
cf.(Shaffer, 2012))._

Applied to the context of human oversight, this means that _the human overseer
has sufficient knowledge about the risk and how to mitigate it, especially
knowing a risk mitigation action_. A human overseer will, for instance, need
to have sufficient understanding of what the system is doing, in which state
it is, which possible risks and benefits there are, which means of influencing
the system exist, which effects will result from possible interventions, and
which of these effects are most desirable. In some cases, this might include
knowledge of normative, social, or cultural domains, without which it would be
infeasible for the overseer to correctly assess a certain risk. The overseer
does not need to have perfect knowledge, though. They might be unsure or even
ignorant about some aspects of their decision situation – as long as they know
enough to achieve their relevant objectives, they have enough epistemic
access. Notably, we assume that the overseer has certain basic knowledge, such
as what a stop button is, or how to operate a computer on a consumer-level.

####  3.1.3. Self-control

To be effective, the human overseer must be in charge of their own doing. More
precisely:

###### Definition 3.3 (Self-control).

_The agent can decide for any path of action in their decision situation and,
if they do so, follow through with it._ 444In the sense of guidance control in
(Fischer and Ravizza, 1998).

Applied to the context of human oversight, this means that _if here is a risk-
mitigating path of action, the human can decide for that path, and if they do
so, they actually perform the corresponding action(s)_. For this, the human
overseer needs to be in their right state of mind and free in their actions.
They also have to be in a position to retain attention and purposefully target
their actions during the oversight. If, for example, the job of overseeing a
fleet of autonomous vehicles is so uneventful and dull that a normal person
cannot help but let their mind wander off from their task, the oversight would
ultimately become ineffective. Only if human overseers have self-control, they
can purposefully address risks. Other reasons why an overseer could lack self-
control would be when they are severely sleep-deprived, under the influence of
substances, or suffering from an active seizure.

An overseer who lacks self-control might also simultaneously lack sufficient
causal power or epistemic access (e.g., in case of a seizure). Nevertheless,
self-control is distinct from the other two conditions, as an overseer can
have self-control without having causal power or epistemic access. In fact,
self-control may, in many situations, be an antecedent of causal power and
epistemic access – but not vice versa.

Sometimes, human overseers will decide in split-seconds or will instinctively
perform a certain move, and we typically regard this as self-controlled. For
example, a human overseer of a factory robot who realizes that it is just
milliseconds away from severely injuring a human worker might reflexively slam
the stop-button. We consider this to be a real decision, even if made in a
split-second; and since the overseer followed suit with their decision, they
are to be considered self-controlled. Therefore, even overseers deciding
instinctively can be effective according to the upcoming Definition
3.5.555Alternatively, one might say there was not a true decision made in the
example, but rather an instinctive jolt for the stop-button. In this case, the
human overseer would not count as self-controlled according to Definition 3.3
(and therefore also not as effective according to the upcoming Definition
3.5). If one wishes to allow the second interpretation of the situation above
and simultaneously allow that the human in it was exercising effective
oversight, then the definition of self-control might be adapted to say “could”
instead of “can”. In this case, even certain instinctive movements of the
above kind fall under the definition. However, since we want to avoid a
discussion of the conceptual idealization introduced by the “could”, we will
stick to Definition 3.3 as is.

####  3.1.4. Fitting intentions

To be effective, the human must also want to do their job properly. More
precisely:

###### Definition 3.4 (Fitting intentions).

_The agent has intentions that are fitting for their role (or some other
relevant standard)._

Applied to the context of human oversight, this means that _the human overseer
_pro tanto_ intents to mitigate risks, i.e., they intent to mitigate any risks
while taking other relevant factors, such as the interests of the system’s
users and other stakeholders, into consideration in a suitable way._ If the
overseer was unwilling to take the efforts necessary to mitigate a risk, they
would not be effective in their oversight – even if they had causal power,
epistemic access, and self-control. This may, for example, be the case if the
overseer lacks motivation to do their job properly, or has other, conflicting
interests. Even worse, if a human overseer had ill intent, they could use
their position to foster risks instead of mitigating them. For example, a
racist overseeing a university admission system could deliberately overwrite
outputs where people of color are ranked highly. In this case, their oversight
should be considered ineffective in the relevant respects.

####  3.1.5. Effectiveness

The above yields the following conceptualization of effectiveness in human
oversight:

###### Definition 3.5 (Effectiveness).

A human overseer is effective in their human oversight if and only if they
have _causal power_ , _epistemic access_ , _self-control_ , and _fitting
intentions_ in the above senses.

Conceptualized this way, there still is considerable vagueness and generality
to “effective human oversight”, for example, due to the term “sufficient” in
Definitions 3.1 and 3.2, or the “fitting” in Definition 3.4. This intentional
vagueness is not a drawback but rather an essential feature. Not every
oversight context requires the same level of epistemic access or causal power
and, hence, the same exercise of oversight might count as effective in one
context but not in another. Accordingly, our conceptualization of
effectiveness allows for adaptability and versatility, acknowledging the
complexity and diversity of situations where human oversight is applied.

Moreover, “effectiveness” is not a binary concept but a matter of degrees: one
exercise of oversight might be _more effective_ than another. Therefore,
considering effectiveness in human oversight as a multifaceted continuum
suitably encapsulates its nuanced nature. In line with the literature (Green,
2022) and legislation (European Commission, 2021), we nevertheless allow the
use of “effective” as a binary concept. Saying that they are effective is to
mean that they are _effective enough_ for the concrete context they are
operating in. In essence, this is also what Definition 3.5 says. While it is
an open and important question how much effectiveness is enough for a given
context, we will not discuss this issue as it would go far beyond the scope of
this paper.

###  3.2. Further considerations

####  3.2.1. Competence

Another aspect of a human overseer that could potentially come to mind is
their _competence_. Intuitively, a competent human overseer will be better at
being effective. While this is true, there is good reason why competence is
not part of the definition of effectiveness. Roughly, we could say that
someone is competent in doing a task if they know what is the appropriate
thing to do to fulfill the task, and are actually able to do it. So,
competence can be reduced to the conditions of effectiveness, especially to
epistemic access, self-control, and partly to causal power. Accordingly, a
competent human overseer will be more likely to establish and retain these
three conditions in a robust way, even though their competence is not part of
their effectiveness per definition.

####  3.2.2. Responsibility

Our conceptualization of effectiveness in human oversight already accounts for
responsibility while also going beyond that. Taking inspiration from a
definition of moral responsibility in computing in (Noorman, 2020) and being
consistent with the general idea of other sources (Talbert, 2019) we can
define responsibility as follows: An agent is (morally) responsible for some
aspect of the world if and only if  (1) the agent has the power to establish a
sufficient causal connection to this aspect of the world,  (2) the agent has
sufficient knowledge of their decision situation, and  (3) the agent can
decide for any path of action in their decision situation and, if they do so,
follow through with it.  In other words: a human overseer is morally
responsible for risk-mitigation (or any other part of their job) if and only
if they have  (1) causal power,  (2) epistemic access, and  (3) self-control.
This can be easily translated to cover past events: a human overseer was
responsible for, say, mitigating a risk if and only if they had sufficient
causal power, epistemic access, and self-control. These insights leads to the
more concise formula that

A human overseer is effective if and only if they are morally responsible and
have fitting intentions.666If Definition 3.3 were adapted to say “could”,
instead of “can” as suggested in footnote 5, this slogan would need to be
altered to exclude cases of split-second, reflexive movements.

This account also allows for shared responsibility between multiple agents,
which is desirable in the context of high-risk AI systems. If, for example,
both a human overseer and the deployer had causal power, epistemic access, and
self-control with regard to a certain risk, they would both be responsible.
For reasons of simplicity, the definition does not allow for group
responsibility, but can be adapted to do so.777 Our conceptualization of
responsibility should not be considered a blueprint for the legal domain.
While it may share similarities with some (though not all) existing legal
definitions of responsibility, caution is warranted against equating the two,
since legal definitions of responsibility vary significantly across diverse
areas of law, legal systems, and legal cultures.

##  4\. Facilitators and Inhibitors of Effective Human Oversight

Beyond providing a general conceptualization, our proposed definition of
effectiveness in human oversight has a practically useful dimension.
Specifically, its conditions can stimulate ideas about possible facilitators
and inhibitors of effectiveness. Facilitators are expected to promote
effective oversight, while inhibitors are expected to hinder it. We organize
them into three categories: technical design features, individual factors of
human overseers, and environmental circumstances (see, e.g., (Parker and
Grote, 2020) for a similar scheme). The purpose of the discussion in this
section is to provide readers with a more concrete understanding of possible
factors regarding effective human oversight in accordance with our
suggestions. The layout of Table 1 could serve as a template for thinking
through possible facilitators and inhibitors, which can be useful to designers
and deployers of AI-based systems. The following overview of facilitators and
inhibitors is not meant to be exhaustive, and some of them may partially
overlap each other. Also, they generally do not refer to a specific use case
and some may not be applicable to every situation.

###  4.1. Technical design features

As “technical design features” we subsume facilitators and inhibitors whose
influence on effective human oversight results from the design of the system.
This list could also include aspects such as interface and interaction design
(Norman, 1983), runtime monitoring systems (Baumeister et al., 2020), fault
tolerance (Koren and Krishna, 2020), or automated anomaly detection systems
(Nassif et al., 2021).

Intervention options. For an overseer to have causal power over the system,
there must be options that allow the overseer to intervene with, control,
overwrite, or undo system decisions and actions. For example, an intervention
option might be a stop button that allows human overseers to stop system
operation at any time. Other examples include options to take over manual
control or to initiate emergency protocols that result in fail-safe modes.

System adaptability. It can be beneficial to oversight, if overseers can adapt
the system’s processes or its interface (Evangelista Belo et al., 2022; Sauer
and Chavaillaz, 2017). For example, the overseer could reduce the speed at
which the system produces its outputs or adapt the system’s interface to
display additional information. This increases the overseer’s epistemic access
if the adaptations help him to acquire and process the information better.

System understandability. Under “system understandability” we subsume design
aspects associated with the area of explainable AI (Langer et al., 2021). For
example, this includes the use of inherently transparent algorithms, the use
of approaches to explain system outputs, the use of what-if analyses, or the
provisioning of log files to enhance traceability of system processes (Arrieta
et al., 2020; Speith, 2022). All options for enhancing system
understandability aim at improving epistemic access (Langer et al., 2021;
Hoffman et al., 2018).

Interpretability of inputs and outputs. By “interpretability”, we mean a
representation of inputs or outputs that is appropriate for human
understanding (Rudin, 2019). This could include, for example, the provision of
information about what input features mean, or visualizing input and output
features in a way that makes them easier to grasp. Interpretability thus aims
at improving epistemic access.

Preselection of outputs to review. There may be cases where oversight tasks
require a mechanism for selecting system outputs that are to be reviewed by
the overseer. By defining boundary conditions or by using oversight support
tools, human overseers could gain better epistemic access if only having to
review a subset of outputs. For example, in a hiring context, such a tool
could be used identify applicants for whom the AI-based evaluation was
obviously fair and free of bias, leaving considerably fewer cases to the
overseer (Biewer et al., 2024).

|  intervention options |  system adaptability |  system understandability |  interpretability of in- and outputs |  preselection of outputs to review |  overseer training |  domain expertise |  conscientiousness |  exhaustion |  motivation |  automation bias |  adequate job design |  role conflicts |  independent thinking |  accountability |  time pressure
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| technical design | individual factors | environment
causal power | • | • |  |  |  | • | • |  |  |  |  |  |  |  |  | ◦
epistemic access |  | • | • | • | • | • | • | • | ◦ | • | ◦ | • |  | • | • | ◦
self-control |  |  |  |  |  | • |  | • | ◦ | • | ◦ | • |  |  | • |
fitting intentions |  |  |  |  |  | • |  | • | ◦ | • | ◦ | • | ◦ |  |  •/◦ |
Table 1. Overview of how exemplary facilitators (•) and inhibitors (◦) usually
contribute to the effectiveness of human oversight.

###  4.2. Individual factors of human overseers

Under “individual factors of human overseers” we subsume traits and states of
human overseers as well as interventions _on_ human overseers that could
facilitate or inhibit effective oversight. This discussion may cover aspects
like vigilance (Warm et al., 2008), cognitive abilities (Spearman, 1904; Warne
and Burningham, 2018), overseers’ propensity to trust in automation (Hoff and
Bashir, 2015), and additional cognitive biases (Roese and Vohs, 2012) and
heuristics (Tversky and Kahneman, 1973).

Overseer Training. Training to prepare for or enhance the role of the human
overseer will be crucial. For instance, a training in which people are
confronted with system errors or erroneous automated decisions can help them
to become better at identifying these errors when later acting on the real
system (Bahner et al., 2008). Overseer training could in principle target all
of the conditions of effective human oversight. Such training could aim to
enable overseers to maintain epistemic access, to recognize earlier when they
are losing self-control (e.g., become inattentive (Perels et al., 2005)), to
perform certain maneuvers that increase their causal power, or to know what
the job requires them to do and thereby improve the fit of their intentions.

Domain expertise. Overseers may require a certain degree of expertise in the
task performed by the system they are supposed to oversee. For instance,
physicians with substantial expertise in diagnosing a disease may have better
epistemic access (e.g., they will know better when a system is making
incorrect diagnoses) than physicians with little expertise (Gaube et al.,
2023). Domain expertise might sometimes also enable more causal power.

Conscientiousness. Conscientiousness is a character trait of people who are
self-disciplined, orderly, goal-directed, who make and follow plans, and who
adhere to social norms (Costa et al., 1991). Highly conscientious overseers
will be more likely to maintain self-control, sufficient epistemic access, and
fitting intentions.

Exhaustion. Exhaustion is one of many psychological strain symptoms (Sonnentag
et al., 2023). It is a state where an overseer feel tired and worn out
(Maslach et al., 1997) and thus may not be good at maintaining self-control
and sufficient epistemic access.

Motivation. Motivation refers to internal and external factors that drive and
direct an individual’s behavior towards a goal (Ryan and Deci, 2000).
Motivation leads people to initiate, intensify, and persist with goal-directed
behavior. Motivation is often influenced by individual factors, environmental
factors, and the perceived significance of the desired outcomes. In the
context of human oversight, motivation is crucial to initiate and maintain
fitting intentions and self-control for the goal of trying to mitigate risk.
It is also important for persisting in maintaining sufficient epistemic
access.

Automation bias. Automation bias is usually considered a cognitive bias where
people use the outcome of an automated decision aid “as a heuristic
replacement for vigilant information seeking and processing” ((Mosier et al.,
1996), p. 205). Automation bias becomes especially likely when facing a system
that works reliably and with high performance because in such cases it is
rational for people to divert their attention to other tasks (Parasuraman and
Manzey, 2010). This, however, will contribute to insufficient epistemic
access. If there are indeed no negative consequences for some time, this may
lead to what Parasuraman and Manzey (Parasuraman and Manzey, 2010) called
“learned carelessness”. This phenomenon could be interpreted as a problem of
self-control or as an issue that leads to intentions no longer fitting the
duties of a human overseer.

###  4.3. Environmental circumstances

Under “environmental circumstances” we subsume factors that lie outside the
technical system or the human overseer, yet they can influence the
effectiveness of oversight. This discussion could also include aspects such as
stakes of the situation (Langer and Landers, 2021), single vs. multitasking
environments (Lyons and Guznov, 2018), or whether oversight is organized as a
team task (Gao et al., 2014; Ulfert et al., 2023).

Adequate job design. The job of a human overseer needs to be well-designed
(Parker and Grote, 2020) meaning that the job is, e.g., motivating,
satisfying, and not overly demanding. For example, the job characteristics
model888See also other models from psychology that inspire job design to make
the work more motivating and satisfying (e.g., (Humphrey et al., 2007)), or
less demanding (e.g. the job-demands-resources model (Demerouti et al.,
2001)). from work psychology (Hackman and Oldham, 1976) highlights that
certain job characteristics promote motivation and satisfaction with the job,
namely skill variety , task identity , perceived task significance , autonomy
, and feedback from the job. The design of the job is important for promoting
self-control, epistemic access, and fitting intentions, for example via
motivation (cf. Section 4.2). Also, ill-designed jobs can lead to
counterproductive work behavior, i.e., behavior at work that is inconsistent
with or even intentionally disrespecting job duties (e.g., not even trying to
maintain a sufficient epistemic access, or intentionally overwriting outputs
to feel “needed” (Strich et al., 2021)).

Role conflicts. Role conflicts could arise if the overseer is at the same time
a decision-maker who uses the AI-based system to support their decisions (King
and King, 1990). Consider, for example, a hiring scenario where an individual
juggles the roles of both human overseer, tasked with mitigating unfairness,
and an HR manager, who is responsible for inviting only the best candidates to
a job interview. A conflict arises when the only method to address unfairness
involves inviting additional, apparently less suitable candidates, conflicting
with the HR manager’s goal of selecting the best candidates only. This
misalignment of duties may give rise to intentions that are unfitting for an
overseer.

Independent thinking. Processing a situation under the impression of a
system’s output can affect an overseer’s judgment of such an output (Furnham
and Boo, 2011). For example, seeing a recidivism risk score for a defendant at
court in addition to the information about the defendant’s case may affect how
people judge this case. Seeing a “high anchor” (i.e., the system’s prediction
of a high risk for recidivism) will make it more likely that the judge will
end up deciding that the defendant has a high risk for recidivism. Such
anchors make overseers selectively aware of anchor-consistent information
(Englich et al., 2006) which undermines epistemic access.

In the area of AI-supported decision-making, there are several ideas of how to
possibly prevent such effects. For instance, although leading to less
acceptance of the system, it seems to partly counter such anchoring effects if
people are forced to think about a given case themselves for some time before
they make a final decision (Buçinca et al., 2021). Alternatively, overseers
may only receive the AI-based output after they have made up their own mind
about a specific case because then they would not initially be affected by the
AI-based output and may be in a better position to judge whether the AI-based
output is appropriate (Green, 2022; Zerilli et al., 2019).

Accountability. Here we refer to accountability as “an obligation to explain
and to justify [one’s] conduct” ((Bovens, 2007), p.447). If overseers feel
accountable for their actions this may influence their attentiveness or may
lead to them feeling more stressed (Hall et al., 2015). A feeling of
accountability could thus promote efforts of overseers to bring themselves in
a sufficient epistemic position. It can also be hypothesized that a felt
obligation to explain oneself will be motivating to maintain high self-control
in order to keep up with the expectations of the job, and that it will
incentivize to uphold intentions that fit one’s duties. However, it could also
lead to unintended consequences such as leading to unfit intentions: if
overseers believe they would have no good explanation for overriding a system
output, they may not do so in cases where they know that they have to file a
report after overwriting system outputs.

Time pressure. Significant time pressure may undermine various conditions for
effectiveness (Rieger and Manzey, 2022). For example, if an overseer only has
a few seconds to react to warnings by the system, this could hinder sufficient
epistemic access. In extreme cases, time pressure could also undermine an
overseers’ causal power. For instance, if the option to overwrite an output is
only available for split seconds, the overseer may not be able to press the
button.

##  5\. Application to the AI Act Proposal

Finally, our analysis turns to effective human oversight as outlined in the
final version of the AI Act (European Parliament, 2024)999At the time of
writing, the trilogue negotiations between the European Commission, the
Parliament and the Council have been concluded and a finalized version has
been approved by the European Parliament. However, the final version of the AI
Act has not yet been approved by the Council of the EU.. According to Article
14(1) of the proposal, what the AI Act defines as high-risk AI systems shall
be designed and developed in such a way that they can be effectively overseen
during their use. As stated in Article 14(2), this oversight shall aim to
prevent or minimize risks to health, safety or fundamental rights. Risk
mitigation is therefore defined as a key objective of human oversight measures
which should contribute to the overall trustworthiness of high-risk systems
(Enqvist, 2023). Consequently, the effectiveness of human oversight, as
outlined in Article 14(1), serves as a foundational principle that requires
oversight measures to be capable of achieving their risk mitigation objective.
This principle can be understood as a condition for a minimum level of
effectiveness of the oversight measures implemented, comparable to a threshold
that must be reached.

###  5.1. Conditions for effectiveness in our proposal and the AI Act

Article 14(3) obliges the provider to enable its deployer to perform effective
oversight by identifying “appropriate measures”.101010It should be noted that
the Commission proposal (European Commission, 2021) referred to the deployer
as the “user”. The European Parliament (European Parliament, 2023) rightly
proposed to use the term “deployer” instead to avoid any confusion with the
end user of the AI system. This proposal found its way in the version approved
by the Parliament (European Parliament, 2024). We therefore use the term
“deployer”. Article 14(4) contains certain aspects of what the proposed AI Act
envisages as indicators of effectiveness of these measures. These require to
enable the individuals to whom human oversight is assigned to do the
following, as appropriate and proportionate to the circumstances:

> (a) to properly understand the relevant capacities and limitations of the
> high-risk AI system and be able to duly monitor its operation, including in
> view of detecting and addressing anomalies, dysfunctions and unexpected
> performance;
>
> (b) to remain aware of the possible tendency of automatically relying or
> over-relying on the output produced by a high-risk AI system (‘automation
> bias’), in particular for high-risk AI systems used to provide information
> or recommendations for decisions to be taken by natural persons;
>
> (c) to correctly interpret the high-risk AI system’s output, taking into
> account, for example, the interpretation tools and methods available;
>
> (d) to decide, in any particular situation, not to use the high-risk AI
> system or to otherwise disregard, override or reverse the output of the
> high-risk AI system;
>
> (e) to intervene in the operation of the high-risk AI system or interrupt
> the system through a ‘stop’ button or a similar procedure that allows the
> system to come to a halt in a safe state.
>
> (quoted from Article 14(4) of the final version approved by the European
> Parliament(European Parliament, 2024))

These measures reflect our conditions of effectiveness. Requirements (d) and
(e) underscore the EU’s recognition of the importance of human overseers being
able to influence the AI system, either by modifying the system’s output or by
stopping its operation. These requirements manifest the need for both causal
power and self-control, ensuring that individuals can influence the decisions
of the AI system, or decide not to use the system’s output or to modify it.
Conditions (a) and (c) relate to the understanding of the system, as they
require the provider to enable the overseer to understand the limitations of
the system (though not necessarily all details of the system), to monitor it
effectively, and to interpret its results correctly. This is consistent with
the concept of epistemic access, which enables the overseer to make informed
decisions. Awareness of automation bias, as described in (b), is intertwined
with epistemic access and self-control, as described above in Section 4.2.
Furthermore, the conditions for effectiveness must also be implemented by the
deployer. In this regard, Article 26(2) (European Commission, 2021) emphasizes
the need for the necessary competence, training, authority, and support in
oversight functions, hence requiring epistemic access and causal power.
111111This requirement was mentioned only in the non-binding recital (48) of
the EU Commission’s proposal (European Commission, 2021), but was ultimately
stipulated more directly in Article 29 of the proposals by the EU Council
(Council of the European Union, 2022), the Parliament (European Parliament,
2023) and Article 26 of the approved version (European Parliament, 2024). The
criterion of fitting intentions of the agents involved has little direct
equivalent in the legal text, which is not focused on subjective intentions
but on actions. The parties’ underlying intentions and motivations are not
directly relevant. One might consider, however, that the automation bias
targeted in Article 14(4) (b) may interfere with fitting intentions of the
human overseer (cf. Section 4.2). Furthermore, the significant penalties
according to Article 99 (European Parliament, 2024) strongly incentivize norm-
compliant behavior.

###  5.2. Further analysis of the EU legislation’s approach

##### Strongly varying levels of abstraction

It is striking that the measures appearing in Article 14(4) are of varying
degrees of abstraction: While some measures contain specific requirements for
the design of the AI-system, other measures are less specific. For example,
the requirement of a “stop” button in (e) is highly concrete. In contrast, the
requirement in (a) to create an understanding of the limitations of the system
is vague and leaves a lot of room for interpretation. Overall, the
requirements for the various players – providers of the AI system, deployers,
the person entrusted with oversight – could have been made clearer in the
structure of the AI Act. Thus, Article 14(4) appears to be a loose collection
of items that were deemed useful in the legislative process without much
structural consistency. The lack of regulatory clarity could hinder the
practical applicability of Article 14 and thus the introduction of truly
effective human oversight.

##### Significance of Psychological Biases

Measures in accordance with (b), which are intended to maintain an awareness
of the automation bias, seem out of place: it introduces a concrete
psychological aspect that sticks out from the more technical and operational
aspects outlined in the other provisions, especially considering that
automation bias is just one of many relevant biases (e.g., hindsight bias,
confidence bias (Hilbert, 2012; Roese and Vohs, 2012)), heuristics (e.g.,
availability heuristic, (Tversky and Kahneman, 1973)), and psychological
phenomena (e.g., exhaustion, motivation, (Ryan and Deci, 2000; Sonnentag et
al., 2023)) that would need to be accounted for.

##### Legal uncertainties

Not every measure in Article 14(4) will have to be fully implemented in every
high-risk system. This already follows from the restriction that the measures
should be implemented “as appropriate and proportionate to the circumstances”.
As a result, the specific measures taken must be assessed on a case-by-case
basis. Not only _how_ these requirements apply, but also _which_ requirements
are applicable therefore depends heavily on the individual case and the design
of the system. While this approach provides flexibility that accommodates the
sometimes beneficial vagueness of the effectiveness of human oversight also
mentioned in Section 3.1.5, it may also be associated with greater legal
uncertainty. The legislation will be complemented by technical standards
developed by European standardization organisations to at least partially
remove this legal uncertainty and to facilitate the implementation of
requirements for high-risk systems, including human oversight (Veale and
Borgesius, 2021). However, this standardization process is still in its early
stages and is limited to a few general guidelines (Laux, 2023). Moreover, the
question arises as to what requirements these standards must meet to satisfy
the effectiveness requirement of Article 14(1). While Article 14(4) offers
some guidance, its vagueness and varying levels of abstraction are
insufficient for determining true effectiveness in specific cases. Thereby,
the standardization organisations risk developing technical standards that may
not withstand legal scrutiny. So, the EU’s reliance on the AI Act’s effective
implementation through practical technical standards poses a potential threat
to the Regulation’s requirement for human oversight.

##### The practical need for a definition

Also, the technical standards created in this process only have a presumptive
effect for providers and deployers of AI systems that are subject to the AI
Act. In the event of a legal dispute, the courts will therefore have to
examine whether the specific design of the human oversight measures meets the
overarching requirement of effectiveness in Article 14(1). As mentioned above,
this will depend on the circumstances of the individual case. To ensure
uniform application of the AI Act by lower courts, a comprehensible definition
of effective human oversight is essential. In this respect, the definition of
effectiveness will be one of the key tasks of the deciding courts in the
coming years, leaving these courts also in need of further clarification on
the conditions of effectiveness.

###  5.3. Utilizing our definition of effectiveness

Our objective-first approach to effectiveness can be used to integrate the
piecemeal list of different requirements listed in the AI Act into a coherent
system. This is advantageous for a legally sound application of the AI Act, as
it helps to structure the requirements for and responsibilities of the
different parties involved. One of the benefits of the breadth and structure
of our conceptualization is that it simplifies the assessment process for
regulators, making it easier to evaluate different AI systems against a
structured set of principles that are framed by the four conditions. The three
domains of facilitators and inhibitors can aid in giving further structure for
certain use cases. This streamlining may also decrease the legal uncertainty
in Article 14, fostering a more consistent and predictable environment for AI
providers and deployers alike, while providing a practical definition of
effectiveness for deciding courts. This could also help to uncover further
shortcomings in Article 14(4): For instance, while understanding the system’s
output (c) can enhance epistemic access, understanding of the system’s _input_
can be equally crucial. Our conditions for effectiveness, when applied with
expertise, make this need evident; whereas this need can be challenging to
identify when relying solely on Article 14(4), which does not mention the
system’s input at all.

Consequently, the points listed in Table 1 can be understood as examples of
how an assessment of the effectiveness of measures could be carried out.
Defining effectiveness using our four conditions provides a practical
framework that highlights the key aspects of effective oversight. Our approach
is therefore well suited to assist standard-setting organizations and to
clarify the rather broad requirements of Article 14(4). In addition, our
proposed framework provides guidance to practitioners who do not wish to rely
solely on technical standards. Finally, our approach can assist courts in the
future by providing a framework for defining and assessing the effectiveness
of human oversight.

##  6\. Limitations

As discussed above, we see advantages brought about by our conceptualization
of effective human oversight. Nevertheless, we also want to point out three
limitations as well as future points to address. _First_ , our
conceptualization is designed for broad applicability across various AI-based
systems, and thus shares a common criticism with the AI Act, namely that of
residing on a high level of abstraction and that of vagueness, as discussed in
Section 3.1.5. Dealing with these issues, especially in complex scenarios,
requires (a) involving individuals with extensive expertise in the task at
hand and (b) fostering collaboration in multidisciplinary teams to assess the
impact on human oversight practice. _Second_ , when we came up with the
facilitators and inhibitors spelled out in Section 4, we realized that our
conceptualization is helpful for thinking through conditions that could
promote or hinder effective oversight. However, for some of the facilitators
(or inhibitors), it depends on the concrete context which of the four
conditions they contribute to (or hinder). Moreover, for certain cases, the
mapping from facilitators and inhibitors to conditions needs considerable
empirical research, outside the scope of this conceptual paper. _Third_ , we
did not discuss the concept of human oversight itself in much detail, but
focused only _effectiveness_ of human oversight. So, while our paper is
helpful in determining whether an instance of human oversight is effective, it
is less helpful in assessing whether something is human oversight. However,
our definition of effectiveness is broad enough to be applied to many roles
along the AI pipeline, not only to overseers. Some more conceptual discussion
of human oversight itself is for instance provided by (Laux, 2023).

##  7\. Conclusion

This paper has proposed an approach to effective human oversight –
encompassing causal power, epistemic access, self-control, and fitting
intentions – to remedy the lack of clarity surrounding the concept of
effectiveness. This is a joint contribution of researchers in psychology, law,
philosophy, and computer science. We argued that, in essence, a morally
responsible overseer with fitting intentions is generally suitable for
mitigating risks associated with high-risk AI systems. We have identified
facilitators and inhibitors of effectiveness in three categories, namely the
technical design of the system, individual characteristics of human overseers,
and the environmental circumstances in which oversight occurs. Thereby, we
have provided inspiration for steps towards a successful implementation of
effective human oversight. We also discussed the extent to which our
understanding of effective human oversight aligns with the European Union’s
upcoming AI Act, in particular Article 14 on human oversight. Our analysis
suggests that the AI Act could benefit from incorporating more nuanced and
structured conditions, such as those proposed in this paper.

In closing, we want to advocate for a balanced approach to effective human
oversight. Without a concrete idea on what effectiveness in human oversight
is, no productive discussion of it is possible. On the one hand, we urge to
not throw out the baby with the bathwater by per se dismissing human oversight
as a safeguard without the necessary conceptual clarity. On the other hand,
caution is warranted as well when emphatically including a notion of effective
human oversight in regulations and guidelines without this conceptual clarity.
Our interdisciplinary approach is intended to aid both the advocates and the
critics of human oversight. For many relevant oversight contexts, the four
conditions of effectiveness seem to provide a better picture on when and how
human oversight is useful. We, therefore, believe that these conditions could
form a pivot for future research on human oversight.

## Acknowledgements

This work is partially funded by DFG grant 389792660 as part of TRR 248 –
CPEC, by VolkswagenStiftung as part of grants AZ 98514, 98513 and 98512 – EIS,
and by the European Regional Development Fund and the Saarland within the
scope of (To)CERTAIN.

## References

  * (1)
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020\.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ 58 (2020), 82–115.
  * Bahner et al. (2008) J. Elin Bahner, Monika F. Elepfandt, and Dietrich Manzey. 2008.  Misuse of Diagnostic Aids in Process Control: The Effects of Automation Misses on Complacency and Automation Bias: (578262012-006).  https://doi.org/10.1037/e578262012-006
  * Bansal et al. (2019) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019.  Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ 7 (Oct. 2019), 2–11.  https://doi.org/10.1609/hcomp.v7i1.5285
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021.  Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. ACM.  https://doi.org/10.1145/3411764.3445717
  * Bartlett and McCarley (2020) Megan L. Bartlett and Jason S. McCarley. 2020.  Ironic efficiency in automation-aided signal detection.  _Ergonomics_ 64, 1 (Aug. 2020), 103–112.  https://doi.org/10.1080/00140139.2020.1809716
  * Baumeister et al. (2020) Jan Baumeister, Bernd Finkbeiner, Sebastian Schirmer, Maximilian Schwenger, and Christoph Torens. 2020.  RTLola Cleared for Take-Off: Monitoring Autonomous Aircraft. In _CAV 2020_ _(LNCS, Vol. 12225)_. Springer, 28–39.  https://doi.org/10.1007/978-3-030-53291-8_3
  * Biewer et al. (2024) Sebastian Biewer, Kevin Baum, Sarah Sterz, Holger Hermanns, Sven Hetmank, Markus Langer, Anne Lauber-Rönsberg, and Franz Lehr. 2024.  Software doping analysis for human oversight.  _Formal Methods in System Design_ (April 2024).  https://doi.org/10.1007/s10703-024-00445-2
  * Bovens (2007) Mark Bovens. 2007.  Analysing and Assessing Accountability: A Conceptual Framework1.  _European Law Journal_ 13, 4 (June 2007), 447–468.  https://doi.org/10.1111/j.1468-0386.2007.00378.x
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021.  To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (April 2021), 1–21.  https://doi.org/10.1145/3449287
  * Costa et al. (1991) Paul T. Costa, Robert R. McCrae, and David A. Dye. 1991.  Facet Scales for Agreeableness and Conscientiousness: A Revision of the NEO Personality Inventory.  _Personality and Individual Differences_ 12, 9 (1991), 887–898.  https://doi.org/10.1016/0191-8869(91)90177-d
  * Council of the European Union (2022) Council of the European Union. 2022.  Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts – General Approach, 14954/22.  https://data.consilium.europa.eu/doc/document/ST-14954-2022-INIT/en/pdf.
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020.  A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM.  https://doi.org/10.1145/3313831.3376638
  * Demerouti et al. (2001) Evangelia Demerouti, Arnold B. Bakker, Friedhelm Nachreiner, and Wilmar B. Schaufeli. 2001.  The job demands-resources model of burnout.  _Journal of Applied Psychology_ 86, 3 (2001), 499–512.  https://doi.org/10.1037//0021-9010.86.3.499
  * Ehsan et al. (2022) Upol Ehsan, Philipp Wintersberger, Q. Vera Liao, Elizabeth Anne Watkins, Carina Manger, Hal Daumé III, Andreas Riener, and Mark O Riedl. 2022.  Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_ _(CHI ’22)_. ACM.  https://doi.org/10.1145/3491101.3503727
  * Englich et al. (2006) Birte Englich, Thomas Mussweiler, and Fritz Strack. 2006.  Playing Dice With Criminal Sentences: The Influence of Irrelevant Anchors on Experts’ Judicial Decision Making.  _Personality and Social Psychology Bulletin_ 32, 2 (Feb. 2006), 188–200.  https://doi.org/10.1177/0146167205282152
  * Enqvist (2023) Lena Enqvist. 2023.  ‘Human oversight’in the EU artificial intelligence act: what, when and by whom?  _Law, Innovation and Technology_ 15, 2 (2023), 508–535.
  * European Commission (2021) European Commission. 2021.  Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts (proposal for a regulation) no 0106/2021.  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206.
  * European Parliament (2023) European Parliament. 2023.  Amendments adopted by the European Parliament on 14 June 2023 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts (COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)), P9_TA(2023)0236.  https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.pdf.
  * European Parliament (2024) European Parliament. 2024.  European Parliament legislative resolution of 13 March 2024 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts (COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)), P9_TA(2024)0138.  https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.pdf.
  * European Union (2016) European Union. 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation).  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679.
  * Evangelista Belo et al. (2022) João Marcelo Evangelista Belo, Mathias N. Lystbæk, Anna Maria Feit, Ken Pfeuffer, Peter Kán, Antti Oulasvirta, and Kaj Grønbæk. 2022.  AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ _(UIST ’22)_. ACM.  https://doi.org/10.1145/3526113.3545651
  * Fischer and Ravizza (1998) John Martin Fischer and Mark Ravizza. 1998.  _Responsibility and Control: A Theory of Moral Responsibility_.  Cambridge University Press, New York.
  * Furnham and Boo (2011) Adrian Furnham and Hua Chu Boo. 2011.  A literature review of the anchoring effect.  _The Journal of Socio-Economics_ 40, 1 (Feb. 2011), 35–42.  https://doi.org/10.1016/j.socec.2010.10.008
  * Gao et al. (2014) Fei Gao, Mary L. Cummings, and Erin Treacy Solovey. 2014.  Modeling Teamwork in Supervisory Control of Multiple Robots.  _IEEE Transactions on Human-Machine Systems_ 44, 4 (Aug. 2014), 441–453.  https://doi.org/10.1109/thms.2014.2312391
  * Gaube et al. (2023) Susanne Gaube, Harini Suresh, Martina Raue, Eva Lermer, Timo K. Koch, Matthias F. C. Hudecek, Alun D. Ackery, Samir C. Grover, Joseph F. Coughlin, Dieter Frey, Felipe C. Kitamura, Marzyeh Ghassemi, and Errol Colak. 2023.  Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays.  _Scientific Reports_ 13, 1 (Jan. 2023).  https://doi.org/10.1038/s41598-023-28633-w
  * Government of Canada (2023) Government of Canada. 2023.  Directive on Automated Decision-Making.  https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32746&section=html
  * Green (2022) Ben Green. 2022.  The flaws of policies requiring human oversight of government algorithms.  _Computer Law & Security Review_ 45 (2022), 105681.  https://doi.org/10.1016/j.clsr.2022.105681
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ _(FAT* ’19)_. ACM.  https://doi.org/10.1145/3287560.3287563
  * Hackman and Oldham (1976) J.Richard Hackman and Greg R. Oldham. 1976.  Motivation through the design of work: test of a theory.  _Organizational Behavior and Human Performance_ 16, 2 (Aug. 1976), 250–279.  https://doi.org/10.1016/0030-5073(76)90016-7
  * Haggenmüller et al. (2021) Sarah Haggenmüller, Roman C. Maron, Achim Hekler, Jochen S. Utikal, Catarina Barata, Raymond L. Barnhill, Helmut Beltraminelli, Carola Berking, Brigid Betz-Stablein, Andreas Blum, Stephan A. Braun, Richard Carr, Marc Combalia, Maria-Teresa Fernandez-Figueras, Gerardo Ferrara, Sylvie Fraitag, Lars E. French, Frank F. Gellrich, Kamran Ghoreschi, Matthias Goebeler, Pascale Guitera, Holger A. Haenssle, Sebastian Haferkamp, Lucie Heinzerling, Markus V. Heppt, Franz J. Hilke, Sarah Hobelsberger, Dieter Krahl, Heinz Kutzner, Aimilios Lallas, Konstantinos Liopyris, Mar Llamas-Velasco, Josep Malvehy, Friedegund Meier, Cornelia S.L. Müller, Alexander A. Navarini, Cristián Navarrete-Dechent, Antonio Perasole, Gabriela Poch, Sebastian Podlipnik, Luis Requena, Veronica M. Rotemberg, Andrea Saggini, Omar P. Sangueza, Carlos Santonja, Dirk Schadendorf, Bastian Schilling, Max Schlaak, Justin G. Schlager, Mildred Sergon, Wiebke Sondermann, H. Peter Soyer, Hans Starz, Wilhelm Stolz, Esmeralda Vale, Wolfgang Weyers, Alexander Zink, Eva Krieghoff-Henning, Jakob N. Kather, Christof von Kalle, Daniel B. Lipka, Stefan Fröhling, Axel Hauschild, Harald Kittler, and Titus J. Brinker. 2021.  Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts.  _European Journal of Cancer_ 156 (2021), 202–216.  https://doi.org/10.1016/j.ejca.2021.06.049
  * Hall et al. (2015) Angela T. Hall, Dwight D. Frink, and M. Ronald Buckley. 2015.  An accountability account: A review and synthesis of the theoretical and empirical research on felt accountability.  _Journal of Organizational Behavior_ 38, 2 (Sept. 2015), 204–224.  https://doi.org/10.1002/job.2052
  * Hilbert (2012) Martin Hilbert. 2012.  Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making.  _Psychological Bulletin_ 138, 2 (March 2012), 211–237.  https://doi.org/10.1037/a0025940
  * Hoff and Bashir (2015) Kevin Anthony Hoff and Masooda Bashir. 2015.  Trust in automation: Integrating empirical evidence on factors that influence trust.  _Human factors_ 57, 3 (2015), 407–434.
  * Hoffman et al. (2018) Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for explainable AI: Challenges and prospects.  _arXiv preprint arXiv:1812.04608_ (2018).
  * Humphrey et al. (2007) Stephen E. Humphrey, Jennifer D. Nahrgang, and Frederick P. Morgeson. 2007.  Integrating motivational, social, and contextual work design features: A meta-analytic summary and theoretical extension of the work design literature.  _Journal of Applied Psychology_ 92, 5 (2007), 1332–1356.  https://doi.org/10.1037/0021-9010.92.5.1332
  * Jobin et al. (2019) Anna Jobin, Marcello Ienca, and Effy Vayena. 2019.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (Sept. 2019), 389–399.  https://doi.org/10.1038/s42256-019-0088-2
  * Jones (2017) Meg Leta Jones. 2017.  The right to a human in the loop: Political constructions of computer automation and personhood.  _Social Studies of Science_ 47, 2 (2017), 216–239.  https://doi.org/10.1177/0306312717699716 PMID: 28406392.
  * King and King (1990) Lynda A. King and Daniel W. King. 1990.  Role conflict and role ambiguity: A critical assessment of construct validity.  _Psychological Bulletin_ 107, 1 (1990), 48–64.  https://doi.org/10.1037//0033-2909.107.1.48
  * Koren and Krishna (2020) Israel Koren and C Mani Krishna. 2020.  _Fault-tolerant systems_.  Morgan Kaufmann.
  * Koulu (2020) Riikka Koulu. 2020.  Proceduralizing control and discretion: Human oversight in artificial intelligence policy.  _Maastricht Journal of European and Comparative Law_ 27 (2020), 720 – 735.  https://api.semanticscholar.org/CorpusID:229412494
  * Lai et al. (2023) Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023.  Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In _2023 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’23)_. ACM.  https://doi.org/10.1145/3593013.3594087
  * Langer and Landers (2021) Markus Langer and Richard N. Landers. 2021.  The future of artificial intelligence at work: A review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers.  _Computers in Human Behavior_ 123 (Oct. 2021), 106878.  https://doi.org/10.1016/j.chb.2021.106878
  * Langer et al. (2021) Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. 2021.  What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research.  _Artificial Intelligence_ 296 (July 2021), 103473.  https://doi.org/10.1016/j.artint.2021.103473
  * Laux (2023) Johann Laux. 2023.  Institutionalised Distrust and Human Oversight of Artificial Intelligence: Toward a Democratic Design of AI Governance under the European Union AI Act.  _Available at SSRN 4377481_ (2023).
  * Lee and See (2004) John D Lee and Katrina A See. 2004.  Trust in automation: Designing for appropriate reliance.  _Human factors_ 46, 1 (2004), 50–80.
  * Longoni et al. (2019) Chiara Longoni, Andrea Bonezzi, and Carey K Morewedge. 2019.  Resistance to Medical Artificial Intelligence.  _Journal of Consumer Research_ 46, 4 (May 2019), 629–650.  https://doi.org/10.1093/jcr/ucz013
  * Lyons and Guznov (2018) Joseph B. Lyons and Svyatoslav Y. Guznov. 2018.  Individual differences in human–machine trust: A multi-study look at the perfect automation schema.  _Theoretical Issues in Ergonomics Science_ 20, 4 (Nov. 2018), 440–458.  https://doi.org/10.1080/1463922x.2018.1491071
  * Maslach et al. (1997) Christina Maslach, Susan E Jackson, and Michael P Leiter. 1997.  _Maslach burnout inventory_.  Scarecrow Education.
  * McBride et al. (2013) Sara E. McBride, Wendy A. Rogers, and Arthur D. Fisk. 2013.  Understanding human management of automation errors.  _Theoretical Issues in Ergonomics Science_ 15, 6 (Aug. 2013), 545–577.  https://doi.org/10.1080/1463922x.2013.817625
  * Methnani et al. (2021) Leila Methnani, Andrea Aler Tubella, Virginia Dignum, and Andreas Theodorou. 2021.  Let Me Take Over: Variable Autonomy for Meaningful Human Control.  _Frontiers in Artificial Intelligence_ 4 (2021).  https://doi.org/10.3389/frai.2021.737072
  * Mosier et al. (1996) Kathleen L. Mosier, Linda J. Skitka, Mark D. Burdick, and Susan T. Heers. 1996.  Automation Bias, Accountability, and Verification Behaviors.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 40, 4 (Oct. 1996), 204–208.  https://doi.org/10.1177/154193129604000413
  * Nassif et al. (2021) Ali Bou Nassif, Manar Abu Talib, Qassim Nasir, and Fatima Mohamad Dakalbab. 2021.  Machine Learning for Anomaly Detection: A Systematic Review.  _IEEE Access_ 9 (2021), 78658–78700.  https://doi.org/10.1109/ACCESS.2021.3083060
  * Nishida et al. (2022) Naoshi Nishida, Makoto Yamakawa, Tsuyoshi Shiina, Yoshito Mekada, Mutsumi Nishida, Naoya Sakamoto, Takashi Nishimura, Hiroko Iijima, Toshiko Hirai, Ken Takahashi, Masaya Sato, Ryosuke Tateishi, Masahiro Ogawa, Hideaki Mori, Masayuki Kitano, Hidenori Toyoda, Chikara Ogawa, and Masatoshi Kudo. 2022.  Artificial intelligence (AI) models for the ultrasonographic diagnosis of liver tumors and comparison of diagnostic accuracies between AI and human experts.  _Journal of Gastroenterology_ 57, 4 (Feb. 2022), 309–321.  https://doi.org/10.1007/s00535-022-01849-9
  * Noorman (2020) Merel Noorman. 2020.  Computing and Moral Responsibility.  In _The Stanford Encyclopedia of Philosophy_ (Spring 2020 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  * Norman (1983) Donald A. Norman. 1983.  Design principles for human-computer interfaces. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1983, Boston, Massachusetts, USA, December 12-15, 1983_ , Raoul N. Smith, Richard W. Pew, and Ann Janda (Eds.). ACM, 1–10.  https://doi.org/10.1145/800045.801571
  * Parasuraman and Manzey (2010) Raja Parasuraman and Dietrich H. Manzey. 2010.  Complacency and Bias in Human Use of Automation: An Attentional Integration.  _Human Factors: The Journal of the Human Factors and Ergonomics Society_ 52, 3 (June 2010), 381–410.  https://doi.org/10.1177/0018720810376055
  * Parker and Grote (2020) Sharon K. Parker and Gudela Grote. 2020.  Automation, Algorithms, and Beyond: Why Work Design Matters More Than Ever in a Digital World.  _Applied Psychology_ 71, 4 (Feb. 2020), 1171–1204.  https://doi.org/10.1111/apps.12241
  * Perels et al. (2005) Franziska Perels, Tina Gürtler, and Bernhard Schmitz. 2005.  Training of self-regulatory and problem-solving competence.  _Learning and Instruction_ 15, 2 (April 2005), 123–139.  https://doi.org/10.1016/j.learninstruc.2005.04.010
  * Rieger and Manzey (2022) Tobias Rieger and Dietrich Manzey. 2022.  Understanding the Impact of Time Pressure and Automation Support in a Visual Search Task.  _Human Factors: The Journal of the Human Factors and Ergonomics Society_ (June 2022), 001872082211112.  https://doi.org/10.1177/00187208221111236
  * Roese and Vohs (2012) Neal J. Roese and Kathleen D. Vohs. 2012.  Hindsight Bias.  _Perspectives on Psychological Science_ 7, 5 (Sept. 2012), 411–426.  https://doi.org/10.1177/1745691612454303
  * Rudin (2019) Cynthia Rudin. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1, 5 (May 2019), 206–215.  https://doi.org/10.1038/s42256-019-0048-x
  * Ryan and Deci (2000) Richard M. Ryan and Edward L. Deci. 2000.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions.  _Contemporary Educational Psychology_ 25, 1 (Jan. 2000), 54–67.  https://doi.org/10.1006/ceps.1999.1020
  * Sauer and Chavaillaz (2017) Juergen Sauer and Alain Chavaillaz. 2017.  How operators make use of wide-choice adaptable automation: observations from a series of experimental studies.  _Theoretical Issues in Ergonomics Science_ 19, 2 (March 2017), 135–155.  https://doi.org/10.1080/1463922x.2017.1297866
  * Schoeffer et al. (2023) Jakob Schoeffer, Johannes Jakubik, Michael Voessing, Niklas Kuehl, and Gerhard Satzger. 2023.  _On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making_.  IOS Press.  https://doi.org/10.3233/faia230074
  * Shaffer (2012) Michael J. Shaffer. 2012.  _Epistemic Access, Confirmation, and Idealization_.  Palgrave Macmillan UK, London, 101–144.  https://doi.org/10.1057/9781137271587_4
  * Sonnentag et al. (2023) Sabine Sonnentag, Louis Tay, and Hadar Nesher Shoshan. 2023.  A review on health and well-being at work: More than stressors and strains.  _Personnel Psychology_ 76, 2 (Jan. 2023), 473–510.  https://doi.org/10.1111/peps.12572
  * Spearman (1904) Charles Spearman. 1904.  " General Intelligence" Objectively Determined and Measured.  15 (1904), 201–292.
  * Speith (2022) Timo Speith. 2022.  A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’22)_. ACM.  https://doi.org/10.1145/3531146.3534639
  * Straitouri and Rodriguez (2023) Eleni Straitouri and Manuel Gomez Rodriguez. 2023.  Designing Decision Support Systems Using Counterfactual Prediction Sets.  _CoRR_ abs/2306.03928 (2023).  https://doi.org/10.48550/ARXIV.2306.03928 arXiv:2306.03928
  * Straitouri et al. (2023) Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. 2023.  Improving Expert Predictions with Conformal Prediction. In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ _(Proceedings of Machine Learning Research, Vol. 202)_ , Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 32633–32653.  https://proceedings.mlr.press/v202/straitouri23a.html
  * Strich et al. (2021) Franz Strich, Anne-Sophie Mayer, and Marina Fiedler. 2021.  What Do I Do in a World of Artificial Intelligence? Investigating the Impact of Substitutive Decision-Making AI Systems on Employees’ Professional Role Identity.  _Journal of the Association for Information Systems_ 22, 2 (2021), 304–324.  https://doi.org/10.17705/1jais.00663
  * Tahaei et al. (2023) Mohammad Tahaei, Marios Constantinides, Daniele Quercia, Sean Kennedy, Michael Muller, Simone Stumpf, Q Vera Liao, Ricardo Baeza-Yates, Lora Aroyo, Jess Holbrook, et al. 2023\.  Human-Centered Responsible Artificial Intelligence: Current & Future Trends. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–4.
  * Talbert (2019) Matthew Talbert. 2019.  Moral Responsibility.  In _The Stanford Encyclopedia of Philosophy_ (Winter 2019 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  * Tversky and Kahneman (1973) Amos Tversky and Daniel Kahneman. 1973.  Availability: A heuristic for judging frequency and probability.  _Cognitive Psychology_ 5, 2 (Sept. 1973), 207–232.  https://doi.org/10.1016/0010-0285(73)90033-9
  * Ulfert et al. (2023) Anna-Sophie Ulfert, Eleni Georganta, Carolina Centeio Jorge, Siddharth Mehrotra, and Myrthe Tielman. 2023.  Shaping a multidisciplinary understanding of team trust in human-AI teams: a theoretical framework.  _European Journal of Work and Organizational Psychology_ (April 2023), 1–14.  https://doi.org/10.1080/1359432x.2023.2200172
  * Vasconcelos et al. (2023) Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S. Bernstein, and Ranjay Krishna. 2023.  Explanations Can Reduce Overreliance on AI Systems During Decision-Making.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (April 2023), 1–38.  https://doi.org/10.1145/3579605
  * Veale and Borgesius (2021) Michael Veale and Frederik J. Zuiderveen Borgesius. 2021.  Demystifying the Draft EU Artificial Intelligence Act.  _CoRR_ abs/2107.03721 (2021).  arXiv:2107.03721 https://arxiv.org/abs/2107.03721
  * Warm et al. (2008) Joel S. Warm, Raja Parasuraman, and Gerald Matthews. 2008.  Vigilance Requires Hard Mental Work and Is Stressful.  _Human Factors: The Journal of the Human Factors and Ergonomics Society_ 50, 3 (June 2008), 433–441.  https://doi.org/10.1518/001872008x312152
  * Warne and Burningham (2018) Russell T. Warne and Cassidy Burningham. 2018.  Spearman’s g Found in 31 Non-Western Nations: Strong Evidence that g is a Universal Phenomenon.  (March 2018).  https://doi.org/10.31234/osf.io/uv673
  * Zerilli et al. (2019) John Zerilli, Alistair Knott, James Maclaurin, and Colin Gavaghan. 2019.  Algorithmic Decision-Making and the Control Problem.  _Minds and Machines_ 29, 4 (Dec. 2019), 555–578.  https://doi.org/10.1007/s11023-019-09513-7
  * Zhang et al. (2020) Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020.  Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. ACM.  https://doi.org/10.1145/3351095.3372852