# Expressing Accountability Patterns using Structural Causal Models

Severin Kacianka<EMAIL_ADDRESS>Technical University of
MunichBoltzmannstr. 3MunichGermany85748 ,  Amjad Ibrahim<EMAIL_ADDRESS>Technical University of MunichBoltzmannstr. 3MunichGermany85748 and  Alexander
Pretschner<EMAIL_ADDRESS>Technical University of
MunichBoltzmannstr. 3MunichGermany85748

(2020)

###### Abstract.

While the exact definition and implementation of accountability depend on the
specific context, at its core accountability describes a mechanism that will
make decisions transparent and often provides means to sanction “bad”
decisions. As such, accountability is specifically relevant for Cyber-Physical
Systems, such as robots or drones, that embed themselves into a human society,
take decisions and might cause lasting harm. Without a notion of
accountability, such systems could behave with impunity and would not fit into
society. Despite its relevance, there is currently no agreement on its meaning
and, more importantly, no way to express accountability properties for these
systems. As a solution we propose to express the accountability properties of
systems using Structural Causal Models. They can be represented as human-
readable graphical models while also offering mathematical tools to analyze
and reason over them. Our central contribution is to show how Structural
Causal Models can be used to express and analyze the accountability properties
of systems and that this approach allows us to identify accountability
patterns. These accountability patterns can be catalogued and used to improve
systems and their architectures.

Accountability, Structural Causal Models, Cyber-Physical Systems

††copyright: acmcopyright††journalyear: 2020††doi: XXX††conference: ; ;
††price: 15.00††isbn: 978-1-4503-9999-9/18/06††ccs: Computing methodologies
Causal reasoning and diagnostics††ccs: Applied computing Evidence collection,
storage and analysis††ccs: Software and its engineering Design languages††ccs:
Computer systems organization Robotics

##  1\. Introduction

Accountability is a central pillar of human societies. Ever since John Locke
and Adam Smith, writers on political philosophy have tried to capture and
refine this slippery concept. Yet, details of definitions vary and there
exists no singular unified definition. Following (Lindberg, 2013), who
surveyed the social science literature on accountability, the central idea of
accountability is that _“when decision-making power is transferred from a
principal (e.g. the citizens) to an agent (e.g. government), there must be a
mechanism in place for holding the agent accountable for their decisions and
tools for sanction”._ Accountability works by ensuring that agents might
suffer consequences for their behavior and will thus avoid “bad” behavior
(Mitchell et al., 1998; Hall et al., 2017).

Causality, simply put, is about asking one question: Why? It is at the core of
human understanding and reasoning, but has only recently been given thorough
mathematical foundation in the from of _Structural Causal Models_ (SCMs)
(Pearl and Mackenzie, 2018). SCMs allow us to precisely describe and reason
over the causal structures of systems. They are uniquely suitable to capture
accountability properties of systems, because an agent can only be held
accountable for an effect that it caused; i.e., causality is a prerequisite
for accountability. Moreover, a recently published a survey of works in the
social sciences (Miller, 2018) shows that Halpern and Pearl’s model of
causality (Halpern, 2015) is a suitable formalization to capture the concept
of an “explanation”. Explanations are tightly linked to accountability and
often seen as the “account” part of “accountability”.

As examples and use cases, we use Cyber-Physical Systems (CPS). CPS are
systems in which software controls hardware to affect the physical world (Lee,
2008). Such systems usually have sensors to perceive the world, employ
actuators to influence their surroundings and use software to plan their next
action. CPS are open systems that are designed to work in unknown environments
and interact with any other system or object. This uncertainty makes it
impractical to specify all “legal” action a CPS can take and implies that
failures and unwanted events will happen. Similar to the transfer of decision-
making power from citizens to government, owners and operators of CPS transfer
some, and in some cases even all, their decision-making to the CPS. Hence, we
want similar _accountability mechanisms_ in place. Such accountability
mechanisms help to avoid “bad” or unwanted behavior by ensuring that its
causes can be identified and there could be consequences. Similar to humans,
the expectation of possible consequences will motivate CPS and their owners
and operators to comply with existing laws and mores. Thus, accountable CPS
will improve society’s trust in their behavior and increase their acceptance
in their respective social context.

In this paper, we focus on the problem of expressing accountability patterns
and the related problem of what data to log. Expressing accountability is
difficult, because there are many differing definitions of accountability,
there is no agreement an what constitutes an “accountable system”, and there
exists no common language to express accountability patterns. Our solution
uses the fact that accountability is build on the notion of causality and
leverages recent advances in the mathematical treatment of causality to
express accountability properties as SCMs. We use the structure of SCMs to
inform logging choices in the system design. Our contribution is (1) to show
how SCMs can be used to express accountability patterns, (2) that they can be
used to analyze and improve the system design, especially regarding the
logging choices, and (3) that this systematic treatment makes it possible to
identify accountability patterns and apply them to systems or system
architectures.

##  2\. What is Accountability?

Across the sciences, there is no consensus on how accountability should be
defined. Its definition is always dependent on the context of a system or
agent and the expectations of its principals. (Kacianka et al., 2017)
conducted a systematic mapping study in computer science and found that there
is no agreement on a definition and that most research papers use a
“dictionary definition” of accountability, or give no definition at all. While
prescribing what accountability is (e.g., (Baldoni et al., 2018) or (Küsters
et al., 2010)) works for specific contexts, in this paper we focus on how to
best express its meaning in general. In this section, we will give a brief
overview of accountability and related concepts such as causality,
responsibility and transparency.

Following (Kacianka and Pretschner, 2018), we see accountability as a relation
between a system (e.g., a CPS such as a drone or a robot; it might, in
principle, also be human), called an agent by (Lindberg, 2013), and a natural
or juridical person, called a principal, that the agent is accountable to. The
exact form of this relation depends of the definition of accountability used.
Systems can be part of a “stack” and one technical system might monitor and
correct another system. However, “things”, such as technical systems, without
legal standing cannot be principals. A principal serves as a link to society
and the legal system. Having a principal that cannot be sued and be held
liable for actions and decisions would sever that link and violate a core
requirement for accountability111There are some ideas of giving legal standing
to machines (“e-person”), however this idea comes with its own set of
problems. See for example http://www.robotics-openletter.eu/. This argument is
similar to the reasons why the executive in many countries cannot be replaced
with “liquid democracy” based voting systems. At some point a decision must be
justified and the decision maker be held accountable222 This point is
eloquently made by J.S. Mill (Mill, 1861): _“Responsibility is null when
nobody knows who is responsible; nor, even when real, can it be divided
without being weakened. To maintain it at its highest, there must be one
person who receives the whole praise of what is well done, the whole blame of
what is ill.”_ . This view is further strengthened by experiments confirming
that _“(…) people, not objects, are held accountable for a negative outcome”_
(Samland and Waldmann, 2016).

Accountability additionally requires the concept of actions that can or cannot
be taken by the agent and requires an account (or explanation) to be provided
to the principal. Actions are the linchpin that connects accountability to
causality. If an agents takes an action, it will cause effects. Not acting
might also cause effects. Principles judge these effects and might demand an
account from the agent. An agent that provides a meaningful account is
accountable, while an agent whose actions cannot be questioned or reviewed is
not accountable. An accountability mechanism is the means to hold the agent
accountable.

Causality and accountability are tightly interlinked and have some complex
interactions. While causality is necessary for accountability attributions,
people will often use the term “cause” for both a causal relationship and an
accountability relationship (Samland and Waldmann, 2016). The accountability
hypothesis states that _“(…) causal queries are generally ambiguous. They
might refer either to causal relations in the narrow sense, or they might
request assessment of moral accountability”_ (Samland and Waldmann, 2016). If
humans understand “cause” to mean “moral accountability”, they will associate
it, for example, with norm violations and consider the subject’s mental state.
Competing hypotheses suggest that norms might alter the causal model (i.e.,
what causes what) or that they might alter causal selection (i.e., what are
possible causes). To illustrate the point: (Samland and Waldmann, 2016)
conducted several experiments in which subjects were presented with different
vignettes and asked to attribute causality. In their experiments they, for
example, presented participants with a vignette in which a plant lover employs
two gardeners to look after his plants. In this scenario the gardeners could
apply different fertilizers on the plants; one would help the plants to
flourish and one would make them perish. The gardeners were explicitly told
which fertilizer to use, i.e., a norm was established. When the participants
were asked to assess the causal effect of the fertilizer on the plants, they,
with one exception, did not attribute the death of the plants to the
fertilizer, but to the norm-violating gardener. If the vignette was altered to
have one gardener intentionally use the wrong fertilizer he was picked more
frequently and if it was modified to say that the gardener accidentally picked
the wrong fertilizer, he was picked less frequently.

Studies like this show how tricky it is to define the meaning of causal
relationship and accountability. They also highlight how important it is to
spell out the causal relationships in a system, to express expected causal
relationships and to compare them to actually observed behavior. In Section
3.4, we show how SCMs are a tool to capture these relationships. However, we
want to caution that the actual modeling is highly subjective and context
sensitive and that reasoning over SCMs is always model relative.

Accountability is not limited to just answering questions, often called
responsiveness. The agent must be able to make a decision, because _“(a)
puppet acting as an extension of someone else’s will is not a legitimate
object of accountability”_(Lindberg, 2013) . Here it is also important to note
that there is a lively debate if machines are even capable of making decisions
and if they can have agency (see for example (Floridi and Sanders, 2004;
Sayes, 2014; Simon, 2015)). In our view, machines are objects, but can make
decisions. While they might not, like a human, suddenly decide to not do their
task, they have enough degrees of freedom and enough autonomy (Clough, 2002)
to make accountability necessary and meaningful. For an automatic and
deterministic system, accountability makes little sense, as any blame will lie
with the natural or legal person who operated or constructed the system. For
such a system, fault localization or debugging is enough. For accountability
to make sense, systems have to be autonomous agents that make decisions and
explore counterfactual scenarios without human intervention.

It is important to be aware that it is very difficult to compare
accountability definitions and define “levels of accountability”. The problem
is that definitions of accountability often disagree on properties and their
meaning. Even if we agree on properties, they are often nominal. For example,
we should be able to name an agent and a principal, but it is not easy to find
a way to compare two agents or find a relation between two agent-principal
pairs such that the excess of the agent in one pair makes up for the lack of
the principal in the other. For example, how should we decide if one person is
a good principal for a robot and compare them to another person’s way of being
a principal of an autonomous car? SCMs and accountability patterns based on
them offer a solution to systematically compare accountability notions.

Accountability will often be used in conjunction, and is sometimes confused,
with the term responsibility. Following (Singh et al., 2018a), accountability
is attributing responsibility in addition to the more traditional definition
of capturing the relation between principals and agents given by (Lindberg,
2013). Seen this way, a responsibility assignment is the output of an
accountability mechanism. Simultaneously, (Lindberg, 2013) points out that
responsibility is a necessary prerequisite for accountability. You cannot be
held accountable for an action or effect for which you are not responsible.
Here “responsible” is not seen as an output, but is seen very close to the
term “caused”. The difference being that you can also _accidentally cause_ an
effect. As “responsibility” requires conscious thoughts and actions, you are
not responsible for this effect and thus also not accountable. A common
example is to become unconscious while driving a car and causing an accident.
You caused the accident, but you are neither responsible nor will you be held
to account. Reviewing the literature in psychology, (Hall et al., 2017), find
that while “accountability” and “responsibility” are used interchangeably by
some researchers, they are usually distinguished with accountability imposing
the additional requirement of an external audience. In line with (Samland and
Waldmann, 2016), (Hall et al., 2017) also point to literature that sees
responsibility as a prerequisite for causality and causality as the core of
accountability. (Sytsma et al., 2012), on the other hand, hold the view that
causality is a normative concept and that _“(c)ausal attributions are
typically used to indicate something more akin to who is responsible for a
given outcome than who caused the outcome in the descriptive sense of the term
used by philosophers”_. (Floridi and Sanders, 2004) differentiate
responsibility and accountability along the line of intentionality. For them,
a non-human agent can be held accountable, but to be responsible for an action
would require them to have actual intentions.

In the recent works on algorithmic accountability (e.g., (Diakopoulos, 2015;
Singh et al., 2018b)), transparency is also often linked to accountability,
and sometimes even seen “as a type of accountability”. Here the idea is that
if the inner workings of machines and algorithms are open to the public, i.e.,
they are transparent, we can attribute responsibility for failures. One
problem, for example, is that, in contrast to decisions of a committee of
people, transparency for source code or machine learning models is not
necessarily useful, because they are often next to impossible to understand.
Another problem for us is that transparency does not make the principle
explicit. It roughly follows the definition of “felt accountability” (see
e.g., (Hall et al., 2017)) that agents are accountable if they think that
someone might check their actions. While this is enough in some instances, we
believe that, especially for CPS, principals should be known. (Ananny and
Crawford, 2018) give ten further limitation of transparency as a tool for
accountability and suggest these limitations might serve as requirements for
algorithmic accountability systems. We believe that causal models offer a
useful way to fulfill those requirements and ensure the accountability of a
system. Causal models are especially good at facilitating explanations, which
are often much more useful than a fully transparent system.

To summarize, causality is a relation between objects. Responsibility, for us,
requires some intention on part of the causing agent. Accountability is a
societal framework that involves autonomous agents and third parties.
Transparency, while close to accountability, lacks the concept of a principal.
In the rest of this paper, we will show how SCMs can be used to express
accountability properties (Section 3), to identify patterns (Section 3.4) and
to improve the system design (Section 4).

##  3\. Expressing Accountability

To illustrate how difficult it is to express the accountability properties and
find common features, we will use two entirely different examples. The first
is the story of Titus Manlius, as told by Livy (Book 8, chapter 7; translation
(Foster, 1926) and a summary (Pauw, 1991)). In 349 BC, Titus Manlius Torquatus
led a campaign against the Latins. To increase the military discipline, he
ordered his troops to strictly obey orders, not to act on their own and not to
leave their post. His son (and namesake) Titus Manlius, while on orders to
patrol, engaged an enemy commander who insulted, ridiculed and challenged him
to a duel. After defeating the enemy heroically in the duel, he rode back to
his camp and proudly told his father of his exploits. Titus Manlius summoned
all his men and rebuked his son for abandoning his post and breaking military
discipline, summarizing the dilemma as _“(…) ut aut rei publicae mihi aut mei
meorum obliviscendum sit (…)”_ (roughly: “(…) that I must either forget the
republic, or myself and mine (…)”). With this he had his son bound to a stake,
beheaded him with an axe and had the body burned with full military honors.

What does this bloody tale from ancient Rome now have to do with our modern
understanding of accountability? First of all, we have all major components
present. Titus Manlius, the father, is the _principal_ who ensures a norm is
not violated. His son is the _agent_ , that (literally) gives an _account_ of
his actions. The execution of the son is then the ultimate _sanction_ for
violating the given norm. The _accountability mechanism_ is a bit harder to
identify. Its goal is to keep up the norm, ensuring military discipline, and
it is embedded in the military as its reporting framework.

As a second example, we look at the fatal 2018 AD crash of an autonomously
driving Uber car (NTSB, 2018). In this accident, the car, equipped with
cameras, radars, a LIDAR and other sensors, was driving autonomously and hit a
pedestrian crossing the road. Crucially, the Uber developers disabled the
car’s collision avoidance functionality, because it interfered with the
computer control. While the car had a safety driver, responsible to intervene
in case of system failures, the safety driver was focused on monitoring the
system and did not notice the pedestrian.

Similar to the story of Titus Manlius above, we can identify components that
can be used for accountability. The car is the _agent_ , its video feeds and
log data are the _account_ and should provide an explanation. The immediate
_sanction_ taken by Uber was to stop their fleet of autonomous cars. Where it
gets tricky is the question of the _principal_. Depending on how we want to
model the problem, the principal could be Uber, the safety driver, a
regulatory agency, a number of other entities, or multiple such entities at
once. Since the principal sets the norm, the norm depends heavily on who to
consider as principal. A regulatory body’s goal might be to ensure road
safety, Uber might wish to avoid public relation disasters and so forth.
Without a clear way to express these intricacies, it is needlessly difficult
to identify the actual and intended principal. Because a principal needs to
know of its role to conduct any oversight, the knowledge of being a principal
is a requirement for accountability and the related logging choices.
Expressing this fact clearly allows regulatory bodies to spell out who they
expect to look at accounts and also gives manufactures, owner and operators
the ability to show that they comply with the accountability requirements and
have the necessary logging infrastructure.

###  3.1. Structural Causal Models

As both the story of Titus Manlius and the story of the Uber car have similar
features, such as a principal. We now want to look at definitions of
accountability and identify patterns that allow us to match them to these
features. This allows us to determine if a given pattern of accountability is
suitable for a given system or if another pattern might be a better fit. We
propose to use _Structural Causal Models_ (SCMs) (Pearl et al., 2016)[p. 26f.]
to capture the relevant accountability features of systems and model the world
they interact with.

###### Definition 0.

Structural Causal Model (Pearl et al., 2016)
A structural causal model $M$ is a tuple $M=\mathcal{(U,V,F)}$, where

  * •

$\mathcal{U}$ is a set of _exogenous_ variables,

  * •

$\mathcal{V}$ is a set of _endogenous_ variables,

  * •

$\mathcal{F}$ associates with each variable $X\in\mathcal{V}$ a function that
determines the value of $X$ given the values of all other variables.

Every SCM is associated with a _graphical causal model_ called the “graphical
model” or the “graph”. While the graph does not include the details of
$\mathcal{F}$, its structure alone is enough to identify patterns and causes.
In an SCM, exogenous variables are external to the model, meaning that we
chose not to explain how they are caused. They are the root nodes of the
_causal graph_ and are not descendant of any other variable. Endogenous
variables, on the other hand, are descendants of at least one exogenous
variable and model components of our system and world for which we want to
explain causes. $\mathcal{F}$ describes the relationships between all those
variables. If we knew the value of every exogenous variable, we could use
$\mathcal{F}$ to determine the value of every endogenous variable. In a
graphical model every node represents an _endogenous_ variable and arrows
represent functions from $\mathcal{F}$ between those variables. SCM are
derived from structural equation models (SEMs) (e.g., (Lomax and Schumacker,
2004)), but in contrast to them, their relation have a direction.

If we want to investigate the cause of Titus Manlius son’s violation of the
norm, we might model the behavior of the enemy commander as an exogenous
variable, which we accept as given and external to the model. His son’s
reaction to those insults are _caused by_ those insults and would thus be
modeled as an endogenous variable. Figure 1 depicts the graphical model for
the story of Titus Manlius. In the SCM, $\mathcal{U}$ would contain _Insults
(I)_ , meaning that our model does not investigate why the enemy commander
chose to insult Titus Manlius’ son. $\mathcal{V}$ would contain the fact
_Titus Manlius’ son reacted (TM)_ , _Engaged in a Duel (ED)_ and _Broke
Discipline (BD)_. For these endogenous variables, we can perform causal
reasoning and give explanations. Note that $\mathcal{U}$ and $\mathcal{V}$
should contain facts that can be observed and measured. If we cannot measure
some influence, but suspect it is there, we can model it as a background
variable.

In our example, $I$ is given by some exogenous input, meaning that our model
cannot explain it any further. For $TM$ the equation would be
$TM=\mathcal{F}_{TM}(I)$. Often this function will include an error term to
account for unknown background factors, called $U$ that might affect a
variable, giving us $TM=\mathcal{F}_{TM}(I,U_{TM})$; accordingly,
$ED=\mathcal{F}_{ED}(TM,U_{ED})$ and $BD=\mathcal{F}_{BD}(ED,U_{BD})$. For
simplicity, we model the functions in $\mathcal{F}$ so that they just transmit
the value and take on either the value “true” or “false”. In practice, the
functions in $\mathcal{F}$ can be any mathematical function and will often be
probability distributions.

###  3.2. Causal Reasoning

Figure 1 can now be used to answer causal questions (Pearl and Mackenzie,
2018)[p. 40]. The first level of questions are questions of association, i.e.,
how one fact relates to another. For example, if we know that the discipline
was broken, did Titus Manlius’ son engage in a duel? To answer this question,
we can inspect the graph and come up with the answer. This task can of course
also be automated (Tikka and Karvanen, 2018) and the graphs can be a lot more
complex.

Figure 1. The story of Titus Manlius as graphical causal model.

Next, we might want to ask if the discipline is broken if anyone engaged in a
duel without being provoked. While questions of association can be answered
with standard logic, here we ask a hypothetical question about something we
have no data on. To analyze this, we first remove all arrows leading into the
node of interest and, second, set this variable to the desired value. Then we
can analyze the resulting model with conventional logic. The reason to do this
is that we want to analyze what would happen if we made a specific event
happen. Removing all arrows leading into the variable makes it independent
from all other influences. Forcing it to the desired value then ensures that
we analyze the world in which the event happens. Figure 2 depicts this model
after the intervention. We disconnect $ED$ from all influences and then set it
to true. Analyzing the model, we can then see that this would also break the
discipline ($BD$). However, our model does not contain any information on who
might have engaged in the duel. If we wanted to answer this question, we would
need a new model containing additional variables.

Figure 2. Simulating an intervention by fixing the value of an variable.

Finally, we can also use causal reasoning to ask counterfactual (“what if”)
questions. If we know that the discipline was broken, what would have happened
if Titus Manlius’ son had not reacted to the insults? Would the discipline
still have been broken? Figure 3 depicts this model of the counterfactual. We
remove the link from $I$ to $TM$ and force its value to “false”, i.e. to not
be provoked. The variable $I$ keeps its original value. The question then is,
what would the value of $ED$ and $BD$ be? We could again use automatic
reasoning engines to deduce these values.

Figure 3. Counterfactual reasoning.

Computing interventions and counterfactuals is what sets causal reasoning
apart from machine learning and other statistical approaches. If we were to
use machine learning, we would not have an explicit model of the world333Of
course, creating a causal model in the first place is a challenge. (Kacianka
et al., 2019; Ibrahim et al., 2019a) provide means to derive models for CPS
from existing system models. . We would only gather one data point where the
insult lead to a break of discipline and a myriad of data points where there
was no insult and also no break in discipline. To understand Titus Manlius’
causal contribution to the effect, we need a model of the interactions and the
ability to analyze it. SCMs give us a way to express these models and then use
well defined mathematical operations to ask questions of association, analyze
interventions and compute counterfactuals.

###  3.3. Structures in SCMs

To give a quick overview444For detailed introductions, see (Pearl and
Mackenzie, 2018) or (Pearl and Bareinboim, 2014). , a causal graph structure
such as a chain, $A\rightarrow B\rightarrow C$, means that $B$ is only
influenced by $A$, $C$ is only influenced by $B$ and $A$ is determined by
external forces not part of this model (Pearl and Mackenzie, 2018)[p. 129]. If
we inverted the chain, $A\leftarrow B\leftarrow C$, the causal meaning would
change, but the independence of the variables would remain the same, i.e., $A$
is still independent of $C$, provided we know $B$. Put another way, if there
were an arrow from $A$ to $C$, then changing $A$ would change $C$. However, if
there is no direct arrow and we keep the parents of $C$, i.e., $B$, constant,
no matter how we change $A$, $C$ will not be affected. A major implication of
this is that we can use data to test causal models. If we have data that show
that $C$ changes when we change $A$, despite $B$ not changing, then we need to
revise our model. (Pearl and Mackenzie, 2018)[p.157] provides four rules that
apply to any causal graph:

  1. (1)

In a chain junction, $A\rightarrow B\rightarrow C$, controlling for $B$
prevents information about $A$ from getting to $C$ or vice versa.

  2. (2)

Likewise, in a fork or confounding junction, $A\leftarrow B\rightarrow C$,
controlling for $B$ prevents information about $A$ from getting to $C$ and
vice versa.

  3. (3)

In a collider, $A\rightarrow B\leftarrow C$, exactly the opposite rules hold.
The variables $A$ and $C$ start out independent, so that information about $A$
tells you nothing about $C$. But if you control for $B$, then information
starts flowing through the “pipe”, due to the explain-away effect.

  4. (4)

Controlling for descendants (or proxies) of a variable is like “partially”
controlling for the variable itself. Controlling for a descendant of a
mediator partly closes the pipe; controlling for a descendant of a collider
partly opens the pipe.

In causal graphs, we often want to know if two nodes are independent, i.e.,
changes in one node will not affect another node. Two variables are
independent if every path between them is _blocked_. Formally:

###### Definition 0 (d-seperation).

(Pearl et al., 2016)[p. 46]

A path $p$ is blocked by a set of nodes $Z$ if and only if

  1. (1)

$p$ contains a chain of nodes $A\rightarrow B\rightarrow C$, or a fork
$A\leftarrow B\rightarrow C$, such that the middle node $B$ is in $Z$ (i.e.,
$B$ is conditioned on), or

  2. (2)

$p$ contains a collider $A\rightarrow B\leftarrow C$ such that the collosion
node $B$ is not in $Z$, and no descendant of $B$ is in $Z$.

If $Z$ blocks every path between two nodes $X$ and $Y$, then $X$ and $Y$ are
d-separated, conditional on $Z$, and thus are independent conditional on $Z$.

From do-calculus, we can also derive two graphical criteria. The front-door
and the back-door criterion allow us to analyze if a causal effect between two
variables is identifiable or not. Both of them can be checked automatically
(Tikka and Karvanen, 2018).

###### Definition 0 (The Back-Door Criterion).

(Pearl et al., 2016)[p. 61] Given an ordered pair of variables $(X,Y)$ in a
directed acyclic graph $G$, a set of variables $Z$ satisfies the back-door
criterion relative to $(X,Y)$ if no node in $Z$ is a descendant of $X$, and
$Z$ blocks every path between $X$ and $Y$ that contains an arrow into $X$.

Intuitively, the back-door criterion ensures that (1) all spurious path
between $X$ and $Y$ are blocked, (2) all directed path from $X$ to $Y$ are not
perturbed, and (3) no new spurious paths are added.

Building on the back-door criterion, the front-door criterion can applied in
structures not suitable for the back-door criterion.

###### Definition 0 (The Front-Door Criterion).

(Pearl et al., 2016)[p. 69]

A set of variables $Z$ is said to satisfy the front-door criterion relative to
an ordered pair of variables $(X,Y)$ if

  1. (1)

$Z$ intercepts all directed paths from $X$ to $Y$.

  2. (2)

There is no unblocked path from $X$ to $Z$.

  3. (3)

All back-door paths from $Z$ to $Y$ are blocked by $X$.

Intuitively, the front-door criterion relies on the fact that one can identify
the effect of $X$ on $Z$ and the effect of $Z$ on $Y$ separately. Having
identified the separate effect, we can then calculate the effect from $X$ on
$Y$.

###  3.4. Accountability Patterns

Accountability becomes necessary when a principal transfers power (for
machines often: delegates tasks) to an agent and that agent then causes some
effect. The principal then has the right to understand how the agent used this
power and why it made a certain decision. We can now use SCMs to express
definitions of accountability and identify the underlying pattern. The
definitions given in this section are abstractions and can usually not be used
in real systems directly. However, as they are condensed to their core, they
are not obscured by implementation details and can be used as patterns for
real systems. If we analyze the SCM of an actual system, we can look for these
patterns, and if we identify one, we can then use the knowledge about the
pattern to improve the actual system at hand. Similarly to a design pattern in
software engineering, an accountability pattern is a reusable solution to a
common problem, but must be implemented anew in each system.

Accountability patterns show us which nodes in a graph are actually relevant
to ensure a given notion of accountability. Using SCMs, we can then analyze
how the variables are connected and interact, and decide which variables need
to be logged. Without this clear understanding, we always run the risk of
logging too little data, or suffer from the burden of logging too much data,
most of it irrelevant. For example, if a path between two nodes is blocked,
they are independent and cannot influence each other. If one of those nodes is
irrelevant for accountability, we do not need to spend resources to log it.
Looking at the Uber example, the color of the Uber car was, presumably,
irrelevant for the accident and all accountability questions associated with
it. So logging this data would not help us clearing up any questions of
accountability.

In this paper, we use the definitions for Lindberg- and RACI- accountability,
taken from (Kacianka and Pretschner, 2018). As causality is at the core of
accountability, we are convinced that any other definition of accountability
could be similarly expressed as an SCM. Our two examples describe specific and
very different notions of accountability. We show that they can be expressed
as a SCM, however, we currently do not have a way to verify that the
translation from the theory to the model is correct. On the contrary, since
accountability is a human and social concept, the correctness of a model will
always depend on a specific context. The big advantage of causal models is
that our assumptions about the causal structure are explicit in the model and
can be discussed and improved upon by others. We consider SCMs ideally suited
to express notions of accountability and foster a discussion about their
differences.

####  3.4.1. Lindberg’s Pattern

Lindberg (Lindberg, 2013) surveyed the social science literature and provided
the following synthesized definition of accountability:

  1. (1)

An agent or institution who is to give an account (A for agent);

  2. (2)

An area, responsibilities, or domain subject to accountability (D for domain);

  3. (3)

An agent or institution to whom A is to give account (P for principal);

  4. (4)

The right of P to require A to inform and explain/justify decisions with
regard to D; and

  5. (5)

The right of P to sanction A if A fails to inform and/or explain/justify
decisions with regard to D.

Lindberg’s definition assumes that an agent in an organization will cause some
effect, usually indirectly through mediation555 A mediator is a variable
between a cause and an effect. It might amplify or modify the initial input.
(Baron and Kenny, 1986) define mediation as a _“generative mechanism through
which (…) [an] independent variable is able to influence the dependent
variable of interest”_.  (see Figure 4). If our system, or a part of our
system, has such a structure, regardless of the actual functions, we can apply
Lindberg’s definitions to set up a framework that allows some principal to get
more information from the agent, i.e., to improve the cause model. The
principal is not part of the model, but will use the output of a system
following that model to sanction an agent. In the same vein, term _felt
accountability_ (Hall et al., 2017) is used in psychology to mean that actors
think there is a possibility that their actions will be evaluated by a third
party.

Figure 4. The causal model for the Lindberg accountability pattern; the
principal is not part of the pattern.

####  3.4.2. RACI Pattern

In contrast to social sciences, organizational sciences often apply a
_Responsible-Accountable-Consult-Inform (RACI)_ framework (Smith et al., 2005)
to visualize the roles of people in an organization. The elements of the
framework are described as follows (adapted from (Smith et al., 2005)):

  * •

Responsible: The individual who completes a task. Responsibility can be
shared.

  * •

Accountable: The person who answers for an action or decision. There can be
only one such person.

  * •

Consult: Persons who are consulted prior to a decision. Communication must be
bidirectional.

  * •

Inform: Persons who are informed after a decision or action is taken. This is
unidirectional communication.

Figure 5 shows the accountability pattern based on the RACI framework. In
contrast to Lindberg above, RACI includes variables that provide additional
information. So, instead of just needing data on the cause, the mediator, and
the effect, we additionally need information on the _accountable agent_ , any
_consulted agent(s)_ , the content of the consultation, called _discussion_
here, and the _informed agent(s)_. While this pattern has much higher logging
and reporting requirements, it also allows us to model the delegation of
decisions and common workflows in organizations, like meetings, that cannot be
captured in the Lindberg model.

Figure 5. The RACI accountability pattern.

##  4\. Using Accountability Patterns

We now show a possible causal model for the story of Titus Manlius and the
Uber example. If we find that they comply with a know accountability pattern,
we can be certain that the causal effect of the agent can be identified and
know which events we need to log. Figure 6 depicts the familiar model of Titus
Manlius’ story. Here we highlight the structure of the Lindberg accountability
pattern in grey. A mapping to the RACI pattern would be far less convincing,
mainly because Titus Manlius’ son did not consult anyone or delegated the task
and we could not map the nodes required by the RACI pattern. He was not an
organization, which is why a model for organizational accountability is no
natural fit. Knowing that in the Linderberg pattern, we need to look at the
behavior of the agent, we can, as shown in the examples on causal reasoning in
Section 3.2, ask the counterfactual question “What would have happened if
Titus Manlius’ son had not engaged in a duel?”. As in this case the military
discipline would not have been broken, holding Titus Manlius’s son to account
is the correct action.

Figure 6. The story of Titus Manlius as causal model, with Lindberg
accountability pattern highlighted in grey.

Figure 7 models the Uber accident. In it, we depict that Uber hired and
trained a safety driver for its cars. The safety driver is aided by operating
manuals and procedures set by Uber’s experts. Only with these prerequisites
the driver is allowed to let the car drive autonomously. Still, this led to an
accident that was reported to the police. In contrast to the story of Titus
Manlius, this model fits very nicely to the RACI patten. With Uber being an
organization, this is of course very natural.

Figure 7. A causal model for the Uber accident with the RACI pattern
highlighted in grey.

Similar to above, after an accident we can ask different counterfactual
questions. To find the responsible agent, we could ask “Would the accident
have happened if the driver had not started the car?”, and, looking at the
model, we can see that the driver is on one direct path to the accident, so
changing _Driver_ would affect _Accident_. However, there is also a path from
_Developers_ via _Emergency Braking disabled_ to _Accident_. In such a case,
we say that the effects of the two nodes are _confounded_ , i.e., without
additional data it is impossible to say what influence each node has. The
accountable entity, on the other hand, is unambiguous: If we change the value
of _Uber_ , all child nodes would change, clearly preventing the accident. If
we chose the Linderberg accountability pattern for this graph, the nodes
_Driver_ would be the agent, _Car Software_ the mediator and _Accident_ the
effect. In this case, we could not make an unambiguous attribution of
accountability and would need to either change the accountability pattern or
change the system to fit the desired pattern.

Of course, many other models are possible for these examples and a plethora of
competing definitions of accountability can be applied. However, our goal is
not to find the “one true model”; our goal here is to show that causal models
are a useful tool to express and identify accountability pattern and analyze
if a systems fulfils them. To stick with our examples, a regulatory body could
publish the RACI pattern and then check if Uber confirmed to it. If, on the
other hand, Uber said they used the Lindberg pattern, this could be criticized
and another pattern be proposed. Without SCMs to express these fine points, a
discussion is next to impossible, and, most importantly, will lose the
technical details. This example also illustrates that SCMs without
$\mathcal{F}$ can answer some queries, but they will often not be able to give
clear answers. Defining $\mathcal{F}$ is one of the major open challenges when
working with SCMs.

###  4.1. Choosing an Accountability Pattern

In the previous section, we showed that the Lindberg pattern would not be a
good fit for the Uber example. In this section, we will look at the SCM for a
part of a system and will show how we can use the strucures in the SCM
(Section 3.3) to check if an accountability pattern fits the model. As in the
Uber example, for a pattern to be applicable, it is important that its causal
effect is identifiable and not confounded. As an example we use another CPS
and discuss the development of an accountability mechanism for an Unmanned
Aerial Vehicle (UAV). The goal is to find out who is accountable if the UAV
crashes in bad weather for which the pilot does not have the necessary flight
permissions.

Figure 8666This model structure is similar to the second structure given by
(Pearl and Mackenzie, 2018)[p.159].  depicts one possible causal model for our
UAV; the Lindberg pattern is highlighted in grey. It captures four “pre-
flight” variables, _weather_ , the legal _visibility limit_ , the type of
_permission_ obtained (e.g., allowed to fly in bad weather) and resulting from
those, _permitted to fly_ , that signals whether a flight was legally allowed
or not. Additionally, it models the pilot’s decision, the fact that the UAV
took off, that it is in flight and whether it crashed.

Figure 8. Causal Model with the accountability model highlighted in grey.

Under the assumption that this model reflects reality correctly, we can now
use the back-door criterion to decide what data to log and what data to use
for analysis. We want to make sure that the effect of the pilot on the crash
is identifiable and not confounded by the other factors. To ensure that, we
need to make sure that every noncausal path is blocked, while not perturbing
any causal path (Pearl and Mackenzie, 2018)[p. 158]. A back-door path between
two nodes $X$ and $Y$ is “ _any path from $X$ to $Y$ that starts with an arrow
pointing into $X$”_ (Pearl and Mackenzie, 2018)[p. 158]. The effect of $X$ on
$Y$ will be de-confounded if every back-door path is blocked. In Figure 8 we
have one such path, _Pilot_ $\leftarrow$ _Weather_ $\rightarrow$ _Visibility
Limit_ $\leftarrow$ _Permission_ $\rightarrow$ _Take-off_ $\rightarrow$ _UAV
in flight_ $\rightarrow$ _UAV crash_. Looking at rule 3 above, we can see that
the _Visibility Limit_ is a collider, meaning we do not need to control for
(log) anything except _Pilot_ , _Take-off_ , _UAV in flight_ and _UAV crash_.
This finding is in line with our intuition: while the weather might have an
influence on the pilot, it also cannot make the UAV take off and crash. The
permission status might influence the take-off decision, but in itself it
cannot prevent a UAV from taking off. As child nodes of _Weather_ and
_Permission_ , _Visibility Limit_ and the resulting fact that a flight is
permitted or not have even less influence on the observed behavior of the UAV.
Note that other cause for a crash, such as an engine failure, are impossible
in this model. To consider them, we would need to extend the model (Ibrahim et
al., 2019a).

If we do not want to model hobby pilots who can start their UAVs, possible
disregarding laws and regulations, where- and whenever they want, and want to
capture a more formal military setup, we might design our overall system to
follow the RACI accountability pattern. Figure 9 depicts such a setup. In such
a RACI setting, pilots have less freedom of choice. They may only start the
UAV if they receive an order to do so. This order is given by a _commander_ ,
after consulting with _meteorologists_ who observe the _weather_ and prepare a
_weather forecast_. If the UAV is still lost due to bad weather, the pilot
will still be responsible, but accountability will, following the RACI
definition, lie with the commander giving the take-off order. As in the
previous example, we have a collider in _weather forecast_ , that ensures that
the _commander_ and the _meteorologist_ are independent.

Figure 9. Using RACI to model the bad weather example.

###  4.2. Accountability Patterns and System Design

Here, we will show how accountability patterns can help us when designing a
system. As an example, suppose we are tasked with attributing accountability
in the case where a UAV is controlled by a pilot, but both the pilot and the
UAV might be influenced by some attacker. To complicate matters, the attacker
is an unobservable confounding variable. That means that it will affect two
observed variables, but can itself not be observed. As such it might confound
the effect of the pilot on the UAV, which is a problem for the Linderberg
accountability pattern. Figure 10 depicts the corresponding causal model. In
contrast to the previous examples, there is the open path, _Pilot_
$\leftarrow$ _Attacker_ $\rightarrow$ _UAV_. Looking at rule 2, we could close
that path by controlling for the _Attacker_. One option to do this would be to
install an intrusion detection system (IDS) and make the _IDS_ a proxy for the
_Attacker_. In an ideal world, a _proxy variable_ will behave like the real
variable, but will in reality often induce some uncertainty (e.g, because it
might miss some attacks).

Figure 10. The unobservable attacker will confound the effect of the pilot on
the UAV.

Figure 11 depicts this causal model; boxes in grey highlight the Linderberg
accountability pattern and the dashed box marks unobservable variables. If we
know what the attacker is doing, we can disentangle their action from the
pilot’s actions and close the back-door path. The downside is that we actually
have to invest time and money into such an IDS and need to be reasonably
certain that it will log all relevant information about the pilot and the
attacker.

Figure 11. Using an IDS to control for the attacker.

However, if we can ensure that the RC cannot be affected by the attacker, we
can use the front-door criterion777See (Thoemmes, 2018) for a detailed example
with linear models. to estimate the causal effect of the pilot on the UAV,
even without installing an IDS. The intuition behind this solution is that,
because there is no back-door path from _Pilot_ to _RC_ , we can estimate the
effect of _Pilot_ on _RC_. Similarly, we can estimate the effect of _RC_ on
_UAV_ and calculate the effect of _Pilot_ on _UAV_. With such a causal
structure, we can do away with an IDS while still being able to ensure
accountability. The downside is that, beside the need to be sure that the
causal structure reflects the real world, the attacker must not be able to
influence the RC and the pilot must have no other way of affected the UAV than
via the RC.

Beside the back-door and front-door criterion, do-calculus offers us many
different options of analyzing and manipulating SCMs. If we express our system
as an SCM, similar to the examples given above, we can use do-calculus and
tools build for it (Tikka and Karvanen, 2018) to either adapt the system to
the necessary structure or even find structures that make our system simpler.

##  5\. Discussion

When looking at any of the figures in the paper, it is easy to suggest
additional variables, different connections, or new structures. Some readers
might look at the figures and argue that the model might be unrealistic or
even wrong. We do not claim that our examples are perfect or “the one true
model”. On the contrary, we want to emphasize how easy these models are to
inspect, understand and criticize. The main advantage of using SCMs to express
accountability properties of systems is that they make definitions of
accountability easy to communicate and foster discussions. If we compare
Figure 7 to the story of the Uber accident in Section 3, we can see how a long
and involved paragraph of text can be expressed as an easy-to-discuss diagram.
The exact details, such as how Uber influences the driver, can be omitted in
the graphical model and expressed in the underlying SCM. For most use cases,
first determining that there is a causal influence from one node on another is
enough. Especially the knowledge that some nodes do _not_ affect the outcome
is useful to establish accountability relationships between principals and
agents. The core advantage of using SCMs to express accountability properties,
however, is that they enable us to identify and communicate accountability
patterns.

Expressing accountability in SCMs is not without its problems. One obvious
problem is the creation of the models in the first place. There exists some
work on using other models of systems such as fault trees or Timed Failure
Propagation Graphs (Ibrahim et al., 2019a), as well as models of human
interaction with systems (Kacianka et al., 2019), to generate the SCMs;
however, so far these efforts are semi-automatic and will require human input.
Looking at the problems that accountable algorithms have with fairness and
bias, we think that this manual input will not go away anytime soon. Without
fully automatic solutions, we are convinced that SCMs offer the best
compromise between being understandable for humans, while offering a rigid
formal framework to analyze and reason over them. The other major problem is
defining the functions $\mathcal{F}$. While they can take on any form, in many
examples binary functions are used. The main reason for this is that such
models are computationally easier to analyze and also straight-forward to
explain. As in the model about Titus Manlius, binary models can be interpreted
as “caused” or “do not cause”. If we now look at the Uber example in Figure 7,
what could the function between Uber and the driver look like? A binary
function might just express that there is an influence. This is good to know,
but not necessarily enough. We might use a probability and draw from
observations that, for example888All these examples are made up for
illustrative purposes and are not true., 95% of the drivers were instructed by
Uber to do long hours and were thus not attentive. But how does this help us,
if we actually want to express the more subtle point that Uber was rewarding
miles driven and drivers were thus motivated to keep driving when they were
exhausted? When considering human interactions, it is not hard to come up with
examples that are hard to capture in neat closed-form formulas. Here the big
advantage of SCMs is that much analysis can be done on models that do not
specify $\mathcal{F}$ explicitly. That is why we look at the patterns of the
graph and ignore the actual functions. If we ensure at the design time of a
system its the causal graph has a structure in which the causal effects
relevant for accountability are identifiable, we will usually be able to find
evidence to do so in a ex-post analysis. In other words, if we at least know
that Uber has an effect on the driver, and build our system to log their
interaction, we will be able to find the exact effect in a post mortem
analysis where we have data on the system, as well as external observations
available.

##  6\. Related Work

Beside the works on accountability that we detail in Section 2, accountability
in computer science was popularized by by Weitzner et al. (Weitzner et al.,
2008), who proposed “Information Accountability” as a step beyond preventive
data control measures. They argue that it is impossible to keep data,
especially personal data, secret. Instead of trying to prevent data leaks,
they suggest to build accountable systems that make it easy to trace data
leaks and use the legal system to punish misbehavior. This work was continued
by Feigenbaum et al. (Feigenbaum et al., 2011; Feigenbaum et al., 2012) with a
focus on security. Their definition of accountability is, for example, close
to the Linderberg pattern. Feigenbaum et al. are, to the best of our
knowledge, the first to make a connection with causality, by pointing to the
Halpern and Pearl (HP) definition of causality (Halpern and Pearl, 2005a, b).
The HP definition (Halpern, 2015) of causality is closely related to the
definition of causality we use here. However, while we here use a _type
causal_ definition, HP is used to define _actual causality_. Type causality is
forward-looking and will often be used by scientific models. A classic example
of type causality is a statement like “ _Cigarettes cause lung cancer_ ”
(Halpern, 2016). Actual causality, on the other hand, is used to identify
specific courses of events in the past. A classic example is the statement “
_the fact that David smoked like a chimney for 30 years caused him to get
cancer last year_ ” (Halpern, 2016). To build systems, we need forward-
looking, type causal models. Fortunately a good type causality model will also
be a good actual causality model and make it easy to analyze events ex-post.
For example, (Halpern and Kleiman-Weiner, 2018) build on actual causality to
give definitions of blameworthiness and moral responsibility. An interesting
property of type causal models is transportability (Pearl and Bareinboim,
2014). It describes under what assumptions we can transfer causal knowledge
gained in experiments to new contexts where only observations are possible.

There are various implementations of accountability mechanisms in the computer
science literature. Kacianka et al. (Kacianka et al., 2017) conducted a
systematic mapping study on implementation, but found that many papers use no
explicit definition of accountability. For the cloud domain, the A4Cloud
project (Felici and Pearson, 2014) published an extensive report on
accountability for cloud providers. From other domains, Küsters et al.
(Küsters et al., 2010) give a mathematical definition of accountability in the
context of cryptography for e-voting systems, Datta et al. (Datta et al.,
2016) suggest to base accountability in CPS on causal information flow
analysis and Baldoni et al. (Baldoni et al., 2018) published an information
model of accountability as an Object-Role Model. As in computer science
finding the causes of problems and blaming the responsible party is an old
problem, many techniques that are not specifically called accountability, yet
solve similar problems, were developed. For example, logging is used to
understand system behavior (Amir-Mohammadian et al., 2016), runtime
verification is used to ensure correct system executing (Leucker and
Schallhart, 2009), fault trees (Vesely et al., 1981) are a technique from the
safety domain to express ways in which a system may fail, fault localization
(Renieres and Reiss, 2003) describes techniques to find faults in source code
and Forensics-by-Design (Rahman et al., 2016) is a technique from the security
domain to build systems so that, if unexpected attacks happen, they can still
be detected. We are positive that all approaches that, in essence, ask
questions of accountability, can be express as an SCM and then analyzed for
patterns.

A fairly new branch of accountability in computer science is algorithmic
accountability (Diakopoulos, 2015). Here, the goal is to understand the
behavior of, often, machine-learning algorithms. Ananny and Crawford (Ananny
and Crawford, 2018) have shown that mere transparency is not sufficient for
accountability. Miller (Miller, 2018) gives a thorough overview of the concept
of an explanation and suggests that actual causality is useful tool to create
and give explanation. Doshi-Velez et al. (Doshi-Velez et al., 2017) point to
the importance of explanations of AI systems in the legal context and
highlight that explanations need to be causal. Mittelstadt et al. (Mittelstadt
et al., 2016) elaborate on the ethical implications of AI systems and also
point to the relevance of causality and causal knowledge. Wachter et al.
(Wachter et al., 2017) also suggest using counterfactuals to explain the
behavior of black box AI systems. Mittelstadt et al. (Mittelstadt et al.,
2019) recently struck a more careful note and suggest that we need to be
careful with models used to explain AI systems. They especially highlight the
utility of contrastive explanations for human understanding. As SCM make it
easy to explore counterfactual alternatives, we expect that our approach of
expressing accountability with SCMs can be adapted to the problems of
algorithmic accountability and explainable AI. Combining these models with
efficient actual causality reasoning (Ibrahim et al., 2019b), work on
abstracting causal models (Beckers and Halpern, 2019) and efforts to
automatically derive SCMs for CPS (Ibrahim et al., 2019a), will allow us to
identify root causes, give counterfactual explanations, exchange models
between different domains of computer science and, most importantly, other
fields of science.

##  7\. Conclusions

CPS will continue to be deployed in shared spaces with humans. This will lead
to questions of accountability of those systems for their actions and for
those systems by their manufactures, owners and operators. In the first part
of this paper, we looked at the concept of accountability from different
disciplines and concluded that causality is at the core of any accountability
definition. Following this, we turned to current research on causality, and
especially SCMs, and showed with examples that they are a useful tool to
express accountability definitions and that SCMs allow us to identify
accountability patterns. Using methods to check if a cause is identifiable, we
explored how different system designs can be adapted to fit specific patterns.
Using an accountability pattern will ensure that the relevant causal effects
in our systems are identifiable. We can then use this knowledge to decide what
values to log and which events are safe to ignore.

While we have shown that SCMs are useful to communicate accountability
definitions in a system’s design, actually developing these SCMs is still an
open problem. Additionally, reasoning about _actual causes_ in such models
suffers from a high computational complexity and can currently only be done
automatically for binary models. In that regard, SCMs have the advantage that
they are already used in fields such as statistics, data science, or
epidemiology and that we can use theoretical results and, especially, tools
from these field to analyze accountability models. We are convinced that the
adaptation of SCMs to express accountability definitions will improve the
communication with fields such as philosophy, sociology or psychology that
have a long history of analyzing and understanding the concept and that these
discussions will lead to the emergence of new accountability patterns,
insights and best practices.

###### Acknowledgements.

This work was supported by the Deutsche Forschungsgemeinschaft (DFG) under
grant no. PR1266/3-1, Design Paradigms for Societal-Scale Cyber-Physical
Systems.

## References

  * (1)
  * Amir-Mohammadian et al. (2016) Sepehr Amir-Mohammadian, Stephen Chong, and Christian Skalka. 2016\.  Correct audit logging: Theory and practice. In International Conference on Principles of Security and Trust. Springer, 139–162.
  * Ananny and Crawford (2018) Mike Ananny and Kate Crawford. 2018.  Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability.  New Media & Society 20, 3 (2018), 973–989.
  * Baldoni et al. (2018) Matteo Baldoni, Cristina Baroglio, Katherine M May, Roberto Micalizio, and Stefano Tedeschi. 2018\.  An information model for computing accountabilities. In International Conference of the Italian Association for Artificial Intelligence. Springer, 30–44.
  * Baron and Kenny (1986) Reuben M Baron and David A Kenny. 1986.  The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.  Journal of Personality and Social Psychology 51, 6 (1986), 1173.
  * Beckers and Halpern (2019) Sander Beckers and Joseph Y Halpern. 2019.  Abstracting causal models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2678–2685.
  * Clough (2002) Bruce T Clough. 2002\.  Metrics, schmetrics! How the heck do you determine a UAV’s autonomy anyway.  Technical Report. Air Force Research Lab Wright-Patterson AFB OH.
  * Datta et al. (2016) Anupam Datta, Soummya Kar, Bruno Sinopoli, and Sean Weerakkody. 2016.  Accountability in cyber-physical systems. In 2016 Science of Security for Cyber-Physical Systems Workshop (SOSCYPS). IEEE, 1–3.
  * Diakopoulos (2015) Nicholas Diakopoulos. 2015\.  Algorithmic accountability: Journalistic investigation of computational power structures.  Digital journalism 3, 3 (2015), 398–415.
  * Doshi-Velez et al. (2017) Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O’Brien, Stuart Schieber, James Waldo, David Weinberger, and Alexandra Wood. 2017\.  Accountability of AI under the law: The role of explanation.  arXiv preprint arXiv:1711.01134 (2017).
  * Feigenbaum et al. (2011) Joan Feigenbaum, Aaron D Jaggard, and Rebecca N Wright. 2011\.  Towards a formal model of accountability. In Proceedings of the 2011 workshop on New security paradigms. ACM, 45–56.
  * Feigenbaum et al. (2012) Joan Feigenbaum, Aaron D Jaggard, Rebecca N Wright, and Hongda Xiao. 2012.  Systematizing “Accountability” in Computer Science (Version of Feb. 17, 2012).  Technical Report. YALEU/DCS/TR-1452, Yale University, New Haven, CT.
  * Felici and Pearson (2014) M Felici and S Pearson. 2014.  D: C-2.1 Report detailing conceptual framework.  Deliverable D32 1 (2014), A4CLOUD.  http://cloudaccountability.eu/sites/default/files/D32.1%20Conceptual%20Framework.pdf
  * Floridi and Sanders (2004) Luciano Floridi and Jeff W Sanders. 2004.  On the morality of artificial agents.  Minds and machines 14, 3 (2004), 349–379.
  * Foster (1926) Benjamin Oliver Foster. 1926\.  Titus Livius (Livy), The History of Rome, Book 8, chapter 7 .  (1926).  http://www.perseus.tufts.edu/hopper/text?doc=Liv.+8+7 [Online; acc. 2019-07-02].
  * Hall et al. (2017) Angela T. Hall, Dwight D. Frink, and M. Ronald Buckley. 2017\.  An accountability account: A review and synthesis of the theoretical and empirical research on felt accountability.  Journal of Organizational Behavior 38, 2 (2017), 204–224.  DOI:http://dx.doi.org/10.1002/job.2052 JOB-13-0646.R4.
  * Halpern (2015) Joseph Y Halpern. 2015\.  A Modification of the Halpern-Pearl Definition of Causality.. In International Joint Conference on Artificial Intelligence. 3022–3033.  https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/view/11058/11085
  * Halpern (2016) Joseph Y Halpern. 2016\.  Actual causality.  MIT Press.
  * Halpern and Kleiman-Weiner (2018) Joseph Y Halpern and Max Kleiman-Weiner. 2018.  Towards formal definitions of blameworthiness, intention, and moral responsibility. In 32nd AAAI Conference on Artificial Intelligence.
  * Halpern and Pearl (2005a) Joseph Y. Halpern and Judea Pearl. 2005a.  Causes and Explanations: A Structural-Model Approach. Part I: Causes.  The British Journal for the Philosophy of Science 56, 4 (Dec. 2005), 843–887.  DOI:http://dx.doi.org/10.1093/bjps/axi147
  * Halpern and Pearl (2005b) Joseph Y. Halpern and Judea Pearl. 2005b.  Causes and Explanations: A Structural-Model Approach. Part II: Explanations.  The British Journal for the Philosophy of Science 56, 4 (Dec. 2005), 889–911.  DOI:http://dx.doi.org/10.1093/bjps/axi148
  * Ibrahim et al. (2019a) Amjad Ibrahim, Severin Kacianka, Alexander Pretschner, Charles Hartsell, and Gabor Karsai. 2019a.  Practical Causal Models for Cyber-Physical Systems. In 11th Annual NASA Formal Methods Symposium.
  * Ibrahim et al. (2019b) Amjad Ibrahim, Simon Rehwald, and Alexander Pretschner. 2019b.  Efficient Checking of Actual Causality with SAT Solving.  Engineering Secure and Dependable Software Systems 53 (2019), 241\.  https://arxiv.org/abs/1904.13101
  * Kacianka et al. (2017) Severin Kacianka, Kristian Beckers, Florian Kelbert, and Prachi Kumari. 2017.  How accountability is implemented and understood in research tools. In International Conference on Product-Focused Software Process Improvement. Springer, 199–218.
  * Kacianka et al. (2019) Severin Kacianka, Amjad Ibrahim, Alexander Pretschner, Alexander Trende, and Andreas Lüdtke. 2019\.  Extending Causal Models from Machines into Humans. In 4th Workshop on Formal Reasoning about Causation, Responsibility, & Explanations in Science & Technology.
  * Kacianka and Pretschner (2018) Severin Kacianka and Alexander Pretschner. 2018.  Understanding and Formalizing Accountability for Cyber-Physical Systems. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3165–3170.
  * Küsters et al. (2010) Ralf Küsters, Tomasz Truderung, and Andreas Vogt. 2010\.  Accountability: definition and relationship to verifiability. In Proceedings of the 17th ACM Conference on Computer and Communications Security. ACM, 526–535.
  * Lee (2008) Edward A Lee. 2008\.  Cyber physical systems: Design challenges. In 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC). IEEE, 363–369.
  * Leucker and Schallhart (2009) Martin Leucker and Christian Schallhart. 2009.  A brief account of runtime verification.  The Journal of Logic and Algebraic Programming 78, 5 (2009), 293–303.
  * Lindberg (2013) Staffan I Lindberg. 2013\.  Mapping accountability: core concept and subtypes.  International Review of Administrative Sciences 79, 2 (2013), 202–226.  DOI:http://dx.doi.org/10.1177/0020852313477761
  * Lomax and Schumacker (2004) Richard G Lomax and Randall E Schumacker. 2004.  A beginner’s guide to structural equation modeling.  Psychology Press.
  * Mill (1861) John Stuart Mill. 1861\.  Consideration of representative government.  [Online: https://www.gutenberg.org/ebooks/5669 acc. 2019-07-02].
  * Miller (2018) Tim Miller. 2018\.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial Intelligence (2018).
  * Mitchell et al. (1998) Terence R. Mitchell, Heidi Hopper, Denise Daniels, Jane George Falvy, and Gerald R. Ferris. 1998\.  Power, Accountability, and Inappropriate Actions.  Applied Psychology 47, 4 (1998), 497–517.  DOI:http://dx.doi.org/10.1111/j.1464-0597.1998.tb00040.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1464-0597.1998.tb00040.x
  * Mittelstadt et al. (2019) Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019\.  Explaining Explanations in AI. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). ACM, New York, NY, USA, 279–288.  DOI:http://dx.doi.org/10.1145/3287560.3287574
  * Mittelstadt et al. (2016) Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016\.  The ethics of algorithms: Mapping the debate.  Big Data & Society 3, 2 (2016), 2053951716679679\.
  * NTSB (2018) NTSB. 2018.  Car with automated vehicle controls crashes into pedestrian .  (2018).  https://www.ntsb.gov/investigations/Pages/HWY18FH010.aspx [Online; acc. 2019-08-20].
  * Pauw (1991) Dirk Anton Pauw. 1991\.  The dramatic elements in Livy’s history.  Acta Classica 34 (1991), 33–49.  http://www.casa-kvsa.org.za/1991/AC34-08-Pauw.pdf
  * Pearl and Bareinboim (2014) Judea Pearl and Elias Bareinboim. 2014.  External validity: From do-calculus to transportability across populations.  Statist. Sci. 29, 4 (2014), 579–595.
  * Pearl et al. (2016) Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016\.  Causal inference in statistics: A primer.  John Wiley & Sons.
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  The Book of Why.  New York, NY: Basic Books.
  * Rahman et al. (2016) N. H. Ab Rahman, W. B. Glisson, Y. Yang, and K. K. R. Choo. 2016\.  Forensic-by-Design Framework for Cyber-Physical Cloud Systems.  IEEE Cloud Computing 3, 1 (Jan 2016), 50–59.  DOI:http://dx.doi.org/10.1109/MCC.2016.5
  * Renieres and Reiss (2003) Manos Renieres and Steven P Reiss. 2003.  Fault localization with nearest neighbor queries. In 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings. IEEE, 30–39.
  * Samland and Waldmann (2016) Jana Samland and Michael R Waldmann. 2016.  How prescriptive norms influence causal inferences.  Cognition 156 (2016), 164–176.
  * Sayes (2014) Edwin Sayes. 2014\.  Actor–Network Theory and methodology: Just what does it mean to say that nonhumans have agency?  Social Studies of Science 44, 1 (2014), 134–149.
  * Simon (2015) Judith Simon. 2015\.  Distributed epistemic responsibility in a hyperconnected era.  In The Onlife Manifesto. Springer, Cham, 145–159.
  * Singh et al. (2018a) Jatinder Singh, Jennifer Cobbe, and Chris Norval. 2018a.  Decision Provenance: Harnessing data flow for accountable systems.  IEEE Access 7 (2018), 6562–6574.
  * Singh et al. (2018b) Jatinder Singh, Christopher Millard, Chris Reed, Jennifer Cobbe, and Jon Crowcroft. 2018b.  Accountability in the IoT: Systems, Law, and Ways Forward.  Computer 51, 7 (2018), 54–65.
  * Smith et al. (2005) Michael L Smith, James Erwin, and Sandra Diaferio. 2005\.  Role & responsibility charting (RACI). In Project Management Forum (PMForum). 5\.
  * Sytsma et al. (2012) Justin Sytsma, Jonathan Livengood, and David Rose. 2012\.  Two types of typicality: Rethinking the role of statistical typicality in ordinary causal attributions.  Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 43, 4 (2012), 814–820.
  * Thoemmes (2018) Felix Thoemmes. 2018\.  The front-door criterion in linear, parametric models.  (2018).  http://www.felixthoemmes.com/blog/the-front-door-criterion-in-linear-parametric-models/ [Online; acc. 2019-08-10].
  * Tikka and Karvanen (2018) Santtu Tikka and Juha Karvanen. 2018.  Identifying causal effects with the R package causaleffect.  arXiv preprint arXiv:1806.07161 (2018).
  * Vesely et al. (1981) W.E. Vesely, F.F. Goldberg, N.H. Roberts, and D.F. Haasl. 1981.  Fault Tree Handbook.  (1981).
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017\.  Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GPDR.  Harv. JL & Tech. 31 (2017), 841.
  * Weitzner et al. (2008) Daniel J. Weitzner, Harold Abelson, Tim Berners-Lee, Joan Feigenbaum, James Hendler, and Gerald Jay Sussman. 2008.  Information Accountability.  Commun. ACM 51, 6 (June 2008), 82–87.  DOI:http://dx.doi.org/10.1145/1349026.1349043