Causal Abstraction for Faithful Model Interpretation

Atticus Geiger<EMAIL_ADDRESS>
Department of Linguistics

Stanford University
Christopher Potts<EMAIL_ADDRESS>
Department of Linguistics

Stanford University
Thomas Icard<EMAIL_ADDRESS>
Department of Philosophy

Stanford University

A faithful and interpretable explanation of an AI model's behavior and internal structure is a high-level explanation that is human-intelligible but also consistent with the known, but often opaque low-level causal details of the model. We argue that the theory of causal abstraction provides the mathematical foundations for the desired kinds of model explanations. In causal abstraction analysis, we use interventions on model-internal states to rigorously assess whether an interpretable high-level causal model is a faithful description of an AI model. Our contributions in this area are: (1) We generalize causal abstraction to cyclic causal structures and typed high-level variables. (2) We show how multi-source interchange interventions can be used to conduct causal abstraction analyses. (3) We define a notion of approximate causal abstraction that allows us to assess the degree to which a high-level causal model is a causal abstraction of a lower-level one. (4) We prove constructive causal abstraction can be decomposed into three operations we refer to as marginalization, variable-merge, and value-merge. (5) We formalize the XAI methods of LIME, causal effect estimation, causal mediation analysis, iterated nullspace projection, and circuit-based explanations as special cases of causal abstraction analysis.

§ INTRODUCTION

We take the fundamental question in explainable artificial intelligence (XAI) to be why a deep learning model makes the predictions it does. The gold standard for explaining model behavior and the internal reasoning that underlies it should be a causal analysis [Woodward, 2003, Pearl, 2019].

However, not just any causal explanation will be a satisfying answer to this question. After all, the parameters of deep learning models give us perfect ground-truth knowledge of the causal relationships between all their components, so low-level causal explanations of behavior and internal reasoning can be easily provided in terms of neurons, activation functions, and weight tensors. Ironically, `black box' models are in some sense easily explainable.  The problem is that these explanations are not interpretable to humans—they fail to instill a sense of understanding [Lipton, 2018].

Interpretable causal explanations in terms of human-intelligible concepts are easy to come by, but difficult to trust. How do we know a high-level explanation isn't a `just-so' story that is  unfaithful [Jacovi and Goldberg, 2020] to the known low-level details of the model? XAI needs a theory for when a high-level causal explanation is harmonious with a low-level causal explanation.

This need is not unique to XAI. Modern deep learning models are akin to the weather or a brain: they contain a large number of densely connected microvariables that have complex, non-linear dynamics. In response to these challenges, Chalupka et al., 2017,  Rubenstein et al., 2017, Beckers and Halpern, 2019 and others pioneered the theory of causal abstraction. Causal abstraction provides a mathematical framework for analyzing a system at multiple levels of detail simultaneously by defining when a human-intelligible, high-level causal explanation is a faithful interpretation of opaque low-level details. Specifically, a high-level (possibly symbolic) model is a faithful proxy for a low-lever (in our setting, usually neural) model when we can align high-level variables with sets of low-level variables that play the same causal role. Causal abstraction has been applied to the study of deep learning AI models [Chalupka et al., 2015, Geiger et al., 2020, Geiger et al., 2021], weather patterns [Chalupka et al., 2016], and human brains [Dubois et al., 2020, Dubois et al., 2020]. Furthermore, approximate causal abstraction [Beckers et al., 2019] supports a graded notion of faithfulness appropriate for a wide range of settings in which a low-level model is unlikely to be perfectly aligned with a high-level one.

We are optimistic about the success of causal abstraction as a theoretical framework for XAI. In stark contrast to weather and brains, we can measure and manipulate the microvariables of deep learning models with perfect precision and accuracy, and thus empirical claims about their structure can be held to the highest standard of rigorous falsification through experimentation.

This Paper
We develop the theory of causal abstraction as a mathematical framework for XAI.

* We generalize a familiar notion of causal abstraction from the literature [Beckers and Halpern, 2019] to cyclic causal models and typed high-level variables.

* We flesh out the full theory of interchange interventions, a method used to conduct causal abstraction analysis of deep learning models [Geiger et al., 2020, Geiger et al., 2021]. An interchange intervention sets the variables of a model to be the value they would take for a different input. The current theory only supports high-level causal explanations with a single intermediate variable and has no connection to approximate causal abstraction [Beckers et al., 2019]. We define a general theory of interchange interventions for XAI in which high-level explanations can involve several variables (Section <ref>), and we define a notion of approximate causal abstraction that corresponds and this gives rise to a graded faithfulness metric of interchange intervention accuracy (Section <ref>).

* We prove that the relation of constructive abstraction holds exactly when the high-level model can be constructed from the low-level model by marginalizing away variables, merging sets of variables, and merging values of variables (Section <ref>). This constructive characterization decomposes abstraction into constituent parts and makes explicit the relationship between marginalization and abstraction. Variable merge, value merge, and marginalization are basic and ubiquitous operations that demystify causal abstraction.

* We show that LIME, causal effect estimation, causal mediation analysis, iterated nullspace projection, and circuit-based explanations are all special cases of causal abstraction analysis, and that integrated gradients [Sundararajan et al., 2017] can be used to compute interchange interventions and conduct causal abstraction analysis (Section <ref>). This lends further support to the thesis that causal abstraction provides a broad foundation for developing explanation methods that are faithful and human-interpretable.

§ RELATED WORK

§.§ Causal Abstraction

Our approach to structural analysis invokes an important concept from the literature on causal models, namely causal abstraction. Much attention has been devoted to the concept in recent years, with a host of alternative approaches. All of this work attempts to specify exactly when a `high-level causal model' can be seen as an abstract characterization of some `low-level causal model'.

[Rubenstein et al., 2017] introduce the notion of an `exact transformation' between models, which maps values of low-level variables to values of high-level variables. Building on this work, [Beckers and Halpern, 2019] consider further restrictions on these maps, e.g., constraining the relationship between transformations of variable values and of interventions. The last definition they present is called constructive abstraction, and our work here focuses on this simple notion. Indeed, because we are working in a deterministic setting of neural network analysis, we can simplify the notation and definitions considerably (see Remark <ref>). Along the way, we also offer an alternative, constructive characterization of constructive abstraction (Section <ref>).

The basic idea of constructive abstraction is that low-level variables are partitioned into clusters, each cluster being associated with a high-level variable. This fundamental idea has been invoked numerous times in the literature, under different names (e.g., Iwasaki and Simon, 1994, Chalupka et al., 2017).  Finally, beyond perfect abstraction, several authors have explored notions of approximate causal abstraction [Beckers et al., 2019, Rischel and Weichwald, 2021]. Another contribution in this paper is to connect interchange intervention analysis with this line of work. Indeed, we see that a natural definition of interchange intervention accuracy previously employed in the literature emerges as a minor variation on existing definitions of approximate abstraction (Theorem <ref>).

§.§ Faithful and Interpretable Causal Explanations of AI

Causal Explanation The philosophical literature on scientific explanation recognizes a variety of  modes in which phenomena can be explained. In the specific instance that we are explaining how an artifact works, there is wide consensus that causal or mechanistic explanations enjoy a privileged status [Machamer et al., 2000]. Particularly when the components of mechanisms are understood in counterfactual terms, such explanations have the virtue of providing (at least in principle) manipulation and control of the system or artifact in question [Woodward, 2002, Woodward, 2003]. In a similar vein, they allow a researcher to  answer a range of `What if?' questions about the topic.

A prominent theme in this literature is the idea that causal/mechanistic explanations should be pitched at an appropriate level of abstraction. As we shall see (in Section <ref>), the operative notion of abstraction adopted here relates closely to desiderata identified in this literature concerning what makes for an appropriate level of causal analysis [Woodward, 2021].

The fundamental operation underlying causal explanations is the intervention, which sets some number of variables in a causal model to fixed values. The nature of this intervention in the context of XAI is left open. The intervention may be on variables for model inputs, internal model components, real-world data generation processes [Feder et al., 2021, Abraham et al., 2022], or model training data statistics [Elazar et al., 2022]. When intervention is on internal neural vector representations, the values of the intervention might be zero vectors, a perturbed or jittered version of the original vector, a learned binary mask applied to the original vector [Csordás et al., 2021, De Cao et al., 2021], the projection of the original vector onto the null space of a linear probe [Ravfogel et al., 2020, Elazar et al., 2020], a tensor product representation [Soulos et al., 2020], or values realized by the vector on some other actual input [Geiger et al., 2019, Geiger et al., 2020, Vig et al., 2020, Meng et al., 2022].

Adopting the vocabulary of [Lipton, 2018], causal abstraction supports interpretable explanations of AI by allowing black-box AI models to be explained by high-level causal models that are transparent in two senses. First, the high-level causal model has fewer variables and sparser causal connections, making it simulatable in the sense that a human could contemplate the entire model at once. Second, the high-level causal model is decomposable in the sense that each variable in the model admits an intuitive explanation with intelligible high-level concepts.

Informally, faithfulness has been defined as the degree to which an explanation accurately represents the `true reasoning process behind a model’s behavior' [Jacovi and Goldberg, 2020, Lyu et al., 2022]. We provide a technical definition for faithfulness by defining a high-level explanation to be faithful to the degree that the high-level explainer model is an approximate causal abstraction of the low-level model being explained. A shared technical definition of a graded notion of faithfulness will allow for apples-to-apples comparisons between different high-level explanations of model behavior and internal reasoning.

Often XAI methods are evaluated based on how plausible or useful a human finds them to be [Herman, 2017]. There is, however, no guarantee that evaluations based on human judgments will be faithful to the true internal reasoning process of a model [Jacovi and Goldberg, 2020].

§.§ Methods for Explaining AI Behavior

The behavior of an AI model is simply the function from inputs to outputs that the model implements. Behavior is trivial to characterize in causal terms. Any input–output behavior can be represented by a two-variable causal model with an input variable that causes an output variable.

Training Interpretable Models to Approximate the Behavior of Black Boxes
Among the most popular XAI methods are LIME [Ribeiro et al., 2016] and SHAP [Lundberg and Lee, 2017], which both learn interpretable models that approximate an uninterpretable model. These methods define an explanation to be faithful to the degree that the interpretable model agrees with local input–output behavior. While not advertised as causal explanation methods, when we interpret priming a model with an input as an intervention, it becomes obvious that two models having the same local input–output behavior is fundamentally a matter of causality.

Crucially, however, the interpretable model lacks any connection to the internal causal dynamics of the uninterpretable model. In fact, it is often presented as a benefit that these explanations are model-agnostic methods that provide the same explanations for models with identical behaviors, but different internal structures. Without further grounding in causal abstraction, methods like LIME and SHAP cannot be trusted to tell us anything meaningful about the abstract causal structure between input and output.

Estimating the Causal Effect of Real-World Concepts on Modal Behavior

CausaLM [Feder et al., 2021] estimates the causal effect that real-world concepts have on the predictions of language models by training analysis models that remove a particular concept, while accounting for a set of confounding concepts. [Elazar et al., 2022] develop a causal framework for understanding the impact of training data on model outputs, which they employ to demonstrate that several co-occurrence statistics have causal effects on the factual knowledge of language models.

[Abraham et al., 2022] introduce an XAI benchmark where the task is to estimate the causal effect of food, service, noise, and ambiance on the sentiment label assigned by a deep learning model to a naturalistic restaurant review. Human generated interventional data is used to evaluate explainer methods, and they find a simple baseline is superior to all the methods they evaluated. Such starkly negative results highlight the need to ground notions of faithfulness in causality to support comparisons between explanation methods.

§.§ Methods for Explaining the Internal Structure of AI

The internal reasoning of an AI model is the intermediate computational process that underlies input–output behavior. Internal reasoning can be represented as a program or algorithm [Putnam, 1960], which in turn can be represented as a causal model [Icard, 2017].

There is a diverse body recent research with the common aim of illuminating the causal mechanisms inside black box models by understanding the human-intelligible concepts represented by neural activations. Causal abstraction provides a mathematical foundation for understanding the high-level semantics of neural representations. In Section <ref>, we return to the methods of LIME, causal effect estimation, causal mediation analysis, iterated nullspace projection, and  circuit-based explanations to formally characterize how these methods can be understood as special cases of causal abstraction. We also show how integrated gradients can be used to compute interchange interventions.

Interchange Interventions
An interchange intervention is a method where a model is provided some input and has an internal representation $h$ fixed to be the value that $h$ would have realized if a different input were provided. Such interventions are central to a strand of research that proposes that symbolic algorithms and neural networks can be related through causal abstraction. We briefly review that strand of research here.

[Geiger et al., 2020] initially used interchange interventions to show that a BERT-based natural language inference model creates a neural representation of lexical entailment, arguing that the ability to create modular representations underlies the capacity for systematic generalization. [Li et al., 2021] use interchange interventions to show that neural representations represent propositional content that systematically alters text generated by a neural language model. [Geiger et al., 2021] explicitly ground interchange intervention analysis in a theory of causal abstraction.

Finally, [Geiger et al., 2022] use interchange interventions to define differentiable training objectives that incentivize neural models to implement high-level algorithms that enable the networks to solve systematic generalization tasks. [Wu et al., 2022] use interchange intervention training during language model distillation, defining a training objective that incentivizes a small `student' model to be abstracted by a large `teacher' model.
[Wu et al., 2022] use interchange intervention training to train causal proxy models that can be queried to predict the causal effects of real-world concepts on a separate original model, achieving state-of-the-art performance on the CEBaB dataset [Abraham et al., 2022]. Additionally, causal proxy models explain themselves and retain performance on the original task, so they can simply replace original models entirely.

Iterative Nullspace Projection [Ravfogel et al., 2020] intervene on a neural representation to remove information about a concept by projecting the representation onto the nullspaces of linear probes trained to predict the given concept. [Elazar et al., 2020] apply iterative nullspace projection to the analysis of pretrained language models, finding that probing techniques do not correlate with the causal presence of a concept. [Lovering and Pavlick, 2022] use iterative nullspace projection to evaluate whether neural representations encode concepts with `mental' causes and effects.

Causal Mediation Analysis [Vig et al., 2020] conduct a causal mediation analysis to analyze gender bias in pretrained language models with transformer architectures [Vaswani et al., 2017, Devlin et al., 2019]. They found that gender bias was mediated by a small set of neurons whose total effect on the output could be linearly decomposed into their direct and indirect effects. [Meng et al., 2022] introduce the technique of causal tracing for mediation analysis. Perturbations are made to input neurons and then intermediate neurons are restored to determine the flow of causation from input to output. This technique was used to locate and edit the factual knowledge stored in large language models. [Csordás et al., 2021] and [De Cao et al., 2021] use differentiable binary masking to identify modular structure in neural networks by performing causal mediation analysis.

Circuit-Based Explanations A recent research program aims to provide explanations of neural networks by reverse engineering the mechanisms of a networks at the level of individual neurons. Two fundamental claims made by [Olah et al., 2020] are that (1) the direction of a neural vector representation encodes high-level concept(s) (e.g., curved lines or object orientation) and (2) the `circuits' defined by the model weights connecting neural representations encode meaningful algorithms. An extensive study of a deep convolutional network [Cammarata et al., 2020] support these core claims with a variety of qualitative and quantitative experiments.

[Elhage et al., 2021] extend this circuit framework to attention-only transformers, discovering `induction heads' that search for previous occurrences of the present token and copy the token directly after the previous occurence. In follow-up work, [Olsson et al., 2022] provides indirect evidence that induction heads underlie the ability of standard transformers to achieve in-context learning. Finally, [Elhage et al., 2022] propose softmax linear units as an activation function that can be used in transformers to incentivize the creation of interpretable neurons.

Other Intervention-Based Research

[Giulianelli et al., 2018] intervene on an LSTM language model at prediction time to encourage better performance on English subject–verb agreement; probes help them home in on the most productive intervention points, but it is the intervention itself that positively affects model behavior.

[Soulos et al., 2020] use interventions on encoder output representations to support claims about the compositional structure of recurrent neural networks being explained by a tensor product representation [McCoy et al., 2019].

[Bau et al., 2019] and [Besserve et al., 2020] search clusters of neurons that perform modular functions in generative adversarial vision networks. They use intervention experiments to illuminate the purpose of the neural modules, discovering that they encode intuitive high-level concepts such as the presence of trees in the generated image.

Probing is the technique of using a supervised or unsupervised model to determine whether a concept is present in a neural representation of a separate model. Probes are a popular tool for analyzing deep learning models, especially pretrained language models [Hupkes et al., 2018, Conneau et al., 2018, Peters et al., 2018, Tenney et al., 2019, Clark et al., 2019].

Although probes are quite simple, our theoretical understanding of probes has greatly improved since their recent introduction into the field. From an information-theoretic point of view, we can observe that using arbitrarily powerful probes is equivalent to measuring the mutual information between the concept and the neural representation [Hewitt and Liang, 2019, Pimentel et al., 2020]. If we restrict the class of probing models based on their complexity, we can measure how usable the information is [Xu et al., 2020, Hewitt et al., 2021]. Regardless of what probe models are used, successfully probing a neural representation does not guarantee that the representation plays a causal role in model behavior [Ravichander et al., 2020, Elazar et al., 2020, Geiger et al., 2020, Geiger et al., 2021].

Feature Attribution
Feature attribution methods ascribe scores to neural representations that capture the `impact' of the representation on model behavior. Gradient-based feature attribution methods [Zeiler and Fergus, 2014, Springenberg et al., 2014, Shrikumar et al., 2016, Binder et al., 2016] measure causal properties when they satisfy some basic axioms [Sundararajan et al., 2017].  [Chattopadhyay et al., 2019] argue for a direct measurement of a neuron's individual causal effect, and [Geiger et al., 2021] provide a natural causal interpretation of the integrated gradients method.

§ CAUSAL MODELS

We start with some basic notation.
Throughout we use $\mathbf{V}$ to denote a fixed set of variables, each $X \in \mathbf{V}$ coming with a range $\mathsf{Val}(X)$ of possible values.

We assume $\mathsf{Val}(X) \cap \mathsf{Val}(Y) = \varnothing$ whenever $X\neq Y$, meaning no two variables can take on the same value.[This is akin to representing partial and total settings as vectors where a value can occur multiple times. For instance, we can simply take any causal model where variables share values, and then `tag' the shared values with variable names to make them unique.] This assumption allows representing the values of a set of variables $\mathbf{X} \subseteq \mathbf{V}$ as sets $\mathbf{x}$ of values, with exactly one value $x \in \mathsf{Val}(X)$ in $\mathbf{x}$ for each $X \in \mathbf{X}$.
We refer to the values $\mathbf{x} \in \mathsf{Val}(\mathbf{X}) $  as  partial settings.
In the special case where $\mathbf{v} \in \mathsf{Val}(\mathbf{V})$, we call $\mathbf{v}$ a total setting.

Capital letters (e.g., $X$) are used for variables and lower case letters (e.g., $x$) are used for values. Bold faced letters (e.g. $\mathbf{X}$ or $\mathbf{x}$) are used for sets of variables and sets of values.
When a variable (or set of variables) and a value (or set of values) have the same letter, the values correspond to the variables (e.g., $x \in \mathsf{Val}(X)$ or $\mathbf{x} \in \mathsf{Val}(\mathbf{X})$). We  use $\mathsf{Domain}(f)$ to denote the domain of a function $f$, $\mathsf{Uniform}(\mathcal{X})$ to be a uniform distribution on (a finite set) $\mathcal{X}$, and $\mathbbm{1}[\varphi]$ to be an indicator function that takes in a proposition $\varphi$ and outputs $1$ if $\varphi$ is true, $0$ otherwise.

Another useful construct in this connection is the projection of a partial setting:

Given a partial setting $\mathbf{u}$ for a set of variables $\mathbf{U} \supseteq \mathbf{X}$, we define $\mathsf{Proj}(\mathbf{u},\mathbf{X})$ to be the restriction of $\mathbf{u}$ to the variables in $\mathbf{X}$. Given a partial setting $\mathbf{x}$ and a set $\mathbf{U} \supseteq \mathbf{X}$:

\[ \mathsf{Proj}^{-1}(\mathbf{x}, \mathbf{U}) = \{\mathbf{u} \in \mathsf{Val}(\mathbf{U}): \mathsf{Proj}(\mathbf{u},\mathbf{X}) = \mathbf{x} \}. \]

A (deterministic) causal model is a pair $\mathcal{M} = (\mathbf{V},\mathcal{F})$, such that $\mathbf{V}$ is a set of variables and  $\mathcal{F} = \{f_{V}\}_{V \in \mathbf{V}}$ is a set of structural functions, with $f_V: \mathsf{Val}(\mathbf{V}) \rightarrow \mathsf{Val}(V)$ assigning a value to $V$  as a function of the values of all the variables.

Observe that our definition of causal model makes no explicit reference to a graphical structure defining a causal ordering on the variables. While the structural function for a variable takes in total settings, it might be the case that the output of a structural function depends only on a subset of values. We can use the structural functions to induce a causal ordering among the variables by saying that $Y \prec X$—or $Y$ is a parent of $X$—just in case there is a setting $\mathbf{w}$ of all the variables $\mathbf{W} = \mathbf{V}\setminus\{X,Y\}$ other than $X$ and $Y$, and two settings $y,y'$ of $Y$ such that $f_X(\mathbf{w},y) \neq f_X(\mathbf{w},y')$ (see, e.g., Woodward, 2003). The resulting order $\prec$ captures a general type of causal dependence. Throughout the paper, we will define structural functions to take in partial settings of parent variables, though technically they take in total settings and ignore all values other than those of the parent variables.

When $\prec$ is irreflexive, we say the causal model is acyclic. Most of our examples of causal models will have this property. However, it is often also possible to give causal interpretations of cyclic models (see, e.g., Bongers et al., 2021). Indeed, the abstraction operations to be introduced generally create cycles among variables, even from initially acyclic models (see  for an example). In Section <ref>, we provide an example where we abstract a causal model representing the bubble sort algorithm into a cyclic model where any sorted list is a solution satisfying the equations.

Our running example will be causal abstraction holding between two finite, acyclic causal models. We will call variables that depend on no other variable input variables ($ \mathbf{X}_{\mathcal{M}}^{\text{In}}$), and variables on which no other variables depend output variables ($ \mathbf{X}_{\mathcal{M}}^{\text{Out}}$). The remaining variables are intermediate variables. We can intervene on input variables $ \mathbf{X}_{\mathcal{M}}^{\text{In}}$ to “prime” the model with a particular input. As such, we will define the functions for input variables in our examples to be constant functions that will be overwritten.

As $\M$ can also be interpreted simply as a set of equations, we can define the set of solutions, which may be empty.

Given $\M = (\mathbf{V},\mathcal{F})$, the set of solutions, called $\sols(\M)$, is the set of all $\mathbf{v} \in \mathsf{Val}(\mathbf{V})$ such that all the equations $v = f_V(\mathbf{v})$ are satisfied for each $v \in \mathbf{v}$. When $\mathcal{M}$ is acyclic, each intervention results in a model with a single solution and we use $\sols(\mathcal{M}_\mathbf{i})$ to refer interchangeably to a singleton set of solutions and its sole member, relying on context to disambiguate.

We can also give a general definition of intervention on a model (see, e.g., Spirtes et al., 2000, Woodward, 2003, Pearl, 2009).

Let $\M$ be a model. We define an intervention to be a partial setting $\mathbf{i} \in  \mathsf{Val}(\mathbf{I})$ for  $\mathbf{I} \subseteq \mathbf{V}$.
We define $\M_{\mathbf{i}}$ to be just like $\M$, except that we replace $f_X$ with the constant function $\mathbf{v} \mapsto \mathsf{Proj}(\mathbf{i}, X)$ for each $X \in \mathbf{I}$.

Due to our focus on fully observable, deterministic systems, we forego the opportunity to incorporate probability into our causal models. However, we discuss extensions of the present work to the probabilistic setting in Section <ref>.

§ EXAMPLE OF CAUSAL MODELS: A SYMBOLIC ALGORITHM AND NEURAL NETWORK

For our running example, we define two basic examples of causal models that demonstrate a potential to model a diverse array of computational processes; the first causal model represents a tree-structured algorithm and the second causal model represents a fully-connected feed-forward neural network. Both the network and the algorithm solve the same `hierarchical equality' task.

§.§ Hierarchical Equality Task

A basic equality task is to determine whether a pair of objects are identical. A hierarchical equality task is to determine whether a pair of pairs of objects have identical relations.

The input to the hierarchical task is two pairs of objects, and the output is $\mathsf{True}$ if both pairs are equal or both pairs are unequal, and $\mathsf{False}$ otherwise. For illustrative purposes, we define the domain of objects to consist of a triangle, square, and pentagon. For example, the input $\Large(\pentagon, \pentagon, \bigtriangleup, \square)$ is assigned the output $\mathsf{False}$ and the inputs $\Large(\pentagon, \pentagon, \bigtriangleup, \bigtriangleup)$ and $\Large(\pentagon, \square, \bigtriangleup, \square)$ are both labeled $\mathsf{True}$.

We chose this task for two reasons. First, there is an obvious tree-structured symbolic algorithm that solves the task: compute whether the first pair is equal, compute whether the second is equal, then compute whether those two outputs are equal. We will encode this algorithm as a causal model. Second, equality reasoning is ubiquitous and has served as a case study for broader questions about the representations underlying relational reasoning in biological organisms [Marcus et al., 1999, Alhama and Zuidema, 2019, Geiger et al., 2022].

For present purposes, hierarchical equality will serve as a case study for explaining how abstract tree-structured composition can be implemented by a fully-connected neural network. We will define a causal model of the symbolic algorithm and a causal model of a neural network that labels hierarchical equality inputs. The neural network implements the hierarchical equality task because we trained it to [Geiger et al., 2022].

§.§ A Tree-Structured Algorithm for Hierarchical Equality

We define a tree structured algorithm $\mathcal{A}$ consisting of four `input' variables $\mathbf{X}^{\text{In}}_{\mathcal{A}} =$ $ \{X_1,X_2,X_3,X_4\}$ each with possible values[Strictly speaking, these variable values are indexed by their variables. We omit the indices for readability.] $\{\Large\pentagon, \bigtriangleup, \square \normalsize\}$, two `intermediate' variables $V_1, V_2$ with values $\{\texttt{True}, \texttt{False}\}$, and one `output' variable $\mathbf{X}^{\text{Out}}_{\mathcal{A}} = \{O\}$ with values $\{\texttt{True}, \texttt{False}\}$. The acyclic causal graph is depicted in Figure <ref>,
where each $f_{X_i}$ (with no arguments) is a constant function to $\large \pentagon$ (which will be overwritten, per Remark <ref>), and $f_{V_1},f_{V_2},f_{O}$ are all identity over their respective domains, e.g., $f_{V_1}(x_1,x_2) = \mathbbm{1}\big [x_1=x_2\big ]$. A total setting can be captured by a vector $[x_1,x_2,x_3,v_1,v_2,o]$ of values for each of the variables.

The default total setting that results from no intervention is $[\large \pentagon,\pentagon,\pentagon,\pentagon,\normalsize{}\texttt{True}, \texttt{True}, \texttt{True}]$. We can also ask what would have occurred had we intervened to fix $X_3$,$X_4$, and $V_2$ to be $\bigtriangleup$, $\square$, and $\texttt{True}$, for example. The counterfactual result is $[\large \pentagon,\pentagon,\bigtriangleup,\square\normalsize, \texttt{True}, \texttt{True}, \texttt{True}]$.

[scale=0.6, every node/.style=scale=0.6]

in 0,1,2,3
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=12pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.309*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (A) at (0+ ,0+ ) $I_1$;

[fill=green!20, draw, circle, minimum size=20pt] (B) at (1*+ ,0+ ) $I_2$;

[fill=green!20, draw, circle, minimum size=20pt] (C) at (2*+ ,0+ ) $I_3$;

[fill=green!20, draw, circle, minimum size=20pt] (D) at (3*+ ,0+ ) $I_4$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) $V_1$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) $V_2$;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) $O$;

() at (0.5*+ ,0.87*+ ) $\mathbf{Id_1}(i_1,i_2)$;

() at (2.5*+ ,0.87*+ ) $\mathbf{Id_1}(i_3,i_4)$;

() at (1.5*+ ,1.37*+ ) $\mathbf{Id_2}(v_1,v_2)$;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

[scale=0.6, every node/.style=scale=0.6]

(mat) [matrix of nodes,
row sep=-,
column sep=-,
nodes=rectangle,text width=2em,align=center,
text depth=1ex,
text height=2ex,
nodes in empty cells] at (0,0)

in 1

([xshift=-.5,opacity=]mat--1.south west) –
([xshift=-.5,opacity=]mat--4.south east);

in 1

([yshift=.5,opacity=]mat-1-.north east) –
([yshift=.5,opacity=]mat-4-.south east);

in 2,...,3

([xshift=-.5]mat--1.south west) –
([xshift=-.5]mat--4.south east);

in 2,...,3

([yshift=.5]mat-1-.north east) –
([yshift=.5]mat-4-.south east);

[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=12pt,inner sep=0pt] () at (mat-1-2) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (mat-1-3) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (mat-1-4) ;

[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=12pt,inner sep=0pt] () at (mat-2-1) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (mat-3-1) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (mat-4-1) ;

(mat) [matrix of nodes,
row sep=-,
column sep=-,
nodes=rectangle,text width=2em,align=center,
text depth=1ex,
text height=2ex,
nodes in empty cells] at (5.05,0)

in 1

([xshift=-.5,opacity=]mat--1.south west) –
([xshift=-.5,opacity=]mat--3.south east);

in 1

([yshift=.5,opacity=]mat-1-.north east) –
([yshift=.5,opacity=]mat-3-.south east);

in 2

([xshift=-.5]mat--1.south west) –
([xshift=-.5]mat--3.south east);

in 2

([yshift=.5]mat-1-.north east) –
([yshift=.5]mat-3-.south east);

The algorithm.

[scale=0.6, every node/.style=scale=0.6]
[fill=green!20, draw, circle, minimum size=20pt] (A) at (0+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (B) at (1*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (C) at (2*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (D) at (3*+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (0+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (1*+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (2*+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (3*+ ,0+ ) ;

[fill=blue!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) ;

[fill=blue!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) ;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) ;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

The total setting of $\mathcal{A}$ determined by the empty intervention.

[scale=0.6, every node/.style=scale=0.6]
[fill=green!20, draw, circle, minimum size=20pt] (A) at (0+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (B) at (1*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (C) at (2*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (D) at (3*+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (0+ ,0+ ) ;

[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (1*+ ,0+ ) ;

[fill=orange!80, draw,regular polygon, regular polygon sides=4, minimum size=12pt,inner sep=0pt] () at (2*+ ,0+ ) ;

[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=12pt,inner sep=0pt] () at (3*+ ,0+ ) ;

[fill=blue!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) ;

[fill=red!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) ;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) ;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

The total setting of $\mathcal{A}$ determined by the intervention fixing $X_3$, $X_4$, and $V_2$ to be $\bigtriangleup$, $\square$, and $\texttt{True}$.

A tree-structured algorithm that perfectly solves the hierarchical equality task with a compositional solution.

§.§ A Fully Connected Neural Network for Hierarchical Equality

Define a neural network $\mathcal{N}$, consisting of eight `input' neurons $\mathbf{X}^{\text{Out}}_{\mathcal{N}} = \{R_1, \dots, R_8\}$, twenty-four `intermediate' neurons $H_{(i,j)}$ for $1 \leq i \leq 3$ and $1 \leq i \leq 8$, and two `output' neurons $\mathbf{X}^{\text{Out}}_{\mathcal{N}} = \{\text{O}_{\mathsf{True}},\text{O}_{\mathsf{False}}\}$. The values for each of these variables are the real numbers $\mathbb{R}$.   We depict the causal graph in Figure <ref>.

[thick,scale=0.6, every node/.style=scale=0.6]

in 1.5,3.5,5.5,7.5
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.316*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (4 * ,3.75 * ) $\text{O}_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (5 * ,3.75 * ) $\text{O}_{\mathsf{False}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * ,0 * ) ${R_{1}}$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * ,0 * ) ${R_{2}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * ,0 * ) ${R_{3}}$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * ,0 * ) ${R_{4}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * ,0 * ) ${R_{5}}$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * ,0 * ) ${R_{6}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * ,0 * ) ${R_{7}}$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT08) at (8 * ,0 * ) ${R_{8}}$;

in  1,2,3
in 1,2,3,4,5,6,7,8
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* ,* ) $H_{(\row, \col)}$ ;

in 1,2,3,4,5,6,7,8
in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT0.north) to (BERT1.south);
[opacity=0.3, ->] (BERT1.north) to (BERT2.south);
[opacity=0.3, ->] (BERT2.north) to (BERT3.south);

in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT3.north) to (out.south);
[opacity=0.3, ->] (BERT3.north) to (out2.south);

\[\footnotesize
\begin{tabular}{ccc}
$\mathbf{r}_{\large \pentagon} = [0.012, -0.301]$ & $\mathbf{r}_{\large \square} = [-0.812, 0.456]$ & $\mathbf{r}_{\large \bigtriangleup} = [0.682, 0.333]$\\
\end{tabular}\]

$W_1 = \begin{bmatrix*}[r]
2.1754\mathrm{e}+00  & -7.1769\mathrm{e}-01 &  -2.1649\mathrm{e}+00 &   6.8584\mathrm{e}-01 &  -3.5393\mathrm{e}-02 &  6.8148\mathrm{e}-03 &  -3.1675\mathrm{e}-02 &  -2.9090\mathrm{e}-02 \\
7.3452\mathrm{e}-03 &  -7.1636\mathrm{e}-03 &  -3.2291\mathrm{e}-04 &  -3.0044\mathrm{e}-03 &   2.2651\mathrm{e}+00 &  -1.9941\mathrm{e}+00 &  -2.2740\mathrm{e}+00 &   2.0176\mathrm{e}+00 \\
-2.1238\mathrm{e}+00 &  -1.6024\mathrm{e}+00 &   2.1323\mathrm{e}+00 &   1.5860\mathrm{e}+00 &   8.2233\mathrm{e}-04 &  5.6526\mathrm{e}-03 &   3.6328\mathrm{e}-02 &   2.9736\mathrm{e}-03 \\
1.4659\mathrm{e}-02 &   8.5846\mathrm{e}-03 &  -7.3199\mathrm{e}-03 &   6.3861\mathrm{e}-03 &  -2.5226\mathrm{e}+00 &  -1.2776\mathrm{e}+00 &   2.5292\mathrm{e}+00 &   1.2696\mathrm{e}+00\\
3.9916\mathrm{e}+00 &  -1.3048\mathrm{e}+00 &  -3.9189\mathrm{e}+00 &   1.2616\mathrm{e}+00 &  -3.4642\mathrm{e}-02 &  -1.4530\mathrm{e}-02 &   7.7398\mathrm{e}-03 &  -5.3017\mathrm{e}-02\\
-3.6872\mathrm{e}+00 &   4.1371\mathrm{e}-01 &   3.7053\mathrm{e}+00 &  -4.0671\mathrm{e}-01 &   1.3940\mathrm{e}-02 &  -1.1155\mathrm{e}-03 &  -1.0047\mathrm{e}-03 &   6.5350\mathrm{e}-03\\
-2.5656\mathrm{e}-01 &   8.9275\mathrm{e}-01 &   2.4427\mathrm{e}-01 &  -8.2323\mathrm{e}-01 &  -3.2437\mathrm{e}-02 &  3.5100\mathrm{e}-02 &   3.6350\mathrm{e}-04 &  -5.6254\mathrm{e}-02\\
-8.2522\mathrm{e}-03 &   3.7633\mathrm{e}-03 &  -7.6974\mathrm{e}-03 &   6.7534\mathrm{e}-03 &  -4.2970\mathrm{e}-01 & 3.0961\mathrm{e}+00 &   4.1024\mathrm{e}-01 &  -3.1124\mathrm{e}+00\\
\end{bmatrix*}$

$b_1 =
\begin{bmatrix}-0.0475 & -0.0151 &  0.0692 & -0.0451 & -0.0777 &  0.0627 & -0.0591 & -0.0196\\
\end{bmatrix}$

$W_2 = \begin{bmatrix*}[r] -4.9015e-01 & -4.9981e-03 & -1.0025e+01 & -1.7210e-02 & -7.1630e-01 &
-4.7900e-01 & -1.0036e-01 &  1.1562e-02\\
9.6085e+00 & -1.5522e-02 &  2.9515e+00 & -4.7480e-03 &  1.3156e+01 &   1.2285e+01 &  3.3859e-01 &  1.7905e-02\\
1.0056e+00 & -5.7124e+00 &  2.5843e-02 & -3.5200e+00 & -5.1716e-01 & 5.0762e-02 & -6.9766e-02 & -3.6073e+00\\
-1.0586e+00 &  9.8556e+00 & -2.9570e-02 &  8.4745e+00 &  5.6010e-01 &2.2497e-02 & -3.9709e-02 &  8.3327e+00\\
1.2459e+00 & -3.9747e+00 & -2.7281e-01 & -2.1867e+00 & -1.0082e+00 &2.6007e-01 & -1.2949e+00 & -9.1204e-01\\
-7.8904e-01 & -1.1877e-01 &  1.9272e-01 & -2.1950e-02 &  1.1614e-02 &-4.2569e+00 & -1.8045e-01 & -4.6835e-02\\
-1.5830e+00 & -1.3206e+00 & -1.2504e-01 & -5.1255e-01 & -3.0161e+00 &-1.5335e+00 & -6.2201e-02 & -8.8656e-01\\
-1.0493e+00 &  9.5044e-02 & -3.9964e-01 &  9.0716e-02 & -2.4861e+00 &-7.6741e-02 & -2.2552e-01 &  5.3066e-02\\
\end{bmatrix*}

$b_2 =
\begin{bmatrix}  0.2746 & -0.1292 &  0.2347 &  0.2833 & -0.3366 & -0.1478 &  0.0980 &  0.1673\\
\end{bmatrix}$

$W_3 =
\begin{bmatrix*}[r]   4.2248e-01 &  2.7740e+00 &  1.0335e+00 & -4.5362e+00 &  8.7838e-01 &     -3.1472e-01 & -4.0193e-01 & -3.6744e-01\\
-2.8812e-01 & -3.1679e-01 &  1.5999e-02 & -1.9106e-01 &  7.9891e-03 &      6.2744e-02 & -3.2715e-01 & -2.8038e-01\\
3.1208e+00 &  1.4330e+00 & -5.8909e+00 &  3.6747e-02 & -1.7160e+00 &     2.0737e+00 & -1.1598e+01 &  2.9503e-01\\
2.9027e+00 & -3.8384e+00 & -1.2183e+01 &  4.0099e-02 & -6.1718e-01 &     9.6990e-01 & -1.2437e+01 &  1.9529e-01\\
-4.8936e+00 &  1.7727e+00 &  3.4905e+00 & -2.3871e+00 &  6.8049e-01 &
-2.7405e+00 &  2.5876e+00 &  1.1965e+00\\
-2.6034e-01 & -2.5041e-01 & -1.7218e-01 & -3.5845e-03 &  1.1953e-01 &  -6.4683e-02 &  2.8834e-02 &  5.3538e-02\\
4.8684e-01 & -3.9053e-03 & -1.3724e-01 & -9.7928e+00 &  7.0597e-02 &1.4577e-01 & -1.0602e+00 & -5.7370e-02\\
1.7658e+00 &  1.9812e+00 &  9.3662e-01 & -1.6880e-02 &  1.0065e+00 &    -9.9469e-01 &  2.9824e+00 & -1.3896e+00\\
\end{bmatrix*}$

$b_3 =
\begin{bmatrix} -0.8259 & -0.1688 & -2.4404 &  1.8469 &  0.9255 & -0.3003 & -0.0837 & -0.2753\\
\end{bmatrix}$

$W_4 =
\begin{bmatrix*}[r]  3.8760 &  -0.1366 &  -3.1693 &  -2.7446 &  -1.8302 &  -0.1413 & -12.2571 & -1.7698\\
-3.8155 &  -0.0268 &   2.9001 &   2.4848 &   1.8365 &  -0.0793 &  12.3972 &1.2339\\
\end{bmatrix*}$

$b_4 =
\begin{bmatrix} 3.2679 & -3.5912\\
\end{bmatrix}$

A fully-connected feed-forward neural network that labels inputs for the hierarchical equality task. We define weights for a network that was created with interchange intervention training to implement the tree-structured solution to the task.

Define $\mathbf{R}$, $\mathbf{H}_1$, $\mathbf{H}_2$, $\mathbf{H}_3$ to be the sets of variables for the first four layers, respectively.

We define $f_{R_k}$ (with no arguments) to be a constant function to $0$, for $1 \leq k \leq 8$. The intermediate and output neurons are determined by the network weights $W_{1},W_2,W_3 \in \mathbb{R}^{8\times8}$, ${W}_4 \in \mathbb{R}^{8\times2}$, and bias terms $b_1, b_2, b_3 \in \mathbb{R}^{8}$, $b_4 \in \mathbb{R}$.  For $1 \leq j \leq 8$, we define

$f_{H_{(1,j)}}(\mathbf{r}) = \mathsf{ReLU}((\mathbf{r}{W}_1 + {b}_1)_j)$     $f_{H_{(2,j)}}(\mathbf{h}_1) = \mathsf{ReLU}((\mathbf{h}_1{W}_2 + {b}_2)_j)$

\[f_{H_{(3,j)}}(\mathbf{h}_2) = \mathsf{ReLU}((\mathbf{h}_2{W}_3 + {b}_3)_j) \]

$f_{O_{\mathsf{True}}}(\mathbf{h}_3) = \mathsf{ReLU}((\mathbf{h}_3{W}_4 + {b}_4)_0)$     $f_{O_{\mathsf{False}}}({h}_3) = \mathsf{ReLU}((\mathbf{h}_3{W}_4 + {b}_4)_1)$

The four shapes that are the input for the hierarchical equality task are represented in $\mathbf{r}_{\large \pentagon}, \mathbf{r}_{\large \square}, \mathbf{r}_{\large \bigtriangleup} \in \mathbb{R}^2$ by a pair of neurons with randomized activation values. The network outputs $\mathsf{True}$ if the value of the output logit $O_{\mathsf{True}}$ is larger than the value of $O_{\mathsf{False}}$, and $\mathsf{False}$ otherwise. We can simulate a network operating on the input $\large (\square, \pentagon, \square, \bigtriangleup)$ by performing an intervention setting $(R_1, R_2)$ and $(R_5,R_6)$ to $\mathbf{r}_{\large \square}$, $(R_3, R_4)$ to $\mathbf{r}_{\large \pentagon}$, and $(R_7,R_8)$ to $\mathbf{r}_{\large \bigtriangleup}$.

In Figure <ref>, we define neural representations for the shapes and the weights of a network $\mathcal{N}$ trained to implement our tree-structured algorithm $\mathcal{A}$ [Geiger et al., 2022]. While the network $\mathcal{N}$ was trained to implement $\mathcal{A}$, looking at the network weights provides no insight into this relationship; the theoretical tool of causal abstraction is needed to illuminate `black-box' neural networks with intervention experiments.

§ CAUSAL ABSTRACTION AND INTERCHANGE INTERVENTION ANALYSIS

Suppose we have a `low-level model' $\mathcal{L} = (\mathbf{V}_{\mathcal{L}},\mathcal{F}_{\mathcal{L}})$ built from `low-level variables' $\mathbf{V}_{\mathcal{L}}$ and a `high-level model' $\mathcal{H} = (\mathbf{V}_{\mathcal{H}},\mathcal{F}_{\mathcal{H}})$ built from `high level variables' $\mathbf{V}_{\mathcal{H}}$. (For instance, these might be $\mathcal{N}$ and $\mathcal{A}$, respectively, from the previous section.)
What structural conditions must be in place for $\mathcal{H}$ to be  a high-level abstraction of the low-level model $\mathcal{L}$?

§.§ Alignments Between Causal Models

A prominent intuition about abstraction is that it may involve associating specific high-level variables with clusters of low-level variables. To systematize this, we introduce a notion of an alignment between a low-level and high-level causal model:

An alignment between $\mathcal{L}$ and $\mathcal{H}$ is given by a pair $\langle \Pi,\tau \rangle$ of a partition $\Pi=\{\Pi_{X_{\mathcal{H}}}\}_{X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}} \cup \{\bot \}}$ and a family $\tau = \{\tau_{X_{\mathcal{H}}}\}_{X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}}}$ of maps, such that:

* The partition $\Pi$ of $\mathbf{V}_{\mathcal{L}}$ consists of non-overlapping, non-empty cells $\Pi_{X_{\mathcal{H}}} \subseteq \mathbf{V}_{\mathcal{L}}$ for each $X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}}$, in addition to a (possibly empty) cell $\Pi_{\bot}$,

* There is a partial surjective map $\tau_{X_{\mathcal{H}}}:\mathsf{Val}(\Pi_{X_\mathcal{H}})\rightarrow\mathsf{Val}(X_{\mathcal{H}})$ for each $X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}}$.

In words, the set $\Pi_{X_{\mathcal{H}}}$ consists of those low-level variables that are `aligned' with the high-level variable $X_{\mathcal{H}}$, and $\tau_{X_{\mathcal{H}}}$ tells us how a given setting of the low-level cluster $\Pi_{X_{\mathcal{H}}}$ corresponds to a setting of the high-level variable $X_{\mathcal{H}}$. The remaining set $\Pi_\bot$ consists of those low-level variables that are `forgotten', not mapped to any high-level variable.

[thick,scale=0.6, every node/.style=scale=0.6]

in 1.5,3.5,5.5,7.5
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.316*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

[draw, ellipse,  minimum width=220pt, minimum height=44pt] (piO) at ( 4.5 * ,4 * + ) ;

[draw, ellipse,  minimum width=180pt, minimum height=44pt] (piV1) at ( 2.5 * ,2.25 * + ) ;
[draw, ellipse,  minimum width=180pt, minimum height=44pt] (piV2) at ( 7.5 * ,2.25 * + ) ;

[draw, ellipse,  minimum width=190pt, minimum height=50pt] (piI1) at ( 1.5 * ,0.25 * + ) ;
[draw, ellipse,  minimum width=190pt, minimum height=50pt] (piI2) at ( 3.5 * ,0.25 * + ) ;
[draw, ellipse,  minimum width=190pt, minimum height=50pt] (piI3) at ( 5.5 * ,0.25 * + ) ;
[draw, ellipse,  minimum width=190pt, minimum height=50pt] (piI4) at ( 7.5 * ,0.25 * + ) ;

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (4 * ,3.75 * ) $O_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (5 * ,3.75 * ) $O_{\mathsf{False}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * ,0 * ) $X_1$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * ,0 * ) $X_2$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * ,0 * ) $X_3$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * ,0 * ) $X_4$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * ,0 * ) $X_5$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * ,0 * ) $X_6$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * ,0 * ) $X_7$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT08) at (8 * ,0 * ) $X_8$;

in  1,2,3
in 1,2,3,4,5,6,7,8
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* ,* ) $H_{(\row, \col)}$;

in 1,2,3,4,5,6,7,8
in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT0.north) to (BERT1.south);
[opacity=0.3, ->] (BERT1.north) to (BERT2.south);
[opacity=0.3, ->] (BERT2.north) to (BERT3.south);

in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT3.north) to (out.south);
[opacity=0.3, ->] (BERT3.north) to (out2.south);

in 0,1,2,3
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.309*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (A) at (0+ ,0+ ) $I_1$;

[fill=green!20, draw, circle, minimum size=20pt] (B) at (1*+ ,0+ ) $I_2$;

[fill=green!20, draw, circle, minimum size=20pt] (C) at (2*+ ,0+ ) $I_3$;

[fill=green!20, draw, circle, minimum size=20pt] (D) at (3*+ ,0+ ) $I_4$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) $V_1$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) $V_2$;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) $O$;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

[opacity=0.8, -, dotted] (piO.north) to (G.south);

[-, dotted] (piI1) to[in=180, out=160] (A);
[-, dotted] (piI2) to[in=-160, out=90] (B);
[-, dotted] (piI3) to[in=-20, out=90] (C);
[-, dotted] (piI4) to[in=0, out=20] (D);

[-, dotted] (piV1) to[in=135, out=90] (E);
[-, dotted] (piV2) to[in=45, out=90] (F);

An alignment between the causal graphs of a low-level fully-connected neural network (bottom) and a high-level tree structured algorithm (top).

An alignment $\langle \Pi, \tau \rangle$ induces a unique translation, viz. a (partial) function  $\tau:\mathsf{Val}(\mathbf{V}_{\mathcal{L}}) \rightarrow \mathsf{Val}(\mathbf{V}_{\mathcal{H}})$. To wit,
for any $\mathbf{v}_{\mathcal{L}} \in \mathsf{Val}(\mathbf{V}_{\mathcal{L}})$:

\begin{eqnarray} \tau(\mathbf{v}_{\mathcal{L}}) & \overset{\textnormal{def}}{=} & \bigcup_{X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}}} \tau_{X_{\mathcal{H}}} \big( \mathsf{Proj}(\mathbf{v}_{\mathcal{L}},\Pi_{X_{\mathcal{H}}}) \big). \label{eq:transl}
\end{eqnarray}

The notion of an alignment (or of a translation) by itself does not tell us anything about how the causal laws, as specified by $\mathcal{F}_{\mathcal{L}}$ and $\mathcal{F}_{\mathcal{H}}$, relate. We also want these to be in harmony, in the sense that (counterfactually) intervening on one produces `the same' result in the other, where `the same' means that the resulting states remain aligned.

In that direction, note that $\tau$ extends naturally from a partial function $\mathsf{Val}(\mathbf{V}_{\mathcal{L}}) \rightarrow \mathsf{Val}(\mathbf{V}_{\mathcal{H}})$, defined just on total settings, to a partial function ${\tau}: \bigcup_{\mathbf{X}_{\mathcal{L}} \subseteq \mathbf{V}_{\mathcal{L}}} \mathsf{Val}(\mathbf{X}_{\mathcal{L}}) \rightarrow \bigcup_{\mathbf{X}_{\mathcal{H}} \subseteq \mathbf{V}_{\mathcal{H}}} \mathsf{Val}(\mathbf{X}_{\mathcal{H}})$, defined on the larger domain of interventions (viz. partial settings). We only define ${\tau}$ on low-level interventions that correspond to high-level interventions. We set ${\tau}(\mathbf{x}_{\mathcal{L}}) =\mathbf{x}_{\mathcal{H}}$ if and only if
\begin{eqnarray}\label{eq:tauext}
\tau\big(\mathsf{Proj}^{-1}(\mathbf{x}_{\mathcal{L}}, \mathbf{V}_{\mathcal{L}})\big) & = &  \mathsf{Proj}^{-1}(\mathbf{x}_{\mathcal{H}}, \mathbf{V}_{\mathcal{H}}).
\end{eqnarray}

Thus, the cell-wise maps $\tau_{X_{\mathcal{H}}}$ canonically give us this partial function $\tau$. Note that we are overloading the notation $\tau$, letting it refer to both the function on total settings in Eq. (<ref>) and the function on partial settings specified by Eq. (<ref>).

§.§ Causal Consistency and Constructive Abstraction

An alignment $\langle \Pi, \tau \rangle$ between $\mathcal{L}$ and $\mathcal{H}$ is  consistent if for all $\mathbf{i} \in \mathsf{Domain}(\tau)$ such that $\sols(\mathcal{L}_{\mathbf{i}})$ is non-empty, the following diagram commutes:

(n1) at (0,0) $\mathbf{i}$;
(n2) at (4,0) $\tau(\mathbf{i})$;
(n1) edge[->] (n2);
at (2,.25) ${\tau}$;
(n3) at (0,-1.5) $\sols\big(\mathcal{L}_{\mathbf{i}}\big)$;
(n4) at (4,-1.5) $\sols\big(\mathcal{H}_{\tau(\mathbf{i})}\big) $;
(n3) edge[->] (n4);
at (2,-1.25) $\tau$;
(n1) edge[->] (n3);
(n2) edge[->] (n4);

That is, the high-level intervention $\tau(\mathbf{i})$ corresponding to low-level intervention $\mathbf{i}$ results in the same high-level total settings as the result of first determining a low-level setting from $\mathbf{i}$ and then applying the translation to obtain a high-level setting. In a single equation:

\begin{eqnarray}\tau\big(\sols\big(\mathcal{L}_{\mathbf{i}}\big)\big)  & = & \sols\big(\mathcal{H}_{{\tau}(\mathbf{i})}\big).\label{eq:cc}\end{eqnarray}

We thus arrive at a definition of causal abstraction:

$\mathcal{H}$ is a constructive  abstraction of $\mathcal{L}$ under an alignment $\langle \Pi, \tau \rangle$ if the causal consistency condition is satisfied.

Though the idea was implicit in much earlier work,
[Beckers and Halpern, 2019] and [Beckers et al., 2019] explicitly introduced the notion of a constructive abstraction in the setting of probabilistic causal models. The definition here is considerably simplified and streamlined, in part because we are not concerned with probability  (though see Section <ref> below).

In the field of program analysis, abstract interpretation is a framework that can be understood as a special case of causal abstraction where models are acyclic and high-level variables are aligned with individual low-level variables rather than sets of low-level variables [Cousot and Cousot, 1977]. The functions $\tau$ and $\tau^{-1}$ are the abstraction and concretization operators that form a Galois connection, and causal consistency guarantees that abstract transfer functions are consistent with concrete transfer functions.

While we assumed in Section <ref> that variables have non-overlapping spaces of values, it is often useful to assume that variable do share values, and we may want to consider more restrictive varieties of causal abstraction that ensure a kind of type consistency. In that direction, define:
Fix a causal model $(\mathbf{V},\mathcal{F})$ and a set $\mathcal{T}$ of types. A typing is a function $T:\mathbf{V}\rightarrow \mathcal{T}$, together with an equivalence relation $\sim$, such that for each $X,Y$ of the same type—i.e., $T(X)=T(Y)$—and each $x \in \mathsf{Val}(X)$, there is exactly one $y \in \mathsf{Val}(Y)$ such that $x \sim y$.
In other words, there is a bijective correspondence $\sim$ between values of variables of the same type, where intuitively $x \sim y$ means that values $x$ and $y$ should behave the same way. For instance, if $X$ and $Y$ are both Boolean variables with values $\{0_X,1_X\}$ and $\{0_Y,1_Y\}$, respectively, then we would naturally have $0_X \sim 0_Y$ and $1_X \sim 1_Y$.  This relation $\sim$ lifts to sets of values in the obvious way: for disjoint sets of variables, $\mathbf{Z}$ and $\mathbf{W}$, we say $\mathbf{z} \sim \mathbf{w}$ if, for each $z \in \mathbf{z}$ there is exactly one $w \in \mathbf{w}$ such that $z \sim w$, and vice versa. With this much can introduce:
Fix causal models  $\mathcal{L} = (\mathbf{V}_{\mathcal{L}},\mathcal{F}_{\mathcal{L}})$ and $\mathcal{H} = (\mathbf{V}_{\mathcal{H}},\mathcal{F}_{\mathcal{H}})$ with typing relations $\sim_{\mathcal{L}}$ and $\sim_{\mathcal{H}}$, respectively. $\mathcal{H}$ is a typed causal abstraction of $\mathcal{L}$ if there is an alignment $\langle \Pi, \tau \rangle$ that is causally consistent (Def. <ref>) and also type consistent: whenever $X$ and $Y$ have the same type in $\mathcal{H}$, then for any $\mathbf{z} \in \mathsf{Val}(\Pi_X)$ and $\mathbf{w} \in \mathsf{Val}(\Pi_Y)$, we have,

\begin{eqnarray*} \mathbf{z}\sim_{\mathcal{L}}\mathbf{w} & \Leftrightarrow & \tau(\mathbf{z}) \sim_{\mathcal{H}} \tau(\mathbf{w}). \end{eqnarray*}

We mention Definition <ref> because typed abstraction played a crucial role in the ability of [Geiger et al., 2022] to use interchange intervention training to solve a vision-based systematic generalization task. Typing assumptions have also been useful in causal discovery [Brouillard et al., 2022]. In Section <ref>, we provide an example involving a typing where variables are split into Booleans and natural numbers to define a causal model representing the bubble sort algorithm.

§.§ Interchange Intervention Analysis

Equipped with the general notion of (constructive) causal abstraction, we now turn to one concrete means of operationalizing claims of causal abstraction. Interchange intervention analysis is one such method.
[Geiger et al., 2022] provides a specialized theory of interchange interventions that only covers cases where the high-level causal model has a single intermediate variable. Furthermore, they formulate the metric of interchange intervention accuracy to capture partially successful model-internal explanations. Here, we expand on this work, presenting a general theory of interchange interventions for high-level causal models with multiple intermediate variables.

When applying causal abstraction to explain neural networks, the central question is how to find an alignment $\langle \Pi,\tau \rangle$ between the algorithm and network. The alignment between the input and output variables in the network and algorithm is simply stipulated by the researcher who knows how they will interpret the inputs and outputs of the network. In contrast, the researcher must search for the best alignment between intermediate variables.

In this search, we partition the low-level intermediate variables, then with $\tau$ already stipulated for input and output, we determine the value of $\tau$ for remaining low-level cells.

Consider some neural network $\mathcal{N}$ and high-level algorithm $\mathcal{A}$ with input and output variables $\mathbf{X}_{\mathcal{L}}^{\text{In}}, \mathbf{X}_{\mathcal{L}}^{\text{Out}} \subseteq \mathbf{V}_{\mathcal{L}}$ and $\mathbf{X}_{\mathcal{H}}^{\text{In}}, \mathbf{X}_{\mathcal{H}}^{\text{Out}} \subseteq \mathbf{V}_{\mathcal{H}}$. Suppose we have $\{\Pi_X\}_{X \in \mathbf{V}_{\mathcal{H}} \cup \bot}$, which is a partition of $\mathbf{V}_{\mathcal{L}}$ where input, output, and intermediate variables are aligned across the two levels:

\[
\begin{tabular}{cc}
$ X \in \mathbf{X}_{\mathcal{H}}^{\text{Out}} \Leftrightarrow \Pi_X \subseteq \mathbf{X}_{\mathcal{L}}^{\text{Out}}$ &  $X \in \mathbf{X}_{\mathcal{H}}^{\text{In}} \Leftrightarrow \Pi_X \subseteq \mathbf{X}_{\mathcal{L}}^{\text{In}}$\\
\end{tabular}\]

\[ X \in \mathbf{V}_{\mathcal{H}} \setminus (\mathbf{X}_{\mathcal{H}}^{\text{Out}} \cup \mathbf{X}_{\mathcal{H}}^{\text{In}}) \Leftrightarrow \Pi_X \subseteq \mathbf{V}_{\mathcal{L}} \setminus (\mathbf{X}_{\mathcal{L}}^{\text{Out}} \cup \mathbf{X}_{\mathcal{L}}^{\text{In}}) .\]

with predefined input and output alignment $\tau_{X}$ for $X \in \mathbf{V}_{\mathcal{H}}^{\text{In}} \cup  \mathbf{V}_{\mathcal{H}}^{\text{Out}}$.
We can induce the intermediate semantics from the input semantics, output semantics, aligned partitions, and the two causal models. For $X_{\mathcal{H}} \in \mathbf{V}_{\mathcal{H}} $ and $\mathbf{z}_{\mathcal{L}} \in \Pi_{X_{\mathcal{H}}}$, if there exists $\mathbf{x}_{\mathcal{L}} \in \mathbf{X}^{\text{In}}_{\mathcal{L}}$ such that $\mathbf{z}_{\mathcal{L}} = \mathsf{Proj}(\sols(\mathcal{N}_{\mathbf{x}_{\mathcal{L}}}), \Pi_{X_{\mathcal{H}}})$,
then we define the intermediate semantics as follows

\[\tau_{X_{\mathcal{H}}}(\mathbf{z}_{\mathcal{L}}) =
\mathsf{Proj}(\sols(\mathcal{A}_{\tau(\mathbf{x}_{\mathcal{L}})}), X_{\mathcal{H}})
\]

and otherwise, we leave $\tau_{X_{\mathcal{H}}}$ undefined for $\mathbf{z}_{\mathcal{L}}$.[
This is a subtle point, but, in general, this construction is not well defined because two different input can produce the same intermediate representation with different high-level interpretations. If this were to happen, the causal abstraction relationship simply wouldn't hold for the alignment.

Once an alignment $\langle \Pi, \tau \rangle$ is constructed, aligned interventions must be performed to experimentally verify the alignment is a witness to the algorithm being an abstraction of the network.  Observe that $\tau$ will be only be defined for values of intermediate partition cells that are realized when some input is provided to the network. This greatly constrains the space of low-level interventions to intermediate partitions that will correspond with high-level interventions.

However, we will still be able to interpret certain low-level interchange interventions [Geiger et al., 2020]. Originally, an interchange intervention was relative to a base input and source input. The network is run with the base input, while a set of neurons is fixed to be the value they would have taken if the source input were provided instead. However, having a single source input restricts analyses to high-level causal models with a single intermediate variable. To support arbitrary high-level models, we extend the definition to multiple source inputs.

Because neural networks involve vectors of real numbers, there are in principle many possible interventions one could perform on neural variables. Most of these settings, however, need not have any intuitive significance when it comes to explaining model behavior on a range of possible inputs. For this reason, we restrict our causal abstraction relation to multi-source interchange interventions that are realized for some possible input.

Consider a neural network $\mathcal{N}$, with input and output variables $\mathbf{X}_{\mathcal{L}}^{\text{In}}, \mathbf{X}_{\mathcal{L}}^{\text{Out}} \subseteq \mathbf{V}_{\mathcal{L}}$, disjoint subsets of intermediate variables $ \mathbf{X}_{\mathcal{L}}^1, \dots, \mathbf{X}_{\mathcal{L}}^k \subseteq \mathbf{V}_{\mathcal{L}} \setminus  (\mathbf{X}_{\mathcal{L}}^{\text{In}} \cup \mathbf{X}_{\mathcal{L}}^{\text{Out}})$, and base input $\mathbf{b}$ and source inputs $\mathbf{s}_1, \dots, \mathbf{s}_k \in \mathbf{X}_{\mathcal{L}}^{\text{In}}$. We define an interchange intervention to be the partial setting that sets the input variables to the base input $\mathbf{b}$ and the intermediate variables $\mathbf{X}_{\mathcal{L}}^i$ to the values they would take on if the source input $\mathbf{s}_i$ were provided to $\mathcal{N}$

\[\label{eq:interchange}
\mathsf{IntInv}(\mathcal{N}, \mathbf{b}, \langle \mathbf{s}_1, \dots ,\mathbf{s}_k \rangle, \langle \mathbf{X}_{\mathcal{L}}^1, \dots, \mathbf{X}_{\mathcal{L}}^k\rangle) \overset{\textnormal{def}}{=}
\mathbf{b}\ \cup \ \mathsf{Proj}((\sols(\mathcal{N}_{\mathbf{s}_1}), \mathbf{X}_{\mathcal{L}}^1) \cup \dots \cup \mathsf{Proj}(\sols(\mathcal{N}_{\mathbf{s}_k}), \mathbf{X}_{\mathcal{L}}^k)\]

[thick,scale=0.6, every node/.style=scale=0.6]

in 0.5,2.5
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

in 4.5
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+  ,-0.316*+ ) ;

in 6.5
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

[draw, ellipse,  minimum width=180pt, minimum height=44pt] (piV1) at ( 1.5 * + ,2.25 * + ) ;

(label) at (4 * + ,4.25 * ) $\texttt{False}$;

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (3 * + ,3.75 * ) $O_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (4 * + ,3.75 * ) $O_{\mathsf{False}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT00) at (0 * + ,0 * ) $x_1$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * + ,0 * ) $x_2$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * + ,0 * ) $x_3$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * + ,0 * ) $x_4$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * + ,0 * ) $x_5$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * + ,0 * ) $x_6$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * + ,0 * ) $x_7$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * + ,0 * ) $x_8$;

in  1,2,3
in 0,1,2,3,4,5,6,7
[fill=red!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* ) $h_{(\row, \col)}$ ;

in 0,1,2,3,4,5,6,7
in 0,1,2,3,4,5,6,7
[opacity=0.3, ->] (BERT0.north) to (BERT1.south);
[opacity=0.3, ->] (BERT1.north) to (BERT2.south);
[opacity=0.3, ->] (BERT2.north) to (BERT3.south);

in 0,1,2,3,4,5,6,7
[opacity=0.3, ->] (BERT3.north) to (out.south);
[opacity=0.3, ->] (BERT3.north) to (out2.south);

[fill=green!20, draw, circle, minimum size=40pt] (A) at (0+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (B) at (1*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (C) at (2*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (D) at (3*+ ,0+ ) ;

[fill=red!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) ;

[fill=red!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) ;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) ;

in 0,1
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

in 2
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+  ,-0.309*+ ) ;

in 3
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

in 0.5
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

in 2.5
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

in 4.5,6.5
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+  ,-0.316*+ ) ;

[draw, ellipse,  minimum width=180pt, minimum height=44pt] (pi2V1) at ( 1.5 * + ,2.25 * + ) ;

(label) at (3 * + ,4.25 * ) $\texttt{True}$;

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (3 * + ,3.75 * ) $O_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (4 * + ,3.75 * ) $O_{\mathsf{True}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT00) at (0 * + ,0 * ) $r_1$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * + ,0 * ) $r_2$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * + ,0 * ) $r_3$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * + ,0 * ) $r_4$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * + ,0 * ) $r_5$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * + ,0 * ) $r_6$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * + ,0 * ) $r_7$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * + ,0 * ) $r_8$;

in  1
in 0,1,2,3,4,5,6,7
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* ) $h_{(\row, \col)}$ ;

in  2
in 0,3,4,5,6,7
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* ) $h_{(\row, \col)}$ ;

in  2
in 1,2
[fill=red!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* )  $h_{(\row, \col)}$;

in  3
in 0,1,2,3,4,5,6,7
[fill=purple!60, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* )  $h_{(\row, \col)}$;

in 0,1,2,3,4,5,6,7
in 0,1,2,3,4,5,6,7
[opacity=0.3, ->] (BERT0.north) to (BERT1.south);
[opacity=0.3, ->] (BERT1.north) to (BERT2.south);
[opacity=0.3, ->] (BERT2.north) to (BERT3.south);

in 0,1,2,3,4,5,6,7
[opacity=0.3, ->] (BERT3.north) to (out.south);
[opacity=0.3, ->] (BERT3.north) to (out2.south);

[fill=green!20, draw, circle, minimum size=40pt] (A) at (0+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (B) at (1*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (C) at (2*+ ,0+ ) ;

[fill=green!20, draw, circle, minimum size=40pt] (D) at (3*+ ,0+ ) ;

[fill=red!20, draw, ellipse, minimum size=20pt] (E2) at (0.5*+ ,0.5*+ ) ;

[fill=blue!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) ;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) ;

in 0
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;

in 1
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ ,-0.3*+ ) ;

in 2,3
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+  ,-0.309*+ ) ;

[->,opacity=] (A) – (E2);
[->,opacity=] (B) – (E2);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E2) – (G);
[->,opacity=] (F) – (G);

[dashed,  ->, thick] (E) to[out=7, in=170] (E2);
[dashed, ->, thick] (piV1) to[out=20, in=160] (pi2V1);

The result of aligned interchange intervention on the low-level fully-connected neural network and a high-level tree structured algorithm under the alignment in Figure <ref>. Observe the equivalent counterfactual behavior across the two levels.

The interchange interventions used by Geiger et al., 2021 are limited in a second way: they consider only interventions that, for each low-level partition cell, either fix the entirety of the cell or fix none of the cell. The resulting set of interventions will verify that every actually realized value of a low-level partition cell can be used in every actually realized context. Formally, the restricted set of interchange interventions can be represented as

\[\mathsf{IntInv}(\mathcal{N}, \mathbf{b}, \langle \mathbf{s}_1, \dots ,\mathbf{s}_k \rangle, \langle \Pi_{\mathbf{X}_{\mathcal{H}}^1}, \dots, \Pi_{\mathbf{X}_{\mathcal{H}}^k} \rangle)\]

where $\mathbf{b}$ and $\mathbf{s}_1, \dots, \mathbf{s}_k$ are base and source inputs and $X_{\mathcal{H}}^1, \dots, X_{\mathcal{H}}^k$ are high level variables.

§.§ Explanation and Generalization

Interchange interventions set variables to values they actually realize on an input. Crucially, this input can come from training data, testing data, or be a theoretical input from some implicitly defined space of well-formed inputs (e.g., the space of all English questions and answers). Definition <ref> requires a commuting diagram hold for all interventions in the range of the partial function $\tau$. So, if a high-level causal model is an abstraction of a deep learning model where $\tau$ is only defined on training data, the abstraction may not hold when $\tau$ is expanded to be defined on testing data.

If we want to develop interpretable explanation methods that are faithful when applied to unseen real-world input, we should seek explanations that generalize to test examples. This problem of generalizations is in no way unique to XAI methods; generalizing from training to testing data is a central question of machine learning as a field [Hinton, 1989].

§ DECOMPOSING CONSTRUCTIVE CAUSAL ABSTRACTION

Given the importance and prevalence of this relatively simple notion of abstraction, it is worth understanding the notion from different angles. Constructive causal abstractions can be fully characterized in terms of three fundamental operations on a causal model.

Marginalization removes a set of variables from a causal model. Variable merge collapses a partition of variables from a causal model, each partition cell becoming a single variable. Value merge collapses a partition of values for each variable from a causal model, each partition cell becoming a single value. The first and third operations relate closely to concepts identified in the philosophy of science literature as being critical to addressing the problem of variable choice [Woodward, 2021].

For the following definitions, we will need a choice function $\mathsf{Choose}$ that takes in a set and outputs an element of that set. We also employ a function $\mathsf{Filter}$ that takes a variable $Y$ and a value $y \in \mathsf{Val}(Y)$, and returns any value in $\mathsf{Val}(Y)$ not equal to $y$.

Marginalization removes a set of variables from a causal model, directly linking the parents and children of each variable.

Choose some $\mathbf{X} \subset \mathbf{V}$ and define $\marg{\mathcal{M}}{\mathbf{X}}$ as follows. The variables are $\mathbf{W} = \mathbf{V} \setminus \mathbf{X}$, with unaltered value spaces.
For each variable $Y \in \mathbf{W}$, define a new function $\marg{f_Y}{\mathbf{X}}$ on $\mathsf{Val}(\mathbf{W})$ as follows:

\[\marg{f_Y}{\mathbf{X}}(\mathbf{w}) =  \begin{cases}
\mathsf{Proj}(\mathbf{w},Y) &  \text{if  } \mathsf{Proj}^{-1}(\mathbf{w}, \mathbf{V}) \cap \sols(\mathcal{M}) \neq \varnothing, \\
\mathsf{Filter}(Y,\mathsf{Proj}(\mathbf{w}, Y))& \text{otherwise}. \\
\end{cases}
\]

Marginalization is essentially a matter of ignoring a subset $\mathbf{X}$ of variables. Philosophers of science have been concerned with scenarios in which a cause of some effect is relatively insensitive to, or stable under, changes in `background variables' [Lewis, 1986, Woodward, 2006]. That is, if we simply ignored other variables that also make a difference for some effect, how reliably would a given factor cause the effect? The definition of marginalization we give here essentially guarantees perfect insensitivity/stability in this sense. Specifically:

The model $\marg{\mathcal{M}}{\mathbf{x}}$ is a constructive causal abstraction of $\mathcal{M}$ with translation map $\tau(\mathbf{v}) = \mathsf{Proj}(\mathbf{v}, \mathbf{V} \setminus \mathbf{X})$. Equivalently, for all $\mathbf{i} \in \mathsf{Domain}(\tau)$,

\[
\mathsf{Proj}(\sols(\mathcal{M}_{\mathbf{i}}), \mathbf{V} \setminus \mathbf{X}) = \sols(\marg{\mathcal{M}}{\mathbf{X}}_{\mathsf{Proj}(\mathbf{i}, \mathbf{V}\setminus \mathbf{X})}),  \]
i.e., solution sets are preserved under marginalization, for all interventions. (Proof in Appendix <ref>.)

Variable merge collapses each cell of a partition into single variables that depend on the parents of their partition and determines the children of their partition. We define $\varmerge{\mathcal{M}}{\mathbf{W}}$ to be the model where the variables merged according to a partition $\{\Pi_{X}\}_{X \in \mathbf{W}}$ with cells indexed by a new set of variables $\mathbf{W}$.

Choose some partition $\{\Pi_{X}\}_{X \in \mathbf{W}}$ of $\mathbf{V}$ and define $\varmerge{\mathcal{M}}{\mathbf{W}}$ as follows. The variables are $\mathbf{W}$. Each variable $X \in \mathbf{W}$ is assigned the values $\mathsf{Val}(\Pi_X)$. Let $\pi^{-1}$ be the natural bijective function from $\bigcup_{\mathbf{X} \subseteq \mathbf{W}}\mathsf{Val}(\mathbf{X})$ to $\bigcup_{\mathbf{X} \subseteq \mathbf{V}}\mathsf{Val}(\mathbf{X})$ where the value assigned to each $X \in \mathbf{W}$ is decomposed into values for each variable in $\Pi_{X}$. This means $\pi$ is a bijection from $\bigcup_{\mathbf{X} \subseteq \mathbf{V}}\mathsf{Val}(\mathbf{X})$ to $\bigcup_{\mathbf{X} \subseteq \mathbf{W}}\mathsf{Val}(\mathbf{X})$. Each $X \in \mathbf{W}$ is assigned the function

\[\varmerge{f_X}{\mathbf{W}}(\mathbf{v}) = \pi(\bigcup_{Y \in \Pi_X} f_{Y}(\pi^{-1}(\mathbf{v})))\]

$\varmerge{\mathcal{M}}{\mathbf{W}}$ is a constructive causal abstraction of $\mathcal{M}$ with $\tau = \pi$. Equivalently, for all $\mathbf{i} \in \mathsf{Domain}(\tau)$,

\[\pi(\sols (\mathcal{M}_{\mathbf{i}})) = \sols(\varmerge{\mathcal{M}}{\mathbf{W}}_{\pi(\mathbf{i})})\]
i.e., solution sets are preserved under variable merge, for all interventions. (Proof in Appendix <ref>.)

Value merge alters the value space of each variable, potentially collapsing values. Crucially, value merges are valid only if the partition cells respect the causal dynamics of $\mathcal{M}$.

Consider some family $\delta = \{\delta_{X}\}_{X \in \mathbf{V}}$ of functions $\delta_X:\mathsf{Val}(X)\rightarrow B_X$; thus, $\delta$ is a function $\mathsf{Val}(\mathbf{V})\rightarrow \bigcup_{X \in \mathbf{V}} B_X$. We will say $\delta$ is viable for value merge when each $\delta_X$ fails to be injective only when the collapsed values play the same role in $\mathcal{M}$. More formally, suppose $\mathbf{i}, \mathbf{i}' \in \mathsf{Val}(\mathbf{I})$ differ only in that $x \in \mathbf{i}$ and $x' \in \mathbf{i}'$. Then we require that,

\[\delta_X(x) = \delta_X(x') \mbox{ implies }  \delta(\sols(\mathcal{M}_{\mathbf{i}})) =  \delta(\sols(\mathcal{M}_{\mathbf{i}'})).\]

Choose some viable family of functions $\delta_{X}: \mathsf{Val}(X) \to B_X$.
Define $\delta$ to be the natural function applying this family of functions to partial settings. Define $\valmerge{\mathcal{M}}{\delta}$ of $\mathcal{M}$ as follows. Every variable $X$ has new values $B_X$ and a new function

\[\valmerge{f_X}{\delta}(\mathbf{v}) = \delta_X(f_X(\mathsf{Choose}(\delta^{-1}(\mathbf{v}))))\]

Observe that our viability condition on $\delta$ makes it so that the choice of the $\mathsf{Choose}$ function has no effect on the solutions of $\Delta(\mathcal{M})$

$\valmerge{\mathcal{M}}{\delta}$ is a constructive causal abstraction of $\mathcal{M}$ with $\tau = \delta$. Equivalently, for all $\mathbf{i} \in \mathsf{Domain}(\tau)$,

\[\delta(\sols(\mathcal{M}_{\mathbf{i}})) =\sols(\valmerge{\mathcal{M}}{\delta}_{\delta(\mathbf{i})})\]
i.e., solution sets are preserved under value merge, for all interventions. (Proof in Appendix <ref>.)

The notion of value merge also relates to an important concept in the philosophy of causation. The range of values $\mathsf{Val}(X)$ for a variable $X$ can be more or less coarse-grained, and some levels of grain seem to be better causal-explanatory targets. For instance, to use a famous example from [Yablo, 1992], if a bird is trained to peck any target that is a shade of red, then it would be misleading, if not incorrect, to say that crimson causes the bird to peck. Roughly, the reason is that this suggests the wrong counterfactual contrasts: if the target were not crimson (but instead, say, scarlet), the bird would still peck. Thus, for a given explanatory purpose, the level of grain in a model should guarantee that cited causes can be proportional to their effects [Yablo, 1992, Woodward, 2021].

As the following result shows, a low-level model being a constructive causal abstraction of a high-level model is simply a matter of being able to construct the high-level model from the low-level model with marginalization, variable merge, and value merge.

[thick,scale=0.6, every node/.style=scale=0.6]

in 1.5,3.5,5.5,7.5
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.316*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

in 4.5
(start) at (*+  ,-0.746*+ ) ;

in 4.5
(end) at (*+  ,-2.316*+ ) ;

[->, line width=2mm] (start) to (end);

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (4 * + ,3.75 * ) $O_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (5 * + ,3.75 * ) $O_{\mathsf{False}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * + ,0 * ) $R_1$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * + ,0 * ) $R_2$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * + ,0 * ) $R_3$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * + ,0 * ) $R_4$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * + ,0 * ) $R_5$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * + ,0 * ) $R_6$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * + ,0 * ) $R_7$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT08) at (8 * + ,0 * ) $R_8$;

in  1,2,3
in 1,2,3,4,5,6,7,8
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* ) $H_{(\row, \col)}$;

in 1,2,3,4,5,6,7,8
in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT0.north) to (BERT1.south);
[opacity=0.3, ->] (BERT1.north) to (BERT2.south);
[opacity=0.3, ->] (BERT2.north) to (BERT3.south);

in 1,2,3,4,5,6,7,8
[opacity=0.3, ->] (BERT3.north) to (out.south);
[opacity=0.3, ->] (BERT3.north) to (out2.south);

[thick,scale=0.6, every node/.style=scale=0.6]

in 1.5,3.5,5.5,7.5
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.316*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

in 7
(start) at (*+  ,4.5*+ ) ;

in 8
(end) at (*+  ,5.8*+ ) ;

[->, line width=2mm] (start) to (end);

[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out) at (4 * + ,3.75 * ) $O_{\mathsf{True}}$;
[fill=yellow!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt](out2) at (5 * + ,3.75 * ) $O_{\mathsf{False}}$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT01) at (1 * + ,0 * ) $R_1$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT02) at (2 * + ,0 * ) $R_2$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT03) at (3 * + ,0 * ) $R_3$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT04) at (4 * + ,0 * ) $R_4$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT05) at (5 * + ,0 * ) $R_5$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT06) at (6 * + ,0 * ) $R_6$;

[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT07) at (7 * + ,0 * ) $R_7$;
[fill=green!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT08) at (8 * + ,0 * ) $R_8$;

in  2
in 2,3,7,8
[fill=blue!20, draw, rectangle, rounded corners=3pt, minimum width=40pt, minimum height=20pt] (BERT) at (* + ,* )  $H_{(\row, \col)}$;

in 1,2,3,4,5,6,7,8
in 2,3,7,8
[opacity=0.3, ->] (BERT0.north) to (BERT2.south);

in 2,3,7,8
[opacity=0.3, ->] (BERT2.north) to (out.south);
[opacity=0.3, ->] (BERT2.north) to (out2.south);

[thick,scale=0.6, every node/.style=scale=0.6]

in 0.5,2.5,4.5,6.5
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=16pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.316*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

in 3.5
(start) at (*+  ,-0.746*+ ) ;

in 3.5
(end) at (*+  ,-2.316*+ ) ;

[->, line width=2mm] (start) to (end);

[fill=yellow!20,draw, ellipse, minimum size=60pt] (piO) at ( 3.5 * ,4 * + ) $O$;

[fill=blue!20,draw, ellipse, minimum size=60pt] (piV1) at ( 1.5 * ,2.25 * + ) $V_1$;
[fill=blue!20,draw, ellipse, minimum size=60pt] (piV2) at ( 6.5 * ,2.25 * + ) $V_2$;

[fill=green!20,draw, ellipse, minimum size=60pt] (piI1) at ( 0.5 * ,0.25 * + ) $I_1$;
[fill=green!20,draw, ellipse, minimum size=60pt] (piI2) at ( 2.5 * ,0.25 * + ) $I_2$;
[fill=green!20,draw, ellipse, minimum size=60pt] (piI3) at ( 4.5 * ,0.25 * + ) $I_3$;
[fill=green!20,draw, ellipse, minimum size=60pt] (piI4) at ( 6.5 * ,0.25 * + ) $I_4$;

[opacity=0.3, ->] (piI1.north) to (piV1.south);
[opacity=0.3, ->] (piI2.north) to (piV1.south);
[opacity=0.3, ->] (piI3.north) to (piV1.south);
[opacity=0.3, ->] (piI4.north) to (piV1.south);

[opacity=0.3, ->] (piI1.north) to (piV2.south);
[opacity=0.3, ->] (piI2.north) to (piV2.south);
[opacity=0.3, ->] (piI3.north) to (piV2.south);
[opacity=0.3, ->] (piI4.north) to (piV2.south);

[opacity=0.3, ->] (piV1.north) to (piO.south);
[opacity=0.3, ->] (piV2.north) to (piO.south);

[scale=0.6, every node/.style=scale=0.6]

in 0,1,2,3
[] () at (*+ -0.45*+ ,-0.309*+ ) {;
[] () at (*+ 0.45*+ ,-0.309*+ ) };
[fill=orange!80, draw,regular polygon, regular polygon sides=3, minimum size=12pt,inner sep=0pt] () at (*+ -0.3*+ ,-0.309*+ ) ;
[fill=green!80, draw,regular polygon, regular polygon sides=4, minimum size=10pt] () at (*+ 0*+ ,-0.3*+ ) ;
[fill=purple!80, draw,regular polygon, regular polygon sides=5, minimum size=10pt] () at (*+ 0.3*++ ,-0.3*+ ) ;

[fill=green!20, draw, circle, minimum size=20pt] (A) at (0+ ,0+ ) $I_1$;

[fill=green!20, draw, circle, minimum size=20pt] (B) at (1*+ ,0+ ) $I_2$;

[fill=green!20, draw, circle, minimum size=20pt] (C) at (2*+ ,0+ ) $I_3$;

[fill=green!20, draw, circle, minimum size=20pt] (D) at (3*+ ,0+ ) $I_4$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (E) at (0.5*+ ,0.5*+ ) $V_1$;

[fill=blue!20, draw, ellipse, minimum size=20pt] (F) at (2.5*+ ,0.5*+ ) $V_2$;

[fill=yellow!20, draw, ellipse, minimum size=20pt] (G) at (1.5*+ ,1*+ ) $O$;

[->,opacity=] (A) – (E);
[->,opacity=] (B) – (E);
[->,opacity=] (C) – (F);
[->,opacity=] (D) – (F);
[->,opacity=] (E) – (G);
[->,opacity=] (F) – (G);

An illustration of a fully-connected neural network being transformed into a tree structured algorithm by (1) marginalizing away neurons aligned with no high-level variable, (2) merging sets of variables aligned with high level variables, and (3) merging the continuous values of neural activity into the symbolic values of the algorithm.

$\mathcal{H}$ is a constructive abstraction of $\mathcal{L}$ if and only if $\mathcal{H}$ can be obtained from $\mathcal{L}$ by a variable merge, value merge, and marginalization. (Proof in Appendix <ref>.)

§ EXAMPLE OF CAUSAL ABSTRACTION: TREE-STRUCTURE IN NEURAL COMPUTATION

In Section <ref>, we introduced the hierarchical equality task, highlighting the crucial role that equality reasoning has played in our understanding of how artificial and biological neural networks relate to modular symbolic algorithms. We then defined a tree-structured algorithm that solves the task, and a neural network that was trained to implement the tree structured algorithm [Geiger et al., 2022]. Here, we show how causal abstraction theory can illuminate precisely what it means for this implementation relationship to hold between the network and algorithm.

§.§ An Alignment Between the Algorithm and the Neural Network

Examining the neural network parameters in Figure <ref> reveals no obvious relationship between the network $\mathcal{N}$ and the algorithm $\mathcal{A}$. However, the network  $\mathcal{N}$ was explicitly constructed to be abstracted by the algorithm $\mathcal{A}$ under the alignment written formally below and depicted visually in Figure <ref>.

$\Pi_O = \{O_{\mathsf{True}}, O_{\mathsf{False}}\}$     $\Pi_{X_k} = \{R_{2k-1}, R_{2k}\}$     $\Pi_{V_1} = \{H_{(2,2)},H_{(2,3)}\}$

$\Pi_{V_2} = \{H_{(2,7)},H_{(2,8)}\}$     2c$\Pi_{\bot} = \mathbf{V} \setminus (\Pi_O \cup \Pi_{V_1} \cup \Pi_{V_2} \cup \Pi_{X_1} \cup \Pi_{X_2} \cup \Pi_{X_3} \cup \Pi_{X_4})$

$\tau_{O}(o_{\mathsf{True}},o_{\mathsf{False}} ) = \begin{cases}
\mathsf{True}  & o_{\mathsf{True}} >o_{\mathsf{False}} \\
\mathsf{False} & \text{otherwise} \\
\end{cases}$     $\tau_{X_k}(r_{2k-1}, r_{2k}) = \begin{cases}
{\large \square}& (r_{2k-1}, r_{2k}) = \mathbf{r}_{\large \square} \\
\vspace{-2pt}
{\large \pentagon}& (r_{2k-1}, r_{2k}) = \mathbf{r}_{\large \pentagon} \\
{\large \bigtriangleup}& (r_{2k-1}, r_{2k}) = \mathbf{r}_{\large \bigtriangleup} \\
\vspace{-1pt}
\textsf{Undefined} & \text{otherwise}\\
\end{cases}$

If there exists $ \mathbf{r} \in \{\mathbf{r}_{\large \pentagon}, \mathbf{r}_{\large \square}, \mathbf{r}_{\large \bigtriangleup} \}^{4}$ such that $\{h_{(2,2)}, h_{(2,3)}\} = \mathsf{Proj}(\sols(\mathcal{N}_{\mathbf{r}}), \{ H_{(2,2)}, H_{(2,3)}\} )$,  define

\[\tau_{V_1}(h_{(2,2)}, h_{(2,3)}) = \mathsf{Proj}(\sols(\mathcal{A}_{\tau(\mathbf{r}})), V_1)\]

If there exists $\mathbf{r} \in \{\mathbf{r}_{\large \pentagon}, \mathbf{r}_{\large \square}, \mathbf{r}_{\large \bigtriangleup} \}^{4}$ such that $\{h_{(2,7)}, h_{(2,8)}\} = \mathsf{Proj}(\sols(\mathcal{N}_{\mathbf{r}}), \{ H_{(2,7)}, H_{(2,8)}\}$, define

\[\tau_{V_2}(h_{(2,7)}, h_{(2,8)}) = \mathsf{Proj}(\sols(\mathcal{A}_{\tau(\mathbf{r}})), V_2)\]

Otherwise leave $\tau_{V_1}$ and $\tau_{V_2}$ undefined.

Consider an intervention $\mathbf{i}$ in the domain of $\tau$. We have a fixed alignment for the input and output neurons, where $\mathbf{i}$ can have output values from the real numbers and input values from $\{\mathbf{r}_{\large \pentagon}, \mathbf{r}_{\large \square}, \mathbf{r}_{\large \bigtriangleup} \}^{4}$. The intermediate neurons are assigned high-level alignment by stipulation; $\mathbf{i}$ can only have intermediate variables that are realized on some input intervention $\mathcal{M}_{\mathbf{x}}$ for $\mathbf{x} \in \{\mathbf{r}_{\large \pentagon}, \mathbf{r}_{\large \square}, \mathbf{r}_{\large \bigtriangleup} \}^{4}$. Constructive abstraction will hold only if these stipulative alignments to intermediate variables do not violate the causal laws of $\mathcal{A}$.

§.§ The Algorithm Abstracts the Neural Network

For our running example, the inputs are a sequence of four shapes from the set $\{\large \bigtriangleup, \square, \pentagon \normalsize\}$. Following Def. <ref>, the domain of $\tau$ is restricted to $3^4$ input interventions, $(3^4)^2$ single-source interchange interventions for  high-level interventions fixing either $V_1$ or $V_2$, and $(3^4)^3$ double-source interchange interventions for high-level interventions fixing both $V_1$ and $V_2$.

This neural network was created using interchange intervention training with the alignment  $\langle \Pi, \tau \rangle$ and the high-level model $\mathcal{A}$, which is why the relation of constructive causal abstraction holds between the high-level model $\mathcal{A}$ and the low-level model $\mathcal{N}$. Equivalently, for all $\mathbf{i} \in \mathsf{Domain}(\tau)$ we have

\begin{eqnarray}\tau\big(\sols\big(\mathcal{N}_{\mathbf{i}}\big)\big)  & = & \sols\big(\mathcal{A}_{{\tau}(\mathbf{i})}\big)\label{eq:networkalg}\end{eqnarray}

We provide code[<https://github.com/atticusg/InterchangeInterventions>] for the interchange intervention training and verifying that the network $\mathcal{N}$ is abstracted by the algorithm $\mathcal{A}$. In Figure <ref>, we depict an aligned interchange intervention performed on $\mathcal{A}$ and $\mathcal{N}$ with the base input $(\large \pentagon, \pentagon, \bigtriangleup, \square \normalsize)$ and a single source input $(\large \square, \pentagon, \bigtriangleup, \bigtriangleup)$. The central insight is that the network and algorithm have the same counterfactual behavior.

Crucially, while our example contains a neural network trained to implement a symbolic algorithm, in practice researchers use interchange intervention analysis to investigate whether a network implements an algorithm.

§.§ The Algorithm can be Constructed from the Neural Network

Our constructive characterization provides a new lens through which to view this result. The network $\mathcal{N}$ can be transformed into the algorithm $\mathcal{A}$ through a marginalization, variable merge, and value merge. We visually depict the algorithm $\mathcal{A}$ being constructed from the network $\mathcal{N}$ in Figure <ref>.

§ APPROXIMATE ABSTRACTION AND INTERCHANGE INTERVENTION ACCURACY

Constructive causal abstraction is an all-or-nothing notion. Either the network and algorithm have identical  (counterfactual) behavior for all inputs and interchange interventions, or they do not. This binary concept prevents us from having a graded notion of faithful interpretation that is more useful in practice [Jacovi and Goldberg, 2020]. Early applications of interchange interventions avoided this theoretical limitation by finding subsets of the input space on which the causal abstraction holds in full force [Geiger et al., 2020, Geiger et al., 2021]. Later [Geiger et al., 2022] proposed interchange intervention accuracy, which is simply the proportion of interchange interventions where the neural network and high-level algorithm have the same input–output behavior.

Consider a neural network $\mathcal{N}$ aligned to a high-level algorithm $\mathcal{A}$ with a partition $\Pi$ and fixed input and output alignment $\tau$. As in Def. <ref>, define the intermediate semantics of $\tau$ through stipulation and restrict the domain of $\tau$ to multi-source interchange interventions. We define the interchange intervention accuracy as follows:[Depending on the application, it may be useful to consider distributions over interchange interventions other than the uniform distribution. ]

\[\mathsf{IIA}(\mathcal{N}, \mathcal{A}, \tau) = \mathbb{E}_{\mathbf{i} \sim \mathsf{Uniform}(\dom(\tau))}\big[  \mathbbm{1}[\tau(\mathsf{Proj}(\mathcal{N}_{\mathbf{i}}, \mathbf{X}_{\mathcal{L}}^{\text{Out}})) = \mathsf{Proj}(\mathcal{A}_{\tau(\mathbf{i})} , \mathbf{X}_{H}^{\text{Out}})] \big ].\]

Interchange intervention accuracy is equivalent to behavioral accuracy if we further restrict $\tau$ to be defined only on interchange interventions where base and sources are all the same single input.

Interchange intervention accuracy is an intuitive metric, but it
cannot be analyzed in terms of any existing notion of approximate abstraction. We define a new notion of approximate abstraction, $\alpha$-on-average constructive abstraction, that is tightly connected to interchange intervention accuracy:
Fix a metric $\mathsf{Distance}_{\mathcal{H}}$ between high-level total settings.
An alignment $\langle \Pi, \tau \rangle$ between $\mathcal{L}$ and $\mathcal{H}$  is $\alpha$-on-average causally consistent if the following holds for all $\mathbf{i} \in \mathsf{Domain}(\tau)$ such that $\sols(\mathcal{L}_{\mathbf{i}})$ is non-empty:

\[ \mathbb{E}_{\mathbf{i} \sim \mathsf{Uniform}(\mathsf{Domain}(\tau))}[\mathsf{Distance}_{\mathcal{H}}\big( \tau(\sols(\mathcal{L}_{\mathbf{i}}) ), \sols(\mathcal{H}_{\tau(\mathbf{i})} )\big) ] \leq  \alpha \label{eq:acc}\]

That is, on average, the high-level intervention ${\tau}(\mathbf{i})$ results in a high-level total setting that is within $\alpha$ of the high-level total setting mapped to from the low-level total setting resulting from the low-level intervention $\mathbf{i}$.

We thus arrive at a definition of approximate causal abstraction:

Fix a metric $\mathsf{Distance}_{\mathcal{H}}$ on high-level total settings.
$\mathcal{H}$ is a $\alpha$-on-average constructive abstraction of $\mathcal{L}$  if there is an alignment $\langle \Pi, \tau \rangle$ between $\mathcal{L}$ and $\mathcal{H}$ satisfying the $\alpha$-on-average causal consistency condition.

Note that causal consistency (Def. <ref>) arises as a special case of Def. <ref> where $\alpha=0$, and thus constructive abstraction is likewise the limit case of $0$-on-average constructive abstraction.

A different notion of approximate abstraction is explored in work by [Beckers et al., 2019] where the maximum distance between high-level total settings is used rather than the average distance. Their notion of $d_{\text{max}}$-$\alpha$-approximation amounts to a kind of `worst case' analysis, which may be more appropriate in some contexts. The most general definition subsuming both $d_{\text{max}}$-$\alpha$-approximation and $\alpha$-on-average approximation would allow arbitrary distributions on interventions.

The notion of $\alpha$-on-average abstraction theoretically grounds the metric of interchange intervention accuracy (as defined by Geiger et al., 2022) in causal abstraction.

If a neural network $\mathcal{N}$ is aligned with $\langle \Pi, \tau \rangle$ to a high-level algorithm $\mathcal{A}$ and we have an interchange intervention accuracy of $\mathsf{IIA}(\mathcal{N}, \mathcal{A}, \langle \tau, \Pi \rangle) = \alpha$, then $\mathcal{A}$ is a $\alpha$-on-average constructive abstraction of $\mathcal{N}$ with $\mathsf{Distance}_{\mathcal{H}}(\mathbf{v}_{\mathcal{H}}, \mathbf{v}'_{\mathcal{H}})$ being equal to $0$ if $\mathbf{v}_{\mathcal{H}}$ and $\mathbf{v}'_{\mathcal{H}}$ are equal and $1$ otherwise.

§ XAI METHODS GROUNDED IN CAUSAL ABSTRACTION

To support the claim that causal abstraction can be taken as a general theoretical foundation for XAI, we will show that many popular XAI methods can be viewed as special cases of causal abstraction analysis.

Explanation methods purely grounded in the behavior of a model can be directly interpreted in causal terms when we interpret the act of providing an input to a model as an intervention to the input variable. Methods that learn interpretable models like LIME or SHAP only consider input–output behavior, so causal abstraction can accurately capture these simple methods using two-variable causal models. Causal mediation analysis and amnesiac probing intervene on a single intermediate neural representation, and can be captured with a three-variable high-level model. Only circuit-based explanations require a high-level model with more than three variables. Finally, integrated gradients is not a special case of causal abstraction analysis, but it can be used to compute interchange interventions and therefore can be used to conduct causal abstraction analysis.

Causal abstraction can easily capture a variety of popular XAI methods with high-level models containing no more than three variables. However, the general theory supports arbitrarily complex high-level causal models, highlighting exciting new directions for research. An one example, future XAI methods leveraging multi-source interchange interventions might be able to practically operationalize complex hypotheses such as whether a transformer model trained to sort numbers implements bubble sort (see sec:future).

§.§ LIME: Behavioral Fidelity as Approximate Abstraction by a Two-Variable Chain

LIME is a popular XAI method that learns an interpretable model $\mathcal{A}$ that locally approximates the behavior of an uninterpretable model $\mathcal{N}$ on a neighborhood of inputs $\psi_{\mathbf{x}}$ around a single input $\mathbf{x}$. The fidelity of the explainer model $\mathcal{A}$ is a measure of how the input-output behavior of $\mathcal{A}$ differs from that of $\mathcal{N}$ on the neighborhood of inputs $\psi_{\mathbf{x}}$.

LIME Fidelity of Interpretable Model

Let $\mathcal{A}$ and $\mathcal{N}$ be models with identical input and output spaces. Define $\mathsf{Distance}(\cdot, \cdot)$ to be a function that computes the distance between outputs.

\[\mathsf{LIME}(\mathcal{N}, \mathcal{A}, \psi_{\mathbf{x}}) = \frac{1}{|\psi_{\mathbf{x}}|}\sum_{\mathbf{x}' \in \psi_{\mathbf{x}}}\mathsf{Distance}(\mathsf{Proj}(\mathsf{Solve}(\mathcal{N}_{\mathbf{x}}), X_{\text{Out}}^{\mathcal{N}}), \mathsf{Proj}(\mathsf{Solve}(\mathcal{A}_{\mathbf{x}}), X_{\text{Out}}^{\mathcal{A}}))\]

The uninterpretable model $\mathcal{N}$ will often be a deep learning model with fully-connected causal structure as shown below.

[node distance = 2.0cm]
nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[->, thick, dashed]
[] (In1) $X^{\mathcal{N}}_{\text{In}}$;
[nnNode, above right = 0.25cm and 0.3cm of In1] (n11) $N_{11}$;
[nnNode, below= 0.05cm of n11] (n21) $N_{21}$;
[nnNode, below= 0.3cm of n11] (n31) $\large \vdots$;
[nnNode, below= 0.95cm of n11] (nm1) $N_{d1}$;

[nnNode, right = 0.5cm of n11] (n12) $N_{12}$;
[nnNode, below right = 0.05cm and 0.5cm of n11] (n22) $N_{22}$;
[nnNode, below right= 0.3cm and 0.65cm  of n11] (n32) $\large \vdots$;
[nnNode, below right= 0.95cm and 0.5cm  of n11] (nm2) $N_{d2}$;

[nnNode, right = 1.25cm of n11] (n13) $\large \hdots$;
[nnNode, below right = 0.1cm and 1.25cm of n11] (n23) $\large \hdots$;
[nnNode, below right = 1cm and 1.25cm of n11] (n23) $\large \hdots$;

[nnNode, right = 2cm of n11] (n14) $N_{1l}$;
[nnNode, below right = 0.05cm and 2cm of n11] (n24) $N_{2l}$;
[nnNode, below right= 0.3cm and 2.15cm  of n11] (n34) $\large \vdots$;
[nnNode, below right= 0.95cm and 2cm  of n11] (nm4) $N_{dl}$;

[nnNode, right = 4.5cm of In1] (Out1) $X^{\mathcal{N}}_{\text{Out}}$;

[nnline] (In1.east) – (n11.west);
[nnline] (In1.east) – (n21.west);
[nnline] (In1.east) – (nm1.west);

[nnline] (n11.east) – (n12.west);
[nnline] (n11.east) – (n22.west);
[nnline] (n11.east) – (nm2.west);

[nnline] (n21.east) – (n12.west);
[nnline] (n21.east) – (n22.west);
[nnline] (n21.east) – (nm2.west);

[nnline] (nm1.east) – (n12.west);
[nnline] (nm1.east) – (n22.west);
[nnline] (nm1.east) – (nm2.west);

[nnline] (n14.east) – (Out1);
[nnline] (n24.east) – (Out1);
[nnline] (nm4.east) – (Out1);

The interpretable model $\mathcal{A}$ will often also have rich internal causal structure such as a decision tree model. For instance:

[node distance = 2.0cm]
nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[->, thick, dashed]
[] (In1) $X^{\mathcal{A}}_{\text{In}}$;
[nnNode, above right = 0.15cm and 0.3cm of In1] (a1) $A_{1}$;
[nnNode, right= 0.3cm of In1] (a2) $A_{2}$;
[nnNode, below right = 0.15cm and 0.3cm of In1] (a3) $A_{3}$;

[nnNode, above right = -0.1cm and 2cm of In1] (a4) $A_{4}$;

[nnNode, right = 4.5cm of In1] (Out1) $X^{\mathcal{A}}_{\text{Out}}$;

[nnline] (In1.east) – (a1.west);
[nnline] (In1.east) – (a2.west);
[nnline] (In1.east) – (a3.west);

[nnline] (a1.east) – (a4);
[nnline] (a2.east) – (a4);

[nnline] (a4.east) – (Out1);
[nnline] (a3.east) – (Out1);

However, LIME only guarantees a correspondence between the input–output behaviors of the interpretable and uninterpretable models. Therefore, representing both $\mathcal{N}$ and $\mathcal{A}$ as a two-variable causal models connecting inputs to outputs is sufficient to describe the fidelity measure in LIME.

LIME Fidelity is Approximate Abstraction Between Two Variable Chains

Let $\mathcal{N}$ and $\mathcal{A}$ each be causal models with variables $X^{\mathcal{A}}_{\text{In}},X^{\mathcal{A}}_{\text{Out}}$ and $X^{\mathcal{N}}_{\text{In}},X^{\mathcal{N}}_{\text{Out}}$, respectively, such that $\mathsf{Val}(X^{\mathcal{A}}_{\text{In}}) = \mathsf{Val}(X^{\mathcal{N}}_{\text{In}}) = \psi_{\mathbf{x}}$ and  $\mathsf{Val}(X^{\mathcal{A}}_{\text{Out}}) =  \mathsf{Val}(X^{\mathcal{N}}_{\text{Out}})$. The following two statements are equivalent:

* $\mathsf{LIME}(\mathcal{N},\mathcal{A}, \psi_{\mathbf{x}}) \leq \alpha $

* $\mathcal{A}$ is an $\alpha$-on-average abstraction of $\mathcal{N}$ with distance function $\mathsf{Distance}$ from LIME and an alignment where $ \Pi_{X^{\mathcal{A}}_{\text{In}}} = \{X^{\mathcal{N}}_{\text{In}}\}$, $\Pi_{X^{\mathcal{A}}_{\text{Out}}} = \{X^{\mathcal{N}}_{\text{Out}}\}$, and $\tau$ is the identity function.

[node distance = 2.0cm]
nnline=[->, thick,  opacity=1.0]
cpmline=[-, dashed,  opacity=1.0]
lossline=[->, thick, dashed]
[] (In1) $X^{\mathcal{A}}_{\text{In}}$;

[nnNode, right = 4.5cm of In1] (Out1) $X^{\mathcal{A}}_{\text{Out}}$;

[below = 1cm of In1] (In2) $X^{\mathcal{N}}_{\text{In}}$;

[nnNode, right = 4.5cm of In2] (Out2) $X^{\mathcal{N}}_{\text{Out}}$;

[nnline] (In1.east) – (Out1.west);
[nnline] (In2.east) – (Out2.west);
[cpmline] (In1.south) – (In2.north);
[cpmline] (Out1.south) – (Out2.north);

§.§ Causal Effect Estimation as Abstraction by a Two-Variable Chain

What was the effect of some real-world concept on the prediction of deep learning model? The CEBaB benchmark [Abraham et al., 2022] presents the estimation of  such causal effects as an XAI task. Specifically, CEBaB evaluates explainer models on their ability to estimate the causal effect of changing the quality of food, service, ambiance, and noise in a real-world dining experience on the prediction of a sentiment classifier given a restaurant review as input data. We represent the real-world data generating process and the neural network with a single causal model $\mathcal{M}_{\text{CEBaB}}$. The real-world concepts $C_{\text{service}}$, $C_{\text{noise}}$, $C_{\text{food}}$, and $C_{\text{ambiance}}$ can take on three values $+$, $-$, and $\textsf{Unknown}$, the input data $X_{\text{In}}$, prediction output $X_{\text{Out}}$, and neural representations $N_{ij}$ can take on real-valued vectors, and the two exogenous variables $U$ and $V$ that represent the real world noise.[Crucially, the models in causal effect estimation are probabilistic models. See Section <ref> for details.]

[node distance = 2.0cm]
nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[->, thick, dashed]
[] (In1) $X_{\text{In}}$;
[nnNode, above right = 0.25cm and 0.3cm of In1] (n11) $N_{11}$;
[nnNode, below= 0.05cm of n11] (n21) $N_{21}$;
[nnNode, below= 0.3cm of n11] (n31) $\large \vdots$;
[nnNode, below= 0.95cm of n11] (nm1) $N_{d1}$;

[nnNode, right = 0.5cm of n11] (n12) $N_{12}$;
[nnNode, below right = 0.05cm and 0.5cm of n11] (n22) $N_{22}$;
[nnNode, below right= 0.3cm and 0.65cm  of n11] (n32) $\large \vdots$;
[nnNode, below right= 0.95cm and 0.5cm  of n11] (nm2) $N_{d2}$;

[nnNode, right = 1.25cm of n11] (n13) $\large \hdots$;
[nnNode, below right = 0.1cm and 1.25cm of n11] (n23) $\large \hdots$;
[nnNode, below right = 1cm and 1.25cm of n11] (n23) $\large \hdots$;

[nnNode, right = 2cm of n11] (n14) $N_{1l}$;
[nnNode, below right = 0.05cm and 2cm of n11] (n24) $N_{2l}$;
[nnNode, below right= 0.3cm and 2.15cm  of n11] (n34) $\large \vdots$;
[nnNode, below right= 0.95cm and 2cm  of n11] (nm4) $N_{dl}$;

[nnNode, right = 4.5cm of In1] (Out1) $X_{\text{Out}}$;

[nnline] (In1.east) – (n11.west);
[nnline] (In1.east) – (n21.west);
[nnline] (In1.east) – (nm1.west);

[nnline] (n11.east) – (n12.west);
[nnline] (n11.east) – (n22.west);
[nnline] (n11.east) – (nm2.west);

[nnline] (n21.east) – (n12.west);
[nnline] (n21.east) – (n22.west);
[nnline] (n21.east) – (nm2.west);

[nnline] (nm1.east) – (n12.west);
[nnline] (nm1.east) – (n22.west);
[nnline] (nm1.east) – (nm2.west);

[nnline] (n14.east) – (Out1);
[nnline] (n24.east) – (Out1);
[nnline] (nm4.east) – (Out1);
[above =0.75cm of In1] (V) $V$;

[above left =-0.1cm and 1cm of In1] (C2) $C_{\text{noise}}$;
[below left =-0.1cm and 1cm of In1] (C1) $C_{\text{food}}$;
[above left =0.5cm and 1cm of In1] (C3) $C_{\text{service}}$;
[below left =0.5cm and 1cm of In1] (Ck) $C_{\text{ambiance}}$;

[left =2.75cm of In1] (U) $U$;

[->, line width = 1] (C1.east) – (In1);
[->, line width = 1] (C2.east) – (In1);
[->, line width = 1] (C3.east) – (In1);
[->, line width = 1] (Ck.east) – (In1);
[->, line width = 1, dashed] (U) – (C1.west);
[->, line width = 1,dashed] (U) – (C2.west);
[->, line width = 1,dashed] (U) – (C3.west);
[->, line width = 1,dashed] (V) – (In1);
[->, line width = 1,dashed] (U) – (Ck.west);

If we are interested in the causal effect of food quality on model output, then we can marginalize away every variable other than the real-world concept $C_{\text{food}}$ and the neural network output $X_{\text{Out}}$ to get a causal model with two endogenous variables. This marginalized causal model is a high-level abstraction of $\mathcal{M}_{\text{CEBaB}}$ that contains a single causal mechanism describing how food quality in a dining experience affects the neural network output.

[node distance = 2.0cm]
nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[->, thick, dashed]

[nnNode, right = 4.5cm of In1] (Out1) $X_{\text{Out}}$;

[above =0.75cm of In1] (V) $V$;

[above left =-0.1cm and 1cm of In1] (C2) ;
[below left =-0.1cm and 1cm of In1] (C1) $C_{\text{food}}$;
[above left =0.5cm and 1cm of In1] (C3) ;
[below left =0.5cm and 1cm of In1] (Ck) ;

[left =2.75cm of In1] (U) $U$;

[->, line width = 1] (C1.east) – (Out1);

[->, line width = 1, dashed] (U) – (C1.west);

[->, line width = 1,dashed] (V) – (Out1);

§.§ Causal Mediation As Abstraction by a Three-Variable Chain

Suppose that changing the value of a variable $X$ from $x$ to $x'$ has an effect on a second variable $Y$. Causal mediation analysis determines how this causal effect is mediated by a third intermediate variable $Z$. The fundamental notions to mediation are total, direct, and indirect effects, which can be defined with interchange interventions.

Total, Direct, and Indirect Effects

Suppose we have a causal model $\mathcal{M}$ with sets of variables $\mathbf{X}, \mathbf{Y}, \mathbf{Z} \in \mathcal{M}$ such that addition and subtraction are well-defined on values of $\mathbf{Y}$. The total causal effect of changing the values of $\mathbf{X}$ from $\mathbf{x}$ to $\mathbf{x'}$ on $\mathbf{Y}$ is

\[\mathsf{TotalEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{Y}) = \mathsf{Proj}(\mathcal{M}_{\mathbf{x}'}, \mathbf{Y}) - \mathsf{Proj}(\mathcal{M}_{\mathbf{x}}, \mathbf{Y})\]

The direct causal effect of changing the value of $\mathbf{X}$ from $\mathbf{x}$ to $\mathbf{x}'$ on $\mathbf{Y}$ through mediator $\mathbf{Z}$ is

\[\mathsf{DirectEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{Y}, \mathbf{Z}) = \mathsf{Proj}(\mathcal{M}_{\mathbf{x}'}, \mathbf{Y}) - \mathsf{Proj}(\sols(\mathcal{M}_{\mathsf{IntInv}(\mathbf{x}, \langle \mathbf{x}' \rangle, \langle \mathbf{Z} \rangle)}, \mathbf{Y}) \]

The indirect causal effect of changing the value of $\mathbf{X}$ from $\mathbf{x}$ to $\mathbf{x}'$ on $\mathbf{Y}$ through mediator $\mathbf{Z}$ is

\[\mathsf{IndirectEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{Y}, \mathbf{Z}) = \mathsf{Proj}(\mathcal{M}_{\mathbf{x}'}, \mathbf{Y}) - \mathsf{Proj}(\sols(\mathcal{M}_{\mathsf{IntInv}(\mathbf{x}', \langle \mathbf{x} \rangle, \langle \mathbf{Z} \rangle)}, \mathbf{Y}) \]

This method has been applied to the analysis of neural networks to characterize how the causal effect of inputs on outputs are mediated by intermediate neural representations. A central goal of such research is identifying sets of neurons that completely mediate the causal effect of an input value change on the output. This is equivalent to a simple causal abstraction analysis.

Complete Mediation is Abstraction by a Three Variable Chain

Consider a neural network $\mathcal{M}$ with inputs $\mathbf{x}$ and $\mathbf{x}'$ and a set of intermediate neurons $\mathbf{H}$.
Define $\Pi_A = \mathbf{X}^{\text{In}}$, $\Pi_C = \mathbf{X}^{\text{Out}}$, $\Pi_{B} = \mathbf{H}$, and $\Pi_{\emptyset} = \mathcal{V} \setminus (\mathbf{X}^{\text{In}} \cup \mathbf{H} \cup \mathbf{X}^{\text{Out}})$.
The following two statements are equivalent:

* The variables $\mathbf{H}$ completely mediate the causal effect of changing $\mathbf{x}$ to $\mathbf{x}'$ on the output:

\[\mathsf{IndirectEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{X}^{\text{Out}}, \mathbf{H}) = \mathsf{TotalEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{X}^{\text{Out}})\]

* The high-level causal model $\Pi(\epsilon_{\Pi_{\emptyset}}(\mathcal{N}))$ has a structure such that $A$ is not a child of $C$

[] (x) at (0,0) $A$ ;
[] (y) at (1.5,0) $B$ ;
[] (z) at (3,0) $C$ ;
[->] (x) – (y);
[->] (y) – (z);

Partial Mediation is Approximate Abstraction by a Three Variable Chain

Consider a neural network $\mathbf{H}$ with inputs $\mathbf{x}$ and $\mathbf{x}'$ and a set of intermediate neurons $\mathbf{H}$. Define $\Pi_X = \mathbf{X}^{\text{In}}$, $\Pi_Y = \mathbf{X}^{\text{Out}}$, $\Pi_{Z} = \mathbf{H}$, and $\Pi_{\emptyset} = \mathcal{V} \setminus (\mathbf{X}^{\text{In}} \cup \mathbf{H} \cup \mathbf{X}^{\text{Out}})$.  The following two statements are equivalent:

* On average, the variables $\mathbf{H}$ mediate a changing $\mathbf{x}$ to $\mathbf{x}'$ by at least $\alpha$:

\[\mathbb{E}_{\mathbf{x},\mathbf{x}' \sim \mathsf{Uniform}(\mathsf{Vals}(\mathsf{X}^{\text{In}}))}[ \mathsf{DirectEffect}(\mathcal{M},\mathbf{x} \to \mathbf{x}', \mathbf{X}^{\text{Out}}, \mathbf{H})] \leq \alpha \]

* If you edit the causal mechanism for the variable $Y$ in $\Pi(\epsilon_{\Pi_{\emptyset}}(\mathcal{N}))$ to be

\[F'_{Y}(z, x) = \mathbb{E}_{x' \in \mathsf{Vals}(X)}[F_{Y}(z,x'))]\]

the resulting model is an $\alpha$-on-average abstraction of $\mathcal{N}$ where the distance function is

\[\mathsf{Distance}(\mathbf{v}, \mathbf{v}') = \mathsf{Proj}(\mathbf{v}, Y ) - \mathsf{Proj}(\mathbf{v}', Y ).\]

§.§ Iterative Nullspace Projection As Abstraction by a Three Variable Chain

Iterative nullspace projection investigates how removing a concept $C$ from a target hidden representation $\mathbf{H}$ of a neural network $\mathcal{N}$ affects the predictions of $\mathcal{N}$. Specifically, the hidden representation is intervened upon to `remove' a concept $C$ by projecting the representation onto the nullspaces of linear probes $\mathcal{L}_1, \dots, \mathcal{L}_k$ trained to predict the value of $C$. We define $\mathcal{L}(\cdot)$ to be the function computing this iterated projection. The idea is to conclude that $\mathcal{N}$ makes use of the concept $C$ if the performance of $\mathcal{N}$ on a task degrades under these interventions.

We can model iterative nullspace projection as abstraction by a three-variable causal model. Say there is a high-level model $\mathcal{A}$ with input variable $X^{\mathcal{A}}_{\text{In}}$, output variable ${X}^{\mathcal{A}}_{\text{Out}}$, and a binary variable $L$ that indicates whether information has been removed. When $l=1$, the causal mechanism $f_{{X}^{\mathcal{A}}_{\text{Out}}}({x}_{\text{In}}, l)$ defines the desired input–output behavior for a neural model. When $l=0$, the causal mechanism $f_{{X}^{\mathcal{A}}_{\text{Out}}}({x}_{\text{In}}, l)$ defines degraded input-output behavior.

Define the alignment between the neural model as follows. Let $\Pi_{X_{\text{In}}}^{\mathcal{A}} = \{X_{\text{In}}^{\mathcal{N}}\}$, $\Pi_{X_{\text{Out}}}^{\mathcal{A}} = \{X_{\text{Out}}^{\mathcal{N}}\}$, $\Pi_{L} = \mathbf{H}$. Furthermore, let $\tau_{X_{\text{In}}}^{\mathcal{A}}$ and $\tau_{X_{\text{Out}}}^{\mathcal{A}}$ be identify functions and define

\[\tau_{L}(\mathbf{h}) = \begin{cases}
0 & \exists x_{\text{In}} \ \mathbf{h} = \mathcal{L}(\mathsf{Proj}(\mathcal{N}_{x_{\text{In}}}, \mathbf{H})) \\
1 & \exists x_{\text{In}} \ \mathbf{h} =\mathsf{Proj}(\mathcal{N}_{x_{\text{In}}}, \{N_{12}, N_{22}\})\\
\end{cases}\]

The high-level causal model $\mathcal{A}$ is an abstraction of the low-level neural model $\mathcal{N}$ under alignment $\langle \Pi, \tau \rangle$ exactly when the iterative nullspace projection removing the concept $C$ resulted in degraded performance defined by $f_{{X}^{\mathcal{A}}_{\text{Out}}}$. We depict an example of this below, where $\mathbf{H} = \{N_{12}, N_{22} \}$.

[] (x) at (0,0) ${X}^{\mathcal{A}}_{\text{In}}$ ;
[] (y) at (2.45,0) $L$ ;
[] (z) at (5.5,0) ${X}^{\mathcal{A}}_{\text{Out}}$ ;
[->] (x) – (y);
[->] (x) to[out=45, in=135] (z);
[->] (y) – (z);
nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[thick, dashed]
[below = 1.75cm of x] (In1) $X^{\mathcal{N}}_{\text{In}}$;
[below = 1.7cm of x, circle, draw, minimum size=0.8cm] (loel) ;
[nnNode, above right = 0.25cm and 0.3cm of In1] (n11) $N_{11}$;
[nnNode, below= 0.05cm of n11] (n21) $N_{21}$;
[nnNode, below= 0.3cm of n11] (n31) $\large \vdots$;
[nnNode, below= 0.95cm of n11] (nm1) $N_{d1}$;

[nnNode, right = 0.5cm of n11] (n12) $N_{12}$;

[draw, ellipse, minimum width=0.8cm, minimum height=1.25cm, below right = -0.4cm and 0.63cm of n11] (pi) ;

[nnNode, below right = 0.05cm and 0.5cm of n11] (n22) $N_{22}$;
[nnNode, below right= 0.3cm and 0.65cm  of n11] (n32) $\large \vdots$;
[nnNode, below right= 0.95cm and 0.5cm  of n11] (nm2) $N_{d2}$;

[nnNode, right = 1.25cm of n11] (n13) $\large \hdots$;
[nnNode, below right = 0.1cm and 1.25cm of n11] (n23) $\large \hdots$;
[nnNode, below right = 1cm and 1.25cm of n11] (n23) $\large \hdots$;

[nnNode, right = 2cm of n11] (n14) $N_{1l}$;
[nnNode, below right = 0.05cm and 2cm of n11] (n24) $N_{2l}$;
[nnNode, below right= 0.3cm and 2.15cm  of n11] (n34) $\large \vdots$;
[nnNode, below right= 0.95cm and 2cm  of n11] (nm4) $N_{dl}$;

[nnNode, right = 4.5cm of In1] (Out1) $X^{\mathcal{N}}_{\text{Out}}$;
[nnNode, right = 4.6cm of In1, circle, draw, minimum size=0.83cm] (crum) ;

[nnline] (In1.east) – (n11.west);
[nnline] (In1.east) – (n21.west);
[nnline] (In1.east) – (nm1.west);

[nnline] (n11.east) – (n12.west);
[nnline] (n11.east) – (n22.west);
[nnline] (n11.east) – (nm2.west);

[nnline] (n21.east) – (n12.west);
[nnline] (n21.east) – (n22.west);
[nnline] (n21.east) – (nm2.west);

[nnline] (nm1.east) – (n12.west);
[nnline] (nm1.east) – (n22.west);
[nnline] (nm1.east) – (nm2.west);

[nnline] (n14.east) – (Out1);
[nnline] (n24.east) – (Out1);
[nnline] (nm4.east) – (Out1);

[lossline] (x) – (loel);
[lossline] (y) – (pi);
[lossline] (z) – (crum);

Observe that the high-level model $\mathcal{A}$ does not have a variable encoding the concept $C$ and the values it might take on. Iterative nullspace projection attempts to determine whether a concept is used by a model, not characterize how that concept is used.

Furthermore, iterative nullspace projection is not guaranteed to be a minimal intervention that removes any and all information about $C$, and therefore cannot causally implicate $C$ in model behavior without further assumptions. First, projecting onto the nullspace of probes trained to predict the value of the concept $C$ is an intervention that removes information about the value $C$ that is linearly accessible. Deep learning models are highly non-linear, and therefore can make decisions using information that is not linearly accessible.  Second, linear probes trained on $C$ may also target other concepts that are correlated with $C$.

These are not fatal flaws of the method. [Elazar et al., 2022] present several control tasks that may mitigate these concerns, and [Lovering and Pavlick, 2022] work around the presence of non-linear information about $C$ by using a linear prediction function after the intervention site in the model they analyze.

§.§ Operationalizing Circuit-Based Explanations with Causal Abstraction

Two fundamental claims made by [Olah et al., 2020] are that (1) linear combinations of neural activations encode high-level concept(s) (e.g., curved lines or object orientation), and (2) the `circuits' defined by the model weights connecting neural representations encode meaningful algorithms over high-level concepts. Causal abstraction helps shed additional lighton these claims, where a low-level causal model encodes neural representations and circuits, and a high-level causal model encodes high-level concepts and meaningful algorithms.

Consider the following example of a circuit-based explanation with a neuron. Suppose we have a neural network $\mathcal{N}$ that takes in an image $x_{\text{In}}$ and outputs three binary labels $x_{\text{cat}}$, $x_{\text{dog}}$, and $x_{\text{car}}$ that indicate whether the image contain a cat, dog, or car.

nnline=[->, thick,  opacity=1.0]
cpmline=[->, thick,  opacity=1.0]
lossline=[thick, dashed]
[] (In1) $X^{\mathcal{N}}_{\text{In}}$;
[nnNode, above right = 0.25cm and 0.3cm of In1] (n11) $N_{11}$;
[nnNode, below= 0.05cm of n11] (n21) $N_{21}$;
[nnNode, below= 0.3cm of n11] (n31) $\large \vdots$;
[nnNode, below= 0.95cm of n11] (nm1) $N_{d1}$;

[nnNode, right = 0.5cm of n11] (n12) $N_{12}$;

[nnNode, below right = 0.05cm and 0.5cm of n11] (n22) $N_{22}$;
[nnNode, below right= 0.3cm and 0.65cm  of n11] (n32) $\large \vdots$;
[nnNode, below right= 0.95cm and 0.5cm  of n11] (nm2) $N_{d2}$;

[nnNode, right = 1.25cm of n11] (n13) $\large \hdots$;
[nnNode, below right = 0.1cm and 1.25cm of n11] (n23) $\large \hdots$;
[nnNode, below right = 1cm and 1.25cm of n11] (n23) $\large \hdots$;

[nnNode, right = 2cm of n11] (n14) $N_{1l}$;
[nnNode, below right = 0.05cm and 2cm of n11] (n24) $N_{2l}$;
[nnNode, below right= 0.3cm and 2.15cm  of n11] (n34) $\large \vdots$;
[nnNode, below right= 0.95cm and 2cm  of n11] (nm4) $N_{dl}$;

[nnNode, above right = 0cm and 4.5cm of In1] (Out1) $X^{\mathcal{N}}_{\text{cat}}$;
[nnNode, right =  4.5cm of In1] (Out2) $X^{\mathcal{N}}_{\text{dog}}$;
[nnNode, below right = 0cm and 4.5cm of In1] (Out3) $X^{\mathcal{N}}_{\text{car}}$;

[nnline] (In1.east) – (n11.west);
[nnline] (In1.east) – (n21.west);
[nnline] (In1.east) – (nm1.west);

[nnline] (n11.east) – (n12.west);
[nnline] (n11.east) – (n22.west);
[nnline] (n11.east) – (nm2.west);

[nnline] (n21.east) – (n12.west);
[nnline] (n21.east) – (n22.west);
[nnline] (n21.east) – (nm2.west);

[nnline] (nm1.east) – (n12.west);
[nnline] (nm1.east) – (n22.west);
[nnline] (nm1.east) – (nm2.west);

[nnline] (n14.east) – (Out1.west);
[nnline] (n24.east) – (Out1.west);
[nnline] (nm4.east) – (Out1.west);

[nnline] (n14.east) – (Out2.west);
[nnline] (n24.east) – (Out2.west);
[nnline] (nm4.east) – (Out2.west);

[nnline] (n14.east) – (Out3.west);
[nnline] (n24.east) – (Out3.west);
[nnline] (nm4.east) – (Out3.west);