# Interpretability is in the Mind of the Beholder: A Causal Framework for
Human-interpretable Representation Learning

Emanuele Marconato
DISI, University of Trento, Italy
DI, University of Pisa, Italy
<EMAIL_ADDRESS>
&Andrea Passerini
DISI, University of Trento, Italy
<EMAIL_ADDRESS>
&Stefano Teso
CIMeC and DISI, University of Trento, Italy
<EMAIL_ADDRESS>

(September 10, 2023)

###### Abstract

Focus in Explainable AI is shifting from explanations defined in terms of low-
level elements, such as input features, to explanations encoded in terms of
interpretable concepts learned from data. How to reliably acquire such
concepts is, however, still fundamentally unclear. An agreed-upon notion of
concept interpretability is missing, with the result that concepts used by
both post-hoc explainers and concept-based neural networks are acquired
through a variety of mutually incompatible strategies. Critically, most of
these neglect the human side of the problem: a representation is
understandable only insofar as it can be understood by the human at the
receiving end. The key challenge in Human-interpretable Representation
Learning (hrl) is how to model and operationalize this human element. In this
work, we propose a mathematical framework for acquiring interpretable
representations suitable for both post-hoc explainers and concept-based neural
networks. Our formalization of hrl builds on recent advances in causal
representation learning and explicitly models a human stakeholder as an
external observer. This allows us derive a principled notion of alignment
between the machine’s representation and the vocabulary of concepts understood
by the human. In doing so, we link alignment and interpretability through a
simple and intuitive name transfer game, and clarify the relationship between
alignment and a well-known property of representations, namely disentanglment.
We also show that alignment is linked to the issue of undesirable correlations
among concepts, also known as concept leakage, and to content-style
separation, all through a general information-theoretic reformulation of these
properties. Our conceptualization aims to bridge the gap between the human and
algorithmic sides of interpretability and establish a stepping stone for new
research on human-interpretable representations.

_Keywords_ explainable AI $\cdot$ causal representation learning $\cdot$
alignment $\cdot$ disentanglement $\cdot$ causal abstractions $\cdot$ concept
leakage

##  1 Introduction

The field of Explainable AI (XAI) has developed a wealth of attribution
techniques for unearthing the reasons behind the decisions of black-box
machine learning models (Guidotti et al., 2018). Traditionally, explaining a
prediction involves identifying and presenting those low-level atomic elements
– like input variables (Štrumbelj and Kononenko, 2014; Ribeiro et al., 2016)
and training examples (Kim et al., 2016; Koh and Liang, 2017) – that are
responsible for said prediction. Explanations output by white-box models, such
as sparse linear classifiers (Ustun and Rudin, 2016) and rule-based predictors
(Wang et al., 2017), follow the same general setup. These atomic elements,
however, are not very expressive and, as such, can be ambiguous (Rudin, 2019).
To see this, consider an image of a red sports car that is tagged as
“positive” by a black-box predictor. In this example, a saliency map would
highlight those pixels that are most responsible for this prediction: these do
not say whether the prediction depends on the image containing a “car”, on the
car being “red”, or on the car being “sporty”. As a consequence, it is
impossible to understand what the model is “thinking” and how it would behave
on other images based on this explanation alone (Teso et al., 2023).

This is why focus in XAI has recently shifted toward explanations expressed in
terms of higher-level symbolic representations, or concepts for short. These
promise to ensure explanations are rich enough they can capture the machine’s
reasoning patterns, while being expressed in terms that can be naturally
understood by stakeholders (Rudin, 2019; Kambhampati et al., 2022).

This trend initially emerged with (post-hoc) concept-based explainers (CBEs)
like TCAV (Kim et al., 2018) and Net2Vec (Fong and Vedaldi, 2018), among
others (Ghorbani et al., 2019; Zhang et al., 2021; Fel et al., 2023a), which
match the latent space of a deep neural network to a vocabulary of pre-trained
concept detectors.111The idea of using higher-level concepts was foreshadowed
in the original LIME paper (Ribeiro et al., 2016). These were quickly followed
by a variety of concept-based models (CBMs) – including Self-Explainable
Neural Networks (Alvarez-Melis and Jaakkola, 2018), Part-Prototype Networks
(Chen et al., 2019), Concept-Bottleneck Models (Koh et al., 2020), GlanceNets
(Marconato et al., 2022), and Concept Embedding Models (Espinosa Zarlenga et
al., 2022) – that support representation learning while retaining
interpretability. Specifically, these approaches learn a neural mapping from
inputs to concepts, and then leverage the latter for both computing
predictions – in a simulatable manner (Lipton, 2018) – and providing ante-hoc
explanations thereof. See (Schwalbe, 2022) for a review. Since concepts act as
a bottleneck through which all information necessary for inference must flow,
CBMs hold the promise of avoiding the lack of faithfulness typical of post-hoc
techniques, while enabling a number of useful operations such as interventions
(Koh et al., 2020) and debugging (Stammer et al., 2021; Bontempelli et al.,
2023) using concepts as a human-friendly interface.

###  1.1 Limitations of Existing Works

The promise of conceptual explanations rests on the assumption that learned
concepts are themselves interpretable. This begs the question: what does it
mean for a vocabulary of concepts to be interpretable?

Researchers have proposed a variety of practical strategies to encourage the
interpretability of the learned concepts, but no consistent recipe. Some CBMs
constrain their representations according to intuitive heuristics, such as
similarity to concrete training examples (Chen et al., 2019) or activation
sparsity (Alvarez-Melis and Jaakkola, 2018). However, the relationship between
these properties and interpretability is unclear, and unsurprisingly there are
well known cases in which CBMs acquire concepts activating on parts of the
input with no obvious semantics (Hoffmann et al., 2021; Xu-Darme et al.,
2023). A more direct way of controlling the semantics of learned concepts is
to leverage supervision on the concepts themselves, a strategy employed by
both CBEs (Kim et al., 2018) and CBMs (Koh et al., 2020; Chen et al., 2020;
Marconato et al., 2022). Unfortunately, this is no panacea, as doing so cannot
prevent concept leakage (Margeloiu et al., 2021; Mahinpei et al., 2021),
whereby information from a concept “leaks” into another, seemingly unrelated
concept, compromising its meaning.

At the same time, concept quality is either assessed qualitatively in a rather
unsystematic fashion – e.g., by inspecting the concept activations or saliency
maps on a handful of examples – or quantitatively, most often by measuring how
well learned concepts match annotations. This so-called concept accuracy,
however, is insufficient to capture issues like concept leakage.

Besides these complications, existing approaches neglect a critical aspect of
this learning problem: that interpretability is inherently subjective. For
instance, explaining a prediction to a medical doctor requires different
concepts than explaining it to a patient: the notion of “intraepithelial” may
be essential for the former, while being complete gibberish to the latter.
However, even when concept annotations are employed, they are gathered from
offline repositories and as such they may not capture concepts that are
meaningful to a particular expert, or that despite being associated with a
familiar name follow semantics incompatible with those the user attaches to
that name.222Of course, there are exceptions to this rule. These are discussed
in section 6.

###  1.2 Our Contributions

Motivated by these observations, we propose to view interpretability as the
machine’s ability to communicate with a specific human-in-the-loop.
Specifically, we are concerned with the problem of learning conceptual
representations that enable this kind of communication for both post- and
ante-hoc explanations. We call this problem human-interpretable representation
learning, or hrl for short. Successful communication is essential for ensuring
human stakeholders can understand explanations based on the learned concepts
and, in turn, realizing the potential of CBEs and CBMs. This view is
compatible with recent interpretations of the role of symbols in neuro-
symbolic AI (Silver and Mitchell, 2023; Kambhampati et al., 2022). The key
question is how to model this human element in a way that can be actually
operationalized. We aim to fill this gap.

Our first contribution is a conceptual and mathematical model – resting on
techniques from causal representation learning (Schölkopf et al., 2021) – of
hrl that explicitly models the human-in-the-loop.

As a second contribution, we leverage our formalization to develop an
intuitive but sound notion of alignment between the conceptual representation
used by the machine and that of the human observer. Alignment is strictly
related to disentanglement, a property of learned representations frequently
linked to interpretability (Bengio et al., 2013; Higgins et al., 2018), but
also strictly stronger, in the sense that disentanglement alone is
insufficient to ensure concept interpretability. We propose that alignment is
key for evaluating interpretability of both CBEs and CBMs.

Our formalization improves on the work of Marconato et al. (2022) and looks at
three settings of increasing complexity and realism: (i) a simple but non-
trivial setting in which the human’s concepts are disentangled (i.e.,
individual concepts can be changed independently from each other without
interference). (ii) a more general setting in which the human’s concepts are
constrained to be disentangled in blocks; (iii) an unrestricted setting in
which the human concepts can influence each other in arbitrary manners. In
addition, we identify a and previously ignored link between interpretability
of representations and the notion of causal abstraction (Beckers and Halpern,
2019; Beckers et al., 2020; Geiger et al., 2023a).

As a third contribution, we formally show that concept leakage can be viewed
as a lack of disentanglement, and therefore of alignment. This strengthens
existing results and allows to reinterpret previous empirical observations
(Marconato et al., 2022; Lockhart et al., 2022).

As a fourth contribution, we discuss key questions arising from our
mathematical framework, including whether perfect alignment is sufficient and
necessary for interpretability, how to measure it, how to implement it in
representation learning, and how to collect the necessary concept annotations.

###  1.3 Outline

The remainder of this paper is structured as follows. In the next section, we
introduce prerequisite material and then proceed in section 3 to formalize the
problem of human-interpretable representation learning and cast concept
interpretability in terms of alignment between representations. Next, in
section 4 we analyze in depth the notion of alignment in three settings of
increasing complexity and study its relationship to the issue of concept
leakage, and then look at the consequences of our formalization in section 5.
Finally, we discuss related works in section 6 and offer some concluding
remarks in section 7.

##  2 Preliminaries

In the following, we indicate scalar constants $x$ in lower-case, random
variables $X$ in upper case, ordered sets of constants ${\bm{\mathrm{x}}}$ and
random variables ${\bm{\mathrm{X}}}$ in bold typeface, and index sets
$\mathcal{I}$ in calligraphic typeface. We also use the shorthand
$[n]:=\\{1,\ldots,n\\}$. Letting ${\bm{\mathrm{X}}}=(X_{1},\ldots,X_{n})$ and
$\mathcal{I}\subseteq[n]$, we write
${\bm{\mathrm{X}}}_{\mathcal{I}}:=(X_{i}:i\in\mathcal{I})$ to indicate the
ordered subset indexed by $\mathcal{I}$ and
${\bm{\mathrm{X}}}_{-{\mathcal{I}}}:={\bm{\mathrm{X}}}\setminus{\bm{\mathrm{X}}}_{\mathcal{I}}$
to denote its complement, and abbreviate
${\bm{\mathrm{X}}}\setminus\\{X_{i}\\}$ as ${\bm{\mathrm{X}}}_{-i}$.

###  2.1 Structural Causal Models and Interventions

A structural causal model (SCM) is a formal description of the causal
relationships existing between parts of a (stochastic) system (Pearl, 2009;
Peters et al., 2017). Formally, an SCM $\mathfrak{C}$ specifies a set of
structural assignments encoding direct causal relationships between
variables,333As customary, we work with SCMs that are acyclic, causally
sufficient (i.e., there are no external, hidden variables influencing the
system), and causally Markovian (i.e., each variable $X_{i}$ is independent of
its non-descendant given its parents in the SCM) (Pearl, 2009). in the form:

$X_{i}\leftarrow f_{i}(\bm{\mathrm{Pa}}_{i},N_{i})$ (1)

where ${\bm{\mathrm{X}}}=(X_{1},\ldots,X_{n})$ are variables encoding the
state of the system, $\bm{\mathrm{Pa}}_{i}\subseteq{\bm{\mathrm{X}}}$ are the
direct causes of $X_{i}$, and $N_{i}$ are noise terms. Variables without
parents are exogenous, and play the role of inputs to the system, while the
others are endogenous. The full state of the system can be sampled by
propagating the values of the exogenous variables through the structural
assignments in a top-down fashion. SCMs can be viewed as graphs in which nodes
represent variables, arrows represent assignments, and noise variables are
usually suppressed, cf. fig. 1.

Following common practice, we assume the noise terms to be mutually
independent from each other and also independent from the variables not
appearing in the corresponding structural equations, that is, it holds that
$N_{i}\mathrel{\text{\scalebox{1.07}{$\perp\mkern-10.0mu\perp$}}}N_{j}$ for
all $i\neq j$ and
$N_{i}\mathrel{\text{\scalebox{1.07}{$\perp\mkern-10.0mu\perp$}}}X_{j}$ for
all $i,j$. This is equivalent to assuming there are no hidden confounders.
This assumption carries over to all SCMs used throughout the paper.

An SCM $\mathfrak{C}$ describes both a joint distribution
$p({\bm{\mathrm{X}}})=\prod_{i}p(X_{i}\mid\bm{\mathrm{Pa}}_{i})$ and how this
distribution changes upon performing interventions on the system. These are
modifications to the system’s variables and connections performed by an
external observer. Using Pearl’s $do$-operator (Pearl, 2009), (atomic)
interventions can be written as $do(X_{i}\leftarrow x_{i})$, meaning that the
value of the variable $X_{i}$ is forcibly changed to the value $x_{i}$,
regardless of the state of its parents and children. Carrying out an atomic
intervention yields a manipulated SCM identical to $\mathfrak{C}$ except that
all assignments to $X_{i}$ are deleted (i.e., the corresponding links in the
graph disappear) and all occurrences of $X_{i}$ in the resulting SCM are
replaced by the constant $x_{i}$. The resulting manipulated distribution is
$p({\bm{\mathrm{X}}}\mid do(X_{i}\leftarrow
x_{i}))=\mathbbm{1}\\!\left\\{X_{i}=x_{i}\right\\}\cdot\prod_{j\neq i}\
p(X_{j}\mid\bm{\mathrm{Pa}}_{j})$. Non-atomic interventions of the form
$do({\bm{\mathrm{X}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{x}}}_{\mathcal{I}})$
work similarly. Expectations of the form $\mathbb{E}[\cdot\mid
do(X_{j}\leftarrow x_{j})]$ are just regular expectations evaluated with
respect to the manipulated distribution.

###  2.2 Disentanglement

Central to our work is the notion of disentanglement (Higgins et al., 2018;
Eastwood and Williams, 2018; Schölkopf et al., 2021) in both its two
acceptations, namely disentanglement of variables and disentanglement of
representations. We henceforth rely on the causal formalization given by Suter
et al. (2019) and Reddy et al. (2022). We refer the reader to those papers for
more details.

${\bm{\mathrm{C}}}$$\ldots$$G_{1}$$G_{n}$

${\bm{\mathrm{C}}}$$\ldots$$G_{1}$$G_{n}$${\bm{\mathrm{X}}}$$\ldots$$M_{1}$$M_{k}$

Figure 1:  SCMs illustrating two different notions of disentanglement. Left:
The variables ${\bm{\mathrm{G}}}=\\{G_{1},\ldots,G_{n}\\}$ are disentangled.
Right: Typical data generation and encoding process used in deep latent
variable models. The machine representation
${\bm{\mathrm{M}}}=\\{M_{1},\ldots,M_{k}\\}$ is disentangled with respect to
the generative factors ${\bm{\mathrm{G}}}$ if and only if each $M_{j}$ encodes
information about at most one $G_{i}$.

Intuitively, a set of variables ${\bm{\mathrm{G}}}=(G_{1},\ldots,G_{n})$ is
disentangled if the variables can be changed independently from one another.
For instance, if $G_{1}$ represents the “color” of an object and $G_{2}$ its
“shape”, disentanglement of variables implies that changing the object’s color
does not impact its shape. This should hold even if the variables
${\bm{\mathrm{G}}}$ have a common set of parents ${\bm{\mathrm{C}}}$ – playing
the role of counfounders, such as sampling bias or choice of source domain
(Pearl, 2009) – meaning that they can be both disentangled and correlated (via
${\bm{\mathrm{C}}}$). From a causal perspective, disentanglement of variables
can be defined as follows:

######  Definition 1 (Disentanglement of variables)

A set of variables ${\bm{\mathrm{G}}}$ are disentangled if and only if
$p(G_{i}\mid{\bm{\mathrm{C}}},do({\bm{\mathrm{G}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}^{\prime}_{\mathcal{I}}))\equiv
p(G_{i}\mid{\bm{\mathrm{C}}})$ for all possible choices of
$\mathcal{I}\subseteq[n]\setminus\\{i\\}$ and
${\bm{\mathrm{g}}}^{\prime}_{\mathcal{I}}$.

Now, consider the SCM in fig. 1 (left). It is easy to see that the variables
${\bm{\mathrm{G}}}$ are disentangled: any intervention
$do({\bm{\mathrm{G}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}^{\prime}_{\mathcal{I}})$
breaks the links from ${\bm{\mathrm{C}}}$ to
${\bm{\mathrm{G}}}_{\mathcal{I}}$, meaning that changes to the latter will not
affect $G_{i}$. In this case, the variables ${\bm{\mathrm{G}}}$ are also
conditionally independent from one another given ${\bm{\mathrm{C}}}$, or
equivalently
$G_{i}\mathrel{\text{\scalebox{1.07}{$\perp\mkern-10.0mu\perp$}}}G_{j}\mid{\bm{\mathrm{C}}}$
for every $i\neq j$.

Later on, we will be concerned with data generation processes similar to the
one illustrated in fig. 1 (right). Here, a set of generative factors
${\bm{\mathrm{G}}}=(G_{1},\ldots,G_{n})$ with common parents
${\bm{\mathrm{C}}}$ cause an observation ${\bm{\mathrm{X}}}$, and the latter
is encoded into a representation ${\bm{\mathrm{M}}}=(M_{1},\ldots,M_{k})$ by a
machine learning model $p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{X}}})$.
Specifically, ${\bm{\mathrm{M}}}$ is obtained by marginalizing over the inputs
${\bm{\mathrm{X}}}$:

$p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{G}}}):=\mathbb{E}_{{\bm{\mathrm{x}}}\sim
p({\bm{\mathrm{X}}}\mid{\bm{\mathrm{G}}})}[p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{x}}})]$
(2)

This can also be viewed as a stochastic map
$\alpha:{\bm{\mathrm{g}}}\mapsto{\bm{\mathrm{m}}}$. Maps of this kind are
central to our discussion.

Since ${\bm{\mathrm{G}}}$ is disentangled (cf. 1), we can talk about
disentanglement of representations for ${\bm{\mathrm{M}}}$. We say that
${\bm{\mathrm{M}}}$ is disentangled with respect to ${\bm{\mathrm{G}}}$ if,
roughly speaking, each $M_{j}$ encodes information about at most one $G_{i}$,
or – more precisely – as long as $G_{i}$ is kept fixed, the value of $M_{j}$
does not change even when the remaining factors
${\bm{\mathrm{G}}}\setminus\\{G_{i}\\}$ are forcibly modified via
interventions. The degree by which a representation violates disentanglement
of representations can be measured using the $\mathsf{PIDA}$ metric:

######  Definition 2 ($\mathsf{PIDA}$ (Suter et al., 2019))

Let $G_{i}$ be a generative factor and $M_{j}$ an element of the machine
representation. $\mathsf{PIDA}$ measures how much fixing $G_{i}$ to a given
value $g_{i}$ insulates $M_{j}$ from changes to the other generative factors
${\bm{\mathrm{G}}}_{-i}$, and it is defined as:

$\mathsf{PIDA}(G_{i},M_{j}\mid
g_{i},{\bm{\mathrm{g}}}_{-i}):=d\big{(}p_{\theta}(M_{j}\mid do(G_{i}\leftarrow
g_{i})),p_{\theta}(M_{j}\mid do(G_{i}\leftarrow
g_{i},{\bm{\mathrm{G}}}_{-i}\leftarrow{\bm{\mathrm{g}}}_{-i}))\big{)}$ (3)

where $d$ is a divergence.444The original definition (Suter et al., 2019)
fixes $d$ to be the difference between means. Here, we slightly generalize
$\mathsf{PIDA}$ to arbitrary divergences, as doing so can account for changes
in higher-order moments too. The average worst case over all possible choices
of $g_{i}$ and ${\bm{\mathrm{g}}}_{-i}$ is given by:

$\textstyle\mathsf{EMPIDA}(G_{i},M_{j}):=\mathbb{E}_{g_{i}}[\max_{{\bm{\mathrm{g}}}_{-i}}\mathsf{PIDA}(G_{i},M_{j}\mid
g_{i},{\bm{\mathrm{g}}}_{-i})]$ (4)

######  Definition 3 (Disentanglement of representations)

We say that a representation ${\bm{\mathrm{M}}}$ is disentangled with respect
to ${\bm{\mathrm{G}}}$ if and only if
$\max_{j}\min_{i}\mathsf{EMPIDA}(G_{i},M_{j})$ is exactly zero.

In other words, ${\bm{\mathrm{M}}}$ is disentangled with respect to
${\bm{\mathrm{G}}}$ if, for every $M_{j}$ there exists a $G_{i}$ such that
fixing the latter insulates $M_{j}$ from changes to the other generative
factors ${\bm{\mathrm{G}}}_{-i}$. In section 4, we will build on both types of
disentanglement to derive our notion of alignment between representations.

Another important notion is that of context-style separation, which can be
viewed as a special case of disentanglement of representations von Kügelgen et
al. (2021). Let the generative factors ${\bm{\mathrm{G}}}$ be partitioned into
two disentangled sectors ${\bm{\mathrm{G}}}_{\mathcal{I}}$ and
${\bm{\mathrm{G}}}_{-\mathcal{I}}$, representing task-relevant information
(content) and task-irrelevant factors of variations (style), respectively.
Then, ${\bm{\mathrm{M}}}$ satisfies content-style separation if the following
holds:

######  Definition 4 (Content-style separation)

Let $({\bm{\mathrm{G}}}_{\mathcal{I}},{\bm{\mathrm{G}}}_{-\mathcal{I}})$ be
two disentangled sectors. Then, ${\bm{\mathrm{M}}}$ separates content from
style iff it can be partitioned into
$({\bm{\mathrm{M}}}_{\mathcal{J}},{\bm{\mathrm{M}}}_{-\mathcal{J}})$ such
that:

$\mathsf{EMPIDA}({\bm{\mathrm{G}}}_{\mathcal{I}},{\bm{\mathrm{M}}}_{\mathcal{J}})=0$
(5)

This means that, if the content ${\bm{\mathrm{G}}}_{\mathcal{I}}$ is fixed,
the machine representation ${\bm{\mathrm{M}}}_{\mathcal{J}}$ are isolated from
changes to the style ${\bm{\mathrm{G}}}_{-\mathcal{I}}$. This property is
asymmetrical: it holds even if ${\bm{\mathrm{M}}}_{-\mathcal{J}}$ is affected
by interventions to ${\bm{\mathrm{G}}}_{\mathcal{I}}$. Also, there is no
requirement that the elements of ${\bm{\mathrm{M}}}_{\mathcal{J}}$ are
disentangled with respect to ${\bm{\mathrm{G}}}_{\mathcal{I}}$.

##  3 Human Interpretable Representation Learning

We are concerned with acquiring interpretable machine representations. Our key
intuition is that a representation is only interpretable as long as it can be
understood by the human at the receiving end. Based on this, we formally state
our learning problem as follows:

###### Definition 5

Human-interpretable representation learning (hrl) is the problem of learning a
(possibly stochastic) mapping between inputs
${\bm{\mathrm{x}}}\in\mathbb{R}^{d}$ and a set of machine representations
${\bm{\mathrm{z}}}\in\mathbb{R}^{k}$ that enables a machine and a specific
human stakeholder to communicate using those representations.

This mapping can be modeled without loss of generality as a conditional
distribution $p_{\theta}({\bm{\mathrm{Z}}}\mid{\bm{\mathrm{X}}})$, whose
parameters $\theta$ are estimated from data. While 5 encompasses both CBEs and
CBMs, the meaning of ${\bm{\mathrm{Z}}}$ differs in the two cases, as we show
next.

${\bm{\mathrm{G}}}$${\bm{\mathrm{X}}}$${\bm{\mathrm{M}}}_{\mathcal{J}}$${\bm{\mathrm{M}}}_{-\mathcal{J}}$${\bm{\mathrm{H}}}$$Y$

${\bm{\mathrm{G}}}$${\bm{\mathrm{X}}}$$\hat{{\bm{\mathrm{H}}}}$${\bm{\mathrm{M}}}$${\bm{\mathrm{H}}}$$Y$

Figure 2: Left: generative process followed by concept-based models CBMs. A
prediction is inferred based on a subset of “interpretable” concepts
${\bm{\mathrm{M}}}_{\mathcal{J}}\subseteq{\bm{\mathrm{M}}}$, so it is to
${\bm{\mathrm{M}}}_{\mathcal{J}}$ that our notion of alignment (section 4, in
red) applies. Right: generative process followed by concept-based explainers
(CBEs). Here, the machine representation ${\bm{\mathrm{M}}}$ is not required
to be interpretable. Rather, the explainer maps it to extracted concepts
$\hat{{\bm{\mathrm{H}}}}$ and then infers how these contribute to the
prediction. Here it is $\hat{{\bm{\mathrm{H}}}}$ that alignment applies to.

###  3.1 Machine Representations: The Ante-hoc Case

CBMs are neural predictors that follow the generative process shown in fig. 2
(left). During inference, a CBM observes an input ${\bm{\mathrm{x}}}$, caused
by generative factors ${\bm{\mathrm{G}}}$, and extracts a representation
${\bm{\mathrm{M}}}$ by performing MAP inference (Koller and Friedman, 2009) on
a distribution $p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{x}}})$
implemented as a neural network. This representation is partitioned into two
subsets: ${\bm{\mathrm{M}}}_{\mathcal{J}}$ are constrained to be
interpretable, while ${\bm{\mathrm{M}}}_{-\mathcal{J}}$ are not. As shown in
fig. 2, only the interpretable subset is used for inferring a prediction
$\hat{y}$, while ${\bm{\mathrm{M}}}_{-\mathcal{J}}$ – if present – is used for
other tasks, such as reconstruction (Marconato et al., 2022). Specifically,
the predicted concepts ${\bm{\mathrm{M}}}_{\mathcal{J}}$ are fed to a
simulatable top layer $p_{\theta}(Y\mid{\bm{\mathrm{M}}})$ – most often a
sparse linear layer – from which an explanation can be easily derived.
Assuming ${\bm{\mathrm{M}}}_{\mathcal{J}}$ is in fact interpretable, CBMs can
provide local explanations summarizing what concepts are responsible for a
particular prediction in an ante hoc fashion and essentially for free
Bontempelli et al. (2021); Schwalbe (2022). For instance, if
$p_{\theta}(Y\mid{\bm{\mathrm{M}}}_{\mathcal{J}})$ is a linear mapping with
parameters $w_{yj}$, the explanation for predicting $\hat{y}$ is given by
Alvarez-Melis and Jaakkola (2018); Chen et al. (2019, 2020); Koh et al.
(2020); Marconato et al. (2022); Zarlenga et al. (2022):

$\mathcal{E}=\\{(w_{\hat{y}j},m_{j})\ :\ j\in\mathcal{J}\\}$ (6)

where each concept activation $m_{j}$ is associated with a “level of
responsibility” inferred from the top layer’s weights. Specific CBMs are
outlined in section 6.

Summarizing, in the case of CBMs the concepts ${\bm{\mathrm{Z}}}$ used for
communicating with users (cf. 5) are embodied by the interpretable machine
representation ${\bm{\mathrm{M}}}_{\mathcal{J}}$.

###  3.2 Machine Representations: The Post-hoc Case

For CBEs, the generative process is different, see fig. 2 (right). In this
case, the internal representation ${\bm{\mathrm{M}}}$ of the model mapping
from inputs ${\bm{\mathrm{X}}}$ to labels $Y$ is not required to be
interpretable. For instance, it might represent the state of all neurons in a
neural network or that of the neurons in the second-to-last layer. CBEs
explain the reasoning process in a post hoc fashion by extracting the
activations of high-level concepts $\hat{{\bm{\mathrm{H}}}}$ from
${\bm{\mathrm{M}}}$, and then inferring a concept-based explanation
$\mathcal{E}$ specifying the contribution of each $\hat{H}_{i}$ to the model’s
prediction, often in the same form as eq. 6.

Here, we are concerned with the interpretability of $\hat{{\bm{\mathrm{H}}}}$.
Some approaches extract them by (indirectly) relying on concept annotations.
For instance, TCAV (Kim et al., 2018) takes a set of linear classifiers, one
for each concept, pre-trained on a densely annotated dataset, and then adapts
them to work with machine representation ${\bm{\mathrm{M}}}$. Unsupervised
approaches instead mine the concepts directly in the space of machine
representations through a linear decomposition (Ghorbani et al., 2019; Zhang
et al., 2021; Fel et al., 2023a, b). Specific examples are discussed in
section 6. In general, there is no guarantee that the symbolic and sub-
symbolic representations $\hat{{\bm{\mathrm{H}}}}$ and ${\bm{\mathrm{M}}}$
capture exactly the same information. This introduces a faithfulness issue,
meaning that CBE explanations may not portray a reliable picture of the
model’s inference process (Kim et al., 2018; Teso, 2019; Pfau et al., 2021;
Fel et al., 2023b).

However, the issue we focus on is that the representation
${\bm{\mathrm{Z}}}=\hat{{\bm{\mathrm{H}}}}$ used by CBEs to communicate with
users is in fact interpretable, regardless of whether it is also faithful.

###  3.3 From Symbolic Communication to Alignment

What makes symbolic communication possible? While a complete answer to this
question is beyond the scope of this paper, we argue that communication
becomes challenging unless the concepts ${\bm{\mathrm{Z}}}$ with which the
machine and the human communicate are “aligned”, in the sense that concepts
having the same name share the same (or similar enough) semantics. Other
factors contributing to interpretability will be discussed in section 5.

In order to formalize this intuition, we focus on the generative process shown
in fig. 3. In short, we assume observations ${\bm{\mathrm{x}}}$ – e.g., images
or text observed during training and test – are obtained by mapping generative
factors ${\bm{\mathrm{G}}}\sim p^{*}({\bm{\mathrm{G}}}\mid{\bm{\mathrm{C}}})$
through a hidden ground-truth distribution
$p^{*}({\bm{\mathrm{X}}}\mid{\bm{\mathrm{G}}})$.

The observations ${\bm{\mathrm{x}}}$ are then received by two observers: a
machine and a human. The machine maps them to its own learned representation
${\bm{\mathrm{M}}}$, which may or may not be interpretable. The interpretable
representations ${\bm{\mathrm{Z}}}$ – which correspond to
${\bm{\mathrm{M}}}_{\mathcal{J}}$ for CBMs (cf. section 3.1) and to
$\hat{{\bm{\mathrm{H}}}}$ for CBEs (section 3.2) – is then derived from
${\bm{\mathrm{M}}}$.

At the same time, the human observer maps the same observations to its own
vocabulary of concepts ${\bm{\mathrm{H}}}$. For instance, if
${\bm{\mathrm{x}}}$ is an image portraying a simple object on a black
background, ${\bm{\mathrm{h}}}$ may encode the “color” or “shape” of that
object, or any other properties deemed relevant by the human. The choice and
semantics of these concepts depend on the background and expertise of the
human observer and possibly on the downstream task the human may be concerned
with (e.g., medical diagnosis or loan approval), and as such may vary between
subjects. It is to these concepts that the human associates names – like in
fig. 4 – and it is these concepts that they would use for communicating the
properties of ${\bm{\mathrm{x}}}$ to other people.

Notice that the human concepts ${\bm{\mathrm{H}}}$ may be arbitrarily
different from the ground-truth factors ${\bm{\mathrm{G}}}$: whereas the
latter include all information necessary to determine the observations, and as
such may be complex and uninterpretable (Gabbay et al., 2021), the former are
those aspects of the observation that matter to the human observer. A concrete
example is that of color blindness: an observer may be unable to discriminate
between certain wavelengths of visible light, despite these being causes of
the generated image ${\bm{\mathrm{X}}}$. Another, more abstract, example are
the generative factors that cause a particular apple to appear ripe, e.g.,
those biological processes occurring during the apple tree’s reproductive
cycle, which are beyond the understanding of most non-experts.555They are so
opaque that a whole science had to be developed to identify and describe them.
In stark contrast, the concept of “redness” is not causally related to the
apple’s appearance, and yet easily understood by most human observers,
precisely because it is a feature that is evolutionarily and culturally useful
to those observers. In this sense, the concepts ${\bm{\mathrm{H}}}$ are
understandable by definition.

We argue that symbolic communication is feasible whenever the names associated
(by the human) to elements of ${\bm{\mathrm{H}}}$ can be transferred to the
elements of ${\bm{\mathrm{Z}}}$ in a way that preserves semantics. That is,
concepts with the same name should have the same meaning. In order to ensure
information expressed in terms of ${\bm{\mathrm{Z}}}$ – say, an explanation
stating that $Z_{1}$ is irrelevant for a certain prediction – is understood by
the human observer, we need to make sure that ${\bm{\mathrm{Z}}}$ itself is
somehow “aligned” with the human’s representation ${\bm{\mathrm{H}}}$.

${\bm{\mathrm{X}}}$$\ldots$$G_{1}$$G_{n}$${\bm{\mathrm{C}}}$$M_{k}$$\ldots$$M_{1}$$H_{1}$$\ldots$$H_{\ell}$“color”“shape”
Figure 3: Graphical model of our data generation process. In words, $n$
(correlated) generative factors exist in the world
${\bm{\mathrm{G}}}=(G_{1},\ldots,G_{n})$ that cause an observed input
${\bm{\mathrm{X}}}$. The machine maps these to an internal representation
${\bm{\mathrm{M}}}=(M_{1},\ldots,M_{k})$, while the human observer maps them
to its own internal concept vocabulary
${\bm{\mathrm{H}}}=(H_{1},\ldots,H_{\ell})$. Notice that the observer’s
concepts ${\bm{\mathrm{H}}}$ may and often do differ from the ground-truth
factors ${\bm{\mathrm{G}}}$. The concepts ${\bm{\mathrm{H}}}$ are what the
human can understand and attach names to, e.g., the “color” and “shape” of an
object appearing in ${\bm{\mathrm{X}}}$. The association between names and
human concepts is denoted by dotted lines. We postulate that communication is
possible if the machine and the human representations are aligned according to
6.

##  4 Alignment as Name Transfer

###  4.1 Alignment: The Disentangled Case

What does it mean for two representations to be aligned? We start by looking
at the simplest (but non-trivial) case in which the ground-truth factors
${\bm{\mathrm{G}}}$ are disentangled, cf. 1. For ease of exposition, let us
also temporarily assume that some of the generative factors are inherently
interpretable, as in (Marconato et al., 2022). Namely, we assume all factors
in ${\bm{\mathrm{G}}}_{\mathcal{I}}\subseteq{\bm{\mathrm{G}}}$, where
$\mathcal{I}\subseteq[n]$, can be understood by the human observer, while
those in ${\bm{\mathrm{G}}}_{-{\mathcal{I}}}$ cannot. The corresponding data
generation process is illustrated in fig. 4. Under these assumptions, we aim
to recover machine representations ${\bm{\mathrm{M}}}$ that are aligned to the
interpretable factors ${\bm{\mathrm{G}}}_{\mathcal{I}}$.

To this end, we generalize the notion of alignment introduced by Marconato et
al. (2022).666Our definition extends that of (Marconato et al., 2022) to the
general case in which the mapping $\alpha$ – which is defined as a marginal
distribution in eq. 2 – is stochastic rather than deterministic. Doing so
allows us to cater to more realistic applications and to draw an explicit
connection with $\mathsf{PIDA}$ in proposition 1. As anticipated, our
definition revolves around the conditional distribution on ${\bm{\mathrm{M}}}$
given by ${\bm{\mathrm{G}}}$, or equivalently the stochastic map
$\alpha:{\bm{\mathrm{g}}}\mapsto{\bm{\mathrm{m}}}$ defined in eq. 2 and shown
in red in fig. 4. The key intuition is that two concept vocabularies
${\bm{\mathrm{G}}}$ and ${\bm{\mathrm{M}}}$ are aligned if and only if
$\alpha$ preserves the semantics of the interpretable generative factors
${\bm{\mathrm{G}}}_{\mathcal{I}}$.

More specifically, alignments holds if $\alpha$ allows to transfer the names
of the interpretable factors in a way that preserves semantics. If
$p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{X}}})$ is learned in an
unsupervised fashion, names are generally transferred by collecting or
constructing inputs annotated with the corresponding human concepts, feeding
them to the concept extractor, and looking for matches between the annotations
and the elements of ${\bm{\mathrm{M}}}_{\mathcal{J}}$.777If concept-level
annotations are used, the names are automatically transferred along with them,
but we still wish the user to be able to match the learned concepts with its
own. In a sense, this process is analogous to giving the human observer access
to a set of “knobs”, each one controlling the value of one
$G_{i}\in{\bm{\mathrm{G}}}_{\mathcal{J}}$, and to a visualization of the
machine representation ${\bm{\mathrm{M}}}_{\mathcal{I}}$. Turning a knob is
akin to intervening on the corresponding factor $G_{i}$. If, by turning a
knob, the user is able to figure out what $G_{i}$ corresponds to what $M_{j}$,
then they will associate them the same name. Since we are assuming
${\bm{\mathrm{G}}}_{\mathcal{I}}$ is disentangled, turning one knob does not
affect the others, which simplifies the process.

The formal definition of alignment is as follows:

######  Definition 6 (Alignment)

Given generative factors ${\bm{\mathrm{G}}}$ of which
${\bm{\mathrm{G}}}_{\mathcal{I}}$ are interpretable, a machine representation
${\bm{\mathrm{M}}}$ is aligned iff the map $\alpha$ between
${\bm{\mathrm{G}}}$ and ${\bm{\mathrm{M}}}$ can be written as:

${\bm{\mathrm{M}}}_{\mathcal{J}}=\alpha({\bm{\mathrm{G}}},{\bm{\mathrm{N}}})_{\mathcal{J}}=(\mu_{j}(G_{\pi(j)},N_{j})\
:\ j\in\mathcal{J})$ (7)

where ${\bm{\mathrm{M}}}_{\mathcal{J}}\subseteq{\bm{\mathrm{M}}}$ are the
machine representations that ought to be interpretable, ${\bm{\mathrm{N}}}$
are independent noise variables, and $\pi$ and $\mu$ satisfy the following
properties:

  * D1.

The index map $\pi:\mathcal{J}\mapsto\mathcal{I}$ is surjective and, for all
$j\in\mathcal{J}$, it holds that, as long as $G_{\pi(j)}$ is kept fixed,
$M_{j}$ remains unchanged even when the other generative factors
${\bm{\mathrm{G}}}\setminus\\{G_{\pi(j)}\\}$ are forcibly modified.

  * D2.

Each element-wise transformation $\mu_{j}$, for $j\in\mathcal{J}$, is
monotonic in expectation over $N_{j}$:

$\exists\bowtie\in\\{>,<\\}\;\text{such that}\;\forall
g^{\prime}_{\pi(j)}>g_{\pi(j)},\;\big{(}\mathbb{E}_{N_{j}}[\mu_{j}(g_{\pi(j)},N_{j})]-\mathbb{E}_{N_{j}}[\mu_{j}(g^{\prime}_{\pi(j)},N_{j})]\big{)}\bowtie
0$ (8)

${\bm{\mathrm{X}}}$${\bm{\mathrm{G}}}_{\mathcal{I}}$${\bm{\mathrm{G}}}_{-{\mathcal{I}}}$${\bm{\mathrm{C}}}$${\bm{\mathrm{M}}}_{\mathcal{J}}$${\bm{\mathrm{M}}}_{-\mathcal{J}}$

Figure 4: Simplified generative process with a single observer, adapted from
(Marconato et al., 2022). Here, ${\bm{\mathrm{C}}}$ are unobserved confounding
variables influencing the generative factors ${\bm{\mathrm{G}}}$, and
${\bm{\mathrm{M}}}$ is the latent representation learned by the machine. The
red arrow represents the map $\alpha$.

Let us motivate our two desiderata. In line with prior work on disentangled
representations (Bengio et al., 2013; Higgins et al., 2018), D1 requires that
$\alpha$ should not “mix” multiple $G_{i}$’s into a single $M_{j}$, regardless
of whether the former belong to ${\bm{\mathrm{G}}}_{\mathcal{I}}$ or not. For
instance, if $M_{j}$ blends together information about both color and shape,
or about color and some uninterpretable factor, human observers would have
trouble pinning down which one of their concepts it matches. If it does not,
then turning the $G_{\pi(j)}$ knob only affects $M_{j}$, facilitating name
transfer.888The converse is not true: as we will see in section 4.4,
interpretable concepts with “compatible semantics” can in principle be blended
together without compromising interpretability. We will show in section 4.2
that this is equivalent to disentanglement.

D2 is also related to name transfer. Specifically, it aims to ensure that,
whenever the user turns a knob $G_{\pi(j)}$, they can easily understand what
happens to $M_{j}$ and thus figure out the two variables encode the same
information. To build intuition, notice that both D1 and D2 hold for the
identity function, as well as for those maps $\alpha$ that reorder or rescale
the elements of ${\bm{\mathrm{G}}}_{\mathcal{I}}$, which clearly preserve
semantics and naturally support name transfer. Monotonicity captures all of
these cases and also more expressive non-linear element-wise functions, while
conservatively guaranteeing a human would be able to perform name transfer.
Notice also that D2 can be constrained further based on the application.

A couple of remarks are in order. Most importantly, notice that our definition
of alignment immediately applies also to the mapping between
${\bm{\mathrm{Z}}}$ and the human vocabulary ${\bm{\mathrm{H}}}$. In this
case, $\alpha$ is the map between human concepts ${\bm{\mathrm{h}}}$ and
machine representations ${\bm{\mathrm{z}}}$, obtained by marginalizing over
${\bm{\mathrm{X}}}$, ${\bm{\mathrm{G}}}$, and ${\bm{\mathrm{C}}}$ (see fig.
3), and it is only aligned if it satisfies D1 and D2. More generally,
alignment can hold for any mapping between representations. We also observe
that, since $\pi$ maps $\mathcal{J}$ exclusively into $\mathcal{I}$, alignment
entails a form of content-style separation (4), in that
${\bm{\mathrm{M}}}_{\mathcal{J}}$ does not encode any information about
${\bm{\mathrm{G}}}_{-\mathcal{I}}$. We will show in section 4.3 that
representations that do not satisfy this condition can be affected by concept
leakage, while aligned representations cannot. Finally, we note that
${\bm{\mathrm{M}}}$ can be aligned and still contain multiple transformations
of the same $G_{i}\in{\bm{\mathrm{G}}}_{\mathcal{I}}$. This does not
compromise interpretability in that all “copies” can always be traced back to
the same $G_{i}$.

###  4.2 Disentanglement Does Not Entail Alignment

Next, we clarify the relationship between alignment and disentanglement of
representations by showing that the latter is exactly equivalent to D1:

###### Proposition 1

Assuming noise terms are independent, as per section 2, D1 holds if and only
if the representations are disentangled in
$({\bm{\mathrm{G}}}_{\mathcal{I}},{\bm{\mathrm{M}}}_{\mathcal{J}})$ (cf. 3.)

All proofs can be found in appendix A. The equivalence between disentanglement
of representations and D1 implies that disentanglement is insufficient for
interpretability: even if ${\bm{\mathrm{M}}}$ is disentangled, i.e., each
$M_{j}$ encodes information about at most one
$G_{i}\in{\bm{\mathrm{G}}}_{\mathcal{I}}$, nothing prevents the transformation
from $G_{i}$ to its associated $M_{j}$ from being arbitrarily complex,
complicating name transfer. In the most extreme case, $\alpha(\cdot)_{j}$ may
not be injective, making it impossible to distinguish between different
$g_{i}$’s, or could be an arbitrary shuffling of the continuous line: this
would clearly obfuscate any information present about $G_{i}$. This means
that, during name transfer, a user would be unable to determine what value of
$M_{j}$ corresponds to what value of $G_{i}$ or to anticipate how changes to
the latter affect the former.

This is why D2 in 6 requires the map between each
$G_{i}\in{\bm{\mathrm{G}}}_{\mathcal{I}}$ and its associated $M_{j}$ to be
“simple”. This extra desideratum makes alignment strictly stronger than
disentanglement.

###  4.3 Alignment Entails No Concept Leakage

Concept leakage is a recently discovered phenomenon whereby the
“interpretable” concepts ${\bm{\mathrm{M}}}_{\mathcal{J}}$ unintentionally end
up encoding information about extraneous concepts (Mahinpei et al., 2021).
Empirically, leaky concepts are predictive for inference tasks that – in
principle – do not depend on them. Situations like the following occur in
practice, even if full concept supervision is used (Marconato et al., 2022;
Mahinpei et al., 2021; Margeloiu et al., 2021):

###### Example 1

Let ${\bm{\mathrm{X}}}$ be a dSprites image (Matthey et al., 2017) picturing a
white sprite, determined by generative factors including “position”, “shape”,
and “size”, on a black background. Now imagine training a concept extractor
$p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{X}}})$ so that
${\bm{\mathrm{M}}}_{\mathcal{J}}$ encodes shape and size – but not position –
by using full concept-level annotations for shape and size. The concept
extractor is then frozen. During inference, the goal is to classify sprites as
either positive ($Y=1$) or negative ($Y=0$) depending on whether they are
closer to the top-right corner or the bottom-left corner. When concept leakage
occurs, the label – which clearly depends only on position – can be predicted
with above random accuracy from ${\bm{\mathrm{M}}}_{\mathcal{J}}$, meaning
these concepts somehow encode information about position, which they are not
supposed to.

The only existing formal account of concept leakage was provided by Marconato
et al. (2022), who view it in terms of (lack of) out-of-distribution (OOD)
generalization. Other works instead focus on in-distribution behavior and
argue that concept leakage is due to encoding discrete generative factors
using a continuous representation (Lockhart et al., 2022; Havasi et al.,
2022). We go beyond these works by providing the first general formulation of
concept leakage and showing that it is related to alignment. Specifically, we
propose to view concept leakage as a (lack of) content-style separation, and
show that this explains how concept leakage can arise both in- and out-of-
distribution.

${\bm{\mathrm{X}}}$${\bm{\mathrm{C}}}$${\bm{\mathrm{G}}}_{\mathcal{I}}$${\bm{\mathrm{G}}}_{-\mathcal{I}}$$do({\bm{\mathrm{g}}}_{-\mathcal{I}})$$Y$${\bm{\mathrm{M}}}_{\mathcal{J}}$${\bm{\mathrm{M}}}_{-\mathcal{J}}$

Figure 5: Generative process for Concept Leakage. A predictor observes
examples $({\bm{\mathrm{X}}},Y)$ and infers $Y$ from its interpretable
representation ${\bm{\mathrm{M}}}_{\mathcal{J}}$ using a learnable conditional
distribution $q_{\lambda}(Y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})$, indicated in
orange. Since the label $Y$ depends solely on
${\bm{\mathrm{G}}}_{-\mathcal{I}}$, we would expect that it cannot be
predicted better than at random: intuitively, if this occurs it means that
information from ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ has leaked into the
interpretable concepts ${\bm{\mathrm{M}}}_{\mathcal{J}}$. Any intervention
$do({\bm{\mathrm{G}}}_{-\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}_{-\mathcal{I}})$
on the uninterpretable/unobserved concepts detaches these from
${\bm{\mathrm{C}}}$, meaning that the label truly only depends on
${\bm{\mathrm{G}}}_{-\mathcal{I}}$.

We start by formalizing the intuition that concept leakage is excess
prediction accuracy – gained by leveraging leaky concepts – compared to a
leak-free baseline (Marconato et al., 2022; Havasi et al., 2022). The
corresponding generative process is reported in fig. 5. We assume the
generative factors ${\bm{\mathrm{G}}}$ are partitioned as
$({\bm{\mathrm{G}}}_{\mathcal{I}},{\bm{\mathrm{G}}}_{-\mathcal{I}})$ such that
only ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ are informative for predicting a label
$Y$, mediated by the conditional distribution
$p(Y\mid{\bm{\mathrm{G}}}_{-\mathcal{I}})$. This implies that their mutual
information is positive, that is,
$I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)>0$.999In this section we are mostly
concerned with the non-informativeness of ${\bm{\mathrm{G}}}_{\mathcal{I}}$,
hence we allow ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ to potentially contain also
interpretable factors. Now, fix a concept encoder
$p_{\theta}({\bm{\mathrm{M}}}_{\mathcal{J}}\mid{\bm{\mathrm{X}}})$ and let
$q_{\lambda}(Y\mid{\bm{\mathrm{M}}}_{\mathcal{J}})$ be a predictor learned on
top of it (in orange in the figure). To quantify concept leakage, we look at
how well the best possible such predictor can infer the label $Y$ using
${\bm{\mathrm{M}}}_{\mathcal{J}}$ after intervening on
${\bm{\mathrm{G}}}_{-\mathcal{I}}$. Analogously to $\mathsf{EMPIDA}$ (2), the
intervention detaches ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ from
${\bm{\mathrm{C}}}$, thus ensuring the label $Y$ cannot be influenced by the
irrelevant factors ${\bm{\mathrm{G}}}_{\mathcal{I}}$. The resulting
manipulated distribution on ${\bm{\mathrm{G}}}$ is:

$p^{\prime}({\bm{\mathrm{G}}})=p({\bm{\mathrm{G}}}\mid
do({\bm{\mathrm{G}}}_{-\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}_{-\mathcal{I}}))q({\bm{\mathrm{g}}}_{-\mathcal{I}}):=\mathbb{E}_{\bm{\mathrm{C}}}[p({\bm{\mathrm{G}}}_{\mathcal{I}}\mid{\bm{\mathrm{C}}})]\mathbbm{1}\\!\left\\{{\bm{\mathrm{G}}}_{-\mathcal{I}}={\bm{\mathrm{g}}}_{-\mathcal{I}}\right\\}q({\bm{\mathrm{g}}}_{-\mathcal{I}})$
(9)

where $q({\bm{\mathrm{g}}}_{-\mathcal{I}})$ is a distribution over possible
interventions. This can be any distribution, with the only requirement that
under any intervention
$do({\bm{\mathrm{G}}}_{-\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}_{-\mathcal{I}})$
the model observes different variations of $Y$.101010If $Y$ is constant,
leakage is impossible, since $I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)=0$.

From the causal factorization in fig. 5, the joint probability of
$({\bm{\mathrm{X}}},Y)$ resulting from the post-interventional distribution
$p^{\prime}({\bm{\mathrm{G}}})$ is given by:

$p({\bm{\mathrm{X}}},Y)=\mathbb{E}_{{\bm{\mathrm{g}}}\sim
p^{\prime}({\bm{\mathrm{G}}}\mid
do({\bm{\mathrm{G}}}_{-\mathcal{I}}\leftarrow{\bm{\mathrm{g}}}_{-\mathcal{I}}))}[p(Y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})p({\bm{\mathrm{X}}}|{\bm{\mathrm{g}}})]$
(10)

Data of this kind appear, for example, in the dSprites experiment (Marconato
et al., 2022) outlined in 1. Here, during training the “position” of the
sprite is fixed (i.e.,
${\bm{\mathrm{G}}}_{pos}={\bm{\mathrm{G}}}_{-\mathcal{I}}$ are fixed to the
center), while at test time the data contains different interventions over the
position ${\bm{\mathrm{G}}}_{pos}={\bm{\mathrm{G}}}_{-\mathcal{I}}$, and free
variations of the other factors ${\bm{\mathrm{G}}}_{\mathcal{I}}$ (e.g.,
“shape” and “size”). Essentially, these interventions move the sprite around
the top-right and bottom-left borders, where the factors
${\bm{\mathrm{G}}}_{pos}$ are extremely informative for the label $Y$.

In order to measure the degree of concept leakage in
$p_{\theta}({\bm{\mathrm{M}}}_{\mathcal{J}}\mid{\bm{\mathrm{X}}})$, we compare
the prediction performance of the best possible predictor
$q_{\lambda}(Y\mid{\bm{\mathrm{M}}}_{\mathcal{J}})$ with that of the best
possible predictor $r_{\gamma}(Y)$ that does not depend on
${\bm{\mathrm{M}}}_{\mathcal{J}}$ at all. This is equivalent to comparing the
behavior of two Bayes optimal predictors, one of which has access to the
learned (possibly leaky) concepts whereas the other does not. In the
following, we assume the distributions $q_{\lambda}$ an $r_{\gamma}$ to be
sufficiently expressive, i.e., they can encode any sufficiently well behaved
stochastic function. This is the case, for instance, when they are implemented
as deep neural networks. We are now ready to define concept leakage:

######  Definition 7 (Concept Leakage)

Given a classifier $q_{\lambda}(y\mid{\bm{\mathrm{z}}})$, an uninformed Bayes
optimal predictor $r_{\gamma}(y)$, and data samples
$({\bm{\mathrm{x}}},y)\in\mathcal{D}$, concept leakage $\Lambda$ is the
difference:

$\Lambda=\max_{\lambda}[\mathcal{L}_{CL}(\lambda)]-\max_{\gamma}[\mathcal{L}_{r}(\gamma)]$
(11)

where:

$\mathcal{L}_{CL}=\mathbb{E}_{({\bm{\mathrm{x}}},y)\sim
p({\bm{\mathrm{X}}},Y)}\log
q_{\lambda,\theta}(y|{\bm{\mathrm{x}}})\qquad\mathcal{L}_{r}=\mathbb{E}_{({\bm{\mathrm{x}}},y)\sim
p({\bm{\mathrm{X}}},Y)}\log r_{\gamma}(y)$ (12)

are the average log-likelihood of the classifier
$q_{\lambda,\theta}(Y|{\bm{\mathrm{X}}}):=\mathbb{E}_{{\bm{\mathrm{m}}}_{\mathcal{J}}\sim
p_{\theta}({\bm{\mathrm{M}}}_{\mathcal{J}}\mid{\bm{\mathrm{X}}})}p(Y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})$
and of the uninformed Bayes optimal classifier, respectively.

By 7, concept leakage occurs if and only if there exists a $\lambda$ that
allows to predict $Y$ better than the best uninformed predictor. In the
following analysis, we characterize concept leakage evaluated on the ground-
truth distribution $p({\bm{\mathrm{X}}},Y)$. We proceed to show that this
quantity is bounded by two terms:

###### Proposition 2

Assuming the causal factorization in fig. 5, it holds that:

$I({\bm{\mathrm{M}}}_{\mathcal{J}},Y)\leq\Lambda\leq
I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)$ (13)

where $I({\bm{\mathrm{A}}},{\bm{\mathrm{B}}})$ denotes the mutual information
between ${\bm{\mathrm{A}}}$ and ${\bm{\mathrm{B}}}$.

The bounds in eq. 13 are useful for understanding how concept leakage behaves.
They show, for instance, that $\Lambda$ cannot exceed the mutual information
between ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ and $Y$. Second, applying the data-
processing inequality (Cover, 1999) to the lower bound yields
$I({\bm{\mathrm{M}}}_{\mathcal{J}},Y)\geq
I({\bm{\mathrm{M}}}_{\mathcal{J}},{\bm{\mathrm{G}}}_{-\mathcal{I}})$. The
latter quantifies the information contained in
${\bm{\mathrm{M}}}_{\mathcal{J}}$ about ${\bm{\mathrm{G}}}_{-\mathcal{I}}$. In
other words, concept leakage can only be zero if indeed the machine concepts
${\bm{\mathrm{M}}}_{\mathcal{J}}$ contain no information about them, because
$I({\bm{\mathrm{M}}}_{\mathcal{J}},{\bm{\mathrm{G}}}_{-\mathcal{I}})\leq\Lambda=0$.
Next, we also show that if ${\bm{\mathrm{M}}}_{\mathcal{J}}$ does not encode
information about ${\bm{\mathrm{G}}}_{-\mathcal{I}}$ – or equivalently, it
satisfies content-style separation (4) – then it has zero concept leakage.

###### Proposition 3

Suppose that ${\bm{\mathrm{M}}}_{\mathcal{J}}$ does not encode any information
of ${\bm{\mathrm{G}}}_{-\mathcal{I}}$, consistently with content-style
separation (4), then $\Lambda$ is zero.

This result leads to two consequences. Let us start by looking at the out-of-
distribution case investigated in (Marconato et al., 2022). Here, the concept
extractor is trained only on some fixed variations of
${\bm{\mathrm{G}}}_{-\mathcal{I}}$. However, when the support of
${\bm{\mathrm{G}}}_{-\mathcal{I}}$ changes drastically, the model is not
likely to ensure content-style separation outside of the support of the
training distribution, even if D1 holds in-distribution. This failure can be
explained by the difficulty of disentanglement techniques to ensure
disentanglement for out-of-distribution samples– in the context of
combinatorial generalization – by Montero et al. (2020, 2022). Consider for
dSprites example. Here, during training, sprites are located in the dead
center of the background, and when observing the sprites on the borders of the
image, which is far away from the support of the training set, the concept
encoder fails to ensure their representations are disentangled. This failure
of disentanglement techniques to ensure disentanglement for out-of-
distribution inputs was also observed – in the context of combinatorial
generalization – by Montero et al. (2020, 2022). Our results show that if
content-style separation does not hold, concept leakage may be non-zero,
meaning that techniques like open-set recognition (Sun et al., 2020) must be
adopted to detect OOD inputs and process them separately.

Next, we look at concept leakage for in-distribution scenarios. Following
Havasi et al. (2022), consider a model leveraging two concepts – presence of
“tail” and “fur“ – and the task of distinguish between images of cats and dogs
using these (clearly non-discriminative) concepts. According to (Havasi et
al., 2022), concept leakage can occur when binary concepts like these are
modelled using continuous variables, meaning the concept extractor can
unintentionally encode “spurious” discriminative information. In light of our
analysis, we argue that concept leakage is instead due to lack of content-
style separation, and thus of alignment. To see this, suppose there exists a
concept $G_{k}\in{\bm{\mathrm{G}}}_{-\mathcal{I}}$ useful for distinguishing
cats from dogs and that it is disentangled as in 1 from the concepts of fur
$G_{fur}$ and of tail $G_{tail}$. Then, by content-style separation, any
representation ${\bm{\mathrm{M}}}_{\mathcal{J}}$ that is aligned to $G_{fur}$
and $G_{tail}$ does not encode any information about $G_{k}$, leading to zero
concept leakage.

In both cases, concept leakage arises as a failure in content-style separation
between relevant and irrelevant generative factors, and as such it can be used
as a proxy for measuring the latter. Moreover, since alignment implies
content-style separation, aligned representations cannot suffer from concept
leakage.111111Note that the converse is not true: while alignment entails
content-style separation, the latter can hold independently from alignment.

An extension of this result includes the case where
$G_{k}\in{\bm{\mathrm{G}}}_{\mathcal{I}}$, i.e., the ground-truth factor
$G_{k}$ is relevant for in-distribution predictions $Y$ and representations
${\bm{\mathrm{M}}}_{\mathcal{J}}$ encode it somewhere. In this case, concept
leakage is evaluated among the elements of ${\bm{\mathrm{M}}}_{\mathcal{J}}$
that should not encode $G_{k}$. That is, if a subset of
${\bm{\mathrm{M}}}_{\mathcal{J}}$ encodes only the concepts $G_{fur}$ and
$G_{tail}$ it must not be discriminative for $Y$. Without loss of
generality,121212The general case includes all representations
${\bm{\mathrm{M}}}_{\pi^{-1}(k)}$, where $\pi^{-1}$ is the pre-image of the
map $\pi$. we suppose that only a single $M_{j^{\prime}}$ is aligned to
$G_{k}$, that is $\pi(j^{\prime})=k$, whereas other
${\bm{\mathrm{M}}}_{\mathcal{J}}\setminus M_{j^{\prime}}$ are aligned to other
concepts, among which $G_{fur}$ and $G_{tail}$. Then, the following holds:

###### Corollary 1

Consider a representation ${\bm{\mathrm{M}}}_{\mathcal{J}}$ that is aligned to
a set of disentangled concepts ${\bm{\mathrm{G}}}_{\mathcal{I}}$, among which
only $G_{k}$ is discriminative for the label $Y$. Then, all
$M_{j}\in{\bm{\mathrm{M}}}_{\mathcal{J}}$ that are not associated by $\alpha$
to $G_{k}$, i.e., $\pi(j)\neq k$, do not suffer from concept leakage.

Ultimately, an aligned representation prevents concept leakage among the
encoded concepts. If some of the representations
${\bm{\mathrm{M}}}_{\mathcal{J}}$ are aligned only to the concepts $G_{fur}$
and $G_{tail}$, they cannot be used to discriminate between cats and dogs.

###  4.4 Alignment: The Block-wise Case

So far, we assumed the generative factors ${\bm{\mathrm{G}}}_{\mathcal{I}}$ –
or, equivalently, the human concepts ${\bm{\mathrm{H}}}$ – are disentangled.
We now extend alignment to more complex cases in which the human concepts can
be mixed together without compromising interpretability. This covers
situations in which, for instance, the machine captures a single categorical
generative factor using multiple variables via one-hot encoding, or uses polar
coordinates to represent the 2D position of an object.

To formalize this setting, we assume ${\bm{\mathrm{G}}}_{\mathcal{I}}$ and
${\bm{\mathrm{M}}}_{\mathcal{J}}$ are partitioned into non-overlapping
“blocks” of variables
${\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}\subseteq{\bm{\mathrm{G}}}_{\mathcal{I}}$
and
${\bm{\mathrm{M}}}_{\mathcal{J}^{\prime}}\subseteq{\bm{\mathrm{M}}}_{\mathcal{J}}$,
respectively. The idea is that each block
${\bm{\mathrm{M}}}_{\mathcal{J}^{\prime}}$ captures information about only a
single block ${\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}$, and that while mixing
across blocks is not allowed, mixing the variables within each block is. From
the human’s perspective, this means that name transfer is now done block by
block. With this in mind, we define block alignment as follows:

######  Definition 8 (Block-wise Alignment)

A machine representation ${\bm{\mathrm{M}}}$ is block-wise aligned to
${\bm{\mathrm{G}}}_{\mathcal{I}}$ if and only if there exists a subset
${\bm{\mathrm{M}}}_{\mathcal{J}}\subseteq{\bm{\mathrm{M}}}$, a partition
$\mathcal{P}_{\bm{\mathrm{M}}}$ of $\mathcal{J}$, and a mapping
$\alpha:({\bm{\mathrm{g}}},{\bm{\mathrm{N}}})\mapsto{\bm{\mathrm{m}}}$ such
that:

${\bm{\mathrm{M}}}_{\mathcal{J}^{\prime}}=\alpha({\bm{\mathrm{G}}},{\bm{\mathrm{N}}})_{\mathcal{J}^{\prime}}:=\mu_{\mathcal{J}^{\prime}}({\bm{\mathrm{G}}}_{\Pi(\mathcal{J}^{\prime})},{\bm{\mathrm{N}}}_{\mathcal{J}^{\prime}})\qquad\forall\mathcal{J}^{\prime}\in\mathcal{P}_{\bm{\mathrm{M}}}$
(14)

where the maps $\Pi$ and $\mu$ satisfy the following properties.

  * D1

There exists a partition $\mathcal{P}_{\bm{\mathrm{G}}}$ of
$\mathcal{I}$131313In principle, we can extend this notion to a family of
subsets $\mathcal{P}_{\bm{\mathrm{G}}}$ of $\mathcal{I}$. As an example, for
$xyz$ positions, one can consider blocks $\\{xy,yz,xz\\}$ that are mapped to
respectively block aligned representations. such that
$\Pi:\mathcal{P}_{\bm{\mathrm{M}}}\to\mathcal{P}_{\bm{\mathrm{G}}}$. We call
this condition block-wise disentanglement

  * D2

Each map $\mu_{\mathcal{J}^{\prime}}$ is simulatable and invertible141414For
continuous variables, we require it to be a diffeomorphism. on the first
statistical moment, that is, there exists a unique pre-image $\alpha^{-1}$
defined as:

${\bm{\mathrm{G}}}_{\Pi(\mathcal{J}^{\prime})}=\alpha^{-1}(\mathbb{E}[{\bm{\mathrm{M}}}_{\mathcal{J}}])_{\mathcal{J}^{\prime}}:=\big{(}\mathbb{E}_{{\bm{\mathrm{N}}}_{\mathcal{J}}}[\mu_{\mathcal{J}^{\prime}}(\cdot,{\bm{\mathrm{N}}}_{\mathcal{J}^{\prime}})]\big{)}^{-1}({\bm{\mathrm{G}}}_{\Pi(\mathcal{J}^{\prime})})$
(15)

By D1, changes to any block of human concepts only impact a single block of
machine concepts, and by D2 the change can be anticipated by the human
observer, that is the human interacting with the machine grasps what is the
general mechanism behind the transformation from ${\bm{\mathrm{G}}}$ (or
${\bm{\mathrm{H}}}$) and ${\bm{\mathrm{M}}}$ (and vice versa). Both properties
support name transfer.

A priori, it is not clear to say what transformations are simulatable (Lipton,
2018), as this property depends crucially on the human’s cognitive limitations
and knowledge. However, simulatability can be assessed via user studies in
practice. We remark that D2 implicitly constraints the variables within each
block to be “semantically compatible” because this property impacts
simulatable. In the context of image recognition, for instance, placing
concepts such as “nose shape” and “sky color” in the same block is likely to
make name transfer substantially more complicated, as changes to “nose shape”
might end affecting the representation of “‘sky color”. Semantic compatibility
is fundamentally a psychological issue. An example of semantic compatibility
is that of rototranslation of the coordinates followed by element-wise
rescaling151515This condition is identical to “weak identifiability” in
representation learning Hyvarinen and Morioka (2017); Khemakhem et al.
(2020a).. A counter example would be a map $\alpha$ given by a conformal map
for the 2D position of an object in a scene. Albeit invertible, it may not be
simple at all to simulate.

Notice that with this definition we include two possible scenarios: (i) the
case where some of the ground-truth concepts belonging to the same block are
transformed in a single block, and (ii) the case where semantically
compatible, but disentangled, concepts ${\bm{\mathrm{G}}}_{I}$ are mixed
together in ${\bm{\mathrm{M}}}_{\mathcal{J}}$, which is often neglected in
current disentanglement literature. The latter includes and extends the
special case of alignment for disentangled ${\bm{\mathrm{G}}}$.

The limitation of 8 reflects the fact that taking into account the possible
user’s grasp of the representation is not straightforward to define and poses
a challenge to provide a uniquely accepted definition that considers the human
factor.

###  4.5 Alignment: The General Case

In the most general case the generative factors ${\bm{\mathrm{G}}}$ are
causally related to each other according to an arbitrary ground-truth SCM
$\mathfrak{C}_{\bm{\mathrm{G}}}$. This entails that
${\bm{\mathrm{G}}}_{\mathcal{I}}$ is no longer (block) disentangled. Hence,
during name transfer turning a “knob” (i.e., a variable $G_{i}$) affects those
knobs that are causally dependent on it.

Naturally, the semantics of the user’s comprises the causal relations between
them. To see this, let $G_{1}$ be the temperature and $G_{2}$ the color of a
metal object: the user knows that temperature affects color, and would not
assign the same name to a representation of temperature that does not have a
similar effect on the representation of color. In order ensure preservation of
these semantics, we say that a machine representation ${\bm{\mathrm{M}}}$ is
aligned to ${\bm{\mathrm{G}}}$ if, whenever the human intervenes on one
$G_{i}$, affecting those generative factors
${\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}$ that depend on it, an analogous
change occurs in the machine representation.

We now show that block-alignment is sufficient to satisfy this desideratum.
Even in this more general case, the distribution over the representation can
be obtained by marginalizing over the input variables:

$p({\bm{\mathrm{M}}}):=\mathbb{E}_{{\bm{\mathrm{x}}}\sim
p({\bm{\mathrm{X}}})}p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{x}}})\equiv\mathbb{E}_{{\bm{\mathrm{g}}}\sim
p({\bm{\mathrm{G}}})}p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{g}}})$ (16)

Notice that the definition of block alignment does not make any assumption
about absence or presence of causal relations between blocks of generative
factors, meaning that we it is still well-defined in this more general
setting.161616This generalizes block alignment beyond disentangled factors
(Suter et al., 2019). Specifically, a map $\alpha$ can be block aligned if the
variables $G$ within each block are disentangled with each other, although
there may exist causal relations across blocks.

Now, imagine having a stochastic map $\alpha$ between ${\bm{\mathrm{G}}}$ and
${\bm{\mathrm{M}}}$ that does satisfy block alignment, and also that there
exist causal relations between the blocks
${\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}$. Whenever the user turns a “knob”
corresponding to a ground-truth block, this yields an interventional
distribution $p({\bm{\mathrm{G}}}\mid
do({\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}\leftarrow{\bm{\mathrm{g}}}_{\mathcal{I}^{\prime}}))$.
Through $\alpha$, this determines a new interventional distribution on the
machine representations, namely:

$p({\bm{\mathrm{M}}}\mid
do({\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}\leftarrow{\bm{\mathrm{g}}}_{\mathcal{I}^{\prime}}))=\mathbb{E}_{{\bm{\mathrm{g}}}\sim
p({\bm{\mathrm{G}}}\mid
do({\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}\leftarrow{\bm{\mathrm{g}}}_{\mathcal{I}^{\prime}}))}p_{\theta}({\bm{\mathrm{M}}}\mid{\bm{\mathrm{g}}})$
(17)

This implies a representation ${\bm{\mathrm{M}}}$ where the (interventional)
distribution is obtained by mapping the state
${\bm{\mathrm{g}}}_{\mathcal{I}^{\prime}}$ through $\alpha$. The same
operation can be performed to obtain the state of all other machine
representations aligned with the blocks that are causally related to
${\bm{\mathrm{G}}}_{\mathcal{I}^{\prime}}$ and affected by the intervention.

Note that this distribution automatically takes causal relations between
generative factors into account and treats them as causal relations between
machine representations. To see this, consider the following example:

###### Example 2

Consider two generative factors $G_{1}$ and $G_{2}$ causally connected via a
structural assignment $G_{2}\leftarrow f(G_{1},N_{2})$, as in fig. 6. As
before, $G_{1}$ could be the temperature and $G_{2}$ the color of a metal
solid. Correspondingly, the aligned representation ${\bm{\mathrm{M}}}$ encodes
the temperature in two distinct variables, $M_{1}$ and $M_{3}$, respectively
to the temperature, say, measured in Celsius and Fahrenheit degrees. $M_{2}$
encodes the color variable.

The consequence of block-wise alignment is sketched in the fig. 6 in three
distinct cases: (left) intervening on the temperature $G_{1}$ affects both the
aligned variables $(M_{1},M_{3})$ and the color $G_{2}$. Correspondingly this
has an effect also on $M_{2}$ that changes according to $G_{2}$. (center) An
intervention on $G_{2}$ influences only $M_{2}$ through $\alpha$ and it does
not affect $M_{1}$ and $M_{3}$. (right) The effect of an intervention on whole
variables ${\bm{\mathrm{G}}}$ is localized such that interventions on the
temperature factor $G_{1}$ will affect $M_{1}$ and $M_{3}$, whereby the
interventions on $G_{2}$ only affect $M_{2}$, isolating it from the
intervention on $G_{1}$.

$G_{1}$$do(G_{1}\leftarrow g_{1})$$G_{2}$$M_{1}$$M_{2}$$M_{3}$ | $G_{1}$$G_{2}$$do(G_{2}\leftarrow g_{2})$$M_{1}$$M_{2}$$M_{3}$ | $G_{1}$$do(G_{1}\leftarrow g_{1})$$G_{2}$$do(G_{2}\leftarrow g_{2})$$M_{1}$$M_{2}$$M_{3}$
---|---|---
Figure 6: Block-aligned representation when $\mathfrak{C}_{\bm{\mathrm{G}}}$
has causal connections. (left) An intervention on $G_{1}$ affects all
representation (displayed in blue), since $(M_{1},M_{3})$ are block-aligned to
$G_{1}$ and $M_{2}$ is aligned to $G_{2}$. (center) Conversely, an
intervention on $G_{2}$ only affects $M_{2}$, leaving the remaining
representations untouched. (right) Intervening on all ${\bm{\mathrm{G}}}$ has
the effect of isolating the corresponding aligned representations from other
interventions. In this case, intervening on $G_{2}$ removes the causal
connection with $G_{1}$, so that $M_{2}$ does not depend on the intervention
of $G_{1}$. Refer to 2 for further details.

Next, we formalize our observation that, thanks to block alignment,
interventions on ${\bm{\mathrm{G}}}$ are automatically mirrored on
${\bm{\mathrm{M}}}$:

###### Proposition 4

Given a block-wise aligned representation ${\bm{\mathrm{M}}}$ to
${\bm{\mathrm{G}}}$, it holds that for each distinct block
${\bm{\mathrm{M}}}_{\mathcal{K}}$ of representations ${\bm{\mathrm{M}}}$, an
intervention on ${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$ isolates
${\bm{\mathrm{M}}}_{\mathcal{K}}$ from interventions of other ground-truth
blocks ${\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}$. Moreover, distinct
interventions on ${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$ corresponds on average
to different interventions on ${\bm{\mathrm{M}}}_{\mathcal{K}}$.

Importantly, this means that the effect of an intervention on the whole
${\bm{\mathrm{G}}}$ isolates each block in ${\bm{\mathrm{M}}}$ from the
others, i.e., there is no explicit causal relation appearing in the learned
representation. This matches the intuition that an intervention on a specific
generative factor affects the corresponding block and removes the dependencies
on other blocks of the representation.

Summarizing, block alignment entails interventions to the ground-truth
concepts are mapped properly. At the same time, alignment between blocks
ensures the transformation $\alpha$ is simulatable, meaning that users can
understand changes happening to all of the variables involved. This is
sufficient to guarantee name transfer can be completed successfully in the
general case, assuming not too many factors are changed at a time.

###  4.6 Alignment and Causal Abstractions

One important observation is that the form of name transfer we have considered
is asymmetrical, in the sense that the user intervenes on its own
representation ${\bm{\mathrm{H}}}$ only, to then check how this impacts
${\bm{\mathrm{M}}}$. The other direction is not considered: it is not
necessary to consider how intervening on ${\bm{\mathrm{M}}}$ impacts
${\bm{\mathrm{M}}}$. This leads to the setup depicted in fig. 7 (right) in
which, given $\mathfrak{C}_{\bm{\mathrm{H}}}$, the effects of interventions on
$H_{i}$ are propagated to ${\bm{\mathrm{M}}}$ via a map
$\beta:{\bm{\mathrm{H}}}\mapsto{\bm{\mathrm{M}}}$, which may or may not be
block-aligned.

We now consider a scenario in which the SCM of the representation
$\mathfrak{C}_{\bm{\mathrm{M}}}$ is also provided171717In practice,
$\mathfrak{C}_{\bm{\mathrm{M}}}$ can be uncovered from data via causal
discovery (Pearl, 2009).  and the effects of interventions on
${\bm{\mathrm{M}}}$ can be propagated leveraging its structural assignments.

Ideally, we would expect that, as long as ${\bm{\mathrm{M}}}$ is block-aligned
to ${\bm{\mathrm{H}}}$, we can always find analogous post-interventional
effects when intervening on $H_{i}$ and on its aligned variable $M_{j}$. This
underlies a consistency condition between the two “worlds” that are described
with $\mathfrak{C}_{\bm{\mathrm{H}}}$ and $\mathfrak{C}_{\bm{\mathrm{M}}}$,
respectively, by requiring that they both lead to similar conclusions when
intervened in a equivalent manner. Clearly, this does not depend solely on the
nature of the map $\beta$ but also on the structure of the machine SCM
$\mathfrak{C}_{\bm{\mathrm{M}}}$.

The presence of a consistency property between
$\mathfrak{C}_{\bm{\mathrm{H}}}$ and $\mathfrak{C}_{\bm{\mathrm{M}}}$ is what
defines a causal abstraction (Beckers and Halpern, 2019; Beckers et al., 2020;
Rubenstein et al., 2017), cf. (Zennaro, 2022) for an overview. Causal
abstractions have been proposed to define (approximate) equivalence between
causal graphs and have recently been employed in the context of explainable AI
(Geiger et al., 2023b, a). The existence of a causal abstraction ensures two
systems are interventionally equivariant: interventions on one system can
always be mapped (modulo approximations) to equivalent interventions in the
other and lead to the same interventional distribution.

All causal abstractions check the consistency between two maps under the same
intervention $do({\bm{\mathrm{H}}}_{\mathcal{I}})$: one is defined by the
post-interventional distribution of $\mathfrak{C}_{\bm{\mathrm{H}}}$ that is
mapped on ${\bm{\mathrm{M}}}$ via $\beta$, the other one consists of first
matching on ${\bm{\mathrm{M}}}$ the correspondent action
$do({\bm{\mathrm{M}}}_{\mathcal{J}})$ and propagate it via
$\mathfrak{C}_{\bm{\mathrm{M}}}$. Intuitively, this means that, under $\beta$,
interventions on ${\bm{\mathrm{H}}}$ lead to the same conclusion as
interventions on ${\bm{\mathrm{M}}}$. We formalize this idea from constructive
causal abstractions181818Existing works on causal abstractions (Beckers and
Halpern, 2019; Geiger et al., 2023b) do not impose the map between values or
interventions are simulatable, meaning that even if
$\mathfrak{C}_{\bm{\mathrm{M}}}$ is a causal abstraction of
$\mathfrak{C}_{\bm{\mathrm{H}}}$, it may be impossible for users to understand
the mapping between the two. (Geiger et al., 2023b) by adapting it to the case
where ${\bm{\mathrm{H}}}$ and ${\bm{\mathrm{M}}}$ are connected by block-
alignment:

######  Definition 9 ($\beta$-Aligned Causal Abstraction)

The $\mathfrak{C}_{\bm{\mathrm{M}}}$ is a causal abstraction of
$\mathfrak{C}_{\bm{\mathrm{H}}}$ under block-alignment $\beta$ if, for all
possible interventions
$do({\bm{\mathrm{H}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{h}}}_{\mathcal{I}})$
with ${\bm{\mathrm{H}}}_{\mathcal{I}}\subseteq{\bm{\mathrm{H}}}$, the
following diagram commutes:

${do({\bm{\mathrm{H}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{h}}}_{\mathcal{I}})}$${p({\bm{\mathrm{H}}}\mid{do({\bm{\mathrm{H}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{h}}}_{\mathcal{I}})})}$${do({\bm{\mathrm{M}}}_{\mathcal{J}}\leftarrow{\bm{\mathrm{m}}}_{\mathcal{J}})}$${p({\bm{\mathrm{M}}}\mid{do({\bm{\mathrm{M}}}_{\mathcal{J}}\leftarrow{\bm{\mathrm{m}}}_{\mathcal{J}})})}$$\scriptstyle{\mathfrak{C}_{\bm{\mathrm{H}}}}$$\scriptstyle{\beta}$$\scriptstyle{\beta_{*}}$$\scriptstyle{\mathfrak{C}_{\bm{\mathrm{M}}}}$
(18)

where $\beta_{*}$ denotes the push-forward operation applied to the
probability
$p({\bm{\mathrm{H}}}\mid{do({\bm{\mathrm{H}}}_{\mathcal{I}}\leftarrow{\bm{\mathrm{h}}}_{\mathcal{I}})})$,
and $\mathcal{J}=\Pi^{-1}(\mathcal{I})$ is the pre-image of $\mathcal{I}$
under $\Pi$.

In other words, aligned causal abstractions extend block alignment by
enforcing a symmetrical consistency condition over interventions when both
SCMs $\mathfrak{C}_{\bm{\mathrm{H}}}$ and $\mathfrak{C}_{\bm{\mathrm{M}}}$ are
known: interventions on ${\bm{\mathrm{M}}}$ have analogues on
${\bm{\mathrm{H}}}$ and vice-versa. This becomes relevant in situations where
the user cannot parse the effect of an intervention on $H_{i}$ on the input
${\bm{\mathrm{X}}}$, i.e., they do not have access to $p({\bm{\mathrm{X}}}\mid
do(H_{i}\leftarrow h_{i}))$, and they are left to validate the effects of
their actions through $\beta$. In this case, leveraging on the SCM
$\mathfrak{C}_{\bm{\mathrm{M}}}$, the user can check how the mirrored
intervention on $M_{j}$ spreads in the machine representations, and compare it
with the corresponding representations given by $\beta$ when the intervention
is propagated on the user’s factors ${\bm{\mathrm{H}}}_{-i}$.

Therefore, while a map $\beta$ being aligned is a necessary condition, it is
not sufficient to guarantee a successful name transfer if
$\mathfrak{C}_{\bm{\mathrm{M}}}$ is highly dissimilar from
$\mathfrak{C}_{\bm{\mathrm{H}}}$. We show this situation explicitly where,
despite having alignment between the user and the machine, the consistency
condition in eq. 18 does not hold.

$H_{1}$$M_{1}$$\mathfrak{C}_{\bm{\mathrm{H}}}$: $\mathfrak{C}_{\bm{\mathrm{M}}}$: $H_{2}$$M_{2}$ | $\mathfrak{C}_{{\bm{\mathrm{H}}}\to{\bm{\mathrm{M}}}}$: $H_{1}$$M_{1}$$H_{2}$$M_{2}$
---|---
Figure 7: Absence of Aligned Causal Abstraction. (left) The user’s
$\mathfrak{C}_{\bm{\mathrm{H}}}$ incorporates a causal connection between
$H_{1}$ to $H_{2}$, while the machine one $\mathfrak{C}_{\bm{\mathrm{M}}}$
presents no causal connections. (right) The total SCM
$\mathfrak{C}_{{\bm{\mathrm{H}}}\to{\bm{\mathrm{M}}}}$ of user’s and machine’s
concepts resulting from an aligned map
$\beta:{\bm{\mathrm{H}}}\to{\bm{\mathrm{M}}}$ (in blue). Refer to 3 for
further discussion.

###### Example 3

We consider two SCMs, one over user variables $\mathfrak{C}_{\bm{\mathrm{H}}}$
and one over the machine ones $\mathfrak{C}_{\bm{\mathrm{M}}}$. As shown in
fig. 7 (left), the two SCMs have a different structure and for ease of
reference we refer to $H_{1}$ and $M_{1}$ as the temperature variable and to
$H_{2}$ and $M_{2}$ as the color variable. Despite the different structure, we
suppose $M_{1}$ and $M_{2}$ are aligned to $H_{1}$ and $H_{2}$, respectively,
via an aligned map $\beta$. We indicate the overall causal graph as
$\mathfrak{C}_{{\bm{\mathrm{H}}}\to{\bm{\mathrm{M}}}}$, see fig. 7 (right).

We can now check that $\mathfrak{C}_{\bm{\mathrm{M}}}$ is not an aligned
abstraction of $\mathfrak{C}_{\bm{\mathrm{H}}}$ under $\beta$. In fact,
intervening on $H_{1}$ leads to different results on
$\mathfrak{C}_{\bm{\mathrm{M}}}$ and
$\mathfrak{C}_{{\bm{\mathrm{H}}}\to{\bm{\mathrm{M}}}}$. For the former,
changing the temperature amounts to modify only the corresponding variable
$M_{1}$ and does not affect $M_{2}$, as evident in fig. 7 (left). Conversely,
a change in the temperature under alignment corresponds also to a change in
color for the variable $M_{2}$, as depicted in fig. 7 (right). The two
interventional effects, hence, do not coincide and
$\mathfrak{C}_{\bm{\mathrm{M}}}$ is not an aligned causal abstraction of
${\bm{\mathrm{H}}}$.

##  5 Discussion and Limitations

Our work provides a crisp requirement that machine representations should
satisfy to ensure interpretability, namely alignment with the human’s concept
vocabulary. Next, we address important issues arising from this requirement.

###  5.1 Is Perfect Alignment Sufficient and Necessary?

It is natural to ask whether perfect alignment is a sufficient and necessary
condition for interpretability of machine concepts. Recall that alignment is
born out of two desiderata. The first one is that of subjectivity: a concept
is understandable to a particular human observer, with different observers
having different expertise and knowledge. This is captured by the human’s
vocabulary ${\bm{\mathrm{H}}}$ in our definition. The second one is that of
guaranteeing that machine and human concepts sharing the same name also share
the same semantics, translated into the desideratum that whenever a human
concept changes the human can anticipate how this will change the machine
representation. For instance, if the human and the machine see the same
picture of a dog, the human can easily figure out what concept encodes the
notion of “dog” and how it would change if they were to delete the dog from
the picture.191919This last point takes into account, at least partially, the
limited cognitive processing abilities of human agents.

Is alignment sufficient? Simply ensuring that two agents share aligned
representations does not automatically entail that symbolic communication will
be successful. For instance, a human observer may misinterpret a machine
explanation built out of aligned concepts simply due to inattention,
confusion, or information overload. These are all important elements in the
equation of interpretability, and we do not intend to dismiss them. The way in
which information is presented is about as important as the contents of the
information being conveyed. The problem of designing interfaces that ensure
the presentation is engaging and easy to understand is however beyond the
scope of this paper. This does not impact our core message, that is, that lack
of alignment can severely hamper communication and that therefore approaches
for learning and evaluating conceptual representations should be designed with
this requirement in mind.

Is alignment necessary? We also point out that perfect alignment is not
strictly necessary, for two reasons. First, it is enough that alignment holds
only approximately. Slight differences in semantics between machine and human
concepts are unlikely to have major effects on communication. This is
compatible with the empirical observation that people can often successfully
communicate even without fully agreeing on the semantics of the words they
exchange (Marti et al., 2023). In practice, the degree of misalignment, and
its impact of the communication, can be defined and measured, at which point
the maximum allowed misalignment becomes an application-specific variable.
Second, it may not be necessary that alignment holds everywhere. If two agents
exchange only a subset of possible messages (e.g., explanations), concepts not
appearing in those messages need not be aligned. For instance, ensuring a CBM
classifying apples as ripe or not to be interpretable only requires the
concepts appearing in its explanations to be aligned, and possibly only those
values that actually occur in the explanations (e.g.,
$\texttt{color}=\texttt{red}$ but not $\texttt{color}=\texttt{blue}$). This
can be understood as a more lax form of alignment applying only to a certain
subset of (values of) the generative factors
${\bm{\mathrm{g}}}_{\mathcal{I}}$, e.g., those related to apples. It is
straightforward to relax 6 in this sense by restricting it to a subset of the
support of $p^{*}({\bm{\mathrm{G}}}_{\mathcal{I}})$ from which the inputs
${\bm{\mathrm{X}}}$ are generated, as these constrain the messages that the
two agents can exchange.

###  5.2 Measuring Alignment

While there exist several metrics for measuring interpretability of concepts
(discussed in section 6.4), here we are concerned with techniques for
assessing alignment.

Considering the relation between alignment and disentanglement (D1), one
option is to leverage one of the many measures of disentanglement proposed in
the literature (Zaidi et al., 2020). The main issues is that most of them
provide little information about how simple the map $\alpha$ (D2) is and as
such they cannot be reused as-is. However, for the disentangled case (cf.
section 4.1), Marconato et al. (2022) noted that one can measure alignement
using the linear DCI (Eastwood and Williams, 2018). Essentially, this metric
checks whether there exists a linear regressor that, given
${\bm{\mathrm{m}}}_{\mathcal{J}}$, can predict
${\bm{\mathrm{g}}}_{\mathcal{I}}$ with high accuracy, such that each $M_{j}$
is predictive for at most one $G_{i}$. In practice, doing so involves
collecting a set of annotated pairs
$\\{({\bm{\mathrm{m}}}_{\mathcal{J}},{\bm{\mathrm{g}}}_{\mathcal{I}})\\}$,
where the $m_{j}$’s and $g_{i}$’s are rescaled in $[0,1]$, and fitting a
linear regressor on top of them using $L_{1}$ regularization. DCI then
considers the (absolute values of the) regressor coefficients
$B\in\mathbb{R}^{|\mathcal{J}|\times|\mathcal{I}|}$ and evaluates average
dispersion of $B_{j:}$ for each machine representation $M_{j}$. In short, if
each $M_{j}$ predicts only a single $G_{i}$, and with high accuracy, then
linear DCI is maximal. The key insight is that the existence of such a linear
map implies both disentanglement (D1) and monotonicity (D2), and therefore
also alignment. The main downside is that the converse does not hold, that is,
linear DCI cannot account for non-linear monotonic relationships.

The alternative we advocate is that of decoupling the measurement of D1 and
D2, and to leverage causal notions for the former. D1 can for instance be
measured using the interventional robustness score (IRS) (Suter et al., 2019),
an empirical version of $\mathsf{EMPIDA}$ (2) that – essentially – measures
the average effect of interventions on ${\bm{\mathrm{G}}}_{\mathcal{I}}$ on
the machine representation. Alternatives include, for instance, DCI-ES
Eastwood et al. (2022), which can better capture the degree by which factors
are mixed and the mutual information gap (MIG) (Chen et al., 2018). These
metrics allow to establish an empirical map $\pi$ between indices of the human
and machine representations, using which it is possible to evaluate D2
separately. One option is that of evaluating Spearman’s rank correlation
between the distances:

$|g_{i}-g_{i}^{\prime}|^{2}\quad\text{and}\quad\|\mathbb{E}[{\bm{\mathrm{M}}}_{\mathcal{J}}\mid
do(G_{i}\leftarrow g_{i})]-\mathbb{E}[{\bm{\mathrm{M}}}_{\mathcal{J}}\mid
do(G_{i}\leftarrow g_{i}^{\prime})]\|^{2}_{2}$ (19)

for interventions $g_{i}$ and $g_{i}^{\prime}$, leaving
${\bm{\mathrm{G}}}_{-i}$ fixed, for each $i\in\mathcal{I}$ and multiple
traversals $(g_{i},g_{i}^{\prime})$.

Unfortunately, none of the existing metrics are suited for non-disentangled
generative factors ${\bm{\mathrm{G}}}_{\mathcal{I}}$ or human representations
${\bm{\mathrm{H}}}$, which are central for alignment in the block-wise
(section 4.4) and general (section 4.5) cases. We leave an in-depth study of
more generally applicable metrics to future work.

###  5.3 Consequences for Concept-based Explainers

Recall that CBEs explain the predictions of black-box models by extracting
interpretable concepts $\hat{{\bm{\mathrm{H}}}}$ from the model’s internal
representation ${\bm{\mathrm{M}}}$ and then evaluating their contribution to
the prediction (cf. section 3.1). In this case, the requirement is that
$\hat{{\bm{\mathrm{H}}}}$ is aligned to the human’s concept vocabulary
${\bm{\mathrm{H}}}$ – irrespective of how the former is extracted. Notice that
alignment is orthogonal to faithfulness, in the sense that an aligned
representation can be unfaithful to the model, and a faithful representation
misaligned with the human. In other words, alignment is a property of the map
from ${\bm{\mathrm{H}}}$ to $\hat{{\bm{\mathrm{H}}}}$, while faithfulness is a
property of the map between ${\bm{\mathrm{M}}}$ and $\hat{{\bm{\mathrm{H}}}}$.

If the mapping from ${\bm{\mathrm{M}}}$ to $\hat{{\bm{\mathrm{H}}}}$ is
invertible, then it is always possible map back and forth – in a lossless
manner – from the machine representations ${\bm{\mathrm{M}}}$ to the surrogate
$\hat{{\bm{\mathrm{H}}}}$. This is a solid basis for faithfulness: whatever
information is conveyed by an explanation built on $\hat{{\bm{\mathrm{H}}}}$
can always be cast in terms of the machine representation itself,202020The
resulting explanation may no longer be simple or understandable, but it still
contains all the information of the original message. and that whatever
relation the latter has with the prediction can be mapped in terms of human
concepts.

In the general case, however, it is non-trivial to find a suitable invertible
function. Suppose the user provides the machine with annotated examples
$({\bm{\mathrm{x}}}_{i},{\bm{\mathrm{h}}}_{i})$ and that these are used – as
is common with supervised CBEs, see section 6.2 – to learn the mapping from
${\bm{\mathrm{M}}}$ to $\hat{{\bm{\mathrm{H}}}}$. Ensuring that this is
invertible requires potentially an enormous amount of examples. To see this,
consider a simple case in which the human concepts ${\bm{\mathrm{H}}}$ are
binary and disentangled and that ${\bm{\mathrm{M}}}$ and ${\bm{\mathrm{H}}}$
are related by a (possibly complex) invertible mapping that is not an
alignment. Even in this ideal case, it might take up to $2^{\ell}$ examples –
where $\ell$ is the dimension of ${\bm{\mathrm{H}}}$ – to align the two
representations, as this involves finding the correct permutation from
${\bm{\mathrm{M}}}$ to ${\bm{\mathrm{H}}}$. Alignment can help in this regard.
In fact, if ${\bm{\mathrm{M}}}$ is aligned to ${\bm{\mathrm{H}}}$, the number
of required examples scales as $\mathcal{O}(\ell)$, because a single
intervention to each user concept $H_{i}$ is sufficient to find the
corresponding aligned element $M_{j}$.

In summary, not only do unaligned (black-box) models imply CBEs require more
supervision on the user concepts to acquire a invertible transformation
ensuring faithfulness, but also it is likely that the representation
${\bm{\mathrm{M}}}$ mixes together the interpretable factors
${\bm{\mathrm{G}}}_{\mathcal{I}}$ with the non-interpretable ones
${\bm{\mathrm{G}}}_{-\mathcal{I}}$, making it more difficult to extract a
concepts $\hat{\bm{\mathrm{H}}}$ aligned to ${\bm{\mathrm{H}}}$.

###  5.4 Consequences for Concept-based Models

As discussed in section 6.1, most CBMs acquire concepts using a variety of
heuristics that do not guarantee alignment. To the best of our knowledge,
GlanceNets (Marconato et al., 2022) are the only CBM that explicitly optimizes
for alignment, and as such avoids concept leakage. They do so by combining a
variational auto-encoder mapping from the input ${\bm{\mathrm{X}}}$ to a
machine representation
${\bm{\mathrm{M}}}=({\bm{\mathrm{M}}}_{\mathcal{J}},{\bm{\mathrm{M}}}_{-\mathcal{J}})$
where only the first partition is used for prediction. These are computed
using a simple linear layer, as is customary. The variational-auto encoder is
trained with some amount of concept-level annotations. This encourages both
disentanglement (Locatello et al., 2019) and monotonicity – and hence
alignment – for in-distribution data. In turn, this also prevents concept
leakage. In order to avoid leakage for out-of-distribution data, GlanceNets
also implement an open-set recognition step (Sun et al., 2020). This is
responsible for detecting inputs encoding concepts that have never been
observed during training. Whenever these are detected, GlanceNets refuse to
output a prediction for them, thus avoiding leakage altogether.

From our perspective, GlanceNets have two major downsides. First, they are
designed to seek alignment with respect to the generative factors underlying
the observations. As we argued, however, interpretability requires alignment
with respect to the human’s concept vocabulary. Second, GlanceNets require a
moderate but non-trivial number of annotations. How to acquire them from the
human observer remains an open problem, discussed in section 5.5.

Summarizing, GlanceNets could be repurposed for solving alignment in the
disentangled case discussed in section 4.1 by combining them with a suitable
annotation elicitation procedure. They are however insufficient to solve
disentanglement when the ground-truth concepts are not disentangled, and new
solutions will be necessary to tackle these more complex and realistic
settings.

###  5.5 Collecting Human Annotations

Both metrics and learning strategies for alignment require some amount of
annotations for the human factors ${\bm{\mathrm{H}}}$. This is a core
requirement related to the subjective nature of interpretability. One option
is that of distributing the annotation effort among crowd-workers, which
however is impractical for prediction tasks that require specific types of
expertise, like medical diagnosis. An alternative is that of gathering
together annotations from different online resources of large language models
(Oikarinen et al., 2022). Doing so, however, can lead to a lack of
completeness (a necessary concept might be missing) and ambiguity (concepts
annotations might mix together different views or meanings). This kind of
supervision cannot guarantee alignment to a specific human observer.

Reducing the annotation effort for personalized supervision is challenging.
Under the assumption that of leveraging generic concept annotations obtained
using the above methods to pre-train the concept extractor, and then fine-tune
the resulting model using a small amount of personalized annotations. This
strategy can save annotation effort as long as the generic annotations contain
most of the information necessary to retrieve the observer’s concepts. An
alternative is to leverage concept-level interactive learning (Lage and Doshi-
Velez, 2020; Chauhan et al., 2023), to request annotations only for those
concepts that are less aligned. Naturally, one might also consider combining
these two strategies, that is, interleaving fine-tuning with interactive
learning, for additional gains. How to estimate alignment (or some lower bound
thereof) in absence of full concept annotations is however an open research
question and left to future work.

##  6 Related Work

While concepts lie at the heart of AI (Muggleton and De Raedt, 1994), the
problem of acquiring interpretabile concepts has historically been neglected
in representation learning (Bengio et al., 2013). Recently concepts have
regained popularity in many areas of research, including explainable AI
(Guidotti et al., 2018), neuro-symbolic AI (De Raedt et al., 2020), and
causality (Pearl, 2009; Schölkopf et al., 2021), yet most concept acquisition
strategies developed in these areas are only concerned with task accuracy,
rather than interpretability. Next, we briefly overview strategies for
acquiring interpretable representations and highlight their shortcomings for
properly solving human-interpretable representation learning.

###  6.1 Unsupervised Approaches

A first group of strategies learn concepts directly from unlabeled data. Well-
known theoretical results in deep latent variable models cast doubts on the
possibility of acquiring representations satisfying any property of interest –
including disentanglement and interpretability – in a fully unsupervised
manner in absence of a strong architectural bias (Locatello et al., 2019;
Khemakhem et al., 2020b). This stems from the fact that, as long as the
concept extraction layers are “flexible enough” (i.e., have no strong
architectural bias), predictors relying interpretable and uninterpretable
concepts can achieve the very same accuracy (or likelihood) on both the
training and test sets. As a consequence, unsupervised strategies that only
maximize for accuracy cannot guarantee interpretability unless they are guided
by an appropriate bias. The main challenge is determining what this bias
should be.

Several, mutually incompatible alternatives have been proposed. Unsupervised
CBEs discover concepts in the space of neuron activations of a target model.
One common bias is that concepts can be retrieved by performing a linear
decomposition of the machine’s representation (Fel et al., 2023b). Specific
techniques include k-means (Ghorbani et al., 2019), principal component
analysis (Graziani et al., 2023), and non-negative matrix factorization (Zhang
et al., 2021; Fel et al., 2023a). Concept responsibility is then established
via feature attribution methods.

Two common biases used in CBMs are sparsity and orthonormality. Self-
Explainable Neural Networks (Alvarez-Melis and Jaakkola, 2018) encourage the
former by pairing an autoencoder architecture for extracting concepts from the
input together with a (simulatable (Lipton, 2018)) task-specific prediction
head, and then combining a cross-entropy loss with a penalty term encouraging
concepts to have sparse activation patterns. Concept Whitening (Chen et al.,
2020) implements a special bottleneck layer that ensures learned concepts are
orthogonal, so as to minimize mutual information between them and facilitate
acquiring concepts with disjoint semantics, as well as normalized within
comparable activation ranges. The relationship between sparsity,
orthonormality, and interpretability is however unclear.

Based on the observation that humans tend to reason in terms of concrete past
cases (Kim et al., 2016), other CBMs constrain concepts to capture salient
training examples or parts thereof, i.e., prototypes. Methods in this group
include Prototype Classification Networks (Li et al., 2018), Part-Prototype
Networks (Chen et al., 2019), and many others (Rymarczyk et al., 2021; Nauta
et al., 2021; Singh and Yow, 2021; Davoudi and Komeili, 2021). At a high
level, they all memorize one or more prototypes (i.e., points in latent space)
that match training examples of their associated class only. Predictions are
based on the presence or absence of a match with the learned prototypes. The
interpretability of this setup has however been called into question (Hoffmann
et al., 2021; Xu-Darme et al., 2023). The key issue is the matching step,
which is carried out in latent space. The latter is generally
underconstrained, meaning that prototypes can end up matching parts of
training examples that carry no useful semantics (e.g., arbitrary combinations
of foreground and background) as long as doing so yields high training
accuracy.

None of these approaches takes the human’s own concept vocabulary
${\bm{\mathrm{H}}}$ into account.

###  6.2 Supervised Strategies

A second family of approaches leverages concept annotations (or some form of
weak supervision). Among supervised CBEs, Net2vec (Fong and Vedaldi, 2018)
defines linear combinations of convolutional filters, and fits a linear model
to decide whether their denoised saliency maps encode a given concept or not,
yielding a binary segmentation mask. TCAV (Kim et al., 2018) defines concepts
as directions – or concept-activation vectors (CAVs) – in latent space. These
are obtained by adapting the parameters of per-concept linear classifiers
trained on a separate densely annotated data set to the machine’s embedding
space. Concept attributions are proportional to the degree by which changing
their activations affects the prediction. Zhou et al. (2018) also relies on
CAVs, but computes explanation by solving an optimization problem. A second
group of supervised CBEs makes use of non-linear maps instead (Kazhdan et al.,
2020; Gu and Tresp, 2019; Esser et al., 2020). For instance, CME (Kazhdan et
al., 2020) uses all activations of the model to learn categorical concepts via
semi-supervised multi-task learning, while INN (Esser et al., 2020) fits a
normalizing flow from the machine representation to the concepts so as to
guarantee their relationship is bijective. Similarly, supervised CBMs like
Concept-Bottleneck Models (Koh et al., 2020), Concept Whitening (Chen et al.,
2020), and GlanceNets (Marconato et al., 2022), among others (Yuksekgonul et
al., 2022; Sawada and Nakamura, 2022; Zarlenga et al., 2022) define a loss
training penalty, for instance a cross-entropy loss, encouraging the extracted
concepts to predict the annotations.

This solution seems straightforward: there is no more direct way than concept
supervision to guide the model toward acquiring representations with the
intended semantics. It also circumvents the negative theoretical results
outlined in section 6.1.

However, models that accurately match the supervision do not necessarily
satisfy content-style separation or allow to have disentangled
representations, which – as discussed in section 4.3 – would lead to a non-
negligible amount of concept leakage (Margeloiu et al., 2021; Mahinpei et al.,
2021). In contrast, alignment explicitly takes both properties into account.
Another major issue is the supervision itself, which is frequently obtained
from general sources rather than from the human observer themselves, meaning
the learned concepts may not be aligned to the concept vocabulary of the
latter. Two notable exceptions are the interactive concept learning approaches
of Lage and Doshi-Velez (2020) and of Erculiani et al. (2023), which are
however unconcerned with concept leakage.

To the best of our knowledge, GlanceNets (Marconato et al., 2022) are the only
CBM that explicitly optimizes for alignment, and as such avoid leakage, yet
they do so with respect to generative factors rather than human concepts. As
discussed in section 5.4, however, GlanceNets can in principle be adapted to
solve human-interpretable representation learning by combining them with a
suitable annotation acquisition strategy. We plan to pursue this possibility
in future work.

###  6.3 Disentanglement

Another relevant area of research is that on learning disentangled
representations. Here, the goal is to uncover “meaningful”, independent
factors of variation underlying the data (Higgins et al., 2016, 2018;
Locatello et al., 2019), with the hope that these are also interpretable
(Bengio et al., 2013). Most current learning strategies rely on extensions of
variational auto-encoders (VAEs) (Kingma and Welling, 2014; Higgins et al.,
2016; Kim and Mnih, 2018; Chen et al., 2018; Esmaeili et al., 2019; Rhodes and
Lee, 2021). As anticipated in section 6.1, unless suitable architectural bias
is provided, unsupervised learning cannot guarantee the learned
representations are disentangled. Motivated by this, follow-up works seek
disentanglement via either concept supervision (Locatello et al., 2020a), weak
supervision (Shu et al., 2020; Locatello et al., 2020b), and other techniques
(Lachapelle et al., 2022; Horan et al., 2021; Stammer et al., 2022).
Disentanglement however is unconcerned with the human’s concept vocabulary,
and furthermore it is weaker than alignment, in that is does not readily
support name transfer.

Independent component analysis (ICA) also seeks to acquire independent factors
of variation (Comon, 1994; Hyvärinen et al., 2001; Naik and Kumar, 2011).
These assume the generative factors are independent from each other and
determine an observation via an injective or invertible map. The objective of
ICA is to recover the generative factors from the observations. While the
linear case is well understood (Comon, 1994), the non-linear case is arguably
more difficult. It was shown that identifying the ground truth factor is
impossible in the unsupervised setting (Hyvärinen and Pajunen, 1999). This is
analogous to the results mentioned in section 6.1 and in fact a formal link
between deep latent variable models and identifiability has recently been
established. (Khemakhem et al., 2020b). On the positive side, it is possible
to show that providing auxiliary supervision on the factors guarantees
identification up to permutation and negation, a property known as strong
identifiability. Weak identifiability (Buchholz et al., 2022) relaxes it
whereby the generative factors are recovered up to a transformation of the
form $A{\bm{\mathrm{g}}}+{\bm{\mathrm{b}}}$, where
$rank(A)\geq\min(\dim{\bm{\mathrm{G}}},\dim{\bm{\mathrm{M}}})$ and
${\bm{\mathrm{M}}}$ is the machine representation and ${\bm{\mathrm{b}}}$ is
an offset. Hyvarinen and Morioka (2017) also contemplate identifiability up to
element-wise non-linearities, that it, given by the class of transformations
$A\sigma[{\bm{\mathrm{g}}}]+{\bm{\mathrm{b}}}$, where $\sigma$ can be a non-
linear. If $\sigma$ is restricted to be monotonic and $A$ is an element-wise
transformation, according to condition D1 in 6, then this form of
identifiability matches that of alignment in the disentangled case. However,
this formulation refers to identification of the generative factors, while
alignment is defined specifically in terms of human concepts. Moreover, we do
not assume to the map from human to machine concepts to be injective, nor to
be exact.

###  6.4 Metrics of Concept Quality

Several metrics have been proposed for assessing the quality of extracted
concepts and of explanations built on them. Standard measures include accuracy
and surrogates thereof (Kim et al., 2018), Jaccard similarity (Fong and
Vedaldi, 2018), sparsity, stability and ability to reconstruct the model’s
internal representation (Fel et al., 2023b), and the degree by which concepts
constitute a sufficient statistics for the prediction (Yeh et al., 2020). We
refer to (Schwalbe, 2022) for an overview. These metrics, however, either
entirely neglect the role of the human observer – in that concept annotations
are either not used or not obtained from the observer themselves – or fail to
account for disentanglement and concept leakage. Alignment fills these gaps.
Recently, two new metrics have been proposed to measure the concept impurity
across individual learnt concepts and among sets of representations (Zarlenga
et al., 2023), but the relation with alignment has not been uncovered yet.

There also exist a number of metrics for measuring disentanglement, such as
$\beta$-VAE score (Higgins et al., 2016), Factor-VAE score (Kim and Mnih,
2018), mutual information gap (Chen et al., 2018), DCI (Eastwood and Williams,
2018), and IRS (Suter et al., 2019). DCI provides also information about the
informativeness of its estimate, and – following (Marconato et al., 2022), it
can be repurposed to measure a form of alignment where the $\mu$
transformations are linear 6. Suter et al. (2019) propose $\mathsf{EMPIDA}$ to
analyze disentanglement from a causal perspective, upon which we base the
construction of alignment. As mentioned in section 5.2, these metrics can be
used to evaluate D1 in the definition of alignment, and therefore alignment
itself when paired with a metric for measuring the complexity of $\alpha$
(D2). Their properties are extensively discussed in Zaidi et al. (2020).

###  6.5 Neuro-Symbolic Architectures

The decomposition between low-level perception – that is, mapping inputs to
concepts, also known as neural predicates in this setting – and high-level
inference outlined in section 3.1 applies also to many neuro-symbolic (NeSy)
models. Examples include DeepProblog (Manhaeve et al., 2018), Logic Tensor
Networks (Donadello et al., 2017), and related architectures (Diligenti et
al., 2017; Fischer et al., 2019; Giunchiglia and Lukasiewicz, 2020; Yang et
al., 2020; Huang et al., 2021; Marra and Kuželka, 2021; Ahmed et al., 2022;
Misino et al., 2022; Winters et al., 2022; van Krieken et al., 2022; Ciravegna
et al., 2023). The biggest differences between CBMs and NeSy architectures is
how they implement the top layer: the former rely on simulatable layers, while
the latter on reasoning layers that take prior symbolic knowledge into account
and are not necessarily simulatable.

Recent works (Marconato et al., 2023a, b) showed that learning a NeSy model
consistent with prior knowledge using only label supervision is insufficient
to guarantee the neural predicates capture the intended semantics. For
instance, it is not uncommon that NeSy architectures attain high prediction
accuracy by acquiring neural predictes that encode information about distinct
and unrelated concepts. Interpretability of the neural predicates however also
requires alignment, meaning that our results apply to these NeSy architectures
as well.

##  7 Conclusion

Motivated by the growing importance of interpretable representations for both
post-hoc and ante-hoc explainability, we have introduced and studied the
problem of human-interpretable representation learning. Our key intuition is
that concepts are interpretable only as long as they support symbolic
communication with an interested human observer. Based on this, we developed a
formal notion of alignment between distributions, rooted in causality, that
ensures concepts can support symbolic communication and that applies to both
post-hoc concept-based explainers and concept-based models. In addition, we
clarified the relationship between alignment and the well-known notions of
disentanglement, illustrating why the latter is not enough for
interpretability, and uncovered a previously unknown link between alignment
and concept leakage. Finally, looking at alignment in the most general case,
we also unearthed its link to causal abstractions, which further cements the
link between interpretability and causality and that we plan to expand on in
future work. With this paper, our aim is that of bridging the gap between the
human and the algorithmic sides of interpretability, with the hope of
providing a solid, mathematical ground on which new research on human-
interpretable representation learning can build.

## Acknowledgements

We acknowledge the support of the MUR PNRR project FAIR - Future AI Research
(PE00000013) funded by the NextGenerationEU. The research of ST and AP was
partially supported by TAILOR, a project funded by EU Horizon 2020 research
and innovation programme under GA No 952215.

##  Appendix A Proofs

###  A.1 Proof of Proposition 1

The proof requires to average over the confounds ${\bm{\mathrm{C}}}$,
encompassing the general case where different $G$’s may be correlated. To this
end, we define the distributions
$p({\bm{\mathrm{g}}})=\mathbb{E}_{{\bm{\mathrm{C}}}}[p({\bm{\mathrm{G}}})]$
and $p({\bm{\mathrm{G}}}\mid do(G_{i}\leftarrow
g_{i}))=\mathbbm{1}\\!\left\\{G_{i}=g_{i}\right\\}\mathbb{E}_{\bm{\mathrm{C}}}[p({\bm{\mathrm{G}}}_{-i}\mid{\bm{\mathrm{C}}})]$.

The proof is split into two parts: (i) proving that D1 implies
disentanglement, and (ii) the other way around.

(i) Assume that D1 holds. Then, the conditional distribution of
${\bm{\mathrm{M}}}$ can be written as:

$\textstyle
p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}})=\prod_{j\in\mathcal{J}}p_{\theta}(m_{j}\mid
g_{\pi(j)})$ (20)

We proceed to show that eq. 20 is disentangled in
$({\bm{\mathrm{G}}}_{\mathcal{I}},{\bm{\mathrm{M}}}_{\mathcal{J}})$. For each
$j\in\mathcal{J}$, it holds that the minimum value of
$\mathsf{EMPIDA}(G_{i},M_{j})$ is obtained when $i=\pi(j)$. That is because:

$\displaystyle p_{\theta}(M_{j}\mid do(G_{\pi(j)}\leftarrow g_{\pi(j)}))$
$\displaystyle=\mathbb{E}_{{\bm{\mathrm{g}}}_{-\pi(j)}}[p_{\theta}(M_{j}\mid
g_{\pi(j)})]$ (21) $\displaystyle p_{\theta}(M_{j}\mid do(G_{\pi(j)}\leftarrow
g_{\pi(j)},{\bm{\mathrm{G}}}_{-\pi(j)}\leftarrow{\bm{\mathrm{g}}}_{-\pi(j)}))$
$\displaystyle=p_{\theta}(M_{j}\mid g_{\pi(j)})$

Note that the first distribution is independent of
${\bm{\mathrm{g}}}_{-\pi(j)}$, so it is equivalent to the latter. Hence,
$\mathsf{EMPIDA}(G_{\pi(j)},M_{j})$ vanishes $\forall j\in\mathcal{J}$,
yielding the claim.

(ii) Let now ${\bm{\mathrm{M}}}_{\mathcal{J}}$ be disentangled with respect to
${\bm{\mathrm{G}}}_{\mathcal{I}}$, that is:

$\textstyle\max_{j\in\mathcal{J}}\min_{i\in\mathcal{I}}\mathsf{EMPIDA}(G_{i},M_{j})=0$
(22)

which is verified iff it holds that
$\min_{i\in\mathcal{I}}\mathsf{EMPIDA}(G_{i},M_{j})=0$ for all $j$. We now
proceed by contradiction to show that vanishing $\mathsf{EMPIDA}$ is only
consistent with D1. Suppose there exist at least one $j\in\mathcal{J}$ such
that:

$\alpha({\bm{\mathrm{m}}}_{\mathcal{J}})_{j}=\mu_{j}({\bm{\mathrm{g}}}_{-\mathcal{I}},N_{j})$
(23)

where $\mathcal{K}\subseteq\mathcal{I}$ containing at least two elements.
Therefore, the probability distribution for $M_{j}$ can be written in general
as $p(m_{j}\mid{\bm{\mathrm{g}}}_{\mathcal{K}})$. Plugging this condition in
the evaluation of $\mathsf{EMPIDA}$ we obtain for every $k\in\mathcal{K}$:

$\displaystyle p(M_{j}\mid do(G_{k}\leftarrow g_{k}))$
$\displaystyle=\mathbb{E}_{{\bm{\mathrm{g}}}_{\mathcal{K}\setminus\\{k\\}}}[p_{\theta}(m_{j}\mid
g_{k},{\bm{\mathrm{g}}}_{\mathcal{K}\setminus\\{k\\}})]$ (24) $\displaystyle
p(M_{j}\mid do(G_{k}\leftarrow
g_{k},{\bm{\mathrm{G}}}_{-k}\leftarrow{\bm{\mathrm{g}}}^{\prime}_{-k}))$
$\displaystyle=p_{\theta}(m_{j}\mid
g_{k},{\bm{\mathrm{g}}}^{\prime}_{\mathcal{K}\setminus\\{k\\}})$

Then, the two distributions coincide, and $\mathsf{EMPIDA}$ is zero, iff there
exists a $k\in\mathcal{K}$ such that all possible interventions
${\bm{\mathrm{G}}}_{\mathcal{K}\setminus\\{k\\}}\leftarrow{\bm{\mathrm{g}}}^{\prime}_{\mathcal{K}\setminus\\{k\\}}$
do not deviate from the expected distribution, formally:

$\forall{\bm{\mathrm{g}}}^{\prime}_{\mathcal{K}\setminus\\{k\\}}\quad
p(m_{j}\mid
g_{k},{\bm{\mathrm{g}}}^{\prime}_{\mathcal{K}\setminus\\{k\\}})=\mathbb{E}_{{\bm{\mathrm{g}}}_{\mathcal{K}\setminus\\{k\\}}}p_{\theta}(m_{j}\mid{\bm{\mathrm{g}}}_{\mathcal{K}})$
(25)

which holds iff
$p_{\theta}(m_{j}\mid{\bm{\mathrm{g}}}_{k},{\bm{\mathrm{g}}}_{\mathcal{K}\setminus\\{k\\}})=p_{\theta}(m_{j}\mid
g_{k})$, which is a contradiction. This proves the claim.

###  A.2 Proof of Proposition 2

In the following, we adopt the shorthand
${\bm{\mathrm{m}}}={\bm{\mathrm{m}}}_{\mathcal{J}}$, and reintroduce the
dependency on ${\bm{\mathrm{m}}}_{-\mathcal{J}}$ at the end. First, we show
that the maximum of the second term in $\Lambda$ in eq. 12 coincides with the
Shannon entropy of $Y$:

$\displaystyle\mathcal{L}_{r}(\gamma)$
$\displaystyle=\mathbb{E}_{p({\bm{\mathrm{x}}},y)}\big{[}\log
r_{\gamma}(y)\big{]}$ (26) $\displaystyle=\int p({\bm{\mathrm{x}}},y)\log
r_{\gamma}(y)\,\mathrm{d}{\bm{\mathrm{x}}}\mathrm{d}y$ $\displaystyle=\int
p(y)\log\frac{r_{\gamma}(y)p(y)}{p(y)}\,\mathrm{d}y$
$\displaystyle=-H(Y)-\mathsf{KL}(p(Y)\mid\mid r_{\gamma}(Y))$

where $p(Y)$ denotes the marginal distribution of $Y$, $H(Y)$ is the Shannon
entropy given by $p(Y)$, and $\mathsf{KL}$ is the Kullback-Leibler divergence.
Since the KL is always non-negative, the previous equation yields the upper
bound:

$\max_{\gamma}[\mathcal{L}_{r}(\gamma)]=-H(Y)$ (27)

We proceed similarly to obtain a lower-bound:

$\displaystyle\mathcal{L}_{CL}(\lambda)$ $\displaystyle=\int
p({\bm{\mathrm{x}}},y)\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}})p_{\theta}({\bm{\mathrm{m}}}\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{m}}}\Big{)}\mathrm{d}{\bm{\mathrm{x}}}\mathrm{d}y$
(28) $\displaystyle\geq\int
p({\bm{\mathrm{x}}})p_{\theta}({\bm{\mathrm{m}}}\mid{\bm{\mathrm{x}}})p(y\mid{\bm{\mathrm{x}}})\log
q_{\lambda}(y\mid{\bm{\mathrm{m}}})\,\mathrm{d}{\bm{\mathrm{x}}}\
\mathrm{d}{\bm{\mathrm{m}}}\ \mathrm{d}y$ $\displaystyle=\int
p_{\theta}({\bm{\mathrm{m}}},y)\log
q_{\lambda}(y\mid{\bm{\mathrm{m}}})\,\mathrm{d}{\bm{\mathrm{m}}}\mathrm{d}y$
$\displaystyle=\int
p_{\theta}({\bm{\mathrm{m}}},y)\log\frac{q_{\lambda}(y\mid{\bm{\mathrm{m}}})p_{\theta}({\bm{\mathrm{m}}})p(y)p_{\theta}({\bm{\mathrm{m}}},y)}{p_{\theta}({\bm{\mathrm{m}}})p(y)p_{\theta}({\bm{\mathrm{m}}},y)}\,\mathrm{d}{\bm{\mathrm{m}}}\mathrm{d}y$
$\displaystyle=\int p_{\theta}({\bm{\mathrm{m}}},y)\log
p(y)\,\mathrm{d}{\bm{\mathrm{m}}}\mathrm{d}y+\int
p_{\theta}({\bm{\mathrm{m}}},y)\Big{[}\log\frac{q_{\lambda,\theta}({\bm{\mathrm{m}}},y)}{p_{\theta}({\bm{\mathrm{m}}},y)}+\log\frac{p_{\theta}({\bm{\mathrm{m}}},y)}{p_{\theta}({\bm{\mathrm{m}}})p(y)}\Big{]}\,\mathrm{d}{\bm{\mathrm{m}}}\mathrm{d}y$
$\displaystyle=-H(Y)-\mathsf{KL}(p_{\theta}({\bm{\mathrm{M}}},Y)\mid\mid
q_{\lambda,\theta}({\bm{\mathrm{M}}},Y))+\mathrm{I}({\bm{\mathrm{M}}},Y)$

where $p_{\theta}({\bm{\mathrm{m}}},y)=\int
p({\bm{\mathrm{x}}})p_{\theta}({\bm{\mathrm{m}}}\mid{\bm{\mathrm{x}}})p(y\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{x}}}$,
$p_{\theta}({\bm{\mathrm{m}}})$ is the posterior of the encoding distribution,
$q_{\lambda,\theta}({\bm{\mathrm{m}}},y):=q_{\lambda}(y\mid{\bm{\mathrm{m}}})p_{\theta}({\bm{\mathrm{m}}})$
denotes the joint probability, and $\text{I}({\bm{\mathrm{M}}},Y)$ is the
mutual information for the random variables ${\bm{\mathrm{M}}}$ and $Y$,
distributed according to $p_{\theta}({\bm{\mathrm{M}}},y)$. Maximizing the
lower-bound implies learning a predictor $q_{\lambda}(y\mid{\bm{\mathrm{m}}})$
that minimizes the $\mathsf{KL}$ term. By the previous equation this happens
iff $q_{\lambda}(y,{\bm{\mathrm{m}}})$ matches
$p_{\theta}({\bm{\mathrm{m}}},y)$. The lower-bound for the first term of
$\Lambda$ hence becomes:

$\max_{\lambda}[\mathcal{L}_{CL}(\lambda)]\geq-H(Y)+\mathrm{I}({\bm{\mathrm{M}}},Y)$
(29)

Adding this term to the second one shows retrieves the definition of concept
leakage and shows that it is lower-bounded by:

$\Lambda\geq\mathrm{I}({\bm{\mathrm{M}}}_{\mathcal{J}},Y)$ (30)

We now proceed deriving the upper-bound for the first term:

$\displaystyle\mathcal{L}_{CL}(\lambda)$ $\displaystyle=\int
p({\bm{\mathrm{x}}},y)\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}})p_{\theta}({\bm{\mathrm{m}}}\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{m}}}\Big{)}\mathrm{d}{\bm{\mathrm{x}}}\mathrm{d}y$
(31) $\displaystyle=\int
p({\bm{\mathrm{g}}}_{I})q({\bm{\mathrm{g}}}_{-\mathcal{I}})\Big{[}\int
p({\bm{\mathrm{x}}}\mid{\bm{\mathrm{g}}})p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}})p_{\theta}({\bm{\mathrm{m}}}\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{m}}}\Big{)}\mathrm{d}{\bm{\mathrm{x}}}\
\mathrm{d}y\Big{]}\ \mathrm{d}{\bm{\mathrm{g}}}_{\mathcal{I}}\
\mathrm{d}{\bm{\mathrm{g}}}_{-\mathcal{I}}$ $\displaystyle\leq\int
q({\bm{\mathrm{g}}}_{-\mathcal{I}})\Big{[}\int
p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log
q_{\lambda,\theta}(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\mathrm{d}y\Big{]}\,d{\bm{\mathrm{g}}}_{-\mathcal{I}}$
$\displaystyle=\int q({\bm{\mathrm{g}}}_{-\mathcal{I}})\Big{[}\int
p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log\frac{q_{\lambda,\theta}(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})p(y)p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})q({\bm{\mathrm{g}}}_{-\mathcal{I}})}{p(y)p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})q({\bm{\mathrm{g}}}_{-\mathcal{I}})}\mathrm{d}y\Big{]}\,d{\bm{\mathrm{g}}}_{-\mathcal{I}}$
$\displaystyle=\int p(y)\log p(y)\,\mathrm{d}y+\int
q({\bm{\mathrm{g}}}_{-\mathcal{I}})p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log\Big{[}\frac{q_{\lambda,\theta}(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})}{p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})}+\frac{p(y,{\bm{\mathrm{g}}}_{-\mathcal{I}})}{p(y)q({\bm{\mathrm{g}}}_{-\mathcal{I}})}\Big{]}\,\mathrm{d}y\mathrm{d}{\bm{\mathrm{g}}}_{-\mathcal{I}}$
$\displaystyle=-H(Y)-\mathbb{E}_{{\bm{\mathrm{g}}}_{-\mathcal{I}}\sim
q({\bm{\mathrm{g}}}_{-\mathcal{I}})}[\mathsf{KL}(p(Y\mid{\bm{\mathrm{G}}}_{-\mathcal{I}})\mid\mid
q_{\lambda,\theta}(Y\mid{\bm{\mathrm{G}}}_{-\mathcal{I}}))]+I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)$

where in the second line we decomposed $p({\bm{\mathrm{x}}},y)$ with the data
generation process, and in the third line we made use of Jensen inequality
when bringing $\int
p({\bm{\mathrm{x}}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}})p({\bm{\mathrm{g}}}_{I})\mathrm{d}{\bm{\mathrm{x}}}\,\mathrm{d}{\bm{\mathrm{g}}}_{\mathcal{I}}$
in the logarithm, and we denoted with
$q_{\lambda,\theta}(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})$ the conditional
distribution obtained by marginalizing over all expectations in the logarithm.
Overall, the only part depending on $\lambda$ appears in the $\mathsf{KL}$
term and $I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)$ is the mutual information for
the probability distribution
$p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})q({\bm{\mathrm{g}}}_{-\mathcal{I}})$.
Notice that the maximum of the upper-bound for $\mathcal{L}_{CL}(\lambda)$
corresponds to a vanishing $\mathsf{KL}$ term and hence the upper-bound for
$\Lambda$ results in:

$\Lambda\leq I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)$ (32)

Finally, we arrive at the claim:

$I({\bm{\mathrm{M}}}_{\mathcal{J}},Y)\leq\Lambda\leq
I({\bm{\mathrm{G}}}_{-\mathcal{I}},Y)$ (33)

which concludes the proof.

###  A.3 Proof of Proposition 3

D1 in 6 entails that the conditional probability of
${\bm{\mathrm{M}}}_{\mathcal{J}}$ can be written in general as:

$p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}})=p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}})$
(34)

The same holds for D1 in 8. We make use of this fact for deriving a different
upper-bound for $\Lambda$. We focus only on the first term of eq. 11; the
analysis of the second one does not change.

$\displaystyle\mathcal{L}_{CL}(\lambda)$ $\displaystyle=\int
p({\bm{\mathrm{x}}},y)\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{m}}}_{\mathcal{J}}\Big{)}\,\mathrm{d}{\bm{\mathrm{x}}}\mathrm{d}y$
(35) $\displaystyle=\int p^{\prime}({\bm{\mathrm{g}}})\Big{[}\int
p({\bm{\mathrm{x}}}\mid{\bm{\mathrm{g}}})p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{x}}})\,\mathrm{d}{\bm{\mathrm{m}}}_{\mathcal{J}}\Big{)}\,\mathrm{d}{\bm{\mathrm{x}}}\mathrm{d}y\Big{]}\,\mathrm{d}{\bm{\mathrm{g}}}$
$\displaystyle\leq\int q({\bm{\mathrm{g}}}_{-\mathcal{I}})\Big{[}\int
p(y\mid{\bm{\mathrm{g}}}_{-\mathcal{I}})\log\Big{(}\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}})p({\bm{\mathrm{g}}}_{\mathcal{I}})\,\mathrm{d}{\bm{\mathrm{m}}}_{\mathcal{J}}\mathrm{d}{\bm{\mathrm{g}}}_{\mathcal{I}}\Big{)}\mathrm{d}y\Big{]}\,\mathrm{d}{\bm{\mathrm{g}}}_{-\mathcal{I}}$
$\displaystyle=\int p(y)\log p_{\lambda,\theta}(y)\,\mathrm{d}y$
$\displaystyle=\int
p(y)\log\frac{p_{\lambda,\theta}(y)p(y)}{p(y)}\,\mathrm{d}y$
$\displaystyle=-H(Y)-\mathsf{KL}(p(Y)\mid\mid p_{\lambda,\theta}(Y))$

In the second line we decomposed the data generation process, in the third
line we made use of Jensen’s inequality to introduce in the logarithm the term
$\int
p({\bm{\mathrm{g}}}_{\mathcal{I}})\,\mathrm{d}{\bm{\mathrm{g}}}_{\mathcal{I}}\int
p({\bm{\mathrm{x}}}\mid{\bm{\mathrm{g}}})\mathrm{d}{\bm{\mathrm{x}}}$. The
marginalization of
$p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{x}}})$ with
$p({\bm{\mathrm{x}}}\mid{\bm{\mathrm{g}}})$ gives
$p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}})$, that by D1
reduces to
$p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}})$,
hence the term appearing in the third line. In the fourth line, we denoted
with $p_{\lambda,\theta}(y)=\int
q_{\lambda}(y\mid{\bm{\mathrm{m}}}_{\mathcal{J}})p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{J}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}})p({\bm{\mathrm{g}}}_{\mathcal{I}})\,\mathrm{d}{\bm{\mathrm{m}}}_{\mathcal{J}}\mathrm{d}{\bm{\mathrm{g}}}_{\mathcal{I}}$
and reduced the first integral in $p(y)$. Finally, we obtain the upper bound
for the first term of $\Lambda$, where the maximum implies having a vanishing
$\mathsf{KL}$ term. Therefore, we have that:

$\Lambda\leq 0$ (36)

Now, since $\Lambda$ is lower bounded by the mutual information
$I({\bm{\mathrm{M}}}_{\mathcal{J}},Y)$, it cannot be negative and hence must
be zero. This concludes the proof.

###  A.4 Proof of Corollay 1

The result of the corollary follows from proposition 3 by considering only the
subset of representations ${\bm{\mathrm{M}}}_{\mathcal{J}}$ that are not
aligned to $G_{k}$. Denote them with ${\bm{\mathrm{M}}}_{\mathcal{K}}$, where
$\mathcal{K}=\\{j:\pi(j)\neq k\\}$ and set
$\bar{\bm{\mathrm{G}}}_{-\mathcal{I}}={\bm{\mathrm{G}}}_{-\mathcal{I}}\cup
G_{k}$. Then, we have:

$p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{K}}\mid{\bm{\mathrm{g}}})=p_{\theta}({\bm{\mathrm{m}}}_{\mathcal{K}}\mid{\bm{\mathrm{g}}}_{\mathcal{I}\setminus\\{g_{k}\\}})$
(37)

Similarly to proposition 3, we then obtain that $\Lambda=0$, i.e., concept
leakage vanishes. This proves the claim.

###  A.5 Proof of Proposition 4

For a given block ${\bm{\mathrm{M}}}_{\mathcal{K}}$ aligned to
${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$, recall that by D1 in 8 it holds that:

${\bm{\mathrm{M}}}_{\mathcal{K}}=\mu_{\mathcal{K}}({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{N}}}_{\mathcal{K}})$
(38)

To prove the first claim, we have to show that after intervening on
${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$ interventions on distinct
${\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}$ do not affect
${\bm{\mathrm{M}}}_{\mathcal{K}}$. Fix
$do({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})})$.
Upon performing a second intervention on the remaining variables
$do({\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{-\Pi(\mathcal{K})})$,
we get:

$\textstyle
p({\bm{\mathrm{G}}}|do({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{-\Pi(\mathcal{K})}))=\mathbbm{1}\\!\left\\{({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})})=({\bm{\mathrm{g}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{g}}}_{-\Pi(\mathcal{K})})\right\\}$
(39)

By D1 of 8, it holds that the corresponding probability distribution on
${\bm{\mathrm{M}}}_{\mathcal{K}}$ can be written as:

$p({\bm{\mathrm{M}}}_{\mathcal{K}}\mid
do({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}))=p({\bm{\mathrm{M}}}_{\mathcal{K}}\mid{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})})$
(40)

which by a similar argument to proposition 1 leads to a vanishing
$\mathsf{PIDA}({\bm{\mathrm{G}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{M}}}_{\mathcal{K}}\mid{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})},{\bm{\mathrm{g}}}_{-\Pi(\mathcal{K})})$,
for all possible interventions
$do({\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}\leftarrow{\bm{\mathrm{g}}}_{-\Pi(\mathcal{K})})$.
This proves that after intervening on ${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$,
arbitrary interventions on ${\bm{\mathrm{G}}}_{-\Pi(\mathcal{K})}$ do not
affect ${\bm{\mathrm{M}}}_{\mathcal{K}}$.

For the second claim, we consider two different intervened values
${\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime}$ and
${\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime\prime}$ for
${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$. Recall that by D2 in 8 it holds that
the mean value of ${\bm{\mathrm{M}}}_{\mathcal{K}}$ is connected to
${\bm{\mathrm{G}}}_{\Pi(\mathcal{K})}$ by an invertible map. Therefore, it
holds that:

${\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime}\neq{\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime\prime}\implies\mathbb{E}_{{\bm{\mathrm{N}}}_{\mathcal{K}}}[\mu_{\mathcal{K}}({\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime},{\bm{\mathrm{N}}}_{\mathcal{K}})]\neq\mathbb{E}_{{\bm{\mathrm{N}}}_{\mathcal{K}}}[\mu_{\mathcal{K}}({\bm{\mathrm{g}}}_{\Pi(\mathcal{K})}^{\prime\prime},{\bm{\mathrm{N}}}_{\mathcal{K}})]$
(41)

by invertibility. This concludes the proof.

## References

  * Guidotti et al. [2018] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ , 51(5):1–42, 2018\.
  * Štrumbelj and Kononenko [2014] Erik Štrumbelj and Igor Kononenko.  Explaining prediction models and individual predictions with feature contributions.  _Knowledge and information systems_ , 41(3):647–665, 2014.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “Why should I Trust You?” Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144, 2016.