# Interpretability Needs a New Paradigm

Andreas Madsen  Himabindu Lakkaraju  Siva Reddy  Sarath Chandar

###### Abstract

Interpretability is the study of explaining models in understandable terms to
humans. At present, interpretability is divided into two paradigms: the
intrinsic paradigm, which believes that only models designed to be explained
can be explained, and the post-hoc paradigm, which believes that black-box
models can be explained. At the core of this debate is how each paradigm
ensures its explanations are faithful, i.e., true to the model’s behavior.
This is important, as false but convincing explanations lead to unsupported
confidence in artificial intelligence (AI), which can be dangerous. This
paper’s position is that we should think about new paradigms while staying
vigilant regarding faithfulness. First, by examining the history of paradigms
in science, we see that paradigms are constantly evolving. Then, by examining
the current paradigms, we can understand their underlying beliefs, the value
they bring, and their limitations. Finally, this paper presents 3 emerging
paradigms for interpretability. The first paradigm designs models such that
faithfulness can be easily measured. Another optimizes models such that
explanations become faithful. The last paradigm proposes to develop models
that produce both a prediction and an explanation.

Interpretability, Explanations, Transparency, Paradigms, Post-hoc, Intrinsic,
Ethics, Future work, Faithfulness measurable models, Self-explanations, Self-
explaining models

##  1 Introduction

There was a time in physics, in the late 17th century, when Isaac Newton
insisted that light is a particle and Christiaan Huygens insisted that light
is a wave (Huygens, 1690). These ideas were seemingly irreconcilable at the
time. Of course, now we have a much better theory, and we understand that
light can be seen as both a wave and a particle.111Known as the wave-particle
duality concept (Messiah, 1966).

In 1874, Georg Cantor proposed set theory and showed there exists at least two
kinds of infinity. This divided the mathematical field. The Intuitionists, who
named Cantor’s theory nonsense, thought that math was a pure creation of the
mind and that these infinities weren’t real. Henri Poincaré said: “Later
generations will regard Mengenlehre (set theory) as a disease from which one
has recovered” (Gray, 1991). Leopold Kronecker called Cantor a “scientific
charlatan” and “corruptor of the youth” (Dauben, 1977).

The other group, the Formalists, thought that by using Cantor’s set theory,
all math could be proven from this fundamental foundation. David Hilbert said:
“No one shall expel us from the paradise that Candor has created” and “In
opposition to the foolish Ignoramus (we will not know; i.e., intuitionists),
our slogan shall be: We must know – we will know” (Smith, 2014).

Today, we know infinities are important concepts; thus, the Intuitionists were
wrong. However, Kurt Gödel showed that the Formalists were also wrong.
Unfortunately, there exist true statements which can never be proven (Gödel,
1931, Gödel’s incompleteness theorem).

These are just two examples in science and mathematics where there have been
strong debates and beliefs due to conflicting paradigms. Science historian
Thomas Kuhn defines a scientific paradigm as: “universally recognized
scientific achievements that, for a time, provide model problems and solutions
to a community of practitioners” (Kuhn, 1996).

Time and time again, when there are conflicting paradigms, it is only “for a
time”. Eventually, we find neither paradigm is true, or both paradigms are
true (under a more nuanced understanding). In retrospect, it is more
constructive to develop an understanding as to which paradigms may be right
under what conditions, as opposed to an all-or-nothing approach of arguing
about a singular right paradigm. Alternatively, we could come up with a new
paradigm, a new school of thought, a new direction; which replaces or bridges
the old way of thinking.

In this paper, we re-examine the current direction and paradigms of
interpretability and invite the reader to consider whether it is time for a
new paradigm.

###  1.1 Interpretability and faithfulness

Interpretability is the ability to explain a model in understandable terms to
humans (Doshi-Velez & Kim, 2017). Model explanations have particularly become
important for AI safety, as machine learning is increasingly being used by the
industry and affects the lives of most humans. This and additional motivations
are elaborated on in Section 2.

Within interpretability, there currently exist two paradigms, called _post-
hoc_ and _intrinsic_ (Lipton, 2018). Section 3 properly describe their stance.
Put briefly, the _intrinsic_ paradigm believes that only models designed to be
explained can be explained (Rudin, 2019);. In contrast, the _post-hoc_
paradigms believe this constraint is unnecessary and too restrictive to
achieve competitive performance (Madsen et al., 2022b).

The position in this paper is that neither paradigm has been fruitful because
their underlying beliefs are problematic and we should therefore look for new
directions. Section 4 contains the primary support for this position. To prove
that other paradigms can exist, Section 5 then presents three emerging
paradigms for interpretability and discusses how they might overcome past
challenges, their beliefs, drawbacks, and future directions. Importantly,
while this demonstrates that developing new paradigms is possible, Section 5
should not be considered a final list of directions.

At the core of this discussion is how each paradigm approaches faithfulness. A
faithful explanation means the explanation accurately reflects the model’s
logic, and ensuring and validating this often presents a major challenge
because the model’s logic is inaccessible to humans (Jacovi & Goldberg, 2020).
Faithfulness is particularly important, as false but convincing explanations
can lead to unsupported confidence in models, increasing the risk of AI.

New paradigms, such as those described in Section 4, bring new perspectives
regarding how to achieve faithfulness. This creates a new opportunity to do
interpretability research centered around ensuring faithfulness. However, it
also creates a new risk as we may take faithfulness for granted once again, as
has been the case with both the intrinsic (Jacovi & Goldberg, 2020) and post-
hoc paradigms (Madsen et al., 2022b). To prevent this, this paper also takes
the position that we should be vigilant about faithfulness when it comes to
these new paradigms to prevent repeating past mistakes.

##  2 Why interpretability is needed

Before discussing the current paradigms and their shortcomings, it’s necessary
to first consider if interpretability is needed at all. Many ethical
motivations for interpretability are also served by bias and fairness metrics,
so if the current paradigms of interpretability do not work (as we argue in
Section 4), perhaps we should drop the idea of interpretability completely. If
the models can be made accurate, unbiased, and fair enough, do we need to
explain the models? In this section, we will argue that interpretability is
required, by examining the limitations of bias and fairness metrics and the
scientific motivations for interpretability.

###  2.1 Limitations of bias and fairness metrics

There is no doubt that bias and fairness metrics present a vital role in
validating models’ behavior. However, a shared limitation is that they always
measure known attributes (Barocas et al., 2019). For example, gender-bias
metrics use gender attributes. This presents two challenges. Can we procure
such attributes (known as protected attributes)? How do we prevent
unanticipated biases?

####  2.1.1 Protected attribute procurement

Attributes like gender, race, age, disability, etc., are under U.S. law known
as “protected attributes” (Xiang & Raji, 2019), and collecting and using these
attributes is heavily regulated in most of the world. Andrus et al. (2021)
writes, “In many situations, however, information about demographics can be
extremely difficult for practitioners to even procure.”. Therefore,
systematically measuring bias and fairness is not always practical (Andrus et
al., 2021).

On the other hand, explanations often don’t depend on knowing these protected
attributes in advance and can provide a more qualitative analysis. For
example, suppose an explanation tells us that the word “Woman” from “Member of
Woman’s Chess Club” in a resume is important for making a hiring
recommendation. In that case, there is a potential harmful bias (Kodiyan,
2019). Therefore, explanations can serve a similar practical purpose to a
fairness or bias metric without performing systematical correlations.

####  2.1.2 Unknown attribute bias

Although protected attributes are important to consider and are often legally
protected, many more relevant attributes are involved in ensuring a fair and
unbiased system. Unfortunately, it is impossible to consider every possible
bias in advance. As an alternative, interpretability offers a more qualitative
and explorative validation.

Continuing the example with resumes and automated hiring recommendations,
during investigations by Fuller et al. (2021), the authors found that a
hospital only accepted candidates with computer programming experience when
they needed workers to enter patient data into a computer. Another example was
a clerk position where applicants were rejected if they did not mention floor-
buffing (i.e., a cleaning method for floors) (Fuller, 2021).

These examples present cases of systematic unintended bias. However, they do
not relate to any protected attributes, and they are so specific they can only
be discovered through qualitative explanations and investigations. That said,
systematic fairness/bias metrics can quantify the damage once potential biases
are identified using interpretability. Afterwards, those metrics can be
integrated into a quality assessment system to prevent future harm.

###  2.2 Interpretability for scientific discovery and understanding

Interpretability is not only used for ethics and adjacent purposes, where bias
and fairness metrics have an important role. Interpretability is also used for
scientific discovery and learning about what makes models work.

####  2.2.1 Scientific Discovery

An example of scientific discovery is interpretability in drug discovery
(Preuer et al., 2019; Jiménez-Luna et al., 2020; Dara et al., 2022). A common
approach is to use importance measures to identify regions in genomic
sequences responsible for a particular behavior, such as producing a protein.
While these explanations do not guarantee that such connections exist in
reality, they can provide important initial hypotheses for scientists enabling
them to make more informed choices about the direction of their research.

####  2.2.2 Model understanding

An emerging field of interpretability is mechanistic interpretability, which
identifies parts of a neural network that have a particular responsibility
(Cammarata et al., 2020). For example, identifying a collection of neurons
responsible for copying content in a generative language model, etc. (Elhage
et al., 2021). Such insights may not be directly relevant to downstream tasks,
but they help us understand current model limitations and can lead to better
model design.

##  3 The current paradigms of interpretability

Table 1: Comparison of the definitions and underlying beliefs of the intrinsic
and post-hoc paradigms. The beliefs relate to a) requirements for a faithful
explanation and b) model capabilities. It should be apparent that these two
views are seemingly incompatible. Intrinsic paradigm The model is designed to
provide explanations by making the explanation part of the model architecture.
Only models that were designed to be explained can be explained. Intrinsic
models can have the same performance as a black-box model. Post-hoc paradigm
The model is produced without regard for explanation, and the explanations are
then created after model training. Although it may be very challenging, black-
box models can be explained. Black-box models will be more generally
applicable than intrinsic models. $a^{a}_{b}$defintion$a^{a}_{b}$underlying
beliefs

This paper uses a common definition of interpretability, “the ability to
explain or to present in understandable terms to a human” by (Doshi-Velez &
Kim, 2017). However, even this definition of interpretability is not agreed
upon.

Lipton says, “the term interpretability holds no agreed upon meaning, and yet
machine learning conferences frequently publish papers which wield the term in
a quasi-mathematical way” (Lipton, 2018). In 2017, a UK Government House of
Lords review of AI noted after substantial expert evidence that “the
terminology used by our witnesses varied widely. Many used the term
transparency, while others used interpretability or explainability, sometimes
interchangeably” (House of Lords, 2017, 91).

For this reason, there are also no clearly agreed-upon definitions of the
current paradigms of interpretability (Carvalho et al., 2019; Flora et al.,
2022). As such, this section defines the _intrinsic_ and _post-hoc_ paradigms,
as well as describe their underlying beliefs, which are summarized in Table 1.

###  3.1 Definitions

Jacovi & Goldberg (2020) write: “A distinction is often made between two
methods of interpretability: (1) interpreting existing models via post-hoc
techniques; and (2) designing inherently interpretable models. (Rudin, 2019)”.
Based on this and other sources (Madsen et al., 2022b; Arya et al., 2019;
Carvalho et al., 2019; Murdoch et al., 2019), this paper refers to these two
ideas as 1) the _intrinsic_ paradigm and 2) the _post-hoc_ paradigm.

####  3.1.1 The intrinsic paradigm

The intrinsic paradigm works on creating so-called _inherently interpretable
models_. These models are architecturally constrained, such that the
explanation emerges from the architecture itself.

Classical examples are decision trees or linear regression. In the field of
neural networks some examples are: 1) “Old-school” attention (Bahdanau et al.,
2015; Jain & Wallace, 2019), where attention points to which input tokens are
important. 2) Neural Modular Networks (Andreas et al., 2016; Gupta et al.,
2020; Fashandi, 2023), which produce a prediction via a sequence of sub-
models, each with known behavior. 3) Prototypical Networks (Bien & Tibshirani,
2009; Kim et al., 2014; Chen et al., 2019), which predicts by finding similar
training observations.

####  3.1.2 The post-hoc paradigm

_Post-hoc_ explanations are computed after the model has been trained. They
are developed independently of the model’s architecture and how it was
trained. However, there are often some simple criteria, like “the model should
be differentiable”, “the training dataset is known”, or “inputs are
represented as tokens” (Madsen et al., 2022b). Although general applicability
is technically not a requirement, if a method is so specific that it only
works on one specific model, it’s likely an _intrinsic explanation_.

As an example, a common post-hoc explanation is gradient-based importance
measures. Importance measures explain which input features (words, pixels,
etc.) are important for making a prediction. This is archived by
differentiating the prediction with respect to the input. The idea is that if
a small change in input causes a big change in the output, then that input is
important (Baehrens et al., 2010; Seo et al., 2018; Karpathy et al., 2015).

###  3.2 Beliefs

As with all paradigms, there are fundamental underlying beliefs, which are why
the paradigm’s followers partake in their paradigm of choice. At the core of
these beliefs are two central questions. When are explanations faithful and
what are the requirements for faithfulness? And, how do these requirements
affect the model’s general performance capabilities?

####  3.2.1 When are explanations faithful?

The intrinsic paradigm believes that: _only models designed to be explained,
can be explained_ , which their _inherently interpretable models_ try to
satisfy. Therefore, they argue that using black-box models is too risky, as
these models can never be faithfully explained (Rudin, 2019).

However, although their models are designed to be intrinsically explainable,
this claim and their faithfulness should still be questioned (Jacovi &
Goldberg, 2020), as many inherently interpretable model ideas are later
revealed not to provide faithful explanations. For example, attention-based
explanations have received notable criticism for not being faithful (Jain &
Wallace, 2019; Serrano & Smith, 2019; Vashishth et al., 2019; Meister et al.,
2021; Madsen et al., 2022a). This is discussed more in Section 4.1.

The _post-hoc explanation_ paradigm takes a less strict stance and believes
that even models that were not designed to be explained (i.e., black-box
models) can still be explained. However, as this paradigm has no control over
the model, achieving faithful explanations is very challenging; this is
discussed more in Section 4.2.

In conclusion, the intrinsic paradigm considers explanations to be part of the
model design, and post-hoc explanations are always applied after the model
design. Madsen et al. (2022b) frame intrinsic as proactive and post-hoc as
retroactive. Hence, the two schools of thought are incompatible frameworks,
and they can philosophically be considered as paradigms (Kuhn, 1996).

####  3.2.2 What is the effect on the model’s general performance
capabilities?

It would seem that _intrinsic explanation_ is the obvious choice. If we can
control the model such that the faithfulness of explanations can be
guaranteed, why consider _post-hoc explanation_?

The commonly mentioned idea is that the _post-hoc_ paradigm believes that by
constraining the models in the manners that the _intrinsic paradigm_ requires,
there is a trade-off in performance (DARPA, 2016). However, this trade-off
does not have to be the case in practice (Rudin, 2019, section 2).

A more accurate take, which is rarely explicitly discussed, is that the common
industry prefers off-the-shelf general-purpose models and only later thinks
about interpretability (Bhatt et al., 2019). Additionally, most research only
considers predictive performance, not interpretability. Therefore, _intrinsic_
researchers are always catching up to black-box models. From the _post-hoc_
perspective, it would make more sense to work on generally applicable
interpretability methods for both off-the-shelf and future black-box models.

From the intrinsic perspective, while the industry might prefer off-the-shelf
models now, they shouldn’t. Not validating models through intrinsic
explanations can have serious consequences (Rudin, 2019) and eventually damage
their business. Additionally, with increasing legal requirements to provide
explanations, the industry may have to use inherently explainable models
(Goodman & Flaxman, 2017).

For these reasons, the _intrinsic_ paradigm believes we should not let the
industry’s needs dictate our research direction, as their goals may be too
short-sighted. In the long run, intrinsic models may be the only reasonable
option.

In conclusion, the _post-hoc_ paradigm has good intentions with providing
general explanations for general-purpose models. However, from the _intrinsic_
paradigm perspective, those good intentions are meaningless if it is
fundamentally impossible to provide guaranteed faithful explanations without
an _inherently interpretable model_.

##  4 Why interpretability needs a new paradigm

It tends to be the case that when there are multiple paradigms, it is because
neither of the paradigms fits the needs. However, for the case of the _post-
hoc_ and _intrinsic_ paradigms, it could be argued that they serve different
needs. For example, _intrinsic_ explanations should be preferred for critical
applications (Rudin, 2019), and _post-hoc_ explanations could be used for
verifiable situations, such as drug discovery, where the hypothesis generated
by the explanations is verified using physical experiments.

###  4.1 The case against the intrinsic paradigm

The industry primarily uses post-hoc explanations, including for high-stakes
applications such as insurance risk assessment and financial loan assessment
(Bhatt et al., 2019; Krishna et al., 2022). This is because such industries
usually do not have the in-house expertise to develop custom inherently
interpretable models. They must rely on basic inherently interpretable models,
like decision trees, which are not competitive, or use more advanced off-the-
shelf neural black-box models, like pre-trained language models, which will be
competitive. In practice, the industry is thus not in a position to choose
inherently interpretable models.

Another challenge with the intrinsic paradigm is that its models are often not
completely interpretable because only a part of the model is aritectually
constrained to be interpretable. The rest of the neural network, still use
black-box components (e.g. Dense layer, Recurrent layer, etc.) which are not
interpretable. As such, the intrinsic promise should not be taken at face
value (Jacovi & Goldberg, 2020).

An example of this, is classic attention-based models (Bahdanau et al., 2015;
Jain & Wallace, 2019). Attention itself is interpretable, as it’s a weighted
sum, and explains the importance of each intermediate representation. However,
attention is often used as token-importance. This does not work, as the
intermediate representations are produced by a black-box recurrent neural
network (e.g., LSTM Hochreiter & Schmidhuber 1997) which can mix or move the
relationship between tokens and the intermediate representations. Therefore,
the attention scores do not necessary represent token-importance (Bastings &
Filippova, 2020).

Likewise, Neural Modular Networks produce an executable problem composed of
sub-networks, such as find-max-num(filter(find())), which is interpretable
(Fashandi, 2023; Andreas et al., 2016; Gupta et al., 2020). However, each sub-
networks (find-max-num, filter, find) is itself a black-box model with little
guarantee that it operates as intended (Amer & Maul, 2019; Subramanian et al.,
2020; Lyu et al., 2022).

Overall, there are few success stories with intrinsic explanations. They are
either not performance-wise competitive, general-purpose enough for the
industry (Bhatt et al., 2019), or their intrinsic claims are unsupported
(Jacovi & Goldberg, 2020).

###  4.2 The case against the post-hoc paradigm

Although post-hoc explanations directly address the interpretability challenge
of black-box components and models, and could therefore provide more complete
explanations, there are also few success stories with post-hoc, where post-hoc
explanations are consistently faithful.

Most notable is perhaps post-hoc importance measure (IM) explanations, where
the explanation indicates which input features are the most important for
making a prediction. The pursuit of such explanations have produced countless
papers (Binder et al., 2016; Ribeiro et al., 2016; Li et al., 2016; Shrikumar
et al., 2017; Smilkov et al., 2017; Sundararajan et al., 2017; Ahern et al.,
2019; Thorne et al., 2019; ElShawi et al., 2019; Sangroya et al., 2020).

However, repeatedly, the faithfulness of these IM explanations is criticized
(Adebayo et al., 2018, 2021; Kindermans et al., 2019; Hooker et al., 2019;
Slack et al., 2020; Yeh et al., 2019). For example, there is great
disagreement between alleged faithful IMs, which is hard to reconcile (Jain &
Wallace, 2019; Krishna et al., 2022). Other works show their faithfulness is
both task and model-dependent and thus don’t provide the generality that the
_post-hoc_ paradigm desires (Bastings et al., 2022; Madsen et al., 2022a).
Finally, theoretical works suggest that IMs are subject to a _no free lunch
thorem_ (Han et al., 2022), or it may be impossible to provide faithful post-
hoc IMs (Bilodeau et al., 2024).

Similar to the work of IM, is visualization of neurons in computer vision,
which shows that neurons represent high-level concepts, such as nose or dog.
This is done by visualizing convolutional weights or the input image that
maximizes a neuron’s activation (Olah et al., 2017; Nguyen et al., 2016;
Yosinski et al., 2015), which provides very convincing evidence. However, it
has been shown empirically, theoretically, and through human-computer-
interaction (HCI) studies that these visualizations do not provide useful
explanations regarding the neurons’ responsibility (Geirhos et al., 2023;
Borowski et al., 2021; Zimmermann et al., 2021)222Neural networks likely do
encode high-level concepts, but these visualizations are not useful for
identifying the responsibility of specific neurons..

Another notable example is probing explanations, where models are verified by
relating the model’s behavior or intermediate representation to, for example,
linguistic properties (part-of-speech, etc.) (Belinkov & Glass, 2019; Belinkov
et al., 2020). This idea has produced an entire subfield called BERTology
(Rogers et al., 2020). BERTology in particular has attained substantial
attention (Coenen et al., 2019; Clark et al., 2019; Rogers et al., 2020;
Clouatre et al., 2022; McCoy et al., 2019; Conneau et al., 2018; Tenney et
al., 2019), with most of the works finding that neural networks can learn
linguistic properties indirectly.

Unfortunately, like post-hoc importance measures, there are many reasons to be
highly skeptical (Belinkov, 2021). For example, using an untrained model or a
randomized dataset shows an equally high correlation with linguistic
properties, compared with training a regular model (Zhang & Bowman, 2018;
Hewitt & Liang, 2019). These discoveries have put the entire methodology into
question, although there is work trying to adapt to these new critics (Voita &
Titov, 2020).

###  4.3 Conclusion

Post-hoc importance measures and probing explanations are just two cases where
post-hoc shows initial promise through countless papers, only to be debunked
repeatedly. The trend is oscillating between proposing new explanation methods
and debunking them. Of course, it’s impossible to prove that there will never
be a great post-hoc method. However, the lack of guarantees also makes it
impossible to know when a faithful post-hoc method is proposed. Similarly,
intrinsic explanations also receive criticism after a while, as has been the
case with attention and Neural Modular Networks.

Although both the intrinsic and post-hoc paradigms have significant issues,
parts of their underlying beliefs have merit. The intrinsic paradigm believes
that _we can’t expect models that were not designed to be explained to be
explained_ , while post-hoc believes _black-box models tend to be more general
purpose and while providing high predictive performance_. These beliefs have
merit, and it’s worth considering how to incorporate their spirit into new
paradigms.

##  5 Possible new paradigms

Table 2: Comparison of the definitions and underlying beliefs of the new
paradigms. The beliefs relate to a) explanation requirements and b) model
capabilities. These new paradigms can be compared with the old paradigms in
Table 1. Faithfulness measurable model paradigm The model is designed to
measure the faithfulness of a category of explanations. It is computationally
feasible to optimize explanations for optimal faithfulness. Models can be
optimized to be faithfulness measurable without loss of predictive
performance. Learn-to-faithfully-explain paradigm The model is optimized such
that an explanation method (e.g., post-hoc method) becomes faithful. The
relaxed faithfulness metric used for optimization is a sufficient
approximation. Models can be optimized such that explanations become faithful
without losing performance. Self-explaining model paradigm The model can
produce both its prediction and an explanation for that prediction. Models can
be trained to model and articulate their own reasoning accurately and will
generalize. Self-explanation capabilities do not negatively impact regular
predictions. $a^{a}_{b}$defintion$a^{a}_{b}$underlying beliefs

Given the shortcomings of the intrinsic and post-hoc paradigms, we are now in
a situation where it’s meaningful to look beyond these current paradigms. To
prove that developing alternative paradigms is possible and to highlight some
new directions of research, this section discusses new directions for
interpretability, frames them as paradigms, and summarizes them in Table 2.
However, these new paradigms are not the only directions one could take for
interpretability, and in general, we will encourage researchers to consider
alternatives that have not yet been explored.

###  5.1 Inherently Faithfulness Measurable Models

The faithfulness of an explanation can be extremely difficult to measure
because the true explanation is inaccessible to us (Jacovi & Goldberg, 2020).
Therefore, faithfulness metrics often have to make use of a proxy measure.

For example, the erasure metric for importance measures says: if information
(pixels, tokens, etc.) is truly important, then when removing it the
prediction should change significantly (Samek et al., 2017). However, this has
various limitations, because removing information causes out-of-distribution
issues, which other works have solved by repeatedly retraining the model on
datasets with allegedly important information removed (Hooker et al., 2019).
Unfortunately, retraining is computationally expensive, and measuring on a
retrained model means faithfulness is no longer measured on the original
model. Other faithfulness metrics have similar limitations (Madsen et al.,
2024b).

Therefore, rather than considering a faithfulness metric as something that
exists externally of the model, the idea with _Inherently Faithfulness
Measurable Models_ (FMMs) is a model that inherently provides the means to
cheaply and accurately measure faithfulness (Madsen et al., 2024b). Then,
because the faithfulness measure is cheap and accurate, it’s possible to
optimize an explanation to maximize this faithfulness metric (Zhou & Shah,
2023).

This is different from how inherently explainable models in the intrinsic
paradigm which are directly designed to be explained because their
architectural constraints produce the explanation (Jain & Wallace, 2019;
Rudin, 2019). The central idea in FFMs is the model does not produce an
explanation directly but enables a faithfulness metric, thus making FFMs
indirectly designed to be explainable (Madsen et al., 2024b).

In essence, Madsen et al. (2024b) reformulates the interpretability problem
from ‘inherently explainable” to “inherently measurable” and show that this
reformulation is much easier to design for, as it does not require
architectural constraints. The concrete demonstration that Madsen et al.
(2024b) provide, uses a RoBERTa model (Liu et al., 2019) where the loss
function is modified such that removing tokens becomes in-distribution,
thereby enabling the erasure metric (Samek et al., 2017) to be applied
directly without retraining. This achieves state-of-the-art faithfulness
metrics without affecting performance or inference time.

Because the work done by Madsen et al. (2024b) only applies to masked language
models. Thus, the immediate next step is to extend these ideas to causal
language models which may be a challenging task as their idea depends on mask-
tokens. Additionally, new work may explore more ways of optimizing
explanations given a cheap faithfulness metric (Zhou & Shah, 2023).

Beyond these immediate applications, the principle of enabling models to be
faithfulness measurable can be applied to other explanations. For example,
concepts (Kim et al., 2018) and influential examples (Yeh et al., 2018; Pruthi
et al., 2020; Koh & Liang, 2017) may be good candidates for this. Both the
general idea and the specific approach by Madsen et al. (2024b), can also be
applied to other domains such as computer vision.

To summarize, while previous interpretability works have explained first and
then measure faithfulness later, this paradigm does the opposite; it designs
for faithfulness measures first and then explains. This reformulation of the
interpretability problem is easier to design for, thus satifing both the
black-box and designed-to-be-explained desirables.

###  5.2 Learn to faithfully explain

Rather than optimizing the predictive model to be faithfulness measurable, a
more long-standing idea in interpretability has been to optimize the
predictive model such that an explanation method applied to that model
produces faithful explanations (Yoon et al., 2019; Chen et al., 2018). This
also does not require the architectural constraints that the intrinsic
paradigm applies, as the explanation comes from an external explanation
method, not the architectural design.

The explanation method can be similar or even identical to those from the
post-hoc paradigm. However, because the model is optimized to enable these
explanations to be faithful, it’s not post-hoc, and there are more reasons to
think that the explanations should be faithful.

Early work on this used an explanation model, which will produce an
explanation of the prediction model. The explanation and prediction models are
then trained jointly to produce faithful explanations and high-accuracy
predictions (Yoon et al., 2019; Chen et al., 2018). This direction has been
called joint amortized explanation methods (JAMs). However, Jethani et al.
(2021) point out that the explanation model often learns to encode the
prediction, which means the explanation model becomes part of the black-box
problem rather than the solution.

Jethani et al. (2021) solve this by using a disjoint setup, where the
explanation model can’t encode the prediction, a setup that following works
have adapted (Jethani et al., 2022; Covert et al., 2022). However, this still
presents a problem as it assumes the explanation model is 100% robust. For
example, an out-of-distribution observation could break the explanation model
and produce unfaithful explanations. This is particularly relevant, as it is
often the cases where models break that are desirable to explain.

Instead, a better direction is likely to use a simple explanation algorithm
that is easy to understand rather than an explanation model. Bhalla et al.
(2023) do this by jointly training a predictive model with an explanation
algorithm, where explanations indicate unnecessary features and the model
learns to support sparse features.

Regardless of the specific approach used to produce the explanation, the
optimization criteria used to make the predictive model converge, such that
the explanations are faithful, is either a relaxation of a faithfulness metric
or an underlying criterion necessary to produce faithful explanations. For
example, masking inputs should be in-distribution (Covert et al., 2022; Bhalla
et al., 2023). Therefore, this field needs to work on how to ensure and
validate that the explanations are actually faithful and that the model
properties that make explanations faithful also hold for out-of-distribution
data.

Another direction in this paradigm is the idea that regularization can produce
more faithful explanations. For example, a more locally-linear model may be
more suitable for a gradient-based explanation (Ross & Doshi-Velez, 2018;
Srinivas et al., 2022; Geirhos et al., 2023), or training a computer vision
model with added input-noise reduces noise in importance measure explanations
because the model becomes less sensitive to one-pixel variations (Bansal et
al., 2020). As of now, these observations are mostly hypothetical or
anecdotal, and we advocate for a systematic study from an interpretability
perspective.

###  5.3 Self-explaining models

Rather than using external algorithms or models to produce explanations, Elton
(2020) proposes that models should explain themselves, meaning they become
_self-explaining_. Because there are now two black-box modalities (the self-
explaining and the normal prediction modality), this can be quite dangerous,
and there must be solid evidence that the explanations are faithful for this
approach to be valid.

Despite the immediate dangers with a self-explaining model, the best tool to
explain a black-box model may be the model itself. In particular, unlike the
JAM approach (mentioned in Section 5.2), a self-explaining model has full
access to its own internal weights and activations. The explanations can also
be much more informative. This particularly holds for instruction-tuned large-
language models (LLMs) (OpenAI, 2023; Jiang et al., 2023; Meta, 2023), which
are allegedly able to explain themselves in great detail and very convincingly
(Chen et al., 2023; Agarwal et al., 2024).

When LLMs explain themselves, this is referred to as self-explanations.
Importantly, self-explanations must relate to the model’s reasoning logic, not
just the world or abstract concepts (e.g., mathematics), which is how most
Chain-of-Thought (CoT) explanations are evaluated (Chu et al., 2023). As
humans don’t know the model’s logic, evaluating self-explanations is difficult
(Huang et al., 2023).

Chen et al. (2023) show although self-explanations are very convincing to
humans (called plausibility), they do not help humans to guess model
predictions of new observations (called simulatability) and, therefore, can be
dangerous. They argue that “naively optimizing human approvals (e.g., RLHF)”
does not improve self-explanations’ simulatability.

Simulatability is important for measuring the usefulness of explanations.
However, it is difficult to ensure that the human test subjects make their
guesses based on the model’s self-explanation and not their world knowledge.
Therefore, measuring faithfulness is equally important, as this evaluates
explanations independently of humans.

To measure faithfulness in LLMs, one can use self-consistency checks (Madsen
et al., 2024a; Huang et al., 2023; Lanham et al., 2023). For example, Lanham
et al. (2023) considers CoT explanations for mathematical reasoning, edits
facts in the CoT (e.g., from “2+2=4” to “2+3=5”), and then checks that the
prediction changes based on these edits. Another example is Madsen et al.
(2024a), who evaluate counterfactual and importance measure explanations by
asking a model to change the input such it gives a different prediction; this
can be verified by re-evaluating with the counterfactual input. Both Lanham et
al. (2023) and Madsen et al. (2024a) find that the faithfulness of LLMs’ self-
explanations is model and task-dependent.

Unfortunately, self-consistency checks are impossible for most explanations
(Madsen et al., 2024a; Lanham et al., 2023). An alternative may be evaluating
the model’s ability to model itself (called self-modeling). The hypothesis is
that self-modeling is a necessary condition for general faithful self-
explanations (Madsen et al., 2024a).

Anthropic Team (2022) evaluate self-modeling capabilities (not in a self-
explanation context) by asking the LLM if it knows the answer to a question
and comparing it with simply asking the LLM to answer the question. They find
that LLMs cannot perform this task without additional fine-tuning. Even with
fine-tuning, the LLM cannot generalize to other domains. As such, developing
general self-modeling capabilities could be a first step towards general self-
explanation. Such work should also show that self-modeling capabilities make
concrete explanations faithful too, using self-consistency checks (Madsen et
al., 2024a).

###  5.4 Comparison

All three mentioned paradigms work with what-would-be black-box models.
However, their idea is to optimize these models to guarantee a faithful
explanation. They optimize for either faithfulness measurability, explanation
faithfulness, or self-model/self-explanation capabilities.

In conclusion, they are different from the post-hoc paradigm because
interpretability is considered during optimization, and they are different
from the intrinsic paradigm because they are not architecturally constrained.
They are also mutually distinct, each taking a different perspective on
achieving faithful interpretability via optimization.

At first, this sounds like a natural and simple idea. However, recall that we
can’t know what a faithful explanation is or what it looks like, as the
model’s internal logic is inaccessible to us. Achieving interpretability via
optimization is thus extremely challenging and involves proxies, assumptions,
etc., which is how they differ.

##  6 Limitations

As with any other position paper, this paper presents one position: that we
must seek out new paradigms of interpretability. However, one could make other
valid positions or directions about the future of interpretability.

In particular, this position paper primarily focuses on faithfulness. However,
as interpretability is about how we can explain models to humans in
understandable terms, one must also consider how well-understandable an
explanation is. This concern is known as human-groundedness (Doshi-Velez &
Kim, 2017), simulatability (Lipton, 2018), or comprehensibility (Robnik-
Šikonja & Bohanec, 2018).

This concern is orthogonal to the faithfulness concern and is therefore not
discussed in this paper. However, readers are encouraged to study recent works
like Schut et al. (2023); Kim (2022), which proposes the new idea that it is
not enough to frame explanations in terms that humans already understand. We
should also develop new language and mental abstractions for humans to
understand machines.

##  7 Conclusion

Although some evidence exists for the new paradigms that were presented in
Section 5, these are, first and foremost, just ideas. It’s only in retrospect
that we can truly know if one paradigm results in meaningful progress in the
field. It is also entirely possible that neither of these ideas is what moves
the interpretability field forward.

For these reasons, the core position of this paper is that we should develop
new directions and paradigms in interpretability, instead of focusing on the
existing post-hoc and intinsic paradigms, which are currently dominating.

That being said, we must also be vigilant regarding faithfulness to avoid
repeating past mistakes (Jacovi & Goldberg, 2020). These new paradigms will
present new arguments for why their method is faithful. As we are unfamiliar
with these arguments, identifying their flaws is difficult, and it will be
easy to get swayed by them.

Historically, a common tactic in post-hoc works was convincing visualization
of explanations that aligned with our intuitions (Olah et al., 2017; Yosinski
et al., 2015; Nguyen et al., 2016). However, such visualizations are empty
arguments, as humans can’t know what a true explanation looks like (Geirhos et
al., 2023). Likewise, intrinsic works have made seemingly strong theoretical
arguments for why their methods were faithful, but these arguments failed to
capture the whole model. Even the new _learn-to-faithfully-explain paradigm_
have already shown sharp corners, where the explainer models unintentionally
encode the prediction and are therefore unfaithful (Jethani et al., 2021).

To prevent false arguments, a sound start is to always have a specific and
measurable definition of faithfulness, which works for all methods within a
given explanation category (e.g., counterfactual or importance measures).

Finally, while these new paradigms are promising, it’s unlikely they will
completely erase the current paradigms. We still teach both the particle and
wave perspectives in physics. Most scientists don’t worry about whether there
are true statements in math that cannot be proven.

Likewise, there will likely always be situations where intrinsic or post-hoc
interpretability makes sense. For example, basic statistics and linear
regressions can be framed as intrinsic interpretability. Hence, if a company
or researchers decide to use a model because of its intrinsically explainable
properties, then we should only praise them – as long as they also measure the
faithfulness of the explanations.

## References

  * Adebayo et al. (2018) Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , volume 2018-Decem, pp. 9505–9515. Curran Associates, Inc., 10 2018.  URL http://arxiv.org/abs/1810.03292.
  * Adebayo et al. (2021) Adebayo, J., Muelly, M., Abelson, H., and Kim, B.  Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation.  In _International Conference on Learning Representations_ , pp. 1–13, 2021.  URL https://openreview.net/forum?id=xNOVfCCvDpM.
  * Agarwal et al. (2024) Agarwal, C., Tanneru, S. H., and Lakkaraju, H.  Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models.  _arXiv_ , 2024.  URL http://arxiv.org/abs/2402.04614.
  * Ahern et al. (2019) Ahern, I., Noack, A., Guzmán-Nateras, L., Dou, D., Li, B., and Huan, J.  Normlime: A new feature importance metric for explaining deep neural networks.  _arXiv_ , 9 2019.  ISSN 23318422.  URL http://arxiv.org/abs/1909.04200.
  * Amer & Maul (2019) Amer, M. and Maul, T.  A review of modularization techniques in artificial neural networks.  _Artificial Intelligence Review_ , 52(1):527–561, 6 2019.  ISSN 0269-2821.  doi: 10.1007/s10462-019-09706-7.  URL http://link.springer.com/10.1007/s10462-019-09706-7.
  * Andreas et al. (2016) Andreas, J., Rohrbach, M., Darrell, T., and Klein, D.  Neural Module Networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 39–48. IEEE, 6 2016.  ISBN 978-1-4673-8851-1.  doi: 10.1109/CVPR.2016.12.  URL http://ieeexplore.ieee.org/document/7780381/.
  * Andrus et al. (2021) Andrus, M., Spitzer, E., Brown, J., and Xiang, A.  What we can’t measure, We can’t understand: Challenges to demographic data procurement in the pursuit of fairness.  _FAccT 2021 - Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 249–260, 2021.  doi: 10.1145/3442188.3445888.
  * Anthropic Team (2022) Anthropic Team.  Language Models (Mostly) Know What They Know.  _Anthropic_ , 7 2022.  URL http://arxiv.org/abs/2207.05221.
  * Arya et al. (2019) Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., Mourad, S., Pedemonte, P., Raghavendra, R., Richards, J., Sattigeri, P., Shanmugam, K., Singh, M., Varshney, K. R., Wei, D., and Zhang, Y.  One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques.  _arXiv_ , 9 2019.  ISSN 23318422.  URL http://arxiv.org/abs/1909.03012.
  * Baehrens et al. (2010) Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and Müller, K. R.  How to explain individual classification decisions.  _Journal of Machine Learning Research_ , 11:1803–1831, 12 2010.  ISSN 15324435.  URL http://arxiv.org/abs/0912.1128.
  * Bahdanau et al. (2015) Bahdanau, D., Cho, K. H., and Bengio, Y.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ , pp. 1–15. International Conference on Learning Representations, ICLR, 9 2015.  URL https://arxiv.org/abs/1409.0473.
  * Bansal et al. (2020) Bansal, N., Agarwal, C., and Nguyen, A.  SAM: The Sensitivity of Attribution Methods to Hyperparameters.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 11–21. IEEE, 6 2020.  ISBN 978-1-7281-9360-1.  doi: 10.1109/CVPRW50498.2020.00009.  URL https://ieeexplore.ieee.org/document/9150607/.
  * Barocas et al. (2019) Barocas, S., Hardt, M., and Narayanan, A.  _Fairness and Machine Learning: Limitations and Opportunities_.  fairmlbook.org, 2019.  URL https://fairmlbook.org/.
  * Bastings & Filippova (2020) Bastings, J. and Filippova, K.  The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pp. 149–155, Stroudsburg, PA, USA, 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.blackboxnlp-1.14.  URL https://www.aclweb.org/anthology/2020.blackboxnlp-1.14.
  * Bastings et al. (2022) Bastings, J., Ebert, S., Zablotskaia, P., Sandholm, A., and Filippova, K.  “Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 976–991, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.64.  URL https://aclanthology.org/2022.emnlp-main.64.
  * Belinkov (2021) Belinkov, Y.  Probing Classifiers: Promises, Shortcomings, and Advances.  _arXiv_ , pp. 1–12, 2 2021.  URL http://arxiv.org/abs/2102.12452.
  * Belinkov & Glass (2019) Belinkov, Y. and Glass, J.  Analysis Methods in Neural Language Processing: A Survey.  _Transactions of the Association for Computational Linguistics_ , 7:49–72, 4 2019.  ISSN 2307-387X.  doi: 10.1162/tacl–“˙˝a–“˙˝00254.  URL https://doi.org/10.1162/tacl_a_00254.
  * Belinkov et al. (2020) Belinkov, Y., Gehrmann, S., and Pavlick, E.  Interpretability and Analysis in Neural NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pp. 1–5, Stroudsburg, PA, USA, 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-tutorials.1.  URL https://www.aclweb.org/anthology/2020.acl-tutorials.1.
  * Bhalla et al. (2023) Bhalla, U., Srinivas, S., and Lakkaraju, H.  Verifiable Feature Attributions: A Bridge between Post Hoc Explainability and Inherent Interpretability.  _arXiv_ , pp. 1–14, 2023.  URL http://arxiv.org/abs/2307.15007.
  * Bhatt et al. (2019) Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., and Eckersley, P.  Explainable Machine Learning in Deployment.  _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pp. 648–657, 9 2019.  doi: 10.1145/3351095.3375624.  URL https://dl.acm.org/doi/10.1145/3351095.3375624.
  * Bien & Tibshirani (2009) Bien, J. and Tibshirani, R.  Classification by Set Cover: The Prototype Vector Machine.  _arXiv_ , pp. 1–24, 2009.  URL http://arxiv.org/abs/0908.2284.
  * Bilodeau et al. (2024) Bilodeau, B., Jaques, N., Koh, P. W., and Kim, B.  Impossibility theorems for feature attribution.  _Proceedings of the National Academy of Sciences_ , 121(2):1–36, 1 2024.  ISSN 0027-8424.  doi: 10.1073/pnas.2304406120.  URL https://pnas.org/doi/10.1073/pnas.2304406120.
  * Binder et al. (2016) Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., and Samek, W.  Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers.  In _Artificial Neural Networks and Machine Learning – ICANN 2016_ , volume 9887 LNCS, pp. 63–71, 2016.  doi: 10.1007/978-3-319-44781-0–“˙˝8.  URL http://link.springer.com/10.1007/978-3-319-44781-0_8.
  * Borowski et al. (2021) Borowski, J., Zimmermann, R. S., Schepers, J., Geirhos, R., Wallis, T. S., Bethge, M., and Brendel, W.  Exemplary Natural Images Explain Cnn Activations Better Than State-of-the-Art Feature Visualization.  _ICLR 2021 - 9th International Conference on Learning Representations_ , pp. 1–41, 2021.
  * Cammarata et al. (2020) Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., and Schubert, L.  Thread: Circuits.  _Distill_ , 5(3), 3 2020.  ISSN 2476-0757.  doi: 10.23915/distill.00024.  URL https://distill.pub/2020/circuits.
  * Carvalho et al. (2019) Carvalho, D. V., Pereira, E. M., and Cardoso, J. S.  Machine Learning Interpretability: A Survey on Methods and Metrics.  _Electronics_ , 8(8):832, 7 2019.  ISSN 2079-9292.  doi: 10.3390/electronics8080832.  URL https://www.mdpi.com/2079-9292/8/8/832.
  * Chen et al. (2019) Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., and Rudin, C.  This looks like that: Deep learning for interpretable image recognition.  _Advances in Neural Information Processing Systems_ , 32, 6 2019.  ISSN 10495258.  URL http://arxiv.org/abs/1806.10574.
  * Chen et al. (2018) Chen, J., Song, L., Wainwright, M. J., and Jordan, M. I.  Learning to explain: An information-theoretic perspective on model interpretation.  _35th International Conference on Machine Learning, ICML 2018_ , 2:1386–1418, 2 2018.  URL http://arxiv.org/abs/1802.07814.
  * Chen et al. (2023) Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., and McKeown, K.  Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations.  _arXiv_ , 2023.  URL http://arxiv.org/abs/2307.08678.
  * Chu et al. (2023) Chu, Z., Chen, J., Chen, Q., Yu, W., and He, T.  A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future.  _arXiv_ , 2023.
  * Clark et al. (2019) Clark, K., Khandelwal, U., Levy, O., and Manning, C. D.  What Does BERT Look at? An Analysis of BERT’s Attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 276–286, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-4828.  URL https://www.aclweb.org/anthology/W19-4828.
  * Clouatre et al. (2022) Clouatre, L., Parthasarathi, P., Zouaq, A., and Chandar, S.  Local Structure Matters Most: Perturbation Study in NLU.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 3712–3731, Stroudsburg, PA, USA, 7 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.293.  URL https://aclanthology.org/2022.findings-acl.293.
  * Coenen et al. (2019) Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., Wattenberg, M., Kim, A. Y. B., Pearce, A., Viégas, F., Wattenberg, M., Yuan, A., Wattenberg, M., Viegas, F. B., Coenen, A., Pearce, A., and Kim, B.  Visualizing and Measuring the Geometry of BERT.  In Wallach, H., Larochelle, H., Beygelzimer, A., d\textquotesingle Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 32, pp. 8594–8603. Curran Associates, Inc., 6 2019.  URL https://proceedings.neurips.cc/paper/2019/file/159c1ffe5b61b41b3c4d8f4c2150f6c4-Paper.pdf.
  * Conneau et al. (2018) Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2126–2136, Stroudsburg, PA, USA, 2018. Association for Computational Linguistics.  ISBN 9781948087322.  doi: 10.18653/v1/P18-1198.  URL http://aclweb.org/anthology/P18-1198.
  * Covert et al. (2022) Covert, I. C., Kim, C., and Lee, S.-I.  Learning to Estimate Shapley Values with Vision Transformers.  _The Eleventh International Conference on Learning Representations_ , pp. 1–48, 2022.  URL http://arxiv.org/abs/2206.05282https://openreview.net/forum?id=5ktFNz_pJLK.
  * Dara et al. (2022) Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. H. M., and Ahsan, M. J.  Machine learning in drug discovery: a review.  _Artificial Intelligence Review_ , 55(3):1947–1999, 2022.
  * DARPA (2016) DARPA.  Explainable Artificial Intelligence (XAI) DARPA-BAA-16-53.  _Defense Advanced Research Projects Agency (DARPA)_ , pp. 1–52, 2016.  ISSN 15580644.  URL https://www.darpa.mil/attachments/DARPA-BAA-16-53.pdf.
  * Dauben (1977) Dauben, J. W.  Georg Cantor and Pope Leo XIII: Mathematics, Theology, and the Infinite.  _Journal of the History of Ideas_ , 38(1):85, 1 1977.  ISSN 00225037.  doi: 10.2307/2708842.  URL https://www.jstor.org/stable/2708842?origin=crossref.
  * Doshi-Velez & Kim (2017) Doshi-Velez, F. and Kim, B.  Towards A Rigorous Science of Interpretable Machine Learning.  _arXiv_ , 2 2017.  URL http://arxiv.org/abs/1702.08608.
  * Elhage et al. (2021) Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C.  A Mathematical Framework for Transformer Circuits.  _Anthropic_ , 2021.  URL https://transformer-circuits.pub/2021/framework/index.html.
  * ElShawi et al. (2019) ElShawi, R., Sherif, Y., Al-Mallah, M., and Sakr, S.  ILIME: Local and Global Interpretable Model-Agnostic Explainer of Black-Box Decision.  In Welzer, T., Eder, J., Podgorelec, V., and Kamišalić Latifić, A. (eds.), _Advances in Databases and Information Systems_ , pp. 53–68. Springer International Publishing, Cham, 2019.  ISBN 978-3-030-28730-6.  doi: 10.1007/978-3-030-28730-6–“˙˝4.  URL http://link.springer.com/10.1007/978-3-030-28730-6_4.
  * Elton (2020) Elton, D. C.  Self-explaining AI as an Alternative to Interpretable AI.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 12177 LNAI:95–106, 2 2020.  ISSN 16113349.  doi: 10.1007/978-3-030-52152-3–“˙˝10.  URL http://link.springer.com/10.1007/978-3-030-52152-3_10.
  * Fashandi (2023) Fashandi, H.  Neural module networks: A review.  _Neurocomputing_ , 552:126518, 2023.  ISSN 18728286.  doi: 10.1016/j.neucom.2023.126518.  URL https://doi.org/10.1016/j.neucom.2023.126518.
  * Flora et al. (2022) Flora, M., Potvin, C., McGovern, A., and Handler, S.  Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreement.  _arXiv_ , pp. 1–22, 2022.  URL http://arxiv.org/abs/2211.08943.
  * Fuller (2021) Fuller, J.  Companies Need More Workers. Why Do They Reject Millions of Résumés?  _The project on workforce_ , 2021.  URL https://www.pw.hks.harvard.edu/post/companies-need-more-workers-wsj.
  * Fuller et al. (2021) Fuller, J. B., Ramen, M., Sage-gavin, E., and Hines, K.  Hidden Workers: Untapped Talent.  _Harvard Business School Project on Managing the Future of Work and Accenture_ , 2021.  URL https://www.pw.hks.harvard.edu/post/hidden-workers-untapped-talent.
  * Geirhos et al. (2023) Geirhos, R., Zimmermann, R. S., Bilodeau, B., Brendel, W., and Kim, B.  Don’t trust your eyes: on the (un)reliability of feature visualizations.  _arXiv_ , 2023.  URL http://arxiv.org/abs/2306.04719.
  * Gödel (1931) Gödel, K.  On Formally Undecidable Propositions of Principia Mathematica and Related Systems I.  _Monatshefte für Mathematik_ , 1931.
  * Goodman & Flaxman (2017) Goodman, B. and Flaxman, S.  European union regulations on algorithmic decision making and a ”right to explanation”.  _AI Magazine_ , 38(3):50–57, 2017.  ISSN 07384602.  doi: 10.1609/aimag.v38i3.2741.
  * Gray (1991) Gray, J.  Did poincaré say “set theory is a disease”?  _The Mathematical Intelligencer_ , 13(1):19–22, 12 1991.  ISSN 0343-6993.  doi: 10.1007/BF03024067.  URL https://maa.org/press/periodicals/convergence/quotations/poincare-jules-henri-1854-1912-0http://link.springer.com/10.1007/BF03024067.
  * Gupta et al. (2020) Gupta, N., Lin, K., Roth, D., Singh, S., and Gardner, M.  Neural Module Networks for Reasoning over Text.  In _International Conference on Learning Representations (ICLR)_ , 12 2020.  URL https://openreview.net/forum?id=SygWvAVFPr.
  * Han et al. (2022) Han, T., Srinivas, S., and Lakkaraju, H.  Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations.  _Advances in Neural Information Processing Systems_ , 35(NeurIPS), 2022.  ISSN 10495258.  URL http://arxiv.org/abs/2206.01254.
  * Hewitt & Liang (2019) Hewitt, J. and Liang, P.  Designing and Interpreting Probes with Control Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2733–2743, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics.  ISBN 9781950737901.  doi: 10.18653/v1/D19-1275.  URL https://www.aclweb.org/anthology/D19-1275.
  * Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, 11 1997.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.8.1735.  URL https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735.
  * Hooker et al. (2019) Hooker, S., Erhan, D., Kindermans, P.-J. J., and Kim, B.  A benchmark for interpretability methods in deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 32, 6 2019.  URL http://arxiv.org/abs/1806.10758.
  * House of Lords (2017) House of Lords, U. G.  AI in the UK: Ready, Willing and Able?, 2017.  URL https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/10007.htm.
  * Huang et al. (2023) Huang, S., Mamidanna, S., Jangam, S., Zhou, Y., and Gilpin, L. H.  Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations.  _arXiv_ , 2023.  URL http://arxiv.org/abs/2310.11207.
  * Huygens (1690) Huygens, C.  _Traité de la lumière où sont expliquées les causes de ce qui luy arrive dans la reflexion, & dans la refraction, et particulierement dans l’etrange refraction du cistal d’Islande_.  Pieter van der Aa, Leiden, Netherlands, 1690.  URL https://archive.org/details/bub_gb_X9PKaZlChggC/mode/2up.
  * Jacovi & Goldberg (2020) Jacovi, A. and Goldberg, Y.  Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4198–4205, Stroudsburg, PA, USA, 4 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.386.  URL https://www.aclweb.org/anthology/2020.acl-main.386.
  * Jain & Wallace (2019) Jain, S. and Wallace, B. C.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North_ , volume 1, pp. 3543–3556, Stroudsburg, PA, USA, 2 2019. Association for Computational Linguistics.  ISBN 9781950737130.  doi: 10.18653/v1/N19-1357.  URL http://aclweb.org/anthology/N19-1357.
  * Jethani et al. (2021) Jethani, N., Sudarshan, M., Aphinyanaphongs, Y., and Ranganath, R.  Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations.  _Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 130:1459–1467, 2021.  ISSN 26403498.
  * Jethani et al. (2022) Jethani, N., Sudarshan, M., Covert, I., Lee, S. I., and Ranganath, R.  Fastshap: Real-Time Shapley Value Estimation.  _ICLR 2022 - 10th International Conference on Learning Representations_ , pp. 1–23, 2022.
  * Jiang et al. (2023) Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E.  Mistral 7B.  _arXiv_ , pp. 1–9, 2023.  URL http://arxiv.org/abs/2310.06825.
  * Jiménez-Luna et al. (2020) Jiménez-Luna, J., Grisoni, F., and Schneider, G.  Drug discovery with explainable artificial intelligence.  _Nature Machine Intelligence_ , 2(10):573–584, 2020.
  * Karpathy et al. (2015) Karpathy, A., Johnson, J., and Fei-Fei, L.  Visualizing and Understanding Recurrent Networks.  _arXiv_ , pp. 1–12, 6 2015.  URL http://arxiv.org/abs/1506.02078.
  * Kim (2022) Kim, B.  Beyond interpretability: developing a language to shape our relationships with AI.  In _The International Conference on Learning Representations_ , 2022.  URL https://iclr.cc/Conferences/2022/Schedule?showEvent=7237.
  * Kim et al. (2014) Kim, B., Rudin, C., and Shah, J.  The Bayesian case model: A generative approach for case-based reasoning and prototype classification.  _Advances in Neural Information Processing Systems_ , 3(January):1952–1960, 2014.  ISSN 10495258.
  * Kim et al. (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., and Sayres, R.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  _35th International Conference on Machine Learning, ICML 2018_ , 6:4186–4195, 11 2018.  URL http://arxiv.org/abs/1711.11279.
  * Kindermans et al. (2019) Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., and Kim, B.  The (Un)reliability of Saliency Methods.  In _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , volume 11700 LNCS, pp. 267–280. Springer, 11 2019.  doi: 10.1007/978-3-030-28954-6–“˙˝14.  URL http://link.springer.com/10.1007/978-3-030-28954-6_14.
  * Kodiyan (2019) Kodiyan, A. A.  An overview of ethical issues in using AI systems in hiring with a case study of Amazon’s AI based hiring tool.  _Researchgate Preprint_ , pp. 1–19, 2019.
  * Koh & Liang (2017) Koh, P. W. and Liang, P.  Understanding Black-box Predictions via Influence Functions.  _34th International Conference on Machine Learning, ICML 2017_ , 4:2976–2987, 3 2017.  URL http://arxiv.org/abs/1703.04730.
  * Krishna et al. (2022) Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., and Lakkaraju, H.  The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective.  _arXiv_ , 2022.  URL http://arxiv.org/abs/2202.01602.
  * Kuhn (1996) Kuhn, T. S.  _The Structure of Scientific Revolutions_.  University of Chicago Press, 3rd editio edition, 1996.  ISBN 978-0-226-45807-6.
  * Lanham et al. (2023) Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Telleen-Lawton, T., Hume, T., Hatfield-Dodds, Z., Kaplan, J., Brauner, J., Bowman, S. R., and Perez, E.  Measuring Faithfulness in Chain-of-Thought Reasoning.  _arXiv_ , 2023.  URL http://arxiv.org/abs/2307.13702.
  * Li et al. (2016) Li, J., Monroe, W., and Jurafsky, D.  Understanding Neural Networks through Representation Erasure.  _arXiv_ , 2016.  URL http://arxiv.org/abs/1612.08220.
  * Lipton (2018) Lipton, Z. C.  The mythos of model interpretability.  _Communications of the ACM_ , 61(10):36–43, 9 2018.  ISSN 0001-0782.  doi: 10.1145/3233231.  URL https://dl.acm.org/doi/10.1145/3233231.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv_ , 7 2019.  ISSN 23318422.  URL http://arxiv.org/abs/1907.11692.
  * Lyu et al. (2022) Lyu, Q., Apidianaki, M., and Callison-Burch, C.  Towards Faithful Model Explanation in NLP: A Survey.  _arXiv_ , 2022.  URL http://arxiv.org/abs/2209.11326.
  * Madsen et al. (2022a) Madsen, A., Meade, N., Adlakha, V., and Reddy, S.  Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 1731–1751, Abu Dhabi, United Arab Emirates, 12 2022a. Association for Computational Linguistics.  URL https://aclanthology.org/2022.findings-emnlp.125.
  * Madsen et al. (2022b) Madsen, A., Reddy, S., and Chandar, S.  Post-hoc Interpretability for Neural NLP: A Survey.  _ACM Computing Surveys_ , 55(8):1–42, 8 2022b.  ISSN 0360-0300.  doi: 10.1145/3546577.  URL https://dl.acm.org/doi/10.1145/3546577.
  * Madsen et al. (2024a) Madsen, A., Chandar, S., and Reddy, S.  Are self-explanations from Large Language Models faithful?  _arXiv_ , 1 2024a.  URL http://arxiv.org/abs/2401.07927.
  * Madsen et al. (2024b) Madsen, A., Reddy, S., Chandar, S., and Anonymous.  Faithfulness Measurable Masked Language Models.  In _Forty-first International Conference on Machine Learning_ , 2024b.  URL https://openreview.net/forum?id=tw1PwpuAuNhttp://arxiv.org/abs/2310.07819.
  * McCoy et al. (2019) McCoy, T., Pavlick, E., and Linzen, T.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 3428–3448, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics.  ISBN 9781950737482.  doi: 10.18653/v1/P19-1334.  URL https://www.aclweb.org/anthology/P19-1334.
  * Meister et al. (2021) Meister, C., Lazov, S., Augenstein, I., and Cotterell, R.  Is Sparse Attention more Interpretable?  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 122–129, Stroudsburg, PA, USA, 8 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.17.  URL http://arxiv.org/abs/2106.01087https://aclanthology.org/2021.acl-short.17.
  * Messiah (1966) Messiah, A.  _Quantum Mechanics_.  North Holland, John Wiley & Sons, 1966.  ISBN 0486409244.
  * Meta (2023) Meta.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _arXiv_ , 2023.  URL http://arxiv.org/abs/2307.09288.
  * Murdoch et al. (2019) Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B.  Definitions, methods, and applications in interpretable machine learning.  _Proceedings of the National Academy of Sciences of the United States of America_ , 116(44):22071–22080, 10 2019.  ISSN 10916490.  doi: 10.1073/pnas.1900654116.  URL http://www.pnas.org/lookup/doi/10.1073/pnas.1900654116.
  * Nguyen et al. (2016) Nguyen, A., Yosinski, J., and Clune, J.  Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks.  _Visualization for Deep Learning workshop at ICML_ , 2016.  URL http://arxiv.org/abs/1602.03616.
  * Olah et al. (2017) Olah, C., Mordvintsev, A., and Schubert, L.  Feature Visualization.  _Distill_ , 2(11), 11 2017.  ISSN 2476-0757.  doi: 10.23915/distill.00007.  URL https://distill.pub/2017/feature-visualization.
  * OpenAI (2023) OpenAI.  GPT-4 Technical Report.  _OpenAI_ , 4:1–100, 3 2023.  URL http://arxiv.org/abs/2303.08774.
  * Preuer et al. (2019) Preuer, K., Klambauer, G., Rippmann, F., Hochreiter, S., and Unterthiner, T.  Interpretable deep learning in drug discovery.  _Explainable AI: interpreting, explaining and visualizing deep learning_ , pp. 331–345, 2019.
  * Pruthi et al. (2020) Pruthi, G., Liu, F., Sundararajan, M., and Kale, S.  Estimating Training Data Influence by Tracing Gradient Descent.  In _Advances in Neural Information Processing Systems_ , 2 2020.  URL http://arxiv.org/abs/2002.08484.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  ”Why should i trust you?” Explaining the predictions of any classifier.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , volume 13-17-Augu, pp. 1135–1144, New York, NY, USA, 8 2016. ACM.  ISBN 9781450342322.  doi: 10.1145/2939672.2939778.  URL https://dl.acm.org/doi/10.1145/2939672.2939778.
  * Robnik-Šikonja & Bohanec (2018) Robnik-Šikonja, M. and Bohanec, M.  _Perturbation-Based Explanations of Prediction Models_.  Springer International Publishing, 2018.  ISBN 9783319904030.  doi: 10.1007/978-3-319-90403-0–“˙˝9.  URL http://dx.doi.org/10.1007/978-3-319-90403-0_9.
  * Rogers et al. (2020) Rogers, A., Kovaleva, O., and Rumshisky, A.  A Primer in BERTology: What We Know About How BERT Works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866, 12 2020.  ISSN 2307-387X.  doi: 10.1162/tacl–“˙˝a–“˙˝00349.  URL https://direct.mit.edu/tacl/article/96482.
  * Ross & Doshi-Velez (2018) Ross, A. and Doshi-Velez, F.  Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1):1660–1669, 4 2018.  ISSN 2374-3468.  doi: 10.1609/aaai.v32i1.11504.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11504.
  * Rudin (2019) Rudin, C.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.  ISSN 2522-5839.  doi: 10.1038/s42256-019-0048-x.  URL http://www.nature.com/articles/s42256-019-0048-x.
  * Samek et al. (2017) Samek, W., Binder, A., Montavon, G., Lapuschkin, S., and Muller, K.-R.  Evaluating the Visualization of What a Deep Neural Network Has Learned.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(11):2660–2673, 11 2017.  ISSN 2162-237X.  doi: 10.1109/TNNLS.2016.2599820.  URL https://ieeexplore.ieee.org/document/7552539/.
  * Sangroya et al. (2020) Sangroya, A., Rastogi, M., Anantaram, C., and Vig, L.  Guided-LIME: Structured sampling based hybrid approach towards explaining blackbox machine learning models.  In _CEUR Workshop Proceedings_ , volume 2699, 2020.
  * Schut et al. (2023) Schut, L., Tomasev, N., McGrath, T., Hassabis, D., Paquet, U., and Kim, B.  Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero.  _arXiv_ , pp. 1–61, 10 2023.  URL http://arxiv.org/abs/2310.16410.
  * Seo et al. (2018) Seo, J., Choe, J., Koo, J., Jeon, S., Kim, B., and Jeon, T.  Noise-adding Methods of Saliency Map as Series of Higher Order Partial Derivative.  In _2018 ICML Workshop on Human Interpretability in Machine Learning_ , 6 2018.  URL http://arxiv.org/abs/1806.03000.
  * Serrano & Smith (2019) Serrano, S. and Smith, N. A.  Is Attention Interpretable?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 2931–2951, Stroudsburg, PA, USA, 6 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1282.  URL https://www.aclweb.org/anthology/P19-1282.
  * Shrikumar et al. (2017) Shrikumar, A., Greenside, P., and Kundaje, A.  Learning important features through propagating activation differences.  In _34th International Conference on Machine Learning, ICML 2017_ , volume 7, pp. 4844–4866, 2017.  ISBN 9781510855144.  URL https://arxiv.org/.
  * Slack et al. (2020) Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H.  Fooling LIME and SHAP.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 180–186, New York, NY, USA, 2 2020. ACM.  ISBN 9781450371100.  doi: 10.1145/3375627.3375830.  URL https://dl.acm.org/doi/10.1145/3375627.3375830.
  * Smilkov et al. (2017) Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M.  SmoothGrad: removing noise by adding noise.  _ICML workshop on visualization for deep learning_ , 2017.  ISSN 23318422.  URL https://goo.gl/EfVzEE.
  * Smith (2014) Smith, J.  David Hilbert’s Radio Address.  _Convergence_ , 2014.  doi: 10.4169/convergence20140202.  URL https://www.maa.org/node/326611.
  * Srinivas et al. (2022) Srinivas, S., Matoba, K., Lakkaraju, H., and Fleuret, F.  Efficiently Training Low-Curvature Neural Networks.  _Advances in Neural Information Processing Systems_ , 35(NeurIPS):1–21, 6 2022.  ISSN 10495258.  URL http://arxiv.org/abs/2206.07144.
  * Subramanian et al. (2020) Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., and Gardner, M.  Obtaining faithful interpretations from compositional neural networks.  _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pp. 5594–5608, 2020.  ISSN 0736587X.  doi: 10.18653/v1/2020.acl-main.495.  URL https://www.aclweb.org/anthology/2020.acl-main.495.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks.  In _34th International Conference on Machine Learning, ICML 2017_ , volume 7, pp. 5109–5118, 3 2017.  ISBN 9781510855144.  URL http://arxiv.org/abs/1703.01365.
  * Tenney et al. (2019) Tenney, I., Das, D., and Pavlick, E.  BERT Rediscovers the Classical NLP Pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4593–4601, Stroudsburg, PA, USA, 5 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1452.  URL https://www.aclweb.org/anthology/P19-1452.
  * Thorne et al. (2019) Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A.  Generating Token-Level Explanations for Natural Language Inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , volume 1, pp. 963–969, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics.  ISBN 9781950737130.  doi: 10.18653/v1/N19-1101.  URL http://aclweb.org/anthology/N19-1101.
  * Vashishth et al. (2019) Vashishth, S., Upadhyay, S., Tomar, G. S., and Faruqui, M.  Attention Interpretability Across NLP Tasks.  _arXiv_ , 9 2019.  URL http://arxiv.org/abs/1909.11218.
  * Voita & Titov (2020) Voita, E. and Titov, I.  Information-Theoretic Probing with Minimum Description Length.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 183–196, Stroudsburg, PA, USA, 3 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.14.  URL https://www.aclweb.org/anthology/2020.emnlp-main.14.
  * Xiang & Raji (2019) Xiang, A. and Raji, I. D.  On the Legal Compatibility of Fairness Definitions.  _Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems_ , 2019.  URL http://arxiv.org/abs/1912.00761.
  * Yeh et al. (2018) Yeh, C.-K., Kim, J. S., Yen, I. E. H., and Ravikumar, P.  Representer Point Selection for Explaining Deep Neural Networks.  In _Advances in Neural Information Processing Systems_ , pp. 9291–9301, 11 2018.  URL http://arxiv.org/abs/1811.09720.
  * Yeh et al. (2019) Yeh, C.-K., Hsieh, C.-Y., Suggala, A., Inouye, D. I., Ravikumar, P. K., Sai Suggala, A., Inouye, D. I., and Ravikumar, P. K.  On the (In)fidelity and Sensitivity of Explanations.  In Wallach, H., Larochelle, H., Beygelzimer, A., d\textquotesingle Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 32_ , pp. 10967–10978. Curran Associates, Inc., Vancouver, Canada, 2019.  URL https://arxiv.org/abs/1901.09392.
  * Yoon et al. (2019) Yoon, J., Jordon, J., and Van Der Schaar, M.  Invase: Instance-wise variable selection using neural networks.  _7th International Conference on Learning Representations, ICLR 2019_ , pp. 1–24, 2019.
  * Yosinski et al. (2015) Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H.  Understanding Neural Networks Through Deep Visualization.  In _Deep Learning Workshop at 31st International Conference on Machine Learning_ , 2015.  URL http://arxiv.org/abs/1506.06579.
  * Zhang & Bowman (2018) Zhang, K. and Bowman, S.  Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 359–361, Stroudsburg, PA, USA, 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5448.  URL http://aclweb.org/anthology/W18-5448.
  * Zhou & Shah (2023) Zhou, Y. and Shah, J.  The Solvability of Interpretability Evaluation Metrics.  In _Findings of the Association for Computational Linguistics: EACL_ , 2023.  URL http://arxiv.org/abs/2205.08696.
  * Zimmermann et al. (2021) Zimmermann, R. S., Borowski, J., Geirhos, R., Bethge, M., Wallis, T. S., and Brendel, W.  How Well do Feature Visualizations Support Causal Understanding of CNN Activations?  _Advances in Neural Information Processing Systems_ , 14(NeurIPS):11730–11744, 2021.  ISSN 10495258.