#  Faithful Explanations of Black-box NLP Models Using LLM-generated
Counterfactuals

Yair Ori GatT∗, Nitay CalderonT∗, Amir FederC,
Alexander ChapaninT, Amit SharmaM and Roi ReichartT
TFaculty of Data and Decision Sciences, Technion, IIT
CColumbia University Data Science Institute, MMicrosoft Research India
∗Equal contribution. Corresponding author<EMAIL_ADDRESS>
###### Abstract

Causal explanations of the predictions of NLP systems are essential to ensure
safety and establish trust. Yet, existing methods often fall short of
explaining model predictions effectively or efficiently and are often model-
specific. In this paper, we address model-agnostic explanations, proposing two
approaches for counterfactual (CF) approximation. The first approach is CF
generation, where a large language model (LLM) is prompted to change a
specific text concept while keeping confounding concepts unchanged. While this
approach is demonstrated to be very effective, applying LLM at inference-time
is costly. We hence present a second approach based on matching, and propose a
method that is guided by an LLM at training-time and learns a dedicated
embedding space. This space is faithful to a given causal graph and
effectively serves to identify matches that approximate CFs. After showing
theoretically that approximating CFs is required in order to construct
faithful explanations, we benchmark our approaches and explain several models,
including LLMs with billions of parameters. Our empirical results demonstrate
the excellent performance of CF generation models as model-agnostic
explainers. Moreover, our matching approach, which requires far less test-time
resources, also provides effective explanations, surpassing many baselines. We
also find that Top-K techniques universally improve every tested method.
Finally, we showcase the potential of LLMs in constructing new benchmarks for
model explanation and subsequently validate our conclusions. Our work
illuminates new pathways for efficient and accurate approaches to interpreting
NLP systems.

##  1 Introduction

Providing faithful explanations for Natural Language Processing (NLP) model
predictions is imperative to guarantee safe deployment, establish trust, and
foster scientific discoveries (Amodei et al., 2016; Goodman & Flaxman, 2017;
Guidotti et al., 2019; Jacovi & Goldberg, 2020). These aspects are
particularly significant in NLP, where the complexity of language and the
opaque behavior of black-box models. Many past works focus on what knowledge a
model encodes (Lyu et al., 2022). However, just because a model encodes a
myriad of features does not mean all are utilized in decision-making. For an
explanation to be genuinely faithful and accurately depict a model’s
underlying reasoning, it is crucial to establish causality. Recognizing this
inherent link and following previous works (Vig et al., 2020; Geiger et al.,
2020; Feder et al., 2021b), this paper introduces a theoretical framework that
binds the two together, providing another evidence that non-causal explanation
methods can occasionally fall short of being truly faithful.

In contrast to model explanation techniques that often conflate correlation
with causation, causal-inspired methods often contrast predictions for an
input example with those of its counterfactual (Soulos et al., 2020; Elazar et
al., 2021; Finlayson et al., 2021). Indeed, counterfactuals are at the highest
level of Pearl’s causal hierarchy (Pearl, 2009), highlighting how changes lead
to a different prediction. However, they cannot be acquired without knowing
the complete data-generating process (or structural model) of the text (Balke
& Pearl, 1994), which is not practical for a given real-world problem. Hence,
we turn to counterfactual approximations (CFs): imagining how a given text
would look like if a certain concept in its generative process were different
(e.g., a different domain for an Amazon product review, as in Calderon et al.
(2022)). Nevertheless, acquiring CFs in past work was limited to simple local
manipulations (Ribeiro et al., 2020; Ross et al., 2021; Wu et al., 2021) or
costly manual annotation (Gardner et al., 2020; Kaushik et al., 2020a),
hindering practical causal effect estimation of high-level concepts on NLP
models.

To address the above limitations, we rely on Large Language Models (LLMs) and
introduce two practical approaches for model explanation. The first approach
is Counterfactual Generation, in which a generative model (an LLM) is prompted
to change a concept of a text while holding confounding concepts fixed. The
first contribution of this paper is our empirical evidence that CF generating
LLMs are SOTA explainers. However, utilizing LLMs comes with a high
computational and financial cost. Moreover, the slow autoregressive generation
process may limit the application of CF generation methods, especially in
scenarios demanding real-time explanations or when explaining vast quantities
of data (Calderon et al., 2023). Therefore, we introduce an efficient
alternative: Matching. In comparison to LLM-based CF generation, the matching
technique is up to 1000 times faster (§B)

We use matching to estimate the causal effect, pairing each treated unit with
one or more control units with similar observed characteristics, i.e.,
identifying CF approximations within a pre-defined candidate set. To enhance
matching quality, we propose in §3.3 a novel method for learning a causal
embedding space. Given a causal graph (e.g., section 3.1), we first employ an
LLM (ChatGPT) for generating CFs (note that the LLM is used only during the
learning phase). We also identify possible valid matches: Instances with
identical values in observable adjustment variables (that satisfy the back-
door criterion). Subsequently, we train our causal representation model by
minimizing an objective composed of contrastive loss components. The causal
representation model learns to encode examples similarly to their LLM-
generated CFs or matched examples. Conversely, it is trained to produce
dissimilar representations for the query example and any misspecified CFs. As
a result, representations remain faithful to the causal graph and can
effectively identify matching candidates.

In §3.1, we propose a simple, intuitive, and essential criterion for
explanation methods: Order-faithfulness – If the explanation method ranks one
concept’s impact above another’s, its true causal effect should genuinely be
greater. We then theoretically show that approximating CF methods are always
order-faithful, in contrast to non-causal explanation methods (that overlook
confounders, for example). Moreover, since CFs do not depend on the explained
model, they enable causal estimation in a model-agnostic manner. Model-
agnostic explanation methods offer numerous advantages, especially during
model selection, debugging, and deployment. For example, developers juggling
multiple models can effortlessly rank them based on their vulnerability to
confounding biases (such as gender bias). Hence, we focus only on model-
agnostic explanations in this study.

We undertake rigorous experiments to compare our causal language
representation method against various generative and matching baselines. For
this purpose, we employ CEBaB (Abraham et al., 2022), the only established
benchmark for evaluating causal explanation methods. We estimate the causal
effect of 24 concept interventions on the predictions of five models, three
small fine-tuned LMs, and two zero-shot LLMs with billions of parameters. In
§5.1 we also conduct an ablation study to examine the impact of each design
choice within our causal model.

Our empirical results reveal three key findings: (1) The counterfactual
generation approach provides strong black-box model explanations. (2) Our
method for learning causal representations for matching outperforms all the
matching baselines, including the best-performing method from the CEBaB paper
(see Figure 5). (3) Top-$K$ matching universally greatly enhances the
explanatory capabilities of all examined methods, including generative models,
when generating multiple CFs.

Finally, building on our findings that LLMs can produce high-quality CFs, we
address a pressing challenge: The lack of benchmarks for NLP model explanation
methods. In §6 we demonstrate how to construct such a benchmark by leveraging
GPT-4 to generate pairs of inputs and their CFs (used as “golden CF”). We
introduce a new CEBaB-like dataset, focusing on Stance Detection, an NLP task
that determines the position expressed by the speaker towards a particular
topic (e.g., abortions, climate change) (Küçük & Can, 2021; Viehmann et al.,
2023), where model explanations may guide decision-making processes or reveal
societal biases. Our novel dataset allows evaluation in an out-of-distribution
setting, such as when the matching candidate set has topics differing from the
test examples. Utilizing the new dataset, we observe similar patterns to our
main conclusions. We hope that this study will pave the way for safer, more
transparent, and more accountable AI systems.

##  2 Related work

Explanation methods and causality. Probing is a common technique for
understanding what model internal representations encode. In probing, a small
supervised (Conneau et al., 2018; Tenney et al., 2019) or unsupervised (Clark
et al., 2019; Manning et al., 2020; Saphra & Lopez, 2019) model is used to
measure whether specific concepts are encoded at specific places in a network.
While probes have helped illuminate what models have learned from data, Geiger
et al. (2021) show that probes cannot reliably provide causal explanations.
Feature importance tools can also be seen as explanation methods (Molnar,
2020), often restricted to input features, although gradient-based methods can
quantify the relative importance of hidden states as well (Zeiler & Fergus,
2014; Binder et al., 2016; Shrikumar et al., 2017; Gat et al., 2022). The
Integrated Gradients method of Sundararajan et al. (2017) has a natural causal
interpretation stemming from its exploration of baseline (CF) inputs (Geiger
et al., 2021). However, even where these methods can focus on internal states,
it remains challenging to connect their analyses with real-world concepts that
do not reduce to simple properties of inputs. In this study, we focus on the
explanation of high-level real-world concepts.

Intervention-based methods involve modifying inputs or internal
representations and studying the effects that this has on model behavior
(Lundberg & Lee, 2017; Ribeiro et al., 2016). Recent methods perturb input or
hidden representations to create CF states that can then be used to estimate
causal effects (Elazar et al., 2021; Finlayson et al., 2021; Soulos et al.,
2020; Vig et al., 2020; Geiger et al., 2021). However, these methods are prone
to generating implausible inputs or network states unless the interventions
are carefully controlled (Geiger et al., 2020). Recently, Wu et al. (2023)
proposed the Causal Proxy Model, a novel explanation method inspired by
S-learner (Künzel et al., 2019), which mimics the behavior of the explained
model when the input is a CF. Although CPM is effective, the need to train a
distinct explainer for each explained model is a major disadvantage.
Conversely, our study focuses on model-agnostic explanation methods.

Approximating counterfactuals. A common use-case for CF examples in machine
learning is for _data augmentation_. These CFs involve perturbations to
confounding factors (Garg et al., 2019), or to the label (Kaushik et al.,
2019; 2020b; Jha et al., 2020). CF examples can be generated through manual
editing, heuristic keyword replacement, or automated text rewriting (Kaushik
et al., 2019; Gardner et al., 2020; Shekhar et al., 2017; Garg et al., 2019;
Feder et al., 2021a; Zmigrod et al., 2019; Riley et al., 2020; Wu et al.,
2021; Mao et al., 2021; Rosenberg et al., 2021). Manual editing is accurate
but expensive, while keyword-based methods can be limited in coverage and
difficult to generalize across languages (Antoniak & Mimno, 2021). Generative
approaches offer a balance of fluency and coverage (Zhou & Wu, 2023). CF
examples help address causal inference’s missing data issues, but generating
meaningful CFs without LLMs is challenging. Our work uses causal
representations to match examples from the treatment group (i.e., the concept
for which the causal model effect is estimated) with similar examples from the
control group.

Faithful Explanations. Faithfulness is a desirable property of any explanation
method, broadly defined as the ability of the method to provide accurate
descriptions of the underlying reasoning of the explained model (Jain &
Wallace, 2019; Jacovi & Goldberg, 2020). As of yet, there is no universally
accepted or formal definition of this term (Lyu et al., 2022). Instead, many
opt to characterize faithfulness using a variety of evaluation metrics,
including Sensitivity (Sundararajan et al., 2017; Adebayo et al., 2018;
Kindermans et al., 2018), Consistency (Liu et al., 2022), Feature Importance
Agreement (Wiegreffe et al., 2021), Robustness (Ghorbani et al., 2019; Yin et
al., 2022) and Simulatability (Sushil et al., 2018; Sia et al., 2023; Li et
al., 2020; Pruthi et al., 2022). Most of these works focus on feature-level
explanations, rely on access to the internal mechanism of the model, train an
external model, and occasionally conflate between correlation and causation.
In this work, we focus on high-level concept explanations. Relying on the
causality literature, in §3.1 we propose a simple, intuitive formal criterion,
which is also a necessary condition for the above broad definition of
faithfulness: Order-Faithfulness.

##  3 Method

###  3.1 Model explanation with counterfactuals

This study focuses on model explanation with counterfactuals (CFs), which
enables causal estimation in a model-agnostic manner (because CFs do not
depend on the explained model). While we are aware of the causal graph that
encodes our causal assumptions, for a given problem we cannot access the
complete data-generating process (DGP) nor the variable values of an example,
and therefore, we cannot produce gold CFs. Given this, we propose two
approaches for approximating them.

The first approach we introduce is Counterfactual generation, in which a
generative model (an LLM) is prompted to change an attribute conveyed in the
text while holding confounding attributes fixed. Nevertheless, utilizing an
LLM at inference time is frequently impractical because of its prolonged
latency, high financial costs, or privacy concerns that prevent data from
being sent to external servers (see the discussion in Calderon et al. (2023)
and our latency measurements in §B).

Considering these challenges, we introduce an efficient alternative, the
matching technique. This approach aims to find CF approximations within a pre-
defined candidate set. To further enhance the matching process, we propose a
novel method that leverages CFs generated by an LLM during training time for
learning causal representations that are faithful to the causal graph.

Table 1: Toy illustration of examples (and their concept values) sampled from
the four sets of a given query example and an intervention of changing service
to positive ($S\leftarrow+$). The misspecified CF
(${\mathbb{X}}_{\texttt{¬CF}}$) resulted from a wrong intervention of removing
the ambiance concept ($A\leftarrow\texttt{?}$).

Query Excellent lobster and decor, but rude waiter. $F:+\hfill S:-\hfill
N:\texttt{?}\hfill A:+$ ${\mathbb{X}}_{\texttt{CF}}$ Excellent lobster and
decor, and friendly waiter. $F:+\hfill S:+\hfill N:\texttt{?}\hfill A:+$
${\mathbb{X}}_{\texttt{M}}$ A cozy atmosphere with great pizza and service.
$F:+\hfill S:+\hfill N:\texttt{?}\hfill A:+$ ${\mathbb{X}}_{\texttt{¬CF}}$
Excellent lobster and rude waiter. $F:+\hfill S:+\hfill N:\texttt{?}\hfill
A:\texttt{?}$ ${\mathbb{X}}_{\texttt{¬M}}$ Too expensive for a loud chaos with
bland food. $F:-\hfill S:\texttt{?}\hfill N:-\hfill A:-$

Figure 1: Our illustration of the causal graph of the CEBaB benchmark (Abraham
et al., 2022). $U$ and $V$ are (unobserved) exogenous variables representing
the state of the world. For example, $V$ may account for style, syntax, and
length. The four concepts $F$ (food), $S$ (service), $A$ (ambiance) and $N$
(noise) affect both the textual review $X$ and its five-star rating $Y$. The
relationship between $X$ and $Y$ can be causal or anti-causal. $f$ is the
model we wish to explain, which is trained to predict $Y$ from $X$. In this
study, we are interested in estimating the causal effect of changing a concept
on the predictive model $f$ (e.g., changing the value of the red variable $S$
from positive to negative).

To provide an accurate causal explanation, we should estimate the Average
Treatment Effect (ATE) (Pearl, 2009) or the Individual Treatment Effect (ITE)
(Shpitser & Pearl, 2006; Shalit et al., 2017). Specifically, the treatment is
a high-level concept describing the text (such as ambiance in a restaurant
review), and the outcome variable is a prediction of a text classifier (e.g.,
see the causal graph in Figure 3.1). When discussing model explanation in the
sense of the causal effect of a concept on the model prediction, the common
versions of the ATE and the ITE are the Causal Concept Effect (CaCE) (Goyal et
al., 2019) and the Individual Causal Concept Effect (ICaCE) (Abraham et al.,
2022).

Given a DGP ${\mathcal{G}}$, an intervention on a treatment variable
$T:t\rightarrow t^{\prime}$ (for simplicity we sometimes write $T\leftarrow
t^{\prime}$), a model $f$, and an input example ${\bm{x}}_{t}$, the CaCE and
ICaCE are defined to be:

$\displaystyle\texttt{CaCE}_{f}(T,t,t^{\prime})$
$\displaystyle=\mathbb{E}_{{\bm{x}}^{\prime}\sim{\mathcal{G}}}\left[f({\bm{x}}^{\prime})|do(T=t^{\prime})\right]-\mathbb{E}_{{\bm{x}}^{\prime}\sim{\mathcal{G}}}\left[f({\bm{x}}^{\prime})|do(T=t)\right]$
(1) $\displaystyle\texttt{ICaCE}_{f}({\bm{x}}_{t},T,t^{\prime})$
$\displaystyle=\mathbb{E}_{{\bm{x}}^{\prime}\sim{\mathcal{G}}}\left[f({\bm{x}}^{\prime})|{\bm{x}}_{t},do(T=t^{\prime})\right]-f({\bm{x}}_{t})$

If $\widetilde{{\bm{x}}}_{t^{\prime}}$ is an approximated CF of the text
${\bm{x}}_{t}$ resulted from the intervention $T:t\rightarrow t^{\prime}$ (can
be either a model-generated CF or a match), then the causal effect estimators
are:

$\displaystyle\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},T,t,t^{\prime})$
$\displaystyle:=\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}_{t^{\prime}})=f(\widetilde{{\bm{x}}}_{t^{\prime}})-f({\bm{x}}_{t})$
(2) $\displaystyle\widehat{\texttt{CaCE}_{f}}(T,t,t^{\prime})$
$\displaystyle=\frac{1}{|{\mathbb{D}}|}\left(\sum_{{\bm{x}}_{t}}{\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}_{t^{\prime}})}+\sum_{{\bm{x}}_{t^{\prime}}}{\widehat{\texttt{ICaCE}_{f}}(\widetilde{{\bm{x}}}_{t},{\bm{x}}_{t^{\prime}})}+\sum_{{\bm{x}}}{\widehat{\texttt{ICaCE}_{f}}(\widetilde{{\bm{x}}}_{t},\widetilde{{\bm{x}}}_{t^{\prime}})}\right)$

###  3.2 Counterfactuals as an ideal model explanation

While definitions of faithfulness vary across literature, there is a consensus
that a faithful explanation method should accurately describe a model’s
underlying reasoning (Rudin, 2019; Jacovi & Goldberg, 2020; Dasgupta et al.,
2022). Lyu et al. (2022) distinguish between two primary avenues of model
explanations: what knowledge a model encodes and why a model makes certain
predictions. Nevertheless, a prevailing misconception is that the what avenue
directly informs what the model uses in making predictions — a flawed notion.
While a model can encode a myriad of features, it does not imply that all are
utilized in decision-making. Explanations derived from the what avenue, thus,
tend to be correlational rather than causal. Focusing on the why is imperative
for understanding a model’s reasoning mechanisms, making causality
indispensable. Consequently, causality must be embedded at the core of the
definition of faithfulness.

Therefore, relying on the causality literature, we propose a simple,
intuitive, and essential criterion: Order-faithfulness. This implies that if
the explanation method ranks the impact of one concept (or intervention)
higher than another, the true causal effect of the first concept should
genuinely exceed that of the second. Order-faithfulness is crucial in
scenarios like model selection and deployment. For instance, to determine the
fairness of a hiring model, it is essential to know whether it incorrectly
prioritizes non-relevant concepts (e.g., gender) over genuine concepts (such
as experience). We next formally define this necessary property.

######  Definition (Order-Faithfulness).

Given an i.i.d. dataset ${\mathbb{D}}$ sampled from a DGP ${\mathcal{G}}$, an
explained model $f$ and a pair of interventions $C_{1}:c_{1}\rightarrow
c^{\prime}_{1},C_{2}:c_{2}\rightarrow c^{\prime}_{2}$, an explanation method
$S$ is order-faithful if:

$\displaystyle\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S(f,C_{1},c_{1},c^{\prime}_{1})]>\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S(f,C_{2},c_{2},c^{\prime}_{2})]\Longleftrightarrow\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})>\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})$
(3)

Note that the definition above does not mandate the explanation method to
estimate the causal effect accurately or generate scores proportional to the
causal effects. Instead, it simply requires the method to be rank-preserving.
In addition, it holds for comparing the importance of two distinct concepts,
as many attribution methods like LIME do (Ribeiro et al., 2016), and it also
holds when assessing the importance of changes in the same concept. We believe
that order-faithfulness is a necessary condition for the broad definition of
faithfulness above: to accurately describe the underlying reasoning of the
explained model (Jacovi & Goldberg, 2020; Lyu et al., 2022). If an explanation
method ranks the importance of one concept above another, but in fact its
causal effect is smaller, then the method does not accurately reflect the
actual reasoning process of the model being explained.

In Appendix §A we provide detailed formal definitions of two explanation
methods: Approximated CF explanation methods and Non-causal explanation
methods (which we define as any function of the observed data). Furthermore,
we prove the following theorem, which elucidates why approximated CF methods
are always order-faithful, unlike non-causal ones.

######  Theorem.

For an explained model $f$, (1) The approximated CF explanation method
$S_{CF}$ is order-faithful for every DGP ${\mathcal{G}}$ and a pair of
interventions; (2) For every DGP ${\mathcal{G}}$, there exist a DGP
${\mathcal{G}}^{\prime}$ resulting from a small modification of
${\mathcal{G}}$, a model $f^{\prime}$, and a pair of interventions
$C_{1}:c_{1}\rightarrow c^{\prime}_{1},C_{2}:c_{2}\rightarrow c^{\prime}_{2}$
such that the non-causal explanation method $S_{NC}$ is not order-faithful.

Proof sketch: We first prove that approximated CF methods are always faithful
by showing the expected prediction of an approximated CF is equal to the
interventional one (conditioned on the do operator). To prove the second part,
we construct ${\mathcal{G}}^{\prime}$ by introducing a new unobserved
confounder variable (the “small modification”) that reverses the order of two
interventions’ causal effect on a new model $f^{\prime}$. In addition, we
ensure that the non-causal method produces the same explanations as it
produces for ${\mathcal{G}}$ since the generation process of ${\bm{x}}$ is the
same in the two DGPs. Hence, $S_{NC}$ cannot be order-faithful in
${\mathcal{G}}$ and ${\mathcal{G}}^{\prime}$ at the same time.

The theorem underscores the importance of developing causal-inspired
explanation methods and suggests why our CF-based approaches are more
effective than other baselines.

###  3.3 Causal representation learning for matching

Although counterfactual generation is a valuable model explanation approach,
employing LLMs during inference can be infeasible. Conversely, there is an
opportunity to harness model-generated CFs to train an efficient method that
seeks approximations within a pre-defined set of candidate texts, mitigating
the need for direct generation. This approach is known as matching (Stuart,
2010) and endeavors to match each treatment unit with its most similar control
unit. To allow effective matching, we introduce a novel method that learns to
encode texts in a space faithful to the causal graph, i.e., each treatment
unit resonates closely with its CF counterparts and remains distinct from
confounding texts. Formally, a matched example of ${\bm{x}}_{t}$ is defined to
be:

$\displaystyle
m_{1}({\bm{x}}_{t})=\operatorname*{arg\,max}_{{\bm{x}}_{t^{\prime}}\in{\mathbb{D}}(T=t^{\prime})}{s\left(\phi({\bm{x}}_{t}),\phi({\bm{x}}_{t^{\prime}})\right)}$
(4)

The candidate set is denoted by ${\mathbb{D}}(T=t^{\prime})$ and is also known
as the control group, contains only texts whose treatment variable equals
$t^{\prime}$, and is known in advance (i.e., both the treatment and control
groups are pre-defined and there is no need to infer which text belong to
which group). $\phi(\cdot)$ is a feature extractor function (the language
representation model we aim to learn) that maps discrete text to the
hypersphere space where matching is conducted by ranking candidates according
to their cosine similarity (see Eq. 11) with the query example:
$s(\phi({\bm{x}}_{t}),\phi(\texttt{candidate}))$. Notice that Eq 4 defines 1:1
matching, but we also experiment with Top-$K$ matching:
$\frac{1}{K}\sum_{i=1}^{K}\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},m_{i}({\bm{x}}_{t}))$.

Our causal representation method is based on an Encoder-only language model
(e.g., RoBERTa (Liu et al., 2019) or a SentenceTransformer (Reimers &
Gurevych, 2019) in our case) fine-tuned to minimize an objective consisting of
six contrastive loss components. The goal of the objective is to enhance the
similarity with approximate counterfactuals and matches while reducing
similarity with misspecified counterfactuals and matches.

Given a causal graph, an intervention on a treatment variable $T:t\rightarrow
t^{\prime}$ and an example ${\bm{x}}_{t}$, we define the four following sets
(see toy examples in Table 3.1):

  1. 1.

${\mathbb{X}}_{\texttt{CF}}$ \- This set contains Counterfactuals, obtained by
modifying ${\bm{x}}_{t}$ through the intervention on $T$, while the other
variables are held unchanged. We utilize an LLM (ChatGPT) to simulate several
CFs (up to ten).

  2. 2.

${\mathbb{X}}_{\texttt{M}}$ \- This set contains Matched samples. These
instances share identical values for the adjustment variables (that satisfy
the back-door criterion, see Appendix §C). Our study employs a smaller model
to predict their values and construct a set of valid matches.

  3. 3.

${\mathbb{X}}_{\texttt{¬CF}}$ \- This set contains Misspecified
Counterfactuals. These examples are derived from editing ${\bm{x}}_{t}$
through interventions on variables other than the treatment variable. The term
misspecification is well-known in the causal literature and indicates a wrong
assumption of the DGP (Vansteelandt et al., 2012). In our case, a wrong
intervention.

  4. 4.

${\mathbb{X}}_{\texttt{¬M}}$ \- This set contains Misspecified Matched
samples. This is the complementary set to ${\mathbb{X}}_{\texttt{M}}$,
containing all those instances that do not qualify as valid matches due to
differing values of the adjustment variables.

We next formulate the contrastive loss Wang & Liu (2021), which aims to
attract some input ${\bm{x}}$ to the “positive set” ${\mathbb{X}}_{+}$, and
separate it from the “negative set” ${\mathbb{X}}_{-}$. $\tau$ is a
temperature hyperparameter.

$\displaystyle\mathcal{L}({\bm{x}},{\mathbb{X}}_{+},{\mathbb{X}}_{-})=-\log\left[\frac{\sum_{{\bm{x}}_{+}}{\exp(s(\phi({\bm{x}}),\phi({\bm{x}}_{+}))/\tau)}}{\sum_{{\bm{x}}_{+}}{\exp(s(\phi({\bm{x}}),\phi({\bm{x}}_{+}))/\tau)}+\sum_{{\bm{x}}_{-}}{\exp(s(\phi({\bm{x}}),\phi({\bm{x}}_{-}))/\tau})}\right]$
(5)

The final loss consists of six contrastive loss components:

$\displaystyle\mathcal{L}({\bm{x}}_{t})=$
$\displaystyle\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{CF}},{\mathbb{X}}_{\texttt{¬M}})+\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{CF}},{\mathbb{X}}_{\texttt{¬CF}})+\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{CF}},{\mathbb{X}}_{\texttt{M}})$
$\displaystyle+$
$\displaystyle\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{M}},{\mathbb{X}}_{\texttt{¬M}})+\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{M}},{\mathbb{X}}_{\texttt{¬CF}})$
(6) $\displaystyle+$
$\displaystyle\mathcal{L}({\bm{x}}_{t},{\mathbb{X}}_{\texttt{¬CF}},{\mathbb{X}}_{\texttt{¬M}})$

We hypothesize (and verify) that omitting components from the objective risks
favoring inaccurate matches, as we highlight in the ablation study at §5.1.
Accordingly, the objective prioritizes text matches that closely resemble CF
approximations within a candidate set, defaulting to adjusting variable values
in their absence. When choosing between misspecified CFs or misspecified
matches, the methods favor the latter, given their potential shared traits
with the CF, such as syntax or style linked to exogenous variables. Formally,
given a query example ${\bm{x}}_{t}$ and four matching candidates
${\bm{x}}_{\texttt{CF}}\in{\mathbb{X}}_{\texttt{CF}},{\bm{x}}_{\texttt{M}}\in{\mathbb{X}}_{\texttt{M}},{\bm{x}}_{\texttt{¬CF}}\in{\mathbb{X}}_{\texttt{¬CF}}$,
the candidates’ ranking, which is based on their similarity with
${\bm{x}}_{t}$, should be
${\bm{x}}_{\texttt{¬M}}\preceq{\bm{x}}_{\texttt{¬CF}}\preceq{\bm{x}}_{\texttt{M}}\preceq{\bm{x}}_{\texttt{CF}}$.
This desirable order is not arbitrary and is a direct product of the six
components, ensuring robustness to variations of the candidate set.

#### Training procedure.

We start with a small set of textual examples (the train set) labeled with the
adjusting variables (e.g., review concepts such as food and service).
Initially, we employ a small Encoder-only model (RoBERTa) and train a concept
predictor for each variable. The predictor models are then harnessed to
construct the four sets: By predicting the concept values of each training
example, we divide them (according to their concept values) and construct the
${\mathbb{X}}_{\texttt{M}}$ and ${\mathbb{X}}_{\texttt{¬M}}$ sets.

For constructing ${\mathbb{X}}_{\texttt{CF}}$ and
${\mathbb{X}}_{\texttt{¬CF}}$, we use a few-shot LLM (ChatGPT) to generate
approximate CFs and misspecified CFs (see prompts at §D.1). We filter out
misspecified CFs from ${\mathbb{X}}_{\texttt{CF}}$ if the concept predictors
indicate an adjusted variable was also changed during the treatment
intervention. Additionally, we use simple rules to filter unsuccessful
generations (e.g., empty strings, “As an AI LM…”).

To train our language representation model (aimed to explain a specific
concept), we proceed as follows: For every training example ${\bm{x}}_{t}$, we
randomly sample four examples:
${\bm{x}}_{\texttt{CF}}\in{\mathbb{X}}_{\texttt{CF}},{\bm{x}}_{\texttt{M}}\in{\mathbb{X}}_{\texttt{M}},{\bm{x}}_{\texttt{¬CF}}\in{\mathbb{X}}_{\texttt{¬CF}}$.
The training goal is to minimize the objective of Eq. 6. We repeat this
process for several epochs (15) and select the checkpoint that achieves the
lowest loss on the dev. set.

##  4 Experimental setup

###  4.1 Model explanation pipeline

Causal Estimation-Based Benchmark (CEBaB). CEBaB (Abraham et al., 2022)
comprises high-quality, labeled CFs designed to benchmark model explanation
methods. This benchmark originated from thousands of original restaurant
reviews obtained from OpenTable. For every original review, human annotators
were tasked to edit the review and write an approximate CF that reflects a
specific intervention, e.g., “change the service concept from positive to
negative”. Each entry (original reviews and approximate CFs) in CEBaB received
a 5-star sentiment rating, labeled by five annotators. Furthermore, every text
was annotated at a conceptual level, as positive, negative or neutral w.r.t.
four central mediating concepts: Food, Service, Ambiance, and Noise (see
Figure 3.1). CEBaB contains two train sets, exclusive ($N=1463$) and inclusive
(which we do not use), development ($N=1672$), and test ($N=1688$) sets. We
randomly split the exclusive set into two equal sets: the train set (for
training the causal representation model and the concept predictors) and a
matching candidates set.

Evaluation measures. CEBaB allows estimating the CaCE and ICaCE by plugging
the human-written CF, $\widetilde{{\bm{x}}}^{h}_{t^{\prime}}$, into the
$\widehat{\texttt{ICaCE}_{f}}$ estimator:
$\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}^{h}_{t^{\prime}})$
This is the difference between the outputs of the model $f$ (a five-dimension
vector in CEBaB). The CEBaB ICaCE allows evaluating an explanation method $m$
by computing errors w.r.t. an intervention:

$\displaystyle\texttt{Err}(f,m,T,t,t^{\prime})=\frac{1}{|{\mathbb{D}}(T\leftarrow
t^{\prime})|}\sum_{({\bm{x}}_{t},\widetilde{{\bm{x}}}^{h}_{t^{\prime}})\in{\mathbb{D}}}{\texttt{Dist}\left(\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}^{h}_{t^{\prime}}),\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}^{m}_{t^{\prime}})\right)}$
(7)

Where Dist can be, for example, the $L2$ distance, the cosine distance
($1-\cos$), or the norm difference (see Abraham et al. (2022) for more
details). The Err scores we present in this study are the mean over 24
interventions: four concepts with three values (six interventions for each
concept).

Evaluation pipeline. For the generative approach, we generate a CF
$\widetilde{{\bm{x}}}^{m}_{t^{\prime}}$ using an LLM and calculate
$\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},\widetilde{{\bm{x}}}^{m}_{t^{\prime}})$.
For matching, given an intervention $T\leftarrow t^{\prime}$ and a set of
candidates that share the new treatment value:
${\mathbb{D}}(T=t^{\prime})=\left\\{{\bm{x}}|{\bm{x}}\in{\mathbb{D}},T({\bm{x}})=t^{\prime}\right\\}$,
we utilize a matching model (e.g., our causal representation model) to
represent each candidate with a high-dimensional continuous vector. To this
end, we extract the last layer hidden states of the candidate’s tokens and
compute their mean vector. Finally, given a query example ${\bm{x}}_{t}$, we
extract its representation, rank the candidates according to their cosine
similarity, select a match, and calculate the
$\widehat{\texttt{ICaCE}_{f}}({\bm{x}}_{t},m_{1}({\bm{x}}_{t}))$.

###  4.2 Models and baselines

In Appendix D we provide additional implementation details, including
hyparameters and prompts.

Generative approach. We employ three different CF generation techniques: Zero-
shot Generative, Few-shot Generative (with three demonstrations) and _Fine-
tune Generative_. The first two are based on the Decoder-only ChatGPT (Ouyang
et al., 2022), while the third is based on an Encoder-decoder T5-base model
(Raffel et al., 2019) trained using a parallel dataset, comprised of original
reviews and their human-written CFs (the inclusive train set of CEBaB). The
zero-shot and few-shot prompts are described in Appendix §D.1.

Matching baselines. We benchmark against six matching methods: (1) Random
Match, which randomly selects an example from the candidate set; (2)
Propensity (Benedetto et al., 2018), which first estimates the propensity
score $P(T=t^{\prime}|x)$ using a concept predictor, and then conducts
matching based on this score; (3) Approx, which randomly selects an example
from ${\mathbb{X}}_{\texttt{M}}$. Notice that ${\mathbb{X}}_{\texttt{M}}$ can
be an empty set, thus, unlike the other baselines, matching may not be
performed; (4) PT RoBERTa, which utilizes a pre-trained LM to represent texts
and finds matches using their cos similarity; (5) PT S-Transformer, same as PT
RoBERTa, expect the backbone model is a SentenceTransformer model (Reimers &
Gurevych, 2019) trained to maximize semantic textual similarity; and (6) FT
S-Transformer, a SentenceTransformer fine-tuned to predict the five-star
sentiment;
Notice that Abraham et al. (2022) found that Approx outperforms all seven
tested (non-matching) baselines (see Figure 5). Therefore we do not include
them in this study.

Interpreted models. We interpret five different models of varying sizes. The
first three are Encoder-only models fine-tuned using only the original reviews
from the train set of CEBaB to predict the five-star sentiment. These models
include DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2019), and
RoBERTa (Liu et al., 2019). The other two are the zero-shot chat versions of
the Decoder-only Llama-2 model (Touvron et al., 2023) with 7 and 13 billion
parameters. We extract the five-star sentiment distribution of the Llama
models using the prompt described in Figure 6.

##  5 Results

Table 2: Comparison between different methods and baselines. The columns
present Err scores (Eq. 7) when explaining different five-class sentiment
models. The sub-columns present three different measures: Euclidean distance
(L2), Cosine distance (Cos), and norm difference (ND). The top table presents
scores using a single match ($K=1$), and the bottom table when $K=10$. The
Generative rows are not matching methods and thus are not comparable to such
methods. Another non-comparable row is the first, which presents the
performance of our causal model with a candidate set that also includes
ground-truth (GT) CFs. Numbers are means over 24 interventions, $\downarrow$
is better.

DistilBERT BERT RoBERTa Llama2-7b Llama2-13b AVG $K=1$ L2 Cos ND L2 Cos ND L2
Cos ND L2 Cos ND L2 Cos ND Causal Model (+GT) .13 .09 .06 .14 .10 .07 .13 .09
.06 .12 .07 .06 .11 .07 .06 .09 Fine-tune Generative .38 .28 .21 .42 .28 .23
.39 .27 .21 .36 .23 .22 .31 .21 .19 .28 Few-shot Generative .42 .32 .23 .45
.29 .23 .41 .30 .21 .38 .26 .23 .33 .23 .20 .30 Zero-shot Generative .43 .35
.24 .47 .35 .25 .44 .34 .23 .40 .27 .25 .35 .26 .21 .32 Random Match .88 .64
.47 .96 .62 .49 .95 .63 .49 .84 .54 .50 .84 .55 .49 .66 Propensity .90 .68 .48
.99 .69 .56 .99 .68 .56 .88 .61 .53 .86 .58 .52 .70 Approx .74 .60 .41 .81 .58
.45 .79 .60 .44 .71 .49 .43 .70 .54 .49 .58 PT RoBERTa .75 .61 .39 .83 .59 .42
.81 .61 .41 .74 .53 .43 .73 .51 .43 .59 PT S-Transformer .75 .60 .40 .83 .60
.45 .81 .61 .43 .74 .55 .44 .69 .52 .41 .59 FT S-Transformer .72 .78 .46 .82
.79 .59 .80 .84 .56 .73 .63 .45 .68 .63 .39 .66 Causal Model .66 .55 .36 .70
.55 .39 .68 .56 .37 .64 .47 .39 .59 .45 .36 .52 DistilBERT BERT RoBERTa
Llama2-7b Llama2-13b AVG $K=10$ L2 Cos ND L2 Cos ND L2 Cos ND L2 Cos ND L2 Cos
ND Fine-tune Generative .32 .21 .20 .36 .23 .23 .34 .21 .21 .30 .18 .20 .27
.16 .18 .24 Few-shot Generative .38 .30 .22 .42 .28 .24 .38 .29 .21 .34 .24
.22 .30 .22 .19 .28 Zero-shot Generative .38 .32 .23 .42 .31 .25 .40 .31 .23
.36 .24 .23 .31 .22 .19 .29 Random Match .70 .52 .40 .78 .51 .47 .77 .51 .45
.67 .45 .40 .67 .45 .40 .54 Propensity .72 .54 .43 .80 .52 .51 .79 .52 .50 .68
.47 .42 .69 .47 .43 .57 Approx .61 .49 .37 .68 .48 .43 .66 .48 .41 .58 .40 .37
.57 .41 .36 .49 PT RoBERTa .64 .50 .36 .71 .49 .44 .70 .49 .42 .61 .44 .37 .60
.43 .36 .51 PT S-Transformer .64 .51 .38 .72 .50 .45 .71 .50 .44 .61 .45 .38
.59 .44 .36 .51 FT S-Transformer .61 .56 .43 .69 .56 .50 .67 .56 .47 .60 .48
.41 .56 .50 .36 .53 Causal Model .56 .47 .35 .63 .46 .41 .60 .45 .38 .55 .40
.36 .52 .38 .33 .46

Our main results are provided in Table 2. In Appendix §5.1 we provide a
thorough ablation study, demonstrating the effect of each of our design
choices, including, among others, the impact of variations of the candidate
set and and the causal model objective. Our three main findings are:

#### 1\. Counterfactual generation models are the SOTA model-agnostic
explainers.

The first promise of this study is that CF generation models are strong
explainers. As can be seen from Table 2, the generative models achieve much
lower errors compared to the other methods, making them the SOTA model-
agnostic explainers. In addition, the table provides two additional findings:
(1) Few-shot prompts are better than zero-shot prompts for generating CFs; and
(2) The small fine-tuned Encoder-decoder T5 is the best generative model. This
suggests that when a parallel dataset consisting of pairs of input examples
and their human-written CFs is available, then it is better to fine-tune a
small model. Moreover, the small model is computationally (and financially)
less expensive than LLMs (but much slower than matching methods).
Nevertheless, consider that collecting a parallel dataset for fine-tuning is
labor intensive.

#### 2\. The causal representation model is the best-performing matching
method.

Table 2 also sheds light on the performance of our novel representation method
in comparison to various explainers and matching baselines. Notably, our
causal model consistently outperforms all the matching baselines, achieving
substantially lower errors across five explained models. Noteworthy, Abraham
et al. (2022) proposed the Approx method (the only matching method they
examined) and found it outperforms common NLP explainers. Our results show
that all matching baselines (including Approx, excluding Random Match and
Propensity) are competitive and perform similarly. Nevertheless, they fall
short of our causal representation model.

Although the generative approach exhibits better scores than the matching
approach, it is important to note that the quality of the
$\widehat{\texttt{ICaCE}_{f}}$ estimation of any matching method is highly
dependent on the matching set. Therefore, we also include the Err scores for
our causal model when the candidate set contains the ground-truth human-
written CFs (see the first row in Table 2). In that case, our causal model
outperforms even the generative models. This suggests that our method is
potentially the SOTA explainer (although this is impractical since access to
CFs is a strong assumption).

Finally, Figure 5, which plots the Err score when selecting the $k$th match,
$m_{k}(x)$, reveals three desirable properties of our causal model: (1) The
lowest Err score belongs to the first match, indicating that, on average, our
causal model indeed selects the best match; (2) A monotonically increasing
pattern (i.e., a larger $k$ results in a higher Err) demonstrates a strong
association between representation similarity and explainability capacity: The
more similar a match is, the more accurate the $\widehat{\texttt{ICaCE}_{f}}$
estimator becomes; and (3) The top-ranked matches selected by our causal model
outperform other baselines, achieving notably lower Err scores.

Figure 2: The L2 Err score for explaining the DistilBERT model (Y-axis) as a
function of selecting the $k$th match, $m_{k}(x)$. For Figures 5 and 3, we
present average Err scores only for interventions with a candidate set of size
larger than 250, making the numbers differ from Table 2.

Figure 3: The L2 Err score for explaining the DistilBERT model (Y-axis) as a
function of Top-$K$ matching. The figure illustrates the beneficial impact of
considering multiple matches. However, beyond a certain point, adding more
matches negatively affects causal estimation.

#### 3\. Top-$K$ matching improves the explanation capabilities of every
method.

When comparing the top rows of Table 2 ($k=1$) to the bottom rows ($k=10$), it
is easy to observe that Top-$K$ matching lowers the Err of every examined
explanation method. This observation also holds for the generative models,
which accordingly generate multiple CFs via sampling. The definitive results,
combined with the simplicity and cost-effectiveness of the Top-$K$ extension
(especially for matching methods, see Appendix §B), suggest that top-$K$
should be a standard for explainers. Given this, the question that arises is
what should the value of K be?

In Figure 3, we plot the L2 Err as a function of $k$ for our method and three
other baselines when explaining DistilBERT (the trends are similar for the
other metrics and explained models). The prominent ‘✓’ shape curve of the
causal model is desirable and indicates that the model selects good matches at
first. Then, around $k=20$, the error begins to surge and only after $k=100$
that it matches the error at $k=1$. Conversely, other baselines rank
candidates less effectively and exhibit a more gradual increase after their
minimum. Nonetheless, the Top-$K$ approach has a beneficial effect for all
methods and a broad range of values can be picked before the effect fades. We
arbitrarily opted for $k=10$ in this study, even though $k=20$ would have been
more advantageous for our causal model. A future direction could explore how
to tune the $k$ parameter for each individual example.

###  5.1 Ablation study

Table 3: Ablation study. Row 1: Our model. Row 2: With a different backbone.
Row 3: Causal S-Transformer (ours) without filtering misspecified CFs from
${\mathbb{X}}_{\texttt{CF}}$. Rows 4-9: Different variations of the objective
(Eq. 6) when discarding components that include the specified sets. Columns
present different matching candidate sets: original, original with ground-
truth CFs, and original with misspecified CFs. Numbers are the mean over 5
explained models and 24 interventions. $\downarrow$ is better.

Original \+ GT CFs \+ Miss. CFs L2 Cos norm L2 Cos norm L2 Cos norm 1 Causal
S-Trans. .65 .52 .38 .13 .09 .06 .70 .58 .46 2 Causal Roberta .66 .52 .37 .16
.11 .08 .70 .59 .46 3 w/o filtering .66 .51 .38 .12 .08 .05 .77 .63 .52 4 w/o
${\mathbb{X}}_{\texttt{CF}}$ .67 .54 .39 .26 .18 .13 .69 .58 .45 5 w/o
${\mathbb{X}}_{\texttt{M}}$ .66 .51 .37 .09 .06 .04 .81 .68 .57 6 w/o
${\mathbb{X}}_{\texttt{¬CF}}$ .66 .51 .38 .06 .04 .03 .83 .68 .60 7 w/o
${\mathbb{X}}_{\texttt{¬M}}$ .66 .51 .38 .16 .11 .07 .70 .59 .46 8 w/o
${\mathbb{X}}_{\texttt{M}}\cup{\mathbb{X}}_{\texttt{¬M}}$ .66 .53 .39 .19 .13
.09 .74 .61 .50 9 w/o
${\mathbb{X}}_{\texttt{CF}}\cup{\mathbb{X}}_{\texttt{¬CF}}$ .66 .52 .38 .15
.09 .07 .72 .60 .49

Figure 4: Average similarity between query examples and examples from the four
sets (the bars), plotted for different variants of the causal model (Y-axis).

This subsection aims to shed more light on the effect of each of our design
choices. To this end, we examine variations or omissions of components from
our causal representation model. Noteworthy, with the original candidate set,
which is relatively small, all of the ablation models are competitive, and the
performance difference is insignificant. Therefore, we also experiment with
variations of the candidate set that reveal the pitfalls of each ablation
model. Table 3 presents the performance when using three candidate sets: (1)
The original candidate set, containing only original reviews from the matching
split; (2) The original set augmented by the ground-truth human-written CFs;
and (3) The original set augmented by the misspecified model-generated CFs.

We start by examining the backbone encoder model of the causal model:
SentenceTransformer and RoBERTa. Comparing the first two rows of Table 3, we
observe that both encoders are competitive and can be utilized as the
backbone. We decided to use SentenceTransformer because its development loss
was lower. By comparing the first row to the third, we examine the impact
filtering unsuccessful or misspecified CFs from ${\mathbb{X}}_{\texttt{CF}}$.
Without filtering, the causal model prioritizes misspecified CFs above valid
matches (see Figure 4), and as a result, when misspecified CFs are present in
the candidate set, the performance degrades (third row, right column, gray
cells).

We next examine the impact of discarding components from the model objective
Eq 6. To this end, for each set of examples, ${\mathbb{X}}_{\texttt{CF}}$,
${\mathbb{X}}_{\texttt{M}}$, ${\mathbb{X}}_{\texttt{¬CF}}$, and
${\mathbb{X}}_{\texttt{¬M}}$, we train an encoder model while ignoring the
three components from the objective that utilize examples from the discarded
set. In addition, we also train two encoder models using only model-generated
CFs (w/o ${\mathbb{X}}_{\texttt{M}}$ and ${\mathbb{X}}_{\texttt{¬M}}$), and
without any (w/o ${\mathbb{X}}_{\texttt{CF}}$ and
${\mathbb{X}}_{\texttt{¬CF}}$, i.e., training only with human-written reviews
– not CFs).

The behavior of each ablation model is as one would expect. For example, when
${\mathbb{X}}_{\texttt{CF}}$ is discarded from the objective (row 4), the
ablation model struggles to identify CFs and the performance declines when the
matching sets include ground-truth CFs. Conversely, when
${\mathbb{X}}_{\texttt{¬CF}}$ or ${\mathbb{X}}_{\texttt{M}}$ are excluded
(rows 6, 7), the model fails to distinguish between CFs and misspecified ones,
often favoring the latter over valid matches. Consequently, the performance
drops when the candidate set contains misspecified CFs. Only our causal model
consistently performs well across both candidate sets. Notably, we can see
that even without model-generated CFs (row 9), the causal model performs well.
Although the causal estimation is less precise, achieving a good explainer
without utilizing an LLM is possible.

Finally, Figure 4 presents the average similarity between query examples and
the four sets for the causal and selected ablative models. As can be seen, our
causal model learns the desirable ranking order:
${\bm{x}}_{\texttt{¬M}}\preceq{\bm{x}}_{\texttt{¬CF}}\preceq{\bm{x}}_{\texttt{M}}\preceq{\bm{x}}_{\texttt{CF}}$.
In contrast, the model learns to favor misspecified CFs over matches when
discarding ${\mathbb{X}}_{\texttt{M}}$ or ${\mathbb{X}}_{\texttt{¬CF}}$ or
when not performing filtering of misspecified CFs from
${\mathbb{X}}_{\texttt{CF}}$. Although these ablation models have a
competitive performance to our model and some outperform it when the candidate
set contains ground-truth CFs, they fail when the set contains misspecified
CFs, highlighting the importance of learning the right ranking order.

##  6 New setup: explaining stance detection models

Table 4: Examples from the new Stance Detection setup. In the first stage of
the dataset construction (top table), we split the original tweets into the
train, dev, matching, and test sets. In the second stage (middle table), we
keep 30% of the original tweets, and for the remaining 70%, we randomly assign
values for the three writer’s concepts. In this example, we ask GPT-4 to edit
the tweet “to make it sound as if a teenage female wrote it”. In the third
(bottom table), we generate CFs for the test set (and are used for
benchmarking). In this example, we ask GPT-4 to change the Job concept to
Farmer. S1: Original tweet. I need feminism because ”what were you wearing”
shouldn’t be a question when I tell my stepmom I was cat called. #SemST
---
$Subject:\texttt{Feminism}\hfill Age:\texttt{?}\hfill Gender:\texttt{?}\hfill
Job:\texttt{?}\hfill Label:\texttt{Favor}$
S2: Modified tweet. OMG, like, I totally need feminism cuz ’what were you
wearing?’ shouldn’t even be a thing when I tell my stepmom some guy shouted
stuff at me. #JustSaying
$Subject:\texttt{Feminism}\hfill Age\rightarrow\texttt{Teen}\hfill
Gender\rightarrow\texttt{\female}\hfill Job:\texttt{?}\hfill
Label:\texttt{Favor}$
S3: CF (test set). Gosh, I reckon I’d holler for feminism, cause ’what were ya
wearin?’ ought not be a question when I tell ma about some fella yelled stuff
at me while I was out in the fields. #JustSaying
$Subject:\texttt{Feminism}\hfill Age:\texttt{Teen}\hfill
Gender:\texttt{\female}\hfill Job\rightarrow\texttt{Farmer}\hfill
Label:\texttt{Favor}$

While we aim to validate our earlier conclusions across multiple benchmarks,
CEBaB remains the sole high-quality benchmark for NLP explainers. On the other
hand, we show that LLMs generate CFs that resemble human-written CFs.
Consequently, we harness LLM capabilities to emulate CEBaB and propose a new
benchmark for model explanation, eliminating the high cost of manually
annotating such benchmarks. We decided to focus on Stance Detection, which is
an important and well-studied NLP task that determines the position (support,
oppose, neutral) expressed by the writer towards a particular topic (e.g.,
abortions, climate change) (Küçük & Can, 2021; Hardalov et al., 2022; Viehmann
et al., 2023). This task is intriguing because understanding public opinions
and beliefs can guide decision-making, policy formulation, and marketing
strategies. Estimating the causal effect of high-level concepts like gender or
age may shed light on the societal biases and perceptions that drive social
trends.

We rely on the stance detection dataset from the SemEval 2016 shared task
(Mohammad et al., 2016). This dataset contains tweets about five subjects:
Abortion, atheism, climate change, feminism, and Hillary Clinton. Humans
annotated the stance of the tweets with three possible labels: support,
oppose, and neutral. For simplicity of the new setup, we embrace the same
causal graph of CEBaB (see Figure 3.1), but replace the four review concepts
with new relevant concepts: (1) Tweet’s subject (specified above); (2)
Writer’s age - teenage, elder or unknown; (3) Writer’s gender - female, male
or unknown; and (4) Writer’s Job - farmer, professor or unknown. The four
concepts have an exogenous variable $U$ as their parent and are the direct
cause of the text $X$ and the label (stance) $Y$, which is naturally affected
by the four concepts. The explanation methods should estimate the effect of 9
interventions: changing the value of the three writer’s concepts (age, gender,
job) to three values (we do not explain the tweet’s subject concept).

Constructing the new setup. We follow three steps for constructing the new
benchmark. See the examples in Table 4. In the first stage, we randomly split
the data into four sets: train ($N=1000$), matching ($N=2250$), dev ($N=250$),
and test ($N=500$). Each subject is precisely one-fifth of each split. For
each set and subject, we keep 30% of the original tweet and assign an
‘unknown’ value to the writer’s concepts. For the remaining 70%, we randomly
sample two of the three writer’s concepts and assign new values (e.g., male,
unknown age, professor). Noteworthy, we do not uniformly sample the writer’s
profile since we want to introduce correlations between the concepts and the
label. Table 7 presents the joint probabilities of different concept and label
values.

In the second stage, we utilize GPT-4 to generate new tweets according to the
new value assignment of 70% of the tweets. The prompts we used are described
in Figure 9. In the third and final stage, we utilize GPT-4 to generate the
ground-truth CFs for the test set, which enables calculating the Err for
evaluating explanation methods. Accordingly, for each test example, we prompt
GPT-4 and generate a single CF for each of the six possible interventions (3
writer’s concepts with two possible values that are different from the
assignment). The prompt is provided in Figure 10. In addition, we ask GPT-4 to
predict the new label of the tweet (after the edit) and add this information
to the dataset.

Table 5: Results for the new stance detection setup we introduce in an out-of-
distribution scenario (for each test example, the candidate set compromise
texts with different subjects). We explain two stance detection models: New \-
RoBERTa fine-tuned on new stance labels extracted by GPT-4; and Original \-
DistilBERT fine-tuned with the stance labels of the original tweets. The
description of the rows is given in the caption of Table 2. Numbers are the
mean over 18 interventions, $\downarrow$ is better.

$K=1$ $K=10$ New Labels Original New Labels Original AVG L2 Cos ND L2 Cos ND
L2 Cos ND L2 Cos ND Causal Model (+GT) .08 .16 .06 .10 .13 .08 - - - - - -
0.10 Zero-shot Generative .22 .74 .16 .36 .65 .30 .19 .68 .15 .34 .61 .30 .39
Random Match .57 .85 .45 .85 .77 .71 .46 .82 .36 .76 .73 .65 .66 Propensity
.55 .87 .44 .83 .80 .70 .44 .83 .34 .74 .74 .64 .66 Approx .54 .87 .43 .73 .75
.60 .43 .82 .34 .65 .72 .56 .66 PT RoBERTa .47 .89 .36 .76 .75 .63 .41 .84 .31
.70 .73 .60 .62 PT S-Transformer .47 .86 .36 .77 .75 .64 .40 .81 .30 .73 .73
.63 .62 Causal Model .46 .84 .36 .71 .74 .60 .40 .80 .30 .64 .71 .58 .59

Experimental details. We interpret two stance detection models. The first one
is a RoBERTa fine-tuned on the new stance labels extracted by GPT-4. The
second model is DistilBERT fine-tuned on the stance labels of the original
tweets. Noteworthy, the causal effect of the writer’s concepts is more
pronounced in the second model. This arises because we designed the dataset to
have a large correlation between the original labels and the concepts, which
the fine-tuned model captures. In contrast, when fine-tuning with the new
labels produced by GPT-4, this causal effect becomes weak. This is mainly
because GPT-4 tends to predict the neutral label far more frequently than its
representation within the original labels (64.8% vs. 25.5%, refer to Table 7),
thereby diminishing the correlation we initially established.

In contrast to CEBaB, our new setup enables testing of our causal matching
method in an out-of-distribution scenario. A prevalent approach in the stance
detection literature is assessing model robustness against distribution shifts
arising from changes in the subject of the texts. For instance, a model might
be trained on texts discussing abortion and then tested on texts about climate
change (Hardalov et al., 2021; Ben-David et al., 2022; Deng et al., 2022). In
our context, such a distribution shift might happen when the subject of the
test samples is distinct from the subjects within the matching candidate set.
Consequently, we conduct experiments to gauge the performance of our method
under circumstances where the distribution shift is evident (i.e., for each
test example, the matching set compromises only candidates with different
subjects).

For the CF generation approach, we employ ChatGPT (and not GPT-4, which
generates the ground-truth CFs of the test set) and only with zero-shot
prompts. We benchmark our causal representation model with the same matching
baselines from §4.2 with the same hyperparameters and training procedure
described in §D.

Results. We report the results on Table 5. As can be seen, we reproduce our
main findings from the previous section on the new benchmark, confirming the
robustness of our conclusions, even in out-of-distribution settings.
Specifically, the generative approach outperforms the matching methods
(although consider that in this setup, the ground-truth CFs are also model-
generated). In addition, the causal representation model surpasses the other
matching baseline and outperforms the generative model when the candidate set
contains ground-truth CFs. Finally, the Top-$K$ technique universally improves
all the explainers.

##  7 Discussion

In this study, we explored the terrain of explaining the impact of high-level
concepts on the predictions of NLP models. We focused on model-agnostic
techniques, which do not require access to the explained model during the
training time and hence can explain numerous models, a property that is
crucial for model selection, debugging, and deployment of safer, transparent,
and accountable systems. We provided a theoretical framework linking
faithfulness to causality, showing that CF approximation explanation methods
are always order-faithful. We further motivate the utilization of LLMs for
generating CFs, either as part of the explanation or for learning an embedding
space that enables effective matching. We hope our theoretical and empirical
results might broadly impact the field, and suggest these four following
directions:

Explanation methods should be causal-inspired. Throughout this paper, our
discussions, and theoretical and empirical findings collectively underscore a
pivotal message: true interpretability is inextricably linked with causality.
As NLP systems increasingly weave themselves into the fabric of our daily
lives, influencing critical decision-making processes in sectors like law,
healthcare, politics, and education, the imperative for faithful explanations
has never been greater. This realization calls for a paradigm shift towards
exploring and developing more causal-inspired explanation methods.

Understanding when LLM-generated counterfactuals fail. Our empirical results
demonstrate that the best method for explaining the causal effect of high-
level concepts on model prediction is by employing multiple (Top-$K$) LLM-
generated CFs. However, this does not mean that the problem of
interpretability has been solved. There exists an impending need for the
community to design more challenging benchmarks capable of identifying the
areas where the LLMs fall short, misunderstand causality, and fail to provide
correct CF explanations (for example, see Kiciman et al. (2023)). Moreover, as
we emphasize throughout the paper, utilizing an LLM at inference time is
frequently impractical because of its prolonged latency, high financial costs,
or privacy concerns, which leads us to the next point.

Closing the gap of efficient explanation methods. As the pursuit of
understanding model behavior intensifies, it is paramount that we do not trade
off efficiency for quality. The noticeable performance gap between the two
approaches we introduced in the paper: the generative approach and the
matching approach, underscores a ripe avenue for research. For example, the
candidate set is crucial for the success of the matching approach, and given
the right examples within this set, it has the potential to surpass its
generative counterpart. Therefore, an interesting line of work could explore
techniques for enriching the candidate set. Bridging this gap could lead to
methods that encapsulate the best of both worlds: high quality and operational
efficiency.

Constructing new CF-based benchmarks. As we shift to causal-inspired
explanation methods, how we evaluate them must evolve in tandem. Accordingly,
it is imperative to benchmark explanations against the true causal effect.
High-quality CFs, which can serve as a proxy for ground truth in this context,
are the basis for such evaluation, as exemplified by the CEBaB benchmark
(Abraham et al., 2022). Nevertheless, the traditional approach for
constructing such benchmarks relies on human experts and is both financially
costly, labor-intensive and fraught with inherent difficulties due to the
cognitive demands of the task. Our work demonstrates that LLMs can effectively
facilitate this process. We encourage the community to craft benchmarks that
resonate with real-world scenarios. These LLM-guided benchmarks can serve as
catalyzers for appraising efficient methods as we work towards bridging the
existing performance gap.

#### Ethics statement.

No human subjects were directly involved in our research. While explanation
methods offer deeper transparency of NLP models, researchers and practitioners
should remain vigilant and mindful. These methods, if misused, could over-
emphasize or amplify biases present in the model or data. Our novel dataset,
derived from tweets taken from SemEval16 (Mohammad et al., 2016), could
inherently contain societal biases, including, but not limited to, social
prejudices or racism. We acknowledge that tweets can sometimes reflect or
amplify societal sentiments, both positive and negative. Furthermore, by
leveraging GPT-4 to edit these tweets (e.g., changing the gender of the
writer), we recognize the potential for introducing additional biases.

#### Reproducibility.

We have made comprehensive efforts to ensure that researchers can effectively
reproduce the results presented in our paper: (1) Theoretical Foundations: All
pertinent theoretical results, encompassing definitions, assumptions, and
proofs, are comprehensively detailed in Appendix A. (2) Prompts: The specific
prompts we employed for generating examples and counterfactuals are laid out
in Appendix D.1. (3) Code: The code utilized in our study is accessible as an
attachment to our paper, bundled within a ZIP file. The hyperparameters we
used are described in D. (4) Future Code Documentation: We recognize the
importance of a structured and comprehensively documented codebase. To this
end, we are preparing an organized and well-documented GitHub repository. A
link to this repository will be accessible in the foreseeable future. (5) Data
and Model Access: We plan to upload our datasets to the HuggingFace hub,
including the new dataset we curated and all the model-generated
counterfactuals. In addition, the models we interpreted within the paper will
also be hosted.

## References

  * Abraham et al. (2022) Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu.  Cebab: Estimating the causal effects of real-world concepts on NLP model behavior.  In _NeurIPS_ , 2022.  URL http://papers.nips.cc/paper_files/paper/2022/hash/701ec28790b29a5bc33832b7bdc4c3b6-Abstract-Conference.html.
  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pp. 9525–9536, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html.
  * Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané.  Concrete Problems in AI Safety.  _arXiv:1606.06565 [cs]_ , 2016.  URL http://arxiv.org/abs/1606.06565.
  * Antoniak & Mimno (2021) Maria Antoniak and David Mimno.  Bad seeds: Evaluating lexical methods for bias measurement.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1889–1904, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.148.  URL https://aclanthology.org/2021.acl-long.148.
  * Balke & Pearl (1994) Alexander Balke and Judea Pearl.  Probabilistic evaluation of counterfactual queries.  In Barbara Hayes-Roth and Richard E. Korf (eds.), _Proceedings of the 12th National Conference on Artificial Intelligence, Seattle, WA, USA, July 31 - August 4, 1994, Volume 1_ , pp. 230–237. AAAI Press / The MIT Press, 1994.  URL http://www.aaai.org/Library/AAAI/1994/aaai94-035.php.
  * Ben-David et al. (2022) Eyal Ben-David, Nadav Oved, and Roi Reichart.  PADA: example-based prompt learning for on-the-fly adaptation to unseen domains.  _Trans. Assoc. Comput. Linguistics_ , 10:414–433, 2022\.  doi: 10.1162/tacl“˙a“˙00468.  URL https://doi.org/10.1162/tacl_a_00468.
  * Benedetto et al. (2018) Umberto Benedetto, Stuart J Head, Gianni D Angelini, and Eugene H Blackstone.  Statistical primer: propensity score matching and its alternatives.  _European Journal of Cardio-Thoracic Surgery_ , 53(6):1112–1117, 2018.
  * Binder et al. (2016) Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In _International Conference on Artificial Neural Networks_ , pp. 63–71. Springer, 2016.
  * Calderon et al. (2022) Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart.  Docogen: Domain counterfactual generation for low resource domain adaptation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7727–7746, 2022\.
  * Calderon et al. (2023) Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, and Amir Kantor.  A systematic study of knowledge distillation for natural language generation with pseudo-target training.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pp. 14632–14659. Association for Computational Linguistics, 2023.  doi: 10.18653/v1/2023.acl-long.818.  URL https://doi.org/10.18653/v1/2023.acl-long.818.
  * Cinelli et al. (2022) Carlos Cinelli, Andrew Forney, and Judea Pearl.  A crash course in good and bad controls.  _Sociological Methods & Research_, pp. 00491241221099552, 2022\.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning.  What does BERT look at? An analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 276–286, Florence, Italy, August 2019. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W19-4828.
  * Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2126–2136, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1198.  URL https://www.aclweb.org/anthology/P18-1198.
  * Dasgupta et al. (2022) Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz.  Framework for evaluating faithfulness of local explanations.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 4794–4815. PMLR, 2022.  URL https://proceedings.mlr.press/v162/dasgupta22a.html.
  * Deng et al. (2022) Ruofan Deng, Li Pan, and Chloé Clavel.  Domain adaptation for stance detection towards unseen target on social media.  In _10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022, Nara, Japan, October 18-21, 2022_ , pp. 1–8. IEEE, 2022.  doi: 10.1109/ACII55700.2022.9953818.  URL https://doi.org/10.1109/ACII55700.2022.9953818.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pp. 4171–4186. Association for Computational Linguistics, 2019\.  doi: 10.18653/v1/n19-1423.  URL https://doi.org/10.18653/v1/n19-1423.
  * Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg.  Amnesic probing: Behavioral explanation with amnesic counterfactuals.  _Transactions of the Association for Computational Linguistics_ , 9:160–175, 2021.
  * Feder et al. (2021a) Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart.  Causalm: Causal model explanation through counterfactual language models.  _Computational Linguistics_ , 47(2):333–386, 2021a.
  * Feder et al. (2021b) Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart.  CausaLM: Causal model explanation through counterfactual language models.  _Computational Linguistics_ , 47(2):333–386, June 2021b.  doi: 10.1162/coli˙a˙00404.  URL https://aclanthology.org/2021.cl-2.13.
  * Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov.  Causal analysis of syntactic agreement mechanisms in neural language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1828–1843, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.144.  URL https://aclanthology.org/2021.acl-long.144.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1307–1323, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.117.  URL https://aclanthology.org/2020.findings-emnlp.117.
  * Garg et al. (2019) Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel.  Counterfactual fairness in text classification through robustness.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 219–226, 2019.
  * Gat et al. (2022) Itai Gat, Nitay Calderon, Roi Reichart, and Tamir Hazan.  A functional information perspective on model interpretation.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 7266–7278. PMLR, 2022.  URL https://proceedings.mlr.press/v162/gat22a.html.
  * Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts.  Neural natural language inference models partially embed theories of lexical entailment and negation.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pp. 163–173, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.blackboxnlp-1.16.  URL https://aclanthology.org/2020.blackboxnlp-1.16.
  * Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts.  Causal abstractions of neural networks.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Ghorbani et al. (2019) Amirata Ghorbani, Abubakar Abid, and James Y. Zou.  Interpretation of neural networks is fragile.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pp. 3681–3688. AAAI Press, 2019\.  doi: 10.1609/aaai.v33i01.33013681.  URL https://doi.org/10.1609/aaai.v33i01.33013681.
  * Goodman & Flaxman (2017) Bryce Goodman and Seth R. Flaxman.  European union regulations on algorithmic decision-making and a ”right to explanation”.  _AI Mag._ , 38(3):50–57, 2017.  doi: 10.1609/aimag.v38i3.2741.  URL https://doi.org/10.1609/aimag.v38i3.2741.
  * Goyal et al. (2019) Yash Goyal, Uri Shalit, and Been Kim.  Explaining classifiers with causal concept effect (cace).  _CoRR_ , abs/1907.07165, 2019.  URL http://arxiv.org/abs/1907.07165.
  * Guidotti et al. (2019) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  _ACM Comput. Surv._ , 51(5):93:1–93:42, 2019\.  doi: 10.1145/3236009.  URL https://doi.org/10.1145/3236009.
  * Hardalov et al. (2021) Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein.  Cross-domain label-adaptive stance detection.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pp. 9011–9028. Association for Computational Linguistics, 2021.  doi: 10.18653/v1/2021.emnlp-main.710.  URL https://doi.org/10.18653/v1/2021.emnlp-main.710.
  * Hardalov et al. (2022) Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein.  A survey on stance detection for mis- and disinformation identification.  In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pp. 1259–1277. Association for Computational Linguistics, 2022\.  doi: 10.18653/v1/2022.findings-naacl.94.  URL https://doi.org/10.18653/v1/2022.findings-naacl.94.
  * Jacovi & Goldberg (2020) Alon Jacovi and Yoav Goldberg.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4198–4205, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.386.  URL https://aclanthology.org/2020.acl-main.386.
  * Jain & Wallace (2019) Sarthak Jain and Byron C Wallace.  Attention is not explanation.  _arXiv preprint arXiv:1902.10186_ , 2019.
  * Jha et al. (2020) Rohan Jha, Charles Lovering, and Ellie Pavlick.  Does data augmentation improve generalization in nlp?  _arXiv preprint arXiv:2004.15012_ , 2020.
  * Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton.  Learning the difference that makes a difference with counterfactually-augmented data.  _arXiv preprint arXiv:1909.12434_ , 2019.
  * Kaushik et al. (2020a) Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton.  Learning the Difference that Makes a Difference with Counterfactually-Augmented Data.  _arXiv:1909.12434 [cs, stat]_ , February 2020a.  URL http://arxiv.org/abs/1909.12434.  arXiv: 1909.12434.
  * Kaushik et al. (2020b) Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton.  Explaining the efficacy of counterfactually-augmented data.  _arXiv preprint arXiv:2010.02114_ , 2020b.
  * Kiciman et al. (2023) Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan.  Causal reasoning and large language models: Opening a new frontier for causality.  _CoRR_ , abs/2305.00050, 2023.  doi: 10.48550/arXiv.2305.00050.  URL https://doi.org/10.48550/arXiv.2305.00050.
  * Kindermans et al. (2018) Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne.  Learning how to explain neural networks: Patternnet and patternattribution.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=Hkn7CBaTW.
  * Küçük & Can (2021) Dilek Küçük and Fazli Can.  Stance detection: A survey.  _ACM Comput. Surv._ , 53(1):12:1–12:37, 2021\.  doi: 10.1145/3369026.  URL https://doi.org/10.1145/3369026.
  * Künzel et al. (2019) Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu.  Metalearners for estimating heterogeneous treatment effects using machine learning.  _Proceedings of the national academy of sciences_ , 116(10):4156–4165, 2019.
  * Li et al. (2020) Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, and Shuming Shi.  Evaluating explanation methods for neural machine translation.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pp. 365–375. Association for Computational Linguistics, 2020.  doi: 10.18653/v1/2020.acl-main.35.  URL https://doi.org/10.18653/v1/2020.acl-main.35.
  * Liu et al. (2022) Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang.  Rethinking attention-model explainability through faithfulness violation test.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 13807–13824. PMLR, 2022.  URL https://proceedings.mlr.press/v162/liu22i.html.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019.  URL http://arxiv.org/abs/1907.11692.
  * Lundberg & Lee (2017) Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, pp. 4768–4777, Red Hook, NY, USA, December 2017. Curran Associates Inc.  ISBN 978-1-5108-6096-4.
  * Lyu et al. (2022) Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch.  Towards faithful model explanation in NLP: A survey.  _CoRR_ , abs/2209.11326, 2022.  doi: 10.48550/arXiv.2209.11326.  URL https://doi.org/10.48550/arXiv.2209.11326.
  * Manning et al. (2020) Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy.  Emergent linguistic structure in artificial neural networks trained by self-supervision.  _Proceedings of the National Academy of Sciences_ , 117(48):30046–30054, 2020.  ISSN 0027-8424.  doi: 10.1073/pnas.1907367117.  URL https://www.pnas.org/content/117/48/30046.
  * Mao et al. (2021) Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick.  Generative interventions for causal learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3947–3956, 2021.
  * Mohammad et al. (2016) Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry.  Semeval-2016 task 6: Detecting stance in tweets.  In Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (eds.), _Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_ , pp. 31–41. The Association for Computer Linguistics, 2016.  doi: 10.18653/v1/s16-1003.  URL https://doi.org/10.18653/v1/s16-1003.
  * Molnar (2020) Christoph Molnar.  _Interpretable machine learning_.  Lulu. com, 2020.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In _NeurIPS_ , 2022.  URL http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Pruthi et al. (2022) Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen.  Evaluating explanations: How much do explanations from the teacher aid students?  _Trans. Assoc. Comput. Linguistics_ , 10:359–375, 2022\.  doi: 10.1162/tacl“˙a“˙00465.  URL https://doi.org/10.1162/tacl_a_00465.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683, 2019.  URL http://arxiv.org/abs/1910.10683.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pp. 3980–3990. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/D19-1410.  URL https://doi.org/10.18653/v1/D19-1410.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 1135–1144, San Francisco California USA, August 2016. ACM.  ISBN 978-1-4503-4232-2.  doi: 10.1145/2939672.2939778.  URL https://dl.acm.org/doi/10.1145/2939672.2939778.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4902–4912, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.442.  URL https://www.aclweb.org/anthology/2020.acl-main.442.
  * Riley et al. (2020) Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh.  Textsettr: Label-free text style extraction and tunable targeted restyling.  _arXiv preprint arXiv:2010.03802_ , 2020.
  * Rosenberg et al. (2021) Daniel Rosenberg, Itai Gat, Amir Feder, and Roi Reichart.  Are vqa systems rad? measuring robustness to augmented data with focused interventions.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 61–70, 2021.
  * Ross et al. (2021) Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner.  Tailor: Generating and perturbing text with semantic controls.  _arXiv preprint arXiv:2107.07150_ , 2021.
  * Rudin (2019) Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nat. Mach. Intell._ , 1(5):206–215, 2019.  doi: 10.1038/s42256-019-0048-x.  URL https://doi.org/10.1038/s42256-019-0048-x.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108, 2019.  URL http://arxiv.org/abs/1910.01108.
  * Saphra & Lopez (2019) Naomi Saphra and Adam Lopez.  Understanding learning dynamics of language models with SVCCA.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 3257–3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1329.  URL https://www.aclweb.org/anthology/N19-1329.
  * Shalit et al. (2017) Uri Shalit, Fredrik D. Johansson, and David A. Sontag.  Estimating individual treatment effect: generalization bounds and algorithms.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 3076–3085. PMLR, 2017.  URL http://proceedings.mlr.press/v70/shalit17a.html.
  * Shekhar et al. (2017) Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi.  FOIL it! find one mismatch between image and language caption.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 255–265, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1024.  URL https://aclanthology.org/P17-1024.
  * Shpitser & Pearl (2006) Ilya Shpitser and Judea Pearl.  Identification of conditional interventional distributions.  In _22nd Conference on Uncertainty in Artificial Intelligence, UAI 2006_ , pp. 437–444, 2006.
  * Shpitser et al. (2010) Ilya Shpitser, Tyler J. VanderWeele, and James M. Robins.  On the validity of covariate adjustment for estimating causal effects.  In Peter Grünwald and Peter Spirtes (eds.), _UAI 2010, Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, July 8-11, 2010_ , pp. 527–536. AUAI Press, 2010.  URL https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=2078&proceeding_id=26.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pp. 3145–3153. PMLR, 2017.
  * Sia et al. (2023) Suzanna Sia, Anton Belyy, Amjad Almahairi, Madian Khabsa, Luke Zettlemoyer, and Lambert Mathias.  Logical satisfiability of counterfactuals for faithful explanations in NLI.  In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_ , pp. 9837–9845. AAAI Press, 2023.  doi: 10.1609/aaai.v37i8.26174.  URL https://doi.org/10.1609/aaai.v37i8.26174.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mpnet: Masked and permuted pre-training for language understanding.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html.
  * Soulos et al. (2020) Paul Soulos, R. Thomas McCoy, Tal Linzen, and Paul Smolensky.  Discovering the compositional structure of vector representations with role learning networks.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pp. 238–254, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.blackboxnlp-1.23.  URL https://aclanthology.org/2020.blackboxnlp-1.23.
  * Stuart (2010) Elizabeth A Stuart.  Matching methods for causal inference: A review and a look forward.  _Statistical science: a review journal of the Institute of Mathematical Statistics_ , 25(1):1, 2010.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, pp. 3319–3328. JMLR.org, 2017.
  * Sushil et al. (2018) Madhumita Sushil, Simon Suster, Kim Luyckx, and Walter Daelemans.  Patient representation learning and interpretable evaluation using clinical notes.  _J. Biomed. Informatics_ , 84:103–113, 2018.  doi: 10.1016/j.jbi.2018.06.016.  URL https://doi.org/10.1016/j.jbi.2018.06.016.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/P19-1452.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288, 2023.  doi: 10.48550/arXiv.2307.09288.  URL https://doi.org/10.48550/arXiv.2307.09288.
  * Vansteelandt et al. (2012) Stijn Vansteelandt, Maarten Bekaert, and Gerda Claeskens.  On model selection and model misspecification in causal inference.  _Statistical methods in medical research_ , 21(1):7–30, 2012.
  * Viehmann et al. (2023) Christina Viehmann, Tilman Beck, Marcus Maurer, Oliver Quiring, and Iryna Gurevych.  Investigating opinions on public policies in digital media: Setting up a supervised machine learning tool for stance classification.  _Communication Methods and Measures_ , 17(2):150–184, 2023.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber.  Investigating gender bias in language models using causal mediation analysis.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html.
  * Wang & Liu (2021) Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pp. 2495–2504. Computer Vision Foundation / IEEE, 2021.  doi: 10.1109/CVPR46437.2021.00252.  URL https://openaccess.thecvf.com/content/CVPR2021/html/Wang_Understanding_the_Behaviour_of_Contrastive_Loss_CVPR_2021_paper.html.
  * Wiegreffe et al. (2021) Sarah Wiegreffe, Ana Marasovic, and Noah A. Smith.  Measuring association between labels and free-text rationales.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pp. 10266–10284. Association for Computational Linguistics, 2021.  doi: 10.18653/v1/2021.emnlp-main.804.  URL https://doi.org/10.18653/v1/2021.emnlp-main.804.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld.  Polyjuice: Automated, general-purpose counterfactual generation.  _arXiv preprint arXiv:2101.00288_ , 2021.
  * Wu et al. (2023) Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts.  Causal proxy models for concept-based model explanations.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 37313–37334. PMLR, 2023.  URL https://proceedings.mlr.press/v202/wu23b.html.
  * Yin et al. (2022) Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, and Kai-Wei Chang.  On the sensitivity and stability of model interpretations in NLP.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pp. 2631–2647. Association for Computational Linguistics, 2022.  doi: 10.18653/v1/2022.acl-long.188.  URL https://doi.org/10.18653/v1/2022.acl-long.188.
  * Zeiler & Fergus (2014) Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), _Computer Vision – ECCV 2014_ , pp. 818–833, Cham, 2014. Springer International Publishing.
  * Zhou & Wu (2023) Xiaoling Zhou and Ou Wu.  Implicit counterfactual data augmentation for deep neural networks.  _arXiv preprint arXiv:2304.13431_ , 2023.
  * Zmigrod et al. (2019) Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 1651–1661, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1161.  URL https://aclanthology.org/P19-1161.

##  Appendix A A faithful explanation

In this section, we prove our main theorem from §3.1. We start by providing
definitions for explanation methods and then continue to define the order-
faithfulness property. For simplicity, assume the image of the explained
models is ${\mathbb{R}}$.

######  Definition 1 (Approximated Counterfactual).

Given a DGP ${\mathcal{G}}$, an explained model $f$, an example ${\bm{x}}$
whose treatment value is $T({\bm{x}})=t$, and an intervention $T:t\rightarrow
t^{\prime}$, we call $\tilde{{\bm{x}}}_{t^{\prime}}$ approximated
counterfactual if:

$\displaystyle
f(\tilde{{\bm{x}}}_{t^{\prime}})=\begin{cases}f({\bm{x}})&t^{\prime}=t\\\
f({\bm{x}}_{T=t^{\prime}})+\epsilon&otherwise\end{cases}$
$\displaystyle\mathbb{E}[\epsilon]=0$

Where ${\bm{x}}_{T=t^{\prime}}$ is the golden CF of ${\bm{x}}$ in
${\mathcal{G}}$ and $\epsilon$ is an approximation error.

This definition also suggests why the Top-$K$ technique improves the causal
effect estimation: Averaging the prediction of $K$ approximated CFs reduces
the variance of the approximation error, making the estimator more robust.

######  Definition 2 (Approximated Counterfactual Explanation Method).

Given a dataset ${\mathbb{D}}$ sampled from a DGP ${\mathcal{G}}$, an
explained model $f$ and an intervention $T:t\rightarrow t^{\prime}$, the
approximated counterfactual explanation method $S_{CF}$ is defined to be:

$S_{CF}(f,T,t,t^{\prime})=\frac{1}{|{\mathbb{D}}|}\sum_{x\in{\mathbb{D}}}{f(\tilde{{\bm{x}}}_{t^{\prime}})-f(\tilde{{\bm{x}}}_{t})}$

######  Definition 3 (Non-causal Explanation Method).

Given a dataset ${\mathbb{D}}$ sampled from a DGP ${\mathcal{G}}$, an
explained model $f$ and an intervention $T:t\rightarrow t^{\prime}$, let
$D[X,T,f]$ be defined to be a set that contains for each unit from
${\mathbb{D}}$ a triplet of model input, model output, and treatment
assignment:

$D[X,f,T]=\\{\left(x({\bm{u}}),f(x({\bm{u}}),T({\bm{u}}))\right)|{\bm{u}}\in{\mathbb{D}}\\}$

An explanation method is called non-causal explanation method $S_{NC}$ if it
is a function of $D[X,f,T]$:

$S_{NC}(f,T,t,t^{\prime})=h\left(D[X,f,T]\right)$

For simplicity, we notate the triplet
$\left(x({\bm{u}}),f(x({\bm{u}}),T({\bm{u}}))\right)$ with
$\left({\bm{x}},f({\bm{x}}),T({\bm{x}})\right)$. According to Def. 3, the
training data of $S_{NC}$ is $D[X,f,T]$, meaning it can be, for example, an
unbiased estimator of:
$\mathbb{E}_{{\mathcal{G}}}[f(x)|T=t^{\prime}]-\mathbb{E}_{{\mathcal{G}}}[f(x)|T=t]$.
However, it may also overlook other concepts than the treatment, such as
confounding concepts, which may hint why $S_{NC}$ potentially fails.

######  Definition 4 (Order-Faithfulness).

Given an i.i.d. dataset ${\mathbb{D}}$ sampled from a DGP ${\mathcal{G}}$, an
explained model $f$ and a pair of interventions $C_{1}:c_{1}\rightarrow
c^{\prime}_{1},C_{2}:c_{2}\rightarrow c^{\prime}_{2}$, an explanation method
$S$ is order-faithful if:

$\displaystyle\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S(f,C_{1},c_{1},c^{\prime}_{1})]>\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S(f,C_{2},c_{2},c^{\prime}_{2})]\Longleftrightarrow\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})>\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})$
(8)

######  Lemma 1.

For an explained model $f$, $S_{CF}$ is order-faithful for any DGP
${\mathcal{G}}$ and any interventions.

###### Proof.

We start by connecting the approximated CF to the do operator. From Def. 1, it
follows that the expected prediction of an approximated CF is equal to the
interventional one:

$\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}[f(\tilde{{\bm{x}}}_{t^{\prime}})]=\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}[f({\bm{x}}_{T=t^{\prime}})+\epsilon]=\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}[f({\bm{x}}_{T=t^{\prime}})]=\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}\left[f({\bm{x}})|do(T=t^{\prime})\right]$

Combining this result with the Def. 2 and the fact that
$P_{{\mathbb{D}}}=P_{{\mathcal{G}}}$, we get:

$\displaystyle\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S_{CF}(f,T,t,t^{\prime})]$
$\displaystyle=\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}\left[\frac{1}{|{\mathbb{D}}|}\sum_{x\in{\mathbb{D}}}{f(\tilde{{\bm{x}}}_{t^{\prime}})-f(\tilde{{\bm{x}}}_{t})}\right]$
$\displaystyle=\mathbb{E}_{x\sim{\mathbb{G}}}[f(\tilde{{\bm{x}}}_{t^{\prime}})-f(\tilde{{\bm{x}}}_{t})]$
$\displaystyle=\mathbb{E}_{x\sim{\mathcal{G}}}[f(\tilde{{\bm{x}}}_{t^{\prime}})]-\mathbb{E}_{x\sim{\mathcal{G}}}[f(\tilde{{\bm{x}}}_{t})]$
$\displaystyle=\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}\left[f({\bm{x}})|do(T=t^{\prime})\right]-\mathbb{E}_{{\bm{x}}\sim{\mathcal{G}}}\left[f({\bm{x}})|do(T=t)\right]$
$\displaystyle=\texttt{CaCE}_{f}(T,t,t^{\prime})$

Thus, $S_{CF}$ is an unbiased estimator of $\texttt{CaCE}_{f}$, and it holds
that:

$\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S_{CF}(f,C_{1},c_{1},c^{\prime}_{1})]-\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}}[S_{CF}(f,C_{2},c_{2},c^{\prime}_{2})]=\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})$

Meaning that $S_{CF}$ is order-faithful for any ${\mathcal{G}}$ and any
interventions. ∎

######  Theorem 1.

For an explained model $f$, (1) The approximated CF explanation method
$S_{CF}$ is order-faithful for every DGP ${\mathcal{G}}$ and pair of
interventions; (2) For every DGP ${\mathcal{G}}$, there exist a DGP
${\mathcal{G}}^{\prime}$ resulting from a small modification of
${\mathcal{G}}$, a model $f^{\prime}$, and a pair of interventions
$C_{1}:c_{1}\rightarrow c^{\prime}_{1},C_{2}:c_{2}\rightarrow c^{\prime}_{2}$
such that the non-causal explanation method $S_{NC}$ is not order-faithful.

###### Proof.

From Lemma 1 we know that $S_{CF}$ is order-faithful for any DGP and (1)
holds.

For part (2) of the theorem, let ${\mathcal{G}}$ be a DGP with at least two
concepts $C_{1},...,C_{K}$ (in case ${\mathcal{G}}$ has only one concept, we
can add concept to the DGP and continue with the proof). If $S_{NC}$ is not
order-faithful for some two interventions, set $f^{\prime}=f$ and we are done.
Therefore, we assume that $S_{NC}$ is order-faithful for all interventions in
${\mathcal{G}}$ and w.l.o.g. assume that:

  1. (a)

$C_{1}$ has no parent nodes (except exogenous variables). It is guaranteed
that at least one concept has no parents since the DGP is a DAG.

  2. (b)

The causal effect of $C_{1}$ is bigger than of $C_{2}$:
$\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})>\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})$.
Otherwise, inverse the corresponding signs of the proof.

Proof sketch of the second part of the theorem: We will construct a new DGP
${\mathcal{G}}^{\prime}$ by introducing a new unobserved confounding concept
$C_{0}$ (this is the “small modification of ${\mathcal{G}}$” from the theorem)
and a new explained model $f^{\prime}$, such that:

$\forall i\geq
1:P_{{\mathcal{G}}^{\prime}}({\bm{x}},f^{\prime}({\bm{x}}),C_{i})=P_{{\mathcal{G}}}({\bm{x}},f({\bm{x}}),C_{i})$
(9)
$\texttt{CaCE}_{f^{\prime}}(C_{1},c_{1},c^{\prime}_{1})<\texttt{CaCE}_{f^{\prime}}(C_{2},c_{2},c^{\prime}_{2})$
(10)

Notice that the first condition Eq. 9 ensures that $S_{NC}$ produces the same
explanations also in the new DGP. This is because for an intervention of a
concept $C_{i}$, $S_{NC}$ is a function of $D[X,f,C_{i}]$, and the joint
distribution from which it is sampled,
$P_{{\mathcal{G}}^{\prime}}({\bm{x}},f^{\prime}({\bm{x}}),C_{i})$, is the same
as in the original DGP ${\mathcal{G}}$.

Moreover, according to the second condition Eq. 10, for the new
${\mathcal{G}}^{\prime}$ and $f^{\prime}$, the causal effect of $C_{1}$ and
$C_{2}$ is reversed (compared to the original ${\mathcal{G}}$ and $f$).
However, $S_{NC}$ produces the same explanations as before (for
${\mathcal{G}}$ and $f$), meaning it cannot be order-faithful also for
${\mathcal{G}}^{\prime}$ and $f^{\prime}$.

We next show that such ${\mathcal{G}}^{\prime}$ and $f^{\prime}$ exist.

Construction: Let $f^{\prime}$ be a new explained model and
${\mathcal{G}}^{\prime}$ be a copy of ${\mathcal{G}}$ with a new node $C_{0}$,
and new edges $(C_{0},C_{1}),(C_{0},f^{\prime}(X))$ (i.e., $C_{0}$ is a
confounder). $C_{0}$ accepts three values: 0, 1 and 2. Let
$p=P_{\mathcal{G}}(C_{1}=c_{1})$ and
$p^{\prime}=P_{\mathcal{G}}(C_{1}=c^{\prime}_{1})$. Define the following
probabilities:

$\displaystyle P_{{\mathcal{G}}^{\prime}}(C_{0}=0)=p$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{0}=1)=p^{\prime}$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{0}=2)=1-p-p^{\prime}$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1}|C_{0}=0)=P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1}^{\prime}|C_{0}=1)=1$
$\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1}|C_{0}=2)=P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1}^{\prime}|C_{0}=2)=0$
$\displaystyle\forall c\neq c_{1},c_{1}^{\prime}:$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c|C_{0}=2)=\frac{P_{{\mathcal{G}}}(C_{1}=c)}{1-p-p^{\prime}}$

We conclude that the marginal distribution of $C_{1}$ is equal in both DGPs,
$P_{{\mathcal{G}}^{\prime}}(C_{1})=P_{{\mathcal{G}}}(C_{1})$:

$\displaystyle P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1})=$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c_{1}|C_{0}=0)P_{{\mathcal{G}}^{\prime}}(C_{0}=0)=p\cdot
1=P_{{\mathcal{G}}}(C_{1}=c_{1})$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c^{\prime}_{1})=$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c^{\prime}_{1}|C_{0}=1)P_{{\mathcal{G}}^{\prime}}(C_{0}=1)=p^{\prime}\cdot
1=P_{{\mathcal{G}}}(C_{1}=c^{\prime}_{1})$ $\displaystyle\forall c\neq
c_{1},c^{\prime}_{1}:P_{{\mathcal{G}}^{\prime}}(C_{1}=c)=$ $\displaystyle
P_{{\mathcal{G}}^{\prime}}(C_{1}=c|C_{0}=2)P_{{\mathcal{G}}^{\prime}}(C_{0}=2)$
$\displaystyle=$
$\displaystyle(1-p-p^{\prime})\cdot\frac{P_{{\mathcal{G}}}(C_{1}=c)}{1-p-p^{\prime}}=P_{{\mathcal{G}}}(C_{1}=c)$

Let $d$ be a difference in causal effects between $C_{1}$ and $C_{2}$ for $f$,
which is positive following assumption (b) from the beginning of the proof.

$d=\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})>0$

Let $f^{\prime}$ be the new explained model. $f^{\prime}$ has an oracle
access111In practice, it can be that the unobserved variable $C_{0}$ controls
for some modification $\psi({\bm{x}})$ and the path $X\rightarrow f(X)$ is
replaced in the new DGP ${\mathcal{G}}^{\prime}$ by
$X\rightarrow\psi(X)\rightarrow f^{\prime}(X)$ and $C_{0}\rightarrow\psi(X)$.
Nevertheless, $S_{NC}$ has only access to realizations of $X$ and
$f^{\prime}$, and therefore, for simplicity of the notations we assume an
oracle. to the underlying concepts and is defined to be:

$f^{\prime}({\bm{x}})=f({\bm{x}})-2d\cdot{\mathbb{I}}_{[C_{1}({\bm{x}})=c_{1}]}+2d\cdot{\mathbb{I}}_{[C_{0}({\bm{x}})=0]}$

Where ${\mathbb{I}}$ is the indicator function. If $C_{0}({\bm{x}})=0$ then
$C_{1}({\bm{x}})=c_{1}$ with probability 1 and:

$f^{\prime}({\bm{x}})=f({\bm{x}})-2d\cdot{\mathbb{I}}_{[True]}+2d\cdot{\mathbb{I}}_{[True]}=f({\bm{x}}).$

Conversely, if $C_{0}({\bm{x}})=1$ then $C_{1}({\bm{x}})=c^{\prime}_{1}$ with
probability 1 and:

$f^{\prime}({\bm{x}})=f({\bm{x}})-2d\cdot{\mathbb{I}}_{[False]}+2d\cdot{\mathbb{I}}_{[False]}=f({\bm{x}})$

Finally, if $C_{0}({\bm{x}})=2$ then $C_{1}({\bm{x}})\neq
c_{1},c^{\prime}_{1}$ and:

$f^{\prime}({\bm{x}})=f({\bm{x}})-2d\cdot{\mathbb{I}}_{[False]}+2d\cdot{\mathbb{I}}_{[False]}=f({\bm{x}})$

From the above equations, it is clear that in the DGP ${\mathcal{G}}^{\prime}$
without any interventions, for all values of $C_{0}$ given ${\bm{x}}$ we have
$f^{\prime}({\bm{x}})=f({\bm{x}})$. Meaning that:

$P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|C_{0},X)=P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|X)=P_{{\mathcal{G}}}(f(X)|X)$

We have shown that
$P_{{\mathcal{G}}^{\prime}}(C_{1})=P_{{\mathcal{G}}}(C_{1})$. Since
${\mathcal{G}}^{\prime}$ is a copy of ${\mathcal{G}}$ (except $C_{0}$), and
since neither the marginal distribution of the concepts nor the generation
process of $X$ did not change, then:

$\displaystyle\forall i=1,...,K:$ $\displaystyle
P_{{\mathcal{G}}}(C_{i})=P_{{\mathcal{G}}^{\prime}}(C_{i})$
$\displaystyle\forall i=1,...,K:$ $\displaystyle
P_{\mathcal{G}}(X|C_{i})=P_{{\mathcal{G}}^{\prime}}(X|C_{i})$

Finally, from definition of $f$, it depends only on ${\bm{x}}$, thus: $\forall
i:P_{{\mathcal{G}}}(f(X)|X,C_{i})=P_{{\mathcal{G}}}(f(X)|X)$ (the same holds
for $f^{\prime}$, and we have shown above that
$P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|C_{0},X)=P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|X)$).
Therefore:

$\displaystyle\forall i=1,...,K:$ $\displaystyle
P_{\mathcal{G}}(X,f(X),C_{i})$
$\displaystyle=P_{\mathcal{G}}(C_{i})P_{\mathcal{G}}(X|C_{i})P_{\mathcal{G}}(f(X)|X,C_{i})$
$\displaystyle=P_{\mathcal{G}}(C_{i})P_{\mathcal{G}}(X|C_{i})P_{\mathcal{G}}(f(X)|X)$
$\displaystyle=P_{{\mathcal{G}}^{\prime}}(C_{i})P_{{\mathcal{G}}^{\prime}}(X|C_{i})P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|X)$
$\displaystyle=P_{{\mathcal{G}}^{\prime}}(C_{i})P_{{\mathcal{G}}^{\prime}}(X|C_{i})P_{{\mathcal{G}}^{\prime}}(f^{\prime}(X)|X,C_{i})$
$\displaystyle=P_{{\mathcal{G}}^{\prime}}(X,f^{\prime}(X),C_{i})$

Eq. 9 holds and accordingly,
$P_{{\mathcal{G}}^{\prime}}(D[X,f,C_{i}])=P_{{\mathcal{G}}}(D[X,f,C_{i}])$,
meaning that $S_{NC}$ produces the same explanations for ${\mathcal{G}}$ and
${\mathcal{G}}^{\prime}$. Since $S_{NC}$ is order-faithful in ${\mathcal{G}}$,
and we know from our assumptions on ${\mathcal{G}}$ that
$\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})>\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})$,
in follows then that:

$\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}^{\prime}}[S(f^{\prime},C_{1},c_{1},c^{\prime}_{1})]>\mathbb{E}_{{\mathbb{D}}\sim{\mathcal{G}}^{\prime}}[S(f^{\prime},C_{2},c_{2},c^{\prime}_{2})]$

We next show that Eq. 10 holds (reversed causal effect), thus $S_{NC}$ cannot
be order-faithful in ${\mathcal{G}}^{\prime}$:

$\displaystyle\texttt{CaCE}_{f^{\prime}}(C_{1},c_{1},c^{\prime}_{1})$
$\displaystyle=\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-2d\cdot\texttt{CaCE}_{{\mathbb{I}}[C_{1}(x)=c_{1}]}(C_{1},c_{1},c^{\prime}_{1})+2d\cdot\texttt{CaCE}_{{\mathbb{I}}[C_{0}(x)=1]}$
$\displaystyle=\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-2d$
$\displaystyle=\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-(\texttt{CaCE}_{f}(C_{1},c_{1},c^{\prime}_{1})-\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2}))-d$
$\displaystyle=\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})-d<\texttt{CaCE}_{f}(C_{2},c_{2},c^{\prime}_{2})=\texttt{CaCE}_{f^{\prime}}(C_{2},c_{2},c^{\prime}_{2})$

The last equation is true as $f^{\prime}$ is not affected by $C_{2}$ in any
way except through the $f$.

∎

##  Appendix B Inference time efficiency

Table 6: Comparison between the inference time latency (in seconds) of
different methods. The top rows present counterfactual generation models, and
the middle and bottom tables for matching methods. Notice that all the
baselines described in §4.2 have the same latency as the Causal Model and thus
are not specified.

Method Backbone Explaining a single example Explaining 100 examples Top-$K=1$
Top-$K=10$ Top-$K=100$ Top-$K=1$ Top-$K=10$ Top-$K=100$ Fine-tune Generative
T5-Base 0.84 1.03 3.09 84 102.2 308.2 Zero-shot Generative ChatGPT (turbo)
2.45 2.95 4.47 245.3 295.3 447.2 Few-shot Generative ChatGPT (turbo) 2.52 2.98
4.49 252.3 298.3 449.2 250 candidates Top-$K=1$ Top-$K=10$ Top-$K=100$
Top-$K=1$ Top-$K=10$ Top-$K=100$ Approx – 0.86 0.86 0.86 1.95 1.95 1.95 Causal
Model S-Transformer 0.03 0.03 0.03 0.27 0.27 0.27 1000 candidates Top-$K=1$
Top-$K=10$ Top-$K=100$ Top-$K=1$ Top-$K=10$ Top-$K=100$ Approx – 0.86 0.86
0.86 1.95 1.95 1.95 Causal Model S-Transformer 0.03 0.03 0.03 0.27 0.27 0.27

As discussed in § 1, we seek three attributes in the explanation method: it
should be model-agnostic, effective, and efficient. In the main body of this
paper, we evaluate the effectiveness of two model-agnostic approaches:
Counterfactual generation and matching. We demonstrate the effectiveness and
precision of the generative models and our causal matching method compared to
other baselines. In this subsection we focus on inference efficiency, an
aspect that is vital for providing real-time explanations and explaining vast
amounts of data.

To compare the computational efficiency of various explanation methods, we
present the inference time latency in Table 6. This latency represents the
time (in seconds) required to compute the $\widehat{\texttt{ICaCE}_{f}}$ for
either a single example or a batch of 100 examples. We measure this latency
for Top-$K=1,10,100$ using the following methods: (1) The Approx baseline,
which employs three fine-tuned RoBERTa models. We first use these models to
predict the three confounder concepts of the query examples. Following this,
we randomly select matches that correspond to the values of each query
example; and (3) Our causal matching method, which first represents the query
examples, then computes their cosine similarity with the candidates and
finally finds the most similar matches. Notice that other matching baselines
have the same latency as our method. All the query and candidate examples are
50 tokens in length, which is also the length of the model-generated
counterfactuals. For the matching baselines, we utilized a candidate set
comprised of 1000 examples.

As can be seen in Table 2, the latency of the explanation methods we evaluated
varies dramatically between them. Interestingly, Top-$K$ does not influence
the latency of the matching methods since it merely involves an argmax
operation. Similarly, the size of the candidate set has no noticeable impact
on latency. Our method also outpaces the Approx baseline, as it utilizes fewer
models and avoids the need to filter the candidate set based on the values of
the confounder concepts.

The generative baselines are significantly slower than the matching methods,
primarily due to the autoregressive token-by-token generation process. For
instance, when explaining a single query example, the fine-tuned model is
around 30 times slower than matching methods for $K=1$ and 100 times slower
for $K=100$. In contrast, it is three times faster than zero-shot and few-shot
ChatGPT, which has an enormous number of parameters and also suffers from API
latency. For batch processing (explaining 100 queries), the benefits of
parallelism offered by GPUs make matching methods exceptionally fast. Our
method is 1000 times faster than the generative baselines.

In summary, our causal matching method consistently demonstrates superior
inference efficiency across various scenarios, making it an ideal choice for
applications requiring real-time explanations or the processing of large
datasets.

##  Appendix C Additional formulations

#### Cosine similarity.

The cosine similarity between two the representations of ${\bm{x}}$ and
${\bm{x}}^{\prime}$ is:

$\displaystyle
s(\phi({\bm{x}}),\phi({\bm{x}}^{\prime}))=\frac{\phi({\bm{x}})^{T}\phi({\bm{x}}^{\prime})}{||\phi({\bm{x}})||||\phi({\bm{x}}^{\prime})||}$
(11)

#### Causal paths.

The following three triplets (or $X$, $Y$ and $Z$) are the main patterns of a
causal graph:

  * •

Chains (or mediators): $X\rightarrow Z\rightarrow Y$. Conditioning on $Z$
blocks the flow from $X$ to $Y$.

  * •

Forks (or common causes, confounders): $X\leftarrow Z\rightarrow Y$.
Conditioning on $Z$ blocks the flow from $X$ to $Y$.

  * •

Colliders (or common effects): $X\rightarrow Z\leftarrow Y$. The flow from $X$
to $Y$ is blocked by default. However, conditioning on $Z$ opens the flow and
induces an association between $X$ and $Y$.

#### Back-door criterion (Pearl, 2009).

A set of variables ${\mathbb{Z}}$ satisfies the back-door criterion relative
to $(X,Y)$ in a directed acyclic graph ${\mathcal{G}}$ if:

  * •

No node in ${\mathbb{Z}}$ is a descendant of $X$.

  * •

$Z$ blocks every path between $X$ and $Y$ that contains an arrow into $X$.

The adjustment criterion (Shpitser et al., 2010) was later devised to handle
cases in which Z may explicitly contain descendants of $X$; however, it is
unnecessary for our causal graph described in Figure 3.1. We term the set
${\mathbb{Z}}$, which satisfies the back-door criterion as the adjustment set.

###  C.1 CEBaB - Causal Estimation-Based Benchmark (OpenTable Reviews)

#### The adjustment set of CEBaB.

In our causal graph (Figure 3.1), when estimating the causal effect of an
aspect, for example, $S$, on the model $f$, the following paths should be
taken into account ($F$ and $N$ w.l.o.g):

  * •

$S\rightarrow X\rightarrow f(X)$

  * •

$S\leftarrow U\rightarrow F\rightarrow X\rightarrow f(X)$

  * •

$S\leftarrow U\rightarrow F\rightarrow Y\leftarrow N\rightarrow X\rightarrow
f(X)$

  * •

$S\rightarrow Y\leftarrow F\rightarrow X\rightarrow f(X)$

  * •

$S\rightarrow Y\leftarrow F\leftarrow U\rightarrow N\rightarrow X\rightarrow
f(X)$

  * •

(when $Y\rightarrow X$) $S\rightarrow Y\rightarrow X\rightarrow f(X)$

  * •

(when $Y\leftarrow X$) $S\rightarrow Y\leftarrow X\rightarrow f(X)$

The adjustment set is $F,N,A$, and $Y$ must not be adjusted. Accordingly, the
set of matches ${\mathbb{X}}_{\texttt{M}}$ should contain texts with the same
aspect values (excluding the treatment) as the query example. Moreover, it
clarifies why we use the term misspecified to describe the sets
${\mathbb{X}}_{\texttt{¬M}}$ and ${\mathbb{X}}_{\texttt{¬CF}}$. This is
because they contain texts that at least one of their aspect values from the
query is different or was changed to another.

#### Increased precision when conditioning on $V$.

The exogenous variable $V$ is a direct cause of the text $X$ that mediates
between concepts (e.g., $S$) and the model prediction $f(X)$. For example, the
variable $V$ can account for the syntax, writing style, or length of $X$.
Controlling for $V$ (hypothetically, since it is not observed) will not bias
the causal effect estimation (of CaCE, and not of ICaCE, which by definition
should control for $V$) since it does not open any back-door paths from $S$ to
$Y$. On the other hand, controlling for $V$ can increase the estimation
precision of CaCE (Cinelli et al., 2022).

Consider the case where the model $f$ learns spurious correlations between $V$
and $Y$ (e.g., the sentiment of long texts tends to be negative). We use the
Approx matching technique (which controls all the adjusted variables: $F$,
$N$, and $A$ and sample a unit from the control group which shares the same
adjustment values as the query example) to calculate the
$\widehat{\texttt{CaCE}_{f}}$ (see Eq.2). Since $V$ is independent of any
other variable (except $X$), the $\widehat{\texttt{CaCE}_{f}}$ estimator is
not be biased asymptotically. However, when the queries and the candidate sets
are finite (and small), the spurious correlations $f$ learned might be
amplified by the non-representing sample. Controlling for $V$ (finding matches
that also share the same attributes, e.g., writing style) mitigates this bias
and increases the precision of the estimation.

Therefore, utilizing counterfactuals while training the causal representation
model used for matching is vital. In contrast to ${\mathbb{X}}_{\texttt{M}}$,
the set of counterfactuals ${\mathbb{X}}_{\texttt{CF}}$ should share the same
values of $V$, and the causal model learns to consider it when finding a
match.

Figure 5: A comparison between different explanation methods adapted from
Abraham et al. (2022). The best performing methods are the simple Approx
baseline and the S-learner.

##  Appendix D Implementation details

We use $\tau=0.1$, train the causal representation models for 12 epochs, and
select the model checkpoint which minimizes the objective of Eq. 6 (or its
modified version for the ablation models). Notice that when using the Err as
the selection criterion, one might be able to select a more robust checkpoint.
However, this violates the model-agnostic property, as calculating the Err
requires access to the explained model. The backbone SentenceTransformer of
our model is MPNet (all-mpnet-base-v2) (Song et al., 2020). We use a learning
rate of 5e-6 for training the causal representation models. We use a learning
rate of 1e-5 and a batch size of 16 for the fine-tuned baselines, explained
models and concept predictors. The concept predictor models are used for
filtering misspecified CFs (for constructing the ${\mathbb{X}}_{\texttt{CF}}$
and ${\mathbb{X}}_{\texttt{¬CF}}$ sets) and for the Approx and Propensity
score baselines. We fine-tune a RoBERTa model using the annotated train set
for each of the four concepts.

For generating counterfactuals, we use the OpenAI API with the following
models: gpt-3.5-turbo (referred to as ChatGPT in the paper) and gpt-4 (GPT-4).

###  D.1 Prompts and examples

Figure 6: The prompt we use for rating CEBaB reviews and extracting the five-
way sentiment distribution (of the Llamas models). We extract the next token
probabilities of 1-5 tokens. [TEXT] is a review. Figure 7: The zero-shot
prompt we use for generating CFs for CEBaB examples. [TEXT] is the review. The
instruction is changed according to the treatment (e.g., in this prompt, we
change the value of the food aspect from positive to negative. Figure 8: The
few-shot prompt we use for generating CFs for CEBaB examples. [TEXT] is a
review and [COUNTERFACTUAL] is a human-written CF. Every prompt consists of
three demonstrations where the aspect of the treatment is changed. Since there
are four aspects, only 12 human-written CFs are required. Figure 9: The prompt
we use for generating examples for the new stance detection setup. Notice -
these examples are used for enriching the dataset with more writer profiles.
[DOMAIN] is the tweet topic/subject (e.g., abortions), and [TEXT] is the
original tweet which we ask the LLM (GPT-4) to edit. The [DESCRIPTION]
describes the new profile of the writer. For example, it can be: “an elder
(60+) farmer wrote it”. In addition, we ask the LLM to predict the label of
the new tweet. Figure 10: The prompt we use for generating ground-truth CFs
for examples from the new stance detection setup. [DOMAIN] is the tweet
topic/subject (e.g., abortions), and [TEXT] is the original tweet which we ask
the LLM (GPT-4) to edit. The [DESCRIPTION] describes the old profile of the
writer (e.g., “Age: Unknown, Gender: Female, Job: professor”). The [EDIT
DESCRIPTION] describes the new profile, for example, “an elder (60+) farmer
wrote it”, or “the age, gender, or job of the writer is unknown”. In addition,
we ask the LLM to predict the label of the new tweet. Figure 11: The zero-shot
prompt we use for generating CFs for examples from the new stance detection
setup. This prompt is used for benchmarking the LLM (ChatGPT), and not for
generating instances for the test set. See Figure 10 caption for more details.
Table 7: Fractions in percentages of the variables (joint probability) in the
new setup (Stance Detection, §6). For example, according to row CC and column
T, 7.3% of the data are tweets about Climate Change (Subject concept) written
by a teenager (Age concept). There are a total of 4000 examples divided into
the train ($N=1000$), matching ($N=2250$), dev ($N=250$), and test ($N=500$)
sets. Orig. Label is the label of the original tweet before any modifications,
and Label is the label predicted by GPT-4 after modifying the tweet. Legend:
Subject – Ab (Abortions), At (Atheism), CC (Climate Change) Fe (Feminism) HC
(Hillary Clinton); Age – ? (Unknown), T (Teenager), E (Elder); Job – ?
(Unknown), ♀ (Female), ♂ (Male); Job – ? (Unknown), F (Farmer), P (Professor);
Label – ? (Neutral), X (Oppose), ✓ (Support); Orig. Label – ? (Neutral), X
(Oppose), ✓ (Support).

Subject Age Gender Job Label Orig. Label Ab At CC Fe HC ? T E ? ♀ ♂ ? F P ?
2613 ✓ ? 2613 ✓ Ab 20.0 0.0 0.0 0.0 0.0 7.2 3.6 9.1 10.1 3.8 6.2 8.6 7.2 4.1
13.4 5.2 1.4 4.9 11.6 3.5 At 0.0 20.0 0.0 0.0 0.0 7.7 5.4 7.0 9.7 4.2 6.0 9.7
6.5 3.8 13.1 5.6 1.3 4.0 12.4 3.6 CC 0.0 0.0 20.0 0.0 0.0 8.7 7.3 4.0 9.5 6.2
4.3 8.4 4.0 7.6 12.5 1.0 6.5 7.6 0.8 11.5 Fe 0.0 0.0 0.0 20.0 0.0 9.2 5.4 5.4
8.3 4.6 7.2 8.4 6.6 4.9 13.1 1.8 5.0 3.4 10.8 5.8 HC 0.0 0.0 0.0 0.0 20.0 8.2
4.5 7.3 9.4 4.1 6.5 8.0 8.4 3.6 12.7 5.3 2.0 5.5 11.2 3.3 ? 7.2 7.7 8.7 9.2
8.2 41.0 0.0 0.0 11.2 13.0 16.8 8.4 18.1 14.6 28.2 6.5 6.3 11.0 18.9 11.1 T
3.6 5.4 7.3 5.4 4.5 0.0 26.2 0.0 14.9 6.2 5.2 16.6 4.2 5.4 16.5 2.6 7.1 8.7
4.9 12.6 E 9.1 7.0 4.0 5.4 7.3 0.0 0.0 32.8 20.9 3.6 8.2 18.2 10.4 4.1 20.1
9.9 2.8 5.8 23.0 4.0 ? 10.1 9.7 9.5 8.3 9.4 11.2 14.9 20.9 47.0 0.0 0.0 11.6
21.2 14.1 30.4 9.8 6.8 11.4 23.3 12.3 ♀ 3.8 4.2 6.2 4.6 4.1 13.0 6.2 3.6 0.0
22.9 0.0 14.4 3.4 5.1 14.0 2.6 6.3 6.1 6.4 10.4 ♂ 6.2 6.0 4.3 7.2 6.5 16.8 5.2
8.2 0.0 0.0 30.1 17.2 8.2 4.8 20.6 6.5 3.1 8.0 17.2 5.0 ? 8.6 9.7 8.4 8.4 8.0
8.4 16.6 18.2 11.6 14.4 17.2 43.2 0.0 0.0 26.0 9.1 8.2 11.4 19.0 12.7 F 7.2
6.5 4.0 6.6 8.4 18.1 4.2 10.4 21.2 3.4 8.2 0.0 32.8 0.0 22.1 8.2 2.5 8.5 21.8
2.5 P 4.1 3.8 7.6 4.9 3.6 14.6 5.4 4.1 14.1 5.1 4.8 0.0 0.0 24.0 16.8 1.6 5.6
5.5 6.0 12.5 ? 13.4 13.1 12.5 13.1 12.7 28.2 16.5 20.1 30.4 14.0 20.6 26.0
22.1 16.8 64.8 0.0 0.0 24.3 27.2 13.3 2613 5.2 5.6 1.0 1.8 5.3 6.5 2.6 9.9 9.8
2.6 6.5 9.1 8.2 1.6 0.0 19.0 0.0 0.5 17.4 1.1 ✓ 1.4 1.3 6.5 5.0 2.0 6.3 7.1
2.8 6.8 6.3 3.1 8.2 2.5 5.6 0.0 0.0 16.2 0.7 2.2 13.3 ? 4.9 4.0 7.6 3.4 5.5
11.0 8.7 5.8 11.4 6.1 8.0 11.4 8.5 5.5 24.3 0.5 0.7 25.5 0.0 0.0 2613 11.6
12.4 0.8 10.8 11.2 18.9 4.9 23.0 23.3 6.4 17.2 19.0 21.8 6.0 27.2 17.4 2.2 0.0
46.8 0.0 ✓ 3.5 3.6 11.5 5.8 3.3 11.1 12.6 4.0 12.3 10.4 5.0 12.7 2.5 12.5 13.3
1.1 13.3 0.0 0.0 27.7