# Explaining Language Models’ Predictions with High-Impact Concepts

Ruochen Zhao1 Shafiq Joty1,2 Yongjie Wang1 Tan Wang1
1Nanyang Technological University, Singapore
2Salesforce AI
{ruochen002, yongjie002<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

The emergence of large-scale pretrained language models has posed
unprecedented challenges in deriving explanations of why the model has made
some predictions. Stemmed from the compositional nature of languages, spurious
correlations have further undermined the trustworthiness of NLP systems,
leading to unreliable model explanations that are merely correlated with the
output predictions. To encourage fairness and transparency, there exists an
urgent demand for reliable explanations that allow users to consistently
understand the model’s behavior. In this work, we propose a complete framework
for extending concept-based interpretability methods to NLP. Specifically, we
propose a post-hoc interpretability method for extracting predictive high-
level features (concepts) from the pretrained model’s hidden layer
activations. We optimize for features whose existence causes the output
predictions to change substantially, _i.e_. generates a high impact. Moreover,
we devise several evaluation metrics that can be universally applied.
Extensive experiments on real and synthetic tasks demonstrate that our method
achieves superior results on predictive impact, usability, and faithfulness
compared to the baselines. Our codebase is available at
https://github.com/RuochenZhao/HIConcept.

##  1 Introduction

Over the past few years, progress achieved by large language models (LLMs) has
led them to be widely applied in sensitive applications such as personalized
recommendation bots and recruitment. However, many users are still reluctant
to adopt large-scale NLP models due to their opaque decision processes
(Mathews, 2019). To increase transparency and user trust, it is crucial to
utilize Explainable AI (XAI) to derive effective model interpretations.
Following interpretability definitions proposed by Miller (2019) and Kim et
al. (2016), we hope to allow humans to understand the cause of a model
prediction, thus increasing the degree to which a human can consistently
predict the model’s results.

Figure 1: Illustration of local and global explanation. On the right, impact
of the concept “Directing” is the change in prediction after its removal.

Current XAI methods fall into two categories based on model scope: (i) local
or instance-based, and (ii) global or model-based (Adadi and Berrada, 2018).
Illustrated in Fig. 1, to explain a sentiment classification model, local
explanations focus on answering “why the model predicts negative for this
particular instance”. Local methods such as counterfactuals have been widely
used to find the minimal change to an instance that results in an alternative
prediction Wachter et al. (2017); Wu et al. (2021). Global explanations, on
the other hand, attempt to explain the decision process at the model level,
such as using feature clusters like ‘movie plot’ and ‘acting’. Among them,
concept-based models, such as Koh et al. (2020) and Kim et al. (2018), extract
high-level concepts to characterize the global behavior of deep neural models.
When understanding texts, humans often employ _concept-based reasoning_ , such
as grouping words into topics (Tenenbaum, 1998). Therefore, concept-based
interpretability provides a natural way of deriving explanations for NLP
models.

However, one shortcoming of the existing concept-based methods is the lack of
consideration of the explanations’ “impact” on output predictions. We define
the “impact” of a feature to be the change in output predictions without it.
For example, the impact of the _directing_ concept in Fig. 1 is the change in
prediction after removing it. Current concept-based methods emphasize learning
features that can recover a model’s predictions accurately. Thus, these
features highly correlate with the predictions. However, as correlation does
not always imply causation, we find that such explanations often have low
impact, _i.e_. excluding it will not cause the predictions to be different.
This undermines the validity of such explanations as users cannot utilize them
to consistently predict the model’s behavior when a feature changes. This is
especially concerning for NLP. Stemming from the compositional nature of
languages, LLMs contain a large number of _spurious correlations_ – features
that are useful for training but not causal, which have become a serious
threat (Feder et al., 2021; McCoy et al., 2019; Eisenstein, 2022). Therefore,
simply applying correlation-based concept mining methods is problematic, as it
may often generate correlational explanations.

In this work, we propose _HI-Concept_ , a complete framework for explaining
LLMs based on high-level concepts that directly impact the model’s predictions
by loss construction. As a post-hoc approach, our method trains a lightweight
concept model that discovers latent features from hidden activations as global
concepts, which can also be mapped back to generate local explanations.
Specifically, to optimize for a high-impact set of concepts, we design a
_causal loss_ to be optimized during training the concept model. To evaluate
the faithfulness, usability, and impact of the discovered concepts, we devise
several evaluation metrics and conduct human studies. In our experiments with
2 classification datasets and 3 pretrained models, we observe that HI-Concept
consistently finds high-impact concepts that are more faithful and usable. Our
contributions can be summarized as follows:

  * •

We first propose HI-Concept, a method to derive both global concepts and their
corresponding local explanations that result in high output changes. Both
forms of generated explanations can complement each other while conforming to
the ‘mindset’ of the model.

  * •

We propose evaluation metrics that stem from theoretical definitions of
treatment effects in literature (Pearl, 2009) and design a human study to
prove the usability of the derived explanations.

  * •

We show that HI-Concept is impactful, usable, and faithful by constructing
reliable and extensive experiments.

##  2 Related Work

###  2.1 Explainability methods

#### Concept-based (General-domain):

Concept-based methods Kim et al. (2018) have been a popular group of
interpretability methods as it derives user-friendly, high-level concepts as
explanations. Among them, the most recent method is ConceptShap Yeh et al.
(2020), which discovers concepts in the intermediate layer with a bottleneck-
shaped extraction model and proposes an adapted Shapley value metric to
evaluate completeness scores. However, as we will show in our experiments
(Section 5), because the existing concept-based methods do not differentiate
between correlational and causal information, their performances on NLP tasks
is problematic: the discovered concepts often have little impact on final
model predictions. Especially on complex transformer models with stronger
confounding effects brought by pretraining, their performances may further
decrease.

#### NLP:

Many NLP explainability methods only discover correlational features, such as
induction-based methods Ling et al. (2017), explainability-aware architectures
Rajani et al. (2019), and feature importance scores Croce et al. (2019).

###  2.2 Causal explainability methods

#### General-domain

Harradon et al. (2018) attempt to intervene in an unsupervised way on the
hidden space by constructing several even-spaced Variational Autoencoders
(VAEs) throughout a CNN, but they only train with a reconstruction loss
instead of explicitly optimizing for impact. Probing methods (Conneau et al.,
2018; Belinkov et al., 2020) train an external model - a _probe_ \- to predict
some properties of interest from the latent representations. To further
investigate causal effects of the features learned from probing, Elazar et al.
(2021) assess the influence of a causal intervention by removing a feature.
However, subsequent work Barrett et al. (2019) shows that such methods
generalize poorly to unseen samples. Moreover, as Belinkov (2022) points out,
the disconnect between the probing model and the original model may result in
the properties not being utilized in the original model’s prediction task.
Causal Mediation Analysis (CMA) Pearl (2022) measures the change in an output
following a counterfactual intervention in an intermediate variable, or
mediator.

#### NLP

According to Feder et al. (2021), causality shows a promising path forward for
NLP research, which can offer insights into the model’s inner workings. Most
current methods attempt to causally explain an NLP model by generating
_counterfactual_ inputs Alvarez-Melis and Jaakkola (2017); Veitch et al.
(2021); Wu et al. (2021). Vig et al. (2020) apply CMA to examine gender bias
by changing pronouns in the input. We argue that both probing and CMA could be
limited to a few aspects as they rely on human-constructed features (_e.g_.,
linguistic, gender features), requiring expertise on the datasets and tasks.
Thus, it might be beneficial to develop unsupervised explanation features.

##  3 Methodology

###  3.1 Task formulation

We follow the definition of interpretability by Kim et al. (2016): “the degree
to which a human can consistently predict the model’s result.” Given a
pretrained model $f$, we aim to explain why it makes a certain predictions
$f({\mathbf{x}})$ for an input ${\mathbf{x}}$. To ensure validity, the
explanation should significantly contribute to the overall prediction process,
thus enabling humans to “consistently predict” the results. As $f$ is fixed,
its prediction process is deemed deterministic and reproducible, allowing us
to conduct counterfactual experiments.

We generally follow the setup in traditional concept extraction models (Yeh et
al., 2020), where explanations are high-level features (concepts) discovered
on the hidden activation space with a concept extraction model. The pretrained
model $f$ can be viewed as a composite of two functions, divided at an
intermediate layer $f=\psi\circ\phi$. $\phi(\cdot)$ encodes the input text
${\mathbf{x}}$ to a hidden representation $\phi({\mathbf{x}})$, and
$\psi(\cdot)$ maps $\phi({\mathbf{x}})$ to classification probabilities
$\psi(\phi({\mathbf{x}}))$. Without loss of generality, we assume that, for an
input ${\mathbf{x}}$, which consists of $T$ tokens $[x_{1},\dots,x_{T}]$,
$\phi({\mathbf{x}})$ can be represented as a concatenation of
$[\phi(x_{1}),\ldots,\phi(x_{T})]$, where each $\phi(x_{t})\in\mathbb{R}^{d}$
denotes a representation of an input token $x_{t}$. Depending on the model
architecture, $\phi(x_{t})$ can be encoded from a local receptive field as in
convolutional nets or a global one as in Transformers (Vaswani et al., 2017).
As the hidden space $\phi({\mathbf{x}})$ reflects the input distribution
${\mathbf{x}}$, $n$ concepts
${\mathcal{C}}=\\{{\mathbf{c}}_{1},\dots,{\mathbf{c}}_{n}\\}$ could be
extracted on $\phi({\mathbf{x}})$ that correspond to different input features.

###  3.2 Correlated VS. predictive explanations

The general approach to concept-based explanation has been to use a concept
mining model to discover high-level features from $\phi({\mathbf{x}})$.
However, it only finds _correlational_ features, as it does not force
explanations to be used in final predictions. The failure cases can be
explained by drawing inspiration from causality analysis (Fig. 2). A real-life
analogy is that, while the hot weather ($X$) creates high demand for ice cream
($E$), it also produces intense UV light exposure ($Z$), causing more sunburns
($Y$). However, it is obvious that high ice cream sales ($E$) do not cause
sunburns ($Y$).

Figure 2: Causal graph illustration.

In pretrained LLMs, the hidden activation space consists of both $E$ and $Z$.
Although only $Z$ truly affects prediction $Y$, $E$ and $Z$ may be correlated
due to the confounding effects brought by $X$. However, a traditional concept
mining model does not differentiate between them and considers both as valid,
which is problematic. In our experiments (Section 5), we also observe that the
discovered concepts based on current methods often have little to no impact on
output predictions, _i.e_. generate tiny changes in predictions when a concept
is turned off, especially when interpreting LLMs like BERT (Devlin et al.,
2018) and T5 (Raffel et al., 2020). To tackle this challenge, we enforce
explanations to be predictive by considering their “impact”.111We assume that,
as the concept vectors coexist in the hidden embedding space of $E\cup Z$,
there is no causal relationship among the concepts
$\\{{\mathbf{c}}_{1},\dots,{\mathbf{c}}_{n}\\}$ themselves.

###  3.3 Defining impact

To formally define the _impact_ of a feature, we draw inspiration from causal
analysis, namely Individual Treatment Effect (ITE) and Average Treatment
Effect (ATE), which measure the effect of interventions in randomized
experiments. Given a binary treatment variable $T$ that indicates whether an
intervention is performed, ATE and ITE are defined as the change in expected
outcome with the do-operation:

$\begin{split}\text{ITE}(x):=&~{}~{}\mathbb{E}[y|\textbf{X}=x,\text{do}(T=1)]\\\
&~{}~{}~{}~{}-\mathbb{E}[y|\textbf{X}=x,\text{do}(T=0)];\\\
\text{ATE}:=&~{}~{}\mathbb{E}[\text{ITE}(x)]\end{split}$ (1)

In our case, a concept ${\mathbf{c}}_{i}$ is discovered as a direction in the
latent space, corresponding to a feature in the input distribution. If
${\mathbf{c}}_{i}$ is used by the model for prediction, the removal of
${\mathbf{c}}_{i}$ should cause a change in prediction. Thus, we define
_impact_ I of a concept ${\mathbf{c}}_{i}$ on an instance $({\mathbf{x}},y)$
as:

$\begin{split}\text{I}({\mathbf{c}}_{i},{\mathbf{x}})=\mathbb{E}[y|\textbf{X}={\mathbf{x}},{\mathbf{c}}_{i}={\bm{0}}]-\mathbb{E}[y|\textbf{X}={\mathbf{x}},{\mathbf{c}}_{i}={\mathbf{c}}_{i}]\end{split}$
(2)

Then, _individual impact_ ($\text{I}({\mathbf{c}}_{i}))$ of a concept
${\mathbf{c}}_{i}$ and _average impact_ ($\text{I}({\mathcal{C}})$) for a
concept set ${\mathcal{C}}$ could be written as:

$\begin{split}\text{I}({\mathbf{c}}_{i}):=\mathbb{E}[\text{I}({\mathbf{c}}_{i},{\mathbf{x}})];~{}~{}\hskip
4.30554pt\text{I}({\mathcal{C}})=&\mathop{\mathbb{E}}_{{\mathbf{c}}_{i}\in{\mathcal{C}}}[\text{I}({\mathbf{c}}_{i})]\end{split}$
(3)

###  3.4 Concept vector extraction

We approximate the distribution over the concepts by encoding the hidden
activations $\phi(x_{t})\in\mathbb{R}^{d}$ into concept probabilities
$p_{{\mathcal{C}}}(x_{t})=[p_{c}^{1}(x_{t}),\dots,p_{c}^{n}(x_{t})]$ with a
bottleneck-shaped network while optimizing for _impact_. For a visualization
of the overall model architecture, we refer readers to Appendix A.

#### Utilizing the original concept model

After initializing the concepts
${\mathcal{C}}=\\{{\mathbf{c}}_{1},\dots,{\mathbf{c}}_{n}\\}$ uniformly, the
probability for concept ${\mathbf{c}}_{i}$ is calculated as
$p_{c}^{i}(x_{t})=\text{TH}((\phi(x_{t})^{\top}{\mathbf{c}}_{i}),\beta)$,
where TH is a threshold function that forces all inputs smaller than $\beta$
to be $0$. To get the concept distribution for the entire sequence
${\mathbf{x}}$, we concatenate the token-level distributions:
$p_{{\mathcal{C}}}({\mathbf{x}})=[p_{{\mathcal{C}}}(x_{1}),\ldots,p_{{\mathcal{C}}}(x_{T})]\in\mathbb{R}^{T\times
n}$. For transformer-based models such as BERT and T5, $\phi({\mathbf{x}})$
consists of only the sequence-level representation (e.g., [cls]) that is used
for prediction. After encoding $\phi({\mathbf{x}})$ into
$p_{{\mathcal{C}}}({\mathbf{x}})$, the bottleneck-shaped network reconstructs
$\phi({\mathbf{x}})$ with a 2-layer perceptron $g_{\theta}$ such that
$g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}))\approx\phi({\mathbf{x}})$.

Then, two loss terms are proposed to train the concept model in an end-to-end
way:

$\bullet$ Reconstruction loss: To faithfully recover the original DNN model’s
predictions, we optimize a surrogate loss with cross-entropy (CE) defined as:

$\begin{split}{\mathcal{L}}_{\text{rec}}{(\theta,{\mathcal{C}})}&=\text{CE}\Big{(}\psi\big{(}\phi({{\mathbf{x}}})\big{)},\psi\big{(}g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}))\big{)}\Big{)}\\\
&=-\sum_{{b}\in{\mathcal{B}}}\psi\big{(}\phi({\mathbf{x}})\big{)}_{b}\log\big{(}\psi(g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}})))_{b}\big{)}\end{split}$
(4)

where ${\mathcal{B}}$ is the set of class labels and $\psi(.)_{b}$ denotes the
prediction score corresponding to label $b$.

$\bullet$ Regularization loss: To ensure user-friendliness, we encourage each
concept vector to correspond to actual examples and concepts to be distinct
from each other. Formally,

$\begin{split}{\mathcal{L}}_{\text{reg}}({\mathcal{C}})=&-\lambda_{1}\frac{\sum_{i=1}^{n}\sum_{x_{t}\in{\mathcal{R}}_{i}}{\mathbf{c}}_{i}^{\top}\phi(x_{t})}{nN}\\\
&+\lambda_{2}\frac{\sum_{i_{1}\neq
i_{2}}{\mathbf{c}}_{i_{1}}^{\top}{\mathbf{c}}_{i_{2}}}{n(n-1)}\end{split}$ (5)

#### Optimizing for impact

We introduce two losses to the original framework: (i) Auto-Encoding loss,
which guarantees that the discovered concepts can serve as latent features
representing the original distribution of $\phi({{\mathbf{x}}})$; (ii)
Causality Loss, which mimics an estimation of “impact” proposed in Eq. 3. By
directly optimizing these two new objectives, HI-Concept ensures that the
discovered concepts generate a high impact on the prediction process.

$\bullet$ Auto-encoding loss: we define the following mean-squared error (MSE)
loss for auto-encoding:

$\begin{split}{\mathcal{L}}_{\text{enc}}{(\theta,{\mathcal{C}})}&=\text{MSE}\Big{(}\phi({{\mathbf{x}}}),g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}))\Big{)}\\\
&=\frac{1}{d}||\phi({{\mathbf{x}}})-g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}})||_{2}^{2}\end{split}$
(6)

$\bullet$ Causality loss: This loss is designed to disentangle concept
directions that have a greater impact. Following Eq. 3, our intuition is that
a merely correlational concept should have an impact close to $0$. Therefore,
we optimize the following loss.

$\begin{split}{\mathcal{L}}_{\text{cau}}(\theta,{\mathcal{C}})=&-\mathop{\sum}_{{\mathbf{c}}_{i}\in{\mathcal{S}}}\mathop{\sum}_{{\mathbf{x}}_{j}\in{\mathcal{D}}}\Big{|}\psi\Big{(}g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}_{j})|{\mathbf{c}}_{i}={\bm{0}})\Big{)}\\\
&-\psi\Big{(}g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}_{j})|{\mathbf{c}}_{i}={\mathbf{c}}_{i})\Big{)}\Big{|}\approx-|I_{\text{avg}}({\mathcal{C}})|\end{split}$
(7)

Here, ${\mathcal{S}}\subseteq{\mathcal{C}}$ denotes a set of concepts to
remove.222During experiments, we have found that random selection yields the
best performance. As we perturb on all inputs
${\mathbf{x}}_{j}\in{\mathcal{D}}$, the training dataset ${\mathcal{D}}$
serves both as the treatment group and the nontreatment group, ensuring that
no divergence is present. As the term
$|\psi(\cdot|{\mathbf{c}}_{i}={\bm{0}})-\psi(\cdot|{\mathbf{c}}_{i}={\mathbf{c}}_{i})|$
in Eq. 7 approximates the impact $I({\mathbf{c}}_{i},{\mathbf{x}}_{j})$ in Eq.
2, minimizing the designed causality loss is a close approximation to
maximizing the average impact in Eq. 3. Intuitively, this loss encourages the
concepts to incorporate directions that result in more significant changes in
the output predictions.

#### Total loss for HI-Concept

Finally, the overall loss function that we minimize becomes:

$\begin{split}{\mathcal{L}}(\theta,{\mathcal{C}})=&{\mathcal{L}}_{\text{rec}}{(\theta,{\mathcal{C}})}+{\mathcal{L}}_{\text{reg}}({\mathcal{C}})\\\
&+\lambda_{e}{\mathcal{L}}_{\text{enc}}(\theta,{\mathcal{C}})+\lambda_{c}{\mathcal{L}}_{\text{cau}}(\theta,{\mathcal{C}})\end{split}$
(8)

where $\lambda_{e}$, $\lambda_{c}$ are the weights for the auto-encoding loss
and the causal loss respectively. In practice, the hyperparameters require
minimal tuning. Specifically, we recommend fixing $\lambda_{1}=0.1$ and
$\lambda_{2}=0.5$ for regularizer loss in Eq. 5, and $\lambda_{e}=1$ for
reconstruction loss. The only hyperparameter to tune is $\lambda_{c}$, whose
optimal level can be found within a few steps. Futher details on
implementation and the training process could be found in Appendix B.

#### Mapping concepts back to word tokens

To derive corresponding local explanations, we employ existing techniques to
map activation-space features back to the discrete input tokens when the
receptive field of a concept is larger than token-level. For explaining BERT,
we employ the transformer visualization method proposed in Chefer et al.
(2021) to map back from the [CLS] activation concepts to input tokens.
Specifically, it visualizes classification with a combination of layer-wise
propagation (LRP), gradient backpropagation, and layer aggregation with
rollout. As a result, for each sample ${\mathbf{x}}$ and concept
${\mathbf{c}}_{i}$, we go from having only one concept similarity score
$p_{c}^{i}({\mathbf{x}})$ to having a list of normalized token importance
scores $s_{1}({\mathbf{c}}_{i}),\dots,s_{T}({\mathbf{c}}_{i})$. For the
intermediate layers of BERT, we simply use the corresponding token
representation (i.e., $\phi(x_{t})$ and its corresponding $p_{c}^{i}(x_{t})$).
For CNNs, we employ the GradCam approach (Selvaraju et al., 2017), which rolls
out the gradients to produce scores for each token.

###  3.5 Evaluating impact

To evaluate _impact_ for a feature, we devise three metrics based on its
definition in Eq. 3, which can be globally applied to other feature-based
interpretability methods.

$\bullet$ Individual and average impacts: We approximate Eq. 3 on the test set
${\mathcal{D}}_{\text{test}}$:

$\begin{split}\text{I}({\mathbf{c}}_{i})&:=\mathop{\mathbb{\sum}}_{{\mathbf{x}}_{j}\in{\mathcal{D}}_{\text{test}}}|\psi\big{(}g_{\theta}(p_{\mathcal{C}}({\mathbf{x}}_{j}))\big{)}-\psi\big{(}g_{\theta}(p_{{\mathcal{C}}\setminus\\{i\\}}({\mathbf{x}}_{j}))\big{)}|;\\\
\text{I}({\mathcal{C}})&=\frac{1}{\lvert{\mathcal{C}}\rvert}\mathop{\sum}_{{\mathbf{c}}_{i}\in{\mathcal{C}}}\text{I}({\mathbf{c}}_{i})\end{split}$

$\bullet$ Recovering accuracy change: Intuitively, if a concept
${\mathbf{c}}_{i}$ is a crucial factor used by the model to make predictions,
omitting it will disrupt the ability to faithfully recover predictions.
Denoting
$\text{Acc}({\mathcal{C}})=\text{Accuracy}(\psi\big{(}g_{\theta}(p_{\mathcal{C}}({\mathbf{x}}))\big{)},\phi(\psi({\mathbf{x}}))$:

$\begin{split}\Delta\text{Acc}_{i}&=|\text{Acc}({\mathcal{C}})-\text{Acc}({\mathcal{C}}\setminus\\{{\mathbf{c}}_{i}\\})|;\\\
\Delta\text{Acc}&=\frac{1}{\lvert{\mathcal{C}}\rvert}\mathop{\sum}_{{\mathbf{c}}_{i}\in{\mathcal{C}}}\Delta\text{Acc}_{i}\end{split}$

$\bullet$ Impact score: To measure the impact of an individual text token
$x_{t}$, we take the top-3 most similar concepts ${\mathcal{C}}_{\text{top}}$
to the input ${\mathbf{x}}$ using the normalized similarity score
$p_{{\mathcal{C}}}({\mathbf{x}})$. For each concept
${\mathbf{c}}_{i}\in{\mathcal{C}}_{\text{top}}$, the transformer visualization
method (Section 3.4) produces normalized token importance scores
$\\{s_{1}({\mathbf{c}}_{i}),\dots,s_{\text{T}}({\mathbf{c}}_{i})\\}$. The
impact score for a token $x_{t}$ is defined as:

$\begin{split}\text{I}(x_{t})=\mathop{\sum}_{c_{i}\in{\mathcal{C}}_{\text{top}}}p_{c}^{i}({\mathbf{x}})s_{t}({\mathbf{c}}_{i})\end{split}$

##  4 Experiment Settings

###  4.1 Datasets and classification models

We test the effectiveness of our method with two standard text classification
datasets: IMDB (Maas et al., 2011) and AG-news (Zhang et al., 2015). The IMDB
dataset consists of movie reviews labeled with positive or negative
sentiments. The AG-news dataset consists of news articles categorized with 4
topics. Table 6 in Appendix D gives a dataset summary. We explain three
classification models: (i) a $6$-layer transformer encoder trained from
scratch, (ii) a pre-trained BERT with finetuning, (iii) a pre-trained T5
(Raffel et al., 2020) with finetuning.

Dataset | Model | Cls.Acc | Metric |  $\beta$-TCVAE | K-means | PCA | ConceptSHAP | HI-Concept
---|---|---|---|---|---|---|---|---
IMDB | Transformer | 81.74% | $\text{I}({\mathcal{C}})$ | 0.037 | 0.047 | 0.001 | 0.031 | 0.150
$\Delta$Acc | 1.24% | 2.59% | 0.01% | 1.30% | 11.06%
RAcc | 52.08% | 83.64% | 85.18% | 84.36% | 88.78%
BERT | 89.14% | $\text{I}({\mathcal{C}})$ | 0.057 | 0.038 | 0.002 | 0.050 | 0.104
$\Delta$Acc | 4.10% | 1.56% | 0.02% | 0.06% | 9.47%
RAcc | 93.86% | 98.69% | 96.68% | 95.84% | 94.53%
T5 | 72.98% | $\text{I}({\mathcal{C}})$ | 0.000 | 0.025 | 0.000 | 0.000 | 0.094
$\Delta$Acc | 0.00% | 1.06% | 0.02% | 20.21% | 38.34%
RAcc | 0.00% | 75.85% | 98.86% | 60.20% | 99.50%
AG | Transformer | 88.33% | $\text{I}({\mathcal{C}})$ | 0.049 | 0.044 | 0.000 | 0.000 | 0.045
$\Delta$Acc | 6.62% | 0.07% | 0.03% | 0.00% | 7.12%
RAcc | 98.90% | 98.16% | 99.99% | 73.01% | 99.50%
BERT | 93.75% | $\text{I}({\mathcal{C}})$ | 0.044 | 0.028 | 0.001 | 0.025 | 0.058
$\Delta$Acc | 5.32% | 7.15% | 0.01% | 4.44% | 10.54%
RAcc | 92.30% | 86.83% | 99.79% | 93.46% | 99.90%
T5 | 94.30% | $\text{I}({\mathcal{C}})$ | 0.000 | 0.011 | 0.000 | 0.000 | 0.054
$\Delta$Acc | 0.00% | 1.49% | 0.01% | 0.00% | 52.20%
RAcc | 0.00% | 24.87% | 97.38% | 0.00% | 99.46%

Table 1: Faithfulness (RAcc$\uparrow$) and causality
($\text{I}({\mathcal{C}})$$\uparrow$, $\Delta\text{Acc}\uparrow$) evaluation
of different text classification methods.

###  4.2 Evaluation measures

We evaluate the explanation methods based on three important aspects as
described below.

$\bullet$ Causality: Causality is an important consideration to evaluate
explainability methods, especially where spurious correlations are strong.
Doshi-Velez and Kim (2017) state: “Causality implies that the predicted change
in output due to a perturbation will occur in the real system”. We evaluate
$\text{I}({\mathcal{C}})$ and $\Delta\text{Acc}$ proposed in Section 3.5, as a
higher $\text{I}({\mathcal{C}})$ and $\Delta\text{Acc}$ represent a higher
change in model predictions, thus a more impactful set of concepts.

$\bullet$ Usability: Proposed in (Doshi-Velez and Kim, 2017), an important
desideratum of explainability is to make sure that it provides usable
information that assists users to accomplish a task. With the concepts
generating a high impact on predictions, we expect that it can enable end-
users to better understand the model’s reasoning process, thus being useful
for debugging and fairness. We include visualizations and human studies to
test it qualitatively.

$\bullet$ Faithfulness: Faithfulness evaluates whether our surrogate model can
accurately mimic the original model’s prediction process. In other words, we
want to ensure that the captured concept probabilities
$p_{{\mathcal{C}}}({\mathbf{x}})$ can recover the original model’s predictions
$\psi\big{(}\phi({{\mathbf{x}}})\big{)}$. We report the _recovering accuracy_
for the set of concepts ${\mathcal{C}}$:

$\begin{split}\text{RAcc}=\frac{1}{|{\mathcal{D}}_{\text{test}}|}\mathop{\sum}_{{\mathbf{x}}_{j}\in{\mathcal{D}}_{\text{test}}}\mathbbm{1}\Big{(}\psi\big{(}\phi({{\mathbf{x}}_{j}})\big{)}=\psi\big{(}g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}_{j}))\big{)}\Big{)}\end{split}$

###  4.3 Baselines and hyperparameters

As fair comparisons to our method, we can only consider _unsupervised_ feature
discovery algorithms. Thus, we use conceptSHAP (Yeh et al., 2020) as a
baseline. To compare to VAE methods, we include the disentanglement VAE
($\beta$-TCVAE) by Chen et al. (2018). Moreover, we include comparisons to
popular non-parametric clustering techniques, including PCA and k-means.

The full list of hyperparameters used for training the HI-Concept model can be
found in Appendix D. Briefly, we use the causal coefficient
$\lambda_{c}\in[1,3]$, depending on the level of confounding within the
dataset. During training, perturbation is performed on the most similar
concept to the input. All experiments are conducted on the penultimate layer
with 10 concepts. The hyperparameters are chosen as an optimal default through
grid search. To make the comparison fair, PCA, K-means, and $\beta$-TCVAE also
use 10 dimensions to encode.

##  5 Results and Analysis

To first provide a sanity check for our method, we conduct a toy experiment
with a synthetic graphic dataset where the level of confounding can be
controlled with ground truth concepts. Appendix C gives details of the
experiment. The results show that our method discovers concepts that align
with human understanding and consistently outperforms the baseline by deriving
more impactful features. As confounding levels in the dataset increase, the
performance gap also widens.

Method | $\text{I}({\mathcal{C}})$ | Keywords
---|---|---
ConceptSHAP | 0.000 | one, two, gt, new, cl, lt, first, world, mo, last, b, san, tuesday, soccer, time,nhl, Australia, red, bryant
ConceptSHAP | 0.000 | first, new, red, world, Yankees, Australia, giants, nl, as, two, one, ga, last, b, u, tuesday, quo, men
HI-Concept | 0.108 | update, us, fed, wal, op, u, stocks, oil, dollar, delta, hr, ex, Wednesday, world, percent, crude
HI-Concept | 0.151 | red, NBA, football, Yankees, sports, NFL, team, baseball, olympic, league, game, season, coach

Table 2: Generated concept keywords with Average Impact
($\text{I}({\mathcal{C}})$) from AG-News dataset, BERT model.

###  5.1 Results on text classification datasets

The experiment results on text classification datasets are presented in Table
1. Concepts discovered by the baseline methods lead to tiny changes in
prediction outputs, which undermine their reliability. On the contrary, on all
models, especially pretrained BERT and T5, concepts discovered by HI-Concept
induce a larger impact than all the baseline methods, while maintaining
faithfulness. This observation consolidates our intuition that pretrained
complex language models with more confounding correlations can benefit more
from HI-Concept. As a seq2seq text generation model, the pretrained T5 is
especially hard to learn for the surrogate models, as the output vocabulary
has a class size of 32,128. For better calculation of impact, we simplify
outputs by filtering to only the classification classes (e.g., words
“Positive”, “Negative” for IMDB) and summing all other vocab probabilities as
“Other”. Some models collapse completely in this case. HI-Concept, however,
excels in maintaining both faithfulness and a large impact.

To qualitatively examine the discovered concepts, we take an example of BERT
on AG-News. In Table 2, 2 out of 10 concepts from both HI-Concept and the
baseline are shown. For ConceptSHAP, both concepts have low
$\text{I}({\mathcal{C}})$, corresponding to lower impact. Although vaguely
hinting at the category “Sports”, they consist of words that are less
indicative, such as “one”, “two”, and “new”. When looking at HI-Concept
discovered topics, the first talks about “World”, especially in the global
finance topic. The second clearly points to the American sports leagues,
indicating category “Sports”. Thus, instead of merely pointing to class
information, the concepts discovered contain more information that aligns with
human understandable concepts. The $\text{I}({\mathcal{C}})$ score shown here
is also consistent with what humans perceive as important and causal words to
the classification, thus indicating the metric’s validity. Moreover, in
Section G.1, we show more concepts discovered in former layers of BERT, which
show that the concepts are not only separable in semantic meanings, but also
syntactical information (such as nouns and adjectives).

Method |  Visualization
---|---
ConceptSHAP |  dream team leads spain 44 - 42 at halftime athens, greece \- as expected, the u.s. men’s basketball team had its hands full in a quarterfinal game against spain on thursday…
HI-Concept |  dream team leads spain 44 \- 42 at halftime athens, greece \- as expected, the u.s. men’s basketball team had its hands full in a quarterfinal game against spain on thursday …

Figure 3: Qualitative comparison from AG-News: “World” news misclassified as
“Sports” by BERT.

The usability of our method is visualized in Fig. 3, which shows the same
failure case (labeled as “World” news but misclassified as “Sports”)
highlighted with the top concept discovered. ConceptSHAP discovers a top
concept related to the keywords “leads”, “as expected”, or “on thursday”,
which are not informative as to why the model classified this input as
“Sports”. On the contrary, HI-Concept could precisely point out why: BERT is
looking at keywords such as “dream team”, “game”, and country names. Such
examples show the potential of HI-Concept being used in understanding the
model’s failure processes, which we further investigate in Section 5.3 with a
carefully designed human study.

###  5.2 Ablation study

To ensure that the designated 4 objectives behave as expected, we conduct
ablation studies for BERT on AG-News and report the results in Table 3. As
observed, eliminating prediction loss leads to a large decrease in RAcc,
resulting in an unfaithful model. Thus, even though the model leads to large
accuracy changes, the results cannot be trusted. Without auto-encoding loss or
regularizer loss, the model has lower performances both in faithfulness and
impact. Without causality loss, RAcc is the highest, indicating an accurate
reconstruction of the original predictions. However, the discovered set of
concepts results in lower output changes. Finally, the full HI-Concept method
discovers a set of concepts that both generate high impact and maintain a good
level of faithfulness.

###  5.3 Human study

To validate that HI-Concept can identify words that are more impactful than
the baseline ConceptSHAP, we design the following human study setup: 100
randomly selected examples from AG’s testset are shown, where each example
consists of the text input and the model’s prediction. The annotator is asked
to select up to three most causal words for the predicted label. We collect
annotations from 4 different annotators proficient in English in order to
obtain a diverse set of causal keywords. We consider the keywords selected by
the annotators to be the ground-truth and calculate the average impact score
($\text{I}(x_{t})$) (Section 3.5) for all unique words (superset) selected by
the annotators, with the baseline and our method trained on the penultimate
layer of BERT. Details about how the human study is conducted can be found in
Appendix F.

Method |  RAcc $\uparrow$ |  $\text{I}({\mathcal{C}})$ $\uparrow$ | $\Delta\text{Acc}\uparrow$
---|---|---|---
No Auto-Encoding Loss | 93.46% | 0.028 | 6.11%
No Prediction Loss | 68.00% | 0.035 | 17.41%
No Regularizer Loss | 95.76% | 0.041 | 6.23%
No Causality Loss | 99.92% | 0.029 | 2.95%
HI-Concept | 99.90% | 0.058 | 10.54%

Table 3:  Ablation on BERT for IMDB with faithfulness (RAcc) and impact
($\text{I}({\mathcal{C}})$, $\Delta\text{Acc}$) evaluation.

| # Examples | Cohen $\kappa$ | ConceptSHAP | Ours
---|---|---|---|---
| (Impact) | (Impact)
Total | 100 | 0.41 | 0.41 | 0.83

Table 4: Human study for causality/usability evaluation.

As shown in Table 4, the average impact score produced by HI-Concept is 2
times higher than the baseline, demonstrating that our model selects the right
(causal) tokens. The annotators have a Cohen’s Kappa agreement of 0.41, which
is considered as moderate agreement (Landis and Koch, 1977). Therefore, even
though the annotators prefer slightly different keywords as causal, our model
still assigns a higher score to them compared to the baseline, covering causal
keywords preferred by all annotators. For example, Fig. 4 shows a correctly
classified “Business” news article. The human annotators, with a small
disagreement, provide 4 unique keywords, which are all highlighted by HI-
Concept. Such ability to identify causal tokens gives HI-Concept the potential
to be used for debugging applications.

HI-Concept |  google shares, once devalued, just may be winners after all wall street, which forced google, the internet search engine, to sharply lower the price of its shares in its initial public offering in august, has decided that the company is worth a lot more today than it was then.
---|---
Human 1 |  devalued, shares, price
Human 2 |  devalued, shares, google

Figure 4: Example sentence in the human study: “Business” news correctly
classified.

###  5.4 Hyper-parameter comparisons

We perform further studies on two hyperparameters: the layer(s) to interpret
and number of concepts. We conduct text experiments and evaluate in terms of
both impact and concept quality. In this section, we summarize our main
findings and refer the readers to Appendix G for experiment details and
results (charts and wordcloud visuals).

For layer-wise comparisons, we experiment on the 3rd, 6th, 9th, and 12th layer
respectively, all with 10 concepts. In terms of impact, the intermediate
layers (3, 6 and 9) have a higher impact $\text{I}({\mathcal{C}})$ (around
0.150), while the penultimate layer (12) has a lower $\text{I}({\mathcal{C}})$
(around 0.058). This is because the concepts discovered at the penultimate
layer are sentence-level (using the [CLS] token), while the intermediate layer
concepts are token-level. Thus, the sentence-level concepts have less fine-
grained control. In terms of topic quality, the later layers tend to discover
more coherent concepts, where each concept mostly corresponds to one class
label. The beginning layers, on the contrary, tend to discover concepts that
are more abstract with mixed class labels. The earlier layers can also
discover lexical concepts, such as concepts with only nouns or adjectives.
Similarly, Dalvi et al. (2021) observe that BERT finds more lexical
information in the earlier layers. This interesting observation could lead to
future studies in investigating how information flows through different layers
in BERT.

For number of concepts, we experiment with 3, 5, 10, 50, and 100 concepts on
the penultimate layer. We find that a concept number close to the number of
output classes usually gives higher prediction changes, while increasing the
number results in higher recovering accuracy. When the number of concepts
becomes larger, concepts usually become more coherent, although the
performance will decrease, as too large a number of concepts introduces more
noise into the training process.

##  6 Conclusions

We have proposed a complete framework to derive impactful concepts that
explain a black-box language model’s decisions. Our framework addresses 3
important challenges in NLP explainability: (i) Impactful: it derives concepts
that generate high output changes and minimize confounding explanations
through a causal loss objective. (ii) Counterfactuals: it proposes an
innovative substitute to the traditional input counterfactual. By producing
latent counterfactuals that are designed to remove features within input
texts, we avoid the input space search. The concern of interpreting the hidden
activations is also addressed by incorporating visualization methods. (iii)
User-friendly: as demonstrated with visualizations and human studies, HI-
Concept leads to human-friendly explanations in NLP tasks that contain high-
level text attributes and semantically meaningful concepts.

## Limitations

Regarding potential concerns, HI-Concept only encourages high impact in post-
hoc model explanations and should serve as an assistive tool instead of being
accepted as ground-truth.

As a future venue to our work, we believe that our framework will set a good
foundation for future research on causal NLP explainability methods,
especially those that hope to derive human-friendly explanations. To improve
it further, a similar causal objective could be used to address spurious
correlations during training. It also has the potential of being carried over
to other domains, such as vision or tabular tasks. The high-level attributes
in the hidden space can also be used in downstream applications to provide
better controllability for the users.

## Ethics Statement

HI-Concept demonstrates the potential to play an important role in practical
scenarios such as debugging and transparency. As AI ethics have become a major
concern in real-life applications, such explanations can help users better
identify bias and promote fairness.

## References

  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking inside the black-box: a survey on explainable artificial intelligence (xai).  _IEEE access_ , 6:52138–52160.
  * Alvarez-Melis and Jaakkola (2017) David Alvarez-Melis and Tommi S Jaakkola. 2017.  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  _arXiv preprint arXiv:1707.01943_.
  * Barrett et al. (2019) Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. 2019.  Adversarial removal of demographic attributes revisited.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6330–6335.
  * Belinkov (2022) Yonatan Belinkov. 2022.  Probing classifiers: Promises, shortcomings, and advances.  _Computational Linguistics_ , 48(1):207–219.
  * Belinkov et al. (2020) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020\.  On the linguistic representational power of neural machine translation models.  _Computational Linguistics_ , 46(1):1–52.
  * Chefer et al. (2021) Hila Chefer, Shir Gur, and Lior Wolf. 2021.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 782–791.
  * Chen et al. (2018) Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018.  Isolating sources of disentanglement in variational autoencoders.  _Advances in neural information processing systems_ , 31.
  * Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single vector: Probing sentence embeddings for linguistic properties.  _arXiv preprint arXiv:1805.01070_.
  * Croce et al. (2019) Danilo Croce, Daniele Rossini, and Roberto Basili. 2019.  Auditing deep learning processes through kernel-based explanatory models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4037–4046.
  * Dalvi et al. (2021) Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2021.  Discovering latent concepts learned in bert.  In _International Conference on Learning Representations_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_.
  * Eisenstein (2022) Jacob Eisenstein. 2022.  Uninformative input features and counterfactual invariance: Two perspectives on spurious correlations in natural language.  In _NAACL_.
  * Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021.  Amnesic probing: Behavioral explanation with amnesic counterfactuals.  _Transactions of the Association for Computational Linguistics_ , 9:160–175.
  * Feder et al. (2021) Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2021.  Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.  _CoRR_ , abs/2109.00725.
  * Harradon et al. (2018) Michael Harradon, Jeff Druce, and Brian Ruttenberg. 2018.  Causal learning and explanation of deep neural networks via autoencoded activations.  _arXiv preprint arXiv:1802.00541_.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016.  Examples are not enough, learn to criticize! criticism for interpretability.  _Advances in neural information processing systems_ , 29.
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677. PMLR.
  * Koh et al. (2020) Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020.  Concept bottleneck models.  In _International Conference on Machine Learning_ , pages 5338–5348. PMLR.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mathews (2019) Sherin Mary Mathews. 2019.  Explainable artificial intelligence applications in nlp, biomedical, and malware classification: a literature review.  In _Intelligent computing-proceedings of the computing conference_ , pages 1269–1292. Springer.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Miller (2019) Tim Miller. 2019.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ , 267:1–38.
  * Pearl (2009) Judea Pearl. 2009.  _Causality: Models, Reasoning and Inference_ , 2nd edition.  Cambridge University Press.
  * Pearl (2022) Judea Pearl. 2022.  Direct and indirect effects.  In _Probabilistic and Causal Inference: The Works of Judea Pearl_ , pages 373–392.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  _arXiv preprint arXiv:1906.02361_.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015.  Exploring the space of topic coherence measures.  In _Proceedings of the eighth ACM international conference on Web search and data mining_ , pages 399–408.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626.
  * Tenenbaum (1998) Joshua Tenenbaum. 1998.  Bayesian modeling of human concept learning.  In _Advances in Neural Information Processing Systems_ , volume 11. MIT Press.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veitch et al. (2021) Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021.  Counterfactual invariance to spurious correlations in text classification.  _Advances in Neural Information Processing Systems_ , 34.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020.  Investigating gender bias in language models using causal mediation analysis.  _Advances in Neural Information Processing Systems_ , 33:12388–12401.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  _arXiv preprint arXiv:2101.00288_.
  * Yeh et al. (2020) Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020.  On completeness-aware concept-based explanations in deep neural networks.  _Advances in Neural Information Processing Systems_ , 33:20554–20565.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _NIPS_.

Appendix for “Explaining Language Models’ Predictions with High-Impact
Concepts”

##  Appendix A Overall method visualization

Fig. 5 shows the overall visualization of the concept generation process. The
neural network $f$ is divided at the intermediate layer into two parts: $\phi$
and $\psi$. The black-arrow path shows the original neural network prediction
process: $y=f({\mathbf{x}})=\psi(\phi({\mathbf{x}}))$, where ${\mathbf{x}}$ is
the input and $y$ is the classification output.

To generate concepts at the intermediate layer, instead of feeding
$\phi({\mathbf{x}})$ directly into $\psi$, we first pass it through a concept
network: Firstly, $\phi({\mathbf{x}})$ is condensed into concept probabilities
$p_{{\mathcal{C}}}({\mathbf{x}})$ by multiplying the normalized activations
$\phi({\mathbf{x}})$ with normalized concept vectors
${\mathcal{C}}=\\{{\mathbf{c}}_{1},\dots,{\mathbf{c}}_{n}\\}$ and going
through the threshold (TH) function. Then, a 2-layer perceptron $g_{\theta}$
is used to reconstruct the original activation: $\phi({\mathbf{x}})\approx
g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}}))$. The reconstruction is then
passed into $\psi$ to get the prediction
$y^{\prime}=\psi(g_{\theta}(p_{{\mathcal{C}}}({\mathbf{x}})))$. To train the
network, we use reconstruction loss, regularizer loss, and causality loss.

The green path indicates the mapping back process from concept probabilities
$p_{{\mathcal{C}}}({\mathbf{x}})$ to input tokens in
${\mathbf{x}}=[x_{1},\dots,x_{t}]$. We use the transformer visualization
approach (Chefer et al., 2021) and Grad-CAM (Selvaraju et al., 2017), which
rely on the gradients generated from the red path.

Figure 5: The overall concept generation process.

##  Appendix B Training details

In practice, we only turn on the causal loss after a certain number of epochs
(usually half of the overall number of epochs) to make sure that the surrogate
model first learns to faithfully reconstruct from the set of concepts before
optimizing for the impactful ones. This is because learning the two
conflicting objectives at once will usually result in low accuracy. We also
note that some contextual information is still needed to maximize the accurate
reconstruction of hidden activations $\phi({\mathbf{x}})$. Thus, the causality
loss is enforced on all concepts except the last one ${\mathbf{c}}_{n}$, which
is used as a ‘context concept’. During model inference, the last (non-
impactful) concept is unused.

After training, we post-process discovered concepts to filter out unused ones.
While the number of concepts $n$ is user-selected, as in many topic models, it
is an inherent flaw as it requires a certain level of domain expertise. For
example, in a movie review dataset with only 2 output classes, if an
unfamiliar user sets $n$ to 200, the model will naturally discover many noisy
concepts and only a few useful ones. To ensure that the noisy concepts are
eliminated, we post-process the concepts and filter out the unused ones (with
an impact $I_{\text{ind}}({\mathbf{c}}_{i})$ close to $0$). Thus, a more
desirable number of concepts is returned even if the user provides an
overestimate of $n$. In our experiments, we see that, after filtering, the
model always achieves a better or same prediction-reconstruction performance
as before. However, even with this post-processing, specifying too large a
number of concepts can still be dangerous as it harms the concept model’s
training process.

##  Appendix C Toy example

We conduct experiments on a synthetic (toy) image dataset with ground truth
concepts in order to test the validity of our method and confirm the claim
that higher confounding effects within the dataset lead to more correlational
explanations, thus calling for a more causal explainability approach.
Specifically, We extend the toy dataset design of Yeh et al. (2020) to make it
more realistic by inserting spurious correlations.

###  C.1 Data generation

As a synthetic setup, at most 15 shapes are randomly scattered on a blank
canvas at random locations with random color selections (as noise). For each
image sample $x_{j}$, $\bm{z^{j}_{\\{1:15\\}}}$ are binary variables of
whether or not a shape is present in $x_{j}$ with each $\bm{z}^{j}_{s}$
sampling from a Bernoulli distribution with probability $0.5$. Then, a
15-class target $\bm{y_{j}}$ is constructed with respect to whether the first
5 shapes ($\bm{z^{j}_{\\{1:5\\}}}$) are present or not with human-designed
rules. For example, $\bm{y_{1}}=\sim(\bm{z_{1}}\cdot\bm{z_{3}})+\bm{z_{4}}$. A
total of $60,000$ examples are generated as the toy dataset using a seed of 0.

The setup mentioned above is, in fact, far away from realistic scenarios, as
it does not consider possible confounding. Thus, to make it more realistic, we
insert spurious correlations between the pairs
$(\bm{z^{j}_{\\{1:5\\}}},\bm{z^{j}_{\\{6:10\\}}}),(\bm{z^{j}_{\\{6:10\\}}},\bm{z^{j}_{\\{11:15\\}}})$
with a correlation factor $p_{\text{cor}}$. For example, when $z_{1}=1$,
$z_{6}=\text{Bernoulli}(p_{\text{cor}})$; when $z_{1}=0$,
$z_{6}=\text{Bernoulli}(1-p_{\text{cor}})$.

###  C.2 CNN classification model used for the toy example

Figure 6: Convolutional Neural Network used for classifying the toy dataset.

The CNN classification model used for the toy dataset is shown in Fig. 6.
Specifically, 3 convolutional layers with a kernel size of 5 and 64 output
channels were used, each followed by a ReLU activation and max pooling layer.
Then, the result is flattened into a linear vector, followed by 2 linear
layers and a sigmoid activation function. The output is a 15-dimensional
binary classification probability. The model is trained for 100 epochs with an
Adam optimizer with learning rate $3e-4$. For reproducibility purposes, the
model is initialized and trained with a seed of 0.

###  C.3 Visualizations

Figure 7: Examples from the toy dataset and concepts discovered.

As an example visualization, in Fig. 7, two random images from the toy dataset
are displayed on the left, while three example concepts discovered by HI-
Concept are plotted on the right. We could observe that HI-Concept is able to
derive meaningful clusters as concepts, which provide a sanity check for
usability of the latent concepts.

###  C.4 Results on toy dataset

$p_{\text{cor}}$ | Cls.Acc | Method |  RAcc $\uparrow$ |  $\text{I}({\mathcal{C}})$ $\uparrow$ | $\Delta\text{Acc}\uparrow$
---|---|---|---|---|---
0.50 | 95.4% | ConceptSHAP | 97.6% | 0.070 | 6.1%
HI-Concept | 98.4% | 0.102 |  9.4% (+3.3%)
0.65 | 99.0% | ConceptSHAP | 99.7% | 0.038 | 3.5%
HI-Concept | 99.3% | 0.084 |  6.8% (+3.4%))
0.75 | 96.1% | ConceptSHAP | 98.3% | 0.069 | 6.0%
HI-Concept | 98.9% | 0.123 |  12.16% (+6.16%)

Table 5:  Faithfulness (RAcc) and Causality ($\text{I}({\mathcal{C}})$,
$\Delta\text{Acc}$) evaluation on the toy dataset. Cls.Acc denotes model’s
classification accuracy.

From the results shown in Table 5, we could observe that, as we increase
$p_{\text{cor}}$ to mimic an increase in confounding levels in real life, our
HI-Concept consistently outperforms the baseline by a bigger margin. HI-
Concept achieves higher impacts ($\text{I}({\mathcal{C}})$) and higher
accuracy change ($\Delta\text{Acc}$), while maintaining the best RAcc,
indicating faithfulness to the original predictions. Moreover, we note that
the improvement is even stronger in real data experiments, as the added
artificial confounding is more complicated in real-life scenarios.

##  Appendix D Hyperparameters used

Dataset | Train | Test | Label dim. | Avg. size
---|---|---|---|---
Toy (image) | 48k | 12k | 15 | (240, 240)
IMDB (text) | 37.5k | 2.5k | 2 | 215
AG (text) | 120k | 7.6k | 4 | 43

Table 6:  A summary of the datasets.

For all concept experiments, the following parameters are universally applied
as a selected default, which demonstrated better performances during
experiments: For regularizer losses, $\lambda_{1}=0.1$ and $\lambda_{2}=0.5$.
In $\text{TH}(\cdot,\beta)$ function, threshold is set to be
$\beta=0.1=\frac{1}{n}$, where n is the number of concepts selected. For the
top-$N$ neighborhood, $N=\frac{1}{4}\text{BS}$, where BS is the effective
batch size, which we have set as 128 during the experiments. For the masking
strategy, we always recommend masking random concepts with a probability of
$0.2$ as the optimal strategy, as masking maximum concepts may lead to a
highly uneven distribution of $\text{I}({\mathcal{C}})$ among discovered
concepts.

As all dataset class sizes are small (2 in IMDB/toy or 4 in AG-News), the
number of concepts is chosen to be 10 for all experiments. When the number of
classes is larger, we recommend choosing a larger number of concepts to ensure
a faithful reconstruction of the original input.

For training the concept model, we always use an Adam optimizer with a
learning rate of $3e-4$. All models are all trained using 100 epochs. In the
HI-Concept models, causal loss is always turned on at half of the overall
number of epochs. After turning on causal loss, all parameters are set to
untrainable except for the concept vectors, which ensures that the
reconstruction ability is not forgotten.

The same hyperparameters are set for the conceptSHAP models, which are also
found to generate the optimal performances. The threshold is set to be
$\beta=0.3$, as recommended by the original paper on NLP datasets.

For the causal loss regularizer, $\lambda_{\text{c}}=1$ is set for all
experiments, except for $\lambda_{\text{c}}=3$ in the case of IMDB with BERT.
A higher $\lambda_{\text{c}}$ will usually lead to a higher output change
($\text{I}({\mathcal{C}})$ and $\Delta\text{Acc}$), accompanied by a decrease
in faithfulness (RAcc).

To reproduce, all experiments were run with a random seed of 0.

A summary of the datasets is provided in 6. IMDB and AG-news are both licensed
for non-commercial use.

##  Appendix E Run-time

Dataset |  $\beta$-TCVAE | kmeans | PCA | conceptSHAP | HI-Concept
---|---|---|---|---|---
IMDB | 475.9 | 37.7 | 0.8 | 199.3 | 227.2
AG | 1525.6 | 15.51 | 2.5 | 1749.65 | 2242.1

Table 7:  A summary of runtime (in seconds) on datasets for BERT.

As our model optimizes for causality loss, the run-time is slightly longer
than the baseline method ConceptSHAP (Yeh et al., 2020), but is still short. A
summary of runtime is shown in Table 7. All models shown are run on the GTX
1080Ti graphic card with 12 GB memory. Generally, as post-hoc explainability
methods, the runtimes are very light and, therefore, a concern that is less
important than the model quality. For example, on a dataset of size 50k such
as IMDB, it only takes 227.2 seconds (3.8) minutes to train our HI-Concept
model.

##  Appendix F Human study setup

Figure 8: Human study instructions with a demonstration. Figure 9: Human study
question and answer.

For the human study, 100 examples are randomly selected from the test set
${\mathcal{D}}_{\text{test}}$. The questionnaire takes the format of a google
form, where the instructions in Fig. 8 are shown to the participants. An
example question looks like the one in Fig. 9. For the 100 questions repeated
twice, 4 volunteers (Ph.D. students) have answered them. The volunteers are
all proficient in English. The volunteers report an average time of 30 minutes
for answering 50 questions. As the volunteers are working also in AI-related
areas and are briefed about the purpose and usage of survey data beforehand,
they understand fully the data collection and usage. Thus, implicit consent is
granted by participation.

##  Appendix G Hyperparameter comparisons

The proposed method of HI-Concept includes many tunable hyperparameters,
including the top-N neighborhood, threshold, etc. While these parameters are
set at the default mentioned in Appendix D, there are two hyperparameters that
users can customize the most: the layer to interpret at and number of concepts
. To better understand how these two parameters may affect the generated
concepts, we conduct comparisons on both. We evaluate in terms of impact and
topic quality. For impact, we have reported the number of effective concepts
left after post-processing, the recovering accuracy (RAcc), the Average Impact
($\text{I}({\mathcal{C}})$), and the induced change in accuracy
($\Delta\text{Acc}$). For topic quality, we have reported coherence scores,
including averaged Pointwise Mutual Information (PMI) (c_uci score),
normalized PMI (c_npmi score), c_v score which measures how often the topic
words appear together in the corpus, and word2vec similarity (Röder et al.,
2015).

The following comparisons are all conducted on the AG-news dataset with BERT,
where the other hyperparameters mentioned in Appendix D stay the same.

###  G.1 Layer-wise comparison

Figure 10: Layer-wise effective number of concepts, RAcc $\uparrow$,
$\text{I}({\mathcal{C}})$ $\uparrow$, and $\Delta$ Acc $\uparrow$.

To compare what each layer discovered, as BERT has 12 layers, we experimented
on the 3rd, 6th, 9th, and penultimate layer respectively, all with 10
concepts.

Quantitatively, we plotted out the effective number of concepts, recovering
accuracy, impact and accuracy change in Fig. 10. All layers demonstrate
similar performances in recovering accuracy, which is close to 100%. The
intermediate layers, especially the 6th layer, produce a higher average impact
and recovering accuracy. This is because the intermediate layers discover
concepts on the token-level, while the penultimate layer concepts are
sentence-level (on the [CLS] token). Thus, the token-level concepts will have
more fine-grained control.

Figure 11: Wordclouds of concepts generated on the 12th layer, including a
sports concept, a technology concept, and a political concept.

Figure 12: Wordclouds of concepts generated on the 9th layer, including a
government concept, a China concept, and an Adjective (mostly) concept.

Qualitatively, we plotted some wordclouds of the keywords in discovered
concepts in Fig. 11 and Fig. 12. From Fig. 11, we could see that, in the
penultimate layer, concepts are more concentrated on each class. For example,
the first concept would correspond to the class “Sports”, the second to
“Sci/Tech”, and the third to “World” news. The emphasis on events is also
clearer, such as the third one talking about the Iraq War. However, When we
move to earlier layers, the concepts’ class labels are more mixed together. In
Fig. 12, the first concept concerns government, which includes terms such as
“government”, “internet”, “security”, “bomb”, “baseball”, etc. It could,
however, correspond to many class labels, such as “Sci/Tech”, “World”, or even
“Sports”. Similarity, the second concept talks about China, including “china”,
“billion”, “people”, “activitists”, “announcement”, etc. The third concept is
interesting as it covers mostly adjective words which do not seem to correlate
too much in semantic meanings, such as “low”, “big”, “closer”, and “third”.
Similar observations are also confirmed in papers such as (Dalvi et al.,
2021), which derives concepts using agglomerative hierarchical clustering
combined with human annotations in BERT latent representations. They observe
that BERT finds more lexical information in the earlier layers.

Figure 13: Layer-wise Topic Coherence Comparison.

In terms of topic quality, we evaluated the concept keywords using coherence
metrics. As shown in Fig. 13, all coherence scores showed a general trend of
concepts becoming more coherent as the layer number increases. The conclusion
is consistent with the wordcloud visualizations.

Thus, in real-life debugging scenarios, we recommend using the penultimate
layer, which will find more coherent topics. However, there could be continued
work to discover information learned in the prior layers and to investigate
how information flows through layers in a hierarchical way.

###  G.2 Number of concepts

In the penultimate layer of BERT, we experiment with 3, 5, 10, 50, and 100
concepts.

Figure 14: Concept-wise effective number of concepts, RAcc $\uparrow$,
$\text{I}({\mathcal{C}})$ $\uparrow$, and $\Delta$ Acc $\uparrow$. Figure 15:
Concept-wise Topic Coherence Comparison.

From Fig. 14, we could see that the performance is very dependent on the
number of concepts. The effective number of concepts, recovering accuracy,
average impact, and accuracy change all appear to be elbow-shaped. In this
case, 5 concepts provided the highest impact on output predictions, as it is
close to the number of classes (4) in the AG-News dataset. Increasing the
number of concepts to 10 would yield a better recovering accuracy. As the
number of concepts increases to 50 and 100, we observe that the model fails to
learn completely. In practice, we have often observed the best number to be
positively correlated with the number of dataset classes. In other words, a
dataset with more classes will require a higher number of concepts for
faithful reconstruction. In terms of topic coherence, we could observe from
Fig. 15 that the topic coherence scores usually oscillate, but mostly display
a generally upward trend of becoming more coherent as the number of concepts
increases.

##  Appendix H Classification models used for text experiments

###  H.1 Transformer classification model trained from scratch

The self-trained transformer model used during text experiments follows a
simple structure: the input text is truncated to max length 512 and passed to
an embedding layer of dimension 200. Then, the embeddings are passed through a
positional encoding layer with dropout rate 0.2. Then, 6 transformer layers
follow with a hidden dimension of 200 and 2 heads. Finally, we mean pool the
transformed embeddings and pass through a linear classifier head. The linear
outputs are activated with a Sigmoid function to produce class probabilities.

To train the transformer model, we use either the IMDB or AG-News dataset. We
train for 10 epochs with a batch size of 128 and an Adam optimizer with
learning rate $3e-4$. We also use a learning rate step scheduler with step
size 1 and gamma of 0.95.

Dataset | AG-News | IMDB
---|---|---
LR | $5e-5$ | $3e-4$
train BS | 8 | 8
eval. BS | 16 | 16
seed | 42 | 42
optimizer | Adam | Adam
| $\text{betas}=(0.9,0.999)$ | $\text{betas}=(0.9,0.999)$
| $\text{epsilon}=1e-8$ | $\text{epsilon}=1e-8$
LR scheduler | linear | linear
warmup steps | 7425 | 1546
training steps | 74250 | 15468

Table 8:  Hyperparameters for finetuning BERT model.

###  H.2 Pretrained and finetuned BERT model

For AG-News, we take the finetuned version of bert-base-uncased model on
huggingface: “fabriceyhc/bert-base-uncased-ag_news”. For IMDB, we finetuned by
ourselves on the bert-base-uncased model. The hyperparameters used for both
finetuning are reported in Table 8, where LR stands for learning rate and BS
stands for batch size.

The huggingface code and models are all licensed under Apache 2.0, which
allows for redistribution and modification. Similarly, the codebase used for
replicating the visualization method (Chefer et al., 2021) and the baseline
method (Chen et al., 2018) are licensed under the MIT license, which allows
for redistribution of the code.