# Multi-Level Explanations for Generative Language Models

Lucas Monteiro Paes* 1††denotes equal contribution in alphabetical order.,
Dennis Wei* 2, Hyo Jin Do2, Hendrik Strobelt2,
Ronny Luss2, Amit Dhurandhar2, Manish Nagireddy2,
Karthikeyan Natesan Ramamurthy2, Prasanna Sattigeri2, Werner Geyer2, Soumya
Ghosh2
1Harvard University 2IBM Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>hjdo@, hendrik.strobelt@,
<EMAIL_ADDRESS><EMAIL_ADDRESS>manish.nagireddy@,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Perturbation-based explanation methods such as LIME and SHAP are commonly
applied to text classification. This work focuses on their extension to
_generative_ language models. To address the challenges of text as output and
long text inputs, we propose a general framework called MExGen that can be
instantiated with different attribution algorithms. To handle text output, we
introduce the notion of scalarizers for mapping text to real numbers and
investigate multiple possibilities. To handle long inputs, we take a multi-
level approach, proceeding from coarser levels of granularity to finer ones,
and focus on algorithms with linear scaling in model queries. We conduct a
systematic evaluation, both automated and human, of perturbation-based
attribution methods for summarization and context-grounded question answering.
The results show that our framework can provide more locally faithful
explanations of generated outputs.

##  1 Introduction

Perturbation-based explanation methods, such as LIME [22] and SHAP [14], are
popular for explaining individual predictions of black-box classification and
regression models. The explanations take the form of attribution scores
assigned to each part of the input, quantifying the effect of that part on the
prediction when perturbed.111We use the broader term “input attribution” in
this paper rather than “feature attribution” because “features” may no longer
be the best term for input parts such as text spans. For text input, the input
parts are spans of text (often words), and applications to text classification
tasks such as sentiment analysis have been well-studied [2, 13, 17, 12].

This work focuses on perturbation-based explanations for _generative_ language
tasks such as summarization and question answering. There has been little work
on such explanations for text generation as corroborated by a survey paper on
SHAP-based methods [17], the most relevant work being a collection of text
examples in the SHAP package documentation [25] and not a paper. Concurrent
with this work, the Captum library has added new capabilities for generative
language models [16].

Perturbation-based input attribution for generative tasks presents challenges
related to having text as output and long texts as input. We propose a general
framework called MExGen (Multi-level Explanations for Generative Language
Models) to address these challenges. The framework can be instantiated with
different attribution algorithms, and we do so using SHAP- and LIME-like
algorithms.

Figure 1: Example of using MExGen C-LIME (BERT scalarizer) to generate
explanations. The example shows how the algorithm captures copied text
(further skeletal remains) but also abstract concept mappings (M54
$\rightarrow$ motorway). (a)  Figure 2: Diagram showing the workflow of
MExGen.

#### Challenge of output text

The first challenge is that language models output text rather than a real
number (e.g., the predicted log probability of a class). Attribution
algorithms require a real-valued function to quantify that function’s
sensitivity to different inputs. We address this through the concept of
_scalarizers_ , functions that map output text to real numbers. We investigate
the use of multiple scalarizers and compare the resulting attributions.

Importantly, most scalarizers that we consider address the truly “black-box”
setting in which we receive only text as output from the model. This is common
with large language models (LLM) that only provide API access or are
proprietary. In this case, scalarizers can use only the output text from the
model being explained. The text-only setting is not handled by Captum and is
only briefly alluded to in SHAP [25].

#### Challenge of input length

The second challenge is that the input text for generative tasks can be long,
particularly for summarization and context-grounded question answering. Longer
inputs require more resources, in terms of input perturbations and model
queries, and impose financial costs for LLMs with API-only access. Long inputs
also pose interpretation issues, since it is not clear what level of
granularity in the attributions is most useful to a user. We address the
challenge in three ways:

Linguistic segmentation. We segment the input text into linguistic units at
multiple levels, for example sentences, phrases, and words, taking advantage
of linguistic structure present in the text.

Multi-level attributions. We use a refinement strategy that proceeds from
attributing at a coarser level like sentences to a finer level like phrases or
words, with optional user input. This controls the number and granularity of
units, and limits the compute.

Linear-complexity algorithms. We instantiate our framework with attribution
methods whose number of perturbations and model queries scale linearly with
the number of units at any level (e.g., number of sentences). In the SHAP
category, we choose Local Shapley (L-Shapley) [4]. For LIME-type methods, we
propose a linear-complexity variant that also limits the number of units
perturbed at one time.

We evaluate the instantiations of MExGen on the tasks of summarization and
context-based question answering. The automated evaluation shows that
instantiations of MExGen are more locally faithful; particularly, they assign
the highest importance to the input parts that are more relevant to the model
output with higher frequency than the baselines. Automated evaluations also
show that different scalarizers may yield similar explanations depending on
the model being explained and the task being performed. Specifically, we found
that some scalarizers — which only use API access to the model — generate
explanations that closely resemble explanations using the log probabilities of
the model being explained. Human evaluation results not only corroborated the
automated evaluation findings but also provided additional insights into how
certain scalarizers or attribution methods, previously considered similar to
other methods in automatic evaluation, were perceived as more faithful by
users.

Below is a summary of our main contributions:

  * •

We propose the MExGen framework to extend perturbation-based input attribution
to generative language models, with a multi-level strategy to combat the
challenges of long inputs.

  * •

We investigate several scalarizers for mapping output text to real numbers,
notably handling the case of text-only output from the model.

  * •

We conduct a systematic evaluation, both automated and human, of input
attribution methods for summarization and question answering, showing that
MExGen can provide more locally faithful explanations of generated output than
the alternative methods available.

##  2 Related Work

#### Post Hoc Explanations for LLMs

We focus on perturbation-based post hoc explanation methods for LLMs, notably
those with only API access where text is the only available output.

Gradient-based methods provide input attribution explanations [27, 29, 26, 1],
but they require access to model gradients with respect to the input.

While there is a significant literature on explaining text classification
models, the literature on perturbation-based explanations for natural language
generation is scarce, as noted in a survey of SHAP-based methods [17]. Below
we discuss the two most relevant works that we have found. One common trait is
that they require model logits (possibly from a proxy model) to generate
explanations.

PartitionSHAP, the default algorithm for text in the SHAP library, handles
long inputs by partitioning them into clusters of tokens and attributing the
same score to tokens within a cluster. We have not found a paper describing
PartitionSHAP. The description of the SHAP library’s text examples [25] is
also cursory. However, it is apparent that PartitionSHAP produces one set of
attributions for each token in the output. This approach is less interpretable
because it requires the selection of an output token and assigns multiple
attribution scores to each input span. PartitionSHAP can work with API-only
access [24], but understanding how it does so required significant
investigation on our part (see Appendix A.1).

CaptumLIME is our name for a modification of LIME [22] tailored for text
generation tasks using new features in the Captum library [16].222Captum also
has variants of SHAP but we found them slow to run and obviated by the
availability of PartitionSHAP. CaptumLIME allows the user to manually define
units for attribution within the input. It handles sequence outputs by
computing log probabilities for tokens in a given output and then summing.
This does, however, depend on access to output probabilities, so CaptumLIME is
not suitable for the text-only setting.

Method | LT Output | LT Input | API Access
---|---|---|---
LIME | ✗ | ✗ | ✗
SHAP | ✗ | ✗ | ✗
HEDGE | ✗ | ✓ | ✗
P-SHAP | ⚫ | ✓ | ✓
Captum | ✓ | ✗ | ✗
MExGen | ✓ | ✓ | ✓
Table 1:  Comparing the features (Long Text Output/Input, API access) of our
MExGen framework with existing methods.

#### Hierarchical Explanations

A line of work [28, 11, 2, 12] has developed hierarchical explanations for
sequence models, including LLMs, which can reveal compositional interactions
between words and phrases. In particular, the HEDGE algorithm of Chen et al.
[2] was identified by Mosca et al. [17] as “arguably the most suitable choice”
for NLP input attribution, in part because it builds its hierarchy in a top-
down, divisive fashion (as opposed to bottom-up agglomeration [28, 12]), which
is more practical for long texts. However, HEDGE is specific to classification
because it measures feature importance based on classification margin. Table 1
shows that PartitionSHAP, Captum, and HEDGE each lack capabilities compared to
MExGen.

#### Natural Language Self-Explanations

We also note works have utilized the generative model itself to provide
explanations in line with subsequent outputs, referred to as chain-of-thought
or CoT [33]. These methods have been shown to be both unstable [32] and highly
variable in quality [18]. Further, CoT provides self-explanations in natural
language, and not input attribution explanations.

##  3 Multi-Level Explanations for Generative Language Models

We describe the proposed MExGen framework for input attribution for generative
language models. Figure 2 provides an overview.

In the setting of perturbation-based input attribution, we are given a
generative LLM $f$, an input text sequence of interest $x^{o}$ (left side of
Figure 2, superscript $o$ for “original”), and a generated output
$y^{o}=f(x^{o})$ that is also a text sequence (top middle of Figure 2). Our
goal is to explain the output $y^{o}$ by attributing to parts of the input
$x^{o}$. Each part of the input, denoted $x_{s}$, $s=1,\dots,d$, is to be
assigned an attribution score $\xi_{s}$ (represented by color on the right of
Figure 2) quantifying the importance of $x_{s}$ in generating the output, in
the sense that if important parts are perturbed, then the output will change
significantly. As the second through fourth paths in Figure 2 indicate, model
$f$ can be queried on perturbations $x$ of $x^{o}$, with no further access.

Generative language tasks pose two main challenges: having text as output, and
potentially long text as input. The following two subsections discuss our
solutions to these challenges.

###  3.1 Handling Text Outputs

Input attribution methods require a numerical target function as the object to
explain. Since an LLM $f$ may only output text, we introduce scalarizers,
which are functions $S$ that map output text back to real numbers (shown as
blue boxes in Figure 2). We consider two types of access to $f$: (a) _full
logit access_ , where $f$ can provide predicted logits for all tokens in its
vocabulary, at each position in the output sequence, (b) _text-only_ setting,
where we are limited to text outputs. See Appendix A.1 for why vocabulary-wide
access is assumed for (a).

#### Full logit access

When all logits are available, we use the probability of generating the
original output sequence $y^{o}$ as the function to explain. We refer to this
as the Log Prob scalarizer. Given output sequence $y^{o}$ of length $\ell$ and
an arbitrary input sequence $x$, we compute from the model’s logits the log
probability of generating each original output token $y^{o}_{t}$ conditioned
on $x$ and previous output tokens $y^{o}_{<t}$. We then average over the
output sequence to obtain the scalarized output for $x$,

$S(x;y^{o},f)=\frac{1}{\ell}\sum_{t=1}^{\ell}\log
p\left(y^{o}_{t}\>|\>y^{o}_{<t},x;f\right).$ (1)

Here the scalarizer $S$ is parameterized by $y^{o}$ and $f$ since the latter
is providing predicted probabilities.

The Log Prob scalarizer generalizes the log probability used in explaining
text classification. This is seen from (1) by setting $\ell=1$ (single
prediction) and identifying $y^{o}_{1}$ with the predicted class.

#### Text-only access

Using the output sequence $y=f(x)$ generated from the input $x$, we consider
similarity measures $S(y;y^{o})$ between $y$ and the original output $y^{o}$
as scalarizers. These now depend on $f$ only via composition, i.e.,
$S(f(x);y^{o})$, and are not parameterized by $f$ as in (1).

  * •

Sim: Computes the cosine similarity between embedding vectors for $y$ and
$y^{o}$.

  * •

BERT: Computes the BERTScore [35] between $y$ and $y^{o}$.

  * •

BART: Computes the “faithfulness” version of the BARTScore between $y$ and
$y^{o}$ [34]. The formula for BARTScore is the same as (1) except with $y$ in
place of $x$ and an auxiliary model $f_{\textit{BART}}$ in place of the model
$f$ being explained. It thus measures the probability of $f_{\textit{BART}}$
generating $y^{o}$ when given $y$ as input.

  * •

SUMM: This scalarizer was added based on our research into the summarization
example [24] in the SHAP library, specifically the API case. As our
experiments show, it is very similar to the BART scalarizer with
$f_{\textit{BART}}$ as a summarization model.

  * •

Log NLI: Uses a natural language inference (NLI) model to predict the log-odds
of entailment given $y^{o}$ as premise and $y$ as hypothesis, and optionally
in the other direction as well (see Appendix A.1).

###  3.2 Handling Long Text Inputs

Some generative language tasks require long input texts, such as in
summarization and context-grounded QA. We address this challenge through a
combination of three techniques: segmenting the input into linguistic units at
multiple levels, using attribution algorithms with linear complexity in the
number of input units, and obtaining attributions to units in a coarse-to-fine
manner.

#### Linguistic segmentation

We segment the input text into linguistic units at multiple granularities:
paragraphs, sentences, phrases, and words (“Select Units” box in Figure 2).
This approach takes advantage of linguistic and other structure present in the
input. For example, the input may already be broken into paragraphs or contain
multiple distinct retrieved documents, in which case these paragraphs or
documents can form the units at the highest level. In contrast, many existing
methods rely on the model’s tokenizer, which can yield units (tokens) that are
too fine, or treat the text as a flat sequence of tokens and let the algorithm
decide how to segment it [2, 25].

We use tokenization and dependency parsing from spaCy v3.6 [9] to segment
paragraphs into sentences and words. To segment sentences into phrases, we
propose an algorithm that uses the dependency parse tree from spaCy. In the
first pass, the algorithm recursively segments the tree and its subtrees into
phrases that are no longer than a maximum phrase length. In the second pass,
some short phrases are re-merged. More details are in Appendix A.2.

Our framework allows for units at any level to be marked as not of interest
for attribution. For example at the word or token level, punctuation and stop
words may be ignored. In question answering, prompts for LLMs often follow a
template, e.g. prefacing the context with the string “Context: ”. Such
elements of the template are usually not of interest. We may also not
attribute to the question and focus only on the context.

#### Linear-complexity algorithms

Given an input segmentation (which may have mixed units as in Figure 2), we
have the actual task of attributing to units $x_{1},\dots,x_{d}$ (the “Post
Hoc Explainer” block in Figure 2). For this we consider only attribution
algorithms that scale linearly with the number of units $d$ in terms of model
queries, to control this cost. We instantiate MExGen with three such
algorithms: Local Shapley (L-SHAP), a LIME-like algorithm with further
constraints (C-LIME), and leave-one-out (LOO) as a baseline.

L-SHAP: This _local_ variation of SHAP was proposed by Chen et al. [4] to
decrease the number of model inferences relative to SHAP, which requires
exponentially many inferences. In our context, L-SHAP does so by only
perturbing units that are within a constant-size neighborhood of the current
unit being attributed to. This makes the number of inferences scale linearly
with the number of units. More precisely, for unit of interest $s\in[d]$, we
consider only the radius-$M$ neighborhood $s-M,\dots,s+M$ (truncated to
$1,\dots,d$ if necessary), and define $\Omega^{M,K}_{s}$ to be the set of all
subsets of $s-M,\dots,s+M$ up to cardinality $K$. Then the attribution score
$\xi_{s}$ for unit $s$ is given by

$\displaystyle\xi_{s}$
$\displaystyle=\sum_{A\in\Omega^{M,K}_{s}}\frac{S(x^{(A)};y^{o},f)-S(x^{(A\cup\\{s\\})};y^{o},f)}{Z_{|A|}},$
(2)

where $x^{(A)}$ is a perturbation of $x^{o}$ in which units $j\in A$ are
replaced (e.g., by a baseline value such as [MASK] or using a LM), and
$Z_{|A|}$ is the number of subsets in $\Omega^{M,K}_{i}$ with cardinality
$|A|$.

C-LIME: Similar to LIME, we use a linear model that operates on interpretable
features $z$ and approximates the model $f$ in the vicinity of original input
$x^{o}$. In our case, the interpretable features $z\in\mathbb{R}^{d}$
correspond to the units $x_{1},\dots,x_{d}$, with $z_{s}=0$ if unit $s$ is
perturbed and $z_{s}=1$ otherwise. The linear model is fit using perturbations
$x^{(1)},\dots,x^{(n)}$ of $x^{o}$, with corresponding interpretable
representations $z^{(1)},\dots,z^{(n)}$, and model outputs (scalarized in our
case) $S(x^{(1)};y^{o},f),\dots,S(x^{(n)};y^{o},f)$:

$\xi=\operatorname*{arg\,min}_{w}\sum_{i=1}^{n}\pi\bigl{(}z^{(i)}\bigr{)}\bigl{(}w^{T}z^{(i)}-S(x^{(i)};y^{o},f)\bigl{)}^{2}+\lambda
R(w),$ (3)

where $\pi(z)$ is a sample weighting function and $R$ is a regularizer. The
best-fit linear model coefficients yield the attribution scores $\xi$.

We make two main departures from LIME, in addition to the use of a general
scalarizer $S$ as seen in (3). Firstly, we limit the number of perturbations
$n$ to a multiple of the number of units $d$. Since the regression parameters
$w$ are $d$-dimensional (not including a possible intercept), the multiplier
could be set to $10$ for example to have $10$ times as many samples (i.e.,
perturbations) as parameters to fit. In contrast, LIME by default sets $n$ to
be in the thousands independently of $d$, which can be prohibitive for LLMs.
Secondly, we limit the number of units $K$ that can be perturbed
simultaneously to a small integer. This concentrates the smaller number of
perturbations on inputs that are closer to $x^{o}$, which has been shown to
improve the local fidelity of attributions [30]. LIME by contrast samples the
number of units to perturb uniformly from $\\{1,\dots,d\\}$ [15]. Other
details are in Appendix A.3.

LOO: We also consider a LOO variant of MExGen as a baseline. Here, units
$x_{1},\dots,x_{d}$ are perturbed one at a time to yield perturbations
$x^{(1)},\dots,x^{(d)}$ (i.e., $n=d$). The attribution score for $x_{s}$ is
the corresponding decrease in scalarizer score from its original value,
$\xi_{s}=S(x^{o};y^{o},f)-S(x^{(s)};y^{o},f)$.

#### Multi-level explanations

Once we have computed attribution scores at a given level, we may choose to
refine the input units and repeat the process (feedback path at the bottom of
Figure 2). For example, given sentence-level attributions for a long input, we
may wish to obtain phrase- or word-level attributions for a few sentences and
keep attributions at the sentence level for the remainder to not introduce too
many new units. The few sentences can be selected by any number of automatic
criteria or by the user, or the user can even choose to stop at sentence
level. See Appendix B for sentence selection criteria used in our experiments.

##  4 Automated Evaluation

We evaluate MExGen across two text generation tasks: Context-grounded QA and
summarization.

###  4.1 Setup

#### Datasets

For summarization, we use two popular datasets, Extreme Summarization (XSUM)
[19] and CNN/Daily Mail (CNN/DM) [23, 8]. For question answering, we use the
Stanford Question Answering Dataset (SQuAD) [21] because of its relatively
short, paragraph-long contexts, which allow us to test mixed sentence-and-
word-level attributions.

#### Models

For summarization, we use two models, a 306M-parameter distilbart-xsum-12-6
model333https://huggingface.co/sshleifer/distilbart-xsum-12-6, and a larger
20B-parameter Flan-UL2 model [31]. We treat the DistilBart model as one with
full logit access, thus enabling use of the Log Prob scalarizer. We call Flan-
UL2 through an API.444We use Flan-UL2 via the IBM Generative AI Python SDK
(Tech Preview) [10]. For QA, we use the 770M-parameter Flan-T5-Large
model555https://huggingface.co/google/flan-t5-large [5] and treat it as
providing output logits.

#### Attribution algorithms

We instantiate MExGen with the attribution algorithms discussed in Section
3.2: L-SHAP, C-LIME, and LOO, where the third is intended more as a baseline.
For summarization, we obtain mixed sentence and phrase attributions while for
QA, we obtain mixed sentence and word attributions. Algorithm parameter
settings are given in Appendix B. We compare against PartitionSHAP (P-SHAP)
[25] and CaptumLIME [16]. For PartitionSHAP, recall from Section 2 that it
requires the selection of an output token to explain, whereas we are
interested in the output sequence as a whole. For this reason, we slightly
modify PartitionSHAP by summing across all attribution scores, corresponding
to different output tokens, that it gives to each input span. This is
equivalent to explaining the sum of the log probabilities of output tokens
because of the linearity of Shapley values, and it corresponds to our Log Prob
scalarizer. For CaptumLIME, since it accepts user-defined units for
attribution, we provide it with the same sentence-level and mixed-level units
used by our algorithms. Thus we can directly compare Captum’s attribution
algorithm (i.e., LIME) with ours, controlling for input segmentation. Captum’s
default for the target function to explain also corresponds to the Log Prob
scalarizer. Additionally, PartitionSHAP and CaptumLIME take the number of
model queries as an input; for a fair comparison, we allow them the greater of
the numbers of model calls used by MExGen L-SHAP and C-LIME.

(a) MExGen C-LIME (b) MExGen L-SHAP

Figure 3:  Spearman’s rank correlation between attribution scores using
different scalarizers. Attributions were computed using multi-level (a) MExGen
C-LIME for the DistilBart model on the XSUM dataset and (b) MExGen L-SHAP for
Flan-T5-Large on the SQuAD dataset.

#### Metrics

To measure local fidelity of explanations, we use perturbation curves as in
previous work on explaining text classification [2, 12]. The general idea
behind perturbation curves is to rank input units in decreasing importance
according to a set of attribution scores, perturb the top $k$ units with $k$
increasing, and plot the resulting change in some output score that measures
how much the perturbed output deviates from the original output. For the
output score, we select an _evaluation scalarizer_ from the set of scalarizers
in Section 3.1. To accommodate PartitionSHAP and CaptumLIME, we choose an
evaluation scalarizer corresponding to the target function that they use for
explanation. For the DistilBart summarization and Flan-T5-Large QA models,
this means the Log Prob scalarizer, while for Flan-UL2, we choose SUMM in
keeping with PartitionSHAP. Appendix C has further details on how we compute
perturbation curves, accounting for the different units used by different
attribution algorithms.

###  4.2 Scalarizer Evaluation

#### Unit Ranking Similarity Across Scalarizers

We evaluate the similarity between attribution score vectors $\xi_{S}$ and
$\xi_{S^{\prime}}$ at the same granularity level for all pairs of scalarizers
$S,S^{\prime}$. We present Spearman rank correlation to measure similarity in
unit rankings. See Appendix C for further results.

Figure 3 shows Spearman correlation matrices, averaged across examples from
the respective datasets. Certain pairs are highly similar, for example BART
and SUMM as mentioned in Section 3.1, as well as Sim and BERT. Log Prob, the
only one that uses logits from the model being explained, clearly differs from
the others. There are also noticeable differences between panels (a) and (b)
in Figure 3, which feature different tasks, datasets, and models. For example,
Spearman correlation between BERT and SUMM is $0.75$ in (a) and $0.83$ in (b).
This suggests the necessity of exploring different scalarizers to explain
different tasks.

(a) BERT Score as Evaluation (b) Log Prob as Evaluation (c) BART Score as
Evaluation

Figure 4:  Perturbation curves for MExGen C-LIME with different scalarizers,
used to explain the DistilBart model on the XSUM dataset. The curves show the
decrease in (a) BERTScore, (b) log probability, and (c) BARTScore when
removing the most important $p$% of tokens according to each explanation
scalarizer. Shading shows standard error.

#### Perturbation Curves Across Scalarizers

Figure 4 shows perturbation curves with different evaluation scalarizers for
MExGen C-LIME using different scalarizers to compute attribution scores (i.e.,
a cross-evaluation of scalarizers). Results are again averaged over examples
from the datasets (see Appendix C for how). In panels (b) and (c), the case
where the attribution scalarizer is matched with the evaluation scalarizer is
generally the best. Additionally, the Log Prob scalarizer performs well across
all three evaluation scalarizers. This implies that the Log Prob scalarizer
gives the most universal (least dependent on the evaluation scalarizer)
explanation; hence, when log probabilities are available, we suggest using
them.

In the case where model logits are not available (e.g., when only API access
is given), the choice of scalarizer is not clear. However, if (hypothetically)
one were to use Log Prob as the evaluation scalarizer, our results in Figures
3 and 4(b) indicate that BERT score is the scalarization method that best
approximates explanations with Log Prob as the scalarizer. We include results
for other tasks, models, and evaluation scalarizers in Appendix C.

###  4.3 Comparison Between Explainers

(a) Flan-UL2 on CNN/DM (b) DistilBart on XSUM (c) Flan-T5-Large on SQuAD

Figure 5:  Perturbation curves for different explanation methods, using the
same scalarizer as the evaluation indicated on the y-axis. Models and
datasets: (a) Flan-UL2 on the CNN/DM dataset, (b) DistilBart on XSUM, and (c)
Flan-T5-Large on SQuAD. MExGen provides multi-level explanations down to
phrase-level for XSUM and CNN/DM and word-level for SQuAD. We also show the
perturbation curve of MExGen C-LIME using mismatched scalarizers: BARTScore
(a) and BERTScore (b)(c). Shading shows standard error in the mean.

#### Perturbation Curves

We compare the performance of the MExGen instantiations with P-SHAP and
CaptumLIME. We were only able to generate results for CaptumLIME in Figure 5
(b) because (i) we had API access to Flan-UL2 and CaptumLIME needs output
logits, and (ii) CaptumLIME does not support Flan-T5-Large. Figure 5 shows
that perturbing units (i.e., the tokens that they comprise) ranked as most
important by MExGen has the largest influence on the evaluation scalarization,
compared to P-SHAP and CaptumLIME. This indicates that the methods using
MExGen identified more important tokens according to the evaluation
scalarizer, while P-SHAP and CaptumLIME give higher scores for less important
tokens.

We found that MExGen C-LIME consistently outperforms the rest of the methods
across all tested models and datasets. In particular, the curve for MExGen
C-LIME is always higher than that of P-SHAP up to at least the top 10% of
tokens; the gray vertical lines in Figure 5 show the fraction of tokens where
the cross-over occurs. We also show in Figure 5 the perturbation curve for
MExGen C-LIME using a different scalarizer than used for evaluation.
Surprisingly, we find that even when using a mismatched scalarizer (notably
BERT in (b)(c), which does not even use logits), MExGen C-LIME can outperform
P-SHAP in local fidelity.

Datasets | Models | MExGen C-LIME | MExGen L-SHAP | MExGen LOO | P-SHAP
---|---|---|---|---|---
XSUM |  DistilBart
| 14.7 | 14.7 | 14.1 | 10.7
| Flan-UL2 | 21.3 | 21.2 | 18.4 | 20.0
CNN/DM |  DistilBart
| 13.5 | 14.7 | 13.2 | 9.7
| Flan-UL2 | 31.7 | 30.8 | 28.3 | 30.7
SQUaD | Flan-T5-Large | 61.2 | 59.9 | 59.9 | 59.4
Table 2:  Area under the perturbation curve (AUPC, higher is better) up to
$20$% of tokens. For DistilBart and Flan-T5-Large, log probability is used as
both the explanation and evaluation scalarizer. For Flan-UL2, which does not
provide access to log probabilites, SUMM is used as the explanation and
evaluation scalarizer.

#### Area Under the Perturbation Curve

Table 2 shows the area under the perturbation curve (AUPC) to summarize
performance over all dataset-model pairs that we tested, not just those in
Figure 5. We evaluate AUPC up to $p=20\%$ as done in Chen et al. [2]. Across
all dataset-model pairs, methods that use the MExGen framework (sometimes
including the baseline LOO) performed better (i.e., higher AUPC) than P-SHAP.
Moreover, the second highest AUPC is always a MExGen method.

##  5 User Study

We conducted a user study to understand how humans perceive explanations
provided by different scalarizers and attribution methods, and whether they
can discern performance differences akin to the quantitative evaluations in
Section 4. To ease interpretation, we developed a visualization tool that
highlights input text spans based on a color-coded scale for the attribution
scores. We addressed the following research questions for summarization:

  1. 1.

Fidelity: Which method is perceived to be better at explaining how the
language model generated the summary?

  2. 2.

Preference: Which method do people prefer?

  3. 3.

Concentration: What are people’s perceptions regarding the concentration of
attribution scores (i.e., do the darker colors cover less of the text)?

  4. 4.

Granularity: Which granularity level (sentence-level vs. multi-level) do
people prefer?

We selected ten examples from the XSUM dataset for the user study with
diversity in topics, while ensuring that they do not contain sensitive issues
or obvious errors. We designed an online survey consisting of three parts.
Each part showed an input text, randomly drawn from the ten examples, and its
summary, generated by the DistilBart model. This user study focused on
algorithms and scalarizers that are of most interest based on the automated
evaluation in Section 4. Specifically, we compared two scalarizers (Log Prob,
BERT) in Part 1, three attribution algorithms (C-LIME, L-SHAP, PartitionSHAP)
in Part 2, and two levels of granularity (sentence-level, multi-level) in Part
3. The presentation order of the attribution algorithms was randomized to
mitigate order effects. The survey consisted of seven pairwise comparisons in
total followed by questions for the participants.

We recruited participants from a large technology company who self-identify as
machine learning practitioners using language models and collected data from
88 of them after filtering. Here, we report a summary of key results only. See
Appendix D for details including survey questions, analysis, and statistical
results.

Scalarizers. Significantly more participants perceived BERT to be higher in
fidelity than Log Prob. They also preferred BERT over Log Prob. Participants
perceived that the attribution concentration was adequate in both BERT and Log
Prob.

Attribution methods. Significantly more participants perceived C-LIME to be
higher in fidelity than L-SHAP. They also preferred C-LIME over L-SHAP.
Participants perceived that the attribution concentration was adequate in all
three methods.

Granularity. Regarding user preference between sentence-level and multi-level
granularities, no statistically significant difference was found.

##  6 Concluding Remarks

We proposed MExGen, a framework to provide explanations for generative
language models. Our experiments show that instances of MExGen provide
faithful explanations and outperform the baselines we compare against —
PartitionSHAP and CaptumLIME. Our numerical results also indicate that C-LIME
outperforms the other explanation methods in the MExGen framework and
PartitionSHAP. The user study results align with the automated evaluation
overall, and additionally reveal that people perceive the BERT scalarizer as
more locally faithful than the Log Prob scalarizer, which was not seen in the
automated evaluation. This result implies that in some cases, there may be no
loss in having text-only access compared to full logit access.

## Limitations

We see three main limitations in our work. First, MExGen is a framework for
post hoc explanations. Although providing such explanations can be helpful for
practitioners to understand the model behavior, it does not fully characterize
how models make predictions and only provides local explanations. Second, our
automatic evaluations use three models (DistilBart, Flan-UL2, and
Flan-T5-Large) and three datasets (XSUM, CNN/DM, and SQUaD). Although our
findings are consistent across the models and datasets, the results reported
in Section 4 could still change in different experimental settings. Third, our
user study analyzes the _perception_ of participants of how well a method
explains the predictions of a given model, and not necessarily the fidelity of
the explanation itself — fidelity is measured more directly in the automated
evaluation. However, we believe the fact that the participant pool was
composed of people with experience in ML and LLMs improves the quality of
evaluation of the explanation methods. Finally, with post hoc explanations,
there is the risk that the explanation algorithm could be steered to obfuscate
undesirable behavior from the model.

## Ethics Statement

MExGen is a framework to explain generative language models. Hence, its
objective is to elucidate how a model made a specific prediction. Methods that
aim to understand how black-box models generate their output are essential for
guaranteeing transparency during decision-making. For example, a generative
language model can be used to summarize dialog and create minutes of meetings
that can later be used to perform high-stakes decisions. Then, it is necessary
to understand how the model generated the summary and ensure that the output
content is based on the input dialog. Therefore, for such high-stakes
applications, methods that can provide explanations for text generated by
language models are necessary, highlighting the importance of MExGen.

## Acknowledgements

The authors thank the following people: Inkit Padhi for suggestions on parsing
sentences into phrases, using language models for perturbing words, and
participation in multiple discussions; Ella Rabinovich and Samuel Ackerman for
discussions on and recommendations of similarity measures as scalarizers, and
Ella in particular for also providing suggestions on parsing sentences into
phrases and using language models for perturbation; Pratap Kishore Varma
Vemulamanda for helping to investigate the API-only case of SHAP’s
summarization example [24]; Subhajit Chaudhury for suggestions of similarity
measures as scalarizers; Keshav Ramji for discussion on related work and
quality evaluation for the explanations; Kush Varshney and Eitan Farchi for
general encouragement and support.

## References

  * Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140, 2015.
  * Chen et al. [2020] Hanjie Chen, Guangtao Zheng, and Yangfeng Ji.  Generating hierarchical explanations on text classification via feature interaction detection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5578–5593, July 2020.  doi: 10.18653/v1/2020.acl-main.494.  URL https://aclanthology.org/2020.acl-main.494.
  * Chen and Jordan [2020] Jianbo Chen and Michael Jordan.  LS-Tree: Model interpretation when the data are linguistic.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(04):3454–3461, Apr. 2020.  doi: 10.1609/aaai.v34i04.5749.  URL https://ojs.aaai.org/index.php/AAAI/article/view/5749.
  * Chen et al. [2019] Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan.  L-Shapley and C-Shapley: Efficient model interpretation for structured data.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=S1E3Ko09F7.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models, 2022.
  * Firth [2005] David Firth.  Bradley-Terry models in R.  _Journal of Statistical software_ , 12:1–12, 2005.
  * He et al. [2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=XPZIaotutsD.
  * Hermann et al. [2015] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  In _NIPS_ , pages 1693–1701, 2015.  URL http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.
  * Honnibal et al. [2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd.  spaCy: Industrial-strength natural language processing in python.  2020\.  doi: 10.5281/zenodo.1212303.
  * IBM [2024] IBM.  IBM Generative AI Python SDK (Tech Preview).  https://github.com/IBM/ibm-generative-ai, 2024.  Accessed: 2024-02-01.
  * Jin et al. [2020] Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren.  Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=BkxRRkSKwr.
  * Ju et al. [2023] Yiming Ju, Yuanzhe Zhang, Kang Liu, and Jun Zhao.  A hierarchical explanation generation method based on feature interaction detection.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 12600–12611, July 2023.  doi: 10.18653/v1/2023.findings-acl.798.  URL https://aclanthology.org/2023.findings-acl.798.
  * Kim et al. [2020] Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon.  Interpretation of NLP models through input marginalization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3154–3167, November 2020.  doi: 10.18653/v1/2020.emnlp-main.255.  URL https://aclanthology.org/2020.emnlp-main.255.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc., 2017.  URL http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf.
  * Mardaoui and Garreau [2021] Dina Mardaoui and Damien Garreau.  An analysis of LIME for text data.  In Arindam Banerjee and Kenji Fukumizu, editors, _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ , volume 130 of _Proceedings of Machine Learning Research_ , pages 3493–3501. PMLR, 13–15 Apr 2021.  URL https://proceedings.mlr.press/v130/mardaoui21a.html.
  * Miglani et al. [2023] Vivek Miglani, Aobo Yang, Aram H. Markosyan, Diego Garcia-Olano, and Narine Kokhlikyan.  Using Captum to explain generative language models, 2023.
  * Mosca et al. [2022] Edoardo Mosca, Ferenc Szigeti, Stella Tragianni, Daniel Gallagher, and Georg Groh.  SHAP-based explanation methods: A review for NLP interpretability.  In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4593–4603, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.406.
  * Nagireddy et al. [2024] Manish Nagireddy, Lamogha Chiazor, Moninder Singh, and Ioana Baldini.  SocialStigmaQA: A benchmark to uncover stigma amplification in generative language models.  In _Proceedings of the 2024 AAAI Conference on Artificial Intelligence_ , 2024.  URL https://arxiv.org/abs/2312.07492.
  * Narayan et al. [2018] Shashi Narayan, Shay B. Cohen, and Mirella Lapata.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1206.  URL https://aclanthology.org/D18-1206.
  * Pham et al. [2022] Thang Pham, Trung Bui, Long Mai, and Anh Nguyen.  Double trouble: How to not explain a text classifier’s decisions using counterfactuals synthesized by masked language models?  In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, editors, _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 12–31, Online only, November 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.aacl-main.2.
  * Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In Jian Su, Kevin Duh, and Xavier Carreras, editors, _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics.  doi: 10.18653/v1/D16-1264.  URL https://aclanthology.org/D16-1264.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "Why should I trust you?" Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144, 2016.
  * See et al. [2017] Abigail See, Peter J. Liu, and Christopher D. Manning.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1099.  URL https://www.aclweb.org/anthology/P17-1099.
  * SHAP [2024a] SHAP.  Text to text explanation: Abstractive summarization example.  https://shap.readthedocs.io/en/latest/example_notebooks/text_examples/summarization/Abstractive%20Summarization%20Explanation%20Demo.html, 2024a.  Accessed February 2024.
  * SHAP [2024b] SHAP.  Text examples.  https://shap.readthedocs.io/en/latest/text_examples.html, 2024b.  Accessed February 2024.
  * Shrikumar et al. [2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pages 3145–3153. PMLR, 2017.
  * Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Singh et al. [2019] Chandan Singh, W. James Murdoch, and Bin Yu.  Hierarchical interpretations for neural network predictions.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=SkEqro0ctQ.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR, 2017.
  * Tan et al. [2023] Zeren Tan, Yang Tian, and Jian Li.  GLIME: General, stable and local LIME explanation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=3FJaFElIVN.
  * Tay et al. [2023] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler.  UL2: unifying language learning paradigms.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023.  URL https://openreview.net/pdf?id=6ruVLB727MC.
  * Turpin et al. [2023] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=bzs4uPLXvi.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=_VjQlMeSB_J.
  * Yuan et al. [2021] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  BARTScore: Evaluating generated text as text generation.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=5Ya8PbvpZ9.
  * Zhang et al. [2020] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  BERTScore: Evaluating text generation with BERT.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SkeHuCVFDr.

##  Appendix A Further Details on MExGen

###  A.1 Scalarizers

#### Vocabulary-wide logits

The inputs $x$ in our case are perturbed versions of the original input
$x^{o}$. Some of these perturbations however may be significantly different
semantically and cause the probability of generating the original output
$y^{o}$ to decrease dramatically. For this reason, the Log Prob scalarizer may
require access to logits for improbable tokens conditioned on $x$, which can
be ensured if logits are available for the entire vocabulary, but not if only
the top $k$ logits are provided.

#### Aggregation for Log Prob scalarizer

As alternatives to the average over the output sequence in (1), other ways of
aggregating include using the sum or taking the product or geometric mean of
the probabilities. We choose the average to normalize for sequence length and
because log probabilities tend to be a more “linear” function of inputs than
probabilities.

#### BERTScore

BERTScore uses an LM to obtain contextual embeddings for the tokens in $y$ and
$y^{o}$, matches the two sets of embeddings based on cosine similarity, and
computes a score to quantify the degree of match.

####  SUMM scalarizer

We investigated the code behind the abstractive summarization example [24] in
the SHAP library, and specifically the TeacherForcing class that it uses in
the text-only API setting. Our understanding of the code is that it obtains
proxy log-odds for tokens in the original output $y^{o}$ by taking log-odds
from an auxiliary summarization model $f_{\textit{SUMM}}$, with input $y$ and
output set to $y^{o}$. If we then average the log-odds over the tokens in
$y^{o}$ (similar to (1)), the result is similar to the BART scalarizer with
$f_{\textit{BART}}=f_{\textit{SUMM}}$ (with a possible discrepancy between
log-odds versus log probabilities). Our experiments show that the BART and
SUMM scalarizers are indeed very similar.

####  Log NLI scalarizer

This scalarizer is based on the intuition that Log NLI entailment is a kind of
similarity. We use the NLI model to predict the log-odds of entailment given
$y^{o}$ as premise and $y$ as hypothesis, and optionally in the other
direction as well. If both directions are used, we take the geometric mean of
the two entailment probabilities and then convert back to log-odds.

###  A.2 Phrase Segmentation

Here we describe the phrase segmentation algorithm mentioned in Section 3.2.
The algorithm starts with the root token of the tree and checks whether each
child subtree of the root is shorter (in terms of tokens) than a maximum
phrase length parameter. If it is shorter, then the subtree constitutes a
phrase, and if it is not, the algorithm is recursively applied to the subtree.
The root of each (sub)tree is also taken to be a phrase. Once the sentence has
been recursively segmented into phrases in this manner, a second pass is
performed to re-merge some phrases that have become too fragmented, thus
controlling the number of phrases which is desirable for computation and
interpretation. Specifically, phrases that constitute noun chunks (as
identified by spaCy) are merged, and certain single-token phrases are merged
with their neighbors. Further notes:

  * •

Subtrees of the dependency parse tree are usually contiguous spans of text (in
English), but sometimes they correspond to multiple spans. In this case, we
treat each span as its own subtree since we wish to have contiguous phrases.

  * •

In measuring the token length of a span, we do not count punctuation or
spaces.

  * •

In merging phrases that fall within a noun chunk, we check conditions that are
consistent with noun chunks, for example that there is a root phrase that is a
single token (the noun), and that the other phrases are children of the root
phrase.

  * •

For merging single-token phrases with their neighbors, we use the following
criteria:

    * –

The single-token phrase (_singleton_) is a non-leaf phrase (is the parent of
other phrases) or a coordinating conjunction.

    * –

If the singleton is a coordinating conjunction (e.g. “and”), the neighbor is a
corresponding conjunct (e.g. “Bob” in “Alice and Bob”).

    * –

If the singleton is a preposition (e.g. “to”), the neighbor is a child of the
preposition (e.g. “the store” in “to the store”).

    * –

If the singleton is of some other type, the neighbor is a leaf phrase and is
either adjacent to the singleton or a singleton itself.

    * –

The merged phrases do not exceed the maximum phrase length parameter.

###  A.3 C-LIME

In addition to the departures from LIME described in Section 3.2, we discuss
two other aspects:

#### Sample weighting

Since we limit the number of units that are simultaneously perturbed to a
small integer $K$, we also no longer use LIME’s sample weighting scheme.
Instead, we give each subset cardinality $k=0,\dots,K$ the same total weight,
and then distribute this weight uniformly over the subsets of that
cardinality.

#### Regularization

Different regularizers $R(w)$ can be used in (3), e.g. $\ell_{2}$ or
$\ell_{1}$. In our experiments however, we do not regularize and compute a
fully dense solution. This allows all units to be ranked to facilitate
evaluation of perturbation curves.

##  Appendix B Algorithm Parameter Settings

This appendix documents the settings of key algorithm parameters and other
choices made in the experiments.

#### Scalarizer models

The text-only scalarizers presented in Section 3.1 can be instantiated with
different models. The ones used in our experiments are as follows:

  * •

“Sim”: We use the all-MiniLM-L6-v2 embedding model from the
SentenceTransformers package666this model can be found at
https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.

  * •

“BERT”: We use the Log NLI model deberta-v2-xxlarge-mnli777this model can be
found at https://huggingface.co/microsoft/deberta-v2-xxlarge-mnli [7] to
compute BERTScore, which is the same Log NLI model that we use for the Log NLI
scalarizer. Our initial reason for doing so was to see whether the two
scalarizers would be very similar because of this choice (they are not as they
operate on different principles). We take the F1-score output as the
BERTScore.

  * •

“BART” and “SUMM”: For “SUMM”, we follow SHAP [24] in using the same
distilbart-xsum-12-6 model as both the scalarizing summarization model as well
as the primary summarization model to explain. For “BART”, we also use
distilbart-xsum-12-6 to determine whether the “BART” and “SUMM” scalarizers
are very similar when instantiated with the same model. This is indeed the
case.

  * •

“Log NLI”: As mentioned above, we use deberta-v2-xxlarge-mnli as the Log NLI
model. We also choose to compute the Log NLI entailment probability in both
directions and take the geometric mean of the two before taking the logit.

All other parameters of these scalarizers are kept at default values.

#### Linguistic segmentation

The phrase segmentation algorithm has one main parameter, the maximum phrase
length, which we set to $10$ spaCy tokens (not counting spaces and
punctuation). We exclude non-alphanumeric units (generally punctuation) from
attribution. For QA, we allow only the context to be attributed to.

#### Multi-level explanations

The decision here is how many top sentences are refined into phrases in the
summarization experiments and into words in the QA experiments. For
summarization, this is determined by first normalizing the sentence-level
attribution scores to the range $[-1,1]$ and then choosing sentences with
absolute attribution scores above a threshold. The threshold is $1/3$ for
C-LIME and LOO with the distilbart-xsum-12-6 model, $0.3$ for L-SHAP with
distilbart-xsum-12-6, $0.5$ for C-LIME and LOO with the Flan-UL2 model, and
$0.4$ for L-SHAP with Flan-UL2. If more than three sentences have scores above
the threshold, we take only the top three. For QA, we simply take the top $1$
sentence as the context paragraphs in SQuAD tend to have only a handful of
sentences.

#### L-SHAP

For L-SHAP, the two main parameters are the local neighborhood radius $M$,
which we take to be $M=2$, and the maximum number of units $K$ perturbed at
one time, also set to $2$. Note that the latter does not include the unit of
interest, so altogether the maximum is $3$.

#### C-LIME

For C-LIME, the main parameters controlling the perturbations are the constant
of proportionality between the number of perturbations $n$ and number of units
$d$, and the maximum number of units $K$ perturbed at one time. The former is
set to $10$ and the latter to $3$, except for sentence-level attributions on
SQuAD where it is $2$.

#### PartitionSHAP

We set the number of model queries (parameter max_evals) to be approximately
equal to the number used by our L-SHAP and C-LIME algorithms. More
specifically, since obtaining mixed-level attributions with MExGen requires
first performing sentence-level attribution, we add the numbers of queries
used during sentence-level and mixed-level attribution. We then take the
larger of these two sums for L-SHAP and C-LIME as the number of queries
allowed for PartitionSHAP.

##  Appendix C Automated Evaluation Details and More Quantitative Results

#### Computing environment

Experiments were run on a computing cluster providing V100 and A100 GPUs with
32 GB of CPU memory and 32, 40, or occasionally 80 GB of GPU memory. One CPU
and one GPU were used at a time. The total computation time is estimated to be
on the order of 100–200 hours.

#### Datasets

For summarization, we limit our evaluation to the first 100 examples from the
test sets of XSUM and CNN/Daily Mail, due to the computation involved in
running multiple perturbation-based attribution algorithms that query
summarization models. For question answering on SQuAD, we evaluate 200
randomly sampled examples from the validation set.

#### Perturbation curves

MExGen can attribute to mixed units of different lengths in terms of the
number of tokens, and these units also differ from those produced by P-SHAP.
To account for these differences in computing perturbation curves, we consider
both the attribution score and the number of tokens for each unit. We rank
units in decreasing order of the unit attribution score divided by the number
of tokens (since we plot perturbation curves as functions of the number of
tokens perturbed, this ratio can be seen as the slope in the score-tokens
plane). We then perturb (more precisely remove) the top $k$ units according to
these ratios, compute the output score given by the evaluation scalarizer, and
increase $k$ until at least $50$% of the tokens have been removed. To average
perturbation curves over examples with different numbers of tokens, we divide
the numbers of tokens perturbed by the total number to obtain percentages. We
then linearly interpolate onto a common grid of percentages before averaging.

(a) (C-LIME, DistilBart, XSUM) (b) (L-SHAP, DistilBart, XSUM) (c) (C-LIME,
DistilBart, XSUM) (d) (L-SHAP, DistilBart, XSUM)

(e) (C-LIME, Flan-T5, SQUaD) (f) (L-SHAP, Flan-T5, SQUaD) (g) (C-LIME,
Flan-T5, SQUaD) (h) (L-SHAP, Flan-T5, SQUaD)

Figure 6: Spearman’s rank correlation and cosine similarity for different
explanation methods in varying datasets.

#### On the Cosine Similarity of Explanations

Figure 6 shows the cosine similarity between all the pairs of secalarizers we
use. We show the cosine similarity between the explanations of each method to
analyze how aligned the explanations are from different scalarizers. Each text
input $x=x_{1},...,x_{d}$ receives a multi-level explanation given by
$\xi_{S}(x)=(\xi_{S}(x)_{1},...,\xi_{S}(x)_{d})\in\mathbb{R}^{d}$ where each
$\xi_{S}(x)_{i}$ represents the contribution of unit $i$ to the model
prediction scalarization computed using the scalarizer $S$. We define the
cosine similarity between scalarizers $S$ and $S^{\prime}$ as the average of
the cosine similarities between the explanations for all available input
texts, i.e.,

$\texttt{CosSim}(S,S^{\prime})\triangleq\frac{1}{|X|}\sum_{x\in
X}\frac{\langle\xi_{S}(x),\xi_{S^{\prime}}(x)\rangle}{||\xi_{S}(x)||||\xi_{S^{\prime}}(x)||}.$

Figure 6 (a) shows the cosine similarities for the scalarizers used by MExGen
C-Lime to explain the predictions of the DistilBert model in the XSUM dataset.
Figure 6 (b) shows the cosine similarities for the scalarizers used by MExGen
L-SHAP to explain the predictions of the Flan-T5-Large model on the SQUaD
dataset.

Similarity Across Scalarizers. Figure 6 indicates that, although some
scalarizers lead to similar model explanations, there are occasions where
scalarizers are more dissimilar. Moreover, the similarities between
scalarizers not only depend on the scalarizer itself but also on the model and
dataset being explained. For example, Figure 6 (b) shows that when using
MExGen L-SHAP to provide explanations for the predictions in the XSUM dataset
using DistilBrat, the scalarizers BERT and SUMM are fairly similar
($\texttt{CosSim}(S_{\textit{BERT}},S_{\textit{SUMM}})=0.96$). On the other
hand, looking at the same pair of scalarizers but for MExGen LIME to provide
explanations to the predictions in the SQUaD dataset using Flan-T5-Large, BERT
and SUMM are more dissimilar
($\texttt{CosSim}(S_{\textit{BERT}},S_{\textit{SUMM}})=0.82$). This result
highlights the necessity of exploring different scalarizers to explain natural
language generation, taking into account the task being performed and the main
objective of the explanation, i.e., target scalarization.

Similarity to Logist. In the SHAP library, SUMM is proposed to provide
explanations to LLMs that do not provide access to the logits — hence, the
main objective of SUMM is to approximate the explanations for the logit when
it is not available. However, Figure 6 shows that SUMM is not always the best
scalarizer for approximating the explanations that would be given if logits
were available. For example, Figure 6 (a) shows that the similarity between
SUMM and logit is near $0.61$. In contrast, Figure 6 (b) shows that the
similarity between the explanations generated using SUMM as scalarizer has a
similarity of $0.78$ with the onex generated using logits.

We are also aware that only comparing the similarities between explanations
might not be enough; once, many researchers use the scores to compute the
ranking of the importance across all input features (input text units here).
For this reason, next, we compare Spearman’s rank correlation to measure the
rank stability across different scalarizers.

Similarity to Log Probability Scalarizer. In the SHAP library, SUMM is
proposed to provide explanations when access to logits is not available.
However, Figure 3 shows that SUMM is not always the best scalarizer for
approximating the explanations that would be given if logits were available.
For example, Figure 3 (a) shows the ranking correlation between the scalarizer
and the Log Prob scalarizer is higher for BERT score. Figure 3 (b) shows that
the rank generated by both the explanations generated using SUMM and BERT
scalarizer are equally similar to the rank of explanations using the Log Prob
scalarizer.

#### Perturbation curves for different combinations of scalarizers

Figures 7, 8, 9 show the perturbation curve for different scalarizations.

(a) BERT Score as Evaluation (b) Log Prob as Evaluation (c) BART Score as
Evaluation

Figure 7:  Perturbation curves for MExGen L-SHAP with different scalarizers,
used to explain the DistilBart model on the XSUM dataset. The curves show the
decrease in (a) BERTScore, (b) log probability, and (c) BARTScore when
removing the most important $p$% of tokens according to each explanation
scalarizer. Shading shows standard error.

(a) BERT Score as Evaluation (b) Log Prob as Evaluation (c) BART Score as
Evaluation

Figure 8:  Perturbation curves for MExGen C-LIME with different scalarizers,
used to explain the Flan-T5-Large model on the SQUaD dataset. The curves show
the decrease in (a) BERTScore, (b) log probability, and (c) BARTScore when
removing the most important $p$% of tokens according to each explanation
scalarizer. Shading shows standard error.

(a) BERT Score as Evaluation (b) Log Prob as Evaluation (c) BART Score as
Evaluation

Figure 9:  Perturbation curves for MExGen L-SHAP with different scalarizers,
used to explain the Flan-T5-Large model on the SQUaD dataset. The curves show
the decrease in (a) BERTScore, (b) log probability, and (c) BARTScore when
removing the most important $p$% of tokens according to each explanation
scalarizer. Shading shows standard error.

##  Appendix D User Study

###  D.1 Participants

We recruited 96 participants from a large technology company. Those who self-
identified as machine learning practitioners using language models were
eligible for the study. We filtered out 8 participants who did not pass
eligibility checks or did not provide valid responses, resulting in data from
88 participants for our analysis.

###  D.2 Scalarizers

Participants perceived BERT to be higher in fidelity than Log Prob. We ran a
binomial test and found that the selection of BERT was significantly higher
than the random chance ($p<.05$, 95% CI [.50, .65]). They also preferred BERT
over Log Prob and the choice was statistically significant ($p<.05$, 95% CI
[.56, .71]). The type of attribution methods (e.g., C-LIME, L-SHAP) did not
affect the results. Participants perceived that the attribution concentration
was adequate overall, as the average ratings (Log Prob: M=4.32, SD=1.65; BERT:
M=4.66, SD=1.44) were close to the median of 4 on the 7-point Likert scale. A
paired t-test revealed that the difference in the concentration perceptions
between scalarizers was not statistically significant.

Selected Option | C-LIME | L-SHAP
---|---|---
Log Prob | 35.2% | 34.1%
BERT | 54.5% | 60.2%
Identical | 10.2% | 5.7%
Table 3: The proportions of participants who selected one of the three options – Log Prob, BERT, or ‘They are identical’. Regardless of attribution methods, significantly more participants chose BERT over Log Prob when asked to select the one with higher perceived fidelity. Preferred Option | C-LIME | L-SHAP
---|---|---
Log Prob | 29.5% | 31.8%
BERT | 62.5% | 64.8%
Identical | 8% | 3.4%
Table 4: The proportions of participants who selected one of the three options
– Log Prob, BERT, or ‘They are identical’ based on their preference.
Regardless of attribution methods, significantly more participants preferred
BERT over Log Prob.

###  D.3 Attribution Methods

We fitted a Bradley-Terry model [6] for the outcome of pairwise comparisons
between attribution methods. The model computes an ‘ability’ estimate of each
method, yielding a complete ranking of methods. Regarding the perceived
fidelity, we found that there is a significant difference between C-LIME and
L-SHAP ($p<.05/3$ with Bonferroni adjustment), with C-LIME having the highest
ability and L-SHAP having the lowest ability. The preference data showed the
same pattern in which there is a significant difference between C-LIME and
L-SHAP ($p<.05/3$ with Bonferroni adjustment), with C-LIME having the highest
ability and L-SHAP having the lowest ability. Other pairs of methods were not
significantly different. Participants perceived that the attribution
concentration was adequate overall, as the average rating was close to the
median on the 7-point Likert scale (M=4.32, SD=1.48). A repeated ANOVA showed
that the differences in perceived concentration among the attribution methods
were not significant.

Selected vs. Rejected Options |  $p$-value
---|---
C-LIME vs. L-SHAP | 0.0107 **
PartitionSHAP vs. L-SHAP | 0.0455
C-LIME vs. PartitionSHAP | 0.5691
Table 5: There is a significant difference in perceived fidelity between C-LIME and L-SHAP. Significant p-values after Bonferroni adjustment are noted with ** (p<0.05/3). Selected vs. Rejected Options |  $p$-value
---|---
C-LIME vs. L-SHAP | 0.0074 **
PartitionSHAP vs. L-SHAP | 0.1758
C-LIME vs. PartitionSHAP | 0.1758
Table 6: There is a significant difference in preference between C-LIME and
L-SHAP. Significant p-values after Bonferroni adjustment are noted with **
(p<0.05/3).

###  D.4 Granularity Preference

Participants were asked to select their preferred granularity of attributions
(sentence-level vs. multi-level). While the number of participants who
preferred multi-level granularity (56.2%) was slightly higher than those who
preferred sentence-level granularity (43.8%), binomial tests indicated that
their granularity choice was not statistically significant. The preference for
granularity did not significantly vary across attribution algorithms (C-LIME,
L-SHAP).

###  D.5 Survey

Figure 10 and Figure 11 show some questions we asked in the survey.

Figure 10: Survey questions. Participants answered a series of questions
related to perceived fidelity and general preference comparing a pair of
attribution methods, followed by questions related to perceived concentrations
of each method. Figure 11: Granularity question. Participants were asked to
select their preferred granularity of attributions (sentence-level vs. multi-
level).

##  Appendix E Future Directions

#### Hierarchical explanations

It could be profitable in future work to incorporate the hierarchical
explanations discussed in Section 2 into the multi-level MExGen framework. The
method of Chen and Jordan [3] may be especially relevant since it leverages a
constituency parse tree to compute word-level importances, which may be
related to our use of dependency parse trees.

#### Word infilling with BERT

We have explored perturbing words by masking them and then calling a BERT
model to fill the masks with different words that fit within the sentence.
However, we have thus far not seen a quantifiable benefit to using BERT
compared to replacing with a fixed baseline value (such as an empty string).
Our experience is in line with the the mixed results reported by Pham et al.
[20] on using BERT in this manner.

#### Phrase segmentation

Segmentation of sentences into phrases could of course be done in ways other
than our dependency parsing algorithm, for example using constituency parsing
instead. A possible advantage of using dependency parsing is that each phrase
can be labeled with the dependency label of its root token and treated
differently on this basis.