# ReAGent: A Model-agnostic Feature Attribution Method for Generative Language
Models

Zhixue Zhao, Boxuan Shan

###### Abstract

Feature attribution methods (FAs), such as gradients and attention, are widely
employed approaches to derive the importance of all input features to the
model predictions. Existing work in natural language processing has mostly
focused on developing and testing FAs for encoder-only language models (LMs)
in classification tasks. However, it is unknown if it is faithful to use these
FAs for decoder-only models on text generation, due to the inherent
differences between model architectures and task settings respectively.
Moreover, previous work has demonstrated that there is no ‘one-wins-all’ FA
across models and tasks. This makes the selection of a FA computationally
expensive for large LMs since input importance derivation often requires
multiple forward and backward passes including gradient computations that
might be prohibitive even with access to large compute. To address these
issues, we present a model-agnostic FA for generative LMs called Recursive
Attribution Generator (ReAGent). Our method updates the token importance
distribution in a recursive manner. For each update, we compute the difference
in the probability distribution over the vocabulary for predicting the next
token between using the original input and using a modified version where a
part of the input is replaced with RoBERTa predictions. Our intuition is that
replacing an important token in the context should have resulted in a larger
change in the model’s confidence in predicting the token than replacing an
unimportant token. Our method can be universally applied to any generative LM
without accessing internal model weights or additional training and fine-
tuning, as most other FAs require. We extensively compare the faithfulness of
ReAGent with seven popular FAs across six decoder-only LMs of various sizes.
The results show that our method consistently provides more faithful token
importance distributions.111Our code: https://github.com/casszhao/ReAGent

##  1 Introduction

Feature attribution (FA) techniques are popular post-hoc methods to provide
explanations for model predictions. FAs generate token-level importance scores
to highlight the contribution of each token to a prediction (Denil, Demiraj,
and De Freitas 2014; Li et al. 2015; Jain et al. 2020; Kersten et al. 2021).
The top-k important tokens are typically considered as the prediction
rationale (Zaidan, Eisner, and Piatko 2007; Kindermans et al. 2016;
Sundararajan, Taly, and Yan 2017; DeYoung et al. 2020). The quality of a
rationale is often evaluated using faithfulness metrics which measure to what
extent the rationale accurately reflects the decision mechanisms of the model
(DeYoung et al. 2020; Jacovi and Goldberg 2020; Ding and Koehn 2021; Zhao and
Aletras 2023).222Plausibility is another aspect of rationale quality,
measuring how understandable it is to humans (Jacovi and Goldberg 2020) and it
is out of the scope of our study.

Figure 1: Input importance distributions for a generative task (top) and a
classification task (bottom) using a toy FA.

The faithfulness of FAs is well-studied in the context of text classification
tasks, particularly with encoder-only language models (LMs), e.g. BERT-based
models (Ding and Koehn 2021; Zhou, Ribeiro, and Shah 2022; Zhao et al. 2022;
Attanasio et al. 2023). A range of FAs has been developed and evaluated in
such a context, e.g. propagation-based methods that rely on forward-backward
passes to the model for computing importance (Kindermans et al. 2016;
Sundararajan, Taly, and Yan 2017; Atanasova et al. 2020), attention-based
methods that use attention weights as an indication of importance (Serrano and
Smith 2019; Jain et al. 2020; Abnar and Zuidema 2020), and occlusion-based
methods that probe the model behavior with corrupted input (Lei, Barzilay, and
Jaakkola 2016; Bastings, Aziz, and Titov 2019; Bashier, Kim, and Goebel 2020).

However, it is non-trivial and not guaranteed to be faithful to apply these
FAs on decoder-only LMs for text generation directly, due to the inherent
differences between encoder and decoder-only models and task settings
respectively, e.g. different attention mechanisms. Moreover, FAs should
produce an attribution distribution per token position for decoder-only LMs in
generation tasks, while only a single attribution distribution needs to be
computed for fine-tuned encoder-only models. Figure 1 shows the difference in
complexity of applying a toy FA on a decoder and encoder-only LM respectively.
Previous work has demonstrated that there is no ‘one-wins-all’ FA across
models and tasks (Atanasova et al. 2020). Therefore, a separate comparison of
various FAs is required for each task and model. However, this is
computationally expensive for large LMs (e.g. some FAs may additionally
require forward-backward passes). In some cases, FAs might not even be
applicable to black-box models that are available only through an API that
might provide access to the predictive likelihood but not the internal
weights, e.g. the OpenAI API.333https://platform.openai.com/docs/api-
reference/introduction

Vafa et al. (2021) proposed using a greedy search to find the shortest subset
of the input as the rationale for generation tasks. However, their method only
highlights important tokens in a binary manner, i.e. important or not. More
importantly, this method requires model fine-tuning which is prohibitive for
large LMs and it might not truly reflect the original models’ inner reasoning
process but the new fine-tuned one.

To address these challenges, we propose Recursive Attribution Generator
(ReAGent), a FA for decoder-only LMs that computes the input importance
distribution for each token position recursively by replacing tokens from the
original input with tokens predicted by RoBERTa where the rest are masked. Our
intuition is that replacing important tokens should result in larger
predictive likelihood differences. We make the following contributions:

  * •

ReAGent does not require accessing the model internally and can be easily
applied to any generative LM;

  * •

We empirically show that ReAGent is consistently more faithful than seven
popular FAs by conducting a comprehensive suite of experiments, covering three
tasks and six LMs of varying sizes from two different model families;

  * •

Our method does not need to retain any gradients for computing input
importance but only relies on forward passes (Bach et al. 2015; Lundberg and
Lee 2017; Selvaraju et al. 2017; Ancona et al. 2018). This minimizes the
memory footprint and compute required.

##  2 Related Work

###  2.1 Post-hoc FAs

Post-hoc explanation methods such as FA techniques are applied retrospectively
by seeking to extract explanations after the model makes a prediction. Most
FAs have been proposed in the context of classification tasks, where a
sequence input $x_{1},\ldots,x_{t-1}$ is associated with a true label $y$ and
a predicted label $\hat{y}$. The underlying goal is to identify which parts of
the input contribute more toward the prediction $\hat{y}$ (Lei, Barzilay, and
Jaakkola 2016). Figure 1 (bottom) shows an example. Most FAs generally fall
into propagation, attention, and occlusion-based categories.

Propagation-based (i.e. gradient-based) methods derive the importance for each
token by computing gradients with respect to the input (Denil, Demiraj, and De
Freitas 2014). Building upon this, the input token vector is multiplied by the
gradient (Input x Gradient) (Denil, Demiraj, and De Freitas 2014), while
Integrated Gradients compares the input with a null baseline input when
computing the gradients with respect to the input (Sundararajan, Taly, and Yan
2017). Other propagation-based FAs include Layer-wise Relevance Propagation
(Bach et al. 2015), GradientSHAP (Lundberg and Lee 2017), and DeepLIFT
(Selvaraju et al. 2017; Ancona et al. 2018). Nielsen et al. (2022) offer a
comprehensive overview of different propagation-based FAs.

Attention-based FAs are applied to models that include an attention mechanism
to weigh the input tokens. The assumption is that the attention weights
represent the importance of each token. These FAs include scaling the
attention weights by their gradients, taking the attention scores from the
last layer, and recursively computing the attention in each layer (Serrano and
Smith 2019; Jain et al. 2020; Abnar and Zuidema 2020).

Occlusion-based methods measure the difference in model prediction between
using the original input and a corrupted version of the input by gradually
removing tokens (Lei, Barzilay, and Jaakkola 2016; Nguyen 2018; Bastings,
Aziz, and Titov 2019; Bashier, Kim, and Goebel 2020) The underlying idea is
that removing important tokens will lead the model to flip its prediction or a
significant drop in the prediction confidence.

Another line of work has focused on learning feature attributions with a
modified model or a separate explainer model (Ribeiro, Singh, and Guestrin
2016; Lundberg and Lee 2017; Kindermans et al. 2019; Bashier, Kim, and Goebel
2020; Kokhlikyan et al. 2021; Hase, Xie, and Bansal 2021). For example, LIME
trains a linear classifier that approximates the model behavior in the
neighborhood of a given input. SHAP (SHapley Additive exPlanations) learns the
predictive probability of using different combinations of tokens and then
computes the marginal contribution of each token towards the target prediction
(Lundberg and Lee 2017).

The methods above have specific model dependencies (e.g. attention) or require
specific training or fine-tuning, which makes it difficult to apply them to
decoder-only models for generation tasks directly.

###  2.2 FAs for Generative Models

In text generation tasks, the model (often decoder-only) needs $n$ forward
passes to generate a sequence of $n$ tokens. As shown in Figure 1, a different
token importance distribution is computed for each predicted token.

Vafa et al. (2021) propose a greedy search approach to find a minimum subset
of the input that maximizes the probability of the target token which is
considered as the rationale for generative LMs. Their method identifies if a
token belongs to the rationale or not, i.e. binary prediction, rather than an
importance distribution. This makes it hard to evaluate if such a rationale is
more faithful than others. Furthermore, this method requires fine-tuning the
original model to support blank positions (i.e. removed tokens) which is
rather challenging and computationally inefficient for large LMs. Cífka and
Liutkus (2023) estimate the importance of each token by exploring how
different lengths of their left-hand side context impact the predictive
probability of the target token. They describe the estimated importance output
as “differential importance scores”, and point out that a higher score should
not be interpreted as the token is more important for the prediction, but it
should be considered as how much “new information” the token adds to the
context that the other tokens have not covered yet.

To the best of our knowledge, no previous work has proposed a model-agnostic
FA for generative LMs.

##  3 Preliminaries

###  3.1 Generative Language Modeling

In generative language modeling, the input is a sequence of tokens,
$\bm{X}=\left[x_{1},\ldots,x_{t-1}\right]$, i.e. the initial input (often
called a prompt). The goal is to learn a model $f_{\theta}$ that approximates
the probability $p_{\theta}$ of the token sequence $x_{1},\ldots,x_{t-1}$.
Here, we assume that $f_{\theta}$ is a specific pre-trained generative LM,
such as GPT (Brown et al. 2020), for predicting the probability of the next
token $x_{t}$ conditioned to the context $x_{1},\ldots,x_{t-1}$ and
parameterized by $\theta$:

$p_{\theta}\left(x_{1},\ldots,x_{t-1}\right)=f_{\theta}\left(x_{1}\right)\prod_{t=2}^{T}f_{\theta}\left(x_{t}\mid
x_{1},\ldots,x_{t-1}\right)$ (1)

###  3.2 Input Importance for Generative LMs

Assuming an underlying model $f_{\theta}$, we want to compute the input
importance distribution for each predicted token $\hat{x}\subset X$ given the
input tokens $X=x_{1},\ldots,x_{t-1}$.

A FA method $e_{t}$ applied on position $t$ produces an importance
distribution $S_{t}=s_{1},\ldots,s_{t-1}$ for a target token $x_{t}$:

$e_{t}(f,\theta,x_{1},\ldots,x_{t-1},x_{t})\rightarrow
S_{t},t\in\\{1,\ldots,n\\}$ (2)

We assume that a higher $s_{i}$ denotes higher importance. $e$ is applied $n$
times for generating importance distributions for $n$ tokens. For a model
agnostic $e_{t}$, access to model parameters $\theta$ is not required.

##  4 Recursive Attribution Generator (ReAGent)

Our aim is to compute the input importance distribution per predicted token by
being (1) model-agnostic (i.e. we should not rely on an attention mechanism);
and (2) able to support black-box LMs with no access to their weights and
architecture details (i.e. we should not rely on backward passes for computing
gradients).

Inspired by occlusion-based FA methods (Zeiler and Fergus 2014; Nguyen 2018),
we assume that replacing more important tokens in the input context
$x_{1},\ldots,x_{t-1}$ will result in larger drops in the predictive
likelihood of the target token $x_{t}$. Therefore, we propose ReAGent, a new
method that iteratively replaces tokens from the context
$x_{1},\ldots,x_{t-1}$ to compute token importance.

The overall process (shown in Algorithm 1) consists of: (1) initializing
importance scores randomly (Step 1); (2) starting an iteration loop for
updating importance scores (Steps 3 to 6) till meeting the stopping condition
(Step 2).

Algorithm 1 Recursive Attribution Generator

Input: LM $f$, context $x_{1},\ldots,x_{t-1}$, target token $x_{t}$

Output: $\textbf{S}_{t}=\\{{s_{1},\dots,s_{t-1}}\\}$

1: Randomly initialize importance scores $\bm{S}_{t}$

2: while !StoppingCondition ($\bm{S}_{t}$, $x_{t}$) do

3: $\mathcal{R}$ $\leftarrow$ randomly select tokens $\mathcal{R}\in
x_{1},\ldots,x_{t-1}$

4: $\hat{x}_{1},\ldots,\hat{x}_{t-1}$ $\leftarrow$ replace $\mathcal{R}$ on
$x_{1},\ldots,x_{t-1}$ with tokens predicted by RoBERTa

5: $\Delta p$ $\leftarrow$
$p(x_{t}|x_{1},\ldots,x_{t-1})-p(x_{t}|\hat{x}_{1\dots t-1})$

6: update importance scores $s_{1},\dots,s_{t-1}$ by $\Delta p$ and
$\mathcal{R}$

7: end while

8: return $\bm{S}_{t}$

###  4.1 Computing Importance Scores

To compute the importance of the context tokens $x_{1},\ldots,x_{t-1}$, we
repeat the following steps until a stopping condition is met.

#### Step 3: Context tokens to be replaced

We first randomly choose a set of tokens $\mathcal{R}$ ($r\%$ of the entire
sequence, $r=30$) from $x_{1},\ldots,x_{t-1}$ to be replaced.

#### Step 4: Replacement tokens

A replacing function $\bm{C}$ replaces each token in $\bm{X}$ with the token
predicted by RoBERTa.444We also test another two methods providing tokens to
replace. However, they empirically perform poorly. We include the results of
these two methods in Section A in the Appendix. A new sequence is generated
for the occlusion of $\bm{X}$ later.

$\begin{split}\bm{C}(\bm{X})=&[\hat{x}_{1},\hat{x}_{2},\dots\hat{x}_{t-1}]\\\
&\quad\sim U(\mathcal{X}^{(g)}([x_{1},x_{2},\dots,x_{t-1}]))\end{split}$ (3)

#### Steps 5 & 6: Updating importance scores

The importance distribution $\bm{S}_{t}$ is computed based on the selection of
tokens, $\mathcal{R}$, and the predictive difference, $\Delta p$ in Eq. 6,
after $\mathcal{R}$ replaced by $\bm{C}(\bm{X})$ as following:

$\displaystyle p^{(o)}_{t}$ $\displaystyle=p(x_{t}|\bm{X})$ (4) $\displaystyle
p^{(r)}_{t}$
$\displaystyle=p(x_{t}|(\bm{M}(\bm{X},\overline{\mathcal{R}})+\bm{M}(\bm{C}(\bm{X}),\mathcal{R})))$
(5) $\displaystyle\Delta p_{t}$ $\displaystyle=p^{(o)}_{t}-p^{(r)}_{t}$ (6)
$\displaystyle\Delta\bm{S}_{t}$ $\displaystyle=\bm{M}(\Delta
p_{t}\cdot\mathds{1}^{|\bm{X}|},\mathcal{R})+\bm{M}(-\Delta
p_{t}\cdot\mathds{1}^{|\bm{X}|},\overline{\mathcal{R}})$ (7)
$\displaystyle\bm{S}^{(l)}_{t}$
$\displaystyle=\bm{S}^{(l)}_{n-1}+\text{logit}\left(\frac{\Delta\bm{S}_{t}+1}{2}\right)$
(8) $\displaystyle\bm{S}_{t}$ $\displaystyle=\text{softmax}(\bm{S}^{(l)}_{t})$
(9)

where $p^{(o)}_{t}$ and $p^{(r)}_{t}$ are predictive probabilities for target
$x_{t}$ when using the original sequence and the sequence with replaced
tokens, respectively. $\bm{M}(\bm{X},\mathcal{R})$ is a masking function that
results in token replacements only for $x_{i}$ selected in step 3. Finally,
$\bm{S}_{t}$ is obtained by adding $\Delta\bm{S}_{t}$ with $\bm{S}_{t-1}$ in
logistic scale.

###  4.2 Step 2: Stopping Condition

A stopping condition is set up for the iterative updating process. When
prompting the model with top-$n$ (default $70\%$) unimportant tokens replaced
with RoBERTa predictions, if the top-k (default set to 3) predicted candidate
tokens (by the model to explain) contain the target token $x_{t}$, the
iterative process stops.

##  5 Experimental Setup

###  5.1 Datasets

Dataset | Length | #Data | Prompt Example
---|---|---|---
LongRA | 36 | 37–149 |  “When my flight landed in Japan, I converted my currency and slowly fell asleep. (I had a terrifying dream about my grandmother, but that’s a story for another time). I was staying in the capital,  ”
TellMeWhy | 50 | 200 |  “Joe ripped his backpack. He needed a new one. He went to Office Depot. They had only one in stock. Joe was able to nab it just in time.Why did He need a new one?”
WikiBio | 35 | 238 |  “Rudy Fernandez (1941–2008) was a labor leader and civil rights activist from the United States. He was born in San Antonio, Texas, and was the son of Mexican immigrants.”

Table 1: Summary of Tasks. Note that different models have different LongRA
data as we only keep ones that have the same predictions with and without
distractors (sentences in parenthesis). Length refers to the average token
count of input sequences.

We experiment with three generic text generation datasets for prompting LMs
with different contextual levels, following recent related work (Vafa et al.
2021; Lal et al. 2021; Gehrmann et al. 2021; Manakul, Liusie, and Gales 2023).

  * •

Long-Range Agreement (LongRA) is designed by Vafa et al. (2021) to allure the
model to predict a specific word. The dataset uses template sentences based on
word pairs from Mikolov et al. (2013) such that the two words, e.g. Japan and
Tokyo in Table 1, are semantically or syntactically-related. The input
includes one word from a pair and aims to prompt the model to predict the
other word in the pair. A distractor sentence that contains no pertinent
information about the word pair is also inserted, e.g. in () in Table 1.
Following Vafa et al. (2021), we only keep examples where the prediction is
the same both with and without the distractor.

  * •

TellMeWhy is a generation task for answering why-questions in narratives. The
input contains a short narrative and a question concerning why characters in a
narrative perform certain actions (Lal et al. 2021). We randomly sample 200
examples.

  * •

WikiBio is a dataset of Wikipedia biographies. We take the first two sentences
as a prompt similar to Manakul, Liusie, and Gales (2023). The model is
expected to continue generating the biography. This task is intuitively more
open than the two tasks above.

###  5.2 Models

We test six large LMs, three models from the GPT (Radford et al. 2019; Brown
et al. 2020; Wang 2021; Wang and Komatsuzaki 2021) and three from the OPT
family (Zhang et al. 2022). We choose our models to span from hundreds of
millions to a few billions of parameters (354M, 1.5B and 6B parameters for
GPT; and 350M, 1.3B, and 6.7B parameters for OPT), as we want to explore how
the model size affects the faithfulness of each FA. All models used are
publicly available. 555We use checkpoints from the Huggingface library for
each model with the following identifiers: gpt2- medium, gpt2-xl,
EleutherAI/gpt-j-6b, facebook/opt-350m, facebook/opt-1.3b,
KoboldAI/OPT-6.7B-Erebus.

###  5.3 Feature Attribution Methods

For comparison, we test seven widely-used FAs (Atanasova et al. 2020; Vafa et
al. 2021):

  * •

Input X Gradient: embedding gradients multiplied by the embeddings (Denil,
Demiraj, and De Freitas 2014).

  * •

Integrated Gradients: Integrate overall gradients using a linear interpolation
between a baseline input (all zero embeddings) and the original input
(Sundararajan, Taly, and Yan 2017).

  * •

Gradient Shap: Compute the gradient with respect to randomly selected points
between the inputs and a baseline distribution (Lundberg and Lee 2017).

  * •

Attention: It is computed averaged attentions across all layers and heads
(Jain et al. 2020).

  * •

Last Attention: The last-layer attention weights averaged across heads (Jain
et al. 2020).

  * •

Attention rollout: Recursively computing the token attention in each layer,
e.g. computing the attention from all positions in layer $l_{i}$ to all
positions in layer $l_{j}$, where $j<i$ (Abnar and Zuidema 2020).

  * •

LIME: For each input, train a linear surrogate model using data points
randomly sampled locally around the prediction (Ribeiro, Singh, and Guestrin
2016).

  * •

Random Following Chrysostomou and Aletras (2022), we take a random attribution
baseline (i.e. randomly assigning importance scores) as a yardstick to have a
comparable analysis across data.

We exclude FAs that demand additional model modifications, e.g. greedy search
(Vafa et al. 2021).666We provide a comparison with the greedy search on
GPT2–354M in the Appendix. Due to the expensive fine-tuning, we do not apply
it to other models.

###  5.4 Faithfulness Evaluation

For evaluation, we use normalized Soft-Sufficiency (Soft-NS) and
comprehensiveness (Soft-NC) that measure the faithfulness of the full
importance distribution proposed by Zhao and Aletras (2023). The metrics are
defined as follows:

$\displaystyle\text{Soft-S}(\mathbf{X},x_{t},\mathbf{X^{\prime}})=1-\max(0,p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathbf{X^{\prime}}))$
(10) $\displaystyle\text{Soft-
NS}(\mathbf{X},\hat{y},\mathbf{X^{\prime}})=\frac{\text{Soft-S}(\mathbf{X},\hat{y},\mathbf{X^{\prime}})-\text{S}(\mathbf{X},\hat{y},0)}{1-\text{S}(\mathbf{X},\hat{y},0)}$
$\displaystyle\text{Soft-C}(\mathbf{X},\hat{y},\mathbf{X^{\prime}})=\max(0,p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathbf{X^{\prime}}))$
$\displaystyle\text{Soft-
NC}(\mathbf{X},\hat{y},\mathbf{X^{\prime}})=\frac{\text{Soft-C}(\mathbf{X},\hat{y},\mathbf{X^{\prime}})}{1-\text{S}(\mathbf{X},\hat{y},0)}$

where $\mathbf{X^{\prime}}$ is obtained by using $q=1-s_{i}$ in Eq. 11 for
each token vector $\mathbf{x^{\prime}_{i}}$. Given a token vector
$\mathbf{x}_{i}$ from the input $\mathbf{X}$ and its FA score $s_{i}$, the
input is soft-perturbed as follows:

$\displaystyle\mathbf{x^{\prime}_{i}}=\mathbf{x_{i}}\odot\mathbf{e_{i}},\>\mathbf{e_{i}}\sim\operatorname{Ber}(q)$
(11)

where $Ber$ a Bernoulli distribution and $\mathbf{e}$ a binary mask vector of
size $n$. $Ber$ is parameterized with probability $q$:

$q=\begin{cases}s,&\text{if retaining elements}\\\ 1-s,&\text{if removing
elements}\end{cases}$ (12)

Adapting Soft-NS & NC for Generative LMs

Soft-NS and Soft-NC have been proposed for evaluating FAs in the context of
encoder-only models. To evaluate FAs on generation tasks, there is one issue
to address when adapting Soft-NS and Soft-NC to generative tasks and models:
the lack of predictive likelihood of the predicted label. Naturally, we can
treat each token as a label. However, in empirical experiments, we find that
the high dimension of the output leads to high sensitivity of the predictive
probability of the specific token. In other words,
$p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathcal{R})$ and
$p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathbf{X}_{\backslash\mathcal{R}})$ often
approximate to $p(\hat{y|\mathbf{X}})$. Re-scaling is a potential but hassle
solution due to the fact that different inputs might need different scaling.

Therefore, we propose to measure the Hellinger distance between prediction
distributions over the vocabulary to assess the changes in model predictions.
Essentially, we replace $p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathbf{X^{\prime}})$
in Eq. 10 with the Hellinger distance. Considering two discrete probability
distributions $\bm{P}_{X,t}=\left[p_{1,t},\ldots,p_{v,t}\right]$ and
$\bm{P}_{X^{\prime},t}=\left[p^{\prime}_{1,t},\ldots,p^{\prime}_{v,t}\right]$,
the Hellinger distance is formally defined as:

$\displaystyle\Delta\bm{P}_{X^{\prime},t}$
$\displaystyle=H(\bm{P}_{X,t},\bm{P}_{X^{\prime},t})$ (13)
$\displaystyle=\frac{1}{\sqrt{2}}\sqrt{\sum_{i=1}^{v}{\left(\sqrt{p_{i,t}}-\sqrt{p^{\prime}_{i,t}}\right)}^{2}}$

where $\bm{P}_{X,t}$ is the probability distribution over the entire
vocabulary (of size $v$) when prompting the model with the full-text, $X$.
$\bm{P}_{X^{\prime},t}$ is for prompting the model with soft-perturbed text.
For a given sequence input $X={x_{1},\ldots,x_{t-1}}$ and a model of
vocabulary size $v$, at time step $t$, the model generates a distribution
$\bm{P}_{X,t}$ for the next token $x_{t}$. The final Soft-NS and Soft-NC at
step $t$ for text generation are formulated as:

$\displaystyle\text{Soft-
NS}(\mathbf{X},x_{t},\mathcal{R})=\frac{\max(0,\Delta\bm{P}_{0,t}-\Delta\bm{P}_{X^{\prime},t})}{\Delta\bm{P}_{0,t}}$
(14) $\displaystyle\text{Soft-
NC}(\mathbf{X},x_{t},\mathcal{R})=\frac{\Delta\bm{P}_{\mathbf{X^{\prime}}_{\backslash\mathcal{R}},t}}{\Delta\bm{P}_{0,t}}$
(15)

where $\Delta\bm{P}_{0,t}$ is Hellinger distance between a zero input’s
probability distribution and full-text input’s probability distribution.
${\mathbf{X^{\prime}}_{\backslash\mathcal{R}},t}$ is the case of “if removing
elements” described in Eq. 12.

Cífka and Liutkus (2023) adopt KL divergence to measure the impact on the
model probability distribution when using different lengths of left-hand
context. Noted by Cífka and Liutkus (2023), this metric in their study is not
related to the marginal effect of each token on the model prediction (Bastings
and Filippova 2020; Pham et al. 2022), but rather to measure how much “new
information” the token adding to the input that other tokens have not
covered.777Several metrics of measuring the distance for discrete
distributions exist, such as the Bhattacharyya distance, the Hellinger
distance, and Kullback-Leibler divergence (Lebret and Collobert 2014; Grave,
Obozinski, and Bach 2014; Weeds, Weir, and McCarthy 2004; Ó Séaghdha and
Copestake 2008). We use Hellinger distance for its simplicity and symmetry
property (as it is a true distance) (Lebret and Collobert 2014). Also, its
range is [0, 1], considering the
$p(\hat{y}|\mathbf{X})-p(\hat{y}|\mathcal{R})$ range [-1, 1].

We evaluate the token-level faithfulness on the dataset LongRA and the
sequence-level faithfulness on TellMeWhy and WikiBio. Specifically, on LongRA,
we evaluate Soft-NS/NC on the target word (from the word pair) for a token-
level faithfulness. On TellMeWhy and WikiBio, we evaluate Soft-NS/NC for every
five tokens888That is, evaluating on a stride size of five tokens. The purpose
is to save computing. In our code, it is a changeable hyperparameter. from the
sequential prediction, and take the average as the sequence-level
faithfulness.

###  5.5 Implementation Details

Our method has four hyperparameters: ratio of replaced tokens $r$ for
updating, validating tolerance $k$ and rationale size $n$ for the stopping
condition. We empirically find the faithfulness of our method is not sensitive
to the hyperparameters of the following ranges: replacing ratio
$r\in({0.1,0.3})$, validating tolerance $k\in({3,5})$, and rationale size
$n\in({5,8})$. A faithfulness comparison of different hyperparameters using
GPT2–354M and further discussion is presented in Table 6 in Section 6.4. We
recommend practitioners use validating tolerance $k=3$, replacing ratio
$r=0.1$, and replacing ratio 0.3, which is the fastest combination, and its
faithfulness is not necessarily decreased.

Our method relies on the random selection of to-be-replaced tokens. To handle
the stochasticity and the potential coverage issues, we perform three separate
runs with different random seeds to obtain three importance distributions. We
average distributions that stop before a maximum of 1,000 steps (loops in
Algorithm 1).

We use pre-trained generative LMs from the Huggingface library (Wolf et al.
2020). We do not fine-tune any model and always use vanilla settings.
Experiments are run on a single NVIDIA A100 GPU with 80GB memory (HBM2e). We
use a Dell PowerEdge XE8545 machine with CUDA parallelism.

##  6 Results

###  6.1 Overall Faithfulness

Figure 2 shows results on token-level faithfulness. Figure 3 shows sequence-
level faithfulness. All Soft-NS/NC are presented as the log of scores divided
by the random baseline. Accordingly, FAs with Soft-NS/NC below zero are less
faithful than the random baseline, i.e. unfaithful. Exhaustive results with
significant tests can be found in Table 8 in the Appendix B.

Figure 2: Token-level faithfulness on LongRA. Values that are close to zero
indicate its faithfulness is on par with the random baseline.

Figure 3: Sequence-level faithfulness, i.e. Soft-NS and Soft-NC, on the two
datasets: WikiBio and TellMeWhy. Values that are close to zero indicate its
faithfulness is on par with the random baseline.

Overall, no single FA outperforms the others consistently across metrics,
datasets, and models. ReAGent outperforms the other FAs in more than half of
evaluation cases. For example, on LongRA (Figure 2), ReAGent always achieves
the highest Soft-NC across models, and on WikiBio, ReAGent consistently
outperforms the rest of the FAs on all OPT models. Further, ReAGent is the
only FA that consistently outperforms the random baseline. Therefore, we
suggest that the faithfulness of a FA varies across generative models and
generation tasks. Compared with popular FAs, our FA is more faithful as it
shows faithfulness consistency, and it outperforms the other FAs across tasks
and models overall.

Between tasks, ReAGent shows a more apparent advantage over other FAs on the
token-level task, LongRA. ReAGent consistently outperforms the others on
LongRA except for being on par with the best, integrated gradients, on Soft-NS
on OPT-1.3B and GPT-J-6B. Between models, we observe that FAs tend to have
comparable faithfulness performance on models from the same model family than
non-family models. For example, on WikiBio, Integrated gradient always has
higher Soft-NS and Soft-NC on GPT family than on OPT family. Similarly, Lime
is consistently on par with the random baseline on OPT models but is the top-
two highest on GPT models. We further investigate these in later sections.

###  6.2 Faithfulness across Tasks and Models

We average Soft-NS/NC over tasks (Table 2) and models (Table 3) respectively.
Overall, ReAGent outperforms the other FAs in 17 column-wise comparisons out
of 18 in total in the two tables. On the only one exceptional, ReAGent is on
par with the corresponding best, i.e. 1.008 to 1.147 (Soft-NS, GPT-J-6B).

| Soft-NC | Soft-NS
---|---|---
| LongRA | TellMeWhy | WikiBio | LongRA | TellMeWhy | WikiBio
Attention | -0.28 | 2.161 | 1.176 | 0.099 | -0.3 | 0.302
Attention Last | 0.048 | 0.07 | 0.092 | -0.222 | -0.102 | -0.151
Attention Rollout | 0.209 | 0.047 | 0.211 | -0.099 | -0.085 | -0.023
Gradient Shap | 1.101 | 1.892 | 0.108 | -0.116 | -0.029 | 0.51
Input X Gradient | 1.423 | 1.463 | 0.49 | 0.03 | -0.22 | -0.081
Integrated Gradients | 1.865 | 1.536 | 1.384 | 0.451 | 0.045 | 0.765
Lime | 0.412 | 0.249 | 1.906 | -0.012 | -0.091 | 0.461
ReAGent | 5.402 | 4.504 | 1.982 | 1.136 | 1.024 | 1.087

Table 2: Soft-NS and Soft-NC averaged over tasks. The best FA on the model
(column) is highlighted in bold.

Soft-NC | OPT-350M | OPT-1.3B | OPT-6.7B | GPT2–354M | GPT2–1.5B | GPT-J-6B
---|---|---|---|---|---|---
Attention | 0.011 | 0.865 | 1.167 | 1.142 | 1.542 | 1.387
Attention Last | -0.161 | 0.163 | 0.431 | -0.114 | 0.204 | -0.104
Attention Rollout | -0.059 | 0.241 | 0.457 | 0.192 | 0.138 | -0.034
Gradient Shap | -0.051 | 0.222 | -0.022 | 1.645 | 2.449 | 1.959
Input x Gradient | 0.243 | -0.12 | 0.188 | 1.939 | 2.64 | 1.86
Integrated Gradients | 0.323 | 0.328 | 0.129 | 2.408 | 3.442 | 2.94
Lime | 0.221 | -1.431 | -1.48 | 2.269 | 3.47 | 2.086
ReAGent | 2.187 | 3.753 | 5.247 | 3.202 | 5.471 | 3.916
Soft-NS | OPT-350M | OPT-1.3B | OPT-6.7B | GPT2–354M | GPT2–1.5B | GPT-J-6B
Attention | 0.068 | 0.233 | 0.581 | 0.039 | 0.448 | -1.168
Attention Last | -0.085 | -0.173 | -0.397 | -0.21 | -0.005 | -0.079
Attention Rollout | 0.089 | -0.151 | -0.214 | -0.077 | 0.006 | -0.068
Gradient Shap | -0.334 | -0.035 | -0.107 | 0.586 | 0.026 | 0.593
Input x Gradient | -0.149 | -0.371 | -0.133 | -0.079 | -0.181 | 0.371
Integrated Gradients | -0.384 | -0.002 | 0.134 | 0.657 | 0.971 | 1.147
Lime | -0.16 | -0.649 | 0.01 | 0.377 | 0.523 | 0.614
ReAGent | 0.693 | 0.759 | 1.306 | 1.192 | 1.535 | 1.008

Table 3: Soft-NS and Soft-NC averaged over models. The best FA on the model
(column) is highlighted in bold.

Dataset | Full Output | FA | Input
---|---|---|---
WikiBio |  developed by Nintendo for the Nintendo Entertainment System. |  ReAGent |  Super Mario Land is a side sc rolling platform video game developed by
Lime |  Super Mario Land is a side sc rolling platform video game developed by
TellMeWhy |  He went to see his old college. |  ReAGent |  Jay took a trip to his old college Jay is an alumni He visited his friends He went and got drunk He had a good time Why did He go ? He
Lime |  Jay took a trip to his old college Jay is an alumni He visited his friends He went and got drunk He had a good time Why did He go ? He

Table 4: Importance distribution is given by ReAGent and Lime, for Model
GPT2–1.5B.

#### Tasks

As shown in Figure 2, ReAGent is the most faithful FA in terms of both Soft-NS
and Soft-NC across task-wise comparison. Particularly, for the token-level
task, LongRA, ReAGent overwhelmingly outperforms the second-highest, i.e.
5.402 to 1.865 (Integrated Gradients). In task LongRA, intuitively, the model
takes more hints from the indicator token in the input, e.g. “Tennessee” to
“Nashville”. Between TellyMeWhy and WikiBio, ReAGent exhibits greater
advantages on TellyMeWhy, where the prompts provide richer contextual
information. We conduct a qualitative analysis of the importance distributions
by each FA and observe that ReAGent more effectively captures the hint token
within the context of the input prompt, examples from GPT2–1.5B are shown in
Figure 4 and Table 4.

Figure 4: Importance distribution over the input: “As soon as I arrived in
Tennessee, I checked into my hotel, and watched a movie before falling asleep.
(I had a great call with my husband, although I wish it were longer). I was
staying in my favorite city, ”. The sentence in () is the distractor. The
model predicts “Nashville” regardless of whether the input includes the
distractor or not.

Figure 4 shows the importance distribution given by the best (ReAGent), the
second-best (Integrated gradients), and the worst (Attention) FAs on LongRA.
For this example, ReAGent is able to better catch the importance of
“Tennessee” and “city” than Integrated Gradients and Attention. Although
Integrated gradients, the second-best FA, recognizes the importance of
“Tennessee”, it fails to ignore the contribution of the distractor sentence as
ReAGent does. Likely, we see the WikiBio example in Table 4, ReAgent more
emphasizes the importance of “Super Mario” than Lime when predicting
“Nintendo”.

We conclude that for tasks provided with contextualized information and
expecting specific predictions, ReAGent is more faithful than the other FAs.
It is capable to pick up the link between the hint in the input and the
predictions that the model utilizes to make the specific prediction.

#### Models

As shown in Figure 2, between OPT and GPT family, ReAGent shows greater
advantages on OPT models than on GPT models, i.e. ReAGent consistently
outperforms other FAs on OPT models on both metrics while only achieving the
top-two consistently on GPT models. This indicates the possible difference in
inner operation between model families and the similarity within model
families.

###  6.3 Evaluation on Long-Range Agreement with Greedy Search

Following Vafa et al. (2021), we conduct a Long-Range Agreement evaluation of
rationales given by each FA and the greedy search method (i.e. greedy
rationalization (Vafa et al. 2021)).The greedy search aims to discover the
smallest subset of input tokens that would predict the same output as the full
sequence (Vafa et al. 2021). Therefore, rationales given by the greedy search
are in various lengths, i.e. each input has its own rationales in different
lengths. We reproduce their experiment and apply the input-specific rationale
length by greedy search to extract the rationale for each FA, and then
evaluate the Ante ratio and No.D ratio of rationales. We acknowledge the
potential bias that the setup of the rationale length is biased towards the
greedy search due to the limitation of the greedy search itself. Specifically,
greedy search only provides binary rationales, i.e. a token is a rationale or
not, rather than an importance distribution. Further, for a given output, two
selected rationales by greedy search will be taken as equally important. We
only experiment with GPT2–354M as greedy search requires further fine-tuning.

Table 5 shows the Ante ratio and No.D ratio of each FA when using varying
lengths of rationales by greedy search, on dataset LongRA. For both metrics,
the higher, the better the rationales are, according to the faithfulness
definition (the smallest subset for predicting the same output as the full
sequence) by Vafa et al. (2021). As shown in the table, although the setup of
rationale length favours greedy search by nature, Attention Rollout
outperforms greedy search on No.D and Gradient Shap is on par with Greedy
Search on Ante.

| Ante Ratio | No.D Ratio
---|---|---
Gradient Shap | 1 | 0.592
Input x Gradient | 0.471 | 0.471
Integrated Gradients | 0.987 | 0.503
Attention Rollout | 0.357 | 0.930
Attention Last | 0.898 | 0.745
Attention | 0.936 | 0.707
Greedy Search | 1 | 0.667
ReAGent | 0.930 | 0.710
Table 5: FAs faithfulness on long-range agreement with templates analogies.
“Ratio” refers to the approximation ratio of each method’s rationale length to
the exhaustive search minimum. “Ante” refers to the percentage of rationales
that contain the true antecedent. “No D” refers to the percentage of
rationales that do not contain any tokens from the distractor.

###  6.4 Impact of Hyper-Parameters

All of our experiments do not involve training or fine-tuning any language
models. Gradient-based FAs and Lime are built upon Inseq library (Sarti et al.
2023). All reported results of ReAGent in the main body are with a replacing
ratio of 30%, max step of 3000, three runs, and top five unimportant tokens
for stopping conditions.

To examine the sensitivity of hyper-parameters of ReAGent, we experiment on
GPT2–354M with different settings of these hyperparameters. The result shows
that our method is always more faithful than the random baseline (above zero),
as shown in Table 6.

Replacing Ratio r | Max Step m | Runs | Soft-NS | Soft-NC
---|---|---|---|---
0.1 | 5000 | 5 | 1.910 | 1.623
0.1 | 5000 | 3 | 2.089 | 3.009
0.1 | 3000 | 5 | 1.752 | 2.170
0.3 | 3000 | 3 | 1.752 | 2.170
0.3 | 3000 | 5 | 2.09 | 4.009
0.3 | 3000 | 3 | 2.406 | 1.481
0.3 | 5000 | 3 | 2.039 | 1.639

Table 6: Soft-NS and Soft-NC of ReAGent on LongRA, with different
hyperparameters.

##  7 Conclusion

We proposed ReAGent, a model-agnostic feature attribution method for text
generation tasks. Our method does not require fine-tuning, any model
modification, or accessing the internal weights. We conducted extensive
experiments across six decoder-only models of two model families in different
sizes. The results demonstrate that our method is consistently more faithful
than other commonly-used FAs.

In future work, we plan to evaluate our method on different generative models,
e.g. encoder-decoder and diffusion models, and different text generation
tasks, e.g. translation and summarization.

##  Appendix A Replacement tokens

Model | Token replacing method | Steps | Steps std | Soft-NS | Soft-NC
---|---|---|---|---|---
GPT2 354M | RoBERTa | 244 | 633 | 2.358 | 2.269
Random | 4041 | 1645 | 1.95 | 5.578
POS | 2154 | 1142 | 1.35 | 2.028
GPT2 1.5B | RoBERTa | 239 | 701 | 1.215 | 5.105
Random | 4379 | 1345 | 1.344 | 8.173
POS | 2434 | 928 | 0.039 | 3.419
GPT-J 6B | RoBERTa | 364 | 972 | 1.593 | 2.739
Random | 3126 | 2328 | 0.943 | 6.908
POS | 2105 | 1285 | 1.758 | 3.217
OPT 350M | RoBERTa | 108 | 106 | 1.483 | 3.141
Random | 3776 | 1686 | 1.527 | 3.947
POS | 1554 | 1132 | 2.055 | 1.723
OPT 1.3B | RoBERTa | 265 | 379 | 0.812 | 4.657
Random | 4291 | 1362 | 1.141 | 7.511
POS | 2642 | 827 | 0.865 | 3.308
OPT 6.7B | RoBERTa | 259 | 391 | 0.876 | 7.486
Random | 4878 | 522 | 0.884 | 9.426
POS | 2810 | 534 | 1.825 | 4.725

Table 7: Soft-NS and Soft-NC results of different token replacing methods. As
other Soft-NS and Soft-NC results presented in the paper, results are
presented as the log of scores divided by the random baseline. Maximum steps
are set to 3000. Figure 5: Sufficiency and Comprehensiveness scores on
different updating steps.

ReAGent measures the predictive likelihood changes after top-k important
tokens are perturbed. The perturbation operation replaces these top-k tokens
with RoBERTa predicted tokens. We also experimented with replacing them with
random tokens from the vocabulary ($\hat{x}_{i}=U(\mathcal{V})$, where
$\mathcal{V}$ is the vocabulary and $i\in[0,t]$), noted as “Random”, and
replacing them with random tokens of the same POS, noted as “POS”.

When replacing top-k with random tokens or tokens of the same POS, we found it
often takes thousands of steps to meet the stopping condition, as shown by the
average “Steps” in Table 7.999In experiments, we set up a maximum updating
steps of 5000. Figure 5 shows the faithfulness scores at different steps
(evaluating per 50 steps). We see that using RoBERTa predicted tokens to
replace, both sufficiency and comprehensiveness coverage much earlier than
Random and POS replacements, as shown in the black vertical line in Figure 5.
On the other hand, replacing top-k important tokens with random tokens or
tokens of the same POS does not bring consistently higher faithfulness.

##  Appendix B Full Faithfulness Results

|  | LongRA | TellMeWhy | Wikitext
---|---|---|---|---
Model | FAs | Soft-NS | Soft-NC | Soft-NS | Soft-NC | Soft-NS | Soft-NC
GPT2–354M | Input x Gradient | 0.145 | 1.793 | -0.062 | 3.3 | -0.32* | 0.725
GPT2–354M | Integrated Gradients | 0.266 | 1.827 | 1.293* | 3.152 | 0.411* | 2.244
GPT2–354M | Gradient Shap | -0.419 | 1.867 | 1.405 | 3.046 | 0.773* | 0.023*
GPT2–354M | Attention Rollout | -0.125* | 0.487* | -0.07* | 0.069* | -0.035* | 0.02*
GPT2–354M | Attention Last | -0.098* | -0.281* | -0.054* | -0.036* | -0.477* | -0.027*
GPT2–354M | Attention | -0.137* | -0.226* | 0.53* | 3.141 | -0.275* | 0.511
GPT2–354M | Lime | -0.404 | 0.168 | 0.67* | 3.057 | 0.865 | 3.581
GPT2–354M | ReAGent | 2.09 | 4.009 | 1.404 | 4.483 | 0.081 | 1.113
GPT2–1.5B | Input x Gradient | -0.648 | 2.64 | -0.256* | 4.529 | 0.363* | 0.752*
GPT2–1.5B | Integrated Gradients | -0.513* | 2.165 | -0.011* | 4.443 | 3.437 | 3.72
GPT2–1.5B | Gradient Shap | -0.398* | 2.823 | -0.183* | 4.186 | 0.66* | 0.339*
GPT2–1.5B | Attention Rollout | -0.08 | -0.008* | 0.015* | 0.348* | 0.083* | 0.073*
GPT2–1.5B | Attention Last | 0.17* | 0.052* | -0.093* | 0.533* | -0.091* | 0.029*
GPT2–1.5B | Attention | -0.034* | 0.046* | -0.099* | 4.285 | 1.479* | 0.295*
GPT2–1.5B | Lime | 0.131* | 0.366 | -0.041* | 4.257 | 1.477 | 5.787
GPT2–1.5B | ReAGent | 0.492 | 7.556 | 1.735 | 5.927 | 2.378 | 2.931
GPT-J-6B | Input x Gradient | 0.784 | 1.652 | 0.173* | 3.366 | 0.157* | 0.56
GPT-J-6B | Integrated Gradients | 1.923 | 4.885 | 0.68* | 1.89* | 0.838 | 2.045
GPT-J-6B | Gradient Shap | 0.497 | 0.948 | 0.945 | 4.592 | 0.337 | 0.337
GPT-J-6B | Attention Rollout | 0.114* | 0.015* | -0.27* | 0.694* | -0.039* | 0.128*
GPT-J-6B | Attention Last | 0.303* | -0.35* | -0.225* | -0.337* | -0.292* | -0.014*
GPT-J-6B | Attention | 0.68 | -0.671* | -2.128 | 3.544 | -2.056 | 1.288
GPT-J-6B | Lime | 1.04 | 1.264 | 0.398 | 3.278 | 0.404 | 1.717
GPT-J-6B | ReAGent | 1.764 | 5.477 | 0.875 | 4.386 | 0.384 | 1.885
OPT-350M | Input x Gradient | 0.152* | 0.49 | -0.879* | 0.11 | 0.281* | 0.131*
OPT-350M | Integrated Gradients | -0.067* | 0.79 | -0.915* | 0.019* | -0.17* | 0.161*
OPT-350M | Gradient Shap | -0.681* | 0.151 | -0.481 | -0.474 | 0.16* | 0.169*
OPT-350M | Attention Rollout | 0.211* | -0.192* | -0.076* | -0.015* | 0.134* | 0.032*
OPT-350M | Attention Last | -0.739* | -0.295* | 0.209 | -0.152* | 0.275* | -0.037*
OPT-350M | Attention | -0.248* | -0.313* | -0.412* | 0.033 | 0.864 | 0.312
OPT-350M | Lime | 0.197* | 0.491 | -0.834* | -0.029* | 0.156* | 0.202*
OPT-350M | ReAGent | 0.932 | 4.074 | 0.147 | 1.6 | 0.999 | 0.887
OPT-1.3B | Input x Gradient | -0.617 | 0.497 | -0.065* | -1.049* | -0.43* | 0.193
OPT-1.3B | Integrated Gradients | 0.618 | 0.858 | -0.603* | 0* | -0.022* | 0.127*
OPT-1.3B | Gradient Shap | -0.588* | 0.696 | -0.647* | 0.002* | 1.132 | -0.034*
OPT-1.3B | Attention Rollout | -0.193* | 0.353 | 0.064* | 0.035 | -0.324* | 0.333
OPT-1.3B | Attention Last | 0.082* | 0.321* | -0.256* | -0.008* | -0.345* | 0.176
OPT-1.3B | Attention | 0.008* | -0.092* | -0.331* | 0.8 | 1.024 | 1.888
OPT-1.3B | Lime | -1.032* | 0.014 | -0.452* | -4.489* | -0.462* | 0.183*
OPT-1.3B | ReAGent | 0.497 | 7.104 | 0.365 | 2.208 | 1.413 | 1.947
OPT-6.7B | Input x Gradient | 0.365* | 1.466 | -0.229 | -1.478 | -0.536 | 0.576
OPT-6.7B | Integrated Gradients | 0.481* | 0.667 | -0.175* | -0.289* | 0.096* | 0.011*
OPT-6.7B | Gradient Shap | 0.896 | 0.122* | -1.214 | -0.002* | -0.003* | -0.186*
OPT-6.7B | Attention Rollout | -0.523* | 0.6 | 0.009* | 0.02 | 0.045* | 0.681
OPT-6.7B | Attention Last | -1.048 | 0.839 | -0.022* | 0.074 | 0.024* | 0.424
OPT-6.7B | Attention | 0.327* | -0.426* | 0.641 | 1.166 | 0.775 | 2.76
OPT-6.7B | Lime | -0.006* | 0.17* | -0.288* | -4.578* | 0.323* | -0.033*
OPT-6.7B | ReAGent | 1.039 | 4.188 | 0.574 | 2.36 | 1.264 | 3.13

Table 8: Full Soft-NS and Soft-NC results of different FAs on different
Models. For consistency and comparison across data and models, results are
presented as the log of scores divided by the random baseline. Asterisk (*)
denotes the case where its p-value is greater than .05 in the Wilcoxon signed-
rank Test, i.e. the score, Soft-NS or Soft-NC, is not significantly different
from the random baseline.

## Acknowledgments

We thank Professor Nikolaos Aletras and Dr George Chrysostomou for their
helpful comments, thoughts, and discussions. We acknowledge IT Services at The
University of Sheffield for the provision of services for High Performance
Computing.

## References

  * Abnar and Zuidema (2020) Abnar, S.; and Zuidema, W. 2020.  Quantifying Attention Flow in Transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4190–4197. Online: Association for Computational Linguistics.
  * Ancona et al. (2018) Ancona, M.; Ceolini, E.; Öztireli, C.; and Gross, M. 2018.  Towards better understanding of gradient-based attribution methods for Deep Neural Networks.  In _6th International Conference on Learning Representations (ICLR)_ , 1711.06104, 0–0. Arxiv-Computer Science.
  * Atanasova et al. (2020) Atanasova, P.; Simonsen, J. G.; Lioma, C.; and Augenstein, I. 2020.  A Diagnostic Study of Explainability Techniques for Text Classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 3256–3274. Online: Association for Computational Linguistics.
  * Attanasio et al. (2023) Attanasio, G.; Pastor, E.; Di Bonaventura, C.; and Nozza, D. 2023.  ferret: a Framework for Benchmarking Explainers on Transformers.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , 256–266. Dubrovnik, Croatia: Association for Computational Linguistics.
  * Bach et al. (2015) Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; and Samek, W. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7): e0130140.
  * Bashier, Kim, and Goebel (2020) Bashier, H. K.; Kim, M.-Y.; and Goebel, R. 2020.  RANCC: Rationalizing Neural Networks via Concept Clustering.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 3214–3224. Barcelona, Spain (Online): International Committee on Computational Linguistics.
  * Bastings, Aziz, and Titov (2019) Bastings, J.; Aziz, W.; and Titov, I. 2019.  Interpretable Neural Predictions with Differentiable Binary Variables.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2963–2977. Florence, Italy: Association for Computational Linguistics.
  * Bastings and Filippova (2020) Bastings, J.; and Filippova, K. 2020.  The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , 149–155. Online: Association for Computational Linguistics.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Chrysostomou and Aletras (2022) Chrysostomou, G.; and Aletras, N. 2022.  Flexible instance-specific rationalization of nlp models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 10545–10553.
  * Cífka and Liutkus (2023) Cífka, O.; and Liutkus, A. 2023.  Black-box language model explanation by context length probing.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 1067–1079. Toronto, Canada: Association for Computational Linguistics.
  * Denil, Demiraj, and De Freitas (2014) Denil, M.; Demiraj, A.; and De Freitas, N. 2014.  Extraction of salient sentences from labelled documents.  _arXiv preprint arXiv:1412.6815_.
  * DeYoung et al. (2020) DeYoung, J.; Jain, S.; Rajani, N. F.; Lehman, E.; Xiong, C.; Socher, R.; and Wallace, B. C. 2020.  ERASER: A Benchmark to Evaluate Rationalized NLP Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4443–4458. Online: Association for Computational Linguistics.
  * Ding and Koehn (2021) Ding, S.; and Koehn, P. 2021.  Evaluating Saliency Methods for Neural Language Models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5034–5052. Online: Association for Computational Linguistics.
  * Gehrmann et al. (2021) Gehrmann, S.; Adewumi, T.; Aggarwal, K.; Ammanamanchi, P. S.; Anuoluwapo, A.; Bosselut, A.; Chandu, K. R.; Clinciu, M.; Das, D.; Dhole, K. D.; et al. 2021.  The gem benchmark: Natural language generation, its evaluation and metrics.  _arXiv preprint arXiv:2102.01672_.
  * Grave, Obozinski, and Bach (2014) Grave, É.; Obozinski, G.; and Bach, F. 2014.  A Markovian approach to distributional semantics with application to semantic compositionality.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , 1447–1456. Dublin, Ireland: Dublin City University and Association for Computational Linguistics.
  * Hase, Xie, and Bansal (2021) Hase, P.; Xie, H.; and Bansal, M. 2021.  The out-of-distribution problem in explainability and search methods for feature importance explanations.  _Advances in neural information processing systems_ , 34: 3650–3666.
  * Jacovi and Goldberg (2020) Jacovi, A.; and Goldberg, Y. 2020.  Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4198–4205. Online: Association for Computational Linguistics.
  * Jain et al. (2020) Jain, S.; Wiegreffe, S.; Pinter, Y.; and Wallace, B. C. 2020.  Learning to Faithfully Rationalize by Construction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4459–4473. Online: Association for Computational Linguistics.
  * Kersten et al. (2021) Kersten, T.; Wong, H. M.; Jumelet, J.; and Hupkes, D. 2021.  Attention vs non-attention for a Shapley-based explanation method.  In _Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , 129–139. Online: Association for Computational Linguistics.
  * Kindermans et al. (2019) Kindermans, P.-J.; Hooker, S.; Adebayo, J.; Alber, M.; Schütt, K. T.; Dähne, S.; Erhan, D.; and Kim, B. 2019.  The (un) reliability of saliency methods.  _Explainable AI: Interpreting, explaining and visualizing deep learning_ , 267–280.
  * Kindermans et al. (2016) Kindermans, P.-J.; Schütt, K.; Müller, K.-R.; and Dähne, S. 2016.  Investigating the influence of noise and distractors on the interpretation of neural networks.  _arXiv preprint arXiv:1611.07270_.
  * Kokhlikyan et al. (2021) Kokhlikyan, N.; Miglani, V.; Alsallakh, B.; Martin, M.; and Reblitz-Richardson, O. 2021.  Investigating sanity checks for saliency maps with image and text classification.  _arXiv preprint arXiv:2106.07475_.
  * Lal et al. (2021) Lal, Y. K.; Chambers, N.; Mooney, R.; and Balasubramanian, N. 2021.  TellMeWhy: A Dataset for Answering Why-Questions in Narratives.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , 596–610. Online: Association for Computational Linguistics.
  * Lebret and Collobert (2014) Lebret, R.; and Collobert, R. 2014.  Word Embeddings through Hellinger PCA.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , 482–490. Gothenburg, Sweden: Association for Computational Linguistics.
  * Lei, Barzilay, and Jaakkola (2016) Lei, T.; Barzilay, R.; and Jaakkola, T. 2016.  Rationalizing Neural Predictions.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 107–117. Austin, Texas: Association for Computational Linguistics.
  * Li et al. (2015) Li, J.; Chen, X.; Hovy, E.; and Jurafsky, D. 2015.  Visualizing and understanding neural models in NLP.  _arXiv preprint arXiv:1506.01066_.
  * Lundberg and Lee (2017) Lundberg, S. M.; and Lee, S.-I. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30.
  * Manakul, Liusie, and Gales (2023) Manakul, P.; Liusie, A.; and Gales, M. J. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Mikolov et al. (2013) Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Nguyen (2018) Nguyen, D. 2018.  Comparing Automatic and Human Evaluation of Local Explanations for Text Classification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 1069–1078. New Orleans, Louisiana: Association for Computational Linguistics.
  * Nielsen et al. (2022) Nielsen, I. E.; Dera, D.; Rasool, G.; Ramachandran, R. P.; and Bouaynaya, N. C. 2022\.  Robust Explainability: A tutorial on gradient-based attribution methods for deep neural networks.  _IEEE Signal Processing Magazine_ , 39(4): 73–84.
  * Ó Séaghdha and Copestake (2008) Ó Séaghdha, D.; and Copestake, A. 2008.  Semantic Classification with Distributional Kernels.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , 649–656. Manchester, UK: Coling 2008 Organizing Committee.
  * Pham et al. (2022) Pham, T.; Bui, T.; Mai, L.; and Nguyen, A. 2022.  Double Trouble: How to not Explain a Text Classifier’s Decisions Using Counterfactuals Synthesized by Masked Language Models?  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 12–31. Online only: Association for Computational Linguistics.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Ribeiro, Singh, and Guestrin (2016) Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016.  ” Why should i trust you?” Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 1135–1144.
  * Sarti et al. (2023) Sarti, G.; Feldhus, N.; Sickert, L.; and van der Wal, O. 2023.  Inseq: An Interpretability Toolkit for Sequence Generation Models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , 421–435. Toronto, Canada: Association for Computational Linguistics.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Serrano and Smith (2019) Serrano, S.; and Smith, N. A. 2019.  Is Attention Interpretable?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2931–2951. Florence, Italy: Association for Computational Linguistics.
  * Sundararajan, Taly, and Yan (2017) Sundararajan, M.; Taly, A.; and Yan, Q. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , 3319–3328. PMLR.
  * Vafa et al. (2021) Vafa, K.; Deng, Y.; Blei, D.; and Rush, A. 2021.  Rationales for Sequential Predictions.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 10314–10332. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Wang (2021) Wang, B. 2021.  Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang and Komatsuzaki (2021) Wang, B.; and Komatsuzaki, A. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Weeds, Weir, and McCarthy (2004) Weeds, J.; Weir, D.; and McCarthy, D. 2004.  Characterising Measures of Lexical Distributional Similarity.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , 1015–1021. Geneva, Switzerland: COLING.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Le Scao, T.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Zaidan, Eisner, and Piatko (2007) Zaidan, O.; Eisner, J.; and Piatko, C. 2007.  Using “Annotator Rationales” to Improve Machine Learning for Text Categorization.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , 260–267. Rochester, New York: Association for Computational Linguistics.
  * Zeiler and Fergus (2014) Zeiler, M. D.; and Fergus, R. 2014.  Visualizing and understanding convolutional networks.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_ , 818–833. Springer.
  * Zhang et al. (2022) Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022.  OPT: Open Pre-trained Transformer Language Models.  arXiv:2205.01068.
  * Zhao and Aletras (2023) Zhao, Z.; and Aletras, N. 2023.  Incorporating Attribution Importance for Improving Faithfulness Metrics.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 4732–4745. Toronto, Canada: Association for Computational Linguistics.
  * Zhao et al. (2022) Zhao, Z.; Chrysostomou, G.; Bontcheva, K.; and Aletras, N. 2022.  On the Impact of Temporal Concept Drift on Model Explanations.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , 4039–4054. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  * Zhou, Ribeiro, and Shah (2022) Zhou, Y.; Ribeiro, M. T.; and Shah, J. 2022.  ExSum: From Local Explanations to Model Understanding.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5359–5378. Seattle, United States: Association for Computational Linguistics.