#  XPrompt: Explaining Large Language Model’s Generation via Joint Prompt
Attribution

Yurui Chang, Bochuan Cao, Yujia Wang, Jinghui Chen, Lu Lin
College of Information Sciences and Technology
Pennsylvania State University
University Park, PA 16802
{yuruic, bccao, yjw5427, jzc5917<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) have demonstrated impressive performances in
complex text generation tasks. However, the contribution of the input prompt
to the generated content still remains obscure to humans, underscoring the
necessity of elucidating and explaining the causality between input and output
pairs. Existing works for providing prompt-specific explanation often confine
model output to be classification or next-word prediction. Few initial
attempts aiming to explain the entire language generation often treat input
prompt texts independently, ignoring their combinatorial effects on the
follow-up generation. In this study, we introduce a counterfactual explanation
framework based on joint prompt attribution, XPrompt, which aims to explain
how a few prompt texts collaboratively influences the LLM’s complete
generation. Particularly, we formulate the task of prompt attribution for
generation interpretation as a combinatorial optimization problem, and
introduce a probabilistic algorithm to search for the casual input combination
in the discrete space. We define and utilize multiple metrics to evaluate the
produced explanations, demonstrating both faithfulness and efficiency of our
framework.

##  1 Introduction

Large Language Models (LLMs), such as GPT4 [1], LLaMA [2] and Claude [3], have
shown excellent performance in various natural language generation tasks
including question answering, document summarization, and many more. Despite
the great success of LLMs, we still have very limited understanding on the LLM
generation behavior – which parts in the input cause the model to generate a
certain sequence. Unable to explain the causality between the input prompt and
the output generation could cause failure in recognizing potential unintended
consequences, such as harmful response [4, 5, 6, 7] and biased generation [8]
attributed to a specific malicious description in the input. These issues
undermine human trust in model usage, thus highlighting a pressing need for
developing an interpretation tool that attributes how an input prompt leads to
the generated content.

Explaining LLM generation through prompt attribution involves extracting the
most influential prompt texts on the model’s entire generation procedure, a
realm that remains relatively under-explored in current research. While
extensive works on input attribution are proposed for text classification
interpretation [9, 10] and next-word generation rationale [11, 12], they can
not be directly applied to explain the full generation sequence due to its
complicated joint probability landscape and the autoregressive generation
procedure. The complexity for interpreting LLM generation compounds as the
model size increases.

Another line of works involves prompting LLMs to self-explain their behaviors
[13]. This method relies on the model’s innate reasoning capabilities,
although current findings suggest that these capabilities may not always be
faithful [14, 15].

There are limited existing attempts focusing on explaining the relationship
between the input prompts and the complete generated sequence. The most
relevant approach, Captum [16], sequentially determines the importance score
for each token by calculating the variations in the joint probability of
generating the targeted output sequence when the token is dropped from the
model input. While being straightforard, this approach treats tokens as
independent features, ignoring their joint semantic influence on the generated
output. In fact, tokens may contain overlapping or complementary information.
For example, given the input prompt: “Write a story about the doctor and his
patient”, the most influential components are “doctor” and “patient”.
Individually removing either of these words would not significantly alter the
generated output, resulting in inaccurately low importance score for each of
them. This is caused by the semantic interaction among these components,
allowing the model to infer the meaning of the omitted one. Existing
attribution methods that ablate each token individually fail to capture such
combinatorial effect. A straightforward remedy might involve exhaustively
assessing all possible combinations to observe the variations in the model
generations, which however is impractical due to the vast search space with
long-context input prompts.

To efficiently search the space for generating accurate prompt explanations,
we develop our framework XPrompt, which provides the counterfactual
explanation to highlight which components of input prompts have the
fundamental effect on the generated context via solving a combinatorial
optimization problem.

We aim to explain the generation behavior of model outputs for any given
prompt while take the joint effects of the prompt components into account.
Assuming that removing the essential parts of the prompt would result in a
significant variation in the model’s output, we propose the novel objective
function and formulate our task of providing faithful counterfactual
explanations for the input prompt as an optimization problem. To quantify the
influence of token combinations in the prompt on the generations, we
incorporate a mask approach for joint prompt attribution. Thus, our goal of
extracting the explanations has been converted to finding the optimal mask of
the input prompt. We solve this problem by a probabilistic search algorithm,
equipped with gradient guidance and probabilistic updaten for efficient
exploration in the discrete solution space. Our main contributions could be
summarized as follows:

  * •

We propose a general interpretation scheme for LLM generation task that
attributes input prompts to the entire generation sequence. Notably, this
recipe considers the joint influence of input token combinations on the
generation. This motivation naturally formulates a combinatorial optimization
problem for explaining generation with the most influential prompt texts.

  * •

We demonstrate XPrompt, an efficient probabilistic search algorithm to solve
the optimization problem. XPrompt works by searching better token combinations
that lead to larger generation changes. It takes the advantage of both the
gradient information and the probabilistic search-space strategy, thereby
achieving an efficient prompt interpretation tool.

  * •

Our framework demonstrates strong performance on language generation tasks
including text summarization, question-answering, and general instruction
datasets. The faithfulness of our explanations is evaluated based on a suit of
comprehensive metrics considering generation probability, word frequency, and
semantic similarity, verifying the transferability and effectiveness of our
methods across a variety of tasks.

##  2 Related Work

While there are extensive works devoted to explaining language models in the
context of text classification tasks [17, 18, 19], relatively few attempts
[11, 12] are proposed to investigate the importance of prompt texts on the
entire generation procedure especially for LLMs. This demonstrates a research
gap that this work aims to fill in.

#### Explaining Language Generation

There are many work explaining the predictions generated by LLMs by measuring
the importance of the input features to the model’s prediction on the
classification tasks. One group of studies perturb the specific input by
removing, masking, or altering the input features, and evaluate the model
prediction changes [20, 21]. The other group of works, such as integrated
gradients (IG) [22] and first-derivative saliency [23], leverage the gradients
of the output with respect to the input to determine the input feature
importance.

Moreover, a few studies utilize the surrogate model to explain the individual
predictions of the black-box models, and the representative method is called
LIME [24]. As for explaining the model’s generation behavior, Captum [16]
calculates the token’s importance score by sequentially measuring the
contribution of input token to the output, which lacks of the accounting for
the semantic relationships between tokens. Another work, ReAGent [11], focuses
on the next-word generation task, computing the importance distribution for
the next token position. This method ignores the contextual dependencies in
the the generated output and could not adequately account for dynamically
changing generations. Our framework aims to interpret the joint effects of the
input prompts on the entire output contexts with considering of the textual
information covered the input prompt.

#### Self-explaining by Prompting

As language models increase in scale, prompting-based models demonstrate
remarkable abilities in reasoning [25], creativity [26], and adaptability
across a range of tasks [27]. However, the complex reasoning processes of
these models remain elusive and require tailored paradigm to better understand
the prompting mechanism. For instance, the chain-of-thought (CoT) paradigm
could explain the LLM behaviors by prompting the model to generate the
reasoning chain along with the answers [13], as pre-trained LLMs have
demonstrated a certain ability to self-explain their behaviors. However,
recent studies have also suggested that the reasoning chain does not guarantee
faithful explanations of the model’s behavior [28] and the final answer might
not always follow the generated reasoning chain [14]. xLLM [29] enhances the
fidelity of explanation derived from LLMs via a fidelity evaluator, which
however is designed for classification tasks. Our efforts are concentrated on
explaining LLM generation by analyzing the attribution of the input prompts to
the output content, without depending on the model’s innate reasoning ability
that are not yet satisfactory.

##  3 Preliminaries

We first introduce necessary notions depicting the LLM generation process, and
then discuss limitations of prior attempts explaining the entire generation
via prompt attribution.

#### Notions on LLM Generation

Denote a specific input prompt as a sequence of tokens
$\bm{x}=(x_{1},\dots,x_{T})$, $x_{i}\in\\{1,2,\dots,|V|\\}$, where $|V|$
represents the vocabulary size, $T$ is the length of the input sequence and
the set of all token indices is $\mathcal{I}=\\{1,2,\dots,T\\}$. The
corresponding generated output $\bm{y}$ could be represented as a sequence of
tokens $\bm{y}=(y_{1},\dots,y_{S})$ with $y_{j}\in\\{1,2,\dots,|V|\\}$. The
output tokens $\\{y_{i}\\}_{i=1}^{S}$ are generated from the LLM
$f_{\bm{\theta}}$ parameterized by $\bm{\theta}$ in an autoregressive manner
with the probability $p_{\bm{\theta}}(\bm{y}|\bm{x})$ as:

$\displaystyle
p_{\bm{\theta}}(\bm{y}|\bm{x})=p_{\bm{\theta}}(y_{1}|\bm{x})\prod_{i=1}^{S-1}p_{\bm{\theta}}(y_{i+1}|\bm{x},y_{i}).$
(1)

The probability of generating the output text $\bm{y}$ given the input prompt
$\bm{x}$ illustrates that the coherence and generation of the output texts
heavily rely on the input prompt $\bm{x}$, indicating an implicit causal
relationships between the input prompt $\bm{x}$ and the output $\bm{y}$.

#### Limitation of Prior Interpretation Attempt

There are limited prior works focus on explaining the relationship between the
individual input prompts and the entire generated sequence, and their
faithfulness is limited by treating input tokens independently. Specifically,
Captum [16], calculates the importance score for each token by slightly
perturbing the input $\bm{x}$ at the $i$-th token:

$G_{i}(\bm{x};\bm{\theta})=p_{\bm{\theta}}(\bm{y}|\bm{x})-p_{\bm{\theta}}(\bm{y}|\bm{x}_{\mathcal{I}\setminus\\{i\\}}),$
(2)

where $\mathcal{I}$ denotes all token indices. Through Eq. (2), one can
calculate an attribution score for each token $i$ indicating its importance
towards the original generation result $\bm{y}$. Such method is direct and
simple, however, it treats tokens as independent features, ignoring their
semantic interaction and joint effect on the generated output. To make it more
obvious, let’s recall the previous example of “Write a story about a doctor
and his patient.”. Note that since “doctor” and “patient” are semantically
correlated, and removing any of the two words would not have a significant
influence on the generated response, their corresponding importance score will
not be too high. On the contrary, the most important token detected by this
approach would be “his”, which is clearly not ideal.

These observations inspire us to develop a new prompt attribution method that
considers the semantic correlation among tokens. Specifically, we assume that
the content generated by LLMs is primarily influenced by a subset of tokens
jointly. The remaining tokens serve as auxiliary or potentially less relevant
information. Those subset of tokens, which fundamentally shape the model’s
output, are viewed as the explanatory tokens for the generated content. In
other words, explanatory tokens do not affect the model output independently,
but jointly contribute to the generated responses.

##  4 Proposed Method

In this section, we propose XPrompt, a simple yet effective framework designed
for generative tasks to explain the attribution of the input prompt by solving
a discrete optimization problem. We start with formulating the general
objective for the discrete optimization problem, and then we introduce our
proposed probabilistic search algorithm for solving the problem.

Figure 1: Overview of XPrompt. Left: Demonstrating the pipline of the
algorithm. Right: Illustrating the process of mask $\bm{m}$ sampling.

#### Problem Formulation for Joint Prompt Attribution

In order to explain language generation via prompt attribution, instead of
treating the prompt tokens independently as in previous works [16], we propose
to evaluate the joint effect of $k$ prompt tokens on the generated sequence.

Specifically, consider a binary prompt mask $\bm{m}=(m_{1},\dots,m_{T})$ in
which $m_{i}\in\\{0,1\\}$ indicates whether the $i$-th token is important or
not. The joint probability of generating the original output sequence $\bm{y}$
given an input masked context $\bm{m}\odot\bm{x}$ is computed as
$p_{\bm{\theta}}(\bm{y}|\bm{m}\odot\bm{x})$, where $\odot$ denotes the
Hadamard product. In our paper, we aim to identify a binary mask $\bm{m}$ that
maximizes the discrepancy in the probability of generating the same output
$\bm{y}$ when comparing the masked input $\bm{m}\odot\bm{x}$ to the original
input $\bm{x}$. A large discrepancy indicates that the masked token
combination are the most influential components for generating $\bm{y}$, thus
should jointly serve as the attributed explanation. Consequently, this
involves training the binary mask $\bm{m}$ to optimize the following objective
function:

$\displaystyle\max_{\bm{m}\in\\{0,1\\}^{T}}\mathcal{L}(\bm{m},\bm{x},\bm{y};\bm{\theta}):$
$\displaystyle=p_{\bm{\theta}}\left(\bm{y}|\bm{x}\right)-p_{\bm{\theta}}\left(\bm{y}|\bm{m}\odot\bm{x}\right)$
(3) $\displaystyle\text{s.t.}\quad|\bm{m}|_{1}$ $\displaystyle=T-k,$

where $k$ represents the number of explanatory tokens. Intuitively, optimizing
the objective outlined in Eq. (3) suggests that we want to find the top $k$
important tokens which, if they are masked, lead to a substantial variation in
model’s output probabilities. Compared with prior methods for individual token
attribution stated in Eq. (2), our formulation in Eq. (3) measures the joint
attribution of a subset of tokens, capturing token interactions to enable more
accurate prompt explanations.

#### Challenges in Solving Eq. (3)

Note that Eq. (3) is a constraint discrete optimization problem that is non-
trivial to solve. One naive solution would be transforming the discrete
optimization problem into a continuous one, i.e., let $\bm{m}\in[0,1]^{T}$,
and formulate the constraint into a regularization term. Then one can adopt
traditional gradient descent based optimization solutions to solve the
problem. However, such a strategy would require extensive gradient
calculations and backward steps on the original inputs, which can be
inefficient in practice especially for LLMs. Moreover, the obtained continuous
mask is not the final explanation we want. In fact, the approximation error
when transforming the continuous mask back into the discrete space can also be
quite significant, leading to worse performances. Another straightforward
solution involves searching through all possible token combinations
exhaustively. However, this could be impractical as well due to the enormous
search space especially while facing long-context inputs. Therefore, we hope
to develop a new method that adopts a search-based strategy to simplify the
algorithm design and satisfy our constraints, while also leveraging gradient
information for efficient optimization.

Algorithm 1 Explainable Prompt Generator: XPrompt

Input: Input tokens $\bm{x}$, output tokens $\bm{y}$, the integer $k$ denoting
the number of explanatory tokens, $1\leq k\leq T$, input mask
$\bm{m}^{(0)}=\textbf{1}$, and the sampling numbers $N$.

Output: Optimal mask $\bm{m}^{(N)}$

1:$\bm{g}=|\nabla_{\bm{m}^{(0)}}\mathcal{L}(\bm{m}^{(0)},\bm{x},\bm{y};\bm{\theta})|$

2:Set $\bm{m}^{(1)}$ as the $\text{top-}k$ value mask for $\bm{g}$:
$\bm{m}^{(1)}=\bm{m}^{(0)};m_{i}^{(1)}=0,\forall
i\in\mathop{\mathrm{arg\,top}\,k}(\bm{g})$

3:for $n=1$ to $N$ do

4: Sample $l\sim\text{softmax}\big{(}\bm{m}^{(n)}\odot\bm{g}\big{)}$

5: Sample
$v\sim\text{softmax}\big{(}(\mathbf{1}-\bm{m}^{(n)})\odot\bm{g}\big{)}$

6: $\bm{m}^{\text{tmp}}=\bm{m}^{(n)}.\text{copy}()$; switch the value of
${m}^{\text{tmp}}_{l}$ and ${m}^{\text{tmp}}_{v}$

7: if
$p_{\bm{\theta}}(\bm{y}|{\bm{m}^{\text{tmp}}}\odot\bm{x})<p_{\bm{\theta}}(\bm{y}|{\bm{m}^{(n)}}\odot\bm{x})$
then

8: $\bm{m}^{(n+1)}=\bm{m}^{\text{tmp}}$

9: else

10: $\bm{m}^{(n+1)}=\bm{m}^{(n)}$

11: end if

12:end for

#### Proposed Probabilistic Search Algorithm

To tackle this challenge, we propose XPrompt, a novel explainable prompt
generator, for efficiently obtaining an optimal solution for Eq. (3). We
summarize our XPrompt in Algorithm 1. The high-level pipeline is illustrated
in Figure 1 left: we initialize and maintain exactly $k$ entries in the mask
$\bm{m}$ to be zero to enforce the constraint and capture their joint
influence of being masked; the mask is then iteratively updated by sampling
indexes for value swapping, searching towards the direction with increased
generation loss $\mathcal{L}(\bm{m},\bm{x},\bm{y};\bm{\theta})$. At the core
of this pipeline is the sampling and update of the discrete mask, which
demands an efficient exploration in the vast search space. Figure 1 right
shows this step, which is featured by the following two essential components,
the gradient-guided masking and the probabilistic search update.

Gradient-Guided Masking: Trivial solutions that set the mask $\bm{m}$ by
uniformly random could cost massive sampling to hit the right optimization
direction. Gradient is a common indicator of feature importance, as evidenced
in existing practices [30, 6, 31]. Therefore, we propose to use gradient as a
guidance to set and sample the mask for more efficient optimization.
Specifically, we begin with the binary mask $\bm{m}^{(0)}=\bm{1}$ which
indicates that all tokens in the input $\bm{x}$ are marked as non-explanatory
ones. Then we compute the gradient
$\nabla_{\bm{m}^{(0)}}\mathcal{L}(\bm{m}^{(0)},\bm{x},\bm{y};\bm{\theta})$ of
the loss function in Eq. (3), and denote the magnitude of gradients as
$\bm{g}=|\nabla_{\bm{m}^{(0)}}\mathcal{L}(\bm{m}^{(0)},\bm{x},\bm{y};\bm{\theta})|$.
Note that the components with larger gradient magnitudes in $\bm{g}$ imply
that altering the corresponding tokens could result in a sharper change to the
generated outputs. In order to initialize the explanatory $k$ tokens guided by
the gradient magnitude, we set the binary mask $\bm{m}^{(1)}$ where
$m^{(1)}_{i}=0$ indicates $g_{i}$ is the top-$k$ value in $\bm{g}$. The
gradient guidance and mask initialization are obtained following Line 1-2 in
the algorithm.

Probabilistic Search Update: While gradient is informative, greedily
determining explanatory tokens by top gradients lacks exploration, leading to
suboptimal solutions. Therefore, we propose a probabilistic search mechanism
for mask sampling and update. Specifically when updating the current binary
mask $\bm{m}$, we iteratively sample a non-zero entry $l$ from the mask and
swap its value with a sampled zero entry $v$ to explore a new (and potentially
better) solution for the mask. Particularly, the sampling is also guided by
the gradient calculated before: the non-zero entry in the mask (representing
non-explanatory tokens) is sampled following probabilities calculated by the
normalized gradient magnitudes, i.e., softmax($\bm{m}^{(n)}\odot\bm{g}$), and
we employ a similar sampling strategy for zero entries (explanatory tokens).
After swapping the mask indicators for the two sampled tokens $l$ and $v$, we
generate a temporary mask $\bm{m}^{\text{tmp}}$. We then evaluate whether this
temporary mask leads to a decrease in output probability. If it does, we
update the binary mask to $\bm{m}^{\text{tmp}}$, otherwise we leave the binary
mask as is. Consequently, without requiring intensive gradient computations,
these sampling iterations keep discovering improved solutions for the discrete
optimization problem shown in Eq. (3). We conclude this update by sampling
process in Line 3-12.

While capturing the joint influence of being masked, the proposed XPrompt
takes the advantage of both the gradient information and the search-space
strategy, thereby achieving better efficiency than adopting either method
alone. XPrompt requires only a single step of gradient calculation and avoids
the need to convert between discrete and continuous masks. The obtained
gradient information provides a favorable initial searching direction and
reliable sampling probabilities that enhances the search efficiency.

##  5 Experiment

This section aims to verify the effectiveness and efficiency of our proposed
framework for interpreting the LLMs on the generation task. We conduct the
experiments to answer the following questions:

  * •

Q1: Do the generated prompt attributions play a predominant role in the
model’s generation, thereby serving as faithful counterfactual explanations
for the generated output?

  * •

Q2: Is our interpretation algorithm efficient for practical usage?

  * •

Q3: Could the proposed algorithm effectively identify a combination of tokens
that impose joint influence on the model generation?

We provide quantitative studies to evaluate the faithfulness of the
explanatory prompt fragments generated by XPrompt, comparing with existing
interpretation baselines and ablation variants.

###  5.1 Experiment Settings

#### Models & Baselines

In the experiment, we employ two LLMs as our targeted
$f_{\bm{\theta}}(\cdot)$: LLaMA2 (7B-Chat) [2] and Vicuna (7B) [32]. There are
relatively few methods attributing prompts on the entire language generation,
and we choose random removal (Random), Integrated-Gradient [22] and Captum
[16] as our baseline for comparison. We implement these models using the
PyTorch framework and pretrained weights from the transformers Python library
[33], and conduct our experiments on an Nvidia RTX A6000-48GB platform with
CUDA version 12.0.

#### Datasets

We employ three distinct text generation datasets: Alpaca [34], tldr_news
[35], and MHC [36], to evaluate the effectiveness of our method across various
generation tasks. As we aim to capture the joint influence of prompts on the
model generations, we focus on relatively long-context prompts rather than
simple one-sentence prompts. Longer prompts tend to contain more diverse
vocabularies, convey more information, and thus more likely to exhibit high
textual correlation.

For evaluation purposes, we randomly select approximately 110 data samples
with at least 15 words from each dataset. All datasets are publicly available
and more details about the dataset are illustrated in Appendix A.1.

###  5.2 Evaluation Metrics

Faithfulness scores are a key metric for assessing the quality of
explanations, with faithful explanations accurately reflecting the model’s
decision-making process [28]. Studies [37, 38] suggest that if a certain input
tokens are truly important, their removal should lead to a more significant
change in model output than the removal of random tokens [39]. Therefore,
after removing explanatory tokens identified by a faithful method, the model’s
new generation would significantly differ from using the original output.
Moreover, the input prompt with masking these explanatory tokens are less
likely to reproduce the original model response $\bm{y}$, indicating a lower
value of $p_{\bm{\theta}}(\bm{y}|\bm{m}\odot\bm{x})$.

To quantitatively evaluate explanation faithfulness, we measure the change of
model generation behavior from two dimensions. On one hand, we compare the
model’s original and new generations: the originally generated sequence
$\bm{y}$ is based on the complete input prompt $\bm{x}$ (e.g.,
$\bm{y}=f_{\bm{\theta}}(\bm{x})$), while the new generation is conditioned on
the masked prompt (e.g.,
$\bm{y}^{\prime}=f_{\bm{\theta}}(\bm{m}\odot\bm{x})$).

We thus measure the similarity of the original generation $\bm{y}$ and the new
generation $\bm{y}^{\prime}$ based on their word frequency and semantics. A
smaller similarity reflects a larger change in model generation, suggesting a
better explanation. On the other hand, we measure the likelihood of generating
the original output $\bm{y}$ when the model uses the masked prompt, e.g.,
changes on $p_{\bm{\theta}}(\bm{y}|\bm{m}\odot\bm{x})$. A smaller likelihood
indicates better explanations whose mask prevents the model from generating
its original output. Detailed definitions of these metrics are explained
below.

Word Frequency: BLEU [40] is widely used to measure how close the candidate
text $\bm{y}^{\prime}$ is to the reference text $\bm{y}$. The score measures
the precision of matching $n$-grams from the text $\bm{y}^{\prime}$ to
$\bm{y}$ by a clipping method to avoid overcounting and adjusting for brevity
of $\bm{y}^{\prime}$ if it is shorter than $\bm{y}$. ROUGE-L [41] is the
metric for measuring the overlap of sequences of words between the two texts,
evaluating how much of the $\bm{y}^{\prime}$ matches with the reference
$\bm{y}$ (precision), how much the reference $\bm{y}$ is covered by the
candidate $\bm{y}^{\prime}$ (recall), and combine them into an F1 score.

Semantic Similarity: SentenceBert [42] is a variation of the BERT model which
is designed to generate high-quality sentence embeddings for the pairs of
sentences. We leverage SentenceBert to transform the text $\bm{y}$ and
$\bm{y}^{\prime}$ into fixed-length embedding vectors, and calculate the
cosine similarity between their embeddings to quantify their semantic
similarity.

Probability Measurement: We define two measurements to reflect how the
likelihood of generating the original text $\bm{y}$ changes before and after
applying the explanation mask. We first define the Probability Ratio (PR) to
indicate how less likely to generate the original output $\bm{y}$ when
explanatory tokens are masked:
$\text{PR}(\bm{x},\bm{y},\bm{m})=\frac{p_{\bm{\theta}}(\bm{y}|\bm{m}\odot\bm{x})}{p_{\bm{\theta}}(\bm{y}|\bm{x})}$.
If the PR score is far below the random baseline, we can conclude that the
masked tokens are indeed important to cause the model generating $\bm{y}$. In
addition, we also calculate the KL-divergence between these two distributions
to measure their difference:
$D_{\text{KL}}\big{(}p_{\bm{\theta}}(\bm{y}|\bm{m}\odot\bm{x})||p_{\bm{\theta}}(\bm{y}|\bm{x})\big{)}$,
and a larger score indicates a larger change in generation likelihood and a
more accurate explanation.

###  5.3 Main Results on Faithfulness (Q1)

We evaluate the faithfulness of the explanatory prompt tokens generated by our
framework on five aforementioned metrics. Table 1 shows the results of
different methods on three datasets by masking $k=3$ identified tokens. For
all metrics except the KL-divergence, a lower score is better which is
annotated as $\downarrow$. In general, we observe that our method consistently
demonstrates better interpretation faithfulness on all datasets compared with
baselines, with a clear margin. In particular, we have the following
observations demonstrating the advantage of our method:

Table 1: Faithfulness Measurement Results for LLaMA-2(7B-Chat) and Vicuna(7B)

Model | Dataset | Methods | Metric(@3)
---|---|---|---
BLEU$\downarrow$ | ROUGE-L$\downarrow$ | SentenceBert$\downarrow$ | PR$\downarrow$ | KL$\uparrow$
Precision | Recall | F1
LLaMA-2 (7B-Chat) | Alpaca | Random | 0.601 | 0.527 | 0.533 | 0.522 | 0.825 | 0.842 | 0.134
Captum | 0.515 | 0.409 | 0.421 | 0.409 | 0.680 | 0.602 | 0.417
Integrated-Gradient | 0.541 | 0.424 | 0.435 | 0.424 | 0.725 | 0.726 | 0.242
XPrompt | 0.482 | 0.388 | 0.386 | 0.379 | 0.642 | 0.549 | 0.504
tldr_news | Random | 0.794 | 0.742 | 0.741 | 0.741 | 0.923 | 0.944 | 0.037
Captum | 0.759 | 0.701 | 0.703 | 0.701 | 0.900 | 0.910 | 0.069
Integrated-Gradient | 0.713 | 0.641 | 0.642 | 0.640 | 0.866 | 0.817 | 0.149
XPrompt | 0.692 | 0.619 | 0.610 | 0.612 | 0.841 | 0.604 | 0.394
MHC | Random | 0.723 | 0.617 | 0.614 | 0.615 | 0.787 | 0.958 | 0.023
Captum | 0.640 | 0.497 | 0.493 | 0.494 | 0.663 | 0.760 | 0.189
Integrated-Gradient | 0.646 | 0.500 | 0.496 | 0.498 | 0.687 | 0.836 | 0.117
XPrompt | 0.575 | 0.403 | 0.405 | 0.403 | 0.602 | 0.710 | 0.238
Vicuna (7B) | Alpaca | Random | 0.587 | 0.541 | 0.552 | 0.535 | 0.786 | 0.891 | 0.079
Captum | 0.466 | 0.435 | 0.427 | 0.418 | 0.649 | 0.654 | 0.347
Integrated-Gradient | 0.541 | 0.495 | 0.503 | 0.488 | 0.744 | 0.835 | 0.135
XPrompt | 0.433 | 0.395 | 0.390 | 0.377 | 0.639 | 0.589 | 0.459
tldr_news | Random | 0.781 | 0.736 | 0.746 | 0.737 | 0.921 | 0.891 | 0.091
Captum | 0.556 | 0.470 | 0.461 | 0.459 | 0.775 | 0.376 | 0.829
Integrated-Gradient | 0.705 | 0.669 | 0.672 | 0.666 | 0.874 | 0.821 | 0.183
XPrompt | 0.536 | 0.456 | 0.454 | 0.448 | 0.772 | 0.317 | 1.006
MHC | Random | 0.715 | 0.625 | 0.623 | 0.623 | 0.810 | 0.972 | 0.012
Captum | 0.579 | 0.438 | 0.432 | 0.433 | 0.627 | 0.811 | 0.120
Integrated-Gradient | 0.672 | 0.559 | 0.555 | 0.556 | 0.762 | 0.941 | 0.030
XPrompt | 0.575 | 0.431 | 0.425 | 0.427 | 0.620 | 0.783 | 0.141

#### Slight random perturbations on the input prompt do not significantly
alter the model’s output, validating the soundness of our approach.

This is evident from the high PR value of around 0.958 and the low KL value of
only 0.023 on the MHC dataset when randomly masking tokens. Only when
essential tokens are perturbed, particularly when masked, does the generation
of content and probability change substantially. The noticeable gaps in these
metrics between XPrompt and Random underscore the genuine importance of the
masked tokens and their role as counterfactual explanations for the model
output.

####  XPrompt can effectively capture semantically important prompt fragments,
and the advantage stands out for long context.

Compared to Captum and Integrated-Gradient, our method generally yields lower
values for BLEU and ROUGE-L across all datasets. The discrepancies between
XPrompt and Captum are more pronounced for datasets with longer text, such as
tldr_news and MHC. Specifically, on the tldr_news dataset, the F1-score of
XPrompt is 12.7% lower than Captum for the LLaMA-2 model, and on the MHC
dataset, it is 17.85% lower, this all indicates that XPrompt finds and removes
the more important token. Furthermore, the SentenceBert results are better on
all datasets, indicating larger semantic variations in the generated output
after removing tokens by XPrompt.

####  XPrompt works even better on stronger LLMs.

The gaps between XPrompt and Captum are less apparent on the Vicuna-7B model
compared to the LLaMA-2-7B-chat model, which could be attributed to the
model’s inherent inference capabilities. Our method is based on the premise
that there are textual correlations among the input tokens, allowing the model
to infer from the remaining content when a portion of the tokens is masked. If
the model has a poor ability to infer the masked token, it may struggle to
capture the contextual information. Consequently, the effectiveness of our
method is tied to the LLM’s proficiency in inference and understanding.

###  5.4 Time Efficiency (Q2)

Table 2: Time Efficiency on LLaMA-2(7B-Chat)

| Method | Dataset
---|---|---
| Alpaca | tldr_news | MHC
Time(s) | Captum | 1169.648 | 1727.602 | 1806.551
XPrompt | 15.225 | 15.397 | 14.473

In Table 2, we compare the average time cost of XPrompt and Captum for
generating $k=3$ explanatory tokens for each prompt instance. Note that since
Captum’s design requires sequentially appending the next token to the input
prompt to re-generate the new output, the time consumption of Captum would
increase significantly when the prompt is long. As the average length of MHC
is longer than the Alpaca shown in Table 3, the computational time increases
from 1169.648$s$ to 1806.551$s$ for Captum, which is quite inefficient and
impractical. As for comparison, our algorithm solves the combinatorial
optimization problem efficiently via the proposed probabilistic search
algorithm, which significantly reduce the computational cost of the
explanation generation. And also since our design only needs to perform the
gradient-guided probabilistic search step for a certain number of times, the
computation time remains consistent regardless of the prompt length as can be
shown form Table 3.

###  5.5 Qualitative Visualization (Q3)

Figure 2 uses a case study to showcase our method can effectively identify
interacted tokens. This figure illustrates the identified explanatory tokens
by Captum and our method XPrompt. While there are few overlaps in the found
tokens, compared with our method, those found by Captum are mostly not
important to the generated response. This demonstrates the Captum’s limited
ability to explain the relationship between input prompt and output response,
especially considering the textual correlations within input tokens. For
instances, the tokens “publishing”, “publish” and “book” have semantic
correlations, thus Captum masking one by one individually cannot really
eliminate this information complementarily provided by the others, especially
when the LLM has a certain context inference ability. Similar issues exist in
the second case, where “doctor” and “patient” has semantic interactions.
Therefore, treating tokens independently and ignoring their joint influence on
the generation is not a favored choice for prompt attribution.

This case study highlights the importance of our formulation of joint
attribution, and verifies the effectiveness of our algorithm in discovering
important token combinations on the generated output. More examples are shown
in Appendix. A.2

Figure 2: Case study for visualizing the explanation of the model responses.

###  5.6 Ablation Study

#### Number of Explanatory Tokens.

Figure 3 shows the Probability Ratio (PR) score as the number of masked tokens
$k$ changes on each dataset for LLaMA-2(7B-Chat). Our algorithm consistently
outperforms other baseline methods even with larger $k$, demonstrating the
stable performance and effectiveness of XPrompt. After masking $k=2$ tokens,
the value of PR would decrease dramatically, indicating that these two tokens
are crucial for generating the output $\bm{y}$ and they act as the triggers
that alter the probability distribution of the output. As the number of tokens
increase to $k=5$, the PR keeps decreasing but with a less sharp slope. This
suggests that the probability of generating the original output $\bm{y}$ is
significantly determined by a few predominant tokens.

Figure 3: Probability Ratio with varying number of masked tokens $k$.

Figure 4: Convergence plot of XPrompt.

#### Number of Sampling Iterations.

We now demonstrate the convergence of our search algorithm. For each sampling
iteration, we sample entries in the mask for value swapping; the mask will be
updated if the swapping leads to a drop in the generation log-likelihood $\log
p_{\bm{\theta}}(\bm{y}|\bm{m}\odot{\bm{x}})$. Figure 4 empirically shows how
the log-likelihood decreases as the sampling and mask update continue. The
quickly decreasing trend demonstrates that our algorithm is successfully
performed to improve the quality of mask in locating the predominant tokens on
generating $\bm{y}$.

#### Effectiveness of Gradient Guidance.

Recall that in our algorithm, we use gradient as guidance to initialize the
mask (e.g., $\bm{m}^{(1)}$) and calculate sampling probability (i.e.,
softmax($\bm{m}^{(n)}\odot\bm{g}$)). To verify the efficiency of gradient
guidance, we compare XPrompt with two variants: w/o Initialization that
initializes the mask by uniformly random instead of gradient, and w/o
Probability that samples swapping entries by uniformly random instead of
gradient. Table 4 in the Appendix reports their resulting generation log-
likelihood $\log p_{\bm{\theta}}(\bm{y}|\bm{m}\odot{\bm{x}})$ in different
iterations.

We observe that without using gradient for initialization, the randomly
initialized mask starts from a worse point with a high generation likelihood;
and without using gradient to guide the sampling, the mask is updated in less
effective direction to explore the search space, resulting in a high
generation likelihood in the end. These results show that gradient is a useful
and efficient tool for initializing the optimization from a better starting
point and guiding the search to a better optimal point.

##  6 Conclusions

In this study, we introduce XPrompt, an efficient probabilistic search
algorithm designed to generate the prompt attributions that elicit the model
outputs for the generation tasks. We tackle the challenge of explaining the
generation behavior for any given prompt by analyzing the joint effect of the
prompt attributions on the output. We frame this explanation task as a
discrete optimization problem, which can be efficiently solved by our proposed
probabilistic search algorithm. This methodology enables efficient generation
of any arbitrary number of explanatory prompt attributions that are
deterministic to the generated content. Our framework is rigorously evaluated
across extensive language generation tasks, including text summarization,
question-answering, and general instruction datasets. The faithfulness of the
explanatory prompt attributions is thoroughly analyzed and assessed by
comprehensive metrics. The results demonstrate that our proposed method
efficiently generates explanatory attributions that faithfully reflect the
model’s generation behavior for the specific prompt. These explanatory
attributions interact in semantically and jointly influence the output
generation. Furthermore, the overall excellent performance of our method on
these diverse datasets highlights its remarkable transferability.

## References

  * [1] OpenAI, J. Achiam, S. Adler, et al.  Gpt-4 technical report, 2024.
  * [2] Touvron, H., L. Martin, K. Stone, et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [3] Anthropic.  Introducing the next generation of claude, 2024.  Accessed: 2024-05-09.
  * [4] DAN.  Chat gpt "dan" (and other "jailbreaks"), 2023.  GitHub repository.
  * [5] Liu, X., N. Xu, M. Chen, et al.  Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2023.
  * [6] Zou, A., Z. Wang, J. Z. Kolter, et al.  Universal and transferable adversarial attacks on aligned language models, 2023.
  * [7] Zhu, S., R. Zhang, B. An, et al.  AutoDAN: Automatic and interpretable adversarial attacks on large language models.  In _Socially Responsible Language Modelling Research_. 2023.
  * [8] Wang, B., W. Chen, H. Pei, et al.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.  _arXiv preprint arXiv:2306.11698_ , 2023.
  * [9] Chen, H., Y. Ji.  Learning variational word masks to improve the interpretability of neural text classifiers.  _arXiv preprint arXiv:2010.00667_ , 2020.
  * [10] Modarressi, A., M. Fayyaz, E. Aghazadeh, et al.  Decompx: Explaining transformers decisions by propagating token decomposition.  _arXiv preprint arXiv:2306.02873_ , 2023.
  * [11] Zhao, Z., B. Shan.  Reagent: Towards a model-agnostic feature attribution method for generative language models.  In _Proceedings of AAAI Workshop on Responsible Language Models_. 2024.
  * [12] Vafa, K., Y. Deng, D. M. Blei, et al.  Rationales for sequential predictions.  In _Empirical Methods in Natural Language Processing_. 2021.
  * [13] Wei, J., X. Wang, D. Schuurmans, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * [14] Turpin, M., J. Michael, E. Perez, et al.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  _ArXiv_ , abs/2305.04388, 2023.
  * [15] Xu, Z., S. Jain, M. Kankanhalli.  Hallucination is inevitable: An innate limitation of large language models, 2024.
  * [16] Miglani, V., A. Yang, A. H. Markosyan, et al.  Using captum to explain generative language models.  _arXiv preprint arXiv:2312.05491_ , 2023.
  * [17] Shi, W., J. Michael, S. Gururangan, et al.  Nearest neighbor zero-shot inference.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3254–3265. 2022.
  * [18] Han, X., W. Zhao, N. Ding, et al.  Ptr: Prompt tuning with rules for text classification, 2021.
  * [19] Shi, Y., H. Ma, W. Zhong, et al.  Chatgraph: Interpretable text classification by converting chatgpt knowledge to graphs.  In _2023 IEEE International Conference on Data Mining Workshops (ICDMW)_ , pages 515–520. IEEE, 2023.
  * [20] Kommiya Mothilal, R., D. Mahajan, C. Tan, et al.  Towards unifying feature attribution and counterfactual explanations: Different means to the same end.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’21, page 652–663. Association for Computing Machinery, New York, NY, USA, 2021.
  * [21] Wu, Z., Y. Chen, B. Kao, et al.  Perturbed masking: Parameter-free probing for analyzing and interpreting BERT.  In D. Jurafsky, J. Chai, N. Schluter, J. Tetreault, eds., _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4166–4176. Association for Computational Linguistics, Online, 2020.
  * [22] Sundararajan, M., A. Taly, Q. Yan.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR, 2017.
  * [23] Li, J., X. Chen, E. Hovy, et al.  Visualizing and understanding neural models in NLP.  In K. Knight, A. Nenkova, O. Rambow, eds., _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 681–691. Association for Computational Linguistics, San Diego, California, 2016.
  * [24] Ribeiro, M. T., S. Singh, C. Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144. 2016.
  * [25] Brown, T., B. Mann, N. Ryder, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * [26] Oppenlaender, J.  The creativity of text-to-image generation.  In _Proceedings of the 25th International Academic Mindtrek Conference_ , pages 192–202. 2022.
  * [27] Khattak, M. U., H. Rasheed, M. Maaz, et al.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19113–19122. 2023.
  * [28] Jacovi, A., Y. Goldberg.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?  In D. Jurafsky, J. Chai, N. Schluter, J. Tetreault, eds., _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205. Association for Computational Linguistics, Online, 2020.
  * [29] Chuang, Y.-N., G. Wang, C.-Y. Chang, et al.  Large language models as faithful explainers, 2024.
  * [30] Ebrahimi, J., A. Rao, D. Lowd, et al.  Hotflip: White-box adversarial examples for text classification, 2018.
  * [31] Shin, T., Y. Razeghi, R. L. Logan IV, et al.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In B. Webber, T. Cohn, Y. He, Y. Liu, eds., _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235. Association for Computational Linguistics, Online, 2020.
  * [32] Zheng, L., W.-L. Chiang, Y. Sheng, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _ArXiv_ , abs/2306.05685, 2023.
  * [33] Wolf, T., L. Debut, V. Sanh, et al.  Transformers: State-of-the-art natural language processing.  In Q. Liu, D. Schlangen, eds., _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45. Association for Computational Linguistics, Online, 2020.
  * [34] Taori, R., I. Gulrajani, T. Zhang, et al.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [35] Belvèze, J.  TL;DR News: A Large Dataset for Abstractive Summarization.  https://huggingface.co/datasets/JulesBelveze/tldr_news, 2022.  Accessed: May 22, 2024.
  * [36] Amod.  mental_health_counseling_conversations (revision 9015341), 2024.
  * [37] Samek, W., A. Binder, G. Montavon, et al.  Evaluating the visualization of what a deep neural network has learned.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(11):2660–2673, 2017.
  * [38] Hooker, S., D. Erhan, P.-J. Kindermans, et al.  A benchmark for interpretability methods in deep neural networks, 2019.
  * [39] Madsen, A., N. Meade, V. Adlakha, et al.  Evaluating the faithfulness of importance measures in NLP by recursively masking allegedly important tokens and retraining.  In Y. Goldberg, Z. Kozareva, Y. Zhang, eds., _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1731–1751. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022.
  * [40] Papineni, K., S. Roukos, T. Ward, et al.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318. Association for Computational Linguistics, USA, 2002.
  * [41] Lin, C.-Y.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81. Association for Computational Linguistics, Barcelona, Spain, 2004.
  * [42] Reimers, N., I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  * [43] Zhang, J.  Cognitive functions of the brain: Perception, attention and memory, 2019.

##  Appendix A Appendix

###  A.1 Dataset Details

We evaluate explanations on three datasets: Alpaca, tldr_news and MHC. Due to
the extensive computational cost, we randomly select approaximately 110 data
samples with at least 15 words from each datasets. The statistics of the data
used in our experiment can be found in Table 3.

  * •

Alpaca [34] is a dataset with 52000 unique examples consisting of instructions
and demonstrations generated by OpenAI’s text-davinci-003 engine. Each example
in the dataset includes an instruction that describes the task the model
should perform, accompanied by optional input context for that task. In our
experiment, we randomly select a subset of these examples for verification
purposes.

  * •

tldr_news [35] dataset is constructed by collecting a daily tech newsletter.
For every piece of data, there is a headline and corresponding content
extracted. The task is to ask the model to simplify the extracted content and
then generate a headline from the input.

  * •

mental_health_counseling (MHC) [36] includes broad pairs of questions and
answers derived from online counseling and therapy platforms. It covers a wide
range of mental-health related questions and concerns, as well as the advice
provided by the psychologists. It is utilized for used to fine-tuning the
model to generate the metal health advice. Here, we prompt the model to
generate an advice based on the provided question.

Dataset | Length | Number | Prompt Examples
---|---|---|---
Alpaca | 16-137 | 108 |  Pretend you are a project manager of a construction company. Describe a time when you had to make a difficult decision.
tldr_news | 14-138 | 120 |  Reddit aired a five-second long ad during the Super Bowl. The ad consisted of a long text message that hinted at the GameStop stocks saga. A screenshot of the ad is available in the article.
MHC | 18-478 | 100 |  I cannot help myself from thinking about smoking. What can I do to get rid of this addiction?

Table 3: Data Statistics

###  A.2 Case Study

In this section, we visualize three examples which are selected from the
dataset Alpaca, tldr_news, and MHC respectively, shown in Figure. 5. The
tokens in the red frame are the $k$ explanatory tokens extracted by the model
LLaMA-2(7B-Chat), and the output responses framed in blue are the most
informative results selected by human. In the first data sample, the
recognized token: "brains" and "cognitive", has some semantic relationship
intuitively. It is a common sense the cognitive ability is underpinned by the
brain’s function and structure. There are numerous reports and studies focused
on the functionally relationship between the brain and coginitive ability
[43]. Due to the extensive amount of data on which the LLMs are trained, the
model can integrate their relationship which is reflected in its response.

As for the second example, the visualized tokens are almost related to the
"book" and "publish", which aligns with the theme of the output content.
Removing either "publishing" or "publish" would not greatly alter the meaning
of the input prompt. This example also shows that our method could detect some
fix expressions like "whether…or..", whose relation is recognized by the token
"vs." in the output.

In the last example, we could figure out the model infers or "guesses" that
the kid feels frustrated and helpless, a sentiment partly related to the token
"never". The generated output does not mention the reason to explain how the
model makes such inference. From this point, our method could generate human-
intelligible explanations to assist the users in identifying which parts of
the prompts fundamentally lead to the undesired response, providing guidance
on how to revise the prompts effectively.

Figure 5: Case study for visualizing the top-$k$ tokens in each example.
Figure 6: Case study for visualizing the variation of the model responses
after the specific tokens are masked.

In Figure 6, we also illustrate the changes in the model response as the
targeted top-$k$ token is removed from the input prompt. Compared with the
top-$k$ tokens identified by our method, those found by Captum are mostly not
important to the response. This demonstrates the Captum’s limited ability to
explain the relationship between input prompt and output response with the
consideration of the textual information. Moreover, when masking the token
"publishing" or "publish", as identified by our method XPrompt, the generated
outputs do not exhibit significant variations in either word frequency or
semantics. However, substantial changes in the generated outputs occur only
when both tokens are simultaneously masked. This highlights the effectiveness
of our methodology in explaining the joint effect of the prompt attributions
on the generated output.

###  A.3 Gradient Design

The Table. 4 displays the log-likelihood without using the gradient as the
guidance to initialize the optimization process or to do sampling at different
stage when searching for the optimal results.

Methods | Number of Iterations
---|---
1 | 5 | 10 | 15 | 20 | 30 | 40 | 50
XPrompt | -42.448 | -51.661 | -56.378 | -59.852 | -62.303 | -64.812 | -67.274 | -69.426
OO w/o Initialization | -33.906 | -44.555 | -50.828 | -55.652 | -58.483 | -62.694 | -64.793 | -66.415
OO w/o Probability | -41.759 | -49.867 | -53.784 | -57.107 | -59.639 | -63.339 | -65.848 | -67.496

Table 4: Ablation study showing the log-likelihood without using gradient for
initialization or sampling with different number of iterations.

###  A.4 Limitations

One possible limitation of our work is that while the explanations demonstrate
the counterfactual correlation between the input prompt and the output
sequence, the underlying generation mechanism remains a black box and could
not be explained by our method, which we believe is an important and
challenging problem. In the future work, we aim to extend the proposed
framework to explore more aspects of the generation mechanism in LLMs.