# Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era

Xuansheng Wu ${}^{1^{*}}$<EMAIL_ADDRESS>
Haiyan Zhao ${}^{2^{*}}$<EMAIL_ADDRESS>
Yaochen Zhu ${}^{3^{*}}$<EMAIL_ADDRESS>
Yucheng Shi ${}^{1^{*}}$<EMAIL_ADDRESS>
Fan Yang 4<EMAIL_ADDRESS>
Tianming Liu 1<EMAIL_ADDRESS>
Xiaoming Zhai 1<EMAIL_ADDRESS>
Wenlin Yao 5<EMAIL_ADDRESS>
Jundong Li 3<EMAIL_ADDRESS>
Mengnan Du 2<EMAIL_ADDRESS>
Ninghao Liu 1<EMAIL_ADDRESS>
1University of Georgia 2New Jersey Institute of Technology 3University of
Virginia 4Wake Forest University 5Tencent AI Lab (Seattle)

###### Abstract

Explainable AI (XAI) refers to techniques that provide human-understandable
insights into the workings of AI models. Recently, the focus of XAI is being
extended towards Large Language Models (LLMs) which are often criticized for
their lack of transparency. This extension calls for a significant
transformation in XAI methodologies because of two reasons. First, many
existing XAI methods cannot be directly applied to LLMs due to their
complexity and advanced capabilities. Second, as LLMs are increasingly
deployed across diverse industry applications, the role of XAI shifts from
merely opening the “black box” to actively enhancing the productivity and
applicability of LLMs in real-world settings. Meanwhile, unlike traditional
machine learning models that are passive recipients of XAI insights, the
distinct abilities of LLMs can reciprocally enhance XAI. Therefore, in this
paper, we introduce Usable XAI in the context of LLMs by analyzing (1) how XAI
can benefit LLMs and AI systems, and (2) how LLMs can contribute to the
advancement of XAI. We introduce 10 strategies, introducing the key techniques
for each and discussing their associated challenges. We also provide case
studies to demonstrate how to obtain and leverage explanations. The code used
in this paper can be found at: https://github.com/JacksonWuxs/UsableXAI_LLM.

**footnotetext: Equal contribution

###### Contents

  1. 1 Introduction
  2. 2 LLM Diagnosis via Attribution Methods
    1. 2.1 Literature Review of Attribution Methods
    2. 2.2 Case Studies: Usability of Attribution Methods for LLMs
    3. 2.3 Challenges
  3. 3 LLM Diagnosis and Enhancement via Interpreting Model Components
    1. 3.1 Understanding the Self-Attention Module
    2. 3.2 Understanding the Feed-Forward Module
    3. 3.3 Challenges
  4. 4 LLM Debugging with Sample-based Explanation
    1. 4.1 Literature Review of Sample-based Explanation
    2. 4.2 Case Study: EK-FAC-based Influence Estimation
    3. 4.3 Challenges
  5. 5 Explainability for Trustworthy LLMs and Human Alignment
    1. 5.1 Security
    2. 5.2 Privacy
    3. 5.3 Fairness
    4. 5.4 Toxicity
    5. 5.5 Truthfulness
    6. 5.6 Challenges
  6. 6 LLM Enhancement via Explainable Prompting
    1. 6.1 Chain of Thoughts (CoT) Prompting
    2. 6.2 Extended Methods of Explainable Prompting
    3. 6.3 Case Study: Is CoT Really Making LLM Inferences Explainable?
    4. 6.4 Challenges
  7. 7 LLM Enhancement via Knowledge-Augmented Prompting
    1. 7.1 Preliminaries: Retrieval-Augmented Generation
    2. 7.2 Enhancing Decision-Making Control with Explicit Knowledge
    3. 7.3 Challenges
  8. 8 Training Data Augmentation with Explanation
    1. 8.1 Explanation-guided Data Augmentation for Mitigating Shortcuts
    2. 8.2 Explanation-enhanced Data Enrichment
    3. 8.3 Challenges
  9. 9 Generating User-Friendly Explanation for XAI
    1. 9.1 User-friendly Data Explanation with LLMs
    2. 9.2 Explaining Small Models with LLMs
    3. 9.3 Self-Explanation of LLMs
    4. 9.4 Challenges
  10. 10 LLMs for Interpretable AI System Design
    1. 10.1 Designing Interpretable Network Architectures with LLMs
    2. 10.2 Designing Interpretable AI Workflows with LLM Agents
    3. 10.3 Challenges
  11. 11 Emulating Humans with LLMs for XAI
    1. 11.1 Emulating Human Annotators for Training Explainable Models
    2. 11.2 Emulating Human Feedback for Evaluating Explainable Models
    3. 11.3 Challenges
  12. 12 Discussion and Conclusion

##  1 Introduction

Explainability holds great promise in understanding machine learning models
and providing directions for improvement. In practice, users have high
expectations for model explainability:

1\. Through explanation, can we know if a model works properly?

2\. Does explainability help developing better models?

First, explanations are expected to illuminate whether a model operates in
accordance with human expectations. For example, does the model leverage
reliable evidence and domain knowledge in its decision-making? Does the model
contain bias and discrimination? Does the model show any vulnerabilities to
potential attacks? Will the model output harmful information? Second, in
recognition of model imperfections, we aspire for explainability to inform the
development of better models. For example, how to adjust the behaviors of a
model if we find it is using unreliable or unreasonable features in making
predictions? Can we improve the performance of a model by aligning its
behavior with human preferences?

Therefore, the question arises: Have these expectations been met? In recent
years, the body of literature on Explainable AI (XAI) has expanded rapidly to
improve model transparency (Du et al., 2019a; Murdoch et al., 2019; Tjoa &
Guan, 2020; Došilović et al., 2018; Rudin et al., 2022), encompassing a wide
array of methods customized for different data modalities, including visual
(Zhang & Zhu, 2018), textual (Danilevsky et al., 2020), graph (Yuan et al.,
2022), and time-series data (Zhao et al., 2023c). Some literature delves into
specific techniques, such as attention methods, generalized additive models,
and causal models. Additionally, some offer reviews on general principles and
categorizations or initiate discussions on evaluating the faithfulness of
explanations (Yang et al., 2019). Despite the progress, the last mile of XAI –
making use of explanations – has not received enough attention. In many cases,
we seem to be satisfied with just acquiring explanations and their associated
visualizations, sometimes followed by qualitative analysis of the model’s
strengths and weaknesses. While these explanations can reveal a model’s
imperfections, quantifying model properties (e.g., fairness, security,
rationality) or taking the next concrete steps toward model improvement
remains a difficult task.

The challenges in achieving usable explainability are twofold. First, there is
an inherent conflict between AI automation and human engagement in XAI. On one
hand, humans need to define explainability that the model should follow, or
scrutinize explanation to identify if any vulnerabilities exist in the model.
On the other hand, the requirement for human oversight introduces substantial
costs, posing challenges to the scalability and practical implementation of
model debugging and improvement in AI workflows. Second, many of the current
approaches view explainability as a purely technical matter, ignoring the
needs of practitioners and non-technical stakeholders. Existing XAI methods
are mainly developed as statistical and mathematical tools. However, there
exists a noticeable disparity between the objectives of these tools and the
expectations of practitioners across various application domains (Malizia &
Paternò, 2023). An explanation that satisfies a technical audience might offer
little value to a non-technical audience.

While the opacity issues have not yet been fully resolved for traditional deep
models (e.g., multi-layer perceptrons, convolutional and recurrent neural
network), the recent advancements of Large Language Models (LLMs) (Brown et
al., 2020; Achiam et al., 2023; Touvron et al., 2023b; Chiang et al., 2023)
appear to have exacerbated the challenge we are facing. Firstly, LLMs
typically possess a significantly larger model size and a greater number of
parameters. This increased model complexity intensifies the difficulty of
explaining their inner workings. Second, different from traditional ML models
that primarily focus on low-level pattern recognition tasks such as
classification and parsing, LLMs can handle more complex tasks such as
generation, reasoning and question answering. Understanding the exclusive
abilities of LLMs presents novel challenges for XAI techniques. Considering
the transformative impact of LLMs across various applications, ensuring the
explainability and ethical use of LLMs has become an imminent and pressing
need. Meanwhile, the emergent capabilities of LLMs also present new
opportunities for XAI research. Their human-like communication and commonsense
reasoning skills offer prospects for achieving explainability in ways that
could potentially augment or replace human involvement.

Figure 1: The contributions and outline of this paper. We define Usable XAI in
the context of LLMs with seven strategies of enhancing LLMs with XAI, and
three strategies of enhancing XAI with LLMs.

Defining “Usable XAI”. In light of the above considerations, in the context of
LLMs, we define Usable XAI which includes two aspects as follows. (1)
Utilizing Explainability to Enhance LLM and AI Systems. Beyond just producing
explanations or enhancing the transparency of LLMs, we explore whether these
explanations can pinpoint issues for model debugging or improve the overall
performance of LLMs or AI models at large, such as accuracy, controllability,
fairness, and truthfulness. (2) Utilizing LLMs to Enhance XAI Frameworks. The
human-like communication ability of LLMs can enhance model explanations in
terms of user-friendliness, by converting the numerical values into
understandable language. Also, the commonsense knowledge stored in LLMs can
significantly boost the practicality of existing XAI frameworks, by playing
the role of humans and alleviating the need for real human involvement in AI
workflows.

Contribution of this paper. In this paper, we investigate 10 strategies
towards usable XAI techniques in the context of LLMs. These strategies are
organized into two major categories: (1) Usable XAI for LLMs; (2) LLM for
Usable XAI, as shown in Figure 1. Additionally, we conduct case studies to
substantiate the discussion on selected techniques. For each strategy, we also
explore the open challenges and areas that require further investigation in
future work.

  * •

Usable XAI for LLMs. We introduce how interpretation can be utilized to
enhance AI pipelines, including LLMs and small models. First, we investigate
how explanations could be utilized to diagnose and enhance LLMs in terms of
utility. We study three types of post-hoc explanation methods, targeting LLM
predictions (Section 2), LLM components (Section 3), and training samples
(Section 4), respectively. Second, we focus on how explanations could be
leveraged to scrutinize and boost model trustworthiness (Section 5), including
security, fairness, toxicity, and truthfulness, which is crucial to achieving
human alignment. Third, we discuss how explainability could guide the
augmentation of data, including both inference data (i.e., prompts) and
training data. Specifically, we discuss two strategies of crafting explainable
prompts for LLMs: Chain-of-Thought prompts (Section 6) and knowledge-enhanced
prompts (Section 7). Furthermore, we introduce leveraging LLM explanations to
augment training data for improving small models (Section 8).

  * •

LLM for Usable XAI. In this part, we investigate strategies for leveraging the
advanced capabilities of LLMs to address the challenges in traditional XAI
domains, thus enhancing the usability of XAI in practice. First, we examine
ways to enhance the user-friendliness of explanations through the generative
capabilities of LLMs (Section 9). Second, we introduce how to automate the
design of interpretable AI workflows by leveraging the planning abilities of
LLMs (Section 10). Third, we introduce how to facilitate the evaluation of XAI
methods by utilizing the unique property of LLMs in emulating human cognition
processes (Section 11).

Differences between this paper and existing surveys. Many surveys have been
conducted to examine Explainable AI (Du et al., 2019a; Tjoa & Guan, 2020;
Došilović et al., 2018) or Interpretable Machine Learning (Murdoch et al.,
2019). This paper differs from existing work as we focus on explanation
methods for large language models. Meanwhile, different from the existing
survey (Zhao et al., 2023b) that mainly reviews explanation methods for LLMs,
our paper puts an emphasis on the XAI usability in LLM studies. To the best of
our knowledge, the most related paper to our survey is (Luo & Specia, 2024),
which also discusses several aspects where explanations can improve LLM
performance. Nevertheless, this light-weight investigation lacks a thorough
examination of XAI methods (e.g., sample-based explanation, interpretable
workflows, explainable prompts) and how LLMs can benefit existing XAI
frameworks (e.g., data augmentation, improving user-friendliness, XAI
evaluation). Finally, our paper contributes further by providing detailed case
studies and open-sourced codes, fostering future research in applying
explanations effectively within the LLM context.

##  2 LLM Diagnosis via Attribution Methods

This section introduces attribution methods as post-hoc explanations for LLMs,
and how we can discover model defects with attribution scores. We start with
revisiting existing attribution methods, and then discuss which methods are
still suitable for explaining LLMs. Since LLMs widely serve both
classification and generation tasks, our discussion categorizes the
attribution methods accordingly. After that, we explore case studies of
applying attribution methods to assess LLM-generated output quality. Finally,
we discuss future work of designing novel post-hoc explanation methods for
LLMs.

###  2.1 Literature Review of Attribution Methods

The attribution-based explanation quantifies the importance of each input
feature that contributes to making predictions. Given a language model $f$
with a prediction $\hat{y}=f(x)$ according to the $N$-words input prompt $x$,
the explainer $g$ assesses the influence of input words in $x$ as
$\mathbf{a}=g(x,\hat{y},f)\in\mathbb{R}^{N}$. Typically, the sign of
$a_{n}\in\mathbf{a}$ indicates word $x_{n}$ positively or negatively
influences $\hat{y}$, and a greater value of $|a_{n}|$ indicates a stronger
impact. In text classification, $\hat{y}$ denotes a specific class label. In
text generation, $\hat{y}$ represents a varying length of generated text.

Many existing attribution-based explanation methods focus on classification
tasks and cannot be directly applied to the generation task. The primary
distinction between them is that: classification is limited to a specific set
of predictions, while generation encompasses an endless array of
possibilities. For instance, in sentiment analysis, a language model can be
instructed to output a number between 0 and 1 that indicates the positivity of
input text by adding a linear layer and a sigmoid function on top of the
language model. However, in the generative setting, the model can express this
positivity in numerous expressions, such as “the reviewer definitely loves
this movie” and “it is a strong positive movie review”. This distinction poses
a unique challenge in adapting explanation methods from classification to
generation tasks. In the following, we review related works based on the
scenarios they are applicable to.

####  2.1.1 Attributing Inputs for Label Classification

Common attribution methods (Du et al., 2019a; Murdoch et al., 2019) developed
for traditional deep models include gradient-based methods, perturbation-based
methods, surrogate methods, and decomposition methods. We introduce the
general idea and representative examples for each category, followed by the
analysis of their suitability for explaining large language models.

Perturbation-based Explanation. Perturbation-based methods assess the
importance of input features by perturbing them and monitoring changes in
prediction confidence, i.e.,
$a_{n}=p(\hat{y}|x)-p(\hat{y}|\widetilde{x}_{n})$, where $\widetilde{x}_{n}$
refers to the input sequence with the $n$-th feature being perturbed. Each
feature could refer to a word (Li et al., 2016a), a phrase (Wu et al., 2020b),
or a word embedding (Li et al., 2016b). The underlying principle is that
perturbing a more important feature should result in a more pronounced
alteration in the model’s prediction confidence. However, this method has
limitations, particularly in its assumption that features are independent,
which is not always the case with textual data due to word inter-dependencies.
Additionally, it is computationally intensive for explaining LLMs, requiring
$N$ inferences for an input of $N$ words.

Gradient-based Explanation. Gradient-based methods offer a computationally
efficient approach for estimating model sensitivity to input features based on
gradients $\frac{\partial p(\hat{y}|x)}{\partial\mathbf{x}_{n}}$, where
$\mathbf{x}_{n}$ refers to the embedding of word $x_{n}$. Some methods employ
the $L_{2}$-norm of gradients to assess word importance (Li et al., 2016a),
i.e., $a_{n}=\|\frac{\partial p(\hat{y}|x)}{\partial\mathbf{x}_{n}}\|_{2}$.
This approach only requires a single inference and one backpropagation pass.
Some extended methods multiply the gradient with the word embedding
(Kindermans et al., ; Ebrahimi et al., 2018; Mohebbi et al., 2021), i.e.,
$a_{n}=\frac{\partial
p(\hat{y}|x)}{\partial\mathbf{x}_{n}}\cdot\mathbf{x}_{n}$. These methods may
yield explanations with limited faithfulness for deep models (Shrikumar et
al., 2017), as gradients only reflect the local relationship between input
variation and output variation. To address this, Integrated Gradients (IG) has
been proposed (Sundararajan et al., 2017; Sikdar et al., 2021; Sanyal & Ren,
2021; Enguehard, 2023), which accumulates gradients as input transitions from
a reference point to the actual data point. Nevertheless, IG entails multiple
rounds of inference and backpropagation, thus significantly increasing
computational demands.

Surrogate-based Explanation. Surrogate-based explanation methods understand
complex models by constructing a simpler model $g$ trained on
$\mathcal{D}(x,\hat{y})=\\{(\widetilde{x}_{k},\widetilde{y}_{k})\\}_{k=1}^{K}$,
where $\mathcal{D}(x,\hat{y})$ denotes a dataset constructed for the target
instance $(x,\hat{y})$; $\widetilde{x}_{k}$ is usually obtained by perturbing
$x$, and $\widetilde{y}_{k}=f(\widetilde{x}_{k})$. The surrogate model $g$,
ranging from basic linear models to sophisticated decision trees, serves as a
proxy to approximate the decision boundary of the target model $f$ for a
specific instance $(x,\hat{y})$. Notable examples include LIME (Ribeiro et
al., 2016), SHAP (Lundberg & Lee, 2017), and TransSHAP (Kokalj et al., 2021),
where the first two are designed for general deep neural networks and the last
one is tailored for Transformer-based language models. Nevertheless, a
significant limitation of them is their intensive reliance on repeated
interactions with the target model, a process that is impractical for LLMs.

Decomposition-based Explanation. Decomposition-based methods assign linearly
additive relevance scores to inputs, effectively breaking down the model’s
prediction. Layer-wise Relevance Propagation (Montavon et al., 2019) and
Taylor-type Decomposition (Montavon et al., 2017) are well-known techniques
for computing these relevance scores. These methods have been adapted for
Transformer-based language models in various research (Voita et al., 2019;
2020; Wu & Ong, 2021). However, a primary challenge in implementing
decomposition-based explanations is the need for tailored decomposition
strategies to accommodate different model architectures. Although many large
language models are based on the Transformer framework, there are key
variations between them, such as LLaMA (Touvron et al., 2023a) and GPT
(OpenAI, 2023), particularly in aspects like positional encoding strategy and
feed-forward network design. This challenge poses a limitation on the
universal applicability of decomposition methods for general-purpose
interpretation.

Table 1: Time complexity analysis on different attribution methods for the generative task. Method | Forward | Backward | Notes
---|---|---|---
Mask Perturbation | $\mathcal{O}(N)$ | 0 | -
Gradient$\times$Input | $\mathcal{O}(1)$ | $\mathcal{O}(M)$ | -
Integrated Gradients | $\mathcal{O}(N_{step})$ | $\mathcal{O}(N_{step}\cdot M)$ |  $N_{step}$ is the number of steps for integrating gradients.
LIME | $\mathcal{O}(N_{aug})$ | 0 |  $N_{aug}$ is the number of augmented samples.
SHAP | $\mathcal{O}(2^{N})$ | 0 | -

To summarize, the traditional explanation methods are not always suitable for
LLMs. In particular, the perturbation-based and gradient-based explanations
are relatively easy to extend for attributing LLM responses to the input
prompts, while the surrogate-based and decomposition-based methods become
significantly challenging to do so. Specifically, surrogate-based methods
suppose that an explainable small model could approximate the decision
boundary of the target model around a local example, but there are limited
explainable models for the text generation task. Meanwhile, decomposition-
based methods require designing decomposition strategies for different layers,
which is challenging for big LLM architectures. Another primary concern is
their significant demand for computing resources. Given an $N$-words input
prompt and an $M$-words output response, we present the time complexity of
several representative explanation methods in Table 1. It demonstrates that
existing methods either require a large number of forward operations or
backward operations. Therefore, improving the efficiency of the attribution-
based explanation is an important direction for future research and
development.

####  2.1.2 Attributing Inputs for Text Generation

Figure 2: An example of attribution saliency map between input and output
tokens.

The explanation of generative models can be defined as attributing the overall
confidence $p(\hat{y}|x)$ to the input $x$, where $\hat{y}$ denotes the
generated response $\hat{y}=[\hat{y}_{1},...,\hat{y}_{M}]$ with $M$ words. One
method to achieve this is by treating the text generation process as a
sequence of word-level classification tasks. This perspective allows for the
application of existing classification-based explanation techniques to assess
the influence of each input word $x_{n}$ in relation to each output word
$\hat{y}_{m}$, resulting in a corresponding attribution score $a_{n,m}$. After
gathering the attributions $a_{n,m}$ for $m=1,...,M$, we perform an
aggregation to determine the overall contribution of each input word $x_{n}$.
This is accomplished by aggregating the individual attributions for all output
words corresponding to the input word, denoted as
$a_{n}=\mathrm{Aggregate}([a_{n,1},...,a_{n,M}])$. The simplest approach for
this aggregation is to average the attributions assigned to each input word
across the different output words (Selvaraju et al., 2016). However, Wu et al.
(2023) observe that attribution scores from different output words are not
inherently comparable. For example, the attribution scores for function words
(e.g., “the”, “is”, “have”) are often disproportionately larger than the
scores for content words with clear semantic meaning (e.g., verbs and nouns).
Therefore, it is necessary to normalize the scores prior to the aggregation,
so that the scores $[a_{n,1},...,a_{n,M}]$ become comparable for $1\leq m\leq
M$. Figure 2 plots the normalized scores of an example case, where each index
in the Y-axis refers to an input prompt token, while that in the X-axis is an
output response token. A greater normalized attribution score is brighter. In
this example, the user attempts to direct the model to output information that
does not exist, namely the French president in 1250. The model successfully
realizes that this thing does not exist and refuses to answer. The model
response can be realized as three parts, “There was no”, “president in
France”, and “in 1250”. According to the figure, the first span is generated
heavily because of the tokens “Who” and “president”, while the model uses both
“France” and “1250” to respond to the second span “president in France”.
Finally, the model emphasizes the date “1250” again by referencing the same
information from the prompt. Overall, these explanations align with human
understanding and highlight the usage of this method in the future. However,
current research on attribution-based explaining for generative LLMs is still
in its early stages, and only a limited number of methods have been proposed.

###  2.2 Case Studies: Usability of Attribution Methods for LLMs

Figure 3: A general pipeline of model diagnosis with attribution explanations.

The attribution map offers a partial insight into the operational mechanics of
LLMs (Chandrasekaran et al., 2018; Hase & Bansal, 2020; Ye & Durrett, 2022a).
Accordingly, we propose a general pipeline that leverages attribution scores
to analyze LLM behaviors, as shown in Figure 3. First, given the target LLM
and an input prompt, we compute attribution scores of input tokens relative to
the output tokens. Second, we extract a feature vector from the attribution
map, tailored to the requirements of the diagnostic task at hand. Third, we
train a light-weight predictor (e.g., a classifier) to diagnose whether the
model behaves appropriately based on the feature vector. In the following, we
provide case studies to illustrate how attribution scores could be utilized to
assess LLM response quality (Adlakha et al., 2023).

####  2.2.1 LLM Response Quality Evaluation with Explanations

This case study explores the use of attribution-based explanations as evidence
for assessing the quality of LLM-generated responses. Here, “quality” is
specifically measured by the responses’ accuracy. We hypothesize that
responses generated from correct rationales are likely to be more accurate.
Our method involves comparing the model’s explanations with underlying
rationales as a means to evaluate response accuracy. This approach could
enhance the reliability of conversational systems, which is crucial for
applications such as medical question answering.

##### Dataset.

We employ the Multi-Sentence Reading Comprehension (MultiRC) dataset (Khashabi
et al., 2018) in this case study. MultiRC presents a more significant
challenge than other datasets: it requires the system to answer questions
based on multiple sentences from a given paragraph. The answers may not
directly correspond to specific sentence spans, reflecting a more realistic
scenario of modern LLMs. Each MultiRC instance is associated with a human-
annotated rationale indicating which sentences are relevant for answering the
question. Our study specifically employs its test subset with 950 samples in
total. We consider 80% in which are the training set and we report the results
evaluated on the rest samples.

##### Implementation.

We apply the Importance Density Score method (Wu et al., 2023) to estimate the
importance of each word in the provided paragraph. Specifically, given
$N$-word paragraph $X$ and the $M$-word response $\hat{Y}$, the attribution
from each input word $x_{n}$ to each output word $\hat{y}_{m}$ is defined as
$a_{n,m}=\frac{\partial
f(y_{m}|Z_{m})}{\partial\,\mathbf{E}_{i}[x_{n}]}\cdot\mathbf{E}_{i}[x_{n}]^{\top}$,
where $Z_{m}$ indicates the formatted input prompt concatenated with the first
$m-1$ response word, $f$ is the language model, and $\mathbf{E}_{i}[x_{n}]$
indicates the input static embedding of $x_{n}$. This pairwise attribution
score is then normalized as $\widetilde{a}_{n,m}=\lceil L\times
a_{n,m}/\text{max}_{n^{\prime}}(a_{n^{\prime},m})\rceil$. Any normalized
attribution $\widetilde{a}_{n,m}$ is equal or less than $b$ will is forced to
0. The overall attribution score on word $x_{n}$ is defined as its attribution
density, i.e.,
$a_{n}=||[a_{n,1},...,a_{n,M}||_{1}/||[a_{n,1},...,a_{n,M}]||_{p}$. In our
experiments, we let $L=10$, $b=2$, $p=5$, and consider Vicuna-7B-v1.1 as our
language model $f$. By averaging these word attribution scores, we reach the
importance of each sentence. The top-K sentences with the highest importance
scores are selected as the explanation for each instance. The explanation is
then concatenated with the output response and fed to a classifier to predict
whether the output is correct or not. We initialize the classifier with
DistillBERT-base (Sanh et al., 2019), and train it over 3 epochs with learning
rate $5e^{-5}$ and weight decay $1e^{-3}$. To compare with this, we also
consider the Full Paragraph or the human-annotated rationales as the
explanation part to train the classifier. Following previous studies (Khashabi
et al., 2018; DeYoung et al., 2019), we evaluate the accuracy of the generated
responses by exactly matching all correct answers with the generated
responses. Table 2 reports the macro precision, recall, F1, and AUC scores on
the test set over five random seeds.

##### Results.

Table 2: Utilizing attribution for response quality evaluation Setting | Precision | Recall | F1 | AUC
---|---|---|---|---
Random | 49.40 | 51.79 | 49.61 | 49.03
Human Rationale | 68.73 | 66.88 | 67.57 | 73.11
Full Paragraph | 58.02 | 58.47 | 56.89 | 63.44
Attribution (ours) | 63.25 | 67.69 | 64.12 | 71.53

In Table 2, we observe that isolating the rationales from the full context
could best help the classifier identify the response quality. It is evident
that explanations align more closely with human-annotated rationales when
associated with correct responses. In particular, the precision and recall
metrics for explanations derived from correct answers surpass those associated
with incorrect responses. This finding serves as empirical evidence
underscoring the effectiveness of attribution-based explanations in estimating
the accuracy of responses generated by LLMs.

####  2.2.2 Hallucination Detection with Attribution Explanations

This case study explores a different aspect of LLM generation quality,
focusing on the presence of hallucinations in LLM-generated responses. We show
that attribution-based explanations can serve as indicators to detect LLM
hallucinations. Hallucinations are defined as responses that contain
information conflicting with or unverifiable by factual knowledge (Li et al.,
2023c; Ji et al., 2023). For instance, if a model is asked about a fictitious
entity like “King Renoit” and responds with a narrative about “The Three
Musketeers”, claiming it pertains to the nonexistent king, it illustrates a
hallucination. This tendency, particularly pronounced in instruction-tuned
models, stems from their earnest endeavor to fulfill user requests. The issue
often arises when direct commands (“tell me a story”) significantly influence
the generation process, while the instruction’s subject (“about King Renoit”)
is neglected. Based on this insight, we develop a hallucination detector
according to the distribution of attribution scores over different types of
prompting words.

##### Dataset.

We use the Hallucination Evaluation Benchmark (Li et al., 2024a) in this case
study. Each instance from this dataset includes an input prompt, a response
generated by ChatGPT (Ouyang et al., 2022), and the knowledge related to the
response. Each piece of knowledge has a human annotation about whether the
knowledge is valid or not. Each instance having at least one invalid knowledge
is considered a hallucination response. This study focuses on 632 less
confused examples from the benchmark, each of which has almost all knowledge
correct or almost all wrong. We randomly select 80% of the samples to form a
training set and the rest form the testing set.

##### Implementation.

Given a query prompt and its ChatGPT response, we aim to build a classifier to
detect if the response contains hallucination. Since the gradients of ChatGPT
is inaccessible, we apply the Vicuna-7B model as the substitute to compute the
attribution scores. Specifically, we adopt Importance Density Scores (Wu et
al., 2023) to compute the attribution scores of Vicuna. We then use the NLTK
package to identify the part-of-speech (POS) tag of each query word. Finally,
each query-response pair is represented with an 82-dimensional vector, where
each dimension indicates the average attribution scores over a certain type of
POS tagging. We develop a Support Vector Machine classifier based on the POS
tagging attribution scores on the training set and report the precision,
recall, F1 scores on the hallucination samples, as well as accuracy over all
samples in Table 3. To compare with this, we also leverage a fine-tuned model
and a prompting-based method to serve as the hallucination detector baselines.

##### Results.

Table 3: Utilizing attribution for hallucination detection on ChatGPT responses. Method | Language Model | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---
Random | - | 88.41 | 50.34 | 64.11 | 50.59
FacTool | GPT-4 | 95.30 | 72.93 | 82.62 | 73.04
Vectara | DeBERTa-base | 90.29 | 60.54 | 72.40 | 59.45
AttrScore (ours) | Vicuna-7B | 90.15 | 74.21 | 81.36 | 70.20
Mistral-7B | 88.74 | 75.04 | 81.26 | 69.57

In the table, we first observe that all methods have demonstrated a greater
performance than the Random strategy, indicating that the proposed methods and
the baselines are valid hallucination detectors. In addition, we observe that
the proposed methods achieve a competitive performance with FacTool using
GPT-4, establishing a new benchmark in this domain. It is worth noting that,
instead of using GPT-4, the attribution score for our method is calculated
using a smaller 7-billion-parameter language model. This demonstrates the
practicality and efficiency of our approach in weak-to-strong generalization
as we could diagnose large language models with smaller models. Future work
could consider extracting more effective features and using more powerful
classifiers.

###  2.3 Challenges

####  2.3.1 How to Identify and Explain the Semantics of Output?

The attribution function $a_{n}=p(\hat{y}|x)-p(\hat{y}|\widetilde{x}_{n})$ is
no longer faithfully attributing the model predictions in the human-interested
semantic level since the model could express the same semantic meaning with
various responses. Specifically, the model could assign a lower confidence to
its original response than the new one, while both responses share the same
semantic meaning. This is a significant difference compared with the
traditional classification problem, where target label sets are manually
designed so that a lower $p(\hat{y}|\cdot)$ indicates the model is less
confident in predicting a specific semantic concept. Taking the sentiment
analysis task as an example, an LLM may generate two different responses
sharing the same predicted concepts, such as “it is a positive review” and
“the audience thinks the movie is great”. Current attribution-based
explanations concentrate on the literal changes in generated responses, but
they do not study how the semantic meanings of these responses change.
Therefore, they do not provide sufficient explanations of model-generated
responses at a semantic level. In this case, the semantic level is which words
of the input review lead the model to believe it is positive. Future work may
tackle this challenge by proposing metrics to evaluate the semantic
differences in responses.

Figure 4: Review of interpretation methods for LLM components and their
applications. We categorize methods according to the target LLM modules: self-
attention layers and feed-forward layers.

####  2.3.2 Explaining LLM Predictions Beyond Attribution

The versatility of LLM generation will inspire a variety of explanation
paradigms beyond traditional attribution methods. Attribution methods aim to
explain model output with the contribution of input features. This explanation
task is meaningful for conventional machine learning (ML) models whose outputs
are usually individual decisions with clear formats (e.g., classification,
regression, object detection). The decisions are highly dependent on the input
features. However, LLMs differ from traditional ML models in two aspects.
First, the generative process of LLMs is stochastic, e.g., we may obtain
different outputs by running an LLM twice even with the same input. Second,
LLMs encode rich knowledge within their parameters, which are independent of
inputs. These unique properties give rise to novel explanation paradigms. For
the first aspect, an explanation task of interest would be to understand the
uncertainty of LLM generation. For example, researchers (Ahdritz et al., 2024;
Varshney et al., 2023; Su et al., 2024) leverage the prediction perplexity to
check whether the LLM is confident during generation, identifying potential
errors in less confident predictions. Second, attributing LLM predictions to
their encoded knowledge instead of input patterns could provide a new
perspective. Some researchers (Yin et al., 2024a) propose the knowledge-
boundary detection task to detect whether a model has specific knowledge for a
given question. Nevertheless, it did not attribute the prediction to specific
knowledge, so humans cannot verify the prediction process with their results
yet.

##  3 LLM Diagnosis and Enhancement via Interpreting Model Components

This section discusses the XAI methods that interpret the internal components
of large language models. Additionally, it delves into the insights these
methodologies offer, which can be instrumental in refining and enhancing the
design of language models. LLMs adopt transformers as the basic architecture,
which typically comprises two types of major components: self-attention layers
and feed-forward layers. In the following, we review the research that focuses
on interpreting each of these components respectively.

###  3.1 Understanding the Self-Attention Module

A multi-head self-attention module contains multiple self-attention heads
capturing different kinds of word-word relations, which are modeled with
weights $\mathbf{W}_{q},\mathbf{W}_{k}\in\mathbb{R}^{D_{1}\times D_{2}}$.
Specifically, the relation of words $w_{i}$ and $w_{j}$ is computed as
$\mathbf{A}_{i,j}\propto(\mathbf{x}_{i}\mathbf{W}_{q})\cdot(\mathbf{x}_{j}\mathbf{W}_{k})^{\top}$,
where $\mathbf{x}_{i},\mathbf{x}_{j}\in\mathbb{R}^{1\times D_{1}}$ are
contextual embeddings of the words. The most straightforward interpretation is
analyzing the attention score matrix $\mathbf{A}$ given an input sequence to
study the relations between words (Vig, 2019; Hoover et al., 2020). In
practice, these intuitive explanations would be majorly used to present case
studies via visualization. With this strategy, Wang et al. (2023b) conduct
case studies on in-context sentiment analysis, where they find that the label
words from the in-context examples serve as anchors of the final prediction.
In specific, these anchors aggregate information from the examples to generate
informative representations from the lower layers, while the deeper layers
utilize these representations to make final predictions. This insight
motivates them to reweight the attention scores of these anchors to achieve
better inference accuracy. Some researchers (Dar et al., 2023; Wu et al.,
2023) extend this framework to globally analyze the attention weights
$\mathbf{W}_{q}$ and $\mathbf{W}_{k}$ by feeding the static word embeddings of
words from an interested vocabulary, instead of their contextual embeddings.
For example, with this approach, Wu et al. (2023) find that instruction tuning
empowers LLMs to follow human intentions by encouraging them to encode more
word-word relations related to instruction words. On the other hand, some
mathematical models are proposed to theoretically explain the self-attention
mechanism, such as Sparse Distributed Memory (Bricken & Pehlevan, 2021) and
Transformer Circuits (Elhage et al., 2021). In particular, the Transformer
Circuits provide mechanistic interpretability to transformer-based models,
which breaks models down into human-understandable pieces. Although these
theoretical analyses on self-attention solids a foundation for future
research, their direct application is largely underexplored.

###  3.2 Understanding the Feed-Forward Module

A feed-forward network is formalized as
$\mathbf{x}^{\prime}=\sigma(\mathbf{x}\mathbf{W}_{u})\mathbf{W}_{v}^{\top}$,
where $\mathbf{x}\in\mathbb{R}^{1\times D_{1}}$ is the intermediate contextual
representation of an input word, $\sigma$ is a non-linear operation, and
$\mathbf{W}_{u},\mathbf{W}_{v}\in\mathbb{R}^{D_{1}\times D_{3}}$ are model
parameters. Feed-forward networks can be understood as key-value memories
(Sukhbaatar et al., 2015; Geva et al., 2021), where each key or value is
defined as $\mathbf{W}_{u}[d]\in\mathbb{R}^{D_{1}}$ and
$\mathbf{W}_{v}[d]\in\mathbb{R}^{D_{1}}$, respectively. That is, each feed-
forward network obtains $D_{3}$ key-value pairs, called memories. One simple
way to interpret the semantic meaning of memory is collecting the words that
could maximally activate the key or value vector of that piece of the memory
(Geva et al., 2021; Dar et al., 2023), which has demonstrated strong
interpretability of the extracted word lists. However, it is critical to be
aware that the key or value vectors are polysemantic (Arora et al., 2018;
Scherlis et al., 2022; Bricken et al., 2023), indicating that this simple
approach might not provide concise explanations for each key-value pair. It
has been shown that the word list of each key-value pair has 3.6 human
interpretable patterns on average (Geva et al., 2021). To alleviate the
limited interpretability caused by the nature of polysemantic, Wu et al.
(2023) propose to interpret the principal components of these key or value
vectors, leading to a more concise explanation for each word list, such as
“medical abbreviations” and “programming tasks and actions”. Other work
examines individual memories by measuring the changes of predictions after
perturbing their corresponding activations, where it reveals that some
memories encode specific knowledge (Dai et al., 2022) and some others capture
general concepts (Wang et al., 2022b). By leveraging the explanations of key-
value memories, we could locate and update the memories associated with a
specific piece of knowledge to perform model editing (Dai et al., 2022; Meng
et al., 2022a; b; Hase et al., 2024), i.e., modifying outdated or incorrect
knowledge. Another usage of these weight explanations is model pruning, where
LLMs can be condensed up to 66.6% of their initial parameters by exclusively
maintaining redundant neurons, resulting in an approximate 1.4x increase in
inference speed (Dalvi et al., 2020).

Besides interpreting and analyzing model weights, some work studies model
activations (e.g., $\mathbf{x}\mathbf{W}_{u}$ or $\mathbf{x}^{\prime}$) to
interpret their functionality. The probing technique is the most popular way
for this purpose (Belinkov et al., 2018; Tenney et al., 2018; Jawahar et al.,
2019; Rogers et al., 2021), identifying whether a specific concept is encoded
within the representations. The basic idea is developing an auxiliary
classifier $g$ to map from the representations $\mathbf{x}^{\prime}$ to the
interested concept space $\mathcal{C}$, such as syntax and part-of-speech
knowledge, and the performance of $g$ interprets how much information encoded
in $\mathbf{x}^{\prime}$ is related to the concepts in $\mathcal{C}$. This
technique motivates developing better parameter-efficient (Chen et al., 2022),
domain-specific (Das et al., 2023), and robust (Bai et al., 2021; Wang et al.,
2023a) LLMs. Recent studies (Chen et al., 2023a; Ahdritz et al., 2024) also
apply the probing method to detect the knowledge boundary of a LLM so that the
hallucinated responses could be reduced. Some researchers (Bricken et al.,
2023; Cunningham et al., 2023) point out another direction to interpret the
model hidden activations, called dictionary learning, which is motivated by
the assumption of superposition (Elhage et al., 2022; Sharkey et al., 2022).
The superposition assumes that LLMs would learn an over-complete set of non-
orthogonal features, thereby transcending the limitations imposed by the
dimensionality of the representation space. Therefore, the researchers aim to
reconstruct and interpret these features to understand the internals of the
model. Practically, they develop a sparse auto-encoder $g$ to reconstruct the
representations $\\{\mathbf{x}_{n}\\}$, which shows that humans could well
interpret the learned sparse features of $g$ according to their most
activation words. Their research shows that this method could be used for more
controllable generation. Specifically, if forcing a sparse feature to be
activated, then the language model $f$ would change its response to perform
the particular behavior of that sparse feature. For example, given
“1,2,3,4,5,6,7,8,9,10” as input, the model originally generates numbers as
output. However, when they are forced to magnify the activations of a sparse
feature called “DNA”, the model changes its output to “AGACCAGAGAGAAC”. In
general, while the explanation techniques for feed-forward networks primarily
offer insights for model development, they have also demonstrated promising
applications in areas such as model editing and controllable generation.

###  3.3 Challenges

Interpreting the functionality of internal modules is still in its infancy,
and we identify two challenges to be tackled in this direction.

####  3.3.1 Complexity of Individual Models and Their Interactions

The transformer-based language model contains two types of modules that
collaborate based on the residual mechanism (He et al., 2016), which enables
later modules to utilize, enhance, and/or discard outputs from preceding
modules. Formally, the output of the $l$-th module is denoted as
$x^{l}=f^{l}(x^{l-1})+x^{l-1}$, where $f^{l}$ could be a self-attention module
or a feed-forward network. Research in this area aims to interpret how
different modules $f^{i}$ and $f^{j}$ work together for $i\neq j$. Pilot
studies (Elhage et al., 2021; Olsson et al., 2022) find that stacked self-
attention modules could form Induction Heads, which demonstrate a strong
correlation with the in-context learning capability. Specifically, the
induction head encourages the model to predict the word “B” followed by a
sequence “AB…A”. Their study finds a specific phase during pre-training LLMs
where both induction heads and the in-context learning capability emerge from
the model. Following this track, researchers observe diverse functional heads
within LLMs for different tasks, such as “Name Mover Head” and “Duplicate
Token Head” for the object identification task (Wang et al., 2022a), “Single
Letter Head” and “Correct Letter Head” for the multiple-choice question
answering task (Lieberum et al., 2023), and “Capitalize Head” as well as
“Antonym Head” for the general purpose tasks (Todd et al., 2023). Although
these studies have indeed deepened our understanding of cross-module effects,
their analyses are grounded on specific tasks or scenarios, leaving it
uncertain whether these findings possess broad generalizability.

####  3.3.2 Nature of Polysemantic and Superposition Assumption

Interpreting the functionality of a single neuron (one row/column vector of a
weight matrix) fails in analyzing large language models since a single neuron
could be activated by multiple and diverse meanings, called polysemantic
(Arora et al., 2018; Scherlis et al., 2022; Bricken et al., 2023). This nature
leads to poor interpretability: explaining a single neuron usually does not
reflect a concise human concept. Some researchers (Elhage et al., 2022;
Sharkey et al., 2022) assume that this phenomenon is caused by the
superposition of an over-complete set of features learned by the models. Based
on this assumption, we may reach another level of explanation by decomposing
the model weights to reconstruct a large number of features. However, the two
critical problems of this approach are still unclear: (1) How do we ensure our
reconstructed features faithfully represent the latent features learned by the
models? (2) How do we interpret our reconstructed features with human
language?

##  4 LLM Debugging with Sample-based Explanation

In this section, we discuss sample-based explanation strategies for LLMs,
which aim to trace back the answers generated by LLMs to specific training
samples (i.e., documents) or document fragments in the corpora. The utility of
sample-based explanations for LLMs is multifaceted. First, tracing back the
predictions of LLM to the training samples can provide evidence for the
generation results, which facilitates model debugging in cases of errors and
increases the trustworthiness of the model from users when the outcomes are
accurate. In addition, it can also help researchers understand how LLMs
generalize from training samples. If the outputs of LLMs can be traced back to
exact subsequences directly spliced from the training data, it might suggest
that the LLM is simply memorizing the data. In contrast, if the generation
results and the influencing training samples are abstractly related, it could
indicate that LLMs can understand the concepts and generate responses by
reasoning from input prompts.

In this section, we start by systematically reviewing traditional sample-based
explanation strategies, including gradient-based methods and embedding-based
methods, as well as some preliminary explorations to generalize them to LLMs.
We then analyze the challenges associated with generalizing the above
strategies to LLMs with unique transformer structures and unprecedented
numbers of parameters. Finally, we discuss the insights to address the
challenges, as well as open challenges worthy of further investigation.

###  4.1 Literature Review of Sample-based Explanation

In this section, we denote the input space and output space as $\mathcal{X}$
and $\mathcal{Y}$, respectively. In the context of large language models
(LLMs), $\mathcal{X}$ is the space of token sequences known as the prompts,
and $\mathcal{Y}$ could be the space of discrete labels in classification
tasks or the space of token sequences as output in generation tasks***Here, we
should note that, in the pretraining phase of LLM with language modeling, the
model either predicts masked tokens in $x_{i}$ (i.e., masked language
modeling), or autoregressively predicts the next token (i.e., causal language
modeling) in $x_{i}$. Therefore, in some works, $y_{i}$ is omitted, and only
$x_{i}$ is included for discussions.. Accordingly, we have a training dataset
$\mathcal{D}_{train}=\\{z_{i}=(x_{i},y_{i})\\}_{i=1}^{N}$ with $N$ samples
drawn from the joint space $\mathcal{X}\times\mathcal{Y}$, on which an LLM
model $f_{\theta}$ is trained with pretrained parameters
$\hat{\theta}\in\mathbb{R}^{P}$. We also have a test sample $z=(x,y)$ of
interest, where we want to explain the generation of $y$ from $x$ based on
training samples in $\mathcal{D}_{train}$ (which can be viewed as the
information source). The goal of sample-based explanation is to measure the
influence of a training sample $z_{i}\in\mathcal{D}_{train}$ or a certain
segment within $z_{i}$, such that the generation of LLMs can be well-explained
and backed up by the selected training samples.

####  4.1.1 Influence Function-based Methods

One strategy to quantify the influence of a training sample $z_{i}$ in the
dataset $\mathcal{D}_{train}$ to a test sample $z$ is through the influence
function (Koh & Liang, 2017; Han et al., 2020). It measures the change of the
prediction loss $\mathcal{L}\left(z,\theta\right)$ for the test sample $z$,
when the training sample $z_{i}$ undergoes a hypothetical modification in the
dataset $\mathcal{D}_{train}$ during model training. This modification results
in an altered set of optimal model parameters, denoted as
$\hat{\theta}_{-z_{i}}$. The most common modification of a training sample is
to remove it from the dataset, where the influence of the removal of a
training sample $z_{i}$ on the loss at test sample $z$ can be computed as
follows:

$\mathcal{I}(z_{i},z)=-\nabla_{\theta}\mathcal{L}(z,\hat{\theta})^{\top}\mathbf{H}_{\hat{\theta}}^{-1}\nabla_{\theta}\mathcal{L}(z_{i},\hat{\theta}),$
(1)

where $\nabla_{\theta}\mathcal{L}(z,\hat{\theta})$ is the gradient of the loss
function $\mathcal{L}$ on the test sample $z$ evaluated at the optimal
parameters $\hat{\theta}$, and
$\mathbf{H}_{\hat{\theta}}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}^{2}\mathcal{L}(z_{i},\hat{\theta})$
denotes the Hessian matrix of the LLM model at parameter $\hat{\theta}$. If we
denote the number of parameters in $\hat{\theta}$ as $P$, the naïve inversion
of the Hessian matrix $\mathbf{H}_{\hat{\theta}}$ leads to
$\mathcal{O}(NP^{2}+P^{3})$ time complexity and $\mathcal{O}(P^{2})$ space
complexity (Schioppa et al., 2022), which is clearly infeasible for large
models. To improve efficiency, Koh & Liang (2017) adopt an iterative
approximation process, i.e., LiSSA (Linear time Stochastic Second-Order
Algorithm), to calculate the Hessian-Vector Product (HVP) in Eq. (1), where
the memory complexity can be reduced to $\mathcal{O}(P)$ and time complexity
to $\mathcal{O}(NPr)$ ($r$ is the number of iterations). To further reduce the
complexity, Pruthi et al. (2020) propose an alternative to Eq. (1), i.e.,
TracIn, which measures the influence of $z_{i}$ on $z$ by calculating the
total reduction of the loss on $z$ whenever $z_{i}$ is included in the
minibatch during model training. The TracIn measurement can be formulated as
follows:

$\mathcal{I}_{\operatorname{TracIn}}\left(z_{i},z\right)=\sum_{t:z_{i}\in\mathcal{B}_{t}}\mathcal{L}\left(z,\theta_{t}\right)-\mathcal{L}\left(z,\theta_{t+1}\right)\approx\frac{1}{b}\sum_{t:z_{i}\in
B_{t}}\eta_{t}\nabla_{\theta}\mathcal{L}\left(z_{i},\theta_{t}\right)\cdot\nabla_{\theta}\mathcal{L}\left(z,\theta_{t}\right),$
(2)

where $\mathcal{B}_{t}$ is the $t$-th mini-batch fed into the model during
training, $\theta_{t}$ is the parameter checkpoint at the $t$-th step,
$\eta_{t}$ is the step size, and $b$ is the size of the mini-batch. According
to the above equation, TracIn only leverages gradient terms, where Hessian
$\mathbf{H}_{\hat{\theta}}$ is removed from the influence measurement. This
substantially improves the efficiency. However, such complexity is still
prohibitive for large models from both the computational and memory
perspectives. In addition, TracIn can only estimate the influence of
adding/removing the sample to the loss, where variants of the vanilla
influence function defined in Eq. (1) can measure the influence of other
modifications of the training sample $z_{i}$, such as perturbation (e.g.,
masking out a segment of a document $x_{i}$). To adapt the vanilla influence
function of Eq. (1) to explain transformers, Schioppa et al. (2022) propose to
use Alnordi iteration (Arnoldi, 1951) to find the dominant eigenvalues and
eigenvectors of the Hessian matrix on randomly sampled subsets
$\mathcal{D}_{sub}$, with $|\mathcal{D}_{sub}|\ll|\mathcal{D}_{train}|$. In
such a case, the diagonalized Hessian can be cheaply cached and inverted,
where the computational and memory complexity can be substantially reduced.
Previous work mainly focuses on reducing the complexity of calculating the
influence of a single training sample. Observing that finding the most
influential training sample on $z$ needs to iterate Eq. (1) overall $N$
training samples, Guo et al. (2021) propose to use fast KNN to pre-filter a
small subset of influence-worthy data points from $\mathcal{D}_{train}$ as
candidates to explain small pretrained language models, whereas Han & Tsvetkov
(2022) propose to iteratively find a small subset
$\mathcal{D}_{sub}\subset\mathcal{D}_{train}$ whose gradient is the most
similar to that of the downstream task examples. Recently, Grosse et al.
(2023) propose to use the Eigenvalue-corrected Kronecker-Factored Approximate
Curvature (EK-FAC) approximation to scale influence functions to LLMs up to
52B parameters. For adaptation, only influences mediated by the multi-layer
perceptron (MLP) layers are considered, whereas weights from the self-
attention layers are fixed, as previous work has demonstrated that knowledge
is mainly encoded in the MLP layers (Wang et al., 2023d). In addition, based
on the assumption that weights from different MLP layers are independent, the
EK-FAC approximated influence can be formulated as the sum of influences
mediated by each layer:

$\mathcal{I}_{\text{EKFAC}}(z_{i},z)=\sum_{l}\nabla_{\theta^{(l)}}\mathcal{L}(z,\hat{\theta})^{\top}(\hat{\mathbf{G}}_{\hat{\theta}^{(l)}}+\lambda^{(l)}\mathbf{I})^{-1}\nabla_{\theta^{(l)}}\mathcal{L}(z_{i},\hat{\theta}),$
(3)

where $\theta^{(l)}$ denotes the weights of the $l$-th MLP layer, and
$\hat{\mathbf{G}}_{\hat{\theta}^{(l)}}$ is the EK-FAC approximated Gauss-
Newton Hessian for $\theta^{(l)}$. Since the inversion of $L$ small
${K_{l}\times K_{l}}$ matrices (i.e., $\mathcal{O}(L\times K_{l}^{3})$) is
substantially more efficient than the inversion of a large $L{K_{l}\times
LK_{l}}$ matrix (i.e., $\mathcal{O}{((LK_{l}})^{3})$),
$\mathcal{I}_{\text{EKFAC}}$ can be adaptable to very large models, let alone
the HVP can be further simplified by utilizing the EK-FAC properties.

Recently, influence function-based explanation has been used in efficient
finetuning of LLMs, where influence function has been used to select a small
subset of training samples given few-shot validation samples for a specific
downstream task, where the training overhead can be substantially improved
(Xia et al., 2024).

####  4.1.2 Embedding-based Methods

Another strategy for sample-based explanation involves leveraging the hidden
representations within the transformer architecture, which is recognized for
encoding high-level semantics from textual data, to calculate the semantic
similarity between $z$ and $z_{i}$. The similarity can also be used to measure
the influence of $z_{i}$ on $z$ as explanations (Rajani et al., 2019).
Specifically, Akyurek et al. (2022) propose to represent the training sample
$z_{i}$ and test sample $z$ by concatenating the input and output as
$z^{cat}_{i}=[x_{i}||y_{i}]$, $z^{cat}=[x||y]$. The concatenation is feasible
for generation tasks where the output $y$ lies in the same token sequence
space as the input prompt $x$. The similarity between $z_{i}$ and $z$ can then
be calculated as follows:

$\mathcal{I}_{\text{emb}}\left(z_{i},z\right)=\frac{f^{(l)}_{\hat{\theta}}(z^{cat}_{i})^{\top}\cdot
f^{(l)}_{\hat{\theta}}\left(z^{cat}\right)}{\left\|f^{(l)}_{\hat{\theta}}(z^{cat}_{i})^{\top}\right\|\left\|f^{(l)}_{\hat{\theta}}\left(z^{cat}\right)\right\|},$
(4)

where $f^{(l)}_{\hat{\theta}}$ is the sub-network that outputs the $l$-th
layer intermediate activation of the pretrained LLM $f_{\hat{\theta}}$. The
Eq. (4) has a similar form as the vanilla influence function defined in Eq.
(1) as well as its TracIn alternative defined in Eq. (2), which assigns a
score $\mathcal{I}$ for the explainee $z$ for each training sample $z_{i}$ in
the dataset $\mathcal{D}_{train}$ as the explanation confidence of the sample
$z_{i}$.

Compared with the influence function methods introduced in the previous part,
embedding-based methods are computationally efficient, as for each explainee
$z$, the explanation score from a training sample $z_{i}$ requires only one
forward pass of the transformer network. In addition, the calculation can be
easily paralleled for different training samples. However, the disadvantage is
also evident: These methods lack a theoretical foundation and may fail to
identify important training samples that may not be semantically similar to
the test sample. Consider the following toy example: Training samples $z_{i}$
= (“1+1=”, “2”) and $z_{j}$ = (“2+2=”, “4”) make the LLM gain the ability to
conduct arithmetic calculation, which explains why prompting the model with
$x$ = “100+100” gives the results $y$=“200”. However, the embeddings between
the test sample $z$ and the two training samples $z_{i}$ and $z_{j}$ can be
very different when calculated via Eq. (4) (Akyurek et al., 2022). Therefore,
embedding-based methods may not be able to faithfully find the training
samples where the explanations require generalization ability beyond semantic
similarity.

###  4.2 Case Study: EK-FAC-based Influence Estimation

In this part, we implement the EK-FAC-approximated influence function proposed
in Grosse et al. (2023), and verify its scalability and effectiveness on LLMs
with billions of parameters, including GPT2-1.5B (Radford et al., 2019),
LLaMA2-7B (Touvron et al., 2023b), Mistral-7B (Jiang et al., 2023), and
LLaMA2-13B.

####  4.2.1 Experimental Design

We use the SciFact dataset (Wadden et al., 2020) as the corpora, which
contains the abstract of 5,183 papers from the domain of basic science and
medicine. The explainee LLMs are obtained by finetuning the pretrained LLMs
for 20,000 iterations, where AdamW (Loshchilov & Hutter, 2018) is used as the
optimizer, and the learning rate and weight decay are set to 1e-5, 1e-3,
respectively. Then, we use 500 samples from the corpora to estimate the (i)
uncentered covariance matrices of the activations and pre-activation pseudo-
gradients $\mathbf{Q}^{(l)}_{A}$, $\mathbf{Q}^{(l)}_{S}$, and (ii) the
variances of the projected pseudo-gradient $\bm{\Lambda}^{(l)}$ for each
selected dense layer $l$, and cache them on the hard disk (details see Eqs.
(16) and (20) in Grosse et al. (2023)). We select the c_fc layer for
GPT2-1.5B, and gate_proj layer for LLaMA2-7B, Mistral-7B, and LLaMA2-13B***All
the implementation and layer names are based on the huggingface transformers,
where the details can be found in
https://huggingface.co/docs/transformers/en/index..

For evaluation, we randomly select 200 samples from the corpora to construct
the test set, which we name SciFact-Inf. Specifically, for the $j$-th selected
sample $z_{j}=(x_{j},x_{j})$ (here $y_{j}=x_{j}$ as label equals the input in
language modeling), we use the first three sentences in $x_{j}$, i.e.,
$\hat{x}_{j}$, to generate a completion $\hat{y}_{j}$ with the finetuned LLM
(here, $\hat{y}_{j}$ does not equal the remaining sentences in $x_{j}$), and
we aim to explain the generation of $\hat{y}_{i}$ from $\hat{x}_{j}$ with the
finetuned LLM with the training samples via EK-FAC approximated influence
scores defined in Eq. (3). Ideally, the $j$-th training sample $z_{j}$ itself
should be the most influential sample w.r.t. the generation of $\hat{y}_{j}$
for test sample $\hat{z}_{j}$, which facilitates quantitative analysis of the
effectiveness of Eq. (3).

In our implementation, for each test sample $\hat{z}_{j}$, we first calculate
the EK-FAC approximated HVP part of the influence
$\mathcal{I}_{\text{EKFAC}}(z_{i},\hat{z}_{j})$, i.e.,
$\sum_{l}\nabla_{\theta^{(l)}}\mathcal{L}(\hat{z}_{j},\hat{\theta})^{\top}(\hat{\mathbf{G}}_{\hat{\theta}^{(l)}}+\lambda^{(l)}\mathbf{I})^{-1}$,
which is shared for all training samples $z_{i}$. Specifically, we record the
layer-wise gradient
$\nabla_{\theta^{(l)}}\mathcal{L}(\hat{z}_{j},\hat{\theta})$ and calculate the
HVP with the cached $\mathbf{Q}_{A}^{(l)}$, $\mathbf{Q}_{S}^{(l)}$ as Eq. (21)
in Grosse et al. (2023). We then go through candidate training samples (1
positive and 99 negative), calculate the gradient
$\nabla_{\theta^{(l)}}\mathcal{L}(z_{i},\hat{\theta})$, and take inner-product
with the approximate HVP as the layer-wise influence. Finally, the layer-wise
influences are summed up as Eq. (3) as the total influence
$\mathcal{I}_{\text{EKFAC}}(z_{i},\hat{z}_{j})$. We rank the influence and
calculate the top-$K$ hit rate of the positive training sample.

####  4.2.2 Results and Analysis

Table 4: Effectiveness of EK-FAC approximated influence function on the established SciFact-Inf dataset. Time (Pre.) stands for the time for precomputing the $\mathbf{Q}_{A}$, $\mathbf{Q}_{S}$, and $\bm{\Lambda}$. Time (Inf.) stands for the time for calculating the influence of 100 training samples per test sample. Experiments for GPT2-1.5B, LLaMA2-7B, Mistral-7B are done on 4 A100 GPUs, whereas experiments for LLaMA2-13B are done with 8 A100 GPUs. Strategy | LLM | Recall@5 | Recall@10 | Time (Pre.) | Time (Inf.)
---|---|---|---|---|---
Random | - | 0.0100 | 0.0200 | - | -
Inf. Func. | GPT2-1.5B | 0.6368 | 0.7363 | 0h 27min | 0min 28sec
Mistral-7B | 0.6418 | 0.6866 | 2h 05min | 1min 47sec
LLaMA2-7B | 0.8063 | 0.8308 | 1h 37min | 1min 34sec
LLaMA2-13B | 0.7811 | 0.8940 | 3h 11min | 3min 08sec

The experimental results are summarized in Table 4. From Table 4 we can find
that, the EK-FAC approximated influence function achieves a good accuracy in
finding the training sample that has the greatest influence on the generation
of a test sample, even if only the influences mediated by a small part of
dense layers are considered. In addition, we find that the main computational
bottleneck in calculating the EK-FAC-based influence is to estimate the
covariances $\mathbf{Q}^{(l)}_{A}$, $\mathbf{Q}^{(l)}_{S}$ and variance
$\bm{\Lambda}^{(l)}$, which can take hours when 500 training samples are used
for the estimation. However, after the estimation, it is relatively cheap to
calculate the influence of 500 training samples for each test sample, which
takes around 3 minutes for a 13B LLaMA2 model on 8 A100 GPUs. This
demonstrates the scalability of EK-FAC-based influence by assuming the
independence of different dense layers and using EK-FAC to simplify the
computation.

###  4.3 Challenges

Overall, explaining the generation of LLMs by tracing back to the training
samples is still an emerging area. Open questions need to be addressed to
further advance the field. In this section, we identify three main challenges
as follows, which can serve as directions for future explorations.

####  4.3.1 Strong Assumptions for Scalability

The unprecedented number of parameters in modern LLMs causes severe
scalability issues for sample-based explanation strategies. This is especially
evident for the gradient-based methods, as the HVP in Eq. (1) induces both
high computational and space complexity. To address the bottleneck, strong
assumptions are usually required to make it feasible for large models. For
example, TracIn (Pruthi et al., 2020) simplifies the second-order term in Eq.
(1) via first-order approximation. Schioppa et al. (2022) assume the Hessian
to be low rank. Grosse et al. (2023) that assume that the weights from
different layers of the LLMs are independent, as well as the tokens in
different steps, such that EK-FAC can be appropriately applied to approximate
the influence function. From the above analysis, we can find that while the
method from Grosse et al. (2023) has the best scalability, it also has the
strongest assumption, which may fail to hold in practice. While highly
efficient to compute, embedding-based methods make the implicit assumption
that semantics similarly implies explainability, which we have demonstrated
may not always be the case. Therefore, how to improve the scalability with
weak assumptions needs to be investigated in the future.

####  4.3.2 Explainability v.s. Understandability

Despite the advantage of influence/embedding similarity to provide a
qualitative measurement of a specific training sample as the explanation for
LLM generation, the understandability of the identified sample can still be
weak, where the connection between the selected training samples and the
generation may not be understandable to human beings. Specifically, Grosse et
al. (2023) cautions that the sign of influence score of the training tokens
may be difficult for humans to connect to the positive or negative influence
on the generation results. This severely jeopardizes the usability of the
identified training samples. In addition, Grosse et al. (2023) also found
that, since LLMs are usually not trained to the minimum to avoid overfitting
(and due to overparameterization, the number of local minimums may be large),
the connection between influence defined in Eq. (1) with the counterfactual
loss of removing the sample $z_{i}$ at $z$ is also weak. For the embedding-
based methods, since most LLM models are black box transformer models, the
similarity of embeddings can also be hard to interpret by human beings;
therefore, it is imperative to improve the interpretability of the identified
training samples, such that tracing back becomes more meaningful.

####  4.3.3 LLM-Oriented Sample-based Explanations

Finally, we observed that both gradient-based and embedding-based methods are
loosely connected to the LLM, as well as the backbone transformer networks.
For example, algorithms like TracIn (Pruthi et al., 2020) are designed to
scale up influence functions to large models, which are not specific for LLMs.
Similarly, the embedding-based method proposed in Akyurek et al. (2022) is
applicable to most machine learning models with latent representations. Grosse
et al. (2023) considers the specialty of LLMs by utilizing the knowledge
neuron assumption of the backbone transformers (Wang et al., 2023d) to
simplify the influence function, where the weights considered are constrained
to the MLP layers, which may not fully utilize the property of transformers.
Therefore, how to further utilize the property of the LLM and the backbone
transformer to design LLM-tailored sample-based influence/similarity (either
to reduce the computational/space overhead or to improve the explanation
quality) is highly promising for future work.

##  5 Explainability for Trustworthy LLMs and Human Alignment

In previous sections, we explore the use of explanation techniques for
assessing and improving the performance of LLMs. In this section, we shift the
focus towards examining LLM trustworthiness. As LLMs are increasingly
integrated into various applications of daily life, including high-stakes
areas like healthcare, finance, and legal advice, it is crucial that their
responses not only are accurate but also align with human ethical standards
and safety protocols (Liu et al., 2023b; Li et al., 2023f). Thus, the need
arises to extend the scope of explanations from merely evaluating the accuracy
of LLMs to scrutinizing their trustworthiness. Herein, we delve into how
explanation techniques, discussed in the previous sections, can be
instrumental in assessing LLMs across key aspects of trustworthiness like
security, privacy, fairness, toxicity, and honesty. It is worth noting that
while explainability itself is an aspect of trustworthiness, it holds the
promise of serving as a foundational tool for addressing other trustworthiness
concerns.

###  5.1 Security

LLMs are known to be vulnerable to attacks and exploitation, such as spreading
misinformation, launching phishing attacks, and poisoning training data
(Derner et al., 2023). For enhanced safety, LLMs are designed to reject
certain types of prompts that may result in the generation of harmful content,
e.g., by precluding prompts likely to elicit unsafe outputs from the training
phase. However, jailbreak techniques can circumvent these restriction measures
and manipulate LLMs into producing malicious contents. Malevolent users (i.e.,
attackers) can craft special prompts that compel or induce LLMs to prioritize
instruction following over rejections (Liu et al., 2023c; Li et al., 2023a).
For example, through Prefix Injection, attackers can use out-of-distribution
prompt prefixes that are less likely to be rejected (walkerspider, 2022; Wei
et al., 2023). Another approach, called Refuse Suppression, involves directing
or persuading models to ignore established safety protocols (Wei et al., 2023;
Zeng et al., 2024), where the instruction following ability is then employed
to perform the attack.

Existing methods mainly rely on prompt engineering for attacking LLMs, but
they usually have low attack success rates and significant time costs (Li et
al., 2024c). Thus, by understanding and engineering latent representations of
LLMs, explanation methods provide a viable way to design advanced attacks and
discover the potential vulnerabilities of LLMs (Liu et al., 2021). For
example, a recent work extracts “safety patterns” via explaining the latent
space of LLMs using representation engineering. Specifically, these patterns
can be captured from the activation differences between malicious queries and
benign queries. The salient portion of difference vectors’ dimensions is
localized and utilized to generate features of safety patterns. The safety
patterns reflect the internal protection mechanisms within LLMs. Circumventing
these patterns leads to novel attacks, which helps exploring potential
vulnerabilities of LLMs (Li et al., 2024c). Besides, a deeper understanding of
fine-tuning can shed light on the reliability of existing safety measures. In
particular, Jain et al. (2023) use networking pruning, attention map
activation, and probing classifiers to track the changes of model capabilities
from pre-training to fine-tuning. These tools are helpful in finding salient
weights that identify crucial neurons to restore pre-training capabilities.
These neurons have demonstrated that the capabilities gained during fine-
tuning can be removed easily through fine-tuning on other unrelated tasks.
This finding casts doubt on the robustness of current safety alignments in
LLMs.

###  5.2 Privacy

Recent studies have revealed that LLMs such as ChatGPT can leak extensive
amounts of training data through a method known as divergence attacks. These
attacks utilize specially crafted prompts to lead the model away from its
standard chatbot-style generation (Nasr et al., 2023). The risk of private
data exposure through such means poses a serious challenge to the development
of ethically responsible models. This issue is compounded by strategies
similar to jailbreak attacks, where misalignment is exploited to induce LLMs
into operating in an unconventional “developer mode” via out-of-distribution
prompts (Li et al., 2023a). Traditional data pre-processing techniques, such
as data sanitization (Lison et al., 2021), are impractical as defenses for
LLMs due to the enormous scale of training data.

Enhancing LLM privacy involves two strategic approaches: (1) preventing the
models from memorizing sensitive data, and (2) establishing safeguards against
the release of sensitive information during content generation. The latter can
employ techniques used in jailbreak defenses, treating prompts that solicit
private information as potentially malicious. The former approach requires
identifying whether LLMs possess specific knowledge, which is traditionally
tested by crafting question-answering (QA) tasks to determine if LLMs can
provide answers. However, this approach faces limitations due to LLMs’
sensitivity to the phrasing of QA prompts, while the optimal prompt is usually
unknown. To tackle the challenge, explanatory techniques can serve as a tool
to confirm whether LLMs have internalized certain knowledge. For instance, via
explaining the relation between factual knowledge and neuron activations (Meng
et al., 2022a; Dai et al., 2022; Hase et al., 2024), we may investigate
whether and where a piece of factual knowledge is stored within transformers.
In addition, Yin et al. (2024a) recently proposes the concept of “knowledge
boundary” and develops a gradient-based method to explore whether LLMs master
certain knowledge independent of the input prompt.

###  5.3 Fairness

Despite LLMs’ powerful generation capabilities, their widespread applications
also bring concerns about exacerbating bias issues in society, as LLMs are
able to learn social biases within human-generated corpus (Gallegos et al.,
2023). For example, in a gender bias case, “[He] is a doctor” is much more
likely than “[She] is a doctor” because of the gender stereotype. In this
subsection, we focus on fairness issues that refer to biases related to race,
gender, and age within human communities (Li et al., 2023g). There is a rich
body of literature quantifying fairness issues within LLMs through various
tests (Adebayo et al., 2023). Interpretation complements these methods by
providing a unique perspective for bias mitigation, focusing on unraveling the
mechanisms through which biases are embedded into LLMs. A research direction
within this domain is the examination of biased attention heads. For instance,
Ma et al. (2023) detect stereotype encodings through probing attention heads
and evaluating their attributions via Shapley values (Lundberg & Lee, 2017).
The results unveil that approximately 15% to 30% of attention heads across six
transformer-based models are linked to stereotypes. These attention heads tend
to specialize in maintaining various stereotypes, offering a potential pathway
for developing effective de-biasing techniques. Moreover, scoring head biases
with gradient-based metrics offers another way for identifying biased heads
(Yang et al., 2023a). Furthermore, recent work has placed LLM representations
under scrutiny (Zou et al., 2023a). Typically, specific templates relevant to
a given concept or function are designed beforehand. Then, representations
closely aligned with the concepts or functions are examined using principal
component analysis (PCA). From this analysis, a vector is derived from the
first principal component to predict a certain bias.

To achieve fair model predictions, a diverse range of mitigation techniques
have been proposed to debias models. One stream of work proposes to debias
LLMs at the embedding level, refining embeddings through various approaches.
For example, a recent work attempts to alter biased embeddings with minimal
alterations to make them orthogonal to neutral embeddings (Rakshit et al.,
2024). Additionally, some studies concentrate on removing biases at the level
of attention heads. Ma et al. (2023) address this by pruning attention heads
that significantly contribute to certain biases. Similarly, another study
adapts movement pruning (Sanh et al., 2020) to dynamically choose an optimal
subset of low-bias attention heads (Joniak & Aizawa, 2022). Beyond modifying
embeddings and pruning attention heads, another strategy involves targeting a
specific group of neurons known to propagate biases. It unlearns the biases by
retraining weight vectors for these neurons (Yu et al., 2023). Besides, bias
mitigation can also be approached from a data-centric perspective using a few
training samples (Thakur et al., 2023). This work uses a pre-trained model to
find the most biased training examples, and then modifies these examples to
fine-tune the model.

###  5.4 Toxicity

Toxicity is another form of harmful content that LLMs may produce. This issue
arises because LLMs are often trained on extensive online text corpora that
have not been thoroughly filtered, containing elements of toxicity that can
hardly be fully eliminated. Toxicity can be identified by interpreting LLM
components like the feed-forward layers and attention heads. For instance,
recent work reveals how toxicity is represented within LLMs by identifying
multiple vectors promoting toxicity within the MLP layers and further
recognizes relevant dimensions through singular value decomposition (Lee et
al., 2024). Furthermore, the exploration of geometric structures in per-layer
representations offers another way to detect toxicity. Balestriero et al.
(2023) applied spline formulations within MLPs to extract seven characteristic
input features, demonstrating their utility in describing the domain of
prompts and classifying toxic remarks.

The insights into how toxicity manifests within LLMs also shed light on
mitigation strategies. Motivated by the finding that toxicity can be reduced
by manipulating relevant vectors, Lee et al. (2024) develops a method called
direct preference optimization (DPO), where paired toxic and non-toxic samples
are utilized to fine-tune models so that non-toxic content is promoted. By
examining the changes in the parameter matrices during the fine-tuning
process, it substantiates that even minor adjustments to these critical
vectors can reduce toxicity. Built on the observation that LLMs’
representations are updated by outputs from attention layers (Elhage et al.,
2021), another work attempts to reduce toxicity by identifying the “toxicity
direction” and then adjusting representations in the opposite direction (Leong
et al., 2023).

###  5.5 Truthfulness

One prominent drawback of LLMs is their tendency to confidently produce false
statements. These statements fall into two main categories: 1) statements that
contradict learned knowledge within models, a problem often related to models’
honesty; 2) statements that are factually incorrect and appear to be
fabricated by models, a phenomenon commonly referred to as hallucination. In
the following, we delve into various approaches that aim to understand
aforementioned two behaviors by leveraging explainability tools.

####  5.5.1 Honesty

Honesty of LLMs describes models’ ability to produce true statements based on
their learned information, where dishonest behaviors significantly compromise
the perceived trustworthiness of LLMs. Numerous studies have been undertaken
to understand how and why dishonest behaviors happen by looking inside LLMs.
One notable work attempts to distinguish dishonesty by training a classifier
to predict the accuracy of statements (Azaria & Mitchell, 2023). The
classifier is simply trained on top of activations from the hidden layers of
LLMs. These activations are generated from true and false statements. The
classifier reaches an accuracy range between 60% and 80%, which suggests that
LLMs may be internally aware of the truthfulness of their outputs (Azaria &
Mitchell, 2023). Furthermore, research by Campbell et al. (2023) localizes
dishonesty behaviors at the level of attention heads. This study employs
specially crafted prompts to induce dishonest responses, and then trains
logistic classifiers on models’ activations over true/false statements. It
also employs activation patching to substitute lying activations with honesty
ones. Intriguingly, both approaches have witnessed the importance of layer
23$-$29 in flipping dishonesty behaviors. Besides, another popular method
tries to study the geometric structure of true/false statements(Marks &
Tegmark, 2023). Typically, these structures are visualized by projecting
representations of statements onto two principal components. A clear linear
structure and the truth directions can be derived to mitigate the dishonest
behaviors.

####  5.5.2 Hallucinations

Hallucinations in LLMs can arise due to poor data quality, biases, outdated
knowledge, and the lack of explicit knowledge (Xu et al., 2024; Zhu et al.,
2023b). However, whether LLMs are aware of their hallucination behaviors
remains an open question. Recent work investigates this question by examining
models’ hidden representation space (Duan et al., 2024). It examines three
hidden states involving a question, its correct answer and its incorrect
answer, which are used to compute an “awareness” score. This metric quantifies
the uncertainty of LLMs regarding their own answers, finding that
adversarially induced hallucination can increase models’ awareness.
Additionally, Li et al. (2024b) illustrates the major differences between
models’ output and their inner activations, identifying these discrepancies as
a potential source of hallucination. By training linear probing classifiers on
each attention head’s activations, the most specialized attention head is
identified. An orthogonal probe is subsequently trained with the identified
specialized probe. The experiments reveal that “truth” might exist in a
subspace instead of a single direction (Li et al., 2024b). Another work
investigates the source of hallucination by analyzing patterns of source token
contributions through perturbations (Xu et al., 2023c). Their findings suggest
that hallucinations may stem from the models’ excessive dependence on a
restricted set of source tokens. Besides, the static distribution of source
token contribution, termed as “source contribution staticity”, can be used as
another indicator of hallucinations.

Building on the above insights into LLM hallucinations, Duan et al. (2024)
apply PCA to derive the direction of the correct answer’s final hidden state,
and enhance the hidden representations with this direction to reduce
hallucinations. In contrast, Li et al. (2024b) adopts a different approach, by
intervening on top-$K$ specialized attention heads, while minimizing the
influence of the rest attention heads within models. Different from PCA that
identifies a single principle direction, this work adopts two distinct
techniques to find multiple directions of intervention. First, they use
orthogonal vectors of each probe’s hyperplane, which is similar to PCA.
Second, they leverage vectors that connect the mean of the true and false
distributions (Li et al., 2024b). The vectors derived from mean shift has been
demonstrated more effective than those from probe classifiers, which presents
another feasible strategy for identifying directions of truth.

###  5.6 Challenges

We discuss the challenges in employing explanations to improve models’
trustworthiness and enhance alignment from two aspects: 1) limitations of
existing detection techniques, and 2) shortcomings of mitigation strategies
based on explanations.

####  5.6.1 Challenges of Existing Detection Methods

Current detection methods primarily focus on the level of LLM modules, such as
layers, attention heads, and representations. However, we still lack a finer-
grained understanding of how knowledge is encoded within LLMs, such as at the
level of individual neurons and weight parameters. Furthermore, we lack
general and robust strategies to identify these knowledge reliably. For
instance, to identify gender biases, attention heads might be examined and
then the related heads pruned (Li et al., 2024b). This approach requires
analyzing each model individually, rather than adopting a general approach.
Moreover, existing localization approaches rely either on probing classifiers
or casual scrubbing, which might not be reliable. For probing classifiers, the
pre-designed biases used to train these classifiers are crucial to their
performance. On the other hand, casual cleaning usually introduces new
variables that complicate the analysis.

####  5.6.2 Challenges of Mitigation Strategies

Since LLMs are trained on vast corpora of data, it is impractical to mitigate
the aforementioned trustworthiness issues from the data perspective. It is
widely believed that XAI, i.e. understanding the inner mechanisms of LLMs, is
a fundamental way to solve these issues. Thus, mitigation methods for LLMs are
typically developed based on explanations. Existing explanations are
implemented using techniques from mechanistic interpretability and
representation engineering (Zhao et al., 2024). While both streams of methods
can alleviate these issues, they fail to fully address them. For example,
principal component analysis (PCA) is often used to find geometric structures
related to these issues, but cannot fix these issues. Another popular
mitigation method is casual scrubbing, which introduces certain "positive"
activations to replace those demonstrated to be responsible for specific
issues. However, the identified directions and patched activations can only
mitigate issues to a certain extent. Moreover, the changes to either
representations or activations could also influence other aspects of models’
capabilities, which we are yet unable to evaluate.

##  6 LLM Enhancement via Explainable Prompting

A key distinction between LLMs and traditional machine learning models lies in
the LLMs’ ability to accept flexibly manipulated input data, namely prompts,
during model inference (Liu et al., 2023a). LLMs generally give precedence to
the information presented in these prompts when generating outputs. Therefore,
to mitigate the opacity issue in LLM predictions, we can enhance prompts with
understandable content, which is then prioritized over the LLMs’ inherent and
implicit knowledge. These enriched prompts can include domain-specific
insights, contextual information, or a step-by-step reasoning chain. In
response, LLMs might reveal their decision-making processes during inference,
which improves the explainability of their behaviors.

###  6.1 Chain of Thoughts (CoT) Prompting

The Chain of Thought (CoT) approach significantly enhances LLMs in tackling
complex tasks (Wei et al., 2022). While LLMs are adept at generating human-
like responses, they often lack transparency in their reasoning processes.
This limitation makes it difficult for users to assess the credibility of the
responses, especially for questions that require detailed reasoning.

To bridge this gap, recent efforts have incorporated in-context learning with
human-crafted explanations directly into prompts (Wei et al., 2022; Huang et
al., 2023a; Yao et al., 2023b; Besta et al., 2023). Among these approaches,
the CoT method stands out by employing explicit knowledge to guide the
reasoning process. Formally, we define the language model as $f_{\theta}$, and
input prompt as $X=\\{x_{1},y_{1},x_{2},y_{2},...,x_{n}\\}$, where
$x_{1},y_{1},x_{2},y_{2},...,x_{n-1},y_{n-1}$ denote the example question-
response pairs for in-context learning, and $x_{n}$ is the actual question. In
a standard question-answering scenario, we have the model output as
$y_{n}=\operatorname*{arg\,max}_{Y}p_{\theta}(Y|x_{1},y_{1},x_{2},y_{2},...,x_{n})$.
This approach, however, does not provide insights into the reasoning process
behind the answer $y_{n}$. Therefore, the CoT method proposes to include
human-crafted explanations $e_{i}$ for the $i$-th in-context example,
resulting in a modified input format
$X=\\{x_{1},e_{1},y_{1},x_{2},e_{2},y_{2},...,x_{n}\\}$. Given the input, the
model will output not only $y_{n}$ but also the generated explanation $e_{n}$:

$e_{n},y_{n}=\operatorname*{arg\,max}_{Y}p_{\theta}(Y|x_{1},e_{1},y_{1},x_{2},e_{2},y_{2},...,x_{n}).$
(5)

Besides allowing for a more transparent and understandable interaction with
LLMs, the CoT approach is also practically useful as it augments LLMs’
functionality by opening a window for users to control the models’ thought
processes. Specifically, the usefulness of CoT methods lies in several key
aspects:

  * •

Reducing Errors in Reasoning: By breaking down complex problems into a series
of smaller tasks, CoT reduces errors in complex calculations and logic-
oriented tasks, leading to a more precise resolution of intricate problems
(Wei et al., 2022; Qin et al., 2023; Zhang et al., 2023; Wang & Zhou, 2024b).

  * •

Providing Adjustable Intermediate Steps: CoT enables the outlining of
traceable intermediate steps within the problem-solving process. This feature
enables users to trace the model’s thought process from inception to
conclusion, and to adjust the prompts if undesirable model behaviors are
observed (Lyu et al., 2023; Wang et al., 2023d).

  * •

Facilitating Knowledge Distillation: The step-by-step reasoning processes
derived from larger LLMs can serve as a specialized fine-tuning dataset for
smaller LLMs. It allows smaller models to learn complex problem-solving by
following explanations, effectively teaching them to tackle intricate
questions with enhanced reasoning capabilities (Magister et al., 2022).

###  6.2 Extended Methods of Explainable Prompting

Advanced techniques beyond CoT methods have been developed to broaden the
range of reasoning paths available to LLMs towards enhancing the transparency
and understandability of the decision-making process (Yao et al., 2023b; Besta
et al., 2023; Yao et al., 2023a; Dhuliawala et al., 2023; Lyu et al., 2023).
We introduce several notable examples below.

Tree-of-Thoughts (ToT). Proposed by Yao et al. (2023b), ToT advances beyond
the traditional linear Chain of Thought reasoning, offering a more versatile
structure that allows models to navigate through multiple reasoning paths. ToT
makes the reasoning process of LLMs more interpretable by closely aligning it
with human thought processes, as humans naturally consider multiple options
and possible outcomes in both forward planning and retrospective analysis to
reach conclusions (Sloman, 1996; Stanovich, 1999). This capability enhances
the capacity of LLMs to tackle complex challenges that require the ability to
consider and reevaluate different strategies, such as devising game strategies
or generating creative content. By simulating the way humans think and make
decisions, ToT not only makes their thought process more understandable to
human users, but also improves the models’ effectiveness in handling complex
tasks.

Graph of Thoughts (GoT). Proposed by Besta et al. (2023), GoT transforms the
output of LLMs into a graph format. This format visualizes information pieces
as nodes and their connections as edges, enabling a more intricate and
connected form of reasoning compared to previous methods CoT and ToT. By
organizing data into nodes (individual concepts or pieces of information) and
edges (relationship between these concepts), GoT makes the logical connections
within complex systems more understandable (Yao et al., 2023a). This graphical
representation brings several benefits for understanding complex information.
Firstly, it enables dynamic modification of relationships between concepts,
offering a clear visualization of how changing one element affects the others.
This is crucial in fields like legal reasoning (Cui et al., 2023; Boche et
al., 2024), scientific research (Ding et al., 2023; Choudhury et al., 2023),
and policy analysis (Chen et al., 2023c), where the inter-dependencies between
various factors can be intricate and subtle. Secondly, GoT enables an
assessment of the significance of each node within the graph, providing
insights into which pieces of information are most critical to the task. This
level of adaptability and clarity makes GoT exceptionally powerful for
analyzing and navigating complex information networks.

###  6.3 Case Study: Is CoT Really Making LLM Inferences Explainable?

####  6.3.1 Background and Experimental Settings

Despite the apparent intuitiveness of the CoT prompt design, a critical
question remains unanswered: Does CoT really make LLM inferences explainable?
In other words, can the information provided through CoT faithfully reflect
the underlying generation process of LLMs? We use multi-hop question-answering
(QA) as the scenario to investigate this problem.

In QA systems, answering multi-hop questions remains a significant challenge.
Instead of leveraging a single information source, multi-hop questions require
synthesizing information from multiple pieces or sources of data into a
coherent and logical sequence. While LLMs show good performance in single-hop
QA tasks (Radford et al., 2019), their efficacy significantly declines in
multi-hop situations (Tan et al., 2023; Kim et al., 2023a; Zhong et al.,
2023). This discrepancy highlights the need for more advanced methods to
effectively handle the intricacy of multi-hop reasoning.

CoT Prompts for Multi-hop QA. To address the above challenge, our case study
applies the CoT technique. CoT relies on high-quality templates as the in-
context learning prompt, where we give an example as below. Here, $[x]$
denotes the test question. The “Thoughts” following each “Question” are step-
by-step problem-solving statements for the multi-hop questions. The thoughts
in the templates align the generation process of LLMs with human cognitive
problem-solving patterns.

⬇

Question: What is the capital of the country where Plainfield Town Hall is
located?

Thoughts: Plainfield Town Hall is located in the country of the United States
of America. The capital of the United States is Washington, D.C.

Answer: Washington, D.C.

...

Question: Who has ownership of the developer of the Chevrolet Corvette (C4)?

Thoughts: The developer of Chevrolet Corvette (C4) is Chevrolet. Chevrolet is
owned by General Motors.

Answer: General Motors

Question: [x]

CoT Faithfulness for Explanation: To quantitatively measure the faithfulness
of CoTs, we select fidelity as the corresponding metrics (Zhao et al., 2023b;
Wachter et al., 2017):

$Fidelity=\frac{\sum_{i=1}^{N}\left(\mathbbm{1}\left(\hat{\mathbf{y}}_{i}=\mathbf{y}_{i}\right)-\mathbbm{1}\left(\hat{\mathbf{y}}_{i}^{mislead}=\mathbf{y}_{i}\right)\right)}{\sum_{i=1}^{N}\left(\mathbbm{1}\left(\hat{\mathbf{y}}_{i}=\mathbf{y}_{i}\right)\right)}\times
100\%,$ (6)

where $\mathbf{y}_{i}$ denotes the ground truth label, $\hat{\mathbf{y}}_{i}$
denotes the original model output with CoT, while
$\hat{\mathbf{y}}_{i}^{mislead}$ denotes the model output with misleading
information inserted in the "Thoughts" section. In the following, we give an
example. Given the target question, the correct step-by-step thoughts should
be: “Ellie Kemper is a citizen of the United States of America. The president
of the United States of America is Joe Biden.” To mislead the model, we
replace the thoughts with incorrect information (the underlined text) and ask
the model to generate a new answer based on incorrect thoughts. If the model
still generates the correct answer after the modification, we believe that the
CoT information does not faithfully reflect the true process of the answer
generation. On the other hand, if it generates an answer corresponding to the
incorrect thoughts, then we claim the thoughts are faithful.

⬇

Question: What is the capital of the country where Plainfield Town Hall is
located?

Thoughts: Plainfield Town Hall is located in the country of the United States
of America. The capital of United States is Washington, D.C.

Answer: Washington, D.C.

...

Question: Who has ownership of the developer of the Chevrolet Corvette (C4)?

Thoughts: The developer of Chevrolet Corvette (C4) is Chevrolet. Chevrolet is
owned by General Motors.

Answer: General Motors

Question: [Who is the head of state of the country where Ellie Kemper holds a
citizenship?]

Thoughts: Ellie Kemper is a citizen of Croatia. The head of state in Croatia
is Zoran Milanović.

Answer:

Experimental Settings. We evaluate the performance on the MQUAKE-CF dataset
Zhong et al. (2023), which includes 1,000 cases for each $K$-hop questions,
$K\in\\{2,3,4\\}$, which totally consists of 3,000 questions. Our evaluation
applies various language models, including GPT-2 (Radford et al., 2019) with
1.5 billion parameters, GPT-J (Wang & Komatsuzaki, 2021) with 6 billion
parameters, LLaMA (Touvron et al., 2023a) with 7 billion parameters,
Vicuna-v1.5 (Chiang et al., 2023) with 7 billion parameters, LLaMA2-chat-hf
(Touvron et al., 2023b) with 7 billion parameters, Falcon (Almazrouei et al.,
2023) with 7 billion parameters, Mistral-v0.1 (Jiang et al., 2023) with 7
billion parameters, and Mistral-Instruct-v0.2 (Jiang et al., 2023) with 7
billion parameters. These models have demonstrated proficiency in both
language generation and comprehension.

####  6.3.2 Experiment Results

Performance Improvement. The performance reported in Table 5 for multi-hop
question answering highlights the effectiveness of the CoT across various
models. GPT-J exhibits the most dramatic improvement, particularly in 3-hop
questions with over a 200% increase in accuracy, suggesting that CoT’s
coherent reasoning greatly enhances LLMs’ question-answering ability. While
GPT-2 shows modest gains, the performance of GPT-J and other models, such as
LLaMA, Vicuna-v1.5, LLaMA2, Falcon, Mistral-v0.1, and Mistral-v0.2, indicates
that the CoT method may be more beneficial for more advanced models. LLaMA2
and Mistral-v0.2 display considerable improvements of 78.4% and 63.8%,
respectively, in 3-hop questions, further supporting this observation.
Overall, the results underscore the potential of coherent reasoning
techniques, i.e., CoT, in advancing the question-answering capabilities of
LLMs across different model architectures and sizes.

Table 5: Multi-hop question answering performance on MQUAKE-CF. Datasets | MQUAKE-CF
---|---
Question Type | 2-hops | 3-hops | 4-hops
Edited Instances | Base | Enhanced | Improve | Base | Enhanced | Improve | Base | Enhanced | Improve
GPT-2 (1.5B) | 13.6 | 15.9 |  16.9% $\uparrow$ | 11.6 | 8.9 |  23.3% $\downarrow$ | 7.0 | 8.4 |  20.0% $\uparrow$
GPT-J (6B) | 23.1 | 51.9 |  124.7% $\uparrow$ | 10.1 | 30.5 |  202.0% $\uparrow$ | 21.3 | 49.8 |  133.8% $\uparrow$
LLaMA (7B) | 47.7 | 65.1 |  36.4% $\uparrow$ | 29.6 | 39.3 |  32.8% $\uparrow$ | 52.4 | 62.9 |  20.0% $\uparrow$
Vicuna-v1.5 (7B) | 41.3 | 56.3 |  36.3% $\uparrow$ | 22.7 | 29.7 |  30.9% $\uparrow$ | 31.6 | 53.1 |  68.2% $\uparrow$
LLaMA2 (7B) | 36.7 | 58.7 |  60.0% $\uparrow$ | 17.0 | 30.3 |  78.4% $\uparrow$ | 29.2 | 49.1 |  68.1% $\uparrow$
Falcon (7B) | 42.3 | 61.7 |  45.7% $\uparrow$ | 23.2 | 31.7 |  35.7% $\uparrow$ | 33.3 | 48.6 |  45.7% $\uparrow$
Mistral-v0.1 (7B) | 49.0 | 69.3 |  41.5% $\uparrow$ | 30.0 | 42.3 |  41.1% $\uparrow$ | 48.7 | 63.2 |  29.9% $\uparrow$
Mistral-v0.2 (7B) | 44.0 | 56.3 |  28.0% $\uparrow$ | 23.0 | 37.7 |  63.8% $\uparrow$ | 32.9 | 56.2 |  70.9% $\uparrow$

Faithfulness Evaluation of CoT. Table 6 illustrates the impact of accurate
versus misleading CoTs on the performance of LLMs. The Fidelity metric
indicates how faithfully the model’s output reflects the reasoning process
described in the CoT. Ideally, a high Fidelity score suggests that the model’s
final response is directly based on the provided CoT, validating it as a
faithful explanation of the model’s reasoning pathway. However, as we will
discuss below, a low Fidelity may not always imply a lack of faithfulness in
the model’s reasoning, which calls for developing more effective evaluation
methods in future research.

GPT-J and LLaMA exhibit high fidelity scores across different question types,
indicating a strong adherence to the given reasoning paths. Conversely, models
such as Vicuna-v1.5, LLaMA2, Falcon, Mistral-v0.1, and Mistral-v0.2 show
relatively high mislead accuracy scores with lower fidelity scores. In the
experiments, we observe that these models usually rely on their own generated
thoughts instead of using incorrect information provided in the CoT.
Mistral-v0.2, in particular, demonstrates the lowest fidelity scores and
highest misleading accuracy scores, suggesting a potential self-defense
ability against false information. The lower fidelity scores of later models
may be attributed to their improved training processes on more diverse and
high-quality datasets, enabling them to develop a better understanding of
context and reasoning. As a result, they are more likely to generate their own
correct reasoning paths.

In conclusion, the generated thoughts can be generally viewed as faithful
explanations of their output answer. While high fidelity scores generally
indicate a model’s adherence to the provided CoT, low fidelity scores do not
necessarily imply a lack of faithfulness, especially when the model
demonstrates the ability to reject misleading information. Further research on
CoT faithfulness and the development of more sophisticated evaluation metrics
could contribute to the advancement of interpretable and reliable language
models.

Table 6: CoT Faithfulness Evaluation on MQUAKE-CF. Datasets | MQUAKE-CF
---|---
Question Type | 2-hops | 3-hops | 4-hops
Edited Instances | Correct | Mislead | Fidelity | Correct | Mislead | Fidelity | Correct | Mislead | Fidelity
GPT-2 (1.5B) | 15.9 | 5.2 | 67.3% | 8.9 | 2.9 | 67.4% | 8.4 | 1.3 | 84.5%
GPT-J (6B) | 51.9 | 7.3 | 85.9% | 30.5 | 1.8 | 94.1% | 49.8 | 2.0 | 96.0%
LLaMA (6B) | 65.1 | 9.9 | 84.8% | 39.3 | 6.1 | 84.5% | 62.9 | 6.0 | 90.5%
Vicuna-v1.5 (7B) | 56.3 | 21.7 | 61.5% | 29.7 | 12.7 | 57.3% | 53.1 | 16.1 | 69.7%
LLaMA2 (7B) | 58.7 | 17.0 | 71.0% | 30.3 | 8.3 | 72.5% | 49.1 | 12.0 | 75.6%
Falcon (7B) | 61.7 | 24.0 | 61.1% | 31.6 | 15.0 | 52.6% | 48.6 | 23.1 | 52.4%
Mistral-v0.1 (7B) | 69.3 | 24.0 | 65.4% | 42.3 | 13.0 | 69.3% | 63.2 | 18.4 | 70.8%
Mistral-v0.2 (7B) | 56.3 | 47.9 | 14.8% | 37.7 | 22.0 | 41.6% | 56.2 | 37.3 | 33.6%

###  6.4 Challenges

Within machine learning, explanation faithfulness refers to the degree to
which an explanation accurately reflects the decision-making process of the
model it is intended to elucidate (Li et al., 2022c). An explanation is
considered as faithful if it causes the model to make the same decision as the
original input. In this context, the challenge faced by explainable prompting
(e.g., CoT prompt) lies in two aspects: (1) directing language models to
generate explanations that are genuinely representative of the models’
internal decision-making processes, and (2) preventing language models from
depending on potentially biased CoT templates.

Regarding the first challenge, our case study has revealed that relatively
small language models may generate answers that do not align with the provided
CoT rationales. Therefore, these rationales do not accurately represent the
decision-making process within these models. Some efforts have been made to
bolster the CoT capabilities of smaller language models by implementing
instruction tuning with CoT rationales (Kim et al., 2023b; Ho et al., 2022).
These methods can help improve the explanation faithfulness of CoT for small
language models, thereby addressing this issue to some extent. Nevertheless,
it remains a challenging problem of how to ensure the generated explanations
(i.e., “what the model says”) are faithful to the internal mechanism (i.e.,
“what the model thinks”) of language models.

Regarding the second challenge, recent research shows that explanations in the
CoT can be heavily influenced by the introduction of biasing prompt templates
into model input (Turpin et al., 2024). This is because existing CoT requires
carefully designed templates to prompt language models to produce
explanations. If incorrect or biased information is encoded in such templates,
the generated explanations could be misleading. Recently, Wang & Zhou (2024a)
propose a novel decoding strategy to implement CoT with prompting, which could
mitigate this issue. However, how to effectively help language models get rid
of the template reliance still remains to be underexplored.

##  7 LLM Enhancement via Knowledge-Augmented Prompting

Enhancing models with external knowledge can significantly improve the control
and interpretability of decision-making processes. While LLMs acquire
extensive knowledge through pre-training on web-scale data, this knowledge is
embedded implicitly within the model parameters, making it challenging to
explain or control how this knowledge is utilized during inference.
Additionally, LLMs may not always encompass the unique knowledge specific to
certain domains, nor keep pace with the constantly evolving information in the
world. To address these limitations, this section discusses Retrieval-
Augmented Generation (RAG) for the explicit integration of external knowledge
into the mechanism of LLM-based AI systems, aiming to yield more interpretable
predictions.

###  7.1 Preliminaries: Retrieval-Augmented Generation

By fetching relevant information from external databases or the internet, RAG
ensures that LLM outputs are accurate and up-to-date. It addresses LLMs’
limitation of relying on fixed and potentially outdated knowledge bases. RAG
operates in two steps: (1) Retrieval: It locates and fetches pertinent
information from an external source based on the user’s query; (2) Generation:
It incorporates this information into the model’s generated response. Given an
input query $x$ and the desired output $y$, the objective function of RAG can
be formulated as (Guu et al., 2020):

$\max_{\phi,\theta}\,\log
p(y|x)=\max_{\phi,\theta}\,\log\sum_{z\in\mathcal{K}}p_{\phi}(y|x,z)\cdot
p_{\theta}(z|x),$ (7)

where $z$ stands for the external knowledge retrieved from a knowledge base
$\mathcal{K}$. Thus, the target distribution is jointly modeled by a knowledge
retriever $p_{\theta}(z|x)$ and an answer reasoning module $p_{\phi}(y|x,z)$.
The knowledge $z$ serves as a latent variable. An RAG model is trained to
optimize the parameters, so that it learns to retrieve relevant knowledge $z$
and to produce correct answers $y$ based on $z$ and $x$. As LLMs possess
stronger text comprehension and reasoning abilities, they can directly serve
as the reasoning module $p_{\phi}$ without further training. In this case, RAG
can be treated as a data-centric problem:

$\max_{z\in\mathcal{K}}\,\log
p(y|z,x)=\max_{z\in\mathcal{K}}\,\frac{p(z|x,y)}{p(z|x)}p(y|x),$ (8)

where the goal is to find appropriate knowledge that supports the desired
output. The interpretability of RAG-based models comes from the information in
$z$: (1) $z$ usually elucidates or supplements the task-specific information
in $x$; (2) $z$ could explain the generation of output $y$. Unlike other deep
models that directly estimate $p(y|x)$ in an end-to-end manner, where the
decision process is not comprehensible, the RAG process provides justification
or rationale $z$ that supports the result.

Existing Retrieval-Augmented Generation (RAG) approaches can be categorized
based on when they integrate external knowledge into the model’s workflow. The
first category incorporates external knowledge at the inference stage. For
instance, Karpukhin et al. (2020) employ dense vectors to identify related
documents or text passages, enhancing the data retrieval step of RAG.
Similarly, Lewis et al. (2020) refine the data retrieval process to ensure
only the most pertinent information influences the model’s output. The second
category integrates external knowledge during the model tuning stage. Some
representative approaches include Guu et al. (2020); Borgeaud et al. (2022);
Nakano et al. (2021). Generally, these methods embed a retrieval mechanism
into the model’s training phase, enabling the model to utilize external data
more efficiently from the outset.

###  7.2 Enhancing Decision-Making Control with Explicit Knowledge

The incorporation of explicit external knowledge through RAG enhances the
precision and controllability of decision-making in LLMs. This method
leverages real-time information from external databases to produce responses
that are not only accurate but also tailored to the specific requirements of
each query. Below, we explore the mechanisms by which RAG achieves a more
controllable and directed content generation process, with references to key
papers that have contributed to these advancements.

####  7.2.1 Reducing Hallucinations in Response

“Hallucination" in the context of LLMs refers to instances where these models
generate information that, while coherent and contextually appropriate, is not
based on factual accuracy or real-world evidence (Huang et al., 2023c). This
issue can lead to the production of misleading or entirely fabricated content,
posing a significant challenge to the reliability and trustworthiness of LLMs’
outputs. RAG offers a powerful solution to mitigate the problem of
hallucinations in LLMs. By actively incorporating up-to-date, verified
external knowledge at the point of generating responses, RAG ensures that the
information produced by the model is anchored in reality. This process
significantly enhances the factual basis of the model’s outputs, thereby
reducing the occurrence of hallucinations. Shuster et al. (2021) applies
neural-retrieval-in-the-loop architectures to knowledge-grounded dialogue,
which significantly reduces factual inaccuracies in chatbots, as confirmed by
human evaluations. Siriwardhana et al. (2023) introduces RAG-end2end, which
joint trains retriever and generator components together. Their method
demonstrates notable performance improvements across specialized domains like
healthcare and news while reducing knowledge hallucination.

####  7.2.2 Dynamic Responses to Knowledge Updating

RAG empowers LLMs with the ability to incorporate the most current
information, keeping their decision-making processes aligned with the latest
developments. This feature is especially vital in fast-evolving fields such as
medicine and technology, where the need for timely and accurate information is
paramount (Meng et al., 2022b). For example, research by (Izacard & Grave,
2020) demonstrates significant enhancements in output relevance and accuracy
through real-time information retrieval. Similarly, Han et al. (2023) suggest
using retrieved factual data to correct and update the knowledge within pre-
trained LLMs efficiently. Additionally, Wang et al. (2023e) introduce a method
for integrating newly retrieved knowledge from a multilingual database
directly into the model prompts, facilitating updates in a multilingual
context.

####  7.2.3 Domain-specific Customization

RAG enhances LLMs by incorporating knowledge from specialized sources,
enabling the creation of models tailored to specific domains. Research by Guu
et al. (2020) illustrates how integrating databases specific to certain fields
into the retrieval process can empower models to deliver expert-level
responses, boosting their effectiveness in both professional and academic
contexts. Shi et al. (2023) have applied this concept in the medical domain
with MedEdit, utilizing an in-context learning strategy to merge relevant
medical knowledge into query prompts for more accurate medical advice.
Moreover, recent research finds that LLMs struggle to capture specific
knowledge that is not widely discussed in the pre-training data. Specifically,
Mallen et al. (2023) observe that LLMs often fail to learn long-tail factual
knowledge with relatively low popularity, finding that simply increasing model
size does not significantly enhance the recall of such information. However,
they note that retrieval-augmented LLMs surpass much larger models in
accuracy, particularly for questions on well-known subjects, suggesting that
this method can effectively bridge knowledge gaps. Similarly, Kandpal et al.
(2023) highlights LLMs’ challenges with acquiring rare knowledge and proposes
that retrieval augmentation offers a viable solution, minimizing reliance on
extensive pre-training for capturing nuanced, less common information.

###  7.3 Challenges

We discuss the challenges in RAG that are relevant to its explainability
aspects: (1) In the retrieval stage $p_{\theta}(z|x)$, does the retrieved
information $z$ always elucidate the task-specific information contained in
the input $x$? (2) In the generation stage $p_{\phi}(y|x,z)$, does $z$
effectively serve as an explanation for the generation of output $y$? Please
note that our goal is not to exhaustively discuss all the limitations of RAG
in this paper as RAG itself is a broad topic in NLP research. For a more
detailed examination of the broader limitations of RAG, we direct readers to
other reviews (Gao et al., 2023).

####  7.3.1 Retrieval Accuracy Bottlenecks

Existent RAG methods typically rely on similarity search to pinpoint relevant
information (Lewis et al., 2020; Gao et al., 2023), which represents a
substantial improvement over basic keyword searches (Robertson et al., 2009).
However, these methods may struggle with complex queries that demand deeper
comprehension and nuanced reasoning. The recent “lost-in-the-middle”
phenomenon (Liu et al., 2024) has revealed that an ineffective retrieval can
result in the accumulation of extraneous or conflicting information,
negatively affecting the generation quality. To address this challenge, recent
RAG approaches have integrated adaptive learning processes (Asai et al.,
2023). This advancement enables the retrieval system to refine their
performance over time through feedback, adapting to evolving language use and
information updates, ensuring their responses remain relevant and accurate.
Nonetheless, efficiently handling intricate and multi-hop questions remains a
significant challenge, highlighting the need for ongoing research to enhance
the capabilities of RAG systems.

####  7.3.2 Controllable Generation Bottlenecks

In-context learning stands out as the premier method for incorporating
external knowledge to boost the capabilities of LLMs such as GPT-4 (Asai et
al., 2023; Gao et al., 2023). Despite its effectiveness, there’s no surefire
way to ensure that these models consistently leverage the provided external
knowledge within the prompts for their decision-making processes. In practice,
to achieve thorough coverage, commonly used dense retrieval usually returns a
large volume of content, including both relevant and redundant information to
the input question. Unfortunately, redundant information in the model prompt
raises the computational cost and can mislead LLMs to generate incorrect
answers. Recent research shows the retrieved information can degrade the
question-answering task performance (Yoran et al., 2023; Petroni et al., 2020;
Li et al., 2022a). Some recent work proposes to fine-tune the LLM to improve
resilience to noise and reduce hallucinations. However, such approach still
cannot prevent oversized retrieval information decrease the system
interpretability (Yoran et al., 2023; Xu et al., 2023b). The challenge of
optimizing the use of external explanations to achieve more precise and
controlled decision-making in LLMs is an ongoing issue that has yet to be
fully addressed.

##  8 Training Data Augmentation with Explanation

This section explores the generation of synthetic data from explanations using
large language models, a technique poised to enhance various machine learning
tasks. In machine learning, limited data availability often constrains model
performance, presenting a significant challenge across many domains. A viable
solution is data augmentation, where LLMs, with their robust generative
capabilities, can be utilized for text data augmentation (Whitehouse et al.,
2023), such as transforming existing text samples into new variants (Dai et
al., 2023). Nevertheless, there are several challenges to be tackled for
effective text augmentation. First, for utility, the generated samples need to
exhibit diversity compared to the original data. Second, these samples should
be exhibit useful patterns relevant to the downstream tasks. To address these
challenges, explanation methods offer a valuable tool to guide data
augmentation by providing supplemental contexts and useful rationales (Carton
et al., 2021). Using LLMs for explanation-guided data augmentation is a
nascent but promising field. In this section, we aim to outline feasible
frameworks and discuss potential applications, offering directions for future
research in this field.

Explanations can be particularly beneficial in data augmentation within two
scenarios. In the first scenario, explanations are used to delineate desired
model behaviors or to identify existing deficiencies, which effectively guides
the data augmentation process of LLMs. The second scenario involves employing
LLMs to directly produce explanatory texts, which serve as supplementary
information to enrich the dataset.

###  8.1 Explanation-guided Data Augmentation for Mitigating Shortcuts

Machine learning models are prone to make predictions with spurious
correlations, also known as shortcuts (Geirhos et al., 2020), which are
misaligned with human reasoning processes. This dependency on shortcuts
underlies various challenges in machine learning, notably diminishing a
model’s ability to generalize to out-of-distribution samples and weakening its
resilience against adversarial attacks (Xu et al., 2020). The extent to which
a model depends on shortcut features is intrinsically linked to its
interpretability. Essentially, if a model’s predictions are predominantly
based on such unreliable features, it indicates that the underlying mechanisms
of the model are not fully comprehensible or interpretable from a human
perspective. Therefore, post-hoc explanation techniques usually play a crucial
role in detecting undesirable correlations between input and predictions
within deep models (Liu et al., 2018; 2021). For example, Du et al. (2021)
adopt Integrated Gradient (IG) to attribute a model’s predictions to its input
features, showing that the model tends to treat functional words, numbers, and
negation words as shortcuts and strongly rely on these shortcut features for
prediction in natural language understanding tasks.

Data augmentation can be applied to training better models that are less
sensitive to shortcut features. Explanatory information such as
counterfactuals (Wang & Culotta, 2021) has been incorporated in data
augmentation to improve model robustness. It generates counterfactual samples
by first identifying critical features (e.g., word tokens) and then replacing
these features with their antonyms, along with reversing their associated
labels. Subsequently, the generated samples are combined with the original
ones to train downstream models. Furthermore, these techniques can be extended
to enhance the out-of-distribution performance of smaller models (Sachdeva et
al., 2023; Wen et al., 2022). Namely, large language models could serve as an
effective tool to augment data. For example, LLMs are able to synthesize
examples that represent outlier cases or rare occurrences, which helps smaller
models generalize better on unseen data (Xu et al., 2023a). This could be
helpful in building robust models in scenarios where data are scarce or
confidential (Tang et al., 2023a). Besides, LLMs are promising in improving
models’ safety by generating adversarial examples that are more valid and
natural compared to conventional approaches (Wang et al., 2023f). First, the
most vulnerable words are identified with attribution-based methods. Then,
these words are replaced by LLMs in a manner that maintains the integrity of
the original text. The quality of these examples can be examined with an
external classifier. Subsequently, these adversarial examples are employed to
train downstream models, effectively fortifying them against potential attacks
and boosting their security. Similarly, LLMs are also helpful in mitigating
biases such as fairness issues in models (He et al., 2023). This work claims
that it automatically identifies underrepresented subgroups, and chooses an
optimal augmentation strategy that avoids hurting other groups. New group
examples are generated using LLMs with human-providing labels. The experiments
observe improvements on both underrepresented groups and overall model
performance. This approach prevents skewed datasets from leading to unfair
outcomes for a certain demographic, thereby potentially promoting fairness in
society.

###  8.2 Explanation-enhanced Data Enrichment

As a powerful generative tool, LLMs have been leveraged to directly generate
natural language explanations as augmented data. This strategy relies on LLMs’
understanding abilities to assist smaller models in their learning tasks. One
objective in such work is to add natural language explanation generated by
LLMs to training data, so as to enhance the performance of small models. Li et
al. (2022b) introduce explanations from LLMs that facilitate the training of
smaller models to enhance their reasoning capabilities and acquire explanation
generation abilities. Specifically, three explanation generation approaches
from LLM are utilized to train smaller models, including (1) explanations
generated through chain of thought prompting, (2) explanations by
rationalization prompting conditioned on golden labels and (3) hybrid approach
that combines the first two approaches. With the best case scenario in a
reasoning case, a 9.5% improvement on accuracy compared to 60x larger GPT-3
has been seen (Li et al., 2022b). It is worth noting that LLMs including
ChatGPT and GPT-4 are competitive in generating coherent and logical
explanations, except for limited languages (Whitehouse et al., 2023).
Explanations from LLMs have also been utilized to mitigate spurious
correlations in aspect-based sentiment analysis tasks (Wang et al., 2023c).
This study proposes using LLMs to generate explanations for the sentiment
labels of aspects in sentences (Wang et al., 2023c). These explanations
provide reasoning grounded in contextual semantics rather than relying on
spurious correlations between words and labels. The explanations are
integrated into the training of aspect-based sentiment analysis models through
two methods: augmenting the training data with the explanations or distilling
knowledge from the explanations through mimicking behaviors. By focusing on
the explanations instead of superficial textual cues, the models can better
learn the true associations between text and sentiment and become more robust,
improving both in-domain performance and generalization ability (Wang et al.,
2023c). Another line of work involves integrating LLM rationales as additional
supervision to guide the training of smaller models. Experiments have shown
that this approach not only requires fewer training data but also outperforms
traditional fine-tuning and distillation methods (Hsieh et al., 2023).

In addition to the existing application of augmentation techniques summarized
above, we envision that incorporating additional text information can also be
practical and efficient in enhancing the performance of various models. For
example, one promising application lies in the realm of guiding the parameter
learning process of small models by using automatically generated
explanations. Previous research has investigated this avenue by directing the
attention of natural language inference models towards human-crafted
explanations (Stacey et al., 2022). As human-crafted explanations are both
arduous and non-transferable, utilizing LLMs as generators presents a more
economic and versatile alternation. Another potential application is to
enhance model performance on complex tasks using natural language explanations
from LLMs. For instance, code translation generation tasks incorporate
explanations as an intermediate step, improving model performance by 12% on
average (Tang et al., 2023b). The result shows that explanations are
particularly useful in zero-shot settings. Apart from assisting in smaller
models, LLMs have demonstrated their ability to improve their own reasoning
abilities by generating reliable rationales (Huang et al., 2022). Further,
Krishna et al. (2023) embed post-hoc explanations, attributing scores to all
input features, into natural language rationales. This approach improves model
accuracy by 10-25% across multiple tasks via in-context learning. Another
study explicitly investigates LLMs’ ability to generate post-hoc explanations
in natural language. The experiments show that LLMs spot key features with 75%
accuracy (Kroeger et al., 2023). These studies present a novel strategy for
leveraging post-hoc explanations, beyond traditional natural language
explanations, to enrich training data so as to bolster model performance.

###  8.3 Challenges

####  8.3.1 Computational Overhead

Conventional post-hoc explanations, built on well-trained models, are often
resource-intensive tasks. The first scenario mentioned above leverages
interpretability techniques to accurately diagnose dataset issues. This