# Explainable Generative AI (GenXAI): A Survey, Conceptualization, and
Research Agenda

Johannes Schneider<EMAIL_ADDRESS>University of Liechtenstein,
Vaduz, Liechtenstein

###### Abstract

Generative AI (GenAI) marked a shift from AI being able to “recognize” to AI
being able to “generate” solutions for a wide variety of tasks. As the
generated solutions and applications become increasingly more complex and
multi-faceted, novel needs, objectives, and possibilities have emerged for
explainability (XAI). In this work, we elaborate on why XAI has gained
importance with the rise of GenAI and its challenges for explainability
research. We also unveil novel and emerging desiderata that explanations
should fulfill, covering, for instance, verifiability, interactivity,
security, and cost aspects. To this end, we focus on surveying existing works.
Furthermore, we provide a taxonomy of relevant dimensions that allows us to
better characterize existing XAI mechanisms and methods for GenAI. We discuss
different avenues to ensure XAI, from training data to prompting. Our paper
offers a short but concise technical background of GenAI for non-technical
readers, focusing on text and images to better understand novel or adapted XAI
techniques for GenAI. However, due to the vast array of works on GenAI, we
decided to forego detailed aspects of XAI related to evaluation and usage of
explanations. As such, the manuscript interests both technically oriented
people and other disciplines, such as social scientists and information
systems researchers. Our research roadmap provides more than ten directions
for future investigation.

###### keywords:

Generative Artificial Intelligence, Explainability, Conceptualization, Survey,
Explainable Artificial Intelligence, Research agenda.

##  1 Introduction

Generative AI (GenAI) has shown remarkable capabilities that have shaken up
the world on a broad basis – ranging from regulators [36], educators [8],
programmers [161] onto medical staff [173]. For businesses [120], GenAI has
the potential to unlock trillions of dollars annually [96]. At the same time,
it is said to threaten mankind [170]. These opposing views are a key drive for
understanding and explaining GenAI. Generative AI constitutes the next level
of AI driven by foundation models [146], where AI can create text, images,
audio, 3D solutions, and videos [47, 14] controllable by humans, specifically
via textual prompts [186] – see also Table 1 for examples of public GenAI
systems. This is a major step forward from AI that was primarily able to
“recognize” to AI that “generates”. GenAI has shown unprecedented capabilities
like passing university-level exams [22, 65]. It also achieves remarkable
results even in areas deemed non-suitable for machines, such as creativity
[17]. It is accessible to everyone, as witnessed by commercial systems like
ChatGPT [1] and Dall-E [11, 124]. Early generative AI methods, such as
Generative Adversarial networks (GANs), can also generate artifacts but are
typically more difficult to control than modern models such as transformers
and diffusion architectures.

Explainable AI for GenAI (GenXAI) techniques produce explanations that help
comprehend AI, for example, outputs for individual inputs or the model as a
whole. Traditionally, explanations have served many purposes due to multiple
needs; for instance, they can increase trust and support the debugging of
models [100]. The need for understanding AI is even larger than in pre-GenAI
eras. For example, explanations can support the verifiability of generated
content and, thereby, contribute to combatting one of the major problems of
GenAI: hallucinations (as argued in Section 3.1). Unfortunately, Explainable
AI (even for pre-GenAI models) is characterized by several open problems
despite many attempts to design solutions to address them over the last few
years [84, 100]. For example, a recent comparison [155] among methods on the
impact of XAI on human-agent interaction found that the difference in scores
between the best (counterfactuals) and worst method (using simply probability
scores) was only 20%, hinting that complex existing methods yield limited
benefits over more complex ones. Thus, XAI techniques are still far from being
optimal. Other works have even openly called the “status quo in
interpretability research largely unproductive” [125]. As such, there is much
to be done, and it is essential to understand current efforts to learn and
improve on them – especially to mitigate high risks [170] while leveraging
opportunities [146].

Figure 1: Article outline. Following important factors and challenges of XAI
for GenAI, desiderata and, in turn, a taxonomy is derived. All three together
inform our research agenda.

This research manuscript is a genuine attempt to progress in this direction.
Our goal is not to (only) list and structure existing XAI techniques, as in
the current stage of the field, more basic questions need to be addressed,
such as identifying key challenges and desiderata for GenXAI. To this end, we,
therefore, opted for a more narrative review methodology [69] accompanied by a
taxonomy development approach from the field of information systems
[110].111Details on the methodology are given in the Appendix (Section 7.1).

There are several surveys on XAI focusing on the pre-GenAI era with primary
technical focus [2, 209, 31, 148, 125, 131, 163, 102, 12, 172, 52, 51] and an
interdisciplinary or social science focus [101, 100, 84]. In particular, by
building upon them, we perform a meta-survey to structure our methods
leveraging also knowledge from pre-GenAI. However, we also uncover novel
aspects that have not yet been covered related to GenAI. Many works surveyed
various aspects of GenAI (not including XAI) [191, 80, 190, 193, 200, 203,
117]. We leverage such surveys for our technical background. Some sub-areas of
GenAI, e.g., on knowledge identification and editing [202], use isolated XAI
techniques as a tool, but do not aim at elaborating on it generally. While we
could not identify any review discussing XAI for GenAI, some research
manuscripts take more of a holistic, partially opinionated view on XAI for
large language models (LLMs) [158, 78] or explicitly survey XAI for LLMs [204,
89]. None of the prior work has provided a comprehensive list of desiderata,
motivation, and challenges for XAI for GenAI, and taxonomy. In particular,
many of our novel aspects cannot be found in prior works. Aside from that,
even when focusing on LLMs only, we differ considerably from prior works.

We begin by providing a technical background. To derive the contributions, we
proceed as outlined in Figure 1: Then, we provide motivation and challenges
for XAI for GenAI, especially pointing towards novel aspects that emerge with
GenAI, such as the increased reach of GenAI throughout society and the need
for interactive adjustment of often complex, difficult to evaluate outputs by
users. Based on this, we derive desiderata, i.e., requirements that
explanations should ideally fulfill, such as the support of interactivity and
verification of outputs. Then, we derive a taxonomy for existing and future
XAI techniques for GenAI. To categorize XAI we use dimensions related to the
in-, the outputs, and internal properties of GenXAI techniques that
distinguish them from pre-GenAI such as self-explanation as well as different
sources and drivers for XAI such as prompts and training data.

Using the identified challenges and desiderata, the remainder of this
manuscript focuses on discussing novel dimensions for GenXAI and the resulting
taxonomy, discussing XAI methods in conjunction with GenAI. Finally, we
provide future directions Our key contributions include describing the need
for XAI for GenAI, desiderata for explanations, and a taxonomy for mechanisms
and algorithms including novel dimensions for categorization.

Table 1: Examples of In-/Outputs for GenAI. For more examples, see [47]. Input/Output |  Description |  Example
---|---|---
_Text to Text_ |  Input: Raw text. Output: Processed or generated text. |  ChatGPT 3.5 [1]
_Text to Image/video_ |  Input: Descriptive text or prompt. Output: Generated image/text. |  DALL-E[11], Sora[13]
_Image/video to Text_ |  Input: Image/video and text. Output: Textual interpretation and answer. |  GPT 4 with Dall-E [1], Gemini [127]
_Images, Actions to Actions_ |  Input: Images depicting actions. Output: Generated action sequences. |  Gato [126]
_Text to 3D_ |  Input: Text describing object Output: 3D representation of object. |  Magic3d [79]

##  2 Technical Background

Here, we provide a short technical introduction to generative AI, covering key
ideas on system and model architectures and training procedures. We restrict
ourselves to text and image data to illustrate multi-modality. For video and
audio, please refer to other surveys (for example [149, 201]).

###  2.1 System Architectures

GenAI models can be used as stand-alone applications accessible through a
simple user interface, essentially allowing textual inputs or uploads as for
OpenAI’s ChatGPT[1] and displaying responses. Thus, a system might be
essentially one large model, where a model is almost exclusively based on deep
learning taking an input processed by a neural network yielding an output. For
multi-modal applications, systems that consist of an LLM and other generative
models, such as diffusion models, are typically employed. However, GenAI-
powered systems might involve external data sources and external applications
that interact in complex patterns, as illustrated in Figure 2. An
orchestration application might decide what actions to take based on GenAI
outputs or user inputs. For example, in ChatGPT-4 a user can include a term
like “search the internet” in the prompt, which implies that first an Internet
search is conducted, and retrieved content from the web is then fed into the
GenAI model. The orchestration application is responsible for actually
performing the web search and modifying the prompt to the GenAI model, e.g.,
enhancing it with an instruction like “Answer based on the following content:”
followed by the retrieved information from the web.

Figure 2: Overview of GenAI system architectures comprising a single or
combined model (with a GUI) as well as GenAI systems interacting with other
applications

###  2.2 GenAI Model Architectures

We discuss key aspects of the transformer architecture and diffusion models
and briefly elaborate on other generative models.

Figure 3: Transformer architecture [176]

####  2.2.1 Transformers

Transformers are the de-facto standard for LLMs, while GenAI models involving
images might also encompass other models such as diffusion models, variational
autoencoders (VAEs), and generative adversarial networks (GANs). The
transformer model makes little assumption upon the input data, which makes it
a very flexible model. Assumptions on the data (priors) also help to reduce
the amount of data needed to train a model. Thus, transformers often require
more data than other models to reach the same performance though simpler
models might never reach the same top-level performance. The transformer
architecture [176] (Figure 3) comes with many variations [80] mostly by using
different implementations of individual elements, for example, using different
types of positional embeddings [30] or different types of attention [134] or
even replacing some components, for instance, Hyena [119] provides a drop-in
replacement for attention based on convolutions. Commonly, the objectives of
these adjustments are better performance and faster computation; for example,
the original transformer requires quadratic run-time in the number of inputs,
which makes it prohibitive for very long inputs. The vanilla transformer
architecture (Figure 3) consists of an encoder and a decoder, where the
decoder processes the outputs of the encoder and (shifted) targets. One might
think of a translation scenario, where the encoder takes a sentence expressed
in the source language and the decoder generates one output word after the
other in the desired language. Each generated word also becomes an input to
the decoder when the next word is generated. Decoder-only architectures (such
as the GPT-series [121, 1]) do not have an encoder. In contrast, encoder-only
architectures do not have a decoder and typically produce contextualized
embeddings of single words or text fragments (for example, BERT [29]).

Both the encoder and the decoder take tokens as input. The embeddings are
vectors in a latent space. While raw tokens can only be compared for equality,
vectors in a latent space allow a more nuanced similarity computation and,
potentially, the extraction of specific token attributes such as sentiment.
Thus, many current GenAI systems leverage encoders to obtain vector embeddings
and use them to retrieve relevant information to enhance prompts by searching
in a vector database [73]. A positional encoding is added to the (text)
embedding, so the network has information about where a word occurred. After
that, inputs are processed through multi-head attention. A single-head
attention provides a mechanism to focus (and select) a specific aspect of the
input (for instance, syntax or sentiment), which is then further processed
through a Feed Forward Network (FFN). Attention can be masked for the decoder
so that it cannot access the actual or future targets it wants to predict; for
example, for the common task of next-word prediction, the triangular matrix
constituting the mask prevents the network has access to the next word to
predict as well as words following the next word.

####  2.2.2 Diffusion Models

Diffusion models learn to reconstruct noisy data. They first distort inputs by
repeatedly adding small amounts of noise until the image appears to be noise
following, e.g., a Gaussian distribution one can sample from to generate
images. They reverse the process for sample generation by taking “noise” as
input and reconstructing samples. There are multiple rather mathematical
intricate methods for diffusion models [193]. Here, we discuss key steps of a
prominent technique highly relevant for text-to-image generation, i.e.
Denoising Diffusion Probabilistic Model (DDPM) [57]. In the forward pass
(input distortion) DDPM acts as a Markov chain, meaning that only the current
state (or input) is relevant for the next output, i.e., for a given data
distribution $x_{0}\sim q(x_{0})$, DDPM yields output $x_{T}$ in a sequence of
$T$ sequential steps by computing in step $t$:
$q(x_{t}|x_{t-1})=N(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I)$. Thus, the
overall output becomes $q(x_{T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})$. Here
$N$ is the Gaussian distribution and $\beta_{t}$ is a hyperparameter. The
reverse pass for generation starts from $p_{\theta}(x_{T})$ and yields
$p_{\theta}(x_{0})$, which is supposed to follow the true data distribution
$q_{x_{0}}$. Thus, outputs of diffusion models are not easily controllable,
i.e., additional input must be provided to the reconstruction process to guide
the generation process towards user-desired images as discussed in Section
2.4.

###  2.3 Other Generative Models: VAEs, GANs

We also discuss generative models aside from diffusion models but refer to
[193] that performs a detailed comparison for in-depth elaboration. Generative
Adversarial Networks (GAN) [45] are trained using a generator that constructs
an output from a random vector and a discriminator that aims to distinguish
generated outputs $\hat{x}$ from actual samples of the true data distribution
$x\sim q_{x}$. An autoencoder [77] comprises an encoder and a decoder that
learns to output a given input that is compressed by the encoder into a latent
space before being attempted to be reconstructed by the decoder. A Variational
Autoencoder (VAE) constrains the latent space towards a given prior
distribution through a regularization term as part of the optimization
objective. If the latent space follows a known distribution, sampling, i.e.,
sample generation is facilitated.

###  2.4 Controlling Outputs

Diffusion and generative models like VAEs and GANs yield high-quality outputs
by providing random inputs. Controlling outputs is not easily possible and
requires additional effort. Early techniques on controllable image synthesis
generated outputs conditioned on class labels or by disentangling dimensions
of the latent space so that altering a dimension corresponds to a human
interpretable operation [19]. However, these approaches provide only very
limited ways to customize outputs. Text-to-image models are more versatile.
They commonly encode text and use it as input for generation. Text encoding
might be done using a frozen encoder [133] or by altering the text encoder as
part of the training process (GLIDE [109]). Text-to-image models require
(image, text) pairs for training, in turn, the textual encoder can stem from
the text of the (text-image) pairs only or from a broader corpus and larger
models. Text-to-image models can either start generation from a low-
dimensional space such as Dall-E [124, 11] or aim to construct directly from
pixel space such as GLIDE [109]. For illustration, we briefly discuss Dall-E
[124, 11]. It uses a multimodal contrastive model where image and text
embeddings are matched. It also contains a text-to-image generator before
closing the gap between CLIP text and image latent space, which can be learned
using a diffusion prior.

###  2.5 LLM Training

LLM training consists of at least one and up to three phases. The phases
differ in the training methodology, goal, and amount of data.

Self-supervised pre-training: The raw model is trained on very large amounts
of data using self-supervised pre-training, for instance, next word prediction
(GPT-2) or tasks such as predicting masked words and sentence order (BERT).
The goal is to learn a flexible and broad representation of all forms of text
that can set the foundation for many different tasks. The data can be a
composition of many different kinds of text data as described for LLama-2
[174], e.g., covering code, Wikipedia articles, or a dump of parts of the
internet [24].

Instruction tuning adapts a pre-trained LLM through supervised fine-tuning
[86, 203]. The term “instruction” is often used interchangeably with prompt.
Instructions tend to be more explicit and directive with more precise
guidance. The goal of instruction tuning is to improve performance on the most
common use cases for LLMs: to follow (short) instructions. Though multiple
variations exist, the training data can consist of task instructions, input
(task instance), and desired outputs. Instruction tuning helps performance on
both seen and unseen tasks and across architectures of different scales [85].
It can go beyond mere adaption of an LLM and also provide a means to encode
additional knowledge for domain specialization, such as medical knowledge
[159].

Alignment Tuning: LLMs might not incorporate human values and preferences,
i.e., they can produce harmful, misleading, and biased outputs [153, 185].
Alignment criteria can be diverse, covering helpfulness, honesty, and
harmlessness [116]. Training data typically stems from humans, for instance,
by ranking LLM-generated answers or producing their own answers. Occasionally,
also (more powerful) and already aligned LLMs produce training data[167].
Training can take place by fine-tuning the LLM using supervised learning and
the alignment dataset. Training can also be indirect in the sense that human
feedback is used to learn a reward model serving as a human critic, i.e.,
predicting the human scoring of an output. The resulting reward model can be
used to adjust the LLM using reinforcement learning. The LLM generates outputs
and receives feedback from the reward model.

##  3 The importance, challenges and desiderata of GenXAI

This section motivates the need to use eXplainable Artificial Intelligence for
GenAI and its challenges, followed by describing important desiderata that
explanations should fulfill.

###  3.1 Importance of XAI for GenAI

Need to adjust outputs -  A larger need for explainability for GenAI emerges
as humans effectively control and tailor the generated outputs. Generative AI
blurs the boundary between users and developers. It can be instructed to solve
tasks based on auxiliary knowledge [73] as witnessed by OpenAI’s GPT-4 store
[114], which allows ordinary users without programming skills to build and
offer applications leveraging uploaded knowledge and GPT-4. A new skill of
prompt engineering has emerged that users need to master [199]. XAI has also
been identified as a key requirement to support prompt engineering by experts
[103]. Users need to understand better how to control outputs, handle
limitations, and mitigate risks [185]. Thus, the stakeholders must understand
GenAI to create solutions aligned with their preferences.

Need of output verification -  GenAI, in particular LLMs, are known for many
shortcomings, ranging from generating harmful, toxic responses to misleading,
incorrect content [185]. LLMs can be said to be “fluent but non-factual,”
which makes their verification even harder. Their outputs are generally non-
trustworthy and need some form of verification or validation (as also
acknowledged by regulators [36]). An explanation provides a mechanism to
identify errors such as hallucinations in outputs.

Increased reach -  GenAI is easy to access and use through an ordinary browser
with a web interface facilitating widespread adoption. ChatGPT was also the
fastest product to reach 100 million users and still rapidly grows [120]. It
is used throughout society, including vulnerable groups such as
schoolchildren, the elderly with limited IT knowledge, and corporate
employees.

High impact applications -  GenAI is a general-purpose technology with
applications that might have severe immediate and long-term impacts. Users
might seek advice for pressing personal problems [152] or turn to GenAI for
educational purposes. In an educational context, using ChatGPT once and
receiving a slightly biased response (towards gender, race, or minority) might
have limited impact, but receiving such responses over a prolonged time might
profoundly impact future generations. Aside from such a long-term view, users
might turn to GenAI like ChatGPT while being in psychological distress and
seeking immediate advice. ChatGPT is even known to outperform humans on
emotional awareness [33]. Given such high-stake applications, understanding
GenAI becomes crucial.

Unknown Applications -  As generative AI can process any text, image, and
other medium as input and output, it is hard to anticipate all possible
applications. As such, the need to understand the models more holistically to
ensure they align with higher-level principles becomes relevant.

Difficult to evaluate automatically -  Simple strategies, which means counting
the number of correct and incorrect answers (accuracy), provide only a very
limited picture of GenAI’s behavior, as many tasks yield responses that are
difficult to score. For example, for summarization, human ratings of gold
standards are worse than those of benchmarks, hinting at the challenge in
designing benchmarks [162]. Thus, a thorough systematic quantitative
evaluation using classical input and output test datasets is difficult,
resulting in an increased demand for better model understanding to anticipate
potential shortcomings, as automated tests are difficult and possibly
insufficient.

Security and safety concerns -  Various problems exist related to the safety
and security of GenAI-based technologies, such as witnessed by adversarial
examples [45]. However, GenAI allows for novel forms of attacks and abuse [53]
targeting a vast number of people through social engineering at a large scale
to manipulate elections. Also, individuals might leverage GenAI for malicious
activities, such as receiving detailed instructions on performing terrorist
attacks.

Accountability and legal concerns -  The need for accountability arises due to
multiple questions, often driven by legal concerns. GenAI exhibits a complex
supply chain of data providers and developers, which means data might stem
from many sources, and multiple companies might be involved in building a
GenAI system. These include those that make a foundation model [146], those
that fine-tune it to a task, and ultimately, a user who adjusts it through
prompting. This makes accountability a challenging task that is needed as
GenAI systems have the potential to cause harm. Thus, the question of who is
responsible and what causes harm becomes more relevant. This can lead to
forensic questions like “Why did an AI trigger a certain action [138]? Was it
the training, data, or model?” And even if it can be tied to one aspect, for
example, data, further questions come up such as “Was it the data from a
third-party provider, public data, or data from users?”. The need for
accountability arises due to the use of copyrighted material or, potentially,
patented material. In lawsuits, a key question is if a patent is valid because
it is highly original. If GenAI can explain that the solution is not “a copy
from an existing patent” but emerges through basic reasoning given “prior
art,” the judge might be more inclined to rule the patent invalid.

Table 2: Why is explainability important for GenAI? Reason |  Description
---|---
_Need to adjust_ |  Users want to create artifacts aligned with their preferences
_Need to verify_ |  Ensure correctness or identify errors in outputs, for example, due to hallucinations
_Increased reach_ |  Growth in the number and diversity of people, including vulnerable groups exposed to GenAI
_High impact applications_ |  GenAI is used for applications with profound impact on individuals and society, for instance, in the medical domain and education
_Unknown Applications_ |  GenAI can be used for many tasks in ways that are hard to anticipate
_Difficult to evaluate_ |  Many applications of GenAI are hard to evaluate using conventional train/test set approaches, as the quality of outputs is hard to quantify
_Security and safety_ |  GenAI suffers from novel vulnerabilities (such as prompt hacks), and it has the potential to cause harm at a large scale
_Accountability and legal concerns_ |  GenAI involve more actors (data provider, foundation model creator, fine-tuner, end-user), making accountability more challenging while at the same time being subject to legal risks
Table 3: Why is XAI more challenging for GenAI? Reason |  Description
---|---
_Lack of access_ |  _No access to inner workings_
_Interactivity_ |  Outputs are due to multiple rounds of interaction, necessitating understanding humans and AI and their mutual understanding.
_Complex systems, models, data, and training_ |  Understanding gets more difficult as complexity increases, e.g., due to network and training data size, the diversity of data, and the combination of multiple training procedures. GenAI models might also be part of a complex system with multiple closely interacting non-AI components such as code interpreters and search engines.
_Complex outputs_ |  Generated artifacts can comprise millions of bits.
_Hard to evaluate explanations_ |  As automatic evaluation of outputs is hard, so is (function-grounded) evaluation of XAI
_Diverse users_ |  More varied user base
_Risk of ethical violations_ |  As GenAI suffers from biases and offensiveness, so might its explanations
_Technical shortcomings_ |  GenAI models hallucinate and exhibit reasoning errors. In particular, also self-explanations by LLMs are subject to known shortcomings such as hallucinations and limited reasoning

###  3.2 Why is XAI more challenging for GenAI?

Lack of access: Commercial GenAI models from large corporations such as OpenAI
and Google are among the most widely used systems. However, users,
researchers, and other organizations interested in understanding the
generative process of specific artifacts or models cannot access model
internals and training data of these models. This rules out many XAI
approaches.

Interactivity: Some tasks involving humans and GenAI are inherently
interactive, including negotiations that can be conducted between GenAI and
humans [143]. Thus, explanations need to focus not just on the model, but also
on how the model impacts humans and vice versa over the course of an
interaction.

Complex systems, models, data, and training: Understanding AI gets
increasingly difficult as models grow and digest more training data. GenAI
based on very large foundation models constitutes the largest AI models today
with hundreds of billions of parameters [146]. They also lead to novel, more
complex supply chains of AI. Pre-GenAI models were often built based on
company-internal data, possibly by fine-tuning a model trained on a moderate-
sized public dataset such as ImageNet. GenAI systems are built using much
larger data from many sources, including public and third-party providers.
Often, a foundation model is further adjusted through fine-tuning or
retrieving information from external data sources [73] to yield a GenAI
system. Thus, the final output of a GenAI system might be generated not only
through a deep learning model but also by engaging with other tools such as
code interpreters [135].

Complex outputs: Generated artifacts are complex, i.e. typically consisting of
thousands up to millions of bits in an information-theoretic sense. Classical
supervised learning yielding classes often produce just a single bit in case
of binary decisions and at most a few dozen bits, i.e., mostly less than a few
million classes. In some sense, before GenAI a label of a classifier
constitutes a single decision, while GenAI makes a multitude of decisions –
one for each aspect of the artifact. Thus, textual outputs and images
naturally lead to investigations of several aspects of the output, such as
tone, style, or semantics of texts [196]. There are many possible questions
about why an artifact exhibits a certain property.

It is hard to evaluate explanations, especially for function-grounded
approaches, where GenAI itself is hard to evaluate. Function-grounded
evaluation focuses on assessing an XAI method based on predefined benchmarks,
for example, to use an explanation to classify an object and see whether the
explanation is equal to the output of the model used to create the
explanation. As such benchmarks might not exist and are difficult to create
for certain GenAI tasks, evaluation of explanations also becomes more
challenging.

Diverse users: A model might be utilized by a diverse set of users across all
age and knowledge groups covering many needs – similar to a search engine. In
contrast, pre-GenAI systems are more often tailored to a specific user group
and task.

Risk of ethical violations: Even commercial GenAI models are known for
producing potentially offensive, harmful, and biased content [1]. GenAI models
also commonly self-explain, and, in turn, the possibility arises that while
the outputs might be ethical, the explanations are offensive, for example, due
to inadequate phrasing or visual depictions.

Technical shortcomings: GenAI suffers from hallucinations and limited
reasoning capability. In turn, if GenAI models self-explain, such explanations
are subject to the same shortcomings.

###  3.3 Desiderata of Explanations for GenAI

Novel Desiderata |  Description
---|---
_Verifiability_ |  Explanation should support verification of AI outputs.
_Lineage_ |  Explanations might include information on tracking and documenting data, algorithms, and processes in AI models to aid accountability, transparency, and reproducibility.
_Interactivity and personalization_ |  With complex AI systems, explanations must be customizable by users and their preferences.
_Dynamic explanations_ |  Explanations might differ in content and structure depending on the sample and specified objectives (e.g., max. plausibility, explain surprising aspects) as it is impossible to explain all details of complex artifacts.
_Costs_ |  Economics of XAI include costs associated with risks of value leakage and the costs for implementing XAI.
_Alignment criteria_ |  Explanations should align with criteria like helpfulness, honesty, and harmlessness, crucial for trust and ethical AI.
_Security_ |  Providing explanations should not compromise security, as insights on GenAI models might facilitate attacks or exploit vulnerabilities.
_Uncertainty_ |  Understanding and conveying the confidence and uncertainty of AI outputs is vital for decision-making and trust.
Table 4: Overview of novel and emerging desiderata for GenXAI

There are also several novel and emerging aspects as well as a big change in
the relevance of desiderata:

  * •

Verifiability has been recently discussed as an important aspect of
explanations [38]. But the problem that outputs of LLMs cannot be verified
(due to hallucinations) has been discussed years earlier [95]. Suppose
explanations cannot be verified, and an explainee must trust the explanation.
In that case, the consequence can be the rejection of a correct answer due to
an incorrect explanation or failure to detect incorrect outputs. A key concern
concerning verification is the effort to verify. Efforts to understand and
tailor explanations have been discussed for XAI in general, for example, for
time efficiency see [148].

  * •

Lineage ensures that model decisions can be tracked to their origins. It
refers to tracking and documenting data, algorithms, and processes throughout
the lifecycle of an AI model. It is highly relevant for accountability,
transparency, reproducibility, and, in turn, governance of artificial
intelligence [142]. It concerns the “who” and the “what”, for example, “Who
provided the data or made the model?” or “What data or aspects thereof caused
a decision?”. While the latter is a well-known aspect of XAI, as witnessed by
sample-based XAI techniques, the former has not been emphasized significantly
in the context of XAI. [37] set forth data traceability as a requirement in
the context of Machine Learning Operations (MLOPs) for XAI in industrial
applications. The need for lineage emerges as GenAI supply chains get more
complex often involving multiple companies [146] rather than just a single
one. Furthermore, multiple lawsuits have been undertaken in the context of
generative AI, for example, related to copyright issues [50]. Regulators have
also set stringent demands on AI providers [36]. Thus, employing GenAI poses
legal risks to organizations. In turn, ensuring lineage-supported
accountability can serve as a risk mitigation.

  * •

Interactivity and personalization have been discussed previously in XAI [148,
139]. However, as GenAI outputs and systems are more complex, the number of
options for how and what to explain has increased drastically. While there is
only one bit to explain in a binary classification system, it amounts to
millions for a generative AI system generating images. That is, it is nearly
impossible for a user to understand the reasons behind all possible details of
an output. Thus, depending on user preferences and the purpose of the
explanation, some aspects might be more relevant to understand than others,
necessitating the need for a user to engage with the system to obtain
explanations and for the system to provide adequate explanations tailored to
the user’s demand. As discussed in Section 4.1.3, systems supporting
interactive XAI are increasingly emerging.

  * •

Dynamic explanations aim at choosing the explanation qualities and content
(e.g., What output properties are explained?) automatically depending on the
sample, explanation objective and possibly other information. For example, an
explanation might elaborate on why a positive or negative tone for a generated
text is chosen. Tonality might only be discussed for some samples and not for
others. When to include it, should be aligned with a specified objective,
e.g., the explanation should satisfy the explainee (plausibility) or to
explain the most surprising aspects of the generated artifacts or to explain
the most impactful properties of the artifact on its consumers. The goal of
dynamic explanations is to maximize the specified objectives while being free
to choose the explanation content, including meta-characteristics such as what
aspects are included in the explanation, its structure, etc.

  * •

Costs related to XAI might also become an emerging area. Economics of XAI has
been elaborated in [10], where concerns related to the costs of implementing
XAI (and transparency) are mentioned. For cooperations, GenXAI adds the risk
of leaking value. That is, competitors (or academics) might prompt a model to
save on the costs of training data. For example, the Alpaca model [167] was
trained on data extracted from one of OpenAI’s models, allowing them to forego
the costly and time-consuming task of collecting data from humans. Using
explanations, for instance, as part of chain-of-thought (CoT) [184], can
further enhance performance and is thus of value.

  * •

Alignment criteria such as helpfulness, honesty, and harmlessness [6] that are
relevant for GenAI in general also play a role for XAI. Some of these criteria
partially overlap with existing ones, such as plausibility (with helpfulness)
and faithfulness (with honesty, for instance, explanations should not be
deceptive [144]). Aspects such as harmlessness have received less attention,
as explanations were commonly simpler (for example, attribution-based).
Harmlessness implies that explanations do not contain offensive information or
information concerning potentially dangerous activities [6]. It also relates
to security, discussed next:

  * •

Security increasingly evolves as a desideratum. Explanations should not
jeopardize the security of the user and the organizations operating the GenAI
model. Providing insights into the reasoning process might facilitate attacks
or might be leveraged in competitive situations, leading to poorer outcomes of
GenAI. For example, in a recent study on price negotiations of humans against
LLMs [143], humans asked an LLM what decision criteria it used for its
decisions and, in turn, systematically exploited this knowledge to obtain
better outcomes against the LLM. A human negotiator might not disclose such
information. Thus, openness can also be abused. “Security through obscurity”
is one protection mechanism against attacks. Cooperation might also aim to
protect their intellectual property. For example, customer support employees
(and GenAI models) might have access to some relevant information on a product
to help customers. Still, they might not be allowed to obtain explanations on
how exactly the product is manufactured as this might constitute a valuable
secret for the company.

  * •

Uncertainty: Understanding the confidence of outputs is an important
desideratum of XAI. While most works aim to explain a decision, explaining the
uncertainty of a prediction has also attracted some attention [106, 41]. Deep
learning models such as image classifiers are known to be overconfident, with
multiple attempts to address the issue [99, 41]. LLMs arguably take this a
step further, as they commonly generate answers eloquently even if they are
wrong, i.e., they are “fluent but non-factual”. Still, LLMs have some (though
not perfect) understanding of whether they can answer a question [64] or not.
They can also be enhanced with uncertainty estimation techniques [60].

Prior research has extensively discussed principles and desiderata of
explanations. Here, we briefly summarize key characteristics explanations
should exhibit drawing on prior surveys [90, 139, 148, 12, 51] and briefly
discuss based on novel characteristics of GenAI. Among the most important,
well-known desiderata are:

  * •

Faithfulness (= fidelity = reliability): An explanation should precisely
reflect the reasoning process of a model. Due to the complexity of GenAI
(models), a higher level of abstraction seems necessary to keep explanations
comprehensible in a limited amount of time.

  * •

Plausibility (= persuasiveness = understandability): An explanation should be
understandable and compelling to the target audience. Textual explanations
(especially, self-explanations) are often easy to understand (e.g., compared
to classical explanations such as SHAP values).

  * •

Completeness (= coverage) and minimality: An explanation should contain all
relevant factors for a prediction (completeness) but no more (minimality).
Asking for complete coverage becomes less feasible with the growth of output,
data, and model sizes. Personalized, interactive explanations allowing a user
to control explanations as well as XAI techniques automatically selecting only
interesting properties to be explained could be the way forward.

  * •

Complexity: Total amount of conveyed information in an explanation, typically
measured relative to an explainer’s knowledge, i.e., subjectively. This aspect
gains in importance as stakeholders become more diverse.

  * •

Input and model sensitivity and robustness: Changes in the input (or model)
that impact model outputs should also lead to changes of explanations
(sensitivity), but if changes (of inputs) do not alter model behavior or
changes in the model do not alter processing and outputs significantly then
explanations should not change disproportionately (robustness). This still
holds for GenAI.

Table 5: Dimensions of our taxonomy for GenXAI algorithms Dimension
---
_Explanation (=Output) properties:_
$\triangleright$ Scope
$\circ$ Explaining single vs. all attributes of the output $\circ$ Explaining
single input-output vs. entire interaction
$\triangleright$ Modality
$\circ$ Unimodal vs. multi-modal explanations
$\triangleright$ Dynamics
$\circ$ Interactive vs. non-interactive explanations $\circ$ Static (sample-
independent) vs. dynamic (sample-dependent) explanation qualities and content
_Input and internal properties:_
$\triangleright$ Foundational source for XAI
$\circ$ Model $\circ$ Optimization $\circ$ Training data $\circ$ Prompt
$\triangleright$ Required access by XAI Method
$\circ$ Black-box vs. white-box
$\triangleright$ Model (self-)explainers
$\circ$ Self- vs. explanations by other models/algorithms
$\triangleright$ Sample difficulty
$\circ$ Explaining simple vs. difficult samples
$\triangleright$ Dimensions of pre-GenAI

##  4 Taxonomy of XAI techniques for GenAI

Our taxonomy provides a scheme for the classification of XAI mechanisms and
algorithms supporting the understanding of GenAI (Section 4.1), which we
utilize to classify existing techniques (Section 4.2).

###  4.1 Dimensions of taxonomy

The key characteristics of our taxonomy are summarized in Table 5. We
distinguish between GenXAI algorithms’ output (= explanation) properties,
input, and internal properties. Explanation properties characterize the XAI
algorithms outputs, i.e., the explanations, in terms of scope, i.e., what
(fraction of) samples, attributes and part of the interaction they explain,
modality, that means, unimodal or multi-modal, and interactivity, i.e., can
user engage in obtaining additional explanations or tailor explanations. Input
and internal properties relate to what the XAI algorithms require producing
explanations and how they are obtained. While many ideas on structuring XAI
are still valid in the context of GenAI, we focus primarily on novel
dimensions such as the foundational source for XAI, which is one of data,
model, training, and prompt. The source forms the key mechanism or artifact
leveraged by XAI techniques to generate explanations, as elaborated in Section
4.1.4. One might classify the first three (data, model, and training) under
the category of intrinsic methods as they impact the resulting model. However,
data can also be extrinsic, particularly for Retrieval-Augmented Generation
(RAG). Similarly, for prompts, Chain-of-Thought (CoT) encourages and guides
XAI, but it also relies to some extent on training. That is, if training data
lacks any form of explanation, CoT prompting will not work. This can be most
easily seen in an extreme case, where words like “because” are removed from
the training data, implying that they will never be generated.

####  4.1.1 Output, Interaction and Input Scope

Often, scope [148] only refers to what inputs are explained by a method, i.e.,
a single sample (local) versus all samples (global), which means the model
behavior in general [52, 12]. Thus, the scope states the quantity of the input
samples, which are explained. We call it _input scope_. For GenAI, we say that
scope also refers to the quantity of the output that is explained, i.e., a
single attribute of the output (focused) vs. all attributes (holistic). We
call this _output scope_. This is illustrated with an example in Figure 4. As
outputs are significantly more complex for GenAI, they raise more options for
questions. For example, why did the response contain some information and not
another? Why was the sentiment of a generated sentence positive, neutral, or
negative? While some of our methods touch on these questions, overall, there
is limited work in this direction. Similarly, there is limited understanding
of how to relate training data to predictions beyond classical approaches such
as influence functions, i.e., how did a specific piece of knowledge in the
training data impact the generation process?

Figure 4: Illustration of input and output scope, where we use “tonality” as
an example of a text-related attribute. Traditionally, scope only referred to
output scope.

_Interaction scope_ : For GenAI, interactions are no longer commonly in the
form of dialogues, and understanding both sides of the system becomes much
more relevant. As such, we distinguish two goals:

  * •

Explaining (single) input-output relations: This is the classical notion,
where an AI system processes one input to produce one output. The goal can be
to understand a particular instance (local explanation) or the model as a
whole. While explanations might consider user-specific aspects and personalize
explanations [139], such personalization is typically independent of the
interaction.

  * •

Explaining the entire interaction: It focuses on human-AI interaction and its
dynamics, for example, the communication and actions between an AI and a human
when a human solves a task using an AI. Interactions are characterized by
multiple rounds of outputs generated by both sides conditioned on prior
outputs. In this case, the goal is to more holistically explain (i) the
dynamics of the interaction, i.e., not just one single input-output but the
entire sequence of in- and outputs and (ii) the outcome of the interaction,
which could be why a particular artifact such as an image was generated in a
certain way, but also why a user did not complete a task, for example,
abandoned it prematurely or achieved an unsatisfactory output. Interaction
dynamics are influenced by a series of technical (such as model behavior
including classical performance measures but also latency, user interface,
etc.) and non-technical factors (such as human attitudes and policies). As
such, human-AI interaction cannot easily be associated with one scientific
field but is inherently interdisciplinary. Explainability, which aims at
understanding AI technology, should focus on how technical factors related to
model behavior impact the interaction. While many existing works touch on the
subject, the change in interactivity brought along by prompting due to GenAI
is not well understood. A study that investigated interactivity in
negotiations was [143]. However, the explanations for outcomes of negotiations
and interaction behavior are not through algorithms but rather through a
manual, qualitative investigation of interaction – a common technique in
social sciences but less prevalent in computer science.

The two goals are illustrated with an example in Figure 5.

Figure 5: Illustration of input and interaction scope, where we use “tonality”
as an example of a text-related attribute. Traditionally, scope only referred
to output scope.

The field of human-computer interaction has aimed at explaining human
interactions for a significant amount of time [92, 15] with some effort also
on discussing human-AI interaction. For example, guidelines for human-AI
interaction (before GenAI) are well-studied [5], also the topic of explanation
in collaborative human-AI systems [171]. Furthermore, explanations in human-AI
systems typically encompass objectives that are driven by non-technical
concerns [101, 108] but there is less work on explaining human-AI interactions
themselves [164], in particular, targeted towards GenAI. To provide two
examples from the pre-GenAI area spanning from in-depth technical study to
broader organizational studies: [136] discussed how human-AI interaction could
be optimized, accounting for long-term goals such as preserving human
diversity. The work would explain how the user can improve her interaction to
reduce error rates and become more efficient. [48] investigated the dynamics
of AI adoption within an organization, explaining how error rates and learning
behavior of an AI impact the complexity of processes. Explanations in
interactive systems in pre-GenAI [122] and GenAI (see Section 4.1.3) are
typically more concerned about supporting interactive explorations of a
decision, for instance, querying a system to better understand it rather than
obtaining explanations on how a sequence of inputs and outputs emerged.

####  4.1.2 Explanation Modality

Commonly, explanations are unimodal, for example, textual, visual, or numeric.
However, also multi-modal explanations have been investigated [118] by
collecting a dataset containing textual and visual justifications. They show
that multi-modal explanations yield favorable outcomes, for instance, each
modality improves due to the presence of the other.

####  4.1.3 Dynamics

_Interactivity:_ In a classical setting, XAI is often non-interactive, meaning
an explainee has limited options to control explanations or request additional
information. However, the idea of interactivity has existed in XAI for some
time, as shown in prior taxonomies [148]. One idea is to rethink XAI as a
dialogue [158]. [160] supports the explainability of a machine learning model
by using an LLM to translate natural language queries into a pre-defined set
of operations related to explainability, for example, getting the most
important features, changing a feature, etc.. [40] uses LLMs to create an
interactive and explainable recommender system. In terms of classical metrics
such as recall and precision, the system is not outperforming. Improving
models due to human explanations in an interactive setting has also been
studied for image recognition [147]. While LLMs are known for their superior
performance in many Natural Language Processing (NLP) tasks, they might also
be employed without improving classical metrics such as accuracy or precision
and recall but rather to yield other benefits such as explainable, interactive
systems requiring less (training) data [40, 28]. [28] showed that binary risk
classification can be done with 40x fewer data to roughly match the
performance of a classical machine learning system through explainable-guided
prompts.
_Static vs. dynamic explanation qualities and content_ relates to how the
structure and content of explanation vary based on the sample due to the
algorithm. That is, which and how properties of the generated artifact are
explained. This choice can be independent of the sample(static), i.e.,
identical for each sample, or it can be dependent on the sample
itself(dynamic). For example, classical methods like attribution maps are
static, as they only explain relevance and they always assign a relevance
value for each input pixel. In contrast, textual self-explanations can be
dynamic. The explained properties can be selective, e.g., for a generated
image an explanation like “A moving, red car was shown to make the image more
vivid and engage viewer” only describes one part of the image and one property
of that part. Explanations are dynamic if the structure and content of the
explanation varied with the sample. Say the XAI technique yielded a generated
image of a mountain landscape “A snowy landscape was chosen as the prompt
contained the word “bright”. Explanations were dynamic as the two exemplary
explanations refer to a different object (car vs landscape), differ in
explained properties (explaining emotional intention vs. not discussing it),
and differ in causal factors discussed (mentioning what inputs caused output
properties, i.e., “bright” in prompt vs. not doing so).

####  4.1.4 Foundational Source for XAI techniques

We consider the data, model, and optimization (training) and prompt the
foundational sources for XAI methods. That is, each source can be modified or
tailored to improve XAI.

Model induced XAI refers to intrinsic XAI methods (also named model-specific
XAI [52]), where the design of the model is altered to foster explainability.
With GenAI, novel intrinsic methods have emerged.
_Using interpretable components_ : Deep learning is a composition of multiple
layers and components, such as activation functions and attention mechanisms.
These components can be more or less complex, limiting the overall
explainability of the model.For example, interpretable activation functions
might substitute conventional less interpretable functions, for instance, SoLU
[32] is said to enhance interpretability. These functions are built on the
idea that there are more features than neurons (per layer), and, in turn,
superposition yields additional features [112]. The paper also provides
evidence for this hypothesis. The interpretability makes some neurons more
interpretable but hides others. Thus, while the method overall claims to
provide a net benefit, it is not without costs. Classical attention layers are
also commonly used, though their value for XAI is debated (see Sec. 4.2.1).

_Interpretable GenAI models through additional models_ : Model
interpretability can be achieved by combining LLMs with other models and often
external knowledge. For example, in [21] an LLM is enhanced with a GNN and
external knowledge that generates an explanation and prediction jointly. [26]
fostered explicit multistep reasoning by chaining responses of two fine-tuned
LLMs; one performs selection, the other inference. [180] uses the idea to
incorporate external knowledge using classical internet search (as is done in
commercial products such as Bing Chat). In addition, it also uses first-order
logic; for example, it creates easier-to-verify subclaims (that jointly lead
to the overall claim). It also generates explanations simply by querying the
LLM. [177] developed and trained a decoder for faithful and explainable online
shopping product comparisons.

Optimization: Adjusting the optimization objective is a common technique to
foster explainability. Common strategies involve disentanglement of latent
dimensions [129] as well as training also for XAI relevant criteria, such as
citing evidence [98]. In the context of GANs, [19] trained explicitly
disentangled representations by maximizing the mutual information between a
small subset of the latent variables and the sample. This allowed the so-
called InfoGAN to separate writing styles from digit shapes on the MNIST
dataset. Furthermore, in [129], explanations have been used to constrain
training, such as “to be right for the right reasons”. Note that while
disentanglement is commonly achieved through training objectives and
regularization, special network architectures like capsule networks [130] and
others (see Section on Disentanglement in [148]) can also support
disentanglement. Diffusion models already have a semantic latent space [72]. A
special reverse process can leverage it for image editing using CLIP [124],
which iteratively improves reconstructed images. [98] trains (more than one)
model to cite evidence for claims, facilitating answer understanding and,
especially, fact-checking.

Training data can support XAI in multiple ways. While the exact impact of
training data and dynamics is not yet fully explored [168], current findings
indicate that training data is an important factor to support explainability –
particularly for textual data.
_Training data composition:_ GenAI training data is usually complex – even for
a single modality. For example, text data might consist of programming code,
books, dialogues, etc. The composition of the training data can impact
reasoning, i.e., code can improve certain reasoning task [91]. While less
explicit statements are known for explanations, the overall data composition
is also likely impacting XAI.
_Explanation quality and quantity_ : Aside from the overall composition of
training data, the presence and absence of explanations in the training data
also impacts XAI. While it is well-known that the quality of training data has
a strong impact on the performance of a model, the impact on XAI has received
less attention. However, as GenAI can self-explain as witnessed by CoT [184],
i.e., generate explanations as part of outputs, training data strongly impacts
performance. For illustration, assume that a model is trained with erroneous
explanations though potentially correct results, it could in principle perform
well on tasks but provide poor explanations. Also, if the training data does
not contain any data explaining the reasoning, the model might perform worse
at explanations. In an extreme case, if the training data does not contain
explanations including explanatory words such as “due to”, “for that reason”
and “because” the model will also never generate such words (and the
corresponding explanations).
_Domain specialization_ [81] can tailor the model more towards a specific
domain, potentially at the cost of abilities in other domains [18]. It can
also contribute towards explainability, as a narrower model focus implies
fewer potential options for explanations and, thus, a lower risk of errors and
easier verifiability.

Prompts can also induce explanations: LLMs can be prompted (i) to provide
explanations in a preferred manner or (ii) be constrained to rely on a limited
set of given facts to create responses, which facilitates verification. The
idea of “rationalization” dates back to the early 2000s [198]. One type of
explainability is “justifying a model’s output by providing a natural language
explanation” [54]. Commonly, explanations clarifying the reasoning process are
elicited through the use of chain-of-thought(CoT) prompting [184]. Such
prompting allows the structure of the reasoning process, implicitly shaping
explanations and utilizing external knowledge for each reasoning step to yield
more faithful explanations [184, 55]. However, explanations can also contain
hallucinations, as shown in the context of few-shot prompting [195] and
sensitivity to inputs for CoT prompting [175]. Despite their unreliability (to
explain the true inner workings due to lack of faithfulness), they might still
be of value to verify output correctness or, in the training of (smaller)
models [207]. Additionally, LLMs can be constrained to rely on information,
which is part of the user-given prompt or extracted from an external database
or the internet, known as Retrieval-Augmented Generation (RAG) [73]. RAG
facilitates understanding the response of an LLM and, especially, verifying
it, as the source of information for the answer is known and typically very
small compared to the entire training data of a GenAI model. To facilitate
explainability for recommender systems, personalized prompt learning, for
example, soft-prompt tuning to yield vector IDs, has been conducted [75].

####  4.1.5 Required Model Access by XAI Method

Depending on what is to be explained and the XAI method, different information
is needed to obtain explanations [148]. Black-box access does not provide
information on likelihoods other than the predicted output, and potentially
even such probabilities might not be given. That is, internal access, e.g., to
activations and gradients, is commonly unavailable. White-box access refers to
having complete access to the model, its training data, and its training
procedure. In between, there is a wide range of grey-box access [138]. An
important restriction for XAI techniques investigating commercial models is
that models are typically only accessible as a black box, such as GPT-4
through an API. Similarly, commercial vendors do not share their training data
and often do not even disclose a coarse summary of the training dataset. Thus,
also XAI techniques leveraging training data are not easily employable.

####  4.1.6 Model (Self-)Explainers

GenAI, in particular, LLMs provide explanations for their own decisions (self-
explain) or serve as explainers in general. That is, the model itself provides
explanations rather than a dedicated XAI technique. This contrasts the
classical notion of intrinsic XAI that often denotes understandable, simple
models such as decision trees and linear regression. While self-explanations
are not necessarily faithful [175], attempts have been made to improve them.
For example, [23] proposed to use an evaluator to quantify faithfulness and
optimizing faithfulness scores iteratively. Though self-explanations are not
necessarily accurate or faithful, they can be helpful, as demonstrated in a
complex environment, where agents performed multistep planning and improved
through self-explanation [182].

LLMs can serve themselves as explainers by providing explanations for
essentially anything, for instance, they can self-explain by generating
explanations tailored to their outputs [184, 175] or support explaining other
machine learning models [115, 160, 157], provide explanations by analyzing
data, for example, pattern in data through autoprompting [156], support people
in self-diagnosis [152], or yield interpretable autonomous driving systems
[93]. In fact, on free-form coding tasks (generation), LLMs produce
explanations that often exceed the quality of crowd workers’ gold references
[208]. For mental health analysis, explanations of LLMs approach those of
humans in quality [192].

####  4.1.7 Explanation Sample Difficulty

Not all input samples are equivalently challenging to explain [132]. The idea
that some samples and interactions are easier to explain than others gains in
relevance, as there is a larger variance in possible inputs and outputs. For
example, LLMs allow us to ask the most simple to the most complex questions
from a human perspective. LLM-generated texts might be a “lookup” of a fact
learned from the training data onto long stories or solutions to complex
tasks. Explanations (as judged by humans) might be worse for difficult samples
than for simple ones. More precisely, this has been observed in explaining
data labels with GPT-3, where GPT-3 explanations degraded much more with
example hardness than human explanations [132]. Generally, the idea of
distinguishing XAI tasks based on their difficulty has received little
attention so far. From a computational perspective, current explanation
algorithms require an identical amount of computation independent of the
difficulty. Also, forward computations typically proceed identically
independent of sample difficulty, though the notion of reflection (and
thinking fast and slow) has been mentioned in the literature [141]. LLMs can
benefit from ensemble methods, for example, by combining multiple outputs into
a single output [59]. There are numerous works on self-correction [117]. For
example, self-debugging using self-generated explanations as feedback has been
associated with reducing coding errors by LLMs [20]. However, the ability of
LLMs to self-correct reasoning by “reflecting” on their responses without
additional information has also been questioned [58]. While state-of-the-art
systems like GPT-4 improve scores on causal reasoning benchmarks, which can
serve as explanations, they also exhibit unpredictable failure modes [67].

####  4.1.8 Dimensions of Pre-GenAI

There is also a long list of concepts relevant to our taxonomy that originate
from pre-GenAI [2, 209, 31, 148, 125, 131, 163, 102], which we do not
elaborate on in detail. For example, XAI methods can also be classified
according to what to understand, which includes the system, model-related
information such as representations (layers, vectors, embeddings) and outputs,
training dynamics, and (impact) of data. This distinction has already been
made before GenAI [148]. The most common ways to structure XAI techniques in
prior works are, unfortunately, not conceptually clean; for example, a common
distinction is between mechanistic and feature attribution-based techniques.
But conceptually, feature attribution relates to how the explanation looks
(i.e., relevance scores for output). At the same time, mechanistic
interpretability aims more at what is being investigated (i.e., neurons and
interactions) and how the techniques work (through reverse engineering). As
the names of the existing categories are well-established and, thus, easy to
comprehend for a reader, we shall also use them in our classification of
techniques shown in the next section, but also discuss classification based on
our novel dimensions shown in Table 5.

###  4.2 Classification of Techniques

We place XAI techniques into four categories commonly found in existing
literature (see prior surveys in Section 1). Figure 6 shows an overview of
techniques. Our focus lies on techniques that have been developed particularly
for GenAI models or constitute classical techniques that have been adjusted
towards GenAI (mostly by addressing computational issues) or could be employed
without much change. We also structure existing techniques in terms of our
novel dimensions (Section 4.2.5).

Figure 6: Overview of categories of techniques with illustrative examples
(Figures are from cited references)

####  4.2.1 Feature Attribution

Feature attribution provides a relevance score for each input feature, such as
a word or pixel.
Perturbation-based techniques alter inputs partially, for example, by removing
or changing features and investigating output (changes). In the context of
NLP, alterations such as removing tokens have been investigated [188] as well
as negating and intensifying statements [74].
Gradient-based methods require a backward pass from outputs to inputs to
obtain derivatives. While not all gradient-based techniques are said to work
reliably [3, 44], some techniques like Grad-CAM [150] that compute a function
of gradients and activations have shown to be valuable on a pixel level in
images but also to compute token-level attribution [105]. Directional
gradients have also been used in NLP models [154, 34]. Integrated gradients
have been used to attribute knowledge to internal neurons [88, 27] as
discussed under “Neuron activation explanation”. Simple first derivatives
concerning embedding dimensions are shown in [74].
Surrogate models approximate large models using much simpler models - often to
understand individual predictions. Classical methods are LIME [128] and SHAP
[87], which have been ported for transformers, that means, for SHAP, see [71].
Furthermore, attention flows in NLP models have been shown to be related to
SHAP values [35]. Explain Any Concept (EAC) [166] presents an approach for
concept explanation, utilizing Segment Anything Model (SAM) [70] for initial
segmentation and introducing a surrogate model to enhance the efficiency of
the explanation process. SAM excels at producing object masks from input
prompts like partial masks, points, or boxes. It can generate masks for all
objects in an image. SAM is trained on a vast dataset that includes 11 million
images and 1.1 billion masks.
Decomposition-based methods: Decomposition traditionally refers to attributing
relevance from outputs towards inputs or decomposing vectors, but in the
context of GenAI, it can also refer to decomposing the reasoning process and
attributing outputs to specific reasons. [83] aims at explaining the reasoning
process for question answers using entailment trees constructed using
reinforcement learning. An entailment tree has a hypothesis as its root,
reasoning steps as intermediate nodes, and facts as leaves.

Common decomposition techniques compute relevance scores on a layer-by-layer
level so that contributions of upper layers emerge as a combination of lower-
level contributions. A classic example is Layer-wise relevance propagation
(LRP) [107]. Decomposition-based methods have also been applied to
transformers[4]. [4] claimed that their adaptation of LRP mitigates
shortcomings of gradient methods, which are said to arise due to layernorm and
attention layers. Linear decomposition has also been suggested for local
interpretation for transformers [194], where a decomposition is considered
interpretable if it is orthogonal and linear. Decompositions are often vector-
based [89]. They express a vector (such as a token embedding) in terms of more
elementary vectors. For example, [104] decomposed (token) vectors and
propagated them through the network while maintaining accurate attribution.
[209] surveyed XAI methods for (word) embeddings. One idea is to express
embedding vectors in terms of an (orthogonal) basis of interpretable basis
(concept) vectors; another technique employs external knowledge and
sparsification of (dense) vectors.
Attention-based: Attention is a key element within neural networks. It
provides an importance score for inputs, where inputs are not necessarily
inputs to the network but those of a prior layer. For LLMs, attention scores
are commonly obtained between all input token pairs for a single attention
layer, which can be visualized using a heatmap or bipartite graph [178].
Relevance scores have also been computed by combining attention information
and gradients [9]. Attention-based methods have been scrutinized, as they
might not identify the most relevant features for predictions [151, 61].
However, the debate has not yet been settled. [165] focus on explaining
language models for long texts leveraging sparse attention by developing a
masked sampling procedure to identify text blocks contributing to a
prediction.
Some listed techniques leverage several ideas, for instance, [104] can be
considered a vector-based and decomposition-based method.

####  4.2.2 Sample-based

Sample-based techniques investigate output changes for different inputs. In
contrast to perturbation-based methods that change more selectively individual
features to investigate their impact, sample-based techniques focus more on
the sample in its entirety to understand the relationship between various
inputs and their corresponding outputs rather than attributing the output to
specific features within a single input.
“Training data influence” measures the impact of a specific training sample on
the model, typically on the output for a particular input to investigate. [49]
addressed computational issues to employ influence functions for LLMs.
Explainability has been transferred from a large natural language inference
dataset to other tasks [197].
Adversarial samples are input alterations due to small, hard-to-perceive
changes for humans that lead to a change in outputs. They are typically
discussed in the context of cybersecurity, where an attacker aims to alter
model outputs, while a human should not notice the input change. However, also
schemes that aim to “trick” humans and classifiers alike have been proposed
[137]. For example, SemAttack [179] perturbs embeddings of (tokens) of BERT,
while other attacks exchange words [63]. Parts of inputs can also be occluded
to understand model behavior better [140].
Counterfactual explanations seek to identify minimal changes to an input so
that the output changes from a class $y$ to a specific class $y^{\prime}$. In
contrast to adversarial samples, changes can be noticed by humans. For
example, GPT-2 has been fine-tuned to provide counterfactuals based on pairs
of original and perturbed input sentences[187]. Exploring LLM capabilities
through counterfactual task variations, [189] has shown that LLMs commonly
rely on narrow, context-specific procedures that do not transfer well across
tasks. [7, 62] use diffusion model guided by classifiers to create
counterfactual explanations. Contrastive explanations explain why a model
predicted $y$ rather than $y^{\prime}$. Contrastive explanations are said to
better disentangle different aspects (such as part of speech, tense,
semantics) by analyzing why a model outputs one token instead of another
[196].

####  4.2.3 Probing-based

Probing-based methods aim at understanding what knowledge an LLM has captured
through “queries” (probes). A classifier(probe) is commonly trained on a
model’s activations to distinguish different types of in- and outputs.
Knowledge-based: For example, encoders such as BERT, MiniXX, and T5 that
produce vectors can be probed by training a classifier on their outputs that
aims to identify the presence of properties of outputs or abilities that
emerge from the inputs, such as syntax knowledge [16] and semantic knowledge
[169]. Alternatively to using classifiers, datasets focusing on specific
aspects such as grammar [94] can be created. The model’s performance on the
dataset indicates the ability to capture the property. The design of datasets
requires care as regularities might provide an opportunity for shortcut
learning [206] that foregoes learning the properties in favor of identifying
such dataset-specific regularities. [56] learns how to map statements in
natural language to fact encodings (in an LLMs representation). In turn, they
allow a new way to detect (and explain) when LLMs fail to integrate
information from context. The research argues that untruthful texts result
from not integrating textual information into specific internal
representations. For example, [82] investigated the training of RoBERTa over
time through probing. They found that local information, such as parts of
speech, is acquired before long-distance dependencies such as topics. [46]
analyzes the learning of text summarization capability of a large language
model by obtaining summaries and output probabilities for a fixed set of
articles at different points during training. They show n-gram overlap between
generated summaries and original articles over time, concluding, for example,
that models learn to copy early during training (leading to high overlap),
which gets smaller over time (decrease in overlap).
Concept-based explanation: Typically, given a set of concepts, concept-based
explanations provide relevance scores of these concepts within inputs [68].
More recently, it has also been proposed to uncover concepts based on what
input information is still present at specific layers (or embeddings) [140].
While the latter investigates images, high-impact concepts as a source of
explanation have also been used for LLMs [205]. [39] allows to interpret a
large set of individual neurons by constructing a visualizable graph using the
training data and truncation and saliency methods. However, concept-based
methods must be designed carefully, as merely investigating interactions among
input variables might be insufficient to show that symbolic concepts are
learnt [76].
Neuron activation explanation: Individual neurons can also be understood using
their activations for inputs. Recently, GPT-4 has been used to generate
textual explanations for individual neuron activations of GPT-2 [115]. For
example, GPT-4 summarizes the text triggering large activations for a neuron.
[27] uncovered “knowledge neurons” that store particular facts, i.e., they
performed knowledge attribution by setting neuron weights to 0 and increasing
it to its original value, while summing up the gradient. If the neuron is
relevant for a particular fact, the sum should be large.

####  4.2.4 Mechanistic interpretability

Mechanistic interpretability investigates neurons and their interconnections.
It aims at reverse-engineering model components into human-understandable
algorithms [111]. To this end, models can be viewed as graphs [42]. Circuits,
i.e., subgraphs, can be identified that yield certain functionality [181].
Common approaches fall into three categories [89], i.e., circuit discovery,
causal tracing, and vocabulary lens. The typical workflow to discover a
circuit is often manual and involves [25]: (i) Observing a behavior (or task)
of a model, creating a dataset to reproduce it, and choosing a metric to
measure the extent to which the model performs the task. (ii) Define the scope
of interpretation (for example the layers of the model), and (iii) perform
experiments to prune connections and components from the model. Circuit-based
analysis can, thus, also focus on specific architectural elements; for
instance, feedforward layers have been assessed and associated with human-
understandable concepts [43]. Also, two-layer attention-only has been
investigated, leading to conjectures about how in-context learning might work
[113]. Recent work has automated finding connections between (abstract) neural
network units that constitute a circuit [25].

(Modern) causal tracing commonly estimates the impact of intermediate
activation on their output [97]. While causal tracing moves from activation
toward outputs, the vocabulary lens focuses on establishing relations to the
vocabulary space. For example, [43] projects weights and hidden states to the
vocabulary space. Individual tokens have also been assessed [123, 66]. [66]
create information flow graphs showing (human-readable) tokens based on
processed vectors within attention heads and memory values. That is, these
vectors are mapped to tokens.

####  4.2.5 Structuring based on novel dimensions

We also classify existing techniques based on uncovered characteristics, shown
in Table 5. Concerning scope, no existing technique explicitly focuses on the
entire interaction. However, LLMs providing self-explanations could be used
for that purpose. When it comes to explaining one, multiple or all properties
of the output, a feature attribution map explains typically at most one
attribute of the output, e.g., by highlighting positive and negative words or
phrases sentiment might be explained. Sample-based techniques can potentially
explain multiple attributes, e.g., if the chosen samples share different
characteristics, e.g., for classification if the most similar images show the
same object but within a different pose and of different color, it can be
concluded that the latter two attributes are irrelevant. Mechanistic
interpretability and probing often aim at isolating a single concept, but this
is not a must.

Existing XAI techniques are typically non-dynamics. Explanations from LLMs can
be interactive, personalized and are often also sample-dependent.

Figure 6 shows a concept matrix linking the foundational source to XAI
methods. Feature attribution techniques are typically posthoc methods that
leverage the model to generate explanations. However, LLMs could also be
prompted to generate feature attribution techniques. Sample-based techniques
commonly rely on the training data and the model, for example, to determine
which samples most strongly activate a particular neuron. While mechanistic
approaches commonly rely on a dataset, it does not have to be data from the
training. Probing can be performed naturally through prompts and other forms
of providing inputs (and analyzing corresponding outputs).

| Existing Categories | Selected GenAI Techniques
---|---|---
| Feat. Att. | Sample | Mechan. | Prob. | CoT | RAG
Training data |  | x | (x) |  | (x) | (x)
Model | x | x | x | x |  |
Optimization |  |  |  | x |  |
Prompts | (x) |  |  | x | x | x
Table 6: Mapping of XAI categories and selected techniques to foundational
sources for XAI

Most existing techniques require white-box access 7. It might be possible to
gain limited insights using black-box access only, for example, using
occlusion (or masking) for feature attribution techniques. It might be
investigated if predictions change, but such an approach typically yields
coarser and less accurate explanations compared to having access to the output
probabilities.

| Existing Categories | Selected GenAI Techniques
---|---|---
| Feat. Att. | Sample | Mechan. | Prob. | Self-explain |
Required Access |  |
Black-Box | (x) |  |  |  | x |
White-Box | x | x | x | x |  |
Table 7: Mapping of XAI categories and selected techniques to required access
by XAI techniques

##  5 Research Agenda of XAI for GenAI, Discussion and Conclusions

As the field of XAI for GenAI is quickly emerging, there are many
opportunities for research covering more technical algorithmic avenues onto
economic and psychological aspects. Bridging the gap between AI research and
other disciplines, such as cognitive science, psychology, and humanities, is
likely the way forward for many topics.

  1. 1.

Explaining interactions rather than single input-outputs is particularly
interesting for interdisciplinary research such as conducted in the field of
information systems and human-computer interaction as it requires both
understanding of humans, the model, and the system (See “Interaction Scope” in
Section 4.1.1).

  2. 2.

Real-time and interactive explanation based on user queries and feedback could
be further explored, necessitating also insights from multiple disciplines
(Section 4.1.3).

  3. 3.

Multimodal GenXAI: Today, most explanations are of a single modality, i.e.,
commonly either text or visual. There is a lack of techniques providing
explanations using more than one modality (Section 4.1.2).

  4. 4.

Adding XAI to novel directions of GenAI, for instance, GenXAI for video, 3D
content generation, and actions (see Table 1) is urgently needed.

  5. 5.

Deriving novel XAI techniques. In particular, the field of mechanistic
interpretability (Section 4.2.4) investigating the inner workings of GenAI
models is a promising yet challenging frontier.

  6. 6.

Addressing verifiability and hallucinations using AI. Hallucinations are
arguably one of the biggest challenges of GenAI. GenXAI can support to
mitigate them as seen through techniques such as Chain-of-thought, explaining
the reasoning process, but more techniques are needed.

  7. 7.

Porting of pre-GenAI XAI techniques by addressing computational concerns of
classical techniques to apply them to large models, as was done for techniques
like SHAP (see Section 4.2.1).

  8. 8.

Personalization of explanations: Recognizing the diversity in users’
backgrounds, expertise, and needs, future research should focus on
personalizing explanations as an important desideratum (see Section 3.3).

  9. 9.

Explanation difficulty has received little attention. A more thorough
understanding in the context of GenAI, quantifying the phenomenon and
potentially even leading to techniques accounting for such difficulty, is
another future avenue (see Section 4.1.7).

  10. 10.

Dealing with the complex nature of outputs of GenAI (compared to simple
classification in the context of pre-GenAI) remains under-explored. While
interactive, user-driven investigation is one avenue in this regard (Section
4.1.3) as well as isolating particular facets using mechanistic
interpretability techniques focusing on circuits (Section 4.2.4), in general,
little is known as of today. For example, it is not clear what facets should
be explained for text or images (see “Output Scope” in Section 4.1.1) and how
to relate actual explanations to high-level objectives such as maximizing
plausbability, e.g., “What attributes should be explained so that an
explanation appears plausible?” (Section 4.1.3).

  11. 11.

Building GenXAI to target ethical, societal, and regulatory concerns,
including contributing to the mitigation of biases and enhancing fairness by
identifying and quantifying them through XAI techniques (see second last
paragraph in this section).

  12. 12.

GenXAI itself raises a number of critical issues with respect to ethical
concerns. For example, in legal cases an explanation why someone performed an
action can decide over life and death. As such explanation themselves need to
carefully assessed with respect to alignment criteria(Section 3.1).

As with any other work, this paper employs a particular perspective on the
matter of study, foregoing some aspects, and emphasizing others. More
concretely, we did not aim to be comprehensive in all regards. Due to the vast
array of works on GenAI and XAI, we decided to forego detailed aspects of XAI
related to evaluation and usage of explanations. Furthermore, we did not
reiterate all existing XAI techniques before GenAI; rather, we focused on
novel aspects and methods for GenAI and referred to other surveys for this
purpose. Concerning modalities, our core focus was on images and text, driven
by the fact that these are (as of now) the two most prevalent modalities.
However, others, like video and 3D content generation, are quickly emerging,
and audio-to-text (and vice versa) has already been established for some time.
We conceptualized existing techniques and discussed some technical aspects,
but a more mathematical treatment could be undertaken. As AI quickly evolves,
our work can only be seen as a snapshot in time. Our conceptualization is
subject to further evolution, though we believe the uncovered dimensions will
pass the test of time, albeit with some enhancements and modifications.
Furthermore, GenAI (in combination with XAI) touches on many ethical and
societal implications, which we partially touched upon in Sections 3.2 and
3.1. However, we refrained from a detailed discussion. We opted to provide a
more neutral stance pointing towards concerns rather than being prescriptive
and elaborating on actions to be taken to address ethical issues. For example,
the lack of access to (commercial) models hinders transparency on the one hand
but protects company know-how on the other hand. As such, one might advocate
for demanding regulation to increase transparency or double down on allowing
companies to be secretive about details regarding their models to protect
their intellectual property. From an organizational perspective, more
transparency might also support user trust. Thus, the preferred governance of
GenAI concerning transparency by companies or from a societal level is
debatable [145]. We do not take any particular stance in such debates.

To conclude, GenXAI is an important area for AI in general. Given the rapid
advancement of GenAI and its widespread implications on individuals, society,
and economics, more work should be dedicated to this area – especially given
the many research gaps highlighted in our research roadmap elaborating on
aspects such as interactivity and scope of explanations. Our work contributes
in this direction by providing a thorough motivation for the study of XAI for
GenAI, structuring existing knowledge through an enhanced conceptualization of
XAI, uncovering novel dimensions, and setting forth a research agenda calling
for a joint effort of the research community to address open issues that
hopefully contribute to the well-being of all of us.

##  6 Acknowledgments

Two very humble and reputable researchers also contributed to this manuscript
but still felt that their contribution did not deserve co-authorship or even
be mentioned explicitly in the acknowledgments, which is remarkable in the
competitive field of research. Both of whom I want to thank cordially.

##  7 Appendix

###  7.1 Research Methodology

Our approach combines aspects of a systematic and a narrative review. However,
as explained, XAI for GenAI is rather novel, with a relatively limited number
of technical works available. A more qualitative approach was our main focus,
for example, a narrative literature review [69]. While we describe our search
process below, our goal was not to quantitatively assess peer-reviewed works
by showing the evolution and counts of publications over the last years.
Currently, works on XAI for GenAI lack a comprehensive conceptualization of
important aspects that inform technical works, such as challenges and
desiderata for explanations for GenAI. Therefore, the importance and
challenges we identified are a synthesis of existing works and a creative
combination taking essential properties of GenAI and prior XAI desiderata into
account. Our taxonomy is based on (i) a meta-survey of (pre-)GenAI taxonomies
and surveys, as many concepts and characteristics are still relevant for
GenAI, as well as (ii) novel works contributing towards XAI for GenAI. We
searched for several terms on Google Scholar between the 15th of February 2024
and the 15th of March 2024 using either “survey” or “review” in “Survey
explainability”, “Survey XAI for generative AI”, “Survey explainability for
large language models”, “explainability large language models”,
“explainability transformer”, “explainability diffusion model”. This was
followed by forward and backward search [183]. We filtered results based on
title, abstract, and full-text as follows: We preferred peer-reviewed works
but also commonly included articles from arxiv.org after performing our
quality assessment as reviewers. We were stricter in inclusion with older
works (on pre-GenAI) and more open to works on XAI for GenAI containing novel
aspects as there are fewer works, and we deem it important to include ideas
that are still in the making. Our taxonomy development process followed that
of [110] that is we alternated between looking at concepts (mostly synthesized
in earlier surveys) and empirical data (primary research papers on XAI for
GenAI) to derive our dimensions iteratively and, ultimately, our taxonomy.

## References

  *   * Achiam et al [2023] Achiam J, Adler S, Agarwal S, et al (2023) Gpt-4 technical report. arXiv preprint arXiv:230308774
  * Adadi and Berrada [2018] Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access
  * Adebayo et al [2018] Adebayo J, Gilmer J, Muelly M, et al (2018) Sanity checks for saliency maps. Advances in Neural Information Processing Systems
  * Ali et al [2022] Ali A, Schnake T, Eberle O, et al (2022) XAI for transformers: Better explanations through conservative propagation. In: International Conference on Machine Learning
  * Amershi et al [2019] Amershi S, Weld D, Vorvoreanu M, et al (2019) Guidelines for human-ai interaction. In: Proceedings of the CHI conference on human factors in computing systems
  * Askell et al [2021] Askell A, Bai Y, Chen A, et al (2021) A general language assistant as a laboratory for alignment. arXiv preprint arXiv:211200861
  * Augustin et al [2022] Augustin M, Boreiko V, Croce F, et al (2022) Diffusion visual counterfactual explanations. Advances in Neural Information Processing Systems
  * Baidoo-Anu and Ansah [2023] Baidoo-Anu D, Ansah LO (2023) Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI
  * Barkan et al [2021] Barkan O, Hauon E, Caciularu A, et al (2021) Grad-sam: Explaining transformers via gradient self-attention maps. In: Proceedings of the ACM International Conference on Information & Knowledge Management
  * Beaudouin et al [2020] Beaudouin V, Bloch I, Bounie D, et al (2020) Flexible and context-specific AI explainability: a multidisciplinary approach. arXiv preprint arXiv:200307703
  * Betker et al [2023] Betker J, Goh G, Jing L, et al (2023) Improving image generation with better captions. Computer Science URL https://cdn.openai.com/papers/dall-e-3.pdf
  * Bodria et al [2023] Bodria F, Giannotti F, Guidotti R, et al (2023) Benchmarking and survey of explanation methods for black box models. Data Mining Knowledge Discovery
  * Brooks et al [2024] Brooks T, Peebles B, Holmes C, et al (2024) Video generation models as world simulators. URL https://openai.com/research/video-generation-models-as-world-simulators, accessed on 2024-03-15
  * Cao et al [2023] Cao Y, Li S, Liu Y, et al (2023) A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:230304226
  * Carroll and Olson [1988] Carroll JM, Olson JR (1988) Mental models in human-computer interaction. Handbook of human-computer interaction
  * Chen et al [2021] Chen B, Fu Y, Xu G, et al (2021) Probing BERT in hyperbolic spaces. arXiv preprint arXiv:210403869
  * Chen et al [2023a] Chen L, Sun L, Han J (2023a) A comparison study of human and machine generated creativity. Journal of Computing and Information Science in Engineering
  * Chen et al [2020] Chen S, Hou Y, Cui Y, et al (2020) Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
  * Chen et al [2016] Chen X, Duan Y, Houthooft R, et al (2016) Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems
  * Chen et al [2023b] Chen X, Lin M, Schärli N, et al (2023b) Teaching large language models to self-debug. arXiv preprint arXiv:230405128
  * Chen et al [2023c] Chen Z, Singh AK, Sra M (2023c) LMExplainer: a Knowledge-Enhanced Explainer for Language Models. arXiv preprint arXiv:230316537
  * Choi et al [2021] Choi JH, Hickman KE, Monahan AB, et al (2021) Chatgpt goes to law school. Journal of Legal Educaction
  * Chuang et al [2024] Chuang YN, Wang G, Chang CY, et al (2024) Large Language Models As Faithful Explainers. arXiv preprint arXiv:240204678
  * Common Crawl Foundation [2024] Common Crawl Foundation (2024) Common Crawl. https://commoncrawl.org/, accessed: 2024-02-20
  * Conmy et al [2024] Conmy A, Mavor-Parker A, Lynch A, et al (2024) Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems
  * Creswell and Shanahan [2022] Creswell A, Shanahan M (2022) Faithful reasoning using large language models. arXiv preprint arXiv:220814271
  * Dai et al [2022] Dai D, Dong L, Hao Y, et al (2022) Knowledge neurons in pretrained transformers. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics
  * Deldjoo [2023] Deldjoo Y (2023) Fairness of chatgpt and the role of explainable-guided prompts. arXiv preprint arXiv:230711761
  * Devlin et al [2019] Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics
  * Dufter et al [2022] Dufter P, Schmitt M, Schütze H (2022) Position information in transformers: An overview. Computational Linguistics
  * Dwivedi et al [2023] Dwivedi R, Dave D, Naik H, et al (2023) Explainable AI (XAI): Core ideas, techniques, and solutions. ACM Computing Surveys
  * Elhage et al [2022] Elhage N, Hume T, Olsson C, et al (2022) Softmax linear units. Transformer Circuits Thread, URL {https://transformer-circuits.pub/2022/solu/index.html}
  * Elyoseph et al [2023] Elyoseph Z, Hadar-Shoval D, Asraf K, et al (2023) Chatgpt outperforms humans in emotional awareness evaluations. Frontiers in Psychology
  * Enguehard [2023] Enguehard J (2023) Sequential Integrated Gradients: a simple but effective method for explaining language models. arXiv preprint arXiv:230515853
  * Ethayarajh and Jurafsky [2021] Ethayarajh K, Jurafsky D (2021) Attention flows are shapley value explanations. arXiv preprint arXiv:210514652
  * European Union [2023] European Union (2023) Eu ai act. https://artificialintelligenceact.eu/, accessed: 2024-15-02
  * Faubel et al [2023] Faubel L, Woudsma T, Methnani L, et al (2023) Towards an MLOps Architecture for XAI in Industrial Applications. arXiv preprint arXiv:230912756
  * Fok and Weld [2023] Fok R, Weld DS (2023) In search of verifiability: Explanations rarely enable complementary performance in ai-advised decision making. arXiv preprint arXiv:230507722
  * Foote et al [2023] Foote A, Nanda N, Kran E, et al (2023) Neuron to graph: Interpreting language model neurons at scale. arXiv preprint arXiv:230519911
  * Gao et al [2023] Gao Y, Sheng T, Xiang Y, et al (2023) Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:230314524
  * Gawlikowski et al [2023] Gawlikowski J, Tassi CRN, Ali M, et al (2023) A survey of uncertainty in deep neural networks. Artificial Intelligence Review
  * Geiger et al [2021] Geiger A, Lu H, Icard T, et al (2021) Causal abstractions of neural networks. Advances in Neural Information Processing Systems
  * Geva et al [2022] Geva M, Caciularu A, Wang KR, et al (2022) Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:220314680
  * Ghorbani et al [2019] Ghorbani A, Abid A, Zou J (2019) Interpretation of neural networks is fragile. In: Proceedings of the AAAI conference on artificial intelligence
  * Goodfellow et al [2015] Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (ICLR)
  * Goyal et al [2022] Goyal T, Xu J, Li JJ, et al (2022) Training dynamics for text summarization models. In: Findings of the Association for Computational Linguistics
  * Gozalo-Brizuela and Garrido-Merchan [2023] Gozalo-Brizuela R, Garrido-Merchan EC (2023) ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv preprint arXiv:230104655
  * Grisold and Schneider [2023] Grisold T, Schneider J (2023) Dynamics of human-ai delegation in organizational routines. In: Proceedings of the International Conference on Information Systems
  * Grosse et al [2023] Grosse R, Bae J, Anil C, et al (2023) Studying large language model generalization with influence functions. arXiv preprint arXiv:230803296
  * Grynbaum and Mac [2023] Grynbaum MM, Mac R (2023) The times sues openai and microsoft over a.i. use of copyrighted work. https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html, accessed: 2024-02-15
  * Guidotti [2022] Guidotti R (2022) Counterfactual explanations and how to find them: literature review and benchmarking. Data Mining and Knowledge Discovery
  * Guidotti et al [2019] Guidotti R, Monreale A, Ruggieri S, et al (2019) A survey of methods for explaining black box models. ACM Computer Surveys
  * Gupta et al [2023] Gupta M, Akiri C, Aryal K, et al (2023) From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy. IEEE Access
  * Gurrapu et al [2023] Gurrapu S, Kulkarni A, Huang L, et al (2023) Rationalization for explainable nlp: A survey. Frontiers in Artificial Intelligence
  * He et al [2022] He H, Zhang H, Roth D (2022) Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:230100303
  * Hernandez et al [2023] Hernandez E, Li BZ, Andreas J (2023) Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:230400740
  * Ho et al [2020] Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems
  * Huang et al [2023a] Huang J, Chen X, Mishra S, et al (2023a) Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:231001798
  * Huang et al [2023b] Huang J, Gu SS, Hou L, et al (2023b) Large language models can self-improve. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
  * Huang et al [2023c] Huang Y, Song J, Wang Z, et al (2023c) Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:230710236
  * Jain and Wallace [2019] Jain S, Wallace BC (2019) Attention is not Explanation. In: Proceedings of NAACL-HLT
  * Jeanneret et al [2022] Jeanneret G, Simon L, Jurie F (2022) Diffusion models for counterfactual explanations. In: Proceedings of the Asian Conference on Computer Vision
  * Jin et al [2020] Jin D, Jin Z, Zhou JT, et al (2020) Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In: Proceedings of the AAAI conference on artificial intelligence
  * Kadavath et al [2022] Kadavath S, Conerly T, Askell A, et al (2022) Language models (mostly) know what they know. arXiv preprint arXiv:220705221
  * Katz et al [2024] Katz DM, Bommarito MJ, Gao S, et al (2024) Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A
  * Katz and Belinkov [2023] Katz S, Belinkov Y (2023) Interpreting Transformer’s Attention Dynamic Memory and Visualizing the Semantic Information Flow of GPT. arXiv preprint arXiv:230513417
  * Kıcıman et al [2023] Kıcıman E, Ness R, Sharma A, et al (2023) Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:230500050
  * Kim et al [2018] Kim B, Wattenberg M, Gilmer J, et al (2018) Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: International conference on machine learning
  * King and He [2005] King WR, He J (2005) Understanding the role and methods of meta-analysis in is research. Communications of the Association for Information Systems
  * Kirillov et al [2023] Kirillov A, Mintun E, Ravi N, et al (2023) Segment anything. arXiv preprint arXiv:230402643
  * Kokalj et al [2021] Kokalj E, Škrlj B, Lavrač N, et al (2021) BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation
  * Kwon et al [2022] Kwon M, Jeong J, Uh Y (2022) Diffusion models already have a semantic latent space. arXiv preprint arXiv:221010960
  * Lewis et al [2020] Lewis P, Perez E, Piktus A, et al (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems
  * Li et al [2016] Li J, Chen X, Hovy E, et al (2016) Visualizing and understanding neural models in NLP. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pp 681–691, 10.18653/v1/N16-1082, URL https://aclanthology.org/N16-1082
  * Li et al [2023a] Li L, Zhang Y, Chen L (2023a) Personalized prompt learning for explainable recommendation. ACM Transactions on Information Systems
  * Li and Zhang [2023] Li M, Zhang Q (2023) Does a neural network really encode symbolic concepts? In: International Conference on Machine Learning, PMLR, pp 20452–20469
  * Li et al [2023b] Li P, Pei Y, Li J (2023b) A comprehensive survey on design and application of autoencoder in deep learning. Applied Soft Computing
  * Liao and Vaughan [2023] Liao QV, Vaughan JW (2023) AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap. arXiv preprint arXiv:230601941
  * Lin et al [2023] Lin CH, Gao J, Tang L, et al (2023) Magic3d: High-resolution text-to-3d content creation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
  * Lin et al [2022] Lin T, Wang Y, Liu X, et al (2022) A survey of transformers. AI Open
  * Ling et al [2023] Ling C, Zhao X, Lu J, et al (2023) Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:230518703
  * Liu et al [2021] Liu LZ, Wang Y, Kasai J, et al (2021) Probing across time: What does RoBERTa know and when? In: Findings of the Association for Computational Linguistics: EMNLP 2021
  * Liu et al [2022] Liu T, Guo Q, Hu X, et al (2022) RLET: A reinforcement learning based approach for explainable QA with entailment trees. arXiv preprint arXiv:221017095
  * Longo et al [2024] Longo L, Brcic M, Cabitza F, et al (2024) Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary research directions. Information Fusion
  * Longpre et al [2023] Longpre S, Hou L, Vu T, et al (2023) The flan collection: Designing data and methods for effective instruction tuning. In: International Conference on Machine Learning
  * Lou et al [2023] Lou R, Zhang K, Yin W (2023) Is prompt all you need? no. A comprehensive and broader view of instruction learning. arXiv preprint arXiv:230310475
  * Lundberg and Lee [2017] Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Advances in neural information processing systems
  * Lundstrom et al [2022] Lundstrom DD, Huang T, Razaviyayn M (2022) A rigorous study of integrated gradients method and extensions to internal neuron attributions. In: International Conference on Machine Learning
  * Luo and Specia [2024] Luo H, Specia L (2024) From Understanding to Utilization: A Survey on Explainability for Large Language Models. arXiv preprint arXiv:240112874
  * Lyu et al [2024] Lyu Q, Apidianaki M, Callison-Burch C (2024) Towards faithful model explanation in nlp: A survey. Computational Linguistics
  * Ma et al [2023] Ma Y, Liu Y, Yu Y, et al (2023) At which training stage does code data help llms reasoning? arXiv preprint arXiv:230916298
  * MacKenzie [2024] MacKenzie IS (2024) Human-computer interaction: An empirical research perspective, 2nd edn. Morgan Kaufmann
  * Mao et al [2023] Mao J, Ye J, Qian Y, et al (2023) A language agent for autonomous driving. arXiv preprint arXiv:231110813
  * Marvin and Linzen [2018] Marvin R, Linzen T (2018) Targeted syntactic evaluation of language models. arXiv preprint arXiv:180809031
  * Maynez et al [2020] Maynez J, Narayan S, Bohnet B, et al (2020) On faithfulness and factuality in abstractive summarization. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics
  * McKinsey & Company [2023] McKinsey & Company (2023) The economic potential of generative ai: The next productivity frontier. https://www.mckinsey.com/featured-insights/mckinsey-live/webinars/the-economic-potential-of-generative-ai-the-next-productivity-frontier, accessed: 2024-02-13
  * Meng et al [2022] Meng K, Bau D, Andonian A, et al (2022) Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems
  * Menick et al [2022] Menick J, Trebacz M, Mikulik V, et al (2022) Teaching language models to support answers with verified quotes. arXiv preprint arXiv:220311147
  * Meronen et al [2024] Meronen L, Trapp M, Pilzer A, et al (2024) Fixing overconfidence in dynamic neural networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
  * Meske et al [2022] Meske C, Bunde E, Schneider J, et al (2022) Explainable artificial intelligence: objectives, stakeholders, and future research opportunities. Information Systems Management
  * Miller [2019] Miller T (2019) Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence
  * Minh et al [2022] Minh D, Wang HX, Li YF, et al (2022) Explainable artificial intelligence: a comprehensive review. Artificial Intelligence Review
  * Mishra et al [2023] Mishra A, Soni U, Arunkumar A, et al (2023) PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. arXiv preprint arXiv:230401964
  * Modarressi et al [2023] Modarressi A, Fayyaz M, Aghazadeh E, et al (2023) DecompX: Explaining Transformers Decisions by Propagating Token Decomposition. arXiv preprint arXiv:230602873
  * Mohebbi et al [2021] Mohebbi H, Modarressi A, Pilehvar MT (2021) Exploring the role of BERT token representations to explain sentence probing results. arXiv preprint arXiv:210401477
  * Molnar [2020] Molnar C (2020) Interpretable Machine Learning. URL https://christophm.github.io/interpretable-ml-book/
  * Montavon et al [2019] Montavon G, Binder A, Lapuschkin S, et al (2019) Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning
  * Mueller et al [2021] Mueller ST, Veinott ES, Hoffman RR, et al (2021) Principles of explanation in human-ai systems. arXiv preprint arXiv:210204972
  * Nichol et al [2022] Nichol AQ, Dhariwal P, Ramesh A, et al (2022) Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: Proceedings of the International Conference on Machine Learning
  * Nickerson et al [2013] Nickerson RC, Varshney U, Muntermann J (2013) A method for taxonomy development and its application in information systems. European Journal of Information Systems
  * Olah [2022] Olah C (2022) Mechanistic interpretability, variables, and the importance of interpretable bases. URL https://www.transformer-circuits.pub/2022/mech-interp-essay, accessed: 2024-02-15
  * Olah et al [2020] Olah C, Cammarata N, Schubert L, et al (2020) Zoom in: An introduction to circuits. Distill (3)
  * Olsson et al [2022] Olsson C, Elhage N, Nanda N, et al (2022) In-context learning and induction heads. arXiv preprint arXiv:220911895
  * OpenAI [2023a] OpenAI (2023a) Introducing the gpt store. https://openai.com/blog/introducing-the-gpt-store, accessed: 2024-15-02
  * OpenAI [2023b] OpenAI (2023b) Language models can explain neurons in language models. https://openai.com/research/language-models-can-explain-neurons-in-language-models?s=09, accessed: 2024-15-02
  * Ouyang et al [2022] Ouyang L, Wu J, Jiang X, et al (2022) Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems
  * Pan et al [2023] Pan L, Saxon M, Xu W, et al (2023) Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:230803188
  * Park et al [2018] Park DH, Hendricks LA, Akata Z, et al (2018) Multimodal explanations: Justifying decisions and pointing to the evidence. In: Proceedings of the IEEE conference on computer vision and pattern recognition
  * Poli et al [2023] Poli M, Massaroli S, Nguyen E, et al (2023) Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:230210866
  * Porter [2023] Porter J (2023) Chatgpt continues to be one of the fastest-growing services ever. https://www.theverge.com/2023/11/6/23948386/chatgpt-active-user-count-openai-developer-conference, accessed: 2024-02-19
  * Radford et al [2019] Radford A, Wu J, Child R, et al (2019) Language models are unsupervised multitask learners. OpenAI blog
  * Rago et al [2021] Rago A, Cocarascu O, Bechlivanidis C, et al (2021) Argumentative explanations for interactive recommendations. Artificial Intelligence
  * Ram et al [2022] Ram O, Bezalel L, Zicher A, et al (2022) What are you token about? dense retrieval as distributions over the vocabulary. arXiv preprint arXiv:221210380
  * Ramesh et al [2022] Ramesh A, Dhariwal P, Nichol A, et al (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220406125
  * Räuker et al [2023] Räuker T, Ho A, Casper S, et al (2023) Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In: IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)
  * Reed et al [2022] Reed S, Zolna K, Parisotto E, et al (2022) A generalist agent. arXiv preprint arXiv:220506175
  * Reid et al [2024] Reid M, Savinov N, Teplyashin D, et al (2024) Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:240305530
  * Ribeiro et al [2016] Ribeiro MT, Singh S, Guestrin C (2016) ”Why should i trust you?” Explaining the predictions of any classifier. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining
  * Ross et al [2017] Ross AS, Hughes MC, Doshi-Velez F (2017) Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:170303717
  * Sabour et al [2017] Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. Advances in neural information processing systems 30
  * Saeed and Omlin [2023] Saeed W, Omlin C (2023) Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems
  * Saha et al [2022] Saha S, Hase P, Rajani N, et al (2022) Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
  * Saharia et al [2022] Saharia C, Chan W, Saxena S, et al (2022) Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems
  * de Santana Correia and Colombini [2022] de Santana Correia A, Colombini EL (2022) Attention, please! a survey of neural attention models in deep learning. Artificial Intelligence Review
  * Schick et al [2024] Schick T, Dwivedi-Yu J, Dessì R, et al (2024) Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems
  * Schneider [2022] Schneider J (2022) Optimizing human hand gestures for ai-systems. AI Communications
  * Schneider and Apruzzese [2023] Schneider J, Apruzzese G (2023) Dual adversarial attacks: Fooling humans and classifiers. Journal of Information Security and Applications
  * Schneider and Breitinger [2023] Schneider J, Breitinger F (2023) Towards ai forensics: Did the artificial intelligence system do it? Journal of Information Security and Applications
  * Schneider and Handali [2019] Schneider J, Handali J (2019) Personalized explanation in machine learning: A conceptualization. In: Proceedings of the European Conference on Information Systems (ECIS)
  * Schneider and Vlachos [2023a] Schneider J, Vlachos M (2023a) Explaining classifiers by constructing familiar concepts. Machine Learning
  * Schneider and Vlachos [2023b] Schneider J, Vlachos M (2023b) Reflective-net: Learning from explanations. Data Mining and Knowledge Discovery
  * Schneider et al [2023a] Schneider J, Abraham R, Meske C, et al (2023a) Artificial intelligence governance for businesses. Information Systems Management 40(3)
  * Schneider et al [2023b] Schneider J, Haag S, Kruse LC (2023b) Negotiating with llms: Prompt hacks, skill gaps, and reasoning deficits. arXiv preprint arXiv:231203720
  * Schneider et al [2023c] Schneider J, Meske C, Vlachos M (2023c) Deceptive xai: Typology, creation and detection. SN Computer Science
  * Schneider et al [2024a] Schneider J, Abraham R, Meske C (2024a) Governance of generative artificial intelligence for companies. arXiv preprint arXiv:240308802
  * Schneider et al [2024b] Schneider J, Meske C, Kuss P (2024b) Foundation models: A new paradigm for artificial intelligence. Business & Information Systems Engineering
  * Schramowski et al [2020] Schramowski P, Stammer W, Teso S, et al (2020) Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence
  * Schwalbe and Finzel [2023] Schwalbe G, Finzel B (2023) A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining and Knowledge Discovery
  * Selva et al [2023] Selva J, Johansen AS, Escalera S, et al (2023) Video transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence
  * Selvaraju et al [2017] Selvaraju RR, Cogswell M, Das A, et al (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision
  * Serrano and Smith [2019] Serrano S, Smith NA (2019) Is Attention Interpretable? In: Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics
  * Shahsavar and Choudhury [2023] Shahsavar Y, Choudhury A (2023) User intentions to use chatgpt for self-diagnosis and health-related purposes: Cross-sectional survey study. JMIR Human Factors
  * Shen et al [2023] Shen T, Jin R, Huang Y, et al (2023) Large language model alignment: A survey. arXiv preprint arXiv:230915025
  * Sikdar et al [2021] Sikdar S, Bhattacharya P, Heese K (2021) Integrated directional gradients: Feature interaction attribution for neural NLP models. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing
  * Silva et al [2023] Silva A, Schrum M, Hedlund-Botti E, et al (2023) Explainable artificial intelligence: Evaluating the objective and subjective impacts of xai on human-agent interaction. International Journal of Human–Computer Interaction
  * Singh et al [2022] Singh C, Morris JX, Aneja J, et al (2022) Explaining patterns in data with language models via interpretable autoprompting. arXiv preprint arXiv:221001848
  * Singh et al [2023] Singh C, Hsu AR, Antonello R, et al (2023) Explaining black box text modules in natural language with language models. arXiv preprint arXiv:230509863
  * Singh et al [2024] Singh C, Inala JP, Galley M, et al (2024) Rethinking Interpretability in the Era of Large Language Models. arXiv preprint arXiv:240201761
  * Singhal et al [2023] Singhal K, Azizi S, Tu T, et al (2023) Large language models encode clinical knowledge. Nature
  * Slack et al [2023] Slack D, Krishna S, Lakkaraju H, et al (2023) Explaining machine learning models with interactive natural language conversations using TalkToModel. Nature Machine Intelligence
  * Sobania et al [2023] Sobania D, Briesch M, Hanna C, et al (2023) An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:230108653
  * Sottana et al [2023] Sottana A, Liang B, Zou K, et al (2023) Evaluation metrics in the era of gpt-4: Reliably evaluating large language models on sequence to sequence tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
  * Speith [2022] Speith T (2022) A review of taxonomies of explainable artificial intelligence (XAI) methods. In: Proceedings of the ACM Conference on Fairness, Accountability, and Transparency
  * Sreedharan et al [2022] Sreedharan S, Kulkarni A, Kambhampati S (2022) Explainable Human-AI Interaction: A Planning Perspective. Springer Nature
  * Stremmel et al [2022] Stremmel J, Hill BL, Hertzberg J, et al (2022) Extend and explain: Interpreting very long language models. In: Machine Learning for Health
  * Sun et al [2024] Sun A, Ma P, Yuan Y, et al (2024) Explain any concept: Segment anything meets concept-based explanation. Advances in Neural Information Processing Systems
  * Taori et al [2023] Taori R, Gulrajani I, Zhang T, et al (2023) Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca
  * Teehan et al [2022] Teehan R, Clinciu M, Serikov O, et al (2022) Emergent structures and training dynamics in large language models. In: Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models
  * Tenney et al [2019] Tenney I, Xia P, Chen B, et al (2019) What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:190506316
  * The Guardian [2023] The Guardian (2023) Elon musk calls ai one of the biggest threats to humanity at summit. https://www.theguardian.com/technology/2023/nov/01/elon-musk-calls-ai-one-of-the-biggest-threats-to-humanity-at-summit, accessed: 2024-02-26
  * Theis et al [2023] Theis S, Jentzsch S, Deligiannaki F, et al (2023) Requirements for explainability and acceptance of artificial intelligence in collaborative work. In: International Conference on Human-Computer Interaction
  * Theissler et al [2022] Theissler A, Spinnato F, Schlegel U, et al (2022) Explainable AI for time series classification: A review, taxonomy and research directions. IEEE Access
  * Thirunavukarasu et al [2023] Thirunavukarasu AJ, Ting DSJ, Elangovan K, et al (2023) Large language models in medicine. Nature Medicine
  * Touvron et al [2023] Touvron H, Martin L, Stone K, et al (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230709288
  * Turpin et al [2024] Turpin M, Michael J, Perez E, et al (2024) Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems
  * Vaswani et al [2017] Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in neural information processing systems
  * Vedula et al [2023] Vedula N, Collins M, Agichtein E, et al (2023) Generating explainable product comparisons for online shopping. In: Proceedings of the ACM International Conference on Web Search and Data Mining
  * Vig [2019] Vig J (2019) A Multiscale Visualization of Attention in the Transformer Model. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics: System Demonstrations
  * Wang et al [2022a] Wang B, Xu C, Liu X, et al (2022a) Semattack: natural textual attacks via different semantic spaces. arXiv preprint arXiv:220501287
  * Wang and Shu [2023] Wang H, Shu K (2023) Explainable claim verification via knowledge-grounded reasoning with large language models. arXiv preprint arXiv:231005253
  * Wang et al [2022b] Wang K, Variengien A, Conmy A, et al (2022b) Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:221100593
  * Wang et al [2023] Wang Z, Cai S, Chen G, et al (2023) Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:230201560
  * Webster and Watson [2002] Webster J, Watson RT (2002) Analyzing the past to prepare for the future: Writing a literature review. MIS quarterly
  * Wei et al [2022] Wei J, Wang X, Schuurmans D, et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems
  * Weidinger et al [2022] Weidinger L, Uesato J, Rauh M, et al (2022) Taxonomy of risks posed by language models. In: Proceedings of the ACM Conference on Fairness, Accountability, and Transparency
  * White et al [2023] White J, Fu Q, Hays S, et al (2023) A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:230211382
  * Wu et al [2021] Wu T, Ribeiro MT, Heer J, et al (2021) Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. arXiv preprint arXiv:210100288
  * Wu et al [2020] Wu Z, Chen Y, Kao B, et al (2020) Perturbed masking: Parameter-free probing for analyzing and interpreting bert. arXiv preprint arXiv:200414786
  * Wu et al [2023] Wu Z, Qiu L, Ross A, et al (2023) Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:230702477
  * Xing et al [2023] Xing Z, Feng Q, Chen H, et al (2023) A survey on video diffusion models. arXiv preprint arXiv:231010647
  * Xu et al [2023] Xu P, Zhu X, Clifton DA (2023) Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence
  * Yang et al [2023a] Yang K, Ji S, Zhang T, et al (2023a) Towards interpretable mental health analysis with large language models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
  * Yang et al [2023b] Yang L, Zhang Z, Song Y, et al (2023b) Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys
  * Yang et al [2023c] Yang S, Huang S, Zou W, et al (2023c) Local interpretation of transformer based on linear decomposition. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
  * Ye and Durrett [2022] Ye X, Durrett G (2022) The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems
  * Yin and Neubig [2022] Yin K, Neubig G (2022) Interpreting language models with contrastive explanations. arXiv preprint arXiv:220210419
  * Yordanov et al [2021] Yordanov Y, Kocijan V, Lukasiewicz T, et al (2021) Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup. arXiv preprint arXiv:211206204
  * Zaidan et al [2007] Zaidan O, Eisner J, Piatko C (2007) Using “annotator rationales” to improve machine learning for text categorization. In: Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference
  * Zamfirescu-Pereira et al [2023] Zamfirescu-Pereira J, Wong RY, Hartmann B, et al (2023) Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In: Proceedings of the CHI Conference on Human Factors in Computing Systems
  * Zhang et al [2023a] Zhang C, Zhang C, Zhang M, et al (2023a) Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:230307909
  * Zhang et al [2023b] Zhang C, Zhang C, Zheng S, et al (2023b) A survey on audio diffusion models: Text to speech synthesis and enhancement in generative ai. arXiv preprint arXiv:230313336
  * Zhang et al [2024] Zhang N, Yao Y, Tian B, et al (2024) A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:240101286
  * Zhang et al [2023c] Zhang S, Dong L, Li X, et al (2023c) Instruction tuning for large language models: A survey. arXiv preprint arXiv:230810792
  * Zhao et al [2023a] Zhao H, Chen H, Yang F, et al (2023a) Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology
  * Zhao et al [2023b] Zhao R, Joty S, Wang Y, et al (2023b) Explaining Language Models’ Predictions with High-Impact Concepts. arXiv preprint arXiv:230502160
  * Zhong et al [2021] Zhong Z, Friedman D, Chen D (2021) Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:210405240
  * Zhou et al [2023] Zhou Y, Zhang Y, Tan C (2023) FLamE: Few-shot Learning from Natural Language Explanations. arXiv preprint arXiv:230608042
  * Ziems et al [2023] Ziems C, Held W, Shaikh O, et al (2023) Can Large Language Models Transform Computational Social Science? arXiv preprint arXiv:230503514
  * Zini and Awad [2022] Zini JE, Awad M (2022) On the explainability of natural language processing deep models. ACM Computing Surveys