# A Practical Review of Mechanistic Interpretability for Transformer-Based
Language Models

Daking Rai1, Yilun Zhou2, Shi Feng3,4, Abulhair Saparov3,5, Ziyu Yao1
1George Mason University, 2Salesforce Research,
3New York University, 4George Washington University, 5Purdue University
{drai2<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/Dakingrai/awesome-mechanistic-interpretability-lm-papers

###### Abstract

Mechanistic interpretability (MI) is an emerging sub-field of interpretability
that seeks to understand a neural network model by reverse-engineering its
internal computations. Recently, MI has garnered significant attention for
interpreting transformer-based language models (LMs), resulting in many novel
insights yet introducing new challenges. However, there has not been work that
comprehensively reviews these insights and challenges, particularly as a guide
for newcomers to this field. To fill this gap, we present a comprehensive
survey outlining fundamental objects of study in MI, techniques that have been
used for its investigation, approaches for evaluating MI results, and
significant findings and applications stemming from the use of MI to
understand LMs. In particular, we present a roadmap for beginners to navigate
the field and leverage MI for their benefit. Finally, we also identify current
gaps in the field and discuss potential future directions.

A Practical Review of Mechanistic Interpretability for Transformer-Based
Language Models

Daking Rai1, Yilun Zhou2, Shi Feng3,4, Abulhair Saparov3,5, Ziyu Yao1 1George
Mason University, 2Salesforce Research, 3New York University, 4George
Washington University, 5Purdue University {drai2<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>https://github.com/Dakingrai/awesome-mechanistic-interpretability-lm-papers

##  1 Introduction

In recent years, transformer-based language models (LMs) have achieved
remarkable success in a wide range of tasks Radford et al. (2019); Brown et
al. (2020); Achiam et al. (2023); Touvron et al. (2023); Bubeck et al. (2023).
Alongside these advancements, there are growing concerns over the safety,
alignment, reliability, and robustness of their usage and development Chang et
al. (2024); Yao et al. (2024); Weidinger et al. (2022), especially as they are
increasingly implemented in real-world applications. These concerns primarily
stem from our limited understanding of these LMs and the difficulty in
interpreting their behavior.

Recently, mechanistic interpretability (MI) has emerged as a promising
technique that fills the gap in the research field of interpretability. This
is a line of methods that interpret a model by reverse-engineering the
underlying computation into human-understandable mechanisms Olah et al.
(2020); Elhage et al. (2021). It has shown promise in providing insights into
the functions of LM components (e.g., neurons, attention heads), offering
mechanistic explanations for various LM behaviors, and enabling users to
leverage the explanations to enhance an LM’s utilization Wang et al. (2022a);
Marks et al. (2024); Templeton et al. (2024). Despite the promise, however,
there are concerns on the scalability and generalizability of MI and its
helpfulness in addressing critical problems in AI safety Räuker et al. (2023);
Casper (2023).

Observing these promises and challenges, we aim to provide a comprehensive
review of MI in its applications to interpret transformer-based LMs. A
_taxonomy_ is present in Figure 1. There exist surveys on relevant topics, but
they differ from ours in a number of key aspects. For example, while Räuker et
al. (2023) discussed interpretability approaches broadly for all types of deep
neural networks, we target an in-depth review of MI for transformer-based LMs.
Similarly, Zhao et al. (2024) presented a broad survey of LM interpretability
but did not focus on MI. Concurrent with our work, Ferrando et al. (2024)
reviewed techniques developed for understanding the inner workings of
transformer-based LMs but did not particularly focus on MI, and Bereska and
Gavves (2024) emphasized MI advances for AI safety.

Our paper is organized as follows. We first present the background knowledge
needed to understand this survey (Section 2). Then, we review the key
developments in MI for LMs, including the fundamental objects of study,
techniques, and evaluation (Sections 3-5). We then particularly present a
_beginner’s roadmap_ in Section 6, which summarizes actionable items for
newcomers to use MI for their benefits. This is then followed by an overview
of findings and applications of MI in Section 7. Finally, we conclude the
survey by discussing challenges and future work in this field.

Probing  Automated Feature Explanation  Sparse Autoencoder (SAE)  Logit Lens
Visualization  Causal Mediation Analysis (CMA)  Knockout  Zero, Mean,
Resampling Ablation, Causal Scrubbing Activation, Path, Attribution Patching,
ACDC, EAP  Techniques Features  Universality  Circuits  Transformer Components
(RS, MHA, FF)  Fundamental Objects Model Capabilities Learning Dynamics
Polysemanticity  Superposition  LM Behavior  LM Components  Applications  AI
Safety  LM Enhancement  LM Generation Steering  Knowledge Editing  Findings
Intrinsic Evaluation  Extrinsic Evaluation  Faithfulness Completeness
Minimality Plausibility Evaluation

Figure 1: Taxonomy of the survey. Blocks are clickable to be directed to the
corresponding sections. Figure 2: Architecture of transformer-based LMs.

##  2 Background: Transformer-based LM

A transformer-based LM Vaswani et al. (2017) $M$ takes input tokens
$X=(x_{1},...,x_{n})$ and outputs a vector in $\mathbb{R}^{|\mathcal{V}|}$, a
probability distribution over the vocabulary $\mathcal{V}$, to predict the
next token $x_{n+1}$. The model refines the representation of each token
$x_{i}$ layer by layer (Figure 2). In the first layer, $h_{i}^{0}$ is an
embedding vector of $x_{i}$, resulting from a lookup operation in an embedding
matrix $W_{E}\in\mathbb{R}^{|\mathcal{V}|\times d}$.111For brevity, we omit
components such as position embedding and layer normalization in transformer,
as they will not affect our discussion of MI. Readers should refer to Vaswani
et al. (2017) for a complete description. This representation is then updated
layer-by-layer through the calculations of _multi-head attention (MHA)_ and
_feed-forward (FF)_ sublayers in each layer, i.e.,

$h_{i}^{l}=h_{i}^{l-1}+a_{i}^{l}+f_{i}^{l},$ (1)

where $h_{i}^{l}$ denotes the representation of token $x_{i}$ at layer $l$,
$a_{i}^{l}$ is the attention output from the MHA sublayer, and $f_{i}^{l}$ is
the output from the FF sublayer. The sequence of $h_{i}^{l}$ across the layers
is also referred to as the _residual stream_ (RS) of transformer in literature
Elhage et al. (2021).

Briefly, the MHA sublayer with $H$ attention heads is implemented via

$\displaystyle a_{i}^{l}$
$\displaystyle=\texttt{concat}(a_{i,0}^{l},..,a_{i,H}^{l})W_{O}^{l},$ (2)
$\displaystyle a_{i,h}^{l}$
$\displaystyle=\texttt{softmax}\Bigg{(}\frac{(h_{i}^{l-1}W_{Q}^{l,h})(h_{\leq
i}^{l-1}W_{K}^{l,h})^{\top}}{\sqrt{d_{k}}}\Bigg{)}$ $\displaystyle\hskip
25.60747pt\cdot(h_{\leq i}^{l-1}W_{V}^{l,h}),$ (3)

where $a_{i,h}^{l}$ is the attention output from the $h$-th head,
$W_{Q}^{l,h},W_{K}^{l,h},W_{V}^{l,h}$ are the query, key, and value (learned)
projection matrices, respectively, and $W_{O}^{l}$ projects the concatenated
attention outputs from all heads to the model dimension $d$.

The FF sublayer then performs two linear transformations over $a_{i}^{l}$ with
an element-wise non-linear function $\sigma$ between them, i.e.,

$f_{i}^{l}=W_{v}^{l}\sigma(W_{k}^{l}a_{i}^{l}+b^{l}_{k})+b^{l}_{v},$ (4)

where $W_{v}^{l}$, $W_{k}^{l}$, $b^{l}_{k}$, and $b^{l}_{v}$ are learned
parameter matrices and biases. Finally, the RS of $x_{n}$ at the final layer,
$h_{n}^{L}$, is projected into a probability distribution over $\mathcal{V}$
by applying an _unembedding_ matrix
$W_{U}\in\mathbb{R}^{d\times|\mathcal{V}|}$ and a softmax operation.

Figure 3: An example of an induction circuit discovered by Elhage et al.
(2021) that consists of outputs of two attention heads (previous token head
and induction head) as nodes and connection between them as edges.

##  3 Fundamental Objects of Study

MI is a bottom-up approach that interprets LMs by decomposing them into
smaller components and more elementary computations. Following Olah et al.
(2020), one of the earliest studies in MI, we categorize research of MI into
three areas: the study of features, circuits, and their universality.

###  3.1 Features

A _feature_ is a human-interpretable input property that is encoded in a
model’s activation.222We use “activations” and “representations”
interchangeably to refer to the intermediate outputs of LM components. For
instance, a neuron or a set of neurons that consistently activates for French
text can be interpreted as a “French text detector” feature Gurnee et al.
(2023).333We refer to each element in the activation as a neuron, which
implies that a neuron is a basis-aligned direction in the activation space
Marks et al. (2024). MI aims to interpret LMs in terms of the independent
features or components representing these features. To this end, Elhage et al.
(2022b) proposed the _linear representation hypothesis_ , which posits that
neural networks have two properties – _linearity_ , i.e. the network’s
activation space consists of meaningful (linear) vectors, each representing a
feature, and _decomposability_ , i.e., network activations can be decomposed
and described in terms of these independent features. This hypothesis
resonates with earlier research that also shows linearity in word embeddings
(e.g.
$\overrightarrow{man\vphantom{k}}-\overrightarrow{woman\vphantom{k}}\approx\overrightarrow{king}-\overrightarrow{queen\vphantom{k}}$)
Mikolov et al. (2013). In Section 7.1, we will review some identified features
in LMs and how they are represented.

###  3.2 Circuits

While the study of features helps us to understand what information is encoded
in a model’s activations, it does not inform us of how these features are
extracted from the input and then processed to implement specific LM behaviors
(e.g., reasoning). MI fills this gap by studying “circuits”, which are
meaningful computational pathways that connect features.

More formally, if we view an LM $M$ as a computational graph with features as
nodes and the weighted connections between them as edges, a circuit is a sub-
graph of $M$ responsible for implementing specific LM behaviors Wang et al.
(2022a). Additionally, although circuits were initially defined as connections
between features Olah et al. (2020), subsequent studies have generalized them
as connections between the activation outputs of transformer components Olsson
et al. (2022); Wang et al. (2022a). Therefore, we include research on
interpreting transformer components, both individually (e.g., interpreting RS
in Eq 1 connecting the activations of MHA, FF, and the RS from earlier layers)
and across multiple components (e.g., the induction circuit in Figure 3), as
circuit study.

An example circuit discovered by Elhage et al. (2021) in a toy LM is shown in
Figure 3. This is an induction circuit consisting of outputs of two attention
heads (previous token head and induction head) as nodes and connection between
them as edges of the circuit. The circuit implements the task of detecting and
continuing repeated subsequences in the input (e.g., _Mr D urs ley was thin
and bold. Mr D - > urs_), where the previous token head encodes the
information _“urs” follows the “D” token_ in the RS, which is then read by the
induction head to promote _“urs”_ as the next token prediction.

###  3.3 Universality

For any feature or circuit that we have identified in an LM in one task, the
critical question arises: Do they similarly exist in other LMs or in other
tasks? The investigation into this question has then given rise to the notion
of _universality_ , i.e., the extent to which similar features and circuits
are formed across different LMs and tasks Olah et al. (2020); Gurnee et al.
(2024). The implications of universal features and circuits can be
significant. For instance, many studies Olsson et al. (2022); Elhage et al.
(2022a, b) on features and circuits were performed with only toy or small LMs.
If these features and circuits are universal, the insights from these studies
can be transferred to other unexamined LMs and potentially state-of-the-art
large LMs (LLMs). However, if they are not universal, then a significant
amount of independent effort will be required to interpret each LM.

##  4 Techniques

Next, we review the techniques that have been developed to study the
fundamental objects described in Section 3 and to perform MI analysis on LMs.

###  4.1 Logit Lens

Figure 4: Logit lens implementation at (1) RS, (2) attention head, and (3) FF
sublayer.

The _logit lens_ (Figure 4) was first introduced by nostalgebraist (2020),
which provides insights on how LMs refine their prediction across layers. This
approach has been applied to interpret _activations_ in both feature Geva et
al. (2021) and circuit discovery Lieberum et al. (2023); Wang et al. (2022a).
Specifically, the activation $h_{i}^{l}$ in each layer $l$ can be projected to
the vocabulary space by multiplying it with the unembedding matrix $W_{U}$,
yielding the logits over the vocabulary $\mathcal{V}$. In doing so, tokens
with the highest logits can be used to infer what information is encoded in
$h_{i}^{l}$.

The logit lens can also be used to interpret the _weights_ of transformer
components. For example, Geva et al. (2022) used it to understand the role of
the FF sublayer in the LM’s prediction by projecting the columns of
$W_{v}^{l}$ parameter matrix to the vocabulary space. Dar et al. (2023)
applied the technique to project query-key ($W_{Q}^{l,h}W_{K}^{l,h}$) and
value-output ($W_{V}^{l,h}W_{O}^{l}$) interaction parameter matrices to
vocabulary space to study how attention heads transfer and mix information
from source tokens to the target token.

However, vocabulary space projection may yield seemingly nonsensical results
for some models. For instance, the top-1 projected token for BLOOM (Scao et
al., 2022) in many intermediate layers is often the input token Belrose et al.
(2023). The authors posited that this may be caused by the discrepancy between
the intermediate layers and the last layer (where $W_{U}$ was trained to apply
to).To address it, Belrose et al. (2023); Din et al. (2023); Pal et al. (2023)
transformed the intermediate activations to align with the representation
space of the final layer.

###  4.2 Probing

The probing technique (Figure 5) is used extensively in (and also before) MI
to investigate whether specific information, such as the part-of-speech
linguistic property, is encoded in given intermediate activations (a single
neuron or set of neurons) Conneau et al. (2018); Tenney et al. (2019a, b). To
do so, it trains a shallow (or linear) classifier, known as a _probe_ , to
predict whether a feature is present in that activations Gurnee et al. (2023);
Dalvi et al. (2019); Durrani et al. (2020); Antverg and Belinkov (2021).
However, it is important to note that the results of probing analyses only
indicate a correlation, not a causal relation, between the feature and
activations.

Figure 5: Probing on RS to detect whether it encodes a “French text” feature.

###  4.3 Sparse Autoencoder (SAE)

SAEs (Figure 6) serve as an unsupervised technique for discovering features
from activations, especially those that demonstrate _superposition_ , a
phenomenon in LMs where their $d$-dimensional representation encodes more than
$d$ features Elhage et al. (2022b); Sharkey et al. (2023); Cunningham et al.
(2023); Bricken et al. (2023); Yun et al. (2021). In contrast to
dimensionality reduction techniques (e.g., Principal Component Analysis), SAEs
seek to embed the activation vectors into a much higher-dimensional space, but
with strong sparsity. Specifically, an encoder maps the $d$-dimensional input
into an $s$-dimensional vector ($s>d$), which the decoder then maps back to
the $d$-dimension. The encoder and decoder are jointly trained for input
reconstruction and sparsity of the $s$-dimensional representation. This
$s$-dimensional representation, owing to its sparsity, makes the discovery of
independent (or _monosemantic_) features more easily.

Figure 6: Sparse Autoencoder (SAE) applied to activation on RS.

###  4.4 Visualization

Visualization (Figure 7) is employed across various stages of an MI
investigation, from generating initial hypotheses to refining them, conducting
qualitative analyses, and validating results. For instance, attention patterns
are often visualized to understand attention heads Lieberum et al. (2023);
Olsson et al. (2022); a neuron activation across the input text is visualized
to identify its functionality Elhage et al. (2022a); Bricken et al. (2023).
While visualization can be highly useful, it requires human effort to
interpret results and carries the risks of overgeneralization. Thus, any
claims need to be substantiated with further experimentation and analysis.

Figure 7: Attention visualization, created using Cooney and Nanda (2023).

###  4.5 Automated Feature Explanation

For feature discovery, human effort is required to annotate neurons with
interpretable feature labels based on their activation patterns Elhage et al.
(2022a); Conmy et al. (2024) or the projected tokens when using logits lens
Geva et al. (2022). To reduce human labor, Bills et al. (2023) proposed using
LLMs to generate feature labels automatically (Figure 8). Additionally, they
also introduced a quantitative automatic explanation score to measure the
quality of these explanations, using LLMs to simulate activations based on the
automatically generated labels and comparing them with the ground-truth
activations.

Figure 8: Automated feature explanation, where the highlighted text represents
the activation strength for a neuron. An LLM can then be used to generate a
textual explanation of the neuron.

###  4.6 Knockout / Ablation

_Knockout_ or _ablation_ (Figure 9) is primarily used to identify components
in a circuit that are important to a certain LM behavior. It removes specific
components and analyzes the change in the LM behavior, where a significant
change suggests importance. There are three main ablation methods: (1) zero:
replacing the output of the component with a zero vector Olsson et al. (2022);
(2) mean: replacing the output with the mean value of randomly sampled inputs
from the same input distribution Wang et al. (2022a); (3) resampling:
replacing the output with that of another random input Chan et al. (2022).

Chan et al. (2022) introduced causal scrubbing which employs resampling
ablation for hypothesis verification. Rather than a direct ablation, they
formalize a hypothesis as $(G,I,c)$, where $G$ is the model’s computational
graph, $I$ is an interpretation graph reflecting the hypothesis, and $c$ is a
function that maps between nodes of $I$ to $G$. Resampling ablation is then
performed on $G$ to validate $I$. The model’s behavior should be preserved
when one ablates activations in $G$ with random samples that follow the same
hypothesis.

Figure 9: An example of ablation, where the output activation from an FF
sublayer is replaced with a zero/mean/resampled vector.

###  4.7 Causal Mediation Analysis (CMA)

Figure 10: (1) Activation Patching, where in the counterfactual run, an
intermediate RS activation is patched (i.e. replaced) by the corresponding
activation from the clean run. (2) Path Patching, where all intermediate
activations between the green highlighted attention head and FF sublayer are
patched by the corresponding activations from the clean run.

Causal mediation analysis (CMA) is popular for circuit discovery, including
two main patching approaches (Figure 10).

Activation patching localizes important _components_ in a circuit Vig et al.
(2020); Meng et al. (2022), which performs three rounds of model inference.
First, the clean run uses a prompt that demonstrates the LM capability in
investigation. Then, the corrupt run uses the same prompt, but with a few
important tokens corrupted with noise such that the LM fails to demonstrate
the capability.444Alternatively, counterfactual prompts can be used as the
corruption procedure Wang et al. (2022a). Finally, the patch run uses all
activation values of the corrupt run, except those of the localized component,
which uses the clean run values. The recovery of the lost capability indicates
that the patched component is important.

Path patching localizes important _connections_ between components Wang et al.
(2022a); Goldowsky-Dill et al. (2023). To assess whether the connection
between two components, $C_{1}$ and $C_{2}$, is significant, path patching
applies activation patching to the output of $C_{1}$ but only along paths
serve as the input to $C_{2}$. If a change in behavior is observed, then the
connection between the two components is considered important.

Circuit discovery via patching is often computationally inefficient, as one
has to iteratively patch each component or connection in an LM to measure its
importance. To address it, Conmy et al. (2024) proposed ACDC (Automatic
Circuit DisCovery) to automate the iterative localization process. Nanda et
al. (2022); Kramár et al. (2024) proposed attribution patching to approximate
activation patching, which requires only two forward passes and one backward
pass for measuring all model components. Syed et al. (2023) further extended
it to edge attribution patching (EAP), which outperforms ACDC in circuit
discovery.

##  5 Evaluation

Existing work has performed two types of evaluation, i.e., _intrinsic
evaluation_ that seeks to establish the quality of the explanations on their
own, without the context of any particular application, and _extrinsic
evaluation_ , which measures the quality of the interpretation by applying the
obtained insights to a relevant downstream task. This section will focus on
the former, whereas the latter was driven by the applications which we will
present in Section 7.6.

##### Faithfulness

Faithfulness measures the degree to which an MI interpretation accurately
reflects the actual decision-making process underlying a model’s behavior
under study Jacovi and Goldberg (2020). For feature discovery, Elhage et al.
(2022a) and Bricken et al. (2023) recruited human annotators to rate the
interpretation of a feature based on its activations over texts, and Bills et
al. (2023) automated it with GPT-4. For circuit discovery, localization is
evaluated by comparing the full model vs. the partial model with the localized
circuit alone Olsson et al. (2022); Wang et al. (2022a); Marks et al. (2024),
while the explanation of circuit components is often validated by knocking out
the component and manually examining its effect.

##### Completeness and Minimality

Besides faithfulness, completeness and minimality are also desirable Wang et
al. (2022a). For example, when studying the model’s ability to recognize
French text, an explanation algorithm may identify a group of neurons $N$ or a
circuit $C$. A complete explanation would mean that no other neurons outside
of $N$ are responsible (e.g., highly activated), or no other circuits besides
$C$ implement this functionality. It can be measured by ablation-based
techniques Wang et al. (2022a); Marks et al. (2024). Specifically, if the
model behavior remains unchanged after removing the neuron or circuit, then
the explanation is incomplete. On the other hand, minimality measures whether
all parts of the explanation (e.g., $N$ or $C$) are necessary, often by
ablating parts of the explanation and computing the change in model behavior
(Wang et al., 2022a).

##### Plausibility

Jacovi and Goldberg (2020) defined plausibility as “how convincing the
interpretation is to humans”. In MI, for example, attributing a model behavior
to polysemantic neurons may be less plausible as compared to monosemantic ones
Bricken et al. (2023); Marks et al. (2024), as humans may have trouble
understanding the former during the evaluation.

##  6 A Beginner’s Roadmap to MI

Feature Study  Circuit Study  Targeted Feature Hypothesis (e.g., LM has
feature $X$) Open-Ended Feature Discovery (e.g., Find features encoded in LM
activations) Observe (e.g., logits distribution, attention visualization)
Logit Lens, VisualizationSAE, Visualization Explain (e.g., Does the activation
pattern encode feature $X$?) Human Explainer or Automated Feature Explanation
Interpret an LM Behavior (e.g., Find circuit for IOI task) Interpret an LM
Component (e.g., attention head) (1) Localization (e.g., Find all nodes and
edges of circuit) CMA, Knockout (2) Explain Circuit Components (2.1) Generate
Hypothesis (e.g., Attention head $X$ is a copy suppression head?) Logit Lens,
Visualization (2.2) Validation (e.g., Does the copy suppression disappear when
attention head $X$ is knocked out?) CMA, Knockout, Logit LensEvaluation
ProbingYesNo

Figure 11: A Beginner’s Roadmap showing the workflow and common techniques for
conducting a feature or circuit study. Both types of studies start with
formulating a research question, followed by various steps (enclosed within
the blue rectangle) to answer it. Techniques employed to execute each step are
indicated along the incoming edge.

A key motivation for this survey is to provide a friendly guide for
researchers and developers interested in MI to quickly pick up the field. To
this end, we provide a _beginner’s roadmap_ in Figure 11.

###  6.1 Feature Study

The study of features can be broadly divided into two categories, i.e.,
_targeted feature discovery_ , which aims to determine if a pre-defined
feature (e.g., a “French text” feature) is encoded in the activations of LMs,
and _open-ended feature discovery_ , which aims to discover all interpretable
features in the activations of LMs.

Probing, logit lens, and visualization are common techniques used for targeted
feature discovery. For probing, a labeled dataset indicating the presence or
absence of the target feature is required to train the probe. If the trained
probe achieves high accuracy on a held-out test set, it can be inferred that
the activation encodes the target feature. On the other hand, the logit lens
or visualization requires a human explainer to interpret the logit lens
projection or the visualization of the highlighted text. Alternatively, LLMs
(e.g., GPT-4) can also be employed to automatically interpret the logit lens
projection or highlighted text to confirm if the feature is encoded in the
activations. Open-ended feature discovery is typically conducted via SAEs and
visualization. Interpretation results of these approaches are then passed to
human explainers or LLMs (if going with automatic feature explanation) for
annotating them with interpretable feature labels, similar to the annotation
process in targeted feature discovery.

Finally, the faithfulness of both categories can be measured by humans based
on how interpretable the discovered features are (e.g., “on a scale of 0–3,
rate your confidence in this interpretation” as executed by Bricken et al.
(2023)) and using automatic explanation scores proposed by Bills et al.
(2023).

###  6.2 Circuit Study

Similarly, the study of circuits can be broadly divided into two categories,
i.e., _interpret an LM behavior (e.g., IOI)_ and _interpret an LM component
(e.g., attention head)_.

The first category involves finding a circuit that is responsible for the LM
behavior. To do so, we start with _(1) localization_ i.e. identifying all the
important components and the edges connecting them via CMA and knockout.
Subsequently, the second step is to _(2) explain each component and its
connection_ one by one. The functional interpretation of each component or
edge can be further divided into two sub-steps — _(2.1) Generating a
hypothesis:_ The behavior of the component under study is analyzed under
relevant input using various visualization techniques (e.g., attention
visualization) and logit lens vocabulary projection. A human can then generate
a plausible hypothesis based on the analysis (e.g., attention head X is
hypothesized to function as a copy suppression head, when it was observed to
decrease the logit of the token it attends to). _(2.2) Validating the
hypothesis:_ The hypothesis can be validated using techniques such as
knockout, CMA, and logit lens. For instance, if the hypothesis suggests that
attention head X is a copy suppression head, then knocking out attention head
X should remove the suppression effect. If the hypothesis is validated, the
model component is considered interpreted; otherwise, further analysis and new
hypotheses are needed. Finally, the discovered circuit, with or without the
second-step explanation, can be evaluated in terms of faithfulness,
completeness, and minimality (Section 5).

The second category of the circuit study, i.e., interpreting an LM component,
follows similar steps but starts with the second step of the first category,
given that the target component to interpret has been provided.

##  7 Findings and Applications

###  7.1 Findings on Features

##### Monosemantics vs. Polysemantics

Earlier work studied neurons as a natural candidate for encoding features,
leading to the discovery of “sentiment neurons” Radford et al. (2017),
“knowledge neurons” Dai et al. (2022), “Base64 neurons” Elhage et al. (2022a),
“skill neurons” Wang et al. (2022b), “positional neurons” Voita et al. (2023),
etc. However, many of these studies found that neurons within LMs are not
_monosemantic_ , i.e. they do not activate only for a single feature. In
contrast, they are _polysemantic_ , i.e. they activate in response to multiple
unrelated features Elhage et al. (2022a); Gurnee et al. (2023); Elhage et al.
(2022b). For instance, a single neuron could activate for both French texts
and texts encoded in Base64. Due to its polysemantic nature, mapping a neuron
to a specific feature becomes challenging.

##### Superposition

The discovery of polysemantic neurons has led to the hypothesis of
_superposition_ , i.e., a model can represent a greater number of independent
features than the number of available neurons. Specifically, features
represented in superposition are encoded in activations by a linear
combination of multiple neurons, rather than by a single neuron, at the cost
of some interference with each other. This hypothesis has been verified by
Elhage et al. (2022b), where the authors showed that when features are sparse,
the model tends to encode features in activation space using superposition. To
extract features from such representations, SAEs have become a widely adopted
approach Bricken et al. (2023); Sharkey et al. (2023); Riggs (2023);
Cunningham et al. (2023), as discussed in Section 4.3. Early studies on SAEs
have shown promising results, with the extracted features from SAEs appearing
more interpretable than those from neurons, according to both human analysis
and automatic explanation scores Bricken et al. (2023).

###  7.2 Findings on Circuits

##### Interpreting LM Behaviors

Specific circuits have been identified for various LM behaviors such as in-
context learning Olsson et al. (2022), indirect object identification (IOI)
Wang et al. (2022a), greater-than operations Conmy et al. (2024), Python
docstring formatting Heimersheim and Janiak (2023), modular addition Nanda et
al. (2023a); Zhong et al. (2024), and more Lieberum et al. (2023); Marks et
al. (2024). These circuits are defined at varying levels of granularity, where
nodes in the computation graph are defined as the outputs of MHA and FF
sublayers (Olsson et al., 2022; Wang et al., 2022a) or as SAE features Marks
et al. (2024); Cunningham et al. (2023). In addition, Merullo et al.  and
Quirke et al. (2024) showed that the same components (e.g., induction heads)
are reused by different circuits (e.g., IOI and induction circuits) to
implement different tasks. Finally, to demonstrate the scalability of current
techniques for identifying circuits in LLMs, Lieberum et al. (2023) identified
the circuit used for the multiple-choice question-answering task on the 70B
Chinchilla LLM Hoffmann et al. (2022)

##### Interpreting Transformer Components

The study of circuits has also yielded insights into the functionalities of
transformer components.

The RS can be viewed as a one-way communication channel that transfers
information from earlier to later layers. Furthermore, Elhage et al. (2021)
hypothesized that MHA and FF in different layers write their output in
different subspaces of the RS, which prevents interference of information. In
addition, nostalgebraist (2020) proposed to view the RS as an LM’s current
“guess” for the output, which is iteratively refined layer-by-layer..

The MHA sublayers are responsible for moving information between tokens, which
enables information from other tokens (i.e., context) to be incorporated into
each token’s representation. Elhage et al. (2021) showed that each attention
head in a layer operates independently and can be interpreted independently.
Several MI studies have shown that these attention heads seem to have
specialized roles. For instance, “negative heads” discovered in GPT2-small by
McDougall et al. (2023) are responsible for reducing the logit values of the
tokens that have already appeared in the context. Other notably identified
attention heads include previous token heads Wang et al. (2022a), duplicate
token heads Wang et al. (2022a), copying heads Elhage et al. (2021), induction
heads Olsson et al. (2022), and successor heads Gould et al. (2023).

The FF sublayers are attributed for the majority of feature extraction Gurnee
et al. (2023), storing and recalling pre-trained knowledge Meng et al. (2022)
and arithmetic computation Stolfo et al. (2023a). More generally, Geva et al.
(2021) viewed FF sublayers as key-value stores where the outputs of the first
layer ($W_{k}^{l}$) of the FF sublayer serves as keys that activate values
(stored knowledge) within the weight matrices of the second layer
($W_{v}^{l}$). Furthermore, they demonstrated that earlier FF layers typically
process shallow (syntactic or grammatical) input patterns, while later layers
focus more on semantic patterns (e.g., text related to TV shows).

###  7.3 Findings on Universality

Studies on universality have yielded mixed results. Early circuit analyses
identified components such as induction heads Olsson et al. (2022), successor
heads Gould et al. (2023), and duplication heads Wang et al. (2022a) across
multiple LMs. Similarly, Merullo et al.  found that different circuits
implementing different tasks (IOI and colored objects task) reuse the same
components (e.g., induction head), demonstrating universality across tasks.
However, Zhong et al. (2024) discovered that two LMs trained with different
initialization can develop qualitatively different circuits for the modular
addition task. Similarly, Chughtai et al. (2023) found that LMs trained to
perform group composition on finite groups with different random weight
initializations on the same task do not develop similar representations and
circuits. Finally, Gurnee et al. (2024) found that only about 1-5% of neurons
from GPT-2 models trained with random initialization exhibit universality.
Understanding the degrees of feature and circuit universality and their
dependency on various aspects of model training (e.g., initialization, model
size, and loss function) remains a crucial open problem.

###  7.4 Findings on Model Capabilities

##### In-Context Learning (ICL)

ICL is an emergent ability of LLMs that enables them to adapt to new tasks
based solely on instructions or a few demonstrations at inference time Wei et
al. (2022). Elhage et al. (2021) studied a simplified case of ICL and
discovered an induction circuit composed of attention heads with specialized
roles (e.g., induction heads), which were then found to be crucial even for
general cases of ICL Olsson et al. (2022); Bansal et al. (2023). Ren et al.
(2024) further investigated few-shot ICL and identified “semantic induction
heads”, which, unlike prior induction heads, model the semantic relationship
between the input and the output token (e.g., “I have a nice pen for writing.
The pen is nice to” $\rightarrow$ “write”).

##### Reasoning

Stolfo et al. (2023b) studied arithmetic reasoning and found that attention
heads are responsible for transferring information from operand and operator
tokens to the RS of the answer or output token, with FF modules subsequently
calculating the answer token. Dutta et al. (2024) studied chain-of-thought
(CoT) multi-step reasoning over fictional ontologies and found that LLMs seem
to deploy multiple different neural pathways in parallel to compute the final
answer. Concurrently, Rai and Yao (2024) investigated neuron activation Geva
et al. (2022) as a unified lens to explain how CoT elicits arithmetic
reasoning of LLMs, including phenomena that were only empirically discussed in
prior work Wang et al. (2023); Ye et al. (2023); Madaan and Yazdanbakhsh
(2022). In particular, the authors found neurons representing various
arithmetic reasoning concepts and showed that the activation of these neurons
is necessary but not sufficient for an LLM’s arithmetic reasoning. Finally,
Brinkmann et al. (2024) discovered an interpretable algorithm in LM for the
task of pathfinding in trees.

##### Others

As listed in Section 7.2, prior work has also studied LM capabilities in tasks
such as IOI Wang et al. (2022a), modular addition Nanda et al. (2023b), and
greater-than operations Conmy et al. (2024), leading to the discovery of
circuits that implement these tasks. Compared with the interpretation of ICL
and reasoning, these studies not only justified the rationale of a capability
but also revealed its underlying algorithm through circuits.

###  7.5 Findings on Learning Dynamics

##### Phase Changes during LM Training

Prior studies have observed sudden shifts in LMs’ capabilities, called “phase
changes” Olsson et al. (2022); Power et al. (2022); Wei et al. (2022). These
changes are considered key steps during the LM training. MI has been applied
to examine the relationship between the emergence of features and circuits and
these phase changes. For example, Olsson et al. (2022) found correlations
between phase changes and the formation of induction circuits, suggesting that
the development of these circuits underlies the phase change. In the task of
symbol manipulation, Nanda et al. (2023a); Varma et al. (2023) discovered a
similar correlation contributing to LM grokking Power et al. (2022), a
phenomenon of LM generalizing beyond memorization. Chen et al. (2024) found
that sudden drops in the loss during training correspond to the acquisition of
attention heads that recognize specific syntactic relations. Finally, Huang et
al. (2024b) provided a unified explanation for grokking, double descent
Nakkiran et al. (2021), and emergent abilities Wei et al. (2022) as a
competition between memorization and generalization circuits.

##### Learning Dynamics during LM Fine-Tuning

Prakash et al. (2024) investigated the underlying changes in mechanisms (e.g.,
task-relevant circuits) to understand performance enhancements in fine-tuned
LMs. The authors found that fine-tuning does not fundamentally change the
mechanisms but enhances existing ones.

###  7.6 Applications of MI

####  7.6.1 Model Enhancement

##### Knowledge Editing

LMs are known to store factual knowledge encountered during pre-training
(Petroni et al., 2019; Cohen et al., 2023). For instance, when an LM is
prompted with “The space needle is in the city of”, it may retrieve the stored
facts and correctly predict “Seattle”. However, these stored facts may be
incorrect or outdated over time, leading to factually incorrect generation
Cohen et al. (2024). MI has been found to be a helpful tool for addressing the
problem, including understanding where and how facts are stored within LMs,
how they are recalled during the inference time, and the approaches for
knowledge editing Meng et al. (2022, 2023); Geva et al. (2023); Sharma et al.
(2024). For instance, Meng et al. (2022) used path patching to localize
components that are responsible for storing factual knowledge, and then edited
the fact (e.g., replacing “Seattle” with “Paris”) by only updating the
parameters of those components.

##### LM Generation Steering

LM generation steering involves controlling an LM’s output by manipulating its
activations at inference time. For instance, Geva et al. (2022) proposed a
method to suppress toxic language generation by identifying and manually
activating neurons in FF layers responsible for promoting non-toxic or safe
words. Similarly, Templeton et al. (2024) identified safety-related features
(e.g., unsafe code, gender bias) and manipulated their activations to steer
the LM towards (un)desired behaviors (e.g., safe code generation, unbiased
text generation). Additionally, Nanda et al. (2023b) demonstrated that an LM’s
output can be altered (e.g., flipping a player turn in the game of Othello
from YOURS to MINE) by pushing its activation in the direction of a linear
vector representing the desired behavior, which was identified using a linear
probe.

####  7.6.2 AI Safety

AI safety is an important concern that MI aims to address. At present, the
exact role MI can play in addressing AI safety is unclear Casper (2023).
Within MI, “enumerative safety” aims to address AI safety by enumerating all
features in LMs and inspecting those related to dangerous capabilities or
intentions Elhage et al. (2022b); Olah and Jermyn (2023). To this end,
Templeton et al. (2024) identified several safety-relevant features that not
only activate when the LM exhibits specific behaviors but also causally
influence the LM’s output; however, the specific circuits that use these
features to implement the behavior have not yet been identified. As we have
discussed, Geva et al. (2022) encouraged language safety by steering the LM’s
generation of non-toxic tokens. Finally, insights from circuit studies were
used to detect prompt injection Belrose et al. (2023) and find adversarial
examples for the IOI task Wang et al. (2022a).

####  7.6.3 Others

Insights from MI have also been used for other downstream tasks. Marks et al.
(2024) improved the generalization of classifiers by identifying and ablating
spurious features that humans consider to be task-irrelevant. Geva et al.
(2022) proposed self-supervised early exit prediction for efficient inference,
drawing insights from their investigations of the FF sublayers’ role in token
prediction.

##  8 Discussion and Future Work

##### Automated Hypothesis Generation

At a high level, MI study is a process with two stages: generating hypotheses
on the underlying mechanisms of LM behavior and validating them. Although
various techniques have automated hypothesis validation (Section 4.5), the
generation part is mainly left to humans, a potential scalability bottleneck
to LLMs and complex behaviors. To address it, automated hypothesis generation,
potentially with humans in the loop, is instrumental.

##### Studies on Complex Tasks and LLMs

Current MI studies are mostly performed on simpler tasks, often criticized as
“streetlight interpretability” Casper (2023); Wang (2022). For instance, Wang
(2022) intentionally selected the IOI task Wang et al. (2022a) because it is a
simple algorithmic task. Similarly, although a few studies were done on
“production-level” LMs Lieberum et al. (2023); Templeton et al. (2024), most
still used small LMs, which may have limited generalizability, given the mixed
results on universality.

##### Practical Utility

While people acknowledge the importance of actionable insights to downstream
applications Doshi-Velez and Kim (2017), MI studies that highlight such
applications, such as adversarial example generation (Wang et al., 2022a),
often lack the depth of investigation when they do not compare against
alternative methods or baselines. As an exception, Marks et al. (2024) showed
that their MI technique improves model generalization better than several
baselines.

##### Standardized Benchmarks and Metrics

Evaluating interpretability results is inherently challenging due to the lack
of ground truth (Zhou et al., 2022), and current MI studies also employ
diverse ad hoc evaluation approaches, potentially leading to inconsistent
comparisons Zhang and Nanda (2023). There are some proposals for standardized
evaluation benchmarks. RAVEL Huang et al. (2024a) evaluates techniques (e.g.,
SAEs) that disentangle polysemantic neurons into monosemantic features. Tracr
Lindner et al. (2024) converts human-readable RASP programs Weiss et al.
(2021) into transformers to enable ground-truth features and algorithms.
However, these proposed benchmarks are still insufficient Räuker et al.
(2023). For instance, success on synthetic transformers in the Tracr benchmark
may not imply that on naturally trained transformers. Thus, more effort is
needed to develop standard evaluation techniques and to ensure their wide
adoption.

##  9 Limitations

Our survey paper discusses MI studies conducted only on decoder-only
transformer LMs, omitting studies on other transformer LM variants such as
encoder-only and encoder-decoder transformer LMs. Additionally, due to space
constraints, some techniques may not have been presented with full technical
details (e.g., causal scrubbing). Furthermore, given the fast-paced nature of
the field, our survey may not reflect the latest advancements that were
published close to or after the survey was conducted. Despite these
limitations, however, our survey provides a comprehensive taxonomy of the
current state of the MI field as well as a roadmap, specifically curated to
help newcomers quickly pick up the field. Both the taxonomy and the roadmap
can serve as guides for future researchers interested in encoder-only or
encoder-decoder LMs to build up their survey. In addition, we have created a
GitHub repository at https://github.com/Dakingrai/awesome-mechanistic-
interpretability-lm-papers to sustainably maintain this survey, and we welcome
researchers sharing the same interest to add new papers and extend the
taxonomy and the roadmap.

## Acknowledgements

DR and ZY were sponsored by the National Science Foundation (SHF2311468) and
the College of Computing and Engineering and the Department of Computer
Science at George Mason University. This project was also supported by
resources provided by the Office of Research Computing at George Mason
University (https://orc.gmu.edu) and funded in part by grants from the
National Science Foundation (2018631).

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Antverg and Belinkov (2021) Omer Antverg and Yonatan Belinkov. 2021.  On the pitfalls of analyzing individual neurons in language models.  In _International Conference on Learning Representations_.
  * Bansal et al. (2023) Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2023.  Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11833–11856.
  * Belrose et al. (2023) Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023.  Eliciting latent predictions from transformers with the tuned lens.  _arXiv preprint arXiv:2303.08112_.
  * Bereska and Gavves (2024) Leonard Bereska and Efstratios Gavves. 2024.  Mechanistic interpretability for ai safety–a review.  _arXiv preprint arXiv:2404.14082_.
  * Bills et al. (2023) Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023.  Language models can explain neurons in language models.  https://openai.com/index/language-models-can-explain-neurons-in-language-models/.
  * Bricken et al. (2023) Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023.  Towards monosemanticity: Decomposing language models with dictionary learning.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2023/monosemantic-features/index.html.
  * Brinkmann et al. (2024) Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. 2024.  A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task.  _arXiv preprint arXiv:2402.11917_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Casper (2023) Stephen Casper. 2023.  The engineer’s interpretability sequence.  https://www.lesswrong.com/s/a6ne2ve5uturEEQK7.
  * Chan et al. (2022) Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. 2022.  Causal scrubbing, a method for rigorously testing interpretability hypotheses.  _AI Alignment Forum_.  https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing.
  * Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_ , 15(3):1–45.
  * Chen et al. (2024) Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. 2024.  Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms.  In _The Twelfth International Conference on Learning Representations_.
  * Chughtai et al. (2023) Bilal Chughtai, Lawrence Chan, and Neel Nanda. 2023.  A toy model of universality: Reverse engineering how networks learn group operations.  In _International Conference on Machine Learning_ , pages 6243–6267. PMLR.
  * Cohen et al. (2024) Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024.  Evaluating the ripple effects of knowledge editing in language models.  _Transactions of the Association for Computational Linguistics_ , 12:283–298.
  * Cohen et al. (2023) Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023.  Crawling the internal knowledge-base of language models.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1856–1869.
  * Conmy et al. (2024) Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2024.  Towards automated circuit discovery for mechanistic interpretability.  _Advances in Neural Information Processing Systems_ , 36.
  * Conneau et al. (2018) Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136.
  * Cooney and Nanda (2023) Alan Cooney and Neel Nanda. 2023.  Circuitsvis.  https://github.com/TransformerLensOrg/CircuitsVis.
  * Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023.  Sparse autoencoders find highly interpretable features in language models.  _arXiv preprint arXiv:2309.08600_.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8493–8502.
  * Dalvi et al. (2019) Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2019.  What is one grain of sand in the desert? analyzing individual neurons in deep nlp models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6309–6317.
  * Dar et al. (2023) Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023.  Analyzing transformers in embedding space.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16124–16170.
  * Din et al. (2023) Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2023.  Jump to conclusions: Short-cutting transformers with linear transformations.  _arXiv preprint arXiv:2303.09435_.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_.
  * Durrani et al. (2020) Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020.  Analyzing individual neurons in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4865–4880.
  * Dutta et al. (2024) Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. 2024.  How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning.  _arXiv preprint arXiv:2402.18312_.
  * Elhage et al. (2022a) Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. 2022a.  Softmax linear units.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2022/solu/index.html.
  * Elhage et al. (2022b) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022b.  Toy models of superposition.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2022/toy_model/index.html.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2021/framework/index.html.
  * Ferrando et al. (2024) Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. 2024.  A primer on the inner workings of transformer-based language models.  _arXiv preprint arXiv:2405.00208_.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12216–12235.
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 30–45.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495.
  * Goldowsky-Dill et al. (2023) Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023.  Localizing model behavior with path patching.  _arXiv preprint arXiv:2304.05969_.
  * Gould et al. (2023) Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2023.  Successor heads: Recurring, interpretable attention heads in the wild.  In _The Twelfth International Conference on Learning Representations_.
  * Gurnee et al. (2024) Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024.  Universal neurons in gpt2 language models.  _arXiv preprint arXiv:2401.12181_.
  * Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023.  Finding neurons in a haystack: Case studies with sparse probing.  _Transactions on Machine Learning Research_.
  * Heimersheim and Janiak (2023) Stefan Heimersheim and Jett Janiak. 2023.  A circuit for python docstrings in a 4-layer attention-only transformer.  https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/acircuit-for-python-docstrings-in-a-4-layer-attention-only.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_.
  * Huang et al. (2024a) Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024a.  Ravel: Evaluating interpretability methods on disentangling language model representations.  _arXiv preprint arXiv:2402.17700_.
  * Huang et al. (2024b) Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024b.  Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition.  _arXiv preprint arXiv:2402.15175_.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online. Association for Computational Linguistics.
  * Kramár et al. (2024) János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. 2024.  Atp*: An efficient and scalable method for localizing llm behaviour to components.  _arXiv preprint arXiv:2403.00745_.
  * Lieberum et al. (2023) Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023.  Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.  _arXiv preprint arXiv:2307.09458_.
  * Lindner et al. (2024) David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Tom McGrath, and Vladimir Mikulik. 2024.  Tracr: Compiled transformers as a laboratory for interpretability.  _Advances in Neural Information Processing Systems_ , 36.
  * Madaan and Yazdanbakhsh (2022) Aman Madaan and Amir Yazdanbakhsh. 2022.  Text and patterns: For effective chain of thought, it takes two to tango.  _arXiv preprint arXiv:2209.07686_.
  * Marks et al. (2024) Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024.  Sparse feature circuits: Discovering and editing interpretable causal graphs in language models.  _arXiv preprint arXiv:2403.19647_.
  * McDougall et al. (2023) Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. 2023.  Copy suppression: Comprehensively understanding an attention head.  _arXiv preprint arXiv:2310.04625_.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023.  Mass-editing memory in a transformer.  In _The Eleventh International Conference on Learning Representations_.
  * (53) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick.  Circuit component reuse across tasks in transformer language models.  In _The Twelfth International Conference on Learning Representations_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26.
  * Nakkiran et al. (2021) Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2021.  Deep double descent: Where bigger models and more data hurt.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2021(12):124003.
  * Nanda et al. (2023a) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023a.  Progress measures for grokking via mechanistic interpretability.  _arXiv preprint arXiv:2301.05217_.
  * Nanda et al. (2023b) Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023b.  Emergent linear representations in world models of self-supervised sequence models.  _arXiv preprint arXiv:2309.00941_.
  * Nanda et al. (2022) Neel Nanda, Chris Olah, Catherine Olsson, Nelson Elhage, and Hume Tristan. 2022.  Attribution patching: Activation patching at industrial scale, 2023.  https://www.neelnanda.io/mechanistic-interpretability/attribution-patching.
  * nostalgebraist (2020) nostalgebraist. 2020.  Interpreting gpt: the logit lens.  _AI Alignment Forum_.  https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens.
  * Olah et al. (2020) Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020.  Zoom in: An introduction to circuits.  _Distill_.  https://distill.pub/2020/circuits/zoom-in.
  * Olah and Jermyn (2023) Chris Olah and Adam Jermyn. 2023.  What would be the most safety-relevant features in language models? (circuits updates - july 2023).  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2023/july-update/index.html#safety-features.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022.  In-context learning and induction heads.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
  * Pal et al. (2023) Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. 2023.  Future lens: Anticipating subsequent tokens from a single hidden state.  _arXiv preprint arXiv:2311.04897_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Power et al. (2022) Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022.  Grokking: Generalization beyond overfitting on small algorithmic datasets.  _arXiv preprint arXiv:2201.02177_.
  * Prakash et al. (2024) Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024.  Fine-tuning enhances existing mechanisms: A case study on entity tracking.  In _The Twelfth International Conference on Learning Representations_.
  * Quirke et al. (2024) Philip Quirke, Clement Neo, and Fazl Barez. 2024.  Increasing trust in language models through the reuse of verified circuits.  _arXiv preprint arXiv:2402.02619_.
  * Radford et al. (2017) Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017.  Learning to generate reviews and discovering sentiment.  _arXiv preprint arXiv:1704.01444_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rai and Yao (2024) Daking Rai and Ziyu Yao. 2024.  An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms.  _arXiv preprint arXiv:2406.12288_.
  * Räuker et al. (2023) Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. 2023.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks.  In _2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)_ , pages 464–483. IEEE.
  * Ren et al. (2024) Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Xipeng Qiu, and Dahua Lin. 2024.  Identifying semantic induction heads to understand in-context learning.  _arXiv preprint arXiv:2402.13055_.
  * Riggs (2023) Logan Riggs. 2023.  Found 600+ monosemantic features in a small lm using sparse autoencoders.  https://www.lesswrong.com/posts/wqRqb7h6ZC48iDgfK/tentatively-found-600-monosemantic-features-in-a-small-lm.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Sharkey et al. (2023) Lee Sharkey, Dan Braun, and Beren Millidge. 2023.  Taking features out of superposition with sparse autoencoders, 2022.  https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition.
  * Sharma et al. (2024) Arnab Sen Sharma, David Atkinson, and David Bau. 2024.  Locating and editing factual associations in mamba.  _arXiv preprint arXiv:2404.03646_.
  * Stolfo et al. (2023a) Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023a.  A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7035–7052.
  * Stolfo et al. (2023b) Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023b.  Understanding arithmetic reasoning in language models using causal mediation analysis.  _arXiv preprint arXiv:2305.15054_.
  * Syed et al. (2023) Aaquib Syed, Can Rager, and Arthur Conmy. 2023.  Attribution patching outperforms automated circuit discovery.  _arXiv preprint arXiv:2310.10348_.
  * Templeton et al. (2024) Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024.  Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.  _Transformer Circuits Thread_.  https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.
  * Tenney et al. (2019a) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.  Bert rediscovers the classical nlp pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601.
  * Tenney et al. (2019b) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019b.  What do you learn from context? probing for sentence structure in contextualized word representations.  _arXiv preprint arXiv:1905.06316_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Varma et al. (2023) Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. 2023.  Explaining grokking through circuit efficiency.  _arXiv preprint arXiv:2309.02390_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020.  Investigating gender bias in language models using causal mediation analysis.  _Advances in neural information processing systems_ , 33:12388–12401.
  * Voita et al. (2023) Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2023.  Neurons in large language models: Dead, n-gram, positional.  _arXiv preprint arXiv:2309.04827_.
  * Wang et al. (2023) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2717–2739, Toronto, Canada. Association for Computational Linguistics.
  * Wang (2022) Kevin Wang. 2022.  Gears-level mental models of transformer interpretability.  https://www.lesswrong.com/posts/X26ksz4p3wSyycKNB/gears-level-mental-models-of-transformer-interpretability.
  * Wang et al. (2022a) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022a.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  In _The Eleventh International Conference on Learning Representations_.
  * Wang et al. (2022b) Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022b.  Finding skill neurons in pre-trained transformer-based language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11132–11152.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.
  * Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022.  Taxonomy of risks posed by language models.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 214–229.
  * Weiss et al. (2021) Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021.  Thinking like transformers.  In _International Conference on Machine Learning_ , pages 11080–11090. PMLR.
  * Yao et al. (2024) Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024.  A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.  _High-Confidence Computing_ , page 100211.
  * Ye et al. (2023) Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Veselin Stoyanov, Greg Durrett, and Ramakanth Pasunuru. 2023.  Complementary explanations for effective in-context learning.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4469–4484, Toronto, Canada. Association for Computational Linguistics.
  * Yun et al. (2021) Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann Lecun. 2021.  Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.  In _Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 1–10.
  * Zhang and Nanda (2023) Fred Zhang and Neel Nanda. 2023.  Towards best practices of activation patching in language models: Metrics and methods.  In _The Twelfth International Conference on Learning Representations_.
  * Zhao et al. (2024) Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024.  Explainability for large language models: A survey.  _ACM Transactions on Intelligent Systems and Technology_ , 15(2):1–38.
  * Zhong et al. (2024) Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. 2024.  The clock and the pizza: Two stories in mechanistic explanation of neural networks.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhou et al. (2022) Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. 2022.  Do feature attribution methods correctly attribute features?  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 9623–9633.