# Not All Language Model Features Are Linear

Joshua Engels
MIT
<EMAIL_ADDRESS>
&Isaac Liao*
MIT
<EMAIL_ADDRESS>
&Eric J. Michaud
MIT & IAIFI
<EMAIL_ADDRESS>
Wes Gurnee
MIT
<EMAIL_ADDRESS>
&Max Tegmark
MIT & IAIFI
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Recent work has proposed the linear representation hypothesis: that language
models perform computation by manipulating one-dimensional representations of
concepts (“features”) in activation space. In contrast, we explore whether
some language model representations may be inherently multi-dimensional. We
begin by developing a rigorous definition of irreducible multi-dimensional
features based on whether they can be decomposed into either independent or
non-co-occurring lower-dimensional features. Motivated by these definitions,
we design a scalable method that uses sparse autoencoders to automatically
find multi-dimensional features in GPT-2 and Mistral 7B. These auto-discovered
features include strikingly interpretable examples, e.g. circular features
representing days of the week and months of the year. We identify tasks where
these exact circles are used to solve computational problems involving modular
arithmetic in days of the week and months of the year. Finally, we provide
evidence that these circular features are indeed the fundamental unit of
computation in these tasks with intervention experiments on Mistral 7B and
Llama 3 8B, and we find further circular representations by breaking down the
hidden states for these tasks into interpretable components.

##  1 Introduction

Language models trained for next-token prediction on large text corpora have
demonstrated remarkable capabilities, including coding, reasoning, and in-
context learning [7, 1, 3, 45]. However, the specific algorithms models learn
to achieve these capabilities remain largely a mystery to researchers; we do
not understand how language models write poetry. Mechanistic interpretability
is a field that seeks to address this gap by reverse-engineering trained
models from the ground up into variables (features) and the programs
(circuits) that process these variables [37].

One mechanistic interpretability research direction has focused on
understanding toy models in detail. This work has found multi-dimensional
representations of inputs such as lattices [30] and circles [27, 35], and has
successfully reverse-engineered the algorithms that models use to manipulate
these representations. A separate direction has identified one-dimensional
representations of high level concepts and quantities in large language models
[15, 29, 17, 6]. These findings have led to the linear representation
hypothesis: that all representations in pretrained large language models are
one-dimensional lines, and that we can understand model behavior as nonlinear
manipulations of these linear representations [38, 6]. In this work, we bridge
the gap between these two regimes by providing evidence that language models
also use multi-dimensional representations.

###  1.1 Contributions

  1. 1.

In Section 3, we generalize the one-dimensional definition of a language model
feature to multi-dimensional features, provide an updated superposition
hypothesis to account for these new features, and analyze the reduction in a
model’s representation space implied by using multi-dimensional features. We
also develop a theoretically grounded and empirically practical test for
irreducible features and run this test on some sample distributions.

  2. 2.

In Section 4, we present a method for finding multi-dimensional features using
sparse autoencoders and identify multi-dimensional representations
automatically in GPT-2 and Mistral 7B, including circular representations for
the day of the week and month of the year. To the best of our knowledge, we
are the first to find an emergent circular representation in a large language
model.

  3. 3.

In Section 5, we propose two tasks, modular addition in days of the week and
in months of the year, that we hypothesize will cause models to use these
circular representations. We perform intervention experiments on Mistral 7B
and Llama 3 8B to show that models do indeed use circular representations for
these tasks. Finally, we present novel methods for decomposing LLM hidden
states and reveal circles in the computed day of the week and month of the
year.

Figure 1: Circular representations of days of the week, months of the year,
and years of the 20th century in layer 7 of GPT-2-small. These representations
were discovered via clustering SAE dictionary elements, described in Section
4. Points are colored according to the token which created the representation.
See Fig. 12 for other axes and Fig. 13 for similar plots for Mistral 7B.

##  2 Related Work

Linear Representations: Early word embedding methods such as GloVe and
Word2vec, although only trained using co-occurrence data, contained directions
in their vector spaces corresponding to semantic concepts, e.g. the well-known
f(king) - f(man) + f(woman) = f(queen) [32, 40, 31]. Recent research has found
similar evidence of linear representations in sequence models trained only on
next token prediction, including Othello board positions [36, 26], the truth
value of assertions [29], and numeric quantities such as longitude, latitude,
birth year, and death year [15, 17]. These results have inspired the linear
representation hypothesis [38, 11] defined above. Recent theoretical work
provides evidence for this hypothesis, assuming a latent (binary) variable-
based model of language [22]. Empirically, dictionary learning has shown
success in breaking down a model’s feature space into a sparse over-complete
basis of linear features using sparse autoencoders [6, 9]. These works assume
that the number of linear features stored in superposition exceeds the model
dimensionality [11].

Nonlinear Representations: There has been comparatively little research on
nonlinear features. One recent paper [24] proves that a one-layer swapped
order (MLP before attention) transformer can in-context-learn a nonlinear
mapping function followed by a linear regression, implying that the “features”
between the MLP and attention blocks are nonlinear. Another work [42] finds
that a transformer trained on a hidden Markov model uses a fractal structure
to represent the probability of each next token. These works analyze toy
models and it is not clear if large language models will have similar
nonlinear features. A separate idea [4] argues for interpreting neural
networks through the polytopes they split the input space into, and identifies
regions of low polytope density as “valid” regions for a potential linear
representation. Finally, recent work on dictionary learning [6] has speculated
about multi-dimensional feature manifolds; our work is most similar to this
direction and develops the idea of feature manifolds theoretically and
empirically .

Circuits: Circuits research seeks to identify and understand circuits, subsets
of a model (usually represented as a directed acyclic graph) that explain
specific behaviors [37]. The base units that form a circuit can be layers,
neurons [37], or sparse autoencoder features [28]. The first circuits-style
work looked at the InceptionV1 image model and found line features that were
combined into curve detection features [37]. More recent work has examined
language models, for example the indirect object identification circuit in
GPT-2 [47]. Given the difficulty of designing bespoke experiments, there has
been increased research in automated circuit discovery methods [28, 8, 44].

Interpretability for Arithmetic Problems: Prior work studies models trained on
modular arithmetic problems $a+b=c\ (\mathrm{mod}\ m)$ and finds that models
that generalize well have circular representations for $a$ and $b$ [27].
Further work shows that models use these circular representations to compute
$c$ via a “clock” algorithm [35] and a separate “pizza” algorithm [49]. These
papers are limited to the case of a small model trained only on modular
arithmetic. Another direction has studied how large language models perform
basic arithmetic, including a circuits level description of the greater-than
operation in GPT-2 [16] and addition in GPT-J [43]. These works find that to
perform a computation, models copy pertinent information to the token before
the computed result and perform the computation in the subsequent MLP layers.
Finally, recent work [14] investigates language models’ ability to increment
numbers and finds linear features that fire on tokens equivalent modulo $10$.

##  3 Definitions and Theory

In this section, we focus on $L$ layer transformer models $M$ that take in
token input ${\bf t}=(t_{1},\ldots,t_{n})$, have hidden states
$\mathbf{x}_{1,l},\ldots,\mathbf{x}_{n,l}$ for layers $l$, and output logit
vectors $\mathbf{y}_{1},\ldots,\mathbf{y}_{n}$. Given a set of inputs $T$, we
let $X_{i,l}$ be the set of all corresponding $\mathbf{x}_{i,l}$. This section
focuses on hypotheses that describe how to decompose hidden states
$\mathbf{x}_{i,l}$ into sums of functions of the input (features). While this
is always possible if $M$ is deterministic via the “trivial” evaluation of $M$
itself, we are interested in decomposable, interpretable hypotheses for the
construction of $\mathbf{x}_{i,l}$. We write matrices in capital bold, vectors
and vector valued functions in lowercase bold, and sets in capital non-bold.

###  3.1 Multi-Dimensional Features

######  Definition 1 (Feature).

We define a $d_{f}$-dimensional feature of sparsity $s$ as a function
${\mathbf{f}}$ that maps a subset of the input space of probability $1-s>0$
into a $d_{f}$-dimensional point cloud in $\mathbb{R}^{d_{f}}$. We say that a
feature is active on the aforementioned subset.

As an example, let $n=1$ (so inputs are single tokens) and consider a feature
${\mathbf{f}}$ that maps integer tokens to equispaced points in
$\mathbb{R}^{1}$. Then ${\mathbf{f}}$ is a $1$-dimensional feature that is
active on integer tokens, and if integer tokens occur $1\%$ of the time across
the input distribution, ${\mathbf{f}}$ has sparsity $s=0.99$.

For features to be meaningful, they should be irreducible. Here, we focus on
statistical reducibility: ${\mathbf{f}}$ is reducible if it is composed of two
statistically independent co-occurring features (in which case ${\mathbf{f}}$
is “separable”) or if it is composed of two non-co-occurring features (in
which case ${\mathbf{f}}$ is a “mixture”). We compare this with another way to
define multi-dimensional irreducibility in Appendix C.

The probability distribution over input tokens $t$ induces a
$d_{f}$-dimensional probability distribution over feature vectors
${\mathbf{f}}(t)$ — Fig. 2 shows two examples. Note that ${\mathbf{f}}(t)$ is
a random vector since $t$ is a random variable; we use $p({\mathbf{f}})$ to
denote the probability density function of ${\mathbf{f}}(t)$.

######  Definition 2.

A feature ${\mathbf{f}}$ is reducible into features ${\mathbf{a}}$ and
${\mathbf{b}}$ if there exists an affine transformation

$\displaystyle{\mathbf{f}}\mapsto{\mathbf{R}}{\mathbf{f}}+{\mathbf{c}}\equiv\left({{\mathbf{a}}\atop{\mathbf{b}}}\right)$
(1)

for some orthonormal $d_{f}\times d_{f}$ matrix ${\mathbf{R}}$ and additive
constant ${\mathbf{c}}$, such that the transformed feature probability
distribution $p({\mathbf{a}},{\mathbf{b}})$ satisfies at least one of these
conditions:

  1. 1.

$p$ is separable, i.e., factorizable as a product of its marginal
distributions:
$p({\mathbf{a}},{\mathbf{b}})=p({\mathbf{a}})p({\mathbf{b}})$.

  2. 2.

$p$ is a mixture
$p({\mathbf{a}},{\mathbf{b}})=wp_{1}({\mathbf{a}},{\mathbf{b}})+(1-w)p_{2}({\mathbf{a}},{\mathbf{b}})$
of disjoint probability distributions for $w>0$, and $p_{1}$ is lower-
dimensional such that
$p_{1}({\mathbf{a}},{\mathbf{b}})=p_{1}({\mathbf{a}})\delta({\mathbf{b}})$.

Here $\delta$ is the Dirac delta function. By two probability distributions
being disjoint, we mean that they have disjoint support (there is no set where
both have positive probability measure, or equivalently the two features
${\mathbf{a}}$ and ${\mathbf{b}}$ cannot be active at the same time). In Eq.
1, ${\mathbf{a}}$ is the first $k$ components of the vector
${\mathbf{R}}{\mathbf{f}}+{\mathbf{c}}$ and ${\mathbf{b}}$ is the remaining
$d_{f}-k$ components. When $p$ is separable or a mixture, we also say that
${\mathbf{f}}$ is separable or a mixture. We term a feature irreducible if it
is not reducible, i.e., if no rotation and translation makes it separable or a
mixture.

(a) Testing $S({\mathbf{a}})$ and $M_{\epsilon}({\mathbf{a}})$ on a reducible
feature ${\mathbf{a}}$.

(b) Testing $S({\mathbf{b}})$ and $M_{\epsilon}({\mathbf{b}})$ on an
irreducible feature ${\mathbf{b}}$

Figure 2: Testing irreducibility of synthetic features. Left in each
subfigure: Distributions of $\mathbf{x}$. For feature ${\mathbf{a}}$, 36.21%
lies within the narrow dotted lines, indicating the feature is likely a
mixture. For feature ${\mathbf{b}}$, 17.84% lies within the wide lines,
indicating the feature is unlikely to be a mixture. The green cross indicates
the angle $\theta$ that minimizes mutual information. Middle in each
subfigure: Histograms of the distribution of $\mathbf{v}\cdot\mathbf{x}$ with
red lines indicating a $2\epsilon$-wide region. Right in each subfigure:
Mutual information between $\mathbf{a}$ and $\mathbf{b}$ as a function of the
rotation angle $\theta$ of matrix $\mathbf{R}$. Both features have a large
($\geq 0.5$ bits) minimum mutual information and so are likely not separable.

Fig. 2a shows an example of a 2D feature that is a mixture, because it can be
decomposed into features ${\mathbf{a}}$ and ${\mathbf{b}}$ where
${\mathbf{b}}$ is a 1D line distribution (marked in red) and ${\mathbf{a}}$ is
the remainder (a 2D cloud and a line, which can in turn be decomposed). An
example of a feature that is separable is a normal distribution (since any
multidimensional Gaussian can be rotated to have a diagonal covariance
matrix). In natural language, a mixture might be a one hot encoding of
“language of the current token”, while a separable distribution might be the
“latitude” and “longitude” of location tokens.

In practice, because of noise and finite sample size, the mixture and
separability definitions may not be precisely satisfied. Thus, we soften our
definitions to permit degrees of reducibility:

######  Definition 3 (Separability Index and $\epsilon$-Mixture Index).

Consider a feature ${\mathbf{f}}$. The separability index $S({\mathbf{f}})$
measures the minimal mutual information between all possible ${\mathbf{a}}$
and ${\mathbf{b}}$ defined in Eq. 1:

$\displaystyle S({\mathbf{f}})\equiv$ $\displaystyle\min
I({\mathbf{a}};{\mathbf{b}})$ (2)

where $I$ denotes the mutual information. Smaller values of $S({\mathbf{f}})$
mean that ${\mathbf{f}}$ is more separable and therefore more reducible. Note
that we can solely minimize over how many components to split off as
${\mathbf{a}}$ and over orthonormal matrices ${\mathbf{R}}$, since the
additive offset ${\mathbf{c}}$ does not affect the mutual information.

The $\epsilon$-mixture index $M_{\epsilon}({\mathbf{f}})$ tests how often
${\mathbf{f}}$ can be projected near zero while it is active:

$\displaystyle M_{\epsilon}({\mathbf{f}})=$
$\displaystyle\max_{\mathbf{v}\in\mathbb{R}^{d_{f}},\;c\in\mathbb{R}}\mathbb{P}\left(|\mathbf{v}\cdot{\mathbf{f}}+c|<\epsilon\sqrt{\mathbb{E}[(\mathbf{v}\cdot{\mathbf{f}}+c)^{2}]}\right)$
(3)

Larger values of $M_{\epsilon}({\mathbf{f}})$ mean that ${\mathbf{f}}$ is more
of a mixture and is therefore more reducible.

We develop an optimization procedure to solve for the $\min$ and $\max$ in
Definition 3 and apply it to synthetic feature distributions in Fig. 2a.
Further details on this optimization process, more synthetic feature
distributions, and more intuition about these definitions can be found in
Appendix B.

###  3.2 Superposition

We now examine the implications of multi-dimensional features for the
superposition hypothesis [11]. We first restate the original superposition
hypothesis using our above definitions:

######  Definition 4 ($\delta$-orthogonal matrices).

Two matrices ${\mathbf{A}}_{1}\in\mathbb{R}^{d\times d_{1}}$ and
${\mathbf{A}}_{2}\in\mathbb{R}^{d\times d_{2}}$ are $\delta$-orthogonal if
$|\mathbf{x}_{1}\cdot\mathbf{x}_{2}|\leq\delta$ for all unit vectors
$\mathbf{x}_{1}\in\text{colspace}({\mathbf{A}}_{1})$ and
$\mathbf{x}_{2}\in\text{colspace}({\mathbf{A}}_{2})$.

######  Hypothesis 1 (One-Dimensional Superposition Hypothesis, paraphrased
from [11]).

Hidden states $\mathbf{x}_{i,l}$ are the sum of many ($\gg d$) sparse one-
dimensional features $f_{i}$ and pairwise $\delta$-orthogonal vectors
$\mathbf{v}_{i}$ such that
$\mathbf{x}_{i,l}(t)=\sum_{i}\mathbf{v}_{i}f_{i}(t)$. We set $f_{i}(t)$ to
zero when $t$ is outside the domain of $f_{i}$.

We now present our new superposition hypothesis, which posits independence
between irreducible multi-dimensional features instead of unknown levels of
independence between one-dimensional features:

######  Hypothesis 2 (Our Superposition Hypothesis, changes underlined).

Hidden states $\mathbf{x}_{i,l}$ are the sum of many ($\gg d$) sparse low-
dimensional irreducible features ${\mathbf{f}}_{i}$ and pairwise
$\delta$-orthogonal matrices $\mathbf{V}_{i}\in\mathbb{R}^{d\times
d_{{\mathbf{f}}_{i}}}$ such that
$\mathbf{x}_{i,l}(t)=\sum_{i}\underline{\mathbf{V}_{i}}{\mathbf{f}}_{i}(t)$.
We set ${\mathbf{f}}_{i}(t)$ to zero when $t$ is outside the domain of
${\mathbf{f}}_{i}$. Any subset of features must be mutually independent on
their shared domain.

The Johnson-Lindenstrauss (JL) Lemma [23] implies that we can choose
$e^{Cd\delta^{2}}$ pairwise one-dimensional $\delta$-orthogonal vectors to
satisfy 1 for some constant $C$, thus allowing us to build the model’s feature
space with a number of one-dimensional $\delta$-orthogonal features
exponential in $d$. In Appendix A, we prove a similar result for low-
dimensional projections (the main idea of the proof is to combine
$\delta$-orthogonal vectors as guaranteed from the JL lemma):

######  Theorem 1.

For any $d$, $d_{\max}$, and $\delta$, it is possible to choose
$\frac{1}{d_{\max}}e^{C(d/d_{\max}^{2})\delta^{2}}$ pairwise
$\delta$-orthogonal matrices ${\mathbf{A}}_{i}\in\mathbb{R}^{n_{i}\times d}$
where $1\leq n_{i}\leq d_{\max}$ for some constant $C$.

This exponential reduction in the number of features that are representable
$\delta$-orthogonally (as opposed to a multiplicative factor reduction, which
is also present) suggests that models will employ higher-dimensional features
only for representations that necessitate detailed multi-dimensional
descriptions. Moreover, these representations will be highly compressed to fit
within the smallest dimensional space possible, potentially leading to
interesting encodings; for example, recent work [33] finds that maximum-margin
solutions for problems like modular arithmetic consist of Fourier features.
Note that the proof assumes the “worst case” scenario that all of the features
are dimension $d_{\max}$, while in practice many of the features may be $1$ or
low dimensional, so the effect on the capacity of a real model that represents
multi-dimensional features is unlikely to be this extreme.

##  4 Sparse Autoencoders Find Multi-Dimensional Features

Sparse autoencoders (SAEs) deconstruct model hidden states into sparse vector
sums from an over-complete basis [6, 9]. For hidden states $X_{i,l}$, a one-
layer SAE of size $m$ with sparsity penalty $\lambda$ minimizes the following
dictionary learning loss [6, 9]:

$\displaystyle\texttt{DL}(X_{i,l})=\operatorname*{arg\,min}_{\begin{subarray}{c}{\mathbf{E}}\in\mathbb{R}^{m\times
d},{\mathbf{D}}\in\mathbb{R}^{d\times
m}\end{subarray}}\sum_{\mathbf{x}_{i,l}\in
X_{i,l}}\left[\left\lVert\mathbf{x}_{i,l}-{\mathbf{D}}\cdot\texttt{ReLU}({\mathbf{E}}\cdot\mathbf{x}_{i,l})\right\rVert_{2}^{2}+\lambda\lVert\texttt{ReLU}({\mathbf{E}}\cdot\mathbf{x}_{i,l})\rVert_{0}\right]$
(4)

In practice, the $L_{0}$ loss on the last term is relaxed to $L_{p}$ for
$0<p\leq 1$ to make the loss differentiable. We call the $m$ columns of
${\mathbf{D}}$ (vectors in $\mathbb{R}^{d}$)  dictionary elements.

We propose that SAEs can discover irreducible multi-dimensional features by
locating point sets with a low $\epsilon$-mixture index. For instance, if
$X_{i,l}$ contains an irreducible two-dimensional feature ${\mathbf{f}}$ (see
2), and ${\mathbf{D}}$ includes just two dictionary elements spanning
${\mathbf{f}}$, both elements must have nonzero activations post-ReLU to
perfectly reconstruct ${\mathbf{f}}$ (otherwise ${\mathbf{f}}$ is a mixture).
Thus the Jaccard similarity of the sets of tokens that these two dictionary
elements fire on is likely to be high. On the other hand, if ${\mathbf{D}}$
contains more than two dictionary elements that span ${\mathbf{f}}$, then the
Jaccard similarity may be lower. However, since there are now many dictionary
elements with a high projection in the two dimensional feature space, the
cosine similarity of the dictionary elements is likely to be high.

Thus for a two dimensional irreducible feature ${\mathbf{f}}$, we expect there
to be groups of dictionary elements with either high cosine or Jaccard
similarity corresponding to ${\mathbf{f}}$. We expect this observation to be
true for higher dimensional irreducible features as well. Note that some
clusters may correspond to separable features ${\mathbf{f}}$, as this
technique only finds features that are not mixtures. This suggests a natural
approach to using sparse autoencoders to search for irreducible multi-
dimensional features:

1\. Cluster dictionary elements by their pairwise cosine similarity or Jaccard
similarity.

2\. For each cluster, run the SAEs on all $\mathbf{x}_{i,l}\in X_{i,l}$ and
ablate all dictionary elements not in the cluster. This will give the
reconstruction of each $\mathbf{x}_{i,l}$ restricted to the cluster found in
step $1$ (if no cluster dictionary elements are non-zero for a given point, we
ignore the point).

3\. Examine the resulting reconstructed activation vectors for irreducible
multi-dimensional features, especially ensuring that the reconstruction is not
separable. This step can be done manually by visually inspecting the PCA
projections for known irreducible multi-dimensional structures (e.g. circles,
see Fig. 2) or automatically by passing the PCA projections to the tests for
Definition 3.

Table 1: Aggregate model accuracy on days of the week and months of the year
modular arithmetic tasks. Performance broken down by problem instance in
Appendix H.

Model | Weekdays | Months
---|---|---
Llama 3 8B | 29 / 49 | 143 / 144
Mistral 7B | 31 / 49 | 125 / 144
GPT-2 | 8 / 49 | 10 / 144

Figure 3: Top two PCA components on the $\alpha$ token. Colors show $\alpha$.
Left: Layer $30$ of Mistral on Weekdays. Right: Layer $5$ of Llama on Months.

This method succeeds on toy datasets of synthetic irreducible multi-
dimensional features; see Appendix D.111Code for reproducing experiments:
https://github.com/JoshEngels/MultiDimensionalFeatures We apply this method to
language models using GPT-2 [41] SAEs trained by Bloom for every layer [5] and
Mistral 7B [21] SAEs that we train on layers 8, 16, and 24 (training details
in Appendix E). Clustering details are in Appendix F, including comments on
scalability.

Strikingly, we reconstruct irreducible multi-dimensional features that are
interpretable circles: in GPT-2, days, months, and years are arranged
circularly in order (see Fig. 1); in Mistral 7B, days and months are arranged
circularly in order (see Fig. 13).

##  5 Circular Representations in Large Language Models

In this section, we seek tasks in which models use the multi-dimensional
features we discovered in Section 4, thereby providing evidence that these
representations are indeed the fundamental unit of computation for some
problems. Inspired by prior work studying circular representations in modular
arithmetic [27], we define two prompts that represent “natural” modular
arithmetic tasks:

Weekdays task: “Let’s do some day of the week math. Two days from Monday is”
Months task: “Let’s do some calendar math. Four months from January is”

For Weekdays, we range over the $7$ days of the week and durations between $1$
and $7$ days to get $49$ prompts. For Months, we range over the $12$ months of
the year and durations between $1$ and $12$ months to get $144$ prompts.
Mistral 7B and Llama 3 8B [2] achieve reasonable performance on the Weekdays
task and excellent performance on the Months task (measured by comparing the
highest logit valid token against the ground truth answer), as summarized in
Table 1. Interestingly, although these problems are equivalent to modular
arithmetic problems $\alpha+\beta\equiv\;?\ (\mathrm{mod}\ m)$ for $m=7,12$,
both models get trivial accuracy on plain modular addition prompts, e.g.
“$5+3\ (\mathrm{mod}\ 7)\equiv$”. Finally, although GPT-2 has circular
representations, it gets trivial accuracy on Weekdays and Months.

To simplify discussion, let $\alpha$ be the day of the week or month of the
year token (e.g. “Monday” or “April”), $\beta$ be the duration token (e.g.
“four” or “eleven”), and $\gamma$ be the target ground truth token the model
should predict, such that (abusing notation) we have $\alpha+\beta=\gamma$.
Let the prompts of the task be parameterized by $j$, such that the $j$th
prompt asks about $\alpha_{j}$, $\beta_{j}$, and $\gamma_{j}$.

###  5.1 Intervening on Circular Day and Month Representations

We first confirm that Llama 3 8B and Mistral 7B have circular representations
of $\alpha$ by examining the PCA of hidden states across prompts at various
layers on the $\alpha$ token. We plot two of these in Fig. 3 and show all
layers in Fig. 14 in the appendix. These plots show circular representations
as the highest varying two components in the model’s representation of
$\alpha$ at many layers.

Figure 4: Visual representation of the intervention process.

Figure 5: Mean and 96% error bars for intervention methods.

We now experiment with intervening on these circular representations. We base
our experiments on the common interpretability technique of activation
patching, which replaces activations from a “dirty” run of the model with the
corresponding activations from a “clean” run [48]. Activation patching
empirically tests whether a specific model component, position, and/or
representation has a causal influence on the model’s output. We employ a
custom subspace patching method to allow testing for whether a specific
circular subspace of a hidden state is sufficient to causally explain model
output. Specifically, our patching technique relies on the following steps
(visualized in Fig. 5):

1\. Find a subspace with a circle to intervene on: Using a PCA reduced
activation subspace to avoid overfitting, we train a “circular probe” to
identify representations which exhibit strong circular patterns. More
formally, let $\mathbf{x}^{j}_{i,l}$ be the hidden state at layer $l$ token
position $i$ for prompt $j$. Let ${\mathbf{W}}_{i,l}\in\mathbb{R}^{k\times d}$
be the matrix consisting of the top $k$ principal component directions of
$\mathbf{x}^{j}_{i,l}$. In our experiments, we set $k=5$. We learn a linear
probe ${\mathbf{P}}\in\mathbb{R}^{2,k}$ from ${\mathbf{W}}_{i,l}\cdot X_{i,l}$
to a unit circle in $\alpha$. In other words, if
$\texttt{circle}(\alpha)=[\cos(2\pi\alpha/7),\sin(2\pi\alpha/7)]$ for Weekdays
and $\texttt{circle}(\alpha)=[\cos(2\pi\alpha/12),\sin(2\pi\alpha/12)]$ for
Months, ${\mathbf{P}}$ is defined as follows:

$\displaystyle{\mathbf{P}}=\operatorname*{arg\,min}_{{\mathbf{P}}^{\prime}\in\mathbb{R}^{2,k}}\sum_{\mathbf{x}^{j}_{i,l}}\left\lVert{\mathbf{P}}^{\prime}\cdot{\mathbf{W}}_{i,l}\cdot\mathbf{x}^{j}_{i,l}-\texttt{circle}(\alpha)\right\rVert_{2}^{2}$
(5)

2\. Intervene on the subspace: Say our initial prompt had $\alpha=\alpha_{j}$
and we are intervening with $\alpha=\alpha_{j^{\prime}}$. In this step, we
replace the model’s projection on the subspace
${\mathbf{P}}\cdot{\mathbf{W}}_{i,l}$, which will be close to
$\texttt{circle}(\alpha_{j}$), with the “clean” point
$\texttt{circle}(\alpha_{j^{\prime}})$. Note that we do not use the hidden
state $\mathbf{x}^{j^{\prime}}_{i,l}$ from the “clean” run, only the “clean”
label $\alpha_{j^{\prime}}$. In practice, other subspaces of
$\mathbf{x}^{j}_{i,l}$ may be used by the model in alternate pathways to
compute the answer. To avoid this affecting the intervention, we average out
all subspaces not in the intervened subspace. Letting $\overline{{\bf
x}_{i,l}}$ be the average of ${\bf x}^{j}_{i,l}$ across all prompts indexed by
$j$ and ${\mathbf{P}}^{+}$ be the pseudoinverse of ${\mathbf{P}}$, we
intervene via the formula

$\displaystyle\mathbf{x}^{j^{*}}_{i,l}=\overline{{\bf
x}_{i,l}}+{{\mathbf{W}}_{i,l}}^{T}{\mathbf{P}}^{+}(\texttt{circle}(\alpha_{j^{\prime}})-\overline{{\bf
x}_{i,l}})$ (6)

We run our patching on all $49$ Weekday problems and $144$ Month problems and
use as “clean” runs the $6$ or $11$ other possible values for $\beta$,
resulting in a total of $49*6$ patching experiments for Weekdays and $144*11$
patching experiments for Months. We also run baselines where we

Figure 6: Off distribution interventions on Mistral layer $5$ on the Weekdays
task. The color corresponds to the highest logit $\gamma$ after performing our
circular subspace intervention process on that point.

(1) replace the entire subspace corresponding to the first $5$ PCA dimensions
with the corresponding subspace from the clean run, (2) replace the entire
layer with the corresponding layer from the clean run, and (3) replace the
entire layer with the average across the task. The metric we use is average
logit difference across all patching experiments between the original correct
token ($\alpha_{j}$) and the target token ($\alpha_{j^{\prime}}$). See Fig. 5
for these interventions on all layers of Mistral 7B and Llama 3 8B on Weekdays
and Months.

The main takeaway from Fig. 5 is that circular subspaces are causally
implicated in computing $\gamma$, especially for Weekdays. Across all models
and tasks, early layer interventions on the circular subspace have almost the
same intervention effect as patching the entire layer, and are sometimes even
better than patching the top PCA dimensions from the clean problem. Note that
patching experiments in Appendix I show $\alpha$ is copied to the final token
on layers $15$ to $17$, which is why interventions drop off there.

To investigate exactly how models use the circular subspace, we perform off
distribution interventions. We modify Eq. 6 so that instead of intervening on
the circumference $\texttt{circle}(\alpha)$, we sweep over a grid of positions
$(r,\theta)$ within the circle:

$\displaystyle\mathbf{x}^{j^{*}}_{i,l}=\overline{{\bf
x}_{i,l}}+{{\mathbf{W}}_{i,l}}^{T}{\mathbf{P}}^{+}[r\cos(\theta),r\sin(\theta)]^{T}-\overline{{\bf
x}_{i,l}})$ (7)

We intervene with
$r\in[0,0.1,\ldots,2],\theta\in[0,2\pi/100,\ldots,198\pi/100]$ and record the
highest logit $\gamma$ after the forward pass. Fig. 6 displays these results
on Mistral layer $5$ for $\beta\in[2,3,45]$. They imply that Mistral treats
the circle as a multi-dimensional representation with $\alpha$ encoded in the
angle.

Figure 7: Top two PCA components of residual errors after EVR with one-hot in
$\alpha$ and $\beta$. Mistral 7B Weekdays, layer $25$, final token. Colored by
$\gamma$.

###  5.2 Decomposing Hidden States

To isolate the rough circuit for Weekdays and Months, we perform layer-wise
activation patching on $40$ random pairs of prompts. The results, displayed in
Appendix I, show that the circuit to compute $\gamma$ consists of MLPs on top
of the $\alpha$ and $\beta$ tokens, a copy to the token before $\gamma$, and
further MLPs there (roughly similar to prior work studying arithmetic circuits
[43]). Moreover, fine-grained patching in Appendix I shows that there are just
a few responsible attention heads for the writes to the token before $\gamma$.
However, patching alone cannot tell use how or where $\gamma$ is represented.

We now introduce a new technique for empirically explaining hidden
representations in algorithmic problems: explanation via regression (EVR).
Given a set of tokens $T$ with a corresponding set of hidden states $X_{i,l}$,
we explain the variance in $X_{i,l}$ by adding together hand-chosen functions
of $t$. This gives us an explanation of what the transformation $T\rightarrow
X_{i,l}$ computes. For a given choice of explanation functions
$\\{{\mathbf{g}}_{i}(t)\\}$, the $r^{2}$ value of a linear regression from
$\\{{\mathbf{g}}_{i}(t)\\}$ to $X_{i,l}$ gives a measure of the explained
variance in $X_{i,l}$. But what functions ${\mathbf{g}}_{i}$ should we choose?

Figure 8: EVR residual RGB plots on Mistral hidden states on the Weekdays
final token, layers 17 to 29. From top to bottom, we show each residual RGB
plot after adding the function(s) ${\mathbf{g}}_{i}$ labelled just underneath,
as well as the resulting $r^{2}$ value. We write “tmr” meaning “tomorrow” for
$\beta=1$. We also write “circle for $x$” meaning the inclusion of two
functions ${\mathbf{g}}_{i}(x)=\\{\cos,\sin\\}(2\pi x/7)$.

We build a list of ${\mathbf{g}}_{i}$ iteratively and greedily. At each
iteration, we perform a linear regression with the current list
${\mathbf{g}}_{1}\dots\mathbf{g}_{k}$, visualize and interpret the residual
prediction errors, and build a new function ${\mathbf{g}}_{k+1}$ representing
these errors to add to the list. Once most variance is explained, we can
conclude that ${\mathbf{g}}_{1},\dots,{\mathbf{g}}_{k}$ constitutes the
entirety of what is represented in the hidden states. This information tells
us what can and cannot be extracted via a linear probe, without having to
train any probes. Furthermore, if we treat each ${\mathbf{g}}_{i}$ as a
feature (see Definition 1), then the linear regression coefficients tell us
which directions in $X_{i,l}$ these features are represented in, connecting
back to 2.

We apply EVR to Months and Weekdays. Since $X_{i,l}$ consists of modular
addition problems with two inputs $\alpha$ and $\beta$, we can visualize the
errors as we iteratively construct ${\mathbf{g}}_{1},\ldots,{\mathbf{g}}_{k}$
by making a heatmap with $\alpha$ and $\beta$ on the two axes, where the color
shows what kind of error is made. More specifically, we take the top 3 PCA
components of the error distribution and assign them to the colors red, green,
and blue. We call the resulting heatmap a residual RGB plot. Errors that
depend primarily on $\alpha$, $\beta$, or $\gamma$ show up as horizontal,
vertical, or diagonal stripes on the residual RGB plot.

In Fig. 8, we perform EVR on the layer 17-29 hidden states of Mistral 7B on
the Weekdays task; additional deconstructions are in Appendix I. We find that
a circle in $\gamma$ develops and grows in explanatory power; we plot the
layer $25$ residuals after explaining with one hot functions in $\alpha$ and
$\beta$ (i.e.
${\mathbf{g}}_{1}=[\alpha=0],{\mathbf{g}}_{2}=[\beta=1],{\mathbf{g}}_{3}=[\alpha=1],\ldots$)
in Fig. 7 to show this incredibly clear circle in $\gamma$. This suggests that
the models may be generating $\gamma$ by using a trigonometry based algorithm
like the “clock” [35] or “pizza” [49] algorithm in late MLP layers.

##  6 Discussion

Our work proposes a significant refinement to the simple one-dimensional
linear representation hypothesis. While previous work has convincingly shown
the existence of one-dimensional features, we find evidence for multi-
dimensional representations that are non-separable and irreducible, requiring
us to generalize the notion of a feature to higher dimensions. Fortunately, we
find that existing feature extraction methodologies like sparse autoencoders
can readily be applied to discover multi-dimensional representations. Although
multi-dimensional representations may be more complicated, we believe that
uncovering the true (perhaps multi-dimensional) nature of model
representations is necessary for discovering the underlying algorithms that
use these representations. Ultimately, we aim to turn complex circuits in
future more-capable models into formally verifiable programs [46, 10], which
requires the ground truth “variables” of language models; we believe this work
takes an important step towards discovering these variables. Finally, we do
not anticipate adverse impacts of our work, as it focuses only on deepening
our understanding of language model representations.

##### Limitations:

It is unclear why we did not find more interpretable multi-dimensional
features: are there truly not that many, or is our clustering technique
failing to find them? Our definition for an irreducible feature (Definition 2)
also had to be relaxed to hold in practice (Definition 3). Thus, although this
work provides preliminary evidence for the multi-dimensional superposition
hypothesis (2), it is still unclear if this theory provides the best
description for the representations models use. We also did not find a small
subset of MLP neurons implementing the “clock” algorithm, leaving as an open
question to what extent models use multi-dimensional representations for
algorithmic tasks. Finally, we only ran experiments on models up to size 8B;
however, recent work [20] implies representations may become universal with
growing model size.

## Acknowledgments and Disclosure of Funding

We thank (in alphabetical order) Dowon Baek, Kaivu Hariharan, Vedang Lad,
Ziming Liu, and Tony Wang for helpful discussions and suggestions. This work
is supported by Erik Otto, Jaan Tallinn, the Rothberg Family Fund for
Cognitive Science, the NSF Graduate Research Fellowship (Grant No. 2141064),
and IAIFI through NSF grant PHY-2019786.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] AI@Meta.  Llama 3 model card, 2024.
  * [3] Anthropic.  The claude 3 model family: Opus, sonnet, haiku.  Technical report, Anthropic, 2024.
  * [4] Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, et al.  Interpreting neural networks through the polytope lens.  arXiv preprint arXiv:2211.12312, 2022.
  * [5] Joseph Bloom.  Open source sparse autoencoders for all residual stream layers of gpt2 small.  https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream, 2024.
  * [6] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah.  Towards monosemanticity: Decomposing language models with dictionary learning.  Transformer Circuits Thread, 2023.  https://transformer-circuits.pub/2023/monosemantic-features/index.html.
  * [7] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [8] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso.  Towards automated circuit discovery for mechanistic interpretability.  Advances in Neural Information Processing Systems, 36:16318–16352, 2023.
  * [9] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey.  Sparse autoencoders find highly interpretable features in language models.  arXiv preprint arXiv:2309.08600, 2023.
  * [10] David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, et al.  Towards guaranteed safe ai: A framework for ensuring robust and reliable ai systems.  arXiv preprint arXiv:2405.06624, 2024.
  * [11] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.  Toy models of superposition.  Transformer Circuits Thread, 2022.  https://transformer-circuits.pub/2022/toy_model/index.html.
  * [12] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2020.
  * [13] Semyon Aranovich Gershgorin.  über die abgrenzung der eigenwerte einer matrix.  Izvestiya Rossiĭskoi akademii nauk. Seriya matematicheskaya, (6):749–754, 1931.
  * [14] Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy.  Successor heads: Recurring, interpretable attention heads in the wild.  arXiv preprint arXiv:2312.09230, 2023.
  * [15] Wes Gurnee and Max Tegmark.  Language models represent space and time.  arXiv preprint arXiv:2310.02207, 2023.
  * [16] Michael Hanna, Ollie Liu, and Alexandre Variengien.  How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.  Advances in Neural Information Processing Systems, 36, 2024.
  * [17] Benjamin Heinzerling and Kentaro Inui.  Monotonic representation of numeric properties in language models.  arXiv preprint arXiv:2403.10381, 2024.
  * [18] Nicholas J. Higham.  Singular value inequalities.  https://nhigham.com/2021/05/04/singular-value-inequalities/, May 2021.
  * [19] Bill Johnson (https://mathoverflow.net/users/2554/bill johnson).  Almost orthogonal vectors.  MathOverflow.  URL:https://mathoverflow.net/q/24873 (version: 2010-05-16).
  * [20] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.  The platonic representation hypothesis.  arXiv preprint arXiv:2405.07987, 2024.
  * [21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [22] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch.  On the origins of linear representations in large language models.  arXiv preprint arXiv:2403.03867, 2024.
  * [23] William B. Johnson and Joram Lindenstrauss.  Extensions of lipschitz mappings into a hilbert space.  In Conference in modern analysis and probability (New Haven, Conn., 1982), volume 26 of Contemporary Mathematics, pages 189–206. American Mathematical Society, Providence, RI, 1984.
  * [24] Juno Kim and Taiji Suzuki.  Transformers learn nonlinear features in context: Nonconvex mean-field dynamics on the attention landscape.  arXiv preprint arXiv:2402.01258, 2024.
  * [25] Silvio Lattanzi, Thomas Lavastida, Kefu Lu, and Benjamin Moseley.  A framework for parallelizing hierarchical clustering methods.  In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part I, pages 73–89. Springer, 2020.
  * [26] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Emergent world representations: Exploring a sequence model trained on a synthetic task.  arXiv preprint arXiv:2210.13382, 2022.
  * [27] Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams.  Towards understanding grokking: An effective theory of representation learning.  Advances in Neural Information Processing Systems, 35:34651–34663, 2022.
  * [28] Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller.  Sparse feature circuits: Discovering and editing interpretable causal graphs in language models.  arXiv preprint arXiv:2403.19647, 2024.
  * [29] Samuel Marks and Max Tegmark.  The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.  arXiv preprint arXiv:2310.06824, 2023.
  * [30] Eric J Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark.  Opening the ai black box: program synthesis via mechanistic interpretability.  arXiv preprint arXiv:2402.05110, 2024.
  * [31] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  Advances in neural information processing systems, 26, 2013.
  * [32] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig.  Linguistic regularities in continuous space word representations.  In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013.
  * [33] Depen Morwani, Benjamin L Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade.  Feature emergence via margin maximization: case studies in algebraic tasks.  arXiv preprint arXiv:2311.07568, 2023.
  * [34] Neel Nanda and Joseph Bloom.  Transformerlens.  https://github.com/TransformerLensOrg/TransformerLens, 2022.
  * [35] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt.  Progress measures for grokking via mechanistic interpretability.  arXiv preprint arXiv:2301.05217, 2023.
  * [36] Neel Nanda, Andrew Lee, and Martin Wattenberg.  Emergent linear representations in world models of self-supervised sequence models.  arXiv preprint arXiv:2309.00941, 2023.
  * [37] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.  Zoom in: An introduction to circuits.  Distill, 2020.  https://distill.pub/2020/circuits/zoom-in.
  * [38] Kiho Park, Yo Joong Choe, and Victor Veitch.  The linear representation hypothesis and the geometry of large language models.  arXiv preprint arXiv:2311.03658, 2023.
  * [39] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [40] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 2019.
  * [42] Adam Shai, Paul Riechers, Lucas Teixeira, Alexander Oldenziel, and Sarah Marzen.  Transformers represent belief state geometry in their residual stream.  https://www.alignmentforum.org/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their, 2024.
  * [43] Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan.  A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035–7052, 2023.
  * [44] Aaquib Syed, Can Rager, and Arthur Conmy.  Attribution patching outperforms automated circuit discovery.  arXiv preprint arXiv:2310.10348, 2023.
  * [45] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [46] Max Tegmark and Steve Omohundro.  Provably safe systems: the only path to controllable agi.  arXiv preprint arXiv:2309.01933, 2023.
  * [47] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  arXiv preprint arXiv:2211.00593, 2022.
  * [48] Fred Zhang and Neel Nanda.  Towards best practices of activation patching in language models: Metrics and methods.  arXiv preprint arXiv:2309.16042, 2023.
  * [49] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas.  The clock and the pizza: Two stories in mechanistic explanation of neural networks.  Advances in Neural Information Processing Systems, 36, 2024.

##  Appendix A Proofs

We will first prove a lemma that will help us prove 1.

######  Lemma 1.

Pick $n$ pairwise $\delta$-orthogonal unit vectors in
$\mathbf{v}_{1},\ldots,\mathbf{v}_{n}\in\mathbb{R}^{d}$. Let
$\mathbf{y}\in\mathbb{R}^{d}$ be a unit norm vector that is a linear
combination of unit norm vectors $\mathbf{v}_{1},\ldots,\mathbf{v}_{n}$ with
coefficients $z_{1}\ldots,z_{n}\in\mathbb{R}$. We can write
${\mathbf{A}}=[\mathbf{v}_{1},\ldots,\mathbf{v}_{n}]$ and
$\mathbf{z}=[z_{1},\ldots,z_{n}]^{T}$, so that we have
$\mathbf{y}=\sum_{k=1}^{n}z_{k}\mathbf{v}_{k}={\mathbf{A}}\mathbf{z}^{T}$ with
$\lVert\mathbf{y}\rVert_{2}=1$. Then,

$\left|\sum_{k=1}^{n}\mathbf{z}_{k}\right|=\lVert\mathbf{z}\rVert_{1}\leq\sqrt{\frac{n}{1-\delta
n}}$

###### Proof.

We will first bound the $L_{2}$ norm of $\mathbf{z}$. If $\sigma_{n}$ is the
minimum singular value of ${\mathbf{A}}$, then we have via standard singular
value inequalities [18]

$\sigma_{n}\leq\frac{\lVert\mathbf{y}\rVert_{2}}{\lVert\mathbf{z}\rVert_{2}}\implies\lVert\mathbf{z}\rVert_{2}\leq\frac{\lVert\mathbf{y}\rVert_{2}}{\sigma_{n}}=\frac{1}{\sigma_{n}}$

Thus we now lower bound $\sigma_{n}$. The singular values are the square roots
of the eigenvalues of the matrix ${\mathbf{A}}^{T}{\mathbf{A}}$, so we now
examine ${\mathbf{A}}^{T}{\mathbf{A}}$. Since all elements of ${\mathbf{A}}$
are unit vectors, the diagonal of ${\mathbf{A}}^{T}{\mathbf{A}}$ is all ones.
The off diagonal elements are dot products of pairs of $\delta$-orthogonal
vectors, and so are within the range $[-\delta,\delta]$. Then by the
Gershgorin circle theorem [13], all eigenvalues $\lambda_{i}$ of
${\mathbf{A}}^{T}{\mathbf{A}}$ are in the range

$(1-\delta(n-1),1+\delta(n-1))$

In particular, $\sigma_{n}^{2}=\lambda_{n}\geq 1-\delta(n-1)$, and thus
$\sigma_{n}\geq\sqrt{1-\delta(n-1)}$. Plugging into our upper bound for
$\lVert\mathbf{z}\rVert_{2}$, we have that $\lVert\mathbf{z}\rVert_{2}\leq
1/\sqrt{1-\delta(n-1)}$. Finally, the largest $L_{1}$ for a point on an
$n$-hypersphere of radius $r$ is when all dimensions are equal and such a
point has magnitude $\sqrt{n}r$, so

$\lVert\mathbf{z}\rVert_{1}\leq\sqrt{\frac{n}{1-\delta(n-1)}}\leq\sqrt{\frac{n}{1-\delta
n}}$

∎

See 1

###### Proof.

By the JL lemma [23, 19], for any $d$ and $\delta$, we can choose
$e^{Cd\delta^{2}}$ $\delta$-orthogonal unit vectors in $\mathbb{R}^{d}$
indexed as $\mathbf{v}_{i}$, for some constant $C$. Let
${\mathbf{A}}_{i}=[\mathbf{v}_{d_{\max}*i},\ldots,\mathbf{v}_{d_{\max}*i+n_{i}-1}]$
where each element in the brackets is a column. Then by construction all
${\mathbf{A}}_{i}$ are matrices composed of unique $\delta$-orthogonal vectors
and there are $\frac{1}{d_{\max}}e^{Cd\delta^{2}}$ matrices
${\mathbf{A}}_{i}$.

Now, consider two of these matrices
${\mathbf{A}}_{i}=[\mathbf{v}_{1},\ldots,\mathbf{v}_{n_{i}}]$ and
${\mathbf{A}}_{j}=[\mathbf{u}_{1},\ldots,\mathbf{u}_{n_{j}}]$, $i\neq j$; we
will prove that they are $f(\delta)$-orthogonal for some function $f$. Let
$\mathbf{y}_{i}=\sum_{k=1}^{n_{i}}z_{i,k}\mathbf{v}_{k}$ be a vector in the
colspace of ${\mathbf{A}}_{i}$ and
$\mathbf{y}_{j}=\sum_{k=1}^{n_{j}}z_{j,k}\mathbf{u}_{k}$ be a vector in the
colspace of ${\mathbf{A}}_{j}$, such that $\mathbf{y}_{i}$ and
$\mathbf{y}_{j}$ are unit vectors. To prove $f(\delta)$-orthogonality, we must
bound the absolute dot product between $\mathbf{y}_{i}$ and $\mathbf{y}_{j}$:

$\displaystyle\left|\mathbf{y}_{i}\cdot\mathbf{y}_{j}\right|$
$\displaystyle=\left|\left(\sum_{k=1}^{n_{i}}z_{i,k}\mathbf{v}_{k}\right)\cdot\left(\sum_{k=1}^{n_{j}}z_{j,k}\mathbf{u}_{k}\right)\right|$
$\displaystyle=\left|\sum_{k_{1}=1}^{n_{i}}\sum_{k_{2}=1}^{n_{j}}\left(z_{i,k_{1}}\mathbf{v}_{k_{1}}\right)\cdot\left(z_{j,k_{2}}\mathbf{u}_{k_{2}}\right)\right|$
$\displaystyle\leq\sum_{k_{1}=1}^{n_{i}}\sum_{k_{2}=1}^{n_{j}}\left|z_{i,k_{1}}z_{j,k_{2}}\right|\left|\mathbf{v}_{k_{1}}\cdot\mathbf{u}_{k_{2}}\right|$
Triangle Inequality
$\displaystyle\leq\sum_{k_{1}=1}^{n_{i}}\sum_{k_{2}=1}^{n_{j}}\left|z_{i,k_{1}}z_{j,k_{2}}\right|\delta$
All $\mathbf{v}_{i},\mathbf{u}_{j}$ are $\delta$ orthogonal
$\displaystyle=\delta\sum_{k_{1}=1}^{n_{i}}\sum_{k_{2}=1}^{n_{j}}\left|z_{i,k_{1}}z_{j,k_{2}}\right|$
$\displaystyle=\delta\left|\sum_{k=1}^{n_{i}}z_{i,k}\right|\left|\sum_{k=1}^{n_{j}}z_{j,k}\right|$
Factoring the product $\displaystyle\leq\delta\sqrt{\frac{n_{i}}{1-\delta
n_{i}}}\sqrt{\frac{n_{j}}{1-\delta n_{j}}}$ By Lemma 1
$\displaystyle\leq\frac{\delta d_{\max}}{1-\delta d_{\max}}$ $n_{i},n_{j}\leq
d_{\max}$ by assumption

Thus ${\mathbf{A}}_{i}$ and ${\mathbf{A}}_{j}$ are $f(\delta)$-orthogonal for
$f(\delta)=\delta d_{\max}/(1-\delta d_{\max})$, and so it is possible to
choose $\frac{1}{d_{\max}}e^{Cd\delta^{2}}$ pairwise $f(\delta)$-orthogonal
projection matrices. Remapping the variable $\delta$ with $\delta\mapsto
f^{-1}(\delta)=\delta/(d_{\max}(1+\delta))$, we find that it is possible to
choose $\frac{1}{d_{\max}}e^{Cd\delta^{2}/((1+\delta)^{2}d_{\max}^{2})}$
pairwise $\delta$-orthogonal projection matrices. Because $1+\delta$ is at
most $2$ with $\delta\in(0,1)$, we can further simplify the exponent and find
that it is possible to choose
$\frac{1}{d_{\max}}e^{C(d/d_{\max}^{2})\delta^{2}/4}$ pairwise
$\delta$-orthogonal projection matrices. Absorbing the $4$ into the constant
$C$ finishes the proof. ∎

##  Appendix B More on Reducibility

###  B.1 Additional Intuition for Definitions

Here, we present some extra intuition and high level ideas for understanding
our definitions and the motivation behind them. Roughly, we intend for our
definitions in the main text to identify representations in the model that
describe an object or concept in a way that fundamentally takes multiple
dimensions. We operationalize this as finding a subspace of representations
that 1. has basis vectors that “always co-occur” no matter the orientation 2.
is not made up of combinations of independent lower-dimensional features.

1\. The first condition is met by the mixture part of our definition. The
feature in question should be part of an irreducible manifold, and so should
“fill” a plane or hyperplane. There shouldn’t be any part of the plane where
the probability distribution of the feature is concentrated, because this
region is then likely part of a lower dimensional feature. The idea of this
part of the definition is to capture multi-dimensional objects; if the entire
multi-dimensional space is truly being used to represent a high-dimensional
object, then the representations for the object should be “spread out”
entirely through the space.

2\. The second condition is met by the separability part of our definition.
This part of the definition is intended to rule out features that co-occur
frequently but are fundamentally not describing the same object or concept.
For example, latitude and longitude are not a mixture in that they frequently
co-occur, but we do not think it is necessarily correct to say they are part
of the same multi-dimensional feature because they are independent.

###  B.2 Empirical Irreducible Feature Test Details

Our tests for reducibility require the computation of two quantities
$S({\mathbf{f}})$ for the separability index and $M_{\epsilon}({\mathbf{f}})$
for the $\epsilon$-mixture index. We describe how we compute each index in the
following two subsections.

####  B.2.1 Separability Index

We define the separability index in Equation 2 as

$\displaystyle S({\mathbf{f}})=$ $\displaystyle\min
I({\mathbf{a}};{\mathbf{b}})$

where the min is over rotations $\mathbf{R}$ used to split
${\mathbf{f}}^{\prime}=\mathbf{R}{\mathbf{f}}+{\mathbf{c}}$ into
${\mathbf{a}}$ and ${\mathbf{b}}$. In two dimensions, the rotation is defined
by a single angle, so we can iterate over a grid of 1000 angles and estimate
the mutual information between ${\mathbf{a}}$ and ${\mathbf{b}}$ for each
angle. We first normalize ${\mathbf{f}}$ by subtracting off the mean and then
dividing by the root mean squared norm of ${\mathbf{f}}$ (and multiplying by
$\sqrt{2}$ since the toy datasets are in two dimensions). To estimate the
mutual information, we first clip the data ${\mathbf{f}}$ to a 6 by 6 square
centered on the origin. We then bin the points into a 40 by 40 grid, to
produce a discrete distribution $p(a,b)$. After computing the marginals $p(a)$
and $p(b)$ by summing the distribution over each axis, we obtain the mutual
information via the formula

$\displaystyle
I({\mathbf{a}};{\mathbf{b}})=\sum_{a,b}p(a,b)\log\frac{p(a,b)}{p(a)p(b)}$ (8)

####  B.2.2 $\epsilon$-Mixture Index

We define the $\epsilon$-mixture index in Equation 3 as

$\displaystyle M_{\epsilon}({\mathbf{f}})=$
$\displaystyle\max_{\mathbf{v}\in\mathbb{R}^{d_{f}},\;c\in\mathbb{R}}\mathbb{P}\left(|\mathbf{v}\cdot{\mathbf{f}}+c|<\epsilon\sqrt{\mathbb{E}[(\mathbf{v}\cdot{\mathbf{f}}+c)^{2}]}\right)$

The challenge with computing $M_{\epsilon}({\mathbf{f}})$ is to compute the
maximum. We opted to maximize via gradient descent; and we guaranteed
differentiability by softening the inequality $<$ with a sigmoid,

$\displaystyle M_{\epsilon,T}({\mathbf{f}},\mathbf{v},c)=$
$\displaystyle\mathbb{E}\left(\sigma\left(\frac{1}{T}\left(\epsilon-\frac{|\mathbf{v}\cdot{\mathbf{f}}+c|}{\sqrt{\mathbb{E}[(\mathbf{v}\cdot{\mathbf{f}}+c)^{2}]}}\right)\right)\right)$
(9)

where $T$ is a temperature, which we linearly decay from $1$ to $0$ throughout
training. We optimize for $\mathbf{v}$ and $c$ using this loss
$M_{\epsilon,T}({\mathbf{f}},\mathbf{v},c)$ using full batch gradient descent
over 10000 steps with learning rate $0.1$. With the solution
$(\mathbf{v}^{*},c^{*})$, the final value of
$M_{\epsilon,T=0}({\mathbf{f}},\mathbf{v}^{*},c^{*})$ is then our estimate of
$M_{\epsilon}({\mathbf{f}})$.

We also run the irreducibility tests on additional synthetic feature
distributions in Fig. 9a and Fig. 9b.

(a) Testing $S({\mathbf{c}})$ and $M_{\epsilon}({\mathbf{c}})$ on a reducible
feature ${\mathbf{c}}$.

(b) Testing $S(\mathbf{d})$ and $M_{\epsilon}(\mathbf{d})$ on an irreducible
feature $\mathbf{d}$

Figure 9: Testing irreducibility of synthetic features. Left in each
subfigure: Distributions of $\mathbf{x}$. For feature ${\mathbf{c}}$, 7.94%
lies within the narrow dotted lines, indicating the feature is unlikely to be
a mixture. For feature $\mathbf{d}$, 25.90% lies within the wide lines,
indicating the feature is likely a mixture. The green cross indicates the
angle $\theta$ that minimizes mutual information. Middle in each subfigure:
Histograms of the distribution of $\mathbf{v}\cdot\mathbf{x}$ with red lines
indicating a $2\epsilon$-wide region. Right in each subfigure: Mutual
information between $\mathbf{a}$ and $\mathbf{b}$ as a function of the
rotation angle $\theta$ of matrix $\mathbf{R}$. Both features have a small
($<0.5$ bits) minimum mutual information and so are likely separable.

##  Appendix C Alternative Definitions

In this section, we present an alternative definition of a reducible feature
that we considered during our work. This chiefly deals with multi-dimensional
features from the angle of computational reducibility as opposed to
statistical reducibility. In other words, this definition considers whether
representations of features on a specific set of tasks can be split up without
changing the accuracy of the task. This captures an interesting (and
important) aspect of feature reducibility, but because it requires a specific
set of prompts (as opposed to allowing unsupervised discovery) we chose not to
use it as our main definition.

Our alternative definitions consider representation spaces that are possibly
multi-dimensional, and defines these spaces through whether they can
completely explain a function $h$ on the output logits. We consider a group
theoretic approach to irreducible representations, via whether computation
involving multiple group elements can be decomposed.

###  C.1 Alternative Definition: Interventions and Representation Spaces

Assume that we restrict the input set of prompts $T=\\{\mathbf{t}^{j}\\}$ to
some subset of prompts and that we have some evaluation function $h$ that maps
from the output logit distribution of $M$ to a real number. For example, for
the Weekdays problems, $T$ is the set of $49$ prompts and $h$ could be the
$\operatorname*{arg\,max}$ over the days of week logits. Abusing notation, we
let $M$ also be the function from the layer we are intervening on; this is
always clear from context. Then we can define a representation space of
$\mathbf{x}^{j}_{i,l}$ as a subspace in which interventions always work:

######  Definition 5 (Representation Space).

Given a prompt set $T=\\{\mathbf{t}^{j}\\}$, a rank-$r$ dimensional
representation space of intermediate value $\mathbf{x}^{j}_{i,l}$ is a rank
$r$ projection matrix $P$ such that for all $j,j^{\prime}$,
$h(M((I-P)\mathbf{x}^{j}_{i,l}+P\mathbf{x}^{j^{\prime}}_{i,l}))=h(M(\mathbf{x}^{j^{\prime}}_{i,l}))$.

Note that it immediately follows that the rank $d$ dimensional matrix $I_{d}$
is trivially a rank $d$ representation space for all prompt sets $T$.

######  Definition 6 (Minimality).

A representation space $P$ of rank $r$ is minimal if there does not exist a
lower rank representation space.

A minimal representation with rank > 1 is a multi-dimensional representation.

######  Definition 7 (Alternative Reducibility).

A representation space $P$ of rank $r$ is reducible if there are orthonormal
representation spaces $P_{1}$ and $P_{2}$ (such that $P_{1}+P_{2}=P$,
$P_{1}P_{2}=0$) where

$h(M(P_{1}\mathbf{x}_{i,l}^{j})+M(P_{2}\mathbf{x}_{i,l}^{j}))=h(M(P_{1}\mathbf{x}_{i,l}^{j}+P_{2}\mathbf{x}_{i,l}^{j}))$

for all $j,j^{\prime}$.

Suppose $T$, $h$ and $M$ define the multiplication of two elements in a finite
group $G$ of order $n$. Then if we interpret the embedding vectors as the
group representations, our definition of reducibility implies to the standard
group-theoretical definition of irreducibility –– specifically, reducibility
into a tensor product representation.

##  Appendix D Toy Case of Training SAEs on Circles

To explore how SAEs behave when reconstructing irreducible features of
dimension $d_{f}>1$, we perform experiments with the following toy setup.
Inspired by the circular representations of integers that networks learn when
trained on modular addition [35, 27], we create synthetic datasets of
activations containing multiple features which are each 2d irreducible
circles.

First however, consider activations for a single circle – points uniformly
distributed on the unit circle in $\mathbb{R}^{2}$. We train SAEs on this data
with encoder
$\texttt{Enc}(\mathbf{x})=\texttt{ReLU}(\mathbf{W_{e}}(\mathbf{x}-\mathbf{b_{d}})+\mathbf{b_{e}})$
and decoder
$\texttt{Dec}(\mathbf{f})=\mathbf{W_{d}}\mathbf{f}+\mathbf{b_{d}}$. We train
SAEs with $m=2$ and $m=10$ with the Adam optimizer and a learning rate of
$10^{-3}$, sparsity penalty $\lambda=0.1$, for 20,000 steps, and a warmup of
1000 steps. In Fig. 10 we show the dictionary elements of these SAEs. When
$m=2$, the network must use both SAE features on each input point, and uses
$\mathbf{d_{b}}$ to shift the reconstructed circle so it is centered at the
origin. When $m=10$, $\mathbf{d_{b}}\approx 0$ and the features spread out
across the circle having close neighbors, with only a subset being active on
any one input.

Figure 10: SAEs trained to reconstruct a single 2d circle with $m=2$ (left)
and $m=10$ (middle and right) dictionary elements. When there are several SAE
features, there is not a natural choice feature directions, and the dictionary
elements spread out across the circle.

We now consider synthetic activations with multiple circular features. Our
data consists of points in $\mathbb{R}^{10}$, where we choose two orthogonal
planes spanned by $(\mathbf{e_{1}},\mathbf{e_{2}})$ and
$(\mathbf{e_{3}},\mathbf{e_{4}})$, respectively. With probability one half a
points is sampled uniformly on the unit circle in the
$\mathbf{e_{1}}$-$\mathbf{e_{2}}$ plane, otherwise the point will be sampled
uniformly on the unit circle in the $\mathbf{e_{3}}$-$\mathbf{e_{4}}$ plane.
We train SAEs with $m=64$ on this data with the same hyperparameters as the
single-circle case.

We now apply the procedure described in Section 4 to see if we can
automatically rediscover these circles. Encouragingly, we first find that the
alive SAE features align almost exactly with either the
$\mathbf{e_{1}}$-$\mathbf{e_{2}}$ or the $\mathbf{e_{3}}$-$\mathbf{e_{4}}$
plane. When we apply spectral clustering with $\texttt{n\\_clusters}=2$ to the
features with the pairwise angular similarities between dictionary elements as
the similarity matrix (Fig. 11, left), the two clusters correspond exactly to
the features which span each plane. As described in Section 4, given a cluster
of dictionary elements $S\subset\\{1,\ldots,m\\}$, we run a large set of
activations through the SAE, then filter out samples which don’t activate any
element in $S$. For samples which do activate an element of $S$, reconstruct
the activation while setting all SAE features not in $S$ to have a hidden
activation of zero. If some collection of SAE features together represent some
irreducible feature, we want to remove all other features from the activation
vector, and so we only allow SAE features in the collection to participate in
reconstructing the input activation. We find that this procedure almost
exactly recovers the original two circles, which encouraged us to apply this
method for discovering the features shown in Fig. 1 and Fig. 13.

Figure 11: Automatic discovery of synthetic circular features by clustering
SAE dictionary elements.

##  Appendix E Training Mistral SAEs

Our Mistral 7B [21] sparse autoencoders (SAEs) are trained on over one billion
tokens from a subset of the Pile [12] and Alpaca [39] datasets. We train our
SAEs on layers 8, 16, and 24 out of 32 total layers to maximize coverage of
the model’s representations. We use a $16\times$ expansion factor, yielding a
total of 65536 dictionary elements for each SAE.

To train our SAEs, we use an $L_{p}$ sparsity penalty for $p=1/2$ with
sparsity coefficient $\lambda=0.012$. Before an SAE forward pass, we normalize
our activation vectors to have norm $\sqrt{d_{model}}=64$ in the case of
Mistral. We do not apply a pre-encoder bias. We use an AdamW optimizer with
weight decay $10^{-3}$ and learning rate 0.0002 with a linear warm up. We
apply dead feature resampling [6] five times over the course of training to
converge on SAEs with around 1000 dead features.

##  Appendix F GPT-2 and Mistral 7B Dictionary Element Clustering

###  F.1 GPT-2-small methods and results

For GPT-2-small, we perform spectral clustering on the roughly 25k layer 7 SAE
features from [5], using pairwise angular similarities between dictionary
elements as the similarity matrix. We use $\texttt{n\\_clusters}=1000$ and
manually looked at roughly 500 of these clusters. For each cluster, we looked
at projections onto principal components 1-4 of the reconstructed activations
for these clusters. In Fig. 12, we show projections for the most interesting
clusters we identified, which appear to be circular representations of days of
the week, months of the year, and years of the 20th century.

Figure 12: Projections of days of week, months of year, and years of the 20th
century representations onto top four principal components, showing additional
dimensions of the representations than Fig. 1.

###  F.2 Mistral 7B methods and results

For Mistral 7B, our SAEs have 65536 dictionary elements and we found it
difficult to run spectral clustering on all of these at once. We therefore
develop a simple graph based clustering algorithm that we run on Mistral 7B
SAEs:

  1. 1.

Create a graph $G$ out of the dictionary elements by adding directed edges
from each dictionary element to its $k$ closest dictionary elements by cosine
similarity. We use $k=2$.

  2. 2.

Make the graph undirected by turning every directed edge into an undirected
edge.

  3. 3.

Prune edges with cosine similarity less than a threshold value $\tau$. We use
$\tau=0.5$.

  4. 4.

Return the connected components as clusters.

We run this algorithm on the Mistral 7B layer $8$ SAE ($2^{16}$ dictionary
elements) and find roughly $2700$ clusters containing between $2$ and $1000$
elements. We manually inspected roughly 2000 of these. From these, we re-
discover circular representations of days of the week and months of the year,
shown in Fig. 13. However, we did not find other obviously interesting and
clearly irreducible features.

Figure 13: Circular representations of days of the week and months of the year
which we discover with our unsupervised SAE clustering method in Mistral 7B.
Unlike similar features in GPT-2, we also find an additional “weekend”
representation in between Saturday and Sunday representations (left) and
additional representations of seasons among the months (right). For instance,
“winter” tokens activate a region of the circle in between the representation
of January and December.

As future work, we think it would be exciting to develop better clustering
techniques for SAE features. Our graph based clustering technique could likely
be improved by more recent efficient and high-quality graph based clustering
techniques, e.g. hierarchical agglomerate clustering with single-linkage [25].
Additionally, we believe we would see a large improvement by setting edge
weights to be a combination of both the cosine and Jaccard similarity of the
dictionary elements, e.g. max(cosine, Jaccard).

##  Appendix G Further Experiment Details

###  G.1 Assets Information

We use the following open source models for our experiments: Llama 3 8B [2]
(custom Llama 3 license https://llama.meta.com/llama3/license/), Mistral 7B
[21] (released under the Apache 2 License), and GPT-2 [41] (modified MIT
license, see https://github.com/openai/gpt-2/blob/master/LICENSE).

###  G.2 Machine Information

Intervention experiments were run on two V100 GPUs using less than $64$ GB of
CPU RAM; all experiments can be reproduced from our open source repository in
less than a day with this configuration. We use the TransformerLens library
[34] for intervention experiments. $\epsilon$-mixture index measurements on
toy datasets took about one minute each, on 8GB of CPU RAM. EVR experiments
take seconds on 8GB of CPU RAM and are dominated by time taken to human-
interpret the RGB plots.

GPT-2 SAE clustering and plotting was run on a cluster of heterogeneous
hardware. Spectral clustering and computing reconstructions + plotting was
done on CPUs only. We made reconstruction plots for 500 clusters, with each
taking less than 10 minutes. Mistral 7B SAE reconstruction plots were made on
the same cluster. We made roughly 2000 reconstruction plots for Mistral 7B
(and manually inspected each), with each taking less than 20 minutes to
generate. Jobs were allocated 64GB of memory each.

Mistral SAE training was run on a single V100 GPU. Initially caching
activations from Mistral 7B on one billion tokens took approximately 60 hours.
Training the SAEs on the saved activations took another 36 hours.

###  G.3 Error Bar Calculation

In Fig. 5 we report 96% error bars for all intervention methods. To compute
these error bars, we loop over all intervention methods and all layers and
compute a confidence interval for each (method, layer) pair across all
prompts. Assuming normally distributed errors, we compute error bars with the
following standard formula:

$EB=\mu\pm z*SE$

where $\mu$ is the sample mean, $z$ is the z score (slightly larger than $2$
for 96% error bars), and $SE$ is the standard error (the standard deviation
divided by the square root of the number of samples). We use standard Python
functions to compute this value.

The reason that the Months error bars are smaller than the Weekdays error bars
is because there are more Months prompts: there are $12*12*11=1584$
intervention effect values, rather than $7*7*6=294$ intervention effect
values.

(a) Mistral 7B, Weekdays (b) Llama 3 8B, Weekdays

(c) Mistral 7B, Months (d) Llama 3 8B, Months

Figure 14: Projections onto the top two PCA dimensions of model hidden states
on the $\alpha$ token show that circular representations of $\alpha$ are
present in various layers.

(a) Mistral 7B MLP Patching (b) Mistral 7B attention patching (c) Llama 3 8B
MLP patching (d) Llama 3 8B attention patching

Figure 15: Attention and MLP patching results on Weekdays. Results are
averaged over 20 different runs with fixed $\alpha$ and varying $\beta$ and 20
different runs with fixed $\beta$ and varying $\alpha$.

(a) Mistral 7B MLP Patching (b) Mistral 7B attention patching (c) Llama 3 8B
MLP patching (d) Llama 3 8B attention patching

Figure 16: Attention and MLP patching results on Months. Results are averaged
over 20 different runs with fixed $\alpha$ and varying $\beta$ and 20
different runs with fixed $\beta$ and varying $\alpha$.

##  Appendix H More Weekdays and Months Plots and Details

We show the results of Mistral 7B and Llama 3 8B on all individual instances
of Weekdays that at least one of the models get wrong in Table 2 and present a
similar table for Months in Table 3.

We show projections onto the top two PCA directions for both Mistral 7B and
Llama 3 8B in Fig. 14 on the hidden layers on top of the $\alpha$ token,
colored by $\alpha$. These are similar plots to Fig. 3, except they are on all
layers. The circular structure in $\alpha$ is visible on many—but not
all—layers. Much of the linear structure visible is due to $\beta$.

In Fig. 15 and Fig. 16, we report MLP and attention head patching results for
Weekdays and Months. We experiment on 20 pairs of problems with the same
$\alpha$ and different $\beta$ and 20 pairs of problems with the same $\beta$
and different $\alpha$, for a total of 40 pairs of problems. For each pair of
problems, we patch the MLP/attention outputs from the "clean" to the "dirty"
problem for each layer and token, and then complete the forward pass. Defining
the logit difference as the logit of the clean $\gamma$ minus the logit of the
dirty $\gamma$, we record what percent of the difference between the original
logit difference of the dirty problem and the logit difference of the clean
problem is recovered upon intervening, and average across these $40$
percentages for each layer and token. This gives us a score we call the
Average Intervention Effect.

For simplicity of presentation, we clip all of the (few) negative intervention
averages to $0$ (prior work [48] has also found negative-effect attention
heads during patching experiments).

Table 2: Weekdays finegrained results. Row ommited if both models get it correct. $\alpha$ | $\beta$ | Ground truth $\gamma$ | Mistral top $\gamma$ | Mistral correct? | Llama top $\gamma$ | Llama correct?
---|---|---|---|---|---|---
1 | 1 | Wednesday | Wednesday | Yes | Thursday | No
3 | 1 | Friday | Friday | Yes | Tuesday | No
4 | 1 | Saturday | Saturday | Yes | Thursday | No
3 | 2 | Saturday | Saturday | Yes | Tuesday | No
4 | 2 | Sunday | Sunday | Yes | Wednesday | No
5 | 2 | Monday | Monday | Yes | Tuesday | No
2 | 3 | Saturday | Friday | No | Saturday | Yes
3 | 3 | Sunday | Sunday | Yes | Tuesday | No
4 | 3 | Monday | Monday | Yes | Tuesday | No
0 | 4 | Friday | Thursday | No | Friday | Yes
3 | 4 | Monday | Monday | Yes | Tuesday | No
0 | 5 | Saturday | Friday | No | Saturday | Yes
1 | 5 | Sunday | Saturday | No | Wednesday | No
2 | 5 | Monday | Sunday | No | Monday | Yes
4 | 5 | Wednesday | Tuesday | No | Tuesday | No
6 | 5 | Friday | Thursday | No | Thursday | No
1 | 6 | Monday | Sunday | No | Thursday | No
2 | 6 | Tuesday | Monday | No | Tuesday | Yes
3 | 6 | Wednesday | Tuesday | No | Tuesday | No
4 | 6 | Thursday | Thursday | Yes | Tuesday | No
5 | 6 | Friday | Friday | Yes | Thursday | No
6 | 6 | Saturday | Thursday | No | Thursday | No
0 | 7 | Monday | Sunday | No | Tuesday | No
1 | 7 | Tuesday | Sunday | No | Tuesday | Yes
2 | 7 | Wednesday | Sunday | No | Wednesday | Yes
3 | 7 | Thursday | Sunday | No | Thursday | Yes
4 | 7 | Friday | Thursday | No | Tuesday | No
5 | 7 | Saturday | Friday | No | Saturday | Yes
6 | 7 | Sunday | Friday | No | Thursday | No
Table 3: Months finegrained results. Row ommited if both models get it correct. $\alpha$ | $\beta$ | Ground truth $\gamma$ | Mistral top $\gamma$ | Mistral correct? | Llama top $\gamma$ | Llama correct?
---|---|---|---|---|---|---
0 | 4 | May | April | No | May | Yes
6 | 4 | November | October | No | November | Yes
0 | 6 | July | June | No | July | Yes
0 | 7 | August | July | No | August | Yes
1 | 7 | September | October | No | September | Yes
3 | 7 | November | October | No | November | Yes
5 | 7 | January | December | No | January | Yes
6 | 7 | February | January | No | February | Yes
7 | 7 | March | February | No | March | Yes
9 | 7 | May | April | No | May | Yes
4 | 9 | February | February | Yes | January | No
2 | 10 | January | December | No | January | Yes
8 | 10 | July | June | No | July | Yes
1 | 11 | January | December | No | January | Yes
2 | 11 | February | December | No | February | Yes
3 | 11 | March | February | No | March | Yes
7 | 11 | July | June | No | July | Yes
8 | 11 | August | July | No | August | Yes
9 | 11 | September | August | No | September | Yes
0 | 12 | January | December | No | January | Yes

##  Appendix I Further EVR Results

In this section, we present results to support a claim that MLPs (and not
attention blocks) are responsible for computing $\gamma$. In Fig. 17, we
deconstruct states on top of the final token (before predicting $\gamma$) on
Llama 3 8B Months (we show a similar plot for the states on the final token of
Mistral 7B on Weekdays in the main text in Fig. 8. These plots show that the
value of $\gamma$ is computed on the final token around layers $20$ to $25$.
To show that this computation of occurs in the MLPs, we must show that no
attention head is copying $\gamma$ from a prior token or directly computing
$\gamma$.

We first perform a patching experiment with the same setup Fig. 16 and Fig. 15
on individual attention heads on the final token. From the patching results we
identify the top $10$ attention heads by average intervention effect. For each
attention head, we compute one EVR run with explanatory functions equal to
one-hot functions of $\alpha$ and $\beta$ (resulting in $14$ functions
${\mathbf{g}}_{i}$ for Weekdays and $24$ for Months) and one with explanatory
functions equal to one-hot functions of $\alpha$, $\beta$, and $\gamma$. We
find that for all layers before $25$, adding $\gamma$ to the explanatory
functions adds almost no explanatory power. Since we established above that
the model has already computed $\gamma$ at this point, we know that attention
heads do not participate in computing $\gamma$.

Table 4: Highest intervention effect attention heads from fine-grained
attention head patching, as well as EVR results with one hot $\alpha,\beta$
and one hot $\alpha,\beta,\gamma$.

L | H |  Average Intervention Effect |  EVR $R^{2}$ One Hot $\alpha$, $\beta$ |  EVR $R^{2}$ One Hot $\alpha$, $\beta$, $\gamma$
---|---|---|---|---
28 | 18 | 0.22 | 0.39 | 0.73
18 | 30 | 0.17 | 0.95 | 0.96
15 | 13 | 0.17 | 0.94 | 0.95
22 | 15 | 0.11 | 0.77 | 0.82
16 | 21 | 0.09 | 0.92 | 0.93
28 | 16 | 0.08 | 0.42 | 0.69
15 | 14 | 0.06 | 0.98 | 0.99
30 | 24 | 0.05 | 0.43 | 0.79
21 | 26 | 0.04 | 0.53 | 0.63
14 | 2 | 0.04 | 0.93 | 0.95
(a) Mistral 7B, Weekdays.

L | H |  Average Intervention Effect |  EVR $R^{2}$ One Hot $\alpha$, $\beta$ |  EVR $R^{2}$ One Hot $\alpha$, $\beta$, $\gamma$
---|---|---|---|---
17 | 0 | 0.18 | 0.98 | 0.99
17 | 1 | 0.08 | 0.98 | 0.98
19 | 10 | 0.08 | 0.95 | 0.96
30 | 17 | 0.07 | 0.85 | 0.90
17 | 3 | 0.07 | 0.93 | 0.95
17 | 27 | 0.06 | 1.00 | 1.00
31 | 22 | 0.05 | 0.37 | 0.78
21 | 9 | 0.04 | 0.73 | 0.78
20 | 28 | 0.04 | 1.00 | 1.00
30 | 16 | 0.04 | 0.73 | 0.85
(b) Llama 3 8B, Weekdays.

L | H |  Average Intervention Effect |  EVR $R^{2}$ One Hot $\alpha$, $\beta$ |  EVR $R^{2}$ One Hot $\alpha$, $\beta$, $\gamma$
---|---|---|---|---
20 | 28 | 0.15 | 0.76 | 0.76
17 | 0 | 0.10 | 0.77 | 0.77
25 | 14 | 0.08 | 0.19 | 0.61
17 | 1 | 0.07 | 0.80 | 0.82
17 | 3 | 0.06 | 0.71 | 0.71
31 | 22 | 0.06 | 0.12 | 0.67
17 | 27 | 0.05 | 0.58 | 0.58
19 | 4 | 0.05 | 0.40 | 0.66
19 | 10 | 0.04 | 0.62 | 0.62
30 | 26 | 0.04 | 0.51 | 0.62
(c) Mistral 7B, Months.

L | H |  Average Intervention Effect |  EVR $R^{2}$ One Hot $\alpha$, $\beta$ |  EVR $R^{2}$ One Hot $\alpha$, $\beta$, $\gamma$
---|---|---|---|---
15 | 13 | 0.26 | 0.62 | 0.62
16 | 21 | 0.17 | 0.76 | 0.76
18 | 30 | 0.13 | 0.77 | 0.77
28 | 18 | 0.11 | 0.13 | 0.52
28 | 16 | 0.07 | 0.13 | 0.52
21 | 25 | 0.05 | 0.65 | 0.70
15 | 14 | 0.03 | 0.72 | 0.72
17 | 26 | 0.02 | 0.77 | 0.77
31 | 1 | 0.02 | 0.11 | 0.57
21 | 24 | 0.02 | 0.30 | 0.45
(d) Llama 3 8B, Months.

Figure 17: Iterative deconstruction of hidden state representations on the
final token on Llama 3 8B, Months.