# Uncovering Intermediate Variables in Transformers using Circuit Probing

Michael A. Lepori1, Thomas Serre2, & Ellie Pavlick1
1Department of Computer Science, 2Carney Institute for Brain Science
Brown University
Providence, RI 02906, USA
{michael_lepori, thomas_serre<EMAIL_ADDRESS>

###### Abstract

Neural network models have achieved high performance on a wide variety of
complex tasks, but the algorithms that they implement are notoriously
difficult to interpret. In order to understand these algorithms, it is often
necessary to hypothesize intermediate variables involved in the network’s
computation. For example, does a language model depend on particular syntactic
properties when generating a sentence? However, existing analysis tools make
it difficult to test hypotheses of this type. We propose a new analysis
technique – circuit probing – that automatically uncovers low-level circuits
that compute hypothesized intermediate variables. This enables causal analysis
through targeted ablation at the level of model parameters. We apply this
method to models trained on simple arithmetic tasks, demonstrating its
effectiveness at (1) deciphering the algorithms that models have learned, (2)
revealing modular structure within a model, and (3) tracking the development
of circuits over training. We compare circuit probing to other methods across
these three experiments, and find it on par or more effective than existing
analysis methods. Finally, we demonstrate circuit probing on a real-world use
case, uncovering circuits that are responsible for subject-verb agreement and
reflexive anaphora in GPT2-Small and Medium.

##  1 Introduction

Transformer models are the workhorse of modern machine learning, driving
breakthroughs in subfields as disparate as NLP (Devlin et al., 2018; Radford
et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020),
and reinforcement learning (Chen et al., 2021). Despite their success, little
is known about the algorithms that they learn to implement. This central
problem has inspired a flurry of analysis and interpretability research trying
to “open the black box” (Rogers et al., 2021; Elhage et al., 2021; Belinkov,
2022). Despite considerable effort, these models remain almost entirely
opaque.

One challenge that is inherent to interpreting a model that succeeds at a
complex task is that researchers often do not have a complete picture of the
algorithm that they are attempting to uncover. However, they may be able to
propose high-level causal variables that are constituents of such an
algorithm. For example, one may intuit that computing the syntactic number of
the subject noun of a sentence might be useful for language modeling (Chomsky,
1965; Linzen et al., 2016). This variable must be causally implicated in an
algorithm that solves the language modeling task, as it constrains the rest of
the sentence due to agreement rules (i.e. the syntactic number of the main
verb must match the syntactic number of the subject). However, it leaves open
infinite possibilities in which other variables influence the prediction of
the next token. Though we focus on language modeling, this discussion applies
more generally to any complex domain where neural networks are applied, from
vision (Dosovitskiy et al., 2020) to astronomy (Ćiprijanović et al., 2020) to
protein folding (Jumper et al., 2021).

We propose circuit probing to enable the investigation of intermediate
variables in Transformers (Vaswani et al., 2017). Circuit probing introduces a
trainable binary mask over model weights, which is optimized to uncover a
circuit that computes a high-level intermediate variable (if one exists). This
technique enables researchers to (1) test whether high-level intermediate
variables are represented by the model, (2) test whether they are causally
implicated in model behavior (rather than simply decodable from model
representations), and (3) reveal the particular subset of model weights that
compute it.

We first demonstrate this method on a series of simple arithmetic tasks,
showing that it is faithful to the underlying model (i.e. it only provides
evidence in support of causal variables that are actually represented by the
model) and reveals circuits that are causally implicated in model behavior. We
then use circuit probing to analyze two syntactic phenomena on GPT2-Small and
Medium (Radford et al., 2019), uncovering particular model components
responsible for subject-verb agreement and reflexive anaphora agreement111We
release our code at: https://github.com/mlepori1/Circuit_Probing. Circuit
probing is implemented using the NeuroSurgeon package (Lepori et al., 2023a)..

#### Limitations of Existing Causal Methods

We compare circuit probing to two other techniques that seek to test for the
existence of intermediate variables in neural networks: probing and causal
abstraction analysis. Probing involves learning a classifier to decode
information about a hypothesized intermediate variable from model activations
(Tenney et al., 2019; Hewitt & Manning, 2019; Ettinger, 2020; Li et al., 2022;
Nanda et al., 2023). Prior work has demonstrated that probing oftentimes
mischaracterizes the underlying computations performed by the model (Hewitt &
Liang, 2019; Zhang & Bowman, 2018; Voita & Titov, 2020)). Among other methods
(Ravfogel et al., 2020; 2022; Belrose et al., 2023), counterfactual embeddings
were introduced to fix this problem, enabling causal analysis using probes
(Tucker et al., 2021). We empirically demonstrate that circuit probing is more
faithful to a model’s computation than linear or nonlinear probing (See
Sections 3.2, 3.3), and that counterfactual embeddings do not always solve
this problem (See Sections 3.1, 3.2).

Causal abstraction analysis intervenes on the activation vectors produced by
Transformer layers to localize intermediate variables to particular vector
subspaces (Geiger et al., 2021; 2023; Wu et al., 2023). However, causal
abstraction analysis requires hypothesizing a complete causal graph – a high-
level description of how inputs are mapped to predictions, including all
interactions between intermediate variables. This is impossible for most real-
world tasks on which we want to apply neural networks (such as language
modeling). We empirically demonstrate the utility of recent causal abstraction
analysis techniques when full causal graphs are available, and show that
circuit probing arrives at the same results (See Sections 3.1, 3.2).

Figure 1: Schematic visualization of circuit probing for an intermediate
variable representing the syntactic number of the subject of a sentence.
Plural subjects are represented in red, singular subjects in blue. At step T0,
prior to training a binary mask, the model component (Attention block or MLP)
produces residual stream updates (Elhage et al., 2021) that are not
partitioned by syntactic number. Circuit probing optimizes a binary mask over
model weights, resulting in a circuit whose updates are partitioned by
syntactic number at step TN.

##  2 Circuit Probing

The key intuition behind circuit probing is that model components that compute
an intermediate variable should produce outputs that are partitioned according
to that variable. For example, if a model component is computing the syntactic
number of the subject noun, then that component should produce outputs that
fall into one of two equivalence classes, corresponding to singular subjects
and plural subjects.

The first step in circuit probing is to formalize the equivalence classes that
we expect our intermediate variable to induce within a model component. To do
so, we first label a dataset according to the intermediate variable that we
are interested in. For example, singular-subject sentences are mapped to one
label, and plural-subject sentences are mapped to another. Then, for a dataset
of size $N$, we construct an $N$ by $N$ idealized similarity matrix, which
maps pairs of inputs with the same label to 1 and pairs of inputs with
different labels to 0. If the outputs produced by a model component share this
pairwise similarity structure, then we can say that this model component
computes (something isomorphic to) the intermediate variable that we are
testing (Kriegeskorte et al., 2008). Circuit probing is illustrated in Figure
1.

It is unlikely that a model component will be fully dedicated to one
hypothesized intermediate variable. Recent work has shown that neural networks
often exhibit structure at the level of subnetworks (Csordás et al., 2020;
Lepori et al., 2023b; Hod et al., 2021). Thus, we freeze the model’s
parameters and optimize a binary mask over neurons in the model (i.e., over
columns in the matrices of the linear transformations that comprise MLP and
attention blocks). Formally, given a model component $C$, with parameters
$\theta$, we introduce a set of mask parameters $m$, where the values of $m$
are tied across weights in a neuron. Model weights are elementwise multiplied
with mask parameters, which results in the circuit $C_{\theta\odot m}$. During
training, we freeze $\theta$ and only optimize the parameters in $m$. We use
continuous sparsification (Savarese et al., 2020), a model pruning technique
that anneals a soft mask into a discrete mask over training, to learn $m$ (See
Appendix B). Our optimization objective is soft nearest neighbors loss (See
Appendix C), a contrastive loss that minimizes (cosine) distance between
inputs of the same class, and maximizes (cosine) distance between inputs of
different classes (Salakhutdinov & Hinton, 2007; Frosst et al., 2019). We also
train with $l_{0}$ regularization to encourage sparse binary masks. If this
process is successful, it results in a sparse circuit within a model component
that computes the hypothesized intermediate variable.

We note that Transformers consist of a series of attention and MLP blocks that
produce additive updates to the residual stream (Elhage et al., 2021). Circuit
probing looks within these attention and MLP blocks to find circuits that
compute high-level variables. In practice, we will search over all blocks
iteratively.

##  3 Experiments

We present four diverse experiments in order to illustrate the breadth of
questions that circuit probing can help address. Experiments 1, 2, and 3
investigate toy models trained on simple arithmetic tasks, where full causal
graphs are easy to construct. These experiments both (1) nuance or reproduce
results from prior work and (2) contrast circuit probing with existing
analysis methods. Experiment 4 applies circuit probing to language models to
demonstrate that the method generalizes to a more realistic model and setting.

Experiment 1 uses circuit probing to distinguish between two algorithms that a
Transformer might implement to solve an arithmetic task, and demonstrates that
circuit probing agrees with all existing analysis methods in this simple
setting. Experiment 2 investigates the internal structure of Transformer
representations and demonstrates the superiority of circuit probing over
nonlinear probing and causal abstraction analysis for this purpose. Experiment
3 investigates the formation of circuits throughout training, and finds that
circuit probing is more faithful to those circuits than linear probing.
Finally, Experiment 4 analyzes a prominent language model (GPT2) on a task
with an unknown causal graph (language modeling)222See Appendix A for all data
and hyperparameter details..

#### Circuit Probing Evaluation

Across these four experiments, we evaluate the success of circuit probing in
two ways: (1) We train a 1-nearest neighbor classifier on the output vectors
produced by the discovered circuit, and then test this classifier on held-out
data. If circuit probing has succeeded at finding a circuit that computes a
particular intermediate variable, then its output vectors will be partitioned
by the possible labels of this variable, and we expect this classifier to
achieve high performance. We employ a rather conservative strategy here,
randomly sampling only 1 output vector for each label to train the nearest
neighbors classifier. (2) We ablate the discovered circuit (i.e. invert the
learned binary mask) and analyze how the model’s behavior changes. This allows
us to investigate whether the circuit is causal with respect to the underlying
model.

###  3.1 Experiment 1: Deciphering Neural Network Algorithms

#### Goal

One of the central goals of interpretability research is to characterize the
algorithms that models implement Olah (2022). This lofty goal is made
substantially more tractable when we can adjudicate between two hypothesized
alternatives. We demonstrate that circuit probing, linear and nonlinear
probing, and causal abstraction analysis all produce converging results when
characterizing the algorithm implemented by a model trained on a simple
arithmetic task.

Figure 2: Two hypothesized causal graphs that represent solutions to the task
in Experiment 1.

Method | $a^{2}$ | $-1*b^{2}$ | $a+b$ | $a-b$
---|---|---|---|---
Circuit | 99% | 99% | 1% | 1%
Linear | 100% | 100% | 0% | 0%
Nonlinear | 100% | 100% | 1% | 1%

Table 1: Accuracy of all probing methods for each intermediate variable in
Experiment 1. All methods converge to the conclusion that the model is
representing $a^{2}-b^{2}$, rather than $(a+b)*(a-b)$.

#### Task

We train a 1-layer GPT2 model to solve a task defined by the function
${(a^{2}-b^{2})\text{(mod P)}}$, where P is set to 113, and $a$ and $b$ are
input variables. The input sequences are of the form $[a,b,P]$, and the model
is tasked with predicting the answer based on the output embedding of the
final token. All inputs are symbolic - they are one hot vectors mapping to
learnable embeddings. We exhaustively generate all possible data points with
$a$ and $b$ taking values $0-112$. Note that this input-output mapping permits
at least two possible solutions, because $a^{2}-b^{2}=(a+b)\times(a-b)$. This
is visualized by the two causal graphs depicted in Figure 2.

#### Analysis Methods

We use circuit probing to determine which of the two alternative solutions the
trained neural network adopts. Specifically, we search for the intermediate
variables $a^{2}$, $(-1*b^{2})$, $(a+b)$, and $(a-b)$, all mod 113. We
optimize binary masks using the output vectors from the attention and MLP
blocks when they are operating on the $P$ token, which is where the final
prediction is made.

We compare circuit probing to linear and nonlinear probing. We train either a
linear or nonlinear classifier to map from embeddings of the $P$ token to the
value of the intermediate variable that we are attempting to decode. We train
counterfactual embeddings using the trained linear and nonlinear probing
classifiers for this task (Tucker et al., 2021) in order to assess whether the
probes are decoding information that is causally implicated in model behavior.
We also compare to boundless distributed alignment search (boundless DAS; Wu
et al. (2023)), a state-of-the-art causal abstraction analysis technique that
is effective when one can fully specify the expected causal graph (as is the
case here). Causal abstraction analysis measures whether model behavior
changes as a result of interventions on intermediate variables, rather than
whether the variable can be decoded directly from model representations. Thus,
we can answer the same research question with causal abstraction analysis, but
the results are not directly comparable to probing results (i.e., cannot be
easily displayed in the same table).

#### Results

We expect all methods to achieve high performance on either $a^{2}$ and
$-1*b^{2}$ or $a+b$ and $a-b$. This would adjudicate between the two
alternative solutions to the arithmetic task. Indeed, we see that circuit
probing, linear probing, and nonlinear probing all converge to show that the
model computes $a^{2}-b^{2}$, rather than $(a+b)\times(a-b)$ (See Table 1).
With circuit probing, we can ablate the discovered circuits to ensure that
they change model behavior, and thus are causally implicated in the algorithm
that the model is implementing. We confirm that the circuits are causal in
Appendix E. Additionally, all methods converge toward more negative results
from the MLP block, indicating that the intermediate variables are computed by
the attention block (See Appendix D for MLP results).

Causal abstraction analysis also supports these conclusions (See Appendix F).
On the other hand, we find that counterfactual embeddings fail to elicit
counterfactual behavior in all cases, and thus do not provide evidence in
either direction (See Appendix G). Finally, we run a transfer learning
experiment, which behaviorally demonstrates that the model is representing the
task as $a^{2}-b^{2}$, rather than $(a+b)\times(a-b)$ (See Appendix H).
Overall, our results demonstrate that circuit probing agrees with existing
analysis techniques when characterizing the algorithm implemented by a small
model trained on a simple task.

###  3.2 Experiment 2: Modularity of Intermediate Variables

Figure 3: Causal graphs representing the Multitask dataset in Experiment 2.
Note that Task 1 and Task 2 share one intermediate variable (blue) and differ
by one intermediate variable (red and green).

#### Goal

We now apply circuit probing to analyze the internal organization of
Transformer models, which has been the subject of several recent studies
(Lepori et al., 2023b; Csordás et al., 2020; Hod et al., 2021; Mittal et al.,
2022). We show that circuit probing can be used to characterize whether
computations are implemented in a modular and reusable manner within a
Transformer, and that nonlinear probing and counterfactual embeddings do not
reveal such structure.

#### Task

We train a model on a simple multitask modular arithmetic task, represented by
the causal graph in Figure 3. We set $P=29$, $P^{\prime}=31$,
$P^{\prime\prime}=23$. The input sequences are of the form $[T,a,b,c,N]$, and
the model is tasked with predicting the solution. $N$ is a separator token,
and $T$ is a task token. For Task 1, we exhaustively generate all possible
data points with $a$ and $b$ taking values $0-112$, and $c$ being a random
token in the same range. Similarly for Task 2. Note that both tasks share one
intermediate variable (a “shared variable”), and each task has one
intermediate variable that the other does not (a “free variable”).

Figure 4: All probing results for Experiment 2. We find that circuit probing and linear probing generally converge to find evidence of the Free and Shared variable for both tasks, and no evidence of the irrelevant variable (“other”). On the other hand, nonlinear probing suggests that the irrelevant variable is represented. Train Task | Variable | Task 1 Ablation Accuracy | Task 2 Ablation Accuracy
---|---|---|---
Task 1 | Shared | 3.7% | 3.7%
Task 1 | Free | 3.2% | 28.1%
Task 2 | Shared | 3.6% | 3.6%
Task 2 | Free | 48% | 4.8%
Table 2: Experiment 2 circuit probing ablation results. Targeted ablations of
the circuits returned by circuit probing reveal a stark difference between the
internal representations of the Free and Shared variables. A targeted ablation
of the shared variable circuit destroys performance on both tasks, whereas a
targeted ablation of Task 1’s free variable harms performance on Task 1 far
more (underlined) than Task 2 (bolded). The same is true in the opposite
direction.

#### Analysis Methods

First, we probe for intermediate variables. We use circuit probing to probe
each task individually, assessing which intermediate variables are computed
when solving Task 1, and which are computed when solving Task 2. Specifically,
we search for the intermediate variables $a\text{(mod P)}$, $b\text{(mod
P')}$, and $c\text{(mod P'')}$. We optimize binary masks using the output
vectors from the attention and MLP blocks when they are operating on the $N$
token, which is where the final prediction is made. We compare circuit probing
to linear and nonlinear probing. We train either a linear or nonlinear
classifier to map from embeddings of the $N$ token to the value of the
intermediate variable that we are attempting to decode. We also run causal
abstraction analysis using boundless DAS.

Next, we investigate modularity directly. We hypothesize that the model
implements a reusable computation for the shared variable $a\text{(mod P)}$ –
that the same computation is used in both Task 1 and Task 2. Similarly, we
hypothesize that the free variables are implemented modularly, – that Task 1’s
free variable computation can be ablated without completely destroying
performance on Task 2, and vice-versa. We use targeted ablations of the
circuits returned by subnetwork probing to provide evidence in support of this
hypothesis. We compare against a similar analysis using counterfactual
embeddings. Note that one cannot address this question using causal
abstraction analysis – causal abstractions measures specific changes in model
behavior, rather than investigating model representations directly.

#### Results

We first present our probing results. We expect models to compute their free
and shared variables ($a\text{(mod P)}$ and $b\text{(mod P')}$ for Task 1, and
$a\text{(mod P)}$ and $c\text{(mod P'')}$ for Task 2), and not to compute the
other variable. Our results for the attention block are shown in Figure 4. All
three methods decode free and shared variables with high accuracy, indicating
that the relevant variables for a given task are computed in the attention
block. However, nonlinear probing (and only nonlinear probing) decodes the
other variable with high accuracy. Causal Abstraction analysis agrees with
circuit probing and linear probing (see Appendix F). This accords with prior
work questioning whether expressive probes accurately reflect the causal
structure of neural networks (Voita & Titov, 2020; Zhang & Bowman, 2018).

Next, we present modularity results. We expect that ablating the circuit
computing the shared variable for either task should destroy performance
across both tasks. On the other hand, we expect that ablating the circuit
computing the free variable for Task 1 should destroy performance on Task 1
while having less of an effect on Task 2 performance (and vice-versa). In
Appendix I, we analyze the morphology of the two free variable circuits
returned by circuit probing and find that the circuits are only distinct in
one tensor within the attention block. Thus, we only ablate the circuits
within that tensor. From Table 2, we see that ablating the circuit that
computes the shared variable for either Task 1 or Task 2 destroys performance
on both tasks. On the other hand, ablating the free variable in Task 1
destroys performance on Task 1, while maintaining some performance on Task 2.
Similarly for the free variable in Task 2.

We investigate the internal structure of this model using counterfactual
embeddings in Appendix J, and find that counterfactual embeddings fail to
provide evidence of modularity. We conclude that counterfactual embeddings act
more as adversarial examples for our trained nonlinear probe, rather than as
meaningful counterfactual inputs to the model. Overall, our results
demonstrate that circuit probing is superior to competing techniques at
characterizing how models structure their computations.

###  3.3 Experiment 3: Circuit Probing as a Progress Measure

Figure 5: (Left) Experiment 3 training curve on train and test sets. We see
generalization long after overfitting. (Right) Linear and circuit probing for
$a^{2}$ throughout training. Linear probing converges to perfect accuracy very
early in training, while circuit probing reveals that the circuit for $a^{2}$
is formed gradually through training, but achieves high performance before the
overall model generalizes perfectly.

#### Goal

Circuit probing allows us to gain insight into the training dynamics of
Transformers at the level of intermediate variables. Recent work has shown
that models may abruptly learn to generalize long after they overfit to the
training data (Power et al., 2022). Despite this rather discontinuous switch
from overfitting to generalization (often called grokking), Nanda et al.
(2022) revealed that the circuit that computes the generalizable algorithm is
formed continuously throughout training. While their work required reverse-
engineering the entire algorithm to gain this insight into circuit formation,
we reproduce their finding in a slightly different setting using circuit
probing (which only requires us to hypothesize a high-level intermediate
variable). In this setting, we demonstrate that circuit probing is more
faithful to the circuits implemented by the underlying model than linear
probing.

#### Task

We train a model on the task $(a^{2}+b)\text{(mod P)}$, with $P=113$. The
input sequences are of the form $[a,b,P]$, and the model is tasked with
predicting the solution. Our model exhibits grokking on this task - it
generalizes long after it overfits. Generalization performance increases
rather slowly from epoch 0 until 10000, then rapidly climbs to near-perfect
accuracy by epoch 17500. See Figure 5 (Left).

#### Analysis Methods

First, we probe for the development of the intermediate variable $a^{2}$
throughout training using circuit probing and linear probing. Next, we perform
a selectivity analysis on the trained model – we probe for a variable that is
not causally implicated in model behavior ($b^{2}$), and verify that circuit
probing does not decode this intermediate variable.

#### Results

First, we investigate the development of the circuit computing $a^{2}$. We
expect the performance of circuit probing to increase steadily throughout
training, converging to a high value before the overall model generalizes.
This finding would align with Nanda et al. (2022)’s finding that “circuit
formation” occurs continuously, and that it completes before the overall model
generalizes. We provide results from circuit probing on the attention block
(See Appendix K for MLP results). From Figure 5 (Right), we see that circuit
probing accuracy increases throughout training, achieving $>$90% accuracy
before epoch 10000. Thus, we can conclude that the circuit that allows the
model to generalize is formed before overall model’s generalization behavior
would imply. However, this circuit is not available from the outset and is
developed throughout training. Linear probing tells a markedly different
story, implying that the variable required to generalize was present nearly
from the beginning of training. Next, we present results from our selectivity
analysis. We expect our probing methods to achieve poor accuracy when probing
for $b^{2}$, a variable that is not causally implicated in model behavior.
Circuit probing achieves 54.9% accuracy333 Random chance for circuit probing
is 50%, as only two distinct integers in our dataset map to the same value of
$b^{2}\text{(mod 113)}$, and we are using a 1-nearest neighbor classifier. at
decoding $b^{2}$ from the fully trained model, whereas linear probing achieves
100% accuracy. See Appendix L for results throughout training. Overall these
results demonstrate that circuit probing is more faithful to the underlying
circuitry than linear probing.

###  3.4 Experiment 4: Circuit Probing for Language Models

#### Goal

The previous experiments focused on toy tasks in which a full causal graph
could be specified. However, the reason that we are interested in developing
interpretability tools is to analyze models that are used in practice, on
tasks where a causal graph cannot be constructed. Here, we use circuit probing
to investigate how pretrained GPT2-Small and GPT2-Medium perform language
modeling. In particular, we investigate two linguistic dependencies that rely
on syntactic number: subject-verb agreement and reflexive anaphora. Subject-
verb agreement refers to the English-language phenomenon where the subject of
a sentence must agree with the main verb of a sentence in syntactic number.
For example: The keys are on the table is grammatical, whereas The keys is on
the table is ungrammatical. We hypothesize that an intermediate variable
representing the syntactic number of the subject noun is computed when
predicting the main verb of a sentence. Reflexive Anaphora ensures that
reflexive pronouns agree with their referents. For example: the consultants
injured themselves is grammatical, and the consultants injured herself is
ungrammatical. We hypothesize that an intermediate variable representing the
syntactic number of the referent is computed when predicting a reflexive
pronoun.

#### Task

For both phenomena, we use the templates from Marvin & Linzen (2018) to
generate sentence prefixes, where the continuation of the prefixes are likely
to be either a main verb (when studying subject-verb agreement) or a reflexive
pronoun (when studying reflexive anaphora). See Appendix M for example
prefixes. We run circuit probing on sentences containing one distractor, which
is a non-subject or non-referent noun (e.g. “cabinet” in the keys to the
cabinet are on the table).

#### Analysis Methods

For each phenomenon, we run circuit probing on the last token of sentence
prefixes to uncover the circuit that computes the syntactic number of the
subject noun or referent. We then ablate the discovered circuit, and evaluate
the model’s ability to grammatically continue held-out sentence prefixes.
Specifically, we assess whether the model is more likely predict tokens that
are consistent or inconsistent with the syntactic number of the
subject/referent. For subject-verb agreement, we inspect the logits for the
tokens is and are – if the logit for is is higher than the logit for are when
the subject is singular (e.g. The officer…), then we consider the model to
have succeeded on that sentence prefix. For reflexive anaphora, we run the
analysis twice, once comparing the logits of herself and themselves, and again
comparing the logits of himself and themselves.

We evaluate models on IID sentences with one distractor noun and on an OOD
dataset of sentences that contain two distractor nouns (See Appendix M for two
distractor examples). We hypothesize that the same circuit computes the
relevant linguistic dependency for both sentence structures. Even if we
recover positive results from this analysis, it could be the case that we are
simply removing too many parameters from the model – rather than ablating a
specialized circuit, we could be destroying the entire model. As a control, we
sample 5 random subnetworks from the complement set of neurons that are in our
circuit and rerun this analysis. We ensure that the random subnetworks contain
the same number of parameters as our circuit in each tensor.

#### Results

For both syntactic dependencies, we expect that ablating the discovered
circuit will render the model worse at distinguishing the syntactic number of
the subject/referent. We expect that ablating random subnetworks should not
harm model performance on any dataset. We run circuit probing over all model
components for both GPT2-Small and Medium. We present results from GPT2-Small
in the main body, and present GPT2-Medium results in Appendix P444In brief,
our results on GPT2-Medium are largely reproduce what we find in GPT2-Small
for reflexive anaphora. We fail to identify a circuit that computes subject-
verb agreement dependencies.. For both phenomena, we find that the dependency
is computed in layer 6’s attention block555For reflexive anaphora, we note
that the underlying model achieves higher accuracy when predicting the
masculine pronoun. This is clear evidence of gender bias in language models,
which has been well-documented elsewhere (Marvin & Linzen, 2018; May et al.,
2019; Rudinger et al., 2018; Weidinger et al., 2021).. Ablating the circuit
returned by circuit probing drops performance substantially for IID and OOD
datasets for both phenomena, while ablating random subnetworks does not impact
model performance (See Figure 6). Other blocks do not exhibit this
characteristic pattern as strongly (See Appendix O). This accords with prior
work suggesting that syntactic dependencies are represented in middle layers
of Transformers (Tenney et al., 2019; Vig & Belinkov, 2019). See Appendix N
for an investigation of KNN performance, Appendix Q for an analysis of circuit
overlap, and Appendix R for qualitative results. Additionally, see Appendix S
for results demonstrating that syntactic number of individual tokens is
computed much earlier in the model.

Figure 6: GPT2-Small ablation results on layer 6’s attention block. Across
both Subject-Verb Agreement (Left) and Reflexive Anaphora evaluated using the
masculine (Middle) and feminine (Right) pronoun, we see that ablating the
discovered circuit renders the model worse at distinguishing the syntactic
number of the subject/referent. Ablating randomly sampled subnetworks has does
not hurt the model’s ability to distinguish singular and plural
subjects/referents.

##  4 Discussion

#### Related Work

Circuit probing is related to recent efforts in mechanistic interpretability –
a burgeoning field that attempts to reverse-engineer neural network
algorithms. Through substantial manual effort, researchers have uncovered the
algorithms that both toy models (Olsson et al., 2022; Nanda et al., 2022;
Chughtai et al., 2023) and more realistic models (Wang et al., 2022; Hanna et
al., 2023; Merullo et al., 2023) are implementing. More broadly, there has
been substantial work analyzing the syntactic (Linzen & Baroni, 2021;
Goldberg, 2019; Tenney et al., 2018; McCoy et al., 2018) and semantic
capabilities Pavlick (2022; 2023); Yu & Ettinger (2020); Hupkes et al. (2020);
Dziri et al. (2023) of language models. Circuit probing is most directly
related to work that attempts to decompose neural networks into functional
subnetworks Csordás et al. (2020); Hamblin et al. (2022); Lepori et al.
(2023b); Zhang et al. (2021); Panigrahi et al. (2023); Hod et al. (2021); Cao
et al. (2021). The success of circuit probing across the diverse set of
experiments presented here is further evidence that subnetworks are a useful
lens through which to analyze models.

#### Limitations

Circuit probing is weaker than both counterfactual embeddings and causal
abstraction analysis in one key respect: it does not allow for counterfactual
interventions. It is currently unknown how multiple circuits compose within a
given block to create one additive update to the residual stream, so one
cannot replace individual variables to elicit counterfactual behavior. Future
work might seek to understand how circuits compose with one another.

#### Conclusion

Through four experiments, we have demonstrated the effectiveness of circuit
probing for investigating the internal representations of models. With circuit
probing, one can gain insights into the underlying algorithms the model is
implementing, how these algorithms are structured within the model, and how
they develop throughout training. Circuit probing is more faithful to the
underlying model than linear or nonlinear probing, and does not require
specifying a full causal graph before testing hypotheses.

##  5 Ethics Statement

Circuit probing can be used to uncover computations that a neural network is
performing. This may have future implications for bias, fairness, and safety
of neural network models. However, we emphasize that the current iteration of
circuit probing should not be used in isolation to assess models for social
biases in real-world systems. Circuit probing can provide positive evidence
that a computation is implemented, but cannot yet be used to provide evidence
that a computation is definitely not implemented in a real-world system.

##  6 Reproducibility Statement

To foster reproducibility, we provide details on model training and
hyperparameters in Appendices A and B. We provide details of our experimental
design for all experiments throughout the main text, as well as in our
appendices. Additionally, we provide a high-level explanation of the circuit
probing algorithm in Section 2, and provide details on the loss function in
Appendix C. Finally, we make our code publicly available.

## References

  * Belinkov (2022) Yonatan Belinkov.  Probing classifiers: Promises, shortcomings, and advances.  _Computational Linguistics_ , 48(1):207–219, 2022.
  * Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman.  Leace: Perfect linear concept erasure in closed form.  _arXiv preprint arXiv:2306.03819_ , 2023.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cao et al. (2021) Steven Cao, Victor Sanh, and Alexander M Rush.  Low-complexity probing via finding subnetworks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 960–966, 2021.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Chomsky (1965) Noam Chomsky.  _Aspects of the Theory of Syntax_.  Number 11. MIT press, 1965.
  * Chughtai et al. (2023) Bilal Chughtai, Lawrence Chan, and Neel Nanda.  A toy model of universality: Reverse engineering how networks learn group operations.  _ArXiv_ , abs/2302.03025, 2023.  URL https://api.semanticscholar.org/CorpusID:256615287.
  * Ćiprijanović et al. (2020) Aleksandra Ćiprijanović, Gregory F Snyder, Brian Nord, and Joshua EG Peek.  Deepmerge: Classifying high-redshift merging galaxies with deep neural networks.  _Astronomy and Computing_ , 32:100390, 2020.
  * Csordás et al. (2020) Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber.  Are neural nets modular? inspecting functional modularity through differentiable weight masks.  In _International Conference on Learning Representations_ , 2020.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al.  Faith and fate: Limits of transformers on compositionality.  _arXiv preprint arXiv:2305.18654_ , 2023.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 1, 2021.
  * Ettinger (2020) Allyson Ettinger.  What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48, 2020.
  * Frosst et al. (2019) Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton.  Analyzing and improving representations with the soft nearest neighbor loss.  In _International conference on machine learning_ , pp. 2012–2020. PMLR, 2019.
  * Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts.  Causal abstractions of neural networks.  _Advances in Neural Information Processing Systems_ , 34:9574–9586, 2021.
  * Geiger et al. (2023) Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D Goodman.  Finding alignments between interpretable causal variables and distributed neural representations.  _arXiv preprint arXiv:2303.02536_ , 2023.
  * Goldberg (2019) Yoav Goldberg.  Assessing bert’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_ , 2019.
  * Hamblin et al. (2022) Chris Hamblin, Talia Konkle, and George Alvarez.  Pruning for interpretable, feature-preserving circuits in cnns.  _arXiv preprint arXiv:2206.01627_ , 2022.
  * Hanna et al. (2023) Michael Hanna, Ollie Liu, and Alexandre Variengien.  How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023.
  * Hewitt & Liang (2019) John Hewitt and Percy Liang.  Designing and interpreting probes with control tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2733–2743, 2019.
  * Hewitt & Manning (2019) John Hewitt and Christopher D Manning.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4129–4138, 2019.
  * Hod et al. (2021) Shlomi Hod, Daniel Filan, Stephen Casper, Andrew Critch, and Stuart Russell.  Quantifying local specialization in deep neural networks.  _arXiv preprint arXiv:2110.08058_ , 2021.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni.  Compositionality decomposed: How do neural networks generalise?  _Journal of Artificial Intelligence Research_ , 67:757–795, 2020.
  * Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kriegeskorte et al. (2008) Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini.  Representational similarity analysis-connecting the branches of systems neuroscience.  _Frontiers in systems neuroscience_ , pp. 4, 2008.
  * Lepori et al. (2023a) Michael A Lepori, Ellie Pavlick, and Thomas Serre.  Neurosurgeon: A toolkit for subnetwork analysis.  _arXiv preprint arXiv:2309.00244_ , 2023a.
  * Lepori et al. (2023b) Michael A Lepori, Thomas Serre, and Ellie Pavlick.  Break it down: evidence for structural compositionality in neural networks.  _arXiv preprint arXiv:2301.10884_ , 2023b.
  * Li et al. (2022) Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Emergent world representations: Exploring a sequence model trained on a synthetic task.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Linzen & Baroni (2021) Tal Linzen and Marco Baroni.  Syntactic structure from deep learning.  _Annual Review of Linguistics_ , 7:195–212, 2021.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.  Assessing the ability of lstms to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535, 2016.
  * Marvin & Linzen (2018) Rebecca Marvin and Tal Linzen.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 1192–1202, 2018.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 622–628, 2019.
  * McCoy et al. (2018) R Thomas McCoy, Robert Frank, and Tal Linzen.  Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks.  In _40th Annual Meeting of the Cognitive Science Society: Changing Minds, CogSci 2018_ , pp. 2096–2101. The Cognitive Science Society, 2018.
  * Merullo et al. (2023) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick.  Language models implement simple word2vec-style vector arithmetic, 2023.
  * Mittal et al. (2022) Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie.  Is a modular architecture enough?  _Advances in Neural Information Processing Systems_ , 35:28747–28760, 2022.
  * Nanda et al. (2022) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt.  Progress measures for grokking via mechanistic interpretability.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Nanda et al. (2023) Neel Nanda, Andrew Lee, and Martin Wattenberg.  Emergent linear representations in world models of self-supervised sequence models.  _arXiv preprint arXiv:2309.00941_ , 2023.
  * Olah (2022) Christopher Olah.  Mechanistic interpretability, variables, and the importance of interpretable bases.  _Transformer Circuits Thread_ , 2022.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, T. J. Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom B. Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Christopher Olah.  In-context learning and induction heads.  _ArXiv_ , abs/2209.11895, 2022.  URL https://api.semanticscholar.org/CorpusID:252532078.
  * Panigrahi et al. (2023) Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora.  Task-specific skill localization in fine-tuned language models.  _arXiv preprint arXiv:2302.06600_ , 2023.
  * Pavlick (2022) Ellie Pavlick.  Semantic structure in deep learning.  _Annual Review of Linguistics_ , 8:447–471, 2022.
  * Pavlick (2023) Ellie Pavlick.  Symbols and grounding in large language models.  _Philosophical Transactions of the Royal Society A_ , 381(2251):20220041, 2023.
  * Power et al. (2022) Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra.  Grokking: Generalization beyond overfitting on small algorithmic datasets.  _arXiv preprint arXiv:2201.02177_ , 2022.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7237–7256, 2020.
  * Ravfogel et al. (2022) Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell.  Linear adversarial concept erasure.  In _International Conference on Machine Learning_ , pp. 18400–18421. PMLR, 2022.
  * Rogers et al. (2021) Anna Rogers, Olga Kovaleva, and Anna Rumshisky.  A primer in bertology: What we know about how bert works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866, 2021.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 8–14, 2018.
  * Salakhutdinov & Hinton (2007) Ruslan Salakhutdinov and Geoff Hinton.  Learning a nonlinear embedding by preserving class neighbourhood structure.  In _Artificial intelligence and statistics_ , pp. 412–419. PMLR, 2007.
  * Savarese et al. (2020) Pedro Savarese, Hugo Silva, and Michael Maire.  Winning the lottery with continuous sparsification.  _Advances in neural information processing systems_ , 33:11380–11390, 2020.
  * Tenney et al. (2018) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_ , 2018.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick.  Bert rediscovers the classical nlp pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4593–4601, 2019.
  * Tucker et al. (2021) Mycal Tucker, Peng Qian, and Roger Levy.  What if this modified that? syntactic interventions with counterfactual embeddings.  Association for Computational Linguistics, 2021.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vig & Belinkov (2019) Jesse Vig and Yonatan Belinkov.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 63–76, 2019.
  * Voita & Titov (2020) E Voita and I Titov.  Information-theoretic probing with minimum description length.  In _EMNLP 2020-2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference_ , pp. 183–196, 2020.
  * Wang et al. (2022) Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  _ArXiv_ , abs/2211.00593, 2022.  URL https://api.semanticscholar.org/CorpusID:253244237.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_ , 2021.
  * Wu et al. (2023) Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman.  Interpretability at scale: Identifying causal mechanisms in alpaca.  _arXiv preprint arXiv:2305.08809_ , 2023.
  * Yu & Ettinger (2020) Lang Yu and Allyson Ettinger.  Assessing phrasal representation and composition in transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4896–4907, 2020.
  * Zhang et al. (2021) Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville.  Can subnetwork structure be the key to out-of-distribution generalization?  In _International Conference on Machine Learning_ , pp. 12356–12367. PMLR, 2021.
  * Zhang & Bowman (2018) Kelly Zhang and Samuel Bowman.  Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 359–361, 2018.

##  Appendix A Data and Hyperparameter Details

All 1-layer GPT2 models have 4 attention heads, embedding size of 128, and MLP
dimension of 512. We use the Adam optimizer (Kingma & Ba, 2014) with a
learning rate of 0.001, and train with weight decay.

#### Experiment 1

We train a 1-layer GPT2 model on 60% of all possible datapoints for this task,
leaving the other 40% held out as a test set. For circuit probing, we train
our mask on all examples from the train set (See Appendix B for those training
hyperparameters) using a batch size of 500, and use updates generated from
train set examples to train the 1-nearest neighbors classifier. We evaluate on
the held-out test set.

For linear and nonlinear probing, we train for 100 epochs, using a learning
rate of 0.1. We use this same learning rate for generating counterfactual
embeddings. Our nonlinear classifier uses ReLU nonlinearity, and has a hidden
size of 256.

For Boundless DAS, we train for 250 epochs, with a 2500 training examples
using the Adam optimizer with a learning rate of 0.01.

For transfer experiments, we finetune on 60% of each dataset, using the Adam
optimizer with a learning rate of 0.001. We train with weight decay. We train
for fewer epochs with $a^{2}$ because the models converge very early in
training.

#### Experiment 2

All details are the same as in Experiment 1.

#### Experiment 3

We train a 1-layer GPT2 model on 33.3% of all possible datapoints for this
task, which gives us the grokking behavior that we wish to investigate. All
other details are the same as Experiment 1.

#### Experiment 4

We train circuit probing on 2000 examples for each dataset and test on 1000
examples, otherwise all details are the same.

##  Appendix B Continuous Sparsification Details

Continuous sparsification enables us to train binary masks over model weights.
Our loss function is defined as:

$\min_{m_{i}\in\\{0,1\\}}L_{soft\\_neighbors}(C_{\theta\odot
m_{i}})+\lambda||m||$ (1)

Where $m$ is our binary mask, $C_{\theta}$ is a model component, with weights
$\theta$, and $||m||$ is our $l_{0}$ regularizer, and $L_{soft\\_neighbors}$
is the soft nearest neighbors loss described in Appendix C

Typically, optimizing such a binary mask is intractable, given the
combinatorial nature of a discrete binary mask over a large parameter space.
Instead, continuous sparsification reparameterizes the loss function by
introducing another variable, $s\in\mathbb{R}^{d}$:

$\min_{s_{i}\in\mathbb{R}^{d}}L_{soft\\_neighbors}(C_{\theta\odot\sigma(\beta\cdot
s_{i})}+\lambda||\sigma(\beta\cdot s_{i})||_{1}$ (2)

In Equation 2, $\sigma$ is the sigmoid function, applied elementwise, and
$\beta$ is a temperature parameter. During training $\beta$ is increased after
each epoch according to an exponential schedule to a large value
$\beta_{max}$. Note that, as $\beta\xrightarrow{}\infty$, $\sigma(\beta\cdot
s_{i})\xrightarrow{}H(s_{i})$, where $H(s_{i})$ is the heaviside function.

$H(s)=\left\\{\begin{array}[]{lr}0,s<0\\\ 1,s>0\end{array}\right\\}$ (3)

Thus, during training, we interpolate between a soft mask ($\sigma$) and a
discrete mask ($H$). During inference, we simply substitute
$\sigma(\beta_{max}\cdot s_{i}))$ for $H(s_{i})$. Notably, we apply continuous
sparsification to a frozen model in an attempt to reveal the internal
structure of this model, whereas the original work introduced continuous
sparsification in the context of model pruning, and jointly trained $\theta$
and $s$.

For all experiments, we train binary masks with the Adam optimizer (Kingma &
Ba, 2014), with a learning rate of $0.001$. We fix $\beta_{max}=200$,
initialize the mask parameters to 0, and train for 90 epochs. The $\lambda$
parameter must scale with the number of parameters per layer. For all 1-layer
GPT2 experiments, we set $\lambda=1E-6$. For GPT2-Small and Medium, we set
$\lambda=1E-7$ and $1E-8$, respectively.

##  Appendix C Soft Nearest Neighbors Loss

Given input embeddings $x$ to a model component $C$ and intermediate variable
labels $y$, in a batch with $b$ samples, Equation 4 defines the full
optimization objective. $\lambda$ is a hyperparameter that scales the $l_{0}$
regularization strength. Intuitively, this loss function pushes members of the
same class towards each other, according to some distance metric, and members
of different classes far from each other. Concretely, this partitions the
output space of transformer layers into equivalence classes defined by the
variable that we are searching for.

$min_{m_{n}\in\\{0,1\\}}-\frac{1}{b}\sum_{i\in
1..b}\frac{\sum_{\begin{subarray}{c}j\in 1...b,\\\ j\neq i,\\\
y_{i}=y_{j}\end{subarray}}e^{cosine\\_dist(C_{\theta\odot
m}(x_{i}),C_{\theta\odot m}(x_{j}))}}{\sum_{\begin{subarray}{c}k\in 1...b,\\\
k\neq i\end{subarray}}e^{cosine\\_dist(C_{\theta\odot m}(x_{i}),C_{\theta\odot
m}(x_{k}))}}+\lambda\sum_{n\in 1...|m|}m_{n}$ (4)

##  Appendix D Experiment 1: MLP Results

In Figure 7, we present results from Experiment 1 for all probing methods on
the MLP block. First, we note that all methods perform worse at decoding
$a^{2}$ and $-1*b^{2}$. We note that chance accuracy for circuit probing is
effectively 50%, whereas chance for probing methods is 0.8% (1 out of 113).
Circuit probing results are generated by a 1-nearest neighbors classifier
trained on the outputs of the MLP block after masking. For the variable
$a^{2}$ and $-1*b^{2}$, there are only two distinct integers that map to the
same value of that variable (i.e. $4^{2}\text{(mod 113)}=111^{2}\text{(mod
113) = 4})$. Because we are training the classifier with 1 vector per variable
label, 50% of underlying integers are represented in the 1-NN training set.
Thus, circuit probing accuracy of 50% means that the block is merely decoding
the identity of the underlying token, rather than meaningfully computing an
intermediate variable.

Figure 7: MLP probe accuracy for Experiment 1. All methods decode $a^{2}$ and
$-1*b^{2}$ worse in the MLP than in the attention block. Note that chance
accuracy for circuit probing is effectively 50%.

##  Appendix E Experiment 1: Circuit Ablation Results

Table 3 contains the results from running an ablation analysis on the circuits
discovered in Experiment 1. We note two things: (1) ablating the circuits for
$a^{2}$ and $-1*b^{2}$ destroy model performance, and (2) circuit probing
returns empty subnetworks for $a+b$ and $a-b$. This is a useful feature of
using $l_{0}$ regularization when training binary masks – if there is no
signal for a given variable, circuit probing is encouraged to return a
maximally sparse (i.e. empty) subnetwork.

Variable | Full Test Acc. | Ablated Test Acc. | % Parameters in Circuit
---|---|---|---
$a^{2}$ | 100% | 0.8% | 53.3%
$-b^{2}$ | 100% | 0.9% | 53.5%
$a+b$ | 100% | 100% | 0%
$a-b$ | 100% | 100% | 0%
Table 3: Experiment 1 task performance after ablating the circuit returned by
circuit probing. We see that ablating the circuit responsible for either
$a^{2}$ or $-b^{2}$ destroys test accuracy. However, we see that circuit
probing returns an empty circuit for both $a+b$ and $a-b$, due to $l_{0}$
regularization. Thus, ablating this empty circuit has no effect.

##  Appendix F Causal Abstraction Analysis Results

In Table 4, we provide results from running causal abstraction analysis using
Boundless DAS (Wu et al., 2023) on both Experiment 1 and Experiment 2. Causal
abstraction analysis creates interventions on model representations in order
to elicit counterfactual behavior in the downstream model. For example, for
the case of $a^{2}-b^{2}$, the model might intervene to change the value of
$a^{2}$ to $a^{\prime 2}$. The intervention is considered successful of the
overall model outputs the answer to $a^{\prime 2}-b^{2}$. Causal abstraction
analysis reports statistics in terms of the success of its counterfactual
embeddings, rather than its ability to decode model representations directly.
Thus, these results answer the same questions as, but are not directly
comparable to, circuit probing results.

Nevertheless, these results support the results generated by circuit probing
in both Experiment 1 and Experiment 2. In Experiment 1, causal abstraction
analysis reveals evidence for $a^{2}$ and $-1*b^{2}$, but not $a+b$ and $a-b$.
In Experiment 2, causal abstraction analysis reveals evidence that the model
is using variables $a\text{(mod P)}$ and $b\text{(mod P')}$ when solving Task
1, and $a\text{(mod P)}$ and $c\text{(mod P'')}$ when solving Task 2.

Component | $a^{2}$ | $-b^{2}$ | $a+b$ | $a-b$
---|---|---|---|---
Attn. | 98% | 99% | 1% | 1%
MLP | 2% | 2% | 2% | 2%
Component | Task | $a\text{ (mod P)}$ | $b\text{ (mod P')}$ | c (mod P”)
Attn. | 1 | 93% | 93% | -
MLP | 1 | 3% | 3% | -
Attn. | 2 | 93% | - | 94%
MLP | 2 | 4% | - | 4%
Table 4: Causal abstraction analysis results for Experiment 1 (top) and
Experiment 2 (bottom). Using boundless distributed alignment search, we reveal
that the attention blocks in both models contain the same causal intermediate
variables that circuit probing discovers.

##  Appendix G Experiment 1: Counterfactual Embeddings

We present counterfactual embedding results from Experiment 1 in Table 5.
Counterfactual embeddings are embeddings that are optimized to fool a probing
classifier. Formally, given a probe, $P_{\theta}$ trained to decode an
intermediate variable, $V$, consider a residual stream state $e$ such that
$P_{\theta}(e)=V_{i}$. We freeze $P_{\theta}$ and optimize $e$ such that
$P_{\theta}(e^{\prime})=V_{j}$. If $P_{\theta}$ is decoding information that
is causally implicated in the underlying model, then replacing $e$ with
$e^{\prime}$ should change the output to the output one would expect from
setting variable $V$ to $V_{j}$. We report counterfactual embedding success –
the percent of embeddings that are successfully optimized to fool the probing
classifier. We see that all linear probing classifiers can be fooled by
counterfactual embeddings. We see that nonlinear classifiers can be fooled by
counterfactual embeddings only for $a^{2}$ and $-1*b^{2}$. Recall that all
classifiers performed poorly at decoding $a+b$ and $a-b$.

Next, we analyze counterfactual behavior success – the percent of
counterfactual embeddings that actually elicit counterfactual behavior in the
overall model. We see that all sets of counterfactual embeddings fail to
elicit counterfactual behavior. Taken in isolation, one might conclude that
these probes are not decoding causally-relevant information, and thus that
models are not actually computing $a^{2}$ and $-1*b^{2}$. However, given the
success of every other analysis technique at causally implicating $a^{2}$ and
$-1*b^{2}$, we may instead conclude that counterfactual embeddings are acting
as adversarial examples to the probing classifier, and are destroying the
embedding with respect to the underlying model.

Counterfactual Embedding Success
---
Probe | $a^{2}$ | $-b^{2}$ | $a+b$ | $a-b$
Linear | 100% | 100% | 100% | 100%
Nonlinear | 100% | 100% | 1% | 1%
Counterfactual Behavior Success
Probe | $a^{2}$ | $-b^{2}$ | $a+b$ | $a-b$
Linear | 1% | 1% | 1% | 1%
Nonlinear | 1% | 1% | 0% | 0%
Table 5: Experiment 1 counterfactual embedding results. (Top) Counterfactual
embedding success – the percent of examples where the counterfactual
optimization procedure creates an example that changes probe outputs to a
particular class. We see that this optimization process largely succeeds,
except for $a+b$ and $a-b$ in nonlinear probes. (Bottom) Counterfactual
behavior success – the percent of counterfactual embeddings that elicit
counterfactual behavior in the model. We see very poor performance on this
metric, indicating that counterfactual embeddings are not producing the
expected behavioral outcomes.

##  Appendix H Experiment 1: Transfer Learning

To further confirm our findings in Experiment 1, we analyze whether training
on $a^{2}-b^{2}$ confers any benefits when finetuning on different tasks. In
particular, we finetune the GPT2 model on a task defined by $a^{2}\text{(mod
113)}$, and separately on a task defined by $a+b\text{(mod 113)}$. If the
model is solving the task using $a^{2}-b^{2}$, we expect that finetuning
should help the model solve $a^{2}$ faster than training a randomly
initialized model, because the model already represents the variable necessary
to solve the finetuning task. Similarly, we expect the finetuning to $a+b$
will be slower than training a randomly initialized model, because the model
represents variables that are explicitly not useful for solving the finetuning
task. From Figure 8, that is exactly what we see.

Figure 8: (Left) Transfer performance for $a^{2}$. We see that pretraining on
$a^{2}-b^{2}$ confers a benefit to the model when finetuning on $a^{2}$.
(Right) Transfer performance for $a+b$. We see that pretraining on
$a^{2}-b^{2}$ is a detriment when finetuning on $a+b$.

##  Appendix I Experiment 2: Circuit Overlap Analysis

From Figure 9, we see that the two circuits computing the free variables in
Task 1 and Task 2 are largely distinct in `attn.c_attn`, but nearly completely
overlapping in `attn.c_proj`. With this insight, we ablate circuit parameters
just within `attn.c_attn`. We can also visualize the distribution of circuits
through attention heads. In Figure 10, we see that circuit probing recovers
structures that exist between particular attention heads (i.e. no single
attention head is fully devoted to an intermediate variable) , but also
partially localizes the two Free variables into specific heads (head 1 for
Task 1, head 2 for Task 2).

Figure 9: Experiment 2: Visualizing the distribution of circuits throughout
the tensors comprising an attention block. We see that the circuits computing
the free variables for Task 1 and Task 2 almost completely overlap in the
c_proj tensor, but are mostly distinct in the c_attn tensor. Figure 10:
Circuit probing recovers some elements of known structure within Transforemrs.
In particular, we see that Head 1 largely computes the Free variable in Task
1, and Head 2 largely computes the Free variable in Task 2. However, we also
note that circuits extend beyond individual attention heads.

##  Appendix J Experiment 2: Counterfactual Embedding Modularity Analysis

Here, we test for modularity using counterfactual embeddings. Counterfactual
embeddings are designed to reveal whether probes are reflecting information
that is causally implicated in model behavior. We summarize the relevant
details of their technique here, but defer to Tucker et al. (2021) for a full
treatment. Given a probe, $P_{\theta}$ trained to decode an intermediate
variable, $V$, consider a residual stream state $e$ such that
$P_{\theta}(e)=V_{i}$. We freeze $P_{\theta}$ and optimize $e$ such that
$P_{\theta}(e^{\prime})=V_{j}$. If $P_{\theta}$ is decoding information that
is causally implicated in the underlying model, then replacing $e$ with
$e^{\prime}$ should change the output. If this information is not causally
implicated in the underlying model, then replacing $e$ with $e^{\prime}$
should have no effect. For both tasks, we generate counterfactual embeddings
using the nonlinear probes trained to decode the “free” and “other” variables
from the residual stream after the attention block. From Table 6, we see that
model performance drop to near-zero after patching in counterfactual
embeddings for either “free” or “other” variables. This suggests that that
counterfactual embeddings act more as adversarial examples to the probe,
rather than providing information about causally-relevant variables.

Train Task | Test Task | Probe Var. | CE Success | Model Acc.
---|---|---|---|---
1 | 1 | Free | 100% | 2.1%
1 | 1 | Other | 100% | 1.8%
2 | 2 | Free | 100% | 2.1%
2 | 2 | Other | 100% | 2.3%
Table 6: Experiment 2 counterfactual embedding modularity results. If the
theory behind counterfactual embeddings is correct, only counterfactual
embeddings optimized to change the prediction of a probe that decodes causal
information should produce different output when patched into the underlying
model. Concretely, counterfactual embeddings affect the “free” variable probe
should result in a different prediction being made in the overall model.
Counterfactual embeddings that affect the “other” variable probe should have
no effect on the overall model’s prediction. We see that this does not happen.
Though all counterfactual embeddings succeed at changing the probe prediction
(CE Sucess), they also all change the overall model prediction (CE Acc.).

##  Appendix K Experiment 3: MLP Probe Accuracy

Here we present the linear and circuit probe accuracy on the MLP block
throughout training for Experiment 3. In Figure 11 see very messy results,
further reinforcing that the intermediate variable $a^{2}$ is computed in the
attention block.

Figure 11: Experiment 3: MLP Probing results. We see chaotic results from both
linear and circuit probes, indicating that the intermediate variable $a^{2}$
is not computed in the MLP block.

##  Appendix L Experiment 3: Selectivity Throughout Training

We present results of our selectivity analysis in Experiment 3 throughout
training. We see that linear probing consistently decodes the non-causal
variable $b^{2}$, while circuit probing is consistently at chance performance.
This illustrates a key problem with linear probing – it can learn to represent
variables that the model does not explicitly represent.

Circuit probing results are generated by a 1-nearest neighbors classifier
trained on the outputs of the MLP block after masking. For the variable
$b^{2}$, there are only two distinct integers that map to the same value of
that variable (i.e. $4^{2}\text{(mod 113)}=111^{2}\text{(mod 113) = 4})$.
Because we are training the classifier with 1 vector per variable label, 50%
of underlying integers are represented in the 1-NN training set. Thus, circuit
probing accuracy of 50% means that the block is merely decoding the identity
of the underlying token, rather than meaningfully computing an intermediate
variable.

Figure 12: Experiment 3: Selectivity Analysis. Linear and circuit probing
results for $b^{2}$ – a variable that is not causally implicated in the task
$a^{2}+b$. Circuit probing reveals that this variable is not represented at
any point during training, whereas linear probing would imply that it is
represented from the start of training.

##  Appendix M Experiment 4: Language Prefix Examples

Here, we present several examples of sentence prefixes for subject-verb
agreement and reflexive anaphora.

Subject-Verb Agreement:

IID: 1-Distractor

  1. 1.

the farmers that the taxi driver admires (are)

  2. 2.

the authors behind the assistants (are)

  3. 3.

the consultant that the skaters like (is)

OOD: 2-Distractors

  1. 1.

the books by the architects next to the executives (are)

  2. 2.

the customer that the skaters like and the ministers hate (is)

  3. 3.

the pilots in front of the dancers to the side of the parents (is)

Reflexive Anaphora:

IID: 1-Distractor

  1. 1.

the consultants that the parents loved doubted (themselves)

  2. 2.

the senators that the taxi drivers hate congratulated (themselves)

  3. 3.

the mechanics thought the pilot hurt (herself/himself).

OOD: 2-Distractors

  1. 1.

the surgeon that the chefs love and the parents admire hated (herself/himself)

  2. 2.

the mechanic knew the banker said the farmer embarrassed (herself/himself)

  3. 3.

the teachers that the architect hates and the minister admires hurt
(themselves)

##  Appendix N Experiment 4: GPT2-Small Language KNN Results

We present circuit probing KNN evaluations for both subject-verb agreement and
reflexive anaphora over all model components. Recall that causal analyses
indicate that a circuit that is causally implicated in computing syntactic
number is located in layer 6’s attention block. We note that KNN accuracy
increases for every MLP block after layer 6. Because MLP blocks are applied
token-wise, this suggets that the information required to decode syntactic
number of both subjects and referents is present in the residual stream after
this layer, but not before. See Figure 13.

Figure 13: GPT2-Small KNN results for subject-verb agreement (Top) and
reflexive anaphora (Bottom). We notice that KNN accuracy increases for MLP
block after layer 6, which is where our ablation analysis located the causal
circuits for both phenomena.

##  Appendix O Experiment 4: GPT2-Small Full Ablation Results

We present circuit probing ablation results for both subject-verb agreement
and reflexive anaphora over all model components. In all cases, we note that
ablating the circuit in attention block in layer 6 provides the greatest drop
in model performance. Randomly ablating subnetworks of the same size does not
harm model performance.

Figure 14: GPT2-Small subject-verb agreement ablation results for every model
component. We note that the attention block in layer 6 provides the greatest
drop in performance after ablating the discovered circuit.

Figure 15: GPT2-Small reflexive anaphora ablation results for every model
component, evaluated using the masculine pronoun. We note that the attention
block in layer 6 provides the greatest drop in performance after ablating the
discovered circuit.

Figure 16: GPT2-Small reflexive anaphora ablation results for every model
component, evaluated using the feminine pronoun. We note that the attention
block in layer 6 provides the greatest drop in performance after ablating the
discovered circuit.

##  Appendix P Experiment 4: GPT2-Medium Results

###  P.1 Reflexive Anaphora

We present results analyzing GPT2-Medium’s ability to compute the syntactic
number of the referent of a reflexive pronoun. We find that the attention
block in layer 7 is most causally implicated in this computation. See Figure
17. Ablating the discovered circuits harms model performance, regardless of
the pronoun used for evaluation. Ablating random subnetworks of the same size
does not harm model performance.

Figure 17: GPT2-Medium reflexive anaphora ablation results for the attention
block in layer 7.

Turning to the reflexive anaphora KNN evalation, we see that the KNN accuracy
of circuits trained on MLP blocks increases during and after layer 7. Because
MLP blocks operate token-wise, this indicates that the information required to
decode the syntactic number of referents is present in the residual stream
after this layer, but not before. This strengthens our causal results
analysis. See Figure 18.

Figure 18: GPT2-Medium reflexive anaphora KNN evaluation results across all
model components.

For completeness, we include ablation results across all model components in
Figures 19 and 20.

Figure 19: GPT2-Medium reflexive anaphora ablation results across all model
components, evaluated using the masculine pronoun. Note that the largest drop
in performance due to ablation occurs at the attention block in layer 7.

Figure 20: GPT2-Medium reflexive anaphora ablation results across all model
components, evaluated using the feminine pronoun. Note that the largest drop
in performance due to ablation occurs at the attention block in layer 7.

###  P.2 Subject-Verb Agreement

We do not find any particular circuits that drop subject-verb agreement
performance substantially when ablated (See Figure 21). This might indicate
that multiple circuits across several blocks are redundantly computing the
syntactic number of the subject noun.

Figure 21: GPT2-Medium subject-verb agreement ablation results across all
model components. These results do not implicate any specific circuit in
computing the syntactic number of the subject noun.

Turning to the subject-verb agreement KNN evalation, we see that the KNN
accuracy of circuits trained on MLP blocks increases after layer 7 (see Figure
22). Because MLP blocks operate token-wise, this might indicate that the
information required to decode the syntactic number of referents is present in
the residual stream after this layer, but not before. However, our causal
analysis does not provide strong evidence of this. From Figure 21, we see a
small drop at layer 7’s attention block, and another at layer 18’s attention
block.

Figure 22: GPT2-Medium subject-verb agreement KNN results. We see KNN
performance increase for MLP blocks around layer 7.

##  Appendix Q Experiment 4: Circuit Overlap

Surprisingly, we see that there is very little overlap between the circuits
used to compute the syntactic numbers of subjects and referents in GPT2-Small,
despite both circuits being present in the same block. See Figure 23

Figure 23: Circuit overlap between syntactic number and reflexive anaphora in
GPT2-Small, attention block 7. We see that the discovered circuits are largely
distinct. We also note that certain attention heads (0, 3, and 7) appear to be
most important in computing syntactic number for both subject nouns and
referents.

##  Appendix R Subject Verb Agreement GPT2 Medium Qualitative Results

We present qualitative results of ablating the subject-verb agreement circuit
discovered by running circuit probing on the attention block in layer 6 of
GPT2-Small (See Table 7). We note that the types of tokens predicted by model
qualitatively stay the same before and after ablation. This suggests that we
have not destroyed the model by ablating the discovered circuit.
Interestingly, we see more tokens that are explicitly consistent with the
syntactic number of the subject before ablation, and fewer after ablation.
This provides qualitative evidence that we have indeed ablated a circuit that
was responsible for tracking the subject-verb agreement dependency.

Prefix | Original Output | Ablate Attn-6 Output
---|---|---
The surgeons behind the dancer |  |
| ’s | ’s
| , | ,
| were | .
| . | and
| and | _
| are | )
| said | to
| _ | in
| had | who
| have | was
| to | ).
| who | is
| in | said
| was | ),
| ) | were
The book from the executives |  |
| of | of
| at | at
| , | who
| ’ | ’
| who | and
| , | ,
| . | in
| and | .
| in | to
| that | that
| was | )
| is | on
| to | I
| themselves | were
| on | are
Table 7: Qualitative examples of the effect of ablating the circuit discovered
in GPT2-Small, layer 6. We record the top 15 next-token predictions. Words
that are explicitly consistent with the syntactic number of the subject are
bolded, words that are inconsistent are underlined.

##  Appendix S Investigating Syntactic Number Feature Extraction

#### Goal

The experiments presented in Section 3.4 demonstrate that linguistic
dependencies (e.g. the influence of a previous word’s syntactic number on main
verb prediction) are computed in the middle layers of both GPT2-Small and
Medium. However, this leaves open the question: where in the model are the
syntactic number features extracted in the first place? In this section, we
uncover a circuit that computes syntactic number features and demonstrate that
ablating this circuit completely destroys a model’s ability to perform
subject-verb agreement.

#### Task

We use a subset of the 1-distractor subject-verb agreement dataset from
Section 3.4 for this experiment. Specifically, we use the subset that of data
that includes prepositional phrases (e.g. the farmer near the parents). In
these sentences, the final word before the main verb is always a non-subject
noun.

#### Analysis Methods

We run circuit probing on the last token of the non-subject noun in the
sentence prefixes to uncover the circuit that computes the syntactic number of
the non-subject noun (rather than the subject noun). Thus, we are effectively
uncovering a syntactic number feature extractor circuit. To ensure that our
discovered circuit truly is a general syntactic number feature extractor, we
ablate the discovered circuit and test the model’s ability to generate
syntactically valid next-token predictions using the same evaluation as
described in Section 3.4. The experimental logic is that if the discovered
circuit is responsible for syntactic number feature extraction in general,
then ablating it will also destroy a model’s ability to extract the syntactic
number of the subject noun, rendering the subject-verb agreement task
impossible.

#### Results

We expect that ablating the discovered circuit will render the model worse at
extracting the syntactic number of all nouns, destroying performance on our
subject-verb agreement evaluation. We expect that ablating random subnetworks
should not harm performance on this task. We run circuit probing over all
model components for both GPT2-Small and Medium. We present results from both
models (See Figure 24). For both models, we find that the syntactic number is
computed in early MLPs (layer 0 for GPT2-Small, layer 1 for GPT2-Medium).
Ablating the circuit returned by circuit probing drops subject-verb agreement
performance nearly to chance, while ablating random subnetworks does not
impact model performance. We present results from all layers for GPT2-Small in
Figure 25 and for GPT2-Medium in Figure 26. We note that it seems that
GPT2-Medium also performs syntactic number extraction in layer 0’s MLP.
Finally, we include KNN evaluation results in Figures 27 and 28.

Figure 24: GPT2-Small (left) and Medium (right) ablation results on early
MLPs. We find that the circuit that extracts syntactic number features from
tokens is computed in early MLPs. Ablating this circuit destroys subject-verb
agreement performance, while ablating random circuits of the same size does
not substantially impact performance.

Figure 25: GPT2-Small syntactic number ablation results across all model
components. Note that the largest drop in performance due to ablation occurs
at the MLP block in layer 0.

Figure 26: GPT2-Medium syntactic number ablation results across all model
components. Note that the largest drop in performance due to ablation occurs
at the MLP blocks in layer 1 and layer 0. However, ablating random subnetworks
has less of an impact on model performance in layer 1’s MLP. Figure 27:
GPT2-Small syntactic number KNN results. We see KNN performance reach ceiling
after MLP 0, and stay at ceiling until late in the model. Figure 28:
GPT2-Medium syntactic number KNN results. We see KNN performance reach ceiling
after MLP 0, and stay at ceiling until late in the model.