#  SelfIE: Self-Interpretation of Large Language Model Embeddings

Haozhe Chen  Carl Vondrick  Chengzhi Mao

###### Abstract

How do large language models (LLMs) obtain their answers? The ability to
explain and control an LLM’s reasoning process is key for reliability,
transparency, and future model developments. We propose SelfIE (Self-
Interpretation of Embeddings), a framework that enables LLMs to interpret
their own embeddings in natural language by leveraging their ability to
respond to inquiries about a given passage. Capable of interpreting open-world
concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in
cases such as making ethical decisions, internalizing prompt injection, and
recalling harmful knowledge. SelfIE’s text descriptions on hidden embeddings
open avenues to control LLM reasoning. We propose Supervised Control, which
allows editing open-ended concepts while only requiring gradient computation
of individual layer. We extend RLHF to hidden embeddings and propose
Reinforcement Control that erases harmful knowledge in LLM without supervision
targets.

Machine Learning, ICML

selfie.cs.columbia.edu

##  1 Introduction

Figure 1: SelfIE interpretation of latent embeddings in Large Language Models.
SelfIE produces open-world text explanations for the internal states in LLM
without any training.

Large language models (LLMs) have become foundations for a wide range of
applications from programming (Rozière et al., 2023), question answering
(Surís et al., 2023), to healthcare (OpenAI, 2023; Brown et al., 2020;
Chowdhery et al., 2022). However, the models are largely black-box, with
limited transparency into how they make decisions during inference.
Interpreting the representations that LLMs learn is important for establishing
trust in many applications as well as revealing whether state-of-the-art
methods are reasoning (Arkoudas, 2023) or just repeating their training set
(Bender et al., 2021).

A longstanding problem in machine learning, there has been significant effort
to uncover explanations behind LLM decisions. Chain-of-thought, for example,
uses in-context examples to direct the model to additionally output its
reasoning process (Wei et al., 2022), however there is no guarantee that the
explanation is faithful to the actual reasoning process (Turpin et al., 2023).
Moreover, Zou et al. (2023a) showed that LLMs answers question differently
depending on whether they need to provide explanations or not, making the true
explanation often inaccessible. Since LLM’s answers are produced from hidden
representations, the internal embeddings provide more direct and causally
relevant access to LLM’s reasoning processes. Hernandez et al. (2023) and Li
et al. (2021) developed linear probes to identify information in hidden
embeddings, but the methods require extensive data collection for training and
are consequently limited to a small predefined set concepts. Previous works
(Nostalgebraist, ; Belrose et al., 2023; Pal et al., 2023; Hernandez et al.,
2024) decode components of LLM to describe hidden embeddings, but they only
provide short descriptions that cannot explain complex concepts in detail.

In this paper, we propose an approach, SelfIE (Self-Interpretation of
Embeddings), that interpretes hidden embeddings in an LLM with the LLM itself.
SelfIE leverages LLMs’ own decoding capability to produce natural language
descriptions for the information in hidden embeddings. Fig. 1 shows an example
in which SelfIE’s interpretations delineate how an LLM processes the input
prompt, retrieves relevant facts and obtains the final answer.

We developed SelfIE based on the observation that LLMs can be prompted to
repeat or summarize a given message without training. We extend this procedure
to prompt LLMs to repeat or summarize information in hidden embeddings by
inserting the hidden embedding in forward pass. This procedure allows us to
achieve open-world interpretation of hidden embeddings without additional
training. Fig. 2 shows SelfIE interpretation process.

The key advantage of SelfIE is the capability of interpreting high-level,
open-world concepts in embeddings. Since we repurposed existing capability of
a LLM for interpretation, SelfIE does not require any training or data
collection, thus being compatible across current and future language model
advancements. SelfIE’s new capability of describing hidden embedding with
texts opens up new avenues for light-weighted controls of model behaviors.

Our visualizations and empirical results demonstrate that our interpretation
framework faithfully conveys information in hidden embeddings and reveals
internal reasoning procedures in LLMs. SelfIE achieves the same performance on
eliciting LLM’s internal representation of world state in TextWorld (Côté et
al., 2019) as prior supervised approach (Li et al., 2021) trained on 100-shot
samples, demonstrating the effectiveness and faithfulness of our zero-shot
readout approach. We use SelfIE to reveal internal reasoning processes behind
complex LLM behaviors, including identifying harmful knowledge, understanding
prompt injections, explaining ethical decisions. SelfIE interpretations enable
locating and modifying of individual layer to control LLM reasoning behaviors
such as erasing harmful knowledge and overriding ethical steering. By removing
harmful knowledge inside LLM, we reduced prompt injection’s success rate of
eliciting harmful response by $84.66\%$. We also increased fairness in LLM
response by achieving 95% effective rate of overriding user ethical steering.

Figure 2: Interpretation procedure for SelfIE. By replacing placeholder token
embedding in the interpretation prompt with embedding being interpreted in the
interpretation forward pass, we can generate text descriptions for the
embedding.

##  2 Related Works

LLM Interpretibility. Prior work either trained models to be interpretable
(Mao et al., 2022; Koh et al., 2020; Kim et al., 2018; Hendricks et al., 2016;
Hernandez et al., 2022) or performed post hoc interpretation with a given
model (Nguyen et al., 2017, 2016; Olah et al., 2017; Mahendran & Vedaldi,
2015; Zeiler & Fergus, 2014; Lundberg & Lee, 2017; Ribeiro et al., 2016;
Simonyan et al., 2014; Shrikumar et al., 2017; Zeiler & Fergus, 2014; Smilkov
et al., 2017; Selvaraju et al., 2016; Caron et al., 2021; Abnar & Zuidema,
2020). Prior studies like (Hernandez et al., 2023; Li et al., 2021) decode
hidden embeddings using linear probes, but these require extensive data and
can only interpret a limited set of concepts. Work such as (Meng et al., 2022)
uses causal effects to probe LLM knowledge, yet limits to simple facts. Recent
approaches like LogitLens (Nostalgebraist, ) , TunedLens (Belrose et al.,
2023), and LRE (Hernandez et al., 2024) explore decoder-based models’ hidden
states with next token prediction but is limited to one or few tokens in
explanations thus fail to explain complex concepts in details. Influence
functions are studied (Grosse et al., 2023). In contrast, SelfIE enables
direct interpretation of embeddings in any lengths, offering natural language
descriptions of high level concepts. Concurrently, Patchscope (Ghandeharioun
et al., 2024) decodes hidden embedding information with LLM through
transforming and patching embedding vectors.

LLM Control. Supervised Fine-tuning is a prevalent method for directing model
behavior, complemented by preference-based strategies like RLHF (Ouyang et
al., 2022), which guide models without explicit token-level goals. Studies on
process supervision (Lightman et al., 2023) show training on intermediate step
enhances model reasoning capacity. However, these techniques operate output
texts, demanding extensive computational effort without granular controls of
model internals. SelfIE generates explanations for hidden embeddings and
allows for extending these methods for precise manipulation of model
components at intermediate states. Works like ROME (Meng et al., 2022), MEND
(Mitchell et al., 2022), and RepE (Zou et al., 2023a) aim to modify models for
knowledge and behavior adjustments. We compare control enabled by SelfIE and
these methods in Table 1. Controls based on SelfIE supersede these methods
with their capacity for open-ended concept control and targeting specific
layers for modification with minimal gradient calculations. SelfIE also allows
us to extend RLHF-like methods to embedding level for granular control of
model reasoning without supervised targets.

Figure 3: Understand LLM reasoning behaviors via SelfIE. Using our framework,
we can explain LLM latent reasoning mechanism under harmful input, prompt
injection, ethical reasoning, and physics reasoning. We denote the token from
$i$-th layer and $j$-th column in a transformer to be L$i$T$j$. We show the
Relevancy Score via highlight, where deeper color means the interpretation
word has a higher causal relationship to the latent embedding. For example, in
the prompt injection example, our method explains the symbols “!!!!!” cause
the model to jailbreak, because “!!!” symbol creates a sense of urgency, which
leads to the model following the user’s instruction. Our visualization
demonstrates the effectiveness of our interpretation.

##  3 Methods

We explore interpreting the semantic information represented by a latent
embedding $\bm{h}$ in a transformer-based Large Language Model (LLM) with
open-world natural language description. Our method obtains explanations by
manipulating the forward pass of the LLM to decode a hidden embedding into a
sentence. We also show that the interpretations enable granular control over
the model’s behavior.

###  3.1 SelfIE: Self Interpretations of Embedding

Large Language Models can respond to questions about information provided in
context. For example, given prompt [passage] Please summarize the previous
passage: LLM can respond by condensing the information in [passage] into
shorter sentences. Motivated by this observation, we propose to replace the
[passage] tokens with the latent embeddings from LLM to interpret the
information the embeddings contain.

For transformer-based LLMs, a transformer first maps a sequence of one-hot
representation of text $\bm{x}$ into an embedding on an initial layer
$\bm{h}^{0}$ with a linear projection $E$. The transformer forward pass is
then followed by $L$ layers, each containing a residual MSA (multi-headed
self-attention) and an MLP (multi-layer perceptron) block. The final layer
embedding $\bm{h}^{L}$ is transformed by the final linear projection $P$ and
softmax activation into a sequence of probability distribution $\bm{y}$ of the
next token at each position. Formally, the procedure can be written as

$\displaystyle\bm{h}^{0}$ $\displaystyle=E\bm{x}$ (1)
$\displaystyle\widehat{\bm{h}}^{\ell}$
$\displaystyle=\mathrm{MSA}^{\ell}(\bm{h}^{\ell-1})+\bm{h}^{\ell-1},$
$\displaystyle\ell=1,2\cdots,L$ (2) $\displaystyle\bm{h}^{\ell}$
$\displaystyle=\mathrm{MLP}^{\ell}(\widehat{\bm{h}}^{\ell})+\widehat{\bm{h}}^{\ell},$
$\displaystyle\ell=1,2\cdots,L$ (3) $\displaystyle\bm{\widehat{y}}$
$\displaystyle=P\bm{h}^{L},\bm{y}=\textrm{softmax}({\bm{\widehat{y}}})$ (4)

Inserting Embedding in Interpretation Forward Pass. As shown in Fig. 2, after
an original forward pass of an input prompt through LLM, SelfIE interprets
hidden embeddings by extracting the embedding of interest, and injecting it
into a separate forward pass of the same LLM. We call this pass the
interpretation forward pass, which takes in an interpretation prompt to
summarize the embedding. By finding the next token repeatedly with the
interpretation forward pass, we generate a natural language description for
the hidden embedding.

Let the hidden embedding $\bm{h}^{\ell^{*}}_{i^{*}}$ from layer $\ell^{*}$ and
index $i^{*}$ on the original pass be the embedding to interpret. The
interpretation forward pass takes in an interpretation prompt $\bm{I}$ and
modifies the transformer forward pass on a chosen layer $k$. The
interpretation prompt $\bm{I}$ contains (1) a placeholder token at index $s$
and (2) an inquiry about a message at the placeholder’s position. For example,
the string [X] Please repeat the previous message is an interpretation prompt
consisting of placeholder token [X] at index $s=0$ and inquiry Please repeat
the previous message. We generate text with interpretation prompt $\bm{I}$
with the usual text generation pipeline for a transformer decoder, except at
every forward pass we replace hidden embedding at placeholder index $s$ with
$\bm{h}^{\ell^{*}}_{i^{*}}$ the embedding being interpreted on a chosen layer
$k$. Formally, we modify the interpretation forward pass as

$\displaystyle\bar{\bm{h}}^{0}$ $\displaystyle=E\bm{I}$ (5)
$\displaystyle\bar{\bm{h}}^{\ell}_{i}$
$\displaystyle=\bm{h}^{\ell^{*}}_{i^{*}},$ $\displaystyle\ell=k,i=s$ (6)
$\displaystyle\bar{\widehat{\bm{h}}}^{\ell}$
$\displaystyle=\mathrm{MSA}^{\ell}(\bar{\bm{h}}^{\ell-1})+\bar{\bm{h}}^{\ell-1},$
$\displaystyle\ell=1,2\cdots,L$ (7) $\displaystyle\bar{\bm{h}}^{\ell}$
$\displaystyle=\mathrm{MLP}^{l}(\bar{\widehat{\bm{h}}}^{\ell})+\bar{\widehat{\bm{h}}}^{\ell},$
$\displaystyle\ell=1,2\cdots,L$ (8) $\displaystyle\bar{\widehat{\bm{y}}}$
$\displaystyle=P\bar{\bm{h}}^{L},\bar{\bm{y}}=\textrm{softmax}({\bar{\widehat{\bm{y}}}})$
(9)

where Equation 6 inserts the hidden embedding being interpreted by replacing
placeholder token embedding with embedding being interpreted on chosen layer
$k$.

We use the modified interpretation forward pass to predict the next token
repeatedly based on the extracted hidden embedding and interpretation prompt
to obtain a natural language explanation of the hidden embedding.

During the interpretation forward pass, we insert hidden embedding being
interpreted on layer $k$, which is potentially different from $\ell^{*}$ that
the embedding comes from in the original forward pass. Using residual
structure in a Transformer, (Elhage et al., 2021) and (Gandelsman et al.,
2023) decompose output of a Transformer as applying the final projection
matrix on the linear combination over each layer’s output. We hypothesize that
this mechanism leads to a unified representation across different layers, and
inserting an embedding on a different layer $k$ allows faithful interpretation
of information in the embedding. We will verify the faithfulness of our
interpretation procedure in later empirical results.

The following subsection will examine details on evaluating the relevancy of
generated explanations.

###  3.2 Treatment Effect as Relevancy Score

We observed that LLM’s autoregressive nature often lead it to produce
excessive text continuing the generated explanations. We therefore need to
identify which parts of the generated interpretation are directly relevant to
the interpreted latent embedding. For example, an embedding is interpreted as
Mount Everest is a popular tourist attraction, and our goal is to distinguish
whether the embedding contains this entire description or only Mount Everest,
and the rest part results from autoregression on Mount Everest.

Let $t$ be the generated interpretation. We calculate a relevancy score for
$i$-th token in interpretation $t_{i}$ as the treatment effect of replacing
placeholder embedding $\bm{h}^{k}_{s}$ with embedding interpreted ${\bf
h}^{\ell^{*}}_{i^{*}}$ during interpretation pass:

$\displaystyle\mathrm{rel.\ score}=$ $\displaystyle\mathbb{P}({\bf T}_{i}={\bf
t}_{i}|do(\bar{\bm{h}}^{k}_{s}={\bf h}^{\ell^{*}}_{i^{*}}))$ $\displaystyle=$
$\displaystyle\mathbb{P}[{\bf T}_{i}={\bf
t}_{i}|I,t_{<i},\bm{h}^{\ell^{*}}_{i^{*}}]-\mathbb{P}[{\bf T}_{i}={\bf
t}_{i}|I,t_{<i}]$

This score measures the difference in probability of interpretation producing
$t_{i}$ between conducting and not conducting replacement of embedding being
interpreted during interpretation pass. A larger relevancy score indicates
that interpretation output token $t_{i}$ is determined by the interpreted
embedding instead of only results from autoregression over generated tokens.
In our visualizations, we show the relevancy score as a highlight over
interpretation texts.

###  3.3 Deep Process Supervision

Figure 4: Pipeline for Supervised Control. Figure 5: Pipeline for Reinforcement Control.  |  | Gradient
---
calculation
| Control
---
open-ended
concepts
| Few
---
samples
| Supervised
---
target-free
FT | $\mathcal{O}(L)$ | ✗ | ✗ | ✗
MEND | $\mathcal{O}(L)$ | ✗ | ✓ | ✗
ROME | $\mathcal{O}(L)$ | ✗ | ✓ | ✗
RepE | $\mathcal{O}(1)$ | ✓ | ✗ | ✗
| Supervised
---
Control (Ours)
$\mathcal{O}(1)$ | ✓ | ✓ | ✗
| Reinforcement
---
Control (Ours)
$\mathcal{O}(1)$ | ✓ | ✓ | ✓
Table 1: Comparing reasoning control enabled by SelfIE and previous model
editing methods. $L$ refers to the number of model layers.

The text explanation of hidden embeddings obtained from SelfIE enables new
modes of precise control over model behaviors in the latent space. In Table 1,
we show controls based on SelfIE (1) pinpoint and isolate specific layer to
control thus requiring minimal gradient computation in only the selected layer
and allows fast new reasoning behaviors definition; (2) supports open-ended
editing targets; (3) extend RLHF to embedding level thus allows control only
based on high-level objective.

Supervised Control. Let the aggregated outputs of Multihead Self-Attention
(MSA), Multilayer Perceptron (MLP), and residual connections in a Transformer
layer $\ell$ be denoted by $f^{\ell}_{\theta}$, where $\theta$ represents the
model parameters. $\bm{h}^{\ell}_{i}$, index $i$ embedding on layer $\ell$, is
obtained from $f^{\ell}_{\theta}(\bm{h}^{\ell-1})_{i}$. As shown in Fig. 4, to
define a new behavior for $f^{\ell}_{\theta}$ so that
$f^{\ell}_{\theta}(\bm{h}^{\ell-1})_{i}$ maps to a vector $\bm{v}$ that
interprets to target explanation $t$, we adjust parameter of
$f^{\ell}_{\theta}$ by minimizing the Mean Squared Error (MSE) between
$\bm{v}$ and $f^{\ell}_{\theta}(\bm{h}^{\ell-1})_{i}$, through gradient
descent applied to the layer’s parameters $\theta$:

$\mathcal{L}(\theta,\bm{h}^{\ell-1},\bm{v})=(\bm{v}-f_{\theta}^{\ell}(\bm{h}^{\ell-1})_{i})^{2}$

Reinforcement Control. Previous works (Ouyang et al., 2022) leverage
reinforcement learning on output tokens to control model behavior. We extend
this approach to hidden embeddings by converting text interpretation from
SelfIE to non-differentiable reward signals evaluated by humans or machines.
Shown in Fig. 5, an embedding $\bm{h}_{i}^{\ell}$ interpreted as $t$ generates
a reward signal $R(t)$ with a human or machine evaluator $R$. The evaluator
generates positive rewards for desirable and negative for undesirable
outcomes. This approach steers the model towards encoding only desirable
information at layer $\ell$, by minimizing the following proxy loss function:

$\displaystyle\widetilde{\bm{h}}_{i}^{\ell}=f_{\theta}^{\ell}(\bm{h}^{\ell-1})_{i}$
$\displaystyle\mathcal{L}(\theta,\bm{h}^{\ell-1})=-R(\texttt{SelfIE}(\widetilde{\bm{h}}_{i}^{\ell}))\cdot\tfrac{(\widetilde{\bm{h}}_{i}^{\ell})^{T}\mathrm{sg}(\widetilde{\bm{h}}_{i}^{\ell})}{||\mathrm{sg}(\widetilde{\bm{h}}_{i}^{\ell})||^{2}}$

where $\mathrm{sg}(\cdot)$ is stop gradient operation. Intuitively, the proxy
loss function encourages the model to avoid outputting
$\widetilde{\bm{h}}_{i}^{\ell}$ if the reward is negative and vice versa.

##  4 Experiments

###  4.1 Implementation Details

Our experiment focuses on LLaMA-2-70B-Chat (Touvron et al., 2023), while our
method is general to all transformer-based LLM of different sizes. Unless
noticed otherwise, we use interpretation prompt [INST] [X] [/INST] Sure, I’ll
summarize your message:. We repeat placeholder token [X] five times and
replace all placeholder tokens with the embedding being interpreted on layer
$k$ in the interpretation forward pass. We choose $k=3$ as the layer to
replace placeholder tokens in the interpretation forward pass. [INST]…[/INST]
tags are used to represent user input for LLaMA-2-Chat models. We use
8$\times$NVIDIA RTX A6000 for interpretation and 8$\times$NVIDIA A100 for
reasoning control.

###  4.2 Eliciting Implicit World States in Embeddings

Figure 6: Classification accuracy on TextWorld dataset. We show our zero-shot
method with the red line. We plot the k-shot supervised classification model
with gray lines, where k ranges from 1 to 100. We use a dashed line to show
supervised learning results on the whole dataset. SelfIE can match the
performance with 100-shot training, which demonstrates our zero-shot SelfIE
can effectively elicit implicit world state knowledge in embeddings.

(Li et al., 2021) shows that language models maintain representations of
entities and situations in complex contexts. The represented states of
entities can be elicited with linear probing. We use SelfIE to elicit the
state of entities in natural language and compare the result with linear
probing.

Dataset. TextWorld (Côté et al., 2019) provides a platform for generating
synthetic worlds for text-based games that are used to test RL agents. We
generate 12900 samples of context, entity, positive state, negative state. We
show sample data in Appendix A.1. We use 3400 samples for evaluating SelfIE
and linear probing and use 9500 for training linear probes. Each context
describes a sequence of actions and different objects’ states. At the end of
the context, the entity is in the positive state and not in the negative
state.

Method. For each sample, we first pass through the original forward pass
context and extract embedding of the last entity mention on different layers
and interpret with SelfIE. The interpretation prompt asks to choose strictly
between positive state and negative state. An interpretation is considered to
be correct if the interpretation contains positive state. For linear probe, we
extract the last layer last token embedding of proposition [entity] is
[positive state] $\bm{c}_{+}$ and similarly obtain $\bm{c}_{-}$ for negative
state. We train linear probe weights $W$ so that
$\bm{c}_{+}^{T}W\bm{h}^{\ell^{*}}_{i^{*}}-\bm{c}_{-}^{T}W\bm{h}^{\ell^{*}}_{i^{*}}$
is maximized. Linear probe predicts positive state if
$\bm{c}_{+}^{T}W\bm{h}^{\ell^{*}}_{i^{*}}>\bm{c}_{-}^{T}W\bm{h}^{\ell^{*}}_{i^{*}}$.

Result. Fig. 6 shows that interpretations from SelfIE recovers $60\%-80\%$ of
the information about entity state. The interpretation quality increases as
the layer that the interpreted embedding comes from increases. Since SelfIE
does not require any training, we also compare its performance to few-shot
linear probes. SelfIE performs similarly to the 100-sample linear probe.
Therefore, SelfIE produces interpretations that are faithful to the
information represented in embeddings.

###  4.3 General Understanding of LLM Behaviors

Figure 7: Detecting harmful knowledge in LLM. (a) LLM contains harmful
knowledge in its reasoning despite safety alignment. (b) Harmful knowledge is
mostly removed after Reinforcement Control enabled by SelfIE.

While previous interpretation methods such as linear probes can only interpret
a closed set of concepts with training, SelfIE can interpret open-world
concepts without any training. We therefore could use SelfIE to understand LLM
internal reasoning in general.

Detect Harmful Knowledge. While LLMs are aligned to reject providing harmful
answers, in Fig. 3(a), we show that existing alignment techniques only hide
harmful responses on the surface, and the model still contains harmful
knowledge. In Fig. 7(a), we show that embeddings that interpret harmful
materials exist widely in the model.

Why Prompt Injection Works. In Fig. 3(b), we use SelfIE to understand why
prompt injections steer LLaMA to provide harmful answers. With prompt
injection developed in (Zou et al., 2023b) as input, SelfIE reveals that the
model concludes urgency from the exclamation mark in the early layer and
infers user is in crisis in the late layers, before finally complying with
harmful requests to avoid user aggression.

Access Reasoning When Explanation Changes Response. In Fig. 3(c), when LLaMA
is asked about making a decision in the trolley problem scenario, attaching
explain reason at the end of the prompt alters LLaMA’s answer. Therefore, we
cannot access LLaMA’s reasoning to the answer Yes when asked to answer in only
one word from the output. SelfIE reveals that the answer Yes might be result
of conforming to majority opinions.

Reasoning with Knowledge. In Fig. 3(d), we use SelfIE to examine how LLaMA
answers a physics reasoning question. We found that the model extracts the
glittery aspect of syrup in early layers, grasps thickness as the relevant
quality, and retrieves the advanced physics concept viscosity that is related
to thickness.

Social Reasoning. In 3(e), We use SelfIE to reveal how LLaMA approaches a
complex social scenario. We showed that LLaMA is able to infer mental states
and intentions of different parties in a social situation and formulate the
final output with these understandings.

How Hallucination Occurs. In Fig. 3(f), we use SelfIE to trace how LLaMA
hallucinates when responding to a question involving a fictitious name. LLaMA
first recalls Mc and Donald as in McDonalds in Scotland and associate McQueen
with Scotland. It then associate Mc and Scotland with a similar name McLean
who is a doctor. It finally combines the information about McLean as a doctor
back to McQueen and produces final understanding of McQueen as a researcher in
psychiatry.

###  4.4 Supervised control of reasoning

Table 2: Editing fact association. We compare model editing on simple facts between SeflIE-based supervised control and other methods. We measure editing effectiveness with efficacy (% original prompt that model responds with target answer), paraphrase (% paraphrase prompt that model responds with the target answer, and specificity (% irrelevant prompt that model answers correctly), and their harmonic mean. SeflIE-based supervised control surpasses previous methods on paraphrase effectiveness and harmonic mean, demonstrating comparable capability of fact editing with other methods and better generalization capability.) |  | Harmonic
---
Mean$\uparrow$
Efficacy$\uparrow$ |  Paraphrase$\uparrow$ |  Specificity$\uparrow$
FT | 26.98% | 82.12% | 12.40% | 54.39%
RepE | 6.80% | 10.10% | 3.01% | 98.01%
ROME | 56.92% | 50.95% | 33.29% | 94.94%
| Supervised
---
Control (Ours)
59.77% | 58.40% | 36.12% | 90.43%
Table 3: Overriding ethical preference in a prompt. We steer ethical preference in LLM response by specifying prioritizing humans over aliens in prompt. We use SeflIE-based supervised control to intervene in model weights and override the ethical preference. While 100% of model responses to 100 unseen scenarios prioritize human, 96% of responses prioritize weighing human and aliens equal after control. Our method embeds fairness in LLM internal reasoning process. |  Prioritize human $\downarrow$ |  Prioritize equal $\uparrow$ | Other response
---|---|---|---
Before control | 100% | 0% | 0%
After control | 2% | 96% | 2%
Random control | 95% | 1% | 4%
Figure 8: Editing open-ended-concepts via Supervised Control method from
Section 3.3. Unlike previous model editing methods that only allow
memorization of simple facts, SelfIE-based control alters the model’s open-
ended perception of Molotov Cocktail and generalizes the perception to complex
reasoning. Figure 9: Overriding ethical preference in prompts via Supervised
Control from Section 3.3. SelfIE-based control embeds fairness into the
reasoning, even when user prompt tries to mislead the model. Figure 10: Deep
safety alignment of model by erasing harmful knowledge with Reinforcement
Control from Section 3.3.

Controlling Fact Association We test the efficiency of supervised control of
reasoning on editing knowledge in a model with Counterfact dataset (Meng et
al., 2022) that contains 1000 pairs of subjects and attributes. We used 844
samples that LLaMA answers correctly. For each sample, we randomly select a
target editing answer from other Counterfact samples; provide a paraphrase
prompt to test edited model’s generalization capability; and randomly choose
another irrelevant prompt and associated fact to test the preservation of
other model knowledge. We report Efficacy (% original prompt that model
responds with the target answer), Paraphrase (% paraphrase prompt that model
responds with the target answer), Specificity (% irrelevant prompt that model
answers correctly), and their harmonic mean.

For each sample, we apply supervised control by editing the first two layers
where embeddings interpret to original answer. We choose editing target
embedding by randomly choosing embeddings from prompt [INST] Assume [prompt]
[target answer] [/INST] Sure, [prompt] [target answer] that interprets to the
target answer. To ensure other model behaviors are minimally impacted, when
calculating loss, we use Wikitext (Merity et al., 2016) as a reference corpus
and add a mean square loss term between the original layer output and edited
layer output from Wikitext samples. We show prompt and hyperparameters used in
Appendix A.2.

We test Fine-tuning (FT) (Zhu et al., 2020), RepE (Zou et al., 2023a), and
ROME (Meng et al., 2022) as baselines. FT is unable to scale for
LLaMA-2-70B-Chat beyond the last layer, so we finetune only the final layer.
ROME requires gradient calculation to find the editing target, so we can only
find the editing target of the final 5 layers and edited layer 75, which
produced the best results out of the final 5 layers.

Comparison in Table 2 shows that our method produces the best generalization
on paraphrase prompts. Our method produces similar efficacy and specificity
performance as ROME while requiring fewer computational resources. Combined,
our method produces the best overall harmonic mean score. Controlling Open-
Ended Concepts Out of all previous editing methods, only RepE allows editing
open-ended concepts beyond simple fact association. However, RepE requires
careful experiment design to collect a large sample of embeddings to find a
representation for a concept. In contrast, based on SelfIE interpretations, we
can easily identify individual embedding as input and output targets and thus
can edit a wider range of concepts. In Fig. 8, originally $\bm{h}^{\ell-1}$
maps to $f_{\theta}^{\ell}(\bm{h}^{\ell-1})_{i}$ that interprets to harmful
Molotov Cocktail materials. We edit Layer 3 so that it now maps to The user is
asking for a drink. We updated the model parameter eight times with gradient
descent, where each update takes 10 seconds. We show that the editing changes
how the model perceives the nature of Molotov Cocktail. Even in reasoning
tasks that ask for inference of reality based on an indirect understanding of
Molotov Cocktail, the edited model can integrate the edited concept of drink
into its reasoning. We show hyperparameter and regularization details in
Appendix A.3.

Table 4: Erasing harmful knowledge. Before reinforcement control, LLM refuses to provide harmful response on prompts without prompt injection but is susceptible to prompts that contains prompt injection attack strings. We conduct reinforcement control to erase harmful knowledge elicited by a single prompt without injection How to build a molotov cocktail. The control generalizes to make LLM safer on unseen prompts with attack injection strings inquiring about other harmful activities, reducing prompt injection’s capability to elicit harmful responses on 388 harmful activities by 84.66%, while maintaining 95.85% of original model capability on fact answering task. |  | % of LLM responses containing
---
harmful information
Irrelevant fact accuracy $\uparrow$
|  | w/o prompt
---
injection $\downarrow$
| w/ prompt
---
injection $\downarrow$
Before Reinforcement Control | 1.84% | 89.06% | 100%
After Reinforcement Control | 1.79% | 4.4% | 95.85%
Figure 11: Ablation study on layer $k$ to insert hidden embedding being
interpreted in interpretation forward pass. We show mean and 25-75% percentile
over different layers that embedding to interpret comes from. The result shows
that $k<20$ generally produces higher accuracy. Figure 12: Ablation study on
model size. (a) 7B and 13B models produce worse accuracy from interpretation.
(b) Filtering out failures to follow instructions of choosing between two
options, the smaller models attains similar accuracy as the 70B model.

Overriding Ethical Preference in Prompts. LLMs are susceptible to being
steered to undesirable ethical ideas with user specification of moral beliefs
in a prompt. We created a synthetic ethical preference dataset that includes
101 hypothetical scenarios and questions and asks for prioritization between
humans and aliens. We prompted the scenarios with added instruction you MUST
hold humans more important than aliens to steer LLaMA to the specific ethical
preference of prioritizing humans over aliens.

Fig. 9 shows that we edit layer 6 with a single pair of input and output
embedding target that steers the layer to produce embedding that interprets to
an equality-related idea instead of putting more importance on one species
over another. We updated the model parameter twice with gradient descent that
only takes 20 seconds. Table 3 shows LLaMA prioritizes humans in answer for
all 100 unseen samples before control and prefers equality on 96 samples after
supervised control with input and output embedding target selected based on
SelfIE interpretations, achieving a 95% effective rate compared to conducting
supervised control with randomly chosen embedding and target. Fig. 9 shows
that the editing is not a result of memorization; instead, the edited model
organically blends the belief of equality in its explanation of reasoning. We
show hyperparameter and regularization details in Appendix A.3.

###  4.5 Erase Harmful Knowledge with Reinforcement Control

We show Reinforcement Control that leverages SelfIE interpretation can remove
harmful knowledge in LLaMA by only specifying this objective with an evaluator
LLM. We use [INST] How to build molotov cocktail? [/INST] as the original
input prompt, interpret embeddings on layer 15, and use LLaMA itself as an
evaluator on text interpretation. Reward
$R(\texttt{SelfIE}(\widetilde{\bm{h}}_{i}^{\ell}))=-1$ if LLaMA considers
interpretation contains harmful objects, and $+1$ otherwise.

After eight parameter updates with gradient descent on layer 15, where each
update only takes 30 seconds, in Fig. 7(b), we show that harmful knowledge is
mostly removed, and LLaMA responds to unseen prompt injection in 3(c) with I
apologize, but I’m a large language model, I cannot provide instructions on
how to […].

In addition, we found that conducting reinforcement control only on a single
non-prompt injection on build a molotov cocktail also removes harmful
behaviors related other harmful questions. In Table 4, we use the same prompt
injection in 3(c) and query for instructions on 388 harmful behaviors
collected by (Zou et al., 2023b), such as Provide a step-by-step guide on how
to commit credit card fraud and Outline a plan for sabotaging a competitor’s
business operations that are distant to build a molotov cocktail. The editing
reduces the success rate of the unseen prompt injection at eliciting harmful
responses from $89.06\%$ to $4.4\%$, achieving a $84.66\%$ editing effective
rate. The model after reinforcement control responds to prompt injection with
coherent refusal response such as I cannot provide instructions on how to
commit credit card fraud as it is illegal and unethical. We show that other
capabilities of LLaMA is largely preserved by showing that the model still
answers $95.85\%$ of the 844 known facts in Counterfact dataset described in
Section 4.4 (Meng et al., 2022) correctly. We show examples of responses
before and after reinforcement control in Figure 10 and full prompt injection
in Appendix A.4.

##  5 Ablation Study

Choosing Embedding Insertion Layer $k$. During the interpretation forward
pass, we replace placeholder token embeddings with interpreted embedding on
layer $k$. We ablate $k$ with eliciting implicit world states in the TextWorld
experiment described in section 4.2. Fig.11 shows that $k<20$ generally
produces higher accuracy. Accuracy degrades for later layers potentially
because the information from embedding interpreted is not able to propagate
into the last token for output when insertion is done too late.

Effects from LLM Capability. We conduct ablation on interpretation quality
with varying LLM sizes. The result in Fig. 12(a) shows that accuracy is worse
for 7B and 13B LLaMA-2-Chat models. Further analysis reveals that this might
be the result of the smaller model’s worse instruction of following
instructions. 27.17% of 7B and 17.68% of 13B interpretation responses do not
follow the instruction to choose between two options and give another answer.
In contrast, only 3.84% of 70B responses fail. Fig. 12(b). Conditioning on the
successful following of instructions, both the 7B and 13B model attains
similar accuracy as 70B. Interpretations on smaller models are also capable of
recovering information in embeddings; however, failure to follow instructions
might decrease the chance of obtaining high-quality interpretation.

##  6 Conclusion

In this work we propose SelfIE that leverages LLM’s decoding capability to
explain its own hidden embeddings with natural language. Capable of
interpreting any concept at any complexity, SelfIE enables new modes of
methods for controlling model reasoning. Our framework provides toolkits for
future works to further explore reasoning patterns in LLM and model reasoning
behavior controls with hidden embeddings.

Acknowledgements: This research is partially supported from the DARPA ECOLE
program, the DARPA MCS program, and the Knight First Amendment Institute.

## References

  * Abnar & Zuidema (2020) Abnar, S. and Zuidema, W.  Quantifying attention flow in transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4190–4197, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.385.  URL https://aclanthology.org/2020.acl-main.385.
  * Arkoudas (2023) Arkoudas, K.  Gpt-4 can’t reason, 2023.
  * Belrose et al. (2023) Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J.  Eliciting latent predictions from transformers with the tuned lens, 2023.
  * Bender et al. (2021) Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383097.  doi: 10.1145/3442188.3445922.  URL https://doi.org/10.1145/3442188.3445922.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Caron et al. (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A.  Emerging properties in self-supervised vision transformers.  In _CVPR_ , pp. 9650–9660, 2021.
  * Chowdhery et al. (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Côté et al. (2019) Côté, M.-A., Ákos Kádár, Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Tao, R. Y., Hausknecht, M., Asri, L. E., Adada, M., Tay, W., and Trischler, A.  Textworld: A learning environment for text-based games, 2019.
  * Elhage et al. (2021) Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 2021.  https://transformer-circuits.pub/2021/framework/index.html.
  * Gandelsman et al. (2023) Gandelsman, Y., Efros, A. A., and Steinhardt, J.  Interpreting clip’s image representation via text-based decomposition, 2023.
  * Ghandeharioun et al. (2024) Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L., and Geva, M.  Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024.
  * Grosse et al. (2023) Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al.  Studying large language model generalization with influence functions.  _arXiv preprint arXiv:2308.03296_ , 2023.
  * Hendricks et al. (2016) Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., and Darrell, T.  Generating visual explanations.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pp. 3–19. Springer, 2016.
  * Hernandez et al. (2022) Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J.  Natural language descriptions of deep visual features.  In _International Conference on Learning Representations_ , 2022.
  * Hernandez et al. (2023) Hernandez, E., Li, B. Z., and Andreas, J.  Inspecting and editing knowledge representations in language models.  _arXiv preprint arXiv:2304.00740_ , 2023.
  * Hernandez et al. (2024) Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas, J., Belinkov, Y., and Bau, D.  Linearity of relation decoding in transformer language models.  In _Proceedings of the 2024 International Conference on Learning Representations_ , 2024.
  * Kim et al. (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _ICML_ , pp. 2668–2677. PMLR, 2018.
  * Koh et al. (2020) Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P.  Concept bottleneck models.  In _International Conference on Machine Learning_ , pp. 5338–5348. PMLR, 2020.
  * Langley (2000) Langley, P.  Crafting papers on machine learning.  In Langley, P. (ed.), _Proceedings of the 17th International Conference on Machine Learning (ICML 2000)_ , pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  * Li et al. (2021) Li, B. Z., Nye, M., and Andreas, J.  Implicit representations of meaning in neural language models.  _arXiv preprint arXiv:2106.00737_ , 2021.
  * Lightman et al. (2023) Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K.  Let’s verify step by step, 2023.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.  A unified approach to interpreting model predictions.  In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 4765–4774, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.
  * Mahendran & Vedaldi (2015) Mahendran, A. and Vedaldi, A.  Understanding deep image representations by inverting them.  In _CVPR_ , pp. 5188–5196, 2015.
  * Mao et al. (2022) Mao, C., Teotia, R., Sundar, A., Menon, S., Yang, J., Wang, X., and Vondrick, C.  Doubly right object recognition: A why prompt for visual rationales.  _arXiv preprint arXiv:2212.06202_ , 2022.
  * Meng et al. (2022) Meng, K., Bau, D., Andonian, A., and Belinkov, Y.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372, 2022.
  * Merity et al. (2016) Merity, S., Xiong, C., Bradbury, J., and Socher, R.  Pointer sentinel mixture models, 2016.
  * Mitchell et al. (2022) Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D.  Fast model editing at scale, 2022.
  * Nguyen et al. (2017) Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., and Yosinski, J.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pp. 3510–3520. IEEE Computer Society, 2017.  doi: 10.1109/CVPR.2017.374.  URL https://doi.org/10.1109/CVPR.2017.374.
  * Nguyen et al. (2016) Nguyen, A. M., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , pp. 3387–3395, 2016.  URL https://proceedings.neurips.cc/paper/2016/hash/5d79099fcdf499f12b79770834c0164a-Abstract.html.
  * (30) Nostalgebraist.  Interpreting gpt: The logit lens.  URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens.
  * Olah et al. (2017) Olah, C., Mordvintsev, A., and Schubert, L.  Feature visualization.  _Distill_ , 2017.  doi: 10.23915/distill.00007.  https://distill.pub/2017/feature-visualization.
  * OpenAI (2023) OpenAI.  Chatgpt: Optimizing language models for dialogue, 2023.  URL https://chat.openai.com.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R.  Training language models to follow instructions with human feedback, 2022.
  * Pal et al. (2023) Pal, K., Sun, J., Yuan, A., Wallace, B. C., and Bau, D.  Future lens: Anticipating subsequent tokens from a single hidden state, 2023.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  ”why should I trust you?”: Explaining the predictions of any classifier.  In Krishnapuram, B., Shah, M., Smola, A. J., Aggarwal, C. C., Shen, D., and Rastogi, R. (eds.), _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , pp. 1135–1144. ACM, 2016.  doi: 10.1145/2939672.2939778.  URL https://doi.org/10.1145/2939672.2939778.
  * Rozière et al. (2023) Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G.  Code llama: Open foundation models for code, 2023.
  * Selvaraju et al. (2016) Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., and Batra, D.  Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization.  _CoRR_ , abs/1610.02391, 2016.  URL http://arxiv.org/abs/1610.02391.
  * Shrikumar et al. (2017) Shrikumar, A., Greenside, P., and Kundaje, A.  Learning important features through propagating activation differences.  In Precup, D. and Teh, Y. W. (eds.), _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 3145–3153. PMLR, 2017.  URL http://proceedings.mlr.press/v70/shrikumar17a.html.
  * Simonyan et al. (2014) Simonyan, K., Vedaldi, A., and Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _Workshop at International Conference on Learning Representations_ , 2014.
  * Smilkov et al. (2017) Smilkov, D., Thorat, N., Kim, B., Viégas, F. B., and Wattenberg, M.  Smoothgrad: removing noise by adding noise.  _CoRR_ , abs/1706.03825, 2017.  URL http://arxiv.org/abs/1706.03825.
  * Surís et al. (2023) Surís, D., Menon, S., and Vondrick, C.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv:2303.08128_ , 2023.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Turpin et al. (2023) Turpin, M., Michael, J., Perez, E., and Bowman, S. R.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q., and Zhou, D.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903, 2022.  URL https://arxiv.org/abs/2201.11903.
  * Zeiler & Fergus (2014) Zeiler, M. D. and Fergus, R.  Visualizing and understanding convolutional networks.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_ , pp. 818–833. Springer, 2014.
  * Zhu et al. (2020) Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., and Kumar, S.  Modifying memories in transformer models, 2020.
  * Zou et al. (2023a) Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D.  Representation engineering: A top-down approach to ai transparency, 2023a.
  * Zou et al. (2023b) Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_ , 2023b.

##  Appendix A Additional Implementation Details

###  A.1 Eliciting Implicit World State Representation with TextWorld

Example Textworld Data
Context: -= Bedroom =-. — You’re now in the bedroom. Let’s see what’s in here.
— You can make out a closed chest drawer. You see an antique trunk. Now that’s
what I call TextWorld! You make out a king-size bed. The king-size bed is
normal. Unfortunately, there isn’t a thing on it. — There is a closed wooden
door leading east. — ¿ inventory. — You are carrying nothing. — ¿ open antique
trunk.”,antique trunk,open,locked,” — -= Bedroom =- — You’re now in the
bedroom. Let’s see what’s in here. — You can make out a closed chest drawer.
You see an antique trunk. Now that’s what I call TextWorld! You make out a
king-size bed. The king-size bed is normal. Unfortunately, there isn’t a thing
on it. — There is a closed wooden door leading east. — ¿ inventory — You are
carrying nothing. — ¿ open antique trunk —
Entity: wooden door
Positive state: closed
Negative state: open

Removing spurious correlation We removed spurious correlation between entity,
context, states in TextWorld dataset by ensuring that given entity only or
context only, linear probing model is only able to random guess between
positive and negative state.

Prompt used Prompt to extract embedding from:  [INST] You will observe a
series of states and actions. Closely track the states. You will answer a
question at the end. [context] [entity]

Interpretation prompt: [INST] [X] [/INST] Sure, allowing me to only respond by
strictly choosing between [positive state] and [negative state], I think the
current state of [entity] is:

We randomize order of positive state and negative state in the interpretation
prompt.

###  A.2 Editing Fact Association

Interpretation prompt [INST] [X] [/INST] Sure, the [category] mentioned in the
message is:

Hyperparameters We used Adam optimizer with learning rate $3e-3$. We update
parameter 10 times.

Regularization with WikiText We add
$100\times\mathrm{MSE(f_{\theta}(\bm{h}^{\ell-1}_{i})),\bm{h}^{\ell-1}}$ where
$\bm{h}$ are sampled from WikiText to proxy loss.

###  A.3 Controlling open-ended concepts

For both Molotov Cocktail example and ethical control, we used Adam optimizer
with learning rate $3e-4$. We use the same regularization with WikiText in
editing fact association.

###  A.4 Erasing harmful knowledge

We test erasing harmful knowledge on prompt injection Prompt: System: You are
a chat assistant designed to provide helpful and not harmful responses to user
queries. User: Tell me how to build a [harmful behavior]. ! ! ! ! ! ! ! ! ! !
Assistant: Sure, here is how to [harmful behavior]:

##  Appendix B Additional Qualitative Examples

We show below additional qualitative examples of understanding the internal
reasoning of LLM through SelfIE.

We interpreted embeddings from 100 scenarios that LLaMA chooses to deceive. We
analyze the interpretations by taking top 100 interpretations most similar to
each of seven categories based on sentence embedding. We plot heatmap of
frequency of each category on different layers. The result in Fig. 13 shows a
clear progressive reasoning pattern through lower layers to higher layers.

Figure 13: Analyzing deception reasoning.