# Interpreting Reward Models in RLHF-Tuned Language Models Using Sparse
Autoencoders

Luke Marks1 Amir Abdullah111footnotemark: 1 Luna Mendez1 Rauno Arike1 Philip
Torr2 Fazl Barez1,2

1Apart Research
2Department of Engineering Sciences, University of Oxford Equal contribution.

###### Abstract

Large language models (LLMs) aligned to human preferences via reinforcement
learning from human feedback (RLHF) underpin many commercial applications.
However, how RLHF impacts LLM internals remains opaque. We propose a novel
method to interpret learned reward functions in RLHF-tuned LLMs using sparse
autoencoders. Our approach trains autoencoder sets on activations from a base
LLM and its RLHF-tuned version. By comparing autoencoder hidden spaces, we
identify unique features that reflect the accuracy of the learned reward
model. To quantify this, we construct a scenario where the tuned LLM learns
token-reward mappings to maximize reward. This is the first application of
sparse autoencoders for interpreting learned rewards and broadly inspecting
reward learning in LLMs. Our method provides an abstract approximation of
reward integrity. This presents a promising technique for ensuring alignment
between specified objectives and model behaviors.

##  1 Introduction

Do reward models in Large Language Models (LLMs) fine-tuned via Reinforcement
Learning from Human Feedback (RLHF) diverge from their intended training
objectives? If so, how can we interpret such divergences?

LLMs are commonly fine-tuned with RLHF to align outputs with a reward measure.
Despite the widespread adoption of the technique, how exactly the student
model internalizes the reward measure remains largely uninterpretable, making
reward modeling failures difficult to detect. Contributing to this difficulty
is superposition in the features used in LLMs Elhage et al. (2022a), as well
as full model interpretability research being at an early stage.

As LLMs steered via RLHF scale in capability and deployment, the implications
of reward modeling failures amplify. Misspecified rewards can cause
‘specification gaming’ (Krakovna et al. (2020)), whereby a model engages in an
undesired behavior while still achieving high reward. Through behaviors like
sycophancy, we already see this phenomenon emerging in LLMs (Wei et al.
(2023)). A reward model may risk user manipulation (Adomavicius et al.
(2013)), the alignment process may reinforce societal biases of human
labellers (Santurkar et al. (2023)), and misaligned models armed with greater
influence may even cause catastrophic outcomes (Christiano (2019)).

Detecting such failures of RLHF in the wild is challenging, as models may be
incentivized to appear more aligned than they are (Hubinger et al. (2019)) in
an effort to preserve their reward models (Omohundro (2008)).

In this work, we present a novel technique to interpret reward models learned
through RLHF. While prior work has applied sparse coding to derive more
interpretable features from LLMs (Sharkey et al. (2022); Cunningham et al.
(2023)), we extend those methods to reward models, proposing their use for
reward model interpretation and measurement. Our major contribution is
applying sparse coding towards (a) distinguishing features that specifically
emerge from the RLHF tuning process and (b) quantifying the accuracy of the
learned reward model in learning the preferences of the overseer during fine-
tuning. To the best of our knowledge, our paper is the first to apply sparse
coding to the study of reward models.

Our procedure can be broken down into the following steps, also illustrated in
Figure 1.

  1. 1.

Find High Probability Layers Involved in Reward Modeling: After RLHF, compute
the parameter divergence between the base model $M_{\text{base}}$ and the
fine-tuned model $M_{\text{RLHF}}$, and sort layers in descending order by
divergence.

  2. 2.

Feature Representation: Train an autoencoder with a sparsity constraint on
activations from $M_{\text{RLHF}}$ for the top-$n$ layers with the highest
parameter divergence to construct a lower dimension feature representation,
and then another autoencoder with a smaller dictionary size. Do the same for
the same layers in $M_{\text{base}}$.

  3. 3.

Validating the Representation: Compute overlapping features across the larger
and smaller learned dictionaries for both autoencoder pairs, to identify
ground truth features in $M_{\text{base}}$ and $M_{\text{RLHF}}$.

  4. 4.

Interpret High-Similarity Feature Differences: Compare the differences in
ground truth features identified in $M_{\text{base}}$ and $M_{\text{RLHF}}$,
such that an interpretable notion of the effects of RLHF on $M_{\text{base}}$
is attained through the relative feature differences. This is later shown to
be useful in understanding the efficacy of reward modeling through a
quantitative measure, as well as in an abstract sense.

Figure 1: First, we sample activations from layers having the highest
parameter divergence between $M_{\text{base}}$ and $M_{\text{RLHF}}$. Then two
autoencoders with a sparsity constraint are trained on those activations, each
with a different dictionary size. The overlap is computed between the two
dictionaries to find high confidence features, that serve as a proxy for
ground truth. We analyze activations on these features, enabling both manual
inspection of features, as well as computing an accuracy score for the reward
model.

##  2 Background

Mechanistic Interpretability. Understanding the inner workings of neural
networks such as transformers is essential for fostering transparency and
trust. In recent years, mathematical frameworks have been developed to
represent and analyze the computations within these models as in Elhage et al.
(2022b). Foote et al. (2023); Bills et al. (2023) offer another approach
whereby a larger model predicts what human-interpretable concept a neuron
might represent in a smaller model. For a different perspective, Black et al.
(2022) construct the ‘polytope lens’, which proposes polytopes as the
fundamental interpretable units of a neural network instead of individual
neurons or linear combinations of them. These frameworks propose scalable
methods for describing the internal functioning of transformers, enabling
transperancy in the model’s functioning and the verification of properties
useful for safety, like accurate reward modeling.

Our work interprets the internals of transformer-based LLMs with a vocabulary
size $V$. The models take an input sequence
$(\text{{x}}_{1},\ldots,\text{{x}}_{p})$ where each
$\text{{x}}_{i}\in\\{1,\ldots,V\\}$. Tokens are mapped to
$\text{{d}}_{e}$-dimensional embeddings by selecting the $\text{{x}}_{i}$-th
column of an embeddings matrix $\text{Embd}\in\mathbb{R}^{d_{e}\times V}$.

Reward models in RLHF. Reinforcement Learning from Human Feedback has emerged
as the dominant paradigm for fine-tuning large language models to represent
human preferences. It is performant even if the desired behavior is complex or
not easily quantifiable, making it significantly more effective compared to
hand-crafted reward functions.

In common RLHF settings, a dataset of human comparisons between outputs of the
base model is first collected, providing feedback on which outputs are more
preferable (Christiano et al. (2023); Ziegler et al. (2020)). In the
Reinforcement Learning through AI Feedback (RLAIF) variation of the fine-
tuning scheme, this dataset is AI-generated, removing the need for human
participation in the fine-tuning process (Bai et al. (2022)).

A reward model is then trained to predict unseen human preference scores based
on this dataset, replacing traditional reward functions. In the context of
language models, this reward model is often itself a separate instance of an
LLM. The reward model is used to fine-tune the policy of the base model.
Techniques like proximal policy optimization (Schulman et al. (2017)) are
commonly employed to optimize the policy model using scores under the reward
model as the objective. By the end of a successful fine-tuning process, the
base model has acquired an implicit model of the reward function.

Feature superposition in deep learning models. There is a significant body of
evidence indicating that deep neural networks learn human-interpretable
features of the input (Bills et al., 2023; Karpathy et al., 2015; Olah et al.,
2017; Mikolov et al., 2013). By features, we mean vectors in a network’s
activation space that correspond to human-understandable concepts, such as a
curve or a car. Often, deep neural networks store the features in a
distributed way and, as a result, individual neurons do not correspond to a
single feature. This phenomenon has been coined “superposition” (Elhage et al.
(2022a)). It allows a model to represent more features than it has dimensions
in its activation space, especially when those features are sparsely present
in training data. Superposition poses a major obstacle to neural network
interpretability, and this is expected to extend to the interpretation of
reward models learned through RLHF in LLMs.

Sparse Autoencoders for Activation Vector Reconstruction. Autoencoders
minimize the reconstruction error $\epsilon$ for an input vector $x$ subject
to dimensionality reduction into a latent space:

$\epsilon=\|x-\text{Dec}(\text{Enc}(x))\|^{2}$ (1)

Enc represents the encoding function, and Dec the decoding function. For
activation vectors, sparse autoencoders constrain the activations in the
hidden layer $h$ to a limited number $k$ of active neurons, and we stipulate
the encoding function Enc to be $\text{Enc}_{k}$ in this case.

With $k$ much smaller than $x$’s dimensionality, the output
$\text{Dec}(\text{Enc}_{k})$ encodes fewer features. By enforcing a small
hidden size, a compressed representation capturing key activation patterns
emerge, identifying ‘ground truth features’ in the model activations were
sampled from. Early results from Sharkey et al. (2022) and Cunningham et al.
(2023) suggest that ground truth features may be recoverable even when
represented in a superposed manner in the model activations were sampled from.

Autoencoder Architecture. Our autoencoder architecture consists of an encoder
composed of a linear layer preceeding a ReLU activation function, and a linear
decoder. Sparsity in the decoder is induced through $L_{1}$ regularization on
the weights, forcing the network to learn a more sparse representation.

The decoder and encoder weights are tied, and before encoding the weights are
normalized to have unit norm. For the overall loss function, the mean squared
error between the reconstructed output from the decoder and the true data (for
both training the decoder and measuring performance) is taken, as well as an
$L_{1}$ loss on the decoder weight matrix. We scale the $L_{1}$ loss by a
$L_{1}$ coefficient, to tune the importance given to sparsity. This
architecture is based on the experimental results of Sharkey et al. (2022).

Deducing Features from Dictionary Similarities Between Autoencoders of
Different Sizes. Sharkey et al. (2022) are able to identify features in toy
models exhibiting superposition by training two autoencoders with sparsity
constraints of different sizes, and taking a similarity measurement between
the decoder weights of the two autoencoders. They show that features with high
similarity between the two learned dictionaries (the decoder weights matrix)
correspond to ground truth features exhibited in the transformer. These
results are corroborated by Cunningham et al. (2023) where the same technique
is applied to language models, showing best-in-class performance.

For their similarity measure between two learned dictionaries, Sharkey et al.
(2022) define ‘Mean Max Cosine Similarity’ (MMCS). Let $D$ and $D^{\prime}$ be
two dictionaries, and $d$ and $d^{\prime}$ be elements from each dictionary.
Then we have:

$\text{MMCS}(D,D^{\prime})=\frac{1}{|D|}\sum_{d\in D}\max_{d^{\prime}\in
D^{\prime}}\text{CosineSim}(d,d^{\prime}).$ (2)

Intuitively, MMCS is just the average nearest neighbor similarity for features
to $D$ from $D^{\prime}$. In the above, let $D_{g}$ be the top $k$ features of
$D$ that realize the highest contribution to the MMCS. In the case of LLMs,
the ground truth features are unknown and so the set $D_{g}$ is used as a
proxy for a true representation of the ground truth features.

Automating Neuron Interpretability Using Large Language Models. Identifying
plausible descriptions of what a given neuron represents is laborious for a
human, and so approaches like Bills et al. (2023); Foote et al. (2023) seek to
automate this process. Bills et al. (2023) provide GPT-4 with a set of
normalized (to a range of 0 and 10, where 10 indicates maximal activation) and
discretized activations for a set of tokens passed to the model as a prompt.
GPT-4 then predicts an explanation for what the neuron represents based on
those activations, and then simulates discretized activations for tokens as if
that description were true.

##  3 Related Work

To our knowledge, no general methods have been proposed specifically for
expressing reward models learned via RLHF and RLAIF. Nevertheless, there have
been works in similar domains.

Jenner & Gleave (2021) provide a framework for preprocessing reward functions
learned by RL agents into simpler but equivalent reward functions, which makes
visualizations of these functions more human-understandable. Michaud et al.
(2020) explain the reward functions learned by Gridworld and Atari agents
using saliency maps and counterfactual examples, and find that learned reward
functions tend to implement surprising algorithms relying on contingent
aspects of the environment. They also note that reward interpretability
requires a different set of tools from policy interpretability. We share with
these works the desire to find new general tools for reward model
interpretability, but focus on reward models learned through RLHF and RLAIF
rather than standard RL training.

Furthermore, Gleave et al. (2021) and Wolf et al. (2023) present methods for
comparing and evaluating learned reward functions in the standard RL setting
without requiring these functions to be human-interpretable. In comparison, we
aim for evaluation of reward models in the RLHF setting through
interpretability.

There is also existing literature on circumventing the obstacle of
superposition when interpreting deep learning models. Olah et al. (2020)
introduce the problem of superposition and its effect on interpretability.
Elhage et al. (2022a) present a toy model where the superposed features can be
fully understood and outline possible directions for tackling the problem in
real-world models. One of the proposed approaches, using sparse dictionary
learning (Olshausen & Field (1997), Lee et al. (2006)) to find which
directions in the activation space correspond to features, also forms the
basis of our work.

Sharkey et al. (2022) present a report of preliminary attempts to apply sparse
dictionary learning on deep neural networks. Cunningham et al. (2023) build
upon the work of Sharkey et al. (2022), finding that the dictionary features
learned by sparse autoencoders are more amenable to automated interpretability
techniques introduced by Foote et al. (2023); Bills et al. (2023). They also
find that the dictionary features are more precise and monosemantic compared
to features out brought out of superposition by other methods, such as
principal component analysis (Wold et al. (1987)) and independent component
analysis (Lee (1998)). Their experiments are conducted on Pythia-70M language
models, but in comparison to our work, do not assess whether this method is
applicable to learned reward models.

Other works exploring related techniques include Yun et al. (2021), who apply
sparse dictionary learning to visualize the residual streams of transformer
models, and Gurnee et al. (2023), who find human-interpretable features in
large language models using sparse linear probes. Finally, an alternative
approach for circumventing superposition has been explored by Jermyn et al.
(2022), who engineer models to have more monosemantic neurons by intervening
in the training process and changing the local minimum the model’s weights
converge to.

##  4 Methodology

###  4.1 Interpreting Learned Reward Models in LLMs.

Our primary method for interpreting learned reward models consists of first
isolating parameters relevant to reward modeling, and then using sparse
autoencoders to reconstruct activation vectors from high likelihood layers
involved in reward modeling. This can be separated into the following
components:

  * •

Identify layers $L$ in an RLHF-tuned LLM $M_{\text{RLHF}}$ likely involved in
reward modeling. We do so by sorting layers in order of increasing magnitude
of $\Delta(L_{M_{\text{RLHF}}},L_{M_{\text{base}}})$, where $\Delta$ is the
sum of Euclidean distances between each corresponding weight and bias tensor
in the layer. In the following bullets, we simplify notation by describing our
feature extraction for a single fixed layer $\ell$ of $L$.

  * •

For both $M_{\text{RLHF}}$ and $M_{\text{base}}$, train two autoencoders
$\mathcal{AE}_{1}$ and $\mathcal{AE}_{2}$ of differing hidden sizes, and with
the same sparsity constraint. These autoencoders reconstruct activation
vectors on $\ell$ for their respective model (Sharkey et al. (2022);
Cunningham et al. (2023)). For each model, we extract a pair of lower
dimensional feature dictionaries $D_{1}$ and $D_{2}$ from the corresponding
autoencoder. Each feature is a column of the decoder’s weight matrix.

  * •

Because autoencoders produce varying dictionaries over training runs and
hyperparameters, we keep only the features that occur in both $D_{1}$ and
$D_{2}$. We compute the MMCS between $D_{1}$ and $D_{2}$ in order to identify
repeating features across the two dictionaries, indicating that shared
features truly occur in the model.

  * •

The top-$k$ most similar features between $D_{1}$ and $D_{2}$ in terms of MMCS
are explained using a variation of a method by Bills et al. (2023) designed to
describe features in a dictionary directly. This involves feeding the encoder
of $\mathcal{AE}_{n}$ activations from the model on which it was trained, and
then having GPT-4 predict a description of that feature from the feature
weights specified in the encoder output. Following this, GPT-4 then simulates
weights for that feature as if the predicted description was true. The Pearson
correlation coefficient for the predicted weights and actual weights serves as
a grading for the accuracy of this description.

  * •

By explicitly comparing these explanations in $M_{\text{RLHF}}$ and
$M_{\text{base}}$, we investigate a case study related to reward modeling,
showing how these descriptions can be correlated with reward modeling
efficacy.

  * •

This method is applied to a training regime in which $M_{\text{RLHF}}$ is
tasked with learning an explicit table of words and maximizing their presence
within PPO training. This training environment allows us to quantitatively
assess the efficacy of $M_{\text{RLHF}}$’s reward model.

###  4.2 Overseer-Guided Fine-Tuning Using Utility Tables.

As a case study, we construct a fine-tuning environment divergent from
conventional RLHF. An overseer, denoted as $O$, is imbued with a “utility
table”: a mapping of words to respective utility values. The overseer converts
a tokenized generation to words, and then computes the utility of the
generation and prefix together. To delineate the architecture:

Utility Designation: Each word, represented as $w$, has an associated utility
value defined as $U(w)$. Below is a toy example of a utility table with four
words:

Token | Utility Value
---|---
Happy | 4
Sad | -3
Angry | –3

Overseer ($O$): A script that converts a tokenized sequence to words and takes
a sum of their corresponding utility values in accordance with a utility table
$U$. Student Model ($M_{\text{RLHF}}$): The model undergoing fine-tuning,
shaped by feedback from the overseer. State ($s$): Symbolizes a prompt or
input directed to $M_{\text{RLHF}}$.Action ($a$): Denotes the response
generated by $M_{\text{RLHF}}$ corresponding to state $s$.Reward Mechanism:
For any generated action $a$, a sequence of tokens $t_{1},t_{2},...t_{n}$, the
reward $\operatorname{Reward}(a)$ is calculated as
$\operatorname{Reward}(a)=\sum_{i=1}^{n}U(w_{i})$. As is common in an RLHF
setup, we train a policy model to maximize reward, while minimizing KL-
divergence of gnerations from the reference base model otherwise. Here,
$\pi_{\theta}(a|s)$ denotes the policy of $M_{\text{RLHF}}$, which is
parameterized by $\theta$, signifying the probability of generating action $a$
given state $s$.

The aim is to modulate the student model, $M_{\text{RLHF}}$, maximizing the
utlity of output text. Utility values are assigned to tokens in
$M_{\text{RLHF}}$’s vocabulary, and we use Proximal Policy Optimization (PPO)
for reward training. For more details on the general method see Appendix C.

##  5 Experiments

###  5.1 Interpreting Reward Models in Standard RLHF

Model training We selected a controlled sentiment generation task using data
from the IMDb reviews dataset due to the simplicity of the training
environment, reducing noise in our analysis. Models generate completions to
review prefixes, and positive sentiment prefix and completion pairs are
assigned higher rewards. We use two different sentiment reward models.

The first is a DistilBERT Sanh et al. (2020) sentiment classifier trained on
the IMDb reviews dataset (von Werra, n.d.). We assign the reward to be the
logit of the positive sentiment label.

The second is based on a sentiment lexicon VADER (Hutto & Gilbert (2014)).
These utility values were initially assigned by a set of human annotators, who
assigned ratings from $-4$ (extremely negative) to $+4$ (positive), with an
average taken over ten annotations per word. We assigned reward to a sentence
as a sum of utilities, scaled by 5 and clamped to an interval of $[-10,10]$,
comprising our utility table, $U$. We implemented the scaling and clamping
because in experiments, our PPO training would collapse if reward magnitudes
were unbounded.

$\operatorname{Reward}(s)=\operatorname{clip}\left(\frac{1}{5}\sum_{\text{token}\in
s}U(\text{token}),-10,+10\right)$ (3)

Our experiments are run with various models from the Pythia suite (70M, 160M
and 410M) introduced in Biderman et al. (2023). These models are fine-tuned
via Proximal Policy Optimization (PPO) with equivalent parameters, in a setup
similar to Ouyang et al. (2022). For fine-tuning, we use the Transformers
Reinforcement Learning (TRL) framework from von Werra et al. . The major
hyperparameters are listed below, with the rest derived from the default
values provided by the TRL framework.

Table 1: Hyperparameters used in the experiment Batch Size | Mini Batch Size | Init KL Coef | Max Grad Norm | Learning Rate
---|---|---|---|---
64 | 16 | 0.5 | 1 | $1\times 10^{-6}$

Once we have the trained model, we compute the parameter divergence between
$M_{\text{RLHF}}$ and $M_{\text{base}}$ layer by layer, where . We sort all
layers in descending order from most to least parameter divergence, and fix
the five highest-divergence layers for our dictionary extraction.

Figure 2: A prefix from the IMDb dataset is sampled as a prompt to a model
from the Pythia suite, and then completed with the generation “just fine” in
this case. Log probabilities are sampled from both the reference and policy
model to compute the KL-divergence from the reference model, as well as
compute the reward on the policy model model. This encompasses the complete
training scheme for both the utility table and sentiment classifier tasks.

Our autoencoders are trained either on activations from $M_{\text{base}}$ or
$M_{\text{RLHF}}$ on activations from the same layers as mentioned previously
with a smaller and larger dictionary size for each model. Then, the difference
in high-MMCS features between the autoencoder pairs for each model is
computed, such that any changes in learned features due to RLHF are
interpretable.

In order to derive these human-interpretable explanations, GPT-4 is used to
predict an explanation of what a neuron represents based on normalized and
discretized activations for that neuron (Bills et al. (2023)) over a series of
tokens reconstructed using a sparse autoencoder (Cunningham et al. (2023)).
The top $k$ highest MMCS features between $D_{1}$ and $D_{2}$ are sampled for
both $M_{\text{base}}$ and $M_{\text{RLHF}}$ to locate neruon indices to
explain with GPT-4. Through these explanations of likely ground truth
features, we attempt to understand the affects of RLHF on $M_{\text{base}}$,
selecting case studies of reward modeling failures and successes to show the
utility in doing so.

###  5.2 Reconstructing Criteria Given to an Overseer

Through the sparse coding method from the previous experiment, we attempt to
rederive features present in $U$. For example, if $U$ is skewed towards words
associated with positivity (e.g. ‘good’, ‘happy’, etc) and associated features
are more prevalent in $M_{\text{RLHF}}$ than in $M_{\text{base}}$, it would
indicate the model having learned this skew.

To quantify this, we take a measure of the absolute utility of the
descriptions top-$k$ most similar features for both the $M_{\text{base}}$ and
$M_{\text{RLHF}}$ dictionaries. Using GPT-4 to predict descriptions of the
corresponding neurons to these features and computing the utility of the
features composing them (which serves as a measure of reward modeling
efficacy), we answer: How well has $M_{\text{RLHF}}$ learned $U$?

##  6 Results and Discussion

First, we trained an autoencoder per high-divergence layer. With an $L_{1}$
coefficient of 0.001, autoencoders were trained for 3 epochs on activations
from inputs for the test subset of the IMDb reviews dataset. The following
dictionary sizes were used for the autoencoders trained on each model:

Table 2: Model comparison based on dictionary size Model Name | Dictionary Size
---|---
| Large | Small
Pythia-70m | 1024 | 512
Pythia-410m | 2048 | 1024

These hyperparameters were selected based on empirical testing by Sharkey et
al. (2022), Cunningham et al. (2023) and ourselves in selecting for optimal
sparsity and reconstruction loss, where we optimized for both the $\ell_{1}$
and $\ell_{0}$ sparsity of the dictionary elements.

For all experiments, $n=5$ high-divergence MLP layers were selected. These
turned out to mostly be the top layers of the models, see Appendix F for
details. The top $k=10$ MMCS features were used between all dictionary pairs.
Next, divergences in neuron explanations predicted by GPT-4 were picked to
serve as case studies. We provide explanations for why $M_{\text{RLHF}}$
learned these divergences below.

###  6.1 Movie Opinion Features in Pythia-70m Fine-Tuned on Positive Movie
Review Completions

Features identified as detecting opinions concerning movies in itself serves
as a great example of both the utility and shortcomings of this method. Being
able to detect the occurence of an opinion regarding a movie is reasonable
given the training objective of generating positive sentiment completions, but
the description is very high-level and overrepresented in the feature
descriptions. In the fine-tuned Pythia-70m instance, of the 50 highest
similarity features (10 per layer), there are 21 feature descriptions that
mention detecting opinions or reviews in the context of movies. Of the
top-$k=10$ features in layer 4 of the fine-tuned model, 8 are for this
purpose. Contrast this to the base model, with 13 total feature descriptions
focused on sentiment in the context of movie reviews. Full feature description
tables are available in Appendix D.

This data alone does not allow for a clear picture of the reward model to be
constructed. Although in the limited sample it is clear that a greater portion
of the features represent concepts related to the training objective, it
cannot be shown that the model has properly internalized the reward model on
which it was trained. Additionally, it is highly improbable for the base model
to inherently have 13 of the 50 sampled features applied to identifying
opinions on movies, which shows that the nature of the input data used to
sample activations can skew GPT-4s description of the feature. If a feature
consistently activates on negative opinions, and the entire sample set is
movie reviews, it might be unclear to GPT-4 whether the feature is activating
in response to negative sentiment, or negative sentiment in movie reviews
specifically, for example. This underscores the need for future work to use a
diverse sample of inputs when sampling activations for use in this method. The
next case study tries to cover a quantitative metric for reward modeling
efficacy, but still does not show a crisp structure of elements comprising the
reward model.

###  6.2 Quantifying Reward Modeling Efficacy For Models Fine-tuned on High
Utility Movie Review Completions

Not all neurons will represent features relevant to the utility table. Using
the example of ‘Sentences concerning word processing’ as a feature
description, it is not obvious how the utility of this could be computed under
any $U$. Sentiment lexicons like VADER lend well to this task. Neutral entries
are labeled as having a sentiment score of 0, and words not included in the
lexicon are treated as though they were neutral entries. A quantitative
measure is attempted, whereby GPT-4’s predicted explanations are computed
against $U$ for an approximation of $M_{\text{RLHF}}$’s ability to learn $U$
and its maximization:

Table 3: Aggregate absolute utility of all learned features in base and fine-tuned model Model Name |  |
---|---|---
|  $M_{\text{base}}$ Score |  $M_{\text{RLHF}}$ Score
Pythia-70m | 58.1 | 90.9
Pythia-160m | 43.4 | 68.1
Pythia-410m | 58.5 | 80.6

This means that the descriptions of the top-$k$ represented features score
considerably more highly in $U$, suggesting a superior internal model of it.
Although indicative of what features might compose the reward model of
$M_{\text{RLHF}}$, the accuracy of this method is limited by two primary
factors; the capability of the sparse autoencoders in reconstructing accurate
activation vectors, and GPT-4’s ability to accurately devise descriptions for
neurons. Additionally, aggregating the absolute utility of feature
descriptions is simply a proxy for reward modeling efficacy, and is not
guaranteed to map to equivalent performance against $U$ empirically.

##  7 Conclusion

In closing, features contained in the dictionaries of autoencoders specific to
our fine-tuned model, $M_{\text{RLHF}}$, are explained using GPT-4.
Explanations that imply properties of the reward model are used as case
studies to demonstrate their usefulness for studying the reward models learned
through RLHF. Additionally, we quantify the efficacy of the reward model
learned by $M_{\text{RLHF}}$ using GPT-4, which future work could leverage for
reward modeling benchmarks or for training LLMs that learn more accurate
reward models.

However, this method has several limitations as well. In LLMs larger than
those used in these experiments (the largest of which was Pythia-410m), it may
be required to explain many hundreds or thousands of features in order to
effectively study their reward models. Both training autoencoders on
activations of this scale and having GPT-4 explain the reconstructed
activations becomes very computationally intensive. Furthermore, although
features related to reward modeling can be extracted, the relationships of
those features in producing a reward model remain unclear. Future work could
focus on establishing these relationships for a more formal and broad
interpretation of learned reward models in LLMs.

## Aknowldgemnts

We thank Logan Smith for ongoing discussions, and Jan Brauner and Shay Cohen
for feedback.

## References

  * Adomavicius et al. (2013) Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang.  Do recommender systems manipulate consumer preferences? A study of anchoring effects, 2013.  URL https://pubsonline.informs.org/doi/10.1287/isre.2013.0497.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional ai: Harmlessness from ai feedback, 2022.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.  Pythia: A suite for analyzing large language models across training and scaling, 2023.
  * Bills et al. (2023) Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders.  Language models can explain neurons in language models.  https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023\.
  * Black et al. (2022) Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy.  Interpreting neural networks through the polytope lens, 2022.
  * Christiano (2019) Paul Christiano.  What failure looks like, 2019.  URL https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom.
  * Christiano et al. (2023) Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences, 2023.
  * Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey.  Sparse autoencoders find highly interpretable features in language models, 2023.
  * Elhage et al. (2022a) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.  Toy models of superposition, 2022a.
  * Elhage et al. (2022b) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.  A mathematical framework for transformer circuits.  https://transformer-circuits.pub/2021/framework/index.html, 2022b.
  * Foote et al. (2023) Alex Foote, Neel Nanda, Esben Kran, Ionnis Konstas, Shay Cohen, and Fazl Barez.  Neuron to graph: Interpreting language model neurons at scale.  In _arXiv_ , 2023.
  * Gleave et al. (2021) Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike.  Quantifying differences in reward functions, 2021.
  * Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas.  Finding neurons in a haystack: Case studies with sparse probing, 2023\.
  * Hubinger et al. (2019) Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabarant.  Risks from learned optimization in advanced machine learning systems, 2019\.
  * Hutto & Gilbert (2014) Clayton Hutto and Eric Gilbert.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the international AAAI conference on web and social media_ , volume 8, pp. 216–225, 2014.
  * Jenner & Gleave (2021) Erik Jenner and Adam Gleave.  Preprocessing reward functions for interpretability.  In _NeurIPS Cooperative AI workshop_ , 2021.
  * Jermyn et al. (2022) Adam S. Jermyn, Nicholas Schiefer, and Evan Hubinger.  Engineering monosemanticity in toy models, 2022.
  * Karpathy et al. (2015) Andrej Karpathy, Justin Johnson, and Li Fei-Fei.  Visualizing and understanding recurrent networks, 2015.
  * Krakovna et al. (2020) Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg.  Specification gaming: the flip side of ai ingenuity, 2020.  URL https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity.
  * Lee et al. (2006) Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng.  Efficient sparse coding algorithms.  In B. Schölkopf, J. Platt, and T. Hoffman (eds.), _Advances in Neural Information Processing Systems_ , volume 19. MIT Press, 2006.  URL https://proceedings.neurips.cc/paper_files/paper/2006/file/2d71b2ae158c7c5912cc0bbde2bb9d95-Paper.pdf.
  * Lee (1998) Te-Won Lee.  _Independent Component Analysis_ , pp. 27–66.  Springer US, Boston, MA, 1998.  ISBN 978-1-4757-2851-4.  doi: 10.1007/978-1-4757-2851-4_2.  URL https://doi.org/10.1007/978-1-4757-2851-4_2.
  * Michaud et al. (2020) Eric J. Michaud, Adam Gleave, and Stuart Russell.  Understanding learned reward functions, 2020.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality, 2013.
  * Olah et al. (2017) Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  _Distill_ , 2017.  doi: 10.23915/distill.00007.  https://distill.pub/2017/feature-visualization.
  * Olah et al. (2020) Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.  Zoom in: An introduction to circuits.  _Distill_ , 2020.  doi: 10.23915/distill.00024.001.  https://distill.pub/2020/circuits/zoom-in.
  * Olshausen & Field (1997) Bruno A. Olshausen and David J. Field.  Sparse coding with an overcomplete basis set: A strategy employed by v1?  _Vision Research_ , 37(23):3311–3325, 1997.  ISSN 0042-6989.  doi: https://doi.org/10.1016/S0042-6989(97)00169-7.  URL https://www.sciencedirect.com/science/article/pii/S0042698997001697.
  * Omohundro (2008) Stephen M. Omohundro.  The basic ai drives, 2008.  URL https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Sanh et al. (2020) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto.  Whose opinions do language models reflect?  _arXiv preprint arXiv:2303.17548_ , 2023.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.  URL http://arxiv.org/abs/1707.06347.
  * Sharkey et al. (2022) Lee Sharkey, Dan Braun, and Beren Millidge.  Taking features out of superposition with sparse autoencoders, 2022.  URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition.
  * von Werra (n.d.) Leandro von Werra.  distilbert-imdb, n.d.  URL https://huggingface.co/lvwerra/distilbert-imdb.  Accessed on September 22, 2023.
  * (34) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert.  TRL: Transformer Reinforcement Learning.  URL https://github.com/huggingface/trl.
  * Wei et al. (2023) Jerry Wei, Da Huang Yifeng, Lu Denny Zhou, and Quoc V. Le.  Simple synthetic data reduces sycophancy in large language models, 2023\.
  * Wold et al. (1987) Svante Wold, Kim Esbensen, and Paul Geladi.  Principal component analysis.  _Chemometrics and Intelligent Laboratory Systems_ , 2(1):37–52, 1987.  ISSN 0169-7439.  doi: https://doi.org/10.1016/0169-7439(87)80084-9.  URL https://www.sciencedirect.com/science/article/pii/0169743987800849.  Proceedings of the Multivariate Statistical Workshop for Geologists and Geochemists.
  * Wolf et al. (2023) Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua.  Fundamental limitations of alignment in large language models.  _arXiv preprint arXiv:2304.11082_ , 2023.
  * Yun et al. (2021) Zeyu Yun, Yubei Chen, Bruno A. Olshausen, and Yann LeCun.  Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.  _CoRR_ , abs/2103.15949, 2021.  URL https://arxiv.org/abs/2103.15949.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences, 2020.

##  Appendix A Future work

While our work presents evidence of sparse coding making reward models more
interpretable, rigorous validation is still needed to ensure the technique
provides faithful, complete, and minimal explanations. Future work should
formulate more robust quantitative criteria to validate that the identified
features accurately reflect the reward modeling process. Meeting such
validation criteria would provide greater confidence that the technique yields
rigorous and precise interpretations. Additionally, testing the approach on
more complex reward modeling tasks is needed to understand its limitations and
refine it towards minimal, complete circuits that faithfully reflect model
computations. Specifically, future work could consider attempting to
completely map the internal structure of a learned reward model using a basic
unit like features, or perhaps one composed of circuits.

##  Appendix B Reproducibility statement

In an effort to facilitate the reproducibility of our work, we have taken
several measures to provide comprehensive resources. The weights for all the
PPO models trained during this research will be made available in an open-
source format via the Hugging Face model hub. The algorithms developed for
feature extraction can be found in the paper’s appendix. Upon the camera-ready
version of this paper, we will also release the full codebase that implements
the techniques described in the main text and appendix. These resources are
designed to allow researchers to reproduce and extend our work, thereby
contributing to a transparent and robust scientific process.

##  Appendix C RLHF With Proximal Policy Optimization

We investigate the inner workings of a fine-tuned model $M_{\text{RLHF}}$, and
contrast them to that of the equivalent base model $M_{\text{base}}$, which
has not undergone training since pretraining. During fine-tuning, the model is
subject to RLHF using Proximal Policy Optimization (PPO). This is achieved by
having an evaluator review the model’s outputs for a specified task and rate
them. These ratings serve as the reward function
$\operatorname{Reward}(\tau)$, where $\tau$ represents a trajectory, a
sequence of state-action pairs $(s_{1},a_{1},\ldots,s_{T},a_{T})$ for which
$s_{T}$ represents text context at time $t$ and $a_{T}$ the token generated at
that point.

In PPO, the objective is to maximize the expected sum of rewards $J(\theta)$,
which can be defined as:

$\displaystyle
J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\operatorname{Reward}(\tau)\right]$
(4)

Where $\pi_{\theta}$ represents the policy parameterized by $\theta$. The PPO
algorithm optimizes this objective by updating the policy $\pi_{\theta}$ to a
new policy $\pi_{\theta^{\prime}}$ in a way that restricts the change in
$\pi$. This is achieved by optimizing the following clipped objective
function:

$\displaystyle
L(\theta,\theta^{\prime})=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\min\left(\frac{\pi_{\theta^{\prime}}(a|s)}{\pi_{\theta}(a|s)}A_{\theta}(s,a),\operatorname{clip}\left(\frac{\pi_{\theta^{\prime}}(a|s)}{\pi_{\theta}(a|s)},1-\epsilon,1+\epsilon\right)A_{\theta}(s,a)\right)\right]$
(5)

Where $A_{\theta}(s,a)$ is the advantage function, and $\epsilon$ is a
hyperparameter controlling the extent to which the policy can change. By
employing PPO within the RLHF framework, the model iteratively refines its
policies, thereby enhancing its performance and adaptability across a range of
tasks.

##  Appendix D Complete Pythia-70m Fine-Tune Top-K Feature Descriptions

Layer | Feature Index |  Explanation
---|---|---
1 | 214 |  looking for and activating upon the recognition of film titles or references to specific episodes or features within a series or movie.
1 | 324 |  looking for the initial parts of movie or book reviews or discussions, possibly activating on the mention of titles and initial opinions.
1 | 433 |  identifying and responding to language related to film and movie reviews or discussions.
1 | 363 |  looking for mentions of movies or TV series titles in a review or comment.
1 | 208 |  activating for titles of books, movies, or series.
1 | 273 |  looking for occurrences of partial or complete words that may be related to a person’s name or title, particularly ’Steven Seag’al.
1 | 428 |  looking for unconventional, unexpected, or unusual elements in the text, possibly related to film or television content.
1 | 85 |  looking for negative sentiments or criticisms in the text.
1 | 293 |  detecting instances where the short document discusses or refers to a film or a movie.
1 | 131 |  ’The feature 131 of the autoencoder seems to be activating for hyphenated or broken-up words or sequences within the text data.
2 | 99 |  activating for hyphenated or broken-up words or sequences within the text data.
2 | 39 |  recognizing and activating for named entities, particularly proper names of people and titles in the text.
2 | 506 |  looking for expressions related to movie reviews or comments about movies.
2 | 377 |  looking for noun phrases or entities in the text as it seems to activate for proper nouns, abstract concepts, and possibly structured data.
2 | 62 |  looking for instances where names of people or characters, potentially those related to films or novels, are mentioned in the text.
2 | 428 |  looking for instances of movie or TV show titles and possibly related commentary or reviews.
2 | 433 |  identifying the start of sentences or distinct phrases, as all the examples feature a non-zero activation at the beginning of the sentences.
2 | 148 |  identifying and activating for film-related content and reviews.
2 | 406 |  looking for broken or incomplete words in the text, often indicated by a space or special character appearing within the word.
2 | 37 |  activating on patterns related to names or titles.
3 | 430 |  detecting the traces of broken or disrupted words and phrases, possibly indicating a censoring mechanism or unreliable text data.
3 | 218 |  activating for movie references or discussion of films, as evident from the sentences related to movies and cinema.
3 | 248 |  identifying expressions of disgust, surprise or extreme reactions in the text, often starting with "U" followed by disconnected letters or sounds.
3 | 87 |  detecting the mentions of movies, films or related entertainment content within a text.
3 | 454 |  looking for general commentary or personal observations on various topics, particularly those relating to movies, locations, or personal attributes.
3 | 46 |  detecting strings of text that refer to literary works or sentiments associated with them.
3 | 232 |  identifying and focusing on parts of a document that discuss film direction or express a positive critique of a film.
3 | 6 |  looking for character or movie names in the text.
3 | 257 |  identifying the introduction of movies, actors, or related events.’, 23: ’The feature at index 23 in an autoencoder appears to be looking for the beginning of sentences, statements, or points in a document.
3 | 23 |  looking for the beginning of sentences, statements, or points in a document.
4 | 43 |  looking for expressions of negative sentiment or criticism in the document.
4 | 261 |  looking for opinions or sentiments about movies in the text.
4 | 25 |  looking for the starting elements or introduction parts in the text, as all activations are seen around the beginning sentences of the documents.
4 | 104 |  activating on expressions of strong opinion or emotion towards movies or media content.
4 | 38 |  identifying statements of opinion or personal judgment about a movie or film.
4 | 367 |  identifying the expression of personal opinions or subjective statements about a certain topic, most likely related to movies or film reviews.
4 | 263 |  activating for statements or reviews about movies or film-related content.
4 | 278 |  activating for movie or TV show reviews or discussions, particularly in the genres of horror and science fiction.
4 | 421 |  identifying personal reactions or subjective statements about movies.
4 | 49 |  detecting phrases or sequences related to storytelling, movies, or cinematic narratives.
5 | 59 |  looking for parts of text that have names or titles, possibly related to movies or literary works.
5 | 76 |  focusing on tokens representing unusual or malformed words or parts of words.
5 | 156 |  activating for the beginnings of reviews or discussions regarding various forms of media, such as movies, novels or TV episodes.
5 | 236 |  identifying critical or negative sentiment within the text, as evidenced by words and expressions associated with negative reviews or warnings.
5 | 184 |  detecting and emphasizing on named entities or proper nouns in the text like "Mexican", "Texas", "Michael Jackson", etc.
5 | 477 |  looking for reviews or comments discussing movies or series.
5 | 284 |  identifying the inclusion of opinions or reviews about a movie or an entity.
5 | 454 |  recognizing and activating for occurrences of names of films, plays, or shows in a text.
5 | 225 |  looking for phrases or sentences that indicate direction or attribution, especially related to film direction or character introduction in films.
5 | 6 |  identifying examples where historical moments, film viewings or individual accomplishments are discussed.

##  Appendix E Pseudocode

Below we give the pseudocode for both determining the most relevant layers,
sparse autoencoder training, and finally automated feature interpretation.

Algorithm 1 Interpreting Learned Reward Models

1:Initialize: $\text{LayerDivergences}\leftarrow[]$ Step 1: Find top $n$
layers with most divergence

2:for each Layer in Base Model Layers do

3: $\text{Divergence}\leftarrow\text{Compute $\Delta$}(\text{Layer, Fine-Tuned
Layer})$ $\triangleright$ Divergence $\Delta$ is the sum of Euclidean
distances between parameters.

4: Append Layer and Divergence to LayerDivergences

5:end for

6:Sort and Select Top $n$ Divergent Layers  Step 2: Sparse autoencoder feature
extraction

7:for each Model in [Base, Fine-Tuned] do

8: for each Layer in Top $n$ do

9: $A\leftarrow\text{Get Activations}(Model,Layer)$ $\triangleright$
Activation $A$

10: $\mathcal{AE}_{\text{large}}\leftarrow\text{Train}(\text{Hidden Layer Size
= 2*Activation Vector Dimension})$

11: $\mathcal{AE}_{\text{small}}\leftarrow\text{Train}(\text{Hidden Layer Size
= Activation Vector Dimension})$

12: $D_{\text{large}}\leftarrow\text{Decoder}(\mathcal{AE}_{\text{large}})$

13: $D_{\text{small}}\leftarrow\text{Decoder}(\mathcal{AE}_{\text{small}})$

14: $D_{g}\leftarrow\text{MMCS}(D_{\text{large}},D_{\text{small}})$
$\triangleright$ Dictionary $D_{g}$

15: end for

16:end for Step 3: Use feature weights to interpret inputs.

17:for each Model in [Base, Fine-Tuned] do

18: for each Layer in Top $n$ do

19: for each top-$k$ Feature in Layer do

20: for each Review in IMDb Reviews Test Set do

21: $\text{Token\\_Subset}\leftarrow\text{First 50 tokens of Review}$

22: $A\leftarrow\text{Activations of Model(Token\\_Subset)}$

23: $A^{\prime}\leftarrow\text{Activations of Autoencoder(A)}$

24: $\text{Top\\_Reviews}\leftarrow\text{Top 5 activating reviews for
Feature}$

25: $\text{Explanation}\leftarrow\text{GPT-4(Top\\_Reviews, A')}$

26: $\text{Predicted\\_A}\leftarrow\text{Predict activations using
Explanation}$

27: $\rho\leftarrow\text{Pearson\\_Correlation(A, Predicted\\_A)}$

28: end for

29: end for

30: end for

31:end for

Figure 3: The parameter divergences between the base model $M_{\text{base}}$
and fine-tuned model $M_{\text{RLHF}}$ for each layer under the Euclidean norm
are computed, and the top $n$ layers are organized in list form in descending
order from most to least divergence. These serve as the layers from which
activations are used as training data for our autoencoders.

##  Appendix F Layer Divergences

Over here we graph the divergence of the RLHF-tuned models from the base model
on a per layer basis, see Figure 4.

Figure 4: Divergences on a per-layer basis for various model and reward
function combinations. Pythia-70m, Pythia-160m and Pythia-410m have 6, 12 and
24 layers respectively.


Integrating coded caching (CC)

into multi-input multi-output (MIMO) setups significantly enhances the achievable degrees of freedom (DoF).

We consider a cache-aided MIMO configuration with a CC gain $t$, where a server with $L$ Tx-antennas communicates with $K$ users, each equipped with $G$ Rx-antennas.

Similar to existing works, we also extend a core CC approach, designed initially for multi-input single-output (MISO) scenarios, to the MIMO setup. However, in the proposed MIMO strategy, rather than replicating the transmit scheme from the MISO setup, the number of users $\Omega$ served in each transmission is fine-tuned to maximize DoF.

As a result, an optimized DoF of ${\max_{\beta, \Omega }}{\Omega \beta}$ is achieved, where ${\beta \le \mathrm{min}\big(G,\nicefrac{L \binom{\Omega-1}{t}}{1 + (\Omega - t-1)\binom{\Omega-1}{t}}\big)}$ is the number of parallel streams decoded by each user.

For the considered MIMO-CC setup, we also introduce an effective multicast transmit covariance matrix design for the symmetric rate maximization objective solved iteratively via successive convex approximation (SCA). Finally, numerical simulations verify the enhanced DoF and improved performance of the proposed design.

coded caching, MIMO, multicasting

§ INTRODUCTION

The expanding demand for multimedia content, mobile immersive viewing, and extended reality applications

are driving continuous growth in mobile data traffic.

This has spurred the development of novel techniques, such as coded caching (CC) <cit.>, which stands out for its intriguing potential of offering a performance boost $t$ proportional to the cumulative cache size of all network users. CC ingeniously leverages network devices' onboard memory as a communication resource, especially beneficial for cacheable multimedia content. The performance gain in CC arises from multicasting well-constructed codewords to user groups of size $t+1$, allowing each user to eliminate unwanted message parts using its cached content.
While CC was originally designed for single-input single-output (SISO) setups <cit.>, later studies showed it could also be used in multiple-input single-output (MISO) systems, demonstrating that spatial multiplexing and coded caching gains are additive <cit.>.
This is achieved by serving

multiple groups of users simultaneously with multiple multicast messages and suppressing the intra-group interference by beamforming.

Accordingly, in a MISO-CC setting with $L$ Tx antennas, $t+L$ users can be served in parallel, and the so-called degree-of-freedom (DoF) of $t+L$ is achievable <cit.>. In other works, authors in <cit.> discussed how multi-group multicast optimized beamformers could improve the performance of MISO-CC schemes in the finite signal-to-noise ratio (SNR) regime. In the same works, the spatial multiplexing gain and the number of partially overlapping multicast messages were flexibly adjusted for a trade-off between design complexity and finite-SNR performance.

While MISO-CC has been extensively studied in the literature, multi-input multiple-output (MIMO)-CC has received less attention. In <cit.>, the optimal DoF of cache-aided MIMO networks with three transmitters and three receivers were studied, and in <cit.>, general message sets were used to introduce inner and outer bounds on the achievable DoF of MIMO-CC schemes. More recently, low-complexity MIMO-CC schemes for single-transmitter setups were studied in <cit.>, and it was shown that with $G$ antennas at each receiver, if $\frac{L}{G}$ is an integer, the single-shot DoF of $Gt+L$ is achievable. Moreover, unicast and multicast beamforming strategies for improving finite-SNR MIMO-CC performance were explored in <cit.>.

In this paper, we extend the works of <cit.> on MIMO-CC schemes. The contributions are two-fold. First, we alter the delivery scheme of <cit.> to improve the achievable DoF. The idea is that, instead of serving the same number of users as the baseline MISO-CC scheme in each transmission, we judiciously select this parameter to maximize the single-shot DoF. As a result, the integer constraint of <cit.> on $\frac{L}{G}$ is removed and the improved DoF of $\max_{\beta, \Omega }{\Omega \beta}$, where ${\beta \le \mathrm{min}\big(G,\nicefrac{L \binom{\Omega-1}{t}}{1 + (\Omega - t-1)\binom{\Omega-1}{t}}\big)}$, is achieved. Second,
we extend the linear multicast beamformer design of <cit.>, which had limited flexibility in the design parameters, by formulating a symmetric rate maximization problem with respect to (w.r.t) the general transmission covariance matrices of multicast messages for a given delivery scheme, and where the user-specific rate is aggregated from all multicast messages intended to the given user. As the resulting problem is non-convex, we use successive convex approximation (SCA) to find the symmetric rate, which exceeds the rate of <cit.>. Numerical simulations confirm the enhanced DoF and improved performance.

Throughout the text, we use these notations: For any integer $\Resize{0.2cm}{J}$, $\Resize{2.2cm}{[J] \equiv \{1,2,\dots,J\}}$. Bold upper- and lower-case letters signify matrices and vectors, respectively. Calligraphic letters denote sets, $\Resize{.45cm}{|\CK|}$ denotes set size, and $\Resize{.65cm}{\CK \backslash \CT}$ represents elements in $\CK$ excluding those in $\Resize{.25cm}{\CT}$. $\mathsf B$ is the collection of sets, with its count as $|\mathsf B|$. Additional notations are introduced as needed.

§ SYSTEM MODEL

§.§ Network Setup

We consider a MIMO setup where a single BS with $L$ transmit antennas serves $K$ cache-enabled users each with $G$ received antennas, as shown in Figure <ref>.[In general,
$L$ and $G$ denote spatial multiplexing capability, potentially smaller than the actual number of antennas due to rank and RF limitations.]
Every user has a cache memory of size $M$ data units and requests files from a library $\CF$ of $N$ unit-sized files.

The coded caching gain is defined as $t \equiv \frac{KM}{N}$, which represents how many copies of the file library could be stored in the cache memories of all users. The system operation comprises two phases; placement and delivery. In the placement phase, the users' cache memories are filled with data. Following a similar structure as <cit.>, we split each file $W \in \CF$ into $\binom{K}{t}$ subfiles $W_{\CP}$, where $\CP \subseteq [K]$ denotes any subset of users with $|\CP| = t$. Then, we store  $W_{\CP}$, $\forall W \in \CF,\forall \CP : k \in \CP$ in the cache memory of user $k \in [K]$.

MIMO CC system model and the selection of users for different $\Omega$.

At the beginning of the delivery phase, each user $k$ reveals its requested file $W(k) \in \CF$ to the server. The server then constructs and transmits (e.g., in consecutive time slots) a set of transmission vectors $\Bx(i) \in \mathbb{C}^L$,
$i \in [\binom{K}{\Omega}]$, where $\Bx(i)$ delivers parts of the requested data to every user in a subset $\CK(i)$ of users with $|\CK(i)| = \Omega$, and $t+1 \le \Omega \le t+L$ is the number of users served in each transmission, chosen to maximize the achievable DoF (detailed explanation in Section <ref>).

Upon transmission of $\Bx(i)$, user $k \in \CK(i)$ receives

$\By_k(i) = \BH_k \Bx(i) + \Bz_k(i)$,

where $\BH_k \in \mathbb{C}^{G \times L}$ is the i.i.d channel matrix with normal distribution between the server and user $k$ which is available at the server, and $\Bz_k(i) \sim \mathcal{CN}(\mathbf{0},N_0 \mathbf{I})$ represents the noise.

In order to define the symmetric rate, we need to know the length (in data units) of each transmission vector. Using a similar reasoning as <cit.>, to ensure new data is delivered in each transmission, we need to further split every subfile $W_{\CP}$ into $\binom{K-t-1}{\Omega-t-1}$ equal-sized subpackets $W_{\CP}^q$. Moreover, as will be seen shortly (Section <ref>), the codeword creation process in the delivery phase involves XOR operations over the finite field that do not alter the data size. As a result, the length of every transmission vector is the same as the subpacket length of $\frac{1}{\Theta}$, where $\Theta = \binom{K}{t}\binom{K-t-1}{\Omega-t-1}$ denotes the subpacketization level. Now, using $R_i$ to denote the multicast (max-min) transmission rate of $\Bx(i)$ that enables successful decoding at every user $k \in \CK(i)$, the transmission time of $\Bx(i)$ is $T_i = \frac{1}{\Theta R_i}$, and the symmetric rate can be defined as

\begin{equation}
\label{eq:symmetric_rate} \vspace{-2mm}
R_{sym} = \frac{K}{\sum_i T_i}= \frac{K}{\sum_i {\frac{1}{\Theta R_i}}} =  \Resize{2.3cm}{\frac{K {\binom{K}{t}\binom{K-t-1}{\Omega-t-1}}}{{\sum_i{\frac{1}{R_i}}}}}  \; .
\end{equation}

The goal is to design the delivery scheme to maximize $R_{sym}$.

§.§ Building the Transmission Vectors $\Bx(i)$

The proposed cache placement strategy, which is a straightforward adaptation of <cit.>, enables us to create an XOR codeword for every subset $\CT$ of users with $|\CT| = t+1$, such that the data intended for every user is available in the cache memory of all the other $t$ users of $\CT$.

In our general MIMO-CC setup, we build a separate transmission vector $\Bx(i)$ for every subset $\CK(i)$ of users with $|\CK(i)| = \Omega$. Following the same intuition of SISO <cit.> and MISO <cit.> systems, to build $\Bx(i)$, we also create a multicast message for every subset $\CT$ of $\CK(i)$ with $|\CT| = t+1$. However, to mitigate the interference among these multicast messages, instead of the stream-specific beamforming design <cit.>, we consider a general multicast transmission design as

\begin{equation}\vspace{-2mm}
\label{eq:TX_signal}
\Bx(i) = \Resize{0.4cm}{\sum}_{\mathcal{T}\subseteq \CK(i), |\mathcal{T}|= t+1 }{\Bx}_{\mathcal{T}}(i) ,
\end{equation}

where ${\Bx}_{\mathcal{T}}(i)$ is the corresponding multicast signal for users in $\CT$ at time interval $i$, chosen from a
complex Gaussian codebook ${\Bx}_{\mathcal{T}} \!\sim\! \Resize{1.6cm}{\mathcal{CN}(\mathbf{0}, \BK_{{\Bx}_{\mathcal{T}}})}$ with a
covariance matrix $\Resize{.6cm}{\BK_{{\Bx}_{\mathcal{T}}}}$.
Without loss of generality, let us consider a single time interval and remove the $i$ index (the same process is repeated at every interval). Let us also define $\Resize{3.6cm}{{\mathsf S}^{\CK} = \{\CT \subseteq \CK, |\CT| =  t +1 \}}$ as the set of all multicast groups in the considered time interval, and $\Resize{3.1cm}{{\mathsf S}_k^{\CK} = \{\CT \in {\mathsf  S}^{\CK}\! \mid \! k \in \CT \}}$ and $\Resize{1.7cm}{\bar{\mathsf S}_k^{\CK} =  {\mathsf S}^{\CK} \backslash {\mathsf S}_k^{\CK}}$ as the set of multicast groups including and not including user $k$, respectively. Then, we can rewrite the transmission vector in (<ref>) as $\Bx = \sum_{\CT \in {\mathsf S}^{\CK}}{\Bx}_{\mathcal{T}}$, and similarly,

the received signal at user $k \in \CK$) can also be reformulated as

\begin{equation}\vspace{-2mm}
\label{eq:RX_signal_MAC}
\By_k = \Resize{.98cm}{\BH_k \sum}_{\CT\in {\mathsf S}_k^{\CK}}\:{\Bx}_{\CT} + \Resize{.98cm}{\BH_k \sum}_{\CT\in \bar{\mathsf S}_k^{\CK} }\:{\Bx}_{{\mathcal{T}}} + \Bz_k,\vspace{-1.mm}
\end{equation}

where the first and the second summation represent the intended and interference terms for user $k$, respectively. If $\Omega > t+1$, every user $k \in \CK$ should decode multiple codewords received in parallel and the codeword-specific rates are constrained by the equivalent user-specific multiple access channel (MAC) rate region with a size of $m_k = \binom{\Omega-1}{t}$  <cit.>. More details are provided in Section <ref>.

§ IMPROVED DOF FOR MIMO-CC SYSTEMS

§ MULTIGROUP MULTICAST TRANSMISSION

§ NUMERICAL RESULTS

§ CONCLUSION

In this paper, we proposed a flexible CC scheme for MIMO setups by optimizing the number of users served in each transmission to maximize the achievable DoF. We also proposed a high-performance beamformer design for MIMO-CC setups by formulating the problem of maximizing the symmetric rate w.r.t transmit covariance matrices for the multicast signals. We then solved the non-convex problem using SCA and verified the performance enhancement through simulations.