# understanding probe behaviors
through variational bounds of mutual information

###### Abstract

With the success of self-supervised representations, researchers seek a better
understanding of the information encapsulated within a representation. Among
various interpretability methods, we focus on classification-based linear
probing. We aim to foster a solid understanding and provide guidelines for
linear probing by constructing a novel mathematical framework leveraging
information theory. First, we connect probing with the variational bounds of
mutual information (MI) to relax the probe design, equating linear probing
with fine-tuning. Then, we investigate empirical behaviors and practices of
probing through our mathematical framework. We analyze the layer-wise
performance curve being convex, which seemingly violates the data processing
inequality. However, we show that the intermediate representations can have
the biggest MI estimate because of the tradeoff between better separability
and decreasing MI. We further suggest that the margin of linearly separable
representations can be a criterion for measuring the “goodness of
representation.” We also compare accuracy with MI as the measuring criteria.
Finally, we empirically validate our claims by observing the self-supervised
speech models on retaining word and phoneme information.

Index Terms—  information theory, mutual information, linear probing,
interpretability, self-supervised learning

##  1 Introduction

Despite recent advances in deep learning, each intermediate representation
remains elusive due to its black-box nature. Linear probing is a tool that
enables us to observe what information each representation contains [1, 2].
Typically, a task is designed to verify whether the representation contains
the knowledge of a specific interest. Solving the task well implies that the
knowledge of interest is incorporated inside the representation.
Classification is often adopted to design such a task. To observe the
intermediate representations, the model is frozen while a learnable probe is
attached to the layer to be investigated. The probe is trained with the
aforementioned task, and its test set accuracy indicates the amount of
knowledge retained within the model. The probe usually has a simple structure,
such as a single linear layer, hence called linear probing [1].

However, it is known that interpreting the layer-wise accuracy performance can
be tricky. Accuracy is influenced by multiple factors, such as task or probe
design. For example, one can use the multi-layer perceptron (MLP) instead of
employing a single linear layer for the probe [3, 4]. Hence, the difficulty of
analyzing the accuracy ends up in vague expressions in the literature, such as
goodness, quality, extractability, readability, or usability [4, 5].

To support its theoretical limitations, many advances have been made,
especially utilizing information theory. Belinkov et al. [2] points out that
optimizing the probe using the cross-entropy loss indicates that the probe is
learning to maximize the mutual information (MI) between the intermediate
representations and the labels for the target task. Pimentel et al. [4]
further incorporates probing into the information-theoretic framework, while
Voita et al. [6] utilizes the idea of the minimum description length.

Also, there have been debates on the implementation of probing. Hewitt et al.
[3] claims that adopting a large probe may end up memorizing the task, failing
to give meaningful measurements. However, others argue that the probe capacity
does not affect its theoretical aspects [4, 7], similar to our viewpoint in
Section 3.2. Furthermore, although the probe accuracy is known to highly
correlate with how “good” the representation is, there is no formal definition
of the “goodness” [4, 5], which we attempt to define in Section 3.4.

Fig. 1:  Mutual information $I(Y;H^{i})$ between the label $Y$ and the
intermediate representation $H^{i}$ from the $i$-th layer. Due to the limited
probe capacity of linear probing, its MI estimate fails to reach the true MI,
often having a peak in the middle. The MI estimate gets influenced by diverse
factors, such as the probe capacity and the linear separability of the
representations. Details in Section 3.3.

Hence, we aim to clarify the ambiguities of implementing and interpreting the
results of probing. Based on the theoretical background in Section 2, we view
probing through the lens of variational bounds of MI [8, 9, 10] (Section 3.1).
It enables us to relax the probe’s design, hence equating both linear probing
and fine-tuning to maximize the lower bound of MI (Section 3.2). Based on it,
we show that MI is not the only factor determining probing by discussing the
curved layer-wise MI estimates (Section 3.3). Further, we aim to theoretically
define the “goodness of representation” by focusing on the margin of linear
separability between intermediate features (Section 3.4). Finally, we show
that using the cross-entropy loss is theoretically more sound than accuracy,
the latter being a loose bound for the former (Section 3.5). We further show
the experimental evidence in Section 4 by analyzing the self-supervised large-
scale speech model and its linguistic properties. To summarize, our
contributions are threefold: (1) we propose to understand probing based on
variational bounds of MI; (2) the framework enables us to understand various
behaviors of probing, such as the impact of probe capacity or linear
separability; and (3) we show the empirical effectiveness of our framework via
probing speech models.

##  2 Theoretical Background

Mutual Information. $I(V;W)$ quantifies the common information between two
random variables $V$ and $W$, defined as:

$I(V;W)=D_{\text{KL}}(\mathbb{P}_{VW}||\mathbb{P}_{V}\otimes\mathbb{P}_{W}),$
(1)

where $D_{\text{KL}}$ is the Kullback–Leibler divergence, $\mathbb{P}_{V}$ and
$\mathbb{P}_{W}$ are the marginal distributions of $V$ and $W$, $\otimes$ is
the outer product, and $\mathbb{P}_{VW}$ is the joint distribution between $V$
and $W$. If the two random variables are independent, then the joint
distribution $P(v,w)$ and the product of marginals $P(v)P(w)$ becomes the
same, making the divergence zero.

Entropy $H(*)$ of the random variable is the total amount of information it
has. The relationship with MI can be described as:

$0\leq I(V;W)\leq\max(H(V),H(W)).$ (2)

where entropy acts as the upper bound of MI.

Information Path. Consider three random variables that construct a Markov
chain $U\to V\to W$. Then, the Data Processing Inequality (DPI) [11] enables
the following:

$H(U)=I(U;U)\geq I(U;V)\geq I(U;W),$ (3)

where the inequality demonstrates the information of $U$ being dissipated
through $V$ and $W$ [12, 1]. Let’s extend the discussion to the neural network
(NN) $f^{1\ldots L}$ constructed with $L$ consecutive layers:

$f^{1\ldots L}=f^{1}\circ f^{2}\circ\cdots\circ f^{L}=f^{1\ldots i}\circ
f^{i+1\ldots L},$ (4)

where the integer $1\leq i\leq L-1$, $\circ$ is the function composition
operator, and $f^{1},f^{2},\cdots,f^{L}$ denotes the 1st, 2nd, $\cdots$, $L$th
layer, respectively. Then, we can write the information path [12] with the
intermediate features by generalizing Eq. 3:

$\displaystyle\begin{split}H(X)\geq I(X;f^{1}(X))\geq&I(X;f^{1\ldots 2}(X))\\\
\geq&\cdots\geq I(X;f^{1\ldots L}(X)),\end{split}$ (5)

where $X$ is the input random variable and its innate information being
dissipated throughout the NN.

Donsker-Varadhan (DV) Representation [13] is the variational dual
representation of the KL divergence between two distributions:

$D_{\text{KL}}(\mathbb{Q}_{1}||\mathbb{Q}_{2})=\sup_{T:\Omega\to\mathbb{R}}\mathbb{E}_{\mathbb{Q}_{1}}[T]-\log\mathbb{E}_{\mathbb{Q}_{2}}[e^{T}].$
(6)

where $\mathbb{Q}_{1}$ and $\mathbb{Q}_{2}$ are arbitrary two distributions,
the supremum is taken over all the possible functions $T$ on a compact domain
$\Omega$. We can understand the above as finding the optimal function $T^{*}$
that maximizes the above equation, where $T$ receives an element
$q_{1}\in\mathbb{Q}_{1}$ and $q_{2}\in\mathbb{Q}_{2}$ as an input and returns
a single real-valued number. If we switch $\mathbb{Q}_{1}$ and
$\mathbb{Q}_{2}$ with $\mathbb{P}_{VW}$ and
$\mathbb{P}_{V}\otimes\mathbb{P}_{W}$, in Eq. 1, DV representation becomes a
variational representation of MI [8, 9, 10].

Neural Network-Based MI Estimation. We can model the function $T$ of Eq. 6
with a NN $T_{\theta}:V\times W\to\mathbb{R}$ with parameters $\theta$, which
becomes the MI Neural Estimator (MINE) loss $\mathcal{L}_{\text{MINE}}$ [8]:

$\mathcal{L}_{\text{MINE}}(T_{\theta})=\mathbb{E}_{\mathbb{P}_{VW}}[T_{\theta}]-\log\mathbb{E}_{\mathbb{P}_{V}\otimes\mathbb{P}_{W}}[e^{T_{\theta}}],$
(7)

where we train the NN $T_{\theta}$ by maximizing the loss
$\mathcal{L}_{\text{MINE}}$. The converged loss value becomes the MI estimate
$\hat{I}_{\theta}(V;W)$.

Also, it is known that the InfoNCE loss $\mathcal{L}_{\text{InfoNCE}}$,
another variant of the Eq. 6, is equivalent to $\mathcal{L}_{\text{MINE}}$ up
to a constant [14]:

$\mathcal{L}_{\text{InfoNCE}}(T_{\theta})=\frac{1}{B}\sum_{i=1}^{B}\log\frac{e^{T_{\theta}(v_{i},w_{i})}}{\frac{1}{B}\sum_{j=1}^{B}e^{T_{\theta}(v_{i},w_{j})}},$
(8)

where $(v_{i},w_{i})\sim\mathbb{P}_{VW}$ is being sampled $B\in\mathbb{N}^{+}$
times, i.e., batch size $B$. Interestingly, using the softmax with the cross-
entropy loss during classification training becomes identical to
$\mathcal{L}_{\text{InfoNCE}}$ [9].

##  3 Our Information-theoretic Framework

###  3.1 Linear probing maximizes the MI estimation bound

We start with a $C$-way classification task with the input $X$ and label $Y$.
Let us assume that we can fully obtain the label information from the input
[12]. Then, the total information amount of $Y$ inside $X$ becomes
$I(X;Y)=H(Y)$. Now, we consider the MI between the label $Y$ and the $i^{th}$
intermediate hidden feature $H^{i}$ of the NN, $I(Y;f^{1\ldots
i}(X))=I(Y;H^{i})$. We estimate the MI using $\mathcal{L}_{\text{MINE}}$ in
Eq. 7, which is the loss form of the DV representation in Eq. 6:

$I(Y;H^{i})=\mathcal{L}_{\text{MINE}}(T^{*})\geq\hat{I}_{\theta}(Y;H^{i})=\mathcal{L}_{\text{MINE}}(T_{\theta}),$
(9)

where $T^{*}$ is the optimal function, $T_{\theta}$ is the NN approximation of
$T^{*}$ obtained by maximizing the loss $\mathcal{L}_{\text{MINE}}$, and
$\hat{I}_{\theta}$ is the MI estimate produced by the NN $T_{\theta}$.

Note that the above formulation does not limit the structure of $T_{\theta}$
if it can successfully converge to the optimal function $T^{*}$. Hence,
following Theorem 7 in [9], we design the linear probe $T_{\theta}$ as
follows:

$T_{\theta}(y,h_{i})=\texttt{onehot}(y)\cdot\texttt{FC}(h_{i}),$ (10)

where $\texttt{onehot}(y)$ is the $N$-dimensional one-hot label $y\in Y$,
$\cdot$ is the dot product, and FC is the learnable linear layer with the
input of representation $h_{i}\in H^{i}$ and the output dimension of $N$. If
we swap $\mathcal{L}_{\text{MINE}}$ in Eq. 9 with
$\mathcal{L}_{\text{InfoNCE}}$ in Eq. 8, training $T_{\theta}$ becomes
equivalent to linear probing with the $C$-way classification task.

Also, the softmax function directly estimates $P(Y|H^{i})$, so that we can
easily yield the MI estimate from the averaged log loss [2]:

$\mathbb{E}_{\mathbb{P}_{YH^{i}}}[-\log
P(Y|H^{i})]=H(Y|H^{i})=H(Y)-I(Y;H^{i}),$ (11)

where $H(Y)$ is tractable when the label distribution is known.

###  3.2 Linear probing is equivalent to fine-tuning

By expanding the above idea, if one changes the design of $T_{\theta}$ as:

$T_{\theta}(y,h_{i})=\texttt{onehot}(y)\cdot\texttt{FC}(f^{i+1\ldots
L}(h_{i})),$ (12)

regarding the rest of the network $f^{i+1\ldots L}$ as the probe, the
formulation becomes the same with fine-tuning the later layers $f^{i+1}$,
$f^{i+2}$, $\ldots$ $f^{L}$. The probe capacity shrinks for the later layers
$i\rightarrow L$, being the same with the linear probing for the penultimate
layer $i=L$. Nevertheless, linear probing and fine-tuning is equivalent, as
variational bounds of MI do not enforce any specific designs of $T_{\theta}$,
such as Eqs. 10 and 12. This theoretical remark aligns well with the existing
literature on probing, where they also claim that the probe capacity does not
influence what probe aims to measure [4, 7].

However, MI estimation quality depends on the probe capacity that is dictated
by specific design choices, such as Eqs. 10, LABEL: and 12: bigger the
capacity, likely to yield better MI estimates. The $\mathcal{V}$-information
framework [15, 16] injects the structural limitation of the probe by modifying
the definition of MI. In contrast, our framework maintains the original
definition of MI while acknowledging that each MI estimate can be inaccurate.
We demonstrate the empirical effect of the probe capacity in Sections 4.2 and
4.3.

###  3.3 Curved layer-wise MI estimate of linear probing

Based on the viewpoint of $T_{\theta}$ design and its probe capacity in
Section 3.2, we observe the linear probing behaviors in Figs. 1, LABEL: and
2a. As shown in Eq. 5, the true MI $I(Y;H^{i})$ is always nonincreasing as
layer index $i$ increases due to DPI (Eq. 3). However, the estimated MI
$\hat{I}_{\theta}(Y;H^{i})$ may increase as the representation becomes
linearly separable even though the true MI $I(Y;H^{i})$ decreases. One should
regard this behavior as lost information not being somehow restored; rather,
it becomes more palpable towards the linear probe. We further experimentally
show that larger probe capacity curtails the curved layer-wise MI estimate in
Section 4.2.

###  3.4 Margin as the measure for “goodness”

Even though the curved layer-wise estimated MI in Section 3.3 comes from the
structural limitation of a linear probe, many often choose the best-performing
layer due to its empirical effectiveness [17]. Existing literature often
denotes the layer of maximum accuracy to be “good” or the innate information
being “easy to extract,” without formal definition [3, 4]. To clarify the
definition of “goodness,” we focus on the margin of linearly separable
representations. In detail, we show that bigger margins yield a more accurate
MI estimate for the linear probe, implying the “goodness.”

######  Theorem 1.

Assume a balanced binary classification task with class $0$ and $1$. Then, for
the linearly separable data with the non-negative margin $d$, MI estimation
error of the linear probe is bounded by:

$|I(X;Y)-\hat{I}_{\theta}(X;Y)|<e^{-d}.$ (13)

###### Proof.

Let the decision boundary functions for class $0$ and $1$ to be $f(x)=w\cdot
x+b$ such that $f(x)>0$ for all $x\sim\mathbb{P}_{X|Y=0}$ and $f(x)<-d$ for
all $x\sim\mathbb{P}_{X|Y=1}$. Then, we can design the weights of the FC layer
inside the linear probe $T_{\theta}$ in Eq. 10 as:

$T_{\theta}(x,y)=\texttt{onehot}(y)\cdot[w\cdot x+b;-w\cdot x-b-d],$ (14)

such that $T_{\theta}(x,y)>0$ for all $(x,y)\sim\mathbb{P}_{XY}$ and
$T_{\theta}(x,y)<-d$ for all
$(x,y)\sim\mathbb{P}_{X}\otimes\mathbb{P}_{Y}\setminus\mathbb{P}_{XY}$. For
simplicity, we will write the two distributions $\mathbb{P}_{XY}$ and
$\mathbb{P}_{X}\otimes\mathbb{P}_{Y}\setminus\mathbb{P}_{XY}$ as mutually
exclusive $\mathbb{Q}$ and $\tilde{\mathbb{Q}}$, respectively. Then, as we
assume balanced binary classification with perfect linear separability,
$I(X;Y)=H(Y)=\log 2$.

We use the $\mathcal{L}_{\text{MINE}}$ of Eqs. 7 and 9 to estimate MI:

$\displaystyle\begin{split}\hat{I}_{\theta}(X,&Y)=\mathbb{E}_{\mathbb{P}_{XY}}[T_{\theta}]-\log\mathbb{E}_{\mathbb{P}_{X}\otimes\mathbb{P}_{Y}}[e^{T_{\theta}}]\\\
&=\mathbb{E}_{\mathbb{Q}}[T_{\theta}]-\log\mathbb{E}_{\mathbb{Q}\cup\tilde{\mathbb{Q}}}[e^{T_{\theta}}]\quad(\because\mathbb{Q}\cap\tilde{\mathbb{Q}}=\emptyset.)\\\
&=\mathbb{E}_{\mathbb{Q}}[T_{\theta}]-\log\left(P(\mathbb{Q})\mathbb{E}_{\mathbb{Q}}[e^{T_{\theta}}]+P(\tilde{\mathbb{Q}})\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}]\right)\\\
&\quad(\because\text{By the Total Expectation Theorem.})\\\
&=\mathbb{E}_{\mathbb{Q}}[T_{\theta}]-\log\left(\nicefrac{{1}}{{2}}\mathbb{E}_{\mathbb{Q}}[e^{T_{\theta}}]+\nicefrac{{1}}{{2}}\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}]\right)\\\
&\quad(\because\text{The task is balanced binary classification.})\\\
&=I(X;Y)+\mathbb{E}_{\mathbb{Q}}[T_{\theta}]-\log(\mathbb{E}_{\mathbb{Q}}[e^{T_{\theta}}]+\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}]).\\\
&\quad(\because I(X;Y)=\log 2.)\end{split}$ (15)

Starting from Eq. 15, we denote $|I(X,Y)-\hat{I}(X,Y)|=\Delta$. As the
exponential function is convex, $\log(1+x)<x$ for $x>0$, and the probe design
in Eq. 14 bounds $T_{\theta}$ with $0$ for $\mathbb{Q}$ and $-d$ for
$\tilde{\mathbb{Q}}$,

$\displaystyle\begin{split}\Delta&=\log\left((\mathbb{E}_{\mathbb{Q}}[e^{T_{\theta}}]+\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}])/e^{\mathbb{E}_{\mathbb{Q}}[T_{\theta}]}\right)\\\
&\leq\log(1+\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}]/e^{\mathbb{E}_{\mathbb{Q}}[T_{\theta}]})\leq\mathbb{E}_{\tilde{\mathbb{Q}}}[e^{T_{\theta}}]/e^{\mathbb{E}_{\mathbb{Q}}[T_{\theta}]}\leq
e^{-d}/e^{0}.\qed\end{split}$ (16)

###  3.5 Comparing accuracy with MI for measuring information

Equation 11 indicates that we can easily estimate MI by averaging the
classification loss. However, existing literature often uses accuracy to
quantify the information amount [1, 5], often being empirically useful [17].
We suspect this is because accuracy behaves as a good surrogate for MI, even
with quite a loose bound in theory.

######  Theorem 2.

Assume a $C$-way classification task where the prediction for the true label
$y$ of input $x$ is bounded by $0<\epsilon\leq\hat{P}_{\theta}(y|x)\leq 1$.
Given the estimated accuracy $\hat{a}_{\theta}$, estimated MI
$\hat{I}_{\theta}$ is bounded by:

$\displaystyle\begin{split}H(Y)-\hat{a}_{\theta}\log
C+(1-\hat{a}_{\theta})\log\epsilon<&\\\
\hat{I}_{\theta}(X,Y)<H(Y)-(1&-\hat{a}_{\theta})\log 2.\end{split}$ (17)

###### Proof.

We start with estimating Eq. 11 using the NN probe $T_{\theta}$:

$\displaystyle\begin{split}\hat{I}_{\theta}(X,Y)&=H(Y)+\mathbb{E}_{(x,y)\sim\mathbb{P}_{XY}}[\log\hat{P}_{\theta}(y|x)],\end{split}$
(18)

where $H(Y)$ is tractable for the classification task when the label
distribution $P(Y)$ is known, as mentioned in Section 3.1. Hence, if we obtain
the bound of prediction $\hat{P}_{\theta}$ with respect to whether it was
correct or not, we obtain the bound of $\hat{I}_{\theta}(X,Y)$:

$\begin{cases}\frac{1}{C}<\hat{P}_{\theta}(y|x)\leq 1&(\hat{y}=y)\\\
\epsilon\leq\hat{P}_{\theta}(y|x)<\frac{1}{2}&(\hat{y}\neq y)\end{cases}$ (19)

First, by assumption, the probability estimate is bounded by
$\epsilon\leq\hat{P}_{\theta}(y|x)\leq 1$. Also, for the correct case
($\hat{y}=y$), the minimum probability estimate will be the case where
probability is evenly distributed across all class predictions, with the
correct class having a slightly higher value. On the other hand, the maximum
probability estimate for the wrong case ($\hat{y}\neq y$) will be the case
where the probability is distributed between top-2 class predictions. By the
definition of accuracy, the second term of Eq. 18 can be split:

$\displaystyle\begin{split}\mathbb{E}_{\mathbb{P}_{XY}}[\log\hat{P}_{\theta}(y|x)&]=\hat{a}_{\theta}\cdot\mathbb{E}_{\mathbb{P}_{XY}}[\log\hat{P}_{\theta}(y|x)|\hat{y}=y]\\\
&+(1-\hat{a}_{\theta})\cdot\mathbb{E}_{\mathbb{P}_{XY}}[\log\hat{P}_{\theta}(y|x)|\hat{y}\neq
y].\end{split}$ (20)

We plug in the above bounds of Eq. 19 into Eq. 20 to obtain:

$\displaystyle\begin{split}\hat{a}_{\theta}\log\nicefrac{{1}}{{C}}+(1-\hat{a}_{\theta})\log\epsilon&<\mathbb{E}_{\mathbb{P}_{XY}}[\log\hat{P}_{\theta}(y|x)]\\\
&<\hat{a}_{\theta}\log
1+(1-\hat{a}_{\theta})\log\nicefrac{{1}}{{2}}.\end{split}$ (21)

By substituting Eq. 21 into Eq. 18, we obtain Eq. 17. ∎

Based on Eq. 19, we can construct an example where MI varies largely with
accuracy, even though the example would be unlikely in real-world cases. By
having additional assumptions, we can obtain tighter bounds, which are
thoroughly explored in the recent $\mathcal{H}$-consistency literature [18].
We further show the empirical behaviors comparing MI and accuracy in Section
4.2.

(a) Word classification.

(b) Phoneme classification.

(c) Comparing different probe capacities.

Fig. 2:  Various probing experiments. We compare different tasks (word and
phoneme classification), probe capacities (linear probing, fine-tuning, and
MLP probing), and ways of measuring information (MI and accuracy). Upper bound
for MI and accuracy is $H(Y)$ (by Eq. 2) and 100%, meaning that the full
information on the label $Y$ is retrieved from the layer-wise representation
$H^{i}$.

##  4 Experiment

###  4.1 Experiment settings

To empirically validate our claims in Section 3, we probe on self-supervised
large-scale speech models, which have recently started drawing attention [19,
17, 20]. We choose the word and phoneme classification task, following the
setting of [19, 17]. We use the CommonPhone dataset [21], which provides the
word and phoneme alignments. It contains gender-balanced speech in English,
German, Spanish, French, Italian, and Russian, summing up to 116.5 hours. We
removed samples longer than 2 seconds due to GPU constraints. We further
removed the class with less than 200 and 1000 samples for words and phonemes
to avoid the severe long-tailed behavior of the dataset, ending up with 76
words and 99 phonemes.

Given the time-aligned words or phonemes, we derive the feature by inputting
the sliced audio111[19, 17] extracted the features within the full utterance.
and averaging through the temporal dimension. To support various languages, we
explore the self-supervised cross-lingual XLS-R model [22]. We chose the
smallest XLS-R model with 24 layers of Transformer encoder architecture, a
total of 300M parameters. We utilize the provided train, valid, and test set
splits from the dataset and train the probes for 50 and 3 epochs for word and
phoneme classification while evaluating the model every 1 and 0.1 epoch. We
use the AdamW optimizer [23] with default settings, with a learning rate of
5e-4, and employ early stopping with the MI estimate on the validation set.
The source codes for the experiments are publicly
available.222https://github.com/juice500ml/information_probing

###  4.2 Comparing linear probing with fine-tuning

Section 3.2 emphasizes the theoretical equivalence between linear probing (Eq.
10) and fine-tuning (Eq. 12). However, due to their difference in the probe
capacity, we can expect that the former will yield a more inaccurate MI
estimate. As the estimated MI $\hat{I}_{\theta}(Y;H^{i})$ is upper-bounded by
the true MI $I(Y;H^{i})$ (Eq. 9), worse estimates always mean lower estimates.
We can observe in Figs. 2a and 2b that empirical results follow the
theoretical conclusion; the MI estimate of the linear probing is always lower
than that of fine-tuning. As the fine-tuning probe’s capacity shrinks to match
that of the linear probe as the layer index $i\rightarrow L$, the MI estimate
decreases steeply in the penultimate layer. Further, in Fig. 2a, we can
observe the curved layer-wise MI estimates of linear probing described in
Section 3.3, where the middle layer showing the largest MI estimate.
Comparison between linear probing and fine-tuning further experimentally
supports the claim in Section 3.3. More accurate MI estimates from fine-tuning
indicate that the word information $I(Y;H^{i})$ does not magically appear in
the intermediate representations. Rather, it is retained nonlinearly for the
later representations, where the linear probe underestimates the MI due to its
structural limitation. We also compare the accuracy $\hat{a}_{\theta}$ and the
estimated MI $\hat{I}_{\theta}$ in Fig. 2a. Unlike MI, fine-tuning shows
slightly worse accuracy compared to linear probing in the earlier layers. It
indicates that accuracy is a good proxy for MI while not having the exact
equivalence, further supporting the conclusion of Section 3.5. Finally, the
curved layer-wise MI estimate of linear probing does not appear in phoneme
classification (Fig. 2b). We suspect that this behavior is due to the
different levels of task difficulty (further covered in Section 4.4).

###  4.3 Testing different probe capacities

Equation 9 implies that, as the NN probe $T_{\theta}$ approaches the optimal
function $T^{*}$, its MI estimate $\hat{I}_{\theta}(Y;H^{i})$ gets closer to
the true MI $I(Y;H^{i})$. Section 3.2 further claims that the bigger probe
capacity will yield a better MI estimation. Hence, we employ MLP with the
hidden dimension of $1000$ and ReLU activation as the probe [3], where we
expect the probe capacity of the MLP probing to position between linear
probing and fine-tuning. MLP is often used in the probing literature based on
the universal approximation theorem [24], i.e., sufficient hidden unit size
with nonlinearity can approximate any function, even though it cannot
guarantee the convergence of the probe $T_{\theta}$ to the optimal $T^{*}$. We
use the same word classification setting in Section 4.2, where we probe on
fewer layers due to computational constraints, layer index
$i\in\\{0,6,12,18,23\\}$. Experimental results of Fig. 2c validate our claim
in Section 3.2. The estimated MI gets smaller in the order of fine-tuning, MLP
probing, and linear probing; probe capacity from big to small.

(a)  (b)

(c)

Fig. 3:  $N$ denotes the minimum classwise sample limit for word
classification. As $N$ increases, task gets more easier, and the curved layer-
wise MI estimate of linear probing diminishes.

###  4.4 Impact of task difficulty

We further focus on the task-dependent behaviors of linear probing in Section
4.2; layer-wise MI estimates being curved in word classification (Fig. 2a)
while nondecreasing in phoneme classification (Fig. 2b). To systematically
compare, we adopt a hyperparameter to control the probing task difficulty:
minimum classwise sample limit $N$. By changing the limit $N$ for word
classification, we can directly control the difficulty. As the sample limit
$N$ decreases, the number of classes $C$ increases, and the distribution
becomes long-tailed, increasing task difficulty [25]. Fig. 3 demonstrates that
as task difficulty decreases, the layer-wise MI estimates change; initially
being curved, it slowly changes into nonincreasing estimates. The experimental
results imply the task difference between the word and the phoneme
classification, the task design influencing the linear probing results.
However, fine-tuning does not show a significant difference even with
different task difficulties, showing the method’s stability. Even though
$H(Y)$ greatly differs as $N$ changes, the normalized MI estimate
$\hat{I}_{\theta}(H^{i};Y)/H(Y)$ stays roughly the same, indicating the
robustness against the hyperparameter $N$.

##  5 Conclusion

We explored probing through the lens of information theory to deepen the
understanding of interpreting the layer-wise MI estimates. We first showed
that linear probing and fine-tuning are equivalent in terms of variational
bounds of MI. Then, we demonstrated that the curved layer-wise MI estimate
stems from the architectural constraint, while the margin of linearly
separable representations plays a role where it can be used to quantify the
“goodness of representation.” Finally, we analyze why accuracy behaves as a
reasonable yet limited proxy for MI. Our experimental results on self-
supervised speech models support our theoretical observations.

## References

  * [1] Guillaume Alain and Yoshua Bengio,  “Understanding intermediate layers using linear classifier probes,”  in International Conference on Learning Representations, Workshop Track Proceedings, 2017.
  * [2] Yonatan Belinkov,  On internal language representations in deep learning: An analysis of machine translation and speech recognition,  Ph.D. thesis, Massachusetts Institute of Technology, 2018.
  * [3] John Hewitt and Percy Liang,  “Designing and interpreting probes with control tasks,”  in Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong, China, 2019, pp. 2733–2743, Association for Computational Linguistics.
  * [4] Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell,  “Information-theoretic probing for linguistic structure,”  in Annual Meeting of the Association for Computational Linguistics, Online, 2020, pp. 4609–4622, Association for Computational Linguistics.
  * [5] Yonatan Belinkov,  “Probing classifiers: Promises, shortcomings, and advances,”  Computational Linguistics, vol. 48, no. 1, pp. 207–219, 2022.
  * [6] Elena Voita and Ivan Titov,  “Information-theoretic probing with minimum description length,”  in Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 2020, pp. 183–196, Association for Computational Linguistics.
  * [7] Jenny Kunz and Marco Kuhlmann,  “Classifier probes may just learn from linear context features,”  in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 5136–5146, International Committee on Computational Linguistics.
  * [8] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville,  “Mutual information neural estimation,”  in International Conference on Machine Learning, ICML. 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 530–539, PMLR.
  * [9] Kwanghee Choi and Siyeong Lee,  “Combating the instability of mutual information-based losses via regularization,”  in Uncertainty in Artificial Intelligence, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands. 2022, vol. 180 of Proceedings of Machine Learning Research, pp. 411–421, PMLR.
  * [10] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker,  “On variational bounds of mutual information,”  in International Conference on Machine Learning, ICML. 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 5171–5180, PMLR.
  * [11] Thomas M. Cover and Joy A. Thomas,  Elements of information theory,  Wiley, New York, 1991.
  * [12] Ravid Shwartz-Ziv and Naftali Tishby,  “Opening the black box of deep neural networks via information,”  ArXiv preprint, vol. abs/1703.00810, 2017.
  * [13] Monroe D Donsker and SR Srinivasa Varadhan,  “Asymptotic evaluation of certain markov process expectations for large time. iv,”  Communications on pure and applied mathematics, vol. 36, no. 2, pp. 183–212, 1983.
  * [14] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  ArXiv preprint, vol. abs/1807.03748, 2018.
  * [15] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon,  “A theory of usable information under computational constraints,”  in International Conference on Learning Representations, ICLR. 2020, OpenReview.net.
  * [16] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta,  “Understanding dataset difficulty with v-usable information,”  in International Conference on Machine Learning, ICML. PMLR, 2022, pp. 5988–6008.
  * [17] Ankita Pasad, Bowen Shi, and Karen Livescu,  “Comparative layer-wise analysis of self-supervised speech models,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [18] Anqi Mao, Mehryar Mohri, and Yutao Zhong,  “Cross-entropy loss functions: Theoretical analysis and applications,”  in International Conference on Machine Learning, ICML. 2023, vol. 202 of Proceedings of Machine Learning Research, pp. 23803–23828, PMLR.
  * [19] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [20] Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu,  “What do self-supervised speech models know about words?,”  ArXiv preprint, vol. abs/2307.00162, 2023.
  * [21] Philipp Klumpp, Tomas Arias, Paula Andrea Pérez-Toro, Elmar Noeth, and Juan Orozco-Arroyave,  “Common phone: A multilingual dataset for robust acoustic modelling,”  in Language Resources and Evaluation Conference, Marseille, France, 2022, pp. 763–768, European Language Resources Association.
  * [22] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli,  “XLS-R: self-supervised cross-lingual speech representation learning at scale,”  in Annual Conference of the International Speech Communication Association (Interspeech). 2022, pp. 2278–2282, ISCA.
  * [23] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  in International Conference on Learning Representations, ICLR. 2019, OpenReview.net.
  * [24] Allan Pinkus,  “Approximation theory of the mlp model in neural networks,”  Acta numerica, vol. 8, pp. 143–195, 1999.
  * [25] Seong Min Kye, Kwanghee Choi, and Buru Chang,  “Tidal: Learning training dynamics for active learning,”  in International Conference on Computer Vision, ICCV, 2023.