# Fluctuation Theorems with Retrodiction rather than Reverse Processes

Clive Cenxin Aw  Centre for Quantum Technologies, National University of
Singapore, 3 Science Drive 2, Singapore 117543, Singapore  Francesco Buscemi
Graduate School of Informatics, Nagoya University, Chikusa-ku, 464-8601
Nagoya, Japan  Valerio Scarani  Centre for Quantum Technologies, National
University of Singapore, 3 Science Drive 2, Singapore 117543, Singapore
Department of Physics, National University of Singapore, 2 Science Drive 3,
Singapore 117542, Singapore

###### Abstract

Irreversibility is usually captured by a comparison between the process that
happens and a corresponding “reverse process”. In the last decades, this
comparison has been extensively studied through fluctuation relations. Here we
revisit fluctuation relations from the standpoint, suggested decades ago by
Watanabe, that the comparison should involve the prediction and the
retrodiction on the unique process, rather than two processes. We identify a
necessary and sufficient condition for a retrodictive reading of a fluctuation
relation. The retrodictive narrative also brings to the fore the possibility
of deriving fluctuation relations based on various statistical divergences,
and clarifies some of the traditional assumptions as arising from the choice
of a reference prior.

##  I Processes versus inferences

Quantitative approaches to irreversibility traditionally involve a comparison
between the process physically happening, usually called forward process, and
a corresponding reverse (or backward) process. The definition of the latter is
intuitive only for some processes: somewhat ironically, not for those that are
paradigmatic of irreversibility. Indeed, consider the erasure channel, which
sends every possible input state to a unique, fixed output state: what should
one take as its reverse process?

In a previous paper Buscemi and Scarani (2021), two of us proposed to look at
irreversibility as arising out of our logical inference, rather than out of
physical processes. Specifically, we proposed to define the reverse process in
terms of Bayesian retrodiction. This is a universal recipe. This retrodictive
element can be identified a posteriori in all previously reported fluctuation
relations that we checked, including the most famous ones, both classical
Bochkov and Kuzovlev (1977); Jarzynski (1997); Crooks (1998); Jarzynski (2000)
and quantum Tasaki (2000), that are highlighted in the many available reviews
Campisi, Hänggi, and Talkner (2011); Jarzynski (2011); Seifert (2012);
Gawedzki (2013). Besides recovering “intuitive” reverse processes,
retrodiction provides a definition for the non-intuitive ones, which smoothly
removes anomalies that were reported with other tentative definitions. Thus it
seems plausible that all fluctuation relations can be understood in terms of
retrodiction (though the literature is too vast and sparse to make a
definitive call, we shall strengthen the evidence with Result 3 below).

In the pursuit of this line of research, we recognize that our previous paper
was not radical enough. If the retrodictive origin of irreversibility is
assumed, the narrative of the two processes becomes superfluous: using
retrodiction to define a reverse process is an unnecessary step. There is only
one process, the one that happens; what is being compared are our forward and
backward inferences on it: prediction and retrodiction.

The replacement of irreversibility with irretrodictability was pioneered by
Watanabe Watanabe (1955, 1965), though prior to our previous work no
connection had been drawn with the fluctuation theorems derived in the last
twenty years. Under this change of viewpoint, it is the same physics that is
being described, freed from an excess baggage in the narrative (and thus,
possibly, on the interpretation). Besides epistemological economy, we are
going to show that this viewpoint is fruitful as it opens previously unnoticed
vistas.

The plan of the paper is as follows. In Section II we present a self-contained
introduction to retrodiction, both classical and quantum; and Section III
describes two case studies in detail. Section IV deals with fluctuation
relations: we show that these relations are intimately related with
statistical distances (“divergences”) and that Bayesian retrodiction arises
from the requirement that the fluctuating variable can be computed locally. We
also compare the fluctuation relations obtained in the retrodictive narrative
with those obtained in the reverse-process narrative. Section V reflects back
on the structure of retrodiction, elaborating on the role of the unavoidable
reference prior.

A word on the presentation. This paper covers topics from statistics,
thermodynamics, and quantum information. We have tried to keep the
presentation self-contained. We have also adopted a compact approach to
references: besides those that prove specific results, we shall cite mostly
reference books and reviews, and occasionally a few works that we consider
clear and exemplary, useful as entry points for the reader, without any
expectation of being exhaustive.

##  II Retrodiction: generalities

In this paper, we consider processes with discrete alphabets. The input of the
channel is denoted $x\in\\{1,2,...,d_{x}\\}$ and the output
$y\in\\{1,2,...,d_{y}\\}$. We shall always have in mind $d_{x}=d_{y}=d$,
keeping the notation different only when clarity demands it.

Also, throughout the paper, we assume that all probability vectors have
strictly positive elements; and also all channels have only strictly positive
entries (with the exception of the permutation channels studied in subsection
III.1). Arbitrarily small entries would be indistinguishable from an exact
zero, certainly in practice, and perhaps also in principle depending on one’s
understanding of probabilities.

###  II.1 Bayesian retrodiction on classical information

As basic setting of retrodiction, consider the most elementary form of
statistical inference: at the output of a known channel $\varphi(y|x)$, one
observes the outcome $y=y^{*}$, and wants to infer something about the input
$x$. In this paper, we focus on Bayesian retrodiction, whose goal is to update
one’s belief on the distribution of $x$. This requires a prior belief, the
reference prior, denoted $\xi(x)$. The total prior knowledge is therefore
captured by the joint probability distribution
$P_{\xi}(x,y)=\xi(x)\varphi(y|x)$; in particular, the prior knowledge about
$y$ is $\hat{\xi}(y)=\sum_{x}\xi(x)\varphi(y|x)$. When the knowledge on $y$ is
updated to $y=y^{*}$, one performs the Bayes’ update

$\displaystyle P_{\xi}(x,y)$ $\displaystyle\stackrel{{\scriptstyle
y=y^{*}}}{{\longrightarrow}}$ $\displaystyle
P_{\xi}^{\prime}(x,y)=P_{\xi}(x|y^{*})\delta_{y,y^{*}}$ (1)

on the total knowledge, whence the updated knowledge on $x$ follows as
$P_{\xi}(x|y^{*})=\xi(x)\varphi(y^{*}|x)/\hat{\xi}(y^{*})$. This is the most
elementary example of retrodiction.

Slightly less basic, though also widely discussed in the statistical
literature, is the retrodiction on $x$ based on “soft evidence” on $y$. This
refers to the situation, in which the update on $y$ is not a sharp value
$y=y^{*}$, but a distribution $u(y)$. In real life, soft evidence may arise by
sheer uncertainty (e.g. reading the outcome $y$ in very dim light) or by
virtual evidence (e.g. the doctor told me a definite result $z=z^{*}$ for my
test, but I saw that he was tired and fear that he may have misread the actual
result $y$ written on the sheet). The translation of such uncertainties into a
quantitative $u(y)$ is not trivial Chan and Darwiche (2005); Jacobs (2019),
but we take it for granted. For such situations, Bayes’ update (1) is
generalised to Jeffrey’s update Jeffrey (1965)

$\displaystyle P_{\xi}(x,y)$ $\displaystyle\,\stackrel{{\scriptstyle
u(y)}}{{\longrightarrow}}\,$ $\displaystyle
P_{\xi}^{\prime}(x,y)=P_{\xi}(x|y)\,u(y)\,.$ (2)

In the case of virtual evidence, Jeffrey’s update is a direct consequence
Pearl (1988); Jaynes (2003) of Bayes’ update starting from $z=z^{*}$, under
the assumption that the variable $z$ influences directly only $y$ and not $x$
(c.f. the example above: the tiredness of the doctor has no direct influence
on whether I am actually sick). In other cases, it may be considered as an
actual addition to the rules of Bayesian inference (this was Jeffrey’s own
view).

Thus, the conditional probability $P_{\xi}(x|y)$ plays the role of channel for
the retrodiction, in short retrodiction channel. For the remainder of the
paper, we change the notation to

$\displaystyle\hat{\varphi}_{\xi}(x|y)$ $\displaystyle=$
$\displaystyle\frac{\xi(x)}{\hat{\xi}(y)}\,\varphi(y|x)\,.$ (3)

We shall make use at our convenience of a matrix representation. The channel
$\varphi(y|x)$ is represented by the column-stochastic matrix

$\displaystyle
M^{\varphi}=\begin{pmatrix}\varphi(0|0)&&\cdots&&\varphi(0|d_{x})\\\
&\ddots&&&\\\ \vdots&&\varphi(y|x)&&\vdots\\\ &&&\ddots&\\\
\varphi(d_{y}|0)&&\cdots&&\varphi(d_{y}|d_{x})\\\ \end{pmatrix}$

Similarly, the retrodiction channel $\hat{\varphi}_{\xi}(x|y)$ is represented
by the column stochastic matrix

$\displaystyle
M^{\hat{\varphi}_{\xi}}=\begin{pmatrix}\hat{\varphi}_{\xi}(0|0)&&\cdots&&\hat{\varphi}_{\xi}(0|d_{y})\\\
&\ddots&&&\\\ \vdots&&\hat{\varphi}_{\xi}(x|y)&&\vdots\\\ &&&\ddots&\\\
\hat{\varphi}_{\xi}(d_{x}|0)&&\cdots&&\hat{\varphi}_{\xi}(d_{x}|d_{y})\\\
\end{pmatrix}\,.$

In this notation, we similarly define inputs and outputs distributions $p(x)$
as column vectors $v^{p}$. For instance, the relations that define the
reference prior can be written as

$\displaystyle M^{\varphi}v^{\xi}=v^{\hat{\xi}}$ $\displaystyle\;,\;$
$\displaystyle M^{\hat{\varphi}_{\xi}}v^{\hat{\xi}}=v^{\xi}\,.$ (4)

###  II.2 Two remarks

Before continuing, we bring up two crucial remarks. The first is about the
reference prior. It is well known Watanabe (1955, 1965); Jaynes (2003), and
our presentation above leaves it clear once again, that this element of
subjectivity is an unavoidable feature of Bayesian retrodiction for a generic
channel. The question of the choice of the prior is a recurring topic in
Bayesian statistics. The literature on fluctuation relations does not mention
it as such, the assumption being stated in more physical language. We shall
get back to this point in Section V. Here, for the sake of definiteness we
just mention two possible choices. One is the uniform prior
$\xi(x)=\frac{1}{d}$ for all $x$. Another is the steady state of $\varphi$,
defined by $\gamma=\hat{\gamma}$. Every stochastic map has at least one steady
state, and exactly one if all its entries are strictly positive. It follows
immediately from
$\hat{\varphi}_{\gamma}(x|y)=\frac{\gamma(x)}{\gamma(y)}\,\varphi(y|x)$ that
the uniform prior is a steady state if and only if $\varphi$ is bistochastic.

The second remark, that also others felt the need to highlight Barnett,
Jeffers, and Pegg (2021), is that retrodiction is not inversion. A channel has
a linear inverse if there exists $M$ such that $MM^{\varphi}=\mathbb{1}$. In
the case of an invertible channel, given a valid output distribution
$\hat{p}(y)=\sum_{x}\varphi(y|x)p(x)$, one is able to recover the input
distribution $p(x)$. But for most invertible channels, $M$ is not a channel
itself: there exist $u(y)$ such that $v^{u}\neq M^{\varphi}v^{p}$. In
particular, since the image of the probability simplex by $M^{\varphi}$ is
convex, there exists $y^{*}$ such that no input distribution $p(x)$ is mapped
by $M^{\varphi}$ to $\delta_{y,y^{*}}$ – while retrodicting from
$\delta_{y,y^{*}}$ is the most basic example of Bayesian inference. Ultimately
of course the difference is in the task: retrodiction does not aim at
reconstructing the prior through repeated sampling, but at updating one’s
belief after a single run of the process 111This is particularly clear in the
example of the test for a sickness given above. We can also refer the reader
to the studies of retrodiction of quantum temporal dynamics Gammelmark,
Julsgaard, and Mølmer (2013); Bao  _et al._ (2020)..

In subsection III.1, we shall see a remarkable coincidence: the channels for
which the retrodiction channel coincides with the inverse, and those for which
the retrodiction channel is independent of the reference prior, are exactly
the same.

###  II.3 Retrodiction on “quantum-inside” classical channels

According to our current knowledge, the most general description of the inner
working of any classical input-output channel is given by quantum theory. The
“quantum-inside” description of a classical channel is as follows (Fig.1). The
classical input $x$ prepares a system in a state $\rho_{x}$. The system is
then sent through a quantum channel (a completely positive, trace-preserving,
CPTP map) $\mathcal{E}$, and eventually measured with the positive operator-
valued measure (POVM) $\\{\Pi_{y}\\}$, leading to the classical outcome $y$.
All in all:

$\displaystyle\varphi(y|x)$ $\displaystyle=$
$\displaystyle\textrm{Tr}(\Pi_{y}\ \mathcal{E}[\rho_{x}])\,.$ (5)

We want to derive the quantum description of the associated classical
retrodiction channel (3): that is, finding states $\hat{\rho}_{y}$, a CPTP map
$\hat{\mathcal{E}}$, and a POVM $\\{\hat{\Pi}_{x}\\}_{x}$, such that

$\displaystyle\varphi_{\xi}(x|y)$ $\displaystyle=$
$\displaystyle\textrm{Tr}(\hat{\Pi}_{x}\
\hat{\mathcal{E}}[\hat{\rho}_{y}])\,.$ (6)

For this, we first need to define the adjoint $\mathcal{E}^{\dagger}$ of the
channel, that is the operator such that
$\textrm{Tr}(Y\mathcal{E}[X])=\textrm{Tr}(\mathcal{E}^{\dagger}[Y]X)$ for all
operators $X,Y$. Inserting this definition in (3), one finds

$\displaystyle\varphi_{\xi}(x|y)$ $\displaystyle=$
$\displaystyle\textrm{Tr}\left((\xi(x)\rho_{x})\
\mathcal{E}^{\dagger}[\Pi_{y}/\hat{\xi}(y)]\right)\,.$

This looks like (3), but in general one has
$\textrm{Tr}(\Pi_{y}/\hat{\xi}(y))\neq 1$,
$\sum_{x}\xi(x)\rho_{x}\neq\mathbb{1}$, and $\mathcal{E}^{\dagger}$ is a CPTP
map only if $\mathcal{E}$ is unital. In order to identify proper states,
channel and measurement, one has to introduce a reference state

$\displaystyle\Xi$ $\displaystyle=$ $\displaystyle\sum_{x}\xi(x)\rho_{x}\,.$
(7)

As in the classical case, we assume that $\Xi$ and
$\hat{\Xi}=\mathcal{E}[\Xi]$ have full rank, to skip caveats for situations of
measure zero. Then one possible construction of the quantum elements of (6)
uses

$\displaystyle\hat{\rho}_{y}$ $\displaystyle=$
$\displaystyle\frac{1}{\hat{\xi}(y)}\mathcal{S}(\mathcal{E}[\Xi])[\Pi_{y}]\,,$
(8) $\displaystyle\hat{\mathcal{E}}\equiv\hat{\mathcal{E}}_{\Xi}$
$\displaystyle=$
$\displaystyle\mathcal{S}(\Xi)\circ\mathcal{E}^{\dagger}\circ\mathcal{S}^{-1}(\mathcal{E}[\Xi])\,,$
(9) $\displaystyle\hat{\Pi}_{x}$ $\displaystyle=$
$\displaystyle\xi(x)\mathcal{S}^{-1}(\Xi)[\rho_{x}]\,,$ (10)

where we have introduced the notation
$\mathcal{S}(A)[B]=\sqrt{A}\,B\,\sqrt{A}$ for a positive operator $A$.
Starting from this basic constructions, one can obtain others:

$\displaystyle\hat{\rho}_{y}$ $\displaystyle\longrightarrow$
$\displaystyle\mathcal{U}_{s}[\hat{\rho}_{y}]\,,$ (11)
$\displaystyle\hat{\mathcal{E}}_{\Xi}$ $\displaystyle\longrightarrow$
$\displaystyle\mathcal{U}_{m}\circ\hat{\mathcal{E}}_{\Xi}\circ\mathcal{U}_{s}^{-1}\,,$
(12) $\displaystyle\hat{\Pi}_{x}$ $\displaystyle\longrightarrow$
$\displaystyle\hat{\Pi}_{x}\circ\mathcal{U}_{m}^{-1}$ (13)

also lead to (6) for any pair of unitary channels
$(\mathcal{U}_{s},\mathcal{U}_{m})$.

The key observation is that the retrodiction channel $\hat{\mathcal{E}}$ turns
out to be the Petz recovery or Petz transpose map Petz (1986, 1988) of
$\mathcal{E}$ for the reference state $\Xi$ [Eq. (9)], or a rotated version
thereof [Eq. (12)].

The Petz map, a widely used tool in quantum information Wilde (2013); Sutter,
Fawzi, and Renner (2016); Alhambra and Woods (2017), was previously identified
on formal grounds as the generalisation of retrodiction within the quantum
formalism Fuchs (2002); Leifer and Spekkens (2013); Crooks (2008); Fields,
Sajia, and Bergou (2020). First of all, in the case where all the states and
the channels are diagonal in the same basis, (9) reduces to (3). Furthermore,
just as the Bayesian retrodiction $\hat{\varphi}_{\xi}$ depends on a reference
prior $\xi$, the Petz map $\hat{\mathcal{E}}_{\alpha}$ depends on a reference
state $\alpha$ 222Petz discovered this map in the context of the monotonicity
property of the quantum relative entropy
$D(\rho||\alpha)=\Tr{\rho\ln\rho-\rho\ln\alpha}$ Umegaki (1961). For any CPTP
linear map $\mathcal{E}$, one has
$D(\rho||\alpha)-D(\mathcal{E}[\rho]||\mathcal{E}[\alpha])\geq 0$. Petz showed
that $D(\rho||\alpha)=D(\mathcal{E}[\rho]||\mathcal{E}[\alpha])$ if and only
if $\hat{\mathcal{E}}_{\alpha}\circ\mathcal{E}[\rho]=\rho$. In other words,
$\hat{\mathcal{E}}_{\alpha}$ reconstructs not only $\alpha$ after the channel,
but every state whose relative entropy with respect to $\alpha$ has not
changed.. Interestingly, the Petz map was also used for quantum fluctuation
relations Kwon and Kim (2019), but the connection with retrodiction was not
noticed.

Figure 1: A “quantum-inside” classical channel and the corresponding
retrodiction channel. The construction, described in Eqs. (7)-(10), is valid
for every set of states $\\{\rho_{x}\\}$, every CPTP map $\mathcal{E}$ and
every POVM $\\{\Pi_{y}\\}$.

##  III Retrodiction: Two case studies

In this Section, we present first retrodiction on Hamiltonian channels (both
classical and quantum), which are provably the only ones for which the
retrodictive map is independent of the reference prior and is identical to the
inverse. Then we discuss retrodiction for all classical bit channels ($d=2$):
precisely because it is elementary, this case study is useful to clarify
features and dissipate possible confusions about retrodiction.

###  III.1 Case study: Hamiltonian channels

We call Hamiltonian channels, both classical and quantum, channels that are
both deterministic and invertible (Watanabe Watanabe (1965) referred to these
channels as “bilaterally deterministic”). The flows do not cross, and each
state belongs to one and only one trajectory.

For classical information, we have $y=f(x)$ with $f$ a bijection (in the
discrete case, a permutation), and so $x=f^{-1}(y)$ is uniquely defined. In
this case, it is absolutely natural to expect

$\displaystyle\varphi(y|x)=\delta_{y,f(x)}$ $\displaystyle\,\Longrightarrow\,$
$\displaystyle\hat{\varphi}_{\xi}(x|y)=\delta_{x,f^{-1}(y)}$ (14)

independent of the reference prior. It is readily verified that this is indeed
the case from Eq. (3), since for a bijection we have
$\hat{\xi}(y)=\xi(x)\delta_{y,f(x)}$.

This result has very appealing features: the retrodiction channel coincides
with the inverse and is independent of the arbitrary choice of reference
prior. Appealing as they are, these features cannot be taken as paradigmatic,
because they are actually unique to this case.

######  Result 1.

The following three statements are equivalent:

  1. (I)

The channel $\varphi$ is a permutation.

  2. (II)

The retrodiction channel $\hat{\varphi}$ is independent of the reference
prior.

  3. (III)

There exists a reference prior $\xi$, for which the retrodiction channel
$\hat{\varphi}_{\xi}$ is the inverse channel $\varphi^{-1}$.

Proof. We present a full proof here, putting on record that the equivalence of
(I) and (II) was already proved in Watanabe’s pioneering study Watanabe
(1965).

Eq. (14) proves (I) $\to$ (II,III). The implication (II) $\to$ (III) goes as
follows: Eq. (4) implies $M^{\hat{\varphi}_{\xi}}M^{\varphi}v^{\xi}=v^{\xi}$.
If $M^{\hat{\varphi}_{\xi}}=M^{\hat{\varphi}}$ for all $\xi$, then
$M^{\hat{\varphi}}M^{\varphi}v^{\xi}=v^{\xi}$ for all vectors: then
$M^{\hat{\varphi}}M^{\varphi}=\mathbb{1}$, that is
$\hat{\varphi}=\varphi^{-1}$.

We are left to prove (III) $\to$ (I). Let’s assume that there is a reference
prior such that $\hat{\varphi}_{\xi}=\varphi^{-1}$ i.e.
$M^{\hat{\varphi}_{\xi}}M^{\varphi}=\mathbb{1}$. Let us spell out this
condition:

$\displaystyle\quad M^{\hat{\varphi}_{\xi}}M^{\varphi}=\mathbb{1}$
$\displaystyle\Leftrightarrow$
$\displaystyle\quad\sum_{y}M^{\hat{\varphi}_{\xi}}_{x^{\prime}y}M^{\varphi}_{yx}=\delta_{x^{\prime}x}\quad\forall_{x^{\prime}x}$
$\displaystyle\Leftrightarrow$
$\displaystyle\quad\sum_{y}\hat{\varphi}_{\xi}(x^{\prime}|y)\varphi(y|x)=\delta_{x^{\prime}x}\quad\forall_{x^{\prime}x}$
$\displaystyle\Leftrightarrow$
$\displaystyle\quad\sum_{y}\frac{\xi(x^{\prime})}{\hat{\xi}(y)}\varphi(y|x^{\prime})\varphi(y|x)=\delta_{x^{\prime}x}\quad\forall_{x^{\prime}x}\,.$

All the terms are products of non-negative numbers, $\xi(x)>0$ for all $x$ by
assumption, and $1/\hat{\xi}(y)\neq 0$ for $0\leq\hat{\xi}(y)\leq 1$. Thus,
for all the off-diagonal terms to be zero we need

$\displaystyle\varphi(y|x^{\prime})\varphi(y|x)=0\quad\forall_{x^{\prime}\neq
x,\,y}\,.$ (15)

This means the product of any two entries of a given $y$-row will always be
zero. Hence, there can be at most one non-zero entry for that row, which means
that the matrix $M^{\varphi}$ can have at most $d$ non-zero entries. But there
are $d$ columns, and the sum of all the elements of each column must be 1.
Thus, the only possibility is that each row and each column have exactly one
non-zero entry, and the value of the entry is 1. This defines a permutation
matrix and concludes the proof. ∎

Incidentally, condition (15) shows that
$M^{\hat{\varphi}_{\xi}}M^{\varphi}=\mathbb{1}$ is not determined by the
reference prior; so at that point we had proved directly (III) $\to$ (II).

The same result holds for retrodiction on quantum information – in fact,
Result 1 was presented first for reasons of clarity, but can be seen as a
special case of the following:

######  Result 2.

The following three statements are equivalent:

  1. (I)

The channel $\mathcal{E}$ is unitary.

  2. (II)

The retrodiction channel $\hat{\mathcal{E}}$ is independent of the reference
prior.

  3. (III)

There exists a reference state $\alpha$, for which the retrodiction channel
$\hat{\mathcal{E}}_{\alpha}$ is the inverse channel $\mathcal{E}^{-1}$.

Proof. The implications (I) $\to$ (II,III) follow from the direct calculation
of (9) for a unitary channel:

$\displaystyle\hat{\mathcal{U}}_{\alpha}$ $\displaystyle=$
$\displaystyle\mathcal{S}(\alpha)\circ\mathcal{U}^{\dagger}\circ\mathcal{S}^{-1}(\mathcal{U}[\alpha])$
$\displaystyle=$
$\displaystyle(\mathcal{U}^{\dagger}\circ\mathcal{U})\circ\mathcal{S}(\alpha)\circ\mathcal{U}^{\dagger}\circ\mathcal{S}^{-1}(\mathcal{U}[\alpha])$
$\displaystyle=$ $\displaystyle\mathcal{U}^{\dagger}=\mathcal{U}^{-1}$

where we have used $\mathcal{U}^{\dagger}\circ\mathcal{U}=\mathcal{I}$ the
identity channel, and
$\mathcal{U}\circ\mathcal{S}(\alpha)\circ\mathcal{U}^{\dagger}=\mathcal{S}(\mathcal{U}[\alpha])$
that is $U\sqrt{\alpha}U^{\dagger}=\sqrt{U\alpha U^{\dagger}}$.

The proof of (II) $\to$ (III) is analog to that for classical information.
Trivially, $\hat{\mathcal{E}}_{\alpha}\circ\mathcal{E}[\alpha]=\alpha$ holds
by definition of $\hat{\mathcal{E}}_{\alpha}$. Therefore, if
$\hat{\mathcal{E}}_{\alpha}=\hat{\mathcal{E}}$ for all $\alpha$, then
$\hat{\mathcal{E}}[\mathcal{E}[\rho]]=\rho$ for all $\rho$; whence
$\hat{\mathcal{E}}=\mathcal{E}^{-1}$.

Finally, for the proof of (III) $\to$ (I): since any Petz map is CPTP, the
starting assumption $\hat{\mathcal{E}}_{\alpha}=\mathcal{E}^{-1}$ implies that
$\mathcal{E}^{-1}$ is a CPTP map. But it is known that a CPTP map
$\mathcal{E}$ with the same input and output space has a CPTP inverse (that
is, it is invertible, and the inverse is itself a channel) if and only if it
is unitary Buscemi  _et al._ (2005)333If the input and output spaces are
allowed to differ, a very similar results hold with an obvious generalisation
Nayak and Sen (2006): one has to append a system to match the dimensions
before performing the unitary; and the inverse consists in undoing the unitary
and tracing the additional system.. ∎

###  III.2 Classical one-bit channels

As a second case study, we consider classical stochastic processes for $d=2$
(Fig. 2). We write a generic channel as

$\displaystyle M^{\varphi}=\begin{pmatrix}\varphi(0|0)&\varphi(0|1)\\\
\varphi(1|0)&\varphi(1|1)\end{pmatrix}\,\equiv\,\begin{pmatrix}1-a&b\\\
a&1-b\end{pmatrix}$ (16)

with $0\leq a,b\leq 1$. Its steady state is

$\displaystyle\gamma(0)=\frac{b}{a+b}\,$ ,
$\displaystyle\;\gamma(1)=\frac{a}{a+b}\,,$

unique unless $a=b=0$ (this being expected, since every state is a steady
state for the identity channel).

The corresponding retrodiction channel with generic reference prior is

$\displaystyle M^{\hat{\varphi}_{\xi}}$ $\displaystyle=$
$\displaystyle\begin{pmatrix}\hat{\varphi}_{\xi}(0|0)&\hat{\varphi}_{\xi}(0|1)\\\
\hat{\varphi}_{\xi}(1|0)&\hat{\varphi}_{\xi}(1|1)\end{pmatrix}$ (17)
$\displaystyle=$
$\displaystyle\begin{pmatrix}(1-a)\frac{\xi(0)}{\hat{\xi}(0)}&a\frac{\xi(0)}{\hat{\xi}(1)}\\\
b\frac{\xi(1)}{\hat{\xi}(0)}&(1-b)\frac{\xi(1)}{\hat{\xi}(1)}\end{pmatrix}$

with $\hat{\xi}(0)\,=\,(1-a)\xi(0)+b\xi(1)$ and $\hat{\xi}(1)=1-\hat{\xi}(0)$.
Interestingly, the retrodiction channel built on the steady state has the same
stochastic matrix as the channel itself:

$\displaystyle M^{\hat{\varphi}_{\gamma}}=M^{\varphi}$ $\displaystyle[d=2]\,.$
(18)

This can be verified without calculation, noticing that
$\hat{\varphi}_{\gamma}(x|x)=\varphi(x|x)$ and that
$M^{\hat{\varphi}_{\gamma}}$ must also be column-stochastic.

(a) 0in10out1$\varphi(0|0)$pt$\varphi(0|1)$pt$\varphi(1|0)$$\varphi(1|1)$

(b)
0in10out1$\hat{\varphi}_{\xi}(0|0)$pt$\hat{\varphi}_{\xi}(1|0)$pt$\hat{\varphi}_{\xi}(0|1)$$\hat{\varphi}_{\xi}(1|1)$

Figure 2: Bit channels (a) and their retrodiction (b) can be depicted as the
respective maps above.

The channel (16) is invertible if and only if $a+b\neq 1$. Result 1 of course
holds: the retrodiction channel will be the inverse if and only if $a=b=0$
(identity channel) or $a=b=1$ (bit-flip channel). For all the other invertible
channels, retrodiction and inversion do not coincide, whatever the choice of
the reference prior.

The non-invertible channels, $a+b=1$, make for an interesting case study; we
change the notation $a\rightarrow\epsilon$, so that

$\displaystyle M^{\varphi}=\begin{pmatrix}1-\epsilon&1-\epsilon\\\
\epsilon&\epsilon\end{pmatrix}\,.$ (19)

First notice that

$\displaystyle M^{\varphi}v^{p}=\begin{pmatrix}1-\epsilon\\\
\epsilon\end{pmatrix}$ (20)

for all input $p$. In other words, these channels erase whatever information
is present in the input, and produce a fixed output distribution (which, of
course, coincides with their steady state). In this sense, they could all be
called erasure channels, though the name is usually given to the case
$\epsilon=0$.

Because at the output all information on the input has been destroyed, one may
naively expect the retrodiction channel to produce a completely random
outcome. But this forgetting the importance of the reference prior in
retrodiction. Plugging the expressions in the equations, one readily finds

$\displaystyle M^{\hat{\varphi}_{\xi}}$ $\displaystyle=$
$\displaystyle\begin{pmatrix}\xi(0)&\xi(0)\\\ \xi(1)&\xi(1)\end{pmatrix}\,.$
(21)

The retrodiction channel of an erasure channel is the erasure channel that
returns the reference prior  – a result that can be easily extended to any
alphabet dimension 444 Consider a generic erasure channel
$\varphi(y|x)=\beta(y)$ for all $x,y\in[1,...,d]$, that sends any input state
to the state $\beta$. This will be the fate of any reference prior too:
$\hat{\xi}(y)=\beta(y)$. Thus Eq. (3) yields $\hat{\varphi}(x|y)=\xi(x)$.. In
agreement with (18), if the reference prior is the steady state, the
retrodiction channel is the same erasure channel555The fact that one can
associate a thermodynamically reversible process to a logically irreversible
channel like the erasure channel was noted for instance by Sagawa Sagawa
(2014). The identity between the forward and the reverse erasure channels was
noticed by Riechers and coworkers Riechers  _et al._ (2020) for a specific toy
model of physical erasure. We see here that it is an unavoidable feature,
having defined the reverse process from the steady state.. These observations
are summarized in Fig. 3.

(a) 0in10out1$1-\epsilon$pt$1-\epsilon$pt$\epsilon$$\epsilon$

(b) 0in10out1$1-\epsilon$pt$\epsilon$pt$1-\epsilon$$\epsilon$

(c) 0in10out1$\xi(0)$pt$\xi(1)$pt$\xi(0)$$\xi(1)$

Figure 3: (a) An $\epsilon$-erasure channel, (b) the retrodiction channel with
steady state as reference prior, (c) the retrodiction channel for a generic
reference prior.

##  IV Fluctuation relations from retrodiction

The topic of this section, fluctuation relations, originated in statistical
thermodynamics. As we shall see, the formal structure of these relations can
be derived without any reference to that branch of physics. As it happens, we
shall mention thermodynamics only in the very last paragraph of the section.
The explicit application of these formulas to important situations in
thermodynamics was discussed in our previous paper Buscemi and Scarani (2021).

###  IV.1 The process and its statistics

As we noted in the introduction, it customary in studies of irreversibility to
define the physical process as the forward process, and to compare it to its
corresponding reverse process. Here we adopt a different narrative:

  * •

There is only one process, the one that is happening.

  * •

A (forward) prediction on the process starts with a prior $p(x)$ on the input,
and infers the predicted distribution

$\displaystyle P_{F}(x,y)$ $\displaystyle=$ $\displaystyle
p(x)\varphi(y|x)\,.$ (22)

  * •

A retrodiction on the process starts with a prior $q(y)$ on the output, and
infers the retrodicted distribution

$\displaystyle P_{R}[\xi](x,y)$ $\displaystyle=$ $\displaystyle
q(y)\hat{\varphi}_{\xi}(x|y)\,.$ (23)

The explicit mention of $\xi$ will be dropped for simplicity in the remainder
of this Section and resumed in Section V.

We proceed to derive fluctuation relations with our narrative, and later we
show the comparison with the reverse-process narrative.

###  IV.2 Derivation of the fluctuation relations

Consider a variable $\Omega(x,y)$ that depends on the initial and final
states, and may be determined by the process. Its predicted distribution is

$\displaystyle\mu_{F}(\omega)$ $\displaystyle=$
$\displaystyle\sum_{x,y}\delta(\omega-\Omega(x,y))\,P_{F}(x,y)\,,$ (24)

while its retrodicted distribution is

$\displaystyle\mu_{R}(\omega)$ $\displaystyle=$
$\displaystyle\sum_{x,y}\delta(\omega-\Omega(x,y))\,P_{R}(x,y)$ (25)
$\displaystyle=$
$\displaystyle\sum_{x,y}\delta(\omega-\Omega(x,y))\,R(x,y)\,P_{F}(x,y)$

where

$\displaystyle R(x,y)$ $\displaystyle=$
$\displaystyle\frac{P_{R}(x,y)}{P_{F}(x,y)}\,.$ (26)

So, the difference between $\mu_{F}(\omega)$ and $\mu_{R}(\omega)$ is encoded
this ratio of probabilities, which is exactly the quantity that appears in the
statistical $f$-divergence Csiszár (1967); Liese and Miescke (2008)

$\displaystyle D_{f}(P_{R}||P_{F})$ $\displaystyle=$
$\displaystyle\sum_{x,y}P_{F}(x,y)\,f(R(x,y))$ (27)

where the function $f(r)$ must be convex for $r\in\mathbb{R}^{+}$ and satisfy
$f(1)=0$. The “entropy production”, on which the thermodynamical literature
bases fluctuation relations, uses $f(r)=-\ln(r)$, which generates the reverse
Kullback-Leibler distance $D_{KL}(P_{F}||P_{R})$. But we don’t need to choose
that particular function at this stage: for any function $f(r)$
invertible666Not all $f$ that are used for $f-$divergences are invertible. For
instance, $f(r)=r\ln(r)$ for the Kullback-Leibler distance, or
$f(r)=\frac{1}{2}|r-1|$ for the total variation distance. For such choices,
one can still study the comparison between $\mu_{R}(\omega)$ and
$\mu_{R}(\omega)$; it just won’t take the compact form (30). for
$r\in\mathbb{R}^{+}$, if we set

$\displaystyle\Omega(x,y)$ $\displaystyle=$ $\displaystyle f(R(x,y))$ (28)

we have by definition

$\displaystyle\left\langle\omega\right\rangle_{F}$ $\displaystyle=$
$\displaystyle D_{f}(P_{R}||P_{F})\,.$ (29)

Besides, there immediately follows from (24) and (25) the fluctuation relation

$\displaystyle\mu_{R}(\omega)$ $\displaystyle=$ $\displaystyle
f^{-1}(\omega)\mu_{F}(\omega)$ (30)

that is the generalisation of Crooks’ theorem Crooks (1998). By integrating
over $\omega$, one obtains the integral fluctuation relation

$\displaystyle\left\langle f^{-1}(\omega)\right\rangle_{F}$ $\displaystyle=$
$\displaystyle\sum_{x,y}P_{R}(x,y)\,=\,1$ (31)

that depends only on the process. This is the generalisation of Jarzynski’s
equality Jarzynski (1997).

###  IV.3 Comparison between retrodiction and reverse process

In all the literature we are aware of, fluctuation relations are presented as
a measure of the statistical difference between the forward and the reverse
process, not between the predicted and retrodicted distributions of a single
process. The difference between the two narratives has mathematical
manifestations that we are going to discuss now.

For the sake of definiteness, let us start with a canonical example. Suppose
that the variable of interest is entropy, and that in the process under study
it changes by $\Delta S$. In a retrodictive approach, (23) defines the
retrodicted distribution for that same process. But if one looks at (23) as
defining a reverse process, for that process the change of entropy will be
rather $-\Delta S$.

Generalising this observation, the distribution of the variable $\omega$ in
the reverse process reads

$\displaystyle\tilde{\mu}_{R}(\omega)$ $\displaystyle=$
$\displaystyle\sum_{x,y}\delta(\omega-\tilde{\Omega}(x,y))\,P_{R}(x,y)\,,$
(32)

where, under assumption (28),

$\displaystyle\tilde{\Omega}(x,y)$ $\displaystyle=$ $\displaystyle
f\left(\frac{1}{R(x,y)}\right)\,\equiv\,g(\Omega(x,y))$ (33)

because the roles of $P_{F}$ and $P_{R}$ are exchanged between the forward and
the reverse process [for the choice $f(r)=-\ln(r)$, there follows the expected
minus sign $g(\omega)=f(1/r)=-f(r)=-\omega$]. The resulting fluctuation
relation then reads Buscemi and Scarani (2021)

$\displaystyle\tilde{\mu}_{R}(g(\omega))|g^{\prime}(\omega)|$ $\displaystyle=$
$\displaystyle f^{-1}(\omega)\mu_{F}(\omega)\,.$ (34)

As expected, $\mu_{F}$ evaluated at $\omega$ is now related to
$\tilde{\mu}_{R}$ evaluated at $g(\omega)$. The Jacobian factor, which comes
from the change of variable in the $\delta$-function, ensures that the
integral fluctuation relation takes exactly the same form as (31).

A comparison of (30) and (34) for various choices of $f$ is given in Table 1.
For the thermodynamical case $f(r)=-\ln(r)$, we have $|g^{\prime}(\omega)|=1$,
and therefore the only difference between (30) and (34) is that $\mu_{R}$ is
evaluated at $\omega$ while $\tilde{\mu}_{R}$ is evaluated at $-\omega$. Thus
in thermodynamics not only the Jarzynski equality, but also the Crooks
fluctuation theorem is the same in both narratives (up to that sign change).
Interestingly, even when reporting experiments in which the reverse process
was actually implemented, it is the retrodictive version that is usually
plotted for its visual convenience: see for instance the pioneering
verification of Crooks’ fluctuation theorem with folding and unfolding of RNA
Collin  _et al._ (2005).

$\omega=f(r)$ |  $f$-divergence | FR for retrodiction (30) | FR for reverse process (34) | Integral FR (31)
---|---|---|---|---
$-\ln(r)$ | Reverse Kullback-Leibler | ${\mu}_{R}(\omega)=e^{-\omega}\mu_{F}(\omega)$ | $\tilde{\mu}_{R}(-\omega)=e^{-\omega}\mu_{F}(\omega)$ | $\left\langle e^{-\omega}\right\rangle_{F}=1$
$(1-\sqrt{r})^{2}$ | Squared Hellinger | $\mu_{R}(\omega)=(1-\sqrt{\omega})^{2}\mu_{F}(\omega)$ | $\tilde{\mu}_{R}\left(\frac{\omega}{(1-\sqrt{\omega})^{2}}\right)\frac{1+\sqrt{\omega}}{(1-\sqrt{\omega})^{2}}=(1-\sqrt{\omega})^{2}\mu_{F}(\omega)$ | $\left\langle(1-\sqrt{\omega})^{2}\right\rangle_{F}=1$
$1/r-1$ | Neyman $\chi^{2}$ | $\mu_{R}(\omega)=\frac{1}{1+\omega}\mu_{F}(\omega)$ | $\tilde{\mu}_{R}\left(\frac{\omega}{1+\omega}\right)\frac{1}{(1+{\omega})^{2}}=\frac{1}{1+\omega}\mu_{F}(\omega)$ | $\left\langle\frac{1}{1+\omega}\right\rangle_{F}=1$
Table 1: Fluctuation relations obtained in the retrodictive and in the
reverse-process narratives, for a few choices of $\omega$ satisfying (29) for
the corresponding $f-$divergence. The fourth column is kept in the form (34)
without possible algebraic simplifications, to facilitate the identification
of $g(\omega)$ and $|g^{\prime}(\omega)|$.

###  IV.4 Fluctuation relations and Bayesian retrodiction

In the retrodictive narrative, the fluctuation relation (30) and its derivate
(31) are statistical properties of the random variable $\omega$ defined by
(29). They are formally valid for the statistical comparison between arbitrary
$P_{F}$ and $P_{R}$, with no reference to the notion of retrodiction, let
alone to its mathematical expression (3). In the reverse process narratives,
one studies the distribution of the values of the variable when the roles of
$P_{F}$ and $P_{R}$ are swapped [Eq. (33)]; but even then, the fluctuation
relations follow without having specified any mathematical relation between
$P_{F}$ and $P_{R}$. So, what is the role of Bayesian retrodiction, or that of
a proper definition of the reverse process? We are going to prove that it
singles out a specific structure for $R(x,y)$, and that this simple result has
far-reaching consequences in the context of thermodynamics.

######  Result 3.

The ratio $R(x,y)$ [Eq. (26)] is of the form $F(x)G(y)$, for some functions
$F$ and $G$, if and only if $P_{F}$ and $P_{R}$ are related as (22) and (23),
with the latter constructed from Bayesian retrodiction [Eq. (3)]. In this
case,

$\displaystyle R(x,y)$ $\displaystyle=$
$\displaystyle\frac{\xi(x)}{p(x)}\frac{q(y)}{\hat{\xi}(y)}\,.$ (35)

Proof. If $P_{F}$ and $P_{R}$ are given by (22) and (23), using (3) it is
trivial to derive (35). In the other direction: without loss of generality we
keep the form (22) for $P_{F}$ and, using the product rule of joint
probabilities, we write $P_{R}(x,y)=q(y)\eta(x|y)$ for the conditional
distribution (channel) $\eta$ and marginal $q$. The assumption reads

$\displaystyle\frac{\eta(x|y)}{\varphi(y|x)}\,=\,\tilde{F}(x)\tilde{G}(y)$
$\displaystyle\forall_{x,y}$

with $\tilde{F}(x)=p(x)F(x)$ and $\tilde{G}(y)=G(y)/q(y)$. Since the l.h.s. is
strictly positive777Recall our assumption that all the entries of the channel
are positive; they could be as small as desired.,
$\textrm{sign}(\tilde{F}(x))=\textrm{sign}(\tilde{G}(y))$ must hold for all
$(x,y)$. Now, being a channel, $\eta$ must satisfy $\sum_{x}\eta(x|y)=1$, that
is $1/\tilde{G}(y)=\sum_{x}\tilde{F}(x)\varphi(y|x)$. So finally

$\displaystyle\eta(x|y)$ $\displaystyle=$
$\displaystyle\frac{\tilde{F}(x)}{\sum_{x}\tilde{F}(x)\varphi(y|x)}\varphi(y|x)\,\equiv\,\hat{\varphi}_{\xi}(x|y)$

where $\xi(x)=\tilde{F}(x)/\sum_{x}\tilde{F}(x)$ is a valid probability
distribution because all the $\tilde{F}(x)$ have the same sign. ∎

While this result may look purely anecdotal or formal, let us recall that in
the usual thermodynamical interpretation $\Omega(x,y)=-\ln(R(x,y))$ is the
(non-adiabatic) stochastic entropy production Esposito and Van den Broeck
(2010). Thus, whenever the stochastic entropy production can be computed
locally (that is, independently of the correlations between microstates $x$
and $y$), a structure of Bayesian retrodiction is unavoidable (in the reverse
process narrative: the reverse process must be defined through Bayesian
retrodiction).

##  V On the choice of the reference prior

In the literature on fluctuation relations, based on thermodynamics, the
wording “reference prior” is absent. Its role is usually taken by an
assumption of “detailed-balance”. In all the examples that we have looked into
Buscemi and Scarani (2021), this corresponds to the choice of the steady state
as reference prior. The operational interpretation of this choice is very
physical: one takes as reference the process in which nothing changes. It has
also a very neat consequence when it comes to fluctuation relations: the ratio
$R(x,y)$ given in (35), and thus the variable that enters the fluctuation
relations, depends on the channel $\varphi$ only through its steady state
$\gamma$. With this choice, one is clearly studying fluctuations around
equilibrium888We used the wording “steady state” instead of “equilibrium”
because the latter has become almost synonymous of thermal equilibrium. Our
channels have nothing thermal a priori, and even for channels involving
contact with a thermal bath, the thermal state may not be the steady state
(e.g. if the Hamiltonian changes during the process).

Inspired by statistical comparisons, one may opt for a different definition of
the reference prior. One possibility is trying to keep prediction and
retrodiction as close as possible. With such goals, let us take as a figure of
merit

$\displaystyle D_{KL}(P_{F}||P_{R}[\xi])$ $\displaystyle=$
$\displaystyle\sum_{x,y}P_{F}(x,y)\ln\left(\frac{P_{F}(x,y)}{P_{R}[\xi](x,y)}\right)$
$\displaystyle=$
$\displaystyle\sum_{x,y}P_{F}(x,y)\ln\frac{p(x)}{q(y)}+\sum_{x,y}P_{F}(x,y)\ln\frac{\hat{\xi}(y)}{\xi(x)}\,,$

which we called the reverse Kullback-Leibler distance in the previous Section.
Only the second term depends on the reference prior; besides, it does not
depend on $q(y)$, but does depend on $p(x)$ that may be arbitrary. We may then
choose $\xi$ as to minimize the average of $D_{KL}(P_{F}||P_{R}[\xi])$ over
all possible choices of $p$. Upon such averaging,
$p(x)\rightarrow\frac{1}{d}$. Thus we want to find
$\xi_{D}=\text{argmin}F[\xi]$ with

$\displaystyle F[\xi]$ $\displaystyle=$
$\displaystyle\sum_{x,y}\varphi(y|x)\ln\frac{\hat{\xi}(y)}{\xi(x)}\,.$ (36)

Interestingly, we have:

######  Result 4.

The reference prior that minimises (36) is $\xi_{D}(x)=\frac{1}{d}$ the
uniform prior, for every $\varphi$.

We present the proof in Appendix A.

In the same spirit, one can study the reference priors that minimize other
figures of merit averaged over the possible priors $p(x)$ and $q(y)$. We run
some simple numerical checks at $d=2$ for two other figures of merit. For the
Kullback-Leibler distance $D_{KL}(P_{R}[\xi]||P_{F})$, the steady state is
generically not optimal, while the uniform prior seems to be optimal again,
even though the dependence on $\xi$ is different from (36). For the guessing
probability, i.e. the probability that
$\textrm{argmax}P_{F}(x,y)=\textrm{argmax}P_{R}[\xi](x,y)$, neither the steady
state nor the uniform prior are generically optimal.

##  VI Final considerations: Do we need a reverse physical process?

The everyday meaning of (ir)reversibility in nature is captured by the
perceived “arrow of time”: if the video of the evolution played backward makes
sense, the process is reversible; if it doesn’t make sense, it is
irreversible.

Science has gone very far in bringing this intuition on quantitative ground.
The standard underlying narrative still involves two processes: the one that
we observe, and the associated reverse process (not deemed to be strictly
impossible, but very unlikely). This reverse process is generically not the
video played backward: to cite an extreme example, nobody conceives bombs that
fly upward to their airplanes while cities are being built from rabble Earman
(1974); Barrett and Sober (1992). In the case of controlled protocols in the
presence of an unchanging environment, the reverse process is implemented by
reversing the protocol. If the environment were to change (in an uncontrolled
way, by definition of environment), the connection between the physical
process and the associated reverse one becomes thinner.

With our line of research, we are exploring the possibility that the narrative
of the reverse process may not be needed at all. In the wording pioneered by
Watanabe, irreversibility may be rather irretrodictability. So far, this
program has found no obstacle, and has even clarified situations that were
deemed puzzling in the case of some quantum channels Buscemi and Scarani
(2021). The vistas opened by this approach also allow to expand the scope of
fluctuation relations (Section IV) and discuss the choice of a reference prior
(Section V).

Barring surprises à la John Bell, this conflict of narratives won’t be
discriminated by experiments. Indeed, on the one hand, the retrodiction
channels (both classical and quantum) are by construction valid channels:
nothing forbids the physical implementation of the corresponding processes, as
indeed was done in the experimental verifications of Crooks’ theorem Collin
_et al._ (2005). On the other hand, to falsify the retrodictive narrative, one
would have to find a reverse process related to its original process in a way
that cannot be expressed by (or worse, contradicts) logical reasoning: it is
hard to see how such a claim could ever be made. So, one’s narrative of choice
will depend on the fruitfulness of the intuition, the economy of concepts, the
elegance of the formulas… In this paper, we have hinted at the superiority of
the retrodictive narrative in all these respects.

## Acknowledgments

We acknowledge the help of Eugene Koh in completing the proof of Result 4.
F.B. acknowledges support from the Japan Society for the Promotion of Science
(JSPS) KAKENHI, Grants Nos.19H04066 and 20K03746, and from MEXT Quantum Leap
Flagship Program (MEXT Q-LEAP), Grant Number JPMXS0120319794. V.S.
acknowledges support from the National Research Foundation and the Ministry of
Education, Singapore, under the Research Centres of Excellence programme.

Data sharing is not applicable to this article as no new data were created or
analyzed in this study.

##  Appendix A The reference prior that minimizes the average Kullback-Leibler
distance

Consider a generic classical channel $\varphi$ with $d$-dimensional input and
output alphabets. Denote the reference prior as $\xi(x)=\frac{1}{d}(1+u_{x})$
with $-1\leq u_{x}\leq d-1$ and $\sum_{x}u_{x}=0$. With this parametrisation,
$F[\xi]\equiv F(\underline{u})$ with

$\displaystyle F(\underline{u})$ $\displaystyle=$
$\displaystyle\sum_{xy}\varphi(y|x)\ln\left(\frac{\sum_{x^{\prime}}\varphi(y|x^{\prime})(1+u_{x^{\prime}})}{1+u_{x}}\right)\,.$

On the uniform prior ($u_{x}=0$ for all $x$), this takes the value

$\displaystyle F(\underline{0})$ $\displaystyle=$
$\displaystyle\sum_{xy}\varphi(y|x)\ln\left(\sum_{x^{\prime}}\varphi(y|x^{\prime})\right)\,.$

Thus, $\Delta\equiv F(\underline{u})-F(\underline{0})$ is equal to

$\displaystyle\Delta$ $\displaystyle=$
$\displaystyle\sum_{xy}\varphi(y|x)\ln\left(\frac{\sum_{x^{\prime}}\varphi(y|x^{\prime})(1+u_{x^{\prime}})}{\sum_{x^{\prime}}\varphi(y|x^{\prime})}\right)-\sum_{x}\ln(1+u_{x})$
$\displaystyle\geq$
$\displaystyle\sum_{xy}\varphi(y|x)\frac{\sum_{x^{\prime}}\varphi(y|x^{\prime})\ln(1+u_{x^{\prime}})}{\sum_{x^{\prime}}\varphi(y|x^{\prime})}-\sum_{x}\ln(1+u_{x})$
$\displaystyle=$
$\displaystyle\sum_{x,y}\varphi(y|x)\ln(1+u_{x})-\sum_{x}\ln(1+u_{x})\,=\,0$

where the inequality is Jensen’s inequality on the first term. Thus we have
proved that

$\displaystyle F(\underline{u})-F(\underline{0})\geq 0\;$
$\displaystyle\;\textrm{for all }\underline{u}\,.$ (37)

## References

## References

  * Buscemi and Scarani (2021) F. Buscemi and V. Scarani, “Fluctuation theorems from bayesian retrodiction,” Phys. Rev. E 103, 052111 (2021).
  * Bochkov and Kuzovlev (1977) G. N. Bochkov and Y. E. Kuzovlev, “General theory of thermal fluctuations in nonlinear systems,” Sov. Phys. JETP 45, 125–130 (1977).
  * Jarzynski (1997) C. Jarzynski, “Nonequilibrium equality for free energy differences,” Phys. Rev. Lett. 78, 2690–2693 (1997).
  * Crooks (1998) G. E. Crooks, “Nonequilibrium Measurements of Free Energy Differences for Microscopically Reversible Markovian Systems,” Journal of Statistical Physics 90, 1481–1487 (1998).
  * Jarzynski (2000) C. Jarzynski, “Hamiltonian derivation of a detailed fluctuation theorem,” Journal of Statistical Physics 98, 77–102 (2000).
  * Tasaki (2000) H. Tasaki, “Jarzynski relations for quantum systems and some applications,” (2000), arXiv:cond-mat/0009244 [cond-mat.stat-mech] .
  * Campisi, Hänggi, and Talkner (2011) M. Campisi, P. Hänggi, and P. Talkner, “Colloquium: Quantum fluctuation relations: Foundations and applications,” Rev. Mod. Phys. 83, 771–791 (2011).
  * Jarzynski (2011) C. Jarzynski, “Equalities and inequalities: Irreversibility and the second law of thermodynamics at the nanoscale,” Annual Review of Condensed Matter Physics 2, 329–351 (2011).
  * Seifert (2012) U. Seifert, “Stochastic thermodynamics, fluctuation theorems and molecular machines,” Reports on Progress in Physics 75, 126001 (2012).
  * Gawedzki (2013) K. Gawedzki, “Fluctuation relations in stochastic thermodynamics,” (2013), arXiv:1308.1518 [math-ph] .
  * Watanabe (1955) S. Watanabe, “Symmetry of physical laws. part iii. prediction and retrodiction,” Rev. Mod. Phys. 27, 179–186 (1955).
  * Watanabe (1965) S. Watanabe, “Conditional probabilities in physics,” Progr. Theor. Phys. Suppl. E65, 135–160 (1965).
  * Chan and Darwiche (2005) H. Chan and A. Darwiche, “On the revision of probabilistic beliefs using uncertain evidence,” Artificial Intelligence 163, 67 – 90 (2005).
  * Jacobs (2019) B. Jacobs, “The mathematics of changing one’s mind, via Jeffrey’s or via Pearl’s update rule,” Journal of Artificial Intelligence Research 65, 783–806 (2019).
  * Jeffrey (1965) R. Jeffrey,  _The logic of decision_ (McGraw-Hill, 1965).
  * Pearl (1988) J. Pearl,  _Probabilistic reasoning in intelligent systems: networks of plausible inference_ (Morgan Kaufmann, 1988).
  * Jaynes (2003) E. T. Jaynes,  _Probability Theory: The Logic of Science_, edited by G. L. Bretthorst (Cambridge University Press, 2003).
  * Barnett, Jeffers, and Pegg (2021) S. M. Barnett, J. Jeffers, and D. T. Pegg, “Quantum retrodiction: Foundations and controversies,” Symmetry 13 (2021), 10.3390/sym13040586.
  * Note (1) This is particularly clear in the example of the test for a sickness given above. We can also refer the reader to the studies of retrodiction of quantum temporal dynamics Gammelmark, Julsgaard, and Mølmer (2013); Bao  _et al._ (2020).
  * Petz (1986) D. Petz, “Sufficient subalgebras and the relative entropy of states of a von neumann algebra,” Comm. Math. Phys. 105, 123–131 (1986).
  * Petz (1988) D. Petz, “Sufficiency of channels over von Neumann algebras,” The Quarterly Journal of Mathematics 39, 97–108 (1988).
  * Wilde (2013) M. M. Wilde,  _Quantum Information Theory_ (Cambridge University Press, 2013).
  * Sutter, Fawzi, and Renner (2016) D. Sutter, O. Fawzi, and R. Renner, “Universal recovery map for approximate markov chains,” Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 472, 20150623 (2016).
  * Alhambra and Woods (2017) A. M. Alhambra and M. P. Woods, “Dynamical maps, quantum detailed balance, and the petz recovery map,” Physical Review A 96 (2017), 10.1103/physreva.96.022118.
  * Fuchs (2002) C. A. Fuchs, “Quantum mechanics as quantum information (and only a little more),” (2002), arXiv:quant-ph/0205039 [quant-ph] .
  * Leifer and Spekkens (2013) M. S. Leifer and R. W. Spekkens, “Towards a formulation of quantum theory as a causally neutral theory of bayesian inference,” Phys. Rev. A 88, 052130 (2013).
  * Crooks (2008) G. E. Crooks, “Quantum operation time reversal,” Phys. Rev. A 77, 034101 (2008).
  * Fields, Sajia, and Bergou (2020) D. Fields, A. Sajia, and J. A. Bergou, “Quantum retrodiction made fully symmetric,” (2020), arXiv:2006.15692 [quant-ph] .
  * Note (2) Petz discovered this map in the context of the monotonicity property of the quantum relative entropy $D(\rho||\alpha)=\Tr{\rho\ln\rho-\rho\ln\alpha}$ Umegaki (1961). For any CPTP linear map $\mathcal{E}$, one has $D(\rho||\alpha)-D(\mathcal{E}[\rho]||\mathcal{E}[\alpha])\geq 0$. Petz showed that $D(\rho||\alpha)=D(\mathcal{E}[\rho]||\mathcal{E}[\alpha])$ if and only if $\mathaccentV{hat}05E{\mathcal{E}}_{\alpha}\circ\mathcal{E}[\rho]=\rho$. In other words, $\mathaccentV{hat}05E{\mathcal{E}}_{\alpha}$ reconstructs not only $\alpha$ after the channel, but every state whose relative entropy with respect to $\alpha$ has not changed.
  * Kwon and Kim (2019) H. Kwon and M. S. Kim, “Fluctuation theorems for a quantum channel,” Phys. Rev. X 9, 031029 (2019).
  * Buscemi  _et al._ (2005) F. Buscemi, G. M. D’Ariano, M. Keyl, P. Perinotti, and R. Werner, “Clean positive operator valued measures,” J. Math. Phys. 46, 082109 (2005).
  * Note (3) If the input and output spaces are allowed to differ, a very similar results hold with an obvious generalisation Nayak and Sen (2006): one has to append a system to match the dimensions before performing the unitary; and the inverse consists in undoing the unitary and tracing the additional system.
  * Note (4) Consider a generic erasure channel $\varphi(y|x)=\beta(y)$ for all $x,y\in[1,...,d]$, that sends any input state to the state $\beta$. This will be the fate of any reference prior too: $\mathaccentV{hat}05E{\xi}(y)=\beta(y)$. Thus Eq. (3) yields $\mathaccentV{hat}05E{\varphi}(x|y)=\xi(x)$.
  * Note (5) The fact that one can associate a thermodynamically reversible process to a logically irreversible channel like the erasure channel was noted for instance by Sagawa Sagawa (2014). The identity between the forward and the reverse erasure channels was noticed by Riechers and coworkers Riechers  _et al._ (2020) for a specific toy model of physical erasure. We see here that it is an unavoidable feature, having defined the reverse process from the steady state.
  * Csiszár (1967) I. Csiszár, “Information type measures of difference of probability distribution and indirect observations,” Studia Scient. Math. Hungar. 2, 299–381 (1967).
  * Liese and Miescke (2008) F. Liese and K.-J. Miescke,  _Statistical Decision Theory_ (Springer, 2008).
  * Note (6) Not all $f$ that are used for $f-$divergences are invertible. For instance, $f(r)=r\ln(r)$ for the Kullback-Leibler distance, or $f(r)=\frac{1}{2}|r-1|$ for the total variation distance. For such choices, one can still study the comparison between $\mu_{R}(\omega)$ and $\mu_{R}(\omega)$; it just won’t take the compact form (30).
  * Collin  _et al._ (2005) D. Collin, F. Ritort, C. Jarzynski, S. B. Smith, I. Tinoco, and C. Bustamante, “Verification of the crooks fluctuation theorem and recovery of rna folding free energies,” Nature 437, 231–234 (2005).
  * Note (7) Recall our assumption that all the entries of the channel are positive; they could be as small as desired.
  * Esposito and Van den Broeck (2010) M. Esposito and C. Van den Broeck, “Three detailed fluctuation theorems,” Phys. Rev. Lett. 104, 090601 (2010).
  * Note (8) We used the wording “steady state” instead of “equilibrium” because the latter has become almost synonymous of thermal equilibrium. Our channels have nothing thermal a priori, and even for channels involving contact with a thermal bath, the thermal state may not be the steady state (e.g. if the Hamiltonian changes during the process).
  * Earman (1974) J. Earman, “An attempt to add a little direction to "the problem of the direction of time",” Philosophy of Science 41, 15–47 (1974).
  * Barrett and Sober (1992) M. Barrett and E. Sober, “Is entropy relevant to the asymmetry between retrodiction and prediction?” The British Journal for the Philosophy of Science 43, 141 – 160 (1992).
  * Gammelmark, Julsgaard, and Mølmer (2013) S. Gammelmark, B. Julsgaard, and K. Mølmer, “Past quantum states of a monitored system,” Phys. Rev. Lett. 111, 160401 (2013).
  * Bao  _et al._ (2020) H. Bao, S. Jin, J. Duan, S. Jia, K. Mølmer, H. Shen, and Y. Xiao, “Retrodiction beyond the Heisenberg uncertainty relation,” Nature Communications 11, 5658 (2020).
  * Umegaki (1961) H. Umegaki, “On information in operator algebras,” Proc. Japan Acad. 37, 459–461 (1961).
  * Nayak and Sen (2006) A. Nayak and P. Sen, “Invertible quantum operations and perfect encryption of quantum states,” (2006), arXiv:quant-ph/0605041 .
  * Sagawa (2014) T. Sagawa, “Thermodynamic and logical reversibilities revisited,” Journal of Statistical Mechanics: Theory and Experiment 2014, P03025 (2014).
  * Riechers  _et al._ (2020) P. M. Riechers, A. B. Boyd, G. W. Wimsatt, and J. P. Crutchfield, “Balancing error and dissipation in computing,” Phys. Rev. Research 2, 033524 (2020).


# Information Guided Regularization for Fine-tuning Language Models

Mandar Sharma1, Nikhil Muralidhar2, Shengzhe Xu1
Raquib Bin Yousuf1 & Naren Ramakrishnan1
1Department of Computer Science, Virginia Tech
2Department of Computer Science, Stevens Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

The pretraining-fine-tuning paradigm has been the de facto strategy for
transfer learning in modern language modeling. With the understanding that
task adaptation in LMs is often a function of parameters shared across tasks,
we argue that a more surgical approach to regularization needs to exist for
smoother transfer learning. Towards this end, we investigate how the
pretraining loss landscape is affected by these task-sensitive parameters
through an information-theoretic lens. We then leverage the findings from our
investigations to devise a novel approach to dropout for improved model
regularization and better downstream generalization. This approach, named
guided dropout, is both task & architecture agnostic and adds no computational
overhead to the fine-tuning process. Through empirical evaluations, we
showcase that our approach to regularization yields consistently better
performance, even in scenarios of data paucity, compared to standardized
baselines. Our codebase for reproducibilty is hosted
here111https://github.com/Mandar-Sharma/Guided-Dropout.

##  1 Introduction

Although highly-parameterized LMs have recently demonstrated incredible few-
shot and zero-shot capabilities (OpenAI, 2023; Touvron et al., 2023), transfer
learning via fine-tuning still remains crucial for building task-specific
models (Sharma et al., 2022; Min et al., 2023). As such, mechanisms to improve
the efficiency of fine-tuning parameter-heavy LMs have garnered much attention
leading to frameworks like LoRA (Hu et al., 2021). However, the aspect of
transfer learning that doesn’t receive as much attention is regularization. As
over-parameterized models are made to adapt to niche tasks with few data
points, effective regularization can go a long way into building models that
generalize better to downstream tasks. Specially in light of recent findings
that LMs tend to share parameters depending on the specific downstream task
(Sharma et al., 2023), we argue against regularizers that treat all LM
parameters the same - all neurons are equal but some neurons are more equal
than others. Here, we empirically study the effects that LM parameters have on
their loss geometry and propose a more surgical and effective approach to
regularization.

As our proposed approach is independent of the target task, it simply involves
appending a few lines of pre-determined code to the LM training loop to obtain
a slight performance improvement on virtually any downstream task. This could
have significant implications as foundational models are often used for a
colossal number of downstream applications - the bert-base-uncased model has
been downloaded more than 1.5 billion times on HuggingFace (Wolf, 2024).
Concisely, we offer the following contributions:

  * •

We study the effects that task-sensitive parameters have on the LM loss
landscape through their visual geometry.

  * •

Through our findings, we propose a novel information guided approach to L2
regularization that its both task & architecture agnostic and adds no
computational overhead to the fine-tuning process.

  * •

We empirically showcase that our approach to guided dropout yields
consistently better performance compared to standardized baselines, specially
in scenarios of data paucity.

  * •

Alongside, we also showcase that a reliable estimate of the model information
can be obtained in a cost-effective manner simply through a small sub-sample
of the training corpus.

##  2 Theoretical Foundations

The theoretical properties underpinning the mini-batch based stochastic
gradient descent (SGD) has received thorough exploration from the scientific
community (Ge et al., 2015; Hardt et al., 2016; Bottou et al., 2018).
Specially relevant to us is the investigation of how loss converges through
the approximation of the eigenvalues of the Hessian (Keskar et al., 2016) and
through its visualized geometry (Li et al., 2018; Yao et al., 2020). With
Fisher information as a proxy to the Hessian (§2.1), we investigate how this
computationally cheaper first-order metric can lend us an understanding of how
the LM loss landscape geometry is affected by its task-specific parameters
(§2.2). Further, we leverage these insights to devise an information-guided
approach to L2 regularization via dropout (Wan et al., 2013) for transfer
learning (§2.3).

Figure 1: The 3D loss landscape geometry of the pretrained $BERT_{BASE}$ model
when: (a) all model parameters are perturbed, (b) 50% of the model parameters
with the highest Fisher scores are perturbed, and (c) 50% of the model
parameters with the lowest Fisher scores are perturbed. Plot (b) resembles the
typical sharp minimizers that generalize poorly compared to the wide
minimizers of base plot (a) and control plot (c).

###  2.1 A Tale of Two Matrices: The Fisher & the Hessian

Observation 1: At $\theta\rightarrow\theta_{o}$, the Fisher information
approximates the Hessian and thus can be used to understand the training
dynamics and loss landscapes of LMs.

For an LM parameterized by $\theta$, the Fisher score function $s(\theta)$
represents the gradient of the log-likelihood function $\log p(x|\theta)$. The
Fisher information $I(\theta)$ is understood as the covariance matrix $K$ of
the score function (1). Characteristically, the Fisher score evaluated at the
true value of the model parameter $\theta_{o}$ is 0, giving us (2). However,
as this requires computing the expectation over the distribution $\theta$,
which is often intractable, the Empirical Fisher (3) is computed given a
dataset $x\in X$ with $N$ samples.

$\displaystyle I(\theta)=K_{s(\theta)}$
$\displaystyle=\mathbb{E}[(s(\theta)-\mathbb{E}[s(\theta_{o})])(s(\theta)-\mathbb{E}[s(\theta_{o})])^{T}]$
(1) $\displaystyle=\mathbb{E}[s(\theta)s(\theta)^{T}]$ (2)
$\displaystyle\equiv\frac{1}{N}\sum_{i=1}^{N}\frac{\partial\log
p(x|\theta)}{\partial\theta}\frac{\partial\log
p(x|\theta)}{\partial\theta^{T}}$ (3)

Similarly, the Hessian for the LM characterized by $\theta$ can be written as:

$H=\frac{1}{N}\sum_{i=1}^{N}\frac{\partial\log
p(x|\theta)}{\partial\theta}\frac{\partial\log
p(x|\theta)}{\partial\theta^{T}}-\frac{1}{p(x|\theta)}\frac{\partial^{2}p(x|\theta)}{\partial\theta\partial\theta^{T}}$
(4)

Jastrzebski et al. (2017) note that the second term in (4) is negligible in
the case when the conditional probability distribution of the model coincides
with the conditional distribution of the training data. Mathematically, this
occurs when the model parameters of the LM $\theta$ is close to the optimum
such that $\theta\rightarrow\theta_{o}$. Thus, for a converged pretrained LM,
the covariance represented by the Empirical Fisher (3) is approximately the
Hessian.

| Top 5% of Parameters | Top 1% of Parameters
---|---|---
| $I_{N}\ast I_{S}$ | $DKL(I_{N},I_{S})$ | $I_{N}\ast I_{S}$ | $DKL(I_{N},I_{S})$
Sample | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$
33% | 0.9864 | 0.0409 | 0.0055 | 0.0304 | 0.9767 | 0.0825 | 0.0044 | 0.0243
10% | 0.9428 | 0.1219 | 0.0164 | 0.0451 | 0.9305 | 0.1665 | 0.0159 | 0.0415
5% | 0.9661 | 0.0788 | 0.0126 | 0.0544 | 0.9547 | 0.1171 | 0.01 | 0.0407
1% | 0.9233 | 0.1338 | 0.0261 | 0.0923 | 0.9111 | 0.1732 | 0.0212 | 0.0665
0.10% | 0.7564 | 0.2448 | 0.1149 | 0.2272 | 0.7349 | 0.3035 | 0.1011 | 0.1757
| Top 0.5% of Parameters | Top 0.25% of Parameters
| $I_{N}\ast I_{S}$ | $DKL(I_{N},I_{S})$ | $I_{N}\ast I_{S}$ | $DKL(I_{N},I_{S})$
| $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$
33% | 0.9643 | 0.1427 | 0.004 | 0.0208 | 0.9399 | 0.3163 | 0.0034 | 0.0169
10% | 0.9136 | 0.2057 | 0.014 | 0.0348 | 0.8784 | 0.4215 | 0.0112 | 0.0274
5% | 0.9445 | 0.1597 | 0.0092 | 0.0351 | 0.8998 | 0.4003 | 0.0072 | 0.0271
1% | 0.8916 | 0.2504 | 0.0196 | 0.0574 | 0.8424 | 0.4822 | 0.017 | 0.048
0.10% | 0.7208 | 0.3792 | 0.0898 | 0.1599 | 0.2908 | 0.3956 | 0.0612 | 0.1451
Table 1: The layer-wise mean $\mu$ and standard deviation $\sigma$ of the
cross-correlation ($\ast$) and the KL-Divergence (DKL) between the Fisher
information computed with the entire WikiText-103 dataset ($I_{N}$) vs that
computed with the sub-sampled dataset ($I_{S}$, where $S<<N$) for the base
BERT model. As noted in section §2.2, since only a fraction of the parameters
contain the highest concentration of Fisher scores, we evaluate the
consistently of the Fisher information matrix based on the top {5, 1, 0.5,
0.25} percent of the LM parameters. Based on this table, we conclude that
reliable estimates of the Fisher information can be obtained through as little
as 1-5 % of the training data.

Observation 2: For LMs, reliable estimates of the Fisher information can be
obtained from a fractional sub-sampled corpus.

While the eigenvalues of the Fisher matrix can certainly be computed
effectively in terms of memory and run-time (Liao et al., 2018), we showcase
that for LMs, the entire Fisher information matrix $I(\theta)$ can be
approximated through a fractional sub-sample of the original training corpus.
The pretraining corpus for $BERT_{BASE}$ consists of a combination of the
Wikipedia corpus and the book corpus ($\sim$ 7 GB) (Devlin et al., 2019). We
attain our approximation of the Empirical Fisher $I(\theta)$ for $BERT_{BASE}$
through WikiText-103 (Merity et al., 2016), a much smaller cut ($\sim$ 4%) of
BERT’s original pretraining corpus. The effective results we present in
section §3 are all based on this approximation. We attribute this observation
to the nature of the pretraining LMs - the redundancy that exits as a
consequence of being trained on large unstructured corpora of free-form text
(Dalvi et al., 2020). To further illustrate this point, in Table 1, we
approximate $I(\theta)$ with incrementally smaller sub-samples from
WikiText-103. We find reliable estimates of $I(\theta)$ even with just 1% to
5% of the samples from WikiText (0.04% - 0.2% of the original pretraining
corpus), although the quality of the estimate drops if we sub-sample below
0.01% (0.004 % of the original corpus).

###  2.2 Fisher Information & the LM Loss Landscape

Prior studies have used the Fisher information as a substitute to the Hessian
for characterizing the properties of stochastic gradient descent (Liao et al.,
2018; Martens, 2020). To attain complementary insights, we perform an
empirical study of the loss geometry of LMs as a function of the Fisher
information $I(\theta)$. Borrowing the filter-normalization based
visualization scheme of Li et al. (2018), we plot the loss landscape of the
pretrained $BERT_{BASE}$ (Devlin et al., 2019) as $f(\alpha,\beta)$ (5) where
the cross-entropy loss $L$ is computed as the converged model parameters
$\theta_{o}$ are perturbed with directional vectors $\delta$ and $\eta$ scaled
by $\alpha$ and $\beta$ respectively. The loss landscape for the $BERT_{BASE}$
is depicted in Figure 1 (a).

$f(\alpha,\beta)=L(\theta_{o}+\alpha\delta+\beta\eta)$ (5)

Observation 3: For LMs, an exceedingly tiny fraction of the model parameters
have significantly high Fisher scores and they have a pronounced effect on the
loss geometry.

LMs are known to have task-specific peculiarities in the parametric
distribution of their Fisher information (Sharma et al., 2023). We observe
similar behaviors where only an exceedingly tiny fraction of the model
parameters have significantly high Fisher scores (see Figure 3 a). To observe
the effects of these parameters on the training loss geometry, we selectively
perturb half of the parameters of $\theta_{o}$ that lie on the higher end of
the Fisher score spectrum (see Figure 1 b). The loss geometry so formed
resembles a typical sharp minimizer. These have been known to lead to poor
generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016;
Jastrzebski et al., 2017). On the other hand, the loss geometry of the BERT
model with the entirety of $\theta_{o}$ perturbed (Figure 1 a) resembles that
of a wide minimizer that are known to generalize better. To act as a control
experiment to strengthen findings from Figures 1 a & b, we plot the loss
geometry of the same model but this time perturbing those half of $\theta_{o}$
that lie on the lower end of the Fisher score spectrum (see Figure 1 c). This
control plot does not deviate from the wide shape of the loss geometry of the
$BERT_{BASE}$ model in Figure 1 a. Thus, the half of $BERT_{BASE}$ parameters
with higher Fisher scores contribute drastically more to the loss geometry
than the half with lower Fisher scores.

Figure 2: The loss landscape geometry of the pretrained $BERT_{BASE}$ model
when: (a) & (b) $n$ % of parameters with the highest Fisher scores are
perturbed, (c) $n$ % of randomly chosen parameters are perturbed. Plot (a)
showcases worsening minimas as we narrow down on the parameters deemed
essential by the Fisher score. Plot (b) showcases that the worsening stops and
the landscape remains consisten as the we take $\leq$ 40% of the highest
scoring parameters. Plot (c) acts as the control plot.

Further, incrementally focusing the model perturbation on those selective LM
parameters that hold the highest Fisher scores leads to an incremental
degradation in the loss landscape (see Figure 2 a). Here, as we perturb fewer
and fewer parameters while focusing on the ones with highest Fisher scores
(from 80% of the parameters with the highest Fisher scores all the way to
0.25% of the parameters with the highest scores), we see the loss landscape
incrementally converging to flatter regions in the neighborhood of the
minimum, again leading to poor generalization (Hochreiter & Schmidhuber, 1997;
Keskar et al., 2016; Jastrzebski et al., 2017). It is interesting to note that
as we go below 40%, the effect of parameter perturbation on the loss landscape
seems to remain consistent. This implies that the 220,000 (0.25%) out of
160,000,000 BERT parameters are so vital to the model convergence that
perturbing them produces as pronounced of an effect as perturbing 44,000,000
(40%) of the model parameters. Similar to the 3D loss geometries, to further
solidify our observations, we introduce a control plot (see Figure 2 c). In
this plot, there is no consistent signal of worsening loss geometry as we
perturb randomly chosen 80%, 60%, 40% of the model parameters.

###  2.3 Fisher Information & Guided Regularization

Observation 4: For transformers, a minority of the layers hold a significant
concentration of Fisher sensitive parameters. This observation is used to
devise a surgical approach to regularization.

Shifting our attention from the atomic parameter-wise view to an aggregate
layer-wise view, we yet again observe similar findings: a minority of the
transformer layers hold a significant concentration of Fisher sensitive
parameters (see Figure 3b, see Appendix §A.2 for individual non-normalized
plots). Based on this observation and our findings from §2.2, we devise an
approach to L2 regularization for transfer learning which we term as
information-guided regularization. Our idea is rather straightforward:
implement L2 regularization such that the sway from current model convergence
is minimal. In other words, during transfer learning, focus the regularization
more on the LM layers deemed less important by the Fisher information and
relax regularization on those layers deemed as vital by the same. Based on
conventional popularity, we choose dropout (Srivastava et al., 2014) as our L2
implementation. For simplicity, our approach to regularization is
algorithmically illustrated in appendix §A.4.

Figure 3: The sorted Fisher scores (normalized) for BERT, GPT2, and T5 based
their (a) model parameters and (b) model layers (aggregated) - see appendix
§A.1 and §A.2 for the raw (non-normalized) plots. From (a) we observe that a
fraction of LM parameters are often attributed the highest Fisher scores.
Similarly, (b) provides an aggregate view showcasing how a minority of the LM
layers hold the majority of training information. Please see appendix §A.3 for
the unsorted layer-wise Fisher score distributions of these models.

Assuming a LM with $l\in\\{1,...,|L|\\}$ layers such that the output of a
given layer $l$ with weights $W^{(l)}$ and biases $b^{(l)}$ can be represented
as:

$y^{(l)}=f(W^{(l)}y^{(l-1)}+b^{(l)})$ (6)

Here, $f$ is the activation function. If layer $l$ is followed by a standard
dropout layer in the LM network, the feed-forward operation becomes:

$\displaystyle r^{(l)}$ $\displaystyle\sim\mathrm{Bernoulli}(p)$ (7)
$\displaystyle\tilde{y}^{(l)}$ $\displaystyle=r^{(l)}\ast y^{(l)}$ (8)

Conventional fine-tuning with dropout has $p$ as a constant. Given that we
have an ordered collection of layers
$S=\\{I(l_{1})<I(l_{2})<...<I(l_{|L|})\\}$ sorted based on their Fisher scores
$I(l)$, we propose an incremental setting of layer-wise $p^{(l)}$ such that
the masking of $y^{(l)}$ through $Bernoulli(p^{(l)})$ follows a layer-wise
masking schedule. We define our Bernoulli sampling probability $p^{(l_{i})}$
for the layer $l$ at the $i^{th}$ position in $S$ as:

$\displaystyle p^{(l_{i})}$ $\displaystyle=\sum_{j=1}^{i}\Delta s$ (9)
$\displaystyle r^{(l_{i})}$ $\displaystyle\sim\mathrm{Bernoulli}(p^{(l_{i})})$
(10)

where $\Delta s=\frac{P_{upper}-P_{lower}}{n_{d}}$, the step size followed by
the _linear increment schedule_. Here, $P_{upper}\in(0,1)$ and
$P_{lower}\in(0,1)$ are constants that represent the upper and lower bounds
for the value that $p^{(l_{i})}$ can take and $n_{d}$ represents the number of
dropout layers in the LM architecture. Our approach amounts to sampling sub-
networks from a larger network where the guidance imposes a sampling bias
towards retaining highly informative neurons in the sub-network. This guided
sampling bias leads to improved generalization by learning through a better
behaved loss landscape.

The determination of our layer-wise masking schedule is a one-time task. After
$p^{(l_{i})}$ is determined for a model, the same schedule yields improvements
in generalization for all downstream tasks for the model (as we demonstrate in
section §3). Thus, not restricted by specific model architectures nor specific
downstream tasks, our guided dropout is both task and architecture agnostic.
Further, as the the layer-wise dropout probabilities $p^{(l_{i})}$ are
predecided ahead of fine-tuning, guided dropout adds no computational overhead
to the fine-tuning process. The $p^{(l_{i})}$ implementation used for this
study is linear, however, users are free to utilize their own non-linear
schedules. We empirically evaluate the efficacy of our approach in the section
that follows.

##  3 Fine-tuning LMs with Information Guided Regularization

Phang et al. (2018) state that fine-tuning $BERT_{LARGE}$ sometimes fails when
a target dataset has fewer than 10,000 training instances. This was specially
apparent on a subset of the GLUE dataset (Wang et al., 2018) that have
relatively low data samples - CoLA, MRPC, RTE, and STS-B. Intuitively, this
would make sense as an overparameterized model would fail to converge on a
small dataset. Lee et al. (2019) evaluate their regularization technique by
fine-tuning $BERT_{LARGE}$ on these 4 tasks and showcase a high rate of
failure among random restarts222Based on Lee et al. (2019), random restarts
refer to using a different train-test split of the dataset for each fine-
tuning run. for standard dropout - specially for the CoLA task where 7 out 20
of their runs fail to converge. In our attempts to replicate this, using the
exact same specifications and the original $BERT_{LARGE}$ model, we found that
all 30/30 of our random restarts for CoLA converged to a respectable Matthew’s
correlation score (see appendix §A.5 for the full table). The only
differentiating factor between our experiment and theirs was the batch-size,
our 8 vs their 32. There have been numerous investigations into how large
batch sizes lead to generalization failures (Keskar et al., 2016; Jastrzebski
et al., 2017).

However, we do borrow the idea to simulate model performance under data
paucity. Instead of having an over-parameterized model $BERT_{LARGE}$
evaluated on smaller GLUE datasets, we evaluate $BERT_{BASE}$ on progressively
smaller cuts of those same four datasets - CoLA, MRPC, RTE, and STS-B. We
progressively shrink the size of the training dataset from 100% to 10% and
evaluate our regularizer with standardized baselines all the way through. To
establish consistency, we report results as averages across 5 random restart
runs along with the corresponding inter-quartile ranges of the runs. Figure 4
showcases the performance of guided dropout (our regularizer) against two
popular contemporary choices for regularization for the 4 GLUE tasks.

Baselines: The standard dropout regularizer (Srivastava et al., 2014) follows
the masking of the input vector as shown in equation (6). Gaussian dropout
(Wang & Manning, 2013) follows a similar idea where instead of zero-ing out
parameters in the input vector, it injects Gaussian noise into the vector. The
noise $\epsilon$ draws from the Gaussian distribution $\mathcal{N}(1,\alpha)$
where $\alpha=\frac{p}{1-p}$ with $p$ as the dropout probability. We choose
$p$ = 10% as Lee et al. (2019) report Dropout(0.1) to have be optimal for all
GLUE tasks.

Experimental Setup: For fair comparisons to the baselines, we use the same
training hyperparameters as those recommended by Devlin et al. (2019): an adam
optimizer with learning rate = 2 x $10^{-5}$, $\beta_{1}$ = 0.9, $\beta_{2}$ =
0.999, learning rate warm-up over the first 10% of steps and a linear decay of
the learning rate after the warm-up steps. For each GLUE task, the exact
train-test split is decided by a random seed and the output layer of
$BERT_{BASE}$ for the task is randomly initialized from a distribution
$\mathcal{N}(0,0.02^{2})$. The fine-tuning is performed for 3 epochs for all
tasks and models and the results are evaluated on the development set.

Figure 4: The performance of Guided dropout (our regularizer) vs standardized baselines on fine-tuning $BERT_{BASE}$ on decreasing cuts of the training datasets for (a) MRPC (b) STS-B (c) RTE and (d) CoLA. Each data point in the grid is an average across 5 random restarts, as shown by the boxplots. Guided dropout yields consistently better results for all the tasks, specially under data paucity. Regularizer | CoLA | STS-B | MNLI | MNLI (MM) | MRPC
---|---|---|---|---|---
Dropout (10%) | 0.56170.5446 | 0.86050.8572 | 84.5284.37 | 84.9584.47 | 80.4579.52
Gaussian Dropout (10%) | 0.57820.5585 | 0.86300.8599 | 84.7884.26 | 85.3184.84 | 81.1180.01
Guided Dropout (10%) | 0.60960.6045 | 0.88130.8743 | 85.1184.42 | 85.1484.81 | 82.4181.44
Regularizer | QNLI | QQP | RTE | SST-2 | WNLI
Dropout (10%) | 89.5989.50 | 89.5389.47 | 64.8062.92 | 94.9894.93 | 45.0743.40
Gaussian Dropout (10%) | 89.0888.90 | 89.4589.39 | 66.7864.25 | 95.1094.97 | 44.3643.66
Guided Dropout (10%) | 89.7189.64 | 89.6289.58 | 68.7763.03 | 95.2395.23 | 50.7045.42
Table 2: The performance of guided dropout (our regularizer) vs standard and
Gaussian dropouts at $p=10\%$ on GLUE tasks. For consistency, the numbers are
presented as $Max_{Mean}$ of results across 5 random restart runs. For CoLA,
the metric presented is the Matthew’s correlation coefficient and that for
STS-B is an average of the Pearson’s and Spearman-r coefficient. For all other
tasks, the figures presented are averages of the accuracy and f1 scores. Our
regularizer outperforms both baselines on 9/10 tasks.

GLUE Tasks: Although we evaluate the performance of our model vs the baselines
under simulated data-paucity for the select 4 GLUE tasks (following Phang et
al. (2018) and Lee et al. (2019), Figure 4), we still perform the standard
evaluation our models with respect to the baselines for all 9 tasks in the
GLUE benchmark (see Table 2):

  * •

Single Sentence Tasks: CoLA (the Corpus of Linguistic Acceptability) (Warstadt
et al., 2019) for grammatical fidelity, SST-2 (the Stanford Sentiment
Treebank) (Socher et al., 2013) for sentiment prediction.

  * •

Similarity and Paraphrase Tasks: MRPC (the Microsoft Research Paraphrase
Corpus) (Dolan & Brockett, 2005), QQP (the Quora Question Pairs), and STS-B
(the Semantic Textual Similarity Benchmark) (Cer et al., 2017) for semantic
equivalence.

  * •

Inference Tasks: MNLI (the Multi-Genre Natural Language Inference Corpus)
(Williams et al., 2018) and RTE (Recognizing Textual Entailment) for textual
entailment, QNLI (the Stanford Question Answering Dataset) (Rajpurkar et al.,
2016) for Q&A, and WNLI (the Winograd Schema Challenge) (Levesque et al.,
2012) for pronoun referent selection.

##  4 Conclusion

Fisher information is a useful tool for characterizing the properties of
stochastic gradient descent (Liao et al., 2018; Martens, 2020). We extend its
implementation to further understand the training dynamics of LMs through the
visual geometry of their loss landscape. Upon observing the pronounced effect
of a select few sets of parameters on the LM loss geometry, we devised a
surgical approach to L2 regularization that dissuades the sway from optimal
model convergence, leading to better generalization in downstream tasks. We
term this regularization as information-guided regularization. Applied to
dropout, this regularization can be comprehended as sampling sub-networks from
a larger network where the guidance imposes a sampling bias towards retaining
highly informative neurons in the sub-network. This guided sampling bias leads
to improved generalization by learning through a better behaved loss
landscape. Although surgical, each pretrained model requires this estimate to
be computed only once - the same estimate is then applied when fine-tuning to
any downstream task. Thus, making our approach both task & model agnostic with
no computational overhead added to the fine-tuning process. We experimentally
validate the effectiveness of our regularizer and showcase its prowess,
specially in scenarios of data paucity. We also showcase how for LMs, the
Fisher information can be frugally approximated with a tiny fraction of the
training corpus.

##  5 Limitations

In our study, we extend the use of Fisher information to further understand
the training dynamics of LMs through the visual geometry of their loss
landscape. From our findings we devise a regularization technique which we
showcase to be effective for a multitude of downstream tasks. Due to the
exhaustive nature of our experiments, replicating this study for multiple
variants of transformer-based LMs was not viable given our in-house
computational resources, thus we chose the most popular LM - BERT (Wolf et
al., 2019) for our empirical evaluations. However, in section §2, we showcase
that the theoretical motivations that make this applicable for the encoder-
only BERT model (Devlin et al., 2019) also holds true for the decoder-only
GPT2 model (Radford et al., ) and the encoder-decoder based model T5 (Raffel
et al., 2020). Thus, we believe our regularizer to be universally applicable.

##  6 Ethics Statement

While many LLM applications are fraught with ethical concerns, we can
ascertain that our study does not bring forth additional complications. The
datasets we use in this study are established benchmark datasets from publicly
accessible websites and do not contain any personally identifiable
information. Our analyses does not constitute human subjects and thus is not
within the purview of the IRB. A silver lining from our work is that, by
adopting regularization techniques as proposed here, we can achieve better
downstream generalization without added computational costs.

## References

  * Bottou et al. (2018) Léon Bottou, Frank E Curtis, and Jorge Nocedal.  Optimization methods for large-scale machine learning.  _SIAM review_ , 60(2):223–311, 2018.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , 2017.
  * Dalvi et al. (2020) Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov.  Analyzing redundancy in pretrained transformer models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4908–4926, 2020.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, 2019.
  * Dolan & Brockett (2005) Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Ge et al. (2015) Rong Ge, Furong Huang, Chi Jin, and Yang Yuan.  Escaping from saddle points—online stochastic gradient for tensor decomposition.  In _Conference on Learning Theory_ , pp. 797–842. PMLR, 2015.
  * Hardt et al. (2016) Moritz Hardt, Ben Recht, and Yoram Singer.  Train faster, generalize better: Stability of stochastic gradient descent.  In _International Conference on Machine Learning_ , pp. 1225–1234. PMLR, 2016.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Flat minima.  _Neural computation_ , 9(1):1–42, 1997.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Jastrzebski et al. (2017) Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey.  Three factors influencing minima in sgd.  _arXiv preprint arXiv:1711.04623_ , 2017.
  * Keskar et al. (2016) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _International Conference on Learning Representations_ , 2016.
  * Lee et al. (2019) Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang.  Mixout: Effective regularization to finetune large-scale pretrained language models.  In _International Conference on Learning Representations_ , 2019.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_ , 2012.
  * Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  _Advances in neural information processing systems_ , 31, 2018.
  * Liao et al. (2018) Zhibin Liao, Tom Drummond, Ian Reid, and Gustavo Carneiro.  Approximate fisher information matrix to characterize the training of deep neural networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(1):15–26, 2018.
  * Martens (2020) James Martens.  New insights and perspectives on the natural gradient method.  _Journal of Machine Learning Research_ , 21(146):1–76, 2020.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_ , 2016.
  * Min et al. (2023) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth.  Recent advances in natural language processing via large pre-trained language models: A survey.  _ACM Computing Surveys_ , 56(2):1–40, 2023.
  * OpenAI (2023) OpenAI.  Chatgpt, 2023.  URL https://chat.openai.com.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_ , 2018.
  * (21) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2016.
  * Sharma et al. (2022) Mandar Sharma, Ajay Gogineni, and Naren Ramakrishnan.  Innovations in neural data-to-text generation: A survey.  _arXiv preprint arXiv:2207.12571_ , 2022.
  * Sharma et al. (2023) Mandar Sharma, Nikhil Muralidhar, and Naren Ramakrishnan.  Learning non-linguistic skills without sacrificing linguistic proficiency.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 6178–6191, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.340.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.  Regularization of neural networks using dropconnect.  In _International Conference on Machine Learning_ , pp. 1058–1066. PMLR, 2013.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 353–355, 2018.
  * Wang & Manning (2013) Sida Wang and Christopher Manning.  Fast dropout training.  In _International Conference on Machine Learning_ , pp. 118–126. PMLR, 2013.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 2019.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018.
  * Wolf (2024) Thomas Wolf.  Did you know bert-based-uncased has been downloaded 1.5 billion times on the hugging face hub – top ai models are hitting youtube viral videos number ranges.  https://x.com/Thom_Wolf/status/1801536742211682542, 2024.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  2019\.
  * Yao et al. (2020) Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.  Pyhessian: Neural networks through the lens of the hessian.  In _2020 IEEE international conference on big data (Big data)_ , pp. 581–590. IEEE, 2020.

##  Appendix A Appendix

###  A.1 LM Parameters based on their Fisher Scores - Sorted

Figure 5: When LM parameters are sorted based on their increasing Fisher
scores, we observe a that only an exceedingly tiny fraction of the model
parameters have significantly high scores. This observation is consistent for
all popular transformer architectures: encoder only (BERT with 0.001% of
parameters having $\geq$ 0.01 score), decoder only (GPT2 with 0.02% of
parameters having $\geq$ 0.01 score), and encoder-decoder models (T5 with
0.0009% of parameters having $\geq$ 0.01 score).

###  A.2 LM Layers based on their Fisher Scores - Sorted

Figure 6: Similar to our observations in §A.1, when the layers of LM are
sorted based on the aggregate Fisher score of their respective layer
parameters, we observe that certain layers tend to have a concentration of
parameters with high Fisher scores. Again, this observation holds true for all
popular transformer architectures: (a) encoder only BERT, (b) decoder only
GPT2, and (c) encoder-decoder based T5.

###  A.3 LM Layers based on their Fisher Scores - Unsorted

Figure 7: Complementary to figure 6, while only certain layers of the model
have significantly high scores, these vital layers are spread throughout the
network and not concentrated towards the beginning or end of the network as
traditionally believed.

###  A.4 Algorithmic Representation of Information Guided Fine-tuning

###  A.5 Fine-tuning BERT Large on CoLA

Restart Seed | Matthews Correlation | Restart Seed | Matthews Correlation
---|---|---|---
1 | 0.6357 | 16 | 0.6074
2 | 0.6164 | 17 | 0.6657
3 | 0.6127 | 18 | 0.6186
4 | 0.6412 | 19 | 0.6437
5 | 0.6060 | 20 | 0.6280
6 | 0.6730 | 21 | 0.6266
7 | 0.6482 | 22 | 0.5982
8 | 0.6206 | 23 | 0.6567
9 | 0.6475 | 24 | 0.6205
10 | 0.6555 | 25 | 0.6390
11 | 0.6357 | 26 | 0.6378
12 | 0.6348 | 27 | 0.6461
13 | 0.6710 | 28 | 0.6275
14 | 0.6470 | 29 | 0.6759
15 | 0.6396 | 30 | 0.6394
Table 3: The seed values and the respective Matthew’s Correlation score for 30
random restart runs of $BERT_{LARGE}$ fine-tuned on CoLA.