# A Technical Critique of Some Parts of the Free Energy Principle

Martin Biehl1111These authors contributed equally to this work. Felix A.
Pollock211footnotemark: 1 Ryota Kanai1

(1Araya Inc., Tokyo 105-0003, Japan
2School of Physics and Astronomy, Monash University, Clayton, Victoria 3800,
Australia
)

###### Abstract

We summarize the original formulation of the free energy principle, and
highlight some technical issues. We discuss how these issues affect related
results involving generalised coordinates and, where appropriate, mention
consequences for and reveal, up to now unacknowledged, differences to newer
formulations of the free energy principle. In particular, we reveal that
various definitions of the “Markov blanket” proposed in different works are
not equivalent. We show that crucial steps in the free energy argument which
involve rewriting the equations of motion of systems with Markov blankets, are
not generally correct without additional (previously unstated) assumptions. We
prove by counterexample that the original free energy lemma, when taken at
face value, is wrong. We show further that this free energy lemma, when it
does hold, implies equality of variational density and ergodic conditional
density. The interpretation in terms of Bayesian inference hinges on this
point, and we hence conclude that it is not sufficiently justified.
Additionally, we highlight that the variational densities presented in newer
formulations of the free energy principle and lemma are parameterised by
different variables than in older works, leading to a substantially different
interpretation of the theory. Note that we only highlight some specific
problems in the discussed publications. These problems do not rule out
conclusively that the general ideas behind the free energy principle are worth
pursuing.

## Overview

In [6] it is argued that the internal coordinates of an ergodic random
dynamical system with a Markov blanket necessarily appear to engage in active
Bayesian inference. Here, we reproduce the argument supporting this
interpretation in detail and highlight at which points it faces technical
issues. In the course of our critique, we also mention issues of some closely
related alternative arguments. In cases where our results have clear
consequences for the more recent related publications [7, 14] we also mention
those. In particular, we point out a conceptual difference in these latter
works that has not previously been acknowledged. However, our analysis thereof
does not go beyond a few remarks. In an additional section we discuss the
effect of our argument on [9]. The logical structure of the present paper is
depicted in Figure 1. We note that the technical issues presented here do not
affect the validity of approaches where an (expected) free energy minimizing
agent is assumed a priori, as presented in, e.g., [8]. None of [6, 9, 7, 14]
make this assumption, they instead aim to identify the conditions under which
such agents will emerge within a given stochastic process. We criticize
specific formal issues in the latter publications but leave open whether they
can be fixed. We now briefly introduce the setting of [6] and then sketch the
content of this paper.

Equation (1), ergodicity,Conditions 1 and 2Condition 2Condition 1Condition
3$\begin{pmatrix}f_{\psi}(\psi,s)\\\ f_{s}(\psi,s,a)\\\ f_{a}(s,a,\lambda)\\\
f_{\lambda}(a,\lambda)\end{pmatrix}$Gaussian and Markov $\omega$Equation
(4)Equations (19) and (20)Equations (21) and (22)Equations (17) and
(18)$\exists q(\Psi|\lambda)$ s.t. Equations (29) and (30)$\exists
q(\Psi|\lambda)$ s.t. Equations (37) and (38)$\exists q(\Psi|\lambda)$ s.t.
Equations (39) and (40)Equations (35) and (36)Equation (42)Interpretation as
Bayesian inferenceSection 1Section 2Section 2[7]Section 2, generalSection
3Section 4Section 5Section 6 [14] Figure 1: Argument visualization. Numbers
labelling edges indicate corresponding steps in this paper. Struck out edges
indicate implications that we prove incorrect. The main argument in [6] takes
the left path. The box in the top right indicates the relations between
Conditions 1, 2 and 3 and their role in [14]. Merged edges indicate a logical
AND combination of the parent nodes.

The starting point is a random dynamical system whose evolution is governed by
the stochastic differential equation

$\displaystyle\dot{x}=f(x)+\omega,$ (1)

where the system state $x$ and vector field $f(x)$ are multi-dimensional, and
$\omega$ is a Gaussian noise term. There is an additional assumption that the
system is ergodic, such that the steady state probability density $p^{*}(x)$
is well defined. (In the original paper, the ergodic density is simply denoted
$p(x)$. We here add a star to highlight that it is a time independent
probability density.) In this case, $-\ln p^{*}(x)$ plays the role of a
potential function, in the sense that $f$ can be formulated in terms of its
gradients [1, 11].

It is then assumed that there is a coordinate system $x=(\psi,s,a,\lambda)$
with $\psi=(\psi_{1},...,\psi_{n_{\psi}})$, $s=(s_{1},...,s_{n_{s}})$,
$a=(a_{1},...,a_{n_{a}})$, and
$\lambda=(\lambda_{1},...,\lambda_{n_{\lambda}})$, referred to as external,
sensory, active, and internal coordinates (these are called “states” in [6])
respectively, such that the following condition holds:

######  Condition 1.

The function $f(x)$ can be written as

$\displaystyle\begin{split}f(x)&=\begin{pmatrix}f_{\psi}(\psi,s,a)\\\
f_{s}(\psi,s,a)\\\ f_{a}(s,a,\lambda)\\\
f_{\lambda}(s,a,\lambda)\end{pmatrix}.\end{split}$ (2)

This particular structure is described as “[formalizing] the dependencies
implied by the Markov blanket” [6]. In contrast, more recent works [7, 14]
formulate the Markov blanket in terms of statistical dependencies of the
ergodic density $p^{*}(x)=p^{*}(\psi,s,a,\lambda)$. Specifically, the
following condition is presented:

######  Condition 2.

The ergodic density factorises as

$\displaystyle
p^{*}(\psi,s,a,\lambda)=p^{*}(\psi|s,a)p^{*}(\lambda|s,a)p^{*}(s,a).$ (3)

In other words, the internal and external coordinates are independently
distributed when conditioned on the sensory and active coordinates. This means
we have two different formal expressions of what constitutes a Markov blanket
in these publications, and their relationship has not previously been
established.

Taking Condition 1 to hold, the argument of [6] then proceeds along the
following steps:

Section 1

Rewrite the vector field $f(\psi,s,a,\lambda)$ describing the dynamics of the
system in terms of the gradient of negative logarithm of the ergodic density
$p^{*}(\psi,s,a,\lambda)$ of that system.

Section 2

Rewrite the components $f_{\lambda}(s,a,\lambda)$ and $f_{a}(s,a,\lambda)$ of
the vector field $f(\psi,s,a,\lambda)$ in terms of only partial gradients of
the negative logarithm of $p^{*}(\psi,s,a,\lambda)$.

Section 3

Assert (in the Free Energy Lemma) the existence of a density $q(\psi|\lambda)$
over the external coordinates $\psi$ parameterized by the internal coordinates
$\lambda$, and that $f(\psi,s,a,\lambda)$ can again be rewritten, this time in
terms of a free energy depending on $q(\Psi|\lambda)$. (Here, and whenever it
would otherwise be ambiguous, we use a capitalized $\Psi$ to indicate full
distributions, rather than the probability density for specific value of
$\psi$.)

Section 4

Claim that equivalence of the equations of motion in Section 2 and Section 3
implies that certain partial gradients of the KL divergence between
$q(\Psi|\lambda)$ and the conditional ergodic density
$p^{*}(\Psi|s,a,\lambda)$ must vanish.

Section 5

Claim that it follows from Section 4 that $q(\Psi|\lambda)$ and
$p^{*}(\Psi|s,a,\lambda)$ are “rendered” equal.

Section 6

Interpret

  * •

$p^{*}(\Psi|s,a,\lambda)$ as a posterior over external coordinates given
particular values of sensor, active, and internal coordinates,

  * •

$q(\Psi|\lambda)$ as encoding Bayesian beliefs about the external coordinates
by the internal coordinates, and

  * •

their equality as the internal coordinates appearing to “solve the problem of
Bayesian inference”.

In the present paper, we make the following main observations

  * •

The re-expression of Equation (1) in the form chosen in Section 1 is derived
under restrictive assumptions, including that the system is subject to
Gaussian and Markov noise.

  * •

Condition 1 and Condition 2 are independent from each other.

  * •

Condition 1 and Condition 3 together lead to a system where the interpretation
of $s$ and $a$ as sensory and active coordinates is questionable.

  * •

Under both Conditions 1 and 2, the expressions of $f_{\lambda}(s,a,\lambda)$
and $f_{a}(s,a,\lambda)$ resulting from Section 2 are not as general as those
contained in the result of Section 1. The more general alternative expression
derived in [7] remains insufficiently general.

  * •

Under both Conditions 1 and 2, the Free Energy Lemma, when taken at face
value, is wrong and cannot be salvaged by using alternatives in Section 2.

  * •

Under both Conditions 1 and 2, contrary to Section 5 the vanishing of the
gradient of the KL divergence does not imply equality of $q(\Psi|\lambda)$ and
$p^{*}(\Psi|s,a,\lambda)$.

  * •

As a consequence, the basic preconditions for the interpretations in Section 6
are not implied by either of the two proposed Markov blanket Conditions 1 and
2.

The later [9] presents an argument almost identical to the one in the original
[6]. In Section 7 we discuss how our observations apply to this publication.

##  1 Expression via the Gradient of the Ergodic Density

Here we introduce the expression of the system’s dynamics Equation (1) in the
form used for the Free Energy Lemma (Lemma 2.1 in [6]). This form expresses
the dynamics of internal and active coordinates of the given ergodic random
dynamical system in terms of the gradient of the ergodic density $p^{*}(x)$.
In accordance with the results of [11], $f(x)$ is rewritten as (see Equation
(2.5) in [6]):

$\displaystyle f(x)$ $\displaystyle=(\Gamma+R)\cdot\nabla\ln p^{*}(x),$ (4)

where $\Gamma$ is the diffusion matrix, which we will take to be block
diagonal, (in [6], and later work such as [7], $\Gamma$ is taken to be
proportional to the identity matrix) and $R$ is an antisymmetric matrix,
defined through the relation

$\displaystyle MR+RM^{T}=M\Gamma-\Gamma M^{T},$ (5)

with

$\displaystyle M_{ij}=\nabla_{j}f_{i}(x).$ (6)

Here, and in all of [6, 9, 7, 14], both $\Gamma$ and $R$ are assumed constant.
We emphasise here that, for general nonlinear models, these matrices can vary
with the coordinates and Equation (5) holds only approximately [10, 12]. (The
exact conditions under which these matrices can be chosen to be constant can
be found in [12, 16] and, for the discrete state case, [2]). Moreover,
Equation (4) is derived in the literature under the explicit assumption that
the fluctuations $\omega$ be Gaussian and Markov [1, 11]. For the
counterexamples we present here, we restrict ourselves to the class of
Ornstein-Uhlenbeck processes, for which $R$ and $\Gamma$ are always constant,
and the ergodic density $p^{*}(x)=p^{*}(\psi,s,a,\lambda)$ is necessarily a
multivariate Gaussian with zero mean. Specifically, following [11],

$\displaystyle
p^{*}(\psi,s,a,\lambda):=\frac{1}{Z}\exp\left[{-\frac{1}{2}(\psi,s,a,\lambda)U(\psi,s,a,\lambda)^{\top}}\right],$
(7)

where $(\psi,s,a,\lambda)$ is a row vector and $Z$ is a suitable normalisation
constant. From Equation (4) it can be seen that,

$\displaystyle U=-(\Gamma+R)^{-1}M;$ (8)

though we emphasise here that strict relations between $M$ and $U$ can only be
made because of the assumption that $\Gamma$ and $R$ are coordinate
independent [17]. This concludes Section 1.

Before moving on to Section 2, we note that, under the assumptions implicit in
Section 1, we can express Condition 1 and Condition 2 in terms of the matrices
$M$ and $U$. (In the nonlinear case, these matrices can still be defined in
terms of derivatives of the force vector field and potential, respectively;
however, they will be generally coordinate-dependent, even when $\Gamma$ and
$R$ are not [10].) Firstly, since it effectively states that
$\nabla_{\psi}f_{a}(x)=\nabla_{\psi}f_{\lambda}(x)=\nabla_{\lambda}f_{s}(x)=\nabla_{\lambda}f_{\psi}(x)=0$,

$\displaystyle{\rm\lx@cref{creftype~refnum}{con:flow}}\;\;\Leftrightarrow\;\;M_{a\psi}=M_{\lambda\psi}=M_{s\lambda}=M_{\psi\lambda}=0,$
(9)

with $M_{\alpha\beta}$ a block sub-matrix of $M$ in general. Secondly, because
of the multivariate Gaussian nature of $p^{*}(\psi,s,a,\lambda)$, the
dependencies of conditional distributions are encoded in the inverse $U$ of
the covariance matrix; we therefore have that

$\displaystyle{\rm\lx@cref{creftype~refnum}{con:fact}}\;\;\Leftrightarrow\;\;U_{\psi\lambda}=U_{\lambda\psi}=0,$
(10)

where $U_{\alpha\beta}$ is a block sub-matrix of $U$. These implications bring
us to our first observation:

######  Observation 1.

Neither one of Condition 1 (the vector field dependency structure) or
Condition 2 (conditional independence in the ergodic distribution) implies the
other:

$\displaystyle{\rm\lx@cref{creftype~refnum}{con:flow}}\;\;\nRightarrow\;\;{\rm\lx@cref{creftype~refnum}{con:fact}}$
(11)
$\displaystyle{\rm\lx@cref{creftype~refnum}{con:flow}}\;\;\nLeftarrow\;\;{\rm\lx@cref{creftype~refnum}{con:fact}}.$
(12)

###### Proof.

In Appendix A, we provide direct counterexamples, using the equivalent
constraints on the matrices $M$ and $U$ in Equations (9) and (10), to
implication in either direction. That is, there exists a system obeying
Condition 1 that does not obey Condition 2 (proving Equation (11)), and there
exists one obeying Condition 2 that does not obey Condition 1 (proving
Equation (12)). ∎

Henceforth, unless otherwise stated, we will assume both Condition 1 and
Condition 2. Any implications that fail to hold in this special case cannot
hold generally.

##  2 Re-Expression Using only Partial Gradients

For Section 2 we focus on the components
$f_{\lambda}=(f_{\lambda_{1}},...,f_{n_{\lambda}})$ and
$f_{a}=(f_{a_{1}},...,f_{n_{a}})$ of $f$. Without loss of generality we can
rewrite them from Equation (4) as:

$\displaystyle f_{a}(s,a,\lambda)=$
$\displaystyle\left(R_{a\psi}\cdot\nabla_{\psi}+R_{as}\cdot\nabla_{s}+(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}\right.$
$\displaystyle\;\left.+R_{a\lambda}\cdot\nabla_{\lambda}\right)\ln
p^{*}(\psi,s,a,\lambda),$ (13) $\displaystyle f_{\lambda}(s,a,\lambda)=$
$\displaystyle\left(R_{\lambda\psi}\cdot\nabla_{\psi}+R_{\lambda
s}\cdot\nabla_{s}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right.$
$\displaystyle\;\left.+R_{\lambda a}\cdot\nabla_{a}\right)\ln
p^{*}(\psi,s,a,\lambda),$ (14)

where $\Gamma_{nm}$ ($R_{nm}$) is the block of $\Gamma$ ($R$) connecting
derivatives with respect to the $m$ coordinates to the time derivatives of the
$n$ coordinates. The expectation value with respect to
$p^{*}(\psi|s,a,\lambda)$ leaves the left hand side of these equations
unchanged. A few manipulations ([7] cf. Equation (12.14), p. 129) reveal that,
on the right hand side, this leads to the ergodic density
$p^{*}(\psi,s,a,\lambda)$ being replaced by the marginalised ergodic density
$p^{*}(s,a,\lambda)$ so that we get

$\displaystyle f_{a}(s,a,\lambda)=$
$\displaystyle\left(R_{a\psi}\cdot\nabla_{\psi}+R_{as}\cdot\nabla_{s}+(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}\right.$
$\displaystyle\;\left.+R_{a\lambda}\cdot\nabla_{\lambda}\right)\ln
p^{*}(s,a,\lambda)$ (15) $\displaystyle f_{\lambda}(s,a,\lambda)=$
$\displaystyle\left(R_{\lambda\psi}\cdot\nabla_{\psi}+R_{\lambda
s}\cdot\nabla_{s}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right.$
$\displaystyle\;\left.+R_{\lambda a}\cdot\nabla_{a}\right)\ln
p^{*}(s,a,\lambda).$ (16)

Since $\nabla_{\psi}\ln p^{*}(s,a,\lambda)=0$, the terms involving
$\nabla_{\psi}$ drop out:

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=\left(R_{as}\cdot\nabla_{s}+(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)\ln
p^{*}(s,a,\lambda),$ (17) $\displaystyle f_{\lambda}(s,a,\lambda)$
$\displaystyle=\left(R_{\lambda s}\cdot\nabla_{s}+R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)\ln
p^{*}(s,a,\lambda).$ (18)

We are not aware of how to further simplify this equation without additional
assumptions. However, in (Equations (2.5) and (2.6) of [6] ) all of the off-
diagonal terms are implicitly assumed to vanish, i.e., Equation (4) is equated
with:

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}\ln p^{*}(s,a,\lambda),$
(19) $\displaystyle f_{\lambda}(s,a,\lambda)$
$\displaystyle=(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\ln
p^{*}(s,a,\lambda).$ (20)

This equation is the result of Section 2.

In the more recent (Appendix B of [7]) a more detailed discussion of Equation
(4) is presented, where it is claimed that Condition 1 implies Condition 2
(cf. our Observation 1) along with the following simplification of Equations
(17) and (18) ([7], Equations (12.8)–(12.11), (12.15), pp. 126–129):

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=\left((\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)\ln
p^{*}(s,a,\lambda),$ (21) $\displaystyle f_{\lambda}(s,a,\lambda)$
$\displaystyle=\left(R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)\ln
p^{*}(s,a,\lambda).$ (22)

However, Equations (21) and (22) are still provably less general than
Equations (13) and (14), even when both Condition 1 and Condition 2 are
satisfied.

######  Observation 2.

Given a random dynamical system obeying Equation (1), ergodicity, and both
Condition 1 and Condition 2, none of Equations (19)–(22) generally hold.

###### Proof.

By counterexample, see Appendix B. There, we show explicitly that a model
satisfying the above assumptions does not satisfy the equations in question. ∎

In order to arrive at Equations (21) and (22) from Equations (17) and (18) in
general, one must remove the offending “solenoidal flow” terms by fiat. That
is, one assumes $R_{as}=R_{\lambda s}=0$. In ([7], Equation (12.4)), the
following, even stronger, condition is assumed as an alternative starting
point (along with Condition 2):

######  Condition 3.

The blocks of the $R$ matrix appearing in Equation (4) coupling $(s,a)$
coordinates to $\lambda$ and $\psi$ coordinates, and $\psi$ coordinates to
$\lambda$ coordinates vanish, i.e.,

$\displaystyle R_{\psi s}=R_{\psi
a}=R_{\psi\lambda}=R_{s\lambda}=R_{a\lambda}=0.$ (23)

This is claimed to imply $M_{\psi\lambda}=M_{\lambda\psi}=0$, but not the full
Condition 1. However, in [14], both Condition 1 and Condition 3 are assumed
(along with $R_{as}=0$). This prompts our next observation.

######  Observation 3.

In a system satisfying both Condition 1 and Condition 3, the internal
coordinates cannot be directly influenced by the sensory coordinates:
$f_{\lambda}(s,a,\lambda)=f_{\lambda}(a,\lambda)$, and the external
coordinates cannot be directly influenced by the active coordinates:
$f_{\psi}(\psi,s,a)=f_{\psi}(\psi,s)$.

###### Proof.

From Equation (5), it follows that

$\displaystyle M=(\Gamma+R)M^{T}(\Gamma-R)^{-1},$ (24)

with the inverse replaced by a pseudoinverse if $\Gamma-R$ is not invertible.
Therefore, if $\Gamma_{\alpha\beta}=\delta_{\alpha\beta}\Gamma_{\alpha\alpha}$
and $R_{\alpha\beta}=\delta_{\alpha\beta}R_{\alpha\alpha}$ for blocks of
coordinates labelled by $\alpha$ and $\beta$, then

$\displaystyle
M_{\alpha\beta}=(\Gamma_{\alpha\alpha}+R_{\alpha\alpha})M^{T}_{\beta\alpha}(\Gamma_{\beta\beta}-R_{\beta\beta})^{-1},$
(25)

and $M_{\beta\alpha}=0\Rightarrow M_{\alpha\beta}=0$.

Condition 3 implies that the only nonzero blocks of $R$ are $R_{\psi\psi}$,
$R_{ss}$, $R_{sa}$, $R_{as}$, $R_{aa}$, and $R_{\lambda\lambda}$, and $\Gamma$
is assumed to be block diagonal. As noted in Equation (9), Condition 1
requires that $M_{a\psi}=M_{\lambda\psi}=M_{s\lambda}=M_{\psi\lambda}=0$.
Through Equation (25), these together imply that $M_{\lambda s}=M_{\psi a}=0$,
and hence that

$\displaystyle\begin{split}f(x)&=\begin{pmatrix}f_{\psi}(\psi,s)\\\
f_{s}(\psi,s,a)\\\ f_{a}(s,a,\lambda)\\\
f_{\lambda}(a,\lambda)\end{pmatrix},\end{split}$ (26)

as was to be shown. ∎

In this case, the four sets of coordinates interact in a chain, and it is
questionable whether the $s$ and $a$ coordinates can be meaningfully
interpreted, respectively, as sensory inputs to the internal coordinates or
their boundary-mediated influence on the external coordinates.

##  3 Free Energy Lemma

The relation of the dynamics of the internal coordinates to Bayesian beliefs
is made by introducing a density (called the variational density)
$q(\Psi|\lambda)$ that is then interpreted as encoding a Bayesian belief. It
is parameterized by the internal coordinates $\lambda$ and claimed to be
“arbitrary”. We take this “at face value” and consider $q(\Psi|\lambda)$ to be
parameterized _only_ by $\lambda$ and, therefore, to be independent of
$(s,a)$. (We note that there is a convention in the literature on variational
Bayesian inference e.g., in [3] to drop the observed variables/data in the
variational density. It is possible that in [6], $(s,a)$ is seen as observed
variables and dropped from the variational density $q(\Psi|\lambda)$ as in
this convention. However, the reason that dropping the observed variables is
justified in the established convention is that those observed variables are
fixed throughout the minimization of the variational free energy and the
parameters of the variational density do not influence the observed data in
any way. In other words the variational density is optimized for a single
datapoint. In [6] the datapoint is continuously changing and partially doing
so in dependence on the parameter $\lambda$ as $\dot{a}=f_{a}(s,a,\lambda)$.
These differences and their consequences are non-trivial and beyond the scope
of this paper so we assume that the variational density does not depend on
$(s,a)$.) If $q(\Psi|\lambda)$ is allowed to depend on $(s,a)$, Observation 4
does not apply and the Free Energy Lemma is made trivially true by setting
$q(\psi|s,a,\lambda):=p^{*}(\psi|s,a,\lambda)$. The existence of the
variational density $q(\Psi|\lambda)$ is asserted by the Free Energy Lemma
(see Lemma 2.1 in [6]). (Explicitly, the Free Energy Lemma asserts the
existence of a free energy $F(s,a,\lambda)$ in terms of which
$f(\psi,s,a,\lambda)$ can be expressed and not the existence of
$q(\Psi|\lambda)$. However, since the free energy is defined as a functional
of $q(\Psi|\lambda)$, it exists if and only if a suitable $q(\Psi|\lambda)$
exists.)

More precisely, the Free Energy Lemma (and Section 3) asserts that for every
ergodic density (equivalently as expressed in [6], for every Gibbs energy
$G(x):=-\ln p^{*}(\psi,s,a,\lambda)$) $p^{*}(\psi,s,a,\lambda)$ of a system
obeying Equations (19) and (20) there is a free energy $F(s,a,\lambda)$,
defined as

$\displaystyle F(s,a,\lambda):$ $\displaystyle=-\ln p^{*}(s,a,\lambda)+\int
q(\psi|\lambda)\ln\frac{q(\psi|\lambda)}{p^{*}(\psi|s,a,\lambda)}\mathop{\kern
0.0pt\mathrm{d}}\\!{}\psi$ (27) $\displaystyle=-\ln
p^{*}(s,a,\lambda)+\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)],$
(28)

in terms of the “posterior density” $p^{*}(\Psi|s,a,\lambda)$, (here, we keep
the conditioning argument $\lambda$, as in [6], and do not explicitly assume
Condition 2, though our conclusions are unaffected by it) such that Equations
(19) and (20) can be rewritten as:

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=-(\Gamma+R)_{aa}\cdot\nabla_{a}F(s,a,\lambda),$ (29)
$\displaystyle f_{\lambda}(s,a,\lambda)$
$\displaystyle=-(\Gamma+R)_{\lambda\lambda}\cdot\nabla_{\lambda}F(s,a,\lambda).$
(30)

It is worth considering what a proof of the Free Energy Lemma could look like.
A proof of existence of a free energy (and therefore of the Free Energy Lemma)
would need to show that, for every system satisfying the given assumptions,
there always exists a $q(\Psi|\lambda)$ such that the right hand sides of
Equations (29) and (30) are equal to the right hand sides of Equations (19)
and (20). Expanding Equations (29) and (30) using (28) leads to:

$\displaystyle\begin{split}f_{a}(s,a,\lambda)&=(\Gamma+R)_{aa}\cdot\nabla_{a}\ln
p^{*}(s,a,\lambda)\\\
&\phantom{(\Gamma+R)_{aa}}-(\Gamma+R)_{aa}\cdot\nabla_{a}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)],\end{split}$
(31)
$\displaystyle\begin{split}f_{\lambda}(s,a,\lambda)&=(\Gamma+R)_{\lambda\lambda}\cdot\nabla_{\lambda}\ln
p^{*}(s,a,\lambda)\\\
&\phantom{(\Gamma+R)_{\lambda\lambda}}-(\Gamma+R)_{\lambda\lambda}\cdot\nabla_{\lambda}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)].\end{split}$
(32)

For equality of the right hand sides to those of Equations (19) and (20) we
need:

$\displaystyle(\Gamma+R)_{aa}\cdot\nabla_{a}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0$ (33)
$\displaystyle(\Gamma+R)_{\lambda\lambda}\cdot\nabla_{\lambda}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0.$ (34)

In words, these equations say that the Free Energy Lemma holds if any of the
following three conditions (of strictly increasing strengths) are given:

  1. 1.

There is a $q(\Psi|\lambda)$ such that the partial gradients $\nabla_{a}$ and
$\nabla_{\lambda}$ of the KL divergence between the variational density and
the conditional ergodic density are elements of the nullspaces of
$(\Gamma+R)_{aa}$ and $(\Gamma+R)_{\lambda\lambda}$ respectively.

  2. 2.

There is a $q(\Psi|\lambda)$ such that the gradients of the KL divergence to
$p^{*}(\Psi|s,a,\lambda)$ are equal to the nullvector:

$\displaystyle\nabla_{a}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]=0,$
(35)
$\displaystyle\nabla_{\lambda}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]=0,$
(36)

Then they are always elements of the nullspaces of $(\Gamma+R)_{aa}$ and
$(\Gamma+R)_{\lambda\lambda}$ respectively.

  3. 3.

There is a $q(\Psi|\lambda)$ such that
$q(\Psi|\lambda)=p^{*}(\Psi|s,a,\lambda)$ (and hence
$p^{*}(\Psi|s,a,\lambda)=p^{*}(\Psi|\lambda)$) which implies that the KL
divergence to $p^{*}(\Psi|s,a,\lambda)$ vanishes for all $a,\lambda$ and the
two partial gradients are always nullvectors and therefore elements of the
according nullspaces.

The Free Energy Lemma can then be proven by showing that one of these three
cases follows from the conditions of the lemma. However, no attempt is made in
[6] to establish this. Instead the given proof discusses purported
_consequences_ of the existence of a suitable $q(\Psi|\lambda)$. These will be
discussed in Sections 4 and 5.

Even if the Free Energy Lemma does not hold for systems obeying Equations (19)
and (20), one might expect that systems that instead only satisfy the more
general Equations (21) and (22) or the most general Equations (17) and (18).
For these systems the Free Energy Lemma would require that there is a
$q(\Psi|\lambda)$ such that

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=\left((\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)F(s,a,\lambda),$
(37) $\displaystyle f_{\lambda}(s,a,\lambda)$ $\displaystyle=\left(R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)F(s,a,\lambda).$
(38)

or

$\displaystyle f_{a}(s,a,\lambda)$
$\displaystyle=\left(R_{as}\cdot\nabla_{s}+(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)F(s,a,\lambda),$
(39) $\displaystyle f_{\lambda}(s,a,\lambda)$ $\displaystyle=\left(R_{\lambda
s}\cdot\nabla_{s}+R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)F(s,a,\lambda),$
(40)

hold respectively. However, we find this not to be the case in general.

######  Observation 4.

Given a random dynamical system obeying Equation (1), ergodicity, Condition 1
and Condition 2, there need not exist a free energy expressed in terms of a
variational density $q(\Psi|\lambda)$ such that:

(i)

Equations (29) and (30) hold if Equations (19) and (20) do;

(ii)

Equations (37) and (38) hold if Equations (19) and (20) don’t hold but
Equations (21) and (22) do;

(iii)

Equations (39) and (40) hold if neither Equations (19) and (20) norEquations
(21) and (22) hold but Equations (17) and (18) do.

###### Proof.

In Appendix C, we derive a set of conditions on the $R$ and $U$ matrices, and
on the putative variational density $q(\Psi|\lambda)$, that follow from each
of the pairs of equations in cases (i–iii). We show that, in general, each
pair leads to a contradiction and, in each case, provide a counterexample that
falls in the according system class. ∎

Before proceeding, we note that later works present an alternative version of
the Free Energy Lemma, where the conditioning argument of $q(\Psi|\lambda)$ is
replaced by the most likely value of $\lambda$ conditional on the $(s,a)$
coordinates [7, 14]. We here concern ourselves with the version apparent in
[6], where $q(\Psi|\lambda)$ is parameterised by the internal states
themselves, but will briefly comment on the interpretation of the alternative
approach in Section 6.

##  4 Vanishing Gradients

As mentioned in Section 3, the proof of the Free Energy Lemma in [6] only
discusses its consequences. The first proposed consequence is that expressing
the vector field in terms of a free energy as in Equations (29) and (30)
“requires” that the gradients with respect to $a$ and $\lambda$ of the KL
divergence vanish, i.e., that Equations (35) and (36) hold.

We mentioned in Section 3 that the implication in the opposite direction
holds. This can be seen from Equations (33) and (34). However, if the
nullspace of $(\Gamma+R)_{aa}$ or $(\Gamma+R)_{\lambda\lambda}$ is non-
trivial, then the gradient may be a non-zero element of this subspace and
Equations (29) and (30) will still hold. In that case the vanishing gradients
would not be necessary for the Free Energy Lemma.

The conditions under which a non-trivial nullspace exists are discussed in
[11]. In short, the nullspace is guaranteed to be trivial in the special case
where $\Gamma$ is positive definite. Whether or not ergodic systems with a
Markov blanket can ever admit a non-trivial nullspace, and hence divergences
in Equations (31) and (32) with non-vanishing gradients, is not immediately
clear. However, in order to establish the necessity of Equations (35) and (36)
this remains to be proven.

##  5 Equality of $Q(\Psi|\lambda)$ and $P^{*}(\Psi|s,a,\lambda)$

The proof of the Free Energy Lemma in [6] also proposes that the vanishing of
gradients of the KL divergence, of the variational density $q(\Psi|\lambda)$
from the conditional ergodic density $p^{*}(\Psi|s,a,\lambda)$, implies the
equality of these densities. We mentioned in Equations (4) that the
implication in the opposite direction holds. This can also be seen from
Equations (33) and (34). Concerning the implication in the direction proposed
by [6], let us now assume that for a given system Equations (19) and (20)
hold, a variational density $q(\Psi|\lambda)$ does exist, and the gradients of
the KL divergence of the variational and ergodic densities vanish i.e.,
Equations (35) and (36) hold. Then consider the argument by [6] in this direct
quote (comments in square brackets by us):

> “However, Equation (2.6) [Equations (19) and (20) above] requires the
> gradients of the divergence to be zero [Equations (35) and (36)], which
> means the divergence must be minimized with respect to internal states. This
> means that the variational and posterior densities must be equal:
>
> $\displaystyle
> q(\psi|\lambda)=p^{[*]}(\psi|s,a,\lambda)\Rightarrow\operatorname{D_{KL}}=0\Rightarrow\begin{cases}&(\Gamma+R)\cdot\nabla_{\lambda}\operatorname{D_{KL}}=0,\\\
> &(\Gamma+R)\cdot\nabla_{a}\operatorname{D_{KL}}=0.\end{cases}$
>
> In other words, the flow of internal and active states minimizes free
> energy, rendering the variational density equivalent to the posterior
> density over external states.”

The first problem in the above quote is that the minimization of the
divergence does not follow from the vanishing gradients. On the contrary,
since Equations (35) and (36) must hold for all $(s,a,\lambda)$, the KL
divergence

$\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$

cannot depend on $(\lambda,a)$; it therefore has no extremum (and thus no
minimum) with respect to either of these coordinates.

The second problem pertains to the identification of the two distributions at
a minimum. In general, if we try to find the minimum of a KL divergence
between a given probability density $p_{1}(Y)$ and a family of densities
$p_{2}(Y|\theta)$ parameterized by $\theta$, then the lowest possible value of
zero is achieved only if there is a parameter $\theta_{1}$ such that
$p_{2}(Y|\theta_{1})=p_{1}(Y)$. If there is no such $\theta_{1}$, then the
minimum value will be larger than zero. So, even if the divergence were
minimized, it would not need to be zero. More generally, the divergence $K(s)$
need not be zero for any value of $s$.

There is therefore no satisfactory reason given why the variational density
$q(\Psi|\lambda)$ and the posterior density $p^{*}(\Psi|s,a,\lambda)$ should
be equal or have low KL divergence. In fact they need not be. (Note that,
since any $q(\Psi|\lambda)$ that doesn’t depend on $(s,a)$ is an element of
the set of those that do, Observation 5 remains true for the case where we
allow this dependence. In that case, the Free Energy Lemma holds because we
can set $q(\Psi|s,a,\lambda):=p^{*}(\Psi|s,a,\lambda)$, and thus a $q$ exists
for which the densities are actually equal. However, the claim here is that
for _every_ $q$ that obeys the conditions in Observation 5 we must have
equality.)

######  Observation 5.

Given a random dynamical system obeying Equation (1), ergodicity, Condition 1
and Condition 2. Then if, additionally,

(i)

Equations (19) and (20) hold and the Free Energy Lemma holds i.e., there
exists a probability density $q(\Psi|\lambda)$ such that Equations (29) and
(30) hold, or

(ii)

Equations (21) and (22) hold and there exists $q(\Psi|\lambda)$ such that
Equations (37) and (38) hold, or

(iii)

Equations (17) and (18) hold and there exists $q(\Psi|\lambda)$ such that
Equations (39) and (40) hold,

then there is no $c\geq 0$ for which it can be guaranteed that

$\displaystyle\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]<c.$
(41)

In particular, it does not follow from these conditions that

$\displaystyle q(\Psi|\lambda)=p^{*}(\Psi|s,a,\lambda).$ (42)

###### Proof.

By example; see Appendix D. To show that the implication does not generally
hold for given system and densities $q(\Psi|\lambda)$ that obey Equations
(19), (20), (29) and (30), Equations (21), (22), (37) and (38), or Equations
(17), (18), (39) and (40) we only have to consider a system that obeys all
three pairs of equations, Equations (19) and (20), Equations (21) and (22),
and Equations (21) and (22), and for which suitable $q(\Psi|\lambda)$ exist.
For this system we then need to show that the $q(\Psi|\lambda)$ that obey
Equations (29) and (30) are not necessarily equal (or similar) to
$p^{*}(\Psi|s,a,\lambda)$.

We use a variant of the model used in Appendix B as such a counterexample.
This system obeys all three of Equations (19) and (20), Equations (21) and
(22), and Equations (21) and (22) and the nullspace of the associated
$\Gamma+R$ is trivial. We identify a set of possible $q(\Psi|\lambda)$
satisfying Equations (29) and (30) which implies that the _gradients of the KL
divergence between those $q(\Psi|\lambda)$ and $p^{*}(\Psi|s,a,\lambda)$_
vanish i.e., Equations (35) and (36) hold. We then demonstrate that for the
$q(\Psi|\lambda)$ in this set the _value of the KL divergence to
$p^{*}(\Psi|s,a,\lambda)$_ can be arbitrarily large. ∎

##  6 Interpretation

Finally, we turn our attention to the interpretation in terms of Bayesian
inference, i.e., Section 6. We again quote directly from [6]:

> Because (by Gibbs inequality) this divergence [
> $\operatorname{D_{KL}}[q(\psi|\lambda)||p^{*}(\psi|s,a,\lambda)]$] cannot be
> less than zero, the internal flow will appear to have minimized the
> divergence between the variational and posterior density. In other words,
> the internal states will appear to have solved the problem of Bayesian
> inference by encoding posterior beliefs about hidden (external) states,
> under a generative model provided by the Gibbs energy.

We have shown that, in general, there is no suitable variational density that
is only parameterized by the internal coordinate $\lambda$. We then showed
that, even if there is a suitable variational density (including those
parameterized by all of $(s,a,\lambda)$), it can be arbitrarily different from
the posterior density. Since the arguments for the internal flow appearing to
minimize the divergence between variational and posterior density are
therefore incorrect, there is no reason why the internal states should appear
to have solved the problem of Bayesian inference.

As mentioned in Section 3, some newer works (e.g., [7, 14]) formulate a
different Free Energy principle, where the variational density of beliefs is
parameterised not by the internal coordinates $\lambda$ but by
$\bar{\lambda}(s,a)=\operatorname*{arg\,max}_{\lambda}p^{*}(\lambda|s,a)$, the
most likely value of the internal coordinates given the sensory and active
ones. In this case, Observations 4 and 5 do not apply. However, the new
parameters $\bar{\lambda}(s,a)$ are strictly a function of the sensory and
active coordinates. This means we have a Markov chain (with capitalisations
indicating random variables associated to the corresponding lower case
coordinates (or functions of coordinates))
$\Lambda\rightarrow(S,A)\rightarrow\bar{\Lambda}$ and, by the data processing
inequality [5], the mutual information between the both sensory and active
coordinates and the belief parameter $\bar{\lambda}$ upper bounds that between
the internal coordinates and the belief parameter. It is therefore not clear
to what extent the internal coordinates $\lambda$, rather than the active and
sensory coordinates $(s,a)$ themselves, can be said to be encoding beliefs
about the external coordinates. Note also that, on any given trajectory,
unless the distribution $p^{*}(\lambda|s,a)$ is sufficiently peaked and
unimodal, the internal coordinates are not guaranteed to spend most of their
time close to their most likely conditional value, and (by definition if
Condition 2 holds), they will not be better predictors of the external
coordinates than those in the Markov blanket.

Generally, $\lambda\neq\bar{\lambda}$ and $\bar{\lambda}$ is the solution to
an optimization problem that is assumed to be solved in these later works.
Using this optimized variable to parameterise beliefs is therefore a
considerable departure from [6]. Contrary to the impression created by the way
it is referenced in [7, 14], the older theory in [6] should be clearly
distinguished from the newer ones in these more recent papers.

##  7 Consequences for [9]

The reference [9] argues for the same interpretation as [6] but there are some
differences in the argument.

The differences are the following:

  * •

In [9], Equation (1) is formulated for “generalized states,” which we refer to
here as generalized coordinates. This means that the variable $x$ is replaced
by a multidimensional variable denoted
${\tilde{x}}=(x,x^{\prime},x^{\prime\prime},...)$.

  * •

The Markov blanket structure is not explicitly defined via Equation (2).
Formally, it is introduced directly (see [9] Equation (10)) in a less general
form corresponding to Equations (19) and (20). (At the same time [6] is
referenced in connection to the Markov blanket so there seems to be no
intention to replace the original definition with the stronger one.)
Therefore, our observations concerning Sections 1, 2 and 3 are not directly
relevant to this paper.

  * •

The internal coordinate $\lambda$ is renamed to $r$ and the role of matrix $R$
is played by the matrix $-Q$.

  * •

The proof of the Free Energy Lemma given in [9] is different. It (implicitly)
suggests to set the variational density equal to the ergodic conditional
posterior.

  * •

The proof of the Free Energy Lemma no longer contains the proposition that the
gradient of the KL divergence of the variational density and the ergodic
conditional density vanish i.e., Section 4.

  * •

The proof also no longer contains the claim that the vanishing gradients of
the KL divergence of the variational density and the ergodic conditional
density imply equality of those densities i.e., Section 5 is not present.

The interpretation in terms of Bayesian inference is unchanged and still
relies on the equality of the variational and the ergodic conditional density.

Since there are no explicit generalized coordinate versions of Sections 1, 2,
4 and 5 in [9] we do not discuss those steps here. We only disprove the Free
Energy Lemma and the claim that when the Free Energy Lemma holds the
variational and ergodic conditional density become equal. For this we present
a way to translate the counterexamples used in Observations 4 and 5 into
counterexamples in generalized coordinates. The interpretation in terms of
Bayesian inference given in [9] is therefore equally as unjustified as the one
in [6].

For completeness, we first state the generalized coordinate versions of the
stochastic differential equation Equation (1)

$\displaystyle\dot{{\tilde{x}}}=f({\tilde{x}})+{\tilde{\omega}},$ (43)

the less general version of the Markov blanket structure Equation (2)

$\displaystyle\begin{split}f_{\tilde{\psi}}({\tilde{\psi}},{\tilde{s}},{\tilde{a}})&=(\Gamma-Q)_{{\tilde{\psi}}{\tilde{\psi}}}\nabla_{\tilde{\psi}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
f_{\tilde{s}}({\tilde{\psi}},{\tilde{s}},{\tilde{a}})&=(\Gamma-Q)_{{\tilde{s}}{\tilde{s}}}\nabla_{\tilde{s}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
f_{\tilde{a}}({\tilde{s}},{\tilde{a}},{\tilde{r}})&=(\Gamma-Q)_{{\tilde{a}}{\tilde{a}}}\nabla_{\tilde{a}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
f_{\tilde{r}}({\tilde{s}},{\tilde{a}},{\tilde{r}})&=(\Gamma-Q)_{{\tilde{r}}{\tilde{r}}}\nabla_{\tilde{r}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}}),\end{split}$ (44)

the expression of the ${\tilde{a}}$ and ${\tilde{r}}$ components of the
vectorfield in terms of the marginalised ergodic density Equations (19) and
(20)

$\displaystyle f_{\tilde{a}}({\tilde{s}},{\tilde{a}},{\tilde{r}})$
$\displaystyle=(\Gamma-Q)_{{\tilde{a}}{\tilde{a}}}\cdot\nabla_{\tilde{a}}\ln
p^{*}({\tilde{s}},{\tilde{a}},{\tilde{r}}),$ (45) $\displaystyle
f_{\tilde{r}}({\tilde{s}},{\tilde{a}},{\tilde{r}})$
$\displaystyle=(\Gamma-Q)_{{\tilde{r}}{\tilde{r}}}\cdot\nabla_{\tilde{r}}\ln
p^{*}({\tilde{s}},{\tilde{a}},{\tilde{r}}),$ (46)

and in terms of free energy Equations (29) and (30):

$\displaystyle f_{\tilde{a}}({\tilde{s}},{\tilde{a}},{\tilde{r}})$
$\displaystyle=(Q-\Gamma)_{{\tilde{a}}{\tilde{a}}}\cdot\nabla_{\tilde{a}}F({\tilde{s}},{\tilde{a}},{\tilde{r}}),$
(47) $\displaystyle f_{\tilde{r}}({\tilde{s}},{\tilde{a}},{\tilde{r}})$
$\displaystyle=(Q-\Gamma)_{{\tilde{r}}{\tilde{r}}}\cdot\nabla_{\tilde{r}}F({\tilde{s}},{\tilde{a}},{\tilde{r}}).$
(48)

The Free Energy Lemma then requires that there exists
$q({\tilde{\Psi}}|{\tilde{r}})$ such that the KL divergence between
$p^{*}({\tilde{\Psi}}|{\tilde{s}},{\tilde{a}},{\tilde{r}})$ vanishes. Without
going into further details of the difference between the proof in [9] and that
in [6], we can prove the former wrong by translating the counterexample used
for the latter into generalised coordinates.

######  Observation 6.

There is a general way to translate a system in ordinary coordinates into a
system of generalised coordinates that corresponds to an infinite number of
independent copies of the original system. This means all properties of the
original system (e.g., linearity, ergodicity, the Gaussian and Markovian
property of the noise, Conditions 1 and 2, properties of $\Gamma,R,U$) are
preserved during this translation.

###### Proof.

By construction, see Appendix E. ∎

This implies that the counterexamples used in proving Observations 4 and 5
directly translate to the setting of the generalised coordinates. The Free
Energy Lemma is therefore also wrong for generalised coordinates and the
variational density $q({\tilde{\Psi}}|{\tilde{r}})$ is not “ensured” [9] to be
equal to the conditional ergodic density
$p^{*}({\tilde{\Psi}}|{\tilde{s}},{\tilde{a}},{\tilde{r}})$.

##  8 Conclusions

We found that the two different Markov blanket conditions proposed in [6, 7,
14] are independent from each other. We then showed that under both of those
Markov blanket conditions, among the six steps contained in the argument in
[6], three do not hold independently from each other. We also showed that
fixing the second of those steps (Section 2) does not provide an valid
alternative. The line of reasoning of [6] therefore does not support its claim
that the internal coordinates of a Markov blanket “appear to have solved the
problem of Bayesian inference by encoding posterior beliefs about hidden
(external) [coordinates], …”. We have also shown that using generalised
coordinates as in [9] does not remedy the situation. Additionally, we
identified a technical error in [7] and an interpretational issue resulting
from possibly too strong assumptions (both Conditions 1 and 3) in [14]. We
also highlighted that the latter publications both argue that it is the most
likely internal coordinates given sensory and active coordinates that encode
posterior beliefs about external states instead of the internal coordinates
themselves. The resulting free energy principle and lemma are therefore a
different proposal. This is not subject to our technical critique.

## Contributions

Conceptualization, M.B., F.A.P. and R.K.; Formal analysis, M.B. and F.A.P.;
Funding acquisition, F.A.P. and R.K.; Methodology, M.B., F.A.P. and R.K.;
Visualization, M.B. and F.A.P.; Writing—original draft, M.B. and F.A.P.;
Writing—review & editing, M.B., F.A.P. and R.K. All authors have read and
agreed to the published version of the manuscript.

## Funding

The work by Martin Biehl and Ryota Kanai on this publication was made possible
through the support of a grant from Templeton World Charity Foundation, Inc.
The opinions expressed in this publication are those of the authors and do not
necessarily reflect the views of Templeton World Charity Foundation, Inc.
Martin Biehl and Ryota Kanai are also funded by the Japan Science and
Technology Agency (JST) CREST project. Felix A. Pollock acknowledges support
from the Monash University Network of Excellence for Consciousness and
Complexity in the Conscious Brain.

## Acknowledgments

All authors are grateful to Karl Friston and Thomas Parr for constructive
feedback on an earlier version of this work. We also want to thank Danijar
Hafner for pointing us to [12]. Martin Biehl wants to thank Yen Yu for helpful
discussions on generalized coordinates.

## Conflicts of interest

The authors declare no conflict of interest. The funders had no role in the
design of the study; in the collection, analyses, or interpretation of data;
in the writing of the manuscript, or in the decision to publish the results.

##  Appendix A Counterexamples for Observation 1

Consider a four dimensional linear system obeying Equation (1) for which there
are coordinates $x=(\psi,s,a,\lambda)$ with
$n_{\psi}=n_{s}=n_{a}=n_{\lambda}=1$ and

$\displaystyle f(x)=Mx,$ (49)

with the parameterisation

$\displaystyle M=\left(\begin{array}[]{cccc}-1&m_{1}&m_{2}&m_{3}\\\
m_{2}&-1&m_{2}&m_{3}\\\ m_{3}&m_{2}&-1&m_{2}\\\
m_{3}&m_{2}&m_{1}&-1\end{array}\right).$ (54)

From Equation (9), it is clear that the system obeys Condition 1 if $m_{3}=0$.
In this case, taking $\Gamma$ to be the identity matrix, it is possible to
show that

$\displaystyle
U_{\psi\lambda}=-\frac{m_{2}(m_{1}-m_{2}+2)(m_{2}^{3}+m_{1}^{2}m_{2}-2m_{1}m_{2}^{2}-4m_{1}-2)}{(m_{1}^{2}+m_{2}^{2}-4m_{2}+4)(m_{1}^{2}+5m_{2}^{2}-4m_{1}m_{2}+4m_{2}+4)}.$
(55)

For fixed, finite $m_{2}$, this is zero only for a few discrete values of
$m_{1}$, such as $m_{1}=m_{2}-2$; that it is generically non-zero proves
Equation (11). As a concrete example, the following

$\displaystyle
M=\left(\begin{array}[]{cccc}-1&-\nicefrac{{2}}{{3}}&-\nicefrac{{2}}{{3}}&0\\\
-\nicefrac{{2}}{{3}}&-1&-\nicefrac{{2}}{{3}}&0\\\
0&-\nicefrac{{2}}{{3}}&-1&-\nicefrac{{2}}{{3}}\\\
0&-\nicefrac{{2}}{{3}}&-\nicefrac{{2}}{{3}}&-1\\\ \end{array}\right),$ (60)

has

$\displaystyle
R=\left(\begin{array}[]{cccc}0&-\nicefrac{{1}}{{8}}&\nicefrac{{3}}{{8}}&0\\\
\nicefrac{{1}}{{8}}&0&0&-\nicefrac{{3}}{{8}}\\\
-\nicefrac{{3}}{{8}}&0&0&\nicefrac{{1}}{{8}}\\\
0&\nicefrac{{3}}{{8}}&-\nicefrac{{1}}{{8}}&0\\\ \end{array}\right)$ (65)

and (full rank and hence ergodic)

$\displaystyle
U=\left(\begin{array}[]{cccc}\nicefrac{{236}}{{255}}&\nicefrac{{127}}{{255}}&-\nicefrac{{31}}{{85}}&-\nicefrac{{12}}{{85}}\\\
\nicefrac{{127}}{{255}}&\nicefrac{{274}}{{255}}&\nicefrac{{206}}{{255}}&\nicefrac{{31}}{{85}}\\\
\nicefrac{{31}}{{85}}&\nicefrac{{206}}{{255}}&\nicefrac{{274}}{{255}}&\nicefrac{{127}}{{255}}\\\
-\nicefrac{{12}}{{85}}&\nicefrac{{31}}{{85}}&\nicefrac{{127}}{{255}}&\nicefrac{{236}}{{255}}\\\
\end{array}\right),$ (70)

and hence ergodic density

$\displaystyle p^{*}(\psi,s,a,\lambda)=$
$\displaystyle\sqrt{\frac{28}{2295\pi^{4}}}\,\exp\left[-\frac{1}{255}\left(137(a^{2}+s^{2})+118(\psi^{2}+\lambda^{2})\right.\right.$
$\displaystyle\qquad\left.\left.+127(\psi s+a\lambda)+93(\psi
a+s\lambda)+206as-36\psi\lambda\vphantom{137(a^{2}+s^{2})+118(\psi^{2}+\lambda^{2})}\right)\vphantom{-\frac{1}{255}\left(137(a^{2}+s^{2})+118(\psi^{2}+\lambda^{2})\right.}\right],$
(71)

which does not conditionally factorise.

Taking the same parameterisation as in Equation (54), and fixing
$m_{1}=m_{2}=-\nicefrac{{1}}{{2}}$, we can search for a non-zero value of
$m_{3}$ that leads to $U_{\psi\lambda}=0$ (equivalent to Condition 2 through
Equation (10)). We find such a value in the real root $c\simeq-0.08$ of the
quintic equation $8c^{5}-4c^{4}-6c^{3}+31c^{2}+40c+3=0$. That is, with

$\displaystyle
M=\left(\begin{array}[]{cccc}-1&-\nicefrac{{1}}{{2}}&-\nicefrac{{1}}{{2}}&c\\\
-\nicefrac{{1}}{{2}}&-1&-\nicefrac{{1}}{{2}}&c\\\
c&-\nicefrac{{1}}{{2}}&-1&-\nicefrac{{1}}{{2}}\\\
c&-\nicefrac{{1}}{{2}}&-\nicefrac{{1}}{{2}}&-1\\\ \end{array}\right),$ (76)

which does not satisfy Condition 1, we have

$\displaystyle
R=\left(\begin{array}[]{cccc}0&-0.06\dots&\hphantom{-}0.22\dots&0\\\
\hphantom{-}0.06\dots&0&0&-0.22\dots\\\
-0.22\dots&0&0&\hphantom{-}0.06\dots\\\
0&\hphantom{-}0.22\dots&-0.06\dots&0\\\ \end{array}\right),$ (81)

and

$\displaystyle U=\left(\begin{array}[]{cccc}0.96\dots&0.43\dots&0.30\dots&0\\\
0.43\dots&1.03\dots&0.58\dots&0.30\dots\\\
0.30\dots&0.58\dots&1.03\dots&0.43\dots\\\ 0&0.30\dots&0.43\dots&0.96\dots\\\
\end{array}\right),$ (86)

which has non-zero determinant (i.e., the dynamics is ergodic) and an ergodic
density satisfying Condition 2. This proves Equation (12).

##  Appendix B Counterexample for Section 2

Here, we consider a linear system, as in the previous appendix. We again
assume $\Gamma$ equal to the identity matrix and choose a force matrix of the
form

$\displaystyle
M=\left(\begin{array}[]{cccc}-1&-\frac{1}{2\sqrt{2}}&-\frac{1}{2\sqrt{2}}&0\\\
-\frac{1}{2\sqrt{2}}&-1&-\frac{1}{16}&0\\\
0&\frac{1}{16}&-1&-\frac{1}{2\sqrt{2}}\\\
0&\frac{1}{2\sqrt{2}}&-\frac{1}{2\sqrt{2}}&-1\\\ \end{array}\right)$ (91)

which explicitly satisfies Condition 1 and has full rank such that the system
is ergodic. Using Equation (5) this leads to

$\displaystyle
U=\left(\begin{array}[]{cccc}\frac{1023}{1057}&\frac{260\sqrt{2}}{1057}&\frac{136\sqrt{2}}{1057}&0\\\
\frac{260\sqrt{2}}{1057}&\frac{1091}{1057}&0&-\frac{136\sqrt{2}}{1057}\\\
\frac{136\sqrt{2}}{1057}&0&\frac{1091}{1057}&\frac{260\sqrt{2}}{1057}\\\
0&-\frac{136\sqrt{2}}{1057}&\frac{260\sqrt{2}}{1057}&\frac{1023}{1057}\\\
\end{array}\right)$ (96)

which shows that this system also satisfies Condition 2 since
$U_{\psi\lambda}=U_{\lambda\psi}=0$. We also find

$\displaystyle
R=\left(\begin{array}[]{cccc}0&-\frac{17}{1786\sqrt{2}}&\frac{479}{1786\sqrt{2}}&-\frac{62}{893}\\\
\frac{17}{1786\sqrt{2}}&0&-\frac{65}{14288}&\frac{479}{1786\sqrt{2}}\\\
-\frac{479}{1786\sqrt{2}}&\frac{65}{14288}&0&\frac{17}{1786\sqrt{2}}\\\
\frac{62}{893}&-\frac{479}{1786\sqrt{2}}&-\frac{17}{1786\sqrt{2}}&0\\\
\end{array}\right),$ (101)

which shows that all entries or $R$ that can be non-zero for an anti-symmetric
matrix are non-zero. For the marginal ergodic density we find

$\displaystyle\begin{split}p^{*}(s,a,\lambda)=\frac{239}{16\sqrt{2415}\pi^{3/2}}\exp&\left[-\frac{69a^{2}}{140}-\frac{37as}{70\sqrt{2}}+\frac{1}{35}\sqrt{2}2a\psi\right.\\\
&\left.\phantom{\left[\vphantom{-\frac{4867s^{2}}{9660}}\right.}-\frac{4867s^{2}}{9660}+\frac{74s\psi}{2415}-\frac{8429\psi^{2}}{19320}\right]\end{split}$
(102)

The difference between the right hand sides of Equations (17) and (19) is

$\displaystyle R_{as}\nabla_{s}\ln
p^{*}(s,a,\lambda)+R_{a\lambda}\nabla_{\lambda}\ln
p^{*}(s,a,\lambda)=\frac{37a+69\sqrt{2}\lambda-2563s}{4830}\neq 0,$ (103)

which shows that Equation (19) is wrong in this example and therefore not
generally equivalent to Equation (17). Similarly, computing the difference
between the right hand sides of Equations (18) and (20), one finds

$\displaystyle R_{\lambda s}\nabla_{s}\ln p^{*}(s,a,\lambda)+R_{\lambda
a}\nabla_{a}\ln
p^{*}(s,a,\lambda)=\frac{2a-\sqrt{2}\lambda+27s}{70\sqrt{2}}\neq 0,$ (104)

and hence Equation (20) is also incorrect in general.

Performing the same comparison for the difference between the general
expression in Equations (17) and (18) and the expressions taken from [7], one
finds

$\displaystyle R_{a_{i}s}\nabla_{s}\ln
p^{*}(s,a,\lambda)=\frac{73\left(296a+552\sqrt{2}\lambda-8429s\right)}{1,154,370}\neq
0$ (105)

for the difference between the right hand sides of Equations (17) and (21),
and

$\displaystyle R_{\lambda s}\nabla_{s}\ln
p^{*}(s,a,\lambda)=-\frac{53\left(296a+552\sqrt{2}\lambda-8429s\right)}{1,154,370\sqrt{2}}\neq
0,$ (106)

for the difference between the right hand sides of Equations (18) and (22).
Therefore, Equations (21) and (22) are also incorrect in general, even when
Condition 1 and Condition 2 both hold.

##  Appendix C Counterexamples for Section 3

We saw in Appendix B that Equations (19) and (20) are not generally equivalent
to Equation (4), even when Condition 1 and Condition 2 hold simultaneously. We
now show that if we instead use Equations (17) and (18), which are generally
equivalent to Equation (4), the Free Energy Lemma does not hold in general.

The original Free Energy Lemma requires that (see Equations (31) and (32))

$\displaystyle(\Gamma+R)_{aa}\cdot\nabla_{a}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0$ (107)
$\displaystyle(\Gamma+R)_{\lambda\lambda}\cdot\nabla_{\lambda}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0.$ (108)

replacing the partial gradient in Equations (29) and (30) with the full
gradient and including the entire matrix $(\Gamma+R)$ leads to the
corresponding requirement for the more general case:

$\displaystyle\left(R_{as}\cdot\nabla_{s}+(\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0$ (109) $\displaystyle\left(R_{\lambda
s}\cdot\nabla_{s}+R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0.$ (110)

Similarly, the version based on the equations taken from [7] implies

$\displaystyle\left((\Gamma_{aa}+R_{aa})\cdot\nabla_{a}+R_{a\lambda}\cdot\nabla_{\lambda}\right)\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0$ (111) $\displaystyle\left(R_{\lambda
a}\cdot\nabla_{a}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\cdot\nabla_{\lambda}\right)\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\Psi|s,a,\lambda)]$
$\displaystyle=0.$ (112)

Using the rules of Gaussian integration, we can write the logarithm of the
conditional ergodic density as

$\displaystyle\ln
p^{*}(\psi|s,a,\lambda)=-\frac{1}{2}|U_{\psi\psi}^{\frac{1}{2}}\psi+U_{\psi\psi}^{-\frac{1}{2}}U_{\psi
s}s+U_{\psi\psi}^{-\frac{1}{2}}U_{\psi
a}a+U_{\psi\psi}^{-\frac{1}{2}}U_{\psi\lambda}\lambda|^{2}+C,$ (113)

with $C$ a constant (and remembering each of $\psi$, $s$, $a$ and $\lambda$ is
a vector of coordinates in general). We can then expand out the derivatives of
the KL divergence to express them in terms of the coordinates:

$\displaystyle\nabla_{s}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\psi|s,a,\lambda)]=$
$\displaystyle-\int{\rm d}\psi\,q(\psi|\lambda)\nabla_{s}\ln
p^{*}(\psi|s,a,\lambda)$ $\displaystyle=$ $\displaystyle
U_{s\psi}U_{\psi\psi}^{-1}\big{(}U_{\psi s}s+U_{\psi a}a$
$\displaystyle\qquad\qquad+U_{\psi\lambda}\lambda+U_{\psi\psi}\langle\psi\rangle_{q(\Psi|\lambda)}\big{)},$
(114)
$\displaystyle\nabla_{a}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\psi|s,a,\lambda)]=$
$\displaystyle-\int{\rm d}\psi\,q(\psi|\lambda)\nabla_{a}\ln
p^{*}(\psi|s,a,\lambda)$ $\displaystyle=$ $\displaystyle
U_{a\psi}U_{\psi\psi}^{-1}\big{(}U_{\psi s}s+U_{\psi a}a$
$\displaystyle\qquad\qquad+U_{\psi\lambda}\lambda+U_{\psi\psi}\langle\psi\rangle_{q(\Psi|\lambda)}\big{)},$
(115)
$\displaystyle\nabla_{\lambda}\operatorname{D_{KL}}[q(\Psi|\lambda)||p^{*}(\psi|s,a,\lambda)]=$
$\displaystyle\int{\rm d}\psi\,\Big{[}\left(\ln q(\psi|\lambda)-\ln
p^{*}(\psi|s,a,\lambda)+1\right)\nabla_{\lambda}q(\psi|\lambda)$
$\displaystyle\qquad\quad-q(\psi|\lambda)\nabla_{\lambda}\ln
p^{*}(\psi|s,a,\lambda)\Big{]}$ $\displaystyle=$ $\displaystyle
U_{\lambda\psi}U_{\psi\psi}^{-1}\big{(}U_{\psi s}s+U_{\psi a}a$
$\displaystyle\qquad\qquad+U_{\psi\lambda}\lambda+U_{\psi\psi}\langle\psi\rangle_{q(\Psi|\lambda)}\big{)}$
$\displaystyle+\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\big{(}U_{\psi
s}s+U_{\psi a}a+U_{\psi\lambda}\lambda\big{)}$
$\displaystyle+\nabla_{\lambda}\left(\langle\psi^{T}U_{\psi\psi}\psi\rangle_{q(\Psi|\lambda)}-H[q(\Psi|\lambda)]\right),$
(116)

with $\langle g(\psi)\rangle_{q(\Psi|\lambda)}:=\int{\rm
d}\psi\,q(\psi|\lambda)g(\psi)$ and $H$ the Shannon entropy.

Substituting Equations (115) and (116) into Equations (107) and (108) leads to

$\displaystyle(\Gamma_{aa}+R_{aa})U_{a\psi}U_{\psi\psi}^{-1}\left(U_{\psi
s}s+U_{\psi
a}a+U_{\psi\lambda}\lambda+U_{\psi\psi}\langle\psi\rangle_{q(\Psi|\lambda)}\right)=0,$
(117)

and

$\displaystyle 0=$
$\displaystyle(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}U_{\psi\psi}^{-1}\big{(}U_{\psi
s}s+U_{\psi
a}a+U_{\psi\lambda}\lambda+U_{\psi\psi}\langle\psi\rangle_{q(\Psi|\lambda)}\big{)}$
$\displaystyle+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\big{(}U_{\psi
s}s+U_{\psi a}a+U_{\psi\lambda}\lambda\big{)}$
$\displaystyle+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\left(\langle\psi^{T}U_{\psi\psi}\psi\rangle_{q(\Psi|\lambda)}-H[q(\Psi|\lambda)]\right).$
(118)

Since these must hold for all values of the coordinates, they put strong
requirements on the $U$ and $R$ matrices. Specifically,

$\displaystyle(\Gamma_{aa}+R_{aa})U_{a\psi}U_{\psi\psi}^{-1}U_{\psi s}$
$\displaystyle=0,$ (119)
$\displaystyle(\Gamma_{aa}+R_{aa})U_{a\psi}U_{\psi\psi}^{-1}U_{\psi a}$
$\displaystyle=0,$ (120)
$\displaystyle(\Gamma_{aa}+R_{aa})U_{a\psi}U_{\psi\psi}^{-1}U_{\psi\lambda}$
$\displaystyle=0.$ (121)

In other words, since $U_{\psi\psi}$ and $\Gamma_{aa}$ must be nonzero for the
dynamics to be ergodic, it must be that $U_{\psi a}=0$. (This is equivalent to
$p^{*}(\Psi|s,a,\lambda)=p^{*}(\Psi|s,\lambda)$. So if Condition 2 also holds
we must have $p^{*}(\Psi|s,a,\lambda)=p^{*}(\Psi|s)$ in order for there to be
a suitable $q(\Psi|\lambda)$.) Specifically, consider the system specified by
the force matrix

$\displaystyle M=\left(\begin{array}[]{cccc}-1&0&\frac{1}{2}&0\\\ 0&-1&0&0\\\
0&0&-1&0\\\ 0&0&0&-1\\\ \end{array}\right)$ (126)

leads to

$\displaystyle R=\left(\begin{array}[]{cccc}0&0&-\frac{1}{4}&0\\\ 0&0&0&0\\\
\frac{1}{4}&0&0&0\\\ 0&0&0&0\\\ \end{array}\right)$ (131)

and

$\displaystyle U=\left(\begin{array}[]{cccc}\frac{16}{17}&0&-\frac{4}{17}&0\\\
0&1&0&0\\\ -\frac{4}{17}&0&\frac{18}{17}&0\\\ 0&0&0&1\\\ \end{array}\right).$
(136)

Here $M$ is full rank so the system is ergodic, clearly it also satisfies
Condition 1 due to the structure of $M$. Since $R_{as}=R_{a\lambda}=R_{\lambda
s}=0$ it obeys Equations (19) and (20) and since $Q_{\psi\lambda}=0$ it also
obeys Condition 2. Additionally, we find $U_{\psi a}=-\nicefrac{{4}}{{17}}$
which is a contradiction.

For the more general version, substituting Equations (114)–(116) into Equation
(109), one finds

$\displaystyle 0=$
$\displaystyle\left(\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}+R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\right)U_{\psi
s}s$
$\displaystyle+\left(\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}+R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\right)U_{\psi
a}a$
$\displaystyle+\left(\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}+R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\right)U_{\psi\lambda}\lambda$
$\displaystyle+\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)\langle\psi\rangle_{q(\Psi|\lambda)}$
$\displaystyle+R_{a\lambda}\nabla_{\lambda}\left(\langle\psi^{T}U_{\psi\psi}\psi\rangle_{q(\Psi|\lambda)}-H[q(\Psi|\lambda)]\right),$
(137)

which, considering that the coordinates can take any values, implies that

$\displaystyle\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}+R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}$
(138)

lies in a common (left) nullspace of $U_{\psi s}$, $U_{\psi a}$ and
$U_{\psi\lambda}$. However, the existence of such a nontrivial nullspace would
imply that the corresponding subspace of $\psi$ coordinates is independent of
the $s$, $a$ and $\lambda$ coordinates (to see this, consider marginalising
over their complement in Equation (113)). In other words, if only $\psi$
coordinates that play a nontrivial role in the dynamics are considered, then
Equation (109) must imply the quantity in Equation (138) is zero, and hence
that

$\displaystyle
R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=-\left(R_{as}U_{s\psi}+(R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}.$
(139)

However, through a similar procedure, one finds that Equation (110) is
equivalent to

$\displaystyle 0=$ $\displaystyle\Big{(}\left(R_{\lambda
s}U_{s\psi}+R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)U_{\psi\psi}^{-1}$
$\displaystyle\qquad+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\Big{)}U_{\psi
s}s$ $\displaystyle+\Big{(}\left(R_{\lambda s}U_{s\psi}+R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)U_{\psi\psi}^{-1}$
$\displaystyle\qquad+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\Big{)}U_{\psi
a}a$ $\displaystyle+\Big{(}\left(R_{\lambda s}U_{s\psi}+R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)U_{\psi\psi}^{-1}$
$\displaystyle\qquad+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}\Big{)}U_{\psi\lambda}\lambda$
$\displaystyle+\left(R_{\lambda s}U_{s\psi}+R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)\langle\psi\rangle_{q(\Psi|\lambda)}$
$\displaystyle+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\left(\langle\psi^{T}U_{\psi\psi}\psi\rangle_{q(\Psi|\lambda)}-H[q(\Psi|\lambda)]\right),$
(140)

implying that

$\displaystyle(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=-\left(R_{\lambda
s}U_{s\psi}+R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)U_{\psi\psi}^{-1}.$
(141)

Unless $R_{a\lambda}$ and $(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})$ share
a common nullspace, or the $U$ and $R$ matrices are finely tuned, then
Equations (139) and (141) contradict one another. In this case, there cannot
exist a $q(\Psi|\lambda)$ that satisfies both Equations (109) and (110), and
hence the modified Free Energy Lemma is invalid in general. In particular,
using the example from Appendix B, if we solve Equation (139) for
$\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}$ we find

$\displaystyle\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=-\frac{53}{5},$
(142)

and from Equation (141) we get

$\displaystyle\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=\frac{29}{239},$
(143)

which is a contradiction.

If we now perform the same procedure for Equations (111) and (112), we arrive
at the following conditions on the gradient of the variational density:

$\displaystyle
R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=-\left((R_{aa}+\Gamma_{aa})U_{a\psi}+R_{a\lambda}U_{\lambda\psi}\right)U_{\psi\psi}^{-1}.$
(144)

and

$\displaystyle
R_{a\lambda}\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=-\left(R_{\lambda
a}U_{a\psi}+(\Gamma_{\lambda\lambda}+R_{\lambda\lambda})U_{\lambda\psi}\right)U_{\psi\psi}^{-1}.$
(145)

Even when Condition 2 holds and $U_{\psi\lambda}=0$, these will be
inconsistent in general. As a specific counterexample, take the system with
force matrix

$\displaystyle M=\left(\begin{array}[]{cccc}-1&0&-\frac{1}{2}&0\\\ 0&-1&0&0\\\
0&0&-1&\frac{\sqrt{3}}{2}\\\ 0&0&0&-\frac{1}{2}\\\ \end{array}\right),$ (150)

with correposponding

$\displaystyle
R=\left(\begin{array}[]{cccc}0&0&\frac{1}{3}&\frac{1}{3\sqrt{3}}\\\ 0&0&0&0\\\
-\frac{1}{3}&0&0&-\frac{1}{\sqrt{3}}\\\
-\frac{1}{3\sqrt{3}}&0&\frac{1}{\sqrt{3}}&0\\\ \end{array}\right),$ (155)

and

$\displaystyle U=\left(\begin{array}[]{cccc}\frac{9}{10}&0&\frac{3}{10}&0\\\
0&1&0&0\\\ \frac{3}{10}&0&\frac{17}{20}&-\frac{\sqrt{3}}{4}\\\
0&0&-\frac{\sqrt{3}}{4}&\frac{3}{4}\\\ \end{array}\right).$ (160)

This model is ergodic (full rank $U$), and it satisfies both Condition 1 and
Condition 2. Moreover, the forces satisfy Equations (21) and (22). However,
substituting the relevant elements of $U$ and $R$ matrices into Equation
(144), we find

$\displaystyle\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=\frac{1}{\sqrt{3}},$
(161)

but doing the same for Equation (145) gives

$\displaystyle\nabla_{\lambda}\langle\psi\rangle_{q(\Psi|\lambda)}=\frac{1}{3},$
(162)

which is a contradiction.

##  Appendix D Counterexample for Section 5

Here we provide an example system for which Conditions 1 and 2 as well as
Sections 1, 2, 3 and 4 are valid but Section 5 fails. We use a system with

$\displaystyle f(x)=Mx$ (163)

where

$\displaystyle M:=\left(\begin{array}[]{cccc}-1&\nicefrac{{1}}{{2}}&0&0\\\
\nicefrac{{1}}{{2}}&-1&\nicefrac{{1}}{{2}}&0\\\
0&\nicefrac{{1}}{{2}}&-1&\nicefrac{{1}}{{2}}\\\ 0&0&\nicefrac{{1}}{{2}}&-1\\\
\end{array}\right).$ (168)

This system is ergodic, satisfies Condition 1 and as we will will see
satisfies Equations (19) and (20) as well. Using Equation (5) we find

$\displaystyle R=\left(\begin{array}[]{cccc}0&0&0&0\\\ 0&0&0&0\\\ 0&0&0&0\\\
0&0&0&0\\\ \end{array}\right)$ (173)

and from Equation (8)

$\displaystyle U=-M$ (174)

which means that Condition 2 is also satisfied.

This leads to the ergodic density

$\displaystyle p^{*}(\psi,s,a,\lambda)=$
$\displaystyle\frac{\sqrt{5}}{16\pi^{2}}e^{-\frac{1}{2}\left(\psi^{2}-\psi
s+s^{2}-sa+a^{2}-a\lambda+\lambda^{2}\right)}$ (175)

which can be used to check that Equations (19) and (20) hold for this example.
The conditional ergodic density is

$\displaystyle
p^{*}(\psi|s,a,\lambda)=p^{*}(\psi|s)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\left(\psi-\frac{1}{2}s\right)^{2}}.$
(176)

If we now define $q(\psi|\lambda)=q(\psi)=\exp(-(\psi-\mu)^{2}/2)/\sqrt{2\pi}$
as a Gaussian distribution with mean $\mu$ and variance one, we can compute
the KL divergence to get:

$\displaystyle\operatorname{D_{KL}}[q(\Psi)||p^{*}(\Psi|s,a,\lambda)]=K(s)=\frac{1}{2}\left(\mu-\frac{1}{2}s\right)^{2}.$
(177)

Clearly, for this choice of $q(\psi|\lambda)$ the gradients with respect to
$a$ and $\lambda$ of the KL divergence vanish everywhere (Equations (35) and
(36) hold). This also means we can express $f_{a},f_{\lambda}$ in terms of a
free energy i.e., the Free Energy Lemma holds for this system. However, for
any proposed bound $c\geq 0$ on the KL divergence, there is a value of $s$ for
which it is exceeded, whatever the choice of $\mu$. Moreover, we can choose a
$\mu$ such that the KL divergence is larger than any given $c$, even when
$s=0$.

##  Appendix E Translating Systems into Generalized Coordinates Systems

We show how to get a generalized coordinate system from a finite dimensional
system. By definition the generalized coordinates are infinite dimensional.
For all $n\in\mathbb{N}$ and a coordinate $x$ they also include the $n$-th
time derivative of $x$.

Assume as given an ergodic, linear, random dynamical system described by

$\displaystyle\dot{x}=Mx+\omega$ (178)

where $x=(x_{1},...,x_{k})$ is a $k$-dimensional vector, $M$ is a $k\times k$
real valued matrix, and $\dot{x}:={\frac{d}{dt}}x$. We can look at the second
time derivative of the state by differentiating both sides:

$\displaystyle{\frac{d}{dt}}\dot{x}$ $\displaystyle={\frac{d}{dt}}(Mx+\omega)$
(179) $\displaystyle\ddot{x}$ $\displaystyle=M\dot{x}+\dot{\omega}$ (180)

Similarly for the third time derivative:

$\displaystyle{\frac{d}{dt}}\ddot{x}$
$\displaystyle={\frac{d}{dt}}(M\dot{x}+\dot{\omega})$ (181)
$\displaystyle=M\ddot{x}+\ddot{\omega}$ (182)

Similarly for all higher derivatives:

$\displaystyle\frac{d^{n}}{dt^{n}}x$
$\displaystyle=M\frac{d^{n-1}}{dt^{n-1}}x+\frac{d^{n}}{dt^{n}}\omega.$ (183)

Now define the generalized coordinates
${\tilde{x}}=(x,x^{\prime},x^{\prime\prime},...)$ as

$\displaystyle x$ $\displaystyle=x$ (184) $\displaystyle x^{\prime}$
$\displaystyle={\frac{d}{dt}}x$ (185) $\displaystyle x^{\prime\prime}$
$\displaystyle=\frac{d^{2}}{dt^{2}}x$ (186) $\displaystyle\vdots$ (187)
$\displaystyle x^{(n)}$ $\displaystyle=\frac{d^{n}}{dt^{n}}x$ (188)
$\displaystyle\vdots$ (189)

Define also

$\displaystyle\tilde{\omega}:=(\omega,{\frac{d}{dt}}\omega,\frac{d^{2}}{dt^{2}}\omega,...,\frac{d^{n}}{dt^{n}}\omega,...).$
(190)

Without further clarification, the derivatives of $\omega$ are not well
defined when the latter is a Gaussian white noise process, as explicitly
assumed in writing the vector field $f(x)$ in terms of the ergodic density [1,
11, 10]. As discussed in [15], delta-correlated Markovian noise is always a
limiting approximation of noise with a finite correlation time. Meaningfully
taking the derivatives requires first choosing a functional form for the
(co)variance whose limit is a delta function (Another, more direct approach
would be in terms of generalized functions, but here too additional
information is required to specify the derivatives [13].) However, different
choices can lead to vastly different central moments of the generalized noise
distribution, including those that vanish or diverge at all orders. In the
former case, the process in terms of generalized coordinates may not be
ergodic [4]; in the latter case, the process is not well defined. In general,
it is not clear that Equation (4) holds in the non-Markovian case, since the
standard derivations in [1, 11] and related works rely on delta-correlated
noise.

Here, we can therefore assume that the noise is such that the derivatives in
Equation (190) can be treated as Markov and Gaussian. We also assume that
$\frac{d^{n}}{dt^{n}}\omega$ is independently and identically distributed to
$\frac{d^{n-1}}{dt^{n-1}}\omega$ for all $n$. Finally, we can then define the
(infinite) matrix ${\bar{M}}$ as the block diagonal matrix with all blocks
equal to $M$:

$\displaystyle{\bar{M}}:=\begin{pmatrix}M&0&\dots&\\\ 0&M&&\\\
\vdots&&\ddots&\\\ \end{pmatrix}$ (191)

The time derivative of $\omega$ is independent of $\omega$, as the changes are
independent of the value of $\omega$. So we actually get an infinite number of
independent and identically distributed systems. Using these definitions we
have:

$\displaystyle\dot{{\tilde{x}}}={\bar{M}}{\tilde{x}}+\tilde{\omega}.$ (192)

These equations describe a random dynamical system composed of an infinite
number of independent linear random dynamical systems, all governed by the
same matrix $M$ and driven by independently and identically distributed noise.
Since the first of these systems (for the variables $x$) is ergodic by
assumption, all of the subsystems are also ergodic and, therefore, the whole
system is ergodic with the ergodic density equal to a product of the original
ergodic density:

$\displaystyle\bar{p}^{*}({\tilde{x}})=p^{*}(x)p^{*}(x^{\prime})p^{*}(x^{\prime\prime})\cdots
p^{*}(x^{(n)})\cdots.$ (193)

Additionally, if $M$ is such that

$\displaystyle\begin{split}M_{\psi}\cdot(\psi,s,a,r)^{\top}=f_{\psi}(\psi,s,a)&=(\Gamma-Q)_{\psi\psi}\nabla_{\psi}\ln
p^{*}(\psi,s,a,r)\\\
M_{s}\cdot(\psi,s,a,r)^{\top}=f_{s}(\psi,s,a)&=(\Gamma-Q)_{ss}\nabla_{s}\ln
p^{*}(\psi,s,a,r)\\\
M_{a}\cdot(\psi,s,a,r)^{\top}=f_{a}(s,a,r)&=(\Gamma-Q)_{aa}\nabla_{a}\ln
p^{*}(\psi,s,a,r)\\\
M_{r}\cdot(\psi,s,a,r)^{\top}=f_{r}(s,a,r)&=(\Gamma-Q)_{rr}\nabla_{r}\ln
p^{*}(\psi,s,a,r),\end{split}$ (194)

(which is the case for the $M$ in the counterexample to Section 5) then for

$\displaystyle\begin{split}(x_{1},x_{2},x_{3},x_{4}):&=(\psi,s,a,r)\\\
(x^{\prime}_{1},x^{\prime}_{2},x^{\prime}_{3},x^{\prime}_{4}):&=(\psi^{\prime},s^{\prime},a^{\prime},r^{\prime})\\\
(x^{\prime\prime}_{1},x^{\prime\prime}_{2},x^{\prime\prime}_{3},x^{\prime\prime}_{4}):&=(\psi^{\prime\prime},s^{\prime\prime},a^{\prime\prime},r^{\prime\prime})\\\
&\vdots\\\
(x^{(n)}_{1},x^{(n)}_{2},x^{(n)}_{3},x^{(n)}_{4}):&=(\psi^{(n)},s^{(n)},a^{(n)},r^{(n)})\\\
&\vdots,\end{split}$ (195)
$\displaystyle{\bar{Q}}:=\begin{pmatrix}Q&0&\dots&\\\ 0&Q&&\\\
\vdots&&\ddots&\\\ \end{pmatrix},$ (196)
$\displaystyle{\bar{\Gamma}}:=\begin{pmatrix}\Gamma&0&\dots&\\\ 0&\Gamma&&\\\
\vdots&&\ddots&\\\ \end{pmatrix},$ (197)

and using Equation (8) and that the inverse of a block diagonal matrix is
block diagonal

$\displaystyle\bar{U}:=\begin{pmatrix}U&0&\dots&\\\ 0&U&&\\\
\vdots&&\ddots&\\\ \end{pmatrix},$ (198)

we also have:

$\displaystyle\begin{split}{\bar{M}}_{\tilde{\psi}}\cdot({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})^{\top}=f_{\tilde{\psi}}({\tilde{\psi}},{\tilde{s}},{\tilde{a}})&=({\bar{\Gamma}}-{\bar{Q}})_{{\tilde{\psi}}{\tilde{\psi}}}\nabla_{\tilde{\psi}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
{\bar{M}}_{\tilde{s}}\cdot({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})^{\top}=f_{\tilde{s}}({\tilde{\psi}},{\tilde{s}},{\tilde{a}})&=({\bar{\Gamma}}-{\bar{Q}})_{{\tilde{s}}{\tilde{s}}}\nabla_{\tilde{s}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
{\bar{M}}_{\tilde{a}}\cdot({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})^{\top}=f_{\tilde{a}}({\tilde{s}},{\tilde{a}},{\tilde{r}})&=({\bar{\Gamma}}-{\bar{Q}})_{{\tilde{a}}{\tilde{a}}}\nabla_{\tilde{a}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})\\\
{\bar{M}}_{\tilde{r}}\cdot({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})^{\top}=f_{\tilde{r}}({\tilde{s}},{\tilde{a}},{\tilde{r}})&=({\bar{\Gamma}}-{\bar{Q}})_{{\tilde{r}}{\tilde{r}}}\nabla_{\tilde{r}}\ln
p^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}}).\end{split}$ (199)

The ergodic density of such a system is a product of ergodic densities of the
original system Equation (176):

$\displaystyle{\bar{p}}^{*}({\tilde{\psi}},{\tilde{s}},{\tilde{a}},{\tilde{r}})=p^{*}(\psi,s,a,r)p^{*}(\psi^{\prime},s^{\prime},a^{\prime},r^{\prime})p^{*}(\psi^{\prime\prime},s^{\prime\prime},a^{\prime\prime},r^{\prime\prime})\cdots.$
(200)

Thus any property of the original system is also a property of the generalized
coordinate system.

## References

  * Ao, [2004] Ao, P. (2004).  Potential in stochastic differential equations: novel construction.  Journal of Physics A: Mathematical and General, 37(3):L25–L30.
  * Ao et al., [2013] Ao, P., Chen, T.-Q., and Shi, J.-H. (2013).  Dynamical Decomposition of Markov Processes without Detailed Balance.  Chinese Physics Letters, 30(7):070201.  Publisher: IOP Publishing.
  * Bishop, [2006] Bishop, C. (2006).  Pattern Recognition and Machine Learning.  Information Science and Statistics. Springer-Verlag, New York.
  * Cornfeld et al., [1982] Cornfeld, I. P., Fomin, S. V., and Sinai, Y. G. (1982).  Ergodic Theory.  Springer-Verlag, New York.
  * Cover and Thomas, [2006] Cover, T. M. and Thomas, J. A. (2006).  Elements of information theory.  Wiley-Interscience, Hoboken, N.J.
  * Friston, [2013] Friston, K. (2013).  Life as we know it.  Journal of The Royal Society Interface, 10(86):2013.0475.
  * Friston, [2019] Friston, K. (2019).  A free energy principle for a particular physics.  arXiv:1906.10184 [q-bio].  arXiv: 1906.10184.
  * Friston et al., [2015] Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., and Pezzulo, G. (2015).  Active inference and epistemic value.  Cognitive Neuroscience, 6(4):187–214.
  * Friston et al., [2014] Friston, K., Sengupta, B., and Auletta, G. (2014).  Cognitive Dynamics: From Attractors to Active Inference.  Proceedings of the IEEE, 102(4):427–445.
  * Kwon and Ao, [2011] Kwon, C. and Ao, P. (2011).  Nonequilibrium steady state of a stochastic system driven by a nonlinear drift force.  Physical Review E, 84:061106.
  * Kwon et al., [2005] Kwon, C., Ao, P., and Thouless, D. J. (2005).  Structure of stochastic dynamics near fixed points.  Proceedings of the National Academy of Sciences, 102(37):13029–13033.
  * Ma et al., [2015] Ma, Y.-A., Chen, T., and Fox, E. B. (2015).  A complete recipe for stochastic gradient MCMC.  In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NeurIPS’15, pages 2917–2925, Montreal, Canada. MIT Press.
  * Oberguggenberger, [1995] Oberguggenberger, M. (1995).  Generalized functions and stochastic processes.  In Bolthausen, E., Dozzi, M., and Russo, F., editors, Seminar on Stochastic Analysis, Random Fields and Applications. Progress in Probability, vol 36., page 215–230. Birkhäuser, Basel.
  * Parr et al., [2019] Parr, T., Da Costa, L., and Friston, K. (2019).  Markov blankets, information geometry and stochastic thermodynamics.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 378(4):2019.0159.
  * van Kampen, [1981] van Kampen, N. G. (1981).  Stochastic Processes in Physics and Chemistry.  North-Holland, Amsterdam.
  * Yuan et al., [2017] Yuan, R., Tang, Y., and Ao, P. (2017).  SDE decomposition and A-type stochastic interpretation in nonequilibrium processes.  Frontiers of Physics, 12(6):120201.
  * Yuan et al., [2014] Yuan, R.-S., Ma, Y.-A., Yuan, B., and Ao, P. (2014).  Lyapunov function as potential function: A dynamical equivalence.  Chinese Physics B, 23(1):010505.