# The Two Kinds of Free Energy and the Bayesian Revolution

Sebastian Gottwald Institute of Neural Information Processing, Ulm University,
89081 Ulm, Germany<EMAIL_ADDRESS>Daniel A. Braun Institute
of Neural Information Processing, Ulm University, 89081 Ulm, Germany

> ## Abstract
>
> The concept of free energy has its origins in 19th century thermodynamics,
> but has recently found its way into the behavioral and neural sciences,
> where it has been promoted for its wide applicability and has even been
> suggested as a fundamental principle of understanding intelligent behavior
> and brain function. We argue that there are essentially two different
> notions of free energy in current models of intelligent agency, that can
> both be considered as applications of Bayesian inference to the problem of
> action selection: one that appears when trading off accuracy and uncertainty
> based on a general maximum entropy principle, and one that formulates action
> selection in terms of minimizing an error measure that quantifies deviations
> of beliefs and policies from given reference models. The first approach
> provides a normative rule for action selection in the face of model
> uncertainty or when information processing capabilities are limited. The
> second approach directly aims to formulate the action selection problem as
> an inference problem in the context of Bayesian brain theories, also known
> as Active Inference in the literature. We elucidate the main ideas and
> discuss critical technical and conceptual issues revolving around these two
> notions of free energy that both claim to apply at all levels of decision-
> making, from the high-level deliberation of reasoning down to the low-level
> information processing of perception.
>
> Keywords: free energy, intelligent agency, bayesian inference, maximum
> entropy, utility theory, active inference

This is a preprint of the article The two kinds of free energy and the
Bayesian revolution, PLOS Computational Biology 16(12), 2020.

##  1 Introduction

There is a surprising line of thought connecting some of the greatest
scientists of the last centuries, including Immanuel Kant, Hermann von
Helmholtz, Ludwig E. Boltzmann, and Claude E. Shannon, whereby model-based
processes of action, perception, and communication are explained with concepts
borrowed from statistical physics. Inspired by Kant’s Copernican revolution
and motivated from his own studies of the physiology of the sensory system,
Helmholtz was one of the first proponents of the analysis-by-synthesis
approach to perception (Yuille and Kersten, , 2006), whereby a perceiver is
not simply conceptualized as some kind of _tabula rasa_ recording raw external
stimuli, but rather relies on internal models of the world to match and
anticipate sensory inputs. The internal model paradigm is now ubiquitous in
the cognitive and neural sciences and has even led some researchers to propose
a Bayesian brain hypothesis, whereby the brain would essentially be a
prediction and inference engine based on internal models (Kawato, , 1999;
Flanagan et al., , 2003; Doya, , 2007). Coincidentally, Helmholtz also
invented the notion of the Helmholtz free energy that plays an important role
in thermodynamics and statistical mechanics, even though he never made a
connection between the two concepts in his lifetime.

This connection was first made by Dayan, Hinton, Neal, and Zemel in their
computational model of perceptual processing as a statistical inference engine
known as the Helmholtz machine (Dayan et al., , 1995). In this neural network
architecture, there are feed-forward and feedback pathways, where the bottom-
up pathway translates inputs from the bottom layer into hidden causes at the
upper layer (the recognition model), and top-down activation translates
simulated hidden causes into simulated inputs (the generative model). When
considering log-likelihood in this setup as energy in analogy to statistical
mechanics, learning becomes a relaxation process that can be described by the
minimization of variational free energy. While it should be emphasized that
variational free energy is not the same as Helmholtz free energy, the two free
energy concepts can be formally related. Importantly, variational free energy
minimization is not only a hallmark of the Helmholtz machine, but of a more
general family of inference algorithms, such as the popular expectation-
maximization (EM) algorithm (Neal and Hinton, , 1998; Beal, , 2003). In fact,
over the last two decades, variational Bayesian methods have become one of the
foremost approximation schemes for tractable inference in the machine learning
literature. Moreover, a plethora of machine learning approaches use loss
functions that have the shape of a free energy when optimizing performance
under entropy regularization in order to boost generalization of learning
models (Williams and Peng, , 1991; Mnih et al., , 2016).

In the meanwhile, free energy concepts have also made their way into the
behavioral sciences. In the economic literature, for example, trade-offs
between utility and entropic uncertainty measures that take the form of free
energies have been proposed to describe decision-makers with stochastic choice
behavior due to limited resources (McKelvey and Palfrey, , 1995; Sims, , 2003;
Mattsson and Weibull, , 2002; McFadden, , 2005; Wolpert, , 2006) or robust
decision-makers with limited precision in their models (Maccheroni et al., ,
2006; Hansen and Sargent, , 2008). The free energy trade-off between entropy
and reward can also be found in information-theoretic models of biological
perception-action systems (Still, , 2009; Tishby and Polani, , 2011; Ortega
and Braun, , 2013), some of which have been subjected to experimental testing
(Ortega and Stocker, , 2016; Sims, , 2016; Schach et al., , 2018; Lindig-León
et al., , 2019; Bhui and Gershman, , 2018; Ho et al., , 2020). Finally, in the
neuroscience literature the notion of free energy has risen to recent fame as
the central puzzle piece in the Free Energy Principle (Friston, , 2010) that
has been used to explain a cornucopia of experimental findings including
neural prediction error signals (Sales et al., , 2019), synaptic plasticity
rules(Bogacz, , 2017), neural effects of biased competition and attention
(Friston et al., , 2012; Parr and Friston, , 2017), visual exploration in
humans (Mirza et al., , 2018), and more—see the references in (Parr and
Friston, , 2019). Over time, the Free Energy Principle has grown out of an
application of the free energy concept used in the Helmholtz machine, to
interpret cortical responses in the context of predictive coding (Friston, ,
2005), and has gradually developed into a general principle for intelligent
agency, also known as _Active Inference_ (Friston et al., , 2013, 2015; Parr
and Friston, , 2019). Consequences and implications of the Free Energy
Principle are discussed in neighbouring fields like psychiatry (Schwartenbeck
and Friston, , 2016; Linson et al., , 2020) and the philosophy of mind (Clark,
, 2013; Colombo and Wright, , 2018).

Given that the notion of free energy has become such a pervasive concept that
cuts through multiple disciplines, the main rationale for this discussion
paper is to trace back and to clarify different notions of free energy, to see
how they are related and what role they play in explaining behavior and neural
activity. As the notion of free energy mainly appears in the context of
statistical models of cognition, the language of probabilistic models
constitutes a common framework in the following discussion. Section 2
therefore starts with preliminary remarks on probabilistic modelling. Section
3 introduces two notions of free energy that are subsequently expounded in
Section 4 and Section 5, where they are applied to models of intelligent
agency. Section 6 concludes the paper.

Figure 1: Graphical representation of an exemplary probabilistic model. The
arrows (edges) indicate causal relationships between the random variables
(nodes). The full joint distribution $p_{0}$ over all random variables is
sometimes also referred to as a _generative model_ , because it contains the
complete knowledge about the random variables and their dependencies and
therefore allows to generate simulated data. Such a model could for example be
used by a farmer to infer the soil quality $S$ based on the crop yields $X$
through Bayesian inference, which allows to determine a priori unknown
distributions such as $p(S|X)$ from the generative model $p_{0}$ via
marginalization and conditionalization.

##  2 Probabilistic models and perception-action systems

Systems that show stochastic behavior, for example due to randomly behaving
components or because the observer ignores certain degrees of freedom, are
modelled using probability distributions. This way, any behavioral,
environmental, and hidden variables can be related by their statistics, and
dynamical changes can be modelled by changes in their distributions.

Consider, for example, the simple probabilistic model illustrated in Fig 1,
consisting of the (for simplicity, discrete) variables past and future soil
quality $\mathbf{S}\coloneqq(S,S^{\prime})$, past and future crop yields
$\mathbf{X}\coloneqq(X,X^{\prime})$, and fertilization $A$. The graphical
model shown in the figure corresponds to the joint probability
$p_{0}(\mathbf{X},\mathbf{S},A)$ given by the factorization

$p_{0}(X^{\prime}|S^{\prime})\,p_{0}(X|S)\,p_{0}(S^{\prime}|S,A)\,p_{0}(S)\,p_{0}(A)\,,$
(1)

where $p_{0}(S)$ is the base probability of the past soil quality $S$,
$p_{0}(X|S)$ is the probability of crop yields $X$ depending on the past soil
quality $S$, and so forth. Given the joint distribution we can now ask
questions about each of the variables. For example, we could ask about the
probability distribution $p(S|X\hskip 1.0pt{=}\hskip 2.0ptx)$ of soil quality
$S$ if we are told that the crop yields $X$ are equal to a value $x$. We can
obtain the answer from the probabilistic model $p_{0}$ by doing _Bayesian
inference_ , yielding the Bayes’ posterior

$p(S|X)=\frac{p(S,X)}{\sum_{s}p(s,X)}=\frac{p_{0}(X|S)p_{0}(S)}{\sum_{s}p_{0}(X|s)p_{0}(s)}\,,$
(2)

where the dependencies on $X^{\prime}$, $S^{\prime}$, and $A$ have been summed
out to calculate the marginal $p(S,X)$. In general, Bayesian inference in a
probabilistic model means to determine the probability of some queried
unobserved variables given the knowledge of some observed variables. This can
be viewed as transforming the prior probabilistic model $p_{0}$ to a posterior
model $p$, under which the observed values have probability one and unobserved
variables have probabilities given by the corresponding Bayes’ posteriors.

In principle, Bayesian inference requires only two different kinds of
operations, namely marginalization, i.e. summing out unobserved variables that
have not been queried, such as $X^{\prime},S^{\prime}$ and $A$ above, and
conditionalization, i.e. renormalizing the joint distribution over observed
and queried variables—that may itself be the result from a previous
marginalization such as $p(S,X)$ above—to obtain the required conditional
distribution over the queried variables. In practice, however, inference is a
hard computational problem and many more efficient inference methods are
available that may provide approximate solutions to the exact Bayes’
posteriors, including belief propagation (Pearl, , 1988), expectation
propagation (Minka, , 2001), variational Bayesian inference (Hinton and van
Camp, , 1993), and Monte Carlo algorithms (MacKay, , 2002). Also note that
inference is trivial if the sought-after conditional distribution of the
queried variable is already given by one of the conditional distributions that
jointly specify the probabilistic model, e.g., $p(X|S)=p_{0}(X|S)$.

Probabilistic models can be used not only as _external (observer) models_ ,
but also as _internal models_ that are employed by the agent itself, or by a
designer of the agent, in order to determine a desired course of action. In
this latter case, actions could either be thought of as deterministic
parameters of the probabilistic model that influence the future (influence
diagrams) or as random variables that are part of the probabilistic model
themselves (prior models) (Boutilier et al., , 1999). Either way, internal
models allow making predictions over future consequences in order to find
actions or distributions over actions that lead to desirable outcomes, for
example actions that produce high rewards in the future. In mechanistic or
process model interpretations, some of the specification procedures to find
such actions are themselves meant to represent what the agent is actually
doing while reasoning, whereas as if interpretations simply use these methods
as tools to arrive at distributions that describe the agent’s behavior. Free
energy is one of the concepts that appears in both types of methods.

##  3 The two notions of free energy

Vaguely speaking, free energy can refer to any quantity that is of the form

$\text{free energy}\ =\
\text{energy}\,\pm\,\text{const.}\times\text{entropy},$ (3)

where energy is an expected value of some quantitity of interest, entropy
refers to a quantity measuring disorder, uncertainty, or complexity, that must
be specified in the given context, and const. is a constant term that
translates between units of entropy and energy, and is related to the
temperature in physically motivated free energy expressions. From relation
(3), it is not surprising that free energy sometimes appears enshrouded by
mystery, as it relies on an understanding of entropy, and “nobody really knows
what entropy is anyway”, as John Von Neumann famously quipped (Feynman et al.,
, 1996).

Historically, the concept of free energy goes back to the roots of
thermodynamics, where it was introduced to measure the maximum amount of work
that can be extracted from a thermodynamic system at a constant temperature
and volume. If, for example, all the molecules in a box move to the left, we
can use this kinetic energy to drive a turbine. If, however, the same kinetic
energy is distributed as random molecular motion, it cannot be fully
transformed into work. Therefore, only part of the total energy $E$ is usable,
because the exact positions and momenta of the molecules, the so-called
microstates, are unknown. In this case, the maximum usable part of the energy
$E$ is the _Helmholtz free energy_ , defined as

$F=E-TS\,,$ (4)

where $S$ is the thermodynamic entropy. In general, the transformation between
two macrostates with free energies $F_{1}$ and $F_{2}$ allows the extraction
of work $W\leq F_{2}-F_{1}$.

While the two notions of free energy that we discuss in the following are
vaguely inspired by the physical original, their motivations are rather
distinct and the main reason they share the nomenclature is due to their
general form (3) resembling the Helmholtz free energy (4).

###  3.1 Free energy from constraints

The first notion of free energy is closely tied to the principle of maximum
entropy (Jaynes, , 1957), which virtually appears in all branches of science.
From this vantage point, the physical free energy is merely a special instance
of a more general inference problem where we hold probabilistic beliefs about
unknown quantities (e.g., the exact energy values of the molecules in a gas)
and we can only make coarse measurements or observations (e.g., the
temperature of the gas) that we can use to update our beliefs about these
hidden variables. The principle of maximum entropy suggests that, among the
beliefs that are compatible with the observation, we should choose the most
“unbiased” belief, in the sense that it corresponds to a maximum number of
possible assignments of the hidden variables.

####  3.1.1 Wallis’ motivation of the maximum entropy principle

Consider the random experiment of distributing $N$ elements randomly in $n$
equally probable buckets with $N\gg n$, where the resulting number of elements
$N_{i}$ in bucket $i\in\\{1,\dots,N\\}$ determines the probability
$p(z_{i})\coloneqq\frac{N_{i}}{N}$. In principle, this way we could generate
any distribution $p$ over a finite set $\Omega=\\{z_{1},\dots,z_{n}\\}$ that
we like, however, a uniform distribution that reflects the equiprobable
assignment clearly is much more likely than a Dirac distribution where all the
probability mass is concentrated in one bucket. Here, the reason is that there
are many possible assignments of elements among the buckets that generate the
uniform distribution, whereas there is only one for a Dirac distribution. In
fact, the number of possibilities of how to distribute $N$ elements among $n$
buckets with $N_{i}$ elements in the $i$th bucket is

$\omega\coloneqq\frac{N!}{N_{1}!\cdots N_{n}!}\,,$ (5)

because $N!$ is the number of possible permutations of all $N$ elements, which
overcounts by the number of permutations of elements inside the same bucket
and thus has to be divided by the number of permutations $N_{i}!$ for all
$i=1,\dots,n$. In the absence of any further measurement constraints, the
number of possibilities (5) is maximized by $N_{i}=N/n$ for all $i$, and thus
the typical distribution $p^{\ast}$ over $\Omega$ in this case is the uniform
distribution, i.e., $p^{\ast}(z_{i})=\frac{1}{n}$ for all $i$.

Consider now the problem of having to determine a typical distribution
$p^{\ast}$ over $\Omega$ such that the expected value
$\mathbb{E}_{p^{\ast}}[\mathcal{E}]\eqqcolon\langle\mathcal{E}\rangle_{p^{\ast}}$
of some quantity $\mathcal{E}$ equals a measured value $\varepsilon$. A simple
example would be the experiment of throwing $N$ dice and taking $\mathcal{E}$
to be the number of dots, i.e.,
$\mathcal{E}(z_{1})=1,\dots,\mathcal{E}(z_{6})=6$, and trying to find the
typical distribution $p^{\ast}$ over outcomes $z_{1},\dots,z_{6}$ under the
constraint that the average number of dots is, say $\varepsilon=2$. The
solution to this problem is analogous to the case of no constraints, but this
time we only consider realizations that are compatible with the measurement
constraint, that is we let $(N_{1},\dots,N_{n})$ belong to the set of
permissible occupation vectors

$\Gamma_{\varepsilon}\coloneqq\big{\\{}(N_{1},\dots,N_{n})\,\big{|}\,\langle\mathcal{E}\rangle_{p}=\varepsilon,\
p(z_{i})=\tfrac{N_{i}}{N}\ \forall i\big{\\}}.$

A typical distribution $p^{\ast}$ for a constraint $\varepsilon$ can then be
determined by a candidate in $\Gamma_{\varepsilon}$ with the maximum number
$\omega$ of possibilities (5). By assumption, $N$ is much larger than $n$, so
that we can get rid of the faculties by making use of Stirling’s approximation
$\ln N!=N\ln N-N+\mathcal{O}(\ln N)$. In particular, when letting
$N,N_{i}\to\infty$ such that $p(z_{i})=\frac{N_{i}}{N}$ remains finite, we
obtain

$\frac{1}{N}\log\omega=\underbrace{-\sum_{i=1}^{n}\frac{N_{i}}{N}\log\frac{N_{i}}{N}}_{=H(p)}+\,\mathcal{O}\left(\frac{\log
N}{N}\right)\ \stackrel{{\scriptstyle N\to\infty}}{{\longrightarrow}}\
H(p)\,.$

where $H(p)\coloneqq-\sum_{z\in\Omega}p(z)\log p(z)$ denotes the (Gibbs or
Shannon) entropy of $p$. Thus, instead of assessing typicality by maximizing
(5) in $\Gamma_{\varepsilon}$ for large but fixed $N$, we can get rid of the
$N$-dependency by simply maximizing $H$,

$p^{\ast}=\mathop{\mathrm{argmax}}_{p,\langle\mathcal{E}\rangle_{p}=\varepsilon}\
H(p)\,.$ (6)

This constrained optimization problem is known as the principle of maximum
entropy. The motivation given here is essentially the Wallis derivation
presented by Jaynes (Jaynes, , 2003).

####  3.1.2 Free energy from constraints and the Boltzmann distribution

The constrained optimization problem (6) can be translated into an
unconstrained problem by introducing a Lagrange multiplier $\beta$, known as
the inverse temperature due to the analogy to thermodynamics and the Helmholtz
Free Energy (4), which has to be chosen post hoc such that the constraint is
satisfied. This results in the minimization of the Lagrangian

$F(p)\coloneqq\langle\mathcal{E}\rangle_{p}\,{-}\,\tfrac{1}{\beta}H(p),$ (7)

which takes the form of a free energy (3). As we shall see later, $F$ takes
its minimum at the Boltzmann distribution known from statistical mechanics,
given by

$p^{\ast}(z)\coloneqq\frac{1}{\mathcal{Z}}\,e^{-\beta\mathcal{E}(z)},$ (8)

where $\mathcal{Z}=\sum_{z\in\Omega}e^{-\beta\mathcal{E}(z)}$ denotes the
normalization constant.

Note that, the argument in the previous section implicitly assumes a uniform
reference distribution, because the buckets are assumed to be equiprobable.
When replacing this assumption by the assumption of a general distribution
$p_{0}$ over $\Omega$, we obtain the principle of minimum relative entropy
(Rosenkrantz, , 1983), where the so-called Kullback-Leibler (KL) divergence
$D_{\mathrm{KL}}(p\|p_{0})=\langle\log(p/p_{0})\rangle_{p}$ is minimized with
respect to $p$ subject to a constraint
$\langle\mathcal{E}\rangle_{p}=\varepsilon$. Analogous to the maximum entropy
principle, this translates to the unconstrained minimization of the Lagrangian

$F(p,p_{0})\coloneqq\langle\mathcal{E}\rangle_{p}\,{+}\,\tfrac{1}{\beta}D_{\mathrm{KL}}(p\|p_{0}),$
(9)

with solution given by
$p^{\ast}(z)=\frac{1}{\mathcal{Z}}\,p_{0}(z)\,e^{-\beta\mathcal{E}(z)}$.

####  3.1.3 The trade-off between energy and uncertainty

Figure 2: Minimizing the free energy from constraints (7) requires to trade
off the competing terms of energy $\langle\mathcal{E}\rangle_{p}$ and entropy
$H(p)$, here shown exemplarily for the case of three elements. Assuming there
exists a unique minimal element
$z^{\ast}=\mathop{\mathrm{argmin}}_{z}\mathcal{E}(z)$, then minimizing only
$\langle\mathcal{E}\rangle_{p}$ over all probability distributions $p$ results
in the (Dirac delta) distribution $\delta_{z^{\ast}}$ that assigns zero
probability to all $z_{i}\not=z^{\ast}$ and probability one to $z_{i}\hskip
2.0pt{=}\hskip 2.0ptz^{\ast}$, and therefore has zero entropy. In contrast,
minimizing only the term $-H(p)/\beta$ is equivalent to maximizing $H(p)$ and
therefore would result in the uniform distribution that gives equal
probability to all elements. The resulting Boltzmann distribution $p^{\ast}$
interpolates between these two extreme solutions of minimal energy
($\beta\to\infty$) and maximum entropy ($\beta\to 0$).

An important feature of the minimization of the free energies (7) and (9)
consists in the balancing of the two competing terms of energy and entropy
(cf. Fig 2). This trade-off between maximal uncertainty (uniform distribution,
or $p_{0}$) on the one hand and minimal energy (e.g. a delta distribution) on
the other hand is the core of the maximum entropy principle. The inverse
temperature $\beta$ plays the role of a trade-off parameter that controls how
these two counteracting forces are weighted.

The maximum entropy principle goes back to the principle of insufficient
reason (Bernoulli, , 1713; de Laplace, , 1812; Poincaré, , 1912), which states
that two events should be assigned the same probability if there is no reason
to think otherwise. It has been hailed as a principled method to determine
prior distributions and to incorporate novel information into existing
probabilistic knowledge. In fact, Bayesian inference can be cast in terms of
relative entropy minimization with constraints given by the available
information (Williams, , 1980). Applications of this idea can also be found in
the machine learning literature, where subtracting (or adding) an entropy term
from an expected value of a function that must be optimized is known as
entropy regularization and plays an important role in modern reinforcement
learning algorithms (Williams and Peng, , 1991; Mnih et al., , 2016) to
encourage exploration (Haarnoja et al., , 2017) as well as to penalize overly
deterministic policies resulting in biased reward estimates (Fox et al., ,
2016).

From now on, we refer to a free energy expression that is motivated from a
trade-off between an energy and an entropy term, such as (7) and (9), as free
energy from constraints, in order to discriminate it from the notion of free
energy introduced in the following section, which—despite of its
resemblance—has a different motivation.

###  3.2 Variational free energy

There is another, distinct appearance of the term “free energy” outside of
physics, which is a priori not motivated from a trade-off between an energy
and entropy term, but from possible efficiency gains when representing Bayes’
rule in terms of an optimization problem. This technique is mainly used in
variational Bayesian inference (Koller, , 2009, Ch. 11), originally introduced
by Hinton and van Camp (Hinton and van Camp, , 1993). As before, for
simplicity all random variables are discrete, but most expressions can
directly be translated to the continuous case by replacing probability
distributions by probability densities and sums by the corresponding
integrals.

Figure 3: The normalization of a functon $\phi$ to obtain a probability
distribution $p_{\phi}$ is equivalent to fitting trial distributions $q$ to
the shape of $\phi$ by minimizing free energy. In two dimensions, the
normalization of a point $\phi=(\phi_{1},\phi_{2})$ corresponds to a (non-
orthogonal) projection onto the plane of probability vectors (A). For
continuous domains, where probability distributions are represented by
densities, normalization corresponds to a rescaling of $\phi$ such that the
area below the graph equals $1$ (B). Instead, when minimizing variational free
energy (red colour), the trial distributions $q$ are varied until they fit to
the shape of the unnormalized function $\phi$ (perfectly at $q=p_{\phi}$).

####  3.2.1 Variational Bayesian inference

As we have seen in Section 2, Bayesian inference consists in the calculation
of a conditional probability distribution over unknown variables given the
values of known variables. In the most simple case of two variables, say $X$
and $Z$, and a probabilistic model of the form
$p_{0}(X,Z)=p_{0}(X|Z)p_{0}(Z)$, Bayesian inference applies if $X$ is observed
and $Z$ is queried. Analogous to (2), the exact Bayes’ posterior $p(Z|X\hskip
1.0pt{=}\hskip 2.0ptx)$ is defined by the renormalization of $p_{0}(x,Z)$ in
order to obtain a distribution over $Z$ that respects the new information
$X\hskip 1.0pt{=}\hskip 2.0ptx$,

$p(Z|X\hskip 1.0pt{=}\hskip
2.0ptx)\,=\,\frac{p_{0}(x,Z)}{\mathcal{Z}(x)}\,\,=\,\frac{p_{0}(x|Z)\,p_{0}(Z)}{\mathcal{Z}(x)}\,\,,$
(10)

with the normalization constant $\mathcal{Z}(x)=\sum_{z}p_{0}(x,z)$.

In variational Bayesian inference, however, this Bayes’ posterior is not
calculated directly by renormalizing $p_{0}(x,Z)$ with respect to $Z$, but
indirectly by approximating it by a distribution $q(Z)$ that is adjusted
through the minimization of an error measure that quantifies the deviation
from the exact Bayes’ posterior. Importantly, the value of this error measure
can be determined without having to know the exact Bayes’ posterior. To see
this, note that the KL divergence between $q(Z)$ and $p(Z|X\hskip
1.0pt{=}\hskip 2.0ptx)$ can be written as

$\underbrace{\left\langle\log\frac{q(Z)}{p(Z|X\hskip 1.0pt{=}\hskip
2.0ptx)}\right\rangle_{q(Z)}}_{=\,D_{\mathrm{KL}}(q(Z)\|p(Z|X\hskip
1.0pt{=}\hskip 2.0ptx))}\ =\
\underbrace{\log\mathcal{Z}(x)}_{\text{indep.~{}of
}q}+\underbrace{\left\langle\log\frac{q(Z)}{p_{0}(x,Z)}\right\rangle_{q(Z)}}_{\eqqcolon\,F(q(Z)\|p_{0}(x,Z))}\,,$
(11)

i.e., it can be decomposed into the sum of a constant term and a term that
does not depend on the normalization $\mathcal{Z}(x)$. In particular, a good
approximation $q(Z)$ of the exact Bayes’ posterior (10) will effectively
minimize this KL divergence, which—due to (11)—can be done by minimizing
$F(q(Z)\|p_{0}(x,Z))$. In particular, the optimium of this minimization is
exactly achieved at the Bayes’ posterior (10),

$\mathop{\mathrm{argmin}}_{q(Z)}\left\langle\log\frac{q(Z)}{p_{0}(x,Z)}\right\rangle_{q(Z)}=\
p(Z|X\hskip 1.0pt{=}\hskip 2.0ptx)\,\,,$ (12)

which is known as the variational characterization of Bayes’ rule. This result
is a special case of (14) in the following section.

Figure 4: In variational Bayesian inference, the operation of renormalizing
the probabilistic model $p_{0}$ evaluated at an observation $X\hskip
1.0pt{=}\hskip 2.0ptx$ (Bayes’ rule), is replaced by an optimization problem.
In practice, this variational representation is often exploited to simplify a
given inference problem, either by reducing the seach space of distributions,
for example through a restrictive parametrization resulting in approximate
inference, or by splitting up the optimization into multiple partial
optimization steps that are potentially easier to solve than the original
problem but might still converge to the exact solution. These two
simplifications can also be combined, for example in the case of mean-field
assumptions where the space of distributions is reduced and an efficient
iterative inference algorithm is obtained at the same time.

####  3.2.2 Variational free energy, an extension of relative entropy

Any non-negative function $\phi$ on a finite space $\Omega$, can be normalized
to obtain a probability distribution $p_{\phi}=\phi/\sum_{z}\phi(z)$ on
$\Omega$ that differs from $\phi$ only by a scaling constant. In cases when it
is not beneficial to carry out the sum $\sum_{z}\phi(z)$ explicitly, such a
normalization might be replaced by the minimization of the variational free
energy

$F(q\|\phi)\coloneqq\left\langle\log\frac{q(Z)}{\phi(Z)}\right\rangle_{q(Z)},$
(13)

with respect to the so-called trial distributions $q$, because we have

$\displaystyle\mathop{\mathrm{argmin}}_{q}F(q\|\phi)$ $\displaystyle\ =\
\frac{\phi(Z)}{\sum_{z}\phi(z)}\ =\ p_{\phi}(Z)\,.$ (14)

Thus, instead of normalizing $\phi$ directly, one fits auxiliary distributions
$q$ to approximate the shape of $\phi$ in the space of probability
distributions (cf. Fig 3). If this optimization process has no constraints,
then the trial distributions are adjusted until $p_{\phi}$ is achieved. In the
case of constraints, for instance if the trial distributions are parametrized
by a non-exhaustive parametrization (e.g., Gaussians), then the optimized
trial distributions approximate $p_{\phi}$ as close as possible within this
parametrization. The minimal value of $F(q\|\phi)$ is

$F(p_{\phi}\|\phi)\,=\,\min_{q}F(q\|\phi)\,=\,-\log\sum_{z}\phi(z)\,.$ (15)

In particular, this implies that $-F(q\|\phi)\leq\log\sum_{z}\phi(z)$ for all
$q$, so that varying $-F(q\|\phi)$ with arbitrary trial distributions $q$
always provides a lower bound to the unknown normalization constant
$\sum_{z}\phi(z)$. In Bayesian inference this is the normalization constant in
Bayes’ rule and called the model evidence, which is why the negative
variational free energy is also called evidence lower bound (ELBO).

The proof of (14) and (15) directly follows from Jensen’s inequality and only
relies on the concavity of the logarithm. As we have seen in the previous
section, in variational Bayesian inference, the reference $\phi$ usually takes
the form of a joint distribution evaluated at the observed variables, e.g.,
$\phi(Z)=p_{0}(x,Z)$ in which case (14) recovers (12). The variational free
energy (13) is a free energy in the sense of (3) since by the additivity of
the logarithm under multiplication ($\log ab=\log a+\log b$),

$F(q\|\phi)=\langle-\log\phi\rangle_{q}-H(q)$ (16)

with energy term $\langle-\log\phi\rangle_{q}$ and entropy term $H(q)$. Note
that, for the choice $\phi=e^{-\beta\mathcal{E}}$, Equation (14) becomes the
Boltzmann distribution (8) and the variational free energy (16) formally
corresponds to the free energy from constraints (7).

Variational free energy can be regarded as an extension of relative entropy
with the reference distribution being replaced by a non-normalized reference
function, since in the case when $\phi$ is already normalized, that is if
$\sum_{z}\phi(z)=1$, then the free energy (13) coincides with the KL
divergence $D_{\mathrm{KL}}(q\|\phi)$. In particular, while relative entropy
is a measure for the dissimilarity of two probability distributions, where the
minimum is achieved if both distributions are equal, variational free energy
is a measure for the dissimilarity between a probability distribution $q$ and
a (generally non-normalized) function $\phi$, where the minimum with respect
to $q$ is achieved at $p_{\phi}$. Accordingly, we can think of the variational
free energy as a specific error measure between probability distributions and
reference functions. In principle, one could design many other error measures
that have the same minimum. This means that, a statement in a probabilistic
setting that a distribution $q^{\ast}$ minimizes a variational free energy
$F(q\|\phi)$ with respect to a given reference $\phi$, is analogous to a
statement in a non-probabilistic setting that some number $x=x^{\ast}$
minimizes the value of an error measure $\epsilon(x,y)$ (e.g., the squared
error $\epsilon(x,y)=(x-y)^{2}$) with respect to a given reference value $y$.

####  3.2.3 Approximate and iterative inference

Representing Bayes’ rule as an optimization problem over auxiliary
distributions $q$ has two main applications that both can simplify the
inference process (cf. Fig 4). First, it allows to approximate exact Bayes’
posteriors by restricting the optimization space, for example using a non-
exhaustive parametrization, e.g. an exponential family. Second, it enables
iterative inference algorithms consisting of multiple simpler optimization
steps, for example by optimizing with respect to each term in a factorized
representation of $q$ separately. A popular choice is the mean-field
approximation, which combines both of these simplifications, as it assumes
independence between hidden states, effectively reducing the search space from
joint distributions to factorized ones, and moreover it allows to optimize
with respect to each factor alternatingly. Note, however, that mean-field
approximations have limited use in sequential environments, where independence
of subsequent states cannot be assumed and therefore less restrictive
assumptions must be used instead (Opper and Saad, , 2001).

Many efficient iterative algorithms for exact and approximate inference can be
viewed as examples of variational free energy minimization, for example the EM
algorithm (Dempster et al., , 1977; Neal and Hinton, , 1998), belief
propagation (Pearl, , 1988; Yedidia et al., , 2001), and other message passing
algorithms(Minka, , 2001; Wainwright et al., , 2005; Winn and Bishop, , 2005;
Minka, , 2005; Yedidia et al., , 2005). While the (Bayesian) EM algorithm
(Beal, , 2003) and Pearl’s belief propagation (Yedidia et al., , 2001) both
can be seen as minimizing the same variational free energy, just with
different assumptions on the approximate posteriors, in (Minka, , 2005), it is
shown that also many other message passing algorithms such as (Minka, , 2001;
Wainwright et al., , 2005; Winn and Bishop, , 2005) can be cast as minimizing
some type of free energy, the only difference being the choice of the
divergence measure as the entropy term. Simple versions of these algorithms
have often existed before their free energy formulations were available, but
the variational representations usually allowed for extensions and
refinements—see (Csiszár and Tusnády, , 1984; Hathaway, , 1986; Neal and
Hinton, , 1998; Beal, , 2003) in case of EM and (Yedidia et al., , 2001;
Heskes, , 2003; Yuille, , 2002; Yedidia et al., , 2005) in case of message
passing.

We are now turning to the question of how the two notions of free energy
introduced in this section are related to recent theories of intelligent
agency.

##  4 Free energy from constraints in information processing

###  4.1 The basic idea

The concept of free energy from constraints as a trade-off between energy and
uncertainty can be used in models of perception-action systems, where entropy
quantifies information processing complexity required for decision-making
(e.g., planning a path for fleeing a predator) and energy corresponds to
performance (e.g., distinguishing better and worse flight directions). The
notion of decision in this context is very broad and can be applied to any
internal variable in the perception-action pipeline (Kahneman, , 2002), that
is not given directly by the environment. In particular, it also subsumes
perception itself, where the decision variables are given by the hidden causes
that are being inferred from observations.

In rational choice theory (von Neumann and Morgenstern, , 1944), a decision-
maker selects decisions $x^{\ast}$ from a set of options $\Omega$ such that a
utility function $U$ defined on $\Omega$ is maximized,

$x^{\ast}=\mathop{\mathrm{argmax}}_{x\in\Omega}\,U(x)\,.$ (17)

The utility values $U(x)$ could either be objective, for example a monetary
gain, or subjective in which case they represent the decision-maker’s
preferences. In general, the utility does not have to be defined directly on
$\Omega$, but could be derived from utility values that are attached to
certain states, for example to the configurations of the playboard in a board
game. In the case of perception, utility values are usually given by
(log-)likelihood functions, in which case utility maximization without
constraints corresponds to greedy inference such as maximum likelihood
estimation. Note that, for simplicity, in this section we consider one-step
decision problems. Sequential tasks can either be seen as multiple one-step
problems where the utility of a given step might depend on the policy over
future steps, or as path planning problems where an action represents a full
action path or policy (Whittle, , 1990; Tishby and Polani, , 2011; Grau-Moya
et al., , 2016; Gottwald and Braun, 2019b, ).

Figure 5: A: Decision-making can be considered as a search process in the
space of options $\Omega$, where options are progressively ruled out.
Deliberation costs are defined to be monotone functions under such uncertainty
reduction. B: Exemplary efficiency curve resulting from the trade-off between
utility and costs, that separates non-optimal from non-admissible behavior.
The points on the curve correspond to bounded-optimal agents that optimally
trade off utility against uncertainty, analogous to the rate-distortion curve
in information theory.

While ideal rational decision-makers are assumed to perfectly optimize a given
utility function $U$, real behavior is often stochastic, meaning that multiple
exposures to the same problem lead to different decisions. Such non-
deterministic behavior could be a consequence of model uncertainty, as in
Bayesian inference or various stochastic gambling schemes, or a consequence of
_satisficing_ (Simon, , 1955), where decision-makers do not choose the single
best option, but simply one option that is good enough. Abstractly, this means
that, the choice of a single decision is replaced by the choice of a
distribution over decisions. More generally, also considering prior
information that the decision-maker might have from previous experience, the
process of deliberation during decision-making might be expressed as the
transformation of a prior $p_{0}$ to a posterior distribution $p$.

When assuming that deliberation has a cost $C(p,p_{0})$, then arriving at
narrow posterior distributions should intuitively be more costly than choosing
distributions that contain more uncertainty (cf. Fig 5A). In other words,
deliberation costs must be increasing with the amount of uncertainty that is
reduced by the transformation from $p_{0}$ to $p$. Uncertainty reduction can
be understood as making the probabilities of options less equal to each other,
rigorously expressed by the mathematical concept of majorization (Marshall et
al., , 2011). This notion of uncertainty can also be generalized to include
prior information, so that the degree of uncertainty reduction corresponds to
more or less deviations from the prior (Gottwald and Braun, 2019a, ).

Maximizing expected utility $\langle U\rangle_{p}$ with respect to $p$ under
restrictions on processing costs $C(p,p_{0})$ is a constrained optimization
problem that can be interpreted as a particular model of bounded rationality
(Simon, , 1955), explaining non-rational behavior of decision-makers that may
be unable to select the single best option by their limited information
processing capability. Similarly to the free energy trade-off between energy
and entropy (cf. Fig 2), this results in a trade-off between utility $\langle
U\rangle_{p}$ and processing costs $C(p,p_{0})$,

$F_{\beta}(p)\coloneqq\langle U\rangle_{p}-\tfrac{1}{\beta}C(p,p_{0}).$ (18)

Here, the trade-off parameter $\beta$ is analogous to the inverse temperature
in statistical mechanics (cf. Equation (7)) and parametrizes the optimal
trade-offs $p_{\beta}^{\ast}\hskip 2.0pt{=}\hskip
2.0pt\mathop{\mathrm{argmax}}_{p}F_{\beta}(p)$ between utility and cost, that
define an efficiency frontier separating the space of perception-action
systems into bounded-optimal, non-optimal, and non-admissible systems (cf. Fig
5).

When assuming that the total transformation cost is the same independent of
whether a decision problem is solved in one step or multiple sub-steps
(additivity under coarse-graining) the trade-off in (18) takes the general
form (3) of a free energy in the sense of energy (utility) minus entropy
(cost), because then the cost function is uniquely given by the relative
entropy

$C(p,p_{0})=D_{\mathrm{KL}}(p\|p_{0}).$ (19)

Note that the additivity of (19) also implies a coarse-graining property of
the free energy (18) in the case when the decision is split into multiple
steps, such that the utility of preceding decisions is effectively given by
the free energy of following decisions. Therefore, in this case, free energy
can be seen as a _certainty-equivalent_ value of the subordinate decision
problems, i.e. the amount of utility the agent would have to receive to be
indifferent between this guaranteed utility and the potential expected utility
of the subsequent decision steps taking account the associated information
processing costs. The special case (19) has been studied extensively in
multiple contexts, including quantal response equilibria in the game-theoretic
literature (McKelvey and Palfrey, , 1995; Wolpert, , 2006), rational
inattention and costly contemplation (Sims, , 2003; Ergin and Sarver, , 2010),
bounded rationality with KL costs (Mattsson and Weibull, , 2002; Ortega and
Braun, , 2013), KL control (Todorov, , 2009; Kappen et al., , 2012), entropy
regularization (Williams and Peng, , 1991; Mnih et al., , 2016), robustness
(Maccheroni et al., , 2006; Hansen and Sargent, , 2008), the emergence of
heuristics (Binz et al., , 2020), thermodynamic models of computation
(Wolpert, , 2019), and the analysis of information flow in perception-action
systems (Tishby and Polani, , 2011; Still, , 2009). While (19) is often
regarded as an abstract measure of uncertainty reduction or a generic proxy
for information processing costs, it can also be viewed as a physical capacity
constraint, where the information that is required to achieve a certain
expected utility is considered to be sent over a channel to the actuator
(Miller, , 1956; Garner, , 1962; MacRae, , 1970; Tatikonda and Mitter, , 2004;
Bhui and Gershman, , 2018). This view is also consistent with the maximum
entropy principle, as (18) and (19) favor distributions $p$ that can be
generated from $p_{0}$ most easily in terms of statistics, and therefore with
minimum communication complexity between $p_{0}$ and $p$ (Harsha et al., ,
2010).

Figure 6: Overview of how to apply utility maximization with information
processing costs to the example from Section 2.

###  4.2 A Simple Example

Ingredients. Consider the probabilistic model shown in Fig 1 with the joint
distribution $p_{0}(\mathbf{X},\mathbf{S},A)$ that is specified by the factors
in the decomposition (1). Here, $S$ and $X$ denote the current environmental
state and the corresponding observation, and $A$ denotes the action that must
be determined in order to drive the system into a new state $S^{\prime}$ with
observation $X^{\prime}$. The decision-making problem is specified by assuming
that we have given a utility function $U$ over future observations
$X^{\prime}$ which the decision-maker seeks to maximize by selecting an action
$A$, while only having access to the current observation $X$. This means that
the decision-maker has control over the distribution $p(A|X)$, which replaces
the prior $p_{0}(A)$ in the factorization (1) of the prior model
$p_{0}(\mathbf{X},\mathbf{S},A)$ to determine the factorization of the
posterior model $p(\mathbf{X},\mathbf{S},A)$ in terms of the fixed components
in $p_{0}$ (cf. Fig 6) as

$p(\mathbf{X},\mathbf{S},A)=\underbrace{p_{0}(X^{\prime}|S^{\prime})\,p_{0}(X|S)\,p_{0}(S^{\prime}|S,A)\,p_{0}(S)}_{p_{0}(\mathbf{X},\mathbf{S}|A)}\,p(A|X)\,.$
(20)

Free energy from constraints. Further assuming that the decision-maker is
subject to an information processing constraint
$D_{\mathrm{KL}}(p\|p_{0})\,{\leq}\,C_{0}$, for some non-negative bound
$C_{0}$, results in the unconstrained optimization problem $\max_{p}F(p)$ with
free energy given by (18), where the trade-off parameter $\beta$ is tuned to
comply with the bound $C_{0}$. Since the action distribution $p(A|X)$ is the
only distribution in the posterior model (20) that changes during decision-
making, i.e., during the transformation from prior to posterior, the total
free energy simplifies to

$\displaystyle F(p)$ $\displaystyle=\langle
U\rangle_{p(\mathbf{X},\mathbf{S},A)}-\tfrac{1}{\beta}D_{\mathrm{KL}}(p(\mathbf{X},\mathbf{S},A)\|p_{0}(\mathbf{X},\mathbf{S},A))$
$\displaystyle=\langle
V(X,A)\rangle_{p(A|X)p(X)}-\tfrac{1}{\beta}\big{\langle}D_{\mathrm{KL}}(p(A|X)\|p_{0}(A))\big{\rangle}$
$\displaystyle=\langle F_{A}(p(A|X))\rangle_{p(X)}\,,$

where we have written $p_{0}(x|s)p_{0}(s)=p(s|x)p(x)$ using Bayes rule (2),
and

$\displaystyle V(X,A)$
$\displaystyle\coloneqq\sum\nolimits_{s,s^{\prime},x^{\prime}}p(s|X)\,p_{0}(s^{\prime}|s,A)\,p_{0}(x^{\prime}|s^{\prime})\,U(x^{\prime})\,,$
$\displaystyle F_{A}(p(A|X))$ $\displaystyle\coloneqq\langle
V(X,A)\rangle_{p(A|X)}-\tfrac{1}{\beta}D_{\mathrm{KL}}(p(A|X)\|p_{0}(A))\,.$

Note that, here the expectation with respect to $p(X)$ does not affect the
optimization with respect to $p(A|X)$ since it can be performed pointwise for
each particular realization $x$ of $X$. In fact, we would have obtained the
same result when conditioning on an arbitrary value $X\hskip 1.0pt{=}\hskip
2.0ptx$ from the outset. However, in general, optimal information processing
strategies may depend on the entire distribution $p(X)$ and can therefore not
be obtained from only considering single observations $x$, for example when
also optimizing with respect to the prior $p_{0}(A)$, see e.g., (Genewein et
al., , 2015).

Free energy maximization. The optimal action distribution $p^{\ast}(A|X)$
maximizing $F_{A}$ is a Boltzmann distribution (8) with “energy” $V(X,A)$ and
prior $p_{0}(A)$,

$p^{\ast}(A|X)=\frac{1}{\mathcal{Z}(X)}\,p_{0}(A)\,e^{\beta V(X,A)}\,,$ (21)

where $\mathcal{Z}(X)\coloneqq\sum_{a}p_{0}(a)e^{\beta V(X,a)}$. Note that in
order to evaluate the utility $V$, it is required to determine the Bayes’
posterior $p(S|X)$. This shows how in a utility-based approach, the need to
perform Bayesian inference results directly from the assumption about which
variables are observed and which are not.

###  4.3 Critical points

The main idea of free energy in the context of information processing with
limited resources is that any computation can be thought of abstractly as a
transformation from a distribution $p_{0}$ of prior knowledge to a posterior
distribution $p$ that encapsulates an advanced state of knowledge resulting
from deliberation. The progress that is made through such a transformation is
quantitatively captured by two measures: the expected utility $\langle
U\rangle_{p}$ that quantifies the quality of $p$ and $C(p,p_{0})$ that
measures the cost of uncertainty reduction from $p_{0}$ to $p$. Clearly, the
critical point of this framework is the choice of the cost function $C$. In
particular, we could ask whether there is some kind of universal cost function
that is applicable to any perception-action process or whether there are only
problem-specific instantiations. Of course, having a universal measure that
allows applying the same concepts to extremely diverse systems is both a boon
and a bane, because the practical insights it may provide for any concrete
instance could be very limited. This is the root of a number of critical
issues:

  1. $(i)$

What is the cost $C$? An important restriction of all deliberation costs of
the form $C(p,p_{0})$ is that they only depend on the initial and final
distributions and ignore the process of how to get from $p_{0}$ to $p$. When
varying a single resource (e.g., processing time) we can use $C(p,p_{0})$ as a
process-independent _proxy_ for the resource. However, if there are multiple
resources involved (e.g., processing time, memory, and power consumption), a
single cost cannot tell us how these resources are weighted optimally without
making further process-dependent assumptions. In general, the theory makes no
suggestions whatsoever about mechanical processes that could implement
resource-optimal strategies, it only serves as a baseline for comparison.
Finally, simply requiring the measure to be monotonic in the uncertainty
reduction, does not uniquely determine the form of $C$, as there have been
multiple proposals of uncertainty measures in the literature (see e.g.
(Csiszár, , 2008)), where relative entropy is just one possibility. However,
relative entropy is distinguished from all other uncertainty measures in its
additivity property, that for example allows to express optimal probabilistic
updates from $p_{0}$ to $p$ in terms of additions or subtractions of
utilities, such as log-likelihoods for evidence accumulation in Bayesian
inference.

  2. $(ii)$

What is the utility? When systems are engineered, utilities are usually
assumed to be given such that desired behavior is specified by utility
maximization. However, when we observe perception-action systems, it is often
not so clear what the utility should be, or in fact, whether there even exists
a utility that captures the observed behavior in terms of utility
maximization. This question of the identifiability of a utility function is
studied extensively in the economic sciences, where the basic idea is that
systems reveal their preferences through their actual choices and that these
preferences have to satisfy certain consistency axioms in order to guarantee
the existence of a utility function. In practice, to guarantee unique
identifiability these axioms are usually rather strong, for example ignoring
the effects of history and context when choosing between different items, or
ignoring the possibility that there might be multiple objectives. When not
making these strong assumptions, utility becomes a rather generic concept,
like the concept of probability, and additional assumptions like soft-
maximization are necessary to translate from utilities to choice
probabilities.

  3. $(iii)$

The problem of infinite regress. One of the main conceptual issues with the
interpretation of $C$ as a deliberation cost is that the original utility
optimization problem is simply replaced by another optimization problem that
may even be more difficult to solve. This novel optimization problem might
again require resources to be solved and could therefore be described by a
higher-level deliberation cost, thus leading to an infinite regress. In fact,
any decision-making model that assumes that decision-makers reason about
processing resources are affected by this problem (Russell and Subramanian, ,
1995; Gigerenzer and Selten, , 2001). A possible way out is to consider the
utility-information trade-off simply an as if description, since perception-
action systems that are subject to a utility-information trade-off do not
necessarily have to reason or know about their deliberation costs. It is
straightforward, for example, to design processes that probabilistically
optimize a given utility with no explicit notion of free energy, but for an
outside observer the resulting choice distribution looks like an optimal free
energy trade-off (Ortega and Braun, , 2014).

In summary, the free energy trade-off between utility and information
primarily serves as a normative model for optimal probability assignments in
information-processing nodes or networks. Like other Bayesian approaches, it
can also serve as a guide for constructing and interpreting systems, although
it is in general not a mechanistic model of behavior. In that respect it
shares the fate of its cousins in thermodynamics and coding theory (Shannon, ,
1948) in that they provide theoretical bounds on optimality but devise no
mechanism for processes to achieve these bounds.

##  5 Variational free energy in Active Inference

###  5.1 The basic idea

Variational free energy is the main ingredient used in the _Free Energy
Principle_ for biological systems in the neuroscience literature (Friston, ,
2005, 2010; Friston et al., , 2015, 2006), which has been considered as
“arguably the most ambitious theory of the brain available today” (Gershman, ,
2019). Since variational free energy in itself is just a mathematical
construct to measure the dissimilarity between distributions and functions—see
Section 3—, the biological content of the Free Energy Principle must come from
somewhere else. The basic biological phenomenon that the Free Energy Principle
purports to explain is homeostasis, the ability to actively maintain certain
relevant variables (e.g., blood sugar) within a preferred range. Usually,
homeostasis is applied as an explanatory principle in physiology whereby the
actual value of a variable is compared to a target value and corrections to
deviation errors are made through a feedback loop. However, homeostasis has
also been proposed as an explanatory principle for complex behavior in the
cybernetic literature (Wiener, , 1948; Ashby, , 1960; Powers, , 1973; Cisek, ,
1999)—for example, maintaining blood sugar may entail complex feedback loops
of learning to hunt, to trade and to buy food. Crucially, being able to
exploit the environment in order to attain favorable sensory states, requires
implicit or explicit knowledge of the environment that could either be pre-
programmed (e.g., insect locomotion) or learnt (e.g., playing the piano).

The Free Energy Principle was originally suggested as a theory of cortical
responses (Friston, , 2005) by promoting the free energy formulation of
predictive coding that was introduced by Dayan and Hinton with the Helmholtz
machine (Dayan et al., , 1995). It found its most recent incarnation in what
is known as Active Inference that attempts to extend variational Bayesian
inference to the problem of action selection. Here, the target value of
homeostasis is expressed through a probability distribution $p_{\mathrm{des}}$
under which desired sensory states have a high probability. The required
knowledge about the environment is expressed through a generative model
$p_{0}$ that relates observations, hidden causes, and actions. As the
generative model allows to make predictions about future states and
observations, it enables to choose actions in such a way that the predicted
consequences conform to the desired distribution. In Active Inference, this is
achieved by merging the generative and the desired distributions, $p_{0}$ and
$p_{\mathrm{des}}$, into a single reference function $\phi$ to which trial
distributions $q$ over the unknown variables are fitted by minimizing the
variational free energy $F(q\|\phi)$. This free energy minimization is
analogous to variational Bayesian inference, where the reference is always
given by a joint distribution evaluated at observed quantities (cf. Section
3.2.1). In the resulting homeostatic process, the trial distributions $q$ play
the role of internal variables that are manipulated in order to achieve
desired sensory consequences that are not directly controllable. Minimizing
variational free energy by the alternating variation of trial distributions
over actions $q_{\text{Actions}}$ and trial distributions over hidden states
$q_{\text{States}}$,

$\displaystyle\underbrace{\min_{q_{\text{Actions}}}F(q\|\phi)}_{\textrm{Action}}\quad\textrm{and}\quad\underbrace{\min_{q_{\text{States}}}F(q\|\phi)}_{\textrm{Perception}},$
(22)

is then equated with processes of action and perception.

Figure 7: Overview of the Active Inference recipe, applied to our example from
Fig 1.

In a nutshell, the central tenet of the Free Energy Principle states that
organisms maintain homeostasis through minimization of variational free energy
between a trial distribution $q$ and a reference function $\phi$ by acting and
perceiving. Sometimes the even stronger statement is made that minimizing
variational free energy is _mandatory_ for homeostatic systems (Friston, ,
2013; Corcoran and Hohwy, , 2018).

###  5.2 A Simple Example

Ingredients. Applying the Active Inference recipe (cf. Fig 7) to our running
example from Fig 1 with current and future states $S,S^{\prime}$, current and
future observations $X,X^{\prime}$, and action $A$, we need a generative model
$p_{0}$, a desired distribution $p_{\mathrm{des}}$, and trial distributions
$q$. The generative model $p_{0}(\mathbf{X},\mathbf{S},A)$ is specified by the
factors in the decomposition (1), the desired distribution
$p_{\mathrm{des}}(X^{\prime})$ is a given fixed probability distribution over
future sensory states $X^{\prime}$, and the trial distributions $q$ are
probability distributions over all unknown variables,
$S,S^{\prime},X^{\prime}$, and $A$.

In most treatments of Active Inference in the literature, the trial
distributions $q$ are simplified, either by a full mean-field approximation
over states and actions (Friston et al., , 2013, 2015), by a partial mean-
field approximation where the dependency on actions is kept but the states are
treated independently of each other (Friston et al., , 2016; Friston et al.,
2017a, ), or more recently (Schwöbel et al., , 2018; Parr et al., , 2019) by
the so-called Bethe approximation (Yedidia et al., , 2001; Heskes, , 2003),
where subsequent states are allowed to interact. In the partial mean-field
assumption of (Friston et al., , 2016), the trial distribution over
$X^{\prime}$ is fixed and given by $p_{0}(X^{\prime}|S^{\prime})$, while for
$A$, $S$ and $S^{\prime}$ the trial distributions are variable but restricted
to be of the mean-field form for $S$ and $S^{\prime}$,

$q(\mathbf{S},A)=q(S)\,q(S^{\prime}|A)\,q(A),$ (23)

i.e., the hidden states $S$ and $S^{\prime}$ are assumed to be independent
given $A$. While mean-field approximations can be good enough for simple
perceptual inference, where a single hidden cause might be responsible for a
set of observations, they can be too strong simplifications for sequential
decision-making problems where the next state $S^{\prime}$ depends on the
previous state $S$. In fact, as can be seen for example in S.2, mean-field
assumptions may fail to show goal-directed behavior even for very simple tasks
such as the navigation in a grid world. A less restrictive assumption would be
a Bethe approximation, a special case of Kikuchi’s cluster variation method
(Kikuchi, , 1951), which allows $S$ and $S^{\prime}$ as well as $S^{\prime}$
and $X^{\prime}$ to be stochastically dependent—cf. Section C in Appendix A.1,
where we derive the update equations under the Bethe assumption for the simple
example of this section. In general, the Bethe approximation achieves exact
marginals in tree-like models, such as the models that are considered in the
Active Inference literature, because it results in update equations that are
equivalent to Pearl’s belief propagation algorithm (Yedidia et al., , 2001;
Pearl, , 1988).

Reference function. The reference $\phi$ is constructed by combining the two
distributions $p_{\mathrm{des}}$ and $p_{0}$. To do so, there have been
several proposals in the Active Inference literature, which fall into one of
two categories: either a specific value function $Q$ is defined (containing
$p_{\mathrm{des}}$), which is multiplied to the generative model using a soft-
max function (Friston et al., , 2015, 2016; Friston et al., 2017a, ),

$\phi(X^{\prime},\mathbf{S},A)\coloneqq p_{0}(X\hskip 1.0pt{=}\hskip
2.0ptx,X^{\prime},\mathbf{S}|A)\,\tfrac{1}{\mathcal{Z}}\,p_{0}(A)e^{Q(A)}\,,$
(24)

or the desired distribution is multiplied directly to the generative model
(Schwöbel et al., , 2018),

$\phi(X^{\prime},\mathbf{S},A)\coloneqq
p_{\mathrm{des}}(X^{\prime})\,p_{0}(X\hskip 1.0pt{=}\hskip
2.0ptx,X^{\prime},\mathbf{S},A).$ (25)

While the reference function in (25) is already completely specified, we still
need to know how to determine the value function $Q$ in the case of (24). For
the partial mean-field assumption (23) it is defined in the literature
(Friston et al., , 2016; Friston et al., 2017a, ) as

$Q(a)\coloneqq\langle
U(X^{\prime},S^{\prime})\rangle_{q(X^{\prime},S^{\prime}|A\hskip
1.0pt{=}\hskip 2.0pta)}+H\big{(}q(X^{\prime}|A\hskip 1.0pt{=}\hskip
2.0pta)\big{)},$ (26)

where $U(x^{\prime},s^{\prime})\coloneqq\log p_{\mathrm{des}}(x^{\prime})+\log
p_{0}(x^{\prime}|s^{\prime})$ favors both desirable and plausible future
observations $x^{\prime}$. While here desirability and plausibility is built
into the value function $Q$ idiosyncratically, in utility-based approaches
(cf. Section 4.2) only desirability has to be put into the design of the
utility function, because there the likelihood $p_{0}(X^{\prime}|S^{\prime})$
of future observations is automatically taken into account by the expected
utility $V$ that is (soft-)maximized by (21). Moreover, since $Q$ can be
rewritten as

$Q(a)=-D_{\mathrm{KL}}\big{(}q(X^{\prime}|A)\|p_{\mathrm{des}}(X^{\prime})\big{)}-\big{\langle}H\big{(}p_{0}(X^{\prime}|S^{\prime})\big{)}\big{\rangle}_{q(S^{\prime}|A)}\,,$

the extra entropy term in (26) has the effect of actions leading to
consequences that more or less _match_ the desired distribution, while also
explicitly punishing actions that lead to a high variability of observations
(by requiring a low average entropy of $p_{0}(X^{\prime}|S^{\prime})$), rather
than trying to produce the single most desired outcome—see the discussion at
the end of Section 5.3. Note also that the value function $Q$ depends (non-
linearly) on the trial distribution $q(S^{\prime}|A)$, because
$q(X^{\prime}|A)=\sum_{s^{\prime}}p_{0}(X^{\prime}|s^{\prime})q(s^{\prime}|A)$
is itself a function of $q(S^{\prime}|A)$, which is problematic during free
energy minimization (see $(ii)$ in Section 5.3).

Free energy minimization. Once the form of the trial distributions $q$—e.g. by
a partial mean-field assumption (23) or a Bethe approximation (see Derivation
of exemplary update equations)—and the reference $\phi$ are defined, the
variational free energy is simply determined by $F(q\|\phi)$. In the case of a
mean-field assumption, the resulting free energy minimization problem is
solved approximately by performing an alternating optimization scheme, in
which the variational free energy is minimized separately with respect to each
of the variable factors in a factorization of $q$, for example by alternating
between $\min_{q(S)}F$, $\min_{q(S^{\prime}|A)}F$, and $\min_{q(A)}F$ in the
case of the partial mean-field assumption (23), where in each step the factors
that are not optimized are kept fixed (cf. Fig 7). In Derivation of exemplary
update equations we derive the update equations for the cases (24) and (25)
under mean-field and Bethe approximations for the one-step example discussed
in this section. Mean-field solutions for the general case of arbitrarily many
timesteps together with their exact solutions can be found in Notebook:
Comparison of different formulations of Active Inference, where we also
highlight the theoretical differences between various proposed formulations of
Active Inference. The effect of some of these differences can be seen in the
grid world simulations in Notebook: Grid world simulations.

###  5.3 Critical points

The main idea behind Active Inference is to express the problem of action
selection in a similar manner to the perceptual problem of Bayesian inference
over hidden causes. In Bayesian inference, agents are equipped with likelihood
models $p_{0}(X|Z)$ that determine the desirability of different hypotheses
$Z$ under known data $X$. In Active Inference, agents are equipped with a
given desired distribution $p_{\mathrm{des}}(X^{\prime})$ over future outcomes
that ultimately determines the desirability of actions $A$. An important
difference that arises is that perceptual inference has to condition on past
observations $X\hskip 2.0pt{=}\hskip 2.0ptx$, whereas naive inference over
actions would have to condition on desired future outcomes $X^{\prime}\hskip
2.0pt{=}\hskip 2.0ptx^{\prime}$.

For a single desired future observation $x^{\prime}$, Bayesian inference could
be applied in a straightforward way by simply conditioning the generative
model $p_{0}$ on $X^{\prime}\hskip 1.0pt{=}\hskip 2.0ptx^{\prime}$. Similarly,
one could condition on a desired distribution $p_{\mathrm{des}}(X^{\prime})$
using Jeffrey’s conditioning rule (Jeffrey, , 1965), resulting in
$p(A|p_{\mathrm{des}})\hskip 2.0pt{=}\hskip
2.0pt\sum_{x^{\prime}}p(A|x^{\prime})\,p_{\mathrm{des}}(x^{\prime})$, which
could be implemented by first sampling a goal
$x^{\prime}\,{\sim}\,p_{\mathrm{des}}(X^{\prime})$ and then inferring
$p(A|x^{\prime})$ given the single desired observation $x^{\prime}$. However,
one of the problems with such a naive approach is that the choice of a goal is
solely determined by its desirability, whereas its realizability for the
decision-maker is not taken into account. This is because by conditioning on
$p_{\mathrm{des}}$, the decision-maker effectively seeks to choose actions in
order to reproduce or match the desired distribution.

To overcome this problem, Control as Inference or Planning as Inference
approaches in the machine learning literature (Toussaint and Storkey, , 2006;
Todorov, , 2008; Kappen et al., , 2012; Levine, , 2018; O’Donoghue et al., ,
2020) do not directly condition on desired future observations but on future
success by introducing an auxiliary binary random variable $R$ such that $R=1$
encodes the occurence of desired outcomes. The auxiliary variable $R$ comes
with a probability distribution $p_{0}(R|X^{\prime},...)$ that determines how
well the outcomes satisfy desirability criteria of the decision-maker, usually
defined in terms of the reward or utility attached to certain outcomes—see the
discussion in $(iii)$ below. The extra variable gives the necessary
flexibility to infer successful actions by simply conditioning on $R\hskip
1.0pt{=}\hskip 2.0pt1$. The advantage of such an approach over direct Jeffrey
conditionalization given a desired distribution over future observations can
be seen in the grid world simulations in Notebook: Grid world simulations,
especially the ability of choosing a desired outcome that is not only
desirable but also achievable—see also Fig 8.

Active Inference tries to overcome the same problem of reconciling
realizability and desirability, but without explicitly introducing extra
random variables and without explicitly conditioning on the future. Instead,
the desired distribution is combined with the generative model to form a new
reference function $\phi$ such that the posteriors $q^{\ast}$ resulting from
the minimization of the free energy $F(q\|\phi)$ contain a baked-in tendency
to reach the desired future encoded by $\phi$. This approach is the root of a
number of critical issues with current formulations of Active Inference:

  1. $(i)$

How to incorporate the desired distribution into the reference?

Instead of using Bayesian conditioning directly in order to condition the
generative model $p_{0}$ on the desired future, in Active Inference it is
required that the reference $\phi$ contains the desired distribution in a way
such that actions sampled from the resulting posterior model are more likely
if they lead to the desired future. As can be seen already for the one-step
case in (24) and (25), the method of how to incorporate the desired
distribution into the reference function is not unique and does not follow
from first principles. There have been essentially two different proposals in
the literature on Active Inference of how to combine the two distributions
$p_{\mathrm{des}}$ and $p_{0}$ into $\phi$ (cf. Fig 7): Either a hand-crafted
value function $Q$ is designed that specifically modifies the action
probability of the generative model, or the probability over futures
$X^{\prime}$ under the generative model $p_{0}$ is modified by directly
multiplying $p_{\mathrm{des}}$ to the likelihood
$p_{0}(X^{\prime}|S^{\prime})$. We discuss both of these proposals in $(ii)$
and $(iii)$ below.

  2. $(ii)$

Proposal 1: $Q$-value Active Inference (Friston et al., , 2013, 2015, 2016;
Friston et al., 2017a, ) In the most popular formulation of Active Inference,
the probability over actions in the reference $\phi$ is defined by
$\frac{1}{\mathcal{Z}}\,p_{0}(A)e^{Q(A)}$, where the value function $Q$ (also
called the “expected free energy”) depends non-linearly on the trial
distributions $q$, as can be seen exemplarily in (26) for the one-step case
under the partial mean-field assumption of Friston et al.,  (2016); Friston et
al., 2017a , where $q(S^{\prime}|A)$ enters $Q$ through
$q(X^{\prime}|A)=\sum_{s^{\prime}}p_{0}(X^{\prime}|s^{\prime})q(s^{\prime}|A)$.
Note that, because of this non-linearity the alternating free energy
minimization would have no closed-form solutions (cf. Derivation of exemplary
update equations). This means that both the trial distributions $q$ and the
reference $\phi=\phi(q)$ will change when $q$ is varied during the
minimization of the _total_ variational free energy $F(q\|\phi(q))$, as would
be required when stipulating a single free energy functional for optimization.
This highlights an important conceptual difference to variational Bayesian
inference, where one assumes a fixed reference $\phi$—resulting from the
evaluation of a fixed probabilistic model $p_{0}$ at known variables (see
Section 3.2.1)—to which distributions $q$ are fitted by minimizing
$F(q\|\phi)$. In contrast, when changing the reference $\phi(q)$ during the
optimization process, it is no longer clear what is actually achieved by this
minimization. As demonstrated by Notebook: Grid world simulations, this issue
has immediate practical implications, as respecting or ignoring the extra $q$
dependency can result in very different behavior even in simple grid world
simulations.
In the Active Inference literature, however, the extra $q$-dependency of $Q$
is largely ignored. Instead of optimizing the full free energy $F(q\|\phi(q))$
with respect to state and action distributions, one alternatingly optimizes
the free energy over states $F_{A}$ for each action $A$ and then the full free
energy with respect to action distributions only, so that action and
perception effectively optimize two different free energies. It is crucial to
note, however, that unlike in variational Bayesian inference with fixed
reference, this separation does not follow from the formalism of variational
free energy, but is a design choice of the Active Inference framework that
imposes this separation by force (see the Appendix Separation of model and
state variables for more details). This way, both separate optimizations can
be considered as variational inference in each single update, even though when
alternating them the reference $\phi$ still changes across the combined
optimization process. This is in contrast to alternating optimization schemes
in variational inference (e.g., in the Bayesian EM algorithm) where the
reference $\phi$ does not change between optimization steps. Thus, there are
two choices: Either Q-value Active Inference is regarded as some kind of
approximation to variational inference under a single total free energy, or
one has to give up the idea of a single free energy function that is
optimized. Either way, the combined process of action and perception does not
correspond to a single variational inference process.
Finally, another important practical issue with $Q$-value Active Inference
models is that the definition of $Q$ relies on a mean-field approximation of
the trial distributions $q$, under which hidden states are assumed to be
stochastically independent. This simplification is too strong for sequential
decision-making tasks, which renders the approach unfit for environments where
the current state depends stochastically on previous states (see Notebook:
Grid world simulations for a demonstration).

  3. $(iii)$

Proposal 2: direct Active Inference (Schwöbel et al., , 2018)

When multiplying $p_{\mathrm{des}}$ to the generative model directly, as in
(25), then the resulting reference $\phi$ is no longer given by a joint
distribution of observations, states, and actions (since in general
$\sum_{x^{\prime}}p_{\mathrm{des}}(x^{\prime})p_{0}(x^{\prime}|S^{\prime})\not=1$).
Instead, this formulation of Active Inference turns out to be a special case
of previous Control as Inference approaches in the machine learning literature
(Toussaint and Storkey, , 2006; Levine, , 2018), where one conditions on an
auxiliary success variable $R$. In particular, for our running example from
Fig 1 with a probabilistic model of the form (1), Control as Inference defines

$p_{0}(R=1|X^{\prime},S^{\prime},A)\coloneqq
e^{r(X^{\prime},S^{\prime},A)}=1-p_{0}(R\hskip 1.0pt{=}\hskip
2.0pt0|X^{\prime},S^{\prime},A)\,,$

where $r=r(X^{\prime},S^{\prime},A)$ denotes a general (negative) reward
function determining desirability. The full joint of the new set of variables
is then given by

$p_{0}(R,\mathbf{X},\mathbf{S},A)=p_{0}(R|X^{\prime},S^{\prime},A)\,p_{0}(\mathbf{X},\mathbf{S},A).$
(27)

Control as Inference then conditions actions on both, the history and future
success ($R=1$). For our one-step example, this results in the Bayes’
posterior

$p(A|X\hskip 1.0pt{=}\hskip 2.0ptx,R\hskip 1.0pt{=}\hskip
2.0pt1)=\frac{1}{\mathcal{Z}}\sum_{x^{\prime},s,s^{\prime}}p_{0}(R=1|x^{\prime},s^{\prime},A)\
p_{0}(\mathbf{x},\mathbf{s},A)\,.$ (28)

It is straightforward to identify $p_{\mathrm{des}}(X^{\prime})$ of Active
Inference as a particular choice of a success probability $p_{0}(R\hskip
1.0pt{=}\hskip 2.0pt1|X^{\prime})$, or equivalently, $\log
p_{\mathrm{des}}(X^{\prime})$ as a reward function $r=r(X^{\prime})$, so that
the joint distribution (27) reduces to the reference function $\phi$ in (25).
Thus, the version of Active Inference in (Schwöbel et al., , 2018) is simply a
variational formulation of Control as Inference that approximates exact
posteriors of the form (28), like other previous variational Bayes’ approaches
(Toussaint, , 2009; Ziebart, , 2010; Levine, , 2018).

Figure 8: Consequences of assuming a desired distribution $p_{\mathrm{des}}$
for action planning under purely inference-based methods, expected utility,
and Active Inference, in the case of a simple example with two actions, one
with a deterministic outcome and one with random outcomes. As can be seen from
the displayed equations, conditioning on $p_{\mathrm{des}}$ (Jeffrey
conditionalization) and conditioning on success (Control as Inference/direct
Active Inference) only differ in the order of normalizing and taking the
expectation over $X^{\prime}$. While conditioning on $p_{\mathrm{des}}$
requires to first sample a target outcome from $p_{\mathrm{des}}$ before an
action from $p(A|x^{\prime})$ can be planned, conditioning on success directly
weighs the desirability of an outcome $p_{\mathrm{des}}(x^{\prime})$ by its
realizability $p(x^{\prime}|A)$. From this point of view, the expected utility
approach is very similar to Control as Inference (which can also be seen in
the grid world environment Notebook: Grid world simulations), since it also
weighs the utility of an outcome with its realizability before soft-
maximizing. It only differs in how it treats the desired distribution as an
exponentiated utility, moving the utility values closer together so that
option $A=1$ is slightly preferred. The early version (Friston et al., , 2013)
of Active Inference is similar to Jeffrey conditioning, because decision-
makers are also assumed to match the desired distribution, by defining the
value function $Q$ as a KL divergence between the predicted and desired
distributions. In later versions of $Q$-value Active Inference (Friston et
al., , 2015, 2016; Friston et al., 2017a, ), the value function $Q$ is
modified by an additional entropy term that explicitly punishes observations
with high variability. Consequently, even when the effect of the action on
future observations is kept the same, i.e., the predictive distribution
$p(X^{\prime}|A)=\sum_{s^{\prime}}p_{0}(X^{\prime}|s^{\prime})p_{0}(s^{\prime}|A)$
remains as depicted in the left-hand column, the preference over actions now
changes completely depending on $p_{0}(X^{\prime}|S^{\prime})$—whereas in the
other approaches, only the predictive distribution $p(X^{\prime}|A)$ and
$p_{\mathrm{des}}(X^{\prime})$ influence planning. While there might be
circumstances where this extra punishment of high outcome variability could be
beneficial, it is questionable from a normative point of view why anything
else other than the predicted outcome probability $p(X^{\prime}|A)$ should be
considered for planning. See Details on the example in Fig 8 for details about
the choices made in the example.

In summary, the assumption of a desired distribution $p_{\mathrm{des}}$ over
future outcomes has led to various attempts in the Active Inference literature
of using probabilistic inference to determine profitable actions. Either an
action distribution $\frac{1}{\mathcal{Z}}\,p_{0}(A)e^{Q(A)}$ is built into
the reference function, which presupposes optimal behavior by designing a
value function $Q$ that leads to desired consequences, or the outcome
probability under the generative model $p_{0}$ is modified directly by
multiplying $p_{\mathrm{des}}$ to $p_{0}$. The latter case is the variational
version of Control as Inference, well-known in the machine learning literature
(Toussaint and Storkey, , 2006; Todorov, , 2008; Toussaint, , 2009; Ziebart, ,
2010; Kappen et al., , 2012; Levine, , 2018; O’Donoghue et al., , 2020).
Considering the issues of $Q$-value Active Inference discussed above, and the
fact that Control as Inference does not rely on a desired distribution over
outcomes, we could ask whether formulating preferences by assuming a desired
distribution is well-advised. As can be seen from Fig 8, the difference
between purely inference-based methods, expected utility approaches, and
Active Inference is mainly in how they treat the desired distribution. Should
$p_{\mathrm{des}}$ be matched or is it good enough if actions are chosen that
lead to a high desired outcome probability? While Control as Inference and
utility-based models essentially take the latter approach, $Q$-value Active
Inference answers this question by requiring that the desired distribution
should be matched as long as the average entropy of
$p_{0}(X^{\prime}|S^{\prime})$ is small.

##  6 So What Does Free Energy Bring To the Table?

###  6.1 A Practical Tool

It is unquestionable that the concept of free energy has seen many fruitful
practical applications outside of physics in the statistical and machine
learning literature. As has been discussed in Section 3, these applications
generally fall into one of two categories, the principle of maximum entropy,
and a variational formulation of Bayesian inference. Here, the principle of
maximum entropy is interpreted in a wider sense of optimizing a trade-off
between uncertainty (entropy) and the expected value of some quantity of
interest (energy), which in practice often appears in the form of regularized
optimization problems (e.g., to prevent overfitting) or as a general inference
method allowing to determine unbiased priors and posteriors (cf. Section 3.1).
In the variational formulation of Bayes’ rule, free energy plays the role of
an error measure that allows to do approximate inference by constraining the
space of distributions over which free energy is optimized, but can also
inform the design of efficient iterative inference algorithms that result from
an alternating optimization scheme where in each step the full variational
free energy is optimized only partially, such as the Bayesian EM algorithm,
belief propagation, and other message passing algorithms (cf. Section 3.2).

It is important to realize that, while the mathematical expressions of a free
energy from constraints with “energy” $\mathcal{E}$ and trade-off parameter
$\beta$ and a variational free energy with reference $\phi$ can formally be
transformed into each other by $\phi=e^{-\beta\mathcal{E}}$, the two kinds of
free energy are inherently distinct, both methodically and by their
motivation. In the case of the free energy from constraints, we are given a
constraint on some quantity $\mathcal{E}$ and we are trying to fulfil this
constraint with minimum bias by selecting a distribution that trades off the
two competing terms $\mathcal{E}$ and entropy. This trade-off also gives the
reason for the existence of the Lagrange multiplier $\beta$ that has to be
determined according to the constraint. In this sense the free energy from
constraints is just a special case of the far more general Lagrangian method
when applied to the optimization of expected values
$\langle\mathcal{E}\rangle_{p}$ under entropy constraints (or the other way
around). In contrast, variational free energy is simply a tool to represent
the normalization of a reference function $\phi$ in terms of an optimization
problem, and therefore does a priori not assume the existence of some quantity
$\mathcal{E}$ that we may have observed in an experiment or that has any other
constraints attached, nor does one explicitly consider entropy to be
constrained or optimized. Therefore, even though starting from a (positive)
reference function $\phi$ we can always invent the existence of some quantity
$\mathcal{E}$ and some multiplier $\beta$ such that
$\phi=e^{-\beta\mathcal{E}}$, this does not explain why these quantities
should exist or why they should be mapped into each other in that particular
way. The Lagrangian method, on the other hand, explains why for a given
constraint on $\mathcal{E}$ we have a Lagrange multiplier $\beta$, how it is
determined, and why the equilibrium distribution has the form
$p^{\ast}=\frac{1}{\mathcal{Z}}e^{-\beta\mathcal{E}}$.

###  6.2 Theories of Intelligent Agency

These practical use-cases of free energy formulations have also influenced
models of intelligent behavior. In the cognitive and behavioral sciences,
intelligent agency has been modelled in a number of different frameworks,
including logic-based symbolic models, connectionist models, statistical
decision-making models, and dynamical systems approaches. Even though
statistical thinking in a broader sense can in principle be applied to any of
the other frameworks as well, statistical models of cognition in a more narrow
sense have often focused on Bayesian models, where agents are equipped with
probabilistic models of their environment allowing them to infer unknown
variables in order to select actions that lead to desirable consequences
(Tenenbaum and Griffiths, , 2001; Wolpert, , 2006; Todorov, , 2009).
Naturally, the inference of unknown variables in such models can be achieved
by a plethora of methods including the two types of free energy approaches of
maximum entropy and variational Bayes. However, both free energy formulations
go one step further in that they attempt to extend both principles from the
case of inference to the case of action selection: utility optimization with
information constraints based on free energy from constraints and Active
Inference based on variational free energy.

While sharing similar mathematical concepts, both approaches differ in syntax
and semantics. An apparent apple of discord is the concept of utility
(Gershman and Daw, , 2012). Utility optimization with information constraints
requires the determination of a utility function, whereas Active Inference
requires the determination of a reference function. In the economic
literature, subjective utility functions that quantify the preferences of
decision-makers are typically restrictive in order to ensure identifiability
when certain consistency axioms are satisfied. In contrast, in Active
Inference the reference function involves determining a desired distribution
given by the preferred frequency of outcomes. However, these differences start
to vanish when weakening the utility concept to something like log-
probabilities, such that the utility framework becomes more similar to the
concept of probability that is able to explain arbitrary behavior. Moreover,
Active Inference has to solve the additional problem of marrying up the
agent’s probabilistic model with its desired distribution into a single
reference function (cf. Section 5.3). The solution to this problem is not
unique, in particular it lies outside the scope of variational Bayesian
inference, but it is critical for the resulting behavior because it determines
the exact solutions that are approximated by free energy minimization. In
fact, as can be seen in simple simulations such as S.2, the various proposals
for this merging that can be found in the Active Inference literature behave
very differently.

Also, both approaches differ fundamentally in their motivation. The motivation
of utility optimization with information constraints is to capture the trade-
off between precision and uncertainty that underlies information processing.
This trade-off takes the form of a free energy once an informational cost
function has been chosen (cf. Section 4.3). Note that Bayes’ rule can be seen
as the minimum of a free energy from constraints with log-likelihoods as
utilities, even though this equivalence is not the primary motivation of this
trade-off. In contrast, Active Inference is motivated from casting the problem
of action selection itself as an inference process (Friston et al., , 2013),
as this allows to express both action and perception as the result of
minimizing the same function, the variational free energy. However, there is
no mystery in having such a single optimization function, because the
underlying probabilistic model already contains both action and perception
variables in a single functional format and the variational free energy is
just a function of that model. Moreover, while approximate inference can be
formulated on the basis of variational free energy, inference in general does
not rely on this concept, in particular inference over actions can easily be
done without free energy (Dayan and Hinton, , 1997; Toussaint and Storkey, ,
2006; Todorov, , 2008; Kappen et al., , 2012; Levine, , 2018).

However, there are also plenty of similarities between the two free energy
approaches. For example, the assumption of a soft-max action distribution in
Active Inference is similar to the posterior solutions resulting from utility
optimization with information constraints. Moreover, the assumption of a
desired future _distribution_ relates to constrained computational resources,
because the uncertainty constraint in a desired distribution over future
states may not only be a consequence of environmental uncertainty, but could
also originate from stochastic preferences of a satisficing decision-maker
that accepts a wide range of outcomes. In fact, as we have seen in the
discussion around Fig 8, various methods for inference over actions differ in
how they treat preferences given by a distribution over desired outcomes: Some
of them try to match the predictive and desired distributions, while others
simply seek to reach states whose outcomes have a high desired probability. In
S.2, we provide a comparison of the discussed methods using grid world
simulations, in order to see their resulting behavior also in a sequential
decision-making task.

A remarkable resemblance among both approaches is the exclusive appearance of
relative entropy to measure dissimilarity. In the Active Inference literature
it is often claimed that every homeostatic system must minimize variational
free energy (Friston, , 2013), which is simply an extension of relative
entropy for non-normalized reference functions (cf. Section 3.2.2). In
utility-based approaches, the relative entropy (19) is typically used to
measure the amount of information processing, even though theoretically other
cost functions would be conceivable (Gottwald and Braun, 2019a, ). For a given
homeostatic process, the KL divergence measures the dissimilarity between the
current distribution and the limiting distribution and therefore is reduced
while approximating the equilibrium. Similarly, in utility-based decision-
making models, relative entropy measures the dissimilarity between the current
posterior and the prior. In the Active Inference literature the stepwise
minimization of variational free energy that goes along with KL minimization
is often equated with the minimization of sensory surprise (see Surprise
minimization for a more detailed explanation), an idea that stems from maximum
likelihood algorithms, but that has been challenged as a general principle
(see (Biehl et al., , 2020) and the response (Friston et al., , 2020)).
Similarly, one could in principle rewrite free energy from constraints in
terms of informational surprise, which would however simply be a rewording of
the probabilistic concepts in log-space. The same kind of rewording is well-
known between probabilistic inference and the minimum description length
principle (Grünwald, , 2007) that also operates in log-space, and thus
reformulates the inference problem as a surprise minimization problem without
adding any new features or properties.

###  6.3 Biological Relevance

So far we have seen how free energy is used as a technical instrument to solve
inference problems and its corresponding appearance in different models of
intelligent agency. Crucially, these kinds of models can be applied to any
input-output system, be it a human that reacts to sensory stimuli, a cell that
tries to maintain homeostasis, or a particle trapped by a physical potential.
Given the existing literature that has widely applied the concept of free
energy to biological systems, we may ask whether there are any specific
biological implications of these models.

Considering free energy from constraints, the trade-off between utility and
information processing costs provides a normative model of decision-making
under resource constraints, that extends previous optimality models based on
expected utility maximization and Bayesian inference. Analogous to rate-
distortion curves in information theory, optimal solutions to decision-making
problems are obtained that separate achievable from non-achievable regions in
the information-utility plane (cf. Fig 5). The behavior of real decision-
making systems under varying information constraints can be analyzed
experimentally by comparing their performance with respect to the
corresponding optimality curve. One can experimentally relate abstract
information processing costs measured in bits to task-dependent resource costs
like reaction or planning times (Schach et al., , 2018; Ortega and Stocker, ,
2016). Moreover, the free energy trade-off can also be used to describe
networks of agents, where each agent is limited in its ability, but the system
as a whole has a higher information processing capacity—for example, neurons
in a brain or humans in a group. In such systems different levels of
abstraction arise depending on the different positions of decision-makers in
the network (Lindig-León et al., , 2019; Genewein et al., , 2015; Gottwald and
Braun, 2019b, ). As we have discussed in Section 4.3, just like coding and
rate-distortion theory, utility theory with information costs can only provide
optimality bounds but does not specify any particular mechanism of how to
achieve optimality. However, by including more and more constraints one can
make a model more and more mechanistic and thereby gradually move from a
normative to a more descriptive model, such as models that consider the
communication channel capacity of neurons with a finite energy budget (Bhui
and Gershman, , 2018).

Considering variational free energy, there is a vast literature on biological
applications mostly focusing on neural processing (e.g., predictive coding,
dopamine) (Schwartenbeck et al., , 2015; Friston et al., 2017b, ; Parr et al.,
, 2019), but there are also a number of applications aiming to explain
behavior (e.g., human decision-making, hallucinations) (Parr et al., , 2018).
Similarly to utility-based models, Active Inference models can be studied in
terms of _as if_ models, so that actual behavior can be compared to predicted
behavior as long as suitable prior and likelihood models can be identified
from the experiment. When applied to brain dynamics, the as if models are
sometimes also given a mechanistic interpretation by relating iterative update
equations that appear when minimizing variational free energy with dynamics in
neuronal circuits. As discussed in Section 3.2.3, the update equations
resulting for example from mean-field or Bethe approximations, can often be
written in message passing form in the sense that the update for a given
variable only has contributions that requires the current approximate
posterior of neighbouring nodes in the probabilistic model. These
contributions are interpreted as local messages passed between the nodes and
might be related to brain signals (Parr et al., , 2019). Other interpretations
(Friston et al., , 2006; Friston et al., 2017a, ; Bogacz, , 2017) obtain
similar update equations by minimizing variational free energy directly
through gradient descent, which can again be related to neural coding schemes
like predictive coding. As these coding schemes have existed irrespective of
free energy (Rao and Ballard, , 1999; Aitchison and Lengyel, , 2017),
especially since minimization of prediction errors is already seen in maximum
likelihood estimation (Rao and Ballard, , 1999), the question remains whether
there are any specific predictions of the Active Inference framework that
cannot be explained with previous models (see (Colombo and Wright, , 2018;
Hohwy, , 2020) for recent discussions of this question).

###  6.4 Conclusion

Any theory about intelligent behavior has to answer three questions: _Where am
I?_ , _where do I want to go?_ , and _how do I get there?_ , corresponding to
the three problems of inference and perception, goals and preferences, and
planning and execution. All three problems can be addressed either in the
language of probabilities or utilities. Perceptual inference can either be
considered as finding parameters that maximize probabilities or likelihood
utilities. Goals and preferences can either be expressed by utilities over
outcomes or by desired distributions. The third question can be answered by
the two free energy approaches that either determine future utilities based on
model predictions, or infer actions that lead to outcomes predicted to have
high desired probability or match the desired distribution. In standard
decision-making models actions are usually determined by a utility function
that ranks different options, whereas perceptual inference is determined by a
likelihood model that quantifies how probable certain observations are. In
contrast, both free energy approaches have in common that they treat all types
of information processing, from action planning to perception, as the same
formal process of minimizing some form of free energy. But the crucial
difference is not whether they use utilities or probabilities, but how
predictions and goals are interwoven into action.

This article started out by tracing back the seemingly mysterious connection
between Helmholtz free energy from thermodynamics and Helmholtz’ view of
model-based information processing that led to the analysis-by-synthesis
approach of perception, as exemplified in predictive coding schemes, and in
particular to discuss the role of free energy in current models of intelligent
behavior. The mystery starts to dissolve when we consider the two kinds of
free energies discussed in this article, one based on the maximum entropy
principle and the other based on variational free energy—a dissimilarity
measure between distributions and (generally unnormalized) functions that
extends the well-known KL divergence from information theory. The Helmholtz
free energy is a particular example of an energy information trade-off that
results from the maximum entropy principle (Jaynes, , 1957). Analysis-by-
synthesis is a particular application of inference to perception, where
determining model parameters and hidden states can either be seen as a result
of maximum entropy under observational constraints or of fitting parameter
distributions to the model through variational free energy minimization. Thus,
both notions of free energy can be formally related as entropy-regularized
maximization of log-probabilities.

Conceptually, however, utility-based models with information constraints serve
primarily as _ultimate_ explanations of behavior, this means they do not focus
on mechanism, but on the goals of behavior and their realizability under ideal
circumstances. They have the appeal of being relatively straightforward
generalization of standard utility theory, but they rely on abstract concepts
like utility and relative entropy that may not be so straightforwardly related
to experimental settings. While these normative models have no immediate
mechanistic interpretation, their relevance for mechanistic models may be
analogous to the relevance of optimality bounds in Shannon’s information
theory for practical codes (Shannon, , 1948). In contrast, Active Inference
models of behavior often mix ultimate and _proximate_ arguments of explaining
behavior (Alcock, , 1993; Tinbergen, , 1963), because they combine the
normative aspect of optimizing variational free energy with the mechanistic
interpretation of the particular form of approximate solutions to this
optimization. While mean-field approaches of Active Inference may be
particularly amenable to such mechanistic interpretations, they are often too
simple to capture complex behavior. In contrast, the solutions of direct
Active Inference resulting from a Bethe assumption are equivalent to previous
Control as Inference approaches (Toussaint and Storkey, , 2006; Todorov, ,
2008; Toussaint, , 2009; Ziebart, , 2010; Kappen et al., , 2012; Levine, ,
2018; O’Donoghue et al., , 2020) that allow for Bayesian message passing
formulations whose biological implementability can be debated irrespective of
the existence of a free energy functional.

Finally, both kinds of free energy formulations of intelligent agency are so
general and flexible in their ingredients that it might be more appropriate to
consider them languages or tools to phrase and describe behavior rather than
theories that explain behavior, in a sense similar to how statistics and
probability theory are not biological or physical theories but simply provide
a language in which we can phrase our biological and physical assumptions.

## Funding

This study was funded by the European Research Council (ERC-StG-2015-ERC
Starting Grant, Project ID: 678082, “BRISC: Bounded Rationality in
Sensorimotor Coordination”).

## References

  * Aitchison and Lengyel, (2017) Aitchison, L. and Lengyel, M. (2017).  With or without you: predictive coding and bayesian inference in the brain.  Current Opinion in Neurobiology, 46:219–227.  Computational Neuroscience.
  * Alcock, (1993) Alcock, J. (1993).  Animal behavior: an evolutionary approach.  Sinauer Associates.
  * Ashby, (1960) Ashby, W. (1960).  Design for a Brain: The Origin of Adaptive Behavior.  Springer Netherlands.
  * Beal, (2003) Beal, M. J. (2003).  Variational Algorithms for Approximate Bayesian Inference.  PhD thesis, University of Cambridge, UK.
  * Bernoulli, (1713) Bernoulli, J. (1713).  Ars conjectandi.  Basel, Thurneysen Brothers.
  * Bhui and Gershman, (2018) Bhui, R. and Gershman, S. J. (2018).  Decision by sampling implements efficient coding of psychoeconomic functions.  Psychological Review, 125(6):985–1001.
  * Biehl et al., (2020) Biehl, M., Pollock, F. A., and Kanai, R. (2020).  A technical critique of the free energy principle as presented in "life as we know it" and related works.  arXiv:2001.06408.
  * Binz et al., (2020) Binz, M., Gershman, S. J., Schulz, E., and Endres, D. (2020).  Heuristics from bounded meta-learned inference.
  * Bogacz, (2017) Bogacz, R. (2017).  A tutorial on the free-energy framework for modelling perception and learning.  Journal of Mathematical Psychology, 76:198–211.  Model-based Cognitive Neuroscience.
  * Boutilier et al., (1999) Boutilier, C., Dean, T., and Hanks, S. (1999).  Decision-theoretic planning: Structural assumptions and computational leverage.  J. Artif. Int. Res., 11(1):1–94.
  * Cisek, (1999) Cisek, P. (1999).  Beyond the computer metaphor: behaviour as interaction.  Journal of Consciousness Studies, 6(11-12):125–142.
  * Clark, (2013) Clark, A. (2013).  Whatever next? predictive brains, situated agents, and the future of cognitive science.  Behavioral and Brain Sciences, 36(3):181–204.
  * Colombo and Wright, (2018) Colombo, M. and Wright, C. (2018).  First principles in the life sciences: the free-energy principle, organicism, and mechanism.  Synthese.
  * Corcoran and Hohwy, (2018) Corcoran, A. W. and Hohwy, J. (2018).  Allostasis, interoception, and the free energy principle: Feeling our way forward.  Oxford University Press.
  * Csiszár, (2008) Csiszár, I. (2008).  Axiomatic characterizations of information measures.  Entropy, 10(3):261–273.
  * Csiszár and Tusnády, (1984) Csiszár, I. and Tusnády, G. (1984).  Information geometry and alternating minimization procedures.  Statistics and Decisions, Supplement Issue, 1:205–237.
  * Dayan and Hinton, (1997) Dayan, P. and Hinton, G. E. (1997).  Using expectation-maximization for reinforcement learning.  Neural Computation, 9(2):271–278.
  * Dayan et al., (1995) Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995).  The helmholtz machine.  Neural Comput., 7(5):889–904.
  * de Laplace, (1812) de Laplace, P. S. (1812).  Théorie analytique des probabilités.  Ve. Courcier, Paris.
  * Dempster et al., (1977) Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1–38.
  * Doya, (2007) Doya, K. (2007).  Bayesian Brain: Probabilistic Approaches to Neural Coding.  MIT Press, Cambridge, Mass.
  * Ergin and Sarver, (2010) Ergin, H. and Sarver, T. (2010).  A unique costly contemplation representation.  Econometrica, 78(4):1285–1339.
  * Feynman et al., (1996) Feynman, R., Hey, A., and Allen, R. (1996).  Feynman Lectures on Computation.  Advanced book program. Addison-Wesley.
  * Flanagan et al., (2003) Flanagan, J. R., Vetter, P., Johansson, R. S., and Wolpert, D. M. (2003).  Prediction precedes control in motor learning.  Current Biology, 13(2):146–150.
  * Fox et al., (2016) Fox, R., Pakman, A., and Tishby, N. (2016).  Taming the noise in reinforcement learning via soft updates.  In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, UAI’16, pages 202–211, Arlington, Virginia, United States. AUAI Press.
  * Friston, (2013) Friston, K. (2013).  Life as we know it.  Journal of The Royal Society Interface, 10(86):20130475.
  * Friston, (2018) Friston, K. (2018).  Does predictive coding have a future?  Nature Neuroscience, 21(8):1019–1021.
  * Friston et al., (2020) Friston, K., Costa, L. D., and Parr, T. (2020).  Some interesting observations on the free energy principle.  arXiv:2002.04501.
  * Friston et al., (2016) Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O’Doherty, J., and Pezzulo, G. (2016).  Active inference and learning.  Neuroscience & Biobehavioral Reviews, 68:862–879.
  * Friston et al., (2013) Friston, K., Schwartenbeck, P., Fitzgerald, T., Moutoussis, M., Behrens, T., and Dolan, R. (2013).  The anatomy of choice: active inference and agency.  Frontiers in Human Neuroscience, 7:598.
  * Friston, (2005) Friston, K. J. (2005).  A theory of cortical responses.  Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456):815–836.
  * Friston, (2010) Friston, K. J. (2010).  The free-energy principle: a unified brain theory?  Nature Reviews Neuroscience, 11:127–138.
  * (33) Friston, K. J., FitzGerald, T. H. B., Rigoli, F., Schwartenbeck, P., and Pezzulo, G. (2017a).  Active inference: A process theory.  Neural Computation, 29:1–49.
  * Friston et al., (2006) Friston, K. J., Kilner, J., and Harrison, L. M. (2006).  A free energy principle for the brain.  Journal of Physiology-Paris, 100:70–87.
  * (35) Friston, K. J., Parr, T., and de Vries, B. (2017b).  The graphical brain: Belief propagation and active inference.  Network Neuroscience, 1(4):381–414.
  * Friston et al., (2015) Friston, K. J., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., and Pezzulo, G. (2015).  Active inference and epistemic value.  Cognitive Neuroscience, 6(4):187–214.
  * Friston et al., (2012) Friston, K. J., Shiner, T., FitzGerald, T., Galea, J. M., Adams, R., Brown, H., Dolan, R. J., Moran, R., Stephan, K. E., and Bestmann, S. (2012).  Dopamine, affordance and active inference.  PLoS Computational Biology, 8(1):e1002327.
  * Garner, (1962) Garner, W. R. (1962).  Uncertainty and structure as psychological concepts.  Wiley.
  * Genewein et al., (2015) Genewein, T., Leibfried, F., Grau-Moya, J., and Braun, D. A. (2015).  Bounded rationality, abstraction, and hierarchical decision-making: An information-theoretic optimality principle.  Frontiers in Robotics and AI, 2.
  * Gershman, (2019) Gershman, S. J. (2019).  What does the free energy principle tell us about the brain.  Neurons, Behavior, Data Analysis, and Theory.
  * Gershman and Daw, (2012) Gershman, S. J. and Daw, N. D. (2012).  Perception, action and utility: The tangled skein.  In Principles of Brain Dynamics. MIT Press.
  * Gigerenzer and Selten, (2001) Gigerenzer, G. and Selten, R. (2001).  Bounded Rationality: The Adaptive Toolbox.  MIT Press: Cambridge, MA, USA.
  * (43) Gottwald, S. and Braun, D. A. (2019a).  Bounded rational decision-making from elementary computations that reduce uncertainty.  Entropy, 21(4).
  * (44) Gottwald, S. and Braun, D. A. (2019b).  Systems of bounded rational agents with information-theoretic constraints.  Neural Computation, 31(2):440–476.
  * Grau-Moya et al., (2016) Grau-Moya, J., Leibfried, F., Genewein, T., and Braun, D. A. (2016).  Planning with information-processing constraints and model uncertainty in markov decision processes.  In Machine Learning and Knowledge Discovery in Databases, pages 475–491. Springer International Publishing.
  * Grünwald, (2007) Grünwald, P. (2007).  The Minimum Description Length Principle.  MIT Press, Cambridge, Mass.
  * Haarnoja et al., (2017) Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. (2017).  Reinforcement learning with deep energy-based policies.  In ICML.
  * Hansen and Sargent, (2008) Hansen, L. P. and Sargent, T. J. (2008).  Robustness.  Princeton University Press.
  * Harsha et al., (2010) Harsha, P., Jain, R., McAllester, D., and Radhakrishnan, J. (2010).  The communication complexity of correlation.  IEEE Transactions on Information Theory, 56(1):438–449.
  * Hathaway, (1986) Hathaway, R. J. (1986).  Another interpretation of the em algorithm for mixture distributions.  Statistics & Probability Letters, 4(2):53–56.
  * Heskes, (2003) Heskes, T. (2003).  Stable fixed points of loopy belief propagation are local minima of the bethe free energy.  In Becker, S., Thrun, S., and Obermayer, K., editors, Advances in Neural Information Processing Systems 15, pages 359–366. MIT Press.
  * Hinton and van Camp, (1993) Hinton, G. E. and van Camp, D. (1993).  Keeping the neural networks simple by minimizing the description length of the weights.  In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT ’93, pages 5–13, New York, NY, USA. ACM.
  * Ho et al., (2020) Ho, M. K., Abel, D., Cohen, J. D., Littman, M. L., and Griffiths, T. L. (2020).  The efficiency of human cognition reflects planned information processing.  Proceedings of the 34th AAAI Conference on Artificial Intelligence.
  * Hohwy, (2020) Hohwy, J. (2020).  Self-supervision, normativity and the free energy principle.  Synthese.
  * Jaynes, (1957) Jaynes, E. T. (1957).  Information theory and statistical mechanics.  Phys. Rev., 106:620–630.
  * Jaynes, (2003) Jaynes, E. T. (2003).  Probability Theory.  Cambridge University Press.
  * Jeffrey, (1965) Jeffrey, R. C. (1965).  The Logic of Decision.  University of Chicago Press, 1 edition.
  * Kahneman, (2002) Kahneman, D. (2002).  Maps of bounded rationality: A perspective on intuitive judgement.  In Frangsmyr, T., editor, Nobel prizes, presentations, biographies, & lectures, pages 416–499. Almqvist & Wiksell, Stockholm, Sweden.
  * Kappen et al., (2012) Kappen, H. J., Gómez, V., and Opper, M. (2012).  Optimal control as a graphical model inference problem.  Machine Learning, 87(2):159–182.
  * Kawato, (1999) Kawato, M. (1999).  Internal models for motor control and trajectory planning.  Current Opinion in Neurobiology, 9(6):718–727.
  * Kikuchi, (1951) Kikuchi, R. (1951).  A theory of cooperative phenomena.  Physical Review, 81(6):988–1003.
  * Koller, (2009) Koller, D. (2009).  Probabilistic graphical models : principles and techniques.  The MIT Press, Cambridge, Massachusetts.
  * Levine, (2018) Levine, S. (2018).  Reinforcement learning and control as probabilistic inference: Tutorial and review.  arXiv:1805.00909.
  * Lindig-León et al., (2019) Lindig-León, C., Gottwald, S., and Braun, D. A. (2019).  Analyzing abstraction and hierarchical decision-making in absolute identification by information-theoretic bounded rationality.  Frontiers in Neuroscience, 13:1230.
  * Linson et al., (2020) Linson, A., Parr, T., and Friston, K. J. (2020).  Active inference, stressors, and psychological trauma: A neuroethological model of (mal)adaptive explore-exploit dynamics in ecological context.  Behavioural Brain Research, 380:112421.
  * Maccheroni et al., (2006) Maccheroni, F., Marinacci, M., and Rustichini, A. (2006).  Ambiguity aversion, robustness, and the variational representation of preferences.  Econometrica, 74(6):1447–1498.
  * MacKay, (2002) MacKay, D. J. C. (2002).  Information Theory, Inference & Learning Algorithms.  Cambridge University Press, USA.
  * MacRae, (1970) MacRae, A. W. (1970).  Channel capacity in absolute judgment tasks: An artifact of information bias?  Psychological Bulletin, 73(2):112–121.
  * Marshall et al., (2011) Marshall, A. W., Olkin, I., and Arnold, B. C. (2011).  Inequalities: Theory of Majorization and Its Applications.  Springer New York, 2nd edition.
  * Mattsson and Weibull, (2002) Mattsson, L.-G. and Weibull, J. W. (2002).  Probabilistic choice and procedurally bounded rationality.  Games and Economic Behavior, 41(1):61–78.
  * McFadden, (2005) McFadden, D. L. (2005).  Revealed stochastic preference: a synthesis.  Economic Theory, 26(2):245–264.
  * McKelvey and Palfrey, (1995) McKelvey, R. D. and Palfrey, T. R. (1995).  Quantal response equilibria for normal form games.  Games and Economic Behavior, 10(1):6–38.
  * Miller, (1956) Miller, G. A. (1956).  The magical number seven, plus or minus two: some limits on our capacity for processing information.  Psychological Review, 63(2):81–97.
  * Minka, (2005) Minka, T. (2005).  Divergence measures and message passing.  Technical Report MSR-TR-2005-173, Microsoft.
  * Minka, (2001) Minka, T. P. (2001).  Expectation propagation for approximate bayesian inference.  In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, UAI ’01, pages 362–369, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Mirza et al., (2018) Mirza, M. B., Adams, R. A., Mathys, C., and Friston, K. J. (2018).  Human visual exploration reduces uncertainty about the sensed world.  PLOS ONE, 13(1):e0190429.
  * Mnih et al., (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.  In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928–1937, New York, New York, USA. PMLR.
  * Neal and Hinton, (1998) Neal, R. M. and Hinton, G. E. (1998).  A view of the em algorithm that justifies incremental, sparse, and other variants.  In Jordan, M. I., editor, Learning in Graphical Models, pages 355–368. Springer Netherlands, Dordrecht.
  * O’Donoghue et al., (2020) O’Donoghue, B., Osband, I., and Ionescu, C. (2020).  Making sense of reinforcement learning and probabilistic inference.  In International Conference on Learning Representations, ICLR ’20.
  * Opper and Saad, (2001) Opper, M. and Saad, D. (2001).  Comparing the Mean Field Method and Belief Propagation for Approximate Inference in MRFs, pages 229–239.
  * Ortega and Braun, (2013) Ortega, P. A. and Braun, D. A. (2013).  Thermodynamics as a theory of decision-making with information-processing costs.  Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469(2153):20120683.
  * Ortega and Braun, (2014) Ortega, P. A. and Braun, D. A. (2014).  Generalized thompson sampling for sequential decision-making and causal inference.  Complex Adaptive Systems Modeling, 2(1):2.
  * Ortega and Stocker, (2016) Ortega, P. A. and Stocker, A. (2016).  Human decision-making under limited time.  In 30th Conference on Neural Information Processing Systems.
  * Parr et al., (2018) Parr, T., Benrimoh, D. A., Vincent, P., and Friston, K. J. (2018).  Precision and false perceptual inference.  Frontiers in Integrative Neuroscience, 12:39.
  * Parr and Friston, (2017) Parr, T. and Friston, K. J. (2017).  Working memory, attention, and salience in active inference.  Scientific reports, 7(1):14678–14678.
  * Parr and Friston, (2019) Parr, T. and Friston, K. J. (2019).  Generalised free energy and active inference.  Biological Cybernetics.
  * Parr et al., (2019) Parr, T., Markovic, D., Kiebel, S. J., and Friston, K. J. (2019).  Neuronal message passing using mean-field, bethe, and marginal approximations.  Scientific Reports, 9(1):1889.
  * Pearl, (1988) Pearl, J. (1988).  Belief updating by network propagation.  In Pearl, J., editor, Probabilistic Reasoning in Intelligent Systems, pages 143–237. Morgan Kaufmann, San Francisco (CA).
  * Poincaré, (1912) Poincaré, H. (1912).  Calcul des probabilités.  Gauthier-Villars, Paris.
  * Powers, (1973) Powers, W. T. (1973).  Behavior: The Control of Perception.  Aldine, Chicago, IL.
  * Rao and Ballard, (1999) Rao, R. P. N. and Ballard, D. H. (1999).  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  Nature Neuroscience, 2(1):79–87.
  * Rosenkrantz, (1983) Rosenkrantz, R. D. (1983).  E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics.  Springer Netherlands, Dordrecht.
  * Russell and Subramanian, (1995) Russell, S. J. and Subramanian, D. (1995).  Provably bounded-optimal agents.  Journal of Artificial Intelligence Research, 2(1):575–609.
  * Sales et al., (2019) Sales, A. C., Friston, K. J., Jones, M. W., Pickering, A. E., and Moran, R. J. (2019).  Locus coeruleus tracking of prediction errors optimises cognitive flexibility: An active inference model.  PLOS Computational Biology, 15(1):e1006267.
  * Saul and Jordan, (1996) Saul, L. K. and Jordan, M. I. (1996).  Exploiting tractable substructures in intractable networks.  In Touretzky, D. S., Mozer, M. C., and Hasselmo, M. E., editors, Advances in Neural Information Processing Systems 8, pages 486–492. MIT Press.
  * Schach et al., (2018) Schach, S., Gottwald, S., and Braun, D. A. (2018).  Quantifying motor task performance by bounded rational decision theory.  Frontiers in Neuroscience, 12:932.
  * Schwartenbeck et al., (2015) Schwartenbeck, P., FitzGerald, T. H. B., Mathys, C., Dolan, R., and Friston, K. (2015).  The dopaminergic midbrain encodes the expected certainty about desired outcomes.  Cerebral cortex (New York, N.Y. : 1991), 25(10):3434–3445.
  * Schwartenbeck and Friston, (2016) Schwartenbeck, P. and Friston, K. (2016).  Computational phenotyping in psychiatry: A worked example.  eNeuro, 3(4):ENEURO.0049–16.2016.
  * Schwöbel et al., (2018) Schwöbel, S., Kiebel, S., and Marković, D. (2018).  Active inference, belief propagation, and the bethe approximation.  Neural Computation, 30(9):2530–2567.
  * Shannon, (1948) Shannon, C. E. (1948).  A mathematical theory of communication.  The Bell System Technical Journal, 27:379–656.
  * Simon, (1955) Simon, H. A. (1955).  A behavioral model of rational choice.  The Quarterly Journal of Economics, 69(1):99–118.
  * Sims, (2003) Sims, C. A. (2003).  Implications of rational inattention.  Journal of Monetary Economics, 50(3):665–690.  Swiss National Bank/Study Center Gerzensee Conference on Monetary Policy under Incomplete Information.
  * Sims, (2016) Sims, C. R. (2016).  Rate–distortion theory and human perception.  Cognition, 152:181–198.
  * Still, (2009) Still, S. (2009).  Information-theoretic approach to interactive learning.  Europhysics Letters, 85(2):28005.
  * Tatikonda and Mitter, (2004) Tatikonda, S. and Mitter, S. (2004).  Control under communication constraints.  IEEE Transactions on Automatic Control, 49(7):1056–1068.
  * Tenenbaum and Griffiths, (2001) Tenenbaum, J. B. and Griffiths, T. L. (2001).  Generalization, similarity, and bayesian inference.  Behavioral and Brain Sciences, 24(4):629–640.
  * Tinbergen, (1963) Tinbergen, N. (1963).  On aims and methods of ethology.  Zeitschrift für Tierpsychologie, 20:410–433.
  * Tishby and Polani, (2011) Tishby, N. and Polani, D. (2011).  Information theory of decisions and actions.  In Cutsuridis, V., Hussain, A., and Taylor, J. G., editors, Perception-Action Cycle: Models, Architectures, and Hardware, pages 601–636. Springer New York.
  * Todorov, (2008) Todorov, E. (2008).  General duality between optimal control and estimation.  In 2008 47th IEEE Conference on Decision and Control. IEEE.
  * Todorov, (2009) Todorov, E. (2009).  Efficient computation of optimal actions.  Proceedings of the National Academy of Sciences, 106(28):11478–11483.
  * Toussaint, (2009) Toussaint, M. (2009).  Robot trajectory optimization using approximate inference.  In Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09. ACM Press.
  * Toussaint and Storkey, (2006) Toussaint, M. and Storkey, A. (2006).  Probabilistic inference for solving discrete and continuous state markov decision processes.  In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pages 945–952, New York, NY, USA. Association for Computing Machinery.
  * von Neumann and Morgenstern, (1944) von Neumann, J. and Morgenstern, O. (1944).  Theory of Games and Economic Behavior.  Princeton University Press, Princeton, NJ, USA.
  * Wainwright et al., (2005) Wainwright, M., Jaakkola, T., and Willsky, A. (2005).  Map estimation via agreement on (hyper)trees: Message-passing and linear-programming approaches.  IEEE Transactions on Information Theory, 51(11):3697–3717.
  * Whittle, (1990) Whittle, P. (1990).  Risk-sensitive optimal control.  Wiley, Chichester New York.
  * Wiener, (1948) Wiener, N. (1948).  Cybernetics: Or Control and Communication in the Animal and the Machine.  John Wiley.
  * Williams, (1980) Williams, P. M. (1980).  Bayesian conditionalisation and the principle of minimum information.  The British Journal for the Philosophy of Science, 31(2):131–144.
  * Williams and Peng, (1991) Williams, R. J. and Peng, J. (1991).  Function optimization using connectionist reinforcement learning algorithms.  Connection Science, 3(3):241–268.
  * Winn and Bishop, (2005) Winn, J. and Bishop, C. M. (2005).  Variational message passing.  J. Mach. Learn. Res., 6:661–694.
  * Wolpert, (2006) Wolpert, D. H. (2006).  Information Theory – The Bridge Connecting Bounded Rational Game Theory and Statistical Physics, pages 262–290.  Springer Berlin Heidelberg.
  * Wolpert, (2019) Wolpert, D. H. (2019).  The stochastic thermodynamics of computation.  Journal of Physics A: Mathematical and Theoretical, 52(19):193001.
  * Yedidia et al., (2001) Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2001).  Generalized belief propagation.  In Leen, T. K., Dietterich, T. G., and Tresp, V., editors, Advances in Neural Information Processing Systems 13, pages 689–695. MIT Press.
  * Yedidia et al., (2005) Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2005).  Constructing free-energy approximations and generalized belief propagation algorithms.  IEEE Transactions on Information Theory, 51(7):2282–2312.
  * Yuille and Kersten, (2006) Yuille, A. and Kersten, D. (2006).  Vision as bayesian inference: analysis by synthesis?  Trends in Cognitive Sciences, 10(7):301–308.  Special issue: Probabilistic models of cognition.
  * Yuille, (2002) Yuille, A. L. (2002).  Cccp algorithms to minimize the bethe and kikuchi free energies: Convergent alternatives to belief propagation.  Neural Computation, 14(7):1691–1722.
  * Ziebart, (2010) Ziebart, B. D. (2010).  Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy.  PhD thesis, Carnegie Mellon Unversity.

##  Appendix A Appendices

###  A.1 Derivation of exemplary update equations

####  A.1.1 $Q$-value Active Inference

In the simple example of Section 5.2 under the partial mean-field assumption
(23), and in the case when the desired distribution $p_{\mathrm{des}}$ is
combined with the generative model $p_{0}$ via the value function $Q$ as shown
in Equation (24), i.e. if $\phi\propto
p_{0}(x,X^{\prime},\mathbf{S},A)\,e^{Q(A)}$, then the full free energy
$F(q\|\phi)$ can be written as

$\displaystyle
F(q\|\phi)=F(q(\mathbf{S}|A)q(A)\|p_{0}(x|S)p_{0}(\mathbf{S}|A)p_{0}(A)e^{Q(A)})$
$\displaystyle=\big{\langle}F_{\mathbf{S}}(A)-Q(A)\big{\rangle}_{q(A)}+D_{\mathrm{KL}}(q(A)\|p_{0}(A))$
(29)

where, $F_{\mathbf{S}}(A)-Q(A)$ is given by

$\left\langle\log\frac{~{}\qquad q(S)\quad
q(S^{\prime}|A)\quad\sum_{s^{\prime}}p_{0}(X^{\prime}|s^{\prime})q(s^{\prime}|A)}{p_{0}(x|S)p_{0}(S)\,p_{0}(S^{\prime}|S,A)\,p_{\mathrm{des}}(X^{\prime})p_{0}(X^{\prime}|S^{\prime})}\right\rangle$
(30)

where the expectation is with respect to $q(X^{\prime},\mathbf{S}|A)$. Thus,
optimizing (29) over $q(A)$, while keeping $q(\mathbf{S}|A)$ fixed, results in
a Boltzmann distribution with prior $p_{0}(A)$ and energy
$F_{\mathbf{S}}(A)-Q(A)$. When optimizing $F(q\|\phi)$ with respect to $q(S)$
while keeping $q(S^{\prime}|A)$ and $q(A)$ fixed, we have

$\displaystyle
q^{\ast}(S)=\mathop{\mathrm{argmax}}_{q(S)}\,F(q\|\phi)=\mathop{\mathrm{argmax}}_{q(S)}\,\langle
F_{\mathbf{S}}(A)\rangle_{q(A)}$
$\displaystyle=\mathop{\mathrm{argmax}}_{q(S)}\,\underbrace{\left\langle\log\frac{q(S)}{p_{0}(x|S)p_{0}(S)e^{\langle
T\rangle_{q(S^{\prime}|A)q(A)}}}\right\rangle_{q(S)}}_{F\big{(}q(S)\big{\|}p_{0}(x|S)p_{0}(S)e^{\langle
T\rangle}\big{)}}\,,$ (31)

where $T\,{\coloneqq}\,\log p_{0}(S^{\prime}|S,A)$ is shorthand for the log-
transition probability. Hence, from (31) we can read off the solution
$q^{\ast}(S)$ in virtue of the general optimum (14) of variational free
energy. While here it was enough to optimize $\langle
F_{\mathbf{S}}\rangle_{q}$, because in contains the only dependencies of
$F(q\|\phi)$ on $q(S)$, this is not the case for $q(S^{\prime}|A)$, since also
$Q$ depends on $q(S^{\prime}|A)$. Thus, when optimizing (29) over
$q(S^{\prime}|A)$ while keeping $q(A)$ and $q(S)$ fixed, one has to optimize
$\langle F_{\mathbf{S}}-Q\rangle$ which does not take the form of a free
energy in $q(S^{\prime}|A)$ due to the functional dependency of
$q(X^{\prime}|A)=\sum_{s^{\prime}}p_{0}(X^{\prime}|s^{\prime})q(s^{\prime}|A)$
on $q(S^{\prime}|A)$ that appears in (30). However, this type of dependency is
largely ignored in the Active Inference literature (as for example noted in
the appendix of (Friston et al., , 2015)), since the optimization with respect
to $q(S^{\prime}|A)$ would not have a closed-form solution otherwise.

Once this term is ignored, then the objective for $q(S^{\prime}|A)$ takes a
very simple form,

$\displaystyle q^{\ast}(S^{\prime}|A)$
$\displaystyle=\mathop{\mathrm{argmax}}_{q(S^{\prime}|A)}\,F(q\|\phi)$
$\displaystyle\approx\mathop{\mathrm{argmax}}_{q(S^{\prime}|A)}\left\langle\log\frac{q(S^{\prime}|A)}{e^{\langle
T\rangle_{q(S)}}}\right\rangle_{q(S^{\prime}|A)},$ (32)

from which we can again read off the resulting update equation. In total, from
(29),(31), and (32) we obtain the set of equations

$\displaystyle q^{\ast}(S)$
$\displaystyle=\tfrac{1}{\mathcal{Z}}\,p_{0}(x|S)p_{0}(S)e^{\langle
T\rangle_{q(S^{\prime}|A)q(A)}}$ (33a) $\displaystyle q^{\ast}(S^{\prime}|A)$
$\displaystyle\approx\tfrac{1}{\mathcal{Z}(A)}\,e^{\langle T\rangle_{q(S)}}$
(33b) $\displaystyle q^{\ast}(A)$
$\displaystyle=\tfrac{1}{\mathcal{Z}}\,p_{0}(A)e^{-F_{\mathbf{S}}(A)+Q(A)},$
(33c)

where $\mathcal{Z}$ denotes the respective normalization constants and
$T\hskip 1.0pt{=}\hskip 2.0pt\log p_{0}(S^{\prime}|S,A)$.

It is important to note, however, that update equations in Active Inference
resulting from a mean-field assumption (even if it is a partial mean-field
assumption such as (23)) should be taken with care, since—as is demonstrated
in the grid world simulations in S2 Notebook—even in very simple situations
the resulting agents fail to correctly plan actions that lead to desired
states.

####  A.1.2 Direct Active Inference (variational Control as Inference)—mean-
field assumption

Here, we derive the update equations resulting from the minimization of the
variational free energy for the reference defined in Equation (25a), i.e. a
variational formulation of Control as inference (Toussaint and Storkey, ,
2006), under the mean-field assumption (23). We start by writing the
variational free energy $F(q\|\phi)$ in a form analogous to (29), where now
$\phi$ is given by
$p_{0}(X^{\prime}|S^{\prime})p_{0}(x,\mathbf{S}|A)p_{0}(A)$,

$\displaystyle F(q\|\phi)$
$\displaystyle=\Big{\langle}\underbrace{F(q(\mathbf{S}|A)\|p_{0}(x,\mathbf{S}|A))}_{=F_{\mathbf{S}}(A)}-G(A)\Big{\rangle}_{q(A)}$
$\displaystyle\quad+D_{\mathrm{KL}}(q(A)\|p_{0}(A))\,,$

where

$G(A)\coloneqq\Big{\langle}\underbrace{\langle\log
p_{\mathrm{des}}(X^{\prime})\rangle_{p_{0}(X^{\prime}|S^{\prime})}}_{\eqqcolon
g(S^{\prime})}\Big{\rangle}_{q(S^{\prime}|A)}\,.$