1,T\rrbracket,\,\,\forall\pi\in\llbracket 1,|\pi|\rrbracket$ where
$\tilde{\bm{D}}_{\tau}$ is an $\text{m}_{5}$ message and $\tilde{\alpha}_{k}$
along with $\tilde{\bm{D}}_{\tau-1}$ are $\text{m}_{4}$ messages.

##### Messages for $\pi$:

$\displaystyle Q^{*}(\pi)$ $\displaystyle=\text{Cat}(\pi;\alpha^{*})\,\,\text{
where
}\,\,\alpha^{*}=\sigma\Bigg{(}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bar{\alpha}}+\sum_{\tau=1}^{T}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\mathbb{F}_{\tau}}\Bigg{)}\,\,\text{
and
}\,\,\mathbb{F}_{\tau}=\begin{bmatrix}\langle\tilde{\bm{D}}_{\tau}\otimes\tilde{\bm{D}}_{\tau-1},\bar{\bm{B}}[U_{\tau-1}^{1}]\rangle_{F}\\\
...\\\
\langle\tilde{\bm{D}}_{\tau}\otimes\tilde{\bm{D}}_{\tau-1},\bar{\bm{B}}[U_{\tau-1}^{|\pi|}]\rangle_{F}\end{bmatrix}$

If we keep applying the same reasoning, we see that
$\mu_{\pi}(\bar{\alpha})=\bar{\alpha}$ is an $\text{m}_{2}$ message, which is
a function of the sufficient statistics of the parent variable $\alpha$
($\text{m}_{1}$ message). Moreover, $\pi$ has one child variable ($S_{\tau}$)
for each time step $\tau\in\llbracket 1,T\rrbracket$, and for each of those
child variables, $\pi$ has two co-parent variables ($S_{\tau-1}$ and
$\bm{B}$). Therefore,
$\mu_{S_{\tau}\rightarrow\pi}=\mathbb{F}_{\tau}\,\,\forall\tau\in\llbracket
1,T\rrbracket$ correspond to $T\,\,\text{m}_{3}$ messages. Those messages are
function of two $\text{m}_{4}$ messages ($\tilde{\bm{D}}_{\tau-1}$ and
$\bar{\bm{B}}$) and one $\text{m}_{5}$ message ($\tilde{\bm{D}}_{\tau}$).

##### Messages for $S_{\tau}$:

$\displaystyle Q^{*}(S_{\tau})$
$\displaystyle=\text{Cat}(S_{\tau};\sigma(\mu_{S_{\tau}}^{*}))$
$\mu_{S_{\tau}}^{*}=[\tau=0]{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bar{\bm{D}}}+[\tau\neq
0]\sum_{k}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau-1}^{k}]\tilde{\bm{D}}_{\tau-1}}+[\tau\leq
t]{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{o}_{\tau}\cdot\bar{\bm{A}}}+[\tau\neq
T]\sum_{k}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau+1}\cdot\bar{\bm{B}}[U_{\tau}^{k}]}$

To understand the above equation, we can consider two cases: $\tau=0$ and
$\tau\neq 0$. In the first case, $S_{0}$ only has one parent variable
($\bm{D}$), and $\mu_{S_{0}}(\bar{\bm{D}})=\bar{\bm{D}}$ where
$\bar{\bm{D}}=\langle u_{D}(\bm{D})\rangle_{Q_{D}}$ is a message from a parent
variable ($\text{m}_{1}$ message). In the second case, $S_{\tau}$ has three
parent variables ($S_{\tau-1}$, $\bm{B}$ and $\pi$), and
$\mu_{S_{\tau}}(\tilde{\bm{D}}_{\tau-1},\bar{\bm{B}},\tilde{\alpha})=\sum_{k}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau}^{k}]\tilde{\bm{D}}_{\tau-1}$
where $\tilde{\bm{D}}_{\tau-1}$, $\bar{\bm{B}}$ and $\tilde{\alpha}$ are also
$\text{m}_{1}$ messages. Let us now think about the child variable(s) of
$S_{\tau}$. If $\tau\leq t$, then $S_{\tau}$ has a child variable from the
likelihood mapping and $\mu_{O_{\tau}\rightarrow
S_{\tau}}(\bm{o}_{\tau},\bar{\bm{A}})=\bm{o}_{\tau}\cdot\bar{\bm{A}}$, where
$\bm{o}_{\tau}$ is a message from the child variable ($\text{m}_{5}$ message)
and $\bar{\bm{A}}$ is a message from the co-parent variable ($\text{m}_{4}$
message). Additionally, if $\tau\neq T$, then $S_{\tau}$ receives a message
from the future $\mu_{S_{\tau+1}\rightarrow
S_{\tau}}(\tilde{\alpha}_{k},\tilde{\bm{D}}_{\tau+1},\bar{\bm{B}})=\sum_{k}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau+1}\cdot\bar{\bm{B}}[U_{\tau}^{k}]$,
where $\tilde{\alpha}_{k}$ and $\bar{\bm{B}}$ are $\text{m}_{4}$ messages and
$\tilde{\bm{D}}_{\tau+1}$ is a $\text{m}_{5}$ message. Figure 18 concludes
this section with an illustration the message passing procedure for $S_{0}$.

Figure 18:  This figure illustrates the passing of messages required to update
the posterior over $S_{0}$. The messages of type
${\color[rgb]{0.56,0.14,0.56}\definecolor[named]{pgfstrokecolor}{rgb}{0.56,0.14,0.56}\text{m}_{1}}$,
${\color[rgb]{0.828125,0.69140625,0.05859375}\definecolor[named]{pgfstrokecolor}{rgb}{0.828125,0.69140625,0.05859375}\text{m}_{2}}$,
${\color[rgb]{0.01,0.75,0.24}\definecolor[named]{pgfstrokecolor}{rgb}{0.01,0.75,0.24}\text{m}_{3}}$,
${\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\text{m}_{4}}$
and
${\color[rgb]{0,0,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.5}\text{m}_{5}}$
come from the parent variables, parent factors, child factors, co-parent
variables and child variables, respectively.

###  7.8 Messages vs update equations

In this section, we present a side by side comparison of the messages obtained
using variational message passing and the update equations that underwrite
belief updating in the active inference literature. Throughout this section,
the messages will always be presented first, followed by the equivalent update
equations. Let us start with the random variable $\bm{D}$:

$\displaystyle Q^{*}(\bm{D})$
$\displaystyle=\text{Dir}(\bm{D};{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}d}+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{0}})$
$\displaystyle Q^{*}(\bm{D})$ $\displaystyle=\text{Dir}(\bm{D};d+\bm{s}_{0})$

These two equations only differ in terms of labels, i.e. $\bm{s}_{0}$ and
$\tilde{\bm{D}}_{0}$ conceptually represent the same quantity. Similarly, the
updates of $\bm{A}$ are recovered up to a change of label:

$\displaystyle Q^{*}(\bm{A})$
$\displaystyle=\prod_{i}\text{Dir}(\bm{A}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i},\bm{a}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})\quad\text{
where
}\quad\bm{a}={\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}a}+\sum_{\tau=0}^{t}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{o}_{\tau}}\otimes{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{\tau}}$
$\displaystyle Q^{*}(\bm{A})$
$\displaystyle=\prod_{i}\text{Dir}(\bm{A}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i},\bm{a}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})\quad\text{
where }\quad\bm{a}=a+\sum_{\tau=0}^{t}\bm{o}_{\tau}\otimes\bm{s}_{\tau}$

The update of $\bm{B}$ slightly differs from the messages obtained from
variational message passing, which follows from the fact that we modified the
variational distribution:

$\displaystyle Q^{*}(\bm{B})$
$\displaystyle=\prod_{u,i}\text{Dir}(\bm{B}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i},\bm{b}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})\quad\text{
where
}\quad\bm{b}[u]={\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[u]}+\sum_{(k,\tau)\in\Omega_{u}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{\tau}}\otimes{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{\tau-1}}$
$\displaystyle Q^{*}(\bm{B})$
$\displaystyle=\prod_{u,i}\text{Dir}(\bm{B}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i},\bm{b}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})\,\quad\text{
where
}\quad\bm{b}[u]=b[u]+\sum_{(k,\tau)\in\Omega_{u}}\bm{\pi}_{k}\bm{s}^{k}_{\tau}\otimes\bm{s}^{k}_{\tau-1}$

The only conceptual difference here is that $\bm{s}^{k}_{\tau}$ depended upon
the policy, while $\tilde{\bm{D}}$ does not. Concerning $S_{\tau}$, we have
re-arranged the update equation to highlight the similarity with the messages:

$\displaystyle Q^{*}(S_{\tau})$
$\displaystyle=\text{Cat}(S_{\tau};\sigma(\mu_{S_{\tau}}^{*}))$
$\displaystyle\mu_{S_{\tau}}^{*}$
$\displaystyle=[\tau=0]{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bar{\bm{D}}}+[\tau\neq
0]\sum_{k}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau-1}^{k}]\tilde{\bm{D}}_{\tau-1}}+[\tau\leq
t]{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{o}_{\tau}\cdot\bar{\bm{A}}}+[\tau\neq
T]\sum_{k}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau+1}\cdot\bar{\bm{B}}[U_{\tau}^{k}]}$
$\displaystyle\mu_{S_{\tau}}^{*}$
$\displaystyle=[\tau=0]\bm{\bar{D}}+[\tau\neq
0]\,\quad\quad\quad\bm{\bar{B}}[U_{\tau-1}^{\pi}]\bm{s}^{\pi}_{\tau-1}\,+[\tau\leq
t]\bm{o}_{\tau}\cdot\bm{\bar{A}}+[\tau\neq
T]\quad\quad\,\,\,\,\,\,\,\bm{s}^{\pi}_{\tau+1}\cdot\bm{\bar{B}}[U_{\tau}^{\pi}]$

There are two main differences here. First, as for $\bm{B}$,
$\bm{s}^{k}_{\tau}$ is replaced by $\tilde{\bm{D}}$, which does not depend on
the policies. Second, the past and future messages have an average over the
policies, while the updates do not. Unsurprisingly, since we replaced $\gamma$
by $\alpha$ and changed the type of distributions, the updates are quite
different:

$\displaystyle Q^{*}(\alpha)$
$\displaystyle=\text{Dir}\Big{(}\alpha;{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\theta}+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}}\Big{)}$
$\displaystyle Q^{*}(\gamma)$
$\displaystyle=\Gamma\Big{(}\gamma;1,\beta+\bm{G}\cdot(\bm{\pi}-\bm{\pi}_{0})\Big{)}$

We conclude this section with the messages and updates of $\pi$, which are
formally distinct. These differences come from the fact that we moved $\bm{G}$
from $P(\pi|\gamma)$ to $P(\alpha)$ and turned $P(\pi|\gamma)$ into a
categorical distribution $P(\pi|\alpha)$:

$\displaystyle Q^{*}(\pi)$ $\displaystyle=\text{Cat}(\pi;\alpha^{*})$
$\displaystyle\alpha^{*}$
$\displaystyle=\sigma\Bigg{(}\,\quad{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bar{\alpha}}\,\quad+\sum_{\tau=1}^{T}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\mathbb{F}_{\tau}}\Bigg{)}\,\,\text{
and
}\,\,\mathbb{F}_{\tau}=\begin{bmatrix}\langle\tilde{\bm{D}}_{\tau}\otimes\tilde{\bm{D}}_{\tau-1},\bar{\bm{B}}[U_{\tau-1}^{1}]\rangle_{F}\\\
...\\\
\langle\tilde{\bm{D}}_{\tau}\otimes\tilde{\bm{D}}_{\tau-1},\bar{\bm{B}}[U_{\tau-1}^{|\pi|}]\rangle_{F}\end{bmatrix}$
$\displaystyle\alpha^{*}$
$\displaystyle=\sigma\Bigg{(}-\frac{1}{\bm{\beta}}\bm{G}+\sum_{\tau=1}^{T}\mathbb{F}_{\tau}\Bigg{)}\,\,\text{
and
}\,\,\mathbb{F}_{\tau}=\bm{s}^{\pi}_{\tau}\cdot\bm{\bar{B}}[U]\bm{s}^{\pi}_{\tau-1}$

However, the general form of the updates remains unchanged with information
coming from the parent through $\bar{\alpha}$ and
$-\frac{1}{\bm{\beta}}\bm{G}$, and from each child through the summation over
time steps.

##  8 Conclusion

The increasing use of active inference in neuroscience has cast many brain
processes as Bayesian inference, the update equations of which can be thought
of as a message passing procedure. The first goal of this paper was to present
a complete overview of the active inference framework in discrete time and
state space (Section 5) as well as a formal introduction to the variational
message passing literature (Section 6). Then, we simplified the generative
model and the variational distribution usually adopted in the active inference
to derive a new set of update equations using the method of Winn and Bishop
(2005) — and highlight the connection between active inference and variational
message passing (Section 7).

We hope that the first few sections of this paper could be useful as an
introduction to variational inference, Forney factor graphs, active inference
or/and variational message passing. Section 7 might also be of interest to
researchers searching for a clear link between active inference and
variational message passing or researchers seeking to derive the update
equations of new generative models. Section 7 explains why a fully factorised
variational distribution simplifies the expected free energy in a way that
precludes risk sensitive behaviour but preserves ambiguity avoidance. Finally,
we note that this issue does not confound generative models implementing tree
search.

One might ask why previous formulations of belief updating or message passing
in active inference have not exploited the simplifications considered in the
current paper. For example, using a Dirichlet distribution to parameterise
Bayesian beliefs over policies — or a fully factorised variational
distribution that would simplify message passing. One answer is that much of
the legacy literature in active inference is concerned with neuronal process
theories and biological implementation. For example, the only reason a Gibbs
form was used for the distribution over policies was to link the implicit
temperature or sensitivity parameter to dopaminergic discharges. Similarly,
the minimisation of variational free energy — using a gradient descent to
implement structured variational message passing — was motivated by the need
to cast belief updating in terms of differential equations that could be
plausibly associated with neuronal dynamics (and accompanying
electrophysiological responses to observations). However, if one frees oneself
from the constraints of biological implementation, the repertoire of
established schemes in machine learning and Bayesian statistics can, in
principle, be leveraged to reproduce kinds of choice behaviour active
inference is trying to explain and emulate. This paper has highlighted the
putative usefulness of variational message passing under a rationalisation of
generative models.

It is interesting to consider whether the simplified expected free-energy —
resulting from our message passing formulation of active inference — can be
linked in any sense to human behaviour, whether normative or pathological. In
particular, the free-energy we have obtained reflects a very specific
functional impoverishment. The full factorisation that is necessary for
vanilla message passing precludes the ability to conditionalize the
variational posterior on policies. This suggests a particular deficit in the
ability to plan, and a blindness to future possibilities, the uncertainty
associated with those possibilities and their potential to satisfy
preferences. As a result, the agent’s objective becomes to seek out
unambiguous cues, with no concern for outcome.

In fact, humans do exhibit patterns of behaviour that — due to their
repetitiveness — seem to reflect a desire for high predictability.
Additionally, some of these patterns do not seem obviously connected to
rewarding or punishing outcomes. For example, those with autism can exhibit
very stereotyped repetitive behaviour: hand flapping, hand clapping, rocking,
etc (Gabriels, 2005), which is often described as stimming (Sundar Rajagopalan
et al., 2013). These repetitive and ritualistic behaviours (Lam, 2007) suggest
an objective to avoid exploration and the associated uncertainty.

This work naturally leads to future directions of research. For example, one
could implement the new generative model proposed in this paper and compare
its performance with the model presented in Section 5. Furthermore, additional
research needs to be done to connect the original update equations of active
inference to the cluster variational message passing literature. Much work has
already been done on structured variational message passing; particularly
relation to marginal message passing — and its advantages over related
approaches based upon Bethe free energy (Yedidia, 2005; Parr et al., 2019).
Another interesting direction of research would be to design new generative
models that can tackle more complex tasks, such as playing Atari games, human-
machine interaction using natural language and automatic structure learning.
Partial answers to these directions of research have already been provided
with the use of deep active inference (Fountas et al., 2020; Ueltzhöffer,
2018; Tschantz et al., 2020), deep temporal models (Friston et al., 2018;
Heins et al., 2020) and Bayesian model reduction (Friston et al., 2018, 2017a;
Wauthier et al., 2020). Nevertheless, we anticipate that additional work will
pursue these avenues of research. Finally, one could also compare the update
schemes under VMP to belief propagation (Yedidia, 2011) or marginal message
passing (Parr et al., 2019).

Acknowledgments

We would like to thank Karl Friston as well as the reviewers for their
valuable feedback, which greatly improved the quality of the present paper.

## Appendix A: Active Inference, KL Control and Reinforcement Learning.

This appendix focuses on the relationship between Active Inference, KL Control
and Reinforcement Learning (cf. Da Costa et al. (2020b) and Levine (2018) for
more details). Let us restart with the expected free energy given by Equation
5:

$\displaystyle\bm{G}(\pi)\approx\sum_{\tau=t+1}^{T}\underbrace{D_{\mathrm{KL}}[\overbrace{Q(O_{\tau}|\pi)}^{\text{expected
outcomes}}||\overbrace{P(O_{\tau})}^{\text{prior
preferences}}]}_{\text{expected
risk}}\,\,+\,\,\underbrace{\mathbb{E}_{Q(S_{\tau}|\pi)}[\text{H}[P(O_{\tau}|S_{\tau})]]}_{\text{expected
ambiguity}}.$

If the expected ambiguity is equal to zero, then the expected free energy
reduces to the expected risk, which is the cost function minimised in the KL
control literature. This highlights that active inference generalises KL
control (Rawlik et al., 2013) by taking into account the ambiguity of the
mapping between the hidden states and the observations. Active inference
therefore selects policies leading to unambiguous states. Furthermore, the
expected risk can be re-written as follows:

$\displaystyle\text{expected
risk}=D_{\mathrm{KL}}[Q(O_{\tau}|\pi)||P(O_{\tau})]=\underbrace{\mathbb{E}_{Q(O_{\tau}|\pi)}[\ln
Q(O_{\tau}|\pi)]}_{\text{negative
entropy}}-\,\underbrace{\mathbb{E}_{Q(O_{\tau}|\pi)}[P(O_{\tau})]}_{\text{expected
rewards}}.$

If the negative entropy is zero, then the expected free energy reduces to the
negative expected prior preference. Those preferences encode the notion of
good outcomes, or equivalently, the notion of rewarding observations. This
highlights why active inference can be thought of as a generalisation of
reinforcement learning (Mnih et al., 2013). Another view on the expected free
energy is:

$\displaystyle\bm{G}(\tau,\pi)$
$\displaystyle=\overbrace{\mathbb{E}_{\tilde{Q}}[\ln Q(S_{\tau}|\pi)-\ln
P(S_{\tau}|O_{\tau},\pi)]}^{\text{(-ve) epistemic
value}}-\overbrace{\mathbb{E}_{\tilde{Q}}[\ln
P(O_{\tau}|\pi)]}^{\text{extrinsic value}},$ (106)

where $\tilde{Q}=P(O_{\tau}|S_{\tau})Q(S_{\tau})$. The extrinsic value is
another term for expected prior preferences, which is equivalent to expected
rewards in reinforcement learning. It is worth looking in more detail at the
negative epistemic value (-EV), which differentiates the learning objectives
of reinforcement learning and active inference:

$\displaystyle-EV$ $\displaystyle=-\overbrace{-\mathbb{E}_{\tilde{Q}}[\ln
Q(S_{\tau}|\pi)-\ln P(S_{\tau}|O_{\tau},\pi)]}^{\text{epistemic value}}$
$\displaystyle\Leftrightarrow\quad\quad EV$
$\displaystyle=\underbrace{\mathbb{E}_{\tilde{Q}}[\ln
P(S_{\tau}|O_{\tau},\pi)-\ln Q(S_{\tau}|\pi)]}_{\text{mutual information
between }S_{\tau}\text{ and }O_{\tau}}.$

Thus, the epistemic value is approximately equal to the mutual information
between $S_{\tau}$ and $O_{\tau}$. The mutual information encodes the expected
information gain over one variable by knowing the value of another. Therefore,
the epistemic value tells us how knowing future observations reduces our
uncertainty over future hidden states. The following should help to see that
the epistemic value is approximately equal to the mutual information between
$S_{\tau}$ and $O_{\tau}$:

$\displaystyle I(S;O)$
$\displaystyle=D_{\mathrm{KL}}\left[\left.\left.P(S_{\tau},O_{\tau})\right|\right|P(S_{\tau})P(O_{\tau})\right]$
$\displaystyle=\mathbb{E}_{P(S_{\tau},O_{\tau})}[\ln P(S_{\tau}|O_{\tau})+\ln
P(O_{\tau})-\ln P(S_{\tau})-\ln P(O_{\tau})]$
$\displaystyle=\mathbb{E}_{P(S_{\tau},O_{\tau})}[\ln P(S_{\tau}|O_{\tau})-\ln
P(S_{\tau})].$

Intuitively, the more an observation tells us about future states, the more
valuable this observation is. The negative epistemic value from equation 106
directly reflects this intuition, and favours the policies with high mutual
information. More importantly, equation 106 allows the agent to compare the
information gain and the reward on the same scale, i.e. using nats from
information theory. This creates a sense in which an active inference agent
deals optimally with the trade-off between exploration and exploitation.

## Appendix B: Useful Properties.

This appendix quickly reviews the properties used throughout this paper.

##### Product rule:

$P(X,Y)=P(X|Y)P(Y)$, where $X$ and $Y$ are random variables.

##### Linearity of expectation:

$\mathbb{E}_{P(Y)}[aY+b]=a\mathbb{E}_{P(Y)}[Y]+b$, where $a$ and $b$ are
constants, and $Y$ is a random variable.

##### Expectation of a constant:

$\mathbb{E}_{P(Y)}[a]=a$, where $a$ is a constant, and $Y$ is a random
variable

##### Log property:

$\ln(ab)=\ln(a)+\ln(b)$, where $a$ and $b$ are real numbers

##### Exponential product property:

$\exp(a+b)=\exp(a)\exp(b)$, where $a$ and $b$ are real numbers

##### Exponential power property:

$\exp(ab)=\exp(a)^{b}$, where $a$ and $b$ are real numbers

## Appendix C: Definition and Justification of the Expected Free Energy.

In this appendix, we focus on the definition of the expected free energy and
the justification of Equation 5. Another good resource on the subject is the
“expected free energy” appendix of Smith et al. (2021). For the sake of
simplicity, we assume the following generative model and variational
distribution:

$P(O_{0:T},S_{0:T},\bm{B}|\pi)=P(\bm{B})P(S_{0})\prod_{\tau=1}^{T}P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\prod_{\tau=0}^{T}P(O_{\tau}|S_{\tau})$
$Q(S_{0:T},\bm{B}|\pi)=Q(\bm{B})\prod_{\tau=0}^{T}Q(S_{\tau}|\pi).$

Furthermore, we let $X=\\{\bm{B},S_{0:T}\\}$ denote the set of hidden
variables of the model. Note that in this appendix, we restrict ourself to the
hidden variables $X$ but new variables such as $\bm{A}$ and $\bm{D}$ can be
added without changing the idea of the following derivation. Initially, the
expected free energy was defined as the variational free energy conditioned on
the policy, i.e.

$\bm{G}(\pi)=D_{\mathrm{KL}}\left[\left.\left.Q(X|\pi)\right|\right|P(O_{0:t},X|\pi)\right].$

However, the above definition does not take into account that observations
will be made in the future. To make up for this, the expected free energy can
be extended as follows:

$\displaystyle\bm{G}(\pi)=\mathbb{E}_{\tilde{Q}}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(X|\pi)\right|\right|P(O_{0:T},X|\pi)\right]\Big{]}\text{
where
}\tilde{Q}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\tilde{Q}(O_{t+1:T}|\pi).$
(107)

Since the future observations ($O_{t+1:T}$) have not been made yet, we need to
predict what they could look like. This prediction relies on a predictive
distribution $\tilde{Q}(O_{t+1:T}|\pi)$ that encodes our best guess about
future outcomes, and is generally defined as follows:

$\displaystyle\tilde{Q}(O_{t+1:T}|\pi)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\prod_{\tau=t+1}^{T}\tilde{Q}(O_{\tau}|\pi),$
$\displaystyle\text{where
}\,\,\,\,\tilde{Q}(O_{\tau}|\pi)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\sum_{S_{\tau}}\tilde{Q}(O_{\tau},S_{\tau}|\pi)\,\,\,\,\text{
and
}\,\,\,\,\tilde{Q}(O_{\tau},S_{\tau}|\pi)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}P(O_{\tau}|S_{\tau})Q(S_{\tau}|\pi).$

Note that the definition of $\tilde{Q}(O_{t+1:T}|\pi)$ assumes independence
between time steps and $\tilde{Q}(O_{\tau}|\pi)$ is obtained by
marginalisation of $\tilde{Q}(O_{\tau},S_{\tau}|\pi)$. By recalling the
definition of the generative model as well as the definition of the
variational distribution, we obtain the following from Equation 107:

$\displaystyle\bm{G}(\pi)$
$\displaystyle=\mathbb{E}_{\tilde{Q}}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{0:T},\bm{B}|\pi)\right|\right|P(O_{0:T},S_{0:T},\bm{B}|\pi)\right]\Big{]}$
$\displaystyle=D_{\mathrm{KL}}\left[\left.\left.Q(\bm{B})\right|\right|P(\bm{B})\right]+D_{\mathrm{KL}}\left[\left.\left.Q(S_{0}|\pi)\right|\right|P(S_{0})\right]{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum}$
$\displaystyle\quad+\,\,\sum_{\tau=1}^{t}\,\,\,\mathbb{E}_{Q(S_{\tau-1},\bm{B}|\pi)}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{\tau}|\pi)\right|\right|P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\right]\Big{]}$
$\displaystyle\quad+\,\,\sum_{\tau=0}^{t}\,\,\,\mathbb{E}_{Q(S_{\tau}|\pi)}\Big{[}\text{H}[P(O_{\tau}|S_{\tau})]\Big{]}$
$\displaystyle\quad+\sum_{\tau=t+1}^{T}\mathbb{E}_{Q(S_{\tau-1},\bm{B}|\pi)}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{\tau}|\pi)\right|\right|P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\right]\Big{]}+\mathbb{E}_{Q(S_{\tau}|\pi)}\Big{[}\text{H}[P(O_{\tau}|S_{\tau})]\Big{]}.$

It must now be mentioned that the policy does not have much of an impact on
the past and current hidden states ($S_{0:t}$). The terms relying on those
states are then removed from the expected free energy to avoid unnecessary
computational costs. Additionally, the divergence between $Q(\bm{B})$ and
$P(\bm{B})$ does not depend on the policy and can be safely ignored, leading
to:

$\displaystyle\bm{G}(\pi)=\sum_{\tau=t+1}^{T}\bm{G}(\pi,\tau)$ (108)

where:

$\bm{G}(\pi,\tau)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\mathbb{E}_{Q(S_{\tau-1},\bm{B}|\pi)}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{\tau}|\pi)\right|\right|P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\right]\Big{]}+\mathbb{E}_{Q(S_{\tau}|\pi)}\Big{[}\text{H}[P(O_{\tau}|S_{\tau})]\Big{]}.$

We now focus on $\bm{G}(\pi,\tau)$ to bridge the gap between Equations 5 and
108. First, we merge the two terms of the above equation together:

$\displaystyle\bm{G}(\pi,\tau)\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\mathbb{E}_{P(O_{\tau}|S_{\tau})Q(S_{\tau},S_{\tau-1},\bm{B}|\pi)}\Big{[}\ln
Q(S_{\tau}|\pi)-\ln P(O_{\tau},S_{\tau}|S_{\tau-1},\bm{B},\pi)\Big{]}.$

Then, we break the second term within the expectation using the product rule.
Additionally, we realise that the following equation can be obtained from the
product rule:

$P(O_{\tau}|S_{\tau-1},\bm{B},\pi)=\frac{P(O_{\tau},S_{\tau-1},\bm{B},\pi)}{P(S_{\tau-1},\bm{B},\pi)}=\frac{P(S_{\tau-1},\bm{B},\pi|O_{\tau})}{P(S_{\tau-1},\bm{B},\pi)}P(O_{\tau})\approx
P(O_{\tau}),$

where we assumed that the fraction is equal to one. Doing this assumption
means that the observation $O_{\tau}$ brings us very little information, i.e.
the posterior is close to the prior. Using the above result we get:

$\displaystyle\bm{G}(\pi,\tau)$ $\displaystyle=\mathbb{E}\Big{[}\ln
Q(S_{\tau}|\pi)-\ln P(S_{\tau}|O_{\tau},S_{\tau-1},\bm{B},\pi)-\ln
P(O_{\tau}|S_{\tau-1},\bm{B},\pi)\Big{]}$
$\displaystyle\approx\mathbb{E}\Big{[}\ln Q(S_{\tau}|\pi)-\ln
P(S_{\tau}|O_{\tau},S_{\tau-1},\bm{B},\pi)-\ln P(O_{\tau})\Big{]},$

where the expectation is still over
$P(O_{\tau}|S_{\tau})Q(S_{\tau},S_{\tau-1},\bm{B}|\pi)$. Then, we uses Bayes
theorem on the second term, the fact that
$(O_{\tau}\perp\\!\\!\\!\perp{}{S_{\tau-1},\bm{B},\pi})|S_{\tau}$ and the log
properties to get:

$\displaystyle\bm{G}(\pi,\tau)$ $\displaystyle=\mathbb{E}\Big{[}\ln
Q(S_{\tau}|\pi)-\ln P(S_{\tau}|O_{\tau},S_{\tau-1},\bm{B},\pi)-\ln
P(O_{\tau})\Big{]}$ $\displaystyle=\mathbb{E}\Big{[}\ln
Q(S_{\tau}|\pi)-\ln\frac{P(O_{\tau}|S_{\tau},S_{\tau-1},\bm{B},\pi)P(S_{\tau}|S_{\tau-1},\bm{B},\pi)}{P(O_{\tau}|S_{\tau-1},\bm{B},\pi)}-\ln
P(O_{\tau})\Big{]}$ $\displaystyle\approx\mathbb{E}\Big{[}\ln
Q(S_{\tau}|\pi)-\ln\frac{P(O_{\tau}|S_{\tau})Q(S_{\tau}|\pi)}{Q(O_{\tau}|\pi)}-\ln
P(O_{\tau})\Big{]}$ $\displaystyle=\mathbb{E}\Big{[}\ln Q(O_{\tau}|\pi)-\ln
P(O_{\tau})-\ln P(O_{\tau}|S_{\tau})\Big{]},$

where we assumed that $P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\approx
Q(S_{\tau}|\pi)$ and $P(O_{\tau}|S_{\tau-1},\bm{B},\pi)\approx
Q(O_{\tau}|\pi)$. The first assumption can be supported by the variational
free energy (VFE) decomposition in term of accuracy and complexity. Indeed,
the VFE penalises the divergence between $Q(S_{\tau}|\pi)$ and
$P(S_{\tau}|S_{\tau-1},\bm{B},\pi)$. The second assumption can be supported as
follows:

$\displaystyle P(O_{\tau}|S_{\tau-1},\bm{B},\pi)$
$\displaystyle=\sum_{S_{\tau}}P(O_{\tau},S_{\tau}|S_{\tau-1},\bm{B},\pi)$
$\displaystyle\approx\sum_{S_{\tau}}Q(O_{\tau},S_{\tau}|\pi)$
$\displaystyle=Q(O_{\tau}|\pi).$

Assuming that the posterior $P(O_{\tau},S_{\tau}|S_{\tau-1},\bm{B},\pi)$ can
be approximated by $Q(O_{\tau},S_{\tau}|\pi)$. The last step relies on the
linearity of expectation and the expectation of a constant, leading to the
final result:

$\displaystyle\bm{G}(\pi,\tau)$
$\displaystyle=D_{\mathrm{KL}}\left[\left.\left.Q(O_{\tau}|\pi)\right|\right|P(O_{\tau})\right]+\mathbb{E}_{Q(S_{\tau}|\pi)}\Big{[}\text{H}[P(O_{\tau}|S_{\tau})]\Big{]}.$

## Appendix D: The simplest generative model.

This appendix provides the reader with the smallest generative model that can
be considered as an active inference agent and aims to solve the k-armed
bandit problem. As shown in Figure 19, this problem is composed of k slot
machines or equivalently k actions that the agent can perform. Each machine
has a different probability of producing a reward and the agent must chose the
action to perform to maximize the rewards obtained. The agent only observes
either a reward or a punishment after the execution of an action. Additional
information related to the usage of active inference in the context of the
multi-arms bandit (MAB) task can be found in (Markovic et al., 2021) where
active inference was compared to other major algorithms for solving MABs such
as UCB sampling and Thompson sampling.

Figure 19: This figure illustrates the 3-armed bandit problem and the
generative model used by the agent. Three slot machines are available to the
agent and each machine has a different probability of producing a reward.
Additionally, there are two possible outcomes when pulling a lever, the agent
either wins plenty of money or gets nothing. The generative model is composed
of two nodes representing the possible outcomes and actions. Finally, the
agent’s goal is to maximize the rewards obtained, by picking the best
strategy.

To solve the bandit problem using active inference, the first step is to
create the generative model that encodes the agent’s beliefs of the
environment. Two random variables are used for this purpose, $O$ represents
the possible outcomes and $U$ the available actions. Furthermore, $P(O|U)$
determines how the observation depends on the action performed by the agent,
and $P(U)$ encodes any prior preference over the available actions. More
precisely, $P(O|U)$ and $P(U)$ are categorical distributions defined as
follows:

$\displaystyle P(O=i|U=j)=A_{ij}\quad\quad\text{and}\quad\quad P(U=j)=a_{j},$

where $A_{ij}$ defines the probability of the i-th outcome given that the j-th
action is performed, and $a_{j}$ encodes the prior over the j-th action. Note
that even if the active inference framework provides a way to learn the matrix
$\bm{A}$, this section assumes that it is given to the agent. The next step is
to pick an inference method to compute the posterior over the hidden state
$U$. This section keeps things simple and uses Bayes theorem:

$\displaystyle
P(U=j|O=1)=\frac{P(O=1|U=j)P(U=j)}{P(O=1)}=\frac{P(O=1|U=j)P(U=j)}{\sum_{k}P(O=1|U=k)P(U=k)}=\frac{A_{1j}a_{j}}{\sum_{k}A_{1k}a_{k}},$

where the definition of the generative model has been used in the last step
and we conditioned on $O=1$ to infer the action that is more likely to be
rewarding. At this point, it is possible to act in our environment either by
sampling the next action to perform from the posterior $P(U|O=1)$ or by
picking the action with the highest posterior probability. Additionally, the
posterior can be reused as an empirical prior for the next time step as
follows:

$\displaystyle P(U=j)\leftarrow
P(U=j|O=1)=\frac{A_{1j}a_{j}}{\sum_{k}A_{1k}a_{k}}.$

This simple example does not capture the entire theoretical power of the
active inference framework. Nevertheless, it illustrates four important
concepts related to the design and use of an active inference agent, namely,
the design of a generative model, the inference of the latent variable(s), the
action selection process, and the use of the posterior as an empirical prior.

## Appendix E: Possible future research.

In this appendix, we propose future research directions aiming to understand
the relationship between $P(\pi|\gamma)$ and $P(\pi|\alpha)$. The first
direction relies on the following link between Dirichlet and gamma
distributions. If we let $X_{1}$,$...$,$X_{k}$ be mutually independent random
variables, each having a gamma distribution with parameters $\theta_{i}$ for
$i=1,...,k$ and if we define $Y_{i}=\frac{X_{i}}{X_{1}+...+X_{k}}$ for
$i=1,...,k$, then
$(Y_{1},...,Y_{k})\sim\text{Dir}(\theta_{1},...,\theta_{k})$. This naturally
leads to the hypothesis that the new generative model might be a
generalisation of the old generative model when all $\theta_{i}$ are equal.

Another interesting fact that could be studied in more detail comes from
studying the variance of the Dirichlet distribution. Recall that the variance
of the random variable $Y_{i}$ is given by:

$\text{Var}[Y_{i}]=\frac{\tilde{\theta}_{i}(1-\tilde{\theta}_{i})}{\theta_{0}+1},$

where $\tilde{\theta}_{i}=\frac{\theta_{i}}{\theta_{0}}$ and
$\theta_{0}=\sum_{j=1}^{k}\theta_{j}$. If we stick to our definition of
$\theta$, i.e. $\theta_{j}=c-\bm{G}_{j}$ with
$c=\overrightarrow{c}_{j}\,\,\forall j$, then we can study how the variance of
$Y_{j}$ behaves as $c$ goes to infinity. Let us begin with:

$\lim_{c\rightarrow+\infty}\tilde{\theta}_{i}=\lim_{c\rightarrow+\infty}\frac{\theta_{i}}{\sum_{j=1}^{k}\theta_{j}}=\lim_{c\rightarrow+\infty}\frac{c-\bm{G}_{i}}{\sum_{j=1}^{k}c-\bm{G}_{j}}=\lim_{c\rightarrow+\infty}\frac{c-\bm{G}_{i}}{kc-\sum_{j=1}^{k}\bm{G}_{j}}=\lim_{c\rightarrow+\infty}\frac{c}{kc}=\frac{1}{k},$

where we note that $\bm{G}_{i}$ and $\sum_{j=1}^{k}\bm{G}_{j}$ become
negligible as $c\rightarrow+\infty$. Returning to the limit of the variance:

$\lim_{c\rightarrow+\infty}\text{Var}[Y_{i}]=\lim_{c\rightarrow+\infty}\frac{\tilde{\theta}(1-\tilde{\theta})}{\theta_{0}+1}=\lim_{c\rightarrow+\infty}\frac{\tilde{\theta}(1-\tilde{\theta})}{\Big{(}\sum_{j=1}^{k}c-\bm{G}_{j}\Big{)}+1}=0,$

where we used the fact that $\tilde{\theta}_{i}$ tends towards $\frac{1}{k}$
(i.e. a constant w.r.t $c$) and therefore the variance is only influenced by
the $c$ in the denominator, which tends towards $+\infty$. Additionally, from
the definition of the mode of the Dirichlet, we see that as
$c\rightarrow+\infty$ then the mode of the distribution tends towards the
centre of the simplex because the $\bm{G}_{i}$ becomes negligible, i.e.

$\lim_{c\rightarrow+\infty}m_{\alpha}=\begin{bmatrix}\frac{1}{k}&\ldots&\frac{1}{k}\end{bmatrix}.$

Combining the behaviour of the variance and the mode as $c\rightarrow+\infty$,
we see that as $c$ increases the prior becomes more and more compact around
the centre of the simplex. In other words, the policy selection becomes more
and more stochastic as $c$ increases. This is not without recalling the role
of $\gamma$ as highlighted previously in the caption of Figure 9.

## Appendix F: Messages for B.

In this appendix, we provide the derivation of the messages for $\bm{B}$,
which relies on the conjugacy between a categorical and a Dirichlet
distribution. Let us start with the definition of $P(\bm{B};b)$, which is a
product of Dirichlet distributions that can be written in the following form:

$\displaystyle\ln P(\bm{B};b)$
$\displaystyle=\ln\prod_{i,u}P(\bm{B}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i};b[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})=\sum_{i,u}\ln\text{Dir}(\bm{B}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i};b[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})$
$\displaystyle=\sum_{i,u}\underbrace{\begin{bmatrix}b[u]_{1i}-1\\\ ...\\\
b[u]_{|S|i}-1\end{bmatrix}\cdot\begin{bmatrix}\ln\bm{B}[u]_{1i}\\\ ...\\\
\ln\bm{B}[u]_{|S|i}\end{bmatrix}-\ln
B(b[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})}_{\text{Logartithm
of Dirichlet}}$ $\displaystyle=\underbrace{\begin{bmatrix}b[1]_{11}-1\\\
...\\\
b[|U|]_{|S||S|}-1\end{bmatrix}}_{\mu_{B}(b)}\cdot\underbrace{\begin{bmatrix}\ln\bm{B}[1]_{11}\\\
...\\\
\ln\bm{B}[|U|]_{|S||S|}\end{bmatrix}}_{u_{B}(\bm{B})}-\underbrace{\sum_{i,u}\ln
B(b[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})}_{z_{B}(b)},$
(109)

where $|U|$ is the number of possible actions. Let $\llbracket a,b\rrbracket$
denotes all the natural numbers between $a$ and $b$ (inclusive). The random
matrix $\bm{B}[u]$ has one child $S_{\tau}$ for each time step
$\tau\in\llbracket 1,T\rrbracket$ where action $u$ has been predicted by the
$m$-th policy, and its probability mass function is given by Equation 7.4.
Similarly, the probability mass function of $S_{\tau-1}$ is obtained from
Equation 7.4 by decreasing all indexes $\tau$ by one. The first step requires
us to re-write Equation 7.4 as a function of $u_{B}(\bm{B})$. This can be done
by using the definition of the dot product and re-arranging to obtain:

$\displaystyle\ln
P(S_{\tau}=k|\bm{B},S_{\tau-1}=l,\pi=m)=\underbrace{\begin{bmatrix}\sum_{k}[\pi=k][U_{\tau-1}^{k}=1][S_{\tau-1}=1][S_{\tau}=1]\\\
...\\\
\sum_{k}[\pi=k][U_{\tau-1}^{k}=|U|][S_{\tau-1}=|S|][S_{\tau}=|S|]\end{bmatrix}}_{\mu_{S_{\tau}\rightarrow\bm{B}}(S_{\tau},S_{\tau-1},\pi)}\cdot
u_{B}(\bm{B}).$ (110)

The second step aims to substitute Equations Appendix F: Messages for B. and
110 within the variational message passing equation (85), i.e.

$\displaystyle\ln Q^{*}(\bm{B})$
$\displaystyle=\Big{\langle}\begin{bmatrix}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[1]_{11}-1}\\\
...\\\
{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[|U|]_{|S||S|}-1}\end{bmatrix}\cdot
u_{B}(\bm{B})\Big{\rangle}$
$\displaystyle+\sum_{\tau=1}^{T}\Big{\langle}\begin{bmatrix}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\sum_{k}[\pi=k][U_{\tau-1}^{k}=1][S_{\tau-1}=1][S_{\tau}=1]}\\\
...\\\
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\sum_{k}[\pi=k][U_{\tau-1}^{k}=|U|][S_{\tau-1}=|S|][S_{\tau}=|S|]}\end{bmatrix}\cdot
u_{B}(\bm{B})\Big{\rangle}+\text{Const},$

where
$\langle\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}\rangle$
refers to
$\langle\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}\rangle_{\sim
Q_{\bm{B}}}$. Note that in the above Equation, $b[u]_{ij}$ are hyper
parameters that can therefore be considered as constants with respect to the
expectation
$\langle\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}\rangle_{\sim
Q_{\bm{B}}}$. The third step builds on this insight, by pulling the summation
over time steps inside the vector, factorising by $u_{\bm{B}}(\bm{B})$, using
the linearity of expectation and by taking the exponential of both sides to
obtain:

$\displaystyle Q^{*}(\bm{B})$ $\displaystyle\propto\exp\\{\mu_{B}^{*}\cdot
u_{B}(\bm{B})\\}$ $\displaystyle\mu_{B}^{*}$
$\displaystyle=\begin{bmatrix}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[1]_{11}-1}+\sum_{k,\tau}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k][U_{\tau-1}^{k}=1][S_{\tau-1}=1][S_{\tau}=1]}\rangle\\\
...\\\
{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[|U|]_{|S||S|}-1}+\sum_{k,\tau}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k][U_{\tau-1}^{k}=|U|][S_{\tau-1}=|S|][S_{\tau}=|S|]}\rangle\end{bmatrix}.$

By looking at Equations 7.4, one can see that $\langle[S_{\tau}=i]\rangle$ and
$\langle[S_{\tau-1}=j]\rangle$ are the i-th and j-th elements of the vector
$\langle u_{S_{\tau}}(S_{\tau})\rangle$ and $\langle
u_{S_{\tau}-1}(S_{\tau}-1)\rangle$, respectively. Furthermore, because
$P(\pi)$ is a categorical distribution it can be expressed as:

$\displaystyle P(\pi|\alpha)=\underbrace{\begin{bmatrix}\ln\alpha_{1}\\\
...\\\
\ln\alpha_{|\pi|}\end{bmatrix}}_{\mu_{\pi}(\alpha)}\cdot\underbrace{\begin{bmatrix}[\pi=1]\\\
...\\\ [\pi=|\pi|]\end{bmatrix}}_{u_{\pi}(\pi)},$ (111)

where $|\pi|$ is the number of policies. The above equation highlights that
$\langle[\pi=k]\rangle$ is the k-th element of $\langle u_{\pi}(\pi)\rangle$.
Using those three insights, we proceed with the following re-parameterization
(i.e. the fourth step):

$\displaystyle\mu_{B}^{*}=\begin{bmatrix}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[1]_{11}-1}+\sum_{k,\tau}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau-1}^{k}=1]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau}}(S_{\tau})}\rangle_{1}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau}-1}(S_{\tau}-1)}\rangle_{1}\\\
...\\\
{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[|U|]_{|S||S|}-1}+\sum_{k,\tau}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau-1}^{k}=|U|]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau}}(S_{\tau})}\rangle_{|S|}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau}-1}(S_{\tau}-1)}\rangle_{|S|}\end{bmatrix},$
(112)

where we focused on the optimal parameters because the rest remains unchanged.
The last step consists of computing the expectation of $\langle
u_{S_{\tau}-1}(S_{\tau}-1)\rangle_{i}$, $\langle
u_{S_{\tau}}(S_{\tau})\rangle_{j}$, and $\langle u_{\pi}(\pi)\rangle_{k}$ for
all $i$, $j$ and $k$:

  * •

$\langle
u_{S_{\tau}-1}(S_{\tau}-1)\rangle_{i}=\langle[S_{\tau}-1=i]\rangle=\tilde{\bm{D}}_{(\tau-1)i}$

  * •

$\langle
u_{S_{\tau}}(S_{\tau})\rangle_{j}=\langle[S_{\tau}=j]\rangle=\tilde{\bm{D}}_{\tau
j}$

  * •

$\langle u_{\pi}(\pi)\rangle_{k}=\langle[\pi=k]\rangle=\tilde{\alpha}_{k}$

One last thing we need to look at is the interaction between the summation and
the indicator function in the i-th line of Equation 112. Indeed, the sum
iterates over all time steps $\tau$ and all policies $k$, but the indicator
function filters out all elements where the k-th policy does not predict the
i-th action at time $\tau-1$. Building on this insight, we can now substitute
the above results in Equation 112:

$\displaystyle Q^{*}(\bm{B})$
$\displaystyle\propto\exp\Bigg{\\{}\begin{bmatrix}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[1]_{11}-1}+\sum_{(k,\tau)\in\Omega_{1}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau
1}\tilde{\bm{D}}_{(\tau-1)1}}\\\ ...\\\
{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[|U|]_{|S||S|}-1}+\sum_{(k,\tau)\in\Omega_{|U|}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau|S|}\tilde{\bm{D}}_{(\tau-1)|S|}}\end{bmatrix}\cdot
u_{B}(\bm{B})\Bigg{\\}}.$

Finally, one can recognise in the above equation the logarithm of a product of
Dirichlet distributions written into their exponential form, i.e.

$\displaystyle Q^{*}(\bm{B})$
$\displaystyle=\prod_{u,i}\text{Dir}(\bm{B}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i},\bm{b}[u]_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}i})\quad\text{
where
}\quad\bm{b}[u]={\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}b[u]}+\sum_{(k,\tau)\in\Omega_{u}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{\tau}}\otimes{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\bm{D}}_{\tau-1}}.$

## Appendix G: Messages for $S_{\tau}$.

This appendix shows how to derive the messages for $S_{\tau}$ for all time
steps. We will use Equations 93 and 7.4 that describe $P(S_{0}|\bm{D})$ and
$P(S_{\tau}|S_{\tau-1},\bm{B},\pi)$ as a function of $u_{S_{\tau}}(S_{\tau})$.
The first step requires us to re-arrange Equation 7.4 and
$P(S_{\tau+1}|S_{\tau},\bm{B},\pi)$ as a functions of
$u_{S_{\tau}}(S_{\tau})$, where $P(S_{\tau+1}|S_{\tau},\bm{B},\pi)$ is
obtained by adding one to all instances of $\tau$ in Equation 7.4. Those two
re-arrangements lead to the following results:

$\displaystyle\ln P(O_{\tau}=k|\bm{A},S_{\tau}=l)$
$\displaystyle=\begin{bmatrix}\sum_{i}[O_{\tau}=i]\ln\bm{A}_{i1}\\\ ...\\\
\sum_{i}[O_{\tau}=i]\ln\bm{A}_{i|S|}\end{bmatrix}\cdot\underbrace{\begin{bmatrix}[S_{\tau}=1]\\\
...\\\ [S_{\tau}=|S|]\end{bmatrix}}_{u_{S_{\tau}}(S_{\tau})}$ (113)

$\displaystyle\ln P(S_{\tau+1}=k|\bm{B},S_{\tau}=l,\pi=m)$
$\displaystyle=\begin{bmatrix}\sum_{j,k,u}[\pi=k][U_{\tau}^{k}=u][S_{\tau+1}=j]\ln\bm{B}[u]_{1j}\\\
...\\\
\sum_{j,k,u}[\pi=k][U_{\tau}^{k}=u][S_{\tau+1}=j]\ln\bm{B}[u]_{|S|j}\end{bmatrix}\cdot
u_{S_{\tau}}(S_{\tau}).$ (114)

For the second step, we need to substitute Equations 93, 7.4, 113 and 114 into
the variational message passing equation. If $\tau=0$, the parent message will
come from the prior (i.e. Equation 93) otherwise from the past (i.e. Equation
7.4). Also, for all time steps such that $\tau\leq t$ there is a message from
the likelihood mapping (i.e. Equation 113) and for all time steps except
$\tau=T$ there is a message from the future (i.e. Equation 114). Putting
everything together we obtain:

$\displaystyle\ln Q^{*}(S_{\tau})$
$\displaystyle=\Big{\langle}[\tau=0]\begin{bmatrix}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{D}_{1}}\\\
...\\\
{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{D}_{|S|}}\end{bmatrix}\cdot
u_{S_{\tau}}(S_{\tau})\Big{\rangle}$ $\displaystyle+\Big{\langle}[\tau\neq
0]\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[\pi=k][U_{\tau-1}^{k}=u][S_{\tau-1}=j]\ln\bm{B}[u]_{1j}}\\\
...\\\
\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[\pi=k][U_{\tau-1}^{k}=u][S_{\tau-1}=j]\ln\bm{B}[u]_{|S|j}}\end{bmatrix}\cdot
u_{S_{\tau}}(S_{\tau})\Big{\rangle}$ $\displaystyle+\Big{\langle}[\tau\leq
t]\begin{bmatrix}\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]\ln\bm{A}_{i1}}\\\
...\\\
\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]\ln\bm{A}_{i|S|}}\end{bmatrix}\cdot
u_{S_{\tau}}(S_{\tau})\Big{\rangle}$ $\displaystyle+\Big{\langle}[\tau\neq
T]\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k][U_{\tau}^{k}=u][S_{\tau+1}=j]\ln\bm{B}[u]_{1j}}\\\
...\\\
\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k][U_{\tau}^{k}=u][S_{\tau+1}=j]\ln\bm{B}[u]_{|S|j}}\end{bmatrix}\cdot
u_{S_{\tau}}(S_{\tau})\Big{\rangle}$ $\displaystyle+\text{Const}.$

The third step requires us to factorise by $u_{S_{\tau}}(S_{\tau})$, use the
linearity of expectation and take the exponential of both sides:

$\displaystyle Q^{*}(S_{\tau})$
$\displaystyle\propto\exp\Bigg{\\{}\Big{[}[\tau=0]\mu_{1}^{*}+[\tau\neq
0]\mu_{2}^{*}+[\tau\leq t]\mu_{3}^{*}+[\tau\neq T]\mu_{4}^{*}\Big{]}\cdot
u_{S_{\tau}}(S_{\tau})\Bigg{\\}},$ (115)

where:

$\displaystyle\mu_{1}^{*}=\begin{bmatrix}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{D}_{1}}\rangle\\\
...\\\
\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{D}_{|S|}}\rangle\end{bmatrix}$
$\displaystyle\mu_{2}^{*}=\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[U_{\tau-1}^{k}=u]}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[\pi=k]}\rangle\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[S_{\tau-1}=j]}\rangle\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{B}[u]_{1j}}\rangle\\\
...\\\
\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[U_{\tau-1}^{k}=u]}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[\pi=k]}\rangle\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[S_{\tau-1}=j]}\rangle\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\ln\bm{B}[u]_{|S|j}}\rangle\end{bmatrix}$
$\displaystyle\mu_{3}^{*}=\begin{bmatrix}\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\ln\bm{A}_{i1}}\rangle\\\
...\\\
\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\ln\bm{A}_{i|S|}}\rangle\end{bmatrix}$
$\displaystyle\mu_{4}^{*}=\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau}^{k}=u]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k]}\rangle\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[S_{\tau+1}=j]}\rangle\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\ln\bm{B}[u]_{1j}}\rangle\\\
...\\\
\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau}^{k}=u]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[\pi=k]}\rangle\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[S_{\tau+1}=j]}\rangle\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\ln\bm{B}[u]_{|S|j}}\rangle\end{bmatrix}.$

The fourth step is the re-parameterization relying on the fact that
$\langle\ln\bm{D}_{i}\rangle$, $\langle[\pi=j]\rangle$,
$\langle[S_{\tau-1}=k]\rangle$, $\langle\ln\bm{B}[l]_{mn}\rangle$,
$\langle\ln\bm{A}_{op}\rangle$ and $\langle[S_{\tau+1}=q]\rangle$ are elements
of $\langle u_{D}(\bm{D})\rangle$, $\langle u_{\pi}(\pi)\rangle$, $\langle
u_{S_{\tau-1}}(S_{\tau-1})\rangle$, $\langle u_{B}(\bm{B})\rangle$, $\langle
u_{A}(\bm{A})\rangle$ and $\langle u_{S_{\tau+1}}(S_{\tau+1})\rangle$,
respectively. Focusing on the $\mu_{i}^{*}$ because the rest remains
unchanged, the result of the the re-parameterisation is:

$\displaystyle\mu_{1}^{*}=\begin{bmatrix}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{D}(\bm{D})}\rangle_{1}\\\
...\\\
\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{D}(\bm{D})}\rangle_{|S|}\end{bmatrix}$
$\displaystyle\mu_{2}^{*}=\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[U_{\tau-1}^{k}=u]}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{S_{\tau-1}}(S_{\tau-1})}\rangle_{j}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{B}(\bm{B})}\rangle_{u1j}\\\
...\\\
\sum_{j,k,u}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}[U_{\tau-1}^{k}=u]}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{S_{\tau-1}}(S_{\tau-1})}\rangle_{j}\langle{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}u_{B}(\bm{B})}\rangle_{u|S|j}\rangle\end{bmatrix}$
$\displaystyle\mu_{3}^{*}=\begin{bmatrix}\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{A}(\bm{A})}\rangle_{i1}\\\
...\\\
\sum_{i}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[O_{\tau}=i]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{A}(\bm{A})}\rangle_{i|S|}\end{bmatrix}$
$\displaystyle\mu_{4}^{*}=\begin{bmatrix}\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau}^{k}=u]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau+1}}(S_{\tau+1})}\rangle_{j}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{B}(\bm{B})}\rangle_{u1j}\\\
...\\\
\sum_{j,k,u}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}[U_{\tau}^{k}=u]}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{\pi}(\pi)}\rangle_{k}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{S_{\tau+1}}(S_{\tau+1})}\rangle_{j}\langle{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}u_{B}(\bm{B})}\rangle_{u|S|j}\end{bmatrix}.$

Finally, the last step consists of computing the expectations of all
sufficient statistics as follows:

  * •

$\langle
u_{D}(\bm{D})\rangle_{i}=\langle\ln\bm{D}_{i}\rangle=\psi(\bm{d}_{i})-\psi(\sum_{r}\bm{d}_{r})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bar{\bm{D}}_{i}$

  * •

$\langle u_{\pi}(\pi)\rangle_{j}=\langle[\pi=j]\rangle=\tilde{\alpha}_{j}$

  * •

$\langle
u_{S_{\tau-1}}(S_{\tau-1})\rangle_{k}=\langle[S_{\tau-1}=k]\rangle=\tilde{\bm{D}}_{(\tau-1)k}$

  * •

$\langle
u_{B}(\bm{B})\rangle_{lmn}=\langle\ln\bm{B}[l]_{mn}\rangle=\psi(\bm{b}[l]_{mn})-\psi(\sum_{r}\bm{b}[l]_{rn})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bar{\bm{B}}[l]_{mn}$

  * •

$\langle
u_{A}(\bm{A})\rangle_{op}=\langle\ln\bm{A}_{op}\rangle=\psi(\bm{a}_{op})-\psi(\sum_{r}\bm{a}_{rp})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\bar{\bm{A}}_{op}$

  * •

$\langle
u_{S_{\tau+1}}(S_{\tau+1})\rangle_{q}=\langle[S_{\tau+1}=q]\rangle=\tilde{\bm{D}}_{(\tau+1)q}$

Substituting those expectations into the equations for the $\mu_{i}^{*}$ leads
to the following results: $\mu_{1}^{*}=\bar{\bm{D}}$,
$\mu_{2}^{*}=\sum_{k}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau}^{k}]\tilde{\bm{D}}_{\tau-1}$,
$\mu_{3}^{*}=\bm{o}_{\tau}\cdot\bar{\bm{A}}$ and
$\mu_{4}^{*}=\sum_{k}\tilde{\alpha}_{k}\tilde{\bm{D}}_{\tau+1}\cdot\bar{\bm{B}}[U_{\tau}^{k}]$.
Where $\bm{o}_{\tau}$ is a one hot vector containing the observation made by
the agent and we used the fact that the indicator function $[U_{\tau}^{k}=u]$
filters out elements from the sum where $u\neq U_{\tau}^{k}$. The final result
is obtained by substituting the values of the $\mu_{i}^{*}$’s in Equation 115
to obtain the following categorical distribution:

$\displaystyle Q^{*}(S_{\tau})$
$\displaystyle\propto\exp\Big{\\{}\mu_{S_{\tau}}^{*}\cdot
u_{S_{\tau}}(S_{\tau})\Big{\\}}$
$\mu_{S_{\tau}}^{*}=[\tau=0]{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bar{\bm{D}}}+[\tau\neq
0]\sum_{k}{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau-1}^{k}]\tilde{\bm{D}}_{\tau-1}}+[\tau\leq
t]{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{o}_{\tau}\cdot\bar{\bm{A}}}+[\tau\neq
T]\sum_{k}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\tilde{\alpha}_{k}\bar{\bm{B}}[U_{\tau}^{k}]\tilde{\bm{D}}_{\tau+1}}.$

## Appendix H: Derivation of the new expected free energy.

In this appendix, we derive the expected free energy of our new model. First,
we restate the factorisation of the generative model and the variational
distribution:

$\displaystyle
P(O_{0:t},S_{0:T},\pi,\bm{A},\bm{B},\bm{D},\alpha)\,\,=\,\,\,\,$
$\displaystyle
P(\pi|\alpha)P(\alpha)P(\bm{A})P(\bm{B})P(S_{0}|\bm{D})P(\bm{D})$
$\displaystyle\prod^{t}_{\tau=0}P(O_{\tau}|S_{\tau},\bm{A})\prod^{T}_{\tau=1}P(S_{\tau}|S_{\tau-1},\bm{B},\pi)$
(116) $\displaystyle
Q(S_{0:T},\pi,\bm{A},\bm{B},\bm{D},\alpha)=Q(\pi)Q(\bm{A})Q(\bm{B})Q(\bm{D})Q(\alpha)\prod_{\tau=0}^{T}Q(S_{\tau}).$
(117)

Remembering from Appendix C that the expected free energy is defined as:

$\displaystyle\bm{G}(\pi)=\mathbb{E}_{\tilde{Q}}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(X|\pi)\right|\right|P(O_{0:T},X|\pi)\right]\Big{]},$
(118)

where the latent variables are $X=\\{S_{0:T},\bm{A},\bm{B},\bm{D},\alpha\\}$,
$\tilde{Q}=\tilde{Q}(O_{t+1:T})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\prod_{\tau=t+1}^{T}\tilde{Q}(O_{\tau})$
and
$\tilde{Q}(O_{\tau})\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\sum_{S_{\tau}}\tilde{Q}(O_{\tau},S_{\tau})$.
Now we substitute Equation Appendix H: Derivation of the new expected free
energy. and 117 into Equation 118 and simplify by removing the terms that are
constant w.r.t the policy $\pi$:

$\displaystyle\bm{G}(\pi)$
$\displaystyle=\mathbb{E}_{\tilde{Q}}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{0:T},\bm{A},\bm{B},\bm{D},\alpha|\pi)\right|\right|P(O_{0:T},S_{0:T},\bm{A},\bm{B},\bm{D},\alpha|\pi)\right]\Big{]}{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum}$
$\displaystyle=D_{\mathrm{KL}}\left[\left.\left.Q(\bm{A})\right|\right|P(\bm{A})\right]+D_{\mathrm{KL}}\left[\left.\left.Q(\bm{B})\right|\right|P(\bm{B})\right]+D_{\mathrm{KL}}\left[\left.\left.Q(\bm{D})\right|\right|P(\bm{D})\right]{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum}$
$\displaystyle+D_{\mathrm{KL}}\left[\left.\left.Q(\alpha)\right|\right|P(\alpha)\right]+\mathbb{E}_{Q(\bm{D})}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{0})\right|\right|P(S_{0}|\bm{D})\right]\Big{]}{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum}$
$\displaystyle+\sum_{\tau=1}^{T}\mathbb{E}_{Q(S_{\tau-1},\bm{B})}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{\tau})\right|\right|P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\right]\Big{]}$
$\displaystyle-\sum_{\tau=0}^{T}\mathbb{E}_{Q(S_{\tau},\bm{A})\tilde{Q}(O_{t+1:T})}\Big{[}\ln
P(O_{\tau}|S_{\tau},\bm{A})\Big{]}$
$\displaystyle=\sum_{\tau=1}^{T}\mathbb{E}_{Q(S_{\tau-1},\bm{B})}\Big{[}D_{\mathrm{KL}}\left[\left.\left.Q(S_{\tau})\right|\right|P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\right]\Big{]}+C{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum_{\tau=0}^{T}}$
$\displaystyle=\sum_{\tau=1}^{T}\mathbb{E}_{Q(S_{\tau},S_{\tau-1},\bm{B})}\Big{[}\ln
Q(S_{\tau})-\ln
P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\Big{]}+C{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum_{\tau=0}^{T}}$
$\displaystyle=\sum_{\tau=1}^{T}\mathbb{E}_{Q(S_{\tau-1},\bm{B})}\Big{[}\underbrace{-\mathbb{E}_{Q(S_{\tau})}\big{[}\ln
P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\big{]}}_{H[\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}]}\Big{]}+C{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum_{\tau=0}^{T}}$
$\displaystyle=\sum_{\tau=1}^{T}\mathbb{E}_{Q(S_{\tau-1},\bm{B})}\Big{[}H\big{[}P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\big{]}\Big{]}+C{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}\sum_{\tau=0}^{T}},$

where
$H[\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}]$
refer to $-\mathbb{E}_{Q(S_{\tau})}\big{[}\ln
P(S_{\tau}|S_{\tau-1},\bm{B},\pi)\big{]}$ in the last equation.

## References

  * Berridge (2007) Kent C. Berridge.  The debate over dopamine’s role in reward: the case for incentive salience.  _Psychopharmacology_ , 191(3):391–431, Apr 2007\.  ISSN 1432-2072.  doi: 10.1007/s00213-006-0578-x.  URL https://doi.org/10.1007/s00213-006-0578-x.
  * Bishop and Winn (2003) Christopher Bishop and John Winn.  Structured variational distributions in vibes.  In _Proceedings Artificial Intelligence and Statistics_. Society for Artificial Intelligence and Statistics, Society for Artificial Intelligence and Statistics, January 2003.  URL https://www.microsoft.com/en-us/research/publication/structured-variational-distributions-in-vibes/.  ISBN 0-9727358-0-1.
  * Blei et al. (2017) David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American Statistical Association_ , 112(518):859–877, 2017.  doi: 10.1080/01621459.2017.1285773.  URL https://doi.org/10.1080/01621459.2017.1285773.
  * Blumer et al. (1987) Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth.  Occam’s razor.  _Information Processing Letters_ , 24(6):377 – 380, 1987.  ISSN 0020-0190.  doi: https://doi.org/10.1016/0020-0190(87)90114-1.  URL http://www.sciencedirect.com/science/article/pii/0020019087901141.
  * Botvinick and Toussaint (2012) Matthew Botvinick and Marc Toussaint.  Planning as inference.  _Trends in Cognitive Sciences_ , 16(10):485 – 488, 2012.  ISSN 1364-6613.  doi: https://doi.org/10.1016/j.tics.2012.08.006.
  * Bowman and Li (2011) Howard Bowman and Su Li.  Cognition, concurrency theory and reverberations in the brain: in search of a calculus of communicating (recurrent) neural systems.  In Andrei Voronkov and Margarita Korovina, editors, _Higher-Order Workshop on Automated Runtime Verification and Debugging, EasyChair Proceedings, Festschrift celebrating Howard Barringer’s 60th Birthday_ , volume 1. EasyChair, December 2011.  URL https://kar.kent.ac.uk/30708/.
  * Browne et al. (2012) C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton.  A survey of monte carlo tree search methods.  _IEEE Transactions on Computational Intelligence and AI in Games_ , 4(1):1–43, 2012.
  * Buckley et al. (2017) Christopher L. Buckley, Chang Sub Kim, Simon McGregor, and Anil K. Seth.  The free energy principle for action and perception: A mathematical review.  _Journal of Mathematical Psychology_ , 81:55 – 79, 2017\.  ISSN 0022-2496.  doi: https://doi.org/10.1016/j.jmp.2017.09.004.
  * Champion et al. (2021) Théophile Champion, Howard Bowman, and Marek Grześ.  Active inference and tree search, 2021.
  * Cox et al. (2019) Marco Cox, Thijs van de Laar, and Bert de Vries.  A factor graph approach to automated design of Bayesian signal processing algorithms.  _Int. J. Approx. Reason._ , 104:185–204, 2019.  doi: 10.1016/j.ijar.2018.11.002.  URL https://doi.org/10.1016/j.ijar.2018.11.002.
  * Cozman (2000) F. G. Cozman.  Generalizing variable elimination in Bayesian networks.  _Proc. IBERAMIA/SBIA-2000 Workshops (Workshop on Probabilistic Reasoning in Artificial Intelligence)_ , 2000.  doi: 10.1016/S0004-3702(00)00029-1.  URL https://ci.nii.ac.jp/naid/30008396546/en/.
  * Da Costa et al. (2020a) Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston.  Active inference on discrete state-spaces: a synthesis, 2020a.
  * Da Costa et al. (2020b) Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith.  The relationship between dynamic programming and active inference: the discrete, finite-horizon case, 2020b.
  * FitzGerald et al. (2015) Thomas H. B. FitzGerald, Raymond J. Dolan, and Karl Friston.  Dopamine, reward learning, and active inference.  _Frontiers in Computational Neuroscience_ , 9:136, 2015\.  ISSN 1662-5188.  doi: 10.3389/fncom.2015.00136.  URL https://www.frontiersin.org/article/10.3389/fncom.2015.00136.
  * Fodor and Pylyshyn (1988) Jerry A. Fodor and Zenon W. Pylyshyn.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1):3 – 71, 1988.  ISSN 0010-0277.  doi: https://doi.org/10.1016/0010-0277(88)90031-5.  URL http://www.sciencedirect.com/science/article/pii/0010027788900315.
  * Forney (2001) G. D. Forney.  Codes on graphs: normal realizations.  _IEEE Transactions on Information Theory_ , 47(2):520–548, 2001.
  * Fountas et al. (2020) Zafeirios Fountas, Noor Sajid, Pedro A. M. Mediano, and Karl Friston.  Deep active inference agents using Monte-Carlo methods, 2020.
  * Fox and Roberts (2012) Charles W. Fox and Stephen J. Roberts.  A tutorial on variational bayesian inference.  _Artificial Intelligence Review_ , 38(2):85–95, Aug 2012.  ISSN 1573-7462.  doi: 10.1007/s10462-011-9236-8.  URL https://doi.org/10.1007/s10462-011-9236-8.
  * Friston (2010) Karl Friston.  The free-energy principle: a unified brain theory?  _Nature Reviews Neuroscience_ , 11(2):127–138, Feb 2010.  ISSN 1471-0048.  doi: 10.1038/nrn2787.  URL https://doi.org/10.1038/nrn2787.
  * Friston (2019) Karl Friston.  A free energy principle for a particular physics, 2019.
  * Friston et al. (2013) Karl Friston, Philipp Schwartenbeck, Thomas Fitzgerald, Michael Moutoussis, Tim Behrens, and Raymond Dolan.  The anatomy of choice: active inference and agency.  _Frontiers in Human Neuroscience_ , 7:598, 2013.  ISSN 1662-5161.  doi: 10.3389/fnhum.2013.00598.  URL https://www.frontiersin.org/article/10.3389/fnhum.2013.00598.
  * Friston et al. (2015) Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Giovanni Pezzulo.  Active inference and epistemic value.  _Cognitive Neuroscience_ , 6(4):187–214, 2015\.  doi: 10.1080/17588928.2015.1020053.  URL https://doi.org/10.1080/17588928.2015.1020053.  PMID: 25689102.
  * Friston et al. (2016) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, John O Doherty, and Giovanni Pezzulo.  Active inference and learning.  _Neuroscience & Biobehavioral Reviews_, 68:862 – 879, 2016.  ISSN 0149-7634.  doi: https://doi.org/10.1016/j.neubiorev.2016.06.022.
  * Friston et al. (2018) Karl Friston, Thomas Parr, and Peter Zeidman.  Bayesian model reduction.  _arXiv e-prints_ , art. arXiv:1805.07092, May 2018.
  * Friston et al. (2020) Karl Friston, Lancelot Da Costa, Danijar Hafner, Casper Hesp, and Thomas Parr.  Sophisticated inference, 2020.
  * Friston et al. (2017a) Karl J. Friston, Marco Lin, Christopher D. Frith, Giovanni Pezzulo, J. Allan Hobson, and Sasha Ondobaka.  Active Inference, Curiosity and Insight.  _Neural Computation_ , 29(10):2633–2683, 10 2017a.  ISSN 0899-7667.  doi: 10.1162/neco˙a˙00999.  URL https://doi.org/10.1162/neco_a_00999.
  * Friston et al. (2017b) Karl J. Friston, Thomas Parr, and Bert de Vries.  The graphical brain: Belief propagation and active inference.  _Network Neuroscience_ , 1(4):381–414, 2017b.  doi: 10.1162/NETN“˙a“˙00018.  URL https://doi.org/10.1162/NETN_a_00018.
  * Friston et al. (2018) Karl J. Friston, Richard Rosch, Thomas Parr, Cathy Price, and Howard Bowman.  Deep temporal models and active inference.  _Neuroscience & Biobehavioral Reviews_, 90:486 – 501, 2018.  ISSN 0149-7634.  doi: https://doi.org/10.1016/j.neubiorev.2018.04.004.  URL http://www.sciencedirect.com/science/article/pii/S0149763418302525.
  * Gabriels (2005) Michael L; Hill Dina E; Ivers Bonnie J; Goldson Edward Gabriels, Robin L; Cuccaro.  Repetitive behaviors in autism: relationships with associated clinical features.  _Research in developmental disabilities_ , 2005.  ISSN 0891-4222.
  * Heins et al. (2020) R. Conor Heins, M. Berk Mirza, Thomas Parr, Karl Friston, Igor Kagan, and Arezoo Pooresmaeili.  Deep active inference and scene construction.  _Frontiers in Artificial Intelligence_ , 3:81, 2020.  ISSN 2624-8212.  doi: 10.3389/frai.2020.509354.  URL https://www.frontiersin.org/article/10.3389/frai.2020.509354.
  * Itti and Baldi (2009) Laurent Itti and Pierre Baldi.  Bayesian surprise attracts human attention.  _Vision Research_ , 49(10):1295 – 1306, 2009\.  ISSN 0042-6989.  doi: https://doi.org/10.1016/j.visres.2008.09.007.  URL http://www.sciencedirect.com/science/article/pii/S0042698908004380.  Visual Attention: Psychophysics, electrophysiology and neuroimaging.
  * Kojima and Kangawa (2005) Masayasu Kojima and Kenji Kangawa.  Ghrelin: Structure and function.  _Physiological Reviews_ , 85(2):495–522, 2005\.  doi: 10.1152/physrev.00012.2004.  URL https://doi.org/10.1152/physrev.00012.2004.  PMID: 15788704.
  * Koller and Friedman (2009) D Koller and N Friedman.  Probabilistic graphical models, massachusetts, 2009.
  * Kschischang et al. (2001) F. R. Kschischang, B. J. Frey, and H. . Loeliger.  Factor graphs and the sum-product algorithm.  _IEEE Transactions on Information Theory_ , 47(2):498–519, 2001.  doi: 10.1109/18.910572.
  * Lam (2007) K. S. Lam.  The repetitive behavior scale-revised : Independent validation in individuals with autism spectrum disorders.  _Journal of Autism and Developmental Disorders_ , 37:855–866, 2007.  URL https://ci.nii.ac.jp/naid/20001501751/en/.
  * Lample and Chaplot (2016) Guillaume Lample and Devendra Singh Chaplot.  Playing fps games with deep reinforcement learning, 2016.
  * LeCun and Cortes (2010) Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.  URL http://yann.lecun.com/exdb/mnist/.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
  * Lin et al. (2018) Wu Lin, Nicolas Hubacher, and Mohammad Emtiyaz Khan.  Variational message passing with structured inference networks, 2018.
  * Markovic et al. (2021) Dimitrije Markovic, Hrvoje Stojic, Sarah Schwoebel, and Stefan J. Kiebel.  An empirical evaluation of active inference in multi-armed bandits, 2021\.
  * Millidge et al. (2020) Beren Millidge, Alexander Tschantz, and Christopher L Buckley.  Whence the expected free energy?, 2020.
  * Mirza et al. (2016) M. Berk Mirza, Rick A. Adams, Christoph D. Mathys, and Karl J. Friston.  Scene construction, visual foraging, and active inference.  _Frontiers in Computational Neuroscience_ , 10:56, 2016\.  ISSN 1662-5188.  doi: 10.3389/fncom.2016.00056.  URL https://www.frontiersin.org/article/10.3389/fncom.2016.00056.
  * Mirza et al. (2018) M. Berk Mirza, Rick A. Adams, Christoph Mathys, and Karl J. Friston.  Human visual exploration reduces uncertainty about the sensed world.  _PLOS ONE_ , 13(1):1–20, 01 2018.  doi: 10.1371/journal.pone.0190429.  URL https://doi.org/10.1371/journal.pone.0190429.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning, 2013.
  * Murphy et al. (2013) Kevin Murphy, Yair Weiss, and Michael I. Jordan.  Loopy belief propagation for approximate inference: An empirical study, 2013.
  * Ognibene and Baldassare (2015) D. Ognibene and G. Baldassare.  Ecological active vision: Four bioinspired principles to integrate bottom–up and adaptive top–down attention tested with a simple camera-arm robot.  _IEEE Transactions on Autonomous Mental Development_ , 7(1):3–25, 2015.
  * Parr and Friston (2018) Thomas Parr and Karl J Friston.  Generalised free energy and active inference: can the future cause the past?  _bioRxiv_ , 2018.  doi: 10.1101/304782.  URL https://www.biorxiv.org/content/early/2018/04/23/304782.
  * Parr et al. (2019) Thomas Parr, Markovic Dimitrije, Stefan J. Kiebel, and Karl J. Friston.  Neuronal message passing using mean-field, Bethe, and marginal approximations.  _Scientific Reports (Nature Publisher Group)_ , 9(1), Dec 2019.  URL http://library.kent.ac.uk/cgi-bin/resources.cgi?url=https://www.proquest.com/scholarly-journals/neuronal-message-passing-using-mean-field-bethe/docview/2179737260/se-2?accountid=7408.  Copyright - This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
  * Parr et al. (2020) Thomas Parr, Lancelot Da Costa, and Karl Friston.  Markov blankets, information geometry and stochastic thermodynamics.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , 378(2164):20190159, 2020.  doi: 10.1098/rsta.2019.0159.  URL https://royalsocietypublishing.org/doi/abs/10.1098/rsta.2019.0159.
  * Rawlik et al. (2013) Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar.  On stochastic optimal control and reinforcement learning by approximate inference (extended abstract).  In _Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence_ , IJCAI ’13, page 3052–3056. AAAI Press, 2013.  ISBN 9781577356332.
  * Schultz et al. (1997) Wolfram Schultz, Peter Dayan, and P. Read Montague.  A neural substrate of prediction and reward.  _Science_ , 275(5306):1593–1599, 1997.  ISSN 0036-8075.  doi: 10.1126/science.275.5306.1593.  URL https://science.sciencemag.org/content/275/5306/1593.
  * Schwartenbeck et al. (2018) Philipp Schwartenbeck, Johannes Passecker, Tobias U Hauser, Thomas H B FitzGerald, Martin Kronbichler, and Karl Friston.  Computational mechanisms of curiosity and goal-directed exploration.  _bioRxiv_ , 2018.  doi: 10.1101/411272.  URL https://www.biorxiv.org/content/early/2018/09/07/411272.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.  doi: 10.1038/nature16961.  URL https://doi.org/10.1038/nature16961.
  * Smith et al. (2021) Ryan Smith, Karl J. Friston, and Christopher J. Whyte.  A step-by-step tutorial on active inference and its application to empirical data, 2021.  URL https://psyarxiv.com/b4jm6/.
  * Solopchuk (2018) Oleg Solopchuk.  Tutorial on active inference, 2018.  URL https://medium.com/@solopchuk/tutorial-on-active-inference-30edcf50f5dc.
  * Sundar Rajagopalan et al. (2013) Shyam Sundar Rajagopalan, Abhinav Dhall, and Roland Goecke.  Self-stimulatory behaviours in the wild for autism diagnosis.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops_ , June 2013.
  * Tschantz et al. (2020) A. Tschantz, M. Baltieri, A. K. Seth, and C. L. Buckley.  Scaling active inference.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8, 2020.  doi: 10.1109/IJCNN48605.2020.9207382.
  * Ueltzhöffer (2018) Kai Ueltzhöffer.  Deep active inference.  _Biological Cybernetics_ , 112(6):547–573, Dec 2018.  ISSN 1432-0770.  doi: 10.1007/s00422-018-0785-7.  URL https://doi.org/10.1007/s00422-018-0785-7.
  * van de Laar and de Vries (2019a) Thijs van de Laar and Bert de Vries.  Simulating active inference processes by message passing.  _Front. Robotics and AI_ , 2019, 2019a.  doi: 10.3389/frobt.2019.00020.  URL https://doi.org/10.3389/frobt.2019.00020.
  * van de Laar and de Vries (2019b) Thijs W. van de Laar and Bert de Vries.  Simulating active inference processes by message passing.  _Frontiers in Robotics and AI_ , 6:20, 2019b.  ISSN 2296-9144.  doi: 10.3389/frobt.2019.00020.  URL https://www.frontiersin.org/article/10.3389/frobt.2019.00020.
  * Van de Maele et al. (2021) Toon Van de Maele, Tim Verbelen, Ozan Çatal, Cedric De Boom, and Bart Dhoedt.  Active vision for robot manipulators using the free energy principle.  _Frontiers in Neurorobotics_ , 15:14, 2021.  ISSN 1662-5218.  doi: 10.3389/fnbot.2021.642780.  URL https://www.frontiersin.org/article/10.3389/fnbot.2021.642780.
  * van Hasselt et al. (2015) Hado van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double Q-learning, 2015.
  * Wauthier et al. (2020) Samuel T. Wauthier, Ozan Çatal, Cedric De Boom, Tim Verbelen, and Bart Dhoedt.  Sleep: Model reduction in deep active inference.  In Tim Verbelen, Pablo Lanillos, Christopher L. Buckley, and Cedric De Boom, editors, _Active Inference_ , pages 72–83, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-64919-7.
  * Wiegerinck (2000) Wim Wiegerinck.  Variational approximations between mean field theory and the junction tree algorithm.  In Craig Boutilier and Moisés Goldszmidt, editors, _UAI ’00: Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence, Stanford University, Stanford, California, USA, June 30 - July 3, 2000_ , pages 626–633. Morgan Kaufmann, 2000.  URL https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=73&proceeding_id=16.
  * Winn and Bishop (2005) John Winn and Christopher Bishop.  Variational message passing.  _Journal of Machine Learning Research_ , 6:661–694, 2005\.
  * Xing et al. (2012) Eric P. Xing, Michael I. Jordan, and Stuart J. Russell.  A generalized mean field algorithm for variational inference in exponential families.  _CoRR_ , abs/1212.2512, 2012.  URL http://arxiv.org/abs/1212.2512.
  * Yedidia (2005) J. S. Yedidia.  Constructing free-energy approximations and generalized belief propagation algorithms.  _IEEE Trans. Information Theory_ , 51(7):2282–2312, 2005.  doi: 10.1109/TIT.2005.850085.  URL https://ci.nii.ac.jp/naid/30019661350/en/.
  * Yedidia (2011) Jonathan S. Yedidia.  Message-passing algorithms for inference and optimization.  _Journal of Statistical Physics_ , 145(4):860–890, Nov 2011.  ISSN 1572-9613.  doi: 10.1007/s10955-011-0384-7.  URL https://doi.org/10.1007/s10955-011-0384-7.
  * Yedidia et al. (2000) Jonathan S. Yedidia, William T. Freeman, and Yair Weiss.  Generalized belief propagation.  In _Proceedings of the 13th International Conference on Neural Information Processing Systems_ , NIPS’00, page 668–674, Cambridge, MA, USA, 2000\. MIT Press.