# Reinforcement Learning through
Active Inference

Alexander Tschantz
Sackler Centre for Consciousness Science
Evolutionary & Adaptive Systems Research Group
University of Sussex
Brighton, UK
<EMAIL_ADDRESS>
&Beren Millidge
University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>
&Anil K. Seth
Sackler Centre for Consciousness Science
Evolutionary & Adaptive Systems Research Group
University of Sussex
Brighton, UK
Canadian Institute for Advanced Research
&Christopher L. Buckley
Evolutionary & Adaptive Systems Research Group
University of Sussex
Brighton, UK

###### Abstract

The central tenet of reinforcement learning (RL) is that agents seek to
maximize the sum of cumulative rewards. In contrast, active inference, an
emerging framework within cognitive and computational neuroscience, proposes
that agents act to maximize the evidence for a biased generative model. Here,
we illustrate how ideas from active inference can augment traditional RL
approaches by (i) furnishing an inherent balance of exploration and
exploitation, and (ii) providing a more flexible conceptualization of reward.
Inspired by active inference, we develop and implement a novel objective for
decision making, which we term the free energy of the expected future. We
demonstrate that the resulting algorithm successfully balances exploration and
exploitation, simultaneously achieving robust performance on several
challenging RL benchmarks with sparse, well-shaped, and no rewards.

##  1 Introduction

Both biological and artificial agents must learn to make adaptive decisions in
unknown environments. In the field of reinforcement learning (RL), agents aim
to learn a policy that maximises the sum of expected rewards (Sutton et al.,
1998). This approach has demonstrated impressive results in domains such as
simulated games (Mnih et al., 2015; Silver et al., 2017), robotics (Polydoros
& Nalpantidis, 2017; Nagabandi et al., 2019) and industrial applications
(Meyes et al., 2017).

In contrast, active inference (Friston et al., 2016; 2015; 2012; 2009) \- an
emerging framework from cognitive and computational neuroscience - suggests
that agents select actions in order to maximise the evidence for a model that
is biased towards an agent’s preferences. This framework extends influential
theories of Bayesian perception and learning (Knill & Pouget, 2004; L
Griffiths et al., 2008) to incorporate probabilistic decision making, and
comes equipped with a biologically plausible process theory (Friston et al.,
2017a) that enjoys considerable empirical support (Friston & Kiebel, 2009).

Although active inference and RL have their roots in different disciplines,
both frameworks have converged upon similar solutions to the problem of
learning adaptive behaviour. For instance, both frameworks highlight the
importance of learning probabilistic models, performing inference and
efficient planning. This leads to a natural question: can insights from active
inference inform the development of novel RL algorithms?

Conceptually, there are several ways in which active inference can inform and
potentially enhance the field of RL. First, active inference suggests that
agents embody a generative model of their preferred environment and seek to
maximise the evidence for this model. In this context, rewards are cast as
prior probabilities over observations, and success is measured in terms of the
divergence between preferred and expected outcomes. Formulating preferences as
prior probabilities enables greater flexibility when specifying an agent’s
goals (Friston et al., 2012; Friston, 2019a), provides a principled (i.e.
Bayesian) method for learning preferences (Sajid et al., 2019), and is
consistent with recent neurophysiological data demonstrating the
distributional nature of reward representations (Dabney et al., 2020). Second,
reformulating reward maximisation as maximizing model evidence naturally
encompasses both exploration and exploitation under a single objective,
obviating the need for adding ad-hoc exploratory terms to existing objectives.
Moreover, as we will show, active inference subsumes a number of established
RL formalisms, indicating a potentially unified framework for adaptive
decision-making under uncertainty.

Translating these conceptual insights into practical benefits for RL has
proven challenging. Current implementations of active inference have generally
been confined to discrete state spaces and toy problems (Friston et al., 2015;
2017b; 2017c) (although see (Tschantz et al., 2019a; Millidge, 2019; Catal et
al., 2019)). Therefore, it has not yet been possible to evaluate the
effectiveness of active inference in challenging environments; as a result,
active inference has not yet been widely taken up within the RL community.

In this paper, we consider active inference in the context of decision
making111A full treatment of active inference would consider inference and
learning, see (Buckley et al., 2017) for an overview.. We propose and
implement a novel objective function for active inference - the free energy of
the expected future \- and show that this quantity provides a tractable bound
on established RL objectives. We evaluate the performance of this algorithm on
a selection of challenging continuous control tasks. We show strong
performance on environments with sparse, well-shaped, and no rewards,
demonstrating our algorithm’s ability to effectively balance exploration and
exploitation. Altogether, our results indicate that active inference provides
a promising complement to current RL methods.

##  2 Active Inference

Both active inference and RL can be formulated in the context of a partially
observed Markov decision process POMDPs (Murphy, 1982). At each time step $t$,
the true state of the environment $\mathbf{s}_{t}$ evolves according to the
stochastic transition dynamics
$\mathbf{{\textnormal{s}}}_{t}\sim\mathrm{p}(\mathbf{s}_{t}|\mathbf{s}_{t-1},\mathbf{a}_{t-1})$,
where $\mathbf{a}\in\mathbb{R}^{d_{a}}$ denotes an agent’s actions. Agents do
not necessarily have access to the true state of the environment, but may
instead receive observations ${\textnormal{o}}_{t}\in\mathbb{R}^{d_{o}}$,
which are generated according to
${\textnormal{o}}_{t}\sim\mathrm{p}({\textnormal{o}}_{t}|\mathbf{s}_{t})$. In
this case, agents must operate on beliefs
${\textnormal{s}}_{t}\in\mathbb{R}^{d_{s}}$ about the true state of the
environment $\mathbf{s}_{t}$. Finally, the environment generates rewards
${\textnormal{r}}_{t}$ according to
${\textnormal{r}}_{t}\sim\mathrm{p}({\textnormal{r}}_{t}|\mathbf{s}_{t})$222We
use $\mathbf{x}$ and $\mathrm{p}(\cdot)$ to denote the generative process and
x and $p(\cdot)$ to denote the agent’s generative model..

The goal of RL is to learn a policy that maximises the expected sum of rewards
$\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}{\textnormal{r}}_{t}]$ (Sutton et
al., 1998). In contrast, the goal of active inference is to maximise the
Bayesian model evidence for an agent’s generative model
$p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)$, where $\theta\in\Theta$
denote model parameters.

Crucially, active inference allows that an agent’s generative model can be
biased towards favourable states of affairs (Friston, 2019b). In other words,
the model assigns probability to the parts of observation space that are both
likely and beneficial for an agent’s success. We use the notation
$p^{\Phi}(\cdot)$ to represent an arbitrary distribution encoding the agent’s
preferences.

Given a generative model, agents can perform approximate Bayesian inference by
encoding an arbitrary distribution $q({\textnormal{s}},\theta)$ and minimising
variational free energy
$\mathcal{F}=D_{\mathrm{KL}}\Big{(}q({\textnormal{s}},\theta)\|p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)\Big{)}$.
When observations o are known, $\mathcal{F}$ can be minimized through standard
variational methods (Bishop, 2006; Buckley et al., 2017), causing
$q({\textnormal{s}},\theta)$ to tend towards the true posterior
$p({\textnormal{s}},\theta|{\textnormal{o}})$. Note that treating model
parameters $\theta$ as random variables casts learning as a process of
inference (Blundell et al., 2015).

In the current context, agents additionally maintain beliefs over policies
$\pi=\\{a_{0},...,a_{T}\\}$, which are themselves random variables. Policy
selection is then implemented by identifying $q(\pi)$ that minimizes
$\mathcal{F}$, thus casting policy selection as a process of approximate
inference (Friston et al., 2015). While the standard free energy functional
$\mathcal{F}$ is generally defined for a single time point $t$, $\pi$ refers
to a temporal sequence of variables. Therefore, we augment the free energy
functional $\mathcal{F}$ to encompass future variables, leading to the free
energy of the expected future $\mathcal{\tilde{F}}$. This quantity measures
the KL-divergence between a sequence of beliefs about future variables and an
agent’s biased generative model.

The goal is now to infer $q(\pi)$ in order to minimise $\mathcal{\tilde{F}}$.
We demonstrate that the resulting scheme naturally encompasses both
exploration and exploitation, thereby suggesting a deep relationship between
inference, learning and decision making.

##  3 Free energy of the expected future

Let ${\textnormal{x}}_{t:T}$ denote a sequence of variables through time,
${\textnormal{x}}_{t:T}=\\{{\textnormal{x}}_{t},...,{\textnormal{x}}_{T}\\}$.
We wish to minimize the free energy of the expected future
$\mathcal{\tilde{F}}$, which is defined as:

$\mathcal{\tilde{F}}=D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}_{0:T},{\textnormal{s}}_{0:T},\theta,\pi)\|p^{\Phi}({\textnormal{o}}_{0:T},{\textnormal{s}}_{0:T},\theta)\Big{)}$
(1)

where $q({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta,\pi)$ represents
an agent’s beliefs about future variables, and
$p^{\Phi}({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta)$ represents an
agent’s biased generative model. Note that the beliefs about future variables
include beliefs about future observations, ${\textnormal{o}}_{t:T}$, which are
unknown and thus treated as random variables333For readers familiar with the
active inference framework, we highlight that the free energy of the expected
future differs from expected free energy (Friston et al., 2015). We leave a
discussion of the relative merits to future work..

In order to find $q(\pi)$ which minimizes $\mathcal{\tilde{F}}$ we note that
(see Appendix C):

$\displaystyle\mathcal{\tilde{F}}=0\Rightarrow D_{\mathrm{KL}}\Big{(}q(\pi)\
\|\big{(}-e^{-\mathcal{\tilde{F_{\pi}}}}\big{)}\Big{)}=0$ (2)

where

$\displaystyle\mathcal{\tilde{F_{\pi}}}=D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}_{0:T},{\textnormal{s}}_{0:T},\theta|\pi)\
\|\ p^{\Phi}({\textnormal{o}}_{0:T},{\textnormal{s}}_{0:T},\theta)\Big{)}$ (3)

Thus, the free energy of the expected future is minimized when
$q(\pi)=\sigma(-\mathcal{\tilde{F_{\pi}}})$, or in other words, policies are
more likely when they minimise $\mathcal{\tilde{F_{\pi}}}$.

###  3.1 Exploration & exploitation

In order to provide an intuition for what minimizing
$\mathcal{\tilde{F_{\pi}}}$ entails, we factorize the agent’s generative
models as
$p^{\Phi}({\textnormal{o}}_{0:T},{\textnormal{s}}_{0:T},\theta)=p({\textnormal{s}}_{0:T},\theta|{\textnormal{o}}_{0:T})p^{\Phi}({\textnormal{o}}_{0:T})$,
implying that the model is only biased in its beliefs over observations. To
retain consistency with RL nomenclature, we treat ‘rewards’ r as a separate
observation modality, such that $p^{\Phi}({\textnormal{o}}_{t:T})$ specifies a
distribution over preferred rewards. We describe our implementation of
$p^{\Phi}({\textnormal{o}}_{t:T})$ in Appendix E. In a similar fashion,
$q({\textnormal{o}}_{t:T}|{\textnormal{s}}_{t:T},\theta,\pi)$ specifies
beliefs about future rewards, given a policy.

Given this factorization, it is straightforward to show that
$-\mathcal{\tilde{F_{\pi}}}$ decomposes into an expected information gain term
and an extrinsic term (see Appendix B)444The approximation in Eq. 4 arises
from the approximation
$q({\textnormal{s}}_{0:T},\theta|{\textnormal{o}}_{0:T},\pi)\approx
p({\textnormal{s}}_{0:T},\theta|{\textnormal{o}}_{0:T},\pi)$, which is
justifiable given that $q(\cdot)$ represents a variational approximation of
the true posterior (Friston et al., 2017a).:

$\displaystyle-\mathcal{\tilde{F_{\pi}}}\approx$
$\displaystyle-\underbrace{\mathbb{E}_{q({\textnormal{o}}_{0:T}|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{s}}_{0:T},\theta|{\textnormal{o}}_{0:T},\pi)\|q({\textnormal{s}}_{0:T},\theta|\pi)\Big{)}\Big{]}}_{\text{Expected
information gain}}$ (4)
$\displaystyle+\underbrace{\mathbb{E}_{q({\textnormal{s}}_{0:T},\theta|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}_{0:T}|{\textnormal{s}}_{0:T},\theta,\pi)\|p^{\Phi}({\textnormal{o}}_{t:T})\Big{)}\Big{]}}_{\text{Extrinsic
term}}$

Maximizing Eq.4 has two functional consequences. First, it maximises the
expected information gain, which quantifies the amount of information an agent
expects to gain from executing some policy. As agents maintain beliefs about
the state of the environment and model parameters, this term promotes
exploration in both state and parameter space.

Second, it minimizes the extrinsic term - which is the KL-divergence between
an agent’s (policy-conditioned) beliefs about future observations and their
preferred observations. In the current context, it measures the KL-divergence
between the rewards an agent expects from a policy and the rewards an agent
desires. In summary, selecting policies to minimise $\tilde{\mathcal{F}}$
invokes a natural balance between exploration and exploitation.

###  3.2 Relationship to probabilistic RL

In recent years, there have been several attempts to formalize RL in terms of
probabilistic inference (Levine, 2018), such as KL-control (Rawlik, 2013),
control-as-inference (Kappen et al., 2012), and state-marginal matching (Lee
et al., 2019). In many of these approaches, the RL objective is broadly
conceptualized as minimising
$D_{\mathrm{KL}}\Big{(}p({\textnormal{o}}_{0:T}|\pi)\|\
p^{\Phi}(o_{0:T})\Big{)}$555We acknowledge that not all objectives follow this
exact formulation.. In Appendix D, we demonstrate that the free energy of the
expected future $\mathcal{\tilde{F}}$ provides a tractable bound on this
objective:

$\displaystyle\mathcal{\tilde{F}}\geq
D_{\mathrm{KL}}\Big{(}p({\textnormal{o}}_{t:T}|\pi)\|\
p^{\Phi}(o_{t:T})\Big{)}$ (5)

These results suggest a deep homology between active inference and existing
approaches to probabilistic RL.

##  4 Implementation

In this section, we describe an efficient implementation of the proposed
objective function in the context of model-based RL. To select actions, we
optimise $q(\pi)$ at each time step, and execute the first action specified by
the most likely policy. This requires (i) a method for evaluating beliefs
about future variables
$q({\textnormal{s}}_{t:T},{\textnormal{o}}_{t:T},\theta|\pi)$, (ii) an
efficient method for evaluating $\mathcal{F}_{\pi}$, and (iii) a method for
optimising $q(\pi)$ such that $q(\pi)=\sigma(-\mathcal{F}_{\pi})$

#### Evaluating beliefs about the future

We factorize and evaluate the beliefs about the future as:

$\displaystyle q({\textnormal{s}}_{t:T},{\textnormal{o}}_{t:T},\theta|\pi)$
$\displaystyle=q(\theta)\prod_{t=\tau}^{T}q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi)q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)$
(6) $\displaystyle
q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi)$
$\displaystyle=\mathbb{E}_{q({\textnormal{s}}_{\tau}|\theta,\pi)}\big{[}p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau})\big{]}$
$\displaystyle
q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)$
$\displaystyle=\mathbb{E}_{q({\textnormal{s}}_{\tau-1}|\theta,\pi)}\big{[}p({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)\big{]}$

where we have here factorized the generative model as
$p({\textnormal{o}}_{\tau},{\textnormal{s}}_{\tau},\theta|\pi)=p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\pi)p({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)p(\theta)$.
We describe the implementation and learning of the likelihood
$p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\pi)$, transition model
$p({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)$ and
parameter prior $p(\theta)$ in Appendix E.

#### Evaluating $\mathcal{\tilde{F}_{\pi}}$

Note that
$-\mathcal{\tilde{F}}_{\pi}=\sum_{\tau=t}^{t+H}-\mathcal{\tilde{F}}_{\pi_{\tau}}$,
where $H$ is the planning horizon. Given beliefs about future variables, the
free energy of the expected future for a single time point can be efficiently
computed as (see Appendix G):

$\displaystyle-\mathcal{\tilde{F}}_{\pi_{\tau}}$ $\displaystyle\approx
E_{q({\textnormal{s}}_{\tau},\theta|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi)\|p^{\Phi}({\textnormal{o}}_{\tau})\Big{)}\Big{]}$
(7)
$\displaystyle+\underbrace{\mathbf{H}[q({\textnormal{o}}_{\tau}|\pi)]-\mathbb{E}_{q({\textnormal{s}}_{\tau}|\pi)}\Big{[}\mathbf{H}[q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\pi)]\Big{]}}_{\text{State
information gain}}$
$\displaystyle+\underbrace{\mathbf{H}[q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)]-\mathbb{E}_{q(\theta)}\Big{[}\mathbf{H}[q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\pi,\theta)]\Big{]}}_{\text{Parameter
information gain}}$

In the current paper, agents observe the true state of the environment
${\textnormal{s}}_{t}$, such that the only partial observability is in rewards
${\textnormal{r}}_{t}$. As as a result, the second term of equation 7 is
redundant, as there is no uncertainty about states. The first (extrinsic) term
can be calculated analytically (see Appendix E). We describe our approximation
of the final term (parameter information gain) in Appendix G.

#### Optimising the policy distribution

We choose to parametrize $q(\pi)$ as a diagonal Gaussian. We use the CEM
algorithm (Rubinstein, 1997) to optimise the parameters of $q(\pi)$ such that
$q(\pi)\propto-\mathcal{F}_{\pi}$. While this solution will fail to capture
the exact shape of $-\mathcal{F}_{\pi}$, agents need only identify the peak of
the landscape to enact the optimal policy.

The full algorithm for inferring $q(\pi)$ is provided in Algorithm 1.

Input: Planning horizon $H$ — Optimisation iterations $I$ — Number of
candidate policies $J$ — Current state ${\textnormal{s}}_{t}$ — Likelihood
$p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau})$ — Transition distribution
$p({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi)$ — Parameter
distribution $P(\theta)$ — Global prior $p^{\Phi}({\textnormal{o}}_{\tau})$
Initialize factorized belief over action sequences
$q(\pi)\leftarrow\mathcal{N}(0,\mathbb{I})$.

for  _$\mathrm{optimisation\ iteration}\ i=1...I$_ do

Sample $J$ candidate policies from $q(\pi)$

for  _$\mathrm{candidate\ policy}\ j=1...J$_ do

$\pi^{(j)}\sim q(\pi)$

$-\mathcal{\tilde{F}}_{\pi}^{j}=0$

for  _$\tau=t...t+H$_ do

$q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi^{(j)})=\mathbb{E}_{q({\textnormal{s}}_{\tau-1}|\theta,\pi^{(j)})}\big{[}p({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi^{(j)})\big{]}$

$q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi^{(j)})=\mathbb{E}_{q({\textnormal{s}}_{\tau}|\theta,\pi^{(j)})}\big{[}p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau})\big{]}$

$-\mathcal{\tilde{F}}_{\pi}^{j}\leftarrow-\mathcal{\tilde{F}}_{\pi}^{j}+E_{q({\textnormal{s}}_{\tau},\theta|\pi^{(j)})}\big{[}D_{\mathrm{KL}}\big{(}q({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi^{(j)})\|p^{\Phi}({\textnormal{o}}_{\tau})\big{)}\big{]}+\mathbf{H}[q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\theta,\pi^{(j)})]-\mathbb{E}_{q(\theta)}\big{[}\mathbf{H}[q({\textnormal{s}}_{\tau}|{\textnormal{s}}_{\tau-1},\pi^{(j)},\theta)]\big{]}$

end for

end for

$q(\pi)\leftarrow\mathrm{refit}(-\mathcal{\tilde{F}}_{\pi}^{j})$

end for

return $q(\pi)$

Algorithm 1 Inference of $q(\pi)$

##  5 Experiments

To determine whether our algorithm successfully balances exploration and
exploitation, we investigate its performance in domains with (i) well-shaped
rewards, (ii) extremely sparse rewards and (iii) a complete absence of
rewards. We use four tasks in total. For sparse rewards, we use the Mountain
Car and Cup Catch environments, where agents only receive reward when the goal
is achieved. For well-shaped rewards, we use the challenging Half Cheetah
environment, using both the running and flipping tasks. For domains without
reward, we use the Ant Maze environment, where there are no rewards and
success is measured by the percent of the maze covered (see Appendix H for
details on all environments).

Figure 1: (A) Mountain Car: Average return after each episode on the sparse-
reward Mountain Car task. Our algorithm achieves optimal performance in a
single trial. (B) Cup Catch: Average return after each episode on the sparse-
reward Cup Catch task. Here, results amongst algorithms are similar, with all
agents reaching asymptotic performance in around 20 episodes. (C & D) Half
Cheetah: Average return after each episode on the well-shaped Half Cheetah
environment, for the running and flipping tasks, respectively. We compare our
results to the average performance of SAC after 100 episodes learning,
demonstrating our algorithm can perform successfully in environments which do
not require directed exploration. Each line is the mean of 5 seeds and filled
regions show +/- standard deviation.

For environments with sparse rewards, we compare our algorithm to two
baselines, (i) a reward algorithm which only selects policies based on the
extrinsic term (i.e. ignores the parameter information gain), and (ii) a
variance algorithm that seeks out uncertain transitions by acting to maximise
the output variance of the transition model (see Appendix E). Note that the
variance agent is also augmented with the extrinsic term to enable comparison.
For environments with well-shaped rewards, we compare our algorithm to the
maximum reward obtained by a state-of-the-art model-free RL algorithm after
100 episodes, the soft-actor-critic (SAC) Haarnoja et al. (2018), which
encourages exploration by seeking to maximise the entropy of the policy
distribution. Finally, for environments without rewards, we compare our
algorithm to a random baseline, which conducts actions at random.

The Mountain Car experiment is shown in Fig. 1A, where we plot the total
reward obtained for each episode over 25 episodes, where each episode is at
most 200 time steps. These results demonstrate that our algorithm rapidly
explores and consistently reaches the goal, achieving optimal performance in a
single trial. In contrast, the benchmark algorithms were, on average, unable
to successfully explore and achieve good performance. We qualitatively confirm
this result by plotting the state space coverage with and without exploration
(Fig. 2B). Our algorithm performs comparably to benchmarks on the Cup Catch
environment (Fig. 1B). We hypothesize that this is because, while the reward
structure is technically sparse, it is simple enough to reach the goal with
random actions, and thus the directed exploration afforded by our method
provides little benefit.

Figure 1 C&D shows that our algorithm performs substantially better than a
state of the art model-free algorithm after 100 episodes on the challenging
Half Cheetah tasks. Our algorithm thus demonstrates robust performance in
environments with well-shaped rewards and provides considerable improvements
in sample-efficiency, relative to SAC.

Finally, we demonstrate that our algorithm can perform well in environments
with no rewards, where the only goal is exploration. Figure 2B shows that our
algorithms rate of exploration is substantially higher than that of a random
baseline in the ant-maze environment, resulting in a more substantial portion
of the maze being covered. This result demonstrates that the directed
exploration afforded by minimising the free energy of the expected future
proves beneficial in environments with no reward structure.

Taken together, these results show that our proposed algorithm - which
naturally balances exploration and exploitation - can successfully master
challenging domains with a variety of reward structures.

Figure 2: (A & B) Mountain Car state space coverage: We plot the points in
state-space visited by two agents - one that minimizes the free energy of the
expected future (FEEF) and one that maximises reward. The plots are from 20
episodes and show that the FEEF agent searches almost the entirety of state
space, while the reward agent is confined to a region that be reached with
random actions. (C) Ant Maze Coverage: We plot the percentage of the maze
covered after 35 episodes, comparing the FEEF agent to an agent acting
randomly. These results are the average of 4 seeds.

##  6 Discussion

Despite originating from different intellectual traditions, active inference
and RL both address fundamental questions about adaptive decision-making in
unknown environments. Exploiting this conceptual overlap, we have applied an
active inference perspective to the reward maximization objective of RL,
recasting it as minimizing the divergence between desired and expected
futures. We derived a novel objective that naturally balances exploration and
exploitation and instantiated this objective within a model-based RL context.
Our algorithm exhibits robust performance and flexibility in a variety of
environments known to be challenging for RL. Moreover, we have shown that our
algorithm applies to a diverse set of reward structures. Conversely, by
implementing active inference using tools from RL, such as amortising
inference with neural networks, deep ensembles and sophisticated algorithms
for planning (CEM), we have demonstrated that active inference can scale to
high dimensional tasks with continuous state and action spaces.

While our results have highlighted the existing overlap between active
inference and RL, we end by reiterating two aspects of active inference that
may be of utility for RL. First, representing preferences as a distribution
over observations allows for greater flexibility in modelling and learning
non-scalar and non-monotonic reward functions. This may prove beneficial when
learning naturalistic tasks in complex nonstationary environments. Second, the
fact that both intrinsic and extrinsic value are complementary components of a
single objective - the free energy of the expected future - may suggest new
paths to tackling the exploration-exploitation dilemma. Our method also admits
promising directions for future work. These include investigating the effects
of different distributions over reward, extending the approach to models which
are hierarchical in time and space (Friston et al., 2018; Pezzulo et al.,
2018), and investigating the deep connections to alternative formulations of
probabilistic control.

## Acknowledgements

AT is funded by a PhD studentship from the Dr. Mortimer and Theresa Sackler
Foundation and the School of Engineering and Informatics at the University of
Sussex. BM is supported by an EPSRC funded PhDS Studentship. CLB is supported
by BBRSC grant number BB/P022197/1. AT and AKS are grateful to the Dr.
Mortimer and Theresa Sackler Foundation, which supports the Sackler Centre for
Consciousness Science. AKS is additionally grateful to the Canadian Institute
for Advanced Research (Azrieli Programme on Brain, Mind, and Consciousness).

## Author Contributions

A.T, B.M and C.L.B contributed to the conceptualization of this work. A.T and
B.M contributed to the coding and generation of experimental results. A.T,
B.M, C.L.B, A.K.S contributed to the writing of the manuscript.

## References

  * Bellemare et al. (2016) Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.  Unifying count-based exploration and intrinsic motivation.  2016\.  URL http://arxiv.org/abs/1606.01868.
  * Bishop (2006) Christopher M Bishop.  _Pattern recognition and machine learning_.  springer, 2006.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural networks.  2015\.  URL http://arxiv.org/abs/1505.05424.
  * Buckley et al. (2017) Christopher L Buckley, Chang Sub Kim, Simon McGregor, and Anil K Seth.  The free energy principle for action and perception: A mathematical review.  _Journal of Mathematical Psychology_ , 81:55–79, 2017.
  * Catal et al. (2019) Ozan Catal, Johannes Nauta, Tim Verbelen, Pieter Simoens, and Bart Dhoedt.  Bayesian policy selection using active inference.  2019\.  URL http://arxiv.org/abs/1904.08149.
  * Chatzilygeroudis et al. (2018) Konstantinos Chatzilygeroudis, Vassilis Vassiliades, Freek Stulp, Sylvain Calinon, and Jean-Baptiste Mouret.  A survey on policy search algorithms for learning robot controllers in a handful of trials.  2018\.  URL http://arxiv.org/abs/1807.02303.
  * Chentanez et al. (2005) Nuttapong Chentanez, Andrew G. Barto, and Satinder P. Singh.  Intrinsically motivated reinforcement learning.  In L. K. Saul, Y. Weiss, and L. Bottou (eds.), _Advances in Neural Information Processing Systems 17_ , pp. 1281–1288. MIT Press, 2005\.  URL http://papers.nips.cc/paper/2552-intrinsically-motivated-reinforcement-learning.pdf.
  * Chitta et al. (2018) Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski.  Deep probabilistic ensembles: Approximate variational inference through KL regularization.  2018\.  URL http://arxiv.org/abs/1811.02640.
  * Chua et al. (2018a) Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  In _Advances in Neural Information Processing Systems_ , pp. 4754–4765, 2018a.
  * Chua et al. (2018b) Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  2018b.  URL http://arxiv.org/abs/1805.12114.
  * Cullen et al. (2018) Maell Cullen, Ben Davey, Karl J Friston, and Rosalyn J Moran.  Active inference in openai gym: A paradigm for computational investigations into psychiatric illness.  _Biological psychiatry: cognitive neuroscience and neuroimaging_ , 3(9):809–818, 2018.
  * Dabney et al. (2020) Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick.  A distributional code for value in dopamine-based reinforcement learning.  _Nature_ , pp. 1–5, 2020.
  * de Abril & Kanai (2018) Ildefons Magrans de Abril and Ryota Kanai.  A unified strategy for implementing curiosity and empowerment driven reinforcement learning.  _arXiv preprint arXiv:1806.06505_ , 2018.
  * Fort et al. (2019) Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  2019\.  URL http://arxiv.org/abs/1912.02757.
  * Friston (2019a) Karl Friston.  A free energy principle for a particular physics.  _arXiv preprint arXiv:1906.10184_ , 2019a.
  * Friston (2019b) Karl Friston.  A free energy principle for a particular physics.  2019b.  URL https://arxiv.org/abs/1906.10184v1.
  * Friston & Kiebel (2009) Karl Friston and Stefan Kiebel.  Predictive coding under the free-energy principle.  364(1521):1211–1221, 2009.  ISSN 1471-2970.  doi: 10.1098/rstb.2008.0300.
  * Friston et al. (2012) Karl Friston, Spyridon Samothrakis, and Read Montague.  Active inference and agency: optimal control without cost functions.  _Biological cybernetics_ , 106(8-9):523–541, 2012\.
  * Friston et al. (2015) Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Giovanni Pezzulo.  Active inference and epistemic value.  6(4):187–214, 2015.  ISSN 1758-8936.  doi: 10.1080/17588928.2015.1020053.
  * Friston et al. (2016) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, Giovanni Pezzulo, et al.  Active inference and learning.  _Neuroscience & Biobehavioral Reviews_, 68:862–879, 2016\.
  * Friston et al. (2017a) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo.  Active inference: a process theory.  _Neural computation_ , 29(1):1–49, 2017a.
  * Friston et al. (2017b) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo.  Active inference: A process theory.  29(1):1–49, 2017b.  ISSN 1530-888X.  doi: 10.1162/NECO˙a˙00912.
  * Friston et al. (2009) Karl J Friston, Jean Daunizeau, and Stefan J Kiebel.  Reinforcement learning or active inference?  _PloS one_ , 4(7), 2009.
  * Friston et al. (2017c) Karl J. Friston, Marco Lin, Christopher D. Frith, Giovanni Pezzulo, J. Allan Hobson, and Sasha Ondobaka.  Active inference, curiosity and insight.  29(10):2633–2683, 2017c.  ISSN 0899-7667.  doi: 10.1162/neco˙a˙00999.  URL https://doi.org/10.1162/neco_a_00999.
  * Friston et al. (2018) Karl J. Friston, Richard Rosch, Thomas Parr, Cathy Price, and Howard Bowman.  Deep temporal models and active inference.  90:486–501, 2018.  ISSN 0149-7634.  doi: 10.1016/j.neubiorev.2018.04.004.  URL http://www.sciencedirect.com/science/article/pii/S0149763418302525.
  * Ha & Schmidhuber (2018) David Ha and Jürgen Schmidhuber.  Recurrent world models facilitate policy evolution.  In _Advances in Neural Information Processing Systems_ , pp. 2450–2462, 2018.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018.
  * Hafner et al. (2018) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  _arXiv preprint arXiv:1811.04551_ , 2018.
  * Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019.
  * Houthooft et al. (2016a) Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.  Curiosity-driven exploration in deep reinforcement learning via bayesian neural networks.  _arXiv preprint arXiv:1605.09674_ , 2016a.
  * Houthooft et al. (2016b) Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.  VIME: Variational information maximizing exploration.  2016b.  URL http://arxiv.org/abs/1605.09674.
  * Kaiser et al. (2019) Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski.  Model-based reinforcement learning for atari.  2019\.  URL http://arxiv.org/abs/1903.00374.
  * Kappen et al. (2012) Hilbert J Kappen, Vicenç Gómez, and Manfred Opper.  Optimal control as a graphical model inference problem.  _Machine learning_ , 87(2):159–182, 2012.
  * Kim et al. (2018) Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song.  EMI: Exploration with mutual information.  2018\.  URL http://arxiv.org/abs/1810.01176.
  * Kingma & Welling (2013) Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  2013\.  URL http://arxiv.org/abs/1312.6114.
  * Knill & Pouget (2004) David C Knill and Alexandre Pouget.  The bayesian brain: the role of uncertainty in neural coding and computation.  _TRENDS in Neurosciences_ , 27(12):712–719, 2004\.
  * L Griffiths et al. (2008) Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum.  Bayesian models of cognition.  2008\.
  * Lee et al. (2019) Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.  Efficient exploration via state marginal matching.  _arXiv preprint arXiv:1906.05274_ , 2019.
  * Leibfried et al. (2019) Felix Leibfried, Sergio Pascual-Diaz, and Jordi Grau-Moya.  A unified bellman optimality principle combining reward maximization and empowerment.  In _Advances in Neural Information Processing Systems_ , pp. 7867–7878, 2019.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Lindley (1956) D. V. Lindley.  On a measure of the information provided by an experiment.  27(4):986–1005, 1956.  ISSN 0003-4851, 2168-8990.  doi: 10.1214/aoms/1177728069.  URL https://projecteuclid.org/euclid.aoms/1177728069.
  * Meyes et al. (2017) Richard Meyes, Hasan Tercan, Simon Roggendorf, Thomas Thiele, Christian Büscher, Markus Obdenbusch, Christian Brecher, Sabina Jeschke, and Tobias Meisen.  Motion planning for industrial robots using reinforcement learning.  _Procedia CIRP_ , 63:107–112, 2017.
  * Millidge (2019) Beren Millidge.  Deep active inference as variational policy gradients.  2019\.  URL http://arxiv.org/abs/1907.03876.
  * Mirchev et al. (2018) Atanas Mirchev, Baris Kayalibay, Maximilian Soelch, Patrick van der Smagt, and Justin Bayer.  Approximate bayesian inference in spatial environments.  2018\.  URL http://arxiv.org/abs/1805.07206.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Mohamed & Rezende (2015) Shakir Mohamed and Danilo Jimenez Rezende.  Variational information maximisation for intrinsically motivated reinforcement learning.  2015\.  URL http://arxiv.org/abs/1509.08731.
  * Murphy (1982) KP Murphy.  A survey of pomdp solution techniques: Theory.  _Models, and algorithms, management science_ , 28, 1982.
  * Nagabandi et al. (2018) Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine.  Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 7559–7566. IEEE, 2018.
  * Nagabandi et al. (2019) Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar.  Deep dynamics models for learning dexterous manipulation.  _arXiv preprint arXiv:1909.11652_ , 2019.
  * O’Donoghue et al. (2017) Brendan O’Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih.  The uncertainty bellman equation and exploration.  _arXiv preprint arXiv:1709.05380_ , 2017.
  * Okada & Taniguchi (2019) Masashi Okada and Tadahiro Taniguchi.  Variational inference MPC for bayesian model-based reinforcement learning.  2019\.  URL http://arxiv.org/abs/1907.04202.
  * Oudeyer & Kaplan (2009) Pierre-Yves Oudeyer and Frederic Kaplan.  What is intrinsic motivation? a typology of computational approaches.  _Frontiers in neurorobotics_ , 1:6, 2009.
  * Parr & Friston (2017) Thomas Parr and Karl J Friston.  The active construction of the visual world.  _Neuropsychologia_ , 104:92–101, 2017.
  * Parr et al. (2019) Thomas Parr, Dimitrije Markovic, Stefan J Kiebel, and Karl J Friston.  Neuronal message passing using mean-field, bethe, and marginal approximations.  _Scientific reports_ , 9(1):1–18, 2019.
  * Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 16–17, 2017.
  * Pezzulo et al. (2016) Giovanni Pezzulo, Emilio Cartoni, Francesco Rigoli, Léo Pio-Lopez, and Karl Friston.  Active inference, epistemic value, and vicarious trial and error.  _Learning & Memory_, 23(7):322–338, 2016.
  * Pezzulo et al. (2018) Giovanni Pezzulo, Francesco Rigoli, and Karl J. Friston.  Hierarchical active inference: A theory of motivated control.  _Trends in Cognitive Sciences_ , 22(4):294 – 306, 2018.  ISSN 1364-6613.  doi: https://doi.org/10.1016/j.tics.2018.01.009.  URL http://www.sciencedirect.com/science/article/pii/S1364661318300226.
  * Polydoros & Nalpantidis (2017) Athanasios S Polydoros and Lazaros Nalpantidis.  Survey of model-based reinforcement learning: Applications on robotics.  _Journal of Intelligent & Robotic Systems_, 86(2):153–173, 2017.
  * Rawlik et al. (2013) Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar.  On stochastic optimal control and reinforcement learning by approximate inference.  In _Twenty-Third International Joint Conference on Artificial Intelligence_ , 2013.
  * Rawlik (2013) Konrad Cyrus Rawlik.  On probabilistic inference approaches to stochastic optimal control.  2013\.
  * Rubinstein (1997) Reuven Y Rubinstein.  Optimization of computer simulation models with rare events.  _European Journal of Operational Research_ , 99(1):89–112, 1997.
  * Sajid et al. (2019) Noor Sajid, Philip J Ball, and Karl J Friston.  Demystifying active inference.  _arXiv preprint arXiv:1909.10863_ , 2019.
  * Schmidhuber (1991) Jürgen Schmidhuber.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pp. 222–227, 1991.
  * Schmidhuber (2007) Jürgen Schmidhuber.  Simple algorithmic principles of discovery, subjective beauty, selective attention, curiosity & creativity.  In _International Conference on Discovery Science_ , pp. 26–38. Springer, 2007.
  * Schwartenbeck et al. (2019) Philipp Schwartenbeck, Johannes Passecker, Tobias U Hauser, Thomas HB FitzGerald, Martin Kronbichler, and Karl J Friston.  Computational mechanisms of curiosity and goal-directed exploration.  8:e41703, 2019.  ISSN 2050-084X.  doi: 10.7554/eLife.41703.  URL https://doi.org/10.7554/eLife.41703.
  * Shyam et al. (2018) Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez.  Model-based active exploration.  _arXiv preprint arXiv:1810.12162_ , 2018.
  * Shyam et al. (2019) Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez.  Model-based active exploration.  In _International Conference on Machine Learning_ , pp. 5779–5788, 2019.  URL http://proceedings.mlr.press/v97/shyam19a.html.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  _Nature_ , 550(7676):354–359, 2017.
  * Still & Precup (2012) Susanne Still and Doina Precup.  An information-theoretic approach to curiosity-driven reinforcement learning.  131(3):139–148, 2012.  ISSN 1611-7530.  doi: 10.1007/s12064-011-0142-z.
  * Storck et al. (1995) Jan Storck, Sepp Hochreiter, and Jürgen Schmidhuber.  Reinforcement driven information acquisition in non-deterministic environments.  In _Proceedings of the international conference on artificial neural networks, Paris_ , volume 2, pp. 159–164. Citeseer, 1995.
  * Sun et al. (2011) Yi Sun, Faustino Gomez, and Juergen Schmidhuber.  Planning to be surprised: Optimal bayesian exploration in dynamic environments.  2011\.  URL http://arxiv.org/abs/1103.5708.
  * Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al.  _Introduction to reinforcement learning_ , volume 135.  MIT press Cambridge, 1998.
  * Teigen (2018) Bjørn Ivar Teigen.  An active learning perspective on exploration in reinforcement learning.  2018\.  URL https://www.duo.uio.no/handle/10852/62823.
  * Tschantz et al. (2019a) Alexander Tschantz, Manuel Baltieri, Anil K. Seth, and Christopher L. Buckley.  Scaling active inference.  2019a.  URL http://arxiv.org/abs/1911.10601.
  * Tschantz et al. (2019b) Alexander Tschantz, Anil K. Seth, and Christopher L. Buckley.  Learning action-oriented models through active inference.  pp. 764969, 2019b.  doi: 10.1101/764969.  URL https://www.biorxiv.org/content/10.1101/764969v1.
  * Tschiatschek et al. (2018) Sebastian Tschiatschek, Kai Arulkumaran, Jan Stühmer, and Katja Hofmann.  Variational inference for data-efficient model learning in POMDPs.  2018\.  URL http://arxiv.org/abs/1805.09281.
  * Ueltzhöffer (2018) Kai Ueltzhöffer.  Deep active inference.  112(6):547–573, 2018.  ISSN 0340-1200, 1432-0770.  doi: 10.1007/s00422-018-0785-7.  URL http://arxiv.org/abs/1709.02341.
  * Watter et al. (2015) Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller.  Embed to control: A locally linear latent dynamics model for control from raw images.  2015\.  URL http://arxiv.org/abs/1506.07365.
  * Williams et al. (2016) Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou.  Aggressive driving with model predictive path integral control.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1433–1440. IEEE, 2016.
  * Yarin Gal et al. (2016) Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen.  Improving PILCO with bayesian neural network dynamics models.  In _Data-Efficient Machine Learning workshop_ , 2016.

##  Appendix A Related work

#### Active inference

There is an extensive literature on active inference within discrete state-
spaces, covering a wide variety of tasks, such as epistemic foraging in
saccades (Parr & Friston, 2017; Friston, 2019b; Schwartenbeck et al., 2019),
exploring mazes (Friston et al., 2015; Pezzulo et al., 2016; Friston et al.,
2016), to playing Atari games (Cullen et al., 2018). Active inference also
comes equipped with a well-developed neural process theory (Friston et al.,
2017a; Parr et al., 2019) which can account for a substantial range of neural
dynamics. There have also been prior attempts to scale up active inference to
continuous RL tasks (Tschantz et al., 2019a; Millidge, 2019; Ueltzhöffer,
2018), which we build upon here.

#### Model based RL

Model based reinforcement learning has been in a recent renaissance, with
implementations vastly exceeding the sample efficiency of model-free methods,
while also approaching their asymptotic performance (Ha & Schmidhuber, 2018;
Nagabandi et al., 2018; Chua et al., 2018a; Hafner et al., 2018). There have
been recent successes on challenging domains such as Atari (Kaiser et al.,
2019), and high dimensional robot locomotion (Hafner et al., 2018; 2019) and
manipulation (Nagabandi et al., 2019) tasks. Key advances include variational
autoencoders (Kingma & Welling, 2013) to flexibly construct latent spaces in
partially observed environments, Bayesian approaches such as Bayes by backprop
(Houthooft et al., 2016a), deep ensembles (Shyam et al., 2018; Chua et al.,
2018a), and other variational approaches (Okada & Taniguchi, 2019;
Tschiatschek et al., 2018; Yarin Gal et al., 2016), which quantify uncertainty
in the dynamics models, and enable the model to learn a latent space that is
useful for action (Tschantz et al., 2019b; Watter et al., 2015). Finally,
progress has been aided by powerful planning algorithms capable of online
planning in continuous state and action spaces (Williams et al., 2016;
Rubinstein, 1997).

#### Intrinsic Measures

Using intrinsic measures to encourage exploration has a long history in RL
(Schmidhuber, 1991; 2007; Storck et al., 1995; Oudeyer & Kaplan, 2009;
Chentanez et al., 2005). Recent model-free and model based-intrinsic measures
that have been proposed in the literature include policy-entropy (Rawlik,
2013; Rawlik et al., 2013; Haarnoja et al., 2018),state entropy (Lee et al.,
2019), information-gain (Houthooft et al., 2016b; Okada & Taniguchi, 2019; Kim
et al., 2018; Shyam et al., 2019; Teigen, 2018), prediction error (Pathak et
al., 2017), divergence of ensembles (Shyam et al., 2019; Chua et al., 2018b),
uncertain state bonuses (Bellemare et al., 2016; O’Donoghue et al., 2017), and
empowerment (de Abril & Kanai, 2018; Leibfried et al., 2019; Mohamed &
Rezende, 2015). Information gain additionally has a substantial history
outside the RL framework, going back to (Lindley, 1956; Still & Precup, 2012;
Sun et al., 2011).

##  Appendix B Derivation for the free energy of the expected future

We begin with the full free energy of the expected future and decompose this
into the free energy of the expected future given policies, and the negative
policy entropy:

$\displaystyle\mathcal{\tilde{F}}$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta,\pi)}[\log
q({\textnormal{o}},{\textnormal{s}},\theta,\pi)-\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)]$ (8)
$\displaystyle=\mathbb{E}_{q(\pi)}[\mathcal{\tilde{F}}_{\pi}]-\mathbf{H}[q(\pi)]$

We now show the free energy of the expected future given policies can be
decomposed into extrinsic and information gain terms:

$\displaystyle\mathcal{\tilde{F}}_{\pi}$
$\displaystyle=E_{q({\textnormal{o}},{\textnormal{s}},\theta,\pi)}[\log
q({\textnormal{o}},{\textnormal{s}},\theta,\pi)-\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)]$ (9)
$\displaystyle=\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{s}},\theta|\pi)+\log
q({\textnormal{o}}|{\textnormal{s}},\theta,\pi)-\log
p({\textnormal{s}},\theta|{\textnormal{o}})-\log p^{\Phi}({\textnormal{o}})]$
$\displaystyle\approx\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{s}},\theta|\pi)+\log
q({\textnormal{o}}|{\textnormal{s}},\theta,\pi)-\log
q({\textnormal{s}},\theta|{\textnormal{o}},\theta)-\log
p^{\Phi}({\textnormal{o}})]$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{s}},\theta|\pi)-\log
q({\textnormal{s}},\theta|{\textnormal{o}},\pi)]+\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{o}}|{\textnormal{s}},\theta,\pi)-\log
p^{\Phi}({\textnormal{o}})]$ $\displaystyle-\mathcal{\tilde{F}}_{\pi}$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{s}},\theta|{\textnormal{o}},\pi)-\log
q({\textnormal{s}},\theta|\pi)]+\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
p^{\Phi}({\textnormal{o}})-\log
q({\textnormal{o}}|{\textnormal{s}},\theta,\pi)]$
$\displaystyle=\underbrace{\mathbb{E}_{q({\textnormal{o}}|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{s}},\theta|{\textnormal{o}},\pi)\|q({\textnormal{s}},\theta|\pi)\Big{)}\Big{]}}_{\text{Expected
Information
Gain}}-\underbrace{\mathbb{E}_{q({\textnormal{s}},\theta|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}|{\textnormal{s}},\theta,\pi)\|p^{\Phi}({\textnormal{o}})\Big{)}\Big{]}}_{\text{Extrinsic
Value}}$

Where we have assumed that $p({\textnormal{s}},\theta|{\textnormal{o}})\approx
q({\textnormal{s}},\theta|{\textnormal{o}},\pi)$. We wish to minimize
$\mathcal{\tilde{F}}_{\pi}$, and thus maximize $-\mathcal{\tilde{F}}_{\pi}$.
This means we wish to maximize the information gain and minimize the KL-
divergence between expected and preferred observations.

By noting that $q({\textnormal{s}},\theta|{\textnormal{o}},\pi)\approx
q({\textnormal{s}}|{\textnormal{o}},\pi)q(\theta|{\textnormal{s}})$, we can
split the expected information gain term into state and parameter information
gain terms:

$\displaystyle\mathbb{E}_{q({\textnormal{o}}|\pi)}\Big{[}D_{\mathrm{KL}}\Big{(}q({\textnormal{s}},\theta|{\textnormal{o}},\pi)\|q({\textnormal{s}},\theta|\pi)\Big{)}\Big{]}$
(10)
$\displaystyle=\mathbb{E}_{q({\textnormal{o}}|\pi)q({\textnormal{s}},\theta|{\textnormal{o}},\pi)}\big{[}\log
q({\textnormal{s}},\theta|{\textnormal{o}},\pi)-\log
q({\textnormal{s}},\theta|\pi)\big{]}$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}}|\pi)q({\textnormal{s}},\theta|{\textnormal{o}},\pi)}\big{[}\log
q({\textnormal{s}}|{\textnormal{o}},\pi)+\log q(\theta|{\textnormal{s}})-\log
q({\textnormal{s}}|\theta,\pi)-\log q(\theta)\big{]}$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}}|\pi)q({\textnormal{s}},\theta|{\textnormal{o}},\pi)}\big{[}\log
q({\textnormal{s}}|{\textnormal{o}},\pi)-\log
q({\textnormal{s}}|\theta,\pi)]\big{]}+\mathbb{E}_{q({\textnormal{o}}|\pi)q({\textnormal{s}},\theta|{\textnormal{o}},\pi)}\big{[}\log
q(\theta|{\textnormal{s}})-\log q(\theta)\big{]}$
$\displaystyle=\underbrace{\mathbb{E}_{q({\textnormal{o}}|\pi)q(\theta)}\Big{[}D_{\mathrm{KL}}\big{(}q({\textnormal{s}}|{\textnormal{o}},\pi)\|q({\textnormal{s}}|\theta)\big{)}\Big{]}}_{\text{Expected
State Information
Gain}}+\underbrace{\mathbb{E}_{q({\textnormal{s}}|\theta)}\Big{[}D_{\mathrm{KL}}\big{(}q(\theta|{\textnormal{s}})\|q(\theta)\big{)}\Big{]}}_{\text{Expected
Parameter Information Gain}}$

##  Appendix C Derivation of the optimal policy

We derive the distribution for $q(\pi)$ which minimizes $\mathcal{\tilde{F}}$:

$\displaystyle\mathcal{\tilde{F}}$
$\displaystyle=D_{\mathrm{KL}}\Big{(}q({\textnormal{o}},{\textnormal{s}},\theta,\pi)\|p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)\Big{)}$
(11)
$\displaystyle=\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta,\pi)}[\log
q({\textnormal{o}},{\textnormal{s}},\theta|\pi)+\log q(\pi)-\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta,\pi)]$
$\displaystyle=\mathbb{E}_{q(\pi)}\Big{[}\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q(\pi)-[\log p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)-\log
q({\textnormal{o}},{\textnormal{s}},\theta|\pi)]\Big{]}$
$\displaystyle=\mathbb{E}_{q(\pi)}\Big{[}\log
q(\pi)-\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)-\log
q({\textnormal{o}},{\textnormal{s}},\theta|\pi)]\Big{]}$
$\displaystyle=\mathbb{E}_{q(\pi)}\Big{[}\log
q(\pi)-\big{[}-\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{o}},{\textnormal{s}},\theta|\pi)-\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)]\big{]}\Big{]}$
$\displaystyle=\mathbb{E}_{q(\pi)}\Big{[}\log q(\pi)-\log
e^{-\big{[}-\mathbb{E}_{q({\textnormal{o}},{\textnormal{s}},\theta|\pi)}[\log
q({\textnormal{o}},{\textnormal{s}},\theta|\pi)-\log
p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)]\big{]}}\Big{]}$
$\displaystyle=\mathbb{E}_{q(\pi)}\Big{[}\log q(\pi)-\log
e^{-D_{\mathrm{KL}}\big{(}q({\textnormal{o}},{\textnormal{s}},\theta|\pi)\|p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)\big{)}}\Big{]}$
$\displaystyle=D_{\mathrm{KL}}\Big{(}q(\pi)\|e^{-D_{\mathrm{KL}}\big{(}q({\textnormal{o}},{\textnormal{s}},\theta|\pi)\|p^{\Phi}({\textnormal{o}},{\textnormal{s}},\theta)\big{)}}\Big{)}$
$\displaystyle=D_{\mathrm{KL}}\Big{(}q(\pi)\
\|e^{-\mathcal{\tilde{F_{\pi}}}}\Big{)}$

##  Appendix D Derivation of RL bound

Here we show that the free energy of the expected future is a bound on the
divergence between expected and desired observations. The proof proceeds
straightforwardly by importance sampling on the approximate posterior and then
applying Jensen’s inequality:

$\displaystyle D_{\mathrm{KL}}\big{(}q({\textnormal{o}}_{t:T}|\pi)\|\
p^{\Phi}(o_{t:T})\big{)}$
$\displaystyle=\mathbb{E}_{q({\textnormal{o}}_{t:T}|\pi)}\big{[}\log
q({\textnormal{o}}_{t:T}|\pi)-\log p^{\Phi}({\textnormal{o}})\big{]}$ (12)
$\displaystyle=\mathbb{E}_{q({\textnormal{o}}_{t:T}|\pi)}\bigg{[}\log\big{(}\int
dx_{1:T}\int
d\theta_{1:T}\frac{q({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta_{t:T}|\pi)q({\textnormal{s}}_{t:T},\theta_{t:T}|{\textnormal{o}}_{t:T})}{p^{\Phi}({\textnormal{o}}_{t:T})q({\textnormal{s}}_{t:T},\theta_{t:T}|{\textnormal{o}}_{t:T})}\big{)}\bigg{]}$
$\displaystyle\leq\mathbb{E}_{q({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta_{t:T}|\pi)}\Big{[}\log\big{(}\frac{q({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta_{t:T}|\pi)}{p^{\Phi}({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta_{t:T})}\big{)}\Big{]}$
$\displaystyle\leq
D_{\mathrm{KL}}\Big{(}q({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta|\pi)\|p^{\Phi}({\textnormal{o}}_{t:T},{\textnormal{s}}_{t:T},\theta)\Big{)}=\mathcal{\tilde{F}}$

##  Appendix E Model details

In the current work, we implemented our probabilistic model using an ensemble-
based approach (Chua et al., 2018a; Fort et al., 2019; Chitta et al., 2018).
Here, an ensemble of point-estimate parameters
$\theta=\\{\theta_{0},...,\theta_{B}\\}$ trained on different batches of the
dataset $\mathcal{D}$ are maintained and treated as samples as from the
posterior distribution $p(\theta|\mathcal{D})$. Besides consistency with the
active inference framework, probabilistic models enable the active resolution
of model uncertainty, capture both epistemic and aleatoric uncertainty, and
help avoid over-fitting in low data regimes (Fort et al., 2019; Chitta et al.,
2018; Chatzilygeroudis et al., 2018; Chua et al., 2018b).

This design choice means that we use a trajectory sampling method when
evaluating beliefs about future variables (Chua et al., 2018a), as each pass
through the transition model
$p({\textnormal{s}}_{t}|{\textnormal{s}}_{t-1},\theta,\pi)$ evokes $B$ samples
from ${\textnormal{s}}_{t}$.

#### Transition model

We implement the transition model as
$p({\textnormal{s}}_{t}|{\textnormal{s}}_{t-1},\theta,\pi)$ as
$\mathcal{N}({\textnormal{s}}_{t};f_{\theta}({\textnormal{s}}_{t-1}),f_{\theta}({\textnormal{s}}_{t-1}))$,
where $f_{\theta}(\cdot)$ are a set of function approximators
$f_{\theta}(\cdot)=\\{f_{\theta_{0}}(\cdot),...,f_{\theta_{B}}(\cdot)\\}$. In
the current paper, $f_{\theta_{i}}({\textnormal{s}}_{t-1})$ is a two-layer
feed-forward network with 400 hidden units and swish activation function.
Following previous work, we predict state deltas rather than the next states
(Shyam et al., 2018).

#### Reward model

We implement the reward model as
$p({\textnormal{o}}_{\tau}|{\textnormal{s}}_{\tau},\theta,\pi)=\mathcal{N}({\textnormal{o}}_{\tau};f_{\lambda}({\textnormal{s}}_{\tau}),\mathbf{1})$,
where $f_{\lambda}({\textnormal{s}}_{\tau})$ is some arbitrary function
approximator666Formally, this is an observation model, but we retain RL
terminology for clarity.. In the current paper,
$f_{\lambda}({\textnormal{s}}_{\tau})$ is a two layer feed forward network
with 400 hidden units and ReLU activation function. Learning a reward model
offers several plausible benefits outside of the active inference framework,
as it abolishes the requirement that rewards can be directly calculated from
observations or states (Chua et al., 2018a).

#### Global prior

We implement the global prior $p^{\Phi}({\textnormal{o}})$ as a Gaussian with
unit variance centred around the maximum reward for the respective
environment. We leave it to future work to explore the effects of more
intricate priors.

##  Appendix F Implementation details

For all tasks, we initialize a dataset $\mathcal{D}$ with a single episode of
data collected from a random agent. For each episode, we train the ensemble
transition model and reward model for 100 epochs, using the negative-log
likelihood loss. We found cold-starting training at each episode to lead to
more consistent behaviour. We then let the agent act in the environment based
on Algorithm 1, and append the collected data to the dataset $\mathcal{D}$.

We list the full set of hyperparameters below:

Hyperparameters
---
Hidden layer size |  400
Learning rate |  0.001
Training-epochs |  100
Planning-horizon |  30
N-candidates (CEM) |  700
Top-candidates (CEM) |  70
Optimisation-iterations (CEM) |  7

##  Appendix G Expected information gain

In Eq. 4, expected parameter information gain was presented in the form
$\mathbb{E}_{q({\textnormal{s}}|\theta)}D_{\mathrm{KL}}\big{(}q(\theta|{\textnormal{s}})\|q(\theta)\big{)}$.
While this provides a nice intuition about the effect of the information gain
term on behaviour, it cannot be computed directly, due to the intractability
of identifying true posteriors over parameters. We here show that, through a
simple application of Bayes’ rule, it is straightforward to derive an
equivalent expression for the expected information gain as the divergence
between the state likelihood and marginal, given the parameters, which
decomposes into an entropy of an average minus an average of entropies:

$\displaystyle\mathbb{E}_{q({\textnormal{s}}|\theta)}D_{\mathrm{KL}}\big{(}q(\theta|{\textnormal{s}})\|q(\theta)\big{)}$
(13)
$\displaystyle=\mathbb{E}_{q({\textnormal{s}}|\theta)q(\theta|{\textnormal{s}})}\big{[}\log
q(\theta|{\textnormal{s}})-\log q(\theta)\big{]}$
$\displaystyle=\mathbb{E}_{q({\textnormal{s}},\theta)}\big{[}\log
q({\textnormal{s}}|\theta)+\log q(\theta)-\log q({\textnormal{s}})-\log
q(\theta)\big{]}$
$\displaystyle=\mathbb{E}_{q({\textnormal{s}},\theta)}\big{[}\log
q({\textnormal{s}}|\theta)-\log q({\textnormal{s}})\big{]}$
$\displaystyle=\mathbb{E}_{q(\theta)q({\textnormal{s}}|\theta)}\big{[}\log
q({\textnormal{s}}|\theta)\big{]}-\mathbb{E}_{q(\theta)q({\textnormal{s}}|\theta)}\big{[}\log\mathbb{E}_{q(\theta)}q({\textnormal{s}}|\theta)\big{]}$
$\displaystyle=-\mathbb{E}_{q(\theta)}\mathbf{H}\big{[}q({\textnormal{s}}|\theta)\big{]}+\mathbf{H}\big{[}\mathbb{E}_{q(\theta)}q({\textnormal{s}}|\theta)\big{]}$

The first term is the (negative) average of the entropies. The average over
the parameters $\theta$ is achieved simply by averaging over the dynamics
models in the ensemble. The entropy of the likelihoods
$\mathbf{H}[p({\textnormal{s}}|\theta)]$ can be computed analytically since
each network in the ensemble outputs a Gaussian distribution for which the
entropy is a known analytical result. The second term is the entropy of the
average $\mathbf{H}[\mathbb{E}_{p(\theta)}p({\textnormal{s}}|\theta)]$.
Unfortunately, this term does not have an analytical solution. However, it can
be approximated numerically using a variety of techniques for entropy
estimation. In our paper, we use the nearest neighbour entropy approximation
(Mirchev et al., 2018).

##  Appendix H Environment details

The Mountain Car environment
($\mathcal{S}\subseteq\mathbb{R}^{2}\mathcal{A}\subseteq\mathbb{R}^{1}$)
requires an agent to drive up the side of a hill, where the car is
underactuated requiring it first to gain momentum by driving up the opposing
hill. A reward of one is generated when the agent reaches the goal, and zero
otherwise. The Cup Catch environment
($\mathcal{S}\subseteq\mathbb{R}^{8}\mathcal{A}\subseteq\mathbb{R}^{2}$)
requires the agent to actuate a cup and catch a ball attached to its bottom. A
reward of one is generated when the agent reaches the goal, and zero
otherwise. The Half Cheetah environment
($\mathcal{S}\subseteq\mathbb{R}^{1}7\mathcal{A}\subseteq\mathbb{R}^{6}$)
describes a running planar biped. For the running task, a reward of
$v-0.1||a||^{2}$ is received, where $v$ is the agent’s velocity, and for the
flipping task, a reward of $\epsilon-0.1||a||^{2}$ is received, where
$\epsilon$ is the angular velocity. The Ant Maze environment
($\mathcal{S}\subseteq\mathbb{R}^{2}9\mathcal{A}\subseteq\mathbb{R}^{8}$)
involves a quadruped agent exploring a rectangular maze.