# Differentiable Generalised Predictive Coding

Andre Ofner  Sebastian Stober

This paper deals with differentiable dynamical models congruent with neural
process theories that cast brain function as the hierarchical refinement of an
internal generative model explaining observations. Our work extends existing
implementations of gradient-based predictive coding with automatic
differentiation and allows to integrate deep neural networks for non-linear
state parameterization. Gradient-based predictive coding optimises inferred
states and weights locally in for each layer by optimising precision-weighted
prediction errors that propagate from stimuli towards latent states.
Predictions flow backwards, from latent states towards lower layers. The model
suggested here optimises hierarchical and dynamical predictions of latent
states. Hierarchical predictions encode expected content and hierarchical
structure. Dynamical predictions capture changes in the encoded content along
with higher order derivatives. Hierarchical and dynamical predictions interact
and address different aspects of the same latent states. We apply the model to
various perception and planning tasks on sequential data and show their mutual
dependence. In particular, we demonstrate how learning sampling distances in
parallel address meaningful locations data sampled at discrete time steps. We
discuss possibilities to relax the assumption of linear hierarchies in favor
of more flexible graph structure with emergent properties. We compare the
granular structure of the model with canonical microcircuits describing
predictive coding in biological networks and review the connection to Markov
Blankets as a tool to characterize modularity. A final section sketches out
ideas for efficient perception and planning in nested spatio-temporal
hierarchies. We open source the Torch code for the suggested generalized
predictive coding optimizer, GPC and hope that our work initiates more
research on generalized state-space models in the context of deep neural
networks, reinforcement learning and aspects of biological plausibility.

##  1 Introduction

###  1.1 Generalized Filtering and Dynamical Generative Models

The hierarchical dynamical generative models (HDMs) discussed by Friston and
Kiebel [1] in the context of explaining brain function describe a powerful
class of Bayesian generative models that are hierarchical in structure and
dynamics and allow to perform Bayesian inference on complex observations. HDMs
capture latent states, encoding estimated causes for observations and their
associated motion in generalised coordinates, i.e. the derivatives of each
encoded state’s trajectory. In the context of HDMS, _perception_ is referred
to as the process of inverting a particular model and refining its parameters,
in order to more accurately explain the observations. As we will see later in
this article, planning and _actions_ are closely intertwined with perception
when applying HDMs to sensory data prediction.

Interestingly, HDMs generalize most established statistical models, e.g.
(extended) Kalman filters can be interpreted as a particular first order HDM.
More generally speaking, it is possible to characterize most existing Bayesian
model optimisation schemes as a variant of _Generalized Filtering_ [2].
Generalised filtering describes a generic approach to compute posterior
densities for latent states and model parameters (i.e. addressing inference
and learning) based on a gradient descent on the variational Free Energy (also
known as the Evidence Lower bound or ELBO in the machine learning domain).

Without any additional assumptions about the underlying probability
distributions, the variational Free Energy can be expressed as:

$\displaystyle\mathcal{F}$ $\displaystyle=D_{\mathrm{KL}}[q(x\mid
o;\phi)\|p(o,x;\theta)]$ $\displaystyle=\underbrace{\mathbb{E}_{q(x\mid
o;\phi)}[\ln q(x\mid o;\phi)]}_{\text{Entropy
}}-\underbrace{\mathbb{E}_{q(x\mid o;\phi)}[\ln p(o,x;\theta)]}_{\text{Energy
}}$ (1)

with data $o$, latent state $x$ and a generative model that generates observed
data $p(o,x)=p(o\mid x)p(x)$. In variational inference, the true psosterior of
latent state x $p(x\mid o)=\frac{p(o,x)}{p(o)}$ is inferred indirectly using
an approximate posterior $q(x\mid o;\phi)$, with known parameters $\phi$. To
do so, fitting the approximate posterior to the true posterior is achieved by
minimizing the divergence between the true and approximate posterior with
respect to the parameters

$D_{\mathrm{KL}}(q(x\mid o;\phi)\|p(o,x;\theta))$ (2)

Existing models that implement Generalized Gradient Descent on the Free Energy
take different forms and with that, different degrees of biological
plausibility - at different levels of analysis. Important variants
implementing generalized filtering are variational filtering [3], dynamic
expectation maximization [3] and generalized predictive coding [4]. Of these,
generalized (Bayesian) predictive coding is a particularly interesting
candidate, since it allows relatively straightforward mapping to neural
mechanism.

We will focus on generalized predictive coding in this article, but keep the
similarities to related, more generic filtering schemes in mind. Finally, HDMs
typically rest on functions operating in continuous time, which requires
additional mechanisms for updates with respect to discrete time intervals,
such as they are present in many machine learning problems of the kind that we
want to look at here.

Dynamical generative models $p(y,\vartheta)=p(y\mid\vartheta)p(\vartheta)$
describe the likelihood $p(y\mid\vartheta)$ of observing data $y$ given causes
$\vartheta=\\{x,v,\theta\\}$, and priors on causes $p(\vartheta)$. With non-
linear functions $h$ and $d$, parameterized $\theta_{h}$ and $\theta_{d}$,
these models generate outputs (responses)
$\widetilde{y}=\left[y,y^{\prime},y^{\prime\prime},\ldots\right]$
characterised by the first derivatives of their trajectory, the generalized
coordinates of motion:

$\displaystyle y$ $\displaystyle=h(x,v)+z$ (3) $\displaystyle y^{\prime}$
$\displaystyle=h_{x}x^{\prime}+h_{v}v^{\prime}+z^{\prime}$ $\displaystyle
y^{\prime\prime}$
$\displaystyle=h_{x}x^{\prime\prime}+h_{v}v^{\prime\prime}+z^{\prime\prime}$
$\displaystyle\vdots$

with stochastic observation noise z. Similarly, the generalized coordinates of
the motion of cause states are

$\displaystyle x^{\prime}$ $\displaystyle=d(x,v)+w$ (4) $\displaystyle
x^{\prime\prime}$ $\displaystyle=d_{x}x^{\prime}+d_{v}v^{\prime}+w^{\prime}$
$\displaystyle x^{\prime\prime\prime}$
$\displaystyle=d_{x}x^{\prime\prime}+d_{v}v^{\prime\prime}+w^{\prime\prime}$
$\displaystyle\vdots$

with stochastic transition noise w. Here, the cause states
$\vartheta=\\{x,v,\theta\\}$ split into input states $v(t)$ at time $t$ and
hidden states $x(t)$ that are part of a transition function $d$ that projects
input states to output states. From a machine learning perspective, we can
directly associate $x(t)$ with the hidden states or memory in Recurrent Neural
Networks (RNNs) and $d(x)$ with the corresponding transition function in RNNs.
Differently to the single-order transition function in classical RNNs, the
function $d$ couples multiple orders of dynamics.

Next to a hierarchy of the cause state dynamics, the models described in [4]
model a hierarchical structure of the cause states themselves:

$\begin{array}[]{c}v^{i-l}=g\left(x^{l},v^{l}\right)+z^{l}\\\
\dot{x}^{l}=f\left(x^{l},v^{l}\right)+w^{l}\end{array}$ (5)

where input and hidden states $v^{l}$ and $x^{l}$ at layer $l$ link states and
dynamics between layers $l$. In such hierarchical models, the outputs of a
layer are the inputs to the next lower layer and the fluctuations $z^{l}$ and
$w^{l}$ influence the fluctuations of states at the next higher level.

###  1.2 Inverting hierarchical dynamical models with Generalized Predictive
Coding

The described general structure of hierarchical dynamical models does not yet
entail the specific process that actually learns parameters and states inside
the model from observations. This process, called model inversion, can be
described at different levels of description, each of which leads to different
assumptions at the implementations level. A general approach for model
inversion is based on variational Bayesian inference, where the conditional
density $p(\vartheta\mid y,m)$ for causes $\vartheta$, model $m$ and data $y$
is approximated using a recognition density $q(\vartheta)$ with respect to a
lower bound on the evidence $p(y\mid m)$ of the model. More detailed
descriptions of hierarchical model inversion can be found, for example, in
[1]. When this Bayesian inversion is done with respect to (explicitly
represented) prediction errors, this process can be implemented with
predictive coding. (Bayesian) predictive coding is a algorithmic motif that is
plausibly represented in biological neurons and thus provides an interesting
candidate for improving machine learning systems.

Generalized predictive coding can be described as a gradient descent on
precision weighted prediction errors [1, 5]:

$\displaystyle\dot{\tilde{\mu}}_{v}^{l}$
$\displaystyle=\tilde{\mu}_{v}^{l}-\partial_{\widetilde{v}}\widetilde{\varepsilon}^{l}\cdot\xi^{l}-\xi_{v}^{(l+1)}$
(6) $\displaystyle\dot{\tilde{\mu}}_{x}^{l}$
$\displaystyle=\tilde{\mu}_{x}^{l}-\partial_{\tilde{x}}\widetilde{\varepsilon}^{l}\cdot\xi^{l}$
$\displaystyle\widetilde{\varepsilon}_{v}^{l}$
$\displaystyle=\widetilde{\mu}_{v}^{(l-1)}-g^{l}\left(\widetilde{\mu}_{x}^{l},\widetilde{\mu}_{v}^{l}\right)$
$\displaystyle\widetilde{\varepsilon}_{x}^{l}$
$\displaystyle=\tilde{\mu}_{x}^{l}-f^{l}\left(\widetilde{\mu}_{x}^{l},\widetilde{\mu}_{v}^{l}\right)$
$\displaystyle\xi_{v}^{l}$
$\displaystyle=\Sigma^{-1}_{{v}^{l}}\widetilde{\varepsilon}_{v}^{l}$
$\displaystyle\xi_{x}^{l}$
$\displaystyle=\Sigma^{-1}_{{x}^{l}}\widetilde{\varepsilon}_{x}^{l}$

where $l$ is the current layer and $\Sigma^{-1}$ is the inverse variance (the
precision) of the prediction errors $\widetilde{\varepsilon}$ in generalized
coordinates. The precision weighted prediction errors $\xi_{v}$ and $\xi_{x}$
are generated with respect to the difference between current expectations
about hidden causes and states $\tilde{\mu}_{v}$ and $\tilde{\mu}_{x}$ and
their predicted values. Predicted values of causes and states are computed
with non-linear feedback functions $f$ and $g$. Crucially, the dynamics of the
hidden states in a particular layer are computed with respect to the states in
the same layer. In contrast, cause states are updated with respect to
predictions that map from states of the next higher layer. Effectively, this
means that hidden states model the dynamics within layers, while cause states
link different layers. 111This restriction to roles in intrinsic and extrinsic
connectivity for cause and hidden states comes from assumptions about the
underlying graphical model, where the dynamics of different nodes can strictly
be separated. In biological and artificial implementations, these boundaries
might be much looser and under constant change. As we’ll see in the discussion
of this article, the interpretation of neural (cortical) function in terms of
canonical predictive coding microcircuits might be substantially less
straightforward than it appears from the math [5].

Existing models that implement a particular form of generalized filtering need
to make specific assumptions, particularly about the nature of modelled
functions and the computation of their generalized coordinates. A popular
approach is to resort to numerical differentiation with the finite difference
method. Numerical differentiation, however, often struggles with rounding and
discretisation problems and the exact computation of higher order derivatives
in general. Automatic differentiation, which drives a substantial amount of
state-of-the art machine learning solves these problems by operating on
functions with known exact gradients. It is this technique that we want to
adapt here in the context of HDMs.

While several implementations of generalized filtering variants exist, they do
not quite yet scale to complex applications such as perception of complex
data, or reinforcement learning problems such as dealt with in the deep
learning domain. Here, we want to focus on implementing hierarchical dynamical
models with update rules described by gradient based predictive coding.
Specifically, we look at Stochastic Gradient Descent (SGD) on prediction
errors with exact gradients in differentiable deep neural network (DNN)
models.

###  1.3 Differentiable predictive coding and deep neural networks

Much effort has recently been spent on developing scaled up versions of
gradient based predictive coding networks and comparing their (relatively)
biologically plausible updates with the exact gradient computation involved in
the backpropagation algorithm that drives a substantial amount of current
state-of-the art machine learning [6, 7, 8, 9, 10]. In particular, it has been
shown that, under specific assumptions 222These assumptions include fixing
predictions during weights updates or the inversion of data and target inputs
to the model - so called ”discriminative” predictive coding. the weight
updates in predictive coding networks use approximations of the exact
gradients in backpropagation [11, 7, 12]. Similar experiments for the
gradients in HDMs have been, so far, not been conducted. Related efforts have
resulted in optimization strategies that allow to turn (deep) neural networks
into gradient based predictive coding networks [11, 7]. PredProp, a DNN
optimiser proposed in [9] jointly optimizes each layer’s learnable parameters
in parallel using exact gradients. This approach removes the need to wait for
individual variables to converge (such as the fixed prediction assumption when
training weight parameters) and allows for constant interaction of the
estimated precision errors between layers. Another aspect of PredProp is the
possibility to include complex DNN architectures within the backward
prediction weights of a single predictive layer, while layer-wise predictive
coding 333Which is arguably more biologically plausible due to the
significantly reduced computational complexity in the connections between
state variables. is still an option. The model presented here is an extension
to the PredProp optimiser that focuses on adding a dynamical pathway in models
with simple feedback and transition weights. Outside of gradient based
predictive coding, there are various models that include predictive coding
inspired mechanisms into state-of-the-art deep neural architectures, although
many times with substantial differences to the mechanisms described in the
predictive coding theory [13, 6, 14]. Detailed reviews on predictive coding
variants and their connection to established methods in machine learning, such
as variational autoencoders and normalizing flows can be found in [6, 15].

##  2 Inferring content and dynamics with generalised predictive coding

We introduce an optimisation method based on Stochastic Gradient Descent,
called GPC, that implements Generalised Predictive Coding with exact gradients
along the hierarchical and dynamical dimension, while still offering the
possibility to estimate the true dynamics underlying raw sensory data. GPC
augments an existing predictive coding based optimisation method called
PredProp [9] and employs exact gradients to optimize inferred states, weights
and prediction error precision in a layer-wise fashion and in parallel (i.e.
without having to wait for other variables to converge) for arbitrary neural
network architectures. Figure 1 summarizes the main components of the proposed
optimisation scheme in a predictive coding model with two layers.

a) b)

Figure 1: a) Hierarchical and dynamical predictions in Differentiable GPC.
Blue and green colors indicate dynamical and hierarchical pathways
respectively. Dotted arrows indicate precision-weighted prediction errors that
drive local updates of learnable parameters. Here, cause states and hidden
states $v_{l}$ and $x_{l}$ are summarized as states $z_{l}=[v_{l},x_{l}]$ in
_layer_ l. b) Different _levels_ of causes can be decoupled from their
dynamics, when higher layers predict inputs (green), but do not predict their
dynamics (blue). For example, the motion ($x_{l}$) of birds ($v_{l}$) flying
might be associated with the relatively static concept of nature ($v_{l+1}$)
as a cause.

For the sake of completeness, we summarize the core functions of PredProp
here. For a more detailed description of PredProp, the reader is referred to
[9]. PredProp implements stochastic gradient descent on the negative Free
Energy. Prediction errors $\epsilon_{l}$ at each layer, weighted by their
_precision_ (inverse covariance) $\Sigma_{l}^{-1}$ are minimized for all
layers _individually_.

More precisely, under the Laplace assumption, the variational Free Energy
simplifies to a Free Energy term based on prediction errors, which can be
noted as:

$\mathcal{F}=-\sum_{l=1}^{L}\Sigma_{l}^{-1}\epsilon_{l}^{2}+\ln
2\pi\Sigma_{l}^{-1}$ (7)

with prediction errors $\epsilon_{l}$ between an observed activity $\mu_{l}$
and predicted mean activity $\hat{\mu}_{l}=f_{l}(\theta_{l+1},\mu_{l+1})$ at
layer $l$:

$\epsilon_{l}=\mu_{l}-f_{l}\left(\theta_{l+1},\mu_{l+1}\right)$ (8)

More details on the variational Free Energy and the Laplace assumption in the
context of predictive coding are discussed in [16, 6, 8].

Effectively, a cost function is optimized for each layer in parallel. As
predictive coding are linked via shared cause states, the summed cost function
of the entire network is optimized simultaneously. PredProp optimises networks
bidirectionally, so propagated values constantly iterate between inferring the
most likely cause (i.e. target) of observed data and the most likely data
given the currently estimated cause. Each predictive coding layer $l$ has at
least one or multiple weights parameterizing the backwards connection
$\theta$, followed by a (non-)linear activation function $f$. Theoretically,
these multi-layer backward connections can incorporate entire DNN structures,
since exact gradients can be computed with automatic differentiation.

In a single-layer predictive coding network, the observed activity $\mu$
simply is the input data. Layers in multi-layer predictive coding networks
predict the activity $\mu_{l-1}$ of the next lower layer. The backward
predictions of the next lower layer are computed using a (non-)linear
activation function $f_{l}$ and learnable weights $\theta_{l}$. The resulting
update at time $t$ for the backward weights $\theta_{l}$ at layer $l$ is

$\displaystyle\frac{d\theta_{l+1}}{dt}=\frac{\partial\mathcal{F}}{\partial\theta_{l}}=-\Sigma_{l-1}^{-1}\epsilon_{{l-1}}\frac{\partial
f}{\partial\theta_{l}}\mu_{l}^{T}$ (9)

Next to updating the weights (i.e. parameter learning), PredProp performs a
simultaneous Gradient Descent on activity (i.e. state inference) and precision
in each layer directly. The corresponding update rules for activity $\mu_{l}$
and precision $\Sigma_{l}^{-1}$ at layer $l$ are:

$\displaystyle\frac{d\mu}{dt}$
$\displaystyle=-\frac{\partial\mathcal{F}}{\partial\mu}=\Sigma_{l}^{-1}\epsilon_{l}\frac{\partial
f}{\partial\mu_{l+1}}\theta_{l+1}^{T}-\Sigma_{l}^{-1}\epsilon_{l}$ (10)
$\displaystyle\frac{d\Sigma}{dt}$
$\displaystyle=-\frac{\partial\mathcal{F}}{\partial\Sigma}={\Sigma_{l}^{-1}\epsilon}^{2}-\Sigma_{l}$

As discussed in more detail in [6, 17], the estimated precision of the
prediction error plays an important role in predictive coding, since it
estimates second-order changes in the partial objective functions and weights
parameter updates accordingly.

###  2.1 From static to dynamic predictive coding models

In contrast to PredProp, which so far has been applied to static perception
tasks, GPC addresses changes in observed inputs. Like many deep neural
architectures, GPC operates on discrete time steps. However, the transitions
inside the model have adaptive interval sizes, which are modulated only by the
respective layer individually, in order to improve its own predictions as well
as its top-down predictability. Since deeper dynamical layers in GPC encode
_known_ analytic functions via their feedback weights, discrete changes over
such adaptive intervals can efficiently be encoded. Importantly, at each
hierarchical layer, multiple instance operate in parallel and randomly sample
new intervals modulated by the precision of prediction errors. Figure 2 shows
the learnable variables that are relevant to the transition function at an
individual predictive coding layer.

Each layer $l$ estimates a state $x$ at discrete time $t$ for time increments
$dt$ of arbitrary size. Additionally, each layer learns transition weights
$W_{d}$ that paramterize the state transition function $d$

$x_{l,t}=d(x_{l,t-dt},dt)+s$ (11)

where $u$ is the stochastic noise driving the selection of sampling intervals
(the step size or stride in the context of DNN models with discrete
timesteps).

Weights $W_{h}$ parameterize the hierarchical prediction function $h$:

$x_{l,t}=h(x_{l+1,t})=W_{h}(x_{l+1,t})$ (12)

Figure 2: Temporal transitions and associated dynamical prediction errors
occurring in the transition function $d_{l}$ at each dynamical layer $l$ of
the predictive coding network.

Functions $d,h$ parameterize a single connection along the dynamical and
hierarchical pathway respectively. Layers along the hierarchical dimension are
connected as each layer’s hierarchical prediction is the expected state at
thse next lower layer $x_{l,t}$. Neighboring layers along the dynamical
dimension are connected as each layer predicts the derivative of the
transition function of the next lower layer $d(x_{l,t})$. The discrete time
increment $dt$ plays an important role, since it influences the estimated
derivative of the observed activity, by paying attention to changes across
different temporal scales.

This estimated derivative is approximated by transition weights $\theta_{t}$.
Since the learned transition function is known (in contrast to the function
generating the observed data), so is its exact gradient. In the dynamical
pathway, neighboring layers predict this gradient of the next lower layer.
Alternatively to computing the gradient, it is possible to simply predict the
change that the transition function applies to known state inputs:

$\displaystyle d^{\prime}(x_{l,t})$
$\displaystyle=\frac{d(x_{l,t},dt)-d(x_{l,t-dt},dt)}{dt}$ (13)
$\displaystyle=\frac{d(x_{l,t},dt)-x_{l,t}}{dt}$

.

Here, we use that $x_{l,t}=W_{d}x_{l,t-dt}$, i.e. we require the transition
weights to be estimated with respect to a known time increment $dt$. Initially
$dt$ is selected randomly based on each layer’s prior on the sampling interval
fluctuation $s$. Since multiple instances of a layer can predict the same
input, different sampling intervals $dt$ are covered at any point in time. The
interval $dt$ is known in the local context of a layer, but is not propagated
to distant layers. As each layer’s states are predicted top-down, the optimal
inferred sampling interval $dt$ balances a low prediction error for the
outgoing prediction with the incoming top-down prediction.

Dynamical and hierarchical pathways are mutually interacting:

$\displaystyle x_{l,t}$ $\displaystyle=d(x_{l,t-dt},dt)=d(h(x_{l+1,t-dt}),dt)$
(14) $\displaystyle=h(x_{l+1,t})=h(d(x_{l+1,t},dt))$

At discrete time step $t$, the total prediction error in each layer is the sum
of dynamical and hierarchical prediction errors:

$\displaystyle e_{l}$ $\displaystyle=e_{x,l}+e_{d,l}+e_{d^{\prime},l}$ (15)
$\displaystyle=(x-h(x_{l+1,t}))+(x-d(x_{l,t-dt})+(d^{\prime}(x_{l,t})-\hat{d^{\prime}(x_{l,t}}))$
$\displaystyle=(x-\hat{x}_{h})+(x-\hat{x}_{d})+(dx-\hat{dx_{d}})$

where $\hat{x}_{h},\hat{x}_{d}$ and $\hat{dx_{d}}$ refer to the hierarchically
predicted state, the transitioned state and the predicted state change of the
last state transition respectively.

Given this objective function for each predictive coding layer, we can apply
the optimised network to various learning and inference tasks. Optimising
multiple sample intervals in parallel is thought to reduce the need for more
complex autoregressive modelling, by trading computational complexity within
single layers with increased parallelism. To test this hypothesis, a first
step is to apply the network to the prediction of sequential data. It should
be noted that, although we focus on sampling temporal inputs, the same method
can be applied to spatial dimensions. The following section evaluate dynamics
learning from sequential data, with and without sensory noise and non-linear
activation functions.

##  3 Learning dynamics from sequential data

###  3.1 Learning dynamics with linear and non-linear activation

With linear activations, GPC infers observed $f(x)$ and the first two
derivatives $f^{\prime}(x),f^{\prime\prime}(x)$ relatively exactly. When a
ReLU non-linearity is used, the represented function has less complex
generalised coordinates, but still matches the observed $f(x)$ well. Figure 3
show examples for dynamical state estimation on sequential inputs with and
without non-linear activation functions.

Figure 3: Approximated representation of a sine function in a GPC model with
three hidden dynamical layers (black and green lines). The feedback
predictions are linearly activated (top) or ReLU activated (bottom).
Prediction errors from the transition functions $d_{1}$ and $d^{\prime}_{2}$
as well as their top-down predictions $\hat{d_{2}}$ and $\hat{d^{\prime}_{2}}$
are indicated in red and yellow respectively.

###  3.2 Learning dynamics with parallel sampling

With changing interval size (stride), the inferred generalised coordinates
change significantly. In the example shown in Figure 4, a stride of 10 leads
to a relatively constant stationary input, where the second represented
derivative remains nearly constant. For strides other than 5 or 10, the
prediction error is substantially higher.

Figure 4:  Dynamical state estimation on a modulated sine wave sampled at
three different strides (1,9,10) in the lowest predictive coding layer.

##  4 Relation to the canonical microcircuit for predictive coding in
biological networks

Figure 5:  Comparison of Differentiable Generalized Predictive Coding with the
canonical microcircuit for predictive coding that is hypothesized to drive
Bayesian inference in the human brain.

A lot of effort has been spent on describing canonical microcircuits that
capture the core functionality of neuronal processing in human brains. In the
context of interpreting neuronal computations as performing Bayesian
inference, multiple similar models of canonical computations have been
suggested that show close correspondence to the connectivity described by
generalised predictive coding [18, 19, 20, 5].

Figure 5 shows a comparison between GPC, the Differentiable Generalised
Predictive Coding model suggested here and the canonical microcircuit (CMC)
described by Bastos et al. [5]. When comparing the structure for a single
predictive coding layer $l$, there is a clear correspondence between cause
states, hidden states and associated prediction errors in biological and
artificial networks. In the context of cortical function, a layer in CMC sends
forward prediction errors to higher cortical areas and backward predictions
towards lower areas. While this structure is visible in GPC, it lacks the
explicit differentiation of states into superficial pyramidal cells and deep
pyramidal cells. In the CMC, this differentiation can be connected to
different sampling frequencies associated with incoming and outgoing
information [5].

The comparison with canonical circuitry and concrete neurophysiological
evidence provides a useful starting point for more in depth analysis of the
biological plausibility of computations in deep predictive coding networks
(and for DNNs in general). One striking aspect in CMC is that feedforward
information typically is propagated via _linear_ connections, while backward
predictions feature _non-linear_ connectivity. GPC and PredProp consider this
differentiation by including DNNs into the backward weights, while keeping
precision and transition pathways linear.444PredProp, however, theoretically
allows to to include non-linearities and complex autoregressive DNN models
into the remaining connections between state variables. Finally, comparing
with the biological interpretation in CMC allows to address inhibitory and
excitatory jointly with associated sampling frequencies in deep predictive
coding explicitly.

The hypothesized canonical circuitry provides an important bridge between
analysis of predictive coding from a functional and computational level of
analysis to concrete biological implementation. Similarly, GPC provides a
concrete implementation of generalised predictive coding in the context of
deep neural networks.

One interesting line of research could address more exact modelling of the CMC
and its implications on performance in machine learning tasks. Another, quite
elegant way of characterising aspects of functional modularity in concrete
implementations is to define functional modules in terms of statistical
independence, i.e. by their Markov Blankets. We will use this approach in the
following section in order to analyse differences between probabilistic
graphical models describing generalized predictive coding and its concrete
DNN-based implementation in GPC.

###  4.1 From neurons to graphical models via Markov blankets

Figure 6: Left: Neuronal Markov blankets describe that separate individual
neurons and their surrounding. Regional Markov Blankets characterize the
effective connectivity between cortical layers based on canonical
microcircuits for predictive coding. Right: Similarly, regional Markov
blankets can be mapped to the connectivity between predictive coding layers in
Differentiable Generalised Predictive Coding.

A relatively recent line of research addresses the formalisation of dynamical
coupling in biological systems across (spatial) scales via Markov Blankets
[21, 22, 23, 24, 25]. Markov blankets describe a partitioning of complex
dynamical systems in terms of statistical boundaries. When applied to
characterising functional modules in biological brains, this entails
partitions appearing at multiple scales, such as at the level of individual
neurons, at the level of canonical micro-circuitry or at the level of brain
regions and larger networks [26]. Markov blankets are an effective way to
characterise statistical modularity in complex systems and can directly be
mapped to the effective connectivity described by the model suggested here.
Figure 6 shows a simplified overview of Markov blankets at the neuronal and
micro-circuit level, next to a mapping to the connectivity structure of GPC.
While we characterize Markov Blankets at the regional level here and focus on
relatively shallow GPC models, future work might characterize Markov Blankets
appearing at larger spatio-temporal scales in more complex implementations.

Crucially, Markov Blankets divide the states of a systems into external,
sensory, active and internal states [26]. Of these, sensory and active states
can be summarized into blanket states, which shield internal and external
partitions. Internal states are updated based on internal and blanket states,
but are independent of external states. Similarly, active states do not depend
upon external states. As shown in Figure 6 the level of canonical micro-
circuits, one can separate neighboring layers in GPC into internal and
external states, where information is propagated exclusively through active
and sensory state, the state predictions and prediction errors respectively.
An important aspect Markov blankets is that the described modularity is
characterised exclusively by statistical dependencies and is not directly
dependent on the physical structure of the systems.

This differentiation between a system’s structure in terms of Markov Blankets
and the implementation can be applied in straightforward fashion to artificial
systems modelled to mimic particular biological function or are tailored to a
particular graphical model. The computations in the model presented here is
based on the graphical model described by Generalised Predictive Coding. As
visible in Figure 1 the implementation of the graph described by Generalised
Predictive Coding seems to require exact connectivity to maintain a valid
structure. For example, interconnecting variables within dynamical and
hierarchical pathways intuitively seems to change the implemented graphical
model. Future work, however, could relax the assumption of such strict mapping
between connectivity in the graphical model and the implementation of GPC by
allowing more flexible (i.e. random up to a certain degree) connectivity
between variables. We can then characterize particular nodes in the model via
their Markov blankets instead of a one-to-one mapping from functional to
implementational level.

##  5 Planning with Generalised Predictive Coding

The hierarchical-dynamical transition model learned with GPC can be used for
planning task in relatively straightforward matter. While arbitrarily complex
algorithms could be employed on top of the learned model-based predictive
model. Here, we focus on planning similar to recent formulations of Active
Inference agents that optimise the _Expected_ Free Energy of with respect to
policies over discrete actions [27, 28, 29, 30]. Policies $\pi$ are sequences
of actions $a_{t}$ at discrete time step $t$. In many reinforcement learning
algorithms, one would optimise on sequences of actions and observed rewards
$r_{t}$:

$\tau=(a_{0},r_{0},a_{1},r_{1},...,a_{T},r_{T})$ up to a planning horizon T.
Typically, reward based methods require additional regularization that enables
exploration in order to improve generalization. Such exploration often refers
to testing new policies that might not be connected to (immediate) optimal
reward and is typically modelled by including a random sampling process or
some form of expected information gain.

The approach in active inference is slightly different and does not directly
operate on a known reward function.555External reward, however, can be
integrated by defining a prior preference on maximising observed reward.
Active inference models optimises the Expected Free Energy (EFE), i.e. the
Free Energy of expected observations for planned policies. Instead of
resolving to external rewards, we can define actions over adaptive time
intervals $dt_{a}$ and the associated Expected Free Energy:

$\displaystyle\tau_{dt_{1}}$ $\displaystyle=(a_{1},EFE_{a_{1}})$ (16)
$\displaystyle\tau_{dt_{2}}$ $\displaystyle=(a_{2},EFE_{a_{2}})$
$\displaystyle...$ $\displaystyle=...$ $\displaystyle\tau_{dt_{N}}$
$\displaystyle=(a_{N},EFE_{a_{N}})$

in N parallel policies with discrete actions $a_{n}$ over planning horizons
$dt_{n}$ with adaptive size. The adaptive sampling intervals in GPC implies
that the temporal666Alternatively, the spatial interval size, when planning is
done with respect to spatial dimensions. In hidden layers, the adaptive
sampling size refers to the predicted motion of causes represented at the
respective lower layer. interval size for each action $a_{t}$ depends on the
variational Free Energy of the previously inferred state.

We can interpret this EFE objective as a sort of performance measure and
simply optimise it with Stochastic Gradient Descent, similar to existing work
on active inference interpreted as a policy gradient method [31].

##  6 Nested hierarchies and perceptual actions

(a) A predictive coding network with four local and one distant nodes with
nested hierarchical-dynamical structure.

(b) Example trajectory with redundancy that can be reduced during perceptual
model reduction in mental simulation.

Figure 7: Perception and planning in generalized predictive coding networks
with learned connectivity.

## References

  * [1] Karl Friston  “Hierarchical models in the brain”  In _PLoS computational biology_ 4.11 Public Library of Science San Francisco, USA, 2008, pp. e1000211
  * [2] Karl Friston, Klaas Stephan, Baojuan Li and Jean Daunizeau  “Generalised filtering”  In _Mathematical Problems in Engineering_ 2010 Hindawi, 2010
  * [3] Karl J Friston, N Trujillo-Barreto and Jean Daunizeau  “DEM: a variational treatment of dynamic systems”  In _Neuroimage_ 41.3 Elsevier, 2008, pp. 849–885
  * [4] Karl Friston and Stefan Kiebel  “Predictive coding under the free-energy principle”  In _Philosophical transactions of the Royal Society B: Biological sciences_ 364.1521 The Royal Society London, 2009, pp. 1211–1221
  * [5] Andre M Bastos et al.  “Canonical microcircuits for predictive coding”  In _Neuron_ 76.4 Elsevier, 2012, pp. 695–711
  * [6] Beren Millidge, Anil Seth and Christopher L Buckley  “Predictive Coding: a Theoretical and Experimental Review”  In _arXiv preprint arXiv:2107.12979_ , 2021
  * [7] Robert Rosenbaum  “On the relationship between predictive coding and backpropagation”  In _arXiv preprint arXiv:2106.13082_ , 2021
  * [8] Rafal Bogacz  “A tutorial on the free-energy framework for modelling perception and learning”  In _Journal of mathematical psychology_ 76 Elsevier, 2017, pp. 198–211
  * [9] André Ofner and Sebastian Stober  “PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Coding”  In _arXiv preprint arXiv:2111.08792_ , 2021
  * [10] David E Rumelhart, Geoffrey E Hinton and Ronald J Williams  “Learning representations by back-propagating errors”  In _nature_ 323.6088 Nature Publishing Group, 1986, pp. 533–536
  * [11] Beren Millidge, Alexander Tschantz and Christopher L Buckley  “Predictive coding approximates backprop along arbitrary computation graphs”  In _arXiv preprint arXiv:2006.04182_ , 2020
  * [12] James CR Whittington and Rafal Bogacz  “An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity”  In _Neural computation_ 29.5 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2017, pp. 1229–1262
  * [13] William Lotter, Gabriel Kreiman and David Cox  “Deep predictive coding networks for video prediction and unsupervised learning”  In _arXiv preprint arXiv:1605.08104_ , 2016
  * [14] Roshan Prakash Rane et al.  “PredNet and Predictive Coding: A Critical Review”  In _Proceedings of the 2020 International Conference on Multimedia Retrieval_ , 2020, pp. 233–241
  * [15] Joseph Marino  “Predictive coding, variational autoencoders, and biological connections”  In _arXiv preprint arXiv:2011.07464_ , 2020
  * [16] Karl Friston, James Kilner and Lee Harrison  “A free energy principle for the brain”  In _Journal of physiology-Paris_ 100.1-3 Elsevier, 2006, pp. 70–87
  * [17] Andre Ofner, Raihan Kabir Ratul, Suhita Ghosh and Sebastian Stober  “Predictive coding, precision and natural gradients”  In _arXiv preprint arXiv:2111.06942_ , 2021
  * [18] Rodney J Douglas and KA Martin  “A functional microcircuit for cat visual cortex.”  In _The Journal of physiology_ 440.1 Wiley Online Library, 1991, pp. 735–769
  * [19] Rodney J Douglas and Kevan AC Martin  “Neuronal circuits of the neocortex”  In _Annu. Rev. Neurosci._ 27 Annual Reviews, 2004, pp. 419–451
  * [20] Rodney J Douglas, Kevan AC Martin and David Whitteridge  “A canonical microcircuit for neocortex”  In _Neural computation_ 1.4 MIT Press, 1989, pp. 480–488
  * [21] Karl Friston  “A free energy principle for a particular physics”  In _arXiv preprint arXiv:1906.10184_ , 2019
  * [22] Maxwell James Désormeau Ramstead, Paul Benjamin Badcock and Karl John Friston  “Answering Schrödinger’s question: A free-energy formulation”  In _Physics of life reviews_ 24 Elsevier, 2018, pp. 1–16
  * [23] Ensor Rafael Palacios et al.  “On Markov blankets and hierarchical self-organisation”  In _Journal of theoretical biology_ 486 Elsevier, 2020, pp. 110089
  * [24] Michael Kirchhoff et al.  “The Markov blankets of life: autonomy, active inference and the free energy principle”  In _Journal of The royal society interface_ 15.138 The Royal Society, 2018, pp. 20170792
  * [25] Judea Pearl  “Graphical models for probabilistic and causal reasoning”  In _Quantified representation of uncertainty and imprecision_ Springer, 1998, pp. 367–389
  * [26] Inês Hipólito et al.  “Markov blankets in the brain”  In _Neuroscience & Biobehavioral Reviews_ Elsevier, 2021
  * [27] Kai Ueltzhöffer  “Deep active inference”  In _Biological cybernetics_ 112.6 Springer, 2018, pp. 547–573
  * [28] Ozan Çatal et al.  “Learning perception and planning with deep active inference”  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 3952–3956  IEEE
  * [29] Otto Himst and Pablo Lanillos  “Deep active inference for partially observable MDPs”  In _International Workshop on Active Inference_ , 2020, pp. 61–71  Springer
  * [30] Zafeirios Fountas, Noor Sajid, Pedro AM Mediano and Karl Friston  “Deep active inference agents using Monte-Carlo methods”  In _arXiv preprint arXiv:2006.04176_ , 2020
  * [31] Beren Millidge  “Deep active inference as variational policy gradients”  In _Journal of Mathematical Psychology_ 96 Elsevier, 2020, pp. 102348