11institutetext: Università degli studi di Roma ‘Tor Vergata’, Via della
ricerca scientifica, 1; 00133 Roma, Italy

# Pressure profiles of distant Galaxy clusters with Planck-SPT data

Filippo Oppizzi 11<EMAIL_ADDRESS>Federico De Luca 11 HervÃ Bourdin
11 Pasquale Mazzotta 11 the CHEX-MATE collaboration

###### Abstract

We present a full set of numerical tools to extract Galaxy Cluster pressure
profiles from the joint analysis of Planck and South Pole Telescope (SPT)
observations. Pressure profiles are powerful tracers of the thermodynamic
properties and the internal structure of the clusters. Tracing the pressure
over the cosmic times allows to constraints the evolution of the cluster
structure and the contribution of astrophysical phenomena. SPT and Planck are
complementary to constrain the cluster structure at various spatial scales.
The SPT cluster catalogue counts 677 cluster candidates up to redshift 1.7, it
is a nearly mass limited sample, an ideal benchmark to test cluster evolution.
We developed a pipeline to first separate the cluster signal from the
background and foreground components and then jointly fit a parametric profile
model on a combination of Planck and SPT data. We validate our algorithm on a
sub-sample of six clusters, common to the SPT and the CHEX-MATE catalogues,
comparing the results with the profiles obtained from X-ray observations with
XMM-Newton.

##  1 Introduction

Galaxy clusters form from the largest gravitational overdensities in the
density field and track the evolution of the Universe at the largest scales
(2012ARA&A..50..353K, ). A large amount of ionised hot gas lies in hydrostatic
equilibrium in the potential well of the Dark Matter (DM) halos of the
clusters. As a consequence of the tight correlation between the gas and the DM
density field, the scale invariance of the latter reflects in the self-
similarity of the pressure profiles. The intra-cluster gas pressure is thus a
sensitive tracer of the internal structure of galaxy clusters.
The Sunyaev-Zeldovic effect (SZ) arises from the interaction of the free
electrons in the ionised gas with the Cosmic Microwave Background (CMB)
photons. It is a direct tracer of the gas pressure and is independent of the
redshift. (1972CoASP…4..173S, ). The gas also emits bremsstrahlung radiation
in X-ray, providing a complementary probe of the cluster structure that is
sensitive to different density regimes.
In this work, we present a pipeline to extract cluster pressure profiles from
a combination of Planck and the South Pole Telescope (SPT) data. The high
sensitivity of Planck and the SPT high resolution makes the two instruments
complementary in constraining the cluster structure from the core to the
outskirts. We compare the results from this SZ analysis with the profiles
derived from X-ray measurements. We analyse six clusters extracted from the
sample of the Cluster HEritage project with XMM-Newton-Mass Assembly and
Thermodynamics at the Endpoint of structure formation (CHEX-MATE) (CHEX-MATE,
).
In section 2 we present the data-sets and the cluster sample, in section 3 we
describe the analysis techniques and in section 3 we present our results.
Finally, section 5 is dedicated to the conclusions.

##  2 Data-sets and Cluster Sample

SPT and Planck represent the state of the art of cosmological surveys at
millimetre and sub-millimetre wavelengths. Combining the information from both
data-sets it is possible to solve the inner core of all clusters in the SPT
catalogue and at the same time to map the faint peripheries with high
precision.
We use the public data from the Planck High-Frequency Instrument (HFI)
(2020A&A…641A…3P, ). The six sky maps with nominal frequencies $100$, $143$,
$217$, $353$, $545$ and $857GHz$ and resolution 9.66, 7.22, 4.90, 4.92, 4.67,
4.22 arcmin respectively.
SPT observes the microwave sky in three frequency bands centred at 95, 150 and
220 GHz with 1.7, 1.2, and 1.0 arcmin resolution, respectively over an area of
2540 $\deg^{2}$ (2011PASP..123..568C, ). Thanks to its resolution it can solve
the core of distant galaxy clusters, inaccessible by Planck
(2015ApJS..216…27B, ). In this work, we use the public SPT
maps111https://pole.uchicago.edu/public/data/chown18/index.html that are
convolved with a common Gaussian beam with 1.75 arcmins FWHM.
CHEX-MATE is a program of X-ray observations with the XMM-Newton satellite of
a selection of 118 galaxy clusters of the Planck PSZ2 catalogue designed to be
unbiased and signal-to-noise-limited (CHEX-MATE, ). To test our technique we
use a set of clusters representing the overlap between the SPT and the CHEX-
MATE catalogue. We use the X-ray counterparts as a benchmark for the pressure
profile derived from our analysis of millimetric data. The sample covers a
redshift range from $z=1.6$ to $z=0.6$, with angular dimension from
$r_{500}=2.8$ to $r_{500}=7.9$ arcmin.

##  3 Method

Figure 1: Flow chart that describe the SPT component separation pipeline
described in section 3.2

We first perform a component separation to remove the contaminant emissions on
the data from the two instruments separately and then we fit the pressure
profiles on both data-sets. We refer to inprep  for detailed description of
the method.

###  3.1 Processing of Planck data

We project the full sky Planck maps on tiles of $512\times 512$ pixels, with
the resolution of 1 arcmin/pixel centred on the cluster. Each image is then
processed with the technique developed in 2017ApJ…843…72B  to isolate the
cluster signal from the background and foreground components. The technique
consists of a four components fit taking into account the cluster SZ signal,
the primary CMB anisotropies, the Galactic Thermal Dust and a correction for
the intra-cluster Dust emission. We use a wavelet thresholding algorithm to
obtain the spatial templates of the CMB and the Galactic dust directly from
the 857 GHz and the 217 GHz Planck channels respectively. To model the thermal
dust spectral energy distribution we use the two-temperature grey body
proposed in 2015ApJ…798…88M . We compute the correction for intra-cluster dust
assuming a Navarro-Frenk-White profile (NFW) and a grey body spectral energy
distribution (SED). All the templates are then convolved with the proper beams
to match the resolution of each channel, fitted and subtracted to the raw data
to obtain the cleaned maps where we will perform the cluster fit.

###  3.2 Processiong of SPT data

As for Planck we select patches of $512\times 512$ pixels around the XMM
cluster centre, but with resolution of 0.5 arcmin/pixel instead of 1
arcmin/pixel. We do not use the same technique applied in the Planck data
reduction, since SPT lacks the high-frequency channels necessary to model the
thermal dust. On the other hand, SPT is insensitive to the large spatial
scales where the dust emission is relevant, our clusters are far from the
dusty galactic plane, and SPT observe at low frequencies where the dust SED is
low. Thus, we do not expect to observe significant dust contamination, and we
do not find any evidences of it in our analysis. Therefore, we focus on the
modelling of the CMB signal using a linear combination (LC) of the three SPT
channels and the 217GHz channel of Planck that we re-project on a map with the
same characteristic of the SPT ones. We compute the weights of the LC
minimising the variance with respect to a signal with a flat SED (i.e. the
CMB) and simultaneously to null the non-relativistic SZ component. We split
then each map in a low-pass filtered map matching the Planck 217 resolution,
and in a high pass filtered map containing the residual small scale features.
We then perform the LC separately on the low-passed maps and high-passed maps,
including the Planck map in the combination of the large scales. We obtain two
maps representing the large and small scales CMB fluctuations, that we re-
combine into a single map. Finally, we denoise the CMB template with a wavelet
thresholding algorithm. Fig. 1 shows the flow chart of the process just
described.

Figure 2: Intensity profiles around PSZ2G259.98-63.43 for the SPT channels (
upper panels) and Planck channels (lower panels). Black dots represent the
radial average on the cleaned maps, the red line is the cluster profile, the
blue line is the background. For Planck the dotted cyan line is the CMB, the
dashed red line is the GTD and the dash-dotted orange line is the cluster
dust.

###  3.3 Joint fit

We combine the SPT and Planck data in a joint fit of a cluster thermal SZ
template derived from the projection along the line of sight of a generalized
NFW (gNFW) pressure profile:

$P(r)=P_{0}\times\frac{P_{500}}{x^{\gamma}(1+x^{\alpha})^{(\beta-\gamma)/\alpha}},$
(1)

where $x=c_{500}r/r_{500}$, $r_{500}$ is the radius where the density is 500
times the critical density of the Universe at the cluster redshift, $c_{500}$
is the concentration parameter and $\alpha,\beta$ and $\gamma$ are the slopes
at $r\sim r_{500}/c_{500}$, $r\gg r_{500}/c_{500}$ and $r\ll r_{500}/c_{500}$
respectively. The template is then converted in SZ emissivity and convolved
with the beam and the SPT window function to build a multi channel model of
the cluster signal. We compute the likelihood on the radially averaged
profiles of the cleaned data and the model. We use circular bins of width
$0.2r_{500}$ for SPT, and of 2 arcmin for Planck, up to $3r_{500}$ for SPT and
$6r_{500}$ for Planck. We assume a Gaussian Likelihood and we perform a MCMC
fit with the Cobaya MCMC sampler (2021JCAP…05..057T, ). For Planck channels,
we obtain the covariance matrices from Monte-Carlo simulations of the noise in
the surroundings of the cluster. For SPT channels instead, we use the sample
covariance of the noise maps provided by the SPT collaboration
(2018ApJS..239…10C, ). We try different combinations of parameters to optimise
the convergence of our chain. We conclude to leave free to vary the amplitude
$P_{0}$ and the inner and outer slopes $\gamma$ and $\beta$ and to keep the
concentration parameter and the intermediate slope fixed to their universal
values identified by 2010A&A…517A..92A  $c_{500}=1.177$ and $\alpha=1.0501$.

###  3.4 X-ray analysis

We based our X-ray analysis on the observations of the European Photon Imaging
Camera (EPIC) public data of the XMM-Newton telescope. We assume a background
model taking into account the quiescent particle background, the cosmic X-ray
background, and the thermal emission associated with our Galaxy. We threat
flares and high-background periods as in Bourdin2013 . We estimate the X-ray
thermodynamical profiles as done in 2017ApJ…843…72B . We use the expressions
introduced in Vikhlinin2006  to parametrise the emission measure
$[n_{p}n_{e}](r)$ and temperature $T(r)$. We then integrate the templates
along the line of sight and we fit them to the background-subtracted X-ray
observable profiles. We refer to Delucainprep  for a complete description of
our X-ray data reduction technique.

##  4 Results

Figure 3: Profiles from XMM-Newton data (dash dotted red lines) and from SPT-
Planck joint fit (blue lines). The shaded regions correspond to the $68\%$
credible intervals. The dotted black lines show the universal profile from
2010A&A…517A..92A . The dashed lines indicate the FWHM of the SPT beam.

We show in Fig. 2 the intensity profile of the estimated cluster signal and
the contaminant components at each frequency for one of the clusters in our
sample, compared with the cleaned data. We compare the pressure profile
derived from the SZ maps with the expectations from X-ray observations. Since
X-ray and SZ data probe the cluster structure at different regimes, we limit
our comparison to the overlapping region between the two, i.e. inside
$r_{500}$. We show a preliminary comparison between the two fits on two
clusters of our sample in Fig. 3. The agreement is remarkable: the credibility
intervals overlap along the whole radius range tested. Some degree of
divergence can be observed in the very inner regions that are not solved by
SPT. The agreement between the two analysis confirms the consistency of our
techniques.

##  5 Conclusions

We presented the preliminary results of the measurement of the pressure
profile of galaxy clusters from the joint analysis of SPT and Planck data. We
developed two independent data reduction pipelines to fully exploit the
potential of each instrument. We fit a spherically symmetric gNFW model of the
profiles, processed by the instrumental response of each channel, to
efficiently exploit the information content of each band. We applied our
algorithm to a sample of six galaxy clusters representing the intersection
between the SPT and CHEX-MATE catalogues. Here we presented the preliminary
results on two clusters and we refer to inprep  for the complete analysis. We
compared the profiles derived from the SPT-Planck joint fit to the ones
obtained from the XMM-Newton data. We found a good agreement between the
results of the two measurements. With the confirmation of the consistency of
our method in hand, we will expand our analysis to a larger number of SPT
clusters to investigates the structure and evolution of the profiles through
cosmic time.

## References

  * (1) A.V. Kravtsov, S. Borgani, Annual Review of Astronomy and Astrophysics 50, 353 (2012), 1205.5556
  * (2) R.A. Sunyaev, Y.B. Zeldovich, Comments on Astrophysics and Space Physics 4, 173 (1972)
  * (3) The CHEX-MATE Collaboration, Arnaud, M., Ettori, S., Pratt, G. W., Rossetti, M., Eckert, D., Gastaldello, F., Gavazzi, R., Kay, S.T., Lovisari, L. et al., A&A 650, A104 (2021)
  * (4) Planck Collaboration, N. Aghanim, Y. Akrami, M. Ashdown, J. Aumont, C. Baccigalupi, M. Ballardini, A.J. Banday, R.B. Barreiro, N. Bartolo et al., A&A 641, A3 (2020), 1807.06207
  * (5) J.E. Carlstrom, P.A.R. Ade, K.A. Aird, B.A. Benson, L.E. Bleem, S. Busetti, C.L. Chang, E. Chauvin, H.M. Cho, T.M. Crawford et al., PASP 123, 568 (2011), 0907.4445
  * (6) L.E. Bleem, B. Stalder, T. de Haan, K.A. Aird, S.W. Allen, D.E. Applegate, M.L.N. Ashby, M. Bautz, M. Bayliss, B.A. Benson et al., The Astrophysical Journal Supplement Series 216, 27 (2015), 1409.0850
  * (7) F. Oppizzi, F. De Luca, H. Bourdin, P. Mazzotta, The CHEX-MATE Collaboration, in prep (2022)
  * (8) H. Bourdin, P. Mazzotta, A. Kozmanyan, C. Jones, A. Vikhlinin, The Astrophysical Journal 843, 72 (2017), 1707.02248
  * (9) A.M. Meisner, D.P. Finkbeiner, The Astrophysical Journal 798, 88 (2015), 1410.7523
  * (10) J. Torrado, A. Lewis, JCAP 2021, 057 (2021), 2005.05290
  * (11) R. Chown, Y. Omori, K. Aylor, B.A. Benson, L.E. Bleem, J.E. Carlstrom, C.L. Chang, H.M. Cho, T.M. Crawford, A.T. Crites et al., The Astrophysical Journal Supplement Series 239, 10 (2018), 1803.10682
  * (12) M. Arnaud, G.W. Pratt, R. Piffaretti, H. Böhringer, J.H. Croston, E. Pointecouteau, A&A 517, A92 (2010), 0910.1234
  * (13) H. Bourdin, P. Mazzotta, M. Markevitch, S. Giacintucci, G. Brunetti, The Astrophysical Journal 764, 82 (2013)
  * (14) A. Vikhlinin, A. Kravtsov, W. Forman, C. Jones, M. Markevitch, S.S. Murray, L. Van Speybroeck, The Astrophysical Journal 640, 691 (2006), astro-ph/0507092
  * (15) F. De Luca, H. Bourdin, P. Mazzotta a, F. Oppizzi, The CHEX-MATE Collaboration, in prep (2022)


# PredNet and Predictive Coding: A Critical Review

Roshan Prakash Rane<EMAIL_ADDRESS>University of Potsdam, Potsdam,
Germany ,  Edit Szügyi<EMAIL_ADDRESS>University of Potsdam, Potsdam,
Germany ,  Vageesh Saxena<EMAIL_ADDRESS>University of Potsdam,
Potsdam, Germany ,  André Ofner<EMAIL_ADDRESS>Otto von Guericke
University, Magdeburg, Germany and  Sebastian Stober<EMAIL_ADDRESS>Otto von
Guericke University, Magdeburg, Germany

(2020)

###### Abstract.

PredNet, a deep predictive coding network developed by Lotter et al., combines
a biologically inspired architecture based on the propagation of prediction
error with self-supervised representation learning in video. While the
architecture has drawn a lot of attention and various extensions of the model
exist, there is a lack of a critical analysis. We fill in the gap by
evaluating PredNet both as an implementation of the predictive coding theory
and as a self-supervised video prediction model using a challenging video
action classification dataset. We design an extended model to test if
conditioning future frame predictions on the action class of the video
improves the model performance. We show that PredNet does not yet completely
follow the principles of predictive coding. The proposed top-down conditioning
leads to a performance gain on synthetic data, but does not scale up to the
more complex real-world action classification dataset. Our analysis is aimed
at guiding future research on similar architectures based on the predictive
coding theory.

deep learning, convolutional neural networks, video classification, video
prediction, semi-supervised, predictive coding

††journalyear: 2020††copyright: acmcopyright††conference: Proceedings of the
2020 International Conference on Multimedia Retrieval; June 8–11, 2020;
Dublin, Ireland††booktitle: Proceedings of the 2020 International Conference
on Multimedia Retrieval (ICMR ’20), June 8–11, 2020, Dublin, Ireland††price:
15.00††doi: 10.1145/3372278.3390694††isbn: 978-1-4503-7087-5/20/06††ccs:
Computing methodologies Machine learning††ccs: Computing methodologies Semi-
supervised learning settings††ccs: Computing methodologies Hierarchical
representations††ccs: Computing methodologies Computer vision problems

##  1\. Introduction

Learning a model of the visual world is a crucial prerequisite to reliably
perform computer vision tasks like object detection and semantic segmentation.
As illustrated by (Jing and Tian, 2019), self-supervised learning allows to
extract this complex structure of the real-world without a need for expensive
labeled data. Videos contain information about how scenes evolve in time and
therefore predicting the future frames of a video is one popular method (Finn
et al., 2016)(Mathieu et al., 2015)(Vondrick et al., 2015)(Wang et al., 2016)
(Wang et al., 2018) of extracting this structure in a self-supervised manner.
Previous research (Lotter et al., 2016)(Mathieu et al., 2015)(Srivastava et
al., 2015) has hypothesized that to accurately predict how the visual world
changes, a model should learn about the object structure and the possible
transformations objects can undergo. Among the various video prediction
models, PredNet by Lotter et al.(Lotter et al., 2016) achieves high video
prediction accuracy with the additional benefit of using a biologically
plausible architecture.

The PredNet architecture is inspired by the predictive coding theory from the
neuroscience literature (Friston, 2009)(Rao and H. Ballard, 1999)(Spratling,
2017) and attempts to implement it with deep neural networks. Predictive
coding is a promising self-supervised learning technique and has shown to
replicate some of the neuronal behavior seen in the mammalian visual cortex.
It posits that the brain is continually making predictions of incoming sensory
stimuli and uses the deviations from these predictions as a learning signal.
It describes hierarchical networks consisting of top-down connections that
carry predictions from higher to lower levels and bottom-up connections that
carry sensory evidence from lower to higher levels at each layer. The error in
prediction is propagated upwards, eventually leading to better predictions in
the future.

In this paper, we first evaluate PredNet’s performance on video prediction by
testing it on a demanding dataset. Then we examine its capability to learn
latent features that are useful for downstream tasks. Specifically, our
contributions in this paper are two-fold:

  1. (1)

Using visualization techniques and experiments, we review PredNet as an
emulation of the predictive coding framework and as a video prediction model.

  2. (2)

We test the features extracted by PredNet by training it in a semi-supervised
setup to perform video action classification. We also evaluate if the
conditioning of top-down predictions on action classes of the video and vice
versa improves the model’s accuracy. We further test this hypothesis on a
simple synthetic dataset by conditioning the predictions of PredNet with
informative top-down class labels.

The paper is organized as follows: Section 2 reviews predictive coding and its
implementations. Section 3 describes our experiment setup, namely our data,
the architecture and the evaluation metrics used. Section 4 is dedicated to
the first phase of our experiments, listing our observations while testing
PredNet. Section 5 gives details on the second phase, the implementation and
evaluation of PredNet+, our proposed extension of the architecture, and in
Section 6 we conclude the paper and list possible directions for future
research.

##  2\. Related work

Rao and Ballard (Rao and H. Ballard, 1999) replicated the ‘extra-classical’
receptive field effects observed in the early stages of cat and monkey visual
cortex with an artificial predictive coding network. These observable effects
are a direct result of the brain trying to efficiently encode sensory data
using prediction. This was accompanied by a rich body of work in neuroscience
(Clark, 2013)(Emberson et al., 2015)(Friston, 2009)(Spratling,
2017)(Summerfield and Koechlin, 2008) and computational modelling (Chalasani
and Principe, 2013)(Lotter et al., 2016)(Song et al., 2019) that explored
different interpretations and implementations of the basic idea, not only in
the visual domain but also in various sensory-motor domains.

Following the success of deep learning in the last decade, many researchers
have attempted to implement the predictive coding model using deep learning
(Han et al., 2018) (Lotter et al., 2016) (van den Oord et al., 2018) (Wen et
al., 2018). Wen et al. (Wen et al., 2018) use predictive coding on static
images to learn optimal feature vectors at each layer for object recognition.
Han et al. (Han et al., 2018) build on this to develop a bidirectional and
dynamic neural network with local recurrent processing. A. Oord et al. (van
den Oord et al., 2018) perform predictive coding in a latent space and use a
probabilistic contrastive loss to learn useful representations. Lotter et. al.
(Lotter et al., 2016) design a video prediction network using the principles
of predictive coding. We chose to evaluate Lotter et al.’s PredNet
architecture for two main reasons: First, it is designed to be structurally
close to the biological predictive coding model with its hierarchical
structure, bottom-up error propagation, and top-down predictions. Second,
PredNet achieves accuracy on-par with state-of-the-art video prediction models
and is a popular baseline used across different spatio-temporal prediction
tasks.

Zhong et al. (Zhong et al., 2018a) extend PredNet into AFA-PredNet within the
robotics domain. They integrate the motor actions of a robot as an additional
signal to condition the top-down generative process. Following this, they
design MTA-PredNet (Zhong et al., 2018b) that has different temporal scales at
different layers in the hierarchy. They developed MTA-PredNet to compensate
for PredNet’s inability to perform reliable long-term predictions which is a
necessity in robotics for planning. Furthermore, researchers have tried to
improve PredNet by adding skip-connections alongside error propagation (Sato
et al., 2018), reducing the number of parameters by using fewer gates in the
top-down ConvLSTM units (Elsayed et al., 2019) and also using inception-type
units within each PredNet layer (Hosseini et al., 2019). Sato et al.(Sato et
al., 2018) evaluate PredNet on weather precipitation dataset and Watanabe et
al. (Watanabe et al., 2018) test PredNet’s response to visual illusions to
examine whether predictive coding models respond to visual illusions just as
humans do. However, none of the above work critically evaluates PredNet as an
implementation of predictive coding and as a reliable self-supervised
pretraining method. Our work aims to provide this in the form of a critical
review of PredNet for future works that intend to use the architecture or
design architectures inspired by it.

##  3\. Experiment setup

###  3.1. Dataset

Most existing large scale video classification datasets have coarse-grained
labels(Heilbron et al., 2015)(Karpathy et al., 2014)(Kuehne et al., 2011).
This means that the models are trained on a relatively easy task and the label
can be detected even from isolated frames, e.g. the ‘soccer’ label can be
inferred from a green field. To overcome this issue and force models to learn
better representations, the Something-something dataset (Goyal et al., 2017)
was introduced. This dataset contains 220,000 videos with 174 fine-grained
action labels. For instance, ‘putting something on a table’, ‘pretending to
put something on a table’, and ‘putting something on a slanted surface so it
slides down’ are three different label classes in the dataset. Mahdisoltani et
al. (Mahdisoltani et al., 2018) provide evidence for the hypothesis that task
granularity is strongly correlated with the quality and generalizability of
learned features. As for the nature of the data, being crowd-sourced, it
includes noise much resembling the real world: thousands of different objects,
variations of lighting conditions, background patterns and camera motion.

###  3.2. PredNet architecture

The PredNet architecture is shown in Figure 1 (Lotter et al., 2016). The
network is composed of stacked hierarchical layers, each of which attempts to
make local predictions of its input. The difference between the actual input
and this prediction is then passed up the hierarchy to the next layer.
Information flows in three ways through the network: (1) the error signal
flows in the bottom-up direction as marked by the red arrows on the right, (2)
the prediction signal flows in the top-down direction as shown by the green
arrow on the left, and (3) the local error signal and prediction estimation
signal flow within each layer. Every layer consists of four units: an input
convolution unit ($A_{i}$), a recurrent representation unit ($R_{i}$) followed
by a prediction unit ($Ahat_{i}$) and an error calculation unit ($E_{i}$) as
labelled in Figure 1. The representation unit is made of a ConvLSTM (Shi et
al., 2015) layer that estimates what the input will be on the next time step.
This input is then fed into the prediction unit that generates the prediction
$Ahat_{i}$. The error units calculate the difference between the prediction
and the input which is fed as input to the next layer. The representation unit
receives a copy of the error signal (red arrow) along with the up-sampled
input from the representation unit of the higher-level (green arrow), which it
uses along with its recurrent memory to perform future predictions.

Figure 1. The original PredNet architecture by Lotter et al. (Lotter et al.,
2016).

###  3.3. Evaluation metrics

Defining a good evaluation metric for the quality of image predictions is a
challenging task by itself (De and Masilamani, 2013)(Mathieu et al., 2015).
There is no universally accepted measurement of image quality and consequently
for image similarity. For the video prediction task, we employ the two
commonly used metrics in literature: Peak Signal Noise Ratio (PSNR) (Mathieu
et al., 2015) and the Structural Similarity Index Measure (SSIM) (Wang et al.,
2004). Like Mathieu et al. (Mathieu et al., 2015), we calculate PSNR and SSIM
only for the frames which have movement with respect to the previous frame and
call them ‘PSNR movement’ and ‘SSIM movement’ respectively. In our case this
is crucial as action videos often contain very few frames with movement and a
metric should not reward a model for simply predicting a still frame. We use a
third metric called ‘conditioned SSIM’, which is calculated as given in
Equation 1. This metric quantifies how different the predictions are from the
previous frame and therefore measures how ‘risky’ the predictions of our model
are in comparison to simply performing a ‘last-frame-copy’.

(1) $SSIM_{cond}=(SSIM_{max}-SSIM(actual_{t-1},pred_{t}))\\\
*SSIM(actual_{t},pred_{t})$

##  4\. Probing PredNet

In the first phase, we review PredNet by evaluating its performance on the
Something-something dataset and visualize different states of the
architecture. For our experiments we use 10 different hyperparameters settings
with a different number of layers, channels per layer, input image size and
frames-per-second (FPS) of the video. These settings are listed in Table 1.
Along with the predicted frame, we visualize the different states of PredNet
at each layer by averaging the activation of all channels in a layer, similar
to Han et al. (Han et al., 2018). We also plot the mean of the error signals
$E_{i}$ and representations $R_{i}$ of every layer to visualize how they
evolve over the span of the video. A sample video with visualizations is shown
in Figure 2. In the following section, we dedicate one paragraph to each of
our findings, and Figure 2 and Table 1 provide further details.

Figure 2. Visualization of evaluated time-specific model states during next frame prediction. Each column corresponds to a single time step, while rows resemble the computed states in each layer. Figure 3. Example of a video with multiple possible future states. Figure 4. Example of a low FPS video and the predictions made by PredNet. Figure 5. Comparison of model performance with respect to the employed frames-per-second rate (FPS). SSIM and PRNS scores show the model’s improvement on a last-frame-copy baseline model. | Frame |  | Image | Number |
---|---|---|---|---|---
Model | rate | Layers | size | of | Loss
| (FPS) |  | (pixel) | param. |
0 | 3 | 4 | 48 X 56 | 6.9 | $L_{0}$
1 | 6 | 4 | 48 X 56 | 6.9 | $L_{0}$
2 | 12 | 4 | 48 X 56 | 6.9 | $L_{0}$
3 | 12 | 4 | 32 X 48 | 6.9 | $L_{0}$
4 | 12 | 5 | 48 X 80 | 5.3 | $L_{0}$
5 | 12 | 6 | 64 X 96 | 5.8 | $L_{0}$
6 | 12 | 7 | 128 X 192 | 6.2 | $L_{0}$
7 | 12 | 6 | 96 X 160 | 7.2 | $L_{0}$
8 | 12 | 5 | 48 X 80 | 5.3 | $L_{all}$
9 | 12 | 6 | 64 X 96 | 5.8 | $L_{all}$
Table 1.  Results obtained from different frame rates, number of layers and
model parameters (in millions) in the model. Listed are the image size and
whether the model was trained with $L_{0}$ loss or $L_{all}$ loss. Similar
models are grouped with horizontal lines and the column that varies is marked
in bold.

###  4.1. Observations

Comparing the frame predictions with input frames in Figure 2, Figure 3 and
Figure 4, we can summarize the working dynamics of PredNet on the action
classification dataset as follows: The model performs previous-frame-copy if
there are no cues for motion in the previous two frames. If there is a cue for
motion and if the direction of the motion is continuous and the motion is
smooth, it interpolates the object in the direction of the motion. Otherwise,
it blurs the region containing the object of motion to keep the L2 loss
minimal by virtue of regression-to-the-mean. The blurring is a result of
PredNet’s inability to learn multi-modal predictions in the sense that it
learns to perform one ideal prediction. It is a typical characteristic of
action-based video sequences that there are multiple possible future states.
For instance, as seen in Figure 3, the thumb can move up or move down or not
move at all in the next frame. The blurring behaviour of PredNet is further
characterized by the experiment we conducted with different sharpness
measures. The predictions by the model are always less sharp than the actual
videos.

PredNet learns relevant features only when trained on videos with continuous
motion. The authors (Lotter et al., 2016) designed and tested PredNet on
videos with continuous motion, such as the KITTI dataset (Geiger et al., 2013)
and their synthetic ‘Rotating Faces’ dataset. This is in stark contrast with
our action dataset which can have a lot of still frames, see e.g. Figure 4. In
this scenario, PredNet resolves to mere last-frame copying, as it is
statistically beneficial to do just that. If the model is not motivated enough
to learn the dynamics of how objects move and scenes evolve then the features
it learns would not be useful for downstream tasks as hypothesized by Lotter
et al. (Lotter et al., 2016).

PredNet’s learning ability is sensitive to the frames-per-second (FPS) rate.
When we compare the performance of models trained on videos with FPS rates 3,
6 and 12 in Figure 5, we can see that the performance varies greatly. In this
and all following figures, we show the improvement in the given evaluation
metric, meaning the difference in the value of the evaluation metric between
the model being evaluated and a baseline model performing last-frame copying,
i.e. how much improvement the given PredNet model shows to simple last-frame
copying. Manual inspection of the predictions further confirms the large
difference in prediction quality. At very high FPS there is minimal motion
between two consecutive frames while at low FPS rates there is abrupt movement
between frames which is challenging to predict. In both of these scenarios,
the model completely resorts to last-frame-copy. Therefore, the FPS of the
video is one of the most important hyperparameters of PredNet.

Two key insights indicate that PredNet is not a comprehensive emulation of
hierarchical predictive coding. Firstly, from the ”mean E activation” plot in
Figure 2, it is evident that the mean bottom-up error increases as one goes up
towards higher layers. This behavior can be observed in all sample videos
being visualized. This is in contrary to the expectations of predictive
coding, which posits that the error decreases as we go up the hierarchy as
parts of the incoming signal are iteratively ‘explained away.’ Secondly,
Lotter et al. (Lotter et al., 2016) demonstrate that models trained with
$L_{0}$ loss perform better than $L_{all}$ loss on the KITTI data. We cross-
check this on our dataset and get similar results. As shown in Figure 6,
models with $L_{0}$ loss perform better on all metrics. Training with $L_{0}$
loss implies that we only minimize the error $E_{0}$ on the lowest layer (see
Figure 1) while in $L_{all}$ loss the model is trained to minimize the
prediction errors in all the layers. Predictive coding suggests that each
layer in the hierarchy minimizes the error signal iteratively. Therefore, an
accurate implementation of predictive coding should improve results when
trained with $L_{all}$ loss. Furthermore, the visualization of mean activation
of PredNet’s states at different layers in Figure 2 shows that the states in
the lowest layer are different from all the higher layers. This indicates that
the model operates with two sub-modules when trained with the $L_{0}$ loss:
the lowest layer $R_{0}$ aims to generate realistic $(t+1)$ predictions, while
the rest of the layers operate as one deep network that regresses $E_{0}$ to
generate the context $R_{1}$. This is also indicated by the fact that the mean
R activation for the lowest layer is higher and follows a different trajectory
than the rest of the layers in Figure 2. We further evaluate PredNet by
examining its ability to extrapolate and predict longer time steps into the
future. As explained in Lotter et al. (Lotter et al., 2016), PredNet can be
used to generate long-term predictions by simply feeding its predictions at
time $t$ back in as input at the next time step $(t+1)$. This can be done
iteratively for $n$ time steps to get a $(t+n)$ prediction into the future. We
test the extrapolation capability of PredNet models that are trained only to
perform $(t+1)$ predictions as well as PredNet models that are trained
exclusively to perform $(t+n)$ predictions. As expected, the results are
marginally better in the latter case as also demonstrated in Lotter et al.
(Lotter et al., 2016). The extrapolated predictions of our best performing
model are given in Figure 7. The extrapolation is started at different time
points in the video as shown by the red marker in the figure. The following
three observations can be made from the above experiments (1) After two-time
steps, the model resorts to last-frame-copying. As already discussed, PredNet
performs predictions by using the movement between consecutive input frames as
an active cue. Therefore while extrapolating, when we feed the predictions
back as input, the model gets a cue that the action has stopped and reverted
to performing last-frame-copying. (2) The predictions get blurrier over time.
This is because the minor blur added by the down-sampling units in the bottom-
up and up-sampling units in the top-down accumulates exponentially with time.
(3) From the metrics in Figure 8 we can infer that the models perform better
if the extrapolation is started in the later stages of the video. This can be
due to the fact that in our dataset, motion generally starts in the middle or
towards the end of the video. In conclusion, the extrapolation experiments
suggest that the network design compels it to learn just short-term
interpolations instead of building long-term predictions.

Finally, we notice that the model delivers improved predictions only when the
topmost layers have a full receptive field. Only this setting allows to
predict object movements instead of just blurring local regions of motion. The
receptive field can be increased either by using deeper layers or by
increasing the kernel size of the convolutions or even by reducing the image
size. We experimented with each of these and found that the prediction scores
improve with the increased receptive field. We show the results of
experimenting with a different number of layers in Figure 9 and it is apparent
that the prediction quality improves with increasing depth.

Figure 6. Influence of $L_{0}$ and $L_{all}$ loss on model performance. SSIM
and PRNS scores show the model’s improvement on a last-frame-copy baseline
model. Figure 7. Extrapolation results for Model 7 extrapolated at different
time steps. The red mark shows the start of the extrapolation. Figure 8.
Comparison of the metrics when starting extrapolation at different stages in
the video. $t$ denotes the total number of frames in the video. SSIM and PRNS
scores show the model’s improvement on a last-frame-copy baseline model.
Figure 9. Comparison of performance with models encompassing 4, 5, 6 and 7
layers. SSIM and PRNS scores show the model’s improvement on a last-frame-copy
baseline model.

##  5\. Label classification with PredNet+

In this section, we describe the second phase of experiments, where we test
the architecture by modifying it to perform supervised label classification
simultaneously with video prediction. For a comparison of the architectures of
PredNet+ and the vanilla PredNet, see Figures 10 and 1 respectively. The model
design, the rationale behind the design and the results are discussed next.

###  5.1. PredNet+ Design

We modify the PredNet architecture such that it can perform video label
classification along with next frame prediction and informally call this
architecture PredNet+. The architecture is shown in Figure 10. As seen here,
PredNet+ contains an additional label classification unit that is attached to
the top-most representation layer. It consists of an encoder and a decoder
part. As displayed in Figure 10 the two ConvLSTM layers form the encoder which
transforms the output of the representation unit $R_{3}$ into label class
probabilities. The two transposed convolution layers make up the decoder that
up-samples and transforms the label classes back to the imaging modality which
is fed back into the top down as shown by the black arrow to $R_{2}$.

Figure 10. The proposed PredNet+ architecture with an additional
classification pathway attached to the deepest prediction layer.

The label prediction unit makes predictions at each incoming frame, whose
weighted sum is passed through a softmax function to get the final class
probabilities for the video. As the model does not have enough context to make
meaningful predictions at the beginning of the video, the weighing-over-time
is done using an exponential function.

PredNet+ is designed such that the latent features at the top-most
representation layer are shared between its two tasks. The future frame
predictions are conditioned on the label predictions made by the label
classification unit (shown in Figure 10 by the black arrow going into
$R_{2}$). We hypothesized that this would improve the results on both sub-
tasks as evident in many multi-task training scenarios (Collobert and Weston,
2008) (Girshick, 2015). Even though in our case, we attach the label
classification unit to the top-most layer, this is not the only approach nor
necessarily the best one as per predictive coding. We decided for this setup
because the top-most layer in the model has both a full receptive field and
access to previous states.

In summary, the label classification unit and the prediction units in PredNet+
are expected to work in tandem in a multitask learning set-up and form a
synergy. However, this is not what we observe in our results.

###  5.2. Results

Table 2 shows our best classification accuracy in comparison to the baseline
model scores of Goyal et al. (Goyal et al., 2017) and the current state-of-
the-art results by Mahdisoltani et al. on the Something-something
dataset(Mahdisoltani et al., 2018).We test the PredNet+ architecture on our
best 4 layer model, 5 layer model and 6 layer model from Table 1. Furthermore,
we test the following minor variations of PredNet+ to further evaluate the
model architecture: First, we remove the recurrent memory in the label
classification unit by replacing the ConvLSTM with Convolution layers. Next,
we extend the label classification loss function such that the model is
rewarded for predicting at least the correct verb in the label. For example,
if the correct label is Pretending to put something behind something, the
model is penalized twice as much if it predicts Showing something to the
camera than if it predicts Putting something behind something, which has the
same verb as the correct label. Surprisingly, the classification results do
not change at all (${\pm 0.6}{\%}$) for any of these model variations. This
suggests that the features from the top-most representation unit do not have
any more information.

Model | Top-1 | Top-5
---|---|---
Baseline (Goyal et al., 2017) | 11.5 | 30.0
Ours | 28.2 | 57.0
Mahdisoltani et al. (Mahdisoltani et al., 2018) | 51.38 | -
Table 2. Classification accuracy on the Something-something dataset with 174
label categories.

The label classification scores suggest that PredNet+ is a long way from the
state-of-the-art architectures. Furthermore, the future frame prediction of
PredNet+ degrade in comparison to its equivalent vanilla PredNet models: Model
5 ($L_{0}$ loss) and Model 8 ($L_{all}$ loss). The metrics in Figure 11 and
the visualization of predictions point this out. To further analyze this, we
experiment with different loss weights for the two tasks. This allows us to
control the relative importance of each task for the model during training. We
find that the model’s future prediction quality degrades when the label
classification task is given increased importance, suggesting that the multi-
task constraint leads to worse future frame predictions.

Figure 11. Comparison of the prediction quality of PredNet+ with equivalent
PredNet models. SSIM and PRNS scores show the model’s improvement on a last-
frame-copy baseline model.

###  5.3. Representation learning with top-down conditioning and synthetic
data

In order to further evaluate our hypothesis that conditioning the top-down
predictions on class labels of the video improves model accuracy, we evaluate
PredNet+ on a synthetic dataset. We employ a moving MNIST dataset with a
static background consisting of randomly generated overlapping geometric
shapes, and a single hand-written digit moving in one of eight directions.
Each frame was annotated with a label representing the future direction of the
digit’s movement. Samples of the dataset are given in the upper rows of Figure
12. We test if adding semantic top-down information helps increasing the
prediction score of the network. We thus keep track of spatiotemporal
prediction performance, while using the movement label classification as an
auxiliary task. The generated predictions generally showed lower confidence in
the moving part of the input frames, especially in the first frames, as seen
in the lower row of Figure 12 (a). Predictions made by the model with
additional label classification pathway are presented in Figure 12 (b).

(

a) PredNet

(

Figure 12. Model predictions on moving MNIST. Shown are examples for the
original model and a model with additional classification pathway.

b) PredNet+

The resulting scores generated by a previous-frame-copy model, plain PredNet
and PredNet+ are displayed in Table 3. The multi-task learning with movement
direction classification improves the MAE score and leads to sharper
predictions in the non-stationary parts of the input images. This indicates
that conditioning the top-down predictions with semantic information can
improve model performance, especially when the additional information can be
related directly to predicted features in the input space.

Model | MAE score
---|---
Previous-frame-copy | 8e-050
PredNet | 7.6e-05
PredNet+ | 7.3e-05
Table 3.  Comparison of the predictive performance of the original and multi-
task PredNet model. Scores are given for next-frame prediction on the moving
MNIST dataset.

##  6\. Conclusion and future work

We have evaluated PredNet (Lotter et al., 2016) on a challenging action
classification dataset in two phases.

In the first phase of our work, we investigate PredNet and derive the
following insights: (1) PredNet does not completely follow the principles of
the predictive coding framework. (2) It can perform only short-term next frame
interpolations, rather than long-term video predictions. This has been further
confirmed by the extrapolation experiments. (3) The representation units are
unable to learn multi-modal distributions and produce blurry predictions. (4)
The models’ learning ability is sensitive to the continuity of motion and the
FPS rate of the videos.

In the second phase, we test PredNet’s ability to learn useful latent features
to perform label classification. We use the features from the highest
representation layer and find that this is not adequate for the task at hand,
namely, the prediction of a complex action classification dataset. We achieve
a classification accuracy of 28.2% in comparison to current state-of-the-art
of 51.38% (Mahdisoltani et al., 2018) and the prediction accuracy also under-
performs the vanilla PredNet. In a further step, we experiment on a synthetic
dataset and show that that top-down conditioning can improve the prediction
scores.

Our results lead to several suggestions for improved models: Firstly, the
network should be trainable with $L_{all}$ loss. This can be done by designing
error estimators that are local to each layer. Secondly, the network should be
redesigned such that it is encouraged to perform long-term predictions rather
than just frame-to-frame interpolation. One way to do this is to have
additional layers higher in the hierarchy, that make predictions at different
temporal scales. Additionally, PredNet’s performance metrics show high
variance while PredNet+ is easily susceptible to over-fitting. These points
signal the need for including regularization techniques and model averaging
methods like dropout within the architecture. Finally, the representation
units should learn multi-modal probability distributions, from which
predictions can be sampled. This could be addressed, for example, with
probabilistic representations in some or all layers.

## References

  * (1)
  * Chalasani and Principe (2013) Rakesh Chalasani and Jose C. Principe. 2013.  Deep Predictive Coding Networks.  _arXiv e-prints_ , Article arXiv:1301.3541 (Jan 2013), arXiv:1301.3541 pages.  arXiv:cs.LG/1301.3541
  * Clark (2013) Andy Clark. 2013\.  Whatever next? Predictive brains, situated agents, and the future of cognitive science.  _Behavioral and Brain Sciences_ 36, 3 (2013), 181–204.  https://doi.org/10.1017/S0140525X12000477 arXiv:0140-525X
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In _Proceedings of the 25th International Conference on Machine Learning_ _(ICML ’08)_. ACM, New York, NY, USA, 160–167.  https://doi.org/10.1145/1390156.1390177
  * De and Masilamani (2013) Kanjar De and V Masilamani. 2013.  Image Sharpness Measure for Blurred Images in Frequency Domain.  _Procedia Engineering_ 64 (12 2013).  https://doi.org/10.1016/j.proeng.2013.09.086
  * Elsayed et al. (2019) N. Elsayed, A. S. Maida, and M. Bayoumi. 2019\.  Reduced-Gate Convolutional LSTM Architecture for Next-Frame Video Prediction Using Predictive Coding. In _2019 International Joint Conference on Neural Networks (IJCNN)_. 1–9.  https://doi.org/10.1109/IJCNN.2019.8852480
  * Emberson et al. (2015) Lauren L. Emberson, John E. Richards, and Richard N. Aslin. 2015.  Top-down modulation in the infant brain: Learning-induced expectations rapidly affect the sensory cortex at 6 months.  _Proceedings of the National Academy of Sciences_ 112, 31 (2015), 9585–9590.  https://doi.org/10.1073/pnas.1510343112 arXiv:https://www.pnas.org/content/112/31/9585.full.pdf
  * Finn et al. (2016) Chelsea Finn, Ian Goodfellow, and Sergey Levine. 2016\.  Unsupervised Learning for Physical Interaction through Video Prediction.  In _Advances in Neural Information Processing Systems 29_ , D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 64–72.  http://papers.nips.cc/paper/6161-unsupervised-learning-for-physical-interaction-through-video-prediction.pdf
  * Friston (2009) Karl Friston. 2009\.  The free-energy principle: a rough guide to the brain?  _Trends in Cognitive Sciences_ 13, 7 (2009), 293 – 301.  https://doi.org/10.1016/j.tics.2009.04.005
  * Geiger et al. (2013) Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013.  Vision meets Robotics: The KITTI Dataset.  _International Journal of Robotics Research (IJRR)_ (2013).
  * Girshick (2015) Ross B. Girshick. 2015\.  Fast R-CNN.  _CoRR_ abs/1504.08083 (2015).  arXiv:1504.08083 http://arxiv.org/abs/1504.08083
  * Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017.  The ”something something” video database for learning and evaluating visual common sense.  _CoRR_ abs/1706.04261 (2017).  arXiv:1706.04261 http://arxiv.org/abs/1706.04261
  * Han et al. (2018) Kuan Han, Haiguang Wen, Yizhen Zhang, Di Fu, Eugenio Culurciello, and Zhongming Liu. 2018.  Deep Predictive Coding Network with Local Recurrent Processing for Object Recognition.  _CoRR_ abs/1805.07526 (2018).  arXiv:1805.07526 http://arxiv.org/abs/1805.07526
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  ActivityNet: A large-scale video benchmark for human activity understanding.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), 961–970.
  * Hosseini et al. (2019) Matin Hosseini, Anthony S. Maida, Majid Hosseini, and Gottumukkala Raju. 2019\.  Inception-inspired LSTM for Next-frame Video Prediction.  _arXiv e-prints_ , Article arXiv:1909.05622 (Aug 2019), arXiv:1909.05622 pages.  arXiv:cs.CV/1909.05622
  * Jing and Tian (2019) Longlong Jing and Yingli Tian. 2019.  Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey.  _CoRR_ abs/1902.06162 (2019).  arXiv:1902.06162 http://arxiv.org/abs/1902.06162
  * Karpathy et al. (2014) A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. 2014\.  Large-Scale Video Classification with Convolutional Neural Networks. In _2014 IEEE Conference on Computer Vision and Pattern Recognition_. 1725–1732.  https://doi.org/10.1109/CVPR.2014.223
  * Kuehne et al. (2011) H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. 2011.  HMDB: A Large Video Database for Human Motion Recognition. In _Proceedings of the 2011 International Conference on Computer Vision_ _(ICCV ’11)_. IEEE Computer Society, Washington, DC, USA, 2556–2563.  https://doi.org/10.1109/ICCV.2011.6126543
  * Lotter et al. (2016) William Lotter, Gabriel Kreiman, and David D. Cox. 2016\.  Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning.  _CoRR_ abs/1605.08104 (2016).  arXiv:1605.08104 http://arxiv.org/abs/1605.08104
  * Mahdisoltani et al. (2018) Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbieh, David J. Fleet, and Roland Memisevic. 2018\.  Fine-grained Video Classification and Captioning.  _CoRR_ abs/1804.09235 (2018).  arXiv:1804.09235 http://arxiv.org/abs/1804.09235
  * Mathieu et al. (2015) Michaël Mathieu, Camille Couprie, and Yann LeCun. 2015\.  Deep multi-scale video prediction beyond mean square error.  _CoRR_ abs/1511.05440 (2015).  arXiv:1511.05440 http://arxiv.org/abs/1511.05440
  * Rao and H. Ballard (1999) Rajesh Rao and Dana H. Ballard. 1999.  Predictive Coding in the Visual Cortex: a Functional Interpretation of Some Extra-classical Receptive-field Effects.  _Nature neuroscience_ 2 (02 1999), 79–87.  https://doi.org/10.1038/4580
  * Sato et al. (2018) Ryoma Sato, Hisashi Kashima, and Takehiro Yamamoto. 2018\.  Short-Term Precipitation Prediction with Skip-Connected PredNet. In _Artificial Neural Networks and Machine Learning – ICANN 2018_ , Věra Kůrková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis (Eds.). Springer International Publishing, Cham, 373–382.
  * Shi et al. (2015) Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015\.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting.  _CoRR_ abs/1506.04214 (2015).  arXiv:1506.04214 http://arxiv.org/abs/1506.04214
  * Song et al. (2019) Z. Song, J. Zhang, G. Shi, and J. Liu. 2019\.  Fast Inference Predictive Coding: A Novel Model for Constructing Deep Neural Networks.  _IEEE Transactions on Neural Networks and Learning Systems_ 30, 4 (April 2019), 1150–1165.  https://doi.org/10.1109/TNNLS.2018.2862866
  * Spratling (2017) M.W. Spratling. 2017\.  A review of predictive coding algorithms.  _Brain and Cognition_ 112 (2017), 92 – 97.  https://doi.org/10.1016/j.bandc.2015.11.003 Perspectives on Human Probabilistic Inferences and the ’Bayesian Brain’.
  * Srivastava et al. (2015) Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. 2015\.  Unsupervised Learning of Video Representations using LSTMs.  _CoRR_ abs/1502.04681 (2015).  arXiv:1502.04681 http://arxiv.org/abs/1502.04681
  * Summerfield and Koechlin (2008) Christopher Summerfield and Etienne Koechlin. 2008.  A Neural Representation of Prior Information during Perceptual Inference.  _Neuron_ 59, 2 (2008), 336 – 347.  https://doi.org/10.1016/j.neuron.2008.05.021
  * van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018\.  Representation Learning with Contrastive Predictive Coding.  _arXiv e-prints_ , Article arXiv:1807.03748 (Jul 2018), arXiv:1807.03748 pages.  arXiv:cs.LG/1807.03748
  * Vondrick et al. (2015) Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2015\.  Anticipating the future by watching unlabeled video.  _CoRR_ abs/1504.08023 (2015).  arXiv:1504.08023 http://arxiv.org/abs/1504.08023
  * Wang et al. (2016) Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition.  _CoRR_ abs/1608.00859 (2016).  arXiv:1608.00859 http://arxiv.org/abs/1608.00859
  * Wang et al. (2018) Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. 2018.  PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning.  _CoRR_ abs/1804.06300 (2018).  arXiv:1804.06300 http://arxiv.org/abs/1804.06300
  * Wang et al. (2004) Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004.  Image Quality Assessment: From Error Visibility to Structural Similarity.  _IEEE TRANSACTIONS ON IMAGE PROCESSING_ 13, 4 (2004), 600–612.
  * Watanabe et al. (2018) Eiji Watanabe, Akiyoshi Kitaoka, Kiwako Sakamoto, Masaki Yasugi, and Kenta Tanaka. 2018.  Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction.  _Frontiers in Psychology_ 9 (2018), 345.  https://doi.org/10.3389/fpsyg.2018.00345
  * Wen et al. (2018) Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Eugenio Culurciello, and Zhongming Liu. 2018.  Deep Predictive Coding Network for Object Recognition.  _CoRR_ abs/1802.04762 (2018).  arXiv:1802.04762 http://arxiv.org/abs/1802.04762
  * Zhong et al. (2018a) Junpei Zhong, Angelo Cangelosi, Xinzheng Zhang, and Tetsuya Ogata. 2018a.  AFA-PredNet: The action modulation within predictive coding.  _CoRR_ abs/1804.03826 (2018).  arXiv:1804.03826 http://arxiv.org/abs/1804.03826
  * Zhong et al. (2018b) Junpei Zhong, Tetsuya Ogata, and Angelo Cangelosi. 2018b.  Encoding Longer-term Contextual Multi-modal Information in a Predictive Coding Model.  _CoRR_ abs/1804.06774 (2018).  arXiv:1804.06774 http://arxiv.org/abs/1804.06774