# Long-horizon Video Prediction
using a Dynamic Latent Hierarchy

Alexey Zakharov1, Qinghai Guo2, Zafeirios Fountas1
Huawei Technologies R&D
1 London, UK
2 Shenzhen, China
<EMAIL_ADDRESS>
###### Abstract

The task of video prediction and generation is known to be notoriously
difficult, with the research in this area largely limited to short-term
predictions. Though plagued with noise and stochasticity, videos consist of
features that are organised in a spatiotemporal hierarchy, different features
possessing different temporal dynamics. In this paper, we introduce Dynamic
Latent Hierarchy (DLH) – a deep hierarchical latent model that represents
videos as a hierarchy of latent states that evolve over separate and fluid
timescales. Each latent state is a mixture distribution with two components,
representing the immediate past and the predicted future, causing the model to
learn transitions only between sufficiently dissimilar states, while
clustering temporally persistent states closer together. Using this unique
property, DLH naturally discovers the spatiotemporal structure of a dataset
and learns disentangled representations across its hierarchy. We hypothesise
that this simplifies the task of modeling temporal dynamics of a video,
improves the learning of long-term dependencies, and reduces error
accumulation. As evidence, we demonstrate that DLH outperforms state-of-the-
art benchmarks in video prediction, is able to better represent stochasticity,
as well as to dynamically adjust its hierarchical and temporal structure. Our
paper shows, among other things, how progress in representation learning can
translate into progress in prediction tasks.

##  1 Introduction

Video data is considered to be one of the most difficult modalities for
generative modelling and prediction, characterised by high levels of noise,
complex temporal dynamics, and inherent stochasticity. Even more so, modelling
long-term videos poses a significant challenge due to the problem of
sequential error accumulation, largely restricting the research in this topic
to short-term predictions.

Deep learning has given rise to generative latent-variable models with the
capability to learn rich latent representations, allowing to model high-
dimensional data by means of more efficient, lower-dimensional states (Kingma
& Welling, 2013; Higgins et al., 2022; Vahdat & Kautz, 2020; Rasmus et al.,
2015). Here, of particular interest are hierarchical latent models, which
possess a higher degree of representational power and expressivity. Employing
hierarchies has so far proved to be an effective method for generating high-
fidelity visual data, as well as concurrently producing more meaningful and
disentangled latent representations in both static (Vahdat & Kautz, 2020) and
temporal (Zakharov et al., 2022) datasets.

Unlike images, videos possess a spatiotemporal structure, in which a
collection of spatial features adhere to the intrinsic temporal dynamics of a
dataset – often evolving at different and fluid timescales. For instance,
consider a simplistic example shown in Figure 1, in which the features of a
video sequence evolve within a strict temporal hierarchy: from the panda
continuously changing its position to the background elements being static
over the entire duration of the video.

Figure 1: Videos can be viewed as a collection of features organised in a
spatiotemporal hierarchy. This graphic illustrates a sequence of frames, in
which the components of the video possess different temporal dynamics (white
circles indicate feature changes). Notice the irregularities in their dynamics
– the panda continuously changes its position, the airplane is seen only for a
few timesteps, while the background remains static throughout. Similar to
this, our model learns hierarchically disentangled representations of video
features with the ability to model their unique dynamics.

Discovering such a temporal structure in videos complements nicely the
research into hierarchical generative models, which have been shown capable of
extracting and disentangling features across a hierarchy of latent states.
Relying on this notion of inherent spatiotemporal organisation of features,
several hierarchical architectures have been proposed to either enforce a
generative temporal hierarchy explicitly (Saxena et al., 2021), or discover it
in an unsupervised fashion (Kim et al., 2019; Zakharov et al., 2022). In
general, these architectures consist of a collection of latent states that
transition over different timescales, which has been shown to benefit long-
term predictions (Saxena et al., 2021; Zakharov et al., 2022).

Building upon these notions, we propose an architecture of a hierarchical
generative model for long-horizon video prediction – Dynamic Latent Hierarchy
(DLH). The principle ideas underlying this work are two-fold. First, we posit
that learning disentangled hierarchical representations and their separate
temporal dynamics increases the model’s expressivity and breaks down the
problem of video modelling into simpler sub-problems, thus benefiting
prediction quality. As such, our model is capable of discovering the
appropriate hierarchical spatiotemporal structure of the dataset, seamlessly
adapting its generative structure to a dataset’s dynamics. Second, the
existence of a spatiotemporal hierarchy, in which some features can remain
static for an arbitrary period of time (e.g. background in Fig. 1), implies
that predicting the next state at every timestep may introduce unnecessary
accumulation of error and computational complexity. Instead, our model learns
to transition between states only if a change in the represented features has
been observed (e.g. airplane in Fig. 1). Conversely, if no change in the
features has been detected, the model clusters such temporally-persistent
states closer together, thus building a more organised latent space. Our
contributions are summarised as follows:

  * •

A novel architecture of a hierarchical latent-variable generative model
employing temporal Gaussian mixtures (GM) for representing latent states and
their dynamics;

  * •

Incorporation of a non-parametric inference method for estimating the discrete
posterior distribution over the temporal GM components;

  * •

The resulting superior long-horizon video prediction performance, emergent
hierarchical disentanglement properties, and improved stochasticity
representation.

##  2 Dynamic Latent Hierarchy

We propose an architecture of a hierarchical latent model for video prediction
– Dynamic Latent Hierarchy. DLH consists of a hierarchy of latent states that
evolve over different and flexible timescales. Each latent state is a mixture
of two Gaussian components that represent the immediate past and the predicted
future in a single belief state. Using this formalisation, the model
dynamically assigns every newly inferred posterior state to one of these
clusters, and thus implicitly learns the temporal hierarchy of the data in an
unsupervised fashion.

###  2.1 Generative model

We consider sequences of observations,
$\\{\textbf{o}_{1},...,\textbf{o}_{T}\\}$, modelled by a hierarchical
generative model with a joint distribution in the form (Fig. 2),

$\displaystyle\prod_{t=1}^{T}p_{\theta}(\textbf{o}_{t},\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})=\prod_{t=1}^{T}p_{\theta}(\textbf{o}_{t}|\vec{\textbf{s}}_{t})\prod_{n=1}^{N}p_{\theta}(\textbf{s}^{n}_{t}|\underbrace{\textbf{e}^{n}_{t}}_{\text{indicator}},\underbrace{\textbf{s}^{n}_{<t}}_{\text{temporal}},\underbrace{\textbf{s}^{>n}_{t}}_{\text{hierarchical}})p_{\theta}(\textbf{e}^{n}_{t}|\textbf{s}^{n}_{<t}),$
(1)

where $\textbf{s}^{n}_{t}\sim\mathcal{N}(\cdot,\cdot)$ is a diagonal Gaussian
latent state, $\textbf{e}^{n}_{t}\sim\text{Ber}(\cdot)$ is the corresponding
Bernoulli variable at a hierarchical level $n$ and timestep $t$,
$\vec{\textbf{s}}=\\{\textbf{s}^{1},...,\textbf{s}^{N}\\}$ and
$\vec{\textbf{e}}=\\{\textbf{e}^{1},...,\textbf{e}^{N}\\}$ denote collections
of all variables in a hierarchy, and $\theta$ are the parameters of the model.
Notice that each state $\textbf{s}^{n}_{t}$ is conditioned on all of the
hierarchical states above, past states in the same level, and an indicator
variable $\textbf{e}^{n}_{t}$.

One of the key features of DLH is the representation of a latent state as a
temporal mixture of Gaussians (MoG). In particular, variables
$\textbf{s}^{n}_{t}$ and $\textbf{e}^{n}_{t}$ together define a MoG,
$p(\textbf{s}^{n}_{t},\textbf{e}^{n}_{t})=p(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t})p(\textbf{e}^{n}_{t})$111Stripping
away the hierarchical and temporal conditioning for clarity., with just two
components such that,

$\displaystyle
p(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t})=\begin{cases}p(\textbf{s}^{n}_{t-1})&\text{if
}\textbf{e}^{n}_{t}=0,\text{ (previous state: {static prior})}\\\
p_{\theta}(\textbf{s}^{n}_{t}|\textbf{s}^{n}_{<t})&\text{if
}\textbf{e}^{n}_{t}=1,\text{ (predicted state: {change prior})}.\end{cases}$
(2)

As such, at every timestep, DLH holds two prior beliefs over the state of the
world: (1) it can remain static, or (2) it can progress through time and thus
change. In this view, variable $\textbf{e}^{n}_{t}$ can be informally
described as the probability of whether state $\textbf{s}^{n}_{t}$ should be
updated or remain fixed at timestep $t$ (Fig. 3). This property allows DLH to
model data as a collection of hierarchical latent states that evolve over
different and flexible timescales, determined by the indicator variable
$\textbf{e}^{n}$.

Figure 2: (a) Generative model of DLH. (b) Inference models of DLH. (c)
Architectural components of DLH, showing the deterministic variables that
correspond to bottom-up, top-down, and temporal information. (d) Example of a
two-level DLH model rolled-out over three timesteps.

###  2.2 Inference

In order to train the model using a variational lower bound, we must estimate
the posterior distribution
$q(\textbf{s}^{n}_{t},\textbf{e}^{n}_{t}|\textbf{o}_{t})$, for which we assume
a mean-field factorisation $q(\textbf{s}^{n}_{t})q(\textbf{e}^{n}_{t})$;
therefore, the two distributions are approximated separately (Fig. 2b).

#### Estimating $q(\textbf{s})$

In DLH, posterior $q(\textbf{s}^{n}_{t})$ is assumed to be a diagonal
Gaussian, amortised using a neural network
$q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})$ with
parameters $\psi$, conditioned on hierarchical states above and the latest
data point $\textbf{o}_{t}$. In line with the established procedure, the
approximate posterior is trained using the reparametrisation trick (Kingma &
Welling, 2013).

#### Estimating $q(\textbf{e})$

Using reparametrisation tricks for discrete latent variables poses a
significant challenge for a stable training procedure of deep learning models,
which can be further exacerbated in hierarchical models (Falck et al., 2021).
To avoid this, we estimate $q(\textbf{e})$ using a non-parametric method.

Inferring distribution $q(\textbf{e}^{n})$ can be conceptualised as a
clustering problem of $q(\textbf{s}^{n})$ with respect to the static and
change priors of the model, with the central question being: under which
temporal mixture component in Eq. 2 is the inferred state most likely? Has the
state of the world changed or has it remained the same? As such , we formulate
the approximation of $p(\textbf{e}^{n}|\textbf{s}^{n})$ as model selection
using expected likelihood ratio, where the two components of the MoG (Eq. 2)
are the competing models. Under the inferred state,
$q^{\prime}(\textbf{s}^{n}_{t})=q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})$,
the expected log-likelihood ratio is,

$\displaystyle\mathbb{E}[\log\Lambda(\textbf{s}^{n}_{t})]=\mathbb{E}_{q^{\prime}(\textbf{s}^{n}_{t})}\log\frac{p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=0)}{p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=1)},$
(3)

where
$p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t})=p_{\theta}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t},\textbf{s}^{n}_{<t},\textbf{s}^{>n}_{t})$
from the definition of the generative model. Assuming the selection of the
most likely component under the inferred posterior, we come to the following
selection criterion (see full derivation and further clarifications in
Appendix B),

$\displaystyle
D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=1)]\text{
}\mathop{\lessgtr}_{\textbf{e}^{n}_{t}=0}^{\textbf{e}^{n}_{t}=1}\text{
}D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=0)],$
(4)

where the most likely component $i$ is approximated to have a probability
$q(\textbf{e}^{n}_{t}=i)=1$. This approximation relates to the VaDE trick,
which is similarly a non-parametric method of estimating the posterior
component variable of a MoG (Jiang et al., 2017; Falck et al., 2021). In
particular, our method can be viewed as taking a sample from the most likely
component of the VaDE-estimated $q(\textbf{e}^{n})$ under the assumption of
equal prior probabilities (see Appendix B.3). Though this method introduces
bias, in practice, we found that it performs better than the VaDE trick. We
hypothesise that this relates to a relatively fast convergence of the
parametrised prior $p_{\theta}(\textbf{e}^{n}_{t}|\cdot)$ model, which becomes
overly confident in its predictions (even before any video features have been
learned), thus irreversibly skewing the approximation of
$q(\textbf{e}^{n}_{t})$. Nevertheless, we believe this direction of future
work may merit further investigation.

###  2.3 Nested timescales

We add a constraint on the hierarchical temporal structure of the generative
model similar to Saxena et al. (2021); Zakharov et al. (2022). In particular,
$q(\textbf{e}^{n+1}=1|\textbf{e}^{n}=0)=0$. Enforcing this constraint has been
shown to be an effective method to promote spatiotemporal disentanglement of
features in hierarchical models, encouraging the representation of
progressively slower features in the higher levels of the model. Furthermore,
to reduce the computational complexity of the model, we block any further
inference above the hierarchical level where $\textbf{e}^{n}=0$ is inferred,
such that:

$\displaystyle\text{if }\textbf{e}^{n-1}_{t}=0,\text{ then
}\textbf{e}^{n}_{t}=0\text{ {and} }q(\textbf{s}^{n}_{t})\leftarrow
q(\textbf{s}^{n}_{t-1}).$ (5)

Lastly, to model continuously changing videos, we assume
$q(\textbf{e}^{1}=1)=1$, which allows for the bottom level of DLH to always be
in use. It is worth noting that the proposed model constraints may be relaxed
in different implementations of DLH, which could be explored in future work.

###  2.4 Lower bound estimation

To train the model, we derive a variational lower bound (ELBO), for which we
introduce an approximate posterior distribution
$q(\vec{\textbf{s}},\vec{\textbf{e}})$ so that

$\displaystyle\sum^{T}_{t=1}\log p(\textbf{o}_{t})$
$\displaystyle=\sum^{T}_{t=1}\log\int_{\vec{\textbf{s}}}\sum_{\vec{\textbf{e}}}q(\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})\frac{p_{\theta}(\textbf{o}_{t},\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})}{q(\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})}$
(6)
$\displaystyle\geq\sum^{T}_{t=1}\mathbb{E}_{q(\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})}\log
p_{\theta}(\textbf{o}_{t}|\vec{\textbf{s}}_{t})+\mathbb{E}_{q(\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})}\Big{[}\log\frac{p_{\theta}(\vec{\textbf{s}}_{t}|\vec{\textbf{e}}_{t})p_{\theta}(\vec{\textbf{e}}_{t})}{q(\vec{\textbf{s}}_{t},\vec{\textbf{e}}_{t})}\Big{]}.$
(7)

Assuming posterior factorisation of
$q(\textbf{s}^{n}_{t},\textbf{e}^{n}_{t})=q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})q(\textbf{e}^{n}_{t}|\textbf{e}^{n-1}_{t})$,
we write the complete formulation of the ELBO,

$\displaystyle\mathcal{L}_{\text{ELBO}}$
$\displaystyle=\sum_{t=1}^{T}\Big{[}\mathbb{E}_{q_{\psi}(\vec{\textbf{s}}_{t})}\log
p_{\theta}(\textbf{o}_{t}|\vec{\textbf{s}}_{t})\Big{]}$ (8a)
$\displaystyle-\sum_{t=1}^{T}\sum^{N}_{n=1}\Big{[}\mathbb{E}_{q(\textbf{e}^{n}_{t})q_{\psi}(\textbf{s}^{n}_{<t},\textbf{s}^{>n}_{t})}D_{\mathrm{KL}}\big{[}q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})||p_{\theta}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t},\textbf{s}^{n}_{<t},\textbf{s}^{>n}_{t})\big{]}\Big{]}$
(8b)
$\displaystyle-\sum_{t=1}^{T}\sum^{N}_{n=1}\Big{[}\mathbb{E}_{q(\textbf{e}^{n-1}_{t})q_{\psi}(\textbf{s}^{n}_{<t})}D_{\mathrm{KL}}\big{[}q(\textbf{e}^{n}_{t}|\textbf{e}^{n-1}_{t})||p_{\theta}(\textbf{e}^{n}_{t}|\textbf{s}^{n}_{<t})\big{]}\Big{]}.$
(8c)

To better understand the optimisation objective and the role of a temporal
Gaussian mixture from Eq. 2, it is useful to break the down the three
components of the ELBO. First, component 8a is the likelihood of the data
under the inferred posteriors $q(\vec{\textbf{s}}_{t})$, which improves the
quality of frame reconstructions. Second, component 8c is the KL divergence
between the posterior and prior Bernoulli distributions over e, allowing the
parametrised prior model to learn the evolution of static and change priors
over time. Lastly, component 8b regularises the latent belief space by
bringing the posterior either closer to the static or to the change component
of a prior MoG. This can be seen more clearly if we expand the expectation,

$\displaystyle=q(\textbf{e}^{n}_{t}=0)\underbrace{D_{\mathrm{KL}}\big{[}q^{\prime}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=0)\big{]}}_{\text{posterior
$\leftrightarrow$ static
prior}}+q(\textbf{e}^{n}_{t}=1)\underbrace{D_{\mathrm{KL}}\big{[}q^{\prime}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=1)\big{]}}_{\text{posterior
$\leftrightarrow$ change prior}}.$ (9)

Depending on the inferred posterior distribution $q(\textbf{e}^{n}_{t})$, the
model will employ the appropriate part of Eq. 9 in the optimisation. For
example, if inferred that state $\textbf{s}^{n}_{t}$ has not changed
($\textbf{e}^{n}_{t}=0$), the model will bring the new posterior and the
static prior closer together, and vice versa. Ultimately, this allows the
model to naturally cluster similar temporal states together, while learning to
transition between states that are sufficiently separated in the belief space.

Figure 3: Sampling MoG components can be seen as changing the generative
structure of the model. The diagram shows a two-level DLH unrolled over three
timesteps – each box indicating a temporal MoG. Bold arrows indicate the
sampled component of a MoG. At $n,t=2$, the model samples the static component
$0\sim p(\textbf{e}^{2}_{2})$, thus state $\textbf{s}^{2}_{1}$ remains fixed
before being updated to $\textbf{s}^{2}_{3}$ (indicated by the ‘Pruned’ label
on the right).

###  2.5 Model components

DLH’s architectural implementation is similar to that of NVAE (Vahdat & Kautz,
2020) and VPR (Zakharov et al., 2022), which employ deterministic variables
for propagating information through the model. More specifically, DLH consists
of the following model components,

Encoder, $\displaystyle x^{n+1}_{t}=f^{n}_{\text{enc}}(x^{n}_{t})$ (10)
Decoder, $\displaystyle
c^{n-1}_{t}=f^{n}_{\text{dec}}(\textbf{s}^{n}_{t},c^{n}_{t})$ (11) Temporal,
$\displaystyle d^{n}_{t+1}=f^{n}_{\text{tem}}(\textbf{s}^{n}_{t},d^{n}_{t})$
(12)

Posterior state, $\displaystyle
q_{\psi}(\textbf{s}^{n}_{t}|x^{n}_{\tau},c^{n}_{t})$ (13) Prior state,
$\displaystyle p_{\theta}(\textbf{s}^{n}_{t}|d^{n}_{t},c^{n}_{t})$ (14) Prior
factor, $\displaystyle p_{\theta}(\textbf{e}^{n}_{t}|d^{n}_{t})$ (15)

where deterministic variables $x^{n}_{t}$, $c^{n}_{t}$, $d^{n}_{t}$ correspond
to the bottom-up, top-down, and temporal variable transformations, as shown in
Figure 2c, and $x^{0}_{t}$ and $c^{0}_{t}$ correspond to the output image
$\textbf{o}_{t}$. Variables $c^{n}_{t}$ and $d^{n}_{t}$ are non-linear
transformations of samples from $\textbf{s}^{>n}_{t}$ and
$\textbf{s}^{n}_{<t}$, respectively. We use a GRU model (Cho et al., 2014) for
the transition and prior factor models, convolutional layers for the encoder
and decoder, and fully-connected MLP layers for all other models.

##  3 Related work

#### Video prediction

Early works in video prediction largely focused on different variants of
deterministic models (Oh et al., 2015; Finn et al., 2016; Byravan & Fox, 2017;
Vondrick & Torralba, 2017); however, it has been widely suggested that these
models are poorly suited for capturing stochasticity that is often present in
video datasets.

The problem of stochastic video prediction has been addressed using a variety
of generative architectures. Models autoregressive in image space (Babaeizadeh
et al., 2021; Reed et al., 2017; Weissenborn et al., 2019; Kalchbrenner et
al., 2017; Denton & Fergus, 2018) demonstrate good results but suffer from
high computational complexity, particularly for long-term predictions. GAN-
based (Goodfellow et al., 2014) approaches have been popular due to their
ability to produce sharp predictions (Clark et al., 2019; Hong et al., 2021;
Mathieu et al., 2016; Lee et al., 2018). More recently, transformers (Vaswani
et al., 2017) have been used to model video datasets, both in latent (Rakhimov
et al., 2020; Yan et al., 2021) and pixel space (Weissenborn et al., 2019). A
fairly large category of video architectures is based on using Variational
Autoencoders (Kingma & Welling, 2013), which have been shown to produce
meaningful latent representations on image (Vahdat & Kautz, 2020; Higgins et
al., 2022) and video data (Zakharov et al., 2022). Variational autoencoders
(VAE)-based models that attempt to learn temporal dependencies in the latent
space (Wu et al., 2021; Villegas et al., 2019; Castrejón et al., 2019;
Franceschi et al., 2020; Saxena et al., 2021; Yan et al., 2021; Zakharov et
al., 2022) generate good performance but generally suffer from blurry
predictions, referred to as the ‘underfitting problem’ (Babaeizadeh et al.,
2021; Wu et al., 2021; Villegas et al., 2019). Nevertheless, these models
benefit from computational efficiency since the learning of temporal video
dynamics commonly happens in a lower-dimensional latent space. Most recently,
diffusion models have been shown to produce great performance on both short
(Yang et al., 2022; Höppe et al., 2022) and long (Harvey et al., 2022) videos.

#### Hierarchical generative models

Hierarchical generative models have been shown to be an effective way of
modelling high-dimensional data, including images (Rasmus et al., 2015;
Sønderby et al., 2016; Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021)
and videos (Saxena et al., 2021; Kim et al., 2019; Pertsch et al., 2020; Hsu
et al., 2019; Zakharov et al., 2022), producing rich latent representations
and demonstrating strong representational power.

#### Temporal abstraction

The topic of learning temporal abstractions from sequential data has been
harmoniously rising in popularity alongside the progress in deep and
hierarchical latent models. Temporal abstraction models often operate a number
of hierarchical latent variables updating over different timescales, with the
goal of capturing the temporal features of a dataset (Chung et al., 2017;
Mujika et al., 2017; Kim et al., 2019; Saxena et al., 2021; Fountas et al.,
2022; Zakharov et al., 2022), though other models learn the relevant
prediction timescales without resorting to hierarchical methods (Chung et al.,
2017; Neitz et al., 2018; Jayaraman et al., 2019; Shang et al., 2019; Kipf et
al., 2019; Pertsch et al., 2020; Zakharov et al., 2021).

Figure 4: Open-loop video prediction given 30 context frames (not shown). GT:
ground-truth sequence, DLH: our model’s prediction. DLH maintains the
important contextual information about the video, without significant
degeneration in the reconstruction quality.

#### Gaussian Mixtures in VAEs

Our work similarly touches on the topic of VAEs with Gaussian Mixture latent
states. Generally, these models are aimed at producing meaningful structure of
the latent space, in which data points are clustered in an unsupervised
fashion (Dilokthanakul et al., 2016; Jiang et al., 2017; Falck et al., 2021).
Though highly relevant conceptually, these works deal with non-temporal data
and therefore have fundamentally different formulations.

##  4 Experiments

In this section, we showcase the representational properties of DLH, and their
resulting impact on the performance of the model for long-term video
prediction. In particular, we demonstrate that DLH: (a) outperforms benchmarks
in long-term video prediction, (b) produces an organised hierarchical latent
space with spatiotemporal disentanglement and temporal abstraction, (c)
generates coherent videos even in datasets characterised by temporal
stochasticity, and (d) dynamically regulates its structural complexity. In the
analysis probing DLH’s expressivity and representations, we emphasise how the
presented formulation of the generative model, in particular the use of
temporal MoG, naturally results in the emergent properties of the model.

Figure 5:  Hierarchical disentanglement in DLH. (A): Random samples
independently drawn from the different hierarchical levels while keeping all
other levels fixed. They qualitatively illustrate what information is
contained within a particular level. In KTH, L1 and L2 tend to encode motion;
L3 encodes the general context of the frame. In M-MNIST, L1 encodes slight
variations in the position and digits; L2 represents position; L3 contains
both style and digit types. (B): Rolling out hierarchical levels in the DML
Mazes under all other levels being fixed. L1 includes minor variations in the
view angle; L2 changes the position of the observer and wall shape; L3
predicts transitions to the different parts of the maze. (C): Inferred
components of the temporal MoGs at every level of the model. Black circles
indicate $\textbf{e}^{n}=1$ (change component) inferred by DLH. L2 infers the
static component when the person’s arms are not moving (green ellipse).
Similarly, L3 is static throughout the duration of the video.

###  4.1 Datasets and benchmarks

#### Datasets

To test the model’s ability in long-term video prediction, we use Moving MNIST
(Srivastava et al., 2015) with 300 timesteps, KTH Action (Schuldt et al.,
2004) with 300 timesteps, and DeepMind Lab (DML) Mazes (Eslami et al., 2018)
with 200 timesteps. For a more detailed analysis of the model’s properties, we
use a toy Moving Ball dataset (Zakharov et al., 2022).

Benchmarks  Clockwork Variational Autoencoder (CW-VAE) (Saxena et al., 2021)
is a hierarchical VAE for video prediction, in which latent variables operate
over fixed-temporal schedules, similarly subject to nested timescales. CW-VAE
demonstrated state-of-the-art performance in long-term video prediction,
indicating the merit of the slower-evolving context states. VTA (Kim et al.,
2019) is a two-level hierarchical model for video prediction that employs a
parametrised boundary detector to learn sub-sequences of a video and generate
temporally-abstracted representations. LMC-Memory (Lee et al., 2021) learns
and stores long-term motion context for better long-horizon video prediction,
which has been shown to outperform other RNN-based approaches.

Table 1:  Open-loop video prediction. Stars denote <5% standard deviation.

M-MNIST | SSIM$\uparrow$ | PSNR$\uparrow$
---|---|---
DLH (Ours) |  0.78* |  15.7*
CW-VAE |  $0.68$* |  $13.11$*
VTA | $0.58$ | $12.18$
LMC-Memory |  $0.75$* |  $13.73$*
KTH Action | SSIM$\uparrow$ | PSNR$\uparrow$
DLH (Ours) |  0.84* | 24.7
CW-VAE | $0.80$ | $22.0$
VTA | $0.77$ | $22.41$
LMC-Memory |  $0.83$* | $23.44$
DML Mazes | SSIM$\uparrow$ | PSNR$\uparrow$
DLH (Ours) | 0.59 | 14.3
CW-VAE | $0.44$ |  $13.71$*
VTA | $0.55$ |  $13.51$*

Parameter count: DLH (7M), CW-VAE (12M), VTA (3M), LMC-Memory (34M).

#### Metrics

To evaluate stochastic video prediction, we employ the standard procedure of
sampling 100 conditionally generated sequences and picking the best one to
report (Denton & Fergus, 2018). For metrics, we use Structural Similarity
(SSIM) and Peak Signal-to-Noise Ratio (PSNR) to test the performance of a
model with respect to the ground-truth videos.

###  4.2 Video prediction and generation

Table 1 shows the evaluation of DLH and its benchmarks in the task of long-
horizon video prediction. As evident, DLH outperforms other models across the
presented datasets. Figure 4 shows some examples of long-horizon open-loop
rollouts. For Moving MNIST, DLH maintains the information about the digits
throughout the sequence, while also accurately predicting their positions. For
DML Mazes, DLH correctly predicts the colours and wall positions, without
switching to a configuration of another maze. Similarly, for KTH, our model
preserves the important contextual knowledge (e.g. background) and accurately
predicts the long sequence of arm swings. Appendices C.1 and C.2 include
qualitative comparisons of the models and the per-frame metric plots.

###  4.3 Hierarchical abstraction and dynamic structure

DLH exhibits characteristics of a model that learns temporally-abstracted and
hierarchically disentangled representations. Figure 5a demonstrates
reconstructed frames retrieved by sampling the different hierarchical levels
of the model. Here, we observe the variations in the samples that correspond
to meaningful and interpretable spatiotemporal features of the videos. In
Figure 5b, we show rollouts of the model’s levels (other levels being fixed)
using DML Mazes, which indicate that DLH learns to transition between
progressively slower features in the higher levels of its hierarchy.

Table 2: Average number of employed levels ($\bar{L}$) and the total KL loss in the instances of DLH with different number of hierarchical levels (Moving Ball) Levels | $\bar{L}$ | KL loss
---|---|---
2 | $1.22\pm 0.05$ | $41.9\pm 0.3$
3 | $1.24\pm 0.05$ | $42.8\pm 1.1$
4 | $1.32\pm 0.12$ | $41.1\pm 1.0$
5 | $1.38\pm 0.07$ | $43.1\pm 2.1$

Figure 5c demonstrates another telling qualitative evaluation of DLH and its
representations – the inferred components of $\textbf{e}^{n}$ (static or
change) for a given video. In particular, it shows that the model continuously
detects feature changes in the second level of its hierarchy (L2) when the
person is moving their arms, and conversely when the person’s are motionless.
Furthermore, it can be seen that the the top level (L3) remains static
throughout. Notably, these results are in agreement with the random samples
shown above, and more clearly illustrate the property of hierarchical
disentanglement present in the model.

The capacity of DLH to learn the spatiotemporal representation of features
along its hierarchy is largely driven by the dynamic manipulation of its
hierarchical and temporal structure (Figure 3). Interestingly, we observe that
DLH consistently converges to similar structures even when possessing
different number of levels. Table 2 shows the average hierarchical depth
employed by the model
($\bar{L}=\frac{1}{T}\sum_{t=1}^{T}\sum_{n=1}^{N}\textbf{e}^{n}_{t}$) over a
video length given the total number of hierarchical levels it has (trained
using the Moving Ball dataset). As evident, the models converge to similar
values despite their size differences, indicating that DLH naturally
simplifies its structure and does not employ more resources than necessary.
This is similarly substantiated by the comparable magnitudes of the total KL
loss, which is commonly used to indicate the amount of information stored in
the latent states (see Appendix C.3 for a more in-depth analysis).

###  4.4 Temporal stochasticity

Table 3: Average predicted probability of $p_{\theta}(\textbf{e}^{2}=1)$ under the different levels of temporal stochasticity ($\lambda$) in the Moving Ball dataset $\lambda$ | $p_{\theta}(\textbf{e}^{2}=1)$
---|---
change | static
0.0 | $.97\pm 0.01$ | $.007\pm.003$
0.1 | $.82\pm 0.02$ | $.074\pm.007$
0.3 | $.73\pm 0.02$ | $.167\pm.011$

Videos often contain temporal stochasticity, where features may change at
seemingly random times. How would a generative model represent such
uncertainty? In the context of employing a Gaussian latent state, the
uncertainty about the next state would have to be reflected in the higher
variance, in order to cover both possible outcomes; however, this necessarily
increases the chance of sampling areas of the latent space that do not
correspond to any meaningful states, harming the prediction performance. In
DLH, by virtue of the temporal MoG, such stochasticity can be effectively
captured by variable $\textbf{e}^{n}$, which decides whether the latent state
$\textbf{s}^{n}$ should be updated or remain fixed, alleviating the need to
sample from degenerate regions of the latent space. To demonstrate this, we
modify the Moving Ball dataset to include random colour changes that can occur
at every timestep with a probability of $\lambda$ and train a two-level DLH
using it. Figure 6 shows a comparison of open-loop rollouts generated by DLH
and CW-VAE, trained on the stochastic Moving Ball with $\lambda=0.1$. While
CW-VAE struggles to generate rollouts with consistent and sharp colour
changes, DLH faces no such problems, producing sequences with both
deterministic and random colour switches. This experiment shows the important
role of MoGs in representing temporal stochasticity, and highlights the
superior representational capacity of DLH.

Similarly, the behaviour of prior $p_{\theta}(\textbf{e}^{2}|\cdot)$ under
temporal stochasticity can be more clearly understood using the results in
Table 3, which shows the average predicted probability of the change
component, at level 2, under the inferred posterior $q(\textbf{e}^{2})$ being
either change ($\textbf{e}^{2}=1$) or static ($\textbf{e}^{2}=0$). More
stochasticity necessarily implies the reduced ability to predict when the
observed video features would change (as would be signalled by the inferred
posterior component), which should be reflected in the average probabilities
predicted by the prior component model. Indeed, in Table 3, we observe that as
the stochasticity of the dataset, $\lambda$, rises, the model becomes more
cautious in its predictions.

Figure 6: Two randomly-selected open-loop rollouts in a stochastic Moving Ball
dataset using DLH (top) and CW-VAE (bottom) models. DLH successfully produces
realistic rollouts, while CW-VAE struggles to produce rollouts with sharp and
random colour changes (frames highlighted in red).

##  5 Discussion

Our work demonstrates that building generative models with better
representational properties, such as spatiotemporal and hierarchical
disentanglement, translates to better predictive capabilities in long and
complex time series. Furthermore, we believe that improving the quality of
latent representations is of high importance for model-based reinforcement
learning agents, where accurate predictions of the future lead to better
planning and offline credit assignment, while a hierarchical and nested
treatment of time could allow for temporally-abstract reasoning. Nevertheless,
one of the limitations facing VAE-based models, and by extension our own, is
the lack of sharpness in the predictions. Though significant progress has been
made in the recent years (Babaeizadeh et al., 2021; Wu et al., 2021),
addressing this problem in DLH can be a significant next step for further
improving the performance of the model.

## References

  * Babaeizadeh et al. (2021) Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan.  Fitvid: Overfitting in pixel-level video prediction.  (arXiv:2106.13195), June 2021.
  * Byravan & Fox (2017) Arunkumar Byravan and Dieter Fox.  Se3-nets: Learning rigid body motion using deep neural networks.  _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 173–180, 2017.
  * Castrejón et al. (2019) Lluís Castrejón, Nicolas Ballas, and Aaron C. Courville.  Improved conditional vrnns for video prediction.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 7607–7616, 2019.
  * Child (2021) Rewon Child.  Very deep vaes generalize autoregressive models and can outperform them on images.  _International Conference on Learning Representations_ , 2021.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _EMNLP_ , 2014.
  * Chung et al. (2017) Junyoung Chung, Sungjin Ahn, and Yoshua Bengio.  Hierarchical multiscale recurrent neural networks.  _International Conference on Learning Representations_ , 2017.
  * Clark et al. (2019) Aidan Clark, Jeff Donahue, and Karen Simonyan.  Adversarial video generation on complex datasets.  _arXiv: Computer Vision and Pattern Recognition_ , 2019.
  * Denton & Fergus (2018) Emily L. Denton and Rob Fergus.  Stochastic video generation with a learned prior.  _International Conference on Machine Learning_ , 2018.
  * Dilokthanakul et al. (2016) Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, M. J. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan.  Deep unsupervised clustering with gaussian mixture variational autoencoders.  _ArXiv_ , abs/1611.02648, 2016.
  * Eslami et al. (2018) S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis.  Neural scene representation and rendering.  _Science_ , 360(6394):1204–1210, June 2018.  doi: 10.1126/science.aar6170.
  * Falck et al. (2021) Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Christopher C. Holmes.  Multi-facet clustering variational autoencoders.  _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=JbqW3KmmE6.
  * Finn et al. (2016) Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  _Advances in neural information processing systems_ , 29, 2016.
  * Fountas et al. (2022) Zafeirios Fountas, Anastasia Sylaidi, Kyriacos Nikiforou, Anil K. Seth, Murray Shanahan, and Warrick Roseboom.  A Predictive Processing Model of Episodic Memory and Time Perception.  _Neural Computation_ , 34(7):1501–1544, 06 2022\.  ISSN 0899-7667.  doi: 10.1162/neco_a_01514.  URL https://doi.org/10.1162/neco_a_01514.
  * Franceschi et al. (2020) Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari.  Stochastic latent residual video prediction.  _International Conference on Machine Learning_ , 2020.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  (arXiv:1406.2661), June 2014.
  * Harvey et al. (2022) William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=0RTJcuvHtIu.
  * Higgins et al. (2022) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  Beta-vae: Learning basic visual concepts with a constrained variational framework.  _International Conference on Learning Representations_ , July 2022\.
  * Hong et al. (2021) Kibeom Hong, Youngjung Uh, and Hyeran Byun.  Arrowgan : Learning to generate videos by learning arrow of time.  _Neurocomputing_ , 438:223–234, 2021.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2021.01.043.  URL https://www.sciencedirect.com/science/article/pii/S0925231221000643.
  * Höppe et al. (2022) Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi.  Diffusion models for video prediction and infilling.  _Transactions on Machine Learning Research_ , 2022.  URL https://openreview.net/forum?id=lf0lr4AYM6.
  * Hsu et al. (2019) Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Z. Chen, Jonathan Shen, P. Nguyen, and Ruoming Pang.  Hierarchical generative modeling for controllable speech synthesis.  _International Conference on Learning Representations_ , 2019.
  * Jayaraman et al. (2019) Dinesh Jayaraman, Frederik Ebert, Alexei Efros, and Sergey Levine.  Time-agnostic prediction: Predicting predictable video frames.  _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=SyzVb3CcFX.
  * Jiang et al. (2017) Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou.  Variational deep embedding: An unsupervised and generative approach to clustering.  _IJCAI_ , 2017.
  * Kalchbrenner et al. (2017) Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu.  Video pixel networks.  volume 70 of _Proceedings of Machine Learning Research_ , pp. 1771–1779, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/kalchbrenner17a.html.
  * Kim et al. (2019) Taesup Kim, Sungjin Ahn, and Yoshua Bengio.  Variational temporal abstraction.  _Advances in Neural Information Processing Systems_ , 32:11570–11579, 2019.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2015.
  * Kingma & Welling (2013) Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  _CoRR_ , abs/1312.6114, 2013.
  * Kipf et al. (2019) Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia.  Compile: Compositional imitation learning and execution.  _International Conference on Machine Learning_ , pp. 3418–3428, 2019.
  * Lee et al. (2018) Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.  Stochastic adversarial video prediction.  (arXiv:1804.01523), April 2018.
  * Lee et al. (2021) Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-Il Kim, and Yong Man Ro.  Video prediction recalling long-term motion context via memory alignment learning.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Maaløe et al. (2019) Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther.  Biva: A very deep hierarchy of latent variables for generative modeling.  _NeurIPS_ , 2019.
  * Mathieu et al. (2016) Michael Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  _International Conference on Learning Representations_ , 2016.
  * Mujika et al. (2017) Asier Mujika, Florian Meier, and Angelika Steger.  Fast-slow recurrent neural networks.  _NeurIPS_ , 2017.
  * Neitz et al. (2018) Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf.  Adaptive skip intervals: Temporal abstraction for recurrent dynamical models.  _NeurIPS_ , 2018.
  * Oh et al. (2015) Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh.  Action-conditional video prediction using deep networks in atari games.  In _NIPS_ , 2015.
  * Pertsch et al. (2020) Karl Pertsch, Oleh Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Daniilidis, Joseph Lim, and Andrew Jaegle.  Keyframing the future: Keyframe discovery for visual prediction and planning.  _Learning for Dynamics and Control_ , pp. 969–979, 2020.
  * Rakhimov et al. (2020) Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.  Latent video transformer.  In _VISIGRAPP_ , 2020.
  * Rasmus et al. (2015) Antti Rasmus, Mathias Berglund, M. Honkala, H. Valpola, and T. Raiko.  Semi-supervised learning with ladder networks.  _NIPS_ , 2015.
  * Reed et al. (2017) Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando de Freitas.  Parallel multiscale autoregressive density estimation.  _International Conference on Machine Learning_ , March 2017.
  * Saxena et al. (2021) Vaibhav Saxena, Jimmy Ba, and Danijar Hafner.  Clockwork variational autoencoders.  In _Neural Information Processing Systems_ , 2021.
  * Schuldt et al. (2004) C. Schuldt, I. Laptev, and B. Caputo.  Recognizing human actions: A local svm approach.  _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , 3:32–36 Vol.3, August 2004.  ISSN 1051-4651.  doi: 10.1109/ICPR.2004.1334462.
  * Shang et al. (2019) Wenling Shang, Alex Trott, Stephan Zheng, Caiming Xiong, and Richard Socher.  Learning world graphs to accelerate hierarchical reinforcement learning.  _arXiv preprint arXiv:1907.00664_ , 2019.
  * Sønderby et al. (2016) Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.  Ladder variational autoencoders.  _NIPS_ , 2016.
  * Srivastava et al. (2015) Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  _International conference on machine learning_ , pp. 843–852, 2015\.
  * Vahdat & Kautz (2020) Arash Vahdat and J. Kautz.  Nvae: A deep hierarchical variational autoencoder.  _Neural Information Processing Systems_ , abs/2007.03898, 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _NIPS_ , December 2017.
  * Villegas et al. (2019) Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee.  High fidelity video prediction with large stochastic recurrent neural networks.  _Neural Information Processing Systems_ , November 2019.
  * Vondrick & Torralba (2017) Carl Vondrick and Antonio Torralba.  Generating the future with adversarial transformers.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2992–3000, 2017.
  * Weissenborn et al. (2019) Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit.  Scaling autoregressive video models.  _International Conference on Learning Representations_ , February 2019\.
  * Wu et al. (2021) Bohan Wu, Suraj Nair, Roberto Martín-Martín, Li Fei-Fei, and Chelsea Finn.  Greedy hierarchical variational autoencoders for large-scale video prediction.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2318–2328, 2021.
  * Yan et al. (2021) Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  (arXiv:2104.10157), September 2021.
  * Yang et al. (2022) Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  (arXiv:2203.09481), May 2022.
  * Zakharov et al. (2021) Alexey Zakharov, Matthew Crosby, and Zafeirios Fountas.  Episodic memory for subjective-timescale models.  _ICML 2021 Workshop on Unsupervised Reinforcement Learning_ , 2021\.
  * Zakharov et al. (2022) Alexey Zakharov, Qinghai Guo, and Zafeirios Fountas.  Variational predictive routing with nested subjective timescales.  _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=JxFgJbZ-wft.

##  Appendix A Training and architectural details

###  A.1 Implementation and training

For all datasets, DLH was trained with video sequences of length 100 and batch
100. We used Adam optimiser (Kingma & Ba, 2015) with a learning rate of
$0.0005$ and $\epsilon=0.0001$. We also find that it is beneficial to multiply
the KL loss with a parameter $\beta$, which is slowly increased to the value
of $1.0$ over the course of the first 10k training iterations. This promotes
the model to learn good reconstructions before being severely restricted in
increasing its latent complexity. The image reconstruction model predicts the
means of a Gaussian with fixed standard deviation, which is optimised
separately for each dataset.

For this paper, we used a three-level DLH for all datasets (except for Moving
Ball, where we used just two levels). Each level possesses deterministic
variables, for which we set $|x^{n}|=|d^{n}|=|c^{n}|=200$, and random
variables, where $|\textbf{s}^{n}|=20$. For encoding and decoding image data,
we use 4 convolutional and 4 transposed-convolutional layers, respectively.
Table 4 shows the architectural details of all other model components,
including the sizes of neural networks. The total number of parameters of a
three-level DLH is 7M, and its training takes $\sim$3 days on a single Tesla
V100.

Table 4: Architecture of models from Eq.10-15. Model | Hidden layers
---|---
Encoder, $f^{n}_{\text{enc}}(x^{n}_{t})$ | MLP [200]
Decoder, $f^{n}_{\text{dec}}(\textbf{s}^{n}_{t},c^{n}_{t})$ | MLP [200]
Temporal, $f^{n}_{\text{tem}}(\textbf{s}^{n}_{t},d^{n}_{t})$ | GRU [200]
Posterior, $q_{\psi}(\textbf{s}^{n}_{t}|x^{n}_{\tau},\textbf{s}^{>n}_{t})$ | MLP [40, 40, 40]
Prior state, $p_{\theta}(\textbf{s}^{n}_{t}|\textbf{s}^{n}_{<t},\textbf{s}^{>n}_{t})$ | MLP [40, 40, 40]
Prior factor, $p_{\theta}(\textbf{e}^{n}_{t}|\textbf{s}^{n}_{<t})$ | GRU [200]

###  A.2 Pseudocode

for  _$\\{\textbf{o}_{1:T}\\}_{i}\sim\mathcal{D}$_ do

for  _$t=1$ to T_ do

for  _$n=1$ to N_ do

$\triangleright$ Compute the log-likelihood ratio under approximate posterior
$\mathbb{E}_{p^{\prime}(\textbf{s}^{>n}_{t}|\textbf{e}^{>n}_{t}=0)}q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})$
using ;

$D_{\mathrm{KL}}[q_{\psi}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=1)]\text{
}$ and $\text{
}D_{\mathrm{KL}}[q_{\psi}(\textbf{s}^{n}_{t})||p^{\prime}(\textbf{s}^{n}_{t}|\textbf{e}^{n}_{t}=0)]$.
;

$\triangleright$ Approximate the posterior
$q(\textbf{e}^{n}_{t}|\textbf{e}^{n-1}_{t})$ using Eq. 4 and the nested
timescales constraint, $q(\textbf{e}^{n}_{t}=1|\textbf{e}^{n-1}_{t}=0)=0$. ;

if  _$0\sim q(\textbf{e}^{n}_{t})$_ then

$\triangleright$ Approximate $q(\textbf{e}^{>n}_{t}=1|\cdot)=0$ and
$q^{\prime}(\textbf{s}^{>n}_{t})\leftarrow q^{\prime}(\textbf{s}^{>n}_{t-1})$
;

$\triangleright$ $K\leftarrow n$ ;

break

for  _$n=K$ to 1_ do

$\triangleright$ Compute the state posterior
$q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t})$ ;

$\triangleright$ Compute the ELBO (Eq. 8) using the inferred posteriors
$q_{\psi}(\vec{\textbf{s}}_{t}|\textbf{o}_{t})q(\vec{\textbf{e}})$. ;

$\triangleright$ Apply a gradient step on $\theta,\psi$.

Algorithm 1 Pseudocode of DLH

##  Appendix B Approximation of the posterior component of MoG

###  B.1 Expected log-likelihood ratio

In Section 2.2, we propose an approximation of the posterior
$q(\textbf{e}^{n}_{t})$ using the expected log-likelihood ratio in Eq. 3. We
can arrive at this formulation via either (a) considering the log-likelihood
ratio of the two components of the MoG, or (b) the VaDE trick from the
Gaussian Mixture VAE literature (Appendix B.3). In this section, we briefly
explain the former perspective.

We start by considering the two components in the prior MoG as competing
models. Their log-likelihood ratio is defined as,

$\displaystyle\log\Lambda(\textbf{s}^{n})=\log\prod_{i=0}^{D}\frac{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=1)}=\sum_{i=0}^{D}\log\frac{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=1)},$
(16)

where $\textbf{s}^{n}_{i}\sim q^{\prime}(\textbf{s}^{n})$, $D$ is the number
of posterior samples, and
$\Lambda(\textbf{s}^{n})=\prod_{i=0}^{D}\frac{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}_{i}^{n}|\textbf{e}^{n}=1)}$.
By applying the Law of Large numbers, taking $D\rightarrow\infty$, we can
write this as an expectation with respect to the posterior
$q^{\prime}(\textbf{s}^{n})$,

$\displaystyle\mathbb{E}[\log\Lambda(\textbf{s}^{n})]=\mathbb{E}_{q^{\prime}(\textbf{s}^{n})}\log\frac{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)},$
(17)

where the static and prior components,
$p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)$ and
$p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)$, are viewed as the competing
models under the inferred posterior $q^{\prime}(\textbf{s}^{n})$. This
expected likelihood ratio can be computed in terms of KL divergences, if we
add and subtract the entropy of $q^{\prime}(\textbf{s}^{n})$,

$\displaystyle\mathbb{E}[\log\Lambda(\textbf{s}^{n})]$
$\displaystyle=\mathbb{E}_{q^{\prime}(\textbf{s}^{n})}\log
p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)-\mathbb{E}_{q^{\prime}(\textbf{s}^{n})}\log
q^{\prime}(\textbf{s}^{n})$ (18)
$\displaystyle-\mathbb{E}_{q^{\prime}(\textbf{s}^{n})}\log
p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)+\mathbb{E}_{q^{\prime}(\textbf{s}^{n})}\log
q^{\prime}(\textbf{s}^{n})$ (19)
$\displaystyle=D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n})||p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)]-D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n})||p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)].$
(20)

Assuming minimum probability of error test, the selection of the most likely
component is realised via

$\displaystyle
D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n})||p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)]-D_{\mathrm{KL}}[q^{\prime}(\textbf{s}^{n})||p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)]\mathop{\lessgtr}_{\textbf{e}^{n}=0}^{\textbf{e}^{n}=1}0.$
(21)

###  B.2 Expected log-likelihood ratio with hierarchical dependencies

In DLH, posteriors are factorised hierarchically, such that

$\displaystyle
q(\vec{\textbf{s}}_{t})=\prod^{N}_{n=0}q_{\psi}(\textbf{s}^{n}_{t}|\textbf{s}^{>n}_{t},\textbf{o}_{t}).$
(22)

In turn, this implies that the expected log-likelihood ratio must be computed
with respect to the hierarchical posteriors,

$\displaystyle\mathbb{E}[\log\Lambda(\textbf{s}^{n})]$
$\displaystyle=\mathbb{E}_{q(\vec{\textbf{s}}_{t})}\log\frac{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)}$
(23)
$\displaystyle=\underbrace{\mathbb{E}_{q^{\prime}(\textbf{s}^{>n}_{t})}}_{\text{hierarchical
context}}\mathbb{E}_{q^{\prime}(\textbf{s}^{n}_{t})}\log\frac{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=0)}{p^{\prime}(\textbf{s}^{n}|\textbf{e}^{n}=1)}.$
(24)

However, this estimation implies that all hierarchical posteriors,
$q^{\prime}(\textbf{s}^{>n}_{t})$, must first be inferred in a top-down
process, which may be computationally expensive, especially for large $N$. To
resolve this, we approximate the hierarchical posteriors, above the level at
which the posterior component $q(\textbf{e}^{n}_{t})$ is being estimated,
using the static priors,

$\displaystyle
q^{\prime}(\textbf{s}^{>n}_{t})\approx\prod^{N}_{j=n+1}p^{\prime}(\textbf{s}^{j}_{t}|\textbf{e}^{j}_{t}=0)=\prod^{N}_{j=n+1}q^{\prime}(\textbf{s}^{j}_{t-1}),$
(25)

which are already known. This simple assumption alleviates the need to infer
all hierarchical posteriors before the estimation of $q(\textbf{e}^{n}_{t})$
and has been shown to work well in practice.

More specifically, the computational savings come from the combination of two
factors: (1) approximating hierarchical context using Eq. 25 during the
inference of $q(\textbf{e}^{n}_{t})$, and (2) the nested timescales assumption
in Eq. 5 that blocks any further inference beyond the level at which
$\textbf{e}^{n}_{t}=0$. As such, the computations pertaining to the inference
procedure are required only up to some level $k$, where $\textbf{e}^{k}_{t}=0$
is first inferred.

###  B.3 Relationship to VaDE trick

VaDE trick is a non-parametric technique for estimating posterior
$p(\textbf{e}|\textbf{s})$ in Gaussian Mixture VAE models (Jiang et al., 2017;
Falck et al., 2021). In particular, Falck et al. (2021) proved that an
approximate posterior distribution $q(\textbf{e})$ that minimises the KL
divergence to the true posterior $p(\textbf{e}|\textbf{s})$ will take the
form,

$\displaystyle\operatorname*{arg\,min}_{q(\textbf{e})}D_{\mathrm{KL}}[q(\textbf{e})||p(\textbf{e}|\textbf{s})]=\frac{\exp(\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}|\textbf{s}))}{\sum_{\textbf{e}\in\textbf{E}}\exp(\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}|\textbf{s}))},$ (26)

where $K=|\textbf{E}|$ is the number of components in a Gaussian mixture
$p(\textbf{s},\textbf{e})$, and $q(\textbf{s})$ is an approximate posterior
over s.

In DLH, the number of components in a MoG is limited to $K=2$. Using Eq. 26,
we can compute the posterior odds and solve for one of the two components
(e.g. $q(\textbf{e}=0)$),

$\displaystyle\frac{q(\textbf{e}=0)}{q(\textbf{e}=1)}$
$\displaystyle=\frac{\exp(\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}=0|\textbf{s}))}{\exp(\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}=1|\textbf{s}))}$ (27)
$\displaystyle=\exp\Big{[}\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}=0|\textbf{s})-\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{e}=1|\textbf{s})\Big{]}$ (28)
$\displaystyle=\exp\Big{[}\mathbb{E}_{q(\textbf{s})}\log\frac{p(\textbf{s}|\textbf{e}=0)p(\textbf{e}=0)}{\cancel{\sum_{\textbf{e}\in\textbf{E}}p(\textbf{s}|\textbf{e})p(\textbf{e})}}-\mathbb{E}_{q(\textbf{s})}\log\frac{p(\textbf{s}|\textbf{e}=1)p(\textbf{e}=1)}{\cancel{\sum_{\textbf{e}\in\textbf{E}}p(\textbf{s}|\textbf{e})p(\textbf{e})}}\Big{]}$
(29) $\displaystyle=\exp\Big{[}\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{s}|\textbf{e}=0)p(\textbf{e}=0)-\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{s}|\textbf{e}=1)p(\textbf{e}=1)\Big{]}$ (30)
$\displaystyle=\exp\Big{[}\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{s}|\textbf{e}=0)-\mathbb{E}_{q(\textbf{s})}\log
p(\textbf{s}|\textbf{e}=1)+\log\frac{p(\textbf{e}=0)}{p(\textbf{e}=1)}\Big{]}$
(31)

Adding and subtracting $\mathbb{E}_{q(\textbf{s})}\log q(\textbf{s})$ inside
the exponent,

$\displaystyle=\exp\Big{[}D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=1)]-D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=0)]+\log\frac{p(\textbf{e}=0)}{p(\textbf{e}=1)}\Big{]}.$
(32)

Finally, we use the fact that $\sum_{\textbf{e}\in\textbf{E}}q(\textbf{e})=1$
to get $q(\textbf{e}=0)$,

$\displaystyle\frac{q(\textbf{e}=0)}{1-q(\textbf{e}=0)}=\exp\Big{[}D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=1)]-D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=0)]+\log\frac{p(\textbf{e}=0)}{p(\textbf{e}=1)}\Big{]},$
(33)

and solving for $q(\textbf{e}=0)$ yields,

$\displaystyle
q(\textbf{e}=0)=\frac{\exp\Big{[}D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=1)]-D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=0)]+\log\frac{p(\textbf{e}=0)}{p(\textbf{e}=1)}\Big{]}}{1+\exp\Big{[}D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=1)]-D_{\mathrm{KL}}[q(\textbf{s})||p(\textbf{s}|\textbf{e}=0)]+\log\frac{p(\textbf{e}=0)}{p(\textbf{e}=1)}\Big{]}}.$
(34)

Equation 34 shows the formulation of distribution $q(\textbf{e})$ in terms of
the VaDE trick. We note that under the assumption that
$p(\textbf{e}=0)=p(\textbf{e}=1)=0.5$, the formulation is a softmax function
of the difference between the KL divergences indicated in Eq. 4. As such, the
proposed estimation of $q(\textbf{e})$ may be seen as taking a sample from the
most likely component in the VaDE-based distribution in Eq. 34, under the
assumption of equal prior probabilities.

##  Appendix C Extra results

###  C.1 Qualitative comparisons

Figure 7: Qualitative comparisons of open-loop predictions using Moving MNIST.
All models are given 30 context frames (36 for CW-VAE). The figures show the
first 50 and last 45 timesteps in the produced rollouts. The context frames
are not shown here. GT denotes the ground-truth sequence.  Figure 8:
Qualitative comparisons of open-loop predictions using KTH. All models are
given 30 context frames (36 for CW-VAE). The figures show the first 50 and
last 45 timesteps in the produced rollouts. The context frames are not shown
here. GT denotes the ground-truth sequence.

###  C.2 Per-frame prediction evaluation

Figure 9: Per-frame video prediction performance against the benchmarks on the
Moving MNIST and KTH datasets. They y-axes correspond to the SSIM$\uparrow$,
PSNR$\uparrow$, and LPIPS$\downarrow$ metrics. The x-axes correspond to the
number of open-loop prediction steps.

###  C.3 Selection of the number of levels

We demonstrate in more detail the ability of DLH to converge its structure and
employ only the necessary amount of resources in processing data. In Section
2.3, using the Moving Ball dataset, we showed that irrespective of the number
of levels the model possesses, it only partially uses its latent hierarchy. In
particular, we reported two metrics: the average number of levels used by the
model and the average total KL divergence. For both metrics, we reported
similar values across the instances of DLH with different total number of
levels.

Table 5: Per-level KL divergences of DLH (Moving Ball dataset) Levels | KL (L1) | KL (L2) | KL (L3) | KL (L4) | KL (L5)
---|---|---|---|---|---
2 | $35.1$ | $6.8$ | - | - | -
3 | $34.7$ | $7.8$ | 0.3 | - | -
4 | $34.3$ | $6.5$ | 0.2 | 0.1 | -
5 | $36.1$ | $6.6$ | 0.1 | 0.1 | 0.2

Notably, the KL divergence is often used as a measure of the amount of
information stored in the latent variables of the model. By breaking down the
contributions of each latent level to the total value of the KL loss, we can
get a glimpse into how DLH employs the different hierarchical levels. Table 5
shows the average per-level KL loss of DLH. As can be observed, levels $>2$
are significantly lower, suggesting that no information is being stored in
those levels, as the model naturally ‘collapses’ them. More visually, we can
sample from these low-KL levels ($>2$), while keeping high-KL levels fixed
($\leq 2$), and vice versa, in order to see if these levels contribute to the
variations in the reconstructed images. Figures 10 and 11 show that, indeed,
samples from the low-KL levels exhibit minimum variations, while high-KL
levels seem to encode most of the important information. These results once
again illustrate how DLH naturally tends to use the minimum amount of
resources necessary for modelling the data. Furthermore, this property hints
at a potential method for selecting the appropriate number of levels – by
monitoring the values of the KL for models with different number of
hierarchical levels.

(a)  3-level DLH
Samples from levels: 3
Fixed levels: 1, 2

(b)  4-level DLH
Samples from levels: 3, 4
Fixed levels: 1, 2

(c) 5-level DLH
Samples from levels: 3, 4, 5
Fixed levels: 1, 2

Figure 10: Samples from levels $>2$. As can be seen, there are no variations
in the observed reconstructions, indicating that no information is stored in
these levels. This is corroborated by the low values of the KL divergence
associated with the hierarchical levels.

(a)  3-level DLH
Samples from levels: 1, 2
Fixed levels: 3

(b) 4-level DLH
Samples from levels: 1, 2
Fixed levels: 3, 4

(c)  5-level DLH
Samples from levels: 1, 2
Fixed levels: 3, 4, 5

Figure 11: Samples from levels $\leq 2$. In contrast to samples in Figure 10,
we now observe high variation in the reconstructed images, indicating that the
data is primarily modelled by the bottom two levels. This is in line with the
high values of the KL divergences associated with these levels.