It suffices to check that $D^s_N(A)$ has the appropriate Calabi-Yau property.  We have that $\nu_A^r \cong id$, hence $\nu_A^s \cong id$.  Then

$$([-1]\nu_A)^s \cong [-s] = [-tN] \cong \Sigma^{-2t}$$

For $N=2$, we have $\Sigma = [1]$, hence $([-1]\nu_A)^r \cong \Sigma^{-r}$.

Suppose $A$ is symmetric.  Then $\stab{}{A}$ is $(-1)$-Calabi-Yau and $\stab{N}{A}$ is $(-2, N)$-Calabi-Yau for all $N > 2$.

Since $A$ is symmetric, $\nu_A = id$ hence $r = 1$.  The statement follows.

The above integer pairs need not be minimal.  The presence of additional relations between the functors $\Omega$, $\nu_{A*}$ and $R$ may allow $\stab{N}{A}$ to be $(x,y)$-Calabi-Yau for smaller values of $x$ and $y$; see below for a concrete example.

§.§ An Example

Let $F$ be any field, let $Q$ be the quiver $\begin{tikzcd} 1 \arrow[r, "\alpha", bend left] & 2 \arrow[l, "\beta", bend left] \end{tikzcd}$, and let $A = FQ/rad^2(FQ)$.  Then $A$ is self-injective with four indecomposable modules: the simple modules $S_1$ and $S_2$ and their two-dimensional injective hulls $I_1$ and $I_2$.

[column sep = small]
[I_1] [dr]         [I_2 ] [dr]

S_1 [ur]         S_2 [ur]         S_1

The Auslander-Reiten quiver of $A$.

Vertices in brackets are projective-injective.

Fix some $N\ge 2$.  For any integers $i, j \ge 0$ satisfying $1 \le i+j \le N-1$, define objects $X(i, j)$ and $Y(i,j)$ in $\MMor_{N-2}(A)$ by

\begin{eqnarray*}
X(i,j) := 0 \rightarrow \cdots \rightarrow 0 \rightarrow S_1 \rightarrow \cdots \rightarrow S_1 \rightarrow I_1 \rightarrow \cdots \rightarrow I_1\\
Y(i,j) := 0 \rightarrow \cdots \rightarrow 0 \rightarrow S_2 \rightarrow \cdots \rightarrow S_2 \rightarrow I_2 \rightarrow \cdots \rightarrow I_2
\end{eqnarray*}

Here each sequence has exactly $i$ simples and $j$ projective-injectives, and each morphism is the canonical inclusion.

In $\rmod{A}$, every monomorphism from an indecomposable module $M$ into a direct sum $Y \oplus Z$ factors through either $Y$ or $Z$, so $(M_\bullet, \alpha_\bullet) \in \MMor_{N-2}(A)$ is indecomposable if and only if each $M_i$ is indecomposable.  Thus the indecomposable objects of $\MMor_{N-2}(A)$ are precisely the $X(i,j)$ and $Y(i,j)$.  The indecomposable projective-injectives are precisely the objects $X(0, j)$ and $Y(0,j)$.

The Nakayama automorphism of $A$ has order $2$, so by Corollary <ref>, $\stab{N}{A}$ is $(-4, 2N)$-Calabi-Yau if $N$ is odd and $(-2, N)$ if $N > 2$ is even.  However, it is easy to check that $\nu_{A*} \cong \Omega \cong \Omega^{-1}$ on $\stab{N}{A}$ for any $N$.  It follows from Proposition <ref> that $R$ and $\Omega^{-1}$ commute, since the corresponding functors $\Sigma$ and $\Sigma[-1]$ commute in $D^s_N(A)$.  Thus $\stab{N}{A}$ has Serre functor $S = \Omega R \nu_{A*} \cong R$, and $D^s_N(A)$ has Serre functor $\Sigma[-1]$.  In particular,

$$S^N \cong \Omega^{-N+2} \cong \begin{cases} \Omega^{-1} & N \text{ odd}\\  id & N \text{ even} \end{cases}$$

Thus for $N > 2$, $\stab{N}{A}$ is $(1,N)$-Calabi-Yau for odd $N$ and $(0, N)$-Calabi-Yau for even $N$.

A straightforward computation shows that for any $i>0$,

\begin{eqnarray*}
S(X(i,j)) &=& \begin{cases} Y(i, j-1) & j > 0 \\ X(N-i, i-1) & j = 0\end{cases}
\\
S(Y(i,j)) &=& \begin{cases} X(i, j-1) & j > 0 \\ Y(N-i, i-1) & j = 0\end{cases}
\\
\Omega (X(i,j)) &=& Y(i,j)
\\
\Omega (Y(i,j)) &=& X(i,j)
\end{eqnarray*}

It follows immediately that $S^n$ is not isomorphic to any power of $\Omega$ for any $0 < n < N$.

We conclude by providing the Auslander-Reiten quiver of $\MMor_{N-2}(A)$ for representative values of $N$.

[column sep = small]
Y(1,0) [dr] [r, dashed]	    [Y(0,1)] [r, dashed]    Y(1,1) [dr] [r, dashed]    [Y(0,2)] [r, dashed]     X(2,0) [dr]

X(1,1)[ur] [r, dashed]    [X(0,2)] [r, dashed]    Y(2,0) [ur]        X(1,0) [ur] [r, dashed]    [X(0,1)] [r, dashed]    X(1,1)

The Auslander-Reiten quiver of $\MMor_{3}(A)$.

Vertices in brackets are projective-injective.

[column sep = small]
Y(1,2) [r, dashed] [dr]    [Y(0,3)] [r, dashed]    X(3,0) [dr]        Y(1,0) [r, dashed] [dr]    [Y(0,1)] [r, dashed]    Y(1,1) [r, dashed] [dr]    [Y(0,2)] [r, dashed]    Y(1,2)

X(2,0) [ur] [dr]        X(2,1) [ur] [dr]        Y(2,0) [ur] [dr]        Y(2,1) [ur] [dr]

X(1,0) [r, dashed] [ur]    [X(0,1)] [r, dashed]    X(1,1) [r, dashed] [ur]    [X(0,2)] [r, dashed]    X(1,2) [r, dashed] [ur]    [X(0,3)] [r, dashed]    Y(3,0) [ur]        X(1,0)

The Auslander-Reiten quiver of $\MMor_{4}(A)$.

Vertices in brackets are projective-injective.

[1]
P. Bahiraei, R. Hafezi, and A. Nematbakhsh.
Homotopy category of $N$-complexes of projective modules.
J. Pure Appl. Algebra, 220(6):2414–2433, 2016.

[2]
Francis Borceux.
Handbook of categorical algebra: volume 1, Basic category
theory, volume 1.
Cambridge University Press, 1994.

[3]
Ragnar-Olaf Buchweitz.
Maximal Cohen-Macaulay modules and Tate-cohomology over
Gorenstein rings.

[4]
Theo Bühler.
Exact categories.
Expositiones Mathematicae, 28(1):1–69, 2010.

[5]
Aaron Chan, Erik Darpö, Osamu Iyama, and René Marczinzik.
Periodic trivial extension algebras and fractionally Calabi-Yau
arXiv preprint arXiv:2012.11927, 2020.

[6]
Raquel Coelho Simões and David Pauksztello.
Simple-minded systems and reduction for negative Calabi-Yau
triangulated categories.
Transactions of the American Mathematical Society,
373(4):2463–2498, 2020.

[7]
Michel Dubois-Violette.
$d^N$= 0: Generalized homology.
K-theory, 14(4):371–404, 1998.

[8]
Michel Dubois-violette and Marc Henneaux.
Generalized cohomology for irreducible tensor fields of mixed Young
symmetry type.
Dubois-Violette M., Henneaux M., Tensor, page 0110088, 1999.

[9]
Ben Elias and You Qi.
Categorifying Hecke algebras at prime roots of unity, part i.
preprint, arXiv:2005.03128.

[10]
Ben Elias and You Qi.
An approach to categorification of some small quantum groups II.
Adv. Math., 288:81–151, 2016.

[11]
Ben Elias and You Qi.
A categorification of quantum $\mathfrak{sl}(2)$ at prime roots of
Adv. Math., 299:863–930, 2016.

[12]
Nan Gao, Julian Külshammer, Sondre Kvamme, and Chrysostomos Psaroudakis.
A functorial approach to monomorphism categories for species i.
preprint, arXiv:1907.04657.

[13]
Sergei I Gelfand and Yuri I Manin.
Methods of homological algebra.
Springer Science & Business Media, 2013.

[14]
Osamu Iyama, Kiriko Kato, and Jun-ichi Miyachi.
Derived categories of $N$-complexes.
Journal of the London Mathematical Society, 96(3):687–716,

[15]
Mikhail M. Kapranov.
On the $q$-analog of homological algebra.
preprint, arXiv:q-alg/9611005.

[16]
Bernhard Keller.
Calabi-Yau triangulated categories.
Trends in representation theory of algebras and related topics,
pages 467–489, 2008.

[17]
Bernhard Keller and Idun Reiten.
Acyclic Calabi-Yau categories.
Compositio Mathematica, 144(5):1332–1348, 2008.

[18]
Mikhail Khovanov.
Hopfological algebra and categorification at a root of unity: the
first steps.
J. Knot Theory Ramifications, 25(3):1640006, 26, 2016.

[19]
Mikhail Khovanov and You Qi.
An approach to categorification of some small quantum groups.
Quantum Topol., 6(2):185–311, 2015.

[20]
Henning Krause.
Homological theory of representations.
<https://www.math.uni-bielefeld.de/ hkrause/HomTheRep.pdf>, 2021.

[21]
W. Mayer.
A new homology theory. I, II.
Ann. of Math. (2), 43:370–380, 594–605, 1942.

[22]
Djalal Mirmohades.
Simplicial structure on complexes.
arXiv preprint arXiv:1404.0628, 2014.

[23]
Amnon Neeman.
Triangulated Categories.
Princeton University Press, 2014.

[24]
Jeremy Rickard.
Derived categories and stable equivalence.
J. Pure Appl. Algebra, 61(3):303–317, 1989.

[25]
Claus Ringel and Markus Schmidmeier.
The Auslander-Reiten translation in submodule categories.
Transactions of the American Mathematical Society,
360(2):691–716, 2008.

[26]
Claus Michael Ringel and Markus Schmidmeier.
Submodule categories of wild representation type.
J. Pure Appl. Algebra, 205(2):412–422, 2006.

[27]
Bao-Lin Xiong, Pu Zhang, and Yue-Hui Zhang.
Auslander-Reiten translations in monomorphism categories.
In Forum Mathematicum, volume 26, pages 863–912. De Gruyter,

[28]
Pu Zhang.
Monomorphism categories, cotilting theory, and
Gorenstein-projective modules.
Journal of Algebra, 339(1):181–202, 2011.

For $(X_\bullet, \alpha_\bullet) \in EMor_k(A)$, define $\Ker_\bullet(X) := ker(\alpha_1) \hookrightarrow ker(\alpha_1^2) \hookrightarrow \cdots \hookrightarrow ker(\alpha_1^{k}) \hookrightarrow X_1$.  For $f_\bullet: X_\bullet \rightarrow Y_\bullet$, let $\Ker_\bullet(f): \Ker_\bullet(X) \rightarrow \Ker_\bullet(Y)$ be given by the component-wise induced maps on the kernels.

It is clear that $\Cok$ defines a functor $MMor_k(A) \rightarrow EMor_k(A)$, that $\Ker$ defines a functor $EMor_k(A) \rightarrow MMor_k(A)$, and that these functors are mutually inverse.  By the Five Lemma, both functors are compatible with the exact structures on each category.  They also preserve projective-injective objects, and therefore induce triangulated equivalences between the stable categories.

Recall that the projective-injective objects of $EMor_k(A)$ are precisely the injective objects of $Mor_k(A)$, hence we have an inclusion $\underline{EMor_k}(A) \hookrightarrow \overline{Mor_k}(A)$.

Let $(X^\bullet, d_X^\bullet) \in C(\mathcal{A})$.  Define an $N$-complex $(\widetilde{X}^\bullet, d_{\widetilde{X}}^\bullet)$ by
$$\widetilde{X}^{kN+i} = \begin{cases}
X^{2k} & i = 0\\
X^{2k+1} & 0 < i < N
\end{cases}$$

with differential

$$d_{\widetilde{X}}^{kN+i} = \begin{cases}
\widetilde{d}_X^{2k} & i = 0\\
id_{X^{2k+1}} & 1 < i < N-1\\
d_X^{2k+1} & i = N-1
\end{cases}$$

This construction yields a functor $\widetilde{\phantom{a}}: C(\mathcal{A}) \rightarrow C_N(\mathcal{A})$.

Let $X \in \mathcal{A}$, viewed as a complex concentrated in degree $0$.  Then $\widetilde{X[2n]} = \mu^{-Nn}_{1}(X)$ and $\widetilde{X[2n+1]} = \mu^{-Nn-1}_{N-1}(X)$.

The functor $\widetilde{\phantom{a}}$ induces functors of triangulated categories $\widetilde{\phantom{a}}: Ho^\natural(\mathcal{A}) \rightarrow Ho^\natural_N(\mathcal{A})$ and $\widetilde{\phantom{a}}: D^\natural(\mathcal{A}) \rightarrow D^\natural_N(\mathcal{A})$, where $^\natural = nothing, b, +, -$.

It is clear that $\widetilde{\phantom{a}}$ preserves chainwise split exact sequences, hence $\widetilde{\phantom{a}}$ is compatible with the exact structure on both categories.  Furthermore, since $\widetilde{\mu_2^{2i}(X)} = \mu_N^{Ni}(X)$ and $\widetilde{\mu_2^{2i-1}(X)} = \mu_N^{Ni -1}(X)$, it follows that $\widetilde{\phantom{a}}$ preserves projective-injective objects.  It then follows immediately that $\widetilde{\phantom{a}}$ descends to a triangulated functor on the homotopy categories.

To show we obtain an induced functor on the derived category, suppose $X^\bullet$ is a $2$-complex.  It is straightforward to check that for any $0 \le i < N$, $0 < r < N$, and $k \in \mathbb{Z}$,

$$H^{kN+i}_r(X) =
\begin{cases}
H^{2k}(X^\bullet) & \text{if } i=0\\
H^{2k+1}(X^\bullet) & \text{if } i>0 \text{ and } i+r = N\\
0 & \text{otherwise}
\end{cases}$$

It follows immediately that $\widetilde{\phantom{a}}$ sends $2$-acyclic complexes to $N$-acyclic complexes, hence descends to a triangulated functor between the derived categories.

Is this needed for anything?

Let $(X_\bullet, \alpha_\bullet), (Y_\bullet, \beta_\bullet) \in MMor_{N-2}(\A)$.  Then the natural map $\Hom_{MMor_{N-2}(\A)}(X_\bullet, Y_\bullet) \rightarrow \Hom_{D^b_N(\A)}(G(X_\bullet), G(Y_\bullet))$ induced by $G$ is surjective.  That is, any morphism $g: G(X_\bullet) \rightarrow G(Y_\bullet)$ in $D^b_N(\A)$ can be represented by a morphism of complexes of the form $G(f_\bullet)$.

Let $g: G(X_\bullet) \rightarrow G(Y_\bullet)$ be a morphism in $D^b_N(\A)$.  By Theorem <ref>, there exists an $N$-acyclic $N$-complex $P^\bullet$ of projective-injectives such that $X_\bullet = Z^0_\bullet(P^\bullet)$.  Let $Q^\bullet = (\tau_{\le -1}P^\bullet)[-N]$.  For each $1 \le i \le N-1$, the differential $d_P^{-N+i, N-i}$ factors as $Q^i = P^{-N+i} \xtwoheadrightarrow{p^i} X_i \hookrightarrow P^0$.  Extending by zero, we obtain a morphism of $N$-complexes $p^\bullet: Q^\bullet \twoheadrightarrow G(X_\bullet)$:

\begin{eqnarray*}
\begin{tikzcd}%[column sep=small]
P^{-N} \arrow[r, "d_P^{-N}"] \arrow[d] & P^{-N+1}\arrow[r, "d_P^{-N+1}"] \arrow[d, two heads, "p^1"] & \cdots \arrow[r, "d_P^{-2}"] & P^{-1} \arrow[r] \arrow[d, two heads, "p^{N-1}"] & 0 \arrow[d]  \\
0 \arrow[r] & X_1 \arrow[r, hook, "\alpha_1"] & \cdots \arrow[r, hook, "\alpha_{N-2}"] & X_{N-1} \arrow[r] & 0
\end{tikzcd}
\end{eqnarray*}

It straightforward to check that $p^\bullet$ is an $N$-quasi-isomorphism, hence $(p^\bullet)^*: \Hom_{D^b_N(\A)}(G(X_\bullet), G(Y_\bullet)) \xrightarrow{\sim} \Hom_{D^b_N(\A)}(Q^\bullet, G(Y_\bullet))$ is an isomorphism.  By Proposition <ref>, the morphism $g \circ p^\bullet : Q^\bullet \rightarrow G(Y_\bullet)$ can be represented by a morphism $f^\bullet$ of $N$-complexes.  Since $P^\bullet$ is $N$-acyclic, for each $1 \le i \le N-1$, $p^i$ is the cokernel of $d_P^{-N, i}$. We obtain a commutative diagram in $\A$ with exact rows:

\begin{eqnarray*}
\begin{tikzcd}[column sep=large]
P^{-N} \arrow[d, "f^0"] \arrow[r, "d_P^{-N, i}"]  & P^{-N+i} \arrow[d, "f^i"] \arrow[r, "p^i"] & X_{i} \arrow[r] \arrow[d, dashed, "f_i"] & 0\\
0 \arrow[r] & Y_{i} \arrow[r, "id"] & Y_{i} \arrow[r] & 0
\end{tikzcd}
\end{eqnarray*}

We claim that $\beta_i f_i = f_{i+1}\alpha_i $ for all $1 \le i < N-1$.  Precomposing with $p^i$, we have:

\begin{eqnarray*}
\beta_i f_i p^i &=& \beta_i f^i = f^{i+1} d_P^{-N+i} = f_{i+1} p^{i+1} d_P^{-N+i} = f_{i+1} \alpha_i p^i
\end{eqnarray*}

By surjectivity of $p^i$ we obtain the desired equation.  It follows that $f_\bullet : X_\bullet \rightarrow Y_\bullet$ is a morphism in $MMor_{N-2}(\A)$ and $G(f_\bullet) \circ p^\bullet = f^\bullet = g \circ p^\bullet$.  Since $(p^\bullet)^*$ is injective, we have that $G(f_\bullet) = g$ in $D^b_N(\A)$.

§ PDG MODULES

Let $k$ be a field of characteristic $p$.  Let $A$ be a finite-dimensional self-injective graded $k$-algebra.

Let $mod-A$ denote the category of fintely-generated right $A$-modules.

A $p$-complex over an abelian category $\mathcal{A}$ is a sequence $X^\bullet$ of objects of $\mathcal{A}$ with a differential $\partial^n: X^n \rightarrow X^{n+1}$ such that any composition of $p$ successive differentials is zero.  The category of $p$-complexes over $\mathcal{A}$ will be denoted $Comp_p(\mathcal{A})$.  Similarly, $Ho_p(\mathcal{A})$ and $D_p(\mathcal{A})$   Give an explicit definition of pdg modules?

Let $H = k[\partial]/(\partial^p)$ as a graded algebra, with $\partial$ in degree 1.  We can give $H$ the structure of a Hopf algebra by defining $\Delta(\partial) = \partial \otimes 1 + 1 \otimes \partial$, $\epsilon(\partial) = 0$, and $S(\partial) = -\partial$.  Let $B = A \# H$, with $H$ acting trivially on $A$.  Then the data of a $p$dg-module is equivalent to the data of a graded $B$-module.

Let $Ho^b_p(grmod-A)$ denote the homotopy category of bounded $p$-complexes.  Let $Ho^p_{pdg}(A) \cong \grstab{}{B}$ denote the homotopy category of finitely-generated $p$dg $A$-modules.

Let $D^b_p(A\grmod)$ denote the derived category of $p$-complexes over $A\grmod$ with bounded homology.   Define $D^b_{pdg}(A)$ the derived category of $p$-dg $A$-modules with finite-dimensional total homology.   Rework this definition.

Let $T: grmod-B \rightarrow grmod-B$ denote the suspension functor for $D^b_{pdg}(A)$.  Be more precise about this.

We have equivalences of categories $Ho^{b,-}_{p}(A\grproj) \rightarrow D^{-}_p(A\grmod)$.  Let $D^{perf}_p(A\grmod)$ be the closure of the image of $K^b_{p}(A\grproj)$ under isomorphisms.
We obtain an equivalence of categories
$$D^b_p(A\grmod)/D^{perf}_p(A\grmod) \cong Ho^{b,-}_p(A\grproj)/Ho^b_p(A\grproj).$$

Let $F$ be any field, and let $A = F[x]/(x^2)$.  Then $A$ has two indecomposable modules:  the one-dimensional simple module $S = x \cdot F$ and the two-dimensional projective-injective module $A$.  The irreducible morphisms in $\mmod A$ are the inclusion $S \hookrightarrow A$ and $x \cdot -:  A \twoheadrightarrow S$.  Every monomorphism from an indecomposable module $X$ into a direct sum $Y \oplus Z$ factors through either $Y$ or $Z$, so $(X_\bullet, f_\bullet) \in MMor_{k}(A)$ is indecomposable if and only if each $X_i$ is indecomposable.

To an indecomposable object $(X_\bullet, f_\bullet) \in MMor_{k}(A)$ we associate a sequence of $k+1$ integers $dim(X_1), \cdots, dim(X_{k+1})$.  The isomorphism class of $X_\bullet$ is determined by this sequence.  Thus the indecomposable objects of $MMor_k(A)$ are in bijection with the set of nondecreasing sequences of length $k+1$ with values in $\{0, 1, 2\}$.

[column sep = small]
11 [dr]         01[dr]

01 [ur] [r]     [02] [r]     12 [ur] [r]     [22] [r]     11

The Auslander-Reiten quiver of $MMor_{3}(A)$.

Vertices in brackets are projective-injective.

[column sep = small]
111 [dr]         001 [dr]

011 [dr] [ur]         112 [ur] [dr]         011 [dr]

001 [ur] [r]     [002] [r]     012 [ur] [r]     [022] [r]     122 [ur] [r]     [222] [r]     111

The Auslander-Reiten quiver of $MMor_{4}(A)$.

Vertices in parentheses are projective-injective.

[column sep = small]
1111 [dr]         0001 [dr]

0111 [ur] [dr]         1112 [ur] [dr]         0011 [dr]

0011 [ur] [dr]         0112 [ur] [dr]         1122 [ur] [dr]         0111 [dr]

0001 [r] [ur]     [0002] [r]     0012 [ur] [r]     [0022] [r]     0122 [ur] [r]     [0222] [r]     1222 [ur] [r]     [2222] [r]     1111

The Auslander-Reiten quiver of $MMor_{5}(A)$.

Vertices in parentheses are projective-injective.


# SLAMP: Stochastic Latent Appearance and Motion Prediction

Adil Kaan Akan1 Erkut Erdem2 Aykut Erdem1 Fatma Güney1
1 Koç University Is Bank AI Center, Istanbul, Turkey
2 Hacettepe University Computer Vision Lab, Ankara, Turkey
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://kuis-ai.github.io/slamp

###### Abstract

Motion is an important cue for video prediction and often utilized by
separating video content into static and dynamic components. Most of the
previous work utilizing motion is deterministic but there are stochastic
methods that can model the inherent uncertainty of the future. Existing
stochastic models either do not reason about motion explicitly or make
limiting assumptions about the static part. In this paper, we reason about
appearance and motion in the video stochastically by predicting the future
based on the motion history. Explicit reasoning about motion without history
already reaches the performance of current stochastic models. The motion
history further improves the results by allowing to predict consistent
dynamics several frames into the future. Our model performs comparably to the
state-of-the-art models on the generic video prediction datasets, however,
significantly outperforms them on two challenging real-world autonomous
driving datasets with complex motion and dynamic background.

##  1 Introduction

Videos contain visual information enriched by motion. Motion is a useful cue
for reasoning about human activities or interactions between objects in a
video. Given a few initial frames of a video, our goal is to predict several
frames into the future, as realistically as possible. By looking at a few
frames, humans can predict what will happen next. Surprisingly, they can even
attribute semantic meanings to random dots and recognize motion patterns [17].
This shows the importance of motion to infer the dynamics of the video and to
predict the future frames.

Figure 1: Comparison of the first prediction frames (11th) SLAMP (left) vs.
state-of-the-art method, SRVP [9] (right) on KITTI [11] (top) and Cityscapes
[4] (bottom) datasets. Our method can predict both foreground and background
objects better than SRVP. Full sequence predictions can be seen in Fig. 32 and
33.

Motion cues have been heavily utilized for future frame prediction in computer
vision. A common approach is to factorize the video into static and dynamic
components [35, 23, 25, 7, 10, 24, 16, 33]. First, most of the previous
methods are deterministic and fail to model the uncertainty of the future.
Second, motion is typically interpreted as local changes from one frame to the
next. However, changes in motion follow certain patterns when observed over
some time interval. Consider scenarios where objects move with near-constant
velocity, or humans repeating atomic actions in videos. Regularities in motion
can be very informative for future frame prediction. In this work, we propose
to explicitly model the change in motion, or _the motion history_ , for
predicting future frames.

Stochastic methods have been proposed to model the inherent uncertainty of the
future in videos. Earlier methods encode the dynamics of the video in
stochastic latent variables which are decoded to future frames in a
deterministic way [5]. We first assume that both appearance and motion are
encoded in the stochastic latent variables and decode them separately into
appearance and motion predictions in a deterministic way. Inspired by the
previous deterministic methods [8, 23, 10], we also estimate a mask relating
the two. Both appearance and motion decoders are expected to predict the full
frame but they might fail due to occlusions around motion boundaries.
Intuitively, we predict a probabilistic mask from the results of the
appearance and motion decoders to combine them into a more accurate final
prediction. Our model learns to use motion cues in the dynamic parts and
relies on appearance in the occluded regions.

The proposed stochastic model with deterministic decoders cannot fully utilize
the motion history, even when motion is explicitly decoded. In this work, we
propose a model to recognize regularities in motion and remember them in the
motion history to improve future frame predictions. We factorize stochastic
latent variables as static and dynamic components to model the motion history
in addition to the appearance history. We learn two separate distributions
representing appearance and motion and then decode static and dynamic parts
from the respective ones.

Our model outperforms all the previous work and performs comparably to the
state-of-the-art method, SRVP, [9] without any limiting assumptions on the
changes in the static component on the generic video prediction datasets,
MNIST, KTH and BAIR. However, our model outperforms all the previous work,
including SRVP, on two challenging real-world autonomous driving datasets with
dynamic background and complex object motion.

Figure 2: Generative Model of SLAMP. The graphical model shows the generation
process of SLAMP with motion history. There are two separate latent variables
for appearance $\mathbf{z}^{p}_{t}$ and motion $\mathbf{z}^{f}_{t}$ generating
frames $\mathbf{x}^{p}_{t}$ and $\mathbf{x}^{f}_{t}$ (black). Information is
propagated between time-steps through the recurrence between frame predictions
(blue), corresponding latent variables (green), and from frame predictions to
latent variables (red). The final prediction $\hat{\mathbf{x}}_{t}$ is a
weighted combination of the $\mathbf{x}^{p}_{t}$ and $\mathbf{x}^{f}_{t}$
according to the mask $\mathbf{m}(\mathbf{x}^{p}_{t},\mathbf{x}^{f}_{t})$.
Note that predictions at a time-step depend on all of the previous time-steps
recurrently, but only the connections between consecutive ones are shown for
clarity.

##  2 Related Work

Appearance-Motion Decomposition: The previous work explored motion cues for
video generation either explicitly with optical flow [35, 34, 22, 23, 25, 7,
10] or implicitly with temporal differences [24] or pixel-level
transformations [16, 33]. There are some common factors among these methods
such as using recurrent models [29, 24, 7], specific processing of dynamic
parts [16, 22, 7, 10], utilizing a mask [8, 23, 10], and adversarial training
[33, 25]. We also use recurrent models, predict a mask, and separately process
motion, but in a stochastic way.

The previous work which explored motion for video generation are mostly
deterministic, therefore failing to capture uncertainty of the future. There
are a couple of attempts to learn multiple future trajectories from a single
image with a conditional variational autoencoder [34] or to capture motion
uncertainty with a probabilistic motion encoder [22]. The latter work uses
separate decoders for flow and frame similar to our approach, however,
predicts them only from the latent vector. We incorporate information from
previous frames with additional modelling of the motion history.

Stochastic Video Generation: SV2P [1] and SVG [5] are the first to model the
stochasticity in video sequences using latent variables. The input from past
frames are encoded in a posterior distribution to generate the future frames.
In a stochastic framework, learning is performed by maximizing the likelihood
of the observed data and minimizing the distance of the posterior distribution
to a prior distribution, either fixed [1] or learned from previous frames [5].
Since time-variance in the model is proven crucial by the previous work, we
sample a latent variable at every time step [5]. Sampled random variables are
fed to a frame predictor, modelled recurrently using an LSTM. We model
appearance and motion distributions separately and train two frame predictors
for static and dynamic parts.

Typically, each distribution, including the prior and the posterior, is
modeled with a recurrent model such as an LSTM. Villegas et al. [32] replace
the linear LSTMs with convolutional ones at the cost of increasing the number
of parameters. Castrejon et al. [3] introduce a hierarchical representation to
model latent variables at different scales, by introducing additional
complexity. Lee et al. [20] incorporate an adversarial loss into the
stochastic framework to generate sharper images, at the cost of less diverse
results. Our model with linear LSTMs can generate diverse and sharp-looking
results without any adversarial losses, by incorporating motion information
successfully into the stochastic framework. Recent methods model dynamics of
the keypoints to avoid errors in pixel space and achieve stable learning [26].
This offers an interesting solution for videos with static background and
moving foreground objects that can be represented with keypoints. Our model
can generalize to videos with changing background without needing keypoints to
represent objects.

Optical flow has been used before in future prediction [21, 25]. Li et al.
[21] generate future frames from a still image by using optical flow generated
by an off-the-shelf model, whereas we compute flow as part of prediction. Lu
et al. [25] use optical flow for video extrapolation and interpolation without
modeling stochasticity. Long-term video extrapolation results show the
limitation of this work in terms of predicting future due to relatively small
motion magnitudes considered in extrapolation. Differently from flow, Xue et
al. [36] model the motion as image differences using cross convolutions.

State-Space Models: Stochastic models are typically auto-regressive, i.e. the
next frame is predicted based on the frames generated by the model. As opposed
to interleaving process of auto-regressive models, state-space models separate
the frame generation from the modelling of dynamics [13]. State-of-the-art
method SRVP [9] proposes a state-space model for video generation with
deterministic state transitions representing residual change between the
frames. This way, dynamics are modelled with latent state variables which are
independent of previously generated frames. Although independent latent states
are computationally appealing, they cannot model the motion history of the
video. In addition, content variable designed to model static background
cannot handle changes in the background. We can generate long sequences with
complex motion patterns by explicitly modelling the motion history without any
limiting assumptions about the dynamics of the background.

##  3 Methodology

Figure 3: SLAMP. This figure shows the components of our SLAMP model including
the prediction model, inference and learned prior models for pixel and then
flow from left to right. Observations $\mathbf{x}_{t}$ are mapped to the
latent space by using a pixel encoder for appearance on each frame and and a
motion encoder for motion between consecutive frames. The blue boxes show
encoders, yellow and green ones decoders, gray ones recurrent posterior,
prior, and predictor models, and lastly red ones show loss functions during
training. Note that $L_{2}$ loss is applied three times for appearance
prediction $\mathbf{x}_{t}^{p}$, motion prediction $\mathbf{x}_{t}^{f}$, and
the combination of the two $\hat{\mathbf{x}}_{t}$ according to the mask
prediction $\mathbf{m}(\mathbf{x}_{t}^{p},\mathbf{x}_{t}^{f})$. We only show
L2 loss between the actual frame $\mathbf{x}_{t}$ and the final predicted
frame $\hat{\mathbf{x}}_{t}$ in the figure. For inference, only the prediction
model and learned prior models are used.

###  3.1 Stochastic Video Prediction

Given the previous frames $\mathbf{x}_{1:t-1}$ until time $t$, our goal is to
predict the target frame $\mathbf{x}_{t}$. For that purpose, we assume that we
have access to the target frame $\mathbf{x}_{t}$ during training and use it to
capture the dynamics of the video in stochastic latent variables
$\mathbf{z}_{t}$. By learning to approximate the distribution over
$\mathbf{z}_{t}$, we can decode the future frame $\mathbf{x}_{t}$ from
$\mathbf{z}_{t}$ and the previous frames $\mathbf{x}_{1:t-1}$ at test time.

Using all the frames including the target frame, we compute a posterior
distribution $q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})$ and
sample a latent variable $\mathbf{z}_{t}$ from this distribution at each time
step. The stochastic process of the video is captured by the latent variable
$\mathbf{z}_{t}$. In other words, it should contain information accumulated
over the previous frames rather than only condensing the information on the
current frame. This is achieved by encouraging
$q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})$ to be close to a
prior distribution $p(\mathbf{z})$ in terms of KL-divergence. The prior can be
sampled from a fixed Gaussian

at each time step or can be learned from previous frames up to the target
frame $p_{\boldsymbol{\psi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})$. We prefer
the latter one as it is shown to work better by learning a prior that varies
across time [5].

The target frame $\mathbf{x}_{t}$ is predicted based on the previous frames
$\mathbf{x}_{1:t-1}$ and the latent vectors $\mathbf{z}_{1:t}$.

In practice, we only use the latest frame $\mathbf{x}_{t-1}$ and the latent
vector $\mathbf{z}_{t}$ as input and dependencies from further previous frames
are propagated with a recurrent model. The output of the frame predictor
$\mathbf{g}_{t}$

contains the information required to decode $\mathbf{x}_{t}$.

Typically, $\mathbf{g}_{t}$ is decoded to a fixed-variance Gaussian
distribution whose mean is the predicted target frame $\hat{\mathbf{x}}_{t}$
[5].

###  3.2 SLAMP

We call the predicted target frame, appearance prediction $\mathbf{x}_{t}^{p}$
in the pixel space. In addition to $\mathbf{x}_{t}^{p}$, we also estimate
optical flow $\mathbf{f}_{t-1:t}$ from the previous frame $t-1$ to the target
frame $t$. The flow $\mathbf{f}_{t-1:t}$ represents the motion of the pixels
from the previous frame to the target frame. We reconstruct the target frame
$\mathbf{x}_{t}^{f}$ from the estimated optical flow via differentiable
warping [15]. Finally, we estimate a mask
$\mathbf{m}(\mathbf{x}_{t}^{p},\mathbf{x}_{t}^{f})$ from the two frame
estimations to combine them into the final estimation $\hat{\mathbf{x}}_{t}$:

$\hat{\mathbf{x}}_{t}=\mathbf{m}(\mathbf{x}_{t}^{p},\mathbf{x}_{t}^{f})\odot\mathbf{x}_{t}^{p}+(\mathbf{1}-\mathbf{m}(\mathbf{x}_{t}^{p},\mathbf{x}_{t}^{f}))\odot\mathbf{x}_{t}^{f}$
(1)

where $\odot$ denotes element-wise Hadamard product and $\mathbf{x}_{t}^{f}$
is the result of warping the source frame to the target frame according to the
estimated flow field $\mathbf{f}_{t-1:t}$. Especially in the dynamic parts
with moving objects, the target frame can be reconstructed accurately using
motion information. In the occluded regions where motion is unreliable, the
model learns to rely on the appearance prediction. The mask prediction learns
a weighting between the appearance and the motion predictions for combining
them.

We call this model SLAMP-Baseline because it is limited in the sense that it
only considers the motion with respect to the previous frame while decoding
the output. In SLAMP, we extend the stochasticity in the appearance space to
the motion space as well. This way, we can model appearance changes and motion
patterns in the video explicitly and make better predictions of future. Fig. 3
shows an illustration of SLAMP (see Appendix Section A for SLAMP-Baseline).

In order to represent appearance and motion, we compute two separate posterior
distributions $q_{\boldsymbol{\phi}_{p}}(\mathbf{z}_{t}^{p}|\mathbf{x}_{1:t})$
and $q_{\boldsymbol{\phi}_{f}}(\mathbf{z}_{t}^{f}|\mathbf{x}_{1:t})$,
respectively. We sample two latent variables $\mathbf{z}_{t}^{p}$ and
$\mathbf{z}_{t}^{f}$ from these distributions in the pixel space and the flow
space. This allows a decomposition of the video into static and dynamic
components. Intuitively, we expect the dynamic component to focus on changes
and the static to what remains constant from the previous frames to the target
frame. If the background is moving according to a camera motion, the static
component can model the change in the background assuming that it remains
constant throughout the video, e.g. ego-motion of a car.

The Motion History: The latent variable $\mathbf{z}_{t}^{f}$ should contain
motion information accumulated over the previous frames rather than local
temporal changes between the last frame and the target frame. We achieve this
by encouraging
$q_{\boldsymbol{\phi}_{f}}(\mathbf{z}_{t}^{f}|\mathbf{x}_{1:t})$ to be close
to a prior distribution in terms of KL-divergence. Similar to [5], we learn
the motion prior conditioned on previous frames up to the target frame:
$p_{\boldsymbol{\psi}_{f}}(\mathbf{z}_{t}^{f}|\mathbf{x}_{1:t-1})$. We repeat
the same for the static part represented by $\mathbf{z}_{t}^{p}$ with
posterior $q_{\boldsymbol{\phi}_{p}}(\mathbf{z}_{t}^{p}|\mathbf{x}_{1:t})$ and
the learned prior
$p_{\boldsymbol{\psi}_{p}}(\mathbf{z}_{t}^{p}|\mathbf{x}_{1:t-1})$.

###  3.3 Variational Inference

For our basic formulation (SLAMP-Baseline), the derivation of the loss
function is straightforward and provided in Appendix Section B. For SLAMP, the
conditional joint probability corresponding to the graphical model in Fig. 2
is:

$\displaystyle p(\mathbf{x}_{1:T})=\prod_{t=1}^{T}$
$\displaystyle~{}~{}p(\mathbf{x}_{t}\lvert\mathbf{x}_{1:t-1},\mathbf{z}_{t}^{p},\mathbf{z}_{t}^{f})$
(2)
$\displaystyle~{}~{}p(\mathbf{z}_{t}^{p}\lvert\mathbf{x}_{1:t-1},\mathbf{z}_{t-1}^{p})~{}~{}p(\mathbf{z}_{t}^{f}\lvert\mathbf{x}_{1:t-1},\mathbf{z}_{t-1}^{f})\vspace{-3mm}$

The true distribution over the latent variables $\mathbf{z}_{t}^{p}$ and
$\mathbf{z}_{t}^{f}$ is intractable. We train time-dependent inference
networks $q_{\boldsymbol{\phi}_{p}}(\mathbf{z}_{t}^{p}\lvert\mathbf{x}_{1:T})$
and $q_{\boldsymbol{\phi}_{f}}(\mathbf{z}_{t}^{f}\lvert\mathbf{x}_{1:T})$ to
approximate the true distribution with conditional Gaussian distributions. In
order to optimize the likelihood of $p(\mathbf{x}_{1:T})$, we need to infer
latent variables $\mathbf{z}_{t}^{p}$ and $\mathbf{z}_{t}^{f}$, which
correspond to uncertainty of static and dynamic parts in future frames,
respectively. We use a variational inference model to infer the latent
variables.

Since $\mathbf{z}_{t}^{p}$ and $\mathbf{z}_{t}^{f}$ are independent across
time, we can decompose Kullback-Leibler terms into individual time steps. We
train the model by optimizing the variational lower bound (see Appendix
Section B for the derivation):

$\displaystyle\mathrm{log}\
p_{\boldsymbol{\theta}}(\mathbf{x})\geq\mathcal{L}_{\boldsymbol{\theta},\boldsymbol{\phi}_{p},\boldsymbol{\phi}_{f},\boldsymbol{\psi}_{p},\boldsymbol{\psi}_{f}}(\mathbf{x}_{1:T})$
(3) $\displaystyle={}\begin{aligned}
\sum\limits_{t}\mathbbm{E}_{\begin{subarray}{c}\mathbf{z}_{1:t}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}_{1:t}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}&~{}\mathrm{log}\
p_{\boldsymbol{\theta}}(\mathbf{x}_{t}\lvert\mathbf{x}_{1:t-1},\mathbf{z}_{1:t}^{p},\mathbf{z}_{1:t}^{f})\\\
&-\beta\Big{[}D_{\mathrm{KL}}(q(\mathbf{z}^{p}_{t}\lvert\mathbf{x}_{1:t})~{}||~{}p(\mathbf{z}^{p}_{t}\lvert\mathbf{x}_{1:t-1}))\\\
&~{}~{}~{}~{}~{}+D_{\mathrm{KL}}(q(\mathbf{z}^{f}_{t}\lvert\mathbf{x}_{1:t})~{}||~{}p(\mathbf{z}^{f}_{t}\lvert\mathbf{x}_{1:t-1}))\Big{]}\end{aligned}$

The likelihood $p_{\boldsymbol{\theta}}$, can be interpreted as an $L_{2}$
penalty between the actual frame $\mathbf{x}_{t}$ and the estimation
$\hat{\mathbf{x}}_{t}$ as defined in (1). We apply the $L_{2}$ loss to the
predictions of appearance and motion components as well.

The posterior terms for uncertainty are estimated as an expectation over
$q_{\boldsymbol{\phi}_{p}}(\mathbf{z}_{t}^{p}\lvert\mathbf{x}_{1:t})$,
$q_{\boldsymbol{\phi}_{f}}(\mathbf{z}_{t}^{f}\lvert\mathbf{x}_{1:t})$. As in
[5], we also learn the prior distributions from the previous frames up to the
target frame as
$p_{\boldsymbol{\psi}_{p}}(\mathbf{z}_{t}^{p}\lvert\mathbf{x}_{1:t-1})$,
$p_{\boldsymbol{\psi}_{f}}(\mathbf{z}_{t}^{f}\lvert\mathbf{x}_{1:t-1})$. We
train the model using the re-parameterization trick [19]. We classically
choose the posteriors to be factorized Gaussian so that all the KL divergences
can be computed analytically.

###  3.4 Architecture

We encode the frames with a feed-forward convolutional architecture to obtain
appearance features at each time-step. In SLAMP, we also encode consecutive
frame pairs into a feature vector representing the motion between them. We
then train linear LSTMs to infer posterior and prior distributions at each
time-step from encoded appearance and motion features.

Stochastic video prediction model with a learned prior [5] is a special case
of our baseline model with a single pixel decoder, we also add motion and mask
decoders. Next, we describe the steps of the generation process for the
dynamic part.

At each time step, we encode $\mathbf{x}_{t-1}$ and $\mathbf{x}_{t}$ into
$\mathbf{h}_{t}^{f}$, representing the motion from the previous frame to the
target frame. The posterior LSTM is updated based on the $\mathbf{h}_{t}^{f}$:

$\displaystyle\mathbf{h}_{t}^{f}$ $\displaystyle=$
$\displaystyle\mathrm{MotionEnc}(\mathbf{x}_{t-1},\mathbf{x}_{t})$ (4)
$\displaystyle\boldsymbol{\mu}_{\boldsymbol{\phi}_{f}(t)},\boldsymbol{\sigma}_{\boldsymbol{\phi}_{f}(t)}$
$\displaystyle=$
$\displaystyle\mathrm{LSTM}_{\boldsymbol{\phi}_{f}}(\mathbf{h}_{t}^{f})\vspace{-5mm}$

For the prior, we use the motion representation $\mathbf{h}_{t-1}^{f}$ from
the previous time step, i.e. the motion from the frame $t-2$ to the frame
$t-1$, to update the prior LSTM:

$\displaystyle\mathbf{h}_{t-1}^{f}$ $\displaystyle=$
$\displaystyle\mathrm{MotionEnc}(\mathbf{x}_{t-2},\mathbf{x}_{t-1})$ (5)
$\displaystyle\boldsymbol{\mu}_{\boldsymbol{\psi}_{f}(t)},\boldsymbol{\sigma}_{\boldsymbol{\psi}_{f}(t)}$
$\displaystyle=$
$\displaystyle\mathrm{LSTM}_{\boldsymbol{\psi}_{f}}(\mathbf{h}_{t-1}^{f})\vspace{-5mm}$

At the first time-step where there is no previous motion, we assume zero-
motion by estimating the motion from the previous frame to itself.

The predictor LSTMs are updated according to encoded features and sampled
latent variables:

$\displaystyle\mathbf{g}_{t}^{f}$ $\displaystyle=$
$\displaystyle\mathrm{LSTM}_{\boldsymbol{\theta}_{f}}(\mathbf{h}_{t-1}^{f},\mathbf{z}_{t}^{f})$
(6) $\displaystyle\boldsymbol{\mu}_{\boldsymbol{\theta}_{f}}$ $\displaystyle=$
$\displaystyle\mathrm{FlowDec}(\mathbf{g}_{t}^{f})$

There is a difference between the train time and inference time in terms of
the distribution the latent variables are sampled from. At train time, latent
variables are sampled from the posterior distribution. At test time, they are
sampled from the posterior for the conditioning frames and from the prior for
the following frames. The output of the predictor LSTMs are decoded into
appearance and motion predictions separately and combined into the final
prediction using the mask prediction (Eq. (1)).

##  4 Experiments

Figure 4: Quantitative Results on MNIST. This figure compares SLAMP to SLAMP-
Baseline, SVG [5], and SRVP [9] on MNIST in terms of PSNR (left) and SSIM
(right). SLAMP clearly outperforms our baseline model and SVG, and performs
comparably to SRVP. Vertical bars mark the length of the training sequences.
Figure 5: SLAMP-Baseline (left) vs. SLAMP (right) on MNIST. The top row shows
the ground truth, followed by the frame predictions by the final, the
appearance, the motion, and the last two rows show the mask and the optical
flow predictions with false coloring. In this challenging case with bouncing
and collisions, the baseline confuses the digits and cannot predict last
frames correctly whereas SLAMP can generate predictions very close to the
ground truth by learning smooth transitions in the motion history, as can be
seen from optical flow predictions. See Fig. 13 for the color wheel showing
the direction of flow.

We evaluate the performance of the proposed approach and compare it to the
previous methods on three standard video prediction datasets including
Stochastic Moving MNIST, KTH Actions [28] and BAIR Robot Hand [6]. We
specifically compare our baseline model (SLAMP-Baseline) and our model (SLAMP)
to SVG [5] which is a special case of our baseline with a single pixel
decoder, SAVP [20], SV2P [1], and lastly to SRVP [9]. We also compare our
model to SVG [5] and SRVP [9] on two different challenging real world
datasets, KITTI [12, 11] and Cityscapes [4], with moving background and
complex object motion. We follow the evaluation setting introduced in [5] by
generating 100 samples for each test sequence and report the results according
to the best one in terms of average performance over the frames. Our
experimental setup including training details and parameter settings can be
found in Appendix Section C. We also share the code for reproducibility.

Figure 6: Quantitative Results on KTH and BAIR. We compare our results to previous work in terms of PSNR, SSIM, and LPIPS metrics with respect to the time steps on KTH (top), and BAIR (bottom) datasets, with 95%-confidence intervals. Vertical bars mark the length of training sequences. SLAMP outperforms previous work including SVG [5], SAVP [20], SV2P [1] and performs comparably to the state of the art method SRVP [9] on both datasets. Table 1:  FVD Scores on KTH and BAIR. This table compares all the methods in terms of FVD scores with their $95\%$-confidence intervals over five different samples from the models. Our model is the second best on KTH and among top three methods on BAIR. Dataset | KTH | BAIR
---|---|---
SV2P |  636 $\pm$ 1 |  965 $\pm$ 17
SAVP |  374 $\pm$ 3 | 152 $\pm$ 9
SVG |  377 $\pm$ 6 |  255 $\pm$ 4
SRVP | 222 $\pm$ 3 |  163$\pm$ 4
SLAMP-Baseline |  236 $\pm$ 2 |  245 $\pm$ 5
SLAMP |  228 $\pm$ 5 | —

Evaluation Metrics: We compare the performance using three frame-wise metrics
and a video-level one. Peak Signal-to-Noise Ratio (PSNR), _higher better_ ,
based on $L_{2}$ distance between the frames penalizes differences in dynamics
but also favors blur predictions. Structured Similarity (SSIM), _higher
better_ , compares local patches to measure similarity in structure spatially.
Learned Perceptual Image Patch Similarity (LPIPS) [37], _lower better_ ,
measures the distance between learned features extracted by a CNN trained for
image classification. Frechet Video Distance (FVD) [31], _lower better_ ,
compares temporal dynamics of generated videos to the ground truth in terms of
representations computed for action recognition.

Stochastic Moving MNIST: This dataset contains up to two MNIST digits moving
linearly and bouncing from walls with a random velocity as introduced in [5].
Following the same training and evaluation settings as in the previous work,
we condition on the first 5 frames during training and learn to predict the
next 10 frames. During testing, we again condition on the first 5 frames but
predict the next 20 frames.

Fig. 4 shows quantitative results on MNIST in comparison to SVG [5] and SRVP
[9] in terms of PSNR and SSIM, omitting LPIPS as in SRVP. Our baseline model
with a motion decoder (SLAMP-Baseline) already outperforms SVG on both
metrics. SLAMP further improves the results by utilizing the motion history
and reaches a comparable performance to the state of the art model SRVP. This
shows the benefit of separating the video into static and dynamic parts in
both state-space models (SRVP) and auto-regressive models (ours, SLAMP). This
way, models can better handle challenging cases such as crossing digits as
shown next.

We qualitatively compare SLAMP to SLAMP-Baseline on MNIST in Fig. 5. The
figure shows predictions of static and dynamic parts as appearance and motion
predictions, as well the final prediction as the combination of the two.
According to the mask prediction, the final prediction mostly relies on the
dynamic part shown as black on the mask and uses the static component only
near the motion boundaries. Moreover, optical flow prediction does not fit the
shape of the digits but expands as a region until touching the motion region
of the other digit. This is due to the uniform black background. Moving a
black pixel in the background randomly is very likely to result in another
black pixel in the background, which means zero-loss for the warping result.
Both models can predict optical flow correctly for the most part and resort to
the appearance result in the occluded regions. However, continuity in motion
is better captured by SLAMP with the colliding digits whereas the baseline
model cannot recover from it, leading to blur results, far from the ground
truth. Note that we pick the best sample for both models among 100 samples
according to LPIPS.

KTH Action Dataset:

Figure 7: Qualitative Results on KTH We visualize the results of SLAMP on KTH
dataset. The top row shows the ground truth, followed by the frame predictions
by the final, the appearance, the motion, and the last two rows show the mask
and the optical flow predictions. The mask prediction combines the appearance
prediction (white) and the motion prediction (black) into the final
prediction. Figure 8: Subject Appearing after the Conditioning Frames. This
figure shows a case where the subject appears after conditioning frames on KTH
with ground truth (top) and a generated sample by our model (bottom). This
shows our model’s ability to capture dynamics of the dataset by generating
samples close to the ground truth, even conditioned on empty frames.

KTH dataset contains real videos where people perform a single action such as
walking, running, boxing, etc. in front of a static camera [28]. We expect our
model with motion history to perform very well by exploiting regularity in
human actions on KTH. Following the same training and evaluation settings used
in the previous work, we condition on the first 10 frames and learn to predict
the next 10 frames. During testing, we again condition on the first 10 frames
but predict the next 30 frames.

Fig. 6 and Table 1 show quantitative results on KTH in comparison to previous
approaches. Both our baseline and SLAMP models outperform previous approaches
and perform comparably to SRVP, in all metrics including FVD. A detailed
visualization of all three frame predictions as well as flow and mask are
shown in Fig. 7. Flow predictions are much more fine-grained than MNIST by
capturing fast motion of small objects such as hands or thin objects such as
legs (see Appendix Section E). The mask decoder learns to identify regions
around the motion boundaries which cannot be matched with flow due to
occlusions and assigns more weight to the appearance prediction in these
regions.

On KTH, the subject might appear after the conditioning frames. These
challenging cases can be problematic for some previous work as shown in SRVP
[9]. Our model can generate samples close to the ground truth despite very
little information on the conditioning frames as shown in Fig. 8. The figure
shows the best sample in terms of LPIPS, please see Appendix Section E for a
diverse set of samples with subjects of various poses appearing at different
time steps.

BAIR Robot Hand: This dataset contains videos of a robot hand moving and
pushing objects on a table [6]. Due to uncertainty in the movements of the
robot arm, BAIR is a standard dataset for evaluating stochastic video
prediction models. Following the training and evaluation settings used in the
previous work, we condition on the first 2 frames and learn to predict the
next 10 frames. During testing, we again condition on the first 2 frames but
predict the next 28 frames.

Figure 9: Qualitative Comparison. We compare SLAMP to SVG [5] and SRVP [9] on
KITTI (top) and Cityscapes (bottom). Our model can better capture the changes
due to ego-motion thanks to explicit modeling of motion history.

We show quantitative results on BAIR in Fig. 6 and Table 1. Our baseline model
achieves comparable results to SRVP, outperforming other methods in all
metrics except SV2P [1] in PSNR and SAVP [20] in FVD. With 2 conditioning
frames only, SLAMP cannot utilize the motion history and performs similarly to
the baseline model on BAIR (see Appendix Section D). This is simply due to the
fact that there is only one flow field to condition on, in other words, no
motion history. Therefore, we only show the results of the baseline model on
this dataset.

Table 2: Results with a Moving Background. We evaluate our model SLAMP in
comparison to SVG and SRVP on KITTI [11] and Cityscapes [4] datasets by
conditioning on 10 frames and predicting 20 frames into the future.

Models | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$)
---|---|---|---
SVG [5] | 12.70 $\pm$ 0.70 | 0.329 $\pm$ 0.030 |  0.594 $\pm$ 0.034
SRVP [9] |  13.41 $\pm$ 0.42 |  0.336 $\pm$ 0.034 | 0.635 $\pm$ 0.021
SLAMP | 13.46 $\pm$ 0.74 | 0.337 $\pm$ 0.034 | 0.537 $\pm$ 0.042

(a) KITTI [12, 11]

Models | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$)
---|---|---|---
SVG [5] | 20.42 $\pm$ 0.63 |  0.606 $\pm$ 0.023 |  0.340 $\pm$ 0.022
SRVP [9] |  20.97 $\pm$ 0.43 | 0.603 $\pm$ 0.016 | 0.447 $\pm$ 0.014
SLAMP | 21.73 $\pm$ 0.76 | 0.649 $\pm$ 0.025 | 0.2941 $\pm$ 0.022

(b) Cityscapes [4]

Real-World Driving Datasets: We perform experiments on two challenging
autonomous driving datasets: KITTI [12, 11] and Cityscapes [4] with various
challenges. Both datasets contain everyday real-world scenes with complex
dynamics due to both background and foreground motion. KITTI is recorded in
one town in Germany while Cityscapes is recorded in 50 European cities,
leading to higher diversity.

Cityscapes primarily focuses on semantic understanding of urban street scenes,
therefore contains a larger number of dynamic foreground objects compared to
KITTI. However, motion lengths are larger on KITTI due to lower frame-rate. On
both datasets, we condition on 10 frames and predict 10 frames into the future
to train our models. Then at test time, we predict 20 frames conditioned on 10
frames.

As shown in Table 2(b), SLAMP outperforms both methods on all of the metrics
on both datasets, which shows its ability to generalize to the sequences with
moving background. Even SVG [5] performs better than the state of the art SRVP
[9] in LPIPS metric for KITTI and on both SSIM and LPIPS for Cityscapes, which
shows the limitations of SRVP on scenes with dynamic backgrounds. We also
perform a qualitative comparison to these methods in Fig. 1 and Fig. 9. SLAMP
can better preserve the scene structure thanks to explicit modeling of ego-
motion history in the background.

Visualization of Latent Space: We visualize stochastic latent variables of the
dynamic component on KTH compared to the static and SVG. (see Fig. 14 and Fig.
15)

##  5 Conclusion

We presented a stochastic video prediction framework to decompose video
content into appearance and dynamic components. Our baseline model with
deterministic motion and mask decoders outperforms SVG, which is a special
case of our baseline model. Our model with motion history, SLAMP, further
improves the results and reaches the performance of the state of the art
method SRVP on the previously used datasets. Moreover, it outperforms both SVG
and SRVP on two real-world autonomous driving datasets with dynamic background
and complex motion. We show that motion history enriches model’s capacity to
predict future, leading to better predictions in challenging cases.

Our model with motion history cannot realize its full potential in standard
settings of stochastic video prediction datasets. A fair comparison is not
possible on BAIR due to the little number of conditioning frames. BAIR holds a
great promise with changing background but infrequent, small changes are not
reflected in current evaluation metrics.

An interesting direction is stochastic motion decomposition, maybe with
hierarchical latent variables, for modelling camera motion and motion of each
object in the scene separately.

#### Acknowledgements.

We would like to thank Jean-Yves Franceschi and Edouard Delasalles for
providing technical and numerical details for the baseline performances, and
Deniz Yuret for helpful discussions and comments. K. Akan was supported by
KUIS AI Center fellowship, F. Güney by TUBITAK 2232 International Fellowship
for Outstanding Researchers Programme, E. Erdem in part by GEBIP 2018 Award of
the Turkish Academy of Sciences, A. Erdem by BAGEP 2021 Award of the Science
Academy.

## References

  * [1] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine.  Stochastic variational video prediction.  In Proc. of the International Conf. on Learning Representations (ICLR), 2018.
  * [2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer.  Scheduled sampling for sequence prediction with recurrent neural networks.  In Advances in Neural Information Processing Systems, 2015.
  * [3] Lluis Castrejon, Nicolas Ballas, and Aaron Courville.  Improved conditional vrnns for video prediction.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [5] Emily Denton and Rob Fergus.  Stochastic video generation with a learned prior.  In Proc. of the International Conf. on Machine learning (ICML), 2018\.
  * [6] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine.  Self-supervised visual planning with temporal skip connections.  In 1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, 2017.
  * [7] Hehe Fan, Linchao Zhu, and Yi Yang.  Cubic lstms for video prediction.  In Proc. of the Conf. on Artificial Intelligence (AAAI), 2019.
  * [8] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In Advances in Neural Information Processing Systems (NeurIPS), 2016\.
  * [9] Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari.  Stochastic latent residual video prediction.  In Proc. of the International Conf. on Machine learning (ICML), 2020\.
  * [10] Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang, Fisher Yu, and Trevor Darrell.  Disentangling propagation and generation for video prediction.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [11] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The KITTI dataset.  International Journal of Robotics Research (IJRR), 2013.
  * [12] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [13] Karol Gregor and Frederic Besse.  Temporal difference variational auto-encoder.  In Proc. of the International Conf. on Learning Representations (ICLR), 2018.
  * [14] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  2018\.
  * [15] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu.  Spatial transformer networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2015\.
  * [16] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2016\.
  * [17] Gunnar Johansson.  Visual perception of biological motion and a model for its analysis.  Perception & Psychophysics, 14(2):201–211, jun 1973.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, 2015.
  * [19] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In Proc. of the International Conf. on Learning Representations (ICLR), 2014.
  * [20] Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.  Stochastic adversarial video prediction.  arXiv.org, 2018.
  * [21] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Flow-grounded spatial-temporal video prediction from still images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 600–615, 2018.
  * [22] Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P. Xing.  Dual motion gan for future-flow embedded video prediction.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
  * [23] Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
  * [24] William Lotter, Gabriel Kreiman, and David Cox.  Deep predictive coding networks for video prediction and unsupervised learning.  In Proc. of the International Conf. on Learning Representations (ICLR), 2017.
  * [25] Chaochao Lu, Michael Hirsch, and Bernhard Scholkopf.  Flexible spatio-temporal networks for video prediction.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [26] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee.  Unsupervised learning of object structure and dynamics from videos.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [27] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In 4th International Conference on Learning Representations, 2016\.
  * [28] Christian Schüldt, Ivan Laptev, and Barbara Caputo.  Recognizing human actions: A local svm approach.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2004.
  * [29] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai kin Wong, and Wang chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In Advances in Neural Information Processing Systems (NeurIPS), 2015\.
  * [30] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [31] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  arXiv.org, 2019.
  * [32] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee.  High fidelity video prediction with large stochastic recurrent neural networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [33] C. Vondrick and A. Torralba.  Generating the future with adversarial transformers.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [34] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert.  An uncertain future: Forecasting from static images using variational autoencoders.  In Proc. of the European Conf. on Computer Vision (ECCV), 2016.
  * [35] Jacob Walker, Abhinav Gupta, and Martial Hebert.  Dense optical flow prediction from a static image.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2015.
  * [36] Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman.  Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks.  In Advances In Neural Information Processing Systems, 2016.
  * [37] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2018.

In this part, we provide additional illustrations, derivations, and results
for our paper “SLAMP: Stochastic Latent Appearance and Motion Prediction”. We
first show the model illustrations of our proposed model (SLAMP) and our
baseline model (SLAMP-Baseline) in comparison to the previous work by Denton
et al. [5] (SVG) in Section A. In Section B, we provide the full derivations
of the variational inference, evidence lower bounds of our baseline model
(Section B.1) and our proposed model (Section B.2). We explain the
architectural choices and training details in Section C. In Section D, we
present detailed versions of the quantitative results in the main paper. In
addition, we present the ablation experiments for our model’s mask component.
We evaluate and compare the predictions of static, dynamic heads of the model,
simple averaging of the two without a mask, and our full model with learned
mask. In Section E, we first provide the color wheel to interpret optical flow
predictions, and a comparison of static and dynamic latent variables. We then
present several qualitative results both with details as in the main paper and
with random samples, showing the diversity of the generated samples on all
datasets.

For video examples, please visit https://kuis-ai.github.io/slamp/.

##  Appendix A Model Illustrations

In Fig. 10, we provide the inference procedure of our model SLAMP, in addition
to the training procedure provided in the main paper. Moreover, we present
graphical illustrations of the training (Fig. 11) and the inference procedures
(Fig. 12) of our baseline model, SLAMP-Baseline, in comparison to SVG [5].

Figure 10: Illustration of the Inference Procedure for SLAMP. This figure
illustrates the difference between the inference time and the train time in
terms of the distributions the latent variables are sampled from. While at
train time, latent variables are sampled from the posterior distribution, at
test time, they are sampled from the posterior for the conditioning frames and
from the prior for the following frames.

Figure 11: Illustration of the Training Procedure for SVG (left) and SLAMP-
Baseline (right). The main difference between SVG and SLAMP-Baseline is that
SLAMP-Baseline has three decoders instead of one pixel decoder. In SLAMP-
Baseline, in addition to the appearance prediction $\mathbf{x}_{t}^{p}$, we
also estimate flow $\mathbf{f}_{t-1:t}$ and warp the previous frame according
to the estimated flow to obtain motion prediction $\mathbf{x}_{t}^{f}$. Mask
decoder takes appearance and motion predictions as input and generates a
weighted combination of the two, $\hat{\mathbf{x}}_{t}$ as the final
prediction. Note that SVG corresponds to only appearance prediction case of
our baseline model.

Figure 12: Illustration of the Inference Procedure for SVG (left) and SLAMP-
Baseline (right). This figure illustrates the inference time in comparison to
the train time in terms of the distribution the latent variables are sampled
from. While at train time, latent variables are sampled from the posterior
distribution, at test time, they are sampled from the posterior for the
conditioning frames and from the prior for the following frames.

##  Appendix B Derivations

Here, we provide derivations of inference steps and variational lower bounds
of the baseline method, SLAMP-Baseline (Section B.1), and our method SLAMP
(Section B.2).

###  B.1 Derivation of the ELBO for SLAMP-Baseline

We first derive the variational lower bound for the baseline model with one
posterior and one learned prior distribution.

$\displaystyle\log p_{\boldsymbol{\theta}}(\mathbf{x})$ $\displaystyle=$
$\displaystyle\log\int\limits_{\mathbf{z}}p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})~{}p(\mathbf{z}|\mathbf{x})$
$\displaystyle=$
$\displaystyle\log\int\limits_{\mathbf{z}}p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})~{}p(\mathbf{z}|\mathbf{x})~{}\frac{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}$
$\displaystyle=$
$\displaystyle\log\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\frac{p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})~{}p(\mathbf{z}|\mathbf{x})}{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}$
$\displaystyle\geq$
$\displaystyle\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\log\frac{p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})~{}p(\mathbf{z}|\mathbf{x})}{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})-\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}{p(\mathbf{z}|\mathbf{x})}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})~{}||~{}p(\mathbf{z}|\mathbf{x}))$

We model the posterior distribution with a recurrent network. The recurrent
network outputs a different posterior distribution,
$q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})$, at every time step.
Due to independence of latent variables across time,
$\mathbf{z}=[\mathbf{z}_{1},\mathbf{z}_{2},\dots,\mathbf{z}_{T}]$, we can
derive the estimation of posterior distribution across time steps as follows:

$\displaystyle
q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})=\prod\limits_{t}q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})$
(8)

Since the latent variables,
$\mathbf{z}=[\mathbf{z}_{1},\mathbf{z}_{2},\cdots,\mathbf{z}_{T}]$, are
independent across time, we can further decompose Kullback-Leibler term in the
evidence lower bound into individual time steps:

$\displaystyle
D_{\mathrm{KL}}\left(\left.\left.q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})\right|\right|p(\mathbf{z}|\mathbf{x}_{1:t-1})\right)$
(9)
$\displaystyle=\int\limits_{\mathbf{z}}q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})~{}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}{p(\mathbf{z}|\mathbf{x}_{1:t-1})}$
$\displaystyle=\int\limits_{\mathbf{z}_{1}}\cdots\int\limits_{\mathbf{z}_{T}}q_{\boldsymbol{\phi}}(\mathbf{z}_{1}|\mathbf{x}_{1})\cdots
q_{\boldsymbol{\phi}}(\mathbf{z}_{T}|\mathbf{x}_{1:T})~{}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}_{1}|\mathbf{x}_{1})\cdots
q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:T})}{p(\mathbf{z}_{1}|\mathbf{x}_{1})\cdots
p(\mathbf{z}_{T}|\mathbf{x}_{1:T-1})}$
$\displaystyle=\int\limits_{\mathbf{z}_{1}}\cdots\int\limits_{\mathbf{z}_{T}}q_{\boldsymbol{\phi}}(\mathbf{z}_{1}|\mathbf{x}_{1})\cdots
q_{\boldsymbol{\phi}}(\mathbf{z}_{T}|\mathbf{x}_{1:T})~{}\sum\limits_{t}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})}{p(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})}$
$\displaystyle=\sum\limits_{t}\int\limits_{\mathbf{z}_{1}}\cdots\int\limits_{\mathbf{z}_{T}}q_{\boldsymbol{\phi}}(\mathbf{z}_{1}|\mathbf{x}_{1})\cdots
q_{\boldsymbol{\phi}}(\mathbf{z}_{T}|\mathbf{x}_{1:T})~{}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})}{p(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})}$
$\displaystyle\text{And because }\int\limits_{x}p(x)=1\text{, this simplifies
to:}$
$\displaystyle=\sum\limits_{t}\int\limits_{\mathbf{z}_{t}}q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})~{}\log\frac{q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})}{p(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})}$
$\displaystyle=\sum\limits_{t}D_{\mathrm{KL}}\left(\left.\left.q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t})\right|\right|p(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})\right)$

At each time step, our model predicts $\mathbf{x}_{t}$, conditioned on
$\mathbf{x}_{t-1}$ and $\mathbf{z}_{t}$. Since our model has recurrence
connections, it considers not only $\mathbf{x}_{t-1}$ and $\mathbf{z}_{t}$,
but also $\mathbf{x}_{1:t-2}$ and $\mathbf{z}_{1:t-1}$. Therefore, we can
further write our inference as:

$\displaystyle\log p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})$
$\displaystyle=\log\prod\limits_{t}p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}_{1:t})$
(10) $\displaystyle=\sum\limits_{t}\log
p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}_{1:t})$

Combining all of them leads to the following variational lower bound:

$\displaystyle\log p_{\boldsymbol{\theta}}(\mathbf{x})$
$\displaystyle\geq\mathcal{L}_{\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\psi}}(\mathbf{x}_{1:T})$
(11)
$\displaystyle=\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z})-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}}(\mathbf{z}|\mathbf{x})~{}||~{}p_{\boldsymbol{\psi}}(\mathbf{z}|\mathbf{x}))$
$\displaystyle=\sum\limits_{t}\big{[}\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}_{1:t}|\mathbf{x}_{1:t})}\log
p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}_{1:t})$
$\displaystyle\hskip 36.98857pt-
D_{\mathrm{KL}}(q_{\boldsymbol{\phi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t}~{}||~{}p_{\boldsymbol{\psi}}(\mathbf{z}_{t}|\mathbf{x}_{1:t-1}))\big{]}$

###  B.2 Derivation of the ELBO for SLAMP

In this section, we derive the variational lower bound for the proposed model
with two posterior and two learned prior distributions.

$\displaystyle\log p_{\boldsymbol{\theta}}(\mathbf{x})$
$\displaystyle=\log\int\limits_{\mathbf{z}^{p}}\int\limits_{\mathbf{z}^{f}}p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})~{}p(\mathbf{z}^{p}|\mathbf{x})~{}p(\mathbf{z}^{f}|\mathbf{x})$
(12)
$\displaystyle=\log\int\limits_{\mathbf{z}^{p}}\int\limits_{\mathbf{z}^{f}}p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})~{}p(\mathbf{z}^{p}|\mathbf{x})~{}p(\mathbf{z}^{f}|\mathbf{x})~{}\frac{q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})}{q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})}~{}\frac{q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})}{q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})}$
$\displaystyle=\log\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\frac{p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})~{}p(\mathbf{z}^{p}|\mathbf{x})~{}p(\mathbf{z}^{f}|\mathbf{x})}{q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})~{}q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})}$
$\displaystyle\geq\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\log\frac{p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})~{}p(\mathbf{z}^{p}|\mathbf{x})~{}p(\mathbf{z}^{f}|\mathbf{x})}{q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})~{}q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})}$
$\displaystyle=\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})-\mathbb{E}_{\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}}\log\frac{q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})}{p(\mathbf{z}^{p}|\mathbf{x})}-\mathbb{E}_{\mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}}\log\frac{q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})}{p(\mathbf{z}^{f}|\mathbf{x})}$
$\displaystyle=\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})~{}||~{}p(\mathbf{z}^{p}|\mathbf{x}))-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})~{}||~{}p(\mathbf{z}^{f}|\mathbf{x}))$

We model the posterior distributions with two recurrent networks. The
recurrent networks output two different posterior distributions,
$q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}_{t}|\mathbf{x}_{1:t})$ and
$q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}_{t}|\mathbf{x}_{1:t})$, at every
time step. Due to the independence of the latent variables across time,
$\mathbf{z}^{p}=[\mathbf{z}^{p}_{1},\mathbf{z}^{p}_{2},\cdots,\mathbf{z}^{p}_{T}]$
and
$\mathbf{z}^{f}=[\mathbf{z}^{f}_{1},\mathbf{z}^{f}_{2},\cdots,\mathbf{z}^{f}_{T}]$,
we can derive the estimation of posterior distributions across time steps as
follows:

$\displaystyle q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})$
$\displaystyle=\prod\limits_{t}q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}_{t}|\mathbf{x}_{1:t})$
$\displaystyle q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})$
$\displaystyle=\prod\limits_{t}q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}_{t}|\mathbf{x}_{1:t})$
(13)

Since the latent variables,
$\mathbf{z}^{p}=[\mathbf{z}^{p}_{1},\mathbf{z}^{p}_{2},\dots,\mathbf{z}^{p}_{T}]$
and
$\mathbf{z}^{f}=[\mathbf{z}^{f}_{1},\mathbf{z}^{f}_{2},\dots,\mathbf{z}^{f}_{T}]$,
are independent across time and independent from each other, we can further
decompose Kullback-Leibler terms in the evidence lower bound into individual
time steps as in Eq. (9).

At each time step, our model predicts $\mathbf{x}_{t}$, conditioned on
$\mathbf{x}_{t-1}$, $\mathbf{z}^{p}_{t}$, $\mathbf{z}^{f}_{t}$. Since our
model has recurrence connections, it considers not only $\mathbf{x}_{t-1}$,
$\mathbf{z}^{p}_{t}$ and $\mathbf{z}^{f}_{t}$, but also $\mathbf{x}_{1:t-2}$,
$\mathbf{z}^{p}_{1:t-1}$ and $\mathbf{z}^{f}_{1:t-1}$. Therefore, we can
further write our inference as:

$\displaystyle\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})$
$\displaystyle=\log\prod\limits_{t}p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}^{p}_{1:t},\mathbf{z}^{f}_{1:t})$
(14) $\displaystyle=\sum\limits_{t}\log
p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}^{p}_{1:t},\mathbf{z}^{f}_{1:t})$

Combining all of them leads to the following variational lower bound:

$\displaystyle\log p_{\boldsymbol{\theta}}(\mathbf{x})$
$\displaystyle\geq\mathcal{L}_{\boldsymbol{\theta},\boldsymbol{\phi}_{p},\boldsymbol{\psi}_{f}}(\mathbf{x}_{1:T})$
(15) $\displaystyle=\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\log
p_{\boldsymbol{\theta}}(\mathbf{x}|\mathbf{z}^{p},\mathbf{z}^{f})-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}|\mathbf{x})~{}||~{}p_{\boldsymbol{\psi}_{p}}(\mathbf{z}^{p}|\mathbf{x}))-D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}|\mathbf{x})~{}||~{}p_{\boldsymbol{\psi}_{f}}(\mathbf{z}^{f}|\mathbf{x}))$
$\displaystyle=\sum\limits_{t}\mathbb{E}_{\begin{subarray}{c}\mathbf{z}^{p}\sim
q_{\boldsymbol{\phi}_{p}}\\\ \mathbf{z}^{f}\sim
q_{\boldsymbol{\phi}_{f}}\end{subarray}}\log
p_{\boldsymbol{\theta}}(\mathbf{x}_{t}|\mathbf{x}_{1:t-1},\mathbf{z}^{p}_{1:t},\mathbf{z}^{f}_{1:t})$
$\displaystyle\hskip 42.67912pt-
D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{p}}(\mathbf{z}^{p}_{t}|\mathbf{x}_{1:t}~{}||~{}p_{\boldsymbol{\psi}_{p}}(\mathbf{z}^{p}_{t}|\mathbf{x}_{1:t-1}))$
$\displaystyle\hskip 42.67912pt-
D_{\mathrm{KL}}(q_{\boldsymbol{\phi}_{f}}(\mathbf{z}^{f}_{t}|\mathbf{x}_{1:t}~{}||~{}p_{\boldsymbol{\psi}_{f}}(\mathbf{z}^{f}_{t}|\mathbf{x}_{1:t-1}))$

##  Appendix C Training Details

We provide training details including scheduled sampling (Section C.1),
architecture details (Section C.2), and the hyper-parameters used in the
optimization (Section C.3).

###  C.1 Scheduled Sampling

Scheduled sampling proposed for sequence prediction [2] has been proven useful
for several tasks where predictions need to be made based on the generated
results from the previous time steps. We also experiment with scheduled
sampling as part of our training procedure. Scheduled sampling prevents the
model from conditioning on ground-truth perfect samples which are not
available at test time. This is achieved by allowing the model to slowly
encounter generated samples instead of ground-truth perfect samples. The ratio
of ground-truth perfect samples over generated samples is decreased throughout
the training. Specifically, we apply inverse sigmoid decay. We report the
scores with and without scheduled sampling for the proposed models, both
SLAMP-Baseline and SLAMP, on all datasets. As can be seen from Table 3,
scheduled sampling is not crucial but it improves the results on KTH,
especially for SLAMP.

Table 3:  Ablation Study. This table shows the quantitative results comparing SLAMP-Baseline and SLAMP with scheduled sampling (\+ SS) and without during training. Following the previous work, we report the results as the mean and the $95\%$-confidence interval in terms of PSNR, SSIM, and LPIPS on all the datasets except LPIPS on MNIST.  | Models | PSNR | SSIM | LPIPS
---|---|---|---|---
MNIST | SLAMP-Baseline |  $16.83$ p m 0.06 |  $0.7537$ p m 0.0017 | —
SLAMP-Baseline + SS |  $16.32$ p m 0.06 |  $0.7343$ p m 0.0016 | —
SLAMP |  $18.07$ p m 0.07 |  $0.7736$ p m 0.0019 | —
SLAMP + SS |  $17.54$ p m 0.08 |  $0.7567$ p m 0.0018 | —
KTH | SLAMP-Baseline |  $28.47$ p m 0.27 |  $0.8527$ p m 0.0053 |  $0.0896$ p m 0.0038
SLAMP-Baseline + SS |  $29.20$ p m 0.28 |  $0.8633$ p m 0.0048 |  $0.0951$ p m 0.0036
SLAMP |  $28.91$ p m 0.28 |  $0.8604$ p m 0.0049 |  $0.0860$ p m 0.0037
SLAMP + SS |  $29.39$ p m 0.30 |  $0.8646$ p m 0.0049 |  $0.0795$ p m 0.0033
BAIR | SLAMP-Baseline |  $19.60$ p m 0.26 |  $0.8175$ p m 0.0083 |  $0.0596$ p m 0.0031
SLAMP-Baseline + SS |  $19.55$ p m 0.26 |  $0.8171$ p m 0.0083 |  $0.0634$ p m 0.0034
SLAMP |  $19.67$ p m 0.26 |  $0.8161$ p m 0.0086 |  $0.0639$ p m 0.0037
SLAMP + SS |  $19.75$ p m 0.26 |  $0.8160$ p m 0.0084 |  $0.0661$ p m 0.0035

###  C.2 Architecture Details

Encoders and Decoders: For all encoders and decoders, we use the same
architectures as the previous work [5, 9]: a DCGAN [27] generator and
discriminator for MNIST, and a VGG16 architecture [30] for KTH and BAIR
datasets. In all datasets, we encode the image into an appearance feature
vector of size $h_{appearance}=128$ and the two consecutive images into a
motion feature vector of size $h_{motion}=128$. Compared to SVG, there are two
more decoders for predicting flow and mask in our models. For flow decoder, we
use the same decoder with two output channels representing motion in
horizontal and vertical direction. See below for the details of the mask
decoder. For SLAMP, motion encoder takes concatenated frame pair as input and
outputs a feature vector encoding motion from one frame to the next.

Similar to previous work [5, 9], we also use skip connections but with a minor
modification. In the previous work, the skip connection from either the last
conditioning frame or last generated frame is used. Instead, we take the
running average of all the skip connections from seen or generated frames. For
example, at time step 15, we use the average of previous 14 skip connections
that are generated.

Mask Predictor: For mask predictor, we use a 5-layer CNN with 2 Squeeze and
Excitation layers (SE-Layer) [14] after each two convolutional layers. In the
CNN, we use 64-channel filters at each layer and do not reduce the resolution
by using $3\times 3$ kernels with padding. We simply concatenate the output of
pixel decoder and warped prediction along their channel axis and feed it into
mask predictor which outputs a one-channel image. We apply sigmoid at the end
to map the output to the range between 0 and 1, representing the weight to
combine the appearance and the motion predictions.

LSTMs and Latent Variables: For prior, posterior, and frame predictor LSTMs,
we use the settings proposed in SVG [5]. All LSTMs have 256 neurons and all
prior and posterior LSTMs have one layer whereas the frame predictor LSTMs
have two layers. For the size of the latent variables, we use $20$, $50$, $64$
for MNIST, KTH, and BAIR, respectively.

###  C.3 Optimization Hyper-Parameters

All the models are trained with Adam optimizer [18], with decay rates
$\beta_{1}=0.9$ and $\beta_{2}=0.999$. We train each model for 300 epochs
where each epoch consists of 1000 updates, unless otherwise is specified. We
take the model which performs the best in the validation set. We will share
the trained models upon publication for replicating the results. Dataset-
specific parameters for each dataset are as follows:

MNIST: The batch size is chosen to be 32, learning rate is $3\times 10^{-4}$
and $\beta=1\times 10^{-4}$. We continue training the models on MNIST with a
lower learning rate, $1\times 10^{-5}$, a lower $\beta=5\times 10^{-5}$, and a
lower $\beta_{1}=0.7$.

KTH: The batch size is chosen to be 20, learning rate is $1\times 10^{-4}$ and
$\beta=1\times 10^{-6}$. We apply scheduled sampling with inverse sigmoid
decay.

BAIR: The batch size is chosen to be 20, learning rate is $1\times 10^{-4}$
and $\beta=1\times 10^{-4}$.

Training details for KITTI and Cityscapes:

We use $92\times 310$ image resolution for KITTI and $128\times 256$ for
Cityscapes. We replaced LSTMs with ConvLSTMs and used $3\times 10$
intermediate feature size for KITTI, $4\times 8$ for Cityscapes. We used a
shared encoder to downsample the image first and then, use two separate
encoders for pixel and motion encoders to make model less powerful. We
increased the number of layers in the shared encoder to downsample the higher
resolution image, and preserve the VGG-basedd structure.

For SVG, we use the same settings as SLAMP. For SRVP, we use the same shared
encoder and use a channel pooling at the end to make the convolutional feature
vector compatible with the rest of the architecture.

We train all the models until the models see $2.4M$ video samples. We use the
largest batch size that we could use and choose the learning rate $1\times
10^{-4}$ for all the models.

##  Appendix D Detailed Quantitative Results

In this section, we provide a detailed version of the quantitative results
presented in the main paper in Figure 3 and 5.

We compare the performance of SLAMP-Baseline and SLAMP to the previous work in
terms of PNSR, SSIM, and LPIPS averaged over all time steps on MNIST (Table
4), KTH (Table 5), and BAIR (Table 6) datasets. Confirming the results in the
main paper, the proposed model SLAMP with motion history outperforms both the
baseline model, SLAMP-Baseline, and the previous work [5, 1, 20] and performs
comparably to the state of the art model SRVP [9]. See the main paper for a
detailed analysis.

Table 4:  Results on MNIST. This table compares the results of SLAMP and SLAMP-Baseline to the previous work on MNIST dataset. Following the previous work, we report the results as the mean and the $95\%$-confidence interval in terms of PSNR and SSIM. Bold and underlined scores indicate the best and the second best performing method, respectively.  Models | PSNR | SSIM
---|---|---
SVG [5] | 14.50 $\pm$ 0.04 | 0.7090 $\pm$ 0.0015
SRVP [9] | $16.93\pm 0.07$ | 0.7799 $\pm$ 0.0020
SLAMP-Baseline | 16.83 $\pm$ 0.06 | 0.7537 $\pm$ 0.0018
SLAMP | 18.07 $\pm$ 0.08 | $0.7736\pm 0.0019$
Table 5:  Results on KTH. This table compares the results of SLAMP and SLAMP-Baseline to the previous work on KTH dataset. Following the previous work, we report the results as the mean and the $95\%$-confidence interval in terms of PSNR, SSIM, and LPIPS. Bold and underlined scores indicate the best and the second best performing method, respectively.  Models | PSNR | SSIM | LPIPS
---|---|---|---
SV2P [1] | 28.19 $\pm$ 0.31 | 0.8141 $\pm$ 0.0050 | 0.2049 $\pm$ 0.0053
SAVP [20] | 26.51 $\pm$ 0.29 | 0.7564 $\pm$ 0.0062 | 0.1120 $\pm$ 0.0039
SVG [5] | 28.06 $\pm$ 0.29 | 0.8438 $\pm$ 0.0054 | 0.0923 $\pm$ 0.0038
SRVP [9] | 29.69 $\pm$ 0.32 | 0.8697 $\pm$ 0.0046 | 0.0736 $\pm$ 0.0029
SLAMP-Baseline | 29.20 $\pm$ 0.28 | 0.8633 $\pm$ 0.0048 | 0.0951 $\pm$ 0.0036
SLAMP | $29.39\pm 0.30$ | $0.8646\pm 0.0050$ | $0.0795\pm 0.0034$
Table 6:  Results on BAIR. This table compares the results of SLAMP and SLAMP-Baseline to the previous work on BAIR dataset. Following the previous work, we report the results as the mean and the $95\%$-confidence interval in terms of PSNR, SSIM, and LPIPS. Bold and underlined scores indicate the best and the second best performing method, respectively.  Models | PSNR | SSIM | LPIPS
---|---|---|---
SV2P [1] | 20.39 $\pm$ 0.27 | 0.8169 $\pm$ 0.0086 | 0.0912 $\pm$ 0.0053
SAVP [20] | 18.44 $\pm$ 0.25 | 0.7887 $\pm$ 0.0092 | 0.0634 $\pm$ 0.0026
SVG [5] | 18.95 $\pm$ 0.26 | 0.8058 $\pm$ 0.0088 | 0.0609 $\pm$ 0.0034
SRVP [9] | 19.59 $\pm$ 0.27 | 0.8196 $\pm$ 0.0084 | 0.0574 $\pm$ 0.0032
SLAMP-Baseline | 19.60 $\pm$ 0.26 | $0.8175\pm 0.0084$ | $0.0596\pm 0.0032$
SLAMP | $19.67\pm 0.26$ | 0.8161 $\pm$ 0.0086 | 0.0639 $\pm$ 0.0037

In addition, we provide detailed results corresponding to the components of
our model. We evaluate the result of the static head, the dynamic head, and
simply their average without the learned mask and compare them to our full
model with the learned mask in Table 7. Our full model, SLAMP, performs the
best in all datasets according to all three evaluation metrics by combining
the two predictions according to the mask prediction.

Table 7:  Mask Ablation Study. This table shows the quantitative results comparing the components of SLAMP model. Static refers to the evaluation of the prediction of the static head directly, Dynamic refers to the evaluation of the prediction of the dynamic head which uses optical flow to predict the next frame, Average refers to average of static and dynamic heads without using the mask prediction. The last row for each dataset show the results of our model which uses the mask prediction to fuse the static and dynamic predictions. Following the previous work, we report the results as the mean and the $95\%$-confidence interval in terms of PSNR, SSIM, and LPIPS on all the datasets except LPIPS on MNIST.  | Models | PSNR | SSIM | LPIPS
---|---|---|---|---
MNIST | SLAMP - Static | 15.03 $\pm$ 0.04 | 0.7273 $\pm$ 0.0014 | —
SLAMP - Dynamic | 17.64 $\pm$ 0.09 | 0.7639 $\pm$ 0.0019 | —
SLAMP - Average | 15.96 $\pm$ 0.05 | 0.7377 $\pm$ 0.0017 | —
SLAMP | 18.07 $\pm$ 0.07 | 0.7736 $\pm$ 0.0019 | —
KTH | SLAMP - Static | 28.12 $\pm$ 0.28 | 0.8410 $\pm$ 0.0056 | 0.0844 $\pm$ 0.0037
SLAMP - Dynamic | 16.14 $\pm$ 0.14 | 0.7614 $\pm$ 0.0064 | 0.3689 $\pm$ 0.0080
SLAMP - Average | 21.61 $\pm$ 0.11 | 0.8359 $\pm$ 0.0048 | 0.2039 $\pm$ 0.0054
SLAMP | 29.39 $\pm$ 0.30 | 0.8646 $\pm$ 0.0049 | 0.0795 $\pm$ 0.0033
BAIR | SLAMP - Static | 19.38 $\pm$ 0.26 | 0.8119 $\pm$ 0.00837 | 0.0606 $\pm$ 0.0033
SLAMP - Dynamic | 16.65 $\pm$ 0.20 | 0.7643 $\pm$ 0.0010 | 0.1176 $\pm$ 0.0048
SLAMP - Average | 18.51 $\pm$ 0.20 | 0.8073 $\pm$ 0.0088 | 0.0897 $\pm$ 0.0044
SLAMP | 19.75 $\pm$ 0.26 | 0.8160 $\pm$ 0.0084 | 0.0661 $\pm$ 0.0035

##  Appendix E Additional Visualizations and Qualitative Results

###  E.1 Optical Flow Visualization

Figure 13: Optical Flow False Coloring. Colors on the wheel indicate the
direction of motion in 2D.

Fig. 13 shows the color wheel used to visualize the optical flow with false
coloring. Colors show the direction of motion and the intensity of color in
the visualizations show the magnitude of motion, i.e. intense colors for large
motions. By following the usual practice in optical flow, we predict flow from
the target frame to the current frame and apply inverse warping to obtain the
target frame. Therefore, the direction of motion is also inverse, i.e. the
opposite direction on the wheel shows the motion from the current frame to the
target frame.

###  E.2 Comparison of Static and Dynamic Latent Variables

In Fig. 8 of the main paper, we provide a visualization of stochastic latent
variables of the dynamic component on KTH using t-SNE. Here, we provide both
the static and the dynamic components for a comparison. The same colors from
the main paper show the semantic classes of video frames plotted. As can be
seen from Fig. 14, static variables on the right are more scattered and do not
from clusters according to semantic classes as in the dynamic variables on the
left (and in the main paper). This shows that our model can learn video
dynamics according to semantic classes with separate modelling of the dynamic
component.

Figure 14: Dynamic (left) vs. Static (right) Latent Variables. This figure
shows the T-SNE visualization of dynamic and static latent variables on 300
test videos from KTH dataset. In dynamic latent variables, different classes
with similar repetitive movements such as walking, running, and jogging are
clustered together. However, in static latent variables, points are more
scattered and do not form clusters according to semantic actions.

In Fig. 15, we visualize the latent variables of SVG [5] to show the
difference between our architecture’s latent variables and SVG’s latent
variables. In our architecture, dynamic branch learns the similar repetitive
movements whereas static branch learns the general image information.
Therefore, dynamic latent variables form cluster around similar movements.
However, in SVG, there is only one branch to predict the future frames, which
only encodes the general image information rather than motion cues. Therefore,
samples do not form clusters according to semantic classes as in our case for
dynamic latent variables shown on the left in Fig. 14.

Figure 15: Latent Variables of SVG. This figure shows the T-SNE visualization
of latent variables of SVG [5] on 300 test videos from KTH dataset. The latent
variables form a normal distribution when visualized with T-SNE because SVG
method uses a standard normal distribution as a fixed prior in KTH dataset.
The model learns the general image information instead of motions groupings.

###  E.3 Diversity of Generated Samples

As proposed in SAVP [20], as a measure of diversity, we visualize the average
over 100 generated samples. According to this measure, if a model is able to
generate diverse results, generated samples should differ where there is
motion, e.g. a moving object appearing in different positions and moving in
different directions, leading to blurring out of the moving object. Therefore,
we expect to see the background without moving objects in the average of the
generated samples. The average samples confirm this for our model as shown for
MNIST Fig. 16, KTH Fig. 17, and BAIR Fig. 18.

There is a special case on KTH which further supports our diversity claim as
shown in Fig. 7 of the main paper. When subject appears after conditioning
frames, our model can handle stochasticity of this challenging case and can
generate diverse sequences. We show the best prediction and three random
predictions in Fig. 19. Generated samples differ in terms of pose and speed of
the subject as well as the time step that the subject appears.

Figure 16: Diversity on MNIST. After a digit hits the wall, it can move in any
direction. Our model successfully models the stochasticity of this case and
generates diverse results, resulting in blurry average images. Figure 17:
Diversity on KTH. Since the running person appears after the conditioning
frames, the model should generate different results for each sample. The
average of the generated samples does not contain any human because our model
can generate diverse results, e.g. person in various poses appearing at
different time steps with different speed. Figure 18: Diversity on BAIR. The
robot hand can move in any direction at each time step, therefore the
generated samples should differ from each other in terms of the position of
the robot hand. The moving robot hand becomes invisible in the average images
after the first few frames, which is an indication of the diversity of the
generated samples. Figure 19: Subject Appearing after Conditioning Frames on
KTH. We show the best sample and three random samples for the case where the
subject appears after conditioning frames. Our model generates different
results at each random sample by learning dataset dynamics.

###  E.4 Additional Qualitative Results

For each dataset, we show random examples with details of the best sample as
well as three random samples generated. The detailed visualizations show
appearance and motion prediction separately as well as the mask prediction and
optical flow with false coloring. Random sample visualizations show the best
sample and three random samples. We also provide full sequences of the samples
in Fig. 1 in Fig. 32 and 33.

Figure 20: No Overlapping Digits. This figure show a regular case with two
non-overlapping digits. Note that predicted flow is from the target frame to
the current frame since we apply inverse warping. The correctness of the
optical flow estimation can be verified by inspecting Fig. 13. Figure 21:
Stationary Digit. This figure shows a case where a digit, i.e. 0, is not
moving. As can be seen from the last row, optical flow is correctly estimated
as zero for that digit. Figure 22: Overlapping Digits. This figure shows a
challenging case where two digits cross each other and continue moving. The
digits start overlapping at around $t=13$. Our model can successfully handle
this challenging case by preserving the appearance of digits. Figure 23:
Overlapping Digits. This figure shows a challenging case where two digits
cross each other and continue moving. The digits start overlapping at around
$t=6$. Our model can successfully handle this challenging case by preserving
the appearance of digits. Figure 24: Random Samples. We show the best sample
and three random samples generated. All of the predictions are sharp-looking
and different than each other, which proves that our model can generate
diverse results. Figure 25: Random Samples. We show the best sample and three
random samples generated. The first and the third samples cannot preserve the
shape of the digits, however, the best sample and the second sample are still
sharp-looking. Figure 26: Person Waving Hands. This figure shows the
challenging case of hand waving with fast motion in a small region. Note that
predicted flow is from the target frame to the current frame since we apply
inverse warping. The correctness of the optical flow estimation can be
verified by checking Fig. 13. Our model focuses on the motion prediction for
moving hands but it recovers the occluded motion boundaries from the
appearance prediction. Figure 27: Person Running. Our model correctly
estimates the optical flow showing the motion of a person running and it can
recover the occluded pixels around the legs from the appearance prediction.
Figure 28: Person Leaving the Frame. Our model correctly estimates optical
flow when the person leaves the scene in the middle of the sequence by
predicting nearly zero flow towards the end. Figure 29: Random Samples. We
show the best sample and three random samples generated. Random samples look
very similar due to the regular motion in the conditioning frames. Our model
can capture the motion from the conditioning frames by generating consistent
samples, only with minor differences in speed. Figure 30: Results on BAIR.
Optical flow consistently changes from one frame to the next, showing that our
model can learn the dataset dynamics. Note that predicted flow is from the
target frame to the current frame since we apply inverse warping. Figure 31:
Random Samples. We show the best sample and three random samples generated.
Random samples look very similar in the beginning due to regular motion in the
conditioning frames. Towards the end of the sequence, samples start looking
different which shows that our model can generate diverse results. Figure 32:
Full sequence of Cityscapes in Fig. 1 We show the full sequence comparisons
with baseline method, SVG [5], and state-of-the-art method, SRVP [9]. Our
model can model the ego-motion while both SRVP and SVG suffers from it
significantly. Figure 33: Full sequence of KITTI in Fig. 1 We show the full
sequence comparisons with baseline method, SVG [5], and state-of-the-art
method, SRVP [9]. Our model can model both the ego-motion and independently
moving objects while both SRVP and SVG cannot reconstruct the future frames
successfully.