# Taylor Swift: Taylor Driven Temporal Modeling for Swift Future Frame
Prediction

Mohammad Saber Pourheydari1,∗ Mohsen Fayyaz1,∗ Emad Bahrami1,∗ Mehdi Noroozi2
Juergen Gall1
1University of Bonn  2Bosch Center for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

While recurrent neural networks (RNNs) demonstrate outstanding capabilities in
future video frame prediction, they model dynamics in a discrete time space
and sequentially go through all frames until the desired future temporal step
is reached. RNNs are therefore prone to accumulate the error as the number of
future frames increases. In contrast, partial differential equations (PDEs)
model physical phenomena like dynamics in continuous time space, however,
current PDE-based approaches discretize the PDEs using e.g., the forward Euler
method. In this work, we therefore propose to approximate the motion in a
video by a continuous function using the Taylor series. To this end, we
introduce TayloSwiftNet, a novel convolutional neural network that learns to
estimate the higher order terms of the Taylor series for a given input video.
TayloSwiftNet can swiftly predict any desired future frame in just one forward
pass and change the temporal resolution on-the-fly. The experimental results
on various datasets demonstrate the superiority of our model.

††⋆ Equal contribution

##  1 Introduction

The ability to predict future frames of a video is essential for many
applications such as weather forecasting [56], autonomous driving [25],
robotics [14], or action recognition [29]. When only the raw video is given,
the task is very challenging since it requires to learn the complex motion of
the objects present in the video. To address this task, several approaches
have been proposed over the last years. In particular, recurrent neural
networks (RNNs) have been popular [56, 53, 51, 52].

While these approaches learn the motions implicitly, recently a new line of
work appeared that leverages partial differential equations (PDEs) and deep
learning for forecasting video frames [31, 30, 17, 42, 40]. PDEs are very
appealing for this task since they are an appropriate tool to model physical
phenomenas like dynamics. These methods model motion not in the discrete space
as recurrent neural networks but in the continuous time space. The latter
provides a more precise model of the motion that is independent of the
sampling rate, which means that a future frame can be directly computed for
any point in time. This is in contrast to recurrent neural networks that are
limited to a fixed sampling rate and sequentially go through all frames until
the desired point in the future is reached, which results in an increased
inference time and error accumulation.

Current PDE-based approaches, however, numerically solve the PDE by the
forward Euler method, which involves a discretization of the continuous PDE as
illustrated in Fig. 2. Since the approximation error is proportional to the
square of the future temporal step size, networks like PDE-Net [31, 30] stack
multiple blocks to minimize the approximation error of the discretization.
While this approach reduces the discretization error for learning a PDE from
data, an error remains and the step size is hard-coded in the network and
cannot be changed after training.

In this work, we therefore propose a novel approach that learns a continuous
representation of the temporal model without the need of any discretization.
Instead of learning a PDE that needs to be numerically solved for training and
inference, we directly infer a continuous function that describes the future.
In contrast to RNNs that forecast the future frame-by-frame or PDE-based
approaches that evaluate the PDEs at discrete step sizes, we infer a
continuous function over time from the observations. As illustrated in Fig. 2,
this avoids discretization artifacts and provides an analytical function that
can be swiftly evaluated for any real number as illustrated in Fig. 1. This
allows, for instance, to generate frames at different future points in
parallel. We can also forecast future frames at a different sampling rate than
the observed frames and the frame-rate of a test video does not need to match
the frame-rate of the training videos. All these properties are very useful
for practical applications and demonstrate the advantages of a continuous
representation.

It is, however, very challenging to infer such a continuous function from a
discrete set of high-dimensional observations as it is the case for video
frame forecasting. We therefore propose to approximate the unknown function by
the Taylor series about the last observation $t$ up to a finite order. As
illustrated in Fig. 1, we first map the observed frames into a learned
embedding space $\mathcal{H}_{t}$ and then estimate each term of the Taylor
expansion $\mathcal{F}_{\mathcal{H}_{t}}$. An important aspect of the proposed
TaylorSwiftNet is that it learns to generate a full Taylor expansion from a
discrete set of observations, i.e., after training, the network is capable of
inferring a Taylor expansion of an unknown function only from a set of
observations until point $t$. We validate the capabilities of the network
using simulated data of known functions where the ground-truth Taylor
expansion is known as well for high-dimensional problems like video frame
forecasting as shown in Fig. 1. Since the network infers a full function over
time $\mathcal{F}_{\mathcal{H}_{t}}:\mathbb{R}\mapsto\mathcal{H}$, we can
evaluate the function for any real value $t{+}\tau\in\mathbb{R}$ without
requiring any additional passes through the network. For instance, we can
swiftly generate frames for $\tau=0.73,1.39,3.89$ as in Fig. 1. Since the
predictions are in the embedding space, the predictions are mapped back to the
image domain by the decoder. The entire TaylorSwiftNet consisting of the
encoder, the estimation of the Taylor expansion, and the decoder is trained
end-to-end.

We compare the proposed approach with the state-of-the-art on four datasets
from different applications domains ranging from video frame forecasting to
forecasting sea surface temperature or traffic flows. On all datasets,
TaylorSwiftNet outperforms RNN-based as well as PDE-based approaches. We
furthermore demonstrate the capabilities of the approach, namely estimating
Taylor expansions of analytical functions and the flexibility to forecast
frames with a different frame-rate than the observed data or even the entire
training data.

Figure 1: Given a sequence of observed frames until time $t$, the encoder maps
them into a latent space. Our network infers from the observations
$\mathcal{H}_{t}$ in the latent space a continuous function
$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ in one pass. The inferred function can
be evaluated for any real number $\tau$ in order to forecast future frames at
$t+\tau$. The decoder maps the forecast frames back to the image domain.

##  2 Related Work

Several approaches have been proposed for video forecasting using unlabeled
videos. In particular, deep neural networks have demonstrated prominent
capabilities for this task. [37, 34, 28] use optical flow to model changes of
temporal dynamics and to better predict the future. Some works [57, 22, 14]
focus on modeling the geometric transformations between frames to predict the
future frames. [13] propose to solve the video prediction task by estimating
and using the transformations of the signal in the frequency domain. [6, 49]
focus on improving the sharpness of the predicted frames by using custom loss
functions. To better handle the future uncertainty and generate sharp
predictions, [35, 50, 25, 10] have used generative adversarial networks or
variational autoencoders. Methods based on 2D or 3D CNNs [3, 35, 50] have also
been proposed. In particular recurrent neural networks (RNNs) have been
popular [46, 53, 51, 14, 32, 36, 54, 56, 52] in recent years.

While these approaches learn the motions implicitly, recently a new line of
work appeared that leverages partial differential equations (PDEs) and neural
networks for forecasting video frames [11, 31, 30, 17, 42, 40]. Some recent
PDE-based works demonstrate substantial improvements compared to recurrent
neural networks for video frame forecasting. [17, 41, 1, 58] define the
dynamics using learned ordinary differential equations following [4]. [44, 27,
15] employ differential equations for stochastic data. Some methods shape the
prediction function or the cost function of their methods using prior physical
knowledge [2, 7]. For instance, [9] use general advection-diffusion principles
as a guideline for designing a network. [2, 43, 45] discover the PDEs by
sparse regression of potential differential terms. [16, 5, 47] introduce non-
regression losses inspired by Hamiltonian mechanics [18]. [38, 12] have
designed specific architectures for predicting and identifying dynamical
systems inspired by numerical schemes for solving PDEs and residual neural
networks [4, 33, 26, 60]. [11] propose the separation of variables as a
general paradigm based on a resolution method for partial differential
equations for video prediction and disentanglement. PDE-Net [31, 30]
discretizes a broad class of PDEs by approximating partial derivatives with
convolutions.

Although partial differential equations model the motion in the continuous
time space, the PDE-based approaches discretize the PDEs using for instance
the forward Euler method. In this work, we propose a different continuous
representation that does not need any discretization.

##  3 PDE based Forecasting

The problem of forecasting requires modeling of the future temporal dynamics
based on the past $k$ observations, i.e.,

$p(x_{t+1}|\mathcal{X}_{t})\quad\text{for}\quad\mathcal{X}_{t}=\\{x_{t-k},\dots,x_{t}\\},$
(1)

where $x_{t}$ denotes the observation at time $t$ and $x_{t+1}$ is the future
frame. For forecasting with a different temporal step, i.e.,
$\tau>1,\,\tau\in\mathbb{N}$, this results in

$\displaystyle p(x_{t+\tau}|\mathcal{X}_{t})$ (2)
$\displaystyle=\int_{x_{t+\tau-1}}\dots\int_{x_{t+1}}p(x_{t+\tau}|x_{t+\tau-1},\mathcal{X}_{t})\dots$
$\displaystyle\quad\quad\quad
p(x_{t+2}|x_{t+1},\mathcal{X}_{t})p(x_{t+1}|\mathcal{X}_{t})dx_{t+\tau-1}\dots
dx_{t+1}.$

When the motion is complex and $x_{t+\tau}$ is high-dimensional, as it is the
case for video frame forecasting, the computation of the integrals is
infeasible. Common auto-regressive approaches therefore approximate the
solution frame-by-frame by taking the argmax of (1) and adding the new
estimate $x_{t+1}$ to the observations. This, however, has the disadvantage
that one needs to iterate over all frames until $x_{t+\tau}$ is reached, the
approximation error increases over time, and $\tau$ is constrained by the
frame-rate of the training data.

Figure 2: Comparing forward Euler and Taylor approximations of a $\sin(t)$
function. The observation until $t=4.75$ is black. The gray curve is the
ground-truth, orange is the Taylor approximation, and blue is the Euler
approximation.

To overcome these issues, it has been recently proposed to model the temporal
dynamics in a different way, namely by a partial differential equation (PDE)
in a learned latent space $\mathcal{H}$ [31, 30, 17, 42, 40]. The continuous
PDE is numerically solved by the forward Euler method, i.e., for a given
starting point $t$ and the derivative at time $t$, the next point is estimated
by

$\mathcal{F}_{\mathcal{H}}(t+\delta
t)\simeq\mathcal{F}_{\mathcal{H}}(t)+\delta
t\frac{\partial\mathcal{F}_{\mathcal{H}}}{\partial t}(t)$ (3)

where $\delta t$ is the step size of the approximation. One of the
disadvantages of this approach is that the error is proportional to the square
of the step size. To address this problem and make the prediction more
accurate, PDE-Net [31] proposes to stack multiple $\delta t$-blocks, i.e.,
$\tau$ is implicitly divided into fixed step sizes $\delta t$. This approach,
however, keeps the disadvantages of a discrete representation with fixed step
sizes. For instance, the step size is determined during training and cannot be
changed for inference. Furthermore, the discretization introduces an
additional approximation error as shown in Fig. 2.

In order to reduce the approximation error and to provide a continuous
representation with respect to $\tau$, we propose a different continuous
approximation. Instead of discretizing the partial differential equation, we
approximate the future dynamics by the Taylor series. In this way, we can
better model the temporal dynamics without pre-defining any step sizes as
illustrated in Fig. 2. Notably, in our model, the temporal step, $\tau$, can
be any positive real number.

##  4 Temporal Dynamics Modeling Through Taylor Series

Figure 3: Overview of the TaylorSwiftNet. The network gets a sequence of
frames $\mathcal{X}_{t}$ as input. The encoder maps all frames into a latent
space. The frames in the latent space are denoted by $\mathcal{H}_{t}$ and
$h_{t}$ denotes the embedding for frame $t$. Using a Taylor series of order
$n$, the function $\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ is approximated to
model the temporal dynamics. To estimate the higher order terms of the Taylor
series, $\mathcal{H}_{t}$ is fed to the DCB blocks. The DCB blocks estimate
sequentially the $\delta_{n}$s, which are the estimated derivatives of
$\mathcal{F}_{\mathcal{H}_{t}}$ at the point $t$. Having the derivatives
$\delta_{n}$s, the Taylor series approximates
$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ at the future time step $\tau$, which
yields $h_{t+\tau}$. Finally, $h_{t+\tau}$ is fed to the decoder and the
future frame $\hat{x}_{t+\tau}$ is predicted.

Our novel approach for continuous forecasting can directly predict
$\hat{x}_{t+\tau}$ for any value $\tau>0$ from the observations
$\mathcal{X}_{t}$. The model consists of three parts as illustrated in Fig. 3:

$\hat{x}_{t+\tau}=\mathcal{D}(h_{t+\tau})\>;\>h_{t+\tau}=\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)\>;\>\mathcal{H}_{t}=\mathcal{E}(\mathcal{X}_{t}).$
(4)

$\mathcal{E}(.)$ first maps the observed frames $\mathcal{X}_{t}$ into the
learned embedding space. Given the frames in the embedding space
$\mathcal{H}_{t}$, our network infers an observation specific function
$\mathcal{F}_{\mathcal{H}_{t}}(.)$ that models the future dynamics in the
embedding space. It is important to note that our approach infers a full
function with respect to $\tau$ and not a single point estimate as it is done
by standard networks. The function $\mathcal{F}_{\mathcal{H}_{t}}(.)$ can
therefore be swiftly evaluated for any values without requiring additional
forward passes through the network. $\mathcal{D}(.)$ finally decodes the
forecast embedding $h_{t+\tau}$ and predicts the future frame
$\hat{x}_{t+\tau}$. While $\mathcal{E}(.)$ and $\mathcal{D}(.)$ will be
discussed in Section 5, we first discuss $\mathcal{F}_{\mathcal{H}_{t}}$.

Our goal is to learn the unknown continuous function
$\mathcal{F}_{\mathcal{H}_{t}}$ that represents the future dynamics in the
embedding space. In case of $\tau=0$,
$\mathcal{F}_{\mathcal{H}_{t}}(t)=h_{t}$, which is equal to the last vector of
$\mathcal{H}_{t}$. For $\tau>0$, however, this function is very complex. We
therefore propose to approximate the function using the Taylor series:

$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)\simeq\sum_{n=0}^{\gamma}\frac{\mathcal{F}_{\mathcal{H}_{t}}^{(n)}(t)}{n!}\tau^{n}\>;\>\mathcal{F}_{\mathcal{H}_{t}}^{(n)}=\frac{\partial^{n}\mathcal{F}_{\mathcal{H}_{t}}}{\partial\tau^{n}}.$
(5)

If $\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ is an analytic function, the Taylor
series with $\gamma=\infty$ is equal to
$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$. However, we can approximate it by
using only a finite number of terms.

Although the approximation (5) requires to compute
$\mathcal{F}_{\mathcal{H}_{t}}^{(n)}$ for higher order terms at $t$ in order
to get a good approximation, this needs to be done only once and the function
can be evaluated for any $\tau>0$. Since computing the derivatives of an
unknown function in a very high dimensional space is impractical, we propose
to learn a network that infers them from $\mathcal{H}_{t}$:

$\mathcal{F}_{\mathcal{H}_{t}}^{(n)}(t)\simeq
f_{n}\left(\Delta_{n}\left(\mathcal{H}^{(n-1)}_{t}\right)\right);\;\mathcal{H}^{(n-1)}_{t}=\Delta_{n-1}\left(\mathcal{H}^{(n-2)}_{t}\right)$
(6)

where $f_{n}$ and $\Delta_{n}$ are trainable blocks that will be described in
Section 5 and $\mathcal{H}^{(0)}_{t}=\mathcal{H}_{t}$. Considering (5) and
(6), we can reformulate (4) as:

$\hat{x}_{t+\tau}=\mathcal{D}\left(h_{t}+\sum_{n=1}^{\gamma}\frac{f_{n}(\Delta_{n}(\Delta_{n-1}(\ldots\Delta_{1}(\mathcal{E}(\mathcal{X}_{t})))))}{n!}\tau^{n}\right).$
(7)

Having the future frame $x_{t+\tau}$ as ground-truth during training, we can
compute the loss between the predicted frame $\hat{x}_{t+\tau}$ and the
ground-truth frame $x_{t+\tau}$ and update the parameters of $\mathcal{D}$,
$f_{n}$, $\Delta_{n}$, and $\mathcal{E}$.

##  5 Proposed Architecture

We now describe the network architecture that learns the model (7). As
illustrated in Fig. 3, the network first encodes the input video frames, i.e.,
$\mathcal{H}_{t}=\mathcal{E}(\mathcal{X}_{t})$. The encoder is described in
Section 5.1. The temporal model, which learns to approximate the function
$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$, is described in Section 5.2. After
training, the temporal model can generate images for any positive continuous
value $\tau$ as illustrated in Fig. 1. We only need to apply the decoder
$\mathcal{D}(\mathcal{F}_{\mathcal{H}_{t}}(t+\tau))$, which is described in
Section 5.3.

###  5.1 Encoder

The encoder $\mathcal{E}(\mathcal{X}_{t})$ maps the input video frames
$\mathcal{X}_{t}\in\mathbb{R}^{C\times T\times H\times W}$ to
$\mathcal{H}_{t}\in\mathbb{R}^{C^{\prime}\times T\times H^{\prime}\times
W^{\prime}}$. $C,T,H,W$ are the video channels, number of frames, height and
width of the frames. $C^{\prime},H^{\prime},W^{\prime}$ are the channels of
the featuremaps and their height and width. In principle, the encoder can be
any 3D convolutional neural network. In this work, we have employed a modified
version of 3DResNet [19, 20]. For the implementation details of our encoder,
we refer to the supplementary materials.

###  5.2 Temporal Model

The temporal model $\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ models the future
temporal dynamics and forecasts the frame at the future temporal step
$t{+}\tau\in\mathbb{R}$ in the embedded space
$h_{t+\tau}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times W^{\prime}}$. As
illustrated in Fig. 3, we estimate $\mathcal{F}_{\mathcal{H}_{t}}^{(n)}$ (6)
recursively. We use for each $\Delta_{n}(.)$ a convolutional block called
delta convolutional block (DCB). The first 2 convolutional layers of DCB use
kernels with size $3\times 3\times 3$ and stride $1\times 1\times 1$. The
input and output size remains the same. The output featuremap is then fed to:
(a) the final convolutional layer $f_{n}$ to output the estimated derivative
$\delta_{n}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times W^{\prime}}$; (b)
the next DCB block to estimate the next order derivative $\delta_{n+1}$.
$f_{n}$ uses kernels with size $T\times 3\times 3$. The estimated derivatives
are then used in (6) to model the temporal dynamics and forecast the embedding
$h_{t+\tau}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times W^{\prime}}$.
While in our experiments the different DCB blocks do not share their weights,
we also evaluate a recurrent version with shared weights in the supplementary
material.

###  5.3 Decoder

The decoder is a convolutional neural network which consists of 6
convolutional layers with kernel size $1\times 3\times 3$. The decoder
$\mathcal{D}(h_{t+\tau})$ decodes the embedding and predicts the future frame
$\hat{x}_{t+\tau}\in\mathbb{R}^{C\times H\times W}$. For more details
regarding the implementation details of the decoder, we refer to the
supplementary materials.

##  6 Experiments

###  6.1 Datasets and Implementation Details

Following the state-of-the-art [17], we also evaluate our method on four
datasets from very different domains, namely Moving MNIST [46], Human 3.6M
[21], Traffic BJ [59], and Sea Surface Temperature [8].

Moving MNIST is a standard dataset for sequence prediction which consists of
two random digits moving inside a $64\times 64$ grid. Data for training were
generated on the fly and a test set of 10,000 sequences were used for
evaluation. We predict 10 unseen future frames given 10 seen input frames.

Human 3.6M contains human actions with their corresponding 3D poses for 17
action scenarios. Following the setting of [17, 54], we select subjects S1,
S5, S6, S7, S8 for training and subjects S9 and S11 for testing using the
walking action. Human 3.6M includes originally RGB images of size $1000\times
1000\times 3$ which we resize to $128\times 128\times 3$ for our experiments.
We predict 4 unseen frames given 4 input seen frames.

Traffic BJ contains the hourly taxi flows of Beijing in a $32\times 32$ grid.
Each frame has two channels which corresponds to the traffic flow entering and
leaving a district. We use 4 input seen frames to predict 4 unseen frames.

Sea Surface Temperature consists of meteorological data of Atlantic ocean
generated by NEMO (Nucleus for European Modeling of the Ocean), which is a
state-of-the-art simulation engine for modeling ocean dynamics. Following the
protocol of [8], we use the Sea Surface Temperature (SST) data of $64\times
64$ sized sub-regions extracted from the original $481\times 781$ sized data.
We predict 4 future frames given 4 unseen input frames.

Evaluation Metrics. Following the state-of-the-art methods [17, 31, 51], we
use the following evaluation metrics: Mean Squared Error (MSE), Mean Absolute
Error (MAE), and the Structural Similarity (SSIM) [55]. We average the metrics
over all frames of the predicted output sequence. While lower MSE and MAE
indicate better performance, a higher SSIM is better.

Training. We use a 3DResNet [19, 20] for the encoder and a version of DCGAN
[39] for the decoder as in [17]. For the Moving MNIST dataset, we use a Taylor
model of order 4 alongside with a 3DResNet18 as the encoder. For the other
datasets, we use a Taylor model of order 2 with a 3DResNet18 as the encoder.
We use Adam [24] with learning rate of 0.0001 to optimize the model through 4K
epochs. For the the SST dataset, we train our model in 1K epochs. To control
the learning rate, we use a scheduler to reduce the learning rate by factor of
0.5 in case of plateau over the SSIM metric on the training set. Following the
previous state-of-the-art methods [17], we use MSE as the loss function. More
implementation details are provided in the supplementary materials.

| Moving MNIST | Traffic BJ | Sea Surface Temperature | Human 3.6
---|---|---|---|---
Method | MSE | MAE | SSIM | MSE $\times 100$ | MAE | SSIM | MSE $\times 10$ | MAE | SSIM | MSE $/10$ | MAE $/100$ | SSIM
Advection-diffusion [9] | - | - | - | - | - | - | 34.1 | 54.1 | 0.966 | - | - | -
DDPAE [23] | 38.9 | 90.7 | 0.922 | - | - | - | - | - | - | - | - | -
ConvLSTM [56] | 103.3 | 182.9 | 0.707 | 48.5 | 17.7 | 0.978 | 45.6 | 63.1 | 0.949 | 50.4 | 18.9 | 0.776
PredRNN [53] | 56.8 | 126.1 | 0.867 | 46.4 | 17.1 | 0.971 | 41.9 | 62.1 | 0.955 | 48.4 | 18.9 | 0.781
Causal LSTM [51] | 46.5 | 106.8 | 0.898 | 44.8 | 16.9 | 0.977 | 39.1 | 62.3 | 0.929 | 45.8 | 17.2 | 0.851
MIM [54] | 44.2 | 101.1 | 0.910 | 42.9 | 16.6 | 0.971 | 42.1 | 60.8 | 0.955 | 42.9 | 17.8 | 0.790
E3D-LSTM [52] | 41.3 | 86.4 | 0.920 | 43.2 | 16.9 | 0.979 | 34.7 | 59.1 | 0.969 | 46.4 | 16.6 | 0.869
PhyDNet [17] | 24.4 | 70.3 | 0.947 | 41.9 | 16.2 | 0.982 | 31.9 | 53.3 | 0.972 | 36.9 | 16.2 | 0.901
TaylorSwiftNet (ours) | 21.2 | 60.8 | 0.952 | 35.3 | 13.7 | 0.992 | 29.8 | 52.2 | 0.978 | 23.1 | 15.8 | 0.910
|  |  |  |  |  |  |  |  |  |  |  |

Table 1: Comparison to the state-of-the-art on four datasets. The results are
the mean over all predicted frames.

###  6.2 Comparison to State-of-the-Art

We compare our TaylorSwiftNet with various state-of-the-art methods. As it can
be seen in Table 1, TaylorSwiftNet significantly outperforms the state-of-the-
art methods on all datasets and for all metrics. Our direct future temporal
forecasting method significantly outperforms state-of-the-art architectures
based on recurrent neural networks, such as PhyDNet [17], ConvLSTM [56],
PredRNN [53], Causal LSTM [51], or Memory in Memory (MIM) [54]. We also
outperform methods that focus only on datasets from one domain: (i) DDPAE [23]
is specialized and state-of-the-art on Moving MNIST; (ii) the advection-
diffusion flow model [9] is the state-of-the-art for the SST dataset. For the
Human 3.6M dataset, some approaches like [48] use additional supervision like
human poses. Although we do not use such additional supervision, we also
outperform [48] as shown in the supplementary materials. The results
demonstrate that the proposed approach learns a very good temporal model from
various videos of different application domains.

Figure 4: Qualitative results. For each dataset, the first row shows the input
of the model, the second row shows the ground-truth, and the third row shows
our prediction. For Traffic BJ, we also show the difference between the
prediction and the ground-truth. In this case, we adjusted the scale to
visualize the differences better.

Fig. 4 shows some qualitative results for all datasets. The qualitative
results demonstrate the high quality of the forecast results that are
generated by the proposed TaylorSwiftNet. For Moving MNIST, TaylorSwiftNet
predicts accurately the digits even when they overlap. For Traffic BJ, the
hourly taxi flows are correctly predicted. The last column shows the
difference between the predicted and ground-truth (target) images. Note that
the scale for the images in last column has been adjusted in order to see the
differences. The sea surface temperature, which depends on phenomena that can
be described by PDEs, is accurately predicted as well. And finally,
TaylorSwiftNet also precisely anticipates the future position and pose of the
person in the video from the Human 3.6M dataset.

###  6.3 Additional Comparison

As mentioned in Sec. 6.2, [48] use human poses as additional supervision.
Although we only use unsupervised training without any additional supervision
in this work, we also outperform [48] on the Human 3.6M dataset. In the
Appendix of [48], the authors evaluate their model with Peak Signal over Noise
Ratio (PSNR) curves with respect to the forecasting horizon for all deciles of
motion in the videos. For time step 4, the PSNR of ‘Ours’ in [48] is below 21
and around 22 for the 1st decile (least human motion). While, our
TaylorSwiftNet obtains a PSNR of 25.75. This shows the superiority of our
method without the need for any additional human pose annotations.

###  6.4 Ablation Experiments

We analyze the impact of $\gamma$, the performance for long-term forecasting,
the ability for continuous forecasting. Furthermore, we compare our approach
with additional baselines and analyze the accuracy of the learned terms of the
Taylor expansion. If not otherwise specified, we use 3DResNet18 as the
encoder.

Impact of $\gamma$. We approximate $\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$ (5)
by $\gamma$ terms where $\gamma$ defines the order of the Taylor series. We
therefore evaluate the effect of using different orders of the Taylor series
on Moving MNIST. As it can be seen in Fig. 5, all three models have
approximately the same prediction performance for $\tau{=}1$ (time step 11).
However, as we increase $\tau$ the difference between the prediction
performance of the 3 models increases. This is expected since having higher
orders of the series will result in a better approximation of
$\mathcal{F}_{\mathcal{H}_{t}}(t+\tau)$.

Figure 5: Comparing different orders of our temporal model using 10 frames as
observation. Figure 6: Comparing future predictions for time horizons that are
larger than the 10 frames used for training. The step size indicates after how
many frames the Taylor approximation is performed. Step size 10 means that the
Taylor approximation is performed every 10 frames.  Figure 7: Predicting
future frames at higher temporal resolution. $\tau$ is increased by 0.3
instead of 1.

Long-term Forecasting. In this experiment, we explore the long-term future
forecasting capability of our TaylorSwiftNet. We use the same setup as in the
previous experiments for the Moving MNIST dataset, but instead of predicting
for the future temporal horizon of 10 frames, we predict 70 frames, i.e.,
evaluating far beyond the prediction range seen during training. We therefore
evaluate our model in a partially auto-regressive mode where we directly
forecast the first 10 frames and feed them back to predict the next 10 frames.
In other words, we do a Taylor approximation every 10 frames. For comparison,
we also perform the Taylor approximation every 7, 5, 2, and each frame. The
latter is a standard auto-regressive setting.

As it can be seen in Fig. 6, performing the Taylor approximation every 10
frames performs best for predicting long-term unseen future frames. In
contrast, the accuracy of the standard auto-regressive setting where the
frames are predicted frame-by-frame performs worst. The reason for such a fast
drop in accuracy is due to the error propagation through the recursive steps.
The models with the longer step size need less auto-regressive steps while the
models with shorter step size need more auto-regressive steps. This shows that
approaches that forecast frames frame-by-frame suffer from error propagation
and are not suitable to forecast longer sequences.

Continuous Prediction. Since our model forecasts frames using a continuous
representation, we do not need to stick to the framerate of the observation.
In Fig. 7, we show qualitative results on Moving MNIST for the future temporal
steps $t{+}\tau\in\\{11,11.3,11.6,...,20.6\\}$, i.e., we increase the
framerate by $1/0.3$. Note that we do not re-train our model for this
experiment. As it can be seen, our TaylorSwiftNet smoothly predicts
intermediate frames. The sharp digits and their accurate location clearly
demonstrate the continuous temporal modeling capability of our model.

For better evaluating the continuous prediction capability of our model, we
conducted also a quantitative evaluation on all four datasets. While keeping
the frames of the test videos as before, we sample only every second frame
during training. This means that the temporal gap between two frames during
training is twice as high as for the test videos. Existing approaches cannot
handle such a change without re-training the models. The results in Table 2
show that changing the sampling rate during training has nearly no impact on
the mean SSIM for all datasets. This demonstrates that our approach learns an
accurate continuous temporal model.

Sub-sampling | Moving MNIST | Traffic BJ | SST | Human 3.6M
---|---|---|---|---
1 | 0.965 | 0.992 | 0.978 | 0.910
2 | 0.952 | 0.991 | 0.974 | 0.901
|  |  |  |

Table 2: SSIM for different frame-rates of the training videos. In case of 1,
we use the same setting as in Table 1, i.e., the training and test videos have
the same frame-rate. In case of 2, we sample during training only every second
frame, i.e., the frame-rates of the training videos differs from the test
videos.

Temporal Modeling. As discussed in Section 4, our approach proposes a complete
novel way for forecasting. Instead of making a point estimate
$h_{t+\tau}=g(\mathcal{X}_{t},\tau)$ for a specific $\tau$, where $g$ is a
network and $h_{t+\tau}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times
W^{\prime}}$, our network infers a continuous function
$\mathcal{F}_{\mathcal{H}_{t}}:\mathbb{R}\mapsto\mathbb{R}^{C^{\prime}\times
H^{\prime}\times W^{\prime}}$ from $\mathcal{X}_{t}$. In order to compare with
point estimates, we evaluate two variants of $g$. For both settings, we use a
fully connected layer to map $\tau$ to a vector of size $C^{\prime}$, which is
denoted by $v_{\tau}$. In the first setting (Expand), we copy $v_{\tau}$ to
get a $C^{\prime}\times H^{\prime}\times W^{\prime}$ tensor that is
concatenated with the temporally-squeezed version of $\mathcal{H}_{t}$. The
resulting $2C^{\prime}\times H^{\prime}\times W^{\prime}$ tensor is processed
by a convolutional layer and a downsampling operation to get
$h_{t+\tau}\in\mathcal{R}^{C^{\prime}\times H^{\prime}\times W^{\prime}}$. In
the second setting (Flatten), we flatten the down-sampled version of
$\mathcal{H}_{t}$ and concatenate it with the vector $v_{\tau}$. We use three
convolutional layers to squeeze the temporal dimension of $\mathcal{H}_{t}$
and down-sample the spatial dimensions 8 times. Afterwards, we use three
transposed convolutional layers to up-sample the concatenated vector to obtain
$h_{t+\tau}$. The results in Table 3 show that learning a function for all
values of $\tau$ performs better than adding $\tau$ as input to the network.

Method | MSE | MAE | SSIM
---|---|---|---
Point Estimate (Expand) | 45.1 | 69.4 | 0.642
Point Estimate (Flatten) | 43.6 | 63.6 | 0.887
TaylorSwiftNet | 21.2 | 60.8 | 0.952
|  |  |

Table 3: Comparison of the proposed temporal model to two variants of point
estimates.

Estimation of Taylor Series. For video data, the function
$\mathcal{F}_{\mathcal{H}_{t}}$ and thus the ground-truth terms of the Taylor
series are unknown. In order to analyze how accurate our network can learn the
terms of the Taylor series, we use the multivariate function $\sin(x+y)$. For
such a function, we can analytically derive all derivatives and compute the
ground-truth terms for the Taylor series. In Fig. 8, we show the prediction,
ground-truth, and absolute difference for the fourth term of the Taylor
expansion. We furthermore provide a comparison to analytically derived
derivatives for three functions in Table 4. The results demonstrate that our
network learns the Taylor series.

Figure 8: Prediction, ground-truth, and absolute difference for the fourth
term of the Taylor expansion for $sin(x+y)$.

| $1^{st}$ | $2^{nd}$ | $3^{rd}$ | $4^{th}$
---|---|---|---|---
$d^{i}\sin/dt^{i}$ | $0.05077$ | $-0.99871$ | $-0.05077$ | $0.99871$
ours | $0.05083$ | $-0.99993$ | $-0.05017$ | $0.99012$
$d^{i}\cos/dt^{i}$ | $-0.9987$ | $-0.0507$ | $0.9987$ | $0.0507$
ours | $-0.9991$ | $-0.0506$ | $0.9900$ | $0.0504$
$d^{i}\exp/dt^{i}$ | $4.5722$ | $4.5722$ | $4.5722$ | $4.5722$
ours | $4.5723$ | $4.5726$ | $4.5721$ | $4.5730$
|  |  |  |

Table 4: Comparing the $1^{st}$, $2^{nd}$, $3^{rd}$, and $4^{th}$ order
derivatives of three functions with the estimated derivatives.

##  7 Conclusion

In this work, we presented an approach that forecasts future frames in one
pass. This has been achieved by modeling the dynamics in the continuous time
space without requiring any discretization. Since the motion can be very
complex in a video, we use the Taylor series as approximation and train a
network to infer the higher order terms of the Taylor series from the observed
frames. We evaluated our approach on four dataset from different domains like
forecasting human motion, hourly taxi flows, or sea surface temperature. For
all datasets, our approach outperformed the state-of-the-art.

## References

  * [1] Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari.  Learning the spatio-temporal dynamics of physical processes from partial observations.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3232–3236. IEEE, 2020.
  * [2] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  In Proceedings of the national academy of sciences, 2016.
  * [3] Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos.  Contextvp: Fully context-aware video prediction.  In ECCV, 2018.
  * [4] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  In Advances in neural information processing systems, 2018.
  * [5] Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou.  Symplectic recurrent neural networks.  In International Conference on Learning Representations, 2019.
  * [6] Marco Cuturi and Mathieu Blondel.  Soft-dtw: a differentiable loss function for time-series.  In ICML, 2017.
  * [7] Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter.  End-to-end differentiable physics for learning and control.  In Advances in Neural Information Processing Systems, pages 7178–7189, 2018.
  * [8] Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari.  Deep learning for physical processes: Incorporating prior scientific knowledge.  In International Conference on Learning Representations (ICLR), 2018\.
  * [9] Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari.  Deep learning for physical processes: Incorporating prior scientific knowledge.  Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124009, 2019.
  * [10] Emily Denton and Rob Fergus.  Stochastic video generation with a learned prior.  In ICML, pages 1174–1183, 2018.
  * [11] Jérémie Donà, Jean-Yves Franceschi, Sylvain Lamprier, and Patrick Gallinari.  Pde-driven spatiotemporal disentanglement.  arXiv preprint arXiv:2008.01352, 2020.
  * [12] Ronan Fablet, Said Ouala, and Cédric Herzet.  Bilinear residual neural network for the identification and forecasting of geophysical dynamics.  In 2018 26th European Signal Processing Conference (EUSIPCO). IEEE, 2018.
  * [13] Hafez Farazi and Sven Behnke.  Frequency domain transformer networks for video prediction.  In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2019.
  * [14] Chelsea Finn, Ian J Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In Advances in neural information processing systems, 2016.
  * [15] Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari.  Stochastic latent residual video prediction.  arXiv preprint arXiv:2002.09219, 2020.
  * [16] Samuel Greydanus, Misko Dzamba, and Jason Yosinski.  Hamiltonian neural networks.  In Advances in Neural Information Processing Systems, pages 15379–15389, 2019.
  * [17] Vincent Le Guen and Nicolas Thome.  Disentangling physical dynamics from unknown factors for unsupervised video prediction.  In CVPR, 2020.
  * [18] William Rowan Hamilton.  Vii. second essay on a general method in dynamics.  Philosophical Transactions of the Royal Society of London, (125):95–144, 1835.
  * [19] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Learning spatio-temporal features with 3d residual networks for action recognition.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [21] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014\.
  * [22] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  In Advances in neural information processing systems, 2016.
  * [23] Hsieh Jun-Ting, Liu Bingbin, Huang De-An, Fei-Fei Li, and Carlos Niebles Juan.  Learning to decompose and disentangle representations for video prediction.  CoRR, 2018.
  * [24] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [25] Yong-Hoon Kwon and Min-Gyu Park.  Predicting future frames using retrospective cycle gan.  In CVPR, 2019.
  * [26] Qianxiao Li, Ting Lin, and Zuowei Shen.  Deep learning via dynamical systems: An approximation perspective.  arXiv preprint arXiv:1912.10382, 2019.
  * [27] Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud.  Scalable gradients for stochastic differential equations.  arXiv preprint arXiv:2001.01328, 2020.
  * [28] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Flow-grounded spatial-temporal video prediction from still images.  In ECCV, 2018.
  * [29] Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P Xing.  Dual motion gan for future-flow embedded video prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1744–1752, 2017.
  * [30] Zichao Long, Yiping Lu, and Bin Dong.  PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network.  In Journal of Computational Physics, 2019.
  * [31] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong.  PDE-Net: Learning PDEs from Data.  In ICML, 2018.
  * [32] Chaochao Lu, Michael Hirsch, and Bernhard Scholkopf.  Flexible spatio-temporal networks for video prediction.  In CVPR, 2017.
  * [33] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong.  Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.  In International Conference on Machine Learning, 2018.
  * [34] Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Li Fei-Fei.  Unsupervised learning of long-term motion dynamics for videos.  In CVPR, 2017.
  * [35] Michael Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  In ICLR, 2016.
  * [36] Marc Oliu, Javier Selva, and Sergio Escalera.  Folded recurrent neural networks for future video prediction.  In ECCV, 2018.
  * [37] Viorica Patraucean, Ankur Handa, and Roberto Cipolla.  Spatio-temporal video autoencoder with differentiable memory.  In ICLR Workshop Track, 2016.
  * [38] Tong Qin, Kailiang Wu, and Dongbin Xiu.  Data driven governing equations approximation using deep neural networks.  In Journal of Computational Physics, 2019.
  * [39] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
  * [40] Maziar Raissi.  Deep hidden physics models: Deep learning of nonlinear partial differential equations.  In The Journal of Machine Learning Research, 2018.
  * [41] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud.  Latent ordinary differential equations for irregularly-sampled time series.  In Advances in neural information processing systems, pages 5320–5330, 2019.
  * [42] Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Data-driven discovery of partial differential equations.  In Science Advances, 2017.
  * [43] Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Data-driven discovery of partial differential equations.  In Science Advances, 2017.
  * [44] Tom Ryder, Andrew Golightly, A Stephen McGough, and Dennis Prangle.  Black-box variational inference for stochastic differential equations.  In International Conference on Machine Learning, pages 4423–4432, 2018.
  * [45] Hayden Schaeffer.  Learning partial differential equations via data discovery and sparse optimization.  In Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2017.
  * [46] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In ICML, 2015.
  * [47] Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins.  Hamiltonian generative networks.  In International Conference on Learning Representations, 2019.
  * [48] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee.  Learning to generate long-term future via hierarchical prediction.  In International Conference on Machine Learning (ICML), 2017.
  * [49] LE Vincent and Nicolas Thome.  Shape and time distortion loss for training deep time series forecasting models.  In Advances in neural information processing systems, 2019.
  * [50] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Generating videos with scene dynamics.  In Advances in neural information processing systems, 2016.
  * [51] Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip.  Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning.  In ICML, 2018.
  * [52] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei.  Eidetic 3d lstm: A model for video prediction and beyond.  In ICLR, 2018.
  * [53] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip.  Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms.  In Advances in neural information processing systems, 2017.
  * [54] Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu.  Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics.  In CVPR, 2019.
  * [55] Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 2004.
  * [56] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In Advances in neural information processing systems, 2015.
  * [57] Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman.  Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks.  In Advances in neural information processing systems, 2016.
  * [58] Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki.  Ode2vae: Deep generative second order odes with bayesian neural networks.  In Advances in neural information processing systems, 2019.
  * [59] Junbo Zhang, Yu Zheng, and Dekang Qi.  Deep spatio-temporal residual networks for citywide crowd flows prediction.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
  * [60] Mai Zhu, Bo Chang, and Chong Fu.  Convolutional neural networks combined with runge-kutta methods.  arXiv preprint arXiv:1802.08831, 2018.