# Learn the Force We Can:
Multi-Object Video Generation from Pixel-Level Interactions

Aram Davtyan
University of Bern
Bern, Switzerland
<EMAIL_ADDRESS>Paolo Favaro
University of Bern
Bern, Switzerland
<EMAIL_ADDRESS>
###### Abstract

We propose a novel unsupervised method to autoregressively generate videos
from a single frame and a sparse motion input. Our trained model can generate
realistic object-to-object interactions and separate the dynamics and the
extents of multiple objects despite only observing them under correlated
motion activities. Key components in our method are the randomized
conditioning scheme, the encoding of the input motion control, and the
randomized and sparse sampling to break correlations. Our model, which we call
YODA, has the ability to move objects without physically touching them. We
show both qualitatively and quantitatively that YODA accurately follows the
user control, while yielding a video quality that is on par with or better
than state of the art video generation prior work on several datasets. For
videos, visit our project website https://araachie.github.io/yoda.

Robot motion |  [width=2.4cm]3Figures/bair_robot_int/image_014 |  |  |  |  |  |
---|---|---|---|---|---|---|---
Object motion |  [width=2.4cm]3Figures/bair_rotate/000007 |  |  |  |  |  |

##  1 Introduction

According to Judea Pearl and Dana Mackenzie [48], “ _a causal learner must
master at least three distinct levels of cognitive ability:_seeing_ , _doing_
and _imagining_.”_ The achievement of these levels seems to correlate well
with the development of skills in organisms and with their chances of
survival. For instance, organisms that learned how to use a tool (an example
of _doing_), have better means to defend themselves beyond their genetic
endowments, to capture preys and to build other useful structures, such as
shelters. Among these levels, _imagining_ is considered the most powerful
capability. _Imagining_ is the ability to ask and answer counterfactual
questions such as “What would happen if X had moved differently than how it
did in the past?”

In this work, we aim at building models that can learn how to answer such
counterfactual questions through the generation of video in a completely
unsupervised fashion. Learning is allowed only through the passive observation
of videos, _i.e._ , without the ability to explicitly interact with the scene,
and without any per-sample manual annotation. We do not aim to learn causal
relationships [48, 69], but rather we aim more conservatively to build a model
that can show some degree of generalization from the training data, _i.e._ ,
that can generate data that was not observed before, that is directly related
to our question, and that is plausible, as shown in Figure
LABEL:fig:bair_teaser. We focus on counterfactual questions that can be
formulated in the form of a motion specification, _i.e._ , “What would happen
if the object containing this pixel would move at this location?” The
questions are posed by providing the current frame, _context_ frames (a subset
of the past frames), and a motion control, _i.e._ , a shift at a pixel (but
the model can also take multiple shifts at once and also at different time
instants). The answer is a generated subsequent frame that shows how the input
frame would change under the specified motion. We train our model without
specifying what objects are (_i.e._ , we do not use information about object
categories), or where they are (_i.e._ , we do not rely on bounding boxes,
landmarks/point annotations or segmentation masks), or how they interact
(_i.e._ , we do not make use of information on object relationships or action
categories or textural descriptions of the scenes). Given two subsequent
frames (the current and following one) from a video in our training data, we
obtain the motion control input by sampling the estimated optical flow at a
few (typically $5$) locations. To generate frames we use flow matching [38,
40, 1], where the model is conditioned on the current and past frames [11],
and feed the motion control input through cross-attention layers in a
transformer architecture. We call our approach YODA, as it shows the ability
of moving objects without touching them.

As shown in Figure LABEL:fig:bair_teaser, an emerging property of our proposed
approach is that the model learns the physical extent of objects in the scene
without ever requiring explicit supervision for it. For example, although the
control is applied to the handle of the brush in the second row, the motion is
applied correctly to the whole brush, and to the whole brush only. A second
learned property can be observed on the first row: When the robot arm is
driven towards other objects, it interacts with them realistically. In the
second row, we can observe a third remarkable capability that the model has
learned. The generated video shows that we can directly rotate a single
object, without using the robot arm to do so. This is a video that has never
been observed in the dataset (all objects are moved directly by the arm or
indirectly via other objects). It does demonstrate empirically that the model
has the ability to _imagine_ novel plausible outcomes when the reality is
modified in ways that were not observed before. Our contributions can be
summarized as follows

  1. 1.

We introduce a model for controllable video synthesis that is trained in a
completely unsupervised fashion, is not domain-specific, and can scale up to
large datasets;

  2. 2.

We introduce an effective way to embed motion information and to feed it to
the model, and show analysis to understand the impact of sampling and the use
of sparsity of the motion field;

  3. 3.

We demonstrate for the first time multi-object interactions in the
unsupervised setting on real data, which has not been shown in other state of
the art methods [44, 43, 10, 29, 6, 5].

##  2 Prior Work

Video generation. An increased interest in video generation has followed the
success of generative models for images [32, 15, 13, 50]. In contrast to image
generation, video generation is plagued by problems such as rendering
realistic motion, capturing diversity (_i.e_., modeling the stochasticity of
the future outcomes) and, most importantly, managing the high computational
and storage requirements. Conventional approaches to video generation are
autoregressive RNN-based models [3, 6]. RNNs are expected to generate
sequences with a consistent motion, because of the conditioning on the
previously generated frames. Other models instead obtain consistency through
the direct generation of a predefined number of frames [24, 5, 54].
Variability (stochasticity) of the generated sequences has been tackled with
GANs [9, 42, 36], variational approaches [2, 12, 3, 63, 37], Transformers [35,
49, 17, 53, 66] and diffusion-based approaches [24, 59, 26, 19, 21, 67]. The
recently proposed autoregressive method RIVER [11] deals with the stochastic
nature of the generative process through flow matching [38]. Among all above
approaches, YODA uses RIVER as a backbone, because of its efficiency and ease
of training.

Controllable video generation work mostly differs in the nature of the control
signals. Control can be defined per frame [8, 16, 33, 45, 46, 27, 44, 10, 29,
51], or as a global label [34, 60, 54]. Some of them are obtained via
supervision [8, 16, 33, 45, 46], or discovered in an unsupervised manner [44,
10, 29, 51]. For instance, CADDY [44] learns a discrete action code of the
agent that moves in the videos. Another model, GLASS [10] decouples the
actions into global and local ones, where global actions, as in YODA, are
represented with 2D shifts, while local actions are discrete action codes, as
in [44]. In [29] the authors explicitly separate the foreground agent from the
background and condition the generation on the transformations of the
segmentation mask. However, all these models are restricted to single agent
videos, while YODA successfully models multiple objects and their
interactions. [27, 18, 6, 5] are the most similar works to ours as they
specify motion control at the pixel-level. However, [27] leverages a pre-
trained object detector to obtain the ground truth control. [18] is based on
warping and therefore does not incorporate memory to model long-range
consequences of actions. II2V [6] uses a hierarchical RNN to allow modeling
higher-order details, but focuses on deterministic prediction. iPOKE [5] aims
to model stochasticity via a conditional invertible neural network, but has to
sacrifice the ability to generate long videos and to intervene into the
generation process at any timestamp. None of those works has demonstrated
controllable video generation on multi-object real scenes.

Multi-object scenes and interactions. Modeling multiple objects in videos and
especially their interactions is an extremely difficult task. It either
requires expensive human annotations [28] or is still limited to simple
synthetic scenes [64, 52, 31]. [43, 65] allow for multi-agent control, but
leverage ground truth bounding boxes during the training. YODA in turn is an
autoregressive generative model for controllable video generation from sparse
motion controls that i) efficiently takes memory into account to simulate
long-range outcomes of the actions, ii) models stochasticity of the future,
iii) does not require human annotation for obtaining the control signal(s),
and iv) demonstrates controllability on a complex multi-object real dataset.

##  3 Training YODA

We denote with $\mathbf{x}=\\{x^{1},\dots,x^{N}\\}$ an RGB video that contains
$N$ frames, where $x^{i}\in\mathbb{R}^{3\times H\times W},i=1,\dots,N$, and
$H$ and $W$ are the height and the width of the frames respectively. The goal
is to build a controllable video prediction method that allows us to
manipulate separate objects in the scene. We formulate this goal as that of
approximating a sampler from the following conditional distribution

$\displaystyle p(x^{k+1}|x^{k},x^{k-1},\dots,x^{1},a^{k}),$ (1)

for $k<N$ and where $a^{k}$ denotes the motion control input. $a^{k}$
specifies the desired shifts at a set of pixels (including the special cases
with a single pixel or none). Our ultimate objective is to ensure through
training that this control implicitly defines the shift(s) for the object(s)
containing the selected pixel(s). The conditioning in eq. (1) allows an
autoregressive generative process at inference time, where the next frame
$x^{k+1}$ in a generated video is sampled conditioned on the current frame
$x^{k}$, the previously generated frames $x^{k-1},\dots,x^{1}$ and the current
control $a^{k}$. To model the conditional distribution in eq. (1), we use
RIVER [11]. This is a recently proposed video prediction method based on
conditional flow matching [38]. We chose RIVER due to its simplicity and
training efficiency compared to conventional RNNs [3, 25, 2, 12, 36] and
Transformers [58, 62, 35, 49, 17, 53] for video prediction. For completeness,
we briefly introduce Flow matching and RIVER in section 3.1. In section 3.2 we
show how the latter is adapted to handle control. In section 3.3, we focus on
how the control signals are obtained and encoded.

###  3.1 Preliminaries: Flow Matching and RIVER

Flow matching [38, 40, 1] was introduced as a simpler, more general and more
efficient alternative to diffusion models [22]. The goal is to build an
approximate sampler from the unknown data distribution $q(y)$, given a
training set of samples of $y$. This is formalized as a continuous normalizing
flow [7] via the following ordinary differential equation

$\displaystyle\dot{\phi}_{t}(y)$ $\displaystyle=v_{t}(\phi_{t}(y))$ (2)
$\displaystyle\phi_{0}(y)$ $\displaystyle=y.$

Eq. (2) defines a flow
$\phi_{t}(y):[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ that pushes
$p_{0}(y)={\cal N}(y\,|\,0,1)$ towards the distribution $p_{1}(y)\approx q(y)$
along the vector field
$v_{t}(y):[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$. Remarkably,
[38] shows that one can obtain $v_{t}(y)$ by solving

$\displaystyle\min_{v_{t}}\;$
$\displaystyle\mathbb{E}_{t,p_{t}(y\,|\,y_{1}),q(y_{1})}{\cal L}(\theta),$ (3)
$\displaystyle\text{with \;}{\cal L}(\theta)$
$\displaystyle=\|v_{t}(y)-u_{t}(y\,|\,y_{1})\|^{2},$ (4)

where one can explicitly define the vector field $u_{t}(y|y_{1})$ and its
corresponding probability density path $p_{t}(y|y_{1})$, with $y_{1}\sim
q(y)$. A particularly simple choice [38] is the Gaussian probability path
$p_{t}(y\,|\,y_{1})={\cal N}(y\,|\,\mu_{t}(y_{1}),\sigma^{2}_{t}(y_{1}))$,
with
$\mu_{0}(y_{1})=0,\mu_{1}(y_{1})=y_{1},\sigma_{0}(y_{1})=1,\sigma_{1}(y_{1})=\sigma_{\text{min}}$.
The corresponding target vector field is then given by

$\displaystyle
u_{t}(y\,|\,y_{1})=\frac{y_{1}-(1-\sigma_{\text{min}})y}{1-(1-\sigma_{\text{min}})t}.$
(5)

Sampling from the learned model can be obtained by first sampling
$y_{0}\sim{\cal N}(y\,|\,0,1)$ and then by numerically solving eq. (2) to
obtain $y_{1}=\phi_{1}(y_{0})$.

RIVER [11] is an extension of the above procedure to the video prediction task
with a computationally efficient conditioning scheme on past frames. The
training objective of RIVER is given by

$\displaystyle{\cal
L}_{\text{R}}(\theta)=\|v_{t}(x\,|\,x^{\tau-1},x^{c},\tau-c\,;\theta)-u_{t}(x\,|\,x^{\tau})\|^{2},$
(6)

where $v_{t}$ is a network with parameters $\theta$, $x^{\tau}$ is a frame
randomly sampled from the training video, $c$ is an index randomly sampled
uniformly in the range $\\{1,\dots,\tau-2\\}$ and $u_{t}$ is calculated with
eq. (5). An additional information provided to $v_{t}$ is the time interval
$\tau-c$ between the target frame $x^{\tau}$ and the past frame $x^{c}$, which
we call the _context frame_.

At test time, during the integration of eq. (2), a new context frame $x^{c}$
is sampled at each step $t$. This procedure enables to condition the
generation of the next frame on the whole past. To further speed up the
training and enable high-resolution video synthesis, RIVER works in the latent
space of a pretrained VQGAN [15]. That is, instead of
$x^{\tau},x^{\tau-1},x^{c}$ in eq. (6) one should write
$z^{\tau},z^{\tau-1},z^{c}$, where $z^{i}$ is the VQ latent code of the $i$-th
frame [11]. Since the use of VQGAN encoding is an optional and separate
procedure, we simply use $x$ in our notation.

###  3.2 Learning to Master the Force

Figure 2: Overview of YODA’s architecture. The noisy target frame $x$, the
reference frame $x^{\tau-1}$ and the context frame $x^{c}$ are concatenated,
reshaped and projected to form a sequence of visual tokens that is augmented
with position encodings and embedded relative temporal distance ($\tau-c$)
between frames and fed to the U-ViT [4] alongside with the embedded time token
($t$). A sequence of control tokens is fused into the pipeline via cross-
attention in the bottleneck of the network.

We now show how to incorporate control into eq. (6) to build a sampler for the
conditioning probability in eq. (1). To do so, we make $v_{t}$ depend on
$a^{\tau-1}$, which is the motion control at time $\tau-1$, as shown in the
following objective

$\displaystyle{\cal
L}_{\text{F}}(\theta)=\|v_{t}(x\,|\,x^{\tau-1},x^{c},\tau-c,a^{\tau-1}\,;\theta)-u_{t}(x\,|\,x^{\tau})\|^{2}.$
(7)

In practice, we implement this conditioning by substituting the bottleneck in
the self-attention layers of the U-ViT [4] architecture of RIVER with cross-
attention blocks (see Figure 2). The control inputs $a^{\tau-1}$ are obtained
by splitting the image domain into a grid of tiles, so that a motion control
can be specified in each tile via a code, and then be fed as keys and values
to the cross-attention layer. More details on $a^{\tau-1}$ will be provided in
the next section.

Inspired by the classifier-free guidance for diffusion models [23], we propose
to switch off the conditioning on both the context and motion control at every
iteration, with some probability $\pi$ (see Algorithm 1). This is done by
substituting the code corresponding to a switched off context or motion
control with noise. A typical value for $\pi$ in our experiments is 0.5. This
serves two purposes: 1) it yields a stronger model that can effectively take
the conditioning into account and 2) it yields a model that can generate a
video by starting from a single frame. To see 1), consider that the
conditioning on both the context frame $x^{c}$ and the control $a^{\tau-1}$ is
redundant when the future frame can be reliably predicted given only one of
the two. In these cases, the model might learn to ignore one of the inputs. To
see 2), consider that when the model generates the first predicted frame,
there is no valid context frame and our procedure allows us to replace the
context frame with noise. Otherwise, one would have to duplicate the first
frame, for example, but this would result in an undesirable training bias.

Algorithm 1 Training of YODA

1: Input: dataset of videos $D$, number of iterations $N$;

2: for $i$ in range(1, $N$) do

3: Sample a video from the dataset $\mathbf{x}\sim D$ of length $L$;

4: Choose a random target frame $x^{\tau},\tau\in\\{3,\dots L\\}$

5: Sample a timestamp $t\sim U[0,1]$;

6: Sample a noisy observation $x\sim p_{t}(x\,|\,x^{\tau})$;

7: Calculate $u_{t}(x\,|\,x^{\tau})$ according to eq. (5);

8: Sample a condition frame $x^{c},c\in\\{1,\dots\tau-2\\}$;

9: Compute the control $a^{\tau-1}$ (see sec. 3.3);

10: With probability $\pi$ set $x^{c}$ to noise;

11: With probability $\pi$ set $a^{\tau-1}$ to noise;

12: Take a gradient descent update
$\theta\leftarrow\theta-\alpha\nabla_{\theta}\mathcal{L}_{\text{F}}(\theta)$
with learning rate $\alpha>0$;

13: end for

###  3.3 Force Embeddings

Ideally, $a^{\tau-1}$ could encode detailed motion information for the objects
in the scene. For instance, $a^{\tau-1}$ could describe that an object is
rotating or pressing against another object or walking (in the case of a
person). Such supervision could potentially provide the ability to control the
video generation in detail and to generalize well to unseen object motion
combinations. However, obtaining such ground truth control signals requires
costly large-scale manual annotation. Similarly to [5], we avoid such costs by
leveraging optical flow. Essentially, instead of using a costly and detailed
motion representation, we use a simpler one that can be computed automatically
and at a large scale.

Given an optical flow $w^{\tau}\in\mathbb{R}^{2\times H\times W}$ between the
frames $x^{\tau}$ and $x^{\tau+1}$ (obtained with a pretrained RAFT [55]), we
define a probability density function $p(i,j)\propto\|w^{\tau}_{ij}\|^{2}$
over the image domain $\Omega$, with $(i,j)\in\Omega$. Then, we randomly
sample a sparse set $\mathcal{S}\subset\Omega$ of $n_{c}=|\mathcal{S}|$ pixel
locations from $p$. This distribution makes it more likely that pixels of
moving objects will be selected. However, in contrast to [5] we do not
introduce additional restrictions to the sampling or explicitly define the
background. This is an essential difference, because in multi-object scenes,
objects that belong to the background in one video might be moving in another.
Thus, in our case, one cannot use the magnitude of the optical flow to
separate objects from the background in each video.

To condition the video generation on the selected optical flow vectors, we
introduce an encoding procedure. First, we construct a binary mask
$m\in\\{0,1\\}^{1\times H\times W}$ such that $m_{ij}=1$,
$\forall(i,j)\in{\cal S}$ and $m_{ij}=0$ otherwise. This mask is concatenated
in the channel dimension with $m\odot w^{\tau}$ to form a tensor
$\tilde{w}^{\tau}$ of shape $(3,H,W)$, which we refer to as the _sparse
optical flow_. $\tilde{w}^{\tau}$ is further tiled into a $16\times 16$ grid.
Each of these tiles is independently projected through an MLP and augmented
with a learnable positional encoding to output a code (see Figure 3). This
particular design of the optical flow encoder is a trade-off between having a
restricted receptive field (because each tile is processed independently) and
efficiency. A small receptive field is needed to ensure that separate controls
minimally interact before being fed to the cross-attention layers (see Figure
2). We found that this is crucial to enable the independent manipulation of
separate objects.

Figure 3: Sparse optical flow (OF) encoder. The dense OF is sampled and tiled
in a $16\times 16$ grid. Each OF tile is fed independently to the MLP, and
then combined with a learnable positional encoding into a code.

##  4 Experiments

In this section, we evaluate YODA to see how controllable the video generation
is, _i.e._ , how much the generated object motion correlates with the input
motion control (see next section), and to assess the image and sequence
quality on three datasets with different scene and texture complexities, as
well as different object dynamics. In the latter case, we report several video
quality metrics, such as FVD [57] and average LPIPS [70], PSNR, SSIM [61] and
FID [47]. Implementation and training details are provided in the
supplementary material.

Figure 4: Violin plots of the control vs generated motion error distributions.
On the top is the distribution of the errors obtained when using a
convolutional encoding of the motion controls (_i.e._ , with a large receptive
field). At the bottom is the distribution obtained with the chosen encoding
(_i.e._ , with a limited receptive field). The chosen encoding decreases the
average error and also makes the model more stable (there is no long tail).

###  4.1 Evaluation of Intended vs Generated Motion

The objective of our training is to build a model to generate videos that can
be controlled by specifying motion through $a^{\tau-1}$ as a set of shifts at
some user-chosen pixels. To evaluate how much the trained model follows the
intended control, we introduce the following metrics: Local and global errors.
To compute them, we sample an image from the test videos. Then, we randomly
select one object in the scene and apply a random motion control input to a
pixel of that object. Because the generated images are of high-quality, we can
use a pre-trained optical flow estimation model [55] to calculate the optical
flow between the first image and the generated one. In principle, one could
measure the discrepancy between the input motion shift and the generated one
at the same pixel. However, since single pixel measurements are too noisy to
use, we assume that all pixels within a small neighborhood around the selected
pixel move in the same way, and then average the estimated optical flow within
that neighborhood to compare it to the input control vector (depending on what
we focus on, we use the relative $L_{2}$ norm or a cosine similarity). We call
this metric the _local error_ (see Figure 5 on the left in blue). Notice that
in some cases, the model could use the motion input to generate a rotated
object, instead of a translated one. Also, the chosen neighborhood (whose size
is fixed) may not fully cover only the object of interest. These issues make
this metric quite noisy. Nonetheless, it still provides a useful approximation
of the average response of the trained model to the control inputs. We also
assume that the generated motion should be zero sufficiently far away from
where the motion control is applied, although in general a local motion could
cause another motion far away. To assess this, we calculate the average
$L_{2}$ norm of the estimated optical flow outside some neighborhood of the
controlled pixel. We call this metric the _global error_ (see Figure 5 on the
right in orange). This measurement tells us the spatial extent of the learned
motion control correlations.

###  4.2 Datasets

We evaluate YODA on the following three datasets:

CLEVRER [68] is a dataset containing 10K training and 1000 test videos
capturing a synthetic scene with multiple simple objects interacting with each
other through collisions. We cropped and downsampled the videos to $128\times
128$ resolution. On CLEVRER we show the ability of our model to model complex
cascading interactions and also to learn the control of long-term motions
(_i.e._ , motions that once started at one frame can last for several frames).

BAIR [14] is a real dataset containing around 44K videos of a robot arm
pushing toys on a flat square table. The resolution of these videos is
$256\times 256$ pixels. The visual complexity of BAIR is much higher than that
of CLEVRER. The objects on the table are diverse and include non-rigid
objects, such as stuffed toys, that have different physical properties.
However, in contrast to CLEVRER, its interactions are simpler and do not
require modeling long-term dynamics.

iPER [39] captures 30 humans with diverse styles performing various movements
of varying complexity. The official train/test split separates the dataset
into 164 training and 42 test clips. Although our main focus is to work with
multi-object datasets, we use this dataset for two reasons: 1) We can test how
YODA learns to control articulated objects, such as humans; 2) We can compare
to the related work iPOKE [5], which has already been tested on this dataset
(and is not designed for multi-object datasets).

###  4.3 Ablations

Sparse optical flow encoder. First, we show the importance of using a sparse
optical flow encoder with a restricted receptive field. We observe that such
an encoder is essential for the model to learn to independently control
different objects in the scene. We manually annotated 128 images from the test
set of the BAIR dataset. For each image we store a list of pixel coordinates
that belong to objects in the scene, 1-2 pixels per object, 3 pixels for the
robot arm. We use the local error to compare our encoder with a convolutional
sparse optical flow encoder from [5]. In Figure 4, we show that our restricted
receptive field optical flow encoder outperforms the convolutional one.

Figure 5: The effect of number of control vectors used during the training. The _local error_ (in blue and on the top) stands for the error of the optical flow in the neighborhood of the controlled pixel, while the _global error_ (in orange and at the bottom) is the average optical flow vector outside of a circle around the interacted pixel. A smaller local error indicates a better response of the model to the control, while a low global error ensures that only the object of interest moves. The boxes at the bottom illustrate the regions across which the error is averaged (left - local error, right - global error). Best viewed in color. $n_{c}$ | $\pi$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$ |  FVD$\downarrow$
---|---|---|---|---|---|---
5 | 0.5 | 0.126 | 30.07 | 0.93 | 5.7 | 70
1 | 0.5 | 0.144 | 28.78 | 0.92 | 6.3 | 83
5 | 0.0 | 0.288 | 24.88 | 0.86 | 9.9 | 401
Table 1: Evaluation of the generated videos on _CLEVRER_ under different input sparsity ($n_{c}$) and randomization ($\pi$). Transition |  [width=2.4cm]3Figures/bair_transition/image_013 |  |  |  |  |  |
---|---|---|---|---|---|---|---
Multi-obj. control |  [width=2.4cm]3Figures/bair_multiobj/image_09 |  |  |  |  |  |
Figure 6: More object manipulation scenarios with YODA on the BAIR dataset.
The vectors in red are the control signals specified per frame in an
interactive regime. The images in the first column can be played as videos in
Acrobat Reader.

Randomized conditioning. In Table 1, we demonstrate the importance of our
randomized conditioning scheme, where we randomly switch off the conditioning
with respect to both the past frame and the control input with probability
$\pi$. The comparisons on the CLEVRER dataset show that randomized
conditioning is crucial for the single frame quality as well as for temporal
consistency (see the FVD metric).

Number of control inputs during the training. $n_{c}$ plays an important role
in enabling the independent control over separate objects. In principle,
responses to the control should be more and more decorrelated as we increase
the number of control inputs. However, values of $n_{c}$ that are too large
would increase the gap between the training and the test conditions, where
often only 1 control is used. At the same time, using fewer optical flow
vectors during the training stimulates the network to learn interactions
(_i.e._ , correlations with other objects). Therefore, we observe a trade-off
between controllability and learning correct dynamics in the choice of $n_{c}$
(see Figure 5). We choose $n_{c}=5$ in the remaining experiments.

###  4.4 Quantitative results

Realism and motion consistency. Following [44], we train YODA on BAIR and then
autoregressively generate 29 frames given the first frame and a set of
controls at each generation step, decoded from the ground truth videos (in our
case from the corresponding optical flows). We report the metrics for 1 and 5
control vectors at each timestamp and show that the 5 controls version
outperforms all prior work (see Table 2). Moreover, notice that prior work on
controllable generation on BAIR only focuses on modeling the actions of the
robot arm, while YODA is able to also effectively control other objects in the
scene.

Method |  LPIPS$\downarrow$ |  FID$\downarrow$ |  FVD$\downarrow$
---|---|---|---
MoCoGAN [56] | 0.466 | 198.0 | 1380
MoCoGAN+ [44] | 0.201 | 66.1 | 849
SAVP [36] | 0.433 | 220.0 | 1720
SAVP+ [44] | 0.154 | 27.2 | 303
CADDY [44] | 0.202 | 35.9 | 423
Huang et al. [29] _positional_ | 0.202 | 28.5 | 333
Huang et al. [29] _affine_ | 0.201 | 30.1 | 292
Huang et al. [29] _non-param_ | 0.176 | 29.3 | 293
GLASS [10] | 0.118 | 18.7 | 411
YODA (ours) 5 controls | 0.112 | 18.2 | 264
YODA (ours) 1 control | 0.142 | 19.2 | 339
Table 2: Video quality comparisons on the _BAIR_ dataset.

We also compare our model on the benchmark introduced in [5] and generate 9
frames starting from a single initial frame. Although our model does not
outperform [5], it does better than all the other prior work, as indicated by
our FVD metric (see Table 3). One should also notice that YODA does not make
use of the same information used in iPOKE [5] (_e.g._ , what a background is).

Method |  FVD$\downarrow$ |  LPIPS$\downarrow$ |  SSIM$\uparrow$
---|---|---|---
Hao [18] | 235.08 | 0.11 | 0.88
Hao [5] w/ KP | 141.07 | 0.04 | 0.93
II2V [6] | 220.34 | 0.07 | 0.89
iPOKE [5] | 77.50 | 0.06 | 0.87
YODA (ours) 1st frame | 133.08 | 0.09 | 0.86
YODA (ours) all frames | 134.35 | 0.09 | 0.87
Table 3: Video quality comparisons on the _iPER_ dataset.

Scene dynamics and interactions. We chose to assess how well YODA models the
physics of the scene and object interactions on the CLEVRER dataset. For this
purpose, we generate 15 frames starting from a single frame and a set of
control vectors. This time we do not specify future controls like in BAIR and
let the model simulate the learned physics. We repeat the experiment with 1
and 5 control vectors. The metrics in Table 1 show that YODA models the
interior object dynamics well, which is further supported by the qualitative
results in section 4.5.
Controllability. We asses how robust YODA is to changes in the control
parameters, such as the direction and the magnitude of the control vectors. We
manually annotated 128 images from the CLEVRER dataset by indicating 1
potential control point per object. We then sample some points and random
control vectors from the annotated ones and feed those to the model to
generate the next frame. We calculate the error between the control vectors
and the average computed optical flow [55] in the neighborhood of the
interacted pixel. We use the cosine distance between the normalized vectors as
an indicator of the accuracy of the control. We show how this metric changes
with the parameters of the control in Figure 7.

Figure 7: Average cosine error between the control vector and the resulting optical flow depending on the magnitude of the control (in pixels) and its direction on CLEVRER.  [width=2.1cm]3Figures/cl_slow/image_09 |  |  |
---|---|---|---
[width=2.1cm]3Figures/cl_fast/image_09 |  |  |
[width=2.1cm]3Figures/cl_other/image_09 |  |  |
[width=2.1cm]3Figures/cl_together/image_09 |  |  |
Figure 8: The leftmost frame is the same across all four rows. In each row YODA is given the controls indicated as a red arrow on the leftmost column. YODA follows accurately the magnitude (first to second rows) and direction (first to third rows) of the motion control input, as well as multiple inputs (fourth row). Notice that since YODA has memory and was trained also to generate frames without control, it can learn to propagate motion across many frames from a single initial control input. The images in the first column can be played as videos in Acrobat Reader.  |  |
---|---|---
|  |
Figure 9: The attention maps from the last layer of the vector field regressor between the interacted location and the rest of the image, averaged over the heads and the flow integration path. It can be seen that the attention maps roughly cover the extents of the objects. [width=2.1cm]3Figures/iper_p1/image_05 |  |  |
---|---|---|---
[width=2.1cm]3Figures/iper_p2/image_06 |  |  |
Figure 10: Examples of YODA simulating human motion on iPER. The images in the
first column can be played as videos in Acrobat Reader.

###  4.5 Qualitative results

In this section we provide some visual examples of the generated sequence. On
the BAIR dataset, we highlight the capability of YODA to move, rotate and
deform separate objects without the robot arm physically touching them.
Figures LABEL:fig:bair_teaser and 6 show different object manipulations on the
BAIR test set. Notice how the model has learned also the 3D representations
and interactions of the objects. Figure 8 shows the diversity of generated
videos that share the same initial frame, but use different control signals.
Notice the high correlation between the intended and generated motions and the
ability of the model to correctly predict the interactions between the
colliding objects. Since our model is based on Transformers [58], we report
the attention maps from the last layer of the network between the interacted
location and the rest of the image (see middle column in Figure 9). These
attention maps often correspond to coarse segmentations of the controlled
objects. Finally, we provide some control sequences on the iPER dataset to
show that YODA simulates realistic motions of humans (see Figure 10). More
qualitative results, including possible applications of YODA, are in the
supplementary materials.

##  5 Conclusion

In this paper we introduced YODA, a novel method for controllable video
generation from sparse motion input. Our experimental evaluation demonstrates
the ability of YODA to generate realistic multi-object videos, which involves
learning the extents and interactions between multiple objects despite only
passively observing their correlated motions.

Acknowledgements This work was supported by grant 188690 of the Swiss National
Science Foundation.

##  Appendix A Architecture and training details

Autoencoder. We trained a VQGAN [15] autoencoder per dataset, using the
official taming transformers repository.111https://github.com/CompVis/taming-
transformers The configurations of the VQGANs can be found in Table 4. Notice
that we did not use a discriminator for the CLEVRER [68] dataset.

Sparse optical flow encoder. The $16\times 16$ tiled grid of sparse optical
flow inputs is reshaped and linearly projected to 256 256-dimensional vectors
that are fed into 5 subsequent blocks of (batch normalization [30], fully-
connected layer and gelu activation [20]). The activation is omitted in the
last block. This procedure results into a representation of the control input
as 256 256-dimensional vectors.

Training. All models are trained for 300K iterations with the AdamW [41]
optimizer with the base learning rate equal to $10^{-4}$ and weight decay
$5\cdot 10^{-6}$. A learning rate linear warmup for 5K iterations is applied
as well as a square root decay schedule. For the CLEVRER [68] dataset,
following [11], random color jittering is additionally used to prevent
overfitting. For the iPER [39] dataset, a random horizontal flip and a random
time reversal is applied to prevent overfitting to the pose of the human. As
suggested in [11], we used $\sigma_{\text{min}}=10^{-7}$ in the flow matching
loss [38] in all the experiments. In the final models, we used $n_{c}=5$
optical flow vectors for the control input.

| CLEVRER $128\times 128$ [68] | BAIR $256\times 256$ [14] | iPER $128\times 128$ [39]
---|---|---|---
embed_dim | 4 | 8 | 4
n_embed | 8192 | 16384 | 16384
double_z | False | False | False
z_channels | 4 | 8 | 4
resolution | 128 | 256 | 128
in_channels | 3 | 3 | 3
out_ch | 3 | 3 | 3
ch | 128 | 128 | 128
ch_mult | [1,2,2,4] | [1,1,2,2,4] | [1,2,2,4]
num_res_blocks | 2 | 2 | 2
attn_resolutions | [16] | [16] | [32]
dropout | 0.0 | 0.0 | 0.0
disc_conditional | - | False | False
disc_in_channels | - | 3 | 3
disc_start | - | 20k | 20k
disc_weight | - | 0.8 | 0.8
codebook_weight | - | 1.0 | 1.0
Table 4: Configurations of VQGAN [15] for different datasets.

##  Appendix B Qualitative results

In this section we provide more qualitative results with YODA. For videos,
please, visit our project’s website https://araachie.github.io/yoda. Figures
11 and 12 contain some selected sequences to demonstrate the quality of
separate frames, which is not visible in the videos on the webpage due to gif
compression. Figure 11 contains more object manipulation scenarios on the BAIR
[14] dataset. Figure 12 provides generated videos capturing long-range
consequences of the input controls on the CLEVRER [68] dataset. Notice the
ability of YODA to realistically model the physics of the scene. YODA was
designed to make it possible to intervene into the generation process at any
timestamp, which allows communicating new impulses to the objects on the fly.

[width=2.5cm]7Figures/supp/bair/rotate_brush/image_028 |  |  |  |  |  |
---|---|---|---|---|---|---
[width=2.5cm]7Figures/supp/bair/jar/image_027 |  |  |  |  |  |
Figure 11: Different scene manipulations with YODA on the BAIR [14] dataset. Please, use Acrobat Reader to play the frames in the leftmost column or visit our project’s website https://araachie.github.io/yoda. [width=2.5cm]7Figures/supp/clevrer/4/image_030 |  |  |  |  |  |
---|---|---|---|---|---|---
[width=2.5cm]7Figures/supp/clevrer/5/image_060 |  |  |  |  |  |
Figure 12: Videos generated with YODA on the CLEVRER [68] dataset. Please, use
Acrobat Reader to play the frames in the leftmost column or visit our
project’s website https://araachie.github.io/yoda.

##  Appendix C Demo

To ease the access to our results, we designed a demo of YODA. The user can
load the pretrained models and directly interact with the scene by dragging
objects with the mouse. The model responds by generating a subsequent frame
taking into account the controls specified by the user as well as the
previously generated images (see the video on the website). The demo and the
code will be released to the public.

##  Appendix D Applications

Pretrained YODA generates realistic responses to control. This opens up the
opportunity to interact with the scene in a counterfactual way, which makes it
possible to apply YODA to downstream tasks. In theory, by optimizing the
control inputs to match the given video, one can solve planning or compress
the video to a single frame and a sequence of controls. Here we discuss
another possible application, object segmentation.

Given an image $x$ of a scene with multiple objects and YODA trained on videos
capturing that scene, the user selects a pixel on the object of interest. A
random control is then applied to that location to generate the response
$x^{\prime}$ with YODA. The optical flow field $w$ between $x$ and
$x^{\prime}$ is calculated. All the vectors in $w$ are compared with the input
control using some similarity measure (can be $l_{2}$ or cosine distance). We
then apply a threshold to the result to obtain the segmentation mask. For
robustness, this procedure can be repeated multiple times and the union of the
resulting masks can be used as the final estimator of the segmentation mask.
For an example, see Figure 13.

---
Figure 13: From left to right: the input image, the optical flow between the
input image and the next one, the estimated mask. The red arrows in the images
in the first column show the control input.

## References

  * [1] Michael S Albergo and Eric Vanden-Eijnden.  Building normalizing flows with stochastic interpolants.  arXiv preprint arXiv:2209.15571, 2022.
  * [2] Mohammad Babaeizadeh, Chelsea Finn, D. Erhan, Roy H. Campbell, and Sergey Levine.  Stochastic variational video prediction.  ArXiv, abs/1710.11252, 2018.
  * [3] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan.  Fitvid: Overfitting in pixel-level video prediction.  arXiv preprint arXiv:2106.13195, 2021.
  * [4] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu.  All are worth words: a vit backbone for score-based diffusion models.  arXiv preprint arXiv:2209.12152, 2022.
  * [5] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Björn Ommer.  ipoke: Poking a still image for controlled stochastic video synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14707–14717, 2021.
  * [6] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Bjorn Ommer.  Understanding object dynamics for interactive image-to-video synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5171–5181, 2021.
  * [7] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  Advances in neural information processing systems, 31, 2018.
  * [8] Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed.  Recurrent environment simulators.  arXiv preprint arXiv:1704.02254, 2017.
  * [9] Aidan Clark, Jeff Donahue, and Karen Simonyan.  Adversarial video generation on complex datasets.  arXiv: Computer Vision and Pattern Recognition, 2019.
  * [10] Aram Davtyan and Paolo Favaro.  Controllable video generation through global and local motion dynamics.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 68–84, Cham, 2022. Springer Nature Switzerland.
  * [11] Aram Davtyan, Sepehr Sameni, and Paolo Favaro.  Randomized conditional flow matching for video prediction.  arxiv, 2022.
  * [12] Emily Denton and Rob Fergus.  Stochastic video generation with a learned prior.  In International conference on machine learning, pages 1174–1183. PMLR, 2018.
  * [13] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  ArXiv, abs/2105.05233, 2021.
  * [14] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine.  Self-supervised visual planning with temporal skip connections.  In CoRL, 2017.
  * [15] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  * [16] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  Advances in neural information processing systems, 29, 2016.
  * [17] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei.  Maskvit: Masked visual pre-training for video prediction.  arXiv preprint arXiv:2206.11894, 2022.
  * [18] Zekun Hao, Xun Huang, and Serge Belongie.  Controllable video generation with sparse trajectories.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7854–7863, 2018.
  * [19] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  arXiv preprint arXiv:2205.11495, 2022.
  * [20] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [21] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [22] Jonathan Ho, Ajay Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  ArXiv, abs/2006.11239, 2020.
  * [23] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [24] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  arXiv preprint arXiv:2204.03458, 2022.
  * [25] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Comput., 9(8):1735–1780, nov 1997.
  * [26] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi.  Diffusion models for video prediction and infilling.  arXiv preprint arXiv:2206.07696, 2022.
  * [27] Qiyang Hu, Adrian Wälchli, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro.  Learning to take directions one step at a time.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 739–746. IEEE, 2021.
  * [28] Yaosi Hu, Chong Luo, and Zhenzhong Chen.  Make it move: Controllable image-to-video generation with text descriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18219–18228, 2022.
  * [29] Jiahui Huang, Yuhe Jin, Kwang Moo Yi, and Leonid Sigal.  Layered controllable video generation.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 546–564, Cham, 2022. Springer Nature Switzerland.
  * [30] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [31] Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu.  Reasoning about physical interactions with object-oriented prediction and planning.  arXiv preprint arXiv:1812.10972, 2018.
  * [32] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34:852–863, 2021\.
  * [33] Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler.  Learning to simulate dynamic environments with gamegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1231–1240, 2020.
  * [34] Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim.  Unsupervised keypoint learning for guiding class-conditional video prediction.  Advances in neural information processing systems, 32, 2019.
  * [35] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid.  Ccvs: Context-aware controllable video synthesis.  Advances in Neural Information Processing Systems, 34:14042–14055, 2021.
  * [36] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.  Stochastic adversarial video prediction.  arXiv preprint arXiv:1804.01523, 2018.
  * [37] Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas E. Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong.  Revisiting hierarchical approach for persistent long-term video prediction.  ArXiv, abs/2104.06697, 2021.
  * [38] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.  Flow matching for generative modeling.  arXiv preprint arXiv:2210.02747, 2022.
  * [39] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5904–5913, 2019.
  * [40] Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  arXiv preprint arXiv:2209.03003, 2022.
  * [41] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [42] Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan.  Transformation-based adversarial video prediction on large-scale data.  ArXiv, abs/2003.04035, 2020.
  * [43] Willi Menapace, Stéphane Lathuiliere, Aliaksandr Siarohin, Christian Theobalt, Sergey Tulyakov, Vladislav Golyanik, and Elisa Ricci.  Playable environments: Video manipulation in space and time.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3584–3593, 2022.
  * [44] Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci.  Playable video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10061–10070, 2021.
  * [45] Manuel Serra Nunes, Atabak Dehban, Plinio Moreno, and José Santos-Victor.  Action-conditioned benchmarking of robotic video prediction models: a comparative study.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8316–8322. IEEE, 2020.
  * [46] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh.  Action-conditional video prediction using deep networks in atari games.  Advances in neural information processing systems, 28, 2015.
  * [47] Gaurav Parmar, Richard Zhang, and Junyan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11400–11410, 2022.
  * [48] Judea Pearl and Dana Mackenzie.  The book of why: the new science of cause and effect.  Basic books, 2018.
  * [49] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.  Latent video transformer.  arXiv preprint arXiv:2006.10704, 2020.
  * [50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [51] Oleh Rybkin, Karl Pertsch, Konstantinos G Derpanis, Kostas Daniilidis, and Andrew Jaegle.  Learning what you can do before doing anything.  arXiv preprint arXiv:1806.09655, 2018.
  * [52] Karl Schmeckpeper, Georgios Georgakis, and Kostas Daniilidis.  Object-centric video prediction without annotation.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13604–13610. IEEE, 2021.
  * [53] Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, and P. Abbeel.  Harp: Autoregressive latent video prediction with high-fidelity image generator.  ArXiv, abs/2209.07143, 2022.
  * [54] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [55] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [56] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz.  Mocogan: Decomposing motion and content for video generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018.
  * [57] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  ArXiv, abs/1812.01717, 2018.
  * [58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [59] Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal.  Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation.  arXiv preprint arXiv:2205.09853, 2022.
  * [60] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva.  Imaginator: Conditional spatio-temporal gan for video generation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1160–1169, 2020.
  * [61] Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13:600–612, 2004.
  * [62] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit.  Scaling autoregressive video models.  arXiv preprint arXiv:1906.02634, 2019.
  * [63] Nevan Wichers, Ruben Villegas, D. Erhan, and Honglak Lee.  Hierarchical long-term video prediction without supervision.  ArXiv, abs/1806.04768, 2018.
  * [64] Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn.  Generative video transformer: Can objects be the words?  In International Conference on Machine Learning, pages 11307–11318. PMLR, 2021.
  * [65] Wilson Yan, Ryo Okumura, Stephen James, and Pieter Abbeel.  Patch-based object-centric transformers for efficient video generation.  arXiv preprint arXiv:2206.04003, 2022.
  * [66] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [67] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  arXiv preprint arXiv:2203.09481, 2022.
  * [68] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum.  Clevrer: Collision events for video representation and reasoning.  ArXiv, abs/1910.01442, 2020.
  * [69] Jiji Zhang and Peter Spirtes.  Intervention, determinism, and the causal minimality condition.  Synthese, 182:335–347, 2011.
  * [70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.