11institutetext: Institute for Applied AI, Hochschule der Medien Stuttgart,
Germany22institutetext: Institute of Industrial Manufacturing and Management
IFF, University of Stuttgart, Germany33institutetext: Fraunhofer Institute for
Manufacturing Engineering and Automation IPA, Stuttgart, Germany

# ViPro: Enabling and Controlling Video Prediction for Complex Dynamical
Scenarios using Procedural Knowledge

Patrick Takenaka 1122 0009-0005-2295-562X Johannes Maucher 11
0000-0002-3804-8937 Marco F. Huber<EMAIL_ADDRESS><EMAIL_ADDRESS>2233 0000-0002-8250-2092

###### Abstract

We propose a novel architecture design for video prediction in order to
utilize procedural domain knowledge directly as part of the computational
graph of data-driven models. On the basis of new challenging scenarios we show
that state-of-the-art video predictors struggle in complex dynamical settings,
and highlight that the introduction of prior process knowledge makes their
learning problem feasible. Our approach results in the learning of a
symbolically addressable interface between data-driven aspects in the model
and our dedicated procedural knowledge module, which we utilize in downstream
control tasks.

###### Keywords:

Informed Machine Learning Procedural Knowledge Video Prediction.

##  1 Introduction

Current advances in deep learning research [2, 8, 19] have shown the
tremendous potential of data-driven approaches. However, upon taking a closer
look, oftentimes domain-specific inductive biases in the training process or
the architecture [8] play a critical role in making the most of the available
data. While an end-to-end generalized architecture that is able to tackle a
wide range of problems is elegant, we argue that this often leads to models
that require large amounts of data to be feasible while still being unable to
generalize well outside of the training distribution [16], leading to limited
applicability for more specialized use cases where data are often scarce, such
as, for instance, in the medical domain. Transfer of deep learning research
into applications usually requires further fine-tuning of the model for the
given use case, and this often corresponds to collecting specialized data. We
believe that providing additional means for domain experts—who may or may not
be experts in machine learning—to represent their knowledge besides data in
deep learning models is crucial for driving AI adaption in more areas.
Ideally, specializing the model to a domain should reduce the complexity of
the learning task and thus, lead to leaner architectures that require less
data than a domain agnostic variant, while providing better predictions for
sparsely observed situations. Furthermore, as we show with our approach, such
grey box modelling approaches also inherently increase the controllability of
the model. Thus, developing a model that benefits from both domain knowledge
and data samples together is our objective.

There are various types of domain knowledge that can be integrated, and ways
how they can be integrated [24], ranging from using logic rules or
differential equations to structure and expand both the loss function and the
learning process [32, 20], to architectural considerations that take into
account the structure of the underlying problem, such as Graph Neural Networks
(GNNs) [28] to model interactions, or more famously to use Convolutional
Neural Networks (CNNs) for spatial data.

We propose to view the knowledge integration types from a different
perspective and group them in either procedural or declarative knowledge.
While the latter encompasses domain facts or rules (“Knowing-That”),
procedural knowledge represents a more abstract level of information by
describing a domain process irrespective of any concrete observation
(“Knowing-How”). A straightforward representative of this paradigm would be a
mathematical formula, which can describe a certain relation between variables
in a concise manner without risk of running into a long-tail problem for less
frequently observed variable assignments as would be the case in data-driven
approaches. Without a doubt, however, data-driven function approximators are
immensely successful in the real world because we are often not able—or it is
impossible—to describe the problem in such a precise way. However, we argue
that currently data-driven approaches are often used without considering
whether the underlying mechanisms of the domain could be described in a more
efficient manner without resorting to arduous data collection.

In this work, we propose a novel architecture design that integrates such
procedural knowledge as an independent module into the overall architecture.
We apply it to video prediction, a task where state-of-the-art models often
still struggle due to the high spatio-temporal complexity involved in scenes.
Its environments often involve understanding complex domain processes that are
hard to robustly learn from observations only and thus are likely to benefit
from domain inductive biases. At the same time, this field is the foundation
for many possible downstream tasks such as Visual Question Answering (VQA)
[30], Model Predictive Control (MPC) [7], or system identification [7]. We
create several scenarios which feature complex dynamics, and integrate the
knowledge about these dynamics. We show that current deep learning models
struggle on their own, but can thrive once enhanced with procedural domain
knowledge. We verify that this is still possible even with very limited data,
and further highlight that such an interface enables control in the target
domain w.r.t. the integrated function parameters at test time, providing a
potential basis for downstream control tasks and allowing flexibility in
adjusting the model for novel scene dynamics.

In summary, our contributions are:

  * •

Specification and analysis of an architectural design for interfacing
procedural knowledge with a data-driven model.

  * •

Introduction of novel challenging scenarios with complex dynamics for video
prediction.

  * •

Application to a downstream control task by relying on the inherently achieved
disentanglement w.r.t. the function parameters.

The paper is structured as follows: First, relevant related work is shown in
Section 2, after which our proposed procedural knowledge integration scheme is
introduced in Section 3, followed by a description of our datasets in Section
4. In Section 5, we first describe our used setup including implementation
details in Section 5.1 and continue by establishing baseline results in
Section 5.2. We then analyze the latent state of the model and the resulting
controllability in Section 5.3. We conclude by discussing limitations and
potential directions for future work in Section 6. Our datasets and code are
available at https://github.com/P-Takenaka/nesy2024-vipro.

##  2 Related Work

Predicting future video frames is a challenging task that requires certain
inductive biases in the training process or model architecture in order to
lead to acceptable prediction outcomes. The most commonly integrated bias is
the modelling of the temporal dependency between individual frames, which
assumes that future frames are dependent on past frames [25, 3, 26]. Some
methods also exploit the fact that the scene is composed of objects by
structuring the latent space accordingly [12, 14, 30, 22], which improved
scene reconstruction performance further compared to approaches that rely on a
single global latent scene representation for predictions.

Since many dynamics in video scenes are of a physical nature, there are also
works that explore the learning of the underlying Partial Differential
Equations (PDEs) to facilitate video reconstruction [13, 4, 33, 29]. Another
line of work—most similar to our approach—considers a more explicit
representation of dynamics knowledge in the model [7, 9]. Here, discretized
PDEs are integrated to calculate a physical state for each frame, which is
decoded back into an image representation. These approaches were, however,
limited to 2D dynamics of sprites, which allowed the dynamics model to operate
directly in the screen space, making the learning problem much easier, while
limiting applicability to more realistic settings. These methods also relied
on the Spatial Transformer Network [6] for decoding purposes. Dynamical
properties of the scene besides the object positions—such as for instance
changing lighting conditions or object orientations—are not modelled with this
approach, since it is based on sampling pixel predictions directly from the
input reference frames. More recently, an architecture was proposed in a
preliminary workshop publication [21] that could in theory handle such dynamic
properties. It was, however, only applied to semantic segmentation in visually
simple settings and we show that it fails at video reconstruction in our
datasets. Our work continues in this line of research and increases the
complexity of video scenarios that can be handled, while broadening the
applicability by bridging the gap to control-based downstream tasks such as
MPC.

##  3 Proposed Architecture

Our objective is to allow domain experts to integrate their knowledge of
underlying domain processes in a data-driven architecture in a domain-
independent way. As such, we embed this knowledge represented as a
_programmatic function_ $F$ within a distinct _procedural knowledge module_
$P$ inside the overall architecture. Instead of learning the domain dynamics
itself, we thus provide the means for the model to learn the interface between
$F$ and its data-driven components. This is possible since we can directly
execute the program code that is $F$—as opposed to for instance natural
language instructions that would need some kind of encoding first—and make it
part of the computational graph.

We opt for an auto-regressive frame prediction scheme in which the model is
exposed to the initial $n$ video frames in order to learn the data sample
dynamics, before it auto-regressively predicts the next $m$ frames on its own.
Learning is guided by the reconstruction loss

$\mathcal{L}_{\mathrm{rec}}=\frac{1}{N}\sum_{i=0}^{N}(\hat{V}_{i}-V_{i})^{2}$
(1)

of all $N=m+n$ predicted RGB frames $\hat{V}$ and the ground-truth $V$.

Our architecture is thus composed of three main components: 1) An initial
video frame encoder, which embeds the $n$ observed frames into a suitable
latent representation, 2) our _procedural knowledge module_ $P$ that
transforms the frame’s latent representation to the next time step, and 3) a
final video frame decoder that transforms the latent representations back into
an image representation, as depicted in Figure 1.

Figure 1: Overview of our auto-regressive video prediction process. The first
$n$ frames are used as reference and are encoded by the model in order to
obtain an initial latent representation of the scene. After this burn-in phase
the model has to rollout future $m$ frames on its own.

The core of our contribution resides in $P$ (see Figure 2): It combines the
integrated _programmatic function_ $F$ with a deep _spatio-temporal prediction
model_ $R$ to obtain the latent frame representation $\hat{z}$ of the next
time step based on the latent representation $z$ of the current step. We
implement this by transforming the latent vector $z$ into a representation
that is separable into three components $z_{a}$, $z_{b}$, and $z_{c}$ via a
latent encoder $P_{\mathrm{in}}$ and a decoder $P_{\mathrm{out}}$. Each part
$a,b,c$ is responsible for encoding different aspects of the frame that are
learned implicitly through our architectural design, which we now describe and
also later verify in our experiments.

Figure 2: Left: Structure of our proposed _procedural knowledge module_ $P$.
Right: Abstract view of the burn-in phase for the object-centric variant of
our architecture.

$z_{a}$ represents features that are relevant for $F$—i.e. which correspond to
its input and output parameters—in latent space. Since $F$ is a programmatic
function, we can directly integrate it into the computational graph as an
individual layer. However, since its input space is usually symbolic and not
in a latent representation, $z_{a}$ is translated into and from its symbolic
pendant $s$ through dedicated mapping layers $F_{\mathrm{in}}$ and
$F_{\mathrm{out}}$, respectively. We further facilitate the transformation
between latent and symbolic representation of the state by introducing a
regularization loss $\mathcal{L}_{s}$ that is based on the symbolic state $s$
and its auto-encoded version for all $N$ predicted frames:

$\mathcal{L}_{s}=\frac{1}{N}\sum_{i=0}^{N}(s_{i}-F_{\mathrm{in}}(F_{\mathrm{out}}(s_{i})))^{2}$
(2)

The final loss is thus
$\mathcal{L}=\mathcal{L}_{\mathrm{rec}}+\lambda\mathcal{L}_{s}$, with
$\lambda$ being a constant weighting coefficient.

The second part $z_{b}$ contains residual scene dynamics that are not covered
by $F$. It corresponds to the output of $R$, which is a typical auto-
regressive frame predictor model [30] with the goal of modelling spatio-
temporal scene properties. It takes into account the whole latent
representation consisting of $z_{a}$, $z_{b}$, and $z_{c}$. Instead of
predicting the full future latent state on its own, we show that when combined
with $F$, this model $R$ focuses on residual scene dynamics only and does not
interfere with dynamics handled by $F$. This property allows us to exert a
certain level of control when predicting frames w.r.t. the inputs and outputs
of $F$, as shown in Section 5.3.

Finally, $z_{c}$ contains static scene features that do not change frame-to-
frame, such as for instance parts of the background or object colors. It is
thus realized as a residual connection directly mapping from $z_{c}$ to
$\hat{z}_{c}$. Auto-regressive approaches often suffer from accumulating
errors, as future predictions build on top of past prediction mistakes. We
were able to alleviate this issue by allowing the model to store scene statics
in $z_{c}$.

Thus, our model has three different pathways available to pass latent features
from one video time step to the next. Each contains meaningful inductive
biases that encourage the model to follow our interpretation of the latent
vectors. $z_{a}$ contains only the encoded state required for $F$ and is
limited by an information bottleneck induced by the symbolic state
transformation for $F$. $z_{c}$ is a direct shortcut from the first encoded
frame, and thus offers an easy to learn path for time-invariant features.
Finally, $z_{b}$ offers the only unrestricted pathway that allows encoding
time-variant features. Keeping representations intact over many frames in
auto-regressive models such as $R$ is a challenging learning objective
however, which 1) makes it less likely that the model will try to encode
static scene features that are better handled by $z_{c}$, and 2) encourages
the model to rely on $z_{a}$ for dynamics that are handled by $F$.

Object-centric representations have shown to further improve video prediction
performance over methods that work with a single latent state for the whole
scene [30]. Since some of our datasets are object based, we also introduce a
variant of our proposed architecture that operates on slot-based object
representations. For our module $P$ the main difference is that the latent
state of a single frame is now separated into multiple object representations.
In practice, this amounts to an additional dimension in the tensors which can
be handled in parallel by all components of $P$. During the burn-in phase, our
module $P$ takes turns with a slot attention [15] module to assemble these
frame object representations. For each frame, the latter refines slot-based
object representations w.r.t. the encoded video frame of the current time step
and the object representation prediction of our module $P$ from the last time
step (cf. Figure 2). Afterwards, $P$ uses this refined representation to
predict the next time step.

With our architecture we aim to concentrate the domain inductive biases within
$P$, and keep the image encoder and decoder parts domain agnostic and in line
with current state-of-the-art models in video prediction. As opposed to models
that use domain-specific decoders such as neural renderers [17], we thus do
not need to obtain a full symbolic representation of the scene and can keep
parts not relevant for $F$ in the latent space of the model. At the same time,
the architecture’s modularity facilitates the integration of and interfacing
with future developments in video prediction, such as better image encoder or
decoders but also better predictors in place of $R$.

##  4 Data

We designed and introduce three new datasets which consist of videos of
rendered 3D scenes using Kubric [5]. They feature complex, nonlinear physical
dynamics and aim to provide suitable testing grounds for our approach. Other
video dynamics datasets in the literature often either happened in 2D screen
space only [7, 4], or involved only short-term dynamics for which linear
approximations of the underlying dynamics were sufficient [30, 11]. As we also
show in our experiments, such models fail once the underlying dynamics
increase in complexity as in our datasets. Thus, our settings are an ideal
candidate for which the introduction of domain knowledge can supplement—or
even enable—purely data-driven approaches. We integrate the dynamics equations
of each dataset as $F$ within the _procedural knowledge module_ $P$ of our
model.

Our main and most challenging dataset “Orbits” is based on the three body
problem [18], in which multiple objects attract each other, resulting in
chaotic movements once three or more objects are involved. A similar dataset
was introduced in previous related work [21], albeit with no background and
only sphere object shapes.

Our second dataset is a rendering of an Acrobot environment, a setting which
is commonly used for benchmarking physics and control models [1]. It features
a double pendulum where one end is fixed in space, resulting in dynamics that
are inherently complex with chaotic movements without further actuation. As we
show in our experiments these environments are easier to predict in pixel-
space for data-driven models compared to our Orbits dataset, however
stabilizing the pole by actuating the joint between the pendulums is a common
control objective that we want to work with as a downstream task.

Our third dataset features a non object-centric variant of the Acrobot
setting. Instead of observing the double pendulum directly we mount the camera
on it, while orienting it towards a static, texturized background, whose
texture is the same for all data samples. We let the double pendulum move
according to the dynamics defined in the Acrobot setting, and the now also
moving camera observes different parts of the background texture in each
frame. The prediction of future frames is thus only successful if the model is
able to correctly estimate and utlize the indirectly observed pendulum
dynamics. The overall dataset setup is otherwise the same as in the Acrobot
setting. This dataset is related to visual Simultaneous Localization And
Mapping (SLAM), in which models establish a map of the environment based on
video input, since an internal representation of the whole—always only
partially observed—scene is necessary. By applying our approach to this
dataset we also show its potential for visual robotics navigation tasks as
future work, for which we could exchange the pendulum dynamics for a robot
dynamics model in $F$.

We use our object-centric architecture variant for the Orbits and Acrobot
dataset, and the non object-centric variant for the Pendulum Camera dataset.
We describe further details of these datasets in Appendix 0.C and their
underlying dynamics equations in Appendix 0.F.

##  5 Experiments

In the following we first describe our experimental setup and continue by
analyzing our proposed approach w.r.t. its performance in contrast to existing
methods. Afterwards we study the feasibility of using our model for downstream
control tasks.

###  5.1 Setup

All models observe the initial six video frames and—where applicable—the
symbolic input for $F$ for the very first frame. We evaluate the performance
based on the prediction performance of the next 24 frames, however during
training only the next twelve frames contribute to the loss in order to
observe generalization performance for more rollouts.

We compare the performance with two groups of relevant state-of-the-art work
in video prediction: 1) Purely data-driven approaches that do not rely on
physical inductive biases in the architecture such as Slot Diffusion [31],
SlotFormer [30] and PredRNN-V2 [26] and 2) approaches that include general
physical inductive biases such as PhyDNet [13], Dona et al. [4], and Takenaka
et al. [21]. We describe the details of the configurations of these models in
Appendix 0.B.

We measure the averaged reconstruction performance for three different random
seeds with the Learned Perceptual Image Patch Similarity (LPIPS), which has
shown better alignment with human perception than other metrics by relying on
a pretrained image encoder. However, for completeness we also report the
standard metrics Structural Similarity (SSIM) and Peak Signal-to-Noise Ratio
(PSNR).

####  5.1.1 Implementation Details

Our used video frame encoder is a standard CNN with a subsequent position
embedding. The modules $P_{\mathrm{in}}$ and $P_{\mathrm{out}}$ within $P$ are
implemented as fully-connected networks including a single hidden layer with
the ReLU activation function, whose non-linearity enables the model to learn
the separable latent space that we require without relying on the learning
capacity of the video frame encoder. We linearly transform $z_{a}$ from the
symbolic to the latent space and back by implementing $F_{\mathrm{in}}$ and
$F_{\mathrm{out}}$ as fully-connected layers without bias neurons. This
enables easy auto-encoding between both spaces while within $P$, and forces
the model to learn the complex transformation between the symbolic state for
$F$ and its latent version that can be used by the video frame decoder in
$P_{\mathrm{in}}$ and $P_{\mathrm{out}}$. The _spatio-temporal prediction
model_ $R$ within $P$ is implemented as a transformer [23] with temporal
position embedding. Finally, we use a Spatial Broadcast Decoder [27] as our
video frame decoder.

For the object-centric variant of our model we use Slot Attention [15] to
obtain object representations in the latent space. For a scene with $M$
objects and $N$ rollout frames, we thus obtain $N\times M$ latent
representations, which are decoded individually into the image space by the
video frame decoder. The final frame prediction is assembled by combining the
$M$ object representations as in Slot Attention for Video (SAVi) [11], which
predicts an auxiliary masking channel in addition to the RGB channels to weigh
the individual object frame decodings.

More implementation details can be found in Appendix 0.A, and details about
the integrated functions in Appendix 0.F.

###  5.2 Video Prediction

Here we establish and discuss the video prediction performance of our proposed
architecture and compare it with related work. More qualitative results for
all datasets can be found in Appendix 0.D.

####  5.2.1 Orbits

Table 1: Performance comparison of our proposed complete architecture (Base) with ablations and relevant related work for the Orbits dataset. |  |  LPIPS$\downarrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$
---|---|---|---|---
Ours |  |  $\mathbf{4.0}$$\pm 1$ |  $\mathbf{97.2}$$\pm 0$ |  $\mathbf{34.7}$$\pm 0$
|  Limited Data |  $19.5$$\pm 2$ |  $87.1$$\pm 0$ |  $27.7$$\pm 0$
|  Learned Parameters in $F$ |  $\mathbf{3.9}$$\pm 1$ |  $\mathbf{97.1}$$\pm 0$ |  $\mathbf{34.7}$$\pm 1$
Architecture Ablations |  |  |
|  1) No $z_{c}$ |  $5.2$$\pm 0$ |  $96.4$$\pm 0$ |  $33.8$$\pm 0$
|  2) No $z_{b},R$ |  $5.7$$\pm 0$ |  $95.8$$\pm 0$ |  $33.0$$\pm 0$
|  3) $F$ as Identity |  $35.5$$\pm 5$ |  $77.9$$\pm 1$ |  $24.2$$\pm 1$
|  4) Only $R$ instead of $P$ |  $41.8$$\pm 0$ |  $76.8$$\pm 0$ |  $23.5$$\pm 0$
Related Work |  |  |
|  Slot Diffusion |  $26.7$$\pm 1$ |  $68.7$$\pm 1$ |  $21.9$$\pm 0$
|  Takenaka et al. |  $36.8$$\pm 0$ |  $78.1$$\pm 0$ |  $24.8$$\pm 0$
|  Slotformer |  $34.2$$\pm 0$ |  $75.3$$\pm 0$ |  $23.2$$\pm 0$
|  PhyDNet |  $35.7$$\pm 1$ |  $77.6$$\pm 0$ |  $24.0$$\pm 0$
|  PredRNN-V2 |  $34.4$$\pm 0$ |  $78.5$$\pm 0$ |  $24.3$$\pm 0$
|  Dona et al. |  $41.1$$\pm 0$ |  $76.7$$\pm 0$ |  $23.5$$\pm 0$
Figure 3: Qualitative performance of different model configurations compared
to the ground-truth (GT). Predictions of selected frame iterations are shown
from left to right. Our model is able to position objects correctly in future
frames, while keeping object shading and overall appearance intact.

For the Orbits dataset we are able to significantly outperform other
approaches (cf. Tables 1 and 3). Our model is able to follow the correct
object trajectories and render the objects accordingly. When comparing the
symbolic state $s$ used by $F$ with its auto-encoded version
$F_{\mathrm{in}}(F_{\mathrm{out}}(s))$ we observe a low Mean Absolute Error
(MAE) of $0.005$. This indicates that frame-to-frame object states can be
accurately recovered from the latent representation without too much error
accumulation.

In order to verify the impact of the integrated procedural knowledge, we train
two variants of our model which replace the integrated function $F$ with an
identity function, and use only the residual model $R$ in place of our
procedural knowledge module $P$ (ablations 3) and 4) in Table 1,
respectively). In both cases the performance decreases substantially and
converges to the same performance as the related work, giving clear evidence
that $F$ has a large positive impact on the performance.

Since we expect that the integration of procedural knowledge should decrease
the complexity of the learning problem, and as such require less data than a
completely uninformed model, we also train our model with a very small dataset
consisting of only 300 training samples instead of 10K to test this
hypothesis. As can be seen in Table 1 the performance significantly decreases,
but stays well above variants that do not include $F$. When looking at the
predictions in Figure 3, the reason for this is a quality loss in object
appearance prediction, however the objects are still rendered at the correct
positions. As we will further show in Section 5.3, the latent vectors $z_{b}$
and $z_{c}$ contain such appearance features, both of which are purely based
on the data-driven aspects of our model. These results indicate that while our
data-driven parts suffer equally to regular data-driven models when trained
with small amounts of data, the inclusion of a procedural knowledge path
$z_{a}$ can enable the model to still provide meaningful predictions. Through
this experiment we thus highlight that our model not only benefits from the
integrated knowledge, but also still benefits from additional data, which
conforms to our goal of providing domain experts with additional avenues to
improve their models without removing options.

We also test our approach in the face of uncertainties in the integrated
function. For this we let the model learn the environment constants object
mass and gravitational constant present within $F$ in an unsupervised manner.
We used a modified learning rate of $1e^{-2}$ instead of the regular $2e^{-4}$
for these parameters to improve and speed up convergence due to the larger
magnitudes compared to regular network weights. The performance is comparable
to our model without any learned parameters (see “Learned Parameters in $F$”
in Table 1). In fact, the model converges on the ground-truth values of the
learned parameters without ever supplying these as a supervision signal,
highlighting the potential of our approach for system and parameter
identification tasks as well.

Finally, we analyze the impact of individual components of our approach. We
train variants of our architecture that do not use the residual path $z_{c}$
and do not use a residual model $R$ and thus $z_{b}$ (see ablations 1) and 2)
in Table 1). Removing either only has a slight negative influence on
performance, which aligns with the finding earlier that using $F$ has the
largest performance impact in the model. It shows however that predictions can
further be improved by using a suitable model $R$, especially when there are
environment processes not covered by $F$, such as dynamic lighting conditions.

####  5.2.2 Acrobot & Pendulum Camera

Table 2: Performance comparison for the Acrobot and the Pendulum Camera dataset. Purely object-centric models such as SlotFormer cannot be applied to the Pendulum Camera dataset as no objects are visible in the scene. | Acrobot | Pendulum Camera
---|---|---
| LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$
Ours |  $\mathbf{3.1}$$\pm 0$ |  $\mathbf{97.9}$$\pm 0$ |  $\mathbf{37.2}$$\pm 2$ |  $\mathbf{26.9}$$\pm 1$ |  $\mathbf{65.4}$$\pm 1$ |  $\mathbf{31.7}$$\pm 0$
Slot Diffusion |  $18.2$$\pm 0$ |  $84.0$$\pm 1$ |  $26.6$$\pm 1$ | N/A | N/A | N/A
Takenaka et al. |  $4.4$$\pm 0$ |  $96.9$$\pm 0$ |  $34.9$$\pm 0$ | N/A | N/A | N/A
Slotformer |  $13.2$$\pm 0$ |  $88.0$$\pm 0$ |  $28.3$$\pm 0$ | N/A | N/A | N/A
PhyDNet |  $14.2$$\pm 2$ |  $90.2$$\pm 1$ |  $29.8$$\pm 0$ |  $50.9$$\pm 0$ |  $39.7$$\pm 1$ |  $21.7$$\pm 1$
Dona et al. |  $31.4$$\pm 9$ |  $85.7$$\pm 4$ |  $23.7$$\pm 8$ |  $50.8$$\pm 0$ |  $41.6$$\pm 2$ |  $21.7$$\pm 0$
PredRNN-V2 |  $17.5$$\pm 2$ |  $88.3$$\pm 0$ |  $28.9$$\pm 0$ |  $50.4$$\pm 0$ |  $39.8$$\pm 0$ |  $21.8$$\pm 0$
Figure 4: Frame predictions of different time steps (left-to-right) for the
Pendulum Camera dataset, with the ground-truth being in the top row (GT).

The Acrobot dataset features a smaller performance gap between our approach
and purely data-driven methods such as SlotFormer (cf. Tables 2 and 4),
indicating that this scenario is easier when relying on data alone. This is
probably due to the restricted movements in the scene, whereas in the Orbits
setting objects are able to move freely in space, making predictions more
difficult. Still, the integration of procedural knowledge was beneficial and
lead to improved performance across the board even when compared to Takenaka
et al. with a similarly informed model. The challenging nature of modelling
indirectly observed dynamics in the Pendulum Camera dataset is reflected in
both the quantitative and qualitative performance observed in Tables 2 and 4.
While our model is able to reconstruct the overall patterns present in each
frame, all comparison models were unable to model the pendulum trajectory
correctly and instead produced uniform blurry predictions.

###  5.3 Control Interface

The model appears to have used the integrated function $F$ correctly in order
to obtain better frame predictions. Since $F$ is based on symbolic inputs and
outputs, the question arises whether we can use these to control the
predictions in an interpretable manner. To evaluate this, we consider two
scenarios: (1) We modify $z_{b}$ and $z_{c}$—i.e. the appearance features—and
observe whether the dynamics do not change; (2) we directly adjust $z_{a}$ and
observe whether the rendered outputs correlate with our modifications of the
symbolic state.

Figure 5: Frame predictions of our model for different time steps (left-to-
right) in the case of no changes to the latent vector (Normal), swapping
$z_{c}$ between the object latent vectors (Swap $\mathbf{z_{c}}$) before
decoding, and swapping both latent vectors $z_{b}$ and $z_{c}$ with the same
permutation (Swap $\mathbf{z_{b},z_{c}}$). Object appearances are swapped, but
the dynamics stay unchanged. Figure 6: Frame predictions of our model for the
Acrobot swing-up control task. We overlay a direct plot of the predicted
physics states on the rendered frames. Our model renders the states with only
slight divergences from the direct plot.

For (1), we leverage the object-centric representations in the Orbits scenario
and simply swap $z_{b}$ and $z_{c}$ between different objects. Figure 5 shows
the qualitative results. First, as we expected in Section 3 we can observe
that $z_{c}$ contains static object properties, such as their size, color, and
overall shape. At the same time, when modifying $z_{b}$ we can see changes in
time variant features in the image such as object shading, occlusions, and
shape changes due to the cameras perspective projection. Finally, all changes
to $z_{b}$ and $z_{c}$ did not change the scene dynamics, as the objects
simply continue their trajectory, albeit with new appearances.

For (2), we interface our model with MPC and by doing this show that our model
not only renders objects at intended locations, but also inherently enables
the integration of downstream tasks that operate in symbolic space. We use an
off-the-shelf MPC controller that interacts with our model through $F$. We
build on top of the Acrobot scenario and define a control task to swing up the
double pendulum by allowing a torque to be applied to the joint between the
poles. We encode the initial six frames with our model as usual and use the
sixth predicted physical state as initialization for the MPC task. We store
the predicted torque action sequence for the next 75 frames and apply the
corresponding torque for each frame prediction in $F$ before decoding. We show
the qualitative results in Figure 6, and more in Appendix 0.G.

We can observe that the model is able to correctly decode predicted object
states into a rendered scene. We conclude that the downstream integration of
symbolic methods into our method is possible and a large benefit over purely
black-box predictors that have no way of interacting in such a manner with
existing symbolic tools. We further verified the controllability for the
Orbits setting in Appendix 0.E and show that the model can adapt to new
dynamics simply by exchanging $F$.

##  6 Limitations and Future Work

Our work puts a spotlight on the benefit of procedural knowledge represented
as programmatic functions and aims to answer some fundamental questions on how
it could be utilized in data-driven video predictors. As such, there are still
many follow-up questions and directions that can be the topic of future
research.

One assumption that we made was to let the model observe the correct function
input of the very first frame in order to give the model a hint of the
magnitude and distribution of possible symbolic states. The prediction of
initial values—especially in physics—is a known problem with a large body of
research behind it. As our focus was on establishing our procedural knowledge
module $P$ in the architecture in a domain-independent way, we excluded this
aspect as it is part of the encoder module of the underlying model, which
should be domain-specific. Still, the integration of a suitable encoder for
predicting initial state values is interesting for future work.

It would also be interesting to combine the given integrated knowledge with
synthesis approaches which might dynamically extend or repair the integrated
function. Neural program synthesis is an active field of research which might
give many insights in this regard. It could also be used to move towards
having a dynamically extending library of domain functions available instead
of only a single procedural knowledge function, greatly increasing the
utility.

##  7 Conclusion

We have introduced a novel architectural scheme to join procedural knowledge
in the form of programmatic functions with data-driven models. By applying it
to video prediction, we show that our approach can enable models to handle
tasks for which data-driven models alone would struggle. While our approach
also works with very limited data, we highlight that it still benefits from
more data, leaving it open for domain experts whether they want to refine
their integrated knowledge, or to collect more data, essentially broadening
the means by which performance improvements can be made. Furthermore, our
grey-box modelling approach increases the transparency of the overall model
and allows direct control of the model predictions through the learned
interface to the integrated procedural knowledge, enabling easy interfacing
with downstream tasks such as MPC.

## References

  * [1] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (Jun 2016). https://doi.org/10.48550/arXiv.1606.01540
  * [2] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] (Jul 2020)
  * [3] Denton, E., Fergus, R.: Stochastic Video Generation with a Learned Prior. In: Proceedings of the 35th International Conference on Machine Learning. pp. 1174–1183. PMLR (Jul 2018)
  * [4] Donà, J., Franceschi, J.Y., Lamprier, S., Gallinari, P.: PDE-Driven Spatiotemporal Disentanglement. In: International Conference on Learning Representations (Jan 2021)
  * [5] Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C., Kipf, T., Kundu, A., Lagun, D., Laradji, I., Liu, H.T., Meyer, H., Miao, Y., Nowrouzezahrai, D., Oztireli, C., Pot, E., Radwan, N., Rebain, D., Sabour, S., Sajjadi, M.S.M., Sela, M., Sitzmann, V., Stone, A., Sun, D., Vora, S., Wang, Z., Wu, T., Yi, K.M., Zhong, F., Tagliasacchi, A.: Kubric: A scalable dataset generator. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3739–3751 (Jun 2022). https://doi.org/10.1109/CVPR52688.2022.00373
  * [6] Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k.: Spatial transformer networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)
  * [7] Jaques, M., Burke, M., Hospedales, T.: Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video. In: International Conference on Learning Representations (Sep 2019)
  * [8] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S.A.A., Ballard, A.J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A.W., Kavukcuoglu, K., Kohli, P., Hassabis, D.: Highly accurate protein structure prediction with AlphaFold. Nature 596(7873), 583–589 (Aug 2021). https://doi.org/10.1038/s41586-021-03819-2
  * [9] Kandukuri, R.K., Achterhold, J., Moeller, M., Stueckler, J.: Physical Representation Learning and Parameter Identification from Video Using Differentiable Physics. International Journal of Computer Vision 130(1), 3–16 (Jan 2022). https://doi.org/10.1007/s11263-021-01493-5
  * [10] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [11] Kipf, T., Elsayed, G.F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., Greff, K.: Conditional Object-Centric Learning from Video. In: International Conference on Learning Representations (Jan 2022)
  * [12] Kosiorek, A., Kim, H., Teh, Y.W., Posner, I.: Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. In: Advances in Neural Information Processing Systems. vol. 31. Curran Associates, Inc. (2018)
  * [13] Le Guen, V., Thome, N.: Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11471–11481. IEEE, Seattle, WA, USA (Jun 2020). https://doi.org/10.1109/CVPR42600.2020.01149
  * [14] Lin, Z., Wu, Y.F., Peri, S., Fu, B., Jiang, J., Ahn, S.: Improving Generative Imagination in Object-Centric World Models. In: Proceedings of the 37th International Conference on Machine Learning. pp. 6140–6149. PMLR (Nov 2020)
  * [15] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., Kipf, T.: Object-Centric Learning with Slot Attention. In: Advances in Neural Information Processing Systems. vol. 33, pp. 11525–11538. Curran Associates, Inc. (2020)
  * [16] Marcus, G., Davis, E.: Rebooting AI: Building Artificial Intelligence We Can Trust. Pantheon Books, USA (2019)
  * [17] Murthy, J.K., Macklin, M., Golemo, F., Voleti, V., Petrini, L., Weiss, M., Considine, B., Parent-Lévesque, J., Xie, K., Erleben, K., Paull, L., Shkurti, F., Nowrouzezahrai, D., Fidler, S.: gradSim: Differentiable simulation for system identification and visuomotor control. In: International Conference on Learning Representations (Oct 2020)
  * [18] Musielak, Z.E., Quarles, B.: The three-body problem. Reports on Progress in Physics 77(6), 065901 (Jun 2014). https://doi.org/10.1088/0034-4885/77/6/065901
  * [19] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (Dec 2022)
  * [20] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686–707 (Feb 2019). https://doi.org/10.1016/j.jcp.2018.10.045
  * [21] Takenaka, P., Maucher, J., Huber, M.F.: Guiding Video Prediction with Explicit Procedural Knowledge. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. pp. 1084–1092 (Oct 2023)
  * [22] Traub, M., Otte, S., Menge, T., Karlbauer, M., Thuemmel, J., Butz, M.V.: Learning What and Where: Disentangling Location and Identity Tracking Without Supervision. In: The Eleventh International Conference on Learning Representations (Feb 2023)
  * [23] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need. CoRR abs/1706.03762 (Dec 2017)
  * [24] von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Pfrommer, J., Pick, A., Ramamurthy, R., Walczak, M., Garcke, J., Bauckhage, C., Schuecker, J.: Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems. IEEE Transactions on Knowledge and Data Engineering 35(1), 614–633 (Jan 2023). https://doi.org/10.1109/TKDE.2021.3079836
  * [25] Wang, Y., Long, M., Wang, J., Gao, Z., Yu, P.S.: PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 879–888. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (Dec 2017)
  * [26] Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P.S., Long, M.: PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(2), 2208–2225 (Feb 2023). https://doi.org/10.1109/TPAMI.2022.3165153
  * [27] Watters, N., Matthey, L., Burgess, C.P., Lerchner, A.: Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (Aug 2019). https://doi.org/10.48550/arXiv.1901.07017
  * [28] Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., Tacchetti, A.: Visual Interaction Networks: Learning a Physics Simulator from Video. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
  * [29] Wu, X., Lu, J., Yan, Z., Zhang, G.: Disentangling Stochastic PDE Dynamics for Unsupervised Video Prediction. IEEE Transactions on Neural Networks and Learning Systems pp. 1–15 (2023). https://doi.org/10.1109/TNNLS.2023.3286890
  * [30] Wu, Z., Dvornik, N., Greff, K., Kipf, T., Garg, A.: SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models. In: The Eleventh International Conference on Learning Representations (Feb 2023)
  * [31] Wu, Z., Hu, J., Lu, W., Gilitschenski, I., Garg, A.: SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models. https://openreview.net/forum?id=ETk6cfS3vk
  * [32] Xu, J., Zhang, Z., Friedman, T., Liang, Y., Broeck, G.: A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In: Proceedings of the 35th International Conference on Machine Learning. pp. 5502–5511. PMLR (Jul 2018)
  * [33] Yang, T.Y., Rosca, J.P., Narasimhan, K.R., Ramadge, P.: Learning Physics Constrained Dynamics Using Autoencoders. In: Advances in Neural Information Processing Systems (Oct 2022)
  * [34] Yi, K., Gan*, C., Li, Y., Kohli, P., Wu, J., Torralba, A., Tenenbaum, J.B.: CLEVRER: Collision Events for Video Representation and Reasoning. In: International Conference on Learning Representations (Sep 2019)

##  Appendix 0.A Further Implementation Details

In the following we describe the core components of our architecture in more
detail.

###  0.A.1 Video Frame Encoder

The used video frame encoder is a standard CNN. The input video frames are
encoded in parallel by merging the temporal dimension $T$ with the batch
dimension $B$. The CNN consists of four convolutional layers, each with a
filter size of 64, kernel size of 5, and a stride of 1. In the non object-
centric variant of our architecture, the output features are flattened and
transformed by a final fully connected network, consisting of initial layer
normalization, a single hidden layer with ReLU activation and a final output
linear layer with $C=768$ neurons each. The result is a latent vector of size
$B\times T\times C$ that serves as input to $P$.

In the object-centric variant, a position embedding is additionally applied
after the CNN, and only the spatial dimensions $H$ and $W$ are flattened
before the transformation of the fully connected network, with $C$ reduced to
$128$. The result is a latent vector of size $B\times T\times C\times H\times
W$. In each burn-in iteration of the object-centric variant, we use the Slot
Attention mechanism [15] to obtain updated object latent vectors before
applying $P$.

###  0.A.2 Procedural Knowledge Module

$P$ is responsible for predicting the latent vector of the next frame. It
consists of the following submodules:

$P_{\mathrm{in}}$. Responsible for transforming the latent vector obtained
from the image frame encoder into a separable latent vector $z$. It is
implemented as a fully connected network with a single hidden layer using the
ReLU activation function. All layers have a subsequent ReLU activation
function. The number of neurons in all layers corresponds to $C$.

$P_{\mathrm{out}}$. Responsible for transforming $z$ back into the latent
image space. It has the same structure as $P_{\mathrm{in}}$.

$F_{\mathrm{in}}$. Responsible for transforming $z_{a}$ within $z$ into the
symbolic space required for $F$. It is a single linear layer without bias
neurons. In the object-centric case, its output size directly corresponds to
the number of parameters required for $F$ $N_{\mathrm{param}}$ for a single
object. In the non object-centric case when there is no separate object
dimension available, it instead corresponds to $N_{\mathrm{param}}\times
N_{\mathrm{objects}}$, where $N_{\mathrm{objects}}$ corresponds to the (fixed)
number of objects (if present in the dataset).

$F$. Contains the integrated function directly as part of the computational
graph. Details about $F$ for the individual data scenarios can be found in
Appendix 0.F.

$F_{\mathrm{out}}$. Same structure as $F_{\mathrm{in}}$, with the input and
output sizes reversed.

$R$. Responsible for modelling residual dynamics not handled by $F$. We
implement it as a transformer [23] with two layers and four heads. We set the
latent size to $C$ and the dimension of its feed-forward network to $512$. It
takes into account the most recent 6 frame encodings. Its output corresponds
to $z_{b}$. A temporal position embedding is applied before the transformer.

We first transform the latent image vector into a separable latent vector $z$
by transforming it with $P_{\mathrm{in}}$. We then split $z$ of size $C$ into
the three equally sized components $z_{a}$, $z_{b}$, and $z_{c}$. We continue
by obtaining their respective next frame predictions $\hat{z}_{a}$,
$\hat{z}_{b}$, and $\hat{z}_{c}$ as follows: $\hat{z}_{a}$ by $F$, $z_{b}$ by
transforming $z$ with $R$, and $\hat{z}_{c}$ directly corresponds to $z_{c}$.
All three components are merged back together and transformed into the image
latent space with $P_{\mathrm{out}}$ before decoding.

###  0.A.3 Video Frame Decoder

We implement the video frame decoder as a Spatial Broadcast Decoder [27]. We
set the resolution for the spatial broadcast to 8, and first apply positional
embedding on the expanded latent vector. We then transform the output by four
deconvolutional layers, each with filter size 64. We add a final convolutional
layer with filter size of 3 to obtain the decoded image. We set the strides to
2 in each layer until we arrive at the desired output resolution of 64, after
which we use a stride of 1. In the object-centric variant, we set the output
filter size to 4 and use the first channel as weights $w$. We then reduce the
object dimension after the decoding as in [15] by normalizing the object
dimension of $w$ via softmax, and using it to calculate a weighted sum with
the object dimensions of the RGB output channels.

###  0.A.4 Training Details

We train all models for at maximum 500k iterations each or until convergence
is observed by early stopping. We clip gradients to a maximum norm of $0.05$
and train using the Adam Optimizer [10] with an initial learning rate of
$2e^{-4}$. We set the loss weighting factor $\lambda$ to 1. We set the batch
size according to the available GPU memory, which was 32 in our case. We
performed the experiments on four NVIDIA TITAN Xp with 12GB of VRAM, taking—on
average—one to two days per run.

##  Appendix 0.B Details for Comparison Models

Takenaka et al. [21]. We apply the training process and configuration as
described in their paper, and instead use RGB reconstruction loss to fit into
our training framework. We integrate the same procedural function here as in
our model.

Slot Diffusion [31]. We use the three-stage training process as described in
the paper with all hyperparameters being set as recommended.

SlotFormer [30]. We use their proposed training and architecture configuration
for the CLEVRER [34] dataset, as its makeup is the most similar to our
datasets and follow their proposed training regimen.

PhyDNet. We use their recommended training and architecture configuration
without changes.

PredRNN-V2. We use their recommended configuration for the Moving MNIST
dataset.

Dona et al. [4]. We report the performance for their recommended configuration
for the Sea Surface Temperature (SST) dataset, as it resulted in the best
performance on our datasets.

##  Appendix 0.C Further Dataset Details

In Table 3 we show further statistics of our introduced datasets.

Table 3: Detailed statistics of our introduced datasets. |  |  Orbits |  Acrobot |  Pendulum Camera
---|---|---|---|---
Number of |  |  |
|  training samples |  10K |  2K |  2K
|  evaluation samples |  256 |  256 |  256
|  burn-in frames |  6 |  6 |  6
|  training rollout frames |  12 |  12 |  12
|  validation rollout frames |  24 |  24 |  24
Spatial size |  $64\times 64$ |  $64\times 64$ |  $64\times 64$
Video FPS |  4 |  10 |  10
Physics FPS |  40 |  40 |  40
Symbolic State |  Position and velocity of each object |  Pole angles and their angular velocities |  As in Acrobot, and also the camera position

##  Appendix 0.D Additional Prediction Visualisations

###  0.D.1 Orbits

This section shows additional qualitative results of our model for the Orbits
dataset.

Figure 7: Qualitative results for the Orbits dataset.

Figure 8: Qualitative results for the Orbits dataset.

####  0.D.1.1 Latent Vector Swaps

This section shows additional qualitative results after swapping parts of the
latent vector between object representations for the Orbits dataset.

Figure 9: Latent vector swap qualitative result.

###  0.D.2 Acrobot

This section shows additional qualitative results of our model for the Acrobot
dataset.

Figure 10: Qualitative results for the Acrobot dataset.

Figure 11: Qualitative results for the Acrobot dataset.

###  0.D.3 Pendulum Camera

This section shows additional qualitative results of our model for the
Pendulum Camera dataset.

Figure 12: Pendulum Camera dataset qualitative result.

##  Appendix 0.E Orbits Control Validation Dataset Details

In the Orbits setting the object positions are part of the symbolic state,
which are an integral factor of correctly rendering the output frame. However,
it is not trivial to measure how well our model is able to decode “hand-
controlled” 3D object positions into a 2D frame in a generalizable manner.
Therefore we chose to setup an empirical evaluation framework by assembling
variations of the Orbits dataset, ranging from different simulation
parameters, over completely novel dynamics, up to non-physics settings such as
trajectory following. For each validation set, we replace $F$ of a model
trained on the default Orbits dataset with the respective version that handles
these new dynamics, and then validate the model without any retraining.

Table 4: LPIPS$\downarrow$ Performance on the default Orbits dataset and the validation settings A-H. A: Increased frame rate; B: Increased gravitational constant; C: Tripled force; D: Repulsion instead of attraction; E: No forces; F: No forces and zero velocities; G: Objects follow set trajectories; H: Objects appear at random locations in each frame. Default | A | B | C | D | E | F | G | H
---|---|---|---|---|---|---|---|---
5.6 | 5.9 | 5.8 | 5.8 | 5.1 | 2.4 | 5.4 | 4.5 | 4.7

As can be seen in Table 4, the performance across all validation settings is
comparable to the default dataset and thus, shows that the outputs of $F$ work
as a reliable control interface at test time. We note that the much lower
LPIPS for test setting E is due to the objects quickly leaving the scene,
resulting in mostly background scenes.

###  0.E.1 Dataset Details

This section shows more details about the Orbits dataset variants that are
used to empirically estimate the control performance. Table 5 describes each
setting in more detail, and qualitative results are shown in Section 0.E.2.

Table 5: Detailed description of the Orbits dataset variants that are used to verify generative control over the integrated function parameters. Setting |  |  Description
---|---|---
Original Dynamics |
|  A |  Increased frame rate from 4 to 10 frames per second
|  B |  Increased gravitational constant for the physical simulation from 7.0 to 20.0
|  C |  Tripled the force applied to objects
Novel Dynamics |
|  D |  Objects are repulsed instead of attracted to each other by inverting the force. However, they are still attracted to the camera focal point in order to stay in view longer
|  E |  No force is applied to the objects, however initial velocities are kept
|  F |  No force is applied to the objects. In addition, all objects have zero velocity, i.e. no object moves
Non-Physical Dynamics |
|  G |  Objects move along a predefined 6 point trajectory
|  H |  Objects are at a randomly sampled location in each frame

###  0.E.2 Qualitative Results

This section shows qualitative results for the Orbits validation dataset
variants.

Figure 13: Qualitative performance of our model for the dataset variant A:
Increased frame rate

Figure 14: Qualitative performance of our model for the dataset variant B:
Increased gravitational constant

Figure 15: Qualitative performance of our model for the dataset variant C:
Tripled force

Figure 16: Qualitative performance of our model for the dataset variant D:
Objects are repulsed

Figure 17: Qualitative performance of our model for the dataset variant E: No
forces

Figure 18: Qualitative performance of our model for the dataset variant F: No
forces, no velocities

Figure 19: Qualitative performance of our model for the dataset variant G:
Trajectory following

Figure 20: Qualitative performance of our model for the dataset variant H:
Random location

##  Appendix 0.F Integrated Function Details

This section shows the functions integrated in our model. All functions first
calculate the appropriate acceleration $a$ before applying it in a semi-
implicit euler integration step with a step size of $\Delta t$.

For the Orbits dataset each objects state consists of position $p$ and
velocity $v$. The environmental constants correspond to the gravitational
constant $g$ and object mass $m$. Given $N$ objects in the scene at video
frame $t$, the object state of the next time step $t+1$ for any object $n$ is
obtained as follows:

$\displaystyle F_{t,n}$ $\displaystyle=\sum_{\begin{subarray}{c}i=0\\\ i\neq
n\end{subarray}}^{N}\frac{(p_{t,i}-p_{t,n})}{|(p_{t,i}-p_{t,n})|}\frac{gm}{|(p_{t,i}-p_{t,n})|^{2}}$
(3) $\displaystyle a_{t,n}$ $\displaystyle=\frac{F_{t,n}}{m}$ (4)
$\displaystyle v_{t+1,n}$ $\displaystyle=v_{t,n}+\Delta ta_{t,n}$ (5)
$\displaystyle p_{t+1,n}$ $\displaystyle=p_{t,n}+\Delta tv_{t+1,n}$ (6)

For the Acrobot dataset the per-frame state consists of the pendulum angles
$\theta_{1}$ and $\theta_{2}$ and their angular velocities $\dot{\theta}_{1}$
and $\dot{\theta}_{2}$. The environmental constants consist of the pendulum
masses $m_{1}$ and $m_{2}$, the pendulum lengths $l_{1}$ and $l_{2}$, the link
center of mass $c_{1}$ and $c_{2}$, the inertias $I_{1}$ and $I_{2}$, and the
gravitational constant $G$. The pendulum state of the next time step $t+1$ is
calculated as follows:

$\displaystyle\delta_{1_{t}}$
$\displaystyle=m_{1}c_{1}^{2}+m_{2}(l_{1}^{2}+c_{2}^{2}+2l_{1}c_{2}\cos{(\theta_{2_{t}})})+I_{1}+I_{2}$
(7) $\displaystyle\delta_{2_{t}}$
$\displaystyle=m_{2}(c_{2}^{2}+l_{1}c_{2}\cos{(\theta_{2_{t}})})+I_{2}$ (8)
$\displaystyle\phi_{2_{t}}$
$\displaystyle=m_{2}c_{2}G\cos{(\theta_{1_{t}}+\theta_{2_{t}}-\frac{\pi}{2})}$
(9) $\displaystyle\phi_{1_{t}}$
$\displaystyle=-m_{2}l_{1}c_{2}\dot{\theta}_{2_{t}}^{2}\sin{(\theta_{2_{t}})}-2m_{2}l_{1}c_{2}\dot{\theta}_{2_{t}}\dot{\theta}_{1_{t}}\sin{(\theta_{2_{t}})}$
(10)
$\displaystyle\;\;\;\;+(m_{1}c_{1}+m_{2}l_{1})G\cos{(\theta_{1_{t}}-\frac{\pi}{2})}+\phi_{2_{t}}$
(11) $\displaystyle\ddot{\theta}_{2_{t}}$
$\displaystyle=\frac{\frac{\delta_{2_{t}}}{\delta_{1_{t}}}\phi_{1,t}-m_{2}l_{1}c_{2}\dot{\theta}_{1,t}^{2}\sin{(\theta_{2,t})}-\phi_{2,t}}{m_{2}c_{2}^{2}+I_{2}-\frac{\delta_{2_{t}}^{2}}{\delta_{1_{t}}}}$
(12) $\displaystyle\ddot{\theta}_{1_{t}}$
$\displaystyle=-\frac{\delta_{2}\ddot{\theta}_{2_{t}}+\phi_{1_{t}}}{\delta_{1_{t}}}$
(13) $\displaystyle\dot{\theta}_{1_{t+1}}$
$\displaystyle=\dot{\theta}_{1_{t}}+\Delta t\ddot{\theta}_{1_{t}}$ (14)
$\displaystyle\dot{\theta}_{2_{t+1}}$
$\displaystyle=\dot{\theta}_{2_{t}}+\Delta t\ddot{\theta}_{2_{t}}$ (15)
$\displaystyle\theta_{1_{t+1}}$ $\displaystyle=\theta_{1_{t}}+\Delta
t\dot{\theta}_{1_{t+1}}$ (16) $\displaystyle\theta_{2_{t+1}}$
$\displaystyle=\theta_{2_{t}}+\Delta t\dot{\theta}_{2_{t+1}}$ (17)

The Pendulum Camera dataset follows the same equations of the Acrobot dataset
to obtain an updated pendulum state. Afterwards, this state is used to obtain
the new camera position $p_{c_{t+1}}$:

$\displaystyle p_{c_{t+1}}=\begin{bmatrix}p_{x}\\\ p_{y}\\\ p_{z}\\\
\end{bmatrix}$
$\displaystyle=\begin{bmatrix}-2l_{1}\sin{(\theta_{1_{t+1}})}-l_{2}\sin{(\theta_{1_{t+1}}+\theta_{2_{t+1}})}\\\
2l_{1}\cos{(\theta_{1_{t+1}})}+l_{2}\cos{(\theta_{1_{t+1}}+\theta_{2_{t+1}})}\\\
10\end{bmatrix}$ (18)

##  Appendix 0.G MPC Details

We set the control objective as the maximization of the potential energy—i.e.
both pendulums oriented upwards—and the minimization of the kinetic
energy—i.e. resting pendulums. The system model corresponds to our integrated
function $F$ and due to already being discretized does not require further
processing. We use a controller with a prediction horizon of 150 steps and
store the predicted torque action sequence for the next 75 frames.

###  0.G.1 Qualitative Results

This section shows additional qualitative results for the MPC task.

Figure 21: Qualitative results for the MPC task.