# Dynamic Visual Reasoning by
Learning Differentiable Physics Models
from Video and Language

Mingyu Ding
MIT CSAIL and HKU
Zhenfang Chen
MIT-IBM Watson AI Lab
Tao Du
MIT CSAIL
Ping Luo
HKU

Joshua B. Tenenbaum
MIT BCS, CBMM, CSAIL
Chuang Gan
MIT-IBM Watson AI Lab

###### Abstract

In this work, we propose a unified framework, called Visual Reasoning with
Differentiable Physics (VRDP) 111Project page: http://vrdp.csail.mit.edu/,
that can jointly learn visual concepts and infer physics models of objects and
their interactions from videos and language. This is achieved by seamlessly
integrating three components: a visual perception module, a concept learner,
and a differentiable physics engine. The visual perception module parses each
video frame into object-centric trajectories and represents them as latent
scene representations. The concept learner grounds visual concepts (_e.g._ ,
color, shape, and material) from these object-centric representations based on
the language, thus providing prior knowledge for the physics engine. The
differentiable physics model, implemented as an impulse-based differentiable
rigid-body simulator, performs differentiable physical simulation based on the
grounded concepts to infer physical properties, such as mass, restitution, and
velocity, by fitting the simulated trajectories into the video observations.
Consequently, these learned concepts and physical models can explain what we
have seen and imagine what is about to happen in future and counterfactual
scenarios. Integrating differentiable physics into the dynamic reasoning
framework offers several appealing benefits. More accurate dynamics prediction
in learned physics models enables state-of-the-art performance on both
synthetic and real-world benchmarks while still maintaining high transparency
and interpretability; most notably, VRDP improves the accuracy of predictive
and counterfactual questions by 4.5% and 11.5% compared to its best
counterpart. VRDP is also highly data-efficient: physical parameters can be
optimized from very few videos, and even a single video can be sufficient.
Finally, with all physical parameters inferred, VRDP can quickly learn new
concepts from few examples.

##  1 Introduction

Dynamic visual reasoning about objects, relations, and physics is essential
for human intelligence. Given a raw video, humans can easily use their common
sense of intuitive physics to explain what has happened, predict what will
happen next, and infer what would happen in counterfactual situations. Such
human-like physical scene understanding capabilities are also of great
importance in practical applications such as industrial robot control [2, 53].

Previous works have made great efforts to build artificial intelligence (AI)
models with such physical reasoning capabilities. One popular strategy is to
develop pure neural-network-based models [65, 20, 39]. These methods typically
leverage end-to-end neural networks [31, 34] with powerful attention modules
such as Transformer [71] to extract attended features from both video frames
and question words, based on which they answer questions directly. Despite
their high question-answering accuracy on CLEVRER [81], a challenging dynamic
visual question-answering benchmark, these black-box models neither learn
concepts nor model objects’ dynamics. Therefore, they lack transparency,
interpretability, and generalizability to new concepts and scenarios. Another
common approach to dynamic visual reasoning is to build graph neural networks
(GNNs) [47] to capture the dynamics of the scenes. These GNN models [54, 81,
16] treat objects in the video as nodes and perform object- and relation-
centric updates to predict objects’ dynamics in future or counterfactual
scenes. Such systems achieve decent performance with good interpretability on
CLEVRER by combining the GNN-based dynamics models with neural-symbolic
execution [60, 82]. However, these dynamic models do not explicitly consider
laws of physics or use concepts encoded in the question-answer pairs
associated with the videos. As a result, they show limitations in
counterfactual situations that require long-term dynamics prediction.

Although (graph-)neural-network-based approaches have achieved competitive
performance on CLEVRER, dynamic visual reasoning is still far from being
solved perfectly. In particular, due to the lack of explicit physics models,
existing models [81, 20, 16] typically struggle to reason about future and
counterfactual events, especially when training data is limited. For this
reason, one appealing alternative is to develop explicit physics-based methods
to model and reason about dynamics, as highlighted in the recent development
of differentiable physics engines [9, 18, 70, 19, 68] and their applications
in robotics [9, 18, 70]. However, these physics engines typically take as
input a full description of the scene (_e.g._ , the number of objects and
their shapes) which usually requires certain human priors, limiting their
availability to applications with well-defined inputs only.

In this work, we take an approach fundamentally different from either network-
based methods or physics-based methods. Noting that deep learning based
methods excel at parsing objects and learning concepts from videos and
language, and physics laws are good at capturing object dynamics, we propose
Visual Reasoning with Differentiable Physics (VRDP), a unified framework that
combines a visual perception module, a concept learner, and a differentiable
physics engine. VRDP jointly learns object trajectories, language concepts,
and objects’ physics models to make accurate dynamic predictions. It starts
with a perception module running an object detector [30] on individual frames
to generate object proposals and connect them into trajectories based on a
motion heuristic. Then, a concept learner learns object- and event-based
concepts, such as ‘shape’, ‘moving’, and ‘collision’ as in DCL [16, 60]. Based
on the obtained object trajectories and attributes, the differentiable physics
engine estimates all dynamic and physical properties (_e.g._ , velocity,
angular velocity, restitution, mass, and the coefficient of resistance) by
comparing the simulated trajectories with the video observations. With these
explicit physical parameters, the physics engine reruns the simulation to
reason about future motion and causal events, which a program executor then
executes to get the answer. The three components of VRDP cooperate seamlessly:
the concept learner grounds physical concepts needed by the physics engine
like ‘shape’ onto the objects detected by the perception module; the
differentiable physics engine estimates all physical parameters and simulates
accurate object trajectories, which in turn help the concept learning process
in the concept learner.

Compared with existing methods, VRDP has several advantages thanks to its
carefully modularized design. First, it achieves the state-of-the-art
performance on both synthetic videos (CLEVRER [81]) and real-world videos
(Real-Billiard [65]) without sacrificing transparency or interpretability,
especially in situations that require long-term dynamics prediction. Second,
it has high data efficiency thanks to the differentiable physics engine and
symbolic representation. Third, it shows strong generalization capabilities
and can capture new concepts with only a few examples.

##  2 Related Work

Visual Reasoning Our model is related to reasoning on vision and natural
language. Existing works can be generally categorized into two streams as end-
to-end approaches [39, 85, 76, 40, 5] and neuro-symbolic approaches [82, 28,
29, 23, 4, 61, 45, 60, 35, 3]. The end-to-end methods [39, 85, 76, 62]
typically tackle the visual question answering (VQA) problem by designing
monolithic multi-modal deep networks [31, 34]. They directly output answers
without explicit and interpretable mechanisms. Beyond question answering,
neuro-symbolic methods [82, 28, 61, 45, 60] propose a set of visual-reasoning
primitives, which manifest as an attention mechanism capable of performing
complex reasoning tasks in an explicitly interpretable manner.

Dynamic visual reasoning in videos has attracted much research attention. Many
video question answering datasets [26, 49, 41, 69, 83] and the methods [84,
51, 37, 77, 80, 21] built on them mainly focus on understanding diverse visual
scenes, such as human actions (MovieQA [69]) or 3D object movements without
physical and language cues (CATER [26]). Differently, CLEVRER [81] targets the
physical and causal relations grounded in dynamic videos of rigid-body
collisions and asks a range of questions that requires the modeling of long-
term dynamic predictions. For this reason, we evaluate our method and compare
it with other state-of-the-arts on CLEVRER.

Both end-to-end [65, 20] and neuro-symbolic methods [81, 16] have been
explored on CLEVRER. However, they either lack transparency or struggle for
long-term dynamic prediction. In this paper, we perform high-performance and
interpretable reasoning by recovering the physics model of objects and their
interactions (_e.g._ , collisions) from visual perception and language
concepts.

Physical Models Physical models are widely used in video prediction [50, 22,
78, 79], neural simulation and rendering [52, 54], and dynamic reasoning [10,
73]. For example, PhysNet and its variants [50, 22, 78, 79] leverage global or
object-centric deep features to predict the physical motion in video frames.
Some other related works [63, 1, 67, 46] extend physical models to predict the
effect of forces and infer the geometric attributes and topological relations
of cuboids.

In this work, we focus on dynamic visual reasoning about object interactions,
dynamics, and physics with question answering, which is central to human
intelligence and a key goal of artificial intelligence. Solving such tasks
requires a good representation and understanding of physics models. A common
choice is to train a deep neural network for physical property estimation
(_e.g._ , location and velocity) based on learned visual and dynamic priors
[15, 10, 73, 72, 42, 54, 55, 33, 32]. However, since these neural networks do
not model physics laws, generalizing them to unseen events or counterfactual
scenarios could result in unexpected results. Our work is different and more
physics-based: Inspired by the recent advances in differentiable physics [9,
18, 70, 19, 68, 36], we implement an impulse-based differentiable rigid-body
simulator and leverage the power of its gradients to infer dynamics
information about the scene.

Physical Scene Understanding Our work is also relevant to studies on physical
scene understanding [7, 75, 68, 66, 8, 24, 25, 50, 38], most of which propose
pure neural-network solutions without explicitly incorporating physics models.
Benchmarks like PHYRE [7] study physical understanding and reasoning based on
pure videos without concept learning and language inference. Based on such
benchmarks, some works [74, 11, 50, 54] learn compositional scene structure
and estimate states through physical motions and visual de-animation.
Recently, two papers propose pure physics-based methods [43, 38] that make
heavy use of differentiable physics simulators, but they typically assume
concepts in the scene are given as input. Our work is unique in that we learn
video and language concepts from raw videos and infer dynamics information
from a differentiable simulator, combining the benefits of both learning and
physics.

Figure 1: VRDP contains three components including a visual perception module,
a concept learner, and a physics model. The perception module first runs an
object detector [30] on individual frames to generate object proposals and
connect them into trajectories based on motion heuristic. Then, the concept
learner learns object- and event-based concepts, such as ‘shape’, ‘moving’,
and ‘collision’, as prior knowledge for the physics model. Based on the
obtained object trajectories and concepts, the differentiable physics engine
estimates all dynamic and physical properties (_e.g._ , velocity $v$, angular
velocity $\alpha$, restitution $r$, mass $m$, and coefficients of resistance
$\lambda$) by comparing the simulated trajectories with the video
observations. With these explicit physical parameters, the physic engine
reruns the simulation to reason about future motion and causal events, which
are then executed by a symbolic executor to get the answer. Stroboscopic
imaging is applied for motion visualization.

##  3 Method

By integrating differentiable physics into the dynamic reasoning framework,
VRDP jointly learns visual perception, language concepts, and physical
properties of objects. The first two provide prior knowledge for optimizing
the third one to reason about the physical world, and the optimized physical
properties in turn help to learn better concepts. In the following, we first
give an overview of our framework and then describe each of its components in
detail.

###  3.1 Framework Overview

An overview of VRDP is illustrated in Fig. 1. It contains three components: a
visual perception module, a concept learner, and a physics model. The input to
the framework is a video and reasoning questions, where the former is
processed by the visual perception module to get object trajectories and
corresponding visual features, and the latter is parsed into executable
symbolic programs with language concepts by the concept learner. Similar to
DCL [60, 16], the concept learner first grounds object properties (_e.g._ ,
color and shape) and event concepts (_e.g._ , collision and moving) by
aligning the visual features and the corresponding concept embeddings in the
(explanatory or descriptive) program that does not require dynamic
predictions, _e.g._ , “what is the shape of …”. With those perceptually
grounded object trajectories and properties, the physical model then performs
differentiable simulation to learn all physical parameters of the scene and
objects by comparing the simulated trajectories with the video observations.
After that, the physics engine simulates unseen trajectories for predictive
and counterfactual scenarios and generates their features, in turn enabling
the concept learner to finetune event concepts from the program that requires
dynamic predictions, _e.g._ , “what will happen …” and “what if …”. Finally, a
symbolic executor executes the parsed programs with the dynamic predictions to
get the answer.

###  3.2 Model Details

Visual Perception Module Given a video with the number of frames $T$, the
visual perception module parses the video frame-by-frame and associate the
parsed objects in each frame into object trajectories
$L=\\{l^{n}\\}^{N}_{n=1}$, where $l^{n}$ denotes the object trajectory of the
$n^{th}$ object and $N$ is the number of the objects in the video.
Specifically, we leverage a pretrained Faster R-CNN [30] as the object
detector to get the Region of Interest (ROI) feature
$f_{t}\in\mathbb{R}^{N\times D}$ and the object location of objects
$b_{t}=[x^{\text{2D}}_{t},y^{\text{2D}}_{t},x^{\text{BEV}}_{t},y^{\text{BEV}}_{t}]\in\mathbb{R}^{N\times
4}$ at frame $t$, where $D$ is the feature dimension,
$(x^{\text{2D}}_{t},y^{\text{2D}}_{t})$ denotes the normalized object bounding
box center in the image coordinate frame, and
$(x^{\text{BEV}}_{t},y^{\text{BEV}}_{t})$ denotes the projected bird’s-eye
view (BEV) location in the BEV coordinate frame using the calibrated camera
matrix. Following works [16, 27], we associate object proposals in adjacent
frames by thresholding their intersection over union (IoU) and obtain the
object trajectory $l^{n}=\\{b^{n}_{t}\\}^{T}_{t=1}$ for the $n^{th}$ object.

The visual perception module then constructs object and interactive
representations for concept learning. The object representation
$F_{\text{obj}}\in\mathbb{R}^{N\times(D+4T)}$ contains both appearance-based
$F_{a}=\mathrm{avg}(\\{f_{t}\\}^{T}_{t=1})$ and trajectory-based feature
$F_{l}=\\{b_{t}\\}^{T}_{t=1}$ for modeling static properties and dynamic
concepts, respectively, where $\mathrm{avg}(\cdot)$ here represents the
average ROI feature over time. The interactive feature
$F_{\text{pair}}\in\mathbb{R}^{T\times N\times N\times 12S}$, where $S$
denotes a fixed temporal window size, is built on every pair of objects. It
contains object trajectories
$\\{b_{t}^{i}\\}_{t_{0}-S/2}^{t_{0}+S/2},\\{b_{t}^{j}\\}_{t_{0}-S/2}^{t_{0}+S/2}$
of the objects $i$ and $j$ and their distance
$\\{\mathrm{abs}(b_{t}^{i}-b_{t}^{j})\\}_{t_{0}-S/2}^{t_{0}+S/2}$ to model the
collision event of the objects at a specific moment $t_{0}$.

Concept Learner The concept learner grounds the physical and event concepts
(_e.g._ , shape and color) as prior knowledge for the physics model from the
video representation and language. It first leverages a question parser to
translate the input questions and choices into executable neuro-symbolic
programs where each language concept in the program is represented by a
concept embedding. Similar to [60, 16], this work adopts a seq2seq model [6]
with an attention mechanism to translate language into a set of symbolic
programs, _e.g._ , retrieving objects with certain colors, getting future or
counterfactual events, finding the causes of an event, thus decomposing
complex questions into step-by-step dynamic visual reasoning processes. The
concept learner assigns each concept in the program (_e.g._ , color, shape,
and collision) a randomly initialized embedding $e\in\mathbb{R}^{C}$ so that
the symbolic program can be formulated as differentiable vector operations.

After that, it projects the visual representation into concept embedding
spaces and performs Nearest Neighbor (NN) search to quantize concepts for the
object attributes and events. We implement the projection through a linear
layer $\mathcal{P}(\cdot)$ and calculate the cosine similarity between two
vectors in the embedding space for NN search. For example, the confidence
score of whether the $n^{th}$ object is a cube can be represented by
$[\mathrm{cos}(\mathcal{P}(F_{a}^{n}),e_{\text{cube}})-\mu]/\sigma$, where
$e_{\text{cube}}$ is the embedding of concept ‘cube’, $\mu$ and $\sigma$ are
the shifting and scaling scalars, and $\mathcal{P}(\cdot)$ maps a
$D$-dimensional visual feature into a $C$-dimensional vector in this case.

Physics Model The differentiable physics model captures objects’ intrinsic
physical properties and makes accurate dynamic predictions for reasoning. With
the perceptually grounded object shapes and trajectories from the above two
components of VRDP, it performs differentiable simulation to optimize the
physical parameters of the scene and objects by comparing the simulation with
the video observations $L$. Our physics model is implemented as an impulse-
based differentiable rigid-body simulator [36, 64, 14]. It iteratively
simulates a small time step of $\Delta t$ based on the objects’ state in the
BEV coordinate through inferring collision events, forces (including
resistance and collision force), and impulses acting on the object, and
updating the state of each object.

When an object moves on the ground with velocity $\overrightarrow{v}$ and
angular velocity $\omega$, we consider three kinds of forces that affect the
movement of the object: sliding friction, rolling resistance, and air
resistance. We use $\lambda_{1},\lambda_{2},\lambda_{3}$ to denote their
coefficients and have:

$\overrightarrow{a}=\begin{cases}-\frac{\overrightarrow{v}}{|\overrightarrow{v}|}(\lambda_{1}\mathrm{g}+\lambda_{3}|\overrightarrow{v}|^{2})~{}~{}~{}~{}~{}\text{if
the shape is not sphere}\\\
-\frac{\overrightarrow{v}}{|\overrightarrow{v}|}(\lambda_{2}\mathrm{g}+\lambda_{3}|\overrightarrow{v}|^{2})~{}~{}~{}~{}~{}\text{if
the shape is sphere}\end{cases}$ (1)

where $\mathrm{g}=9.81\mathrm{m/s^{2}}$ is the standard gravity and
$\overrightarrow{a}$ denotes the acceleration of the object, whose direction
is opposite to the velocity. The velocity $\overrightarrow{v}$ and the
location $\overrightarrow{l^{\prime}}=(x^{\prime},y^{\prime})$ are then
updated accordingly by the second order Runge-Kutta (RK2) algorithm [13].
Similarly, the angular velocity $\omega$ also decreases at each time step due
to the angular drag, and the angle $\alpha$ of the object is updated by the
RK2 algorithm.

The physics engine checks whether the boundaries of two objects with radius
$R$ are overlapped in the BEV coordinate frame to detect collision events.
Based on the fact that the total momentum of an isolated system should be
constant in the absence of net external forces, we compute the impulse of
collided objects and ignore the friction caused by the collision. Let
$(m_{1},m_{2})$, $(r_{1},r_{2})$, $(\alpha_{1}$, $\alpha_{2})$,
$(\overrightarrow{v_{1}},\overrightarrow{v_{2}})$,
$(\overrightarrow{l^{\prime}_{1}},\overrightarrow{l^{\prime}_{2}})$ denote the
mass, restitution, angle, velocity and BEV location of two collided objects at
the moment of the collision, respectively;
$\overrightarrow{d_{1}},\overrightarrow{d_{2}}$ represent their collision unit
directions that the force is acting on, where
$\overrightarrow{d_{1}}+\overrightarrow{d_{2}}=\overrightarrow{0}$. The change
of velocity $\Delta\overrightarrow{v_{1}},\Delta\overrightarrow{v_{2}}$ at the
moment of collision can be obtained by calculating the impulse on the
collision direction:

$\begin{split}&\overrightarrow{\Delta
v_{1}}=-(1+r_{1}r_{2})(m_{2}/(m_{1}+m_{2}))(\overrightarrow{d_{1}}\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}(\overrightarrow{v_{1}}-\overrightarrow{v_{2}}))\overrightarrow{d_{1}}\\\
&\overrightarrow{\Delta
v_{2}}=-(1+r_{1}r_{2})(m_{1}/(m_{1}+m_{2}))(\overrightarrow{d_{2}}\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$\scriptscriptstyle\bullet$}}}}}(\overrightarrow{v_{2}}-\overrightarrow{v_{1}}))\overrightarrow{d_{2}},\end{split}$
(2)

the velocity $\overrightarrow{v}$ is then updated by
$\overrightarrow{v}\leftarrow\overrightarrow{v}+\Delta\overrightarrow{v}$.
Similarly, the angular velocity $\omega$ can be updated by
$\omega\leftarrow\omega+\Delta\omega$, where $\Delta\omega$ is computed based
on conservation of angular momentum.

Given an initial state of the scene and objects, our physics engine simulates
force, impulse, and collision events and iteratively updates the state of each
element. All physical parameters including
$R,\lambda,m,r,\alpha,\overrightarrow{v},\overrightarrow{l}$ are initialized
and then optimized with L-BFGS algorithm [58] by fitting the simulated
trajectories $L^{\prime}=\\{(x^{\prime}_{t},y^{\prime}_{t})\\}_{t=1}^{T}$ into
the perceptual trajectories
$L^{\text{BEV}}=\\{(x^{\text{BEV}}_{t},y^{\text{BEV}}_{t})\\}_{t=1}^{T}$. To
alleviate the difficulty of the optimization, we mark the time frame of each
object’s first collision by calculating the BEV distance between every pair
and decompose the differentiable physical optimization and simulation into the
following steps: 1) Since radius $R$ and resistance coefficients $\lambda$ are
consistent in all videos, we use $K$ videos to jointly learn those physical
parameters and fix them for the optimization of other sample-dependent
parameters. 2) For each video, we then use the frames before the collision to
optimize the collision-independent physical parameters, such as initial
velocity $\overrightarrow{v_{0}}$, initial location $\overrightarrow{l_{0}}$,
and initial angle $\alpha_{0}$. 3) With the above parameters learned and
fixed, we optimize the remaining collision-dependent parameters, including
mass $m$ and resistance $r$ of each object. This process follows the
curriculum learning paradigm [12] by optimizing from fewer to more frames,
_e.g._ , multi-step optimization on [0, 40], [0, 80], and [0, 128] frames,
where the parameters in each step are initialized from the optimization of the
previous step. 4) With all parameters of the physical model learned, the
engine runs simulations and re-calculates the trajectory-based representations
$F_{l}$ for answering counterfactual, descriptive, and explanatory questions.
5) For the predictive case, we leverage the learned physical model as
initialization and re-optimize all sample-dependent parameters with only the
last 20 frames to reduce the cumulative error over time.

Symbolic Execution As in [60, 16], we perform reasoning with a program
executor, which is a collection of deterministic functional modules designed
to realize all logic operations specified in symbolic programs. Its input
consists of the parsed programs, learned concept embeddings, and visual
representations, including the appearance-based feature $F_{a}$ from the
visual perception module and the updated trajectory feature $F_{l}$ from the
physics engine. Given a set of parsed programs, the program executor runs them
step-by-step and derives the answer based on these representations. For
example, the ‘counting’ program outputs the number of objects which meet
specific conditions (_e.g._ , red sphere). In this process, the executor
leverages the concept learner to filter out eligible objects.

Our reasoning process is designed fully differentiable w.r.t. the visual
representations and the concept embeddings by representing all object states,
events, and results of all operators in a probabilistic manner during
training, supporting gradient-based optimization. Moreover, it works
seamlessly with our explicit physics engine, which simulates dynamic
predictions through real physical parameters, forming a symbolic and
deterministic physical reasoning process. The whole reasoning process is fully
transparent and step-by-step interpretable.

###  3.3 Training Objectives

Similar to [16, 81], we train the program parser with program labels using
cross-entropy loss,

$\mathcal{L}_{program}=-\sum_{j=1}^{J}\mathbf{1}\\{y_{p}=j\\}\log(p_{j}),$ (3)

where $J$ is the size of the pre-defined program set, $p_{j}$ is the
probability for the $j$-th program and $y_{p}$ is the ground-truth program
label.

We optimize the physical parameters in the physical model by comparing the
simulation trajectories with the video observations. All physical parameters
including $R,\lambda,m,r,\alpha,\overrightarrow{v},\overrightarrow{l}$ are
initialized and then optimized with L-BFGS algorithm [58] by fitting the
simulated trajectories
$L^{\prime}=\\{(x^{\prime}_{t},y^{\prime}_{t})\\}_{t=1}^{T}$ into the
perceptual trajectories
$L^{\text{BEV}}=\\{(x^{\text{BEV}}_{t},y^{\text{BEV}}_{t})\\}_{t=1}^{T}$. We
have:

$\mathcal{L}_{Physics}=\|L^{\prime}-L^{\text{BEV}}\|^{2}_{2},$ (4)

We optimize the feature extractor and the concept embeddings in the concept
learner by question answering. We treat each option of a multiple-choice
question as an independent boolean question during training. Specifically, we
use cross-entropy loss to supervise open-ended questions and use mean square
error loss to supervise counting questions. Formally, for counting questions,
we have

$\mathcal{L}_{QA,count}=(y_{a}-z)^{2},$ (5)

where $z$ is the predicted number and $y_{a}$ is the ground-truth number
label. For other open-ended questions and multiple-choice questions, we have

$\mathcal{L}_{QA,others}=-\sum_{a=1}^{A}\mathbf{1}\\{y_{a}=a\\}\log(p_{a}),$
(6)

where $A$ is the size of the pre-defined answer set, $p_{a}$ is the
probability for the $a$-th answer and $y_{a}$ is the ground-truth answer
label.

##  4 Experiments

By recovering physics models of objects and their interactions from video and
language, VRDP enjoys the following benefits: 1) high accuracy and full
transparency, 2) superior data efficiency, and 3) high generalizability. In
this section, we first evaluate the accuracy and data efficiency of VRDP on
the widely used dynamic visual reasoning benchmark CLEVRER [81] and its
subsets; we then validate the model’s generalizability on adapting to new
concepts with few-shot data; lastly, we experiment on the real-world dataset
Real-Billiard [65] to show that VRDP works well in real-world dynamic
prediction and reasoning.

Datasets and Evaluation Settings To validate the effectiveness of our method
for reasoning about the physical world, we conduct main experiments on the
CLEVRER [81] dataset, as it contains both language and physics cues such as
rigid body collisions and dynamics, compared to other benchmarks that focus on
either action understanding without physical inferring [49, 41] or temporal
reasoning without language concepts [26, 8]. CLEVRER includes four types of
question answering (QA): descriptive, explanatory, predictive, and
counterfactual, where the first two types concern more on video understanding,
while the latter two types involve physical dynamics and predictions in
reasoning. Therefore, we mainly focus on the predictive and counterfactual
questions in this work and use QA accuracy as the evaluation metric. Note the
multi-choice question (explanatory, predictive, and counterfactual) contains
multiple options. Only if all the options (per opt.) are answered correctly
can it be regarded as a correct question (per ques.).

We then collect a few-shot physical reasoning dataset with novel language and
physical concepts (_e.g._ , “heavier” and “lighter”), termed generalized
CLEVRER, containing 100 videos (split into 25/25/50 for train/validation/test)
with 375 options in 158 counterfactual questions. This dataset is
supplementary to CLEVRER [81] for generalizing to new concepts with very few
samples. For real-world scenarios, we conduct experiments on the Real-Billiard
[65] dataset, which contains three-cushion billiards videos captured in real
games for dynamics prediction. We generate 6 reasoning questions (_e.g._ ,
“will one billiard collide with …?”) for each video and evaluate both the
prediction error and QA accuracy.

Implementation Details We follow the experimental setting in [81, 16] using a
pre-trained Faster R-CNN model [30] to generate object proposals for each
frame and training the language program parser with 1,000 programs for all
question types. We implement three versions of VRDP models, where our
unsupervised VRDP leverage a Slot-Attention model [59] to parse the objects
unsupervisedly, while VRDP ${\dagger}$ use Faster-RCNN [30] as the object
detector. In addition to our standard model that grounds object properties
from question-answer pairs, we also train a variant (VRDP
${\dagger}{\ddagger}$) on CLEVRER with an explicit rule-based program executor
[81] and object attribute supervision. The camera matrix is optimized from 20
training videos. We set $\Delta t=0.004\text{s},K=10,S=10$, and $T=128$ for
CLEVRER [81] and $T=20$ for Real-Billiard [65]. More details of the dataset
and settings can be found in Supplemental Materials.

Methods | Overall | Predictive | Counterfactual | Descriptive | Explanatory
---|---|---|---|---|---
per task | per ques. | per opt. | per ques. | per opt. | per ques. | per opt. | per ques.
TVQA+ [49] | 37.2 | 57.3 | 70.3 | 48.9 | 53.9 | 4.1 | 72.0 | 63.3 | 23.7
Memory [21] | 27.2 | 43.3 | 50.0 | 33.1 | 54.2 | 7.0 | 54.7 | 53.7 | 13.9
IEP (V) [45] | 20.2 | 40.5 | 50.0 | 9.7 | 53.4 | 3.8 | 52.8 | 52.6 | 14.5
TbD-net (V) [61] | 23.6 | 58.6 | 50.3 | 6.5 | 56.1 | 4.4 | 79.5 | 61.6 | 3.8
HCRN [48] | 27.3 | 44.8 | 54.1 | 21.0 | 57.1 | 11.5 | 55.7 | 63.3 | 21.0
MAC (V) [39] | 32.1 | 65.5 | 51.0 | 16.5 | 54.6 | 13.7 | 85.6 | 59.5 | 12.5
MAC (V+) [39] † | 44.2 | 69.8 | 59.7 | 42.9 | 63.5 | 25.1 | 86.4 | 70.5 | 22.3
NS-DR [81] †‡ | 69.7 | 80.7 | 82.9 | 68.7 | 74.1 | 42.2 | 88.1 | 87.6 | 79.6
NS-DR (NE) [81] †‡ | 64.1 | 77.7 | 75.4 | 54.1 | 76.1 | 42.0 | 85.8 | 85.9 | 74.3
DCL [16] † | 75.5 | 84.1 | 90.5 | 82.0 | 80.4 | 46.5 | 90.7 | 89.6 | 82.8
DCL-Oracle [16] †‡ | 75.6 | 84.5 | 90.6 | 82.1 | 80.7 | 46.9 | 91.4 | 89.8 | 82.0
Object-based Attention [20] | 88.3 | 91.7 | 93.5 | 87.5 | 91.4 | 75.6 | 94.0 | 98.5 | 96.0
VRDP (ours) | 82.9 | 86.9 | 91.7 | 83.8 | 89.9 | 75.7 | 89.8 | 89.1 | 82.4
VRDP (ours) † | 86.6 | 89.4 | 94.5 | 89.2 | 92.5 | 80.7 | 91.5 | 90.9 | 85.2
VRDP (ours) †‡ | 90.3 | 92.0 | 95.7 | 91.4 | 94.8 | 84.3 | 93.4 | 96.3 | 91.9

Table 1: Question-answering accuracy of visual reasoning models on CLEVRER
[81]. We report per-task and per-question overall accuracies, as well as per-
option and per-question accuracies for each sub-task. Note that predictive and
counterfactual questions that require dynamics and physical prediction are our
focus. † denotes the method uses a supervised object detector, such as
Faster/Mask R-CNN [30]. ‡ indicates the use of object properties (_i.e._ ,
shape, color, and material) as supervision.

###  4.1 Comparative Results on CLEVRER

We conduct experiments on CLEVRER against several counterparts: TVQA+ [49],
Memory [21], IEP (V) [45], TbD-net (V) [61], HCRN [48], MAC [39], NS-DR [81],
DCL [16], and Object-based Attention [20]. Among them, NS-DR [81] and DCL [16]
are high-performance interpretable symbolic models, while Object-based
Attention [20] is the state-of-the-art end-to-end method.

From Tab. 1 we observe that: 1) Counterfactual and predictive questions are
more difficult than descriptive and explanatory ones as they require accurate
physical dynamics and prediction hence our main focus. By reconstructing the
physical world explicitly, our method outperforms all existing works on these
two types by large margins. For example, VRDP ${\dagger}$ improves the per
question accuracy of counterfactual questions by 11.5% and 79.7% compared to
the best end-to-end [20] and neural-symbolic [16] counterparts.

2) The end-to-end model [20] improves the accuracy at the cost of losing model
transparency and interpretability. However, by leveraging object attribute
supervision and explicit program executors [81], our VRDP
${\dagger}{\ddagger}$ achieves new state-of-the-art overall performance on
CLEVRER. It closes the performance gap between interpretable models and state-
of-the-art end-to-end methods. Moreover, it shows the flexibility of our
physics model that can be combined with various physical concepts and program
executors while achieving impressive performance.

3) We conducted ablative experiments to study the impact of pre-trained object
detection modules of our framework by replacing the supervised visual model
[30] in VRDP ${\dagger}$ with an unsupervised one [59] in VRDP. We observe
that although the use of unsupervised detectors decreases the performance
slightly, our framework still enjoys higher performance than previous methods
in counterfactual and predictive questions.

4) Neither the neuro-symbolic nor end-to-end works employ explicit dynamic
models with physical meanings. In contrast, our model is fully transparent
with step-by-step interpretable programs and meaningful physical parameters
powered by a differentiable engine.

Figure 2: Comparisons of the data efficiency evaluation on four types of
questions with MAC (V) [39] and Object-based Attention [20] trained with
different proportion of the CLEVRER [81] dataset. Our method is highly data-
efficient in that it achieves comparable results with the state-of-the-art
counterpart [20] with $3\times$ fewer data. It improves the reasoning accuracy
significantly when fewer data (_e.g._ , 20%) are used.

###  4.2 Detailed Analysis

Evaluation of Data Efficiency We evaluated the data efficiency of VRDP with
two representative models: MAC (V) [39] and Object-based Attention [20]. From
Fig. 2 we see that: VRDP is highly data-efficient. When the amount of data is
reduced, the accuracy of our model drops slightly, while the performance of
MAC (V) [39] and Object-based Attention [20] drops drastically due to
insufficient data. For example, we improve the counterfactual accuracy of
Object-based Attention [20] by 426% under the setting of 20% data. Notably,
our model uses 20% of the dataset to achieve comparable performance to other
works that use 80% of data. This is because the components of VRDP, _e.g._ ,
perception module and question parser, can be trained with a small amount of
data. More importantly, our physics model is built based on an explicit
physics engine, which can be optimized from the trajectory of a single video.

Figure 3: VRDP learns new concepts and accurately reasons about counterfactual
events from few data on generalized CLEVRER.

Methods | Per opt. | Per ques.
---|---|---
MAC (V) [39] | 63.8 | 22.0
Object-based Attention [20] | 59.5 | 26.7
VRDP (Ours) | 88.1 | 75.6

Table 2: Comparative results of generalizability evaluation under the few-shot
setting. All models are first pretrained on CLEVRER and then finetuned with
only 25 videos for adapting to generalized CLEVRER. VRDP can learn new
concepts quickly with few-shot data.

Evaluation of Generalizability This part studies the generalization
capabilities of VRDP against MAC (V) [39] and Object-based Attention [20] on
the generalized CLEVRER dataset. Tab. 2 shows our model outperforms other
works by a large margin (75.6 vs. 26.7) on per question accuracy,
demonstrating our model can quickly learn new concepts from few examples by
reconstructing the physics world. An example of generalization with few-shot
data is shown in Fig. 3. Our model learns a novel concept “heavier” from only
25 videos and the corresponding question-answer pairs. The simulation is then
run with 5 times the mass to answer the question correctly.

Methods | Overall | Predictive | Counterfactual | Descriptive | Explanatory
---|---|---|---|---|---
per task | per ques. | per opt. | per ques. | per opt. | per ques. | per opt. | per ques.
Baseline | 72.6 | 81.6 | 85.1 | 72.4 | 77.6 | 49.6 | 87.8 | 88.0 | 80.6
\+ Collision-independent First | 81.3 | 87.8 | 86.1 | 72.8 | 89.3 | 74.1 | 91.3 | 91.9 | 86.9
\+ Curriculum Optimization | 85.6 | 90.2 | 87.6 | 76.5 | 94.8 | 84.3 | 92.2 | 93.3 | 89.2
\+ Re-optimization for Prediction | 90.3 | 92.0 | 95.7 | 91.4 | 94.8 | 84.3 | 93.4 | 96.3 | 91.9

Table 3: Ablation study on the optimization of physical parameters on CLEVRER
[81]. The reasoning accuracy for the four types of questions is continuously
increased through a better learning process.

Ablation Study on the Learning of Physics Models In this work, sample-
independent physical parameters ($R$, $\lambda$) are learned from multiple
training videos. In contrast, the sample-dependent parameters, such as
$m,r,\alpha,v,l$, can only be learned with a single video, leading to
difficulties in optimization, especially when there are many collisions. This
part studies the optimization of these sample-dependent parameters by making
comparisons among the following four simplified learning processes on CLEVRER
[81]: 1) Baseline – optimize all target parameters directly from all frames
simultaneously. 2) Collision-independent First – first use the frames before
the collision to optimize collision-independent parameters for each object,
including initial velocity $\overrightarrow{v_{0}}$, initial location
$\overrightarrow{l_{0}}$, and initial angle $\alpha_{0}$; then optimize mass
$m$ and restitution $r$ from all video frames. 3) Curriculum Optimization –
optimize $m$ and $r$ by performing multiple steps on [0, 40], [0, 80], and [0,
128] frames, where each step is initialized from the optimization of the
previous step. 4) Re-optimization for Prediction (Full model) – leverage the
learned physical parameters as initialization and re-optimize all sample-
dependent parameters with the last 20 frames to reduce the cumulative error
over time.

Tab. 3 shows that the performance continuously increases when more
optimization steps are used, demonstrating the contribution of each part. The
“Collision-independent First” rule offers the greatest improvement, especially
for counterfactual questions, as counterfactual simulations only rely on the
initial state. “Curriculum Optimization” improves all types of questions, and
“Re-optimization for Prediction” re-calculates the dynamics of the last 20
frames, thus mainly affect predictive questions.

Failure Analysis VRDP learns the physics model from object trajectories in
videos and language concepts in question-answer pairs. It is data-efficient
and robust enough to work well when there exists inaccurate perception or
incorrect concept learning in some video frames. However, we noticed that the
model might fail in the following cases: 1) If the object collides immediately
after entering the image plane, there are insufficient frames before the
collision to learn the initial velocity $v_{0}$. 2) If no collision occurs on
an object, its restitution $r$ and mass $m$ cannot be optimized (unknown). We
set default values for them. 3) The optimization becomes difficult if there
are many cubes and collisions between them in the scene, because cube
collisions (considering the sides and corners) are more complicated than
sphere and cylinders’. These issues are challenging and will be our future
work.

Figure 4: An example of physical simulation and question-answering on the
real-world billiard dataset [65]. VRDP learns accurate physics parameters and
infers the correct answer by simulation.

Methods | S1$\text{ Err.}\downarrow$ | S2$\text{ Err.}\downarrow$ | QA Acc. (%)$\uparrow$
---|---|---|---
VIN [73] | 1.02 | 5.11 | 58.3
OM [42] | 0.59 | 3.23 | 61.1
CVP [79] | 3.57 | 6.63 | 58.3
IN [10] | 0.37 | 2.72 | 69.4
CIN [65] | 0.30 | 2.34 | 72.2
VRDP (Ours) | 0.24 | 0.88 | 80.6

Table 4: Comparisons of the prediction error and question-answering accuracy
on Real-Billiards. The rollout timesteps are chosen to be the same (S1) and
twice (S2) as the training time ($T=20$). The error is scaled by 1,000.

###  4.3 Comparative Results on Real-World Billiards

We also conduct experiments on the real-world dataset Real-billiard [65] with
our supplemented question-answer pairs. Note that the billiard table is a
chaotic system, and highly accurate long-term prediction is intractable. Fig.
4 shows an example of the ground truth video and our simulated prediction
based on the perceptual grounded physics model. It can be seen that the
predicted collision events and trajectories are of good quality. Tab. 4
evaluates the prediction errors under two different rollout timesteps and QA
accuracy with 5 competitors: VIN [73], OM [42], CVP [79], IN [10], and CIN
[65]. For the prediction task, the rollout timesteps are chosen to be the same
(S1$=[0,T]$) and twice (S2$=[T,2T]$) as the training time, where the training
time $T=20$. We refer interested readers to CIN [65] for more details. We find
that VRDP is superior to these methods on both prediction and question
answering tasks. Moreover, VRDP works well in long-term prediction. It reduces
the S2 error on CIN [65] by 62.4%.

##  5 Conclusion

This work introduces VRDP, a unified framework that integrates powerful
differentiable physics models into dynamic visual reasoning. It contains three
mutually beneficial components: a visual perception module, a concept learner,
and a differentiable physics engine. The visual perception module parses the
input video into object trajectories and visual representations; the concept
learner grounds language concepts and object attributes from question-answer
pairs and the visual representations; with object trajectories and attributes
as prior knowledge, the physics model optimizes all physical parameters of the
scene and objects by differentiable simulation. With these explicit physical
parameters, the physics model reruns the simulation to reason about future
motion and causal events, which are then executed by a symbolic program
executor to get the answer. Equipped with the powerful physics model, VRDP is
of highly data-efficient and generalizable that adapts to novel concepts
quickly with few-shot data. Moreover, both the explicit physics engine and the
symbolic executor are step-by-step interpretable, making VRDP fully
transparent. Extensive experiments on CLEVRER and Real-Billiards show that
VRDP outperforms state-of-the-art dynamic reasoning methods by large margins.

## Acknowledgments and Disclosure of Funding

This work was supported by MIT-IBM Watson AI Lab and its member company
Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR
(N00014-18-1-2847), and Mitsubishi Electric; Ping Luo was supported by the
General Research Fund of Hong Kong No.27208720.

## References

  * [1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine.  Learning to poke by poking: Experiential learning of intuitive physics.  In NeurIPS, 2016.
  * [2] A. Ajay, M. Bauza, J. Wu, N. Fazeli, J. B. Tenenbaum, A. Rodriguez, and L. P. Kaelbling.  Combining physical simulators and object-based networks for control.  In ICRA, 2019.
  * [3] S. Amizadeh, H. Palangi, O. Polozov, Y. Huang, and K. Koishida.  Neuro-symbolic visual reasoning: Disentangling" visual" from" reasoning".  In ICML, 2020.
  * [4] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein.  Neural module networks.  In CVPR, 2016.
  * [5] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh.  Vqa: Visual question answering.  In ICCV, 2015.
  * [6] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  In ICLR, 2015.
  * [7] A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick.  Phyre: A new benchmark for physical reasoning.  In NeurIPS, 2019.
  * [8] F. Baradel, N. Neverova, J. Mille, G. Mori, and C. Wolf.  Cophy: Counterfactual learning of physical dynamics.  In ICLR, 2020.
  * [9] P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum.  Simulation as an engine of physical scene understanding.  Proceedings of the National Academy of Sciences, 2013.
  * [10] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu.  Interaction networks for learning about objects, relations and physics.  arXiv preprint arXiv:1612.00222, 2016.
  * [11] D. M. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. Fei-Fei, J. Wu, J. B. Tenenbaum, et al.  Learning physical graph representations from visual scenes.  In NeurIPS, 2020.
  * [12] Y. Bengio, J. Louradour, R. Collobert, and J. Weston.  Curriculum learning.  In ICML, pages 41–48, 2009.
  * [13] J. C. Butcher.  A stability property of implicit runge-kutta methods.  BIT Numerical Mathematics, pages 358–361, 1975.
  * [14] E. Catto.  Modeling and solving constraints.  In Game Developers Conference, page 16, 2009.
  * [15] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum.  A compositional object-based approach to learning physical dynamics.  In ICLR, 2017.
  * [16] Z. Chen, J. Mao, J. Wu, K.-Y. K. Wong, J. B. Tenenbaum, and C. Gan.  Grounding physical concepts of objects and events through dynamic visual reasoning.  In ICLR, 2021.
  * [17] E. Coumans.  Bullet physics engine.  Open Source Software: http://bulletphysics. org, 1(3):84, 2010.
  * [18] F. de Avila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter.  End-to-end differentiable physics for learning and control.  In NeurIPS, volume 31, pages 7178–7189, 2018.
  * [19] J. Degrave, M. Hermans, J. Dambre, et al.  A differentiable physics engine for deep learning in robotics.  Frontiers in neurorobotics, 13:6, 2019.
  * [20] D. Ding, F. Hill, A. Santoro, and M. Botvinick.  Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures.  arXiv preprint arXiv:2012.08508, 2020.
  * [21] C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In CVPR, 2019.
  * [22] C. Finn, I. Goodfellow, and S. Levine.  Unsupervised learning for physical interaction through video prediction.  In NeurIPS, 2016.
  * [23] C. Gan, Y. Li, H. Li, C. Sun, and B. Gong.  VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation.  In ICCV, pages 1811–1820, 2017.
  * [24] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al.  Threedworld: A platform for interactive multi-modal physical simulation.  NeurIPS, 2021.
  * [25] C. Gan, S. Zhou, J. Schwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. Yamins, J. J. DiCarlo, J. McDermott, A. Torralba, et al.  The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai.  arXiv preprint arXiv:2103.14025, 2021.
  * [26] R. Girdhar and D. Ramanan.  Cater: A diagnostic dataset for compositional actions and temporal reasoning.  In ICLR, 2020.
  * [27] G. Gkioxari and J. Malik.  Finding action tubes.  In CVPR, 2015.
  * [28] C. Han, J. Mao, C. Gan, J. Tenenbaum, and J. Wu.  Visual concept-metaconcept learning.  In NeurIPS, 2019.
  * [29] X. Han, S. Wang, C. Su, W. Zhang, Q. Huang, and Q. Tian.  Interpretable visual reasoning via probabilistic formulation under natural supervision.  In ECCV, 2020.
  * [30] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [32] E. Heiden, D. Millard, E. Coumans, Y. Sheng, and G. S. Sukhatme.  Neuralsim: Augmenting differentiable simulators with neural networks.  arXiv preprint arXiv:2011.04217, 2020.
  * [33] E. Heiden, D. Millard, H. Zhang, and G. S. Sukhatme.  Interactive differentiable simulation.  arXiv preprint arXiv:1905.10706, 2019.
  * [34] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [35] R. Hu, J. Andreas, T. Darrell, and K. Saenko.  Explainable neural computation via stack neural module networks.  In ECCV, 2018.
  * [36] Y. Hu, L. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand.  Difftaichi: Differentiable programming for physical simulation.  In ICLR, 2020.
  * [37] D. Huang, P. Chen, R. Zeng, Q. Du, M. Tan, and C. Gan.  Location-aware graph convolutional networks for video question answering.  In AAAI, pages 11021–11028, 2020.
  * [38] Z. Huang, Y. Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan.  Plasticinelab: A soft-body manipulation benchmark with differentiable physics.  In ICLR, 2021.
  * [39] D. A. Hudson and C. D. Manning.  Compositional attention networks for machine reasoning.  In ICLR, 2018.
  * [40] D. A. Hudson and C. D. Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In CVPR, 2019.
  * [41] Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In CVPR, 2017.
  * [42] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu.  Reasoning about physical interactions with object-oriented prediction and planning.  In ICLR, 2019.
  * [43] K. M. Jatavallabhula, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Considine, J. Parent-Levesque, K. Xie, K. Erleben, et al.  gradsim: Differentiable simulation for system identification and visuomotor control.  In ICLR, 2021.
  * [44] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In CVPR, 2017.
  * [45] J. Johnson, B. Hariharan, L. Van Der Maaten, J. Hoffman, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick.  Inferring and executing programs for visual reasoning.  In ICCV, pages 2989–2998, 2017.
  * [46] T. Kipf, E. van der Pol, and M. Welling.  Contrastive learning of structured world models.  In ICLR, 2020.
  * [47] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * [48] T. M. Le, V. Le, S. Venkatesh, and T. Tran.  Hierarchical conditional relation networks for video question answering.  In CVPR, pages 9972–9981, 2020.
  * [49] J. Lei, L. Yu, M. Bansal, and T. L. Berg.  Tvqa: Localized, compositional video question answering.  In EMNLP, 2018.
  * [50] A. Lerer, S. Gross, and R. Fergus.  Learning physical intuition of block towers by example.  In ICML, 2016.
  * [51] X. Li, J. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan.  Beyond rnns: Positional self-attention with co-attention for video question answering.  In AAAI, pages 8658–8665, 2019.
  * [52] Y. Li, T. Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba.  Visual grounding of learned physical models.  In ICML, pages 5927–5936, 2020.
  * [53] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba.  Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids.  In ICLR, 2019.
  * [54] Y. Li, J. Wu, J.-Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake.  Propagation networks for model-based control under partial observation.  In ICRA, 2019.
  * [55] J. Liang and M. C. Lin.  Differentiable physics simulation.  In ICLR Workshop, 2020.
  * [56] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [57] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [58] D. C. Liu and J. Nocedal.  On the limited memory bfgs method for large scale optimization.  Mathematical programming, pages 503–528, 1989.
  * [59] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf.  Object-centric learning with slot attention.  arXiv preprint arXiv:2006.15055, 2020.
  * [60] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  In ICLR, 2019.
  * [61] D. Mascharka, P. Tran, R. Soklaski, and A. Majumdar.  Transparency by design: Closing the gap between performance and interpretability in visual reasoning.  In CVPR, pages 4942–4950, 2018.
  * [62] I. Misra, R. Girshick, R. Fergus, M. Hebert, A. Gupta, and L. Van Der Maaten.  Learning by asking questions.  In CVPR, pages 11–20, 2018.
  * [63] R. Mottaghi, M. Rastegari, A. Gupta, and A. Farhadi.  “what happens if…” learning to predict the effect of forces in images.  In ECCV. Springer, 2016.
  * [64] M. Müller, J. Stam, D. James, and N. Thürey.  Real time physics: class notes.  In ACM SIGGRAPH 2008 classes, pages 1–90, 2008.
  * [65] H. Qi, X. Wang, D. Pathak, Y. Ma, and J. Malik.  Learning long-term visual dynamics with region proposal interaction networks.  In ICLR, 2021.
  * [66] R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux.  Intphys: A framework and benchmark for visual intuitive physics reasoning.  arXiv preprint arXiv:1803.07616, 2018.
  * [67] T. Shao, A. Monszpart, Y. Zheng, B. Koo, W. Xu, K. Zhou, and N. J. Mitra.  Imagining the unseen: Stability-based cuboid arrangements for scene understanding.  ACM TOG, 2014.
  * [68] K. Smith, L. Mei, S. Yao, J. Wu, E. Spelke, J. Tenenbaum, and T. Ullman.  Modeling expectation violation in intuitive physics with coarse probabilistic object representations.  NeurIPS, 32:8985–8995, 2019.
  * [69] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler.  MovieQA: Understanding Stories in Movies through Question-Answering.  In CVPR, 2016.
  * [70] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum.  Differentiable physics and stable modes for tool-use and manipulation planning.  In IJCAI, 2019.
  * [71] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [72] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and S. Levine.  Entity abstraction in visual model-based reinforcement learning.  In CoRL, pages 1439–1456, 2020.
  * [73] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti.  Visual interaction networks: Learning a physics simulator from video.  NeurIPS, 30:4539–4547, 2017.
  * [74] J. Wu, E. Lu, P. Kohli, B. Freeman, and J. Tenenbaum.  Learning to see physics via visual de-animation.  In NeurIPS, pages 153–164, 2017.
  * [75] J. Wu, I. Yildirim, J. J. Lim, W. T. Freeman, and J. B. Tenenbaum.  Galileo: Perceiving physical object properties by integrating a physics engine with deep learning.  In NeurIPS, 2015.
  * [76] Q. Wu, C. Shen, L. Liu, A. Dick, and A. van den Hengel.  What value do explicit high level concepts have in vision to language problems?  In CVPR, 2016.
  * [77] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In ACM MM, 2017.
  * [78] T. Ye, X. Wang, J. Davidson, and A. Gupta.  Interpretable intuitive physics model.  In ECCV, 2018.
  * [79] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani.  Compositional video prediction.  In ICCV, pages 10353–10362, 2019.
  * [80] Y. Ye, Z. Zhao, Y. Li, L. Chen, J. Xiao, and Y. Zhuang.  Video question answering via attribute-augmented attention network learning.  In ICLR, 2017.
  * [81] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum.  Clevrer: Collision events for video representation and reasoning.  In ICLR, 2020.
  * [82] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. Tenenbaum.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.  In NeurIPS, 2018.
  * [83] A. Zadeh, M. Chan, P. P. Liang, E. Tong, and L.-P. Morency.  Social-iq: A question answering benchmark for artificial social intelligence.  In CVPR, 2019.
  * [84] H. Zhou, A. Kadav, F. Lai, A. Niculescu-Mizil, M. R. Min, M. Kapadia, and H. P. Graf.  Hopper: Multi-hop transformer for spatiotemporal reasoning.  In ICLR, 2021.
  * [85] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei.  Visual7w: Grounded question answering in images.  In CVPR, 2016.

##  Appendix A Appendix

In this section, we provide supplementary details of our VRDP. First, we give
more details of our physics model and the neuro-symbolic operations in the
program executor. We then introduce the datasets we use and build, including a
synthetic dataset (CLEVRER [81]), a real-world dataset (Real-Billiard [65]),
and a newly built few-shot dataset (Generalized CLEVRER). After that, we
detail the training settings and steps.

###  A.1 Details of Physics Model

In this part, we provide supplementary details of our physics model. With the
perceptually grounded object shapes and trajectories from the perception
module and the concept learner of VRDP, our physics model performs
differentiable simulation to optimize the physical parameters of the scene and
objects by comparing the simulation $L^{\prime}$ with the video observations
$L^{\text{BEV}}$. The target bird’s-eye view (BEV) trajectory $L^{\text{BEV}}$
is obtained by projecting the object center to the BEV coordinate. The Camera-
to-BEV projection can be written as:

$\begin{bmatrix}x\\\ y\\\ -\\\ 1\\\
\end{bmatrix}_{BEV}=\mathbf{K}^{-1}\cdot\begin{bmatrix}~{}x\cdot z~{}\\\
~{}y\cdot z~{}\\\ ~{}z~{}\\\ ~{}1~{}\\\ \end{bmatrix}_{camera}$ (7)

where $\mathbf{K}$ is the estimated camera matrix, $[x,y,z]_{camera}$ is the
point in 2D image coordinates ($z_{camera}$ can be calculated from the camera
matrix $\mathbf{K}$), $[x,y]_{BEV}$ denotes the horizontal position and
vertical position of the projected point in BEV coordinates.

Based on the graphics programming language DiffTaichi [36], our physics model
is implemented as an impulse-based differentiable rigid-body simulator. Based
on conservation of momentum and angular momentum, it iteratively simulates a
small time step of $\Delta t$ based on the objects’ state in the BEV
coordinate through inferring collision events, forces and impulses acting on
the object, and updating the state of each object. In addition to calculating
the acceleration based on the conservation of momentum in our main paper, we
also calculate the angular acceleration based on the angular momentum. For
example, we have:
$\overrightarrow{M}=\overrightarrow{r}\times\overrightarrow{F}$ and
$M=I\frac{\mathrm{d}\omega}{\mathrm{d}t}$, where $\overrightarrow{M}$ denotes
moment of force, $\overrightarrow{F}$ is the applied force, and
$\overrightarrow{r}$ is the distance from the applied force to object. The
momentum of inertia $I$ is $1/6m(2R)^{2}$ for the cube, where $m$ represents
its mass.

Figure 5: An illustration of circle-rectangle collision detection. The gray
part denotes the rectangle (cube) and we transform the origin to the center of
the rectangle so that the coordinate axis is parallel to its side. For each
simulation step: we consider three situations: 1) if the center of the circle
is in the orange area, the circle and the rectangle do not collide; 2) if the
circle center is in the red area, the circle collides with the rectangle and
the collision direction is perpendicular to the coordinate axis; 3) if the
circle center falls in the purple area, the circle and the rectangle collide
and the collision direction is perpendicular to the tangent of the collision
position on the circle.

In this work, we perform collision detection between circles and rectangles in
BEV view. Fig. 5 shows the illustration of our circle-rectangle collision
detection algorithm. We project the center of the rectangle (the gray part) to
the origin so that the coordinate axis is parallel to its side. Then the area
outside the rectangle is divided into three parts that the center of the
circle can fall: 1) collision with the sides of the square (red); 2) collision
with the corners of the square (purple); 3) no collision (orange). The
implementation of circle-circle and rectangle-rectangle collisions is similar.

Figure 6: An illustration of the reasoning process of the program executor and
concept learner. The program executor executes the parsed programs (_e.g._ ,
Filter_static_concept (color, shape, material)) step-by-step with the visual
representations and language concepts. For each step, it leverages the concept
learner or physical model to filter specific targets or simulate/predict new
visual trajectories.

###  A.2 Details of Neuro-Symbolic Programs

Following DCL [16], we represent the objects, events and moments through
learnable embeddings and quantize the static and dynamic concepts to perform
temporal and causal reasoning. In this part, we list all the available data
types and operations for CLEVRER in Tab. 5. We refer interested readers to DCL
[16] for more details.

We also visualize the reasoning process of an example step-by-step in Fig. 6.
It shows how we get the correct answer for the counterfactual question
‘Without the green sphere, what will happen?’ with a choice ‘The green
cylinder and the cyan object collide’. After the first program
‘Filter_static_concept(all objects, green sphere)’ is executed, the executor
removes the retrieved object, reruns the simulation to get counterfactual
trajectories, and updates the visual features. After that, the executor runs
the remaining programs and gets the final answer ‘True’ with a probability of
0.96 calculated through the cosine distance in the concept learner.

###  A.3 Details of Datasets

CLEVRER CLEVRER [81] is a diagnostic video dataset for systematic evaluation
of computational models on a wide range of reasoning tasks. Objects in CLEVRER
videos adopt similar compositional intrinsic attributes as in CLEVR [44],
including three shapes (cube, sphere, and cylinder), two materials (metal and
rubber), and eight colours (gray, red, blue, green, brown, cyan, purple, and
yellow). All objects have the same size, same friction coefficient (except the
sphere that rolling on the ground), so no vertical bouncing occurs during the
collision. Each object has a different mass and a different restitution
coefficient. CLEVRER introduces three types of events: enter, exit and
collision, each of which contains a fixed number of object participants: 2 for
collision and 1 for enter and exit. The objects and events form an abstract
representation of the video.

CLEVRER includes four types of question: descriptive (_e.g._ ‘what color’),
explanatory (‘what’s responsible for’), predictive (‘what will happen next’),
and counterfactual (‘what if’), where the first two types concern more on
video understanding and temporal reasoning, while the latter two types involve
physical dynamics and predictions in reasoning. Therefore, we mainly focus on
the predictive and counterfactual questions in this work. CLEVRER consists of
2,000 videos, with a number of 1,000 training videos, 5,000 validation videos,
and 5,000 test videos. It also contains 219,918 descriptive questions, 33,811
explanatory questions, 14,298 predictive questions, and 37,253 counterfactual
questions. In this paper, we tune the model using the validation set and
evaluate it with the test set.

Generalized CLEVRER To evaluate the generalizability of reasoning methods, we
collect a few-shot physical reasoning dataset with novel language and physical
concepts (_e.g._ , ‘heavier’ and ‘lighter’), termed generalized CLEVRER,
containing 100 videos (split into 25/25/50 for train/validation/test) with 375
options in 158 counterfactual questions. This dataset is supplementary to
CLEVRER [81] for generalizing to new concepts (_i.e._ , heavier, lighter) with
very few samples. All videos last for 5 seconds and are generated by a physics
engine [17] that simulates object motion plus a graphs engine that renders the
frames. It has the same video settings (objects and events settings) with
CLEVRER but different questions/concepts, _e.g._ , “What would happen if the
blue sphere were heavier?”, we generate the ground truth video in the
counterfactual case by setting five times the weight and perform the physical
simulation with Bullet [17]. In this work, we evaluate the QA accuracy of this
dataset.

Real-Billiard For real-world scenarios, we conduct experiments on the Real-
Billiard [65] dataset, which contains three-cushion billiards videos captured
in real games for dynamics prediction. There are 62 training videos with
18,306 frames, and 5 testing videos with 1,995 frames. The bounding box
annotations are from an off-the-shelf ResNet-101 FPN detector [56] pretrained
on COCO [57] and fine-tuned on a subset of 30 images from our dataset. Wrong
detections are manually filtered out. We generate 6 reasoning questions
(_e.g._ , “will one billiard collide with …?”) for each video and evaluate
both the prediction error and QA accuracy.

###  A.4 Details of Training Settings

As in [81, 16], we use a pre-trained Faster R-CNN model [30] that is trained
on 4,000 video frames randomly sampled from the training set with object masks
and attribute annotations to generate object proposals for each frame. We
train the language program parser with 1,000 programs for all question types.
All deep modules (concept learner and program executor) are trained using Adam
optimizer for 40 epochs on 8 Nvidia 1080Ti GPUs and the learning rate is set
to $10^{-4}$. The camera matrix is optimized from 20 training videos. We set
$\Delta t=0.004\text{s},D=256,C=64,K=10,S=10$, and $T=128$ for CLEVRER [81]
and $T=20$ for Real-Billiard [65]. In addition to our standard model that
grounds object properties from question-answer pairs, we also train a variant
(VRDP ${\dagger}$) on CLEVRER with an explicit rule-based program executor
[81] and object attribute supervisions (attribute annotation in 4000 frames
learned by the Faster R-CNN model).

For the physical model, we use the L-BFGS optimizer [58] with an adaptive
learning rate to optimize all physical parameters. The optimization terminates
when it reaches a certain number of steps or the loss is less than a certain
value. In all experiments, the number of the optimization step is set to 20.
The loss threshold is set to 0.0005 for the learning of collision-independent
parameters (_i.e._ , initial velocity, initial location, and initial angle),
and 0.0002, 0.001, 0.01 for the optimization of collision-dependent parameters
(mass and restitution) on [0, 40], [0, 80], and [0, 128] frames, respectively.

The training of VRDP can be summarized into three stages. First, we extract
the visual features directly from the video by the visual perception module,
and learn language concepts in the concept learner from all descriptive and
explanatory questions; second, we optimize all physical parameters by using
the perceived trajectories and the learned concepts; third, after obtaining
the physics model, we re-calculate the visual features from the simulated
trajectories and finetune language concept embeddings from all question types,
including predictive and counterfactual questions. During this training
process, the three parts of VRDP are integrated seamlessly and benefit each
other.

###  A.5 Visualizations

We show visualization examples (including failure cases) on CLEVRER [81] in
Fig. 7 and Fig. 8. We also show examples on Real-Billiards [65] in Fig. 9.
These figures show that our model can accurately learn physical parameters
from video and language and perform causal simulations, predictive
simulations, and counterfactual simulations for dynamic visual reasoning. Note
that the billiard table is a chaotic system, and highly accurate long-term
prediction is intractable. For more failure analysis, please refer to our main
paper.

———————————————————————————————————————-

Figure 7: Visualization (1) of the videos and question-answering results of
our VRDP on CLEVRER. We highlighted our failure in red and explained the cause
of it.

———————————————————————————————————————-

Figure 8: Visualization (2) of the videos and question-answering results of
our VRDP on CLEVRER. We highlighted our failure in red and explained the cause
of it. Figure 9: Visualization examples of the videos and question-answering
results of our VRDP on Real-Billiards. Note that the billiard table is a
chaotic system, and highly accurate long-term prediction is intractable.

Type | Operation | Signature
---|---|---
Input Operations | Start | $()\rightarrow\textit{event}$
Returns the special “start” event |
end | $()\rightarrow\textit{event}$
Returns the special “end” event |
Objects | $()\rightarrow\textit{objects}$
Returns all objects in the video |
Events | $()\rightarrow\textit{events}$
Returns all events happening in the video |
UnseenEvents | $()\rightarrow\textit{events}$
Returns all future events happening in the video |
Output Operations | Query_color | $(\textit{object})\rightarrow\textit{color}$
Returns the color of the input object |
Query_material | $(\textit{object})\rightarrow\textit{material}$
Returns the material of the input objects |
Query_shape | $(\textit{object})\rightarrow\textit{shape}$
Returns the shape of the input objects |
Count | $(\textit{objects})\rightarrow\textit{int}$
Returns the number of the input objects/ events | $(\textit{events})\rightarrow\textit{int}$
Exist | $(\textit{objects})\rightarrow\textit{bool}$
Returns “yes” if the input objects is not empty |
| Belong_to | $(\textit{event},\textit{events})\rightarrow\textit{bool}$
| Returns “yes” if the input event belongs to the input event sets |
| Negate | $(\textit{bool})\rightarrow\textit{bool}$
| Returns the negation of the input boolean |
Physics Operations | Counterfactual_simulation | $(\textit{object})\rightarrow\textit{events},\textit{representations}$
Perform simulation with the object removed |
Predictive_simulation | $(\textit{objects})\rightarrow\textit{events},\textit{representations}$
Perform simulation after the video ends |
Apply_heavier | $(\textit{object})\rightarrow\textit{object}$
Assign the object five times its weight before the counterfactual simulation |
Apply_lighter | $(\textit{object})\rightarrow\textit{object}$
Assign the object one-fifth of its weight before the counterfactual simulation |
Object Filter Operations | Filter_static_concept | $(\textit{objects},\textit{concept})\rightarrow\textit{objects}$
Select objects from the input list with the input static concept |
Filter_dynamic_concept | $(\textit{objects},\textit{concept},\textit{frame})\rightarrow\textit{objects}$
Selects objects in the input frame with the dynamic concept |
| Unique | $(\textit{objects})\rightarrow\textit{object}$
| Return the only object in the input list |
Event Filter Operations | Filter_in | $(\textit{events},\textit{objects})\rightarrow\textit{events}$
Select incoming events of the input objects |
Filter_out | $(\textit{events},\textit{objects})\rightarrow\textit{events}$
Select existing events of the input objects |
Filter_collision | $(\textit{events},\textit{objects})\rightarrow\textit{events}$
Select all collisions that involve an of the input objects |
Get_col_partner | $(\textit{event},\textit{object})\rightarrow\textit{object}$
Return the collision partner of the input object |
Filter_before | $(\textit{events},\textit{events})\rightarrow\textit{events}$
Select all events before the target event |
Filter_after | $(\textit{events},\textit{events})\rightarrow\textit{events}$
Select all events after the target event |
Filter_order | $(\textit{events},\textit{order})\rightarrow\textit{event}$
Select the event at the specific time order |
Filter_ancestor | $(\textit{event},\textit{events})\rightarrow\textit{events}$
Select all ancestors of the input event in the causal graph |
Get_frame | $(\textit{event})\rightarrow\textit{frame}$
Return the frame of the input event in the video |
| Unique | $(\textit{events})\rightarrow\textit{event}$
| Return the only event in the input list |

Table 5: All neuro-symbolic operations on the CLEVRER dataset [81]. Our model
contains five types of operations, including input, output, physics, object
filter, and event filter operations. In this table, “order” denotes the
chronological order of an event, _e.g._ “First”, “Second” and “Last”; “static
concept” denotes object-level static concepts like “Blue”, “Cube” and “Metal”;
“dynamic concept” represents object-level dynamic concepts like “Moving” and
“Stationary”; and “representations” denotes the visual features that are
calculated from object trajectories.