# Thermodynamic phase transition of a black hole in rainbow gravity

Zhong-Wen Feng1,2, Email<EMAIL_ADDRESS>Shu-Zheng Yang1,2 Email:
<EMAIL_ADDRESS>1 Physics and Space Science College, China West Normal
University, Nanchong, 637009, China
2 Department of Astronomy, China West Normal University, Nanchong 637009,
China

###### Abstract

In this letter, using the rainbow functions that were proposed by Magueijo and
Smolin, we investigate the thermodynamics and the phase transition of rainbow
Schwarzschild black hole. First, we calculate the rainbow gravity corrected
Hawking temperature. From this modification, we then derive the local
temperature, free energy, and other thermodynamic quantities in an isothermal
cavity. Finally, we analyze the critical behavior, thermodynamic stability,
and phase transition of the rainbow Schwarzschild black hole. The results show
that the rainbow gravity can stop the Hawking radiation in the final stages of
black holes’ evolution and lead to the remnants of black holes. Furthermore,
one can observe that the rainbow Schwarzschild black hole has one first-order
phase transition, two second-order phase transitions, and three Hawking-Page-
type phase transitions in the framework of rainbow gravity theory.

##  I Introduction

The Lorentz symmetry is known as one of the fundamental symmetries in nature,
which produces the standard energy-momentum dispersion relation, i.e.,
$E^{2}-p^{2}=m^{2}$ch1 ; ch2 . However, many works claimed that the standard
energy-momentum dispersion relation would not be held when it nears the Planck
length (or the Planck energy) ch3 ; ch4 ; ch5 ; ch6 ; ch7 . In fact, one of
the intriguing predictions among various quantum gravity theories, such as
loop quantum gravity, loop quantum gravity, and non-commutative geometry, is
the existence of a minimum measurable length that can be identified with the
Planck length ch8 ; ch9 ; ch10 . This idea is supported by many Gedanken
experiments ch11 . Furthermore, the Planck length can be considered as the
dividing line between the quantum and the classical description of spacetime.
Therefore, the Planck length should be taken as an invariant scale. However,
in the special relativity, these features conflict with the Lorentz symmetry
because the Planck scale is not invariant under the linear Lorentz
transformations. To solve this paradoxical situation, the standard energy-
momentum dispersion relation must be changed to the so-called modified
dispersion relation (MDR).

Combining the MDR with the special relativity, Amelino-Camelia proposed the
double special relativity (DSR) ch21 . The DSR is an extension of special
relativity, which contains two fundamental constants: the velocity of light
$c$ and the Planck energy $E_{p}$. Meanwhile, the DSR is also a framework for
encoding properties of flat quantum spacetime ch22 . However, it is observed
that the DSR is typically formulated in momentum space. Therefore, a
nonlinearity of the Lorentz transformation has a very significant effect on
the definition of a dual space in the framework of DSR, i.e., the dual space
is non-trivial ch31 . To overcome this problem, the DSR was generalized to the
curved spacetimes by Magueijo and Smolin. This doubly general theory of
relativity is called as rainbow gravity (or gravity’s rainbow) ch23 . The name
rainbow gravity (RG) comes from the fact that this theory assumes the
spacetime background depending on the energy of a test particle. Therefore,
one should use a family of metrics (rainbow metrics) parameterized by the
ratio ${E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}$ to
describe the background of this spacetime instead of using a single metric.

The rainbow gravity is very important because it can be applied to different
physical systems and modifies many classical theories. For example, by
incorporating the RG with the Friedmann-Robertson-Walker cosmologies, the
possibility of resolving big bang singularity was investigated in ch24 . In
ch26 , with the help of the RG, the authors investigated the modified
Starobinsky model and the inflationary solution to the motion equations, and
the spectral index of curvature perturbation and the tensor-to-scalar ratio
were also calculated. In addition, the deflection of light, photon time delay,
gravitational red-shift, and the weak equivalence principle were also studied
in the framework of the RG ch33 . It is worth mentioning that the RG has a
great effect on the thermodynamics of black holes ch31 ; ch32 ; ch34 ; ch35 ;
ch36 ; ch38 ; ch39 ; ch40 ; ch41 . In 2004, Ali _et al._ showed that the RG
can prevent the black holes from total evaporation, which leads to remnants of
black holes in exactly the same way as done by the generalized uncertainty
principle ch31 ; ch41+ ; ch43 . Hence, the RG may solve the information loss
and naked singularity problems of black holes.

On the other hand, it is well known that black holes have not only
thermodynamic properties, but also rich phase structures and critical
phenomena. Over the past decades, the phase transitions and critical phenomena
of black holes have been widely explored ch43+ . In 1977, Davies found that
the Kerr-Newman black hole exhibits the phenomena of phase transition ch44 .
Later, Pavón discovered a non-equilibrium second order phase transition in the
charged Reissner-Nordström (RN) black hole ch45 . The existence of a certain
phase transition in the asymptotically anti de Sitter (AdS) spacetimes was
proved by Hawking and Page. They demonstrated that the AdS Schwarzschild (SC)
black hole undergoes a phase transition (i.e., Hawking-Page phase transition)
to a thermal AdS space if the temperature reaches a certain value ch46 . This
seminal work has attracted wide attention because it can explained the
confinement/deconfinement phase transition of gauge field in the AdS/CFT
correspondence ch47 . Since then, several works were devoted to investigate
the Hawking-Page phase transition and the critical phenomena for other more
complicated AdS spacetimes ch50 ; ch51 ; ch53 ; ch54 ; ch55 ; ch56 ; ch57 .
From these studies, people found that the phase transitions behavior of
charged black holes is similar to that of the Van der Waals liquid-gas system
ch50 ; ch51 ; ch53 ; ch54 ; ch55 ; ch56 ; ch57 ; ch56+ . In 2012, Kubizňák and
Mann proposed a remarkable new perspective on the relationship between the
phase transition of charged AdS black hole and the Van der Waals liquid-gas
phase transition ch50 . By treating the cosmological constant $\lambda$ as a
thermodynamic pressure $P$ and its conjugate variable as specific volume $V$,
they analyzed the thermodynamic behavior of RN-AdS black hole in the extended
phase space. The results showed that the RN-AdS black hole system has a first-
order small-large black hole phase transition since the free energy $G$
demonstrates a “swallow tail”-type behavior. In addition, Kubizňák and Mann
also studied the “$P-V$” criticality and the critical exponents and proved
that they coincide with those of the Van der Waals system.

From the above discussion, it is interesting to investigate the thermodynamic
criticality and the phase transition of the rainbow black holes. Therefore, in
this letter, we study the thermodynamic criticality and phase transition of
rainbow SC black hole. By utilizing the relation
$E\geq{1\mathord{\left/{\vphantom{1{r_{H}}}}\right.\kern-1.2pt}{r_{H}}}={1\mathord{\left/{\vphantom{1{2GM}}}\right.\kern-1.2pt}{2GM}}$
and a new kind of rainbow functions, we derive the modified thermodynamic
quantities and free energy of the rainbow SC black hole. According to these
modifications, the thermodynamic stability, thermodynamic criticality and
phase transition of the rainbow SC black hole are analyzed. It turns out that
our results are different from those of the standard Hawking-Page phase
transition. However, when the ratio
${E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}$ approaches
zero, our results reduce to the standard forms.

The organization of this paper is as follows. The next section is devoted to
introducing a new kind of rainbow functions, which were proposed by Magueijo
and Smolin. In Section III, using the rainbow functions, we study the
thermodynamics of SC black hole in the context of rainbow gravity, and then
its thermodynamic stability, thermodynamic criticality and phase transition
are discussed. The paper ends with conclusions in Section IV.

##  II A brief introduction on the rainbow functions

To begin with, we briefly review the MDR and the rainbow functions. The
general form of MDR can be expressed as follows:

$\displaystyle
E^{2}f^{2}\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)-p^{2}g^{2}\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)=m^{2},$
(1)

where $E_{p}$ is the Planck energy, and the correction terms
$f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)$
and
$g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)$
are known as rainbow functions, which are responsible for the modification of
the energy-momentum relation in the ultraviolet regime. However, in the limit
${E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}\to 0$, the
rainbow functions satisfy the following relations

$\displaystyle\mathop{\lim}\limits_{{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}\to
0}f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)=1,\mathop{\lim}\limits_{{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}\to
0}g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)=1.$
(2)

Eq. (2) indicates that the MDR reduces to the standard energy-momentum
dispersion relation at low energy scale. It should be mentioned that the
expression of rainbow function is not unique, and people can find a series
expressions for rainbow functions depending on different phenomenological
motivations. According to the varying speed of light theory, Magueijo and
Smolin developed a MDR, which takes the form of
${{E^{2}}\mathord{\left/{\vphantom{{E^{2}}{\left({1-{{\gamma
E}\mathord{\left/{\vphantom{{\gamma
E}{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)^{2}}}}\right.\kern-1.2pt}{\left({1-{{\gamma
E}\mathord{\left/{\vphantom{{\gamma
E}{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)^{2}}}-p^{2}=m^{2}$ ch13 . This
MDR implies that a spacetime has an energy-dependent velocity
$c=\left({1-{{\gamma E}\mathord{\left/{\vphantom{{\gamma
E}{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right){}$. Comparing the MDR with Eq.
(1), the rainbow functions can be fixed as follows:

$f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)=\frac{1}{{1-{{\gamma
E}\mathord{\left/{\vphantom{{\gamma
E}{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}},\quad
g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)=1,$
(3)

where $\gamma$ is the rainbow parameter. Eq. (3) implies that the varying
velocity of light in the RG becomes smaller when the energy of photons
increases. Notably, the bounds on the values of $\gamma$ can be analyzed by
using many theoretical and experimental considerations ch33 ; ch58+ . The
concrete expressions of rainbow functions have a strong effect on the
predictions. Hence, in the subsequent discussions, we scrutinize the
thermodynamics, phase structures, and critical phenomena of rainbow SC black
hole using rainbow functions (3).

##  III Thermodynamics and phase transition of rainbow SC black hole

In this section, we investigate the thermodynamics and phase transition of
rainbow SC black hole using rainbow functions (3). In ch23 ; ch43 , the
authors pointed out that one can construct rainbow spacetimes by replacing
$dt\to{{dt}\mathord{\left/{\vphantom{{dt}{f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}}\right.\kern-1.2pt}{f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}$
for time coordinates and
$dx^{i}\to{{dx^{i}}\mathord{\left/{\vphantom{{dx^{i}}{g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}}\right.\kern-1.2pt}{g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}$
for all spatial coordinates. Therefore, the line element of rainbow SC black
hole is given as follows:

$\displaystyle ds^{2}$
$\displaystyle=-\frac{{1-\left({{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}\right)}}{{f^{2}\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}dt^{2}+\frac{{dr^{2}}}{{\left[{1-\left({{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}\right)}\right]g^{2}\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}$
$\displaystyle+\frac{{r^{2}}}{{g^{2}\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}d\Omega^{2},$
(4)

where $d\Omega^{2}=d\theta^{2}+\sin^{2}\theta d\phi^{2}$ represents the line
element of two-dimensional hypersurfaces. Obviously, the event horizon of
rainbow SC black hole is located at $r_{H}=2GM$. When considering Eq. (2), the
metric of original SC black hole is recovered. According to Eq. (3) and Eq.
(III), the Hawking temperature of rainbow SC black hole can be derived from
the definition of surface gravity as follows: ch62 ; ch63

$\displaystyle
T_{H}^{{\rm{RG}}}=\frac{\kappa}{{2\pi}}=\frac{{g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}{{f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}T_{H}=\frac{1}{{8\pi
GM}}\left({1-\gamma\frac{E}{{E_{p}}}}\right),$ (5)

where $\kappa=-\frac{1}{2}\mathop{\lim}\limits_{r\to
r_{H}}\sqrt{-\frac{{g^{11}}}{{g^{00}}}}\frac{{\left({g^{00}}\right)^{\prime}}}{{g^{00}}}=\frac{{\kappa_{0}g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}{{2\pi
f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}$
is the surface gravity.
$\kappa_{0}={1\mathord{\left/{\vphantom{1{4GM}}}\right.\kern-1.2pt}{4GM}}$ and
$T_{H}={1\mathord{\left/{\vphantom{1{8\pi GM}}}\right.\kern-1.2pt}{8\pi GM}}$
are the original surface gravity and Hawking temperature of SC black hole,
respectively. Obviously, the original surface gravity gets modified in RG.
Meanwhile, it is evident that the rainbow Hawking temperature is very
sensitive to the concrete expression of rainbow functions. Following the
heuristic argument in ch64 ; ch65 , the Heisenberg uncertainty principle (HUP)
$\Delta x\Delta p\geq 1$ still holds in RG. Therefore the HUP can be
translated into a lower bound on the energy, i.e.,
$E\geq{1\mathord{\left/{\vphantom{1{\Delta x}}}\right.\kern-1.2pt}{\Delta
x}}$, where $E$ is the energy of a particle emitted in the Hawking radiation.
From the lower bound on the energy and the uncertainty position $\Delta x$,
one can obtian the following relation: ch31

$E\geq{1\mathord{\left/{\vphantom{1{\Delta x}}}\right.\kern-1.2pt}{\Delta
x}}\approx{1\mathord{\left/{\vphantom{1{r_{H}}}}\right.\kern-1.2pt}{r_{H}}}={1\mathord{\left/{\vphantom{1{2GM}}}\right.\kern-1.2pt}{2GM}}.$
(6)

It should be noted that this bound on the energy plays a key role in the
modification of thermodynamics of rainbow black holes. Substituting Eq. (6)
into Eq. (5), the rainbow Hawking temperature can be rewritten as

$T_{H}^{\rm{RG}}=\frac{1}{{8\pi
GM}}\left({1-\frac{{\gamma}}{{2\sqrt{G}M}}}\right),$ (7)

where we use
$E_{p}=G^{{{-1}\mathord{\left/{\vphantom{{-1}2}}\right.\kern-1.2pt}2}}$ in
natural units. When $\gamma=0$, the rainbow Hawking temperature reaches as
that in the original case. Next, we investigate the entropy of rainbow SC
black hole. Using the first law of black hole thermodynamics, the entropy of
rainbow SC black hole is given by

$\displaystyle S$ $\displaystyle=\int{T_{H}^{-1}dM}$
$\displaystyle=4\sqrt{G}M\pi\left({\sqrt{G}M+\gamma}\right)+2\pi\gamma^{2}{\rm{ln}}\left({2\sqrt{G}M-\gamma}\right).$
(8)

It is clear that the area law of the entropy $S=4\pi M^{2}$ is recovered when
$\gamma=0$. Since the evolution and the phase transition of black holes can be
detected at a finite surface $r$, we need to calculate the local temperature
of rainbow SC black hole. On the basis of the finding from ch66 ; ch59+ ; ch59
, the local temperature in the RG at a finite distance $r$ outside the black
hole can be expressed as follows:

$T_{{\rm{\rm{local}}}}^{\rm{RG}}=T_{H}\left({1-\frac{{2GM}}{r}}\right)^{-\frac{1}{2}}\frac{{g\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}{{f\left({{E\mathord{\left/{\vphantom{E{E_{p}}}}\right.\kern-1.2pt}{E_{p}}}}\right)}}.$
(9)

Considering Eq. (6), Eq. (7), and the rainbow functions (3), the corresponding
local temperature for the observer on the cavity is given by

$T_{{\rm{\rm{local}}}}^{\rm{RG}}=\frac{1}{{8\pi
GM\sqrt{1-\frac{{2GM}}{r}}}}\left({1-\frac{\gamma}{{2\sqrt{G}M}}}\right).$
(10)

Obviously, Eq. (10) is implemented by the redshift factor of the metric. If
one considers the parameter $r$ as an invariable quantity, the critical value
of black hole’s mass, the rainbow parameter, and the local temperature can be
obtained by the following equations

$\left({\frac{{\partial T_{{\rm{\rm{local}}}}}}{{\partial
M}}}\right)_{r}=0,\quad\left({\frac{{\partial^{2}T_{{\rm{\rm{local}}}}}}{{\partial
M^{2}}}}\right)_{r}=0.$ (11)

Using Eq. (11), and setting $r=10$ and $G=1$, the critical mass, the critical
free parameter, and the critical local temperature are

$\displaystyle M_{c}^{\rm RG}=2.36701,\gamma_{c}=1.68082,T_{c}^{\rm
RG}=0.0149398.$ (12)

The local temperature as a function of $\gamma$ is depicted in Fig. 1.

Figure 1: (a) Relationship between the rainbow local temperature and mass for
different $\gamma$. (b) Original and modified local temperature versus mass.
We set $G=1$ and $r=10$.

From Fig. 1, it can be observed that for $\gamma\neq 0$, there is a phase
transition with $0<\gamma<\gamma_{c}$. Therefore, for the convenience of
discussing the critical phenomena and phase transition of rainbow SC black
hole, we set $\gamma=1$ in the following discussions.

By fixing $G=1$ and $r=10$, we plot the rainbow local temperature
$T_{\rm{local}}^{\rm{RG}}$ and original local temperature $T_{\rm{local}}$
versus the mass of SC black hole in Fig. 1. The blue dashed line illustrates
the original local temperature, which is infinite when the mass approaches
zero and $M_{3}$. The minimum value of local temperature $T_{1}^{\prime}$
occurs at $M_{1}^{\prime}$. For the rainbow local temperature case (red solid
line), one can see that the behavior of $T_{\rm{local}}^{\rm{RG}}$ is similar
to that of the original local temperature at $M_{3}$, where the horizon of
black hole is close to $r=2GM_{3}$. This indicates that the black hole is very
hot near the event horizon. As the mass of the black hole decreases, the
rainbow local temperature reduces to $T_{2}$ corresponding to the mass
$M_{2}$, and then increases to its maximum value $T_{1}$ at $M_{1}$. Finally,
the rainbow local temperature becomes zero when the mass of black hole
approaches a finite value $M_{0}$, which leads to a remnant of rainbow SC
black hole, i.e.,
$M_{\rm{res}}=M_{0}={{\gamma\sqrt{G}}\mathord{\left/{\vphantom{{\gamma\sqrt{G}}2}}\right.\kern-1.2pt}2}$.
The values of $\left({M_{1},T_{1}}\right)$, $\left({M_{2},T_{2}}\right)$,
$\left({M_{3},T_{3}}\right)$ and
$\left({M_{1}^{\prime},T_{1}^{\prime}}\right)$ can be numerically obtained if
needed.

Next, using the first law of thermodynamics, the local energy of rainbow SC
black hole within the boundary $r$ is given as follows:

$\displaystyle E_{\rm{local}}^{\rm{RG}}$
$\displaystyle=\int_{M_{0}}^{M}{T_{\rm{local}}^{\rm{RG}}dS^{\rm{RG}}}$
$\displaystyle=\frac{r}{G}\left({\sqrt{1-\frac{{\gamma
G^{{3\mathord{\left/{\vphantom{32}}\right.\kern-1.2pt}2}}}}{r}}-\sqrt{1-\frac{{2GM}}{r}}}\right).$
(13)

When $\gamma\rightarrow 0$, Eq. (III) reduces to the original local energy
$E_{\rm{local}}={{r\left({1-\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}\right)}\mathord{\left/{\vphantom{{r\left({1-\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}\right)}G}}\right.\kern-1.2pt}G}$.
It may be noted that one can investigate the thermodynamic stability of the
black holes from their heat capacity. Hence, using the rainbow local
temperature Eq. (10) and the rainbow local energy Eq. (III), the rainbow heat
capacity at fixed $r$ can be expressed as

$\displaystyle\mathcal{C}^{\rm{RG}}$ $\displaystyle=\left({\frac{{\partial
E_{{\rm{local}}}^{{\rm{RG}}}}}{{\partial
T_{{\rm{local}}}^{{\rm{RG}}}}}}\right)_{r}$
$\displaystyle=\frac{{16M^{3}\pi\left({r-2GM}\right)}}{{2M\left({3GM-r}\right)+\gamma\sqrt{G}\left({2r-5GM}\right)}}.$
(14)

By setting $\mathcal{C}^{\rm{RG}}=0$, the remnant mass is
$M_{{\rm{res}}}={{\gamma\sqrt{G}}\mathord{\left/{\vphantom{{\gamma\sqrt{G}}2}}\right.\kern-1.2pt}2}$.
Obviously, Eq. (III) reduces to the original heat capacity $\mathcal{C}=8\pi
GM^{2}(r-2GM)/(3GM-r)$ when $\gamma$ vanishes. The heat capacity associated
with the mass of SC black hole is plotted in Fig. 2.

Figure 2:  Original and rainbow heat capacity versus mass for $G=1$ and
$r=10$.

In Fig. 2, the red solid line for rainbow heat capacity
$\mathcal{C}^{\rm{RG}}$ vanishes at $M_{0}$, which is different from the blue
dashed line for the original heat capacity $\mathcal{C}$ that reaches zero
when $M\to 0$. It is well known that the black hole is stable if its heat
capacity is positive, while the black hole is unstable if it has negative heat
capacity. Therefore, one can find that the rainbow heat capacity has two
stable regions $M_{0}\leq M\leq M_{1}$ and $M_{2}\leq M\leq M_{3}$, and one
unstable region $M_{1}\leq M\leq M_{2}$. However, the blue dashed line for the
original case has only one stable region $M_{1}^{\prime}\leq M\leq M_{3}$, and
one unstable region $M\leq M_{1}^{\prime}$. Interestingly, the rainbow heat
capacity diverges at the points where the temperature reaches the maximum
value $M_{1}$ and the minimum value $M_{2}$, which indicates that two second-
order phase transitions exist in the canonical ensemble.

According to the discussions on the local temperature and the heat capacity,
we classify the rainbow SC black hole into three branches depending on its
mass scale. The range, state, and stability of the three branches of rainbow
SC black hole are presented in Table. 1.

Table 1:  Region, state, and stability of the three branches of rainbow SC black hole. Branches | Region | State | Stability
---|---|---|---
1 | $M_{0}\leq M\leq M_{1}$ | small | stable
2 | $M_{1}\leq M\leq M_{2}$ | intermediate | unstable
3 | $M_{2}\leq M\leq M_{3}$ | large | stable

From Table. 1, an additional intermediate black hole (IBH) can be found in the
system, which never appears in the original case. Obviously, the stability of
the three black holes depend on their heat capacities. Therefore, it is
obvious that the small black hole (SBH) and the large black hole (LBH) are
stable, showing that they can survive for a long time within the frame of the
RG theory. On the contrary, the IBH is unstable because its heat capacity is
negative, which implies that it would decay into the SBH or LBH quickly. In
addition, it should be noted that the stability of the black holes can be
known from their free energy.

In order to obtain more details on the thermodynamic phase transition and
stability of the rainbow SC black hole, it is necessary to investigate the
free energy of rainbow SC black hole in a cavity. In ch67 ; ch68 , the free
energy is defined as follows:

$F_{\rm{on}}=E_{\rm{local}}-T_{\rm{local}}S.$ (15)

Substituting Eq. (III), Eq. (10) and Eq. (III) into Eq. (15), the rainbow free
energy is given by

$\displaystyle F_{\rm{on}}^{\rm{RG}}$
$\displaystyle=\frac{r}{G}\left({\sqrt{1-\frac{{\gamma
G^{{3\mathord{\left/{\vphantom{32}}\right.\kern-1.2pt}2}}}}{r}}-\sqrt{1-\frac{{2GM}}{r}}}\right)$
$\displaystyle-\frac{{\left({2M-\sqrt{G}\gamma}\right)\chi}}{{8M^{2}\sqrt{1-\frac{{2GM}}{r}}}},$
(16)

where
$\chi={2M\left({M+\sqrt{G}\gamma}\right)+G\gamma^{2}{\rm{ln}}\left({{{2M}\mathord{\left/{\vphantom{{2M}{\sqrt{G}}}}\right.\kern-1.2pt}{\sqrt{G}}}-\gamma}\right)}$.
When $\gamma=0$, Eq. (III) is reduced to the original free energy
$F_{\rm{on}}=r{{\left({1-\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}\right)}\mathord{\left/{\vphantom{{\left({1-\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}\right)}G}}\right.\kern-1.2pt}G}-{M\mathord{\left/{\vphantom{M{2\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}}}\right.\kern-1.2pt}{2\sqrt{1-{{2GM}\mathord{\left/{\vphantom{{2GM}r}}\right.\kern-1.2pt}r}}}}$.
For further investigation of the phase transition of rainbow SC black hole, we
plot Fig. 3.

Figure 3: (a) Variation in free energy of rainbow SC black hole with local
temperature for different $\gamma$. (b) and (c) Original free energy of SC
black hole and rainbow SC black hole as a function of local temperature. Here,
we choose $G=1$ and $r=10$.

Fig. 3 shows $F_{\rm{on}}^{\rm RG}$ curves with $T_{\rm{local}}^{\rm RG}$ for
different $\gamma$, the value of rainbow parameter $\gamma$ decreases from top
to bottom. The swallow tail structure appears when the rainbow parameter
$\gamma$ is smaller than the critical value $\gamma_{c}$, which indicates that
there is a two-phase coexistence state. The results are consistent with the
profile of $T_{\rm{local}}^{\rm RG}-M$ in Fig. 1.

Fig. 3 and Fig. 3 show the free energy of original SC black hole and rainbow
SC black hole, respectively. First, let us focus on the original case for
$\gamma=0$. It is found that the local temperature has a minimum value
$T_{1}^{\prime}$, below which no black hole solution exists. Moreover, the
free energy vanishes when $M\to{\rm{0}}$ because the vacuum state is Minkowski
spacetime. Hence, $F_{\rm{on}}^{{\rm{HFS}}}$ represents the free energy of the
hot flat space (HFS). The Hawking-Page transition occurs at $T_{c}^{\prime}$.
For $T<T_{c}^{\prime}$, the free energies of both unstable small black hole
and stable large black hole are higher than $F_{\rm{on}}^{{\rm{HFS}}}$, this
implies that the HFS is more probable than the unstable small black hole and
stable large black hole. However, when $T>T_{c}^{\prime}$, the free energy of
unstable small black hole is higher than $F_{\rm{on}}^{{\rm{HFS}}}$, while the
free energy of stable large black hole is lower than
$F_{\rm{on}}^{{\rm{HFS}}}$. This indicates that the stable large black hole is
more probable than the HFS. Therefore, one can see that the radiation collapse
to the stable large black hole by phase transition, and the unstable small
black hole eventually decays into the large black hole thermodynamically for
$T>T_{c}^{\prime}$.

Then, turn to look at Fig. 3. For non-zero $\gamma$, the behavior of the
rainbow free energy is different from that of the original case:

(i) Due to the presence of black hole remnants, we need use the hot curved
space (HCS) to investigate phase transition of rainbow SC black hole instead
of using the HFS ch60+ . Therefore, we analyse the phase transition between
the three black hole states and the HCS via studying free energies of rainbow
SC black hole. In addation, the HCS becomes zero when $M\rightarrow M_{\rm
res}$, i.e., $F_{\rm{on}}^{{\rm{HCS}}}=0$.

(ii) The small-intermediate black hole transition occurs at the inflection
point $T_{1}$ corresponding to the mass $M_{1}$, the IBH and LBH meet at
$T_{2}$ corresponding to the mass $M_{2}$. Interestingly, three Hawking-Page-
type critical points can be found easily, which correspond to the temperatures
$T_{c}^{(1)}$, $T_{c}^{(2)}$, and $T_{c}^{(3)}$, respectively, as shown in
Fig. 3, whereas only one Hawking-Page phase transition point can be found in
original case, as shown in Fig. 3.

(iii) A first-order phase transition occurs because the $F_{\rm{on}}$
demonstrates the characteristic swallow tail behavior. It is reminiscent of
the “free energy-Hawking temperature” relation of charged AdS black hole, as
described in ch51 ; ch53 ; ch54 ; ch55 ; ch56 ; ch57 . The intersection point
between the red solid line and the blue solid line is the first-order phase
transition point that correspond to the temperature $T_{c}^{\rm First-order}$.

(iv) For $T_{0}<T<T_{c}^{(1)}$, the free energies of the three black holes are
higher than those of the HCS. Therefore, the HCS is more probable in this
region. Meanwhile, it can be observed that the IBH and LBH collapse into the
SBH for $T_{0}<T<T_{c}^{\rm First-order}$ because
$F_{\rm{on}}^{{\rm{SBH}}}<F_{\rm{on}}^{{\rm{LBH}}}<F_{\rm{on}}^{{\rm{IBH}}}$,
and the IBH and SBH collapse into the LBH for $T_{c}^{\rm First-
order}<T<T_{c}^{\rm(1)}$ since
$F_{\rm{on}}^{{\rm{LBH}}}<F_{\rm{on}}^{{\rm{SBH}}}<F_{\rm{on}}^{{\rm{IBH}}}$.
However, the free energies of the LBH, SBH and IBH decrease below
$F_{\rm{on}}^{\rm HCS}$ one by one when the temperature is higher than
$T_{c}^{(1)}$. For $T_{c}^{(1)}<T<T_{c}^{(2)}$, the relation of free energies
obey
$F_{\rm{on}}^{{\rm{LBH}}}<F_{\rm{on}}^{{\rm{HCS}}}<F_{\rm{on}}^{{\rm{SBH}}}<F_{\rm{on}}^{{\rm{IBH}}}$.
It then changes to
$F_{\rm{on}}^{{\rm{LBH}}}<F_{\rm{on}}^{{\rm{SBH}}}<F_{\rm{on}}^{{\rm{HCS}}}<F_{\rm{on}}^{{\rm{IBH}}}$
for $T_{c}^{(2)}<T<T_{c}^{(3)}$. Finally, the relation becomes
$F_{\rm{on}}^{{\rm{LBH}}}<F_{\rm{on}}^{{\rm{SBH}}}<F_{\rm{on}}^{{\rm{IBH}}}<F_{\rm{on}}^{{\rm{HCS}}}$
for $T_{c}^{(3)}<T<T_{1}$. Therefore, the stable SBH and unstable IBH decay
into the stable LBH eventually for $T_{c}^{(1)}<T<T_{1}$.

##  IV Discussion

In this paper, using the rainbow functions that were proposed by Magueijo and
Smolin, we studied the quantum corrections to the thermodynamics and the phase
transitions of SC black hole. According to the rainbow surface gravity and the
uncertainty principle, we calculated modified thermodynamic quantities such as
the rainbow Hawking temperature, rainbow entropy, rainbow heat capacity, and
rainbow local temperature. From these modifications, we then derived the
critical points of the black hole thermodynamic ensemble, and analyzed the
thermodynamic stability and phase transition of the rainbow SC black hole.

Comparing our results with the original thermodynamic quantities and Hawking-
Page phase transition of SC black hole, many different findings were obtained.
First, the rainbow gravity can stop the Hawking radiation in the final stages
of black holes’ evolution and lead to remnants, which is in agreement with the
predictions of the generalized uncertainty principle. Second, from Eq. (III),
one can obtain two second-order phase transitions because the heat capacity
exhibits two divergences: at $M_{1}$ and $M_{2}$. Third, according to the
“$F_{\rm{on}}^{\rm RG}-T_{\rm{local}}^{\rm RG}$” plane, it was found that an
unstable intermediate black hole, which never appears in the original SC black
hole, interpolates between the stable small black hole and stable large black
hole in the rainbow SC black hole. Fourth, $F_{\rm{on}}^{\rm RG}$ surface
demonstrated the characteristic swallow tail behavior, which implies that the
rainbow SC black hole system has a first-order transition. This behavior is
reminiscent of the “free energy-Hawking temperature” relation in AdS
spacetimes. Finally, three Hawking-Page-type critical points were found for
rainbow SC black hole, as shown in Fig. 3, whereas only one Hawking-Page
critical point was found for the original case (see Fig. 3) and two Hawking-
Page critical points were observed in ch59 . For $T_{0}<T<T_{c}^{(1)}$, the
IBH and LBH will collapse into the stable SBH since the free energy of SBH is
lower than those of the IBH and LBH. However, for $T_{0}<T<T_{c}^{(1)}$, the
SBH and IBH decay into the stable LBH because the free energy of LBH is lower
than those of the IBH and SBH.

## Acknowledgements

The authors thank Yongwan Gim and the anonymous referees for helpful
suggestions and enlightening comments, which helped to improve the quality of
this paper. This work was supported by the Natural Science Foundation of China
(Grant No. 11573022).

## References

  * (1) C. M. Will, Living Rev. Rel. 9 (2006) 3. [arXiv:gr-qc/0510072]
  * (2) V. A. Kostelecký, N. Russell, Rev. Mod. Phys. 83 (2011) 11. [arXiv:0801.0287]
  * (3) G. ’t Hooft, Class. Quantum Gravity 13 (1996) 1023. [arXiv:gr-qc/9601014]
  * (4) V. A. Kostelecký, M. Mewes, Phys. Rev. Lett. 97 (2006) 140401. [arXiv:hep-ph/9703464]
  * (5) R. Iengo, J. G. Russo, M. Serone, JHEP 0911 (2009) 020. [arXiv:0906.3477]
  * (6) H. Belich, K. Bakke, Phys. Rev. D 90 (2014) 025026. [arXiv:1407.6279]
  * (7) J. Alfaro, P. González, R. Ávila, Phys. Rev. D 91 (2015) 105007. [arXiv:1504.04222]
  * (8) D. Amati, M. Ciafaloni, G. Veneziano, Phys. Lett. B 216 (1989) 41. [DOI:10.1016/0370-2693(89)91366-X]
  * (9) R. Gambini, J. Pullin, Phys. Rev. D 59 (1999) 124021. [arXiv:gr-qc/9809038]
  * (10) F. Girelli, E. R. Livine, D. Oriti, Nucl. Phys. B 708 (2005) 411. [arXiv:gr-qc/0406100]
  * (11) F. Scardigl, Phys. Lett. B 452 (1999) 39. [arXiv:gr-qc/0406100]
  * (12) G. Amelino-Camelia, Int. J. Mod. Phys. D 11 (2002) 35. [arXiv:gr-qc/0012051]
  * (13) R.Garattini, E. N. Saridakis, Int. J. Mod. Phys. D 25 (2016) 1650101. [arXiv:1411.7257]
  * (14) A. F. Ali, Phys. Rev. D 89 (2014) 104040. [arXiv:1402.5320]
  * (15) J. Magueijo, L. Smolin, Class. Quantum Gravity 21 (2004) 1725. [arXiv:gr-qc/0305055]
  * (16) A. Awad, A. F. Ali, B. Majumder, JCAP 10 (2013) 052. [arXiv:1308.4343]
  * (17) A. Chatrabhuti, V.Yingcharoenrat, P. Channuie, Phys. Rev. D 93 (2016) 043515. [arXiv:1510.09113]
  * (18) A. F. Ali, M. Faizal,B. Majumder, Europhys. Lett. 109 (2015) 20001. [arXiv:1406.1980]
  * (19) A. F. Ali, M. Faizal, M. M. Khalil, JHEP 1412 (2014) 159. [arXiv:1409.5745]
  * (20) A. F. Ali, M. Faizal, M. M. Khalil, Nucl. Phys. B 894 (2015) 341. [arXiv:1410.5706]
  * (21) Y. Gim, W. Kim, JCAP 05 (2015) 002. [arXiv:1501.04702]
  * (22) S. H. Hendi, M. Faizal, Phys. Rev. D 92 (2015) 044027. [arXiv:1506.08062]
  * (23) S. H. Hendi, M. Faizal, B. E. Panah, S. Panahiyan, Eur. Phys. J. C 76 (2016) 296. [arXiv:1508.00234]
  * (24) S. H. Hendi, B. Eslam Panah, S. Panahiyan, Phys. Lett. B 769 (2017) 191. [arXiv:1602.01832].
  * (25) S. H. Hendi, S. Panahiyan, B. E. Panah, M. Momennia, Eur. Phys. J. C 76 (2016) 150. [arXiv:1512.05192]
  * (26) Y. Gim, W. Kim, Eur. Phys. J. C 76 (2016) 166. [arXiv:1509.06846]
  * (27) Z. W. Feng, L. Zhang, X. T. Zu, Mod. Phys. Lett. A 29 (2014) 1450123. [DOI:10.1142/S0217732314501235]
  * (28) Z. W. Feng, H. L. Li, X. T. Zu, S. Z. Yang, Eur. Phys. J. C 76 (2016) 212. [arXiv:1604.04702]
  * (29) D. Roychowdhury, arXiv:1403.4356 [hep-th].
  * (30) P. C. W. Davies, Class. Quantum Gravity 6 (1989) 1909. [DOI:10.1088/0264-9381/6/12/018]
  * (31) D. Pavón, Phys. Rev. D 43 (1991) 2495. [DOI:10.1103/PhysRevD.43.2495]
  * (32) S. W. Hawking, D. N. Page, Commun. Math. Phys. 87 (1983) 577. [DOI:10.1007/BF01208266]
  * (33) E. Witten, Adv. Theor. Math. Phys. 2(1998) 253. [arXiv:hep-th/9802150]
  * (34) D. Kubizňák, R. B. Mann, JHEP 1207 (2012) 033. [arXiv:1205.0559].
  * (35) S. Gunasekaran, R. B. Mann, JHEP 2012 (2012)110. [arXiv:1208.6251]
  * (36) R. G. Cai, Y. P. Hu, Q. Y. Pan, Y. L. Zhang, Phys. Rev. D 91 (2015) 024032. [arXiv:1409.2369]
  * (37) M. H. Dehghani, S. Kamrani, A. Sheykhi, Phys. Rev. D 90 (2014) 104020. [arXiv:1505.02386]
  * (38) S. W. Wei, Y. X Liu, Phys. Rev. Lett. 115 (2015) 111302. [arXiv:1502.00386]
  * (39) X. X. Zeng, H. B. Zhang, L. F. Li, Phys. Lett. B 756 (2016) 170. [arXiv:1511.00383]
  * (40) X. X. Zeng, L. F. Li, Phys. Lett. B 764 (2017) 100. [arXiv:1512.08855].
  * (41) A. Dehyadegari, A. Sheykhi, A. Montakhab, Phys. Lett. B 768 (2017) 235. [arXiv:1607.05333]
  * (42) J. Magueijo, L. Smolin, Phys. Rev. D 67 (2003) 044017. [arXiv:gr-qc/0207085]
  * (43) Z. W. Feng, S. Z. Yang, H. L. Li, X. T. Zu, Phys. Lett. B 768 (2017) 81. [arXiv:1610.08549]
  * (44) S. W. Hawking, Commun. Math. Phys. 43 (1975) 199. [DOI:10.1007/BF02345020]
  * (45) S. W. Hawking, Phys. Rev. D 13 (1976) 191. [DOI:10.1103/PhysRevD.13.191]
  * (46) R. J. Adler, P. Chen, D. I. Santiago, Gen. Relativ. Gravit. 33 (2001) 2101. [arXiv:gr-qc/0106080]
  * (47) M. Cavaglia, S. Das, Class. Quantum Gravity 21 (2004) 4511. [arXiv:hep-th/0404050]
  * (48) R. C. Tolman, Phys. Rev. 35 (1930) 904. [DOI:10.1103/PhysRev.35.904]
  * (49) J. Man, H. Cheng, Phys. Rev. D 87 (2013) 044002. [arXiv:1301.2739]
  * (50) W. Kim, Y. Kim, Phys. Lett. B 718 (2012) 687. [arXiv:1207.5318]
  * (51) Y. Gim, W. Kim, JCAP 10 (2014) 003. [arXiv:1406.6475]
  * (52) R. G. Cai, Phys. Rev. D 65 (2002) 084014. [arXiv:hep-th/0109133]
  * (53) Y. S. Myung, Phys. Lett. B 690 (2010) 534. [arXiv:1002.4448]


# Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video
Question Answering

Xingrui Wang$1$ Wufei Ma$1$ Angtian Wang$1$ Shuo Chen$2$
Adam Kortylewski$3,4$ Alan Yuille$1$
$1$ Johns Hopkins University $2$ Tsinghua University
$3$ Max Planck Institute for Informatics $4$ University of Freiburg
{xwang378, wma27, angtianwang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

For vision-language models (VLMs), understanding the dynamic properties of
objects and their interactions within 3D scenes from video is crucial for
effective reasoning. Cognitive science research has suggested that humans are
adept at understanding these properties by constructing 3D and temporal (4D)
representations of the world. In this work, we introduce a video question
answering dataset SuperCLEVR-Physics that focuses on the dynamics properties
of objects. We concentrate on physical concepts — velocity, acceleration, and
collisions within 4D scenes, where the model needs to fully understand these
dynamics properties and answer the questions built on top of them. From the
evaluation of a variety of current VLMs, we find that these models struggle
with understanding these dynamic properties due to the lack of explicit
knowledge about the spatial structure in 3D and world dynamics in time
variants.

To demonstrate the importance of an explicit 4D dynamics representation of the
scenes in understanding world dynamics, we further propose NS-4Dynamics, a
Neural-Symbolic model for reasoning on 4D Dynamics properties under explicit
scene representation from videos. Using scene rendering likelihood combining
physical prior distribution, the 4D scene parser can estimate the dynamics
properties of objects over time to and interpret the observation into 4D scene
representation as world states. By further incorporating neural-symbolic
reasoning, our approach enables advanced applications in future prediction,
factual reasoning, and counterfactual reasoning. Our experiments show that our
NS-4Dynamics suppresses previous VLMs in understanding the dynamics properties
and answering questions about factual queries, future prediction, and
counterfactual reasoning. Moreover, based on the explicit 4D scene
representation, our model is effective in reconstructing the 4D scenes and re-
simulate the future or counterfactual events. Our dataset and model will be
released at https://github.com/XingruiWang/SuperCLEVR-Physics

##  1 Introduction

Visual question answering (VQA) is commonly regarded as a comprehensive method
to assess how well machine learning models can identify objects, understand
their relationships, and perform reasoning over the scenes. When it comes to
dynamic scenes in video question answering (VideoQA), we argue that a model
should incorporate a dynamic representation that captures how objects change
and interact over time. This is partly motivated by studies in cognitive
science hamrick2016inferring ; ullman2018learning  which found that humans
excel at understanding the dynamics and interactions in the physical world.
This allows humans to perform complex tasks such as planning and manipulation
in the 3D world.

However, current video understanding models in the computer vision community
focus on VQA tasks in large benchmark datasets of real images. These datasets
lack annotations of 3D/4D properties (which are notoriously hard to annotate),
which makes it impractical to train these models to estimate 3D/4D
annotations. Although these LVMs may have the capability of learning knowledge
of 3D/4D properties, perhaps encoded implicitly, it is impossible to test this
on these datasets. This applies also to the most recent algorithms which
exploit the enormous strengths of large language models.

Cognitive scientists have developed an impressive variety of synthetic
datasets for physical reasoning which enable the development and testing of
models to understand dynamics, physical properties, and interactions. This has
led in many important conceptual findings yi2019clevrer ; chen2022comphy ;
bear2021physion ; tung2023physion++ ; ates-etal-2022-craft ; patel2022cripp ;
girdhar2020cater ; zheng2024contphy . Due to the nature of these datasets and
in particular the simplicity of the objects and environments, it is nontrivial
to extend their models to the complexity of real world. As listed in Table 1,
their environments are limited either to 2D flat planes or overly simplified
objects, leading to a big gap between controlled synthetic datasets and real
image datasets that computer vision community favors.

Table 1: Comparisons with previous dynamical reasoning datasets. From left to right: 3D space, physical dynamics, language understanding, realistic objects. | 3D | Phy. | Lang. | Real
---|---|---|---|---
CRAFT ates-etal-2022-craft  | ✗ | ✗ | ✓ | ✗
CLEVRER yi2019clevrer  | ✗ | ✗ | ✓ | ✗
Comphy chen2022comphy  | ✗ | ✓ | ✓ | ✗
Physion bear2021physion  | ✓ | ✗ | ✗ | ✗
Physion++ tung2023physion++  | ✓ | ✓ | ✗ | ✗
Ours | ✓ | ✓ | ✓ | ✓

To address the understanding of the 4D dynamic in Video Question Answering
(VideoQA) tasks, we introduce SuperCLEVR-Physics. This dataset focuses on the
dynamics and interactions of objects in 3D space — velocities, accelerations,
and collisions, and design questions around these properties in the factual,
future prediction, and counterfactual scenario. As opposed to the simple
objects in previous datasets yi2019clevrer ; li2023super , we improve the
realism of the appearances of objects and backgrounds with new materials and
textures.. For video generation, we first design the dynamics properties in
the initial state and use PyBullet coumans2016pybullet  to simulate the motion
and interaction in the following frames. Then we use Blender blender  to
render the videos. Questions are formulated around dynamic properties and
interactions from three perspectives: factual, predictive, and counterfactual.
Factual questions are answered using the full video frames, predictive
questions are answered based only on the initial portion of the video, and
counterfactual questions rely on modified properties that differ from the
original video. Fig.1 presents examples of video frames and their
corresponding factual, predictive, and counterfactual questions. As a
compositional reasoning dataset, we also provide new operation programs to
answer these questions step by step using a 4D dynamic scene representation.
We test the existing VideoQA models and their variants on SuperCLEVR-Physics
and find they are struggling to understand these dynamic features, due to the
lack of explicit knowledge of the 3D world or 3D dynamics of objects.

This motivates us to develop NS-4Dynamics, a neural-symbolic model that
reasons about dynamics by first estimating explicit 4D scene representation.
Our NS-4Dynamics consists of two key modules, a dynamic 3D compositional model
following by a symbolic reasoning module. The dynamic 3D compositional model
learns a 3D generative model ma2022robust ; NEURIPS2023_b783c44b  of the scene
and captures the dynamics of objects with a physical prior to output a 4D
parse of the video.. Then based on the compositional 4D scene representation,
the symbolic reasoning module executes the reasoning program and obtains
answers step-by-step with high interpretability.

We demonstrate that NS-4Dynamics outperforms previous state-of-the-art models
on video question answering regarding the dynamic properties of objects,
including velocity, acceleration, and collision events on our SuperCLEVR-
Physics. Furthermore, with an explicit 4D scene representation, our model not
only infers objects’ dynamic properties and reconstruct the scenes, but also
predicts future states and reasons about counterfactual conditions.

Our contributions :(1) First, we introduce a VideoQA dataset regarding the
dynamics properties of objects (velocity, acceleration) and multi-object
interactions (collisions). We also improve the visual complexity of objects
and backgrounds with new color or textures for materials, and further design
questions about these properties in factual, future prediction, and
counterfactual scenarios. (2) Second, we propose NS-4Dynamics, a neural-
symbolic model that first reconstructs the 4D scene with a dynamic 3D
generative model with physical priors as the perception module, and then
reasons about dynamics over the explicit 4D scene representation. Third, we
demonstrate that NS-4Dynamics can outperform previous state-of-the-art models
VLMs on SuperCLEVR-Physics, showing the advantages of our approach to infer
objects’ dynamic properties, reconstruct the scenes, imagine future states,
and reason about counterfactual conditions.

Figure 1: Examples of video frames and their corresponding factual,
predictive, and counterfactual questions.

##  2 Related Work

#### Video question answering.

Video questions answering (VideoQA) is a challenging task because models must
not only detect and identify objects from static images, but also track and
infer objects’ changes and interactions over a sequence of frames. A number of
VideoQA datasets annotate question-answer pairs on natural videos 8017608 ;
yang2022learningta ; majumdar2023openeqa . However these datasets are not
suitable for studying dynamic physical properties as natural videos contain
limited object interactions and the annotated questions are hindered by
shortcuts and biases. Another line of works focused on physical reasoning in
simulated environments yi2019clevrer ; chen2022comphy ; ding2021dynamic ;
tung2023physion++ . However, these datasets are either built in restricted
settings yi2019clevrer ; chen2022comphy  or lack of real dynamics and forces
common in real world tung2023physion++  (see Table 1). In order to further
challenge and explore the limitations of current video-text models on dynamics
reasoning, we develop SuperCLEVR-Physics with improved realism for both video
quality and physics simulation.

#### Video-text models.

With the availability web-scale image-text or video-text paired datasets
schuhmann2022laion ; bain2021frozen , recent video-text models
wang2022internvideo ; lin2023video ; xu2024pllava  adopted heavy multi-modal
pretraining and achieved improved results on a wide range video understanding
tasks. However, these models often exploit biases and shortcuts for dynamic
reasoning, failing to capture the physical intrinsics that drive object
movements and interactions. Our NS-4Dynamics incorporated a 4D dynamic scene
representation in our scene parser, allowing compositional reasoning of
various physical events in the video sequence.

#### Physical scene understanding.

Understanding the physical events in a dynamic 4D scene or inferring the
future state is hard as it requires a model to apply physical principles to
explain the observed events or simulate the future outcomes. Previous studies
explored physics engines for simulation and learning kadambi2023incorporating
; battaglia2013simulation , or integrating a differentiable physics engine in
deep learning models wu2015galileo ; wu2017learning ; kyriazis2013physically .
Another line of works learned physical properties from a compositional scene
hamrick2016inferring ; ullman2018learning ; ding2021dynamic ; chen2022comphy ;
zheng2024contphy . However, previous methods were limited to simple scenes
where physics models provide good approximations of the world. We extend our
scope to more realistic scenes with real-world objects and complex
simulations, and enable physical scene reasoning with our dynamic 3D
compositional reasoning.

#### 3D compositional models.

Our work is also related to previous 3D compositional models for image
classification jesslen2023robust , 6D pose estimation ma2022robust , and
3D-aware visual questions answering wang2023daware . 3D compositional models
learns a probabilistic generative model of feature activations of a scene and
enables scene understanding with analysis-by-synthesis. We extended previous
3D compositional models with a dynamic scene representation, allowing us to
reconstruct object trajectories and reason about the physical events.

##  3 Dataset

To study the understanding of dynamics in visual question answering, we
present the SuperCLEVR-Physicsdataset. This dataset is designed to evaluate
the understanding of physical object dynamics — including velocity,
acceleration, and collision events in Video Question Answering (VideoQA). Our
work builds on li2023super , which introduced complex scenes with multiple
object categories to image-based VQA and wang20243d  which expanded the former
with 3D aware scene understanding questions. We follow the established setting
in terms of objects classes and shapes, and extend this line of work to video
data, where we include the dynamic properties of objects to build a 4D scene
representation.

In this section, we first introduce the three key components for designing the
video data in our proposed dataset: (1) A more realistic appearance for the
objects and the background in Sec.3.1; (2) The introduction of object dynamics
over time in Sec.3.2; and (3) the introduction of dynamic events caused by
object interactions in Sec.3.3. Building on this setting, we will then discuss
the physics simulation and video rendering in Sec.3.4 and the automated
generation of questions in Sec.3.5.

Figure 2: (a) We keep the same shape from SuperCLEVR but repaint the color of
the material for objects to make it more realistic. (b) We project the
environment map from HDRI into a dome shape. (c) After assigning dynamics
properties, we use physical simulation (PyBullet) and render (Blender) to
generate the video data, where objects are moving and interacting in the
dynamic 3D space.

###  3.1 Realistic Appearance

#### Objects appearances

We use the same object shapes as in SuperCLEVR, including 5 vehicle categories
(car, plane, bicycle, motorbike, bus) with 18 sub-types. However, like other
synthetic datasets in VQA yi2019clevrer , the original SuperCLEVR colors are
simple monochrome. To achieve higher realism while retaining distinct color
labels, we replace the 3D model material with a more realistic colormap by
following steps. (1) We generate a new colormap for all color labels (gray,
red, brown, yellow, green, cyan, blue, and purple), with each color including
six randomly generated HEX codes created by an LLM. (2) We then apply these
new colors to the main body of the objects. For example, as shown in Figure 2
(a), we choose the largest material area, like a car’s body, and apply a
randomly selected HEX code from the colormap. For other parts, we use colors
that enhance realism: transparent for windows, black for wheels, and red for
lights.

#### Backgrounds

We use highly realistic background images and lights in our dataset. We
utilize real HDRI images (https://polyhaven.com/hdris), containing real
captured images with environment maps in 509 different scenes. We map the
image as material into a dome shape as background to introduce more
variability into data (see Fig. 2 (b)).

###  3.2 Dynamics Properties

A critical feature of our dataset is the dynamic scene properties designed to
test the 4D physical scene understanding of vision-language models. We define
these properties to accurately reflect real-world physics, thereby creating a
realistic and controlled setting for VideoQA challenges. This section details
how we establish and manipulate each dynamic feature to create diverse and
complex scenarios for models to interpret. Detailed settings are provided in
the Appendix.

#### Position and Orientation.

(1) Position: Objects are placed randomly in the scene. we use a beta
distribution along the x and y axes to make then closer to the edge. The
z-axis, which controls height relative to the ground, is set differently to
classes. For airplanes, they have the option to start in the air, where the
z-axis positions range from 2 to 4.5; otherwise, they are positioned on the
ground, same as the other classes. (2) Orientation: All objects are initially
facing the center of the scene and we add a random noise for variability.

#### Velocity.

Each object begins with a predefined velocity state — static (0 m/s), slow (3
m/s), or fast (6 m/s) — setting the initial conditions for movement and
subsequent interactions.

#### Internal Forces and Acceleration.

In our dataset, the velocities of objects are further influenced by both
internal and external forces, which directly impact their acceleration and
subsequent motion, akin to real-world scenarios as follows. (1) Engine Force
for Vehicles: Vehicles are equipped with engines that provide forward
acceleration. We apply a uniform acceleration of 1 m/s2 to vehicles with an
active engine. We also set speed limits for objects with different velocity
states — 8 m/s for "fast" objects and 5.5 m/s for others. (2) Floating Force
for Airplanes: Airplanes may have a floating force to either maintain a static
height or descend due to gravity without the floating.

#### External Forces and other physical properties.

The environment also includes external forces such as gravity, friction, and
elastic forces during collisions, along with additional physical properties of
the objects. For further details, please refer to the Appendix.

###  3.3 Collision Events

In our dataset, we extend prior benchmarks yi2019clevrer ; chen2022comphy  by
focusing on more complex shapes and motions in full 3D space, which makes the
motions of objects harder to track after the collision. The collision event
will be calculated by a physics simulator. For each collision, we will record
the objects involved and the frame index, allowing us to develop questions
from these annotations.

###  3.4 Physical Simulation and Rendering

The video data is generated using the pipeline from the Kubric greff2021kubric
package, which uses the physical engine (PyBullet coumans2016pybullet ) for
simulation and Blender blender  to render videos frame by frame. In our
dataset, each video is 2000 ms long, resulting in 120 frames with a
corresponding frame rate of 60. In the final dataset, we generate 1000 videos
for training, 100 for validation and 100 for testing.

###  3.5 Question Generation

To study 3D dynamic reasoning in visual question-answering tasks, we design
the questions for the SuperCLEVR-physics dataset to address three aspects:
factual, predictive, and counterfactual reasoning, each focused on dynamics
like velocity, acceleration, and collision events. The questions are generated
by predefined templates with functional programs. We have 12752 questions
totally, where 7850 are factual, 2750 are predictive, 989 are counterfactual.
We provide all the question templates and the corresponding programs in the
Appendix.

#### Factual Questions

These questions revolve around the velocity, acceleration, and collisions of
objects. Examples include "Is the object moving?" and "Is object A moving
faster than object B?" We also incorporate questions by defining a specific
moment from the events of collisions happens or objects coming into frames and
ask much like how physics defines instantaneous velocity such as "when object
A collides with object B", or "when object A comes into the frame". Vertical
and horizontal accelerations are also explored, with questions like "Is object
A floating in the air?" or "Is the wagon accelerating?"

#### Predictive Questions

Predictive questions are designed to infer future interactions, especially
collisions, based on current dynamics and positioning. An example question is
“Will object A collide with object B next?”. For predictive questions, only
first 30 frames of videos are given to the model.

#### Counterfactual Questions

Counterfactual questions explore changes in object dynamics and how these
affect interactions, such as changes in velocity or acceleration. For
instance, "If Object A were moving fast, would it collide with Object B?" or
"If Object A’s engine stops, would it collide with Object B?" Such questions
allow us to assess how models respond to hypothetical modifications, providing
deeper insight into their understanding of physical reasoning.

##  4 Model

Figure 3: Our NS-4Dynamicshas three main components. I: A Neural symbolic
scene parser that parses a 4D dynamic scene representation from the video. II:
A physical prior providing a prior distribution for current states. III: A
program executor that parses questions into reasoning programs and then
executes the program over the predicted scene representation to answer the
questions.

In this section, we introduce NS-4Dynamics, to tackle the VideoQA task about
object dynamics in 3D space. As shown in Fig. 3, our model consists of three
main components: a neural symbolic scene parser, a physical prior module and a
program executor with question parser. The neural symbolic scene parser and
the physics prior are combined to first parsing the video sequences into a 4D
scene representation. Then, we integrate the language model and reasoning
program to answer the questions about the dynamic properties of objects,
including velocity, acceleration, and deduce collisions. In this part, we will
introduce the definition of the 4D symbolic scene representation in Section
4.1, then the scene parser with physical prior in Section 4.2, and the
language model and reasoning program in Section 4.3 respectively.

###  4.1 4D Symbolic Scene Representation

For a given video, we denote the image frame at timestep $t$ as
$\mathcal{I}_{t}$. The 4D symbolic scene representation $\mathcal{S}$ is a
sequence of objects states
$\mathcal{S}_{t}=\\{O_{t}^{1},O_{t}^{2},\ldots,O_{t}^{N}\\}$ over time, where
$N$ is the number of objects in the scene. Object $O_{t}^{i}$ is defined by
its translation $T_{t}^{i}\in\mathbb{R}^{3}$, rotation
$R_{t}^{i}\in\mathbb{R}^{3}$, velocity $v_{t}^{i}\in\mathbb{R}^{3}$,
acceleration $a_{t}^{i}\in\mathbb{R}^{3}$, class $cls^{i}$ and color $c^{i}$.

###  4.2 Dynamic Scene Parser with Physics Prior

The scene parser aims to estimate the world state $\mathcal{S}_{t}$ from the
corresponding observation $\mathcal{I}_{t}$ and the previous states
$\mathcal{S}_{<t}$ with a probabilistic model. Similar to the pose estimation
on images wang2021nemo ; ma2022robust , we first use render-and-compare on the
feature level for each frame, (_i.e_. $F(\mathcal{I}_{t})$) and maximizing a
scene likelihood to estimate the states, where $F(\cdot)$ is a CNN backbone.
We further consider the temporal consistency and physical plausibility between
frames by adding a physical prior in the likelihood function for correction.
This maximizing likelihood for world states estimation can be written as:

$\displaystyle\hat{\mathcal{S}_{t}}=\arg\max_{\mathcal{S}_{t}}p\left(\mathcal{S}_{t},F(\mathcal{I}_{t})\right)\cdot
p(\mathcal{S}_{t}|\mathcal{S}_{<t})$ (1)

Preliminaries. Like pose estimation, the scene likelihood compares the
features of images ($\mathcal{I}_{t}$) and the neural mesh model ($M_{cls}$)
from soft rasterization liu2019soft , to estimate the 3D location ($T_{t}$)
and rotation ($R_{t}$) by maximizing likelihood (_i.e_. world states
$\mathcal{S}_{t}$). The neural mesh model represents 3D CAD model as a vertex
set and is trained with constructive learning with images to obtain a
representation of each vertex. The details of the definition of the neural
mesh model and calculation of scene likelihood are put in the Appendix. During
inference, with a given translation $T_{t}$ and rotation $R_{t}$, the neural
mesh model of category $cls$ can be projected to have its features
$f_{cls}(R,T)$.

#### Inference as scene parsing

As the input images are processed by the CNN and get a feature $F_{t}\coloneq
F(\mathcal{I}_{t})$, following the similar structure wang2021nemo ;
ma2022robust , we are able to compare the features and construct a likelihood
function as:

$\displaystyle p(\mathcal{S}_{t},F(\mathcal{I}_{t}))\simeq p(F_{t}\mid
M_{cls},R_{t},T_{t},B),$ (2)

where $B$ is a background parameter learned from training.

#### Physical prior

Scene likelihood alone is insufficient to reconstruct a physically plausible
4D scene. Inspired by the idea from bayesian inference
areasalzmann2011physical , we integrate a physics prior
$p(\mathcal{S}_{t}|\mathcal{S}_{<t})$ into the likelihood function as a
correction term as in Eq. 4

However, it’s nontrivial to directly model the physical functions. Many
previous studies on physical engines within neural networks only focus on
simple shapes like cubes as the complexity in shapes of objects and the
interaction process is hard to model. On the other hand, computational
physical enginecoumans2016pybullet  excel at modeling physical functions for
any known shapes but are not differentiable and challenging to integrate into
the de-rendering process during inference.

In our method, we modify the discriminative physics engineering into a
probabilistic model by introducing an uncertainty term
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$. Denoted a physics engine as
$\text{PE}(\cdot)$, then the prior $p(\mathcal{S}_{t}|\mathcal{S}_{<t})$ can
be computed as:

$\displaystyle p(\mathcal{S}_{t}|\mathcal{S}_{<t})$
$\displaystyle=q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1}),$ (3)
$\displaystyle(R_{t},T_{t})\mid(\hat{R}_{t-1},\hat{T}_{t-1})$
$\displaystyle\sim\mathcal{N}(\text{PE}(\hat{R}_{t-1},\hat{T}_{t-1}),\sigma^{2}I),$
(4) $\displaystyle q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1})$
$\displaystyle=C\exp\left(-\frac{1}{2\sigma^{2}}\left[(R_{t},T_{t})-\mu\right]^{T}\left[(R_{t},T_{t})-\mu\right]\right),$
(5)

where $\mu=\text{PE}(\hat{R}_{t-1},\hat{T}_{t-1})$,
$C=1/\sqrt{(2\pi)^{k}|\sigma^{2}I|}$.

Finally, the rotation $R_{t}$ and translation $T_{t}$ can be estimated by
maximizing the joint likelihood of the rendering likelihood and the physical
prior likelihood:

$\displaystyle\hat{R}_{t},\hat{T}_{t}$
$\displaystyle=\arg\max_{R_{t},T_{t}}p(F_{t}\mid O_{c},R_{t},T_{t},B)\cdot
q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1}).$ (6)

#### Other attributes

By having the estimated world states ($R_{t},T_{t}$), here we introduce how
the other properties and interactions are predicted: (1) Velocity and
acceleration can be calculated from the difference between the translation and
further between the computed velocities at two consecutive time steps. To
avoid the potential noise in the translation estimation, we use a moving
average filter to smooth the velocity and acceleration estimation with the
window size 5. (2) Object class can be obtained from the category of mesh
model achieving the highest likelihoodwang20243d . To predict the color
attribute of objects, we crop the region containing the object from the RGB
image and train an additional CNN classifier.

#### Collisions and resimulation

Collisions are obtained from the physics engine simulator as the time of
estimating the physical prior. For each collision, we record the time and
objects involved. Predictive and counterfactual resimulation: From the
explicit 4D scene representation, we can easily simulate the future or
counterfactual states of the objects. For either two cases, we apply the
physics engine and assign the predicted translation, rotation, velocity or
counterfactual condition to all objects as initial conditions and simulate. We
visualize the resimulation results in Sec. 5.4

###  4.3 Program execution

After having the 4D scene representation from the scene parser, we can parse
the question into reasoning programs and execute the program on the scene
representation to predict the answer. The question parsing follows previous
work (yi2019clevrer, ; chen2022comphy, ), where an LSTM sequence-to-sequence
model is trained to parse the question into its corresponding program.

##  5 Experiments

###  5.1 Experiment Setup

#### Baseline models.

We select 7 representation models from the following 3 categories as baseline
models on SuperCLEVR-Physics. (1) Simple classification-based methods:
CNN+LSTM and FiLM. (2) Neural symbolic models: NS-DR and PO3D-VQA, (3) large
pretrained video-text models: Video-LLaVA, PLLaVA, and InternVideo. Please
refer to the appendix for detailed implementations.

#### Dynamic scene parser implementation.

The dynamic scene parser and CNN classifier are trained on the images and 120k
images are used for training. The dynamic scene parser uses ResNeXT for the
feature extractor and is trained for 50 epochs with batch size 4 on 4 GPUs.
The attributes classifier uses ResNet50 and is trained with the cropped images
by the ground truth bounding box. During inference, we set the variance of the
physical prior as 3.

###  5.2 Video Question Answering Results

We first compare the NS-4Dynamics with baseline models for the VideoQA task of
SuperCLEVR-Physics, the results are shown in Tab. 2.

Table 2: Performance on the SuperCLEVR-Physics testing split for each question
type, _i.e_. factual, predictive and counterfactual. The factual questions
contain sub-type about Velocity, Acceleration, Collision and "All" means its
overall accuracy. † indicates GPT-assisted zero-shot evaluation.

| Average | Factual | Predictive | Counterfactual
---|---|---|---|---
| All | Vel. | Acc. | Col.
CNN+LSTM | 48.03 | 40.63 | 41.71 | 56.79 | 25.37 | 56.04 | 47.42
FiLM perez2018film  | 50.18 | 44.07 | 48.58 | 53.09 | 26.87 | 54.94 | 51.54
NS-DR yi2019clevrer  | 51.44 | 51.44 | 55.63 | 46.34 | 46.86 | - | -
PO3D-VQA wang20243d  | 62.93 | 61.22 | 62.21 | 73.17 | 51.20 | 65.33 | 62.24
Video-LLaVA † lin2023video  | 36.04 | 33.12 | 31.90 | 56.44 | 17.31 | 42.34 | 32.65
PLLaVA † xu2024pllava  | 62.18 | 51.45 | 51.90 | 60.12 | 43.75 | 65.69 | 69.39
InternVideo wang2022internvideo  | 52.62 | 51.07 | 59.29 | 49.08 | 36.06 | 54.74 | 59.18
w/o physics | 75.97 | 79.68 | 81.40 | 81.30 | 74.88 | 78.83 | 69.39
NS-4Dynamics | 82.64 | 87.70 | 88.66 | 83.73 | 88.46 | 85.71 | 74.51

#### Comparison with classification-based methods.

The accuracy of CNN+LSTM only reaches 40.63%, 56.04% and 47.42% for factual,
predictive, and counterfactual questions; and FiLM achieves 44.07%, 54.94% and
51.54%. The performance of these models is notably lower than that of
NS-4Dynamics, which achieves 87.70%, 85.71% and 74.51% for factual,
respectively. As classification-based methods generally rely on extracting
features through CNNs, we demonstrates significant advancements in handling
dynamic content with explicit scene representation.

#### Comparison with the Neural-Symbolic methods

The comparison between the Neural-Symbolic models shows significant
performance improvement as the dimension of scene representation increases.
PO3D-VQA constructs explicit 3D scene representation, leading to a better
estimation to the objects’ position and poses, which is beneficial to infer
the dynamics and interaction compared with the 2D presentation in VR-DR
(9.78%) for the factual questions. As pushing the dimension into 4D and
considering the physical prior knowledge, our NS-4Dynamics achieves 82.64%
overall accuracy which is 19.71% higher than PO3D-VQA.

#### Comparison with the Video-LLM

Although Video-LLMs are found to have strong video understanding abilities and
generalization abilities, they do not perform as well on SuperCLEVR-Physics.
In zero-shot settings, Video-LLaVA obtains an overall accuracy of 36.04%,
while PLLaVA achieves a better but limited performance of 62.18%. Even with
finetuning, the accuracy of InternVideo is only 52.62%, falling far behind our
NS-4Dynamics. Results show existing video foundation models fail to capture
physical principles or perform reasoning on complex dynamic scenes.

###  5.3 4D Scene Reconstruction Results

To evaluate the accuracy of reconstructed scene representation with the
observation, we using the metric is 6D pose estimation xiang2014beyond to
compare the accuracy of the translation and rotation estimation, and compare
with the image level 6D pose estimation in PO3D-VQA, and the dynamics scene
representation without the physics prior (w/o physics). The results are shown
in Tab. 3.

Table 3: Evaluation of scene representation by the metric in 6D pose
estimation.

| FLOPs | Translations | Rotations
---|---|---|---
| RMSE | $Acc_{\frac{\pi}{18}}$ | $Acc_{\frac{\pi}{6}}$ | $\text{Med}_{err}$
PO3D-VQA | 0.18 | 0.233 | 62.21 | 67.21 | 3.498
w/o physics | 0.63 | 0.055 | 78.73 | 96.18 | 2.730
NS-4Dynamics | 0.62 | 0.040 | 89.71 | 98.34 | 1.419

###  5.4 Qualitative results

We visualize how our model and baseline w/o physics reconstruct the scenes and
re-simulate the future scene in predictive questions. In Fig. 4, the questions
ask about the collision, but in (c) the baseline misses the position of the
car and makes mistakes. In Fig. 5, from the input videos (a1), our model
predicts a better position for the blur bus (b1); By resimulation, the result
of our model (b2) is closer to the ground truth future frames (a2), compared
with the results of baseline (c2).

Figure 4: Qualitative examples of factual questions. (a) shows the input
video; (b)We show our NS-4Dynamics can have a better estimation for the motion
with the physical prior. (c) The error of position predicted by baseline w/o
physics in the red box leads to the mistake in answering the question. Figure
5: Qualitative examples of predictive questions. (a1) The first 30 frames are
given to models as input video and (a2) the following are hidden as ground
truth future states; (b1) our NS-4Dynamics has a better estimation of the
poses of objects, and (b2) a plausible imagination by re-simulation. (c1) The
red box shows the error of poses estimation of bus from the baselinew/o
physics, which (c2) makes the red suv collide with the school bus in the
future.

##  6 Conclusion and Discussion

In this work, we study the 3D dynamics properties in video question answering.
We present a new dataset SuperCLEVR-Physics which builds a realistic 3D
simulation with various physical events. We find that existing video-text
models, even with large-scale pretraining, fail to capture important physical
properties that are crucial for temporal or predictive reasoning. Hence we
develop NS-4Dynamics, a neural-symbolic model built on explicit 4D scene
representation. We capture the physical principles by incorporating physical
priors in our neural symbolic scene parser. Experimental results on
SuperCLEVR-Physics show that our approach outperforms previous state-of-the-
art methods by a wide margin, demonstrating the advantages of our approach to
infer dynamic properties and to predict future states. In ongoing work we are
making our dataset more realistic by exploiting big data techniques, such as
generating 3D-DST data with diffusion models ma2023adding  and developing our
neural-symbolic parser to work on real images by extending kaushik2024source .

#### Limitations.

Our neural symbolic scene parser integrates physics as a prior when predicting
future states. For future work we hope to explore differentiable physics
engines that enable scene parsing with multi-frame reasoning. This is a
significant extension of previous methods as we are involving real-world
objects and complex simulations, posing challenges to the differentiable
physics simulator.

## References

  * [1] Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret.  CRAFT: A benchmark for causal reasoning about forces and inTeractions.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2602–2627, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [3] Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum.  Simulation as an engine of physical scene understanding.  Proceedings of the National Academy of Sciences, 110(45):18327–18332, 2013.
  * [4] Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, et al.  Physion: Evaluating physical prediction from vision in humans and machines.  arXiv preprint arXiv:2106.08261, 2021.
  * [5] Blender Online Community.  Blender - a 3D modelling and rendering package, 2021.  Blender Foundation, Stichting Blender Foundation, Amsterdam, The Netherlands.
  * [6] Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan.  Comphy: Compositional physical reasoning of objects and events from videos.  arXiv preprint arXiv:2205.01089, 2022.
  * [7] Erwin Coumans and Yunfei Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  2016\.
  * [8] Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan.  Dynamic visual reasoning by learning differentiable physics models from video and language.  Advances In Neural Information Processing Systems, 34:887–899, 2021.
  * [9] Rohit Girdhar and Deva Ramanan.  CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning.  In ICLR, 2020.
  * [10] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi.  Kubric: a scalable dataset generator.  2022\.
  * [11] Jessica B Hamrick, Peter W Battaglia, Thomas L Griffiths, and Joshua B Tenenbaum.  Inferring mass in complex scenes by mental simulation.  Cognition, 157:61–76, 2016.
  * [12] Artur Jesslen, Guofeng Zhang, Angtian Wang, Alan Yuille, and Adam Kortylewski.  Robust 3d-aware object classification via discriminative render-and-compare.  arXiv preprint arXiv:2305.14668, 2023.
  * [13] Achuta Kadambi, Celso de Melo, Cho-Jui Hsieh, Mani Srivastava, and Stefano Soatto.  Incorporating physics into data-driven computer vision.  Nature Machine Intelligence, 5(6):572–580, 2023.
  * [14] Prakhar Kaushik, Aayush Mishra, Adam Kortylewski, and Alan Yuille.  Source-free and image-only unsupervised domain adaptation for category level object pose estimation.  arXiv preprint arXiv:2401.10848, 2024.
  * [15] Nikolaos Kyriazis and Antonis Argyros.  Physically plausible 3d scene tracking: The single actor hypothesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9–16, 2013.
  * [16] Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille.  Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14963–14973, 2023.
  * [17] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  arXiv preprint arXiv:2311.10122, 2023.
  * [18] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708–7717, 2019.
  * [19] Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Yaoyao Liu, Adam Kortylewski, and Alan Yuille.  Adding 3d geometry control to diffusion models.  arXiv preprint arXiv:2306.08103, 2023.
  * [20] Wufei Ma, Angtian Wang, Alan Yuille, and Adam Kortylewski.  Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features.  In European Conference on Computer Vision, pages 492–508. Springer, 2022.
  * [21] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al.  OpenEQA: Embodied Question Answering in the Era of Foundation Models.  In CVPR, 2024.
  * [22] Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang.  Cripp-vqa: Counterfactual reasoning about implicit physical properties via video question answering.  arXiv preprint arXiv:2211.03779, 2022.
  * [23] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [24] Mathieu Salzmann and Raquel Urtasun.  Physically-based motion models for 3d tracking: A convex formulation.  In 2011 International Conference on Computer Vision, pages 2064–2071, 2011.
  * [25] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [26] Fish Tung, Mingyu Ding, Zhenfang Chen, Daniel M. Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith Fan, and Kevin A. Smith.  Physion++: Evaluating physical scene understanding that requires online inference of different physical properties.  arXiv, 2023.
  * [27] Tomer D Ullman, Andreas Stuhlmüller, Noah D Goodman, and Joshua B Tenenbaum.  Learning physical parameters from dynamic scenes.  Cognitive psychology, 104:57–82, 2018.
  * [28] Angtian Wang, Adam Kortylewski, and Alan Yuille.  Nemo: Neural mesh models of contrastive features for robust 3d pose estimation.  arXiv preprint arXiv:2101.12378, 2021.
  * [29] Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan Yuille.  3d-aware visual question answering about parts, poses and occlusions.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [30] Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan L Yuille.  3d-aware visual question answering about parts, poses and occlusions.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 58717–58735. Curran Associates, Inc., 2023.
  * [31] Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan L Yuille.  3d-aware visual question answering about parts, poses and occlusions.  Advances in Neural Information Processing Systems, 36, 2024.
  * [32] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al.  Internvideo: General video foundation models via generative and discriminative learning.  arXiv preprint arXiv:2212.03191, 2022.
  * [33] Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum.  Learning to see physics via visual de-animation.  Advances in neural information procesing systems, 30, 2017.
  * [34] Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum.  Galileo: Perceiving physical object properties by integrating a physics engine with deep learning.  Advances in neural information processing systems, 28, 2015.
  * [35] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In IEEE winter conference on applications of computer vision, pages 75–82. IEEE, 2014.
  * [36] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng.  Pllava: Parameter-free llava extension from images to videos for video dense captioning.  arXiv preprint arXiv:2404.16994, 2024.
  * [37] Hongyang Xue, Zhou Zhao, and Deng Cai.  Unifying the video and question attentions for open-ended video question answering.  IEEE Transactions on Image Processing, 26(12):5656–5666, 2017.
  * [38] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Learning to answer visual questions from web videos.  IEEE TPAMI, 2022.
  * [39] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum.  Clevrer: Collision events for video representation and reasoning.  arXiv preprint arXiv:1910.01442, 2019.
  * [40] Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B Tenenbaum, and Chuang Gan.  Contphy: Continuum physical concept learning and reasoning from videos.  arXiv preprint arXiv:2402.06119, 2024.

##  Appendix A Dynamic properties setting

Table 4: Dynamic settings of SuperCLEVR-Physics Attribute | Description | Aeroplanes | Others
---|---|---|---
Mass | Calculated from volume | $\rho V$
Position | $(x,y)$ | Beta distribution
$z$ | Uniform distribution | 0
Orientation | Faces to the center with noises | -
Velocity | Initial state: static, slow, fast |  $\\{0,3,6\\}$ m/s
Internal Force | Engine force (Forwards) | $\\{1,0\\}\text{m/s}^{2}$
Floating force(Upwards) | {10, 0} $\text{Mass}\times\text{m/s}^{2}$ | 0 $\text{Mass}\times\text{m/s}^{2}$

#### External Forces and other physical properties

  * •

Gravity: All objects in the dataset are subject to gravity, which influences
their vertical movements and impacts when in flight or during falls. The
gravitational constant is set as 10.

  * •

Friction: Frictional forces affect the movement of all objects, especially
when they interact with the ground or each other, slowing them down and
eventually bringing them to a stop.

  * •

Elastic Force during Collisions: When objects collide, the elastic forces come
into play, defined by elasticity coefficients. These forces affect how objects
bounce off each other or rebound from barriers, significantly altering their
trajectories and speeds.

#### Physical Properties

(1) Mass: The mass for each object is calculated based on its specific shape
model, which is linearly related to its volume with a density of 2.7; (2)
Friction: The friction of objects are set as xx; the friction floor (the dome
shaped background) is set as xx. (2) Restitution: The restitution of objects
are xx and the floor’s are set to xx.

##  Appendix B Question and program executor

We design new program for the 4D dynamcis reasoning VideoQA. For the factual
question, we use query_moving,query_moving_fast,query_moving_direction for get
the moving status. greater_linear_velocities, less_linear_velocities for
comparing the linear velocities. query_engine for querying the physics engine.
query_floating for querying the floating objects. filter_collision for
filtering the collision events of given objects.get_col_partner for getting
the collision partner of the object. get_all_col_partners for getting all the
collision partners of the object.

##  Appendix C Details of Dynamic Scene Parser with Physics Prior

The ultimate goal of the scene parser is to estimate the all dynamic state
$\mathcal{S}_{t}$ from the corresponding observation $\mathcal{I}_{t}$ and the
previous states $\mathcal{S}_{<t}$ using a probabilistic model. The previous
work in 6D pose estimation has proposed to use render-and-compare to
maximizing scene likelihood for static images. For the video data, we further
consider the temporal consistency and physical plausibility of the dynamics
estimation.

$\displaystyle\hat{\mathcal{S}_{t}}$
$\displaystyle=\arg\max_{\mathcal{S}_{t}}p\left(f(\mathcal{I}_{t})|\mathcal{S}_{t}\right)\cdot
p(\mathcal{S}_{t}|\mathcal{S}_{<t}),$ (7) $\displaystyle\hat{\mathcal{S}_{0}}$
$\displaystyle=\arg\max_{\mathcal{S}_{0}}p(f(\mathcal{I}_{0})|\mathcal{S}_{0}).$
(8)

Inspired by the neural-meshed based generative mode for pose estimation
methods on static images [28, 20], we develop a de-rendering based generative
model to reconstruct the 4D dynamic scene representation frame by frame. To
ensure the predictions are not only plausible in 3D positions but also in
physical rules, we incorporate a physical likelihood model and a 3D generative
model with rendering likelihood, as shown in Fig. 3.

Preliminaries. In the previous work for static images[20], Neural Meshes model
were introduced for 6D pose estimation through inverse rendering. For that
task, the goal is to jointly estimate the 6D pose (2D location $(x,y)$,
distance $d$ to the camera and 3D pose ($\alpha,\beta,\gamma$) of objects in
an image by comparing the Neural Meshes feature after rendering with the input
image feature and maximizing the rendering likelihood. More formally, the mesh
for a given object in category $c$ is represented as
$M_{c}=\\{v_{i}\in\mathbb{R}^{3}|i=1\ldots N\\}$ where $v_{i}$ means the
vertex. The corresponding neural texture of the mesh $M_{c}$ is
$T_{c}\in\mathbb{R}^{N\times l}$ where $l$ is the dimension of the feature. So
that the neural mesh model for category $c$ is their aggregation
$O_{c}=\\{M_{c},T_{c}\\}$ .

In this work, we use transform the position into a world coordinate instead.
Given the object 3D rotation $R=(\alpha,\beta,\gamma)$ and translation
$T=(x,y,z)$, we can render the neural mesh model $O_{c}$ into a feature map
$F_{c}$ with soft rasterization [18]:

$\displaystyle F_{c}(R,T)=\mathfrak{R}(O_{c},R,T).$ (9)

#### Inference as scene parsing

For static images, the previous work [28] has shown that the rendering
likelihood can be used to estimate the 6D pose of objects. In our dynamics
scene, similar strategies can be applied to each frame at time step $t$. As
the neural mesh models are probabilistic generative model of neural feature
activation, we can first define the rendering likelihood of the feature map
$F_{t}$ given any 6D pose $R_{t},T_{t}$ as:

$\displaystyle p(\mathcal{I}_{t}|\mathcal{S}_{t})$ $\displaystyle=p(F_{t}\mid
O_{c},R_{t},T_{t},B)$ (10)
$\displaystyle=\prod_{i\in\mathcal{FG}}p(f_{t}^{(i)}\mid
O_{c},R_{t},T_{t})\prod_{j\in\mathcal{BG}}p(f_{t}^{(j)}\mid B),$ (11)

where $\mathcal{FG}$ and $\mathcal{BG}$ are the set of foreground and
background locations on the 2D feature map and $f_{t}^{(i)}$ is the feature
vector of $F$ at location $i$ at timestep $t$. B is a background parameter
learned from training. Here the foreground and background likelihoods are
modeled as Gaussian distributions.

#### Physical prior

(correction)

Rendering likelihood alone is insufficient to reconstruct a physically
plausible 4D dynamic scene representation. We also integrate a physics prior
into our likelihood model. The physical process can be modeled as a Markov
model [24], where the physical prior distribution of rotation $R_{t}$ and
translation $T_{t}$ at time step $t$ can be expressed as:

$\displaystyle
p(\mathcal{S}_{t}|\mathcal{S}_{<t})=q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1}).$
(12)

However, it’s nontrivial to directly model the physical functions. Most
current studies on physical engines within neural networks only focus on
simple shapes like cubes as the complexity in shapes of objects and the
interaction process is hard to model. On the other hand, computational
physical engines such as Bullet excel at modeling physical functions for any
known shapes but are not differentiable and challenging to integrate into the
de-rendering process during inference.

In our method, we modify the discriminative physics engineering into a
probabilistic model by introducing an uncertainty term
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$. Denoted a physics engine as
$\text{PE}(\cdot)$, then the $q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1})$
can be assumed by:

$\displaystyle(R_{t},T_{t})$
$\displaystyle=\text{PE}(\hat{R}_{t-1},\hat{T}_{t-1})+\epsilon,$ (13)
$\displaystyle(R_{t},T_{t})\mid(\hat{R}_{t-1},\hat{T}_{t-1})$
$\displaystyle\sim\mathcal{N}(\text{PE}(\hat{R}_{t-1},\hat{T}_{t-1}),\sigma^{2}I),$
(14) $\displaystyle q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1})$
$\displaystyle=C\exp\left(-\frac{1}{2\sigma^{2}}\left[(R_{t},T_{t})-\mu\right]^{T}\left[(R_{t},T_{t})-\mu\right]\right),$
(15)

where $\mu=\text{PE}(\hat{R}_{t-1},\hat{T}_{t-1})$,
$C=1/\sqrt{(2\pi)^{k}|\sigma^{2}I|}$.

Finally, the rotation $R_{t}$ and translation $T_{t}$ can be estimated by
maximizing the joint likelihood of the rendering likelihood and the physical
prior likelihood:

$\displaystyle\hat{R}_{t},\hat{T}_{t}$
$\displaystyle=\arg\max_{R_{t},T_{t}}p(F_{t}\mid O_{c},R_{t},T_{t},B)\cdot
q(R_{t},T_{t}\mid\hat{R}_{t-1},\hat{T}_{t-1}).$ (16)

#### Relationship to Bayes-Kalman filtering

The conceptual ideas are directly inspired by the classic ideas of Bayes-
Kalman filtering where the goal is to estimate a hidden state based on a
sequence of observations. Bayes-Kalman filtering consists of updating a
probability distribution of the hidden state by a prediction step followed by
a correction step which incorporates evidence from a new observation. It uses
a dynamic model for how the hidden state changes with time which is directly
analogous to our physical prior. It has an observation model, corresponding to
our scene parser, for how new observations give evidence for the hidden state.
Bayes-Kalman, however, is difficult to implement in complex applications, like
ours, because it requires us to represent and update a complex probability
distribution. The standard approach for doing this is particle filtering where
the probability distribution is represented by a set of point particles which
are updated during the prediction and correction steps. This is also
challenging so instead we use a simple approximation which essentially uses a
single particle. In future work we will experiments to see if our model gives
even better results if instead of this approximation we use particle
filtering.

##  Appendix D Baseline Models

We select 7 representative models from the following 3 categories as baseline
models on SuperCLEVR-Physics. (1) Simple classification-based methods encode
videos with a CNN backbone and predict answers with a classifier head.
Specifically we consider CNN+LSTM that aggregates frame-level CNN features and
encode the question by LSTM, and FiLM [23] that incorportes a feature-level
linear modulation module for question answering. (2) Neural symbolic models
first parse the scene into object instances and then executes a program for
question answering. NS-DR [39] adopts a Mask R-CNN for object detection. We
modify it with the new program in SuperCLEVR-Physics and evaluate it on
factual questions. As their 2D simulator is unable to reason the objects in
3D, we do not compare it on predictive and counterfactual questions. PO3D-VQA
[31] uses a 3D detector and reconstructs an explicit 3D scene representation
for each frame. We extend the model for VideoQA by computing dynamic
properties from object locations and predict collision by filtering the
distance between objects. (3) Large pretrained video-text models adopt large-
scale image-text or video-text data for pretraining and achieve strong
generalization abilites to various video-text tasks. We consider Video-LLaVA
[17] and PLLaVA [36] for zero-shot evaluation and choose the model sizes as
7B. For evaluation, we use a GPT model to justify the correctness of free-form
answers. Moreover, we finetune a pretrained InternVideo [32] model that
predicts answers with a classifier head.