# An analogical model for the stationary black holes by the flow field of
space-time fluid around a 3-dimensional point sink

Behzad Ataei<EMAIL_ADDRESS>Department of Aerospace
Engineering, Sharif University of Technology,Tehran, Iran  Ali Ayatollah
Rafsanjani  0000-0002-3717-718X<EMAIL_ADDRESS>Department of
Physics, Sharif University of Technology,Tehran, Iran  School of Physics,
Institute for Research in Fundamental Sciences (IPM), Tehran, Iran  Alireza
Bahrampour  Department of Physics, Sharif University of Technology,Tehran,
Iran  Mohammad Taeibi Rahni  Department of Aerospace Engineering, Sharif
University of Technology,Tehran, Iran  Mohammadreza Salimi  Aerospace Research
Institute, Sharif University of Technology, Tehran, Iran  Mehdi Golshani
School of Physics, Institute for Research in Fundamental Sciences (IPM),
Tehran, Iran

###### Abstract

Recent researches suggest an analogy between the theory of general relativity
(GR) and fluid dynamics. As a result of this analogy, the Navier-Stokes
equations and Einstein field equations are the same, and it is possible to
study the properties of space-time by using fluid mechanics. In this paper, we
present a new model to describe gravitational phenomena by an inviscid and
compressible fluid called space-time fluid (STF). The analogy method is used
to obtain the gravity field of both static and rotating masses from the flow
field of STF around static and rotating point sinks. In addition, event
horizons and the ergosphere of stationary black holes are defined based on our
STF model. Then, we compare hydrodynamic forces exerted on a test particle
with gravitational forces in the gravitoelectromagnetic approximation of the
GR. As a natural consequence, it is shown that inertial and gravitational
masses are equivalent in this analogy. Finally, using the aspect of fluid
dynamics, Mach’s principle, weak equivalence principle, and information
discontinuity on the event horizon are discussed.

Keywords: general relativity; space-time fluid; analogue gravity; acoustic
metric; black hole; emergent gravity; Kerr metric; compressible flow.

##  I Introduction

The analogy is a cognitive process that plays a significant role in problem-
solving. It provides cross-fertilization of ideas between various research
subjects by giving new ways of looking at problems. As examples of valuable
analogies in physics, we can refer to the statistical theory of gases
developed by Maxwell and Boltzmann, which established an analogy between gases
and microscopic Newtonian rigid bodies through which they obtained entire laws
of thermodynamics by those new microscopic degrees of freedom [1]. Another
successful analogy was the gravitomagnetic hypothesis, which is an analogy
between gravitation and electromagnetism that predicted some phenomenological
aspects of general relativity (GR) before that [2]. The most well-known
analogy of gravity in recent decades is the model of gravity and space-time by
various condensed-matter systems, especially the fluid one [3].

Fluid physics is a vibrant field that includes two essential scales,
microscopic and macroscopic. Using the fluid-gravity analogies, we can extract
microscopic and statistical attributes of space-time by the lattice-Boltzmann
method and macroscopic one by Navier-Stokes equations. Therefore, these kinds
of analogies help to understand the phenomenological effects of gravity better
and obtain lessons that could be useful on the way to a quantum gravity
theory.

Furthermore, due to the development of engineering tools in fluid dynamics,
such analogies facilitate obtaining new solutions and simulations for critical
aspects of gravity by using powerful instruments like computational fluid
dynamics (CFD). Also, by establishing the fluid/gravity correspondence, we
would have an exquisite attitude that could be an excellent source of
intuition to solve the gravitational problems. For instance, there is no
satisfactory solution for interior the Kerr metric, and we hope to model the
gravitational field of a black hole by using the CFD to reach a clear picture
of this problem. Moreover, it could be a good ground for experimental
investigation of curved-space quantum field theory in condensed matter
laboratories. It is possible to obtain a new entry for solving cosmological
problems by using broad-range phenomena in fluid physics (like dark matter and
energy).

In 1981 Unruh used a hydrodynamical analogy for a scalar field theory in an
effective space-time using an acoustic wave in an irrotational inviscid
barotropic fluid and then used this model to redescribe the black hole
evaporation [4]. This first clue led us to look at gravitational phenomena,
like Hawking’s radiation, as emergent phenomena. After Unruh’s primary model,
many efforts were made to develop a fluid analogy to describe phenomena that
correspond to fundamental curved-space quantum field theory, like black hole
horizons, ergosphere, and Hawking’s radiation, which has no direct relation
with general relativity. Nowadays, these properties are known as attributes
that emerge from unfamiliar microscopic degrees of freedom [5].

In 1995, Jacobson derived Einstein’s equation from the thermodynamical
property of horizon and showed that one could look to Einstein’s equation as
an equation of state[6]. Thereupon, one of the more explicit works that show
we can look at gravity as an emergent force was 2011 Verlinde’s paper which
represented gravity as an entropic force [7]. In the same year, Padmanabhan
showed that Einstein’s field equations are identical in form to Navier–Stokes
equations of hydrodynamics [8]. His result led us to think that it is possible
to obtain an identical description of large-scale gravitational phenomena by
using the hydrodynamical analogy in addition to the statistical aspects of
emergent gravity.

Also, in several articles, black holes were modeled by a two-dimensional sink
or one-dimensional irrotational fluid dynamics in order to reproduce black
hole thermodynamics [9], which in none of them, the flow field of space-time
fluid (STF) and gravitational forces are described explicitly. Liberati and
Visser have introduced the same idea and attempted to calculate the fluid flow
field using the Schwarzschild metric. [3]

This article starts with time dilation in special relativity and compares it
with gravitational time dilation phenomenologically. Then, using the classical
inviscid barotropic fluid, the flow field of STF around a spherically
symmetric mass distribution will be calculated. In this analogy, a
gravitational mass is modeled by a point sink. We show that the flow field
around a point sink introduces gravitational forces and the equivalence
principle as a natural consequence. To validate our results, we will study all
exerted forces on a test particle in STF flow to show that these forces are
the same as the applied forces on probe B satellite by using the
gravitomagnetic equations in the gravitoelectromagnetic approximation of the
general relativity.

Afterward, in Section IV, the event horizon and ergosphere surface
(ergosurface) of black holes are defined and calculated in static and
rotational cases. We show that the sonic surfaces around the stationary and
rotating point sinks correspond to the event horizon and ergosurface of
stationary black holes. In Section VI, we discuss the meaning of the Mach’s
principle, equivalence principle, preferred reference frame, and information
discontinuity in both our fluid flow model and black holes.

##  II Time dilation

Time dilation is the difference in elapsed time as measured by two observers.
Here, we consider it phenomenologically as an effect observed in several
experiments. It is either due to a relative velocity between observers
(special relativistic time dilation) or a difference in gravitational
potential between their locations (general relativistic gravitational time
dilation)

###  II.1 Relative and absolute time dilation

According to the theory of Special relativity, the proportion of elapsed time
rates for two observers at a speed of $\mathbb{v}$ relative to each other is

$\frac{dt^{\prime}}{dt}=\sqrt{1-\frac{\mathbb{v}^{2}}{c^{2}}},$ (1)

Which $dt$ is the rate of elapsed time measured by observer $O$, and
$dt^{\prime}$ for observer $O^{\prime}$. Notice that this equation is valid
only in the reference frame of $O$ and that $O^{\prime}$ has another
experience for the passage of time. This phenomenon has been confirmed in
numerous experiments [10, 11, 12, 13, 14]

In addition to the time dilation between inertial observers, time passes
slowly when an object is placed in a gravitational field. This phenomenon
became validated in 1976 by gravity probe A of NASA. Unlike the relative
velocity time dilation, observers agree on which clock is slower in the
gravitational time dilation. We can also see this attribute in accelerated
frames of reference. Einstein predicted the gravitational time dilation using
these facts and assuming the equivalence principle.

The amount of time dilation for an object located at distant $r$ from a static
gravitational mass is

$\frac{dt^{\prime}}{dt}=\sqrt{1-\frac{2GM}{rc^{2}}},$ (2)

with $M$ the gravitational mass, $G$ the gravitational constant, and
$\mathrm{c}$ the speed of light. Notice that $dt$ corresponds to the observer
$O$ at infinity, which has no gravitational field, and $dt^{\prime}$ is the
proper time of observer $O^{\prime}$ at position $r$. Assume a rotating
gravitational mass with specific angular momentum $\mathbb{a}=\mathbb{J}/(Mc)$
(with $\mathbb{J}=I\mathbb{\Omega}$ the angular momentum); time dilation is
written as

$\frac{dt^{\prime}}{dt}=\sqrt{1-\frac{2GMr}{c^{2}\left(r^{2}+a^{2}\cos^{2}\theta\right)}}.$
(3)

###  II.2 STF flow field and time dilation

First, we suppose STF as an inviscid fluid has filled the space. We propose
that the relative time dilation is due to the relative velocity of the
observer in the STF. So, it will be shown that by considering the
gravitational time dilation as a phenomenological fact, we can obtain the
equivalence principle and other gravitational phenomena. To reach this goal,
first, we use Eq. 1 and define the relative velocity between the STF flow and
the object as $\mathbb{u}_{stf}$, and the speed of data transfer in STF as
$c_{s}$. So, we can write

$\frac{dt^{\prime}}{dt}=\sqrt{1-\frac{\mathbb{u}_{stf}{}^{2}}{c_{s}{}^{2}}}.$
(4)

Here, $O$ is the observer that is comoved with STF flow, and $O^{\prime}$ has
the relative velocity to the flow field. Due to the similarity of the
expression $\mathbb{u}_{sft}/c_{s}$ to the Mach number in fluid dynamics, we
called this expression the pseudo-Mach number as

${\cal M}:=\frac{\left|\mathbb{u}_{stf}\right|}{c_{s}}.$ (5)

The pseudo-Mach number indicates the ratio of the velocity of STF flow to the
speed of data transfer in STF.

Here, we assume the speed of signaling in the STF is equal to the speed of
light, i.e., $\left(c_{s}=c\right)$. So, the relation between the pseudo-Mach
number and the time dilation is

$\frac{dt^{\prime}}{dt}=\sqrt{1-{\cal M}^{2}}.$ (6)

##  III STF flow field around a static and rotating mass

###  III.1 Static mass

Consider Eq. 6 as the relation between the STF flow field and the
gravitational time dilation. Using this, we can determine the STF flow field
around a static point sink by an analogy between Eqs. 2 and 4, as

$\mathbb{u}_{stf}=-\sqrt{\frac{2GM}{r}}\hat{r}.$ (7)

The spherical symmetry of the mass distribution led us to infer that the
direction of the flow field has no $\hat{\theta}$ and $\hat{\phi}$ components
and is just in the radial direction. Also, we demand an attraction force
toward the source; thus, we assume an inward flow. This equation shows the
radial flow field around a point sink. We can interpret it as a 3-dimensional
point sink swallowing the STF.

###  III.2 Rotating mass

By considering a stationary rotating mass and comparing Eqs. 3 and 4, the
magnitude of the flow field will be given by

$\left|\mathbb{u}_{stf}\right|=\sqrt{\frac{2GMr}{r^{2}+a^{2}\cos^{2}\theta}}.$
(8)

In this case, we cannot use the spherical symmetry to determine the velocity
field components; however, axial symmetry implies that the field is
independent of $\hat{\theta}$. One can suggest that the radial component is
the same as the static case and use the magnitude of the flow to calculate the
$\hat{\phi}$ component. Nevertheless, this is not true because in the poles
$(\theta=0,\pi)$, the flow is just in the radial direction 111Here, the Kerr
metric and consequently the flow field of rotating masses have been written in
the Boyer-Lindquist coordinates. Hence, the radial $\hat{r}$ and transverse
$\hat{\phi}$ components are the confocal ellipses and hyperbolas in the
ellipsoidal coordinate. (at the rotational axis of the poles the flow has only
the radial component). Hence,

$\mathbb{u}_{stf}(\theta=0,\pi)=-\sqrt{\frac{2GMr}{r^{2}+a^{2}}}\hat{r}.$ (9)

##  IV static and rotating black holes

In our analogy, black holes are modeled as powerful point sinks toward which
the STF flows, and at some points, the STF velocity reaches the speed of
light. Thus, we will have sonic surfaces similar to normal fluids by
considering the light same as the acoustic wave in STF. This section
investigates the resulting sonic surfaces formed around the sinks.
Correspondingly, the equivalence of the event horizon and the ergosurface
around black holes with obtained sonic surfaces are being examined.

First of all, we investigate the compressibility of space-time fluid. For this
purpose, the divergence of the flow field in Eq. 7 is calculated as

$\nabla\cdot\mathbb{u}_{stf}\neq 0.$ (10)

Consequently, we conclude that STF is compressible.

###  IV.1 static black hole

In the STF model, the locus of the sonic surface around a static point sink
corresponds to the event horizon of a stationary black hole in the
Schwarzschild metric, which is [16]:

$r_{s}=\frac{2GM}{c^{2}}$ (11)

To obtain the sonic surface around a static point sink, we set the pseudo-Mach
number in Eq. 5 equal to one;

$\displaystyle{\cal M}$
$\displaystyle=\frac{\left|\mathbb{u}_{stf}\right|}{c_{s}}=\frac{\sqrt{\frac{2GM}{r}}}{c_{s}}=1,$
(12)

thus, the surface where the velocity of the STF is equal to the speed of light
(sonic surface) is obtained as

$\displaystyle r=\frac{2GM}{c_{s}^{2}},$ (13)

which is correspond to the Schwarzschild radius.

Figure 1: The STF radial velocity around static point sink

In Fig. 1, the velocity of the STF around the sink is shown for a static point
sink. In the radius $r_{s}$, the velocity of the $\mathrm{STF}$ is equal to
the speed of light.

###  IV.2 rotating black hole

####  IV.2.1 Event horizon and ergosphere in Kerr metric

The event horizon and ergosurface around the rotating black hole are
calculated using the Kerr metric.[17] The radius of the event horizon

$r_{\pm}=\frac{r_{s}\pm\sqrt{r_{s}^{2}-4a^{2}}}{2},$ (14)

and ergosurface are

$r_{s_{\pm}}=\frac{r_{s}\pm\sqrt{r_{s}^{2}-4a^{2}\cos^{2}\theta}}{2}.$ (15)

In Fig. 2, we see the red surface as the outer event horizon with radius
$r_{+}$and the green surface as the inner event horizon with radius $r_{-}$,
where these radii correspond to the values in Eq. 14. In addition, the yellow
surface is the outer ergosphere with radius $r_{s_{+}}$, and the blue surface
corresponds to the inner ergosphere with radius $r_{s_{-}}$. These radii have
corresponded to the values in Eq. 15.

####  IV.2.2 Sonic surface of a rotating point sink

On the sonic surface around a rotating point sink, $\mathbb{u}_{stf}$ in Eqs.
8 and 9 must be equal to $c_{s}$. We are setting Eq. 8, the total velocity of
the STF, equal to the speed of acoustic wave,

${\cal
M}=\frac{\left|\mathbb{u}_{stf}\right|}{c_{s}}=\frac{\sqrt{\frac{2GMr}{r^{2}+a^{2}\cos^{2}\theta}}}{c_{s}}=1$
(16)

results in

$r_{1,2}=\frac{r_{s}\pm\sqrt{r_{s}^{2}-4a^{2}\cos^{2}\theta}}{2}.$ (17)

As one can see, these surfaces are analogous to the ergosurface. An object
within the $r<r_{s_{+}}$ region cannot be stationary with respect to an
outside observer at a far distance unless that object were to move faster than
the speed of the acoustic wave concerning the local spacetime. However,
objects can eject from there with a proper radial velocity.

To obtain the region where the test particles cannot eject from that, setting
Eq. 9 (velocity in the direction of the radius of the STF around the rotating
sink) equal to the speed of sound, we will have

${\cal
M}=\frac{\left|\mathbb{u}_{stf}\right|}{c_{s}}=\frac{\sqrt{\frac{2GMr}{r^{2}+a^{2}}}}{c_{s}}=1,$
(18)

So, points referring to areas where the radial velocity of the fluid becomes
equal to the velocity of the sound wave are determined as

$r_{3,4}=\frac{r_{s}\pm\sqrt{r_{s}^{2}-4a^{2}}}{2}.$ (19)

Eqs. 17 and 19 show surfaces where the total velocity and radial velocity of
the STF flow are equal to the speed of light. In other words, the quasi-Mach
number is equal to one at these surfaces, and information discontinuity occurs
at the sonic surfaces in the fluid flow. Obtained surfaces correspond to the
event horizon and the rotating black hole ergosurface.

According to the Euler equations, simplified forms of the Navier-Stokes
equations, information travels through the pressure wave at the speed of sound
in a compressible and inviscid flow. In subsonic regimes, the information can
propagate both upstream and downstream of the flow; hence the equations have
elliptical behavior. On the other hand, as velocity increases, information
propagation becomes weaker until a singularity occurs at the speed of sound.
In the meantime, the equations show parabolic behavior. This means that the
information could no longer propagate upstream, resulting in information
rupture. In the presented analogy, the sound speed in STF is equivalent to the
speed of light propagation. The sound wave cannot pass through the sonic
surface from downstream. In addition, there is a surface where the magnitude
of the STF velocity is equal to the speed of information transfer in the fluid
[18].

Therefore, according to the Eq. 17, if a sound wave moves in the radial
direction, it can pass through the surface. In contrast, Eq. 19 shows a
surface where the STF velocity is equal to the speed of sound, and the surface
prevents the sound wave from passing through even if it moves in a radial
direction. This phenomenon leads to information disconnection. As mentioned
before, several studies have shown an analogy between the sonic surface of the
point sink and the event horizon of static black holes in two dimensions.
Nevertheless, this article calculates sonic surfaces in three dimensions
around static and rotating point sinks. Additionally, this study better
explains the nature of information transfer in the event horizon and
ergosphere. In the future, we will accurately model this subject with the help
of CFD.

Figure 2: ergosphere and event horizons around the rotating black hole [19]

##  V Hydrodynamics and Gravitation

Here we will study the phenomenological conformity between gravitational and
hydrodynamic forces acting on a test particle in a two-phase inviscid fluid.
Notice that the nature of dynamics in GR is due to the space-time geometry,
and there is no concept of force in this theory. Thus, we should use the
perturbation method to bring the two approaches on the same ground to have a
fair comparison. Hence, utilizing the flow field obtained for the STF, we
calculate the acting hydrodynamic forces on the test particle.

###  V.1 Gravitoelectromagnetism

Gravitoelectromagnetism, GEM for short, refers to a set of formal similarities
between electromagnetic equations and relative gravity. In particular, between
the Maxwell field equations and the valid approximation under certain
conditions, to the Einstein field equations for GR. Gravitomagnetism is a
widely used term that refers explicitly to the kinetic effects of gravity in
comparison with the magnetic effects of a moving electric charge. The most
common version of GEM is only valid away from separate sources and for slow-
moving experimental particles. In this analogy, the GEM field is created
around a rotating mass, the value of which is given by

$\mathbb{B}_{g}=\frac{GI}{2c^{2}r^{3}}\left(\mathbb{\Omega}-{3(\mathbb{\Omega}\cdot\hat{r})\hat{r}}\right),$
(20)

and the force applied to the test particle is

$\mathbb{F}=m\mathbb{g}+4m\left(\mathbb{v}\times\mathbb{B}_{g}\right),$ (21)

which is like the Lorentz force [20, 21, 22, 23, 24, 25]. Here,
$\mathbb{B}_{g}$ is the gravitomagnetic field, $\mathbb{v}$ is the velocity of
the test particle, $\mathbb{\Omega}$ is the angular velocity, and $I$ is the
area moment of inertia.

These approximated forces are due to the frame-dragging effect in GR. In 2011,
NASA’s Gravity Probe B experiment confirmed the forces of the
gravitoelectromagnetic field. In this paper, the probe B satellite is
considered as a spherical test particle orbiting a rotating planet [26].
Therefore we consider a spherical test particle, like the probe B satellite,
orbiting a rotating planet.

###  V.2 Hydrodynamic forces applied to a test particle

The following forces are applied to a single spherical particle, with velocity
$\mathbb{v}$, in the inviscid flow field $\mathbb{u}$ [27, 28, 29, 30, 31,
32]. By assuming the STF density to be equal to the particle density,
$\rho_{c}$, we have

$\displaystyle\mathbb{F}_{vm}+\mathbb{F}_{L}+\mathbb{F}_{p}=\left(\rho_{c}\forall_{p}\right)\frac{d\mathbb{v}}{dt},$
(22)

with

$\displaystyle\mathbb{F}_{vm}=\frac{1}{2}\rho_{c}\forall_{p}\left(\frac{D\mathbb{u}}{Dt}-\frac{d\mathbb{v}}{dt}\right),$
(23)
$\displaystyle\mathbb{F}_{p}=\rho_{c}\forall_{p}\left(\frac{D\mathbb{u}}{Dt}\right),$
(24)
$\displaystyle\mathbb{F}_{L}=C_{L}\rho_{c}\forall_{p}(\mathbb{v}\times\mathbb{\omega}).$
(25)

$\mathbb{F}_{vm}$ stands for the virtual mass, $\mathbb{F}_{L}$ for the
pressure gradient force, and $\mathbb{F}_{p}$ for the lift force in these
equations. The particle fluid-mass is $m_{p}=\rho_{c}\forall_{p}$, with
$\forall{}_{p}$ the effective particle volume. Also, $C_{L}$ is the lift force
coefficient, and $\mathbb{\omega}$ is the vorticity field. Given that the
motion velocity of the test particle is $\mathbb{v}$, the acceleration of the
particle with respect to the comoving observer flows with STF is equal to:

$\mathbb{a}=\frac{d\mathbb{v}}{dt}$ (26)

####  V.2.1 Hydrodynamic forces due to the flow field around the static point
sink:

The radial acceleration field is calculated by the material derivative of the
flow field in spherical coordinates, which for a steady flow is as follows
[33]:

$\frac{D{u}_{r}}{Dt}={u}_{r}\frac{\partial{u}_{r}}{\partial
r}+\frac{{u}_{\theta}}{r}\frac{\partial{u}_{r}}{\partial\theta}+\frac{{u}_{\phi}}{r\sin\theta}\frac{\partial{u}_{r}}{\partial\phi}$
(27)

By placing the velocity field of the static point sink, Eq. 7, in Eq. 27,
we’ll have

$\frac{D\mathbb{u}_{stf}}{Dt}=\frac{-GM}{r^{2}}\hat{r}.$ (28)

Fig. 3 shows the magnitude and direction of the STF acceleration field
correspond to the gravity field. By placing Eqs. 26 and 28 in Eq. 22, the
forces acting on the test particle around the static point sink are acquired
as follows ($\rho_{\text{stf }}$ is STF density):

Figure 3: STF acceleration field around a point sink

$\displaystyle\rho_{stf}\forall_{p}\left(\frac{d\mathbb{v}}{dt}\right)$
$\displaystyle=\rho_{stf}\forall_{p}\left(\frac{-GM}{r^{2}}\right)$ (29)
$\displaystyle+\frac{1}{2}\rho_{stf}\forall_{p}\left(\frac{-GM}{r^{2}}-\frac{1}{2}\rho_{stf}\forall_{p}\frac{d\mathbb{v}}{dt}\right).$

By defining the effective inertial mass,
$m=\left(3/2\right)\rho_{stf}\forall_{p}$, we have

$F=m\left(\frac{d\mathbb{v}}{dt}\right)=m\left(\frac{-GM}{r^{2}}\right).$ (30)

The forces obtained in Eq. 30 correspond with the forces acting on the moving
test particle around the static point sink.

####  V.2.2 Hydrodynamic forces due to the flow field around the rotary point
sink

Now, we study the forces acting on a test particle in the vicinity of a
rotating point sink. Rotation of the point sink produces a vortex field. In
[34, 35], it was shown that there is a connection between acceleration and
vorticity fields in an inviscid and compressible fluid flow. In mentioned
works, acceleration and vorticity fields for the inviscid compressible flow
are defined as

$\mathbb{E}=(\mathbb{u}.\nabla)\mathbb{u}\
,\quad\boldsymbol{\omega}=\nabla\times\mathbb{u}\ .$ (31)

Which $\mathbb{u}$ is the velocity of the fluid, $\mathbb{\omega}$ is the
vorticity field, $\mathbb{E}$ is the acceleration field. The acceleration
field is obtained in the Eq. 28, and the relation between vorticity and
acceleration fields. Suppose that the angular velocity of the rotating sink is
very small so that the specific angular momentum approaches zero
$(a\rightarrow 0)$. In this case, vorticity and velocity fields of STF around
a rotating point sink, respectively $\mathbb{\omega}_{stf}$ and
$\mathbb{u}_{stf}$, will satisfy the below equations.[34]

$\displaystyle\nabla.\boldsymbol{\omega}_{stf}=0,$ (32)
$\displaystyle\nabla\cdot\frac{D\mathbb{u}_{stf}}{Dt}=M,$ (33)
$\displaystyle\nabla\times\frac{D\mathbb{u}_{stf}}{Dt}+\partial_{t}\boldsymbol{\omega}_{stf}=0,$
(34) $\displaystyle
J=c_{s}^{2}\nabla\times\boldsymbol{\omega}_{stf}-\partial_{t}\left(\frac{D\mathbb{u}_{stf}}{Dt}\right).$
(35)

These equations are analogous to the Maxwell equations and its other analogous
gravitoelectromagnetic equations. The vortex field around the rotating point
sink obtains from the above equations as

$\boldsymbol{\omega}_{stf}=\frac{GI}{2c_{s}^{2}r^{3}}\left(\mathbb{\Omega}-{3(\mathbb{\Omega}\cdot\hat{r})\hat{r}}\right).$
(36)

To calculate the hydrodynamic forces applied to the test particle around the
rotating point sink, we have to place Eqs. 26, 28 and 36, in Eq. 22. If we
assume $C_{l}=6$ for the STF flow, we will have

$\displaystyle F=$
$\displaystyle\left(\frac{3}{2}\right)\rho_{fst}\forall_{p}\left(\frac{d\mathbb{v}}{dt}\right)=\left(\frac{3}{2}\right)\rho_{stf}\forall_{p}\left(\frac{-GM}{r^{2}}\right)$
(37)
$\displaystyle+4\left(\frac{3}{2}\right)\rho_{c}\forall\left[\mathbb{v}\times\left(\frac{GI}{2c_{s}^{2}r^{3}}(\mathbb{\Omega}-{3(\mathbb{\Omega}\cdot\hat{r})\hat{r}})\right)\right].$

Again, we use the effective inertial mass definition,
$m=\left(3/2\right)\rho_{stf}\forall_{p}$ to reach the following equation:

$F=m\left(\frac{d\mathbb{v}}{dt}\right)=m\mathbb{E}+4m(\mathbb{v}\times\boldsymbol{\omega}_{stf}).$
(38)

Here, $\boldsymbol{\omega}_{stf}$ corresponds with the GEM field around the
rotating mass, and Eq. 38 corresponds with the quasi-Lorentz force applied to
the NASA probe satellite mentioned in Eq. 21. It can be said that the applied
forces to the test particle located in the STF around the rotating mass are
the same as the forces that are applied to the NASA probe satellite.

##  VI Discussion

In this paper, we introduced an innovative analogy between fluid dynamics and
phenomenological aspects of general relativity, such as black hole physics and
gravitoelectromagnetic (GEM). In this analogy, mass is analogous to a point
sink in a compressible superfluid we call space-time fluid (STF). Black holes
are also modeled as extreme point sinks, around which there are surfaces where
the velocity of the fluid reaches the speed of sound. We investigated the
hydrodynamic forces applied to a test particle floating in the STF flow using
the two-phase flow equations. Also, the acceleration and vorticity fields of
the STF flow in the vicinity of the point sink correspond to the gravitational
and the GEM fields around the rotary mass, respectively. This analogy could
increase our intuitive understanding of the nature of gravity and inertia and
provides us with the privilege of acquiring and using powerful fluid dynamics
tools such as CFD for modeling and research on gravitational phenomena such as
dark matter and Dark energy. According to our analogy, in this section, we
will present the foundational discussions about the Mach’s principle, the
equivalence principle of inertial and gravitational masses, and the lack of
information transfer from the event horizon.

### Equivalence principle and preferred frame

The equivalence between the inertial and the gravitational masses is one of
Einstein’s hypotheses for his general theory of relativity. However, in the
presented analogy, according to Eq. 30, the inertial and the gravitational
masses are proven to be the same. There is no difference in whether the
particle takes to be at rest and the STF has acceleration or the STF is
stationary, and the particle accelerates with respect to it. We get the same
results in both cases, but The latter case is related to the free fall frame
while the first case is related to the accelerator frame. Also, according to
Eq. 28, the acceleration field of the STF flow in the vicinity of the point
sink matches with the obtained gravitational field.

Figure 4: STF vorticity field around a rotating point sink [36]

### Mach’s principle

The origin and nature of the force of inertia have always been an issue for
physicists. Newton postulates the absolute space-time to answer this question.
Regardless, Mach believed in the contextuality in this matter. Mach’s
principle says that inertia is due to the distribution of the whole objects in
the universe. However, Einstein took a mediocre approach in which dynamic
space-time and the distribution of objects together determine the inertial
frame locally. In this research, the STF flow determines the local reference
frame. In this analogy, the relative acceleration between the STF flow and the
objects causes the production of a hydrodynamic force that corresponds with
the inertial force acting on the accelerating mass.

### Information discontinuity in fluid and black hole

There are unique features on the event horizon and ergosurface around
stationary and rotating black holes. For example, light cannot leave the
surface of the black hole event horizon. Therefore, the information inside the
black hole cannot leave the event horizon surface. In the science of fluid
mechanics, there is a discontinuity of information at sonic surfaces because
the flow velocity at those surfaces is equal to the velocity of the
propagation of acoustic waves. According to recent research on acoustic
metrics, the sonic surfaces created inside the converging nozzle correspond
with the event horizon surfaces of black holes. Based on this analogy, we
calculated the sonic surfaces around the strong point sink that corresponds
with the event horizon and the ergosurface around the black holes. This
analogy creates an excellent intuitive understanding of the event horizon and
ergosphere for us. Also, the properties of the event horizon and the
ergosurface around the black hole could be explored.

##  VII Conclusions

This study attempted to investigate the analogy between fluid dynamics and GR.
With the help of this analogy, we were able to calculate the relation between
the time dilation and the STF flow field. It is shown that the sonic surfaces
created around the stationary and rotating point sink correspond with the
event horizon and ergosurface around the stationary and rotating black hole.
The hydrodynamic forces entered into a test particle placed inside the STF
flow field were calculated to validate the obtained flow field equations. The
principle of equivalence was discussed due to results that show the
acceleration of the fluid flow around the point sink is the same as the
gravitational acceleration. Finally, the Mach’s principle was discussed,
assuming the STF flow as a non-rigid reference. One of the results of this
analogy is an increase in the intuitive understanding of gravity and black
holes. In addition, we can use the CFD tools for dark energy and dark matter
problems and get a clearer picture of the structure inside the black holes.[1]

###### Acknowledgements.

We sincerely thank Y. Jafari, S. Alapour, M. S. Nomani, H. Fazeli, H. Afshar,
H. Niasari, S. Shams, N. Riazi, S. Shahsavar, R. Ahmadvand, K. Goharian, F.
Rahmani, M. A. Rafsanjani and Ghadir Jafari for their guidance and assistance.

## References

  * Klein [1973] M. J. Klein, The development of boltzmann’s statistical ideas, in  _The Boltzmann Equation_ (Springer, 1973) pp. 53–106.
  * Mashhoon  _et al._ [2001] B. Mashhoon, F. Gronwald, and H. I. Lichtenegger, Gravitomagnetism and the clock effect, in  _Gyros, Clocks, Interferometers…: Testing Relativistic Graviy in Space_ (Springer, 2001) pp. 83–108.
  * Barceló  _et al._ [2011] C. Barceló, S. Liberati, and M. Visser, Analogue gravity, Living reviews in relativity 14, 1 (2011).
  * Brout  _et al._ [1995] R. Brout, S. Massar, R. Parentani, and P. Spindel, Hawking radiation without trans-planckian frequencies, Physical Review D 52, 4559 (1995).
  * Unruh [1981] W. G. Unruh, Experimental black-hole evaporation?, Physical Review Letters 46, 1351 (1981).
  * Jacobson [1995] T. Jacobson, Thermodynamics of spacetime: the einstein equation of state, Physical Review Letters 75, 1260 (1995).
  * Verlinde [2011] E. Verlinde, On the origin of gravity and the laws of newton, Journal of High Energy Physics 2011, 1 (2011).
  * Padmanabhan [2011] T. Padmanabhan, The hydrodynamics of atoms of spacetime: Gravitational field equation is navier–stokes equation, International Journal of Modern Physics D 20, 2817 (2011).
  * Cadoni [2004] M. Cadoni, Acoustic analogues of two-dimensional black holes, Classical and Quantum Gravity 22, 409 (2004).
  * Mandelberg and Witten [1962] H. I. Mandelberg and L. Witten, Experimental verification of the relativistic doppler effect, JOSA 52, 529 (1962).
  * Hay  _et al._ [1960] H. J. Hay, J. P. Schiffer, T. E. Cranshaw, and P. A. Egelstaff, Measurement of the red shift in an accelerated system using the mössbauer effect in fe 57, Physical Review Letters 4, 165 (1960).
  * Frisch and Smith [1963] D. H. Frisch and J. H. Smith, Measurement of the relativistic time dilation using $\mu$-mesons, American Journal of Physics 31, 342 (1963).
  * Bailey  _et al._ [1977] J. Bailey, K. Borer, F. Combley, H. Drumm, F. Krienen, F. Lange, E. Picasso, W. Von Ruden, F. Farley, J. Field, _et al._ , Measurements of relativistic time dilatation for positive and negative muons in a circular orbit, Nature 268, 301 (1977).
  * Hafele and Keating [1972] J. C. Hafele and R. E. Keating, Around-the-world atomic clocks: Observed relativistic time gains, Science 177, 168 (1972).
  * Note [1] Here, the Kerr metric and consequently the flow field of rotating masses have been written in the Boyer-Lindquist coordinates. Hence, the radial $\hat{r}$ and transverse $\hat{\phi}$ components are the confocal ellipses and hyperbolas in the ellipsoidal coordinate.
  * Rosen [1971] N. Rosen, The complete schwarzschild solution, Annals of Physics 63, 127 (1971).
  * Weinberg [1972] S. Weinberg,  _Gravitation and cosmology: principles and applications of the general theory of relativity_ (1972).
  * Anderson [1990] J. D. Anderson,  _Modern compressible flow: with historical perspective_ , Vol. 12 (McGraw-Hill New York, 1990).
  * [19] R. Davie, Interior structure of the kerr black hole.
  * Heaviside [1893] O. Heaviside, A gravitational and electromagnetic analogy, The Electrician 31, 281 (1893).
  * Tisserand [1890] F. Tisserand, Sur le mouvement des planètes, en supposant l’attraction représentée par l’une des lois électrodynamiques de gauss ou de weber, Comptes Rendus de l’Académie des Sciences (Paris) 100, 313 (1890).
  * Behera and Barik [2018] H. Behera and N. Barik, A new set of maxwell-lorentz equations and rediscovery of heaviside-maxwellian (vector) gravity from quantum field theory, arXiv preprint arXiv:1810.04791 (2018).
  * Mashhoon [2003] B. Mashhoon, Gravitoelectromagnetism: a brief review, arXiv preprint gr-qc/0311030 (2003).
  * Mashhoon  _et al._ [1984] B. Mashhoon, F. W. Hehl, and D. S. Theiss, On the gravitational effects of rotating masses: the thirring-lense papers, General relativity and gravitation 16, 711 (1984).
  * Thorne [1988] K. S. Thorne, Gravitomagnetism, jets in quasars, and the stanford gyroscope experiment, Near zero: new frontiers of physics , 573 (1988).
  * [26] T. Phillips, Nasa announces results of epic space time experiment.
  * Tsuji [2011] Y. Tsuji, Particle-fluid interaction, in  _Multiphase Flows with Droplets and Particles_ (CRC Press, 2011) pp. 73–134.
  * Maneshian  _et al._ [2018] B. Maneshian, K. Javadi, and M. T. Rahni, Bubble dynamics in rotating flow under an accelerating field, Physics of Fluids 30, 082108 (2018).
  * Salari  _et al._ [2013] A. Salari, M. Karmozdi, R. Maddahian, and B. Firoozabadi, Analytical study of single particle tracking in both free and forced vortices, Scientia Iranica 20, 351 (2013).
  * Putra and Lucas [2020] R. A. Putra and D. Lucas, Modeling of the free-surface vortex-driven bubble entrainment into water, Water 12, 709 (2020).
  * Auton [1987] T. Auton, The lift force on a spherical body in a rotational flow, Journal of fluid Mechanics 183, 199 (1987).
  * Van Nierop  _et al._ [2007] E. A. Van Nierop, S. Luther, J. J. Bluemink, J. Magnaudet, A. Prosperetti, and D. Lohse, Drag and lift forces on bubbles in a rotating flow, Journal of Fluid Mechanics 571, 439 (2007).
  * White and Majdalani [2006] F. M. White and J. Majdalani,  _Viscous fluid flow_ , Vol. 3 (McGraw-Hill New York, 2006).
  * Kambe [2010] T. Kambe, A new formulation of equations of compressible fluids by analogy with maxwell’s equations, Fluid dynamics research 42, 055502 (2010).
  * Kambe [2014] T. Kambe, On fluid maxwell equations, in  _Frontiers of fundamental physics and physics education research_ (Springer, 2014) pp. 287–295.
  * [36] studylib, Precision measurement of planetary gravitomagnetic field and future chinese pathfinder mission.


# TIDE: Temporally Incremental Disparity Estimation via Pattern Flow in
Structured Light System

Rukun Qiao1, Hiroshi Kawasaki2 and Hongbin Zha1 Manuscript received: October,
26, 2021; Revised January, 7, 2022; Accepted Januaray, 28, 2022.This paper was
recommended for publication by Editor Sven Behnke upon evaluation of the
Associate Editor and Reviewers’ comments.1Rukun Qiao and Hongbin Zha are with
the Key Lab of Machine Perception (MOE), School of Artificial Intelligence,
Peking University, Beijing, China<EMAIL_ADDRESS><EMAIL_ADDRESS>Kawasaki is with the Graduate School and Faculty of
Information Science and Electrical Engineering, Kyushu University, Fukuoka,
Japan<EMAIL_ADDRESS>Object Identifier (DOI):
10.1109/LRA.2022.3150029. Copyright: © 2022 IEEE.

###### Abstract

We introduced Temporally Incremental Disparity Estimation Network (TIDE-Net),
a learning-based technique for disparity computation in mono-camera structured
light systems. In our hardware setting, a static pattern is projected onto a
dynamic scene and captured by a monocular camera. Different from most former
disparity estimation methods that operate in a frame-wise manner, our network
acquires disparity maps in a temporally incremental way. Specifically, We
exploit the deformation of projected patterns (named _pattern flow_) on
captured image sequences, to model the temporal information. Notably, this
newly proposed pattern flow formulation reflects the disparity changes along
the epipolar line, which is a special form of optical flow. Tailored for
pattern flow, the TIDE-Net, a recurrent architecture, is proposed and
implemented. For each incoming frame, our model fuses correlation volumes
(from current frame) and disparity (from former frame) warped by pattern flow.
From fused features, the final stage of TIDE-Net estimates the residual
disparity rather than the full disparity, as conducted by many previous
methods. Interestingly, this design brings clear empirical advantages in terms
of efficiency and generalization ability. Using only synthetic data for
training, our extensitve evaluation results (w.r.t. both accuracy and
efficienty metrics) show superior performance than several SOTA models on
unseen real data. The code will be available on
https://github.com/CodePointer/TIDENet soon.

###### Index Terms:

Range sensing, RGB-D perception, Structured light systems, Active sensor, Deep
learning methods.

##  I Introduction

Various variants of disparity estimation methods built upon monocular
structured light systems [1, 2] have been proposed in the literature. They
have drawn wide attention from both academia and industry due to their
promises in important application areas like augmented reality, sport
analysis, and medical robots. However, dynamic scene acquisition is still an
unsolved challenging problem. While object scanning in static scenes can
leverage rich information provided by multiple patterns, this setting is not
suitable for dynamic scenes due to the difficulty of feature matching. As
such, for dynamic scenes, single-pattern structured light systems are
preferred, which is refereed to as _one-shot scan_ in the literature. Since
input sequences in the one-shot scan setting tend to be sparse and unstable,
recently several learning-based methods are proposed and widely recognized as
a promising alternative solution to address these two challenges [3, 4, 5].

Although these deep neural network (DNN) models have achieved promising
accuracy, there are two obstacles preventing us from training robust models
with large-scale real-world datasets: 1. getting the ground truth of dense
correspondences of dynamic scenes is virtually impossible; 2. even if we could
get these ground truths, the data would be biased towards the specific pattern
and/or device in the hardware design. Therefore, training on the synthetic
data and testing on real-world scenarios is a more practical choice. In such a
situation, the generalization ability of the model is the most significant
factor to consider.

In this paper, the focus is to exploit the temporal coherence in the image
sequence to improve the generalization ability, while guaranteeing efficiency.
To this end, we propose a neural network named TIDE-Net (Temporally
Incremental Disparity Estimation). Rather than estimating a disparity map for
each frame from scratch, TIDE-Net conducts residual estimation based upon the
former frame. Notably our method can model even earlier frames, through hidden
layers in a recurrent architecture. This incrementally updated scheme allows
the network size to be concise while maintaining accuracy.

Utilizing temporal coherence between frames in a video has been studied
intensively for correspondence search [6, 5]. However, the naive practice of
aggregating features extracted from frames by concatenation ignores the pixel
displacement caused by scene motion, and thus, it usually results in low
accuracy [7]. To address this issue, we propose to leverage a novel
formulation named _pattern flow_ , which is the pattern deformation in the
observed image and a constrained version of generic optical flow. Although a
former study has mentioned the pattern flow concept [8], they treat it as a
feature instead of explicitly using the pattern flow for correspondence search
between a captured image and a projected pattern.

In contrast, we analyzed the pattern flow and found that it has a unique
feature absent in the generic optical flow formulation: The disparity changes
between frames can be computed explicitly given the estimated pattern flow.
Once the pattern flow is estimated, we can pixel-wisely warp both disparity
and hidden layer from the former frame. Thus, features and information can be
aligned along the temporal space credit to this operation. Experimental
results illustrated that our method required less computational cost than SOTA
methods while guaranteeing better accuracy and generalization ability.

In summary, our contributions are threefold:

  * •

We propose an incremental disparity estimation framework based on TIDE-Net
which fully utilized the local and sequential nature of images from dynamic
scenes. By focusing on the incremental non-linear portion, parameter size can
be reduced while guaranteeing accuracy and efficiency.

  * •

We propose a novel algorithm to estimate pattern flow, which represents
correspondences of projected pattern between adjacent frames of structured
light systems, and the information is fed into TIDE-Net.

  * •

Comprehensive experiments proved that, although TIDE-Net is only trained by
synthetic data and evaluated by real one without any adaptation, our method
performs better accuracy than several SOTA models as well as computational
cost, showing efficient domain-invariant generalization ability of the method.

##  II Related Work

The active light systems can achieve robust and accurate depth information
even if there is no texture or high-frequency shape on the object surface.
Thus, they are considered a practical solution for 3D scan [1, 2]. For
reconstruction on dynamic scenes, many spatial coding techniques have been
proposed based on pattern decoding and global matching [9, 10, 11, 8]. The
pattern can be binary dots [12], strips [13], grids [14], etc. These methods
focus on encoding coordinate features into one projected pattern, and then
extracting features from observed images by decoding on the spatial domain.
The projected pattern needs to be recognizable in the images; thus, the
computed disparity map is always sparse.

As for the utilization of temporal information, some methods embedded spatial
features in the multiple different patterns projected periodically. To
compensate moving parts in the scene for a one-shot scan, several methods are
proposed, such as detecting individual motion [6, 15], directly computing
disparity of the scene by the blurs of line-based patterns [8], or utilizing
DNN [5]. In the method [5], they propose a geometric loss between image pairs
for their model training to improve the spatio-temporal consistency. The loss
requires objects to have a rigid motion, and related position $(R,t)$ is
needed for model training. Our research does not have rigid motion
assumptions, and no registration is needed for training.

Stereo matching and optical flow focusing on the correspondences between image
pairs are also related to our problem since a pattern projector is optically
the same as a camera. Many networks designed for those problems can be applied
to structured light systems with several additional modifications [16, 17,
18]. We exploit ideas from the stereo methods to our rectified mono-camera
setup, where the reference pattern is warped to the observed image by
correlation layer for disparity estimation inside the network structure.

The domain generalization ability of networks is also an important issue and
drawing wide attention recently. Some methods apply self-supervised training
[5, 19] or fine-tuning when transfer to a new data domain. However, such
solutions require a large number of data from the target domain with or
without ground truth. Some methods apply online adaptation techniques to
update the network during the inference process [20, 21], while the BP process
results in the loss of efficiency. [22] proposed DSMNet equipped with a novel
normalization layer and graph-based filtering layer to improve the
generalization ability for passive stereo matching. In contrast, we do not
design a specific framework in our network. We utilize the temporal coherence
from the input sequence to guarantee the accuracy and efficiency of our
network.

Figure 1: Pattern flow from adjacent image pairs. We denote red and blue
colors for different moving directions, while intensity stands for the value.
The forehead is moving back and the jaw is moving forward. Top-right is a
sketch map for the geometric property of the pattern flow in structured light
systems.

Figure 2: The general framework of our method. The initialization process is
only applied for the first frame, then we incrementally estimate the disparity
map for every frame using TIDE-Net.

Figure 3: The architecture of the TIDE-Net. | Optical Flow | Pattern Flow
---|---|---
Definition | 3D point | Light ray
Degree of freedoms | 2 | 1
Physical correspondence | Yes | No
Texture-less problem | Yes | No
Intensity model | Passive | Active
TABLE I: The comparison between classic optical flow and proposed pattern
flow.

##  III Methods

In this part, we will illustrate our method in detail. We first go through the
definition of incremental disparity computing from an image sequence in
structured light systems. Then, we explain the definition and analysis of
pattern flow. Finally, we introduce the entire framework combined with the
incremental warping technique.

###  III-A Problem Definition

Our research aims at the 3D depth estimation from sequential frames where
scenes are projected by a static pattern projector and captured by a single
camera. For such systems, the pattern projector can be regarded as a second
camera considering its image plane as the reference pattern. Therefore, we can
define disparity for the camera similar to stereo vision accordingly. We
assume that the camera and the projector are calibrated and rectified by pre-
processing. Generally, scenes are captured by the camera, and captured images
are fed into the algorithm to estimate disparity maps for each frame. Here, we
denote $\mathrm{\mathbf{I}}^{t}$ as the input camera image and
$\mathrm{\mathbf{D}}^{t}$ as the output disparity, where $t$ represents the
frame number. The designed pattern is denoted as $\mathrm{\mathbf{P}}$. In our
system, a pseudo-random dot pattern [23] is used, which guarantees patch
uniqueness in each row and is repeated periodically to the vertical direction.

We assume online disparity estimation for our system, that is, for each input
frame $\mathrm{\mathbf{I}}^{t}$, all the history information before time $t$
is available for calculation, denoted as $H^{t-1}$. Thus, the disparity of the
current frame can be estimated through incremental processing:

$\mathrm{\mathbf{D}}^{t}=\mathrm{\mathbf{D}}^{t-1}+g\left(\mathrm{\mathbf{I}}^{t},\mathrm{\mathbf{P}},H^{t-1}\right),$
(1)

where we assume $t>0$. Our incremental process, denoted as function $g$,
focuses on the residual estimation instead of the full disparity computation
for each frame. This usually helps to improve accuracy and efficiency. Rather
than some method utilizing temporal information as an additional dimension for
temporal-window-based batch matching [24], our method focuses on the step-by-
step updating processing. We utilize a deep neural network (DNN) model with a
temporal module to find the function $g$, where the hidden layer is passed
through the sequence as the history information $H$. This is a straight-
forward idea and has been developed in stereo matching problems.

However, simply concatenating multiple frames usually ignores the motion of
objects in the scene, resulting in the overfitting on specific motion features
in the synthetic dataset. To achieve the network to learn the motion-invariant
consistent features explicitly, we pre-warp the history information consisting
of previous frames to retrieve accurate correspondences between adjacent
frames. We proposed to use pattern flow between frames to accomplish the
warping. The pattern flow can be robustly calculated locally with a simple but
effective algorithm with our proposed method. From the ablation study in Sec.
IV, we can find that the pattern flow can improve the performance compared to
simply passing the hidden layer directly.

###  III-B Pattern Flow in Structured Light Systems

In a structured light system, the projector emits a reference pattern into the
3D space. The projected pattern is deformed by the shape of the object’s
surface and observed by the camera. If objects in the scene are moving, the
observed patterns are deformed, because the shape (depth) of the scene also
changes as shown in Fig. 1(left), where two adjacent frames are shown to
visualize the motion of the projected pattern. We define this kind of light
flow caused by projected pattern as pattern flow, denoted as
$\mathrm{\mathbf{F}}^{t}$, which represents the motion of the projected
patterns. This concept was mentioned by Furukawa et al. , who use strip
pattern flow to estimate depth for fast-moving object [8]. In our research, we
put forward the pattern flow defined on the pixel level to propagate
information precisely.

We first give out the geometric relationship between the disparity map and the
pattern flow. Considering a light ray projected from
$\mathrm{\mathbf{P}}(x_{p},y)$ on reference pattern to the 3D position
$\mathbf{p}(X,Y,Z)$ at time $t$. In the next frame, the projected position
moves to a new place caused by the scene motion, denoted as
$\mathbf{p}(X+\Delta X,Y,Z+\Delta Z)$. Notice that these two positions are not
guaranteed to be the same physical point, but they are on the same light ray
projected from the projector. Therefore the $\mathbf{p}$ can only move along
the epipolar plane, and we do not need to consider vertical direction on the
rectified image. We denote $f$ as the focal length, $b$ as the baseline, and
$(x,y)$ as the observed pixel in camera space. Then we have

$\left\\{\begin{split}&x=\frac{f}{Z}X\\\
&x_{p}=\frac{f}{Z}(X-b)\end{split}\right..$ (2)

By taking the derivative of Eq.2, we have

$\left\\{\begin{split}\frac{\mathrm{d}x}{\mathrm{d}t}&=\frac{f}{Z}\frac{\mathrm{d}X}{\mathrm{d}t}-\frac{fX}{Z^{2}}\frac{\mathrm{d}Z}{\mathrm{d}t}\\\
\frac{\mathrm{d}x_{p}}{\mathrm{d}t}&=\frac{f}{Z}\frac{\mathrm{d}X}{\mathrm{d}t}-\frac{f(X-b)}{Z^{2}}\frac{\mathrm{d}Z}{\mathrm{d}t}\end{split}\right..$
(3)

Since the two projected positions belong to the same projected ray, we have
$\frac{\mathrm{d}x_{p}}{\mathrm{d}t}=0$. The pattern flow can also be denoted
as a horizontal vector $\mathrm{\mathbf{F}}(x,y)=(u,0)$, where
$u=\frac{\mathrm{d}x}{\mathrm{d}t}$. The Eq. 3 can be further simplified as

$u=-\frac{fb}{Z^{2}}\frac{\mathrm{d}Z}{\mathrm{d}t}.$ (4)

On the other hand, suppose $d$ is the disparity value for pixel
$\mathrm{\mathbf{D}}(x,y)$. Therefore we have $d=\frac{fb}{Z}$ according to
the epipolar geometry. Take the derivative of $d$ we have

$\frac{\mathrm{d}d}{\mathrm{d}t}=-\frac{fb}{Z^{2}}\frac{\mathrm{d}Z}{\mathrm{d}t}.$
(5)

Thus from Eq. 4, 5 we can have:

$u=\frac{\mathrm{d}d}{\mathrm{d}t},$ (6)

which means that the estimated pattern flow is the change of the disparity
between images. The geometric relationship is shown in Fig 1(top-right) for
better understanding. With the help of pattern flow, the disparity can be
propagated between frames according to Eq.5. In our framework, we propagate
both disparity map and hidden layer to eliminate the motion factor between
frames.

In structured light systems, the intensity is decided by the projected ray and
object texture together. However, in our practice, the projected light always
holds a dominant position in IR image in most cases. Therefore, the local and
fast Lucas-Kanade method along the epipolar line could be applied for pattern
flow calculation.

Notice that although we apply a classical method from optical flow
calculation, the pattern flow has several differences to classic optical flow
in the passive sensor:

  * •

Firstly, the optical flow is the motion for physical points, while the pattern
flow is the displacement of projected light rays, which means it only appears
along the 1D epipolar line, rather than the 2D image.

  * •

Secondly, pattern flow in active light is always easier to calculate, for the
pattern always has enough features for local matching. No semi-global
optimization is needed for dense pattern flow calculation.

  * •

Lastly, although the pattern flow has the before-mentioned advantage to
optical flow, it does not guarantee the tracked points in two frames belong to
the same physical points. Pattern flow lacks physical correspondence
information and cannot be used for motion estimation. However, it can be used
for disparity propagation from the previous disparity.

The differences between pattern flow and optical flow are listed in Table I
for better illustration.

###  III-C TIDE Network

Given an observed image sequence clip $\mathrm{\mathbf{I}^{t-k:t}}$ from the
camera and pre-designed pattern $\mathrm{\mathbf{P}}$, we estimate the
disparity sequence with iterative updating process frame by frame. Fig. 2
shows an overview of our approach, and the network architecture is shown in
Fig. 3. We now describe all the components in detail.

####  III-C1 Feature extraction

We concatenate the input image $\mathrm{\mathbf{I}}^{t}$ and the normalized
image from local contrast normalization(LCN) method [16, 5] as the input of
feature extraction. For each frame inside the temporal window, two encoders
with the same architecture are applied to the input image
$\mathrm{\mathbf{I}}^{t}$ and pattern $\mathrm{\mathbf{P}}$. We use the
similar feature extraction layers mentioned in [18] but reduce the parameter
numbers. Then, the correlation pyramid is built which will be indexed by
updating block.

####  III-C2 Update block

We use the similar recurrent updating block in [18] for disparity refinement,
where a ConvGRU block is applied to provide disparity residual estimation. The
input of ConvGRU block is concatenated from 3 parts: The warped disparity
$\mathrm{\mathbf{\hat{D}}}^{t-1}$, the feature from the context encoder, and
the retrieved correlated features from the correlation pyramid given the
predicted disparity. The ConvGRU block outputs disparity at 1/8 resolution. We
then upsample the disparity map to full resolution by taking an upsampling
mask as the interpolation weights. Although the network used in [18] can
acquire accurate results on optical flow estimation, it is utilized between
image pairs, resulting in the lack of temporal information usage. Besides, the
infer speed is also harmed by multiple recurrent iterations. Thus, we improve
the framework in two main parts: 1. We passed the hidden layer along the
different frames to provide temporal consistency; 2. Instead of taking
multiple iterations in every frame, we only have one iteration for each frame,
forcing the network to get information from the hidden layer. With those
improvements, the computing time could be reduced while guaranteeing accuracy.

####  III-C3 Warping by pattern flow

As mentioned at the end of Sec.III-A, simply passing the previous hidden layer
and disparity map may result in the bias by objects motion. Therefore, we
first calculate the pattern flow $\mathrm{\mathbf{F}}^{t}$ between
$\mathrm{\mathbf{I}}^{t}$ and $\mathrm{\mathbf{I}}^{t-1}$, then warp the
hidden layer $h^{t-1}$ and disparity map $\mathrm{\mathbf{D}}^{t-1}$ (The ”w”
operator in Fig.2 between frames). We calculate the pattern flow by applying
Lucas–Kanade method for every pixel in $1/8$ resolution, which is the input
resolution for the update block. Thanks to the feature-rich projected pattern,
the pattern flow can be calculated without any semi-global optimization. The
dense and accurate pattern flow map could be acquired in a very short time.

####  III-C4 Initialization for the first frame

Since our method is an incremental function, we need the initial value of the
first frame. In practice, we found a very rough disparity estimation is enough
for TIDE-Net to reconstruct the accurate disparity within several frames.
Thus, we train a small U-Net [5] to give the initial output for the very first
frame. The initialization process is only needed for the very first frame of
the image sequence.

###  III-D Loss Functions

We applied L1 loss for supervised training on the synthetic dataset given the
disparity ground truth:

$L=\sum_{i=t-N+1}^{t}\left|\mathrm{\mathbf{D}}^{i}-\mathrm{\mathbf{D}}^{i}_{GT}\right|$
(7)

where the $N$ denotes the temporal window size for the training.

Figure 4: The experiment platform for data collection.

Figure 5: The experiment results for visualization. All the models are trained
on the original synthetic data, the first row, then evaluated on the others.
We show the disparity map (upper row) and residual error map(lower row) for
comparison. The non-projected area is masked out manually and excluded from
the quantitative analysis.

Figure 6: Our experiment result on the synthetic sequence. With several
processing frames(about 30-50), our method can achieve an accurate result.

##  IV Experiments

In this section, we systematically evaluate our method compared with the
previous technique. We first introduce the experiment platform we used for
data collection, then we give out the details of the training process.
Finally, we present the evaluation result on several unseen data to illustrate
our method’s generalization ability and the contribution of temporal
information.

###  IV-A Experiment Platform

We use the realsense D435i[23], an active stereo system equipped with two
infrared cameras and one emitter, to collect real data for evaluation. In our
research, we only utilize one camera for evaluation. We also manually increase
the baseline to 218mm by utilizing another D435i device, as shown in Fig. 4,
for we find the original baseline is not sensitive enough to the object
motion. We combine one camera from the left D435i and the emitter from another
as the final mono-camera structured light system. The device collects and
stores the images with fps=24.

The pattern we used for our experiments is the pseudo-random dots that were
originally applied in realsense D435i. Since they do not provide the pre-
designed pattern officially, we calibrate the pattern by moving the device
forward and backward towards a white wall. Then we apply rectification to the
camera image and calibrated pattern to get the data for evaluation. We project
that pattern in the scene and captured the images with the left IR camera. All
the methods take the IR image sequence as the network input.

###  IV-B Training Schedule and Implementation Details

Once we calibrated the devices used for data collection, we can generate the
synthetic dataset for model training and then evaluate that on the real
collected data. To generate image sequences of dynamic scenes with non-rigid
motion, we randomly sample 4 objects from ShapeNet [25], each of which has an
individual random rigid motion. The camera and projector are placed towards
those objects, with a white plane as the background. Then, we utilized the
structured light renderer provided in [5] for infrared image generation with
the pre-calibrated parameters used for real data collecting. With those
settings, we generate 2048 sequences with 32 frames for each. We also generate
another 512 sequences for the test, shown in Fig. 5 (Original Dataset).

For model training, we use Adam optimizer with the learning rate at $10^{-3}$.
We trained our model on one 2080Ti GPU with the batch size $N=1$. The temporal
window size is set to 8, for this is the maximum capacity of the GPU memory.

###  IV-C Qualitative Result on Real Data

For non-rigid data evaluation, we take a white wall as the background, and
collect the images of the non-rigid moving human face. We select several SOTA
methods in stereo matching and optical flow for comparison with some small
modifications to structured light systems: 1. The Active Stereo Net [16] for
active stereo; 2. the FADNet [26] for passive stereo matching; 3. the RAFT
[18] framework for optical flow, and 4. CTD [5] framework using temporal
information for structured light systems. Apart from those deep-learning
methods, we also add the Spacetime stereo [24] for comparison. All the
comparison baselines take single frame as input during the evaluation while
ours takes multi-frames for incremental estimation. With the help of temporal
information, ours can perform especially better on the occluded part with a
better prediction from history, as shown in Fig. 5 (Real-world Non-rigid
Motion).

###  IV-D Quantitative Result

To better illustrated our model’s performance, we test our method on several
data with the given ground-truth for quantitative analysis. We compute two
types of metrics for quantitative comparison on every estimated disparity
maps: The percentage of pixels with disparity error larger than
$t\in\\{1.0,2.0,5.0\\}$, denoted as $o(t)$; and the average L1 loss, denoted
as $avg$. We compute those metrics on the whole disparity sequence except the
very first frame and calculate the average as the final performance.

Original Dataset. We first evaluate our method on the same data domain where
the models are trained. Same generation parameters are applied according to
Sec. IV-B, and the result is shown in Tab. II. We can see all the methods can
achieve accurate results after the supervised training. Our method does not
outperform others but also achieves good accuracy.

TABLE II: Evaluation Result: Same Configuration to the Training Dataset) Original Dataset | o(1.0) | o(2.0) | o(5.0) | avg
---|---|---|---|---
Spacetime Stereo | 17.68 | 14.76 | 13.76 | 15.140
CTD | 24.71 | 11.29 | 6.78 | 2.030
ActiveStereoNet | 12.06 | 8.41 | 6.24 | 2.391
FADNet | 7.32 | 5.47 | 3.25 | 0.812
RAFT | 7.76 | 4.60 | 2.59 | 0.916
Ours | 7.93 | 4.89 | 2.81 | 0.955
Synthetic Indoor Scenes | o(1.0) | o(2.0) | o(5.0) | avg
Spacetime Stereo | 55.37 | 38.88 | 30.38 | 39.222
CTD | 99.07 | 98.13 | 95.52 | 105.999
ActiveStereoNet | 77.69 | 66.43 | 53.30 | 50.083
FADNet | 99.30 | 98.63 | 96.79 | 106.427
RAFT | 57.73 | 47.59 | 39.84 | 56.417
Ours | 48.89 | 36.39 | 25.26 | 24.046
Real-World Rigid Motion | o(1.0) | o(2.0) | o(5.0) | avg
Spacetime Stereo | 74.86 | 55.23 | 36.88 | 25.772
CTD | 95.17 | 90.44 | 76.63 | 17.129
ActiveStereoNet | 76.46 | 57.33 | 32.94 | 9.618
FADNet | 96.39 | 92.93 | 83.94 | 14.457
RAFT | 76.01 | 52.84 | 22.60 | 7.666
Ours | 73.52 | 49.65 | 19.16 | 5.507

Real-world Rigid Motion. After the evaluation on the Original Dataset, we test
the performance on the unseen real-world data. For ground truth generation, we
first scan the accurate 3D information of several objects by a precise laser
scanner [27] offline. Then we place those objects on a rotation plate for data
collection. To register the video and scanned model, we use the aruco marker
[28] pasted on the plate to get the accurate position of objects in every
frame. We apply a hard mask to remove the background, for we only want to
focus on the reconstruction result of the object. Fig. 5 shows the estimated
disparity map, and Tab. II illustrate the quantitative comparison. Our method
outperforms all the SOTA methods with better boundary performance.

Synthetic Indoor Scenes. We also infer our model on another synthetic dataset
to test the generalization ability. We use the depth map from ICL-NUIM[29], a
synthetic dataset used for VO/SLAM problems where the camera moves freely
inside a room. We use four sequences from a living room for evaluation. We
first convert all the RGB images into gray ones, then we create the virtual
projector to project the pattern into every image. Different from the original
training dataset, these scenes are static and the camera has complex ego-
motion with large rotation and transportation. The evaluation result, shown in
Tab. II and Fig. 5, illustrate that our method can still get robust results
while others suffer a lot from the domain shifting problem.

TABLE III: Computing time comparison. All the methods are evaluated on a GTX1070 card. | FPS
---|---
CTD | 24.0
ActiveStereoNet | 21.7
FADNet | 3.5
RAFT | 16.5
Ours | 54.7

Efficiency comparison. We also compare the computing time of disparity
estimation, which is shown in Tab. III. The 3D cost volume in ActiveStereoNet
[16] and recurrent iteration module in RAFT [18] requires a longer time for
evaluation. In our method, we extend the iteration process into the temporal
space, resulting in a boost in efficiency. All the results are evaluated on a
GTX1070 card.

TABLE IV: Ablation Study for Pattern Flow (PF) and Temporal Hidden Layer ($h$). | o(1.0) | o(2.0) | o(5.0) | avg
---|---|---|---|---
Ours - $h$ \- PF | 77.55 | 55.51 | 23.58 | 9.811
Ours - PF | 76.18 | 51.49 | 20.53 | 6.825
Ours | 73.52 | 49.65 | 19.16 | 5.507

###  IV-E Ablation Study

To prove the contribution of the temporal module, we apply the ablation study
by removing pattern flow propagation and hidden layer passing. We take the
result from Real-World Rigid Motion data for comparison. We first removed the
pattern flow warping module(-PF). The previous disparity and hidden features
will pass directly to the next frame. By removing the warping, the accuracy
decreases. We then further remove the hidden layer(-$h$) and all the network
only takes previous disparity as the initial estimation. As listed in Tab. IV,
the accuracy decreased when removing the temporal propagation and pattern flow
warping.

We also show several estimated disparity maps in the same sequence of
”Synthetic Indoor Scenes” to illustrate the performance improvements along the
time. In Fig.6, our method can achieve better performance frame by frame in
the 30-50 frames from the beginning. Such accurate results can be maintained
in a relatively long sequence without any initialization processing. The TIDE-
Net takes pattern, a global reference in structured light systems, as the
input for every updating process. Therefore our method can be robust to the
error accumulation problem.

##  V Conclusion and Future Works

In this paper, we proposed TIDE-Net, a temporally incremental disparity
estimation network for mono-camera structured light systems. The network
utilized the temporal coherence for image sequence to improve accuracy and
efficiency. With the TIDE-Net’s help, disparity maps can be incrementally
estimated by focusing on the residual part, resulting in a smaller network
size and better generalization ability. We also first introduce and analyze
the pattern flow, an active light flow in structured light systems, which
could be used to propagate disparity and hidden layers between frames. With
the help of pattern flow, the TIDE-Net could focus on the local non-linear
part optimization and the accuracy could be further improved. Our method
achieves state-of-the-art accuracy on the unseen data. The code of our
research will be available on our project page soon.

Our framework also has several limitations for dynamic scene reconstructions.
For example, fast-moving objects and complex textures, which are classic
challenges in active sensor systems, will harm the correspondence finding in
our methods. Besides, our method relies on the detection of pattern moving.
Such moving may be very trivial when the baseline is too short and the
disparity change is relatively small. In such a situation, our method will
degrade to the naive temporal aggregation method.

In our future work, we plan to focus on the model online adaptation research.
By updating the network online, the model can adapt to different situations.
Therefore the model can perform better on long sequence processing and results
can be further improved.

## References

  * [1] J. Salvi, J. Pages, and J. Batlle, “Pattern codification strategies in structured light systems,” _Pattern Recognition_ , vol. 37, no. 4, pp. 827–849, 2004.
  * [2] J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” _Pattern Recognition_ , vol. 43, no. 8, pp. 2666–2680, 2010.
  * [3] S. Ryan Fanello, C. Rhemann, V. Tankovich, A. Kowdle, S. Orts Escolano, D. Kim, and S. Izadi, “Hyperdepth: Learning depth from structured light without matching,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5441–5450.
  * [4] S. Ryan Fanello, J. Valentin, C. Rhemann, A. Kowdle, V. Tankovich, D. Philip, and S. Izadi, “Ultrastereo: Efficient learning-based matching for active stereo systems,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6535–6544.
  * [5] G. Riegler, Y. Liao, S. Donné, V. Koltun, and A. Geiger, “Connecting the dots: Learning representations for active monocular depth estimation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7616–7625.
  * [6] Y. Taguchi, A. K. Agrawal, and O. Tuzel, “Motion-aware structured light using spatio-temporal decodable patterns,” in _European Conference on Computer Vision_ , 2012, pp. 832–845.
  * [7] A. Hosni, C. Rhemann, M. Bleyer, and M. Gelautz, “Temporally consistent disparity and optical flow via efficient spatio-temporal filtering,” in _PSIVT_ , 2011.
  * [8] R. Furukawa, R. Sagawa, and H. Kawasaki, “Depth estimation using structured light flow - analysis of projected pattern flow on an object’s surface,” in _International Conference on Computer Vision_ , 2017, pp. 4650–4658.
  * [9] L. Zhang, B. Curless, and S. M. Seitz, “Rapid shape acquisition using color structured light and multi-pass dynamic programming,” in _International Symposium on 3D Data Processing Visualization and Transmission_ , 2002, pp. 24–36.
  * [10] H. Kawasaki, S. Ono, Y. Horita, Y. Shiba, R. Furukawa, and S. Hiura, “Active one-shot scan for wide depth range using a light field projector based on coded aperture,” in _International Conference on Computer Vision_ , 2015, pp. 3568–3576.
  * [11] R. Furukawa, H. Morinaga, Y. Sanomura, S. Tanaka, S. Yoshida, and H. Kawasaki, “Shape acquisition and registration for 3D endoscope based on grid pattern projection,” in _European Conference on Computer Vision_ , 2016, pp. 399–415.
  * [12] P. Vuylsteke and A. Oosterlinck, “Range image acquisition with a single binary-encoded light pattern,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 12, no. 2, pp. 148–164, 1990.
  * [13] H. Kawasaki, R. Furukawa, R. Sagawa, and Y. Yagi, “Dynamic scene shape reconstruction using a single structured light pattern,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2008, pp. 1–8.
  * [14] M. Proesmans, L. Van Gool, and A. Oosterlinck, “One-shot active 3D shape acquisition,” in _International Conference on Pattern Recognition_ , 1996, pp. 336–340.
  * [15] Y. Zhang, Z. Xiong, Z. Yang, and F. Wu, “Real-time scalable depth sensing with hybrid structured light illumination,” _IEEE Transactions on Image Processing_ , vol. 23, no. 1, pp. 97–109, 2014.
  * [16] Y. Zhang, S. Khamis, C. Rhemann, J. Valentin, A. Kowdle, V. Tankovich, M. Schoenberg, S. Izadi, T. Funkhouser, and S. Fanello, “Activestereonet: End-to-end self-supervised learning for active stereo systems,” in _European Conference on Computer Vision_ , 2018, pp. 802–819.
  * [17] J. Žbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” _Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 2287–2318, 2016.
  * [18] Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in _European Conference on Computer Vision_ , 2020, pp. 402–419.
  * [19] M. M. Johari, C. Carta, and F. Fleuret, “Depthinspace: Exploitation and fusion of multiple video frames for structured-light depth estimation,” in _International Conference on Computer Vision_ , 2021.
  * [20] A. Tonioni, F. Tosi, M. Poggi, S. Mattoccia, and L. Stefano, “Real-time self-adaptive deep stereo,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 195–204.
  * [21] S. Li, X. Wang, Y. Cao, F. Xue, Z. Yan, and H. Zha, “Self-supervised deep visual odometry with online adaptation,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6338–6347.
  * [22] F. Zhang, X. Qi, R. Yang, V. Prisacariu, B. Wah, and P. Torr, “Domain-invariant stereo matching networks,” in _European Conference on Computer Vision_ , 2020, pp. 420–439.
  * [23] Intel. (2018) Intel realsense D435i. [Online]. Available: https://www.intelrealsense.com/depth-camera-d435i
  * [24] L. Zhang, B. Curless, and S. M. Seitz, “Spacetime stereo: shape recovery for dynamic scenes,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2003, pp. 1983–1992.
  * [25] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “Shapenet: An information-rich 3d model repository,” Stanford University — Princeton University — Toyota Technological Institute at Chicago, Tech. Rep. arXiv:1512.03012 [cs.GR], 2015.
  * [26] Q. Wang, S. Shi, S. Zheng, K. Zhao, and X. Chu, “FADNet: A fast and accurate network for disparity estimation,” in _International Conference on Robotics and Automation_ , 2020, pp. 101–107.
  * [27] Shining3D. (2017) FreeScan x5. [Online]. Available: https://www.aniwaa.com/product/3d-scanners/shining-3d-freescan-x5
  * [28] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and R. M. Carnicer, “Generation of fiducial marker dictionaries using mixed integer linear programming,” _Pattern Recognition_ , vol. 51, no. C, pp. 481–491, 2016.
  * [29] A. Handa, T. Whelan, J. McDonald, and A. J. Davison, “A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM,” in _International Conference on Robotics and Automation_ , 2014, pp. 1524–1531.