# Holographic Einstein Ring of Deformed AdS-Schwarzschild Black Holes

Jin-Yu Gui111Email<EMAIL_ADDRESS>Xiao-Xiong Zeng222Email:
<EMAIL_ADDRESS>(Corresponding author) Ke-Jian He333Email:
<EMAIL_ADDRESS>(Corresponding author) Huan Ye444Email<EMAIL_ADDRESS>
###### Abstract

The Einstein ring of a deformed AdS-Schwarzschild black hole (BH) is
investigated under the wave optics framework. When the source is fixed on the
AdS boundary, we can obtain the corresponding response function generated on
the opposite side of the boundary. Utilizing a specialized optical system
equipped with a convex lens enables us to capture an image of the BH’s
holographic Einstein ring on the screen. The influence of the relevant
physical parameters and the observed inclination on the characteristics of the
Einstein ring can also be studied, which found that the alteration of the
observer’s position, the image displayed on the screen changes from an
axisymmetric ring to an arc, ultimately becoming a solitary point of
luminosity. Furthermore, variations in the relevant physical parameters
naturally exert an influence on the Einstein ring, resulting in various
changes to both its radius and brightness. We also discover the photon ring of
the BH from the perspective of the geometric optics, and the numerical results
demonstrate that the incident angle of the photon ring is consistent with that
of the Einstein ring. It is found that the validity of studying the Einstein
ring by wave optics. The investigation on the Einstein rings formed by the
deformed AdS-Schwarzschild BHs is anticipated to yield more comprehensive
insights into the spacetime geometry, enabling its distinction from
Schwarzschild BH.

###### keywords:

Deformed AdS-Schwarzschild black hole , AdS/CFT Correspondence , Einstein ring
, Wave optics

[first]organization=Department of Mechanics, Chongqing Jiaotong University,
city=Chongqing, postcode=400074, country=China

[second]organization=School of Material Science and Engineering, Chongqing
Jiaotong University, city=Chongqing, postcode=400074, country=China

##  1 Introduction

The AdS/CFT correspondence[1], also known as holographic duality or
gauge/gravity duality, establishes a profound link between classical
gravitational theory in Anti-de Sitter (AdS) spacetime and quantum field
theories (QFTs). Due to its particularity, it has been a research hotspot for
many physicists in the past few decades and a bridge connecting different
disciplinary fields[2]. The relevant research in the field of holographic
duality is not only to test the self-consistency of the theory, but also to
provide a fresh development platform for the study of strongly coupled field
theory. The concept of holographic duality has been extensively utilized in
the investigation of holographic superconductivity[3; 4; 5], Fermi and non-
Fermi liquids[6; 7; 8], exotic metals[9; 10; 11] and Metal insulation phase
transition in quantum phase transition[12; 13; 14].

The black hole (BH) shadow, an apparent feature of BHs, has long attracted the
attention of theoretical physicists and astronomers. The Event Horizon
Telescope (EHT) project successfully captured images of the BH shadow located
at the center of both the M87 galaxy and our Milky Way[15; 16], marking a
significant milestone in astronomical observations. The study of BH shadows is
currently extensive across various spacetime contexts, including binary BH
shadows[17], different configurations of BHs[18; 19], dynamically evolving
spacetimes [20], wormholes [21], and naked singularities[22]. The images and
associated research on BHs have demonstrated the presence of a bright ring
located beyond the event horizon of the BH. This ring structure, formed due to
the extensive accumulation of high-energy matter surrounding the BH, is
commonly known as the photon ring. The photon ring not only can serve as a
probe for testing general relativity (GR), but also contains valuable
information regarding the structural composition of spacetime.

The current research on BH shadows primarily relies on geometric optics, with
a focus on the behavior of photons in close proximity to BHs. Recently,
Hashimoto $et~{}al.$ have employed wave optics to investigate the shadows of
BHs, presenting an intriguing and novel approach[23]. Especially in [24], a
significant breakthrough was made concerning the existence of a sequence of
luminous ring structures known as Einstein rings encircling the AdS
Schwarzschild BH, based on the principle of holographic duality. It is found
that the holographic images of the BH, produced by the gravitational lensing
effect of a BH, can be constructed from the response function for a thermal
state on two-dimensional sphere dual to Schwarzschild-AdS BH, and that the
Einstein rings can be clearly observed. At present this work has also been
applied to holographic superconductivity models[25], in which researchers have
discovered a discontinuous change in the size of the photon ring of the
charged BHs[26] have also been studied accordingly, and the result showed that
the chemical potential has no influence on the radius of the Einstein ring,
and the temperature dependence of the Einstein ring exhibits a unique
characteristic. Then, other studies in the background of modified gravity, can
be referred to [27; 28; 29; 30].

The AdS-Schwarzschild BH, as a classical solution in AdS spacetime, has been
extensively studied due to its valuable insights into the nature of BHs,
gravity, and the structure of the universe in the AdS framework[31; 32; 33;
34; 35]. In modified gravity theories, related studies have been conducted on
AdS-Schwarzschild BHs, including investigations of deformed solutions. In
particular, in [36], the author obtained deformed AdS-Schwarzschild BH
solutions using the gravity decoupling (GD) method, which extends known
solutions of the standard gravitational action to additional sources and
modified gravitational theories. The author further explored the impact of the
deformation parameter on the horizon structure, thermodynamics, and Hawking-
Page phase transition temperature. However, the shadow of the deformed AdS-
Schwarzschild BH remains unexplored. Therefore, this article aims to
investigate the Einstein rings of the deformed AdS-Schwarzschild BH using wave
optics, based on the AdS/CFT correspondence, and explore the impact of
relevant physical parameters on the Einstein rings.

The paper is organized as follows. In Section 2, we briefly introduce the
deformed AdS-Schwarzschild BH and derive the lensing response function that
describes the diffraction of a wave source by the BH in this spacetime
background. In Section 3, we employ an optical system equipped with a convex
lens to observe the Einstein ring formed on the screen, and pay particular
attention to the influence of the relevant physical parameters and the
observer’s position on the Einstein ring. Furthermore, we compare the relevant
conclusions obtained through wave optics with those derived from geometric
optics. Finally, in Section 4, we summarize the key findings and conclusions
of our study.

##  2 The deformed AdS-Schwarzschild BH and its response function

We consider the black hole solution of a deformed AdS-Schwarzschild BH with
the following metric[36]

$\mathrm{d}s^{2}=-G(r)\mathrm{d}t^{2}+\frac{1}{G(r)}\mathrm{d}r^{2}+r^{2}(\mathrm{d}\theta^{2}+\sin^{2}\theta\mathrm{d}\varphi^{2}),$
(1)

with

$G(r)=1-\frac{2M}{r}+\frac{r^{2}}{l^{2}}+\alpha\frac{\beta^{2}+3r^{2}+3\beta
r}{3r(\beta+r)^{3}},$ (2)

where $M$ is the ADM mass, $\alpha$ is referred to as the deformation
parameter, can adjust the strength of the effects of geometric deformations on
the background geometry, and $\beta$ is a constant parameter with length
dimension that can avoid the central singularity, which will be referred as a
control parameter, $l=\sqrt{3/\left|\Lambda\right|}$ is the AdS radius, where
$\Lambda$ is the cosmological constant. In what follows, we take $l=1$.
Substituting the metric function mentioned above with the variable $r=1/y$,
Eq.(1) can be written as

$\mathrm{d}s^{2}=\frac{1}{y^{2}}[-G(y)\mathrm{d}t^{2}+\frac{\mathrm{d}y^{2}}{G(y)}+\mathrm{d}\theta^{2}+\sin^{2}\theta\mathrm{d}\varphi^{2}],$
(3)
$G(y)=1+y^{2}-2My^{3}+\frac{y^{3}\alpha(\frac{3}{y^{2}}+\frac{3\beta}{y}+\beta^{2})}{3(\frac{1}{y}+\beta)^{3}}.$
(4)

At $y=\infty$, a space-time singularity emerges, while at $y=0$, the
corresponding system resides on the AdS boundary. Note that the temperature of
the BH is defined by the Hawking temperature, which is given by the relation
$T=\frac{1}{4\pi}{G}^{\prime}(y_{h})$, where $y=y_{h}$ represents the event
horizon of the BH.

Next, we consider the dynamics of a massless scalar field based on the
following Klein-Gordon equation[24]

$D_{a}D^{a}\tilde{\Psi}=0.$ (5)

In order to solve the Klein-Gordon equation more conveniently, we use the
ingoing Eddington coordinate[26] which is convenient for us to observe the
nature of the BH to ensure that the physical quantity is continuous at the
event horizon. The coordinates can be written as

$\upsilon\equiv t+y_{*}=t-\int\frac{\mathrm{d}y}{G(y)},$ (6)

hence, the metric function in Eq.(3) can be expressed as follows

$\mathrm{d}s^{2}=\frac{1}{y^{2}}\Big{[}-G(y)\mathrm{d}\upsilon^{2}-2\mathrm{d}y\mathrm{d}\upsilon+\mathrm{d}\theta^{2}+\sin^{2}\theta\mathrm{d}\varphi^{2}\Big{]}.$
(7)

Near the AdS boundary, the asymptotic solution of the scalar field is

$\displaystyle\tilde{\Psi}(\upsilon,y,\theta,\varphi)$ $\displaystyle=$
$\displaystyle
J_{\mathcal{Q}}(\upsilon,\theta,\varphi)+y\partial_{\upsilon}J_{\mathcal{Q}}(\upsilon,\theta,\varphi)$
(8) $\displaystyle+$
$\displaystyle\frac{1}{2}y^{2}D_{S}^{2}J_{\mathcal{Q}}(\upsilon,\theta,\varphi)+\left\langle\mathcal{Q}\right\rangle
y^{3}+Q(Y^{4}),$

where $D_{S}^{2}$ denotes the scalar Laplacian on unit $S^{2}$. Based on the
AdS/CFT framework, $J_{\mathcal{Q}}$ can be seen as the material source on the
boundary[37]. We select the mono-chromatic and axis-symmetric Gaussian wave
packet as the wave source, at the South pole $(\theta_{0}=\pi)$ of the AdS
boundary, as shown in Fig.1

Figure 1: Diagram of the response function and the source

$\displaystyle J_{\mathcal{Q}}(\upsilon,\theta)$ $\displaystyle=$
$\displaystyle
e^{-i\omega\upsilon}(2\pi\eta^{2})^{-1}\mathrm{exp}[-\frac{(\pi-\theta)^{2}}{2\eta^{2}}]$
(9) $\displaystyle=$ $\displaystyle
e^{-i\omega\upsilon}\sum_{l=0}^{\infty}C_{l0}X_{l0}(\theta),$

where $\eta$ is the width of the wave generated by the Gaussian source,
$X_{l0}(\theta)$ is the spherical harmonics function, and $\eta\ll\pi$.
$C_{l0}$ that is the coefficients of $X_{l0}(\theta)$ which can be expanded as

$C_{l0}=(-1)^{l}\sqrt{\frac{l+1/2}{2\pi}}\mathrm{exp}[-\frac{(l+1/2)^{2}\eta^{2}}{2}].$
(10)

Then, considering the symmetry of spacetime, the scalar field
$\tilde{\Psi}(\upsilon,y,\theta,\varphi)$ can be decomposed into

$\tilde{\Psi}(\upsilon,y,\theta,\varphi)=\sum_{l=0}^{\infty}\sum_{n=-l}^{l}e^{-i\omega\upsilon}C_{l0}Y_{l}(y)X_{ln}(\theta,\varphi).$
(11)

The corresponding response function $\left\langle\mathcal{Q}\right\rangle$ can
be expanded as

$\left\langle\mathcal{Q}\right\rangle=\sum_{l=0}^{\infty}e^{-i\omega\upsilon}C_{l0}(\mathcal{Q})_{l}X_{l0}(\theta).$
(12)

In Eq.(11), $Y_{l}$ satisfies the equation of motion as follow

$\displaystyle
y^{2}G(y){Y}^{\prime\prime}_{l}+(y^{2}{G}^{\prime}(y)-2yG(y)+2i\omega
y^{2}){Y}^{\prime}_{l}$ $\displaystyle+[-2i\omega y-y^{2}l(l+1)]{Y}_{l}=0.$
(13)

According to the AdS/CFT framework, near the AdS boundary $Y_{l}$ can be
expanded as

$\lim_{y\to 0}Y_{l}=1-i\omega
y+\frac{y^{2}(-l(1+l))}{2}+\left\langle\mathcal{Q}\right\rangle_{l}y^{3}+Q(y^{4}).$
(14)

Obviously, the function $Y_{l}$ has two boundary conditions. One of which is
at the event horizon $y=y_{h}$, and has the following form

$(y_{h}^{2}{G}^{\prime}+2i\omega y_{h}^{2}){Y}^{\prime}_{l}-[(2i\omega
y_{h}+y_{h}^{2}l(l+1))]{Y}_{l}=0.$ (15)

The other is at $y=0$, namely the AdS boundary. In this case, the wave source
$J_{\mathcal{Q}}$ is the asymptotic form of the scalar field at infinity, and
$Y_{l}(0)=1$ which can be seen from Eq.(13). We get the corresponding
numerical solution of $Y_{l}$, and extract
$\left\langle\mathcal{Q}\right\rangle_{l}$ by using the pseudo-spectral
method[23; 24]. Subsequently, using the Eq.(12) to get the value of the total
response function. In Fig.2 to Fig.4, we fix the deformation parameter
$\beta=1$ and change the control parameter $\alpha$ and other parameters to
observe the amplitude of the response function corresponding to different
parameters. It can be observed from Fig.2 that the amplitude of the response
function increases with the increase of the deformation parameter $\alpha$.
Conversely, Fig.3 indicates that the period of the response function decreases
with the increase of the wave source frequency $\omega$. As can be seen from
Fig.4, the amplitude of the response function exhibits a notable variation
with temperature, yet the relationship is not linear. For instance, at
$\mathit{T}=0.310$, the amplitude of the response function attains its peak,
and subsequently diminishes at $\mathit{T}=0.269$ and $\mathit{T}=0.514$.

Figure 2: Effect of different $\alpha$ on the response function, where
$\beta=1$, $y_{h}=1$, $\omega=90$. Figure 3: Effect of different $\omega$ on
the response function, where $\alpha=1.6$, $y_{h}=1$, $\beta=1$. Figure 4:
Effect of different $\mathit{T}$ on the response function, where $\alpha=1.6$,
$\beta=1$, $\omega=90$.

For simplicity and without loss of generality, we fixed the control parameter
$\alpha=1$ and change the deformation parameter $\beta$ and other parameters,
as shown in Fig.5 to Fig.7. The amplitude of the response function decreases
with the increase of the control parameter $\beta$ and the wave source
frequency $\omega$, respectively, as shown in Fig.5 and Fig.6. The amplitude
of the response function is maximum when $\mathit{T}=0.313$ and decreasing
smoothly when $\mathit{T}=0.272$ and $\mathit{T}=0.515$ in Fig.7.

Figure 5: Effect of different $\beta$ on the response function, where
$\alpha=1$, $y_{h}=1$, $\omega=90$. Figure 6: Effect of different $\omega$ on
the response function, where $\alpha=1$, $y_{h}=1$, $\beta=1$. Figure 7:
Effect of different $\mathit{T}$ on the response function, where $\alpha=1$,
$\beta=1$, $\omega=90$.

##  3 The formation of holographic ring

Although we have derived the response function, it doesn’t exhibit holographic
images, to achive this goal, it is necessary to incorporate an optical setup
equipped with a convex lens. The purpose of the convex lens is to convert a
plane wave into a spherical wave, with an observed Angle $\theta_{obs}$ at the
AdS boundary. A new coordinate $(\theta^{{}^{\prime}},\varphi^{{}^{\prime}})$
is obtained by rotating coordinate $(\theta,\varphi)$, and the following
relation is satisfied[26]

$\cos\varphi^{{}^{\prime}}+i\cos\theta^{{}^{\prime}}=e^{i\theta_{obs}}(\sin\theta\cos\varphi+i\cos\theta),$
(16)

where $\theta^{{}^{\prime}}=0,\varphi^{{}^{\prime}}=0$ correspond to the
observational center. In addition, we establish a Cartesian coordinate system
$(x,y,z)$ such that
$(x,y)=(\theta^{{}^{\prime}}\cos\varphi^{{}^{\prime}},\theta^{{}^{\prime}}\sin\varphi^{{}^{\prime}})$
at the boundary of the observational point, for a virtual optical system. The
convex lens is adjusted on a two-dimensional plane $(x,y)$, in which the focal
length of the lens and corresponding radius are denoted by $f$ and $d$,
respectively. Further, we define the coordinates on the spherical screen as
$(x,y,z)=(x_{SC},y_{SC},z_{SC})$, satisfying
$x^{2}_{SC}+y^{2}_{SC}+z^{2}_{SC}=f^{2}$. The relationship between the
incident wave $\Psi(\tilde{x})$ before passing through the convex lens and the
outgoing wave $\Psi_{T}(\tilde{x})$ after passing through the convex lens
satisfies

$\Psi_{T}(\tilde{x})=e^{-i\hat{\omega}\frac{\left|\tilde{x}\right|}{f}}\Psi(\tilde{x}).$
(17)

The wave function imaging on the screen can be represented as[26]

$\displaystyle\Psi_{SC}(\tilde{x}_{SC})$ $\displaystyle=$
$\displaystyle\int_{\left|\tilde{x}\right|\leq
d}d^{2}x\Psi_{T}(\tilde{x})e^{i\omega R}$ (18) $\displaystyle\propto$
$\displaystyle\int_{\left|\tilde{x}\right|\leq
d}d^{2}x\Psi(\tilde{x})e^{-i\frac{\omega}{2f}\tilde{x}\cdot\tilde{x}_{SC}}$
$\displaystyle=$ $\displaystyle\int
d^{2}x\Psi(\tilde{x})\sigma(\tilde{x})e^{-i\frac{\omega}{2f},\tilde{x}\cdot\tilde{x}_{SC}},$

where $R$ is the distance from the lens point $(x,y,0)$ to the screen point
$(x^{2}_{SC},y^{2}_{SC},z^{2}_{SC})$. In the formula, $\sigma(\tilde{x})$ is
the window function, which is defined as

$\sigma(\tilde{x}):=\begin{cases}1,~{}~{}~{}~{}0\leq\tilde{x}\leq d;\\\
0,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\tilde{x}\geq d.\end{cases}$ (19)

From Eq.(18), it is clear that the observed wave on the screen is associated
with the incident wave through the Fourier transform, and we can observe the
holographic images on this screen, by Eq.(18). We will study the influence of
deformation parameter $\alpha$ , control parameter $\beta$, and Gaussian wave
source on holographic images, with the source width $\eta=0.02$ and radius of
convex lens $d=0.6$.

###  3.1 The effect of deformation parameter $\alpha$ on the images

(a) $\alpha=0.1$,$\theta_{obs}=0$

(b) $\alpha=0.1$,$\theta_{obs}=\pi/6$

(c) $\alpha=0.1$,$\theta_{obs}=\pi/3$

(d) $\alpha=0.1$,$\theta_{obs}=\pi/2$

(e) $\alpha=5.1$,$\theta_{obs}=0$

(f) $\alpha=5.1$,$\theta_{obs}=\pi/6$

(g) $\alpha=5.1$,$\theta_{obs}=\pi/3$

(h) $\alpha=5.1$,$\theta_{obs}=\pi/2$

(i) $\alpha=10.1$,$\theta_{obs}=0$

(j) $\alpha=10.1$,$\theta_{obs}=\pi/6$

(k) $\alpha=10.1$,$\theta_{obs}=\pi/3$

(l) $\alpha=10.1$,$\theta_{obs}=\pi/2$

(m) $\alpha=15.1$,$\theta_{obs}=0$

(n) $\alpha=15.1$,$\theta_{obs}=\pi/6$

(o) $\alpha=15.1$,$\theta_{obs}=\pi/3$

(p) $\alpha=15.1$,$\theta_{obs}=\pi/2$

Figure 8: Observational appearance of the response on the screen for different
$\alpha$ at various observation angles, where $y_{h}=5$, $\omega=90$.

Initially, we present the holographic images, which are obtained by varying
the deformation parameter, $\alpha$, and the observer’s position,
$\theta_{obs}$, as depicted in Fig.8. When $\theta_{obs}=0$, it means that the
wave source is fixed at the south pole of the AdS boundary, while the observer
is situated at the north pole, observing a series of axisymmetric concentric
rings. When we consider $\theta_{obs}=\pi/6$ (see the second column), the
holographic ring changes from strictly spherical symmetry to axisymmetric and
some brightness decreases. As it increases to $\theta_{obs}=\pi/3$ and
$\theta_{obs}=\pi/2$, the holographic ring gradually disappears, especially in
$\theta_{obs}=\pi/2$ where only bright spots appear. That is to say, the shape
of the holographic ring heavily depends on the observer’s position. In the
leftmost column of Fig.8, despite the change in the value of $\alpha$, a
brightest circular ring persists, which is referred to as the Einstein ring.
In addition, to better investigate the impact of the deformation parameter
$\alpha$ on the Einstein ring, we have plotted Fig.9 in which the distance
between the two peak trajectories corresponds to the diameter of the Einstein
ring. One can see that as $\alpha$ increases, the radius of the Einstein ring
gradually increases. When $\alpha=0.1$, the radius of the ring is 0.49, and
when $\alpha=15.1$, the radius of the ring is 0.56. Then, from the luminosity
represented on the y-axis of Fig.9, it can be observed that as $\alpha$
increases, the luminosity gradually decreases.

(a) $\alpha=0.1$

(b) $\alpha=5.1$

(c) $\alpha=10.1$

(d) $\alpha=15.1$

Figure 9: The effect of $\alpha$ on the brightness, where $\theta_{obs}=0$,
$y_{h}=5$, $\omega=90$.

Next, we investigate the impact of the event horizon on the Einstein ring of
black hole. For simplicity, we consider only the case where $\theta_{obs}=0$,
as depicted in Fig.10. When $y_{h}=0.5$, a bright spherically symmetric ring
appears on the screen. Additionally, as $y_{h}$ increases from 0.5 to higher
values, such as $y_{h}=3$ and $y_{h}=5.5$, the ring gradually contracts,
notably, when $y_{h}=8$, the ring radius is minimum. The observation indicates
that as $y_{h}$ increases, the ring radius decreases, but there is no apparent
pattern of change in the ring radius with temperature. In literature [26], the
authors discussed the variation of ring radius with temperature. However, here
we argue that the variation of ring radius with the event horizon radius is
more reasonable, as the relationship between the black hole temperature and
the event horizon is not monotonic, as depicted in Fig.11.

(a) $y_{h}=0.5$($\mathit{T}=0.514$)

(b) $y_{h}=3$($\mathit{T}=0.305$)

(c) $y_{h}=5.5$($\mathit{T}=0.469$)

(d) $y_{h}=8$($\mathit{T}=0.657$)

Figure 10: The effect of $y_{h}$ on the Einstein ring, where $\theta_{obs}=0$,
$\alpha=1.6$, $\omega=90$. Figure 11:  The relationship between temperature
and event horizon $y_{h}$, where $\alpha=1.6$, $\beta=1$.

(a) $y_{h}=0.5$($\mathit{T}=0.514$)

(b) $y_{h}=3$($\mathit{T}=0.305$)

(c) $y_{h}=5.5$($\mathit{T}=0.469$)

(d) $y_{h}=8$($\mathit{T}=0.657$)

Figure 12: The effect of $\mathit{T}$ on the brightness, where
$\theta_{obs}=0$, $\alpha=1.6$, $\omega=90$.

Similarly, in Fig.12, we plotted the relationship between luminosity and the
event horizon. When $y_{h}=0.5$, $\mathit{xs/f}=1$, the luminosity is maximum.
As $y_{h}$ gradually increases, the horizontal axis corresponding to the
maximum luminosity gradually decreases, which means that the radius of the
ring decreases as $y_{h}$ increases.

We can also analyze the impact of wave sources, such as the wave source
frequency $\omega$, on the Einstein ring. As demonstrated in Fig.13, a
discernible trend emerges where the resolution of the Einstein ring improves
as the wave source frequency increases. At $\omega=40$, the presence of
multiple diffraction rings is observable, indicating significant interference
effects. However, as the frequency rises to $\omega=100$, these additional
diffraction rings gradually fade, leaving only the primary Einstein ring
distinct. This transition can be attributed to the increasing dominance of
geometric optics at higher frequencies. To complement our understanding of
these findings, we have depicted the correlation between luminosity and
angular frequency in Fig.14. It is evident that high frequencies are crucial
for effectively observing and resolving the radius of the ring.

(a) $\omega=10$

(b) $\omega=40$

(c) $\omega=70$

(d) $\omega=100$

Figure 13: The effect of $\omega$ on the Einstein ring, where
$\theta_{obs}=0$, $\alpha=1.6$, $y_{h}=5$.

(a) $\omega=10$

(b) $\omega=40$

(c) $\omega=70$

(d) $\omega=100$

Figure 14: The effect of $\omega$ on the brightness, where $\theta_{obs}=0$,
$\alpha=1.6$, $y_{h}=5$.

###  3.2 The effect of control parameter $\beta$ on the images

(a) $\beta=0.3$,$\theta_{obs}=0$

(b) $\beta=0.3$,$\theta_{obs}=\pi/6$

(c) $\beta=0.3$,$\theta_{obs}=\pi/3$

(d) $\beta=0.3$,$\theta_{obs}=\pi/2$

(e) $\beta=0.45$,$\theta_{obs}=0$

(f) $\beta=0.45$,$\theta_{obs}=\pi/6$

(g) $\beta=0.45$,$\theta_{obs}=\pi/3$

(h) $\beta=0.45$,$\theta_{obs}=\pi/2$

(i) $\beta=0.6$,$\theta_{obs}=0$

(j) $\beta=0.6$,$\theta_{obs}=\pi/6$

(k) $\beta=0.6$,$\theta_{obs}=\pi/3$

(l) $\beta=0.6$,$\theta_{obs}=\pi/2$

(m) $\beta=0.75$,$\theta_{obs}=0$

(n) $\beta=0.75$,$\theta_{obs}=\pi/6$

(o) $\beta=0.75$,$\theta_{obs}=\pi/3$

(p) $\beta=0.75$,$\theta_{obs}=\pi/2$

Figure 15: Observational appearance of the response on the screen for
different $\beta$ at various observation angles, where $y_{h}=5$, $\omega=90$.

Similarly, with the change of control parameter $\beta$, a series of
concentric rings can be observed in the leftmost column of Fig.15. As the
observer’s position $\theta_{obs}$ varies from $\theta_{obs}=0$ to
$\theta_{obs}=\pi/2$, the Einstein ring transitions from strict spherical
symmetry to axial symmetry, accompanied by a decrease in partial brightness.
Subsequently, the ring gradually disappears, leaving only a single light spot.
Fig.16 indicates that the radius of the Einstein ring gradually decreases with
the increase of $\beta$. Specifically, when $\beta=0.3$, the radius of the
ring is 0.6, and when $\beta=0.75$, the radius of the ring is 0.5.
Furthermore, as $\beta$ augments, the luminosity gradually intensifies.

In Fig.17, we analyze the impact of the event horizon on the Einstein ring of
a BH. The results indicate a reciprocal relationship between the increasing
$y_{h}$ and the corresponding decrease in the ring’s radius. Fig.18 depicts
the relationship between luminosity and the radius of the event horizon. As
$y_{h}$ gradually increases, the horizontal coordinate corresponding to the
maximum luminosity gradually decreases, suggesting that the ring’s radius
diminishes with the augmentation of $y_{h}$.

(a) $\beta=0.3$

(b) $\beta=0.45$

(c) $\beta=0.6$

(d) $\beta=0.75$

Figure 16: The effect of $\beta$ on the brightness, where $\theta_{obs}=0$,
$y_{h}=5$,$\omega=90$

(a) $y_{h}=0.5$,$\mathit{T}=0.515$

(b) $y_{h}=4.5$,$\mathit{T}=0.403$

(c) $y_{h}=8.5$,$\mathit{T}=0.698$

(d) $y_{h}=12.5$,$\mathit{T}=1.009$

Figure 17: The effect of $\mathit{T}$ on the brightness, where
$\theta_{obs}=0$, $\beta=1$, $\omega=90$.

(a) $y_{h}=0.5$,$\mathit{T}=0.515$

(b) $y_{h}=4.5$,$\mathit{T}=0.403$

(c) $y_{h}=8.5$,$\mathit{T}=0.698$

(d) $y_{h}=12.5$,$\mathit{T}=1.009$

Figure 18: The effect of $\mathit{T}$ on the brightness, where
$\theta_{obs}=0$, $\beta=1$, $\omega=90$.

###  3.3 Images from the viewpoint of geometric optics

In this section, we will discuss Einstein rings from the perspective of
geometric optics. We mainly study the incidence angle of photons, based on the
spherically symmetric spacetime background described in Eq.(1) and Eq.(2). We
define conserved quantities in the context of a metric as $\omega=G(r)\partial
t/\partial\lambda$ and $\tilde{L}=r^{2}\partial\varphi/\partial t$, in which
$\omega$ is the energy of the photon, $\tilde{L}$ is the angular momentum of
the photon and $\lambda$ is the affine parameter. Owing to the spherical
symmetry of spacetime, we only consider the coordinate system in which the
photon orbits are located at the equator $\theta\equiv\pi/2$. The four-speed
vector $\upsilon^{\zeta}=(\mathrm{d}/\mathrm{d}\lambda)^{\zeta}$ is satisfied

$\displaystyle-G(r)(\mathrm{d}t/\mathrm{d}\lambda)^{2}+G(r)^{-1}(\mathrm{d}r/\mathrm{d}\lambda)^{2}$
$\displaystyle+r^{2}\sin\theta(\mathrm{d}\varphi/\mathrm{d}\lambda)^{2}=0,$
(20)

and

$\dot{r}^{2}=\omega-\tilde{L}z(r).$ (21)

here $z(r)=G(r)/r^{2}$ and $\dot{r}=\partial r/\partial\lambda$. The incident
angle $n^{\zeta}\equiv\partial/\partial r^{\zeta}$ with boundary $\theta_{in}$
as normal vector is defined as[26; 28]

$\cos\theta_{in}=\frac{g_{\alpha\beta}\upsilon^{\alpha}n^{\beta}}{\left|\upsilon\right|\left|n\right|}\mid_{r=\infty}=\sqrt{\frac{\dot{r}^{2}/G}{\dot{r}^{2}/G+\tilde{L}/r^{2}}}\mid_{r=\infty},$
(22)

further simplification leads to

$\sin\theta_{in}^{2}=1-\cos\theta_{in}^{2}=\frac{\tilde{L}^{2}z(r)}{{\dot{r}^{2}+\tilde{L}^{2}z(r)}}\mid_{r=\infty}=\frac{\tilde{L}^{2}}{\omega^{2}}.$
(23)

When the two endpoints of a geodesic are aligned with the center of the BH,
due to axial symmetry[24], an observer will see a ring image with a radius
corresponding to the incident angle $\theta_{in}$. As a photon reaches the
location of the photon sphere, it neither escapes nor falls into the BH, but
instead begins to orbit around it. At this point, the angular momentum is
denoted by $L$, and the equation of orbital motion for the photon at the
photon ring is determined by the following conditions[23; 24; 26]

$\dot{r}=0,\mathrm{d}z/\mathrm{d}r=0.$ (24)

In the case, we have $\sin\theta_{in}=\mathit{L}/{\omega}$, furthermore, from
Fig.19 it can be observed that $\sin\theta_{R}$ satisfies the relationship

$\sin\theta_{R}=r_{R}/f.$ (25)

The incident angle of the photon and the angle of the photon ring both
describe the angle of the photon ring that can be observed by an observer, and
they should be essentially equal, that is

$r_{R}/f=L/\omega.$ (26)

Figure 19: the relation between ring radius $r_{R}$ and ring angle
$\theta_{R}$.

When relevant physical parameters change, we use numerical methods to validate
the results obtained from Eq.(26). Fig.20 depicts the radius of the Einstein
ring for different parameter $\alpha$, and Fig.21 depicts the radius of the
Einstein ring with different parameter $\beta$. One can see that the observed
angle of the Einstein ring obtained by wave optics is very close to the
incident angle of the photon ring obtained by geometric optics. Despite their
impact on the fitting accuracy, this conclusion is independent of the
deformation parameter $\alpha$ and the constant parameter $\beta$. That is,
the Einstein ring can be constructed holographically via wave optics.

(a) $\alpha=0.1$

(b) $\alpha=1.1$

(c) $\alpha=3$

Figure 20: Comparing the Einstein ring radius between the geometric optics and
wave optics for different values of $\alpha$ with $\omega=90$. Where the
discrete red points represent the Einstein ring radius derived from wave
optics, while the blue curve depicts the variation in the radius of the
circular orbit as the temperature changes, which is based on geometric optics.

(a) $\beta=0.5$

(b) $\beta=1$

(c) $\beta=2$

Figure 21: Comparing the Einstein ring radius between the geometric optics and
wave optics for different values of $\beta$ with $\omega=90$. Where the
discrete magenta points represent the Einstein ring radius derived from wave
optics, while the blue curve depicts the variation in the radius of the
circular orbit as the temperature changes, which is based on geometric optics.

##  4 Summary and conclusions

With the wave optics, we investigated the Einstein rings of the deformed AdS-
Schwarzschild BH, based on AdS/CFT correspondence. By solving for the
asymptotic behavior of the wave function near the boundary, we obtained the
response of the Gaussian wave source at the South Pole at the North Pole. The
results indicate that the response function is the diffraction pattern formed
by the scalar waves emitted from the wave source after being scattered by the
BH. When $\beta=1$, the amplitude of the response function increases with the
increase of the deformation parameter $\alpha$, but the response function
decreases with the increase of the wave source frequency $\omega$, the
amplitude of the response function varies significantly with temperature.
Similarly, when $\alpha=1$, the amplitude of the response function decreases
with the increase of both the control parameter $\beta$ and the wave source
frequency $\omega$. Moreover, the amplitude of the response function also
varies with temperature.

In order to observe the corresponding holographic images, we introduced an
optical system with a convex lens. The result showed that when the control
parameter $\beta$ is fixed, the radius of the holographic ring gradually
increases with the increase of the deformation parameter $\alpha$. When the
event horizon increases, the ring radius decreases. Moreover, as the wave
source frequency increases, the image becomes clearer. On the contrary, when
we fix the deformation parameter $\alpha$, the radius of the Einstein ring
gradually decreases with the increase of control parameter $\beta$. When the
horizon radius increases, the ring radius decreases. Similarly, as the wave
source frequency increases, the ring image becomes clearer.

Under the framework of geometric optics, we further studied the incident angle
of photons ring and found through theoretical analysis and numerical
simulation that this result is consistent with the observation angle of the
Einstein ring obtained through wave optics. This conclusion does not depend on
deformation parameter $\alpha$ and control parameter $\beta$, although the
$\alpha$ and $\beta$ affect the fitting accuracy.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China
(Grants No. 11675140, No. 11705005, and No. 12375043), and Innovation and
Development Joint Foundation of Chongqing Natural Science Foundation (Grant
No. CSTB2022NSCQ-LZX0021) and Basic Research Project of Science and Technology
Committee of Chongqing (Grant No. CSTB2023NSCQMSX0324).

## References

  * [1] J. M. Maldacena, Adv. Theor. Math. Phys. 2 (1998), 231-252 [arXiv:hep-th/9711200 [hep-th]].
  * [2] M. Natsuume, Lect. Notes Phys. 903 (2015), pp.1-294 [arXiv:1409.3575 [hep-th]].
  * [3] S. A. Hartnoll, C. P. Herzog and G. T. Horowitz, JHEP 12 (2008), 015 [arXiv:0810.1563 [hep-th]].
  * [4] S. A. Hartnoll, C. P. Herzog and G. T. Horowitz, Phys. Rev. Lett. 101 (2008), 031601 [arXiv:0803.3295 [hep-th]].
  * [5] S. S. Gubser, Phys. Rev. D 78 (2008), 065034 [arXiv:0801.2977 [hep-th]].
  * [6] M. Cubrovic, J. Zaanen and K. Schalm, Science 325 (2009), 439-444 [arXiv:0904.1993 [hep-th]].
  * [7] H. Liu, J. McGreevy and D. Vegh, Phys. Rev. D 83 (2011), 065029 [arXiv:0903.2477 [hep-th]].
  * [8] T. Faulkner, H. Liu, J. McGreevy and D. Vegh, Phys. Rev. D 83 (2011), 125002 [arXiv:0907.2694 [hep-th]].
  * [9] S. A. Hartnoll, J. Polchinski, E. Silverstein and D. Tong, JHEP 04 (2010), 120 [arXiv:0912.1061 [hep-th]].
  * [10] S. S. Pal, Phys. Rev. D 84 (2011), 126009 [arXiv:1011.3117 [hep-th]].
  * [11] B. S. Kim, E. Kiritsis and C. Panagopoulos, New J. Phys. 14 (2012), 043045 [arXiv:1012.3464 [cond-mat.str-el]].
  * [12] E. Mefford and G. T. Horowitz, Phys. Rev. D 90 (2014) no.8, 084042 [arXiv:1406.4188 [hep-th]].
  * [13] E. Kiritsis and J. Ren, JHEP 09 (2015), 168 [arXiv:1503.03481 [hep-th]].
  * [14] A. Donos and J. P. Gauntlett, JHEP 06 (2014), 007 [arXiv:1401.5077 [hep-th]].
  * [15] K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 875 (2019), L1 [arXiv:1906.11238 [astro-ph.GA]].
  * [16] K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 930 (2022) no.2, L12 [arXiv:2311.08680 [astro-ph.HE]].
  * [17] J. Shipley and S. R. Dolan, Class. Quant. Grav. 33 (2016) no.17, 175001 [arXiv:1603.04469 [gr-qc]].
  * [18] C. Bambi and K. Freese, Phys. Rev. D 79 (2009), 043002 [arXiv:0812.1328 [astro-ph]].
  * [19] F. Atamurotov, A. Abdujabbarov and B. Ahmedov, Astrophys. Space Sci. 348 (2013), 179-188
  * [20] A. K. Mishra, S. Chakraborty and S. Sarkar, Phys. Rev. D 99 (2019) no.10, 104080 [arXiv:1903.06376 [gr-qc]].
  * [21] P. G. Nedkova, V. K. Tinchev and S. S. Yazadjiev, Phys. Rev. D 88 (2013) no.12, 124019 [arXiv:1307.7647 [gr-qc]].
  * [22] R. Shaikh, P. Kocherlakota, R. Narayan and P. S. Joshi, Mon. Not. Roy. Astron. Soc. 482 (2019) no.1, 52-64 [arXiv:1802.08060 [astro-ph.HE]].
  * [23] K. Hashimoto, S. Kinoshita and K. Murata, Phys. Rev. Lett. 123 (2019) no.3, 031602 [arXiv:1906.09113 [hep-th]].
  * [24] K. Hashimoto, S. Kinoshita and K. Murata, Phys. Rev. D 101 (2020) no.6, 066018 [arXiv:1811.12617 [hep-th]].
  * [25] Y. Kaku, K. Murata and J. Tsujimura, JHEP 09 (2021), 138 [arXiv:2106.00304 [hep-th]].
  * [26] Y. Liu, Q. Chen, X. X. Zeng, H. Zhang, W. L. Zhang and W. Zhang, JHEP 10 (2022), 189 [arXiv:2201.03161 [hep-th]].
  * [27] X. X. Zeng, K. J. He, J. Pu, G. p. Li and Q. Q. Jiang, Eur. Phys. J. C 83 (2023) no.10, 897 [arXiv:2302.03692 [gr-qc]].
  * [28] X. X. Zeng, L. F. Li and P. Xu, [arXiv:2307.01973 [hep-th]].
  * [29] X. X. Zeng, M. I. Aslam, R. Saleem and X. Y. Hu, [arXiv:2311.04680 [gr-qc]].
  * [30] X. Y. Hu, X. X. Zeng, L. F. Li and P. Xu, [arXiv:2312.13552 [gr-qc]].
  * [31] A. S. Khan and F. Ali, Phys. Dark Univ. 26 (2019), 100389
  * [32] L. G. C. Gentile, P. A. Grassi and A. Mezzalira, JHEP 10 (2013), 065 [arXiv:1207.0686 [hep-th]].
  * [33] J. Dunn and C. Warnick, Class. Quant. Grav. 33 (2016) no.12, 125010 [arXiv:1507.07367 [gr-qc]].
  * [34] T. K. Dey, Int. J. Mod. Phys. A 33 (2018) no.33, 1850193 [arXiv:1711.07008 [hep-th]].
  * [35] N. Banerjee, A. Bhattacharjee and A. Mitra, JHEP 01 (2021), 038 [arXiv:2008.02828 [hep-th]].
  * [36] M. R. Khosravipoor and M. Farhoudi, Eur. Phys. J. C 83 (2023) no.11, 1045 [arXiv:2311.02456 [gr-qc]].
  * [37] I. R. Klebanov and E. Witten, Nucl. Phys. B 556 (1999), 89-114 [arXiv:hep-th/9905104 [hep-th]].


# Cross-Dimensional Refined Learning for Real-Time 3D Visual Perception from
Monocular Video

Ziyang Hong C. Patrick Yue
Hong Kong University of Science and Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a novel real-time capable learning method that jointly perceives a
3D scene’s geometry structure and semantic labels. Recent approaches to real-
time 3D scene reconstruction mostly adopt a volumetric scheme, where a
Truncated Signed Distance Function (TSDF) is directly regressed. However,
these volumetric approaches tend to focus on the global coherence of their
reconstructions, which leads to a lack of local geometric detail. To overcome
this issue, we propose to leverage the latent geometric prior knowledge in 2D
image features by explicit depth prediction and anchored feature generation,
to refine the occupancy learning in TSDF volume. Besides, we find that this
cross-dimensional feature refinement methodology can also be adopted for the
semantic segmentation task by utilizing semantic priors. Hence, we proposed an
end-to-end cross-dimensional refinement neural network (CDRNet) to extract
both 3D mesh and 3D semantic labeling in real time. The experiment results
show that this method achieves a state-of-the-art 3D perception efficiency on
multiple datasets, which indicates the great potential of our method for
industrial applications.

Figure 1:  Comparison between the proposed approach and baselines. Our model
is more accurate and coherent in real time, compared to two baseline methods
with input from monocular video, Atlas [27] and NeuralRecon [39] \+ Semantic-
Heads. Real-time 3D perception efficiency $\eta_{3D}$ the higher the better.
Color denotes different semantic segmentation labeling.

##  1 Introduction

Recovering 3D geometry and semantics of objects or environment scenes prevails
these days with the advent of ubiquitous digitization. The digitization of the
world where people live can not only help them better understand their
environment scenes, but also enable robots to comprehend what they need to
know about the world and therewith conducting assigned tasks. Generally, with
surrounding environment measurements as input, 3D reconstruction and 3D
semantic segmentation are two key 3D perception techniques [9, 40, 13] in the
computer vision society, which enable a wide range of applications, including
digital twins [19, 3], virtual/augmented reality (VR/AR) [29, 39], building
information modeling [25, 42], and autonomous driving [4, 21].

Tremendous research efforts have been made for 3D perception techniques. Based
on the sensor types, researches on 3D perception can be divided into two main
streams, namely active range sensors that capture surface geometry information
and RGB cameras that capture texture with perspective projection. Originated
from KinectFusion [29], the commodity RGB-D range sensor is used to measure
depth data first and then fuse it into Truncated Signed Distance Function
(TSDF) volume for 3D reconstruction. Although the follow-up depth-based TSDF
fusion methods [45, 46, 1, 47, 37] achieve detailed dense reconstruction
result, they suffer from global incoherence due to the lack of sequential
correlation, the tendency of noise disturbance due to redundant overlapped
calculations, and the incapability of semantic deduction due to the lack of
texture features.

On the other hand, as camera-equipped smartphones become readily available
with built-in inertial measurement units, recent advances have emerged to
explore 3D perception with RGB cameras on mobile devices. The problem of
reconstructing 3D geometry with posed RGB images input only is referred to as
multi-view stereo (MVS). Existing methods for MVS that are based on deep
learning, tend to adopt a volumetric scheme by directly regressing the TSDF
volume [27, 38, 7, 39] either as a whole or in fragments. However, these
volumetric learning methods extract 3D geometric feature representation simply
from the back projection of 2D image features, resulting in the mismatch to
the 2D information priors for the predicted 3D reconstruction. Moreover, the
intrinsic end-to-end learning manner and the lack of local details on the
reconstructed mesh of these volumetric schemes result in an inferior semantic
deduction based on its 3D reconstruction prediction.

What’s worse, these learning-based methods tend to store their entire
computational graphs in memory for aggregation and require prohibitive 3D
convolution operations [27, 33, 38], which keeps them from being deployed on
robots due to the real-time and low-latency requirements in SLAM. These
limitations motivate our key idea to utilize 2D explicit predictions to
further impose a light-weight feature refinement on the 3D features input in a
sparse manner, while keeping the global coherence within the fragments. Unlike
these preceding learning-based volumetric works, we conjecture that the
utilization of 2D prior knowledge coming out of explicit predictions as a
latent feature refinement plays a significant role in learning the feature
representation in 3D perception. In addition, the feature refinement brought
by 2D explicit prediction can be operated within the fragment input for
keeping the computation redundancy and thus overhead low, while having the
global coherence by correlating different fragments to extract the target 3D
semantic mesh.

In this paper, we propose a novel framework, CDRNet, to accomplish both 3D
meshing and 3D semantic labeling tasks in real-time. Our key contributions are
as follows.

  * •

We propose a novel, end-to-end trainable network architecture, which cross-
dimensionally refines the 3D features with the prior knowledge extracted from
the explicit estimations of depths and 2D semantics.

  * •

The proposed cross-dimensional refinements yield more accurate and robust 3D
reconstruction and semantic segmentation results. We highlight that the
explicit estimations of both depths and 2D semantics serve as efficient yet
effective prior knowledge for 3D perception learning.

  * •

To achieve real-time 3D perception capability, our approach performs both
geometric and semantic localized updates to the global map. We present a
progressive 3D perception system that is capable of real-time interaction with
input data streaming from cellphones with a monocular camera.

##  2 Related Work

Figure 2:  Overview of CDRNet. Posed RGB images from monocular videos are
wrapped as fragment input for 2D feature extraction, which is used for both
depth and 2D semantic predictions for cross-dimensional refinement purposes.
To learn the foundational 3D geometry before conducting refinements, the
extracted 2D features are back-projected into raw 3D features
$\mathcal{V}_{s}$ in different resolutions without any 2D priors involved. At
each resolution, after being processed by the GRU, the output feature $L_{s}$
in the local volume is further fed into Depth and Semantics refinement modules
sequentially to have a 2D-prior-refined feature with better representations.

Real-Time 3D Perception.  The prosperity of deep learning hardwares enables
both inference and training at the edge [20, 14], thus it consolidates the
foundation to deploy more and more learning-based 3D perception techniques in
real time. KinectFusion [29] first brought in the concept of handling 3D
reconstruction tasks in real time with commodity RGB-D sensors. Han et al.
[13] presented a real-time 3D meshing and semantic labeling system similar to
our work, however, depth measurements from RGB-D sensors are required as input
in their work. Pham et al. [31] built up 3D meshes with voxel hashing, and
then fuse the initial semantic labeling with super-voxel clustering and a
high-order conditional random field (CRF) to improve labeling coherence.
Menini et al. [26] extended RoutedFusion [45] by merging semantic estimation
in its TSDF extraction scheme for each incoming depth-semantics pair.
NeuralRecon [39] adopted sparse 3D convolutions and the gated recurrent unit
(GRU) to achieve a real-time 3D reconstruction on cellphones, without the
capability of semantic deduction. For depth estimation and semantic
segmentation, there are also works achieving real-time processing capability
[44, 28, 31].

Voxelized 3D Semantic Segmentation.  The learning of semantic segmentation on
the voxelized map started from [5], which extends TSDF fusion pipeline [29]
with per-pixel labels. 3DMV [11] and MVPNet [18] further combined both depth
and RGB modalities to train an end-to-end network with 3D semantics for voxels
and point clouds, respectively. PanopticFusion [28] performed map
regularization based on adopting a CRF on the predicted panoptic labels. Atlas
[27] utilized its extracted 3D features and passed them to a set of semantic
heads for voxel labeling, the pyramid features are proven to have strong
semantics at all scales than the gradient pyramid in nature, as proven in
[22]. BPNet [15] proposed to have a joint-2D-3D reasoning in an end-to-end
learning manner. Two derivative works [26, 17] of RoutedFusion incorporated
semantic priors into their depth fusion scheme and removed their routing
module for less overhead. However, none of these works utilize the prior
knowledge within the estimated 2D semantics as a 3D feature refinement.

Volumetric 3D Surface Reconstruction.  Volumetric TSDF fusion became prevalent
for 3D surface reconstruction starting from the seminal work KinectFusion [29]
due to its high accuracy and low latency. A follow-up work, PSDF-Fusion [30]
augmented TSDF with a random variable to improve its robustness to sensor
noise. Starting from DeepSDF [30], the learned representations of TSDF using
depth input dominates the current fad. These learning-based substitutes [45,
46, 3, 1, 47, 37, 49] to TSDF fusion achieve impressive 3D reconstruction
quality compared to the baseline method with the availability of RGB-D range
sensors.

Given the fact that range sensors have relatively higher cost and energy
consumption than RGB cameras, MonoFusion [32] is one of the first works to
learn TSDF volume from RGB images by fusing the estimated depth into an
implicit model. Atlas [27] started the trend of learning-based methods by a
direct regression on TSDF volume. NeuralRecon [39] achieved a real-time 3D
reconstruction learning capability by utilizing sparse 3D convolutions and
recurrent networks with key frames as input. TransformerFusion [2] and VoRTX
[38] introduced transformers [43] to improve the performance by more relevant
inter-frame correlation. These learning-based methods prevail thanks to the
availability of these general 2D feature extractors, such as FPN [22] and
U-Net [35]. 2D information in RGB images can be effectively extracted and
further utilized for constructing their 3D perception counterparts.

However, the learning of the explicit representations of 2D latent geometric
features, such as depths and semantics, is typically ignored by all the prior
arts. They only treat the 2D feature as an intermediate in the network and
then conduct ray back-projection upon it, without considering the explicit
representations for their 3D embodiment, which we found are significant prior
knowledge for 3D perception. To extract depth as the explicit 2D
representation, VolumeFusion [7] and SimpleRecon [36] performed local MVS and
further fused it into TSDF volume with its customized network, while 3DVNet
[33] performed sparse 3D convolutions on the feature-back-projected point
cloud. Different from above, our method extracts the 2D representations from
light-weight network modules, including a portion of MVSNet [48] for depth and
a simple 2D MLP head for 2D semantics, to conduct the 3D feature refinements.
The refinement incorporates the geometric and semantic prior information to
improve the generalizability of our network by correlating the 2D
representations in their 3D counterparts.

To the best of our knowledge, we present the very first learning-based method
which uses posed RGB images input only to conduct 3D perception tasks in real
time, including 3D meshing and semantic labeling.

##  3 Methods

Given a posed image sequence $\mathbf{I}$, our goal is to extract a 3D mesh
model that can represent both 3D geometry and 3D semantic labeling, i.e., 3D
meshing with vertices $\mathcal{K}\in\mathbb{R}^{3}$, surfaces
$\mathcal{G}\in\mathbb{N}^{3}$, and its corresponding 3D semantic labeling
$\mathcal{S}\in\mathbb{N}$. We achieve this goal by jointly predicting TSDF
value $T\in[-1,1]$ and semantic label $S\in\mathbb{N}$ for each voxel, and
then extracting the mesh with the marching cubes [23]. Meanwhile, our proposed
method aims at establishing a real-time capable deep learning model for these
two 3D perception tasks. To quantitatively evaluate the efficiency of
conducting these two tasks simultaneously, we define a 3D perception
efficiency metric $\eta_{3D}$ by involving frames per second (FPS) in runtime,
as shown in Sec. 4.1.

The proposed network architecture is illustrated in Fig. 2. In Sec. 3.1, we
introduce the joint fragment learning on depth, 2D semantic category,
intermediate TSDF, and occupancy using key frames input, for the following
cross-dimensional refinements of TSDF and 3D semantics. For each fragment, the
geometric features are progressively extracted in a coarse-to-fine hierarchy
using binomial inputs GRU to build the learned representations of 3D. Sec. 3.2
describes the cross-dimensional refinements for 3D features that refines 3D
features with anchored features and semantic pixel-to-vertex correspondences
enabled by the depth and 2D semantic predictions, which helps the learning of
not only the TSDF value, but also the 3D semantic labeling in a sparsified
manner. We also present the implementation details including loss design in
Sec. 3.3. Specifications of the network are elaborated in the supplement.

###  3.1 Sparse Joint Fragment Learning in a Coarse-to-Fine Manner

Given the inherent nature of great sparsity in the ordinary real-world 3D
scene, we utilize sparse 3D convolutions to efficiently extract the 3D feature
from each input scene. However, the memory overhead of processing a 3D scene
is still prohibitive, thus we fragment the whole 3D scene and progressively
handle each of them, to further release the memory burden of holding up the
huge 3D volume data. Inspired by [27, 12, 39, 38, 33], we adopt a coarse-to-
fine learning paradigm for the sparse 3D convolutions to effectively exploit
the representation of 3D features in multiple scales. In each stage of the
hierarchy, the raw features in the fragment bounding volume (FBV) is extracted
from a GRU by correlating local features and global feature volume.

FBV Construction by Image Features.  Following [44, 39], we select a set of
key frames as the input sequences out of a monocular RGB video by querying on
each frame’s pose, namely the relative translation and optical center rotation
with empirical thresholds, $\theta_{key}$ and $t_{key}$. Key frames
$\mathbf{I}$, camera intrinsics $\mathbf{K}$, and transform matrices
$\mathbf{T}\in SE(3)$ which is an inversion of the camera pose, are all
wrapped into a fragment
$\mathbf{F}_{i}=\\{\mathbf{I}_{i,j},\mathbf{K}_{i,j},\mathbf{T}_{i,j}\\}_{j=1}^{N_{k}}$
as the input to the network, where $i$, $j$, and $N_{k}$ denote the fragment
index, the key frame index, and the number of key frames in each fragment,
respectively.

Once the fragment $\mathbf{F}_{i}$ is constructed, it is processed by a 2D
feature extractor pyramid to extract image features. In the decoder part of
the extractor pyramid, three different resolutions of feature maps are
extracted sequentially as $\mathcal{P}_{s}\in\\{P_{2},P_{3},P_{4}\\}$, where
the suffix notation of $P$ denotes the scaling ratio level in $\log_{2}$
similar to [22]. The extracted feature $\mathcal{P}_{s}$ is then back-
projected into a local 3D volume, according to the projection matrix of each
frame in $\mathbf{F}_{i}$. We hereby define FBV as the current local volume
$\mathcal{F}_{s,i}=\\{T_{s,i}^{x\times y\times z},S_{s,i}^{x\times y\times
z}\\}$ that is conditioned on the pyramid layers $\mathcal{P}_{s}$, where all
the 3D voxels that are casted in the view frustums of current $\mathbf{F}_{i}$
are included.

Initial Depth and 2D Semantics Learning.  With the fine feature $P_{2}$ as
input, we build up differentiable homography fronto-parallel planes for the
coarse-level depth prediction $\hat{D}_{4}$. Likewise, 2D semantics prediction
$\hat{S}_{4}^{2D}$ is extracted with a pointwise convolutional decoder as the
2D semantic head using $P_{2}$. The resolution gap between the input and
output feature map provides generalizability. The initial depth estimation is
retrieved from the features using a light-weight multi-view stereo network via
plane sweep [48]. For each source feature map $x$ in $P_{2}$, we conduct the
planar transformation $\mathbf{x}_{j}\sim\mathbf{H}_{j}(d)\cdot x$, where
“$\sim$” denotes the projective equality and $\mathbf{H}_{j}(d)$ is the
homography of the $j^{\text{th}}$ key frame at depth $d$. The $j^{\text{th}}$
homography111For brevity’s sake, the transformation from homogeneous
coordinates to Euclidean coordinates in the camera projection is omitted here.
in a given fragment input $\mathbf{F}_{i}$ is defined as:

$\mathbf{H}_{j}(d)=d\cdot\mathbf{K}_{j}\cdot(\mathbf{T}_{j}\cdot\mathbf{T}_{1}^{-1})\cdot\mathbf{K}_{1}^{T}\enspace.$
(1)

To measure the similarity after conducting homography warping, we calculate
the variance cost of $\mathbf{x}_{j}$ and further process it with an encoder-
decoder-based cost regularization network. The output logit from the
regularization network is treated as the depth probability on each plane and
the soft argmin [48] is conducted to have initial depth predictions.

Geometric and Semantic GRU Fusion.  Meanwhile, as the 2D features are
extracted in different resolutions, they are back-projected from each of the
pyramid level in $\mathcal{P}_{s}$ into raw geometric 3D features
$\mathcal{V}_{s}\in\\{V_{2},V_{3},V_{4}\\}$, which are further sparsified by
sparse 3D convolutions. To improve the global coherence and temporal
consistency of the reconstructed 3D mesh, following [39], we first correlate
the sparse geometric feature $\mathcal{V}_{s}$ in the current
$\mathcal{F}_{s,i}$ using GRU, with the local FBV hidden states $H_{s,i-1}$
whose information coming from all of the previous fragments
$\mathcal{F}_{s,i^{\prime}},i^{\prime}<i$ and coordinates are masked to be the
same as $\mathcal{V}_{s}$. Such correlation outputs a temporal-coherent local
feature $L_{s,i}$ for each stage $s$, which is used to generate dense
occupancy intermediate $o_{s,i}$, and passed to the 2D-to-3D cross-dimensional
refinements. The global feature volume for the entire scene $G_{s,i}$ is fused
by $G_{s,i-1}$ and $L_{s,i}$ given the coordinates of $\mathcal{V}_{s}$ as
masks, and update $H_{s,i}$. Unlike [39], we reuse the same parameters in GRU
to process the back-projected and upsampled 3D semantic features to generalize
better for the semantic prediction $\hat{S}$ in the current FBV. This is
because inputting TSDF and semantic features sequentially into GRU enables its
selective fusion across modalities, thus the feature extracted from the hidden
state incorporates more semantic information, as pointed out in [34].

For the sake of learning 3D features consistently between scales, we update
$\mathcal{V}_{s}$ at each stage by fusing with the upsampled $L_{s+1,i}$.
Inspired by the meta data mechanism proposed in [36], we further concatenate
sparse features, with sparse TSDF, occupancy and semantics after masking with
$o_{s,i}$, as the meta feature $L_{s+1,i}$ to be upsampled. We found the
inclusion of semantic information in the hidden state of GRU helps build up a
good starting point for the upcoming feature refinements, which is verified in
the ablation.

###  3.2 2D-to-3D Cross-Dimensional Refinements

The raw coherent features from GRUs lack detailed geometric descriptions,
leading to unsatisfactory meshing and semantic labeling results. To overcome
these issues, we propose to leverage the 2D feature that is latent after
incorporating the learning of depth and semantic frame for the refinement
purposes. We notice that with the learning of depth and 2D semantics, the 2D
features now reside in the latent space which can generalize to more accurate
3D geometry and semantics via cross-dimensional refinements.

2D-to-3D Prior Knowledge.  Consider a probabilistic prior in the latent space
of the output coherent feature coming from GRU, which accounts for the prior
knowledge that the pixel information in both depth predictions and 2D semantic
predictions should produce high confidence matching with regard to their own
3D representations. The prior conditioned 3D feature for both perception tasks
is defined as:

$X_{prior}=f(L_{s,i})=f\bigl{(}H_{s,i}(\mathcal{V}_{s},H_{s,i-1}\mid\mathcal{F}_{s,i})\bigr{)}\enspace,$
(2)

where $f(\cdot)$ is the 2D-to-3D feature refinement process for either 3D
meshing or 3D semantic labeling, whose input is $L_{s,i}$ extracted from
$\mathcal{V}_{s}$ and $H_{s,i-1}$ given $\mathcal{F}_{s,i}$. We borrow the
notation of $H_{s,i}$ to be a constructor function $H_{s,i}(\cdot)$ indicating
GRU. For each voxel in $\mathcal{F}_{s,i}$, both TSDF and semantic labeling
predictions can be formulated as:

$\hat{I}_{s,i}=\epsilon
h\bigl{(}H_{s,i}(\mathcal{V}_{s},H_{s,i-1}\mid\mathcal{F}_{s,i})\bigr{)}+(1-\epsilon)X_{prior}\enspace,$
(3)

where $\hat{I}_{s,i}\in\mathcal{F}_{s,i}$ is the refined prediction;
$\epsilon$ is a random variable for the respective prior, which is jointly
learned by the feature refinement modules representing the 2D-to-3D priors and
the GRU network trained with maximum likelihood estimation losses; $h(\cdot)$
is the prediction head. The proof of Eq. (3) can be found in the supplement.

The key insight is that the voxels back-projected from either depth prediction
or semantic label prediction of the input images has strong evidence on its 3D
counterparts. We hereby define anchored voxels $\alpha_{i}$, as those voxels
in $\mathcal{F}_{s,i}$ that are incorporating all the back-projected depth
points, given the fact that the 3D reconstruction task is essentially an
inverse problem. We propose two progressive feature refinement modules to
learn the high confidence of the refined features in latent space such that a
more accurate $\hat{I}_{s,i}$ can be extracted with the help of 2D-to-3D prior
knowledge.

Depth-Anchored Occupancy Refinement.  Unlike the volumetric methods [27, 39]
that directly regress on the TSDF volume, we propose a novel module in each
stage $s$ that can explicitly refine the initial depth, predict depths in
resolutions, and further create the 3D anchored features with the depth
prediction, as shown in Fig. 3. The anchored feature is generated by 3D sparse
convolutions with an anchored voxel on the occupancy intermediate $o_{i}$
222The universal stage suffix $s$ is hereinafter omitted for brevity..

Figure 3: Workflow of the depth anchor refinement module. Anchored voxels are
extracted from depth points and further serve as a geometric prior for the
occupancy refinement.

Intuitively, the anchored voxel has higher confidence of achieving a valid
$o_{i}$ and $T_{s,i}$ close to zero. We imposed the anchored feature on the
occupancy feature to reinforce the occupancy information brought by the depth
prior.

Inspired by [6, 33], we conduct PointFlow algorithm for each stage in the
coarse-to-fine structure $\mathcal{V}_{s}$ to determine the depth displacement
on the initial depth prediction such that finer depth prediction can be
achieved. Different from the PointFlow algorithm used in [33], we utilize the
back-projected depth points from all $N_{k}$ views in the fragment to query an
anchored voxel, which can be further aggregated with $o_{i}$. Fig. 4
illustrates how these hypothesis points are selected and turned into depth
displacement prediction, such that the anchored voxel can be generated.

Figure 4:  Anchored voxel generation for occupancy refinement.  An example of
occupancy refinement happening on the middle row of a $3\times 6\times 3$ FBV
is shown with geometrically valid voxel highlighted in green. The initial
depth prediction is back-projected into FBV and displaced by trilinear
interpolation on all depth points, in the range of 6 additional hypothesis
points for each depth point. The voxels on the top are set as half transparent
for clarity.

The anchored voxel index in the 3D volume is sparsified as a mask to update
the occupancy prediction as $\hat{o}_{i}$ in the following:

$\hat{o}_{i}=o_{i}\cap\alpha_{i}\enspace.$ (4)

The enhanced occupancy prediction $\hat{o}_{i}$ is used to condition the TSDF
volume at the current stage to generate the refined $\hat{T}_{i}$, which is
further sparsified with a light-weight pointwise convolution and upsampled to
concatenate with $L_{s,i}$.

Pixel-to-Vertex Matching Semantic Refinement.  In addition to the depth anchor
refinement, we propose a semantic cross-dimensional refinement which utilizes
the semantic prior that lies in the 2D semantic prediction to have a refined
3D voxel semantic prediction, implemented as follows. First, the 2D feature
backbone learns the 2D semantic prior information that is useful for 3D voxel
semantic labeling learning by incorporating the learning of 2D frame semantic
labeling. Second, the sparse 3D feature $L_{s,i}$ is passed to pointwise 3D
convolution layers and comes up with the initial 3D voxel semantic labeling
predictions in respective scales. Third, to conduct the semantic feature
refinement, we observed that there is a sole 3D voxel counterpart in
$\mathcal{F}_{s,i}$ for each pixel on a 2D semantic prediction of
$\mathbf{I}_{i,j}$, since the surface edges are encoded as vertices. We define
these vertices as the one-on-one matching correspondences to their camera-
projected pixels, which are recorded in a matching matrix for masking the 2D
features $\mathcal{P}_{s}$.

The upper part of Fig. 5 illustrates the design of the matching matrix that is
used to correlate the pixel-vertex pairs for each frame $\mathbf{I}_{i,j}$
across all vertices in $\mathcal{F}_{s,i}$. We construct the matching matrix
$\mathbf{M}=\\{\overrightarrow{m}_{idx}\\}^{N}_{idx=1}$ for each semantic
labeling frame, where $N$ is the number of the vertices in the volume
$\mathcal{F}_{s,i}$. Each column of the matching matrix $\mathbf{M}$ is
defined as:

$\mathbf{M}(idx)=\overrightarrow{m}_{idx}=\begin{bmatrix}u_{idx}\\\ v_{idx}\\\
\text{mask}\end{bmatrix}.$ (5)

For each column, each pixel-vertex pair recorded in the matching matrix, i.e.,
the $idx^{\text{th}}$ vertex in the 3D volume on the right-hand side of the
upper part and its correspondence pixel on the left-hand side is recorded. The
last entry of the pixel-vertex pair represents a mask which is recorded as
valid when the 2D correspondence for $\mathbf{M}$ is in the current view
frustum of the frame.

After the matching matrix $\mathbf{M}$ is constructed, it will be used for
masking each of the feature map $\mathcal{P}_{s}$ with the $\log_{2}$ scale of
$s$ to create a refined feature, whose voxel number is the same as the number
of sparse 3D features, as shown in the lower part of Fig. 5. Meanwhile, the
coordinates of the sparse 3D features $L_{s,i}$ are mapped as the coordinate
of the refined feature. By doing so, the underlying semantic information from
the $\mathcal{P}_{s}$ can be incorporated by $L_{s,i}$, such that better 3D
semantic prediction can be achieved. Then we use the sparse pointwise
convolution to extract its underlined feature from 2D semantics and
concatenate it with $L_{s,i}$ to create $L_{s-1,i}$ with semantic information
for the refinement in the next finer stage, so as to ensure the 2D semantic
priors to have reliable refinement on the sparse coherent features.

###  3.3 Implementation Details

Our model is implemented in PyTorch, trained and tested on an NVIDIA RTX3090
graphics card. We empirically set the optimizer as Adam without weight decay
[24], with an initial learning rate of 0.001, which goes through 3 halves
throughout the training. The first momentum and second momentum are set to 0.9
and 0.999, respectively. For key frame selection, following [44, 39], we set
thresholds $\theta_{key}$, $t_{key}$ and fragment input number $N_{k}$ as 15
degrees, 0.1 meters, and 9, respectively. A fraction of FPN [22] is adopted as
the 2D backbone with its classifier as MNasNet [41]. MinkowskiEngine [8] is
utilized as the sparse 3D tensor library. More details are introduced in the
supplement.

Figure 5:  Workflow of the pixel-to-vertex matching feature refinement. Upper:
Matching matrix $\mathbf{M}$ for pixel-to-vertex correspondence is constructed
with camera projection. The red-shaded boxes in the 3D volume denote an
example of valid correspondence pairs of the 2D semantic prediction
$\overrightarrow{m}_{a}$ and its surrounding 3D scene. The green and purple
boxes in the 3D volume view denote the occluded vertex and out-of-view vertex
that is not imaged in the 2D semantic prediction, which correspond to
$\overrightarrow{m}_{b}$ and $\overrightarrow{m}_{z}$, respectively; Lower:
The 2D features are further masked by $\mathbf{M}(a)$ with the mapped
coordinates from the sparse 3D features of the scene that are valid for the
current view.

Loss Design.  Our model is trained in an end-to-end fashion except for the
pre-trained 2D backbone. Since our target is to learn the 3D geometry and
semantic segmentation of the surrounding scene given the posed images input,
we regress the TSDF value with the mean absolute error (MAE) loss, classify
the occupancy value with the binary cross-entropy (BCE) loss and the semantic
labeling with cross-entropy (CE) loss as:

$\displaystyle\mathcal{L}_{3D}=$
$\displaystyle\sum^{4}_{s=2}\alpha_{s}\mathcal{L}_{\text{MAE}}(T_{s},\hat{T}_{s})+\lambda\alpha_{s}\mathcal{L}_{\text{BCE}}(O_{s},\hat{O}_{s})$
$\displaystyle+\beta_{s}\mathcal{L}_{\text{CE}}(S_{s},\hat{S}_{s})\enspace,$
(6)

where $T$, $S$, and $O$ denote TSDF value, semantic labeling, and occupancy
predictions. $\alpha_{s}$, $\beta_{s}$, and $\lambda$ are the weighting
coefficients in different stages for TSDF volume, semantic volume, and
positive weight for BCE loss, respectively. By doing so, the learning process
stays most sensitive and relevant to the supervisory signals in the coarse
stage, and less fluctuating as the 3D features become finer with the
upsampling, after log-transforming the predicted and ground-truth TSDF value
following [27].

To conduct cross-dimensional refinements, we regress the depth estimation with
MAE loss and classify the 2D semantic segmentation with CE loss:

$\displaystyle\mathcal{L}_{2D}=$
$\displaystyle\mathcal{L}_{\text{MAE}}(d_{init},\hat{D}_{init})+\mathcal{L}_{\text{CE}}(S^{2D}_{2},\hat{S}^{2D}_{2})$
$\displaystyle+\sum^{4}_{s=2}\gamma_{s}\mathcal{L}_{\text{MAE}}(D_{s},\hat{D}_{s})\enspace,$
(7)

where $D$ and $\gamma_{s}$ denote depth and the weighting coefficient for
depth estimation in different stages. We further wrap the losses into an
overall loss $\mathcal{L}=\mathcal{L}_{3D}+\mu\mathcal{L}_{2D}$, where $\mu$
is the coefficient to balance the joint learning of 2D and 3D.

Figure 6: Qualitative 3D reconstruction results on ScanNet. Our method is
capable of reconstructing consistent and detailed geometry which is neither
overly smooth as the one from Atlas [27] nor eroded with holes as from
NeuralRecon [39].

##  4 Experiments

###  4.1 Datasets and Metrics

Method |  Acc. $\downarrow$ |  Comp. $\downarrow$ |  Prec. $\uparrow$ |  Recall $\uparrow$ |  F-Score $\uparrow$
---|---|---|---|---|---
Atlas [27] | 0.124 | 0.074 | 0.382 | 0.711 | 0.499
NeuralRecon [39] | 0.073 | 0.106 | 0.450 | 0.609 | 0.516
3DVNet [33] | 0.051 | 0.075 | 0.715 | 0.625 | 0.665
SimpleRecon [36] | 0.061 | 0.055 | 0.686 | 0.658 | 0.671
VoRTX [38] | 0.089 | 0.092 | 0.618 | 0.589 | 0.603
Ours | 0.068 | 0.062 | 0.609 | 0.616 | 0.612
|  |  |  |  |

Table 1: Quantitative 3D reconstruction results on ScanNet. Our method is
superior to two main baselines, Atlas and NeuralRecon, and as competitive as
other SOTAs on 3D reconstruction.

We conduct the experiments on two indoor scene datasets, ScanNet (v2) [10] and
SceneNN [16]. The model is trained on the ScanNet train set, tested and
reported on the ScanNet test set and further verified on SceneNN data set. To
quantify the 3D reconstruction and 3D semantic segmentation capability of our
method, we use the standard metrics following [27, 39]. Completeness Distance
(Comp.), Accuracy Distance (Acc.), Precision, Recall, and F-score, are used
for 3D reconstruction, while mean Intersection over Union (mIoU) is used for
3D semantic segmentation.

To evaluate how much robustness a model can achieve while targeting 3D
perception tasks in real time, we define the 3D perception efficiency metric
$\eta_{3D}=\text{FPS}\times\text{mIoU}\times\text{F-score}$, since F-score is
regarded as the most suitable 3D metric for evaluating 3D reconstruction
quality by considering Precision and Recall at the same time [27, 39, 36]. It
is noteworthy that for fairness across methods, FPS for processing speed is
measured in the inference across all captured frames in a given video sequence
rather than key frames only, since the input is the same for different methods
regardless of their key frame selection scheme.

Method | FPS $\uparrow$ | KFPS $\uparrow$ | FLOPF $\downarrow$ | mIoU $\uparrow$ |  $\eta_{3D}$ $\uparrow$
---|---|---|---|---|---
3DMV [11] | 7.04 | N/A | 65.06G | 44.2 | N/A
BPNet [15] | 4.46 | N/A | 141.06G | 74.9 | N/A
Atlas [27] | 66.3 | N/A | 267.04G | 34.0 | 11.25
NeuralRecon [39] \+ Semantics-Heads | 228 | 30.9 | 42.38G | 27.9 | 32.82
VoRTX [38] \+ Semantic-Heads | 119 | 13.5 | 150.23G | 13.2 | 9.47
Ours | 158 | 21.4 | 90.62G | 39.1 | 37.81
|  |  |  |  |

Table 2: Quantitative 3D voxel semantic segmentation and overall 3D perception
results on ScanNet. Upper: Two representative state-of-the-art methods for
semantic segmentation whose input requires either depth or 3D mesh,
respectively. No key-frame selection and F-score are involved due to their
input modality; Lower: RGB-input-only volumetric methods. Key-frame FPS (KFPS)
is measured with the same selection scheme across all methods. FLOPF is
measured with PyTorch operation counter across operations of neural network’s
learnable modules.  Figure 7: Qualitative 3D semantic segmentation results on
ScanNet. Our method consistently outperforms baseline models and sometimes
even surpasses the ground-truth labeling, e.g., in the bottom row, the photo-
printed curtain above the bed is correctly recognized as “curtain” and
“picture”, whereas the ground truth mistakes it as “other furniture”.

###  4.2 Evaluation Results and Discussion

3D Perception.  To evaluate the 3D perception capability, we mainly compare
our methods against state-of-the-art works in two categories: volumetric 3D
reconstruction and voxelized 3D semantic segmentation methods.

For 3D reconstruction capability, we compare our proposed method with the
canonical volumetric methods [27, 39] and several state-of-the-art 3D
reconstruction methods with posed images input [33, 36]. Fig. 6 demonstrates
the superiority of our method in terms of 3D reconstruction by showing the 3D
meshing results in normal mapping. Table 1 shows that our method outperforms
two main baseline methods in terms of 3D meshing accuracy. We further compare
both state-of-the-art depth estimation methods and volumetric methods in depth
metrics in the supplement to justify from the depth extraction perspective.

For 3D semantic segmentation quality, we compare Atlas, NeuralRecon with
semantic heads, and VoRTX with semantic heads with our methods in Table 2. We
augment three stages of MLP heads on top of the flattened 3D features to
predict the semantic segmentation for both baselines. Due to its lack of 3D
feature extraction, SimpleRecon, as one of the SOTA baselines, is
intrinsically incapable of following this modification for semantics as well
as being combined with our proposed cross-dimensional refinement techniques.
Table 2 shows that our method outperforms these two baselines. Besides mIoU
for semantic segmentation, we include FPS and $\eta_{3D}$ for 3D perception
efficiency in the comparison. We also include two state-of-the-art 3D semantic
segmentation methods, 3DMV [11] and BP-Net [15]. It shows that our method can
achieve mIoU results nearly comparable to 3DMV, but with only RGB images as
input. Overall, our method achieves the best 3D semantic segmentation
performance and highest 3D perception efficiency among all the volumetric
methods. Fig. 7 and Fig. 8 illustrate the 3D semantic labeling results. We
found that the semantic information generation on VoRTX is unsatisfying,
mostly caused by its bias on geometric features brought by the projective
occupancy mentioned in [38].

Efficiency.  Since our main goal is to achieve real-time processing
performance while solving 3D perception tasks, we compare the computational
efficiency of our model against other RGB-input-only volumetric methods in
Table 2. The 3D perception efficiency metric $\eta_{3D}$ for several 3D
semantic segmentation works are shown there. We employ FPS, which is commonly
used to measure efficiency for 2D-input 3D perception methods [27, 39, 38], as
a metric to bring out and emphasize the nature of real-time system. We also
include the floating-point operations per frame (FLOPF) to compare the
learnable parameters’ operations across different methods. The superiority in
$\eta_{3D}$ of our method manifests that it has better deployment potential
for real-life 3D perception applications. From the human user’s and robotic
SLAM’s points of view, our method greatly surpasses the threshold of being
real-time, 90.17 FPS, as elaborated in the supplement. It shows that our
method is more suitable for real-time industrial scenarios with input data
from low-cost portable devices compared to baseline methods.

Method | Acc. $\downarrow$ | Comp. $\downarrow$ | FPS $\uparrow$ | F-Score $\uparrow$ | mIoU $\uparrow$ |  $\eta_{\text{3D}}$ $\uparrow$
---|---|---|---|---|---|---
Atlas [27] | 0.074 | 0.164 | 54.7 | 0.499 | 31.4 | 8.57
NeuralRecon [39] \+ Semantic-Heads | 0.138 | 0.216 | 178 | 0.510 | 15.9 | 14.43
Ours | 0.068 | 0.143 | 121 | 0.611 | 36.7 | 27.09
|  |  |  |  |  |

Figure 8: Qualitative and quantitative 3D pereception results on SceneNN
dataset. Our method is proven to be generalized to SceneNN without pre-
training on the SceneNN train set.

###  4.3 Ablation Study

To analyze the effectiveness of cross-dimensional refinement, we present 3D
perception efficiency $\eta_{3D}$ and its components of with different
modifications in Table 3. In other experiments above, we adopt (e) as our
method.

Binomial GRU Fusion. In (a), we remove the back-projected semantics input to
GRU in the pipeline. Compared with (e), both F-score and mIoU of the removal
degrade since no hidden semantic information from last FBV is fused with GRU
anymore. Although FPS increases due to fewer computations, the efficiency
$\eta_{3D}$ is worse.

Depth Refinement. In (c), we remove the depth anchor refinement in the
pipeline. The loss in F-score and mIoU manifests that the geometric feature
without depth anchor refinement becomes inferior, which means depth anchor
refinement can improve 3D reconstruction performance.

| GRU Input | Depth | Semantics | F-Score$\uparrow$ | mIoU$\uparrow$ | FPS $\uparrow$ | $\eta_{3D}\uparrow$
---|---|---|---|---|---|---|---
| DE | AR | SE | PVR
(a) | Geo. | ✓ | ✓ | ✓ | ✓ | 0.477 | 31.7 | 190 | 28.73
(b) | Geo.+ Sem. | ✓ |  | ✓ |  | 0.479 | 27.1 | 232 | 30.12
(c) | Geo.+ Sem. | ✓ |  | ✓ | ✓ | 0.482 | 34.5 | 169 | 28.10
(d) | Geo.+ Sem. | ✓ | ✓ | ✓ |  | 0.556 | 26.8 | 226 | 33.68
(e) | Geo.+ Sem. | ✓ | ✓ | ✓ | ✓ | 0.612 | 39.1 | 158 | 37.81
|  |  |  |  |  |  |  |  |

Table 3: Ablation study. We assess our method by removing each of the proposed
feature fusion techniques on ScanNet. DE, AR, SE, and PVR denote depth
estimation, anchored refinement, 2D semantics estimation, and point-to-vertex
refinement, respectively.

Semantic Refinement. We validate the semantic refinement in the pipeline by
removing this module and, as shown in (d). The mIoU drops due to the
insufficient learning information from semantic heads only. This result
demonstrates the effectiveness of our semantic refinement scheme based on
pixel-to-vertex matching for improving 3D semantic segmentation performance.
We also experiment with no refinements but depth and 2D semantics learning
setup in (b), which gives the highest FPS but not satisfying 3D perception
performance.

##  5 Conclusion

In this paper, we proposed a lightweight volumetric method, CDRNet, that
leverages the 2D latent information about depths and semantics as the feature
refinement to handle 3D reconstruction and semantic segmentation tasks
effectively. We demonstrated that our method has real-time 3D perception
capabilities, and justified the significance of utilizing 2D prior knowledge
when solving 3D perception tasks. Experiments on multiple datasets justify the
3D perception performance improvement of our method compared to prior arts.
From the application point of view, the scalability of CDRNet supports the
notion that 2D priors should not be disregarded in 3D perception tasks and
opens up new avenues for achieving real-time 3D perception using input data
from readily accessible portable devices such as smartphones and tablets.

Acknowledgements:  This work is in part supported by Bright Dream Robotics
(BDR) and the HKUST-BDR Joint Research Institute Funding Scheme under Project
HBJRI-FTP-005 (OKT22EG06).

## References

  * [1] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies.  Neural rgb-d surface reconstruction.  In CVPR, pages 6290–6301, 2022.
  * [2] Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner.  Transformerfusion: Monocular rgb scene reconstruction using transformers.  In CVPR, volume 34, pages 1403–1414, 2021.
  * [3] Aljaz Bozic, Pablo Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner.  Neural deformation graphs for globally-consistent non-rigid reconstruction.  In CVPR, pages 1450–1459, 2021.
  * [4] Anh-Quan Cao and Raoul de Charette.  Monoscene: Monocular 3d semantic scene completion.  In CVPR, pages 3991–4001, 2022.
  * [5] Tommaso Cavallari and Luigi Di Stefano.  Semanticfusion: Joint labeling, tracking and mapping.  In ECCV, pages 648–664. Springer, 2016.
  * [6] Rui Chen, Songfang Han, Jing Xu, and Hao Su.  Point-based multi-view stereo network.  In ICCV, pages 1538–1547, 2019.
  * [7] Jaesung Choe, Sunghoon Im, Francois Rameau, Minjun Kang, and In So Kweon.  Volumefusion: Deep depth fusion for 3d scene reconstruction.  In ICCV, pages 16086–16095, 2021.
  * [8] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In CVPR, pages 3075–3084, 2019.
  * [9] Manuel Dahnert, Ji Hou, Matthias Nießner, and Angela Dai.  Panoptic 3d scene reconstruction from a single rgb image.  In NeurIPS, volume 34, pages 8282–8293, 2021.
  * [10] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, pages 5828–5839, 2017.
  * [11] Angela Dai and Matthias Nießner.  3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation.  In NeurIPS, pages 452–468, 2018.
  * [12] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In ICCV, pages 2495–2504, 2020.
  * [13] Lei Han, Tian Zheng, Yinheng Zhu, Lan Xu, and Lu Fang.  Live semantic 3d perception for immersive augmented reality.  IEEE Transactions on Visualization and Computer Graphics, 26(5):2012–2022, 2020.
  * [14] Ziyang Hong and C. Patrick Yue.  Efficient-grad: Efficient training deep convolutional neural networks on edge devices with gradient optimizations.  In ACM Transactions on Embedded Computing Systems (TECS), volume 21, pages 1–24. ACM New York, NY, 2022.
  * [15] Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, and Tien-Tsin Wong.  Bidirectional projection network for cross dimension scene understanding.  In CVPR, pages 14373–14382, 2021.
  * [16] Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung.  Scenenn: A scene meshes dataset with annotations.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 92–101. IEEE, 2016.
  * [17] Shi-Sheng Huang, Haoxiang Chen, Jiahui Huang, Hongbo Fu, and Shi-Min Hu.  Real-time globally consistent 3d reconstruction with semantic priors.  IEEE Transactions on Visualization & Computer Graphics, 01:1–1, 2021.
  * [18] Maximilian Jaritz, Jiayuan Gu, and Hao Su.  Multi-view pointnet for 3d scene understanding.  In ICCVW, pages 0–0, 2019.
  * [19] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu.  Ditto: Building digital twins of articulated objects from interaction.  In CVPR, pages 5616–5626, 2022.
  * [20] Yann LeCun.  1.1 deep learning hardware: Past, present, and future.  In 2019 IEEE International Solid-State Circuits Conference (ISSCC), pages 12–19. IEEE, 2019.
  * [21] Chenguang Li, Jia Shi, Ya Wang, and Guangliang Cheng.  Reconstruct from top view: A 3d lane detection approach based on geometry structure prior.  In CVPRW, pages 4370–4379, 2022.
  * [22] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [23] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  SIGGRAPH, 21(4):163–169, 1987.
  * [24] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2018.
  * [25] Jisan Mahmud, True Price, Akash Bapat, and Jan-Michael Frahm.  Boundary-aware 3d building reconstruction from a single overhead image.  In CVPR, pages 441–451, 2020.
  * [26] Davide Menini, Suryansh Kumar, Martin R Oswald, Erik Sandström, Cristian Sminchisescu, and Luc Van Gool.  A real-time online learning framework for joint 3d reconstruction and semantic segmentation of indoor scenes.  In IEEE Robotics and Automation Letters, volume 7, pages 1332–1339. IEEE, 2021.
  * [27] Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich.  Atlas: End-to-end 3d scene reconstruction from posed images.  In ECCV, pages 414–431. Springer, 2020.
  * [28] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji.  Panopticfusion: Online volumetric semantic mapping at the level of stuff and things.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4205–4212. IEEE, 2019.
  * [29] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 127–136. IEEE, 2011.
  * [30] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, pages 165–174, 2019.
  * [31] Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung.  Real-time progressive 3d semantic segmentation for indoor scenes.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1089–1098. IEEE, 2019.
  * [32] Vivek Pradeep, Christoph Rhemann, Shahram Izadi, Christopher Zach, Michael Bleyer, and Steven Bathiche.  Monofusion: Real-time 3d reconstruction of small scenes with a single web camera.  In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 83–88. IEEE, 2013.
  * [33] Alexander Rich, Noah Stier, Pradeep Sen, and Tobias Höllerer.  3dvnet: Multi-view depth prediction and volumetric refinement.  In 2021 International Conference on 3D Vision (3DV), pages 700–709. IEEE, 2021.
  * [34] Luis Roldao, Raoul De Charette, and Anne Verroust-Blondet.  3d semantic scene completion: A survey.  IJCV, 130(8):1978–2005, 2022.
  * [35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [36] Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard.  Simplerecon: 3d reconstruction without 3d convolutions.  In ECCV, 2022.
  * [37] Christiane Sommer, Lu Sang, David Schubert, and Daniel Cremers.  Gradient-sdf: A semi-implicit surface representation for 3d reconstruction.  In CVPR, pages 6280–6289, 2022.
  * [38] Noah Stier, Alexander Rich, Pradeep Sen, and Tobias Höllerer.  Vortx: Volumetric 3d reconstruction with transformers for voxelwise view selection and fusion.  In 2021 International Conference on 3D Vision (3DV), pages 320–330. IEEE, 2021.
  * [39] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao.  Neuralrecon: Real-time coherent 3d reconstruction from monocular video.  In CVPR, pages 15598–15607, 2021.
  * [40] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, pages 2446–2454, 2020.
  * [41] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In CVPR, pages 2820–2828, 2019.
  * [42] Carlos A Vanegas, Daniel G Aliaga, and Bedrich Benes.  Building reconstruction using manhattan-world grammars.  In CVPR, pages 358–365, 2010.
  * [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, volume 30, 2017.
  * [44] Kaixuan Wang and Shaojie Shen.  Mvdepthnet: Real-time multiview depth estimation neural network.  In 2018 International Conference on 3D Vision (3DV), pages 248–257. IEEE, 2018.
  * [45] Silvan Weder, Johannes Schonberger, Marc Pollefeys, and Martin R Oswald.  Routedfusion: Learning real-time depth map fusion.  In CVPR, pages 4887–4897, 2020.
  * [46] Silvan Weder, Johannes L Schonberger, Marc Pollefeys, and Martin R Oswald.  Neuralfusion: Online depth fusion in latent space.  In CVPR, pages 3162–3172, 2021.
  * [47] Yabin Xu, Liangliang Nan, Laishui Zhou, Jun Wang, and Charlie CL Wang.  Hrbf-fusion: Accurate 3d reconstruction from rgb-d data using on-the-fly implicits.  In ACM TOG, volume 41, pages 1–19. ACM New York, NY, 2022.
  * [48] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In ECCV, pages 767–783, 2018.
  * [49] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  In NeurIPS, 2022.