# In-gap band in the one-dimensional two-orbital Kanamori-Hubbard model with
inter-orbital Coulomb interaction

N. Aucar Boidi1<EMAIL_ADDRESS>H. Fernández García1 Current
affiliation: University of Antwerp, 2020 Antwerp, Belgium Y. Núñez-Fernández2
K. Hallberg1 1 Centro Atómico Bariloche and Instituto Balseiro, CNEA and
CONICET, 8400 Bariloche, Argentina  2 CEA-Grenoble, IRIG-PHELIQS, 38000
Grenoble, France

###### Abstract

We study the electronic spectral properties at zero temperature of the one-
dimensional (1D) version of the degenerate two-orbital Kanamori Hubbard model
(KHM), one of the well established frameworks to study transition metal
compounds, using state-of-the-art numerical techniques based on the Density
Matrix Renormalization Group. While the system is Mott insulating for the
half-filled case, as expected for an interacting 1D system, we find
interesting and rich structures in the single-particle density of states (DOS)
for the hole-doped system. In particular, we find the existence of in-gap
states which are pulled down to lower energies from the upper Hubbard band
(UHB) with increasing the inter-orbital Coulomb interaction $V$. We analyze
the composition of the DOS by projecting it onto different local excitations
and we observe that for large dopings these in-gap excitations are formed
mainly by inter-orbital holon-doublon (HD) states and their energies follow
approximately the HD states in the atomic limit. We observe that the Hund
interaction $J$ increases the width of the in-gap band, as expected from the
two-particle fluctuations in the Hamiltonian. The observation of a finite
density of states within the gap between the Hubbard bands for this extended
1D model indicates that these systems present a rich excitation spectra which
could help us understand the microscopic physics behind multi-orbital
compounds.

##  I Introduction

Understanding the microscopic mechanisms in materials with strong electron-
electron correlations due to interactions in local orbitals, stands out as one
of the most challenging problems in condensed matter physics. For example,
materials like transition-metal oxides with partially filled $d$ or $f$ shells
give rise to interesting properties like high temperature superconductivity,
colossal magnetoresistance, correlation-driven metal-insulator transitions,
heavy fermion behavior or an orbital selective Mott phaseVojta ; Vojta2 .

The discovery of a family of materials with similar characteristics in low
dimensions, such as those with correlated electrons in ladders, gives us the
possibility of a more detailed understanding of the underlying physical
mechanisms. This is because of the availability of more accurate theoretical
and numerical tools for one dimensional models, such as the Density Matrix
Renormalization Group techniqueWhite ; Karen1 ; Karen2 ; ramasesha ; KW .
Among these systems we can mention the AFe2X3 family, where A: K, Rb, Cs, and
Ba, and X are chalcogenides X: S, Se, which are formed by double chains of
edge sharing FeX4 tetrahedra.Takubo  In particular BaFe2S3Yamauchi ; Takahashi
and BaFe2Se3Ying ; Zhang  were found to superconduct under high pressure. The
latter compound behaves in a manner compatible with orbital selective Mott
physicsCaron ; Rincon ; Dong . However, in spite of the theoretical progress
made in the calculation of static properties, it is still difficult to obtain
precise and detailed theoretical electronic structure results to compare with
experiments, like angular resolved photoemission (ARPES), inverse photo-
emission experiments (IPE) or optical conductivity measurements.

In this paper we report on results obtained for the zero-temperature local
density of states (DOS) for the model Hamiltonian which decribes these multi-
orbital systems, the two-orbital Kanamori-Hubbard model (KHM) KanamoriHubbard
; reviewGeorges  in one dimension, which includes a ferromagnetic Hund
coupling $J$ between the orbitals, in the doped regime, with equal bandwidths,
using the DMRG for the calculation of static and dynamical properties Karen2 ;
ramasesha ; KW .

By carefully analyzing the local electronic density of states (DOS) for a
large range of parameters and dopings, we find a well defined in-gap band for
large enough values of the inter-orbital Coulomb interaction ($V$). This
structure is additional to the well known upper and lower Hubbard bands (UHB,
LHB). As we show in this work, for large dopings, this in-gap band is formed
mainly by holon-doublon pairs. These results are similar to the findings in
Refs. yurielhd ; yurieldop ; Yashar  for the two-orbital KHM but on a Bethe
lattice using DMFT where an additional peak (in the half-filled case) or an
additional band (for the doped case) were observed in the presence of $V$. In
our case, as we are considering an interacting one-dimensional system, which
is insulating for the half-filled case, we must dope the system to study its
metallic behavior, where this new band appears. Although previous work
reported holon-doublon pairs in related models at higher energiesfulde ; wang
; picos ; karski , and also as metastable states out of equilibriumprelovsek ;
rinconfeiguin , the existence of an in-gap band, like the one we are
presenting in this paper, was not reported before for this model.

##  II Model and Method

We study the degenerate two-orbital Kanamori-Hubbard model in 1D:

$H=\sum_{\left\langle
ij\right\rangle\alpha\sigma}t_{\alpha}c_{i\alpha\sigma}^{\dagger}c_{j\alpha\sigma}-(\mu-\epsilon)\sum_{i}n_{i}+\sum_{i}{H}_{i}\mbox{,}$
(1)

where $\left\langle ij\right\rangle$ are nearest-neighbor sites on a chain,
$\alpha=1,2$ are orbital indices, and $\sigma$ the spin index. The creation
and destruction operators are $c^{\dagger}$ and $c$, respectively and
$n_{i}=\sum_{\alpha\sigma}c_{i\alpha\sigma}^{\dagger}c_{i\alpha\sigma}$ the
on-site number operator. The nearest-neighbor hoppings for orbital 1 and 2
take the value $t_{1}=t_{2}=0.5$ which is taken as the unit of energy. No
inter-orbital hybridization is considered. Here $\mu$ is the chemical
potential where $\mu=0$ leads to half-filled bands. This implies that the site
energies must take the values $\epsilon=-U/2-V+J/2$

The on-site interactions $H_{i}$ are:

$\begin{array}[]{c}{H_{i}}=U\sum_{\alpha}n_{i\alpha\uparrow}n_{i\alpha\downarrow}+\sum_{\sigma\sigma^{\prime}}\left(V-J\delta_{\sigma\sigma^{\prime}}\right)n_{i1\sigma}n_{i2\sigma^{\prime}}-\\\
-J\left(c_{i1\uparrow}^{\dagger}c_{i1\downarrow}c_{i2\downarrow}^{\dagger}c_{i2\uparrow}+c_{i2\uparrow}^{\dagger}c_{i2\downarrow}c_{i1\downarrow}^{\dagger}c_{i1\uparrow}\right)\\\
-J\left(c_{i1\uparrow}^{\dagger}c_{i1\downarrow}^{\dagger}c_{i2\uparrow}c_{i2\downarrow}+c_{i2\uparrow}^{\dagger}c_{i2\downarrow}^{\dagger}c_{i1\uparrow}c_{i1\downarrow}\right)\end{array}$
(2)

where $J>0$ is the local exchange Hund’s coupling and $U$ ($V$) is the intra
(inter)-orbital Coulomb repulsion between electrons (see Fig. 1).

Figure 1:  Sketch of the interactions included in the KHM (Eq. (1)). Also
shown is the representation of the inter-orbital HD excitation.

###  II.1 Method

To calculate the static and dynamical properties of the model we use the
matrix-product-state (MPS) implementation of the DMRG in the OSP order
(orbital-spin-position, varying the position index first) together with the
inversion symmetry along the chain axis and open boundary conditions. However,
for the calculation of the band dispersions in Fig. 7, we use periodic
boundary conditions. This geometry led to the most reliable results.

For the DOS we calculate the following dynamical response functions:

$A^{>}(\omega)=-\frac{1}{\pi}\Im\langle
c_{1\uparrow}(\omega+i\eta-H+E_{0})^{-1}c_{1\uparrow}^{\dagger}\rangle$ (3)
$A^{<}(\omega)=-\frac{1}{\pi}\Im\langle
c_{1\uparrow}^{\dagger}(\omega+i\eta-H+E_{0})^{-1}c_{1\uparrow}\rangle$ (4)

where the expectation is taken for the ground state with energy $E_{0}$ of the
Hamiltonian (1).

All dynamical functions are obtained using the correction-vector
methodramasesha  directly on the real axis (with a very small imaginary offset
or Lorentzian broadening $\nu=0.15$) at zero temperature on a wide scale of
energies. Smaller values of this broadening result in features in the DOS
revealing its finite-size structure. The main results of this paper do not
depend on this value.

We study system sizes of $L=12$ that correspond to $12$ physical lattice sites
with two orbitals each. The number $m$ of states used to calculate ground
states energies (the bond dimension) was set to 1024 and up to 512 for
correlation functions and we performed up to eight sweeps reaching a good
convergence with $m$. We have also done calculations for other system sizes,
showed in Fig. 6, to see possible size effects. In these results, we can
observe that the main features of the DOS remain the same.

##  III Results

####  III.0.1 Zero Hund interaction $J=0$

We will consider first the simpler case in which the Hund coupling is $J=0$
and study its effect later, including the rotationally invariant case defined
when $V=U-2J$.reviewGeorges

For the half-filled case ($\mu=0$) the system is a Mott insulator for finite
values of $U$ as seen in Fig. 2, where the UHB and LHB can be seen. When the
system is doped with holes we observe an additional structure in between these
Hubbard bands. This structure evolves towards higher energies with $\mu$,
following the atomic HD excitations, marked with arrows (see Table I in Ref.
yurieldop ). For $\mu\geq-U/2$ ($J=0$) the atomic limit HD excitation is
$(U-V)$ while for $\mu<-U/2$ it is $(U/2-V-\mu)$. The light blue line
corresponds to the projection of the DOS onto local HD excitations as
explained in Sect. IV.

Figure 2:  DOS for the two orbital KHM for $U=4$, $J=0$, $L=12$, and $V=2$
while varying the chemical potential $\mu$. $\omega=0$ corresponds to the
Fermi energy in all figures. For $\mu=0$ the UHB and LHB are present and the
system is a Mott insulator. When the system is doped with holes an in-gap band
is formed. Also shown is $A_{HD}(\omega)$ (light blue line) and the energies
of the HD state in the atomic limit (arrows).

It is also interesting to analyze the effect of the inter-orbital interaction
$V$, as we show in Fig. 3. When $V=0$ the system consists of two independent
interacting Hubbard chains and both Hubbard bands are clearly recognizeable.
When $V$ is turned on we observe a transfer of weight from the UHB to an in-
gap band which decreases in energy as $V$ grows. The case $V=4=U$ corresponds
to the rotationally invariant situation ($J=0$) and the in-gap band merges
with the central structure. We also observe a dip in the DOS at the Fermi
energy $\omega=0$ for finite values of $V$. As studied below (see Sect. IV),
these states are formed mainly by excitations of the kind $|0,\uparrow\rangle\
$ and $|\uparrow,\uparrow\rangle$ (and all other spin projections) in the
atomic limit.

Figure 3:  DOS for the two orbital KHM for $U=4$, $J=0$, $L=12$, and $\mu=-2$
while increasing the inter-orbital interaction $V$. For $V=0$ the UHB and LHB
are present and the system is a metal. Also shown is $A_{HD}(\omega)$ (light
blue line).

####  III.0.2 Finite Hund interaction $J$

In Fig. 4 we present results for the DOS in the presence of a finite inter-
orbital Hund interaction $J$. We observe changes in the DOS at low energies
and also in the in-gap band. As we show below, the DOS close to the Fermi
energy $\omega=0$ is formed, mainly, by singly occupied stated in both
orbitals, which are affected by the spin flip term in the Hamiltonian (1). The
widening produced by $J$ in the in-gap band is produced, on the other hand, by
the two-particle fluctuations in 1 since this band is formed mainly by HD
pairs.conJ

Figure 4:  DOS for the two orbital KHM for $U=4$, $V=2$, $L=12$, and $\mu=-2$
while increasing the inter-orbital Hund interaction $J$. $J=1$ corresponds to
the rotationally symmetric case $V=U-2J$. Also shown is $A_{HD}(\omega)$
(light blue line).

##  IV Characterization of the excitations

In order to understand the effect of the inter-orbital interaction $V$ on the
DOS we have calculated several dynamical response functions which correspond
to the projection of the DOS onto particular atomic states on each site: one
site is composed by two orbitals and its states is represented as
$|s1,s2\rangle$. In a similar way as done in Ref. yurieldop  we define the
Green’s functions
$A_{s_{1},s_{2}}(\omega)=A_{s_{1},s_{2}}^{>}(\omega)+A_{s_{1},s_{2}}^{<}(-\omega)$
with:

$A_{s_{1},s_{2}}^{>}(\omega)=-\frac{1}{\pi}\Im\langle
c_{1\uparrow}(\omega+i\eta-H+E_{0})^{-1}X_{s_{1},s_{2}}^{\dagger}\rangle$ (5)
$A_{s_{1},s_{2}}^{<}(\omega)=-\frac{1}{\pi}\Im\langle
c_{1\uparrow}^{\dagger}(\omega+i\eta-H+E_{0})^{-1}X_{s_{1},s_{2}}\rangle$ (6)

where the expectation is taken for the ground state with energy $E_{0}$ of the
Hamiltonian (1). The excitations are
$X_{s_{1},s_{2}}^{\dagger}=P_{s_{1},s_{2}}c_{1\uparrow}^{\dagger}$ and their
reverse action $X_{s_{1},s_{2}}=c_{1\uparrow}P_{s_{1},s_{2}}$. The projector
$P_{s_{1},s_{2}}=|s_{1},s_{2}\rangle\langle s_{1},s_{2}|$ is used to select
the corresponding atomic configuration $|s_{1},s_{2}\rangle$. Note that adding
all possible configurations gives the total DOS since
$\sum_{s_{1},s_{2}}P_{s_{1},s_{2}}=1$.

We are particularly interested in the following excitations (and their reverse
actions) for orbital 1 (similarly for orbital 2), where we add over all spin
projections:

(i) HD states
($|\downarrow,0\rangle\to|\uparrow\downarrow,0\rangle=|d,0\rangle$):
$X_{d,0}^{\dagger}=n_{1\downarrow}(1-n_{2\uparrow})(1-n_{2\downarrow})c_{1\uparrow}^{\dagger}$

(ii) $|0,0\rangle\to|\uparrow,0\rangle$:
$X_{\uparrow,0}^{\dagger}=(1-n_{1\downarrow})(1-n_{2\uparrow})(1-n_{2\downarrow})c_{1\uparrow}^{\dagger}$

(iii) $|0,\uparrow\rangle\to|\uparrow,\uparrow\rangle$:
$X_{\uparrow,\uparrow}^{\dagger}=n_{2\uparrow}(1-n_{2\downarrow})(1-n_{1\downarrow})c_{1\uparrow}^{\dagger}$

(iv)
$|\downarrow,\uparrow\rangle\to|\uparrow\downarrow,\uparrow\rangle=|d,\uparrow\rangle$:
$X_{d,\uparrow}^{\dagger}=n_{2\uparrow}(1-n_{2\downarrow})n_{1\downarrow}c_{1\uparrow}^{\dagger}$

In Fig. 5 we present the results for the projection of the DOS onto these
atomic states. It is interesting to observe that for $V=0$ (Fig. 5a) we do not
observe the in-gap structure and all excitations containing doublons
contribute to the UHB, as expected. When we turn on the inter-orbital Coulomb
interaction $V$ (Fig. 5b) there is a transfer of spectral weight towards lower
energies forming a new band which is constituted mainly by HD excitations. The
weight of these HD excitations in the UHB is negligible. When the doping is
increased further (e.g. $\mu=-3$, (Fig. 5c)) the HD character of the new band
is increased and we find, as expected, a reduction in the total weight of the
UHB. The curve corresponding to the HD projection is also shown in the
previous figures to signal the extra band. In Fig. 2 we see how the HD
excitations increase their energy with hole doping (also marked with an arrow
following the projection onto the atomic HD states as previously explained).
Here we find that for larger dopings the in-gap band has a larger weight in
the HD states. In Fig. 3 we see how this HD projection accompanies the new
band towards lower energies with $V$ while in Fig. 4 we see how this new band
widens slightly with $J$.

We also find that the low energy excitations are formed mainly by states of
the form $|0,\uparrow\rangle\ $ and $|\uparrow,\uparrow\rangle$. The
observation of regions with charge density waves (CDW) and superconductivity
for the low hole doping limit of the rotationally invariant case of this model
in Ref. Dagotto  suggests that this low energy structure could be related to
superconducting or CDW fluctuations. We leave the analysis of these low energy
excitations for further study.

Figure 5:  DOS projected onto different atomic states defined in the text for
$U=4$, $J=0$ and $L=12$. a) $V=0$, $\mu=-2$; b) $V=2$, $\mu=-2$ and c) $V=2$,
$\mu=-3$ (other projections have negligible weight and are not shown).

In Fig. 6 we plot the DOS for several system sizes, showing that the existence
of the in-gap band is robust.

Figure 6:  DOS for different system sizes ($L$ is the number of two-orbital
sites) for $U=4$, $V=2$, $J=0$ and $\mu=-2$.

##  V Band dispersions

We also calculate the spectral functions $A(\omega,k)$, which are obtained by
Fourier transforming the single-particle Green’s function in real space with
periodic boundary conditions using DMRG.

The momentum dispersions of the bands are plotted in Fig. 7. In the right
column we show the dispersion of the HD excitations. For $V=0$ (which
corresponds to two independent single-orbital chains) we observe that the HD
excitations are located mainly at the zone boundaries, $k=\pm\pi$, while for
finite $V$ the HD excitations seem to be more extended in $k$. In this latter
case we can clearly see that the HD band is formed at lower energies, as shown
before with the DOS. We also see the dip in the DOS at low energies. For $J=1$
we observe a broadening of the HD band together with a reduction of the dip
close to the Fermi energy ($\omega=0$).

Figure 7:  Heat plots of $A(\omega,k)$ for $U=4$, $\mu=-2$ and $L=12$ and
periodic boundary conditions. a) $V=0$, $J=0$, b) $V=2$, $J=0$ and c) $V=2$,
$J=1$ (rotationally symmetric case). Figures in the right column show the
dispersions projected onto de HD excitations for each case.

##  VI Conclusions

In this work we studied the $T=0$ single electron spectral functions of the 1D
two-orbital Kanamori-Hubbard model which is the basic model to describe a wide
range of correlated multi-orbital materials. We resort to high precision DMRG
numerical calculation for the ground states and the dynamical response
functions. This allowed us to observe a rich structure in the DOS for a wide
range of parameters. A salient feature for the hole doped case is the presence
of an in-gap band with a large component on holon-doublon excitations whose
spectral weight is transfered from the upper Hubbard band for intermediate
values of the inter-orbital Coulomb interaction $V$. We expect that these in-
gap excitations will be also observed in two and three dimensional versions of
this model, as already indicated in previous work using DMFT. yurielhd ;
yurieldop  One main difference is that for higher dimensions one can also
analyze the half-filled case which is metallic for small enough local
interactions. As shown in Ref. yurielhd , the HD band is also observed in this
case, where a metallic band is needed to provide for the holes or the doubly
occupied states which form the HD excitations. For the one-dimensional case
studied in this paper, the holes are provided by doping.

We also studied the energy dispersion and found that these excitations are
less concentrated close to the zone boundaries when compared to the $V=0$
case. The new in-gap band shown in this paper should also show up in other
observables such as optical conductivity. Work is underway to analyze the low
energy features and the consequences of electron doping.

###### Acknowledgements.

We acknowledge support from projects PICT 2016-0402 and PICT 2018-01546 from
the Argentine ANPCyT and PIP2015 - 11220150100538 CO (CONICET). We thank
Cristian Batista, Adriana Moreo and Elbio Dagotto for useful dicussions.

## References

  * (1) H. von Löhneysen, A. Rosch, M.Vojta, and P. Wölfle, Fermi-liquid instabilities at magnetic quantum phase transitions, Rev. Mod. Phys. 79,1015 (2007)
  * (2) M. Vojta, Orbital-Selective Mott Transitions: Heavy Fermions and Beyond, J. Low Temp Phys. 161, 203 (2010).
  * (3) S. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992)
  * (4) K. Hallberg, New trends in density matrix renormalization, Adv. Phys. 55, 477-526 (2006)
  * (5) K. Hallberg, Density-matrix algorithm for the calculation of dynamical properties of low-dimensional systems, Phys. Rev. B 52 R9827 (1995)
  * (6) S. Ramasesha, S.K. Pati, H.R. Krishnamurthy, Z. Shuai, and J.L. Brédas, Low-lying electronic excitations and nonlinear optic properties of polymers via symmetrized density matrix renormalization group method, Synthetic Metals 85, 1019 (1997)
  * (7) T. Kühner and S. White, Dynamical correlation functions using the density matrix renormalization group, Phys. Rev. B 60, 335 (1999)
  * (8) K. Takubo, et al., Orbital order and fluctuations in the two-leg ladder materials BaFe2X3(X=S and Se) and CsFe2Se3, Phys. Rev. B 96, 115157 (2017)
  * (9) T. Yamauchi, Y. Hirata, Y. Ueda, and K. Ohgushi, Pressure-Induced Mott Transition Followed by a 24-K Superconducting Phase in BaFe2S3, Phys. Rev. Lett. 115, 246402 (2015)
  * (10) H. Takahashi, et al., Pressure-induced superconductivity in the iron-based ladder material BaFe2S3, Nat. Mater. 14, 1008 (2015)
  * (11) J. Ying, H. Lei, C. Petrovic, Y. Xiao, and V. Struzhkin, Interplay of magnetism and superconductivity in the compressed Fe-ladder compound BaFe2Se3, Phys. Rev. B 95, 241109(R) (2017)
  * (12) Y. Zhang, L.-F. Lin, J.-J. Zhang, E. Dagotto, and S. Dong, Sequential structural and antiferromagnetic transitions in BaFe2Se3 under pressure, Phys. Rev. B 97, 045119 (2018)
  * (13) J. M. Caron, J. R. Neilson, D. C. Miller, K. Arpino, A. Llobet, and T. M. McQueen, Orbital-selective magnetism in the spin-ladder iron selenides Ba1-xKxFe2Se3, Phys. Rev. B 85, 180405(R) (2012)
  * (14) J. Rincón, A. Moreo, G. Alvarez, and E. Dagotto, Exotic Magnetic Order in the Orbital-Selective Mott Regime of Multiorbital Systems, Phys. Rev. Lett. 112, 106405 (2014)
  * (15) S. Dong, J.-M. Liu, and E. Dagotto, BaFe2Se3: A High TC Magnetic Multiferroic with Large Ferrielectric Polarization, Phys. Rev. Lett. 113, 187204 (2014)
  * (16) J. Kanamori, Electron Correlation and Ferromagnetism of Transition Metals, Prog. Theor. Phys. 30 275 (1963)
  * (17) A. Georges, L. De’ Medici, and J. Mravlje, Strong Correlations from Hund’s Coupling, Annu. Rev. Cond. Matt. Phys. 4, 137 (2013)
  * (18) Y. Núñez- Fernández, G. Kotliar, and K. Hallberg, Emergent low-energy bound states in the two-orbital Hubbard model, Phys. Rev. B 97, 121113(R) (2018)
  * (19) K. Hallberg and Y. Núñez- Fernández, Subbands in the doped two-orbital Kanamori-Hubbard model, Phys. Rev. B 102, 245138 (2020)
  * (20) Y. Komijani, K. Hallberg, and G. Kotliar, Renormalized dispersing multiplets in the spectrum of nearly Mott localized systems, Phys. Rev. B. 99, 125150 (2019)
  * (21) T. A. Kaplan, P. Horsch, and P. Fulde, Close Relation between Localized-Electron Magnetism and the Paramagnetic Wave Function of Completely Itinerant Electrons, Phys. Rev. Lett. 49, 889 (1982)
  * (22) S. Zhou, Y. Wang, and Z. Wang, Doublon-holon binding, Mott transition, and fractionalized antiferromagnet in the Hubbard model, Phys. Rev. B 89, 195119 (2014)
  * (23) S. B. Lee, J. von Delft, and A. Weichselbaum, Doublon-Holon Origin of the Subpeaks at the Hubbard Band Edges, Phys. Rev. Lett. 119, 236402 (2017) and references therein.
  * (24) M. Karski, C. Raas, and G. S. Uhrig, Electron spectra close to a metal-to-insulator transition, Phys. Rev. B 72, 113110 (2005)
  * (25) Z. Lenarčič and P. Prelovšek, Charge recombination in undoped cuprates, Phys. Rev. B. 90, 235136 (2014)
  * (26) J. Rincón, E. Dagotto, and A. E. Feiguin, Photoinduced Hund excitons in the breakdown of a two-orbital Mott insulator, Phys. Rev. B 97, 235104 (2018)
  * (27) Y. Niu, J. Sun, Y. Ni, J. Liu, Y. Song, and S. Feng, Doublon-holon excitations split by Hund’s rule coupling within the orbital-selective Mott phase, Phys. Rev. B 100, 075158 (2019)
  * (28) N. D. Patel, N. Kaushal, A. Nocera, G. Alvarez and E. Dagotto, Emergence of superconductivity in doped multiorbital Hubbard chains, npj Quantum Mater. 5, 27 (2020)


# 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting

Xiaoyang Lyu  ,  Yang-Tian Sun  The University of Hong KongHong Kong ,  Yi-Hua
Huang  ,  Xiuzhe Wu  The University of Hong KongHong Kong ,  Ziyi Yang
Zhejiang UniversityHangzhouChina ,  Yilun Chen  ,  Jiangmiao Pang  Shanghai AI
LabShanghaiChina and  Xiaojuan Qi  The University of Hong KongHong Kong

###### Abstract.

In this paper, we present an implicit surface reconstruction method with 3D
Gaussian Splatting (3DGS), namely 3DGSR, that allows for accurate 3D
reconstruction with intricate details while inheriting the high efficiency and
rendering quality of 3DGS. The key insight is incorporating an implicit signed
distance field (SDF) within 3D Gaussians to enable them to be aligned and
jointly optimized. First, we introduce a differentiable SDF-to-opacity
transformation function that converts SDF values into corresponding Gaussians’
opacities. This function connects the SDF and 3D Gaussians, allowing for
unified optimization and enforcing surface constraints on the 3D Gaussians.
During learning, optimizing the 3D Gaussians provides supervisory signals for
SDF learning, enabling the reconstruction of intricate details. However, this
only provides sparse supervisory signals to the SDF at locations occupied by
Gaussians, which is insufficient for learning a continuous SDF. Then, to
address this limitation, we incorporate volumetric rendering and align the
rendered geometric attributes (depth, normal) with those derived from 3D
Gaussians. This consistency regularization introduces supervisory signals to
locations not covered by discrete 3D Gaussians, effectively eliminating
redundant surfaces outside the Gaussian sampling range. Our extensive
experimental results demonstrate that our 3DGSR method enables high-quality 3D
surface reconstruction while preserving the efficiency and rendering quality
of 3DGS. Besides, our method competes favorably with leading surface
reconstruction techniques while offering a more efficient learning process and
much better rendering qualities. The code will be available at
https://github.com/CVMI-Lab/3DGSR.

Gaussian Splatting, Implicit Function, Signed Distance Function, Volumetric
Rendering

Figure 1. Our method, called 3DGSR, achieves accurate 3D surface
reconstruction with rich details, while maintaining the efficiency and high-
quality rendering of 3DGS. The left part illustrates the performance
comparison of 3DGSR with cutting-edge reconstruction methods. The right part
shows the capability of our method to achieve high-quality reconstruction and
rendering results simultaneously.

##  1\. Introduction

3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has emerged as a new state-
of-the-art approach for high-quality novel view synthesis. This method
represents the geometry and appearance of a 3D scene as a set of Gaussians,
which are then optimized from posed multi-view images. Thanks to its Gaussian
rasterization pipeline, 3DGS achieves real-time efficiency, even when
rendering high-resolution outputs. Despite its impressive rendering quality
and efficiency, 3DGS generates only noisy, incomplete points for 3D geometry
and struggles to accurately reconstruct the 3D surface of a scene (Tang et
al., 2023a). These, however, are highly desired in various geometry-related
applications, such as 3D reconstruction, geometry editing, animation, and
relighting, among others.

This motivates us to investigate how to enable 3DGS for high-quality surface
reconstruction while preserving its rendering capabilities and speed. The
inability of 3DGS to faithfully represent a 3D surface stems from its
unstructured point-based geometry representation (i.e., the center of
Gaussian). During optimization, the positions of these points (Gaussians) are
adjusted to capture the geometry of the scene. However, the learning objective
is to improve the rendering quality, which may not necessarily lead to
accurate positions. The limited number of views utilized for learning further
exacerbates this issue. Besides, the optimization process does not consider
the geometric constraints among the points that are indeed constrained by the
underlying geometry (i.e., surfaces), leading to noisy points. Further, the
discrete and unstructured points make it difficult to extract 3D surfaces
through post-processing, such as Poisson surface reconstruction (Kazhdan and
Hoppe, 2013).

In this paper, we introduce 3DGSR, a novel approach that incorporates a neural
implicit signed distance field (SDF) within Gaussians for geometry modeling,
which is then jointly optimized and aligned with 3D Gaussians. First, we
propose a differentiable SDF-to-opacity transformation function that converts
SDF values into corresponding Gaussians’ opacities, thus connecting SDF and
Gaussians. This association enforces surface constraints on Gaussians, aligns
the SDF with Gaussians, and facilitates their unified optimization. Next,
during learning, optimizing the Gaussians will provide supervisory signals for
updating the SDF, allowing the reconstruction of intricate details. However,
this only offers sparse supervisory signals to the SDF at locations occupied
by the Gaussians, which proves inadequate for optimizing a continuous SDF and
may lead to the formation of redundant structures. To address this issue, we
further integrate volumetric rendering (Mildenhall et al., 2021; Wang et al.,
2021) and align the rendered depth (normal) with that obtained from 3D
Gaussians through consistency loss to optimize the SDF. This approach
effectively regularizes locations not covered by 3D Gaussians, eliminating
unnecessary surface reconstructions beyond the Gaussian sampling range (see
Fig. 3). It is worth noting that since we only sample a few rays during each
learning iteration, this does not introduce significant computational costs.
The coupled representation and learning of the 3D Gaussians and the implicit
SDF field form a positive feedback cycle, enabling them to improve each other
through mutual learning and ultimately yield high-quality rendering and
surface reconstruction. As illustrated in Fig. 1, our approach achieves
accurate reconstruction and high-quality rendering results simultaneously.

Comprehensive experiments conducted on various datasets show that our method
is capable of producing high-quality reconstruction results while
simultaneously preserving the efficiency and qualities of 3DGS. When compared
to state-of-the-art methods for surface reconstruction (Wang et al., 2021;
Yariv et al., 2021; Tang et al., 2023b), our approach substantially surpasses
them in rendering quality and facilitates the reconstruction of intricate
structures with significantly shorter learning time.

##  2\. Related Works

Our method builds upon prior works in 3D reconstruction with multi-view images
to enable fast surface reconstruction and novel view synthesis. We discuss
prior works in multi-view 3D reconstruction, neural implicit representation,
and point-based rendering.

#### Multi-view Stereo

Traditional multi-view stereo (MVS) methods have employed point-based (Barnes
et al., 2009; Stereopsis, 2010; Galliani et al., 2016; Schönberger et al.,
2016) and voxel-based (De Bonet and Viola, 1999; Broadhurst et al., 2001;
Seitz and Dyer, 1999) representation. Point-based methods estimate the 3D
positions of pixels by matching multi-view images and finding pixel
correspondences. The surface can then be obtained by performing screened
Poisson surface reconstruction (Kazhdan and Hoppe, 2013) on the resulting
point clouds. However, these methods heavily rely on matching accuracy and
struggle to reconstruct textureless regions. On the other hand, voxel-based
methods cannot achieve high-quality reconstruction due to the resolution
limitation caused by high memory consumption.

#### Neural Implicit Representations

In recent years, the fields implicitly predicted by neural networks have
gained significant attention in both novel view synthesis (Lombardi et al.,
2019; Sitzmann et al., 2019; Mildenhall et al., 2021) and 3D reconstruction
(Yariv et al., 2020; Niemeyer et al., 2020; Kellnhofer et al., 2021; Jiang et
al., 2020; Liu et al., 2020b). Neural Radiance Field (NeRF) (Mildenhall et
al., 2021) has demonstrated excellent performance in high-quality novel view
synthesis by modeling scenes with a density field and a view-dependent color
field. However, NeRF’s density field cannot extract high-quality surfaces. To
achieve accurate surface reconstruction, NeuS (Wang et al., 2021) represents
the scene with a signed distance function (SDF) and derives the alpha value
from the precondition that the zero-level surface contributes the most to the
volume rendering. Following NeuS, Darmon et al. (Darmon et al., 2022) improves
the reconstruction quality with an additional warping loss that ensures
photometric consistency. Neuralangelo (Li et al., 2023b) introduces multi-
resolution hash grids (Müller et al., 2022) and regularizes the model with
numerical gradients and coarse-to-fine training. However, these methods often
require extensive training time, which can limit their practical
applicability.

Several works have focused on accelerating neural implicit representation by
using implicit-explicit hybrid representation with grids (Sun et al., 2022;
Karnewar et al., 2022; Müller et al., 2022; Wang et al., 2023a; Fridovich-Keil
et al., 2022), trees (Yu et al., 2021; Liu et al., 2020a), points (Xu et al.,
2022; Dai et al., 2023; Chen et al., 2023b), or low-rank decomposition (Chan
et al., 2022; Hu et al., 2023; Chen et al., 2022). These methods speed up the
training process and promote the rendering speed of radiance fields to a
remarkable real-time level. However, fast and high-quality surface
reconstruction from multi-view images remains a challenging problem.

#### Point-based Rendering

Unstructured points (Gross and Pfister, 2011) can be efficiently rendered to
obtain novel-view images. However, directly rasterizing colored point clouds
(Grossman and Dally, 1998) with fixed point sizes can lead to disconnected
artifacts such as discontinuity, holes, and aliasing. To address these issues,
previous works (Botsch et al., 2005; Pfister et al., 2000; Ren et al., 2002;
Zwicker et al., 2001) extend points to larger primitives (e.g., spheres,
ellipsoids, and surfels) and splat them onto the image plane. Differentiable
point-based methods (Rakhimov et al., 2022; Pittaluga et al., 2019; Aliev et
al., 2020) tackle these artifacts by rendering a holed latent feature map and
then decoding it as a full RGB image using neural networks. Alternatively,
some approaches render points as primitives (e.g., surfels and Gaussians) with
trainable parameters and render them differentially (Yifan et al., 2019; Zhang
et al., 2022; Kerbl et al., 2023).

Most recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has
demonstrated remarkable performance in terms of high-level rendering quality
and real-time rendering. Subsequent works have extended 3DGS to handle 3D/4D
generation (Yi et al., 2023; Tang et al., 2023a; Zielonka et al., 2023; Ren et
al., 2023; Yang et al., 2023a), dynamic scenes (Luiten et al., 2023; Yang et
al., 2023b, 2024; Das et al., 2023; Li et al., 2023a; Huang et al., 2023),
relighting (Gao et al., 2023), and animation (Jung et al., 2023; Ye et al.,
2023). Some concurrent works (Guédon and Lepetit, 2023; Chen et al., 2023a)
aim to reconstruct the 3D surface from multi-view images using 3DGS. NeuSG
(Chen et al., 2023a) jointly trains a NeuS (Wang et al., 2021) and 3DGS.
During training, the normals derived from NeuS regularize the rotation of
Gaussians, while NeuS is also regularized to ensure that the SDF values at
Gaussians’ positions are close to zero. SuGaR (Guédon and Lepetit, 2023)
enforces Gaussians to have limited overlap with others during the optimization
of 3DGS. After that, an initial mesh can be extracted by performing Poisson
reconstruction (Kazhdan and Hoppe, 2013) on Gaussians, using the shortest axis
as normals. The mesh is then bound with flat Gaussians distributed on its
faces and optimized to obtain refined results.

##  3\. Preliminaries

#### 3D Gaussian Splatting (3DGS)

3DGS (Kerbl et al., 2023) employs 3D Gaussian points, characterized by a
central point $\mu$ and a covariance matrix $\Sigma$, to effectively render
images from given viewpoints:

(1) $G(x)=\exp\left(-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)\right).$

To facilitate optimization, $\Sigma$ is factorized into the product of a
scaling matrix $S$, represented by scale factors $s$, and a rotation matrix
$R$ encoded by a quaternion $r\in\mathbb{R}^{4}$:

(2) $\Sigma=RSS^{T}R^{T},$

For rendering novel views, the 3D Gaussians are projected onto a 2D image
plane according to elliptical weighted average (EWA) (Zwicker et al., 2001):

(3) $\Sigma^{\prime}=JW\Sigma W^{T}J^{T},$

where $W$ is the viewing transformation matrix, and $J$ is the Jacobian
derived from projective transformations. The final color $C(u)$ at pixel $u$
is rendered by combining $N$ ordered Gaussian splats using point-based
$\alpha$-blending:

(4) $C(u)=\sum_{i\in N}T_{i}\alpha_{i}c_{i},\quad
T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j}).$

Here, $c_{i}$ is determined by the spherical harmonic (SH) function and the
viewing direction, while $\alpha_{i}$ is computed from the 2D covariance
$\Sigma^{\prime}$, modulated by the opacity $\sigma$ of each Gaussian (Kerbl
et al., 2023). In addition to rendering color $c_{i}$, Eq. (4) is also
employed to render normal and depth as detailed below.

#### Normal and Depth Estimation from Gaussian

Typically, each 3D Gaussian can be treated as a conventional point cloud by
using the centers of 3D Gaussians. Then, we can leverage the estimated depth
(normal) and render them into a depth (normal) map using point-based alpha
blending (Gao et al., 2023) as:

(5) $\\{\mathcal{D},\mathcal{N}\\}=\sum_{i\in
N}T_{i}\alpha_{i}\\{d_{i},\mathbf{n}_{i}\\}.$

Here $d_{i}$ and $\mathbf{n}_{i}$ are the depth and normal of $i$th point
respectively.

##  4\. Method

Figure 2. Pipeline of our proposed approach for implicit surface
reconstruction. We model the surface with an implicit SDF field, with which
the SDF value of each 3D Gaussian can be predicted. The opacity of each
Gaussian is modeled as a bell-shaped function of its distance to the implicit
surface. The geometry attributes of 3D Gaussians serve as a regularization for
the SDF field, while the rendered image is supervised by the captured image.

Given posed multiview images $\\{\mathcal{I}_{k}\\}$ and a point cloud
obtained from the Structure from Motion algorithm (Schönberger and Frahm,
2016), our goal is to enable 3DGS with the ability to reconstruct accurate
surface while preserving its rendering quality and efficiency. To accomplish
this, we propose an approach integrating neural implicit signed distance field
(SDF) within 3D Gaussians as shown in Fig. 2. A hash grid and a single-layer
MLP are employed to predict the SDF value at each queried location.
Additionally, we introduce a differentiable SDF-to-opacity transformation
function $\Phi$ to link and align SDF and 3D Gaussians. Subsequently, given a
3D Gaussian, its opacity value is determined by querying the corresponding SDF
value and transforming it using the function $\Phi$. This coupled design
allows the optimization of 3D Gaussians to supply supervisory signals for
updating the SDF, enabling the reconstruction of intricate details. Details
are unfolded in (Sec. 4.1).

However, the above is insufficient for optimizing the continuous SDF, as it
can only provide supervision at locations where Gaussians are present. This
results in floating artifacts in free space due to inadequate supervision (see
Fig.3). To tackle this issue, we utilize volumetric rendering to generate
depth and normals by querying points along the cast ray from the neural
implicit SDF. Subsequently, a depth (normal) consistency loss is employed to
align the rendered depth (normal) with those derived from 3D Gaussians,
effectively regularizing the locations not covered by Gaussians and limiting
floaters. The details are elaborated in 4.2.

###  4.1. Connect 3D Gaussians with Implicit SDF

The vanilla 3DGS method optimizes millions of Gaussians to accurately fit the
training views, enabling photorealistic synthesis of novel views. However, its
main emphasis is on fitting the target views, which does not guarantee high-
quality 3D geometries. Moreover, the 3D Gaussians should be constrained by the
underlying geometry (i.e., surfaces) of the 3D scene, which is absent in
current methods. Consequently, 3DGS faces limitations in accurately
reconstructing 3D surfaces (see Fig. 5).

Here, we introduce neural implicit SDF, denoted as
$f:\mathbb{R}^{3}\rightarrow\mathbb{R}^{1}$, which maps each 3D position
$x\in\mathbb{R}^{3}$ to its signed distance from the target surface. This is
implemented using multi-resolution hash grids (Müller et al., 2022) in
conjunction with a single-layer MLP. The surface $\mathrm{S}$ of a scene can
then be derived by using the zero-level set of its SDF, defined by:

(6) $\mathrm{S}=\\{x\in\mathbb{R}^{3}\>|\>f(\mathbf{x})=0\\}.$

Next, to link the SDF with 3D Gaussians, we introduce a differentiable SDF-to-
opacity transformation function. This function should adhere to the constraint
that points closer to the surface contribute more to rendering a pixel and,
therefore, have a higher opacity value. Consequently, we select a bell-shaped
function $\Phi_{\beta}(f(x))$ controlled by a hyperparameter $\gamma$,
expressed as:

(7) $\Phi_{\beta}(f(x))=\frac{e^{-\beta\cdot f(x)}}{(1+e^{-\beta\cdot
f(x)})^{2}},$

where $\beta$ is a learnable parameter controlling the shape of the function.

This function transforms SDF values at a query location into the opacity of
the corresponding Gaussian at that location. By linking the SDF with 3D
Gaussians using a differentiable function, the SDF field can be updated
through the optimization of 3D Gaussians by minimizing the photometric loss:

(8)
$\mathcal{L}_{c}=\lambda\mathcal{L}_{\text{D-SSIM}}+(1-\lambda)\mathcal{L}_{1},$

This loss function encourages the Gaussian at a location $x$ that contributes
more to rendering a pixel to have a higher opacity value $\Phi_{\beta}(f(x))$.
This, in turn, encourages the SDF $f(x)$ to be closer to $0$ and, therefore,
nearer to the surface. The high-quality rendering capability of 3DGS allows
for learning detailed image structures, which facilitates the reconstruction
of detailed structures (see Fig. 5).

In addition to $\mathcal{L}_{c}$, we also impose a constraint as Eq. (9) to
maintain consistency between the rendered normal $\mathcal{N}$, obtained
directly from Gaussians via Eq. 5, and a pseudo normal $\hat{\mathcal{N}}$,
computed from the rendered depth $\mathcal{D}$ derived from 3D Gaussians under
the local planarity assumption.

(9) $\mathcal{L}_{n}=\|\mathcal{N}-\hat{\mathcal{N}}\|_{2}.$

Further, to encourage $f$ to approximate a valid signed distance function, we
follow the Eikonal equation (Gropp et al., 2020) to penalize the deviation of
$f$:

(10) $\mathcal{L}_{\text{eik}}=\mathbb{E}_{x}[(\|\triangledown f-1\|)^{2}].$

Additionally, we enforce that the distribution of Gaussians adheres to the
surface by constraining the SDF values at their centers to be close to zero:

(11) $\mathcal{L}_{\text{pt}}=\|f(x_{g})\|_{1},$

where $x_{g}$ denotes the centers of 3D Gaussians.

###  4.2. Regularization with Volumetric Rendering

By integrating the SDF field with 3D Gaussians and training them jointly, as
discussed in Section 4.1, we can attain high-fidelity rendering results with
an optimized SDF field. However, due to the sparse and discrete nature of 3D
Gaussians, the aforementioned supervision signals are only available for
locations occupied by 3D Gaussians, predominantly near the surface. This is
insufficient to regularize the continuous SDF, resulting in artifacts at
locations not covered by Gaussians, such as floating artifacts in unoccupied
free spaces, as illustrated in Fig. 3.

Figure 3. Comparison of our method with and without volumetric rendering
regularization. The incorporation of volumetric rendering regularization
proves to be highly effective in eliminating floating artifacts in free space
where Gaussians are absent. Figure 4. Comparison of volume rendering results
and Gaussian splatting results on the TNT dataset. The resemblance
demonstrates the efficacy of our learning process in aligning the rendered
geometric attributes - depth and normal - derived from volumetric rendering
(which is based on the learned SDF) and those obtained from 3DGS.

Since volumetric rendering employs ray-casting to render a pixel, it samples
points along a ray that includes both occupied and free points. We propose
incorporating volumetric rendering for optimization. The depth map and normal
map of volume rendering can effectively represent the occupancy distribution
in the whole space, while those of sparse Gaussians aligns well with the
observed image. This inspires us to adopt the rendering results of Gaussians’
geometry attributes as the guidance of volume rendering, which can be utilized
as the medium for propagating gradients. Specifically, we employ volumetric
rendering to render depth (normals) and supervise them using depth (normals)
derived from Gaussians using Eq. (5). This can provide regularizations to SDF
values at locations uncovered by Gaussians.

More specifically, for each pixel, the ray cast from it is denoted as
$\\{\mathbf{p}(t)=r_{o}+t\cdot r_{d}\>\|\>t>0\\}$, where $r_{o}$ is the camera
center and $r_{v}$ is the unit direction vector of the ray. We accumulate the
depth $\hat{\mathbf{D}}(r)$ and normal $\hat{\mathbf{N}}(r)$ along the cast
ray (Wang et al., 2021):

(12)
$\tilde{\mathbf{D}}(r)=\sum_{i=1}^{M}T^{r}_{i}\alpha_{i}t^{r}_{i}\;,\tilde{\mathbf{N}}(r)=\sum_{i=1}^{M}T^{r}_{i}\alpha_{i}\tilde{n}^{r}_{i}\;,$

where $\tilde{n}$ is the normal computed by the gradient of SDF field
$\triangledown f$, $T^{r}_{i}$ and $\alpha_{i}$ represent the transmittance
and alpha value (a.k.a opacity) of the sample point, and their values can be
computed by

(13)
$T^{r}_{i}=\prod_{j=1}^{i-1}(1-\alpha_{i}),~{}\alpha_{i}=\max\left(\frac{\phi_{s}(f(x_{i}))-\phi_{s}(f(x_{i+1}))}{\phi_{s}(f(x_{i}))},0\right),\;$

where $\delta^{r}_{i}$ is the distance between adjacent sample points and
$\phi_{s}$ is the Sigmoid function.

After that, we utilize the depth $\mathcal{D}$ and normal $\mathcal{N}$
rendered by Gaussian Splatting to supervise the volumetric rendering results
$\mathbf{D}$ and $\mathbf{N}$ by optimizing the following objectives:

(14)
$\mathcal{L}_{\text{vd}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\mathcal{D}(r)-\tilde{\mathbf{D}}(r)\|_{2}\;,$

and

(15)
$\mathcal{L}_{\text{vn}}=\sum_{\mathbf{r}\in\mathcal{R}}\|{\mathcal{N}}(\mathbf{r})-\tilde{\mathbf{N}}(\mathbf{r})\|_{1}+\|1-\mathcal{N}(\mathbf{r})\tilde{\mathbf{N}}(\mathbf{r})\|_{1}\;,$

where $\mathcal{R}$ is the union of all training rays. These constraints
regularize all the sampling points along the training rays, resulting in dense
supervision for the implicit field and eliminating floating artifacts. As
shown in Fig. 2, there exist noisy SDF distributions (negative value) floating
over the free space between the camera and the object, which is then removed
by applying the regularization with volumetric rendering. Accurate geometry
can be obtained through our careful training strategy and model design. We
visualize the optimized depth map and normal map of both volume rendering and
Gaussian rendering in Fig 4.

The overall training objective is defined as a combination of all
aforementioned losses:

(16)
$\mathcal{L}=\mathcal{L}_{c}+\lambda_{1}\mathcal{L}_{n}+\lambda_{2}\mathcal{L}_{\text{eik}}+\lambda_{3}\mathcal{L}_{\text{pt}}+\lambda_{4}\mathcal{L}_{\text{vd}}+\lambda_{5}\mathcal{L}_{\text{vn}},$

where $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5}$ are
hyperparameters that control the relative importance of each loss term.

##  5\. Experiments

###  5.1. Experimental Settings

#### Datasets

Our evaluation encompasses three datasets: the NeRF synthetic dataset
(Mildenhall et al., 2021), the DTU real-captured dataset (Jensen et al.,
2014), and the Tanks&Temples (TNT) large-scale dataset (Knapitsch et al.,
2017). The NeRF synthetic dataset consists of eight objects, each rendered via
path tracing to generate 100 views. For testing, we select an eighth of these
images, allocating the remainder for training. The DTU and TNT datasets are
processed for camera poses as per (Gao et al., 2023), with image and mask
processing following the methodology outlined in (Liu et al., 2020a). The
image resolutions for the DTU and TNT datasets are $1600\times 1200$ and
$1920\times 1080$ pixels, respectively.

#### Comparisons

We evaluated our method from both surface reconstruction quality and rendering
quality of novel-view synthesis. We benchmark our algorithm against several
state-of-the-art methods, including NeRF (Mildenhall et al., 2021), NeuS (Wang
et al., 2021), NeuSG (Chen et al., 2023a), RelighableGaussian (Gao et al.,
2023), and NeRF2Mesh (Tang et al., 2023b), Neuralangelo (Li et al., 2023b),
PET-NeuS (Wang et al., 2023b), BakedSDF (Yariv et al., 2023), VolSDF (Yariv et
al., 2021). The quality of image synthesis is quantified using the Peak
Signal-to-Noise Ratio (PSNR). For surface reconstruction accuracy, we employ
the Chamfer distance and the F1 score as our metrics (Knapitsch et al., 2017).
Furthermore, we also compare the rendering quality of our method with some
works that solely focus on novel view synthesis tasks (Tri-MipRF (Hu et al.,
2023), Zip-NeRF (Barron et al., 2023), 3DGS (Kerbl et al., 2023)).

|  | Chair | Drums | Ficus | Hotdog | Lego | Materials | Mic | Ship | Avg
---|---|---|---|---|---|---|---|---|---|---
PSNR | NeRF (Mildenhall et al., 2021) | 34.17 | 25.08 | 30.39 | 36.82 | 33.31 | 30.03 | 34.78 | 29.30 | 31.74
Ins-NGP (Müller et al., 2022) | 35.00 | 26.02 | 33.51 | 37.40 | 36.39 | 29.78 | 36.22 | 31.10 | 33.18
Mip-NeRF (Barron et al., 2021) | 35.14 | 25.48 | 33.29 | 37.48 | 35.70 | 30.71 | 36.51 | 30.41 | 33.09
3D-GS (Kerbl et al., 2023) | 35.36 | 26.15 | 34.87 | 37.72 | 35.78 | 30.00 | 35.36 | 30.80 | 33.32
NeuS (Wang et al., 2021) | 31.22 | 24.85 | 27.38 | 36.04 | 34.06 | 29.59 | 31.56 | 26.94 | 30.20
NeRO (Liu et al., 2023) | 28.74 | 24.88 | 28.38 | 32.13 | 25.66 | 24.85 | 28.64 | 26.55 | 27.48
BakedSDF (Yariv et al., 2023) | 31.65 | 20.71 | 26.33 | 36.38 | 32.69 | 30.48 | 31.52 | 27.55 | 29.66
NeRF2Mesh (Tang et al., 2023b) | 34.25 | 25.04 | 30.08 | 35.70 | 34.90 | 26.26 | 32.63 | 29.47 | 30.88
Ours | 34.85 | 26.08 | 35.17 | 36.88 | 34.90 | 30.03 | 36.44 | 31.48 | 33.23
C-$\mathcal{L}_{1}$ | VolSDF (Yariv et al., 2021) | 1.18 | 4.03 | 3.01 | 3.22 | 2.26 | 1.57 | 1.13 | 6.42 | 2.86
NeuS (Wang et al., 2021) | 3.99 | 1.27 | 0.94 | 2.12 | 2.56 | 1.39 | 1.00 | 5.38 | 2.33
NeRO (Liu et al., 2023) | 1.27 | 1.97 | 1.22 | 1.88 | 1.90 | 1.33 | 0.87 | 4.95 | 1.92
BakedSDF (Yariv et al., 2023) | 1.83 | 1.43 | 1.09 | 1.68 | 1.13 | 1.36 | 0.84 | 3.88 | 1.66
NeRF2Mesh (Tang et al., 2023b) | 1.62 | 1.11 | 0.65 | 2.73 | 1.93 | 1.42 | 0.78 | 2.20 | 1.55
RelightableGaussian (Gao et al., 2023) | 3.65 | 2.34 | 1.26 | 3.11 | 1.63 | 1.35 | 1.76 | 3.35 | 2.31
Ours | 1.01 | 0.95 | 0.69 | 1.92 | 1.35 | 1.35 | 1.15 | 3.35 | 1.50
Table 1. We assess the quality of synthesized images and the accuracy of surface reconstruction, with each cell colored to indicate the best and second best. Our method is compared against various state-of-the-art (SOTA) approaches in tasks of novel view synthesis and surface reconstruction. It outperforms all competitors in both tasks, achieving the highest PSNR and the lowest Chamfer-$\mathcal{L}_{1}$ (C-$\mathcal{L}_{1}$) distance. Scan ID | 24 | 37 | 40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
COLMAP | 0.81 | 2.05 | 0.73 | 1.22 | 1.79 | 1.58 | 1.02 | 3.05 | 1.40 | 2.05 | 1.00 | 1.32 | 0.49 | 0.78 | 1.17 | 1.36
NeRF (Mildenhall et al., 2021) | 1.90 | 1.60 | 1.85 | 0.58 | 2.28 | 1.27 | 1.47 | 1.67 | 2.05 | 1.07 | 0.88 | 2.53 | 1.06 | 1.15 | 0.96 | 1.49
NeuS (Wang et al., 2021) | 1.00 | 1.37 | 0.93 | 0.43 | 1.10 | 0.65 | 0.57 | 1.48 | 1.09 | 0.83 | 0.52 | 1.20 | 0.35 | 0.49 | 0.54 | 0.84
VolSDF (Yariv et al., 2021) | 1.14 | 1.26 | 0.81 | 0.49 | 1.25 | 0.70 | 0.72 | 1.29 | 1.18 | 0.70 | 0.66 | 1.08 | 0.42 | 0.61 | 0.55 | 0.86
ReGS (Gao et al., 2023) | 1.21 | 1.00 | 1.10 | 0.87 | 1.03 | 1.51 | 1.51 | 1.06 | 1.63 | 0.97 | 1.36 | 1.21 | 0.94 | 1.38 | 1.26 | 1.20
Ours | 0.68 | 0.84 | 0.70 | 0.39 | 1.16 | 0.87 | 0.77 | 1.48 | 1.26 | 0.87 | 0.69 | 0.80 | 0.42 | 0.64 | 0.60 | 0.81
Table 2. Quantitative assessment on the DTU dataset with each cell colored to
indicate the best and second best. Our method achieves the highest average
quality of surface reconstruction and the lowest Chamfer-$\mathcal{L}_{1}$
distance with real-world data. Figure 5. Qualitative comparisons of
reconstructed results between NeRF2Mesh (Tang et al., 2023b), Relightable 3D
Gaussian (Gao et al., 2023), and our method on the NeRF Synthetic dataset. Our
method yields the highest-quality surface reconstruction preserving fine
details and intricate structures. Figure 6. Qualitative comparisons of
reconstructed results between VolSDF (Yariv et al., 2021), Relightable 3D
Gaussian (Gao et al., 2023), and our method on the DTU dataset. Our method
yields the highest-quality surface reconstruction preserving fine details and
intricate structures.

#### Implementation Details

Our work is implemented by PyTorch and CUDA and optimized by AdamW optimizer.
We bind the Gaussian points to the SDF field for the whole training process to
ensure that the Gaussian rendering process can provide a coarse SDF field,
that can boost the converging speed of volumetric rendering. To ensure stable
optimization, we introduce the warm-up strategy to make sure the dynamic
Gaussian points can move around to the surface and provide an accurate
supervised signal. The warm-up steps are set as 3000 for the Nerf synthetic
dataset and 5000 for the TNT dataset. During the volumetric rendering stage,
we sample $N=1024$ rays per step.

For the synthetic data, we initialize the implicit function as a sphere, and
the position of Gaussian is determined by the zero-level surface of the SDF
field. For the real-world scenes, which frequently exhibit intricate geometry
and noised pose, we adopt the SFM points as the initialized Gaussian position,
and the network is still initialized as a sphere. Furthermore, we also apply
the sparse MVS points to ensure the consistency between the rendered depth
$\mathcal{D}$ and projected depth map $D_{mvs}$ with $\mathcal{L}_{1}$ loss.
Our models are trained for 30000 iterations, and all the experiments are
conducted on a single NVIDIA GeForce RTX 3090 GPU.

###  5.2. Ablation Study

We conduct ablation studies on the NeRF Synthetic dataset in Table 3 to
evaluate the effectiveness of each module in our method. Six different
configurations are investigated to train our model.

#### Volumetric Constraints.

The volumetric constraints have a minimal impact on novel view synthesis
(PSNR: 32.83-33.32) yet significantly influence the reconstruction results
(Chamfer-L1: 3.63-1.51). Both 3DGS rasterization and ray-based volumetric
rendering rely on image supervision. However, the depth and normal
representations derived from these dual methods exhibit distinct
characteristics. While 3DGS excels at capturing high-frequency signals, it
often suffers from floaters. Conversely, ray-based volumetric rendering yields
smoother and more stable outcomes, but may struggle to capture fine details.

These two methods can complement each other and help address their respective
weaknesses. Redundant Gaussian points can be effectively eliminated, and
simultaneously, the high-frequency information learned by 3DGS can be employed
to enhance the quality of the SDF field. This is demonstrated in Table 1 and
Figure 3 of the main paper.

#### Learnable $\beta$.

By making $\beta$ trainable results, the conversion from SDF to 3DGS opacity
can be learned automatically, which results in a 0.20 and 0.44 improvement in
rendering and reconstruction performance, respectively.

#### Normal Supervision.

If we use the pseudo normal to supervise the normal of Gaussian points, the
model will get 0.23 and 0.05 improvement in Chamfer-$\mathcal{L}_{1}$ and
PSNR, respectively.

#### Eikonal Loss.

Removing the eikonal loss (Eq. (10)) will relax the constraint of SDF field,
which can damage the surface learning and lead to worse reconstruction
results. Note that better rendering quality (PSNR 33.4) is obtained in this
configuration, which indicates the relaxation of geometric constraints in
specific ways can aid in improving rendering quality.

#### Surface Alignment.

Constraining the Gaussian points close to the zero-level surface will inject
more high-frequency information into the SDF field leading to detailed surface
extracted results.

#### Reconstruction Strategy.

Compared with the marching cube with the SDF field, we also employ Poisson
reconstruction to extract the surface from Gaussians with normals derived from
the implicit SDF field, which exhibits a lower performance. This can be
attributed to the noisy, sparse distribution of Gaussians that significantly
impact the surface reconstruction quality. In contrast, the implicit SDF field
provides a continuous representation, allowing for easy extraction of high-
fidelity surfaces.

Method | Chamfer-$\mathcal{L}_{1}$ | F-score | PSNR | LPIPS | SSIM
---|---|---|---|---|---
w/o Geo | 3.63 | 80.20 | 32.83 | 0.0380 | 0.963
static $\beta$ | 1.95 | 90.20 | 33.02 | 0.0385 | 0.966
w/o eq (9) | 1.74 | 91.36 | 33.18 | 0.0383 | 0.967
w/o eq (10) | 2.00 | 87.89 | 33.40 | 0.0350 | 0.968
w/o eq (11) | 1.69 | 91.70 | 33.06 | 0.0388 | 0.966
Poisson | 2.03 | 87.86 | - | - | -
_Ours (full)_ | 1.51 | 93.50 | 33.23 | 0.0340 | 0.968
Table 3. Ablation study on the NeRF Synthetic dataset, where we evaluate the
effect of each proposed component.

###  5.3. Experimental Results

Tables 1 and 2 present the quantitative results for synthetic and real-world
datasets, respectively. On synthetic data, Table 1 demonstrates that our
method surpasses other state-of-the-art methods in terms of the reconstruction
quality, as measured by Chamfer-$\mathcal{L}_{1}$ distance. Furthermore, our
method also excels in synthesizing high-quality novel view images, which are
comparable to state-of-the-art novel view synthesis methods. This highlights
our method’s superiority over surface reconstruction techniques and its
competitive performance in novel view synthesis tasks. On real-captured DTU
datasets, as shown in Table 2, our method also exhibits the best performance
in terms of average surface reconstruction quality.

Besides, we also found that better reconstruction results have less
contribution to the rendering quality compared to the vanilla 3D-GS. The
slightly lower performance is because 3DGS is optimized solely for rendering
images; while our geometry constraints reduce the freedom of the optimization
of 3DGS for the rendering tasks, the best solution for rendering may not
necessarily be for geometry, as evidenced by performance in Table 1 (Trade-off
between rendering quality and surface reconstruction). But overall, our
formulation achieves the best trade-off between rendering quality and
reconstruction quality.

Additionally, we provide qualitative comparisons of the reconstruction results
qualitatively and present them in Figure 5. The state-of-the-art NeRF-based 3D
reconstruction method NeRF2Mesh (Tang et al., 2023b) fails to preserve fine
details like fine Mic textures, Lego tracks, and stripes on the Hotdog disk.
RelightableGaussian (Gao et al., 2023) incorporates shading equations to
determine Gaussian appearance and trains 3D Gaussians with attributes of
normal and BRDF. 3D surfaces can then be obtained by applying Screened Poisson
Reconstruction (Kazhdan and Hoppe, 2013) on the 3D Gaussians assigned with
normal. However, Gaussians are naturally discontinuous and unstructured,
resulting in broken parts on the Mic and floaters on the Hotdog disk. Due to
the absence of smooth surface modeling, its reconstructed results on Lego
tracks, which contain intricate structures, are noisy and inaccurate.
Combining the fitting power of Gaussian Splatting and the smooth surface
modeling of the implicit SDF field, our method reconstructs surfaces with
geometric details accurately and smoothly.

##  6\. Conclusion

In this paper, we introduce 3DGSR, an efficient method for high-quality
surface reconstruction using 3DGS. Our approach is based on two key
components: 1) the integration of neural implicit SDF and its binding with 3D
Gaussians through a differentiable SDF-to-opacity transformation function, and
2) the utilization of volumetric rendering and the SDF-and-Gaussian geometry
consistency regularization for SDF optimization. The first component allows
for the joint optimization of SDF and 3D Gaussians, with the optimization of
Gaussians providing supervision signals for learning the SDF. The second
component supplies additional supervision signals for refining the SDF in
areas not occupied by Gaussians, using a consistency loss that aligns depth
(normal) from SDF with that derived from Gaussians. Our extensive experiments
showcase the effectiveness of 3DGSR in reconstructing high-quality surfaces
outperforming those obtained from state-of-the-art reconstruction pipelines,
without compromising the rendering capabilities and efficiency of 3D
Gaussians. Our study does have certain limitations caused by the trade-off
between rendering quality and surface smoothness. In order to render high-
quality images, the smoothness of the surface may be compromised in some cases
involving complex texture patterns.

## References

  * (1)
  * Aliev et al. (2020) Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020.  Neural point-based graphics. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_. Springer, 696–712.
  * Barnes et al. (2009) Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009.  PatchMatch: A randomized correspondence algorithm for structural image editing.  _ACM Trans. Graph._ 28, 3 (2009), 24.
  * Barron et al. (2021) Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. 2021.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5855–5864.
  * Barron et al. (2023) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2023.  Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields.  _ICCV_ (2023).
  * Botsch et al. (2005) Mario Botsch, Alexander Hornung, Matthias Zwicker, and Leif Kobbelt. 2005.  High-quality surface splatting on today’s GPUs. In _Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, 2005._ IEEE, 17–141.
  * Broadhurst et al. (2001) Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. 2001.  A probabilistic framework for space carving. In _Proceedings eighth IEEE international conference on computer vision. ICCV 2001_ , Vol. 1. IEEE, 388–393.
  * Chan et al. (2022) Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. 2022\.  Efficient geometry-aware 3D generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16123–16133.
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  Tensorf: Tensorial radiance fields. In _European Conference on Computer Vision_. Springer, 333–350.
  * Chen et al. (2023a) Hanlin Chen, Chen Li, and Gim Hee Lee. 2023a.  NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance.  _arXiv preprint arXiv:2312.00846_ (2023).
  * Chen et al. (2023b) Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. 2023b.  Neurbf: A neural fields representation with adaptive radial basis functions. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4182–4194.
  * Dai et al. (2023) Peng Dai, Yinda Zhang, Xin Yu, Xiaoyang Lyu, and Xiaojuan Qi. 2023.  Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 154–164.
  * Darmon et al. (2022) François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. 2022.  Improving neural implicit surfaces geometry with patch warping. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6260–6269.
  * Das et al. (2023) Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy Ilg, and Jan Eric Lenssen. 2023.  Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction.  _arXiv preprint arXiv:2312.01196_ (2023).
  * De Bonet and Viola (1999) Jeremy S De Bonet and Paul Viola. 1999.  Poxels: Probabilistic voxelized volume reconstruction. In _Proceedings of International Conference on Computer Vision (ICCV)_ , Vol. 2. 2.
  * Fridovich-Keil et al. (2022) Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022.  Plenoxels: Radiance fields without neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5501–5510.
  * Galliani et al. (2016) Silvano Galliani, Katrin Lasinger, and Konrad Schindler. 2016.  Gipuma: Massively parallel multi-view stereo reconstruction.  _Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation e. V_ 25, 361-369 (2016), 2.
  * Gao et al. (2023) Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. 2023.  Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing.  _arXiv preprint arXiv:2311.16043_ (2023).
  * Gropp et al. (2020) Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020.  Implicit Geometric Regularization for Learning Shapes.  In _Proceedings of Machine Learning and Systems 2020_. 3569–3579.
  * Gross and Pfister (2011) Markus Gross and Hanspeter Pfister. 2011.  _Point-based graphics_.  Elsevier.
  * Grossman and Dally (1998) Jeffrey P Grossman and William J Dally. 1998.  Point sample rendering. In _Rendering Techniques’ 98: Proceedings of the Eurographics Workshop in Vienna, Austria, June 29—July 1, 1998 9_. Springer, 181–192.
  * Guédon and Lepetit (2023) Antoine Guédon and Vincent Lepetit. 2023.  SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering.  _arXiv preprint arXiv:2311.12775_ (2023).
  * Hu et al. (2023) Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. 2023.  Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 19774–19783.
  * Huang et al. (2023) Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. 2023.  SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes.  _arXiv preprint arXiv:2312.14937_ (2023).
  * Jensen et al. (2014) Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. 2014.  Large scale multi-view stereopsis evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 406–413.
  * Jiang et al. (2020) Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. 2020.  Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 1251–1261.
  * Jung et al. (2023) HyunJun Jung, Nikolas Brasch, Jifei Song, Eduardo Perez-Pellitero, Yiren Zhou, Zhihao Li, Nassir Navab, and Benjamin Busam. 2023.  Deformable 3D Gaussian Splatting for Animatable Human Avatars.  _arXiv preprint arXiv:2312.15059_ (2023).
  * Karnewar et al. (2022) Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022.  Relu fields: The little non-linearity that could. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Kazhdan and Hoppe (2013) Michael Kazhdan and Hugues Hoppe. 2013.  Screened poisson surface reconstruction.  _ACM Transactions on Graphics (ToG)_ 32, 3 (2013), 1–13.
  * Kellnhofer et al. (2021) Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. 2021.  Neural lumigraph rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4287–4297.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (July 2023).  https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * Knapitsch et al. (2017) Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017.  Tanks and temples: Benchmarking large-scale scene reconstruction.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1–13.
  * Li et al. (2023a) Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2023a.  Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis.  _arXiv preprint arXiv:2312.16812_ (2023).
  * Li et al. (2023b) Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. 2023b.  Neuralangelo: High-Fidelity Neural Surface Reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8456–8465.
  * Liu et al. (2020a) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020a.  Neural sparse voxel fields.  _Advances in Neural Information Processing Systems_ 33 (2020), 15651–15663.
  * Liu et al. (2020b) Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. 2020b.  Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2019–2028.
  * Liu et al. (2023) Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. 2023.  NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images.
  * Lombardi et al. (2019) Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019.  Neural volumes: learning dynamic renderable volumes from images.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–14.
  * Luiten et al. (2023) Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2023.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  _arXiv preprint arXiv:2308.09713_ (2023).
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–15.
  * Niemeyer et al. (2020) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3504–3515.
  * Pfister et al. (2000) Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. 2000.  Surfels: Surface elements as rendering primitives. In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_. 335–342.
  * Pittaluga et al. (2019) Francesco Pittaluga, Sanjeev J Koppal, Sing Bing Kang, and Sudipta N Sinha. 2019.  Revealing scenes by inverting structure from motion reconstructions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 145–154.
  * Rakhimov et al. (2022) Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, and Evgeny Burnaev. 2022.  Npbg++: Accelerating neural point-based graphics. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15969–15979.
  * Ren et al. (2023) Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. 2023.  DreamGaussian4D: Generative 4D Gaussian Splatting.  _arXiv preprint arXiv:2312.17142_ (2023).
  * Ren et al. (2002) Liu Ren, Hanspeter Pfister, and Matthias Zwicker. 2002.  Object space EWA surface splatting: A hardware accelerated approach to high quality point rendering. In _Computer Graphics Forum_ , Vol. 21. Wiley Online Library, 461–470.
  * Schönberger and Frahm (2016) Johannes Lutz Schönberger and Jan-Michael Frahm. 2016.  Structure-from-Motion Revisited. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schönberger et al. (2016) Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016.  Pixelwise view selection for unstructured multi-view stereo. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14_. Springer, 501–518.
  * Seitz and Dyer (1999) Steven M Seitz and Charles R Dyer. 1999.  Photorealistic scene reconstruction by voxel coloring.  _International Journal of Computer Vision_ 35 (1999), 151–173.
  * Sitzmann et al. (2019) Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Stereopsis (2010) Robust Multiview Stereopsis. 2010.  Accurate, Dense, and Robust Multiview Stereopsis.  _IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE_ 32, 8 (2010).
  * Sun et al. (2022) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5459–5469.
  * Tang et al. (2023a) Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. 2023a.  DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation.  _arXiv preprint arXiv:2309.16653_ (2023).
  * Tang et al. (2023b) Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. 2023b.  Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement. In _International Conference on Computer Vision (ICCV)_.
  * Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ (2021).
  * Wang et al. (2023a) Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. 2023a.  Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3295–3306.
  * Wang et al. (2023b) Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. 2023b.  PET-NeuS: Positional Encoding Triplanes for Neural Surfaces.  (2023).
  * Xu et al. (2022) Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. 2022.  Point-nerf: Point-based neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5438–5448.
  * Yang et al. (2023a) Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang, Yi Xu, Xulei Yang, Fayao Liu, and Guosheng Lin. 2023a.  Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting.  _arXiv preprint arXiv:2312.04820_ (2023).
  * Yang et al. (2023b) Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. 2023b.  Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction.  _arXiv preprint arXiv:2309.13101_ (2023).
  * Yang et al. (2024) Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. 2024.  Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting. In _International Conference on Learning Representations (ICLR)_.
  * Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ 34 (2021), 4805–4815.
  * Yariv et al. (2023) Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. 2023.  BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis.  _arXiv_ (2023).
  * Yariv et al. (2020) Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020.  Multiview neural surface reconstruction by disentangling geometry and appearance.  _Advances in Neural Information Processing Systems_ 33 (2020), 2492–2502.
  * Ye et al. (2023) Keyang Ye, Tianjia Shao, and Kun Zhou. 2023.  Animatable 3d gaussians for high-fidelity synthesis of human motions.  _arXiv preprint arXiv:2311.13404_ (2023).
  * Yi et al. (2023) Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. 2023.  Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors.  _arXiv preprint arXiv:2310.08529_ (2023).
  * Yifan et al. (2019) Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–14.
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021.  Plenoctrees for real-time rendering of neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5752–5761.
  * Zhang et al. (2022) Qiang Zhang, Baek Seung-Hwan, Rusinkiewicz Szymon, and Felix Heide. 2022.  Differentiable point-based radiance fields for efficient view synthesis. In _ACM SIGGRAPH Asia 2022 Conference Proceedings_. 1–12.
  * Zielonka et al. (2023) Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. 2023.  Drivable 3d gaussian avatars.  _arXiv preprint arXiv:2311.08581_ (2023).
  * Zwicker et al. (2001) M. Zwicker, H. Pfister, J. van Baar, and M. Gross. 2001.  Surface Splatting. In _ACM Transactions on Graphics (Proc. ACM SIGGRAPH)_. 371–378.