# Interplay of Mechanochemistry and Material Processes in the Graphite to
Diamond Phase Transformation

Brenden W. Hamilton<EMAIL_ADDRESS>Timothy C. Germann  Theoretical
Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA

###### Abstract

The manifestation of intra-molecular strains in covalent systems is widely
known to accelerate chemical reactions and open alternative reaction paths.
This process is moderately understood for isolated molecules and uni-molecular
processes. However, in condensed matter processes such as phase
transformations, material properties and structure may influence typical
mechanochemical effects. Therefore, we utilize steered molecular dynamics to
induce out of plane strains in graphite and compress the system under a
constant strain rate to induce phase transformation. We show that the out of
plane strain allows for phase transformations to initiate at lower amounts of
compressive strain. Yet, in contrast to typical mechanochemical results, the
sum of compressive and out of plane work needed to form diamond has a local
minimum due to altered defect formation processes during phase transformation.
Additionally, these altered processes slow the kinetics of the phase
transformation, taking longer from initiation to total material
transformation.

Mechanochemistry is known to accelerate chemical reactions and alter the
reaction path in covalent systems [1, 2, 3, 4]. By adding additional strain to
chemical bonds, the activation barrier can be directly lowered [5], typically
resulting in faster reaction kinetics. Mechanochemistry can occur in cases in
which a mechanical strain drives a bond to yield and induces chemistry [6, 7],
as well as mechanochemical assistance on a thermally induced process of
interest [8, 9, 4, 10, 11, 12].

While the majority of mechanochemistry research has focused on the linear
strains induced by pulling on covalent bonds [13, 7, 14, 15, 16, 17, 18, 19,
20, 21], it has recently been shown that the dynamics of condensed phase,
molecular materials under complex strain states can induce ’many-body’
deformations: the twisting and bending of molecules [8, 22]. These many-body
deformations can induce significant acceleration of reactions via lowered
energy barriers and cause alternate reaction pathways to become the main
reaction path [23, 24, 25, 26]. In the pulling of mechanophores to induce
isomerization, adding a rotational strain around the central axis of the
molecule was shown to lower the total energy needed to cause reaction. This
monotonic decrease in the _total work_ needed, the pulling work done and the
energy needed to induce the rotational strain, results in a more efficient
reaction path than pure pulling forces [23].

Layered materials, such as graphite, are known to have layer buckling
instabilities under pressure, which can induce local, out of plane strains
[27, 28, 29, 30]. Specifically in graphite, these local deformations can close
the HOMO-LUMO gap to potentially assist in reactivity and phase transformation
[31]. Graphitic defects such as ripplocations[32, 33, 34, 35], kinking[36, 37,
38], and Wigner defects[39, 40] can all result in a wide range of out of plane
strain states of a layer. Carbon phase transformations and diamond formation
have significance for a variety of scientific topics such as detonation
products [41, 42], planetary interiors [43, 44], nuclear fusion [45], and
synthetic materials fabrication [46]. Rapid compression and shear stress
formation have been commonly proposed as a viable route for the synthetic
formation of diamond [47, 48, 49]. The open questions are: (a) how does local
out of plane strain influence the graphite to diamond phase transformation,
beyond typical pressure effects; and (b) can it be utilized to tailor
nanoscale diamond formation, better controlling other carbon allotropes? More
broadly, previous many-body mechanochemistry studies have looked at isolated
molecules and uni-molecular decomposition reactions. In the case of graphite
to diamond phase transformations, significantly more condensed matter
processes are in play, such as defect formation and non-hydrostatic stress
states which will more directly influence the transformation kinetics.

In this work, we utilized Many-Boded Steered Molecular Dynamics (MBsMD) [23]
to induce out of plane strain on graphite layers, followed by a uniaxial
compression to induce phase transformation. All molecular dynamics (MD)
simulations were conducted using the LAMMPS software package [50, 51].
Interatomic forces were calculated with the ReaxFF potential [52]. All
simulations were conducted under NPT conditions with fully periodic boundary
conditions and a 0.1 fs timestep. The temperature was set to 300 K using a
Nose-Hoover thermostat [53] and pressure conditions were set to 1.0 atm. The X
and Y pressure tensor components are coupled, and all shear stresses (box
tilts) are independent of each other. Uniaxial deformations are applied in the
Cartesian Z direction with a constant engineering strain rate. Deformation is
performed at every timestep and atomic coordinates are fractionally remapped.

Figure 1: Initial simulation set up. Left: top-down view, Z axis into page, of
graphite layers with improper centers colored red. Improper centers in the
adjacent layer are shifted by one carbon ring. Right: side-on view of an out
of plane strained system with the set angle as 30∘.

The initial atomic configuration is a hexagonal graphite cell of 12960 atoms
and (300 K, 1 atm) relaxed box dimensions of X = 4.56918 nm, Y = 3.95348 nm,
and Z = 6.58632 nm, with the non-zero box tilt of XY = -2.28255 nm. MBsMD is
used to apply an external field to drive 4-body systems (either proper or
improper dihedrals) to be at specific angles, causing twisting and bending
deformations within molecules [23]. Here, improper dihedrals within the
graphite layers are used to induce out of plane strain. Figure 1 colors the
improper dihedral center atoms red for one graphite layer. Improper angles,
which are 0∘ in the relaxed crystal, are driven to angles between 0∘ and 50∘,
with independent runs for a chosen angle. Figure 1 shows a side on view of
layer impropers driven to 30∘. Before compressive deformation, the cells are
equilibrated with the external field on for 50 ps at 300 K and 1 atm. For
ensembled samples, an additional 20 ps was run, saving a configuration every 5
ps.

In this work, the MBsMD method is extended such that the external field is
only applied to a specific improper if its bonding environment is in its
initial state. Hence, the chosen angle does not directly deform or alter the
diamond phase regions. Additional details are provided in Supplemental
Material section SM-1.

The phase of the material is determined using the polyhedral template matching
as employed in LAMMPS [54]. As shown in Figure 2, the strains (or times) of
initiation and completion of transformation from graphite are defined as the
first step in which the graphite population fails below 80% and 20%,
respectively. The transformed phase populations are taken for all systems as
the state at 50% strain. The compressive work is calculated from the internal
energy rise from the first step of compression to the peak energy state prior
to phase transformation. The total work is calculated using the same energy
peak, but referenced from the initial state with no external field system,
i.e., the combined work done from compression and to initially deform the
layers.

Figure 2: Example of phases present during compression for a case with out of
plane strain at 30∘ and a strain rate of $5\text{\times}{10}^{-9}$/s. Black
points are graphite, blue for cubic diamond, orange for hexagonal diamond.
Circle at 40% strain and 0.8 population represents the point of phase
transformation used in Figure 3. The vertical line at 50% strain indicates the
point in which phase amounts are taken for Figure 4.

For each initial out of plane strain, 5 samples are compressed with a constant
engineering strain rate of $5\text{\times}{10}^{-9}$/s. Figure 3 shows the
strain at which transformation initiates (blue) and finishes (red). The
inclusion of an out of plane strain directly lowers the compressive strain
necessary to begin to form a diamond phase, with out of plane improper angles
of 50∘ inducing the phase transformation at $\approx$10% strain lower than at
0∘ out of plane strain. While the average points show a monotonic decrease,
there is still considerable noise from sample to sample, as expected from
previous works [31]. Figure 2, which shows the phases present for 30∘ out of
plane strain, shows a rapid transformation from 100% graphite to $\approx$35%
graphite, followed by a much slower completion of the transformation to a
fully diamond state (mixture of cubic and hexagonal). At higher out of plane
strains, small regions of a locally disordered lattice form as meta-stable
regions that often become diamond after further compression, leading to more
compression/time overall to complete the phase transformation to a (mostly)
diamond system. This results in the divergence of the 80% graphite strain
(blue) and 20% graphite strain (red) curves, which represents a slowing of the
phase transformation kinetics. While the increased out of plane strain lowers
the activation barrier for phase transformation, it also results in additional
materials processes such as defect formation that slows the overall kinetics,
especially as the systems reach lower amounts of graphite remaining. We devote
the remainder of this paper to understanding the processes underlying the
slower reaction kinetics.

Figure 3: Level of compressive strain needed to induce (blue, 80% graphite)
and complete (red, 20% graphite) phase transformation for out of plane strains
studied. Points are averaged over five independent runs with error bars as
standard deviation.

To better assess the overall process of diamond formation under an induced out
of plane strain, Figure 4 shows the average work done on the system to induce
phase transformation for each out of plane strain, where the total work
includes both the compressive work as well as the work done to create the out
of plane strain states. For no out of plane strain, these values are the same,
and a monotonic decrease is expected for increasing strain, as was shown for
the activation barriers of decomposition reactions [23]. The increasing
divergence of the two is also expected as more work is done to reach higher
levels of out of plane strain. The total work shows an uptick for out of plane
strains from 20∘ to 40∘, which is accompanied by a greatly reduced slope for
the compressive work and continual increasing out of plane strain work,
resulting in a local minimum at 20∘. While the error bars show that this
uptick is not necessarily statistically significant, this trend of a slowed
decrease in the work necessary may physically arise from the onset of
structural damage. This is consistent with the slower transformation kinetics
for the last 30-35% of graphite transformation observed for higher out of
plane strains (Figs. 2 and 3), since additional compressive work is needed to
complete the transformation. The 20∘ represents a local minimum where
significant local disorder has yet to significantly form, leading to minimal
slowing of kinetics and no effect on energetics, letting the system benefit
from a large mechanochemical effect from the out of plane strain, without it
impeding the transformation. The Supplemental Material section SM-2 shows a
graphite population history for each out of plane strain, as well as mid-
transformation images of atomic positions, from snapshots as close to the
onset of local disorder as possible (between 30% and 50% graphite population)
when relevant.

These results differ from previous studies of the isomerization of
mechanophores, in which adding a rotational strain prior to pulling on the
molecule to induce reaction consistently lowered the total work needed for
reaction [23]. This helps to showcase the complex nature of mechanochemistry
in condensed matter systems (compared to isolated molecules and idealized
systems) in which deviatoric stresses, local phase and structure, defect
formation, and the existence of multiple meta-stable states all play a role in
how the reactivity of the system responds. In this case, too much
mechanochemical influence ends up altering other processes after the onset of
phase transformation that partially negates the initial acceleration.

Figure 4: Work done on the system prior to phase transformation. Compressive
work (blue) is from just uniaxial deformation. Total work (red) includes
compressive work and out of plane strain work. Points are averaged over five
independent runs Error bars represent the standard deviation from five
independent runs.

Finally, Figure 5 shows the phase populations at 50% strain, in which graphite
has typically transformed to a majority of some phase of diamond, either cubic
or hexagonal. The hexagonal diamond formed in the low out of plane strain
cases is from the formation of pure twin defects. With increasing out of plane
strain, the amount of hexagonal diamond, as well as the uncertainty, in phase
amounts, increases. This is due to both the presence of twin defects as well
as local disorder and dislocations which can manifest as hexagonal diamond
under continued compression that results in crystallization. Supplemental
Material section SM-3 shows these same population plots for 45% and 55%
compressive strain, in which the same trends arise where 20∘ out of plane
strain is the local maximum of cubic diamond and uncertainty greatly increase
beyond this point. This helps to corroborate the idea that the local minimum
in work done is a result of material influences on kinetics coupled with the
mechanochemical effect and not an artifact of an arbitrary choice in total
strain to assess populations, which are moderately stable from transformation
until a compression induced material failure. The local minimum in total work
at 20∘ may also arise from the lack of pure twin formation at this level of
out of plane strain due to local energetics and/or available transition
states.

Figure 5: Percentage amounts of each phase present at 50% compressive strain.
Points are averaged over five independent runs, and error bars represent the
standard deviation over those runs. Black points are graphite, blue for cubic
diamond, orange for hexagonal diamond.

In summary, we assess how mechanochemical perturbations of the graphite
structure influence the compression induced transition to diamond. By using
steered MD to induce out of plane strains, systems with more out of plane
strain require less total compression to form diamond, in terms of total
strain, but the overall kinetics of total transformation are considerably
slower. Assessing the total work needed to form diamond, the sum of the
compressive work and the work to induce out of plane strain, a local minimum
arises at 20∘ out of plane strain. As out of plane strain is induced, it
lowers the barrier to begin the formation of diamond, however, at higher
levels of strain, defects and local disorder form during phase transformation,
resulting in more compressive work (relative to the work to start
transformation) being needed to complete the phase transformation, causing
slower kinetics. This directly differs from previous mechanochemical studies
in which increased mechanochemcial work decreases the amount of total work
needed to induce reaction [23, 4]. At the 20∘ local minimum, almost no pure
twin defects form, and at higher levels of out of plane strain, the
uncertainty in twin and hexagonal diamond formation increases significantly as
a result of the local disorder induced by the out of plane strain. These
results stem from the mechanochemical decrease in activation coupling with the
local strains’ effect on other material processes slowing the transformation
kinetics. This opens a number of questions into how the various complications
of condensed matter chemistry and phase transformations can influence the role
of mechanochemistry by altering downstream and related processes in materials.

###### Acknowledgements.

Funding for this project was provided by the Director’s Postdoctoral
Fellowship program, project LDRD 20220705PRD1. Partial funding was provided by
the Advanced Simulation and Computing Physics and Engineering Models project
(ASC-PEM). This research used resources provided by the Los Alamos National
Laboratory Institutional Computing Program. This work was supported by the
U.S. Department of Energy (DOE) through the Los Alamos National Laboratory.
The Los Alamos National Laboratory is operated by Triad National Security,
LLC, for the National Nuclear Security Administration of the U.S. Department
of Energy (Contract No. 89233218CNA000001). Approved for unlimited release:
LA-UR-23-21258

## References

  * Ribas-Arino and Marx [2012] J. Ribas-Arino and D. Marx, Covalent mechanochemistry: Theoretical concepts and computational tools with applications to molecular nanomechanics, Chemical Reviews 112, 5412 (2012).
  * Takacs [2013] L. Takacs, The historical development of mechanochemistry, Chemical Society Reviews 42, 7649 (2013).
  * Stauch and Dreuw [2016] T. Stauch and A. Dreuw, Advances in Quantum Mechanochemistry: Electronic Structure Methods and Force Analysis, Chemical Reviews 116, 14137 (2016).
  * Hamilton  _et al._ [2022a] B. W. Hamilton, M. P. Kroonblawd, and A. Strachan, Extemporaneous Mechanochemistry: Shockwave Induced Ultrafast Chemical Reactions Due to Intramolecular Strain Energy, Journal of Physical Chemistry Letters 13, 6657 (2022a).
  * Bell [1978] G. I. Bell, Models for the Specific Adhesion of Cells to Cells, Science 200, 618 (1978).
  * Grandbois  _et al._ [1999] M. Grandbois, M. Beyer, M. Rief, H. Clausen-Schaumann, and H. E. Gaub, How strong is a covalent bond?, Science 283, 1727 (1999).
  * Davis  _et al._ [2009] D. A. Davis, A. Hamilton, J. Yang, L. D. Cremar, D. Van Gough, S. L. Potisek, M. T. Ong, P. V. Braun, T. J. Martínez, S. R. White, J. S. Moore, and N. R. Sottos, Force-induced activation of covalent bonds in mechanoresponsive polymeric materials, Nature 459, 68 (2009).
  * Hamilton  _et al._ [2021a] B. W. Hamilton, M. P. Kroonblawd, C. Li, and A. Strachan, A Hotspot’s Better Half: Non-Equilibrium Intra-Molecular Strain in Shock Physics, Journal of Physical Chemistry Letters 12, 2756 (2021a).
  * Wood  _et al._ [2015] M. A. Wood, M. J. Cherukara, E. M. Kober, and A. Strachan, Ultrafast Chemistry under Nonequilibrium Conditions and the Shock to Deflagration Transition at the Nanoscale, Journal of Physical Chemistry C 119, 22008 (2015).
  * Hamilton  _et al._ [2022b] B. W. Hamilton, M. P. Kroonblawd, and A. Strachan, The Potential Energy Hotspot: Effects from Impact Velocity, Defect Geometry, and Crystallographic Orientation, Journal of Physical Chemistry C 126, 3743 (2022b).
  * Hamilton  _et al._ [2021b] B. W. Hamilton, M. N. Sakano, C. Li, and A. Strachan, Chemistry Under Shock Conditions, Annual Review of Materials Research 51, 101 (2021b).
  * Hamilton and Germann [2023] B. W. Hamilton and T. C. Germann, Energy localization efficiency in 1, 3, 5-trinitro-2, 4, 6-triaminobenzene pore collapse mechanisms, Journal of Applied Physics 133, 035901 (2023).
  * Ribas-Arino  _et al._ [2009] J. Ribas-Arino, M. Shiga, and D. Marx, Understanding covalent mechanochemistry, Angewandte Chemie - International Edition 48, 4190 (2009).
  * Ong  _et al._ [2009] M. T. Ong, J. Leiding, H. Tao, A. M. Virshup, and T. J. Martínez, First principles dynamics and minimum energy pathways for mechanochemical ring opening of cyclobutene, Journal of the American Chemical Society 131, 6377 (2009).
  * Ghanem  _et al._ [2021] M. A. Ghanem, A. Basu, R. Behrou, N. Boechler, A. J. Boydston, S. L. Craig, Y. Lin, B. E. Lynde, A. Nelson, H. Shen, and D. W. Storti, The role of polymer mechanochemistry in responsive materials and additive manufacturing, Nature Reviews Materials 6, 84 (2021).
  * Wang  _et al._ [2015] J. Wang, T. B. Kouznetsova, Z. Niu, M. T. Ong, H. M. Klukovich, A. L. Rheingold, T. J. Martinez, and S. L. Craig, Inducing and quantifying forbidden reactivity with single-molecule polymer mechanochemistry, Nature Chemistry 7, 323 (2015).
  * Deneke  _et al._ [2020] N. Deneke, M. L. Rencheck, and C. S. Davis, An engineer’s introduction to mechanophores, Soft Matter 16, 6230 (2020).
  * Stauch and Dreuw [2014] T. Stauch and A. Dreuw, A quantitative quantum-chemical analysis tool for the distribution of mechanical force in molecules, Journal of Chemical Physics 140, 10.1063/1.4870334 (2014).
  * Wiita  _et al._ [2006] A. P. Wiita, S. R. K. Ainavarapu, H. H. Huang, and J. M. Fernandez, Force-dependent chemical kinetics of disulfide bond reduction observed with single-molecule techniques, Proceedings of the National Academy of Sciences of the United States of America 103, 7222 (2006).
  * Dopieralski  _et al._ [2013] P. Dopieralski, J. Ribas-Arino, P. Anjukandi, M. Krupicka, J. Kiss, and D. Marx, The Janus-faced role of external forces in mechanochemical disulfide bond cleavage, Nature Chemistry 5, 685 (2013).
  * Piermattei  _et al._ [2009] A. Piermattei, S. Karthikeyan, and R. P. Sijbesma, Activating catalysts with mechanical force, Nature Chemistry 1, 133 (2009).
  * Hamilton and Strachan [2022a] B. W. Hamilton and A. Strachan, Rapid activation of non-oriented mechanophores via shock loading and spallation, arXiv preprint arXiv:2211.16258 (2022a).
  * Hamilton and Strachan [2022b] B. W. Hamilton and A. Strachan, Many-Body Mechanochemistry : Intra-molecular Strain in Condensed Matter Chemistry, chemRxiv (2022b).
  * Kroonblawd and Fried [2020] M. P. Kroonblawd and L. E. Fried, High Explosive Ignition through Chemically Activated Nanoscale Shear Bands, Physical Review Letters 124, 206002 (2020).
  * Larsen and Boydston [2013] M. B. Larsen and A. J. Boydston, ”Flex-activated” mechanophores: Using polymer mechanochemistry to direct bond bending activation, Journal of the American Chemical Society 135, 8189 (2013).
  * Klukovich  _et al._ [2013] H. M. Klukovich, T. B. Kouznetsova, Z. S. Kean, J. M. Lenhardt, and S. L. Craig, A backbone lever-arm effect enhances polymer mechanochemistry, Nature Chemistry 5, 110 (2013).
  * Lafourcade  _et al._ [2018] P. Lafourcade, C. Denoual, and J. B. Maillet, Irreversible Deformation Mechanisms for 1,3,5-Triamino-2,4,6-Trinitrobenzene Single Crystal through Molecular Dynamics Simulations, Journal of Physical Chemistry C 122, 14954 (2018).
  * Lafourcade  _et al._ [2020] P. Lafourcade, C. Denoual, and J.-B. Maillet, Elastic instability in graphite single crystal under dynamic triaxial compression: Effect of strain-rate on the resulting microstructure, Journal of Applied Physics 128, 045101 (2020).
  * Plummer  _et al._ [2022] G. Plummer, M. W. Barsoum, C. R. Weinberger, and G. J. Tucker, Basal dislocations in max phases: Core structure and mobility, Materialia 21, 101310 (2022).
  * Barsoum and Radovic [2011] M. W. Barsoum and M. Radovic, Elastic and mechanical properties of the max phases, Annual review of materials research 41, 195 (2011).
  * Kroonblawd and Goldman [2018] M. P. Kroonblawd and N. Goldman, Mechanochemical formation of heterogeneous diamond structures during rapid uniaxial compression in graphite, Physical Review B 97, 1 (2018).
  * Badr and Barsoum [2022] H. O. Badr and M. W. Barsoum, On ripplocations and the deformation of graphite, Carbon (2022).
  * Barsoum and Tucker [2017] M. Barsoum and G. Tucker, Deformation of layered solids: ripplocations not basal dislocations, Scripta Materialia 139, 166 (2017).
  * Barsoum  _et al._ [2019] M. Barsoum, X. Zhao, S. Shanazarov, A. Romanchuk, S. Koumlis, S. Pagano, L. Lamberson, and G. Tucker, Ripplocations: A universal deformation mechanism in layered solids, Physical Review Materials 3, 013602 (2019).
  * Freiberg  _et al._ [2018] D. Freiberg, M. Barsoum, and G. Tucker, Nucleation of ripplocations through atomistic modeling of surface nanoindentation in graphite, Physical Review Materials 2, 053602 (2018).
  * Barsoum  _et al._ [2004] M. Barsoum, A. Murugaiah, S. Kalidindi, T. Zhen, and Y. Gogotsi, Kink bands, nonlinear elasticity and nanoindentations in graphite, Carbon 42, 1435 (2004).
  * Plummer  _et al._ [2021] G. Plummer, H. Rathod, A. Srivastava, M. Radovic, T. Ouisse, M. Yildizhan, P. Å. Persson, K. Lambrinou, M. Barsoum, and G. Tucker, On the origin of kinking in layered crystalline solids, Materials Today 43, 45 (2021).
  * Basu  _et al._ [2009] S. Basu, A. Zhou, and M. Barsoum, On spherical nanoindentations, kinking nonlinear elasticity of mica single crystals and their geological implications, Journal of Structural Geology 31, 791 (2009).
  * Telling  _et al._ [2003] R. H. Telling, C. P. Ewels, A. A. El-Barbary, and M. I. Heggie, Wigner defects bridge the graphite gap, Nature materials 2, 333 (2003).
  * Ewels  _et al._ [2003] C. Ewels, R. Telling, A. El-Barbary, M. Heggie, and P. Briddon, Metastable frenkel pair defect in graphite: source of wigner energy?, Physical review letters 91, 025505 (2003).
  * Hammons  _et al._ [2021] J. A. Hammons, M. H. Nielsen, M. Bagge-Hansen, S. Bastea, C. May, W. L. Shaw, A. Martin, Y. Li, N. Sinclair, L. M. Lauderbach, _et al._ , Submicrosecond aggregation during detonation synthesis of nanodiamond, The Journal of Physical Chemistry Letters 12, 5286 (2021).
  * Armstrong  _et al._ [2020] M. R. Armstrong, R. K. Lindsey, N. Goldman, M. H. Nielsen, E. Stavrou, L. E. Fried, J. M. Zaug, and S. Bastea, Ultrafast shock synthesis of nanocarbon from a liquid precursor, Nature Communications 11, 1 (2020).
  * Hubbard [1981] W. Hubbard, Interiors of the giant planets, Science 214, 145 (1981).
  * Kraus  _et al._ [2017] D. Kraus, J. Vorberger, A. Pak, N. Hartley, L. Fletcher, S. Frydrych, E. Galtier, E. Gamboa, D. Gericke, S. Glenzer, _et al._ , Formation of diamonds in laser-compressed hydrocarbons at planetary interior conditions, Nature Astronomy 1, 606 (2017).
  * Biener  _et al._ [2009] J. Biener, D. Ho, C. Wild, E. Woerner, M. Biener, B. El-Dasher, D. Hicks, J. Eggert, P. Celliers, G. Collins, _et al._ , Diamond spheres for inertial confinement fusion, Nuclear Fusion 49, 112001 (2009).
  * Balmer  _et al._ [2009] R. Balmer, J. Brandon, S. Clewes, H. Dhillon, J. Dodson, I. Friel, P. Inglis, T. Madgwick, M. Markham, T. Mollart, _et al._ , Chemical vapour deposition synthetic diamond: materials, technology and applications, Journal of Physics: Condensed Matter 21, 364221 (2009).
  * Kraus  _et al._ [2016] D. Kraus, A. Ravasio, M. Gauthier, D. O. Gericke, J. Vorberger, S. Frydrych, J. Helfrich, L. B. Fletcher, G. Schaumann, B. Nagler, B. Barbrel, B. Bachmann, E. J. Gamboa, S. Göde, E. Granados, G. Gregori, H. J. Lee, P. Neumayer, W. Schumaker, T. Döppner, R. W. Falcone, S. H. Glenzer, and M. Roth, Nanosecond formation of diamond and lonsdaleite by shock compression of graphite, Nature Communications 7, 1 (2016).
  * Scandolo  _et al._ [1995] S. Scandolo, M. Bernasconi, G. Chiarotti, P. Focher, and E. Tosatti, Pressure-induced transformation path of graphite to diamond, Physical Review Letters 74, 4015 (1995).
  * Hirai and Kondo [1991] H. Hirai and K.-i. Kondo, Modified phases of diamond formed under shock compression and rapid quenching, Science 253, 772 (1991).
  * Plimpton [1995] S. Plimpton, Fast parallel algorithms for short-range molecular dynamics, Journal of Computational Physics 117, 1 (1995).
  * Thompson  _et al._ [2022] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in ’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton, LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales, Computer Physics Communications 271, 108171 (2022).
  * Wood  _et al._ [2018] M. A. Wood, D. E. Kittell, C. D. Yarrington, and A. P. Thompson, Multiscale modeling of shock wave localization in porous energetic material, Physical Review B 97, 1 (2018), arXiv:1711.02769 .
  * Nosé [1984] S. Nosé, A unified formulation of the constant temperature molecular dynamics methods, The Journal of Chemical Physics 81, 511 (1984).
  * Larsen  _et al._ [2016] P. M. Larsen, S. Schmidt, and J. Schiøtz, Robust structural identification via polyhedral template matching, Modelling and Simulation in Materials Science and Engineering 24, 055007 (2016).


# Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction

Xiaoyang Lyu111Equal contribution. Chirui Chang∗ Peng Dai Yang-Tian Sun
Xiaojuan Qi222Corresponding author.
The University of Hong Kong
{shawlyu, chiruichang<EMAIL_ADDRESS>{daipeng<EMAIL_ADDRESS>
###### Abstract

Scene reconstruction from multi-view images is a fundamental problem in
computer vision and graphics. Recent neural implicit surface reconstruction
methods have achieved high-quality results; however, editing and manipulating
the 3D geometry of reconstructed scenes remains challenging due to the absence
of naturally decomposed object entities and complex object/background
compositions. In this paper, we present Total-Decom, a novel method for
decomposed 3D reconstruction with minimal human interaction. Our approach
seamlessly integrates the Segment Anything Model (SAM) with hybrid implicit-
explicit neural surface representations and a mesh-based region-growing
technique for accurate 3D object decomposition. Total-Decom requires minimal
human annotations while providing users with real-time control over the
granularity and quality of decomposition. We extensively evaluate our method
on benchmark datasets and demonstrate its potential for downstream
applications, such as animation and scene editing. The code is available at
https://github.com/CVMI-Lab/Total-Decom.git.

Figure 1:  Indoor scenes consist of complex compositions of objects and
backgrounds. Our proposed method, Total-Decom, (a) performs 3D reconstruction
from posed multiview images, (b) decomposes the reconstructed mesh to generate
high-quality meshes for individual objects and backgrounds with minimal human
annotations. This approach facilitates such applications as (c) object re-
texturing and (d) scene reconfiguration. For additional demonstrations, please
refer to our supplementary materials and videos.

##  1 Introduction

Scene reconstruction from multi-view images is a fundamental problem in
computer vision and graphics [31, 32, 14, 29, 13, 27, 24]. Recently, neural
implicit surface reconstruction methods such as VolSDF [43] and NeuS [39] have
been proposed to address this problem and have achieved high-quality results.
However, editing and manipulating the 3D geometry of a reconstructed scene
remains challenging due to the absence of naturally decomposed object entities
and complex object/background compositions. Such functionality is, however,
desired for many real-world applications, such as editing, animation, and
simulation. Consequently, we are motivated to investigate decomposed 3D
reconstruction, which enables the extraction of desired object-level shapes
and facilitates scene manipulations such as reorganizing objects in a scene
(see Fig. 1).

A few attempts have been made to decompose a reconstructed 3D scene into
individual objects using separate Multi-Layer Perceptron (MLP) layers to
represent specific objects [40, 41, 22, 19]. However, these approaches face
scalability issues when dealing with scenes containing numerous objects [19].
Furthermore, the success of these methods heavily relies on human-annotated
instance masks on multi-view images during training, which poses challenges in
obtaining them for large-scale practical applications. Moreover, even with
ground-truth instance labels, the existing state-of-the-art method [41] still
fails to produce satisfactory results, with multiple objects missing, as shown
by the second row of Fig. 7, due to the inherent difficulties in separating
all objects using implicit representations.

In this paper, we introduce Total-Decom, a novel method designed for
decomposed 3D reconstruction with minimal human interaction. At the core of
our method lies the integration of the Segment Anything Model (SAM) [17]– an
interactive image segmentation model– hybrid implicit and explicit surface
representation, and a mesh-based region-growing approach. This integration
allows the decomposition of a scene into the background and individual objects
with minimal interactions and provides users with control over the granularity
and quality of decomposition via real-time interactions, as shown in Fig. 1.

Specifically, our method first employs an implicit neural surface
representation for its ability to achieve dense and complete 3D reconstruction
from images. At this stage, we also integrate object-aware information by
distilling image features from the SAM model for follow-up efficient
interaction and accurate decomposition. After obtaining the learned implicit
surface and features, our approach further extracts explicit mesh surfaces
while distilling features into their vertices. The explicit representation
provides valuable geometry topology information for scene decomposition and
enables real-time neural rendering to enhance human interactions. Then, in
order to identify and separate the desired object for surface decomposition,
we utilize the SAM decoder and the rendered SAM feature, converting a human-
annotated click on a single rendered image view into corresponding dense
object masks. Thanks to the segmentation capability of SAM and our feature
rendering design, this interactive process also allows users to obtain the
desired objects at different granularities while minimizing human interactions
and avoiding high computational costs. Lastly, with the derived object mask
from a single view with good object boundaries serving as object seeds, we
propose a mesh-based region-growing module that progressively expands these
seeds along the mesh surface to obtain decomposed 3D object surfaces. This
process leverages distilled feature similarities of vertices and 3D mesh
geometry topology for accurate object decomposition, further ensuring
precision by confining the growing process to mask boundaries obtained from
the SAM decoder.

We extensively validate our approach on benchmark datasets. More importantly,
our high-quality decomposed 3D reconstruction enables many downstream
applications in manipulating and animating objects in virtual environments,
including re-texturing [4] with diffusion model, deformation, and motion. (See
Fig. 1 and videos in supplementary).

In sum, our main contributions are as follows:

  * •

We introduce a novel pipeline that seamlessly integrates the segment anything
model with hybrid implicit-explicit neural surface representations for 3D
decomposed reconstruction from sparse posed images. Our approach requires
minimal human annotations (approximately one click per object on average)
while achieving high decomposition quality.

  * •

We propose a new mesh-based region-growing method that leverages the geometry
topology of 3D mesh, feature similarities among vertices, object masks, and
boundaries derived from the SAM model to accurately identify and extract
object surfaces for decomposition.

  * •

We perform an extensive evaluation of our approach on various datasets,
demonstrating its superior ability to decompose objects with high accuracy.
Furthermore, we showcase the potential of our results for numerous downstream
tasks, such as animation and scene editing.

##  2 Related Work

Object Compositional Reconstruction In order to better reconstruct the
geometry of objects in complex scenes, many research efforts have been devoted
to exploring better object-level compositional scene representations. For
instance, ObjSDF [40] proposes a compositional scene representation to assist
in geometry optimization in highly composite scenes and better object-level
extraction with the help of multi-view consistent instance labels. Kong _et
al_. [19] build an object-level scene model from a real-time RGB-D input
stream for object-compositional SLAM. Wu _et al_. build upon ObjSDF and
further mitigate object occlusions in complex scenes through an object
distinction regularization term. Additionally, a new occlusion-aware object
opacity rendering scheme is introduced to reduce the negative impact of
occlusion on scene reconstruction. However, these methods rely heavily on
accurate multi-view consistent ground-truth instance-level labels and cannot
effectively preserve all objects within the reconstruction.

Figure 2: Visualization for distilled generalized features.

Decompositional Neural Rendering Neural radiance fields (NeRFs) offer a
unified representation to represent the appearance [25, 44], and other
spatially varying properties [20]. In the field of decompositional neural
rendering, many research efforts investigate the distillation of generalized
features [16, 8, 18, 35], or segmentation predictions [33, 1] from large-scale
pre-trained image backbones, such as DINO [2, 28], CLIP features [30] and
panoptic segmentation [5], to neural fields. Specifically, Semantic NeRF [47]
explores incorporating regressing semantic labels into the process of Novel
View Synthesis. Building upon this, distilled feature fields (DFFs) [18] and
neural feature fusion fields (N3F) [35] have emerged as pioneering approaches
for distilling [12] semantic features from pretrained models, such as LSeg
[21] and DINO [2], into NeRF for decomposition. ISRF [8] does not rely solely
on feature matching, but instead obtains the final results through region
growing. These approaches leverage feature similarities or predicted
instance/semantic codes for grouping and implicitly decomposing a scene,
requiring no additional human annotation efforts and offering greater
scalability for large-scale scenes. However, when directly applied to the
reconstruction task, these strategies often produce decomposed outputs that
are incomplete and exhibit poorly defined contours (see Fig. 3) due to ill-
defined object boundaries [8] or multiview inconsistent and low-quality
segmentation predictions [37, 5, 11] used during training. In contrast to
existing methods, our work not only avoids the need for extensive manual
interaction through a geometry-guided feature approach but also enables the
extraction of any constituent part of a complex scene.

NeRF with SAM Recently, segmentation methods based on different
representations [5, 21, 17, 23] have developed rapidly. SAM [17], as an
emerging vision foundation model, achieves efficient 2D interactive
segmentation capabilities through extensive high-quality annotated
supervision. It brings forth more possibilities in this field and has the
potential to help achieve thorough decomposition of complex scenes with
plausible 2D segmentation capabilities. Recently, some concurrent works have
explored the combination of SAM and NeRF for decomposition. SA3D [3]
incorporates cross-view self-prompting technique to obtain multi-view
consistent masks. However, running SAM multiple times incurs high
computational costs, and the segmentation results are sensitive to interaction
trajectories, leading to unstable performance.

##  3 Empirical Study on General Visual Features

Figure 3: Comparison on different decomposition methods with SAM feature. SAM
+ region growing represents object extraction with our method. SAM +
similarity indicates object extraction with similarity matching in 3D space,
following [35, 18].

A central challenge in decomposed 3D scene reconstruction is incorporating
object-aware knowledge to accurately separate individual objects and
backgrounds. While existing methods have explored the use of ground-truth
multi-view consistent instance-level annotations [40, 41, 22, 19], these
approaches suffer from high annotation costs and scalability issues. Motivated
by recent advancements in vision foundation models providing generic features,
we investigate their potential for object decomposition and reducing human
annotation requirements. Although this strategy has been examined in neural
rendering [18, 8, 35], it remains under-explored in decomposed 3D
reconstruction, which necessitates more precise boundary information.
Accordingly, we investigate features from three foundation models: CLIP-LSeg
[21], DINO [2], and SAM [17].

We utilize the MonoSDF [45] for implicit neural surface reconstruction,
augment it with a feature rendering head, and distill features from the above
2D backbones. The rendered features after distillation are depicted in Fig. 2.
We observe that: (1) distilled CLIP-LSeg features cannot distinguish objects
of the same categories; (2) distilled DINO features lack accurate object
boundaries; and (3) distilled SAM features preserve object boundaries.
However, none of these features are discriminative enough to accurately
separate different object instances. For example, while SAM features perform
the best, directly grouping 3D objects based on similar feature responses
leads to the merging of distant areas due to the absence of geometry and
object boundary information, as shown in Fig. 3.

Despite the above shortcomings, we observe that the distillation brings the
features of the same object from different views closer than the original 2D
features from SAM, suggesting that high feature similarity often indicates a
high likelihood of belonging to the same object, as shown in Fig. 4.
Consequently, we propose a novel approach that leverages SAM features and a
mesh-based region-growing method to decompose a 3D scene with minimal human
annotations, typically requiring only one click per object.

Figure 4: Visualization of the SAM feature for the same object in different
views with t-SNE [36]. All the features are in the same feature space. Figure
5: Overview of Total-Decom. (1) Foreground and background decomposed neural
reconstruction. We have four networks in this stage to predict the geometry,
appearance, semantic, and SAM features per point. We follow the ObjSDF++ [41]
to use the foreground and background compositional representation with pseudo
geometry priors and apply $\min$ operation to construct the whole scene.
Notably, the foreground is constrained with object distinct loss (Eq. (6)) and
the background is regularized with Manhattan loss (Eq. (7)) and floor
reflection loss (Eq. (8)). Furthermore, we also train a solely feature network
to render the generalized features. (2) Interactive Decomposition. We firstly
extract the SAM feature from the feature network into the vertices of the
reconstruction mesh. Subsequently, for any given pose, we can render a color
image and a feature image. Passing the feature image and user-selected prompt
into the SAM decoder allows us to obtain the 2D mask of the regions of
interest. Utilizing our newly proposed surface region-growing algorithm, we
can then acquire the 3D mesh corresponding to these regions. Our method
enables the user to select objects with varying levels of granularity,
requiring just one or two clicks.

##  4 Overview

Our objective is to reconstruct a 3D scene from multi-view images and
decompose it into individual object entities and the background while
minimizing the need for human annotations. To achieve this, we propose a novel
pipeline that integrates SAM into a hybrid implicit-explicit surface
representation, combined with a mesh-based region-growing method to
effectively identify and decompose arbitrary 3D objects within a scene. The
overview of our approach is illustrated in Fig. 5.

As shown in Fig. 5 (Left) and detailed in Sec. 5, we first adopt an implicit
neural surface representation to achieve dense and complete 3D reconstruction
from images while incorporating object-aware information by distilling image
features from the SAM model into an implicit neural feature field (see Fig. 5:
bottom ). Importantly, we introduce geometry-guided regularization integrated
with semantic priors to disentangle foreground objects and backgrounds,
including occluded and invisible background regions with details unfolded in
Sec. 5.

Upon obtaining the learned implicit surface and features, our approach further
extracts explicit mesh surfaces for foreground objects and background; see
Fig. 5: Background $d_{B}(p)$ and Foreground $d_{F}(p)$; and distills features
into their vertices; see Fig. 5: Neural Mesh. The explicit mesh surface
provides valuable geometry topology information for follow-up scene
decomposition and enables real-time neural rendering to enhance human
interactions. Then, given rendered images and features from the mesh surface,
our method employs the SAM decoder to convert image clicks into dense object
masks to precisely identify the target object in one rendered view (see Fig.
5: Interactive Decomposition ) and allow users control over granularity and
quality while minimizing human interactions. Details are elaborated in Sec. 6.
Notably, by leveraging the rendered SAM features from our model, we only
require the decoding process of the SAM model at this stage, thus
circumventing high computational costs.

Lastly, using the vertices corresponding to dense object masks as seeds, we
propose a mesh-based region-growing module that progressively expands the
seeds along the mesh surface to obtain object surfaces as detailed in Sec. 6.
This module harnesses the feature similarities of vertices and the geometry
topology of the 3D mesh to achieve accurate object decomposition. The growing
process is also confined by the vertices corresponding to mask boundaries,
which further ensures the precision of object decomposition.

##  5 Neural Implicit Feature Distillation and Surface Reconstruction

In this stage, we employ an implicit neural field for reconstruction from
posed images, disentangle foreground and background using geometric priors,
and distill features from the SAM encoder to incorporate object-aware
information. An illustration of our implicit reconstruction is depicted in
Fig. 5. In the following, we first elaborate on our reconstruction network and
rendering formula, and then detail our core design for achieving background
and foreground separation while reconstructing occluded foreground areas.

Reconstruction network and SDF-based neural implicit surface representation.
We use the signed distance function $d(\mathbf{p})$ to represent the geometry
of the surface at each point $\mathbf{p}$. Considering a ray
$\mathbf{r}(t)=\mathbf{o}+t\mathbf{v}$ from a camera position $\mathbf{o}$ in
the direction of $v$, we calculate the signed distance function of each
sampled point $\mathbf{p}$ using the geometry MLP and use three MLPs to
predict the color $C(\mathbf{p},v)$, semantic logits $S(\mathbf{p})$, and
generalized feature $F(\mathbf{p})$ distilled from the SAM encoder,
respectively; refer to Fig. 5 for details. To apply the volume rendering
formula, we follow VolSDF [43] to convert the signed distance function to
volume density $\sigma(\mathbf{p})$:

$\sigma(\mathbf{p})=\begin{cases}\frac{1}{2\beta}\exp\left(\frac{-d(\mathbf{p})}{\beta}\right)&\text{
if }d(\mathbf{p})\geq 0\\\
\frac{1}{\beta}-\frac{1}{2\beta}\exp\left(\frac{d(\mathbf{p})}{\beta}\right)&\text{
if }d(\mathbf{p})<0\end{cases},$ (1)

where $\beta>0$ is a learnable parameter to decide the sharpness of the
surface density. Then, we use the volume rendering formula [15] to obtain
outputs $\mathbf{E}$ of the target pixel,

$\hat{\mathbf{E}}(r)=\sum_{i=1}^{M}T^{r}_{i}\alpha_{i}\hat{e}^{r}_{i}\;,$ (2)

where $\hat{e}\in\\{\hat{c},\hat{n},\hat{d},\hat{s},\hat{f}\\}$ represent the
predicted color, normal, depth, semantic logits, generalized feature.
$T^{r}_{i}$ and $\alpha_{i}$ represent the transmittance and alpha value
(a.k.a opacity) of the sample point, and their values can be computed by

$T^{r}_{i}=\prod_{j=1}^{i-1}(1-\alpha_{i}),~{}\alpha_{i}=1-\exp(-\sigma^{r}_{i}\delta^{r}_{i})\;,$
(3)

where $\delta^{r}_{i}$ is the distance between adjacent sample points.

We follow the loss function $\mathcal{L}_{\text{rgb}}$ and
$\mathcal{L}_{\text{geo}}$ in MonoSDF [45] to optimize the rendered color,
depth, and normal. For the rendered semantic $\hat{S}(r)$, we use the cross-
entropy loss defined as

$\mathcal{L}_{\text{sem}}=-\mathbb{E}_{\mathbf{r}\in\mathcal{R}}\left[\sum_{l=1}^{L}{P}_{l}(r)\log\hat{P}_{l}({r})\right]\;,$
(4)

where $P_{l}(r),\hat{P}_{l}(r)$ are the multi-class semantic probability as
class $l$ of the ground truth map and rendering map for ray $r$, respectively.
Additionally, we use the $L2$ loss $\mathcal{L}_{f}$ to optimize the rendered
generalized feature $\hat{F}(r)$ for distilling the $F(r)$ from the SAM
encoder.

Modeling foreground and background compositional scene geometry. To represent
foreground and background geometry separately, we construct two different SDF
fields $\mathcal{S}=\\{\mathcal{F},\mathcal{B}\\}$ following [41]. The single
scene $\Omega$ is the composition of the
$\Omega=\mathcal{F}\bigcup\mathcal{B}$. The scene SDF can be calculated as the
minimum of two fields SDFs
$d_{\Omega}(\mathbf{p})=\text{min}\\{d_{\mathcal{F}}(\mathbf{p}),d_{\mathcal{B}}(\mathbf{p})\\}$.
To learn the geometry from the supervision of foreground and background masks,
we adopt occlusion-aware opacity rendering [41] to guide the learning of
different field surfaces. The loss function is defined as:

$\mathcal{L}_{O}=\mathbb{E}_{\mathbf{r}\in\mathcal{R}}[\sum_{\mathcal{S}_{i}\in\mathcal{S}}\left\|\hat{O}_{\mathcal{S}_{i}}(\mathbf{r})-O_{\mathcal{S}_{i}}(\mathbf{r})\right\|],$
(5)

where $\hat{O}(r)=\int_{t_{n}}^{t_{f}}T(\mathbf{r}(t))\alpha(\mathbf{r}(t))dt$
to formulate the occlusion-aware object opacity in the depth range
$[t_{n},t_{f}]$.

For reconstructing the clean foreground mesh, we follow the object distinction
regularization term [41] forcing each point in a single scene to be only
located inside one field, which is defined as follows:

$\mathcal{L}_{\text{reg}}=\mathbb{E}_{\mathbf{p}}[\sum_{d_{\mathcal{S}_{i}}(\mathbf{p})\neq
d_{\Omega}(\mathbf{p})}\operatorname{ReLU}\left(-d_{\mathcal{S}_{i}}(\mathbf{p})-d_{\Omega}(\mathbf{p})\right)]\;,$
(6)

Compared with the foreground, the background is more difficult to reconstruct
because it has many occluded areas that are not visible in all captured views.
To regularize the reconstruction of these areas, we follow the Manhattan world
assumption [46, 10], i.e., the surfaces of man-made scenes should be aligned
with three dominant directions. We use this to regularize the reconstruction
of the floor and the wall using,

$\displaystyle\mathcal{L}_{\text{man}}=$
$\displaystyle\mathbb{E}_{r\in\mathfrak{F}}(\hat{p}_{f}(r)|1-\hat{\mathbf{n}}({r})\cdot\mathbf{n}_{f}|)+$
(7)
$\displaystyle\mathbb{E}_{r\in\mathfrak{W}}(\min_{i\in\\{-1,0,1\\}}\hat{p}_{w}(r)|i-\hat{\mathbf{n}}({r})\cdot\mathbf{n}_{w}|),$

where $\hat{p}_{f},\hat{p}_{w}$ represent the probabilities of the pixel being
floor and wall derived from the semantic MLP, $\mathfrak{F},\mathfrak{W}$ are
the sets of camera rays of image pixels that are labeled as floor and wall
regions, $\hat{n}_{r}$ is the rendering normal of rays $r$,
$n_{f}=\left<0,0,1\right>$ represent the assumed normal direction in the floor
regions and $n_{w}$ is a learnable normal to represent the normal direction in
the wall regions.

Figure 6: The effect of different constraint on Replica room_1.

As shown in Fig. 6, applying this constraint to regularize background
reconstruction will yield more regular geometry but there still exist many
undesired structures due to the inaccurate semantic information of these
invisible regions. Fortunately, the majority of occlusions occur on the
ground, and the ceiling corresponds one-to-one with the structure of the
floor. Therefore, we can utilize the structural information of the ceiling to
constrain the unknown areas on the ground. Specifically, when we have a frame
of an image that observes the ceiling, we can start from points on the ceiling
$\mathbf{p}_{c}$, emitting a ray along the direction of gravity
$\mathbf{n}_{g}=\left<0,0,-1\right>$. In the background domain, the first
point $\mathbf{p}_{f}$ hit by this ray is considered as the ground. By
employing the root finding method [27], we locate this point and constrain its
normal vector by

$\mathcal{L}_{\text{floor}}=|1-\mathbf{n}(\mathbf{p}_{f})\cdot\mathbf{n}_{f}|.$
(8)

As shown in Fig. 6, this constraint can help us get the complete and regular
background.

##  6 Interactive Decomposition

Upon obtaining the implicit foreground surface $d_{\mathcal{F}}$ and features,
we focus on decomposing the foreground into individual objects and allowing
users to control the decomposition through interactions in this stage, as
shown in Fig. 5. We propose to use an explicit mesh surface as it can provide
geometry information for better decomposition and allow efficient rendering by
using rasterization or combining with Gaussian splatting. We provide details
on the integration of Gaussian splatting in the supplementary material. Hence,
before conducting decomposition, we extract foreground mesh
$\mathcal{M}_{\mathcal{F}}$ and distill features to its vertices
$\mathcal{V}=\\{v_{1},...,v_{n}\\}$. Then, each vertex $v_{i}$ is associated
with a distilled feature $f_{v_{i}}$ and a 3D location $p_{i}$. The vertices
on the mesh are connected via edges $\mathcal{E}$, which are determined by the
geometry topology of the mesh. As shown in Sec. 3, it is still challenging to
rely solely on distilled features $f_{v_{i}}$ to decompose 3D objects. Thus,
we introduce human annotations to identify each object and aim to minimize
human interactions. The following details how we realize object decomposition
based on a designed mesh-based region-growing method with human interactions
using our designed method. We will first introduce how we obtain seed points
for an object by combining SAM, rendered features, and human clicks. Then we
elaborate on our new mesh-based region-growing algorithm for acquiring object-
level meshes to realize a complete decomposition.

#### Object Seed Generation

Given a $\mathcal{M}_{\mathcal{F}}$, the goal is to obtain a set of initial
seed points for each object $o$ by using human annotations. As shown in Fig.
5, with one image $I$ containing $o$ with corresponding feature map $f_{I}$
rendered from $\mathcal{M}$, the user will produce a click $c$ on the image
identifying the desired object. Based on the click $c$, serving as the prompt,
and $f_{I}$, a 2D mask $m_{o}$ that describes the object can be efficiently
obtained through the lightweight mask decoder of SAM. Users are also allowed
to adjust its clicks according to $m_{o}$ to refine it. Our experiment shows
that most of the objects’ mask $m_{o}$ can be obtained with just one click.
The pixels in mask $m_{o}$ are then mapped to their corresponding vertices to
yield the 3D object seeds denoted as $\mathcal{S}_{o}$ for the follow-up
region-growing algorithm. This process efficiently turns a single click into a
set of vertices to enhance the accuracy of identifying object $o$. Besides, to
locate object boundaries, we also extract the contour pixels $c_{o}$ of
$m_{o}$ and map them to their corresponding vertices $\mathcal{B}_{o}$, which
forms the boundary condition for region growing.

#### Region-growing on Mesh for Decomposition

Given the seeds $\mathcal{S}_{o}$ and boundary condition $\mathcal{B}_{o}$, we
design a region-growing method to obtain the corresponding mesh $M_{o}$ for
object $o$. As illustrated in Algorithm 1 in the supplementary material, the
seeds $\mathcal{S}_{o}$ are progressively expanded along the mesh
$\mathcal{M}_{\mathcal{F}}$ to include their connected neighboring vertices
with high feature similarities. The boundary vertices $\mathcal{B}_{o}$
constrain the propagation process by enforcing it to stop if including
vertices that will be outside of mask $m_{o}$. This helps ensure the boundary
accuracy of the decomposed object. It is worth mentioning that the geometric
relationship between vertices is leveraged in this progressive expanding
process, benefiting the extraction of objects and reducing computation.
Compared to region-growing algorithm that purely relies on spatial location
[8] or features, incorporating edges from the mesh as growth paths introduces
a constrain from topological structure, thereby making the growth process more
consistent with the geometric structure and avoiding including vertices that
are not geometrically related with seed vertices but share similar features
with them, making the extraction of object be accurate. We present more
analysis on the region-growing algorithm in the supplementary material.

##  7 Experiments

Figure 7: Visualized assessments on different datasets. We present the
reconstruction results for the background, foreground and decomposed objects
on Replica [34], ScanNet [6], NICE-SLAM [48] and our self-captured billiard
room. To clearly visualize the decomposed objects, we use different color for
the different instances.

###  7.1 Experiment Setup

#### Implementation Details.

Our method is implemented using Pytorch and uses the Adam optimizer with a
learning rate of $5e-4$ for the tiny MLP part ( 2 layers with 256 channels for
the geometry, appearance, and feature prediction, 1 layer with 128 nodes for
the semantic prediction ) In the reconstruction strategy, we minimize the loss

$\displaystyle\mathcal{L}=$
$\displaystyle\mathcal{L}_{\text{rgb}}+\mathcal{L}_{\text{geo}}+\lambda_{1}\mathcal{L}_{O}+\lambda_{2}\mathcal{L}_{\text{reg}}+$
(9)
$\displaystyle\lambda_{3}\mathcal{L}_{\text{man}}+\lambda_{4}\mathcal{L}_{\text{floor}}+\lambda_{5}\mathcal{L}_{\text{sem}}+\lambda_{6}\mathcal{L}_{f},\;$

to optimize our implicit neural surface, where we set
$\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5},\lambda_{6}$ as
$0.1,0.1,0.01,0.01,0.5,0.1$, respectively. More details can be found in the
supplementary.

#### Dataset and Metrics

Our experiments are mainly conducted on the Replica [34] dataset, which is a
synthetic dataset with each providing accurate geometry, HDR textures and 3D
instance annotations. We follow the selection of ObjSDF++ [41] to evaluate the
effectiveness of our method. We report both instance-level and holistic
reconstruction results on this dataset. The reconstruction results are mainly
evaluated by Chamfer-$L_{1}$ and F-Score. To further demonstrate the
robustness of our method, we also use the ScanNet [6] as the real-world
dataset which provides 1513 scenes. Due to its lack the object object-level
ground truth, we show the visualized assessment in the main paper. Besides the
public dataset mentioned before, we also evaluate the performance of our
method on the self-captured data, one is the room from the NICE-SLAM [48] and
another is the self-captured billiard room. More details and results can be
seen in the supplementary.

#### Compared Methods.

The compared methods are mainly divided into two categories. The first one is
the object compositional reconstruction method that uses multiple fields to
represent each object with the supervision from ground truth instance masks,
like ObjSDF++ [41]. We compare the instance level and holistic scene
reconstruction quality with them. The second category is the volume density
based methods that decompose each scene with generalized features, like ISRF
[8], DFF [18]. Since this type of method does not introduce geometric
constraints, we mainly compare the way of decomposition.

###  7.2 Results

| Scene Reconstruction | Decomposed Reconstruction
---|---|---
Method | Chamfer-$\mathcal{L}_{1}$ $\downarrow$ | F-score $\uparrow$ | Chamfer-$\mathcal{L}_{1}$ $\downarrow$ | F-score $\uparrow$
ObjSDF++ | 3.58 | 85.69 | 3.84 $\pm$ 0.02 | 79.49 $\pm$ 0.08
_Ours_ | 3.53 | 85.82 |  3.58 $\pm$ 0.01 |  81.70 $\pm$ 0.08

Table 1: Quantitative assessments from Replica dataset on scene and decomposed
reconstruction.

#### Scene reconstruction and object decomposition on the Replica dataset.

To evaluate the decomposed reconstruction accuracy of Total-Decom, we conduct
experiments on the Replica dataset as it provides ground-truth objects’ meshes
for evaluation. We mainly compared our approach with the ObjSDF++, the state-
of-the-art method that decomposes the scene structure with pseudo geometry
priors as far as we know. Because the number of decomposed objects from
ObjSDF++ is limited (around 25 objects per scene), we only evaluate the
foreground objects that ObjSDF++ can extract for a fair comparison. In
reality, our approach can generally yield a more complete decomposition of the
scene with more objects. The results are shown in Table 1. Although our
approach doesn’t rely on ground-truth instance masks for decomposition, our
method still surpasses ObjSDF++ in both scene and decomposed object
reconstruction results. It is worth noting that our approach only requires
1.41 clicks on average per object, while ObjSDF++ requires dense human
instance annotations on multi-view images. Our reconstructed results also
outperform ObjSDF++ qualitatively. As shown in Fig. 7, ObjSDF++ tends to
inaccurately reconstruct the vase (Room_0) and trash bin (Office_4) within the
background field or may fail to recover the structure of the chair (Office_4).
With an elaborately mesh-based region-growing method facilitated by SAM, our
method can deliver decomposed results of higher qualities.

Method Geometry Multi-grained Scene-level Single-View Interaction LSeg + DFF
[18] ✗ ✗ ✗ ✗ DINO + DFF/N3F [35] ✗ ✓ ✗ ✗ ISRF [8] ✗ ✓ ✗ ✗ SAMNeRF [3] ✗ ✓ ✗ ✗
Panoptic Lifting [33] ✗ ✗ ✓ ✗ AssetField [42] ✗ ✗ ✓ ✗ vMAP [19] ✓ ✗ ✓ ✗
ObjSDF++ [41] ✓ ✗ ✓ ✗ _Ours_ ✓ ✓ ✓ ✓

Table 2: Comparison of decomposition capabilities between different methods.

#### Background reconstruction.

We demonstrate the quality of background reconstruction in Fig. 6 and Fig. 7.
Our approach consistently delivers high-quality clean background
reconstructions and reconstructs occluded areas.

#### Comparison of Decomposition Capabilities

We compare the decomposition capabilities of our model with existing methods
in NeRF and 3D reconstruction in terms of whether they can support geometric
decomposition, multi-grained decomposition, scene-level decomposition, and
interactive selection. Tab. 2 presents the comparison results, indicating that
our method is the sole approach encompassing all capabilities. Additional
results and applications exploiting these capabilities can be found in the
supplementary material.

##  8 Conclusion

We presented Total-Decom, a novel framework for reconstructing a 3D surface
and decomposing it into individual objects and backgrounds, significantly
reducing the reliance on object annotations. Our approach centers on
integrating SAM with hybrid implicit-explicit surface representations and a
mesh-based region-growing algorithm. SAM provides object-aware features and
facilitates the acquisition of more accurate object seeds for region growth.
Simultaneously, the region-growing algorithm effectively combines geometric
topology from explicit mesh and object-aware features from SAM to accurately
decompose desired objects with minimal human annotations. Qualitative and
quantitative evaluations indicate that our method can yield not only precise
geometry but also extract as many objects as possible. We hope our proposed
method will facilitate the development of environment simulators in the
future.

Acknowledgments: This work has been supported by Hong Kong Research Grant
Council - Early Career Scheme (Grant No. 27209621), General Research Fund
Scheme (Grant No. 17202422), and RGC Matching Fund Scheme (RMGS). Part of the
described research work is conducted in the JC STEM Lab of Robotics for Soft
Materials funded by The Hong Kong Jockey Club Charities Trust.

Supplementary

In the supplementary file, we provide more implementation details and more
results not elaborated in our paper due to the paper length limit:

  * •

Sec. S9: more implementation details.

  * •

Sec. S10: more qualitative comparisons and qualitative results.

  * •

Sec. S11: more ablation results

  * •

Sec. S12: limitation analysis.

##  S9 Implementation Details

###  S9.1 Hash Grid in Implicit Reconstruction

Our entire pipeline primarily consists of four distinct networks: geometry
network, appearance network, semantic network, and feature network. The
geometry, appearance, and semantic networks share the same hash grid, while
the feature network utilizes another. We follow instant-NGP [26] to construct
the hash grid as a replacement for the frequency position encoding used in
vanilla NeRF [25] to accelerate model convergence. Specifically, the 3D space
is represented by multi-level feature grids:

$R_{l}:=\left\lfloor R_{\min}b^{l}\right\rfloor,b:=\exp\left(\frac{\ln
R_{\max}-\ln R_{\min}}{L-1}\right),$ (10)

where $R_{\text{min}}=16$ and $R_{\text{max}}=2048$ represent the coarsest and
finest resolutions, respectively. Each level grid has $T=2$ dimensional
features. Both the feature network and geometry network grids share the same
structure.

###  S9.2 Loss Function

Our training objective consists of different losses including
$\mathcal{L}_{\text{rgb}},\mathcal{L}_{\text{geo}},\mathcal{L}_{O},\mathcal{L}_{\text{reg}},\mathcal{L}_{\text{man}},\mathcal{L}_{\text{floor}},\mathcal{L}_{\text{sem}},\mathcal{L}_{f}$
simultaneously, following [45, 10, 41]. Their detailed formulations are shown
below. (1) $\mathcal{L}_{\text{rgb}}$ is the color loss function defined as

$\mathcal{L}_{\text{rgb}}=\sum_{\mathbf{r}\in\mathcal{R}}||C(r)-\hat{C}(r)||\;,$
(11)

where $C(r)$ is the ground truth color value alone the ray $r$ and
$\hat{C}(r)$ is the rendered color along the ray $r$.
$\mathcal{L}_{\text{geo}}$ is the geometry loss function to constrain the
geometry of the implicit surface which includes three different parts
$\mathcal{L}_{\text{depth}},\mathcal{L}_{\text{normal}},\mathcal{L}_{\text{eik}}$,
defined as

$\mathcal{L}_{\text{geo}}=0.1\mathcal{L}_{\text{depth}}+0.05\mathcal{L}_{\text{normal}}+0.05\mathcal{L}_{\text{eik}}\;.$
(12)

$\mathcal{L}_{\text{depth}}$ is the scale-invariant depth loss to regularize
the rendering depth $\hat{D}(r)$ by the pseudo depth $\bar{D}(r)$ from
Omnidata [7], which is defined as

$\mathcal{L}_{\text{depth
}}=\sum_{\mathbf{r}\in\mathcal{R}}\|w\hat{D}(\mathbf{r})+q-\bar{D}(\mathbf{r})\|^{2}\;,$
(13)

where $w,q$ represents scale and shift solved by the least square method, and
$\mathcal{R}$ is the unit of all rays. $\mathcal{L}_{\text{normal}}$ is the
normal loss defined as:

$\mathcal{L}_{\text{normal
}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\hat{N}(\mathbf{r})-\bar{N}(\mathbf{r})\|_{1}+\|1-\hat{N}(\mathbf{r})\bar{N}(\mathbf{r})\|_{1}\;,$
(14)

where $\hat{N}(r)$ is the predicted normal from Omnidata [7] and $\bar{N}(r)$
is the rendered normal. $\mathcal{L}_{\text{eik}}$ is the eikonal loss
proposed by [9] to regularize the signed distance field, which is defined as

$\mathcal{L}_{\text{eik}}=\sum\mathbb{E}_{d_{\Omega}}\left(\left\|\nabla
d_{\Omega}(\mathbf{p})\right\|-1\right)^{2}\;.$ (15)

###  S9.3 Analysis of Region Growing

Algorithm 1 Mesh-based Region Growing

1:Input: $\mathcal{V}$ associated with $f_{v_{i}}$ for each vertex $v_{i}$,
$\mathcal{S}_{o}$, $\mathcal{B}_{o}$, $\mathcal{E}$, a similarity threshold
$\tau$, an attenuation parameter $\theta$ and a tolerance $\epsilon$

2:Output: the vertex set $\mathcal{V}_{o}$ for the object mesh
$\mathcal{M}_{o}$

3:Initialize the target vertex set $\mathcal{V}_{o}\leftarrow\mathcal{S}_{o}$

4:Initialize the intermediate target vertex set
$\mathcal{V}_{o}^{\prime}\leftarrow\mathcal{V}_{o}$

5:initialize the candidate seed vertex set
$\mathcal{S}_{o}^{\prime}\leftarrow\emptyset$

6:while $\mathcal{S}_{o}\neq\emptyset$ do

7: for each vertex $s$ in $\mathcal{S}_{o}$ do

8: Find all the neighbors of $s$ with $\mathcal{E}$, as $\mathcal{N}$

9: for each vertex $n$ in $\mathcal{N}$ do

10: $\text{sim}(f_{s},f_{n})\leftarrow\frac{{f_{s}\cdot f_{n}}}{{\left\lVert
f_{s}\right\rVert\left\lVert f_{n}\right\rVert}}$

11: if $\text{sim}(f_{s},f_{n})>\tau$ then

12: add $n$ to $\mathcal{V}_{o}^{\prime}$, add $n$ to $\mathcal{S}_{o}$

13: else

14: add $s$ to $\mathcal{S}_{o}^{\prime}$

15: end if

16: end for

17: end for

18: if
$\frac{\lvert\mathbb{\mathcal{missing}}{V}_{o}^{\prime}\cap\mathcal{B}_{o}\rvert}{\lvert\mathbb{\mathcal{missing}}{B}_{o}\rvert}>\epsilon$
then

19: return $\mathcal{V}_{o}$

20: else

21: $\mathcal{V}_{o}\leftarrow\mathcal{V}_{o}^{\prime}$,
$\mathcal{S}_{o}\leftarrow\mathcal{S}_{o}^{\prime}$,
$\mathcal{S}_{o}^{\prime}\leftarrow\emptyset$

22: end if

23: $\tau\leftarrow\tau-\theta$

24:end while

The detailed algorithm of mesh-based region growing is illustrated in
Algorithm 1. The success of the mesh-based region growing algorithm hinges on
propagating 2D guidance into the 3D space. The accurate mesh extraction can be
attributed to the following reasons:

(1) The distilled 3D features are both view-consistent and instance-aware. As
demonstrated in Fig. 4 of the main paper, for features of the same object
under different viewpoints, the inter-class distance of the distilled feature
pairs is significantly smaller than that of the feature pairs derived from the
teacher model [17]. Consequently, with the distilled features used in both 2D
and 3D spaces, 2D seed pixels and boundary pixels can serve as reliable
references for 3D seed vertices and boundary vertices.

(2) The SAM decoder, on top of rendered SAM features from one view, produces a
dense mask that serves as the seeds for the region-growing process and
constrains the boundary for growth.

(3) The explicit geometry information, i.e., vertices and edges, constrains
the growing process by considering the topology of meshes, effectively ruling
out vertices that have high feature similarities or are spatially adjacent but
not geometrically connected.

(4) The separation of the foreground mesh $\mathcal{M}_{F}$ allows the
algorithm to operate effectively in low-noise environments by removing
conflicting mesh vertices from the background. With a pure foreground mesh for
growth, the absence of surfaces originating from the background minimizes
interference during the extraction process.

In summary, these factors collectively enable our method to extract the 3D
mesh of a specified object using only a single viewpoint and a few clicks.

##  S10 More Qualitative Comparisons and Qualitative Results

###  S10.1 ScanNet Results

Figure S8: Visualization assessments of the reconstruction results on the
ScanNet dataset.

To demonstrate the effectiveness of the foreground and background
reconstruction method. We also compare our method against various recent works
on the ScanNet dataset as shown in Table S3. Our method achieves state-of-the-
art on both Chamfer-$\mathcal{L}_{1}$ and F-score. In particular, our method
obtains a significant increase on ”Comp” and ”Recall” because of the
constraint on the invisible regions. However, the ”Acc” and ”Prec” are
slightly decreasing because the ground truth mesh doesn’t have the complete
background mesh which will influence these two metrics.

As shown in Fig. S8, our method can obtain a cleaner background mesh and
detailed foreground mesh on the real scene dataset which is much easier for
cross-scene editing and getting the mesh asset for many other downstream
applications.

Method |  Acc $\downarrow$ |  Comp $\downarrow$ |  C-$\mathcal{L}_{1}$ $\downarrow$ |  Prec $\uparrow$ |  Recall $\uparrow$ |  F-score $\uparrow$
---|---|---|---|---|---|---
COLMAP [32] | 0.047 | 0.235 | 0.141 | 71.1 | 44.1 | 53.7
UNISURF [27] | 0.554 | 0.164 | 0.359 | 21.2 | 36.2 | 26.7
VolSDF [43] | 0.414 | 0.120 | 0.267 | 32.1 | 39.4 | 34.6
NeuS [39] | 0.179 | 0.208 | 0.194 | 31.3 | 27.5 | 29.1
Manhattan-SDF [10] | 0.072 | 0.068 | 0.070 | 62.1 | 56.8 | 60.2
NeuRIS [38] | 0.050 | 0.049 | 0.050 | 71.7 | 66.9 | 69.2
MonoSDF [45] | 0.035 | 0.048 | 0.042 | 79.9 | 68.1 | 73.3
ObjSDF++ | 0.039 | 0.045 | 0.042 | 78.1 | 70.6 | 74.0
Ours | 0.044 | 0.040 | 0.042 | 74.7 | 74.8 | 74.7
Table S3: Quantitative assessments of the proposed model against previous
works on the ScanNet dataset.

###  S10.2 Real Time Interaction on the Replica dataset

In the main paper, we introduce Total-Decom, a method capable of decomposing
an entire scene at any granularity level. We have also designed a graphical
user interface (GUI) to interactively decompose desired objects for downstream
applications. The foreground and background reconstructed meshes are loaded
simultaneously, and the vertices are used as fixed initialization points to
train the Gaussian Splatting model for real-time rendering. Regarding feature
rendering, we utilize the trained grid and apply the rasterization method to
obtain the feature map of the observed view. The rendered features and
selected prompts are then passed into the SAM decoder to generate the mask for
region-growing. Further details are showcased in the accompanying video.

###  S10.3 More results on scenes defying Manhattan assumption

We tested the influence of Manhattan constraints on the TNT auditorium, which
features sloped ground. As shown in Fig. S9, our constraint effectively
eliminates floaters and yields smooth floors. This success is because the
optimization focuses on minimizing overall loss (Eq.(9)), and the
regularization term (Eq.(7$\sim$8)) only penalizes heavily reconstructions
that substantially violate the constraints, such as floaters on the floor. The
combinatorial effects of all loss terms make the optimization robust toward
corner cases.

Figure S9: Background reconstruction results for Auditorium scene, consider
moving this to the supp Figure S10: Visualization of mask2former results.
Figure S11: Visualization decomposed results on the self-captured dataset.

###  S10.4 More Demonstrations and Evaluations on Self-captured Data

We utilize self-captured data to demonstrate the generalization and
practicality of our method. The data are captured using Apple ARKit. Our
method can successfully separate the background and foreground, as well as
interactively decompose the desired objects, as illustrated in Fig. S11.
Moreover, with 16 clicks, we can successfully obtain the 16 very small
billiards. Note that different instances are labeled with different colors. To
our knowledge, no existing works can achieve this level of separation without
relying on exhaustive human annotations, such as ObjSDF and ObjSDF++. Even
when provided with dense annotations across frames, decomposing all small
objects remains a challenging task when relying solely on implicit
reconstruction in existing methods as illustrated in Fig. 7 in the main paper.

We are aware that there are some existing methods for panoptic segmentation,
such as Mask2Former [5], which can generate semantic and instance labels for
each frame. However, when applied to such a challenging scenario, they fail to
segment these small objects, as seen in Fig. S10. This suggests that they
cannot be readily used even for segmenting a single frame, let alone ensuring
view consistency across multiple frames for preparing the data required by
existing methods such as ObjSDF, ObjSDF++, and vMAP.

| ObjSDF++ | Ours
---|---|---
Room0 | 34 | 41
Room1 | 22 | 24
Room2 | 24 | 30
Office0 | 21 | 23
Office1 | 13 | 20
Office2 | 28 | 38
Office3 | 29 | 38
Office4 | 21 | 28
Table S4: The number of decomposed reconstructed foreground objects on
different scenes from Replica following the ObjSDF++ splits (100 images).

###  S10.5 Number of Selected Objects

Table S4 presents the number of extracted objects on the Replica dataset.
Although ObjSDF++ employs ground-truth instance labels for scene
decomposition, it fails to extract certain objects, as illustrated in Fig. 7
of the main paper. Consequently, ObjSDF++ decomposes fewer objects with many
objects missing compared to our proposed method. This highlights that
achieving complete decomposition of 3D objects is challenging when solely
relying on implicit representations, whereas our combined implicit and
explicit design attains higher performance. Additionally, our method attains
such decomposition qualities with merely one or two human clicks per object,
rather than relying on dense masks across multiple views for a single object,
as required in ObjSDF and ObjSDF++.

##  S11 Ablation Studies

| Decomposed Reconstruction
---|---
Method | Chamfer-$\mathcal{L}_{1}$ $\downarrow$ | F-score $\uparrow$
Full Model | 2.59 | 87.35
w/o Foreground and Background Decomposition | 3.10 | 84.75
w/o Region Growing | 6.98 |  54.01

Table S5: Ablation study assessing the influence of different components to
the decomposed reconstruction results on Replica room0.

Component-wise Study We conduct ablation studies on the Replica dataset to
evaluate the effectiveness of our designed modules, as this dataset provides
instance-level ground-truth geometry. We examine the influence of foreground
and background decomposition and the region-growing algorithm on object
selection. Table S5 summarizes the results of our ablation study. “w/o
Foreground and Background Reconstruction” refers to selecting objects on the
whole mesh, while “w/o Region Growing” indicates the use of a simple cosine
similarity to segment objects instead of our designed algorithm. Foreground
and background decomposition methods enhance selection quality, as the
foreground mesh naturally prevents the region-growing method from selecting
the background mesh. The carefully designed region-growing method achieves an
overall improvement of 33.34 in F-score and 4.39 in Chamfer-$\mathcal{L}_{1}$.
These experiments demonstrate the effectiveness of our proposed method.

##  S12 Limitations

While our method is capable of decomposing scenes with minimal human
interaction, it still faces some limitations in handling occluded foreground
areas. For instance, our approach cannot complete the occluded areas of
foreground objects due to the absence of training supervision. In the future,
we plan to explore the integration of generative methods to complete such
invisible 3D objects and obtain high-quality object meshes even in the
presence of occlusions.

## References

  * Bing et al. [2022] WANG Bing, Lu Chen, and Bo Yang.  Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Cen et al. [2023] Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian.  Segment anything in 3d with nerfs.  _arXiv preprint arXiv:2304.12308_ , 2023.
  * Chen et al. [2023] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Text2tex: Text-driven texture synthesis via diffusion models.  _arXiv preprint arXiv:2303.11396_ , 2023.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1290–1299, 2022.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * Eftekhar et al. [2021] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir.  Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10786–10796, 2021.
  * Goel et al. [2022] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan.  Interactive segmentation of radiance fields.  _arXiv preprint arXiv:2212.13545_ , 2022.
  * Gropp et al. [2020] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In _International Conference on Machine Learning_ , pages 3789–3799. PMLR, 2020.
  * Guo et al. [2022] Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou.  Neural 3d scene reconstruction with the manhattan-world assumption.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5511–5520, 2022.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Huang et al. [2023] Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams.  Neural kernel surface reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4369–4379, 2023.
  * Jiang et al. [2020] Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser.  Local implicit grid representations for 3d scenes.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Kajiya and Von Herzen [1984] James T Kajiya and Brian P Von Herzen.  Ray tracing volume densities.  _ACM SIGGRAPH computer graphics_ , 18(3):165–174, 1984.
  * Kerr et al. [2023] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  Lerf: Language embedded radiance fields.  In _International Conference on Computer Vision (ICCV)_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kobayashi et al. [2022] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing nerf for editing via feature field distillation.  _arXiv preprint arXiv:2205.15585_ , 2022.
  * Kong et al. [2023] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison.  vmap: Vectorised object mapping for neural field slam.  _arXiv preprint arXiv:2302.01838_ , 2023.
  * Kundu et al. [2022] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J. Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12871–12881, 2022.
  * Li et al. [2022] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  _arXiv preprint arXiv:2201.03546_ , 2022.
  * Li et al. [2023] Zizhang Li, Xiaoyang Lyu, Yuanyuan Ding, Mengmeng Wang, Yiyi Liao, and Yong Liu.  Rico: Regularizing the unobservable for indoor compositional reconstruction.  In _ICCV_ , 2023.
  * Liu et al. [2023] Jiahui Liu, Chirui Chang, Jianhui Liu, Xiaoyang Wu, Lan Ma, and Xiaojuan Qi.  Mars3d: A plug-and-play motion-aware model for semantic segmentation on multi-scan 3d point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9372–9381, 2023.
  * Lyu et al. [2023] Xiaoyang Lyu, Peng Dai, Zizhang Li, Dongyu Yan, Yi Lin, Yifan Peng, and Xiaojuan Qi.  Learning a room with the occ-sdf hybrid: Signed distance function mingled with occupancy aids scene representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8940–8950, 2023.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, 2022.
  * Oechsle et al. [2021] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5589–5599, 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Peng et al. [2020] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In _European Conference on Computer Vision_ , pages 523–540. Springer, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. [2016] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise View Selection for Unstructured Multi-View Stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Siddiqui et al. [2022] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder.  Panoptic lifting for 3d scene understanding with neural fields.  _arXiv preprint arXiv:2212.09802_ , 2022.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Tschernezki et al. [2022] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi.  Neural feature fusion fields: 3d distillation of self-supervised 2d image representations.  _arXiv preprint arXiv:2209.03494_ , 2022.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.
  * Wang et al. [2022a] Bing Wang, Lu Chen, and Bo Yang.  Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images.  _arXiv preprint arXiv:2208.07227_ , 2022a.
  * Wang et al. [2022b] Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang.  Neuris: Neural reconstruction of indoor scenes using normal priors.  _arXiv preprint arXiv:2206.13597_ , 2022b.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021.
  * Wu et al. [2022] Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng.  Object-compositional neural implicit surfaces.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVII_ , pages 197–213. Springer, 2022.
  * Wu et al. [2023] Qianyi Wu, Kaisiyuan Wang, Kejie Li, Jianmin Zheng, and Jianfei Cai.  Objectsdf++: Improved object-compositional neural implicit surfaces.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 21764–21774, 2023.
  * Xiangli et al. [2023] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Bo Dai, and Dahua Lin.  Assetfield: Assets mining and reconfiguration in ground feature plane representation.  _arXiv preprint arXiv:2303.13953_ , 2023.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 34:4805–4815, 2021.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Yu et al. [2022] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  _arXiv preprint arXiv:2206.00665_ , 2022.
  * Yuille and Coughlan [1999] A. Yuille and J. M. Coughlan.  Manhattan world: Compass direction from a single image by bayesian inference.  In _Computer Vision, IEEE International Conference on Computer Vision_ , page 941, Los Alamitos, CA, USA, 1999. IEEE Computer Society.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15838–15847, 2021.
  * Zhu et al. [2022] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12786–12796, 2022.