#  Deciphering the $\sim 18$ TeV photons from GRB 221009A

Sarira Sahu<EMAIL_ADDRESS>Instituto de Ciencias Nucleares,
Universidad Nacional Autónoma de México,
Circuito Exterior, C.U., A. Postal 70-543, 04510 Mexico DF, Mexico  B. Medina-
Carrillo<EMAIL_ADDRESS>Departamento de Física Aplicada, Centro
de Investigación y de Estudios Avanzados del IPN,
Unidad Mérida. A.P. 73, Cordemex, Mérida, Yucatán 97310, México  G. Sánchez-
Colón<EMAIL_ADDRESS>Departamento de Física Aplicada, Centro de
Investigación y de Estudios Avanzados del IPN,
Unidad Mérida. A.P. 73, Cordemex, Mérida, Yucatán 97310, México  Subhash
Rajpoot<EMAIL_ADDRESS>Department of Physics and Astronomy,
California State University,
1250 Bellflower Boulevard, Long Beach, CA 90840, USA

###### Abstract

On 9 October, 2022, an extremely powerful gamma-ray burst, GRB 221009A, was
detected by several instruments. Despite being obstructed by the Milky Way
galaxy, its afterglow outburst outshone all other GRBs seen before. LHAASO
detected several thousands very high energy photons extending up to 18 TeV.
Detection of such energetic photons are unexpected due to the large opacity of
the Universe. It is possible that in the afterglow epoch the intrinsic very
high energy photon flux from the source might have increased manifolds, which
could compensate the attenuation by pair-production with the extragalactic
background light. We propose such a scenario and show that very high energy
photons can be observed on the Earth from the interaction of very high energy
protons with the seed synchrotron photons in the external forward shock region
of the GRB jet.

Particle astrophysics (96), Blazars (164), Gamma-ray bursts (629),
Relativistic jets (1390)

mainReferences appendixReferences (Appendix)

##  1 Introduction

On October 9, 2022, at T0 = 13 : 16 : 59.000 UT (Veres et al., 2022), a long
duration gamma-ray burst (GRB), identified as GRB 221009A (also known as Swift
J1913.1+1946) was detected in the direction of the constellation Sagitta by
the Gamma-ray Burst Monitor (GBM) (Meegan et al., 2009) onboard the Fermi
Gamma-ray Space Telescope. The prompt emission was also detected by several
other space observatories, such as the Fermi Large Area Telescope (LAT), Swift
(S.Dichiara et al., 2022; Krimm et al., 2022), AGILE (Ursi et al., 2022; Piano
et al., 2022), INTEGRAL (Gotz et al., 2022), Solar Orbiter (Xiao et al.,
2022), SRG (Lapshov et al., 2022), Konus (Frederiks et al., 2022), GRBAlpha
(Ripa et al., 2022), and STPSat-6 (Mitchell et al., 2022). GRB 221009A is
located at the coordinate RA = 288.282 and Dec = 19.495 (Pillera et al.,
2022). The Fermi-LAT detected the most energetic photon of energy 99.3 GeV (at
$t_{0}+240$ s). It is the highest energy photon ever detected by Fermi-LAT
from a GRB in the prompt phase (Bissaldi et al., 2022; Pillera et al., 2022).
The afterglow emission was also observed at different wavelengths (Das &
Razzaque, 2022), and the optical follow-up observation estimated a very low
redshift of $z\simeq 0.151$ (de Ugarte Postigo et al., 2022). The total
emitted isotropic-equivalent gamma-ray energy from GRB 221009A is estimated to
be $(2-6)\times 10^{54}$ erg (de Ugarte Postigo et al., 2022; Kann & Agui,
2022). This is the brightest, long-duration GRB, and arguably, one of the
nearest, and possibly, the most energetic GRB ever observed. It has also been
reported that GRB 221009A produced a significant ionization of the Earth’s
lower ionosphere ($\sim 60-100$ km) (Hayes & Gallagher, 2022) and is the
strongest ionization effect ever recorded from a GRB.

The Large High Altitude Air Shower Observatory (LHAASO) with the water
Cherenkov detector array (WCDA) and the larger air shower kilometer square
area (KM2A) detector observed more than 5000 very high energy (VHE) photons
within $T_{0}+2000$ s in the 500 GeV to 18 TeV energy range, making them the
most energetic photons ever observed from a GRB (Huang et al., 2022).
Surprisingly, the ground-based Cherenkov detector Carpet-2 at Baksan Neutrino
Observatory reported the detection of undoubtedly a very rare air-shower
originating from a 251 TeV photon 4536 s after the GBM trigger from the
direction of the GRB 221009A (Dzhappuev et al., 2022). Observations of these
unusually VHE gamma-rays by LHAASO and Carpet-2 from GRB 221009A are
incomprehensible, and led to the speculations of non-standard physics
explanations of these observed events. However, there is a caveat concerning
the observation of 251 TeV gamma-ray. The angular resolution of the Carpet-2
is several degrees and the two previously reported Galactic VHE sources, 3HWC
J1928+178 and LHASSO J1929+1745, are located close to the position of the GRB
221009A (Fraija et al., 2022). It remains uncertain whether the observed 251
TeV photon is from the GRB 221009A or from either of these Galactic sources.
Nevertheless, the temporal and spatial coincidence of this event with the GRB
221009A is worth exploring (Finke & Razzaque, 2022; Alves Batista, 2022;
Mirabal, 2022). In the present context, we will delve into the VHE emission
observed by LHAASO.

The VHE $\gamma$-rays observed by the Cherenkov telescopes from the
extragalactic sources undergo energy-dependent attenuation by interacting with
the extragalactic background light (EBL) through electron–positron pair
production (Stecker et al., 1992; Ackermann et al., 2012). As a result, the
shape of the spectrum at very high energies changes significantly. Several
well known EBL models have been developed to study the attenuation at
different redshifts.These models have been used successfully by the the highly
sensitive Imaging Atmospheric Cherenkov Telescopes (IACTs) such as VERITAS
(Holder et al., 2009), HESS (Hinton, 2004), and MAGIC (Cortina, 2005), to
analyze the observed VHE gamma-rays from sources of different redshifts. The
observed VHE gamma-ray flux from the source can be written in terms of the
intrinsic flux $F_{in}$ and the survival probability of the VHE photon as
(Hauser & Dwek, 2001)

$F_{\gamma}(E_{\gamma})=F_{in}(E_{\gamma})\,e^{-\tau_{\gamma\gamma}(E_{\gamma})},$
(1)

where $E_{\gamma}$ is the observed VHE photon energy and $\tau_{\gamma\gamma}$
is the optical depth for the pair-production process. The optical depth for a
18 TeV photon at a redshift of $z=0.151$ is 18.3 in EBL model of Franceschini
et al. (2008) and 19.4 in the EBL model of Dominguez et al. (2011) which
corresponds to the survival probability of the VHE photon
$e^{-\tau_{\gamma\gamma}}\sim 1.1\times 10^{-8}$ and $3.6\times 10^{-9}$
respectively in both these models. Thus, for a 18 TeV photon energy, the
observed flux will be suppressed by a factor of $\sim 10^{-9}-10^{-8}$. As the
observation of 18 TeV photon from a source at redshift $z=0.151$ is difficult
to comprehend, it is viewed as signature of new physics such as Lorentz
invariance violation (Zhu & Ma, 2022; Li & Ma, 2022; Baktash et al., 2022;
Finke & Razzaque, 2022), oscillation of photon to a pseudo-scalar particle
(axion-like particle (ALP)) (Galanti et al., 2022; Lin & Yanagida, 2022;
Troitsky, 2022), ALP abundance enhanced with its mass caused by a first-order
phase transition in a hidden sector (Nakagawa et al., 2022), heavy neutrino as
the means of propagation to avoid the energy attenuation (Cheung, 2022), and
sterile neutrinos produced via mixing with active neutrinos (Brdar & Li,
2022). On the other hand, from the standard physics point of view, these
gamma-rays are argued to be the secondaries arising from the interactions
between the ultra-high energy cosmic rays emanating from GRB 221009A and the
cosmological photon background on their way to the Earth (Alves Batista,
2022). Also, observation of neutrinos from such a bright GRB is discussed
(Murase et al., 2022).

Since the VHE spectra of most of the flaring high energy blazars (HBLs) of
different redshifts are explained very well using the EBL models of
Franceschini et al. (2008) and Dominguez et al. (2011), the recent observation
of $\sim 18$ TeV photon from the GRB 221009A falls short of this expectation.
The obvious question is, can it be due to the intrinsic flux from the source?
If we look into Eq.(1), the depletion in the flux due to
$e^{-\tau_{\gamma\gamma}}$ can, in principle, be compensated by increasing the
intrinsic flux. However, this may not be possible in most of the situations.
As noted previously, GRB 221009A is very special as its afterglow outburst
outshone all other GRBs seen before, despite the fact that GRB 221009A is
obstructed by the Milky Way galaxy. Furthermore, the burst was so powerful
that it ionized Earth’s atmosphere and disrupted long wave radio
communications. It is estimated that, at low redshifts, such energetic GRBs
are extremely rare events and may occur once in a century (Atteia, 2022).
Thus, it is possible that the intrinsic VHE flux from the source might have
increased manifolds, which could compensate the depletion from the EBL effect.
In this letter, we would like to pursue such a scenario and its impact on the
observation of $\sim 18$ TeV photons by LHAASO.

##  2 Common features of Blazar and GRB

The emission mechanisms in blazars (a subclass of active galactic nuclei
(AGN)) and GRBs have many features in common (Urry & Padovani, 1995; Gehrels &
Razzaque, 2013). Such common features are found to prevail in the synchrotron
luminosity and Doppler factor between GRBs and active galactic nuclei (AGNs)
(Wu et al., 2011). In several studies it was observed that the jets in blazars
and GRBs share common features despite large differences in their masses and
bulk Lorentz factors (Nemmen et al., 2012; Wang & Wei, 2011; Wu et al., 2016).
It is natural to use such mechanisms and processes to study the multi-TeV
flaring of high energy blazars to study the afterglow phases of GRBs.

Previously we have used the photohadronic process to study the multi-TeV
flaring from HBLs (Sahu, 2019; Sahu et al., 2019, 2020). In the photohadronic
scenario, protons in the blazar jet are accelerated to very high energies and
then collide with the background seed photons to produce $\Delta$-resonance
($p\gamma\rightarrow\Delta^{+}$) with the following kinematical condition
(Sahu, 2019)

$E_{p}\epsilon_{\gamma}=0.32\,\Gamma\,{\cal D}(1+z)^{-2}\,\mathrm{{GeV}^{2}},$
(2)

where $E_{p}$ and $\epsilon_{\gamma}$ are the proton energy and the background
seed photon energy respectively in the observer’s frame. In the process, the
observed VHE photon carries about 10% of the proton energy, $E_{\gamma}\simeq
0.1Ep$. The bulk Lorentz factor and the Doppler factor respectively are given
by $\Gamma$ and ${\cal D}$. As the jets of the observed HBLs and GRBs beam
towards us, $\Gamma\simeq{\cal D}$. The $\Delta$-resonances decay to neutral
pions that subsequently decay to VHE gamma-rays. These are the blueshifted
photons observed by the Cherenkov telescopes on Earth. This model is very
successful in explaining the VHE gamma-ray spectra from several HBLs, and the
intrinsic flux $F_{\mathrm{in}}$ is given by

$F_{in}=F_{0}\,E^{-\delta+3}_{\gamma,TeV},$ (3)

where $E_{\gamma,TeV}$ is the photon energy in TeVs. The normalization
constant $F_{0}$ can be fixed from the observed spectrum and the spectral
index $\delta=\alpha+\beta$ is the free parameter in the model (Sahu, 2019;
Sahu et al., 2019). Note that $F_{in}$ is independent of $\Gamma$ and $\cal
D$. The high energy protons in the jet have a power-law differential spectrum
$dN/dE_{p}\propto E^{-\alpha}_{p}$, $E_{p}$ is the proton energy and we take
$\alpha=2$ (Dermer & Schlickeiser, 1993), a generally accepted value. For
HBLs, the seed photon flux also follows a power-law
$\Phi_{\gamma}\propto\epsilon^{\beta}_{\gamma}\propto E^{-\beta}_{\gamma}$
(Sahu, 2019; Sahu et al., 2019). For HBLs, the value of $\delta$ always lies
in the range $2.5\leq\delta\leq 3.0$, which corresponds to a $\beta$ value in
the range $0.5\leq\beta\leq 1.0$, indicating that the seed photons are in the
low energy tail region of the SSC spectrum (Sahu et al., 2019). Recently, it
has been shown that for GRBs, the value of $\beta$ can be positive or negative
(Sahu & Fortín, 2020). $\beta>0$ implies that the seed photons are in the
self-Compton regime. $\beta<0$ locates seed photons in the synchrotron regime.
It was previously shown that the VHE spectra of the GRB 190114C and GRB
190829A are due to the interaction of the high energy protons with the low
energy tail region of the background synchrotron self Compton (SSC) photons in
the jet with $\beta>0$ (Sahu & López Fortín, 2020; Sahu et al., 2022). Also
shown there was that the VHE spectrum of GRB 180720B is from the interaction
of high energy protons with the synchrotron seed photons in the jet
environment with $\beta<0$ (Sahu & López Fortín, 2020). This negative value of
$\beta$ corresponds to the falling part of the synchrotron spectrum.

##  3 Results

LHAASO, with its two detectors WCDA and KM2A, detected $\geq 5000$ photons
above 500 GeV from the GRB 221009A within $T\sim 2000$ s of the prompt
emission. The number of photons $N_{\gamma}$ detected at a time interval $T$
by any of these detectors at zenith angle $\theta$ and effective area
$A(E_{\gamma},\theta)$ is (Zhao et al., 2022)

$N_{\gamma}=T\int_{0.5\,TeV}\frac{dN_{\gamma}}{dE_{\gamma}}A(E_{\gamma},\theta)\,e^{-\tau_{\gamma}(E_{\gamma})}dE_{\gamma},$
(4)

where the differential photon spectrum can be written as

$\frac{dN_{\gamma}}{dE_{\gamma}}=F_{0}\,E^{-\delta+1}_{\gamma,TeV}\,TeV^{-2}.$
(5)

The source was observed at a zenith angle of $30^{\circ}\lesssim\theta\lesssim
35^{\circ}$ that we adopt in Eq.(4). Taking into account the areas of LHAASO-
WCDA and LHAASO-KM2A (Cao et al., 2022), we evaluate the integral in Eq.(4)
for $\delta=2.5,\,1.7$ and $1.2$. For the present analysis, we consider the
EBL model of Franceschini et al. (2008). We assume that these two detectors
observe photons above 500 GeV in the range $5000\leq N_{\gamma}\leq 6500$. By
fixing the value of $N_{\gamma}$, we calculate the value of $F_{0}$ which is
then used to calculate the VHE photon flux and the integrated flux
$F^{int}_{\gamma}$ in the energy range $100\,\mathrm{GeV}\leq E_{\gamma}\leq
18\,\mathrm{TeV}$.

In Fig. 1 we have shown the predicted spectra for $\delta=2.5,1.7$ and $1.2$
by taking into account the effective area of the LHASSO-WCDA detector and
fixing $N_{\gamma}=5500$. The relative energy resolution of LHAASO-WCDA is
$\simeq 50\%$ at energies around 18 TeV (Fig. 26 of Chapter 1 of Cao et al.
(2022)). For $\delta=2.5$ the flux starts from a maximum value of
$F_{\gamma}\sim 10^{-8}\,\mathrm{erg\,cm^{-2}\,s^{-1}}$ at $E_{\gamma}=100$
GeV, and decreases slowly up to $\sim 4$ TeV. Beyond $\sim 4$ TeV it falls
faster due to the EBL effect. The spectrum intersects with the sensitivity
curve of LHAASO with 2000 s exposure at $E_{cut}=9.94$ TeV which is at the
lower edge of the energy resolution (at 9 TeV). The $\delta=2.5$ value implies
$\beta=0.5$ with the intrinsic flux $F_{\mathrm{in}}\propto
E^{0.5}_{\gamma,TeV}$. This corresponds to seed photons in the lower tail
region of the SSC spectrum in the GRB jet. The accelerated high energy protons
in the jet interact with these seed photons to produce VHE gamma-rays, a
situation very similar to the VHE flaring of HBLs.

We repeat the calculation for $\delta=1.7$ which corresponds to $\beta=-0.3$.
As discussed previously, negative value of the seed photon spectral index
$\beta$ corresponds to photons in the descending part of the synchrotron
spectrum towards higher $\epsilon_{\gamma}$ values and
$\Phi_{\gamma}\propto\epsilon^{-0.3}_{\gamma}$. Thus, in this case, the high
energy protons interact with the seed photons in the synchrotron regime of the
external forward shock region to produce gamma-rays. The spectrum starts with
$F_{\gamma}\sim 10^{-9}\mathrm{erg\,cm^{-2}\,s^{-1}}$ at $E_{\gamma}=100$ GeV
and increases very slowly up to $\sim 4$ TeV and then falls faster as the
exponentially decaying term from the EBL dominates. The curve intersects with
the LHAASO sensitivity curve at $E_{cut}=11.53$ TeV. The intrinsic flux
increases as $F_{\mathrm{in}}\propto E^{1.3}_{\gamma,TeV}$.

Finally, we consider a smaller value of $\delta=1.2$ which is shown in Fig. 1.
This value of $\delta$ gives $\beta=-0.8$. In the photohadronic context, this
corresponds to $\Phi_{\gamma}\propto\epsilon^{-0.8}_{\gamma}$ which is the
descending part of the synchrotron spectrum towards higher $\epsilon_{\gamma}$
values like the ones for $\delta=1.7$. However, in this case, the seed
synchrotron spectrum in the external forward shock region falls faster than
the one for $\delta=1.7$. The spectrum increases and reaches a maximum flux at
$E_{\gamma}\sim 4.5$ TeV, and then decreases exponentially for large values of
$E_{\gamma}$ intersecting the LHAASO curve at $E_{cut}=12.44$ TeV. The
intrinsic flux in this case behaves like $E^{1.8}_{\gamma,TeV}$.

Figure 1: Using the effective area of the detector LHAASO-WCDA the VHE Spectrum for GRB 221009A is given for different values of the spectral index $\delta$, by fixing $N_{\gamma}=5500$. The intrinsic flux for each $\delta$ also shown. The LHAASO sensitivity curve for with 2000 s exposure is also shown. The vertical line corresponds to 18 TeV photon energy. The shaded region is $\pm 50\%$ relative energy resolution of LHAASO-WCDA for $E_{\gamma}\simeq 18$ TeV.  Figure 2: This is same as Fig. 1 but using the detector area of LHAASO-KM2A and the shaded region here is $\pm 36\%$ relative energy resolution of LHAASO-KM2A for $E_{\gamma}\simeq 18$ TeV. Table 1: Using the LHAASO-WCDA ($30^{\circ}\leq\theta\leq 45^{\circ}$) and LHAASO-KM2A effective detector areas and different values of $\delta$ and number of events, $N_{\gamma}$, we have calculated the flux normalization factor $F_{0}$ in units of $10^{-8}\mathrm{erg\,cm^{-2}\,s^{-1}}$, the integrated flux $F^{int}_{\gamma}$ in units of $10^{-8}\mathrm{erg\,cm^{-2}\,s^{-1}}$, in the energy range 100 GeV to 18 TeV along with the corresponding luminosity $L_{\gamma,48}$ in units of $10^{48}\,\mathrm{erg\,s^{-1}}$. $E_{cut}$ is the value of $E_{\gamma}$ in TeV unit where it intersects with the LHAASO sensitivity curve with 2000 s exposure time. The bracketed values are the results using the LHAASO-KM2A detector area.  $\delta$ | $N_{\gamma}$ | $F_{0}$ | $F^{int}_{\gamma}$ | $L_{\gamma,48}$ | $E_{cut}$
---|---|---|---|---|---
2.5 | 5500 | 2.11 (0.54) | 2.49 (0.63) | 1.63 (0.41) | 9.94 (8.17)
| 6500 | 2.50 (0.63) | 2.95 (0.75) | 1.92 (0.49) | 10.18 (8.41)
1.7 | 5500 | 1.41 (0.48) | 1.22 (0.41) | 0.80 (0.27) | 11.53 (10.48)
| 6500 | 1.67 (0.56) | 1.44 (0.49) | 0.94 (0.32) | 11.70 (10.63)
1.2 | 5500 | 0.92 (0.36) | 1.07 (0.42) | 0.70 (0.27) | 12.44 (11.32)
| 6500 | 1.08 (0.42) | 1.26 (0.50) | 0.83 (0.32) | 12.55 (11.50)

We repeat the calculation by using the effective area of LHAASO-KM2A for
$\delta=2.5,\,1.7,\,1.2$ and $N_{\gamma}=5500$. The results are shown in Fig
2. For 18 TeV photons the relative energy resolution of LHAASO-KM2A is $\simeq
36\%$ (Fig. 2, Chapter 1 of Cao et al. (2022)) which put the observed photon
energy in the range $11.52$ TeV to $24.48$ TeV. For a given $\delta$, both
LHAASO-WCDA and LHAASO-KM2A spectra the pattern is similar but the $E_{cut}$
value for LHAASO-KM2A is smaller than that of LHAASO-WCDA. Also, the $E_{cut}$
for LHAASO-KM2A is less than $11.52$ TeV which shows that LHAASO-KM2A may not
be able to detect these photons.

To account for the behavior of the VHE spectrum for different $N_{\gamma}$, we
fix $N_{\gamma}=5500$ and $6500$ to calculate $F_{0}$, the integrated flux
$F^{int}_{\gamma}$, the luminosity $L_{\gamma}$ and $E_{cut}$ values using the
effective areas of the detectors LHAASO-WCDA and LHAASO-KM2A detectors. These
are given in Table 1. The results of LHAASO-KM2A are the bracketed values in
the table. It can be seen that increasing $N_{\gamma}$ from $5500$ to $6500$
leads to increament in all quantities. This implies that by knowing
$N_{\gamma}$ and the maximum value of $E_{\gamma}$ we can predict the VHE
gamma-ray spectrum, provided the EBL contribution is well understood.

For a given value of $N_{\gamma}$, the $E_{cut}$ value increases and
approaches $\sim 18$ TeV as $\delta$ decreases from 2.5 to 1.2. Moreover, by
further decreasing $\delta$, one can reach $E_{cut}\sim 18$ TeV which
corresponds to a very stiff synchrotron spectrum and may be problematic. Also,
for a given $\delta$, by increasing $N_{\gamma}$, the $E_{cut}$ value
increases. From our analysis we observed that, the LHAASO-WCDA is more likely
to observe photons of energy $\sim 18$ TeV than the LHAASO-KM2A. From the
dependence of $E_{cut}$ on $\delta$ we infer that the interaction of high
energy protons with the descending part of the synchroton seed photon spectrum
is more likely to produce $\sim 18$ TeV photons than the high energy protons
interaction with the low energy tail region of the seed SSC photons in the GRB
jet.

Additionally, we calculate the chance probability of $N_{\gamma}\geq 1$ for
$E_{\gamma}\sim 18$ TeV by taking the normalization constant $F_{0}$ as a
variable in the range $10^{-10}\leq F_{0}\,(\mathrm{erg\,cm^{-2}\,s^{-1}})\leq
2\times 10^{-7}$ for a fixed $\delta=1.2$. We fix the total number of observed
photons above 500 GeV to be 5500 and $T=2000$ s. We assume that the noise in
the data has a Gaussian distribution with an unknown standard deviation of
$\sigma$ and variance $\sigma^{2}$ (Gregory, 2010). Bayesian inference is
implemented by using the Markov chain Monte Carlo (MCMC) method to estimate
the posterior probability distribution function (PDF) as a function of $F_{0}$
for a given $\sigma$ value. This we have done for $\sigma$ in the range
$100\leq\sigma\leq 1500$. Using the PDFs for different $\sigma$ values and
$E_{\gamma}\sim 18$ TeV we evaluate

$N_{\gamma}(E_{\gamma})=T\int^{(1+\delta_{E}/2)E_{\gamma}}_{(1-\delta_{E}/2)E_{\gamma}}\frac{dN_{\gamma}}{dE^{\prime}_{\gamma}}A(E^{\prime}_{\gamma},\theta)\,e^{-\tau_{\gamma}(E^{\prime}_{\gamma})}dE^{\prime}_{\gamma},$
(6)

using the Monte Carlo simulation. Here $\delta_{E}=0.5$ is the value for the
$50\%$ uncertainty in the energy around 18 TeV for the LHAASO-WCDA. For a
given value of $\sigma$ we repeat the procedure for $10^{6}$ times. This way
we obtain the percentage of chance probability of $N_{\gamma}\geq
1=10^{-4}\times\text{No\, of\, times\,}N_{\gamma}(18\,TeV)\geq 1$. The results
are plotted in Fig. 3. It is observed that if the posterior PDF is symmetric
around the mid-point, then the percentage of chance probability of detection
of $N_{\gamma}\geq 1$ is very small. Similarly, for an asymmetric PDF with the
weight factor leaning more towards smaller values of $F_{0}$, the percentage
of chance probability is also very small. However, for the asymmetric PDFs
with the weight factor leaning more towards larger values of $F_{0}$, the
percentage of chance probability is large with a maximum value of 40%.

Figure 3: The percentage of chance probability for $N_{\gamma}\geq 1$ for
$E_{\gamma}=18$ TeV is plotted as a function of $\sigma$.

##  4 Conclusion

In conclusion, the recent observation of $\sim 18$ TeV photons by LHAASO from
GRB 221009A sheds doubts on the applicability of the well-known EBL models for
photons of energy $>10$ TeV at a redshift of $\gtrsim 0.151$ even though these
EBL models work very well in explaining the VHE spectra of so many other TeV
sources. This incompatibility has led towards new physics solutions. However,
there is still a conventional way to delve into the problem, which we proposed
here. We argue that high energy protons interacting with the synchrotron
photon background in the GRB jet will be able to produce photons of energy
close to 18 TeV. Assuming that the error in the data has a Gaussian
distribution and using the area of LHAASO-WCDA, we obtain maximum 40% chance
probability of observing $N_{\gamma}\geq 1$ for $E_{\gamma}\sim 18$ TeV. Our
analysis shows that LHAASO-WCDA is more likely to observe photons of energy
$\sim 18$ TeV than the LHAASO-KM2A. We anticipate that the publication of GRB
221009A results should be able to either confirm or rule out either most or
all of the explanations discussed here.

The authors are indebted to Miguel Enrique Iglesias Martínez and Jose Guerra
Carmenate for helping in the Bayesian analysis. We are also thankful to the
anonymous referee for making useful suggestions. S.S. is thankful to S.
Salokya for reading the manuscript. The work of S.S. is partially supported by
DGAPA-UNAM (Mexico) Projects No. IN103522. B. M-C and G. S-C would like to
thank CONACyT (México) for partial support. Partial support from CSU-Long
Beach is gratefully acknowledged.

## References

## References

  * Ackermann et al. (2012) Ackermann, M., Ajello, M., Allafort, A., et al. 2012, Science, 338, 1190, doi: 10.1126/science.1227160
  * Alves Batista (2022) Alves Batista, R. 2022.  https://arxiv.org/abs/2210.12855
  * Atteia (2022) Atteia, J.-L. 2022, GCN Circ. 32793
  * Baktash et al. (2022) Baktash, A., Horns, D., & Meyer, M. 2022, arXiv e-prints, arXiv:2210.07172.  https://arxiv.org/abs/2210.07172
  * Bissaldi et al. (2022) Bissaldi, E., Omodei, N., & Kerr, M. 2022, GCN Circ. 32637
  * Brdar & Li (2022) Brdar, V., & Li, Y.-Y. 2022.  https://arxiv.org/abs/2211.02028
  * Cao et al. (2022) Cao, Z., et al. 2022, Chin. Phys. C, 46, 035001.  https://arxiv.org/abs/1905.02773
  * Cheung (2022) Cheung, K. 2022.  https://arxiv.org/abs/2210.14178
  * Cortina (2005) Cortina, J. 2005, Astrophys. Space Sci., 297, 245, doi: 10.1007/s10509-005-7627-5
  * Das & Razzaque (2022) Das, S., & Razzaque, S. 2022.  https://arxiv.org/abs/2210.13349
  * de Ugarte Postigo et al. (2022) de Ugarte Postigo, A., Izzo, L., Pugliese, G., et al. 2022, GCN Circ. 32648
  * Dermer & Schlickeiser (1993) Dermer, C. D., & Schlickeiser, R. 1993, ApJ, 416, 458, doi: 10.1086/173251
  * Dominguez et al. (2011) Dominguez, A., et al. 2011, MNRAS, 410, 2556, doi: 10.1111/j.1365-2966.2010.17631.x
  * Dzhappuev et al. (2022) Dzhappuev, D. D., Afashokov, Y. Z., Dzaparova, I. M., et al. 2022, The Astronomer’s Telegram, 15669, 1
  * Finke & Razzaque (2022) Finke, J. D., & Razzaque, S. 2022, arXiv e-prints, arXiv:2210.11261.  https://arxiv.org/abs/2210.11261
  * Fraija et al. (2022) Fraija, N., Gonzalez, M., & HAWC Collaboration. 2022, The Astronomer’s Telegram, 15675, 1
  * Franceschini et al. (2008) Franceschini, A., Rodighiero, G., & Vaccari, M. 2008, A&A, 487, 837, doi: 10.1051/0004-6361:200809691
  * Frederiks et al. (2022) Frederiks, D., A.Lysenko, Ridnaia, A., et al. 2022, GCN Circ. 32668
  * Galanti et al. (2022) Galanti, G., Roncadelli, M., & Tavecchio, F. 2022, arXiv e-prints, arXiv:2210.05659.  https://arxiv.org/abs/2210.05659
  * Gehrels & Razzaque (2013) Gehrels, N., & Razzaque, S. 2013, Front. Phys. (Beijing), 8, 661, doi: 10.1007/s11467-013-0282-3
  * Gotz et al. (2022) Gotz, D., Mereghetti, S., Savchenko, V., et al. 2022, GCN Circ. 32660
  * Gregory (2010) Gregory, P. 2010, Bayesian Logical Data Analysis for the Physical Sciences, Cambridge University Press, UK
  * Hauser & Dwek (2001) Hauser, M. G., & Dwek, E. 2001, Ann. Rev. Astron. Astrophys., 39, 249, doi: 10.1146/annurev.astro.39.1.249
  * Hayes & Gallagher (2022) Hayes, L. A., & Gallagher, P. T. 2022, Research Notes of the AAS, 6, 222, doi: 10.3847/2515-5172/ac9d2f
  * Hinton (2004) Hinton, J. A. 2004, New Astron. Rev., 48, 331, doi: 10.1016/j.newar.2003.12.004
  * Holder et al. (2009) Holder, J., et al. 2009, AIP Conf. Proc., 1085, 657, doi: 10.1063/1.3076760
  * Huang et al. (2022) Huang, Y., Hu, S., Chen, S., et al. 2022, GCN Circ. 32677
  * Kann & Agui (2022) Kann, D. A., & Agui, J. F. 2022, GCN Circ. 32762
  * Krimm et al. (2022) Krimm, H. A., Barthelmy, S. D., Dichiara, S., et al. 2022, GCN Circ. 32688
  * Lapshov et al. (2022) Lapshov, I., Molkov, S., Mereminsky, I., et al. 2022, GCN Circ. 32663
  * Li & Ma (2022) Li, H., & Ma, B.-Q. 2022, arXiv e-prints, arXiv:2210.06338.  https://arxiv.org/abs/2210.06338
  * Lin & Yanagida (2022) Lin, W., & Yanagida, T. T. 2022, arXiv e-prints, arXiv:2210.08841.  https://arxiv.org/abs/2210.08841
  * Meegan et al. (2009) Meegan, C., Lichti, G., Bhat, P. N., et al. 2009, ApJ., 702, 791, doi: 10.1088/0004-637X/702/1/791
  * Mirabal (2022) Mirabal, N. 2022, arXiv e-prints, arXiv:2210.14243.  https://arxiv.org/abs/2210.14243
  * Mitchell et al. (2022) Mitchell, L. J., Phlips, B. F., Johnson, W., et al. 2022, GCN Circ. 32746
  * Murase et al. (2022) Murase, K., Mukhopadhyay, M., Kheirandish, A., Kimura, S. S., & Fang, K. 2022, arXiv e-prints, arXiv:2210.15625.  https://arxiv.org/abs/2210.15625
  * Nakagawa et al. (2022) Nakagawa, S., Takahashi, F., Yamada, M., & Yin, W. 2022.  https://arxiv.org/abs/2210.10022
  * Nemmen et al. (2012) Nemmen, R. S., Georganopoulos, M., Guiriec, S., et al. 2012, Science, 338, 1445, doi: 10.1126/science.1227416
  * Piano et al. (2022) Piano, G., Verrecchia, F., Bulgarelli, A., et al. 2022, GCN Circ. 32657
  * Pillera et al. (2022) Pillera, R., Bissaldi, E., Omodei, N., et al. 2022, GCN Circ. 32658
  * Ripa et al. (2022) Ripa, J., Pal, A., Werner, N., et al. 2022, GCN Circ. 32685
  * Sahu (2019) Sahu, S. 2019, Rev. Mex. Fis., 65, 307, doi: 10.31349/revmexfis.65.307
  * Sahu et al. (2019) Sahu, S., Fortín, C. E. L., & Nagataki, S. 2019, ApJ., 884, L17, doi: 10.3847/2041-8213/ab43c7
  * Sahu & Fortín (2020) Sahu, S., & Fortín, C. E. L. 2020, ApJL., 895, L41, doi: 10.3847/2041-8213/ab93da
  * Sahu & López Fortín (2020) Sahu, S., & López Fortín, C. E. 2020, ApJL., 895, L41, doi: 10.3847/2041-8213/ab93da
  * Sahu et al. (2020) Sahu, S., López Fortín, C. E., Iglesias Martínez, M. E., Nagataki, S., & Fernández de Córdoba, P. 2020, MNRAS., 492, 2261, doi: 10.1093/mnras/staa023
  * Sahu et al. (2022) Sahu, S., Polanco, I. A. V., & Rajpoot, S. 2022, ApJ., 929, 70, doi: 10.3847/1538-4357/ac5cc6
  * S.Dichiara et al. (2022) S.Dichiara, Gropp, J. D., Kennea, J., et al. 2022, GCN Circ. 32632
  * Stecker et al. (1992) Stecker, F. W., de Jager, O. C., & Salamon, M. H. 1992, ApJ, 390, L49, doi: 10.1086/186369
  * Troitsky (2022) Troitsky, S. V. 2022, arXiv e-prints, arXiv:2210.09250.  https://arxiv.org/abs/2210.09250
  * Urry & Padovani (1995) Urry, C. M., & Padovani, P. 1995, Publications of the Astronomical Society of the Pacific, 107, 803, doi: 10.1086/133630
  * Ursi et al. (2022) Ursi, A., Panebianco, G., Pittori, C., et al. 2022, GCN Circ. 32650
  * Veres et al. (2022) Veres, P., Burns, E., Bissaldi, E., et al. 2022, GCN Circ. 32636
  * Wang & Wei (2011) Wang, J., & Wei, J. Y. 2011, ApJ, 726, L4, doi: 10.1088/2041-8205/726/1/L4
  * Wu et al. (2016) Wu, Q., Zhang, B., Lei, W.-H., et al. 2016, Mon. Not. Roy. Astron. Soc., 455, L1, doi: 10.1093/mnrasl/slv136
  * Wu et al. (2011) Wu, Q., Zou, Y.-C., Cao, X., Wang, D.-X., & Chen, L. 2011, ApJL., 740, L21, doi: 10.1088/2041-8205/740/1/L21
  * Xiao et al. (2022) Xiao, H., Krucker, S., & Daniel, R. 2022, GCN Circ. 32661
  * Zhao et al. (2022) Zhao, Z.-C., Zhou, Y., & Wang, S. 2022.  https://arxiv.org/abs/2210.10778
  * Zhu & Ma (2022) Zhu, J., & Ma, B.-Q. 2022, arXiv e-prints, arXiv:2210.11376.  https://arxiv.org/abs/2210.11376


# Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation
without Manual Labels

###### Abstract

Current 3D scene segmentation methods are heavily dependent on manually
annotated 3D training datasets. Such manual annotations are labor-intensive,
and often lack fine-grained details. Importantly, models trained on this data
typically struggle to recognize object classes beyond the annotated classes,
i.e., they do not generalize well to unseen domains and require additional
domain-specific annotations. In contrast, 2D foundation models demonstrate
strong generalization and impressive zero-shot abilities, inspiring us to
incorporate these characteristics from 2D models into 3D models. Therefore, we
explore the use of image segmentation foundation models to automatically
generate training labels for 3D segmentation. We propose Segment3D, a method
for class-agnostic 3D scene segmentation that produces high-quality 3D
segmentation masks. It improves over existing 3D segmentation models
(especially on fine-grained masks), and enables easily adding new training
data to further boost the segmentation performance – all without the need for
manual training labels.

\begin{overpic}[width=496.85625pt,abs,unit=1mm,scale={.25}]{figures/teaser_new.jpg}
\par\put(12.0,114.0){\large Rui Huang${}^{1}$ \quad Songyou Peng${}^{2}$ \quad
Ayça Takmaz${}^{2}$ \quad Federico Tombari${}^{3}$ \quad Marc
Pollefeys${}^{2,4}$ } \put(46.0,108.0){\large Shiji Song${}^{1}$ \quad Gao
Huang${}^{1}$${{}}$ \quad Francis Engelmann${}^{2,3}$} \par\put(44.0,100.0){
${}^{1}$Tsinghua University \hskip 2.84526pt ${}^{2}$ETH Zurich \hskip
2.84526pt ${}^{3}$Google \hskip 2.84526pt ${}^{4}$Microsoft}
\par\put(66.0,94.0){ \href http://segment3d.github.io}
\par\put(42.0,12.0){\parbox{113.81102pt}{\centering\it
Mask3D\cite[cite]{[\@@bibref{}{Schult2023ICRA}{}{}]}\\\ trained on {manual}
labels.\@add@centering}} \put(91.0,13.8){\parbox{128.0374pt}{\centering\it
Segment3D (Ours)\\\ trained on {automatic} labels.\@add@centering}}
\end{overpic}

Figure 1:  Fine-Grained Class-Agnostic 3D Point Cloud Segmentation. Segment3D
predicts highly accurate segmentation masks _(right)_ , improves over state-
of-the-art 3D segmentation methods (e.g., Mask3D [43], _left_), and does not
require manually labeled 3D training data. This is achieved through the
automatic generation of high-quality training masks using foundation models
for image segmentation [25].

**footnotetext: Corresponding author.

##  1 Introduction

In this work, we propose Segment3D, a method for fine-grained class-agnostic
3D segmentation. In particular, dividing the space into coherent segments
aligned with both the scene geometry and its semantics is a key challenge.
This ability to accurately segment and interpret 3D scenes is fundamental for
intelligent assistants, such as autonomous robots or AR/VR devices that can
help vision-impaired people navigate and engage with unknown spaces.

Current methods for 3D indoor-scene understanding mostly focus on semantic
[33, 44, 37, 38, 48, 51] and instance segmentation [43, 30, 8, 29, 49, 15].
These approaches, while effective in benchmarking, have limitations stemming
from their training. Primarily, they depend on extensive manually labeled 3D
training sets that are both time-consuming and challenging to annotate.
Importantly, their performance often deteriorates when applied to scenarios
beyond their training data, limiting their effectiveness in diverse, real-
world scenarios. This becomes particularly apparent under the recently
emerging task of _open-vocabulary_ 3D scene understanding [36, 47, 24, 31]
that aims to segment arbitrary user queries, which naturally go beyond the
pre-defined set of training-set classes. Concurrently, the recent surge in
foundation models, particularly 2D vision-language models [39, 22, 25],
demonstrates remarkable potential. Trained on internet-scale data, these
models exhibit an extraordinary ability to generalize, even in a zero-shot
setting, to new and different input distributions. However, their application
has been predominantly confined to 2D data. For instance, SAM [25] has shown
impressive results in 2D image segmentation, but its applicability to 3D scene
understanding remains mostly unexplored. All these factors give rise to the
interesting and significant research question:

_How to leverage 2D foundation models for class-agnostic 3D scene segmentation
without manually labeled 3D data?_

Recently, SAM3D [53] has proposed a straightforward method that uses posed
RGB-D images corresponding to a 3D scene. They first predict segmentation
masks for RGB-D images with SAM [25]. Subsequently, these 2D masks are
projected into the 3D space, and through an iterative bottom-up process, the
3D masks of partial scenes are merged to derive the final segmentation result
for the entire scene (see Fig. 2). However, due to the heuristic merging
rules, there are cases where masks that should be merged are not actually
merged when combining the 3D masks of two adjacent frames. Additionally,
variations in perspectives across frames can lead to conflicting segmentations
in overlapping regions, which introduces noise during the merging. Meanwhile,
the inference speed of SAM3D is significantly slow, attributed to the
application of SAM to a large number of RGB images in the scenes coupled with
the bottom-up merging approach.

To circumvent the cumbersome and noisy merging procedure, we introduce
Segment3D, a _clean and efficient_ approach that directly utilizes a _native_
3D segmentation model to achieve class-agnostic, fine-grained 3D segmentation.
Segment3D employs a two-stage training approach that requires no hand-
annotated labels for supervision. We first pre-train our class-agnostic 3D
segmentation model with the automatically generated 2D masks from SAM which
are projected to partial RGB-D point clouds. Since there is readily available
large-scale RGB-D data, additional data can be effortlessly integrated to
further enhance segmentation performance. This pre-training stage lays the
groundwork for understanding the 3D structure from 2D annotations. However, as
our ultimate objective is the segmentation of full 3D scenes, we must bridge
the domain gap between partial point clouds and the more comprehensive 3D
point clouds obtained from 3D scanners or reconstruction techniques [13, 21].
To this end, in the second stage, we fine-tune the model on full 3D point
clouds in a self-supervised manner, utilizing high-confidence mask predictions
from the pre-trained model as training signal.

In experiments, we show strong performance in class-agnostic segmentation on
ScanNet++ [54] and further show the use of Segment3D for improving open-
vocabulary 3D instance segmentation [47].

Overall, the contributions of this paper are as follows:

  * •

We introduce Segment3D, a novel approach and training strategy for fine-
grained class-agnostic 3D point cloud segmentation without manually annotated
labels.

  * •

Improved segmentation performance compared to a wide range of baselines,
including fully supervised methods trained on carefully annotated datasets.

  * •

We show that utilizing 2D foundation models to automatically generate high-
quality training masks is a viable alternative to costly manual labeling.

##  2 Related Work

#### 3D Instance Segmentation

Current models have seen a significant development over the last years, from
proposal-based [55, 52], over grouping-based [23, 8, 29, 49, 28], to recent
Transformer-based [43, 30] methods. In this work, we follow the currently
best-performing Transformer-based paradigm. Despite impressive advancements, a
shared limitation is the dependence on costly manual ground-truth annotations.
Recently, there have been efforts to automate the annotation process by
bundling state-of-the-art segmentation models [50], but they are still limited
to pre-defined object classes. However, in the context of open-world 3D scene
understanding, the importance of semantic classification for closed-set
categories has diminished. A rare exception are interactive 3D segmentation
methods[27, 56] where a human iteratively collaborates with a deep learning
model to segment arbitrary objects, but naturally requires user input.
Instead, in this work, we propose a general 3D segmentation method trained on
automatically generated labels.

#### Foundation Models

Foundation models, particularly those that are multimodal [39, 22], have
revolutionized the field of AI by leveraging extensive image-text pre-
training. These models can derive rich image representations guided by natural
language descriptions, enabling a variety of downstream tasks [18, 40, 35].
Another line of work [6, 34], based on self-supervised learning, employs image
training and yields high-performance features directly applicable as inputs
for linear classifiers. Segment Anything Model (SAM) [25] has recently
advanced the performance of foundation models for image segmentation. SAM has
undergone training on a diverse, high-quality dataset comprising more than 1
billion masks. This training equips SAM with the ability to generalize to
novel object types and images, surpassing the scope of its observations during
the training process. Additionally, SAM can generate high-quality, fine-
grained masks. Our work leverages the power of SAM for pre-training a 3D
segmentation model, which is later also used for generating supervision
signals for fine-tuning on scene-level point clouds.

Open-Vocabulary 3D Scene Understanding. Recently, there has been an increased
interest in 3D open-vocabulary scene understanding. This new field utilizes
the zero-shot recognition abilities of 2D vision-language models [39, 22],
enabling a more comprehensive understanding of diverse and previously unseen
3D environments [58, 20, 19, 36, 24, 26, 14, 31, 57, 47, 7]. PLA [14] aligns
point cloud features with captions extracted from multi-view images of a scene
to enable open-vocabulary recognition. OpenScene [36] distills per-pixel image
features to 3D point clouds, generating point-wise scene representations co-
embedded with text and image pixels in CLIP feature space. However, it mainly
focuses on semantic segmentation and exhibits a limited understanding of
object instances. To this end, OpenMask3D [47] predicts class-agnostic 3D
instance masks and aggregates per-mask features via multi-view fusion of CLIP-
based image embeddings. Although they have achieved impressive results in
open-vocabulary tasks, a highly versatile segmentation model is required to
accomplish this. Our method contributes to this area by providing a variety of
class-agnostic masks. These masks can serve as foundational inputs for models
like OpenMask3D, enhancing their ability to perform instance-related tasks.

##  3 Method

Figure 2: SAM3D vs. Segment3D. SAM3D [53] _(left)_ merges 2D segmentation
masks of RGB-D images generated by SAM to obtain 3D segmentation of entire
scenes. This merging process introduces noise due to heuristic merging rules
and conflict segmentation results across overlapping frames. Moreover, it is
slow because of extensive image inference and the cumbersome merging
procedure. Instead, our Segment3D _(right)_ utilizes a native 3D model to
directly segment entire 3D scenes.  Figure 3: Method Overview. Training
Segment3D involves two stages: The first stage _(left)_ relies on largely
available RGB-D image sequences and SAM $\blacksquare$$\square$, a pre-trained
foundation model for 2D image segmentation [25]. Segment3D
$\blacksquare$$\square$ is pre-trained on partial RGB-D point clouds and
supervised with pseudo ground-truth masks from SAM projected to 3D. Due to the
domain gap between partial and full point clouds, in the second stage
_(right)_ , Segment3D $\blacksquare$$\square$ is fine-tuned with confident
masks predicted by the pre-trained Segment3D $\blacksquare$$\square$.

We are interested in a method that can segment any object in a given 3D scene.
Hence, depending on existing 3D training datasets cannot accomplish this goal,
as their mask annotations are limited to a predefined set of object classes
[12, 2, 45, 1] and models trained on these datasets might not generalize well
to novel classes. SAM [25] is a foundation model for 2D image segmentation,
which shows extraordinary generalization ability. SAM3D [53] introduced a
straightforward method utilizing posed RGB-D images for 3D scene
understanding. As shown in Fig. 2, it employs SAM to predict segmentation
masks for RGB-D images, followed by projecting these 2D masks into 3D space.
Through an iterative bottom-up process, 3D masks of partial scenes are merged
to obtain the segmentation result for the entire scene. However, heuristic
merging rules sometimes hinder the correct merging of 3D masks, and
conflicting segmentation results across overlapping frames can introduce noise
during the merging. Furthermore, the extensive image processing and the
cumbersome merging procedure lead to the slow inference speed. To avoid the
cumbersome and noisy merging process, we introduce Segment3D, a clean and
efficient approach which utilizes a native 3D model to directly segment the
entire scene.

Our key idea is to automatically generate class-agnostic mask annotations
using pre-trained foundation models [4]. To employ SAM for our 3D segmentation
model we formulate a two-stage training approach as illustrated in Fig. 3.
First, 2D masks generated with SAM are projected onto _partial_ RGB-D point
clouds and serve as supervision signal for pre-training our class-agnostic 3D
segmentation model (Sec. 3.1). Since the final goal is to segment 3D scenes,
we need to bridge the inevitable domain gap between partial RGB-D point clouds
and full 3D point clouds from 3D scanners or reconstruction methods. We
therefore fine-tune our model on _full_ 3D point clouds using high-confident
mask predictions from our pre-trained model (Sec. 3.2).

###  3.1 Stage 1: Pre-Training on RGB-D Point Clouds

In contrast to the relatively scarce availability of 3D data, there is an
abundance of 2D data, particularly RGB-D images, which are readily accessible.
For example, ScanNet [12] comprises merely 1513 3D scans, compared to the
substantially larger collection of $2.5$ million RGB-D images. Leveraging the
automatically generated masks from SAM together with the abundant 2D data, we
pre-train our 3D segmentation model on partial RGB-D point clouds.

Data Preparation. Starting from a collection of RGB-D frames, we create
partial 3D point clouds and their corresponding pseudo ground-truth 3D masks.
Note that the labels can be automatically obtained without manual effort.

For each frame in a large RGB-D dataset, we first transform the 2D depth map
to a partial 3D point cloud. To do so, we need to know the intrinsic matrix
$\mathbf{K}\in\mathbb{R}^{3\times 3}$ and extrinsic matrix
$\mathbf{T}=\begin{bmatrix}\mathbf{R}&\mathbf{t}\end{bmatrix}\in\mathbb{R}^{3\times
4}$. For each pixel $\mathbf{p}=(u,v)$, we can transform it with its depth
value $D_{\mathbf{p}}$ into a 3D point $\mathbf{P}$ in world coordinates as
follows:

$\mathbf{P}=\mathbf{R}^{\top}\cdot(D_{\mathbf{p}}\cdot\mathbf{K}^{-1}\cdot\tilde{\mathbf{p}})-\mathbf{R}^{\top}\mathbf{t}\vspace{-5pt}$
(1)

where $\tilde{\mathbf{p}}$ is the homogeneous coordinate of $\mathbf{p}$. By
applying the process in Eq. (1) to all pixels in the depth map $D$ and
associating their per-pixel RGB value, we obtain the input partial 3D point
cloud. Now, to obtain the pseudo ground-truth 3D segmentation masks for this
point cloud, we prompt SAM [25] with a regular grid on the RGB image and
acquire on average $\sim 50$ high-quality 2D segmentation masks per frame.
Since we know the one-to-one mapping between 2D pixel and 3D points from Eq.
(1), we directly obtain the per-point 3D mask labels.

Model Architecture. We use a model inspired by Mask3D [43] to train a class-
agnostic 3D segmentation model. The model is comprised of a sparse
convolutional backbone derived from MinkowskiUNet [11] and a transformer
decoder, as in MaskFormer [10, 9]. We adopt a set of queries to represent the
masks, each of which is initialized with a positional embedding. Specifically,
we select query positions with furthest point sampling (FPS) and use their
Fourier positional encodings as the query embeddings. Leveraging the
transformer decoder, all the mask queries are refined by progressively
attending to point cloud features across multiple scales in parallel. Each
mask query is subsequently decoded into both a mask feature and a binary label
to predict whether the given query corresponds to a valid object or not. By
computing cosine similarity scores between a mask feature and all point
features within the point cloud, a heatmap is generated over the point cloud.
This heatmap is input to a sigmoid function, and thresholded at 0.5, resulting
in the final binary mask.

Training with SAM Generated Masks. We supervise the model with two losses: the
per-point level supervision loss $\mathcal{L}_{\text{mask}}$ and a per-query
level supervision loss $\mathcal{L}_{\text{obj}}$. The loss
$\mathcal{L}_{\text{mask}}$ enables learning a foreground-background
segmentation for each mask, and is composed of a dice loss
$\mathcal{L}_{\text{dice}}$ [32] and a binary cross-entropy loss
$\mathcal{L}_{\mathrm{ce}}$ for each point. The $\mathcal{L}_{\mathrm{obj}}$
is a binary classification loss that indicates whether a query represents a
valid “object” or “no object”. This mechanism allows for the prediction of a
variable number of masks, depending on the underlying scene content and
geometry. Following prior work [5, 43, 9], we first adopt bipartite graph
matching to establish correspondences between the set of predicted masks and
the set of target masks originating from SAM as described before. If the
predicted instance finds a matching target mask, then we assign it an “object”
label; conversely, if there is no match, we assign “no object”. In summary, we
optimize the following losses:

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{mask}}+\lambda_{\text{obj}}\mathcal{L}_{\text{obj}}$
(2)
$\displaystyle\mathcal{L}_{\text{mask}}=\lambda_{\text{dice}}\mathcal{L}_{\text{dice}}+\lambda_{\mathrm{ce}}\mathcal{L}_{\mathrm{ce}}$
(3)

where $\lambda_{*}$ are hyperparameters that balance the contribution of each
component in the loss. The binary classification loss
$\mathcal{L}_{\text{obj}}$ is applied to all queries, while the mask loss
$\mathcal{L}{{}_{\text{mask}}}$ is specifically applied to masks labeled as
“object”.

###  3.2 Stage 2: Self-Supervised Scene Fine-Tuning

After Stage 1, we obtain a class-agnostic 3D segmentation model by pre-
training solely on RGB-D images and automatically generated labels from SAM.
However, a fundamental domain gap persists between partial point clouds
derived from RGB-D images and full point clouds acquired through a 3D scanner
or reconstruction methods [21, 13] (See Fig. 4). This gap exists mostly
because of object occlusions, but also due to challenges of depth cameras to
capture dark or reflective surfaces from a single viewpoint. Hence, depending
solely on RGB-D frames for training a 3D segmentation model intended for full
3D scans proves inadequate. Therefore, we propose to further fine-tune our
model on scene-level full 3D point clouds. The key idea to obtain 3D mask
annotations for training on full point clouds is to use selected, high-
confidence masks generated by the pre-trained model itself. Note that this
approach does not require any manual labels on full 3D scenes and proves
essential for the performance of Segment3D (see Tab. 3).

Figure 4: Partial vs. Full Point Clouds. We show the difference between
partial _(center)_ and full _(right)_ point clouds from ScanNet [12]. Full
point clouds are more complete and exhibit fewer occlusions due to
reconstruction over multiple viewpoints. Due to this domain gap, fine-tuning
on full 3D point clouds is necessary.

Confidence-Based Mask Generation. Next, we outline the process of generating
the supervision signal for the fine-tuning stage. The pre-trained model
processes point clouds independently of their nature, be it partial or full.
Therefore, when presented with a full 3D point cloud, the pre-trained 3D model
produces a set of masks, each with a binary classification (valid or not) and
a heatmap over all points, just as in Sec. 3.1. To assess the quality of the
predicted masks, we compute a confidence score based on the confidence map
$\sigma(\mathbf{h})$, where $\mathbf{h}$ is the predicted heatmap and $\sigma$
is the sigmoid function. We then compute the average confidence of those
points for which $\sigma(\mathbf{h})>0.5$ as the confidence score of the
predicted mask, denoted as $c_{\text{mask}}$. We also consider the
classification as a valid object and use the probability from the binary
classification assigned to the “object” category as the confidence score,
denote as $c_{\text{obj}}$. The final confidence score for each predicted mask
is then the product of the two scores, $c=c_{\text{mask}}\cdot
c_{\text{obj}}$. For fine-tuning our 3D segmentation model in Stage 2, we
select the most confidently predicted masks above a threshold $\tau_{c}$.

Training with the High-Confidence Generated Masks. For fine-tuning, we follow
the same procedure as before and use $\mathcal{L}_{\text{mask}}$ as defined in
Eq. 3. In contrast to the pre-training stage, the binary classification loss
$\mathcal{L}_{\text{obj}}$, responsible for categorizing queries into valid or
invalid, is omitted. Since we only select masks with high confidence for
supervision, it can happen that some objects in the scene have no assigned
ground truth mask. In such instances, deeming a correctly predicted mask for
those objects as invalid would be detrimental. Table 3 illustrates the
efficacy of the self-supervised fine-tuning process in comparison to pre-
training alone.

|  | without post-processing | with post-processing
---|---|---|---
Model | Ground Truth Labels | AP | AP50 | AP25 | AP | AP50 | AP25
SAM3D [53] | ✗ | $3.9$ | $9.3$ | $22.1$ | $8.4$ | $16.1$ | $30.0$
Felzenszwalb _et al._ [17] | ✗ | $5.8$ | $11.6$ | $27.2$ | – | – | –
Mask3D [43] |  ScanNet200 [42] | $8.7$ | $15.5$ | $27.2$ | $14.3$ | $21.3$ | $29.9$
Mask3D [43] |  ScanNet [12] | 9.4 | 16.8 | 28.7 | 15.4 | 22.7 | 31.6
Segment3D (Ours) | ✗ | $\mathbf{12.0}$ | $\mathbf{22.7}$ | $\mathbf{37.8}$ | $\mathbf{19.0}$ | $\mathbf{29.7}$ | $\mathbf{41.6}$
|  | (+ $2.6$) | (+ $5.9$) | (+ $9.1$) | (+ $3.6$) | (+ $7.0$) | (+ $10.0$)

Table 1:  3D Segmentation Scores on ScanNet++ Val Set. The evaluation metric
is average precision (AP). Similar to [17, 53], Segment3D does not require
manually annotated training labels. We report scores with and without post-
processing (more details in Sec. 4.1).

##  4 Experiments

We firstly compare with fully-supervised and traditional geometric
segmentation baselines in a class-agnostic segmentation setting (Sec. 4.1). We
then provide detailed analysis to understand the importance of fine-tuning,
the effect of training on more data and show the advantage of our method in
segmenting small objects (Sec. 4.2). We also show qualitative results of our
3D segmentation method (Sec. 4.3). Finally, we demonstrate its potential
application for the task of open-set 3D instance segmentation as recently
proposed in OpenMask3D [47] (Sec. 4.4).

###  4.1 Comparing with State-of-the-Art Methods

Setup. We run experiments on four popular datasets including ScanNet [12], its
extension ScanNet200 [42], the Replica[45] dataset, and the newly released
ScanNet++ [54]. For training our model, we employ ScanNet [12, 42], which is
collected through a lightweight RGB-D scanning process. ScanNet comprises
$1513$ indoor scenes, encompassing $\sim$$2\cdot 10^{6}$ views, along with 3D
camera poses, surface reconstruction and instance-level semantic mask
annotations. For Stage 1, we sample every 25${}^{\text{th}}$ frame of the
RGB-D sequences ($\sim$ 1 FPS) and obtain approximately $76\cdot 10^{3}$
training frames. For Stage 2, we use the $\sim$$1.2\cdot 10^{3}$ reconstructed
3D scans of indoor spaces as full point clouds. For evaluation, we use
ScanNet++ [54] which comes with high-fidelity 3D mask annotations including
smaller objects which are not well annotated in ScanNet. It includes high-
resolution 3D scans captured at sub-millimeter precision and annotated
comprehensively, covering objects of varying sizes. See the appendix for the
implementation details of our method.

Methods in Comparison. We compare with a wide range of prior art methods from
different categories. Mask3D [43] is a fully-supervised transformer-based
method trained on manually annotated 3D segmentation masks. Transformer-based
methods [44, 46] currently define the state-of-the-art approach for 3D
instance segmentation. Segment3D has the same backbone as Mask3D but instead
of training on manually annotated 3D masks, it learns from automatically
generated 2D (pre-training) and 3D masks (fine-tuning). Felzenszwalb _et al._
[17] proposed a graph-based method for segmentation which does not require any
training data and operates directly on the 3D geometry. As mentioned in Sec.
3, SAM3D [53] merges the segmentation masks of RGB-D images generated by SAM
to obtain the 3D segmentation result. Similarly to ours, it does not rely on
manual labels.

Metrics. We evaluate all methods on the validation set of ScanNet++ and report
average precision (AP) scores at IoU thresholds of $25$%, $50$%, and averaged
over the range [0.5:0.95:05] between predicted and ground truth masks.
Consistent with common practices in the field [44, 46, 49], we also report
scores after post-processing. This involves smoothing the predicted masks
through graph-based oversegmentation [17], and splitting distant parts of the
same mask via connected component clustering DBSCAN [16].

Results. Scores are reported in Table 1. Segment3D outperforms all previous
methods by at least $\mathbf{+2.6}$ AP and up to $\mathbf{+10.0}$ AP25.
Notably, we achieve such improvements _without ground-truth mask annotations_
as used by Mask3D. In general, the performance of Mask3D (and other fully
supervised methods) depends on the quality of the annotated training dataset;
often the manual annotation of small objects (pens, cellphones) or other fine-
details is challenging. Instead, Segment3D relies on automatically generated
high-quality masks from SAM which can capture fine-grained details without
human annotation effort. Table 2 highlights that Segment3D excels particularly
in predicting the more challenging small object masks.

Finally, our approach significantly outperforms SAM3D despite both methods
relying on the same 2D masks from SAM. Even without fine-tuning Segment3D
already outperforms SAM3D (see Table 3). This shows that Segment3D can
circumvent the noise that arises during the merging process of SAM3D.
Furthermore, Segment3D is more efficient since it only processes a single
input of point cloud, while SAM3D needs to inference a large number of RGB-D
images and involves a bottom-up merging procedure.

Note the comparable performance of Mask3D trained on ScanNet and ScanNet200,
which we attribute to similar mask annotations in both datasets. They differ
only in the labeled classes (20 _vs._ 200), while most categories of
ScanNet200 are labeled as “other” in ScanNet. For both datasets, we utilize
all available masks independent of their semantic class to maximize the number
of training labels.

###  4.2 Analysis Experiments

Mask Size |  | Mask3D[43] |  Segment3D (Ours)
---|---|---|---
Large |  AP50 | $30.6$ |  $30.4$ (- $0.2$)
AP25 | $47.7$ |  $47.0$ (- $0.7$)
Medium |  AP50 | $25.8$ |  $29.2$ (+ $3.4$)
AP25 | $41.8$ |  $49.5$ (+ $7.7$)
Small |  AP50 | $4.5$ |  $15.9$ (+ $11.4$)
AP25 | $11.3$ |  $29.6$ (+ $18.3$)
Table 2:  Segmentation Scores on Different Mask Sizes. The evaluation metric
is average precision (AP) on ScanNet++ validation set. Segmented3D improves
over Mask3D, especially on small and medium-sized object masks. Details are in
Sec. 4.2.

Performance on Different Mask Sizes. We proceed with an analysis of the
performance of our Segment3D and Mask3D (the best-performing baseline) on
object masks of various sizes. We categorize the size of masks based on the
number of points they contain (small: [$0$, $2$k], medium: [$2$k, $15$k]
large: [$15$k, $\infty$]) and exclude the masks of the floor, ceiling and
walls. The results are reported in Table 2. Our method yields significantly
improved segmentation results on small and medium-sized masks, and performs on
par with Mask3D on large masks. This confirms our intuition that Mask3D
performs poorly on small-sized object masks as those are typically harder to
manually annotate. In contrast, Segment3D utilizes masks from SAM as
supervision, which capture fine-grained scene details. In summary, our model,
trained on automatically generated masks by a 2D foundation model, surpasses a
model trained on manually labeled datasets. This showcases the usefulness of
foundation models and raises the question if manually labeled large-scale 3D
datasets are necessary for training 3D point cloud segmentation models.

Figure 5: Qualitative Results on ScanNet++ Val Set. From top to bottom, we
show the colored input 3D scenes, the segmentation masks predicted by SAM3D
[53], Mask3D [43], our Segment3D and the ground truth 3D mask annotations.
\begin{overpic}[width=447.16759pt]{figures/openmask3d_quali.pdf}\put(10.0,0.5){{\small\hbox{\pagecolor{white}{``a
black eraser''}}}} \put(44.0,0.5){{\small\hbox{\pagecolor{white}{``kettle
handle"}}}} \put(74.0,0.5){{\small\hbox{\pagecolor{white}{``copier control
screen"}}}} \end{overpic} Figure 6: Qualitative Results of Adapted OpenMask3D
[47]. Given a text prompt _(bottom)_ , OpenMask3D finds the corresponding
masks $\blacksquare$ in a given 3D scene _(top)_. We show the 3D scene
reconstruction and an RGB image for better visualization _(top left corner)_.
The original OpenMask3D based on Mask3D is unable to recognize any of the
above queries, whether it is a relatively small object or a fine-grained
affordance part of an object. In contrast, our Segment3D adaptation performs
well in these cases.

The Effect of Two-Stage Training. Next, we compare the performance of
Segment3D pre-trained solely on partial RGB-D point clouds (Stage 1) and with
additional fine-tuning on full point clouds (Stage 2). Scores are reported in
Table 3. The additional fine-tuning stage almost doubles the segmentation
performance of our model on the most challenging AP metric. By training with
the predicted high-confidence masks, Segment3D effectively reduces the
inherent domain gap between the partial point clouds derived from RGB-D images
and full 3D point clouds.

Training Stages | AP | AP50 | AP25
---|---|---|---
Pre-Training (Stage 1) | $7.4$ | $15.2$ | $31.2$
\+ Fine-Tuning (Stage 2) |  $\mathbf{12.0}$ (+ $4.6$) |  $\mathbf{22.7}$ (+ $7.5$) |  $\mathbf{37.8}$ (+ $6.6$)

Table 3:  Effect of Two-Stage Training. Fine-tuning on full point clouds
supervised by confident predicted 3D masks significantly surpasses pre-
training on projected 2D SAM masks alone.

Number of Queries. Since our model and Mask3D share the same transformer-based
architecture, we analyse the influence of the number of queries on the
segmentation performance. The results are presented in Fig. 7. The number of
queries is an important aspect of such models as each query represents a mask
and ultimately defines the upper bound of recognizable masks in a 3D scene. In
the case of Mask3D, the segmentation performance remains relatively stable
with changes in the number of queries, whereas Segment3D experiences notable
benefits from an increased number of queries. This is an interesting
observation since the key difference between Segment3D and Mask3D is the
training data, and it shows the benefit of automatically generated SAM masks
which are more diverse than the manually annotated ground truth masks in
ScanNet. Constrained by limited GPU memory, we were unable to assess
performance with more than 400 queries. Nevertheless, the trend of the curves
suggests the potential for further improved performance with increased query
numbers.

$\blacksquare$ Segment3D $\blacksquare$ Mask3D (ScanNet) $\blacksquare$ Mask3D
(ScanNet200)

Figure 7: Performance with Varying Number of Queries. The metric is average
precision with IoU thresholds of 25% and 50%.

Pre-Training with Additional Data. Since RGB-D data is available in abundance
and masks can be automatically generated, it is natural to ask if pre-training
on additional masks will further improve the overall performance. To that end,
we perform a first experiment where we select additional frames from the
training set of ScanNet++ dataset (to increase the variety of training data).
Again, we sample frames at roughly $1$ FPS resulting in 34k frames, in
addition to the previous 76k frames of ScanNet. Table 4 shows an impressive
performance boost of $\mathbf{+2.0}$ AP50. For future work, considering that
this improvement is obtained simply by adding more automatically generated
masks to the pre-training, our approach seems promising to train on even more
data. It could even be plausible to train on internet images combined with
monocular depth estimation, such as ZoeDepth [3], to compute the partial point
clouds.

Pre-Training Dataset (# Frames) | AP | AP50
---|---|---
ScanNet ($76$k) | $12.0$ | $22.7$
ScanNet ($76$k), ScanNet++ ($34$k) |  $13.7$ (+ $1.7$) |  $24.7$ (+ $2.0$)

Table 4:  Performance Increase with Additional Training Data.

###  4.3 Qualitative Results

Fig. 5 shows several representative examples of Segment3D segmentation results
on the ScanNet++ dataset. As can be seen, the scenes are quite diverse,
presenting multiple challenges such as clutter and a wide range of mask sizes.
Despite these challenges, our model predicts quite accurate and well-localized
segmentation masks. For example, compared to Mask3D, our method is able to
segment the finer-grained objects on top of the bed and the shelf. The masks
of the computer screen and the chair in front of it are also less fragmented
than predictions of the baselines. See the appendix (Sec. B) for
visualizations in the wild test.

###  4.4 Application: Open-Set Scene Understanding

A real-world application of our class-agnostic 3D segmentation method is open-
vocabulary 3D scene understanding, as implemented by the recent OpenMask3D
[47]. Given a 3D scene, a user can search for arbitrary objects via text
prompts (see Fig. 6). A core component of OpenMask3D is Mask3D which segments
the scene into masks. Since Mask3D is trained on the closed-set of ScanNet,
its masks are not truly class-agnostic or open-vocabulary. We therefore
replace Mask3D with our class-agnostic Segment3D. We evaluate on the closed-
set labels of Replica and ScanNet++ in Table 5 and report an improvement of up
to $\mathbf{+2.7}$ AP50. Yet, this score serves only as an indicator, since
most masks are not considered in this closed-set evaluation. See the appendix
(Sec. C) for a discussion and more details.

Model | Segmentor | ScanNet++ | Replica
---|---|---|---
OpenMask3D [47] |  Mask3D[43] | $15.0$ | $18.0$
OpenMask3D [47] | Segment3D (Ours) |  $17.7$ (+ $2.7$) |  $18.7$ (+ $0.7$)

Table 5:  Open-Set 3D Scene Understanding Scores. The evaluation metric is
average precision with an IoU threshold of $50$%.

##  5 Conclusion and Discussion

We have presented Segment3D, a simple, yet powerful class-agnostic 3D
segmentation model. The model is trained entirely on automatically generated
masks from SAM, a foundation model for image segmentation. It is competitive
and even outperforms existing 3D segmentation models that rely on hand-labeled
3D training scenes.

Indeed, this raises the question of whether hand-labeled 3D training datasets
are as essential as they were once thought to be. Similarly, in the case of
test labels, our qualitative comparison indicates that Segment3D sporadically
identifies small object masks not annotated in the ground truth. Consequently,
the full performance of Segment3D might not be accurately reflected in the
scores.

Overall, the work shows the potential of foundation models as automatic
training label generators, and our preliminary experiments seem to indicate
that more data will further increase the segmentation performance for
Segment3D.

## References

  * [1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese.  3D Semantic Parsing of Large-Scale Indoor Spaces.  In CVPR, 2016.
  * [2] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al.  ARKitScenes: A Diverse Real-World Dataset for 3D Indoor Scene Understanding using Mobile RGB-D Data.  arXiv preprint arXiv:2111.08897, 2021.
  * [3] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  ZoeDepth: Zero-Shot Transfer by Combining Relative and Metric Depth.  arXiv preprint arXiv:2302.12288, 2023.
  * [4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the Opportunities and Risks of Foundation Models.  arXiv preprint arXiv:2108.07258, 2021.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-End Object Detection with Transformers.  In ECCV, 2020.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging Properties in Self-Supervised Vision Transformers.  In ICCV, 2021.
  * [7] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang.  CLIP2Scene: Towards Label-Efficient 3D Scene Understanding by CLIP.  In CVPR, 2023.
  * [8] Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang.  Hierarchical Aggregation for 3D Instance Segmentation.  In ICCV, 2021.
  * [9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-Attention Mask Transformer for Universal Image Segmentation.  In CVPR, 2022.
  * [10] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-Pixel Classification is Not All You Need for Semantic Segmentation.  In NeurIPS, 2021.
  * [11] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks.  In CVPR, 2019.
  * [12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.  In CVPR, 2017.
  * [13] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt.  BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration.  TOG, 2017.
  * [14] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi.  PLA: Language-Driven Open-Vocabulary 3D Scene Understanding.  In CVPR, 2023.
  * [15] Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner.  3D-MPA: Multi-Proposal Aggregation for 3D Semantic Instance Segmentation.  In CVPR, 2020.
  * [16] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise.  In KDD, 1996.
  * [17] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient Graph-Based Image Segmentation.  IJCV, 2004.
  * [18] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation.  In ICLR, 2022.
  * [19] Huy Ha and Shuran Song.  Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models.  In CoRL, 2022.
  * [20] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo.  CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training.  In ICCV, 2023.
  * [21] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al.  KinectFusion: Real-time 3D Reconstruction and Interaction using a Moving Depth Camera.  In UIST, 2011.
  * [22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up Visual and Vision-Language Representation Learning with Noisy Text Supervision.  In ICML, 2021.
  * [23] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia.  PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation.  In CVPR, 2020.
  * [24] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  LERF: Language Embedded Radiance Fields.  In ICCV, 2023.
  * [25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment Anything.  In ICCV, 2023.
  * [26] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing Nerf for Editing via Feature Field Distillation.  In NeurIPS, 2022.
  * [27] Theodora Kontogianni, Ekin Celikkan, Siyu Tang, and Konrad Schindler.  Interactive Object Segmentation in 3D Point Clouds.  In ICRA, 2023.
  * [28] Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, and Bastian Leibe.  4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation.  2022\.
  * [29] Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia.  Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks.  In CVPR, 2021.
  * [30] Jiahao Lu, Jiacheng Deng, Chuxin Wang, Jianfeng He, and Tianzhu Zhang.  Query Refinement Transformer for 3D Instance Segmentation.  In ICCV, 2023.
  * [31] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang.  Open-Vocabulary Point-Cloud Object Detection without 3D Annotation.  In CVPR, 2023.
  * [32] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.  In 3DV, 2016.
  * [33] Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann.  Mix3D: Out-of-context data augmentation for 3D scenes.  In 3DV, 2021.
  * [34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  DINOv2: Learning Robust Visual Features without Supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [35] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  In ICCV, 2021.
  * [36] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al.  OpenScene: 3D Scene Understanding with Open Vocabularies.  In CVPR, 2023.
  * [37] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  In CVPR, 2017.
  * [38] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In NeurIPS, 2017.
  * [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models from Natural Language Supervision.  In ICML, 2021.
  * [40] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting.  In CVPR, 2022.
  * [41] Xavier Roynard, Jean-Emmanuel Deschaud, and François Goulette.  Paris-Lille-3D: A Large and High-Quality Ground-Truth Urban Point Cloud Dataset for Automatic Segmentation and Classification.  IJRR, 2018.
  * [42] David Rozenberszki, Or Litany, and Angela Dai.  Language-Grounded Indoor 3D Semantic Segmentation in the Wild.  In ECCV, 2022.
  * [43] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe.  Mask3D: Mask Transformer for 3D Semantic Instance Segmentation.  In ICRA, 2023.
  * [44] Jonas Schult, Francis Engelmann, Theodora Kontogianni, and Bastian Leibe.  DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes.  In CVPR, 2020.
  * [45] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The Replica Dataset: A Digital Replica of Indoor Spaces.  arXiv preprint arXiv:1906.05797, 2019.
  * [46] Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu.  Superpoint Transformer for 3D Scene Instance Segmentation.  In AAAI, 2023.
  * [47] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann.  OpenMask3D: Open-Vocabulary 3D Instance Segmentation.  In NeurIPS, 2023.
  * [48] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  KPConv: Flexible and Deformable Convolution for Point Clouds.  In ICCV, 2019.
  * [49] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo.  SoftGroup for 3D Instance Segmentation on Point Clouds.  In CVPR, 2022.
  * [50] Silvan Weder, Hermann Blum, Francis Engelmann, and Marc Pollefeys.  LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories.  In 3DV, 2024.
  * [51] Silvan Weder, Francis Engelmann, Johannes L Schönberger, Akihito Seki, Marc Pollefeys, and Martin R Oswald.  ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction.  arXiv, 2023.
  * [52] Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni.  Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds.  In NeurIPS, 2019.
  * [53] Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu.  SAM3D: Segment Anything in 3D Scenes.  In ICCVW, 2023.
  * [54] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai.  ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes.  In ICCV, 2023.
  * [55] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J Guibas.  GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud.  In CVPR, 2019.
  * [56] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni.  AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation.  In arXiv preprint arXiv:2306.00977, 2023.
  * [57] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu.  CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data.  In CVPR, 2023.
  * [58] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  PointCLIP: Point Cloud Understanding by CLIP.  In CVPR, 2022.

##  A Implementation Details

The feature backbone of Segment3D is a Minkowski Res16UNet34C [11]. We perform
standard data augmentations, including horizontal flipping, random rotations,
elastic distortion and random scaling. In addition, we use color augmentations
including jittering, brightness and contrast augmentation. For Stage 1, we use
AdamW optimizer and a one-cycle learning rate schedule with a peak learning
rate of $2\times 10^{-4}$. The model is trained for 20 epochs with a batch
size of $16$ partial RGB-D point clouds. A training on $2$ cm voxelization
takes approximately $60$ hours with $2$ RTX3090 GPUs. For Stage 2, the initial
learning rate is set to $1\times 10^{-4}$. We train the model for 50 epochs
with a batch size of $8$ full 3D point clouds. Training takes $\sim 10$ hours
with $2$ cm voxels on $4$ A100 GPUs. For both Stage 1 and Stage 2, we set the
number of queries to 150 during training. For Stage 1, following Mask3D [43],
the values of $\lambda_{\text{obj}}$, $\lambda_{\text{dice}}$, and
$\lambda_{\text{ce}}$ are set to 2, 2, and 5, respectively. For Stage 2, the
values of $\lambda_{\text{dice}}$ and $\lambda_{\text{ce}}$ are set to 2 and
5, respectively.

In Stage 1, to generate masks with SAM [25], we use its automatic mask
generation pipeline. We prompt SAM with a regular grid on the RGB image. After
post-processing, we obtain nested masks with different levels of granularity.
In cases where a pixel is encompassed by multiple masks, we assign the mask ID
associated with the highest predicted Intersection over Union (IoU) to that
pixel. Then we project the masks to 3D. In Stage 2, we use DBSCAN [16] to
split erroneously merged instances for the selected high-confidence masks.

##  B Additional Datasets

To further assess the generalization capability of Segment3D, we test on
additional indoor and outdoor data.

###  B.1 Indoor In-the-Wild Scenes

We first perform experiments on in-the-wild indoor scans. Unlike the training
scenes, which are mostly collected in hotel rooms, office spaces and
universities, the scans used in these experiments are recorded from workshops
depicting a wide variety of objects never seen during training, such as
milling machines, saws, turning lathes, grindstones etc., (see Fig. A). We
show qualitative results of our Segment3D and Mask3D [43] which are both pre-
trained on ScanNet [12]. Our Segment3D yields much better masks than Mask3D.
For example, Segment3D yields accurate, sharp and compact segmentation masks
of small individual objects on the table, and posters on the wall while at the
same time being able to segment large tools.

Figure A: Qualitative Results on Indoor _‘in-the-wild’_ Scenes. From top to
bottom, we show the colored input 3D scenes, the segmentation masks predicted
by Mask3D [43] and our Segment3D. Segment3D, trained with unsupervised
learning, demonstrates superior generalization compared to Mask3D, which is
trained with fully supervised learning.

###  B.2 Outdoor Scenarios

Next, we visualize the performance of Segment3D and Mask3D on outdoor scenes
in Fig. B and report scores in Tab. A. Specifically, we carry out experiments
on the Paris-Lille-3D [41] (PL3D) dataset. PL3D is a segmentation dataset
containing more than $2$ km of street recordings of the French cities, Lille
and Paris. The test set labels are not publicly available. However, since both
Mask3D [43] and Segment3D are trained on ScanNet, we can use the entire
training set of the PL3D dataset for evaluating the effectiveness of class-
agnostic segmentation. We directly apply both models trained on room-sized
indoor scenes to the outdoor scenes. The only adaptation we perform is the
split of the large outdoor scenes into multiple smaller crops, due to GPU
memory limitations, and the point cloud coordinates are scaled down by a
factor of $10$ to be better aligned with the room-sized training data.

The qualitative comparison reveals impressive generalization performance of
Segment3D, for example, the individual cars along the street are correctly
segmented while Mask3D oversegments or merges them. The street itself is
correctly recognized as a very large single segment, while Mask3D shows noisy
artifacts. Due to the necessary preprocessing and voxelization, the
quantitative scores are low as many small ground truth segments are merged in
a single voxel. However, it can still be observed that Segment3D performs
significantly better than Mask3D. This indicates that the generalization of
Segment3D is also better when transferring from indoor to outdoor scenes.

Figure B: Qualitative Results on Outdoor Scenes. We show segmentations
predicted by Mask3D [43] _(top)_ and our Segment3D _(bottom)._

Model | AP50 | AP25
---|---|---
Mask3D (Pre-trained on ScanNet) | $1.5$ | $3.3$
Segment3D (Pre-trained on ScanNet) |  $\mathbf{4.0}$ (+ $2.5$) |  $\mathbf{7.7}$ (+ $4.4$)

Table A:  Evaluation on the Paris-Lille-3D Dataset.

##  C Open-Set 3D Scene Understanding

As mentioned in the main paper, a key application of our class-agnostic
segmentation is open-set 3D scene understanding where a user can search for
arbitrary open-vocabulary descriptions. This task is implemented in the
OpenMask3D [47] framework. While the original OpenMask3D relies on Mask3D [43]
as the scene segmentor, we adapt it and replace Mask3D with Segment3D. Below,
in Fig. C we show additional results of the original OpenMask3D based on
Mask3D, compared to our adapted OpenMask3D based on Segment3D.

In addition to performing fine-grained segmentation, Segment3D also offers
more flexible input for text prompts, such as the affordance part of an
object. In Fig. C, we provide two examples. When we refer to the “kettle
handle”, Segment3D can highlight it accurately while Mask3D only segments the
whole “kettle”. Similarly, Segment3D can point out the “copier control screen”
while Mask3D can only segment the whole “copier”. This enables new
applications for intelligent non-human assistants, autonomous robots, and
AR/VR devices.

\begin{overpic}[width=447.16759pt]{figures/openmask3d.pdf}\put(12.0,0.5){{\small\hbox{\pagecolor{white}{``a
black eraser''}}}} \put(45.0,0.5){{\small\hbox{\pagecolor{white}{``kettle
handle"}}}} \put(74.0,0.5){{\small\hbox{\pagecolor{white}{``copier control
screen"}}}} \end{overpic} Figure C: Qualitative Results of Adapted OpenMask3D
[47]. Given a text prompt _(bottom)_ , OpenMask3D finds the corresponding
masks $\blacksquare$$\square$ in a given 3D scene _(top)_. We show the 3D
scene reconstruction and an RGB image for better visualization _(top left
corner)_. The original OpenMask3D based on Mask3D is unable to recognize any
of the above queries, whether it is a relatively small object or a fine-
grained affordance part of an object. In contrast, our Segment3D adaptation
performs well in these cases.