# Insight-HXMT observation on 4U 1608–52: evidence of interplay between
thermonuclear burst and accretion environments

Yu-Peng Chen* Key Laboratory for Particle Astrophysics, Institute of High
Energy Physics, Chinese Academy of Sciences, 19B Yuquan Road, Beijing 100049,
China  Shu Zhang* Key Laboratory for Particle Astrophysics, Institute of High
Energy Physics, Chinese Academy of Sciences, 19B Yuquan Road, Beijing 100049,
China  Long Ji  School of Physics and Astronomy, Sun Yat-Sen University,
Zhuhai, 519082, China  Shuang-Nan Zhang  Key Laboratory for Particle
Astrophysics, Institute of High Energy Physics, Chinese Academy of Sciences,
19B Yuquan Road, Beijing 100049, China  University of Chinese Academy of
Sciences, Chinese Academy of Sciences, Beijing 100049, China  Ling-Da Kong
Key Laboratory for Particle Astrophysics, Institute of High Energy Physics,
Chinese Academy of Sciences, 19B Yuquan Road, Beijing 100049, China
University of Chinese Academy of Sciences, Chinese Academy of Sciences,
Beijing 100049, China  Peng-Ju Wang  Key Laboratory for Particle Astrophysics,
Institute of High Energy Physics, Chinese Academy of Sciences, 19B Yuquan
Road, Beijing 100049, China  University of Chinese Academy of Sciences,
Chinese Academy of Sciences, Beijing 100049, China  Zhi Chang  Key Laboratory
for Particle Astrophysics, Institute of High Energy Physics, Chinese Academy
of Sciences, 19B Yuquan Road, Beijing 100049, China  Jing-Qiang Peng  Key
Laboratory for Particle Astrophysics, Institute of High Energy Physics,
Chinese Academy of Sciences, 19B Yuquan Road, Beijing 100049, China
University of Chinese Academy of Sciences, Chinese Academy of Sciences,
Beijing 100049, China  Jin-Lu Qu  Key Laboratory for Particle Astrophysics,
Institute of High Energy Physics, Chinese Academy of Sciences, 19B Yuquan
Road, Beijing 100049, China  University of Chinese Academy of Sciences,
Chinese Academy of Sciences, Beijing 100049, China  Jian Li  CAS Key
Laboratory for Research in Galaxies and Cosmology, Department of Astronomy,
University of Science and Technology of China, Hefei 230026, China  School of
Astronomy and Space Science, University of Science and Technology of China,
Hefei 230026, China<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A type-I burst could influence the accretion process through radiation
pressure and Comptonization both for the accretion disk and the
corona/boundary-layer of an X-ray binary, and vice versa. We investigate the
temporal evolution of a bright photospheric radius expansion (PRE) burst of 4U
1608–52 detected by Insight-HXMT in 1–50 keV, with the aim of studying the
interplay between the burst and persistent emission. Apart from the emission
from the neutron star (NS) surface, we find the residuals both in the soft
($<$3 keV) and hard ($>$10 keV) X-ray band. Time-resolved spectroscopy reveals
that the excess can be attributed to an enhanced pre-burst/persistent emission
or the Comptonization of the burst emission by the corona/boundary-layer. The
Comptonization model is a convolution thermal-Comptonization model (thcomp in
XSPEC) and the Comptonization parameters are fixed at the values derived from
the persistent emission. We find, during the PRE phase, after the enhanced
pre-burst/persistent emission or the Comptonization of the burst emission is
removed, the NS surface emission shows a plateau, and then a rise until the
photosphere touches down to the NS surface, resulting in a flux peak at that
moment. We speculate that the findings above correspond to that the obscured
lower part of the NS surface by the disk is exposed to the line of sight due
to the inner disk evaporation by the burst emission. The consistency between
the $f_{a}$ model and convolution thermal-Comptonization model indicates the
interplay between thermonuclear bursts and accretion environments. These
phenomena did not usually show up by the conventional blackbody model fitting,
which may be due to low count rate and narrow energy coverage in previous
observations.

stars: coronae — stars: neutron — X-rays: individual (4U 1608–52) — X-rays:
binaries — X-rays: bursts

##  1 Introduction

Type-I X-ray burst, also named thermonuclear bursts, are triggered by unstable
thermonuclear burning of the accreted fuel from a low-mass X-ray binary (LMXB)
hosting a neutron star (NS) (for reviews, see Lewin et al., 1993; Cumming,
2004; Strohmayer & Bildsten, 2006; Galloway et al., 2008). Since its first
detection in 1975 from 3A 1820–30, so far there are 116 Galactic X-ray
binaries observed to produce thermonuclear
bursts111https://personal.sron.nl/$\sim$jeanz/bursterlist.html, manifesting a
sudden increase in the X-ray luminosity followed by an exponential decay and
with a typical duration about tens seconds. The most luminous bursts are the
photospheric radius expansion (PRE) events, for which the peak flux reaches
the Eddington luminosity of the NS.

Among some of the thousands of observed bursts (Galloway et al., 2020),
observations on bursters by RXTE (in’t Zand et al., 2013; Worpel et al., 2013;
Ballantyne & Strohmayer, 2004; Keek et al., 2014), INTEGRAL (Sánchez-Fernández
et al., 2020), NICER (Keek et al., 2018, 2018a), AstroSat (Bhattacharyya et
al., 2018) and Insight-HXMT (Chen et al., 2018, 2019), revealed interactions
between the burst emission and the accretion environment: the continuum
spectrum was observed to have an enhancement at soft X-ray and/or a shortage
at hard X-rays (Worpel et al., 2013, 2015; Chen et al., 2012; Ji et al.,
2013). Such spectral deviations are considered as burst-induced and might be
relevant to disk reflection, accretion rate increase, and corona cooling
(Ballantyne & Strohmayer, 2004; Keek et al., 2014; Degenaar et al., 2018).

Moreover, the reflection spectrum, consisting of discrete lines and a hump
peaking at 20–40 keV, is interpreted as disk refection of an illuminant from
the corona/boundary-layer. The burst emission could also serve as the
illuminant to the disk, and a reflection component is correspondingly observed
during the burst. However, so far only iron line is firmly detected during
bursts, specifically during the long-duration superbursts (Ballantyne &
Strohmayer, 2004; Keek et al., 2014). The observations above are the influence
of the bursts on the accretion environment; however, there are few
observational results reported related to the burst spectral change caused by
the accretion disk/corona.

4U 1608–52 is a prolific burster located at the Galactic plane (Belian et al.,
1976). More than 100 type-I X-ray bursts, inhabited its outbursts which have a
typical frequency of once per 1–2 years since its discovery, were regularly
observed. The distance was estimated as $D\sim$2.9–4.5 kpc based on the peak
flux pf PRE bursts $\sim$1.2–1.5 $\times 10^{-7}~{}{\rm erg}~{}{\rm
cm}^{2}~{}{\rm s}^{-1}$ (e.g. Galloway et al., 2008; Poutanen et al., 2014).
Its spin is around $\nu$=619 Hz (Muno et al., 2001; Galloway et al., 2008),
based on the burst oscillation detection.

In this present investigation, we provide a broad-band spectral view of 4U
1608–52 during its 2020 outburst observed by NICER and Insight-HXMT, both for
its outburst and burst emission. We first describe the data reduction
procedure of NICER and Insight-HXMT in Section 2. We then present an in-depth
spectral analysis and model parameters of its outburst emission in Section
3.1, burst lightcures and spectral evolution in Section 3.2 and Section 3.3.
Finally, we summarize our results and discuss their implications in Section 4.

##  2 Observations and Data reduction

###  2.1 Insight-HXMT

Hard X-ray Modulation Telescope (HXMT, also dubbed as Insight-HXMT, Zhang et
al., 2020) excels in its broad energy band (1–250 keV) and a large effective
area in hard X-rays energy band. It carries three collimated telescopes: the
High Energy X-ray Telescope (HE; poshwich NaI/CsI, 20–250 keV, $\sim$ 5000
cm2), the Medium Energy X-ray Telescope (ME; Si pin detector, 5–40 keV, 952
cm2) and the Low Energy X-ray telescope (LE; SCD detector, 1–12 keV, 384 cm2).
Under the quick read-out system of Insight-HXMT detectors, there is little
pile-up effect event at the PRE burst peak. Insight-HXMT Data Analysis
software (HXMTDAS) v2.04 are used to analyze the data. The data are reduced
following the recommended procedure of the Insight-HXMT Data Reduction Guide
v2.04222http://hxmtweb.ihep.ac.cn/SoftDoc.jhtml, which are screened in the
standard criterion include in Insight-HXMT pipelines: lepipeline, mepipeline
and hepipeline.

Two bursts were detected by three-payloads of Insight-HXMT from 4U 1608–52 in
its 2020 outburst, as shown in Table 1. However, the second burst, which is
only half bright as the first one, did not fall into the good-time-interval of
LE;  thus we only analyzed the first burst occurred at MJD 59069.770768 in
this work.

The persistent spectra adapted the GTI exclude the time span before the burst
peak time 100 s and after the burst peak time 200 s. The persistent spectra of
LE are rebinned by ftool grppha with minimum of 100 counts per grouped bin.
For the ME, the spetcra are binned up by a factor 20, due to its background is
comparable with the source emission.

For the burst, we perform the time-resolved spectroscopy with a time
resolution of 0.25 s, and define the time of the bolometric flux peak as a
time reference (0 point in Fig. 1 and Fig. 3). The burst spectra are rebinned
by ftool grppha with minimum of 10 counts per grouped bin.

In the calibration experiments on ground and the first two years in orbit, the
recommend energy band for spectral fitting of LE is 1-10 keV, except very
bright sources with a flux brighter than several Crab. After the midyear of
2019, the recommend band is shrunken to 2-10 keV, which mostly dues to the
poor background estimation because of an increase of detector temperatures and
something else. However, in the burst spectral analysis, we take the pre-burst
emission as background, so we extend the energy band to 1-10 keV in burst
spectra fitting, but still adapt 2-10 keV in persistent emission spectral
fitting.  We notice that, the LE spectral residuals of the persistent emission
have rather complex structures in the energy band $<$ 1.5 keV, where the
bursts have only a few data points.  We also use the 2-10 keV to analyze the
burst spectra, and get the roughly consistent results within parameter’s error
bar. It is a similar consideration that we extend the ME band to 8–30 keV and
the HE band to 25–50 keV in the burst spectra fitting. In short, for the
persistent emission spectra fitting of LE and ME, the energy bands are limited
to 2–10 keV and 10–20 keV; for the burst spectra fitting of LE, ME and HE, the
energy bands used are 1–10 keV and 8–30 keV and 25–50 keV respectively.

During fittings of the persistent emission, the LE data in 2–10 keV and ME
data in 10–20 keV are used, while the ME data $>$ 20 keV and the HE spectra
are not used to fit because of faint source flux and strong background. As
shown in Fig. 2, the persistent emission detected by ME in 20–30 keV and HE in
25–100 keV is very weak compared with the background. Most of these spectral
channels are close to or fainter than the systematic uncertainty of the
background (1%); e.g., for the count rate detected by HE in 30–50 keV, the
background of $\sim$120 cts/s is comparable to the detected count rate of
$\sim$123 cts/s. Thus the persistent emission has a count rate of $\sim$ 3
cts/s which is close to the systematic uncertainty of the estimated
background.  In addition, we added a systematic uncertainty of 1% to the
Insight-HXMT spectra in 1–100 keV, to account for systematic uncertainties in
the detector calibrations (Li et al., 2020).

###  2.2 NICER

On 2020 August 8, within the same day when Insight-HXMT detected the burst
from 4U 1608–52, NICER also observed the same source. The OBSID is 3657026501,
with a good time interval $\sim$ 2 ks and a count rate $\sim$ 900 cts/s in the
0.3–12 keV band. However, the NICER missed the type-I X-ray burst because of
an observation gap.

The NICER data are reduced using the pipeline tool
nicerl2333https://heasarc.gsfc.nasa.gov/docs/nicer/nicer_analysis.html in
NICERDAS v7a with the standard NICER filtering and using ftool XSELECT to
extract lightcurves and spcectra. The background is estimated using the tool
nibackgen3C50 (Remillard et al., 2022). The Focal Plane Module (FPM) No. 14
and 34 are removed from the analysis because of increased detector noise. The
response matrix files (RMFs) and ancillary response files (ARFs) are generated
with the ftool nicerrmf and nicerarf. The spectra are rebinned by ftool
ftgrouppha (Kaastra & Bleeker, 2016) optimal binning algorithm and plus
minimum of 25 counts per grouped bin. Other rebin method, e.g., minimum of 100
counts per grouped bin by ftool grppha, are adapted. As expected, the fit
results are consistent with each other within parameter’s error bar.

For the ISM absorption, we use tbabs in the spectral model and wilm abundances
(Wilms et al., 2000). To erase the residuals $<$ 1 keV, three absorption edges
are added in spectra fitting: 0.56 keV, 0.71 keV, and 0.87 keV. We added a
systematic uncertainty of 1% to the NICER spectra.

From NICER and Insight-HXMT lightcurves, the none-burst/persistent emission is
stable in our observations. We jointly fit the persistent spectra observed
with NICER and Insight-HMXT, as shown in Fig. 2. The joint fit of the spectra
covers an energy band of 0.4–10 keV, 2–10 keV and 10–20 keV for NICER, LE and
ME, respectively.

The spectra are fitted with XSPEC v12.11.1 and the model parameters are
estimated with a 68% confidence level (1 $\sigma$).

##  3 Analysis and Results

###  3.1 None-burst/persistent emission detected by NICER and Insight-HXMT

The jointed NICER and Insight-HXMT data in a broader energy range 0.4–20 keV,
give us an opportunity to utilize a more physics meaningful model to fit the
persistent emission, rather than the simplified models, i.e., a simple photon
power law, a power law with high energy exponential rolloff (cutoffpl in
xspec) and a broken power law (bknpow in xspec). We fit the joint of NICER and
Insight-HXMT (LE and ME) spectrum with an absorbed convolution thermal
Comptonization model (with an input seed photon spectrum diskbb), available as
thcomp (a more accurate version of nthcomp) (Zdziarski et al., 2021) in XSPEC,
which is described by the optical depth $\tau$, electron temperature $kT_{\rm
e}$, scattered/covering fraction $f_{\rm sc}$.

The hydrogen column (tbabs in XSPEC) accounts for both the line of sight
column density and as well any intrinsic absorption near the source. The seed
photons are in a shape of diskbb, since the thcomp model is a convolution
model and the fraction of Comptonization photons is also given in the model.

Normalization constants are included during fittings to take into account the
inter-calibrations of the instruments. We keep the normalization factor of the
LE data with respect to the ME and NICER data to unity.

Using the model above, we find an acceptable fit:$\chi_{\upsilon}$=0.95
(d.o.f. 846; Fig. 2 and Table 3), with the inner disc radius $R_{\rm diskbb}$
and scattered/covering fraction $f_{\rm sc}$ are found to be $\sim
15.1_{-0.8}^{+0.9}$ km (with distance 4 kpc and inclination angel 40 degree)
and $0.77_{-0.05}^{+0.05}$, respectively. The derived hydrogen column density
$N_{\rm H}$ is $\sim$1.3$\times 10^{22}~{}{\rm cm}^{-2}$, which agrees with
values previously reported in a range of 0.9–1.5 $\times 10^{22}~{}{\rm
cm}^{-2}$ (PenninxW et al., 1989; Armas Padilla et al., 2017). The thcomp
parameters, $\tau$ and $kT_{\rm e}$ are well consistent with a previous
outburst in soft state of 4U 1608–52 (Armas Padilla et al., 2017), which
derived the parameters were derived with the nthcomp model. The inferred
bolometric flux in 1–100 keV is $7.33_{-0.03}^{+0.06}\times 10^{-9}~{}{\rm
erg}~{}{\rm cm}^{-2}~{}{\rm s}^{-1}$. The constant of ME and NICER is
0.93$\pm$0.02 and 1.05$\pm$0.01 respectively.

Using model of cons*tbabs*(diskbb+nthcomp) to fit the spectra, i.e., non-
convolutional Comptonization model, we get a similar results for both the
corona and the disk temperatures but with a smaller normalization of the disk.
The shortage of the disk normalization than the convolution model is
corresponding to the missing part of the disk emission which is supposed to be
scattered in the corona.

Another assumption that the seed photons of the Comptonization are from the NS
surface, i.e., the diskbb component is substituted by a blackbody component in
the aforementioned convolution model, is also attempted. Taking this approach,
spectral fits yield a roughly same thcomp parameters but with
$\chi_{\upsilon}$=1.12 (the same d.o.f.) and soft residuals $<$ 2 keV.
Furthermore, the derived blackbody radius is $34.1\pm 1.0$ km, which is far
greater than the NS radius. A hybrid model (Armas Padilla et al., 2017), i.e.,
three-component model (diskbb+thcomp*bb or bb+thcomp*diskbb) is not attempted,
since the above two-component model is able to fit the data.

Since there are no iron emission line or reflection bump above 10 keV, no
reflection model are used for the spectrum fitting.

###  3.2 Burst lightcurves by Insight-HXMT

We show the LE/ME/HE lightcurves in Fig. 1 with a time resolution of 0.1 s.
The burst profiles exhibit a typically fast rise and slow (exponential) decay
in the soft X-ray band, and manifest a plateau in soft X-ray band (LE) and two
peaks in hard X-ray band (ME&HE), which is a typical characteristic of a PRE
burst.

In the middle of the PRE phase with a constant luminosity $L_{\rm Edd}$, the
burst emission has the lowest blackbody temperature, which could cause a dip
in the HE lightcurves. However, interestingly, there is a peak/excess in the
HE lightcurves. For the highest 6 points in HE lightcurves in its whole energy
band (20–250 keV), the hard excess is 222.3$\pm$39.3 cts/s with 5.6 $\sigma$
detection; meanwhile, the burst emission for HE (for a blackbody with a
temperature of 2.0 keV and a bolometric flux of an Eddington luminosity)
should be less than 35 cts/s in this energy band. The hard excess in 30–50 keV
is 71.5$\pm$18.0 cts/s with 4 $\sigma$ detection; meanwhile, the burst
emission should be negligible with $<$ 0.1 cts/s in this energy band.

This hard X-ray excess in the lightcurve suggests that there is another
provenance except for the burst, which is also visible during the burst
spectra analysis below.

###  3.3 Broad-band spectra of burst emission by Insight-HXMT

When we fit the burst spectra, we estimate the background using the emission
before the burst, i.e., assuming the persistent emission is unchanged during
the burst. To account for the effective area calibration deviation, a constant
is added to the model. At the first attempt, for LE, the constant is fixed to
1, the others are variable during spectra fitting. The fits indicate that most
of the constants of HE and some of the constant of ME are not convergent,
owing to the low-significance data. Under this situation, the constant of
ME&HE is fixed at 1 for the combined-spectra fitting.

We follow the classical approach to X-ray burst spectroscopy by subtracting
the persistent spectrum and fitting the net spectrum with an absorbed
blackbody. In the decay phase, such a spectral model generally results in an
acceptable goodness-of-fit, with a mean reduced $\chi^{2}_{\upsilon}~{}\sim$
1.0 (d.o.f. 20–60). However, we note that a significant residuals are shown
below 3 keV and above 10 keV, as shown in the left panel of Fig. 4, especially
the spectra in the PRE phase, the reduced $\chi^{2}_{\upsilon}$ are above 1.5
(d.o.f. 60–80).

To erase the residuals, we first consider the $f_{a}$ model. Following Worpel
et al. (2013) we then include an addition component for fitting the variable
persistent emission. We assume that during the burst the spectral shape of the
persistent emission is unchanged, and only its normalization (known as a
$f_{a}$ factor) is changeable. As reported earlier by RXTE and NICER, the
$f_{a}$ model provides a better fit than the conventional one (absorbed
blackbody). We compare the above two models using the F-test. In some cases,
the $f_{a}$ model significantly improves the fits with a p-value $\sim
10^{-5}$.

As shown in left panel of Figure 3, the spectral fitting results from these
two models have differences mainly around the PRE phase. By considering an
additional factor $f_{a}$, the burst blackbody flux tends to slightly
decrease, and the temperature becomes higher but the radius shrinks. The
$f_{a}$ factor reaches a maximum of $6.5_{-1.3}^{+1.3}$ when the radius
reaches its peak. During the PRE phase, the radius is up to
$10.8_{-1.0}^{+1.2}$ km, which is two times larger than the radius measured at
touch-down time $5.1_{-0.3}^{+0.3}$ km (assuming a distance of 4 kpc). This is
typical to a moderate photospheric expansion with a bolometric burst peak flux
$F_{\rm bb}$ $15.3_{-0.8}^{+0.8}\times 10^{-8}~{}{\rm erg}~{}{\rm
cm}^{-2}~{}{\rm s}^{-1}$ in 0.1–100 keV.

Since the burst photons could also be affected by the corona/boundary-layer,
we thus check if the model used in the persistent emission could be same with
the burst emission. By taking the pre-burst emission as background emission,
the burst spectra are fitted by the model tbabs*thcomp*bb, in which the thcomp
parameters are fixed at the persistent emission fit results. Thus convolution
thermal Comptonization model (with an input seed photon spectra blackbody) has
the same d.o.f with the canonical blackbody model, and a more d.o.f. than the
$f_{a}$ model. The bb and thcomp represents the burst emission from the NS
photosphere and a corona/boundary-layer influence on the burst emission. This
model allows us to evaluate the contribution from both the up-scattered by the
corona/boundayr-layer and direct photons from the NS surface.

In the PPE phase, this model provides the best fit and yields physically
acceptable spectral parameters; the obtained best-fit parameters are given in
the right panel of Fig. 3. We find that this convolved thermal-Comptonization
model provides an equally good results with $f_{a}$ model but with a more
d.o.f., and statistically preferred to the $f_{a}$ model in the middle of PRE
phase (with a coolest blackbody temperature). However, in the rising and decay
part, such model has a bigger reduced $\chi^{2}_{\upsilon}$ than the $f_{a}$
model and even the canonical blackbody model, which may indicates that the
burst emission suffers few Comptonization during this phase.

As mentioned above, the free/unfixed parameters include the blackbody
temperature $kT_{\rm bb}$ and the normalization $N_{\rm bb}$. The trend of the
parameters are similar with the $f_{a}$ model, but with a greater change.
Compared to the $f_{a}$ model results, the maximum radius $R_{\rm bb}$ is up
to $29.5_{-2.4}^{+2.9}$ km, the minimum temperature $kT_{\rm bb}$ is low to
$1.19_{-0.06}^{+0.06}$ keV.

Other scenarios, i.e., burst reflection by the disk, NS atmosphere model
carbatm/hatm (Suleimanov et al., 2011, 2012, 2018) in Xspec, are also tried to
fit the burst spectra, as we did in Chen et al. (2019). However, neither could
alleviate the residuals at soft X-ray and hard X-ray bands simultaneously.

For the hard X-ray excess detected in the lightcurve during the PRE phase, we
calculate and find that the persistent emission has not enough flux to build
the enhancement. We fake the HE spectra using the aforementioned model
parameters of the persistent emission, the HE flux of the persistent emission
model in 20–250 keV and 30–50 keV is 3 cts/s and 0.7 cts/s. Taking into the
factor $f_{a}$ account, this model predicted enhancement flux only equivalent
to one tens of the hard excess. The spectra residuals in hard X-ray band are
also visible in the middle panel of Fig. 4 ($f_{a}$ model to fit the burst
spectra in the PRE phase). The hard X-ray excess also disfavours the
reflection model because of the faint persistent emission in hard X-ray band.

##  4 Discussion

In this work, we have presented a spectral analysis of a PRE burst and
persistent emission from 4U 1608–52 in its 2020 outburst observed by NICER and
Insight-HXMT. The persistent emission is well fitted by an absorbed
convolution thermal-Comptonization model, in which 77% disk emission is up-
scattered by the corona/boundary-layer. The X-ray burst shows a significant
spectral deviation/excess both at $<$ 3 keV and $>$ 10 keV from an absorpted
blackbody in the PRE phase. This excess is consistent with that the burst
emission is up-scattered by the corona/boundary-layer, only part of the burst
emission without Comptonization is detected, which mimics the Comptonization
of the disk emission in the persistent emission.

###  4.1 X-ray continuum

Based on LE&ME lightcurves and spectral fitting results, the burst locates at
the high/soft state (banana state). Previous works have attempt to fit the
spectra with thermal (diskbb or/and blackbody) plus a Comptonization model,
rather than a convolution thermal-Comptonization model, which will cause an
underestimation of the thermal emission. In this work, adapting thermal-
Comptonization model thcomp in XSPEC, the fit results indicate that most of
the disk emission is involved in Compton upscattering.

Broadly speaking, in an accreting low-magnetic field NS, except for the
emission from the NS surface, there are at least two geometrically distinct
regions to generate X-ray emission (see the review by Done et al., 2007): in
the accretion disc and the boundary/spreading layer (BL/SL) (similar to the
corona of the case of an accreting black hole). BL is supposed to spread in
large radial extent in the disc midplane; whereas SL has a narrower spread but
spreads over a considerable height from the equatorial plane toward higher
stellar latitudes. There are claimed judgment criteria for BL and SL based on
temporal (Gilfanov et al., 2003) and spectral (Grebenev & Sunyaev, 2002;
Suleimanov & Poutanen, 2006) characteristics.

In the burst review, during the decay part of the burst, the burst emission is
well fitted by a blackbody and no strong comptonization/up-scattered emission
detected. Thus the hot electrons plasma should not have a significant coverage
for the NS surface. Add that into the consideration with a big scattering
factor $f_{\rm sc}$ 0.77${}_{-0.05}^{+0.05}$ (the hot electrons plasma on the
disk in the persistent emission), the corona-like geometry of the BL is
favoured.

We find that the persistent emission is 4.8% $L_{\rm Edd}$ and the
corona/boundary-layer temperature is 3.02${}_{-0.08}^{+0.08}$ keV, which is in
the range of Comptonizing temperature expected for NS LMXBs in the soft state
(Armas Padilla et al., 2017). Meanwhile, the scattering factor $f_{\rm sc}$ is
0.77${}_{-0.05}^{+0.05}$, which is too large for the corona/boundary-layer
with a lamp-post geometry. Given those above, we prefer the corona/boundary-
layer with a slab/sandwich geometry, as shown in Figure 7. Regarding that the
temperature and optical depth deviates from the corona canonical value, we
also prefer anther corona pattern–a so-called warm layer (Zhang et al., 2000)
with temperature $\sim$2–3 keV and optical depth $\sim$5–10, which is produced
by the magnetic reconnection. The outburst spectral evolution and its
understandings will be given in our forthcoming paper.

###  4.2 Enhanced Persistent Emission up to 50 keV

In several bursters, during the low/hard state, decrease/deficit in the hard
X-ray band (30–50 keV) have been observed in short-duration bursts which
happened in the low-hard state (Chen et al., 2012 and reference therein). It
is expected that the burst emission (2–3 keV), which is relatively cooler than
the corona (tens keV), cause the change in the corona structure or
temperature.

In this work, conversely, an enhancement hard X-ray emission are observed
during a short-duration burst, which is first reported in GS 1826–238 by
BeppoSAX in 30–60 keV (in’t Zand et al., 1999). However, these two sets of
bursts located different spectral state of LMXBs. In both cases, the soft
X-ray showers of the burst may manifest as an enhancement of the input seed
photons but not a sufficient cooler upon the corona.

Compared with the disk component of the persistent emission, the count rate of
the burst at the PRE phase is 4 times more. The emergent photons of the burst
could be up-scattered to higher energy by the corona/boundary-layer. For the
comptonization of the burst during PRE phase, i.e., a blackbody with
temperature 1.22 keV and normalization 1.33, and a hot corona with temperature
of 3 keV, optical depth 10.2 and cover-factor 0.76, we fake a spectrum induced
by the inverse Compton scattering of the blackbody emission and get a count
rate 289 cts/s in the energy band of 30–50 keV. Thus, the up-scattered photons
of the burst do cause an enhancement hard X-ray emission.

Based on the burst spectra fit results, the enhancement hard X-ray emission
could be related to the up-scattered burst emission by the corona/boundary-
layer, just like the situation in the persistent emission, rather than
enhancement accretion rate manifesting itself by elevating persistent emission
with unaltered spectral shape (Worpel et al., 2013, 2015).

###  4.3 Dynamical evolution of the disk geometry

As a common sense, the burst emission has an increasing and decreasing area
during its rise and decay phase, which corresponding the hot spot spreading in
the NS surface. Meanwhile, there are at least two moments when the hot spot
covers that whole NS surface, the photoshphere lift-up point and the touch-
down point for the PRE burst. As shown in Figure 1 of Shaposhnikov et al.
(2003), the hot-spot spreads on the NS surface and then lifts up the
photonsphere, i.e., from ’a’ stage to ’b’ stage in the Figure. There should be
a moment that hot spot covers that whole NS surface in the rise phase and vice
versa in the decay phase. However, there are some PRE bursts with a short
increase time, i.e., that the increase time is too short for telescope to
accumulating enough counts in the first moment when the hot spot covers that
whole NS surface.

In practice, the latter is usually used to derive the NS radius. As shown in
Figure 5, at the touch-down point, the burst emission reaches its peak flux,
both for the $f_{a}$ model and convolution thermal-Comptonization model. A
dynamical evolution of the disk geometry could cause this phenomenon, i.e.,
the lower NS hemisphere, which is obscured before the burst (the burst PRE
phase), appears from the disk after the burst-disk interaction, as shown in
Figure 1 of Shaposhnikov et al. (2003) and Figure 7 in this work.

In theory, Poynting-Robertson drag could drain the inner-accretion-disk by
taking away the momentum of the accretion matter hence enlarging the local
accretion rate (in’t Zand et al., 2013; Worpel et al., 2013, 2015), which is
faster than it is being refilled (Stahl et al., 2013; Fragile et al., 2020).
At this moment, the inner part of the disk is hollowed out by the burst
emission.

The flux-temperature diagram of the burst also indicates that the inner disk
radius change causes a bigger visible part of the NS surface. If the whole NS
surface shows up as a single-temperature blackbody and a constant color
correction factor, the burst flux $F$ should scale as $kT_{\rm bb}^{4}$ in the
flux-temperature diagram, and the slope represents the emitting area in the
double logarithmic coordinates (G$\ddot{u}$ver et al., 2012). As shown in
Figure 6, the rising phase and decaying phase obey different $F\propto kT_{\rm
bb}^{4}$. We fit the two sets with $F=\frac{R^{2}}{D^{2}}\sigma T^{4}$, the
blackbody radius of the rising and decaying phase is 4.3$\pm$0.11 km and
5.3$\pm$0.05 km with $D=4$ kpc.

Assuming the NS radiates at the Eddington limit in the PRE phase and the disk
reaches the NS surface before the PRE phase, the blackbody flux ratio detected
at the rising phase $F_{\rm rise}$ and decaying phase $F_{\rm decay}$ is
positively associated with inclination angel $i$, i.e.,$\frac{F_{\rm
rise}}{F_{\rm decay}}=(1+{\rm cos}~{}i)/2$ (Shaposhnikov et al., 2003;
Shaposhnikov & Titarchuk, 2004). The inclination angel $i$ is estimated as
$\sim$ 70∘. However, this result is bigger than the value $\sim$ 30∘–40∘
derived from the spectral fit results on an outburst of 4U 1608–52 by a
reflection model (Degenaar et al., 2015).

###  4.4 Corona/boundary-layer reacting on burst

The interaction between the burst and inhabited persistent emission was first
studied from the long-duration, brightest and most vigorous PRE bursts with
moderate/super expansions in 4U 1820–30 and 4U 1636–536 (Ballantyne &
Strohmayer, 2004; Keek et al., 2014) (a factor of $\sim$10–104 increases in
emission area). Then in short-duration bursts, this interaction was mainly
observed as the persistent spectral change, rather than the burst spectral
change, i.e., enhancement accretion rate, deficit at hard X-ray band,
reflection by the disk and driven outflow.

Type-I X-ray burst happens on the NS surface, which is also in the accretion
environment. In principle, the burst spectrum may be influenced due to the
Comptonization of the burst photons by the surrounding corona/boundary-layer
(Chen et al., 2019). A Comptt component was reported in bursts of 4U 1608–52
from RXTE observations above 3 keV (Kajava et al., 2017). However, their
approach resembled the $f_{a}$ model since the Comptt component is added in
the spectra fitting. In this work, the persistent emission is well fitted by a
convolution thermal-Comptonization model. As a result, given the similarity,
for the burst, adapting the convolution model with the parameters in the
persistent emission fitting but with a blackbody emission, this could also fit
the short-duration burst spectra in the PRE phase. The goodness of the fit are
comparable with the $f_{a}$ model, but with a colder $kT$ and larger $R$ in
the middle of the PRE phase. If this is the case, the radius of the
photosphere is underestimated with the canonical blackbody model or the
$f_{a}$ model.

In principle, the interaction between burst and accretion environment, might
be expected to have spectral evolution for both of the burst emission and
accretion emission during burst, with spectral shape deviation from pure
blackbody and the model of the pre-burst emission. The short duration and
rapid spectral change limits the accumulated time and photon counts, which in
turn requires larger detection area and broad band energy coverage which may
be satisfied by the next generation of Chinese mission of so-called eXTP
(enhanced X-ray Timing and Polarimetry mission) (Zhang et al., 2019) or a
contemporary joint observation of the burst by NICER and Insight-HXMT.

We thank the reviewer for the constructive feedback and comments that greatly
improved the quality of this paper. This work made use of the data and
software from the Insight-HXMT mission, a project funded by China National
Space Administration (CNSA) and the Chinese Academy of Sciences (CAS). This
research has made use of data and software provided by of data obtained from
the High Energy Astrophysics Science Archive Research Center (HEASARC),
provided by NASA’s Goddard Space Flight Center. This work is supported by the
National Key R&D Program of China (2021YFA0718500) and the National Natural
Science Foundation of China under grants 11733009, U1838201, U1838202,
U1938101, U2038101.

## References

  * Armas Padilla et al. [2017] Armas Padilla, M., Ueda, Y., Hori, T., Shidatsu, M., Munoz-Darias, T., 2017, MNRAS, 467, 290
  * Ballantyne & Strohmayer [2004] Ballantyne, D. R., & Strohmayer, T. E. 2004, ApJL, 602, L105
  * Bhattacharyya et al. [2018] Bhattacharyya, S. Yadav, J. S., Sridhar, Navin, et al. 2018, ApJ, 860, 88
  * Belian et al. [1976] Belian, R. D., Conner, J. P., & Evans, W. D. 1976, ApJ, 206, L135
  * Chen et al. [2012] Chen, Y. P., Zhang, S., Zhang, S. N., et al. 2012, ApJL, 752, 34
  * Chen et al. [2018] Chen, Y. P., Zhang, S., Qu, J. L., Zhang, S. N., et al. 2018, ApJL, 864, 30
  * Chen et al. [2019] Chen, Y. P., Zhang, S., Zhang, S. N., et al. 2019, Journal of High Energy Astrophysics, 24, 23
  * Cumming [2004] Cumming, A. 2004, Nucl. Phys. B Proc. Suppl., 132, 435
  * Degenaar et al. [2015] Degenaar, N., Miller, J. M., Chakrabarty, D., Harrison, F. A., Kara, E., & Fabian, A. C. 2015, MNRAS, 451, L85
  * Degenaar et al. [2018] Degenaar, N., Ballantyne, D. R., Belloni, T., et al. 2018, SSRv, 214, 15
  * Done et al. [2007] Done, C., Gierliński, M., Kubota, A. 2007, A&AR, 15, 1
  * Fragile et al. [2020] Fragile, P. C., Ballantyne, D. R., & Blankenship, A. 2020, NatAs, 4, 541
  * Galloway et al. [2008] Galloway, D. K., Muno, M. P., Hartman, J. M., et al. 2008, ApJS, 179, 360
  * Galloway et al. [2020] Galloway, D. K., In’t Zand, J., Chenevez, J., et al. 2020, ApJS, 249, 32
  * G$\ddot{u}$ver et al. [2012] G$\ddot{u}$ver, T., Psaltis, D., & Zel, F. 2012, ApJ, 747, 76
  * Gilfanov et al. [2003] Gilfanov, M.; Revnivtsev, M.; Molkov, S.Gilfanov et al. 2003, A&A, 410, 217
  * Grebenev & Sunyaev [2002] Grebenev, S. A., & Sunyaev, R. A. 2002, AstL, 28, 150
  * in’t Zand et al. [1999] in’t Zand, J. J. M., Heise, J., Kuulkers, E., et al. 1999, A&A, 347, 891
  * in’t Zand et al. [2013] in’t Zand, J. J. M., Galloway, D. K., Marshall, H. L., et al. 2013, A&A,553, A83
  * Ji et al. [2013] Ji, L., Zhang, S., Chen, Y. P., et al. 2013, MNRAS, 432, 2773
  * Kaastra & Bleeker [2016] Kaastra, J. S.; Bleeker, J. A. M. 2016, A&A, 587, A151
  * Kajava et al. [2017] Kajava, J. J. E., Koljonen, K. I. I., N$\ddot{a}$ttil$\ddot{a}$, J., Suleimanov, V., & Poutanen, J. 2017, MNRAS, 472, 78
  * Keek et al. [2014] Keek, L., Ballantyne, D. R., Kuulkers, E., & Strohmayer, T. E. 2014b, ApJL, 797, L23
  * Keek et al. [2018] Keek, L., Arzoumanian, Z., Bult, P., et al. 2018, ApJL, 855,4
  * Keek et al. [2018a] Keek, L., Arzoumanian, Z., Chakrabarty, D., et al. 2018, ApJL, 856,37
  * Lewin et al. [1993] Lewin, W. H. G., van Paradijs, J., & Taam, R. E. 1993, Space Sci. Rev., 62, 223
  * Li et al. [2020] Li, X. B., Li, X. F., Tan, Y. et al. 2020, JHEA, 27, 64
  * Muno et al. [2001] Muno, M. P., Chakrabarty, D., Galloway, D. K., & Savov, P. 2001, ApJ, 553, L157
  * PenninxW et al. [1989] PenninxW., Damen E., van Paradijs J., Tan J., Lewin W. H. G., 1989, A&A, 208, 146
  * Poutanen et al. [2014] Poutanen, J., N$\ddot{ai}$ttil$\ddot{a}$, J., Kajava, J. J. E. et al. 2014, MNRAS, 442, 3777
  * Sánchez-Fernández et al. [2020] Sánchez-Fernández C., Kajava J. J. E., Poutanen J., et al., 2020, A&A, 634, A58
  * Remillard et al. [2022] Remillard, R. A., Loewenstein, M., Steiner, J. F. et al. 2022, AJ, 163, 130
  * Suleimanov & Poutanen [2006] Suleimanov, V., & Poutanen, J. 2006, MNRAS, 369, 2036
  * Shaposhnikov et al. [2003] Shaposhnikov, N., Titarchuk, L., Haberl, F., 2003, ApJL, 593, L35
  * Shaposhnikov & Titarchuk [2004] Shaposhnikov, N., & Titarchuk, L. 2004, ApJ, 606, L57
  * Stahl et al. [2013] Stahl, A., Klu$\acute{z}$niak, W., Wielgus, M., & Abramowicz, M. 2013, A&A, 555, A114
  * Strohmayer & Bildsten [2006] Strohmayer, T., & Bildsten, L. 2006, New views of thermonuclear bursts (Compact stellar X-ray sources), 113, 156
  * Suleimanov et al. [2011] Suleimanov, V., Poutanen, J., Werner, K. 2011, A&A, 527, A139
  * Suleimanov et al. [2012] Suleimanov, V., Poutanen, J., Werner, K. 2012, A&A, 545, A120
  * Suleimanov et al. [2018] Suleimanov, V., Poutanen, J., Werner, K. 2018, A&A, 619, A114
  * Wilms et al. [2000] Wilms, J., Allen, A., & McCray, R. 2000, ApJ, 542, 914
  * Worpel et al. [2013] Worpel, H., Galloway, D. K., & Price, D. J. 2013, ApJ, 772, 94
  * Worpel et al. [2015] Worpel, H., Galloway, D. K., & Price, D. J. 2015, ApJ, 801, 60
  * Zhang et al. [2000] Zhang, S. N., et al., 2000, Science, 287, 1239
  * Zhang et al. [2014] Zhang, S., L,u F. J., Zhang, S. N. et al. in Space Telescopes and Instrumentation 2014: Ultraviolet to Gamma Ray, Proc. SPIE, Vol. 9144 (2014) p. 914421
  * Zhang et al. [2019] Zhang, S. N., Santangelo, A., Feroci, M., et al. 2019, Science China Physics, Mechanics & Astronomy, Volume 62, Issue 2, article id. 29502, 25
  * Zhang et al. [2020] Zhang, S.-N., Li, T.-P., Lu, F.-J., et al. 2020, SCPMA, 63, 249502
  * Zdziarski et al. [2021] Zdziarski, A. A., Szanecki, M., Poutanen, J., Gierlinski, M., & Biernacki, P. 2021, MNRAS, 492, 5234

Table 1: The bursts obsid and peak time of 4U 1608–52 detected by Insight/HXMT
in 2020 ourbutst

obsid | Start Time | Elapsed Time (s) | LE GTI (s) | ME GTI (s) | Burst peak time (MJD)
---|---|---|---|---|---
P030402100401∗ | 59069.63746 (2020-08-08T15:16:50) | 14708 | 2480 | 3750 | 59069.77077
P030402100602 | 59079.83751 (2020-08-18T20:04:55) | 11617 | 120 | 480 | 59079.85655

  * ${\mathrm{*}}$

This work.

  * Note:

Both bursts are detected by the three payloads of Insight-HXMT. However, the
second bursts, which is only half bright as the first one, did not located the
good-time-interval.

Table 2: The NICER obsid at the same day when Insight-HXMT detected the burst  obsid | Start Time | Elapsed Time (s) | GTI (s)
---|---|---|---
3657026501 | 59069.61551 (2020-08-08T14:43:40) | 3600 | 1707
Table 3: The results of the spectral fit of the LE, ME and NICER spectra in the 0.4–20 keV range with cons*tbabs*thcomp*diskbb $N_{\rm H}$ | $\tau$ | $kT_{\rm e}$ | $f_{\rm sc}$ | $kT_{\rm in}$ | $N_{\rm diskbb}$ | $\chi_{\nu}^{2}$
---|---|---|---|---|---|---
$10^{22}~{}{\rm cm}^{-2}$ |  | keV |  | keV | $10^{2}$ |
$1.33_{-0.01}^{+0.01}$ | $10.2^{+0.3}_{-0.4}$ | $3.02^{+0.08}_{-0.08}$ | $0.76_{-0.05}^{+0.05}$ | $0.68_{-0.02}^{+0.01}$ | $8.38_{-0.96}^{+1.03}$ | 802/846

Figure 1:  Net lightcurves of the type-I X-ray burst detected in the Insight-
HXMT observation of 4U 1608–52 with time bin 0.1 s. The top, 2nd, 3rd and
bottorm panel is LE, ME, HE results in their full energy band and HE in 30–50
keV respectively. The red lines indicates the PRE phase. The arrows indicates
the hard X-ray excess during the PRE phase.  Figure 2: The spectral fit
results of the persistent emission by LE (black), ME (red) and NICER (green)
with model cons*tbabs*thcomp*diskbb. The embedded panel shows the background
level for ME (red) and HE (blue).

Figure 3:  Spectral fitting result of the burst with time bin 0.25 second with
a pure blackbody (black), $f_{a}$ model (the left panel, red) and and
convolution thermal-Comptonization model (the right panle, red), include the
time evolution of the blackbody bolometric flux $F_{\rm bb}$, the temperature
$kT_{\rm bb}$, the observed radius $R$ of NS surface at 4 kpc, the goodness of
fit $\chi_{v}^{2}$. The bolometric flux of the blackbody model $F_{\rm bb}$ is
in unit of $10^{-8}~{}{\rm erg/cm}^{2}/{\rm s}$.  Figure 4:  Top panel: the
spectral fits results by LE (black), ME (red) and HE (green) when the burst
reaches the maximum emission area by $f_{a}$ model, the blackbody model and
enhancement of the persistent emission are labeled. The three panels below:
residuals of spectral fits results by an absorbed black-body model (the 2nd
panel), $f_{a}$ model (the 3rd panel) and convolution thermal-Comptonization
model (the bottom panel).

Figure 5: Bolometric flux evolution of the bursts detected by Insight-HXMT
with the $f_{a}$ model (left) and convolution thermal-Comptonization model
(right). The bolometric flux (in unit of $10^{-8}~{}{\rm erg/cm}^{2}/{\rm s}$)
evolution of the enhanced persistent emission (green), the burst emission
(red) and the sum of the former two components (blue).  Figure 6: The flux-
temperature diagram for the burst spectra in the rising (blue) and decaying
(red) phase from 4U 1608–52. The diagonal blue and red lines correspond to
constant radius in the rising and decaying phase. The time sequence is that
the burst starts at the lower right, stays at PRE phase (black) at the upper
left and decays at the lower left.

Figure 7: Illustration of the central region of an NS XRB before the PRE phase
(left), in the PRE phase (middle) and after the PRE phase (right) during a
burst, in which the inner part of the disk is swept away by the burst in the
PRE phase.


# 3D Concept Grounding on Neural Fields

Yining Hong
University of California, Los Angeles
&Yilun Du
Massachusetts Institute of Technology
&Chunru Lin
Shanghai Jiao Tong University
&Joshua B. Tenenbaum
MIT BCS, CBMM, CSAIL
&Chuang Gan
UMass Amherst and MIT-IBM Watson AI Lab

###### Abstract

In this paper, we address the challenging problem of 3D concept grounding
(_i.e_. segmenting and learning visual concepts) by looking at RGBD images and
reasoning about paired questions and answers. Existing visual reasoning
approaches typically utilize supervised methods to extract 2D segmentation
masks on which concepts are grounded. In contrast, humans are capable of
grounding concepts on the underlying 3D representation of images. However,
traditionally inferred 3D representations (e.g., point clouds, voxelgrids and
meshes) cannot capture continuous 3D features flexibly, thus making it
challenging to ground concepts to 3D regions based on the language description
of the object being referred to. To address both issues, we propose to
leverage the continuous, differentiable nature of neural fields to segment and
learn concepts. Specifically, each 3D coordinate in a scene is represented as
a high dimensional descriptor. Concept grounding can then be performed by
computing the similarity between the descriptor vector of a 3D coordinate and
the vector embedding of a language concept, which enables segmentations and
concept learning to be jointly learned on neural fields in a differentiable
fashion. As a result, both 3D semantic and instance segmentations can emerge
directly from question answering supervision using a set of defined neural
operators on top of neural fields (e.g., filtering and counting). Experimental
results show that our proposed framework outperforms unsupervised / language-
mediated segmentation models on semantic and instance segmentation tasks, as
well as outperforms existing models on the challenging 3D aware visual
reasoning tasks. Furthermore, our framework can generalize well to unseen
shape categories and real scans***Project page: http://3d-cg.csail.mit.edu.

##  1 Introduction

Consider an image of a table pictured in Figure 1. We wish to construct a
method that is able to accurately ground the concepts and reason about the
image such as the number of legs the pictured table has. Existing works
typically address this problem by utilizing a supervised segmentation model
for legs, and then applying a count operator on extracted segmentation masks
[19]. However, as illustrated in Figure 1, in many visual reasoning tasks, the
correct answer depends on a very small portion of a given image, such as a
highly occluded back leg, which many existing 2D segmentation systems may
neglect. Humans are able to accurately ground the concepts from images and
answer such questions by reasoning on the underlying 3D representation of the
image [32]. In this underlying 3D representation, the individual legs of a
table are roughly similar in size and shape, without underlying issues of
occlusion of different legs, making reasoning significantly easier and more
flexible. In addition, such an intermediate 3D representation further
abstracts confounding factors towards reasoning, such as the underlying
viewing direction from which we see the input shape. To resemble more human-
like reasoning capability, in this paper, we propose a novel concept grounding
framework by utilizing an intermediate 3D representation.

A natural question arises – what makes a good intermediate 3D representation
for concept grounding? Such an intermediate representation should be
discovered in a self-supervised manner and efficient to infer, as well as
maintain correspondences between 3D coordinates and feature maps in a fully
differentiable and flexible way, so that segmentation and concept learning can
directly emerge from this inferred 3D representation with natural supervision
of question answering. While traditional 3D representations (e.g., point
clouds, voxelgrids and meshes) are efficient to infer, they can not provide
continuous 3D feature correspondences flexibly, thus making it challenging to
ground concepts to 3D coordinates based on the language descriptions of
objects being referred to. To address both issues, in this work, we propose to
leverage the continuous, differentiable nature of neural fields as the
intermediate 3D representation, which could be easily used for segmentation
and concept learning through question answering.

Figure 1: Issues with 2D Concept Grounding and Visual Reasoning.  Existing
methods typically answer questions by relying on 2D segmentation masks
obtained from supervised models. However, heavy occlusion leads to incorrect
segmentations and answers.

To parameterize a neural field for reasoning, we utilize recently proposed
Neural Descriptor Fields (NDFs) [30] which assign each 3D point in a scene a
high dimensional descriptor. Such descriptors are learned in a self-supervised
manner, and implicitly capture the hierarchical nature of a scene, by
leveraging implicit feature hierarchy learned by a neural network. Portions of
a scene relevant to a particular concept could be then differentiably
extracted through a vector similarity computation between each descriptor and
a corresponding vector embedding of the concept, enabling concept
segmentations on NDFs to be differentiably learned. In contrast, existing
reasoning approaches utilize supervised segmentation models to pre-defined
concepts, which prevents reasoning on concepts unseen by the segmentation
model [19], and further prevents models from adapting segmentations based on
language descriptions.

On top of NDFs, we define a set of neural operators for visual reasoning.
First, we construct a filter operator, and predict the existence of a
particular concept in an image. We also have a query operator which queries
about an attribute of the image. In addition, we define a neural counting
operator, which quantifies the number of instances of a particular concept. We
find that by integrating our neural operators with NDF, our framework is able
to effectively and robustly answer a set of visual reasoning questions.
Simultaneously, we observe the emergence of natural 3D segmentations of
concepts, at the semantic and instance level, directly from the underlying
supervision of downstream reasoning.

Our contributions can be summed up as follows: 1) we propose 3D-CG, which
utilizes the differentiable nature of neural descriptor fields (NDF) to ground
concepts and perform segmentations; 2) we define a set of neural operators,
including a neural counting operator on top of the NDF; 3) with 3D-CG,
semantic and instance segmentations can naturally emerge from question
answering supervision; 4) our 3D-CG outperforms baseline models in both
segmentation and reasoning tasks; 5) it can also generalize well to unseen
shape categories and real scans.

##  2 Related works

##### Visual Reasoning.

There have been different tasks focusing on learning visual concepts from
natural language, such as visually-grounded question answering [8, 9] and
text-image retrieval [33]. Visual reasoning stands out as a challenging task
as it requires human-like understanding of the visual scene. Numerous visual
reasoning models have been proposed in recent years. Specifically, MAC [13]
combined multi-modal attention for compositional reasoning. LCGN [12] built
contextualized representations for objects to support relational reasoning.
These methods model the reasoning process implicitly with neural networks.
Neural-symbolic methods [37, 19, 1] explicitly perform symbolic reasoning on
the objects representations and language representations. Specifically, they
use perception models to extract 2D masks for 3D shapes as a first step, and
then execute operators and ground concepts on these pre-segmented masks, but
are limited to a set of pre-defined concepts. In this paper, we present an
approach towards grounding 3D concepts in a fully differentiable manner, with
which 3D segmentations can emerge naturally with question answering
supervision.

##### Neural Fields.

Our approach utilizes neural fields also known as neural implicit
representations, to parameterize an underlying 3D geometry of a shape for
reasoning. Such fields have been shown to accurately represent 3D geometry
[23, 3, 26, 27, 28], dynamic scenes [24, 7], and appearance [31, 21, 25, 36,
29], acoustics [17] and more general multi-modal signals [6]. In this work, we
utilize descriptors defined on such fields [30] to differentiably reason in
3D.

##### Language-driven Segmentation.

Recent works have been focused on leveraging language for segmentation.
Specifically, BiLD [10] distills the knowledge from a pre-trained zero-shot
image classification model into a two-stage detector. MDETR [14] is an end-to-
end modulated detector that detects objects in an image conditioned on a raw
text query, like a caption or a question. LSeg [15] uses a text encoder to
compute embeddings of descriptive input labels together with a transformer-
based image encoder that computes dense per-pixel embeddings of the input
image. GroupViT [35] learns to group image regions into progressively larger
arbitrary-shaped segments with a text encode. These methods typically use an
encoder to encode the text and do not have the ability to modify the
segmentations based on the question answering loss. LORL [34] uses the object-
centric concepts derived from language to facilitate the learning of object-
centric representations. However, they can only improve the segmentation
results but cannot generate segmentations from scratch with language
supervision.

##  3 3D Concept Grouding

In Figure 2, we show an overall framework of our 3D Concept Grounding (3D-CG),
which seamlessly bridges the gap between 3D perception and reasoning by using
Neural Descriptor Field (NDF). Specifically, we first use NDF to assign a high
dimensional descriptor vector for each 3D coordinate, and run a semantic
parsing module to translate the question into neural operators to be executed
on the neural field. The concepts, also the parameters of the operators,
possess concept embeddings as well. Upon executing the operators and grounding
concepts in the neural field, we perform dot product attention between the
descriptor vector of each coordinate and the concept embedding vector to
calculate the score (or mask probability) of a certain concept being grounded
on a coordinate. We also propose a count operator which assigns point
coordinates into different slots for counting the number of instances of a
part category. Both the NDF and the neural operator have a fully
differentiable design. Therefore, the entire framework can be end-to-end
trained, and the gradient from the question answering loss can be utilized to
optimize both the NDF perception module and the visual reasoning module and
jointly learn all the embeddings and network parameters.

Figure 2: Our 3D Concept Grounding (3D-CG) framework. Given an input image and
a question, 3D-CG first utilizes a Neural Descriptor Field (NDF) to extract a
descriptor vector for each 3D point coordinate (here we take 5 coordinates
with 5 different colors as examples), and a semantic parsing network to parse
the question into operators. The mask probabilities of the coordinates are
initialized with occupancy values. Concepts are also mapped to embedding
vectors. Attention is calculated between the descriptors and concept
embeddings to yield new mask probabilities and filter the set of coordinates
being referred as the concept. We also define a count operator which segments
a part category into different slots. The sum of the maxpooled mask
probabilities of all slots can be used for counting, and the loss can be back
propagated to optimize NDF as well as the embeddings. Semantic and instance
segmentations naturally emerge with 3D-CG.

###  3.1 Model Details

####  3.1.1 Neural Descriptor Fields

In this paper, we utilize a neural field to map a 3D coordinate $\mathbf{x}$
using a descriptor function $f(\mathbf{x})$ and extract the feature descriptor
of that 3D coordinate:

$f(\mathbf{x}):\mathbb{R}^{3}\rightarrow\mathbb{R}^{n},\quad\mathbf{x}\mapsto
f(\mathbf{x})=\mathbf{v}$ (1)

where $\mathbf{v}$ is the descriptor representation which encodes information
about shape properties (e.g., geometry, appearance, etc).

In our setting, we condition the neural field on the partial point cloud
$\mathcal{P}$ derived from RGB-D images. We use a PointNet-based encoder
$\mathcal{E}$ to encode $\mathcal{P}$. The descriptor function then becomes
$f(\mathbf{x},\mathcal{E}(P))=\mathbf{v}$. This continuous and differentiable
representation maps each 3D coordinate $\mathbf{x}$ to a descriptor vector
$\mathbf{v}$. Since the concepts to be grounded in most reasoning tasks focus
on geometry and appearance, we leverage two self-supervised pre-training tasks
to parameterize $f$ and learn the features concerning these properties.

##### Shape Reconstruction.

Inspired by recent works on using occupancy networks [20] to reconstruct
shapes and learn shape representations, we also use an MLP decoder
$\mathcal{D}_{1}$ which maps each descriptor vector $\mathbf{v}$ to an
occupancy value: $\mathcal{D}_{1}:\mathbf{v}\mapsto\mathbf{o}\in[0,1]$, where
the occupancy value indicates whether the 3D coordinate is at the surface of a
shape.

##### Color Reconstruction.

We use another MLP $\mathcal{D}_{2}$ to decode an RGB color of each
coordinate: $\mathcal{D}_{2}:\mathbf{v}\mapsto\mathbf{c}\in\mathbb{R}^{3}$,
where the color value indicates the color of the 3D coordinate on the shape.

After learning decoders $D_{i}$ through these self-supervised pre-training
tasks, to construct the Neural Descriptor Field (NDF) $f({\bm{x}})$, we
concatenate all the intermediate activations of $D_{i}$ when decoding a 3D
point $\mathbf{x}$ [30]. The resultant descriptor vector $\mathbf{v}$ can then
be used for concept grounding. Since $\mathbf{v}$ consists of intermediate
activations of $D_{i}$, they implicitly capture the hierarchical nature of a
scene, with earlier activations corresponding to lower level features and
later activations corresponding to higher level features.

####  3.1.2 Concept Quantization

Visual reasoning requires determining the attributes (e.g., color, category)
of a shape or shape part, where each attribute contains a set of visual
concepts (e.g., blue, leg). As shown in Figure 2, visual concepts such as
legs, are represented as vectors in the embedding space of part categories.
These concept vectors are also learned by our framework. To ground concepts in
the neural fields, we calculate the dot products $\langle\cdot,\cdot\rangle$
between the concept vectors and the descriptor vectors from the neural field.
For example, to calculate the score of a 3D coordinate belonging to the
category leg, we take its descriptor vector $\mathbf{v}$ and compute
$p=\langle\mathbf{v},v^{leg}\rangle$, where $v^{leg}$ is the concept embedding
of leg.

####  3.1.3 Semantic Parsing

To transform natural language questions into a set of primitive operators that
can be executed on the neural field, a semantic parsing module is incorporated
into our framework. This module utilizes a LSTM-based Seq2Seq to translate
questions into a set of fundamental operators for reasoning. These operators
takes concepts and attributes as their parameters (e.g., Filter(leg),
Query(color)).

####  3.1.4 Neural operators on Neural Descriptor Fields

The operators extracted from natural language questions can then be executed
on the neural field. Due to the differentiable nature of the neural field, the
whole execution process is also fully-differentiable. We represent the
intermediate results in a probabilistic manner: for the $i$-th sampled
coordinate $\mathbf{x}_{i}$, it is represented by a descriptor vector
$\mathbf{v}_{i}$ and has an attention mask $\mathbf{m}_{i}\in[0,1]$.
$\mathbf{m}_{i}$ denotes the probability that a coordinate belongs to a
certain set, and is initialized using the occupancy value output by
$\mathcal{D}_{1}$: $\mathbf{m}_{i}=\mathbf{o}_{i}$.

There are three kinds of operators output by the semantic parsing module, we
will illustrate how each operator executes on the descriptor vectors to yield
the outputs.

##### Filter Operator.

The filter operator “selects outs" a set of coordinates belonging to the
concept $c$ by outputting new mask probabilities:

$\text{Filter}(c):\mathbf{m}_{i}^{c}=min(\mathbf{m}_{i},\langle\mathbf{v}_{i},v^{c}\rangle)$
(2)

##### Query Operator.

The query operator asks about an attribute $a$ on selected coordinates and
outputs concept $\hat{c}$ of the maximum probability:

$\text{Query}(a):\hat{c}=\arg\max
min(\mathbf{m}_{i},\langle\mathbf{v}_{i},v^{c}\rangle),c\in a$ (3)

##### Count Operator.

The count operator intends to segment the coordinates in the same part
category into instances and count the number of instances. Inspired by
previous unsupervised segmentation methods which output different parts in
various slots [16, 2], we also allocate a set of slots for different instances
of a category. The $j$-th slot $s_{j}$ has its own embedding vector
$v^{s}_{j}$. The score $s_{ij}$ of the $i$-th coordinate belonging to the
$j$-th slot is also computed by dot product
$s_{ij}=\langle\mathbf{v},v^{s_{j}}\rangle$. We further take softmax to
normalize the probabilities across slots. Since we want to count the instances
of a part that was previously selected out, we also take the minimum of
previous mask probabilities and the mask probabilities of each slot. The
result of counting (i.e., the number of part instances $n$) is obtained by
summing up the maximum probability in each slot.

$\displaystyle\text{Count}(c):\mathbf{m}_{ij}$
$\displaystyle=min(\mathbf{m}_{i}^{c},\frac{s_{ij}}{\sum s_{i^{\prime}j}})$
(4) $\displaystyle n$ $\displaystyle=\sum_{j}\max_{i}\mathbf{m}_{ij}$ (5)

####  3.1.5 Segmentation

Note that based on the above operators, not only visual reasoning can be
performed, but we can also do semantic segmentation and instance segmentation.

##### Semantic Segmentation.

We can achieve semantic segmentations by executing Query(category) on each
coordinate and output the category $\hat{c}$ with the maximum probability for
each point.

##### Instance Segmentation.

We can perform instance segmentations by first doing semantic segmentation,
and then for each output $\hat{c}$ we further execute count$(\hat{c})$.

###  3.2 Training Paradigm

##### Optimization Objective.

During training, we jointly optimize the NDF and the concept embeddings by
minimizing the loss as:

$\mathcal{L}=\alpha\cdot\mathcal{L}_{\text{NDF}}+\beta\cdot\mathcal{L}_{\text{reasoning}}$
(6)

Specifically, the NDF loss consists of two parts: the binary cross entropy
classification loss between the ground-truth occupancy and the predicted
occupancy, and the MSE loss between the ground-truth rgb value and the
predicted rgb value.

We also define three kinds of losses for three types of questions: 1) For
questions that ask about whether a part exist, we use an MSE loss
$\|a-\max({\mathbf{m}_{i}})\|^{2}$ where $a$ is the ground-truth answer and
$\max({\mathbf{m}_{i}})$ takes the maximum mask probability among all sampled
3D coordinates; 2) For questions that query about an attribute, we take the
cross entropy classification loss between the predicted concept category
$\hat{c}$ and the ground-truth concept category; 3) For counting questions, we
first use an MSE loss $\|a-n\|^{2}$ between the answer and the number output
by summing up the maxpool values of all the slots, and further add a loss to
ensure that the mask probabilities of the top $K$ values in the top $a$ slots
(i.e., the slots with the maximum maxpool values) should be close to 1, where
$K$ is a hyper-parameter. During training, we first train the NDF module only
for $N_{1}$ epochs and then jointly optimize the NDF module and the concept
embeddings. The Seq2seq model for semantic parsing is trained independently
with supervision.

##### Curriculum Learning.

Motivated by previous curriculum strategies for visual reasoning [19], we
employ a curriculum learning strategy for training. We split the questions
according to the length of operators they are parsed into. Therefore, we start
with questions with only one neural operator (e.g., “is there a yellow part of
the chair" can be parsed into Filter(Color)).

##  4 Experiments

###  4.1 Experimental Setup

####  4.1.1 Dataset

Instead of object-based visual reasoning tasks where objects are spatially
disentangled, which makes segmentations quite trivial, we seek to explore
part-based visual reasoning tasks where segmentations and reasoning are both
harder. To this end, we collect a new dataset, PartNet-Reasoning, which
focuses on visual question answering on the PartNet [22] dataset.
Specifically, we render approximately 3K RGB-D images from shapes of 4
categories: Chair, Table, Bag and Cart, with 8 questions on average for each
shape. We have three question types: exist_part queries about whether a
certain part exists by having the filter operator as the last operator;
query_part uses query operator to query about an attribute (e.g., part
category or color) of a filtered part; count_part utilizes the count operator
to count the number of instances of a filtered part. We are interested in
whether the reasoning tasks can benefit the segmentations of the fine-grained
parts, as well as whether the latent descriptor vectors by NDF can result in
better visual reasoning.

####  4.1.2 Evaluation Tasks

##### Reasoning.

We report the visual question answering accuracy on the PartNet-Reasoning
dataset w.r.t the three types of questions on all four categories.

##### Segmentation.

We further evaluate both the performances of semantic segmentation and
instance segmentation on our dataset. As stated in 3.1.5, semantic
segmentation can be performed by querying the part category with the maximum
mark probability of each coordinate, and filtering the coordinates according
to category labels, and instance segmentation can be achieved by using the
count operator on each part category. We report the mean per-label
Intersection Over Union (IOU).

####  4.1.3 Baselines

|  | PointNet+LSTM | MAC | NDF+LSTM | CVX+L | BAE+L | 3D-CG
---|---|---|---|---|---|---|---
Chair | exist_part | 52.3 | 65.2 | 55.7 | 71.9 | 72.4 | 85.4
query_part | 41.6 | 53.9 | 54.2 | 72.3 | 70.5 | 68.7
count_part | 63.4 | 78.1 | 71.6 | 48.8 | 68.0 | 92.2
Table | exist_part | 55.1 | 66.4 | 61.3 | 68.5 | 71.0 | 80.3
query_part | 43.6 | 51.2 | 54.4 | 69.7 | 73.6 | 75.1
count_part | 35.5 | 55.3 | 50.1 | 30.7 | 45.7 | 90.9
Bag | exist_part | 65.4 | 85.4 | 69.2 | 87.3 | 73.8 | 89.2
query_part | 53.1 | 74.8 | 64.0 | 88.1 | 70.6 | 85.2
count_part | 51.2 | 70.9 | 72.3 | 53.4 | 31.4 | 68.3
Cart | exist_part | 49.7 | 75.3 | 61.7 | 79.1 | 91.0 | 90.1
query_part | 50.1 | 64.0 | 57.0 | 72.3 | 81.5 | 86.3
count_part | 41.6 | 82.1 | 67.3 | 46.6 | 47.3 | 74.8
Table 1: Visual question answering accuracies. exist_part queries about
whether a certain part exists by having the filter operator as the last
operator; query_part uses query operator to query about an attribute of a
filtered part; count_part utilizes the count operator to count the number of
instances of a filtered part. Point+LSTM, MAC and NDF+LSTM are methods based
on neural networks. CVX+L and BAE+L are neural-symbolic methods which pre-
segment the masks and ground concepts on the masks. 3D-CG outperforms baseline
models by a large margin.

We compare our approach to a set of different baselines, with details provided
in the supp. material.

##### Unsupervised Segmentation Approaches.

We compare 3D-CG with two additional 3D unsupervised segmentation models, and
further consider how we may integrate such approaches with concept grounding.
First, we consider CVXNet [5], which decomposes a solid object into a
collection of convex polytope using neural implicit fields. In this baseline,
a number of primitives are selected and a convex part is generated in each
primitive. Next, we consider BAENet [2] which decomposes a shape into a set of
pre-assigned branches, each specifying a portion of a whole shape. In
practice, we tuned with the original codes of BAENet on our dataset and found
that nearly all coordinates would be assigned to a single branch. This is
probably because the original model is trained on voxel models, while ours is
on partial point cloud and contains more complex shapes, thus posing more
challenges. Therefore, for BAENet we use an additional loss which enforces
that at least some of the branches should have multiple point coordinates
assigned to positive occupancy value. The primitives in CVXNet and the
branches in BAENet are similar to slots in our paper.

##### Reasoning Baselines.

We compare our approach to a set of different reasoning baselines. First we
consider PointNet+LSTM and NDF+LSTM, which use the features by PointNet or our
NDF module, concatenated with the features of the questions encoded by a LSTM
model. The final answer distribution is predicted with an MLP. Next, we
compare with MAC, a commonly-used attention-based model for visual reasoning.
We add a depth channel to the input to the model. Finally we compare with
CVX+L and BAE+L are neural-symbolic models that use the similar language-
mediated segmentation framework from [34]. Specifically, they first initiate
the segmentations in the slots, and each slot has a slot feature vector. The
operators from the questions are executed symbolically on the slot features.
Question answering loss can be also propagated back to finetune the slot
segmentations and features.

##### Segmentation Baselines.

For the segmentation tasks, we compare our approach with unsupervised
segmentation approaches CVX and BAE described above. We further construct
language-mediated variants CVX-L and BAE-L. For semantic segmentation by CVX
and BAE, we use the manual grouping methods as in [5]. For semantic
segmentation by language-mediated models, we use the filter operator to filter
the slots belonging to the part categories.

###  4.2 Experimental Results

####  4.2.1 Reasoning & Concept Grounding

Figure 3: Qualitative Illustration. Examples of the reasoning process of our
proposed 3D-CG. Given an input image of a shape and a question, we first parse
the question into steps of operators. We visualize the set of points being
referred to by the operator via highlighting the regions where mask
probabilities > 0.5. As is shown,our 3D-CG can make reference to the right set
of coordinates, thus correctly answering the questions.

Table 1 shows the VQA accuracy among all models on our dataset. We can see
that overall, our 3D-CG outperforms baseline modeles by a large margin.
Language-mediated neural-symbolic methods (CVX+L & BAE+L) are better than
neural methods in the exist_part and query_part question types, but in general
they are far worse than our method. This is because the slot-based
representations pre-select the parts and make it easier for the concepts to
attend to specific regions. However, wrong pre-segmentations are also hard to
be corrected during the training of reasoning. The accuracies for the
count_part type further demonstrates this point, where language-mediated
baselines have extremely low accuracies, and segmentations from Figure 4(a)
also show that even with supervision from question answering loss, BAE and CVX
cannot segment the instances of a part category (e.g., legs and wheels).
Neural methods such as MAC and NDF+LSTM perform well in the count_part type in
some categories, probably due to some shortcuts of the PartNet dataset (e.g.,
most bags have two shoulder straps). In comparison to both neural and neural-
symbolic methods, our method performs well in all three question types. This
is because the regions attended by concepts are dynamically improved with the
question answering loss, while the advantage of explicit disentanglement of
concept learning and reasoning process is maintained.

(a) Visualization of segmentation results.

(b) Generalization.

Figure 4: Visualization of segmentation as well as generalization. CVX and BAE
are unsupervised segmentaton methods. CVX+L and BAE+L are language-mediated
methods which utilize question answering loss to finetune the segmentation
module. 3D-CG has better performances in both semantic and instance
segmentations, while other methods suffer from merging parts or segmenting a
part into multiple parts. It can also generalize well to unseen categories and
real scans.

Figure 3 shows some qualitative examples of the reasoning process of our
method. Specifically, the questions are parsed into a set of neural operators.
For each step, the neural operator takes the output of the last step as its
input. For the filter operator, we visualize the attended regions with mask
probabilities greater than 0.5. From the figure, we can see that our method
can attend to the correct region to answer the questions, as well as segment
the instances correctly for counting problems. This attention-based reasoning
pipeline is also closer than previous neural-symbolic methods to the way that
humans perform reasoning. For example, when asked the question “What is the
category of the green part of the chair?", humans would directly pay attention
to the green region regardless of the segmentations of the rest of the chair.
However, previous neural symbolic methods [37, 19, 34] segment the chair into
different parts first and then select the green part, which is very unnatural.

####  4.2.2 Segmentation

Table 2 shows the semantic segmentation and instance segmentation results, and
Figure 4(a) visualizes some qualitative examples. We can see that overall, our
3D-CG can better segment parts than unsupervised or language-mediated methods.
Other methods experience some common problems such as failing to segment the
instances within a part category (e.g., one of the legs is always merged with
the seat), or experience unclear boundaries and segment one part into multiple
parts (e.g., segmenting the chair back or the tabletop into two parts). In
general, language-mediated methods are better than pure unsupervised methods,
which indicates that the natural supervision of question answering does
benefit the segmentation modules. However, a lot of the wrong segmentations
cannot be corrected by reasoning, resulting in bad results for counting
questions for these models. In contrast, our 3D-CG can learn segmentations
well using question answering loss, mainly because the segmentations emerge
naturally from scratch without any restrictions.

####  4.2.3 Generalization

In Figure 4(b), we show some qualitative examples of how our 3D-CG trained on
seen categories can be generalized directly to unseen categories and real
scans. We show results of semantic and instance segmentations, as well as
visualize the parts that are referred to in the questions.

| Chair | Table | Bag | Cart | Mean
---|---|---|---|---|---
CVX | 62.3 | 74.2 | 66.0 | 44.2 | 61.7
CVX+L | 64.6 | 74.5 | 70.2 | 51.3 | 65.2
BAE | 49.5 | 71.0 | 64.1 | 49.7 | 58.6
BAE+L | 56.3 | 72.3 | 69.8 | 46.9 | 61.3
3D-CG | 76.6 | 79.3 | 67.3 | 54.2 | 69.4

(a) Semantic Segmentation IOU

| Chair | Table | Bag | Cart | Mean
---|---|---|---|---|---
CVX | 44.1 | 40.6 | 31.2 | 22.5 | 34.6
CVX+L | 42.6 | 51.2 | 41.3 | 20.4 | 38.9
BAE | 53.3 | 32.2 | 39.8 | 34.0 | 39.9
BAE+L | 50.1 | 47.0 | 34.1 | 36.6 | 42.0
3D-CG | 68.5 | 71.2 | 47.2 | 40.5 | 56.9

(b) Instance Segmentation IOU

Table 2: Segmentation Results. 3D-CG outperforms all unsupervised / language-
mediated baseline models.

For generalizing to new categories, we use the model trained on carts to
ground and count the instances of the concept “wheel" in cars. We can see that
the instance segmentation results on the wheels are not perfect because one
wheel is in wrong position. However, most of the wheels are detected and the
model manages to output the right count. We also use the model trained on
chairs to filter the legs of a bed. All the legs are successfully selected out
by our 3D-CG.

We also use real 3D scans from the RedWood dataset [4] to estimate 3D-CG’s
ability to generalize to real scenes. We use a single-view scan to reconstruct
partial point cloud and remove the background such as the floor. We input the
partial point cloud into our 3D-CG and output the segmentations on the ground-
truth voxels. For generalizing to bicycles, we use the 3D-CG model trained on
carts. We can see that 3D-CG can find all instances in the bicycle and detect
both wheels. Furthermore, 3D-CG trained on chairs can also be generalized to
chairs in real scans.

##  5 Discussion

##### Conclusion.

In this paper, we propose 3D-CG, which leverages the continuous and
differentiable nature of neural descriptor fields to segment and learn
concepts in the 3D space. We define a set of neural operators on top of the
neural field, with which not only can semantic and instance segmentations
emerge naturally, but visual reasoning can also be well performed.

##### Limitations and Future Work.

An limitation of our underlying framework with 3D-CG is that while we show
transfer results on real scenes, our approach is only trained with synthetic
scenes and questions. While we believe our proposed operations is general-
purpose, an interesting direction of future work would be scale our framework
to directly train on complex real-world scenes, and ground more concepts from
natural language on these real scenes is worth delving into in the future.

## Acknowledgments and Disclosure of Funding

This work was supported by MIT-IBM Watson AI Lab and its member company
Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR
(N00014-18-1-2847), and Mitsubishi Electric.

## References

  * [1] Z. Chen, J. Mao, J. Wu, K.-Y. K. Wong, J. B. Tenenbaum, and C. Gan.  Grounding physical concepts of objects and events through dynamic visual reasoning.  ICLR, 2021.
  * [2] Z. Chen, K. Yin, M. Fisher, S. Chaudhuri, and H. Zhang.  Bae-net: Branched autoencoder for shape co-segmentation.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8489–8498, 2019.
  * [3] Z. Chen and H. Zhang.  Learning implicit fields for generative shape modeling.  In Proc. CVPR, pages 5939–5948, 2019.
  * [4] S. Choi, Q.-Y. Zhou, S. D. Miller, and V. Koltun.  A large dataset of object scans.  ArXiv, abs/1602.02481, 2016.
  * [5] B. Deng, K. Genova, S. Yazdani, S. Bouaziz, G. E. Hinton, and A. Tagliasacchi.  Cvxnet: Learnable convex decomposition.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 31–41, 2020.
  * [6] Y. Du, M. K. Collins, B. J. Tenenbaum, and V. Sitzmann.  Learning signal-agnostic manifolds of neural fields.  In Advances in Neural Information Processing Systems, 2021.
  * [7] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu.  Neural radiance flow for 4d view synthesis and video processing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [8] C. Gan, Y. Li, H. Li, C. Sun, and B. Gong.  Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation.  In ICCV, pages 1811–1820, 2017.
  * [9] S. Ganju, O. Russakovsky, and A. K. Gupta.  What’s in a question: Using visual questions as a form of supervision.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6422–6431, 2017.
  * [10] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui.  Zero-shot detection via vision and language knowledge distillation.  ArXiv, abs/2104.13921, 2021.
  * [11] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9:1735–1780, 1997.
  * [12] R. Hu, A. Rohrbach, T. Darrell, and K. Saenko.  Language-conditioned graph networks for relational reasoning.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10293–10302, 2019.
  * [13] D. A. Hudson and C. D. Manning.  Compositional attention networks for machine reasoning.  ICLR, 2018.
  * [14] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion.  Mdetr - modulated detection for end-to-end multi-modal understanding.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1760–1770, 2021.
  * [15] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl.  Language-driven semantic segmentation, 2022.
  * [16] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf.  Object-centric learning with slot attention.  ArXiv, abs/2006.15055, 2020.
  * [17] A. Luo, Y. Du, M. J. Tarr, J. B. Tenenbaum, A. Torralba, and C. Gan.  Learning neural acoustic fields.  arXiv preprint arXiv:2204.00628, 2022.
  * [18] T. Luong, H. Pham, and C. D. Manning.  Effective approaches to attention-based neural machine translation.  In EMNLP, 2015.
  * [19] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu.  The neuro-symbolic concept learner: Interpreting scenes words and sentences from natural supervision.  ArXiv, abs/1904.12584, 2019.
  * [20] L. M. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4455–4465, 2019.
  * [21] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proc. ECCV, 2020.
  * [22] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su.  Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 909–918, 2019.
  * [23] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proc. ICCV, 2019.
  * [24] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5379–5389, 2019.
  * [25] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proc. CVPR, 2020.
  * [26] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proc. CVPR, 2019.
  * [27] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger.  Convolutional occupancy networks.  In Proc. ECCV, 2020.
  * [28] D. Rebain, K. Li, V. Sitzmann, S. Yazdani, K. M. Yi, and A. Tagliasacchi.  Deep medial fields.  arXiv preprint arXiv:2106.03804, 2021.
  * [29] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proc. ICCV, pages 2304–2314, 2019.
  * [30] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann.  Neural descriptor fields: Se (3)-equivariant object representations for manipulation.  arXiv preprint arXiv:2112.05124, 2021.
  * [31] V. Sitzmann, M. Zollhöfer, and G. Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In Proc. NeurIPS 2019, 2019.
  * [32] E. S. Spelke, K. Breinlinger, K. Jacobson, and A. Phillips.  Gestalt Relations and Object Perception: A Developmental Study.  Perception, 22(12):1483–1501, 1993.
  * [33] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun.  Order-embeddings of images and language.  CoRR, abs/1511.06361, 2016.
  * [34] R. Wang, J. Mao, S. J. Gershman, and J. Wu.  Language-mediated, object-centric representation learning.  In FINDINGS, 2021.
  * [35] J. Xu, S. D. Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang.  Groupvit: Semantic segmentation emerges from text supervision.  ArXiv, abs/2202.11094, 2022.
  * [36] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Proc. NeurIPS, 2020.
  * [37] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.  In NeurIPS, 2018.

## Overview

In this appendix, we supplement our paper by providing more qualitative
examples and details about our model to help readers better understand our
paper.

This appendix is organized as follows.

  * •

In Sec. A, we provide more qualitative examples about how 3D-CG performs
reasoning steps.

  * •

In Sec. B, we provide more details about our 3D-CG model, including the
detailed architecture of NDF, the semantic parsing module and the curriculum
learning strategies.

  * •

In Sec. C, we provide more details about the baselines models.

  * •

In Sec. D, we discuss the potential societal impacts of this paper.

##  Appendix A More Qualitative Examples

###  A.1 QA Examples

In Figure 5, we show more examples of the reasoning process of 3D-CG. We can
see that overall, our model can make reference to the correct region to answer
the questions, although sometimes the segmentation might be a little noisy
(e.g., for the bag a part of the shoulder strap is not highlighted, and the
highlighted part of cart contains part of the wheel.). Also, for counting
problems, we can see that our model can segment the legs and make the right
prediction. However, for the chair with five legs, although our model can
assign five colors to five legs, some of the legs are overlapped with each
other by colors.

Figure 5: More qualitative examples that illustrate the reasoning process of
3D-CG. Given an input image of a shape and a question, we first parse the
question into steps of operators. We visualize the set of points being
referred to by the operator via highlighting the regions where mask
probabilities > 0.5. As is shown,our 3D-CG can make reference to the right set
of coordinates, thus correctly answering the questions.

##  Appendix B Details on 3D-CG

###  B.1 NDF

We first train the reconstruction task solely for 50,000 iterations.

For encoder, we use a PointNet-based encoder network with ResNet blocks. We
first map the 6-dim (x,y,z,r,g,b) point cloud to 128-dim features, followed by
several pooling and leaky relu layers. We then go through another fully-
connected layer to get 256-dim features. We use 5 resnet fully connected
blocks, each with input dim 256 and output dim 128. At each step the output is
concatenated with the input to produce the input of the next step. A final
fully-connected layer maps the feature back to 128.

For the decoder, we first concatenate the coordinates and the point cloud
latent feature vector. The new input is passed through 5-layer resnet blocks.
All the resnet blocks have a hidden size of 128. The features of the blocks
are concatenated together and passed through a fully-connected layer to
produce the occupancy value. For color reconstruction, the features further go
through 3 linear layers with hidden size 128 and produce r,g,b value. The
batch size is 4. The learning rate is 1e-4. All embeddings have the dim 2049.

###  B.2 Semantic Parsing

Our semantic parser is an attention-based sequence to sequence (seq2seq) model
with an encoder-decoder structure similar to that in [18]. The encoder is a
bidirectional LSTM [11]. The decoder is a similar LSTM that generates a vector
from the previous token of the output sequence. Both the encoder and decoder
have two hidden layers with a 256-dim hidden vector. We set the dimensions of
both the encoder and decoder word vectors to be 300.

###  B.3 Curriculum learning

Figure 6: The curriculum learning strategy of 3D-CG.

In Figure 6, we show detailed strategies of curriculum training for 3D-CG.
During the whole training process, we gradually add more visual concepts and
more complex question examples into the model, which is shown in Figure 6. In
general, the whole training process is split into 3 stages. First, we learn
color concepts. We want to learn individual concepts like red or seat.
However, since most chairs contain certain parts like back and seat, we
therefore learn the color concepts first by asking questions like “Is there a
red part in the chair?" For the second lesson, after learning about color
concepts like in the figure, we can then attend to the red region, and we can
ground different part categories by asking questions like “What is the
category of the red part?". Finally, we can train more complex questions
altogether. Each lesson is trained for approximately 50,000 iterations.

##  Appendix C Datails on Baseline Models

###  C.1 Details on Language-mediated models

We explain in detail how CVX-L and BAE-L work for segmentation and reasoning,
as is shown for Figure 7. As the original CVX/BAE model, we use the same NDF
as in 3D-CG, except that instead of decoding the descriptor vectors into N*1
occupancy values, we first decode them into N*S occupancy values for different
slots, and then maxpool to get the overall occupancy values for
reconstruction. For reasoning, the descriptor vectors of each slot is mean
pooled to get a part-centric representation. The question also goes through a
semantic parsing module like 3D-CG, and attention is calculated between the
concept embedding vector and the descriptor vector of each slot to identify
whether the part in one slot can be accounted for a concept. The reasoning
loss can also be back-propagated to the NDF module and the concept embeddings.
We use the same curriculum learning strategy as 3D-CG and train for the same
number of epochs.

Figure 7: Detailed framework of CVX-L / BAE-L, where N denotes the number of
sampled points, and S denotes the number of slots.

###  C.2 Neural-network based baselines

For MAC, we use an ImageNet-pretrained ResNet-101 to extract 14 × 14 × 1024
feature maps for MAC. For PointNet+LSTM, we use a 2048 dimensional feature
from the last layer. For NDF+LSTM, we mean-pool the descriptor vectors of all
point coordinates to get 2049-dimensional vectors. All models are trained for
50 epochs.

##  Appendix D Societal Impacts

Our work is of broad interest to the computer vision and machine learning
communities. Our proposed no inherent ethical or societal issues on its own,
but inherit those typical of learning methods, such as capturing the implicit
biases of data. Our work aims to enable more interpretable reasoning, and may
serve as an inspiration for works that aims to perform less black-box
reasoning with gradient-based learning.