# Mass estimation of the very massive galaxy cluster SRGe CL2305.2-2248 from
strong lensing

I.M. Khamitov<EMAIL_ADDRESS>1,2  I.F. Bikmaev  2,3,11  N.S.
Lyskova  4,8  A.A. Kruglov  4  R.A. Burenin  4,11  M.R. Gilfanov  4,5  A.A.
Grokhovskaya  6,7  S.N. Dodonov  6,7,11  S.Yu. Sazonov  4  A.A. Starobinsky  9
R.A. Sunyaev  4,5  I.I. Khabibullin  10,4,5  E.M. Churazov 1TÜBİTAK National
Observatory, Antalya, Turkey 2Kazan Federal University, Kazan, Russia 3Academy
of Sciences of The Republic of Tatarstan, Kazan, Russia 4Space Research
Institute RAS (IKI), Moscow, Russia 5Max Planck Institute for Astrophysics,
Garching, Germany 6Special Astrophysical Observatory of the Russian Academy of
Sciences, Nizhnij Arkhyz, Russia 7Institute of Applied Astronomy RAS, St.
Petersburg, Russia 8P.N.Lebedev Physical Institute RAS, Astro Space Center,
Moscow, Russia 9Landau Institute for Theoretical Physics RAS, Chernogolovka,
Russia 10Observatory of the Ludwig Maximilian University of Munich, Munich,
Germany 11Sternberg Astronomical Institute, Moscow State University, Moscow,
Russia  4,5

###### Abstract

The galaxy cluster SRGe CL2305.2$-$2248 (SPT-CL J2305 $-$ 2248, ACT-CL J2305.1
$-$ 2248) is one of the most massive clusters at high redshifts ($z\simeq
0.76$) and is of great interest for cosmology. For an optical identification
of this cluster, deep images were obtained with the 1.5-m Russian-Turkish
telescope RTT-150. Together with the open archival data of the Hubble Space
Telescope, it became possible to identify candidates for gravitationally
lensed images of distant blue galaxies in the form of arcs and arclets. The
observed giant arc near the brightest cluster galaxies allowed us to estimate
the radius of the Einstein ring, which is $9.8\pm 1.3$ arcseconds. The
photometric redshift of the lensed source was obtained ($z_{s}=2.44\pm 0.07$).
Its use in combination with the Einstein radius estimate made it possible to
independently estimate the SRGe CL2305.2$-$2248 mass. It was done by
extrapolating the strong lensing results to large radii and using the model
density distribution profiles in relaxed clusters. This extrapolation leads to
mass estimates $\sim 1.5-3$ times smaller than those obtained from X-ray and
microwave observations. A probable cause for this discrepancy may be the
process of cluster merging, which is also confirmed by SRGe CL2305.2$-$2248
morphology in the optical range.

###### keywords:

galaxy clusters, galaxy cluster SRGe CL2305.2-2248, strong lensing

December 1, 2021

##  1 Introduction

The very massive cluster of galaxies SRGe CL2305.2$-$2248 was detected in
X-rays from the results of the first all-sky survey (completed in June 2020)
with the eROSITA telescope on board the SRG space observatory (Predehl et al.,
2021; Sunyaev et al., 2021). Based on the results of optical observations with
the Russian-Turkish 1.5-m telescope RTT-150 and the 6-m BTA telescope, an
optical identification of the cluster was carried out, and a spectroscopic
measurement of the redshift of the cluster was obtained, $z=0.7573$ (Burenin
et al., 2021). The presence of strong gravitational lensing in the cluster
field is noted in the paper by Bleem et al. (2020) (see their Table 8) on the
early detection of this cluster (SPT-CL J2305-2248) in the millimeter band of
the South Pole Telescope survey. In this work, we carried out a photometric
estimate of the redshift of the lensed source and obtained an independent
estimate of the mass of the cluster within the Einstein ring. Assuming the
Navarro-Frenk-White density distribution (Navarro et al., 1996), we derived
the cluster mass estimate at $R_{500}$. In our estimates we assume the
standard cosmological model $\Lambda$CDM with the following parameters:
$\Omega_{m}=0.3$, $\Omega_{\Lambda}$ = 0.7, $H_{0}=70$ km/s/Mpc.

##  2 Photometric observations

Deep direct images of the SRGe CL2305.2$-$2248 cluster field were obtained on
the Russian-Turkish 1.5-m telescope RTT-150 as a part of the ground-based
support of the X-ray survey with the eROSITA telescope onboard the Spectrum-
Roentgen-Gamma (SRG) space observatory. The observations were performed in
August 23-26, 2020, in the SDSS _g,r,i,z_ filters. An Andor iKon-L 936
BEX2-DD-9ZQ CCD camera 2Kx2K (2048x2048) pixels with a resolution element of
0.326 was used as a detector. The quantum efficiency of the CCD is about 90%
and higher in the wavelength range from 4000Å to 8500Å. Table 1 shows a log of
observations made. The total moderate quality exposure times were 13200 s,
13800 s, 8400 s, and 8400 s in the SDSS filters _g,r,i,z_ , respectively. The
total exposure in each filter was divided into 600 s exposures, between which
the telescope pointing axis was shifted by 10-20 in an arbitrary direction.
Processing of the direct images was performed in a standard way, using the
_IRAF_ software, as well as our own software, using a standard set of
calibrations.

The photometric calibration of the images was obtained using observations of
the photometric standards (Smith et al., 2002).

Table 1: Observation log of the galaxy cluster SRGe CL2305.2$-$2248 performed at RTT-150. F - SDSS filter, N - number of images, T - total exposure in seconds, $\sigma$ \- the seeing in arcseconds. Date | F | N | T | $\sigma$
---|---|---|---|---
| (SDSS) |  | (s) |  ($\arcsec$)
2020-08-23 | g | 4 | 2400 | 2.4
2020-08-23 | r | 4 | 2400 | 2.0
2020-08-23 | i | 4 | 2400 | 1.7
2020-08-23 | z | 4 | 2400 | 1.7
2020-08-24 | g | 4 | 2400 | 1.6
2020-08-24 | r | 4 | 2400 | 1.7
2020-08-24 | i | 4 | 2400 | 1.6
2020-08-24 | z | 5 | 3000 | 1.6
2020-08-25 | g | 4 | 2400 | 2.0
2020-08-25 | r | 5 | 3000 | 2.2
2020-08-25 | i | 6 | 3600 | 2.1
2020-08-25 | z | 5 | 3000 | 2.0
2020-08-26 | g | 10 | 6000 | 2.0
2020-08-26 | r | 10 | 6000 | 2.2

##  3 Selection of lensed images

For the task of an optical identification of the cluster under consideration,
deep images with a limiting magnitude as faint as 23-24m in the 4 bands
_g,r,i,z_ of the SDSS photometric system were obtained at the RTT-150
telescope. At redshift $z=0.76$ the main radiation flux from the cluster falls
into the _i_ and _z_ bands. The images in the _g,r_ bands were used to cut off
the background galaxies at smaller redshifts. The number of background
galaxies separated by colors _(g-r)_ and _(r-i)_ in the RTT-150 images and
that did not belong to the cluster in the field of 9 x 9 arcminutes was about
1000. An analysis of the images showed that, in addition to the large number
of background galaxies detected in all _g,r,i_ bands in regions close in
position to two brightest cluster galaxies (BCG), about 10 blue sources,
visible only in the _g_ filter, are detected. Given that Bleem et al. (2020)
noted the presence of strong gravitational lensing in the cluster field, we
suggested that the blue sources in the RTT-150 frames near the BCG could be
associated with the gravitational lensing features (arcs). To select the
lensed images of a distant source, we used deep RTT-150 images of the cluster
field obtained in _g,r,i,z_ bands and high spatial resolution images from the
Hubble Space Telescope (HST) open data archive (Whitmore et al., 2016)
obtained on April 23, 2018 in F110W filter with the WFC3/IR detector, an
exposure time of 758.81 s (Fig. 1), and in the F200LP filter with the
WFC3/UVIS detector, an exposure time of 741 s.

Figure 1: HST-image of the cluster field. The giant arc is clearly visible.
The positions of the centers of the two brightest cluster galaxies are marked
with crosses. Figure 2: The color image of the cluster field. The pseudo-
colors are based on RTT-150 data (_g_ \- blue, _r_ \- green, _i_ \- red). The
selected blue sources in the form of arcs and arclets in the HST image are
marked.

The lensed candidates were selected by the morphology of the sources in the
HST image and by brightness excess in the _g_ band compared to the _r,i,z_
bands. Fig. 2 shows the selected candidates in the form of arcs and arclets.
The brightness of the sources is close to the detection limit in the _g_
-image and is $\sim 24^{m}$. The observed giant arc near the brightest
galaxies of the cluster is detected in the _g_ -image in the form of three
separate sources. The arc lies nearly on one circle together with two sources
to the southeast (concentrically aligned with the arc in the HST image), what
indicates a high degree of circular symmetry of the lens and its excellent
position on the line of sight between the observer and the lensed galaxy.
Thus, to estimate the lens mass we can consider a concentrically symmetric 2D
mass distribution of the lens. Fig. 3 shows this region in the _g,r,i,z_
bands. The blue and red circles show the fits of Einstein ring position
obtained, respectively, by analyzing the HST image of the arc and the
southeastern arclets and the identified blue sources based on RTT-150 data.
The radius of the circle determines the angular distance $\Theta_{arc}$
corresponding to the Einstein radius. The centers of the circles are also
marked: blue - according to HST data, red - according to RTT150 data. It can
be seen that the radii of the circles and their centers are in good agreement
with each other. Thus, we demonstrated the possibility of using RTT-150 in
estimates of this kind. We took the distance between the center of the circle
and _BCG_ , $1\arcsec.3$, as the error of the $\Theta_{arc}$ determination in
this system.

Figure 3: Deep $30\arcsec\times 30\arcsec$ _g,r,i,z_ -images of the field
around _BCG_. The positions of the two brightest galaxies of the cluster are
marked with red crosses.

##  4 The redshift estimation of the lensed source

The total exposure times for the deep fields in the _g_ and _r_ bands are the
same. Consequently, taking into account the same quantum efficiency of the
detector and filter transmission in these bands, the depth of the fields is
also approximately the same. The absence of signal in the _r,i,z_ bands
allowed us to constrain the redshift of the lensed source. With a great
probability, it is a galaxy with intense star formation. Therefore, the signal
registered in the _g_ band corresponds to the location of the emission line
$L_{\alpha}$ (1215Å) in this region of the spectrum. Moreover, the strong CIV
line (1549Å) must also fall into this region. Otherwise, the signal from it
would be registered in the _r_ band. Also, the CIV cannot be the only line
falling into the _g_ band. In this case, the strong MgII emission line (2799Å)
would be registered in the _i_ band. Based on these considerations and
analyzing the transmission function of the _g_ filter used on RTT-150, we
determined the lower and upper limits on the redshift of the lensed galaxy. In
Fig. 4, the red dashed line shows the wavelength region of the $L_{\alpha}$
distant source line position, at which the $L_{\alpha}$ and CIV lines
simultaneously fall within the region of this band. When estimating the
wavelength interval, the filter transmittance region above 70% was considered.

Figure 4: Transmission curve of the _g_ filter used at RTT-150. The wavelength
region of the line position $L_{\alpha}$ of the distant source, at which the
lines $L_{\alpha}$ and CIV simultaneously fall in the region of this band, is
marked.

Thus, the redshift of the lensed source was determined: $z_{s}=2.44\pm 0.07$,
which was used in the mass estimate.

##  5 Mass estimation from strong gravitational lensing

In this work, we use the same approach to mass estimation as in the work by
Dahle et al. (2016), performed for the PSZ1 G311.65-18.48 cluster, which was
discovered by the Planck observatory. In the field of this cluster a
gravitationally lensed image of a distant galaxy in the form of a giant arc
was detected. The parameters of the lensed system (Einstein’s radius in
angular $\Theta_{arc}$ and physical units $R_{Ein}$, redshifts to the source
and the lens and the corresponding angular diameter distances111The angular
diameter distance is defined as the ratio of the transverse size of an object
in physical units to its angular size in radians.) for calculating the SRGe
CL2305.2$-$2248 mass are given in Table 2.

The mass of a cluster inside a cylinder of radius $R_{Ein}=72.2$ kpc is
estimated as:

$M^{cyl}_{Ein}=\pi R_{Ein}^{2}\Sigma_{crit}=3.3^{+0.95}_{-0.8}\cdot
10^{13}M_{\odot},$ (1)

where $\Sigma_{crit}$ is the critical surface density (equation (4) in the
Appendix).

The mass of the cluster estimated in Burenin et al. (2021) is $M_{500}\sim
9.03\cdot 10^{14}M_{\odot}$, which means that the radius, within which the
mean density is equal to 500 times the critical density at $z=0.7573$, is
equal to $R^{X}_{500}\sim 1110\,$ kpc. That corresponds to the angular size of
$\sim 2.5^{\prime}$. Thus, to recalculate the strong lensing mass within
$R_{Ein}$ to the value of $M(<R^{X}_{500})$, we need an extrapolation in
radius by the factor of $R^{X}_{500}/R_{Ein}\sim 15$, what limits the accuracy
of such calculations.

A rough estimate of the cluster mass enclosed within a sphere of radius
$R^{X}_{500}$ can be obtained by assuming that the lensing cluster is
described by a singular isothermal sphere (SIS) model with the density
distribution $\rho(r)\propto r^{-\gamma}$, where $\gamma=2$. Similar
calculations were performed for elliptical galaxies by Lyskova et al. (2018).
For a given $\gamma$, the mass (1) within a cylinder of radius $R_{Ein}$ can
be converted to the mass within a sphere of the same radius. For $\gamma=2$,
this mass is $2\cdot 10^{13}M_{\odot}$. In that case, the mass of the cluster
inside the sphere of radius $1110$ kpc is $M=(3.3\pm 0.4)\cdot
10^{14}M_{\odot}$, which is $\sim 3$ times less than the mass derived from the
X-ray luminosity-mass relation in Burenin et al. (2021).

Table 2:  Parameters of the gravitationally lensed system: Einstein radius in angular ($\Theta_{arc}$) and physical ($R_{Ein}$) units, redshifts of the source $z_{s}$ and the lens $z_{d}$, and the corresponding angular diameter distances $D_{s}$ and $D_{d}$ $\Theta_{arc}$ | $9\arcsec.8\pm 1\arcsec.3$
---|---
$R_{Ein}$ |  $72.2\pm 9.6$ kpc
$z_{s}$ | $2.44\pm 0.07$
$z_{d}$ | $0.7573\pm 0.0006$
$D_{s}$ |  $1674_{+9.6}^{-9.9}$ Mpc
$D_{d}$ |  $1519\pm 0.44$ Mpc

There is no doubt that the assumption about the slope $\gamma=2$ over the
entire range of radii from $\sim 0.07$ $R^{X}_{500}$ to $R^{X}_{500}$ is a
rough approximation of the actually observed density profiles. It can be used
only for an order of magnitude estimation. The Navarro-Frank-White profile
(Navarro et al., 1996) or the Einasto profile (Einasto, 1965), which describe
well the numerical simulations of cluster formation when the contribution of
ordinary matter (baryons) can be neglected, are a better approximation. In the
case of a slightly simpler Navarro-Frank-White profile (compared to the
Einasto model), the slope of the density profile changes from $\gamma=1$ at
small radii to $\gamma=3$ at large radii.

Let us estimate the mass of the cluster assuming that the density obeys the
Navarro-Frenk-White model:

$\rho(r)=\frac{\delta_{c}\rho_{crit}}{\left(\frac{r}{R_{s}}\right)\left(1+\frac{r}{R_{s}}\right)^{2}},$
(2)

where $\rho_{crit}=3H^{2}(z)/(8\pi G)$ is the critical density,
$H(z)=H_{0}(\Omega_{m}(1+z)^{3}+\Omega_{\Lambda})^{1/2}$ is the Hubble
constant at the redshift of the cluster, $G$ is the gravitational constant.
The characteristic cluster radius is $R_{s}=R_{200}/c$, where $c$ is the
dimensionless concentration parameter, which is included in the normalization
of the density profile as

$\delta_{c}=\frac{200}{3}\frac{c^{3}}{ln(1+c)-c/(1+c)}.$ (3)

The size of a cluster $R_{200}$ is defined as the radius of a sphere within
which the mean halo density is $200\rho_{crit}$. Therefore, the halo mass is
$M_{200}\equiv M(R_{200})=\frac{4}{3}\pi R^{3}_{200}\times 200\rho_{crit}$.

Studies of the properties of dark matter halos based on numerical simulations
show (i.e.,, Duffy et al., 2008) that the parameters $c$ and $M_{200}$ are
closely related to each other in a wide range of halo masses. For further
estimates, we use the halo concentration-mass relation (taking into account
the redshift of the cluster) from Duffy et al. (2008). Thus, we are left with
one free parameter in the Navarro-Frenk-White model - the halo mass. For a
spherically symmetric lens, the position of the tangential arc should be close
to the tangential critical curve (the properties of the lens with the Navarro-
Frenk-White density profile are described in Bartelmann, 1996, also see
Appendix). By equating $\Theta_{arc}$ to the size of the critical curve in the
lens plane, we obtain an independent estimate for the halo mass. Under the
assumptions above, the best agreement with the observed position of the arc is
achieved at $M_{200}=7.1\cdot 10^{14}M_{\odot}$ ($R_{200}=1388$ kpc). In this
case, the concentration parameter is equal to $c=3.8$ and
$R_{s}=R_{200}/c=365$ kpc. Thus, for comparison with the mass estimates from
X-rays and the Sunyaev-Zeldovich effect, we can calculate the mass of the dark
matter halo $M_{500}=(4.9\pm 0.7)\cdot 10^{14}M_{\odot}$ within the radius
inside which the mean halo density is equal to $500\rho_{crit}(z=0.7573)$, as
well as the mass within $R^{X}_{500}$ = 1110 kpc, which is
$M(<R^{X}_{500})=(5.9\pm 0.6)\cdot 10^{14}M_{\odot}$. Here, the uncertainties
in mass estimates are based on the uncertainty in the position of the halo
center, i.e. the center of the circle (see Fig. 3). The radius of tangential
critical curve for the Navarro-Frenk-White profile is plotted against the dark
halo mass $M(<R^{X}_{500})$ in Fig. 5. The mass estimate that matches best the
position of the tangential arc is marked with a grey cross. For comparison, we
also show the cluster mass within $R^{X}_{500}=1110$ kpc obtained in Burenin
et al. (2021) from X-ray observations, performed within the framework of the
SRG/eROSITA survey. The mass estimates $M_{500}$ of the SRGe CL2305.2-2248
cluster available to date are presented in Table 3. Note that the physical
size of the sphere inside which the mass is calculated can vary for different
methods. As seen from Table 3 and Fig. 2, the mass of the cluster estimated
from gravitational lensing is $\sim 1.5-2$ times smaller than the mass from
the literature.

Note that the size of the critical curve substantially depends on the
derivative of the gravitational potential in the central part of the cluster.
Consequently, the strong lensing mass estimate may turn out to be sensitive to
the contribution of central galaxies and, possibly, hot X-ray emitting gas, to
the total density profile of a cluster. Numerical cosmological simulations
show that taking baryons into account can indeed significantly change the
density profile in the central region of the cluster (for example, Henson et
al., 2017; Shirasaki et al., 2018), mainly due to cooling and ”adiabatic
compression” (for example, Gnedin et al., 2004). These effects can be
partially taken into account by keeping the NFW radial profile (2), but
changing the value of the concentration parameter $c$ compared to expectations
from cosmological dark matter only simulations. Above we obtained the cluster
mass estimate under the assumption that the concentration parameter is
expressed via the halo mass according to the relation presented in Duffy et
al. (2008). However, observations of strong and weak lensing of several well-
studied clusters from the CLASH sample (Merten et al., 2015) give noticeably
larger values of $c$ than predicted in Duffy et al. (2008), where cosmological
simulations involving only dark matter were used. If we fix the concentration
parameter at the value that is two times higher than the predictions from
Duffy et al. (2008), then the estimate of the halo mass is $M_{200}=2.6\cdot
10^{14}M_{\odot}$ , $c=8.3$, $M_{500}=2.1\cdot 10^{14}M_{\odot}$,
$M(<R^{X}_{500})=2.8\cdot 10^{14}M_{\odot}$. Thus, there is a noticeable
uncertainty when extrapolating from $R_{Ein}=72$ kpc to large radii $\sim
1000$ kpc due to the fact that real profiles of the total density of galaxy
clusters can be much more complicated than predicted by numerical simulations
of cluster formation without baryons, and due to possible deviations from
spherical symmetry.

Table 3: SRGe CL2305.2-2248 cluster mass estimates from the X-ray and microwave observations and strong lensing |  $M_{500}$, 10${}^{14}M_{\odot}$ | Source
---|---|---
eROSITA | $9.0\pm 2.6$ | Burenin et al. (2021)
ACT | $9.2\pm 1.5$ | Hilton et al. (2021)
SPT | $7.4\pm 0.8$ | Bleem et al. (2020)
strong | $4.9\pm 0.7$ | this work
lensing |  |
Figure 5: Dependence of the size of the tangential critical curve for the
spherically symmetric Navarro-Frenk-White profile on the cluster mass
estimated within the sphere of radius $R_{500}^{X}=1110$ kpc. The
concentration parameter $c$ is related to the halo mass by the relation from
Duffy et al. (2008) derived from the analysis of cosmological simulations
without taking baryons into account. The gray cross denotes the mass of the
cluster at which the Einstein radius coincides with the radius of the circle
$\Theta_{arc}$. When estimating the uncertainty for the mass, only the error
in the determination of $\Theta_{arc}$ was taken into account. The vertical
dashed line and shaded area show the mass estimate for the SRGe cluster
CL2305.2-2248 from SRG/eROSITA data.

##  6 Discussion

The massive galaxy cluster SRGe CL2305.2-2248 at redshift $z\simeq 0.76$ is a
very rare object in the observable universe. Its mass, obtained from X-ray
observations and from the Sunyaev-Zeldovich effect, $M_{500}\sim 9\cdot
10^{14}M_{\odot}$ is comparable to the mass of the El Gordo cluster. Within
the framework of the standard cosmological model $\Lambda$CDM, only $\sim 10$
of such massive clusters at redshifts $z>0.7$ are expected in the entire sky.
In this work, we investigated deep direct images of the field of the cluster
SRGe CL2305.2$-$2248 obtained with the Russian-Turkish 1.5-m telescope
(RTT-150) as a part of the tasks of ground support for the X-ray survey of the
eROSITA telescope onboard the Spectrum-Roentgen-Gamma (SRG) space observatory.
From the RTT-150 images and high spatial resolution images from the Hubble
Space Telescope (HST) Open Data Archive, probable gravitationally lensed
sources were identified, including the giant arc near the cluster brightest
galaxies. The photometric redshift of the arc is estimated to be
$z_{S}=2.44\pm 0.07$. The arc and several other lensed images lie virtually on
one circle with a radius of $9.8\pm 1.3$ arc seconds, which made it possible
to make an assumption about the spherical symmetry of the lens and obtain a
direct measurement of the mass inside a cylinder with a radius equal to the
radius of the circle (= Einstein’s radius $R_{Ein}$). Extrapolation of mass
measurements from strong lensing, that uses the singular isothermal sphere or
the Navarro-Frenk-White profile approximation for a reasonable range of values
of the concentration parameter $c$, leads to estimates of $M_{500}$
approximately $1.5-3$ times lower than the estimate from the relation
$M-L_{X}$. Values of $c$ that are well below the typical ones for relaxed
clusters are required to reconcile these estimates. A possible explanation is
a merger of clusters that led to a flatter mass distribution in the central
part of the cluster. The presence of two massive galaxies in the center may
also argue for this hypothesis. Another explanation could be a peculiarly high
X-ray luminosity of this cluster. In principle, the luminosity can increase
noticeably during certain merging phases. Consequently, the merger process can
simultaneously lead to an overestimation of the luminosity and an
underestimation of the concentration parameter $c$. Detailed X-ray and
microwave observations are needed to reliably determine the dynamic state of
the cluster. Constructing an accurate lens model using strong and weak lensing
methods will also allow one to draw conclusions about whether we observe one
massive cluster or several mergers, and to estimate the halo mass(es)
regardless of the dynamic state. However, this requires measuring/evaluating
redshifts of a large number of sources and measuring the degree of elongation
of background objects (for weak lensing).

###### Acknowledgements.

This work is based on observations with the eROSITA telescope on board the SRG
observatory. The SRG observatory was built by Roskosmos in the interests of
the Russian Academy of Sciences represented by its Space Research Institute
(IKI) within the framework of the Russian Federal Space Program, with the
participation of the Deutsches Zentrum f’́ur Luft- und Raumfahrt (_DLR_). The
SRG/eROSITA X-Ray telescope was built by a consortium of German Institutes led
by the Institute of Extraterrestrial Physics (_MPE_) with the support from
_DLR_. The SRG spacecraft was designed, built, launched and is operated by the
Lavochkin Association and his subcontractors. The science data are downlinked
via the Deep Space Network Antennae in Medvezhye Lakes, Ussuriysk, and
Baikonur, funded by Roskosmos. The eROSITA data used in this work were
processed using the _eSASS_ software system developed by the German eROSITA
consortium and proprietary data reduction and analysis software developed by
the Russian eROSITA Consortium. The authors thank TUBITAK, IKI, KFU and AN RT
for support of observations on the Russian-Turkish 1.5-m telescope in Antalya
(RTT-150). The study is based on observations made with the NASA/ESA Hubble
Space Telescope and obtained from the Hubble Legacy Archive, which is a joint
project of the Space Telescope Science Institute (STScI/NASA), the Space
Telescope European Coordinating Facility (ST-ECF/ESAC/ESA) and the Canadian
Astronomy Data Center (CADC/NRC/CSA). The work of IFB, RAB, SND was supported
by the Russian Science Foundation, grant 21-12-00210. AAS was partially
supported by the project 0033-2019-0005 of the Ministry of Education and
Science of the Russian Federation.

## References

  * Bartelmann (1996) M. Bartelmann,  A&A, 313, 697 (1996)
  * Bleem et al. (2020) L. E. Bleem, S. Bocquet, B. Stalder, M. D. Gladders, P. A. R. Ade, S. W. Allen, et al.,  ApJS, 247, 25 (2020)
  * Burenin et al. (2021) R. A. Burenin, I. F. Bikmaev, M. R. Gilfanov, A. A. Grokhovskaya, S. N. Dodonov, M. V. Eselevich, et al.,  Astronomy Letters, 47, 467 (2021)
  * Dahle et al. (2016) H. Dahle, N. Aghanim, L. Guennou, P. Hudelot, R. Kneissl, E. Pointecouteau, et al.,  A&A, 590, L4, 1 (2016)
  * Duffy et al. (2008) A. R. Duffy, J. Schaye, S. T. Kay, and C. Dalla Vecchia,  MNRAS, 390, L64 (2008)
  * Einasto (1965) J. Einasto,  Trudy Astrofizicheskogo Instituta Alma-Ata, 5, 87 (1965)
  * Gnedin et al. (2004) O. Y. Gnedin, A. V. Kravtsov, A. A. Klypin, and D. Nagai,  ApJ, 616, 16 (2004)
  * Henson et al. (2017) M. A. Henson, D. J. Barnes, S. T. Kay, I. G. McCarthy, and J. Schaye,  MNRAS, 465, 3361 (2017)
  * Hilton et al. (2021) M. Hilton, C. Sifón, S. Naess, M. Madhavacheril, M. Oguri, E. Rozo, et al.,  ApJS, 253, 3 (2021)
  * Lyskova et al. (2018) N. Lyskova, E. Churazov, and T. Naab,  MNRAS, 475, 2403 (2018)
  * Merten et al. (2015) J. Merten, M. Meneghetti, M. Postman, K. Umetsu, A. Zitrin, E. Medezinski, et al.,  ApJ, 806, 4 (2015)
  * Navarro et al. (1996) J. F. Navarro, C. S. Frenk, and S. D. M. White,  ApJ, 462, 563 (1996)
  * Predehl et al. (2021) P. Predehl, R. Andritschke, V. Arefiev, V. Babyshkin, O. Batanov, W. Becker, et al.,  A&A, 647, A1, 16 (2021)
  * Shirasaki et al. (2018) M. Shirasaki, E. T. Lau, and D. Nagai,  MNRAS, 477, 2804 (2018)
  * Smith et al. (2002) J. A. Smith, D. L. Tucker, S. Kent, M. W. Richmond, M. Fukugita, T. Ichikawa, et al.,  AJ, 123, 2121 (2002)
  * Sunyaev et al. (2021) R. Sunyaev, V. Arefiev, V. Babyshkin, A. Bogomolov, K. Borisov, M. Buntov, et al.,  A&A, 656, A132, 29 (2021)
  * Whitmore et al. (2016) B. C. Whitmore, S. S. Allam, T. Budavari, S. Casertano, R. A. Downes, T. Donaldson, et al.,  AJ, 151, 134 (2016)

## Appendix

Analytical expressions which describe the position of tangential arcs in a
spherically symmetric gravitational lensing system, in which the lens density
profile is described by the Navarro-Frenk-White model, are presented in
Bartelmann (1996). The properties of gravitational lenses with axial symmetry
are determined by their surface density $\kappa(x)=\Sigma(x)/\Sigma_{cr}$,
where

$\Sigma_{\mathrm{cr}}=\frac{c^{2}}{4\pi
G}\frac{D_{\mathrm{s}}}{D_{\mathrm{d}}D_{\mathrm{ds}}},$ (4)

$x$ is the coordinate along the radius in $R_{s}$ units, $c$ is the speed of
light, $G$ is the gravitational constant, $D_{d}$, $D_{s}$ is the angular
diameter distance from the observer to the lens and to the source,
respectively, $D_{ds}$ \- angular diameter distance from the lens to the
source. The mass enclosed within a circle of radius $x$ is given by

$m(x)\equiv 2\int_{0}^{x}dyy\kappa(y).$ (5)

In case of spherically symmetric lenses, the source images in the form of
tangential arcs appear close to the tangential critical curve, which is
determined by the condition $m(x)=x^{2}$. The volumetric density $\rho(r)$,
defined by the formula (2), corresponds to the dimensionless surface density

$\kappa(x)=2\kappa_{\mathrm{s}}\frac{f(x)}{x^{2}-1},$ (6)

where
$\kappa_{\mathrm{s}}\equiv\rho_{\mathrm{s}}r_{\mathrm{s}}\Sigma_{\mathrm{cr}}^{-1}$.
The dimensionless mass $m(x)$ is given by the expression

$m(x)=4\kappa_{\mathrm{s}}g(x),$ (7)

where

$g(x)=\ln\frac{x}{2}+\left\\{\begin{array}[]{ll}\frac{2}{\sqrt{x^{2}-1}}\arctan\sqrt{\frac{x-1}{x+1}}&(x>1)\\\
\frac{2}{\sqrt{1-x^{2}}}\mathrm{arctanh}\sqrt{\frac{1-x}{1+x}}&(x<1)\\\
1&(x=1)\end{array}\right.$


# Structure Aware and Class Balanced 3D Object Detection on nuScenes Dataset

Sushruth Nagesh∗+, Asfiya Baig∗+, Savitha Srinivasan∗+, Akshay Rangesh+, Mohan
Trivedi+
UC San Diego

###### Abstract

3-D object detection is pivotal for autonomous driving. Point cloud based
methods have become increasingly popular for 3-D object detection, owing to
their accurate depth information. NuTonomy’s nuScenes dataset [1] greatly
extends commonly used datasets such as KITTI in size, sensor modalities,
categories, and annotation numbers. However, it suffers from severe class
imbalance. The Class-balanced Grouping and Sampling paper [21] addresses this
issue and suggests augmentation and sampling strategy. However, the
localization precision of this model is affected by the loss of spatial
information in the downscaled feature maps. We propose to enhance the
performance of the CBGS model by designing an auxiliary network [3], that
makes full use of the structure information of the 3D point cloud, in order to
improve the localization accuracy. The detachable auxiliary network is jointly
optimized by two point-level supervisions, namely foreground segmentation and
center estimation. The auxiliary network does not introduce any extra
computation during inference, since it can be detached at test time.

##  1 Introduction

3D object detection serves as an essential basis of visual perception, motion
prediction, and planning for autonomous driving. Many of the current methods
rely on point cloud data for accurate 3D detection. Datasets such as KITTI,
which are widely used for this application have several drawbacks. They are
limited in size, and have fewer categories that are not entirely
representative of real-world scenarios. Many new datasets and testbeds like
LISA-A [12], LISA-Audi[15], LISA-T [2], Waymo[14], nuScenes[1] etc have
contributed significantly in terms of data volumes and complexities.

Out of these datasets, nuScenes dataset has the largest number of lidar sweeps
and it is collected under different environment conditions. nuScenes dataset
has 10 categories for 3D detection as compared to 3 in KITTI, and also has 360
degree coverage in both vision and LIDAR modalities. 360 degree full-surrond
data is important for reliable object detection and tracking as shown by
Rangesh et al [10] where a full-surround camera and LiDAR based approach for
tracking multiple objects in the autonomous driving context is developed. The
3D ground truth boxes also have attributes such as velocities associated with
them, that aids in the classification process. However, a major drawback of
the nuScenes dataset is its inherent class imbalance problem. The number of
examples for common object classes outnumbers that of rare object classes by a
large margin. To address this issue, the CBGS [21] paper proposes grouping and
sampling strategies. It generates a smoother class distribution by improving
the average density of rare classes in the training split.

Figure 1: Struture Aware and Class Balanced 3D object detection overview

Although CBGS addresses the class imbalance problem, the network architecture
does not utilize structure information of point cloud data, which leaves room
for improvement in the model’s localization performance. To enhance the
model’s localization precision, we propose to include an auxiliary network
[3]. The network exploits fine-grained structure information of point cloud by
converting convolutional features back to point level representations. This is
performed in order to learn object-sensitive boundaries and intra-object
relationships. This additional network does not add to the computational
overhead during inference, since it can it can be removed while testing. The
auxiliary network performs two main tasks- foreground segmentation and center
estimation. The segmentation task enables the backbone network to more
precisely detect the object boundary. The center estimation task aids in
learning the relative position of each object point with respect to the object
center, leading to more accurate localization.

The main contributions of this paper can be summarized as below:

  * •

Introducing an auxiliary network into the existing CBGS architecture to
improve localization accuracy.

  * •

Introducing multi-group Non-Maximal Suppression (NMS) in the multi-group head
network to decrease the number of false positive bounding boxes.

##  2 Related Work

###  2.1 3D Object Detection

There are several methods proposed to tackle the task of 3D object detection.
Rangesh et al [11] perform 3D object detection using single monocular images
by leveraging the ground plane. Some works like [6], [18], [4] convert point
clouds to the bird-eye-view format, following which a 2D CNN is applied to
obtain the 3D detection results. Some methods convert the point cloud into
voxels [13], [16], [20] and apply 3D feature extractors using 3D Sparse
Convolutions. Several Point-based methods first predict 2D boxes from the
image, and then estimate the location, size and orientation of the 3D objects
using the cropped point cloud by applying PointNet++ [9]. However, most of the
above methods don’t perform well on all categories, especially the rare
classes such as bicycle. Therefore, the class imbalance problem needs to be
addressed in a more effective manner. Zhu et al [21] address the class
imbalance by designing a class-balanced sampling and augmentation strategy to
generate a well balanced data distribution. Moreover, they use a balanced
grouping head to improve the performance for categories with similar shapes
and dataset sizes.

###  2.2 Auxiliary Task Learning

In the domain of autonomous driving, researchers have proposed to incorporate
additional modules providing contextual information to boost the detection
accuracy. The authors of SA-SSD [3] propose to improve the localization
accuracy of single-stage detectors by exploiting the structure information of
3D point cloud using an auxiliary network. Yang et al [17] suggested to
extract geometric and semantic features from High Definition (HD) Maps to
enhance the network’s awareness about road geometry . Liang et al [5] proposed
feature fusion method which combines information coming from multiple scales
and 3D space. Zhao et al [19] proposed to improve the crowd counting by
leveraging heterogeneous attributes of the density map as guidance to fully
utilize the potential of the underlying representations without explicitly
changing the extracted features. Mordan et al [8] suggested to use auxiliary
supervision to learn the scene-aware feature for improving the robustness of
detecting objects. Our approach is mainly motivated by the auxiliary network
suggested by [3], however differ from them at the same time as we are
augmenting this into a two stage network[21] and using nuscenes dataset[1]

Figure 2: Example of a point cloud with ground truth boxes Figure 3: Model
architecture

##  3 Methodology

The whole 3D object detection architecture flow diagram is presented in figure
1. The main modules of the architecture include Voxelization Unit, 3D Feature
Extractor backbone, Auxiliary Network, Region Proposal Network and Multi head
network. Each of the modules are explained in detail in the following sub
sections. In addition, input data sampling and augmentation strategies are
explained in the following section.

###  3.1 Input and Augmentation

Each Point cloud frame in NuScenes 3D object dataset has the format
$(x,y,z,intensity,ringindex)$. Also each frame is accompanied by a timestamp.
As followed in [1], 10 lidar sweeps are aggregated together to form a dense
point cloud. An example of one such aggregation of lidar sweeps with ground
truth boxes is shown in figure [2].The keyframes (samples) are always
considered and in between keyframes, 9 other non keyframes (sweeps) are used
for aggregation. The input point cloud format used is of the form
$(x,y,z,intensity,\Delta t)$ [21]. $\Delta t$ is the time lag of non-keyframe
with respect to keyframe.

NuScenes dataset has severe class imbalance problem. To attenuate this problem
DS sampling strategy used in CBGS [21] is adopted. If examples of a class is
less, then more number of duplicates are created to sample from. DS sampling
smoothens the class distribution curve. Another common augmentation strategy
GT-AUG proposed in SECOND [16] is used to maintain a database of point clouds
of various objects and augment the training samples with it.

VoxelNet[20] based voxelization strategy is followed. Voxel size of
$[0.1m,0.1m,0.2m]$ is used. The point cloud is divided into voxels and each
voxel is represented byb the average of all the points present in it.

###  3.2 Backbone and detection networks

The backbone network down samples the voxelized representation of the point
cloud into a smaller but semantically strong and dense feature representation.
This is achieved using a deep network of sparse 3D convolutions with skip
connections. Though the final output feature representation of the backbone is
semantically accurate, it lacks spatial accuracy. This is addressed through an
auxiliary network proposed in the following section.

An end to end trainable Region Proposal Network used in VoxelNet [20] is
adopted. The RPN consists of series of down sampling layers and concatenation
of each down sampled output to get an high resolution feature map.

###  3.3 Auxiliary network

As we progressively down sample in the backbone network, the spatial
resolution of the feature maps decrease. So, the features at the boundaries
may get merged with background. Also, there is ambiguity in determining the
scale and shape of bounding boxes since the feature maps are very sparse.

Auxiliary network solves this by making the backbone network structurally
aware. The auxiliary network proposed in SA-SSD [3] is adopted. This auxiliary
network with point wise supervision is used to guide the intermediate layers
of the backbone to extract fine grained structure of the point cloud.Figure
[2]

The auxiliary network maps each non zero index of backbone feature to real
world coordinates so that each backbone feature can be represented in a point-
wise form. This representation is denoted by ${(f_{j},p_{j}):j=1,2,...M}$
where $f$ is the feature vector and $p$ is the point coordinate. Now
interpolation is used to generate full resolution feature vector $\tilde{f}$
at the original point cloud coordinates $(p_{i}:i=1,2,,...N)$. The final
representation is of the form ${(\tilde{f_{j}},p_{i}):i=1,2,...N}$.

The feature vector at each point is calculated by:

$\tilde{f_{i}}=\dfrac{\sum_{j=1}^{M}w_{j}(p_{i})f_{j}}{\sum_{j=1}^{M}w_{j}(p_{i})}$
(1)

where,

$w_{j}(p_{i})=\begin{cases}\dfrac{1}{||p_{i}-p_{j}||},&\text{if $p_{j}\epsilon
N(p_{i})$}\\\ 0,&\text{otherwise}\end{cases}$ (2)

Here $N(p_{i})$ denotes a ball of region which has a radius of 0.2m,0.4m and
0.8m at different steps. We concatenate the point wise features generated
across different down sampling layers and use a shallow fully connected
predictor to generate task specific outputs. The two task specific heads are
segmentation and center point estimation heads.

There are two auxiliary tasks defined. The first task applies sigmoid function
to the output of segmentation head. This helps in calculating foreground
background segmentation loss. This loss helps in making the network more
boundary aware. The next task estimates the center of each object in the point
cloud sample based on the regression head output ($R^{Nx3}$). A center
estimation smooth L1 loss is defined in the following sections. This loss
helps the network learn the inter object relationships.

Finally, the auxiliary loss is used only for training. It is disabled while
testing. So there is no inference time overhead.

###  3.4 Class balanced grouping

We adopt the class balanced grouping strategy [21] to address the class
imbalance in the nuScenes dataset. The idea is that classes of similar shape
or size are easier to learn from the same task. Classes with similar shapes or
sizes share some common features that can compensate for each other during
training. The model first recognizes the superclass or group to which an
object belongs, and within each group there are object classification,
regression and orientation classification heads. Further, in order to balance
the instance numbers in each group, major classes are not included as a part
of groups. For example, since cars account for 43.7% of the dataset, adding
this to a group will dominate the learning process.

The 10 classes are thus split into 6 groups: (Car), (Truck, Construction
Vehicle), (Bus, Trailer), (Barrier), (Motorcycle, Bicycle), (Pedestrian,
Traffic Cone).

###  3.5 Multi-Group Non-Maximum Suppression

During inference, in addition to performing Non-Maximum Suppression (NMS)
within each group, we also apply NMS across all groups to further reduce the
overlapping predictions. A score threshold of 0.1 and an IoU threshold of 0.3
is used. The NMS technique used also takes into account the predicted yaw
angle to compute the IoU.

###  3.6 Loss functions

The three branches in each group are object classification, bounding box
regression and orientation classification. Regression without normalization is
performed for velocity estimation. Different classes have different anchor
dimensions depending on the class mean values. This is adopted from VoxelNet
[20]. The losses used are weighted focal loss, smooth l1 loss and softmax
cross-entropy loss for classification, regression and orientation
classification respectively. The regression terms are:
$(x,y,z,l,w,h,v_{x},v_{y})$.

In the auxiliary network, we use focal loss for the foreground segmentation
task, and smooth-l1 loss for center estimation.

$\small{L_{seg}=\dfrac{1}{N_{pos}}\sum_{1}^{N}-\alpha(1-\hat{s_{i}})^{\gamma}\log(\hat{s_{i}})}$
(3) $\hat{s_{i}}=\begin{cases}\tilde{s_{i}},&\text{if $s_{i}=1$}.\\\
1-\tilde{s_{i}},&\text{otherwise}.\end{cases}$ (4)

where $\tilde{s_{i}}$ denotes the predicted foreground/background probability
of each point and $s_{i}$ is a binary label to indicate whether a point falls
into a ground-truth bounding box. $\alpha$ and $\gamma$ are hyper parameters,
and values of 0.25 and 2 are used for the same.

$L_{ctr}=\dfrac{1}{N_{pos}}\sum_{1}^{N}Smooth-l1(\Delta\tilde{p}-\Delta
p).1[s_{i}=1]$ (5)

where $N_{pos}$ is the number of foreground points and $1[.]$ is an indicator
function.

The detection and auxiliary tasks are jointly optimized using the following
objective:

$L=L_{cls}+\omega L_{box}+\beta L_{orient}+\mu L_{seg}+\lambda L_{ctr}$ (6)

##  4 Implementation details

###  4.1 Network

Voxelization: The voxel size is $s_{x}=0.1,s_{y}=0.1,s_{z}=0.2$ meters. Point
cloud within the range of [-50.4, 50.4], [-51.2, 51.2], [-5.0, 3.0] meters in
X, Y, Z axis respectively is considered, and the number of voxels is 1008 x
1024 x 40.

For the 3D feature extractor, 16, 32, 64, 128 channels of sparse 3D
convolution respectively for each block is used. We take outputs from layers
8,12,16 and 18 for the auxiliary network. Each nonzero index of the backbone
feature is converted to real-world coordinates based on the quantization step
of the current stage, so that it can be represented in a point-wise form. To
generate full-resolution point-wise features, a employ a feature propagation
layer [9] is employed at each stage to interpolate backbone features at the
coordinates of original point cloud. For interpolation, the inverse distance
weighted average among all the points in a neighboring region is used.

Data augmentation is applied during training by using a random flip in the
x-axis. Scaling is performed with a scale factor sampled from [0.95, 1.05].
Rotation is applied around Z axis between [-0.3925, 0.3925] radians and
translation is performed in the range [0.2, 0.2, 0.2] m in all the axes.

###  4.2 Training

The model was trained on Nvidia Gtx 1080 GPU. We use adamW [7] optimizer
together with one-cycle policy with a maximum learning rate of 0.04, with
division factor 10. The momentum ranges from 0.95 to 0.85, and a fixed weight
decay 0.01 is used to achieved faster convergence. The model is trained for 5
epochs with batch size 4. During inference, top 1000 proposals are retained in
each group, then NMS with score threshold of 0.1 is applied, with an IoU
threshold of 0.2. A threshold of 80 is set for the number of boxes in each
group after NMS.

##  5 Experimental Evaluation

###  5.1 Dataset

The nuScenes dataset [1] consists of 1000 driving scenes collected in Boston
and Singapore.

For object detection and tracking problems, 23 objects classes are annotated
with accurate 3D bounding boxes at 2Hz. Additionally, object-level attributes
such as visibility, activity and pose are also annotated. The entire dataset
contains 1.4M camera images, 390K LIDAR sweeps, 1.4M RADAR sweeps, and 1.4M
object bounding boxes in 40k keyframes.

Method | mAP | mATE | mASE | mAOE | mAVE | mAAE | NDS
---|---|---|---|---|---|---|---
CBGS | 18.96 | 0.57 | 0.36 | 1.22 | 1.40 | 0.32 | 26.8
BS-CBGS ($\mu=1,\lambda=0$) | 16.54 | 0.62 | 0.30 | 1.087 | 2.79 | 0.37 | 25.1
SA-CBGS ($\mu=1,\lambda=2$) | 15.56 | 0.588 | 0.314 | 1.23 | 1.73 | 0.223 | 26.5
SA-CBGS ($\mu=0.5,\lambda=1$) | 20.10 | 0.56 | 0.37 | 1.26 | 1.48 | 0.32 | 27.37
SA-CBGS ($\mu=2,\lambda=4$) | 20.67 | 0.527 | 0.286 | 1.10 | 2.23 | 0.284 | 29.36
SA-CBGS+NMS ($\mu=2,\lambda=4,$ t=0.1) | 19.56 | 0.506 | 0.283 | 1.08 | 2.15 | 0.27 | 29.08
Table 1:  Evaluation Metrics-10% data Method | mAP | mATE | mASE | mAOE | mAVE | mAAE | NDS
---|---|---|---|---|---|---|---
CBGS-40GB | 26.05 | 0.48 | 0.29 | 1.24 | 0.59 | 0.23 | 36.95
SA-CBGS-40GB | 26.37 | 0.49 | 0.29 | 1.24 | 0.63 | 0.25 | 36.41
Table 2:  Evaluation Metrics-33% data Method | Car | Truck | Construction Vehicle | Bus | Barrier | Pedestrian | Traffic Cone
---|---|---|---|---|---|---|---
CBGS | 55.79 | 8.80 | 4.10 | 10.76 | 16.17 | 53.90 | 31.21
BS-CBGS ($\mu=1,\lambda=0$) | 43.75 | 5.13 | 3.35 | 5.87 | 19.94 | 49.89 | 34.69
SA-CBGS($\mu=1,\lambda=2$) | 46.7 | 5.79 | 2.60 | 10.74 | 13.98 | 45.03 | 30.01
SA-CBGS ($\mu=0.5,\lambda=1$) | 52.85 | 14.54 | 7.37 | 20.06 | 17.46 | 52.54 | 34.69
SA-CBGS ($\mu=2,\lambda=4$) | 56.30 | 12.8 | 10.0 | 16.27 | 19.83 | 56.70 | 29.6
SA-CBGS+NMS ($\mu=2,\lambda=4$, t=0.1) | 52 | 11.2 | 10.12 | 15.4 | 16.57 | 56.29 | 28.89
Table 3:  mAP values for different classes

(a) PR curve for car on CBGS (b) PR curve for car on SA-CBGS

Figure 4: PR curves for class car for CBGS and SA-CBGS

###  5.2 Metrics

The metrics for the nuScenes detection task will be described below. The final
score, also known as the nuScenes Detection Score is a weighted sum of the
mean Average Precision (mAP) and the True Positive (TP) metrics.

####  5.2.1 Average Precision Metric

For the Average Precision metric, a match is defined based on the 2D center
distance on the ground plane. This is chosen in place of intersection over
union (IoU) in order to account for objects with relatively smaller
footprints, like bikes and pedestrian. If such objects are detected, a small
error in translation can lead to an IoU of 0, which is undesirable.

The AP is computed as the normalized area under the precision recall curve for
precision and recall over 10%. The mAP is obtained by averaging over the AP
obtained for the matching thresholds of D = 0.5, 1, 2, 3 metres and a set of
classes C.

$mAP=\frac{1}{|C||D|}\Sigma_{c\in C}\Sigma_{d\in D}AP_{c,d}$

####  5.2.2 True Positive Metrics

For every prediction that matched with a ground truth box, a set of True
Positive metrics are measured. The metrics are computed using the matching
distance threshold of d = 2 metres.

  * •

Average Translation Error (ATE): 2D Euclidean distance between the predicted
and ground truth centers (units in meters).

  * •

Average Scale Error: Error in 3D IoU after alignment of orientation and
translation. Calculated as (1 - IoU).

  * •

Average Orientation Error: Yaw angle difference between the prediction and
ground truth (units in radians).

  * •

Average Velocity Error: L2 norm of the difference in velocities (units in
m/s).

  * •

Average Attribute Error: 1 - acc, where acc is the attribute classification
accuracy for attributes such as sitting or standing.

For each TP metric, the mean TP metric (mTP) is computed over all classes,

$mTP=\frac{1}{|C|}\Sigma_{c\in C}TP_{c}$

####  5.2.3 nuScenes Detection Score

As the mAP does not capture all aspects of the nuScenes detection tasks, like
velocity and attribute estimation, the nuScenes Detection Score is used as the
final score. It is computed by consolidating the mAP and TP metrics into a
scalar score,

$NDS=\frac{1}{10}[5mAP+\Sigma_{mTP\in TP}(1-min(1,mTP))]$

where the mAP is the mean Average Precision, and TP is the set of the 5 mean
True Positive metrics.

###  5.3 Experiments

A series of experiments were conducted to analyze the effect of incorporating
auxiliary network into CBGS architecture[21]. First a model based on CBGS [21]
was trained to get baseline model to compare with. Then, loss weights for
center estimation ($\mu$) and foreground-background estimation ($\lambda$)
suggested by SA-SSD model [3] was used to train the proposed SA-CBGS network.
Then, a range of experiments for various values of $\mu$ and $\lambda$ were
tried to tune the loss weights. Also, an experiment using only the foreground-
background segmentation loss was tried to evaluate the spatial localization of
the trained model. All these experiments were carried out with 10% of the
nuScenes dataset. Another set of experiments evaluated the baseline and SA-
CBGS model on 33% of the nuScenes dataset. Finally, inter class Non Maximal
Suppression was applied on a trained SA-CBGS model with weights $\lambda$ = 4
and $\mu$ = 2.

###  5.4 Results

Table 1 contains the the evaluation metrics for the experiments described
above. Upon fine-tuning the weights for computing the auxiliary loss, the
weights $\mu=2,\lambda=4$ outperformed the baseline by 1.71 points. The SA-
CBGS networks with any other auxiliary loss weights failed to match the
metrics of the baseline. The reasoning for this could be that the contribution
of the loss in the weight updates was not high enough to have an impact. It
can also be seen that the mATE and mASE have significantly reduced as these
can be directly correlated to the auxiliary tasks of foreground segmentation
and center point estimation. The foreground segmentation task makes the
features more aware of the boundary, and therefore the scale of the objects.
The center point estimation task influences the feature maps to reduce the
translation error in the centers of the predicted bounding boxes.

As the best weight values were found to be $\mu=2,\lambda=4$, the rest of the
experiments were conducted with those as the auxiliary loss weight values.
Inter-group NMS was applied as a post processing step, and although it seems
to have visually improved the results the evaluation metrics have not showed
improvement. This is because the threshold on IoU could lead to False
Negatives, hence deteriorating performance.

The mAP values for each category are compared in Table 3. Although there is no
particular trend among the results of the different experiments, the
structurally aware models in general seem to outperform the baseline.

Another experiment with more amount of data (33%) was performed with loss
weights $\mu=1,\lambda=2$. The results for this are reported in Table 2. Here
the mAP values for both the baseline and SA-CBGS models were comparable. This
can be attributed to using of different non optimal loss weights which was
figured out in the previous experiments. Also the mAP values obtained with 33%
of the data were significantly more than the best performing model on 10% of
the data. This suggests us that the previous model on less amount of data was
under fitting and has scope for improvement.

The precision recall curves for the Car category computed using various
distance values can be seen in Figure 4. The PR curve for for SA-CBGS is
reported for the best performing model with $\mu=2,\lambda=4$. The area under
the curve is slightly higher for the SA-CBGS model, which explains the higher
mAP compared to the baseline. Examples of qualitative detection results can be
seen in Figures 5, and 6. The edge with a line attached in the bounding box
indicates the vehicle’s front. Figure 7 contains the detection results after
applying NMS on the predicted boxes. The green colored boxes are the ground
truth 3D object boxes and the blue boxes are the predicted 3D object boxes.

##  6 Conclusion

We proposed to use an auxiliary network to guide the features learned by the
3D extractor to be more aware of the structure information of the 3D objects.
As the feature maps are more aware of the structure, the object detection
performance outperforms the baseline significantly in terms of multiple
evaluation metrics including the nuScenes Detection Score.

(a) Qualitative results of CBGS - scene 1

(b) Qualitative results of SA-CBGS - scene 1

Figure 5: Qualitative results - scene 1

(a) Qualitative results of CBGS - scene 2

(b) Qualitative results of SA-CBGS - scene 2

Figure 6: Qualitative results - scene 2

(a) Original detection

(b) Detection with NMS

(c) Original detection

(d) Detection with NMS

Figure 7: Effect of multi-group NMS

## References

  * [1] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving, 2019.
  * [2] Nachiket Deo, Nasha Meoli, Akshay Rangesh, and Mohan M. Trivedi.  On control transitions in autonomous driving: A framework and analysis for characterizing scene complexity.  In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 2422–2426. IEEE, 2019.
  * [3] Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang.  Structure aware single-stage 3d object detection from point cloud.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [4] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  CoRR, abs/1812.05784, 2018.
  * [5] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun.  Multi-task multi-sensor fusion for 3d object detection.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [6] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun.  Deep continuous fusion for multi-sensor 3d object detection.  In The European Conference on Computer Vision (ECCV), September 2018\.
  * [7] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  CoRR, abs/1711.05101, 2017.
  * [8] Taylor Mordan, Nicolas Thome, Gilles Henaff, and Matthieu Cord.  Revisiting multi-task learning with rock: a deep residual auxiliary block for visual detection.  In Advances in Neural Information Processing Systems, pages 1310–1322, 2018.
  * [9] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  CoRR, abs/1706.02413, 2017.
  * [10] Akshay Rangesh and Mohan Manubhai Trivedi.  No blind spots: Full-surround multi-object tracking for autonomous vehicles using cameras and lidars.  IEEE Transactions on Intelligent Vehicles, 4:588–599, 2019.
  * [11] Akshay Rangesh and Mohan M Trivedi.  Ground plane polling for 6dof pose estimation of objects on the road.  IEEE Transactions on Intelligent Vehicles, 2020.
  * [12] Akshay Rangesh, Kevan Yuen, Ravi Kumar Satzoda, Rakesh Nattoji Rajaram, Pujitha Gunaratne, and Mohan Manubhai Trivedi.  A multimodal, full-surround vehicular testbed for naturalistic studies and benchmarking: Design, calibration and deployment.  ArXiv, abs/1709.07502, 2017.
  * [13] Shuran Song and Jianxiong Xiao.  Deep sliding shapes for amodal 3d object detection in rgb-d images.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
  * [14] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset, 2019\.
  * [15] A. Tawari, S. Sivaraman, M. M. Trivedi, T. Shannon, and M. Tippelhofer.  Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking.  In 2014 IEEE Intelligent Vehicles Symposium Proceedings, pages 115–120, 2014.
  * [16] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, Oct 2018.
  * [17] Bin Yang, Ming Liang, and Raquel Urtasun.  Hdnet: Exploiting hd maps for 3d object detection.  In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 146–155. PMLR, 29–31 Oct 2018\.
  * [18] Bin Yang, Wenjie Luo, and Raquel Urtasun.  PIXOR: real-time 3d object detection from point clouds.  CoRR, abs/1902.06326, 2019.
  * [19] Muming Zhao, Jian Zhang, Chongyang Zhang, and Wenjun Zhang.  Leveraging heterogeneous auxiliary tasks to assist crowd counting.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [20] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  CoRR, abs/1711.06396, 2017.
  * [21] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu.  Class-balanced grouping and sampling for point cloud 3d object detection.  2019\.