# Renormalizability of the leading order operator for neutrinoless double beta
decay with the effects of finite nucleon size

Tai-Xing Liua, Ri-Guang Huanga,b and Dong-Liang Fanga,b<EMAIL_ADDRESS>aInstitute of Modern Physics, Chinese Academy of Sciences, Lanzhou 730000,
China  bSchool of Nuclear Sciences and Technology, University of Chinese
Academy of Sciences, Beijing 101408, China

###### Abstract

The fundamental process of neutrinoless double beta decay, $nn\to
ppe^{-}e^{-}$, dominated by the exchange of light Majorana neutrinos, is
studied in the framework of chiral effective field theory. Considering
neutrinos as virtual states, we evaluate the contributions of finite nucleon
size to the transition amplitude in a non-perturbative manner, as opposed to
expanding these effects in powers of momentum. Based on the nucleon form
factors expressed in terms of the dipole and Kelly parametrizations, we find
that, at the leading order, the present scheme could renormalize the amplitude
in the context of the standard mechanism and provide predictions consistent
with the previous investigations. Consequently, we argue that the impact of
the effects of finite nucleon size on the amplitude is comparable to that of
the leading-order contact term introduced in Cirigliano et al. (2018a) in a
perturbative scheme. Our results provide not only a benchmark calculation for
the transition amplitude between two schemes but also evidence for the
reasonableness of non-perturbative treatment for the effects of finite nucleon
size in conventional nuclear many-body methods.

††preprint: APS/123-QED

_Introduction._ —Neutrinoless double beta (0$\nu\beta\beta$) decay, a
hypothetical second-order weak transition that converts two neutrons in a
nucleus into two protons through the emission of two electrons but without
neutrinos Furry (1939), is an ideal venue for probing lepton number violation
(LNV). Besides demonstrating that neutrinos are Majorana fermions Schechter
and Valle (1982), its observation provides insightful perspectives to explain
the neutrino mass generation Mohapatra and Senjanovic (1980), determines the
absolute neutrino mass scale and hierarchy Avignone et al. (2008), and
explores the mechanism of the matter-antimatter asymmetry in early universe
Davidson et al. (2008). Currently, numerous underground experiments with high
sensitivity Ni et al. (2019); Agostini et al. (2020); Giachero et al. (2021);
Abe et al. (2023) all over the world are actively searching for
0$\nu\beta\beta$ decay signals for different even-even nuclei. In the future,
more ton-scale facilities Abgrall et al. (2017); Albert et al. (2018); Armatol
et al. (2022); Cao et al. (2024) will be deployed to improve the discovery
potential of experiments and measure the half-life of different candidate
isotopes (for a recent review, see Ref. Agostini et al. (2023)).

These experiments will help us make better understanding of this process. In
general, whatever operator including the LNV physics may induce the nuclear
0$\nu\beta\beta$ decay. To reveal the underlying mechanism of this process,
the precise calculation of nuclear matrix elements (NMEs) that incorporates
various LNV contributions is essential. However, its estimation based on the
different many-body methods suffers from a discrepancy of a factor of three to
five thus far Engel and Menéndez (2017). One should note that the uncertainty
of NMEs is rooted in the realization of nuclear structure and LNV mechanisms.

It has been shown that the chiral effective field theory (ChEFT) Weinberg
(1990, 1991) offers a promising tool to quantify such uncertainties in nuclear
calculations, for a comprehensive review see Ref. Hammer et al. (2020).
Moreover, the electroweak currents can be introduced systematically in ChEFT
Pastore et al. (2009); Baroni et al. (2016); Krebs et al. (2017, 2019). By
using series of EFTs, the various LNV sources at energy well above the
electroweak scale can be expressed as the 0$\nu\beta\beta$ operators at the
nucleon level Prezeau et al. (2003); Cirigliano et al. (2017); Cirigliano et
al. (2018b, c); Dekens et al. (2020); de Vries et al. (2022). With ChEFT, ab
initio calculations under the standard light Majorana neutrino-exchange
mechanism Weinberg (1979) for NMEs were performed Yao et al. (2020); Belley et
al. (2021); Novario et al. (2021); Wirth et al. (2021); Belley et al. (2024).

Recently, within the standard mechanism of 0$\nu\beta\beta$ decay, the ChEFT
description based on naive dimensional analysis (NDA) Cirigliano et al.
(2018b) has been proven to violate the requirement of renormalizability
Cirigliano et al. (2018a); Cirigliano et al. (2019). As a consequence, a
contact operator that encodes the effect of hard-neutrino exchange is promoted
to leading order (LO). The corresponding low-energy constant (LEC) is
currently still undetermined by experiments or lattice QCD simulations,
introducing additional uncertainties to the estimation of NMEs besides those
from nuclear structure.

The prescription to estimate the size of the LEC was first presented by
Cirigliano et al. in Cirigliano et al. (2021a, b). Although this approach by
matching the $nn\to ppe^{-}e^{-}$ amplitude onto the Cottingham formula
Cottingham (1963), is somewhat model-dependent, the uncertainty is acceptable
at LO calculation for ChEFT. On the other hand, Yang et al. Yang and Zhao
(2023) developed a relativistic method that can renormalize the LO decay
amplitude by resorting to the light neutrino exchange rather than utilizing a
contact term. The two distinct approaches show decent consistency in the
prediction of the LO amplitude.

An essential ingredient used to evaluate the 0$\nu\beta\beta$ amplitude is the
coupling of lepton-neutrino current to a nucleon weak current. This coupling
can be decomposed into all structures permitted by Lorentz invariance, whose
kinematics is factorized into the momentum-dependent functions, i.e., nucleon
form factors (FFs), encoding the effects of finite nucleon size. As an
application of ChEFT for the low-energy electroweak processes for nucleons,
the nucleon FFs can be expressed in terms of powers of momentum (see Ref.
Bernard et al. (1995) for a review), in which the LO term corresponds to the
point-like assumption of the nucleon. It is important to emphasize that the
prerequisite for this perturbative expansion is the observability of the
external source particles. If one of the particles is virtual, the nucleon FFs
will vary based on the momentum of the intermediate state. From this
perspective, a non-perturbative approach to incorporate nucleon structure
effects may provide novel insights into the renormalization of the
0$\nu\beta\beta$ amplitude. Furthermore, it is worth stressing that the
aforementioned nucleon FFs are inappropriate for assessing the impact of
nucleon structure on nuclear forces because nuclear forces are so strong that
the internal structure of the nucleon changes during strong process.

In this study, we revisit the $nn\to ppe^{-}e^{-}$ transition mediated solely
by the light Majorana neutrinos to examine the renormalizability of
0$\nu\beta\beta$ operators. The finite size of the nucleon is taken into
account in a non-perturbative manner, similar to methodologies employed in
nuclear many-body calculations Simkovic et al. (1999, 2009); Hyvärinen and
Suhonen (2015); Yao et al. (2022). Using the momentum-dependent FFs, we obtain
the finite and regulator-independent amplitude at LO in the absence of short-
range contribution. It shows the model-free advantage of this method and the
validity of the way to treat nucleon structure in conventional nuclear
calculation.

_Theoretical framework._ —We focus on the scenario of light Majorana neutrino
mediated $nn\to ppe^{-}e^{-}$ transition. The scattering of initial and final
state particles is restricted to the ${}^{1}S_{0}$ channel because it has been
proved in Refs. Cirigliano et al. (2018a); Cirigliano et al. (2019) that the
conflict between NDA and renormalizability only appears in this channel. For
this purpose, the heavy-baryon formulation Jenkins and Manohar (1991); Bernard
et al. (1992) of ChEFT is employed and the Coulomb interaction is ignored.

We start with the chiral Lagrangian, the nucleon-nucleon potential derived
from two-nucleon irreducible diagrams comprises the long-range pion exchange
and the short-range four-nucleon contact terms. At LO, the two-nucleon
potential is given by

$\begin{split}V_{NN}(\boldsymbol{p}^{\prime},\boldsymbol{p})=&-\frac{g_{A}^{2}}{4f_{\pi}^{2}}\vec{\tau}_{1}\cdot\vec{\tau}_{2}\frac{\boldsymbol{\sigma}_{1}\cdot\boldsymbol{q}\,\boldsymbol{\sigma}_{2}\cdot\boldsymbol{q}}{\boldsymbol{q}^{2}+m_{\pi}^{2}}\\\
&+C_{S}+C_{T}\,\boldsymbol{\sigma}_{1}\cdot\boldsymbol{\sigma}_{2}\,,\end{split}$
(1)

where $g_{A}=1.27$ is the nucleon axial coupling, $f_{\pi}=92.2$ MeV the pion
decay constant, $m_{\pi}=138$ MeV the pion mass. The initial and final
relative momenta, denoting as $\boldsymbol{p}$ and $\boldsymbol{p}\,^{\prime}$
respectively, define the momentum transfer
$\boldsymbol{q}=\boldsymbol{p}\,^{\prime}-\boldsymbol{p}$. In the
${}^{1}S_{0}$ channel, the combination of LECs $C_{S}$ and $C_{T}$, i.e.
$\widetilde{C}=C_{S}-3C_{T}$, is needed.

In the context of the standard mechanism of 0$\nu\beta\beta$ decay, the
electron-neutrino Majorana mass term, which is the dominant contribution of
LNV at low energy, can be written as

$\mathcal{L}_{\Delta L=2}=-\frac{m_{\beta\beta}}{2}\nu_{eL}^{T}C\nu_{eL}\,,$
(2)

where $C=i\gamma_{2}\gamma_{0}$ denotes the charge conjugation operator, and
the effective mass $m_{\beta\beta}=\sum_{i}U^{2}_{ei}m_{i}$ is defined with
masses of three generations of light neutrino and elements of the neutrino
mixing matrix. The definition of the neutrino potential is similar to that of
the two-nucleon potential Cirigliano et al. (2017). At LO in NDA, only
following neutrino-exchange potential

$\begin{split}V_{\nu}(\boldsymbol{p}^{\prime},\boldsymbol{p})=&\tau_{1}^{+}\tau_{2}^{+}\frac{1}{\boldsymbol{q}^{2}}\left[G_{V}^{2}\left(\boldsymbol{q}^{2}\right)-G_{A}^{2}\left(\boldsymbol{q}^{2}\right)\,\boldsymbol{\sigma}_{1}\cdot\boldsymbol{\sigma}_{2}\right.\\\
&\left.+G_{A}^{2}\left(\boldsymbol{q}^{2}\right)\,\boldsymbol{\sigma}_{1}\cdot\boldsymbol{q}\,\boldsymbol{\sigma}_{2}\cdot\boldsymbol{q}\frac{2m_{\pi}^{2}+\boldsymbol{q}^{2}}{\left(\boldsymbol{q}^{2}+m_{\pi}^{2}\right)^{2}}\right]\end{split}$
(3)

contributes to the $nn\to ppe^{-}e^{-}$ amplitude. In light of the non-
perturbative property of the nucleon structure in the 0$\nu\beta\beta$
process, we have introduced the momentum-dependent FFs
$G_{V}\left(\boldsymbol{q}^{2}\right)$ and
$G_{A}\left(\boldsymbol{q}^{2}\right)$, which are normalized as the vector and
axial-vector coupling respectively, to take into account the finite-nucleon-
size effects.

The commonly used dipole parametrization
$G_{\alpha}\left(\boldsymbol{q}^{2}\right)=g_{\alpha}/(1+\boldsymbol{q}^{2}/\Lambda_{\alpha}^{2})^{2}$
($\alpha=V,A$) is employed in this work. Here we adopt $\Lambda_{V}=0.84$ GeV
and $\Lambda_{A}=1.05$ GeV respectively. Given that the prediction of dipole
parametrization is inconsistent with data in the high-momentum region Jones et
al. (2000); Gayou et al. (2001, 2002); Puckett et al. (2010); Meyer et al.
(2016), we additionally utilize the parametrization scheme presented by Kelly
Kelly (2004) to describe the structure of the nucleon.

The Kelly parametrization of the axial FF takes the form

$G_{A}(x)=\frac{g_{A}+a_{1}x}{1+b_{1}x+b_{2}x^{2}+b_{3}x^{3}}\,,$ (4)

where $x=\boldsymbol{q}^{2}/(4m_{N}^{2})$ with the nucleon mass $m_{N}=939$
MeV, the dimensionless model parameters $a_{1},b_{1},b_{2}$ and $b_{3}$ are
determined by fitting of data from Ref. Hashamipour et al. (2019). On the
other hand, in conjunction with the fits in Refs. Kelly (2004); Punjabi et al.
(2015) for the Sachs FFs of proton and neutron, one can obtain the vector FF
as a function of momentum via the relation

$G_{V}(x)=\frac{\left[G_{Ep}(x)-G_{En}(x)\right]+x\left[G_{Mp}(x)-G_{Mn}(x)\right]}{1+x}\,,$
(5)

where $G_{Ep}$, $G_{Mp}$, $G_{En}$ and $G_{Mn}$ are the so-called electric and
magnetic Sachs FFs, for proton and neutron, respectively.

The results of the dipole and fitted Kelly forms are shown in Fig.1.

Figure 1: Dependence of the nucleon axial (top) and vector (bottom) FFs on
momentum. The solid circles and squares with error bars represent the
experimental data of $G_{A}$ and $G_{V}$, respectively, from Ref. Hashamipour
et al. (2019) and Ref. Cates et al. (2011). The red and blue lines represent
the predictions of the dipole and Kelly parametrizations, respectively.

The prediction of the Kelly parametrization for the nucleon FFs is in
reasonable agreement with the data. One of the important points is that both
types of parametrization schemes for vector and axial FFs all predict a
general behavior:

$G_{\alpha}\propto\boldsymbol{q}^{-4}\,,\,\,\,\,\text{for}\,\,|\boldsymbol{q}|\to\infty\,.$
(6)

By this scale rule of FFs, we will demonstrate the convergence of the
0$\nu\beta\beta$ amplitude in the ultraviolet (UV) region. Comparing the
transition amplitudes based on both parametrizations, we will demonstrate the
validity of assessing the effects of finite nucleon size with dipole
parametrization on NMEs.

The LO 0$\nu\beta\beta$ amplitude based on the kinematics
$n(\boldsymbol{p})n(-\boldsymbol{p})\to
p(\boldsymbol{p}^{\prime})p(-\boldsymbol{p}^{\prime})e^{-}\left(\boldsymbol{p}_{e1}=0\right)e^{-}\left(\boldsymbol{p}_{e2}=0\right)$
can be schematically expressed as in Cirigliano et al. (2019):

$A_{\nu}=-\left(V_{\nu}+V_{\nu}G_{0}T_{s}+T_{s}G_{0}V_{\nu}+T_{s}G_{0}V_{\nu}G_{0}T_{s}\right)\,,$
(7)

where $G_{0}$ is the two-nucleon free propagator, $T_{s}$ the two-nucleon
scattering amplitude obtained by iterating the LO strong potential $V_{NN}$
through $G_{0}$. We name the four terms in Eq.(7), from left to right,
$A_{\mathcal{A}}$, $A_{\mathcal{B}}$, $\bar{A}_{\mathcal{B}}$, and
$A_{\mathcal{C}}$.

We are interested in the renormalizability of the 0$\nu\beta\beta$ amplitude.
From Eq.(7), the sub-amplitude $A_{\mathcal{A}}$ as a tree-level contribution
must be convergent. Because of the finiteness of the strong amplitude $T_{s}$,
the UV behavior of the remaining terms is dominated by the loop integrals with
the insertion of neutrino potential. The nonrelativistic propagator $G_{0}$
behaves like $\mathcal{O}\left(\Lambda^{-2}\right)$ in the UV region, where
$\Lambda$ denotes a UV energy scale much greater than the typical momentum in
the process. For the neutrino potential in the ${}^{1}S_{0}$ wave, its UV
scale rule is given by

$V_{\nu}\sim\mathcal{O}\left(\Lambda^{-2-2g}\right)\,,$ (8)

where $g=0$ (4) corresponds to the case without (with) the effects of finite
nucleon size. Therefore, in the nonrelativistic case, the superficial degree
of divergence for the last three terms in Eq.(7) has the following form:

$D=L-2N(1+g)\,,$ (9)

where $L$ is the number of loops, and $N=1$ is the number of insertions of the
neutrino potential. Given that $L=1$ and $g\geq 0$ for $A_{\mathcal{B}}$ and
$\bar{A}_{\mathcal{B}}$, it follows that the both sub-amplitudes could reach
the convergence as $\mathcal{O}\left(\Lambda^{-1-2g}\right)$. However,
$A_{\mathcal{C}}\sim\mathcal{O}\left(\Lambda^{-2g}\right)$ means that its
convergence depends on the treatment of the nucleon structure. In the case
without the effects of finite nucleon size, i.e. $g=0$, $A_{\mathcal{C}}$
diverges logarithmically such that a LEC
$g_{\nu}^{NN}\sim\mathcal{O}\left(f_{\pi}^{-2}\right)$ is required to
renormalize the LO amplitude $A_{\nu}$ Cirigliano et al. (2018a); Cirigliano
et al. (2019). With finite nucleon size, however,
$A_{\mathcal{C}}\sim\mathcal{O}\left(\Lambda^{-8}\right)$ is finite so that
$A_{\nu}$ can be renormalized without inclusion of any short-range counter
terms.

_Results and discussions._ —Following the usual numerical strategy, the
renormalizability of $A_{\nu}$ is demonstrated by its independence of the
cutoff value in momentum or coordinate space. In this work, we introduce the
momentum cutoff $\Lambda$ via the way that regulates the UV part of the LO
strong potential with a separable Gaussian regulator:

$V_{NN}(\boldsymbol{p}^{\prime},\boldsymbol{p})\longrightarrow\text{exp}\left(-\frac{\boldsymbol{p}^{\prime}\,{}^{4}}{\Lambda^{4}}\right)V_{NN}(\boldsymbol{p}^{\prime},\boldsymbol{p})\,\text{exp}\left(-\frac{\boldsymbol{p}^{4}}{\Lambda^{4}}\right).$
(10)

To determine the LEC $\widetilde{C}$ in the strong interaction, we fit $T_{s}$
to the ${}^{1}S_{0}$ wave neutron-proton scattering length $a_{np}=-23.74$ fm.
The cutoff independence of the phase shift has been achieved at
$\Lambda\to\infty$.

Choosing the kinematic points $|\boldsymbol{p}|=1$ MeV and
$|\boldsymbol{p}^{\prime}|=38$ MeV, we depict in Fig.2 the cutoff variation of
the LO amplitude $A_{\nu}$.

Figure 2: The LO amplitude $A_{\nu}$ as a function of the cutoff value
$\Lambda$. The open squares are the results without the contribution of finite
nucleon size. The results denoted by red and blue solid squares take into
account the effects of finite nucleon size, via the dipole and Kelly
parametrizations, respectively.

The ChEFT results without the contributions of nucleon structure display a
logarithmic divergence, while the amplitude with the finite-nucleon-size
effects shows a satisfactory convergence, as same as the expectation above. In
addition, the amplitudes obtained from both the dipole and Kelly
parametrizations are remarkably similar at any cutoff value, with a difference
around only 1%. This indicates that the impact of different parametrizations
of nucleon structure on 0$\nu\beta\beta$ decay is negligible, highlighting the
robustness and reliability of the parametrization in calculating the
0$\nu\beta\beta$ amplitude. Therefore, the dipole parametrization is accurate
enough to assess the contribution of the nucleon structure to NMEs.

To validate the present scheme, we calculate the amplitude for
$|\boldsymbol{p}|=25$ MeV and $|\boldsymbol{p}^{\prime}|=30$ MeV and perform a
comparison with the previous results Cirigliano et al. (2021a, b); Yang and
Zhao (2023). As shown in Fig.3,

Figure 3: The LO amplitude $A_{\nu}$ as a function of the cutoff value
$\Lambda$, in comparison with the results taken from Ref. Cirigliano et al.
(2021a) (green line) and Ref. Yang and Zhao (2023) (black line), respectively.

again, we capture the renormalized amplitude without introducing any
undeterminate contact operators. The limit value of absolute amplitude
$|A_{\nu}|=0.01956(8)$ $\text{MeV}^{-2}$ at this kinematic point, where the
error stems from the dependence on parametrization. Although the approach used
to renormalize $A_{\nu}$ in this study differs from that in Refs. Cirigliano
et al. (2021a); Yang and Zhao (2023), our results are consistent with the
estimations of Refs. Cirigliano et al. (2021a); Yang and Zhao (2023) within
the uncertainty 10%, a relatively strict upper bound for ChEFT truncation
error at LO.

The renormalized on-shell and off-shell amplitudes as functions of momentum
are shown in Fig.4. At various kinematic points, the results accounting for
the effects of nucleon structure are in quite good agreement with the
predictions based on the renormalized scheme of Refs. Cirigliano et al.
(2021a, b).

Figure 4: Dependence of the on-shell (top panel) and off-shell (bottom panel)
amplitudes on momentum for cutoff value $\Lambda=6.4$ GeV. The black line
denotes the results based on the long- and short-range transition operators,
in which the LEC is determined by reproducing the amplitude at
$|\boldsymbol{p}|=25$ MeV and $|\boldsymbol{p}^{\prime}|=$ 30 MeV, i.e.,
$A_{\nu}$ = -0.0195 $\text{MeV}^{-2}$ Cirigliano et al. (2021a).

Furthermore, the present scheme can also recover the transition amplitude for
higher partial-wave channels obtained in Ref Cirigliano et al. (2019).

The primary motivation for employing a non-perturbative approach to treat the
finite-size effects of the nucleon arises from the inadequacy of perturbative
expansion of the nucleon FFs caused by virtual neutrinos, rather than from the
behavior of FFs decreasing as $\mathcal{O}(\Lambda^{-4})$ in the UV region.
The numerical results indicate that the approach to including the suppressed
effects of FFs is equivalent to the renormalized scheme of Ref. Cirigliano et
al. (2021a) at LO. Therefore, to ensure the correctness of the result, we
argue that the calculation of the 0$\nu\beta\beta$ amplitude should be
conducted within a uniform renormalization scheme.

We note that conventional many-body calculations estimate the contributions of
finite nucleon size to NMEs in the same manner as this work. Based on the
above discussions, we conclude that these calculations introducing the nucleon
structure effects in a non-perturbative way are consistent with the scheme
that renormalizes the amplitude with short-range terms. In contrast, combining
the nucleon structure with the short-range contribution in many-body
calculation may lead to an overestimation of the amplitude, thereby amplifying
the uncertainty of NMEs.

Moreover, since we evaluate the LO 0$\nu\beta\beta$ amplitude without relying
on any unknown LECs, our approach provides a model-independent description for
0$\nu\beta\beta$ decay. This is crucial for improving the precision of NMEs
calculated in many-body methods. Besides circumventing model dependence, the
realistic size of the short-range operator as discussed in Refs. Cirigliano et
al. (2018a); Cirigliano et al. (2019); Cirigliano et al. (2021a, b) can be
addressed by matching the estimation of the decay rate based on the present
scheme.

Our framework is more straightforward for advancing higher-order calculations,
in contrast to the relativistic scheme Yang and Zhao (2023) within the
framework of manifestly Lorentz-invariant ChEFT. The latter would introduce
certain complexities when moving to higher orders, especially for the
treatment of the renormalization Hammer et al. (2020); Epelbaum et al. (2020).
Meanwhile, our method facilitates a more intuitive combination with existing
many-body techniques compared to the relativistic scheme.

_Summary._ —In this study, we present a nonrelativistic approach to
renormalize the LO 0$\nu\beta\beta$ amplitude for the standard mechanism using
the effects of nucleon structure instead of the short-distance operators with
undetermined LECs. Our calculations of the 0$\nu\beta\beta$ amplitude at
various kinematic points demonstrate the efficacy of the proposed scheme,
showing its comparability with established frameworks Cirigliano et al.
(2021a, b). The most significant implication of this equivalence is the
validation of conventional many-body calculations based on the light Majorana
neutrino mechanism, confirming the absence of any oversight regarding LO LNV
physics in these calculations. Finally, our scheme has the capacity to
transcend the specific scenario of light Majorana neutrino exchanges,
generalizing to the non-standard decay mechanisms and helping to reduce the
uncertainties of corresponding NMEs in many-body calculations.

###### Acknowledgements.

We thank Prof. F. Šimkovic for inspiring us to perform this calculation. This
work is supported by the National Key Research and Development Program of
China (2021YFA1601300), the Chinese Academy of Sciences with Project for Young
Scientists in Basic Research (YSBR- 099), and the Postdoctoral Special
Foundation of Gansu Province.

## References

  * Cirigliano et al. (2018a) V. Cirigliano, W. Dekens, J. De Vries, M. L. Graesser, E. Mereghetti, S. Pastore, and U. Van Kolck, Phys. Rev. Lett. 120, 202001 (2018a), eprint 1802.10097.
  * Furry (1939) W. H. Furry, Phys. Rev. 56, 1184 (1939).
  * Schechter and Valle (1982) J. Schechter and J. W. F. Valle, Phys. Rev. D 25, 2951 (1982).
  * Mohapatra and Senjanovic (1980) R. N. Mohapatra and G. Senjanovic, Phys. Rev. Lett. 44, 912 (1980).
  * Avignone et al. (2008) F. T. Avignone, III, S. R. Elliott, and J. Engel, Rev. Mod. Phys. 80, 481 (2008), eprint 0708.1033.
  * Davidson et al. (2008) S. Davidson, E. Nardi, and Y. Nir, Phys. Rept. 466, 105 (2008), eprint 0802.2962.
  * Ni et al. (2019) K. Ni et al. (PandaX-II), Chin. Phys. C 43, 113001 (2019), eprint 1906.11457.
  * Agostini et al. (2020) M. Agostini et al. (GERDA), Phys. Rev. Lett. 125, 252502 (2020), eprint 2009.06079.
  * Giachero et al. (2021) A. Giachero et al. (CUORE), PoS ICHEP2020, 133 (2021), eprint 2011.09295.
  * Abe et al. (2023) S. Abe et al. (KamLAND-Zen), Phys. Rev. Lett. 130, 051801 (2023), eprint 2203.02139.
  * Abgrall et al. (2017) N. Abgrall et al. (LEGEND), AIP Conf. Proc. 1894, 020027 (2017), eprint 1709.01980.
  * Albert et al. (2018) J. B. Albert et al. (nEXO), Phys. Rev. C 97, 065503 (2018), eprint 1710.05075.
  * Armatol et al. (2022) A. Armatol et al. (CUPID) (2022), eprint 2203.08386.
  * Cao et al. (2024) X.-G. Cao et al. (N$\nu$DEx-100), Nucl. Sci. Tech. 35, 3 (2024), eprint 2304.08362.
  * Agostini et al. (2023) M. Agostini, G. Benato, J. A. Detwiler, J. Menéndez, and F. Vissani, Rev. Mod. Phys. 95, 025002 (2023), eprint 2202.01787.
  * Engel and Menéndez (2017) J. Engel and J. Menéndez, Rept. Prog. Phys. 80, 046301 (2017), eprint 1610.06548.
  * Weinberg (1990) S. Weinberg, Phys. Lett. B 251, 288 (1990).
  * Weinberg (1991) S. Weinberg, Nucl. Phys. B 363, 3 (1991).
  * Hammer et al. (2020) H. W. Hammer, S. König, and U. van Kolck, Rev. Mod. Phys. 92, 025004 (2020), eprint 1906.12122.
  * Pastore et al. (2009) S. Pastore, L. Girlanda, R. Schiavilla, M. Viviani, and R. B. Wiringa, Phys. Rev. C 80, 034004 (2009), eprint 0906.1800.
  * Baroni et al. (2016) A. Baroni, L. Girlanda, S. Pastore, R. Schiavilla, and M. Viviani, Phys. Rev. C 93, 015501 (2016), [Erratum: Phys.Rev.C 93, 049902 (2016), Erratum: Phys.Rev.C 95, 059901 (2017)], eprint 1509.07039.
  * Krebs et al. (2017) H. Krebs, E. Epelbaum, and U. G. Meißner, Annals Phys. 378, 317 (2017), eprint 1610.03569.
  * Krebs et al. (2019) H. Krebs, E. Epelbaum, and U. G. Meißner, Few Body Syst. 60, 31 (2019), eprint 1902.06839.
  * Prezeau et al. (2003) G. Prezeau, M. Ramsey-Musolf, and P. Vogel, Phys. Rev. D 68, 034016 (2003), eprint hep-ph/0303205.
  * Cirigliano et al. (2017) V. Cirigliano, W. Dekens, J. de Vries, M. L. Graesser, and E. Mereghetti, JHEP 12, 082 (2017), eprint 1708.09390.
  * Cirigliano et al. (2018b) V. Cirigliano, W. Dekens, E. Mereghetti, and A. Walker-Loud, Phys. Rev. C 97, 065501 (2018b), [Erratum: Phys.Rev.C 100, 019903 (2019)], eprint 1710.01729.
  * Cirigliano et al. (2018c) V. Cirigliano, W. Dekens, J. de Vries, M. L. Graesser, and E. Mereghetti, JHEP 12, 097 (2018c), eprint 1806.02780.
  * Dekens et al. (2020) W. Dekens, J. de Vries, K. Fuyuto, E. Mereghetti, and G. Zhou, JHEP 06, 097 (2020), eprint 2002.07182.
  * de Vries et al. (2022) J. de Vries, G. Li, M. J. Ramsey-Musolf, and J. C. Vasquez, JHEP 11, 056 (2022), eprint 2209.03031.
  * Weinberg (1979) S. Weinberg, Phys. Rev. Lett. 43, 1566 (1979).
  * Yao et al. (2020) J. M. Yao, B. Bally, J. Engel, R. Wirth, T. R. Rodríguez, and H. Hergert, Phys. Rev. Lett. 124, 232501 (2020), eprint 1908.05424.
  * Belley et al. (2021) A. Belley, C. G. Payne, S. R. Stroberg, T. Miyagi, and J. D. Holt, Phys. Rev. Lett. 126, 042502 (2021), eprint 2008.06588.
  * Novario et al. (2021) S. Novario, P. Gysbers, J. Engel, G. Hagen, G. R. Jansen, T. D. Morris, P. Navrátil, T. Papenbrock, and S. Quaglioni, Phys. Rev. Lett. 126, 182502 (2021), eprint 2008.09696.
  * Wirth et al. (2021) R. Wirth, J. M. Yao, and H. Hergert, Phys. Rev. Lett. 127, 242502 (2021), eprint 2105.05415.
  * Belley et al. (2024) A. Belley et al., Phys. Rev. Lett. 132, 182502 (2024), eprint 2308.15634.
  * Cirigliano et al. (2019) V. Cirigliano, W. Dekens, J. De Vries, M. L. Graesser, E. Mereghetti, S. Pastore, M. Piarulli, U. Van Kolck, and R. B. Wiringa, Phys. Rev. C 100, 055504 (2019), eprint 1907.11254.
  * Cirigliano et al. (2021a) V. Cirigliano, W. Dekens, J. de Vries, M. Hoferichter, and E. Mereghetti, Phys. Rev. Lett. 126, 172002 (2021a), eprint 2012.11602.
  * Cirigliano et al. (2021b) V. Cirigliano, W. Dekens, J. de Vries, M. Hoferichter, and E. Mereghetti, JHEP 05, 289 (2021b), eprint 2102.03371.
  * Cottingham (1963) W. N. Cottingham, Annals Phys. 25, 424 (1963).
  * Yang and Zhao (2023) Y. L. Yang and P. W. Zhao (2023), eprint 2308.03356.
  * Bernard et al. (1995) V. Bernard, N. Kaiser, and U.-G. Meissner, Int. J. Mod. Phys. E 4, 193 (1995), eprint hep-ph/9501384.
  * Simkovic et al. (1999) F. Simkovic, G. Pantis, J. D. Vergados, and A. Faessler, Phys. Rev. C 60, 055502 (1999), eprint hep-ph/9905509.
  * Simkovic et al. (2009) F. Simkovic, A. Faessler, H. Muther, V. Rodin, and M. Stauf, Phys. Rev. C 79, 055501 (2009), eprint 0902.0331.
  * Hyvärinen and Suhonen (2015) J. Hyvärinen and J. Suhonen, Phys. Rev. C 91, 024613 (2015).
  * Yao et al. (2022) J. M. Yao, J. Meng, Y. F. Niu, and P. Ring, Prog. Part. Nucl. Phys. 126, 103965 (2022), eprint 2111.15543.
  * Jenkins and Manohar (1991) E. E. Jenkins and A. V. Manohar, Phys. Lett. B 255, 558 (1991).
  * Bernard et al. (1992) V. Bernard, N. Kaiser, J. Kambor, and U. G. Meissner, Nucl. Phys. B 388, 315 (1992).
  * Jones et al. (2000) M. K. Jones et al. (Jefferson Lab Hall A), Phys. Rev. Lett. 84, 1398 (2000), eprint nucl-ex/9910005.
  * Gayou et al. (2001) O. Gayou et al., Phys. Rev. C 64, 038202 (2001).
  * Gayou et al. (2002) O. Gayou et al. (Jefferson Lab Hall A), Phys. Rev. Lett. 88, 092301 (2002), eprint nucl-ex/0111010.
  * Puckett et al. (2010) A. J. R. Puckett et al., Phys. Rev. Lett. 104, 242301 (2010), eprint 1005.3419.
  * Meyer et al. (2016) A. S. Meyer, M. Betancourt, R. Gran, and R. J. Hill, Phys. Rev. D 93, 113015 (2016), eprint 1603.03048.
  * Kelly (2004) J. J. Kelly, Phys. Rev. C 70, 068202 (2004).
  * Hashamipour et al. (2019) H. Hashamipour, M. Goharipour, and S. S. Gousheh, Phys. Rev. D 100, 016001 (2019), eprint 1903.05542.
  * Punjabi et al. (2015) V. Punjabi, C. F. Perdrisat, M. K. Jones, E. J. Brash, and C. E. Carlson, Eur. Phys. J. A 51, 79 (2015), eprint 1503.01452.
  * Cates et al. (2011) G. D. Cates, C. W. de Jager, S. Riordan, and B. Wojtsekhowski, Phys. Rev. Lett. 106, 252003 (2011), eprint 1103.1808.
  * Epelbaum et al. (2020) E. Epelbaum, H. Krebs, and P. Reinert, Front. in Phys. 8, 98 (2020), eprint 1911.11875.


# GMAIR : Unsupervised Object Detection Based on Spatial Attention and
Gaussian Mixture

Weijin Zhu, Yao Shen, Linfeng Yu, Lizeth Patricia Aguirre Sanchez
Department of Computer Science
Shanghai Jiao Tong University
<EMAIL_ADDRESS>

###### Abstract

Recent studies on unsupervised object detection based on spatial attention
have achieved promising results. Models, such as AIR and SPAIR, output “what”
and “where” latent variables that represent the attributes and locations of
objects in a scene, respectively. Most of the previous studies concentrate on
the “where” localization performance; however, we claim that acquiring “what”
object attributes is also essential for representation learning. This paper
presents a framework, GMAIR, for unsupervised object detection. It
incorporates spatial attention and a Gaussian mixture in a unified deep
generative model. GMAIR can locate objects in a scene and simultaneously
cluster them without supervision. Furthermore, we analyze the “what” latent
variables and clustering process. Finally, we evaluate our model on MultiMNIST
and Fruit2D datasets and show that GMAIR achieves competitive results on
localization and clustering compared to state-of-the-art methods.

##  1 Introduction

The perception of human vision is naturally hierarchical. We can recognize
objects in a scene at a glance and classify them according their appearances,
functions, and other attributes. It is expected that an intelligent agent can
also decompose scenes to meaningful object abstraction, which is known as an
object detection task in machine learning. In the last decade, there have been
significant developments in supervised object detection tasks. However, its
unsupervised counterpart continues to be challenging.

Recently, there has been some progress in unsupervised object detection.
Attend, infer, repeat (AIR, Eslami et al. (2016)), which is a variational
autoencoder (VAE, Kingma and Welling (2013)) based method, achieved
encouraging results. Spatially invariant AIR (SPAIR, Crawford and Pineau
(2019)) replaced the recurrent network in AIR by a convolutional network that
attained better scalability and lower computational cost. SPACE (Lin et al.
(2020)), which combines spatial-attention and scene-mixture approaches,
performed better in background prediction.

Despite the recent progress in unsupervised object detection, results of
previous studies remain unsatisfactory. One of the reasons for this could be
that previous studies on unsupervised object detection were mainly
concentrated on object localization and lacked analysis and evaluation on the
“what” latent variables, which represent the attributes of objects. These
variables are essential for many tasks such as clustering, image generation,
and style transfer. Another important concern is that they do not directly
reason about the category of objects in the scene, which is beneficial to know
in many cases, unlike most of the studies on corresponding supervised tasks.

This paper presents a framework for unsupervised object detection that can
directly reason about the category and localization of objects in the scenes
and provide an intuitive way to analyze the “what” latent variables by simply
incorporating a Gaussian mixture prior assumption. In Sec. 2, we introduce the
architecture of our framework, GMAIR. We introduce related works in Sec. 3. We
analyze the “what” latent variables in Sec. 4.1. We describe our model for
image generation in Sec. 4.2. Finally, we present quantitative evaluation
results of both clustering and localization in Sec. 4.3.

Our main contributions are:

  * •

We combine spatial attention and a Gaussian mixture in a unified deep
generative model, enabling our model to cluster discovered objects.

  * •

We analyze the “what” latent variables, which are essential because they
represent the attributes of the objects.

  * •

Our method achieves competitive results on both clustering and localization
compared to state-of-the-art methods.

##  2 Gaussian Mixture Attend, Infer, Repeat

In this section, we introduce our framework, GMAIR, for unsupervised object
detection. GMAIR is a spatial-attention model with a Gaussian mixture prior
assumption for the “what” latent variables, and this enables the model to
cluster discovered objects. An overview of GMAIR is presented in Fig. 1.

Figure 1:  Architecture of GMAIR. This is a VAE-based model that consists of a
probabilistic encoder, $q_{\phi}(\mathbf{z}|\mathbf{x})$, and a probabilistic
decoder, $p_{\theta}(\mathbf{x}|\mathbf{z})$. In encoder
$q_{\phi}(\mathbf{z}|\mathbf{x})$, feature maps with dimension $H\times
W\times D$ are extracted from data $\mathbf{x}$ going through a backbone
network representing feature of $H\times W$ divided regions. They are then
fetched into three separated modules: pres-head, depth-head, and where-head,
which produce the posterior of $\mathbf{z}^{\mathsf{pres}}$,
$\mathbf{z}^{\mathsf{depth}}$, and $\mathbf{z}^{\mathsf{where}}$,
respectively. A cat-encoder module generates The posterior of
$\mathbf{z}^{\mathsf{cat}}$ with $H\times W$ input glimpses transformed by a
spatial transformer network (STN) as input, and the posterior of
$\mathbf{z}^{\mathsf{what}}$ is generated by a what-encoder module with
$H\times W$ input glimpses and $\mathbf{z}^{\mathsf{cat}}$ as input. In
decoder $p_{\theta}(\mathbf{x}|\mathbf{z})$, each $H\times W$ latent
$\mathbf{z}^{\mathsf{what}}$ is fetched into a glimpse decoder to generate
decoded glimpses rendered by the renderer to recover to the final generated
image. Finally, the priors of
$\mathbf{z}^{\mathsf{pres}},\mathbf{z}^{\mathsf{depth}},\mathbf{z}^{\mathsf{where}}$,
and $\mathbf{z}^{\mathsf{cat}}$ are fixed, whereas the prior of
$\mathbf{z}^{\mathsf{what}}$ is generated by a “what priors” module using
$\mathbf{z}^{\mathsf{cat}}$ as input.

###  2.1 Structured Object-semantic Latent Representation

We follow SPAIR to attain object abstraction latent variables (Crawford and
Pineau (2019)); the image is divided into $H\times W$ regions. Latent
variables $\mathbf{z}=(\mathbf{z}_{1},\mathbf{z}_{2},...,\mathbf{z}_{HW})$ is
a concatenation of $HW$ latent variables where $\mathbf{z}_{i}$ is the latent
variable for the $i$-th region representing the semantic feature of the object
centered in the $i$-th region. Furthermore, for each region we divide
$\mathbf{z}_{i}$ into five seperate latent variables,
$\mathbf{z}_{i}=(\mathbf{z}^{\mathsf{pres}}_{i},\mathbf{z}^{\mathsf{what}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i})$,
where
$\mathbf{z}^{\mathsf{pres}}_{i}\in\\{0,1\\},\mathbf{z}^{\mathsf{what}}_{i}\in\mathbb{R}^{A},\mathbf{z}^{\mathsf{cat}}_{i}\in\\{0,1\\}^{C},\mathbf{z}^{\mathsf{where}}_{i}\in\mathbb{R}^{4},\mathbf{z}^{\mathsf{depth}}_{i}\in\mathbb{R}$,
$A$ is the dimension of “what” latent variables and $C$ is the number of
clusters. The meaning of
$\mathbf{z}^{\mathsf{pres}},\mathbf{z}^{\mathsf{what}},\mathbf{z}^{\mathsf{where}},\mathbf{z}^{\mathsf{depth}}$
are the same as in Crawford and Pineau (2019), while
$\mathbf{z}^{\mathsf{cat}}$ are one-hot vectors for categories.

GMAIR imposes a prior on those latent variables as follow:

$p(\mathbf{z})=\prod_{i=1}^{HW}p(\mathbf{z}^{\mathsf{pres}}_{i})\left(p(\mathbf{z}^{\mathsf{cat}}_{i})p(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{z}^{\mathsf{cat}}_{i})p(\mathbf{z}^{\mathsf{where}}_{i})p(\mathbf{z}^{\mathsf{depth}}_{i})\right)^{\mathbf{z}^{\mathsf{pres}}_{i}}.$
(1)

#### Gaussian Mixture Prior Assumption

Latent variables $\mathbf{z}^{\mathsf{cat}}$ are one-hot vectors that act as
classification indicators. They obey the categorical distribution,
$\operatorname{Cat}(\boldsymbol{\pi})$, where $\boldsymbol{\pi}\in[0,1]^{C}$.
For simplicity, we assume that $\boldsymbol{\pi}_{k}=1/C$ for all $1\leq k\leq
C$.

We assume that $\mathbf{z}^{\mathsf{what}}_{i}$ conditional on
$\mathbf{z}^{\mathsf{cat}}_{i}$ obeys a Gaussian distribution. In that case,
$\mathbf{z}^{\mathsf{what}}_{i}$ obeys a Gaussian mixture model, that is,

$\displaystyle p(\mathbf{z}^{\mathsf{what}}_{i})$
$\displaystyle=\sum_{k=1}^{C}p(\mathbf{z}^{\mathsf{cat}}_{i,k}=1)p(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{z}^{\mathsf{cat}}_{i,k}=1)$
(2)
$\displaystyle=\sum_{k=1}^{C}p(\mathbf{z}^{\mathsf{cat}}_{i,k}=1)f(x;\mu_{k},\sigma^{2}_{k})$

where
$f(x;\mu,\sigma^{2})=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{(x-\mu)^{2}}{2\sigma^{2}}\right)$
is the probability density function of Gaussian distribution,
$\mu_{k},\sigma_{k}$ ($k=1..C$) are the mean and standard derivation of the
$k$-th Gaussian distribution. We let $\mu_{k}$ and $\sigma_{k}$ be learnable
parameters that are jointly trained with other parameters. During the
implementation, $\mu_{k}=\mu(\mathbf{z}^{\mathsf{cat}}_{i})$ and
$\sigma_{k}=\sigma(\mathbf{z}^{\mathsf{cat}}_{i})$ if
$\mathbf{z}^{\mathsf{cat}}_{i,k}=1$ where $\mu$ and $\sigma$ can be modeled as
linear layers. They are called “what priors” module in Figure 1.

For other latent variables, $\mathbf{z}^{\mathsf{pres}}$ are modeled using a
Bernoulli distribution, $\beta(p)$, where $p$ is the present probability.
$\mathbf{z}^{\mathsf{where}}$ and $\mathbf{z}^{\mathsf{depth}}$ are modeled
using normal distributions,
$\mathcal{N}(\mu^{\mathsf{where}}_{\mathsf{prior}},{\sigma^{\mathsf{where}}_{\mathsf{prior}}}^{2})$
and
$\mathcal{N}(\mu^{\mathsf{depth}}_{\mathsf{prior}},{\sigma^{\mathsf{depth}}_{\mathsf{prior}}}^{2})$,
respectively. All priors of latent variables are listed in Table 1.

Table 1: Priors of latent variables Latent Variables | Priors
---|---
$\mathbf{z}^{\mathsf{pres}}$ | $\beta(p)$
$\mathbf{z}^{\mathsf{what}}$ | $\mathcal{N}(\mu(\mathbf{z}^{\mathsf{cat}}),{\sigma(\mathbf{z}^{\mathsf{cat}}_{i})}^{2})$
$\mathbf{z}^{\mathsf{cat}}$ | $\operatorname{Cat}(\boldsymbol{\pi})$
$\mathbf{z}^{\mathsf{where}}$ | $\mathcal{N}(\mu^{\mathsf{where}}_{\mathsf{prior}},{\sigma^{\mathsf{where}}_{\mathsf{prior}}}^{2})$
$\mathbf{z}^{\mathsf{depth}}$ | $\mathcal{N}(\mu^{\mathsf{depth}}_{\mathsf{prior}},{\sigma^{\mathsf{depth}}_{\mathsf{prior}}}^{2})$

###  2.2 Inference and Generation Model

#### Inference Model $q_{\phi}(\mathbf{z}|\mathbf{x})$

In the inference model, latent variables conditional on data $\mathbf{x}$ are
modeled by Eqn. 3.

$q(\mathbf{z}|\mathbf{x})=\prod_{i=1}^{HW}q(\mathbf{z}^{\mathsf{pres}}_{i}|\mathbf{x})\left(q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})q(\mathbf{z}^{\mathsf{depth}}_{i}|\mathbf{x})q(\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i})q(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i})\right)^{\mathbf{z}^{\mathsf{pres}}_{i}}.$
(3)

During implementation, feature maps with dimension $H\times W\times D$ are
extracted from a backbone network using data $x$ as input, where $D$ is the
number of channels of feature maps. Further, the posteriors of
$\mathbf{z}^{\mathsf{pres}},\mathbf{z}^{\mathsf{where}}$, and
$\mathbf{z}^{\mathsf{depth}}$ are reasoned by pres-head, where-head, and
depth-head, respectively. Input images are cropped into $H\times W$ glimpses
by a spatial transformer network, and each of these is transferred to the cat-
encoder module to generate posteriors of $\mathbf{z}^{\mathsf{cat}}$.
Subsequently, we use the concatenation of the $i$-th glimpse and
$\mathbf{z}^{\mathsf{cat}}_{i}$ $(1\leq i\leq HW)$ as the input of the what-
encoder to generate posteriors of $\mathbf{z}^{\mathsf{what}}$.

#### Generation Model $p_{\theta}(\mathbf{x}|\mathbf{z})$

In the generation model, each $\mathbf{z}^{\mathsf{what}}_{i}$ $(1\leq i\leq
HW)$ is changed back into a glimpse by using a glimpse decoder. Then, a
renderer combines $HW$ glimpses to generate $\hat{\mathbf{x}}$. We use the
same render algorithm as in previous studies (Eslami et al. (2016), Crawford
and Pineau (2019)).

###  2.3 The Loss Functions

#### Evidence Lower Bound

In general, we learn parameters of VAE jointly by maximizing the evidence
lower bound (ELBO), which can be formulated as:

$\displaystyle ELBO$
$\displaystyle=\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}\left[\log\left(\frac{p(\mathbf{x},\mathbf{z})}{q(\mathbf{z}|\mathbf{x})}\right)\right]$
(4)
$\displaystyle=\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}\left[\log\left(p(\mathbf{x}|\mathbf{z})\right)\right]-\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}\left[\log\left(\frac{q(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})}\right)\right]$

where, the first term is called the reconstruction term denoted by
$-L_{\mathsf{recon}}$ and the second term, the regularization term. The
regularization term can be further decomposed into five terms by substituting
Eqn. 1 and Eqn. 3 into Eqn. 4, and each of the five terms corresponding to the
Kullback–Leibler divergence (or its expectation) between a type of latent
variables and its prior:

$\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}\left[\log\left(\frac{q(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})}\right)\right]=L_{\mathsf{pres}}+L_{\mathsf{where}}+L_{\mathsf{depth}}+L_{\mathsf{cat}}+L_{\mathsf{what}}.$
(5)

The terms in Eqn. 5 are:

$\displaystyle L_{\mathsf{pres}}\ \ =$
$\displaystyle\sum_{i=1}^{HW}KL\left(q(\mathbf{z}^{\mathsf{pres}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{pres}}_{i})\right)$
(6) $\displaystyle L_{\mathsf{where}}=$
$\displaystyle\sum_{i=1}^{HW}q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})KL\left(q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{where}}_{i})\right)$
(7) $\displaystyle L_{\mathsf{depth}}=$
$\displaystyle\sum_{i=1}^{HW}q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})KL\left(q(\mathbf{z}^{\mathsf{depth}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{depth}}_{i})\right)$
(8) $\displaystyle L_{\mathsf{cat}}\ \ \ =$
$\displaystyle\sum_{i=1}^{HW}q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})}\left[KL\left(q(\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i})||p(\mathbf{z}^{\mathsf{cat}}_{i})\right)\right]$
(9) $\displaystyle L_{\mathsf{what}}\ =$
$\displaystyle\sum_{i=1}^{HW}q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x})}\left[KL\left(q(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i})||p(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{z}^{\mathsf{cat}}_{i})\right)\right].$
(10)

A complete derivation is given in Appendix A.

#### Overlap Loss

During actual implementation, we find that penalizing on overlaps of objects
sometimes helps. Therefore, we introduce an auxiliary loss called overlap
loss. First, we calculate $HW$ images with size $3\times
H^{\mathsf{img}}\times W^{\mathsf{img}}$, where $H^{\mathsf{img}}$ and
$W^{\mathsf{img}}$ are respectively the height and width of the input image,
transformed by $HW$ decoded glimpses by a spatial transformer network. The
overlap loss is then calculated as the average of the sum subtract by the
maximum for each $H^{\mathsf{img}}\times W^{\mathsf{img}}$ pixels.

This loss, inspired by the boundary loss in SPACE (Lin et al. (2020)), is
utilized to penalize if the model tries to split a large object into multiple
smaller ones. However, we achieve this by using a different calculation method
that incurs a lower computational cost.

#### Total Loss

The total loss is:

$L=\sum_{x\in S}\alpha_{x}L_{x}$ (11)

where,
$S=\left\\{\mathsf{recon},\mathsf{overlap},\mathsf{pres},\mathsf{where},\mathsf{depth},\mathsf{cat},\mathsf{what}\right\\}$,
and $\alpha_{:}$ are the coefficients of the corresponding loss terms.

##  3 Related Works

Several studies on unsupervised object detection have been conducted,
including spatial-attention methods such as AIR (Eslami et al. (2016)), SPAIR
(Crawford and Pineau (2019)), and SPACE (Lin et al. (2020)), and scene-mixture
methods such as MONet (Burgess et al. (2019)), IODINE (Greff et al. (2019)),
and GENESIS (Engelcke et al. (2019)). Most of them including our work are
based on a VAE (Kingma and Welling (2013)).

The AIR (Eslami et al. (2016)) framework uses a VAE-based hierarchical
probabilistic model marking a milestone in unsupervised scene understanding.
In AIR, latent variables are structured into groups of latent variables
$\mathbf{z}_{1:N}$, for $N$ discovered objects, each of which consists of
“what,” “where,” and “presence” variables. A recurrent neural network is used
in the inference model to produce $\mathbf{z}_{1:N}$, and there is a decoder
network for decoding the “what” variables of each object in the generation
model. A spatial transformer network (Jaderberg et al. (2015)) is used for
rendering.

Because AIR attends one object at a time, it does not scale well to scenes
that contain many objects. SPAIR (Crawford and Pineau (2019)) attempted to
address this issue by replacing the recurrent network with a convolutional
network that follows a spatially invariant assumption. Similar to YOLO (Redmon
et al. (2016)), in SPAIR, the locations of objects are specified relative to
local grid cells.

Scene-mixture models such as MONet (Burgess et al. (2019)), IODINE (Greff et
al. (2019)), and GENESIS (Engelcke et al. (2019)) perform segmentation instead
of explicitly finding the $\mathbf{z}^{\mathsf{where}}$ location of objects.
SPACE (Lin et al. (2020)) employs a combination of both methods. It consists
of a spatial-attention model for the foreground and a scene-mixture model for
the background.

In the area of deep unsupervised clustering, recent methods include AAE
(Makhzani et al. (2015)), GMVAE (Dilokthanakul et al. (2016)), IIC (Ji et al.
(2019)). AAE combines the ideas of generative adversarial networks and
variational inference. GMVAE uses a Gaussian mixture model as a prior
distribution. In IIC, objects are clustered by maximizing mutual information
of pairs of images. All of them show promising results on unsupervised
clustering.

GMAIR incorporates a Gaussian mixture model for clustering, similar to the
GMVAE framework111We also refer to a blog post
(http://ruishu.io/2016/12/25/gmvae/) published by Rui Shu.. It worth noting
that our attempt may simply be a choice amongst many given options. Unless
previous research, our main contribution is to show the feasibility of
performing clustering and localization simultaneously. Moreover, our method
provides a simple and intuitive way to analyze the mechanics of the detection
process.

##  4 Models and Experiments

The experiments were divided into three parts: a) the analysis of “what”
representation and clustering along with the iterations, b) image generation,
and c) quantitative evaluation of the models.

We evaluate the models on two datasets :

  * •

MultiMNIST: A dataset generated by placing 1–10 small images randomly chosen
from MNIST (a standard handwritten digits dataset, (LeCun (1998))) to random
positions on $128\times 128$ empty images.

  * •

Fruit2D: A dataset collected from a real-world game. In the scenes, there are
$9$ types of fruits of various sizes. There is a large difference between both
the number and the size of small objects and large objects. The ratio of the
size of the largest type of objects to that of the smallest type of objects is
~$6$, and there are ~$31$ times objects in the smallest size than in the
largest size. These settings make it difficult to perform localization and
clustering.

In the experiments, we compared GMAIR to two models, SPAIR and SPACE, both of
which achieve state-of-the-art in unsupervised object detection in
localization performance. Separated Gaussian mixture models are applied to the
“what” latent variables generated by the compared models to obtain the
clustering results. We set the number of clusters $C=10$ and Monte Carlo
samples $M=1$ except as otherwise defined for all experiments. We present the
details of models in Appendix B.

It is worth mentioning that the model sometimes successfully locates an object
and encloses it with a large box. In that case, IoU between the ground truth
and the predicted one will be small, and therefore, will not count to be a
correct bounding box when calculating AP. We fix this issue by removing the
empty area in generated glimpses to obtain the real size of predicted boxes.

###  4.1 “What” Representation and Cluster Analysis

We conducted the experiments using the MultiMNIST dataset. We ran GMAIR for
440k iterations and observed the change in the values of the average precision
(AP) of bounding boxes, accuracy (ACC), and normalized mutual information
(NMI) of clustering until 100k iterations. We also visualized the “what”
latent variables in the latent space during the process, as shown in Fig. 2.
Although all values continued to increase even after 100k iterations, the
visualization results were similar to those at the 100k iteration. For
integrity, we reserved the results from 100k to 440k iterations in Appendix D.
Details of calculating the AP, ACC, and NMI are discussed in Appendix C.

The results showed that at an early stage (~10k iterations) of training,
models can already locate objects well with AP $>0.9$ (Fig. 2(a)). At the same
time, $\mathbf{z}^{\mathsf{what}}$, representations of objects were still
evolving, and the results of clustering (in Fig. 2(b)) was not desirable
((ACC, NMI) was ($0.24,0.15$)); the digits were a blur in Fig. 2(f). After 50k
and 100k iterations of training, the clustering effect of
$\mathbf{z}^{\mathsf{what}}$ was increasingly apparent, and the digits were
clearer (Fig. 2(g), 2(h)). The clustering results ((ACC, NMI) was
($0.55,0.43$) at 50k, and ($0.65,0.55$) at 100k iterations) were improved
(Fig. 2(c), 2(d)).

It should be noted that even if the clustering effect of
$\mathbf{z}^{\mathsf{what}}$ is sufficiently enough, the model may fail to
locate the centers of clusters (for example, the large cluster in light red in
Fig. 2(d)), leading to poor clustering results. In the worst case, the model
may learn to converge all $\mu_{k},\sigma_{k}(1\leq k\leq C)$ to the same
values, $\mu^{*},\sigma^{*}$, and the Gaussian mixture model may degenerate to
a single Gaussian distribution, $\mathcal{N}(\mu^{*},{\sigma^{*}}^{2})$,
resulting in a miserable clustering result. In general, we found that this
phenomenon usually occurs at the early stage of training and can be avoided by
adjusting the learning rate of relative modules and the coefficients of the
loss functions.

(a) AP(IoU=0.5), ACC and NMI during training

(b) “What” latent space, at 10k iterations

(c) “What” latent space, at 50k iterations

(d) “What” latent space, at 100k iterations

(e) Original image

(f) Generated image, at 10k iterations

(g) Generated image, at 50k iterations

(h) Generated image, at 100k iterations

Figure 2: “What” representation and cluster analysis. (a) Average precision
(AP), accuracy (ACC), normalized mutual information (NMI) during training.
(b-d) Visualized “what” latent space by t-SNE (Van der Maaten and Hinton
(2008)) at 10k, 50k, and 100k iterations, respectively. Each small dot
represents a sample of $\mathbf{z}^{\mathsf{what}}$, and different colors
represent the ground-truth categories of the corresponding objects. The large
dots are $\mu_{k}(1\leq k\leq C)$ described in Sec. 2.1, and each of these can
be seen as the center of a cluster. The closures represent results of
clustering, which are closures of the closest $n$ $\mathbf{z}^{\mathsf{what}}$
points to $\mu_{k}$ that are assigned to the $k$-th cluster (where $1\leq
k\leq C$ and we choose $n=200$). The color of $\mu_{k}(1\leq k\leq C)$ and
closures are decided by a matching algorithm such that a maximum number of
$\mathbf{z}^{\mathsf{what}}$ are correctly classified to the ground-truth
label. (e) Sample of original image. (f-h) Samples of generated image at 10k,
50k, and 100k iterations, respectively.

###  4.2 Image Generation

It is expected that $\mu_{k}(1\leq k\leq C)$ represents the average feature of
the $k$-th type of objects, and $\mathbf{z}^{\mathsf{what}}_{i}$ latent
variable can be decomposed into:

$\mathbf{z}^{\mathsf{what}}_{i}=\mathbf{z}^{\mathsf{avg}}_{i}+\mathbf{z}^{\mathsf{local}}_{i}.$
(12)

$\mathbf{z}^{\mathsf{avg}}_{i}=\mu_{k}(1\leq k\leq C)$ if the $i$-th object is
in the $k$-th category and $\mathbf{z}^{\mathsf{local}}$ represents the local
feature of the object. By altering $\mathbf{z}^{\mathsf{avg}}$ or
$\mathbf{z}^{\mathsf{local}}$, we should obtain new objects that belong to
other categories or the same category with different styles, respectively. In
the experiment, we altered $\mathbf{z}^{\mathsf{avg}}$ and
$\mathbf{z}^{\mathsf{local}}$ and observed the generated images for each
object, as shown in Fig. 3. In Fig. 3(a), objects in each cluster correspond
to a type of digit, which is exactly what we expected (except for digit 8 in
column 3). In Fig. 3(b), categories with a large number of objects are grouped
into multiple clusters, while categories with a small number are grouped into
one cluster. This is due to the significant difference in number between
various types. However, objects in a cluster come from a category in general.

The structure of GMAIR ensures its ability to control object categories,
object styles, and the positions of each object of the generated images by
altering $\mathbf{z}^{\mathsf{avg}}$, $\mathbf{z}^{\mathsf{local}}$, and
$\mathbf{z}^{\mathsf{where}}$. Examples are shown in Fig. 4.

(a) MultiMNIST

(b) Fruit2D

Figure 3: Generated objects by varying $\mathbf{z}^{\mathsf{avg}}$ and
$\mathbf{z}^{\mathsf{local}}$. The horizontal axis represents varying
$\mathbf{z}^{\mathsf{avg}}$, and the vertical axis represents varying
$\mathbf{z}^{\mathsf{local}}$, on both (a) and (b).

(a) MultiMNIST

(b) Fruit2D

Figure 4: Generated images by varying attributes and locations of objects.
Columns 1 to 5 are numbered from left to right. Column 1 shows original
images. Column 2 shows the generated images without varying
$\mathbf{z}^{\mathsf{avg}}$, $\mathbf{z}^{\mathsf{local}}$, and
$\mathbf{z}^{\mathsf{where}}$. Column 3 presents images generated by setting
all $\mathbf{z}^{\mathsf{avg}}$ to the same random $\mu_{k}(1\leq k\leq C)$.
Column 4 depicts images generated by varying $\mathbf{z}^{\mathsf{local}}$.
Column 5 shows images generated by applying a random shuffle to
$\mathbf{z}_{i}$.

This could provide a new approach for tasks such as style transfer, image
generation, and data augmentation. Note that previous methods such as AIR,
SPAIR, and its variants can also obtain similar results, but we achieve them
in finer granularity.

###  4.3 Quantitative Evaluations

We quantitatively evaluate the models in terms of the AP of bounding boxes,
ACC and NMI of the clusters, and the results are listed in Table 2. In the
first part, we summarize some results of the state-of-the-art models for
unsupervised clustering on MNIST dataset for comparison. In the second and the
third part, we compare GMAIR to the state-of-the-art models for unsupervised
object detection on MultiMNIST and Fruit2D dataset, respectively. The
clustering results of SPAIR and SPACE are obtained by Gaussian mixture models
(GMMs). Results show that GMAIR achieves competitive results on both
clustering and localization.

Table 2: Quantitative Results on Localization (AP) and Clustering (Accuracy and NMI) Model | Dataset | AP (%, IoU=0.5) | ACC (%) | NMI (%)
---|---|---|---|---
IIC | MNIST | — | $98.4\pm 0.652$ | —
AAE (C=16) | MNIST | — | $90.45\pm 2.05$ | —
AAE (C=30) | MNIST | — | $95.90\pm 1.13$ | —
GMVAE (M=1) | MNIST | — | $77.78\pm 5.75$ | —
GMVAE (M=10) | MNIST | — | $82.31\pm 3.75$ | —
GMAIR | MultiMNIST | $97.3\pm 0.10$ | $80.4\pm 0.48$ | $75.5\pm 0.66$
SPAIR + GMM | MultiMNIST | $90.3$ | $59.4\pm 1.50$ | $56.3\pm 1.41$
SPACE + GMM | MultiMNIST | $96.7$ | $68.8\pm 3.43$ | $65.8\pm 2.85$
GMAIR | Fruit2D | $84.9\pm 1.56$ | $90.9\pm 0.32$ | $85.7\pm 1.25$
SPAIR + GMM | Fruit2D | $83.3$ | $88.1\pm 0.70$ | $78.4\pm 0.51$
SPACE + GMM | Fruit2D | $93.8$ | $95.0\pm 1.99$ | $87.0\pm 2.20$

##  5 Conclusion

We introduce GMAIR, which combines spatial attention and a Gaussian mixture,
such that it can locate and cluster unseen objects simultaneously. We analyze
the “what” latent variables and clustering process, provide examples of GMAIR
application to the task of image generation, and evaluate GMAIR quantitatively
compared with SPAIR and SPACE.

## Acknowledgments and Disclosure of Funding

This work was partially supported by the Research and Development Projects of
Applied Technology of Inner Mongolia Autonomous Region, China under Grant No.
201802005, the Key Program of the National Natural Science Foundation of China
under Grant No. 61932014, and Pudong New Area Science & Technology Development
Fundation under Grant No. PKX2019-R02. Yao Shen is the corresponding author.

## References

  * Burgess et al. (2019) C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Charakorn et al. (2020) R. Charakorn, Y. Thawornwattana, S. Itthipuripat, N. Pawlowski, P. Manoonpong, and N. Dilokthanakul.  An explicit local and global representation disentanglement framework with applications in deep clustering and unsupervised object detection.  _arXiv preprint arXiv:2001.08957_ , 2020.
  * Crawford and Pineau (2019) E. Crawford and J. Pineau.  Spatially invariant unsupervised object detection with convolutional neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3412–3420, 2019.
  * Crawford and Pineau (2020) E. Crawford and J. Pineau.  Exploiting spatial invariance for scalable unsupervised object tracking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 3684–3692, 2020.
  * Dilokthanakul et al. (2016) N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan.  Deep unsupervised clustering with gaussian mixture variational autoencoders.  _arXiv preprint arXiv:1611.02648_ , 2016.
  * Duan et al. (2019) K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian.  Centernet: Keypoint triplets for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6569–6578, 2019.
  * Engelcke et al. (2019) M. Engelcke, A. R. Kosiorek, O. P. Jones, and I. Posner.  Genesis: Generative scene inference and sampling with object-centric latent representations.  _arXiv preprint arXiv:1907.13052_ , 2019.
  * Eslami et al. (2016) S. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hinton.  Attend, infer, repeat: Fast scene understanding with generative models.  _arXiv preprint arXiv:1603.08575_ , 2016.
  * Everingham et al. (2010) M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * Greff et al. (2017) K. Greff, S. Van Steenkiste, and J. Schmidhuber.  Neural expectation maximization.  _arXiv preprint arXiv:1708.03498_ , 2017.
  * Greff et al. (2019) K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner.  Multi-object representation learning with iterative variational inference.  In _International Conference on Machine Learning_ , pages 2424–2433. PMLR, 2019.
  * Gu et al. (2021) C. Gu, H. Xie, X. Lu, and C. Zhang.  Cgmvae: Coupling gmm prior and gmm estimator for unsupervised clustering and disentanglement.  _IEEE Access_ , 2021.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Identity mappings in deep residual networks.  In _European conference on computer vision_ , pages 630–645. Springer, 2016.
  * Higgins et al. (2016) I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  2016\.
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Jaderberg et al. (2015) M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu.  Spatial transformer networks.  _arXiv preprint arXiv:1506.02025_ , 2015.
  * Jang et al. (2016) E. Jang, S. Gu, and B. Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Ji et al. (2019) X. Ji, J. F. Henriques, and A. Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9865–9874, 2019.
  * Kingma and Welling (2013) D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kosiorek et al. (2018) A. R. Kosiorek, H. Kim, I. Posner, and Y. W. Teh.  Sequential attend, infer, repeat: Generative modelling of moving objects.  _arXiv preprint arXiv:1806.01794_ , 2018.
  * LeCun (1998) Y. LeCun.  The mnist database of handwritten digits.  _http://yann.lecun.com/exdb/mnist/_ , 1998.
  * LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lin et al. (2020) Z. Lin, Y.-F. Wu, S. V. Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  _arXiv preprint arXiv:2001.02407_ , 2020.
  * Liu et al. (2016) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , pages 21–37. Springer, 2016.
  * Loper and Black (2014) M. M. Loper and M. J. Black.  Opendr: An approximate differentiable renderer.  In _European Conference on Computer Vision_ , pages 154–169. Springer, 2014.
  * Makhzani et al. (2015) A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey.  Adversarial autoencoders.  _arXiv preprint arXiv:1511.05644_ , 2015.
  * Redmon et al. (2016) J. Redmon, S. Divvala, R. Girshick, and A. Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Springenberg (2015) J. T. Springenberg.  Unsupervised and semi-supervised learning with categorical generative adversarial networks.  _arXiv preprint arXiv:1511.06390_ , 2015.
  * Stelzner et al. (2019) K. Stelzner, R. Peharz, and K. Kersting.  Faster attend-infer-repeat with tractable probabilistic models.  In _International Conference on Machine Learning_ , pages 5966–5975. PMLR, 2019.
  * Sun (2019) S.-H. Sun.  Multi-digit mnist for few-shot learning, 2019.  URL https://github.com/shaohua0116/MultiDigitMNIST.
  * Van der Maaten and Hinton (2008) L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Wang et al. (2019) D. Wang, M. Jamnik, and P. Lio.  Unsupervised and interpretable scene discovery with discrete-attend-infer-repeat.  _arXiv preprint arXiv:1903.06581_ , 2019.
  * Wu and He (2018) Y. Wu and K. He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Zhu et al. (2020) Q. Zhu, J. Su, W. Bi, X. Liu, X. Ma, X. Li, and D. Wu.  A batch normalized inference network keeps the kl vanishing away.  _arXiv preprint arXiv:2004.12585_ , 2020.

##  Appendix A Derivation of The KL Terms

In this section, we derive the KL terms in Eqn. 5. By assumption of
$q(\mathbf{z}|\mathbf{x})$ and $p(\mathbf{z})$ (Eqn. 3 and Eqn. 1), we have:

$\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}\left[\log(\frac{q(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})})\right]=\sum_{i=1}^{HW}\mathbb{E}_{q(\mathbf{z}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}_{i}|\mathbf{x})}{p(\mathbf{z}_{i})}\right).$
(13)

The term
$\mathbb{E}_{q(\mathbf{z}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}_{i}|\mathbf{x})}{p(\mathbf{z}_{i})}\right)$
can further be expanded as follow:

$\displaystyle\mathbb{E}_{q(\mathbf{z}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}_{i}|\mathbf{x})}{p(\mathbf{z}_{i})}\right)$
$\displaystyle=$ $\displaystyle
q(\mathbf{z}^{\mathsf{pres}}_{i}=0|\mathbf{x})\log\left(\frac{q(\mathbf{z}^{\mathsf{pres}}_{i}=0|\mathbf{x})}{p(\mathbf{z}^{\mathsf{pres}}_{i}=0)}\right)+q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})\left(\log\left(\frac{q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})}{p(\mathbf{z}^{\mathsf{pres}}_{i}=1)}\right)\right.$
$\displaystyle\left.+\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}\left[\log\left(\frac{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}{p(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i})}\right)\right]\right)$
(14) $\displaystyle=$ $\displaystyle
KL\left(q(\mathbf{z}^{\mathsf{pres}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{pres}}_{i})\right)$
$\displaystyle+q(\mathbf{z}^{\mathsf{pres}}_{i}=1|\mathbf{x})\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}\left[\log\left(\frac{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}{p(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i})}\right)\right].$

Continue to expand Eqn. A:

$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}\left[\log\left(\frac{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x})}{p(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{depth}}_{i},\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{what}}_{i})}\right)\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})}{p(\mathbf{z}^{\mathsf{where}}_{i})}\right)$
$\displaystyle+$
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{depth}}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}^{\mathsf{depth}}_{i}|\mathbf{x})}{p(\mathbf{z}^{\mathsf{depth}}_{i})}\right)$
(15) $\displaystyle+$
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{cat}}_{i},\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}})}{p(\mathbf{z}^{\mathsf{cat}}_{i})}\right)$
$\displaystyle+$
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{what}}_{i},\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x})}\log\left(\frac{q(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}},\mathbf{z}^{\mathsf{cat}}_{i})}{p(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{z}^{\mathsf{cat}}_{i})}\right).$

By the definition of Kullback–Leibler divergence, the four terms in the RHS of
Eqn. A are indeed

$\displaystyle
KL\left(q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{where}}_{i})\right),$
$\displaystyle
KL\left(q(\mathbf{z}^{\mathsf{depth}}_{i}|\mathbf{x})||p(\mathbf{z}^{\mathsf{depth}}_{i})\right),$
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i}|\mathbf{x})}\left[KL\left(q(\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i})||p(\mathbf{z}^{\mathsf{cat}}_{i})\right)\right],$
and
$\displaystyle\mathbb{E}_{q(\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i}|\mathbf{x})}\left[KL\left(q(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{x},\mathbf{z}^{\mathsf{where}}_{i},\mathbf{z}^{\mathsf{cat}}_{i})||p(\mathbf{z}^{\mathsf{what}}_{i}|\mathbf{z}^{\mathsf{cat}}_{i})\right)\right],$

respectively. Therefore, we complete the proof of Eqn. 5.

During the implementation, we model discrete variables
$\mathbf{z}^{\mathsf{pres}}$ and $\mathbf{z}^{\mathsf{cat}}$ using the Gumbel-
Softmax approximation (Jang et al. (2016)). Therefore, all variables are
differentiable using the reparameterization trick.

##  Appendix B Implementation Details

Our code is available at https://github.com/EmoFuncs/GMAIR-pytorch.

###  B.1 Models

Here, we describe the architecture of each module of GMAIR, as shown in Fig.
1. The backbone is a ResNet18 (He et al. (2016)) network with two
deconvolution layers replacing the fully connected layer, as shown in Table 3.
Pres-head, depth-head, and where-head are convolutional networks that are only
different from the number of output channels, as shown in Table 4. What-
encoder and cat-encoder are multiple layer networks, as shown in Table 5.
Finally, the glimpse decoder is a deconvolutional network, as shown in Table
6.

For other models, we make use of code from
https://github.com/yonkshi/SPAIR_pytorch for SPAIR, and
https://github.com/zhixuan-lin/SPACE for SPACE. We utilize most of the default
configuration for both models, and only change $A$ (the dimension of
$\mathbf{z}^{\mathsf{what}}_{i}$) to $256$ for comparison, the size of the
base bounding box to $72\times 72$ for large objects.

###  B.2 Training and Hyperparameters

The base set of hyperparameters for GMAIR is given in Table 7. The value $p$
(the prior on $\mathbf{z}^{\mathsf{pres}}$) drops gradually from $1$ to the
final value $6\mbox{e-}6$, and the value $\alpha_{\mathsf{overlap}}$ drops
from $2$ to $0$ in the early stage of training for stability. The learning
rate is in the range of $[5\mbox{e-}5,1\mbox{e-}4]$.

###  B.3 Testing

During testing phase, in order to obtain deterministic results, we use the
value with the largest probability (density) for latent variables
$\mathbf{z}$, instead of sampling them from the distributions. To be specific,
we use $\boldsymbol{\pi},\mu^{\mathsf{depth}},p,\mu^{\mathsf{what}},\mbox{ and
}\mu^{\mathsf{where}}$ for
$\mathbf{z}^{\mathsf{cat}},\mathbf{z}^{\mathsf{depth}},\mathbf{z}^{\mathsf{pres}},\mathbf{z}^{\mathsf{what}}$
and $\mathbf{z}^{\mathsf{where}}$, respectively.

##  Appendix C Calculation of AP, ACC and NMI

The value of AP is calculated at threshold $\mbox{IoU}=0.5$ by using the
calculation method from the VOC (Everingham et al. (2010)). Before calculating
the ACC and NMI of clusters, we filter the incorrect bounding boxes. A
predicted box $PB$ is correct iff there is a ground-truth box $GB$ such that
$\operatorname{IoU}(PB,GB)>0.5$, and the class of a correct predicted box $PB$
is assigned to the class of the ground-truth box $GB$ such that
$\operatorname{IoU}(PB,GB)$ is maximized. After filtering, all correct
predicted boxes are used for the calculation of ACC and NMI. Note that we
still have many ways to assign each predicted category to a real category when
calculating the value of ACC. In all of the ways, we select the one such that
ACC is maximized, following Dilokthanakul et al. (2016). Formulas are shown in
Eqn. 16 and Eqn. 17 for the calculation of ACC and NMI:

$\mbox{ACC}=\frac{\sum_{k=1}^{C}\max_{1\leq j\leq
C^{\prime}}|\left\\{i|G_{i}=j,P_{i}=k\right\\}|}{|P|}$ (16)

$\mbox{NMI}=\frac{2I(G,P)}{H(G)+H(P)}$ (17)

where $G$ and $P$ are respectively the ground-truth categories and predicted
categories for all correct boxes, $C$ and $C^{\prime}$ are the number of
clusters and real classes, $H(\cdot)$ and $I(\cdot,\cdot)$ are the entropy and
mutual information function, respectively.

Table 3: Architecture of Backbone Layer | Type | Size | Act./Norm. | Output Size
---|---|---|---|---
resnet | ResNet18 (w/o fc) |  |  | $512\times 4\times 4$
deconv layer 1 | Deconv | $128$ | ReLU/BN | $128\times 8\times 8$
deconv layer 2 | Deconv | $64$ | ReLU/BN | $64\times 16\times 16$
Table 4: Architectures of Pres/Depth/Where-Head Layer | Type | Size | Act./Norm. | Output Size
---|---|---|---|---
Input |  |  |  | $64\times 16\times 16$
Hidden | Conv | $\left[3\times 3,128\right]\times 3$ | ReLU | $128\times 16\times 16$
Output | Conv | $1\times 1,1/1/4$ |  | $1/1/4\times 16\times 16$
Table 5: Architectures of What/Cat-Encoder Layer | Type | Size | Act./Norm. | Output Size
---|---|---|---|---
Input | Flatten |  |  | $(3\times 32\times 32=)3072$
Layer 1 | Linear | $3072\times 128$ | ReLU | $128$
Layer 2 | Linear | $128\times 256$ | ReLU | $256$
Layer 3 | Linear | $256\times 512$ | ReLU | $512$
Output | Linear | $512\times A/C$ |  | $A/C$
Table 6: Architecture of Glimpse-Decoder Layer | Type | Size | Act./Norm. | Output Size
---|---|---|---|---
Input | Linear | $A\times 256$ | ReLU | $256\times 1\times 1$
Layer 1 | Deconv | $128$ | ReLU/GN(8) | $128\times 2\times 2$
Layer 2 | Deconv | $128$ | ReLU/GN(8) | $128\times 4\times 4$
Layer 3 | Deconv | $64$ | ReLU/GN(8) | $64\times 8\times 8$
Layer 4 | Deconv | $32$ | ReLU/GN(8) | $32\times 16\times 16$
Conv | Conv | $3\times 3,32$ | ReLU/GN(8) | $32\times 16\times 16$
Layer 5 | DeConv | $16$ | ReLU/GN(4) | $16\times 32\times 32$
Output | Conv | $3\times 3,3$ |  | $3\times 32\times 32$
Table 7: Base Hyperparameters Description | Variable | Value
---|---|---
Base bbox size | $(a_{h},a_{w})$ | $(72,72)$
Batch size |  | $16$
Dim. of $\mathbf{z}^{\mathsf{what}}_{i}$ | $A$ | $256$
Dim. of $\mathbf{z}^{\mathsf{cat}}_{i}$ | $C$ | $10$
Glimpse size | $(H_{obj},W_{obj})$ | $(32,32)$
Learning rate |  | $[5\mbox{e-}5,1\mbox{e-}4]$
Loss Coef. of $L_{\mathsf{cat}}$ | $\alpha_{\mathsf{cat}}$ | 1
Loss Coef. of $L_{\mathsf{overlap}}$ | $\alpha_{\mathsf{overlap}}$ | $2\rightarrow 0$
Loss Coef. of $L_{\mathsf{depth}}$ | $\alpha_{\mathsf{depth}}$ | 1
Loss Coef. of $L_{\mathsf{pres}}$ | $\alpha_{\mathsf{pres}}$ | 1
Loss Coef. of $L_{\mathsf{recon}}$ | $\alpha_{\mathsf{recon}}$ | 8,16
Loss Coef. of $L_{\mathsf{what}}$ | $\alpha_{\mathsf{recon}}$ | 1
Loss Coef. of $L_{\mathsf{where}}$ | $\alpha_{\mathsf{recon}}$ | 1
Prior on $\mathbf{z}^{\mathsf{cat}}$ | $\boldsymbol{\pi}$ | $(1/C,...,1/C)$
Prior on $\mathbf{z}^{\mathsf{depth}}$ | $(\mu^{\mathsf{depth}}_{\mathsf{prior}},{\sigma^{\mathsf{depth}}_{\mathsf{prior}}})$ | $(0,1)$
Prior on $\mathbf{z}^{\mathsf{pres}}$ | $p$ | $1\rightarrow 6\mbox{e-}6$
Prior on $\mathbf{z}^{\mathsf{where}}$ | $(\mu^{\mathsf{where}}_{\mathsf{prior}},{\sigma^{\mathsf{where}}_{\mathsf{prior}}})$ | $(0,1)$
Prior on $\mathbf{z}^{\mathsf{what}}$ | $(\mu^{\mathsf{what}}_{\mathsf{prior}},{\sigma^{\mathsf{what}}_{\mathsf{prior}}})$ | $(0,1)$

##  Appendix D Additional Experiment Results

The graphs of “what” representation after 100k iterations are shown in Fig. 5.

(a) AP(IoU=0.5), ACC and NMI during training

(b) “What” latent space, at 220k iterations

(c) “What” latent space, at 330k iterations

(d) “What” latent space, at 440k iterations

(e) Original image

(f) Generated image, at 220k iterations

(g) Generated image, at 330k iterations

(h) Generated image, at 440k iterations

Figure 5: “What” representation and cluster analysis after 100k iterations.
(a) Average precision (AP), accuracy (ACC), normalized mutual information
(NMI) during training. (b-d) Visualized “what” latent space by t-SNE (Van der
Maaten and Hinton (2008)) at 220k, 330k, and 440k iterations, respectively.
(e) Sample of original image. (f-h) Samples of generated image at 220k, 330k,
and 440k iterations, respectively.