#  Manipulating spatial structure of high-order quantum coherence with
entangled photons

Shuang-Yin Huang,1,2 Jing Gao,1,2 Zhi-Cheng Ren,1,2 Zi-Mo Cheng,1,2 Wen-Zheng
Zhu,1,2 Shu-Tian Xue,1,2 Yan-Chao Lou,1,2 Zhi-Feng Liu,1,2 Chao Chen,1,2
Fei Zhu,3 Li-Ping Yang,4 Xi-Lin Wang,1,2,5,6 and Hui-Tian Wang1,2,7
1National Laboratory of Solid State Microstructures and School of Physics,
Nanjing University, Nanjing 210093, China
2Collaborative Innovation Center of Advanced Microstructures, Nanjing
University, Nanjing 210093, China
3Intelligent Scientific Systems Co. Limited, Beijing 102208, China
4Center for Quantum Sciences and School of Physics, Northeast Normal
University,
Changchun 130024, China
5Hefei National Laboratory, Hefei 230088, China
6Synergetic Innovation Center of Quantum Information and Quantum Physics,
University of Science and Technology of China, Hefei 230026, China
7Collaborative Innovation Center of Extreme Optics, Shanxi University, Taiyuan
030006, China

###### Abstract

High-order quantum coherence reveals the statistical correlation of quantum
particles. Manipulation of quantum coherence of light in temporal domain
enables to produce single-photon source, which has become one of the most
important quantum resources. High-order quantum coherence in spatial domain
plays a crucial role in a variety of applications, such as quantum imaging,
holography and microscopy. However, the active control of high-order spatial
quantum coherence remains a challenging task. Here we predict theoretically
and demonstrate experimentally the first active manipulation of high-order
spatial quantum coherence by mapping the entanglement of spatially structured
photons. Our results not only enable to inject new strength into current
applications, but also provide new possibilities towards more wide
applications of high-order quantum coherence.

The high-order coherence of light reveals statistical correlation beyond the
first-order coherence indicated by intensity fringes in traditional
interference (for instance, the best known Young’s double-slit interference).
Study on high-order coherence was pioneered by Hanbury Brown and Twiss (HBT)
in a landmark interferometry experiment in 1956 Brown1956 , in which they
observed the bunching of photons in thermal light. This discovery led Glauber
to lay the foundation of quantum optics by developing the quantum theory of
coherence Glauber1963  and found practical application in astronomy to measure
the angular diameter of stars Brown1956-2 . Since then, the HBT interferometry
has been deeply explored with various sources, such as interacting photons in
a nonlinear medium Bromberg2010  and twisted light Loaiza2016 , and has
attracted significant interest beyond photons, including electrons
(antibunching effect for fermions) Henny1999 ; Oliver1999 , atoms
Schellekens2005 ; Ottl2005 ; Jeltes2007 , molecules Rosenberg2022 , matter
wave Perrin2012  and phonons Cayla2020 .

In contrast to the thermal source, the high-order coherence of quantum light
unveils remarkable statistical behaviors among quantum particles. One of the
typical quantum sources is the single-photon source He2013 ; Ding2016 ;
Somaschi2016 ; Tomm2021 , which shows the antibunching effect in temporal
domain and has important applications in quantum information science ranging
from quantum foundation Tang2012 ; Wen2023 , quantum communications Lu2021 ,
quantum metrology Chu2017 ; Wang2020  to quantum computation and quantum
simulation Wang2019 ; Cogan2023 . The bunching and antibunching behaviors of
photons are usually described by the second-order quantum coherence in
temporal domain, which is quantified by $g^{2}(\tau)|_{\tau=0}$, where
$\tau=t_{2}-t_{1}$ indicates the arriving time difference between the two
detected photons. A perfect single-photon source exhibits a vanishing
$g^{2}(\tau)|_{\tau=0}=0$, indicating its intrinsic “single-photon” nature, as
illustrated in Fig. 1(a). The antibunching effect, characterized by
$g^{2}(\tau)|_{\tau=0}<1$, originates from the sub-Poissonian photon
statistics Short1983  and signifies a genuine quantum phenomenon with no
classical counterpart. Therefore, the control of high-order quantum coherence
in temporal domain has brought significant impact on quantum information
science and quantum optics. Consequently, the manipulation of this quantum
feature in spatial domain would be expected to introduce new quantum effects
and applications.

From the perspective of spatial domain, the existing research on the
manipulation of high-order quantum coherence was mainly focused on the uniform
spatial modes. By establishing entanglement in spatial degree of freedom
(DOF), for example, orbital angular momentum (OAM) arising from a helical
phase in the form of $\exp(jm\varphi)$ (where $\varphi$ is the azimuthal angle
in the polar coordinates and the integer index $m$ indicates the topological
charge), one can construct spatially structured entangled photons Mair2001 ;
Kovlakov2017 ; Defienne2018 ; Erhard2020 ; Forbes2021 . The quantum feature
from the nonuniform spatial structure can be utilized to produce various types
of entanglement, including two-photon Dada2011  and multi-photon Zhang2016 ;
Hiesmayr2016 ; Malik2016 ; Zhang2017 ; Erhard2018  high-dimensional
entanglement, high-angular-momenta entanglement Fickler2012 ; Hiekkamaki2021 ,
hyperentanglement in multiple DOFs Barreiro2005 ; Zhao2019 . These novel
quantum entanglements have found many important applications in various
quantum information operations and protocols, such as full Bell state
measurement Barreiro2008 , alignment-free quantum communication D'Ambrosio2012
, direct characterization of quantum dynamics Graham2013 , quantum
teleportation of multiple DOFs of a single photon Wang2015 , and direct
measurement of particle exchange phase Liu2022 . Exploring the spatial
nonuniformity in structured entangled photons has emerged as an appealing and
valuable realm. Moreover, the spatial nonuniformity in high-order quantum
coherence holds great potential for intriguing effects and significant
applications. However, on-demand manipulation of achieving high-order quantum
coherence remains challenging.

In this letter, we theoretically propose and experimentally demonstrate the
active control of the nonuniform high-order quantum coherence of spatially
structured entangled photons, as shown in Fig. 1(b). In theory, we develop an
approach that maps the entanglement of spatially structured photons to the
nonuniform high-order quantum coherence, which is applicable for both two and
more entangled photons. In experiment, we successfully verified our approach
by measuring the second-order spatial quantum coherence
$g^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$ of spatially entangled two photons
at respective positions $\mathbf{r}$ and $\mathbf{r}^{\prime}$. The key to our
experiment is to transfer the polarization entanglement of two spatially
uncorrelated photons to the spatially structured entanglement. Our results
have the potential to pave the way for new possibilities in quantum
information processing and enhance the control over photon-based systems.

Figure 1: Temporal coherence versus spatial coherence. (a) Traditionally, the
second-order temporal coherence function $g^{(2)}(\tau)$ is used to describe
the bunching and antibunching statistics of quasi-one-dimensional photon
pulses. (b) The second-order spatial coherence function
$g^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$ characterizes the quantum spatial
correlations of spatially structured entangled photons.  Figure 2:
Experimental setup of manipulating nonuniform high-order quantum coherence of
spatially structured entangled photons. Using single-mode fibers (SMFs), we
eliminate the momentum correlation of the photon pair, as depicted in the
inset. The two-photon polarization entanglement is converted into either OAM
entanglement or patterned-phase entanglement via the entanglement transfer
units (ETUs), which behaves as a polarization-dependent mode switching device.
The idler photon in path-B is filtered by a $\pi$/4-sector mask and collected
by a multimode fiber (MMF) connected to a single-photon avalanche diode
(SPAD). The electrical signal of SPAD is used to trigger the ICCD camera
realizing the coincidence imaging of the signal photon in path-A.

Manipulation of nonuniform quantum coherence.—The essence of our quantum
coherence control method lies in the conversion of global polarization
entanglement into nonuniform spatial correlation. The resulting patterned-
phase entangled two-photon states are described by a wave-packet function
(WPF) of the form Cui2023

$\Psi(\mathbf{r},\mathbf{r}^{\prime})=\frac{1}{\sqrt{2}}\eta_{A}(\mathbf{r})\eta_{B}(\mathbf{r}^{\prime})\left\\{e^{i[\Phi_{A}(\mathbf{r})-\Phi_{B}(\mathbf{r}^{\prime})]}\pm{\rm
c.c.}\right\\},$ (1)

where the normalized function $\eta_{i}(\mathbf{r})$ $i=\\{A,B\\}$ represents
the probability amplitude distribution of the paraxial photons in path-$i$. By
engineering the patterns in phases $\Phi_{A}(\mathbf{r})$ and
$\Phi_{B}(\mathbf{r})$, we can precisely control the spatial structure of
second-order coherence function of a photon pair (see Supplemental Material
for details SuppMat)

$g^{(2)}(\mathbf{r},\mathbf{r}^{\prime})=\frac{G^{(2)}(\mathbf{r},\mathbf{r}^{\prime})}{n_{A}(\mathbf{r})n_{B}(\mathbf{r}^{\prime})}=1\pm\cos[2\Phi_{A}(\mathbf{r}_{A})-2\Phi_{B}(\mathbf{r}^{\prime}_{B})].$
(2)

Here, $G^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$ represents the second-order
correlation function Glauber1963 . We note that the photon number density
distribution of each individual photon remains almost unchanged despite the
presence of the nonuniform spatial entanglement, i.e.,
$n_{A}(\mathbf{r})\approx|\eta_{A}(\mathbf{r})|^{2}$ and
$n_{B}(\mathbf{r}^{\prime})\approx|\eta_{B}(\mathbf{r}^{\prime})|^{2}$ in the
paraxial case.

Figure 3:  The contrast between photon density distribution, two-photon
correlation $G^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$, and coherence function
$g^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$ for helical phases
$\Phi_{A}(\mathbf{r})=2\varphi$ and
$\Phi_{B}(\mathbf{r}^{\prime})=2\varphi^{\prime}$. (a) The left column shows
the coincidence patterns captured by the ICCD camera at path-A triggered by
the photon in path-B with the sector mask at four different orientation angles
($\varphi^{\prime}=\pi/4,\pi/6,\pi/12,0$). The middle column displays the
corresponding unfolded patterns of the left column. The right column presents
the rescaled two-photon coincidence pattern
$G^{(2)}(\varphi,\varphi^{\prime})$. (b) The rescaled single-photon intensity
distribution directly captured by the ICCD exhibits a doughnut structure and
the corresponding unfolded pattern is displayed below. (c) The obtained
second-order coherence function $g^{(2)}(\varphi,\varphi^{\prime})$.

Experimental realization.—In our experiment, as depicted in Fig. 2, a mode-
locked laser with a repetition rate of 80 MHz and a wavelength of 390 nm pumps
a sandwich-like type-II crystal composed of a 4-mm-long BBO+HWP+BBO setup
Wang2016 . By using the degenerate and non-collinear spontaneous parametric
down-conversion (SPDC) process, we generate the polarization-entangled photon
pairs in the form of $(|H_{A}V_{B}\rangle\pm|V_{A}H_{B}\rangle)/\sqrt{2}$,
where $H$ and $V$ indicate horizontally and vertically polarized states,
respectively. We utilize the single-mode fibers (SMFs) and 3-nm narrow-band
filters to eliminate the momentum and frequency correlations of the photon
pair. The obtained joint momentum spectrum exhibits a Gaussian distribution
Walborn2010 , as shown in the inset at the top-right corner, implying the
elimination of momentum correlation. We achieve the polarization entanglement
fidelity of $\sim$$0.982$ after the two photons passing through the SMFs.

We utilize the entanglement transfer units (ETUs) composed of polarization-
dependent mode switching devices to convert the polarization entanglement into
the patterned-phase entanglement. The phases $\Phi_{i}(\mathbf{r})$
($i={A,B}$) and $-\Phi_{i}(\mathbf{r})$ are applied to the $H$\- and
$V$-polarized photons, respectively, using q-plates Marrucci2006  or spatial
light modulators (SLMs). Next, we select the $H$-polarized photons described
by the WPF in Eq. (1) for coincidence, using the polarization beam splitters
(PBSs).

In the coincidence measurement, the idler photons in path-B are filtered by a
mask and collected by a multimode fiber (MMF), which is connected to a single-
photon avalanche diode (SPAD). The SPAD detector serves as a trigger for a
pixel-intensified charge-coupled device (ICCD) camera (2DSPC-GS-F), and
enables to capture the image of the spatial coincidence distribution of the
signal photons in path-A, i.e., the $G^{(2)}(\mathbf{r},\mathbf{r}^{\prime})$
function with fixed $\mathbf{r}^{\prime}$. To ensure that the detected photons
at the heralding detector and the camera originate from the same correlated
photon pair, an additional ($\sim$30 meter) SMF is used to compensate for the
electrical delay (150 ns) that occurs during the ICCD triggering process. In
experiment, we insert a $\pi/4$-sector mask (instead of a very narrow sector
mask) in path-B to obtain the stronger coincidence signal, in which the
orientation angle of the angular bisector of the sector mask is defined by
$\varphi^{\prime}_{M}$.

To demonstrate the nonuniform coherence of OAM entanglement, we apply helical
phases $\Phi_{A}(\mathbf{r})=m_{A}\varphi$ and
$\Phi_{B}(\mathbf{r}^{\prime})=m_{B}\varphi^{\prime}$ to the two photons via
q-plates. Here, $\varphi$ and $\varphi^{\prime}$ are the azimuthal angles of
the two cylindrical coordinate systems associated with the two paraxial
photons. The resulting OAM-entangled states are given by
$(\left\lvert{m_{A},-m_{B}}\right\rangle\pm\left\lvert{-m_{A},m_{B}}\right\rangle)/\sqrt{2}$.
According to our theory, the second-order correlation function will exhibit
the azimuthal nonuniformity, i.e.,
$g^{(2)}(\mathbf{r},\mathbf{r}^{\prime})=g^{(2)}(\varphi,\varphi^{\prime})=1\pm\cos[2(m_{A}\varphi-
m_{B}\varphi^{\prime})]$.

Figure 3 illustrates the experimental results for the OAM-entangled state
$(\left\lvert{2_{A},-2_{B}}\right\rangle+\left\lvert{-2_{A},2_{B}}\right\rangle)/\sqrt{2}$,
i.e., $m_{A}=m_{B}=2$. In the left column of Fig. 3(a), we present the spatial
coincidence distribution captured by the ICCD camera. Notably, the coincidence
image exhibits a $C_{4}$ symmetry for any fixed
$\varphi^{\prime}=\varphi^{\prime}_{M}$, and the patterns rotate with the
orientation angle $\varphi^{\prime}_{M}$ of the $\pi/4$-sector mask in path-B.
The middle column of Fig. 3(a) displays the corresponding unfolded patterns of
the left column MoreauSA2019 , in which each row corresponds to four different
orientation angles of $\varphi^{\prime}_{M}$. Furthermore, in the right column
of Fig. 3(a), we present the rescaled second-order correlation function
$G^{(2)}(\varphi,\varphi^{\prime})$ as a function of the angles $\varphi$ and
$\varphi^{\prime}$.

Figure 4:  The second-order coherence function
$g^{(2)}(\varphi,\varphi^{\prime})$ for different OAM entangled states
$(\left\lvert{m_{A},-m_{B}}\right\rangle\pm\left\lvert{-m_{A},m_{B}}\right\rangle)/\sqrt{2}$.
Figure 5:  Imposing arbitrary patterned-phase in the coherence function of an
entangled photon pair. (a) The $g^{(2)}$ function for sector-shaped patterned-
phases. The first row illustrates the imposed patterned-phases
$\Phi_{i}(\varphi)$ and $-\Phi_{i}(\varphi)$ ($i=\\{A,B\\}$) for $H$\- and
$V$-polarized photons, respectively. The second row presents the measured
$g^{(2)}$ functions for different scanning phase
$\Phi_{B}=\\{0,\pi/4,\pi/2\\}$ in the first three figures. Neither the signal
photon nor the idler photon exhibits any sector structure in their photon-
number density. (b) The $g^{(2)}$-function of photon pairs subjected to
arbitrary-pattern entanglement.

As predicted by our theory, the $C_{4}$-symmetric pattern does not appear in
the photon-number density distribution. Without coincidence, only the doughnut
structure of the OAM photon is observed, as shown in Fig 3(b). To extract the
$g^{(2)}$ function in our experiment, we employ the following formula

$g^{(2)}(\varphi,\varphi^{\prime})=\frac{P_{AB}(\varphi,\varphi^{\prime})}{P_{A}(\varphi)},$
(3)

where $P_{AB}$ represents the normalized probability distribution of the
path-A photons in coincidence measurement, while $P_{A}$ refers to the
normalized probability distribution of the path-A photon in the absence of
coincidence. The measured coherence function
$g^{(2)}(\varphi,\varphi^{\prime})$, as shown in Fig. 3(c), is in good
agreement with the theoretical prediction given in Eq. (2) and exhibits the
similar fringes to the $G^{(2)}(\varphi,\varphi^{\prime})$ function in panel
(a).

Our approach can be applied to cases when the signal and idler photons have
different OAM quantum numbers. To demonstrate this, we present a comparison
between experimental results and theoretical simulations for various cases in
Fig. 4. The quantity of fringes is determined only by the OAM quantum number
($m_{A}$) of the signal photons and is independent of $m_{B}$ of the idler
photons. The fringes with positive slope are observed for the state of
$(\left\lvert{m_{A},-m_{B}}\right\rangle+\left\lvert{-m_{A},m_{B}}\right\rangle)/\sqrt{2}$,
while the fringes with negative slope are observed for the state of
$(\left\lvert{m_{A},-m_{B}}\right\rangle-\left\lvert{-m_{A},m_{B}}\right\rangle)/\sqrt{2}$.
Obviously, the measured $g^{(2)}$ functions match the theoretical results very
well.

Tailoring quantum coherence.—Our approach offers the precise control over the
spatial structure of quantum coherence in photon pairs. We impose the
conjugate phases to the orthogonally polarized photons. With the aid of the
ETU for polarization to patterned-phase, we are able to tailor arbitrary
structures in the $g^{(2)}$ function of entangled photon pairs.

In Fig. 5(a), the two areas of SLMA in path-A are loaded a binary phase
($0,\pi/2$) for the $H$-polarized photons and its conjugate phase ($0,-\pi/2$)
[which is equivalent to ($0,3\pi/2$)] for the $V$-polarized photons. While the
two areas of SLMB in path-B are loaded a ternary phase ($0,\pi/4,\pi/2$) for
the $H$-polarized photons and its conjugate phase ($0,-\pi/4,-\pi/2$) [which
is equivalent to ($0,7\pi/4,3\pi/2$)] for the $V$-polarized photons,
respectively. By rotating the sector mask, we can change the phase
$\Phi_{B}|_{\varphi^{\prime}=\varphi^{\prime}_{M}=(11\pi/6,\pi/6,\pi)}=(0,\pi/4,\pi/2)$
of the idler photons in path-B, we obtained three different distributions of
the $g^{(2)}$ function as shown by the three images in the second row. The
observed three different values of the coherence function
$g^{(2)}(\varphi,\varphi^{\prime})=0,1,2$ represent the destructive, uniform
and constructive distributions according to Eq. (2). Neither the signal
photons nor the idler photons exhibits any sector structure in their photon-
number density distributions, as shown in the inset at the bottom right
corner.

Similarly, we can apply the binary phases with arbitrary patterns to the
signal photons in path-A, such as the letter “N”. By using the same procedure
as Fig. 5(a), we observe the letter “N”, its complementary pattern, and a
uniform spot in the $g^{(2)}(\varphi,\varphi^{\prime})$ function, as depicted
in Fig. 5(b).

In conclusion, we have for the first time demonstrated the active manipulation
of nonuniform high-order quantum coherence in both theory and experiment. The
manipulation is realized by mapping the entanglement of spatially structured
entangled photons to the high-order spatial quantum coherence. The on-demand
spatially structured entangled photons are generated with polarization-
entangled and spatial-uncorrelated photons through entanglement transferring
from polarization to OAM (using q-plates) or a patterned phase (using SLMs).
The experimental results clearly verified the feasibility of our approach.
High-order quantum coherence in spatial domain has played an key role in many
quantum applications ranging from to imaging Edgar2012 ; Defienne2018-2 ;
Moreau2019 ; Defienne2021 ; CuiYi2023 , holography Chrapkiewicz2016 ;
Defienne2021-2  to microscopy Tenne2019 ; Varnavski2020 ; Lubin2022 ;
Ndagano2022 . Active control of spatial high-order quantum coherence would not
only enable to tap more potential in these applications but also open new
opportunities for quantum information and quantum optics based on high-order
coherence. Furthermore, our approach can be expanded to higher-order quantum
coherence control, such as the fourth-order coherence function of four-photon
pulses.

###### Acknowledgements.

This work was supported by the National Key R&D Program of China (Grants No.
2019YFA0308700 and No. 2020YFA0309500); National Natural Science Foundation of
China (Grants No. 12234009, No.12275048 and No. 12274215); Innovation Program
for Quantum Science and Technology (Grant No. 2021ZD0301400); Program for
Innovative Talents and Entrepreneurs in Jiangsu; Natural Science Foundation of
Jiangsu Province (Grant No. BK20220759); Key R&D Program of Guangdong Province
(Grant No. 2020B0303010001).

## References

  * (1) R. H. Brown and R. Q. Twiss, Correlations between photons in two coherent beams of light, Nature (London) 177, 27 (1956).
  * (2) R. J. Glauber, The Quantum Theory of Optical Coherence, Phys. Rev. 130, 2529 (1963).
  * (3) R. H. Brown and R. Q. Twiss, Test of a new type of stellar interferometer on Sirius, Nature (London) 1787, 1046 (1956).
  * (4) Y. Bromberg, Y. Lahini, E. Small and Y. Silberberg, Hanbury Brown and Twiss interferometry with interacting photons, Nat. Photonics 4, 721 (2010).
  * (5) O. S. Magaña-Loaiza, M. Mirhosseini, R. M. Cross, S. M. H. Rafsanjani, and R. W. Boyd, Hanbury Brown and Twiss interferometry with twisted light, Sci. Adv. 2, e1501143 (2016).
  * (6) M. Henny, S. Oberholzer, C. Strunk, T. Heinzel, K. Ensslin, M. Holland, and C. Sch önenberger, The fermionic Hanbury Brown and Twiss experiment, Science 284, 296 (1999).
  * (7) W. D. Oliver, J. Kim, R. C. Liu, and Y. Yamamoto, Hanbury Brown and Twiss-Type Experiment with Electrons, Science 284, 299 (1999).
  * (8) M. Schellekens, R. Hoppeler, A. Perrin, J. V. Gomes, D. Boiron, A. Aspect, and C. I. Westbrook, Hanbury Brown Twiss effect for ultracold quantum gases, Science 310, 648 (2005).
  * (9) A. Öttl, S. Ritter, M. Köhl, and T. Esslinger, Correlations and Counting Statistics of an Atom Laser, Phys. Rev. Lett. 95, 090404 (2005).
  * (10) T. Jeltes, J. M. McNamara, W. Hogervorst, W. Vassen, V. Krachmalnicoff, M. Schellekens, A. Perrin, H. Chang, D. Boiron, A. Aspect, and C. Westbrook, Comparison of the Hanbury Brown-Twiss Effect for Bosons and Fermions, Nature (London) 445, 402 (2007).
  * (11) J. S. Rosenberg, L. Christakis, E. Guardado-Sanchez, Z. Z. Yan, and W. S. Bakr, Observation of the Hanbury Brown-Twiss effect with ultracold molecules, Nat. Phys. 18,1062 (2022).
  * (12) A. Perrin, R. Bücker, S. Manz, T. Betz, C. Koller, T. Plisson, T. Schumm, and J. Schmiedmayer, Hanbury Brown and Twiss correlations across the Bose-Einstein condensation threshold, Nat. Phys. 8, 195 (2012).
  * (13) H. Cayla, S. Butera, C. Carcy, A. Tenart, G. Hercé, M. Mancini, A. Aspect, I. Carusotto, and D. Clément, Hanbury Brown and Twiss Bunching of Phonons and of the Quantum Depletion in an Interacting Bose Gas, Phys. Rev. Lett. 125, 165301 (2020).
  * (14) Y. M. He, Y. He, Y. J. Wei, D. Wu, M. Atatüre, C. Schneider, S. Höfling, M. Kamp, C. Y. Lu, and J. W. Pan,On-demand semiconductor single-photon source with near-unity indistinguishability, Nat. Nanotechnol. 8, 213 (2013).
  * (15) X. Ding, Y. He, Z. C. Duan, N. Gregersen, M. C. Chen, S. Unsleber, S. Maier, C. Schneider, M. Kamp, S. Höfling, C. Y. Lu, and J. W. Pan, On-Demand Single Photons with High Extraction Efficiency and Near-Unity Indistinguishability from a Resonantly Driven Quantum Dot in a Micropillar, Phys. Rev. Lett. 116, 020401 (2016).
  * (16) N. Somaschi, V. Giesz, L. De Santis, J. C. Loredo, M. P. Almeida, G. Hornecker, S. L. Portalupi, S. L. Portalupi, T. Grange, C. Antn, J. Demory, C. Gmez, I. Sagnes, N. D. Lanzillotti-Kimura, A. Lematre, A. Auffeves, A. G. White, L. Lanco, and P. Senellart, Near-optimal single-photon sources in the solid state, Nat. Photonics 10, 340 (2016).
  * (17) N. Tomm, A. Javadi, N. O. Antoniadis, D. Najer, M. C. Löbl, A. R. Korsch, R. Schott, S. R. Valentin, A. D. Wieck, A. Ludwig, and R. J. Warburton, A bright and fast source of coherent single photons, Nat. Nanotechnol. 16, 399 (2021).
  * (18) J. S. Tang, Y. L. Li, X. Y. Xu, G. Y. Xiang, C. F. Li, and G. C. Guo, Realization of quantum Wheeler’s delayed-choice experiment, Nat. Photonics 6, 600 (2012).
  * (19) Y. J. Wen, Y. Wang, L. M. Tian, S. Zhang, J. Li, J. S. Du, H. Yan, and S. L. Zhu, Demonstration of the quantum principle of least action with single photons, Nat. Photonics 17, XXX (2023). https://doi.org/10.1038/s41566-023-01212-1
  * (20) C. Y. Lu and J. W. Pan, Quantum-dot single-photon sources for the quantum internet, Nat. Nanotechnol. 16, 1294 (2021).
  * (21) X. L. Chu, S. Götzinger, and V. Sandoghdar, A single molecule as a high-fidelity photon gun for producing intensity-squeezed light, Nat. Photonics 11, 58 (2017).
  * (22) H. Wang, J. Qin, S. Chen, M. C. Chen, X. You, X. Ding, Y. H. Huo, Y. Yu, C. Schneider, S. Höfling, M. Scully, C. Y. Lu, and J. W. Pan, Observation of intensity squeezing in resonance fluorescence from a solid-state device, Phys. Rev. Lett. 125, 153601 (2020).
  * (23) H. Wang, J. Qin, X. Ding, M. C. Chen, S. Chen, X. You, Y. M. He, X. Jiang, L. You, Z. Wang, C. Schneider, Jelmer J. Renema, S. Höfling, C. Y. Lu, and J. W. Pan, Boson Sampling with 20 Input Photons and a 60-Mode Interferometer in a $10^{14}$-Dimensional Hilbert Space, Phys. Rev. Lett. 123, 250503 (2019).
  * (24) D. Cogan, Z.-E. Su, O. Kenneth, and D. Gershoni, Deterministic generation of indistinguishable photons in a cluster state, Nat. Photonics 17, 324 (2023).
  * (25) R. Short and L. Mandel, Observation of Sub-Poissonian Photon Statistics, Phys. Rev. Lett. 51, 384 (1983).
  * (26) A. Mair, A. Vaziri, G. Weihs, and A. Zeilinger, Entanglement of the orbital angular momentum states of photons, Nature (London) 412, 313 (2001).
  * (27) E. V. Kovlakov, I. B. Bobrov, S. S. Straupe, and S. P. Kulik, Spatial Bell-State Generation Without Transverse Mode Subspace Postselection, Phys. Rev. Lett. 118, 030503 (2017).
  * (28) H. Defienne, M. Reichert, and J. W. Fleischer, Adaptive Quantum Optics with Spatially Entangled Photon Pairs, Phys. Rev. Lett. 121, 233601 (2018).
  * (29) M. Erhard, M. Krenn, and A. Zeilinger, Advances in high-dimensional quantum entanglement, Nat. Rev. Phys. 2, 365 (2020).
  * (30) A. Forbes, M. Oliveira, and M. R. Dennis, Structured light, Nat. Photonics 15, 253 (2021).
  * (31) A. C. Dada, J. Leach, G. S. Buller, M. J. Padgett, and E. Andersson, Experimental high-dimensional two-photon entanglement and violations of generalized Bell inequalities, Nat. Phys. 7, 677 (2011).
  * (32) Y. Zhang, F. S. Roux, T. Konrad, M. Agnew, J. Leach, and A. Forbes, Engineering two-photon high-dimensional states through quantum interference, Sci. Adv. 2, e1501165, (2016).
  * (33) B. C. Hiesmayr, M. J. A. de Dood, and W. Löffler, Observation of Four-Photon Orbital Angular Momentum Entanglement, Phys. Rev. Lett. 116, 073601 (2016).
  * (34) M. Malik, M. Erhard, M. Huber, M. Krenn, R. Fickler, and A. Zeilinger, Multi-photon entanglement in high dimensions, Nat. Photonics 10, 248 (2016).
  * (35) Y. Zhang, M. Agnew, T. Roger, F. S. Roux, T. Konrad, D. Faccio, J. Leach, and A. Forbes, Simultaneous entanglement swapping of multiple orbital angular momentum states of light, Nat. Commun. 8, 632 (2017).
  * (36) M. Erhard, M. Malik, M. Krenn, and A. Zeilinger, Experimental Greenberger-Horne-Zeilinger entanglement beyond qubits, Nat. Photonics 12, 759 (2018).
  * (37) R. Fickler, R. Lapkiewicz, W. N. Plick, M. Krenn, C. Schaeff, S. Ramelow, and A. Zeilinger, Quantum entanglement of high angular momenta, Science 338, 640 (2012).
  * (38) M. Hiekkamäki, F. Bouchard, and R. Fickler, Photonic Angular Super-Resolution Using Twisted N00N States, Phys. Rev. Lett. 127, 263601 (2021).
  * (39) J. T. Barreiro, N. K. Langford, N. A. Peters, and P. G. Kwiat, Generation of Hyperentangled Photon Pairs, Phys. Rev. Lett. 95, 260501 (2005).
  * (40) T. M. Zhao, Y. S. Ihn, and Y. H. Kim, Direct Generation of Narrow-band Hyperentangled Photons, Phys. Rev. Lett. 122, 123607 (2019).
  * (41) J. T. Barreiro, T. C. Wei and P. G. Kwiat, Beating the channel capacity limit for linear photonic superdense coding, Nat. Phys. 4, 282 (2008).
  * (42) V. D’Ambrosio, E. Nagali, S. P. Walborn, L. Aolita, S. Slussarenko, L. Marrucci, and F. Sciarrino, Complete experimental toolbox for alignment-free quantum communication, Nat. Commun. 3, 961 (2012).
  * (43) T. M. Graham, J. T. Barreiro, M. Mohseni, and P. G. Kwiat, Hyperentanglement-Enabled Direct Characterization of Quantum Dynamics, Phys. Rev. Lett. 110, 060404 (2013).
  * (44) X. L. Wang, X. D. Cai, Z. E. Su, M. C. Chen, D. Wu, L. Li, N. L. Liu, C. Y. Lu, and J. W. Pan, Quantum teleportation of multiple degrees of freedom of a single photon, Nature (London) 518, 516 (2015).
  * (45) Z. F. Liu, C. Chen, J. M. Xu, Z. M. Cheng, Z. C. Ren, B. W. Dong, Y. C. Lou, Y. X. Yang, S. T. Xue, Z. H. Liu, W. Z. Zhu, X. L. Wang, and H. T. Wang, Hong-Ou-Mandel Interference between Two Hyperentangled Photons Enables Observation of Symmetric and Antisymmetric Particle Exchange Phases, Phys. Rev. Lett. 129, 263602 (2022).
  * (46) D. Z. Cui, X. L. Wang, X. X. Yi, and L. P. Yang, Control of quantum coherence and Hong-Ou-Mandel interference of structured photon pairs, arXiv:2305.19825 (2023).
  * (47) L. P. Yang and D. Xu, Quantum theory of photonic vortices and quantum statistics of twisted photons, Phys. Rev. A 105, 023723 (2022).
  * (48) D. Z. Cui, X. X. Yi, and L. P. Yang, Quantum Imaging Exploiting Twisted Photon Pairs, Adv. Quantum Technol. 6, 2370053 (2023).
  * (49) X. L. Wang, L. K. Chen, W. Li, H. L. Huang, C. Liu, C. Chen, Y. H. Luo, Z. E. Su, D. Wu, Z. D. Li, H. Lu, Y. Hu, X. Jiang, C. Z. Peng, L. Li, N. L. Liu, Y. A. Chen, C. Y. Lu, and J. W. Pan, Experimental Ten-Photon Entanglement, Phys. Rev. Lett. 117, 210502 (2016).
  * (50) S. P. Walborn, C. Monken, S. Pádua, and P. S. Ribeiro, Spatial correlations in parametric down-conversion, Phys. Rep. 495, 87 (2010).
  * (51) L. Marrucci, C. Manzo, and D. Paparo, Optical spin-to-orbital angular momentum conversion in inhomogeneous anisotropic media, Phys. Rev. Lett. 96, 163905 (2006).
  * (52) P. A. Moreau, E. Toninelli, T. Gregory, R. S. Aspden, P. A. Morris, and M. J. Padgett, Imaging Bell-type nonlocal behavior, Sci. Adv. 5, eaaw2563 (2019).
  * (53) M. P. Edgar, D. S. Tasca, F. Izdebski, R. E. Warburton, J. Leach, M. Agnew, G. S. Buller, R. W. Boyd, and M. J. Padgett, Imaging high-dimensional spatial entanglement with a camera, Nat. Commun. 3, 984 (2012).
  * (54) H. Defienne, M. Reichert, and J. W. Fleischer, General Model of Photon-Pair Detection with an Image Sensor, Phys. Rev. Lett. 120, 203604 (2018).
  * (55) P. A. Moreau, E. Toninelli, T. Gregory, and M. J. Padgett, Imaging with quantum states of light, Nat. Rev. Phys. 1, 367 (2019).
  * (56) H. Defienne, J. Zhao, E. Charbon, and Daniele Faccio, Full-field quantum imaging with a single-photon avalanche diode camera, Phys. Rev. A 103, 042608 (2021).
  * (57) R. Chrapkiewicz, M. Jachura, K. Banaszek, and W. Wasilewski, Hologram of a single photon, Nat. Photonics 10, 576 (2016).
  * (58) H. Defienne, B. Ndagano, A. Lyons, and D. Faccio, Polarization entanglement-enabled quantum holography, Nat. Phys. 17, 591 (2021).
  * (59) R. Tenne, U. Rossman, B. Rephael, Y. Israel, A. K. Ptaszek, R. Lapkiewicz, Y. Silberberg, and D. Oron, Super-resolution enhancement by quantum image scanning microscopy, Nat. Photonics 13, 116 (2019).
  * (60) O. Varnavski and Th. Goodson, Two-photon fluorescence microscopy at extremely low excitation intensity: The power of quantum correlations, J. Am. Chem. Soc. 142, 12966 (2020).
  * (61) G. Lubin, D. Oron, U. Rossman, R. Tenne, and V. J. Yallapragada, Photon Correlations in Spectroscopy and Microscopy, ACS Photonics 9, 2891(2022).
  * (62) B. Ndagano, H. Defienne, D. Branford, Y. D. Shah, A. Lyons, N. Westerberg, E. M. Gauger, and D. Faccio, Quantum microscopy based on Hong-Ou-Mandel interference, Nat. Photonics 16, 385 (2022).


# Feature Selection using Sparse Adaptive Bottleneck Centroid-Encoder

Tomojit Ghosh<EMAIL_ADDRESS>
Department of Mathematics
Colorado State University
Fort Collins, CO 80523 ,USA Michael Kirby<EMAIL_ADDRESS>
Department of Mathematics
Colorado State University
Fort Collins, CO 80523, USA

###### Abstract

We introduce a novel nonlinear model, Sparse Adaptive Bottleneck Centroid-
Encoder (SABCE), for determining the features that discriminate between two or
more classes. The algorithm aims to extract discriminatory features in groups
while reconstructing the class centroids in the ambient space and
simultaneously use additional penalty terms in the bottleneck layer to
decrease within-class scatter and increase the separation of different class
centroids. The model has a sparsity-promoting layer (SPL) with a one-to-one
connection to the input layer. Along with the primary objective, we minimize
the $l_{2,1}$-norm of the sparse layer, which filters out unnecessary features
from input data. During training, we update class centroids by taking the
Hadamard product of the centroids and weights of the sparse layer, thus
ignoring the irrelevant features from the target. Therefore the proposed
method learns to reconstruct the critical components of class centroids rather
than the whole centroids. The algorithm is applied to various real-world data
sets, including high-dimensional biological, image, speech, and accelerometer
sensor data. We compared our method to different state-of-the-art feature
selection techniques, including supervised Concrete Autoencoders (SCAE),
Feature Selection Networks (FsNet), Stochastic Gates (STG), and LassoNet. We
empirically showed that SABCE features often produced better classification
accuracy than other methods on the sequester test sets, setting new state-of-
the-art results.

##  1 Introduction

Technological advancement has made high-dimensional data readily available.
For example, in bioinformatics, the researchers seek to understand the gene
expression level with microarray or next-generation sequencing techniques
where each point consists of over 50,000 measurements (Pease et al., 1994;
Shalon et al., 1996; Metzker, 2010; Reuter et al., 2015). The abundance of
features demands the development of feature selection algorithms to improve a
Machine Learning task, e.g., classification. Another important aspect of
feature selection is knowledge discovery from data. Which biomarkers are
important to characterize a biological process, e.g., the immune response to
infection by respiratory viruses such as influenza (O’Hara et al., 2013)?
Additional benefits of feature selection include improved visualization and
understanding of data, reducing storage requirements, and faster algorithm
training times.

Feature selection can be accomplished in various ways that can be broadly
categorized as filter, wrapper, and embedded methods. In a filter method, each
variable is ordered based on a score. After that, a threshold is used to
select the relevant features (Lazar et al., 2012). Variables are usually
ranked using correlation (Guyon and Elisseeff, 2003; Yu and Liu, 2003), and
mutual information (Vergara and Estévez, 2014; Fleuret, 2004). In contrast, a
wrapper method uses a model and determines the importance of a feature or a
group of features by the generalization performance of the predetermined model
(El Aboudi and Benhlima, 2016; Hsu et al., 2002). Since evaluating every
possible combination of features becomes an NP-hard problem, heuristics are
used to find a subset of features. Wrapper methods are computationally
intensive for larger data sets, in which case search techniques like Genetic
Algorithm (GA) (Goldberg and Holland, 1988) or Particle Swarm Optimization
(PSO) (Kennedy and Eberhart, 1995) are used. In embedded methods, feature
selection criteria are incorporated within the model, i.e., the variables are
picked during the training process (Lal et al., 2006). Iterative Feature
Removal (IFR) uses the ratio of absolute weights of a Sparse SVM model as a
criterion to extract features from the high dimensional biological data set
(O’Hara et al., 2013).

Mathematically feature selection problem can be posed as an optimization
problem on $\ell_{0}$-norm, i.e., how many predictors are required for a
machine learning task. As the minimization of $\ell_{0}$ is intractable (non-
convex and non-differentiable), $\ell_{1}$-norm is used instead, which is a
convex proxy of $\ell_{0}$ (Tibshirani, 1996). Although the $\ell_{1}$ has
been used in the feature selection task in linear (Fonti and Belitser, 2017;
Muthukrishnan and Rohini, 2016; Kim and Kim, 2004; O’Hara et al., 2013;
Chepushtanova et al., 2014) as well as in nonlinear regime (Li et al., 2016;
Scardapane et al., 2017; Li et al., 2020), it has some disadvantages as well.
For example, when multi-collinearity exists (i.e., two or more independent
features have a high correlation with one another) in data, $\ell_{1}$ selects
one of them and discards the rest, degrading the rest prediction performance
(Zou and Hastie, 2005). Although the problem can be overcome using iterative
feature removal scheme as proposed by (O’Hara et al., 2013). It has been
reported that minimizing Lasso doesn’t satisfy the Oracle property (Zou,
2006). ElasticNet (Zou and Hastie, 2005), on the other hand, overcomes some
limitations of Lasso by combining $\ell_{2}$ norm with $\ell_{1}$-norm.

This paper proposes a new embedded variable selection approach called Sparse
Adaptive Bottleneck Centroid-Encoder (SABCE) to extract features when class
labels are available. Our method modifies the Centroid-Encoder model (Ghosh et
al., 2018; Ghosh and Kirby, 2022) by incorporating two penalty terms in the
bottleneck layer to increase class separation and localization. SABCE applies
a $\ell_{2,1}$ penalty to a sparsity-promoting layer between the input and the
first hidden layer while reconstructing the class centroids. One key attribute
of SABCE is the adaptive centroids update during training, distinguishing it
from Centroid-Encoder, which has a fixed class centroid. We evaluate the
proposed model on diverse data sets and show that the features produce better
generalization than other state-of-the-art techniques.

##  2 Sparse Adaptive Bottleneck Centroid-Encoder (SABCE)

Consider a data set $X=\\{x_{i}\\}_{i=1}^{N}$ with $N$ samples and $M$ classes
where $x_{i}\in\mathbb{R}^{d}$. The classes denoted $C_{j},j=1,\dots,M$ where
the indices of the data associated with class $C_{j}$ are denoted $I_{j}$. We
define centroid of each class as $c_{j}=\frac{1}{|C_{j}|}\sum_{i\in
I_{j}}x_{i}$ where $|C_{j}|$ is the cardinality of class $C_{j}$.

###  2.1 Bottleneck Centroid-Encoder (BCE)

Given the setup mentioned above, we define Bottleneck Centroid-Encoder, which
is the starting point of our proposed algorithm. The objective function of BCE
is given below:

$\displaystyle\mathcal{L}_{bce}(\theta)=\frac{1}{2N}\sum^{M}_{j=1}\sum_{i\in
I_{j}}(\|c_{j}-f(x_{i};\theta)\|^{2}_{2}+\mu_{1}\|g(c_{j})-g(x_{i})\|^{2}_{2})+\mu_{2}\sum_{k<l}\frac{1}{1+\|g(c_{k})-g(c_{l}))\|^{2}_{2}}$
(1)

The mapping $f$ is composed of a dimension-reducing mapping $g$ (encoder)
followed by a dimension-increasing reconstruction mapping $h$ (decoder). The
first term of the objective is minimizing the square of the distance between
$f(x_{i})$ and its class centroid $c_{j}$. Therefore the aim is to map the
sample $x_{i}$ to its corresponding class centroid $c_{j}$, and the mapping
function $f$ is known as Centroid-Encoder (Ghosh and Kirby, 2022). The output
of the encoder $g$ is used as a supervised visualization tool (Ghosh and
Kirby, 2022; Ghosh et al., 2018). Centroid-Encoder calculates its cost on the
output layer; if the centroids of multiple classes are close in ambient space,
the corresponding samples will land close in the reduced space, increasing the
error rate. To remedy the situation, we add two more terms to the bottleneck
layer, i.e., at the output of the encoder $g$, which we call Bottleneck
Centroid-Encoder (BCE). The term $\|g(c_{j})-g(x_{i})\|^{2}_{2}$ will further
pull a sample $x_{i}$ towards it centroid which will improve the class
localization in reduced space. Further, to avoid the overlap of classes in the
latent bottleneck space, we introduce a term that serves to to repel centroids
there. We achieve this by maximizing the distances (equivalently the square of
the $\ell_{2}$-norm) between all class-pairs of latent centroids. We
introduced the third term to fulfill the purpose. Note, as the original
optimization is a minimization problem, we choose to minimize
$\sum_{k<l}\frac{1}{1+\|g(c_{k})-g(c_{l})\|^{2}_{2}}$ which will ultimately
increase the distance between the latent centroids of class $k$ and $l$. We
added $1$ in the denominator for numerical stability. The hyper parameters
$\mu_{1}$ and $\mu_{2}$ will control the class localization and separation in
the embedded space. We use a validation set to determine their values.

###  2.2 Sparse Adaptive Bottleneck Centroid-Encoder for Robust Feature
Selection

The Sparse Bottleneck Centroid-Encoder (SBCE) is a modification to the BCE
architecture as shown in Figure 1. The input layer is connected to the first
hidden layer via the sparsity promoting layer (SPL). Each node of the input
layer has a weighted one-to-one connection to each node of the SPL. The number
of nodes in these two layer are the same. The nodes in SPL don’t have any bias
or non-linearity. The SPL is fully connected to the first hidden layer,
therefore the weighted input from the SPL will be passed to the hidden layer
in the same way that of a standard feed forward network. During training, an
$\ell_{2,1}$ penalty, which is also known as Elastic Net (Zou and Hastie,
2005), will be applied to the weights connecting the input layer and SPL
layer. The sparsity promoting $\ell_{2,1}$ penalty will drive most of the
weights to near zero and the corresponding input nodes/features can be
discarded. Therefore, the purpose of the SPL is to select important features
from the original input. Note we only apply the $\ell_{2,1}$ penalty to the
parameters of the SPL.

Figure 1: The architecture of Bottleneck Centroid-Encoder and Sparse
Bottleneck Centroid-Encoder. Notice that Sparse Bottleneck Centroid-Encoder
employs a sparse layer between the input and the first hidden layer to promote
feature sparsity using $\ell_{2,1}$ norm.

Denote $\theta_{spl}$ to be the parameters (weights) of the SPL and $\theta$
to be the parameters of the rest of the network. The cost function of sparse
bottleneck centroid-encoder is given by

$\displaystyle\mathcal{L}_{sbce}(\theta,\theta_{spl})=\frac{1}{2N}\sum^{M}_{j=1}\sum_{i\in
I_{j}}(\|c_{j}-f(x_{i};\theta)\|^{2}_{2}+\mu_{1}\|g(c_{j})-g(x_{i})\|^{2}_{2})$
(2)
$\displaystyle+\mu_{2}\sum_{k<l}\frac{1}{1+\|g(c_{k})-g(c_{l}))\|^{2}_{2}}+\lambda_{1}\|\theta_{spl}\|_{1}+\lambda_{2}\|\theta_{spl}\|^{2}_{2}$

where $\lambda_{1},\lambda_{2}$ are the hyperparameter which control the
sparsity. A larger value of $\lambda_{1}$ will promote higher sparsity
resulting more near-zero weights in SPL.

####  2.2.1 Sparsification of the Centroids

The targets of the SBCE are the class centroids which are pre-computed from
data and labels. For high-dimensional datasets, the features are noisy,
redundant, or irrelevant (Alelyani et al., 2018); therefore, feature selection
with fixed class centroids, computed on high-dimensional ambient space, may be
impacted by the noise. We can remedy the situation by promoting sparsity in
the class centroid during training. In this approach, we start with $c_{j}$’s
computed in the ambient space; after that, we change $c_{j}$’s by multiplying
it component-wise by $\theta_{spl}$, i.e.,
$[{c_{j}}]_{t}=[{c_{j}}]\odot[{{\theta}_{spl}}]_{t-1}$, where $t$ is the
current epoch. As $\theta_{spl}$ sparsifies the input data by eliminating
redundant and noisy features, updating the centroids as shown above will
reduce noise from the targets, thus improving the discriminative power of
selected features. We call this algorithm as Sparse Adaptive Bottleneck
Centroid-Encoder (SABCE). In Equation 2, we use $[{c_{j}}]_{t}$ (instead of
$c_{j}$) to calculate the cost at each iteration. The comparison in Table 1
shows the performance advantage of SABCE over SBCE.

Models | Data set
---|---
Mice Protein | MNIST | FMNIST | GLIOMA | Prostate_GE
SBCE | $95.8$ | $92.5$ | $85.0$ | $65.6$ | $88.2$
SABCE | 99.8 | 94.0 | 85.4 | 74.2 | 90.2
Table 1: Comparison between SBCE and SABCE on five benchmarking data sets
using top 50 features. We use the same network architecture and hyper
parameters in training. We follow the same experimental set up as in Section
3.

Training Details: We implemented SLCE in PyTorch (Paszke et al., 2017) to run
on GPUs on Tesla V100 GPU machines. We trained SABCE using Adam (Kingma and
Ba, 2015) on the whole training set without minibatch. We pre-train our model
for ten epochs and then include the sparse layer (SPL) with the weights
initialized to 1. Then we train the model for another ten epochs to adjust the
weights of the SPL. After that, we did an end-to-end training applying
$\ell_{2,1}$-penalty on the SPL for 1000 epochs. Like any neural network-based
model, the hyperparameters of SABCE need to be tuned for optimum performance.
Table 2 contains the list with the range of values we used in this paper. We
used validation set to choose the optimal value. Section A of Appendix has
information on reproducibility. We will provide the code with a dataset as
supplementary material.

Hyper parameter | Range of Values
---|---
# Hidden Layers (L) | {1, 2}
# Hidden Nodes (H) | {50,100,200,250,500}
Activation Function | Hyperbolic tangent (tanh)
$\mu_{1},\mu_{2}$ | {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7}
$\lambda_{1},\lambda_{2}$ | {0.01, 0.001, 0.0001, 0.0002, 0.0004, 0.0006, 0.0008}
Table 2: Hyperparameters for Sparse Adaptive Bottleneck Centroid-Encoder.

####  2.2.2 Feature Cut-off

Figure 2: Feature selection cut off geometry.

As shown in Figure 2, the $\ell_{2,1}$-norm of the sparse layer (SPL) drives a
lot of weights to near zero. Often hard thresholding or a ratio of two
consecutive weights is used to pick the nonzero weight (O’Hara et al., 2013).

In this article, we take a different approach to select the set of
discriminatory features as shown in Figure 2. After training SBCE, we arrange
the absolute value of the weights of the sparse layer in descending order
forming a curve (the orange one). We then join the first and the last point of
the curve with a straight line (the blue dotted line). We measure the distance
of each point on the curve to the straight line. The point with the largest
distance is the position (P) of the elbow. We pick all the features whose
absolute weight is greater than that of P. Figure 2 demonstrates the approach
on GLI_85 and MNIST set. The red star indicates the position of point P (the
elbow), and the absolute weight of features on the left of P is higher than on
the right, selecting only 796 out of 22,283 features from GLI_85 and 137 out
of 784 MNIST pixels.

###  2.3 Empirical Analysis of SBCE

In this section we present a series of analyses of the proposed model to
understand its behavior.

1\. Analysis of Hyper-parameters $\mu_{1}$ and $\mu_{2}$:  The hyper-
parameters $\mu_{1}$ and $\mu_{2}$ control the class scatter and separation in
bottleneck space. We ran an experiment on the MNIST digits to understand the
effect of $\mu_{1}$ and $\mu_{2}$ on model’s performance. We put aside $20\%$
of samples from each class as a validation set and the rest of the data set is
used to train a the model for each combination of $\mu_{1}$ and $\mu_{2}$. The
validation set is used to compute the error rate using a 5-NN classifier in
the two-dimensional space. Figure 3 shows the errors for different
combinations of $\mu_{1}$ and $\mu_{2}$ in a heat map.

Figure 3: Analysis of error rate with changes to $\mu_{1}$ and $\mu_{2}$.

Observe that the error rate increases with $\mu_{2}$ when $\mu_{1}$ is zero.
The behavior is not surprising as setting $\mu_{1}$ to zero nullifies the
effect of the second term (see Equation 2), which would hold the samples
tightly around their centroid in reduced space. The gradient from the first
term will exert a pulling force to bind the samples around their centroids,
but the gradient coming from the third term will dominate the gradient of the
first term as $\mu_{2}$ increases. As an effect, the class-scatter increases
in low dimensional space resulting in misclassifications. As soon as $\mu_{1}$
increases to 0.1, the error rate decreases significantly. After that, a higher
value of $\mu_{2}$ doesn’t change the results too much. The minimum validation
error occurs for $\mu_{1}=0.6$ and $\mu_{2}=0.1$. The analysis reveals that
$\mu_{1}$ is relatively more important than $\mu_{2}$.

2\. Analysis of Feature Sparsity:

Figure 4: Sparsity analysis of SABCE on six data sets. MNIST, ISOLET, and
Human Activity have more samples than features while ALLAML, GLI_85, and
Prostate_GE have more features than data points; see Table 3 for more details.
In each case, we plotted the absolute value of weights of the sparse layer in
descending order. In all the experiments, we set $\lambda_{1},\lambda_{2},$ to
0.001. These plots suggest the model promotes sparsity, driving most of the
weights in the sparsity layer to near zero.

Here, we study sparsity promotion on six representative data sets. Three
datasets contain more samples than features: MNIST, ISOLET, and Human
Activity. These data sets are also from different domains, namely image,
speech, and accelerometer sensors. We also use three biological data sets:
ALLAML, GLI_85, and Prostate_GE, where the sample size is significantly
smaller when compared to the number of features (see Table 3). We fit our
model on the training partition of each data set and then plot the absolute
value of the weights of the sparse layer in descending order as shown in
Figure 4. As can be seen, the model promotes sparsity in each case, driving
most of the weights in the sparse layer to near zero ($10^{-4}$ to $10^{-6}$).
The rest of the features have significantly higher values, and our feature
cut-off technique distinguishes them successfully, pointing out the number of
selected features in each case.

3\. Effect of $\lambda_{1}$ and $\lambda_{2}$ on Feature Sparsity:

Figure 5: Effect of $\lambda_{1}$ and $\lambda_{2}$ on sparsity.

In Figure 5, we show how hyperparameters $\lambda_{1}$ and $\lambda_{2}$
control sparsity on Human Activity data. We fix $\lambda_{2}$ to $0.001$ and
run SABCE with different values of $\lambda_{1}$, which we show in the first
row. Observe that the solution become less sparse with the decrease of
$\lambda_{1}$. In the second row, we present a similar plot over different
values of $\lambda_{2}$ while fixing $\lambda_{1}$ to $0.001$. The change of
$\lambda_{2}$ doesn’t contribute too much to the model’s sparsity.

4\. Generalization Property of Selected Features: To investigate the
generalization aspect of the features, we restrict training and validation
sets to the selected variables. We fit a one-hidden layer neural network on
the training set to predict the class label of the validation samples. Figure
6 shows the accuracy as a function of feature count.

Figure 6: Accuracy on the validation set as a function of the number of
features.

The plot suggests that the selected features can accurately predict unseen
samples. We see that the accuracy increases with the number of features for
MNIST, ISOLET Human Activity, and GLI_85; in contrast, adding more features
does not change the accuracy for ALLAML and Prostate_GE significantly.

5\. Feature Selection Stability: In this experiment, we shed light on the
stability of the feature selection process; specifically, we want to compare
and contrast the feature sets across several trials. To this end, We run our
model five times on two high-dimensional biological data, ALLAML, Prostate_GE,
and one high sample size data Human Activity and then compare the feature
sets.

Figure 7: Venn diagram of five sets of features for each of the three high-
dimensional data sets.

Figure 7 shows the results using Venn diagrams. Observe that the number of
selected features over five runs are generally close to each other. In each
case, there is a significant number of overlapping features. For each data
set, we calculate the Jaccard index using the five feature sets to measure
their similarity—the Jaccard index of ALLAML, Prostate_GE, and Human Activity
are 0.6254, 0.6828, and 0.6253, respectively. High Jaccard scores indicate
that the feature sets have a lot of commonality over different runs.

##  3 Experimental Results

We present the comparative evaluation of our model on various data sets using
several feature selection techniques.

###  3.1 Experimental Details

Dataset | No. Features | No. of Classes | No. of Samples | Domain
---|---|---|---|---
ALLAML | 7129 | 2 | 72 | Biology
GLIOMA | 4434 | 4 | 50 | Biology
SMK_CAN | 19993 | 2 | 187 | Biology
Prostate_GE | 5966 | 2 | 102 | Biology
GLI_85 | 22283 | 2 | 85 | Biology
CLL_SUB | 11340 | 3 | 111 | Biology
Mice Protein | 77 | 8 | 975 | Biology
COIL20 | 1024 | 20 | 1440 | Image
Isolet | 617 | 26 | 7797 | Speech
Human Activity | 561 | 6 | 5744 | Accelerometer Sensor
MNIST | 784 | 10 | 70000 | Image
FMNIST | 784 | 10 | 70000 | Image
Table 3: Descriptions of the data sets used for benchmarking experiments.

We used twelve data sets from a variety of domains (image, biology, speech,
and sensor; see Table 3) and five neural network-based models to run three
benchmarking experiments. To this end, we picked the published results from
two papers (Lemhadri et al., 2021; Singh et al., 2020) for benchmarking and we
ran the Stochastic Gate (Yamada et al., 2020) using the code provided by
authors. We followed the same experimental methodology described in (Lemhadri
et al., 2021; Singh et al., 2020) for an apples-to-apples comparison. This
approach permitted a direct comparison of LassoNet, FsNet, Supervised CAE
using the authors’ best results. All three experiments follow the standard
workflow:

  * •

Split each data sets into training and test partition.

  * •

Run SABCE on the training set to extract top $K\in\\{10,50\\}$ features.

  * •

Using the top $K$ features train a one hidden layer ANN classifier with $H$
ReLU units to predict the test samples. The $H$ is picked using a validation
set.

  * •

Repeat the classification 20 times and report average accuracy.

Now we describe the details of the two experiments.

Experiment 1: The first bench-marking experiment is conducted on six publicly
available (Li et al., 2018) high dimensional biological data sets: ALLAML,
GLIOMA, SMK_CAN, Prostate_GE, GLI_85, and CLL_SUB111Available at
https://jundongl.github.io/scikit-feature/datasets.html to compare SABCE with
FsNet, Supervised CAE (SCAE), and Stochastic Gates (STG). Following the
experimental protocol of Singh et al. (Singh et al., 2020), we randomly
partitioned each data into a 50:50 ratio of train and test and ran SABCE, STG
on the training set. After that, we calculated the test accuracy using the top
$K=\\{10,50\\}$ features. We repeated the experiment 20 times and reported the
mean accuracy. We ran a 5-fold cross-validation on the training set to tune
the hyperparameters.

Experiment 2: In the second bench-marking experiment, we compared our approach
with LassoNet(Lemhadri et al., 2021) and Stochastic Gate(Yamada et al., 2020)
on six data sets: Mice Protein222There are some missing entries that are
imputed by mean feature values., COIL20, Isolet, Human Activity, MNIST, and
FMNIST333Available at UCI Machine Learning repository. Following the
experimental set of Lemhadri et al., we split each data set into 70:10:20
ratio of training, validation, and test sets. We ran SCE on the training set
to pick the top $K=50$ features to predict the class labels of the sequester
test set. We extensively used the validation set to tune the hyperparameters.

###  3.2 Results

Now we discuss the results of the benchmarking experiments. In Table 4 we
present the results of the first experiment where we compare SABCE, SCAE, STG,
and FsNet on six high-dimensional biological data sets. Apart from the results
using a subset (10 and 50) of features, we also provide the prediction using
all the features. In most cases, feature selection helps improve
classification performance. Generally, SABCE features perform better than SCAE
and FsNet; out of the twelve classification tasks, SABCE produces the best
result on ten. Notice that the top fifty SABCE features give a better
prediction rate than the top ten in all the cases. Interestingly, the accuracy
of SCAE and FsNet drop significantly on SMK_CAN, GLI_85 and CLL_SUB using the
top fifty features.

Data set | Top 10 features | Top 50 features | All
---|---|---|---
FsNet | SCAE | STG | SABCE | FsNet | SCAE | STG | SABCE | Fea.
ALLAML | $91.1$ | $83.3$ | $81.0$ | 93.7 | $92.2$ | $93.6$ | $88.5$ | 94.6 | $89.9$
Prostate_GE | $87.1$ | $83.5$ | $82.3$ | 89.9 | $87.8$ | $88.4$ | $85.0$ | 90.1 | $75.9$
GLIOMA | $62.4$ | $58.4$ | $62.0$ | 66.8 | $62.4$ | $60.4$ | $70.4$ | 74.2 | $70.3$
SMK_CAN | 69.5 | $68.0$ | $65.2$ | $68.1$ | $64.1$ | $66.7$ | $68.0$ | 69.4 | $65.7$
GLI_85 | $87.4$ | 88.4 | $72.2$ | $84.7$ | $79.5$ | $82.2$ | $81.0$ | 85.7 | $79.5$
CLL_SUB | $64.0$ | $57.5$ | $54.4$ | 70.8 | $58.2$ | $55.6$ | $63.2$ | 72.2 | $56.9$
Table 4: Comparison of mean classification accuracy of FsNet, SCAE,STG and
SABCE features on six real-world high-dimensional biological data sets. The
prediction rates are averaged over twenty runs on the test set. Numbers for
FsNet and SCAE are being reported from (Singh et al., 2020). The last column
reports accuracy using all features using an ANN classifier.

Now we turn our attention to the results of the second experiment, as shown in
Table 5. The features of the SABCE produce better classification accuracy than
LassoNet in all cases. Besides COIL20, our model has better accuracy by a
margin of $4\%-6.5\%$. On the other hand, STG performed slightly better (a
margin of 0.4% to 0.7%) than SABCE on Mice Protein, COIL20, and MNIST. In
contrast, our model is more accurate than STG on FMNIST, ISOLET, and Activity
by 1.6% to 4.2%. Note that LassoNet is the worst-performing model. In this
experiment, STG performed competitively compared to the first, where STG’s
performance was significantly worse than that of SABCE. Upon further
investigation, it turns out that the model fails to induce feature sparsity on
all six high-dimensional biological data sets. We fit the model on the
training partition of each data set and then plot the probability of the
stochastic gates in descending order, which we call the sparsity plot. We run
STG using a wide range on $\lambda$, which controls the sparsity of the model.
In Figure 8, we show the result on ALLAML data. As we can see, the model
doesn’t create a sparse solution for input features for any of the nine
values. Ideally, we should have observed the probability of many variables to
near zero so that those features could be ignored. Changing the activation
function, number of hidden nodes, or depth of the network doesn’t produce a
sparser solution. We kept the similar analysis on other data sets in Appendix
(Section C).

Data set | Top 50 features | All features
---|---|---
LassoNet | STG | SABCE | ANN
Mice Protein | $95.8$ | 99.8 | $99.4$ | $99.00$
MNIST | $87.3$ | 94.7 | $94.0$ | $92.8$
FMNIST | $80.0$ | $83.8$ | 85.4 | $88.30$
ISOLET | $88.5$ | $90.9$ | 93.3 | $95.30$
COIL-20 | $99.1$ | 99.7 | $99.3$ | $99.60$
Activity | $84.9$ | $86.6$ | 90.8 | $85.30$
Table 5: Classification results using LassoNet, STG, and SABCE features on six
publicly available data sets. Numbers for LassoNet and ’All features ANN’ are
reported from (Lemhadri et al., 2021). All the reported accuracies are
measured on the test set. Figure 8: Sparsity analysis of Stochastic Gates on
ALLAML data.

##  4 Related Work

Feature selection has a long history spread across many fields, including
bioinformatics, document classification, data mining, hyperspectral band
selection, computer vision, etc. We describe the literature related to the
embedded methods where the selection criteria are part of a model. The model
can be either linear or non-linear. Adding an $\ell_{1}$ penalty to
classification and regression methods naturally produce feature selectors for
linear model, see (Tibshirani, 1996; Fonti and Belitser, 2017; Muthukrishnan
and Rohini, 2016; Kim and Kim, 2004; Zou and Hastie, 2005; Marafino et al.,
2015; Shen et al., 2011; Sokolov et al., 2016; Lindenbaum and Steinerberger,
2021; Candes et al., 2008; Daubechies et al., 2010; Bertsimas et al., 2017;
Xie and Huang, 2009). Support Vector Machines (Cortes and Vapnik, 1995) have
been used extensively for feature selection, see (Marafino et al., 2015; Shen
et al., 2011; Sokolov et al., 2016; Guyon et al., 2002; O’Hara et al., 2013;
Chepushtanova et al., 2014).

While the linear models are generally fast and convex, they don’t capture the
non-linear relationship among the input features (unless a kernel trick is
applied). Non-linear models based on deep neural networks overcome these
limitations. Here, we will briefly discuss a handful of such models. Group
Sparse ANN (Scardapane et al., 2017) used group Lasso (Tibshirani, 1996) to
impose the sparsity on a group of variables instead of a single variable. Li
et al. proposed deep feature selection (DFS), which is a multilayer neural
network-based feature selection technique (Li et al., 2016). (Kim et al.,
2016) proposed a heuristics based technique to assign importance to each
feature. Using the ReLU activation, (Roy et al., 2015) provided a way to
measure the contribution of an input feature towards hidden activation of next
layer. (Han et al., 2018) developed an unsupervised feature selection
technique based on the autoencoder architecture. (Taherkhani et al., 2018)
proposed a RBM (Hinton et al., 2006; Hinton and Salakhutdinov, 2006) based
feature selection model. Also see, (Balın et al., 2019; Yamada et al., 2020;
Singh et al., 2020).

##  5 Discussion, Conclusion and Limitations

In this paper, we proposed a novel neural network-based feature selection
technique, Sparse Adaptive Bottleneck Centroid-Encoder (SABCE). Using the
basic multi-layer perceptron encoder-decoder neural network architecture, the
model backpropagates the SABCE cost to a feature selection layer that filters
out non-discriminating features by $\ell_{2,1}$-regularization. The setting
allows the feature selection to be data-driven without needing prior
knowledge, such as the number of features to be selected and the underlying
distribution of the input features. The extensive analysis in Section 2.3
demonstrates that the $\ell_{2,1}$-norm induces good feature sparsity without
shrinking all the variables. Unlike other methods, e.g., Stochastic Gates, our
approach promotes feature sparsity for high-dimension and low-sample size
biological datasets, further demonstrating the value of SABCE as a feature
detector. We chose the $\lambda_{1}$ and $\lambda_{2}$ from the validation set
from a wide range of values and saw that smaller values work better for
classification. The plots with the Venn diagrams confirm the consistent and
stable feature detection ability of SABCE.

The rigorous benchmarking with twelve data sets from diverse domains and four
methods provides evidence that the features of SABCE produce better
generalization performance than other state-of-the-art models. We compared
SABCE with FsNet, mainly designed for high-dimensional biological data, and
found that our proposed method outperformed it in most cases. In fact, our
model produced new state-of-the-art results in ten cases out of twelve. The
comparison also includes Supervised CAE, which is less accurate than SABCE. On
the data sets where the number of observations is more than the number of
variables, SABCE features produces better classification results than LassoNet
in all six cases and better than Stochastic Gates on three data sets. The
strong generalization performance, coupled with the ability to sparsify input
features, establishes the value of our model as a nonlinear feature detector.

Although SABCE produces new state-of-the-art results on diverse data sets, our
model won’t be the right choice for the cases where class centroids make
little sense, e.g., natural images. The current scope of the work doesn’t
allow us to investigate other optimization techniques, e.g., proximal gradient
descent, trimmed Lasso, etc., which we plan to explore in the future.

## References

  * Alelyani et al. (2018) Salem Alelyani, Jiliang Tang, and Huan Liu.  Feature selection for clustering: A review.  _Data Clustering_ , pages 29–60, 2018.
  * Balın et al. (2019) Muhammed Fatih Balın, Abubakar Abid, and James Zou.  Concrete autoencoders: Differentiable feature selection and reconstruction.  In _International conference on machine learning_ , pages 444–453. PMLR, 2019.
  * Bertsimas et al. (2017) Dimitris Bertsimas, Martin S Copenhaver, and Rahul Mazumder.  The trimmed lasso: Sparsity and robustness.  _arXiv preprint arXiv:1708.04527_ , 2017.
  * Candes et al. (2008) Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd.  Enhancing sparsity by reweighted l1 minimization.  _Journal of Fourier analysis and applications_ , 14(5):877–905, 2008.
  * Chepushtanova et al. (2014) Sofya Chepushtanova, Christopher Gittins, and Michael Kirby.  Band selection in hyperspectral imagery using sparse support vector machines.  In _Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XX_ , volume 9088, page 90881F. International Society for Optics and Photonics, 2014.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik.  Support vector machine.  _Machine learning_ , 20(3):273–297, 1995.
  * Daubechies et al. (2010) Ingrid Daubechies, Ronald DeVore, Massimo Fornasier, and C Sinan Gunturk.  Iteratively reweighted least squares minimization for sparse recovery.  _Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences_ , 63(1):1–38, 2010.
  * El Aboudi and Benhlima (2016) Naoual El Aboudi and Laila Benhlima.  Review on wrapper feature selection approaches.  In _2016 International Conference on Engineering & MIS (ICEMIS)_, pages 1–5. IEEE, 2016.
  * Fleuret (2004) François Fleuret.  Fast binary feature selection with conditional mutual information.  _Journal of Machine learning research_ , 5(9), 2004.
  * Fonti and Belitser (2017) Valeria Fonti and Eduard Belitser.  Feature selection using lasso.  _VU Amsterdam Research Paper in Business Analytics_ , 30:1–25, 2017.
  * Ghosh and Kirby (2022) Tomojit Ghosh and Michael Kirby.  Supervised dimensionality reduction and visualization using centroid-encoder.  _Journal of Machine Learning Research_ , 23(20):1–34, 2022.
  * Ghosh et al. (2018) Tomojit Ghosh, Xiaofeng Ma, and Michael Kirby.  New tools for the visualization of biological pathways.  _Methods_ , 132:26 – 33, 2018.  ISSN 1046-2023.  doi: https://doi.org/10.1016/j.ymeth.2017.09.006.  URL http://www.sciencedirect.com/science/article/pii/S1046202317300439.  Comparison and Visualization Methods for High-Dimensional Biological Data.
  * Goldberg and Holland (1988) David E Goldberg and John Henry Holland.  Genetic algorithms and machine learning.  _Machine Learning_ , 1988.
  * Guyon and Elisseeff (2003) Isabelle Guyon and André Elisseeff.  An introduction to variable and feature selection.  _Journal of machine learning research_ , 3(Mar):1157–1182, 2003.
  * Guyon et al. (2002) Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik.  Gene selection for cancer classification using support vector machines.  _Machine learning_ , 46(1):389–422, 2002.
  * Han et al. (2018) Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu.  Autoencoder inspired unsupervised feature selection.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2941–2945. IEEE, 2018.
  * Hinton and Salakhutdinov (2006) Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  _science_ , 313(5786):504–507, 2006.
  * Hinton et al. (2006) Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh.  A fast learning algorithm for deep belief nets.  _Neural Comput._ , 18(7):1527–1554, July 2006\.  ISSN 0899-7667.  doi: 10.1162/neco.2006.18.7.1527.  URL http://dx.doi.org/10.1162/neco.2006.18.7.1527.
  * Hsu et al. (2002) Chun-Nan Hsu, Hung-Ju Huang, and Stefan Dietrich.  The annigma-wrapper approach to fast feature selection for neural nets.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , 32(2):207–212, 2002.
  * Kennedy and Eberhart (1995) James Kennedy and Russell Eberhart.  Particle swarm optimization.  In _Proceedings of ICNN’95-international conference on neural networks_ , volume 4, pages 1942–1948. IEEE, 1995.
  * Kim et al. (2016) Seong Gon Kim, Nawanol Theera-Ampornpunt, Chih-Hao Fang, Mrudul Harwani, Ananth Grama, and Somali Chaterji.  Opening up the blackbox: an interpretable deep neural network-based classifier for cell-type specific enhancer predictions.  _BMC systems biology_ , 10(2):243–258, 2016.
  * Kim and Kim (2004) Yongdai Kim and Jinseog Kim.  Gradient lasso for feature selection.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 60, 2004.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Lal et al. (2006) Thomas Navin Lal, Olivier Chapelle, Jason Weston, and André Elisseeff.  Embedded methods.  In _Feature extraction_ , pages 137–165. Springer, 2006.
  * Lazar et al. (2012) Cosmin Lazar, Jonatan Taminau, Stijn Meganck, David Steenhoff, Alain Coletta, Colin Molter, Virginie de Schaetzen, Robin Duque, Hugues Bersini, and Ann Nowe.  A survey on filter techniques for feature selection in gene expression microarray analysis.  _IEEE/ACM transactions on computational biology and bioinformatics_ , 9(4):1106–1119, 2012.
  * Lemhadri et al. (2021) Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani.  Lassonet: A neural network with feature sparsity.  _Journal of Machine Learning Research_ , 22(127):1–29, 2021.
  * Li et al. (2020) Gen Li, Yuantao Gu, and Jie Ding.  The efficacy of $l\\_1$ regularization in two-layer neural networks.  _arXiv preprint arXiv:2010.01048_ , 2020.
  * Li et al. (2018) Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu.  Feature selection: A data perspective.  _ACM Computing Surveys (CSUR)_ , 50(6):94, 2018\.
  * Li et al. (2016) Yifeng Li, Chih-Yu Chen, and Wyeth W Wasserman.  Deep feature selection: theory and application to identify enhancers and promoters.  _Journal of Computational Biology_ , 23(5):322–336, 2016.
  * Lindenbaum and Steinerberger (2021) Ofir Lindenbaum and Stefan Steinerberger.  Randomly aggregated least squares for support recovery.  _Signal Processing_ , 180:107858, 2021.
  * Marafino et al. (2015) Ben J Marafino, W John Boscardin, and R Adams Dudley.  Efficient and sparse feature selection for biomedical text classification via the elastic net: Application to icu risk stratification from nursing notes.  _Journal of biomedical informatics_ , 54:114–120, 2015\.
  * Metzker (2010) Michael L Metzker.  Sequencing technologies—the next generation.  _Nature reviews genetics_ , 11(1):31, 2010.
  * Muthukrishnan and Rohini (2016) R Muthukrishnan and R Rohini.  Lasso: A feature selection technique in predictive modeling for machine learning.  In _2016 IEEE international conference on advances in computer applications (ICACA)_ , pages 18–20. IEEE, 2016.
  * O’Hara et al. (2013) Stephen O’Hara, Kun Wang, Richard A Slayden, Alan R Schenkel, Greg Huber, Corey S O’Hern, Mark D Shattuck, and Michael Kirby.  Iterative feature removal yields highly discriminative pathways.  _BMC genomics_ , 14(1):1–15, 2013.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In _NIPS-W_ , 2017.
  * Pease et al. (1994) A C Pease, D Solas, E J Sullivan, M T Cronin, C P Holmes, and S P Fodor.  Light-generated oligonucleotide arrays for rapid dna sequence analysis.  _Proceedings of the National Academy of Sciences_ , 91(11):5022–5026, 1994.  ISSN 0027-8424.  doi: 10.1073/pnas.91.11.5022.  URL https://www.pnas.org/content/91/11/5022.
  * Reuter et al. (2015) Jason A Reuter, Damek V Spacek, and Michael P Snyder.  High-throughput sequencing technologies.  _Molecular cell_ , 58(4):586–597, 2015.
  * Roy et al. (2015) Debaditya Roy, K Sri Rama Murty, and C Krishna Mohan.  Feature selection using deep neural networks.  In _2015 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–6. IEEE, 2015.
  * Scardapane et al. (2017) Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini.  Group sparse regularization for deep neural networks.  _Neurocomputing_ , 241:81–89, 2017.
  * Shalon et al. (1996) Dari Shalon, Stephen J Smith, and Patrick O Brown.  A dna microarray system for analyzing complex dna samples using two-color fluorescent probe hybridization.  _Genome research_ , 6(7):639–645, 1996.
  * Shen et al. (2011) Li Shen, Sungeun Kim, Yuan Qi, Mark Inlow, Shanker Swaminathan, Kwangsik Nho, Jing Wan, Shannon L Risacher, Leslie M Shaw, John Q Trojanowski, et al.  Identifying neuroimaging and proteomic biomarkers for mci and ad via the elastic net.  In _International Workshop on Multimodal Brain Image Analysis_ , pages 27–34. Springer, 2011.
  * Singh et al. (2020) Dinesh Singh, Héctor Climente-González, Mathis Petrovich, Eiryo Kawakami, and Makoto Yamada.  Fsnet: Feature selection network on high-dimensional biological data.  _arXiv preprint arXiv:2001.08322_ , 2020.
  * Sokolov et al. (2016) Artem Sokolov, Daniel E Carlin, Evan O Paull, Robert Baertsch, and Joshua M Stuart.  Pathway-based genomics prediction using generalized elastic net.  _PLoS computational biology_ , 12(3):e1004790, 2016.
  * Taherkhani et al. (2018) Aboozar Taherkhani, Georgina Cosma, and T Martin McGinnity.  Deep-fs: A feature selection algorithm for deep boltzmann machines.  _Neurocomputing_ , 322:22–37, 2018.
  * Tibshirani (1996) Robert Tibshirani.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Vergara and Estévez (2014) Jorge R Vergara and Pablo A Estévez.  A review of feature selection methods based on mutual information.  _Neural computing and applications_ , 24(1):175–186, 2014.
  * Xie and Huang (2009) Huiliang Xie and Jian Huang.  Scad-penalized regression in high-dimensional partially linear models.  _The Annals of Statistics_ , 37(2):673–696, 2009\.
  * Yamada et al. (2020) Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger.  Feature selection using stochastic gates.  In _International Conference on Machine Learning_ , pages 10648–10659. PMLR, 2020.
  * Yu and Liu (2003) Lei Yu and Huan Liu.  Feature selection for high-dimensional data: A fast correlation-based filter solution.  In _Proceedings of the 20th international conference on machine learning (ICML-03)_ , pages 856–863, 2003.
  * Zou (2006) Hui Zou.  The adaptive lasso and its oracle properties.  _Journal of the American statistical association_ , 101(476):1418–1429, 2006.
  * Zou and Hastie (2005) Hui Zou and Trevor Hastie.  Regularization and variable selection via the elastic net.  _Journal of the royal statistical society: series B (statistical methodology)_ , 67(2):301–320, 2005.

##  A Reproducibility

Table 6 shows the values of the hyperparameters that we use in our
experiments. The tuning of these parameters is done in two ways: for low
sample size biological data sets (first six data sets in the Table 6), we run
five-fold cross-validation on a training partition. We split each of the
remaining data sets into train, validation, and test ratio of 70:10:20 and
used the validation set for tuning.

Dataset | Network topology | Activation | Learning Rate | $\lambda_{1},\lambda_{2}$ | $\mu_{1},\mu_{2}$ | Epoch
---|---|---|---|---|---|---
ALLAML | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
GLIOMA | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
SMK_CAN | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
Prostate_GE | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
GLI_85 | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
CLL_SUB | $d\rightarrow 250\rightarrow d$ | tanh | 0.001 | 0.001,0.001 | 0.6,0.1 | 1050
Mice Protein | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 0.2,0.6 | 1050
COIL20 | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 0.8,0.1 | 1050
Isolet | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 0.8,0.3 | 1050
Human Activity | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 1.0,0.9 | 1050
MNIST | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 0.6,0.1 | 1050
FMNIST | $d\rightarrow 100\rightarrow d$ | tanh | 0.008 | 0.001,0.001 | 0.6,0.1 | 1050
Table 6: Details of network topology and hyperparameters for SABCE. The number
$d$ is the input dimension of the network and is data set dependent.

##  B Compute Time

We present the average compute time of training for each data set in the table
below.

Dataset | Training time (in minutes)
---|---
ALLAML | 0.137
GLIOMA | 0.122
SMK_CAN | 0.889
Prostate_GE | 0.121
GLI_85 | 0.809
CLL_SUB | 0.276
Mice Protein | 1.538
COIL20 | 2.563
Isolet | 4.757
Human Activity | 2.795
MNIST | 20.621
FMNIST | 23.697
Table 7: Average training time for Sparse Adaptive Bottleneck Centroid-Encoder
for each data set.

##  C Sparsity Analysis of Stochastic Gates

In this section, we present a detailed analysis of the input feature sparsity
of Stochastic Gates (STG)(Yamada et al., 2020). We have observed that STG
performed competitively for the data sets with more samples than features
(Table 5 of main paper); in contrast, the model performed relatively weakly on
data sets where the number of variables is very large than the number of input
data (Table 4 of main paper). Upon further investigation, it turns out that
the model fails to induce feature sparsity on all six high-dimensional
biological data sets. We fit the model on the training partition of each data
set and then plot the probability of the stochastic gates in descending order,
which we call the sparsity plot. We run STG using a wide range on $\lambda$,
which controls the sparsity of the model.

In Figure 9, 10,11, 12 and 13, we present the sparsity plot for Prostate_GE,
GLIOMA, SMK_CAN, GLI_85, and CLL_SUB data respectively. These sparsity plots
are qualitatively similar to ALLAML (Figure 8), except for GLIOMA when
$\lambda=5.0$.
It is clear from this analysis that Stochastic Gate cannot sparsify the hi-
dimensional biological data sets, and it’s plausible that the model doesn’t
always pick the proper discriminatory feature set. The classification results
in Table 4 (in the main article) do support the claim.

Figure 9: Sparsity analysis of Prostate_GE data. Figure 10: Sparsity analysis
of GLIOMA data. Figure 11: Sparsity analysis of SMK_CAN data. Figure 12:
Sparsity analysis of GLI_85 data. Figure 13: Sparsity analysis of CLL_SUB
data.