# Fermi surface and mass renormalization in the iron-based superconductor
YFe2Ge2

Jordan Baglo<EMAIL_ADDRESS>Now at Institut quantique and
Département de physique, Université de Sherbrooke, Sherbrooke, Québec, Canada
Cavendish Laboratory, University of Cambridge, Cambridge CB3 0HE, UK  Jiasheng
Chen  Cavendish Laboratory, University of Cambridge, Cambridge CB3 0HE, UK
Keiron Murphy  Now at Clarendon Laboratory, University of Oxford, Oxford, UK
Cavendish Laboratory, University of Cambridge, Cambridge CB3 0HE, UK  Roos
Leenen  High Field Magnet Laboratory (HFML-EMFL), Radboud University, 6525 ED
Nijmegen, The Netherlands  Alix McCollam  High Field Magnet Laboratory (HFML-
EMFL), Radboud University, 6525 ED Nijmegen, The Netherlands  Michael L.
Sutherland  Cavendish Laboratory, University of Cambridge, Cambridge CB3 0HE,
UK  F. Malte Grosche  Cavendish Laboratory, University of Cambridge, Cambridge
CB3 0HE, UK

(August 28, 2024)

###### Abstract

Quantum oscillation measurements in the new unconventional superconductor
YFe2Ge2 resolve all four Fermi surface pockets expected from band structure
calculations, which predict an electron pocket in the Brillouin zone corner
and three hole pockets enveloping the centers of the top and bottom of the
Brillouin zone. The carrier masses are uniformly renormalized by about a
factor of five and broadly account for the enhanced heat capacity Sommerfeld
coefficient
$\simeq$100\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$$.
Our data highlight the key role of the electron pocket, which despite its
small volume accounts for about half the total density of states, and point
towards a predominantly local mechanism underlying the mass renormalization in
YFe2Ge2.

Superconductivity in iron pnictides and chalcogenides involves a rich
interplay of magnetic, structural, and nematic instabilities as well as the
associated quantum critical phenomena [1, 2, 3, 4]. This diverse phenomenology
includes strong quasiparticle mass renormalization, which in the extreme case
of the alkali metal iron arsenides (K/Rb/Cs)Fe2As2 produces heat capacity
Sommerfeld coefficients $\sim
100\relbar$200\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$$,
approaching values usually associated with rare-earth-based heavy fermion
materials. Understanding the origin of this mass renormalization is central to
a comprehensive theory of iron-based materials: is it caused by non-local,
nearly quantum critical fluctuations associated with incipient order, which
also produce the superconducting pairing interaction, or should it be
attributed primarily to local, dynamical correlations [5], as in the Hund’s
metal scenario (e.g. [6])?

The iron germanide superconductor YFe2Ge2 [7, 8, 9] provides a fresh
opportunity to investigate the origin of mass renormalization in strongly
correlated iron-based intermetallics. It shares key aspects with the alkali
metal iron arsenides (K/Rb/Cs)Fe2As2, such as (i) bad metal behavior at high
temperature $T$, with resistivities $\rho$ of several hundred
$\mathrm{\SIUnitSymbolMicro\SIUnitSymbolOhm}\text{\,}\mathrm{cm}$, (ii) low
transition temperatures $T_{c}$ of order a few kelvin, (iii) strongly enhanced
heat capacity Sommerfeld coefficient
$\gamma_{n}\simeq$100\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$$,
(iv) reduced heat capacity jump at $T_{c}$ of order $0.4T_{c}\gamma_{n}$, and
residual extrapolated $C/T$ at low $T$ of order $0.4\gamma_{n}$ [10, 11]. On
the other hand, it lacks the pnictogen or chalcogen constituents of other Fe-
based superconductors, and because the Ge layers in YFe2Ge2 are covalently
bonded along $\hat{c}$, it forms a more compressed, collapsed-tetragonal
structure. Density functional theory (DFT) calculations [12, 13] suggest that
this leads to a much more strongly warped Fermi surface (FS) geometry than the
cylindrical FS sheets found in other iron-based superconductors (Fig. 1).

Figure 1:  Ab initio Fermi surface structure of YFe2Ge2. The Fermi surface
(top center) consists of three concentric hole pockets (A-C), which envelop
the Z point on the Brillouin zone (BZ) boundary, and a single electron pocket
(D) centered on the X point at the corner of the BZ. Lighter colors indicate
higher Fermi velocity.

Figure 2:  Low temperature properties of YFe2Ge2. The in-plane electrical
resistivity $\rho(T)$ displays a superconducting transition (arrow) below an
anomalous normal-state $T^{3/2}$ form. A sharp superconducting anomaly (arrow)
is also observed in the heat capacity Sommerfeld ratio $C/T$ (inset), which in
the normal state reaches
$\simeq$100\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$$.

Key input for any comprehensive theoretical description of iron-based
superconductors derives from the experimental determination of the electronic
structure by photoemission spectroscopy or quantum oscillation measurements
(e.g. [14, 15, 16]). An early ARPES study in YFe2Ge2 [17] investigated key
aspects of the electronic structure, and core-level spectroscopy indicated a
large fluctuating Fe moment $\simeq$1\text{\,}\mu_{\mathrm{B}}$$ [18], which
is consistent with neutron scattering and NMR studies [19, 20]. The precise
determination of Fermi surface and carrier mass by quantum oscillation
techniques, however, has in YFe2Ge2 so far been held back by the lack of
single crystals with the required level of purity [21]. Here, we present a
detailed study of Fermi surface structure and carrier mass by observing de
Haas-van Alphen oscillations in a new generation of high-quality crystals of
YFe2Ge2. We find that the Fermi surface consists of three hole pockets and one
electron pocket, with shapes broadly in line with numerical calculations [12,
13]. Carrier masses are enhanced roughly five-fold over DFT values. The mass
renormalization varies only weakly between Fermi surface pockets and within
each Fermi surface pocket, suggesting that on-site interactions provide the
primary mechanism for the profound mass enhancement and boost to the
electronic heat capacity recorded in YFe2Ge2.

High-purity crystals of YFe2Ge2 were grown by a liquid transport technique
[22, 23] and characterized by electrical transport, magnetic, and
thermodynamic measurements. They display sharp superconducting transition
anomalies (Fig. 2), and their residual resistivities are
$\simeq$0.3\text{\,}\mathrm{\SIUnitSymbolMicro\SIUnitSymbolOhm}\text{\,}\mathrm{cm}$$,
corresponding to residual resistance ratios
$\text{RRR}=\rho($300\text{\,}\mathrm{K}$)/\rho_{0}\simeq 700$. Angle-
dependent de Haas-van Alphen measurements at fields of up to
$18\text{\,}\mathrm{T}$ were performed using a mutual inductance technique
with modulation field amplitude $\simeq$0.2\text{\,}\mathrm{mT}$$ at a
frequency of $\sim$29\text{\,}\mathrm{Hz}$$ on two samples (S1, S2) in a
superconducting cryomagnet/dilution refrigerator system at the Cavendish
Laboratory, and using piezoresistive torque magnetometry on a third sample
(S3) at fields of up to $38\text{\,}\mathrm{T}$ in a resistive
electromagnet/dilution refrigerator system at HFML Nijmegen. Sample S1 (S2)
was mounted with the crystallographic $\hat{c}$ ($\hat{a}$) direction aligned
with the axis of the respective pickup coil, and S3 was mounted with $\hat{c}$
perpendicular to the surface of the cantilever. Quantum oscillation data were
extracted by subtracting a low-order polynomial background from the raw data,
and oscillations periodic in $1/B$ were identified from peaks in the power
spectrum. The peak frequencies $F$ relate to extremal cross-sectional areas
$A_{k}$ of the Fermi surface via the Onsager relation $A_{k}=(2\pi e/\hbar)F$.
The dependence of the signal amplitude $\tilde{y}$ on temperature $T$ at a
fixed magnetic field $B$ provides the effective carrier mass $m^{*}$ via the
Lifshitz-Kosevich expression $\tilde{y}=\alpha
T\left[\sinh\left($14.639\text{\,}\mathrm{T}\mathrm{K}^{-1}$\frac{T}{B}\frac{m^{*}}{m_{e}}\right)\right]^{-1}$,
where $\alpha$ is a temperature-independent factor and $m_{e}$ is the bare
electron mass. The field dependence of the signal envelope provides an
estimate of the electronic mean free path [24, 25]. The electronic structure
was calculated using the Generalized Gradient Approximation [26] in WIEN2k
[27] with $100,000\leavevmode\nobreak\ k$-points in the BZ (6768 $k$-points in
the irreducible BZ) and $Rk_{max}\\!=\\!7$, using the experimentally
determined crystal structure at $100\text{\,}\mathrm{K}$ with
$a=$3.959\,17(3)\text{\,}\mathrm{\SIUnitSymbolAngstrom}$$,
$c=$10.397\,54(13)\text{\,}\mathrm{\SIUnitSymbolAngstrom}$$, and the
fractional vertical Ge position $z=0.378331(7)$ [22]. Extremal orbits, band
masses and pocket-resolved contributions to the density of states (DOS) were
extracted using SKEAF [28], and Fermi surfaces were plotted in FermiSurfer
[29].

Figure 3:  De Haas-van Alphen signal: strong oscillations are observed in
sample S1 for $B||c$ down to fields $<$6\text{\,}\mathrm{T}$$ (upper panel,
black trace and associated black peak in power spectrum). At a tilt angle
$\theta=30^{\circ}$, three fundamental frequencies $\alpha$, $\beta$, $\gamma$
as well as some harmonics can be resolved. The evolution of these frequencies
with tilt angle (lower panel) suggests that they can be assigned to orbits on
sheets A, B, and C, respectively (Fig. 1, see text).

The calculated Fermi surface (Fig. 1) consists of three roughly ellipsoidal
hole pockets (A–C) nested around Z at the top and bottom of the Brillouin zone
and a roughly cylindrical electron pocket (D) around X at the zone corner,
with a duckbill-shaped outgrowth towards the center of the BZ at $\Gamma$.
Moreover, our DFT calculations produce an unrenormalized Sommerfeld
coefficient
$\gamma_{0}\simeq$16.7\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$$,
a factor of six less than the experimentally observed value but significantly
larger than values previously obtained in calculations based on numerically
relaxed $z$ parameters [12, 13].

The quantum oscillation signal observed in S1 (Fig. 3) displays three
fundamental frequencies, $\alpha$, $\beta$, and $\gamma$. The same set of
frequencies is present in sample S2 [25]. The results of a rotation study from
$H\parallel{}\hat{c}$ ($\theta=0^{\circ}$) to $H\parallel{}\hat{a}$
($\theta=90^{\circ}$) are summarized in Figs. 3b and 4a. The three fundamental
frequencies $\alpha$, $\beta$ and $\gamma$ were tracked over most of the
angular range. By comparing to DFT results, they can be assigned to extremal
orbits on the three hole pockets. The highest frequency, $\gamma$, is
unambiguously associated with the largest hole pocket, C. The next highest
frequency, $\beta$, matches predictions for the second hole sheet (B), with
good quantitative agreement near $H\parallel{}\hat{a}$. The third frequency,
$\alpha$, depends weakly on tilt angle, as expected for the smallest, nearly
ellipsoidal hole pocket (A), but at roughly half the predicted frequency. The
effective masses extracted from Lifshitz-Kosevich fits (Fig. 4b) are high,
with $m^{*}$ exceeding $10\leavevmode\nobreak\ m_{e}$ on the $\gamma$ and
$\beta$ sheets. These values markedly exceed the highest masses measured in
the BaFe2(As/P)2 series [30] and are as high as those observed in
(K/Rb/Cs)Fe2As2 [16, 31, 32]. Whereas effective masses for $\alpha$ and
$\beta$ show little angle dependence, that of $\gamma$ rises sharply near
$H\parallel\hat{c}$ (Fig. 4c). A uniform renormalization of DFT band masses by
$\sim 5$ produces good agreement with experimental values on all three hole
pockets.

Figure 4:  Quantum oscillation results for the three hole pockets. (a) Frequencies (filled symbols: S1, open symbols: S2) as function of angle, rotating from $H\parallel{}\hat{c}$ ($\theta=0^{\circ}$) to $H\parallel{}\hat{a}$ ($\theta=90^{\circ}$). Dotted lines show DFT results. A band shift of $-10\text{\,}\mathrm{m}\mathrm{R}\mathrm{y}\mathrm{d}$ was used for $\alpha$ (Tab. 1) (b) Temperature dependence of the oscillation amplitudes extracted in S1 for $$16\text{\,}\mathrm{T}$<B<$18\text{\,}\mathrm{T}$$ at $\theta=30^{\circ}$. Solid lines show Lifshitz-Kosevich fits to the data at $T>$0.12\text{\,}\mathrm{K}$$. (c) Angle dependence of the extracted effective masses for the $\alpha$, $\beta$, and $\gamma$ orbits compared to DFT band masses $m_{\alpha,\beta,\gamma}$, uniformly renormalized by $\times 4.7$ (dashed lines). FS pocket |  A |  B |  C |  $\sum_{A-C}$ |  D
---|---|---|---|---|---
Hole count |  f.u.-1 |  … |  … |  … |  …
Ab initio |  0.098 |  0.208 |  0.836 |  1.142 |  -0.142
QO-corrected |  0.029 |  0.271 |  0.769 |  1.069 |  -0.069
$\Delta E$ (mRyd) |  -10 |  3.5 |  -2.4 |  |  1.6
FS pocket |  A |  B |  C |  D |  $4.7\sum_{A-D}$
$C/T$ |  $\text{mJ}/\text{mol K}^{2}$ |  |  … |  … |  …
Ab initio |  1.68 |  2.95 |  5.30 |  6.81 |  78.5
QO-corrected |  0.79 |  3.34 |  4.43 |  7.58 |  75.9
Table 1: Particle number and DOS budgets in YFe2Ge2. DFT Fermi surface pocket
volumes for the hole pockets A–C (top line) have been corrected by using the
measured cross-sectional areas (second line, see text). The requirement to
reach 1 hole/formula unit (f.u.) fixes the volume of the electron pocket. The
third data line lists the band shifts that bring each pocket in line with the
QO-corrected hole count. The ab initio DOS (fourth data line, converted to
$C/T$) calculated for the experimentally determined fractional vertical Ge
coordinate $z=0.37833$ is dominated by the contribution from the electron
pocket D. The analogous calculation for the shifted bands produces a lower
contribution from the hole pockets, shifting the balance even more towards the
electron pocket. Applying the mass renormalization $\simeq 4.7$ estimated from
QO measurements (Fig. 4) to the calculated DOS produces an overall $C/T$ in
rough agreement with experiment.

The electron pocket D could not be resolved at fields of up to
$18\text{\,}\mathrm{T}$. Its ab initio volume corresponds to 0.142
electrons/formula unit (f.u.), and its expected dominant QO frequency is
$\sim$0.9\text{\,}\mathrm{kT}$$. This estimate can be refined by using the QO
data on the three observed hole pockets in combination with charge neutrality,
which imposes an overall hole count of 1/formula unit. Assuming that the ab
initio geometry and associated hole count $n_{0}$ of each hole pocket can be
scaled to make them consistent with the measured QO frequencies, we estimate
the actual hole count for each pocket as
$n=n_{0}q_{a}^{2}q_{c}/(q_{0,a}^{2}q_{0,c})\propto\sqrt{F_{a}^{2}F_{c}}/\sqrt{F_{0,a}^{2}F_{0,c}}$.
Here, $F_{a}\propto q_{a}q_{c}$, the QO frequency for field along $\hat{a}$,
is given by the extent $q_{a},q_{c}$ of the pocket along $\hat{a}$ and
$\hat{c}$, and $F_{c}\propto q_{a}^{2}$, the QO frequency for field along
$\hat{c}$, is determined by the extent of the pocket $\perp\hat{c}$, or along
$\hat{a}$. $F_{0,a}$ and $F_{0,c}$ are the ab initio frequencies for the same
pocket and $q_{0,a}$, $q_{0,c}$ the corresponding dimensions. In the case of
pocket C, the $\gamma$ frequency was extrapolated to $\theta=0^{\circ}$ and
the resulting hole count checked for consistency against a second calculation
in which $F_{c}$ and $F_{0,c}$ values were taken at $\theta=7.5^{\circ}$, the
lowest angle at which the $\gamma$ oscillation could be observed. The upper
half of Table 1 summarizes the results of these calculations, which suggests
that the total hole count per f.u. arising from A, B and C is about 1.07. This
implies that the volume of the electron pocket is smaller than expected from
ab initio calculations. It corresponds to about $0.07\leavevmode\nobreak\
e^{-}$ per f.u., which leads to a dominant QO frequency of order
$500\text{\,}\mathrm{T}$. A clear signature of oscillations in this frequency
range was seen in torque magnetometry to fields of up to
$38\text{\,}\mathrm{T}$ in sample S3 (Fig. 5). Our initial observation
exploited the phenomenon of torque interaction [33, 24, 25], which mixes
quantum oscillation frequencies via the nonlinear cantilever response at high
magnetic fields. The $\gamma$ peak develops several side lobes, some of which
can be indexed as $\gamma\pm n\delta$ with
$\delta\sim$450\text{\,}\mathrm{T}$$, depending on field angle. At some
angles, a distinct low-frequency peak at the corresponding frequency $\delta$
can also be resolved directly (Fig. 5). Further work will be required to
narrow down the current mass estimate $m^{*}\simeq 8.8\pm
1.8\leavevmode\nobreak\ m_{e}$ obtained at $60^{\circ}$ tilt angle from the
temperature dependence of the directly observed peak intensity (Fig. 5c), and
to track the mass as a function of angle. Because the angle dependence (Fig.
5b) is consistent with a small cylindrical pocket, we interpret $\delta$ as a
signature of the elusive electron pocket D.

The lower part of Table 1 tracks the contributions of the different Fermi
surface pockets to the overall DOS. When the bands are shifted to bring the
DFT pocket volumes into line with QO data, the resulting “QO-corrected” DOS is
dominated by the electron pocket, which contributes nearly half of the total
DOS. Applying a uniform mass renormalization of 4.7 across all pockets A–D, as
suggested by the high-resolution measurements for pockets A–C (Fig. 4) would
account for a Sommerfeld coefficient of about
$76\text{\,}\mathrm{m}\mathrm{J}\mathrm{/}\mathrm{mol}\mathrm{{}^{2}}\mathrm{K}$.
The shortfall of about 20% compared to the experimental value at zero field
may be attributed to a general reduction of the Sommerfeld coefficient in high
magnetic field, as magnetic fluctuations are increasingly suppressed, or we
may have underestimated the contribution of the electron pocket, which is
highly sensitive to details such as band filling and the structural $z$
parameter. This sensitivity is caused by a flat region in the dispersion along
the BZ diagonal, which produces a duckbill outgrowth towards the center of the
BZ. As this feature is quasi-1D (little dispersion within the symmetry plane
of the BZ), its emergence produces a van Hove-like singularity in the DOS
(Fig. 5d). YFe2Ge2 appears to be situated close to the cusp of this anomaly,
similar to the situation of the $\gamma$ sheet in Sr2RuO4 (e.g. [34]). This
sensitivity introduces significant uncertainty into DFT estimates of the
electron pocket DOS contribution.

Figure 5: Electron pocket in YFe2Ge2. (a) Quantum oscillations in the magnetic
torque occur at the frequencies expected for the three hole pockets $\alpha$,
$\beta$, $\gamma$, but the nonlinear response of the cantilever torque sensor
produces side lobes to the $\gamma$-frequency which can be indexed as
$\gamma\pm\delta$, where $\delta\simeq$450\text{\,}\mathrm{T}$$ is attributed
to the electron pocket located in the BZ corner. A peak at the same frequency
$\delta$ is observed at some field angles. (b) Tracking $\delta$ via the side-
lobe frequency (open symbols) and stand-alone peak frequency (closed symbols)
reveals an angle dependence that is consistent with a cylindrical pocket, for
which the cross-sectional area is $\propto 1/\cos(\theta)$. (c) A Lifshitz-
Kosevich fit to the amplitude extracted at an angle of $\simeq 60^{\circ}$
reveals a strongly enhanced carrier mass $m^{*}=8.8\pm 1.8\leavevmode\nobreak\
m_{e}$. (d) The electron pocket DOS peaks sharply at a critical band filling
that depends on $z$, as a duckbill outgrowth appears.

Our findings establish the electronic structure of YFe2Ge2 as consistent with
expectations from DFT calculations, but with strong mass renormalization. Only
less than half of the experimental heat capacity can be attributed to the
three hole pockets, for which the mass renormalization of order 5 can be
comparatively precisely determined (Tab. 1). Attributing the missing heat
capacity to the electron pocket requires a mass enhancement of a similar
magnitude. Torque oscillation data at up to $38\text{\,}\mathrm{T}$ suggest an
enhanced carrier mass which is in line with expectations, depending on the
precise contribution of the unusual duckbill protrusion of the electron
pocket. More detailed measurements will be needed to map out this important
pocket fully. YFe2Ge2 emerges as an unusual iron-based superconductor with a
far more three-dimensional electronic structure than the alkali metal iron
arsenides (K/Rb/Cs)Fe2As2, with which it shares remarkably similar bulk
properties. The observed uniform mass renormalization is reminiscent of the
intensely studied rare-earth-based heavy fermion materials, but it occurs
without their partially filled $f$ states. This suggests an on-site mechanism
like the Kondo effect [35] amplified by Hund’s rule coupling, which forces the
iron $d$ electrons into a high spin state and thereby boosts electronic
correlations [36, 37, 6, 38, 39, 40].

###### Acknowledgements.

We thank, in particular, G. Lonzarich for helpful discussions. The work was
supported by the EPSRC of the UK (grants no. EP/K012894 and EP/P023290/1), by
Trinity College, and by the HFML-RU member of the European Magnetic Field
Laboratory (EMFL).

## References

  * Hosono and Kuroki [2015] H. Hosono and K. Kuroki, Physica C: Superconductivity and its Applications 514, 399 (2015).
  * Si  _et al._ [2016] Q. Si, R. Yu, and E. Abrahams, Nat Rev Mater 1, 16017 (2016).
  * Shibauchi  _et al._ [2014] T. Shibauchi, A. Carrington, and Y. Matsuda, Annu. Rev. Condens. Matter Phys. 5, 113 (2014).
  * Watson  _et al._ [2015] M. D. Watson, T. K. Kim, A. A. Haghighirad, N. R. Davies, A. McCollam, A. Narayanan, S. F. Blake, Y. L. Chen, S. Ghannadzadeh, A. J. Schofield, M. Hoesch, C. Meingast, T. Wolf, and A. I. Coldea, Phys. Rev. B 91, 155106 (2015).
  * Tomczak  _et al._ [2012] J. M. Tomczak, M. van Schilfgaarde, and G. Kotliar, Phys. Rev. Lett. 109, 237010 (2012).
  * Georges  _et al._ [2013] A. Georges, L. de’Medici, and J. Mravlje, Annu. Rev. Condens. Matter Phys. 4, 137 (2013).
  * Zou  _et al._ [2014] Y. Zou, Z. Feng, P. W. Logg, J. Chen, G. Lampronti, and F. M. Grosche, Phys. Status Solidi RRL - Rapid Res. Lett. 8, 928 (2014).
  * Chen  _et al._ [2016] J. Chen, K. Semeniuk, Z. Feng, P. Reiss, P. Brown, Y. Zou, P. W. Logg, G. I. Lampronti, and F. M. Grosche, Phys. Rev. Lett. 116, 127001 (2016).
  * Chen  _et al._ [2019] J. Chen, M. B. Gamża, K. Semeniuk, and F. M. Grosche, Phys. Rev. B 99, 020501(R) (2019).
  * Avila  _et al._ [2004] M. Avila, S. Bud’ko, and P. Canfield, J. Magn. Magn. Mater. 270, 51 (2004).
  * Hardy  _et al._ [2013] F. Hardy, A. E. Böhmer, D. Aoki, P. Burger, T. Wolf, P. Schweiss, R. Heid, P. Adelmann, Y. X. Yao, G. Kotliar, J. Schmalian, and C. Meingast, Phys. Rev. Lett. 111, 027002 (2013).
  * Singh [2014] D. J. Singh, Phys. Rev. B 89, 024505 (2014).
  * Subedi [2014] A. Subedi, Phys. Rev. B 89, 024504 (2014).
  * Coldea  _et al._ [2008] A. I. Coldea, J. D. Fletcher, A. Carrington, J. G. Analytis, A. F. Bangura, J.-H. Chu, A. S. Erickson, I. R. Fisher, N. E. Hussey, and R. D. McDonald, Phys. Rev. Lett. 101, 216402 (2008).
  * Sato  _et al._ [2009] T. Sato, K. Nakayama, Y. Sekiba, P. Richard, Y.-M. Xu, S. Souma, T. Takahashi, G. F. Chen, J. L. Luo, N. L. Wang, and H. Ding, Phys. Rev. Lett. 103, 047002 (2009).
  * Terashima  _et al._ [2013] T. Terashima, N. Kurita, M. Kimata, M. Tomita, S. Tsuchiya, M. Imai, A. Sato, K. Kihou, C.-H. Lee, H. Kito, H. Eisaki, A. Iyo, T. Saito, H. Fukazawa, Y. Kohori, H. Harima, and S. Uji, Phys. Rev. B 87, 224512 (2013).
  * Xu  _et al._ [2016] D. F. Xu, D. W. Shen, D. Zhu, J. Jiang, B. P. Xie, Q. S. Wang, B. Y. Pan, P. Dudin, T. K. Kim, M. Hoesch, J. Zhao, X. G. Wan, and D. L. Feng, Phys. Rev. B 93, 024506 (2016).
  * Sirica  _et al._ [2015] N. Sirica, F. Bondino, S. Nappini, I. Píš, L. Poudel, A. D. Christianson, D. Mandrus, D. J. Singh, and N. Mannella, Phys. Rev. B 91, 121102(R) (2015).
  * Wo  _et al._ [2019] H. Wo, Q. Wang, Y. Shen, X. Zhang, Y. Hao, Y. Feng, S. Shen, Z. He, B. Pan, W. Wang, K. Nakajima, S. Ohira-Kawamura, P. Steffens, M. Boehm, K. Schmalzl, T. R. Forrest, M. Matsuda, Y. Zhao, J. W. Lynn, Z. Yin, and J. Zhao, Phys. Rev. Lett. 122, 217003 (2019).
  * Srpčič  _et al._ [2017] J. Srpčič, P. Jeglič, I. Felner, B. Lv, C. W. Chu, and D. Arčon, Phys. Rev. B 96, 174430 (2017).
  * Kim  _et al._ [2015] H. Kim, S. Ran, E. Mun, H. Hodovanets, M. Tanatar, R. Prozorov, S. Bud’ko, and P. Canfield, Philos. Mag. 95, 804 (2015).
  * Chen  _et al._ [2020] J. Chen, M. B. Gamża, J. Banda, K. Murphy, J. Tarrant, M. Brando, and F. M. Grosche, Phys. Rev. Lett. 125, 237002 (2020).
  * Yan  _et al._ [2017] J.-Q. Yan, B. C. Sales, M. A. Susner, and M. A. Mcguire, Phys. Rev. Mater. 1, 023402 (2017).
  * Shoenberg [2009] D. Shoenberg,  _Magnetic Oscillations in Metals_ , 1st ed. (Cambridge University Press, Cambridge, 2009).
  * Sup [2021] See Supplemental Material at [_URL_], which includes Refs. [33, 24], for further details of QO results in sample S2, the Dingle analysis of the mean free path for both samples S1 and S2, and the torque interaction phenomenon. (2021).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Blaha  _et al._ [2020] P. Blaha, K. Schwarz, F. Tran, R. Laskowski, G. K. H. Madsen, and L. D. Marks, J. Chem. Phys. 152, 074101 (2020).
  * Rourke and Julian [2012] P. Rourke and S. Julian, Computer Physics Communications 183, 324 (2012).
  * Kawamura [2019] M. Kawamura, Computer Physics Communications 239, 197 (2019).
  * Shishido  _et al._ [2010] H. Shishido, A. F. Bangura, A. I. Coldea, S. Tonegawa, K. Hashimoto, S. Kasahara, P. M. C. Rourke, H. Ikeda, T. Terashima, R. Settai, Y. Ōnuki, D. Vignolles, C. Proust, B. Vignolle, A. McCollam, Y. Matsuda, T. Shibauchi, and A. Carrington, Phys. Rev. Lett. 104, 057008 (2010).
  * Zocco  _et al._ [2014] D. A. Zocco, K. Grube, F. Eilers, T. Wolf, and H. v. Löhneysen, JPS Conf. Proc. 3, 015007 (2014).
  * Eilers  _et al._ [2016] F. Eilers, K. Grube, D. A. Zocco, T. Wolf, M. Merz, P. Schweiss, R. Heid, R. Eder, R. Yu, J.-X. Zhu, Q. Si, T. Shibauchi, and H. v. Löhneysen, Phys. Rev. Lett. 116, 237003 (2016).
  * Vanderkooy and Datars [1968] J. Vanderkooy and W. R. Datars, Can. J. Phys. 46, 1215 (1968).
  * Barber  _et al._ [2018] M. E. Barber, A. S. Gibbs, Y. Maeno, A. P. Mackenzie, and C. W. Hicks, Phys. Rev. Lett. 120, 076602 (2018).
  * Wu  _et al._ [2016] Y. P. Wu, D. Zhao, A. F. Wang, N. Z. Wang, Z. J. Xiang, X. G. Luo, T. Wu, and X. H. Chen, Phys. Rev. Lett. 116, 147001 (2016).
  * Haule and Kotliar [2009] K. Haule and G. Kotliar, New J. Phys. 11, 025021 (2009).
  * Yin  _et al._ [2011] Z. P. Yin, K. Haule, and G. Kotliar, Nat. Mater. 10, 932 (2011).
  * de’ Medici  _et al._ [2014] L. de’ Medici, G. Giovannetti, and M. Capone, Phys. Rev. Lett. 112, 177001 (2014).
  * Backes  _et al._ [2015] S. Backes, H. O. Jeschke, and R. Valentí, Phys. Rev. B 92, 195128 (2015).
  * Zhao  _et al._ [2020] D. Zhao, H. L. Wo, J. Li, D. W. Song, L. X. Zheng, S. J. Li, L. P. Nie, X. G. Luo, J. Zhao, T. Wu, and X. H. Chen, Phys. Rev. B 101, 064511 (2020).


# DiffPrune: Neural Network Pruning with Deterministic Approximate Binary
Gates and $L_{0}$ Regularization

Yaniv Shulman<EMAIL_ADDRESS>
###### Abstract

Modern neural network architectures typically have many millions of parameters
and can be pruned significantly without substantial loss in effectiveness
which demonstrates they are over-parameterized. The contribution of this work
is two-fold. The first is a method for approximating a multivariate Bernoulli
random variable by means of a deterministic and differentiable transformation
of any real-valued multivariate random variable. The second is a method for
model selection by element-wise multiplication of parameters with approximate
binary gates that may be computed deterministically or stochastically and take
on exact zero values. Sparsity is encouraged by the inclusion of a surrogate
regularization to the $L_{0}$ loss. Since the method is differentiable it
enables straightforward and efficient learning of model architectures by an
empirical risk minimization procedure with stochastic gradient descent and
theoretically enables conditional computation during training. The method also
supports any arbitrary group sparsity over parameters or activations and
therefore offers a framework for unstructured or flexible structured model
pruning. To conclude experiments are performed to demonstrate the
effectiveness of the proposed approach.

This is a draft version, content may be revisited in subsequent versions.

##  1 Introduction

Modern neural network architectures have achieved state-of-the-art results in
various learning tasks and have the capacity to approximate highly complex
functions. Typically such models have many millions of parameters which are
densely connected. These dense structures enable efficient parallel computing
using specialized software and hardware however they also require substantial
resources to store the models and execute which precludes loading such models
on hardware having modest resources such as low power embedded devices.
Furthermore these large architectures can be pruned significantly without
substantial loss in effectiveness which demonstrates that they are highly
over-parameterized [14]. Due to being over-parameterized they are prone to
overfitting which can lead to poor generalization [32]. Model compression is
an approach that aims to address these aforementioned drawbacks by reducing
the number of parameters while minimizing performance loss to an acceptable
degree. There are many approaches suggested for model pruning or compression
including weight pruning [7], network quantization [6], architecture learning
[27], distilling knowledge [8], structured pruning [30] and $L_{0}$
regularization [16].

Sparse representation is related to the notion that many of the explanatory
factors are irrelevant and therefore are represented by many zeros [3].
Consequntly a particularly useful approach is the structured or unstructured
binary gating of parameters. This approach enables parameters to take on exact
zero values thus enabling conditional computational savings during training
without imposing additional constraints on the parameters [14, 3]. However
many learning algorithms such as neural networks utilize gradient-based
optimizers such as the back-propagation algorithm [25]. Models that are
designed to have a continuous relationship between parameters and the training
objective enable the computation of exact gradients which in turn enable
efficient optimization [3]. The challenge is then combining discrete binary
gates acting on the parameters by means of gate-parameter-wise multiplication
with efficient gradient based optimization. Deterministic discreteness can be
smoothed and approximated reasonably well with the softmax and sigmoid
functions for categorical and binary states respectively. However if a
distribution over discrete states is needed, there is no clear solution [18,
3]. Therefore, the addition of useful continuous approximations to discrete
random variables can be important for the development of new models. A
practical way to overcome the limitations related to including discrete states
in the computation graph is by the inclusion and the optimization of
stochastic nodes and taking samples of their states. A well known method is
the reparameterization of stochastic nodes by deterministic functions of their
parameters and a fixed noise distribution [11, 23].

The main contribution of this work is two-fold. The first is a method for
approximating a multivariate Bernoulli random variable by means of a
deterministic and differentiable transformation of any real-valued
multivariate random variable. The second is a deterministic method for model
selection by element-wise multiplication of parameters with approximate binary
gates that can take on exact zero values. Since the method is differentiable
the gates are learned jointly with the model parameters during training by
means of empirical risk minimization and theoretically offers the benefit of
conditional computation during training. Note the term _differential_ is used
in this paper in the context of training neural networks, i.e. allowing a
small number of points where the first order derivatives do not exist. This is
typically due to the very common use of rectifiers in the calculation graph
such as the ubiquitous Relu activation [21].

##  2 DiffPrune

Let $\boldsymbol{\theta}$ be the parameters of a hypothesis
$h(\cdot,\boldsymbol{\theta})\mathrel{\mathop{\ordinarycolon}}X\rightarrow Y$
such as a neural network and let $\mathcal{D}$ be a training set consisting of
$N$ i.i.d. instances $\\{(x_{1},y_{1}),\ldots,(x_{N},y_{N})\\}$. The empirical
risk $\mathcal{R}$ associated with the regularized hypothesis
$h(\cdot,\boldsymbol{\theta})$ is defined as:

$\displaystyle\mathcal{R}_{h}(\boldsymbol{\theta})=\frac{1}{N}\left(\sum_{i=1}^{N}\mathcal{L}\left(h(x_{i};\boldsymbol{\theta}),y_{i}\right)\right)+\lambda\mathcal{L}_{reg}(\boldsymbol{\theta})$
(1)
$\displaystyle\boldsymbol{\theta}^{*}=\operatorname*{arg\,min}_{\boldsymbol{\theta}}\mathcal{R}_{h}(\boldsymbol{\theta})$
(2)

Where $\mathcal{L}\mathrel{\mathop{\ordinarycolon}}Y\times
Y\rightarrow\mathbb{R}_{\geq 0}$ is a loss function that measures the
discrepancy between the true value $y_{i}$ and the predicted outcome
$\hat{y}_{i}=h(x_{i};\boldsymbol{\theta})$;
$\mathcal{L}_{reg}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{\mathinner{\\!\left\lvert\boldsymbol{\theta}\right\rvert}}\rightarrow\mathbb{R}_{\geq
0}$ is a regularization loss that measures the hypothesis complexity;
$\lambda\in\mathbb{R}_{\geq 0}$ is a scaling factor for the regularization
loss enabling trade-off between model complexity and primary task loss; and
$\mathinner{\\!\left\lvert\boldsymbol{\theta}\right\rvert}$ is the cardinality
of $\boldsymbol{\theta}$. The goal of the optimization learning problem is to
find $\boldsymbol{\theta}^{*}$ given the hypothesis $h$ and data $\mathcal{D}$
for which the empirical risk $\mathcal{R}_{h}(\boldsymbol{\theta})$ is
minimal. The term complexity usually corresponds to a differentiable function
of the model parameters such as the $L_{1}$ and $L_{2}$ regularizers that
impose soft constraints on the parameters.

Consider the hypothesis $h$ and associated empirical risk $\mathcal{R}_{h}$
following re-parametrization of $\boldsymbol{\theta}$ given a partition of
$\boldsymbol{\theta}$ to $M$ subsets
$\boldsymbol{\theta}_{1}\,,\ldots\,,\boldsymbol{\theta}_{M}$:

$\displaystyle\boldsymbol{\theta}=\hat{\boldsymbol{\theta}}\odot\hat{\boldsymbol{\pi}}\,,\quad\hat{\boldsymbol{\theta}}=\bigcup\limits_{j=1}^{M}\boldsymbol{\theta}_{j}\,,\quad\hat{\boldsymbol{\pi}}=\bigcup\limits_{j=1}^{M}\boldsymbol{\pi}_{j}\,,\quad\boldsymbol{\pi}_{j}\in\\{0,1\\}^{\mathinner{\\!\left\lvert\boldsymbol{\theta}_{j}\right\rvert}}\,,\quad\mathinner{\\!\left\lvert\boldsymbol{\theta}_{j}\right\rvert}\geq
2\,,$ (3)
$\displaystyle\mathcal{R}_{h}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\pi}})=\frac{1}{N}\left(\sum_{i=1}^{N}\mathcal{L}\left(h(x_{i};\cup_{j=1}^{M}\boldsymbol{\theta}_{j}\odot\boldsymbol{\pi}_{j}),y_{i}\right)\right)+\sum\limits_{j=1}^{M}\lambda_{j}\mathcal{L}_{reg}(\boldsymbol{\theta}_{j},\boldsymbol{\pi}_{j})$
(4)
$\displaystyle\hat{\boldsymbol{\theta}}^{*},\hat{\boldsymbol{\pi}}^{*}=\operatorname*{arg\,min}_{\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\pi}}}\mathcal{R}_{h}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\pi}})$
(5)

Where $\odot$ is the element-wise Hadamard product. Let $\theta_{jk}$ and
$\pi_{jk}$ denote the $k$-th element of $\boldsymbol{\theta}_{j}$ and
$\boldsymbol{\pi}_{j}$ respectively, then $\pi_{jk}$ enables a binary ”gate”
like function over the corresponding weight $\theta_{jk}$. Unfortunately
$\mathcal{R}_{h}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\pi}})$ is not
differentiable w.r.t. $\hat{\boldsymbol{\pi}}$ due to its binary nature. As an
alternative, the next section describes a deterministic differentiable
relaxation of the binary constraints over the gates that allows for the gates
to take on exact zeros and enables solving a surrogate minimization problem
efficiently and deterministically using common gradient based optimizers.

###  2.1 Approximating a multivariate Bernoulli random variable

To benefit from conditional computation in training it is desirable for
parameters to take on exact zeros. One such way to achieve this is by
utilizing multiplicative gates that can take on exact zero values. Whereas
gates taking on exact zero value is crucial for efficient pruning, it is not
as important for gates to be exactly one when enabled since in inference the
original parameters may be substituted by the multiplication result of the
gate with its corresponding parameter(s). Therefore the hard constraint of
gates being binary may be relaxed and replaced with a soft constraint of being
approximately one when enabled. Let $S\sim q(S|\boldsymbol{\phi})$ be a real-
valued multivariate random variable with parameters $\boldsymbol{\phi}$. And
let $u(\cdot)$ be a differential function from the support of $S$ to the range
$(0,1)^{\mathinner{\\!\left\lvert\mathbf{s}\right\rvert}}$. Equations (6) -
(10) define a deterministic and differentiable transformation $g(\cdot)$ that
transforms samples taken from $S$ to follow an approximate Bernoulli
distribution such that $g(S)=Z\sim\tilde{b}(Z|\boldsymbol{\phi})$.

$\displaystyle\tilde{\mathbf{z}}$
$\displaystyle=\max\left(u(\mathbf{s})-\boldsymbol{\beta}\,,0\right)\,,\qquad\boldsymbol{\beta}\in(0,1)^{\mathinner{\\!\left\lvert\mathbf{s}\right\rvert}}$
(6) $\displaystyle\mathbf{z}^{+}$
$\displaystyle=\\{\tilde{z}_{k}\,|\,\tilde{z}_{k}>0\,,\>k=1,\ldots,\mathinner{\\!\left\lvert\tilde{\mathbf{z}}\right\rvert}\\}$
(7) $\displaystyle\mathbf{z}^{0}$
$\displaystyle=\tilde{\mathbf{z}}-\mathbf{z}^{+}$ (8)
$\displaystyle\mathbf{z}^{1}$
$\displaystyle=\left(\mathbf{z}^{+}-\bar{\mathbf{z}}^{+}\right)*e^{-\zeta}+1\,,\qquad\zeta\in\mathbb{R}_{\geq
0}$ (9) $\displaystyle\mathbf{z}$
$\displaystyle=\mathbf{z}^{0}\cup\mathbf{z}^{1}$ (10)

Where the operator $-$ in equation (8) is the set difference and
$\bar{\mathbf{z}}^{+}$ is the sample mean of $\mathbf{z}^{+}$. The
transformation defined by $g(\cdot)$ conceptually comprises $\mathbf{z}$ of
two partitions: $\mathbf{z}^{0}$ and $\mathbf{z}^{1}$, such that by definition
all $\mathbf{z}^{0}=0$ and assuming that
$\mathinner{\\!\left\lvert\mathbf{z}^{1}\right\rvert}>0$ then
$\operatorname*{\mathbb{E}}(\mathbf{z}^{1})=1$. The variance of
$\mathbf{z}^{1}$ is controlled by $\zeta$ and since $\mathinner{\\!\left\lvert
z_{k}^{+}-z_{l}^{+}\right\rvert}<1$ it may be set as small as practically
useful and can therefore approximate a binary random variable to any finite
practical precision. In addition note that $Z$ becomes exactly binary in the
limit as $\zeta\rightarrow\infty$. It is clear that the distribution
$\tilde{b}(Z|\boldsymbol{\phi})$ depends on the choice of
$q(S|\boldsymbol{\phi}),u(\cdot)$ and $\boldsymbol{\beta}$. It is also notable
that $\tilde{b}(Z|\boldsymbol{\phi})$ is strongly bimodal in the non-
degenerate case even if the distribution $q(S|\boldsymbol{\phi})$ is unimodal.
Finally the gradient of $\mathbf{z}$ w.r.t. $\mathbf{s}$ is non-degenerate
provided that $\mathinner{\\!\left\lvert\mathbf{z}^{1}\right\rvert}\geq 2$
i.e. there are at least two members in $\mathbf{z}^{+}$.

There are many possible choices for $u(\cdot)$ however in this paper the focus
is on two variants. The first is the element-wise $sigmoid(\cdot)$ function
and the second is the $softmax(\cdot)$.

Sigmoid: Let $u\coloneqq sigmoid(\cdot)$, the marginal probability of $z_{k}$
being approximately one is:

$\displaystyle p\left(z_{k}>0\right)$
$\displaystyle=p\left(\frac{1}{1+e^{-s_{k}}}>\beta_{k}\right)=p\left(s_{k}>-\ln{\left(1/\beta_{k}-1\right)}\right)$
(11)
$\displaystyle=1-CDF_{s_{k}}\left(-\ln{\left(1/\beta_{k}-1\right)}\right)$
(12)

Where $CDF_{s_{k}}$ is the marginal cumulative density function of $s_{k}$.
Therefore in the case that $S$ is independent $u\coloneqq sigmoid(\cdot)$
results in the events $z_{k}\approx 1$ being independent, i.e.
$p\left(z_{k}\approx 1,z_{l}\approx 1\right)=p\left(z_{k}\approx
1\right)p\left(z_{l}\approx 1\right)$.

Softmax: Let $u\coloneqq softmax(\cdot)$, the conditional probability of
$z_{k}$ being approximately one given $\\{S_{l}=s_{l}\,|\,l\neq k\\}$ is:

$\displaystyle p\left(z_{k}>0\>|\>\\{S_{l}=s_{l}\,|\,l\neq
k\\}\right)=p\left(e^{s_{k}}/\sum_{l=1}^{\mathinner{\\!\left\lvert
S\right\rvert}}e^{s_{l}}>\beta_{k}\>|\>\\{S_{l}=s_{l}\,|\,l\neq k\\}\right)$
(13)

Rearranging the inequality in equation (13):

$\displaystyle e^{s_{k}}/\sum_{l=1}^{\mathinner{\\!\left\lvert
S\right\rvert}}e^{s_{l}}>\beta_{k}$ (14) $\displaystyle
e^{s_{k}}>\beta_{k}\sum_{l=1}^{\mathinner{\\!\left\lvert
S\right\rvert}}e^{s_{l}}$ (15)
$\displaystyle\left(1-\beta_{k}\right)e^{s_{k}}>\beta_{k}\sum_{l=1,l\neq
k}^{\mathinner{\\!\left\lvert S\right\rvert}}e^{s_{l}}$ (16) $\displaystyle
s_{k}>\ln\left(\frac{\beta_{k}}{1-\beta_{k}}\sum_{l=1,l\neq
k}^{\mathinner{\\!\left\lvert S\right\rvert}}e^{s_{l}}\right)$ (17)

Therefore by combining equations (13) and (17) the conditional probability of
$z_{k}$ being approximately one given $\\{S_{l}=s_{l}\,|\,l\neq k\\}$ is:

$\displaystyle p\left(z_{k}>0\>|\>\\{S_{l}=s_{l}\,|\,l\neq k\\}\right)$
$\displaystyle=p\left(s_{k}>\ln\left(\frac{\beta_{k}}{1-\beta_{k}}\sum_{l=1,l\neq
k}^{\mathinner{\\!\left\lvert S\right\rvert}}e^{s_{l}}\right)\right)$ (18)
$\displaystyle=1-CDF_{s_{k}}\left(\ln\left(\frac{\beta_{k}}{1-\beta_{k}}\sum_{l=1,l\neq
k}^{\mathinner{\\!\left\lvert S\right\rvert}}e^{s_{l}}\right)\right)$ (19)

Note that the choice of $u\coloneqq softmax(\cdot)$ leads to the events
$z_{k}\approx 1$ having inherent dependencies regardless of the independence
of $S$.

The formulation in equations (6) - (10) offers a number of benefits in the
context of pruning which are discussed in subsequent sections. Most
importantly the empirical risk in equation (4) can now be smoothed by
replacing the binary Bernoulli gates $\hat{\boldsymbol{\pi}}$ with the
approximately binary gates $\hat{\mathbf{z}}$ and thus enable the use of
gradient based optimizers:

$\displaystyle\boldsymbol{\theta}=\hat{\boldsymbol{\theta}}\odot\hat{\mathbf{z}}\,,\quad\hat{\boldsymbol{\theta}}=\bigcup\limits_{j=1}^{M}\boldsymbol{\theta}_{j}\,,\quad\hat{\mathbf{z}}=\bigcup\limits_{j=1}^{M}\mathbf{z}_{j}\,,\quad\mathbf{z}_{j}\in\\{0,\tilde{1}\\}^{\mathinner{\\!\left\lvert\boldsymbol{\theta}_{j}\right\rvert}}\,,\quad\mathinner{\\!\left\lvert\boldsymbol{\theta}_{j}\right\rvert}\geq
2\,,\quad Z_{j}\sim\tilde{b}(g(S_{j})\,|\,\boldsymbol{\phi}_{j})$ (20)
$\displaystyle\mathcal{R}_{h}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\phi}})=\frac{1}{N}\left(\sum_{i=1}^{N}\mathcal{L}\left(h(x_{i};\cup_{j=1}^{M}\boldsymbol{\theta}_{j}\odot\mathbf{z}_{j}),y_{i}\right)\right)+\sum\limits_{j=1}^{M}\lambda_{j}\mathcal{L}_{reg}(\boldsymbol{\theta}_{j},\mathbf{z}_{j})$
(21)
$\displaystyle\hat{\boldsymbol{\theta}}^{*},\hat{\boldsymbol{\phi}}^{*}=\operatorname*{arg\,min}_{\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\phi}}}\mathcal{R}_{h}(\hat{\boldsymbol{\theta}},\hat{\boldsymbol{\phi}})$
(22)

###  2.2 Gates

By transforming the samples of continuous distributions that allow for the
reparameterization trick [11, 23] it is possible to express the stochastic
objective in (21) as an expectation over a deterministic differentiable
transformation $f(\cdot)$ of the parameters $\hat{\boldsymbol{\phi}}$ and a
parameter free noise distribution $p(\boldsymbol{\epsilon})$ and perform Monte
Carlo approximation to the intractable expectation over the noise
distribution. Whereas injecting noise to the gradients estimation can be
useful such as in the case of dropout [29], it may also have the undesirable
effect of increasing the variance of the estimator [9]. Furthermore estimation
by means of sampling to produce expectations has the drawback of increasing
the computational requirements. Consequently it is proposed to choose
$q(S|\boldsymbol{\phi})$ such that no sampling is required and the objective
(21) becomes deterministic. Let $S_{j}$ follow an isotropic normal
distribution $\mathcal{N}(\boldsymbol{\mu}_{j},\sigma^{2}I)$, it is thus
possible to perform a deterministic and differentiable maximum likelihood
estimation of the gates values simply by computing
$\mathbf{z}_{j}=g(\boldsymbol{\phi}_{j})=g(\boldsymbol{\mu}_{j})$. Note that
$\boldsymbol{\mu}_{j}$ are nuisance parameters that are only required during
training and may be pruned with the corresponding model parameter(s).

It is perhaps useful to provide an intuitive interpretation of $g(\cdot)$ in
the context of computing gates over the model parameters. One way of many to
select which gates are enabled is to rely on the notion of relative ”utility”
of the parameters in $\boldsymbol{\theta}_{j}$ and to rank them based on the
probability that a parameter $\theta_{jk}$ is the most ”useful” or ”important”
in the set. In the case where $u(\cdot)\coloneqq softmax(\cdot)$ the random
variable $S_{j}$ is then interpreted as the logits of a categorical random
variable denoting which of the elements of $\boldsymbol{\theta}_{j}$ has the
”most utility”. In the case where $u(\cdot)\coloneqq sigmoid(\cdot)$ the
random variable $S_{j}$ is interpreted as the logits of a multivariate
Bernoulli random variable denoting the importance probability for each of the
weights independently. In this context, for simplicity,
$\boldsymbol{\beta}_{j}$ can be reduced to a scaler and act as a
hyperparameter specifying the probability selection threshold.

Given the choice of $S\sim\mathcal{N}(\boldsymbol{\mu},\sigma^{2}I)$ and due
to using deterministic maximum likelihood sampling it follows that:

$\displaystyle
p\left(z_{k}>0\right)=1-CDF_{\mathcal{N}(\mu_{k},\sigma^{2})}\left(-\ln{\left(1/\beta-1\right)}\right)$
(23)

and:

$\displaystyle p\left(z_{k}>0\right)$
$\displaystyle=1-CDF_{\mathcal{N}(\mu_{k},\sigma^{2})}\left(\ln\left(\frac{\beta}{1-\beta}\sum_{l=1,l\neq
k}^{\mathinner{\\!\left\lvert\boldsymbol{\mu}\right\rvert}}e^{\mu_{l}}\right)\right)$
(24)

for when $u(\cdot)\coloneqq sigmoid(\cdot)$ and $u(\cdot)\coloneqq
softmax(\cdot)$ respectively. Note the edge case resulting in degenerate
gradients when there is at most one active gate in a partition i.e.
$\mathinner{\\!\left\lvert\mathbf{z}^{1}\right\rvert}\leq 1$. However in
practice the need to prune all parameters but one is unlikely and therefore
this limitation is not expected to cause issues in application of this method.

###  2.3 Regularization

If no regularization is performed, i.e. only the the first term in equation
(21) is present, the gates will converge to some minima in respect to the
primary objective with no additional restrictions. This is useful for
configuration free model selection that is likely to retain the cumbersome
model’s accuracy. However in many cases it is desirable to encourage the model
to achieve a higher sparsity with the trade-off of lower accuracy by the
introduction of regularization loss. Note that for simplifying the notation in
the remainder of this section it is assumed that the parameters are not
partitioned and therefore the partition subscript is omitted.

$\mathbf{L_{0}}$ regularizer: The $L_{0}$ loss is simply the number of
parameters of the model that are different from zero. This regularisation loss
has desirable attributes in the context of pruning since it ”punishes” the
model for having non-zero parameters with no further restrictions and does not
induce shrinkage. Assuming that $p(w_{k}=0)=0$ and due to
$\operatorname*{\mathbb{E}}(\mathbf{z}^{1})=1$ it is possible to compute the
$L_{0}$ loss simply as $L_{0}(\mathbf{z})=\sum\mathbf{z}$. Unfortunately this
will yield zero gradients w.r.t. $\boldsymbol{\mu}$, or w.r.t. each
$\boldsymbol{\mu}_{j}$ in the case where the parameters are partitioned.
Instead the approach suggested in [16] is adopted and rather than directly
computing the number of enabled gates the expected number of enabled gates,
whether sampling or not, is used as a surrogate loss. The expected $L_{0}$
loss over the parameters is obtained by summing the marginal probability or
marginal conditional probability in the case of $u(\cdot)\coloneqq
softmax(\cdot)$ of every gate being enabled:

$\displaystyle\mathcal{L}_{reg}(\mathbf{z})=\sum_{k=1}^{\mathinner{\\!\left\lvert
Z\right\rvert}}p\left(z_{k}>0\right)$ (25)

Or in the case where the parameters of the model are partitioned:

$\displaystyle\mathcal{L}_{reg}(\mathbf{z})=\sum_{j=1}^{M}\sum_{k=1}^{\mathinner{\\!\left\lvert
Z_{j}\right\rvert}}p\left(z_{jk}>0\right)$ (26)

Gaussian dropout: Experimental results discussed in subsequent sections
demonstrate that applying Gaussian dropout [29] to $\boldsymbol{\mu}$ has a
substantial effect on the results, both the compression rate and the overall
accuracy of the model. From a practical perspective injecting noise has the
side effect of mild stochastic binary dropout due to gates being randomly
enabled and disabled. From a theoretical point of view the product of the two
random variables, $\mathcal{N}(\boldsymbol{\mu},\sigma^{2}I)$ and
$\mathcal{N}(1,\sqrt{p/(1-p)})$ does not follow a multivariate normal
distribution however since in practice the ML sampling yields a constant
$\boldsymbol{\mu}$ the resulting noise follows an uncorrelated multivariate
normal distribution where the standard deviation of each of the components is
proportional to the absolute value of its mean.

###  2.4 Group sparsity

In many applications it is desired to enforce group pruning of parameters
since this can enable more substantial computational savings in practice with
existing hardware and software libraries [30]. A common scenario is the
pruning of the activations computed by a layer, e.g. neuron pruning. Other
applications require jointly pruning groups of activations such as when
performing filter pruning of a CNN [13]. DiffPrune can be used to this extent
by virtue of associating a gate in $\mathbf{z}$ with a group of parameters or
activations. Note the cardinality of nuisance parameters
$\mathinner{\\!\left\lvert\boldsymbol{\mu}\right\rvert}$ is equal to the
number of gates which is typically substantially less than required for
unstructured parameter pruning.

###  2.5 Initialization and configuration

Partitioning and sparsity: The proposed method enables arbitrary pruning
schemes in respect to any combination of parameters, groups of parameters,
activations and groups of activations. Furthermore by including the
regularization loss in the objective different target compression rates for
each partition can be specified by configuration of the parameters
$\boldsymbol{\lambda}$. Larger values of $\lambda_{j}$ will accelerate pruning
by balancing the magnitude of the $L_{0}$ regularization terms against the
model’s primary loss. Both the partition scheme and the values of the
hyperparameters are dependent on the objectives of the practitioner, the data
and the architecture.

Initializing $\boldsymbol{\mu}_{j}$: Let $\boldsymbol{\mu}_{j}^{(0)}$ denote
the initial value of $\boldsymbol{\mu}_{j}$. $\boldsymbol{\mu}_{j}^{(0)}$ can
be considered as defining a priori probability distribution on the ”utility”
of weights in $\boldsymbol{\theta}_{j}$ and for certain values can result in
pruning of parameters pre-training. Typically through application of the
principle of indifference a uniform distribution is the default choice in
cases such as this unless there is a particular reason to do otherwise.
However for effective backpropagation it is required that the parameters
$\boldsymbol{\mu}_{j}$ are initialized randomly and have both positive and
negative values. Therefore it is recommended for $\boldsymbol{\mu}_{j}^{(0)}$
to have a mean of zero and small variance so that it is close to being uniform
and works well in the context of the SGD optimization framework. In
experiments done as part of this work a truncated normal distribution with a
standard deviation of $0.05$ is used.

Selecting $\beta_{j}$: Effectively acting as a pruning selection threshold
where parameters that have a utility probability not greater than $\beta_{j}$
are pruned. Let $\mathbf{p}_{j}^{(0)}=u(\boldsymbol{\mu}_{j}^{(0)})$. Given no
a priori preference as to which parameters are to be pruned it is desirable to
select $\beta_{j}$ such that $p_{jk}-\beta_{j}>0\,,\>\forall
p_{jk}\in\mathbf{p}_{j}^{(0)}$ as this leads to no parameters being pruned
pre-training. Therefore $\beta_{j}$ can be selected heuristically during the
initialization stage without the need to configure it by the practitioner. In
theory it can be said that lower values of $\beta_{j}$ provide the model the
opportunity to train for longer before pruning commences. In experiments done
in this work values of $\beta_{j}=0.99\min(\mathbf{p}_{j}^{(0)})$ are used.

Learning $\zeta_{j}$: Since $\mathinner{\\!\left\lvert
z_{jk}^{+}-z_{jl}^{+}\right\rvert}<1$ it is possible in theory to reduce the
variance of the enabled gates $\mathbf{z}_{j}^{1}$ to an arbitrary value by
increasing $\zeta_{j}$. In this work however $\zeta_{j}$ is learned together
with the other parameters of the model.

Dropout rate: The Gaussian dropout rate is reparameterized as
$\sqrt{\sigma(\eta_{j})/(1-\sigma(\eta_{j}))}$ where $\eta_{j}\in\mathbb{R}$
can be learned jointly with the other parameters of the model and
$\sigma(\cdot)$ is the sigmoid function. To enable backpropagation for
training $\eta_{j}$ through the sampling stage gradient estimates are obtained
by a single sample Monte Carlo and the reparameterization trick [11, 23].

##  3 Related work

Methods for approximating binary discrete random variables that can represent
exact zero by rectifications of continuous random variables have been
suggested before in various contexts. These include rectifying samples taken
from the logistic distribution by a Relu for conditional computation [3]
however the resulting distribution is not strongly bi-modal and therefore
cannot represent effectively binary random variables. The hard concrete
distribution proposed in [16] can emulate binary variables more closely by
hard clipping a stretched concrete random variable [18, 10]. This approach can
represent both exact zero and one however has some drawbacks, the gradients
suffer from high-variance estimates; it is limited only to its own Concrete
distribution and does not generalize to other multivariate distributions [9].
Furthermore it is continuous in $[0,1]$ and the hard rectifiers on both
extremes substantially increase the sparsity of gradients during
backpropagation. Expanding on this framework is the work of [9] where any
mixture pair of distributions converging to $\delta(0)$ and $\delta(1)$ can be
used to construct smoothed binary gates. By contrast the method proposed here
can transform any underlying distribution including unimodal distributions
into practically discrete binary random variables and maintain the
tractability of continuous optimization. Furthermore due to the use of the
$softmax(\cdot)$ function that creates dependency between the components of
the emulated binary random variable even components that are zero are affected
by gradient updates and do not suffer to the same extent from gradient ”dead
zone”. A different approach is proposed in [24] for gradient based
optimization of discrete random variables by smoothing transformations of
binary latent variables with continuous noise in a way that allows for
gradients reparameterization.

Many methods have been proposed for compression and sparsification of neural
networks during the last three decades. Due to the large volume of work it is
not possible to mention the absolute majority of methods and instead the
reader is referred to [5, 4] for a comprehensive review. Most methods proposed
in the literature follow a three stage procedure where (i) an over-
parameterized large network is trained to partial or full convergence; (ii)
following the training redundant weights or neurons are pruned based on
heuristics; (iii) the network is trained again possibly from scratch to ”fine-
tune” the model [14, 7, 20, 17]. This procedure may repeat a number of times
to iteratively increase the model compression. One major shortcoming of many
of the methods that follow this three stage procedure is they require training
the over specified network to convergence which precludes conditional
computational savings during training [14, 3].

Similar to the method here in the context of network pruning is the $L_{0}$
regularization for neural networks proposed in [16, 9] that prunes the network
during training by the introduction of smooth Bernoulli stochastic gates.
These frameworks however resort to Monte Carlo sampling to estimate the
posterior probability distribution over the parameters. Other related methods
are proposed in [28, 26], however the main difference and drawback of these
methods is that they are non-differential and use the high variance straight-
through estimator [3] to compute the gradients of the gate distribution
parameters. A rather differnet method that performs online pruning while
training is proposed in [33] where magnitude based heuristics determine which
weights to prune.

##  4 Experiments

It is generaly not feasible to perform exact comparisons between published
methods due to the discrepancies related to differing frameworks, partial
information, pre-processing pipelines and stochastic aspects [4]. However to
demonstrate the effectiveness of the proposed method an attempt is made to
compare it with the results published in [16, 9] using similar architectures,
datasets and preprocessing. The models are implemented in TensorFlow [2] using
standard Dense and Conv2D layers with additional custom DiffPrune layers added
between the standard layers to perform neuron and group pruning on the
activations of the standard layers. Conditional computation is not performed
in practice however an indicative number of FLOPs per training step is
recorded and used to demonstrate potential theoretical benefits. All image
datasets were taken from TensorFlow Datasets [1] with the default train/test
split. The source code is publicly available at
https://bitbucket.org/YanivShu/diffprune_public. Note there was no attempt to
perform an exhaustive search of hyperparameters for the best possible result
therefore these results should be taken as indicative only.

###  4.1 LeNet5 MNIST classification

The first experiment is the toy classification task of MNIST using the basic
CNN LeNet5 [12] with an original architecture of 20-50-800-500. The results
for two variants of DiffPrune are reported in table 1. The first has no
regularization loss and $\boldsymbol{\eta}$ is trained with $\eta^{(0)}=0$ for
all partitions with $u\coloneqq softmax(\cdot)$ and thus demonstrates the
capabilities of the method to perform configuration free ”no fuss” model
selection during training. The second has a regularization loss term added to
the objective as per equation (26) with $u\coloneqq sigmoid(\cdot)$,
$\boldsymbol{\lambda}=\\{10^{-5},10^{-5},2*10^{-5},2*10^{-5}\\}$ and
$\boldsymbol{\eta}$ was trained with $\eta^{(0)}\approx-1.734$ for all
partitions. Training was performed for 200 epochs with testing following the
completion of each epoch, of which a representative result is reported. The
batch size was 100 and optimization is performed using Adam with default
parameters and a learning rate of $0.0005$.

Method | Pruned Arch. | Error %
---|---|---
Sparse VD [19] | 14-19-242-131 | 1.0
GL [30] | 3-12-192-500 | 1.0
GD [26] | 7-13-208-16 | 1.1
SBP [22] | 3-18-284-283 | 0.9
BC-GNJ [15] | 8-13-88-13 | 1.0
BC-GHS [15] | 5-10-76-16 | 1.0
$L_{0}$ [16] | 20-25-45-462 | 0.9
$L_{0,\,\lambda\text{sep}}$ [16] | 9-18-65-25 | 1.0
MDR-ExpUni [9] | 6-8-72-31 | 1.0
DiffPrune | 11-30-496-309 | 0.6
DiffPrunereg | 10-20-71-35 | 1.0
Table 1: Comparison of architecture learned and error rates by methods
compared in [16, 9] and DiffPrune on the MNIST classification task using
LeNet5. The DiffPrunereg variant has a regularization loss term incorporated
to the objective.

###  4.2 Wide Residual Network CIFAR10 classification

The second experiment is the classification task of the CIFAR10 dataset using
a pruned Wide Residual Network [31]. The baseline architecture is the
WRN-28-10 with no dropout and an attempt is made to maintain as closely as
possible the training procedue and settings described in [31] with some minor
differences to the data augmentation pipeline and a smaller batch size of 120.
Filter pruning is applied in all groups excluding group conv1, and in all
blocks on the hidden Conv2D layer as well as jointly on the outputs of the
second Conv2D layer and the skip connection convolution layer. Applying
pruning jointly to both convolution layers at the same activation plane
results in the same activation dimensions and enables conditional computation
benefits. In practice this is implemented by adding a DiffPrune layer to prune
the addition results of the convolution layers activations. The DiffPrune
layers are all configured with $u\coloneqq sigmoid(\cdot)$ and to enable a
fair comparison no Gaussian or other dropout was applied.

Two results are reported varying in their regularization settings. The first
result demonstrates pruning with no accuracy loss and the second demonstrates
more aggressive regularization with little loss of accuracy. These two
variants are referred to as ”low compression” and ”high compression”
respectively.

For the low compression variant the error rate achieved on the test set was
the same as reported in [31] of $\approx 0.42\%$. The DiffPrune layers in all
blocks are configured with
$\boldsymbol{\lambda}=\\{10^{-5},10^{-5},5*10^{-5}\\}$ for each of the groups
conv2, conv3 and conv4 respectively. The pruned architecture learned is:
16-[(133,122)-(148,105)-(137,101)-(106,58)]-[(301,255)-(316,248)-(310,240)-(318,214)]-[(618,543)-(624,578)-(620,600)-(615,635)].

The high compression variant achieves an error rate of $\approx 4.5\%$. The
DiffPrune layers in all blocks are configured with
$\boldsymbol{\lambda}=\\{2*10^{-5},7*10^{-5},3*10^{-4}\\}$ for each of the
groups conv2, conv3 and conv4 respectively. The pruned architecture learned
during training has substantially less enabled filters in deeper layers of the
network:
16-[(127,120)-(137,112)-(139,102)-(113,61)]-[(286,230)-(296,235)-(289,238)-(304,231)]-[(148,180)-(162,165)-(167,143)-(153,169)].

(a)

(b)

Figure 1: (a) Accuracy measured on the test set for the CIFAR10 classification
task with the original and pruned WRN-28-10 wide residual network. (b) The
number of estimated FLOPs for the forward pass of a mini-batch .

##  5 Conclusion

This paper proposes a novel distribution that approximates efficiently and
effectively the multivariate Bernoulli distribution by employing a
deterministic transformation to random variables with no restrictions on the
underlying distribution. By choosing a continuous underlying distribution that
allows for efficient ML estimation or the reparameterization trick it better
emulates the binary nature of Bernoulli distributions while enabling the use
of efficient gradient based optimization in deterministic or stochastic
computation graphs respectively. Furthermore a deterministic framework for
model selection with approximate binary gates and exact gradients was proposed
which can prune models with or without regularization whilst maintaining the
error rate of the original cumbersome model. Furthermore it is demonstrated
that this method allows for deterministic optimization of the $L_{0}$ norm of
parametric models by smoothing the combinatorial problem with the proposed
approximate Bernoulli distribution. Experiments demonstrate that the proposed
method can competitively sparsify neural networks in comparison to current
approaches while theoretically allowing for conditional computation in
training.

## References

  * [1] TensorFlow Datasets, a collection of ready-to-use datasets.  https://www.tensorflow.org/datasets.
  * [2] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * [3] Y. Bengio, N. Léonard, and A. C. Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  CoRR, abs/1308.3432, 2013.
  * [4] D. W. Blalock, J. J. G. Ortiz, J. Frankle, and J. V. Guttag.  What is the state of neural network pruning?  In I. S. Dhillon, D. S. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020.
  * [5] Y. Cheng, D. Wang, P. Zhou, and T. Zhang.  A survey of model compression and acceleration for deep neural networks.  CoRR, abs/1710.09282, 2017.
  * [6] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, 2016.
  * [7] S. Han, J. Pool, J. Tran, and W. J. Dally.  Learning both weights and connections for efficient neural networks.  In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1135–1143, Cambridge, MA, USA, 2015. MIT Press.
  * [8] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  In NIPS Deep Learning and Representation Learning Workshop, 2015\.
  * [9] C.-T. Huang, J.-C. Chen, and J.-L. Wu.  Learning sparse neural networks through mixture-distributed regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [10] E. Jang, S. Gu, and B. Poole.  Categorical reparameterization with gumbel-softmax.  2017\.
  * [11] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [12] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  In Proceedings of the IEEE, pages 2278–2324, 1998.
  * [13] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf.  Pruning filters for efficient convnets.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [14] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell.  Rethinking the value of network pruning.  In International Conference on Learning Representations, 2019.
  * [15] C. Louizos, K. Ullrich, and M. Welling.  Bayesian compression for deep learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3288–3298. Curran Associates, Inc., 2017.
  * [16] C. Louizos, M. Welling, and D. P. Kingma.  Learning sparse neural networks through ${L}_{0}$ regularization.  In International Conference on Learning Representations, 2018.
  * [17] J. Luo, J. Wu, and W. Lin.  Thinet: A filter level pruning method for deep neural network compression.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5068–5076. IEEE Computer Society, 2017.
  * [18] C. J. Maddison, A. Mnih, and Y. W. Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  2016\.  cite arxiv:1611.00712.
  * [19] D. Molchanov, A. Ashukha, and D. Vetrov.  Variational dropout sparsifies deep neural networks.  volume 70 of Proceedings of Machine Learning Research, pages 2498–2507, International Convention Centre, Sydney, Australia, 06–11 Aug 2017\. PMLR.
  * [20] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz.  Pruning convolutional neural networks for resource efficient inference.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [21] V. Nair and G. E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In J. Fürnkranz and T. Joachims, editors, ICML, pages 807–814. Omnipress, 2010.
  * [22] K. Neklyudov, D. Molchanov, A. Ashukha, and D. P. Vetrov.  Structured bayesian pruning via log-normal multiplicative noise.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6775–6784. Curran Associates, Inc., 2017.
  * [23] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  volume 32 of Proceedings of Machine Learning Research, pages 1278–1286, Bejing, China, 22–24 Jun 2014. PMLR.
  * [24] J. T. Rolfe.  Discrete variational autoencoders, 2017.
  * [25] D. E. Rumelhart, G. E. Hinton, and R. J. Williams.  Learning Representations by Back-propagating Errors.  Nature, 323(6088):533–536, 1986.
  * [26] S. Srinivas and R. V. Babu.  Generalized dropout, 2016.
  * [27] S. Srinivas and R. V. Babu.  Learning neural network architectures using backpropagation.  In R. C. Wilson, E. R. Hancock, and W. A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016.
  * [28] S. Srinivas, A. Subramanya, and R. V. Babu.  Training sparse neural networks, 2016.
  * [29] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15(1):1929–1958, Jan. 2014.
  * [30] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li.  Learning structured sparsity in deep neural networks.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2074–2082. Curran Associates, Inc., 2016.
  * [31] S. Zagoruyko and N. Komodakis.  Wide residual networks.  In E. R. H. Richard C. Wilson and W. A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1–87.12. BMVA Press, September 2016.
  * [32] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals.  Understanding deep learning requires rethinking generalization.  2017\.
  * [33] M. Zhu and S. Gupta.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  CoRR, abs/1710.01878, 2017.