$\hat{\mathcal{O}}(w_{0})=\sum_{j}\left[(\alpha_{j}^{+})^{\dagger}\alpha_{j}^{+}-(\alpha_{j}^{-})^{\dagger}\alpha_{j}^{-}\right]\,.$
(B.25)

The state $\bra{\mathtt{APS}}$ is an eigenstate of this operator, with the
eigenvalue being the $\eta$-invariant of $K$ on $Y$,

$\displaystyle\begin{split}\bra{\mathtt{APS}}\hat{\mathcal{O}}(w_{0})&=\bra{\mathtt{APS}}\left(\sum_{\xi_{j}\geq
0}1-\sum_{\xi_{j}<0}1\right)\\\
&=\bra{\mathtt{APS}}2\eta_{K}(Y)\,.\end{split}$ (B.26)

Thus, we obtain

$\frac{\braket{\mathtt{APS}}{\hat{W}_{1}(m_{0}e^{\mathrm{i}\alpha_{0}})e^{-\frac{\mathrm{i}}{2}\alpha_{0}\hat{\mathcal{O}}(w_{0})}}{W_{0}}}{\braket{\mathtt{APS}}{\hat{W}_{1}(m_{0})}{W_{0}}}=\exp\left(-\mathrm{i}\alpha_{0}\eta_{K}(Y)\right)$
(B.27)

in the limit $m_{0}\to 0$.

Finally, we would like to compute
$\exp\left(\mathrm{i}\text{Tr}(\alpha_{0}\theta(w_{0}-w)\tau^{3})\right)$.
This quantity is analogous to (the exponential of) $\text{Tr}(\gamma^{5})$
appearing in the Fujikawa’s computation of an axial anomaly [Fuj79, Fuj84],
and can be computed by a similar heat kernel regularization method. Since
$\alpha_{0}\theta(w_{0}-w)$ vanishes near the boundary, such a local
computation is still applicable even in the precense of the boundary [KY21].
The computation of
$\exp\left(\mathrm{i}\text{Tr}(\alpha_{0}\theta(w_{0}-w)\tau^{3})\right)$ was
performed in [KS21] using the Fujikawa method. We will simply adopt their
result here:

$\displaystyle\begin{split}\exp\left(\mathrm{i}\text{Tr}(\alpha_{0}\theta(w_{0}-w)\tau^{3})\right)&=\lim_{\Lambda\to\infty}\int_{W_{0}}\mathop{\mathrm{ch}}\nolimits(\mathcal{F})\hat{A}(TW)\\\
&=\lim_{\Lambda\to\infty}\int_{W}\mathop{\mathrm{ch}}\nolimits(\mathcal{F})\hat{A}(TW)\,.\end{split}$
(B.28)

The field strength $\mathcal{F}$ for the superconnection (of odd type since
$d+2$ is odd) and the Chern character
$\mathop{\mathrm{ch}}\nolimits(\mathcal{F})$ were defined in Section 3.1.
Recall the field strength depends on the mass parameters as
$\mathcal{F}=\mathcal{F}(\widetilde{M}=M/\Lambda)$. The cutoff scale $\Lambda$
is introduced through the heat kernel regularization in the Fujikawa method.
Here, it is important that on the right-hand side, we should first evaluate
the integral and then take the limit $\Lambda\to\infty$. The second equality
sign follows from the fact that
$\mathop{\mathrm{ch}}\nolimits(\mathcal{F})\hat{A}(TW)$ vanishes near the
boundary due to the conditions imposed on the collar neighborhood of the
boundary.

Combining Eq. (B.21), Eq. (B.27), and Eq. (B.28), we see that

$\frac{\mathcal{Z}_{d+2}[m_{0}e^{\mathrm{i}\alpha_{0}}]}{\mathcal{Z}_{d+2}[m_{0}]}=\exp\left(\mathrm{i}\alpha_{0}\operatorname{Ind}Q\right)=\exp\left[\mathrm{i}\alpha_{0}\left(-\eta_{K}(Y)+\lim_{\Lambda\to\infty}\int_{W}\mathop{\mathrm{ch}}\nolimits(\mathcal{F})\hat{A}(TW)\right)\right]$
(B.29)

for $m_{0}\to 0$. Since $\alpha_{0}$ is arbitrary, we conclude that

$\eta_{K}(Y)=-\operatorname{Ind}Q+\lim_{\Lambda\to\infty}\int_{W}\mathop{\mathrm{ch}}\nolimits(\mathcal{F})\hat{A}(TW)\,.$
(B.30)

This is the desired APS index theorem.

The same result still holds when $d$ is even, with the appropriately modified
definition of $K$ as in Eq. (B.1) and using the superconnection of even type
from Section 3.1. We will only briefly sketch the derivation for the even $d$
case since all the steps are analogous to the odd $d$ case. Again, we have
$2N_{f}$ species of $(d+2)$-dimensional Dirac fermions $\Upphi$ on a
$(d+2)$-dimensional manifold $W$ with the boundary $\partial W=Y$. The
$(d+2)$-dimensional Dirac operator in the bulk $W$ is given by

$\mathsf{D}_{d+2}=\hat{\Gamma}^{\hat{\mu}}D_{\hat{\mu}}+m_{1}+\mathrm{i}\hat{\Gamma}(m_{2}\tau^{3}+M_{1}\tau^{1}+M_{2}\tau^{2})$
(B.31)

where $\hat{\Gamma}$ is the chirality matrix in $(d+2)$-dimensions which is
now even, $m_{1}$ and $m_{2}$ are real constants, and $M_{1}$ and $M_{2}$ are
$N_{f}\times N_{f}$ hermitian matrix valued modulated masses. On the boundary
$\partial W$, $M_{1}$ and $M_{2}$ reduces to those defining the operator $K$
in Eq. (B.1). As before, we write $m_{1}+im_{2}=m_{0}e^{i\alpha_{0}}$. We may
write
$\mathsf{D}_{d+2}=(-\mathrm{i}Q+m_{0}e^{\mathrm{i}\alpha_{0}\hat{\Gamma}\tau^{3}})$
where

$Q=\mathrm{i}\hat{\Gamma}^{\hat{\mu}}D_{\hat{\mu}}-\hat{\Gamma}(M_{1}\tau^{1}+M_{2}\tau^{2})\,.$
(B.32)

$Q$ again is a hermitian operator on $W$ with appropriately chosen boundary
conditions. The aim is to compute the index of this $Q$. The
$\mathbb{Z}_{2}$-grading that is necessary to define the index in this case is
now given by the matrix $\tau^{3}\hat{\Gamma}$ which acts both on the flavor
and the spinor indices, and it anticommutes with $Q$. The rest of the steps
are essentially the same as the odd $d$ case, and we will not repeat them
here.

##  Appendix C Explicit torus partition functions

In this appendix, we rederive Eq. (4.8) from Section 4.2.1 by explicitly
computing the torus partition functions of the 2d Dirac fermion with the
modulated complex mass parameter. We will consider a slightly more general
configuration where the winding number of the mass term along the $a$-cycle is
an arbitrary integer $\tilde{n}$. If we write the angular coordinate along the
$a$-cycle as $\theta_{a}$, then we have
$\alpha(\theta_{a})=\tilde{n}\theta_{a}$.

We compute the partition function by interpreting the $b$-cycle as the
compactified Euclidean time direction. In the Minkowski signature, the
canoncial quantization on $S^{1}_{a}$ can be done by the expansion

$\psi(t,x)=\sum_{n\in\mathbb{Z}+1/2}\left[a_{n}\phi_{n}^{a}(x)e^{-\mathrm{i}E_{n}^{a}t}+b_{n}\phi_{n}^{b}(x)e^{-\mathrm{i}E_{n}^{b}t}\right]\,,$
(C.1)

where the explicit forms of the functions $\phi_{n}^{a},\,\phi_{n}^{b}$ are
not important. The energies are given by252525In this appendix, we let the
coordinates on the torus to be dimensionless (and $2\pi$ periodic), and thus
all the energies and the mass parameters are also dimensionless.

$\begin{cases}E_{n}^{a}&=\frac{1}{2}\tilde{n}+\frac{1}{2}\sqrt{(\tilde{n}-2n)^{2}+4M_{0}^{2}}\,,\\\
E_{n}^{b}&=\frac{1}{2}\tilde{n}-\frac{1}{2}\sqrt{(\tilde{n}-2n)^{2}+4M_{0}^{2}}\,,\end{cases}$
(C.2)

and in the limit $M_{0}\to\infty$, we have $|E_{n}^{i}|\to\infty$ for all $n$
and both $i=a,b$.

For a single fermionic mode with energy $E$, we pick a regularization scheme
such that the thermal partition function $\text{Tr}[(-1)^{F}e^{-\beta H}]$ for
this mode becomes

$\lim_{|E|\to\infty}\mathcal{Z}_{1}[E]=\begin{cases}1&\text{if}\,\,E>0\,,\\\
-1&\text{if}\,\,E<0\,.\end{cases}$ (C.3)

In the infinite energy limit, the partition function receives a contribution
only from the ground state, and we are setting the ground state energy to be
zero. The sign of the partition function corresponds to the assignment of the
fermion number to the ground state. The sign difference for the two cases is
to account for the level crossing at $E=0$.

Then, the 2d partition function in the infinite mass limit becomes

$\lim_{M_{0}\to\infty}\mathcal{Z}_{2}(T^{2})=\prod_{E_{n}^{i}}\left(\lim_{|E_{n}^{i}|\to\infty}\mathcal{Z}_{1}[E_{n}^{i}]\right)$
(C.4)

which formally corresponds to $(-1)^{\chi}$ where $\chi$ is the number of
negative energy eigenvalues. The number of negative energy eigenvalues for
large $M_{0}$ is infinite, and the partition function is not well-defined
without an additional regularization.

We perform the regularization in the following way. Inside the parameter space
$\widetilde{P}=\mathbb{C}$ including the degenerate point, we can continuously
deform the mass parameter and change its winding number. In such a process,
the energy spectrum will continuously vary, and some of the energy levels
cross the zero energy. Each time an energy level crosses the zero energy, it
changes the sign of the partition function. Consider continuously changing the
winding number from $\tilde{n}$ to zero, first by setting $M_{0}$ to zero and
increasing it again while keeping $\alpha=0$. If the number of energy levels
that cross zero in the process is $N$, then the regularized partition function
becomes

$\lim_{M_{0}\to\infty}\frac{\mathcal{Z}_{2}(T^{2})}{\mathcal{Z}^{(0)}_{2}(T^{2})}=(-1)^{N}\,.$
(C.5)

The remaining task is to compute $N$. At the end of the deformation process,
the energy spectrum is given by

$E=\pm\sqrt{n^{2}+M_{0}^{2}}$ (C.6)

with large $M_{0}$, whereas at the beginning of the process we have Eq. (C.2),
again with large $M_{0}$. We have to glue the two spectra across $M_{0}=0$,
and count the number of energy levels which cross the zero energy. When we
start from the spectrum (C.6) with sufficiently large $M_{0}$, all the
$E_{n}^{a}$’s are positive whereas all the $E_{n}^{b}$’s are negative. As we
decrease $M_{0}$, $E_{n}^{a}$’s always remain positive down to $M_{0}=0$, but
some of the $E_{n}^{b}$’s which were originally negative become positive or
zero at $M_{0}=0$, depending on $\tilde{n}$ and $n$. Now, from $M_{0}=0$, we
evolve the spectrum again by increasing $M_{0}$, according to (C.6). A feature
of the spectrum (C.6) is that, if the energy $E$ was positive (negative) at
$M_{0}=0$, it always remains positive (negative) all the way to
$M_{0}\to\infty$.

From this, we deduce that the number of energy levels that cross the zero
energy while following the spectral flow of energy eigenvalues across the
deformation process is equal to the number of positive $E_{n}^{b}$’s at
$M_{0}=0$. The number of $n$’s satisfying this condition is $N=\tilde{n}$.
Thus,

$\lim_{M_{0}\to\infty}\frac{\mathcal{Z}_{2}(T^{2})}{\mathcal{Z}^{(0)}_{2}(T^{2})}=(-1)^{\tilde{n}}\,,$
(C.7)

and the special case $\tilde{n}=1$ confirms Eq. (4.8). See Figure 5 for an
examplary spectal flow of the energy spectrum across the deformation process.

Figure 5: The spectral flow for the case of $\tilde{n}=5$. The vertical axis
corresponds to the energy, and from the left to the right the deformation of
the mass parameter occurs. The stand-alone red crosses indicate non-degenerate
energy eigenvalues, whereas the red crosses that are overlaid with blue dots
indicate doubly-degenerate energy eigenvalues. The horizontal black dashed
line corresponds to the zero energy. We see that the number of zero energy
crossings across the spectral flow is indeed equal to $\tilde{n}=5$.

## References

  * [AH20] A. Angelescu and P. Huang, _Integrating Out New Fermions at One Loop_ , JHEP 01 (2021) 049, arXiv:2006.16532 [hep-ph].
  * [AIO00] M. Alishahiha, H. Ita, and Y. Oz, _On superconnections and the tachyon effective action_ , Phys. Lett. B 503 (2001) 181–188, arXiv:hep-th/0012222.
  * [APS75a] M. F. Atiyah, V. K. Patodi, and I. M. Singer, _Spectral asymmetry and Riemannian geometry. II_ , Math. Proc. Cambridge Phil. Soc. 78 (1975) 405.
  * [APS75b] , _Spectral asymmetry and Riemannian Geometry. I_ , Math. Proc. Cambridge Phil. Soc. 77 (1975) 43.
  * [APS76] , _Spectral asymmetry and Riemannian geometry. III_ , Math. Proc. Cambridge Phil. Soc. 79 (1976) 71–99.
  * [AW99] A. G. Abanov and P. B. Wiegmann, _Theta terms in nonlinear sigma models_ , Nucl. Phys. B 570 (2000) 685–698, arXiv:hep-th/9911025.
  * [AWH22] D. Aasen, Z. Wang, and M. B. Hastings, _Adiabatic paths of Hamiltonians, symmetries of topological order, and automorphism codes_ , arXiv:2203.11137 [quant-ph].
  * [Ber84] M. V. Berry, _Quantal phase factors accompanying adiabatic changes_ , Proc. Roy. Soc. Lond. A 392 (1984) 45–57.
  * [BGV92] N. Berline, E. Getzler, and M. Vergne, _Heat kernels and dirac operators_ , Die Grundlehren der mathematischen Wissenschaften in Einzeldarstellungen, Springer-Verlag, 1992. https://books.google.com/books?id=vRc-QgAACAAJ.
  * [CFLS19a] C. Córdova, D. S. Freed, H. T. Lam, and N. Seiberg, _Anomalies in the Space of Coupling Constants and Their Dynamical Applications I_ , SciPost Phys. 8 (2020) 001, arXiv:1905.09315 [hep-th].
  * [CFLS19b] , _Anomalies in the Space of Coupling Constants and Their Dynamical Applications II_ , SciPost Phys. 8 (2020) 002, arXiv:1905.13361 [hep-th].
  * [CJH85] C. G. Callan Jr and J. A. Harvey, _Anomalies and fermion zero modes on strings and domain walls_ , Nuclear Physics B 250 (1985) 427–436.
  * [EQV+20] S. A. R. Ellis, J. Quevillon, P. N. H. Vuong, T. You, and Z. Zhang, _The Fermionic Universal One-Loop Effective Action_ , JHEP 11 (2020) 078, arXiv:2006.16260 [hep-ph].
  * [FFM+20a] H. Fukaya, M. Furuta, S. Matsuo, T. Onogi, S. Yamaguchi, and M. Yamashita, _A physicist-friendly reformulation of the Atiyah-Patodi-Singer index and its mathematical justification_ , PoS LATTICE2019 (2019) 061, arXiv:2001.01428 [hep-lat].
  * [FFM+20b] H. Fukaya, M. Furuta, Y. Matsuki, S. Matsuo, T. Onogi, S. Yamaguchi, and M. Yamashita, _Mod-two APS index and domain-wall fermion_ , arXiv:2012.03543 [hep-th].
  * [FH16] D. S. Freed and M. J. Hopkins, _Reflection positivity and invertible topological phases_ , Geom. Topol. 25 (2021) 1165–1330, arXiv:1604.06527 [hep-th].
  * [FKS17] D. S. Freed, Z. Komargodski, and N. Seiberg, _The Sum Over Topological Sectors and $\theta$ in the 2+1-Dimensional $\mathbb{C}\mathbb{P}^{1}$ $\sigma$-Model_, Commun. Math. Phys. 362 (2018) 167–183, arXiv:1707.05448 [cond-mat.str-el].
  * [FOY17] H. Fukaya, T. Onogi, and S. Yamaguchi, _Atiyah-Patodi-Singer index from the domain-wall fermion Dirac operator_ , Phys. Rev. D 96 (2017) 125004, arXiv:1710.03379 [hep-th].
  * [Fuj79] K. Fujikawa, _Path Integral Measure for Gauge Invariant Fermion Theories_ , Phys. Rev. Lett. 42 (1979) 1195–1198.
  * [Fuj84] , _On the Evaluation of Chiral Anomaly in Gauge Theories with Gamma(5) Couplings_ , Phys. Rev. D 29 (1984) 285.
  * [GJF17] D. Gaiotto and T. Johnson-Freyd, _Symmetry Protected Topological phases and Generalized Cohomology_ , JHEP 05 (2019) 007, arXiv:1712.07950 [hep-th].
  * [GKKS17] D. Gaiotto, A. Kapustin, Z. Komargodski, and N. Seiberg, _Theta, Time Reversal, and Temperature_ , JHEP 05 (2017) 091, arXiv:1703.00501 [hep-th].
  * [GKSW14] D. Gaiotto, A. Kapustin, N. Seiberg, and B. Willett, _Generalized Global Symmetries_ , JHEP 02 (2015) 172, arXiv:1412.5148 [hep-th].
  * [GL20] P. Gorantla and H. T. Lam, _Interface junctions in QCD 4_, SciPost Phys. 10 (2021) 085, arXiv:2004.13300 [hep-th].
  * [GW81] J. Goldstone and F. Wilczek, _Fractional Quantum Numbers on Solitons_ , Phys. Rev. Lett. 47 (1981) 986–989.
  * [GY21] K. Gomi and M. Yamashita, _Differential $KO$-theory via gradations and mass terms_, arXiv:2111.01377 [math.KT].
  * [HHN+19] Y. Hidaka, Y. Hirono, M. Nitta, Y. Tanizaki, and R. Yokokura, _Topological order in the color-flavor locked phase of a ( 3+1 )-dimensional U(N) gauge-Higgs system_ , Phys. Rev. D 100 (2019) 125016, arXiv:1903.06389 [hep-th].
  * [HHY21] S.-J. Huang, C.-T. Hsieh, and J. Yu, _Effective field theories of topological crystalline insulators and topological crystals_ , Phys. Rev. B 105 (2022) 045112, arXiv:2107.03409 [cond-mat.str-el].
  * [HKT20] P.-S. Hsin, A. Kapustin, and R. Thorngren, _Berry Phase in Quantum Field Theory: Diabolical Points and Boundary Phenomena_ , arXiv:2004.10758 [cond-mat.str-el].
  * [HT20] M. Honda and Y. Tanizaki, _Topological aspects of 4D Abelian lattice gauge theories with the $\theta$ parameter_, JHEP 12 (2020) 154, arXiv:2009.10183 [hep-th].
  * [JR76] R. Jackiw and C. Rebbi, _Solitons with fermion number $1/2$_, Physical Review D 13 (1976) 3398.
  * [Kah11] A. Kahle, _Superconnections and index theory_ , Journal of Geometry and Physics 61 (2011) 1601–1624.
  * [Kap14] A. Kapustin, _Symmetry Protected Topological Phases, Anomalies, and Cobordisms: Beyond Group Cohomology_ , arXiv:1403.1467 [cond-mat.str-el].
  * [Kit09] A. Kitaev, _Periodic table for topological insulators and superconductors_ , AIP Conf. Proc. 1134 (2009) 22–30, arXiv:0901.2686 [cond-mat.mes-hall].
  * [Kit13] A. Kitaev, _On the classification of short-range entangled states_ , 2013\. http://scgp.stonybrook.edu/video_portal/video.php?id=2010. Talk at Simons Center.
  * [Kit15] , _Homotopy-theoretic approach to spt phases in action: Z16 classification of three- dimensional superconductors_ , 2015. http://www.ipam.ucla.edu/programs/workshops/symmetry-and-topology-in-quantum-matter/?tab=schedule. Symmetry and Topology in Quantum Matter Workshop, Institute for Pure and Applied Mathematics, University of California.
  * [KK19] A. Karasik and Z. Komargodski, _The Bi-Fundamental Gauge Theory in 3+1 Dimensions: The Vacuum Structure and a Cascade_ , JHEP 05 (2019) 144, arXiv:1904.09551 [hep-th].
  * [KL00] P. Kraus and F. Larsen, _Boundary string field theory of the D anti-D system_ , Phys. Rev. D 63 (2001) 106004, arXiv:hep-th/0012198.
  * [KS20a] A. Kapustin and L. Spodyneiko, _Higher-dimensional generalizations of Berry curvature_ , Phys. Rev. B 101 (2020) 235130, arXiv:2001.03454 [cond-mat.str-el].
  * [KS20b] , _Higher-dimensional generalizations of the Thouless charge pump_ , arXiv:2003.09519 [cond-mat.str-el].
  * [KS21] H. Kanno and S. Sugimoto, _Anomaly and Superconnection_ , arXiv:2106.01591 [hep-th].
  * [KS22] A. Kapustin and N. Sopenko, _Local Noether theorem for quantum lattice systems and topological invariants of gapped states_ , arXiv:2201.01327 [math-ph].
  * [KSTZ17] Z. Komargodski, A. Sharon, R. Thorngren, and X. Zhou, _Comments on Abelian Higgs Models and Persistent Order_ , SciPost Phys. 6 (2019) 003, arXiv:1705.04786 [hep-th].
  * [KT17] Y. Kikuchi and Y. Tanizaki, _Global inconsistency, ’t Hooft anomaly, and level crossing in quantum mechanics_ , PTEP 2017 (2017) 113B05, arXiv:1708.01962 [hep-th].
  * [KTTW14] A. Kapustin, R. Thorngren, A. Turzillo, and Z. Wang, _Fermionic Symmetry Protected Topological Phases and Cobordisms_ , JHEP 12 (2015) 052, arXiv:1406.7329 [cond-mat.str-el].
  * [KW99] C. Kennedy and A. Wilkins, _Ramond-Ramond couplings on Brane - anti-Brane systems_ , Phys. Lett. B 464 (1999) 206–212, arXiv:hep-th/9905195.
  * [KY21] S. K. Kobayashi and K. Yonekura, _Atiyah-Patodi-Singer index theorem from axial anomaly_ , arXiv:2103.10654 [hep-th].
  * [LOT21] Y. Lee, K. Ohmori, and Y. Tachikawa, _Matching higher symmetries across Intriligator-Seiberg duality_ , JHEP 10 (2021) 114, arXiv:2108.05369 [hep-th].
  * [OY21] T. Onogi and T. Yoda, _Comments on the Atiyah-Patodi-Singer index theorem, domain wall, and Berry phase_ , JHEP 12 (2021) 096, arXiv:2109.08274 [hep-th].
  * [QSV21] J. Quevillon, C. Smith, and P. N. H. Vuong, _Axion Effective Action_ , arXiv:2112.00553 [hep-ph].
  * [Qui85] D. Quillen, _Superconnections and the chern character_ , Topology 24 (1985) 89–95.
  * [RSFL10] S. Ryu, A. P. Schnyder, A. Furusaki, and A. W. Ludwig, _Topological insulators and superconductors: tenfold way and dimensional hierarchy_ , New Journal of Physics 12 (2010) 065010.
  * [Sha20] A. Sharon, _Global Aspects of Spaces of Vacua_ , JHEP 11 (2020) 083, arXiv:2004.11182 [hep-th].
  * [Shi21] K. Shiozaki, _On adiabatic cycles of quantum spin systems_ , arXiv:2110.10665 [cond-mat.str-el].
  * [STU20] T. Sulejmanpasic, Y. Tanizaki, and M. Ünsal, _Universality between vector-like and chiral quiver gauge theories: Anomalies and domain walls_ , JHEP 06 (2020) 173, arXiv:2004.10328 [hep-th].
  * [Tho14] R. Thorngren, _Framed Wilson Operators, Fermionic Strings, and Gravitational Anomaly in 4d_ , JHEP 02 (2015) 152, arXiv:1404.4385 [hep-th].
  * [Tho17] , _Topological Terms and Phases of Sigma Models_ , arXiv:1710.02545 [cond-mat.str-el].
  * [TK10] J. C. Y. Teo and C. L. Kane, _Topological Defects and Gapless Modes in Insulators and Superconductors_ , Phys. Rev. B 82 (2010) 115120, arXiv:1006.0690 [cond-mat.mes-hall].
  * [TK17] Y. Tanizaki and Y. Kikuchi, _Vacuum structure of bifundamental gauge theories at finite topological angles_ , JHEP 06 (2017) 102, arXiv:1705.01949 [hep-th].
  * [TS18] Y. Tanizaki and T. Sulejmanpasic, _Anomaly and global inconsistency matching: $\theta$-angles, $SU(3)/U(1)^{2}$ nonlinear sigma model, $SU(3)$ chains and its generalizations_, Phys. Rev. B 98 (2018) 115126, arXiv:1805.11423 [cond-mat.str-el].
  * [TTU00] T. Takayanagi, S. Terashima, and T. Uesugi, _Brane - anti-brane action from boundary string field theory_ , JHEP 03 (2001) 019, arXiv:hep-th/0012210.
  * [TU22] Y. Tanizaki and M. Ünsal, _Center vortex and confinement in Yang-Mills theory and QCD with anomaly-preserving compactifications_ , PTEP 2022 (2022) 04, arXiv:2201.06166 [hep-th].
  * [Ü20] M. Ünsal, _Strongly coupled QFT dynamics via TQFT coupling_ , JHEP 11 (2021) 134, arXiv:2007.03880 [hep-th].
  * [WQB+21] X. Wen, M. Qi, A. Beaudry, J. Moreno, M. J. Pflaum, D. Spiegel, A. Vishwanath, and M. Hermele, _Flow of (higher) berry curvature and bulk-boundary correspondence in parametrized quantum systems_ , arXiv preprint arXiv:2112.07748 (2021) .
  * [WWW18] J. Wang, X.-G. Wen, and E. Witten, _A New SU(2) Anomaly_ , J. Math. Phys. 60 (2019) 052301, arXiv:1810.00844 [hep-th].
  * [WY19] E. Witten and K. Yonekura, _Anomaly Inflow and the $\eta$-Invariant_, The Shoucheng Zhang Memorial Workshop, 9 2019. arXiv:1909.08775 [hep-th].
  * [Yam21] M. Yamashita, _Differential models for the Anderson dual to bordism theories and invertible QFT’s, II_ , arXiv:2110.14828 [math.AT].
  * [Yon16] K. Yonekura, _Dai-Freed theorem and topological phases of matter_ , JHEP 09 (2016) 022, arXiv:1607.01873 [hep-th].
  * [Yon18] , _On the cobordism classification of symmetry protected topological phases_ , Commun. Math. Phys. 368 (2019) 1121–1173, arXiv:1803.10796 [hep-th].
  * [YY21] M. Yamashita and K. Yonekura, _Differential models for the Anderson dual to bordism theories and invertible QFT’s, I_ , arXiv:2106.09270 [math.AT].


# Training Multi-bit Quantized and Binarized Networks withA Learnable
Symmetric Quantizer

Phuoc Pham
Incheon National Univeristy
<EMAIL_ADDRESS>Jacob A. Abraham
University of Texas at Austin
<EMAIL_ADDRESS>Jaeyong Chung
Incheon National Univeristy
<EMAIL_ADDRESS>
###### Abstract

Quantizing weights and activations of deep neural networks is essential for
deploying them in resource-constrained devices, or cloud platforms for at-
scale services. While binarization is a special case of quantization, this
extreme case often leads to several training difficulties, and necessitates
specialized models and training methods. As a result, recent quantization
methods do not provide binarization, thus losing the most resource-efficient
option, and quantized and binarized networks have been distinct research
areas. We examine binarization difficulties in a quantization framework and
find that all we need to enable the binary training are a symmetric quantizer,
good initialization, and careful hyperparameter selection. These techniques
also lead to substantial improvements in multi-bit quantization. We
demonstrate our unified quantization framework, denoted as UniQ, on the
ImageNet dataset with various architectures such as ResNet-18,-34 and
MobileNetV2. For multi-bit quantization, UniQ outperforms existing methods to
achieve the state-of-the-art accuracy. In binarization, the achieved accuracy
is comparable to existing state-of-the-art methods even without modifying the
original architectures.

##  1 Introduction

Figure 1: The proposed unified quantization method outperforms the existing
state-of-the-art methods both in binarization and multi-bit quantization by
significant margins.

Deep neural networks have achieved tremendous success in various fields
including computer vision [31], natural language processing [50], and speech
recognition [8], having demonstrated unprecedented predictive performance.
However, the computational complexity and memory access count required by the
existing models pose a challenge in deploying them to resource-constrained
devices, and applying to latency-critical services. To address this challenge,
efficient network architectures, manually designed [44] or automatically
searched [46], and model compression techniques such as pruning [17] and
quantization [5, 24, 45, 54] have been studied. In practical model deployment,
quantization is a necessary step, and often the last means to control the
performance and efficiency trade-off once the model architecture is fixed
[24].

Quantizing weights and activations to a lower precision not only reduces the
computational complexity but also the model size, memory footprint, and memory
access count but at cost of degraded performance. Recent quantization methods
[11, 29] can overcome accuracy degradation from their full-precision
counterparts even with 4-bit weights and activations. However, most of these
methods do not present 1-bit results, possibly due to their severely degraded
performance or because 1-bit training does not converge, ditching the most
efficient option.

While Zhou _et al_. [55] considered binarized neural networks together with
multi-bit quantization, binarized neural networks have been a distinct
research topic from the quantized models. Most studies on binarized models [7,
23, 36, 42] focused on the 1-bit case solely. Binarized networks have gained
attention because of the expensive floating-point multiplications and
additions being replaced by efficient xnor and popcount operations. However,
this is not specific to binarized networks, and any low-precision networks can
be executed efficiently by such operations as shown in [55]. Thus, a binary
kernel can execute any bit-width networks seamlessly depending on the accuracy
and efficiency requirements.

In contrast, the process of building binarized networks is different from
building quantized networks. Binarized network researchers often modify a base
architecture to improve performance. Several studies increased the
representation capacity by using more weight and activation bases [34, 56].
Most studies incorporate changes to improve training efficiency such as dual
skip connections [20, 36, 53]. In addition, more aggressive changes are sought
for via neural architecture search [27, 39, 53]. In addition to model changes,
binarized networks use a quantizer specialized for binary such as the sign
function.

The distinct creation process for binarized networks causes several
difficulties in practice. First, modifications of a base model often affect
the number of floating-point operations and memory access count [10]. Thus,
the actual latency and power consumption can differ from the expected results,
and a binarized model cannot be guaranteed to be better than the 2-bit
counterpart. Second, training binarized networks requires additional skilled
workforce because it needs special expertise, especially a deep understanding
of the model itself. Finally, given a model, binarization provides only one
option for accuracy and efficiency, and the limited exploration of the trade-
off may lead to a suboptimal solution.

Figure 2: The overview about our method. A unified framework is used for both
model quantization and binarization.

This paper proposes a unified quantization framework, denoted as UniQ, for
multi-bit quantized networks and binarized networks. UniQ achieves up to 4.8%
and 2.7% accuracy improvements on the ImageNet dataset over the existing
state-of-the-art quantization and binarization methods, respectively, as shown
in Figure 1. Figure 2 illustrates UniQ in contrast to the conventional
approach.

This paper first considers two popular weight quantizers in quantized
networks. A weight quantizer, first proposed in [55] and adopted in [4, 26]
later, maps weight values into the range [0, 1] first and then performs
quantization and re-maps the quantized values to the range [-1,1]. This method
implies the importance of the symmetry; however, the weight mapping becomes an
impediment to taking advantage of pre-trained models. The other weight
quantizer, used in [2, 11], maps inputs to a real value represented by the
product of a scaling factor and a signed integer. The signed integer is
assumed to be represented by two’s complement, which has asymmetric ranges.
This quantization method does not transform the weights of the pre-trained
models. However, we hypothesize that the asymmetry has a negative impact on
extremely low-precision training. Thus, we design a symmetric quantizer, where
the step size can be learned via the gradient descent procedure.

While the step size is learnable with the task loss, we observe that the
initialization of the learnable parameter has a significant impact on the
final solution. For the initialization of this parameter, prior works used a
heuristic [11] or a numerical method [2]. In contrast, we propose an analytic
initialization method that is optimal in the mean squared sense. According to
our ablation study, our proposed framework shows significant improvements over
prior works as a combined result of the symmetric quantizer and the optimal
initialization.

The proposed quantizer and init method can be applied to the binary case
seamlessly, but the significant improvements demonstrated in multi-bit are not
shown in 1-bit. We scrutinize the training dynamics of the binary case and
find that the binary case receives strong gradient signals at the beginning of
training and the distribution of quantizer input changes extremely fast
compared to multi-bit cases. We hypothesize that this difference is caused as
the initial point after binarization is too far from the pre-trained model
solution. As a simple yet effective solution, we suggest to use warm-up
strategy, which has been used widely for large-batch training. However, we
empirically show that in 1-bit training, this improves accuracy substantially
even when a small batch size is used.

Our major contributions are summarized as follows:

  * •

In multi-bit quantization, the proposed unified method outperforms existing
methods consistently to achieve the state-of-the-art accuracy of ResNet-18,-34
and MobileNetV2 on ImageNet.

  * •

In binarization, the proposed method achieves comparable results to the state-
of-the-art methods. These results have been achieved only by enhancing the
training process without modifying the original network architectures, meaning
that our method can be used in conjunction with network modification
techniques.

  * •

We propose an optimal, analytic initialization for step sizes.

##  2 Related Work

Modern neural networks have increased their computational complexity and
memory requirements. Therefore, recent works have proposed efficient
architectures [21, 22, 44, 47] and model compression techniques such as
network binarization [7, 23, 36, 42], low-bit quantization [5, 24, 45, 54],
and knowledge distillation [29, 40] to reduce the model size and amount of
computation. Among these, low-bit quantization is one of the most popular
methods and is widely used in the research literatures and real-life
applications [6, 35, 49].

Efficient Models. Recently optimized networks such as EfficientNet [47],
MobileNet-v1 [22], -v2 [44], -v3 [21] have achieved high accuracy by replacing
the standard convolutional layers with depth-wise separable convolutions,
thereby significantly reducing the number of parameters. Even for such
efficient architectures, quantization is necessary to deploy them in
specialized hardware [13] and provides further reductions in its size and
number of calculations. Recent works [16, 25, 29] attempted to quantize these
models, but at the expense of the significant loss in prediction accuracy.
Compared to DSQ [16] and QKD [29], our method yields consistently higher
results for all bit-widths when tested with MobileNet-V2, which again,
demonstrates its effectiveness even for highly optimized networks.

Model Binarization. As a special case of quantization, model binarization has
been studied extensively and has received much attention owing to its
efficiency for deployment in edge devices. Using binarized weights and
activations resulted in $\sim$32× memory saving over the full-precision
counterpart and brought $\sim$58× computational efficiency on CPUs by taking
advantage of bitwise operations [42]. Unfortunately, these networks usually
lead to severe accuracy degradation. To mitigate this problem, many existing
methods proposed the idea of modifying the original architecture. In [42], the
order of layers within a block was changed to improve the information flow. In
[36], an additional skip connection was added to each block in the residual
networks. In [28], the input and output widths of each layer were adjusted.
Rather than modifying the original architecture, this study focuses only on
improving the quantizer itself and the training process. Our proposed method
is orthogonal to the aforementioned model modification methods.

Multi-bit Quantization. In contrast, recent works on quantization [5, 12, 14,
48, 51, 54] have achieved substantial efficiency improvements without the need
to re-design or develop the whole new architecture. Thus, it can substantially
reduce the design effort. PACT [5] and LQ-Nets [54] first proposed the idea of
learning quantizer parameters. LQ-Net parameterizes quantization levels for a
non-uniform quantizer. In QIL [14], a non-uniform quantizer was constructed
using a non-linear transformer followed by a uniform quantizer. While these
non-uniform quantizers provide a higher degree of freedom, they usually
require more computation and memory than uniform quantizers. PACT [5]
parameterized the clipping level in a uniform activation quantizer. LSQ [11]
showed better accuracy by making the step size learnable. SAT [11] studied
efficient training for quantized networks. Both PACT [5] and SAT [5] adopted
the weight quantizer of DoReFaNet [55], which is symmetric but transforms the
weights into a new range. This makes it difficult to take advantage of pre-
trained models. In contrast, we design a uniform, symmetric quantizer that
does not require the transformation and can leverage pre-trained models fully.
Besides, none of the recent methods reports their results for binarized neural
networks, due to severe performance degradation, or their quantizers are not
properly designed to support binarized networks. In contrast, we propose a
unified framework that can support all bit-widths including 1-bit
binarization. We obtain new state-of-the-art results for multi-bit
quantization and promising results for model binarization by using the new
quantizer design without any modifications in the original architecture
needed.

Knowledge Distillation. Another popular method is knowledge distillation that
is widely used in many computer vision tasks. The basic idea is that the
knowledge from the teacher networks is transferred to the student networks,
providing an additional guidance signal to the training process of the
smaller-sized student network. Applying distillation methods to low-precision
networks was performed by [29, 38, 40, 52] where a real-valued network is used
as the teacher and a low-precision bit network as a student. QKD [29] reported
competitive results in multi-bit quantization using knowledge distillation.
LSQ [11] also showed that knowledge distillation provided additional
improvements in their quantization results. However, we outperform these
methods even without resorting to the idea of transferring knowledge, by
focusing more on the fundamental issues. In addition, our method is orthogonal
to knowledge distillation, and can be used in conjunction to further boost the
performance.

##  3 Preliminaries

In this section, we first review the weight quantizers commonly used in the
literature. In [2, 11, 32], a weight $w\in\mathbb{R}$ is approximately
represented by

$w\approx\hat{w}\cdot\Delta$ (1)

where $\Delta$ is a scaling factor, called the step size, and $\hat{w}$ is a
signed integer, which is assumed to be represented by two’s complement. This
is often referred to as the fixed-point representation. This representation
has asymmetric ranges; it can represent one more negative number than positive
numbers. For example, when $\Delta=1$, it can represent -2, -1, 0, and 1 for
2-bit weights. We hypothesize that this asymmetry has a negative impact on
low-precision training. While it has an asymmetric range in the strict sense,
it is considered symmetric in [2]. Thus, to avoid confusion, we refer to this
as semi-symmetric.

In [4, 26, 55], a weight $w\in\mathbb{R}$ is quantized into a $k$-bit value by

$2\text{ Quantize
}_{\mathrm{k}}\left(\frac{\tanh\left(w\right)}{2\max\left(\left|\tanh\left(w\right)\right|\right)}+\frac{1}{2}\right)-1$
(2)

where $\text{ Quantize }_{\mathrm{k}}(x)=\text{round}(x(2^{k}-1))/(2^{k}-1)$.
In this quantizer, the weights are first mapped into the range [0,1] and
quantized into a $k$-bit value in the range [-1,1]. This quantizer has the
symmetric property, but is problematic for two reasons. First, it transforms
the weights into a new range and loses the knowledge of pretrained models.
Second, the transformed range may cause the vanishing or exploding gradient
problem because the variance of weights becomes substantially different from
that suggested in Xavier [15] or Kaiming initialization [18]. Thus, SAT [26]
proposed to scale the transformed weights again using the constant in Xavier
initialization. However, the first problem remains. We address these two
problems by using a new symmetric quantizer and optimal initialization that
minimizes the mean square quantization error.

##  4 Unified Quantization

###  4.1 Learnable Symmetric Quantizer

A quantizer $Q:\mathbb{R}\rightarrow\mathbb{R}$ is a piecewise constant
function and each interval is mapped to a corresponding output. The end points
of the intervals are referred to as decision levels and the output is called
the reconstruction level. A uniform quantizer has evenly spaced decision
levels and reconstruction levels. The length of the intervals is called the
step size, denoted as $\Delta$. The total number of reconstruction levels is
denoted by $N$. We denote the clip function to be used by quantizers by
$\text{clip}_{N}(x)=\min(\max(x,0),N-1)$. The round function is denoted by
$\lfloor\cdot\rceil$. In practice, $N$ is usually even, and thus a symmetric
quantizer in the strict sense do not include the value of zero as a
reconstruction level. For example, when $N=4$ and $\Delta=1$, the symmetric
quantizer has -1.5, -0.5, 0.5, and 1.5 as the reconstruction levels. We
quantize weights by the uniform symmetric quantizer

$Q_{w}(x)=\lfloor\text{clip}_{N}\left(\left(x+\alpha\right)/\Delta\right)\rceil\Delta-\alpha$
(3)

where $\alpha=\Delta\cdot(N-1)/2$. Eq. (3) can be rewritten as

$Q_{w}(x)=\hat{q}_{w}(\Delta/2)$ (4)

where
$\hat{q}_{w}=2\lfloor\text{clip}_{N}\left((x+\alpha)/\Delta\right)\rceil-N+1$;
$\hat{q}_{w}$ can be encoded into $\lceil\log_{2}{N}\rceil$ bits using $\pm 1$
encoding.

We consider the ReLU non-linearity, wihch is widely used in the deep learning
literature, as the activation function. Because almost half of the ReLU
responses are zero, we fix the zero value as a reconstruction level instead of
parameterizing an offset. Thus, for activations, we use

$Q_{a}(x)=\lfloor\text{clip}_{N}(x/\Delta)\rceil\Delta.$ (5)

Designing a uniform quantizer usually boils down to deciding one parameter,
the step size $\Delta$. Instead of designing the quantizers manually, we make
$\Delta$ a learnable parameter as in recent prior works [11, 2], and optimize
it with the task loss via the gradient descent procedure. The round function
has a zero derivative almost everywhere, and the exact derivative is not
useful in learning. Thus, we adopt the straight-through estimator (STE), which
assumes a unit derivative for the entire input range of the round function.
Then, we have

$\frac{\partial
Q_{w}}{\partial\Delta}=\left\\{\begin{array}[]{ll}-\frac{x}{\Delta}+\lfloor\frac{x}{\Delta}-\frac{1}{2}\rceil+\frac{1}{2}&\text{
if }|x|<\alpha\\\ \text{sign}(x)\alpha&\text{ otherwise. }\\\
\end{array}\right.$ (6)

We can also find $\frac{\partial Q_{a}}{\partial\Delta}$ similarly. While this
allows us to learn the step size, the initialization of this parameter is
necessary and in our experience, careful initialization improves accuracy
substantially.

###  4.2 Optimal MSE Initialization

The quantized networks are usually initialized with a pre-trained model, and
the learnable step sizes are also initialized depending on the statistics of
the pre-trained model. Let $X$ be the random variable for a quantizer input
and its pdf is denoted by $p(x)$. The optimal step size for $Q_{w}$ is defined
in the mean squared error (MSE) sense by

$\Delta^{*}_{w}=\operatorname*{arg\,min}_{\Delta}D_{w}(\Delta)$ (7)

where $D_{w}(\Delta)=\mathop{\mathbb{E}}[(x-Q_{w}(x;\Delta))^{2}]$. Using
Leibniz integral rule, we take the derivative of $D_{w}$ and obtain

$\displaystyle\frac{dD_{w}}{d\Delta}=-\sum_{i=1}^{\frac{N}{2}-1}(2i-1)\int_{(i-1)\Delta}^{i\Delta}2\left(x-\left[\frac{2i-1}{2}\right]\Delta\right)p(x)dx$
(8)
$\displaystyle-(N-1)\int_{(\frac{N}{2}-1)\Delta}^{\infty}2\left(x-\left[\frac{N-1}{2}\right]\Delta\right)p(x)dx.$

By setting (8) to zero, we can find the optimal step size. In general, this
equation does not have a closed-form solution and a numerical method is
required. However, for common probability distributions such as Gaussian and
Laplace, the step size for each $N$ of interest can be pre-computed assuming a
unit variance, and can be scaled by the standard deviation of the quantizer
input. In our implementation, a Gaussian distribution is assumed for weights.

For the activation quantizer $Q_{a}$, we also define the optimal step size
$\Delta_{a}^{*}$ and the mean squared error $D_{a}$, and derive
$\frac{dD_{a}}{d_{\Delta}}$ similarly. However, in the case of the activation
quantizer, a Gaussian distribution is assumed for pre-activations (activations
prior to the non-linearity). The activations after the ReLU non-linearity
follow a rectified Gaussian, a modification of Gaussian where the negative
elements are reset to zero. While a rectified Gaussian is a mixture of a
discrete distribution for zero and a continuous distribution for the positive
elements, we pre-compute the step size using the continuous part only because
$Q_{a}$ is designed to include zero as a reconstruction level by construction.
When we pre-compute the step sizes, the standard Gaussian is assumed for pre-
activations. We denote the pre-computed step sizes for activations and weights
by $\Delta^{u}_{a}$ and $\Delta^{u}_{w}$, respectively.

Table 1: The optimal unit step size and optimal SQNR for weights and activations. $N$ | Weight | Activation
---|---|---
$\Delta^{u}_{w}$ | SQNR( dB) | $\Delta^{u}_{a}$ | SQNR( dB)
2 | 1.596 | 4.4 | 1.224 | 5.5
4 | 0.996 | 9.3 | 0.651 | 11.6
8 | 0.586 | 14.3 | 0.353 | 17.2
16 | 0.335 | 19.4 | 0.193 | 22.7
|  |  |  |

The constants for each $N$ are summarized in Table 1, which also shows the
optimal signal-to-quantization-noise (SQNR) ratio. We use it later to analyze
the training dynamics of quantized models. Even if the step size is set
optimally, the MSE is proportional to the signal energy (the variance of the
quantizer input) and it is not useful to see the optimality of the step size
during training where the signal energy varies substantially over time. Using
the pre-computed step sizes, we finally obtain

$\Delta^{*}_{w}=\Delta^{u}_{w}\text{Std}(X),$ (9)

and

$\Delta^{*}_{a}=\Delta^{u}_{a}\sqrt{2\mathop{\mathbb{E}}[X^{2}]}.$ (10)

Note that $\sqrt{2\mathop{\mathbb{E}}[X^{2}]}$ is the standard deviation of
the pre-activations because they were assumed to have a symmetric distribution
around zero. The statistics of the quantizer input are estimated by the sample
statistics. For activations, we use a given number of batches to estimate the
standard deviation. For a simple implementation, we calculate the sample
standard deviation of each batch and use its maximum values over the batches.
In order to accurately estimate the input statistics of an activation
quantizer using multiple batches, we need to forward-propagate multiple
batches for each layer in a layer-wise manner. In our experience, this adds
unnecessary complexity to the implementation and the simple method provides
similar performance.

###  4.3 Training Instability in 1-bit

While our symmetric weight quantizer and MSE init support all bits seamlessly
to 1-bit, in our experience, the binary training is not effective in the same
setup as that for other bits. We investigate the training dynamics,can which
leads to the following observations. First, 1-bit SGD training receives strong
gradient signals initially because the initial point after binarization is far
from the solution of the pre-trained model, which we use for init, in contrast
to 2-bit or higher training. Second, the step sizes are not adapted to
maximize the signal-to-quantization-noise ratio (SQNR) or maintain the initial
SQNR during the initial fast learning. While the objective of the learnable
quantizer is not to maximize the SQNR, we observe that the step size usually
changes along with the standard deviation of the quantizer input in 2-bit or
higher training, maintaining a reasonable SQNR. Thus, a SQNR significantly
lower than the optimal level is not considered as desirable. We hypothesize
that abrupt changes in the quantizer input distribution get the step size
stuck in a local minimum. We show empirical evidence that warm-up training
mitigates this issue. In addition, we empirically find that Adam is more
robust to this problem than SGD. This appears to be owing to the gradient
normalization in Adam.

##  5 Experimental Results

To demonstrate the effectiveness of our proposed method, we evaluate it on the
CIFAR-100 [30] and the ImageNet datasets [43]. The CIFAR-100 dataset consists
of 60,000 32x32 color images from 100 classes with a total of 50,000 training
and 10,000 test images. The ImageNet dataset consists of more than 1.2M
training images from 1000 classes and 50K validation images. We use various
popular network architectures such as ResNet-18, -32, -34 [19] and MobieNet-V2
[44] for evaluation. The experiment results are compared with various recent
works on multi-bit quantization and neural network binarization.

Method | ResNet-18 (FP: 71.57) | ResNet-34 (FP: 75.11) | MobileNet-V2 (FP: 71.53)
---|---|---|---
| Bit-width (W/A)
| 4/4 | 3/3 | 2/2 | 1/1 | 4/4 | 3/3 | 2/2 | 1/1 | 4/4 | 3/3 | 2/2 | 1/1
PACT [5] | 69.2 | 68.1 | 64.4 | - | - | - | - | - | 61.4 | - | - | -
DoReFa-Net [55] | 68.1 | 67.5 | 62.6 | - | - | - | - | - | - | - | - | -
DSQ [16] | 69.6 | 68.7 | 65.2 | - | 72.8 | 72.5 | 70.0 | - | - | - | - | -
QIL [14] | 70.1 | 69.2 | 65.7 | - | 73.7 | 73.1 | 70.6 | - | 64.8 | - | - | -
LSQ [11] | 71.1 | 70.2 | 67.6 | - | 74.1 | 73.4 | 71.6 | - | - | - | - | -
LSQ+ [2] | 70.8 | 69.3 | 66.8 | - | - | - | - |  | - | - | - | -
SAT [26] | 70.3 | 69.3 | 65.5 | - | - | - | - | - | - | - | - | -
QKD [29] | 71.4 | 70.2 | 67.4 | - | 74.6 | 73.9 | 71.6 | - | 67.4 | 62.6 | 45.7 | -
UniQ (Ours) | 71.5 | 70.5 | 67.8 | 60.5 | 75.0 | 74.2 | 72.1 | 65.8 | 68.2 | 65.0 | 50.5 |  23.2
|  |  |  |  |  |  |  |  |  |  |  |
Table 2: Top-1 accuracy (%) on ImageNet dataset. Comparison with the existing
state-of-the-art DSQ, QIL, LSQ, LSQ+, QKD and SAT. The result of PACT for
MobileNet-V2 is cited from HAQ paper [51]. The dash symbol ”-” indicates no
data available and ”FP” represents the full precision network accuracy in our
implementation.

###  5.1 ImageNet Results

Implementation details. In the following experiments, we quantize all
convolutional and fully connected layers to ultra-low precision except the
first and last layers, which are represented by 8-bit precision as was done in
[11]. In case of binarized networks, we leave the first, last, and down-
sampling layers the full-precision as were done in prior works [28, 36]. We
quantize weights and activations to the same bit-width for all experiments.
For multi-bit and binarized networks, we use SGD and Adam, respectively. For
SGD, we use 0.01 as the initial learning rate and decay it using the cosine
learning rate schedule without restarts [37]. For Adam, the learning rate is
fixed to 0.001 for 5 epochs as the warm-up and then increase to 0.004 and then
follow the cosine schedule. For all experiments, we use layer-wise and kernel-
wise quantizations for activations and weights, respectively, with an
exception of MobileNetV2, in which layer-wise quantization is used for both
weights and activations considering the relative parameter overhead. In
addition, weight decay is not used for step size parameters. Our
implementation is based on PyTorch.

We use original ResNet-18, ResNet-34, and MobileNetV2 architectures, without
any changes in their structure. For ResNets, we use the pre-activation
version. We follow the commonly used data augmentation strategy as in [54,
11], where the training images are randomly cropped and resized to 224 × 224,
and horizontally flipped half the time. For testing, the single-center crop of
size 224×224 is applied. The transformed images are finally normalized by the
mean and standard deviation. All networks are trained for 90 epochs with a
batch size of 256 (2 GPUs), a momentum of 0.9, and a weight decay of
$10^{-4}$, $0.5\times 10^{-4}$, $0.25\times 10^{-4}$, 0 for 4-bit, 3-bit,
2-bit and 1-bit quantized models, respectively. We use the pre-trained models
available at PytorchCV 111https://pypi.org/project/pytorchcv for weight
initialization. For multi-bit and binarized networks, we use the floating-
point and 2-bit models for initialization, respectively. For step size
initialization, the first 1000 training batches are used to estimate the
statistics of activations.

Network | Method | Acc(%) | Original
---|---|---|---
ResNet-18 (FP: 71.57) | ABC-Net [34] | 42.7 |
XNOR-Net [42] | 51.2 |
BNN+ [9] | 53.0 | $\checkmark$
DoReFa-Net [55]${\dagger}$ | 53.4 | $\checkmark$
Bi-Real [36] | 56.4 |
XNOR-Net++ [3] | 57.1 |
IR-Net [41] | 58.1 | $\checkmark$
ProxyBNN [20] | 58.7 | $\checkmark$
RBNN [33] | 59.9 | $\checkmark$
BinaryDuo [28] | 60.4 |
UniQ (Ours) | 60.5 | $\checkmark$
ResNet-34 (FP: 75.11) | ABC-Net | 52.4 |
Bi-Real | 62.2 |
IR-Net | 62.9 | $\checkmark$
RBNN | 63.1 | $\checkmark$
UniQ (Ours) | 65.8 | $\checkmark$
|  |  |
Table 3: Top-1 accuracy comparison to the existing state-of-the-art
binarization methods on ImageNet. ”Original” is used to denote methods not
requiring architecture modification. ${\dagger}$DoReFa-Net uses 2-bit for
activations.

Comparison with prior works on multi-bit quantization. We compare our method
to existing methods in Table 2. For the existing methods, the results are
directly cited from the original papers unless mentioned otherwise. By looking
at the reported table, we can observe that our method outperforms all the
previous quantization methods in top-1 accuracy. Specifically, we can achieve
significant performance gain over the recent state-of-the-art methods LSQ,
QKD, and SAT on all comparing architectures. The improvements range from 0.1%
to 4.8% compared to the second-best method (QKD). Note that QKD and SAT need a
total of 120 and 150 training epochs, respectively, while our method only
requires 90 epochs to obtain better accuracy. In addition, it is worth to
mention that QKD uses knowledge distillation but UniQ does not. Knowledge
distillation is known to provide additional improvements on quantization
results as shown in LSQ [11]. MobileNetV2 has an architecture already
optimized for efficiency such as depth-wise convolutions and the accuracy is
known to be sensitive to quantization [11, 29]. For MobileNetV2, UniQ
outperforms the existing state-of-the-art method, QKD, by significant margins
of 2.4% and 4.8% for 2-bit and 3-bit quantized models, respectively. A
substantial increase in prediction accuracy can be seen in other 2-bit models.
With ResNet-18, UniQ achieves 67.8% top-1 accuracy, with 0.2% and 0.4%
improvements over LSQ and QKD, respectively. With ResNet-34, it achieves a
top-1 accuracy of 72.1%, which is a 0.4% improvement over QKD and LSQ.

Comparison with prior works on binarized neural networks. We further compare
our method with the state-of-the-art binarization methods in Table 3. As shown
in the last column, many existing methods modify a base architecture and it is
difficult to be compared. While the model size and the number of operations
are the same, the memory access count can be different. For example, the dual
skip connection employed in Bi-Real does not affect the model size and the
number of operations for convolutions but the memory access count.
Nonetheless, to our best knowledge, UniQ outperforms the previous state-of-
the-art accuracy for binarized ResNet-34 by a significant margin of 2.69%. For
ResNet-18, UniQ even achieves a comparable accuracy to BinaryDuo, which
requires to increase the width of the skip connections.

###  5.2 CIFAR-100 Results

Implementation details. We use the pre-activation variant of ResNet-32 for all
the experiments on CIFAR-100. We train for 350 epochs with a mini-batch size
of 128. All quantized models are initialized from the pre-trained full-
precision counterparts, which we train from scratch. For simplicity, we use
the same weight decay of $5\times 10^{-4}$ across all CIFAR-100 experiments.
The standard data augmentation includes random cropping and horizontal
flipping is applied for each training image. The first 100 training batches
are used for step size initialization. We use layer-wise quantization for both
weights and activations. For other settings, we follow the same settings as
described in Section 5.1

Comparison with LSQ. For a fair comparison, we compare LSQ to our method in
our setting. We implement LSQ carefully and cross-check the correctness with
[11, 2]. Our final results are summarized in Table 4. When using ResNet-32, we
can observe that, for 4-bit quantized models, LSQ can match the accuracy of
the full-precision baseline, which is in line with the results reported in the
original paper [11]. It is worth noting that when the bit-width is reduced to
3, our method can still archive the same accuracy of 71.4% compared to the
4-bit LSQ quantized model. For 2-bit, the accuracy drops by only 2.1% when
using our method compared to 2.9% for LSQ. For the most aggressive 1-bit
quantization, we can achieve 62.4%, while no data is reported for LSQ as its
quantizer is not suitable for binarization.

Figure 3: Weight distribution from two different layers of trained, 2-bit
quantized ResNet-32 with LSQ (green) and UniQ (red) quantization scheme. Two
networks are trained with the same hyperparameters on CIFAR-100 for 350
epochs. The dashed lines are the reconstruction levels of each method.

(a) 2-bit SGD.

(b) 1-bit SGD w/o warm-up.

(c) 1-bit SGD w/ warm-up.

(d) 1-bit Adam.

Figure 4: Training dynamics of an activation quantization layer in ResNet32.
The abrupt change of the standard deviation (std) of the quantizer input may
get the step size ($\Delta$) stuck in a poor local minimum. The highlighted
area indicates the warm-up period and the dashed orange line indicates the
optimal SQNR value.

Imbalanced weight distribution.  For UniQ and LSQ, we show the distributions
of the trained weights in Figure 3, for two different layers. We can see that
the trained weights of LSQ have the form of negatively-skewed distribution,
with a long tail on the negative side, and many weights values around the
maximum reconstruction level. In contrast, UniQ has a symmetric distribution
around the zero value, and the weights are relatively evenly distributed. The
higher entropy of the balanced distribution may suggest that UniQ allows the
network to retain more knowledge on weights than LSQ.

Step size initialization. To show how important the initialization of the step
sizes is, we also compare the results when the step size of the proposed
quantizer is initialized with 0.1, 0.2, LSQ’s heuristic, and the proposed
optimal method. In LSQ, the step size is initialized to
$\frac{2\mathop{\mathbb{E}}[X]}{\sqrt{Q_{P}}}$ where $X$ is the quantizer
input and $Q_{P}=2^{\text{bit}}-1$ ($Q_{P}=2^{\text{bit}-1}-1$) for
activations (weights). The results are summarized in Table 5. The accuracy
values vary substantially depending on initialization. It is interesting to
see that, our proposed quantizer with LSQ init performs poorly for all bit-
widths. In contrast, by replacing LSQ init with our init, a significant
performance boost can be seen for all bit-widths. The 1.0% performance boost
over LSQ init is seen in 2-bit. These results suggest that the step size
parameters need to be initialized properly, otherwise it will lead to
performance degradation.

Methods | Bit-width (W/A) |
---|---|---
| 32/32 | 4/4 | 3/3 | 2/2 | 1/1
LSQ | 71.4 | 71.4 | 70.9 | 68.5 | -
UniQ (Ours) | 71.6 | 71.4 | 69.3 | 62.4
|  |  |  |  |
Table 4: Comparison of LSQ and our method (UniQ) with ResNet32 and CIFAR-100 (FP Acc: 71.4%) Bit-width | Step Size Initialization
---|---
(W/A) | 0.1 | 0.2 | LSQ Init | Our Init
2/2 |  67.1 | 68.6 | 68.3 | 69.3
3/3 | 70.7 | 70.9 | 71.0 | 71.4
|  |  |  |
Table 5: Comparison of different methods for step size initialization with ResNet32 and CIFAR-100 (FP Acc: 71.4%) | No warm-up | Constant warm-up
---|---|---
| Learning rate | Warm-up epochs
SGD | 0.01 | 0.005 | 0.001 | 5 | 10
57.0 | 59.1 | 59.5 | 61.3 | 61.1
Adam | 0.004 | 0.001 | 0.0005 | 5 | 10
60.9 | 59.9 | 61.0 | 62.4 | 62.1
|  |  |  |  |
Table 6: Results of ResNet-32 on CIFAR-100 with different learning rate
schedules. For the warm-up period, the initial learning rate is set to 0.001,
and then increases to 0.01 (0.004) for SGD (Adam).

Training dynamics. While our quantizer and init method are general to all bit-
widths, the binarized networks trained by UniQ do not provide a satisfactory
performance at the same hyperparameter setting for 2-bit or higher. We thus
investigate the activation quantizer of a convolutional layer. Specifically,
we observe the evolution of the step size, the standard deviation (SD) of the
quantizer input, and SQNR during 30 epochs of training. Figure (4(a)) and
(4(b)) show 2-bit and 1-bit training, respectively, at the same setting, where
we use SGD and the learning rate of 0.01. In 2-bit, the step size shrinks as
the SD decreases, which maintains the initial SQNR not very far from the
optimal level, shown in the dotted orange line. However, in 1-bit, the SD
jumps sharply at the beginning, whereas the step size does not change
accordingly, dropping the SQNR substantially, suggesting that the step size
might be trapped at a local minimum. Figure (4(c)) shows how the warm-up can
help mitigate this problem. We use 5 epochs of warm-up with a learning rate of
0.001 and observe that the SD changes smoothly and the step size seems to be
adapted to that, maintaining a better SQNR. Figure (4(d)) shows an evidence
that Adam is more robust to this issue. While we use the learning rate of
0.004, which is a high rate for Adam, we do not observe the issue, and the
SQNR approximately stays at the optimal level in this case.

Warm-up for 1-bit Training. We further perform experiments to find the best
optimizer and warm-up strategy for binary training. The empirical results from
Table 6 are in line with our analysis on the training dynamics. The results
indicate that initially training a binarized model with a small learning rate
for some epochs can improve the prediction accuracy substantially. In
contrast, the models trained with a higher rate at the beginning can be easily
trapped in a saddle point or bad local minimum; without increasing the
learning rate at some point, the training may converge too slow. The results
also suggest that Adam may perform better than SGD, but SGD also provides a
decent performance in contrast to the common belief that SGD gives severely
degraded results than Adam in binary training [1]. It is also shown that warm-
up training helps Adam as well as SGD. Moreover, we observe that increasing
the number of warm-up epochs does not improve the accuracy significantly.
Thus, we choose the 5 epoch warm-up period.

##  6 Conclusion

In this paper, we have proposed a quantization method generalized for both
multi-bit quantized and binarized models. We have designed a symmetric
quantizer with a trainable step size and proposed an analytic, optimal
initialization of the step size. In addition, we have investigated the
difficulties in the 1-bit training and suggested practical methods to overcome
them. For multi-bit quantization, the proposed method have achieved new record
accuracies of ResNet-18,-34, MobileNetV2 on ImageNet. For binarization,
without modifying original network architectures, we have achieved better or
comparable accuracy to that of recent binarized networks by focusing on the
fundamental training problems.

## References

  * [1] M. Alizadeh, J. Fernández-Marqués, N. D. Lane, and Y. Gal.  An empirical study of binary neural networks’ optimisation.  In International Conference on Learning Representations, 2018.
  * [2] Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 696–697, 2020.
  * [3] A. Bulat and G. Tzimiropoulos.  Xnor-net++: Improved binary neural networks.  arXiv preprint arXiv:1909.13863, 2019.
  * [4] J. Choi, S. Venkataramani, V. Srinivasan, K. Gopalakrishnan, Z. Wang, and P. Chuang.  Accurate and efficient 2-bit quantized neural networks.  In Proceedings of the 2nd SysML Conference, volume 2019, 2019.
  * [5] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan.  Pact: Parameterized clipping activation for quantized neural networks.  arXiv preprint arXiv:1805.06085, 2018.
  * [6] F. Conti, P. D. Schiavone, and L. Benini.  Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 37(11):2940–2951, 2018.
  * [7] M. Courbariaux, Y. Bengio, and J.-P. David.  Binaryconnect: Training deep neural networks with binary weights during propagations.  In Advances in neural information processing systems, pages 3123–3131, 2015.
  * [8] G. E. Dahl, D. Yu, L. Deng, and A. Acero.  Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition.  IEEE Transactions on audio, speech, and language processing, 20(1):30–42, 2011.
  * [9] S. Darabi, M. Belbahri, M. Courbariaux, and V. P. Nia.  Bnn+: Improved binary network training.  arXiv preprint arXiv:1812.11800, 2018.
  * [10] R. Ding, T.-W. Chin, Z. Liu, and D. Marculescu.  Regularizing activation distribution for training binarized deep networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11408–11417, 2019.
  * [11] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha.  Learned step size quantization.  In International Conference on Learning Representations, 2019.
  * [12] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha.  Learned step size quantization.  arXiv preprint arXiv:1902.08153, 2019.
  * [13] H. Genc, A. Haj-Ali, V. Iyer, A. Amid, H. Mao, J. Wright, C. Schmidt, J. Zhao, A. Ou, M. Banister, et al.  Gemmini: An agile systolic array generator enabling systematic evaluations of deep-learning architectures.  arXiv preprint arXiv:1911.09925, 2019.
  * [14] J. S. Gil, C. T. Son, S. H. Lee, J. W. Son, Y. J. Kwak, J. J. Han, S. J. Hwang, and C. G. Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In Conference on Computer Vision and Pattern Recognition (CVPR 2019), pages 1–10. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [15] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
  * [16] R. Gong, X. Liu, S. Jiang, T. Li, P. Hu, J. Lin, F. Yu, and J. Yan.  Differentiable soft quantization: Bridging full-precision and low-bit neural networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4852–4861, 2019.
  * [17] S. Han, J. Pool, J. Tran, and W. Dally.  Learning both weights and connections for efficient neural network.  In Advances in neural information processing systems, pages 1135–1143, 2015.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] X. He, Z. Mo, K. Cheng, W. Xu, Q. Hu, P. Wang, Q. Liu, and J. Cheng.  Proxybnn: Learning binarized neural networks via proxy matrices.  In Proceedings of the European Conference on Computer Vision, volume 2, 2020.
  * [21] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1314–1324, 2019.
  * [22] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [23] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio.  Binarized neural networks.  In Advances in neural information processing systems, pages 4107–4115, 2016.
  * [24] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704–2713, 2018.
  * [25] S. R. Jain, A. Gural, M. Wu, and C. H. Dick.  Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks.  arXiv preprint arXiv:1903.08066, 2019.
  * [26] Q. Jin, L. Yang, Z. Liao, and X. Qian.  Neural network quantization with scale-adjusted training.
  * [27] D. Kim, K. P. Singh, and J. Choi.  Learning architectures for binary networks.  In European Conference on Computer Vision, pages 575–591. Springer, 2020.
  * [28] H. Kim, K. Kim, J. Kim, and J.-J. Kim.  Binaryduo: Reducing gradient mismatch in binary activation network by coupling binary activations.  In International Conference on Learning Representations, 2020.
  * [29] J. Kim, Y. Bhalgat, J. Lee, C. Patel, and N. Kwak.  Qkd: Quantization-aware knowledge distillation.  arXiv preprint arXiv:1911.12491, 2019.
  * [30] A. Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * [31] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [32] D. Lin, S. Talathi, and S. Annapureddy.  Fixed point quantization of deep convolutional networks.  In International conference on machine learning, pages 2849–2858, 2016.
  * [33] M. Lin, R. Ji, Z. Xu, B. Zhang, Y. Wang, Y. Wu, F. Huang, and C.-W. Lin.  Rotated binary neural network.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [34] X. Lin, C. Zhao, and W. Pan.  Towards accurate binary convolutional neural network.  In Advances in Neural Information Processing Systems, pages 345–353, 2017.
  * [35] J. Liu, B. Zhuang, P. Chen, M. Tan, and C. Shen.  Aqd: Towards accurate quantized object detection.  arXiv preprint arXiv:2007.06919, 2020.
  * [36] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng.  Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm.  In Proceedings of the European conference on computer vision (ECCV), pages 722–737, 2018.
  * [37] I. Loshchilov and F. Hutter.  SGDR: stochastic gradient descent with warm restarts.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [38] A. Mishra and D. Marr.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy.  arXiv preprint arXiv:1711.05852, 2017.
  * [39] H. Phan, Z. Liu, D. Huynh, M. Savvides, K.-T. Cheng, and Z. Shen.  Binarizing mobilenet via evolution-based searching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13420–13429, 2020.
  * [40] A. Polino, R. Pascanu, and D. Alistarh.  Model compression via distillation and quantization.  arXiv preprint arXiv:1802.05668, 2018.
  * [41] H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song.  Forward and backward information retention for accurate binary neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2250–2259, 2020.
  * [42] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In European conference on computer vision, pages 525–542. Springer, 2016.
  * [43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [44] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [45] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer.  Efficient processing of deep neural networks: A tutorial and survey.  Proceedings of the IEEE, 105(12):2295–2329, 2017.
  * [46] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2820–2828, 2019.
  * [47] M. Tan and Q. Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  volume 97 of Proceedings of Machine Learning Research, pages 6105–6114, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [48] S. Uhlich, L. Mauch, K. Yoshiyama, F. Cardinaux, J. A. Garcia, S. Tiedemann, T. Kemp, and A. Nakamura.  Differentiable quantization of deep neural networks.  arXiv preprint arXiv:1905.11452, 2(8), 2019.
  * [49] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott, P. Leong, M. Jahre, and K. Vissers.  Finn: A framework for fast, scalable binarized neural network inference.  In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 65–74, 2017.
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [51] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han.  Haq: Hardware-aware automated quantization with mixed precision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8612–8620, 2019.
  * [52] X. Wu, Y. Wu, and Y. Zhao.  Binarized neural networks on the imagenet classification task.  arXiv preprint arXiv:1604.03058, 2016.
  * [53] J. Ye, S. Zhang, and J. Wang.  Distillation guided residual learning for binary convolutional neural networks.  arXiv preprint arXiv:2007.05223, 2020.
  * [54] D. Zhang, J. Yang, D. Ye, and G. Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In Proceedings of the European conference on computer vision (ECCV), pages 365–382, 2018.
  * [55] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  arXiv preprint arXiv:1606.06160, 2016.
  * [56] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid.  Structured binary neural networks for accurate image classification and semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 413–422, 2019.