# Single-path Bit Sharing for Automatic Loss-aware Model Compression

Jing Liu, Bohan Zhuang, Peng Chen, Chunhua Shen, Jianfei Cai, Mingkui Tan†
Jing Liu is with the School of Software Engineering, South China University of
Technology and also with the Key Laboratory of Big Data and Intelligent Robot
(South China University of Technology), Ministry of Education. E-mail:
<EMAIL_ADDRESS>Mingkui Tan is with the School of Software
Engineering, South China University of Technology. Mingkui Tan is also with
the Pazhou Laboratory, Guangzhou, China. E-mail<EMAIL_ADDRESS>Peng
Chen is with the School of Computer Science, the University of Adelaide,
Australia. E-mail<EMAIL_ADDRESS>Bohan Zhuang, and Jianfei Cai are
with the Faculty of Information Technology, Monash University, Australia.
E-mail: {bohan.zhuang<EMAIL_ADDRESS>Chunhua Shen is with the
College of Computer Science and Technology, Zhejiang University, Hangzhou,
China. E-mail<EMAIL_ADDRESS>† Corresponding author.

###### Abstract

Network pruning and quantization are proven to be effective ways for deep
model compression. To obtain a highly compact model, most methods first
perform network pruning and then conduct network quantization based on the
pruned model. However, this strategy may ignore that the pruning and
quantization would affect each other and thus performing them separately may
lead to sub-optimal performance. To address this, performing pruning and
quantization jointly is essential. Nevertheless, how to make a trade-off
between pruning and quantization is non-trivial. Moreover, existing
compression methods often rely on some pre-defined compression configurations
(_i.e._ , pruning rates or bitwidths). Some attempts have been made to search
for optimal configurations, which however may take unbearable optimization
cost.  To address the above issues, we devise a simple yet effective method
named Single-path Bit Sharing (SBS) for automatic loss-aware model
compression. Equipped with SBS, the trade-off between pruning and quantization
can be automatically determined. More critically, we can also search for
optimal compression configurations in an effective way.  Specifically, we
first consider network pruning as a special case of quantization, which
provides a unified view for pruning and quantization. We then introduce a
single-path model to encode all candidate compression configurations. For
example, a high bitwidth value will be decomposed into the sum of the lowest
bitwidth value and a series of re-assignment offsets. In this way, the
configuration search problem is transformed into a subset selection problem,
which significantly reduces the number of parameters, computational cost and
optimization difficulty. Relying on the single-path model, we further
introduce learnable binary gates to encode the choice of bitwidth. By jointly
training the binary gates in conjunction with network parameters, the
compression configurations of each layer can be automatically determined.
Extensive experiments on both CIFAR-100 and ImageNet show that SBS is able to
significantly reduce computational cost while achieving promising performance.
For example, our SBS compressed MobileNetV2 achieves 22.6$\times$ Bit-
Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.

###### Index Terms:

Network Quantization, Network Pruning, Bit Sharing, Loss-aware Model
Compression.

##  1 Introduction

Deep neural networks (DNNs) [36] have achieved great success in many
challenging computer vision tasks, including image classification [35, 25,
17], object detection [53, 40, 59], image generation [19, 20, 7], and video
analysis [57, 64, 74]. However, a deep model usually has a large number of
parameters and consumes enormous computational resources, which presents great
obstacles for many applications, especially on resource-constraint devices,
such as smartphones. To reduce the number of parameters and computational
overhead, many methods [27, 76, 80] have been proposed to perform model
compression by removing the redundancy while maintaining the performance.

(a) Multi-path search scheme [67]

(b) Single-path search scheme (Ours)

Figure 1: Multi-path vs. single-path search scheme. For clarity, we show the
illustrations of weight quantization only.  (a) Multi-path search scheme [67]:
represents each candidate configuration as a separate path and formulates the
mixed-precision quantization problem as a path selection problem, which gives
rise to a huge number of trainable parameters and high computational overhead
when the search space becomes large. $o_{i}$ and $p_{i}$ are the convolutional
operation and probability for the $i$-th configuration, respectively. $x^{l}$
indicates the input feature maps for layer $l$, $w^{i}$ is the weight for the
$i$-th configuration and $w^{i}_{k}$ is the $k$-bit quantized version of
$w^{i}$. (b) Single-path search scheme (Ours): represents each candidate
configuration as a subset of a super-bitwidth and formulates the mixed-
precision quantization problem as a subset selection problem, which
significantly reduces the number of parameters, computational cost, and
optimization difficulty. The super-bitwidth is the highest bitwidth (_i.e._ ,
32-bit) in the search space, $w$ is the weight for the convolutional
operation, $w_{k}$ is the $k$-bit quantized version of $w$, $r_{k}$ is the re-
assignment offset and $g_{k}$ is the corresponding binary gate to control the
bitwidth decision.

In recent years, we have witnessed the remarkable progress of model
compression methods. Specifically, network pruning [28, 27] removes the
uninformative modules and network quantization [76, 30] maps the full-
precision values to low-precision ones. To obtain a highly compact model, most
methods [24, 47, 71] first perform pruning and then conduct quantization based
on the pruned model.  However, performing pruning and quantization separately
ignores that they would affect each other, which may lead to sub-optimal
performance.  To address this issue, performing pruning and quantization
jointly is essential and urgent in many applications but may pose several new
challenges.

First, finding the optimal trade-off between pruning and quantization is non-
trivial since they would affect each other. For example, if a model is under-
pruned, we can apply aggressive network quantization to the pruned model to
achieve a high compression ratio. In contrast, if a model is over-pruned, the
resulting model is more sensitive to the quantization noise. In this case, the
quantization bitwidth of the pruned model has to be high to preserve the
performance.

Second, to achieve better performance, one may assign different configurations
(_e.g._ , pruning rates and bitwidths) according to the layer’s contribution
to the accuracy and efﬁciency of a network. However, the search space of model
compression grows exponentially with the increasing number of compression
configurations. To handle this, existing differentiable methods [67, 6]
explore the exponential search space using gradient-based optimization. As
shown in Figure 1(a), these methods first construct a multi-path candidate
network wherein each candidate compression configuration is maintained as a
separate path. Then, the optimal path is selected during training. However,
when the search space becomes large, the multi-path scheme leads to a huge
amount of parameters and high computational cost. Due to the parameter
explosion, the optimization of the candidate network would be challenging.

In this paper, we propose to address these issues by devising a simple yet
effective method named Single-path Bit Sharing (SBS) for automatic loss-aware
model compression. Equipped with SBS, the trade-off between pruning and
quantization can be learned. More critically, SBS is able to efficiently
search optimal configurations. Specifically, we first represent pruning as
0-bit quantization, which provides a unified view for pruning and
quantization. We then introduce a novel single-path model to encode all
bitwidths in the search space, as shown in Figure 1(b). Our SBS is built upon
the observation that the quantized values of a high bitwidth can be shared
with those of low bitwidths under some conditions. Therefore, we are able to
decompose a quantized representation into the sum of the lowest bitwidth
representation and a series of re-assignment offsets. In this way, the
quantized values of different bitwidths are different subsets of those of
super-bitwidth (_i.e._ , the highest bitwidth in the search space).  Instead
of choosing a path in the multi-path scheme [67], our SBS casts the mixed-
precision quantization problem to a subset selection problem, which
significantly reduces the number of parameters, computational cost, and
optimization difficulty. Relying on the single-path model, we further
introduce learnable binary gates to encode the choice of bitwidth. By jointly
training the binary gates and network parameters, the configurations of each
layer can be automatically determined.

Our main contributions are summarized as follows:

  * •

We devise a novel single-path scheme that encapsulates multiple configurations
in a unified single-path framework. Relying on this, we further introduce
learnable binary gates to determine the optimal configurations. Our proposed
SBS casts the search problem as a subset selection problem, hence
significantly reducing the number of parameters and search cost.

  * •

We formulate the quantized representation as a gated combination of the lowest
bitwidth representation and a series of re-assignment offsets, in which we
explicitly share the quantized values between different bitwidths. By training
the binary gates and network parameters, the configuration of each layer can
be automatically determined. The proposed SBS enable the candidate operations
to learn jointly rather than separately, hence greatly easing the optimization
of the compressed candidate network.

  * •

We evaluate our SBS on CIFAR-100 and ImageNet over various network
architectures. Extensive experiments show that the proposed method achieves
the state-of-the-art performance. For example, on ImageNet, our SBS compressed
MobileNetV2 achieves 22.6$\times$ Bit-Operation (BOP) reduction with only 0.1%
performance decline in terms of the Top-1 accuracy.

##  2 Related Work

Network quantization. Network quantization represents the weights, activations
and even gradients with low precision to yield compact DNNs. With low-
precision integers [76] or power-of-two representations [38], the heavy matrix
multiplications can be replaced by efficient bitwise operations, leading to
much faster test-time inference and lower power consumption. To improve the
quantization performance, extensive methods have been proposed to learn
accurate quantizers [32, 75, 11, 5, 16]. Specifically, given a convolutional
layer, let $x$ and $w$ be the output activations of the previous layer and the
weight parameters of given layer, respectively. First, following [33, 11, 8],
one can normalize $x$ and $w$ into scale [0, 1] by $T_{x}(\cdot)$ and
$T_{w}(\cdot)$, respectively:

$\displaystyle z_{x}$
$\displaystyle=T_{x}(x)=\mathrm{clip}\left(\frac{x}{v_{x}},0,1\right),$ (1)
$\displaystyle z_{w}$
$\displaystyle=T_{w}(w)=\frac{1}{2}\left(\mathrm{clip}\left(\frac{w}{v_{w}},-1,1\right)+1\right),$
(2)

where $v_{x}$ and $v_{w}$ are trainable quantization intervals indicating the
range of weights and activations to be quantized. Here, the function
$\mathrm{clip}\left(v,v_{\mathrm{low}},v_{\mathrm{up}}\right)={\rm min}({\rm
max}(v,v_{\mathrm{low}}),v_{\mathrm{up}})$ clips any number $v$ into the range
$[v_{\mathrm{low}},v_{\mathrm{up}}]$. Then, one can apply the following
function to quantize the normalized activations and parameters, namely
$z_{x}\in[0,1]$ and $z_{w}\in[0,1]$, to discretized ones:

$D(z,s)=s\cdot\mathrm{round}\left(\frac{z}{s}\right),$ (3)

where $s$ denotes the normalized step size,  $\mathrm{round}(x)=\lceil
x-0.5\rceil$ returns the nearest integer of a given value $x$ and
$\lceil\cdot\rceil$ is the ceiling function.  Typically, for $k$-bit
quantization, the normalized step size $s$ can be computed by

$s=\frac{1}{2^{k}-1}.$ (4)

Last, the quantized activations and weights can be obtained by
$Q_{x}(x)=T_{x}^{-1}(D(z_{x},s))=v_{x}\cdot D(z_{x},s)$ and
$Q_{w}(w)=T_{w}^{-1}(D(z_{w},s))=v_{w}\cdot(2\cdot D(z_{w},s)-1)$, where
$T_{x}^{-1}(\cdot)$ and $T_{w}^{-1}(\cdot)$ denote the inverse functions of
$T_{x}(\cdot)$ and $T_{w}(\cdot)$, respectively.  In general, the function
$D(\cdot,\cdot)$ is non-differentiable. Following [76, 30], one can use the
straight through estimation (STE) [1] to approximate the gradient of
$D(\cdot,\cdot)$ by the identity mapping, namely, $\partial D(z,s)/z\approx
1$. To reduce the optimization difficulty introduced by $D(\cdot,\cdot)$,
several methods have been proposed to approximate the gradients of
$D(\cdot,\cdot)$ [13, 46, 77]. Moreover, most previous works assign the same
bitwidth for all layers [76, 78, 79, 32, 31, 38, 16, 68, 8]. Though attractive
for simplicity, setting a uniform precision places no guarantee on optimizing
network performance, since different layers have different redundancy and
arithmetic intensity. Therefore, several studies proposed mixed-precision
quantization [63, 15, 67, 61] that sets different bitwidths according to the
redundancy of each layer. In this paper, based on the proposed single-path bit
sharing model, we devise an approach that efficiently searches for appropriate
bitwidths for different layers through gradient-based optimization.

NAS and pruning. Neural architecture search (NAS) aims to automatically design
efficient architectures, either based on reinforcement learning [51, 22],
evolutionary search [52, 70] or gradient-based methods [41, 3]. In particular,
gradient-based NAS has gained increased popularity, where the search space can
be divided into the multi-path design [41, 4] and single-path formulation [58,
23], depending on whether each operation is added as a separate path or not.
While prevailing methods optimize the network topology, the focus of this
paper is to search optimal configurations for a given network. Moreover,
network pruning can be treated as fine-grained NAS, which aims at removing
redundant modules to reduce the model size and accelerate the run-time
inference speed, giving rise to methods based on weight pruning [24, 21, 43],
filter pruning [28, 80, 48], or layer pruning [66, 29, 9], _etc._ Apart from
filter pruning, we further take network quantization into consideration to
obtain more compact networks.

AutoML for model compression. Recently, much effort has been devoted to
automatically determining the pruning rate [60, 14], or the bitwidth [45, 6]
of each layer, either based on reinforcement learning [26, 63], evolutionary
search [42, 65], gradient optimization [67, 6], _etc._ To increase the
compression ratio, several methods have been proposed to jointly optimize
pruning and quantization strategies. In particular, some work only support
weight quantization [60, 72] or use fine-grained pruning [69]. However, the
resultant models cannot be implemented efficiently on edge devices. To handle
this, several methods [67, 65, 73] have been proposed to consider filter
pruning, weight quantization, and activation quantization jointly. Compared
with these methods, we carefully design the compression search space by
sharing the quantized values between different candidate configurations, which
significantly reduces the number of parameters, search cost, and optimization
difficulty. Compared with those methods that share the similarities of using
quantized residual errors [10, 18, 39, 62], our proposed method recursively
uses quantized residual errors to decompose a quantized representation into a
set of candidate bitwidths and parameterize the bitwidth selection via a
series of binary gates. Compared with Bayesian Bits [62], our SBS differs in
several aspects: 1) We theoretically verify the theorem of quantization
decomposition and it can be applied to non-power-of-two bitwidths (i.e.,
$b_{1}$ and $\gamma_{j}$ can be set to arbitrary appropriate integer values),
which is a general case of the one in Bayesian Bits. 2) The optimization
problems are different. Specifically, we formulate model compression as a
single-path subset selection problem while Bayesian Bits casts the
optimization of the binary gates to a variational inference problem that
requires more relaxations and hyperparameters. 3) Our compressed models with
less or comparable BOPs outperform those of Bayesian Bits by a large margin on
ImageNet (See Table III).

##  3 Proposed Method

Given a pre-trained model, we focus on automatic model compression for pruning
and quantization jointly, which poses two new challenges. First, finding the
optimal trade-off between pruning and quantization is non-trivial since they
may affect each other. Second, to determine the optimal configurations (_e.g._
, pruning rates and bitwidths) for each layer, one may consider different
configurations as different paths and reformulate the configuration search
problem as a path selection problem [67], as shown in Figure 1(a). However,
when the search space becomes large, it suffers from a huge number of
parameters and high computational cost. As a result, the optimization of the
compressed model becomes more challenging.

In this paper, we consider network pruning as 0-bit quantization, which
provides a unified view for pruning and quantization. In this way, the model
compression can be transformed into a mixed-precision quantization problem.
Based on this, we highlight that the quantized values of a high bitwidth are
shared with those of low bitwidths under some conditions. Relying on this
observation, we propose a simple yet effective method named Single-path Bit
Sharing (SBS) to reduce the number of parameters, search cost and ease the
optimization for the compressed candidates, as shown in Figure 1(b). In the
following, we will illustrate the details of our method.

###  3.1 Single-path bit sharing decomposition

To illustrate the single-path bit sharing decomposition, we begin with an
example of 2-bit quantization for $z\in\\{z_{x},z_{w}\\}$. Specifically, we
use the following equation to quantize $z$ to $2$-bit using Eqs. (3) and (4):

$z_{2}=D(z,s_{2}),\quad~{}s_{2}=\frac{1}{2^{2}-1},$ (5)

where $z_{2}$ and $s_{2}$ are the quantized value and the step size of 2-bit
quantization, respectively. Due to the large step size, the residual error
$z-z_{2}\in[-s_{2}/2,s_{2}/2]$ may be large and results in a significant
performance decline. To reduce the residual error, an intuitive way is to use
a smaller step size, which indicates that we quantize $z$ to a higher
bitwidth. Since the step size $s_{4}=1/(2^{4}-1)$ in 4-bit quantization is a
divisor of the step size $s_{2}$ in 2-bit quantization, the quantized values
of 2-bit quantization are shared with those of 4-bit quantization. Based on
2-bit quantization, the 4-bit counterpart introduces additional unshared
quantized values. In particular, if $z_{2}$ has zero residual error, then
4-bit quantization maps $z$ to the shared quantized values (_i.e._ , $z_{2}$).
In contrast, if $z_{2}$ has non-zero residual error, 4-bit quantization is
likely to map $z$ to the unshared quantized values.  In this case, 4-bit
quantization can be regarded as performing quantized value re-assignment based
on $z_{2}$. Such a re-assignment process can be formulated as

$z_{4}=z_{2}+r_{4},$ (6)

where $z_{4}$ is the 4-bit quantized value and $r_{4}$ is the re-assignment
offset based on $z_{2}$. To ensure that the results of re-assignment fall into
the 4-bit quantized values, the re-assignment offset $r_{4}$ must be an
integer multiple of the step size $s_{4}$. Formally, $r_{4}$ can be computed
by performing 4-bit quantization on the residual error of $z_{2}$:

$r_{4}=D(z-z_{2},s_{4}),\quad~{}s_{4}=\frac{1}{2^{4}-1}.$ (7)

According to Eq. (6), a 4-bit quantized value can be decomposed into the 2-bit
representation and its re-assignment offset. Similarly, an 8-bit quantized
value can also be decomposed into the 4-bit representation and its
corresponding re-assignment offset. In this way, we can generalize the idea of
decomposition to arbitrary effective bitwidths as follows.

######  Theorem 1.

_(Quantization Decomposition)_ Let $z\in\left[0,1\right]$ be a normalized
full-precision input, and $\\{b_{j}\\}_{j=1}^{K}$ be a sequence of candidate
bitwidths. If $b_{j}$ is an integer multiple of $b_{j-1}$, _i.e._ ,
$b_{j}\small{=}\gamma_{j}b_{j-1}(j>1)$, where
$\gamma_{j}\in{\mathbb{Z}}^{+}\backslash\\{1\\}$ is a multiplier, then the
quantized approximation $z_{b_{K}}$ can be decomposed as:

$\displaystyle z_{b_{K}}=z_{b_{1}}+\sum_{j=2}^{K}r_{b_{j}},~{}$ (8)
$\displaystyle\mathrm{where}~{}r_{b_{j}}=D(z-z_{b_{j-1}},s_{b_{j}}),~{}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}z_{b_{j}}=D(z,s_{b_{j}}),~{}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}s_{b_{j}}=\frac{1}{2^{b_{j}}-1}.$

Theorem 1 indicates that the quantized approximation $z_{b_{K}}$ can be
decomposed into the sum of the lowest bitwidth representation $z_{b_{1}}$ and
a series of recursive re-assignment offsets. To measure the quantization
error,  we introduce the following corollary.

Figure 2: The normalized quantization error change and error bound vs.
bitwidth. The normalized quantization error change (red line) is bounded by
$C/(2^{b_{K}}\small{-}1)$ (blue line).

######  Corollary 1.

_(Normalized Quantization Error Bound)_ Given ${\bf z}\in[0,1]^{d}$ being a
normalized full-precision vector, ${\bf z}_{b_{K}}$ being its quantized vector
with bitwidth $b_{K}$, where $d$ is the cardinality of ${\bf z}$. Let
$\epsilon_{K}=\frac{\|{\bf z}-{\bf z}_{b_{K}}\|_{1}}{\|{\bf z}\|_{1}}$ be the
normalized quantization error, then the following error bound w.r.t. $K$
holds:

$\displaystyle|\epsilon_{K}\small{-}\epsilon_{K+1}|\leq\frac{C}{2^{b_{K}}\small{-}1},$
(9)

where $~{}C\small{=}\frac{d}{\|{\bf z}\|_{1}}$ is a constant.

To empirically demonstrate Corollary 1, we perform quantization on a random
toy data ${\bf z}\in[0,1]^{100}$ and show the normalized quantization error
change $|\epsilon_{K}\small{-}\epsilon_{K+1}|$ and error bound
$\frac{C}{2^{b_{K}}\small{-}1}$ in Figure 2. From the results, the normalized
quantization error change decreases quickly as the bitwidth increases and is
bounded by $C/(2^{b_{K}}\small{-}1)$. We put the proof of Corollary 1 in the
supplementary material.

Note that in Theorem 1, both the smallest bitwidth $b_{1}$ and the multiplier
$\gamma_{j}$ can be set to arbitrary appropriate integer values (_e.g._ , 2,
3, etc.). To obtain a hardware-friendly compressed network111More details can
be found in the supplementary material., we set $b_{1}$ and $\gamma_{j}$ to 2,
which ensures that all the decomposition bitwidths are power-of-two. Moreover,
since the normalized quantization error change is small when the bitwidth is
greater than 8 as indicated in Corollary 1 and Figure 2, we only consider
those bitwidths that are not greater than 8-bit (_i.e._ ,
$b_{j}\in\\{2,4,8\\}$) in our paper.

###  3.2 Single-path bit sharing model compression

####  3.2.1 Binary gate for quantization

In a neural network, different layers have diverse redundancy and contribute
differently to the accuracy and efficiency of the network. To determine the
bitwidth for each layer, we introduce a layer-wise binary quantization gate
$g_{b_{j}}\in\\{0,1\\}(j>1)$ on each of the re-assignment offsets in Eq.
(LABEL:eq:bit_decompistion) to encode the choice of the quantization bitwidth
$b_{j}$ as:

$\displaystyle g_{b_{j}}$
$\displaystyle=H\left(||z-z_{b_{j-1}}||-\alpha_{b_{j}}\right),$ (10)
$\displaystyle z_{b_{K}}=z_{b_{1}}+g_{b_{2}}$
$\displaystyle\big{(}r_{b_{2}}+\cdots+g_{b_{K-1}}\big{(}r_{b_{K-1}}+g_{b_{K}}r_{b_{K}}\big{)}\big{)},$

where $\alpha_{b_{j}}(j>1)$ is a layer-wise threshold that controls the choice
of bitwidth and $H(A)$ is the step function which returns 1 if $A\geq 0$ and
returns 0 otherwise. Here, we use the quantization error $||z-z_{b_{j-1}}||$
to determine the choice of bitwidth. Specifically, if the quantization error
$||z-z_{b_{j-1}}||$ is greater than $\alpha_{b_{j}}$, we activate the
corresponding quantization gate to increase the bitwidth so that the residual
error can be reduced.  On the contrary, if quantization error is lower than
$\alpha_{b_{j}}$, the quantization gate will be deactivated, resulting in a
low bit quantization.

####  3.2.2 Binary gate for pruning

Note that in Eq. (10), we can also consider filter pruning as 0-bit filter-
wise quantization, which makes it possible to perform network pruning and
quantization jointly. To avoid the prohibitively large filter-wise search
space, we propose to divide the filters into groups based on channel indices
and consider the group-wise sparsity instead. To be specific, we introduce a
binary gate $g_{c,b_{1}}$ for each group to encode the choice of pruning as:

$\displaystyle g_{c,b_{1}}$ $\displaystyle=H(||w_{c}||-\alpha_{b_{1}}),$ (11)
$\displaystyle z_{c,b_{K}}$
$\displaystyle=g_{c,b_{1}}\cdot\big{(}z_{c,b_{1}}+g_{b_{2}}\big{(}r_{c,b_{2}}+\cdots~{}$
$\displaystyle+g_{b_{K-1}}(r_{c,b_{K-1}}+g_{b_{K}}r_{c,b_{K}})\big{)}\big{)},$

where $z_{c,b_{j}}$ is the $c$-th group filters with $b_{j}$-bit quantization
and $r_{c,b_{j}}$ is the corresponding re-assignment offset obtained by
quantizing the residual error $z_{c}-z_{c,b_{j-1}}$. Here, $\alpha_{b_{1}}$ is
a layer-wise threshold for filter pruning.  Following PFEC [37], we use the
$\ell_{1}$-norm to evaluate the importance of the filter. Specifically, if a
group of filters is important, the corresponding pruning gate will be
activated. In contrast, if a group of filters is uninformative, the
corresponding pruning gate will be deactivated.

###  3.3 Learning for loss-aware compression

####  3.3.1 Gradient approximation for the step function

Note that both quantization and pruning have their corresponding thresholds.
Instead of determining the thresholds manually, we propose to learn them via
gradient descent. Nevertheless, the step function in Eqs. (10) and (11) is
non-differentiable. To address this, we propose to use straight-through
estimator (STE) [1, 76] to approximate the gradient of $H(\cdot)$ by the
gradient of the sigmoid function $S(\cdot)$ , which can be formulated as

$\displaystyle\frac{\partial g}{\partial\alpha}=\frac{\partial
H\left(A-\alpha\right)}{\partial\alpha}$ $\displaystyle\approx\frac{\partial
S\left(A-\alpha\right)}{\partial\alpha}~{}$ (12)
$\displaystyle=-S\left(A-\alpha\right)\left(1-S(A-\alpha)\right),$

where $g$ is the output of a binary gate, $\alpha$ is the corresponding
pruning or quantization threshold and $A$ denotes some metrics (_i.e._ ,
quantization error $||z-z_{b_{j-1}}||$ or the $\ell_{1}$-norm of the c-th
group of filters $||w_{c}||$).

####  3.3.2 Objective function for SBS

To design a hardware-efficient model, the objective function should reflect
both the accuracy and computational cost of a compressed model. Let ${\bf W}$
be the model parameters. Following [4], we incorporate the computational cost
into the objective function and formulate the joint objective as:

$\mathcal{L}({\bf W},\alpha)=\mathcal{L}_{ce}({\bf W},\alpha)+\lambda\log
R(\alpha),$ (13)

where $\mathcal{L}_{ce}(\cdot,\cdot)$ is the cross-entropy loss, $R(\cdot)$ is
the computational cost of the network and $\lambda$ is a hyperparameter that
balances two losses.  Following single-path NAS [58], we use a similar
formulation of computational cost to preserve the differentiability of the
objective function.  For simplicity, we consider performing weight
quantization only. The computational cost $R(\cdot)$ is formulated as follows:

$\displaystyle R(\alpha)$
$\displaystyle=\sum_{l=1}^{L}\sum_{c=1}^{G}g_{c,b_{1}}^{l}\big{(}R_{c,b_{1}}^{l}+g_{b_{2}}^{l}\big{(}R_{c,b_{2}}^{l}-R_{c,b_{1}}^{l}~{}$
(14)
$\displaystyle+\cdots+g_{b_{K}}^{l}(R_{c,b_{K}}^{l}-R_{c,b_{K-1}}^{l})\big{)}\big{)},$

where $R_{c,b_{j}}^{l}$ is the computational cost of the $c$-th group of
filters with $b_{j}$-bit quantization for layer $l$. $g_{b_{j}}^{l}$ and
$g_{c,b1}^{l}$ are the binary gates for $b_{j}$-bit quantization and the
pruning decision for $c$-th group of filters, respectively. Similarly, the
computational cost for activation quantization can be easily derived by
replacing the binary gates of weights with those of activations. Note that in
Eq. (12), we approximate the gradient of the step function $H(\cdot)$ by the
gradient of the sigmoid function $S(\cdot)$. Therefore, the computational cost
$R(\cdot)$ remains differentiable.

By jointly training the binary gates and model parameters, we are able to make
a trade-off between pruning and quantization. More importantly, the
configurations of each layer are automatically determined. However, the
gradient approximation of the binary gate may inevitably introduce noisy
signals, which is more severe when we quantize both weights and activations.
To alleviate this issue, we propose to train the binary gates of weights and
activations in an alternating manner. Let $\\{\alpha_{w}^{l}\\}_{l=1}^{L}$ and
$\\{\alpha_{x}^{l}\\}_{l=1}^{L}$ be the thresholds for weights quantization
and activations quantization, respectively. The training algorithm of the
proposed SBS is shown in Algorithm 1. An empirical study over the effect of
alternating training scheme is put in Section 5.2. Once the training is
finished, we can obtain the resulting compressed model by selecting those
filters and bitwidths with activated binary gates. Then, we fine-tune the
resulting compressed model to compensate for the accuracy loss.

Algorithm 1 Training method for SBS

Input: A pre-trained model $M^{0}$, the number of candidate bitwidths $K$, a
sequence of candidate bitwidths $\\{b_{j}\\}_{j=1}^{K}$, the number of
training epochs $T$, and hyperparameters $\lambda$.
Output: A compressed model $M$.

1: Initialize $M$ using $M^{0}$.

2: Initialize the threshold for weights quantization
$\\{\alpha_{w}^{l}\\}_{l=1}^{L}$ and activations quantization
$\\{\alpha_{x}^{l}\\}_{l=1}^{L}$ to zero.

3: for $t=1,\dots,T$ do

4: Update ${\rm{\bf W}}$ and $\\{\alpha_{w}^{l}\\}_{l=1}^{L}$ by minimizing
Eq. (13).

5: Update ${\rm{\bf W}}$ and $\\{\alpha_{x}^{l}\\}_{l=1}^{L}$ by minimizing
Eq. (13).

6: end for

Figure 3: Comparison between the single-path scheme (Ours) and multi-path
scheme [67].

###  3.4 More discussions on SBS

Our single-path scheme is different from the multi-path scheme in DNAS [67].
To study the difference, for simplicity, we consider a linear regression
scenario and compare the quantization errors incurred by the two schemes over
the regression weights. Formally, we have the following results on the
quantization errors incurred by the two schemes.

######  Proposition 1.

Consider a linear regression problem $\min\limits_{{\bf
w}\in{\mathbb{R}}^{d}}\underset{({\bf
x},y)\sim{\mathcal{D}}}{{\mathbb{E}}}[(y-{\bf w}{\bf x})^{2}]$ with data pairs
$\\{({\bf x},y)\\}$, where ${\bf x}\in{\mathbb{R}}^{d}$ is sampled from
${\mathcal{N}}(0,\sigma^{2}\bf I)$ and $y\in{\mathbb{R}}$ is its response.
Consider using SBS and DNAS to quantize the linear regression weights. Let
${\bf w}_{L}^{t}$ and ${\bf w}_{D}^{t}$ be the quantized regression weights of
SBS and DNAS at the iteration $t$ of the optimization, respectively. Then the
following equivalence holds during the optimization process:

$\displaystyle\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{L}}\,{\bf
x})^{2}]\small{=}\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{D}}\,{\bf x})^{2}],$
(15) $\displaystyle\mathrm{where}~{}{\bf w}^{t}_{\text{L}}\small{=}{\bf
w}^{t}_{b_{1}}\\!\\!\small{+}g_{b_{2}}^{t}\big{(}{\bf
r}^{t}_{b_{2}}\\!\small{+}\\!\cdots+\\!g_{b_{K-1}}^{t}\\!\big{(}{\bf
r}^{t}_{b_{K-1}}\\!\small{+}g_{b_{K}}^{t}\\!{\bf
r}^{t}_{b_{K}}\\!\big{)}\\!\big{)},$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}{\bf r}^{t}_{b_{j}}=D({\bf
w}^{t}_{\text{L}}-{\bf
w}^{t}_{{\text{L}},{b_{j-1}}},s_{b_{j}}),~{}j=2,\cdots,K,$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}{\bf
w}^{t}_{\text{D}}=\sum_{i=1}^{K}p_{i}^{t}{\bf
w}_{b_{i}}^{t},~{}~{}~{}\sum_{i=1}^{K}p^{t}_{i}=1.$

From Proposition 1, the multi-path scheme (DNAS) converges to our single-path
scheme during the optimization. However, our single-path scheme contains fewer
parameters and consumes less computational overhead.  For example, the multi-
path scheme maintains $K$ paths while our proposed single path scheme
maintains only single-path. Thus, the parameters and computational cost
reduction is $(K-1)/K$.

To further study the difference between our single-path scheme and multi-path
scheme, we consider a linear regression scenario $\min\limits_{{\bf
w}\in{\mathbb{R}}^{d}}\underset{({\bf
x},y)\sim{\mathcal{D}}}{{\mathbb{E}}}[(y-{\bf w}{\bf x})^{2}]$ and compare the
quantization errors incurred by the two quantization schemes over the
regression weights. Specifically, we construct a toy linear regression dataset
$\\{({\bf x}_{i},y_{i})\\}_{i=1}^{N}$ where we randomly sample
$N\small{=}10,000$ data ${\bf x}\in{\mathbb{R}}^{10}$ from Gaussian
distribution ${\mathcal{N}}(0,\bf I)$ and obtain corresponding responses
$y={\bf w}^{*}{\bf x}+\Delta$. Here, ${\bf w}^{*}\in{\mathbb{R}}^{10}$ is a
fixed weight randomly sampled from $[0,1]^{10}$ and $\Delta\in{\mathbb{R}}$ is
a noise sampled from ${\mathcal{N}}(0,1)$. We then apply LBS and DNAS to
quantize the regression weights. As shown in Figure 3, our method converges
faster and smoother than the multi-path scheme.

TABLE I: Comparisons of different methods w.r.t. Bit-Operation (BOP) count on
CIFAR-100. “BOP comp. ratio” denotes the BOP compression ratio.

Network | Method | BOPs (M) |  | BOP comp. ratio
---
| Search Cost
---
(GPU hours)
Top-1 Acc. (%) | Top-5 Acc. (%)
ResNet-20 | Full-precision | 41798.6 | 1.0 | – | 67.5 | 90.8
4-bit precision | 674.6 | 62.0 | – | 67.8$\pm$0.3 | 90.4$\pm$0.2
2-7 | DQ [61] | 1180.0 | 35.4 | 2.2 | 67.7$\pm$0.6 | 90.4$\pm$0.5
| HAQ [63] | 653.4 | 64.0 | 5.8 | 67.7$\pm$0.1 | 90.4$\pm$0.3
| DNAS [67] | 660.0 | 62.9 | 2.8 | 67.8$\pm$0.3 | 90.4$\pm$0.2
2-7 | SBS-P (Ours) | 28586.5 | 1.5 | 0.2 | 67.9$\pm$0.1 | 90.7$\pm$0.2
| SBS-Q (Ours) | 649.5 | 64.4 | 0.8 | 68.1$\pm$0.1 | 90.5$\pm$0.0
| SBS (Ours) | 630.6 | 66.3 | 1.0 | 68.1$\pm$0.3 | 90.6$\pm$0.2
ResNet-56 | Full-precision | 128771.7 | 1.0 | – | 71.7 | 92.2
4-bit precision | 2033.6 | 63.3 | – | 70.9$\pm$0.3 | 91.2$\pm$0.4
2-7 | DQ [61] | 2222.9 | 57.9 | 7.0 | 70.7$\pm$0.2 | 91.4$\pm$0.4
| HAQ [63] | 2014.9 | 63.9 | 12.9 | 71.2$\pm$0.1 | 91.1$\pm$0.2
| DNAS [67] | 2016.8 | 63.8 | 7.6 | 71.2$\pm$0.1 | 91.3$\pm$0.2
2-7 | SBS-P (Ours) | 87021.6 | 1.5 | 0.4 | 71.5$\pm$0.1 | 91.8$\pm$0.2
| SBS-Q (Ours) | 1970.7 | 65.3 | 1.3 | 71.5$\pm$0.2 | 91.5$\pm$0.2
| SBS (Ours) | 1918.8 | 67.1 | 1.5 | 71.6$\pm$0.1 | 91.8$\pm$0.4

TABLE II:  Comparisons of different methods w.r.t. memory footprints on
CIFAR-100. “M.f. comp. ratio” denotes the memory footprints compression ratio.

Network | Method | Memory footprints (KB) | M.f. comp. ratio | Top-1 Acc. (%) | Top-5 Acc. (%)
---|---|---|---|---|---
ResNet-56 | Full-precision | 5653.4 | 1.0 | 71.7 | 92.2
4-bit precision | 711.7 | 7.9 | 70.9$\pm$0.3 | 91.2$\pm$0.4
2-7 | DQ [61] | 723.2 | 7.8 | 70.9$\pm$0.4 | 91.7$\pm$0.3
| HAQ [63] | 700.0 | 8.1 | 71.3$\pm$0.1 | 91.1$\pm$0.1
| DNAS [67] | 708.9 | 8.0 | 71.5$\pm$0.2 | 91.3$\pm$0.1
2-7 | SBS-P (Ours) | 4778.0 | 1.2 | 71.5$\pm$0.1 | 91.8$\pm$0.2
| SBS-Q (Ours) | 674.5 | 8.4 | 71.5$\pm$0.2 | 91.6$\pm$0.2
| SBS (Ours) | 657.3 | 8.6 | 71.6$\pm$0.1 | 91.8$\pm$0.4

##  4 Experiments

###  4.1 Compared methods

To investigate the effectiveness of SBS, we consider the following methods for
comparisons: SBS: our proposed method with joint pruning and quantization;
SBS-Q: SBS with quantization only; SBS-P: SBS with pruning only; and several
state-of-the-art model compression methods: DNAS [67]: uses multi-path search
scheme to search for optimal bitwidths; HAQ [63]: uses reinforcement learning
to automatically determine the quantization policies; HAWQ [15]: uses second-
order Hessian information to guide the bitwidth search of each layer; DQ [61]:
uses gradient-based method to learn the bitwidths; DJPQ [73]: combines the
variational information bottleneck method to structured pruning and mixed-bit
precision quantization and learns optimal configurations using gradient-based
method; Bayesian Bits [62]: casts configuration search problem of pruning and
quantization to a variational inference problem and use gradient-based method
to learn configurations. We do not consider other methods since they have been
compared in the methods mentioned above.

###  4.2 Datasets and evaluation metrics

We evaluate the proposed SBS on two image classification datasets, including
CIFAR-100 [34] and ImageNet [12]. CIFAR-100 consists of 50k training samples
and 10k testing images with 100 classes. ImageNet contains 1.28 million
training samples and 50k testing images for 1,000 classes.

We measure the performance of different methods using the Top-1 and Top-5
accuracy. Experiments on CIFAR-100 are repeated 5 times and we report the mean
and standard deviation. For fair comparisons, we measure the computational
cost by the Bit-Operation (BOP) count for all the compared methods following
[23, 73]. The BOP compression ratio is defined as the ratio between the total
BOPs of the uncompressed and compressed model. We can also measure the
computational cost by the total weights and activations memory footprints
following DQ [61]. Similarly, the memory footprints compression ratio is
defined as the ratio between the total memory footprints of the uncompressed
and compressed model. Moreover, following [58, 41], we use the search cost on
a GPU device (NVIDIA TITAN Xp) to measure the time of finding an optimal
compressed model.

###  4.3 Implementation details

To demonstrate the effectiveness of the proposed method, we apply SBS to
various architectures, such as ResNet [25] and MobileNetV2 [55]. All
implementation are based on PyTorch [50]. Following HAQ [63], we quantize all
the layers, in which the first and the last layers are quantized to 8-bit.
Following ThiNet [48], we only perform filter pruning for the first layer in
the residual block. For ResNet-20 and ResNet-56 on CIFAR-100 [34], we set $B$
to 4. For ResNet-18 and MobileNetV2 on ImageNet [54], $B$ is set to 16 and 8,
respectively. We first train the full-precision models and then use the pre-
trained weights to initialize the compressed models. For CIFAR-100, we use the
same data augmentation as in [25], including randomly cropping and horizontal
flipping. For ImageNet, images are resized to $256\times 256$, and then a
$224\times 224$ patch is randomly cropped from an image or its horizontal flip
for training. For testing, a $224\times 224$ center cropped is chosen.

(a) BOPs vs. Top-1 Acc.

(b) Memory vs. Top-1 Acc.

Figure 4: Performance comparisons of different methods with different BOPs and
memory footprints. We use different methods to compress ResNet-56 and report
the results on CIFAR-100.

Following [38], we introduce weight normalization during training. We use SGD
with nesterov [49] for optimization. The momentum term is set to $0.9$. We
first train the uncompressed network for 30 epochs on CIFAR-100 and 10 epochs
on ImageNet. The learning rate is set to $0.001$. We then fine-tune the
searched compressed network to recover the performance drop. On CIFAR-100, we
fine-tune the searched network for $200$ epochs with a mini-batch size of
$128$. The learning rate is initialized to $0.1$ and is divided by 10 at
$80$-th and $120$-th epochs. For ResNet-18 on ImageNet, we fine-tune the
searched network for $15$ epochs with a mini-batch size of 256. For
MobileNetV2 on ImageNet, we fine-tune for 150 epochs. For all models on
ImageNet, the learning rate starts at 0.01 and decays with cosine annealing
[44]. Following [41], we initialize $\\{\alpha_{w}^{l}\\}_{l=1}^{L}$ and
$\\{\alpha_{x}^{l}\\}_{l=1}^{L}$ to zero.

TABLE III:  Comparisons of different methods on ImageNet. “*” denotes that we
get the results from the figures in [62] and “–” denotes that the results are
not reported. “BOP comp. ratio” denotes the BOP compression ratio.

Network | Method | BOPs (G) | BOP comp. ratio | Top-1 Acc. (%) | Top-5 Acc. (%) |
---|---|---|---|---|---|---
ResNet-18 | Full-precision | 1857.6 | 1.0 | 69.8 | 89.1 |
4-bit precision | 34.7 | 53.5 | 69.2 | 89.0 |
2-7 | DJPQ [73] | 35.5 | 52.3 | 69.1 | – |
| DQ [61] | 40.1 | 46.3 | 68.9 | 88.6 |
| HAQ [63] | 34.4 | 54.0 | 69.2 | 89.0 |
| HAWQ [15] | 34.0 | 54.6 | 68.5 | – |
|  Bayesian Bits* [62] | 35.9 | 51.7 | 69.5 | – |
2-7 | SBS-Q (Ours) | 34.4 | 54.0 | 69.5 | 89.0 |
| SBS (Ours) | 34.0 | 54.6 | 69.6 | 89.0 |
MobileNetV2 | Full-precision | 308.0 | 1.0 | 71.9 | 90.3 |
8-bit precision | 19.2 | 16.0 | 71.5 | 90.1 |
2-7 | DQ [61] | 19.6 | 1.9 | 70.4 | 89.7 |
| HAQ [63] | 13.8 | 22.3 | 71.4 | 90.2 |
|  Bayesian Bits* [62] | 13.8 | 22.3 | 71.4 | – |
2-7 | SBS-Q (Ours) | 13.6 | 22.6 | 71.6 | 90.2 |
| SBS (Ours) | 13.6 | 22.6 | 71.8 | 90.3 |

TABLE IV: Effect of the bit sharing scheme. We report the Top-1/Top-5
accuracy, BOPs, search cost, and GPU memory footprints on CIFAR-100. The
search cost and GPU memory are measured on a GPU device (NVIDIA TITAN Xp).

Network | Method | Top-1 Acc. | Top-5 Acc. | BOPs (M) | Search Cost (GPU hours) | GPU Memory (GB)
---|---|---|---|---|---|---
ResNet-20 | w/o bit sharing | 67.8$\pm$0.1 | 90.5$\pm$0.2 | 664.2 | 2.8 | 4.4
w/ bit sharing | 68.1$\pm$0.1 | 90.5$\pm$0.0 | 649.5 | 0.8 | 1.5
ResNet-56 | w/o bit sharing | 71.3$\pm$0.3 | 91.4$\pm$0.4 | 2001.1 | 8.7 | 10.9
w/ bit sharing | 71.5$\pm$0.2 | 91.5$\pm$0.2 | 1970.7 | 1.3 | 3.0

###  4.4 Comparisons on CIFAR-100

We apply SBS to compress ResNet-20 and ResNet-56 and evaluate the performance
on CIFAR-100. We report the results under BOPs and memory footprints
constraints in Tables I and II. From the results, compared with the full-
precision model, 4-bit quantized networks achieve comparable performance. For
example, 4-bit ResNet-20 even outperforms the full-precision network by 0.3%
in the Top-1 accuracy. Compared with full-precision networks, mixed-precision
methods are able to further reduce the BOPs while preserving the performance.
To be specific, for ResNet-56, DNAS achieves 63.8$\times$ BOPs reduction and
8.0$\times$ memory footprints reduction while still achieving nearly lossless
performance. Our proposed SBS-Q outperforms state-of-the-arts DQ, HAQ, and
DNAS with less computational cost and memory footprints. Specifically, our
quantized ResNet-56 outperforms DNAS by $0.3\%$ in terms of the Top-1 accuracy
while achieving 65.3$\times$ BOPs reduction. SBS-Q quantized ResNet-56 with
more memory footprints reduction surpasses those of HAQ by 0.2% on the Top-1
accuracy. By performing pruning and quantization jointly, SBS achieves the
best performance while further reducing computational cost and memory
footprints of the compressed models.

We also show the results of the compressed ResNet-56 with different BOPs and
memory footprints in Figure 4. From the results, SBS consistently outperforms
all the other methods under variant BOPs and memory footprints settings.
Moreover, our proposed SBS achieves significant improvement in terms of BOPs
and memory footprints compared with fixed-precision methods, especially at low
BOPs and memory footprints settings.  For example, compared with the fixed-
precision counterpart, SBS compressed ResNet-56 consumes much less
computational overhead (783.52 vs. 1156.46 BOPs) and fewer memory footprints
(395.25 vs. 536.24 KB) but achieves comparable performance.

To evaluate the efficiency of the proposed method, we also compare the search
cost of different methods and report the results in Table I. From the results,
the search cost of the proposed SBS is much smaller than the state-of-the-art
methods. For example, for ResNet-20, the search cost of SBS is 2.8$\times$
lower than DNAS while the search cost of SBS for ResNet-56 is nearly
5.1$\times$ lower than DNAS. Compared with SBS-Q, SBS only introduces a small
amount of overhead. To be specific, for ResNet-56, SBS only introduces 0.2 GPU
hour compared with SBS-Q. These results show the superior efficiency of SBS.

TABLE V: Resource-constrained compression on BitFusion. We evaluate the
proposed SBS under the latency and energy constraint and report the Top-1 and
Top-5 accuracy on ImageNet.

Network | Method | Latency-constrained | Energy-constrained
---|---|---|---
Acc.-1 (%) | Acc.-5 (%) | Latency (ms) | Acc.-1 (%) | Acc.-5 (%) | Energy (mJ)
MobileNetV2 | 8-bit precision | 71.5 | 90.1 | 24.9 | 71.5 | 90.1 | 37.0
SBS (Ours) | 72.0 | 90.4 | 21.6 | 71.7 | 90.3 | 29.8

###  4.5 Comparisons on ImageNet

To evaluate our proposed SBS on the large-scale dataset, we apply the proposed
method to compress ResNet-18 and MobileNetV2 and report the Top-1/Top-5
accuracy and BOPs in Table III. From the results, 4-bit quantized ResNet only
results in a 0.6% performance drop in the Top-1 accuracy. Also, the 8-bit
MobileNetV2 only leads to a 0.4% performance decline in the Top-1 accuracy.
Compared with fixed-precision quantization, mixed-precision methods are able
to reduce the BOPs while preserving the performance. Notably, our proposed
SBS-Q outperforms the state-of-the-art baselines with less computational cost.
Specifically, SBS-Q compressed MobileNetV2 outperforms the one compressed by
HAQ with more BOPs reduction. Moreover, by combining pruning and quantization,
SBS achieves nearly lossless performance while further reducing the
computational cost of the compressed models. For example, SBS compressed
MobileNetV2 reduces the BOPs by 22.6$\times$ while only resulting in 0.1%
performance degradation in terms of the Top-1 accuracy.

Figure 5: SBS searched bitwidths of ResNet-18 on ImageNet. Figure 6: SBS
searched pruning rates of ResNet-18 on ImageNet.

We also illustrate the detailed compression configurations (_i.e._ , bitwidth
and pruning rate) of each layer from the compressed ResNet-18 in Figures 5 and
6. From the results, our SBS assigns more bitwidths to the weights and
activations in the downsampling layers. One possible reason is that the number
of parameters and computational cost of downsampling layers are much smaller
than other layers. Compressing these layers may lead to a significant
performance decline. For filter pruning, the pruning rate decreases with the
increase of layer index. Moreover, our SBS inclines to prune more filters in
the shallower layers of ResNet-18, which can significantly reduce the number
of parameters and computational overhead. More detailed configurations of the
other compressed models are put in the supplementary material.

##  5 Further Studies

In this section, we conduct further studies for the proposed SBS. 1) We
investigate the effect of the bit sharing scheme in Section 5.1. 2) We explore
the effect of the one-stage compression in Section 5.2. 3) We study the effect
of the alternating training scheme in Section 5.3. 4) We provide more results
on resource-constrained compression in Section 5.4.

###  5.1 Effect of the bit sharing scheme

To investigate the effect of the bit sharing scheme, we apply SBS to quantize
ResNet-20 and ResNet-56 with and without the bit sharing scheme and report the
results on CIFAR-100. Here, SBS without the bit sharing denotes that we
compress the models with the multi-path scheme [67]. We report the Top-1/Top-5
accuracy and BOPs in Table IV. We also present the search cost and GPU memory
footprints measured on a GPU device (NVIDIA TITAN Xp). From the results, SBS
with the bit sharing scheme obtains more compact models and consistently
outperforms the ones without the bit sharing scheme while significantly
reducing the search cost and GPU memory footprints. For example, ResNet-56
with the bit sharing scheme outperforms the counterpart by 0.2% in the Top-1
accuracy and achieves 3.6$\times$ reduction on GPU memory and 6.7$\times$
acceleration during training.

TABLE VI: Effect of the one-stage compression. We report the results of
ResNet-56 on CIFAR-100. “SBS-P $\rightarrow$ SBS-Q” denotes that we
sequentially perform filter pruning and quantization following [24].

Network | Method | Top-1 Acc. | Top-5 Acc. | BOPs (M)
---|---|---|---|---
ResNet-56 | SBS-P $\rightarrow$ SBS-Q | 70.4$\pm$0.1 | 90.8$\pm$0.2 | 1077.7
SBS | 70.8$\pm$0.4 | 91.2$\pm$0.1 | 1042.5

TABLE VII: Effect of the alternating training scheme. We report the results of
ResNet-56 on CIFAR-100.

Network | Method | Top-1 Acc. | Top-5 Acc. | BOPs (M)
---|---|---|---|---
ResNet-56 | Joint training | 71.3$\pm$0.2 | 91.6$\pm$0.3 | 1942.4
Alternating training | 71.6$\pm$0.1 | 91.8$\pm$0.4 | 1918.8

###  5.2 Effect of the one-stage compression

To investigate the effect of the one-stage compression scheme, we perform
model compression with the one-stage and two-stage compression scheme on
ResNet-56. Specifically, one stage compression scheme means that we perform
filter pruning and quantization jointly. The two stage compression scheme is
that we first perform filter pruning and then conduct network quantization
following deep compression [24]. For convenience, we denote the two-stage
compression scheme as SBS-P$\rightarrow$SBS-Q. At the first stage, we have
conduct extensive experiments to find a pruning rate that results in nearly
lossless performance. The Top-1, Top-5 accuracy and BOPs of the pruned model
are 71.7%, 91.5% and 87172.6 M, respectively. Based on the pruned model, we
further apply SBS-Q to obtain the compressed model. From Table VI, the
resulting model obtained by the one-stage scheme with less computational cost
outperforms the two-stage counterpart by 0.4% in both the Top-1 accuracy and
Top-5 accuracy. These results show the superiority of performing pruning and
quantization jointly.

###  5.3 Effect of the alternating training scheme

To investigate the effect of the alternating training scheme introduced in
Algorithm 1, we apply our method to compress ResNet-56 with a joint training
scheme and an alternating training scheme on CIFAR-100. Here, the joint
training scheme denotes that we train the binary gates of weights and
activations jointly. The alternating training scheme indicates that we train
the binary gates of weights and activations in an alternating way, as
mentioned in Section 3.3.2. From the results of Table VII, the model trained
with the alternating scheme achieves better performance than those of the
joint scheme while consuming lower computational overhead, which demonstrates
the effectiveness of the proposed alternating training scheme.

###  5.4 Resource-constrained compression

To investigate the effect of our SBS on hardware devices, we further apply SBS
to compress MobileNetV2 under different resource constraints on the BitFusion
architecture [56] which is a state-of-the-art spatial ASIC design for neural
network accelerator. Instead of using BOPs, we measure the computational cost
by the latency and energy on a simulator of the BitFusion with a batch size of
16. We report the results in Table V. Compared with fixed-precision
quantization, SBS achieves better performance with lower latency and energy.
Specifically, SBS compressed MobileNetV2 with much lower latency and energy
even outperforms 8-bit MobileNetV2 in terms of the Top-1 and Top-5 accuracy.
These results show the effectiveness of our SBS on hardware devices.

##  6 Conclusion and Future Work

In this paper, we have proposed an automatic loss-aware model compression
method called Single-path Bit Sharing (SBS) for pruning and quantization
jointly. The proposed SBS introduces a novel single-path bit sharing model,
which encodes all bitwidths in the search space. Specifically, our SBS is
based on the observation that the quantized values of a high bitwidth are
shared with those of lower bitwidths under some constraints. Therefore, we
have proposed the single-path bit sharing decomposition that decomposes a
quantized representation into the sum of the lowest bitwidth representation
and a series of re-assignment offsets. Based on this, we have further
introduced learnable binary gates to encode the choice of different
compression configurations, including filter-wise 0-bit for filter pruning. By
jointly training the binary gates and network parameters, we are able to make
a trade-off between pruning and quantization. More importantly, the
configuration of each layer can be automatically determined. Experiments on
CIFAR-100 and ImageNet have shown that our method is able to achieve
significant computational cost and memory footprints reduction while
preserving performance. Moreover, our proposed method is able to achieve
latency and energy reduction on BitFusion architecture. In the future, we plan
to combine our proposed SBS with other methods, _e.g._ , neural architecture
search, to find a more compact model with better performance.

## References

  * [1] Y. Bengio, N. Léonard, and A. Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [2] Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. Workshops, pages 2978–2985, 2020.
  * [3] H. Cai, L. Zhu, and S. Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  In Proc. Int. Conf. Learn. Repren., 2019.
  * [4] H. Cai, L. Zhu, and S. Han.  ProxylessNAS: Direct neural architecture search on target task and hardware.  In Proc. Int. Conf. Learn. Repren., 2019.
  * [5] Z. Cai, X. He, J. Sun, and N. Vasconcelos.  Deep learning with low precision by half-wave gaussian quantization.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 5406–5414, 2017\.
  * [6] Z. Cai and N. Vasconcelos.  Rethinking differentiable search for mixed-precision neural networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2346–2355, 2020\.
  * [7] J. Cao, Y. Guo, Q. Wu, C. Shen, J. Huang, and M. Tan.  Improving generative adversarial networks with local coordinate coding.  IEEE Trans. Pattern Anal. Mach. Intell., 2020.
  * [8] P. Chen, J. Liu, B. Zhuang, M. Tan, and C. Shen.  Aqd: Towards accurate quantized object detection.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
  * [9] S. Chen and Q. Zhao.  Shallowing deep networks: Layer-wise pruning based on feature representations.  IEEE Trans. Pattern Anal. Mach. Intell., pages 3048–3056, 2019\.
  * [10] Y. Chen, T. Guan, and C. Wang.  Approximate nearest neighbor search by residual vector quantization.  Sensors, pages 11259–11273, 2010.
  * [11] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan.  PACT: parameterized clipping activation for quantized neural networks.  arXiv preprint arXiv:1805.06085, 2018.
  * [12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 248–255, 2009\.
  * [13] R. Ding, T.-W. Chin, Z. Liu, and D. Marculescu.  Regularizing activation distribution for training binarized deep networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 11408–11417, 2019.
  * [14] X. Dong and Y. Yang.  Network pruning via transformable architecture search.  In Proc. Adv. Neural Inf. Process. Syst., pages 759–770, 2019.
  * [15] Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer.  HAWQ: hessian aware quantization of neural networks with mixed-precision.  In Proc. IEEE Int. Conf. Comp. Vis., pages 293–302, 2019.
  * [16] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha.  Learned step size quantization.  In Proc. Int. Conf. Learn. Repren., 2020.
  * [17] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr.  Res2net: A new multi-scale backbone architecture.  IEEE Trans. Pattern Anal. Mach. Intell., pages 652–662, 2021\.
  * [18] Y. Gong, L. Liu, M. Yang, and L. Bourdev.  Compressing deep convolutional networks using vector quantization.  arXiv preprint arXiv:1412.6115, 2014.
  * [19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Proc. Adv. Neural Inf. Process. Syst., pages 2672–2680, 2014\.
  * [20] Y. Guo, Q. Chen, J. Chen, Q. Wu, Q. Shi, and M. Tan.  Auto-embedding generative adversarial networks for high resolution image synthesis.  IEEE Trans. Multimed., pages 2726–2737, 2019.
  * [21] Y. Guo, A. Yao, and Y. Chen.  Dynamic network surgery for efficient dnns.  In Proc. Adv. Neural Inf. Process. Syst., pages 1379–1387, 2016\.
  * [22] Y. Guo, Y. Zheng, M. Tan, Q. Chen, J. Chen, P. Zhao, and J. Huang.  Nat: Neural architecture transformer for accurate and compact architectures.  In Proc. Adv. Neural Inf. Process. Syst., pages 735–747, 2019.
  * [23] Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun.  Single path one-shot neural architecture search with uniform sampling.  In Proc. Eur. Conf. Comp. Vis., pages 544–560, 2020.
  * [24] S. Han, H. Mao, and W. J. Dally.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  In Proc. Int. Conf. Learn. Repren., 2016.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 770–778, 2016\.
  * [26] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han.  AMC: automl for model compression and acceleration on mobile devices.  In Proc. Eur. Conf. Comp. Vis., pages 815–832, 2018.
  * [27] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang.  Filter pruning via geometric median for deep convolutional neural networks acceleration.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4340–4349, 2019\.
  * [28] Y. He, X. Zhang, and J. Sun.  Channel pruning for accelerating very deep neural networks.  In Proc. IEEE Int. Conf. Comp. Vis., pages 1398–1406, 2017.
  * [29] Z. Huang and N. Wang.  Data-driven sparse structure selection for deep neural networks.  In Proc. Eur. Conf. Comp. Vis., pages 317–334, 2018.
  * [30] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio.  Binarized neural networks.  In Proc. Adv. Neural Inf. Process. Syst., pages 4107–4115, 2016\.
  * [31] Q. Jin, L. Yang, and Z. Liao.  Towards efficient training for neural network quantization.  arXiv preprint arXiv:1912.10207, 2019.
  * [32] S. Jung, C. Son, S. Lee, J. Son, J.-J. Han, Y. Kwak, S. J. Hwang, and C. Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4350–4359, 2019\.
  * [33] S. Jung, C. Son, S. Lee, J. Son, J.-J. Han, Y. Kwak, S. J. Hwang, and C. Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4350–4359, 2019\.
  * [34] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [35] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Proc. Adv. Neural Inf. Process. Syst., pages 1106–1114, 2012\.
  * [36] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Comput., pages 541–551, 1989.
  * [37] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf.  Pruning filters for efficient convnets.  In Proc. Int. Conf. Learn. Repren., 2017.
  * [38] Y. Li, X. Dong, and W. Wang.  Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks.  In Proc. Int. Conf. Learn. Repren., 2020.
  * [39] Z. Li, B. Ni, W. Zhang, X. Yang, and W. Gao.  Performance guaranteed network acceleration via high-order residual quantization.  In Proc. IEEE Int. Conf. Comp. Vis., pages 2603–2611, 2017.
  * [40] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In Proc. IEEE Int. Conf. Comp. Vis., pages 2999–3007, 2017.
  * [41] H. Liu, K. Simonyan, and Y. Yang.  DARTS: Differentiable architecture search.  In Proc. Int. Conf. Learn. Repren., 2019.
  * [42] Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun.  Metapruning: Meta learning for automatic neural network channel pruning.  In Proc. IEEE Int. Conf. Comp. Vis., pages 3295–3304, 2019.
  * [43] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell.  Rethinking the value of network pruning.  In Proc. Int. Conf. Learn. Repren., 2019.
  * [44] I. Loshchilov and F. Hutter.  SGDR: stochastic gradient descent with warm restarts.  In Proc. Int. Conf. Learn. Repren., 2017.
  * [45] Q. Lou, L. Liu, M. Kim, and L. Jiang.  Autoqb: Automl for network quantization and binarization on mobile devices.  arXiv preprint arXiv:1902.05690, 2019.
  * [46] C. Louizos, M. Reisser, T. Blankevoort, E. Gavves, and M. Welling.  Relaxed quantization for discretized neural networks.  In Proc. Int. Conf. Learn. Repren., 2019.
  * [47] C. Louizos, K. Ullrich, and M. Welling.  Bayesian compression for deep learning.  In Proc. Adv. Neural Inf. Process. Syst., pages 3288–3298, 2017\.
  * [48] J.-H. Luo, H. Zhang, H.-Y. Zhou, C.-W. Xie, J. Wu, and W. Lin.  Thinet: Pruning CNN filters for a thinner net.  IEEE Trans. Pattern Anal. Mach. Intell., pages 2525–2538, 2019\.
  * [49] Y. E. Nesterov.  A method for solving the convex programming problem with convergence rate o (1/k^ 2).  In Proceedings of the USSR Academy of Sciences, volume 269, pages 543–547, 1983.
  * [50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Proc. Adv. Neural Inf. Process. Syst., pages 8024–8035, 2019\.
  * [51] H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean.  Efficient neural architecture search via parameter sharing.  In Proc. Int. Conf. Mach. Learn., pages 4092–4101, 2018.
  * [52] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le.  Regularized evolution for image classifier architecture search.  In Proc. AAAI Conf. on Arti. Intel., pages 4780–4789, 2019.
  * [53] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Trans. Pattern Anal. Mach. Intell., pages 1137–1149, 2017\.
  * [54] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  Int. J. Comp. Vis., pages 211–252, 2015.
  * [55] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4510–4520, 2018\.
  * [56] H. Sharma, J. Park, N. Suda, L. Lai, B. Chau, V. Chandra, and H. Esmaeilzadeh.  Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network.  In Int. Symp. Comput. Architecture, pages 764–775, 2018.
  * [57] K. Simonyan and A. Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Proc. Adv. Neural Inf. Process. Syst., pages 568–576, 2014.
  * [58] D. Stamoulis, R. Ding, D. Wang, D. Lymberopoulos, B. Priyantha, J. Liu, and D. Marculescu.  Single-path NAS: designing hardware-efficient convnets in less than 4 hours.  In Eur. Conf. Mach. Learn. Princ. Pract. Knowl. Discovery in Databases, pages 481–497, 2019.
  * [59] Z. Tian, C. Shen, H. Chen, and T. He.  FCOS: Fully convolutional one-stage object detection.  In Proc. IEEE Int. Conf. Comp. Vis., pages 9626–9635, 2019.
  * [60] F. Tung and G. Mori.  CLIP-Q: deep network compression learning by in-parallel pruning-quantization.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 7873–7882, 2018\.
  * [61] S. Uhlich, L. Mauch, F. Cardinaux, K. Yoshiyama, J. A. Garcia, S. Tiedemann, T. Kemp, and A. Nakamura.  Mixed precision dnns: All you need is a good parametrization.  In Proc. Int. Conf. Learn. Repren., 2020.
  * [62] M. van Baalen, C. Louizos, M. Nagel, R. A. Amjad, Y. Wang, T. Blankevoort, and M. Welling.  Bayesian bits: Unifying quantization and pruning.  In Proc. Adv. Neural Inf. Process. Syst., 2020.
  * [63] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han.  HAQ: hardware-aware automated quantization with mixed precision.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 8612–8620, 2019\.
  * [64] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In Proc. Eur. Conf. Comp. Vis., pages 20–36, 2016.
  * [65] T. Wang, K. Wang, H. Cai, J. Lin, Z. Liu, H. Wang, Y. Lin, and S. Han.  Apq: Joint search for network architecture, pruning and quantization policy.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2075–2084, 2020\.
  * [66] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li.  Learning structured sparsity in deep neural networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Proc. Adv. Neural Inf. Process. Syst., pages 2074–2082, 2016.
  * [67] B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer.  Mixed precision quantization of convnets via differentiable neural architecture search.  arXiv preprint arXiv:1812.00090, 2018.
  * [68] S. Xu, H. Li, B. Zhuang, J. Liu, J. Cao, C. Liang, and M. Tan.  Generative low-bitwidth data free quantization.  In Proc. Eur. Conf. Comp. Vis., pages 1–17, 2020.
  * [69] H. Yang, S. Gui, Y. Zhu, and J. Liu.  Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-based approach.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2175–2185, 2020\.
  * [70] Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu.  CARS: continuous evolution for efficient neural architecture search.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1826–1835, 2020\.
  * [71] S. Ye, T. Zhang, K. Zhang, J. Li, J. Xie, Y. Liang, S. Liu, X. Lin, and Y. Wang.  A unified framework of dnn weight pruning and weight clustering/quantization using admm.  arXiv preprint arXiv:1811.01907, 2018.
  * [72] S. Ye, T. Zhang, K. Zhang, J. Li, J. Xie, Y. Liang, S. Liu, X. Lin, and Y. Wang.  A unified framework of DNN weight pruning and weight clustering/quantization using ADMM.  arXiv preprint arXiv:1811.01907, 2018.
  * [73] W. Ying, L. Yadong, and B. Tijmen.  Differentiable joint pruning and quantization for hardware efficiency.  In Proc. Eur. Conf. Comp. Vis., pages 259–277, 2020.
  * [74] R. Zeng, C. Gan, P. Chen, W. Huang, Q. Wu, and M. Tan.  Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization.  IEEE Trans. Image Process., pages 5797–5808, 2019.
  * [75] D. Zhang, J. Yang, D. Ye, and G. Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In Proc. Eur. Conf. Comp. Vis., pages 373–390, 2018.
  * [76] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  arXiv preprint arXiv:1606.06160, 2016.
  * [77] B. Zhuang, L. Liu, M. Tan, C. Shen, and I. Reid.  Training quantized neural networks with a full-precision auxiliary module.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1485–1494, 2020\.
  * [78] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid.  Towards effective low-bitwidth convolutional neural networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 7920–7928, 2018\.
  * [79] B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid.  Structured binary neural networks for accurate image classification and semantic segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 413–422, 2019\.
  * [80] Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, and J. Zhu.  Discrimination-aware channel pruning for deep neural networks.  In Proc. Adv. Neural Inf. Process. Syst., pages 883–894, 2018.

Supplementary Materials for “Single-path Bit Sharing for Automatic Loss-aware
Model Compression”

In the supplementary, we provide the detailed proof of Theorem 1, Corollary 1,
Proposition 1, more details and more experimental results of the proposed SBS.
We organize the supplementary material as follows.

  * •

In Section A, we provide the proof for Theorem 1.

  * •

In Section B, we give the proof for Corollary 1.

  * •

In Section C, we offer more details about the proof of Proposition 1.

  * •

In Section D, we provide more explanations on the hardware-friendly
decomposition.

  * •

In Section E, we give more details about the search space of LBS.

  * •

In Section F, we give more details about the quantization configurations of
different methods.

  * •

In Section G, we offer more results in terms of MobileNetV3 on CIFAR-100.

  * •

In Section H, we report the searched compression configurations of the
compressed models.

##  Appendix A Proof of Theorem 1

To prove Theorem 1, we first provide a lemma of quantization formulas as
follows.

######  Lemma 1.

Let $z\in\left[0,1\right]$ be a normalized full-precision input, and
$\\{b_{j}\\}_{j=1}^{K}$ be a sequence of candidate bitwidths. If $b_{j}$ is an
integer multiple of $b_{j-1}$, _i.e._ ,
$b_{j}\small{=}\gamma_{j}b_{j-1}(j>1)$, where
$\gamma_{j}\in{\mathbb{Z}}^{+}\backslash\\{1\\}$ is a multiplier, then the
quantized $z_{b_{j+1}}$ can be decomposed into the quantized $z_{b_{j}}$ and
quantized value re-assignment $r_{b_{j+1}}$

$\displaystyle z_{b_{j+1}}=z_{b_{j}}$ $\displaystyle+r_{b_{j+1}},$ (A)
$\displaystyle\mathrm{where}~{}r_{b_{j+1}}$
$\displaystyle=D(z-z_{b_{j}},s_{b_{j+1}}),~{}$ $\displaystyle z_{b_{j}}$
$\displaystyle=D(z,s_{b_{j}}),~{}$ $\displaystyle s_{b_{j}}$
$\displaystyle=\frac{1}{2^{b_{j}}-1}.$

###### Proof.

We construct two sequences $\\{A_{m}\\}$ and $\\{B_{n}\\}$ for the
$b_{j}$-quantized value and the $b_{j+1}$-quantized value

$\displaystyle\left\\{0,\frac{1}{2^{b_{j}}-1},\frac{2}{2^{b_{j}}-1},\ldots,\frac{2^{b_{j}}-2}{2^{b_{j}}-1},1\right\\},$
(B)
$\displaystyle\left\\{0,\frac{1}{2^{b_{j+1}}-1},\frac{2}{2^{b_{j+1}}-1},\ldots,\frac{2^{b_{j+1}}-2}{2^{b_{j+1}}-1},1\right\\}.$
(C)

First, we can obtain each value in $\\{A_{m}\\}$ is in $\\{B_{n}\\}$, _i.e._ ,
$\\{A_{m}\\}\subset\\{B_{n}\\}$ since
$2^{b_{j+1}}\small{-}1\small{=}(2^{b_{j}})^{\gamma_{j+1}}\small{-}1$ is
divisible by $2^{b_{j}}\small{-}1$. Then we rewrite the sequence $\\{B_{n}\\}$
as:

$\displaystyle\Bigg{\\{}0,\frac{1}{2^{b_{j+1}}-1},\ldots,\underbrace{\frac{1}{2^{b_{j}}-1}}_{A_{2}},\ldots,\underbrace{\frac{t-1}{2^{b_{j+1}}-1}}_{B_{t}},\underbrace{\frac{t}{2^{b_{j+1}}-1}}_{B_{t+1}},\dots,\underbrace{\frac{2}{2^{b_{j}}-1}}_{A_{3}},\ldots,\frac{2^{b_{j+1}}-2}{2^{b_{j+1}}-1},1\Bigg{\\}}.$
(D)

Note that $A_{1},A_{2},\ldots\in\\{B_{n}\\}$, thus we only focus on the
sequence $B_{n}$. For an attribute $z\in\left[0,1\right]$, it surely falls in
some interval $\left[A_{i},A_{i+1}\right]$. Without loss of generality, we
assume $z\in\left[A_{2},A_{3}\right]$ and $z\in\left[B_{t},B_{t+1}\right]$
(see sequence D).

Next, we will discuss the following cases according to the position of $z$ in
$\left[A_{2},A_{3}\right]$:
1) If $z\in[A_{2},\frac{A_{2}+A_{3}}{2}]$, based on the definition of
$z_{b_{j}}=D(z,s_{b_{j}})$, we have $z_{b_{j}}=A_{2}$. Moreover, if
$z\in[B_{t},\frac{B_{t}+B_{t+1}}{2}]$, then $z_{b_{j+1}}=B_{t}$ and
$r_{b_{j+1}}=D(z-A_{2},s_{b_{j+1}})=B_{t}-A_{2}$. Thus, we get
$z_{b_{j+1}}=z_{b_{j}}+r_{b_{j+1}}$; otherwise if
$z\in(\frac{B_{t}+B_{t+1}}{2},B_{t+1}]$, then $z_{b_{j+1}}=B_{t+1}$ and
$r_{b_{j+1}}=B_{t+1}-A_{2}$. Therefore, we have the same conclusion.
2) If $z\in(\frac{A_{2}+A_{3}}{2},A_{3}]$, similar to the first case, we have
$z_{b_{j}}=A_{3}$. Moreover, if $z\in[B_{t},\frac{B_{t}+B_{t+1}}{2}]$, then
$z_{b_{j+1}}=B_{t}$ and $r_{b_{j+1}}=D(z-A_{3},s_{b_{j+1}})=-(A_{3}-B_{t})$.
Thus we have $z_{b_{j+1}}=z_{b_{j}}+r_{b_{j+1}}$; otherwise if
$z\in(\frac{B_{t}+B_{t+1}}{2},B_{t+1}]$, then $z_{b_{j+1}}=B_{t+1}$ and
$r_{b_{j+1}}=-(A_{3}-B_{t+1})$. Hence we still obtain the same conclusion.

∎

######  Theorem 1.

Let $z\in\left[0,1\right]$ be a normalized full-precision input, and
$\\{b_{j}\\}_{j=1}^{K}$ be a sequence of candidate bitwidths. If $b_{j}$ is an
integer multiple of $b_{j-1}$, _i.e._ ,
$b_{j}\small{=}\gamma_{j}b_{j-1}(j>1)$, where
$\gamma_{j}\in{\mathbb{Z}}^{+}\backslash\\{1\\}$ is a multiplier, then the
following quantized approximation $z_{b_{K}}$ can be decomposed as:

$\displaystyle z_{b_{K}}=z_{b_{1}}+\sum_{j=2}^{K}r_{b_{j}},~{}$ (E)
$\displaystyle\mathrm{where}~{}r_{b_{j}}=D(z-z_{b_{j-1}},s_{b_{j}}),~{}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}z_{b_{j}}=D(z,s_{b_{j}}),~{}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}s_{b_{j}}=\frac{1}{2^{b_{j}}-1}.$

###### Proof.

By summing the equations $z_{b_{j}}\small{=}z_{b_{j-1}}+r_{b_{j}}$ in lemma 1
from $j\small{=}2$ to $K$, we complete the results. ∎

##  Appendix B Proof of Corollary 1

######  Corollary 1.

_(Normalized Quantization Error Bound)_ Given ${\bf z}\in[0,1]^{d}$ being a
normalized full-precision vector, ${\bf z}_{b_{K}}$ being its quantized vector
with bitwidth $b_{K}$, where $d$ is the cardinality of ${\bf z}$. Let
$\epsilon_{K}=\frac{\|{\bf z}-{\bf z}_{b_{K}}\|_{1}}{\|{\bf z}\|_{1}}$ be the
normalized quantization error, then the following error bound w.r.t. $K$ holds

$\displaystyle|\epsilon_{K}\small{-}\epsilon_{K+1}|\leq\frac{C}{2^{b_{K}}\small{-}1},$
(F)

where $~{}C\small{=}\frac{d}{\|{\bf z}\|_{1}}$ is a constant.

###### Proof.

Let ${\bf z}=[z^{1},z^{2},\ldots,z^{d}]$ and ${\bf
z}_{b_{K}}=[z_{b_{K}}^{1},z_{b_{K}}^{2},\ldots,z_{b_{K}}^{d}]$, we rewrite the
normalized quantization error as:

$\epsilon_{K}=\frac{\sum_{i=1}^{d}|z^{i}-z_{b_{K}}^{i}|}{\|{\bf z}\|_{1}}.$
(G)

Then, we get the following result

$\displaystyle|\epsilon_{K}\small{-}\epsilon_{K+1}|\small{=}\frac{1}{\|{\bf
z}\|_{1}}\sum_{i=1}^{d}\left||z^{i}-z_{b_{K}}^{i}|\small{-}|z^{i}-z_{b_{K+1}}^{i}|\right|$
$\displaystyle\small{\leq}\frac{1}{\|{\bf
z}\|_{1}}\sum_{i=1}^{d}\left|z_{b_{K}}^{i}\small{-}z_{b_{K+1}}^{i}\right|\small{=}\frac{1}{\|{\bf
z}\|_{1}}\sum_{i=1}^{d}\left|r_{b_{K+1}}^{i}\right|$ (H)
$\displaystyle\small{\leq}\frac{1}{\|{\bf
z}\|_{1}}\sum_{i=1}^{d}\left|s_{b_{K}}\right|\small{=}\frac{d}{\|{\bf
z}\|_{1}(2^{b_{K}}\small{-}1)}=\frac{C}{2^{b_{K}}-1}.$

∎

To empirically demonstrate Corollary 1, we perform quantization on a random
toy data ${\bf z}\in[0,1]^{100}$ and show the normalized quantization error
change $|\epsilon_{K}\small{-}\epsilon_{K+1}|$ and error bound
$\frac{C}{2^{b_{K}}\small{-}1}$ in Figure I. From the results, the normalized
quantization error change decreases quickly as the bitwidth increases and is
bounded by $C/(2^{b_{K}}\small{-}1)$.

Figure I: The normalized quantization error change and error bound vs.
bitwidth. The normalized quantization error change (red line) is bounded by
$C/(2^{b_{K}}\small{-}1)$ (blue line).

##  Appendix C Proof of Proposition 1

To prove Proposition 1, we derive a lemma of normalized quantization error as
follows.

######  Lemma 2.

Given ${\bf z}\in[0,1]^{d}$ being a normalized full-precision vector and ${\bf
z}_{b_{K}}$ being its quantized vector with bitwidth $b_{K}$, where $d$ is the
cardinality of ${\bf z}$, let $\epsilon_{K}\small{=}\frac{\|{\bf z}-{\bf
z}_{b_{K}}\|_{1}}{\|{\bf z}\|_{1}}$ be the normalized quantization error, then
$\epsilon_{K}$ decreases with the increase of K.

###### Proof.

Let ${\bf z}=[z^{1},z^{2},\ldots,z^{d}]$ and ${\bf
z}_{b_{K}}=[z_{b_{K}}^{1},z_{b_{K}}^{2},\ldots,z_{b_{K}}^{d}]$, we rewrite the
normalized quantization error as:

$\epsilon_{K}=\frac{\sum_{i=1}^{d}|z^{i}-z_{b_{K}}^{i}|}{\|{\bf z}\|_{1}}.$
(I)

Since $z_{b_{K}}$ is the only variable, we focus on one term $z-z_{b_{K}}$ of
the numerator in Eq. (I). Next, we will prove
$|z-z_{b_{j}}|\geq|z-z_{b_{j+1}}|$. Following the sequence (D) in Lemma 1 and
without loss of generality, we assume
$z\in\left[B_{t},B_{t+1}\right]\subset\left[A_{2},A_{3}\right]$, then we have
the following conclusions:
1) If $A_{2}=B_{t}<B_{t+1}<A_{3}$, then
$|z-z_{b_{j}}|=|z-A_{2}|\geq|z-z_{b_{j+1}}|$, where
$|z-A_{2}|{=}|z-z_{b_{j+1}}|$ happens only when
$z\in[B_{t},\frac{B_{t}+B_{t+1}}{2}]$ ;
2) If $A_{2}<B_{t}<B_{t+1}<A_{3}$, then $|z-z_{b_{j}}|\geq
B_{t+1}-B_{t}>\frac{B_{t+1}-B_{t}}{2}\geq|z-z_{b_{j+1}}|$;
3) If $A_{2}<B_{t}<B_{t+1}=A_{3}$, then
$|z-z_{b_{j}}|=|z-A_{3}|\geq|z-z_{b_{j+1}}|$, where
$|z-A_{3}|{=}|z-z_{b_{j+1}}|$ happens only when
$z\in[\frac{B_{t}+B_{t+1}}{2},B_{t+1}]$.

Therefore, we have $|z-z_{b_{j}}|\geq|z-z_{b_{j+1}}|$. Thus, summing from
$i=1$ to $d$ complete the conclusion. ∎

######  Proposition 1.

Consider a linear regression problem $\min\limits_{{\bf
w}\in{\mathbb{R}}^{d}}\underset{({\bf
x},y)\sim{\mathcal{D}}}{{\mathbb{E}}}[(y-{\bf w}{\bf x})^{2}]$ with data pairs
$\\{({\bf x},y)\\}$, where ${\bf x}\in{\mathbb{R}}^{d}$ is sampled from
${\mathcal{N}}(0,\sigma^{2}\bf I)$ and $y\in{\mathbb{R}}$ is its response.
Consider using SBS and DNAS to quantize the linear regression weights. Let
${\bf w}_{L}^{t}$ and ${\bf w}_{D}^{t}$ be the quantized regression weights of
SBS and DNAS at the iteration $t$ of the optimization, respectively. Then the
following equivalence holds during the optimization process

$\displaystyle\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{L}}\,{\bf
x})^{2}]\small{=}\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{D}}\,{\bf x})^{2}],$ (J)
$\displaystyle\mathrm{where}~{}{\bf w}^{t}_{\text{L}}\small{=}{\bf
w}^{t}_{b_{1}}\\!\\!\small{+}g_{b_{2}}^{t}\big{(}{\bf
r}^{t}_{b_{2}}\\!\small{+}\\!\cdots+\\!g_{b_{K-1}}^{t}\\!\big{(}{\bf
r}^{t}_{b_{K-1}}\\!\small{+}g_{b_{K}}^{t}\\!{\bf
r}^{t}_{b_{K}}\\!\big{)}\\!\big{)},$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}{\bf r}^{t}_{b_{j}}=D({\bf
w}^{t}_{\text{L}}-{\bf
w}^{t}_{{\text{L}},{b_{j-1}}},s_{b_{j}}),~{}j=2,\cdots,K,$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}{\bf
w}^{t}_{\text{D}}=\sum_{i=1}^{K}p_{i}^{t}{\bf
w}_{b_{i}}^{t},~{}~{}~{}\sum_{i=1}^{K}p^{t}_{i}=1.$

###### Proof.

We first rewrite the optimization objective as:

$\displaystyle(y\small{-}{\bf w}^{t}\,{\bf x})^{2}\small{=}({\bf
w}^{\ast}\,{\bf x}\small{-}{\bf w}^{t}\,{\bf x}\small{+}\Delta_{{\bf
x}})^{2}\small{=}\left(({\bf w}^{\ast}\small{-}{\bf w}^{t})\,{\bf
x})\right)^{2}\small{+}2\Delta_{{\bf x}}\left\langle{\bf
w}^{\ast}\small{-}{\bf w}^{t},{\bf x}\right\rangle\small{+}\Delta_{{\bf
x}}^{2}\small{=}({\bf r}^{t}\,{\bf x})^{2}\small{+}2\Delta_{{\bf
x}}\left\langle{\bf r}^{t},{\bf x}\right\rangle\small{+}\Delta_{{\bf x}}^{2},$

where ${\bf w}^{*}$ is the optimal value of ${\bf w}$, $\Delta_{{\bf x}}$ is
the regression error that is a constant, and ${\bf r}^{t}$ denotes the weight
quantization error at the $t$-th iteration. Taking expectation, we have
following results

$\displaystyle{\mathbb{E}}_{({\bf x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf
w}^{t}\,{\bf x})^{2}]$ $\displaystyle\small{=}{\mathbb{E}}_{{\bf
x}\sim{\mathcal{N}}(0,\sigma^{2}\bf I)}({\bf r}^{t}\,{\bf
x})^{2}\small{+}2\Delta_{{\bf x}}\left\langle{\bf r}^{t},{\bf
x}\right\rangle\small{+}\Delta_{{\bf x}}^{2}$
$\displaystyle\small{=}{\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf
I)}({\bf r}^{t}\,{\bf x})^{2}\small{+}\Delta_{{\bf x}}^{2}$
$\displaystyle\small{=}{\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf
I)}\left(\sum_{i=1}^{d}{\bf r}_{i}^{t}{\bf
x}_{i}\right)^{2}\small{+}\Delta_{{\bf x}}^{2}$
$\displaystyle\small{=}{\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf
I)}\sum_{i=1}^{d}({\bf r}_{i}^{t}{\bf x}_{i})^{2}\small{+}2\sum\limits_{k\neq
j}{\bf r}_{k}^{t}{\bf r}_{j}^{t}{\bf x}_{k}{\bf x}_{j}\small{+}\Delta_{{\bf
x}}^{2}$ $\displaystyle\small{=}\sum_{i=1}^{d}({\bf
r}_{i}^{t})^{2}{\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf I)}({\bf
x}^{2})\small{+}2\sum\limits_{k\neq j}{\bf r}_{k}^{t}{\bf
r}_{j}^{t}\left({\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf
I)}({\bf x})\right)^{2}\small{+}{\mathbb{E}}_{{\bf
x}\sim{\mathcal{N}}(0,\sigma^{2}\bf I)}\Delta_{{\bf x}}^{2}$
$\displaystyle\small{=}\sum_{i=1}^{d}({\bf
r}_{i}^{t})^{2}{\sigma}^{2}\small{+}{\mathbb{E}}_{{\bf
x}\sim{\mathcal{N}}(0,\sigma^{2}\bf I)}\Delta_{{\bf x}}^{2},$

where we use assumption of Gaussian distribution ${\mathbb{E}}({\bf x})=0$ and
${\mathbb{E}}({\bf x}^{2})=\sigma^{2}$.

The last equation suggests the training quantization error is only related to
the weight quantization error $({\bf r}_{i}^{t})$ in an iteration since the
term ${\mathbb{E}}_{{\bf x}\sim{\mathcal{N}}(0,\sigma^{2}\bf I)}\Delta_{{\bf
x}}^{2}$ is a constant. Followed by Lemma 2, the quantization error decreases
as the bitwidth increases. This means that for each iteration, both the LBS
and the DNAS are able to search for the $K$-bit to lower the quantization
error by gradient descent.

Therefore, with gradient descent, for attribute $\eta/\sigma^{2}>0$, there
exists some iteration $T$ so that when $t>T$, we have

$\displaystyle\left|\sum_{i=1}^{d}(\tilde{{\bf
r}}_{i}^{t})^{2}\small{-}\sum_{i=1}^{d}(\hat{{\bf
r}}_{i}^{t})^{2}\right|\leq\eta/\sigma^{2},$

where $\tilde{{\bf r}}={\bf w}^{\ast}\small{-}{\bf w}_{\text{L}}$ and
$\hat{{\bf r}}={\bf w}^{\ast}\small{-}{\bf w}_{\text{D}}$ denote the weight
quantization error with the LBS and the DNAS, respectively. Then multiplying
$\sigma^{2}$ gives that

$\displaystyle\bigg{|}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}_{L}^{t}\,{\bf
x})^{2}]\small{-}{\mathbb{E}}_{({\bf x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf
w}_{D}^{t}\,{\bf x})^{2}]\bigg{|}\leq\eta.$

Based on the definition of limit, the following result holds

$\displaystyle\lim_{t\rightarrow\infty}\bigg{|}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}_{L}^{t}\,{\bf
x})^{2}]\small{-}{\mathbb{E}}_{({\bf x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf
w}_{D}^{t}\,{\bf x})^{2}]\bigg{|}\small{=}0.$

Therefore, we have

$\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{L}}\,{\bf
x})^{2}]\small{=}\lim_{t\rightarrow\infty}{\mathbb{E}}_{({\bf
x},y)\sim{\mathcal{D}}}[(y\small{-}{\bf w}^{t}_{\text{D}}\,{\bf x})^{2}].$

∎

##  Appendix D More details about hardware-friendly decomposition

In this section, we provide more explanations on hardware-friendly
decomposition. As mentioned in Section 4.1, $b_{1}$ and $\gamma_{j}$ can be
set to arbitrary appropriate integer values (_e.g._ , 2, 3, etc.). By default,
we set $b_{1}$ and $\gamma_{j}$ to 2 for better hardware utilization. On
general-purpose computing devices (_e.g._ , CPU, GPU), $byte$ (8 bits) is the
lowest data type for operations. Other data types and ALU registers are all
composed of multiple bytes in width. By setting $b_{1}$ and $\gamma_{j}$ to 2,
2-bit/ 4-bit/ 8-bit quantization values can be packed into $byte$ (or
$short,int,long$) data type without bit wasting. Otherwise, if $b_{1}$ and
$\gamma_{j}$ set to other values, it is inevitable to have wasted bits when
packing mixed-precision quantized tensors on general-purpose devices. For
example, a 32-bit $int$ data type can be used to store ten 3-bit quantized
values with 2 bits wasted. One might argue that these 2 bits can be leveraged
with the next group of 3-bit data, but it will result in irregular memory
access patterns, which will degrade the hardware utilization more seriously.
Moreover, 8-bit quantization is able to achieve comparable performance with
the full precision counterparts for many networks [16] and the normalized
quantization error change is small when the bitwidth is greater than 8 as
indicated in Corollary 1 and Figure 2. Therefore, there is no need to consider
a bitwidth higher than 8.

##  Appendix E More details about search space for model compression

Given an uncompressed network with $L$ layers, we use $C_{l}$ to denote the
number of filters at the $l$-th layer. Let $B$ be the number of filters in a
group. For any layer $l$, there would be
$G=\left\lceil\frac{C_{l}}{B}\right\rceil$ groups in total. Since we quantize
both weights and activations, given $K$ candidate bitwidths, there are $K^{2}$
different quantization configurations for each layer. Thus, for the whole
network with $L$ layers, the size of the search space $\Omega$ can be computed
by

$|\Omega|=\prod_{l=1}^{L}\left(K^{2}\times G\right).$ (K)

Eq. (K) indicates that the search space is extremely large when we have large
$L$, $K$, and smaller $B$. For example, the search space of ResNet-18 is
$|\Omega|\approx 4.3\times 10^{34}$ with $K=3$ and $B=16$, which is large
enough to cover the potentially good configurations.

TABLE I:  Comparisons of different methods in terms of MobileNetV3 on
CIFAR-100.

Method | BOPs (M) | BOP comp. ratio | Top-1 Acc. (%) | Top-5 Acc. (%)
---|---|---|---|---
Full-precision | 68170.1 | 1.0 | 76.1 | 93.9
6-bit precision | 2412.6 | 28.3 | 76.1$\pm$0.0 | 93.7$\pm$0.0
1-5 DQ | 2136.3 | 31.9 | 75.9$\pm$0.1 | 93.7$\pm$0.1
HAQ | 2191.7 | 31.1 | 76.1$\pm$0.1 | 93.5$\pm$0.0
DNAS | 2051.9 | 33.2 | 76.1$\pm$0.1 | 93.7$\pm$0.1
1-5 SBS-P (Ours) | 59465.8 | 1.1 | 76.0$\pm$0.0 | 93.5$\pm$0.0
SBS-Q (Ours) | 2021.9 | 33.7 | 76.1$\pm$0.1 | 93.7$\pm$0.1
SBS (Ours) | 2006.6 | 34.0 | 76.1$\pm$0.1 | 93.7$\pm$0.1

##  Appendix F More details about quantization configurations

Note that all the compared methods follow the quantization configurations in
their papers. Specifically, for DQ [61], we parameterize the fixed-point
quantizer using case U3 with $\bm{\theta}=\left[d,q_{\mathrm{max}}\right]$. We
initialize the weights using a pre-trained model. The initial step size is set
to $d=2^{\lfloor\log_{2}(\max(|{\bf W}|)/(2^{b-1}-1)\rfloor}$ for weights and
$2^{-3}$ for activations. The remaining quantization parameters are set such
that the initial bitwidth is 4-bit. For HAQ [63], we first truncate the
weights and activations into the range of $\left[-v_{w},v_{w}\right]$ and
$\left[0,v_{x}\right]$, respectively. We then perform linear quantization for
both weights and activations. To find more proper $v_{w}$ and $v_{x}$, we
minimize the KL-divergence between the original weight distribution ${\bf W}$
and the quantized one $Q({\bf W})$. For DNAS [67], we follow DoReFa-Net [76]
to quantize weights and follow PACT [11] to quantize activations. We
initialize the learnable clipping level to 1. For DJPQ [73] and Bayesian Bits
[62], we directly get the results from original papers. For other methods, we
use the quantization function introduced in Section 3. The trainable
quantization intervals $v_{x}$ and $v_{w}$ are initialized to 1.

##  Appendix G More results in terms of MobileNetV3 on CIFAR-100

To investigate the effectiveness of the proposed SBS on the lightweight
models, we apply our methods to MobileNetV3 on CIFAR-100. Following LSQ+ [2],
we introduce a learnable offset to handle the negative activations in hard-
swish. We show the results in Table I. From the results, our proposed SBS
outperforms the compared methods with much fewer computational cost, which
demonstrates the effectiveness of the proposed SBS on the lightweight models.

##  Appendix H More detailed about the compression configurations of the
compressed models

In this section, we illustrate the detailed compression configurations (_i.e._
, bitwidth and/or pruning rate) of each layer from the quantized and
compressed ResNet-18 and MobileNetV2 in Figures II, III, IV and V. From the
results, our SBS assigns more bitwidths to the weights in the downsampling
convolutional layer in ResNet-18 and depthwise convolutional layer in
MobileNetV2. Intuitively, one possible reason is that the number of parameters
and computational cost of these layers are much smaller than other layers.
Compressing these layers may lead to a significant performance decline.
Moreover, our SBS inclines to prune more filters in the shallower layers of
ResNet-18, which can significantly reduce the number of parameters and
computational overhead. To further explore the relationship between pruning
and quantization, we illustrate the detailed configurations of each layer from
a more compact MobileNetV2, as shown in Figure VI. From the figure, if a layer
is set to a high pruning rate, our SBS tends to select a higher bitwidth to
compensate for the performance decline. In contrast, if a layer is set to a
low pruning rate, our SBS tends to select a lower bitwidth to reduce the model
size and computational costs.

Figure II: Detailed configurations of the quantized ResNet-18. The Top-1
accuracy, Top-5 accuracy and BOPs of the quantized ResNet-18 are 69.5%, 89.0%
and 34.4G, respectively.  Figure III: Detailed configurations of the quantized
MobileNetV2. The Top-1 accuracy, Top-5 accuracy and BOPs of the quantized
MobileNetV2 are 71.6%, 90.2% and 13.6G, respectively.

(a) Bitwidth configuration of the compressed ResNet-18.

(b) Pruning rate configuration of the compressed ResNet-18. The pruning rate
is defined as the ratio between $\\#$pruned weights of the compressed model
and $\\#$weights of the uncompressed model.

Figure IV: Detailed configurations of the compressed ResNet-18. The Top-1
accuracy, Top-5 accuracy and BOPs of the compressed ResNet-18 are 69.6%, 89.0%
and 34.0G, respectively.

(a) Bitwidth configuration of the compressed MobileNetV2.

(b) Pruning rate configuration of the compressed MobileNetV2. The pruning rate
is defined as the ratio between $\\#$pruned weights of the compressed model
and $\\#$weights of the uncompressed model.

Figure V: Detailed configurations of the compressed MobileNetV2. The Top-1
accuracy, Top-5 accuracy and BOPs of the compressed MobileNetV2 are 71.8%,
90.3% and 13.6G, respectively.

(a) Bitwidth configuration of the compressed MobileNetV2.

(b) Pruning rate configuration of the compressed MobileNetV2. The pruning rate
is defined as the ratio between $\\#$pruned weights of the compressed model
and $\\#$weights of the uncompressed model.

Figure VI: Detailed configurations of the compressed MobileNetV2. The Top-1
accuracy, Top-5 accuracy and BOPs of the compressed MobileNetV2 are 71.4%,
90.1% and 10.8G, respectively.