# BiTAT: Neural Network Binarization
with Task-dependent Aggregated Transformation

Geon Park1 Jaehong Yoon111footnotemark: 1 Haiyang Zhang3 Xing Zhang3
Sung Ju Hwang${}^{1~{}2}$ Yonina C. Eldar3
KAIST1 AITRICS2 Weizmann Institute of Science3 Equal contribution.

###### Abstract

Neural network quantization aims to transform high-precision weights and
activations of a given neural network into low-precision weights/activations
for reduced memory usage and computation, while preserving the performance of
the original model. However, extreme quantization (1-bit weight/1-bit
activations) of compactly-designed backbone architectures (e.g., MobileNets)
often used for edge-device deployments results in severe performance
degeneration. This paper proposes a novel Quantization-Aware Training (QAT)
method that can effectively alleviate performance degeneration even with
extreme quantization by focusing on the inter-weight dependencies, between the
weights within each layer and across consecutive layers. To minimize the
quantization impact of each weight on others, we perform an orthonormal
transformation of the weights at each layer by training an input-dependent
correlation matrix and importance vector, such that each weight is
disentangled from the others. Then, we quantize the weights based on their
importance to minimize the loss of the information from the original
weights/activations. We further perform progressive layer-wise quantization
from the bottom layer to the top, so that quantization at each layer reflects
the quantized distributions of weights and activations at previous layers. We
validate the effectiveness of our method on various benchmark datasets against
strong neural quantization baselines, demonstrating that it alleviates the
performance degeneration on ImageNet and successfully preserves the full-
precision model performance on CIFAR-100 with compact backbone networks.

##  1 Introduction

Over the past decade, deep neural networks have achieved tremendous success in
solving various real-world problems, such as image/text generation [6, 15],
unsupervised representation learning [11, 4, 38], and multi-modal training
[32, 37, 28]. Recently, network architectures that aim to solve target tasks
are becoming increasingly larger, based on the empirical observations of their
improved performance. However, as the models become larger, it is increasingly
difficult to deploy them on resource-limited edge devices with limited memory
and computational power. Therefore, many recent works focus on building
resource-efficient deep neural networks to bridge the gap between the scale of
deep neural networks and actual permissible computational complexity/memory-
bounds for on-device model deployments. Some of these works consider designing
computation- and memory-efficient modules for neural architectures, while
others focus on compressing a given neural network by either pruning its
weights [7, 12, 19, 36] or reducing the bits used to represent the weights and
activations [3, 8, 18]. The last approach, _neural network quantization_ , is
beneficial for building on-device AI systems since the edge devices oftentimes
only support low bitwidth-precision parameters and/or operations. However, it
inevitably suffers from the non-negligible forgetting of the encoded
information from the full-precision models. Such loss of information becomes
worse with extreme quantization into binary neural networks with 1-bit weights
and 1-bit activations [3, 39, 29, 27].

How can we then effectively preserve the original model performance even with
extremely low-precision deep neural networks? To address this question, we
focus on the somewhat overlooked properties of neural networks for
quantization: the weights in a layer are highly correlated with each other and
weights in the consecutive layers. Quantizing the weights will inevitably
affect the weights within the same layer, since they together comprise a
transformation represented by the layer. Thus, quantizing the weights and
activations at a specific layer will adjust the correlation and relative
importance between them. Moreover, it will also largely impact the next layer
that directly uses the output of the layer, which together comprise a function
represented by the neural network.

Method |  BRECQ [18] |  DBQ [8] |  ReActNet [21] | Ours
---|---|---|---|---
$\text{bit}_{w}$ / $\text{bit}_{a}$ | $2/4$ | $4/8$ | $1/1$ | $1/1$
correlation | block | N/A | N/A | block
task-based $Q$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$
structured | node | $\times$ | $\times$ | dynamic
approach |  PTQ1 |  QAT2 | QAT | QAT
T1@ImgNet | 66.60% | 70.5% | 68.26% | 68.51%
FLOPs $\times 10^{7}$ | 3.31 | 3.60 | 1.2 | 1.2

  * 1

1Post-training Quantization

  * 2

2Quantization-aware Training

Figure 1: Left: An Illustration of our proposed method. Weight elements in a
layer is highly correlated to each other along with the weights in other
layers. Our BiTAT sequentially obtains quantized weights of each layer based
on the importance of disentangled weights to others using a trainable
orthonormal rotation matrix and importance vector. Right: Categorization of
relevant and strong quantization methods to ours.

Despite their impact on neural network quantization, such inter-weight
dependencies have been relatively overlooked. As shown in Figure 1 Right,
although BRECQ [18] addresses the problem by considering the dependency
between filters in each block, it is limited to the Post-Training Quantization
(PTQ) problem, which suffers from inevitable information loss, resulting in
inferior performance. The most recent Quantization-Aware Training (QAT)
methods [8, 21] are concerned with obtaining quantized weights by minimizing
quantization losses with parameterized activation functions, disregarding
cross-layer weight dependencies during training process. To the best of our
knowledge, no prior work explicitly considers dependencies among the weights
for QAT.

To tackle this challenging problem, we propose a new QAT method, referred to
as Neural Network Binarization with Task-dependent Aggregated Transformation
(BiTAT), as illustrated in Figure 1 Left. Our method sequentially quantizes
the weights at each layer of a pre-trained neural network based on chunk-wise
input-dependent weight importance by training orthonormal dependency matrices
and scaling vectors. While quantizing each layer, we fine-tune the subsequent
full-precision layers, which utilize the quantized layer as an input for a few
epochs while keeping the quantized weights frozen. we aggregate redundant
input dimensions for transformation matrices and scaling vectors,
significantly reducing the computational cost of the quantization process.
Such consideration of inter-weight dependencies allows our BiTAT algorithm to
better preserve the information from a given high-precision network, allowing
it to achieve comparable performance to the original full-precision network
even with extreme quantization, such as binarization of both weights and
activations. The main contributions of the paper can be summarized as follows:

  * •

We demonstrate that weight dependencies within each layer and across layers
play an essential role in preserving the model performance during quantized
training.

  * •

We propose an input-dependent quantization-aware training method that
binarizes neural networks. We disentangle the correlation in the weights from
across multiple layers by training rotation matrices and importance vectors,
which guides the quantization process to consider the disentangled weights’
importance.

  * •

We empirically validate our method on several benchmark datasets against
state-of-the-art neural network quantization methods, showing that it
significantly outperforms baselines with the compact neural network
architecture.

##  2 Related Work

#### Minimizing the quantization error.

Quantization methods for deep neural networks can be broadly categorized into
several strategies [26]. We first introduce the methods by minimizing the
quantization error. Many existing neural quantization methods aim to minimize
the weight/activation discrepancy between quantized models and their high-
precision counterparts. XNOR-Net [30] aims to minimize the least-squares error
between quantized and full-precision weights for each output channel in
layers. DBQ [8] and QIL [14] perform layerwise quantization with parametric
scale or transformation functions optimized to the task. Yet, they quantize
full-precision weight elements regardless of the correlation between other
weights. While TSQ [33] and Real-to-Bin [22] propose to minimize the
$\ell_{2}$ distance between the quantized activations and the real-valued
network’s activations by leveraging intra-layer weight dependency, they do not
consider cross-layer dependencies. Recently, BRECQ [18] and the work in a
similar vein on post-training quantization [24] consider the interdependencies
between the weights and the activations by using a Taylor series-based
approach. However, calculating the Hessian matrix for a large neural network
is often intractable, and thus they resort to strong assumptions such as small
block-diagonality of the Hessian matrix to make them feasible. BiTAT solves
this problem by training the dependency matrices alongside the quantized
weights while grouping similar weights together to reduce the computational
cost.

#### Modifying the task loss function.

A line of methods aims to achieve better generalization performance during
quantization by taking sophisticatedly-designed loss functions. BNN-DL [10]
adds a distributional loss term that enforces the distribution of the weights
to be quantization-friendly. Apprentice [23] uses knowledge distillation (KD)
to preserve the knowledge of the full-precision teacher network in the
quantized network. However, such methods only put a constraint on the
distributional properties of the weights, not the dependencies and the values
of the weight elements. CI-BCNN [34] parameterizes bitcount operations by
exploring the interaction between output channels using reinforcement learning
and quantizes the floating-point accumulation in convolution operations based
on them. However, reinforcement learning is expensive, and it still does not
consider cross-layer dependencies.

#### Reducing the gradient error.

Bi-Real Net [20] devises a better gradient estimator for the sign function
used to binarize the activations and a magnitude-aware gradient correction
method. It further modifies the MobileNetV1’s architecture to better fit the
quantized operations. ReActNet [21] achieves state-of-the-art performance for
binary neural networks by training a generalized activation function for
compact network architecture used in [20]. However, the quantizer functions in
these methods conduct element-wise unstructured compression without
considering the change in other correlated weights during quantization
training. This makes the search process converge to the suboptimal solutions
since task loss is the only guide for finding the optimal quantized weights,
which is often insufficient for high-dimensional and complex architectures.
However, our proposed method can obtain a better-informed guide that compels
the training procedure to spend more time searching in areas that are more
likely to contain high-performing quantized weights.

##  3 Weight Importance for Quantization-aware Training

We first introduce the problem of Quantization-Aware Training (QAT) in Section
3.1 and show that the dependency among the neural network weights plays a
crucial role in preserving the performance of a quantized model obtained with
QAT in Section 3.2. We further show that the dependency between consecutive
layers critically affects the performance of the quantized model in Section
3.3.

###  3.1 Problem Statement

We aim to quantize a full-precision neural network into a binary neural
network (BNN), where the obtained quantized network is composed of binarized
$1$-bit weights and activations, which preserves the performance of the
original full-precision model. Let $f(\cdot;\mathcal{W})$ be a $L$-layered
neural network parameterized by a set of pre-trained weights
$\mathcal{W}=\\{\bm{w}^{(1)},\dots,\bm{w}^{(L)}\\}$, where
$\bm{w}^{(l)}\in\mathbb{R}^{d_{l-1}\times d_{l}}$ is the weight at layer $l$
and $d_{0}$ is the dimensionality of the input. Given a training dataset
$\mathcal{X}$ and corresponding labels $\mathcal{Y}$, existing QAT methods
[30, 8, 14, 2, 35, 25] search for optimal quantized weights by solving for the
optimization problem that can be generally described as follows:

$\displaystyle\underset{\mathcal{W},\boldsymbol{\phi}}{\operatorname*{minimize}}~{}~{}\mathcal{L}_{task}\left(f\left(\mathcal{X};Q\left(\mathcal{W};\boldsymbol{\phi}\right)\right),\mathcal{Y}\right),$
(1)

where $\mathcal{L}_{task}$ is a standard task loss function, such as cross-
entropy loss, and $Q(\cdot;\boldsymbol{\phi})$ is the weight quantization
function parameterized by $\boldsymbol{\phi}$ which transforms a real-valued
vector to a discrete, binary vector. The quantization function used in
existing works typically minimize loss terms based on the Mean Squared Error
(MSE) between the full-precision weights and the quantized weights at each
layer:

$\displaystyle Q(\bm{w}):=\alpha^{*}\bm{b}^{*},\text{~{}~{}~{}~{}
where~{}}\alpha^{*},\bm{b}^{*}=\operatorname*{arg\,min}_{\alpha\in\mathbb{R},\bm{b}\in\\{-1,1\\}^{m}}\left\|\bm{w}-\alpha\bm{b}\right\|_{2}^{2},$
(2)

Figure 2: A simple experiment that cross-layer weight correlation is critical
to find well-performing quantized weights during QAT.

where $m$ is the dimensionality of the target weight. For inference, QAT
methods use $\bm{w}_{q}=Q(\bm{w})$ as the final quantized parameters. They
iteratively search for the quantized weights based on the task loss with a
stochastic gradient descent optimizer, where the model parameters converge
into the ball-like region around the full-precision weights $\bm{w}$.

However, the region around the optimal full-precision weights may contain
suboptimal solutions with high errors. We demonstrate such inefficiency of the
existing quantizer formulation through a simple experiment in Figure 2.
Suppose we have three input points, $\bm{x}_{1},\bm{x}_{2}$, and $\bm{x}_{3}$,
and full-precision weights $\bm{w}$. Quantized training of the weight using
Equation 2 successfully reduces MSE between the quantized weight and the full-
precision, but the task prediction loss using $\bm{w}_{q}$ is nonetheless very
high.

We hypothesize that the main source of error comes from the independent
application of the quantization process to each weight element. However,
neural network weights are not independent, but highly correlated and thus
quantizing a set of weights will largely affect the others. Moreover, after
quantization, the relative importance among weights could also largely change.
Both factors lead to high quantization errors in the pre-activations. On the
other hand, our proposed QAT method _BiTAT_ , described in Section 4, achieves
a quantized model with much smaller MSE. This results from the consideration
of the inter-weights dependencies, which we describe in the next subsection.

###  3.2 Disentangling Weight Dependencies via Input-dependent Orthornormal
Transformation

Figure 3: Solid lines: Test accuracy of a MobileNetV2 model on CIFAR-100
dataset, after adding Gaussian noise to the top 5 rows and the bottom 5 rows
of $\bm{\hat{w}}^{(l)}$ for all layers, considering the dependency on the
lower layers. Dashed lines: Not considering the dependency on the lower
layers. The x axis is in log scale.

How can we then find the low-precision subspace, which contains the best-
performing quantized weights on the task, by exploiting the inter-weight
dependencies? The properties in the input distribution give us some insights
into this question. Let us consider a task composed of centered $N$ training
samples {$\bm{x}_{1},\dots,\bm{x}_{N}\\}=\mathcal{X}\in\mathbb{R}^{N\times
d_{0}}$. We can obtain principal components of the training samples
$\bm{v}_{1},\dots,\bm{v}_{d_{0}}\in\mathbb{R}^{d_{0}}$ and the corresponding
coefficients $\lambda_{1},\dots,\lambda_{d_{0}}\geq 0$, in a descending order.
Let us further suppose that we optimize a single-layered neural network
parameterized by $\bm{w}^{(1)}$. Neurons corresponding to the columns of
$\bm{w}^{(1)}$ are oriented in a similar direction to the principal components
with higher variances (i.e., $\bm{v}_{i}$ than $\bm{v}_{j}$, where $i<j$) that
is much more likely to get activated than the others. We apply a change of
basis to the column space of the weight matrix $\bm{w}^{(1)}$ with the bases
$(\bm{v}_{1},\dots,\bm{v}_{d_{0}})$:

$\displaystyle\bm{V}\widetilde{\bm{w}}^{(1)}$ $\displaystyle=\bm{w}^{(1)}$ (3)
$\displaystyle\widetilde{\bm{w}}^{(1)}$
$\displaystyle={\bm{V}}^{\top}\bm{w}^{(1)},$ (4)

where $\bm{V}=[\bm{v}_{1}\ |\ \cdots\ |\
\bm{v}_{d_{0}}]\in\mathbb{R}^{d_{0}\times d_{0}}$ is an orthonormal matrix.
The top rows of the transformed weight matrix $\widetilde{\bm{w}}^{(1)}$ will
contain more important weights, whereas the bottom rows will contain less
important ones. Therefore, the accuracy of the model will be more affected by
the perturbations of the weights at top rows than ones at the bottom rows.
Note that this transformation can also be applied to the convolutional layer
by “unfolding” the input image or feature map into a set of patches, which
enables us to convert the convolutional weights into a matrix (The detailed
descriptions of the orthonormal transformations for convolutional layers is
provided in the supplementary file).

We can also easily generalize the method to multi-layer neural networks, by
taking the inputs for the $l$-th layer as the “training set”, assuming that
all of the previous layer’s weights are fixed, as follows:

$\displaystyle\left\\{\bm{x}^{(l)}_{i}=\sigma\left({\bm{w}^{(l)}}^{\top}\bm{x}^{(l-1)}_{i}\right)\right\\}_{i=1}^{N},$
(5)

where $\sigma(\cdot)$ is the nonlinear transformation defined by both the non-
linear activations and any layers other than linear transformation with the
weights, such as a average-pooling or Batch Normalization. Then, we
straightforwardly obtain the change-of-basis matrix $V^{(l)}$ and
$\bm{s}^{(l)}$ for layer $l$. The impact of transformed weights is shown in
Figure 3. We compute the principal components of each layer in the initial
pre-trained model and measure the test accuracy when adding the noise to the
top-5 highest-variance _(dashed red)_ or lowest-variance components _(dashed
blue)_ per layer. While a model with perturbed high-variance components
degenerates the performance as the noise scale increases, a model with
perturbed low-variance components consistently obtains high performance even
with large perturbations. This shows that preserving the important weight
components that correspond to high-variance input components is critically
important for effective neural network quantization that can preserve the loss
of the original model.

###  3.3 Cross-layer Weight Correlation Impacts Model Performance

So far, we only described the dependency among the weights within a single
layer. However, dependencies between the weights across different layers also
significantly impact the performance. To validate that, we perform layerwise
sequential training from the bottom layer to the top. At the training of each
layer, the model computes the principal components of the target layer and
adds noise to its top-5 high/low components. As shown in Figure 3, progressive
training with the low-variance components  _(solid blue)_ achieves
significantly improved accuracy over the end-to-end training counterpart
_(dashed blue)_ with a high noise scale, which demonstrates the beneficial
effect of modeling weight dependencies in earlier layers. We describe further
details in the supplementary file.

##  4 Task-dependent Weight Transformation for Neural Network Binarization

Our objective is to train binarized weights $\bm{w}_{q}$ given pre-trained
full-precision weights. We effectively mitigate performance degeneration from
the loss of information incurred by binarization by focusing on the inter-
weight dependencies within each layer and across consecutive layers. We first
reformulate the quantization function $Q$ in Equation 2 with the weight
correlation matrix $\bm{V}$ and the importance vector $\bm{s}$ so that each
weight is disentangled from the others while allowing larger quantization
errors on the unimportant disentangled weights:

$\displaystyle
Q(\bm{w};\bm{s},\bm{V})=\operatorname*{arg\,min}_{\bm{w}_{q}\in\mathbb{Q}}\left\|\operatorname*{diag}(\bm{s})\left({\bm{V}}^{\top}\otimes\bm{w}-{\bm{V}}^{\top}\otimes\bm{w}_{q}\right)\right\|^{2}_{F}+\gamma\left\|\bm{w}_{q}\right\|_{1},$
(6)

where $\bm{s}\in\mathbb{R}^{d_{0}}$ is a scaling term that assigns different
importance scores to each row of $\hat{\bm{w}}$. We denote
$\mathbb{Q}=\\{\bm{\alpha}\odot\bm{b}:\bm{\alpha}\in\mathbb{R}^{d_{out}},\bm{b}\in\\{-1,1\\}^{d_{in}\times
d_{out}}\\}$ as the set of possible binarized values for
$\bm{w}\in\mathbb{R}^{d_{in}\times d_{out}}$ with a scalar scaling factor for
each output channel. The operation $\otimes$ denotes permuted matrix
multiplication with index replacement, where we detail the computation in the
following subsection. We additionally include $\ell_{1}$ norm adjusted by a
hyperparameter $\gamma$. At the same time, we want our quantized model to
minimize the empirical task loss (e.g., cross-entropy loss) for a given
dataset. Thus we formulate the full objective in the form of a bilevel
optimization problem to find the best quantized weights which minimizes the
task loss by considering the cross-layer weight dependencies and the relative
importance among weights:

$\displaystyle\begin{split}\bm{w}^{*},\bm{s}^{*},\bm{V}^{*}=&\operatorname*{arg\,min}_{\bm{w},\bm{s},\bm{V}}\mathcal{L}_{task}\left(f\left(\mathcal{X};\bm{w}_{q}\right),\mathcal{Y}\right),\text{~{}~{}~{}~{}where
}\bm{w}_{q}=Q(\bm{w};\bm{s},\bm{V}).\end{split}$ (7)

After the quantized training, the quantized weights $\bm{w}_{q}^{*}$ are
determined by $\bm{w}_{q}^{*}=Q(\bm{w}^{*};\bm{s}^{*},\bm{V}^{*})$.

In practice, directly solving the above bilevel optimization problem is
impractical due to its excessive computational cost. We therefore instead
consider the following relaxed problem:

$\displaystyle\begin{split}\bm{w}^{*},\bm{s}^{*},\bm{V}^{*}=\operatorname*{arg\,min}_{\bm{w},\bm{s},\bm{V}}\mathcal{L}_{task}\left(f(\mathcal{X};\mathrm{sgn}(\bm{w})),\mathcal{Y}\right)+&\lambda\left\|\operatorname*{diag}(\bm{s}){\bm{V}}^{\top}\otimes\left(\bm{w}-\mathrm{sgn}(\bm{w})\right)\right\|^{2}_{F}\\\
+&\gamma\left\|\mathrm{sgn}(\bm{w})\right\|_{1},\end{split}$ (8)

where $\lambda$ is a hyperparameter to balance between the quantization
objective and task loss. Since it is impossible to compute the gradients for
discrete values in quantized weights, we adopt the straight-through estimator
[1] that is broadly used across QAT methods: $\mathrm{sgn}(\bm{w})$ indicates
the sign function applied elementwise to $\bm{w}$. We follow [21] for the
derivative of $\mathrm{sgn}(\cdot)$. Finally, we obtain the desired quantized
weights by $\bm{w}_{q}^{*}=\mathrm{sgn}(\bm{w^{*}})$. In order to obtain the
off-diagonal parts of the cross-layer dependency matrix $V$, we minimize
Equation 8 with respect to $\bm{s}$ and $\bm{V}$ to dynamically determine the
values (we omit $\mathcal{X}$ and $\mathcal{Y}$ from the arguments of
$\mathcal{L}_{train}(\cdot)$ for readability):

$\displaystyle\begin{split}\mathcal{L}_{train}(\bm{w},\bm{s},\bm{V})=\mathcal{L}_{task}\left(f\left(\mathcal{X};\mathrm{sgn}(\bm{w})\right),\mathcal{Y}\right)+&\lambda\left\|\operatorname*{diag}(\bm{s}){\bm{V}}^{\top}\otimes\left(\bm{w}-\mathrm{sgn}(\bm{w})\right)\right\|_{F}^{2}\\\
+&\gamma\left\|\mathrm{sgn}(\bm{w})\right\|_{1}+Reg(\bm{s},\bm{V}),\end{split}$
(9)

where
$Reg(\bm{s},\bm{V}):=\|\bm{V}\bm{V}^{\top}-\bm{I}\|^{2}+|\sigma-\sum_{i}\log(s_{i})|^{2}$
is a regulariztion term which enforces $\bm{V}$ to be orthogonal and keeps the
scale of $\bm{s}$ constant. Here, $\sigma$ is the constant initial value of
$\sum_{i}\log(s_{i})$, which is a non-negative importance score.

---
Figure 4: Quantization-aware Training with BiTAT: We perform a sequential
training process: quantization training of a layer - rapid finetuning for
upper layers. At each layerwise quantization, we also train the importance
vector and orthonormal correlation matrix, which are initialized by PCA
components of the current and lower layer inputs in the target block, and
guide the quantization to consider the importance of disentangled weights.

###  4.1 Layer-progressive Quantization with Block-wise Weight Dependency

While we obtain the objective function in Equation 9, it is inefficient to
perform quantization-aware training while considering the complete
correlations of all weights in the given neural network. Therefore, we only
consider cross-layer dependencies between only few consecutive layers (we
denote it as a block), and initialize $\bm{s}$ and $\bm{V}$ using Principal
Component Analysis (PCA) on the inputs to those layers within each block.

---
Figure 5: Initialization of the block correlation matrix.

Formally, we define a weight correlation matrix in a neural network block
$V^{(block)}\in\mathbb{R}^{(\sum_{i=1}^{k}d_{i})\times(\sum_{i=1}^{k}d_{i})}$,
where $k$ is the number of layers in a block, similarly to the block-diagonal
formulation in [18] to express the dependencies between weights across layers
in the off-diagonal parts. We initialize $\bm{s}^{(l)}$ and in-diagonal parts
$V^{(l)}$ by applying PCA on the input covariance matrix:

$\displaystyle\bm{s}^{(l)}\leftarrow{(\bm{\lambda}^{(l)})}^{\frac{1}{2}},\quad\bm{V}^{(l)}\leftarrow\bm{U}^{(l)},\quad\quad\text{
where
}~{}\bm{U}^{(l)}\bm{\lambda}^{(l)}(\bm{U}^{(l)})^{\top}:=\frac{1}{N}\sum_{i=1}^{N}\bm{o}_{i}^{(l-1)}{\bm{o}_{i}^{(l-1)}}^{\top},$
(10)

where $\bm{o}^{(l)}$ is the output of $l$-th layer and $\bm{o}^{(0)}=\bm{x}$.
This allows the weights at $l$-th layer to consider the dependencies on the
weights from the earlier layers within the same neural block, and we refer to
this method as sequential quantization. Then, instead of having one set of
$\bm{s}$ and $V$ for each layer, we can keep the previous layer’s $\bm{s}$ and
$V$ and expand them. Specifically, when quantizing layer $l$ which is a part
of the block that starts with the layer $m$, we first apply PCA on the input
covariance matrix to obtain $\bm{\lambda}^{(l)}$ and $U^{(l)}$. We then expand
the existing $\bm{s}^{(m:l-1)}$ and $V^{(m:l-1)}$ to obtain
$\bm{s}\in\mathbb{R}^{D+d_{l-1}}$ and $\bm{s}\in\mathbb{R}^{D+d_{l-1}}$ as
follows111$[\cdot]_{i}$ indicates the $i$-th element of the object inside the
brackets.:

$\displaystyle[\bm{s}^{(m:l)}]_{i}:=\begin{cases}[\bm{s}^{(m:l-1)}]_{i},&i\leq
D,\\\ [(\bm{\lambda}^{(l)})^{\frac{1}{2}}]_{i-D},&D<i,\end{cases}$
$\displaystyle\quad\quad\quad[\bm{V}^{(m:l)}]_{i,j}:=\begin{cases}[\bm{V}^{(m:l-1)}]_{i,j},&i,j\leq
D,\\\ [\bm{U}^{(l)}]_{i-D,j-D},&D<i,j,\\\ 0,&\text{otherwise},\end{cases}$
(11)

where $D=\sum_{i=m}^{l-2}d_{i}$, as illustrated in Figure 4. The weight
dependencies between different layers (i.e., off-diagonal areas) are trainable
and zero-initialized. To enable the matrix multiplication of the weights with
the expanded $\bm{s}$ and $\bm{V}$, we define the expanded block weights as
follows222$[A;B]$ indicates vertical concatenation of the matrices A and B.:

$\displaystyle\bm{w}^{(m:l)}$
$\displaystyle=\begin{bmatrix}\mathrm{PadCol}(\bm{w}^{(m:l-1)},d_{l});\bm{w}^{(l)}\end{bmatrix},$
(12)

where $\mathrm{PadCol}(\cdot,c)$ zero-pads the input matrix to the right by
$c$ columns. Then, our final objective from Equation 9 with cross-layer
dependencies is given as follows:

$\displaystyle\begin{split}\mathcal{L}_{train}(\bm{w}^{(l:L)},\bm{s}^{(m:l)},\bm{V}^{(m:l)})=~{}&\mathcal{L}_{task}\left(f\left(\mathcal{X};\\{\mathrm{sgn}(\bm{w}^{(l)}),\bm{w}^{(l+1:L)}\\}\right),\mathcal{Y}\right)\\\
&+\lambda\left\|\operatorname*{diag}(\bm{s}^{(m:l)}){\bm{V}^{(m:l)}}^{\top}\left(\bm{w}^{(m:l)}-\mathrm{sgn}(\bm{w}^{(m:l)})\right)\right\|_{F}^{2}\\\
&+\gamma\left\|\mathrm{sgn}(\bm{w}^{(m:l)})\right\|_{1}+Reg(\bm{s}^{(m:l)},\bm{V}^{(m:l)}).\end{split}$
(13)

Given the backbone architecture with $L$ layers, we minimize
$\mathcal{L}_{train}(\bm{w}^{(l)},\bm{s}^{(l)},\bm{V}^{(l)})$ with respect to
$\bm{w}^{(l)},\bm{s}^{(l)}$, and $\bm{V}^{(l)}$ to find the desired binarized
weights $\bm{w}_{q}^{*(l)}$ for layer $l$ while keeping the other layers
frozen. Next, we finetune the following layers using the task loss function a
few epochs before performing QAT on following layers, as illustrated in Figure
4. This sequential quantization proceeds from the bottom layer to the top and
the obtained binarized weights are frozen during the training.

Algorithm 1 Neural Network Binarization with Task-dependent Aggregated
Transformation

1: Input: Pre-trained weights $\bm{w}^{(1)},\dots,\bm{w}^{(L)}$ for $L$
layers, task loss function $\mathcal{L}$, Maximum size of input-dimension
group $k$, quantization epochs per layer $N_{ep}$.

2: Output: Quantized weights $\bm{w}^{*(1)},\dots,\bm{w}^{*(L)}$.

3: $\mathcal{B}_{1},\dots,\mathcal{B}_{n}$ ←Divide the neural network into $n$
blocks

4: for each block $\mathcal{B}$ do

5: $\bm{s}=[],\bm{V}=[]$

6: for each layer $l$ in $\mathcal{B}$ do

7: $\bm{o}^{(l-1)}\leftarrow$ inputs for layer $l$

8: $\bm{P}\leftarrow$ if $d_{l-1}>k$ then K-Means($\bm{X}^{(l)}$, $k$) else
$\bm{I}_{d_{l-1}}$ COMMENT$\rhd$ Grouping permutation matrix

9:
$\bm{U}\operatorname*{diag}(\bm{\lambda})\bm{U}^{\top}=\mathtt{PCA}(\bm{P}\bm{o}^{(l-1)})$
COMMENT$\rhd$ Initialization values for the expanded part

10: $\bm{s}\leftarrow[\bm{s};\bm{\lambda}^{\frac{1}{2}}]$,
$\bm{V}\leftarrow\begin{bmatrix}\bm{V}&\boldsymbol{0}\\\
\boldsymbol{0}&\bm{U}\end{bmatrix}$ COMMENT$\rhd$ expand $s$ and $V$

11:
$\bm{w}^{(l:L)},\bm{s},\bm{V}\leftarrow\operatorname*{arg\,min}_{\bm{w},\bm{s},\bm{V}}\mathcal{L}_{train}(\bm{w}^{(l:L)},\bm{s},\bm{V})$
COMMENT$\rhd$ Iterate for $N_{ep}$ epochs

12: $\bm{w}_{q}^{(l)}\leftarrow\mathrm{sgn}(\bm{w}^{(l)})$

###  4.2 Cost-efficient BiTAT via Aggregated Weight Correlation using
Reduction Matrix

We derived a QAT formulation which focues on the cross-layer weight dependency
by learning block-wise weight correlation matrices. Yet, as the number of
inputs to higher layers is often large, the model constructs higher-
dimensional $\bm{V}^{(l)}$ on upper blocks, which is costly. In order to
reduce the training memory footprint as well as the computational complexity,
we aggregate the input dimensions into several small groups based on
functional similarity using $k$-means clustering.

First, we take feature vectors, the outputs of the $l$-th layer
$\bm{o}^{(l)}_{1},\dots,\bm{o}^{(l)}_{N}\in\mathbb{R}^{d_{l}}$ for each output
dimension, to obtain $d_{l}$ points
$\bm{p}_{1},\bm{p}_{2},\dots,\bm{p}_{d_{l}}\in\mathbb{R}^{N}$, then aim to
cluster the points to $k$ groups using $k$-means clustering, each containing
$N/k$ points. Let $g_{i}\in\\{1,2,\dots,k\\}$ indicate the group index of
$\bm{p}_{i}$, for $i=1,\dots,d_{l}$. We construct the reduction matrix
$\bm{P}\in\mathbb{R}^{k\times d_{l}}$, where $P_{ij}=\frac{1}{N/k}$ if
$g_{j}=i$, and $0$ otherwise. Each group corresponds to a single row of the
reduced $\widehat{\bm{V}}^{(l+1)}\in\mathbb{R}^{k\times k}$ instead of the
original dimension $d_{l}\times d_{l}$. In practice, this significantly
reduces the memory consumption of the $\bm{V}$ (down to 0.07%). Now, we
replace ${\bm{s}}$ and ${\bm{V}}^{\top}$ in Equation 13 to $\widehat{\bm{s}}$
and $\widehat{\bm{V}}^{\top}\bm{P}$, respectively, initializing
$\widehat{\bm{s}}$ and $\widehat{\bm{V}}$ with the grouped input covariance
$\frac{1}{N}\sum_{i=1}^{N}(\bm{P}\bm{o}^{(l)}_{i})(\bm{P}\bm{o}^{(l)}_{i})^{\top}$.
We describe the full training process of our proposed method in Algorithm 1.
The total number of training epochs taken in training is $O(LN_{ep})$, where
$L$ is the number of layers, and $N_{ep}$ is the number of epochs for the
quantizing step for each layer.

##  5 Experiments

We validate a new quantization-aware training method, BiTAT, over multiple
benchmark datasets; CIFAR-10, CIFAR-100 [17], and ILSVRC2012 ImageNet [9]
datasets. We use MobileNet V1 [13] backbone network, which is a compact neural
architecture designed for mobile devices. We follow overall experimental
setups from prior works [35, 21].

#### Baselines and training details.

While our method aims to solve the QAT problem, we extensively compare our
_BiTAT_ against various methods; Post-training Quantization (PTQ) method:
BRECQ [18], and Quantization-aware Training (QAT) methods: DBQ [8], EBConv
[3], Bi-Real Net [20], Real-to-Bin [22], LCQ [35], MeliusNet [2], ReActNet
[21]. Note that DBQ, LCQ, and MeliusNet, which keep some crucial layers, such
as 1$\times$1 downsampling layers, in full-precision, leading to inefficiency
at evaluation time. Due to the page limit, we provide the details on
baselines, and the training and inference phase during QAT including
hyperparameter setups in the Supplementary file. We also discuss about the
limitations and societal impacts of our work in the Supplementary file.

Table 1: Performance comparison of BiTAT with baselines. We report the
averaged test accuracy across three independent runs. The best results are
highlighted in bold, and results of cost-expensive models ($10^{8}\uparrow$
ImgNet FLOPs) are de-emphasized in gray. We refer to several results reported
from their own papers, denoted as †.

Methods | Architecture |  Bitwidth Weight / Activ. |  ImgNet FLOPs ($\times 10^{7}$) |  ImgNet acc (%) |  Cifar-10 acc (%) |  Cifar-100 acc (%)
---|---|---|---|---|---|---
Full-precision | ResNet-18 | 32 / 32 | 200.0 | 69.8 | 93.02 | 75.61
MobileNet V2 | 32 / 32 | 31.40 | 71.9 | 94.43 | 68.08
BRECQ [18] | MobileNet V2 | 4 / 4 |  3.31 |  66.57† | - | -
DBQ [8] | MobileNet V2 | 4 / 8 |  3.60 |  70.54† | 93.77 | 73.20
LCQ [35] | ResNet-18 | 2 / 2 | 15.00 | 68.9† | - | -
MobileNet V2 | 4 / 4 |  3.31 |  70.8† | - | -
MeliusNet59 [2] | N/A | 1 / 1 | 24.50 | 70.7† | - | -
Bi-Real Net [20] | ResNet-18 | 1 / 1 | 15.00 | 56.4† | - | -
Real-to-Bin [22] | ResNet-18 | 1 / 1 | 15.00 |  65.4† | - | 76.2†
EBConv [3] | ResNet-18 | 1 / 1 | 11.00 | 71.2† | - | 76.5†
ReActNet-C [21] | MobileNet V1 | 1 / 1 | 14.00 | 71.4† | 90.77 | -
ReActNet-A [21] | MobileNet V1 | 1 / 1 |  1.20 | 68.26 | 89.73 | 65.51
BiTAT (Ours) | MobileNet V1 | 1 / 1 |  1.20 | 68.51 | 90.21 | 68.36

###  5.1 Quantitative Analysis

We compare our BiTAT against various PTQ and QAT-based methods in Table 1 on
multiple datasets. BRECQ introduces an adaptive PTQ method by focusing on the
weight dependency via hessian matrix computations, resulting in significant
performance deterioration and excessive training time. DBQ and LCQ suggest QAT
methods, but the degree of bitwidth compression for the weights and
activations is limited to 2- to 8-bits, which is insufficient to meet our
interest in achieving neural network binarization with 1-bits weights and
activations. MeliusNet only suffers a small accuracy drop, but it has a high
OP count. DBQ and LCQ restrict the bit-width compression to be higher at 4
bits so that they cannot enjoy the XNOR-Bitcount optimization for speedup.
Although Bi-Real Net, Real-to-Bin, and EBConv successfully achieve neural
network binarization, over-parameterized ResNet is adopted as backbone
networks, resulting in higher OP count. Moreover, except EBConv, these works
still suffer from a significant accuracy drop. ReActNet binarizes all of the
weights and activations (except the first and last layer) in compact network
architectures while preventing model convergence failure. Nevertheless, the
method still suffers from considerable performance degeneration of the
binarized model. On the other hand, our BiTAT prevents information loss during
quantized training up to 1-bits, showing a superior performance than ReActNet,
0.37 % $\uparrow$ for ImageNet, 0.53% $\uparrow$ for CIFAR-10, and 2.31%
$\uparrow$ for CIFAR-100. Note that BiTAT further achieves on par performance
of the MobileNet backbone for CIFAR-100. The results support our claim on
layer-wise quantization from the bottom layer to the top, reflecting the
disentangled weight importance and correlation with the quantized weights at
earlier layers.

#### Ablation study

We conduct ablation studies to analyze the effect of salient components in our
proposed method in Figure 7 Left. BiTAT based on layer-wise sequential
quantization without weight transformation already surpasses the performance
of ReActNet, demonstrating that layer-wise progressive QAT through an implicit
reflection of adjusted importance plays a critical role in preserving the pre-
trained models during quantization. We adopt intra-layer weight transformation
using the input-dependent orthonormal matrix, but no significant benefits are
observed. Thus, we expect that only disentangling intra-layer weight
dependency is insufficient to fully reflect the adjusted importance of each
weight due to a binarization of earlier weights/activations. This is evident
that BiTAT considering both intra-layer and cross-layer weight dependencies
achieves improved performance than the case with only intra-layer dependency.
Yet, this requires considerable additional

Figure 6: Visualization of the learned $\bm{V}$ matrix and the
$\operatorname*{diag}(\bm{s})\bm{V}^{\top}$ of three blocks of the network,
with the CIFAR-100 dataset. Notice the off-diagonal parts which represent
cross-layer dependencies.

training time to compute with a chunk-wise transformation matrix. In the end,
BiTAT with aggregated transformations, which is our full method, outperforms
our defective variants in both terms of model performance and training time by
drastically removing redundant correlation through reduction matrices. We note
that using $k$-means clustering for aggregated correlation is also essential,
as another variant, BiTAT with filter-wise transformations, which filter-
wisely aggregates the weights instead, results in deteriorated performance.

Method |  Intra-layer Transform |  Cross-layer Transform |  Accuracy (%) |  Train Time (hours)
---|---|---|---|---
ReActNet [21] | N/A | N/A | 65.51 $\pm$ 0.74 | 10.75
BiTAT (Ours) | $\times$ | $\times$ | 68.17 $\pm$ 0.07 | 3.49
$\checkmark$ | $\times$ | 67.82 $\pm$ 0.22 | 3.66
$\checkmark$ | $\checkmark$ | 68.21 $\pm$ 0.24 | 8.50
w/ Filter-wise Transform | 67.86 $\pm$ 0.11 | 3.01
| w/ Aggregated Transform | 68.36 $\pm$ 0.45 | 3.11

Figure 7: Left: Ablation study for analyzing core components in our method. We
report the averaged performance and 95% confidence interval across 3
independent runs and the complete BiTAT result is highlighted in gray
background. Right: Visualization of the weight grouping during sequential
quantization of BiTAT. Each 3$\times$3 square represents a convolutional
filter of the topmost layer ($26^{th}$, excluding the classifier) of our
model, and each unique color represents each group to which weight elements
belong.

###  5.2 Qualitative Analysis

#### Visualization of Reduction Matrix

We visualize the weight grouping for BiTAT in Figure 7 Right to analyze the
effect of the reduction matrix, which groups the weight dependencies in each
layer based on the similarity between the input dimensions. Each 3$\times$3
square represents a convolutional filter, and each unique color in weight
elements represents which group each weight is assigned to, determined by the
$k$-means algorithm, as described in Section 4.2. We observe that weight
elements in the same filter do not share their dependencies; rather, on
average, they often belong to four-five different weight groups. Opposite to
these observations, BRECQ regards the weights in each filter as the same group
for computing the dependencies in different layers, which is problematic since
weight elements in the same filter can behave differently from each other.

#### Visualization of Cross-layer Weight Dependency

In Figure 6, we visualize learned transformation matrices $\bm{V}$ (top row),
which shows that many weight elements at each layer are also dependent on
other layer weights as highlighted in darker colors, verifying our initial
claim. Further, we provide visualizations for their multiplications with
corresponding importance vectors $\operatorname*{diag}(\bm{s})\bm{V}^{\top}$
(bottom row). Here, the row of $\bm{V}^{\top}$ is sorted by the relative
importance in increasing order at each layer. We observe that important
weights in a layer affect other layers, demonstrating that cross-layer weight
dependency impacts the model performance during quantized training.

##  6 Conclusion

In this work, we explored long-overlooked factors that are crucial in
preventing the performance degeneration with extreme neural network
quantization: the inter-weight dependencies. That is, quantization of a set of
weights affect the weights for other neurons within each layer, as well as
weights in consecutive layers. Grounded by the empirical analyses of the node
interdependency, we propose a Quantization-Aware Training (QAT) method for
binarizing the weights and activations of a given neural network with minimal
loss of performance. Specifically, we proposed orthonormal transformation of
the weights at each layer to disentangle the correlation among the weights to
minimize the negative impact of quantization on other weights. Further, we
learned scaling term to allow varying degree of quantization error for each
weight based on their measured importance, for layer-wise quantization. Then
we proposed an iterative algorithm to perform the layerwise quantization in a
progressive manner. We demonstrate the effectiveness of our method in neural
network binarization on multiple benchmark datasets with compact backbone
networks, largely outperforming state-of-the-art baselines.

## References

  * Bengio et al. [2013] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Bethge et al. [2020] Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel.  MeliusNet: Can binary neural networks achieve MobileNet-level accuracy?  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Bulat et al. [2021] Adrian Bulat, Georgios Tzimiropoulos, and Brais Martinez.  High-capacity expert binary networks.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Courbariaux et al. [2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, 2016.
  * Creswell et al. [2018] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath.  Generative adversarial networks: An overview.  _IEEE Signal Processing Magazine_ , 2018.
  * Dai et al. [2018] Bin Dai, Chen Zhu, Baining Guo, and David Wipf.  Compressing neural networks using the variational information bottleneck.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2018.
  * Dbouk et al. [2020] Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag.  DBQ: A differentiable branch quantizer for lightweight deep neural networks.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * Ding et al. [2019] Ruizhou Ding, Ting-Wu Chin, Zeye Liu, and Diana Marculescu.  Regularizing activation distribution for training binarized deep networks.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Gidaris et al. [2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ , 2018.
  * He et al. [2020] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang.  Learning filter pruning criteria for deep convolutional neural networks acceleration.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Jung et al. [2019] Sangil Jung, Changyong Son, Seohyung Lee, JinWoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2015.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Li et al. [2021] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  BRECQ: Pushing the limit of post-training quantization by block reconstruction.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Lin et al. [2020] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao.  Hrank: Filter pruning using high-rank feature map.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Liu et al. [2018] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng.  Bi-real net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Liu et al. [2020] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng.  ReActNet: Towards precise binary neural network with generalized activation functions.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * Martinez et al. [2020] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos.  Training binary neural networks with real-to-binary convolutions.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Mishra and Marr [2018] Asit Mishra and Debbie Marr.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * Nagel et al. [2020] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Park and Yoo [2020] Eunhyeok Park and Sungjoo Yoo.  PROFIT: A novel training method for sub-4-bit MobileNet models.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * Qin et al. [2020a] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe.  Binary neural networks: A survey.  _Proceedings of the International Conference on Pattern Recognition (ICPR)_ , 2020a.
  * Qin et al. [2020b] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe.  Binary neural networks: A survey.  _Pattern Recognition_ , 2020b.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ , 2021.
  * Rastegari et al. [2016a] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In _European conference on computer vision_. Springer, 2016a.
  * Rastegari et al. [2016b] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016b.
  * Sandler et al. [2019] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  MobileNetV2: Inverted residuals and linear bottlenecks.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Su et al. [2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_ , 2019.
  * Wang et al. [2018] Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng.  Two-step quantization for low-bit neural networks.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Wang et al. [2019] Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian.  Learning channel-wise interactions for binary convolutional neural networks.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Yamamoto [2021] Kohei Yamamoto.  Learnable companding quantization for accurate low-bit neural networks.  _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Yoon and Hwang [2017] Jaehong Yoon and Sung Ju Hwang.  Combined group and exclusive sparsity for deep neural networks.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2017.
  * Zareian et al. [2021] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14393–14402, 2021.
  * Zbontar et al. [2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  _arXiv preprint arXiv:2103.03230_ , 2021.
  * Zhuang et al. [2019] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid.  Structured binary neural networks for accurate image classification and semantic segmentation.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] See supplementary file.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See supplementary file.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See supplementary materials.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See supplementary file.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See
supplementary file.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [No]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [No]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  7 Appendix

###  7.1 Details for Problem Setups

#### Baselines.

While our method aims to solve the QAT problem, we extensively compare our
_BiTAT_ against various Post-training Quantization (PTQ)- or QAT methods:
BRECQ [18] is a PTQ method that considers weight dependencies using the
Hessian of the task loss. DBQ [8] is a QAT method based on continuous
relaxation of the quantizer function. EBConv [3] conditionally selects
appropriate binarized weights based on the task information. Bi-Real Net [20]
adds residual connections to propagate full-precision values, preventing
information loss to activation quantization. Real-to-Bin [22] constrains a
loss term at the end of each convolution to minimize the output discrepancy
between the full-precision and the quantized model. LCQ [35] devises a
trainable quantization function in order to reduce the quantization error.
MeliusNet [2] proposes a new architecture that better propagates full-
precision values throughout the network. ReActNet [21] is the state-of-the-art
binary quantization method, which additionally adopts residual connections,
and element-wise shift operations before/after the activation and the sign
operation. Note that DBQ, LCQ, and MeliusNet keep some crucial layers of
MobileNet in full-precision, leading to inefficiency at evaluation time.

#### Training.

Following the setup from ReActNet [21], we quantize all layers’ weights and
activations except the initial and final layers. We use the Adam optimizer
[16]. For the ImageNet experiment, a learning rate is $0.002$ and $0.0002$ for
quantization training and the fine-tuning, respectively, with linear learning
rate decay. We set batch size as $512$ both the quantization phase and the
fine-tuning phase is done for $5$ epochs per layer. For the CIFAR-100
experiment, a learning rate is $3\times 10^{-4}$ for quantization training and
the fine-tuning with linear learning rate decay. We set batch size as $800$
and both the quantization and fine-tuning are done with $40$ epochs per layer.
For all experiments, we set $\lambda=100$, and $\gamma=10^{-5}$, which notes
that simple choice of the hyperparameters for our regularization terms is
sufficient to show impressive performance. The number of input dimension
groups is set $k=256$, applying the grouped weight correlation to layers with
input dimensions smaller than $k$.

#### Inference.

In deployment, the highly efficient XNOR-Bitcounting operations can be used
for the convolutional layers, also used in existing neural network
binarization works [5, 30, 21].

###  7.2 Extension to Convolutional Layers

Let us consider a convolutional layer of size $n_{out}\times n_{in}\times
k\times k$, where $n_{in}$ and $n_{out}$ are the number of input and output
channels, respectively, and $k$ is the kernel size. We define the set
$\mathcal{P}_{\bm{x}}$ as the set of all patches of size $n_{in}\times k\times
k$ extracted from the training image $\bm{x}$. This patch-extracting operation
is sometimes called im2col or F.unfold in PyTorch.

A convolutional layer applied to $\bm{x}$ can be thought of as a fully-
connected layer individually applied to all patches in $\mathcal{P}_{\bm{x}}$
and then concatenated:

$\displaystyle\bm{w}*\bm{x}=\\{\mathrm{Reshape}_{(n_{in}k^{2})\times(n_{out})}(\bm{w})^{\top}\mathrm{vec}(\bm{p})\\}_{\bm{p}\in\mathcal{P}_{\bm{x}}},$
(14)

where $*$ denotes the convolution operation, $\mathrm{Reshape}_{shape}(\cdot)$
denotes the reshaping of the tensor into the specified _shape_ , and
$\mathrm{vec}(\cdot)$ denotes the flattening operation. Each pixel of the
output feature map corresponds to a matrix multiplication between a patch and
the weight matrix. Therefore, we can analogically apply the same
transformation as explained in Section 3 to convolutional layers.

###  7.3 Details on Cross-layer Dependency

In this section, we further explain the detailed experimental setting for
Figure 3. We take the standard MobileNetV2 [31] model and train it to
convergence on the CIFAR-100 dataset with standard SGD with a weight decay.
Then, we add noise to the same pretrained model parameters before evaluating
the test accuracy based on the following two different ways:

  1. 1.

Layer-dependent noise addition. We first compute the covariance of the input
to the first layer and perform PCA using obtained covariance values to compute
$\widetilde{\bm{w}}^{(1)}$ in Equation 4. Now, we add independent gaussian
noise with varying scales to the top five rows of $\widetilde{\bm{w}}^{(1)}$.
Next, we sequentially repeat the process to the consecutive layers, and after
that, we evaluate the model performance, which is shown in solid red lines.
The same process is done but with the bottom five rows of each layer instead
of the top five, shown in solid blue lines.

  2. 2.

Layer-independent noise addition. Before adding noise to model parameters, we
compute the covariance of the input values for all layers. Next, we perform
PCA and compute $\widetilde{\bm{w}}^{(l)}$ with Equation 4 per layer using
these initial covariance values. That is, a layer cannot reflect the weight
change through noise addition in others layers, as different from the first
approach. Independent gaussian noise with varying scales is added to the top
five rows of $\widetilde{\bm{w}}^{(l)}$ for each layer, and then the
performance of the model is evaluated, shown in red dashed lines. The same is
done with the bottom five rows of $\widetilde{\bm{w}}^{(l)}$, shown in blue
dashed lines.

###  7.4 Additional Analysis

Method |  Accuracy (%) |  $Q_{orig}$ |  $Q_{ours}$
---|---|---|---
ReActNet [21] | 65.51 $\pm$ 0.74 | 13.35 | 475.94
BiTAT (Ours) | 68.36 $\pm$ 0.45 | 39.77 | 434.92

Figure 8: Additional ablation studies. Left: The comparison of the final
$Q_{orig}$ (Equation 2) and $Q_{ours}$ (Equation 6) values in ReActNet and
BiTAT. Right: The evolution of the ratio of $Q_{ours}$ to $Q_{orig}$ in
ReActNet, in comparison to the final ratio in our BiTAT. Cross-layer
dependencies not considered in both computations.

This paper suggests that the proposed quantization loss on disentangled
weights is a better indicator for prediction accuracy than the general
quantization loss (Equation 2), which is evident in multiple validation
analyses and the superior model performance in our BiTAT as described in the
main text. Here, we provide the quantitative analysis to show that ReActNet
[21] fails to minimize the quantization loss on disentangled weights while our
BiTAT successfully does. In Figure 8 Left, we show the $Q_{orig}$ (Equation 2)
and $Q_{ours}$ (Equation 6 w/o $\ell_{1}$ norm) between the initial full-
precision weights of the pre-trained model and the obtained binarized weights
from ReActNet and BiTAT. $Q_{orig}$ represents the naive MSE between the full-
precision weights and the binarized weights. $Q_{ours}$ represents the
dependency-weighted MSE between the full-precision weights and the binarized
weights. Note that, in this analysis, we obtain $\bm{s}$ and $\bm{V}$ for each
layer from the initial pre-trained model by Equation 10 to compute $Q_{ours}$
and neglect the weight dependency across different layers, which is hard to be
computed analytically.

We observe that while the value of $Q_{orig}$ is lower in ReActNet than in
BiTAT, $Q_{ours}$ is higher in ReActNet than in BiTAT. As shown in Figure 8
Right, the ratio $r=Q_{ours}/Q_{orig}$ in ReActNet (Red) drastically increases
at the beginning stage and is maintained in a high degree until the completion
of the quantization-aware training, compared to the $r$ value of the model
obtained by BiTAT (Blue dashed). While disregarding the first few epochs of
ReActNet training, where the accuracy is very low, ReActNet’s $r$ value
dominates that of BiTAT. The value slowly decreases as the ReActNet training
proceeds, but never reaches the level of BiTAT, demonstrating the inefficiency
of the ReActNet training procedure compared to ours.

###  7.5 Limitations

We consider two limitations of our work in this section. First, our BiTAT
framework is built based on a sequential quantization strategy, which
progressively quantizes the layers from the bottom to the top. Therefore, the
training time of our algorithm depends on the number of layers in the backbone
network architecture. While we have already validated the cost-efficiency of
our proposed method against ReActNet using MobileNet ($26$ stacked layers) in
Figure 7 Left, we might spend more training time quantizing all layer weights
for the backbones, composed of much more layers like ResNet-$1001$ ($1001$
stacked layers). Next, our method focuses on the cross-layer weight dependency
within each neural block, including a few consecutive layers. We thereby avoid
the excessive computational cost of obtaining the relationship across all
layers in the backbone architecture, yet we consider it a tradeoff between
accurate dependency and computation budgets.