# Sparsity-Control Ternary Weight Networks

Xiang Deng<EMAIL_ADDRESS>Zhongfei Zhang  State University of New York
at Binghamton, Binghamton, NY, US

###### Abstract

Deep neural networks (DNNs) have been widely and successfully applied to
various applications, but they require large amounts of memory and
computational power. This severely restricts their deployment on resource-
limited devices. To address this issue, many efforts have been made on
training low-bit weight DNNs. In this paper, we focus on training ternary
weight {-1, 0, +1} networks which can avoid multiplications and dramatically
reduce the memory and computation requirements. A ternary weight network can
be consider as a sparser version of the binary weight counterpart by replacing
some -1s or 1s in the binary weights with 0s, thus leading to more efficient
inference but more memory cost. However, the existing approaches to training
ternary weight networks cannot control the sparsity (i.e., percentage of 0s)
of the ternary weights, which undermines the advantage of ternary weights. In
this paper, we propose to our best knowledge the first sparsity-control
approach (SCA) to training ternary weight networks, which is simply achieved
by a weight discretization regularizer (WDR). SCA is different from all the
existing regularizer-based approaches in that it can control the sparsity of
the ternary weights through a controller $\alpha$ and does not rely on
gradient estimators. We theoretically and empirically show that the sparsity
of the trained ternary weights is positively related to $\alpha$. SCA is
extremely simple, easy-to-implement, and is shown to consistently outperform
the state-of-the-art approaches significantly over several benchmark datasets
and even matches the performances of the full-precision weight counterparts.

###### keywords:

ternary weight networks, sparsity control, model compression, image
classification

††journal: Journal of Neural Networks Templates

##  1 Introduction

Deep Neural Networks (DNNs) have obtained state-of-the-art performances on
various tasks across a wide range of domains including computer vision [1, 2,
3, 4] and natural language processing [5, 6, 7]. Currently, there is an
increasing interest in implementing DNNs on portable devices, driven by the
strong and demanding needs in a wide spectrum of applications noticeably
including smart sensor networks, telehealthcare, and security. However, as
DNNs need large amounts of memory and computing power, it is still a
challenging problem on how to effectively implement DNNs on the devices with
constrained memory or limited computing power.

One solution to this problem is to reduce the bit-width of DNN weights, such
as training DNNs with binary weights or ternary weights. In this paper, we
focus on training ternary weights without considering discrete activations.
Ternary weight networks are different from binary weight networks in that they
set some values in the binary weights to zeros, which makes them sparser, thus
being more efficient but also requiring more memory. Ternary weight networks
can be divided into three categories, i.e., the ternary weight with values
constrained to {-1, 0, +1}, the ternary weight with {-$\alpha$, 0, +$\alpha$},
and the ternary weight with {-$\alpha$, 0, +$\beta$} where $\alpha$ and
$\beta$ are real numbers and $\alpha\neq\beta$. Among these three kinds of
ternary weights, the ternary weight with {-1, 0, +1} is the most
efficient111{-$\alpha$, 0, +$\alpha$} is more efficient than {-$\alpha$, 0,
+$\beta$} as it can be considered as {-1, 0, +1} $\times\alpha$. as it avoids
multiplications completely, and it is even more efficient than the binary
counterpart as ternary weights {-1, 0, +1} can be considered as setting some
values in the binary weights {-1, +1} to 0s. 0s can be considered as network
pruning [8] that makes networks sparse and thus reduces the computational
cost. In light of this, in this paper, we attempt to train DNNs with ternary
weights {-1, 0 +1} that are able to reduce the computational complexity
substantially, speed up the inference significantly, and lead to about
16$\times$ or 32$\times$ memory requirement reduction compared with the float
(32-bit) or double (64-bit) precision counterparts.

Table 1: Comparison between WDR and the existing regularizers Methods | Controlling sparsity | Free from gradient estimators
---|---|---
The regularizer in [9] | ✗ | ✗
The regularizers in [10] | ✗ | ✗
SCA | ✓ | ✓

The challenge for training ternary or binary weight networks is that the
gradients with respect to the discrete weights do no exist. Substantial
efforts have been made on addressing this issue [11, 12, 13, 14]. These
approaches train discrete weight networks by using stochastic weights or using
the straight-through estimator to estimate the gradients with respect to
discrete weights. However, we notice that most existing efforts on training
ternary weight networks focus on training ternary weights with {-$\alpha$, 0,
+$\alpha$} [15, 12, 16] or {-$\alpha$, 0, +$\beta$} [17, 13], which are unable
to avoid multiplications completely. Most of these approaches still have a
large performance gap to full-precision counterparts, while some approaches
[18, 13] are able to obtain promising results, but they need a complex
training process, which causes the ternary weight networks not widely used.
Moreover, none of these approaches is able to control the sparsity of the
ternary weights, which cannot fully use the advantage of ternary weights. It
is thus necessary and appealing for the development of an accurate, sparsity-
controlling and easy-to-implement approach for converting full-precision
networks to the ternary versions with weights {-1, 0, +1}.

In the paper, we propose a simple yet effective approach (i.e., SCA) to
training ternary weight {-1, 0, +1} networks and it also has the ability to
control the sparsity of the weights. SCA is developed base on the fact that
the ternary solutions to a DNN are also included in the full-precision weight
space. SCA attempts to search for a ternary solution in the full-precision
space. However, it is difficult to find an appropriate ternary solution in
such a large weight space. To address this issue, at training time, SCA limits
the full-precision weight to be in the range between -1 and +1 by
parameterizing them with $\tanh(\Theta)$ where $\Theta$ are the parameters. In
this way, the weight space is largely reduced. By taking advantage of the
properties of the $\tanh(\Theta)$ function, we design a novel weight
discretization regularization (WDR) to force weight ternarization and control
the sparsity. To the end, SCA simply trains a ternary weight network by
optimizing a full-precision network loss plus a WDR term. At test time, the
ternary weights are obtained by simply rounding $\tanh(\Theta)$ to the nearest
integers. Despite its simplicity, it is able to outperform the state-of-the-
art approaches and perform on a par with full-precision counterparts.

SCA is a regularizer-based approach. In the existing literature, three well-
known regularizers [9, 10] have been proposed for training binary weight
networks and they can be easily extended to training ternary weight networks.
We summarize the differences between the regularizer (i.e., WDR) in SCA and
the existing regularizers in Table 1. First, SCA is able to control the
sparsity of the ternary weights while the existing regularizers cannot.
Second, SCA is able to use the real gradients to train ternary weight networks
while the existing regularizers rely on the straight-through gradient
estimator. We provide the technique differences among these regularizers in
more detail in Section 3.6.

The main contributions of our work can be summarized as follows:

  * 1.

We have proposed a simple yet effective approach (i.e., SCA) to training
ternary weight networks by simply minimizing a task loss plus a regularizer.
The simple nature of SCA makes it easy-to-implement and have a high code-
reuse-rate when converting a full-precision network to the ternary version,
which is of much practical values, considering the fact that the state-of-the-
art models in various domains are still built on full-precision weight DNNs.

  * 2.

We have proposed a novel regularizer WDR for training ternary weights. The
shape controller $\alpha$ in WDR enables SCA to control the sparsity of the
trained ternary weights. We theoretically and empirically show that the
sparsity of the ternary weights is positively related to $\alpha$.

  * 3.

The existing literature has only studied how to control the sparsity of full-
precision weights in a DNN. To the best of our knowledge, this is first work
to explore controlling the sparsity of ternary weights in a DNN.

  * 4.

Extensive experiments on several benchmark datasets demonstrate that SCA
outperforms the state-of-the-art approaches and matches the performances of
full-precision counterparts.

##  2 Related Work

Our work is related to the literature on model compression including network
quantization and network pruning. Thus, we first present an overview of the
existing techniques on training DNNs with discrete weights and then review the
existing literature on network pruning.

###  2.1 Network Quantization

Network quantization has drawn numerous research attentions due to its
potential in various applications and low memory and computational power
requirements. The goal is to train DNNs with low-bit weights or activations.
These approaches train discrete neural networks by approximating full-
precision weights or activations in each layer with scaling factors and
discrete values [19, 15, 20, 21, 22, 17, 23, 16, 24, 11, 25, 26], using
stochastic weights [27, 18, 28, 29, 30], using a gradient estimator [31, 32,
33], using the straight-through estimator [34, 35], or using reinforcement
learning [36]. Among these approaches, our work is most related to these
approaches to training binary or ternary weight neural networks.

####  2.1.1 Approaches to Training Binary Neural Networks

Many approaches have been proposed to training binary neural networks. Soudry
et al. [27] propose to train binary neural networks through the variational
Bayesian method that infers networks with binary weights and neurons.
BinaryConnect [34] uses sign function to binarize the weights during the
forward and backward propagation while using full-precision weights in the
parameter update stage. Binarized Neural Networks (BNNs) [35] and XNOR-net
[19] make some extensions to this method by binarizing both weights and
activations. BNNs [35] utilizes the sign function and additional scaling
factors to binarize the real-value weights and the pre-activations at training
time. The straight-through estimator is used to back-propagate through the
binarization operation. XNOR-net takes BNNs one step further by approximating
the real-valued tensor and activation tensor by a binary filter and a scaling
factor. ABC-nets [22] make an extension by using the linear combination of
multiple binary tensors to approximate full-precision weights. Moreover, to
alleviate information loss, multiple binary activations are also employed.
McDonnell [23] follows BinaryConnect [34] and BNNs [35], and applies a layer-
dependent scaling to the sign of the weights. Tang et al. [9] explore how the
learning rate, the scale factor, and the regularizer influence the
performances of BNNs. Shayer et al. [18] propose LR-Net to train binary weight
neural networks by using the central limit theorem and the local
reparameterization trick. They update the weight distribution at training
time. Binary weights are sampled from the learned distribution at test time.
PBNet [37] extends the idea of LR-Net and uses a probabilistic method for
training neural networks with both binary weights and binary activations. Leng
et al. [12] propose to train low-bit neural networks by decoupling the
continuous parameters from the discrete constraints and casting the original
problem into several subproblems. CI-BCNN [36] trains binary neural networks
by using the channel graph structure and channel-wise interactions. LQ-Nets
[11] jointly train the network parameters and its associated quantizers for
DNNs. Bi-real nets [38] use the identity shortcut to connect the real
activations to activations of the consecutive block, thus improving
representational capability of binary neural networks. ELQ [16] trains
discrete weight DNNs through explicitly regularizing the weight approximation
error and the loss perturbation. Zhuang et al. [24] propose to divide the
network into groups of which each can be reconstructed by using a set of
binary branches. Meng et al. [28] propose to use the Bayesian rule to train
binary weights for DNNs. These approaches either have a large performance gap
to the full-precision networks or need a complex training process.

####  2.1.2 Approaches to Training Ternary Neural Networks

Substantial research efforts have been made on training neural networks with
ternary weights or activations. Alemdar et al. [20] use a two-stage teacher-
student approach for training neural networks with ternary activations. First,
they train the teacher network with stochastically firing ternary neurons and
then let the student network learn how to imitate the teacher’s behavior using
a layer-wise greedy algorithm. Mellempudi et al. [21] propose to take
advantage of a fine-grained quantization technique that involves multiple
scaling factors to obtain a ternary neural network. TTQ [17] uses two full-
precision scaling coefficients for each layer and quantizes the weights to
three real values. Li et al. [15] develop an approach to training ternary
weight networks by minimizing the Euclidean distance between the full-
precision weights and the ternary weights along with a non-negative scaling
factor. LR-Net [18] attempts to train a stochastically ternary network by
leveraging the local reparametrization trick and sampling ternary weights at
the test time. We notice that almost all the existing efforts on training
ternary weight networks focus on training ternary weights with {$-\alpha$, 0,
$\alpha$} [15, 12, 21, 16] or {$-\alpha$, 0, $\beta$} [17, 13] that are unable
to avoid multiplications completely and thus are less efficient than {-1, 0,
+1} in SCA. More importantly, none of these approaches is able to control the
sparsity of the ternary weights, which cannot fully use the advantage of
ternary weights.

SCA trains ternary weight networks by using parameterization and a novel
regularizer. Thus, it is also related to regularizer based approaches for
training discrete weight networks. Tang et al. [9] and BNN+ [10] propose three
regularizers for training binary neural networks, which can be easily extended
to train ternary weight networks. However, these regularizers cannot control
the sparsity of the ternary weights and highly rely on the straight-through
estimator to approximate the gradients with respect to the discrete weights.
SCA overcomes all these shortcomings.

###  2.2 Network Pruning

On the other hand, our work is also related to the studies on network pruning,
including but not limited to [39, 40, 41, 42, 43, 44, 45, 46]. Network pruning
aims to compress a DNN into a sparse version by making the full-precision
weights sparser. Luo et al. [43] propose to use the statistics of next layer
to prune weights. However, weight pruning approaches require substantial
iterations to converge and the pruning threshold needs to be set manually.
Low-rank decomposition has also been introduced to network compression. These
approaches [47, 48, 49] use matrix decomposition technique to decompose the
weight in DNNs. The limitation is that these approaches increase the number of
the layers in a DNN and thus are easy to result in gradient-vanishing issues
during the training process.

SCA is different from all these existing studies in that SCA attempts to
control the sparsity of ternary weights rather than full-precision weights,
which is a more challenging problem. To our best knowledge, SCA is first work
to control the sparsity of ternary weight networks.

##  3 Framework

In this section, we introduce SCA for training DNNs with ternary weights whose
values are constrained to {-1, 0, +1}. SCA is developed based on the fact that
the ternary solution to a DNN is also included in the full-precision weight
space. To illustrate the connection between ternary weights and full-precision
weights, we first review the basic process for training full-precision weight
networks. Then we make further derivations of the basic process to show how to
train ternary weight networks.

###  3.1 Deep Neural Networks with Full-Precision Weights

In this part, we review how to train a DNN with full-precision weights. Given
the training data $(X,Y)$ where $X$ are the inputs and $Y$ are the targets,
the output of a neural network $f$ with weights $W$ is $f(X,W)$. The loss can
be written as:

$L_{c}(W)=\mathcal{L}(f(X,W),Y)$ (1)

where $\mathcal{L}(.)$ is any loss function, such as mean square error or
cross entropy, and $W=[w_{1},w_{2},...,w_{n}]$ with $w_{i}$ representing the
weights in the $i$th layer.

With the loss function, we can use backpropagation to compute the gradients
with respect to the weights $W$ and utilize gradient descent to minimize the
loss function. This can be done easily for continuous weight networks while it
is difficult for discrete weight networks due to no gradients. Fortunately,
the discrete solutions to a DNN are also included in the full-precision weight
space. We show below that with modifications to the basic training process
above we can also train DNNs with ternary weights.

###  3.2 Limiting the Weight Value Range with Parameterization

Figure 1: Graph of
$\tanh(\theta)=\frac{\exp\left(\theta\right)-exp\left(-\theta\right)}{exp\left(\theta\right)+exp\left(-\theta\right)}$

DNNs with ternary weights are dramatically efficient at the test time as it
needs less memory and no multiplication operation. The challenge for training
ternary weights is that the gradients with respect to the discrete weights are
0 or do not exist. The existing studies utilize the straight-through estimator
to solve this problem. In this paper, we propose a novel method to use the
real gradient instead of the estimator to do backpropagation. As the ternary
solution to a DNN is also in the full-precision weight space, we are inspired
to maintain continuous value weights during training, enabling us to do normal
backpropagation. However, the full-precision weight space (usually 16 bits or
32 bits) is too large to find an appropriate ternary solution. To alleviate
this issue, we propose to compress the continuous weight space. The possible
values for ternary weights are -1, 0, and +1 that are between -1 and +1. The
range of the function $\tanh(.)$ is between -1 and +1 as shown in Figure 1, so
that we use $\tanh(\Theta)$ to parameterize the continuous weights $W$:

$W=\tanh(\Theta)$ (2)

where $\Theta=[\theta_{1},\theta_{2},...,\theta_{n}]$ are the parameters for
computing $W$ and $\theta_{i}$ is used to compute $w_{i}$.

Then we replace $W$ in loss (1) with $\tanh(\Theta)$, obtaining the following
loss term:

$L_{c}(\tanh(\Theta))=\mathcal{L}(f(X,\tanh(\Theta)),Y)$ (3)

where $\tanh(\Theta)$ can be seen as the instantiations of $W$. The advantage
of doing this is that the weight space is significantly reduced and the
backprobagation can still be applied to computing the gradients with respect
to $\Theta$, enabling us to search for a ternary solution in a much smaller,
continue weight space. Note that although $\tanh(\Theta)$ function has been
widely used in deep learning, how to use it to control the sparsity of ternary
weights has never been studied. We show this below.

###  3.3 Weight Discretization Regularization and Objective Function

Although the weight space is extremely decreased by using parameterization
$\tanh(\Theta)$, it is still difficult to find the discrete solution in this
space. If we directly solve for the minimization problem with (3) as the
objective, we are able to obtain a solution in the continuous space (-1, 1)
that may still be far from the discrete solution. To address this issue, we
develop a novel weight discretization regularizer (WDR) to force the weights
to be ternary and to control the sparsity of the ternary weights by taking
advantage of the property of tanh() function:

$\displaystyle
R(\tanh(\Theta))=\sum_{i=1}^{n}\sum_{j=1}^{|\theta_{i}|}\left[\left(\alpha-\tanh^{2}\left(\theta_{ij}\right)\right)\tanh^{2}\left(\theta_{ij}\right)\right]$
(4)

where $|\theta_{i}|$ denotes the number of the elements in $\theta_{i}$;
$\theta_{ij}$ represents the $j$th element in $\theta_{i}$; and $\alpha$ is
the shape controller of function $R()$. Note that one may have a
misunderstanding at first glance that WDR enforces the weights to converge at
{$-\sqrt{\alpha}$, $+\sqrt{\alpha}$, 0}. We will show below (in Section 3.5)
that this is not true and how WDR leads to the expected ternary weights {-1,
+1, 0}. More essentially, shape controller $\alpha$ is also the sparsity
controller of the trained ternary weights (the sparsity here is measured by
the percentage of 0s in the ternary weights). We will theoretically (in
Section 3.5) and empirically (in Section 4.5) show that the sparsity of the
ternary weights is positively related to $\alpha$.

We combine the loss term (3) and WDR (4) to obtain the final objective
function of SCA:

$J=L_{c}(\tanh(\Theta))+\lambda R(\tanh(\Theta))$ (5)

where coefficient $\lambda$ is a hyperparameter to balance the contributions
between the loss term $L_{c}(\tanh(\Theta))$ and the regularizer
$R(\tanh(\Theta))$.

At the training time, we solve for the minimization problem with objective (5)
using a gradient descent based optimizer as the gradients of $J$ with respect
to $\Theta$ exist. It is observed that SCA only needs two simple operations to
convert a full-precision network to the ternary version, i.e., replacing
weights $W$ with $\tanh(\Theta)$ and adding the WDR term $R(\tanh(\Theta))$.
Thus, it has an extremely high code reuse rate for converting full-precision
networks to the ternary counterparts. To further illustrate this point, we
compare the pseudocode for training full-precision weight networks with that
of SCA. As shown in Figure 2, their differences lie in line 3 and line 10. In
line 3, SCA simply modifies the conv function by adding $\tanh()$ to
parametrize the weights. In line 10, SCA simply adds a regularizer to force
the weights to be ternary. SCA only changes the conv function and then adds a
regularizer to the loss. Therefore, it has a very high code reuse rate when
converting a full-precision network to the ternary counterpart.

Figure 2: Pseudocode compariosn. (a): Pseudocode for training full-precision
networks based on tensorflow. (b): Pseudocode for training ternary weight
networks via SCA based on tensorflow where $b$ in line 10 is the balancing
weight and $R()$ is the proposed regularizer WDR that can be simply
implemented as a function.

###  3.4 Test-Time Inference

After (5) converges, we take the learned $\Theta$ to compute the ternary
weights for inference. Specifically, at the test time, we obtain the ternary
weights $W_{ter}$ by rounding $\tanh(\Theta)$ to the nearest integer:

$W_{ter}=round(\tanh(\Theta))$ (6)

Since the range of $\tanh(\theta_{ij})$ is between -1 and +1, the obtained
nearest integer of $\tanh(\theta_{ij})$ can be -1, 0, or +1. For given test
data $x_{t}$, the prediction is obtained by using $W_{ter}$:

$Pred=f(W_{ter},x_{t})$ (7)

Thanks to the regularizer WDR, simply rounding $\tanh(\Theta)$ to the nearest
integer almost does not lead to any performance dropping. We summarize SCA in
Algorithm 1.

Algorithm 1 SCA

0: Training data (X, Y), a neural network $f$ with initial parameters
$\Theta_{0}$

0: Ternary weights $W_{ter}$

1: Construct objective function (5) and minimize it with gradient descent to
obtain $\Theta_{opt}$

2: Obtain ternary weights $W_{ter}$ by simply rounding $\tanh(\Theta_{opt})$
to nearest integers

Figure 3: Function graph of $R(\tanh(\theta))$
with $\alpha=0$

Figure 4: Function graph of $R(\tanh(\theta))$
with $\alpha\geq 2$

Figure 5: Function graph of $R(\tanh(\theta))$
with $\alpha$ in (0, 2)

###  3.5 Theoretical Analysis

In this part, we further examine the proposed regularizer WDR and
theoretically show why $\alpha$ is able to control the sparsity of the trained
ternary weights. For clarity, we omit the subscripts $i$, $j$ of $\theta_{ij}$
and observe WDR w.r.t. each $\theta_{ij}$:

$R(\tanh(\theta))=\left(\alpha-\tanh^{2}\left(\theta\right)\right)\tanh^{2}\left(\theta\right)$
(8)

The derivative function of (8) is expressed as:

$\displaystyle\frac{\partial
R(\tanh(\theta))}{\partial\theta}=2\tanh\left(\theta\right)\left(1-\tanh^{2}\left(\theta\right)\right)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$
(9)

For training ternary weights {-1, 0, +1}, it is desired that
$\tanh(\theta)=0$, $\tanh(\theta)=-1$ 222Note that $\tanh(\theta)$ cannot
equal to -1 or 1, but can be infinitely close to -1 or 1, and
$\tanh(\theta)=+1$ are three minimum points of the WDR term
$R(\tanh(\theta))$. We provide below the minimum and maximum points of
$R(\tanh(\theta))$ in different cases, i.e., $\alpha=0$, $0<\alpha<2$, and
$\alpha\geq 2$

###### Proposition 1

When $\sqrt{\frac{\alpha}{2}}=0$ (i.e., $\alpha=0$), $R(\tanh(\theta))$ has
three zero-gradient points with $\tanh(\theta)=0$ as the maximum point, and
$\tanh(\theta)=-1$ and $\tanh(\theta)+1$ as the two minimum points.

Proof of of proposition 1: When $\alpha=0$, the derivative function of
$R(\tanh(\theta))$ is written as:

$\displaystyle\frac{\partial
R(\tanh(\theta))}{\partial\theta}=-4\tanh^{3}\left(\theta\right)\left(1-\tanh^{2}\left(\theta\right)\right)$
(10)

By solving $\frac{\partial R(\tanh(\theta))}{\partial\theta}=0$, we obtain
three zero-gradient points, i.e., $\tanh(\theta)$ = 0, $\tanh(\theta)$ = -1,
and $\tanh(\theta)$ = +1. In addition,
$\left(1-\tanh^{2}\left(\theta\right)\right)$ in right hand side of (10) is
always positive as the range of $\tanh(\theta)$ is (-1, 1) as shown in Figure
1. Thus, the sign of (10) is determined by $-4\tanh^{3}\left(\theta\right)$.
When $\tanh(\theta)\in(-1,0)$, $-4\tanh^{3}\left(\theta\right)$ is positive so
that (10) is positive. It means that $R(\tanh(\theta))$ is increasing when
$\tanh(\theta)\in(-1,0)$. Similarly, When $\tanh(\theta)\in(0,1)$,
$-4\tanh^{3}\left(\theta\right)$ is negative so that (10) is negative. It
means that $R(\tanh(\theta))$ is decreasing when $\tanh(\theta)\in(0,1)$. We
provide the function graph of $R(\tanh(\theta))$ with $\alpha=0$ in Figure 5.
Therefore, $\tanh(\theta)=0$ is the maximum point and $\tanh(\theta)=-1$ and
$\tanh(\theta)=+1$ are two minimum points. The above proves Proposition 1.

However, the case in Proposition 1 is not desired for training ternary weights
as the ideal case for training ternary weights is that $\tanh(\theta)=0$,
$\tanh(\theta)=-1$, and $\tanh(\theta)=+1$ are three minimum points.

###### Proposition 2

When $\sqrt{\frac{\alpha}{2}}\geq 1$ (i.e., $\alpha\geq 2$),
$R(\tanh(\theta))$ has three zero-gradient points with $\tanh(\theta)$=0 as
the minimum point, and $\tanh(\theta)$=-1 and $\tanh(\theta)$=+1 as the two
maximum points

Proof of Proposition 2: When $\sqrt{\frac{\alpha}{2}}\geq 1$ (i.e.,
$\alpha\geq 2$), by directly solving (9) = 0, we obtain five zero-gradient
points of $R(\tanh(\theta))$, i.e., $\tanh(\theta)$ = 0, $\tanh(\theta)$ = -1,
$\tanh(\theta)$ = +1, $\tanh(\theta)$ = $-\sqrt{\frac{\alpha}{2}}$, and
$\tanh(\theta)$ = $\sqrt{\frac{\alpha}{2}}$. However, as
$\sqrt{\frac{\alpha}{2}}\geq 1$ but the range of $\tanh(\theta)$ is (-1, 1),
zero-gradient points $\tanh(\theta)$ = $-\sqrt{\frac{\alpha}{2}}$ and
$\tanh(\theta)$ = $\sqrt{\frac{\alpha}{2}}$ do not exist. Thus, in this case,
$\tanh(\theta)$ = 0, $\tanh(\theta)$ = -1, and $\tanh(\theta)$ = +1 are three
zero-gradient points of $R(\tanh(\theta))$. In addition,
$\left(1-\tanh^{2}\left(\theta\right)\right)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$
in (9) is always positive when $\sqrt{\frac{\alpha}{2}}\geq 1$ (i.e.,
$\alpha\geq 2$). Thus, the sign of (9) is determined by $2\tanh(\theta)$. When
$\tanh(\theta)\in(-1,0)$, $2\tanh(\theta)$ is negative so that (9) is
negative. It means that $R(\tanh(\theta))$ is decreasing when
$\tanh(\theta)\in(-1,0)$. Similarly, when $\tanh(\theta)\in(0,1)$,
$2\tanh(\theta)$ is positive so that (9) is positive. It means that
$R(\tanh(\theta))$ is increasing when $\tanh(\theta)\in(0,1)$. We provide the
function graph of $R(\tanh(\theta))$ with $\alpha\geq 2$ in Figure 5.
Therefore, $\tanh(\theta)=0$ is the minimum point and $\tanh(\theta)=-1$ and
$\tanh(\theta)=+1$ are two maximum points. The above proves Proposition 2.

Obviously the case in Proposition 2 is not desired for training ternary
weights, either. Let us pay more attention to the last case.

###### Proposition 3

When $0<\sqrt{\frac{\alpha}{2}}<1$ (i.e, $0<\alpha<2$), $R(\tanh(\theta))$ has
five zero-gradient points. $\tanh(\theta)=0$, $\tanh(\theta)=-1$, and
$\tanh(\theta)=+1$ are three minimum points, and
$\tanh(\theta)=-\sqrt{\frac{\alpha}{2}}$ and
$\tanh(\theta)=\sqrt{\frac{\alpha}{2}}$ are two maximum points. Moreover, the
percentage of 0s in the resulting ternary weights by minimizing
$R(\tanh(\theta))$ is positively related to $\alpha$.

Proof of proposition 3: When $0<\sqrt{\frac{\alpha}{2}}<1$, by directly
solving (9) = 0, we obtain five zero-gradient points for $R(\tanh(\theta))$,
i.e., $\tanh(\theta)$ = 0, $\tanh(\theta)$ = -1, $\tanh(\theta)$ = +1,
$\tanh(\theta)$ = $-\sqrt{\frac{\alpha}{2}}$, and $\tanh(\theta)$ =
$\sqrt{\frac{\alpha}{2}}$, and all the five zero-gradient points are
meaningful. In addition, as $\left(1-\tanh^{2}\left(\theta\right)\right)$ in
(9) is always positive, the sign of (9) is determined by
$2\tanh(\theta)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$. When
$\tanh(\theta)\in(-1,-\sqrt{\frac{\alpha}{2}})$,
$2\tanh(\theta)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$ is positive
so that (9) is positive. It means that $R(\tanh(\theta))$ is increasing when
$\tanh(\theta)\in(-1,-\sqrt{\frac{\alpha}{2}})$. When
$\tanh(\theta)\in(-\sqrt{\frac{\alpha}{2}},0)$,
$2\tanh(\theta)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$ is negative
so that (9) is negative. It means that $R(\tanh(\theta))$ is decreasing when
$\tanh(\theta)\in(-\sqrt{\frac{\alpha}{2}},0)$. When
$\tanh(\theta)\in(0,\sqrt{\frac{\alpha}{2}})$,
$2\tanh(\theta)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$ is positive
so that (9) is positive. It means that $R(\tanh(\theta))$ is increasing when
$\tanh(\theta)\in\left(0,\sqrt{\frac{\alpha}{2}}\right)$. When
$\theta\in(\sqrt{\frac{\alpha}{2}},1)$,
$2\tanh(\theta)\left(\alpha-2\tanh^{2}\left(\theta\right)\right)$ is negative
so that (9) is negative. It means that $R(\tanh(\theta))$ is decreasing when
$\tanh(\theta)\in(\sqrt{\frac{\alpha}{2}},1)$. We provide the function graph
of $R(\tanh(\theta))$ with $0<\sqrt{\frac{\alpha}{2}}<1$ (i.e, $0<\alpha<2$)
in Figure 5. Therefore, $\tanh(\theta)=0$, $\tanh(\theta)=-1$, and
$\tanh(\theta)=+1$ are three minimum points, and
$\tanh(\theta)=-\sqrt{\frac{\alpha}{2}}$ and
$\tanh(\theta)=\sqrt{\frac{\alpha}{2}}$ are two maximum points. Moreover, as
shown in Figure 5, the points with $\tanh(\theta)$ in
$\left(-\sqrt{\frac{\alpha}{2}},\sqrt{\frac{\alpha}{2}}\right)$ are more
inclined to moving toward the minimum point with $\tanh(\theta)=0$ due to
gradient $g_{0}$. The points with $\tanh(\theta)$ in
$\left(-1,-\sqrt{\frac{\alpha}{2}}\right)$ are more inclined to moving toward
the minimum point with $\tanh(\theta)=-1$ due to gradient $g_{1}$, and the
points with $\tanh(\theta)$ in $\left(\sqrt{\frac{\alpha}{2}},1\right)$ are
more inclined to moving toward the minimum point with $\tanh(\theta)=1$ due to
gradient $g_{2}$. Consequently, the percentage of 0s in the resulting ternary
weights is positively related to the length of the range
$\left(-\sqrt{\frac{\alpha}{2}},\sqrt{\frac{\alpha}{2}}\right)$, i.e,
$\sqrt{2\alpha}$, and it is thus positively related to $\alpha$. The above
proves Proposition 3.

It is obvious that the case in Proposition 3 is desired for training ternary
weight networks as $\tanh(\theta)=-1$, $\tanh(\theta)=0$, and
$\tanh(\theta)=+1$ are three minimum points. We also empirically verify the
positive correlation between the sparsity of the ternary weights and the
values of $\alpha$ in the experiment section.

###  3.6 Differences between WDR in SCA and the Existing Regularizers

In this part, we provide the differences between regularizer WDR in SCA with
the regularizers in the existing literature.

Tang et al. [9] proposed a regularizer $1-W^{2}$ for training binary weight
networks, which can be easily extended to $(1-W^{2})W^{2}$ for training
ternary weight networks. However, this simple regularizer cannot control the
sparsity of the ternary weights and it relies on the straight-through
estimator to estimate the gradients with respect to discrete weights $W$. In
contrast, WDR addresses these two limitations by introducing the combination
of a controller $\alpha$ and parameterization $\tanh(\Theta)$ to control the
sparsity of the ternary weights and allow for using the real gradients.

BNN+ [10] introduced two regularizers for training binary neural networks,
i.e., $|\alpha-|W||$ and $(\alpha-|W|)^{2}$, which can be easily extended to
$|\alpha-|W||*|W|$ and $(\alpha-|W|)^{2}*|W|^{2}$, respectively, for training
ternary weight networks. First, it is obvious that the straight-through
estimator is still required to estimate the gradients with respect to discrete
weights $W$. The proposed WDR uses the real gradients by introducing
$\tanh(\Theta)$. Second, besides $|W|=0$, the minimum point for these two
regularizers in BNN+ is $|W|=\alpha$, which leads to ternary weights
{$-\alpha,0,+\alpha$}. Consequently, BNN+ introduces $\alpha$ to control the
scaling of $W$. In contrast, $\alpha$ in our WDR is used to control the
sparsity of the ternary weights, which results in more efficient ternary
weight networks. Moreover, {$-\alpha,0,+\alpha$} in BNN+ is less efficient
than {-1, 0, +1} in WDR.

##  4 Experiments

In this section, we report extensive experiments for evaluating SCA on several
benchmark datasets. Through these experiments, we aim to address the following
research questions:

  * 1.

How much is the performance gap between a ternary weight network trained by
SCA and the full-precision weight counterpart?

  * 2.

Does our approach SCA yield better performances than those of the existing
approaches?

  * 3.

Is the sparsity of the trained ternary weights positively related to $\alpha$
on real datasets?

  * 4.

What is the difference between the features learned by SCA and the features of
full-precision networks?

###  4.1 Datasets

We adopt five benchmark datasets: MNIST [50], CIFAR-10 [51], CIFAR-100 [51],
Tiny ImageNet 333http://tiny-imagenet.herokuapp.com/, and ImageNet
(ILSVRC2012) [52].

MNIST is a handwritten digit image classification dataset with 10 classes,
containing 60,000 training images and 10,000 test images. We do not use any
data augmentation on MNIST.

CIFAR-10 is an image classification dataset with 10 classes, containing 50,000
training images and 10,000 test images with image size 32 $\times$ 32 in the
RGB space. We follow the standard data augmentation on CIFAR-10. During
training time, we pad 4 pixels on each side of an image and randomly flip it
horizontally. Then the image is randomly cropped to 32 $\times$ 32 size.
During test time, we only evaluate the single view of an original 32 $\times$
32 image without padding or cropping.

CIFAR-100 comprises similar images to those in CIFAR-10, but has 100 classes.
We adopt the same data augmentation strategy as that in CIFAR-10.

Tiny ImageNet, i.e., a subset of ImageNet, is an image classification dataset
with 200 classes, containing 100,000 training images and 10,000 test images
with size 64 $\times$ 64 in the RGB space. We adopt the standard data
augmentation strategy on Tiny ImageNet, i.e., randomly padding, flipping, and
cropping. At test time, we only evaluate the original image.

ImageNet is a large-scale image classification dataset with 1000 classes,
containing 1.28 million training images and 50,000 validation images with
different sizes in the RGB space. On ImageNet, we use the standard scale and
aspect ratio augmentation strategy from [53]. Test images are resized so that
the shorter side is set to 256, and then are cropped to size 224 $\times$ 224.

###  4.2 Competitors

For fair comparison, we only compare SCA with the existing approaches444We
only compare SCA with the competitors in term of test accuracy because almost
all the competitors except TTQ [17] only report the accuracy without sparsity.
for training DNNs with binary or ternary weights without discrete activations.
The competitors for training DNNs with binary weights include BinaryConnect
[34], BWN [19], DoReFa [54], and BayesBiNN [28]. The competitors for training
DNNs with ternary weights include TWN [15], TTQ [17], INQ [13], LR-Net [18],
ELQ [16], and ELB [12]. As SCA is a regularizer-based approache, we also
compare SCA with the regularizers in [9, 10] which are extended to training
ternary weight networks. The performances of all the competitors are all taken
from their original papers or obtained from the author-released codes unless
otherwise specified. We follow the same convention as that used in the
existing work to keep the first and last layers in full precision. All the
results below are reported based on 3 runs.

###  4.3 Comparison with State-of-the-art Approaches

We report the performances of different approaches as well as the full-
precision networks on different datasets.

Table 2: Test accuracies (%) on MNIST Methods | Weight Types | Accuracy (%)
---|---|---
BinaryConnect | Binary: {-1 ,+1} | 98.71
BayesBiNN | 98.86
BWN | Ternary: {-$\alpha$, +$\alpha$} | 99.05
Extended [9] | Ternary: {-1, 0, +1} | 99.11
LR-Net | 99.50
SCA (Ours) | 99.56$\pm$0.02
TWN | Ternary: {-$\alpha$, 0, +$\alpha$} | 99.35
Extended [10] | 99.15
Full-precision network | Full-Precision | 99.56

####  4.3.1 Performances on MNIST

On MNIST, we use the same architecture as that in the competitors [15, 18],
i.e., $(32-C5)+MP2+(64-C5)+MP2+512FC+Softmax$, where ($32-C5$) is the
convolutional layer containing 32 filters of size $5\times 5$; $MP2$ is the
max-pooling layer with stride 2; and $512FC$ denotes the fully connected layer
with 512 nodes. We adopt dropout [55] before the last layer with drop rate of
0.5. $\lambda$ and $\alpha$ are set to 1e-7 and 1e-4, respectively. The
weights are initialized with Xavier initializer [56]. The objective function
is minimized with optimizer Adam [57] and mini-batch size 128. The initial
learning rate is 0.01 and is divided by 10 at the 100th epoch and the 160th
epoch. We have trained the network for 200 epochs.

The comparison results on MNIST are reported in Table 2. It is observed that
SCA achieves the best performance among the approaches with ternary weights
{-1, 0 ,+1}, also beats the other state-of-the-art approaches significantly,
and even matches the performance of the full-precision weight counterpart.
This demonstrates that SCA is able to compress a full-precision weight network
to the ternary counterpart without accuracy dropping. We also notice that the
performance gap between SCA (99.56%) and the best competitor (99.50%) is not
large. The reason is that the performances of SCA and the best competitor are
both very close to that of the full-precision network (99.56%). The
performance of the full-precision network can be considered as the upper bound
of the ternary weight network performances. Both SCA and the best competitor
can match this upper bound so that their performance gap is not large.
However, SCA still shows its superiority over the existing approaches as it
achieves the same accuracy as that of the full-precision network.

####  4.3.2 Performances on CIFAR-10

On CIFAR-10, we adopt the same architectures as those in the competitors [15,
18, 17]: VGG-S [15, 18], ResNet-20 [17, 58], and VGG-Variate [59]. For VGG-S,
$\lambda$ and $\alpha$ are 5e-8 and 0.1, respectively; dropout with drop rate
of 0.5 is adopted; weight decay is used in the last layer with parameter 1e-5;
the objective function is minimized with Adam with mini-batch size 128; the
initial learning rate is 0.01 and is divided by 10 at the 200th epoch and the
370th epoch; we have trained the network for 450 epochs. For ResNet-20,
$\lambda$ and $\alpha$ are set to 5e-6 and 0.1, respectively; weight decay is
used in the last layer with parameter 1e-5; the initial learning rate is 0.005
and is divided by 5 at the 150th epoch, then divided by 2 at the 450th epoch;
we have trained the network for 700 epochs with mini-batch size 128. For VGG-
Variate, $\alpha$ is set to 0.1; dropout [55] with drop rate of 0.5 is adopted
before the last layer; the initial $\lambda$ is set to 5e-8 and is multiplied
by 50 after 270 epochs of training; weight decay is used in the last layer
with parameter 1e-5; the objective function is minimized with Adam with mini-
batch size 128; the initial learning rate is 0.005 and is divided by 10 at the
120th epoch and the 270th epoch; we have trained the network for 370 epochs.
We adopt the same initialization strategy as that in LR-Net [18] and TTQ [17]
by using the pretrained full-precision weights as the initialization for VGG-
Variate and all ResNets while VGG-S is initialized with Xavier initializer
[56].

Table 3: Test accuracies (%) on CIFAR-10 Methods | Weight Types | VGG-S | ResNet-20 | VGG-Variate
---|---|---|---|---
BinaryConnect | Binary: {-1 ,+1} | 91.10 | - | -
BWN | Binary: {$\alpha$, +$\alpha$} | 90.18 | - | -
DoReFa | - | 90.00 | -
Extended [9] | Ternary: {-1, 0, +1} | 90.17 | 89.97 | 91.06
LR-Net | 93.26 | 90.08 | 91.47
SCA (Ours) | 93.41$\pm$0.10 | 91.28 $\pm$ 0.15 | 92.75 $\pm$ 0.17
TWN | Ternary: {-$\alpha$, 0, +$\alpha$} | 92.56 | - | -
Extended [10] | 90.32 | 90.10 | 91.13
TTQ | Ternary: {-$\alpha$, 0, +$\beta$} | - | 91.13 | -
Full-precision network | full-precision | 93.42 | 91.76 | 92.75

Table 3 summarizes the comparison results on CIFAR-10. We observe that SCA
compresses the full-precision networks to the ternary versions almost without
accuracy dropping for VGG-S and VGG-Variate but with a little accuracy
dropping for ResNet-20. The reason is that VGG-S and VGG-Variate ($\geq$ 0.84M
parameters) have much more parameters than those in ResNet-20 (0.27M
parameters) and compressing over-parameterized networks may not hurt their
performances. On the other other hand, despite its simplicity, SCA also
significantly outperforms all the binary or ternary competitors on all the
three networks, which demonstrates the effectiveness and superiority of SCA.

Table 4: Performances on ImageNet |  | ResNet-18 | AlexNet
---|---|---|---
Methods | Weight Types | TOP-1 (%) | TOP-5 (%) | TOP-1 (%) | TOP-5 (%)
BWN | Binary: {-$\alpha$, +$\alpha$} | 60.8 | 83.0 | 56.8 | 79.4
Extended [9] |  Ternary: {-1, 0, +1} | 65.8 | 86.7 | 54.8 | 76.9
LR-Net | 63.5 | 84.8 | 55.9 | 76.3
SCA (Ours) | 67.9$\pm$0.11 | 88.0$\pm$0.21 | 59.3$\pm$0.16 | 80.8$\pm$0.19
TWN |  Ternary: {-$\alpha$, 0, +$\alpha$} | 65.3 | 86.2 | 54.5 | 76.8
ELB | 67.0 | 87.5 | 58.2 | 80.6
ELQ | 67.3 | 88.0 | 57.9 | 80.2
Extended [10] | 66.0 | 86.3 | 54.9 | 76.6
TTQ |  Ternary: {-$\alpha$, 0, +$\beta$} | 66.6 | 87.2 | 57.5 | 79.7
INQ | 66.0 | 87.1 | - | -
Full-precision network | Full-Precision | 69.5 | 89.2 | 60.8 | 81.9

####  4.3.3 Performances on ImageNet

To investigate the performances of SCA on large-scale datasets, we conduct a
series of experiments on ImageNet. Limited by computation resources, we only
adopt two networks on ImageNet. We follow the competitors [17, 15] and use
18-layer ResNet [58] and AlexNet [60] architectures. We adopt the same
initialization strategy as that in TTQ [17] and LR-Net [18] by using the full-
precision weights as the initialization. For ResNet-18, $\lambda$ is initially
set to 1e-9 and is multiplied by 1e2 and 10 at the 50th epoch and the 70th
epoch, respectively; $\alpha$ is set to 0.3; weight decay is used in the last
layer with parameter 1e-6; the objective function is minimized with Adam with
mini-batch size 128; the initial learning rate is 0.005 and is divided by 10
after the 30th epoch, the 50th epoch, and the 70th epoch; we have trained the
network for 90 epochs. For AlexNet, $\lambda$ is initially set to 1e-9 and is
multiplied by 1e4 and 1e3 at the 70th epoch and the 100th epoch, respectively;
$\alpha$ is set to 1.0; weight decay is used in the last layer with parameter
$1e-6$; the objective function is minimized with Adam; the initial learning
rate is 0.005 and is divided by 2, 10, and 5 at the 50th epoch, the 70th
epoch, and the 100th epoch, respectively; we have trained the network for 130
epochs.

Table 4 reports the comparison results 555The results of LR-Net on AlexNet and
Extended [9, 10] are obtained from our implementation based on the paper.  on
ImageNet. Clearly, SCA outperforms the competitors with ternary weights {-1,
0, +1} in terms of both Top1 and Top5 accuracies by a large margin, and also
beats the competitors with the other kinds of ternary weights {$\alpha$, 0
$-\alpha$/$\beta$} significantly, which demonstrates the usefulness and
applicability of SCA on large-scale datasets. We also notice that even on
large-scale datae ImageNet, the performances of the ternary weight networks
trained by SCA are still close to those of the full-precision counterparts,
which demonstrates the effectiveness of SCA for converting full-precision
networks to the ternary versions on large-scale datasets.

####  4.3.4 Performances on CIFAR-100 and Tiny ImageNet

To further explore the performances of SCA, we evaluate it on more datasets,
i.e., CIFAR-100 and Tiny ImageNet. As the existing approaches did not report
the results on these two datasets, we only compare the performances of SCA
with the upper bound of ternary weight network performances, i.e., the
performances of full-precision networks. If SCA can match the upper bound, it
means that SCA at least does not perform worse than the existing approaches.
On CIFAR-100, the values of the hyperparameters are the same as those on
CIFAR-10 except that $\alpha$ is set to 1e-4 and 0.5 for VGG-S and VGG-
variate, respectively. On Tiny ImageNet, $\alpha$ is set to 0.1; $\lambda$ is
initially set to 5e-7 and is multiplied by 1e2 at the 90th epoch; weight decay
is used with parameter 1e-5; dropout with drop rate 0.5 is adopted; we have
the networks for 120 epochs.

Table 6 reports the comparison results on CIFAR-100 and Tiny ImageNet. It is
observed that the ternary weight networks trained by SCA consistently match
the performances of the full-precision counterparts on both datasets and both
networks. This demonstrates the effectiveness of SCA for compressing full-
precision weight networks to the ternary weight counterparts.

Table 5: Test accuracies (%) on CIFAR-100 and Tiny ImageNet

|  | VGG-S | VGG-Variate
---|---|---|---
CIFAR-100 | SCA (Ours) | 72.2 | 70.3
Full-precision network | 72.2 | 70.4
Tiny ImageNet | SCA (Ours) | 55.2 | 52.1
Full-precision network | 55.6 | 52.4

Table 6: Comparison between SCA and SCA without parameterization on CIFAR-10

| VGG-S | ResNet-20 | VGG-Variate
---|---|---|---
ES$\rm A_{w}$ | 10.00 | 10.00 | 10.00
SCA | 93.41 | 91.28 | 92.75

###  4.4 Effectiveness of Parameterization with $\tanh()$

Parameterization $\tanh(\Theta)$ in SCA is used to compress the full-precision
weight space so that we can search for a ternary weight solution more easily.
In this part, we explore the effectiveness of the parameterization. We compare
the performance of SCA with that of SCA without using parameterization
$\tanh(\Theta)$ (i.e., the objective function is reduced to
$\mathcal{L}(f(X,W),Y)+\lambda*R(W)$). We denote the latter by ES$\rm A_{w}$.
We use grid search to tune the hyperparameters in ES$\rm A_{w}$. However, we
find that the accuracy of ES$\rm A_{w}$ on CIFAR-10 is always 10% as there are
always some weights that are far from -1, 0, or +1 without the restriction of
$\tanh()$. As shown in Table 6, the accuracies drop substantially on CIFAR-10
for all the three networks without parameterization $\tanh()$, which indicates
the importance and the effectiveness of the parameterization $\tanh()$ for
training ternary weight networks.

Table 7: Sparsity (%) of ternary weights and accuracies (%) on MNIST with different values of controller $\alpha$ for given $\lambda$s |  |  $\alpha$ = 0 |  $\alpha$ = 1e-4 |  $\alpha$ = 1e-2 |  $\alpha$ = 0.1 |  $\alpha$ = 0.2 |  $\alpha$ = 0.5 |  $\alpha$ = 1 |  $\alpha$ = 2
---|---|---|---|---|---|---|---|---|---
$\lambda$ = 1e-5 | Sparsity | 0.008 | 0.096 | 2.89 | 29.69 | 61.31 | 99.63 | 99.89 | 99.97
Accuracy | 99.47 | 99.48 | 99.45 | 99.56 | 99.47 | 99.49 | 99.36 | 60.9
$\lambda$ = 1e-7 | Sparsity | 0.21 | 0.48 | 3.25 | 11.00 | 15.15 | 43.39 | 93.94 | 97.44
Accuracy | 99.53 | 99.56 | 99.40 | 99.48 | 99.50 | 99.50 | 99.40 | 99.42
$\lambda$ = 1e-9 | Sparsity | 11.48 | 12.65 | 13.41 | 18.30 | 25.86 | 33.46 | 51.99 | 67.21
Accuracy | 99.39 | 99.53 | 99.50 | 99.47 | 99.55 | 99.56 | 99.44 | 99.41
Table 8: Sparsity (%) of ternary weights and accuracies (%) of VGG-S on
CIFAR-10 with different values of $\alpha$ for given $\lambda$s

|  |  $\alpha$ = 0 |  $\alpha$ = 1e-4 |  $\alpha$ = 1e-2 |  $\alpha$ = 0.1 |  $\alpha$ = 0.2 |  $\alpha$ = 0.5 |  $\alpha$ = 1 |  $\alpha$ = 1.5 |  $\alpha$ = 2
---|---|---|---|---|---|---|---|---|---|---
$\lambda$ = 1e-7 | Sparsity | 0.0143 | 0.099 | 1.001 | 2.411 | 6.278 | 29.17 | 79.65 | 93.31 | 97.21
Accuracy | 92.77 | 92.66 | 92.42 | 92.83 | 92.68 | 92.61 | 93.23 | 92.36 | 91.11
$\lambda$ = 1e-8 | Sparsity | 0.29 | 0.32 | 1.58 | 3.91 | 5.38 | 8.56 | 33.92 | 70.28 | 85.77
Accuracy | 93.38 | 93.25 | 93.14 | 93.41 | 93.00 | 93.29 | 93.15 | 93.36 | 92.75
$\lambda$ = 1e-9 | Sparsity | 3.02 | 3.05 | 4.34 | 7.10 | 11.60 | 12.89 | 22.62 | 40.04 | 59.35
Accuracy | 93.05 | 92.90 | 92.97 | 92.96 | 92.94 | 93.05 | 92.63 | 93.38 | 92.67

###  4.5 Sparsity Control

In this part, we investigate how the sparsity (i.e., the percentage of 0s) of
the trained ternary weights varies with the values of controller $\alpha$ for
given coefficients $\lambda$. Table 7 and Table 8 report the results on MNIST
and CIFAR-10, respectively. It is observed that the ternary weights are
becoming increasingly sparser as $\alpha$ increases for given $\lambda$s on
both datasets, which verifies that the sparsity of the trained ternary weights
is positively related to $\alpha$. Essentially, when the ternary weights
become sparser, the accuracy only changes a little, which indicates that SCA
is able to control the sparsity of the trained ternary weights through
$\alpha$ with little or no accuracy changes. For example, on MNIST (Table 7),
for given $\lambda$ value 1e-5, when $\alpha$ is increased from 0 to 0.5, the
sparsity is increased from 0.008% (which means that there is almost no 0 in
the ternary weights) to 99.63% (which means that there are almost all 0s in
the ternary weights), but the accuracy is almost not changed (i.e., from
99.47% to 99.49%). Similarly, on CIFAR-10 (Table 8), for given $\lambda$ value
1e-7, when $\alpha$ is increased from 0 to 1, the sparsity is increased from
0.0143% (which means that there is almost no 0 in the ternary weights) to
79.65% (which means that most of the values in the ternary weights are 0s),
but the accuracy is only changed a little (i.e., from 92.77% to 93.23%).
Therefore, SCA can control the sparsity of the trained ternary weights through
$\alpha$ with little or no accuracy changes. We also notice an exception on
MNIST that when $\lambda$=1e-5 and $\alpha$=2, the accuracy drops
substantially due to over-sparsity with sparsity 99.97%. The reason is that
99.97% 0s make the ternary weight network too sparse to fit the data.

Figure 6: Feature visualization of odd layers of ResNet-18 on ImageNet images

###  4.6 Feature Visualization

The experimental results on several benchmarks datasets have shown that the
ternary weight networks trained by SCA perform on a par with the full-
precision counterparts. The natural question that arises is whether their
feature maps are similar. We visualize the average feature map in each odd
layer of ternary weight ResNet-18 and full-precision weight ResNet-18.

Figure 6 shows the feature comparison between each of the odd layers of
ternary ResNet-18 and the full-precision ResNet-18 on ImageNet images. It is
observed that their features in the shallow layers (e.g., layer 1-13) are
extremely similar and those in the top layers differ a little, which indicates
that the ternary weight networks trained by SCA are able to learn effective
features similar to those of the full-precision counterparts and thus
demonstrates the effectiveness of SCA for feature extraction.

###  4.7 Discussion

We have shown that SCA is able to convert a full-precision network to the
ternary version with no or little accuracy decrease. Besides the performance
improvements, the significance of SCA over the existing approaches is more at
its simplicity, effectiveness, and sparsity-control ability. Using a very
simple method to achieve even surpass the performances of the state-of-the-art
approaches and to control the sparsity of the ternary weights is the main
contributions of this work. From practical perspective, current state-of-the-
art models in various applications are built on full-precision weight networks
that require large amounts of computation and memory, which highly limits
their implementation on resource-limited devices. SCA paves the way for
converting full-precision networks to the ternary versions simply and
effectively with a very high code reuse rate, which makes the deployment on
resource-limited devices suitable. Second, SCA is able to control the sparsity
of the ternary weights to further accelerate the inference according to the
demands on portable devices.

##  5 Conclusion

It is well-known that full-precision DNNs have high-demanding memory and
computation requirements. In this work, we have proposed a simple yet
effective approach SCA to training ternary weight {-1, 0, +1} networks that
are dramatically efficient at inference and require much less memories. SCA is
able to control the sparsity of the ternary weights, which is to our best
knowledge the first work alone this line. We have theoretically and
empirically shown that the sparsity of the ternary weights is positively
related to controller $\alpha$. Extensive experiments on five benchmark
datasets have demonstrated the SCA outperforms the existing approaches
significantly and even matches the performances of the full-precision
counterparts.

## References

  * Girshick [2015] R. Girshick,  Fast r-cnn,  in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.
  * Redmon et al. [2016] J. Redmon, S. Divvala, R. Girshick, A. Farhadi,  You only look once: Unified, real-time object detection,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779–788.
  * Kroner et al. [2020] A. Kroner, M. Senden, K. Driessens, R. Goebel,  Contextual encoder–decoder network for visual saliency prediction,  Neural Networks 129 (2020) 261–270.
  * Clay et al. [2021] V. Clay, P. König, K.-U. Kühnberger, G. Pipa,  Learning sparse and meaningful representations through embodiment,  Neural Networks 134 (2021) 23–41.
  * Serban et al. [2016] I. V. Serban, A. García-Durán, C. Gulcehre, S. Ahn, S. Chandar, A. Courville, Y. Bengio,  Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus,  arXiv preprint arXiv:1603.06807 (2016).
  * Andreas et al. [2016] J. Andreas, M. Rohrbach, T. Darrell, D. Klein,  Learning to compose neural networks for question answering,  arXiv preprint arXiv:1601.01705 (2016).
  * Joulin et al. [2016] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov,  Bag of tricks for efficient text classification,  arXiv preprint arXiv:1607.01759 (2016).
  * Frankle and Carbin [2018] J. Frankle, M. Carbin,  The lottery ticket hypothesis: Finding sparse, trainable neural networks,  arXiv preprint arXiv:1803.03635 (2018).
  * Tang et al. [2017] W. Tang, G. Hua, L. Wang,  How to train a compact binary neural network with high accuracy?,  in: Thirty-First AAAI conference on artificial intelligence, 2017.
  * Darabi et al. [2018] S. Darabi, M. Belbahri, M. Courbariaux, V. P. Nia,  Bnn+: Improved binary network training,  arXiv preprint arXiv:1812.11800 (2018).
  * Zhang et al. [2018] D. Zhang, J. Yang, D. Ye, G. Hua,  Lq-nets: Learned quantization for highly accurate and compact deep neural networks,  in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 365–382.
  * Leng et al. [2018] C. Leng, Z. Dou, H. Li, S. Zhu, R. Jin,  Extremely low bit neural network: Squeeze the last bit out with admm,  in: Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  * Zhou et al. [2017] A. Zhou, A. Yao, Y. Guo, L. Xu, Y. Chen,  Incremental network quantization: Towards lossless cnns with low-precision weights,  in: International Conference on Learning Representations, 2017.
  * Xu et al. [2019] Y. Xu, W. Dai, Y. Qi, J. Zou, H. Xiong,  Iterative deep neural network quantization with lipschitz constraint,  IEEE Transactions on Multimedia (2019).
  * Li et al. [2016] F. Li, B. Zhang, B. Liu,  Ternary weight networks,  arXiv preprint arXiv:1605.04711 (2016).
  * Zhou et al. [2018] A. Zhou, A. Yao, K. Wang, Y. Chen,  Explicit loss-error-aware quantization for low-bit deep neural networks,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9426–9435.
  * Zhu et al. [2017] C. Zhu, S. Han, H. Mao, W. J. Dally,  Trained ternary quantization,  in: International Conference on Learning Representations, 2017.
  * Shayer et al. [2018] O. Shayer, D. Levi, E. Fetaya,  Learning discrete weights using the local reparameterization trick,  in: International Conference on Learning Representations, 2018.
  * Rastegari et al. [2016] M. Rastegari, V. Ordonez, J. Redmon, A. Farhadi,  Xnor-net: Imagenet classification using binary convolutional neural networks,  in: European Conference on Computer Vision, Springer, 2016, pp. 525–542.
  * Alemdar et al. [2017] H. Alemdar, V. Leroy, A. Prost-Boucle, F. Pétrot,  Ternary neural networks for resource-efficient ai applications,  in: 2017 International Joint Conference on Neural Networks (IJCNN), IEEE, 2017, pp. 2547–2554.
  * Mellempudi et al. [2017] N. Mellempudi, A. Kundu, D. Mudigere, D. Das, B. Kaul, P. Dubey,  Ternary neural networks with fine-grained quantization,  arXiv preprint arXiv:1705.01462 (2017).
  * Lin et al. [2017] X. Lin, C. Zhao, W. Pan,  Towards accurate binary convolutional neural network,  in: Advances in Neural Information Processing Systems, 2017, pp. 345–353.
  * McDonnell [2018] M. D. McDonnell,  Training wide residual networks for deployment using a single bit for each weight,  in: International Conference on Learning Representations, 2018.
  * Zhuang et al. [2019] B. Zhuang, C. Shen, M. Tan, L. Liu, I. Reid,  Structured binary neural networks for accurate image classification and semantic segmentation,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 413–422.
  * Martinez et al. [2020] B. Martinez, J. Yang, A. Bulat, G. Tzimiropoulos,  Training binary neural networks with real-to-binary convolutions,  in: International Conference on Learning Representations, 2020.
  * Stock et al. [2020] P. Stock, A. Joulin, R. Gribonval, B. Graham, H. Jégou,  And the bit goes down: Revisiting the quantization of neural networks,  in: International Conference on Learning Representations, 2020.
  * Soudry et al. [2014] D. Soudry, I. Hubara, R. Meir,  Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights,  in: Advances in Neural Information Processing Systems, 2014, pp. 963–971.
  * Meng et al. [2020] X. Meng, R. Bachmann, M. E. Khan,  Training binary neural networks using the bayesian learning rule,  International Conference on Machine Learning (2020).
  * Stamatescu et al. [2020] G. Stamatescu, I. Fuss, L. B. White, F. Gerace, C. Lucibello,  Critical initialisation in continuous approxi-mations of binary neural networks,  in: International Conference on Learning Representations, 2020.
  * Shekhovtsov et al. [2020] A. Shekhovtsov, V. Yanush, B. Flach,  Path sample-analytic gradient estimators for stochastic binary networks,  Advances in neural information processing systems (2020).
  * Esser et al. [2020] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, D. S. Modha,  Learned step size quantization,  in: International Conference on Learning Representations, 2020.
  * Li et al. [2020] Y. Li, X. Dong, W. Wang,  Additive powers-of-two quantization: A non-uniform discretization for neural networks,  in: International Conference on Learning Representations, 2020.
  * Bulat et al. [2021] A. Bulat, B. Martinez, G. Tzimiropoulos,  High-capacity expert binary networks,  International Conference on Learning Representations (2021).
  * Courbariaux et al. [2015] M. Courbariaux, Y. Bengio, J.-P. David,  Binaryconnect: Training deep neural networks with binary weights during propagations,  in: Advances in neural information processing systems, 2015, pp. 3123–3131.
  * Hubara et al. [2016] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, Y. Bengio,  Binarized neural networks,  in: Advances in neural information processing systems, 2016, pp. 4107–4115.
  * Wang et al. [2019] Z. Wang, J. Lu, C. Tao, J. Zhou, Q. Tian,  Learning channel-wise interactions for binary convolutional neural networks,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 568–577.
  * Peters and Welling [2018] J. W. Peters, M. Welling,  Probabilistic binary neural networks,  arXiv preprint arXiv:1809.03368 (2018).
  * Liu et al. [2018] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, K.-T. Cheng,  Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm,  in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 722–737.
  * Denton et al. [2014] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, R. Fergus,  Exploiting linear structure within convolutional networks for efficient evaluation,  in: Advances in neural information processing systems, 2014, pp. 1269–1277.
  * Han et al. [2015] S. Han, H. Mao, W. J. Dally,  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,  arXiv preprint arXiv:1510.00149 (2015).
  * Hu et al. [2016] H. Hu, R. Peng, Y.-W. Tai, C.-K. Tang,  Network trimming: A data-driven neuron pruning approach towards efficient deep architectures,  arXiv preprint arXiv:1607.03250 (2016).
  * He et al. [2017] Y. He, X. Zhang, J. Sun,  Channel pruning for accelerating very deep neural networks,  in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1389–1397.
  * Luo et al. [2017] J.-H. Luo, J. Wu, W. Lin,  Thinet: A filter level pruning method for deep neural network compression,  in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 5058–5066.
  * Wang et al. [2016] Y. Wang, C. Xu, S. You, D. Tao, C. Xu,  Cnnpack: Packing convolutional neural networks in the frequency domain,  in: Advances in neural information processing systems, 2016, pp. 253–261.
  * Zhuang et al. [2018] Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, J. Zhu,  Discrimination-aware channel pruning for deep neural networks,  in: Advances in Neural Information Processing Systems, 2018, pp. 875–886.
  * Renda et al. [2020] A. Renda, J. Frankle, M. Carbin,  Comparing rewinding and fine-tuning in neural network pruning,  in: International Conference on Learning Representations, 2020.
  * Denil et al. [2013] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, N. De Freitas,  Predicting parameters in deep learning,  in: Advances in neural information processing systems, 2013, pp. 2148–2156.
  * Kim et al. [2015] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, D. Shin,  Compression of deep convolutional neural networks for fast and low power mobile applications,  arXiv preprint arXiv:1511.06530 (2015).
  * Ren et al. [2018] W. Ren, J. Zhang, L. Ma, J. Pan, X. Cao, W. Zuo, W. Liu, M.-H. Yang,  Deep non-blind deconvolution via generalized low-rank approximation,  in: Advances in Neural Information Processing Systems, 2018, pp. 297–307.
  * LeCun and Cortes [2010] Y. LeCun, C. Cortes,  MNIST handwritten digit database (2010). URL: http://yann.lecun.com/exdb/mnist/.
  * Krizhevsky and Hinton [2009] A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, Technical Report, Citeseer, 2009.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei,  Imagenet: A large-scale hierarchical image database,  in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * Szegedy et al. [2015] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich,  Going deeper with convolutions,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * Zhou et al. [2016] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, Y. Zou,  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,  arXiv preprint arXiv:1606.06160 (2016).
  * Srivastava et al. [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov,  Dropout: A simple way to prevent neural networks from overfitting,  Journal of Machine Learning Research 15 (2014) 1929–1958.
  * Glorot and Bengio [2010] X. Glorot, Y. Bengio,  Understanding the difficulty of training deep feedforward neural networks,  in: Proceedings of the thirteenth international conference on artificial intelligence and statistics, 2010, pp. 249–256.
  * Kingma and Ba [2015] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  in: International Conference on Learning Representations, 2015.
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * Cai et al. [2017] Z. Cai, X. He, J. Sun, N. Vasconcelos,  Deep learning with low precision by half-wave gaussian quantization,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5918–5926.
  * Krizhevsky et al. [2012] A. Krizhevsky, I. Sutskever, G. E. Hinton,  Imagenet classification with deep convolutional neural networks,  in: Advances in neural information processing systems, 2012, pp. 1097–1105.