# Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via
Self-Distillation and Weight Factorization

Yufei Guo111Equal Contributions. Weihang Peng222Equal Contributions. Yuanpei
Chen  Liwen Zhang  Xiaode Liu  Xuhui Huang  Zhe Ma

###### Abstract

Emerged as a biology-inspired method, Spiking Neural Networks (SNNs) mimic the
spiking nature of brain neurons and have received lots of research attention.
SNNs deal with binary spikes as their activation and therefore derive extreme
energy efficiency on hardware. However, it also leads to an intrinsic obstacle
that training SNNs from scratch requires a re-definition of the firing
function for computing gradient. Artificial Neural Networks (ANNs), however,
are fully differentiable to be trained with gradient descent. In this paper,
we propose a joint training framework of ANN and SNN, in which the ANN can
guide the SNN’s optimization. This joint framework contains two parts: First,
the knowledge inside ANN is distilled to SNN by using multiple branches from
the networks. Second, we restrict the parameters of ANN and SNN, where they
share partial parameters and learn different singular weights. Extensive
experiments over several widely used network structures show that our method
consistently outperforms many other state-of-the-art training methods. For
example, on the CIFAR100 classification task, the spiking ResNet-18 model
trained by our method can reach to 77.39% top-1 accuracy with only 4 time
steps.

###### keywords:

Spiking Neural Networks , Artificial Neural Networks , Knowledge Distillation
, Weight Factorization

††journal: Pattern Recognition

##  1 Introduction

The Spiking Neural Networks (SNNs) have been recognized as one of the next-
generation neural networks [1]. This type of neural network is known for its
bio-mimicry of the brain neurons, which utilize “spikes” for information
communication and can process data in a spatial-temporal manner [2, 3]. SNNs
imitate such spike mechanisms and thus received a lot of research attention.

Through the time dimension of spiking neurons, each of them fires a spike only
if a variable called membrane potential transcends the threshold, if not, the
spiking neuron would remain still. As a result, the activation would be either
1 (fire a spike) or 0 (remain silent), which eliminates multiplication in the
neural networks and brings energy efficiency. For example, some work shows
that SNNs can save orders of magnitude energy over Artificial Neural Networks
(ANNs) [4, 5]. In addition, the temporal processing ability of SNNs makes them
unique to ANNs in a way that learns spatial-temporal information.

Despite its energy efficiency and special spatial-temporal processing ability,
training SNNs is challenging because spiking neurons fire discrete spikes
whose gradients are not well-defined. Consequentially, the zero-but-all
gradient makes it impossible to train SNNs via gradient-based optimization
methods. To address this, various training methods have been proposed. First,
spike-timing-dependent plasticity (STDP) [6] approaches [7, 8] leverage an
unsupervised learning algorithm called Hebbian learning [9] to update the
weights. STDP is inspired by biology where the spiking time difference can
determine the connection of the synaptic weight. Yet, STDP is limited to
small-scale datasets, which may be due to the lack of global information about
the error.

Second, the Surrogate Gradient (SG) tries to find an alternative function when
doing back-propagation of the spiking neurons. Therefore, SG can be
incorporated into the current gradient-based optimization framework. SG
provides decent performance and can narrow the time steps into (_i.e_. 5
$\sim$ 20) even on a large-scale dataset such as ImageNet [10]. However, the
SG inevitably introduces some error in calculating the gradient, though
sometimes it is hard to estimate the error, which, makes the convergence
unstable and slower than ANN training.

Last, the ANN-to-SNN conversion [11, 12] offers another approach. It utilizes
the well-trained ANN checkpoint and converts it into SNN by replacing the
activation function from ReLU with spiking activation. It provides a fast way
to obtain an SNN without using gradient descent at all. The disadvantage of
this type of method is that SNN does not have its own learned feature, rather,
all SNN does is to mimic ANN.

In this work, we propose to employ a joint training framework of both ANN and
SNN. During training, the parameters of ANN and SNN are shared partially.
Specifically, we apply the matrix factorization of weights via Singular Value
Decomposition (SVD) and let ANN and SNN optimize their own singular weights
but keep the singular vectors the same. Second, we propose adding multiple
branches in the network to distill the knowledge from ANN to SNN, called self-
distillation. Our method can be viewed as a combination of ANN-SNN conversion
and SG gradient training. We do not alter the SG training algorithm in SNN,
but we provide guidance during the training of SNN. Unlike ANN-SNN conversion,
where the ANN and SNN share the same parameters, we add more degree of
freedom, _i.e_., different singular values in parameters, but restricting them
to be totally different. The overall workflow can be seen in Fig. 1. Extensive
experiments over several popular network structures show that our joint
training framework consistently outperforms state-of-the-art training methods.
For example, on the CIFAR100 classification task, we can train a spiking
ResNet-18 and achieve 77.39% top-1 accuracy.

Figure 1: The overall joint training framework. To better transfer knowledge,
some multiple branches with a global average pooling layer and a fully-
connected layer are added inside the network. Then three kinds of distillation
losses are applied to these branches: the cross-entropy (CE) loss between the
output and the label, the Kullback-Leibler Divergence (KLD) loss between the
ANN’s output and the SNN’s output, and the L2 loss between the output features
of ANN and SNN.

In summary, our contributions are three-fold:

  1. 1.

We propose a joint training framework where the knowledge of ANN is
consistently transferred to the SNN during training.

  2. 2.

We factorize the parameters of both ANN and SNN and restrict them to have the
same singular vectors while learning different singular values.

  3. 3.

Extensive experiments are conducted on vision benchmarks, demonstrating that
our method reaches SOTA.

The remainder of the paper is organized as follows: Section 2 provides a
background for a better understanding of the proposed method. Next in Section
3, some preliminaries related to the method are introduced. In Section 4, the
motivation of the paper, our ANN-SNN joint training framework, and the
corresponding weight factorization method for the parameters in trained ANN
and SNN are introduced in detail sequentially. Moreover, an overall algorithm
pseudocode for better explaining the workflow of the proposed method is given.
In Section 5, experimental settings and performance results of the proposed
method and comparisons with other SoTA methods are presented. Finally, we
summarize the research.

##  2 Related Work

###  2.1 Spiking Neural Networks (SNNs).

The SNN as a biology-inspired method, is recognized as a potential
architecture to reach high-level artificial intelligence. Generally, there are
two routers to obtain the high-performance SNNs: (1) converting a trained ANN
to an SNN [13, 14] and (2) training the SNN from scratch directly [15, 16].
The ANN-SNN conversion method trains a well-performed ANN first and then
modifies the ANN ReLU activation to the SNN spike activation and reuses other
parameters of the ANN to generate the same architecture-based SNN. To further
improve the converted SNN accuracy, some interesting methods are proposed. For
example, in [11] and [12], the conversion error is decomposed to each layer
and is reduced via calibrating the parameters. [17, 18] further advance the
analysis of conversion error and formulate a conversion-aware training
technique for ANN. Though this kind of method can obtain an SNN in a short
time, getting a high-accuracy SNN requires many time steps which will increase
the inference time at the same time. Training an SNN directly from scratch can
greatly reduce time steps, even less than 5 [19, 20, 21], which receives more
research attention recently. Recent works, such as [22, 23], by co-optimizing
parameters, firing threshold, and leaky factor together respectively, have
obtained comparable SNNs as the conversion method. However, all these methods
do not solve the non-differentiability of the firing function and the gradient
explode/vanish well, which will be introduced in detail in Section 4.1. In
this work, we focus on solving this problem.

###  2.2 Knowledge Distillation.

Knowledge distillation (KD) was originally introduced as a kind of model
compression method by [24]. [24] trains a small student model via transferring
the knowledge of a rather large trained teacher model to the small one. In his
work, the knowledge is defined as the teacher’s outputs after the final
softmax layer. It carries richer information than one-hot labels since it can
provide the inter-class similarities information learned by the teacher, which
the labels do not enjoy. Following this, many works extract the intermediate
representations of the teacher as the knowledge to facilitate the optimization
of the student, such as intermediate feature tensors [25] or attention maps
[26]. Another line of KD methods transfers the feature relationships rather
than the actual features themselves [27]. Along with classification tasks,
Many works that use the KD based learning in other tasks, e.g., action
detection [28], text recognition [29], continuous emotion recognition [30],
and object detection [31]. There are also some works that introduce the KD
method in the SNN domain [32]. However, they all adopt a big teacher SNN model
to guide the small SNN counterpart learning. In this paper, rather than
compress the SNN model, we use the KD method to solve the obstacle of the
gradient exploding/varnish which avoids the good training of SNNs.

##  3 Preliminary

###  3.1 Leaky Integrate-and-Fire Model

We first introduce the notation following [33]. Throughout the paper, vectors
are denoted by bold italic letters. For instance, $\bm{x}$ and $\bm{y}$ denote
the input and target output variables. Matrices or tensors as clear from the
text are denoted by bold capital letters (_e.g_. $\mathbf{W}$). Constants are
represented by small upright letters, e.g., $t$. We can denote the time
dimension and the element indices with bracketed superscripts and subscripts,
respectively. For example, $\bm{u}_{i}^{(t)}$ means the $i$-th membrane
potential at time step $t$.

In this paper, the spiking neurons we use for SNNs are the well-known Leaky
Integrate-and-Fire (LIF) neuron model [34]. The variable controlling the
dynamics of LIF is called membrane potential, denoted by $\bm{u}$. At each
time step, the membrane potential is updated by

$\bm{u}^{(t+1),\text{pre}}=\tau\bm{u}^{(t)}+\bm{c}^{(t+1)},\text{where
}\bm{c}^{(t+1)}=\mathbf{W}\bm{x}^{(t+1)},$ (1)

where $\tau$ is a constant within $(0,1)$ and controls the leakage of membrane
potential. $\bm{c}^{(t+1)}$ is the pre-synaptic input at time step $t+1$,
which is charged by input current $\bm{c}^{(t+1)}$. The input current can be
calculated from the dot-product between the weights and the spike from the
previous layer. Once the membrane potential exceeds the firing threshold
$V_{th}$, a spike will be fired from the LIF neuron, given by

$\bm{y}^{(t+1)}=\begin{cases}1&\text{if }\bm{u}^{(t+1),\text{pre}}>V_{th}\\\
0&\text{otherwise}\end{cases},\ \ \
\bm{u}^{(t+1)}=\bm{u}^{(t+1),\text{pre}}\cdot(1-\bm{y}^{(t+1)})$ (2)

After firing, the spike output $\bm{y}^{(t+1)}$ will propagate to the next
layer and become the input $\bm{x}^{(t+1)}$ of the next layer, (note that we
omit the layer index for simplicity).

The Classifier in SNN.  Generally, in a neural network-based classification
model, the final network output will be used to compute the softmax and
predict the desired class object. If we directly output the number of spikes
to compute the probability, it will lose too much information. This is because
the output of the network could be positive and negative as well. The spikes,
however, can only output positive values. For this reason, we choose to only
integrate the network output and do not fire them across time, as did in
recent practice [35].

$\bm{y}_{\text{net}}=\frac{1}{T}\sum_{i=1}^{T}\bm{c}_{\text{net}}^{(t)}=\frac{1}{T}\sum_{i=1}^{T}\mathbf{W}\bm{x}^{(t)},$
(3)

Then, we can compute the cross-entropy loss based on the true label and
$\mathrm{Softmax}(\bm{y}_{\text{net}})$.

##  4 Method

In this section, we introduce our ANN-SNN joint training framework. First, we
discuss the motivation of this work, _i.e_. the gradient computation problem
in spiking neurons. Then, we present self-distilled training with multiple
branches. In the end, we present the weight factorization for the parameters
in ANN and SNN.

###  4.1 Motivation

The most notorious problem in SNN training may be the non-differentiability of
the firing function Eq. (missing) 2. To concretely discuss this problem, we
denote the loss function as $L$ and calculate the gradients w.r.t. weights
using the chain rule:

$\frac{\partial L}{\partial\mathbf{W}}=\sum_{t}\frac{\partial
L}{\partial\bm{y}^{(t)}}\frac{\partial\bm{y}^{(t)}}{\partial\bm{u}^{(t),\text{pre}}}\frac{\partial\bm{u}^{(t),\text{pre}}}{\partial\bm{c}^{(t)}}\frac{\partial\bm{c}^{(t)}}{\partial\mathbf{W}}.$
(4)

Here, the firing function (Eq. (missing) 2) is similar to the $\mathrm{sign}$
function. The gradient of $\mathrm{sign}$ is 0 almost everywhere except for
the threshold. Recall that the gradient descent updates the parameters by
subtracting the learning rate multiplied by the gradient, therefore, the
actual updates for weights would either be 0 or infinity.

As mitigation of this problem, the surrogate gradient was proposed. When
performing the forward pass, the firing function remains exactly the same,
however, when performing the backward pass, the firing function becomes a
surrogate function. And we can compute the surrogate gradient function.

A popular surrogate gradient may refer to the rectangular function proposed in
[16]. Another type is called rectangular function, given by

$\begin{cases}\frac{\partial\bm{y}^{(t)}}{\partial\bm{u}^{(t),\text{pre}}}&=\gamma\max\left(0,1-\left|\frac{\bm{u}^{(t),\text{pre}}}{V_{th}}-1\right|\right),\\\
\frac{\partial\bm{y}^{(t)}}{\partial\bm{u}^{(t),\text{pre}}}&=\frac{1}{a}\mathrm{sign}\left(\left|\bm{u}^{(t),\text{pre}}-V_{th}\right|<\frac{a}{2}\right).\end{cases}$
(5)

Both of them have a hyper-parameter to control the width and the sharpness of
the surrogate gradient. The rectangular function will become the Straight-
Through Estimator [36] if $V_{th}=0.5,a=1$. The triangular is computed based
on the distance between the potential and threshold.

Gradient Explode/Vanish.  The gradient explode or vanish problem in ANN can be
effectively mitigated by residual block, which contains the skip connection
[37]. The skip connection can be formulated by

$\bm{y}=g(f(\bm{x})+\bm{x}),$ (6)

where $g(\cdot)$ is the activation function, _e.g_. ReLU in ANN, and
$f(\cdot)$ is the convolutional layers and activation layers in the main path.
Since ReLU is unbounded for the positive part, the gradient can be passed to
the input of the block. However, in the case of LIF neurons, the gradient will
be reduced through SG. Either $\bm{u}^{(t),\text{pre}}>2V_{th}$ or
$\bm{u}^{(t),\text{pre}}<0$ the gradient of LIF will be 0. Moreover, the
output of $g(\cdot)$ is 0/1, which cannot effectively carry the information
from the input tensor to the network output. Thus, even using ResNet and SG
the SNN still cannot be well-trained due to the potential gradient
exploding/varnish.

###  4.2 Self Distillation

Here, we propose our joint-training framework, which composes of distillation
of partial weight sharing. Let us divide the ResNet into several parts. The
ResNet has a stem (the first convolution), a body, and a head (the global
average pooling layers and the classifier). The body is made of 4 stages,
where each stage composes several residual blocks. The residual blocks have
two or three convolution layers and the skip connection. As we discussed
before, even equipped with residual blocks, the SNN may still suffer from the
gradient explode/vanish problem. Thus, we add knowledge distillation from ANN
to SNN. Particularly, the source of knowledge comes from not only the network
output but also the intermediate output. Similar to Branchynet [38] we add
multiple branches inside the network. In total, there are four intermediate
branches, which are followed right after the end of each stage of the ResNet
with a global average pooling layer and a fully-connected layer.

The overall loss function for this distillation framework is composed of 4
parts:

  1. 1.

Loss 1: The cross-entropy (CE) loss between the output and the label. Note
that, the CE loss is introduced for both ANN and SNN, and also both the
intermediate output from ANN and SNN. Thus, all branches in the framework
receive direct knowledge from the labels. It can be formulated by:

$L_{\text{CE}}=\sum_{i=1}^{4}CrossEntropy(\bm{y}_{\text{ANN},i,\bm{z}})+\sum_{i=1}^{4}CrossEntropy(\bm{y}_{\text{SNN},i,\bm{z}}),$
(7)

where both ANN and SNN have 4 stages and thus there are 8 CE losses.

  2. 2.

Loss 2: The Kullback-Leibler Divergence (KLD) loss between the ANN’s output
and the SNN’s output. This loss function is computed based on the softmax
(_i.e_. class probabilities), and ensures that the output from SNN should
mimic that from ANN. It can be formulated by:

$L_{\text{KLD}}=\sum_{i}^{4}KLD(\mathtt{detach}(\bm{y}_{\text{ANN},i}),\
\bm{y}_{\text{SNN},i}).$ (8)

Again, this loss function is applied to all branches of the network. Note that
the output from ANN is detached from calculating the gradients. Otherwise, the
ANN’s output will be pulled over to the SNN’s output, leading to an accuracy
decrease.

  3. 3.

Loss 3: The L2 loss from hints. Due to the temporal dimension and binary
activation in SNN architecture, the features in ANN and the features in SNN
may have different norms of magnitude. Therefore, we impose L2-norm loss
between the output features of ANN and SNN, given by

$L_{\text{Norm}}=\sum_{i}^{4}||\mathtt{detach}(\bm{y}_{\text{ANN},i})-\
\bm{y}_{\text{SNN},i}||_{F}^{2},$ (9)

Similar to KLD loss, the L2 norm loss also detaches the output from ANN from
computing the gradients.

Together, we optimize these three loss functions simultaneously to transfer
the knowledge from ANN to SNN. Remarkably, the loss function from the first
and second stage’s outputs provides the guiding information for shallow
layers, which alleviates the gradient explode/vanish problems. The overall
loss function can be written by

$L=L_{\text{CE}}+\lambda_{1}L_{\text{KLD}}+\lambda_{2}L_{\text{Norm}},$ (10)

where $\lambda_{1}$ and $\lambda_{2}$ are the hyperparameters for controlling
the distillation loss.

###  4.3 Weight-Factorization Training

The second ingredient of our joint-training framework is weight-factorization
training (WFT). This method is inspired by ANN-SNN conversion. Conventionally,
the ANN-SNN conversion directly copy-pastes the weight and bias parameters
from ANN to SNN [39]. This method is only effective when the time steps in SNN
are large enough. However, in our training framework the SNN is trained with
less than 10 time steps. Thus if we keep the parameters in the ANN and the SNN
identical, the performance of the SNN is limited. Recently, [12] suggests that
a proper _calibration_ of the weights in ANN is needed for conversion to SNN
in extremely low time steps. This finding inspires us to develop a partial
weight-sharing regime for the joint training of ANN and SNN. In order to do
so, we apply the Singular Value Decomposition (SVD) to the weights parameters
and share the eigenvectors for ANN and SNN while keeping the eigenvalues
optimized at the discretion of the ANN and SNN separately. Intuitively, let us
first consider a simple case: a fully-connected layer
$\mathbf{W}\in\mathbb{R}^{c_{in}\times c_{out}}$. We can apply the SVD to the
weight matrix, given by

$\mathbf{W}=\mathbf{U\Sigma
V}^{\top}=\begin{bmatrix}\bm{u}_{1}&\bm{u}_{2}&\cdots&\bm{u}_{c_{in}}\end{bmatrix}\begin{bmatrix}\sigma_{1}&0&\cdots&0\\\
0&\sigma_{2}&\cdots&0\\\ \vdots&0&\ddots&0\\\ 0&0&\cdots&\sigma_{r}\\\
\end{bmatrix}\begin{bmatrix}\bm{v}_{1}\\\ \bm{v}_{2}\\\ \vdots\\\
\bm{v}_{c_{out}}\end{bmatrix},$ (11)

where $\bm{u}_{i}\in\mathbb{R}^{c_{in}}$ and
$\bm{v}_{i}\in\mathbb{R}^{c_{out}}$ are the $i$-th eigenvectors of
$\mathbf{WW}^{\top}$ and $\mathbf{W}^{\top}\mathbf{W}$, respectively. $\sigma$
is the singular value and $r$ is the number of singular value. SVD can
decompose weights into:

$\mathbf{W}=\sum_{i=1}^{r}\sigma_{i}\bm{u}_{i}\bm{v}_{i}^{\top}.$ (12)

In other words, SVD can decompose the weight matrix into a weighted sum of
several rank-1 matrices. In our WFT, we let SNN and ANN have the same
eigenvectors but different eigenvalues, given by

$\mathbf{W}_{\text{ANN}}=\mathbf{U\Sigma}_{\text{ANN}}\mathbf{V},\ \ \
\mathbf{W}_{\text{SNN}}=\mathbf{U\Sigma}_{\text{SNN}}\mathbf{V}.$ (13)

In this case, the SNN and ANN do not share parameters completely, providing
more degree of freedom compared to the ANN-SNN conversion works. In our
implementation, we re-parameterize the weights using Eq. (13) and directly
update $\mathbf{U,V,\Sigma}_{\text{ANN}},$ and $\mathbf{\Sigma}_{\text{SNN}}$
with gradient descent. The initial values are determined by applying the SVD
for the initialized weights. The singular values in the ANN and SNN are also
initialized to the same. We put the overall algorithm pseudocode in Algo. 1.

Input: ANN/SNN to be trained; Training dataset, total training epoch $E$, the
distillation hyper-parameter $\lambda_{1}$ and $\lambda_{2}$;

Initialize the weight parameters $\mathbf{W}$;

Apply the SVD to $\mathbf{W}$ and obtain $\mathbf{U,V,\Sigma}_{\text{ANN}},$
$\mathbf{\Sigma}_{\text{SNN}}$;

for  _all $e=1,2,\dots,E$-th epoch_ do

Compose the weights:
$\mathbf{W}_{\text{ANN}}=\mathbf{U\Sigma}_{\text{ANN}}\mathbf{V}$,
$\mathbf{W}_{\text{SNN}}=\mathbf{U\Sigma}_{\text{SNN}}\mathbf{V}$ ;

Get all outputs $\bm{y}_{\text{ANN},i}$, $\bm{y}_{\text{SNN},i}$;

Compute CE loss $L_{\text{CE}}$ (cf. Eq. (missing) 7) ;

Compute KLD loss $L_{\text{CE}}$ (cf. Eq. (missing) 8) ;

Compute Norm loss $L_{\text{CE}}$ (cf. Eq. (missing) 9) ;

Back-propagation and compute the gradients:
$\nabla_{\mathbf{U}}L,\nabla_{\mathbf{V}}L,\nabla_{\mathbf{\Sigma}_{\text{ANN}}}L,\nabla_{\mathbf{\Sigma}_{\text{SNN}}}L$;

Descend loss function and update weights;

return trained SNN and trained ANN.

Algorithm 1 A joint training framework for ANN and SNN

####  4.3.1 Memory Complexity

Our method does not largely increase the memory of weights training when
compared to the separate training baseline, _i.e_. optimize
$\mathbf{W}_{\text{ANN}}$ and $\mathbf{W}_{\text{SNN}}$ separately. The number
of elements for our method is $c_{in}^{2}+c_{out}^{2}+2r$ where
$r=\min(c_{in},c_{out})$, while the number of elements for the baseline is
$2c_{in}c_{out}$. For most convolutional layers in ResNets, the $c_{in}$ is
the same with $c_{out}$, therefore, our method only has $2c_{in}$ additional
parameters, with the same order of magnitude with bias parameters. For the
downsampling layers, we have $c_{out}=2c_{in}$ and thus our method has 20%
more parameters, which is affordable considering that there are only 3
downsampling layers in ResNets.

####  4.3.2 Generalization to Convolutional Layers

For Conv layers, the weights are 4D tensors, where kernel sizes add up to two
additional dimensions. In our implementation, these two dimensions do not
affect our implementations. We can apply the same SVD decomposition in every
entry of the kernel. As a consequence, all layers can be implemented with WFT.

###  4.4 Extending to Other Networks

In addition to the ResNet we discussed before, our joint learning framework
can be applied to other network architectures like VGG-series [40]. For these
modern convolutional architectures, the network is usually partitioned into
four different stages, each stage corresponding to a downsample of the input
feature map and an increase of the channel numbers. We add the distillation
loss function to the exit of these four stages. Thus, our method can
seamlessly incorporate most convolutional networks.

##  5 Experiments

In this section, we demonstrate the effectiveness and efficiency of our
algorithms. We first briefly illustrate the implementation details of our
experiments and then compare the results with existing state-of-the-art. We
also provide ablation studies.

###  5.1 Implementation Details

Table 1: Comparison between our algorithm and existing algorithms on
CIFAR-10/100 datasets with ResNets. Our method improves network performance
across all tasks.

Dataset | Method | Architecture | Time Steps | Accuracy
---|---|---|---|---
CIFAR10 | Hybrid training [41] | ResNet-20 | 250 | 92.22
TTBR [42] | ResNet-18 | 64 | 95.04
TSSL-BP [43] | CIFARNet | 5 | 91.41
TET [44] | ResNet-19 | 4 | 94.44
RecDis-SNN [45] | ResNet-19 | 2 | 93.64
Real Spike [46] | ResNet-20 | 4 | 92.53
2 | 90.47
Dspike [33] | ResNet-18 | 4 | 93.66
2 | 93.13
STBP-tdBN [19] | ResNet-19 | 4 | 92.92
2 | 92.34
Joint A-SNN (Ours) | ResNet-18 | 4 | 95.45
2 | 94.01
ResNet-34 | 4 | 96.07
2 | 95.13
CIFAR100 | TTBR [42] | ResNet-18 | 64 | 78.45
Diet-SNN [35] | ResNet-20 | 5 | 64.07
TET [44] | ResNet-19 | 4 | 74.47
RecDis-SNN [45] | ResNet-19 | 4 | 74.10
Real Spike [46] | ResNet-20 | 4 | 64.87
2 | 63.40
Dspike [33] | ResNet-18 | 4 | 73.35
2 | 71.68
Joint A-SNN (Ours) | ResNet-18 | 4 | 77.39
2 | 75.79
ResNet-34 | 4 | 79.76
| 2 | 77.11

Table 2: Comparison between our algorithm and existing algorithms on
CIFAR-10/100 datasets with VGGs.

Dataset | Method | Architecture | Time Steps | Accuracy
---|---|---|---|---
CIFAR10 | Hybrid training [41] | VGG-16 | 100 | 91.13
PLIF [22] | VGG-11 | 8 | 93.50
Temp Prune [47] | VGG-16 | 1 | 93.05
Joint A-SNN (Ours) | VGG-16 | 1 | 93.79
CIFAR100 | Hybrid training [41] | VGG-11 | 125 | 67.87
Temp Prune [47] | VGG-16 | 1 | 70.15
Joint A-SNN (Ours) | VGG-16 | 1 | 74.24

We use ResNet-18 and ResNet-34 as our selected architectures for experiments.
We also use VGG-16 [40] in some cases. The distillation hyper-parameters are
set to $\lambda_{1}=1$ and $\lambda_{2}=0.3$. We use Adam optimizer [48] with
a learning rate of $1e-3$. The learning rate is decayed using the cosine
annealing strategy [49]. The weight decay is set to $5e-4$ and we train all
models for 300 epochs and report the test accuracy in the last epoch. We run
the model with 4 GTX 1080Tis. We verify our models on CIFAR-10, CIFAR-100
[50], as well as Tiny-ImageNet [10]. We encode the images to spike using the
first layer in the SNN, as adopted in recent works[46, 44]. We also give an
introduction for each dataset:

CIFAR10.  CIFAR-10 contains 50k $32\times 32$ training images and 10k test
images. It is divided into 10 classes. The network kernel sizes and strides of
neural networks are adjusted to fit the size of this dataset.

CIFAR100.  CIFAR-100 consists of 50k $32\times 32$ training images and 10k
test images. It is divided into 100 classes. Similarly, the network kernel
sizes and strides of neural networks are also adjusted to fit the size of this
dataset.

Tiny-ImageNet.  Tiny-ImageNet is the modified subset of the original ImageNet
dataset [10]. Here, there are 200 different classes of ImageNet dataset, with
100k training and 10k validation images. The resolution of the images is 64×64
pixels.

###  5.2 Comparison with State of the Art

In this section, we compare our method with the existing state-of-the-art
works. We first focus on the evaluation of SNNs on CIFAR-10 and CIFAR-100
datasets. We adopt Hybrid training, TTBR, Temp Prune, Diet-SNN, STBP-tdBN,
TSSL-BP, PLIF, Dspike, Real Spike, RecDis, and TET as our comparison (see
reference in the results table) and summarize the accuracy comparison in Table
1. For the CIFAR-10 dataset, the highest accuracy from prior work is 94.44
with ResNet-19. While our joint-training method achieves 1% absolute
improvements with the same time steps based on the ResNet-18. And the
ResNet-19 has 2x channel numbers and more than 10x computation cost than
ResNet-18. It is also worth noting that with only 2 time steps, our method can
also outperform the Real Spike and the Dspike with 4 time steps with 1.48% and
0.35% accuracy, respectively. These comparison results clearly show the
efficiency and effectiveness of our method.

For the CIFAR-100 dataset, the TET [44] and Dspike [33] achieve the best
accuracies, which are 73.35 and 74.47, respectively. Our joint training method
greatly improves the accuracy to 77.39, nearly 3% improvement. Notably, the
SNN trained by our method with even half-time steps (_i.e_., 2) can obtain
75.8% accuracy, outperforming the state of the arts.

Moreover, we train a 34-layer network with our joint-training framework,
pushing the limit of SNN higher. For example, on CIFAR-10, our ResNet-34
reaches 96.07% accuracy. As for the CIFAR-100 dataset, the ResNet-34 scores a
79.8% accuracy, a result close to 80%. In addition to the ResNet family
architecture, we show that our framework can be applied to other networks like
VGG-16. We run SNN with only 1 time step in order to compare the state-of-the-
art Temporal Pruning [47]. We list the results in Table 2, where we can find
our method outstrips the other SNN works tested with VGG-16. Our method
achieves a much higher improvement on the CIFAR-100 dataset (4% compared to
[47]).

Table 3: Comparison between our algorithm and existing algorithms on Tiny-
ImageNet datasets. Our method improves network performance across all tasks.

Dataset | Method | Architecture | Time Steps | Accuracy
---|---|---|---|---
Tiny-ImageNet | AGC [51] | VGG-16 | 150 | 51.92
DCT-SNN [52] | VGG-13 | 125 | 52.43
Joint A-SNN (Ours) | VGG-16 | 4 | 55.39
2 | 53.91

Next, we conduct experiments on the Tiny-ImageNet dataset in Table 3, which is
a more complex dataset than CIFAR, to verify our method. There aren’t many
baselines on this dataset, yet our method still achieves higher accuracy with
4 or even 2 time steps. This shows the ability of our method to handle the
large-scale dataset.

Table 4: Ablation results for different design choices.  Design Choices |  | Accuracy | Time
---|---|---|---
w./ ANN | CE Loss | KLD Loss | Norm Loss | WFT |
| ✓ |  |  |  |  | 92.39 | 26.7s
✓ |  | ✓ |  |  |  | 93.12 | 31.6s
✓ | ✓ | ✓ |  |  |  | 94.09 | 34.8s
✓ | ✓ | ✓ | ✓ |  |  | 94.52 | 34.9s
✓ |  |  |  | ✓ |  | 93.69 | 32.2s
✓ | ✓ | ✓ | ✓ | ✓ |  | 95.45 | 36.7s

###  5.3 Ablation Studies

In this section, we conduct ablation studies on the design choice of our joint
training frameworks. Here, we have several baselines: (1) Independent trained
SNN without any distillation loss function; (2) trained SNN with vanilla KLD
loss from trained ANN (similar to Knowledge Distillation [24]); (3) jointly
trained ANN and SNN with different choices of loss functions, including CE
Loss, KLD Loss, Norm Loss; (4) jointly trained ANN and SNN with weight-
factorized training yet without distillation. Here, we summarize these cases
together with final test accuracy and training time cost for 1 epoch running.
Note that here we use ResNet-18 for the CIFAR-10 dataset. The results are put
in Table 4, from which we can see that the standard training of SNN is 92.4%,
a similar accuracy level to existing works. Because standard training does not
use the knowledge from ANN, the training is the fastest among other
approaches, amounting to 26.7 seconds per training epoch. If we choose to use
Knowledge Distillation [24] (the second row), the performance would boost to
93.1%, which is a moderate improvement (0.7%). It also leads to a higher
training time (increased by 5 seconds). Using the joint training with CE and
KLD loss, we get a 1.7% accuracy improvement. Moreover, with the L2 norm loss,
one even further gets another performance lift, amounting to 94.5% final
accuracy. These two choices also increase the a bit training time but get
significantly higher accuracy. Finally, we verify the WFT mechanism, which
obtains 93.7% accuracy. This result proves that near-ANN parameters may help
the optimization with SNN. Together, these parts all contribute to the final
accuracy, resulting in 95.45% accuracy.

We also compare the ANN performance here. Since WFT will also affect the
training convergence of ANN, here we report how ANN’s accuracy changes after
using WFT. Our baseline result is 95.61%. Using the CE loss, we add multiple
branches to ANNs, which also helps the optimization, and get 95.98% accuracy.
Finally, using WFT the accuracy of ANN is slightly affected (95.74%).

###  5.4 Energy Efficiency

Table 5: Energy estimation of ANN and SNNs of computation. Method | Model | Acc. | #Add. | #Mult. | Energy
---|---|---|---|---|---
ANN (ours) | Res-18 | 95.74 | 187M | 187M | 1.03$J$
SNN [35](T=5) | Res-20 | 92.70 | 142M | 8.80M | 168$mJ$
SNN ([33], T=2) | Res-18 | 93.13 | 71.2M | 3.52M | 80.3$mJ$
SNN (ours, T=2) | Res-18 | 94.01 | 73.4M | 3.52M | 82.2$mJ$

In this section, we measure the hardware efficiency of our proposed framework
during inference. To measure the inference cost, we measure the energy cost of
both computation and data movement. It is worthwhile to highlight that in some
neuromorphic hardware, the data movement cost could be negligible due to the
in-memory computing design [5]. Yet in the digital circuit, the data accessing
energy cannot be ignored. Hence, we discuss the energy cost of computation and
data movement separately in this section.

Computational Energy Cost. The dot-product in ANN is equal to performing a
multiply-accumulate (MAC) operation, composed of an addition and a
multiplication. As for SNN, the binary nature of spike-based computation can
eliminate the multiplication for all layers except for the first layer.
Moreover, if the spike is not fired, the hardware can even avoid this addition
with sparse computation. Based on this, we estimate the energy consumption
following [35, 53], a technique using 45nm CMOS technology. The MAC operation
in ANN costs 4.6$pJ$ energy and the accumulation in SNN costs 0.9$pJ$ energy.
Based on this, we compute the energy cost and put it in Table 5. Our network
only costs 82.2$mJ$ for a single forward and consumes 12.5$\times$ lower
energy compared with ANN. Compared to existing [33], our model consumes
slightly higher energy, amounting to 1.9$mJ$. Nevertheless, our sacrifices
only trivial energy for a large accuracy improvement over [33] (0.9% accuracy
uplift).

Data Movement Energy Cost. We also estimate the energy cost of data movement
in the case of digital circuit deployment. We follow that moving 64 bits of
data from the cache costs $10-100pJ$ using 45nm CMOS technology too [53]. For
ANN, we calculate the total data moved as two times of activation sizes from
all layers, based on that every feature map will be moved from the
computational unit to the cache after the computation of the previous layer
and moved from the cache to the computational unit again for the computation
of the present layer. For SNN, we multiply the activation sizes by $T$ times
due to the movement of membrane potential, as well as the output spikes
number. Note that one spike is counted as 1 bit while one activation or one
membrane potential is counted as 32 bits. We put the summary of data movement
cost in Table 6 where minimum energy is calculated by 10$pJ$ per 64-bit
movement and maximum energy is calculated by 100$pJ$ per 64-bit movement. We
can find that SNN indeed costs a slightly higher data movement energy than
ANN. However, combing both data movement energy and computational energy still
exhibits better energy efficiency in SNNs.

Table 6: Energy estimation of ANNs and SNNs of data movement. Method | Model | Acc. | Volume | Min. Energy | Max. Energy
---|---|---|---|---|---
ANN (ours) | Res-18 | 95.74 | 6.234 MB | 1.022$mJ$ | 10.22$mJ$
SNN (ours, T=2) | Res-18 | 94.01 | 12.506 MB | 2.062$mJ$ | 20.62$mJ$

##  6 Conclusion

In this paper, we have introduced the joint-training framework of ANN and SNN.
Our framework consists of two core ingredients, the first is self-distillation
from multiple branches, and the second is weight-factorized training assisted
by the Singular Value Decomposition. Both of them aim to provide auxiliary
information during the training of SNNs to circumvent the problem of gradient
vanishing/exploding. Our method can be viewed as a special case of “hybrid
learning”, _i.e_., using ANN for SNN training. We have demonstrated that our
method brings a large improvement over the original baseline and the existing
state of the arts on several widely used network structures. We hope this
framework will enable more advanced neuromorphic intelligence. However, since
ANNs are experts in spatial feature processing but poor at temporal ones,
depending on the guide of ANNs, our framework will limit the spatial-temporal
processing ability of SNNs to some extent. For future work, we will further
study the combination manner of ANNs and SNNs to optimize the underlying code
and retain the spatial-temporal processing ability of SNNs.

## Acknowledgment

This work is supported by grants from the National Natural Science Foundation
of China under contracts No.12202412 and No.12202413.

## References

  * [1] W. Maass, Networks of spiking neurons: the third generation of neural network models, Neural networks 10 (9) (1997) 1659–1671.
  * [2] K. Roy, A. Jaiswal, P. Panda, Towards spike-based machine intelligence with neuromorphic computing, Nature 575 (7784) (2019) 607–617.
  * [3] P. Panda, S. A. Aketi, K. Roy, Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization, Frontiers in Neuroscience 14 (2020) 653.
  * [4] F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G.-J. Nam, et al., Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip, IEEE transactions on computer-aided design of integrated circuits and systems 34 (10) (2015) 1537–1557.
  * [5] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, et al., Loihi: A neuromorphic manycore processor with on-chip learning, Ieee Micro 38 (1) (2018) 82–99.
  * [6] G.-q. Bi, M.-m. Poo, Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type, Journal of neuroscience 18 (24) (1998) 10464–10472.
  * [7] M. Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, S. J. Thorpe, T. Masquelier, Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks, Pattern recognition 94 (2019) 87–95.
  * [8] P. Falez, P. Tirilly, I. M. Bilasco, P. Devienne, P. Boulet, Unsupervised visual feature learning with spike-timing-dependent plasticity: How far are we from traditional feature learning approaches?, Pattern Recognition 93 (2019).
  * [9] D. O. Hebb, The organization of behavior: A neuropsychological theory, Psychology Press, 2005.
  * [10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: CVPR, 2009.
  * [11] S. Deng, S. Gu, Optimal conversion of conventional artificial neural networks to spiking neural networks, in: International Conference on Learning Representations.
  * [12] Y. Li, S. Deng, X. Dong, R. Gong, S. Gu, A free lunch from ann: Towards efficient, accurate spiking neural networks calibration, in: International Conference on Machine Learning, PMLR, 2021, pp. 6316–6325.
  * [13] B. Rueckauer, I.-A. Lungu, Y. Hu, M. Pfeiffer, S.-C. Liu, Conversion of continuous-valued deep networks to efficient event-driven networks for image classification, Frontiers in neuroscience 11 (2017) 682.
  * [14] B. Han, G. Srinivasan, K. Roy, Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13558–13567.
  * [15] Y. Wu, L. Deng, G. Li, J. Zhu, L. Shi, Spatio-temporal backpropagation for training high-performance spiking neural networks, Frontiers in neuroscience 12 (2018) 331.
  * [16] Y. Wu, L. Deng, G. Li, J. Zhu, Y. Xie, L. Shi, Direct training for spiking neural networks: Faster, larger, better, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 1311–1318.
  * [17] L. Wang, L. Hu, J. Gu, Z. Hu, Y. Wu, K. He, J. Hopcroft, Towards understanding learning representations: To what extent do different neural networks learn the same representation, Advances in neural information processing systems 31 (2018).
  * [18] T. Bu, W. Fang, J. Ding, P. DAI, Z. Yu, T. Huang, Optimal ANN-SNN conversion for high-accuracy and ultra-low-latency spiking neural networks, in: International Conference on Learning Representations, 2022.
URL https://openreview.net/forum?id=7B3IJMM1k_M

  * [19] H. Zheng, Y. Wu, L. Deng, Y. Hu, G. Li, Going deeper with directly-trained larger spiking neural networks, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 11062–11070.
  * [20] Y. Guo, Y. Chen, L. Zhang, Y. Wang, X. Liu, X. Tong, Y. Ou, X. Huang, Z. Ma, Reducing information loss for spiking neural networks, in: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XI, Springer, 2022, pp. 36–52.
  * [21] Y. Guo, Y. Chen, L. Zhang, X. Liu, Y. Wang, X. Huang, Z. Ma, IM-loss: Information maximization loss for spiking neural networks, in: A. H. Oh, A. Agarwal, D. Belgrave, K. Cho (Eds.), Advances in Neural Information Processing Systems, 2022.
URL https://openreview.net/forum?id=Jw34v_84m2b

  * [22] W. Fang, Z. Yu, Y. Chen, T. Masquelier, T. Huang, Y. Tian, Incorporating learnable membrane time constant to enhance learning of spiking neural networks, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2661–2671.
  * [23] Y. Kim, P. Panda, Revisiting batch normalization for training low-latency deep spiking neural networks from scratch, Frontiers in neuroscience 1638.
  * [24] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, stat 1050 (2015) 9.
  * [25] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, Y. Bengio, Fitnets: Hints for thin deep nets, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv.org/abs/1412.6550

  * [26] S. Wang, Z. Chen, Z. Ren, H. Liang, Q. Yan, P. Ren, Paying more attention to self-attention: Improving pre-trained language models via attention guiding (2022).
  * [27] J. Yim, D. Joo, J. Bae, J. Kim, A gift from knowledge distillation: Fast optimization, network minimization and transfer learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [28] P. Zhao, L. Xie, J. Wang, Y. Zhang, Q. Tian, Progressive privileged knowledge distillation for online action detection, Pattern Recognition 129 (2022) 108741\.
  * [29] Z. R. Wang, J. Du, Joint architecture and knowledge distillation in cnn for chinese text recognition, Pattern Recognition 111 (2021) 107722.
  * [30] S. Zhang, C. Tang, C. Guan, Visual-to-eeg cross-modal knowledge distillation for continuous emotion recognition, Pattern Recognition: The Journal of the Pattern Recognition Society (2022) 130.
  * [31] R. Tang, Z. Liu, Y. Li, Y. Song, H. Liu, Q. Wang, J. Shao, G. Duan, J. Tan, Task-balanced distillation for object detection, Pattern Recognition (2023) 109320\.
  * [32] R. Kushawaha, S. Kumar, B. Banerjee, R. Velmurugan, Distilling spikes: Knowledge distillation in spiking neural networks (05 2020).
  * [33] Y. Li, Y. Guo, S. Zhang, S. Deng, Y. Hai, S. Gu, Differentiable spike: Rethinking gradient-descent for training spiking neural networks, Advances in Neural Information Processing Systems 34 (2021).
  * [34] M. A. Nahmias, B. J. Shastri, A. N. Tait, P. R. Prucnal, A leaky integrate-and-fire laser neuron for ultrafast cognitive computing, IEEE journal of selected topics in quantum electronics 19 (5) (2013) 1–12.
  * [35] N. Rathi, K. Roy, Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks, arXiv preprint arXiv:2008.03658 (2020).
  * [36] Y. Bengio, N. Léonard, A. Courville, Estimating or propagating gradients through stochastic neurons for conditional computation, arXiv preprint arXiv:1308.3432 (2013).
  * [37] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: CVPR, 2016.
  * [38] S. Teerapittayanon, B. McDanel, H.-T. Kung, Branchynet: Fast inference via early exiting from deep neural networks, in: 2016 23rd International Conference on Pattern Recognition (ICPR), IEEE, 2016, pp. 2464–2469.
  * [39] P. U. Diehl, G. Zarrella, A. Cassidy, B. U. Pedroni, E. Neftci, Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware, in: 2016 IEEE International Conference on Rebooting Computing (ICRC), IEEE, 2016, pp. 1–8.
  * [40] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: ICLR, 2015.
  * [41] N. Rathi, G. Srinivasan, P. Panda, K. Roy, Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation, in: International Conference on Learning Representations, 2020.
  * [42] Q. Meng, S. Yan, M. Xiao, Y. Wang, Z. Lin, Z.-Q. Luo, Training much deeper spiking neural networks with a small number of time-steps, Neural Networks 153 (2022) 254–268.
  * [43] W. Zhang, P. Li, Temporal spike sequence learning via backpropagation for deep spiking neural networks, Advances in Neural Information Processing Systems 33 (2020) 12022–12033.
  * [44] S. Deng, Y. Li, S. Zhang, S. Gu, Temporal efficient training of spiking neural network via gradient re-weighting, in: International Conference on Learning Representations.
  * [45] Y. Guo, X. Tong, Y. Chen, L. Zhang, X. Liu, Z. Ma, X. Huang, Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, IEEE, 2022, pp. 326–335.
  * [46] Y. Guo, L. Zhang, Y. Chen, X. Tong, X. Liu, Y. Wang, X. Huang, Z. Ma, Real spike: Learning real-valued spikes for spiking neural networks, in: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII, Springer, 2022, pp. 52–68.
  * [47] S. S. Chowdhury, N. Rathi, K. Roy, Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning, in: European Conference on Computer Vision, Springer, 2022, pp. 709–726.
  * [48] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [49] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, in: International Conference on Learning Representations, 2016.
  * [50] A. Krizhevsky, V. Nair, G. Hinton, Cifar-10 (canadian institute for advanced research).
URL http://www.cs.toronto.edu/~kriz/cifar.html

  * [51] S. Kundu, G. Datta, M. Pedram, P. A. Beerel, Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 3953–3962.
  * [52] I. Garg, S. S. Chowdhury, K. Roy, Dct-snn: Using dct to distribute spatial information over time for low-latency spiking neural networks, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4671–4680.
  * [53] M. Horowitz, 1.1 computing’s energy problem (and what we can do about it), in: 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), IEEE, 2014, pp. 10–14.