(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: School of Electronic Information and Electrical Engineering,
Shanghai Jiao Tong University, Shanghai, China 22institutetext: Huawei
Technologies, Shenzhen, China
22email<EMAIL_ADDRESS>https://github.com/Intelligent-Computing-Research-Group/BKDSNN

# BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks
Training with Blurred Knowledge Distillation

Zekai Xu 11 Kang You 11 Qinghai Guo 22 Xiang Wang 22 Zhezhi He Corresponding
Author11 0000-0002-6357-236X

###### Abstract

Spiking neural networks (SNNs), which mimic biological neural systems to
convey information via discrete spikes, are well-known as brain-inspired
models with excellent computing efficiency. By utilizing the surrogate
gradient estimation for discrete spikes, learning-based SNN training methods
that can achieve ultra-low inference latency (_i.e_., number of time-step)
have emerged recently. Nevertheless, due to the difficulty of deriving precise
gradient for discrete spikes in learning-based methods, a distinct accuracy
gap persists between SNNs and their artificial neural networks (ANNs)
counterparts. To address the aforementioned issue, we propose a _blurred
knowledge distillation (BKD)_ technique, which leverages randomly blurred SNN
features to restore and imitate the ANN features. Note that, our BKD is
applied upon the feature map right before the last layer of SNNs, which can
also mix with prior logits-based knowledge distillation for maximal accuracy
boost. In the category of learning-based methods, our work achieves state-of-
the-art performance for training SNNs on both static and neuromorphic
datasets. On the ImageNet dataset, BKDSNN outperforms prior best results by
4.51% and 0.93% with the network topology of CNN and Transformer,
respectively.

###### Keywords:

Learning-based Spiking Neural Networks Knowledge Distillation

##  1 Introduction

Inspired by the neural firing mechanism in the biological nervous system,
spiking neural network (SNN) [37] emerges as an alternative framework for the
traditional artificial neural network (ANN). ANNs rely on continuous
activation values to convey information between neurons [31], while SNNs
utilize discrete spiking events to process and propagate information [6, 38].
Given the remarkable energy efficiency achieved by such event-driven
information propagation [18], SNNs show promising prospects on computational
intelligence tasks [44] with strong autonomous learning capabilities and
ultra-low power consumption [2, 11, 41, 58]. Apart from these advantages, SNNs
are also natively compatible with event-based neuromorphic datasets (_e.g_.,
CIFAR10-DVS) [14, 65, 61, 62].

Currently, conversion-based and learning-based are two common methods to train
SNNs. Conversion-based methods attempt to leverage information from ANNs by
transferring the parameters of a pre-trained ANN to its SNN counterpart, which
have achieved promising results in several tasks [2, 8, 33, 34]. However,
event-triggered neurons convey limited information with ultra-low inference
time-steps (_e.g_., 4 time-steps), leading to degraded performance of
conversion-based methods [14, 65]. Consequently, conversion-based methods
usually take a long inference time (_e.g_., 32 time-steps) to get comparable
accuracy as the original ANN [3, 47] based on the equivalence of the ReLU
activation and the firing rate of integrate-and-fire (IF) neuron [16]. In
order to further improve the performance of SNNs with ultra-low inference
time-steps, learning-based methods unfold SNN training over the inference
time-steps and leverage back-propagation through time (known as BPTT) [59, 52]
to train SNNs. Inspired by [27], several works [35, 14, 62] adopt surrogate
gradient estimation as an implicit differentiation for direct training of
SNNs. With the combination of BPTT and surrogate gradient estimation,
learning-based methods exhibit advantages in processing event-based
neuromorphic datasets (_e.g_., CIFAR10-DVS [22]) [14, 65, 61, 62].
Furthermore, direct training of CNN- [23, 14] and Transformer-based [65, 61,
62] SNNs also achieves promising performance with ultra-low time-steps on
static datasets (_e.g_., CIFAR10 [28], CIFAR100 [28] and ImageNet [7]).

Despite that numerous efforts have been made in learning-based methods, SNNs
still need improvement compared with ANNs. [62, 53] claim that imprecise
surrogate gradient estimation of spiking neurons leads to performance
degradation. On the other hand, KDSNN [53] and LaSNN [21] introduce layer-wise
knowledge distillation (KD) to train a student SNN with rich information from
the ANN teacher, achieving promising performance on several classical vision
datasets (MNIST, CIFAR10/100) with CNN-based models. However, according to our
preliminary experimental results, both KDSNN and LaSNN struggle with more
complex datasets (ImageNet) and Transformer-based models. We hypothesize that
the direct distillation between discrete SNN features and continuous ANN
features [53, 21] might be the source of the limitations.

In this paper, to further alleviate performance degradation of learning-based
SNNs, we introduce a blurred knowledge distillation (BKD) technique to
leverage randomly blurred SNN features to restore and mimic ANN features. We
apply BKD to the intermediate feature before the last layer of the student SNN
and prove its superiority over layer-wise knowledge distillation in learning-
based SNNs. Furthermore, we combine prior logits-based distillation with BKD,
which provides mutual benefits for improving performance. We evaluate BKDSNN
on both static datasets (_i.e_., CIFAR10, CIFAR100, and ImageNet) and a
neuromorphic dataset (_i.e_., CIFAR10-DVS), where we achieve state-of-the-art
(SOTA) performance with both CNN- and Transformer-based SNN models.

To summarize, our contributions are threefold:

  * $\bullet$

We introduce a blurred knowledge distillation (BKD) technique to guide the
training of student SNNs through feature imitation and demonstrate the
advantages of BKD on learning-based SNNs.

  * $\bullet$

We apply BKD to the intermediate feature before the last layer and prove its
superiority over previous layer-wise distillation. Combined with prior logits-
based knowledge distillation, our BKDSNN maximizes accuracy boost on learning-
based SNNs.

  * $\bullet$

We benchmark our method on both static datasets (_i.e_., CIFAR10, CIFAR100 and
ImageNet) and a neuromorphic dataset (_i.e_., CIFAR10-DVS). Our method
outperforms prior SOTA learning-based methods for both CNN- and Transformer-
based models.

##  2 Background and Related Work

#### Spiking Neuron Model.

Previous works [3, 8, 14, 65, 61, 62, 2, 24] have widely used the classic
integrate-and-fire (IF) neuron as the fundamental unit in SNNs. IF neuron
integrates input currents through changes in its membrane potential, which is
compared against the threshold for spike generation. The membrane potential is
reset after emitting a spike. The dynamics of the IF neuron is depicted as
follows:

$\begin{gathered}H[t]=V[t-1]+\frac{1}{\tau}\left(I[t]-\left(V[t-1]-V_{\text{reset}}\right)\right)\\\
S[t]=\Theta\left(H[t]-V_{\text{th}}\right)=\begin{cases}1,&H[t]\geq
V_{\text{th }}\\\ 0,&H[t]<V_{\text{th }}\end{cases}\\\
V[t]=H[t](1-S[t])+V_{\text{reset }}S[t]\end{gathered}$ (1)

where $\tau$ is the membrane time constant, $I[t]$ is the receiving current at
time step $t$ and $\Theta\left(\cdot\right)$ is the heaviside function. The
neuron accumulates $I[t]$ to change the membrane potential $H[t]$ at time step
$t$. Once $H[t]$ exceeds the firing threshold $V_{\text{th}}$, the neuron
generates a spike and resets the membrane potential to $V_{\text{reset}}$.

#### Conversion-based Methods.

To fully make use of the knowledge from pre-trained ANN, conversion-based
methods are designed to transfer the learned parameters of ANNs into
corresponding SNNs, without further training of SNNs. Previous works [17, 18]
replace ReLU activations in ANNs with spiking neurons and add scaling
operations like weight normalization and threshold balancing [10] to generate
corresponding SNNs. Although several works [8, 17, 18, 34] have achieved
nearly loss-less accuracy with VGG-16 and ResNet, the converted SNNs still
require long inference time to match the original ANNs in precision. [45]
claims that low firing rates caused by weight normalization result in
increasing latency for the information reaching higher layers, which limits
the practical application of the converted SNNs.

(a) Architecture of BKDSNN. (b) BKD block.

Figure 1: Overview of training SNN with blurred knowledge distillation (BKD).
The SNN is directly trained with back-propagation through time (BPTT) [52],
where we utilize a blurred variant of KD (_i.e_., BKD) to achieve higher
accuracy. BKD highlighted in yellow shaded region differs from prior SNN-
oriented KD in three perspectives: 1) A blurred matrix ${\bm{B}}$ is _randomly
sampled on the fly_ (per input image) to mask out the feature of student SNN;
2) A restoration block $\mathcal{G}$ consisting of two convolutional layers
connected by a ReLU layer is applied on _blurred SNN features_ to _restore and
mimic ANN features_ ; 3) Such blurred knowledge distillation is applied only
to the _intermediate features before the last layer_.

#### Learning-based Methods.

With the development of surrogate gradient estimation [39], learning-based
methods leverage continuous-value-based loss functions to handle
nondifferentiable difficulties in SNN training. Ref. [39] introduces surrogate
gradient estimation to conduct SNN training on the event-based DVS gesture
dataset, which achieves better accuracy after a few training iterations. In an
effort to improve computational resources in SNNs, joint CNN-SNN [54] puts
forward a novel augmented spiking-based framework and achieves promising
results in several image classification tasks. DCT-SNN [15] reduces the number
of inference time-steps by combining frequency-domain learning with surrogate
gradient descent. Spiking ResNet [60] uses the directly-trained deep SNN to
achieve high performance on ImageNet. SEW ResNet [14] proposes the spike-
element-wise block to counter gradient vanishing and gradient exploding, which
successfully extends the depth of directly-trained SNN to more than 100
layers. When it comes to Transformer-based SNN, Spikformer [65] with a novel
spiking self-attention block achieves significant improvement in image
classification on both neuromorphic and static datasets. Transformer-based SNN
is further improved by successive Spikingformer [61], ConvBN-MaxPooling-LIF
(CML) [62] and Meta-SpikeFormer [57], achieving SOTA performance.

#### Knowledge Distillation for SNN Training.

Knowledge distillation (KD) is a classical transfer learning method that has
been proven to be effective on various tasks, such as model compression [20,
49, 36, 56], incremental learning [42, 5, 4, 55], _etc_. Recently, several
works [32, 50, 30] extend KD for SNN training, such as distilling spikes from
a large SNN to a small SNN or logits-based knowledge from a well-trained ANN
to a simple SNN. KDSNN [53] proposes a novel framework embracing both logits-
based and feature-based knowledge distillation, which enhances performance on
several image classification datasets (MNIST, CIFAR10, and CIFAR100). LaSNN
[21] puts forward a layer-wise feature-based ANN-to-SNN distillation
framework, achieving comparable top-1 accuracy to ANNs on the challenging Tiny
ImageNet dataset. Furthermore, KDSNN [53] and LaSNN [21] are applicable to
both homogeneous and heterogeneous networks. Despite the superiority of KDSNN
[53] and LaSNN [21], they struggle with more complex datasets (_e.g_.,
ImageNet) with CNN-based and Transformer-based models according to our
preliminary experimental results in Sec. 4.2.

##  3 Method

In this paper, we propose a novel blurred knowledge distillation SNN (overall
architecture of BKDSNN is in Fig. 1(a)) training method that leverages
knowledge from the teacher ANN to guide the training of the student SNN. As
depicted in Fig. 1(b), our BKD first introduces a randomly blurred matrix
${\bm{B}}$ to mask the student SNN features. Unlike KDSNN [53] and LaSNN [21],
which only apply an adaptive layer to align the SNN feature with the ANN
feature before distillation, BKD introduces a restoration block $\mathcal{G}$
on blurred SNN feature to restore and mimic ANN feature. The proposed BKD is
applied only to the intermediate feature before the last layer, different from
the layer-wise distillation [53, 21]. As shown in Fig. 1(a), we combine the
logits-based distillation with BKD to leverage both logits and feature
knowledge from the teacher ANN.

Table 1: Summary of notations in this paper

Notation | Description
---|---
$T$ |  | time-step
---
$\lambda$ |  | blurred ratio
---
${\bm{R}}\in\mathbb{R}^{b\times c^{\text{tea}}}$ |  | random matrix ${\bm{R}}\sim U\left(0,1\right)$
---
${\bm{B}}\in\mathbb{R}^{b\times c^{\text{tea}}}$ |  | blurred matrix
---
$\hat{{\bm{y}}}\in\mathbb{R}^{b\times n^{\text{cls}}}$ |  | ground truth label
---
${\bm{y}}^{\text{stu}},{\bm{y}}^{\text{tea}}\in\mathbb{R}^{b\times n^{\text{cls}}}$ |  | output logits of student SNN and teacher ANN
---
${\bm{q}}^{\text{stu}},{\bm{q}}^{\text{tea}}\in\mathbb{R}^{b\times n^{\text{cls}}}$ |  |  ${\bm{y}}^{\text{stu}}$ and ${\bm{y}}^{\text{tea}}$ after flattened processing
---
${\bm{\mathsfit{F}}}^{\text{tea}}\in\mathbb{R}^{b\times c^{\text{tea}}\times h\times w}$ |  | feature before the last layer of teacher ANN
---
${\bm{\mathsfit{F}}}^{\text{stu}}\in\mathbb{R}^{T\times b\times c^{\text{stu}}\times h\times w}$ |  | feature before the last layer of student SNN
---
$\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}\in\mathbb{R}^{b\times c^{\text{stu}}\times h\times w}$ |  | average of ${\bm{\mathsfit{F}}}^{\text{stu}}$ at time-step dimension ($T$)
---
$f_{\text{adp}}$ |  | adaptive layer to align ${\bm{\mathsfit{F}}}^{\text{stu}}$ with ${\bm{\mathsfit{F}}}^{\text{tea}}$
---
$\mathcal{G}$ |  | restoration block for ${\bm{\mathsfit{F}}}^{\text{stu}}$
---
$\hat{{\bm{\mathsfit{F}}}}^{\text{stu}}\in\mathbb{R}^{b\times c^{\text{tea}}\times h\times w}$ |  | blurred restoration feature from ${\bm{\mathsfit{F}}}^{\text{stu}}$
---

#### Blurred Knowledge Distillation.

To facilitate understanding, we present important notations in Tab. 1. The
proposed blurred knowledge distillation is illustrated in Fig. 1(b). We first
introduce a random matrix ${\bm{R}}$ which obeys uniform distribution between
0 and 1. Then we apply a blurred ratio $\lambda$ to ${\bm{R}}$ to generate a
blurred matrix ${\bm{B}}$ through Eq. 2.

${\bm{B}}_{i,j}=\begin{cases}0,&\text{ if }{\bm{R}}_{i,j}<\lambda\\\ 1,&\text{
Otherwise }\end{cases}$ (2)

For feature ${\bm{\mathsfit{F}}}^{\text{stu}}$ at the last layer of student
SNN, we first average ${\bm{\mathsfit{F}}}^{\text{stu}}$ at time-step
dimension ($T$) as:

$\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}=\frac{1}{T}\displaystyle\sum_{i=1}^{T}{\bm{\mathsfit{F}}}^{\text{stu}}_{i}$
(3)

After the averaging operation above, we apply a convolutional adaptive layer
$f_{\text{adp}}$ to align $\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}$ with
${\bm{\mathsfit{F}}}^{\text{tea}}_{i}$ when student feature dimension
$c^{\text{stu}}$ mismatches with teacher feature dimension $c^{\text{tea}}$.
After the alignment, we construct a restoration block $\mathcal{G}$ with two
convolutional layers connected by a ReLU layer and apply it to
$\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}$ with the blurred matrix ${\bm{B}}$
to achieve blurred restoration feature
$\hat{{\bm{\mathsfit{F}}}}^{\text{stu}}$ as illustrated in Eq. 4.

$\hat{{\bm{\mathsfit{F}}}}^{\text{stu}}=\begin{cases}\mathcal{G}\left(f_{\text{adp}}(\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}})*{\bm{B}}\right),&c^{\text{stu}}\neq
c^{\text{tea}}\\\
\mathcal{G}\left(\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}*{\bm{B}}\right),&\text{otherwise}\end{cases}$
(4)

We calculate blurred knowledge distillation loss $\mathcal{L}_{\text{BKD}}$
using quadratic norm ($L^{2}$) as follows.

$\mathcal{L}_{\text{BKD}}=\displaystyle\sum_{i}^{b}\sum_{j}^{c^{\text{stu}}}\sum_{k}^{h}\sum_{l}^{w}\left(\hat{{\bm{\mathsfit{F}}}}_{i,j,k,l}^{\text{stu}}-{\bm{\mathsfit{F}}}_{i,j,k,l}^{\text{tea}}\right)^{2}$
(5)

We then combine blurred knowledge distillation loss $\mathcal{L}_{\text{BKD}}$
with the original cross entropy loss $\mathcal{L}_{\text{CE}}$ to get the
total loss $\mathcal{L}_{\text{total}}$ as follows.

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{CE}}({\bm{y}}^{\text{stu}},\hat{{\bm{y}}})+\mathcal{L}_{\text{BKD}}$
(6)

#### Effectiveness of BKD.

(a) Feature visualization of ResNet-18. (b) Feature distribution of SNN and
ANN.

Figure 2: (a) Comparison of feature visualization in ResNet-18 under various
methods; (b) Histogram of SNN and ANN feature.

For both CNN- and Transformer-based models, features of deeper layers have
better representation for the original input image [26, 43, 63]. In other
words, features of deeper layers contain sufficient information of adjacent
pixels (for CNN-based models) [26] or patch pairs (for Transformer-based
models) [43, 63]. Therefore, we can use partial student SNN feature before the
last layer to restore and mimic teacher ANN feature through a restoration
block $\mathcal{G}$, which can help the student SNN achieves better
representation and effectively mimics teacher ANN. We then apply Grad-CAM
which can intuitively demonstrate the feature similarity [46] on each block in
ResNet-18 under various methods. As depicted in Fig. 2(a), compared with
previous SNN (SEW ResNet-18 [14]) and KDSNN [53], our BKDSNN transfers
sufficient knowledge from ANN and effectively mimics ANN feature. Meanwhile,
as illustrated in Fig. 2(b), student SNN feature
$\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}$ is discrete while teacher ANN
feature ${\bm{\mathsfit{F}}}^{\text{tea}}$ is continuous. Therefore, directly
applying distillation to encourage discrete SNN feature
$\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}$ to imitate continuous ANN feature
${\bm{\mathsfit{F}}}^{\text{tea}}$ [53, 21] is not plausible. However, the
proposed BKD leverages the restoration block $\mathcal{G}$ to map discrete
feature $\tilde{{\bm{\mathsfit{F}}}}^{\text{stu}}$ to continuous feature
$\hat{{\bm{\mathsfit{F}}}}^{\text{stu}}$ for imitating continuous ANN feature
${\bm{\mathsfit{F}}}^{\text{tea}}$, which is a more proper spiking-friendly
feature-based distillation method.

#### Mixed Distillation.

As depicted in Fig. 1(a), BKDSNN further applies logits-based distillation
which transfers knowledge from teacher ANN logits to student SNN logits. As
soft logits from the teacher model embrace sufficient hidden knowledge, which
serves as a good regularization to student model [64], we introduce the
temperature $\tau$ to make the logits distribution flatter. The output logits
of both teacher and student are processed as follows:

${\bm{q}}_{i}=\frac{\exp\left({\bm{y}}_{i}/\tau\right)}{\textstyle\sum_{j=1}^{n^{\text{cls}}}\exp\left({\bm{y}}_{j}/\tau\right)},\quad
i\in\\{1,2,\dots,n^{\text{cls}}\\}$ (7)

Then logits-based distillation loss $\mathcal{L}_{\text{LD}}$ is defined as:

$\mathcal{L}_{\text{LD}}=\tau^{2}\cdot\mathcal{L}_{\text{ CrossEntropy
}}\left({\bm{q}}^{\text{stu}},{\bm{q}}^{\text{tea}}\right)$ (8)

We combine $\mathcal{L}_{\text{LD}}$ with $\mathcal{L}_{\text{BKD}}$ to
achieve mixed distillation loss $\mathcal{L}_{\text{MD}}$ (Eq. 9), which
contains both logits and feature information from the teacher ANN.

$\mathcal{L}_{\text{MD}}=w_{\text{LD}}\cdot\mathcal{L}_{\text{LD}}+w_{\text{BKD}}\cdot\mathcal{L}_{\text{BKD}}$
(9)

where $w_{\text{LD}}$ and $w_{\text{BKD}}$ are weights for
$\mathcal{L}_{\text{LD}}$ and $\mathcal{L}_{\text{BKD}}$, respectively. Thus,
the overall loss $\mathcal{L}$ is defined as follows:

$\mathcal{L}=\mathcal{L}_{\text{ CrossEntropy
}}\left({\bm{y}}^{\text{stu}},\hat{{\bm{y}}}\right)+\mathcal{L}_{\text{MD}}=\mathcal{L}_{\text{
task}}+\mathcal{L}_{\text{MD}}$ (10)

##  4 Experiments

###  4.1 Experimental Configuration

#### Platform.

All experiments are evaluated on a server equipped with 32 cores Intel Xeon
Platinum 8352V CPU with 2.10GHz and 8-way NVIDIA GPUs. We use SpikingJelly
[13] to simulate IF neurons.

#### Static Datasets and Settings.

We first evaluate BKDSNN on three static datasets, _i.e_., CIFAR10/100 and
ImageNet.

CIFAR10/CIFAR100 [28] contains 50k training and 10k testing images with
32$\times$32 resolution, and is labeled with 10- or 100-categories
respectively for classification. For CNN-based models, we use ResNet-19 from
TET [9] and lightweight ResNet-20 from SEW ResNet [14] with corresponding ANNs
as teacher models. For Transformer-based models, we train ViT-S/16 on CIFAR10
and CIFAR100 to make it serve as the teacher ANN model for Sformer-4-256,
Sformer-2-384, and Sformer-4-384 (We abbreviate Spikingformer to Sformer for
convenience). Sformer-L-D represents a Sformer model with L Sformer encoder
blocks and D feature embedding dimensions.

ImageNet-1K [7] contains 1.3 millions of training images and 50k validation
images with 224$\times$224 resolution, labeled with 1k classes. We choose
ResNet-18, ResNet-34, and ResNet-50 pre-trained on ImageNet-1K as teacher ANN
models for the corresponding SEW ResNet models [14]. For Transformer-based
models, we introduce ViT-B/16 [12] pre-trained on ImageNet1K as the teacher
ANN model for Sformer-8-384, Sformer-8-512 and Sformer-8-768. We also apply
our BKDSNN on the latest Meta-SpikeFormer [57] and conduct experiments with
Mformer-8-256, Mformer-8-384 and Mformer-8-512 on ImageNet-1K (We abbreviate
Meta-SpikeFormer to Mformer for convenience.).

For both CNN- and Transformer-based models, we set temperature $\tau$ to 2.0,
blurred ratio $\lambda$ to 0.15 and logits-based distillation weight
$w_{\text{LD}}$ to 1. Specifically for CNN-based models, we set blurred
knowledge distillation weight $w_{\text{BKD}}$ to $7e^{-4}$ for CIFAR10/100
and $7e^{-5}$ for ImageNet. We follow all the training configurations (batch
size, learning rate, optimizer, _etc_.) from SEW ResNet [14] except ResNet-19,
where we use training configuration from TET [9]. For Transformer-based
models, we set $w_{\text{BKD}}$ to $2e^{-3}$ for both CIFAR10/100 and
ImageNet, and we follow all the training configurations (batch size, learning
rate, optimizer, etc.) from Sformer+CML [62].

#### Neuromorphic Dataset and Networks.

We also evaluate BKDSNN on CIFAR10-DVS [22], which is a neuromorphic dataset
that leverages dynamic vision sensor (DVS) to convert 10k frame-based images
from the CIFAR10 dataset into 10k event streams, providing an event-stream
dataset with 10 different classes. It includes 9k training samples and 1k
testing samples. Note that, CIFAR10-DVS is an event-stream dataset tailored to
SNN, so we use SNN models with 16 time-steps as teacher models to guide the
training of SNN models with lower time-steps (_i.e_., 4 and 8). We utilize
Wide-7B-Net [14] and Sformer-2-256 [62] with 4 and 8 inference time-steps as
student models for CNN-based and Transformer-based models, respectively.

For both Wide-7B-Net [14] and former-2-256 [62], we set $\tau$ to 2.0,
$\lambda$ to 0.40 and $w_{\text{LD}}$ to 1. Then we set $w_{\text{BKD}}$ to
$2e^{-3}$ for Wide-7B-Net [14] and $7e^{-5}$ for Sformer-2-256 [62]. We follow
all the training configurations from SEW ResNet [14] for Wide-7B-Net [14], and
Sformer+CML [62] for Sformer-2-256 [62].

#### Settings for Prior KD Methods on SNNs.

Furthermore, to compare BKDSNN with previous KD methods on SNNs (_i.e_., KDSNN
[53] and LaSNN [21]), we follow the settings from [53] and [21] to reproduce
the experimental results on all datasets with CNN-based and Transformer-based
models.

###  4.2 Comparison on Static Datasets

#### Comparison on CIFAR10/100

between BKDSNN and previous methods are listed in Tab. 2(a) and Tab. 2(b),
where BKDSNN surpasses prior SOTA methods with both CNN- and Transformer-based
models. Specifically, BKD applied upon Sformer-4-384 outperforms previous SOTA
by 0.18% on CIFAR10 and 0.93% on CIFAR100, which achieves 96.06% and 81.26%
accuracy respectively.

Intuitively, it is difficult for both KDSNN [53] and LaSNN [21] to improve the
performance of CNN-based models on the CIFAR dataset. For ResNet-19, the
knowledge of the ANN teacher is limited for the SNN student, leading to the
above difficulties. When it comes to ResNet-20, the lightweight model conveys
limited information which results in the ineffectiveness of previous knowledge
distillation methods. In contrast, our BKDSNN is effective on both limited
teacher knowledge as well as lightweight student models, which verifies the
superiority of our method.

Table 2:  Comparison with previous works, using various networks and multiple
datasets. P denotes #parameters. KD denotes knowldege distillation. $\star$:
results taken from [14]. $\dagger$: prior state-of-the-art conversion-based
method QCFS [2]. The prior SOTA results are marked in gray, which is used to
compare with our BKDSNN and report the accuracy difference. Best results are
in bold. – indicates results not available (not reported in original article
or cannot reproduce).

Method Architecture |  R-19 (P: 12.63M) |  R-20 (P: 0.27M) |  Time Step
---|---|---|---
|  CF10
---
Acc
|  CF100
---
Acc
|  CF10
---
Acc
|  CF100
---
Acc
w/o KD | QCFS [2]$\dagger$ | - | - | 83.75 | 55.23 | 4
TET [9] | 94.44 | 74.47 | 87.75 | 59.14 | 4
STBP [60] | 92.92 | 70.86 | – | – | 4
Spiking-R [23] | 92.13 | 70.13 | 87.32 | 59.93 | 4
SEW-R [14] | 93.24 | 70.84 | 89.07 | 60.16 | 4
| Teacher ANN | 95.60 | 79.78 | 91.77 | 68.40 | 1
with KD | KDSNN [53] | 94.36 | 74.08 | 89.03 | 60.18 | 4
LaSNN [21] | 94.21 | 73.92 | 88.41 | 59.17 | 4
| BKDSNN
---
(ours)
| 94.64
---
(+0.20)
| 74.95
---
(+0.48)
| 89.29
---
(+0.22)
| 60.92
---
(+0.74)
4

(a) CNN-based SNN on CF10/100 dataset.

Method Architecture |  Sformer-4-256 (P: 4.15M) |  Sformer-2-384 (P: 5.76M) |  Sformer-4-384 (P: 9.32M) |  Time Step
---|---|---|---|---
|  CF10
---
Acc
|  CF100
---
Acc
|  CF10
---
Acc
|  CF100
---
Acc
|  CF10
---
Acc
|  CF100
---
Acc
w/o KD | Spikformer [65] | 93.94 | 75.96 | 94.80 | 76.95 | 95.19 | 77.86 | 4
+CML [62] | 94.82 | 77.64 | 95.63 | 78.75 | 95.93 | 79.65 | 4
Sformer [61] | 94.77 | 77.43 | 95.22 | 78.34 | 95.61 | 79.09 | 4
+CML [62] | 94.94 | 78.19 | 95.54 | 78.87 | 95.81 | 79.98 | 4
| Teacher ANN | 96.75 | 82.22 | 96.75 | 82.22 | 96.75 | 82.22 | 1
with KD | KDSNN [53] | 95.00 | 78.38 | 95.59 | 79.25 | 95.88 | 80.33 | 4
LaSNN [21] | 94.97 | 78.02 | 95.55 | 78.91 | 95.79 | 79.99 | 4
| BKDSNN
---
(ours)
| 95.29
---
(+0.29)
| 79.41
---
(+1.03)
| 95.90
---
(+0.31)
| 80.63
---
(+1.38)
| 96.06
---
(+0.18)
| 81.26
---
(+0.93)
4

(b) Transformer-based SNN on CF10/100 dataset.

Method Architecture |  |  R-18
---
(P: 11.69M)
|  R-34
---
(P: 21.79M)
|  R-50
---
(P: 25.56M)
Time Step
Top1 Acc | Top1 Acc | Top1 Acc
w/o KD | QCFS [2]$\dagger$ | – | 69.37 | – | 32
TET [9] | – | 68.00 | – | 4
STBP [60] | – | 67.04 | – | 4
Spiking-R [23]$\star$ | 62.32 | 61.86 | 57.66 | 4
SEW-R [14] | 63.18 | 67.04 | 67.78 | 4
| Teacher ANN | 69.76 | 73.31 | 76.31 | 1
with KD | KDSNN [53] | 63.42 | 67.18 | 67.72 | 4
LaSNN [21] | 63.31 | 66.94 | 67.81 | 4
| BKDSNN
---
(ours)
| 65.60
---
(+2.18)
| 71.24
---
(+4.06)
| 72.32
---
(+4.51)
4

(c) CNN-based SNN on ImageNet dataset.

Method Architecture |  Sformer-8-384 (P: 16.81M) |  Sformer-8-512 (P: 29.68M) |  Sformer-8-768 (P: 66.34M) |  Time Step
---|---|---|---|---
Top1 Acc. | Top1 Acc. | Top1 Acc.
w/o KD | Spikformer [65] | 70.24 | 73.38 | 74.81 | 4
+CML [62] | 72.73 | 75.61 | 77.34 | 4
Sformer [61] | 72.45 | 74.79 | 75.85 | 4
+CML [62] | 74.35 | 76.54 | 77.64 | 4
| Teacher ANN | 81.78 | 81.78 | 81.78 | 1
with KD | KDSNN [53] | 74.62 | 76.44 | 77.83 | 4
LaSNN [21] | 73.85 | 76.38 | 77.66 | 4
| BKDSNN
---
(ours)
| 75.48
---
(+0.86)
| 77.24
---
(+0.70)
| 79.93
---
(+2.10)
4

(d) Transformer-based SNN on ImageNet dataset.

Table 3: (a) Comparison with Meta-SpikeFormer [57] on ImageNet. (b)
Experimental results on CIFAR10-DVS. Mformer represents Meta-SpikeFormer [57].
$\star$: SNN with high inference time-step (_i.e_., 16) serves as the teacher
model for corresponding SNN with low inference time-step (_i.e_., 4 and 8).
The prior state-of-the-art results are marked in gray. Best results are in
bold.

Method Architecture |  Mformer-8-256 (P: 15.11M) |  Mformer-8-384 (P: 31.33M) |  Mformer-8-512 (P: 55.35M) |  Time Step
---|---|---|---|---
Top1 Acc. | Top1 Acc. | Top1 Acc.
w/o KD | Mformer [57] | 71.82 | 75.40 | 77.96 | 1
74.11 | 77.20 | 78.81 | 4
| Teacher ANN | 81.78 | 81.78 | 81.78 | 1
with KD | Mformer [57] | – | – | 79.11 | 1
– | – | 80.00 | 4
BKDSNN (ours) | 72.93 | 76.10 | 79.82 | 1
| 75.32
---
(+1.21)
| 77.92
---
(+0.72)
| 80.93
---
(+0.93)
4

(a) Meta-SpikeFormer on ImageNet dataset.

Transformer-based | CNN-based
---|---
Method | Acc |  | Time
---
Step
Acc |  | Time
---
Step
Method | Acc |  | Time
---
Step
Acc |  | Time
---
Step
| Sformer-2-256
---
+CML [62]$\star$
81.4 | 16 | 81.4 | 16 |  | Wide-7B-Net
---
[14]$\star$
74.4 | 16 | 74.4 | 16
| Sformer-2-256
---
+CML [62]
76.4 | 4 | 79.2 | 8 |  | Wide-7B-Net
---
[14]
64.8 | 4 | 70.2 | 8
KDSNN [53] | 78.1 | 4 | 79.9 | 8 | KDSNN [53] | 66.1 | 4 | 70.9 | 8
LaSNN [21] | 78.3 | 4 | 80.1 | 8 | LaSNN [21] | 66.6 | 4 | 70.5 | 8
| BKDSNN
---
(ours)
| 79.3
---
(+1.0)
4 |  | 80.8
---
(+0.7)
8 |  | BKDSNN
---
(ours)
| 68.3
---
(+1.7)
4 |  | 72.2
---
(+1.3)
8

(b) Experimental results on CIFAR10-DVS.

(a) Feature map visualization. (b) GPU hours on ImageNet.

Figure 3: (a) Feature map visualization of different methods on SEW ResNet-18
(CNN) and Sformer-8-384+CML (Transformer). Input images are sampled from the
ImageNet validation dataset. Shaded colors from blue to red indicate the
impacts of the regions on the classification scores from low to high. (b) Top1
accuracy versus GPU hours on ImageNet with SEW ResNet-50 and
Spikingformer-8-768.

#### Comparison on ImageNet

between BKDSNN and previous methods are tabulated in Tab. 2(c), Tab. 2(d) and
Tab. 3(a), As anticipated, BKDSNN outperforms previous SOTA methods with both
CNN- and Transformer-based models on ImageNet. BKDSNN achieves 72.32% top1
accuracy with SEW ResNet-50 and 80.93% with Mformer-8-512, with 4.51% and
0.93% improvement, respectively.

As reported in Tab. 2(c), Tab. 2(d) and Tab. 3(a), ANN outperforms previous
learning-based SNN by a large margin, which will serve as the teacher model to
provide adequate knowledge for the student SNN. Based on our experiments, the
accuracy improvement of SNN trained by KDSNN [53] and LaSNN [21] is limited,
due to the inappropriate distillation between student and teacher models.
Nevertheless, thanks to the blurred student features that effectively mimic
and restore the teacher features, BKDSNN achieves promising results on the
large-scale dataset (_i.e_. ImageNet).

#### Feature Visualization.

To further interpret BKDSNN, we apply Grad-CAM [46] to visualize the
intermediate feature before the last layer using different methods. In
$\text{2}^{\text{nd}}$ (ANN) to $\text{3}^{\text{rd}}$ (SNN) column of Fig.
3(a), ANN properly focuses on the regions with vital information (highlighted
in red), while learning-based SNN is distracted by the background area. Such
erroneous attention of SNN ($\text{3}^{\text{rd}}$ column in Fig. 3(a)) does
not get calibrated by the previous KD methods, _i.e_., KDSNN [53]
($\text{4}^{\text{th}}$ column) and LaSNN [21] ($\text{5}^{\text{th}}$
column). However, the attention (shaded) region of our BKDSNN
($\text{6}^{\text{th}}$ column) is quite similar to the pre-trained ANN, which
demonstrates the effectiveness of BKD in learning the knowledge of ANN by
mimicking and restoring teacher ANN features with blurred student SNN
features.

#### Comparison of Training Cost.

We plot the evolution curves of validation top1 accuracy during GPU hours as
depicted in Fig. 3(b). Without applying knowledge distillation (denoted as w/o
KD in blue lines), the training of SEW ResNet-50 and Spikingformer-8-768
consume the least GPU hours. With the introduction of layer-wise distillation
(LaSNN [21] in green lines and KDSNN [53] in yellow lines), the training of
SEW ResNet-50 and Spikingformer-8-768 consume more hours while the improvement
is limited. Compared with the layer-wise distillation methods above, our
single-layer BKDSNN (in red lines) saves GPU hours and boosts the performance
significantly.

###  4.3 Comparison on Neuromorphic Dataset

We also evaluate BKDSNN on CIFAR10-DVS with results reported in Tab. 3(b).
Specifically, BKDSNN outperforms previous SOTA results by 1.0% on models with
4 time-steps and 0.7% on models with 8 time-steps. Note that, the improvement
of BKDSNN on Wide-7B-Net [14] is more pronounced than that on Sformer-2-256
[62], which is probably due to the fact that Wide-7B-Net [14] with 16
inference time-steps provides more knowledge for the corresponding student
model with lower time-steps, compared to Sformer-2-256 [62].

###  4.4 Ablation Study

#### Ablation on multiple losses ($\mathcal{L}_{\textrm{LD}}$,
$\mathcal{L}_{\textrm{BKD}}$ and $\mathcal{L}_{\textrm{MD}}$).

We test different settings of BKDSNN, including versions without any knowledge
distillation (w/o KD), with logit-based KD ($+\mathcal{L}_{\textrm{LD}}$),
with BKD ($+\mathcal{L}_{\textrm{BKD}}$), with mixed distillation (logit-based
+ BKD, $+\mathcal{L}_{\textrm{MD}}$). Results are tabulated in Tab. 4(b).
Compared to logit-based KD, applying BKD to the input feature of the last
layer shows on-par accuracy improvement on both CNN- and Transformer-based
SNNs. Furthermore, thanks to the compatibility between the blurred feature-
based KD and logit-based KD, their mixed version shows the maximal accuracy
improvement.

Table 4:  Ablation study with multiple settings. R, S, and KD denotes ResNet,
Sformer and Knowledge Distillation respectively. S-4-384-400E refers to
Sformer-4-384 after 400 epochs training. $\star$: results from [14],
$\diamond$: results from [62]. Best results are in bold, runner-up results are
marked in gray.

Method | Config |  S-4-384-400E |  R-19
---|---|---|---
$\tau$ | $\lambda$ |  |  CF10
---
Acc
|  CF100
---
Acc
|  CF10
---
Acc
|  CF100
---
Acc
$+\mathcal{L}_{\text{LD}}$ | 1.5 | - | 96.13 | 81.40 | 94.50 | 74.66
2.0 | - | 96.17 | 81.25 | 94.53 | 74.57
4.0 | - | 96.31 | 80.88 | 94.56 | 74.52
$+\mathcal{L}_{\text{BKD}}$ | - | 0.15 | 96.13 | 81.38 | 94.52 | 74.76
- | 0.40 | 96.26 | 81.43 | 94.56 | 74.83
- | 0.60 | 96.31 | 81.53 | 94.61 | 74.89
$+\mathcal{L}_{\text{MD}}$ | 2.0 | 0.15 | 96.18 | 81.63 | 94.64 | 74.95
2.0 | 0.40 | 96.20 | 81.67 | 94.68 | 74.98
2.0 | 0.60 |  96.37 (+0.11) |  81.86 (+0.19) |  94.76 (+0.08) |  75.07 (+0.09)

(a) Ablation of $\tau$ and $\lambda$ on CIFAR.

Method | CNN-based | Transformer-based
---|---|---
| Student
---
SNN
| Teacher
---
Acc
| SNN
---
Acc
| Student
---
SNN
| Teacher
---
Acc
| SNN
---
Acc
w/o KD |  R-18$\star$ | - | 63.18 |  S-8-384$\diamond$ | - | 74.35
R-34$\star$ | - | 67.04 |  S-8-512$\diamond$ | - | 76.54
R-50$\star$ | - | 67.78 |  S-8-768$\diamond$ | - | 77.64
$+\mathcal{L}_{\text{LD}}$ |  R-18 | 69.76 | 65.36(+2.18) |  S-8-384 | 81.78 | 74.69(+0.24)
R-34 | 73.31 | 69.90(+2.86) |  S-8-512 | 76.99(+0.45)
R-50 | 76.13 | 71.78(+4.00) |  S-8-768 | 79.04(+1.40)
$+\mathcal{L}_{\text{BKD}}$ |  R-18 | 69.76 | 65.16(+1.98) |  S-8-384 | 81.78 | 75.08(+0.73)
R-34 | 73.31 | 70.23(+3.19) |  S-8-512 | 77.29(+0.75)
R-50 | 76.13 | 71.24(+3.46) |  S-8-768 | 79.64(+2.00)
$+\mathcal{L}_{\text{MD}}$ |  R-18 | 69.76 | 65.60(+2.42) |  S-8-384 | 81.78 | 75.48(+1.13)
R-34 | 73.31 | 71.24(+4.20) |  S-8-512 | 77.24(+0.70)
R-50 | 76.13 | 72.32(+4.54) |  S-8-768 | 79.93(+2.29)

(b) Ablation of multiple losses on ImageNet.

Method |  SEW R-18 Top1 Acc |  S-8-384 Top1 Acc |  KD location
---|---|---|---
LaSNN [21] | 63.31 | 73.85 | layer-wise
+$\mathcal{L}_{\text{BKD}}$ | 63.82 | 74.47 | layer-wise
KDSNN [53] | 63.42 | 74.62 | layer-wise
+$\mathcal{L}_{\text{BKD}}$ | 63.94 | 74.93 | layer-wise
KDSNN [53] | 63.87 | 74.96 | single-layer
| BKDSNN
---
(ours)
| 65.60
---
(+1.66)
| 75.48
---
(+0.52)
single-layer

(c) Ablation of BKD on ImageNet.

Method | $\lambda$ | Spikingformer-4-384-400E | ResNet-19
---|---|---|---
|  CIFAR10
---
Acc
|  CIFAR100
---
Acc
|  CIFAR10
---
Acc
|  CIFAR100
---
Acc
w $\mathcal{G}$ | w/o $\mathcal{G}$ | w $\mathcal{G}$ | w/o $\mathcal{G}$ | w $\mathcal{G}$ | w/o $\mathcal{G}$ | w $\mathcal{G}$ | w/o $\mathcal{G}$
$+\mathcal{L}_{\text{BKD}}$ | 0.00 | 95.38 | 94.38 | 80.44 | 76.64 | 94.47 | 93.42 | 74.68 | 70.97
0.15 | 96.13 | 94.60 | 81.38 | 76.81 | 94.52 | 93.58 | 74.76 | 71.22
0.40 | 96.26 | 94.82 | 81.43 | 77.14 | 94.56 | 93.64 | 74.83 | 71.29
0.60 | 96.31 | 94.89 | 81.53 | 77.16 | 94.61 | 93.72 | 74.89 | 71.37

(d) Ablation of $\lambda$ and $\mathcal{G}$ on CIFAR.

(a) SEW ResNet-18 (b) SEW ResNet-34 (c) SEW ResNet-50

(d) Sformer-8-384 (e) Sformer-8-512 (f) Sformer-8-768

Figure 4: Evolution curve of validation accuracy during training, with CNN-
(top 3 sub-figures) and Transformer-based (bottom 3 sub-figures) SNNs on
ImageNet. The labels in the legends are the same as in Tab. 4(b).

In addition to the accuracy obtained for SEW ResNet-18 and Sformer-8-384
discussed above, we also plot their evolution curves of validation accuracy
during training as in Fig. 4. We observe that with the introduction of logits-
based distillation $\mathcal{L}_{\text{LD}}$ (yellow curve) and feature-based
blurred knowledge distillation $\mathcal{L}_{\text{BKD}}$ (green curve), the
performance is significantly improved compared to the vanilla counterpart
(blue curve). Moreover, as anticipated, the mixed distillation (red curve) in
KDSNN leads to faster convergence with higher validation accuracy during the
training. These experimental results highlight the important role of logits-
based and feature-based knowledge from the teacher ANN in improving the
performance of student SNNs.

#### Ablation of BKD and its location

We have reported the KDSNN accuracy under different settings in Tab. 4(b)
where KD was applied to logits, the last layer feature, or both. Two more
questions still remain unanswered: 1) whether BKD outperforms the vanilla
implementation? 2) Is applying KD/BKD to the last layer better than the layer-
wise implementation?

To answer these questions, we perform experiments whose results are reported
in Tab. 4(c). LaSNN [21] and KDSNN [53] are two recent works of layer-wise
implementation of KD for SNN training. We replace the original feature-based
distillation loss in LaSNN and KDSNN with BKD loss $\mathcal{L}_{\text{BKD}}$.
In LaSNN, the replacement improves accuracy by 0.51% on SEW ResNet-18 and by
0.62% on Sformer-8-384. In KDSNN, the replacement improves accuracy by 0.52%
on SEW ResNet-18 and by 0.31% on Sformer-8-384. These experiments confirm the
advantages of the blurred implementation of KD. Moreover, we further test
layer-wise (every layer feature) or single-layer (intermediate feature before
the last layer) implementations of KD. We observe similar accuracy boosts for
the two implementations. Thus, we conclude that applying BKD only to the
intermediate feature map before the last layer is the adequate setup.

#### Ablation of temperature coefficient, blurred ratio and restoration block.

To examine the optimal setting of hyper-parameters, _i.e_., temperature $\tau$
in Eqs. 7 and 8, blurred ration $\lambda$ in Eq. 2 and restoration block
$\mathcal{G}$ in Eq. 4, we sweep $\tau$, $\lambda$ and $\mathcal{G}$ with
ResNet-19 and Sformer-4-384, under varying BKDSNN settings. The results of the
CIFAR-10 and CIFAR-100 datasets are reported in Tab. 4(a) and Tab. 4(d). Note
that we increase the training epochs of Sformer-4-384 to 400 for more distinct
results. For logits-based distillation $\mathcal{L}_{\text{LD}}$, we find that
flatter logits distribution (with higher $\tau$) is better for classification
with smaller categories (CIFAR10), while sharper logits distribution (with
lower $\tau$) is more effective for classification with larger categories
(CIFAR100). For BKD $\mathcal{L}_{\text{BKD}}$ and mixed distillation
$\mathcal{L}_{\text{MD}}$, training with a higher blurred ratio $\lambda$
(_i.e_., 0.60) achieves greater accuracy across various datasets and models.
As reported in Tab. 4(d), with the introduction of blurred ration $\lambda$
and restoration block $\mathcal{G}$, student SNN better mimics teacher ANN and
conveys more sufficient information, which enhances performance on CIFAR-10
and CIFAR-100 with both CNN- and Transformer-based models.

###  4.5 Firing Rate Analysis

#### Firing Rate.

We compare the firing rate $r$ of the intermediate feature before last layer
on CIFAR with Sformer-4-384 and ResNet-19 under multiple settings. The firing
rate $r$ is defined as:

$r=\frac{1}{bc^{\text{stu}}hw}\displaystyle\sum_{i}^{b}\sum_{j}^{c^{\text{stu}}}\sum_{k}^{h}\sum_{l}^{w}\tilde{{\bm{\mathsfit{F}}}}_{i,j,k,l}^{\text{stu}}$
(11)

As reported in Tab. 5(a), it reveals that on both CNN- and Transformer-based
models, setting a higher firing rate $r$ achieves higher accuracy. Note that,
with the introduction of $\mathcal{L}_{\text{BKD}}$ and
$\mathcal{L}_{\text{MD}}$, high blurred ratio $\lambda$ encourages model to
fire more spikes. Furthermore, compared with $\mathcal{L}_{\text{LD}}$, our
$\mathcal{L}_{\text{BKD}}$ reduces the firing rate while improving the
performance, which assists the energy consumption reduction. Meanwhile, in
virtue of $\mathcal{L}_{\text{MD}}$, BKDSNN significantly improves the
performance with a slight boost on firing rate, which leads to state-of-the-
art.

###  4.6 Energy Consumption Analysis

Table 5: Firing rate $r$ under multiple settings on CIFAR and Energy
consumption of ANN and SNN on ImageNet. T denotes Transformer.

Method | Config |  S-4-384-400E |  R-19
---|---|---|---
$\tau$ | $\lambda$ |  |  CF10
---
r ($e^{-2}$)
|  CF10
---
Acc
|  CF100
---
r ($e^{-2}$)
|  CF100
---
Acc
|  CF10
---
r ($e^{-2}$)
|  CF10
---
Acc
|  CF100
---
r ($e^{-2}$)
|  CF100
---
Acc
$+\mathcal{L}_{\text{LD}}$ | 1.5 | - | 5.55 | 96.13 | 1.52 | 81.40 | 3.94 | 94.50 | 0.93 | 74.66
2.0 | - | 5.72 | 96.17 | 1.32 | 81.25 | 4.03 | 94.53 | 1.01 | 74.57
4.0 | - | 6.07 | 96.31 | 1.18 | 80.88 | 4.11 | 94.56 | 1.08 | 74.52
$+\mathcal{L}_{\text{BKD}}$ | - | 0.15 | 4.67 | 96.13 | 0.97 | 81.38 | 3.82 | 94.52 | 0.82 | 74.76
- | 0.40 | 5.30 | 96.26 | 1.03 | 81.43 | 3.94 | 94.56 | 0.89 | 74.83
- | 0.60 | 5.55 | 96.31 | 1.14 | 81.53 | 4.08 | 94.61 | 1.02 | 74.89
$+\mathcal{L}_{\text{MD}}$ | 2.0 | 0.15 | 5.39 | 96.18 | 1.14 | 81.63 | 3.92 | 94.64 | 1.03 | 74.95
2.0 | 0.40 | 6.17 | 96.20 | 1.34 | 81.67 | 3.99 | 94.68 | 1.07 | 74.98
2.0 | 0.60 | 6.66 | 96.37 | 1.55 | 81.86 | 4.18 | 94.76 | 1.13 | 75.07

(a) Firing rate $r$ on CIFAR.

Method |  | Time
---
Step
| SOPS(G)
---
($\downarrow$)
| Energy Consum-
---
ption (mJ)($\downarrow$)
| Top1
---
Acc ($\uparrow$)
R-50 [19] | 1 | 4.12 (FLOPS) | 18.95 | 76.13
Spiking R-50 [14] | 4 | 5.31(28.88%$\uparrow$) | 5.44(71.29%$\downarrow$) | 57.66(-13.47)
SEW R-50 [14] | 4 | 4.83(17.23%$\uparrow$) | 4.89(74.20%$\downarrow$) | 67.78(-8.35)
BKDSNN (ours) | 4 | 4.81(16.75%$\uparrow$) | 4.85(74.40%$\downarrow$) | 72.32(-3.81)
T-8-768 [12] | 1 | 18.60 (FLOPS) | 85.56 | 81.16
S-8-768 [61] | 4 | 12.54(32.58%$\downarrow$) | 13.68(84.01%$\downarrow$) | 75.85(-5.31)
S-8-768+CML [62] | 4 | 12.30(33.87%$\downarrow$) | 13.42(84.32%$\downarrow$) | 77.64(-3.52)
BKDSNN (ours) | 4 | 12.60(32.26%$\downarrow$) | 13.72(83.96%$\downarrow$) | 79.93(-1.23)

(b) Energy consumption on ImageNet.

As tabulated in Tab. 5(b), we calculate the energy consumption of BKDSNN
according to the energy model in [29, 65, 25] and compare it with its ANN
counterparts and SNNs trained by previous learning-based methods. In terms of
the OPS, SNN variants are higher than ANNs, while ANNs use FLOPS as the
operation counting metric. Nevertheless, since the energy per synaptic
operation is more efficient than the operation in ANN. For ResNet-50, BKDSNN
consumes the lowest energy while optimally suppressing accuracy loss. For
Sformer-8-768, our BKDSNN consumes more energy to increase firing rates for
maintaining similar performance as SOTA methods for SNN training.

##  5 Conclusion

BKDSNN constructs a knowledge distillation based framework for learning-based
SNN training, achieving promising performance on both static and neuromorphic
datasets supported by comprehensive experiments. We expect that BKDSNN will
contribute to future investigations on brain-inspired SNN, which can further
narrow the performance gap between SNN and ANN. In this work, we mainly focus
on learning-based SNN training methods due to their remarkable performance
under ultra-low inference time-step. We anticipate to extend BKDSNN on
conversion-based methods, which is expected to overcome the existing
bottlenecks of conversion-based methods under ultra-low inference time-step.

## Acknowledgements

This work is partially supported by National Key R&D Program of China
(2022YFB4500200), National Natural Science Foundation of China (No. 62102257),
Biren Technology–Shanghai Jiao Tong University Joint Laboratory Open Research
Fund, Microsoft Research Asia Gift Fund, Lingang Laboratory Open Research Fund
(No.LGQS-202202-11).

## References

  * [1] Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al.: A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7243–7252 (2017)
  * [2] Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., Huang, T.: Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks. arXiv preprint arXiv:2303.04347 (2023)
  * [3] Cao, Y., Chen, Y., Khosla, D.: Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision 113, 54–66 (2015)
  * [4] Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9233–9242 (2020)
  * [5] Chen, L., Yu, C., Chen, L.: A new knowledge distillation for incremental object detection. In: 2019 International Joint Conference on Neural Networks (IJCNN). pp. 1–7. IEEE (2019)
  * [6] Davies, M., Srinivasa, N., Lin, T.H., Chinya, G., Cao, Y., Choday, S.H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al.: Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro 38(1), 82–99 (2018)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [8] Deng, S., Gu, S.: Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476 (2021)
  * [9] Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946 (2022)
  * [10] Diehl, P., Neil, D., Binas, J., Cook, M., Liu, S.C., Pfeiffer, M.: Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing (07 2015). https://doi.org/10.1109/IJCNN.2015.7280696
  * [11] Ding, J., Bu, T., Yu, Z., Huang, T., Liu, J.: Snn-rat: Robustness-enhanced spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems 35, 24780–24793 (2022)
  * [12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [13] Fang, W., Chen, Y., Ding, J., Yu, Z., Masquelier, T., Chen, D., Huang, L., Zhou, H., Li, G., Tian, Y.: Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances 9(40), eadi1480 (2023). https://doi.org/10.1126/sciadv.adi1480, https://www.science.org/doi/abs/10.1126/sciadv.adi1480
  * [14] Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., Tian, Y.: Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems 34, 21056–21069 (2021)
  * [15] Garg, I., Chowdhury, S.S., Roy, K.: Dct-snn: Using dct to distribute spatial information over time for low-latency spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4671–4680 (2021)
  * [16] Gerstner, W., Kistler, W.M.: Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press (2002)
  * [17] Han, B., Roy, K.: Deep spiking neural network: Energy efficiency through time based coding. In: European Conference on Computer Vision. pp. 388–404. Springer (2020)
  * [18] Han, B., Srinivasan, G., Roy, K.: Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13558–13567 (2020)
  * [19] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [20] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [21] Hong, D., Shen, J., Qi, Y., Wang, Y.: Lasnn: Layer-wise ann-to-snn distillation for effective and efficient training in deep spiking neural networks. arXiv preprint arXiv:2304.09101 (2023)
  * [22] Hongmin, L., Hanchao, L., Xiangyang, J., Guoqi, L., Luping, S.: Cifar10-dvs: An event-stream dataset for object classification. Frontiers in Neuroscience 11 (2017)
  * [23] Hu, Y., Tang, H., Pan, G.: Spiking deep residual network (2020)
  * [24] Hu, Y., Zheng, Q., Jiang, X., Pan, G.: Fast-snn: Fast spiking neural network by converting quantized ann. arXiv preprint arXiv:2305.19868 (2023)
  * [25] Hu, Y., Deng, L., Wu, Y., Yao, M., Li, G.: Advancing spiking neural networks towards deep residual learning (2023)
  * [26] Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artificial intelligence review 53, 5455–5516 (2020)
  * [27] Kheradpisheh, S.R., Ganjtabesh, M., Thorpe, S.J., Masquelier, T.: Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks 99, 56–67 (2018)
  * [28] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [29] Kundu, S., Pedram, M., Beerel, P.A.: Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise (2021)
  * [30] Kushawaha, R.K., Kumar, S., Banerjee, B., Velmurugan, R.: Distilling spikes: Knowledge distillation in spiking neural networks. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 4536–4543. IEEE (2021)
  * [31] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * [32] Lee, D., Park, S., Kim, J., Doh, W., Yoon, S.: Energy-efficient knowledge distillation for spiking neural networks. arXiv preprint arXiv:2106.07172 (2021)
  * [33] Li, C., Ma, L., Furber, S.: Quantization framework for fast spiking neural networks. Frontiers in Neuroscience 16, 918793 (2022)
  * [34] Li, Y., Deng, S., Dong, X., Gong, R., Gu, S.: A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In: International conference on machine learning. pp. 6316–6325. PMLR (2021)
  * [35] Liu, F., Zhao, W., Chen, Y., Wang, Z., Yang, T., Li, J.: Sstdp: Supervised spike timing dependent plasticity for efficient spiking neural network training. Frontiers in neuroscience p. 1413 (2021)
  * [36] Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge distillation for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2604–2613 (2019)
  * [37] Maass, W.: Networks of spiking neurons: the third generation of neural network models. Neural networks 10(9), 1659–1671 (1997)
  * [38] Merolla, P.A., Arthur, J.V., Alvarez-Icaza, R., Cassidy, A.S., Sawada, J., Akopyan, F., Jackson, B.L., Imam, N., Guo, C., Nakamura, Y., et al.: A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197), 668–673 (2014)
  * [39] Neftci, E.O., Mostafa, H., Zenke, F.: Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine 36(6), 51–63 (2019)
  * [40] Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience 9, 437 (2015)
  * [41] Ostojic, S.: Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons. Nature neuroscience 17(4), 594–600 (2014)
  * [42] Peng, C., Zhao, K., Maksoud, S., Li, M., Lovell, B.C.: Sid: Incremental learning for anchor-free object detection via selective and inter-related distillation. Computer vision and image understanding 210, 103229 (2021)
  * [43] Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 12179–12188 (2021)
  * [44] Roy, K., Jaiswal, A., Panda, P.: Towards spike-based machine intelligence with neuromorphic computing. Nature 575(7784), 607–617 (2019)
  * [45] Rueckauer, B., Lungu, I.A., Hu, Y., Pfeiffer, M., Liu, S.C.: Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience 11, 682 (2017)
  * [46] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision 128(2), 336–359 (Oct 2019). https://doi.org/10.1007/s11263-019-01228-7, http://dx.doi.org/10.1007/s11263-019-01228-7
  * [47] Sengupta, A., Ye, Y., Wang, R., Liu, C., Roy, K.: Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience 13, 95 (2019)
  * [48] Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: Hats: Histograms of averaged time surfaces for robust event-based object classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1731–1740 (2018)
  * [49] Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355 (2019)
  * [50] Takuya, S., Zhang, R., Nakashima, Y.: Training low-latency spiking neural network through knowledge distillation. In: 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS). pp. 1–3. IEEE (2021)
  * [51] Tan, G., Wang, Y., Han, H., Cao, Y., Wu, F., Zha, Z.J.: Multi-grained spatio-temporal features perceived network for event-based lip-reading. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20094–20103 (2022)
  * [52] Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 1311–1318 (2019)
  * [53] Xu, Q., Li, Y., Shen, J., Liu, J.K., Tang, H., Pan, G.: Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7886–7895 (2023)
  * [54] Xu, Q., Li, Y., Shen, J., Zhang, P., Liu, J.K., Tang, H., Pan, G.: Hierarchical spiking-based model for efficient image classification with enhanced feature extraction and encoding. IEEE Transactions on Neural Networks and Learning Systems (2022)
  * [55] Xu, Z., Zhang, M., Hou, J., Gong, X., Wen, C., Wang, C., Zhang, J.: Delving into transformer for incremental semantic segmentation (2022)
  * [56] Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., Yuan, C.: Focal and global knowledge distillation for detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4643–4652 (2022)
  * [57] Yao, M., Hu, J., Hu, T., Xu, Y., Zhou, Z., Tian, Y., Bo, X., Li, G.: Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In: The Twelfth International Conference on Learning Representations (2023)
  * [58] Zenke, F., Agnes, E.J., Gerstner, W.: Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nature communications 6(1), 6922 (2015)
  * [59] Zenke, F., Vogels, T.P.: The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation 33(4), 899–925 (2021)
  * [60] Zheng, H., Wu, Y., Deng, L., Hu, Y., Li, G.: Going deeper with directly-trained larger spiking neural networks. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 11062–11070 (2021)
  * [61] Zhou, C., Yu, L., Zhou, Z., Zhang, H., Ma, Z., Zhou, H., Tian, Y.: Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954 (2023)
  * [62] Zhou, C., Zhang, H., Zhou, Z., Yu, L., Ma, Z., Zhou, H., Fan, X., Tian, Y.: Enhancing the performance of transformer-based spiking neural networks by improved downsampling with precise gradient backpropagation. arXiv preprint arXiv:2305.05954 (2023)
  * [63] Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., Feng, J.: Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 (2021)
  * [64] Zhou, H., Song, L., Chen, J., Zhou, Y., Wang, G., Yuan, J., Zhang, Q.: Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650 (2021)
  * [65] Zhou, Z., Zhu, Y., He, C., Wang, Y., Yan, S., Tian, Y., Yuan, L.: Spikformer: When spiking neural network meets transformer (2022)

Supplementary

##  A1 Definition of Time-Step.

In spiking neural network (SNN), time-step refers to a discrete unit of time
used in the simulation of the network dynamics [2, 10, 33, 45]. Unlike the
traditional artificial neural network (ANN), which often operates in a
continuous manner, SNN models the behavior of biological neurons by
considering the timing (time-stamp) of spikes. The simulation of SNN typically
involves breaking down time into discrete steps, and for each time-step, the
network processes incoming signals, updates neuron membrane potentials, and
generates spikes based on certain rules and conditions. The time-step is a
fundamental parameter that influences the temporal resolution of the
simulation and is crucial for capturing the dynamics of spiking neurons. The
choice of an appropriate time-step is essential for _balancing computational
efficiency and accuracy in simulating the behavior of the network_. With
greater time-step, SNN can provides a more accurate representation of the
temporal dynamics but the inference consumes more energy. With smaller time-
step, on the other hand, it can speed up simulations but may lead to a loss of
precision (_i.e_., inference accuracy).

Algorithm 1 Training Algorithm of BKDSNN

Input: Dataset D.
Model: Teacher ANN model
$\mathcal{M}_{\text{ANN}}(x;\tilde{{\bm{\mathsfit{W}}}})$ with pretrained
weight $\tilde{{\bm{\mathsfit{W}}}}$; Student SNN model
$\mathcal{M}_{\text{SNN}}(x;{\bm{\mathsfit{W}}})$ with initial weight
${\bm{\mathsfit{W}}}$; Blurred restoration block $\mathcal{G}$.
Parameter: Temperature $\tau$ and weight $w_{\text{LD}}$ for
$\mathcal{L}_{\text{LD}}$; Blurred ratio $\lambda$ and weight $w_{\text{BKD}}$
for $\mathcal{L}_{\text{BKD}}$.
Output: Student SNN model $\mathcal{M}_{\text{SNN}}(x;{\bm{\mathsfit{W}}})$.

1: for $i=1$ to $e$ do

2: for $x,\hat{y}$ in D do

3: # forward propagation

4:
${\bm{y}}^{\text{tea}},{\bm{\mathsfit{F}}}^{\text{tea}}=\mathcal{M}_{\text{ANN}}(x)$;

5:
${\bm{y}}^{\text{stu}},{\bm{\mathsfit{F}}}^{\text{stu}}=\mathcal{M}_{\text{SNN}}(x)$;

6: #calculate $\mathcal{L}_{\text{LD}}$

7: calculate ${\bm{q}}^{\text{stu}}$ and ${\bm{q}}^{\text{tea}}$ using Eq. 7;

8: $\mathcal{L}_{\text{LD}}=\tau^{2}\cdot\mathcal{L}_{\text{ CrossEntropy
}}\left({\bm{q}}^{\text{stu}},{\bm{q}}^{\text{tea}}\right)$;

9: # calculate $\mathcal{L}_{\text{BKD}}$

10: calculate $\hat{{\bm{\mathsfit{F}}}}^{\text{stu}}$ using Eq. 2, Eq. 3 and
Eq. 4;

11:
$\mathcal{L}_{\text{BKD}}=\displaystyle\sum_{i}^{b}\sum_{j}^{c^{\text{stu}}}\sum_{k}^{h}\sum_{l}^{w}\left(\hat{{\bm{\mathsfit{F}}}}_{i,j,k,l}^{\text{stu}}-{\bm{\mathsfit{F}}}_{i,j,k,l}^{\text{tea}}\right)^{2}$;

12: # calculate total loss $\mathcal{L}$

13:
$\mathcal{L}_{\text{MD}}=w_{\text{LD}}\cdot\mathcal{L}_{\text{LD}}+w_{\text{BKD}}\cdot\mathcal{L}_{\text{BKD}}$;

14: $\mathcal{L}=\mathcal{L}_{\text{ CrossEntropy
}}\left({\bm{y}}^{\text{stu}},\hat{{\bm{y}}}\right)+\mathcal{L}_{\text{MD}}=\mathcal{L}_{\text{
task}}+\mathcal{L}_{\text{MD}}$;

15: # backward propagation

16: calculate the gradients
$\frac{\partial\mathcal{L}}{\partial{\bm{\mathsfit{W}}}}$;

17: update ${\bm{\mathsfit{W}}}$;

18: end for

19: end for

##  A2 Algorithm for BKDSNN.

The process of our algorithm is specified in Algorithm 1. In each training
iteration, we first calculate the logits-based distillation loss
$\mathcal{L}_{\text{LD}}$, using output logits ${\bm{y}}^{\text{tea}}$ and
${\bm{y}}^{\text{stu}}$. Then, we extract the intermediate feature
${\bm{\mathsfit{F}}}^{\text{tea}}$ and ${\bm{\mathsfit{F}}}^{\text{stu}}$ in
front of the last layer from the teacher ANN $\mathcal{M}_{\text{ANN}}$ and
the student SNN $\mathcal{M}_{\text{SNN}}$ respectively. Afterwards, we
leverage ${\bm{\mathsfit{F}}}^{\text{tea}}$,
${\bm{\mathsfit{F}}}^{\text{stu}}$ together with the blurred restoration block
$\mathcal{G}$ and the blurred ratio $\lambda$ to calculate a blurred knowledge
distillation loss $\mathcal{L}_{\text{BKD}}$. Combined with the cross entropy
task loss between ${\bm{y}}^{\text{tea}}$ and ground truth label
$\hat{{\bm{y}}}$, we obtain the total loss $\mathcal{L}$ which can be used
with back-propagation through time (BPTT) [52, 59] algorithm to update
${\bm{\mathsfit{W}}}$ in $\mathcal{M}_{\text{SNN}}$.

##  A3 Ablation Study on CIFAR.

Ablation results on CIFAR are shown in Tabs. A1, A2 and A3, where the
introduction of logits-based distillation loss $\mathcal{L}_{\text{LD}}$,
blurred knowledge distillation $\mathcal{L}_{\text{BKD}}$ and mixed
distillation $\mathcal{L}_{\text{MD}}$ significantly improve the performance,
compared with methods without knowledge distillation (denoted as w/o KD).
However, as shown in Tab. A1, the introduction of $\mathcal{L}_{\text{LD}}$
and $\mathcal{L}_{\text{BKD}}$ are more effective than
$\mathcal{L}_{\text{MD}}$ on CIFAR10 with Sformer-2-384 and Sformer-4-384.
_Limited training epochs on a simple dataset lead to the experimental results
above_. Therefore, we increase the training epochs from 300 to 400 for more
distinct results. As shown in Tab. A2, the introduction of
$\mathcal{L}_{\text{MD}}$ achieves maximized accuracy boost under all settings
on CIFAR, leading to state-of-the-art performance on CIFAR (96.37% on CIFAR10
and 81.63% on CIFAR100). Along with the experimental results on CIFAR with
CNN-based models in Tab. A3, the effectiveness of our BKDSNN is further
verified.

Figure A1: Illustration of gradient correction with BKD for IF neuron. Table
A1: Ablation on CIFAR with Transformer-based models after 300 epochs.

Method |  | CIFAR10 | CIFAR100
---|---|---|---
Student SNN | Teacher ANN |  SNN Acc | Teacher ANN |  SNN Acc
model | Acc | model | Acc
w/o KD |  S-4-256 [53] | - | - | 94.94 | - | - | 78.19
S-2-384 [53] | - | - | 95.54 | - | - | 78.87
S-4-384 [53] | - | - | 95.81 | - | - | 79.98
$+\mathcal{L}_{\text{LD}}$ |  S-4-256 | ViT-S/16 | 96.75 | 95.08(+0.14) | ViT-S/16 | 82.22 | 78.73(+0.54)
S-2-384 | 95.96(+0.42) | 80.02(+1.15)
S-4-384 | 96.07(+0.26) | 80.82(+0.84)
$+\mathcal{L}_{\text{BKD}}$ |  S-4-256 | ViT-S/16 | 96.75 | 95.22(+0.28) | ViT-S/16 | 82.22 | 79.22(+1.03)
S-2-384 | 95.75(+0.21) | 80.26(+1.39)
S-4-384 | 96.12(+0.31) | 81.18(+1.20)
$+\mathcal{L}_{\text{MD}}$ |  S-4-256 | ViT-S/16 | 96.75 | 95.29(+0.35) | ViT-S/16 | 82.22 | 79.41(+1.22)
S-2-384 | 95.90(+0.36) | 80.63(+1.76)
S-4-384 | 96.06(+0.25) | 81.26(+1.28)

Table A2: Ablation on CIFAR with Transformer-based models after 400 epochs.

Method |  | CIFAR10 | CIFAR100
---|---|---|---
Student SNN | Teacher ANN |  SNN Acc | Teacher ANN |  SNN Acc
model | Acc | model | Acc
w/o KD |  S-4-256-400E [53] | - | - | 95.31 | - | - | 78.58
S-2-384-400E [53] | - | - | 95.75 | - | - | 79.17
S-4-384-400E [53] | - | - | 95.95 | - | - | 80.37
$+\mathcal{L}_{\text{LD}}$ |  S-4-256-400E | ViT-S/16 | 96.75 | 95.60(+0.29) | ViT-S/16 | 82.22 | 79.54(+0.96)
S-2-384-400E | 95.95(+0.20) | 80.56(+1.39)
S-4-384-400E | 96.17(+0.22) | 81.25(+0.88)
$+\mathcal{L}_{\text{BKD}}$ |  S-4-256-400E | ViT-S/16 | 96.75 | 95.43(+0.12) | ViT-S/16 | 82.22 | 79.63(+1.05)
S-2-384-400E | 95.93(+0.18) | 80.83(+1.66)
S-4-384-400E | 96.13(+0.18) | 81.38(+1.01)
$+\mathcal{L}_{\text{MD}}$ |  S-4-256-400E | ViT-S/16 | 96.75 | 95.76(+0.45) | ViT-S/16 | 82.22 | 79.86(+1.28)
S-2-384-400E | 96.18(+0.43) | 80.83(+1.66)
S-4-384-400E | 96.37(+0.42) | 81.63(+1.26)

Table A3: Ablation on CIFAR with CNN-based models.

Method |  | CIFAR10 | CIFAR100
---|---|---|---
Student SNN | Teacher ANN |  SNN Acc | Teacher ANN |  SNN Acc
model | Acc | model | Acc
w/o KD | TET R-19 [9] | - | - | 94.44 | - | - | 74.47
SEW R-20 [14] | - | - | 89.07 | - | - | 60.16
$+\mathcal{L}_{\text{LD}}$ | TET R-19 |  R-19 | 95.60 | 94.49(+0.05) |  R-19 | 79.78 | 74.58(+0.11)
SEW R-20 |  R-20 | 91.77 | 89.13(+0.06) |  R-20 | 68.40 | 60.36(+0.20)
$+\mathcal{L}_{\text{BKD}}$ | TET R-19 |  R-19 | 95.60 | 94.58(+0.14) |  R-19 | 79.78 | 74.82(+0.35)
SEW R-20 |  R-20 | 91.77 | 89.21(+0.14) |  R-20 | 68.40 | 60.57(+0.41)
$+\mathcal{L}_{\text{MD}}$ | TET R-19 |  R-19 | 95.60 | 94.64(+0.20) |  R-19 | 79.78 | 74.95(+0.48)
SEW R-20 |  R-20 | 91.77 | 89.29(+0.22) |  R-20 | 68.40 | 60.92(+0.76)

(a) Q, K, V and attention firing rate curve of Sformer-4-384 under multiple
settings on CIFAR10.

(b) Q, K, V and attention firing rate curve of Sformer-4-384 under multiple
settings on CIFAR100.

(c) Q, K, V and attention firing rate curve of Sformer with multiple feature
dimension on ImageNet.

Figure A2: Q, K, V and attention firing rate curve of Sformer on both CIFAR
and ImageNet. The top 2 rows are Q, K, V and attention firing rate curve with
multiple blurred ratio $\lambda$ on CIFAR, it is obvious that higher blurred
ratio $\lambda$ leads to higher firing rate. The bottom row is Q, K, V and
attention firing rate curve of Sformer with multiple feature dimension on
ImageNet, Sformer with higher dimension fires more spikes in attention map.

##  A4 Effectiveness of BKD on Gradient Correction.

We further analyse the advantages of proposed BKD on SNN training. In
learning-based SNN training, spiking neuron is a vital component. As
illustrated in Eq. 1, spiking neuron leverages the heaviside function
$\Theta\left(\cdot\right)$ to generate spikes. However, the gradient of
heaviside function $\Theta\left(\cdot\right)$ (Eq. A1) is not applicable to
classical back-propagation based training as the existence of infinity value
results in instability.

$\dfrac{\partial\Theta(v)}{\partial v}=\begin{cases}+\infty,&v=0\\\
0,&\text{otherwise}\end{cases}$ (A1)

Aiming at avoiding infinity value during back-propogation, [39] proposed the
surrogate gradient method which introduced a surrogate function to replace the
Heaviside function. In this paper we use the sigmoid function
$\hat{\Theta}(\cdot,\alpha)$ (Eq. A2) as the surrogate function due to its
versatility and stability [65, 61, 62].

$\begin{gathered}\hat{\Theta}\left(v,\alpha\right)=\frac{1}{1+e^{-\alpha
v}}\\\
\hat{\Theta}^{\prime}\left(v,\alpha\right)=\alpha\hat{\Theta}\left(v,\alpha\right)-\alpha\hat{\Theta}^{2}\left(v,\alpha\right)\end{gathered}$
(A2)

where $\alpha$ refers to smoothing factor and we set it to 4.0 as previous
works [65, 61, 62]. Then we derive the gradient
$\frac{\partial\mathcal{L}_{\text{total}}}{\partial{\bm{\mathsfit{F}}}^{\text{stu}}_{i,j,k,l}}$
generated by $\mathcal{L}_{\text{total}}$ in Eq. A3

$\begin{split}\frac{\partial\mathcal{L}_{\text{total}}}{\partial{\bm{\mathsfit{F}}}^{\text{stu}}_{i,j,k,l}}&=\frac{\partial\mathcal{L}_{\text{CE}}}{\partial{\bm{y}}^{\text{stu}}}\frac{\partial{\bm{y}}^{\text{stu}}}{\partial{\bm{\mathsfit{F}}}^{\text{stu}}_{i,j,k,l}}+\frac{\partial\mathcal{L}_{\text{BKD}}}{\partial\hat{{\bm{\mathsfit{F}}}}_{i,j,k,l}^{\text{stu}}}\frac{\partial\hat{{\bm{\mathsfit{F}}}}_{i,j,k,l}^{\text{stu}}}{\partial{\bm{\mathsfit{F}}}_{i,j,k,l}^{\text{stu}}}\\\
&=\frac{\partial\mathcal{L}_{\text{CE}}}{\partial{\bm{y}}^{\text{stu}}}\hat{\Theta}^{\prime}\left({\bm{\mathsfit{F}}}^{\text{stu}}_{i,j,k,l},\alpha\right)+\delta\end{split}$
(A3)

As depicted in Fig. A1, $\delta$ in Eq. A3 generated by
$\mathcal{L}_{\text{BKD}}$ serves as a correction term which refines the
surrogate gradient for more precise estimation in learning-based SNN training.
The GradCAM visualization in Fig. 2(a) further collaborates that the
introduction of $\mathcal{L}_{\text{BKD}}$ is beneficial for surrogate
gradient estimation.

##  A5 Firing Rate of Q, K, V.

We plot Q, K, V and attention firing rate curve of Sformer-8-768 on ImageNet,
with different methods in Fig. A3. The introduction of distillation increases
the firing rate of Q, K, V and attention compared with method without
distillation (denoted as w/o KD). Compared with logits-based distillation
(denoted as $+\mathcal{L}_{\text{LD}}$), our blurred knowledge distillation
(denoted as $+\mathcal{L}_{\text{BKD}}$) increases the firing rate by a
smaller margin while achieving higher accuracy improvement (+0.60%). Compared
with method without distillation, mixed distillation (denoted as
$+\mathcal{L}_{\text{MD}}$) trades an acceptable firing rate increase for the
maximized accuracy boost (+2.29%). The increment of firing rate leads to
higher energy consumption which is verified in Tab. 5(b). However, with the
significant boost on accuracy (+2.29%), such energy consumption increase
(+2.24%) is acceptable.

##  A6 Energy Consumption Calculation.

We follow previous works [29, 65, 25] to calculate the energy consumption of
our BKDSNN and tabulate the results in Tab. 5(b). Calculating the theoretical
energy consumption requires to count the synaptic operations
$\operatorname{SOPS}$ as follows:

$\operatorname{SOPS}(l)=r_{\text{input}}\times T\times\operatorname{FLOPS}(l)$
(A4)

where $l$ is a block/layer in corresponding models, $r_{\text{input}}$ is the
firing rate of the input spike train of the block/layer, $T$ is the simulation
time-step of spike neuron. $\operatorname{FLOPS}(l)$ refers to the floating
point operations of $l$ block/layer, which is the number of multiply-and-
accumulate (MAC) operations. $\operatorname{SOPS}(l)$ refers to the spike-
based accumulate (AC) operations of $l$ block/layer. Then, we follow the
assumption in [65] that both MAC and AC operations are implemented on the 45nm
hardware, where $E_{\text{MAC}}=4.6\textrm{pJ}$ and
$E_{\text{AC}}=0.9\textrm{pJ}$. The theoretical energy consumption $E$ is
defined as:

$E=E_{\text{MAC}}\times\operatorname{FLOPS}(1)+E_{\text{AC}}\times\displaystyle\sum_{i=2}^{l}\operatorname{SOPS}(i)$
(A5)

where $\operatorname{FLOPS}(1)$ refers to the first layer to encode static RGB
images into the form of spikes.

(a) Firing rate curve of Q in Sformer-8-768 with multiple methods on ImageNet.

(b) Firing rate curve of K in Sformer-8-768 with multiple methods on ImageNet.

(c) Firing rate curve of V in Sformer-8-768 with multiple methods on ImageNet.

(d) Firing rate curve of attention in Sformer-8-768 with multiple methods on
ImageNet.

Figure A3: Q, K, V and attention firing rate curve of Sformer-8-768 on
ImageNet with different methods. The introduction of distillation increases
the firing rate of Q, K, V and attention compared with method without
distillation (denoted as w/o KD). Compared with logits-based distillation
(denoted as $+\mathcal{L}_{\text{LD}}$), our blurred knowledge distillation
(denoted as $+\mathcal{L}_{\text{BKD}}$) increases firing rate by a smaller
margin while achieving higher accuracy improvement (+0.60%). Compared with
method without distillation, mixed distillation (denoted as
$+\mathcal{L}_{\text{MD}}$) trades an acceptable firing rate increase for the
maximized accuracy boost (+2.29%).

##  A7 Experiments on Multiple Neuromorphic Datasets

Table A4: Experiments on Multiple Neuromorphic Datasets.

Method | DVS-Gesture | NCaltech-101 | N-CARS | DVSLip
---|---|---|---|---
Acc | T | Acc | T | Acc | T | Acc | T
Sformer+CML | 98.6 | 16 | 69.91 | 16 | 99.31 | 16 | 54.93 | 15
Sformer+CML | 96.3 | 4 | 69.12 | 4 | 98.42 | 4 | 49.84 | 10
KDSNN | 96.4 | 4 | 69.16 | 4 | 98.51 | 4 | 51.06 | 10
LaSNN | 96.7 | 4 | 69.21 | 4 | 98.44 | 4 | 50.82 | 10
| BKDSNN
---
(ours)
| 97.8
---
(+1.1)
4 |  | 69.68
---
(+0.47)
4 |  | 99.02
---
(+0.51)
4 |  | 52.96
---
(+1.90)
10

As tabulated in Tab. A4, we extend our BKDSNN on datasets of DVS-Gesture [1],
NCaltech-101 [40], N-CARS [48] and DVSLip [51], BKDSNN achieves state-of-the-
arts results on all datasets above, which further verifies the practical
utility of BKDSNN on neuromorphic datasets.

##  A8 Experiments on Object Detection and Semantic Segmentation

Table A5: Experiments with M-Sformer (Meta-SpikeFormer) on object detection
and semantic segmentation. PoolFormer serves as corresponding teacher ANN.
$\dagger$: Reproduce result from the source code of original work.

Method | Object Detection | Semantic Segmentation
---|---|---
P(M) | T | mAP(%) | P(M) | T | MIoU(%)
PoolFormer | 40.6 | 1 | 60.1 | 34.6 | 1 | 42.0
M-Sformer | 75.0 | 1 | 51.2 | 58.9 | 4 | 34.2†
| BKDSNN
---
75.0 | 1 |  | 52.3(+1.1)
---
58.9 | 4 |  | 35.5(+1.3)
---

We extend BKDSNN on object detection and semantic segmentation tasks in Tab.
A5 with Meta-SpikeFormer [57], our BKDSNN surpasses Meta-SpikeFormer on both
object detection and semantic segmentation tasks, which further verifies the
effectiveness of BKDSNN.