# Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for
Event-Based Vision

Alexander Kugele*111Bosch Center for Artificial Intelligence, 71272 Renningen,
GermanyCorresponding author ,222Bio-Inspired Circuits and Systems (BICS) Lab,
Zernike Inst Adv Mat, University of Groningen, Nijenborgh 4, NL-9747 AG
Groningen, Netherlands.
<EMAIL_ADDRESS>
0000-0002-7482-8973  Thomas Pfeil11footnotemark: 1
<EMAIL_ADDRESS>Michael Pfeiffer11footnotemark: 1
<EMAIL_ADDRESS>
0000-0001-7159-3622  Elisabetta Chicca22footnotemark: 2 ,333Groningen
Cognitive Systems and Materials Center (CogniGron), University of Groningen,
Nijenborgh 4, NL-9747 AG Groningen, Netherlands
<EMAIL_ADDRESS>
0000-0002-5518-8990

###### Abstract

Event-based vision sensors encode local pixel-wise brightness changes in
streams of events rather than full image frames and yield sparse, energy-
efficient encodings of scenes, in addition to low latency, high dynamic range,
and lack of motion blur. Recent progress in object recognition from event-
based sensors has come from conversions of successful deep neural network
architectures, which are trained with backpropagation. However, using these
approaches for event streams requires a transformation to a synchronous
paradigm, which not only loses computational efficiency, but also misses
opportunities to extract spatio-temporal features. In this article we propose
a hybrid architecture for end-to-end training of deep neural networks for
event-based pattern recognition and object detection, combining a spiking
neural network (SNN) backbone for efficient event-based feature extraction,
and a subsequent classical analog neural network (ANN) head to solve
synchronous classification and detection tasks. This is achieved by combining
standard backpropagation with surrogate gradient training to propagate
gradients inside the SNN layers. Hybrid SNN-ANNs can be trained without
additional conversion steps, and result in highly accurate networks that are
substantially more computationally efficient than their ANN counterparts. We
demonstrate results on event-based classification and object detection
datasets, in which only the architecture of the ANN heads need to be adapted
to the tasks, and no conversion of the event-based input is necessary. Since
ANNs and SNNs require different hardware paradigms to maximize their
efficiency, we envision that SNN backbone and ANN head can be executed on
different processing units, and thus analyze the necessary bandwidth to
communicate between the two parts. Hybrid networks are promising architectures
to further advance machine learning approaches for event-based vision, without
having to compromise on efficiency.

##  1 Introduction

Figure 1:  Hybrid SNN-ANN models consist of an SNN backbone to compute
features directly from event camera outputs which are accumulated and
processed by an ANN head for classification and object detection. Using
sparse, binary communication (dots) instead of dense tensors (arrows) to
process events enables efficient inference. SNN and ANN can be on completely
different devices.

Event-based vision sensors address the increasing need for fast and energy-
efficient visual perception [1, 2, 3, 4, 5], and have enabled new use cases
such as high-speed navigation [6], gesture recognition [7], visual odometry
and SLAM [8, 9, 10]. These sensors excel at very low latency and high dynamic
range, while their event-based encoding creates a sparse spatio-temporal
representation of dynamic scenes. Every event indicates the position, precise
time, and polarity of a local brightness change.

In conventional frame-based computer vision deep learning-based methods have
led to vastly improved performance in object classification and detection, so
it is natural to expect a boost in performance also from applying deep neural
networks to event-based vision. However, such an approach has to overcome the
incompatibility of machine learning algorithms developed for a synchronous
processing paradigm, and the sparse, asynchronous nature of event-based
inputs. Recent successful approaches for processing event data have therefore
relied on early conversions of events into filtered representations that are
more suitable to apply standard machine learning methods [11, 12, 13].
Biologically inspired spiking neural networks (SNNs) in principle do not
require any conversion of event data and can process data from event-based
sensors with minimal preprocessing.

However, high performing SNNs rely on conversion from standard deep networks
[14, 15], thereby losing the opportunity to work directly with precisely timed
events. Other approaches like evolutionary methods [16] or local learning
rules like STDP [17, 18] are not yet competitive in performance.

Here we describe a hybrid approach that allows end-to-end training of neural
networks for event-based object recognition and detection. It combines sparse
spike-based processing of events in early layers with off-the-shelf ANN layers
to process the sparse, abstract features (see Fig. 1 for an illustration).
This is made possible by combining standard backpropagation with recently
developed surrogate gradient methods to train deep SNNs [19, 20, 21, 22, 23,
24]. The advantage of the hybrid approach is that early layers can operate in
the extremely efficient event-based computing paradigm, which can run on
special purpose hardware implementations [25, 26, 27, 28, 29, 30]. The hybrid
approach is also optimized for running SNN parts and conventional neural
networks on separate pieces of hardware by minimizing the necessary bandwidth
for communication. In our experiments we demonstrate that the hybrid SNN-ANN
approach yields very competitive accuracy at significantly reduced
computational costs, and is thus ideally suited for embedded perception
applications that can exploit the advantages of the event-based sensor
frontend.

Our main contributions are as follows:

  * •

We propose a novel hybrid architecture that efficiently integrates information
over time without needing to transform input events into other
representations.

  * •

We propose the first truly end-to-end training scheme for hybrid SNN-ANN
architectures on event camera datasets for classification and object
detection.

  * •

We investigate how to reduce communication bandwidths for efficient hardware
implementations that run SNNs and ANNs on separate chips.

  * •

We analyze how transfer learning of SNN layers increases the accuracy of our
proposed architecture.

##  2 Related Work

A variety of low level representations for event-based vision data have been
explored: The HOTS method [31] defines a time surface, i.e., a two-dimensional
representation of an event stream by convolving a kernel over the event
stream. This method was improved in [12] by choosing an exponentially decaying
kernel and adding local memory for increased efficiency. In [32], a more
general take on event stream processing is proposed, utilizing general kernel
functions that can be learned and project the event stream to different
representations. Notably, using a kernel on event camera data and aggregating
the result is the same as using a spiking neural network layer. Our approach
allows learning a more general low level representation by using a deep SNN
with an exponentially decaying kernel, compared to only one layer in [32] and
learnable weights compared to [31, 12].

Conversion approaches such as [14, 15] transform trained ANNs into SNNs for
inference, and so far have set accuracy benchmarks for deep SNNs. Conversion
methods train on image frames and do not utilize the membrane potential or
delays to integrate information over time. In [33] networks are unrolled over
multiple time steps, which allows training on event camera datasets. However,
temporal integration is only encoded in the structure of the underlying ANN,
but not in the dynamics of spiking neurons. In their formulation, the
efficiency of the SNN is limited by the rate coding of the neurons, which is
potentially more inefficient than encodings learned via end-to-end training in
our hybrid SNN-ANN approach. In addition, conversion methods typically do not
optimize for energy-efficiency.

SNN training with variants of backpropagation has been demonstrated in [20,
22, 23], albeit on simpler architectures (e.g., only one fully-connected layer
in [20]) and, in general, without delays during simulation. A mixed approach
is used in [21], first training and converting an ANN and then training the
converted SNN. Our approach uses synaptic delays and skip connections,
exploring how complex ANN architectures translate to SNN architectures. The
closest architecture to ours is from [34], which is trained from scratch to
predict optical flow. Their U-Net with an SNN encoder transmits information
from all SNN layers, leading to a high bandwidth from SNN to ANN. We improve
on this by using only a single layer to transmit information from SNN to ANN,
and extend to applications in classification and object detection tasks.

Conventional ANN architectures are used in [35] to solve the task of image
reconstruction from events with a recurrent U-Net, and they subsequently show
that classification and object detection are possible on the reconstructed
images. No SNN layers are used in this case, resulting in a computationally
expensive network and the need to preprocess the event camera data. Faster and
more efficient training and inference for object detection is presented in
[13], who propose a recurrent convolutional architecture that does not need to
explicitly reconstruct frames. As the sparsity of the event stream is not
utilized, it is expected that gains in energy-efficiency are possible with SNN
approaches.

##  3 Methods

Figure 2:  Training a hybrid network with a DenseNet backbone. a) Compact
representation. b) Network rolled out over time steps. The SNN backbone
computes sparse features from event camera data that are accumulated at time
intervals $\Delta T_{i}$. The ANN head integrates features from multiple
outputs (2, in this example) for a prediction (classification or object
detection). We use a time step of $1\text{\,}\mathrm{m}\mathrm{s}$ to
integrate information over small time scales. During inference, the SNN
backbone runs asynchronously without a time-stepped rollout, enabling
potential savings in computation. Layers with the same name share weights.

This section introduces the proposed hybrid SNN-ANN architecture and describes
training, inference and metrics we used to evaluate our method. Our hybrid
network consists of two parts: An SNN backbone that computes features from raw
event camera inputs, and an ANN head that infers labels or bounding boxes at
predefined times (see Fig. 2). The overall task is to find an efficient
mapping from a sequence of event camera data $E$ in a time interval $T$ to a
prediction $P$, which can be a label $l$ or a set of bounding boxes $B$. Our
approach consists of three stages: First, continuously in time, an
intermediate representation $I=S(E)$ is generated using the SNN backbone.
Second, this intermediate representation is accumulated at predefined points
in time. Third, when all accumulators are filled, the accumulated intermediate
representations are mapped via the ANN head $A$ to the final prediction
$P=A(\mathrm{Acc}(I))=A(\mathrm{Acc}(S(E)))$. The following sections describe
all three parts in more detail.

###  3.1 SNN backbone

Spiking neural networks (SNNs) are biologically inspired neural networks,
where each neuron has an internal state. Neurons communicate via spikes, i.e.
binary events in time, to signal the crossing of their firing thresholds. Upon
receiving a spike $i$, the synaptic current $I$ changes proportionally to the
synaptic weight $w$, which in turn leads to a change of the neuron’s internal
state $V$. Because of the binary and sparse communication, these networks can
be significantly more energy-efficient than dense matrix multiplications in
conventional ANNs (see also [36]).

The task of the SNN backbone $S$ is to map a sequence of raw event camera
inputs $E$ into a compressed, sparse, and abstract intermediate representation
$I=S(E)$ in an energy-efficient way. More concretely, the input is a stream of
events $e_{i}=(t_{i},x_{i},y_{i},p_{i})$, representing the time $t_{i}$ and
polarity $p_{i}$ of an input event at location $(x_{i},y_{i})$. Polarity is a
binary $\\{-1,1\\}$ signal that encodes if the brightness change is positive
(brighter) or negative (darker). This stream is processed by the SNN without
further preprocessing. In our implementation, the first layer has two input
channels representing the two polarities. In contrast to previous work such as
[34] and [33], the input events are never transformed into voxel grids or rate
averages, but directly processed by the SNN to compute the intermediate
representation $S(E)$. The spiking neuron model we use is the leaky integrate-
and-fire model [37], simulated using the forward Euler method,

$\displaystyle I_{i}$ $\displaystyle=\alpha
I_{i-1}+\textstyle\sum_{j}w_{j}S_{j}$ (1) $\displaystyle V_{i}$
$\displaystyle=\beta V_{i-1}+I_{i}$ (2) $\displaystyle S_{i}$
$\displaystyle=\Theta(V_{i}-V_{\mathrm{th}})$ (3)

where $V$ is the membrane potential, $I$ is the presynaptic potential,
$S_{j,i}$ are the binary input and output spikes, $V_{\mathrm{th}}$ is the
threshold voltage, and $t_{d}=t_{i}-t_{i-1}$ is an update step. The membrane
leakage $\beta$ and the synaptic leakage $\alpha$ are given as
$\exp{(-t_{d}/\tau_{\mathrm{mem}})}$ and $\exp{(-t_{d}/\tau_{\mathrm{syn}})}$,
respectively. To simulate the membrane reset, we add a term to Eq. 2 which
implements the reset-by-subtraction [14] mechanism,

$V_{i}=V_{i}-V_{\mathrm{th}}S_{i-1}~{}~{}~{}.$ (4)

The threshold is always initialized as $V_{\mathrm{th}}=1$ and trained jointly
with the weights (see appendix for details).

We simulate our network by unrolling it in time with a fixed time step
$t_{d}$. Figure 2b shows the training graph for a rollout of 7 time steps. Our
simulation allows choosing arbitrary delays for the connections of different
layers, which determines how information is processed in time. Inspired by
recent advances, we choose to implement streaming rollouts [38] in all our
simulations. This means that each connection has a delay of one time step, in
accordance with the minimum delay of large-scale simulators for neuroscience
[39]. This allows integrating temporal information via delayed connections, in
addition to the internal state.

The SNN backbone $S(E)$ outputs a set of sequences of spikes in predefined
time intervals $T_{\mathrm{out},i}$: $I=(e_{j})_{t_{j}\in
T_{\mathrm{out},i}}$. An example is shown in the dashed box of Fig. 2. Details
about the backbones used can be found in Section 3.8. The figure shows an
unrolled DenseNet backbone with two blocks (SC0 to SP0 and SP0 to SP1,
respectively), where two output intervals are defined as $\Delta
T_{0}=[t_{2},T_{0}]$ and $\Delta T_{1}=[t_{5},T_{1}]$. This structure is used
during training, where a time-stepped simulator approximates the continuous-
time SNN. During inference, the SNN backbone runs asynchronously, enabling
savings in computation.

###  3.2 Accumulator

Our model connects the sparse, continuous representation of the SNN with the
dense, time-stepped input of the ANN with an accumulator layer for each output
interval $\Delta T_{i}$ (Acc in Fig. 2). The task of this layer is to
transform the sparse data to a dense tensor. We choose the simple approach of
summing all spikes in each time interval $\Delta T_{i}$ to get a dense tensor
with the feature map shape $(c_{\mathrm{f}},h_{\mathrm{f}},w_{\mathrm{f}})$.

###  3.3 ANN head

The ANN head processes the accumulated representations from the accumulators
to predict classes or bounding boxes. The general structure of the ANN head
can be described with three parameters: the number of SNN outputs
$n_{\mathrm{out}}$, the number of stacks $n_{\mathrm{s}}$ and the number of
blocks per stack $n_{\mathrm{l}}$. The exemplary graph in Fig. 2 has
$n_{\mathrm{out}}=2$, $n_{\mathrm{s}}=2$ and $n_{\mathrm{l}}=2$. Having
multiple outputs and stacks allows to increase the temporal receptive field,
i.e., the time interval the ANN uses for its predictions (for details see
appendix). All blocks with the same name share their weights to reduce the
number of parameters. The number of blocks can be different for each stack. In
most experiments we use two stacks with 1 and 3 blocks, respectively. The
dense representation for each $\Delta T_{i}$ is then further processed by each
stack, where results are summed at the end of a stack and used as input for
the next stack. Each block in a stack consists of batch normalization
$\mathrm{BN}$, convolution $\mathrm{C}$, dropout $\mathrm{Drop}$,
$\mathrm{ReLU}$ and pooling $\mathrm{P}$, e.g.,
$\mathrm{B0}(x)=\mathrm{P}(\mathrm{ReLU}(\mathrm{Drop}(\mathrm{C}(\mathrm{BN}(x)))))$.
We use a convolutional layer with kernel size 2, stride 2, learnable weights
and $\mathrm{ReLU}$ activation as pooling layer. Whenever we use dropout in
conjunction with weight sharing, we also share the dropout mask. In the case
of classification, a linear layer is attached to the last block in the last
stack (see Section 3.6). For object detection, an SSD head is attached to
selected blocks in the last stack (see Section 3.7).

###  3.4 Energy-efficiency

We use the same metric as [14], i.e., we count the number of operations of our
network. Due to the sparse processing and binary activations in the SNN, the
number of operations is given by the synaptic operations, i.e., the sum of all
postsynaptic current updates. For ANN layers, we count the number of multiply-
add operations, as the information is processed with dense arrays. For this
metric, it is assumed that both SNN and ANN are run on dedicated hardware.
Then the total energy is proportional to the number of operations plus a
constant offset. We discuss benefits, drawbacks and alternatives to this
metric in the appendix. To regularize the number of spikes we utilize an
$\mathrm{L}_{1}$ loss on all activations of the SNN backbone

$L_{\mathrm{s}}=\sum_{l,b,i,x,y}\frac{\lambda_{s}}{LBTW_{l}H_{l}}|S_{l,i,b,x,y}|.$
(5)

with a scaling factor $\lambda_{s}$, the number of layers $L$, the batch size
$B$, total simulation time $T$, and width $W_{l}$ and height $H_{l}$ of the
respective layer. This also reduces the bandwidth, as discussed in Section
3.5.

###  3.5 Bandwidth

As we expect that special hardware could be used to execute at least the SNN
backbone during inference, we want to minimize the bandwidth between SNN and
ANN to avoid latency issues. We design our architectures such that only the
last layer is connected to the ANN and use $\mathrm{L}_{1}$ loss on the
activations to regularize the number of spikes. This is in contrast to [34],
where each layer has to be propagated to the ANN. We report all bandwidths in
MegaEvents per second ($\mathrm{M}\mathrm{E}\mathrm{v}\mathrm{/}\mathrm{s}$)
and MegaBytes per second ($\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$). One
event equals
$6.125\text{\,}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}$, because we
assume it consists of 32 bit time + 16 bit spatial coordinates + 1 bit
polarity.

###  3.6 Classification

To classify, we attach a single linear layer to the last block in the last
stack of our hybrid network. We use the negative log-likelihood loss of the
output of this layer. Additionally, we use $\mathrm{L_{2}}$-regularization
(weight decay) with a factor of $0.001$. We use the Adam optimizer [40] with a
learning rate of $\eta=0.01$ and a learning rate schedule to divide it by 5 at
$20\text{\,}\%$, $80\text{\,}\%$ and $90\text{\,}\%$ of the total number of
epochs $n_{\mathrm{e}}=100$.

###  3.7 Object detection

We use the SSD (single shot detection) architecture [41], which consists of a
backbone feature extractor and multiple predictor heads. Features are
extracted from the backbone at different scales and for each set of features,
bounding boxes and associated classes are predicted, relative to predefined
prior boxes. In the appendix, we present a novel, general way to tune the
default prior boxes in a fast and efficient way. During inference, non-maximum
suppression is used on the output of all blocks to select non-overlapping
bounding boxes with a high confidence. The performance of the network is
measured with the VOC mean average precision (mAP) [42]. If not otherwise
denoted, we use the same learning hyperparameters as in Section 3.6 but a
smaller learning rate of $0.001$.

###  3.8 Backbone architectures

Figure 3:  The different backbones in compact representation with spiking
convolutional (SC) and pooling (SP) layers. a) VGG. b) DenseNet. c) DenseSep
(DenseNet with depthwise separable convolutions). Depthwise separable
convolutions consist of a depthwise convolution (dw) and a convolution with
kernel size 1x1. Shades of blue mark different blocks. All depicted networks
have 2 blocks and 2 layers per block. Multiple inputs are concatenated.

Three different architecture types are used in our experiments: VGG [43],
DenseNet [44] and DenseSep. A VGG network with $N_{\mathrm{b}}$ blocks and
$N_{\mathrm{l}}$ layers per block is a feed-forward neural network with
$N_{\mathrm{b}}\cdot N_{\mathrm{l}}$ layers, where each output channel is
given by $g\cdot l$ with $l$ the layer index (starting from 1). The DenseNet
structure consists of $N_{\mathrm{b}}$ blocks, where each block consists of
$N_{\mathrm{l}}$ connected layers per block. DenseSep is a combination of the
depthwise separable block of MobileNet [45] and the DenseNet structure (see
Fig. 3).

##  4 Results

###  4.1 Classification on N-MNIST

Figure 4:  a) Accuracy on the N-MNIST test set vs. number of operations for
different architectures. Hybrid SNN-ANN architectures (green) overcome the
efficiency limit of conversion-based architectures (blue diamond) with only a
minor drop in accuracy. SNN-ANN architectures are more efficient than almost
all ANNs (purple). Compared to the SNN and ANN baseline, our hybrid networks
increase accuracy and energy-efficiency. b) Relative accuracy on the N-MNIST
test set vs. relative number of operations. The VGG backbone has the highest
gain in energy-efficiency, while losing significantly in accuracy. Other
backbones show a minor decrease with a significant gain in energy-efficiency.
We report the mean and error of the mean over 6 repetitions.

We train a hybrid SNN-ANN network for classification on the N-MNIST dataset
[46]. In N-MNIST, each MNIST digit is displayed on an LCD monitor while an
event camera performs three saccades within roughly
$300\text{\,}\mathrm{m}\mathrm{s}$. It contains the same $60000$ train and
$10000$ test samples as MNIST at a resolution of $34\times 34$ pixels. Here we
compare the performance of our hybrid network to networks where the SNN
backbone is replaced with ANN layers (ANN-ANN), two baselines and two
approaches from the literature: A conversion approach [33], where a trained
ANN is converted to a rate-coded SNN and an ANN that reconstructs frames from
events and classifies the frames [35]. The first baseline is a feedforward ANN
with the same structure as our VGG SNN-ANNs, but where all time steps are
concatenated and presented to the network as different channels. The second
baseline is an SNN of the same structure, where we accumulate the spikes of
the last linear layer and treat this sum as logits for classification. The SNN
is unrolled over the same number of time steps than the SNN-ANNs. We report
results for multiple network sizes and configurations to show that hybrid SNN-
ANNs generally perform better in terms of energy-efficiency and bandwidth
(detailed architecture parameters can be found in the appendix). Mean values
of accuracy and number of operations are reported together with the error of
the mean over 6 repetitions, using different initial seeds for the weights.
For one DenseSep network only 5 iterations are reported, because training of
one trial did not converge, resulting in a significant outlier compared to the
performance of all other networks. In Fig. 4a, we show the accuracy on the
N-MNIST test set vs. the number of operations for our architectures, baselines
and related approaches from the literature. Our hybrid networks (green, SNN)
reach similar accuracies to the ANNs (purple), while being consistently more
energy-efficient. The best hybrid architecture is a DenseNet with $g=16$,
$N_{\mathrm{l}}=3$ and $n_{\mathrm{out}}=1$. Compared to [33], it performs
slightly worse ($99.10\pm 0.09\text{\,}\%$ vs. $99.56\pm 0.01\text{\,}\%$) in
accuracy, but improves significantly on the number of operations ($94\pm
17\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$ vs. $460\pm
38\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$) despite having more
parameters ($504025$ vs. $319890$). The average bandwidth between SNN and ANN
is $0.250\pm 0.044\text{\,}\mathrm{M}\mathrm{E}\mathrm{v}\mathrm{/}\mathrm{s}$
for this architecture, or approximately
$1.53\text{\,}\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$. Our smallest DenseNet
with $64875$ parameters ($g=8$, $N_{\mathrm{l}}=2$, $n_{\mathrm{out}}=1$) is
even more efficient, needing only $15.9\pm
2.5\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$ at a bandwidth of
$0.144\pm 0.034\text{\,}\mathrm{M}\mathrm{E}\mathrm{v}\mathrm{/}\mathrm{s}$ to
reach $99.06\pm 0.03\text{\,}\%$ accuracy. Our results are mostly above [35]
in terms of accuracy, although our networks are much smaller (their networks
have over $10\text{\,}\mathrm{M}$ parameters). Due to the large parameter
size, we also estimate that we should be more energy-efficient, although the
authors do not provide any numbers in their publication.

In Fig. 4b, we plot the average per backbone over all architectures in Fig.
4a, relative to the averages of the ANN-ANN architectures. All hybrid SNN-ANNs
improve the number of operations significantly over ANN-ANN implementations
with at most a minor loss in accuracy. Hybrid DenseNets provide the best
accuracy-efficiency trade-off with an average improvement of about a factor of
56 in energy-efficiency while also increasing accuracy by approximately
$0.2\text{\,}\%$. For VGG architectures, the energy-efficiency is increased by
a factor of roughly 65 at a loss in accuracy of 0.4 between hybrid and ANN-ANN
architectures. Hybrid DenseSep architectures lose approximately 0.2 accuracy
points, but gain a factor of about 28 in energy-efficiency. We assume that the
DenseSep architectures perform the worst in comparison for two reasons: First,
they are already the most efficient ANN architectures, so improving is harder
than for less optimized architectures. Second, as the effective number of
layers is higher in this architecture compared to the other two, optimizing
becomes more difficult and gradient deficiencies (vanishing, exploding, errors
of surrogate gradient) accumulate more. Hyperparameter optimization (learning
rate, dropout rates, regularization factors for weights and activations) was
not performed for each network separately, but only once for a DenseNet with
$N_{\mathrm{l}}=2,g=16,n_{\mathrm{out}}=1$ on a validation set that consisted
of $10\text{\,}\%$ of the training data, i.e., 6000 samples.

###  4.2 Classification on N-CARS

N-CARS is a binary car vs. background classification task, where each sample
has a length of $100\text{\,}\mathrm{m}\mathrm{s}$. We train the same networks
as in Section 4.1 with a growth factor $g=16$ and compare to the same
baselines and two results from the literature. We see the same trend in
energy-efficiency improvements over ANNs, with factors ranging from 15
(DenseSep) to 110 (VGG). Relative accuracy decreases significantly for
DenseSeps by 1.5 points, but is 0.75 points higher for VGG. These results,
together with the results in Fig. 4b suggest, that the more efficient an
architecture already is, the less can be gained from training an equivalent
hybrid SNN, although the gains of at least a factor of 15 in energy-efficiency
is still significant. In comparison to our ANN and SNN baselines, our SNN-ANNs
perform better in terms of accuracy and number of operations. Two out of four
SNN runs could not go beyond chance level and were therefore excluded from the
validation. In conclusion, using hybrid SNN-ANNs saves energy compared to SNNs
and ANNs of similar structure. In this experiment, our architecture is not
competitive to state-of-the-art [33] in accuracy, but has a similar energy
demand with approximately double the number of parameters. Detailed results
and figures can be found in the appendix.

###  4.3 Object detection on shapes_translation

The shapes_translation dataset [8] contains frames, events, IMU measurements
and calibration information of a scene of different shapes pinned to a wall.
As training data for event-based vision object detection is scarce [47], we
labelled bounding boxes for each of the $1356$ images and $10$ shapes,
resulting in $9707$ ground truth bounding boxes. A detailed description and an
example of the dataset can be found in the appendix. We provide two train/test
splits: shapes_translation-90 where $90\text{\,}\%$ of the data is randomly
assigned to the train set and a more difficult split shapes_translation-30,
where only $30\text{\,}\%$ of the data is used in the train set. In section
Section 4.4 and Section 4.5, we present the results on
shapes_translation-30/90.

###  4.4 Results on shapes_translation-90

Figure 5:  Mean average precision (mAP) over number of operations for
shapes_translation-30/90 (a/b, mean over 4 trials). The architectures with the
best results on the N-MNIST training are either trained from scratch (s) or
initialized with the weights from the training (pretrained, p). Pretrained
architectures improve over random initialization (VGG, DenseSep) or are on par
with it (DenseNet).

For all backbones, we take the architecture parameters from the best network
in the N-MNIST task. See the appendix for details. We want to compare hybrid
networks trained from scratch with networks where the SNN backbone is
initialized with network weights trained on N-MNIST. This allows investigating
the effect of transfer learning during training. Results are shown in Fig. 5.
The networks with pretrained weights always converge to higher mAP than their
non-pretrained counterpart. The DenseNet and DenseSep backbones perform better
than VGG. This is in agreement with the results for classical ANNs, where
DenseNet architectures outperform VGG on image-based datasets like ImageNet.
Our best network is a pretrained SNN-ANN DenseSep with a mean average
precision of $87.37\pm 0.51\text{\,}\%$, $1398.9\pm
2.3\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$ operations and a
bandwidth of $11.0\text{\,}\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$. A
comparable ANN backbone would require a bandwidth of
$1210\text{\,}\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$. A regular SSD
architecture with the same backbone as our VGG network, where all time steps
are concatenated over the channels outperforms our networks in terms of mAP,
but also needs significantly more operations. We report the detailed results
in the appendix.

###  4.5 Results on shapes_translation-30

We do the same evaluation as in Section 4.4, but with a
$30\text{/}70\text{\,}\%$ training/test data split (Fig. 5). The mAP is higher
for networks with pretrained weights for DenseSep and VGG and on par with
DenseNet. As with shapes_translation-90, the backbones with skip connections
perform better than the VGG backbone. One of the four VGG experiments did not
fully converge, explaining the high standard deviation. Our best network is an
SNN-ANN DenseNet trained from scratch with $2790\pm
50\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$ operations, a mean
average precision of $82.0\pm 1.0\text{\,}\%$, and a bandwidth of
$2.68\text{\,}\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$. A comparable ANN
backbone would have a bandwidth of
$864\text{\,}\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$. As in Section 4.4 the
regular SSD architecture is better in mAP but worse in the number of
operations. We report the detailed results in the appendix.

##  5 Conclusion

In this paper, we introduced a novel hybrid SNN-ANN architecture for efficient
classification and object detection on event data that can be trained end-to-
end with backpropagation. Hybrid networks can overcome the energy-efficiency
limit of rate-coded converted SNN architectures, improving by up to a factor
of 10. In comparison to similar ANN architectures, they improve by a factor of
15 to 110 on energy-efficiency with only a minor loss in accuracy. Their
flexible design allows efficient custom hardware implementations for both the
SNN and ANN part, while minimizing the required communication bandwidth. Our
SNN-ANN networks learn general features of event camera data, that can be
utilized to boost the object detection performance on a transfer learning
task.

We expect that the generality of the features can be improved when learning on
larger and more diverse datasets. Our work is particularly suited for datasets
where temporal integration happens on a short time interval, but struggles for
longer time intervals, e.g., multiple seconds due to the immense number of
rollout steps needed. Recent advances in deep learning, particularly C-RBP
[48] can potentially help to overcome this by using recurrent backpropagation
that has a constant memory complexity with number of steps (compared to
backpropagation, where the memory-complexity is linear). This also would
ensure that our networks converge to a fixed point over time, potentially
making predictions more stable. More work on surrogate gradients and methods
to stabilize training can further help to increase both energy-efficiency and
performance of hybrid networks. Our work is a first step towards ever more
powerful event-based perception architectures that are going to challenge the
performance of image-based deep learning methods.

## Acknowledgments

The authors would like to acknowledge the financial support of the CogniGron
research center and the Ubbo Emmius Funds (Univ. of Groningen). Furthermore,
this publication has received funding from the European Union’s Horizon 2020
research innovation programme under grant agreement 732642 (ULPEC project).

## References

  * Lichtsteiner et al. [2008] P. Lichtsteiner, C. Posch, and T. Delbruck.  A 128$\times$128 120 dB 15$\mu$s latency asynchronous temporal contrast vision sensor.  _IEEE Journal of Solid-State Circuits_ , 43(2):566–576, Feb 2008.  ISSN 0018-9200.  doi:10.1109/JSSC.2007.914337.
  * Posch et al. [2011] C. Posch, D. Matolin, and R. Wohlgenannt.  A qVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS.  _IEEE Journal of Solid-State Circuits_ , 46(1):259–275, 2011.  doi:10.1109/JSSC.2010.2085952.
  * Liu and Delbruck [2010] Shih-Chii Liu and Tobi Delbruck.  Neuromorphic sensory systems.  _Current opinion in neurobiology_ , 20(3):288–295, 2010.
  * Serrano-Gotarredona and Linares-Barranco [2013] T. Serrano-Gotarredona and B. Linares-Barranco.  A 128$\,\times$ 128 1.5% contrast sensitivity 0.9% FPN 3 µs latency 4 mW asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers.  _IEEE Journal of Solid-State Circuits_ , 48(3):827–838, 2013.  doi:10.1109/JSSC.2012.2230553.
  * Gallego et al. [2020] Guillermo Gallego, Tobi Delbruck, Garrick Michael Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jorg Conradt, Kostas Daniilidis, and et al.  Event-based vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , page 1–1, 2020.  ISSN 1939-3539.  doi:10.1109/tpami.2020.3008413.  URL http://dx.doi.org/10.1109/TPAMI.2020.3008413.
  * Falanga et al. [2020] Davide Falanga, Kevin Kleber, and Davide Scaramuzza.  Dynamic obstacle avoidance for quadrotors with event cameras.  _Science Robotics_ , 5(40), 2020.  doi:10.1126/scirobotics.aaz9712.
  * Maro et al. [2020] Jean-Matthieu Maro, Sio-Hoi Ieng, and Ryad Benosman.  Event-based gesture recognition with dynamic background suppression using smartphone computational capabilities.  _Frontiers in neuroscience_ , 14:275, 2020.
  * Mueggler et al. [2017] E. Mueggler, H. Rebecq, G. Gallego, T. Delbruck, and D. Scaramuzza.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM.  36:142–149, Feb. 2017.
  * Vidal et al. [2018] A. R. Vidal, H. Rebecq, T. Horstschaefer, and D. Scaramuzza.  Ultimate SLAM? combining events, images, and IMU for robust visual SLAM in HDR and high-speed scenarios.  _IEEE Robotics and Automation Letters_ , 3(2):994–1001, 2018.  doi:10.1109/LRA.2018.2793357.
  * Zhu et al. [2019] Dekai Zhu, Zhongcong Xu, Jinhu Dong, Canbo Ye, Yinbai Hu, Hang Su, Zhengfa Liu, and Guang Chen.  Neuromorphic visual odometry system for intelligent vehicle application with bio-inspired vision sensor.  In _2019 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , pages 2225–2232. IEEE, 2019.
  * Amir et al. [2017] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al.  A low power, fully event-based gesture recognition system.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7243–7252, 2017.
  * Sironi et al. [2018] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman.  HATS: Histograms of averaged time surfaces for robust event-based object classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * Perot et al. [2020] Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi.  Learning to detect objects with a 1 megapixel event camera.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 16639–16652. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/c213877427b46fa96cff6c39e837ccee-Paper.pdf.
  * Rueckauer et al. [2017] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu.  Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.  _Frontiers in Neuroscience_ , 11:682, 2017.  ISSN 1662-453X.  doi:10.3389/fnins.2017.00682.
  * Sengupta et al. [2019] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy.  Going deeper in spiking neural networks: VGG and residual architectures.  _Frontiers in Neuroscience_ , 13:95, 2019.  ISSN 1662-453X.  doi:10.3389/fnins.2019.00095.
  * Opez-Vázquez et al. [2019] G Opez-Vázquez, Manuel Ornelas-Rodriguez, Andres Espinal, Jorge Soria-Alcaraz, Alfonso Rojas-Dominguez, Héctor Soberanes, Juan Carpio Valadez, and Horacio Rostro-Gonzalez.  Evolutionary spiking neural networks for solving supervised classification problems.  _Computational Intelligence and Neuroscience_ , 2019:13, 03 2019.  doi:10.1155/2019/4182639.
  * Kheradpisheh et al. [2018] Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, Simon J Thorpe, and Timothée Masquelier.  STDP-based spiking deep convolutional neural networks for object recognition.  _Neural Networks_ , 99:56–67, 2018.
  * Barbier et al. [2020] Thomas Barbier, Céline Teulière, and Jochen Triesch.  Unsupervised learning of spatio-temporal receptive fields from an event-based vision sensor.  In Igor Farkaš, Paolo Masulli, and Stefan Wermter, editors, _Artificial Neural Networks and Machine Learning – ICANN 2020_ , pages 622–633. Springer International Publishing, 2020.
  * Neftci et al. [2019] E. O. Neftci, H. Mostafa, and F. Zenke.  Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.  _IEEE Signal Processing Magazine_ , 36(6):51–63, 2019.
  * Lee et al. [2016] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer.  Training deep spiking neural networks using backpropagation.  _Frontiers in Neuroscience_ , 10:508, 2016.  ISSN 1662-453X.  doi:10.3389/fnins.2016.00508.
  * Rathi and Roy [2020] Nitin Rathi and Kaushik Roy.  DIET-SNN: Direct input encoding with leakage and threshold optimization in deep spiking neural networks.  _arXiv 2008.03658_ , 2020.
  * Shrestha and Orchard [2018] Sumit Bam Shrestha and Garrick Orchard.  SLAYER: Spike layer error reassignment in time.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems 31_ , pages 1412–1421. Curran Associates, Inc., 2018.  URL http://papers.nips.cc/paper/7415-slayer-spike-layer-error-reassignment-in-time.pdf.
  * Wu et al. [2019] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi.  Direct training of spiking neural networks: Faster, larger, better.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019\.
  * Cramer et al. [2021] Benjamin Cramer, Sebastian Billaudelle, Simeon Kanya, Aron Leibfried, Andreas Grübl, Vitali Karasenko, Christian Pehle, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, Johannes Schemmel, and Friedemann Zenke.  Surrogate gradients for analog neuromorphic computing.  _arXiv 2006.07239_ , 2021.
  * Schemmel et al. [2010] Johannes Schemmel, Daniel Brüderle, Andreas Grübl, Matthias Hock, Karlheinz Meier, and Sebastian Millner.  A wafer-scale neuromorphic hardware system for large-scale neural modeling.  _Proceedings of 2010 IEEE International Symposium on Circuits and Systems_ , pages 1947–1950, 2010.
  * Furber et al. [2013] S. B. Furber, D. R. Lester, L. A. Plana, J. D. Garside, E. Painkras, S. Temple, and A. D. Brown.  Overview of the SpiNNaker system architecture.  _IEEE Transactions on Computers_ , 62(12):2454–2467, Dec 2013.  doi:10.1109/TC.2012.142.
  * Merolla et al. [2014] Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven K. Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron D. Flickner, William P. Risk, Rajit Manohar, and Dharmendra S. Modha.  A million spiking-neuron integrated circuit with a scalable communication network and interface.  _Science_ , 345(6197):668–673, 2014.  ISSN 0036-8075.  doi:10.1126/science.1254642.  URL https://science.sciencemag.org/content/345/6197/668.
  * Qiao et al. [2015] Ning Qiao, Hesham Mostafa, Federico Corradi, Marc Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri.  A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses.  _Frontiers in Neuroscience_ , 9:141, 2015.  ISSN 1662-453X.  doi:10.3389/fnins.2015.00141.
  * Davies et al. [2018] M. Davies, N. Srinivasa, T. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, Y. Liao, C. Lin, A. Lines, R. Liu, D. Mathaikutty, S. McCoy, A. Paul, J. Tse, G. Venkataramanan, Y. Weng, A. Wild, Y. Yang, and H. Wang.  Loihi: A neuromorphic manycore processor with on-chip learning.  _IEEE Micro_ , 38(1):82–99, January 2018.  ISSN 0272-1732.  doi:10.1109/MM.2018.112130359.
  * Billaudelle et al. [2020] S. Billaudelle, Y. Stradmann, K. Schreiber, B. Cramer, A. Baumbach, D. Dold, J. Göltz, A. F. Kungl, T. C. Wunderlich, A. Hartel, E. Müller, O. Breitwieser, C. Mauch, M. Kleider, A. Grübl, D. Stöckel, C. Pehle, A. Heimbrecht, P. Spilger, G. Kiene, V. Karasenko, W. Senn, M. A. Petrovici, J. Schemmel, and K. Meier.  Versatile emulation of spiking neural networks on an accelerated neuromorphic substrate.  In _2020 IEEE International Symposium on Circuits and Systems (ISCAS)_ , pages 1–5, 2020.  doi:10.1109/ISCAS45731.2020.9180741.
  * Lagorce et al. [2017] X. Lagorce, G. Orchard, F. Galluppi, B. E. Shi, and R. B. Benosman.  HOTS: A hierarchy of event-based time-surfaces for pattern recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(7):1346–1359, 2017.  doi:10.1109/TPAMI.2016.2574707.
  * Gehrig et al. [2019] Daniel Gehrig, Antonio Loquercio, Konstantinos G. Derpanis, and Davide Scaramuzza.  End-to-end learning of representations for asynchronous event-based data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Kugele et al. [2020] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca.  Efficient processing of spatio-temporal data streams with spiking neural networks.  _Frontiers in Neuroscience_ , 14:439, 2020.  ISSN 1662-453X.  doi:10.3389/fnins.2020.00439.
  * Lee et al. [2020] Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy.  Spike-FlowNet: Event-based optical flow estimation with energy-efficient hybrid neural networks.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision – ECCV 2020_ , pages 366–382, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-58526-6.
  * Rebecq et al. [2019] Henri Rebecq, Rene Ranftl, Vladlen Koltun, and Davide Scaramuzza.  Events-to-video: Bringing modern computer vision to event cameras.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Rieke [1999] F. Rieke.  _Spikes: Exploring the Neural Code_.  Bradford book. MIT Press, 1999.  ISBN 9780262681087.
  * Gerstner et al. [2014] Wulfram Gerstner, Werner M. Kistler, Richard Naud, and Liam Paninski.  Neuronal dynamics: From single neurons to networks and models of cognition.  2014\.
  * Fischer et al. [2018] Volker Fischer, Jan Koehler, and Thomas Pfeil.  The streaming rollout of deep networks - towards fully model-parallel execution.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems 31_ , pages 4039–4050. Curran Associates, Inc., 2018.  URL http://papers.nips.cc/paper/7659-the-streaming-rollout-of-deep-networks-towards-fully-model-parallel-execution.pdf.
  * Stimberg et al. [2019] Marcel Stimberg, Romain Brette, and Dan FM Goodman.  Brian 2, an intuitive and efficient neural simulator.  _eLife_ , 8:e47314, August 2019.  ISSN 2050-084X.  doi:10.7554/eLife.47314.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2015.
  * Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: Single shot multibox detector.  In _European Conference on Computer Vision_ , page 21–37. Springer International Publishing, 2016.  ISBN 9783319464480.
  * Everingham et al. [2009] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  _International Journal of Computer Vision_ , 88:303–308, September 2009.  URL https://www.microsoft.com/en-us/research/publication/the-pascal-visual-object-classes-voc-challenge/.  Printed version publication date: June 2010.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv 1409.1556_ , 2015.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Howard et al. [2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  MobileNets: Efficient convolutional neural networks for mobile vision applications.  _arXiv 1704.04861_ , 2017.
  * Orchard et al. [2015] Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades.  _Frontiers in Neuroscience_ , 9:437, 2015.  ISSN 1662-453X.  doi:10.3389/fnins.2015.00437.
  * Gehrig et al. [2020] Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carrio, and Davide Scaramuzza.  Video to events: Recycling video datasets for event cameras.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Linsley et al. [2020] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre.  Stable and expressive recurrent vision models.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 10456–10467. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/766d856ef1a6b02f93d894415e6bfa0e-Paper.pdf.
  * Neil et al. [2016] Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu.  Learning to be efficient: Algorithms for training low-latency, low-compute deep spiking neural networks.  In _Proceedings of the 31st Annual ACM Symposium on Applied Computing_ , SAC ’16, page 293–298, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450337397.  doi:10.1145/2851613.2851724.  URL https://doi.org/10.1145/2851613.2851724.
  * Bose et al. [2020] Sumon Kumar Bose, Jyotibdha Acharya, and Arindam Basu.  Is my neural network neuromorphic? taxonomy, recent trends and future directions in neuromorphic engineering, 2020.
  * Davidson and Furber [2021] Simon Davidson and Steve B. Furber.  Comparison of artificial and spiking neural networks on digital hardware.  _Frontiers in Neuroscience_ , 15:345, 2021.  ISSN 1662-453X.  doi:10.3389/fnins.2021.651141.  URL https://www.frontiersin.org/article/10.3389/fnins.2021.651141.
  * Thakur et al. [2018] Chetan Singh Thakur, Jamal Lottier Molin, Gert Cauwenberghs, Giacomo Indiveri, Kundan Kumar, Ning Qiao, Johannes Schemmel, Runchun Wang, Elisabetta Chicca, Jennifer Olson Hasler, Jae-sun Seo, Shimeng Yu, Yu Cao, André van Schaik, and Ralph Etienne-Cummings.  Large-scale neuromorphic spiking array processors: A quest to mimic the brain.  _Frontiers in Neuroscience_ , 12:891, 2018.  ISSN 1662-453X.  doi:10.3389/fnins.2018.00891.  URL https://www.frontiersin.org/article/10.3389/fnins.2018.00891.
  * Zenke and Ganguli [2018] Friedemann Zenke and Surya Ganguli.  SuperSpike: Supervised learning in multilayer spiking neural networks.  _Neural Computation_ , 30(6):1514–1541, 2018\.  doi:10.1162/neco_a_01086.  URL https://doi.org/10.1162/neco_a_01086.  PMID: 29652587.
  * González-Burgos et al. [2000] Guillermo González-Burgos, German Barrionuevo, and David A. Lewis.  Horizontal Synaptic Connections in Monkey Prefrontal Cortex: An In Vitro Electrophysiological Study.  _Cerebral Cortex_ , 10(1):82–92, 01 2000.  ISSN 1047-3211.  doi:10.1093/cercor/10.1.82.
  * Sandin and Nilsson [2020] Fredrik Sandin and Mattias Nilsson.  Synaptic delays for insect-inspired temporal feature detection in dynamic neuromorphic processors.  _Frontiers in Neuroscience_ , 14:150, 2020.  ISSN 1662-453X.  doi:10.3389/fnins.2020.00150.
  * Luo et al. [2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper/2016/file/c8067ad1937f728f51288b3eb986afaa-Paper.pdf.
  * Brandli et al. [2014] C. Brandli, R. Berner, M. Yang, S. Liu, and T. Delbruck.  A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor.  _IEEE Journal of Solid-State Circuits_ , 49(10):2333–2341, 2014.  doi:10.1109/JSSC.2014.2342715.

Appendix

##  Appendix A Metrics for Energy-Efficiency

We chose to use the metric of [14] to compare the number of operations between
ANNs and SNNs as a metric to compare the energy-efficiency of these two types
of neural networks. This is a simplification, because in reality the actual
energy consumption also depends on the specific hardware that the networks run
on. However, this is a widely adopted metric because it is simple to calculate
and allows for an approximate comparison [49, 14, 50, 33, 51]. In [52], an
overview over different neuromorphic architectures is given that shows a range
of $2.8$-$360~{}\mathrm{pJ}$ per synaptic operation, i.e., a factor of 100
between different hardware implementations is possible for SNNs. Also, this
value is dependent on even more factors, like networks size (how much of the
hardware is used by the network) and the base energy needed to operate the
hardware. Comparing actual energy measurements also suffers from the fact that
you have to ensure to utilize the chips in the same way, which is barely
possible. In [21], a factor of approximately 5 is introduced to correct the
differences in energy consumption of an operation of SNN and ANN, which would
significantly increase the energy-efficiency measured in our experiments to
factors of 140 to 325 between SNNs and ANNs. Although our results would look
more impressive, we don’t think that this is a fair comparison based on the
discussion in this section. In [51], a more thorough analysis of the number of
operations is provided that takes all operations on digital hardware into
account and shows that rate-coded SNNs are not more efficient than ANNs with
floating point activations. As the number of operations of our networks is
well below rate-coded SNNs, we also save energy compared to our ANNs.

##  Appendix B Classification on N-CARS

Figure 6:  a) Accuracy on the N-CARS test set vs. number of operations for
different architectures. Hybrid SNN-ANN architectures are more efficient than
all ANNs (purple), the most efficient ones are on par in efficiency with the
conversion approach. Compared to the SNN and ANN baseline, our hybrid networks
increase accuracy and energy-efficiency. b) Relative accuracy on the N-CARS
test set vs. relative number of operations for the three different backbones.
As with N-MNIST, VGG backbones gain the most, followed by DenseNet and
DenseSep. We report the mean and error of the mean over 4 repetitions.

We report the results of our experiments in Fig. 6. Our hybrid networks
improve in energy-efficiency compared to our ANN-ANNs, as well as to the
baselines. Furthermore, our hybrid networks show a significantly higher
accuracy compared to the baselines. They are on par with [35], while using
smaller networks. They are, however, not yet competitive compared to the
conversion approach of [33].

##  Appendix C More information about backbones

In this section, we list more details about the different backbones. Each
layer in a block is connected to all layers in the same block before it and
has $g$ output channels. Multiple inputs are concatenated to preserve the
temporal structure of the data. To match the structure of the DenseNet, we
employ the same hyperparameters also for the other architectures. All pooling
layers are depthwise convolutional layers with kernel size and stride 2,
because then the layer can adapt to the input activation during training. With
regular pooling layers, it can happen that the activation after the pooling
layer dies out. Compared to the DenseNet architecture, adding depthwise
separable convolutions together with 1x1 convolutions should reduce the number
of operations of the ANN significantly. We want to evaluate if this still
holds when using these building blocks in our SNN backbone.

##  Appendix D Surrogate gradients

Figure 7:  Surrogate functions (top) and their corresponding analytical
gradients (right). A tradeoff between accurately approximating the function
$\Theta(V-V_{\mathrm{th}})$ and allowing non-zero gradients has to be made.
The range of the x-axis has been chosen to be approximately in the range of
typical values in our experiments and $V_{\mathrm{th}}=1$.

Surrogate gradients are used to replace the gradient of the activation
function $\frac{\partial\Theta(x)}{\partial x}$ of the spiking network in Eq.
3, because the gradient of this function is 0 everywhere, except for at 0,
where it is ill-defined. Recently, multiple surrogate gradients have been
proposed. The goal is to find a function that has a well-defined gradient,
ideally close to 1 to avoid vanishing and exploding gradient issues, while the
approximate function is as close as possible to the activation function Eq. 3.

A natural approach is to replace the $\Theta$-function with a sigmoid $\sigma$
plus an additional parameter $a$ that controls the steepness (orange and green
line in Fig. 7)

$\sigma(x,a)=\frac{1}{1+\exp{(-ax)}}.$ (6)

The steepness then controls the tradeoff between the range in which the
gradient is non-zero (Fig. 7 bottom) and the accuracy of the approximation
(Fig. 7 top). This approach has two shortcomings: First, it has to be taken
care that the gradient does not over- or underflow, which happens easily
because of the exponential function. Second, the gradient is only close to one
for a very small range of inputs, leading to suboptimal convergence. The
authors of [53] propose to use the fast sigmoid

$\sigma_{f}(x,a)=(\frac{ax}{1+|ax|}+1)/2,$ (7)

a numerically more stable and faster to calculate approximation of the sigmoid
(red and purple lines in Fig. 7). The gradient of the fast sigmoid,

$\frac{\partial\sigma_{f}(x,a)}{\partial x}=\frac{1}{(|ax|+1)^{2}}$ (8)

is in the range $[0,1]$ with the maximum at $0$ for arbitrary steepness. A
similarly numerically stable, but less accurate approximation can be obtained
by using the first or second order Taylor expansion of the sigmoid (yellow
line in Fig. 7)

$T(x,a)=(1/2+ax/4-(ax)^{3}/48).$ (9)

It is also possible to use a linear function

$L(x,a)=ax/2+0.5$ (10)

or $\mathrm{ReLU}$ to approximate the sigmoid (brown and pink lines in Fig.
7). All of the above have to be constraint to a range (typically
$[0,V_{\mathrm{th}}]$ or $[-V_{\mathrm{th}},V_{\mathrm{th}}]$, leading to
constant gradients in a fixed range. The functions themselves are only crude
approximations of the original activation function. The DIET-SNN surrogate
[21] takes a mixed approach and defines only the gradient as a triangular
function with the maximum at $0$, leading to linear gradients symmetrically
around the threshold.

Figure 8:  Comparison of accuracy vs. operations for different surrogate
gradients from Fig. 7. Results are recorded with a hybrid DenseNet ($g=16$,
$n_{\mathrm{out}}=2$, $N_{\mathrm{l}}=2$, $N_{\mathrm{b}}=2$) on a $1800$
samples, 4 class subset of the N-MNIST dataset. We report validation accuracy
and number of operations, each relative to the best achieved result, i.e., the
best architecture is in the top-left. The validation accuracy was chosen as to
prevent optimizing hyperparameters on the test set.

Comparing experimental results for different surrogate gradient methods from
the literature is difficult, because every paper uses different network
architectures. On the other hand, treating the surrogate function as a
hyperparameter to sweep over in conjunction with the architecture
hyperparameters like growth factor or training hyperparameters like the
learning rate would lead to an intractable search space. Therefore, we chose
to evaluate the surrogate functions first, by choosing an average-sized
architecture (hybrid DenseNet with $g=16$, $n_{\mathrm{out}}=2$,
$N_{\mathrm{l}}=2$, $N_{\mathrm{b}}=2$) and train on a $1800$ samples, 4 class
subset of the N-MNIST training dataset (see Fig. 8). The best result in terms
of accuracy is obtained with the ’diet’ gradient from [21], using a scale
factor of 1. A sigmoid with $a=10$ leads to a lower number of operations, but
we decided that this does not compensate for the obtained loss in accuracy.

###  D.1 Training with surrogate gradients

Our goal is to train the hybrid network end-to-end with backpropagation. We
use a surrogate gradient [19] for Eq. 3, to overcome the problem that the
analytical gradient is 0 almost everywhere in SNN layers. Recently, multiple
surrogate gradients have been proposed for SNN training. We provide an
experiment with different surrogate gradients on a small subsample of the
N-MNIST dataset in the appendix, with the result that the DIET surrogate
gradient proposed in [21] gives the best results. It is defined as

$\displaystyle\dfrac{\mathrm{d}\Theta(x)}{\mathrm{d}x}\approx\begin{cases}\lambda(V_{\mathrm{th}}-|x|),&\text{if
}-V_{\mathrm{th}}\leq x\leq V_{\mathrm{th}}\\\ 0,&\text{otherwise}\end{cases}$
(11)

with a scaling factor $\lambda$.

Similar to [21], we train the threshold voltage and the membrane leakage with
backpropagation. Additionally, in the same way, we train the synaptic leakage.
For the leakages and the threshold, we use one global parameter per layer,
because [21] indicated that using one parameter for each neuron in each layer
does not lead to better results.

##  Appendix E Detailed description of N-MNIST networks

Our experiments compared VGG, DenseNet and DenseSep backbones, using different
growth factors $g\in\\{8,16\\}$, layers per block $N_{\mathrm{l}}\in\\{2,3\\}$
and number of output intervals $n_{\mathrm{out}}\in\\{1,8\\}$. All neuron
parameters (threshold and both leakages) are trained jointly with the weights.
They are initialized as $V_{\mathrm{th}}=1.$,
$\tau_{\mathrm{mem}}=$20\text{\,}\mathrm{m}\mathrm{s}$$ and
$\tau_{\mathrm{syn}}=$1\text{\times}{10}^{-8}\text{\,}\mathrm{m}\mathrm{s}$$.
The networks are rolled out for
$n_{\mathrm{t}}=$150\text{\,}\mathrm{m}\mathrm{s}$$, with outputs at
$\\{24,42,60,78,96,114,132,150\\}~{}$\mathrm{m}\mathrm{s}$$ for
$n_{\mathrm{out}}=8$. The ANN head always consists of two stacks, where the
first stack consists of one block and the second stack has three blocks. A
linear classification layer of shape (80, 10) is attached to the flattened
output of the last layer in the last stack.

##  Appendix F Synaptic delays

Synaptic delays are present in biological systems [54] as well as on dedicated
hardware [55], and therefore have to be taken into account during training. We
implement delays by evaluating the maximum delay of out-going connections for
each layer and initialize a ring buffer of this size to store the spiking
activations. With this approach, we save memory compared to simpler
approaches, like storing the activations at the input of each layer or storing
all activations over all time steps. This allows training deeper networks, as
the total available memory on GPU is the limiting factor for the network size.
As a side effect, this introduces an additional way to integrate information
in time, see Appendix G and [38].

##  Appendix G Timescales

Our architecture has three intrinsic timescales: On the lowest level, the
membrane potential integrates information over multiple time steps before
emitting a spike. This timescale is determined by the factors $\beta$ and
$V_{\mathrm{th}}$ in Eq. 2 and Eq. 3. As a rule of thumb, we observe that the
weights in combination with the threshold are often such that only 1-3 spikes
are needed to excite a neuron. On the second level, there are the delayed
connections in the SNN, transporting information forward in time. As each
connection has a delay of at least 1, the delay from input to output without
skip connection is equal to the depth of the SNN. Finally, the SNN can have
multiple output intervals $T_{\mathrm{out},i}$, which enables the ANN to
integrate information over the timescale of multiples of the depth of the
network, if the output intervals are separated by enough time steps.

##  Appendix H Optimizing SSD prior boxes

In the SSD architecture [41], each set of features is processed by two
convolutional layers that predict the bounding box coordinates relative to
pre-defined prior boxes and the class for each prior box. The first
convolutional layer maps each spatial location to $n$ prior boxes and predicts
a class score for each class and box. The second convolutional layer has $4n$
output channels and regresses the $x_{0}$, $y_{0}$, $x_{1}$ and $y_{1}$
coordinates to match each prior box as close as possible to the closest ground
truth box. Only prior boxes with an intersection-over-union (IoU) larger than
a predefined threshold with a bounding box are matched with that box, which
naturally leads to a lot of bounding boxes being assigned to the background
class. Therefore, the loss is only calculated at a specific negative-positive
ratio of all prior boxes, typically set to values aroung $3$. Because the
number of positive training samples is highly influenced by the quality of the
prior boxes, tuning the hyperparameters that define the prior boxes is of
particular significance.

In the following, we are presenting a novel way to optimize the prior box
hyperparameters of the SSD architecture without training the neural network.
The SSD architecture as described in Section 3.7 has multiple Detector Heads,
whereas each head is defined by its prior boxes. For each head $i$, the set of
priors is described by the minimum size $s_{\mathrm{min,i}}$, maximum size
$s_{\mathrm{max,i}}$ and aspect ratio $r_{\mathrm{i}}$ such that four prior
boxes are defined with shapes: $(s_{\mathrm{min,i}},s_{\mathrm{min,i}})$,
$(\sqrt{s_{\mathrm{min,i}}s_{\mathrm{max,i}}},\sqrt{s_{\mathrm{min,i}}s_{\mathrm{max,i}}})$,
$(s_{\mathrm{min,i}}\cdot
r_{\mathrm{i}},\frac{s_{\mathrm{min,i}}}{r_{\mathrm{i}}})$,
$(\frac{s_{\mathrm{min,i}}}{r_{\mathrm{i}}},s_{\mathrm{min,i}}\cdot
r_{\mathrm{i}})$. We argue, that good prior boxes are defined by their overlap
with bounding boxes during training. As the ground truth training bounding
boxes are available, we therefore do not have to optimize a loss defined by
the network predictions, but can define and optimize a function that describes
the relationship between prior boxes and ground truth boxes. This function
should inlcude two objectives: First, it is important to have a high
intersection over union (IoU) between ground truth boxes and prior boxes. This
can in practice be measured by the mean IoU over all combinations between
training and ground truth bounding boxes. However, this can lead to all prior
boxes fitting well to only a subset of the training boxes. Therefore we
instead take the mean over the maximum IoU between a prior box and all ground
truth boxes. Second, to enforce that each prior box is matched to at least one
ground truth box, we also count the number of prior boxes with a maximum IoU
over the detection threshold $t_{\mathrm{d}}$. We combine both objectives with
a $2:1$ weight as

$\displaystyle
L_{\mathrm{priors}}=(\mu_{\mathrm{IoU}}+2N_{\mathrm{priors>t_{\mathrm{d}}}}/N_{\mathrm{priors}})/3$
(12)

We chose to do the simplest approach to optimize this function and just
sampled 2000 random points. Generally, more intricate approaches like Bayesian
Optimization would be possible as well, but the result from random search was
already sufficient for our purposes. To sample suitable points, we also
consider the receptive field of each feature map: Each prior box should be
smaller than their respective receptive field to ensure we only predict
bounding boxes based on information that is available to the predictor. We
incorporate this by calculating the theoretical receptive field [56] for each
feature map and setting the bounds for the maximum sizes to $90\text{\,}\%$ of
the receptive field. Additionally, we want to enforce that the minimum sizes
grow monotonically with the receptive field and always be smaller than the
maximum sizes. We achieve this by using the maximum size of head $i-1$ as
minimum size of head $i$. For the results in all following sections, we
optimized the hyperparameters by selecting four feature maps from our network
with sizes $\\{60,30,15,7\\}$ and receptive fields $\\{34,38,62,110\\}$. We
sample 2000 random points and compare the prior boxes with the smallest loss
to the default prior boxes of the SSD architecture in Fig. 9. The backbone is
a DenseNet SNN architecture with the same parameters as the best model in
Section 4.1. Our mean average precision increases by 0.07 (averaged over two
runs) compared to the default prior boxes. Additionally, the whole
optimization process takes about $1.67\text{\,}\mathrm{h}$ for 2000 random
samples, while training the network once for 50 epochs already takes
$7.8\text{\,}\mathrm{h}$. Our best result, ordered as $s_{\mathrm{min,i}}$,
$s_{\mathrm{max,i}}$, $r_{\mathrm{i}}$ is $\\{10,27,32,49\\}$,
$\\{27,32,49,64\\}$, $\\{2.0,1.07,1.84,2.69\\})$. In Fig. 10, we compare the
prior boxes used for training before and after optimization. We find that
smaller boxes, particularly at the edge, are matched much better. There are
less prior boxes for bigger ground truth boxes. However, there are still
enough well-matching boxes and therefore should not affect training
performance. Having less matches can even help to speed up inference, because
less boxes have to be compared during non-maximum suppression.

Figure 9:  Top: Mean average precision on shapes_translation-90 for the
default prior boxes and the prior boxes found after our optimization. Bottom:
Time to train the network once and time to do the whole optimization with 2000
samples. Our method finds better prior boxes at a fraction of the time of one
training. Usually, multiple training steps are needed to optimize the prior
box hyperparameters.

Figure 10:  Left: Prior boxes _before_ optimization. Right: Prior boxes
_after_ our optimization. Each image shows all prior boxes that are matched to
a ground truth target box during training. After optimization, the prior boxes
for smaller targets fit a lot better, particularly at the edge of the frame.
At the same time, there are less prior boxes for bigger targets, which should
be beneficial for post-processing methods like non-maximum suppression,
because less predictions have to be filtered.

Pseudo-code for the function to be optimized (minimized) is shown in
LABEL:lst:priorBoxes. This function can be straight-forwardly optimized using
e.g., Bayesian optimization.

⬇

1def priors_loss(hyperparameters):

2 # feature maps are defined by backbone

3 feature_map_sizes = get_feature_map_sizes(backbone)

4 min_sizes, max_sizes, aspect_ratios = hyperparameters

5 # generally non-differentiable

6 prior_boxes = define_priors(min_sizes, max_sizes, aspect_ratios,
feature_map_sizes)

7 # normalized between 0 and 1

8 ious_mean = calculate_mean_over_max_ious_per_prior(prior_boxes,
ground_truth_boxes)

9 n_priors_bigger_thresh = calculate_priors_bigger_than_thresh(prior_boxes,
ground_truth_boxes)

10 return -(ious_mean + 2*n_priors_bigger_thresh / len(prior_boxes)) / 3

Listing 1: Pseudocode to optimize prior boxes

##  Appendix I The shapes_translation dataset

Figure 11:  Three samples from the dataset with our ground truth bounding
boxes. Top: Grayscale frames used for labeling. Bottom: Events accumulated
over $1\text{\,}\mathrm{m}\mathrm{s}$.

The scene was recorded by moving a DAVIS240C [57] around with varying speeds
at a resolution of $180\times 240$ pixels for a duration of
$59.75\text{\,}\mathrm{s}$. Three samples of the dataset are shown in Fig. 11.
While the shapes cannot overlap and there are no occlusions in the dataset, it
is still non-trivial to predict the bounding boxes from the accumulated event
data, as their appearance changes due to the camera movement. For example, the
triangle is barely visible on the left , while for the middle sample, it can
be clearly identified. Particularly difficult are those samples where shapes
move in or out of the picture, as in the sample in the middle or right. The
paper on which the shapes are printed on (middle), as well as the background
(middle and right) add additional noise to the data. The task is to predict
bounding boxes in a predefined time interval from the asynchronous event data.
As the whole dataset is small, we decided to provide two train/test splits
together with the bounding boxes: An easier split, called
shapes_translation-90 where $90\text{\,}\%$ of the data is randomly assigned
to the train set and the other part is used as test set, and a more difficult
split shapes_translation-30, where only $30\text{\,}\%$ of the data is used in
the train set. We also considered splitting the sets in two time intervals,
such that the separation between train and test set is higher, but due to the
small size and the fact that only one scene is recorded, we decided this split
does not provide an additional benefit.

###  I.1 Object detection performance without finetuning

Figure 12:  Mean average precision over epochs for a hybrid network with
frozen SNN backbone on shapes_translation-30. The network with a pretrained
SNN backbone improves over random features without further training of the
backbone. Mean and error of the mean over two trials.

To investigate if the features learned for N-MNIST are general enough to be
helpful on other datasets, we train only the ANN head of a network with two
different weight initializations for the backbone. One is initialized with
random weights, while the other was pretrained with a different ANN head on
N-MNIST. It can be seen in Fig. 12 that the pretrained backbone reaches a
significantly higher mAP than the random network. The backbone learned
features on the N-MNIST dataset that are also useful for the
shapes_translation-90 task.

##  Appendix J shapes_translation Details

For all backbones, we take the architecture parameters from the best network
in the N-MNIST task. For VGG and DenseNet, this is a network with $g=16$,
$N_{\mathrm{l}}=2$, $N_{\mathrm{b}}=2$, while the DenseSep architecture has
$g=8$, $N_{\mathrm{l}}=3$, $N_{\mathrm{b}}=2$. For the ANN head, we choose a
network consisting of two stacks, where the first stack has 1 block and the
second stack has 3 blocks. SSD heads are attached after each block of the last
stack, and additionally after the first convolution of the last stack, to
enable a denser set of bounding boxes in space with a small spatial
resolution.

See Table 2 and Fig. 13 for results on shapes_translation-90. See Table 2 and
Fig. 14 for results on shapes_translation-30.

Figure 13:  Mean average precision (mAP) over the number of epochs for
shapes_translation-90. Networks trained from scratch are marked with (s),
pretrained networks with (p).  Figure 14:  Mean average precision (mAP) over
the number of epochs for shapes_translation-30. Networks trained from scratch
are marked with (s), pretrained networks with (p).

architecture | mAP | #ops $[\mathrm{MOps}]$ | #params | bw [$\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$]
---|---|---|---|---
SNN DenseNet (s) | $86.98\pm 0.69$ | $3000\pm 130$ | 744939 | 3.52
SNN DenseSep (s) | $85.57\pm 0.58$ | $1393\pm 1.3$ | 409553 | 2.48
SNN VGG (s) | $83.78\pm 0.44$ | $1361\pm 61$ | 336037 | 3.20
ANN DenseNet | $63.40\pm 0.55$ | $14690$ | 151040 | 864
ANN DenseSep | $64.54\pm 0.18$ | 7841 | 238344 | 1210
ANN VGG | $62.85\pm 0.44$ | 83064 | 336016 | 1380
SNN DenseNet (p) | $87.09\pm 0.20$ | $2679.0\pm 7.1$ | 744939 | 11.5
SNN DenseSep (p) | $87.37\pm 0.51$ | $1398.9\pm 2.3$ | 409553 | 11.0
SNN VGG (p) | $86.70\pm 0.36$ | $1611\pm 32$ | 336037 | 7.49

Table 1:  Results on shapes_translation-90 (4 trials each). We report the mean
and error of the mean. Networks trained from scratch are marked with (s),
pretrained networks with (p). #ops are the number of operations, #params the
number of parameters of the model and bw is the bandwidth between SNN backbone
and ANN head.

architecture | mAP | #ops $[\mathrm{MOps}]$ | #params | bw [$\mathrm{M}\mathrm{B}\mathrm{/}\mathrm{s}$]
---|---|---|---|---
SNN DenseNet (s) | $82.0\pm 1.0$ | $2790\pm 50$ | 744939 | 2.68
SNN DenseSep (s) | $79.4\pm 1.2$ | $1384.6\pm 1.0$ | 409553 | 2.22
SNN VGG (s) | $58\pm 19$ | $1230\pm 89$ | 336037 | 1.82
ANN DenseNet | $54.20\pm 0.64$ | $14690$ | 151040 | 864
ANN DenseSep | $56.78\pm 0.24$ | 7841 | 238344 | 1210
ANN VGG | $57.72\pm 0.36$ | 83064 | 336016 | 1380
SNN DenseNet (p) | $81.76\pm 0.53$ | $2696.5\pm 3.3$ | 744939 | 16.1
SNN DenseSep (p) | $81.74\pm 0.61$ | $1392.3\pm 1.0$ | 409553 | 10.7
SNN VGG (p) | $76.66\pm 0.58$ | $1303\pm 21$ | 336037 | 14.8

Table 2:  Results on shapes_translation-30 (4 trials each). We report the mean
and error of the mean. Networks trained from scratch are marked with (s),
pretrained networks with (p). #ops are the number of operations, #params the
number of parameters of the model and bw is the bandwidth between SNN backbone
and ANN head.

##  Appendix K Regularization for homeostasis

During training we face the problem that sometimes the network does not start
to train, because no spikes reach the last layer. As the surrogate gradient of
$\Theta(V-V_{\mathrm{th}})$ is only non-zero around the threshold, neurons
that are not spiking, also cannot learn and therefore, cannot learn to spike.
We could track this back to the weight initialisation, that sometimes leads to
a suboptimal inital state. We tried to tackle this problem by including
homeostasis during training, i.e., introducing a loss that increases activity

$L_{\mathrm{spikes}}=0.5(\sum_{tln}S_{tln}-r_{\mathrm{target}})^{2},$ (13)

with indices time step $t$, layer $l$ and neuron $n$. Experimentally, we did
not find an improvement. This can be explained by looking at the calculated
weight update for a single-layer network.

$\displaystyle\Delta
w_{ij}=-\eta(\sum_{t}S_{tj}-r_{\mathrm{target}})(\left.\frac{\partial\Theta}{\partial
x}\right|_{x=V_{i}-V_{\mathrm{th}}}\sum^{t}_{k=0}\beta^{k}I_{j-k}),$ (14)

i.e., the update is also always 0 and we cannot force the network to an
average target rate if the activation is too low in the beginning.

##  Appendix L Membrane potential initialization

We tried to utilize the initial membrane potential to increase training
accuracy, because by default, the initial membrane potentials $V_{0}$ are a
free parameter during training. In some works [33, 14] it is set to 0 for each
sample, which covers the case where the initial membrane potentials is reset
after each prediction. To be independent from the starting conditions, it is
possible to start with random values for all $V_{0}$. This can also be seen as
data augmentation. We tried to draw initial membrane potentials from a normal
distribution

$V_{0}\sim\mathcal{N}(V_{\mathrm{th}}/2,\,V_{\mathrm{th}}/10)$ (15)

for each sample during training and also experimented with testing with random
initialization vs. initialization from 0. We found in practice that often the
results on the validation set are worse in both cases and therefore decided to
stick with initialization to 0 for training and during validation and test
time.

##  Appendix M Bandwidth details

We introduce an additional $\mathrm{L}_{1}$ loss on the activations of the
output layer of the SNN,

$L_{\mathrm{out}}=\frac{\lambda_{\mathrm{out}}}{BTWH}\sum_{b,i,x,y}|S_{\mathrm{out},i,b,x,y}|$
(16)

with a scaling factor $\lambda_{\mathrm{out}}$, the batch size $B$, total
simulation time $T$, and width $W$ and height $H$ of the output layer. It is
important to notice that we do not need to take the absolute value, because
all activations are binary, i.e., positive by definition. For the same reason,
the loss is normalized to 1.

##  Appendix N Parameters for classification tasks

Tables 3, 4, 5 and 6 list all used architectures for the classification tasks.
SNN-ANNs have more parameters than ANN-ANNs because we also count the neuron
parameters $V_{\mathrm{th}}$, $\beta$, $\alpha$ that are shared across layers.

backbone | $g$ | $N_{\mathrm{l}}$ | $n_{\mathrm{out}}$ | SNN | $n_{params}$
---|---|---|---|---|---
densenet | 8 | 2 | 1 | False | 64854
densenet | 8 | 2 | 1 | True | 64875
densenet | 8 | 2 | 8 | False | 64854
densenet | 8 | 2 | 8 | True | 64875
densenet | 8 | 3 | 1 | False | 127014
densenet | 8 | 3 | 1 | True | 127041
densenet | 8 | 3 | 8 | False | 127014
densenet | 8 | 3 | 8 | True | 127041
densenet | 16 | 2 | 1 | False | 256414
densenet | 16 | 2 | 1 | True | 256435
densenet | 16 | 2 | 8 | False | 256414
densenet | 16 | 2 | 8 | True | 256435
densenet | 16 | 3 | 1 | False | 503998
densenet | 16 | 3 | 1 | True | 504025
densenet | 16 | 3 | 8 | False | 503998
densenet | 16 | 3 | 8 | True | 504025
Table 3:  All DenseNet architectures for the classification tasks.  backbone | $g$ | $N_{\mathrm{l}}$ | $n_{\mathrm{out}}$ | SNN | $n_{params}$
---|---|---|---|---|---
densenet_sep | 8 | 2 | 1 | False | 60454
densenet_sep | 8 | 2 | 1 | True | 60487
densenet_sep | 8 | 2 | 8 | False | 60454
densenet_sep | 8 | 2 | 8 | True | 60487
densenet_sep | 8 | 3 | 1 | False | 117774
densenet_sep | 8 | 3 | 1 | True | 117819
densenet_sep | 8 | 3 | 8 | False | 117774
densenet_sep | 8 | 3 | 8 | True | 117819
densenet_sep | 16 | 2 | 1 | False | 237374
densenet_sep | 16 | 2 | 1 | True | 237407
densenet_sep | 16 | 2 | 8 | False | 237374
densenet_sep | 16 | 2 | 8 | True | 237407
densenet_sep | 16 | 3 | 1 | False | 464014
densenet_sep | 16 | 3 | 1 | True | 464059
densenet_sep | 16 | 3 | 8 | False | 464014
densenet_sep | 16 | 3 | 8 | True | 464059
Table 4:  All DenseSep architectures for the classification tasks.  backbone | $g$ | $N_{\mathrm{l}}$ | $n_{\mathrm{out}}$ | SNN | $n_{params}$
---|---|---|---|---|---
vgg | 8 | 2 | 1 | False | 50174
vgg | 8 | 2 | 1 | True | 50195
vgg | 8 | 2 | 8 | False | 50174
vgg | 8 | 2 | 8 | True | 50195
vgg | 8 | 3 | 1 | False | 125582
vgg | 8 | 3 | 1 | True | 125609
vgg | 8 | 3 | 8 | False | 125582
vgg | 8 | 3 | 8 | True | 125609
vgg | 16 | 2 | 1 | False | 198254
vgg | 16 | 2 | 1 | True | 198275
vgg | 16 | 2 | 8 | False | 198254
vgg | 16 | 2 | 8 | True | 198275
vgg | 16 | 3 | 1 | False | 498830
vgg | 16 | 3 | 1 | True | 498857
vgg | 16 | 3 | 8 | False | 498830
vgg | 16 | 3 | 8 | True | 498857
Table 5:  All VGG architectures for the classification tasks.  backbone | $g$ | $N_{\mathrm{l}}$ | $n_{\mathrm{out}}$ | SNN | $n_{params}$
---|---|---|---|---|---
ffann | 8 | 2 | 1 | False | 711394
ffsnn | 8 | 2 | 1 | True | 691389
Table 6:  All baseline architectures for the classification tasks.