# Bibliography management: BibTeX

Overleaf  Jeevanshi Sharma
Department of Electrical Engineering
Aligarh Muslim University, Aligarh
<EMAIL_ADDRESS>

# Evaluating CNN with Oscillatory Activation Function

Overleaf  Jeevanshi Sharma
Department of Electrical Engineering
Aligarh Muslim University, Aligarh
<EMAIL_ADDRESS>

###### Abstract

The reason behind CNN’s capability to learn high-dimensional complex features
from the images is the non-linearity introduced by the activation function.
Several advanced activation functions have been discovered to improve the
training process of neural networks, as choosing an activation function is a
crucial step in the modeling. Recent research has proposed using an
oscillating activation function to solve classification problems inspired by
the human brain cortex. [1] This paper explores the performance of one of the
CNN architecture ALexNet on MNIST and CIFAR10 dataset using oscillatory
activation function (GCU) and some other commonly used activation functions
like ReLu, PReLu, Mish.

_K_ eywords Activation Function $\cdot$ Convolutional Neural Networks $\cdot$
Growing Cosine Unit $\cdot$ XOR Function

##  1 Introduction

The convolutional neural network is a type of feed-forward multi-layer neural
network, which is generally used for image datasets. It is composed of a
convolution layer (Conv), a down-sampling layer (or pooling layer), and a
full-connection layer (FC). The convolution layer is responsible for the
extraction of the different features from the input images, which gets
convoluted by several filters on the convolution layer. Then the feature is
blurred by the down-sampling layer. Finally, a set of eigenvectors is acquired
through a full connection layer. Activation functions are an important part of
CNN’s architecture and its performance. CNN’s have been successfully in
practically various fields [2] [3] [4] [5]. The general architecture of a
convolutional neural network is represented in Figure 1.

Figure 1: CNN Architecture

The practical implementation of a CNN model still demands improvement. Many
studies have been done on different image classification methods[6] [7],
design of adaptive learning rate [8], design of drop-out layer [9] [10], and
at the core research on better activation functions for mapping data better
[11] [12] [13]. Neural networks are non-linear. It is their non-linearity that
makes them powerful. As natural images have highly co-related pixel values,
this attribute can only be better exploited with the use of non-linear
functions. Linear activation functions make no use of the multi-layer CNNs.
[14]

Over the years, there has been a lot of significant development in CNN
architectures to mitigate problems regarding computational efficiency, error
rates, and other improvements. Architectures like LeNet-5 [15], AlexNet [16],
Inception [17], VCGNet [18], ResNet [19], etc. were put forward to deal with
issues of overfitting, issues with high-resolution images, etc. Past studies
have shown to achieve better results when these architectures are optimized
with different activation functions [20] and optimization algorithms. Along
with this, many variants of activation functions like ReLu and different
combinations of activation functions have been proposed to achieve better
results with these architectures. [21] [22] [23]. In particular, the use of
the ReLU like activation functions can result in faster training compared to
saturating sigmoidal type activation functions because these activation
functions do not saturate for a wider range of inputs and avoid the vanishing
gradient problem. The classification of activation functions is primarily
based on whether they can modify their shape during training. [24] Oscillatory
activation functions which are inspired by biological comprises of multiple
hyperplanes in their decision boundaries, which enables neurons to make more
complex decisions. In [25], the author suggests that deep neural networks with
oscillatory activation functions might partially bridge the gap present in
biological and artificial neural networks. One such function is GCU (Growing
Cosine Unit). [26] In contrast to the common perception of a single plane
decision boundary, the GCU has infinite evenly spaced solutions resulting in
infinite evenly spaced hyperplanes, which solves the XOR problem in neural
networks.

This study talks about the development of trainable activation function and
their performance on CNN architectures, AlexNet and ResNet.

##  2 Activation Functions

In a neural network, neurons map the weighted sum of their input to a single
output with the help of Activation Functions. The main purpose served by the
activation function is to introduce non-linearity in the network. The
important properties commonly shared by activation functions are non-
linearity, differentiability, continuous, bounded, and zero-centering. [27]
Non-linear functions are universal approximates and using a linear activation
function in a multi-layer network is the same as using a single-layer network.
[28] Continuously differentiable activation functions are desired when using
gradient-based optimization algorithms. Activation functions are preferred to
be centered at zero so that the gradients do not shift in a particular
direction for gradients to be unbiased the output of the activation function
must be symmetrical at zero so that the gradients do not shift in a particular
direction.

###  2.1 ReLU

ReLU [29] was primarily used to overcome the vanishing gradient problem. ReLU
is the most common activation function used for classification problems.

Figure 2: ReLU

###  2.2 PReLU

The disadvantage of ReLU is that it cannot adapt to sudden changes and zero
the negative inputs (dying ReLU problem). PReLU generalises the traditional
with a slope for negative inputs using a learnable parameter that leads to
adaptive constants which can increase accuracy. [30]

Figure 3: PReLU

###  2.3 Mish

Mish is an unbounded in positive direction and a bounded negative domain. Mish
has properties like non-monotonicity and smooth profile. [31]

Figure 4: Mish

###  2.4 GCU

Growing Cosine Unit was introduced in 2021 [32]. It is an oscillatory
activation function whose amplitude increases with increasing values. GCU has
infinite evenly spaced solutions resulting in infinite evenly spaced hyper-
planes, instead of a single plane decision boundary which resolves the long-
rooted XOR problem as for separating classes in XOR datasets two hyper-planes
are needed which required the use activation function with multiple zeros.[33]

Figure 5: GCU

##  3 Dataset and Methodology

###  3.1 Experimental Setup

In this following, a comparison of the AlexNet on CIFAR10 and MNIST with GCU
and other activation functions is presented. The SGD and Adam optimizers are
used on CIFAR and MNIST datasets respectively with sparse categorical
crossentropy loss function (softmax classification head). For both the
datasets, we experiment with the choice of activation functions at two
locations in AlexNet: one for all of the convolutional layer and the other for
the dense layers.

The results for CIFAR 10 are reported on 50 epchos and 40 epochs for MNIST.

##  4 Results

The performance of GCU with various activation functions are visualized and
presented in Table 1 and Table 2 for comparing the oscillating activation
functions with other prominently used non-linear activation fucntions (Mish,
Prelu).

Architecture | Convolutional Layer | Activation Dense Layer | Validation Accuracy | Test Accuracy | Val Loss | Loss
---|---|---|---|---|---|---
AlexNet | ReLU | ReLU | 0.9940 | 0.9974 | 0.0850 | 0.0145
AlexNet | GCU | ReLU | 0.9830 | 0.9791 | 0.0791 | 0.9830
AlexNet | PReLU | ReLU | 0.9945 | 0.9904 | 0.0469 | 0.0435
AlexNet | Mish | ReLU | 0.9940 | 0.9928 | 0.1190 | 0.0625

Table 1: Performance on MNIST

Architecture | Convolutional Layer | Activation Dense Layer | Validation Accuracy | Test Accuracy | Val Loss | Loss
---|---|---|---|---|---|---
AlexNet | ReLU | ReLU | 0.7011 | 0.9777 | 1.6989 | 0.0666
AlexNet | GCU | ReLU | 0.6036 | 0.8256 | 1.3953 | 0.4913
AlexNet | PReLU | ReLU | 0.7015 | 0.9765 | 1.6207 | 0.0722
AlexNet | Mish | ReLU | 0.7143 | 0.9802 | 1.6899 | 0.0591

Table 2: Performance on CIFAR 10

##  5 Conclusion

The activation function is an important part of a Convolutional Neural
Network. Many research have proposed multiple methods which can improve the
performance of CNNs. From the view of activation function, this paper explores
and experiment the use of oscillatory activation function (GCU) in AlexNet,
and compares it with PReLu and Mish. As Mish is more computationally heavy
than ReLU, we have only placed these function in the convolution layers and
ReLU for the deep connected layers. GCU have shown comparable performance on
both MNIST and CIFAR 10 dataset.

## References

  * [1] Albert Gidon, Timothy Adam Zolnik, Pawel Fidzinski, Felix Bolduan, Athanasia Papoutsi, Panayiota Poirazi, Martin Holtkamp, Imre Vida, and Matthew Evan Larkum.  Dendritic action potentials and computation in human layer 2/3 cortical neurons.  Science, 367(6473):83–87, 2020.
  * [2] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, page 160–167, New York, NY, USA, 2008. Association for Computing Machinery.
  * [3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60:84 – 90, 2012.
  * [4] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [6] Wartini Ng, Budiman Minasny, Maryam Montazerolghaem, Jose Padarian, Richard Ferguson, Scarlett Bailey, and Alex B McBratney.  Convolutional neural network for simultaneous prediction of several soil properties using visible/near-infrared, mid-infrared, and their combined spectra.  Geoderma, 352:251–267, 2019.
  * [7] Marios Savvides, Khoa Luu, Yutong Zheng, and Chenchen Zhu.  Methods and software for detecting objects in images using a multiscale fast region-based convolutional neural network, July 16 2019.  US Patent 10,354,362.
  * [8] Huizhen Zhao, Fuxian Liu, Han Zhang, and Zhibing Liang.  Research on a learning rate with energy index in deep learning.  Neural Networks, 110:225–231, 2019.
  * [9] Shaofeng Cai, Yao Shu, Gang Chen, Beng Chin Ooi, Wei Wang, and Meihui Zhang.  Effective and efficient dropout for deep convolutional neural networks.  arXiv preprint arXiv:1904.03392, 2019.
  * [10] Saihui Hou and Zilei Wang.  Weighted channel dropout for regularization of deep convolutional neural network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8425–8432, 2019.
  * [11] Gianluca Maguolo, Loris Nanni, and Stefano Ghidoni.  Ensemble of convolutional neural networks trained with different activation functions.  Expert Systems with Applications, 166:114048, 2021.
  * [12] Arun Kumar Dubey and Vanita Jain.  Comparative study of convolution neural network’s relu and leaky-relu activation functions.  In Applications of Computing, Automation and Wireless Systems in Electrical Engineering, pages 873–880. Springer, 2019.
  * [13] Aman Dureja and Payal Pahwa.  Analysis of non-linear activation functions for classification tasks using convolutional neural networks.  Recent Patents on Computer Science, 12(3):156–161, 2019.
  * [14] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural networks, 2(5):359–366, 1989.
  * [15] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [17] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [18] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [20] Franco Manessi and Alessandro Rozza.  Learning combinations of activation functions.  In 2018 24th international conference on pattern recognition (ICPR), pages 61–66. IEEE, 2018.
  * [21] Mainak Bandyopadhyay.  Multi-stack hybrid cnn with non-monotonic activation functions for hyperspectral satellite image classification.  Neural Computing and Applications, 33(21):14809–14822, 2021.
  * [22] Tongtong Jiang and Jinyong Cheng.  Target recognition based on cnn with leakyrelu and prelu activation functions.  In 2019 International Conference on Sensing, Diagnostics, Prognostics, and Control (SDPC), pages 718–722. IEEE, 2019.
  * [23] Zhiguan Huang, Xiaohao Du, Liangming Chen, Yuhe Li, Mei Liu, Yao Chou, and Long Jin.  Convolutional neural network based on complex networks for brain tumor image classification with a modified activation function.  IEEE Access, 8:89281–89290, 2020.
  * [24] Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete.  A survey on modern trainable activation functions.  Neural Networks, 138:14–32, 2021.
  * [25] Matthew Mithra Noel, Shubham Bharadwaj, Venkataraman Muthiah-Nakarajan, Praneet Dutta, and Geraldine Bessie Amali.  Biologically inspired oscillating activation functions can bridge the performance gap between biological and artificial neurons.  arXiv preprint arXiv:2111.04020, 2021.
  * [26] Mathew Mithra Noel, Advait Trivedi, Praneet Dutta, et al.  Growing cosine unit: A novel oscillatory activation function that can speedup training and reduce parameters in convolutional neural networks.  arXiv preprint arXiv:2108.12943, 2021.
  * [27] Leonid Datta.  A survey on activation functions and their relation with xavier and he normal initialization.  arXiv preprint arXiv:2004.06632, 2020.
  * [28] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural networks, 2(5):359–366, 1989.
  * [29] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [31] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [32] Mathew Mithra Noel, Advait Trivedi, Praneet Dutta, et al.  Growing cosine unit: A novel oscillatory activation function that can speedup training and reduce parameters in convolutional neural networks.  arXiv preprint arXiv:2108.12943, 2021.
  * [33] Ehsan Lotfi and M-R Akbarzadeh-T.  A novel single neuron perceptron with universal approximation and xor computation properties.  Computational intelligence and neuroscience, 2014, 2014.


# OpenSpike: An OpenRAM SNN Accelerator

Farhad Modaresi  Dept. of Electrical Engineering
Allameh Mohaddes Nouri University
Nur, Mazandaran, Iran
<EMAIL_ADDRESS>Matthew Guthaus  Dept. of Computer Science and
Engineering, UC Santa Cruz
Santa Cruz, CA, United States
<EMAIL_ADDRESS>Jason K. Eshraghian  Dept. of Electrical and Computer
Engineering, UC Santa Cruz
Santa Cruz, CA, United States
<EMAIL_ADDRESS>
###### Abstract

This paper presents a spiking neural network (SNN) accelerator made using
fully open-source EDA tools, process design kit (PDK), and memory macros
synthesized using OpenRAM. The chip is taped out in the 130 nm SkyWater
process and integrates over 1 million synaptic weights, and offers a
reprogrammable architecture. It operates at a clock speed of 40 MHz, a supply
of 1.8 V, uses a PicoRV32 core for control, and occupies an area of 33.3 mm2.
The throughput of the accelerator is 48,262 images per second with a wallclock
time of 20.72 $\mu$s, at 56.8 GOPS/W. The spiking neurons use hysteresis to
provide an adaptive threshold (i.e., a Schmitt trigger) which can reduce state
instability. This results in high performing SNNs across a range of benchmarks
that remain competitive with state-of-the-art, full precision SNNs. The design
is open sourced and available online: https://github.com/sfmth/OpenSpike

###### Index Terms:

ASIC, accelerator, open source, OpenRAM, spiking neural network

##  I Introduction

The open source community has enabled advances in deep learning (DL) to
proliferate over the past decade. Much of these advances have successfully
been ported to biologically plausible spiking neural networks (SNNs). SNNs
have been used to model brain function, and can perform DL using gradient-
based or Hebbian learning rules, often on a CUDA backend [1, 2, 3, 4]. To
achieve this, the training methods that have been popularized by DL (e.g.,
error backpropagation) have also been adopted to training SNNs to much success
[5, 4]. The benefits of SNNs are limited when ported onto CUDA-accelerated
backends as the underlying instruction-set is constrained to SIMD/SIMT
(single-instruction multiple-data/thread) processing.

Demonstrating the value of SNNs goes beyond measuring test set accuracy on toy
problems. Evaluating the performance of SNNs extends to its efficiency, often
in terms of energy, latency, or synaptic operations per second. Hardware such
as Loihi and SpiNNaker offer power profiling tools to estimate the overhead of
SNN workloads, and programming packages such as SpikingKeras can provide
coarse estimates based on the number of operations in a model [6, 7, 8]. These
tools have lowered the barrier to access neuromorphic algorithm development
and have provided promising empirical demonstrations of spike-based computing.

Neuromorphic research is still highly exploratory, and despite the various
engineering benefits that have been demonstrated, the research community
remain uncertain with what feature sets, neuron models, and neural
computations should be integrated on-chip. On the one hand, why should we
implement features in silicon that are not commonly used? On the other hand,
how do we know what benefits such features have to offer without hardware
baselines?

The recent expansion of open source electronic design automation (EDA) and
VLSI tooling is primed to do for neuromorphic chip design what has already
been done for DL [9, 10]. The explosion of new DL methods as applied to SNNs
only keeps coming [11, 12, 13, 14, 15, 16, 17, 18, 19], and lowering the
barrier to develop ASICs can ultimately profile these workloads in a way that
can demonstrate or dispute energy and latency advantages [20, 21, 22].

To guide the direction of neuromorphic research along an open and reproducible
trajectory, this paper presents a fully open source SNN accelerator, including
the process design kit (PDK), the tooling used to synthesize the design, and
the memory macros used to store synaptic weights. Our design was done in the
SkyWater 130nm (SKY130) process, cleared all pre-tape-out checks, and includes
1,059,840 synaptic weights, and operates at a clock speed of 40 MHz. Small-
scale SNNs (i.e., on the order of 1,000-10,000 neurons) tend to be memory-
limited in their performance, so we have used OpenRAM macros in the design to
promote further optimization of near-memory compute tooling.

##  II Neuron Model

The most common neuron model used in SNNs trained via gradient descent is the
leaky integrate-and-fire neuron model [23, 24]. It offers a reasonable
approximation of neurons while retaining simplicity. A discrete-time version
can be represented by the following equation:

$u_{t}=\beta u_{t-1}+x_{t-1}w-z_{t-1}u_{\rm thr},$ (1)

where $u$ is the membrane potential of the neuron, $x$ is the input to the
neuron, $w$ is the weight attached to $x$, $\beta$ is the decay rate of the
membrane potential, and the subscript $t$ refers to time. When the membrane
potential exceeds the threshold $u_{\rm thr}$, an output spike is generated:

$z_{t}=\begin{cases}1,&\text{\rm if $u_{t}>u_{\rm thr}$}\\\
0,&\text{otherwise.}\\\ \end{cases}$ (2)

Introducing a Schmitt trigger to the thresholding action of the neuron has the
following effect:

$z_{t}=\begin{cases}1,&\text{\rm if $u_{t}>u_{\rm thr}^{h}$ and $i_{t}=0$}\\\
0,&\text{otherwise.}\\\ \end{cases}$ (3)

where $i_{t}$ refers to spike inhibition:

$i_{t}=\begin{cases}0,&\text{\rm if $z_{t}=0$ and $u_{t}<u_{\rm thr}^{l}$}\\\
1,&\text{otherwise.}\\\ \end{cases}$ (4)

As shown in Refs. [15, 25, 26], SNNs are highly tolerant to weight
quantization. In the extreme, the trainable weights of an SNN can be binarized
to $w\in\\{-1,+1\\}$ with small performance degradation as performance is
‘propped up’ by non-binarized variables, including membrane potential and
time.

##  III Architecture of the Accelerator

We propose a time-multiplexed accelerator designed for the SKY130 process with
an open-source PDK, synthesized and hardened using fully open-source EDA
tools. The accelerator core is illustrated in Figure 1. When carrying out
neuromorphic computations, the memory complexity of DL and SNN workloads
scales with $\mathcal{O}(n^{2})$ where $n$ is the number of neurons, which is
dominated by the network parameters. Fully-integrated accelerators on a
monolithic substrate tend to be bottelenecked by weight access.

To address this in our OpenSpike core, spiking neuron modules are re-used and
time-multiplexed by parallelizing neuronal computations with the loading of
weights, along with time-multiplexing inter-neuron computations. Neuron re-use
optimizes for area with a marginal impact on latency. This is possible due to
the parallelization of weight access and neuronal computations (i.e., state
update, and spike triggering). Furthermore, using binarized weights balances
the cost of weight reads from memory with neuron computations.

###  III-A Network Architecture

To implement a neural network, our accelerator uses 1,024 hardware neurons to
process the network one layer at a time. For example, in an SNN with an
architecture of 1024-1024-10 dense connections across three layers, a total of
1,024 neurons would be needed on chip. A drawback to this approach is the need
to save and load neuron data for each layer, but this can be performed in
parallel with weight read-out with a marginal impact on performance.

###  III-B Neuron Processor

Each of the 1,024 neurons consists of a multiply-accumulate (MAC) unit and a
potential adder. The MAC unit sums the product of inputs $x$ with weights $w$
over time, and stores the result in a register. The final result is then
passed to the potential adder which accumulates present time membrane
potential $u_{t}$ with the previous time decayed potential $\beta u_{t-1}$.

While the MAC unit is computing the incoming potential for the current layer,
the potential adder holds the accumulated potential from the previous layer.
The membrane potential arithmetic unit (MAU) computes the decayed potential,
and then loads the result in the potential adder. As decayed potentials are
loaded to the potential adder, the result is used to calculate the spike
responses which are saved and re-read at a slower rate for MAC units, as they
need the previous layer’s spikes. Upon completion of reading decayed
potentials, the new membrane potential values for each neuron are written as a
neuron selector addresses into necessary neurons. The reset status is
determined by the spike response from the Schmitt trigger module. This
response determines whether a spike has occured, which determines whether the
membrane potential has to be reset prior to saving or not.

Figure 1: System architecture: Data flow between different modules of the
OpenSpike core.

Each MAC unit can accumulate up to four incoming connections at once. This
process is repeated for all fan-in spikes.

###  III-C Control Unit

The control unit is a state machine that has three main states:

####  III-C1 Input Layer State

The input layer state consumes only 3 clock cycles, as the input layer neurons
each have only 1 input. It only takes one accumulation cycle for MAC units to
calculate the accumulated potential, as the input spike cache is already
loaded in the output layer state. Saving results from the output layer also
takes one cycle since it only has 10 output neurons and the accelerator core
is designed to save up to 16 neurons in each cycle. Initialization of the
state in the control unit takes one cycle to complete.

####  III-C2 Hidden Layer State

In this state, the MAC units iterate through 1,024 input connections for each
neuron 4 connections at a time, such that 256 cycles are required to compute
the potential values. While MAC units are processing, the neuron selector
activates the potential adders 16 at a time, and it therefore takes 64 cycles
to finish processing and storing the hidden layer potentials.

####  III-C3 Output Layer State

In this state, only 10 MAC units are needed to compute over 256 cycles. At the
same time, the input spike cache is loaded from the input spike SRAM in 8
cycles. Additionally, the potential adders will complete generating results
from the hidden layer. This completes the operations needed for a single time
step.

###  III-D Spike Processor

The spike processor has two functions: i) spike emission using the Schmitt
trigger as a neuron selector reads the updated membrane potential (Figure 2),
and ii) to act as a 128-byte cache for MAC units to be used in the input layer
and to be pre-loaded in the output layer state.

Figure 2: Spike emission process.

###  III-E Membrane Potential Arithmetic Unit

The MAU receives the final potential value from the neuron selector and resets
the potential if that neuron has triggered a spike in the present time step.
The final value is stored, and the decayed potentials are loaded into the
potential adder, where the membrane potential and the decay rate $\beta$ are
read from SRAM and multiplied with a combinational circuit and routed to its
respective potential adder using the neuron selector. To decrease latency, a
combinational circuit was used for multiplication where the membrane potential
is divided into smaller fractions using shift operations, and then they are
added back together in different combinations based on the decay rate to
compute the final decayed membrane potential value (Figure 3).

Figure 3: Membrane potential arithmetic unit uses shift-and-add to implement
varying decay rates with low computational cost.

###  III-F OpenRAM Macros

Memory macros have classically been highly proprietary and manually optimized
circuits, as memory cells rely on layouts that are so dense they must often
bypass design rule violations. Designs are often limited by the availability
of memory designs and arrays, and memory compilers are scarcely available with
existing PDKs. OpenRAM is an open-source memory compiler that helps with
addressing these issues [27]. By using a high-level Python configuration file,
the layout and netlists of SRAMs can be generated by passing in data word
size, the number of words in memory, for a given PDK. Process, voltage and
temperature corners can also be specified for SRAM characterization.

The proposed design uses approximately 154 kB worth of dual-port 2 kB SRAM
macros that consume a total area of 21.89 mm2 for storing spikes, adaptive
thresholds from the Schmitt triggers which enables inhibition, membrane
potential, the decay rate, weights, and incoming spikes. The SRAMs operate at
40 MHz while the core is clocked at 20 MHz. The memory controller addresses
spikes to all fan-out weights, and interfaces all SRAMs to the core. The
weight read lines are multiplexed within one clock cycle of the accelerator
core to increase bandwidth.

##  IV Results

###  IV-A Backend Flow

The OpenLane flow was used to harden the accelerator core, turning synthesized
Verilog into a GDSII layout [28]. Logic synthesis, floorplanning, placement,
clock routing and optimization, global and detailed routing are performed
within this flow. Hardening the accelerator core took 10 hours with a peak of
42.5 GB of RAM utilized, generated 128,776 physical cells, and the resulting
GDSII layout is illustrated in Figure 4.

###  IV-B Timing

The critical path of the core is 24.56 ns due to the neuron state calculation
in the MAU. To optimize the MAU, the throughput of the membrane potential
decay step is doubled by including a pair of adders. The maximum clock
frequency is 24.39 MHz. The SRAMs can operate at 40 MHz and take two cycles
per memory access, and so to balance weight access with computation, the core
is driven at 20 MHz.

TABLE I: Network Timing for a Dense SNN

Process | Latency
---|---
Input Layer | 0.120 $\mu$s
Hidden Layer | 10.3 $\mu$s
Output Layer | 10.3 $\mu$s
Total | 20.72 $\mu$s

  * *

Network architecture: 1024-1024-10 Dense SNN.

TABLE II: Power consumption at the fastest corner Group |  Internal |  Switching |  Leakage |  Total |  %
---|---|---|---|---|---
Sequential |  31.4 $m$W |  2.32 $m$W |  0.37 $\mu$W |  33.9 $m$W |  28.5%
Combinational |  44.1 $m$W |  40.7 $m$W |  2.37 $\mu$W |  84.8 $m$W |  71.5%
Total |  75.5 $m$W |  43.2 $m$W |  2.74 $\mu$W |  119 $m$W |  100.0%
% |  63.6% |  36.4% |  $\sim$0.0% |  100% |
TABLE III: Accuracy of OpenSpike across different datasets compared to full precision counterparts | Accuracy | Accuracy
---|---|---
Dataset | (OpenSpike) | (Full precision)
MNIST | 99.12% | 99.42%
FashionMNIST | 88.12% | 91.02%
DVSGesture | 92.36% | 93.06%
Figure 4: GDSII of the accelerator core.

The timing of various stages of the network are provided in Table I, noting
that this is only for one case out of any number of possible architectural
configurations. Initialization involves loading firmware onto the PicoRV32
processor to configure on-chip logic analyzers, 38 general purpose IOs
(maximum bandwidth of 50 MHz), and a wishbone bus to interface the accelerator
with the firmware. The input layer takes 0.12 $\mu$s to complete, where input
images pre-loaded on a 16kB bank of SRAM. The input is treated as a flattened
single-channel 32$\times$32 image. The input is then passed to the hidden
layer, where each input is weighted by 1,024 weights and takes a total of 10.3
$\mu$s. The final layer has almost the same latency, as the same number of
neuron inputs must be processed. The wallclock time for a forward-pass is
20.72 $\mu$s. The total throughput without pipelining is therefore 48,262
images per second.

###  IV-C Power

The OpenSpike core and its SRAM macros consume about the same amount of
energy. SRAM macros during read and write operations of the core consume
106.54 $m$W of power in total. The dominant portion of power consumption of
the core arises from dynamic power in combinational logic. A power breakdown
at the fastest corner is provided in Table II. Internal power refers to
dynamic power in standard cells, and switching power accounts for dynamic
power across routing and capacitances external to cells. The dynamic power is
the total of the internal and external switching power, and amounts to 119
$m$W for a fully activated network (i.e., all inputs are spiking) in the
worst-case. This can be significantly reduced by training sparse networks,
e.g., by using sparse input data from event cameras [29], and by imposing
objectives that aim to reduce spike count [5, 30, 4, 12].

###  IV-D Accuracy

The accuracy of SNNs with binarized weights, where the network weights
$w\in\\{-1,+1\\}$, has been measured on several image datasets, including
MNIST, FashionMNIST, and DVSGesture [29]. We tested the performance of small
convolutional SNN architectures (16Conv5-64Conv5-10) on classification tasks
in a supervised learning setup where the best performance for the MNIST
dataset was 99.12%, the FashionMNIST dataset was 88.12%, and for DVSGesture
92.36% was achieved (16Conv5-32Conv5-11). These represent very small
performance hits when compared to their full precision counterpart networks,
where MNIST only degraded by 0.3%, FashionMNIST by 2.9%, and DVSGesture by
0.7%. The same hyperparameters and surrogate gradients as in Ref. [26] are
adopted, where the binarization operator is replaced with a straight-through-
estimator during the backward-pass.

##  V Discussion and Conclusion

The proposed SNN accelerator core based on OpenRAM memories aims to drive
neuromorphic hardware research in the direction of reproducibility, in much
the same way algorithms and software development has gone. The work gone into
making neuromorphic accelerators available for broad, public use are extremely
useful for applications engineers in the pursuit of applied neuromorphic
research, though is of less use in the chip design flow outside of algorithms
exploration. Frenkel et al. moved towards open-sourcing a neuromorphic chip
for online learning, and though it relies on closed-source PDKs and memory
macros, making all other aspects of the design available can help the
neuromorphic community with both education and expanding the reach of custom
hardware [31]. Our accelerator addresses the challenges of open hardware by
using a fully open-source design flow. While open flows are presently
constrained to legacy nodes, industry involvement and more advanced
technologies have already become better integrated within open-EDA toolchains
within the past year, including GlobalFoundries 180 nm and SkyWater’s RRAM-
CMOS process. Open neuromorphic accelerators have the potential to do what the
open-source community has achieved for DL.

## References

  * [1] J. C. Knight and T. Nowotny, “Larger GPU-accelerated brain simulations with procedural connectivity,” _Nature Computational Science_ , vol. 1, no. 2, pp. 136–142, 2021.
  * [2] J. P. Turner, J. C. Knight, A. Subramanian, and T. Nowotny, “mlgenn: accelerating snn inference using gpu-enabled neural networks,” _Neuromorphic Computing and Engineering_ , vol. 2, no. 2, p. 024002, 2022\.
  * [3] M. Stimberg, D. F. Goodman, and T. Nowotny, “Brian2genn: a system for accelerating a large variety of spiking neural networks with graphics hardware,” _bioRxiv_ , p. 448050, 2018.
  * [4] J. K. Eshraghian, M. Ward, E. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu, “Training spiking neural networks using lessons from deep learning,” _arXiv preprint arXiv:2109.12894_ , 2021\.
  * [5] E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” _IEEE Signal Processing Magazine_ , vol. 36, no. 6, pp. 51–63, 2019.
  * [6] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain _et al._ , “Loihi: A neuromorphic manycore processor with on-chip learning,” _Ieee Micro_ , vol. 38, no. 1, pp. 82–99, 2018.
  * [7] S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, “The spinnaker project,” _Proceedings of the IEEE_ , vol. 102, no. 5, pp. 652–665, 2014\.
  * [8] T. Bekolay, J. Bergstra, E. Hunsberger, T. DeWolf, T. C. Stewart, D. Rasmussen, X. Choo, A. R. Voelker, and C. Eliasmith, “Nengo: A python tool for building large-scale functional brain models,” _Frontiers in Neuroinformatics_ , vol. 7, p. 48, 2014.
  * [9] R. T. Edwards, “Google/SkyWater & the promise of the open pdk,” in _Proc. Workshop on Open-Source EDA Technol.(WOSET)_ , 2020.
  * [10] A. Ghazy and M. Shalan, “Openlane: The open-source digital asic implementation flow,” in _Proc. Workshop on Open-Source EDA Technol.(WOSET)_ , 2020.
  * [11] G. Bellec, F. Scherr, A. Subramoney, E. Hajek, D. Salaj, R. Legenstein, and W. Maass, “A solution to the learning dilemma for recurrent networks of spiking neurons,” _Nature Communications_ , vol. 11, no. 1, pp. 1–15, 2020\.
  * [12] A. Henkes, J. K. Eshraghian, and H. Wessels, “Spiking neural network for nonlinear regression,” _arXiv preprint arXiv:2210.03515_ , 2022.
  * [13] Y. Yang, N. D. Truong, J. K. Eshraghian, A. Nikpour, and O. Kavehei, “Weak self-supervised learning for seizure forecasting: a feasibility study,” _Royal Society Open Science_ , vol. 9, no. 8, p. 220374, 2022.
  * [14] K. M. Stewart and E. Neftci, “Meta-learning spiking neural networks with surrogate gradient descent,” _Neuromorphic Computing and Engineering_ , 2022\.
  * [15] J. K. Eshraghian, X. Wang, and W. D. Lu, “Memristor-based binarized spiking neural networks: Challenges and applications,” _IEEE Nanotechnology Magazine_ , vol. 16, no. 2, pp. 14–23, 2022.
  * [16] R.-J. Zhu, Q. Zhao, T. Zhang, H. Deng, Y. Duan, M. Zhang, and L.-J. Deng, “Tcja-snn: Temporal-channel joint attention for spiking neural networks,” _arXiv preprint arXiv:2206.10177_ , 2022.
  * [17] N. Perez-Nieves and D. Goodman, “Sparse spiking gradient descent,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 11 795–11 808, 2021.
  * [18] Y. Venkatesha, Y. Kim, L. Tassiulas, and P. Panda, “Federated learning with spiking neural networks,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 6183–6194, 2021.
  * [19] M. E. Elbtity, P. S. Chandarana, B. Reidy, J. K. Eshraghian, and R. Zand, “APTPU: Approximate computing based tensor processing unit,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 69, no. 12, pp. 5135–5146, 2022.
  * [20] P. Zhou, D.-U. Choi, W. D. Lu, S.-M. Kang, and J. K. Eshraghian, “Gradient-based neuromorphic learning on dynamical rram arrays,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 12, no. 4, pp. 888–897, 2022.
  * [21] S. M. Kang, D. Choi, J. K. Eshraghian, P. Zhou, J. Kim, B.-S. Kong, X. Zhu, A. S. Demirkol, A. Ascoli, R. Tetzlaff _et al._ , “How to build a memristive integrate-and-fire model for spiking neuronal signal generation,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 68, no. 12, pp. 4837–4850, 2021.
  * [22] J. K. Eshraghian, S.-M. Kang, S. Baek, G. Orchard, H. H.-C. Iu, and W. Lei, “Analog weights in reram dnn accelerators,” in _2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2019, pp. 267–271.
  * [23] P. Dayan and L. F. Abbott, _Theoretical neuroscience: Computational and mathematical modeling of neural systems_. MIT press, 2005.
  * [24] L. Lapique, “Recherches quantitatives sur l’excitation electrique des nerfs traitee comme une polarization.” _J. of Physiol. and Pathology_ , vol. 9, pp. 620–635, 1907.
  * [25] J. K. Eshraghian, C. Lammie, M. R. Azghadi, and W. D. Lu, “Navigating local minima in quantized spiking neural networks,” _arXiv preprint arXiv:2202.07221_ , 2022.
  * [26] J. K. Eshraghian and W. D. Lu, “The fine line between dead neurons and sparsity in binarized spiking neural networks,” _arXiv preprint arXiv:2201.11915_ , 2022.
  * [27] M. R. Guthaus, J. E. Stine, S. Ataei, B. Chen, B. Wu, and M. Sarwar, “Openram: An open-source memory compiler,” in _2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 2016, pp. 1–6.
  * [28] T. Ajayi and D. Blaauw, “Openroad: Toward a self-driving, open-source digital layout implementation tool chain,” in _Proceedings of Government Microcircuit Applications and Critical Technology Conference_ , 2019.
  * [29] A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza _et al._ , “A low power, fully event-based gesture recognition system,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7243–7252.
  * [30] F. Zenke and T. P. Vogels, “The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks,” _Neural computation_ , vol. 33, no. 4, pp. 899–925, 2021.
  * [31] C. Frenkel and G. Indiveri, “Reckon: A 28nm sub-mm2 task-agnostic spiking recurrent neural network processor enabling on-chip learning over second-long timescales,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 1–3.