# HPCNeuroNet: Advancing Neuromorphic Audio Signal Processing with
Transformer-Enhanced Spiking Neural Networks

Murat Isik *  —, , Hiruna Vishwamith *  — , Kayode Inadagbo *  — , and I. Can
Dikmen, *  — M. Isik is with Stanford University, Stanford, CA, USA (e-mail:
mrtisik@stanford.edu).H. Vishwamith is with University Of Moratuwa, Moratuwa,
Sri Lanka (e-mail: vishwamithpgh.20@uom.lk).K. Inadagbo is with Prairie View
A&M University, Texas, USA (e-mail: kayodeinadagbo@gmail.com).I. C. Dikmen is
with Temsa Research&Development Center, Adana, Turkey (e-mail:
can.dikmen@temsa.com).

###### Abstract

This paper presents a novel approach to neuromorphic audio processing by
integrating the strengths of Spiking Neural Networks (SNNs), Transformers, and
high-performance computing (HPC) into the HPCNeuroNet architecture. Utilizing
the Intel N-DNS dataset, we demonstrate the system’s capability to process
diverse human vocal recordings across multiple languages and noise
backgrounds. The core of our approach lies in the fusion of the temporal
dynamics of SNNs with the attention mechanisms of Transformers, enabling the
model to capture intricate audio patterns and relationships. Our architecture,
HPCNeuroNet, employs the Short-Time Fourier Transform (STFT) for time-
frequency representation, Transformer embeddings for dense vector generation,
and SNN encoding/decoding mechanisms for spike train conversions. The system’s
performance is further enhanced by leveraging the computational capabilities
of NVIDIA’s GeForce RTX 3060 GPU and Intel’s Core i9 12900H CPU. Additionally,
we introduce a hardware implementation on the Xilinx VU37P HBM FPGA platform,
optimizing for energy efficiency and real-time processing. The proposed
accelerator achieves a throughput of 71.11 Giga-Operations Per Second (GOP/s)
with a 3.55 W on-chip power consumption at 100 MHz. The comparison results
with off-the-shelf devices and recent state-of-the-art implementations
illustrate that the proposed accelerator has obvious advantages in terms of
energy efficiency and design flexibility. Through design-space exploration, we
provide insights into optimizing core capacities for audio tasks. Our findings
underscore the transformative potential of integrating SNNs, Transformers, and
HPC for neuromorphic audio processing, setting a new benchmark for future
research and applications.

###### Index Terms:

neuromorphic computing, audio processing, high-performance computing,
heterogeneous devices.

##  I Introduction

High-Performance Computing (HPC) has become a cornerstone in the evolution of
neuromorphic computing, particularly in the realm of audio processing. The
recent surge in neuromorphic hardware architectures, epitomized by platforms
like HPCNeuroNet, underscores the pivotal role of HPC in pushing the
boundaries of computational capabilities. Neuromorphic systems, inspired by
the intricate workings of the human brain, are uniquely positioned to
transform the landscape of audio processing. Their event-driven computational
approach mirrors biological neural activities, making them particularly adept
at handling audio data. However, the journey towards harnessing the full
potential of neuromorphic computing is not without its challenges. One of the
most pressing issues is the reliance on external software for training
neuromorphic systems. This method is not only time-intensive but also consumes
significant energy, making it a less-than-ideal solution for real-time
applications. The need for more efficient, HPC-driven solutions is evident.

Furthermore, the escalating demand for processing vast datasets, especially in
high-resolution audio processing, calls for systems that combine speed,
adaptability, and energy efficiency. Field-Programmable Gate Arrays (FPGAs)
have emerged as a beacon in this context. Their inherent adaptability,
parallelism, and configurability make them ideal for handling the throughput
rates required by modern applications. Over the past decade, FPGAs have seen
widespread adoption due to their versatility across a myriad of applications.
Their inherent flexibility, parallelism, and configurability make them well-
suited to meet the rigorous throughput requirements of modern audio
applications. Over the past decade, FPGAs have witnessed widespread adoption,
a testament to their adaptability across a spectrum of applications. In the
current technological landscape, FPGA architectures, such as the one
underpinning the HPCNeuroNet system, are increasingly recognized as a viable
alternative to traditional FPGAs. These architectures seamlessly integrate the
robust performance of FPGAs with the versatility of processors, offering an
optimal solution for demanding audio processing applications. While there have
been strides in leveraging FPGAs for various applications, the challenge of
power consumption remains. This highlights the indispensable role of HPC in
achieving performance enhancements while maintaining energy frugality.

###  I-A Contributions of Our Research

This manuscript delves into the synergy between neuromorphic audio processing
and high-performance computing, emphasizing the role of heterogeneous devices.
Our primary contributions are:

  * •

Introduction of the HPCNeuroNet architecture, a novel approach that integrates
HPC, SNN dynamics, and Transformer mechanisms for audio processing.

  * •

Exploration of FPGA architectures in neuromorphic audio processing, detailing
the implementation of HPCNeuroNet on FPGA and its benefits.

  * •

Insights into the combined potential of SNN and Transformer architectures for
audio processing, with a focus on real-world applications like audio
denoising.

  * •

Analysis of neuromorphic hardware integration into devices like GPUs, CPUs,
and FPGAs, highlighting their respective advantages and performance metrics.

###  I-B Organization

This manuscript is structured as follows: Section 2 provides background on
neuromorphic computing, its evolution, and its intersection with high-
performance computing. Section 3 delves into the specifics of neuromorphic
audio processing, discussing the methodologies, challenges, and advancements
in the field. Section 4 presents the methodological approach, detailing the
datasets used, the expressivity of Spiking Neural Networks, and the hardware
architectures employed. Section 5 showcases the results, emphasizing the
performance gains achieved through HPC and the scalability of the proposed
methods. Section 6 offers a discussion on the results, analyzing them in the
broader context of HPC and neuromorphic audio processing. Section 7 concludes
the manuscript, summarizing the main points, and emphasizing the significant
contributions and findings of this study. Section 8 discusses potential future
directions in neuromorphic audio processing with HPC.

##  II Neuromorphic Computing, High-Performance Computing, and Audio
Processing

###  II-A Evolution of Neuromorphic Computing and Intersection with HPC

Neuromorphic computing, rooted in brain-inspired principles, has undergone
significant evolution. While its core remains centered on emulating the human
brain’s functionalities, the methodologies have diversified. Techniques such
as in-memory computing, continuous learning in hardware, and fine-grained
parallelism have been explored. The objectives of neuromorphic research can be
categorized into two main streams: understanding biological nervous systems
and leveraging brain-inspired principles for efficient computing. The
intersection of these objectives is evident in the incorporation of
biologically credible mechanisms, potentially surpassing traditional machine
learning paradigms. At the algorithmic forefront, neuromorphic systems
predominantly employ spiking SNNs or models trained using biologically
plausible learning rules. Recent advancements, such as e-prop and decolle,
have addressed challenges of spatial and temporal locality in error signals,
enabling large-scale SNNs like SpikeGPT [1, 2]. Concurrently, the hardware
community has been focusing on neuromorphic devices that leverage material
physics to emulate biological functionalities, ranging from memristive to
photonic devices. The integration of neuromorphic architectures with HPC has
been pivotal in optimizing resource utilization. The event-driven nature of
neuromorphic systems, characterized by neural ’spikes’, offers energy
efficiency, making them ideal for HPC. However, programming challenges
persist, especially for realizing a broad class of probabilistic algorithms in
SNNs.

###  II-B Role of HPC in Current Audio Processing Techniques

Neuromorphic computers, with their architecture mimicking the human brain,
hold transformative potential for healthcare, especially in point-of-care
devices. However, their adoption has been constrained by training challenges,
primarily the dependence on external software, which is both cumbersome and
energy-inefficient. The integration of neuromorphic systems with HPC can
address these limitations, offering enhanced computational power and energy
efficiency. In the realm of audio processing, neuromorphic systems combined
with HPC can revolutionize techniques, as demonstrated in [3], where an SNN-
based sound recognition system achieved over 91% accuracy even in noisy
environments. Similarly, the work in [4] proposed a low-latency voice
recognition system suitable for IoT applications, highlighting the potential
of integrating neuromorphic systems, HPC, and audio processing.

###  II-C Network Architectures in HPC Contexts

The architecture of neural networks plays a significant role in determining
their efficiency and applicability in HPC contexts. For instance, edge
detection, a critical machine learning task, requires extensive neurons and
software simulations, leading to scalability issues and increased
computational time. The research in [5] proposed a scalable FPGA
implementation strategy to address these challenges. Researchers [6] employed
the Oscillatory Dynamic Link Matcher (ODLM) methods to identify patterns,
underscoring the significance of network structures in high-performance
computing.

##  III Neuromorphic Audio

The potential of neuromorphic hardware in audio processing is underscored by
its promise in energy efficiency and computational performance. This is where
HPC plays a pivotal role. By leveraging the parallel processing capabilities
of HPC, neuromorphic audio processing can handle the computational demands of
real-time audio recognition, filtering, and classification. A salient
advantage of integrating HPC with neuromorphic systems lies in the reduction
of model-loading overhead, significantly enhancing processing speeds.
Furthermore, the introduction of probabilistic neuromorphic programming,
combined with hyper-dimensional computing and vector symbolic architectures
like Spatial Semantic Pointers, further augments the capabilities of
neuromorphic audio processing.

Neuromorphic architectures, with their inherent parallelism,
interconnectedness, and adaptability, are adept at executing intricate
operations in real-time [7]. They strike a balance between quality,
efficiency, and resource utilization. These systems employ signals made up of
brief temporal pulses, known as spikes or events, with data encoded in various
ways, from spike polarity and frequency to time-between-spikes [8, 9]. The
Address-Event Representation (AER), a notable advancement introduced by Mead
lab in 1991 [10], facilitates the interconnection of silicon neurons across
chips representing different neural layers.

For audio recognition, neuromorphic tools emulate the biological cochlea’s
function, breaking down an audio signal into various frequency bands of
spiking data [7]. This mirrors the cochlea’s conversion of pressure signals
into neural signals directed to the brain. The inaugural silicon cochlea,
conceptualized by Lyon and Mead [11], emulated the basilar membrane using a
sequence of filter segments. Numerous VLSI adaptations of this model exist
[12, 13, 14], with digital cochlea models employing traditional Digital Signal
Processing methods [15, 16, 17].

The Neuromorphic Audio System (NAS) stands out due to its unique approach [18,
19, 20, 21]. It processes data directly encoded as spikes using Pulse
Frequency Modulation (PFM), implementing Spike Signal Processing methods and
leveraging AER interfaces. The system ingests digital audio streams,
symbolizing monaural sound signals. A Synthetic Spike Generator transforms
this input into a spike sequence, which is then divided into distinct
frequency bands using a cascading band-pass filter bank. These are amalgamated
into an AER output bus, encoding each spike as per AER, and forwarding this
data to classification mechanisms. The computational demands of neuromorphic
audio, given its intricate operations and real-time requirements, necessitate
the power and parallelism of HPC, ensuring optimal performance and efficiency.
A roadmap diagram is shown in Figure 1.

Figure 1: HPCNeuroNet framework depicted across three dimensions: layers,
tasks, and challenges of the design.

##  IV Method

###  IV-A Dataset

The Intel N-DNS collection, adapted from the Microsoft DNS Challenge,
encompasses diverse human vocal recordings in multiple languages and noise
backgrounds [22]. It offers a generation script producing 30-second segments
of clean, noise, and noisy audio for training and validation. Leveraging HPC,
the collection ensures rapid data processing, especially for large datasets.
Contributors can customize generation parameters, choose specific languages
and noises, or add other datasets. The collection provides 500 hours of audio
with SNRs between 20 dB and -5 dB, recorded at 16 kHz and 16-bit resolution.
The validation set maintains standard parameters for consistency. Evaluation
data will be released after model submissions, ensuring unbiased assessment.
The challenge encourages continuous engagement, positioning it as a dynamic
platform for neuromorphic research. Additionally, the challenge includes HPC-
optimized dataloaders for efficient audio retrieval, with optional metadata
detailing audio origins, noise types, and blend levels.

###  IV-B Expressivity of Spiking Neural Networks

SNNs have emerged as a promising avenue in the realm of neural computation,
primarily due to their ability to mimic the temporal intricacies of biological
neural systems. Recent studies have unveiled fascinating similarities and
contrasts between SNNs and their counterparts, the ReLU-activated artificial
neural networks (ReLU-ANNs). This section offers insights into the functional
realization capabilities of these networks and underscores their expressivity
nuances.

####  IV-B1 Function Dissection

Consider a function, denoted as $f(x)$, which is articulated as:

$f(x)=\begin{cases}x&\text{for }x\leq-\theta,\\\ \frac{x-\theta}{2}&\text{for
}-\theta<x<\theta,\\\ 0&\text{for }x\geq\theta.\end{cases}$ (1)

This function can also be rephrased using the ReLU activation:

$f(x)=-\frac{1}{2}\sigma(-x-\theta)-\frac{1}{2}\sigma(-x+\theta),$ (2)

where $\sigma$ epitomizes the ReLU function.

The function $f(x)$ can be adeptly realized through an SNN, characterized by a
set of weights, delays, and thresholds. This network configuration encompasses
two input neurons and a singular output neuron. Depending on the magnitude and
sign of $x$, the firing dynamics of the output neuron are influenced by the
input spikes. This realization holds true across the entire domain of $x$.

####  IV-B2 Complexity Insights

In the realm of neural network architectures, the function $f(x)$ is
represented in a two-layered ReLU-ANN. This architecture comprises two hidden
units and a singular output unit, with each connection defined by specific
weight and bias parameters. The simplicity of the ReLU-ANN structure is
evident, especially when considering the tri-linear nature of $f(x)$. Given
that a ReLU unit can segment the input space into a maximum of two linear
zones, the necessity for at least two hidden units becomes clear. This
emphasizes the efficiency of the ReLU-ANN in capturing $f(x)$. Transitioning
to SNNs, while their inherent delays weren’t the focal point of our
discussion, they play a crucial role in adjusting spike dynamics, which in
turn influences the neural output. A key insight from our exploration is the
capability of SNNs to emulate the dynamics of a conventional ReLU network. The
function $f(x)$ exemplifies this, suggesting scenarios where SNNs could
potentially surpass ReLU-ANNs in representational efficiency. SNNs,
characterized by their transmission of information through spikes or action
potentials, are aptly suited for neuromorphic hardware implementations. Recent
studies have illuminated the versatility of SNNs, showcasing their ability to
emulate Turing machines, arbitrary threshold circuits, and even sigmoidal
neurons. Their prowess extends to approximating continuous functions with
remarkable precision, underscoring their applicability across diverse domains.
In the context of neuromorphic audio processing, restructuring models are
pivotal. Strategic deployment, such as utilizing fewer cores, can enhance both
energy efficiency and computational throughput. Furthermore, the synergy
between neuromorphic hardware architectures and high-performance computing
platforms promises substantial benefits in audio processing. A noteworthy
mention is that our neural network was supervised training using sweat samples
with known ion concentrations before the primary evaluation. In instances
where the chip’s output was inaccurate, adjustments were made, leading to
modifications in the neural network’s node weights. A groundbreaking feature
in our approach is the on-chip training capability, which obviates the need
for external software, marking a significant stride in the field.

####  IV-B3 SNN Neurons hardware implementation

In Figure 2, the architecture of neural processing components in SNN hardware
is depicted. Depending on the neuron models and encoding strategies used in
SNNs, the design of these neural components can vary. As an example, when
utilizing the LIF model combined with rate coding techniques, the neural
components might include the subsequent submodules: 1)  _Multiplier_ , 2)
_Accumulator_ , 3)  _Thresholds_ , and 4)  _Spiking Encoder_ :

Figure 2: Architecture of Spike-Based Neural Systems [23].

  * •

_Multiplier_ and  _Accumulator_ compute aggregate of weighted spike values
using varied data representations. Yet, retaining neural network weights
demands significant on-board memory capacity within FPGA. Given hardware
constraints, besides the usual floating-point and fixed-point representations,
there’s a growing interest in exploring reduced-precision data types in
contemporary studies, including i)  _Quantization_ , ii)  _Binarization_ ,
iii)  _Approximation Computation_ , and iv)  _Posit-floating Computation_ :

The authors of  _Q-SpiNN_ [24] and work [25] implemented a  _Quantized Spiking
Neural Network_ (QSNN) framework to reduce the memory consumption of network
weights based on the low-precision integer scheme. Moreover, work  _BS4NN_
[26] and work [27] further reduced the weight precision to the  _Binarized
Spiking Neural Network_ (BSNN). Work  _AxSNN_ [28] explored the application of
approximation computation on SNN. The authors of [29] researched the
possibility of applying  _Posit_ -floating computation [30, 31] on FPGA-based
SNN accelerators.

  * •

_Thresholds and Spiking Encoding_ : As depicted in Figure 2, the criteria
include factors like decay rate, recovery duration, activation limits, and so
on. For certain encoding methods, like  _Phase Coding_ , an encoding component
becomes essential.

####  IV-B4 Coding Formats of Spiking Signals

In contemporary studies on artificial intelligence, spiking neural networks
have emerged as a leading subject, aiming to emulate the operational
mechanisms of biological neural systems. As a result, it’s essential to
transform input data into a format of spiking signals. For example, Figure 3
displays five of the most commonly used spiking encoding methods in recent
studies[32, 33, 34]:

Figure 3: Five popular spiking coding formats [23]

  * •

_Rate Coding Format:_ Rate coding is a predominant method in associated
studies. This approach transforms the input data into a Poisson spike train
that reflects the respective firing rate. As illustrated in Figure 3a, a more
substantial input is depicted as a spike train with an increased firing
frequency.

  * •

_Time-to-first Spiking Coding Format:_  _Time-to-first Spiking Coding_ employs
a temporal encoding technique. Rather than utilizing firing rate to depict
varying input values, this method uses the firing delay within a stimulus
period to represent data with a spike. As depicted in Figure 3b, spikes
corresponding to greater inputs occur sooner. Unlike the rate coding method,
time-to-first coding needs just one spike, with data details encoded in the
time delay.

  * •

_Phase Coding Format:_ The two mentioned encoding techniques require a
transformation between data values and firing rate or delay. The choice of
conversion approach can affect the resulting spike trains. As illustrated in
Figure 3c, the study by [35] investigated an alternative spiking encoding
approach that directly translates the binary representation of an input value
into a spike train. In this context, the weight of the spike signifies the
importance of each input bit.

  * •

_Burst Coding Format:_ To minimize the transmission time of spiking signals,
the studies by [36] and [37] investigated an encoding technique involving
burst transmission, as depicted in Figure 3d. This approach represents the
input value through the count of spikes and the interval between spikes (ISI)
within the burst. A denser clustering of spikes in a burst indicates a higher
input value.

  * •

_Rank Order Coding Format:_ Figure 3d shows the scheme of  _Rank Order Coding_
[38]. Rather than using the spike latency in the stimulus for encoding as seen
in the  _Time-to-first Spiking Coding_ approach, this method captures input
data based on the sequence in which spikes are fired, without considering the
time delay between them. Consequently, for $N$ pre-neurons, the outputs can
represent up to $2^{N}$ distinct data values.

##  V Transformers in Audio Processing

###  V-A General Description of Transformers

Transformers, introduced in [39], have revolutionized the field of deep
learning, particularly in natural language processing (NLP) tasks. The core
strength of the Transformer architecture lies in its self-attention mechanism,
which allows the model to weigh input data differently, focusing more on parts
that are contextually relevant. This mechanism enables the model to capture
long-range dependencies and relationships in the data, making it particularly
powerful for sequence-to-sequence tasks.

###  V-B Transformers in Audio Processing

While Transformers were initially designed for NLP tasks, their potential has
been recognized in various other domains, including audio processing. In audio
tasks, Transformers can capture temporal dependencies in audio sequences,
making them suitable for tasks like speech recognition, audio synthesis, and
sound classification. The self-attention mechanism allows the model to focus
on specific parts of an audio clip, such as distinguishing speech from
background noise or identifying specific instruments in a musical piece.
Recent advancements have seen the adaptation of Transformer-based models, like
BERT and GPT, for audio tasks, leading to state-of-the-art performance in
several benchmarks.

###  V-C Hardware Implementation of Transformers

The computational demands of Transformer models, especially when scaled, are
significant. This has led to interest in hardware implementations that can
accelerate their performance while reducing power consumption. Hardware
platforms parallelize the matrix operations central to Transformer models,
offering faster inference times. Moreover, with the advent of edge computing,
there’s a growing interest in deploying compact Transformer models on edge
devices, necessitating efficient hardware implementations. Recent research has
also explored quantization and pruning techniques to reduce the model size
without significantly compromising performance, making them more amenable to
hardware deployment.

###  V-D HPCNeuroNet

HPCNeuroNet is a cutting-edge architecture designed to seamlessly integrate
the computational prowess of HPC with the temporal dynamics of SNN and the
attention mechanisms of Transformers, specifically tailored for audio
processing which are shown in Figure 4. The audio data, in its raw waveform,
is first transformed using the Short-Time Fourier Transform (STFT) to convert
it into a time-frequency representation. This representation captures both the
temporal and frequency characteristics of the audio, making it suitable for
further processing. The transformed data, termed the Noisy Input, represents
the audio with potential noise or unwanted signals. This input is then passed
to the Transformer Embedding layer, which converts the time-frequency
representation into dense vector embeddings. These embeddings encapsulate the
salient features of the audio data and are primed for processing by the
subsequent Transformer layers. The core computational layers of the
architecture are the HPCNeuroNet Transformer Layers. These layers process the
embeddings, discerning patterns, and relationships intrinsic to the audio
data. Within these layers, the Self-Attention Mechanism plays a pivotal role.
It allows the model to focus on different parts of the audio data based on
their relevance. This mechanism involves transformations into Query (Q), Key
(K), and Value (V) matrices, followed by Scaled Dot-Product Attention, Multi-
Head Attention, and Positional Encoding. After processing through the
Transformer layers, the data is passed to the Spiking Self Attention
mechanism, which introduces the temporal dynamics of SNN into the attention
mechanism. The output from this layer is then encoded into spike trains using
the SNN Encode layer, leveraging mechanisms such as Rate Coding and Time-to-
First-Spike Coding. The spike-encoded data is then decoded using the SNN
Decode layer, transforming the spike trains back into a time-frequency
representation. This representation undergoes an Inverse Short-Time Fourier
Transform (ISTFT), converting it back to the time-domain. The Clean Output
represents the denoised or processed audio data, with unwanted signals or
noise mitigated. This output is then further refined using Magnitude and Phase
information, ensuring the audio’s fidelity and clarity. The Delay Unit
introduces temporal dynamics into the network, ensuring that the network’s
response is synchronized with the temporal characteristics of the audio data.
The entire process culminates in the End, marking the completion of the audio
data’s journey through the HPCNeuroNet architecture and producing a processed
audio output that is enhanced, denoised, and ready for playback or further
analysis.

Figure 4: HPCNeuroNet Model Architecture. The left showcases the STFT
transforming raw audio into a time-frequency representation. The middle
highlights the Transformer Embedding layer and the core HPCNeuroNet
Transformer Layers with mechanisms like Self-Attention and Scaled Dot-Product
Attention. The Spiking Self Attention, SNN Encode, and SNN Decode layers
introduce and process SNN dynamics. The final Clean Output is refined using
Magnitude, Phase, and ISTFT, with the Delay Unit ensuring temporal
synchronization.

Table I delineates the salient characteristics of the HPCNeuroNet system. Each
row of the table represents a specific configuration or instantiation of the
system, and the columns provide detailed specifications for various
parameters.

  * •

Number of Bands: This column specifies the number of frequency bands into
which the input audio signal is decomposed. In the given configuration, the
audio signal is divided into 256 distinct frequency bands, allowing for a
granular representation of the signal in the frequency domain.

  * •

Frequency Range: This column indicates the range of frequencies that the
system can process. The provided configuration is capable of processing
signals ranging from 0 Hz to 8 kHz, encompassing most of the human audible
range and ensuring that a wide variety of audio signals can be effectively
handled.

  * •

Dynamic Range: The dynamic range, measured in decibels (dB), represents the
ratio between the largest and smallest values that the system can process. A
dynamic range of 96 dB signifies that the system can handle a vast range of
amplitude variations, from very soft to very loud sounds, without distortion
or loss of fidelity.

  * •

Max. Event Rate: This metric, expressed in Mevents/s (Million events per
second), provides an estimate of the system’s processing capability in terms
of the maximum number of events it can handle per second. An event rate of
8.76 Mevents/s indicates a robust processing capability, ensuring real-time or
near-real-time processing of input signals.

  * •

Clock Frequency: The clock frequency, given in MHz, denotes the operating
frequency of the system’s core components. A clock frequency of 250 MHz
ensures rapid processing and timely response to input signals, facilitating
real-time applications and efficient system performance.

TABLE I: HPCNeuroNet characteristics Number bands | Frequency range | Dynamic range | Max. Event rate | Clock frequency
---|---|---|---|---
256 | 0 Hz–8 kHz | 96 dB | 8.76 Mevents/s | 250 MHz

####  V-D1 CPU/GPU Implementation

We utilized Python to execute implementations on the CPU and GPU. The study
leveraged the computational prowess of NVIDIA’s GeForce RTX 3060 GPU and
Intel’s Core i9 12900H CPU, both of which are optimized for different tasks,
ensuring an efficient execution of our implementations.

####  V-D2 FPGA Implementation

Figure 5: Hardware architecture of HPCNeuroNet.

The architecture design implements the various processes in parallel, thus
consuming less processing time. Xilinx Intellectual Property (IP) cores and
custom modules are used in the implementation which is illustrated in Figure
5. Modules are defined using VHDL based on the AXI Interface. The modules in
the design use $16$-bit data configuration for the real and imaginary parts.
Xilinx FFT core is used to improve the efficiency and speed of the STFT and
ISTFT blocks which have a maximum instantaneous frequency of $250$ MHz and can
operate up to $50$ frames/sec. We used the implementation according to dataset
is 48000 1x 1D vectors.

### STFT:

STFT is a pivotal component in the HPCNeuroNet architecture, responsible for
converting the raw audio waveform into a time-frequency representation. Given
the dataset’s structure, with each audio sample being a 1D vector of length
48000, the FFT core was configured accordingly. The STFT operation uses a
window length of 512 with a hop length of 128, leading to 8 ms per time-step,
given the signal’s 16 kHz sampling rate. This configuration ensures that both
the temporal and frequency characteristics of the audio are captured
efficiently.

### SNN Encode:

The encoding process converts the time-frequency representation into spike
trains, optimized for speed and efficiency. The $16$-bit data configuration
ensures that the spike trains capture the nuances of the audio data while
being suitable for real-time processing. The encoder’s design is influenced by
the neuromorphic denoising network, ensuring efficient operation and synergy
with the subsequent processing stages.

### SNN Decode:

The $16$-bit data configuration is maintained, ensuring that the decoded time-
frequency representation retains the audio data’s original characteristics,
albeit processed. The decoder’s design is tailored to operate in conjunction
with the encoder, ensuring seamless integration and efficient data flow within
the HPCNeuroNet architecture.

### ISTFT:

ISTFT is responsible for converting the time-frequency representation back
into the time-domain audio waveform. Similar to the STFT, the core operates
with a $16$-bit data configuration for both the real and imaginary parts,
ensuring precision in the reconstructed audio. The ISTFT operation mirrors the
STFT’s parameters, using the same window length and hop length. The resulting
output is the processed audio waveform, ready for playback or further
analysis.

The concept of design-space exploration (DSE) has gained traction in
neuromorphic hardware research, offering insights into optimizing audio
processing tasks. The judicious selection of core capacity is paramount to
achieving optimal performance and energy efficiency metrics.

####  V-D3 Hardware Accelerator MAC Components

The number of Multiply-Accumulate (MAC) operations in the HPCNeuroNet
architecture can be quantified for its primary components:

  1. 1.

Transformer Embedding: The embedding layer involves matrix multiplications to
convert input data into dense vector embeddings, where $D$ represents the
number of input features or dimensions in the input data and $S$ denotes the
embedding dimension, which is the number of dimensions in the dense vector.

$\text{MAC}_{\text{Embedding}}=D\times S$ (3)

  2. 2.

HPCNeuroNet Transformer Layers: The transformer’s self-attention mechanism
involves matrix multiplications for the Query (Q), Key (K), and Value (V)
transformations, as well as the Scaled Dot-Product Attention, where $T$
represents the number of attention heads in the Multi-Head Attention mechanism
of the transformer layers.

$\text{MAC}_{\text{Transformer}}=2\times D^{2}\times S+D\times S^{2}\times T$
(4)

  3. 3.

Spiking Self Attention and SNN Encode/Decode: The MAC operations for the SNN
layers, where $N$ signifies the number of neurons/processing units in SNN and
$t_{\text{sim}}$ represents simulation time.

$\text{MAC}_{\text{SNN}}=N\times t_{\text{sim}}$ (5)

The total MAC operations for the HPCNeuroNet architecture is the sum of the
MAC operations of its components:

$\text{MAC}_{\text{Total}}=\text{MAC}_{\text{Embedding}}+\text{MAC}_{\text{Transformer}}+\text{MAC}_{\text{SNN}}$
(6)

####  V-D4 Throughput Analysis

The MAC counts, as derived from neural network libraries [40, 41], and the
latency or simulation time, primarily influenced by the hardware components,
can be used to compute the throughput of the system:

$\text{Simulation Time}=\frac{\text{Total Inference Time}}{\text{Total number
of inference samples}}$ (7)

This relationship can be represented as:

$\text{Throughput}=\frac{\\#\text{MACs}}{\text{Latency (Simulation Time)}}$
(8)

####  V-D5 Hit rate of sound recognition system

The efficacy of the N-DNS Challenge is gauged by the clarity of the produced
audio. We utilize the Scale-Invariant Source-to-Noise Ratio (SI-SNR) to
evaluate this clarity. This metric is widely recognized in the domain of audio
processing, as highlighted in references such as [42, 43]. The SI-SNR
evaluates the prominence of human speech over background noise in the system’s
output, drawing parallels with the conventional Source-to-Noise Ratio (SNR)
[43]. A distinguishing attribute of SI-SNR is its immunity to changes in
amplitude. In essence, this ensures that the metric remains unbiased,
regardless of variations in output loudness. Given an input waveform, denoted
by a zero-mean vector $s$, and its corresponding system output $\hat{s}$ and
$e_{\text{noise}}$ represents the noise or error in the signal, the SI-SNR is
expressed as:

$\text{SI-
SNR}:=10\log_{10}\left(\frac{||s_{\text{target}}||^{2}}{||e_{\text{noise}}||^{2}}\right),$
(9)

with

$s_{\text{target}}:=\frac{\langle\hat{s},s\rangle
s}{||s||^{2}}\quad\text{and}\quad
e_{\text{noise}}:=\hat{s}-s_{\text{target}}.$ (10)

Our preference for SI-SNR stems from its straightforwardness and broad
applicability, contrasting with more complex metrics like the speech-to-text
accuracy in the Microsoft DNS Challenge [44]. The N-DNS challenge primarily
seeks to promote advancements in neuromorphic techniques. We view the audio
denoising task as representative of typical audio processing tasks. While some
commercial ventures might not emphasize perceptual quality for human
listeners, SI-SNR offers a practical metric that can also serve as a loss
function in machine learning approaches.

##  VI Results

TABLE II: Evaluation results on CPU, GPU, and our processor. |  | Technology
---
[nm]
| Frequency
---
[MHz]
| # of MAC
---
[MOP]
| Latency
---
[ms]
| Throughput
---
[GOP/s]
| Power [Watt]
---
| Power Efficiency
---
[GOP/s/W]
Intel i9 12900H (CPU) | 10 | 3700 | 960 | 110.62 | 8.67 | 28 | 0.30
NVIDIA RTX 3060 (GPU) | 8 | 1320 | 960 | 3.15 | 304.76 | 80 | 3.80
Xilinx VU37P (FPGA) | 16 | 100 | 960 | 13.5 | 71.11 | 3.55 | 19.75

Table II covers a number of key metrics, including manufacturing technology,
operating frequency, power consumption, and other parameters such as latency,
throughput, and power efficiency. This is evident when examining the latency
metric for the Xilinx FPGA, which stands at a mere 13.5 ms. In terms of power
consumption, the FPGA demonstrates remarkable energy efficiency with a power
requirement of only 3.55 Watts, significantly less than both the CPU and GPU.
The table further highlights the performance-per-watt of the FPGA with a
throughput of 71.11 GOP/s and power efficiency of 19.75 GOP/s/W, underlining
the suitability of FPGA devices for tasks where power efficiency is critical.
This comparison reveals the distinctive characteristics and advantages of each
technology, and their appropriateness would largely depend on the specifics of
the application at hand.

Figure 6: HPCNeuroNet on Different Heterogeneous Devices

Figure 6 shows that CPUs, with their general-purpose design, offer
unparalleled flexibility, making them suitable for a diverse range of tasks.
However, this versatility often comes at the cost of power efficiency and
throughput, especially when compared to specialized hardware like GPUs and
FPGAs. GPUs, originally designed for graphics rendering, have been repurposed
for parallel computing tasks due to their many-core architecture, granting
them higher throughput for specific parallelizable tasks. Yet, they still lag
behind FPGAs in terms of power efficiency. FPGAs, with their reconfigurable
nature, allow for custom-tailored hardware designs that can achieve superior
power efficiency and throughput for specific applications. However, this
customization introduces a higher development complexity, requiring expertise
in digital design and domain-specific knowledge. In essence, while CPUs
provide broad applicability, GPUs and FPGAs offer specialized performance
advantages, each with its trade-offs in terms of efficiency, throughput, and
development complexity.

TABLE III: Comparison of Results on Audio Processing Tasks

Work |  NN |  HW |  Platform |  Freq. |  MAC Number (OP) |  SNR Rate (db) |  Power (W) |  Latency (ms) |  Throughput (GOP/s) |  Scaled Efficiency (GOP/s/W)
---|---|---|---|---|---|---|---|---|---|---
[45] |  CNN-LSTM |  CPU |  Intel i7-9750H |  2.60GHz |  12.5M |  N/A |  25 |  20 |  6.25 |  0.25
[45] |  CNN-LSTM |  GPU |  NVIDIA RTX 2080 |  1.6GHz |  135M |  N/A |  250 |  5 |  27 |  0.108
[46] |  Transformer |  GPU |  NVIDIA RTX 2080 |  1.6GHz |  13.5G |  N/A |  250 |  20 |  675 |  2.7
[47] |  LSTM |  FPGA |  Xilinx Virtex-7 |  200MHz |  N/A |  N/A |  22 |  16.7us |  13.18 |  0.599
[4] |  DRNN |  FPGA |  Xilinx Zynq-7100 |  100MHz |  N/A |  [0, 5, 10] |  2.78 |  7.90 |  7.22 |  2.597
[48] |  LSTM |  GPU |  NVIDIA TitanX |  1.15 GHz |  4290M |  25.0 |  150 |  10 |  429 |  2.86
[49] |  SNN |  FPGA |  Intel Cyclone-4 |  65.03 MHz |  N/A |  25.0 |  1.684 |  N/A |  0.71 |  0.42
[50] |  SNN |  FPGA |  Intel Cyclone-4 |  107.28 MHz |  N/A |  25.0 |  5.148 |  0.27 |  53.6 |  10.42
[51] |  Sigma-Delta NN |  ASIC |  Intel Loihi 2 |  1 kHz |  N/A |  12.71 |  N/A |  0.36 |  N/A |  N/A
Ours |  Hybrid (SNN& Transformer) |  FPGA |  Xilinx VU37P |  100MHz |  960M |  12.14 |  3.55 |  13.5 |  70.11 |  19.75

As shown in Table III, we compare our Hybrid (SNN&Transformer) implementation
with other audio processing models presented in the literature. The evaluation
of our design is done on the Xilinx VU37P FPGA platform operating at 100MHz.
Our experimental results show that the proposed accelerator achieves a
throughput of 70.11 GOP/s with a 3.55 W on-chip power consumption. The SNR for
our model is 12.14 dB. When compared with off-the-shelf devices and recent
state-of-the-art implementations, our proposed accelerator demonstrates
significant advantages in terms of energy efficiency, achieving a scaled
efficiency of 19.75 GOP/s/W, and offers design flexibility.

##  VII Conclusion

The integration of neuromorphic systems with HPC has ushered in a new era of
audio data processing. This confluence has not only streamlined processing but
also enhanced efficiency and speed, making significant strides in real-time
audio applications. Our research, anchored by the results presented,
emphasizes the pivotal role of HPC in augmenting the capabilities of a Hybrid
(SNN & Transformer) model for neuromorphic audio processing. The salient
contributions of HPC to neuromorphic audio processing, as evidenced by our
study, are manifold. The remarkable throughput and energy efficiency metrics
achieved in our experiments underscore the transformative potential of HPC
when combined with neuromorphic principles. The fusion of SNNs with
transformers, in particular, heralds a novel approach to neuromorphic audio
processing. This synergy capitalizes on the strengths of both paradigms,
paving the way for unprecedented energy efficiency, real-time processing
capabilities, and a more nuanced representation of audio data. The
transformative potential of neuromorphic hardware in the domain of audio
processing, when bolstered by the computational prowess of HPC, is undeniable.
As we stand at this juncture, the horizon is replete with opportunities, and
the future beckons with the promise of innovations that will redefine the
landscape of neuromorphic audio processing.

##  VIII Future Work

The integration of SNNs with transformers introduces complexities, notably in
the realms of training dynamics and system scalability. However, the
anticipated benefits underscore its potential as a promising avenue in
neuromorphic audio processing research. Subsequent investigations will aim to
assess its practicality in real-world audio processing, specifically within
innovative TEMSA Skoda vehicles that are equipped with neuromorphic audio
systems.

## Acknowledgments

The authors would like to thank to Mert Hidayetoglu and Jonathan Naoukin for
the feedback, which made the attainment of the results reported in this paper
possible.

## References

  * [1] R.-J. Zhu, Q. Zhao, and J. K. Eshraghian, “Spikegpt: Generative pre-trained language model with spiking neural networks,” _arXiv preprint arXiv:2302.13939_ , 2023.
  * [2] J. Zhang, B. Dong, H. Zhang, J. Ding, F. Heide, B. Yin, and X. Yang, “Spiking transformers for event-based single object tracking,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8801–8810.
  * [3] E. Cerezuela-Escudero, A. Jimenez-Fernandez, R. Paz-Vicente, J. P. Dominguez-Morales, M. J. Dominguez-Morales, and A. Linares-Barranco, “Sound recognition system using spiking and mlp neural networks,” in _International Conference on Artificial Neural Networks_. Springer, 2016, pp. 363–371.
  * [4] C. Gao, S. Braun, I. Kiselev, J. Anumula, T. Delbruck, and S.-C. Liu, “Real-time speech recognition for iot purpose using a delta recurrent neural network accelerator,” in _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2019, pp. 1–5.
  * [5] B. Glackin, J. Harkin, T. M. McGinnity, L. P. Maguire, and Q. Wu, “Emulating spiking neural networks for edge detection on fpga hardware,” in _2009 International Conference on Field Programmable Logic and Applications_. IEEE, 2009, pp. 670–673.
  * [6] L.-C. Caron, F. Mailhot, and J. Rouat, “Fpga implementation of a spiking neural network for pattern matching,” in _2011 IEEE International Symposium of Circuits and Systems (ISCAS)_. IEEE, 2011, pp. 649–652.
  * [7] A. v. Schaik, T. J. Hamilton, and S.-C. Liu, “Silicon cochleas,” _Event-Based Neuromorphic Systems_ , pp. 71–89, 2015.
  * [8] G. Indiveri, E. Chicca, and R. Douglas, “A vlsi array of low-power spiking neurons and bistable synapses with spike-timing dependent plasticity,” _IEEE transactions on neural networks_ , vol. 17, no. 1, pp. 211–221, 2006.
  * [9] S. J. Thorpe, A. Brilhault, and J.-A. Perez-Carrasco, “Suggestions for a biologically inspired spiking retina using order-based coding,” in _Proceedings of 2010 IEEE International Symposium on Circuits and Systems_. IEEE, 2010, pp. 265–268.
  * [10] K. A. Boahen, “Communicating neuronal ensembles between neuromorphic chips,” in _Neuromorphic Systems Engineering: Neural Networks in Silicon_. Springer, 1998, pp. 229–259.
  * [11] R. F. Lyon and C. Mead, “An analog electronic cochlea,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 36, no. 7, pp. 1119–1134, 1988.
  * [12] B. Wen and K. Boahen, “A silicon cochlea with active coupling,” _IEEE transactions on biomedical circuits and systems_ , vol. 3, no. 6, pp. 444–455, 2009.
  * [13] T. J. Hamilton, C. Jin, A. Van Schaik, and J. Tapson, “An active 2-d silicon cochlea,” _IEEE Transactions on biomedical circuits and systems_ , vol. 2, no. 1, pp. 30–43, 2008.
  * [14] S.-C. Liu, A. Van Schaik, B. A. Minch, and T. Delbruck, “Event-based 64-channel binaural silicon cochlea with q enhancement mechanisms,” in _2010 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2010, pp. 2027–2030.
  * [15] M.-P. Leong, C. T. Jin, and P. H. Leong, “An fpga-based electronic cochlea,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2003, pp. 1–10, 2003.
  * [16] R. V. Dundur, M. Latte, S. Kulkarni, and M. Venkatesha, “Digital filter for cochlear implant implemented on a field-programmable gate array,” in _proceedings of world academy of science, engineering and technology_ , vol. 33. Citeseer, 2008.
  * [17] C. S. Thakur, T. J. Hamilton, J. Tapson, A. van Schaik, and R. F. Lyon, “Fpga implementation of the car model of the cochlea,” in _2014 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2014, pp. 1853–1856.
  * [18] M. Domínguez-Morales, A. Jimenez-Fernandez, E. Cerezuela-Escudero, R. Paz-Vicente, A. Linares-Barranco, and G. Jimenez, “On the designing of spikes band-pass filters for fpga,” in _ICANN 2011, Part II_ , ser. LNCS, T. Honkela, Ed., vol. 6792. Heidelberg: Springer, 2011, pp. 389–396.
  * [19] A. Jimenez-Fernandez, A. Linares-Barranco, R. Paz-Vicente, G. Jiménez, and A. Civit, “Building blocks for spike-based signal processing,” in _International Joint Conference on Neural Networks, IJCNN_ , 2010, pp. 1–8.
  * [20] F. Gomez-Rodriguez, R. Paz, L. Miro, A. Linares-Barranco, G. Jimenez, and A. Civit, “Two hardware implementations of the exhaustive synthetic aer generation method,” in _IWANN 2005_ , ser. LNCS, J. Cabestany, A. Prieto, and F. Sandoval, Eds., vol. 3512. Heidelberg: Springer, 2005, pp. 534–540.
  * [21] E. Cerezuela-Escudero, M. Dominguez-Morales, A. Jiménez-Fernández, R. Paz-Vicente, A. Linares-Barranco, and G. Jiménez-Moreno, “Spikes monitors for fpgas, an experimental comparative study,” in _IWANN 2013, Part I_ , ser. LNCS, I. Rojas, G. Joya, and J. Gabestany, Eds., vol. 7902. Heidelberg: Springer, 2013, pp. 179–188.
  * [22] Y. Xing, G. Di Caterina, and J. Soraghan, “A new spiking convolutional recurrent neural network (scrnn) with applications to event-based hand gesture recognition,” _Frontiers in neuroscience_ , vol. 14, p. 1143, 2020.
  * [23] M. Isik, “A survey of spiking neural network accelerator on fpga,” _arXiv preprint arXiv:2307.03910_ , 2023.
  * [24] R. V. W. Putra and M. Shafique, “Q-spinn: A framework for quantizing spiking neural networks,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [25] C. J. Schaefer and S. Joshi, “Quantizing spiking neural networks with integers,” in _International Conference on Neuromorphic Systems 2020_ , ser. ICONS 2020. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3407197.3407203
  * [26] S. R. Kheradpisheh, M. Mirsadeghi, and T. Masquelier, “Bs4nn: Binarized spiking neural networks with temporal coding and learning,” _Neural Processing Letters_ , pp. 1–19, 2021.
  * [27] M.-L. Wei, M. Yayla, S.-Y. Ho, J.-J. Chen, C.-L. Yang, and H. Amrouch, “Binarized snns: Efficient and error-resilient spiking neural networks through binarization,” in _2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 2021, pp. 1–9.
  * [28] S. Sen, S. Venkataramani, and A. Raghunathan, “Approximate computing for spiking neural networks,” in _Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017_. IEEE, 2017, pp. 193–198.
  * [29] V. H. Silva, J. F. Chaves, R. M. Gomes, and B. A. Santos, “Posit-based spiking neuron in an fpga.”
  * [30] J. L. Gustafson, “Posit arithmetic,” _Mathematica Notebook describing the posit number system_ , vol. 30, 2017.
  * [31] Gustafson and Yonemoto, “Beating floating point at its own game: Posit arithmetic,” _Supercomput. Front. Innov.: Int. J._ , vol. 4, no. 2, p. 71–86, Jun. 2017. [Online]. Available: https://doi.org/10.14529/jsfi170206
  * [32] W. Guo, M. E. Fouda, A. M. Eltawil, and K. N. Salama, “Neural coding in spiking neural networks: A comparative study for robust neuromorphic systems,” _Frontiers in Neuroscience_ , vol. 15, p. 212, 2021.
  * [33] J. von Kügelgen, “On artificial spiking neural networks: Principles, limitations and potential,” _June_ , vol. 18, p. 2017, 2017.
  * [34] A. El Ferdaoussi, J. Rouat, and E. Plourde, “Efficiency metrics for auditory neuromorphic spike encoding techniques using information theory,” _Neuromorphic Computing and Engineering_ , vol. 3, no. 2, p. 024007, 2023.
  * [35] J. Kim, H. Kim, S. Huh, J. Lee, and K. Choi, “Deep neural networks with weighted spikes,” _Neurocomputing_ , vol. 311, pp. 373–386, 2018.
  * [36] E. M. Izhikevich, N. S. Desai, E. C. Walcott, and F. C. Hoppensteadt, “Bursts as a unit of neural information: selective communication via resonance,” _Trends in neurosciences_ , vol. 26, no. 3, pp. 161–167, 2003.
  * [37] H. G. Eyherabide, A. Rokem, A. V. Herz, and I. Samengo, “Bursts generate a non-reducible spike-pattern code,” _Frontiers in Neuroscience_ , vol. 3, p. 2, 2009.
  * [38] S. Thorpe and J. Gautrais, “Rank order coding,” in _Computational neuroscience_. Springer, 1998, pp. 113–118.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [40] “PyTorch-OpCounter,” https://pypi.org/project/thop/, accessed: 2023-08-26.
  * [41] “TensorFlow-Profiler,” https://tensorflow.org/guide/profiler, accessed: 2023-08-26.
  * [42] F. Bahmaninezhad, J. Wu, R. Gu, S.-X. Zhang, Y. Xu, M. Yu, and D. Yu, “A comprehensive study of speech separation: spectrogram vs waveform separation,” _arXiv preprint arXiv:1905.07497_ , 2019.
  * [43] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 626–630.
  * [44] H. Dubey, V. Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper _et al._ , “Icassp 2022 deep noise suppression challenge,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9271–9275.
  * [45] J. H. Ko, J. Fromm, M. Philipose, I. Tashev, and S. Zarar, “Precision scaling of neural networks for efficient audio processing,” _arXiv preprint arXiv:1712.01340_ , 2017.
  * [46] T. Yoshimura, T. Hayashi, K. Takeda, and S. Watanabe, “End-to-end automatic speech recognition integrated with ctc-based voice activity detection,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6999–7003.
  * [47] S. Wang, Z. Li, C. Ding, B. Yuan, Q. Qiu, Y. Wang, and Y. Liang, “C-lstm: Enabling efficient lstm using structured compression techniques on fpgas,” in _Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , 2018, pp. 11–20.
  * [48] G. Diamos, S. Sengupta, B. Catanzaro, M. Chrzanowski, A. Coates, E. Elsen, J. Engel, A. Hannun, and S. Satheesh, “Persistent rnns: Stashing recurrent weights on-chip,” in _International Conference on Machine Learning_. PMLR, 2016, pp. 2024–2033.
  * [49] B. Deng, Y. Fan, J. Wang, and S. Yang, “Reconstruction of a fully paralleled auditory spiking neural network and fpga implementation,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 15, no. 6, pp. 1320–1331, 2021.
  * [50] ——, “Auditory perception architecture with spiking neural network and implementation on fpga,” _Neural Networks_ , vol. 165, pp. 31–42, 2023.
  * [51] J. Timcheck, S. B. Shrestha, D. B. D. Rubin, A. Kupryjanow, G. Orchard, L. Pindor, T. Shea, and M. Davies, “The intel neuromorphic dns challenge,” _Neuromorphic Computing and Engineering_ , vol. 3, no. 3, p. 034005, 2023.