#  : A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient
Edge Devices

Arnab Raha, Deepak A. Mathaikutty, Soumendu K. Ghosh and Shamik Kundu *Shamik
Kundu contributed to this work during his internship in the Advanced
Architecture Research Group during summers ’22 and ’23. He is currently a PhD
student at UT Dallas. (Email<EMAIL_ADDRESS>Advanced Architecture
Research, NPU IP, CGAI (CCG), Intel Corporation, Santa Clara, CA, USA
Email: {arnab.raha, deepak.a.mathaikutty, soumendu.ghosh,
<EMAIL_ADDRESS>
###### Abstract

This paper introduces , a Flexible Neural Network accelerator, which adopts
agile design principles to enable versatile dataflows, enhancing energy
efficiency. Unlike conventional convolutional neural network accelerator
architectures that adhere to fixed dataflows (such as input, weight, output,
or row stationary) for transferring activations and weights between storage
and compute units, our design revolutionizes by enabling adaptable dataflows
of any type through software configurable descriptors. Considering that data
movement costs considerably outweigh compute costs from an energy perspective,
the flexibility in dataflow allows us to optimize the movement per layer for
minimal data transfer and energy consumption, a capability unattainable in
fixed dataflow architectures. To further enhance throughput and reduce energy
consumption in the architecture, we propose a novel sparsity-based
acceleration logic that utilizes fine-grained sparsity in both the activation
and weight tensors to bypass redundant computations, thus optimizing the
convolution engine within the hardware accelerator. Extensive experimental
results underscore a significant enhancement in the performance and energy
efficiency of relative to existing DNN accelerators.

###### Index Terms:

Deep neural network accelerator, flexible data flow, sparsity acceleration,
energy efficiency.

##  I Introduction

The landscape of machine learning is experiencing an unprecedented surge, with
a multitude of artificial intelligence (AI) networks proposed alongside the
development of numerous hardware platforms dedicated to accelerating inference
tasks. As the field progresses, the complexity of neural networks continues to
grow, resulting in the handling of vast amounts of tensor data that exhibit
diverse shapes and dimensions across different layers of existing networks.
Moreover, with the continuous introduction of new models, the dimensions of
these tensor data are in constant flux. Consequently, there is a pressing need
to engineer hardware accelerators with the flexibility to efficiently process
network layers of varying dimensions [1, 2].

Furthermore, the proliferation of edge devices, including wearables, smart
cameras, smartphones, and surveillance platforms, underscores the importance
of energy efficiency in the design of neural network accelerators [3]. Given
that tensor data processing involves traversing multiple levels of memory
hierarchy, minimizing data transfer while maximizing data reuse and resource
utilization emerges as critical imperatives to improve the energy efficiency
of Deep Neural Network (DNN) accelerators [4].

However, the prevailing accelerators for DNN execution, such as Eyeriss [5]
and TPU [6], typically adopt custom memory hierarchies and fixed dataflows.
These architectures dictate the sequence in which the tensor data for the
activations and weights are moved to processing units to execute the tensor
operations for each layer of the network. Although effective, these approaches
may not fully exploit the potential for energy efficiency optimization
inherent in more flexible hardware designs. Therefore, there is a growing
interest in exploring novel architectures and strategies that can better adapt
to the evolving demands of neural network inference while simultaneously
improving energy efficiency across a range of edge devices and applications.

The energy consumption for each layer in neural network inference is heavily
influenced by the movement of data across the memory hierarchy and the level
of reuse within the processing units. Previous studies have endeavored to
characterize energy efficiency through analytical models while stressing the
importance of enabling flexibility in scheduling tensors of varying dimensions
[7] . This flexibility involves optimizing the ordering, blocking, and
partitioning of tensors to maximize reuse from the innermost memory hierarchy,
where the energy cost per unit of data moved is minimized. However, most
existing DNNs, such as ResNet, YOLO, VGG, and GoogLeNet, comprise tens to
hundreds of layers, each with different preferences for scheduling to achieve
energy optimality. Fixed-schedule DNN accelerators can only offer optimal data
reuse and resource utilization for a subset of DNN layers, thus limiting
overall energy efficiency. Moreover, these accelerators exhibit strong network
dependencies, posing challenges in adapting to the rapidly evolving landscape
of DNNs. Existing DNN accelerator designs from both industry and academia
predominantly employ fixed schedules, such as weight stationary (WS), output
stationary (OS), non-local reuse (NR), and row stationary (RS) [8, 9]. The
fixed dataflow characteristic of these accelerators originates from their
tensor data distribution modules, which perform addressing to on-die storage,
data transfer to processing engine arrays, and data storage to SRAM banks in a
predetermined manner.

As a result, existing accelerators lack the flexibility to implement different
schedules (i.e., dataflows), and while software-based solutions on general-
purpose CPUs and GPUs can reshape and load tensor data, fixed-function
accelerators do not support such flexibility. FPGAs, although offering
flexibility, cannot alter the hardware configuration during execution from one
layer to another of a DNN.

In contrast to previous approaches, this paper proposes a dataflow-aware
flexible DNN accelerator that leverages schedule information from DNN layers
to adapt tensor data shape and internal compute configuration on a per-layer
basis. This enables the compiler to configure the DNN accelerator optimally
for handling tensor operations based on tensor dimensions. The key advantage
of our proposed accelerator design lies in its ability to switch among
multiple schedules based on layer characteristics, thereby minimizing memory
accesses for a given tensor operation and resulting in significant energy
savings at the accelerator level.

To further enhance performance and increase energy efficiency in the
accelerator, we capitalize on the inherent sparsity inherent in DNNs. Due to
the nature of DNNs, the weights associated with the network are often
“sparse,” meaning that they contain a significant number of zeros generated
during the training phase [10, 11]. These zero-valued weights do not
contribute to the accumulation of partial sums during multiply-and-accumulate
(MAC) operations. Additionally, highly sparse weights cause activations to
become sparse in subsequent layers of the DNN after passing through non-linear
activation functions like ReLU. Furthermore, network quantization for edge
device inference also results in a high number of zeros in both weights and
activations. The unstructured fine-grained sparsity in weights and activations
offers potential for improved energy efficiency and processing speed in two
ways: (1) MAC computation can be gated or skipped, and (2) weights and
activations can be compressed to reduce storage and data movement. The former
reduces energy consumption, while the latter reduces both energy consumption
and processing cycles. However, designing DNN accelerators to harness these
benefits from sparsity is challenging due to irregular access patterns,
workload imbalances, and under-utilization of MAC-based processing elements
[12]. Hence, in this paper, we develop a novel sparsity acceleration logic
capable of skipping zero-valued compression while simultaneously identifying
non-zero elements in both activation and weight tensors. This will facilitate
the implementation of an efficient convolution engine in the hardware
accelerator at the edge, enabling efficient utilization of resources and
enhancing overall performance and energy efficiency.

In summary, in this paper, we introduce , a Flexible Neural Network
accelerator, designed with agile principles to support versatile dataflows,
thereby enhancing energy efficiency. Recognizing that data movement costs
significantly outweigh compute costs in terms of energy consumption, the
flexibility in dataflow enables us to optimize data transfer per layer,
leading to minimal data movement and reduced energy consumption, an advantage
not achievable in fixed dataflow architectures. Additionally, to further boost
throughput and reduce energy consumption within the architecture, we propose
an innovative sparsity-based acceleration logic. This logic harnesses fine-
grained sparsity in both activation and weight tensors to bypass redundant
computations, effectively optimizing the convolution engine within the
hardware accelerator. This paper makes the following contributions.

  * •

This paper introduces a novel DNN accelerator, , designed to be sensitive to
dataflow, offering flexibility by integrating DNN layer scheduling insights.
By dynamically adjusting tensor data shape and the internal compute
configuration for each layer, the accelerator allows the compiler to optimize
its performance in handling tensor operations, tailoring its configurations
based on tensor dimensions for diverse neural network architectures.

  * •

We introduce a novel sparsity acceleration logic that capitalizes on the
unstructured fine-grained sparsity present in incoming activation and weights,
thereby expediting inference execution within the DNN accelerator. Data are
maintained in a zero-compressed format to mitigate storage and data movement
expenses. Weights and activations are mapped while considering sparsity to
enhance reuse, thereby enhancing overall performance.

  * •

Extensive experimental evaluations conducted on six distinct DNNs spanning
both classification and detection tasks highlight the transformative impact of
our accelerator. Specifically, our architecture showcases substantial
improvements over fixed-schedule accelerators for ResNet101 and YOLOv2,
demonstrating up to $77\%$ and $62\%$ energy reduction over Eyeriss and TPU,
respectively. Furthermore, our accelerator achieves notable sparsity
improvements for four additional DNNs, namely ResNet50, MobileNetV2,
GoogLeNet, InceptionV3. Across these benchmarks, achieves a speedup of
$2.0\times$–$2.7\times$ over dense accelerators and $1.3\times$–$1.8\times$
over semi-sparse accelerators with weight-sparsity support. Sparsity support
also provides $1.9\times$–$2.5\times$ and $1.3\times$–$1.7\times$ improvement
in energy efficiency compared to dense and weight-sided accelerator. These
results underscore the profound impact of our accelerator in enabling
efficient execution of sparse and compact DNNs, significantly enhancing both
speed and energy consumption metrics.

The remainder of the paper is organized as follows. Section II delineates the
need for a flexible dataflow and an efficient two-sided sparsity acceleration
logic. Section III describes the microarchitectural details of the proposed
accelerator. Section IV presents the experimental setup followed by the
results in Section V. The prior art in this domain is outlined in Section VI.
Finally, Section VII concludes the paper.

##  II Motivation

In this section, we delve into the fundamental motivations driving the design
and development of our accelerator architecture, focusing on two key aspects:
the paramount importance of flexibility and the critical need for efficient
sparsity acceleration. By addressing these critical considerations, our
accelerator aims to revolutionize the landscape of deep learning hardware,
offering unparalleled versatility and performance across a wide range of
neural network models and applications. We explore how these foundational
principles drive innovation and shape the architectural decisions that lead to
the design of .

###  II-A Importance of flexibility

Figure 1: Illustration of multi-loop tensor processing during convolution
operation in DNN.

Numerous DNN accelerators utilize spatial architectures comprised of arrays of
processing elements (PEs) alongside local storage like register files (RFs)
for those PEs, as well as external storage such as SRAM banks. In inference
tasks, trained weights, or filters (FL), must be loaded into PE arrays from
storage sources like DRAMs and SRAM buffers. Input images, referred to as
input activations or feature maps (IFs), are also transferred into PE arrays,
where MAC operations occur across multiple input channels (ICs) between
activations and weights, generating output activations (OFs). Multiple sets of
weight tensors (OCs) are commonly used against a specific set of activations
to produce an output tensor volume. Finally, a non-linear function (e.g.,
ReLU) is applied to the output activations, which then become the input
activations for the subsequent layer. The tensor processing involved in a
convolution operation, as shown in Fig. 1, showcases convolution layers
comprising seven nested loops. These layers generate an output tensor, OF map,
from multiple kernel feature maps, FLs, operating on one or more input
tensors, IF map. Each point in the output volume undergoes a MAC operation
during the calculation. For instance, a 1$\times$1 convolution layer, such as
the second convolution layer in ResNet50, illustrates IF map dimensions of IX
= 56, IY = 56, IC = 64, and the filters dimensions of FX = 1, FY = 1, IC = 64,
OC = 256. These dimensions convolve (with a batch size of ON = 1) to produce
an OF map with dimensions OX = 56, OY = 56, OC = 256, accompanied by
appropriate padding values.

The dimensions of the input tensor undergo changes as they transition from one
layer to another within a DNN and across various DNNs. Consequently, the
development of flexible hardware accelerators becomes crucial to maintaining
high utilization of compute units across network layers with arbitrary
dimensions. Attempting to map various tensor dimensions to a fixed PE array
with a consistent tensor mapping pattern can lead to decreased array
utilization. To improve performance and energy efficiency, it is imperative to
minimize data movement by maximizing data reuse from local memory and
improving resource utilization. This optimization is particularly vital, as
the cost of memory accesses often exceeds that of computing, as illustrated in
Fig. 2. Numerous existing DNN accelerators, such as Eyeriss [12], TPU [13],
and SCNN [10], implement novel memory hierarchies and fixed dataflows,
influencing the movement of tensors for activations and weights within the
processing units and the workload assigned to each PE. A fixed dataflow
constrains the types of data movement across the memory hierarchy, limiting
the degree of reuse within processing units. The movement of input activations
(IFs), weights (FLs), and partial sums, as well as the order of reuse,
directly impact the energy consumption for each layer. In the literature[7],
inference accelerators are classified into input stationary, weight
stationary, output stationary, and row stationary based on the dataflow. The
data reuse scheme is based on loop order, loop blocking, and partitioning for
tensor processing, collectively called a “schedule”, as depicted in Fig. 3.
This schedule is described in relation to the dimensions of the tensors in a
convolutional neural network. The loop order dictates the relative order of
IX, IY (spatial), and IC dimensions for activations and FX, FY, IC, OC
dimensions for filters when loading these data into the accelerator. Moreover,
loop partitioning dictates how the overall convolution operation is
distributed among the PEs in the PE array, while loop blocking governs the
allocation of multiple points in each dimension to the same PE.

Figure 2: Relative energy costs of different compute and memory operations at
various precisions in 45 nm technology[14]. Note that x-axis is in logarithmic
scale. Figure 3: Illustration of (1) loop order, (2) loop partitioning and (3)
loop blocking — referred to collectively as scheduling - for optimizing data
loading and distribution in the accelerator.

All existing inference engines operate with fixed loop orders, blocking, and
partitioning for convolution operations. Consequently, each accelerator can
only execute a single predetermined dataflow, where the data remain stationary
in only one aspect. Various schedules require that IFs, FLs, and OFs/partial
sums be mapped and accessed from local register file (RF) storage differently
based on the type of schedule being computed. For instance, in the Input
Stationary (IS) scenario, a single point within the IF RF must undergo
multiplication and accumulation against multiple points in the FL RF. The
frequency of this repetition of operations varies on the basis of the
schedule. Similarly, in the Weight Stationary (WS) situation, a single point
within the FL RF must be multiplied and accumulated against multiple points in
the IF RF. Lastly, for Output Stationary (OS) schedules, the same partial sum
in the OF RF must be retained and used to accumulate the results of
multiplication between distinct IF and FL RF points over multiple cycles.
Additionally, when the size of the SRAM imposes limitations on the number of
IC points that can be stored, incomplete OF points in the form of partial sums
must be transferred to a higher-level memory hierarchy for subsequent
retrieval into PE RFs to complete the OF computation across all input
channels.

Previous research aimed at characterizing the energy efficiency of DNN
accelerators by constructing analytical models underscores the need to
introduce flexibility in scheduling tensor operations of various dimensions to
maximize reuse from the innermost memory hierarchy, where the energy cost per
unit of data moved is minimized [7]. Vision networks such as ResNet, YOLO, and
VGG consist of tens to hundreds of layers, each favoring different dataflows
to achieve optimal energy efficiency. In contrast, fixed dataflows can only
cater to optimal data reuse and resource utilization for a limited subset of
DNN layers. The proposed tensor data computing PE array offers a practical
solution to address the flexibility challenge with minimal hardware overhead.
Realizing a flexible dataflow accelerator requires a dataflow-aware tensor
distribution unit capable of utilizing layer-specific optimal schedules and
dataflow information to distribute data to the array. Moreover, the
accelerator should inherently support flexible mapping and execution of these
data within each PE.

Motivation 1: It is important to develop a flexible dataflow in order to
minimize data movement and maximize reuse in the PE array.

###  II-B Importance of Sparsity Acceleration

Sparse IFs are inherent in DNNs because of several factors. One primary cause
is the prevalent use of the ReLU as an activation function within many DNN
architectures. The nature of ReLU to set negative values to zero contributes
to sparsity, particularly intensifying in deeper layers, where it often
exceeds 90%. Additionally, the rise of auto-encoders and generative
adversarial networks (GANs) further accentuates sparsity trends. These network
feature decoder layers employ zero insertion techniques to up-sample input
feature maps, resulting in more than $75\%$ zeros. In addition, extensive
efforts have focused on inducing FL sparsity within DNNs. Various criteria,
such as saliency, magnitude, and energy consumption, are employed to determine
which weights to prune. As a result, pruned networks exhibit weight sparsity
levels of up to 90% [15].

The translation of the sparsity in weights and activations into enhanced
energy efficiency and processing speed presents a significant opportunity.
However, designing DNN accelerators capable of effectively harnessing these
advantages remains a formidable challenge. Computation gating emerges as a
promising technique for converting sparsity in both FLs and FLs into energy
savings. Its implementation involves recognizing if either the weight or
activation is zero and gating the datapath switching and memory accesses
accordingly, achieving cost-effective solutions. To optimize throughput while
conserving energy, skipping cycles of processing MACs with zero weights or
activations becomes desirable. Yet, this necessitates more intricate read
logic to locate the next non-zero value without expending cycles on zeros. A
natural solution entails maintaining FLs and IFs in a compressed format
indicating the next non-zero location relative to the current one. However,
compressed formats, often of variable length, pose challenges for parallel
processing across PEs without compromising compression efficiency.
Additionally, simultaneous recognition of sparsity in both weights and
activations complicates matters, as efficiently ‘looking ahead’ (e.g.,
skipping non-zero weights when the corresponding activation is zero) proves
challenging with many compression formats. The irregularity introduced by such
jumps precludes the use of pre-fetching to enhance throughput. Consequently,
the control logic for processing compressed data becomes intricate, adding
overhead to the PEs. Addressing these complexities is vital to realize the
full potential of sparsity in DNN accelerators.

As a result, hardware solutions in this domain have been sparse. For example,
Cnvlutin [16] exclusively facilitates skipping cycles for activations without
compressing weights, while Cambricon-X [17] lacks the ability to maintain
activations in compressed format. Given the intricacies involved in skipping
cycles for both weights and activations, existing hardware designed for sparse
processing tends to be tailored to specific layer types. For example, EIE [18]
is tailored for fully connected (FC) layers, while SCNN [10] is optimized for
convolutional (CONV) layers. This specialization underscores the need for
further innovation in developing versatile hardware architectures capable of
efficiently handling sparsity across various layer types in deep neural
networks.

Introducing computation skipping for sparse data fundamentally alters the
workload distribution across PEs, as the workload at each PE becomes
contingent on sparsity levels. As the count of non-zero values fluctuates
across diverse layers, data types, or even within specific regions within the
same filter or feature map, it engenders an inherent imbalance in workload
distribution across PEs [12]. Consequently, the throughput of the entire DNN
accelerator becomes constrained by the PE processing the highest number of
non-zero Multiply-Accumulate (MAC) operations. This imbalance inevitably
results in reduced PE utilization, thereby impeding the overall efficiency and
performance of the DNN accelerator. Addressing this challenge requires
innovative strategies to optimize workload distribution and improve PE
utilization, thus maximizing the potential benefits of computation skipping
for sparse data.

Motivation 2: It is important to develop an acceleration logic that can
leverage both unstructured IF and FL sparsity, in zero-compressed format.

##  III Microarchitecture Design

Figure 4: Top-level schematic of accelerator: (1) Interconnect pathways within
the PE array arranged in N columns, data distribution network, control block,
and SRAM. (2) Architectural intricacies of the individual PEs showing data
storage (IF/FL/OF RF) and sparsity storage (IF/FL SP) and MAC unit.

This section delineates the array of microarchitecture design decisions and
techniques essential for implementing the proposed accelerator.

###  III-A Overview of accelerator

The high-level diagram of the DNN accelerator is shown in Fig. 4, illustrating
the various microarchitectural components that facilitate flexible and
reconfigurable dataflow. Although the proposed DNN accelerator accommodates
any schedule, it remains preferable to configure it according to the optimal
schedule for individual layers of the neural network. The optimal schedule is
obtained per layer, according to existing research [19, 20]. Leveraging the
regularity of DNN computations facilitates efficient data loading into the
accelerator and enables flexible convolution mapping based on the optimal
schedule. In our design, we assume a three-level memory hierarchy. The first
level comprises internal register files within each PE. The second level
consists of SRAM, akin to a small L1 cache, storing input operands, output
points, and partial sums. The third level encompasses DRAM memory, which is
required because of its high capacity to store large amounts of filter weights
and spilled OF points. For brevity, we omit DRAM-level implementation
specifics and assume that the SRAM’s capacity is sufficient to accommodate all
input, intermediate activations, and filter weights. Each memory level gives
us the opportunity to reuse data, thus enhancing energy efficiency. The
flexible DNN accelerator, capable of supporting $INT8,U8,FP16$ and $BF16$
datatypes, comprises three principal components, elaborated in subsequent
sections.

###  III-B Versatile Processing Element (VPE) and Flexible Processing Element
Array (FPA)

The Versatile Processing Element (VPE) serves as the fundamental computational
unit within the proposed accelerator, tasked primarily with executing MAC
operations between IF and FL points [21, 22, 23]. VPE also facilitates the
accumulation of internal/ external partial sums (psums). In the context of the
flexible accelerator, the VPE optimizes the reuse of IF/FL/OF data and selects
the most suitable compute template based on the optimal schedule for each
layer. The Flexible Processing Element Array (FPA) comprises an N$\times$M
array of VPEs, with the array dimension parameterized by design, typically
through synthesis parameters. This array can be conceptualized as being
arranged into M columns, each column consisting of N VPEs. To streamline the
control logic, we deliberately adopted a square grid configuration
(N$\times$N) for the FPA, using N = 16, simplifying the associated control
mechanisms. Fig. 4.1 illustrates the schematic of the DNN FPA, composed of an
array of VPEs that serve as the most important computational units within the
DNN accelerator. Each VPE, as demonstrated in Fig. 4.2, features five 1R1W
register files (RF) to store input operands (IF and FL), their corresponding
sparsity bitmaps, and output values (OF/psum). During a typical MAC operation,
the input operands are fetched from the IF and FL RFs, based on the stored
bitmaps and the sparsity acceleration logic (described later in Section
III-E), and accumulated within the OF RF. Control over read pointers for IF,
FL, and OF RFs, as well as the write pointer for OF, is managed using internal
counters programmed by the DNN control logic.

The proposed VPE, as demonstrated in Fig. 5 executes computations on IF, FL,
and OF/psum tensor data based on the optimal schedule of the current layer.
VPE dynamically adjusts the loading and access patterns of the IF, FL and
OF/psum tensor data within the PE RFs to maximize reuse of the tensor data.
Configuration descriptors are updated at the onset of each convolution layer
based on the optimal schedule of the layer, guiding data redirection during
load, compute, and drain operations throughout the lifetime of input tensor
data. The PE finite-state machine (FSM) incorporates internal counters and
logic to generate read and write control signals for the IF, FL, and OF RFs,
along with multiplexer control signals to route data from the RFs to the
appropriate arithmetic units based on the template (V$\times$V/M$\times$M) or
operation type (MAC/Eltwise) and tensor dimension.

Figure 5: Microarchitecture of the PE demonstrating inter-connectivity among
control units, data register files, and MAC units, alongside support for two
accumulation orders.

Internal registers within the PE FSM track the total number of PE blocks (or
OF/psum points) produced, aiding in addressing IF/FL/OF RFs. Additionally,
counters like ifcount, wcount, and ofcount manage the addresses/indexes for
IF, FL, and OF RFs, increasing or clearing based on the number of input
activations and weights required to calculate each OF point or psum block. The
layer schedule determines the type and extent of IF/FL/OF RF data reuse,
regulated by internal IF/FL/OF block counters controlling the loading of new
IF/FL data and draining OF data each round, as per the layer’s optimal
schedule. These internal structures and associated control logic are pivotal
to supporting flexible schedules within the VPE. The critical role of VPE in
facilitating flexible blocking within the DNN accelerator is realized by
dividing the RF into multiple subbanks (X) and incorporating X MACs (e.g., X =
4) alongside multiplexers, allowing the implementation of V$\times$V,
V$\times$M, and M$\times$M templates [24], based on the optimal blocking
factor of the layer for dimensions ($IC_{B}$, $OC_{B}$), as depicted in Fig.
6.

Figure 6: Vector Processing Element (VPE) accommodating V$\times$V,
M$\times$V, and M$\times$M templates. In V$\times$V, accumulation involves
Inner Channels (ICs) of the same Output Channel (OC) of weights (FL), while
both M$\times$V and M$\times$M entail accumulation of ICs from different OCs.
This illustration presents the dataflow inside the VPE along with loop
blocking and partitioning for weights with FX=1 and FY=1.

For specific schedules, the convolution operation can be partitioned to split
across multiple VPEs based on ICs. Consequently, DNN computations that
generate partial sums (psums) across different sets of ICs for a particular OF
point should be mapped to a single column or row of VPEs. Internal psum
accumulation enables accumulation of all ICs partitioned into multiple VPEs to
generate the final OF point. The FPA facilitates the transmission of psums
formed within the PE to its right or top neighbor, which is essential for the
accumulation of internal psum within the FPA. To mitigate wire congestion and
routing complexity, interconnections between PEs are restricted to their top
and right neighbors. This architectural decision inherently influences how
work can be partitioned among different PEs, particularly in the IC dimension.

In certain optimal schedules, all input channels are not accumulated
simultaneously. Instead, a portion of the input channel set is initially
loaded into the PE RF, and the computed partial sum is extracted to the SRAM
(or even DRAM) to be brought back into the PE RFs later when the remaining
input channels are accumulated. External partial sum accumulation necessitates
a 32-bit wide read and write direct bypass to and from the SRAMs. Sharing
arithmetic units for MAC and Eltwise computation, along with multiplexer
control logic routing appropriate tensor data into these units, reduces area
overhead by enhancing hardware reuse efficiency within the PE. Residual
networks like ResNet require element-wise operations, such as the addition of
OFs from two convolution layers. To support such operations while maximizing
hardware resource reuse, OFs from two different layers are routed into the PE,
using existing load and drain paths. The Eltwise field in the programmable
descriptor signals an eltwise operation, bypassing the multiply operation
within the PE and performing an eltwise addition of the two IF inputs.

###  III-C Schedule-Aware Flexible Depth Adder Tree (FlexTree)

Our accelerator’s core features a tree-based architecture named FlexTree,
designed for partial sum accumulation across numerous PEs within a PE array to
generate the final output point [25]. What sets FlexTree apart is its dynamic
adjustment of the adder tree’s depth, enabling flexible schedules programmed
by the compiler for network layers of diverse dimensions. This hardware
enhancement allows the compiler/scheduler to discover highly compute-efficient
schedules. FlexTree achieves dynamic reconfiguration of the adder tree depth
through software-programmable configuration registers. Unlike existing DNN
accelerators where partial sum accumulation occurs by moving partial sums
among neighboring PEs, FlexTree’s innovative tree-based architecture
significantly enhances partial sum accumulation efficiency (up to $2.17\times$
speedup). In contrast to state-of-the-art DNN accelerators with fixed
schedules and adder tree-based architectures, where the adder tree depth
remains fixed at design time, our FlexTree technique offers dynamic
reconfiguration capabilities, achieving speedups of up to
$4\times$–$16\times$. Thus, our proposed FlexTree architecture enhances
compute efficiency by enabling superior partial sum accumulation techniques
across a wide range of network layers found in modern DNNs.

Before delving into the FlexTree architecture, it is essential to understand
the concept of Input Channel Partition ($IC_{P}$), similar to OF channel
partition as illustrated earlier in Fig. 3. $IC_{P}$ denotes how many ICs are
assigned to a single PE in the PE array. Consequently, this also denotes the
number of PEs that participate in the partial sum accumulation. Let us
elucidate $IC_{P}$ using an example of 64 input channels. When $IC_{P}$ is set
to 1, the computation involves only one processing element (PE), denoted PE0.
All 64 input channels undergo pointwise multiplication and accumulation within
PE0, producing the final output. In the case of $IC_{P}$ = 2, the 64 input
channels are evenly divided between PE0 and PE1, with each processing 32 input
channels. PE0 accumulates partial sums from input channels 0 to 31, while PE1
accumulates those from channels 32 to 63, forming the final output
collectively. Likewise, for $IC_{P}$ = 4, the input channels are distributed
in PE0, PE1, PE2, and PE3, with each PE handling 16 input channels. These
partial sums from the respective set of input channels are accumulated within
each PE to generate the final output. Essentially, the product of $IC_{P}$ and
Input Channel Blocking $IC_{B}$ should be equal to $IC$.

Fig. 7 illustrates the FlexTree architecture, which receives 16 inputs from
the 16 PEs within a column of the PE array in the DNN accelerator. $IC_{P}$
supported by the adder tree network ranges from 1 to 16, inclusively. Even if
$IC_{P}=2$, the output of the computation must still pass through the adder
tree network before yielding the final OF output. This ensures a reduction in
hardware overhead by simplifying hardware design and achieving uniformity
across all $IC_{P}$ values. It is noteworthy that our FlexTree architecture
can accommodate $IC_{P}$ values that are not powers of 2 by inputting zeros
into the FlexTree network of PEs that do not align with powers of 2. Each
module marked with a ‘+’ sign comprises both the INT8 adder and the FP16 adder
to support convolution layers of different precision [26]. Depending on the
input precision (INT8 vs. FP16), the partial sum output from the PEs is routed
to the appropriate hardware resource within FlexTree. In Fig. 7, for $IC_{P}$
values of [1, 2], the flops [A, B, C, D, E, F, G, H] at level 1 serve as the
final OF output tap points. For $IC_{P}$ = [4], the flops [I, J, K, L] at
level 2 act as the final OF output tap points. Similarly, for $IC_{P}$ values
of [8] and [16], the flops [M, N] at level 3 and [O] at level 4, respectively,
serve as the final OF output tap points. The total number of FlexTree output
tap points varies for different $IC_{P}$ values. Therefore, for $IC_{P}$
values of [1, 2, 4, 8, 16], the total number of FlexTree output tap points is
[8, 8, 4, 2, 1], respectively. To simplify the extraction of final OF points
from the FlexTree module into the drain module, we allow a maximum of 4 output
feature map (OF) points to be extracted from the FlexTree output. The
illustration shown in the figure assumes $IC=64$, $IC_{P}=16$ and therefore
Port O is active.

Figure 7: FlexTree architecture details with illustration using 64 input
channels and 16 input channel partition factor.

###  III-D Schedule-aware Tensor Distribution Network (SDN)

The Schedule-aware Tensor Distribution Network (SDN) serves as the core
infrastructure of the proposed accelerator, tasked with efficiently
transferring input data between on-chip memory (SRAM) and PE array, and vice
versa, adhering to the optimal schedule of the layer [27]. These data include
configuration settings, activation and kernel data (IF & FL), sparsity
encodings, as well as bias & scale factors essential for the calculation
within the PE array. Additionally, the SDN manages the transportation of
computational results, including output activation (OF) and partial sums, from
the PE array’s internal storage structure (RF) back to the SRAM, ensuring that
the layout facilitates subsequent tensor layer acceleration. During the
operational phases, the input side of the distribution network is termed the
“load/fill” phase, while the output side is termed the “drain” phase, as
illustrated in Fig. 8.1-2. In fixed hardware accelerators, the pre-determined
data layout in SRAM simplifies the load and drain phases, but compromises
flexibility and optimization in operations. This rigidity restricts reuse
potential, escalates memory accesses, and consequently increases overall
energy and power consumption. Flexible hardware demands dynamic changes in the
SRAM data layout, contingent on the type of reuse and optimal schedule
(blocking and partitioning) for the layer.

Figure 8: accelerator load and drain paths. (1) Activations and weights are
fetched from SRAM and delivered to PE array. (2) Output activations/partial
sums are fetched from PE array and routed to SRAM after compression in Sparse
Encoders in SDN. Flexible schedule support is integrated in load and drain
FSM.

The usual design consideration revolves around simplifying one phase, while
the other manages the complexities associated with rearranging the data to
adhere to the optimal schedule. When the SRAM data layout remains fixed, the
loading process must handle the complexity associated with unpacking the fixed
layout data and arranging them according to the predetermined order and
sequence dictated by the optimal schedule. Furthermore, the loading process
must be hierarchical: initially organizing the input in a manner consumable by
a column of PEs based on the partitioning factor and then, within the column,
determining which input byte corresponds to which PE based on the blocking
factor through a series of multiplexers [28, 29]. For activation data, this
involves retrieving data from memory in a predetermined order and distributing
the IX, IY, and IC in the sequence and quantity specified by the reuse factor
of the optimal schedule. Throughout the reuse process, one set of input
remains resident, while the other circulates multiple times. Typically, the
optimal schedule strives to maximize reuse, thereby reducing the frequency of
fetching from SRAM.

Figure 9: Data distribution patterns through flexible NoC.

For weight data, there exists an opportunity to prearrange the data offline
according to the optimal schedule before storing them in the SRAM. This
approach simplifies the load process by eliminating the need for data
rearrangement during weight distribution. Conversely, in cases where the
SRAM’s data layout is flexible rather than fixed, the load process is
simplified while the drain process is tasked with packing the output
activations according to the predetermined order and sequence outlined by the
optimal schedule. The drain component, responsible for gathering output points
from the PE array, must understand the sequence of OX, OY, and OC in which the
data are transmitted by the PEs. Then it rearranges or packs them based on the
IX, IY, and IC parameters of the optimal schedule for the subsequent layer.
However, if the data layout for the load is fixed, the drain only needs to
rearrange the output points into a predetermined layout (such as Z-major),
regardless of the optimal schedule for the next layer [30]. Another critical
aspect of the distribution network is the interconnect or Network-on-Chip
(NoC) used to link the PE array with the drain and load blocks in the design.
NoC must have the ability to unicast, multicast or broadcast the input data to
one or more PEs based on the order specified by the optimal schedule, as
demonstrated in Fig. 9. This maximizes reuse and minimizes the number of
accesses to SRAM, improving overall efficiency, as defined in existing
research [31].

###  III-E Two-sided Sparsity Acceleration

In the proposed accelerator, our aim is to harness the sparsity in both FLs
and IFs to enhance not only energy efficiency but also throughput. Throughout
the DNN framework, the data remains compressed until reaching the PE.
Operating within the compressed domain offers advantages, such as reducing on-
chip bandwidth requirements and storage demands, potentially leading to energy
savings and throughput enhancements. However, handling compressed data, which
often varies in length, poses challenges in terms of data manipulation, such
as distributing data across PEs and implementing sliding window processing
within the PE [32]. In this section, we will introduce an innovative two-sided
sparsity acceleration logic capable of processing sparse data within the
compressed domain to achieve higher throughput [33, 34, 35, 36].

The core idea is that IFs or FLs with a value of $0$ do not contribute to non-
zero outcomes during MAC operations, allowing them to be skipped during both
the compute and storage phases [37]. The DRAM serves as storage for zero-value
compressed input activations and weights, which are delivered to the load
module in batches [38, 34]. The load module then transmits the compressed
inputs and weights to their respective buffers. Along with these, the load
module also transfers the corresponding bitmaps to the input and weight bitmap
buffers, respectively. The sparsity acceleration module receives all of these
compressed data and bitmaps as input, and through a series of combinational
operations, it determines the exact non-zero values in the inputs and weights
that need to be computed in the dot product. These values are then fed into
the compute module, which generates the output feature maps. Subsequently,
these feature maps are stored in the store module, which incorporates a zero-
value compression module to compress the zero-valued elements. These
compressed feature maps are then stored back in the DRAM for further
processing in subsequent layers of the DNN.

The sparsity acceleration logic partitions the incoming input activations and
weights in blocks, and aims to find the participating pairs in the dot product
computation, by accounting for the fine grained sparsity. The logic starts off
with generating an activation and weight bitmap corresponding to the incoming
values. The bitmaps consist of 1-bit values, represented by either ‘0’ or ‘1’,
instead of 8-bit values present in the original activation and weight sets
(considering an 8-bit quantized network). For every non-zero element in the
activation set, the corresponding position in the activation bitmap consists
of ‘1’. The bitmap consists of a ‘0’ for every zero value in the activation
set. Similarly, the weight bitmap is also obtained from the incoming set of
weights. Subsequently, we also obtained a compressed activation and weight set
of values, which only consists of non-zero elements in the corresponding
original sets. The compressed sets, along with the two bitmaps, are
subsequently stored in the memory to be fetched during computation to perform
the MAC operation during inference, as demonstrated in Fig. 10.

During the compute phase, the two bitmaps, corresponding to activations and
weights, are logically AND-ed to obtain another intermediate bitmap. The total
number of $1$s in this intermediate bitmap depicts the total number of
activation and weight pairs that will result in non-zero partial sums, which
must be accumulated over time. Activations and weights at these non-zero
positions are provided as input to the PE, significantly reducing compute
operations. Through the combined sparsity acceleration logic, our accelerator
achieves enhanced computational speed, improving performance, and throughput
for DNN inference by exploiting two-sided sparsity. This approach notably
decreases the energy consumption of the Processing Element Array, completing
tasks with a reduced number of cycles.

Figure 10: Two-sided combined sparsity acceleration logic.

##  IV Experimental Setup

TABLE I: Comparison of with state-of-the-art fixed schedule accelerator
designs. Identical memory hierarchies and cost ratios are used for evaluation.

|  Eyeriss [5] |  TPU [6] |
---|---|---|---
Memory Hierarchy |  3-levelEyeriss has additional inter-PE communication with RF:PE=1:2 cost ratio |  3-level |  3-level
Num of PEs |  168 |  256 |  256
RF (in each PE) |  512 B |  32 B |  192 B
On-chip Buffer/SRAMSRAM sub-bank size remains constant for all |  108 KB |  64 KB |  1.5 MB
DRAM |  1 GB |  28 MB |  1 GB
Energy cost ratio (PE:RF:SRAM:DRAM) |  1:1:6:200 |  1:0.06:6:200 |  1:0.125:6:200

To evaluate the efficiency of the proposed accelerator, we implemented in
Chisel 3.0, with the generated RTL simulated in Synopsys VCS®. We chose Chisel
because of its ability to facilitate the generation of parametrizable designs
featuring multiple variations of PE, allowing easy modification of RF size,
number of MAC units, etc. As mentioned in Section III-A, the accelerator
supports UINT8, INT8, FP16, BF16 precision. Subsequently, the RTL undergoes
synthesis in the Synopsys Design Compiler (DC), utilizing one of the
industry’s most advanced process technology nodes, (based on 7 nm), to
generate the Gate-Level Netlist (GLS) and corresponding area for each
accelerator component. To estimate the power consumption within the proposed
accelerator, we employed Synopsys Verdi to generate an activity file
(Switching Activity Interchange Format: SAIF), using test benches for
assistance. The accelerator netlist, coupled with the activity file, serves as
input to Synopsys PrimeTimePX (PTPX), enabling power estimation at the gate
level for both block and full-chip designs of the accelerator. An overview of
this workflow is illustrated in Fig. 11. The architecture comprises a unified
tile of 256 PEs organized in a 16×16 grid (16 columns with each column having
16 individual PEs), featuring 8 MAC units within each PE, resulting in a total
of 2048 MACs. This tile encompasses 1.5 MB of SRAM equipped with 32-byte
read/write ports. The PE consists of 4x16 B IF RF Register File (RF), 4x16 B
FL RF, and 16x4 B OF RF, contributing to an overall 192 B RF per PE. The
precision of the IF, FL, and OF points is an 8bit integer. The memory
hierarchy of our design is illustrated in Table IV. Operating at a frequency
of 1.8 GHz and 0.75 volts, the accelerator boasts a dense peak Trillion
Operations Per Second (TOPS) performance, reaching 7.37 TOPS, with efficiency
metrics of 5 TOPS/watt and 4.6 TOPS/mm2.

We conducted a comparative analysis of the performance of in conjunction with
two state-of-the-art dense accelerators, namely Eyeriss [9] and TPU [13]. The
comparison considered various design specifications described in Table IV.
Furthermore, we evaluated the performance of on sparse DNN workloads using
state-of-the-art networks [39]: ResNet50, MobileNetV2, InceptionV3, and
GoogLeNet trained on the ImageNet dataset [40]. The first three models were
compressed using (i) Quantization-Aware Training (QAT) to quantize
weights/activations to INT8 precision and (ii) unstructured pruning using the
regularization-based sparsity algorithm (RB-sparsity). GoogLeNet was quantized
in the same way, but filter pruning with geometric median criterion was
applied. The compressed models were obtained from Intel’s Neural Network
Compression Framework (NNCF)[41]. Per-layer and overall network weight
sparsity were obtained from these models. Furthermore, all models were
subjected to inference on the entire ImageNet2012 validation dataset (50,000
images) and activation sparsity at input and output of each layer was
calculated using PyTorch hooks. The average activation sparsity across the
entire dataset, weight sparsity, and layer statistics were fed into a
framework of , which was used to obtain the layer-wise and overall network
compute acceleration and total energy consumption of the accelerator, reported
in Section V. Specifically, we compared the performance of , which uses two-
sided combined sparsity, against dense accelerators without any sparsity
support and those capable of exploiting fixed weight-sided sparsity [9, 42].
The framework was modified to evaluate the latency and energy of a dense
variant and a weight-sided variant of to allow fair comparison.

Figure 11: Workflow overview for implementation. Please refer to text for
full-form of abbreviations.

##  V Experimental Results

Figure 12: Area and Power Breakdown of for (1) overall accelerator and (2) PE
level granularity.

In this section, we begin by providing a breakdown of the power and area
consumption for our proposed accelerator. Subsequently, we proceed to evaluate
its performance using state-of-the-art network and dataset configurations.

###  V-A Power and Area Results

We assess the power and area cost of the proposed accelerator using an
illustrative implementation in this paper. Fig. 12.1 and 2 show the power and
area breakdown of the entire accelerator as well as the inter-PE breakdowns,
respectively. As shown, the PE array unit consumes $\approx 83\%$ of power and
$\approx 86\%$ of the total area of the overall accelerator design. Moreover,
MAC operation constitutes about $46\%$ of the power and $54\%$ of the area
inside each PE of the accelerator. This shows reasonable power and area impact
when compared to the significant benefits it provides in terms of the ability
to support flexible schedules.

###  V-B Comparison with SOTA Fixed Schedule Accelerators

Fig. 13 shows the improvement in energy efficiency of our flexible schedule
DNN accelerator over two prominent fixed-schedule designs, Eyeriss [5] and TPU
[6] assuming identical memory hierarchies. These results are obtained for two
DNNs used in image recognition and object detection, ResNet101 and YOLOv2,
using our custom DNN accelerator energy estimation framework. We have used
dense models (i.e., with 0 weight sparsity) for these results. Note that we
have scaled the memory hierarchy of the two accelerators to the same level as
for a fair comparison. In this figure, Here, the y-axis represents a %
reduction in energy consumption of compared to these two designs. The left
subplot depicts the layer-wise energy reduction for all layers, sorted in
increasing order of reduction. In the right subplot, we summarize the
distribution of reduction across all layers. The x-axis shows the two
comparative accelerators. Compared to Eyeriss, results in $40\%$–$77\%$
reduction for ResNet101 and $45\%$–$77\%$ for YOLOv2. Compared to TPU,
provides up to $62\%$ and $58\%$ energy savings for ResNet101 and YOLOv2,
respectively. While it is true that in certain layers, exhibits a slight
energy increase (indicated by negative energy reduction) compared to TPU, this
is primarily attributed to the optimized dataflow on TPU for specific layers,
particularly 20 layers in ResNet101 and 4 layers in YOLOv2. However, on
average, still provides notable advantages, offering average energy savings of
$14\%$ and $22\%$ for these respective DNN architectures over TPU. It is
important to note that the heightened energy consumption in these layers stems
from the robust support for flexibility within DNN, which inherently
introduces slightly higher overhead. On the other hand, we see an average
improvement of $57\%$ and $69\%$ over Eyeriss. Despite occasional spikes in
energy consumption for select layers, consistently outperforms these fixed-
schedule accelerators, showcasing its superior efficiency and overall cost-
effectiveness.

Figure 13: Layer-wise distribution of % energy improvement of over fixed
schedule DNN accelerators (Table IV) for ResNet101 and YOLOv2 (dense models).
Optimal schedule used for each layer in . Figure 14: Comparison of layerwise
compute acceleration of and Weight-sided (one-sided) sparse accelerator over
dense accelerator (without any sparsity support) benchmarked with (1)
ResNet50, (2) MobileNetV2, (3) GoogLeNet, (4) InceptionV3. Only a few
representative layers are presented for each network.

###  V-C Sparsity Benefits using

In this section, we present a comprehensive analysis of the layer-wise and
overall network speed-up achieved by compared to two prominent counterparts: a
dense accelerator without any sparsity acceleration support and a fixed
weight-sided sparse accelerator. Fig. 14.1-4 presents the layer-wise compute
acceleration (y-axis) provided by weight-sided and in comparison with the
dense accelerator, for few representative layers (x-axis) of 4 DNN benchmarks.
Note that the activation sparsity numbers reported in the following discussion
are averaged across the entire dataset. For a fair comparison, benchmarking
was performed using the same optimal schedule for all accelerator types.

####  V-C1 ResNet50

The sparse ResNet50 model has $5\%$–$88\%$ unstructured weight sparsity,
$weight\\_sp_{layer}$, resulting in up to $8.1\%$ acceleration across layers
in weight-sided accelerator. However, except before the first conv layer,
ResNet50 has a high activation sparsity, $act\\_sp_{layer}$, at the input of
every convolution layer due to the presence of the ReLU activation function.
On average across the entire ImageNet validation dataset, this amounts to
$act\\_sp_{layer}$ = $14\%$–$83\%$ sparsity. conveniently leverages both
weight and activation sparsity to provide up to $10.3\%$ compute acceleration,
as shown in Fig. 14.1. Overall, gives up to $3.1\times$ better acceleration
than the weight-sided accelerator for ResNet50.

####  V-C2 GoogLeNet

Since GoogLeNet was filter-pruned, maximum $weight\\_sp_{layer}$ = $30\%$.
This contributed to maximum $1.4\times$ speed-up in weight-sided accelerator.
In contrast, the maximum measured $act\\_sp_{layer}$ = $91\%$ resulted in a
maximum acceleration $10.8\times$ in . Fig. 14.3 shows that provides up to
$7.7\times$ better compute acceleration compared to fixed weight-sided
accelerator, even for networks with low weight sparsity.

####  V-C3 InceptionV3

This model is very sparse with a maximum $weight\\_sp_{layer}$ = $96\%$. There
are many layers with large dimensions and filter sizes; therefore, both the
weight-sided accelerator and can leverage weight sparsity and provide up to
$24.7\times$ speed-up for Mixed.7a.branch3x3.2.conv, Layer Id: 72 (not shown
in figure). Although $act\\_sp_{layer}$ for this layer is $78\%$, cannot
provide any additional speed-up for this layer. However, there are many other
layers with activation sparsity higher than weight sparsity, allowing to
leverage both. As depicted in Fig. 14.4, provides a high level of compute
acceleration. Among such layers with sparsity skewed toward activations, the
maximum speed-up is $11.3\times$. Therefore, the proposed design can take the
best of both worlds and give better savings than the weight-sided accelerator.
Across all layers, is up to $4.3\times$ faster than the weight-sided
accelerator, clearly demonstrating superior performance.

####  V-C4 MobileNetV2

MobileNetV2 is a compact and lightweight model compared to the other
benchmarks discussed earlier. Although sparse MobileNetV2 consists up to
$70\%$ $weight\\_sp_{layer}$, the maximum speed-up provided by the weight-
sided accelerator is only $3.3\times$ (last linear layer). Interestingly, the
weight sparsity of all conv layers, except features.18.0, Layer Id: 51 is
$<50\%$ leading to a low overall speed-up. However, leveraging activation
sparsity (maximum $act\\_sp_{layer}$ = $74\%$) in addition to weights can
provide up to $4.1\times$ acceleration. Fig. 14.2 indicates that even for
compact models with small layer sizes, is superior to the weight-sided
accelerator by $3.9\times$.

Figure 15: Comparison of compute acceleration for full network inference in
over dense and weight-sided accelerator benchmarked with 4 DNNs.

####  V-C5 Overall Network Acceleration

The compute acceleration obtained by dense, weight-sided and our proposed
accelerator for the entire end-to-end network inference, depicted in Fig. 15
reveal a significant acceleration advantage conferred by across all evaluated
networks. Here, the y-axis represents the acceleration, whereas the x-axis
represents benchmarks. The dense accelerator does not provide any acceleration
as it cannot leverage weight or activation sparsity, denoted by values $1$.
Evidently, the speed-up for weight-sided accelerator is proportional to the
overall network weight sparsity. Across all these networks, the weight-sided
accelerator provides $1.2\times$–$1.7\times$ speed-up. On the contrary, the
acceleration obtained in is proportional to the relative distribution of
weight and activation sparsity. For ResNet50,
$weight\\_sp_{network},act\\_sp_{network}$ = $61\%,55\%$ and takes advantage
of them to provide $2.2\times$ speed-up. MobileNetV2 and GoogLeNet has
$weight\\_sp_{network},act\\_sp_{network}$ = $52\%,30\%$ and
$weight\\_sp_{network},act\\_sp_{network}$ = $24\%,58\%$, respectively. These
results in $2.1\times$ and $2.0\times$ speed-up, respectively. Clearly, even
with these two networks with low sparsity on one side, provides a similar
speed-up to ResNet50 due to two-sided sparsity support. Finally, InceptionV3
has $weight\\_sp_{network},act\\_sp_{network}$ = $61\%,63\%$ contributing to
$2.7\times$, which is the maximum across the 4 networks. As evident from these
results, our accelerator consistently outperforms both dense and weight-sided
architectures in terms of compute acceleration. This substantial improvement,
$2.2\times$ vs. dense and $1.6\times$ vs. weight-sided accelerator (on
average), underscores the efficacy of our proposed approach in enhancing
overall network speed-up, demonstrating its superiority in accelerating DNN
inference computations. Furthermore, the observed acceleration benefits are
valid across the various architectural complexities and model sizes
represented by the diverse DNNs considered in our evaluation. This robust
performance underscores the versatility and effectiveness of two-sided
sparsity acceleration support in across a spectrum of deep learning model
architectures.

Figure 16: Comparison of energy efficiency for full network inference in over
dense and weight-sided accelerator benchmarked with 4 DNNs.

####  V-C6 Energy Efficiency improvement

Figure 16 presents the improvement in energy efficiency of 3 different
accelerator architectures (y-axis) while evaluating 4 DNN benchmarks (x-axis)
on the ImageNet validation dataset. We considered dense accelerator energy
consumption as the baseline. As evident from the figure, these results largely
correlate with overall network compute acceleration in Fig. 15 since the
accelerator circuits are active for a reduced amount of time. Furthermore,
compared to the weight-sided accelerator, allows for substantial reduction in
memory cycle count as ZVC compressed data flows through the different memory
hierarchies, resulting in reduced memory energy consumption. This is enabled
by the sparsity-aware load and drain path, as explained in Section III-D. Note
that DRAM transactions are not considered in these results. Across all 4
benchmarks, is $2.1\times$ and $1.5\times$ more energy efficient than the
dense and weight-sided accelerators, respectively.

In conclusion, our comprehensive evaluation showcases not only the substantial
compute acceleration achieved by our proposed accelerator, but also its
remarkable energy efficiency improvements compared to existing dense and
weight-sided architectures. This underscores the pivotal role of our approach
in addressing the dual challenges of performance enhancement and energy
conservation in DNN accelerators, paving the way for sustainable and efficient
AI hardware solutions.

##  VI Related Work

Recent years have witnessed a surge in the field of DNN accelerators. Most DNN
accelerator designs only implement fixed schedules with fixed dataflow. Fig.
17 illustrates the different DNN accelerators from industry and academia and
their supported datatypes. For example, NeuFlow [43] and ISAAC [44] implement
a weight stationary schedule, ShiDianNao [45] and Movidius VPU2 [46] implement
an output stationary schedule, Google TPU [6] only implements Nonlocal Reuse
schedule, and Eyeriss [5] from MIT implements a row stationary schedule. A key
challenge arises from the limitations of the tensor data PE module hardware,
which operates solely on a fixed dataflow pattern. It lacks the ability to
dynamically adjust to accommodate diverse schedules, as it lacks awareness of
any schedule information, owing to its restricted functionality. Therefore,
one cannot implement different schedules (i.e. dataflows) in these
accelerators, and till today there are no existing accelerators that can
support flexible schedules. In addition to hardware solutions, software-based
solutions can mimic programmable PE array units that can perform computation
on varying dataflow tensor data in general-purpose CPUs and GPUs, but fixed-
function accelerators do not support this flexibility in design. Therefore,
these software solutions cannot be used in existing accelerators. Moreover,
software solutions are far from being energy optimal to be considered for
adoption in edge inference devices. FPGAs provide an alternative avenue for
DNN acceleration with flexibility, but the hardware configuration of the FPGA
cannot be changed during the execution of one DNN application, which also
implies a fixed schedule during execution. Additionally, FPGAs have lower
energy efficiency compared to ASIC hardware accelerators.

Figure 17: Edge DNN accelerator competitive landscape, plotted with public
data, considering baseline reference [14].

Since the PE array modules in all previous designs have limited
functionalities in the form of a basic MAC structure, which is the compute
kernel for convolution operation, the wide degree of mismatch between the data
patterns required by different input, weight, and output stationary optimal
schedules makes it impossible for the fixed architecture PE array to be able
to handle the tensor data correctly without sacrificing energy and/or
performance. The key disadvantage is that the PE array that performs
convolution computation in these previous solutions is not schedule aware. Due
to this limitation, reformatting the energy optimal dataflow type into a
dataflow that is supported by the underlying fixed architecture PE array
induces severe performance and energy penalty as more SRAM reads are required
to complete the work and prevents the PE array from reaching maximum
utilization if the accesses are serialized. Software solutions can also be
used for rearranging the input, output activation as well as weight tensor
data for different optimal schedules to be fed into the PE array, into the
type of fixed dataflow supported by the PE array, which not only would require
assisting CPUs but would also be highly energy and performance inefficient,
thereby significantly diminishing the energy efficiency gains offered by
flexible scheduling.

In contrast to these approaches, we propose a schedule-aware runtime
configurable PE array module, which can (1) process tensor data (both weights
and activations) that are either input, output, or weight stationary, or even
a mixture of these dataflow types, depending upon the energy optimal schedule
for the current DNN layer, (2) have its microarchitecture be reconfigurable at
runtime based on software-programmable configuration registers, and (3)
leverage maximum activation and weight reuse by having small amount of
distributed local storage close to compute within the PE array itself. The
dataflow support in the proposed PE array module is flexible. It is controlled
by a list of configuration descriptors, which are set at the beginning of the
execution of each layer. This tensor data computation PE array module is a
pure hardware solution that exposes hardware knobs to the compiler and
configures the dataflow during runtime, enabling the flexible schedules of
convolutional layers in DNN accelerators without performance penalty due to
rearranging computation within the PE or having to offload any work to CPU or
software.

##  VII Conclusion

In this paper, we proposed a flexible schedule-aware DNN accelerator , which
can adapt its internal dataflow to the optimal schedule of each layer in DNNs.
Our proposed solution maximizes data reuse at each memory level, resulting in
significant energy savings arising from optimal data reuse. Note that
flexibility works seamlessly on top of existing performance-enhancing features
such as sparsity acceleration and low-precision logic, and it does not
diminish their impact in any manner. It is evident that this flexibility comes
at the cost of additional area overhead compared to fixed dataflow
accelerators, but it also enables us to achieve significant energy savings on
average across a myriad of DNN layers. Furthermore, we propose a novel
approach to improve throughput and reduce energy usage in the architecture.
Taking advantage of fine-grained sparsity in both activation and weight
tensors, we optimize the inference engine within the hardware accelerator.
Experimental results demonstrate significant improvements in both performance
and energy efficiency compared to existing DNN accelerators. This research
contributes to ongoing efforts to develop more efficient hardware accelerators
for executing deep neural networks.

##  VIII Acknowledgements

We would like to sincerely thank Gautham Chinya, Debebrata Mohapatra, Huichu
Liu, Moongon Jung, Sang Kyun Kim, Guruguhanathan Venkataramanan, Raymond Sung,
Hong Wang, and Cormac Brick for their contributions to this work.

## References

  * [1] A. Raha, R. Sung, S. Ghosh, P. K. Gupta, D. A. Mathaikutty, U. I. Cheema, K. Hyland, C. Brick, and V. Raghunathan, “Efficient hardware acceleration of emerging neural networks for embedded machine learning: An industry perspective,” in _Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing: Hardware Architectures_. Springer, 2023, pp. 121–172.
  * [2] A. Raha, S. K. Kim, D. A. Mathaikutty, G. Venkataramanan, D. Mohapatra, R. Sung, C. Brick, and G. N. Chinya, “Design considerations for edge neural network accelerators: An industry perspective,” in _2021 34th International Conference on VLSI Design and 2021 20th International Conference on Embedded Systems (VLSID)_. IEEE, 2021, pp. 328–333.
  * [3] S. K. Ghosh, A. Raha, and V. Raghunathan, “Energy-efficient approximate edge inference systems,” _ACM Transactions on Embedded Computing Systems_ , vol. 22, no. 4, pp. 1–50, 2023.
  * [4] A. Raha, S. Ghosh, D. Mohapatra, D. A. Mathaikutty, R. Sung, C. Brick, and V. Raghunathan, “Special session: Approximate tinyml systems: Full system approximations for extreme energy-efficiency in intelligent edge devices,” in _2021 IEEE 39th International Conference on Computer Design (ICCD)_. IEEE, 2021, pp. 13–16.
  * [5] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in _Proceedings of the 43rd International Symposium on Computer Architecture_ , ser. ISCA ’16, 2016.
  * [6] N. P. Jouppi and et al., “In-datacenter performance analysis of a tensor processing unit,” 2017.
  * [7] H. Kwon, P. Chatarasi, M. Pellauer, A. Parashar, V. Sarkar, and T. Krishna, “Understanding reuse, performance, and hardware cost of dnn dataflow: A data-centric approach,” in _Proc. MICRO_ , ser. MICRO ’52. New York, NY, USA: Association for Computing Machinery, 2019, p. 754–768.
  * [8] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” _ACM SIGARCH computer architecture news_ , vol. 44, no. 3, pp. 367–379, 2016.
  * [9] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” _IEEE journal of solid-state circuits_ , vol. 52, no. 1, pp. 127–138, 2016.
  * [10] A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, “Scnn: An accelerator for compressed-sparse convolutional neural networks,” _ACM SIGARCH computer architecture news_ , vol. 45, no. 2, pp. 27–40, 2017.
  * [11] S. K. Ghosh, S. Kundu, A. Raha, D. A. Mathaikutty, and V. Raghunathan, “Harvest: Towards efficient sparse dnn accelerators using programmable thresholds,” in _2024 37th International Conference on VLSI Design and 2021 20th International Conference on Embedded Systems (VLSID)_. IEEE, 2024.
  * [12] Chen _et al._ , “Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 9, no. 2, pp. 292–308, 2019.
  * [13] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers _et al._ , “In-datacenter performance analysis of a tensor processing unit,” in _Proceedings of the 44th annual international symposium on computer architecture_ , 2017, pp. 1–12.
  * [14] M. Horowitz, “1.1 Computing’s energy problem (and what we can do about it),” in _Proc. ISSCC_ , 2014.
  * [15] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, “Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks,” _JMLR_ , vol. 22, no. 1, pp. 10 882–11 005, 2021.
  * [16] J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, “Cnvlutin: Ineffectual-neuron-free deep neural network computing,” _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3, pp. 1–13, 2016.
  * [17] S. Zhang, Z. Du, L. Zhang, H. Lan, S. Liu, L. Li, Q. Guo, T. Chen, and Y. Chen, “Cambricon-x: An accelerator for sparse neural networks,” in _2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2016, pp. 1–12.
  * [18] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, “Eie: Efficient inference engine on compressed deep neural network,” _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3, pp. 243–254, 2016.
  * [19] X. Yang, M. Gao, Q. Liu, J. Setter, J. Pu, A. Nayak, S. Bell, K. Cao, H. Ha, P. Raina _et al._ , “Interstellar: Using halide’s scheduling language to analyze dnn accelerators,” in _Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2020, pp. 369–383.
  * [20] H. Kwon, P. Chatarasi, V. Sarkar, T. Krishna, M. Pellauer, and A. Parashar, “Maestro: A data-centric approach to understand reuse, performance, and hardware cost of dnn mappings,” _IEEE micro_ , vol. 40, no. 3, pp. 20–29, 2020.
  * [21] D. Mohapatra, A. Raha, G. Chinya, H. Liu, C. Brick, and L. Hacking, “Configurable processor element arrays for implementing convolutional neural networks,” Apr. 30 2020, uS Patent App. 16/726,709.
  * [22] S. Hsu, A. Agarwal, D. Mohapatra, A. Raha, M. Jung, G. Chinya, and R. Krishnamurthy, “Multi-buffered register files with shared access circuits,” Apr. 22 2021, uS Patent App. 17/132,895.
  * [23] A. Raha, D. Mohapatra, G. Chinya, G. Venkataramanan, S. K. Kim, D. Mathaikutty, R. Sung, and C. Brick, “Performance scaling for dataflow deep neural network hardware accelerators,” Sep. 2 2021, uS Patent App. 17/246,341.
  * [24] D. Mohapatra, A. Raha, D. A. Mathaikutty, R. J.-H. Sung, and C. M. Brick, “Runtime configurable register files for artificial intelligence workloads,” Mar. 10 2022, uS Patent App. 17/530,156.
  * [25] D. Mohapatra, A. Raha, D. Mathaikutty, R. Sung, and C. Brick, “Schedule-aware dynamically reconfigurable adder tree architecture for partial sum accumulation in machine learning accelerators,” Apr. 28 2022, uS Patent App. 17/520,281.
  * [26] A. Raha, M. A. Anders, R. J.-H. Sung, D. Mohapatra, D. A. Mathaikutty, R. K. Krishnamurthy, and H. Kaul, “Floating point multiply-accumulate unit for deep learning,” Jun. 16 2022, uS Patent App. 17/688,131.
  * [27] G. Chinya, H. Liu, A. Raha, D. Mohapatra, C. Brick, and L. Hacking, “Schedule-aware tensor distribution module,” Feb. 20 2024, uS Patent 11,907,827.
  * [28] D. Mathaikutty, A. Raha, R. Sung, D. Mohapatra, and C. Brick, “Sparsity-aware datastore for inference processing in deep neural network architectures,” Mar. 3 2022, uS Patent App. 17/524,333.
  * [29] D. A. Mathaikutty, A. Raha, R. J.-H. Sung, and D. Mohapatra, “Data reuse in deep learning,” Jun. 16 2022, uS Patent App. 17/684,764.
  * [30] A. Raha, D. A. Mathaikutty, U. I. Cheema, and D. Kondru, “Output drain path facilitating flexible schedule-based deep neural network accelerator,” Jan. 11 2024, uS Patent App. 18/474,464.
  * [31] H. Kwon, A. Samajdar, and T. Krishna, “Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable interconnects,” _ACM SIGPLAN Notices_ , vol. 53, no. 2, pp. 461–475, 2018.
  * [32] A. Raha, D. Mohapatra, D. A. Mathaikutty, R. J.-H. Sung, and C. M. Brick, “System and method for balancing sparsity in weights for accelerating deep neural networks,” Mar. 17 2022, uS Patent App. 17/534,976.
  * [33] G. Chinya, D. Mathaikutty, G. Venkataramanan, D. Mohapatra, M. Jung, S. K. Kim, A. Raha, and C. Brick, “Accelerated loading of unstructured sparse data in machine learning architectures,” Feb. 11 2021, uS Patent App. 17/081,509.
  * [34] A. Raha, D. Mathaikutty, D. Mohapatra, S. K. Kim, G. Chinya, and C. Brick, “Methods and apparatus to load data within a machine learning accelerator,” Oct. 21 2021, uS Patent App. 17/359,392.
  * [35] A. Raha, M. Langhammer, D. Mohapatra, N. Tunali, and M. Wu, “Methods and apparatus to perform low overhead sparsity acceleration logic for multi-precision dataflow in deep neural network accelerators,” Sep. 15 2022, uS Patent App. 17/709,337.
  * [36] S. Kundu, A. Raha, D. A. Mathaikutty, and K. Basu, “Rash: Reliable deep learning acceleration using sparsity-based hardware,” in _2024 25nd International Symposium on Quality Electronic Design (ISQED)_. IEEE, 2024.
  * [37] F. Connor, D. Bernard, and N. Hanrahan, “Dot product calculators and methods of operating the same,” U.S. Patent 10,768,895 B2, Sep. 8, 2020.
  * [38] G. Chinya, D. Mohapatra, A. Raha, H. Liu, and C. Brick, “Methods, systems, articles of manufacture, and apparatus to decode zero-value-compression data vectors,” Oct. 31 2023, uS Patent 11,804,851.
  * [39] M. AI, “The latest in machine learning — papers with code,” https://paperswithcode.com/.
  * [40] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [41] A. Kozlov, I. Lazarevich, V. Shamporov, N. Lyalyushkin, and Y. Gorbachev, “Neural network compression framework for fast model inference,” _arXiv preprint arXiv:2002.08679_ , 2020.
  * [42] J. Park, H. Yoon, D. Ahn, J. Choi, and J.-J. Kim, “Optimus: Optimized matrix multiplication structure for transformer neural network accelerator,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 363–378, 2020.
  * [43] C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun, “Neuflow: A runtime reconfigurable dataflow processor for vision,” in _CVPR 2011 WORKSHOPS_ , 2011, pp. 109–116.
  * [44] A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,” in _2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA)_ , 2016, pp. 14–26.
  * [45] Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “Shidiannao: Shifting vision processing closer to the sensor,” in _2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA)_ , 2015, pp. 92–104.
  * [46] Intel, “Intel keembay,” https://newsroom.intel.com/wp-content/uploads/sites/11/2019/11/intel-ai-summit-keynote-slides.pdf.