# S2TA: Exploiting Structured Sparsity for
Energy-Efficient Mobile CNN Acceleration

Zhi-Gang Liu  Paul N. Whatmough  Yuhao Zhu  University of Rochester,
Rochester, NY, USA  Matthew Mattina  Arm ML Research Lab, Boston, MA, USA

###### Abstract

Exploiting sparsity is a key technique in accelerating quantized convolutional
neural network (CNN) inference on mobile devices. Prior sparse CNN
accelerators largely exploit unstructured sparsity and achieve significant
speedups. Due to the unbounded, largely unpredictable sparsity patterns,
however, exploiting unstructured sparsity requires complicated hardware design
with significant energy and area overhead, which is particularly detrimental
to mobile/IoT inference scenarios where energy and area efficiency are
crucial.

We propose to exploit structured sparsity, more specifically, Density Bound
Block (DBB) sparsity for both weights and activations. DBB block tensors bound
the maximum number of non-zeros per block. DBB thus exposes statically
predictable sparsity patterns that enable lean sparsity-exploiting hardware
and efficient memory access. We propose new hardware primitives to implement
DBB sparsity for (static) weights and (dynamic) activations, respectively,
with very low overheads.

Building on top of the primitives, we describe S2TA, a systolic array-based
CNN accelerator that exploits joint weight and activation DBB sparsity and new
dimensions of data reuse unavailable on the traditional systolic array. S2TA
in 16nm achieves more than 2$\times$ speedup and energy reduction compared to
a strong baseline of a systolic array with zero-value clock gating, over five
popular CNN benchmarks. Compared to two recent non-systolic sparse
accelerators, Eyeriss v2 (65nm) and SparTen (45nm), S2TA in 65nm uses about
2.2$\times$ and 3.1$\times$ less energy per inference, respectively.

##  1 Introduction

††† Denotes equal contribution.

Convolutional neural network (CNN) inference has quickly become an important
workload in (ultra) low-power mobile [48, 12] and IoT/embedded [21, 10, 11, 5]
devices. CNN accelerators are now a standard component in mobile SoCs [42, 24,
17, 47], where 8-bit integer (INT8) CNN inference is the most widely used [41]
due to the stringent requirements on energy efficiency (TOPS/W) and area
efficiency (TOPS/mm2).

A common strategy to improve CNN accelerator efficiency is to exploit
sparsity, as zeros in the data tensors (both weights and activations) reduce
the theoretical compute and storage requirement significantly. Zeros in CNNs
are statistically distributed in a random pattern. Exploiting the random
sparsity, which is also commonly referred to as unstructured sparsity, has
been the main focus of sparse hardware accelerators to date [15, 30, 36, 13,
6]. Exploiting unstructured sparsity, however, requires complex hardware
structures that introduce significant area and energy overhead, which are
particularly detrimental to mobile/embedded inference scenarios.

Exploiting unstructured sparsity introduces hardware overheads due to
additional buffers used in data manipulation. There are two fundamental
approaches to supporting random sparsity. The first is the inner-product style
[38, 6] which requires an operand gather stage to evenly distribute the
unpredictable workload to the PEs to perform the MAC operations. The second,
is the outer-product style which has more conventional operand distribution
and MAC operations, but then requires a result scatter stage to spread the
non-contiguous results of the individual MACs over the output feature maps.
Both of these approaches introduce significant hardware overheads in the form
of large buffers that drastically degrade the energy and area-efficiency of
the accelerator.

We quantitatively show that the additional buffers required to exploit
unstructured sparsity, which may seem like nitty-gritty engineering details,
in fact increase the energy per MAC of a baseline (systolic array) dense
accelerator by 71%. As a result, random sparse accelerators published to date
have demonstrated high speedup, but due to hardware overheads, have limited
energy and area efficiency gains.

We argue that the sparsity-exploiting microarchitecture structures must be
lightweight for it to be beneficial to exploit sparsity at all in
mobile/embedded CNN accelerators. Otherwise, the energy/area overheads can
easily eclipse the speedup gains. To that end, we propose to exploit
structured sparsity, which has regular sparsity patterns that allow the
hardware additions required to be very lean. In particular, we focus on
Density Bound Block (DBB) [26] format, which tiles data tensors into blocks,
and then introduces a bound on the maximum number of non-zero elements per
block. DBB overcomes both of the challenges with random sparsity. Firstly, the
maximum number of MAC operations is fixed, which significantly achieves high
PE utilization without operand buffers. Secondly, the blocked data limits the
location of output results, eliding distributed accumulators.

We first propose two primitives: W-DBB and A-DBB, which exploit DBB in weight
and activation sparsity, respectively. Weight sparsity is statically known,
and thus the hardware design is relatively straightforward. Activation
sparsity, however, is dynamic. We propose a dynamic pruning scheme co-designed
with a novel time-unrolled architecture to implement A-DBB.

Building on W-DBB and A-DBB, we describe how to integrate the two building
blocks into a fully-sparse CNN accelerator that efficiently exploits sparsity
in both weights and activations. In particular, we focus on the well-known
systolic array architecture, popular in mobile/embedded inference scenarios
due to its extreme efficiency arising from high data reuse. We show that both
W-DBB and A-DBB can be easily integrated into a classic SA architecture by
introducing a tensor PE (TPE) design enabling further efficiency gains.

Fig. 1:  Energy breakdown for a conventional dense INT8 systolic array
accelerator, with typical 50% CNN sparsity. The INT8 MAC datapath itself is
very compact (consuming only 20% of the energy), while the operand/result
buffers dominate.

In summary, this paper makes the following contributions:

  * •

Quantifying Overhead of Exploiting Unstructured Sparsity We provide the first
quantitative analysis of the hardware overheads (area and energy) of
exploiting unstructured sparsity. We show that the additional operand and
accumulator buffers introduce about 50% and 10% energy and area overhead
compared to the datapath of a baseline dense INT8 CNN accelerator. These
overheads are amortized in floating-point designs [27, 32, 38], where the
datapath power is much higher, but in the mobile INT8 case, these overheads
eclipse the gains.

  * •

Joint Weight/Activation DBB Sparsity We propose an architecture that exploits
DBB sparsity in both weights and activations. This is non-trival, because
while weight sparsity is known offline, activation sparsity is not known until
runtime and can vary wildly. We propose Dynamic Activation Pruning (DAP),
which co-designs training time activation DBB pruning and novel run-time
hardware support. DAP compresses activations by 2–3$\times$ with negligible
impact on test accuracy.

  * •

Time-Unrolled Variable DBB Sparsity We propose a new time-unrolled
microarchitecture to serialize the processing of MACs in a DBB block across
time.

  * •

Structured Sparse Tensor Accelerator (S2TA) We show that joint DBB weight and
activation sparsity can be efficiently incorporated into CNN accelerators. As
a case study, we target the systolic array template, by extending the
traditional scalar PE into a tensor PE (TPE) that consumes compressed DBB data
blocks.

  * •

Evaluation in 16nm and 65nm We implement the S2TA design in both 16nm and 65nm
technology. On a range of popular models (AlexNet, MobileNetv1, VGG-16,
ResNet50v1), S2TA demonstrates 2.08$\times$ lower energy compared to the
baseline without DBB. Compared to the state-of-the-art unstructured sparse
accelerator SparTen [13] and Eyeriss-v2 [6], S2TA has 2.2$\times$ and
3.1$\times$ lower energy on AlexNet in 65nm technolgy.

##  2 Motivation

In this section, we review the two fundamental approaches to supporting random
sparsity in embedded CNN accelerators: (1) inner-product style with operand
gather, and (2) outer-product style with result scatter. Both introduce
significant hardware overheads that drastically degrade the energy and area-
efficiency of the accelerator.

###  2.1 Sparsity-Exploiting Structures Must be Efficient for Mobile DNN
Inference

The simplest way to exploit sparsity in hardware is Zero Value Clock Gating
(ZVCG). Fig. 6b shows how ZVCG simply detects zero operands (weights and
activations) and clock-gates the operand and/or result registers to reduce
power dissipation. While ZVCG gives a significant reduction in datapath power
[7, 33], it does not increase throughput, nor does it reduce the SRAM
bandwidth (as the zeros are still stored and read in sequence). More
importantly, ZVCG reduces the hardware utilization and, thus, does not improve
area efficiency (TOPS/mm2), which is critical for mobile.

Sparse GEMM potentially achieves much higher gains than ZVCG by reducing both
the number of MACs that need to be executed and also the memory bandwidth. To
that end, only the non-zero values and a corresponding positional index are
stored. The index encodes the position of each non-zero element in the
expanded matrix, via either compressed sparse row/column (CSR/CSC) [15, 30]
format or a simple bitmask [13]. At runtime, both the SRAM bandwidth, and the
number of MACs executed are significantly reduced.

However, implementing fully sparse GEMM on energy and area-constrained INT8
mobile/IoT accelerators is challenging. Basically, removing MACs with zero
operands breaks the regular compute pattern and requires complex on-chip
buffering and data re-ordering to maximize the hardware utilization. The
additional buffers significantly increase the energy and area overhead. Fig. 1
shows the energy breakdown of an INT8 dense systolic array accelerator for a
typical CNN layer. The data is obtained from the extracted post-layout power
estimation in a 16nm technology node with fully annotated switching activity.

#### Key Insight

The energy consumption of the actual INT8 MAC computation in Fig. 1 is
significantly overshadowed by the buffers used for operands and accumulators.
Therefore, any sparsity-exploiting scheme must not introduce significant
overheads that exacerbate the data buffering cost, as that is already the
dominant energy consumer.

Today’s sparsity-exploiting mechanisms, unfortunately, introduce significant
area and energy overhead for data buffering due to non-trivial data re-
ordering, which can be classified into two fundamental categories, as
illustrated in Fig. 2. The first category (Fig. 2a) performs a gather
operation in the front-end to collect matching pairs of non-zero operands
before buffering and finally performing MAC computations in the conventional
manner. This approach is seen in previous work, including EIE [15], Eyerissv2
[6], and SMT-SA [38]. The second approach (Fig. 2b) avoids re-ordering
operands in the front-end, by computing an outer-product (i.e. multiplying all
non-zero weights with all non-zero activations). This approach is seen in
accelerators such as SCNN [30] and SparTen [13]. However, this requires a
scatter operation on the partial products using a very large number of read-
modify-write accumulators to store the output activations. The rest of this
section quantifies the overhead associated with these two approaches.

###  2.2 Overhead 1: Operand Gather Structure

(a) Inner-Product (Eyerissv2) (b) Outer-Product (SCNN)

Fig. 2:  Hardware structures for sparse GEMM require explicit data re-
ordering, which introduces overheads (blocks in red), in the form of either
(a) an operand gather stage before the MAC compute (SMT-SA [38]), or (b) a
result scatter with a distributed accumulator (SCNN [30]).  Fig. 3:  Effective
energy/area and speedup, with breakdown by component for INT8 systolic array
variants processing a typical convolution with 50% weight and activation
sparsity. The SMT, our INT8 re-implementation of [38], achieves speedup, but
buffering overheads actually result in worse energy efficiency than even the
dense case.

The weight ($W$) and activation ($A$) tensors have independent random sparsity
patterns. Thus, during execution, we must walk the indexes to find matching
pairs of non-zero positions to pass to the MAC units. The number of matches
varies wildly depending on the position and the input data, which gives rise
to an unpredictable number of MACs in any single cycle, leading to variable,
unbalanced PE utilization at run time.

To keep the MAC utilization high, this load imbalance is usually evened out
using a data staging buffer, which collects the matched operand pairs and
packs them into groups of a fixed size that matches the datapath width [27,
32]. The hardware supporting this approach is shown in Fig. 2a. While the non-
zero operand matching itself may be of a reasonable cost [13], the data
staging buffer (typically FIFO) introduces high energy and area overheads,
which are especially significant for low energy/area INT8 mobile/embedded
accelerators.

Fig. 3 quantifies the energy and area overhead of the FIFOs used for
distributing matching pairs for INT8 operands. We compare the energy and area
across four designs: a dense systolic array (SA), a systolic array with ZVCG
optimization (SA-ZVCG), and two variants of SA-SMT, a recent systolic array
that exploits random sparsity using operand FIFOs [38]. The two SA-SMT
variants differ by their FIFO depths; one uses 2-entry FIFOs (SMT-T2Q2) and
the other uses 4-entry FIFOs (SMT-T2Q4). The PPA is obtained for a typical
convolution layer with 50% weight and activation sparsity. The energy and area
are broken down into the two key SA components: MACs (compute) and the on-chip
buffers.

#### Key Insight

We find that the two SA-SMT variants achieve 1.6$\times$ and 1.8$\times$
speedup. However, this requires additional buffering for load balancing,
introducing significant area and energy overheads. Overall, despite the
speedup achieved, SMT shows nearly 50% higher power and roughly same area to
SA-ZVCG, with INT8 operands required for mobile.

Table 1:  Comparison of PE buffer sizes per INT8 MAC. Outer-product style
accelerators (SCNN and SparTen) require large buffers per MAC compared to a
baseline systolic array.

Architecture | Operands | Accumulators | Total
---|---|---|---
SCNN [30] | 1.28 KB | 0.375 KB | 1.65 KB
SparTen [13] | 864 B | 128 B | 0.99 KB
Eyeriss v2 [6] | 165 B | 40 B | 205 B
SA-SMT1 [38] | 16 B | 4 B | 20 B
Systolic Array1 [18] | 2 B | 4 B | 6 B
S2TA-W2 (Ours) | 0.375 B | 0.5 B | 0.875 B
S2TA-AW3 (Ours) | 0.75 B | 4 B | 4.75 B

1Our INT8 implementation. 24/8 W-DBB (4$\times$4$\times$4_4$\times$8).
3Time-Unrolled AW-DBB (8$\times$4$\times$4_8$\times$8, BZ=8).

###  2.3 Overhead 2: Result Scatter Structure

A common alternative to the gather structure for matched operands, is to
multiply every non-zero weight with every non-zero activation (i.e., an outer-
product). In this approach, however, the MAC operations being executed in
parallel correspond to different, typically non-contiguous elements in the
output feature map. Therefore, the results of individual MACs must be properly
distributed to the right elements in the output feature map, using a scatter
operation.

Distributing the results is expensive because, instead of using a simple local
output stationary accumulator register in the dense accelerator, one must
construct a very large accumulator buffer using FF or SRAM, where each
accumulator requires a read-modify-write operation to accumulate the partial
sum in the correct place. The area and power overhead of this large
accumulator buffer is, again, significant for INT8 inference accelerators,
where the datapath logic is relatively cheap compared to the buffers (Fig. 1).

Tbl. 1 quantifies the overhead of the additional buffers that implement the
result scatter function. Specifically, the table compares the buffer size per
PE of the baseline systolic array and a range of different accelerators. Of
these accelerators, SCNN [30] and SparTen [13] implement a scatter structure.
While the systolic array requires only 2B for storing operands and 4B for
storing the accumulators per PE, the storage requirement is significantly
higher for other accelerators: e.g., SCNN [30] reports a total buffer size of
1.65 KB per PE, and SparTen [13] requires about 1 KB per PE.

##  3 Main Idea and Design Overview

(a) Unstructured Sparse Tensor (b) DBB Sparse Tensor

Fig. 4:  Comparison of (a) an unstructured sparse tensor, and (b) a density
bound block (DBB) sparse tensor. DBB constrains the maximum number of non-
zeros (NNZ) per block, such that the max workload is known at design time. The
NNZ shown is 3, and the block size (BZ) is 8.

This section first introduces the DBB structured sparsity, which we argue
enables an efficient sparsity-exploiting architecture (Sec. 3.1). We then
highlight the rationale of our architecture support for DBB, with the detailed
designs to follow in the subsequent sections (Sec. 3.2).

###  3.1 Exploiting Structured Sparsity

The sparsity-exploiting structures must be lightweight to be at all beneficial
in exploiting sparsity in mobile CNN accelerators. To that end, we propose to
exploit structured sparsity, which has regular sparsity patterns that allow
the hardware design to be lean. In particular, we focus on leveraging the
Density Bound Block (DBB) sparsity. DBB essentially divides an
(activation/weight) tensor into blocks and sets the upper bound of the number
of non-zero (NNZ) elements in each block. Fig. 4 compares unstructured
sparsity with DBB sparsity. While the overall sparsity level is the same, DBB
sparsity constrains the maximum number of NNZs in a block such that the
maximum workload is known at design time.

DBB provides two benefits. First, processing DBB blocks in order ameliorates
the load imbalance problem and removes the distributed accumulator problem
encountered with unstructured sparsity, avoiding the energy- and area-hungry
buffers. Second, DBB lends itself to a simple hardware design based around the
movement of small DBB data blocks, as we will illustrate with our S2TA
accelerator (Sec. 6). This compact and efficient hardware architecture is
possible because results are naturally generated in sequential blocks; no data
reordering is required.

Fig. 5:  A $4/8$ DBB example, where BZ=8 and NNZ=4. The tensor is blocked
along the channel dimension. The block is compressed by storing only the non-
zero elements and their positional index bitmask (M). The same compression
applies to weights and activation tensors.

(a) Dense (DP8) (b) ZVCG (DP8) (c) W-DBB (DP4M8) (d) A/W-DBB (DP4M4) (e)
A/W-DBB (DP1M4)

Fig. 6:  Parallel sparse MAC hardware. (a) Dense 8-MAC vector dot product
(DP8). (b) Dense DP8 with zero-value clock gating (ZVCG) reduces power by
exploiting random sparsity. (c) $4/8$ weight DBB (W-DBB) sparsity uses only 4
MACs (DP4), with an 8:1 mux (M8) for each (DP4M8). (d) A/W-DBB uses a 4-way
mux (DP4M4), but is limited to fixed DBB sparsity ratios. (e) W-DBB and Time-
Unrolled, serialized A-DBB allow the hardware to support activation density
from $1/8$ to $8/8$, tuned on a per-layer basis, implemented with a very cheap
single MAC and 4:1 mux (DP1M4). $M$ denotes index bitmask.

Fig. 5 gives a concrete DBB example, where the block size (BZ) is 8 and the
maximum number of non-zero values (NNZ) per block is 4. In this particular
case, the tensor blocking is performed along the channel dimension, which is a
common strategy to avoid all the elements in any single channel falling into
the same block. The compression itself has two steps. First, the non-zero
elements are stored by removing the zeros. Second, a simple bitmask $M$ is
added to encode the presence of a non-zero element at each location in the
expanded block.

We usually refer to a DBB block by the ratio $NNZ/BZ$. The example in Fig. 5
would be a 4/8 block. Note that any blocks that have less than NNZ non-zero
elements will include one or more zeros in the compressed form. For sparse DBB
execution, a block cannot contain more than $NNZ$ non-zeros, achieved via
pruning (Section 4 and 5). However, we also support a conventional dense mode
for unpruned models.

###  3.2 Design Overview

The rest of the paper discusses how to build an efficient accelerator to
exploit weight and activation DBB sparsity, without significant accuracy drop.
Weight sparsity is statically known, and thus the hardware design is
relatively straightforward. Activation sparsity, however, is dynamic. We
propose a dynamic pruning scheme co-designed with a novel time-unrolled
architecture to exploit DBB sparsity in activations.

Building on top of DBB compression for both weight and activation, we describe
how to integrate them into a complete DNN accelerator (Sec. 6). In particular,
we focus on the popular systolic architecture. We show that both weight and
activation DBB support can be easily integrated into a classic SA architecture
by grouping PEs, leading to a tensor PE (TPE) design. Critically, the TPE
design naturally exposes additional dimensions of data reuse that is
unobtainable in the traditional SA design, enabling further efficiency gains.

We note that weight DBB sparsity has been used in a commercial A100 GPU from
Nvidia [28]. Our design differs in two ways. First, A100 supports only fixed
2/4 weight sparsity with up to 2$\times$ speedup, while our design
comprehensively exploits both weight and variable activation DBB sparsity for
up to 8$\times$ speedup using a novel time-unrolled architecture (Section
5.2). We show that activation DBB requires non-trivial extensions from
previous weight DBB due to the dynamic nature of activation sparsity. Second,
our design focuses on the systolic array architecture, which is well suited to
low-power mobile/embedded systems. To fairly compare with A100, we implement
an A100-featured systolic design, S2TA-W, as a baseline in the evaluation.
Sec. 7 and 8.2.

##  4 Static Weight DBB Sparsity

This section describes an architecture to apply DBB to weight sparsity
(W-DBB). Note that weight DBB has been explored in [19, 45] and the
proprietary A100 GPU from Nvidia with fixed 2/4 weight only W-DBB [28, 31].
Here, describing it allows us to explain the general principle of DBB
sparsity, which we additionally apply to activations too.

The key advantage of weight DBB is that the load imbalance problem is greatly
relaxed, as we have bounded the maximum number of non-zero elements per block
and can provision hardware based on this. We illustrate this by modifying the
simple parallel (vector) architecture template given in Fig. 6a, which show an
8-MAC dot product, which we refer to as DP8. Building on top of DP8, Fig. 6c
shows a datapath for a $4/8$ W-DBB block, using only 4 hardware MACs instead
of 8, yielding a 50% reduction in MACs at the same throughput, and a 37.5%
reduction in weight operand bandwidth. Since this design has only 4 MACs
(DP4), with an 8-input mux (M8)111Omitting trivial optimization of mux widths
for the sake of clarity., we call this configuration DP4M8.

The key to the hardware for W-DBB is the 8:1 MUX in front of each MAC. The
MUX, controlled by the positional bitmask($M$) from the weight DBB block, is
used to steer the correct activation element into the MAC. The overhead of
this MUX is negligible, especially compared to the cost required to exploit
unstructured sparsity (Sec. 2). The DP4M8 (Fig. 6c) also accommodates fall
back to dense operation, which is essential to support models that have
density greater than 50%.

##  5 Dynamic Activation DBB Sparsity

Activation DBB compression (A-DBB) can be used together with W-DBB, allowing
us to exploit the full potential of sparse data, without significant
overheads. However, A-DBB is more challenging than W-DBB because: (1)
activation sparsity is unbounded, and (2) optimal activation DBB sparsity
varies significantly across the layers of a network. In this section, we
describe the co-designed Dynamic Activation Pruning (DAP) and time-unrolled
variable DBB architecture.

###  5.1 Dynamic Activation Pruning (DAP)

While weights can be compressed ahead of time (offline), activations are the
result of runtime computation and therefore must be compressed online. To do
this, we propose Dynamic Activation Pruning (DAP), which prunes and compresses
the dense activation tensors into the DBB format. The random sparsity of a
given activation tensor may have more non-zeros in a block than allowed by the
DBB $NNZ$. Therefore, DAP implements simple Top-NNZ pruning to keep the block
elements with the largest magnitude. As with W-DBB (Fig. 5), the activation
tensor is first decomposed into 1$\times$1$\times$$BZ$ blocks along the
channel dimension.

DAP is a lossy scheme, and can degrade test accuracy on some models if
$NNZ/BZ$ is small. For example, MobileNetV1 shows a test accuracy drop from
71% to 56.1% when using $4/8$ DAP for all point-wise CNN layers. We propose an
extension to the conventional DNN training procedure to make up any accuracy
loss from DAP (Sec 8.1).

(a) S2TA with M$\times$N TPE Array (b) 1$\times$1$\times$1 TPE (c) Time-
Unrolled 2$\times$4$\times$2 TPE with 4$\times$ DP1M4

Fig. 7:  Overview of (a) S2TA accelerator with TPE array (N$\times$M), dynamic
activation pruning (DAP), local SRAM and Arm M33 MCUs. Example TPE
configurations are denoted A$\times$B$\times$C, where B is the $NNZ$ of the
weight block. A is the number of activation operand block inputs per cycle,
and C is the number of weight block inputs. TPE configs show for: (b) the
classic scalar MAC PE (1$\times$1$\times$1), (c) A Time-Unrolled
2$\times$4$\times$2 TPE implemented by connecting four DP1M4 units (Fig. 6e).

###  5.2 Architectural Support for A/W-DBB

#### Basic Design

A simple implementation of joint DBB sparsity in a vector datapath is given in
Fig. 6d, where both weights and activations are compressed to reduce operand
bandwidth. As before, the only overheads are a single datapath multiplexer
before each MAC, which this time is reduced to 4:1 (DP4M4). The index bitmasks
can be trivially compared to determine matching non-zero positions, and unused
MACs can be clock gated to further reduce power. Although this doesn’t
increase utilization of MACs, both weight and activations are now compressed
in DBB format, thus the power to load both from SRAM is decreased
dramatically.

#### Challenges to Support Variable A-DBB

However, activation DBB introduces a significant challenge because the maximum
sparsity varies significantly across layers, for example for ResNet50v1, the
per-layer tuned activation DBB ranges from $8/8$ (dense) in early layers down
to $2/8$ towards the end. Therefore, while weight density typically hovers
below 50% and therefore $4/8$ W-DBB is a safe choice, forcing a fixed
activation DBB sparsity (A-DBB) would be a huge compromise and result in
diminished gains. Therefore, it is necessary to support variable A-DBB
sparsity in the architecture. Naively supporting variable DBB sparsity in
hardware reduces utilization and leads to low energy and area efficiency. For
example, if we implement a $4/8$ DBB datapath and run a model with $2/8$ DBB,
the utilization would drop by $\sim$50%.

#### Time Unrolled Architecture for Variable A-DBB

To support variable activation (A-DBB) sparsity efficiently, our key idea is
to switch from unrolling the elements in the A-DBB blocks spatially (Fig. 6d)
to serializing them in time (Fig. 6e). This simply means that we process one
element of the activation block per cycle using a single MAC, over multiple
cycles. For example, a layer with low density activations can use $1/8$ DBB,
requiring just one cycle per block. On the other hand, a layer with denser
$5/8$ DBB activations requires five cycles. In this way, we can now freely
vary the activation block density per layer directly by changing the number of
cycles per block, from $1/8$ to $8/8$. Meanwhile, the datapath utilization and
operand bandwidth remain constant.

##  6 Structured Sparse Tensor Accelerator (S2TA)

This section describes how to integrate the architectural support for W-DBB
and variable A-DBB into a DNN accelerator to exploit weight and activation
sparsity. As a case-study, we focus on systolic array-based DNN accelerators.
Due to their superior efficiency arising from local register-to-register
operand reuse [22], the systolic array has been widely promoted for CNN
accelerators [23, 35, 34, 38, 37], and even used in commercial products [18].
We call our DBB-exploiting systolic array S2TA, with two main variants: (1)
S2TA-W as a comparison baseline targeting W-DBB alone, and (2) our optimal
time-unrolled S2TA-AW exploiting joint A/W-DBB.

###  6.1 Overall Architecture

The key to supporting DBB sparsity in a systolic array is to replace the
traditional scalar PE with a Tensor PE (TPE). A scalar PE (Fig. 7b) accepts a
single pair of operands per cycle and computes a single MAC. A TPE (Fig. 7c),
in contrast, accepts a pair of fixed size operand blocks per cycle.

TPE is a natural design choice to support DBB, since both weights and
activations are naturally blocked in DBB. Each TPE essentially computes the
MACs between an activation tensor and a weight tensor. One can implement the
TPE using the DP4M8 MAC unit in Fig. 6(c) to form S2TA-W. For the time-
unrolled S2TA-AW, the DP1M4 MAC unit in Fig. 6(e) can be directly used as the
TPE implementation.

A TPE can be configured in a variety of ways. For instance, Fig. 7(c) shows a
TPE that accepts a weight tensor and an activation tensor in 4 cycles, with
size A$\times$B${}^{{}^{\prime}}$=2$\times$4 and B$\times$C=4$\times$2, where
B=4 is $NNZ$ of weight DBB blocks. The degenerate case of a
1$\times$1$\times$1 TPE is equivalent to the scalar PE from a traditional SA
(Fig. 7b). Fig. 7c shows a 2$\times$4$\times$2 TPE to exploit joint A/W-DBB,
which is implemented by connecting four time-unrolled scalar DP1M4 datapath in
Fig. 6(e) together in a pure outer-product fashion. In contrast, a TPE
implemented with DP4M8 datapath (Fig. 6(c)) for exploiting W-DBB alone is
reminiscent of a 4-way dot-product with dense A$\times$B activation and sparse
$NNZ$$\times$C weight tensor inputs to the TPE.

At the array level, TPEs operate in the usual systolic fashion as shown in
Fig. 6a: each TPE receives a pair of tensor operands from adjacent neighbors
and passes them on, except that the operands are tensors instead of scalar
values. Networks are mapped onto the array using simple matrix tiling, similar
to the TPU [18], except we use output-stationary.

#### Data Reuse

In addition to naturally supporting DBB sparsity, the TPE organization exposes
two new dimensions of data reuse compared to the $1\times 1\times 1$ PE
organization in classic systolic arrays. First, moving from a scalar MAC to a
tensor-wise product in the TPE introduces intra-TPE accumulator reuse, as we
now achieve multiple MACs per accumulator update, but lower the chance of
ZVCG. Second, moving from a single scalar input to tensors introduces intra-
TPE operand reuse, as each operand arriving at a TPE is used more than once.
This amortizes the cost of moving operands across the array, amongst multiple
MACs.

These two new forms of data reuse result in much smaller on-chip buffer sizes,
as the flip-flops required in the TPE are increasingly shared amongst a larger
number of MAC units. Table 1 shows that S2TA-W with a
4$\times$4$\times$4_4$\times$8 TPE array, and time-unrolled
8$\times$4$\times$4_8$\times$8 S2TA-AW for $BZ$=8 have $\sim$7–1,886$\times$
less total buffers per MAC than previous architectures. As a result, a larger
TPE would also increase the energy efficiency, albeit at a marginally reduced
clock frequency. Note that the outer-product TPE is more efficient than the
dot-product counterpart due to increased data reuse.

###  6.2 Hardware Support for DAP

To support A-DBB, the hardware must implement DAP, which simply selects the
$NNZ$ activation elements with the largest magnitude from the $BZ$-block
(Section 5.1).

The challenge of providing hardware support for DAP is that the $NNZ$ in a
A-DBB block is variable at run time. To support this, the DAP hardware
cascades $NNZ$ number of maxpool stages. Fig. 8 shows the DAP hardware. Each
maxpool selects the largest magnitude elements from the input block through
binary comparison using $BZ$ \- 1 comparators, which in our current design is
fixed to contain 8 elements as a block size of 8 is empirically found to
provide a good balance between accuracy and efficiency (Sec. 8.1).

We cap the maxpool stages at 5, since higher $NNZ$ would usually not lead to
significant efficiency gains. Thus, the DAP hardware (Table 2) supports a
A-DBB sparsity ratio ranging from $1/8$ to $5/8$, bypassing any unused stages.

Fig. 8: Hardware DAP array, consisting of cascaded magnitude (absolute value)
maxpool stages, configured for an input block of $BZ$=$8$, with
$NNZ$$\leq$$5$. The elements in red are discounted in consecutive maxpools.
For $4/8$ DBB, the output will be the elements $[4,5,-7,6]$ and the positional
bit mask $M$=8′h4D.

###  6.3 Other Design and Implementation Details

#### On-chip SRAM

As is commonplace for accelerators, we heavily leverage local software managed
SRAM [25] to provide a low-cost operand supply. The 0.5MB weight buffer (WB)
and the 2MB activation buffer (AB) are separate. Both are double buffered to
overlap computation in the TPE array and DMA data transfer. The SRAM is
grouped, rather than distributed, so we can use large high density SRAMs.

#### Local MCU with SIMD

We implement non-GEMM operations such as activation functions, pooling,
scaling, normalization and data type casting using Arm Cortex-M33 [1]
microcontrollers (MCUs), which have 32-bit SIMD instructions [2]. M33 is very
small (0.008mm2 [1]) and low power (3.9$\mu$W/MHz [1]) in 16 nm. Control and
data movement (DMA) tasks are also performed by the MCUs, e.g. loading the
input image into AB. We use a cluster of 4 MCUs each with a small 64KB control
store SRAM, which is sufficient to ensure that the MCUs are never the
performance bottleneck.

##  7 Methodology

#### Automatic RTL Generation

The S2TA accelerator is highly modular and can be configured to make a
calculated trade-off between area, performance, and power consumption. Instead
of evaluating an arbitrary design point, we implement a parameterized Python
RTL generator to explore the full design space, defined by five main
parameters: the three TPE dimensions (A, B, C in Sec. 6.1) and the dimension
of the entire SA (M, N); altogether denoted as A$\times$B$\times$C_M$\times$N.
Each design can be further configured with any combination of W-DBB, A-DBB,
ZVCG, and time-unrolling.

The RTL generator produces synthesizable Verilog RTL, along with a testbench
suite. Each design is automatically validated in Synopsys VCS using the
generated testbench, which executes inference on a given CNN model. This
generates accurate performance (throughput) metrics. During the simulation, we
also log value change dump (VCD) switching activity traces used for accurate
annotated power simulation.

Table 2:  Area and power breakdown of the S2TA-A/W design using a 16 nm process node, 8$\times$4$\times$4_8$\times$8 TPE configuration for $BZ$=$8$, with 4 TOPS peak throughput for 4/8 weight and dense activation.  Component | Power, mW | Area, mm2
---|---|---
MAC Datapath and Buffers | 317.7 | (58.7%) | 0.72 | (19.1%)
Weight SRAM (512KB) | 69.4 | (12.8%) | 0.54 | (14.3%)
Activation SRAM (2MB) | 93.4 | (17.2%) | 2.16 | (57.3%)
Cortex-M33 MCU [1] $\times$4 | 50.4 | (9.3%) | 0.30 | (8.0%)
DAP Array | 10.4 | (2%) | 0.05 | (1.3%)
Total | 541.3 | (100%) | 3.77 | (100%)

#### Physical Design and Evaluation

To evaluate area and power, each design goes through a complete EDA flow
consisting of Synopsys and Cadence tools, with the TSMC PDKs and Arm multi-Vt
cell libraries and single-ported SRAM compilers. We used both TSMC 16nm FinFET
and TSMC 65nm technology. The clock frequency is constrained to 1GHz in 16nm
(500MHz in 65nm) at the slow corner, with multiple process, voltage and
temperature corners for setup and hold timing. Power analysis was performed at
the typical corner, using Synopsys PrimeTimePX, with the parasitic-annotated
netlist and switching activity from VCD simulation traces.

#### S2TA-A/W Design Point

Based on typical mobile DNN accelerator specs, we set a hard constraint of 4
TOPS peak (dense) throughput and clock frequency of 1GHz in 16nm (500MHz in
65nm). We then sweep the design space to identify designs on the area-vs-power
frontier, from which we identify the TPE array design with the lowest power,
which is the 8$\times$4$\times$4_8$\times$8 outer-product TPE implemented with
the time-unrolled DP1M4 datapath (Fig. 6(e)). This configuration with $4/8$
W-DBB and variable A-DBB is referred to as S2TA-AW, and is used throughout the
evaluation. Full area and power breakdown of S2TA-AW is shown in Table 2.

#### Baselines

We compare against the following baselines:

  * •

SA-ZVCG: classic SA design with zero-value clock gating
(1$\times$1$\times$1_32$\times$64 TPE array in our notation).

  * •

S2TA-W: this is the variant of S2TA that exploits 4/8 W-DBB sparsity alone
(dense activations), using a 4$\times$8$\times$4_4$\times$8 TPE with 4-MAC
dot-product datapaths (DP4M8 in Fig. 6(c)). Comparing against S2TA-W allows us
to understand the gains from exploiting A-DBB and W-DBB jointly. This design
also implements ZVCG to weakly exploit activation sparsity.

  * •

SA-SMT: our INT8 re-implementation of a recent systolic array design that
exploits unstructured sparsity using operand staging FIFOs to distribute
matching operands [38], using the T2Q2 variant from the paper.

  * •

SparTen and Eyeriss-v2: both are recent non-systolic array designs [13, 6]
that exploit unstructured sparsity.

All systolic array designs have 4 TOPS peak (dense) throughput and otherwise
identical configurations. We use the same EDA flow to obtain performance,
power, and area (PPA) metrics for a fair comparison. The PPA metrics for
SparTen and Eyeriss-v2, are directly from the papers.

##  8 Evaluation Results

###  8.1 Accuracy Results

As with virtually all forms of sparsity, DBB sparsity used in S2TA is lossy,
and thus requires DNN fine-tuning to regain any accuracy loss. Here, we first
describe the simple extensions to the conventional DNN training procedure to
support DBB sparsity, followed by the accuracy results. We evaluate INT8
models since we focus on mobile inference, where INT8 is the most widely used
for deployment [41].

#### Training for W-DBB

We apply magnitude based DBB-aware weight pruning, which is similar to random
magnitude pruning [46], but pruning independently within each DBB block. This
typically runs for 20-50 epochs, progressively pruning small-magnitude weights
within each DBB block, until the desired DBB sparsity constraint is met.

Tbl. 3 shows the results of W-DBB (with dense activation) fine-tuning of five
popular CNNs with INT8 quantization: VGG-16, MobileNetV1, ResNet-50V1 on
ImageNet, and LeNet-5 on MNIST. We find that $4/8$ W-DBB density typically
achieves $<$0.5% accuracy loss, on both relatively big (ResNet-50V1) and
compact (MobileNetV1) networks.

In general, a larger block size (BZ) relaxes accuracy loss, but increases the
hardware cost to exploit the sparsity. Meanwhile, a larger NNZ per block
increases accuracy while leaving less room for exploiting sparsity. This is
evident as we increase the NNZ for ResNet-50V1 from 2 to 4 in Tbl. 3. Overall,
we find that $4/8$ DBB density level is a good compromise that achieves low
accuracy loss for both compact and larger models. Compared to previous work,
Kang [19] targets a fixed $2/8$ W-DBB, which is too aggressive to achieve good
accuracy, while the Nvidia A100 [28] uses $2/4$, which is the same sparsity
level as our $4/8$ choice, but less flexible. In this paper, we apply BZ=8
throughout our description without losing generality.

#### Training for A-DBB

Dynamic Activation Pruning (DAP) is lossy and requires fine-tuning to minimize
accuracy impact. We incorporate DAP into DNN fine-tuning by adding DAP in
front of convolution operations, mimicking how it is used at inference. To
back propagate through the DAP layer, we calculate the gradient of DAP with
respect to the activation $a$ $\frac{\partial DAP(a)}{\partial a}$, which is a
binary mask tensor with a value 1 for the Top-NNZ elements and a value 0 for
the pruned ones.

As an example, MobileNetV1 shows a test accuracy drop from 71% to 56.1% when
using $4/8$ A-DBB for all point-wise CNN layers before fine-tuning. A 30-epoch
DAP-aware fine-tuning recovers the accuracy to 70.2%. We find that 50-100
epochs of fine-tuning are typically sufficient.

Tbl. 3 also shows the accuracy of applying A-DBB alone and applying both forms
of DBB sparsity jointly. Note that while W-DBB density is hypertuned on a per-
model basis, the A-DBB density varies wildly from early layers to later layers
and is therefore tuned per-layer (supported by S2TA-AW). We observe that
combined A/W-DBB has a 0.1%–0.4% accuracy loss compared to exploiting W-DBB
alone. Total additional training time is less than 48 hours in all cases.

Finally, we demonstrate A/W-DBB pruning of Transformers, by training I-BERT
[20] on the GLUE dataset [40].

Table 3:  Accuracies of the baseline (INT8) models and various DBB variants.
The accuracy loss of exploiting both A-DBB and W-DBB is generally about 1%.
The accuracy loss of exploiting one form of sparsity is about 0.5%. A-DBB
density varies significantly across layers. We report the weighted average,
which can be a non-integer ratio.

Model | Dataset | Baseline3 |  ———– DBB Pruning ———–
---|---|---|---
|  | Acc. (%) | A-DBB1 | W-DBB2 | Acc. (%)
LeNet-5 | MNIST | 99.0 | 3/8 | – | 98.9
LeNet-5 | MNIST | 99.0 | – | 2/8 | 98.9
LeNet-5 | MNIST | 99.0 | 4/8 | 2/8 | 98.8
MobileNetV1 | ImageNet | 70.1 | 3.8/8 | – | 69.4
MobileNetV1 | ImageNet | 70.1 | – | 4/8 | 69.8
MobileNetV1 ∗ | ImageNet | 70.1 | 4.8/8 | 4/8 | 68.9
AlexNet | ImageNet | 55.7 | 3.8/8 | – | 54.7
AlexNet | ImageNet | 55.7 | – | 4/8 | 54.9
AlexNet ∗ | ImageNet | 55.7 | 3.9/8 | 4/8 | 54.6
VGG-16 | ImageNet | 71.5 | 3.1/8 | – | 71.8
VGG-16 | ImageNet | 71.5 | – | 3/8 | 71.4
VGG-16 ∗ | ImageNet | 71.5 | 3.1/8 | 3/8 | 71.9
ResNet-50V1 | ImageNet | 75.0 | – | 4/8 | 74.5
ResNet-50V1 | ImageNet | 75.0 | – | 3/8 | 74.3
ResNet-50V1 | ImageNet | 75.0 | – | 2/8 | 73.1
ResNet-50V1 | ImageNet | 75.0 | 3.49/8 | – | 74.4
ResNet-50V1 ∗ | ImageNet | 75.0 | 3.49/8 | 3/8 | 73.9
ResNet-50V1 | ImageNet | 75.0 | 3.49/8 | 4/8 | 74.1
I-BERT (base) | GLUE (QQP) | 91.2 |  4/8 4 | – | 91.2
I-BERT (base) | GLUE (QQP) | 91.2 |  3/8 4 | – | 91.0
I-BERT (base) | GLUE (QQP) | 91.2 | – |  4/8 4 | 91.1
I-BERT (base) | GLUE (QQP) | 91.2 |  4/8 4 |  4/8 4 | 90.9
I-BERT (base) | GLUE (SST2) | 94.7 |  4/8 4 | – | 94.3
I-BERT (base) | GLUE (SST2) | 94.7 |  4/8 4 |  4/8 4 | 93.5

1Tuned per-layer, average reported; and $-$ for dense. 2Tuned per-model
(excluding the 1st layer); and $-$ for dense. 38-bit dense models. 4Fully-
connected sub-layers (FC1, FC2) of Encoders. ∗ Used for whole model evaluation
in Sec. 8.3.

(a) SA-ZVCG (b) SA-SMT (c) S2TA-W (d) S2TA-AW

Fig. 9:  Energy (normalized to SA-ZVCG at 0% activation and 50% weight
sparsity) and speedup with sparsity. (a) SA-ZVCG energy falls slowly with
density, no speedup. (b) SA-SMT shows higher energy than SA-ZVCG, up to
2$\times$ speedup. (c) S2TA-W exploiting W-DBB alone provides a fixed
2$\times$ speedup, and 1.2$\times$ energy reduction compared to SA-ZVCG, at
sparsity $\geq$50%. (d) S2TA-AW with A/W-DBB exploits joint sparsity for up to
8$\times$ speedup and significant energy reduction.

###  8.2 Microbenchmarking Results

Fig. 10:  Energy breakdown and speedup of various SA variants for a typical
convolution with 50% ($4/8$-DBB) and 62.5% ($3/8$-DBB) INT8 weight and
activation sparsity, respectively. The results are normalized to SA-ZVCG.

We first use a set of synthetic (microbenchmark) DNNs with specific
weight/activation sparsity to understand the performance, energy, and area of
S2TA relative to the baselines.

Fig. 9a show show the energy and performance of SA-ZVCG vary with weight and
activation density. The $x$-axis increases the weight sparsity from 0% to
87.5%, and the two bars refer to two different activation densities of 50% and
20%, respectively. The energy is normalized to energy per MAC operation.
Naturally, the energy of SA-ZVCG scales weakly as the weight and activation
sparsity increases due to the clock gating, but there is no speedup regardless
of the sparsity. Fig. 9b shows that SA-SMT exploiting unstructured sparsity
for both weight and activation consumes significant higher energy than the SA-
ZVCG in Fig. 9a.

#### Exploiting W-DBB Alone

Fig. 9c shows the energy and performance of S2TA-W normalized to SA-ZVCG.
S2TA-W exploits a $4/8$ W-DBB sparsity and, thus, achieves a maximal 2$\times$
speedup step when weight sparsity is $\geq$50%. The 2$\times$speedup also
gives a corresponding energy reduction. However, the energy reduction at 50%
weight sparsity then plateaus, only scaling weakly with additional DBB weight
sparsity and lower switching activity. Clearly, S2TA-W cannot maximally
exploit the abundant activation sparsity.

#### Exploiting A/W-DBB Jointly

Exploiting both forms of sparsity, Fig. 9d shows that S2TA-AW achieves a
significant energy reduction of up to 9.1$\times$ compared to SA-ZVCG.

While the speedup from S2TA-W is capped at 2$\times$ regardless of the
activation sparsity, S2TA-AW supports variable activation compression and so
the speedup increases with activation sparsity from 1$\times$ at dense to
8$\times$ at 12.5% activation density.

#### Compared to Exploiting Unstructured Sparsity

Fig. 10 shows energy and speedup on a typical convolution layer with 50%
weight and 62.5% activation sparsity across all design variants. SA-SMT [38]
is a SA-based accelerator exploiting unstructured sparsity using staging FIFOs
with same INT8 operands. The results are normalized to SA-ZVCG. We break down
the energy consumption into different components: the datapath of the PE
array, the buffers in the PE array, SRAM, activation layers (M33), and the DAP
logic, which is unique to S2TA-AW. We evaluate two variants of SA-SMT: T2Q2
has an operand staging FIFO depth of two and T2Q4 has FIFO depth 4. While both
SA-SMT variants are faster than SA-ZVCG, they both also increase the effective
energy consumption (43.0% (T2Q2); 41.2% (T2Q4)) versus the baseline SA-ZVCG.
This is due to the energy overhead of the staging buffer for distributing
matched operands, not necessary for S2TA-W and S2TA-AW. This is also evident
in the significantly lower buffer energy for S2TA-W and S2TA-AW.

#### A comparison between S2TA-W and S2TA-AW

Fig. 10 shows that the energy benefits of S2TA-AW mainly come from a
3.1$\times$ reduction in the SRAM energy, as S2TA-AW exploits activation
sparsity using the time-unrolled outer-product TPE, whereas S2TA-W loads dense
activations from SRAM.

Table 4: Comparison of S2TA-AW and baselines, along with previously published
sparse CNN accelerators in 16nm/65nm.

|  |  SparTen [13] |  Eyeriss v2 [6] |  SA-ZVCG1 |  SA-SMT1 [38] |  S2TA-W1 |  S2TA-AW1
---|---|---|---|---|---|---|---
Weight Sparsity | Random | Random | ZVCG | Random |  $\nicefrac{{4}}{{8}}$ DBB, Dense |  $\nicefrac{{4}}{{8}}$ DBB, Dense
Activation Sparsity | Random | Random | ZVCG | Random | Dense |  $\nicefrac{{(1-5)}}{{8}}$ DBB, Dense
SRAM Size (W/A) | – | 246KB | 2MB / 0.5MB | 2MB / 0.5MB | 2MB / 0.5MB | 2MB / 0.5MB
Hardware MACs | 32 (INT8) | 384 (INT8) | 2048 (INT8) | 2048 (INT8) | 2048 (INT8) | 2048 (INT8)
Process Technology |  |  \- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 16nm Implementations \- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Clock Freq. | GHz | – | – | 1.0 | 1.0 | 1.0 | 1.0
Area |  mm2 | – | – | 3.7 | 4.2 | 3.4 | 3.8
Peak Throughput2 | TOPS | – | – | 4 | 8 | 8 |  8 (166)
Peak Energy Eff.2 | TOPS/W | – | – |  10.5 (12.83) |  8.01 (11.93) |  12.4 (13.93) | 14.3 (26.53)
AlexNet |  $\times 10^{3}$ Inf./sec | – | – |  2.2 (3.05) |  2.7 (4.05) |  3.3 (5.05) | 3.7 (6.35)
$\times 10^{3}$ Inf./J | – | – |  4.81 (7.55) |  4.48 (6.735) |  7.2 (8.75) | 9.9 (13.15)
TOPS/W | – | – |  7.2 (9.85) |  6.7 (8.75) |  9.8 (115) | 14 (195)
MobileNet |  $\times 10^{3}$ Inf./sec | – | – |  2.7 (3.65) |  3.5 (5.45) |  4.2 (7.35) | 5.2 (9.75)
$\times 10^{3}$ Inf./J | – | – |  7.9 (8.45) |  7.5 (8.05) |  9.2 (9.95) | 13.3 (14.95)
TOPS/W | – | – |  8.7 (9.25) |  8.2 (9.05) |  10 (115) | 14 (175)
Process Technology | 45nm | 65nm | 65nm | – | 65nm | 65nm
Clock Freq. | GHz | 0.8 | 0.2 | 0.5 | – | 0.5 | 0.5
Area |  mm2 |  0.7666 |  3.386,7 | 21 | – | – | 24
Peak Throughput2 | TOPS | 0.2 | 0.152 | 2 | – | 4 |  4 (86)
Peak Energy Eff.2 | TOPS/W | – | – | 0.78 | – | 0.87 | 1.1
AlexNet |  $\times 10^{3}$ Inf./sec | – |  0.28 (0.345) |  1.1 (1.55) | – |  1.6 (2.55) | 1.8 (3.25)
$\times 10^{3}$ Inf./J |  (0.52 5) |  0.66 (0.745) |  0.44 (0.675) | – |  0.55 (0.665) | 0.77 (1.025)
TOPS/W |  (0.685) |  0.96 (1.15) |  0.67 (0.955) | – |  0.82 (0.955) | 1.2 (1.45)
MobileNet |  $\times 10^{3}$ Inf./sec | – |  0.134 |  1.6 (1.825) | – |  1.82 (3.645) | 3.60 (4.855)
$\times 10^{3}$ Inf./J | – |  0.224 | 0.68 | – | 0.76 | 1.04
TOPS/W | – | 0.24 | 0.75 | – | 0.83 | 1.1

1Our results. 250% sparse weights and activations. 375% sparse weights and
activations. 4Scaled from MobileNet-v1-0.5-128 to 1.0-224. 5Conv only. 6Logic
only. 72.7Mgates.

Fig. 11:  Energy reduction and speedup (normalized to SA-ZVCG) comparison on
ResNet50V1, VGG16, MobileNetV1, and AlexNet (Convolution only) using the same
16nm technology. The S2TA-AW is 2.08$\times$, 1.84$\times$, 2.24$\times$
energy efficient, and 2.11$\times$, 1.26$\times$, 1.43$\times$ speedup than
SA-ZVCG, S2TA-W, SA-SMT baselines, respectively.  Fig. 12:  AlexNet per-layer
energy for Eyeriss-v2 [6], SparTen [13], SA-ZVCG, S2TA-W, and the optimal
S2TA-AW. S2TA-AW in 65nm is about 2.2$\times$ more efficient than the 45nm
SparTen on AlexNet energy per inference. SparTen has low energy only on very
high sparsity layers (i.e. Conv3, 4 and 5). Eyeriss-v2 consumes 3.1$\times$
more total energy than S2TA-AW in the same 65nm technology.

###  8.3 Full Model Inference Results

We compare S2TA-AW with SA, SA-ZVCG, SA-SMT, and S2TA-W across four models:
VGG16, MobileNetv1, ResNet50v1, and AlexNet. Fig. 11 gives energy reduction
(top figure) and speedup (bottom figure) comparison, normalized to SA-ZVCG.
Compared to SA-ZVCG, S2TA-AW reduces energy by 1.76–2.79$\times$, with
1.67–2.58$\times$ speedup.

#### S2TA-W

On average, S2TA-AW consumes 1.84$\times$ lower energy and achieves
1.26$\times$ speedup over S2TA-W (Fig. 11).

#### SA-SMT

SA-SMT [38] exploits unstructured sparsity in a systolic array. SA-SMT suffers
from the overhead of distributing matching pairs, which can stall the data
flow in the systolic array. SA-SMT resolves it using expensive operand buffer
FIFOs (Sec. 2.2). We reimplemented SA-SMT, which achieves 8.01 TOPS/W compared
to 14.3 TOPS/W for S2TA-AW at the same sparsity, as shown in Tbl. 4. This is
due to the high energy cost of the FIFOs (Fig. 9b and 10).

#### Non-SA Accelerators

Next, we compare S2TA with non-SA random-sparse accelerators, SparTen and
Eyeriss-v2.

SparTen [13] is a state-of-the-art CNN accelerator that exploits unstructured
sparsity in weights and activations, with superior results to SCNN [30]. For
the comparison, we re-implemented S2TA-AW in 65nm technology, which is older
than the 45nm used for SparTen and, thus, does not unfairly favor S2TA-AW.
Fig. 12 compares the energy for S2TA-AW and SparTen on AlexNet, with SA-ZVCG
and S2TA-W. The total energy per inference on AlexNet is 2.2$\times$ lower on
S2TA-AW compared to SparTen, even with a one node process disadvantage (65nm
vs 45nm). The gain on MobileNet is even greater, as shown in Tbl. 4. SparTen
performs best on layers with very high sparsity (e.g., Conv3-5), and less well
on layers with dense or more moderate sparsity (e.g., Conv1,2).

A comparison with Eyeriss-v2 [6] is given in Tbl. 4 and Fig. 12. S2TA-AW shows
$4.7\times$ and $1.4\times$ higher energy efficiency on MobileNet and AlexNet
for convolution layers, respectively. Fig. 12 indicates Eyeriss-v2 is
3.1$\times$ less energy efficient than S2TA-AW in the same 65 nm process
technology.

As with any SA-based inference accelerators, fully-connected (FC) and
depthwise (DW) layers are memory bound on S2TA, as batching is typically not
used in inference. However, we do prune FC/DW, included in the full model
results in Tbl. 4.

These results demonstrate that although the speedup gained by S2TA is
moderate, the overheads are low and the architecture is more energy efficient.
In fact, we find that even the baseline SA-ZVCG has lower energy than SparTen
on AlexNet and Eyeriss-v2 on MobileNet. While achieving high speedup from
sparsity is important for certain scenarios, it is essential to also consider
the overheads and start with an energy efficient architecture, which is
especially important for mobile inference.

###  8.4 Summary of Results

Finally, we summarize the key results, based on full-accelerator synthesis
data with typical weight and activation sparsity.

  1. 1.

DBB pruning of both weights and activations reduces the complexity of DNNs
without significant accuracy loss (Table 3).

  2. 2.

SA-ZVCG consumes 25% less energy than a dense SA by exploiting random
sparsity.

  3. 3.

S2TA-W exploiting W-DBB alone only marginally reduces energy (by 1.13$\times$)
compared to SA-ZVCG; exploiting both weight and activation sparsity is far
superior.

  4. 4.

S2TA-AW achieves an average 2.08$\times$ energy reduction and 2.11$\times$
speedup compared to SA-ZVCG on full DNNs.

  5. 5.

Activation DBB sparsity varies widely across layers, thus the S2TA-AW time-
unrolled architecture supports variable activation DBB sparsity, with fixed
weight sparsity. 222 S2TA time-unrolled architecture can also be implemented
to support variable weight DBB sparsity and fixed activation DBB sparsity.

  6. 6.

S2TA-AW on typical CNN microbenchmarks with 50% (75%) weight and activation
sparsity show 8 (16) TOPS, 14.3 (26.5) TOPS/W, and 2.16 (4.21) TOPS/mm2, in
16nm, based on full-accelerator synthesis data in Table 5.

  7. 7.

S2TA-AW has 1.4 – 4.7$\times$ lower energy than state-of-the-art accelerators
exploiting unstructured sparsity, including SparTen, Eyerissv2, and SA-SMT
(Table 5).

Table 5 summarizes key aspects of this work.

Table 5:  Summary of designs evaluated and previous works. S2TA-AW is a very
low overhead fully-sparse architecture that achieves significant speedup and
energy efficiency gains. The optimal design is the Time-unrolled (Variable
DBB) S2TA-AW architecture with up to 8$\times$ speedup.

Architecture | Weight Sparsity | Activation Sparsity | Hardware Overhead | ZVCG | Variable DBB (Time-unrolling)
---|---|---|---|---|---
\- - - - - - - - - Power Savings From Random Sparsity, No Speedup \- - - - - -
- - -
SA [18] | ✗ | ✗ | – | ✗ | ✗
SA-ZVCG | ✗ | ✗ | – | ✓ | ✗
\- - - - - - - - Speedup From Random Sparsity Incurs HW Overheads \- - - - - -
- -
SA-SMT [38] | Random | Random | Gather | ✓ | ✗
SCNN [30] | Random | Random | Scatter | ✗ | ✗
SparTen [13] | Random | Random | Gather | ✗ | ✗
\- - - - - - - - - - Speedup From Structured Sparsity: No Overheads \- - - - -
- - - - -
Kang [19] |  $\nicefrac{{2}}{{8}}$ DBB | ✗ | – | ✗ | ✗
STA [26] |  $\nicefrac{{4}}{{8}}$ DBB | ✗ | – | ✗ | ✗
A100 [28] |  $\nicefrac{{2}}{{4}}$ DBB | ✗ | – |  –1 | ✗
S2TA-W |  $\nicefrac{{4}}{{8}}$ DBB | ✗ | – | ✗ | ✗
S2TA-AW |  $\nicefrac{{4}}{{8}}$ DBB |  $\nicefrac{{(1-5)}}{{8}}$ DBB | – | ✓ | ✓

1 Unpublished proprietary design.

##  9 Related Work

#### Zero Value Clock Gating

(ZVCG) (Sec. 2.1) saves power when having zero operands [7, 33, 18]. We apply
ZVCG to exploit excess sparsity that cannot be exploited by DBB.

#### Indexed Unstructured Sparsity

EIE [15] implements a fine-grained sparse CSR-encoded INT16 matrix-vector
accelerator, and ESE [16] extends this to LSTMs. Doping [39] and MASR [14]
also exploit unstructured sparsity for LSTMs and RNNs, but uses a bitmask
encoding. A number of papers target unstructured sparse matrix multiplication
for very sparse data, such as Outer Space [29], which uses an outer product
scheme, and SpArch [44], which further optimizes for locality. Cnvlutin [4]
skips compute for zero activations, without explicit indexes. SCNN [30]
implements a fully CSR-indexed sparse CNN accelerator using an outer product
to exploit sparse weights and activations. FixyNN [43] demonstrates a fixed-
weight accelerator, that can very efficiently exploit random sparsity. SparTen
[13] and Eyeriss v2 [6] both support fully-sparse inference. We focus on DBB
sparsity, but compare with SparTen, and Eyeriss v2 (Table 4).

#### DBB Weight Sparsity

Kang [19] implements accelerator exploiting a fixed 2/8 W-DBB sparsity. The
design is based on a dot product microarchitecture with limited data reuse.
Similar work[45] also exploited W-DBB in the GPU context. The proprietary
Nvidia A100 GPU implements fixed 2/4 W-DBB, which achieves 1.5$\times$ speedup
and 3.12 TOPS/W (peak)[8], 4$\times$ lower than the S2TA-W baseline at 12.4
TOPS/W (Table 4).

S2TA is the first architecture exploiting both W-DBB and A-DBB, and is the
first to incorporate DBB into a systolic array with the novel time-unrolled
technique exploiting new dimensions of data reuse for up to 8$\times$ peak
speedup.

#### Sparsity in Systolic Arrays

SAs (e.g. Google TPU [18]) are efficient because they have high data reuse and
local communication. SMT-SA [38] is an SA that exploits unstructured FP32
sparsity using data staging FIFOs, which are energy inefficient for INT8
datapath, although acceptable for FP32. Kung et al. [23] showed a
preprocessing step of column combining of sparse weight matrices, before
processing on a dense SA architecture. Liu et al. [26] exploited W-DBB
sparsity for INT8 datapath in systolic architecture. NB-SMT [37] is a sparse
SA with the ability to momentarily halve the MAC precision during to aid load
balancing pipeline hazards.

#### Bit-wise Sparsity

Even sign extension and zero bits within a word can be considered for
optimization: Pragmatic [3] implements weight bit-sparsity, Tactical [9]
implements activation bit-sparsity, and Laconic [36] implements weight and
activation bit-sparsity. While orthogonal to our work, bit-sparsity is an
interesting avenue for future DBB research.

##  10 Conclusion

Density bound block (DBB) exploits structured sparsity, without the overheads
of random sparsity schemes. We describe an architecture to aggressively
exploit DBB sparsity on both weights and activations. For weight DBB, we prune
during training to meet the DBB structured sparsity constraint. However, we
cannot do this for activations, which are not static, but rather computed at
runtime. Therefore, we introduce Dynamic Activation Pruning (DAP), a co-design
solution for activation DBB, which implements activation DBB in hardware
during runtime to force the required DBB sparsity. DAP is lossy, and therefore
must also be incorporated during training to prevent accuracy loss at
inference time.

The proposed novel time-unrolled DBB sparsity architecture, S2TA-AW,
implements joint weight and activation sparsity in an efficient systolic
architecture. The design significantly outperforms other strong baselines,
including the variants S2TA-W and SA-SMT, which exploit weight sparsity alone,
and fully unstructured sparsity, respectively. On the pruned INT8 benchmark
models AlexNet, MobileNetv1, VGG16, ResNet50v1, a 16nm S2TA-AW with joint
weight and activation DBB sparsity demonstrates 2.08$\times$ and 1.84$\times$
energy reduction, on average, compared to the baseline SA-ZVCG and S2TA-W
which exploits weight DBB sparsity alone, respectively. Finally, S2TA-AW has
about 2$\times$ and 3$\times$ the energy efficiency of the state-of-the-art
unstructured sparse accelerator SparTen and Eyersiss-v2 on AlexNet,
respectively.

## References

  * [1] Arm Cortex-M33. [Online]. Available: https://developer.arm.com/ip-products/processors/cortex-m/cortex-m33
  * [2] ARMv8-M Architecture Reference Manual. [Online]. Available: https://static.docs.arm.com/ddi0553/a/DDI0553A_e_armv8m_arm.pdf
  * [3] J. Albericio, A. Delmás, P. Judd, S. Sharify, G. O’Leary, R. Genov, and A. Moshovos, “Bit-pragmatic deep neural network computing,” in _Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO-50 ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 382–394. [Online]. Available: https://doi.org/10.1145/3123939.3123982
  * [4] J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, “Cnvlutin: Ineffectual-neuron-free deep neural network computing,” in _Proceedings of the 43rd International Symposium on Computer Architecture_ , ser. ISCA ’16. Piscataway, NJ, USA: IEEE Press, 2016, pp. 1–13. [Online]. Available: https://doi.org/10.1109/ISCA.2016.11
  * [5] C. Banbury, C. Zhou, I. Fedorov, R. Matas, U. Thakker, D. Gope, V. Janapa Reddi, M. Mattina, and P. Whatmough, “MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers,” in _Proceedings of Machine Learning and Systems_ , A. Smola, A. Dimakis, and I. Stoica, Eds., vol. 3, 2021, pp. 517–532. [Online]. Available: https://proceedings.mlsys.org/paper/2021/file/a3c65c2974270fd093ee8a9bf8ae7d0b-Paper.pdf
  * [6] Y. Chen, T. Yang, J. Emer, and V. Sze, “Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 9, no. 2, pp. 292–308, 2019.
  * [7] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” in _Proceedings of the 43rd International Symposium on Computer Architecture_. Piscataway, NJ, USA: IEEE Press, 2016, pp. 367–379. [Online]. Available: https://doi.org/10.1109/ISCA.2016.40
  * [8] W. J. Dally. Logarithmic Numbers and Asynchronous Accumulators, The Future of DL Chips. MLSys Keynote Talk, 2021. [Online]. Available: https://media.mlsys.org/Conferences/MLSYS2021/Slides/DL_Chips_ML_Sys_0421.pdf
  * [9] A. Delmas Lascorz, P. Judd, D. M. Stuart, Z. Poulos, M. Mahmoud, S. Sharify, M. Nikolic, K. Siu, and A. Moshovos, “Bit-tactical: A software/hardware approach to exploiting value and bit sparsity in neural networks,” ser. ASPLOS ’19. New York, NY, USA: Association for Computing Machinery, 2019, pp. 749–763. [Online]. Available: https://doi.org/10.1145/3297858.3304041
  * [10] I. Fedorov, R. P. Adams, M. Mattina, and P. N. Whatmough, “SpArSe: Sparse architecture search for CNNs on resource-constrained microcontrollers,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 4978–4990.
  * [11] I. Fedorov, M. Stamenovic, C. Jenson, L.-C. Yang, A. Mandell, Y. Gan, M. Mattina, and P. N. Whatmough, “ TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids ,” in _Conference of the International Speech Communication Association (INTERSPEECH)_ , 2020.
  * [12] Y. Feng, P. Whatmough, and Y. Zhu, “ASV: Accelerated Stereo Vision System,” in _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’52. New York, NY, USA: Association for Computing Machinery, 2019, p. 643–656. [Online]. Available: https://doi.org/10.1145/3352460.3358253
  * [13] A. Gondimalla, N. Chesnut, M. Thottethodi, and T. N. Vijaykumar, “Sparten: A sparse tensor accelerator for convolutional neural networks,” in _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’52. New York, NY, USA: Association for Computing Machinery, 2019, p. 151–165. [Online]. Available: https://doi.org/10.1145/3352460.3358291
  * [14] U. Gupta, B. Reagen, L. Pentecost, M. Donato, T. Tambe, A. M. Rush, G. Wei, and D. Brooks, “MASR: A modular accelerator for sparse rnns,” in _28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019, Seattle, WA, USA, September 23-26, 2019_. IEEE, 2019, pp. 1–14. [Online]. Available: https://doi.org/10.1109/PACT.2019.00009
  * [15] S. Han _et al._ , “EIE: Efficient inference engine on compressed deep neural network,” in _Proceedings of the 43rd Int. Symp. on Computer Architecture (ISCA)_ , 2016.
  * [16] S. Han, J. Kang, H. Mao, Y. Hu, X. Li, Y. Li, D. Xie, H. Luo, S. Yao, Y. Wang, H. Yang, and W. B. J. Dally, “Ese: Efficient speech recognition engine with sparse lstm on fpga,” in _Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , ser. FPGA ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 75–84. [Online]. Available: https://doi.org/10.1145/3020078.3021745
  * [17] P. Hansen, A. Vilkin, Y. Khrustalev, J. Imber, D. Hanwell, M. Mattina, and P. N. Whatmough, “ ISP4ML: Understanding the Role of Image Signal Processing in Efficient Deep Learning Vision Systems ,” in _International Conference on Pattern Recognition (ICPR)_ , 2020.
  * [18] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, R. C. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan, R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omernick, N. Penukonda, A. Phelps, J. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Walter, W. Wang, E. Wilcox, and D. H. Yoon, “In-datacenter performance analysis of a tensor processing unit,” _CoRR_ , vol. abs/1704.04760, 2017. [Online]. Available: http://arxiv.org/abs/1704.04760
  * [19] H. Kang, “Accelerator-aware pruning for convolutional neural networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , pp. 1–1, 2019.
  * [20] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-BERT: Integer-only BERT Quantization,” _arXiv e-prints, accepted by ICML21_ , p. arXiv:2101.01321, Jan. 2021. [Online]. Available: https://icml.cc/Conferences/2021/ScheduleMultitrack?event=9811
  * [21] S. Kodali, P. Hansen, N. Mulholland, P. Whatmough, D. Brooks, and G. Wei, “Applications of Deep Neural Networks for Ultra Low Power IoT,” in _2017 IEEE International Conference on Computer Design (ICCD)_ , 2017, pp. 589–592.
  * [22] Kung, “Why systolic architectures?” _Computer_ , vol. 15, no. 1, pp. 37–46, 1982.
  * [23] H. Kung, B. McDanel, and S. Q. Zhang, “Packing sparse convolutional neural networks for efficient systolic array implementations: Column combining under joint optimization,” in _24th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_ , 2019, pp. 821–834.
  * [24] S. K. Lee, P. N. Whatmough, M. Donato, G. G. Ko, D. Brooks, and G.-Y. Wei, “SMIV: A 16-nm 25-mm² SoC for IoT With Arm Cortex-A53, eFPGA, and Coherent Accelerators,” _IEEE Journal of Solid-State Circuits_ , pp. 1–1, 2021.
  * [25] H. Li, M. Bhargav, P. N. Whatmough, and H. . Philip Wong, “On-chip memory technology design space explorations for mobile deep neural network accelerators,” in _2019 56th ACM/IEEE Design Automation Conference (DAC)_ , 2019, pp. 1–6.
  * [26] Z.-G. Liu, P. N. Whatmough, and M. Mattina, “Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference,” _IEEE Computer Architecture Letters_ , vol. 19, no. 1, pp. 34–37, 2020.
  * [27] M. Mahmoud, I. Edo, A. H. Zadeh, O. Mohamed Awad, G. Pekhimenko, J. Albericio, and A. Moshovos, “Tensordash: Exploiting sparsity to accelerate deep neural network training,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2020, pp. 781–795.
  * [28] Nvidia. A100 GPU Datasheet. [Online]. Available: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet.pdf
  * [29] S. Pal _et al._ , “Outerspace: An outer product based sparse matrix multiplication accelerator,” in _Int. Symp. on High Performance Computer Architecture (HPCA)_ , Feb 2018, pp. 724–736.
  * [30] A. Parashar _et al._ , “SCNN: An accelerator for compressed-sparse convolutional neural networks,” in _2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA)_ , June 2017, pp. 27–40.
  * [31] J. Pool. Accelerating sparsity in the Nvidia Ampere architecture. [Online]. Available: https://developer.download.nvidia.com/video/gputechconf/gtc/2020/presentations/s22085-accelerating-sparsity-in-the-nvidia-ampere-architecture%E2%80%8B.pdf
  * [32] E. Qin, A. Samajdar, H. Kwon, V. Nadella, S. Srinivasan, D. Das, B. Kaul, and T. Krishna, “Sigma: A sparse and irregular gemm accelerator with flexible interconnects for dnn training,” in _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_ , 2020, pp. 58–70.
  * [33] B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernández-Lobato, G.-Y. Wei, and D. Brooks, “Minerva: Enabling low-power, highly-accurate deep neural network accelerators,” in _Proceedings of the 43rd International Symposium on Computer Architecture_ , ser. ISCA, 2016.
  * [34] A. Samajdar, J. M. Joseph, Y. Zhu, P. Whatmough, M. Mattina, and T. Krishna, “A systematic methodology for characterizing scalability of dnn accelerators using scale-sim,” in _2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ , 2020, pp. 58–68.
  * [35] A. Samajdar, Y. Zhu, P. Whatmough, M. Mattina, and T. Krishna, “SCALE-Sim: Systolic CNN Accelerator Simulator,” _arXiv preprint arXiv:1811.02883_ , 2018.
  * [36] S. Sharify, A. D. Lascorz, M. Mahmoud, M. Nikolic, K. Siu, D. M. Stuart, Z. Poulos, and A. Moshovos, “Laconic deep learning inference acceleration,” in _Proceedings of the 46th International Symposium on Computer Architecture_ , ser. ISCA ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 304–317. [Online]. Available: https://doi.org/10.1145/3307650.3322255
  * [37] G. Shomron and U. Weiser, “Non-blocking simultaneous multithreading: Embracing the resiliency of deep neural networks,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2020, pp. 256–269.
  * [38] G. Shomron, T. Horowitz, and U. Weiser, “SMT-SA: Simultaneous multithreading in systolic arrays,” _IEEE Comput. Archit. Lett._ , vol. 18, no. 2, pp. 99–102, Jul. 2019.
  * [39] U. Thakker, P. Whatmough, Z. LIU, M. Mattina, and J. Beu, “Doping: A technique for extreme compression of lstm models using sparse structured additive matrices,” in _Proceedings of Machine Learning and Systems_ , A. Smola, A. Dimakis, and I. Stoica, Eds., vol. 3, 2021, pp. 533–549. [Online]. Available: https://proceedings.mlsys.org/paper/2021/file/a3f390d88e4c41f2747bfa2f1b5f87db-Paper.pdf
  * [40] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355. [Online]. Available: https://aclanthology.org/W18-5446
  * [41] P. Warden. Why are eight bits enough for deep neural networks? [Online]. Available: https://petewarden.com/2015/05/23/why-are-eight-bits-enough-for-deep-neural-networks/
  * [42] P. N. Whatmough, S. K. Lee, M. Donato, H. Hsueh, S. Xi, U. Gupta, L. Pentecost, G. G. Ko, D. Brooks, and G. Wei, “A 16nm 25mm2 SoC with a 54.5x Flexibility-Efficiency Range from Dual-Core Arm Cortex-A53 to eFPGA and Cache-Coherent Accelerators,” in _2019 Symposium on VLSI Circuits_ , 2019, pp. C34–C35.
  * [43] P. N. Whatmough, C. Zhou, P. Hansen, S. K. Venkataramanaiah, J. sun Seo, and M. Mattina, “FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning,” in _Proceedings of the 2nd SysML Conference, Palo Alto, CA, USA_ , 2019.
  * [44] Z. Zhang, H. Wang, S. Han, and W. J. Dally, “Sparch: Efficient architecture for sparse matrix multiplication,” in _26th IEEE International Symposium on High Performance Computer Architecture (HPCA)_ , 2020.
  * [45] M. Zhu, T. Zhang, Z. Gu, and Y. Xie, “Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus,” in _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’52. New York, NY, USA: Association for Computing Machinery, 2019, p. 359–371. [Online]. Available: https://doi.org/10.1145/3352460.3358269
  * [46] M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression.” _CoRR_ , vol. abs/1710.01878, 2017. [Online]. Available: http://dblp.uni-trier.de/db/journals/corr/corr1710.html#abs-1710-01878
  * [47] Y. Zhu, M. Mattina, and P. N. Whatmough, “Mobile Machine Learning Hardware at ARM: A Systems-on-Chip (SoC) Perspective,” in _Proceedings of the 1st Conference on Systems and Machine Learning_ , 2018.
  * [48] Y. Zhu, A. Samajdar, M. Mattina, and P. Whatmough, “Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision,” in _Proceedings of the 45th Annual International Symposium on Computer Architecture_ , ser. ISCA ’18. IEEE Press, 2018, p. 547–560. [Online]. Available: https://doi.org/10.1109/ISCA.2018.00052