# Deployment of Energy-Efficient Deep Learning Models on Cortex-M based
Microcontrollers using Deep Compression

Mark Deutel1, Philipp Woller2, Christopher Mutschler2, and Jürgen Teich4 1
Mark Deutel is with Fraunhofer IIS, Fraunhofer Institute for Integrated
Circuits IIS, Nuremberg, Germany and Friedrich-Alexander-Universität Erlangen-
Nürnberg (FAU), Erlangen, Germany<EMAIL_ADDRESS>Phillip Woller and
Christopher Mutschler are with the Fraunhofer IIS, Fraunhofer Institute for
Integrated Circuits IIS, Nuremberg, Germany
<EMAIL_ADDRESS>Jürgen Teich is with Friedrich-
Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany
<EMAIL_ADDRESS>
###### Abstract

Large Deep Neural Networks (DNNs) are the backbone of today’s artificial
intelligence due to their ability to make accurate predictions when being
trained on huge datasets. With advancing technologies, such as the Internet of
Things, interpreting large quantities of data generated by sensors is becoming
an increasingly important task. However, in many applications not only the
predictive performance but also the energy consumption of deep learning models
is of major interest.

This paper investigates the efficient deployment of deep learning models on
resource-constrained microcontroller architectures via network compression. We
present a methodology for the systematic exploration of different DNN pruning,
quantization, and deployment strategies, targeting different ARM Cortex-M
based low-power systems. The exploration allows to analyze trade-offs between
key metrics such as accuracy, memory consumption, execution time, and power
consumption. We discuss experimental results on three different DNN
architectures and show that we can compress them to below 10% of their
original parameter count before their predictive quality decreases. This also
allows us to deploy and evaluate them on Cortex-M based microcontrollers.

###### Index Terms:

Deep Neural Networks, Deep Compression, Network Pruning, Weight Quantization,
Microcontrollers

##  I Introduction

Deep Neural Networks (DNNs) became predominant in many applications that
require autonomous decision-making based on environmental information,
including audio recognition [1, 2], image classification [3, 4], or human
activity monitoring [5]. DNNs are beneficial as they are easy to set up and as
they can be trained to detect correlations even when they are confronted with
high-dimensional data.

However, the execution of DNNs is energy-, resource-, and time-expensive [6,
7]. In situations where the trade-off between resource constraints, execution
time, and predictive quality is key, DNNs often struggle to compete with
classical machine learning approaches [8]. However, with trends like smart
devices and the internet of things (IoT), the demand and interest in deploying
DNNs on microcontrollers grows.

Figure 1: Overview of our methodology.

Deep compression is a relatively young research area that deals with the
compression of DNNs. Prominent techniques include DNN graph pruning [9],
weight quantization [10, 11], and subspace methods [12]. Their goal is to
reduce the resource footprint of a DNN by reducing the number of trainable
weights and computational complexity while preserving the original predictive
performance.

Based on these principles different DNN compression pipelines have been
proposed. Most noticeably, Han et al. [13] who proposed a pipeline combining
network pruning, integer quantization, and Huffman encoding. Others focus on
quantization during network training [10] or on structure-based pruning. This
allows for an immediate removal of pruned weights [14, 15]. However, such
well-established frameworks only trade compression over predictive accuracy
but do not explicitly target energy-efficiency and architecture-specific
constraints like memory availability and processing speed that play an
important role in many embedded applications.

This paper proposes a methodology to systematically train and deploy DNN
architectures on Cortex-M-based microcontrollers. We introduce an automated
pipeline that covers application-specific DNN training and compression, and
that combines it with a target architecture-specific deployment, see Fig. 1.
Our proposed pipeline is composed of two major building blocks. First, from an
application-specific viewpoint we systematically explore compression
techniques, i.e., network pruning and weight quantization, and configurations
during the training of DNNs. Second, incorporating an architecture-specific
view, we realize the mapping from a high-level graph-based DNN representation
to low level code. This step involves an offline code generator and a runtime
library. The former takes care of the data layout, plans memory allocation and
emits executable code while the latter provides implementations of common DNN
layers used by the generated code. Novel methods implemented in our proposed
pipeline include ahead-of-time code generation and memory allocation
scheduling, which eliminates the need for any form of network interpretation
or dynamic memory management at runtime, and the exploitation of sparse
matrices generated as part of our pruning techniques using the _Compressed
Column Storage (CCS)_ format [16].

In our experiments we evaluate both compression and deployment for three
common DNN architectures, i.e., AlexNet [17], ResNet [3] and LeNet [18]. Our
objective is a thorough evaluation of the relation between compressed DNNs and
their predictive quality. In contrast to previous work our results do not only
focus on deployment or compression alone, but provide detailed insight into
the relationship between different pruning, quantization, and deployment
strategies when applied in combination. Furthermore, we deployed the
compressed models on three target systems and discuss their memory
consumption, execution time, and power consumption.

The rest of this paper is structured as follows. Sec. II discusses related
work. Secs. III and IV provide details about our compression and deployment
pipeline. Sec. V discusses our experimental results. Sec. VII concludes.

##  II Related Work

Existing research predominately compresses DNNs via network pruning and weight
quantization. These techniques are well understood as research has been
conducted exploring the effects of pruning and quantization on a network’s
predictive performance [19, 20]. However, when deploying applications for
embedded targets, they are defined by the constraints imposed by the platforms
they use. As a result, the suitability of DNN models for deployment on
microcontrollers is not only determined by their accuracy but also by their
memory footprint and inference time. Therefore, this work extends existing
findings by analysing the effects of DNN compression not only on accuracy but
also on relevant deployment metrics, i.e. memory consumption, latency, and
energy consumption.

Furthermore, research that focuses on the deployment of DNNs to
microcontrollers is often published in an application-oriented way, e.g., to
realize real-time drowsiness detection [21] or to perform motor fault
diagnosis [22]. Those platforms do not serve as general purpose frameworks as
they are tightly optimized to the particular application requirement and as
they do not generalize to a broader set of target architectures.

Nevertheless, recent scientific work has provided some insight into
generalized deployment of DNNs on microcontrollers. The approach most closely
related to our proposed methodology is MCUNet [23]. Similar to our pipeline
the authors describe a two stage process to seamlessly combine model design
(TinyNAS) with an inference engine (TinyEngine). Still MCUNet differs from our
approach in the way it generates suitable DNN candidates for deployment. To
find networks that meet target platform constraints, MCUNet focuses on neural
architecture search (NAS) [24] while our framework starts from well-known
existing DNN architectures and then dynamically scales them down during their
training using pruning and quantization techniques.

A more general approach to a deployment framework for microcontrollers is
_tfl-micro_ [25], which supports the execution of quantized tensorflow lite
models on ARM Cortex-M-based hardware using ARM’s _CMSIS_ library. However,
this also limits the framework as it requires the usage of _tensorflow (TF)_
for model training and also only supports a subset of features implemented in
TF.

Besides that there are also commercial frameworks focusing on embedded
platforms. Noticeable examples are STM’s _X-CUBE-AI_
111https://www.st.com/en/embedded-software/x-cube-ai.html, that allows for the
automatic conversion of pre-trained AI algorithms to deployable code as well
as web-service based end-to-end solutions like _Edge Impulse_
222https://www.edgeimpulse.com/ or _SensiML_ 333https://sensiml.com/. However,
such commercial frameworks are often either black boxes (e.g., X-CUBE-AI) or
they base themselves on already existing underlying solutions (e.g., _Edge
Impulse_ uses _tfl-micro_) and their limitations.

##  III Compression and Deployment Pipeline

Our pipeline is fully integrated and seamlessly covers the complete DNN
training and deployment process. Our methodology uses both network pruning
(Sec. III-A) and weight quantization (Sec. III-B) which can both be controlled
via a set of additional _hyperparameters_. Furthermore, the trained and
compressed DNNs are directly converted from their graph-based representation
to executable architecture-specific program code (see Sec. IV). As a result,
our pipeline can easily be integrated with existing meta-heuristic
optimization frameworks (e.g. Optuna [26]) to conduct automated design space
exploration.

###  III-A Network Pruning

Our pipeline implements configurable elements for network pruning, i.e., (1)
pruning techniques, (2) pruning heuristics, and (3) pruning schedule. which we
describe in the following.

Pruning techniques. Pruning DNNs by removing parameters has been proposed
early [27, 9]. While initially being introduced to improve generalization and
convergence it recently became a standard size reduction technique with no or
low cost of accuracy. Our pipeline implements _element-wise pruning_ and
_structural pruning_. Element-wise pruning removes connections from a DNN’s
compute graph, i.e., parameters of the network are set to zero. Hence, these
parameters do no longer influence the training error and are removed from the
scope of the optimizer that trains the network. Structural pruning sets whole
structures of parameters to zero. This has shown to be very efficient for
pruning filters [14] or channels [15] of 2D-convolutional layers but it can
analogously also be applied to rows and columns of linear layers. Its major
benefit is the removal of complete structures from the weight tensors at once,
which results in a considerable immediate reductions of parameters (which is
in contrast to element-wise pruning that only creates sparse weight tensors).

Pruning heuristics. A critical aspect of pruning is the selection of elements
or structures that, when removed, have the least impact on predictive
performance. Oracle pruning [28] finds an optimal selection by removing every
single structure and element of a network before evaluating its impact on the
prediction quality. In practical applications this approach cannot be applied
as it is too resource- and time-consuming. Fortunately, there have been
proposed a number of heuristics that approximate optimal element-wise or
structural pruning. In our framework we implemented many popular approaches
that are based on different criteria such as magnitude [29], L-norm [14],
gradient [28] or percentage of zeros found in activations [30] to rank
parameters or parameter structures by their approximated importance.

Pruning schedules. The pruning schedule determines when, how often, and to
what extent a DNN will be pruned during training. We implement two well-known
approaches: _One-Shot Pruning_ [9] and _Iterative Pruning_. One-shot pruning
first trains a DNN until it achieves a reasonable accuracy on the test
dataset, and then prunes the DNN (optionally followed by a few epochs of re-
training). Iterative pruning prunes a DNN over the course of training, which
allows for an interleaved re-training. Hence, not all weights are removed at
the same time but step by step over several pruning iterations (finally
enforcing maximal sparsity). We implemented _Automated Gradual Pruning (AGP)_
[31], which gradually increases the number of pruned weights $s_{t}$ starting
at $t_{0}$ from an initial sparsity $s_{i}$ to a final sparsity $s_{f}$ over
$n$ steps:

$s_{t}=s_{f}+(s_{i}-s_{f})\left(1-\frac{t-t_{0}}{n\Delta t}\right)^{3},\
t\in\left\\{t_{0},\dots,t_{0}+n\Delta t\right\\}.$ (1)

###  III-B Weight Quantization

Quantization reduces the numerical resolution of the parameters and their
computation. This not only cuts down the memory footprint but also the
computational complexity of a DNN. However, as parameter quantization comes
with an additional error on the predictive performance a major challenge lies
in the selection of a good trade-off between predictive quality and parameter
resolution.

Our framework uses an _affine mapping_ scheme that transforms an original
floating-point parameter into an 8-bit unsigned integer444See also [32] and
https://onnxruntime.ai/docs/how-to/quantization.html.. We apply a function
$f(x)$ in combination with additional sets of trainable scale and zero point
parameters:

$\begin{split}f(x)&=g\left(\left\lfloor\frac{x}{s}\right\rceil+zp\right),\\\
s&=\frac{max_{data}-min_{data}}{255},\ 0\leq zp\leq 255,\end{split}$ (2)

where $g(x)$ is the clamp-function to avoid data type overflows:

$g(x)=\left\\{\begin{array}[]{rcl}x&\mbox{if}&0\leq x\leq 255\\\
255&\mbox{if}&x>255\\\ 0&\mbox{if}&x<0.\end{array}\right.$ (3)

The scale parameter $s$ defines the step size of the quantization, which is
calculated as the ratio between the span in which values are distributed in
the original floating-point space and the span covered by the quantized
integer space. The zero point parameter $zp$ denotes an exact point in
quantization space that represents zero in the original floating-point space.
The two parameters can be defined either per tensor or per structure.

Quantization can not only be applied to weight tensors but also to activation
tensors. We refer to this as _full integer quantization_. During execution
most computations can then be performed in integer instead of floating-point
space which is beneficial for target systems that are not equipped with
floating-point units. We give an example for applying full-integer
quantization to matrix-multiplications. The general form is defined as:

$\begin{split}c_{ij}=\sum_{k=0}^{n}a_{ik}\cdot b_{ki},\forall
i\in\\{0,\dots,m\\},\forall j\in\\{0,\dots,p\\},\end{split}$ (4)

where the first line describes how the elements of a matrix $C$ are calculated
from the elements of a $m\times n$ matrix $A$ and a $n\times p$ matrix $B$. In
a fully-quantized DNN, both matrices $A$ and $B$ contain integer values and we
first must de-quantize them by rearranging Equation 2 before we multiply them.
As the resulting matrix $C$ is represented in the un-quantized space, we have
to quantize it by applying Eq. 2 again. By substituting and rearranging the
previous computations we obtain

$c_{ij}=g\left(zp_{c}+\left\lfloor{\left(\frac{s_{a}\cdot
s_{b}}{s_{c}}\right)\sum_{k=0}^{n}(a_{ik}-zp_{a})(b_{ik}-zp_{b})}\right\rceil\right).$
(5)

Note that only the scale parameters $\\{s_{a},s_{b},s_{c}\\}\in\mathbb{R}$
while all other parameters $\in\mathbb{N}_{0}$.

Our pipeline implements two popular ways of determining at which point
quantization is applied to a DNN. The first method quantizes as a post process
(PPQ) [11, 33], i.e., after training has finished, and the second method
integrates quantization into the training loop. The latter is denoted by
_Quantization-aware Training (QAT)_ [10]. Both techniques come with their
advantages and disadvantages: PPQ is extremely easy to integrate as it can be
performed completely decoupled from a DNN’s training process and does not
requires any invasive changes to a DNN’s architecture (i.e., no re-training to
fine-tune quantization parameters). However, this usually comes at the cost of
a larger error introduced by quantization as the required scale and zero point
parameters are only roughly approximated. In contrast, QAT adapts quantization
parameters as part of a DNN’s training process and can hence yield better
results. However, QAT only works properly with extensive network augmentation,
which leads to a more complex and computationally expensive training process.

##  IV Architecture-Specific Deployment

Our pipeline provides a deployment framework for targeting microcontrollers,
see Fig. 2. We call this framework _dnnruntime_. It uses a platform-
independent, offline, and ahead of time conversion tool together with a
runtime library. The conversion tool maps pre-trained DNNs stored in the ONNX
format to C code (Sec. IV-A), while the runtime library implements platform-
specific DNN operators that are subsequently used by the code emitted from the
conversion tool (Sec. IV-B). Our implementation is novel in the way that it
exploits static properties of trained DNNs (i.e. fixed layer configurations
and parameters). and therefore removes the necessity of interpreting the DNN
at runtime. This includes dynamic allocation of memory for intermediate
tensors which can be simulated offline allowing heap allocation to be
conducted at compile time. This not only decreases the computational overhead
at runtime but also allows metrics like simulated memory consumption to be
directly fed back into the overall optimization process without having to
evaluate the model on the target system.

Figure 2: Overview of the process implemented by our conversion tool, mapping
trained ONNX DNN models to C-Code and finally compiling them to deployable
binaries.

###  IV-A Conversion Tool

The main functionality of the conversion tool is to generate ANSI C code based
on a given ONNX model of the DNN to be deployed. This involves two steps: (1)
parsing and converting the model to an intermediate representation, and (2)
using this representation to determine a suitable data format, simulate memory
allocation and generate an implementation describing the model’s structure in
code.

The ONNX format stores a DNN’s compute graph as a directed, acyclic graph with
a set of fixed inputs and outputs. Every node in the graph represents an
operation and can have multiple incoming and outgoing edges assigned to it.
Edges describe the dataflow during the DNN’s execution. Besides that, based on
the type of operation a node represents, additional static parameters tensors
can also be assigned to it.

####  IV-A1 Mapping ONNX to target-specific intermediate format

We first map a given ONNX representation to an architecture-specific
intermediate format that can be used to emit program code later on. This
involves three consecutive steps.

First, we concatenate the static tensors of all nodes into a byte-stream. The
single elements of each tensor are stored in the stream by using a little-
endian byte order as this is the default memory format on ARM architectures
(of course, this can be modified easily). Additionally, we add padding bytes
where necessary to avoid triggering the memory protection unit (MPU) when
accessing tensor data at runtime. Afterwards, we generate descriptor
structures containing the location of each tensor in the byte-stream and
additional metadata such as data types and tensor shapes. Sparse tensors are
handled as edge cases as they are generated by element pruning during our
pipeline’s compression stage. To reduce memory usage, our tool applies a
conversion from the original full-sized layout of the tensors to a more
compact _Compressed Row Storage (CRS)_ [16] layout, see Fig. 3 for an example.
CRS reduces the memory footprint, allows for an optimized implementation of
matrix-vector products, and does not pose any requirements on how sparsity is
distributed inside a tensor (hence, element pruning during compression can
ignore the subsequent space-saving storage of pruned tensors). A disadvantage
of CRS is that it can only be applied to 2D-tensors (matrices), which means
that we need to map higher dimensional weight tensors of convolutions to 2D
space before processing them.

$\displaystyle A=\begin{bmatrix}10&0&0&0&1\\\ 0&7&0&2&0\\\ 0&0&8&0&0\\\
14&0&0&0&6\end{bmatrix}$

values | 10 | 1 | 7 | 2 | 8 | 14 | 6
---|---|---|---|---|---|---|---
col_ind | 0 | 4 | 1 | 3 | 2 | 0 | 4
row_ptr | 0 | 2 | 4 | 5 | 7 |  |

Figure 3: Conversion of an asymmetric matrix $A$ (left) into its CRS
representation (right): we emit three smaller arrays to the bytes-stream
instead of one. The first one contains all non-zero values, the second one
contains column indices, and the third one contains row pointers.

Second, the conversion tool generates descriptor structures for all dynamic
activation tensors. This is more compilicated than it is for static parameter
tensors as activation tensors are only represented in the ONNX model’s compute
graph as edges. Edges are not required to provide any meta information like
data types or shapes. However, this information is mandatory for our
conversion tool. Hence, we implement a process called _shape inference_. The
idea is to trace the execution of a DNN through its compute graph from input
to output nodes, and to use these traces to infer the shapes and types of
intermediate tensors assigned to inner edges.

Third, we parse and interpret all operator nodes in the ONNX compute graph and
bring them into a topological and serialized order. This information is then
used during code generation to determine the execution order of operations.

####  IV-A2 Code Generation

Using the intermediate representation generated by the first step the
conversion tool can emit code. We start by estimating the minimal heap size
required for storing activation tensors. This information can be queried
offline as once the training of a DNN is complete, its structure and
dimensionality remains unchanged throughout its lifetime. Using the minimal
heap size, we define a fixed-size memory balloon at compile time (eliminating
the need for dynamic memory management at runtime). A naive approach that
estimates the size of this balloon calculates the product of the shapes of all
activation tensors and multiplies them with the byte sizes of their respective
data types. However, this is not space-efficient as usually the lifetimes of
these tensors are rather short. Hence, parts of the heap memory can be reused
for multiple tensors during an inference pass. This may have a big impact on
the amount of memory required.

We take advantage of this by implementing an offline _memory planning
algorithm_ based on graph tracing and using a first-fit allocation strategy
therein. We estimate optimal heap re-usage in two steps: First, based on
incoming and outgoing edges of nodes (i.e., operators) in the input model, the
algorithm creates two lists per operator: The first list contains all tensors
that have to be allocated for that operator (i.e., allocation list) and the
second list contains all previously allocated tensors that can be discarded
(i.e., release list). Second, the algorithm proceeds iterating through the
sequence of operators starting with an empty, infinitely sized memory balloon.
For each operator, it first iterates the tensors in the corresponding free
list and marks their space in the memory balloon as unoccupied. After that, it
iterates the allocate list and tries to reserve pieces of memory based on the
shapes and data types of the tensors. To find suitable locations in the
balloon, the algorithm compares the required sizes with available segments of
free memory starting from the beginning (i.e., first fit). Once it found
suitable pieces of memory, it marks them as allocated in the balloon and stops
searching. During all steps, the algorithm keeps track of the maximum size of
the memory balloon.

The emitted code implements a two-function API: The first function allows to
setup the converted DNN and the second function executes an inference given an
input sample. The latter is implemented based on the list of topologically
sorted ONNX operator nodes stored in the previously generated intermediate
representation. For each operator a function call is emitted. These functions
are implemented by the _runtime library_. To give context to these function
and pass intermediate results between them, we provide references to constant
tensor descriptor structures generated as part of the intermediate
representation. All static data associated with weight tensor descriptors is
stored in a byte-array in the intermediate representation. Therefore, a
constant C-array (i.e. flash memory) containing all the data is emitted,
accordingly.

The amount of random access memory required for intermediate activation
tensors is based on the minimal memory balloon previously estimated by our
_memory planning algorithm_. Hence our tool emits another accordingly sized
zero initialized non-constant C-array (i.e. heap memory).

###  IV-B Runtime Library

To perform the operations described by the input ONNX model, the code emitted
by our conversion tool relies on additional DNN operator functionality that we
implement by a runtime library. Currently, this includes operators such as
convolutions, linear transformations, batch-normalization, pooling operations,
and activation functions. All our implementations follow the ONNX operator
specification555https://github.com/onnx/onnx/blob/master/docs/Operators.md.
Where required, we also implement quantized versions of these operators.

Based on the target platform there are different possibilities to optimize the
execution of DNNs. During profile tests we found that most execution time is
spent on computing convolutions or matrix-vector products. Hence, an optimal
implementation of these operator types yields significant improvements in both
resource consumption and execution time. Less crucial but still significantly,
some operations can be removed from a DNN’s compute graph by applying graph
optimization, which we apply after DNN training and compression. Notable
optimization techniques include _batch normalization folding_ [10] and the
fusing of ReLU activation functions into preceding quantized linear or
convolutional operations [10].

For our experiments we focus on Cortex-M0+ and Cortex-M4 processors. This is
why our implementation heavily makes use of the information of these processor
architectures. A major algorithmic optimization that we apply is to unroll 2D
convolutions into more CPU- and memory-friendly matrix-vector products
(_im2col mapping_). Hence, during mapping we rearrange both the input tensors
and the parameter tensors of convolutions. This is a common approach used in
digital signal processing.666e.g., Matlab® provides convmtx() and convmtx2()
for this task.. In addition, this mapping also enables our conversion tool to
apply CRS to convolutional layers.

Since DNNs also use matrix-vector products in linear transformations, a nice
additional side-effect is that by using the im2col mapping complete inference
passes can be described by matrix-vector products and non-linearities alone.
Moreover, ARM provides optimized open-source implementations for matrix
products in their _CMSIS_ library777https://github.com/ARM-software/CMSIS_5.
Using them is especially beneficial on architectures like the Cortex-M4 as it
allows to use SIMD instructions provided by ARM’s _Digital Signal Processing
(DSP)_ extension, see Sec. V.

##  V Evaluation

TABLE I: Parameters of the DNN architectures used in our experiments. AlexNet (44.7M) |  ResNet (9.4M) |  LeNet (1.2M)
---|---|---
Input: [3, 32, 32] |  Input: [1, 28, 28]
[3, 64, 2, 2] |  [64, 64, 3, 1] |  [1, 32, 3, 1]
[64, 192, 3, 1] |  [64, 128, 3, 1] |  [32, 64, 3, 1]
[192, 384, 3, 1] |  [128, 256, 3, 1] |
[384, 256, 3, 1] |  [256, 512, 3, 1] |
MaxPool: [2]
[6400, 4096] |  [25088,10] |  [9216, 128]
[4096, 4096] |  |  [128, 10]
[4096, 10] |  |
Output: [10]
SoftMax
Tuples describe linear layers in the form of [num. inputs, num. outputs].
Every layer uses ReLU as their non-linearity except the last ones which are
followed by SoftMax; AlexNet/LeNet: quadruples describe 2D-convolutions by
[channels, filters, kernel size, stride]; ResNet: quadruples describe residual
blocks each with two 2D-convolutions in the form of [block in. channels, block
out. filters, conv. kernel size, conv. stride]; AlexNet/ResNet: every
2D-convolution is followed by batch normalization.
TABLE II: Microcontrollers considered in our evaluation. |  Raspberry Pi Pico |  Arduino Nano 33 BLE Sense |  Raspberry Pi 4B
---|---|---|---
Processor |  RP2040, Cortex-M0+ (Armv6-M) |  nrf52840, Cortex-M4 (Armv7-M) |  BCM2711 SoC, Cortex-A72 (ARMv8-A)
Clock |  133 MHz |  64 MHz |  1.5 GHz
Flash |  2 MB |  1 MB |  16 GB (SD-Card)
RAM |  256 KB (SRAM) |  256 KB (SRAM) |  8 GB (SDRAM)
SIMD |  No |  Yes (ARM DSP) |  Yes (ARM Neon)

(a) AlexNet on CIFAR-10.

(b) ResNet on CIFAR-10.

(c) LeNet on MNIST.

Figure 4: Element-wise and structural pruning applied to the DNN
architectures. The curves describe the relation between theoretical model size
and accuracy of compressed models relative to their uncompressed baselines
when using different pruning techniques in combination with an iterative AGP
pruning schedule. We define theoretical model size to be the number of weights
a model features excluding all weights that have been set to zero by pruning.

To evaluate our pipeline we selected three popular DNN architectures: (1) a
convolutional network similar to the one proposed by Krizhevsky et al. [17] to
classify CIFAR-10 images (AlexNet), (2) a residual network [3] (ResNet), and
(3) a smaller network architecture initially proposed by LeCun et al. [18]
(LeNet) for classifying the MNIST handwritten digit database. See Table I for
more details.

We trained AlexNet and ResNet on the CIFAR10 [4] dataset for 100 epochs with
mini-batches of size 80 and LeNet on the MNIST handwritten digit datasets [34]
for 20 epochs with mini-batches of size 48 (as training converges on MNIST
considerably faster). On all the models we used stochastic gradient descent
(SGD) with a momentum of 0.9 and a learning rate of $1e-3$. We achieved a
maximum accuracy of 86.51% for AlexNet, 85.97% for ResNet, and 98.46% for
LeNet that serve as baselines for our experiments.

To evaluate our deployment pipeline we selected three target systems: (1) a
Raspberry Pi Pico, (2) an Arduino Nano 33 BLE Sense, and (3) a Raspberry Pi 4B
(that serves as a larger reference system), see Table II. As our runtime
library is mainly optimized for Cortex-M architectures, we instead use
onnxruntime [32] for the deployment to the Raspberry Pi 4B.

We compare the performance of different pruning techniques in Sec. V-A and
discuss their combination with quantization in Sec. V-B. We analyze the memory
footprint of our compressed DNNs in Sec. V-C. In Sec. V-D we discuss the
execution time and power/energy consumption w.r.t. the predictive accuracy
from a real-world execution of the compressed DNNs on the target platforms.

###  V-A Comparison of Pruning Techniques

First, we present results of pruning experiments conducted for each of our
three test DNN architectures. We repeated model training from scratch and
increased pruning target rates starting with 0% as the un-pruned baseline and
ending with 99% (i.e., a relative theoretical model size of 1%) as the most
aggressively pruned configuration. For each of the configurations we repeated
the experiment five times and report their means and standard deviations.

Figure 5: Comparison of pruning schedules for both structural and element-wise
pruning with AlexNet on CIFAR-10 (L1: pruning selected parameters based on
$\ell_{1}$-norm; Level: selects parameters based on magnitude level).

Figs. 4 and 5 show the predictive accuracy of the models (relative to the un-
pruned baseline) over percentages of theoretically remaining parameters on the
validation dataset. We first investigate the influence of pruning schedules on
the achievable pruning rate, see AlexNet on CIFAR-10 in Fig. 5. While we
cannot observe major differences between iterative (i.e., Automated Gradual
Pruning, AGP) and one-shot schedules for final parameter counts $>10\%$, we
see that iterative schedules perform slightly better both for element-wise and
structural pruning for parameter counts below $10\%$. We argue that this is
because it is more difficult for the network to retrain when a large number of
parameters are removed at once than when parameters are removed gradually and
retraining is possible in between. Hence, we decided to focus on iterative
pruning schedules for all further experiments.

With the iterative AGP schedule on the same experimental setup we tested
different pruning heuristics for structural and element-wise pruning on all
architectures, see Figs. 4(a) to 4(c). We report four different heuristics for
structural pruning and one for element-wise pruning alongside a random
selection (’Random’) approach as a baseline for the more elaborate heuristics.
For structural pruning, we use both the $\ell_{1}$\- and the $\ell_{2}$-norm
of parameter structures (’L1’ and ’L2’) as well as their gradient size
(’Gradient’) and the average percentage of zeros in their corresponding
activation (’Activation’) as heuristics. For element-wise pruning we use a
magnitude level to decide which elements to prune (’Level’).

While we see a significant improvement of the level-based heuristic over the
corresponding random selection for element-wise pruning on all our three DNN
architectures, we cannot observe a similar behaviour for structural pruning.
Instead, none of the more complex structural pruning heuristics managed to
significantly improve over the random selection approach. There is also only
little variation in the results of the heuristics. We believe that the main
reason for this is the iterative re-training between pruning steps: while
introducing pruning during DNN training can cause degradation in the
predictive quality of a model, it was very often regained in a short number of
epochs when re-training. This is in line with results reported in previous
work [29].

In all our experiments we used the same target compression rates for both
element-wise and structural pruning. However, we see that the structural
pruning experiments resulted in models that exceed their selected targeted
compression rate. In some cases this reduces the parameter count to almost
99.9%. For element-wise pruning, we do not see such an effect. The reason for
this is related to the removal of structures from DNN models during structural
pruning: due to the existence of data dependencies between layers, removing
structures from their parameter tensors also influences the shapes of tensors
in surrounding layers. For element-wise pruning, parameters are not completely
eliminated from the DNN but are instead just set to zero. Therefore, all data
dependencies remain in the network, no tensor shape changes occur, and the
pruning target rate is more precisely met.

###  V-B Combining Pruning and Quantization

We present experimental results for weight quantization in combination with
pruning for our three test DNN architectures. Fig. 6(a) shows the results for
quantization in combination with structural pruning and Fig. 6(b) shows the
results for element-wise pruning. The different colors refer to the models we
trained and quantization strategies are differentiated with the line and
marker style. We aim to give an understanding on how much the additional
quantization error alone influences the prediction quality of quantized
models. Hence, as before, the $x$-axes show the relative theoretical model
size reduction while this time the $y$-axes reports the accuracy decrease of
each pruned and quantized model in relation to its pruned but not quantized
version.888We want to point out that quantization does not change the number
of parameters in a model, only their resolution, so the application of
quantization does not affect the relative theoretical model size, although the
model effectively becomes smaller. We present our findings on actual size
reduction achievable with quantization in Sec. V-C. This allows us to focus on
the additional error that is introduced through quantization alone.

When looking at the results of structural pruning in combination with both
Quantization as a post process after training (PPQ) and Quantization Aware
Training (QAT), we see that the techniques work well together with pruning for
all our three architectures. In Fig. 6(a) we see that the accuracy decrease
consistently is $<5\%$ even when using quantization in combination with
aggressive pruning regimes. The only outliers we monitored were part of our
experiments on the LeNet architecture. Here, for the two most aggressive
pruning configurations, the accuracy decrease between the non-quantized and
quantized models went to around 40% for both tested quantization strategies.
Moreover, we observed an increase in standard deviation as accuracy decreased,
which we believe is related to an increase in variance in the trained weight
values that we observed for LeNet at higher compression rates. The higher the
variance of the values in a weight tensor, the worse quantization can
represent these values in integer space.

(a) L1-norm AGP structural pruning

(b) Level AGP element-wise pruning

Figure 6: Decrease in accuracy on the evaluation dataset observed for our
three DNN architectures when combining pruning and quantization techniques.

We also tested element-wise pruning in combination with PPQ and QAT, see Fig.
6(b). Other than for structural pruning, where PPQ performed consistently well
even in combination with aggressive pruning configurations, for element-wise
pruning we observed accuracy decreases of over 70% in comparison to the un-
quantized versions. In particular, we observe that PPQ noticeably failed for
models that have been compressed by element-wise pruning to 10% or less of
their original parameter count. This is despite the fact that the technique
performed well for pruning configurations that target compression rates above
10%. In contrast, QAT performed significantly better than PPQ even when used
together with aggressive pruning configurations. The technique was able to
keep the accuracy decrease very close to 0% during all conducted experiments.
Hence, we conclude that PPQ generally seems to perform better when being
applied in combination with structural pruning than when used with element
pruning. QAT on the other hand performed consistently well, both in
combination with structural and element-wise pruning.

(a) Flash consumption of the compressed models (left: AlexNet/CIFAR-10;
middle: ResNet/CIFAR-10; right: LeNet/MNIST).

(b) SRAM consumption of the compressed models (left: AlexNet/CIFAR-10; middle:
ResNet/CIFAR-10; right: LeNet/MNIST).

Figure 7: Memory required for the deployment of our three DNN architectures
(AlexNet, ResNet, LeNet) when applying the proposed runtime environment. (a)
shows the required amounts of read-only ”Flash” memory to store the DNNs’
trained weights, while (b) shows the required amounts of random access memory
”SRAM” to store the dynamic activation tensors. The dashed horizontal red and
green lines mark the memory limits of our two considered embedded target
platforms, the Raspberry Pi Pico and the Arduino Nano 33 BLE Sense.

###  V-C From Size Reduction to Memory Consumption

As described in Sec. IV the execution of DNNs requires both static read-only
and dynamic random access memory. We now discuss the memory footprint of the
models when they are being deployed. Fig. 7(a) shows the relation between
relative model size (on the $x$-axes) and ROM/flash consumption in Kibibyte
(on the $y$-axes) and Fig. 7(b) shows the same relation for SRAM. The curves
for the pruned models are drawn with solid lines while the curves for the
models where additional quantization was applied are dashed. To also highlight
the importance of model compression we added the Flash and SRAM limits of out
two microcontroller target platforms (red and green dashed horizontal lines).
On both of these platforms flash and SRAM availability are two of the main
bottlenecks for deployment.

We see a linear correlation between flash consumption and relative theoretical
model size for structural pruning in Fig. 7(a) (note that both axes are in
logarithmic scale). The reason for this is that pruned structures can be
completely removed from the model which immediately decreases its memory
consumption in the process. For element-wise pruning, there is no such direct
correlation. Instead, when looking at the orange curves in Fig. 7(a) we
observe plateaus for higher theoretical model sizes. This is because pruning
only generates sparse weight matrices. Our runtime environment utilizes
compressed row storage (CRS) as a special decoding technique to store sparse
weight matrices space-efficiently. However, a characteristic of the technique
is that decoding a tensor will only result in memory savings after a certain
percentage of sparsity has been reached. For any amount of sparsity below this
threshold, it is better to just use the default memory layout where all
elements are saved sequentially.

An additional observation we made is that for models that are compressed using
quantization and element-wise pruning the threshold at which CRS becomes
feasible is much higher than for just pruned models. Again, this is related to
the properties of CRS decoding. Instead of all values, only values unequal to
zero (or a zero point) are stored. To preserve their position in the original
un-decoded matrix, the row and column indices of the values must be stored as
well. For larger matrices, like they exist in DNNs, these indices usually
require 16- or 32-bit integers to be stored correctly. Therefore, memory
savings made by CRS can be considered as a trade-off between storing some
elements and their indices versus storing all elements without any indices.
When introducing sparsity into matrices with 32-bit floating point values,
this quickly becomes a good trade-off. However, since quantized values require
only 8 bits, while the index values introduced by CRS are still usually at
least 16 bits long, the amount of sparsity that has to be introduced before
memory can be saved is higher.

Fig. 7(b) shows the relationship between relative theoretical model size and
required SRAM. Similar to flash consumption, a correlation between model size
and SRAM consumption can be observed for structural pruning. When structures
are removed from parameter tensors during pruning, data dependencies between
layers are also removed. Therefore, the shapes of the dynamic intermediate
activation tensors stored in RAM and shared between succeeding layers reduce
as well. The reason why the relation is not perfectly linear is because our
runtime library tries to re-use heap memory for several activation tensors.
How good this strategy works and how much memory it can save depends on the
topology of the DNN that it is applied to. We cannot observe a reduction in
SRAM consumption for element-wise pruning. This is because we do not remove
any elements during element-wise pruning. While CRS can compress the sparse
parameter tensors, it cannot change their original shapes. Therefore, as
expected, no data dependencies are removed during decoding and as a result all
intermediate activation tensors retain their original sizes.

TABLE III: Average ($n=8$) execution time, power and energy consumption observed per inference for compressed version of LeNet targeting two relative accuracy levels. The baseline model achieved an accuracy of $\approx$98% for both float and uint8. rel. accuracy | compression | type | system | execution time (ms) | power (mW) | energy (mWs)
---|---|---|---|---|---|---
$>$99.00% | 99.18% | structure, float | arduino |  $44.15$ p m 0.03 |  $141.14$ p m 0.07 |  $6.23$ p m 0.01
|  |  | pico |  $184.28$ p m 0.17 |  $81.42$ p m 0.42 |  $15.00$ p m 0.07
|  |  | pi4b |  $1.81$ p m 0.55 |  $3754.70$ p m 284.92 |  $6.65$ p m 1.35
|  | structure, uint8 | arduino |  $18.84$ p m 0.04 |  $101.06$ p m 1.80 |  $1.90$ p m 0.03
|  |  | pico |  $50.06$ p m 0.03 |  $79.37$ p m 1.03 |  $3.97$ p m 0.05
|  |  | pi4b |  $2.16$ p m 0.60 |  $3624.90$ p m 272.88 |  $7.68$ p m 1.53
| 95.00% | element, float | arduino | — | — | —
|  |  | pico |  $1004.77$ p m 0.07 |  $81.44$ p m 0.10 |  $81.82$ p m 0.11
|  |  | pi4b |  $5.66$ p m 1.39 |  $3772.18$ p m 263.01 |  $21.00$ p m 3.34
|  | element, uint8 | arduino |  $360.95$ p m 0.09 |  $142.28$ p m 0.10 |  $51.36$ p m 0.04
|  |  | pico |  $392.23$ p m 0.07 |  $84.82$ p m 0.37 |  $33.27$ p m 0.15
|  |  | pi4b |  $5.10$ p m 1.27 |  $3673.70$ p m 332.91 |  $18.31$ p m 3.04
$>$97.00% | 99.7% | structure, float | arduino |  $22.48$ p m 0.01 |  $105.32$ p m 0.27 |  $2.37$ p m 0.01
|  |  | pico |  $73.26$ p m 0.33 |  $81.72$ p m 0.48 |  $5.99$ p m 0.03
|  |  | pi4b |  $1.91$ p m 0.56 |  $3554.33$ p m 232.34 |  $6.67$ p m 1.48
|  | structure, uint8 | arduino |  $18.79$ p m 0.03 |  $82.54$ p m 0.80 |  $1.55$ p m 0.02
|  |  | pico |  $24.97$ p m 0.44 |  $79.92$ p m 0.88 |  $2.00$ p m 0.04
|  |  | pi4b |  $2.01$ p m 0.53 |  $3581.79$ p m 240.22 |  $7.11$ p m 1.41
| 97.00% | element, float | arduino | — | — | —
|  |  | pico |  $658.70$ p m 0.10 |  $82.44$ p m 0.68 |  $54.30$ p m 0.45
|  |  | pi4b |  $5.35$ p m 1.23 |  $3816.72$ p m 246.08 |  $20.12$ p m 2.79
|  | element, uint8 | arduino |  $303.15$ p m 0.09 |  $141.58$ p m 0.05 |  $42.92$ p m 0.01
|  |  | pico |  $392.27$ p m 0.10 |  $84.67$ p m 0.45 |  $33.22$ p m 0.18
|  |  | pi4b |  $4.64$ p m 1.39 |  $3787.84$ p m 395.19 |  $17.05$ p m 3.16

###  V-D Deployment Results

As the last step of evaluating our pipeline, we deployed several of the pruned
and quantized models from our previous experiments on our target systems and
monitored key runtime metrics. We especially focused on execution time per
inference, power, and energy consumption. For measuring these metrics on our
test systems, we used an Agilent N6705A DC Power Analyzer to provide them with
a regulated power supply. Furthermore, the Power Analyzer allowed us to
measure the current and power drawn by the systems. To measure the execution
time required for calculating the energy consumption of our DNN models, we
used a GPIO signal. We toggled the signal every time an inference started and
finished and monitored it using an oscilloscope. We present the results of our
measurements for LeNet in Table III and the results for our other two DNN
architectures in Table IV.

TABLE IV: Average ($n=8$) execution time, power and energy consumption
observed per inference for AlexNet and ResNet. To achieve different
compression rates and thus different accuracy scores, we used L1-norm
structural pruning. In addition, we used post training quantization for all
non floating-point measurements.

compression | type | rel. accuracy | system | execution time (ms) | power (mW) | energy (mWs)
---|---|---|---|---|---|---
95.9% | float | 97.79% | arduino | — | — | —
|  |  | pico | — | — | —
|  |  | pi4b |  $6.60$ p m 1.85 |  $3795.81$ p m 334.67 |  $24.43$ p m 4.10
| uint8 | 97.56% | arduino | — | — | —
|  |  | pico |  $1766.58$ p m 0.25 |  $92.40$ p m 0.03 |  $163.23$ p m 0.07
|  |  | pi4b |  $3.96$ p m 1.12 |  $3809.41$ p m 250.59 |  $14.82$ p m 2.71
98.2% | float | 94.88% | arduino | — | — | —
|  |  | pico | — | — | —
|  |  | pi4b |  $2.68$ p m 0.70 |  $3687.37$ p m 221.25 |  $9.71$ p m 1.76
| uint8 | 94.65% | arduino |  $403.11$ p m 0.16 |  $139.96$ p m 0.17 |  $56.42$ p m 0.07
|  |  | pico |  $669.97$ p m 0.19 |  $86.41$ p m 0.02 |  $57.89$ p m 0.03
|  |  | pi4b |  $2.93$ p m 0.41 |  $4187.75$ p m 89.73 |  $12.28$ p m 2.05
99.6% | float | 83.26% | arduino |  $252.40$ p m 0.40 |  $107.83$ p m 0.42 |  $27.22$ p m 0.10
|  |  | pico |  $1435.42$ p m 0.34 |  $81.55$ p m 0.07 |  $117.06$ p m 0.09
|  |  | pi4b |  $2.45$ p m 0.09 |  $4182.59$ p m 66.35 |  $10.25$ p m 0.35
| uint8 | 82.33% | arduino |  $110.00$ p m 0.01 |  $141.17$ p m 0.04 |  $15.53$ p m 0.01
|  |  | pico |  $186.57$ p m 0.12 |  $86.86$ p m 0.21 |  $16.21$ p m 0.04
|  |  | pi4b |  $2.10$ p m 0.14 |  $4134.21$ p m 52.03 |  $8.68$ p m 0.60
(a) AlexNet, CIFAR10, baseline accuracy $\approx$86% for both float and uint8

compression | type | rel. accuracy | system | execution time (ms) | power (mW) | energy (mWs)
---|---|---|---|---|---|---
93.2% | float | 96.25% | arduino | — | — | —
|  |  | pico | — | — | —
|  |  | pi4b |  $29.41$ p m 2.62 |  $4234.81$ p m 121.13 |  $124.24$ p m 7.11
| uint8 | 96.43% | arduino |  $11\,983.40$ p m 2.05 |  $143.74$ p m 0.49 |  $1722.48$ p m 5.69
|  |  | pico |  $25\,997.20$ p m 4.00 |  $91.94$ p m 0.01 |  $2390.24$ p m 0.41
|  |  | pi4b |  $21.32$ p m 3.32 |  $4085.92$ p m 213.90 |  $86.54$ p m 9.62
98.7% | float | 92.50% | arduino |  $6105.20$ p m 1.60 |  $105.76$ p m 1.85 |  $645.69$ p m 11.16
|  |  | pico |  $46\,266.50$ p m 16.82 |  $85.78$ p m 0.02 |  $3968.83$ p m 1.49
|  |  | pi4b |  $9.41$ p m 4.27 |  $4536.99$ p m 183.46 |  $42.06$ p m 16.53
| uint8 | 88.52% | arduino |  $2425.50$ p m 0.92 |  $102.38$ p m 1.10 |  $248.31$ p m 2.70
|  |  | pico |  $4269.78$ p m 2.58 |  $87.81$ p m 0.04 |  $374.92$ p m 0.36
|  |  | pi4b |  $7.74$ p m 2.78 |  $3863.07$ p m 427.70 |  $28.82$ p m 6.98
99.6% | float | 83.48% | arduino |  $1754.75$ p m 0.31 |  $106.57$ p m 2.64 |  $187.00$ p m 4.63
|  |  | pico |  $11\,140.25$ p m 1.85 |  $83.13$ p m 0.24 |  $926.12$ p m 2.61
|  |  | pi4b |  $5.42$ p m 1.66 |  $3560.66$ p m 421.12 |  $18.70$ p m 4.01
| uint8 | 78.28% | arduino |  $806.80$ p m 0.60 |  $104.48$ p m 0.09 |  $84.29$ p m 0.13
|  |  | pico |  $1214.53$ p m 1.00 |  $84.13$ p m 0.06 |  $102.18$ p m 0.10
|  |  | pi4b |  $3.45$ p m 0.22 |  $3881.06$ p m 72.77 |  $13.37$ p m 0.60
(b) ResNet, CIFAR10, baseline accuracy $\approx$85% for float and $\approx$84%
for uint8

For the LeNet architecture, we used the models’ accuracy on its evaluation
dataset to compare the deployment of different pruning and quantization
techniques. The logic behind this was that if models that were compressed
using different approaches can achieve a similar accuracy, then they can be
seen as direct alternatives and are therefore comparable. In our experimental
setup we defined two relative accuracy boundaries for which we selected the
smallest compressed models from our previous tests that meet them: $>$99% and
$>$97%. In the second column of Table III we can see that LeNet can be
compressed to well over 5% of its original parameter count while still passing
both accuracy thresholds. We tested all selected models on our three target
systems, including not only the pruned models but also their quantized
counterparts, see the third and fourth columns. Note that using element
pruning we were not able to deploy all selected models on the Arduino.

For all deployed LeNet models, we monitored execution time, power and energy
consumption over a span of 8 different inferences. Their resulting averages
are presented in the remaining columns of Table III. First, we see that the
execution time per inference on the Pi 4B is significantly lower than on the
Arduino or the Pi Pico. This is expected as the Pi 4B runs between 1.0 to 1.5
GHz while both the Arduino and the Pico run in a lower MHz range. When
comparing the Raspberry Pi Pico and the Arduino Nano we observed a higher
execution time on the Pico than on the Arduino (consistently). This is even
though the Arduino runs with a clock speed only around half as fast as the
Pico. On both systems the quantized models always outperformed their floating-
point counter parts although much more pronounced on the Pico. This can be
explained by the features present on both systems. First, the Arduino’s
Cortex-M4 processor implements a real floating point unit while on the Pi
Pico’s Cortex-M0+ processor floating point arithmetic has to be simulated.
Second, the Arduino’s Cortex-M4 processor supports ARM’s Digital Signal
Processing (DSP) extension giving it access to a subset of SIMD instructions
to accelerate its integer operations. The Pi Pico does not implement the DSP
extension.

Considering the power measured during inference for all deployed models, we
see that all our tested systems on average draw a constant amount of power
while we see a more significant variation in the different samples taken for
each model on the Pi 4B (note that the power consumption is much higher than
on the other two systems). On the Arduino, the measured power consumption was
between 100 to 150 mW on average while on the Pi Pico it was around 80 mW. In
contrast to that, the Pi 4B generally consumed around 4W. However, in addition
to power, execution time is the second factor in calculating a system’s energy
consumption. Looking at the results, we see that for some cases the Pi 4B
scored the best energy consumption per inference. It is often followed by the
Arduino and then the Pi Pico. This is in reverse to the power consumption and
shows that having an excellent runtime can compensate for high power
consumption.

Besides LeNet, we also deployed our AlexNet and ResNet architectures, see
Table IV. For both architectures we evaluated models which were compressed
using structural pruning. This is different from our LeNet experiments where
we tested both structural and element-wise pruning. The reason we did not do
this for AlexNet and ResNet is due to memory limitations on our Arduino and Pi
Pico target systems, which made it impossible for us to deploy any element
pruned models. For structural pruning the situation is different and we were
able to feasibly deploy models. However, we were still forced to select models
trained with aggressive compression rates that removed well over 90% of the
original parameters. As a consequence we had to accept decreases in accuracy
to be able to deploy models, see the third column of Tables 8(a) and 8(b).

To measure execution time, power, and energy consumption, we used the same
approach as before. We again monitored all three metrics over 8 different
inferences and calculated their averages and respective standard deviations.
When looking at the results for AlexNet and ResNet, we see the same patterns
we discussed for LeNet in Table III.

Yet we evaluated the two architectures as a way to explore upper boundaries of
feasible DNN deployment. AlexNet features a very high number of trainable
parameters while ResNet contains a high number of large and computationally
expensive 2D-convolutions. This affects the deployment of the two
architectures differently. While for AlexNet, we need to apply high
compression rates to shrink the model size far enough to fit the it into the
memory of our target microcontrollers (see Sec. V-C) for ResNet execution time
is the primary bottleneck. Even after pruning almost 99% of ResNet’s
parameters we still measure inference times of around 4 seconds on the Pi
Pico, and of around 6 seconds on the Arduino Nano even when applying
quantization, see the fifth column of Table 8(b). Only after removing $>$99%
of all parameters the execution time ended up in an acceptable range (of
around 1 second) on both microcontroller targets. However, these extremely
high compression rates were accompanied by a high loss in accuracy, see the
first column. This may make the usage of such a complex DNN architecture
impractical for microcontroller targets. Therefore, we conclude that not only
a model’s parameter count but also its topology decides if it is deployable on
a target system from a performance standpoint.

##  VI Discussion of Results

By using our compression and deployment pipeline, we were able to
automatically and feasibly deploy DNNs to microcontrollers. Compressing DNNs
allowed us to achieve significant savings in memory consumption, execution
time, and energy consumption at runtime, without sacrificing model accuracy
for it. For pruning, we achieved the best results and most savings with
structural pruning. When comparing different pruning strategies, our
experiments indicate that structural pruning offers better opportunities for
saving memory and execution time than element-wise pruning. In addition, the
execution of DNNs that were compressed using this technique did not require
any special support for sparse matrix formats as it is required for element
pruning. Furthermore, we observed that for structural pruning, using different
state-of-the-art heuristics did not have that much of an impact. Choosing a
reasonable pruning schedule and allowing for retraining has proven to be more
effective. Additionally, applying weight quantization together with structural
pruning resulted in even more savings not only in memory consumption, but also
in execution time. This is due to the fact that our target systems were able
to process much more efficient in integer- than in floating-point space.
Besides that, we noticed that our different compression and deployment
strategies had almost no influence on the power drawn by both the Pi Pico and
the Arduino during inference. This means that the observed energy savings were
mainly the result of execution times.

We come to the conclusion that a DNN model is deployed optimally on a
microcontroller, if it runs on a system where it fits into the available
memory and draws the least amount of power under load while still being able
to run inferences in a reasonable time frame. Furthermore, we argue that a
DNN’s execution time has to be seen in relation to the frequency at which
input samples are generated by connected sensors.

##  VII Conclusion

In this work we presented a configurable pipeline for compressing DNNs and
deploying them on Cortex-M based microcontrollers. To achieve compression, our
pipeline utilizes network pruning and weight quantization techniques. The
deployment is handled by using a proposed runtime environment, which consists
of a code generator for mapping trained networks to executable C-code and a
runtime library which provides optimized implementations of common DNN layers.
We used the introduced pipeline to compare DNNs compressed with different
pruning and quantization techniques. Furthermore, we tested how compression
influences runtime performance on multiple target systems. We were able to
show that even larger DNN architectures like AlexNet or ResNet can be feasibly
deployed on microcontrollers featuring memory footprints of as little as 1-2
MB Flash and 256 Kb SRAM while still achieving good execution time and
accuracy results.

## Acknowledgement

This work was supported by the Bavarian Ministry for Economic Affairs,
Infrastructure, Transport and Technology through the Center for Analytics-
Data-Applications (ADA- Center) within the framework of “BAYERN DIGITAL II”.

## References

  * [1] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “CNN architectures for large-scale audio classification,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Mar. 2017, pp. 131–135.
  * [2] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional Neural Networks for Speech Recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 22, no. 10, pp. 1533–1545, Oct. 2014.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2016, pp. 770–778.
  * [4] A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” _University of Toronto_ , May 2012.
  * [5] T. Kautz, B. H. Groh, J. Hannink, U. Jensen, H. Strubberg, and B. M. Eskofier, “Activity recognition in beach volleyball using a Deep Convolutional Neural Network,” _Data Mining and Knowledge Discovery_ , vol. 31, no. 6, pp. 1678–1705, Nov. 2017.
  * [6] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” _Proceedings of the IEEE_ , vol. 105, no. 12, pp. 2295–2329, Dec. 2017.
  * [7] A. Canziani, A. Paszke, and E. Culurciello, “An Analysis of Deep Neural Network Models for Practical Applications,” _arXiv:1605.07678 [cs]_ , Apr. 2017.
  * [8] N. D. Lane, S. Bhattacharya, A. Mathur, P. Georgiev, C. Forlivesi, and F. Kawsar, “Squeezing Deep Learning into Mobile and Embedded Devices,” _IEEE Pervasive Computing_ , vol. 16, no. 3, pp. 82–88, 2017\.
  * [9] Y. L. Cun, J. S. Denker, and S. A. Solla, “Optimal Brain Damage,” in _Advances in Neural Information Processing Systems_. Morgan Kaufmann, 1990, pp. 598–605.
  * [10] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2018.
  * [11] M. Nagel, R. A. Amjad, M. V. Baalen, C. Louizos, and T. Blankevoort, “Up or Down? Adaptive Rounding for Post-Training Quantization,” in _International Conference on Machine Learning_. PMLR, Nov. 2020, pp. 7197–7206.
  * [12] C. Liu, J. Li, and Y. Gong, “SVD-based universal DNN modeling for multiple scenarios,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [13] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _International Conference on Learning Representations (ICLR)_ , 2016.
  * [14] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning Filters for Efficient ConvNets,” _arXiv:1608.08710 [cs]_ , Mar. 2017.
  * [15] S. Anwar, K. Hwang, and W. Sung, “Structured pruning of deep convolutional neural networks,” _J. Emerg. Technol. Comput. Syst._ , vol. 13, no. 3, Feb. 2017.
  * [16] R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. V. d. Vorst, _Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd Edition_. Philadelphia, PA: SIAM, 1994.
  * [17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, May 2017.
  * [18] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation Applied to Handwritten Zip Code Recognition,” _Neural Computation_ , vol. 1, no. 4, pp. 541–551, Dec. 1989\.
  * [19] P. Hu, X. Peng, H. Zhu, M. M. S. Aly, and J. Lin, “OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, May 2021.
  * [20] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” in _International Conference on Learning Representations_ , 2019.
  * [21] B. Reddy, Y.-H. Kim, S. Yun, C. Seo, and J. Jang, “Real-Time Driver Drowsiness Detection for Embedded System Using Model Compression of Deep Neural Networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , Jul. 2017, pp. 438–445.
  * [22] S. Lu, G. Qian, Q. He, F. Liu, Y. Liu, and Q. Wang, “In Situ Motor Fault Diagnosis Using Enhanced Convolutional Neural Network in an Embedded System,” _IEEE Sensors Journal_ , vol. 20, no. 15, pp. 8287–8296, Aug. 2020.
  * [23] J. Lin, W.-M. Chen, Y. Lin, j. cohn, C. Gan, and S. Han, “MCUNet: Tiny Deep Learning on IoT Devices,” in _Advances in Neural Information Processing Systems_ , vol. 33. Curran Associates, Inc., 2020, pp. 11 711–11 722.
  * [24] B. Zoph and Q. V. Le, “Neural Architecture Search with Reinforcement Learning,” arXiv, Tech. Rep., Feb. 2017.
  * [25] R. David, J. Duke, A. Jain, V. J. Reddi, N. Jeffries, J. Li, N. Kreeger, I. Nappier, M. Natraj, S. Regev, R. Rhodes, T. Wang, and P. Warden, “TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems,” _arXiv:2010.08678 [cs]_ , Mar. 2021.
  * [26] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, ser. KDD ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 2623–2631.
  * [27] S. J. Hanson and L. Y. Pratt, “Comparing biases for minimal network construction with back-propagation,” in _Proceedings of the 1st International Conference on Neural Information Processing Systems_. Cambridge, MA, USA: MIT Press, Jan. 1988, pp. 177–185.
  * [28] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning Convolutional Neural Networks for Resource Efficient Inference,” _arXiv:1611.06440 [cs]_ , Jun. 2017.
  * [29] S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, p. 1135–1143.
  * [30] H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang, “Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures,” _arXiv:1607.03250 [cs]_ , Jul. 2016.
  * [31] M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” _arXiv:1710.01878 [cs, stat]_ , Nov. 2017\.
  * [32] O. R. developers, “ONNX Runtime,” https://onnxruntime.ai/, 2021, version: 1.8.0.
  * [33] I. Hubara, Y. Nahshan, Y. Hanani, R. Banner, and D. Soudry, “Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming,” _arXiv:2006.10518 [cs, stat]_ , Dec. 2020.
  * [34] Y. LeCun, C. Cortes, and C. Burges, “MNIST handwritten digit database,” _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , vol. 2, 2010.