# MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design
Framework for MCUs

Junfeng Gong1,2, Cheng Liu 1,21, Long Cheng3, Huawei Li1,2, Xiaowei Li1,2 1
Corresponding author.This work is supported by the National Key R&D Program of
China under Grant (2022YFB4500405), and the National Natural Science
Foundation of China under Grant 62174162. 1SKLP, Institute of Computing
Technology, Chinese Academy of Sciences, Beijing, China  2Dept. of Computer
Science, University of Chinese Academy of Sciences, Beijing, China  3School of
Control and Computer Engineering, North China Electric Power University,
Beijing, China

###### Abstract

Mixed-precision neural network (MPNN) that utilizes just enough data width for
the neural network processing is an effective approach to meet the stringent
resources constraints including memory and computing of MCUs. Nevertheless,
there is still a lack of sub-byte and mixed-precision SIMD operations in MCU-
class ISA and the limited computing capability of MCUs remains underutilized,
which further aggravates the computing bound encountered in neural network
processing. As a result, the benefits of MPNNs cannot be fully unleashed. In
this work, we propose to pack multiple low-bitwidth arithmetic operations
within a single instruction multiple data (SIMD) instructions in typical MCUs,
and then develop an efficient convolution operator by exploring both the data
parallelism and computing parallelism in convolution along with the proposed
SIMD packing. Finally, we further leverage Neural Architecture Search (NAS) to
build a HW/SW co-designed MPNN design framework, namely MCU-MixQ. This
framework can optimize both the MPNN quantization and MPNN implementation
efficiency, striking an optimized balance between neural network performance
and accuracy. According to our experiment results, MCU-MixQ achieves
2.1$\times$ and 1.4$\times$ speedup over CMix-NN and MCUNet respectively under
the same resource constraints.

###### Index Terms:

Low-bitwidth Quantization, Mixed-precision Neural Network, SIMD, MCUs

##  I Introduction

The application of Artificial intelligence (AI) has become prevalent in
typical Internet of Things (IoT) scenarios such as health monitoring,
mechanical equipment fault diagnosis, and industrial automation. These
applications commonly rely on microcontrollers (MCUs) known for their ultra-
low power consumption and cost as the central processing units. Yet, AI
especially deep learning models demands significant computational and memory
resources, posing great challenges to its deployment on resource-limited MCUs.
While embedding deep learning accelerators within MCUs is a conceivable
strategy, it substantially escalates chip costs and energy consumption,
thereby constraining its use in IoTs. Consequently, there is considerable
interest in deploying lightweight deep learning models on MCUs to attain
efficient inference and empower the intelligence of things[1, 2, 3].

The two major challenges of deploying deep learning on MCUs are insufficient
computational resources and limited memory capacity. For the memory issue,
initiatives such as MCUNet[4, 5] optimizes the memory scheduling during the
inference from the perspective of the overall model topology. MCUNetV2 [6]
further decomposes deep learning computations into smaller block computations
to reduce the memory requirements. For the computation issue, neural network
processing is known to be computing bound for MCUs and many efforts have been
devoted to alleviate this challenge. Mixed-precision neural network (MPNN)
quantization that can reduce both the computational and memory requirements of
the models is a straightforward yet effective approach[7]. However, MCUs
generally lack support for sub-byte instructions[12] and using primitive
instructions for neural network processing directly would waste the limited
computational resources in MCUs and aggravate the computing bound problem.

Prior works [14, 12] investigated the use of SIMD instructions to accelerate
low-bitwidth convolution operations, but they generally spread the low-
bitwidth operations across the different SIMD lanes and the number of low-
bitwidth operations packed in the SIMD is limited to the number of SIMD lanes.
Essentially, they fail to make full use of the SIMD computing fabric because
each SIMD lane is actually underutilized. In fact, the approach of packing
low-bitwidth operations into high-bitwidth operations has also been explored
for integer instructions of CPUs[8] and primitive DSPs of FPGAs[38, 37, 8,
36], but these approaches cannot be applied directly on SIMD instructions of
MCUs. For instance, the shifting required by packing is almost free on FPGAs,
but it is non-trivial for MCUs and inappropriate packing can even result in
considerable performance penalty.

To achieve efficient packing on MCUs, we propose to pack low-bitwidth
operations with the granularity of the SIMD lanes such that each SIMD lane can
be fully utilized. In addition, SIMD fabric usually enables different lane
configurations, so we can also configure the SIMD lane sizes to suit the
different bitwidth requirements of the convolution and achieve higher SIMD
utilization. Furthermore, we rearrange the packing order of the low-bit
operands to reduce the auxiliary instructions of packing and lower the packing
overhead accordingly.

As the bitwidth of the neural network models affects both the model accuracy
and model implementation efficiency on MCUs, optimizing the model quantization
and model implementation independently will lead to sub optimal results. In
this work, we leverage NAS to take the model quantization and implementation
efficiency of low-bitwidth operators into consideration at the same time and
co-optimize the model accuracy and model performance. As mentioned, packing on
MCUs requires additional shifting operations and the implementation efficiency
of different bitwidth is not proportional to the bitwidth. To this end, we
construct a performance model for neural network operators of low bitwidth
such that the influence of different quantization on MCU implementation
efficiency can be predicted immediately for the co-optimization. Finally, we
have the SIMD-based convolution operator supporting various sub-byte
operations added to TinyEngine[4] such that MPNNs can be deployed on top of
TinyEngine efficiently and benefit the memory optimizations in TinyEngine.

The main contributions of this work can be summarized as follows.

  * •

To enable efficient MPNN processing on MCUs, we propose an efficient low-
bitwidth operation packing algorithm to make full into the SIMD fabric of
MCUs. Specifically, we propose to have multiple low-bitwidth operations packed
in the granularity of SIMD lanes and adapt the SIMD lane sizes to fit the
convolution bitwidth requirements at the same time, achieving significantly
higher packing efficiency compared to prior SIMD packing strategies.

  * •

Considering that the bitwidth of neural network models impacts both model
accuracy and implementation efficiency, we leverage NAS to perform a packing-
aware quantization for MCUs, thus co-optimizing model accuracy and performance
concurrently. This framework is open sourced on GitHub
111https://anonymous.4open.science/r/MCU-MixQ-FCD7.

  * •

With both the low-bitwidth packing and packing-aware NAS, we establish a HW/SW
co-optimization framework, MCU-MixQ, for efficient neural network
implementation on MCUs. According to our experiments on a set of neural
network models, MCU-MixQ achieves $2.1\times$ and $1.5\times$ performance
speedup over CMix-NN and MCUNet on average respectively under the same
resource and accuracy constraints.

##  II Related Work

To achieve efficient deep learning on MCUs with limited computing resources,
prior work have proposed various optimization strategies from distinct angles.
Early work mainly investigated mixed-precision quantization that explores
smaller bitwidth for each different neural network layer to reduce both the
compute and memory requirements. Some approaches concentrate on optimizing
fundamental neural network operators by employing low-bitwidth operation
packing and SIMD optimization, thereby increasing the implementation
efficiency of the major neural network operators. Some of the research focused
on HW/SW co-optimization, simultaneously considering operator implementation
efficiency and model quantization. Additionally, there are also strategies [4]
[6] targeting at memory optimization via scheduling and patching to
accommodate larger deep learning models on MCUs with minimal performance loss.
While the memory optimizations are generally orthogonal to the computing
optimization, we mainly illustrate the computing optimization approaches in
the rest of this section.

###  II-A Mixed-Precision Nerual Network Quantization

Quantization is an established method for model compression, effectively
reducing both computing and memory overhead. Currently, unified-precision
quantization method has achieved remarkable success[26], even achieving
lossless precision at 8-bit precision after fine-tuning[32] or other novel
techniques[28]. Nevertheless, since model layers differ in their sensitivity
to compression, conventional uniform precision quantization methods cannot
realize optimal results. Several studies [18, 19, 22] have introduced various
metrics to assess the sensitivity of different model layers, guiding the
configuration of quantization schemes to strike a balance between accuracy and
quantization bitwidth. It is a generic computing optimization approach for
various computing engines including CPUs[8], FPGAs[8, 27], and GPUs[30, 31],
and can be particularly beneficial to MCUs with rather limited computing
resources and energy budgets[29].

###  II-B Network Operator Optimizations

MPNNs that can reduce both the computing and memory requirements without
compromising the model accuracy fit well with MCUs with limited hardware
resources. Some recent works proposed to develop customized computing fabrics
such as dot-product units and vector processing units to support convolution
with lower bitwidth [13] [24] [25] in MPNNs, but there is still a lack of
native low-bitwidth operator, particularly under 8 bit, in mainstream
commodity MCUs. Implementing low-bitwidth deep learning models with primitive
MCU arithmetic instructions directly leads to underutilization of the limited
computing resources. To address this, CMSIS-NN [14] leverages SIMD
instructions to optimize typical fixed point neural network operations like
int8. CMix-NN [12] investigates the use of SIMD instructions for 2 bit, 4 bit,
and 8 bit convolution kernels for efficient MPNN inference. Hikconv [8]
presents a more general approach to pack arbitrary low-bitwidth convolution
kernels on primitive integer arithmetic instructions in MCUs. The authors in
[21] explored the computing redundancy from inputs to improve the convolution
performance.

###  II-C HW/SW Co-Optimization

Despite of the improved computing and memory efficiency of MPNNs, the
performance deployed on the target computing engines can vary substantially
because of the implementation efficiency variations of the low-bitwidth neural
network operations. To address the problem, hardware-aware quantization
approaches have been explored to suit the different computing fabrics. While
NAS [20] [9] provides a unified design framework to search through the model
design space for multi-objective optimization, it has been widely adopted to
co-optimize the accuracy and performance of MPNNs in prior works [10].
TinyEngine [4] explored the model architecture along with the memory
limitation to ensure effective neural network processing on MUCs.

In summary, recent studies have shown considerable promise for leveraging
neural network redundancies through mixed-precision quantization and
developing low-bitwidth convolution on current MCUs. However, dissociating
quantization from operator optimization—targeting accuracy and performance
separately—may result in suboptimal results. While several HW/SW co-
optimization methods exist, there is still a lack of study on SIMD
optimization for MPNNs, and prior efforts still fail to unleash the computing
potential of MCUs. Innovative approaches that can both fully exploit MCU
computing resources and concurrently compress model redundancies through
quantization are highly demanded to perform neural network processing
efficiently on MCUs.

##  III Overall Design Framework

Figure 1: MCU-MixQ Overview, the proposed HW/SW co-optimization design
framework for MPNN on MCUs.

In this work, we present an MPNN design framework for MCUs, namely MCU-MixQ,
as depicted in Fig. 1. This framework comprises SIMD-based low-bitwidth neural
network operators and a hardware-aware quantization explorer based on NAS. The
low-bitwidth neural network operator has multiple low-bitwidth operations
packed into SIMD fabric, which makes full use of the computing resources in
MCUs to mitigate the computing bottleneck. The hardware-aware quantization
explorer is employed to reduce the data width of neural network models as much
as possible, allowing for reduced computing and memory resource usage while
retaining inference accuracy. Given that the implementation efficiency of low-
bitwidth neural network operators also varies with the bitwidth configurations
and significantly impacts network performance, the quantization explorer must
be aware of the operator’s implementation efficiency. This is achieved by
incorporating a performance loss component alongside the standard accuracy
loss component, as illustrated in Fig. 1.

###  III-A Low-bitwidth Network Operators

To fully harness the computational resources, particularly SIMD, in MCUs for
the acceleration of neural network processing, we seek to pack multiple low-
bitwidth operations in a single SIMD fabric. Unlike prior SIMD packing [34,
33, 8] that fits each low-bitwidth operation to an independent SIMD lane, we
propose to conduct the packing in each SIMD lane such that the packing is not
limited to the minimum lane size i.e. 8-bit. Notably, we can adjust the SIMD
lane sizes to the bitwidth requirements of the convolution and make best use
of the SIMD fabric in MCUS. In addition, the packing typically requires
shifting operations which also takes non-trivial overhead and affects the
resulting performance of the network operators. To alleviate the packing
overhead, we further propose a data reordering mechanism to reduce the number
of auxiliary instructions required by the packing. The overall SIMD-based low-
bitwidth convolution packing algorithm, namely SLBC, will produce an optimized
sub-byte convolution operator, which will be utilized to sustain the execution
of MPNNs. SLBC will be detailed in Section IV.

###  III-B Hardware-Aware MPNN Quantization

Motivated by prior NAS-based quantization works [10, 15, 16, 17, 23], we
leverage a differentiable NAS to achieve hardware-aware quantization and co-
optimize the model accuracy and performance. It starts with a pre-trained
floating point model and sets the possible quantization data width as the
initial design space of NAS. Then, it creates a quantization super-net to
cover all the possible quantization configurations. Each layer of the target
model to be quantized will be replaced with a mixed kernel composed of
multiple weighted branches and each branch represents a specific quantization
option. Given the quantization search space $Q=\\{q_{1},q_{2},...,q_{n}\\}$,
the quantization super-net can be denoted as $f(Q)$, while a sub-net sampled
from $Q$ is $f(q_{i})$. The optimization goal is to search for a quantization
sub-net $q^{*}$ to maximize the accuracy and minimize the latency while
fulfilling the design constraints such as model sizes.

With the super-net architecture, we can start the super-net training and have
two loss components included to take both the model accuracy and model
performance of different quantization setups into consideration in training as
shown in Eq. 2 and Eq. 1. Particularly, the performance loss component mainly
characterizes the network performance when deployed on MCUs with the proposed
SLBC packing approach. Since it is expensive to deploy the network with
various quantization configurations on MCUs and extract the performance with
realistic deployment, we have a simplified yet precise performance model for
the NAS. The model is closely coupled with the SLBC packing and it will be
illustrated in Section IV as well. After the quantization optimization, MCU-
MixQ performs quantization aware training (QAT) on the selected mixed-
precision model and the model will be deployed on MCUs, completing the entire
workflow.

$Loss(\alpha_{w},\alpha_{a})=\sum_{l=1}^{L}{C^{l}}$ (1)
$Loss(\alpha_{w},\alpha_{a})=Loss_{acc}(\alpha_{w},\alpha_{a})+Loss_{comp}(\alpha_{w},\alpha_{a})$
(2)

Finally, we deploy the obtained MPNN on MCUs with TinyEngine [4] which is an
memory-efficient inference framework designed for MCUs. It provides all the
major functionalities required to deploy a high-level model on MCUs.
Particularly, it optimizes the memory usage of the model during the process of
code generation and manages the memory scheduling to ensure on-demand
parameter loading. These techniques prevent the out of memory issues during
inference while minimizing its influence on the performance. While TinyEngine
does not support sub-byte convolution operators, we have SLBC integrated to
enable the deployment of MPNNs on TinyEngine. The revised TinyEngine in
combination with the proposed quantization explorer constitutes the
comprehensive HW/SW co-designed MPNN framework for MCUs known as MCU-MixQ.

##  IV Low-bitwidth Network Operator Optimizations

As a compute-intensive operator, convolution is heavily bounded by the
computing capability of MCUs due to the lack of massively parallel computing
fabrics. Therefore, optimizing the convolution, which is the major kernel of
neural networks, is critical to the neural network performance on MCUs.

###  IV-A SIMD-based Low-bitwidth Convolution

Considering the mathematical equivalence of polynomial multiplication and
convolution operation, for an $s_{b}$-bit sequence $s$ and a $k_{b}$-bit
convolution kernel $k$, we can pack multiple low-bitwidth elements of $s$ and
$k$ into one wider hardware unit $R_{1}$ and $R_{2}$ which can be represented
with the following polynomial forms.

$\displaystyle R_{1}$ $\displaystyle=\sum_{i=0}^{N_{s}-1}{s[i]\cdot
2^{iS_{b}}}$ (3) $\displaystyle R_{2}$
$\displaystyle=\sum_{j=0}^{N_{k}-1}{k[j]\cdot 2^{jS_{b}}}$ (4)

With the packing, the product $P$ of a high-precision multiplier can be
simplified to Equation 5 according to the rule of polynomial multiplication.

$\displaystyle P$ $\displaystyle=R_{1}\times R_{2}$
$\displaystyle=(\sum_{i=0}^{N_{s}-1}{s[i]\cdot
2^{iS_{b}}})\cdot(\sum_{j=0}^{N_{k}-1}{k[j]\cdot 2^{jS_{b}}})$ (5)
$\displaystyle=\sum_{k=0}^{N_{s}+N_{k}-2}{(\sum_{i+j=k}{s[i]\cdot k[j]\cdot
2^{kS_{b}}})}$

According to the definition of convolution, the application of the
$N_{k}$-kernel $k$ to a $N_{s}$-element sequence $s$ also yields
$N_{s}+N_{k}-1$ elements and the $n$th element of convolution sequence $y$ can
be represented as:

$\displaystyle y[n]$ $\displaystyle=\sum_{m=0}^{N_{k}-1}{s[n-m]\cdot k[m]}$
(6) $\displaystyle=\sum_{i+j=n}{s[i]\cdot k[j]}$

According to Eq. 5 and Eq. 6, it is evident that the multiplication product
$P$ is composed of convolution sequence $y$, each of which has been left-
shifted by the corresponding number of bits with Eq. 7. As a result, each
element of convolution sequence can be segmented from $P$ through bit
operations. By utilizing a single multiplication instruction along with
multiple bit-wise instructions for packing and segmentation, the overhead of
the convolution can be significantly reduced compared to naive implementation.

$\displaystyle P$ $\displaystyle=\sum_{k=0}^{N_{s}+N_{k}-2}{y[k]\cdot
2^{k_{b}}}$ (7)

The above analysis assumes that the computation unit is long enough to
accommodate all elements from the sequence and kernel. In practice, most
commodity MCUs like Cortex-M incorporate SIMD instructions to enhance parallel
computing capabilities. Thus, we investigate the use of the SIMD instructions
for efficient low-bitwidth convolution and propose SLBC, a SIMD low-bitwidth
convolution optimized for MCUs. The detailed execution flow of SLBC is
presented in Algorithm 1. With SLBC, multiple multiply and add operations in a
convolution operator can be substituted with a single SIMD multiplication
instruction and bit-wise operations. SLBC mainly consists of three processing
stages including packing, SIMD multiplication, and SIMD segmentation as
illustrated in Fig. 2.

Algorithm 1 Naïve SLBC

for $i=0$ to $K$ by $1$ do $\triangleright$ kernel packing

$B\mathrel{|}=k[i]\ll(i\times S)$

end for

$mask=(1\ll S)-1$

$vmask=vdupq\\_n(mask)$ $\triangleright$ vector mask

for $i=0$ to $N_{f}$ by $N\times 2$ do $\triangleright$ elements packing

for $i=0$ to $G$ by $1$ do

for $k=0$ to $N$ by $1$ do

$A[j]\mathrel{|}=A[i+j\times N+k]\ll(k\times S)$

end for

end for

$VA=vld(A)$

$VP=vmul(VA,VB)$ $\triangleright$ SIMD multiplication

for $j=0$ to $N+K-1$ by $1$ do $\triangleright$ extract

$shift\\_vp=vshr(VP,vmask)$

$vresult=vand(shift\\_vp,vmask)$

$scalar0=vget(vresult,0)$

$scalar1=vget(vresult,1)$

$output[i+0\times N+j]\mathrel{+}=scalar0$

$output[i+1\times N+j]\mathrel{+}=scalar1$

end for

end for

Figure 2: SIMD Low-bitwidth Convolution (SLBC) Overview

In packing stage, multiple convolution elements can be packed into a wider
SIMD register file with SIMD shift instruction and vector mask operation.
Suppose each SIMD register has $N_{l}$ lanes and each lane is able to pack
$N_{s}$ sequence elements and $N_{k}$ kernel elements. When $N_{k}$ is equal
to the kernel size $k$, the entire kernel can be packed into a single SIMD
lane. According to the packing strategy defined previously, the packed SIMD
register $vs$ and $vk$ can be denoted as Eq. 8 and Eq. 9 respectively.

$\displaystyle vs$
$\displaystyle=\sum_{l=0}^{N_{l}-1}{(2^{lL_{b}}\cdot(\sum_{i=0}^{N_{s}-1}{s[lN_{s}+i]\cdot
2^{iS_{b}}}))}$ (8) $\displaystyle vk$
$\displaystyle=\sum_{l=0}^{N_{l}-1}{(2^{lL_{b}}\cdot(\sum_{i=0}^{N_{k}-1}{k[i]\cdot
2^{iS_{b}}}))}$ (9)

In SIMD multiplication stage, the packed data $vs$ and $vk$ multiply with an
SIMD instruction and the product is presented in Eq. 10. Note that $\otimes$
denotes the SIMD multiplication and $N_{l}$ denotes the total number of SIMD
lanes. After the SIMD multiplication, the convolution sequence is already
stored in the output vector, which means that we can replace more ADD and MUL
i.e. single instruction single data (SISD) instructions with one SIMD
instruction.

$\displaystyle vp$ $\displaystyle=vs\otimes vk$ (10)
$\displaystyle=\sum_{l=0}^{N_{l}-1}{(2^{lL_{b}}\cdot(\sum_{i=0}^{N_{s}-1}{s[lN_{s}+i]\cdot
2^{iG_{b}}}))}$
$\displaystyle\otimes\sum_{l=0}^{N_{l}-1}{(2^{lL_{b}}\cdot(\sum_{i=0}^{N_{k}-1}{k[i]\cdot
2^{iG_{b}}}))}$
$\displaystyle=\sum_{l=0}^{N_{l}-1}{(2^{lL_{b}}\cdot\sum_{k=0}^{N_{s}+N_{k}-2}{(\sum_{i+j=k}{(vs[lN_{s}+i]\cdot
vk[j]\cdot 2^{kG_{b}}}))})}$

In segmentation stage, we notice that SLBC can be viewed as multiple parallel
packing tasks. As shown in Eq. 11, the last element in a lane will be combined
with the first element of the next lane to form an element of the convolution
sequence, while the other data in each SIMD lane also become elements of the
convolution sequence. Note that $vp$ represents a 2-D array while the first
dimension represents the lane index and the second index represents the
element position in each lane. Finally, SIMD bit-wise operations is utilized
to extract the convolution sequence from the output vector.

$y[i]=\left\\{\begin{aligned} &vp[l][N_{s}+N_{k}-2]+vp[l+1][0],i\neq 0,i\neq
N-1,k=0\\\ &vp[l][k],others\end{aligned}\right.$ (11)

subject to

$\left\\{\begin{aligned} l&=i/(N_{s}+N_{k}-2)\\\ k&=i\%(N_{s}+N_{k}-2)\\\
N&=lN_{s}+N_{k}-1\end{aligned}\right.$

###  IV-B Enhance Locality Through Reordering

Despite the packing efficiency, SLBC requires extra bit operations such as LSR
to extract convolution elements from the output vector and the overhead of
these bit operations is non-trivial. Inspired by ULPPACK [11] that utilizes
local accumulation to combine multiple bit operations together and reduces
segmentation overhead substantially, we propose a new reordering algorithm for
SLBC to improve the register reuse during packing, as shown in Theorem IV.1.

######  Theorem IV.1.

For SIMD registers with $L$ lanes, each lane can accommodate $N$ low-bitwidth
elements, a group of $N\times L^{2}$ elements will be reordered and packed
within $L$ SIMD registers. For the $y$th lane of the $x$th SIMD register, it
will be packed into the $y$th position of the $x$th one.

To illustrate the reordering algorithm, we have two simplified packing
examples presented in Fig. 3 and Fig. 4. Suppose each SIMD register has 2
lanes and each lane can pack 2 elements. Assume that the kernel size is also 2
so that the entire kernel can be fully packed into one SIMD lane. Fig. 3 shows
the processing of the naive packing proposed in Algorithm 2. Since the entire
kernel can be packed into one lane and convolution sequence needs to be
segmented, two different segmentations of sequence will be packed into one
SIMD register and meanwhile the entire kernel will be packed into each lane of
the same SIMD register to perform an optimized convolution through SIMD
multiplication. But according to the details of SLBC, the overlapping part are
distributed in adjacent lanes within the same SIMD register. According to the
principle of SIMD, the overlapping part can not be utilized through shift
operation. As a result, the overlapping part needs to be segmented separately
from adjacent lanes, thus leading to unnecessary overhead in bit-wise
operation instructions.

In order to fully utilize the overlapping portions and merge multiple
segmentation operations together, the arrangement order of elements has been
modified so that the overlapping portions appear in adjacent SIMD registers
rather than between adjacent lanes within the same one. The specific method of
reordered packing SLBC is shown in Fig. 4. It can be observed that since the
order of data rearrangement has been changed, there exists overlap between the
result of adjacent iterations. For SIMD registers with $L$ lanes that store
the product results, which represent the convolution sequence after packing,
the element contained in the boundary lane cannot form one complete element of
the convolution independently. Instead, it needs to be added to the element
held in the first lane in the adjacent SIMD register to become an real
element. In other words, the elements located in boundary position require an
additional segmentation operation. Fig. 4 illustrates the packing positions of
these two elements in SIMD registers. However, After rearranging the packing
order of elements, the boundary elements to jointly form one complete
convolution element are located in corresponding lanes of adjacent SIMD
registers. Therefore, these two SIMD registers can be accumulated after
performing parallel shifting operations, which eliminates the need for
additional splitting overhead. For the configuration discussed above, $L$
segmentation operations will be eliminated for every $N\times L\times L$
elements, thus reducing segmentation overhead to $\frac{1}{N\times L}$ of the
original count.

Figure 3: Output overlapping in naïve SLBC

Figure 4: Output Overlapping in Reordered SLDC Algorithm 2 SLBC with reordered
packing

for $i=0$ to $N_{f}$ by $N\times G\times G$ do

$local=vdup(0)$ $\triangleright$ local accumulator

for $j=0$ to $G$ by $1$ do $\triangleright$ reordered packing

for $k=0$ to $K$ by $1$ do

for $l=0$ to $N$ by $1$ do

$A[k]=f[i+j\times N+4\times k+l]\ll(l\times S)$

end for

end for

$VA=vld(A)$

$VP=vmul(VA,VB)$

$vst(local,VP)$

$local=vadd(local,VP)$

for $k=0$ to $N$ by $1$ do

$shift_{v}p=vshr(local,k\times S)$

$vresult=vand(shift_{v}p,vmask)$

$scalar0=vget(vresult,0)$

$scalar1=vget(vresult,1)$

$output[i+j\times N+k]\mathrel{+}=scalar0$

$output[i+j\times N+G\times N+k]\mathrel{+}=scalar1$

end for

$local=vshr(local,N\times S)$

end for

for $j=0$ to $K-1$ by $1$ do

$scalar0=vget(local,0)$

$scalar1=vget(local,1)$

$output[i+G\times N+j]\mathrel{+}=scalar0$

$output[i+2\times G\times N+j]\mathrel{+}=scalar1$

end for

end for

The complete algorithm for reordered SLBC is illustrated in Algorithm.2.
During packing stage, $N\times L\times L$ elements will be considered as a
group to be packed into $N$ SIMD registers. After the completion of the
multiplication operation, the convolution elements squeezed in the register
will not be segmented immediately. Instead, it will be right-shifted and added
to the local accumulation after each round of multiplication. After the group
of multiplications is completed, the actual elements will be segmented from
the local accumulation. In the subsequent experimental phase, we conducted an
ablation study on reordered SLBC and SLBC within an end-to-end framework to
validate the effectiveness of the improved method in reducing segmentation
overhead.

###  IV-C Adaptive SIMD Packing

SLBC has low-bitwidth operations packed into each SIMD lane independently, but
the packing efficiency depends on both the SIMD lane size and the operation
bitwidth to be packed. Since SIMD usually allows different lane sizes, we can
adjust the SIMD lane size to fit the convolution bitwidth for higher SIMD
utilization. For each convolution of an MPNN, we adaptively decide the
optimized packing and SIMD lane sizes at compilation time to ensures optimized
MPNN performance.

###  IV-D Packing Performance Prediction

The proposed HW/SW co-design framework MUC-MixQ requires a large number of
performance evaluation of MPNNs with different quantization setups which can
be too expensive for evaluation with realistic deployment, so we further build
a performance model for this purpose. As mentioned, the low-bitwidth
convolution implemented with SLBC includes both SIMD MUL instruction and
bitwise operations. Considering the varied execution time of the different
types of instructions, we use SISD instructions as the calibration metric and
align SIMD MUL instruction and SISD bit operations with it. Specifically, as
shown in Eq. 12, the complexity of SISD instructions $C_{SISD}$ is roughly
proportional to the number of SISD accumulation and multiplication operations
where $\alpha$ and $\beta$ refers to the proportion coefficients and they can
be obtained with experiments.

$C=C_{SISD}+\alpha C_{SIMD}+\beta C_{bit}$ (12)

##  V Experiment

To showcase the outstanding performance of MCU-MixQ on MCUs, we conducted
experiments on two datasets: Visual Wake Word (VWW) and CIFAR-10. VWW is a
vision-oriented dataset specifically designed to determine the presence or
absence of a person in an image. CIFAR-10 is a widely adopted benchmark for
image classification tasks. For the hardware platform, we selected ARM
Cortex-M7 microcontroller STM32F746, which is equipped with 320kB of SRAM and
1MB of Flash memory. All the latency measurement is obtained at a clock
frequency of 216MHz.

###  V-A End-to-End Performance Evaluation

We have the neural network benchmark implemented on the target hardware
platform with CMix-NN[12], WPC&DDD[35], TinyEngine[4] and the proposed MCU-
MixQ respectively. Note that CMix-NN and WPC&DDD only supports three different
bitwidth setups i.e. 2bit, 4bit, and 8bit, TinyEngine only supports 8bit while
MCU-MixQ supports all the bitwidth between 2bit and 8bit. Given the same model
accuracy constraint, we compared the end-to-end performance of the resulting
neural network models and the comparison is summarized in Table I. It can be
observed that MCU-MixQ achieves the best performance and outperforms all the
other solutions. This can be attributed to multi-folded reasons including the
more efficient low-bitwidth convolution optimization and more flexible
quantization, as well as the HW/SW co-optimization. They will be analyzed in
detail in the rest of the experiments. Moreover, MCU-MixQ takes advantage of
the memory optimization provided by TinyEngine, so the peak memory usage is
also reduced. On the other hand, we notice that CMix-NN and WPC&DDD with more
flexible quantization setups show even lower performance than TinyEngine with
fixed int8 quantization. This is mainly attributed to other optimization
techniques introduced by TinyEngine, model-adaptive memory scheduling and
computation kernel specialization for example.

TABLE I: End-to-end performance comparison with previous frameworks Backbone | method | Quantization | Peak Memory | Flash Memory | Clocks | Latency | Accuracy
---|---|---|---|---|---|---|---
VGG-Tiny | CMix-NN[12] | Mixed(2,4,8) | 146.33KB | 146.33KB | 5680854 | 26.3ms | 71.4%
WPC& DDD[35] | Mixed(2,4,8) | 228.59KB | 146.33KB | 5140887 | 23.8ms | 70.2%
TinyEngine[4] | 8-bit | 51.93KB | 584.65KB | 3715233 | 17.2ms | 78.1%
MCU-MixQ | Mixed(2-8) | 49.92KB | 591.12KB | 2721615 | 12.6ms | 78.3%
MobileNet-Tiny | CMix-NN[12] | Mixed(2,4,8) | 184.20KB | 184.20KB | 9136773 | 42.3ms | 76.2%
WPC& DDD[35] | Mixed(2,4,8) | 272.38KB | 184.20KB | 7819221 | 36.2ms | 74.3%
TinyEngine[4] | 8-bit | 68.37KB | 704.15KB | 5983279 | 27.7ms | 80.1%
MCU-MixQ | Mixed(2-8) | 62.14KB | 687.36KB | 4600823 | 19.5ms | 79.7%

###  V-B SLBC Efficiency Evaluation

First of all, we compare SLBC with other convolution kernels. In order to
showcase its efficacy on low-bit convolutions, we compare SLBC with naive
convolution, SIMD convolution and CMix-NN. SIMD convolution uses SIMD
instructions to accelerate convolution without other optimization method. Due
to the lack of support for sub-byte in naive and SIMD convolution, the latency
of the convolution operator does not change when executing with different
bitwidths under 8 bits. Fig 5 illustrates the speedups of SLBC under different
bitwidths over the two methods. According to the experimental results, SLBC
achieves an average speedup of $4\times$ and $2\times$ over naive and SIMD
convolution seperately.

Figure 5: Speedup over naive and SIMD convolution

CMix-NN is a flexible mixed-precision inference library, which supports any
combination of 2, 4, 8 bitwidth. It compresses low-bitwidth data for storage
and simultaneously constructs vector instructions using masks in convolution.
In order to demonstrate the superiority of SLBC over CMix-NN in terms of
hardware resource utilization, we compared the theoretical throughput of the
two methods. More specifically, Fig 6 presents the acceleration ratios for
different bitwidth combinations, which represents the equivalent ratio of
operations performed by the one SIMD instruction. According to Fig 6, SLBC can
achieve up to $1.5\times$ speedup over CMix-NN in most quantization
combination.

Figure 6: Speedups over CMix-NN

To demonstrate the effectiveness of our reordered packing SLBC(RP-SLBC), we
conducted ablation experiments on the new method. More specifically, we
integrate SLBC and RP-SLBC into our end-to-end deep learning framework
respectively. For each convolution kernel, we calculate their complexity
according to Algorithm 1 and Algorithm 2 separately, and evaluate the model on
MCU platform. Based on the results in Figure. 7 and theory analysis in
Algorithm 1, RP-SLBC can reach up to nearly $1.1\times$ through reordered
packing.

Figure 7: Latency comparison between SLBC and RP-SLBC

###  V-C HW/SW Co-optimization Evaluation

In order to evaluate our hardware-aware quantization explorer, we choose
EdMIPs as baseline, and utilize them to perform a search for optimal model
quantization configurations. EdMIPs estimates complexity by using MACs as a
proxy approximately. In contrast, according to Eq.12, our hardware-aware
quantization explorer categorizes various operations within the operators, and
adapt them with adjusting parameters. Quantization configurations searched by
EdMIPs and our quantization explorer are illustrated in Fig. 8. Compared to
the quantization configuration searched by EdMIPs, our approach allows for
quantizing to lower average bitwidths for both weights and activations under
the same model architecture. Under the respective given quantization biwidth
configurations, our model can reach up to 78.3% Top-1 accuracy, which is +2.3%
up to EdMIPs, reflecting the effectiveness of our performance prediction model
which can accurately directs NAS to perform hardware-aware quantization.

Figure 8: Quantization configuration searched by EdMIPs and SIMD-aware
quantization explorer respectively

##  VI Conclusion

In this work, we present MCU-MixQ, a HW/SW co-optimized MPNN framework
designed for MCU, which improves inference speed while meeting stringent
hardware resources. We enhance the parallelism of the low-bitwidth convolution
operator through packing and SIMD instructions, and meanwhile implement a low-
bitwidth convolution library designed for MCU. As for model quantization, we
employ differentiable NAS to automatically configure the optimal combination
of quantization bit-widths for the model, while simultaneously considering the
runtime efficiency of the model running on MCU. After quantization search
stage, MPNN will undergo quantization-aware training and be ultimately mapped
onto the optimized MCU kernels. Our experimental results demonstrate that MCU-
MixQ achieves better performance compared to sota TinyEngine framework.

## References

  * [1] C. Banbury, C. Zhou, I. Fedorov, R. M. Navarro, U. Thakker, D. Gope, V. J. Reddi, M. Mattina, and P. N. Whatmough, “Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers,” 2021.
  * [2] A. Burrello, A. Garofalo, N. Bruschi, G. Tagliavini, D. Rossi, and F. Conti, “Dory: Automatic end-to-end deployment of real-world dnns on low-cost iot mcus,” _IEEE Transactions on Computers_ , pp. 1–1, 2021.
  * [3] F. Svoboda, J. Fernandez-Marques, E. Liberis, and N. D. Lane, “Deep learning on microcontrollers: a study on deployment costs and challenges,” in _Proceedings of the 2nd European Workshop on Machine Learning and Systems_ , ser. EuroMLSys ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 54–63.
  * [4] J. Lin, W.-M. Chen, Y. Lin, C. Gan, and S. Han, “Mcunet: Tiny deep learning on iot devices,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [5] E. Liberis and N. D. Lane, “Pex: Memory-efficient microcontroller deep learning through partial execution,” 2023.
  * [6] J. Lin, W.-M. Chen, H. Cai, C. Gan, and S. Han, “Mcunetv2: Memory-efficient patch-based inference for tiny deep learning,” in _Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [7] M. Rakka, M. E. Fouda, P. Khargonekar, and F. Kurdahi, “A review of state-of-the-art mixed-precision neural network frameworks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–20, 2024.
  * [8] X. Liu, Y. Chen, P. Ganesh, J. Pan, J. Xiong, and D. Chen, “Hikonv: High throughput quantized convolution with novel bit-wise management and computation,” in _2022 27th Asia and South Pacific Design Automation Conference (ASP-DAC)_. IEEE Press, 2022, p. 140–146.
  * [9] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” 2019.
  * [10] Z. Cai and N. Vasconcelos, “Rethinking differentiable search for mixed-precision neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2349–2358.
  * [11] J. Won, J. Si, S. Son, T. J. Ham, and J. W. Lee, “Ulppack: Fast sub-8-bit matrix multiply on commodity simd hardware,” in _Proceedings of Machine Learning and Systems_ , D. Marculescu, Y. Chi, and C. Wu, Eds., vol. 4, 2022, pp. 52–63.
  * [12] A. Capotondi, M. Rusci, M. Fariselli, and L. Benini, “Cmix-nn: Mixed low-precision cnn library for memory-constrained edge devices,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 67, no. 5, pp. 871–875, 2020.
  * [13] D. Metz, V. Kumar, and M. Själander, “Bisdu: A bit-serial dot-product unit for microcontrollers,” _ACM Trans. Embed. Comput. Syst._ , vol. 22, no. 5, sep 2023. [Online]. Available: https://doi.org/10.1145/3608447
  * [14] L. Lai, N. Suda, and V. Chandra, “Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus,” 2018.
  * [15] L. Li, C. Yan, and Y. Chen, “Differentiable search for finding optimal quantization strategy,” 2024.
  * [16] B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer, “Mixed precision quantization of convnets via differentiable neural architecture search,” 2019. [Online]. Available: https://openreview.net/forum?id=BJGVX3CqYm
  * [17] Y. Li, C. Hao, X. Zhang, X. Liu, Y. Chen, J. Xiong, W.-m. Hwu, and D. Chen, “Edd: efficient differentiable dnn architecture and implementation co-search for embedded ai solutions,” in _Proceedings of the 57th ACM/EDAC/IEEE Design Automation Conference_ , ser. DAC ’20. IEEE Press, 2020.
  * [18] Y. Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer, “Zeroq: A novel zero shot quantization framework,” 2020.
  * [19] Z. Dong, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer, “Hawq: Hessian aware quantization of neural networks with mixed-precision,” 2019.
  * [20] C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey, and F. Hutter, “Neural architecture search: Insights from 1000 papers,” 2023.
  * [21] J. Liu, F. Zhang, J. Guan, H.-H. Sung, X. Guo, X. Du, and X. Shen, “Space-efficient trec for enabling deep learning on microcontrollers,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_ , ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 644–659. [Online]. Available: https://doi.org/10.1145/3582016.3582062
  * [22] N. P. Pandey, M. Nagel, M. van Baalen, Y. Huang, C. Patel, and T. Blankevoort, “A practical mixed precision algorithm for post-training quantization,” 2023.
  * [23] I. Koryakovskiy, A. Yakovleva, V. Buchnev, T. Isaev, and G. Odinokikh, “One-shot model for mixed-precision quantization,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, jun 2023, pp. 7939–7949. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.00767
  * [24] Angelo Garofalo, Giuseppe Tagliavini, Francesco Conti, Davide Rossi, and Luca Benini. “XpulpNN: Accelerating quantized neural networks on RISC-V processors through ISA extensions,” in _2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, IEEE, 2020, pp. 186–191. [Online]. Available: https://doi.org/10.23919/DATE48585.2020.9116529
  * [25] Angelo Garofalo, Gianmarco Ottavi, Alfio Di Mauro, Francesco Conti, Giuseppe Tagliavini, Luca Benini, and Davide Rossi. “A 1.15 TOPS/W, 16-cores parallel ultra-low power cluster with 2b-to-32b fully flexible bit-precision and vector lockstep execution mode,” in _ESSCIRC 2021-IEEE 47th European Solid State Circuits Conference (ESSCIRC)_ , IEEE, 2021, pp. 267–270.
  * [26] Q. Jin, J. Ren, R. Zhuang, S. Hanumante, Z. Li, Z. Chen, Y. Wang, K. Yang, and S. Tulyakov, “F8net: Fixed-point 8-bit only multiplication for network quantization,” 2022.
  * [27] J. Chen, L. Liu, Y. Liu, and X. Zeng, “A learning framework for n-bit quantized neural networks toward fpgas,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 3, p. 1067–1081, Mar. 2021. [Online]. Available: http://dx.doi.org/10.1109/TNNLS.2020.2980041
  * [28] S. Bai, J. Chen, X. Shen, Y. Qian, and Y. Liu, “Unified data-free compression: Pruning and quantization without fine-tuning,” in _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_. Los Alamitos, CA, USA: IEEE Computer Society, oct 2023, pp. 5853–5862.
  * [29] P.-E. Novac, G. Boukli Hacene, A. Pegatoquet, B. Miramond, and V. Gripon, “Quantization and deployment of deep neural networks on microcontrollers,” _Sensors_ , vol. 21, no. 9, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/9/2984
  * [30] A. Kuzmin, M. Van Baalen, Y. Ren, M. Nagel, J. Peters, and T. Blankevoort, “Fp8 quantization: The power of the exponent,” _arXiv preprint arXiv:2208.09225_ , 2022.
  * [31] P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu, “Fp8 formats for deep learning,” 2022.
  * [32] B. Li, K. Huang, S. Chen, D. Xiong, H. Jiang, and L. Claesen, “Dfqf: Data free quantization-aware fine-tuning,” in _Proceedings of The 12th Asian Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, S. J. Pan and M. Sugiyama, Eds., vol. 129. PMLR, 18–20 Nov 2020, pp. 289–304. [Online]. Available: https://proceedings.mlr.press/v129/li20a.html
  * [33] M. Cowan, T. Moreau, T. Chen, J. Bornholt, and L. Ceze, “Automatic generation of high-performance quantized machine learning kernels,” in _Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization_ , ser. CGO 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 305–316. [Online]. Available: https://doi.org/10.1145/3368826.3377912
  * [34] Y. Umuroglu and M. Jahre, “Work-in-progress: towards efficient quantized neural network inference on mobile devices,” in _2017 International Conference on Compilers, Architectures and Synthesis For Embedded Systems (CASES)_ , 2017, pp. 1–2.
  * [35] A. Mujtaba, W.-K. Lee, and S. O. Hwang, “Low latency implementations of cnn for resource-constrained iot devices,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 69, no. 12, pp. 5124–5128, 2022.
  * [36] J. Sommer, M. Ozkan, O. Keszocze, and J. Teich, “Dsp-packing: Squeezing low-precision arithmetic into fpga dsp blocks,” in _2022 32nd International Conference on Field-Programmable Logic and Applications (FPL)_. Los Alamitos, CA, USA: IEEE Computer Society, sep 2022, pp. 160–166. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/FPL57034.2022.00035
  * [37] Xilinx, “Convolutional neural network with int4 optimization on xilinx devices,” 2020. [Online]. Available: https://docs.xilinx.com/v/u/en-US/wp521-4bit-optimization
  * [38] Xilinx, “Deep learning with int8 optimization on xilinx devices,” 2017. [Online]. Available: https://docs.xilinx.com/v/u/en-US/wp486-deep-learning-int8