# On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M
Microcontrollers

Mark Deutel1, Frank Hannig2, Christopher Mutschler2, and Jürgen Teich1 1
Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)2 Fraunhofer IIS,
Fraunhofer Institute for Integrated Circuits IIS

###### Abstract

On-device training of DNNs allows models to adapt and fine-tune to newly
collected data or changing domains while deployed on microcontroller units
(MCUs). However, DNN training is a resource-intensive task, making the
implementation and execution of DNN training algorithms on MCUs challenging
due to low processor speeds, constrained throughput, limited floating-point
support, and memory constraints. In this work, we explore on-device training
of DNNs for Cortex-M MCUs. We present a method that enables efficient training
of DNNs completely in place on the MCU using fully quantized training (FQT)
and dynamic partial gradient updates. We demonstrate the feasibility of our
approach on multiple vision and time-series datasets and provide insights into
the tradeoff between training accuracy, memory overhead, energy, and latency
on real hardware.

###### Index Terms:

On-Device Training, TinyML, Embedded Machine Learning, Microcontrollers

##  I Introduction

Adapting, fine-tuning, or retraining deep neural networks (DNNs) on
microcontroller units (MCUs) to improve alongside a changing input domain or
application is not a straightforward task [1]. Typically, once the DNN is
deployed on the MCU, its trainable weights are stored read-only in Flash, and
adapting the DNN without reprogramming the MCU is often not feasible. In
addition, the limited memory and computational resources on the MCU require
that the weights and feature maps of a DNN are often stored in quantized form,
which further complicates regular DNN training normally performed in floating-
point. Moreover, DNN training itself adds significant memory and computational
overhead, making on-device training on MCUs for DNNs of meaningfull size and
complexity even more challenging.

However, training DNNs on MCUs has a number of highly desirable properties,
such as (a) a reduced communication of the MCU with remote training servers,
since no data recorded by the MCU needs to be sent away for offline training,
(b) increased privacy, since training with data from the input domain, which
may contain personally identifiable information such as speech or likeness,
can be performed directly on the MCU without any persistent storage of data on
a remote server, (c) energy efficiency, since MCUs have an energy footprint
several orders of magnitude smaller than any GPU-based server, and (d) in
place training with zero downtime, since DNN training can be performed on the
fly alongside the regular inference of recorded samples on the MCU.

###  I-A Challenges

Supervised training of DNNs relies on two algorithms: (a) backpropagation
(BP), which is used to compute local gradients for each intermediate layer of
the DNN based on the chain rule, and (b) stochastic gradient descent (SGD),
which is used to update the trainable weights of each layer based on the local
gradients calculated by BP. Usually, the gradients used by SGD are
stochastically approximated by accumulating them over a small subset of
samples (minibatches) of all available training samples. The combination of BP
and SGD is well-known, versatile, and the backbone of all major AI advances of
recent years. However, it is also extremely memory- and compute-intense,
resulting in DNN training usually being performed on larger server systems and
with computing accelerator cards like GPUs. This results in major challenges
that need to be addressed to enable DNN training on regular MCUs.

The high computational cost of DNN training lies in the ”backward pass”, i.e.,
the BP and SGD algorithms, that must be executed for each training sample in
addition to the ”forward pass”, i.e., the inference. The backward pass is
responsible for propagating the error computed for the last layer of the DNN
by a loss function to calculate local errors for all intermediate layers. As a
result, for each layer, at least one partial derivative with respect to its
input during the forward pass must be computed, while for all trainable layers
another partial derivative with respect to their weights must also be computed
to facilitate the parameter updates performed by SGD. For example, for linear
layers and when using minibatches, this results in two additional transposed
matrix multiplications, while for convolutional layers, two additional
”transposed convolutions” have to be performed in addition to the forward
pass. In conclusion, for trainable layers in a DNN, the added computational
cost of performing their backward pass is about twice as high as the
computational cost of performing their forward pass.

The high memory overhead of DNN training is the result of four aspects: First,
the partial derivatives of most layers commonly used in DNNs depend on
intermediate tensors used in the forward pass of the network, see Fig. 1 for a
schematic overview. If only inference is performed, these tensors are usually
short-lived because they only temporarily store data between successive
layers. Therefore, to save RAM, many inference libraries for MCUs try to
optimize memory requirements by reusing areas on the heap for multiple tensors
during an inference pass [2, 3]. However, when performing backpropagation,
many of these intermediate results need to be kept in memory longer, since
they are required by operations in the backward pass that can only be executed
after the complete forward pass has finished and the backward pass has
propagated the error back to the corresponding layer. As a result,
optimization of heap usage becomes less efficient.

Second, when performing SGD, many inference frameworks process an entire
minibatch of samples at once, adding another ”batch dimension” to the input,
output, and all intermediate tensors, and thus greatly increasing the memory
utilization, which now additionally depends on the minibatch size used.

Third, additional memory must be provided to store training data over time,
either as a labeled dataset for supervised training or a replay buffer for
continual learning. On an MCU, this typically requires external memory from
which data can be swapped in and out of RAM.

Fourth, since the weights of all trained layers are dynamically updated at
runtime, i.e., they are not read-only, they cannot be stored in Flash, where
most MCU inference libraries would place them, but must instead be placed in
RAM.

Figure 1: Data dependencies between forward and backward pass of linear and
convolutional layers in a DNN.

###  I-B Contribution

To address the challenges outlined in Section I-A and to make on-device
training on tiny devices such as Cortex-M MCUs feasible, we propose the
following contributions in this paper:

We introduce (i) an 8-bit fully quantized training (FQT) algorithm that
quantizes both weights and intermediate feature tensors during the forward
pass and errors and gradients tensors during the backward pass by using the
same linear quantization scheme also used for regular DNN inference. As a
result, our approach enables on-device DNN training on MCUs online, in place,
and without any transformations, changes, or conversions to the DNN’s
representation in memory or code between inference and training.

Furthermore, to allow FQT while still being able to achieve stable training
results similar to regular floating-point training on GPUs, we propose (ii) a
memory-efficient version of minibatching with standardized gradients as well
as (iii) a method to optimize the distribution of values in the 8-bit range of
quantized tensors by adapting the quantization parameters of trained weight
tensors dynamically during training, similar to quantization-aware training
(QAT) [4].

To further reduce the computational overhead of our approach, we employ (iv) a
dynamic gradient update strategy that is fully compatible with our FQT method,
which prunes the computational tree of BP per training sample based on an
online computed magnitude of error based heuristic.

Finally, we demonstrate the flexibility of our approach for both transfer
learning and full DNN training for multiple datasets (11 in total), and
provide insights into the trade-off between memory, latency, energy, and DNN
accuracy for three Cortex-M-based MCUs, see Tab. II.

The rest of this paper is structured as follows. Section II outlines work on
on-device training of DNNs on MCUs related to ours. In Section III we discuss
our approach and its implementation in more detail. In Section IV we present
our evaluation and then conclude in Section V.

##  II Related Works

On-device training on MCUs is gaining popularity as it promises a much simpler
and more energy-efficient solution for DNN fine-tuning and adaptation than
training on a remote server and then redeploying the model. As a result, much
research on on-device training so far has focused on achieving a better trade-
off between memory overhead, computing resource requirements, and accuracy.

A popular approach to reduce memory utilization is to recompute intermediate
results from the forward pass, which are required by the backward pass,
instead of keeping them in memory [5, 6]. While this helps to minimize the
additional memory overhead introduced by training, it does so at the cost of a
significant additional computational overhead due to multiple computations of
results. Another proposed approach is to reduce the memory footprint of
intermediate activation tensors by constructing a dynamic and sparse
computational graph [7]. However, the approach as presented by the authors is
only designed to minimize activations of the forward pass and still uses a
matrix-vector product, albeit reduced in size, to decide which part of the
computational graph to eliminate. Other approaches focus on a hierarchical
approach with different levels of granularity at which to drop computations,
such as the data, model, and algorithm levels [8]. Finally, reducing memory
utilization during both inference and training by processing both activations
and weights quantized has also been proposed [9, 10].

Due to the limited memory and computational resources on embedded devices, the
most common DNN training tasks addressed by researchers so far are transfer
learning and fine-tuning tasks, where an already well-trained DNN is adapted
to a changed input domain or application. The reason for the popularity of
these tasks is that they usually do not require training of a DNN from
scratch, but only fine-tuning of its last layers. For example, Tiny-transfer-
learning [11] freezes the weights of the DNN and trains only the biases, for
which updates can be computed much easier and with less memory overhead than
for weights. A similar approach is to train only batch normalization layers
[12, 13]. Alternatively, TinyOL [14] trains only the last layer of the DNN to
achieve reduced memory and computational overhead, while [13] introduces small
additional ”patches” of trainable weights to a well-trained DNN. However,
while training only subsets of a DNN’s weights can lead to adequate training
results, especially for more complex transfer learning tasks or larger domain
shifts, the results can be poor [11], making these strategies not as reliable
as full DNN training.

An approach to enable full DNN training on MCUs is described in [15], which
uses two techniques: first, freeing up memory for activation tensors early at
the cost of having to recompute them later, and second, paging where parts of
the activation tensors are swapped out to a secondary, larger memory. However,
swapping to external memory, in particular, is slow and makes certain
assumptions about the MCU that may now be applicable in all cases.

Alternatively, the authors in [10] use a combination of quantized training,
which they call quantization-aware scaling, combined with updating only parts
of the weights on-device at runtime. This approach is probably the closest to
ours, although there are some concerns that we try to address in this paper.
First, the approach described by the authors seems to be optimized exclusively
for their own DNN architecture MCUNet [2], while our method focuses on general
CNN support for different domains and applications. Second, the authors mostly
provide simulated results for transfer-learning using their on-device training
approach, while our work aims to explore the trade-off between performance and
resource requirements on MCUs of different scale (Cortex-M0+, Cortex-M4, and
Cortex-M7) and for different tasks. Third, the gradient selection method
proposed by the authors for their on-device training is performed entirely at
compile time (offline). This effectively freezes parts of the DNN at runtime,
which, similar to the concerns we mentioned about transfer learning, can lead
to poor training results for certain datasets and domains. To overcome this
problem, our work proposes a method to dynamically evaluate which gradients to
compute for each sample online during on-device training.

Finally, several memory-aware on-device learning frameworks have been proposed
for larger embedded systems, i.e., mobile and IoT edge platforms [16, 17, 18,
19]. While inspiring, the methodologies proposed and implemented by these
frameworks are not directly applicable to Cortex-M MCUs, mainly due to the
much more constrained memory and compute resources present on MCUs compared to
larger mobile and edge platforms.

##  III Concepts

To make supervised training feasible on MCUs, we propose a training framework
centered around two techniques that we will discuss below: Fully quantized
training (FQT), see section III-A, and dynamic sparse gradient updates, see
section III-B.

###  III-A Fully Quantized Training

(a) Quantization Aware Training (QAT)

(b) Fully Quantized Training (FQT)

Figure 2: Schematic comparison of a convolutional block with ReLU activation
and Batchnorm for Quantization Aware Training (QAT) and Fully Quantized
Training (FQT). In FQT the convoltional, Batchnorm and ReLU layers have been
folded into a single monolithic QConv layer.

Typically, DNNs deployed on a MCU for inference are quantized using a linear
quantization. This is done to save memory and because integer arithmetic can
be performed more efficiently on most MCUs than floating-point arithmetic.
Usually, quantization is implemented per tensor in the form of
$v_{q}=\lfloor\frac{v_{f}}{s}\rfloor+z$, where $s$ and $z$ are two parameters
scale and zero point derived from the distribution of floating-point values in
the original unquantized tensor. Therefore, in order to train a DNN in place
without any conversion, training must be performed using the same quantization
scheme already established for inference. In addition, this has the advantage
that all optimizations already established for quantized inference can be
reused for training.

A well-established method for training quantized DNNs on GPU-based systems is
called quantization-aware training (QAT) [4]. However, while the method helps
to train DNNs robust to additional errors introduced by quantization, the DNN
is not actually trained quantized, but still processed in floating-point
arithmetic, with quantization only simulated, see Fig. 2a for a schematic
overview. As a result, in QAT, the DNN is not fully transferred to quantized
space until training is complete. Therefore, QAT does not provide any
immediate memory or computational savings for the training process itself. In
addition, using QAT would require a costly conversion process at the beginning
and end of training.

To overcome the problems of QAT, and to achieve our goal of in place, online
training on fully quantized DNNs, we propose a training method based on FQT,
facilitating training directly with the quantized weights of a DNN and using
the already established quantization scheme for inference, see Fig. 2b. As a
result, our method is flexible, since there is no difference in how the DNN is
represented in memory and code between inference and training. Therefore, the
same forward pass of a DNN executed on-device using our method can be used for
both regular inference and training at the same time. Furthermore, since both
the forward and backward pass of our training method are performed on
quantized weights, feature maps, errors and gradients, the method is
significantly more memory-efficient than traditional floating-point based
training.

BP for DNNs is based on the chain rule, and calculation of local errors and
gradients for both linear and convolutional layers can be described as

$\displaystyle E_{n-1}$ $\displaystyle=W_{n}^{T}\cdot E_{n}$ (1)
$\displaystyle\nabla W_{n}$ $\displaystyle=E_{n}\cdot X_{n}^{T}$ (2)

with the $\cdot$ operator either being a matrix multiplication or a
2D-convolution when using minibatches. Therefore, Eq. (1) calculates the error
$E_{n-1}$ of the $n-1$th layer of the DNN, given the error $E_{n}$ and weight
$W_{n}$ of the $n$th layer, and Eq. (2) calculates the gradient $\nabla W_{n}$
used to update the weights of the $n$th layer given the error $E_{n}$ and the
input of the $n$th layer $X_{n}$, see also Fig. 1. Note that both Eq. (1) and
(2) are based on the same mathematical operation $\cdot$ performed during the
forward pass of the layer

$Y_{n}=W_{n}\cdot X_{n}$ (3)

with the addition of a transpose operator. In fact, Eq. (1) and (2) are the
partial derivatives of Eq. (3) with respect to $X_{n}$ and $W_{n}$
respectively. As a result, quantizing backpropagation is straightforward,
since the same quantization scheme and optimizations that are used in the
forward pass during inference can be directly applied again. Therefore, for
example, the fully quantized version of Eq. (1) for each element $e_{n-1}\in
E_{n-1}$ can be computed from $e_{n}\in E_{n}$ and $w_{n}\in W_{n}$ as

$e_{n-1}=\biggl{\lfloor}\frac{s_{w_{n}}s_{e_{n}}}{s_{e_{n-1}}}\sum(w_{n}-z_{w_{n}})(e_{n}-z_{e_{n}})\biggr{\rfloor}+z_{e_{n-1}}$
(4)

where $s_{w_{n}}$, $s_{e_{n}}$ and $s_{e_{n-1}}$ as well as $z_{w_{n}}$,
$z_{e_{n}}$, and $z_{e_{n-1}}$ are the respective scale and zero point
parameters for the quantized tensors $E_{n}$, $E_{n-1}$, and $W_{n}$. The
quantized version of Eq. (2) looks identical, however in our appraoch we omit
the re-quantization of $\nabla W_{n}$ since we perform gradient descent
locally in floating-point space in the next step.

For updating each element of the quantized weight $W_{n}$ of a given layer $n$
from a current state $i$ to a next state $i+1$ at the end of each mini-batch
we use the gradient descent algorithm

$w_{i+1}=\frac{1}{s_{w_{i+1}}}\left[\left(w_{i}-z_{w_{i}}\right)s_{w_{i}}l\nabla
w_{i}\right]+z_{w_{i+1}}$ (5)

adapted for quantization where $s_{w_{i}}$ and $z_{w_{i}}$ are the current
weight tensor’s scale and zero point parameters and $s_{w_{i+1}}$ and
$z_{w_{i+1}}$ are the updated weight tensor’s parameters. To update the scale
and zero point parameters alongside the trainable weights, $s_{w_{i+1}}$, see
Eq. (6), and $z_{w_{i+1}}$, see Eq. (7), are calculated based on the maximum
and minimum values $f_{min}$ and $f_{max}$ observed in the intermediate
results from $\left(w_{i}-z_{w_{i}}\right)s_{w_{i}}l\nabla w_{i}$ which are in
floating-point space.

$\displaystyle s_{w_{i+1}}$ $\displaystyle=\frac{f_{max}-f_{min}}{255}$ (6)
$\displaystyle z_{w_{i+1}}$
$\displaystyle=\biggl{\lfloor}-\frac{f_{min}}{s_{w_{i+1}}}\biggr{\rfloor}$ (7)

To implement stochastic gradient descent (SGD), i.e., mini-batches, there are
two viable approaches: (a) process all $b$ samples of a mini-batch at once
during a single network pass by introducing an additional dimension to the
input, output, and all intermediate tensors and then perform BP and weight
updates immediately over the entire batch, or (b) accumulate $\nabla W_{n}$ in
a buffer over $b$ successive training steps of individual samples. Both
options increase the amount of memory required compared to performing gradient
descent on each sample, i.e., batch size $b=1$. However, not using SGD and
instead relying on per-sample updates can have a significant impact on the
training performance of the DNN, as negative training effects such as
catastrophic forgetting [20] become much more prevalent.

We have chosen version (b) for our SGD approach. The reason is that by
introducing additional gradient buffers to accumulate $\nabla W_{n}$, the
additional SRAM utilization to support SGD is independent of both the batch
size $b$ and the full network depth $N$, and instead depends only on the
number of layers actually trained in the DNN. Furthermore, to minimize the
additional memory overhead introduced by SGD, we do not consider extensions
such as (Nesterov-) momentum or adaptive learning rate algorithms such as Adam
[21], as these techniques, while generally improving convergence and stability
of training, come at the cost of significantly higher memory requirements due
to the need to store multiple versions of the gradient buffer.

Finally, while testing with quantized backpropagation and SGD, we found that
standardizing the accumulated gradients per channel before performing the
update step, similar to the intuition of RMSProp [22], improves the
convergence stability of our FQT approach. This results in a final per-element
update step of our SGD approach in the form of

$w_{i+1}=\frac{1}{s_{w_{i+1}}}\left[\left(w_{i}-z_{w_{i}}\right)s_{w_{i}}l\frac{\nabla
w_{i}-\mu_{\nabla W}}{\sigma_{\nabla W}}\right]+z_{w_{i+1}}$ (8)

where $\mu_{\nabla W}$ and $\sigma_{\nabla W}$ are calculated as running mean
and standard deviation of the local gradients $\nabla W$ calculated per
sample.

###  III-B Dynamic Sparse Gradient Updates

While quantized BP can help overcome the high memory overhead of BP, it does
not directly contribute to reducing the algorithm’s computational
complexity111Note that from a practical standpoint, just using FQT often
already reduces the latency per training sample, since many MCUs can compute
more efficiently with integers than with floats. Therefore, we propose Dynamic
Sparse Gradient Updates as a technique that can be used in conjunction with
our FQT-based training method to reduce the computational overhead added by BP
with minimal impact on the training performance of our method.

We base our approach to only partially update gradients during BP on the
hypotheses that (a) during BP, the magnitude of values in the gradient tensors
becomes smaller, (b) during BP, the distribution of high magnitude values
becomes sparser for earlier layer, and (c) as training progresses and the DNN
has gained a good understanding of the underlying optimization landscape,
different samples will only activate smaller clusters of neurons in a DNN at
once. As a result, as training progresses, more and more values in
backpropagated error tensors will not be of high magnitude, i.e., they will
not lead to a significant update of the corresponding weights.

We illustrate the intuitions discussed above in Fig. 3, where we visualize the
gradient tensors of the last three linear layers of a DNN trained on the
flowers dataset from our evaluation. The heatmaps show the absolute values of
the gradient updates of a single training sample after the first and the tenth
epoch of training. We want to emphasize three aspects. First, the gradient
tensors become sparser during the backward pass (compare rows $n$ and $n-2$).
Therefore, our approach implements a per-layer gradient update rate. Second,
in general, features are not learned randomly across neurons, but in
structures that are activated at once, i.e., rows and columns for linear
layers, features and channels for convolutional layers. This observation
allows us to estimate the importance of parts of the gradient tensor, i.e.,
the magnitude of values, averaged per structure rather than per element,
thereby decreasing the overhead of our method. Third, the overall magnitude of
the gradient values decreases as training progresses (compare the two
columns), resulting in fewer gradient structures worth updating in later
stages of training than in early stages. As a result, we implement a dynamic
update rate into our algorithm, which can decrease over the course of
training.

Our proposed partial gradient updating algorithm monitors the absolute
magnitude of the values in the error tensors of each layer for each training
sample during BP as a simple heuristic to rank them. To limit the
computational overhead of calculating the heuristic, our algorithm does not
evaluate each value of the error tensors separately, but only observes the
$l1$-norm per channel/column. This optimization is based on the assumption
that trainable DNN layers represent features in structures and not randomly,
so it is worthwhile to update the weights of an entire structure when a high
error is observed.

To calculate a per-layer update rate which can subsequently be used to select
the total number of top-$k$ values that should be kept in an error tensor
based on the magnitude of error ranking achieved before, our approach uses the
following equation

$k=\lfloor\min\\{\lambda_{min}+\lvert\varepsilon\rvert(\lambda_{max}-\lambda_{min}),1\\}N\rfloor$
(9)

with $0<=\lambda_{min}<=\lambda_{max}<=1$ being hyperparameters that control
upper and lower percentage thresholds for the amount of $k$ to be kept.
Furthermore, Eq. (9) considers $\lvert\varepsilon\rvert$ which is the
difference between the current sample’s loss and the maximum loss observed
over the whole training so far. As a result, our approach allows for a dynamic
update rate between $\lambda_{min}$ and $\lambda_{max}$ over the course of
training, where the higher the training success becomes, i.e., the more the
loss converges towards zero, the more the update rate will converge towards
$\lambda_{min}$.

Figure 3: Heatmaps of the absolute values of the gradient tensors of the last three linear layers of a DNN trained on the flowers dataset, exemplarily for a training sample after the first epoch (left column) and the tenth epoch (right column) of training. TABLE I: Datasets considered for transfer learning Dataset | Classes | Input Size | Type
---|---|---|---
cwru [23] | 9 | $1\times 512$ | Time Series
daliac [24] | 13 | $1\times 1024$ | Time Series
speech [25] | 36 | $1\times 16000$ | Time Series
animals [26] | 10 | $3\times 128\times 128$ | Vision
cifar10 [27] | 10 | $3\times 32\times 32$ | Vision
cifar100 [27] | 100 | $3\times 32\times 32$ | Vision
flowers [28] | 102 | $3\times 128\times 128$ | Vision

##  IV Evaluation

We evaluate our approach on two training tasks: Transfer learning, see Secs.
IV-A, IV-B, and IV-C, and full DNN training, see Sec. IV-D. We consider three
DNN configurations for all experiments: uint8 where the complete DNN is fully
quantized, mixed where the DNN is quantized except its classification head
that remains in floating-point, and float32 where the DNN remains completely
in its original floating-point representation as a reference. Since we
observed in our experiments that especially the quantization of tensors in the
last layers of the DNN can significantly affect on-device training, we chose
the mixed configurations as another trade-off between accuracy and resource
utilization besides FQT. For all three DNN configurations, we used the
micromod222https://www.sparkfun.com/micromod platform and considered three
MCUs for testing: IMXRT2062, nrf52840, and RP2040, see Tab. II for detailed
information.

###  IV-A On-Device Transfer Learning

(a) Accuracy

(b) Latency on IMXRT2062

(c) Feature Maps, Trainable Weights, and Gradient Buffers

(d) Static Weights

Figure 4: Results of fully quantized on-device transfer learning (blue)
compared to floating-point (green) and mixed training (orange). The accuracy
results in Fig. 4a are averaged over five training runs. We also show baseline
results trained on a GPU-based server in red. The latency results in Fig. 4b
were measured on the IMXRT2062 MCU and are averaged over 1000 consecutive
training steps. In both plots, we denote the standard deviation with black
bars. Fig. 4c and 4d show the memory utilization for Flash and RAM for all
datasets as returned by our deployment framework. We have marked the memory
constraints of the different MCUs with red dashed lines where relevant, see
also Tab. II.

To validate the effectiveness of our on-device FQT approach, we performed
transfer learning on seven different datasets, see Tab. I for a detailed
overview. We chose problems of different complexity, input size and domains to
show the versatility of our approach in enabling stable FQT.

We use the same DNN architecture for all seven datasets which we dubbed
MbedNet. Its architecture is closely related to MobileNetV3 [29] scaled down
to be more suitable to the hardware constraints on MCUs. To facilitate time
series data, we map the time dimension of input samples to one of the spatial
input dimensions of the DNN while leaving the other spatial dimensions empty.
This way, MbedNet can support non-image input with only little changes to its
architecture.

TABLE II: MCUs considered for our experiments Name | Type | Clock Speed | Amps Draw Idle | Flash/ROM | RAM
---|---|---|---|---|---
RP2040 | Cortex-M0+ | 133 MHz | 31.24 mA | 16 MB (external) | 264 KB
nrf52840 | Cortex-M4 | 64 MHz | 7.27 mA | 1 MB (internal) | 256 KB
IMXRT2062 | Cortex-M7 | 600 MHz | 108.26 mA | 16 MB (external) | 2$\times$512 KB

We conducted our experiments for three DNN configurations (uint8, mixed, and
float). First, we trained a baseline model on ImageNet for all vision
datasets. We then performed transfer learning on each of the target datasets,
including network pruning to adapt the feature extraction of the model. We
used an iterative pruning schedule based on an $L1$-norm heuristic and
automated gradual pruning. Once trained, we performed post-training
quantization for both FQT and mixed training before deploying the models on
the MCU. Once deployed, we set the last five layers of each DNN to random
values, thereby resetting its classification capabilities. We then retrained
the DNNs using the original training datasets for 20 epochs with a learning
rate of $0.001$ and a batch size of $48$.

We report the results for all three DNN configurations and all tested datasets
in Fig. 4. We show the accuracy on the test split after 20 epochs of
retraining on-device in Fig. 4a (blue, orange, and green bars), averaged over
5 independent training runs and additionally provide the initial accuracy
achieved by the model after GPU training (red bar, baseline). In cases where
on-device training was not possible due to memory constraints, we simulated
retraining on a desktop computer333We used the same code and framework as used
on the MCU minus some instruction set specific optimizations, which we had to
disable.. For on-device training, we did not implement any data augmentation
on the MCU, which for some datasets, e.g., speech or flowers, resulted in more
pronounced overfitting effects on device compared to the original GPU-based
training, which included data augmentation.

For ”simpler” problems with fewer classes, i.e., cwru, daliac, animals, and
cifar10, we observed that on-device training was able to retrain or even
improve the accuracy established by the baseline consistently and with almost
no variance, while for ”complex” problems with more classes, i.e., flowers,
cifar100, speech, retraining slightly underperformed compared to the other DNN
configurations and the baseline (compare blue bars with the other bars). Due
to the reduced quantized range of the 8-bit integer gradient tensors,
ambiguous or subtle features may not be correctly learned or discriminated,
resulting in less precise decision boundaries between classes. For all
datasets, this effect was easily resolved by training the classification head
in floating-point, which resulted in a similar accuracy compared to full
floating-point training (compare green and orange bars).

In Figs. 4c, and 4d, we give an overview of the on-device memory utilization
for each of the seven datasets for transfer learning. To execute DNNs on-
device, our training framework requires three separate segments of memory. Two
segments in RAM, see Fig. 4c and one in Flash, see Fig. 4d. The segments in
RAM are required to (a) store feature maps, i.e., intermediate results of the
DNN layers during both the forward and backward pass, and (b) to store
trainable weights and gradient buffers, while the segment in Flash memory is
required for non-trainable weights. Since memory is one of the biggest
limiting factors on all our target MCUs, see Tab. II, we have highlighted
relevant memory constraints with red dashed lines both in Fig. 4c and Fig. 4d.

The differences that can be observed in memory utilization of the DNNs between
the datasets are (a) due to the different sample sizes of the datasets,
resulting in differently sized input and feature tensors, see Tab. I and (b)
the pruning we applied to the DNNs for some of the datasets during their
initial training, resulting in smaller weight tensors. Compared to performing
DNN inference on the device, additional RAM is required for weights and
gradient buffers. Furthermore, the memory required for feature tensors may
also increase, since intermediate results of layers must be kept longer in
memory to facilitate the backward pass, resulting in fewer opportunities to
reuse memory for subsequent intermediate tensors. In addition to trainable
weights that must be stored in RAM, any weight tensor that is not to be
retrained on-device can be stored read-only. Flash usage is shown in Fig. 4d.
We found that especially floating-point training could exceed the Flash of
some MCUs, e.g., the nrf52840 in our experiments. In general, however, this
was much less of a concern than managing RAM utilization.

In addition to memory, Fig. 4b shows the latency per training sample, i.e.,
the time between the DNN receiving an input, producing a prediction, and
completing backpropagation, as a unit of time for all datasets and DNN
configurations on the IMXRT2062 that we could deploy on the MCU given its
hardware constraints. All results are averaged over 1000 consecutive training
samples and broken down by time spent in the forward and backward pass
(compare light and dark colors). We observe that most of the training time was
spent on the forward pass for transfer learning. This seems counterintuitive
at first, since backpropagation theoretically performs roughly twice as many
computations as the forward pass. However, since we are performing transfer
learning rather than full DNN training in our experiments, backpropagation
does not need to be executed for all layers of the DNN, only the last few. In
addition, MBedNet is designed to learn compact representations quickly,
resulting in large, computationally expensive initial layers, but compact,
cheap final layers, allowing for a deep DNN architecture despite given
resource constraints on MCUs. As a result, the time spent computing forward
through the expensive initial layers still dominates the computation of the
cheap final layers, even with the added overhead of backpropagation and
training.

In conclusion, FQT significantly reduces memory requirements, making on-device
training on MCUs feasible. Fine-tuning the final layers comes at little
additional latency cost, and even when quantized, full baseline accuracy can
be retrained for most of the datasets we tested. Where fully quantized
training performed subpar, using a mixed approach where the classification
head is trained unquantized while convolutional layers are trained quantized
consistently addressed this problem while still offering an excellent tradeoff
between memory utilization and latency.

(a) Latency

(b) Energy

Figure 5: Comparison of latency and energy for transfer learning on the CWRU
and Daliac datasets for the three MCU platforms considered. All results are
averaged over 1000 consecutive training steps, with the standard deviation
shown as black bars.

###  IV-B Transfer Learning Across MCUs

As an additional experiment to the transfer learning task, and to get an
understanding of the performance and constraints of FQT across different MCU
platforms, we deployed the daliac and cwru datasets on the nrf52840 and RP2040
MCU and compared our findings with the results we observed on the IMXRT2062,
see Fig. 5. We only deployed the two datasets and not all seven because they
were the only two that met the memory constraints on all three MCUs. As
expected, processing on the IMXRT2062 is significantly faster than on the
other Cortex-M4 and Cortex-M0+ based systems due to the much higher clock
speed of the IMXRT2062, see Fig. 5a. However, when comparing the RP2040 to the
nrf52840 alone, even though the RP2040 has a higher clock speed, the nrf52840
was able to process training samples faster. This is because the nrf52840 has
a dedicated floating-point unit (FPU) and implements the DSP instruction set
extension, which our framework makes extensive use of to implement data-level
parallelism (SIMD) in training sample processing, allowing much higher
throughput than the RP2040, which has no FPU and no support for data-level
parallelism.

In Fig. 5b, we show the energy required per training sample averaged over 1000
consecutive training steps for each training configuration. Since we measured
the entire MCU and not just the processor, we excluded the MCU’s idle draw
listed in Tab. II. On a per-sample basis, the IMXRT2062 is the most energy-
efficient and the NRF52840 is the least. However, this analysis assumes that
samples can be processed as quickly as possible without delay between samples.
In many real-world applications, new samples arrive at a fixed rate, slower
than the MCU can process them, which can cause it to sit idle the rest of the
time. As a result, in such scenarios, the MCU with the lowest idle power is
usually the most energy-efficient, which according to Tab. II would be the
nrf52840 in our experiments.

###  IV-C Dynamic Sparse Gradient Updates

(a) Accuracy float32

(b) Accuracy mixed

(c) Accuracy uint8

(d) Speedup per sample mixed on IMXRT2062

Figure 6: Comparison of accuracy on the test datasets after 20 training epochs
for three different gradient update rates
($\lambda_{min}\in\\{0.1,0.5,1.0\\}$) for the mixed, quantized, and floating-
point DNN configurations in Figs. 6a to 6c. The results we report are averaged
over 5 training runs. We show the standard deviation as black bars.
Furthermore, we show the average speedup per sample for the different gradient
update rates for the mixed training configuration in Fig. 6d.

We validate the ability of our partial gradient updating approach to reduce
the computational complexity of backpropagation by comparing the training
accuracy achieved with different dynamic gradient update rate configurations,
see Fig. 6, with the results of our experiments presented in Section IV-A,
where we used full gradient updates. We show the test accuracy for three
different $\lambda_{min}$ (blue: $\lambda_{min}=1.0$, orange:
$\lambda_{min}=0.5$, and green: $\lambda_{min}=0.1$) for all DNN
configurations and for all seven datasets considered in our transfer learning
experiments, see Figs. 6a, 6b, and 6c. For all experiments, we performed on-
device training in combination with dynamic gradient updates for 20 epochs,
and we repeated each training 5 times and provide averaged results.

For both floating-point and mixed transfer learning (Figs. 6a and 6b), both
$\lambda_{min}=0.5$ and $\lambda_{min}=0.1$ achieved the same level of
accuracy after 20 epochs as $\lambda_{min}=1.0$, showing that, if chosen
appropriately, only a small fraction of gradients need to be actually updated
for each training sample to achieve a similar training result as with fully
updated gradients. For FQT (Fig. 6c), $\lambda_{min}=0.5$ also worked without
significant degradation compared to $\lambda_{min}=1.0$. However, with the
extremely small update rate of $\lambda_{min}=0.1$, we observed that the
training became unstable, i.e., high variance between different training runs,
and performed significantly worse than $\lambda_{min}=1.0$.

To further demonstrate that introducing partial gradient updates actually
reduces the computational complexity of backpropagation and does not just
result in a slower loss convergence over the course of training, we provide
both loss and accuracy curves exemplarily for the flowers dataset and for the
mixed DNN configuration for all three $\lambda_{min}$ tested in Fig. 8. It can
be seen that the loss convergence speed does not decrease for lower
$\lambda_{min}$ and is relatively consistent across all three different
gradient update rates, showing that dynamic gradient updating is able to
reduce computational complexity while minimally impacting training
performance.

Finally, we show the speedup achieved by partial gradient updating with all
three $\lambda_{min}$ considered when using the mixed training configuration
on the IMXRT2062 for all seven datasets, see Fig. 6d. We observed that for the
mixed DNN configuration, reducing the number of updated gradients leads to a
significant speedup per sample for backpropagation, especially for
$\lambda_{min}=0.1$, where an average speedup of about $6.64$ was achieved
across all datasets. Since the overhead of partial gradient updating is the
same for different $\lambda_{min}$, while at the same time, a smaller
$\lambda_{min}$ leads to more operations being skipped, the increase in
speedup is polynomial, depending on the number of loop nests in the layers
trained.

###  IV-D Complete On-Device Training

To evaluate that our FQT approach can also be used to train DNNs on MCUs
completely from scratch, we used a smaller DNN pre-trained on the MNIST
dataset [30], which we then fully retrained, i.e., all layers, on the device
for four MNIST-related datasets, see Tab. III. The DNN we used for this
purpose consisted of 2 convolutional layers, a max-pooling layer, and 2 linear
layers, all with ReLU as activation and BatchNorm. We used the same DNN
configurations as in Sec. IV-A, a learning rate of $0.001$ and a batch size of
$48$. The corresponding results are shown in Fig. 7.

(a) Accuracy

(b) Latency and Energy

Figure 7: Complete On-Device training of FMNIST, KMNIST, EMNIST Letters, and EMNIST Digits for a DNN consisting of two convolutional and two linear layers, both with ReLU and batch normalization, and with max-pooling in between. The accuracy results shown in Fig. 7a are averaged over 5 training runs, while the latency and energy results shown in Fig. 7b are exemplary for EMNIST for all three considered MCU platforms and are averaged over 1000 consecutive training steps. We denote the standard deviation with black bars. Figure 8: Accuracy and loss curves for all three considered gradient update rates, exemplarily for a training run on the flowers dataset using the mixed DNN configuration TABLE III: Datasets considered for full on-device training Dataset | Classes | Input Size | Type
---|---|---|---
FMNIST [31] | 10 | $1\times 28\times 28$ | Vision
KMNIST [32] | 10 | $1\times 28\times 28$ | Vision
EMNIST-Letters [33] | 26 | $1\times 28\times 28$ | Vision
EMNIST-Digits [33] | 10 | $1\times 28\times 28$ | Vision

In Fig. 7a, we show the accuracy results after on-device training for all
three DNN configurations and for all four datasets. On-device full floating-
point training achieved the highest accuracy, closely followed by mixed and
fully quantized training. Different from transfer learning, we observed a more
pronounced gap between quantized and floating-point training. A reason for
this observation is, that while the transfer learning experiments we discussed
earlier only retrained the classification head and the last convolutional
layers of the DNN, and thus could still rely on a well-trained feature
extractor, i.e., the earlier layers, the full on-device experiments shown in
Fig. 7a had to learn both feature extraction and classification from scratch.
As a result, it appears that the limited resolution of the 8-bit data types
used for training resulted in the DNN not being able to adequately learn the
extraction of more complex features in its earlier layers. Therefore, although
the training was still generally successful for FQT, we observed an earlier
stagnation of FQT compared to floating-point or mixed training, and thus a
slightly reduced accuracy. Similar effects can be observed with training of
other kinds of low-bit representation DNNs like binarized DNNs [34]. A more
fine-grained quantization scheme, such as per-channel quantization [35], or
alternative quantization strategies, such as weighted entropy [36] or
stochastic quantization [37], while not as well supported by existing tool
flows on MCUs, could be useful approaches to bridge this gap as long as they
can maintain a reasonable computational overhead.

We also present latency and energy results for our complete on-device training
experiments exemplarily for the EMNIST Digits dataset and for all three
considered MCU platforms and training configurations in Fig. 7b. For the
nrf52840 and RP2040, we were only able to test the uint8 configuration due to
memory constraints on both platforms. As in Fig. 4b above, we mark the
fraction of latency per sample spent in the forward pass in light colors and
in the backward pass in dark colors. Since we are training all layers of the
DNN, the time spent in the backward pass is significantly higher than the time
spent in the forward pass. These observations are the opposite of what we saw
earlier in Fig. 4b, where only the last layers were trained, but they are much
closer to our initial intuition based on the theoretical analysis of FQT
training that we stated at the beginning of the paper. Nevertheless, the
results presented in Fig. 7b clearly show the significant computational impact
that especially training of early feature extracting layers has.

Finally, we show energy results in Fig. 7b. Identical to the results discussed
in Fig. 5b, we subtracted the idle draw of the MCUs shown in Tab. II. As with
the results discussed earlier, we can see that the IMXRT1062 is the most
energy-efficient option per sample compared to the nrf52840 and RP2040, which
instead have a much lower idle power draw.

###  IV-E Comparison with MCUNet-5FPS

TABLE IV: Comparison between our quantized optimizer and SGD+M+QAS [10] for updating the last two blocks of MCUNet-5FPS [2]. For our approach, we provide the accuracy on the test splits of the datasets. Precision | Optimizer | Accuracy (%) (MCUNet backbone: 23M MACs, 0.48M Param) |  Avg. Acc.
---|---|---|---
Cars [38] |  CF10 [27] |  CF100 [27] |  CUB [39] |  Flowers [28] |  Food [40] |  Pets [41] |  VWW [42]
fp32 |  SGD-M [10] | 56.7 | 86.0 | 63.4 | 56.2 | 88.8 | 67.1 | 79.5 | 88.7 | 73.3
int8 |  SGD-M [10] | 31.2 | 75.4 | 64.5 | 55.1 | 84.5 | 52.5 | 79.5 | 88.7 | 64.9
int8 |  SGD+M+QAS [10] | 55.2 | 86.9 | 64.6 | 57.8 | 89.1 | 64.4 | 80.9 | 89.3 | 73.5
uint8 | ours | 54.5 | 89.5 | 65.2 | 58.5 | 85.8 | 66.6 | 79.8 | 89.3 | 73.7

We compare the performance of our FQT optimizer with the results reported by
the authors of the optimizer SGD+M+QAS [10] when training the last two blocks
of MCUNet-5FPS [2], see Tab. IV. The first row in Tab. IV shows the baseline
accuracy reported by [10] for MCUNet trained on eight datasets using a
floating-point SGD optimizer with momentum (SGD-M). We could reproduce these
results using the publicly available sources of MCUNet provided by the
authors. The next two rows show the results obtained by [10] first for naively
using quantized SGD-M without any changes besides the usage of quantized
tensors and second for their SGD+M+QAS approach, which addresses shortcomings
of the SGD-M approach. The last row shows the results we obtained for
retraining MCUNet using our approach, i.e., fully quantized training with
standardized gradients and dynamic adaptation of the zero-point and scale
parameters as described in Sec. III. All our reported results are on unseen
test splits of the datasets. Like [10] we retrained for 50 epochs. We used a
learning rate of $0.001$ and a batch size of $48$.

Our optimizer matched SGD+M+QAS closely, even outperforming it in five of the
eight datasets, see the last two rows in Table IV. On average, this resulted
in our approach having a slightly higher performance in retraining MCUNet, see
last column of Table IV, while exceeding the performance of naively retraining
the quantized weights, i.e. SGD-M.

Figure 9: Memory (green: ROM, orange + blue: RAM) and latency averaged over
1000 consecutive training steps of MBedNet (ours) compared to MCUNet-5FPS
deployed on the IMXRT2062 for cifar10.

However, the main goal of both our optimizer and SGD+M+QAS is to provide a
better tradeoff between memory and latency to meet the resource constraints of
MCUs and enable on-device training. But since the authors of [10] do not
report detailed numbers for SGD+M+QAS, we instead report a comparison between
MCUNet deployed with our framework and MbedNet for cifar10, see Fig. 9.

MCUNet has more trainable parameters than MBedNet, especially in its last
layers. This results in an overall higher memory utilization of MCUNet, see
the left plot in Fig. 9. More specifically, since the last layers are
trainable, this results in (a) larger feature maps to propagate between layers
(blue segment of bars), and (b) more memory required to store larger gradient
and weight buffers to enable training of the last layers (orange segment of
bars). This trend can likewise be observed for latency, see the left plot in
Fig. 9. Because MBedNet’s final layers are smaller, both their forward and
backward passes can be executed significantly faster than MCUNet’s. Since the
backward pass has to compute two partial derivatives for each layer, the
difference is more pronounced than for the forward pass. Finally, when
comparing accuracy, both architectures achieve a similar baseline of around
86%. When using our optimizer, MCUNet could be trained to an accuracy of 89.5%
for cifar10, which is slightly higher than the 85.9% our optimizer achieved
for MBedNet but which is in line with what SGD-M-QAS achieved on MCUNet
(86.9%). Still, we argue that the significantly lower memory and latency cost
of MBedNet compared to MCUNet makes it a more suitable architecture for
efficient on-device training on MCUs. This allows MbedNet not only to run
inference at around 5FPS (200 ms) like MCUNet, but also to retrain.

##  V Conclusion

In this work, we discussed the challenges of on-device training on Cortex-M
microcontrollers, such as computational overhead and memory constraints, and
proposed a method to overcome them. To this end, we proposed a combination of
fully quantized on-device training (FQT) combined with dynamic partial
gradient updates to reduce compute overhead by dynamically adapting to the
distribution and magnitude of gradients values during backpropagation. In an
extensive evaluation, focusing on three different Cortex-M-based MCUs, we show
that on-device transfer learning and full on-device training are both feasible
from an accuracy and resource utilization standpoint. Therefore, our results
indicate that on-device MCU training is possible for a wide range of datasets.
Furthermore, our method is flexible and allows for training configurations,
where some layers of a DNN are trained in floating-point, while others are
trained quantized. This solidifies the capability of our method in adapting to
problems of varying complexity and to different DNN architectures.

## References

  * [1] J. Lin, L. Zhu, W.-M. Chen, W.-C. Wang, and S. Han, “Tiny machine learning: Progress and futures [feature],” _IEEE Circuits and Systems Magazine_ , vol. 23, no. 3, pp. 8–34, 2023.
  * [2] J. Lin, W.-M. Chen, Y. Lin, C. Gan, S. Han _et al._ , “McuNet: Tiny deep learning on IoT devices,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 11 711–11 722, 2020.
  * [3] M. Deutel, P. Woller, C. Mutschler, and J. Teich, “Energy-efficient deployment of deep learning applications on Cortex-M based microcontrollers using deep compression,” in _MBMV 2023; 26th Workshop_. VDE, 2023, pp. 1–12.
  * [4] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2704–2713.
  * [5] A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves, “Memory-efficient backpropagation through time,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [6] T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” _arXiv preprint arXiv:1604.06174_ , 2016.
  * [7] L. Liu, D. Lei, H. Xing, Z. Maohua, L. Guoqi, D. Yufei, and X. Yuan, “Dynamic sparse graph for efficient deep learning,” in _International Conference on Learning Representations_ , 2019.
  * [8] Y. Wang, Z. Jiang, X. Chen, P. Xu, Y. Zhao, Y. Lin, and Z. Wang, “E2-train: Training state-of-the-art cnns with over 80% energy savings,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [9] N. Wang, J. Choi, D. Brand, C.-Y. Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point numbers,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [10] J. Lin, L. Zhu, W.-M. Chen, W.-C. Wang, C. Gan, and S. Han, “On-device training under 256kb memory,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 22 941–22 954, 2022.
  * [11] H. Cai, C. Gan, L. Zhu, and S. Han, “TinyTL: Reduce memory, not parameters for efficient on-device learning,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 11 285–11 297.
  * [12] J. Frankle, D. J. Schwab, and A. S. Morcos, “Training batchnorm and only batchnorm: On the expressive power of random features in CNNs,” in _International Conference on Learning Representations_ , 2020.
  * [13] P. K. Mudrakarta, M. Sandler, A. Zhmoginov, and A. Howard, “K for the price of 1: Parameter-efficient multi-task and transfer learning,” in _International Conference on Learning Representations_ , 2018.
  * [14] H. Ren, D. Anicic, and T. A. Runkler, “TinyOL: TinyML with online-learning on microcontrollers,” in _International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [15] S. G. Patil, P. Jain, P. Dutta, I. Stoica, and J. Gonzalez, “Poet: Training neural networks on tiny devices with integrated rematerialization and paging,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 17 573–17 583.
  * [16] Q. Wang, M. Xu, C. Jin, X. Dong, J. Yuan, X. Jin, G. Huang, Y. Liu, and X. Liu, “Melon: Breaking the memory wall for resource-efficient on-device machine learning,” in _Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services_ , 2022, pp. 450–463.
  * [17] D. Xu, M. Xu, Q. Wang, S. Wang, Y. Ma, K. Huang, G. Huang, X. Jin, and X. Liu, “Mandheling: Mixed-precision on-device dnn training with dsp offloading,” in _Proceedings of the 28th Annual International Conference on Mobile Computing And Networking_ , 2022, pp. 214–227.
  * [18] I. Gim and J. Ko, “Memory-efficient dnn training on mobile devices,” in _Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services_ , 2022, pp. 464–476.
  * [19] X. Qi and C. Liu, “Enabling deep learning on iot edge: Approaches and evaluation,” in _2018 IEEE/ACM Symposium on Edge Computing (SEC)_. IEEE, 2018, pp. 367–372.
  * [20] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of Learning and Motivation_. Elsevier, 1989, vol. 24, pp. 109–165.
  * [21] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [22] G. Hinton, N. Srivastava, and K. Swersky, “Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,” (Date last accessed 04-June-2024). [Online]. Available: http://www.cs.toronto.edu/~hinton/coursera/lecture6/lec6.pdf
  * [23] “Bearing data center,” (Date last accessed 08-March-2024). [Online]. Available: https://engineering.case.edu/bearingdatacenter
  * [24] H. Leutheuser, D. Schuldhaus, and B. M. Eskofier, “Hierarchical, multi-sensor based classification of daily life activities: comparison with state-of-the-art algorithms using a benchmark dataset,” _PloS one_ , vol. 8, no. 10, p. e75196, 2013.
  * [25] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [26] “Animals-10,” (Date last accessed 08-March-2024). [Online]. Available: https://www.kaggle.com/datasets/alessiocorrado99/animals10
  * [27] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009.
  * [28] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_. IEEE, 2008, pp. 722–729.
  * [29] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1314–1324.
  * [30] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [31] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms,” _arXiv preprint arXiv:1708.07747_ , 2017.
  * [32] T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha, “Deep learning for classical japanese literature,” _arXiv preprint arXiv:1812.01718_ , 2018.
  * [33] G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik, “EMNIST: Extending MNIST to handwritten letters,” in _International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2017, pp. 2921–2926.
  * [34] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [35] R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” _arXiv preprint arXiv:1806.08342_ , 2018.
  * [36] E. Park, J. Ahn, and S. Yoo, “Weighted-entropy-based quantization for deep neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5456–5464.
  * [37] Y. Dong, R. Ni, J. Li, Y. Chen, H. Su, and J. Zhu, “Stochastic quantization for learning accurate low-bit deep neural networks,” _International Journal of Computer Vision_ , pp. 1–14, 2019.
  * [38] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2013, pp. 554–561.
  * [39] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-ucsd birds 200,” Technical Report CNS-TR-2010-001, California Institute of Technology, Tech. Rep., 2010.
  * [40] L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_. Springer, 2014, pp. 446–461.
  * [41] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 3498–3505.
  * [42] A. Chowdhery, P. Warden, J. Shlens, A. Howard, and R. Rhodes, “Visual wake words dataset,” _arXiv preprint arXiv:1906.05721_ , 2019.