# BioNetExplorer: Architecture-Space Exploration of Bio-Signal Processing Deep
Neural Networks for Wearables

Bharath Srinivas Prabakaran1, Asima Akhtar1, Semeen Rehman, Osman Hasan, , and
Muhammad Shafique 1 These two authors have contributed to this work equally.B.
S. Prabakaran and S. Rehman are with Technische Universität Wien
(TU Wien), Vienna, Austria.
E-mail: {bharath.prabakaran<EMAIL_ADDRESS>Akhtar and O.
Hasan are with National University of Sciences and Technology (NUST),
Islamabad, Pakistan.
E-mail: {aakhtar.msee17seecs<EMAIL_ADDRESS>Shafique is with
the Division of Engineering, New York University Abu Dhabi (NYU-AD), Abu
Dhabi, United Arab Emirates.
E-mail<EMAIL_ADDRESS>received September 07, 2020;
revised January 24, 2021.Copyright © 2021 IEEE. Personal use of this material
is permitted. However, permission to use this material for any other purposes
must be obtained from the IEEE by sending a request to pubs-
<EMAIL_ADDRESS>
###### Abstract

Deep Learning (DL) has been shown to be highly effective in solving various
problems across numerous applications and domains, such as autonomous driving
and image recognition. Due to the advent of DL, plenty of research works have
explored the applicability of DL, more specifically Deep Neural Networks
(DNNs), to solve pattern recognition and computer vision challenges. More
recently, researchers have focused on the topic of automated generation and
exploration of DNN architectures, which tend to mostly focus on image
recognition or visual datasets, primarily, due to the computer vision-related
DL advancements.

In this work, we propose the BioNetExplorer framework to systematically
generate and explore multiple DNN architectures for bio-signal processing in
wearable devices. Our framework varies key neural architecture parameters to
search for an embedded DNN architecture with a low hardware overhead, which
can be deployed in wearable edge devices to analyze the bio-signal data and to
extract the relevant information, such as arrhythmia and seizure. Furthermore,
BioNetExplorer reduces the exploration time by deploying genetic algorithms,
such as NSGA-II, SPEA-2, etc. Our framework also enables hardware-aware DNN
architecture search by imposing user requirements and hardware constraints
(storage, FLOPs, etc.) during the exploration stage, thereby limiting the
number of networks explored. Moreover, BioNetExplorer can also be used to
search for DNNs based on the user-required output classes; for instance, a
user might require a specific output class, attributed towards ventricular
fibrillation, due to genetic predisposition or a pre-existing heart condition.

The use of genetic algorithms reduces the exploration time, on average, by
$9\times$, compared to exhaustive exploration. We are successful in
identifying Pareto-optimal designs, which can reduce the storage overhead of
the DNN by ~$30$MB for a quality loss of less than $0.5\%$. To enable low-cost
embedded DNNs, BioNetExplorer also employs different model compression
techniques to further reduce the storage overhead of the network by up to
$53\times$ for a quality loss of $<0.2\%$.

###### Index Terms:

Deep Neural Networks, Bio-signals, Wearables, Healthcare, Exploration, LSTM,
Long Short Term Memory, Convolution, DNN, Embedded Systems, Performance,
Efficiency.

##  I Introduction

According to recent estimates from the International Data Corporation (IDC),
by the year $2025$, $41.6$ billion IoT (Internet-of-Things) devices are
expected to generate nearly $79.4$ Zettabytes ($\times 10^{12}$ Gigabytes) of
data annually [1]. This includes the data collected by autonomous cars, home-
automation environments, smart wearables, etc. Recent estimates suggest that
the number of smart wearables and fitness trackers in the market will rise to
~$356$ million by the end of $2020$, with each device generating roughly $560$
Megabytes of data every month (averaging ~$185$ Petabytes, monthly) [2]. These
trends are expected to rapidly escalate due to the current Covid-19 outbreak.
In such situations, low-cost wearables can be widely used for personalized
monitoring and prediction to contain any current/future pandemics [3] [4]. The
data collected from these smart edge-devices is typically transmitted to the
cloud for filtering, processing, and mining to extract valuable information,
which can benefit the end-user or the system developer to provide health and
lifestyle recommendations to improve the users’ quality-of-life. However,
transmitting such a large amount of data to a fog device incurs significant
time, power, and energy overheads on the wearable device [5]. Besides the
overheads associated with these systems, transmitting data to the fog or cloud
may create security and privacy concerns for the users who might not wish to
transfer their physiological information over untrusted networks or store
their data on third-party cloud platforms and data-servers [6].

Figure 1: Storage Overhead and Number of Floating Point Operations of State-
of-the-Art Deep Neural Networks (based on data from [7, 8, 9, 10, 11]). These
networks have a very high hardware overhead and perform ~$10^{12}$ floating
point operations.

To address these issues, the Edge Computing paradigm has emerged [12], which
brings the computing layer closer to the sensor-nodes, i.e., the edge devices
where the data is collected, to improve response times and to reduce energy
consumption. For example, such a system model is also adopted in the so-called
Near-Sensor Computing paradigm [13]. However, these edge devices are typically
not capable of executing compute- and memory-intensive data processing
algorithms like complex Deep Neural Networks (DNNs), which are currently used
for various purposes such as classification, long-/short-term predictions, and
anomaly detection in bio-signal data. Fig. 1 presents an analysis of the
storage overhead and number of floating point operations required for a single
inference of different state-of-the-art DNNs [7, 8, 9, 10, 11].

From these results, one can make the following observations:

  * •

DNNs are application dependent, i.e., each application, based on its
requirement, needs a specific trained DNN architecture that could be very
different from the architecture used by another application. For example,
DenseNet-$190$ is used for image classification, whereas Deep Speech-II is
considered to be state-of-the-art for speech recognition;

  * •

DNNs incur a large on-chip memory overhead for storing the network’s
parameters, which are required for executing the DNN;

  * •

DNNs require a large number of floating-point operations to be executed in a
second, to enable the system to work and process data in real-time;

Based on these observations, the following research challenges need to be
addressed to enable embedded DNNs for resource-constrained wearables:

  * •

Based on the number of DNN parameters and their possible values, the design
space of DNNs can explode, leading to a very large number of networks in the
architecture-space that need to be exhaustively trained and evaluated. This is
a complex problem to solve; requiring a huge amount of resources for network
search and training. These kinds of resources may only be affordable by large-
scale enterprises. The key scientific questions, that we target in this paper
are:

    * [Q1]

How do we traverse the architecture-space of DNNs to effectively reduce the
exploration time, while identifying DNNs that trade-off between output quality
and hardware overhead?

    * [Q2]

Can we effectively and simultaneously optimize both the output quality and the
hardware overhead, instead of just one of these factors, while traversing the
DNN architecture-space?

  * •

Besides exploring the conventional DNN architecture-space, model compression
techniques, such as Pruning and Quantization, can also be used to reduce the
hardware overhead of the network. In this regard, the question is:

    * [Q3]

How would different pruning and quantization techniques impact the quality of
the bio-signal processing application and minimize the hardware overhead of
the DNN?

To address these research challenges, we propose the following novel
contributions:

  * •

We propose a novel BioNetExplorer framework, which systematically varies the
key neural architecture parameters of a DNN to construct an architecture-space
that can be subsequently explored and evaluated to identify networks that
satisfy the user requirements and hardware constraints of the target platform.

  * •

We investigate four well-established genetic algorithms to identify near-
optimal points/designs without exhaustively exploring the complete search-
space and study their outcome when applied to our architecture-space.

  * •

We propose a weighted cost function that can be used to simultaneously
optimize output quality and hardware overhead. The weights allocated to these
two factors will determine the primary optimization goal of the genetic
algorithm-based search and its priority.

  * •

Our BioNetExplorer framework effectively constructs a new architecture-space
of the near-optimal DNN by varying the key pruning and quantization
parameters. It then explores this architecture-space to identify networks that
minimize the hardware overhead with minimal or no quality loss.

Open-source Contributions: The complete framework along with the Pareto-
optimal networks for all the explored DNN architecture-spaces will be made
open-source and available online at https://bionetexplorer.sourceforge.io.

Fig. 2 illustrates an overview of the proposed contributions integrated in our
novel BioNetExplorer framework.

Figure 2: An Overview of the Proposed Contributions (Dark Highlighted Boxes)
in Our BioNetExplorer Framework.

##  II Related Work & Preliminaries

Wearables are highly pervasive smart electronic devices that are used to
improve the users’ quality-of-life. This includes a wide range of devices like
smart-watches, such as the ones manufactured by Apple [14] and Samsung [15],
fitness trackers, like Fitbit [16], sports watches, smart clothing, bio-
implants, etc. One of the most important functions offered by these devices is
physiological signal monitoring, which can be used to track the health of its
users [17]. For example, the Apple Watch and Fitbit can be used to monitor the
electrical signals associated with a heartbeat and record them in the form of
an electrocardiogram (ECG). The data gathered from these devices are
transmitted to the cloud for further processing and detecting anomalies such
as Atrial Fibrillation [18]. There has been a wide range of research works
that have focused on the adoption of wearables for monitoring the user’s
health [19, 20, 21]. For instance, Sun et al. [22] have proposed an optimized
vector transformation approach that reduces the overhead of key generation,
thereby enabling lightweight encryption and decryption in smart-health IoT
infrastructure. Similarly, to enable real-time feedback to the user, a coding-
free control system for an Industrial IoT (IIoT) infrastructure was presented
in [23].

There has been a wide range of works on Neural Architecture Search (NAS), from
the use of basic algorithms to Reinforcement learning for the sole purpose of
designing DNNs. A comprehensive survey of the neural architecture search
approaches based on the search space, the strategy of search, and performance
of the approach is presented in [24]. One of the most popular techniques for
hyper-parameter optimization is Bayesian Optimization (BO) [25]. However, it
has not yet been extensively explored for NAS. This is mainly because the BO
toolboxes are based on Gaussian processes and tend to focus on low-dimensional
continuous optimization problems, and is, thus, not suitable for architecture-
space exploration [24]. A recurrent neural network (RNN), which can be used to
generate DNN architectures for classifying objects in the CIFAR-10 image
dataset, is presented in [26]. The RNN is trained using a reinforcement
learning approach to maximize the accuracy of the networks generated. Liu et
al. [27] presented a sequential model-based optimization strategy, which can
be used to generate convolutional neural networks $5\times$ as efficiently as
the approach presented in [26]. In $2019$, Liu et al. [28] proposed a network-
level structure search combined with the traditional cell-level structure
search to compose a hierarchical architecture space of state-of-the-art DNNs.
Typically, the state-of-the-art approaches for NAS focus on image
classification and image segmentation [29] without much focus on other
application domains such as bio-signal processing. Recently, Tan et al. [30]
proposed a platform-aware neural architecture search approach, which focuses
solely on image classification and object recognition. Similarly, several
other recent works presented platform-aware DNN architecture-space exploration
approaches that are targeted for image classification [31, 32, 33]. These
approaches do not offer the same accuracy-hardware trade-offs for other
application domains, such as bio-signal processing, as illustrated in Section
V-A.

Typically, Model Compression techniques such as Pruning and Quantization have
been shown to be very effective in reducing the hardware overheads and
computational requirements of DNNs. Han et al. [34] proposed a layer-wise
approach to prune and quantize DNNs that are used for image classification.
Anwar et al. [35] presented a technique that can be used to introduce
structured sparsity in the network to reduce the representation effort and
maximize parallel computation. As an alternative approach to structured
pruning, Luo et al. [36] presented a method that eliminates complete filters
based on data from subsequent layers to reduce the number of computations
required for inference and accelerate DNN execution. Marchisio et al. [37]
improved upon the work presented in [34] by introducing an iterative class-
blind approach for pruning weights in DNNs to further reduce the hardware
overhead of the network. Lin et al. [38] proposed a framework that can be used
to prune a DNN dynamically at runtime, based on the input image and feature
maps, to preserve the full ability of the original DNN. Similarly, there is a
wide range of techniques focused on quantizing DNNs to reduce their hardware
overhead [39, 40, 41, 42, 43]. Besides NAS and model compression, Kumar et al.
[44] and Gural et al. [45] presented effective ML-based approaches that can be
deployed on resource-constrained IoT devices to perform search predictions and
image classification.

In this work, we primarily focus on the automatic generation and exploration
of bio-signal processing DNNs that can be deployed on resource-constrained
wearable devices. Plenty of research works have used manually-designed DNNs
(i.e., not obtained through automatic neural architecture search) for
classification of ECG signal components and/or arrhythmia [46, 47, 48]. The
DNN architecture proposed by Hannun et al. [48] can classify $12$ rhythm
classes on their custom ECG dataset and is considered to be the state-of-the-
art for arrhythmia classification. Our BioNetExplorer framework adopts the
basic building block used in [48] to generate and explore DNN architectures
for specific arrhythmia categories, as per user requirement, which can be
deployed on wearables like smart-watches or fitness trackers. Note, we have to
modify the input and output layers of this DNN to ensure that the network can
process data from the available training dataset and classify them into the
required output classes based on the target application. We have also included
a Long Short-Term Memory (LSTM) [49] layer at the end of the DNN to improve
accuracy in worst-case scenarios, i.e., lower number of feature extraction
layers. Our ECG processing DNN architecture, modified from the DNN presented
in [48], is illustrated in Fig. 3. Given such a DNN architecture, the
challenge is to explore the following architectural parameters:  (i) Number of
ResNet Blocks;  (ii) Number of Filters; and  (iii) Number of LSTM Cells.  This
requires us to generate and explore different DNN variants considering these
network parameters to identify a set of Pareto-optimal configurations for the
DNN architecture. This provides a low-complexity solution instead of employing
a full-scale NAS from scratch.

Figure 3: Our ECG Processing DNN Architecture to Detect Anomalies (modified
from the DNN presented in [48]). Figure 4: An Overview of Different Key Steps
in Our BioNetExplorer Framework. We consider the applications’ quality
requirements and hardware constraints of the system as inputs to our
framework. Based on these requirements, we construct and explore the
architecture-space of DNNs. These DNNs are trained using a custom dataset,
which we construct for the target application from existing open-source
datasets. Afterwards, the DNNs are further compressed to obtain a DNN with the
minimum-possible hardware overhead and maximum-possible output quality, under
the imposed requirements and constraints.

##  III The BioNetExplorer Framework

An overview of our BioNetExplorer framework is presented in Fig. 4. It
considers the user requirements (e.g., output classes and quality) to
construct and label the comprehensive bio-signal data-set, and hardware
constraints (e.g., Floating-point Operations (FLOPs) and memory storage
overhead (MB)) to generate and explore the architecture-space of bio-signal
processing DNNs for wearables.

###  III-A User Requirements

The BioNetExplorer framework restricts the output classes of the DNN being
explored. For instance, in case the user requires another output class besides
the two base cases of normal and anomalous, due to pre-existing conditions,
another output class and the associated label can be included by our framework
to search for an efficient DNN architecture in the new architecture-space.
Note, the labeled data associated with the new output class has to be present
in the dataset. We consider four metrics for evaluating the quality
($Q_{DNN}$) of DNNs, namely, Accuracy ($A_{DNN}$), Recall ($R_{DNN}$),
Precision ($P_{DNN}$), and F1-score ($F1_{DNN}$). These metrics are defined
as:

$A_{DNN}=\frac{\text{True Positives + True Negatives}}{\text{Total Number of
Classifications}}$ (1) $R_{DNN}=\frac{\text{True Positives}}{\text{True
Positives + False Negatives}}$ (2) $P_{DNN}=\frac{\text{True
Positives}}{\text{True Positives + False Positives}}$ (3)
$F1_{DNN}=2*\frac{\text{Precision * Recall}}{\text{Precision + Recall}}$ (4)

Recall is the ability of the DNN under consideration to find all relevant
cases of a specific class within the dataset, while Precision defines the
ability of the DNN to identify relevant points in the dataset. F1-score is
defined as the harmonic mean of Precision and Recall. The user can define a
minimum quality constraint using any of the aforementioned metrics to ensure
that the final network achieved by the exploration satisfies this constraint.

###  III-B Hardware Constraints

Besides the quality constraint, a hardware-level constraint on the networks
being explored ensures that these DNNs do not require compute or memory
resources beyond the configuration of the target wearable platform. We can
estimate the hardware requirements of a DNN using metrics such as Storage
Overhead ($S_{DNN}$) based on the number of parameters (weights and biases) of
the network, and Floating-point Operations ($F_{DNN}$) that determines the
execution time of the DNN on the target platform. Due to the hardware
constraints of the wearable device, an efficient DNN that can be executed on
such devices needs to offer the best possible quality with minimum-possible
$S_{DNN}$ and $F_{DNN}$. This typically creates a trade-off, wherein the
quality of the DNN typically decreases when we reduce the number of parameters
to curtail the memory footprint/FLOPs, and vice-versa.

###  III-C Dataset Construction

Our framework constructs a dataset for each specific DNN exploration by
combining labels and creating the desired output classes from the primary bio-
signal dataset. Each of the labels/annotations in the primary dataset should
correspond to one of the output classes of the explored DNN in the final
constructed dataset. For example, in our case study, we create a new dataset
by grouping together a $256$ bio-signal sample window and assigning a specific
label to this window based on the original labels of the signals present. Each
of the $41$ beat and non-beat annotations in the dataset are eventually
classified into one of the output labels in each DNN exploration. We use
$70\%$ of the constructed datasets for training the DNN architecture, $10\%$
for validating the network, and the remaining $20\%$ for testing and
estimating the output quality of the DNN. We have explained this scenario with
help of a few different examples in Section V.

###  III-D Generation of DNN Architectures

After collecting information regarding all the relevant constraints and
requirements that can be used to restrict the architectural-space, and the
dataset required for the target application, we generate the set of possible
DNN architectures ($\psi$) by varying three key neural architecture
parameters, namely,  (1) #ResNet Blocks,  (2) #Filters, and  (3) #LSTM Cells.
These three parameters can potentially take up any value in the domain of
${\rm I\\!R^{+}}$, thereby, creating an unbounded design space that is
impossible to explore. Therefore, in this work, we consider the parameters of
the current state-of-the-art DNN architectures as the upper-bounds to our
architecture-space exploration. Precisely, in our case-study, the ranges of
these parameters are as follows:

  1. 1.

The #ResNet Blocks [50] can vary from $0$ to $15$, with each block being
composed of two 1-D convolution layers, followed by batch normalization [51],
Rectified Linear Unit (ReLU) activation, and dropout [52];

  2. 2.

The #Filters (size 16) in each convolution layer is defined as $32\times
2^{y}$ where the value of $y$, starting from $0$, is incremented by $1$ after
every $x$ ResNet Blocks where $x\in[1,4]$; and

  3. 3.

The #LSTM cells can vary as a function of $2^{z}$ where $z\in[4,8]$.

Due to the upper-bound on the number of parameters and storage size imposed by
the state-of-the-art networks, we can reduce the number of networks to be
explored to realistic limits. For example, in our case study (see Section V),
the number of explored networks is reduced from $320$ to $135$. It is
important to note that, since the exploration of the architecture-space is
limited by the current generation of state-of-the-art architectures, any major
upgrades in the architectures of the state-of-the-art will, in turn, affect
the architecture-space of the explored DNNs.

###  III-E Exploration of DNN Architectures

Each of the DNNs obtained from varying the architectural parameters needs to
be trained using the constructed bio-signal data-set to detect anomalies
and/or specific conditions. Since training is a compute-intensive task, it is
important to  (i) generate fewer DNN models (as discussed above), and  (ii)
explore the search-space fast.  At this stage, there are two ways to explore
the architecture-space:

(1) Exhaustive Search, which involves training all the DNNs in the
architecture-space to identify the set of Pareto-optimal DNN architectures
that trade-off QoS for $S_{DNN}$/$F_{DNN}$ and vice-versa. Training all the
networks in the architecture-space requires hundreds of GPU-hours depending
upon the size of the networks being explored. Due to the practical constraints
imposed by hardware resources present on wearables and the upper bound imposed
by state-of-the-art DNNs, we have a curtailed architecture-space, for which we
can perform an exhaustive search. Therefore, for benchmarking the efficiency
of the genetic algorithms and random search, we additionally implemented the
exhaustive search. However, when the network complexity, number of parameters,
parameter instances, and the number of hyper-parameters increase, thereby
leading to exponentially large architecture-spaces, we usually rely on
genetic-algorithm based architecture-search methods.

(2) Genetic Algorithm-Based Search, which involves effectively training a
small subset of the networks in the architecture-space to reduce the training
time to only tens of GPU-hours. Genetic algorithms have proved to be quite
effective in utilizing the given cost function to obtain near-optimal
solutions while reducing the time required for exploration [53]. Towards this,
we leverage the Genetic Algorithms that rely on the biological principles of
reproduction and evolution to create a new generation of networks that can
potentially perform better than the previous generation to optimize the cost
function. The scope of our work is limited to the investigation of genetic
algorithms, a small subset of meta-heuristics that encompasses other
techniques like Tabu Search [54], Simulated Annealing [55], Ant Colony
Optimizations [56], etc., for the exploration of our DNN architecture-spaces.

Figure 5: Overview of our Constructed Chromosome. The three network parameters
are encoded as genes, which are combined together to form our chromosome. The
number of bits for each gene is selected based on the possible number of
values for each parameter. This leads to $2^{9}$ possible DNN architectures.
Figure 6: Example of Crossover and Mutation in our Chromosome.

Initially, we start with a set of random individuals/DNNs in the architecture-
space, which we refer to as the Initial Population. Based on our experiments
and recommendation from previous works [57], we have identified that a
population size of $30$ offers the best results in terms of designs obtained
for our architecture-space $\psi$. Each of the three key network parameters
(#Filters, #ResNet Blocks, #LSTM cells) that can be varied in the DNN
architecture-space is encoded as a Gene. These genes are joined together in
the form of a string to generate a Chromosome, which can be decoded to
construct the DNN architecture. Based on the number of possible values for
each gene, our chromosome is a binary string of length $9$, as illustrated in
Fig. 5. The next step is to evaluate the viability of an individual (i.e., a
DNN architecture) based on its ability to compete with the other individuals.
This is known as the Fitness Value, which is subsequently evaluated for each
individual in the population and depends on the cost function $\phi$. The
fitness value is computed either as the cost function in scenarios where the
decoded individual (ARCH) corresponds to a DNN in the existing architecture-
space $\psi$ or is taken to be $0$ otherwise and discarded from the search.
For example, even though the #LSTM cells is encoded using $3$-bits, the gene
only has $5$ possible values. This leads to $3\times 2^{5}$ potential DNN
architectures that are not present in the architecture-space $\psi$, which can
be immediately discarded. Next, through a process called Selection, we pick
two individuals, based on their fitness values, and have them pass their genes
to the next generation. At this point, two important factors come into
picture, namely Crossover and Mutation. For each pair of parents selected for
mating, an ordered crossover occurs with a probability of $0.4$, and a
crossover point in the parents’ chromosomes are selected at random. The
offspring are created by exchanging the genes of parents among themselves from
the start of the chromosome until the crossover point. The new offspring
generated by this crossover mechanism are added to the population of the next
generation. Furthermore, the offspring’s genes can undergo a shuffle-index
mutation with a probability of $0.11$, which would flip the bit, thereby
introducing “diversity” in the population and prevent premature convergence.
An example illustrating the mechanism of crossover and mutation is depicted in
Fig. 6. We determine a population of size $30$, based on the individuals’
fitness values, and re-iterate to produce $5$ generations of offspring to
explore and identify the individuals that offer the best fitness value. Fig. 7
presents a flow chart depicting the stages for our genetic algorithm-based DNN
architecture-space exploration.

Figure 7: Flowchart Depicting the Stages for our Genetic Algorithm-Based DNN
Architecture-Space Exploration. BioNetExplorer evaluates the fitness value of
each individual by training them to determine the cost function $\phi$, if
they are a part of the architecture-space $\psi$, and presetting the others to
zero.

Without loss of generality, in this work, we consider 4 well-known genetic
algorithms for exploring our architectural-space, namely, Roulette Wheel [58],
Tournament Selection [59], NSGA-II [60], and SPEA-2 [61]. The time complexity
of our BioNetExplorer framework depends on the size of the architecture-space
($N$) and the computational complexity of the genetic algorithms used for
exploring the architecture-space. Roulette Wheel has a time complexity of
$O(N*\log N)$ when using binary search in the selection process. Similarly,
Tournament Selection, NSGA-II, and SPEA-2 have a time complexity of $O(N)$,
$O(N^{2})$, and $O(N^{2}*\log N)$, respectively. Since each of these
algorithms select and train different DNN architectures as part of their
exploration, we will also compare the efficiency of these algorithms, as
illustrated in Section V. These algorithms require us to propose a cost
function ($\phi$), which the genetic algorithm can optimize
(minimize/maximize) to obtain a set of near-optimal DNN architectures
($\omega$) for the given exploration. To ensure that the obtained design
offers the best quality ($Q_{DNN}$) with minimum storage overhead ($S_{DNN}$)
or $F_{DNN}$, we propose the following weighted cost functions:

$\phi=\alpha*Q+\beta*\left[1-\frac{S_{DNN}}{S_{MAX}}\right]$ (5)

where $\alpha\in[0,1]$ and $\beta\in[0,1]$ denote the weights for quality
$Q_{DNN}$ and storage overhead $S_{DNN}$, respectively, and $S_{MAX}$ denotes
the maximum storage overhead of the state-of-the-art DNN architecture. Without
loss of generality, in this work, we model the hardware constraint as the
storage overhead of the DNN. Similarly, $F$ and $Energy$ of DNN can also be
modeled and constrained. The system designer can specify the importance of the
two factors by varying the weight parameters ($\alpha,\beta$) and if both are
equally important, then we set both values to be equal. The outcome of this
weighted optimization technique is illustrated in Section V. The Weighted DNN
Architecture Search is depicted in Algorithm 1, where the ExplorationAlgorithm
(line 13) function call employs the search algorithm (exhaustive or one of the
four aforementioned genetic algorithms) under the quality ($Q_{Const}$) and
hardware constraints ($S_{Const}$). An overview of the symbols and denotations
used in our work is provided in Table I.

Algorithm 1 Weighted DNN Architecture Search

1:$\psi,~{}\alpha,~{}\beta,~{}S_{MAX}$

2:$Q_{Const},~{}S_{Const}$

3:$\omega$

4:$S_{DNN}=[];$

5:for ARCH in $\psi$ do

6: if $StorageOverhead(\texttt{ARCH})\leq S_{Const}$ then

7: $S_{DNN}.append(StorageOverhead(\texttt{ARCH}));$

8: else

9: $\psi.remove(\texttt{ARCH});$

10: end if

11:end for

12:$\phi=\alpha*Q_{DNN}+\beta*\left[1-\frac{S_{DNN}}{S_{MAX}}\right]$

13:$\omega=ExplorationAlgorithm(\phi,\psi);$

14:for DNN in $\omega$ do

15: if $Q_{DNN}<Q_{Const}$ then

16: $\omega.remove(\texttt{DNN});$

17: end if

18:end for

TABLE I: Symbols and Denotations Symbol | Denotation
---|---
$Q_{DNN}$ |  Quality of the DNN
$Q_{Const}$ |  Quality Constraint of the Application
$A_{DNN}$ |  Accuracy of the DNN (Eq. 1)
$R_{DNN}$ |  Recall of the DNN (Eq. 2)
$P_{DNN}$ |  Precision of the DNN (Eq. 3)
$F1_{DNN}$ |  F1-score of the DNN (Eq. 4)
$S_{DNN}$ |  Storage Overhead of the DNN
$S_{MAX}$ |  Maximum Storage Overhead of State-of-the-Art DNN Architecture
$S_{Const}$ |  Storage Overhead Constraint of the Target Platform
$F_{DNN}$ |  Number of Floating-point Operations Reqd. for the DNN
$\psi$ |  Set of possible DNN Architectures (Architecture-space)
ARCH |  DNN in $\psi$
$N$ |  Size of the Architecture-space $\psi$
$\phi$ |  Cost Function for the Genetic Algorithms (Eq. 5)
$\omega$ |  Set of Near-optimal DNN Architectures (Output of Genetic Algorithm)
$\alpha$ |  Weight for Output Quality $Q_{DNN}$ in the cost function $\phi$
$\beta$ |  Weight for Hardware Overhead $S_{DNN}$ (or $F_{DNN}$) in the cost function $\phi$

###  III-F Model Compression

Besides the use of architecture-space exploration to identify networks with
similar output quality and low hardware overhead, we include the ability to
further compress the network size using model compression techniques, like
Pruning and Quantization, in our framework, and study their impact on the
quality of bio-signal processing applications.

Pruning: This technique involves eliminating redundant and less important
weights/kernels, or at times layers in the network, to further reduce the
DNN’s hardware overhead, thereby increasing their deployability in wearables.
Furthermore, eliminating weights and network connections results in reducing
the number of floating-point operations ($F$), which also decreases the
execution time and energy required for inference. We retrain the network with
the remaining weights to achieve an output quality similar to that of the
original network obtained from the architecture-space exploration. We
integrate different pruning techniques from [35, 36, 34, 37, 38] in our
framework, from which the system designer can select the appropriate technique
based on DNN design and application requirements. For instance, the techniques
presented in [34] and [37] prune the weights based on their magnitude. The
approach presented by Han et al. [34] eliminates the lowest $x\%$ of the
absolute weights in each layer of the DNN and retrains the network to achieve
the original network accuracy. On the other hand, the technique presented in
[37] is class-blind, i.e., it eliminates $x\%$ of the lowest absolute weights
in the entire DNN, irrespective of which layer the weight corresponds to, and
retrains the network to improve its accuracy. We evaluate the effectiveness of
these two pruning techniques in our case-study as well, see Section V. Note,
other pruning techniques can also be integrated into our framework as long as
they comply with the standard input/output interfaces of pruning.

Quantization: The weights and biases in the network are typically stored as
$32$-bit floating-point numbers in memory, which leads to a large storage
overhead, access latency, and energy consumption. Similarly, the energy
required for performing a $32$-bit floating-point ADD operation is $9\times$
greater than the energy required for a $32$-bit integer ADD operation [62].
Therefore, by limiting the set of possible values through a process known as
quantization, we can reduce the hardware overhead of the DNN parameters from
$32$-bit floating-point values to $16$, $8$, or as illustrated in our work,
even lesser number of bits. The key is to analyze the impact of quantization
on the quality of the resulting DNN for our targeted bio-signal processing
application.

Based on our experiments and previous work [34], the optimal approach to
achieve maximum reduction in hardware overhead of the network involves pruning
the network to eliminate redundant or less important weights, followed by DNN
parameter quantization. This involves the formation of $2^{q}$ (where $q$ is
the number of quantized bits) clusters based on the DNN parameters in each
layer of the DNN using the k-means clustering algorithm. Next, we allocate
equally spaced values to each cluster of weights in the layer, starting with
the minimum value allocated to all zeros up to the maximum value allocated to
all ones. For the purpose of uniformity, we use a uniform number of bits to
quantize all layers (both convolutional and fully-connected) in the network.
Note, different quantization techniques can be integrated into our
BioNetExplorer framework if they comply with the input/output interfaces of
standard quantization techniques.

##  IV Experimental Setup

Figure 8: Overview of Our Experimental Tool-Flow. TABLE II: Hyper-Parameter Values For Training DNNs. Parameter | Values
---|---
Weights Initialized | He et al. [63]
Adam Optimizer [64] | $\beta_{1}=0.9$
$\beta_{2}=0.999$
Dropout | $0.2$
Batch Size | $128$
Learning Rate | $0.001$
TABLE III: Genetic Algorithms’ Parameters & Values. Parameter | Values
---|---
Chromosome Length | $9$
Population Size | $30$
Generation Size | $5$
Crossover Probability | $0.4$
Mutation Probability | $0.11$

Fig. 8 presents an overview of our experimental tool-flow for implementation
and validation of the BioNetExplorer framework. The dataset used in this work
is openly accessible and is made available online by the Physionet data bank
[65]. Our framework uses the MIT-BIH ECG data [66] with $41$ beat and non-beat
annotations to construct new datasets and labels based on the application’s
requirement. Furthermore, to illustrate the benefits of including a unique
output class, we include the CU Ventricular Arrhythmia dataset [67] that
contains instances of ECG recordings with labels of ventricular tachycardia,
ventricular flutter, and ventricular fibrillation. We use the Keras
environment, which includes the Tensorflow machine learning platform, in
Python to implement the DNN architectures that we explore in this work. The
hyper-parameters are crucial for evaluating the accuracy of a DNN
architecture. Therefore, we train the DNN architecture over multiple
iterations for various values of the hyper-parameters in each iteration and
select the hyper-parameter values that offer the best accuracy. The hyper-
parameters that offer the maximum accuracy for these networks are presented in
Table II. The four genetic algorithms that are used in this work are
implemented using the DEAP library [68]. The parameters used by these
algorithms and their values are presented in Table III. The exploration
algorithm is implemented on top of the network training stage using multiple
GPU servers with Core i$9$-$9900$ CPUs integrated with two Nvidia RTX $2080$i
GPUs, each. Furthermore, we enable the early-stopping mechanism during the
network training stage to terminate execution when the loss of the DNN does
not change between two consecutive epochs. The trained networks are then
evaluated based on the number of parameters they require, which determines the
storage overhead and the number of floating-point operations required to
perform a single inference, which, in turn, determines the execution time of
the network on the target platform.

##  V Case-Study: Anomaly Detection in ECG Signals

We illustrate the benefits of using the proposed BioNetExplorer framework with
the help of an ECG signal processing DNN, which is used to classify the
following cases:

  * •

DNN1: Normal and Anomaly Classes;

  * •

DNN2: Normal, Premature Ventricular Contraction (PVC), and Other Anomalies;

  * •

DNN3: Normal, Bundle Branch Block (BB), and
Other Anomalies;

  * •

DNN4: Normal, Atrial Anomaly, Ventricular Anomaly, and Other Anomalies;

  * •

DNN5: Normal, Ventricular Fibrillation, and
Other Anomalies.

###  V-A Effectiveness of State-of-the-Art Neural Architecture Search
Approaches in Bio-Signal Processing

First, we evaluate the effectiveness of two state-of-the-art Neural
Architecture Search approaches for our bio-signal processing DNN architecture-
space, namely,  (i) MnasNet [30] and  (ii) MobileNetV3 [33].  The purpose of
this experimental study is to corroborate our claims that  (i) traditional NAS
approaches designed for image classification applications do not work
effectively for bio-signal processing applications and  (ii) there is a need
and lack of NAS frameworks for bio-signal processing applications, like the
BioNetExplorer framework proposed in this work.  Unlike image classification
applications that have two-dimensional inputs, bio-signal data is typically
present in a 1-dimensional format. Therefore, to accommodate this requirement,
we replaced the traditional 2D normal and depth-wise convolution layers with
1D normal and separable convolution, respectively. We have explored five
variants of the MnasNet discussed in [30] and the small and large variants of
the MobileNetV3 discussed in [33], with various filter sizes ([$3$, $5$, $7$,
$8$, $16$]) and depth multipliers ([$1$, $0.5$, $0.25$]).

The results of deploying these NAS techniques on the DNN1 and DNN4
architecture-spaces are presented in Figs. 9 and 10, respectively. The default
blocks used in the MnasNet framework are more suited for images and/or
2-dimensional data because of which they do not perform well for bio-signal
processing applications, as illustrated in Figs. 9(a) and 10(a). The best
performing networks obtained using the MnasNet framework (labeled A in both
figures) consistently perform worse than the networks obtained by our
BioNetExplorer framework, especially in cases where the number of output
classifications is greater than two ( A in Fig. 10). The networks obtained by
MobileNetV3-NAS (labeled B in both figures) perform much better than MnasNet
for both illustrated cases, which is still non-optimal when compared to the
networks obtained by our framework. The basic ResNet block, adopted from [48]
by our BioNetExplorer framework, is more suited for bio-signal processing
applications as compared to the blocks used in state-of-the-art [30, 31, 32,
33]. BioNetExplorer is successful in obtaining networks with better output
quality for the same or reduced hardware overhead and vice-versa when compared
to other existing approaches.

Figure 9: Applicability of MnasNet and MobileNetV3 on the DNN1 Architecture-
Space. Figure 10: Applicability of MnasNet and MobileNetV3 on the DNN4
Architecture-Space.

The approaches presented in [31] and [32] use a basic block structure that is
replicated in different combinations, for exploring the DNN architecture-
space, which is similar to the block structure and approaches presented in
MnasNet [30] and MobileNetV3 [33]. Therefore, these approaches would also
behave similarly to the state-of-the-art techniques presented in Figs. 9 and
10.

###  V-B Training Time Benefits of Genetic Algorithm-Based DNN Architecture-
Space Exploration

Figure 12: Exhaustive Exploration and Analysis of DNN1 Architectures.

Figure 13: Exhaustive Exploration and Analysis of DNN4 Architectures. Figure
11: Training Time Analysis of Exhaustive, Random, and Genetic Algorithm-Based
Search. The time required for training the random networks stay the same for
all optimization goals. Genetic algorithms reduce the exploration time by
~$9\times$ because they select and train a small-subset of the networks.

Second, we illustrate the benefits obtained in training time of the DNN
architecture-space exploration when a genetic algorithm is used instead of the
exhaustive exploration for all five DNNs. Fig. 11 presents the time required
for selecting and training the Pareto-optimal (Exhaustive Search) or near-
optimal (Genetic Algorithm-Based Search) DNNs for various values of $\alpha$
and $\beta$. Similar to recent NAS studies [69], we also implement a random
search as a potential search strategy. It randomly selects and trains $10\%$
of the architecture-space; see results in Fig. 11. Since random search has
practically very low overhead and a fixed number of architectures to train for
all optimization goals, the time required for training the DNN stays the same
for all scenarios. On average, the use of genetic algorithms reduces the
exploration time by $9.03\times$. Note, exhaustive exploration might not be a
viable option in all scenarios. For example, in applications that require the
use of deeper and complex neural networks, the training time for each network
would lead to an exponential increase in the exploration time while requiring
hundreds of GPU-hours. Since genetic algorithms do not train all the networks
in the architecture-space, the exploration time is limited to tens of GPU
hours. Furthermore, as a back-up strategy, our system supports the use of
random search that has a fixed-time complexity to provide a reasonably good
solution.

###  V-C Exhaustive Exploration and Analysis of Deep Neural Networks from the
DNN1 and DNN4 Architecture-Spaces

Next, we exhaustively explore all the architectures of DNN1 and DNN4, i.e.,
train all the architectures using the constructed datasets. We illustrate the
quality-of-service and memory storage overhead trade-off of the DNN1 and DNN4
architecture-spaces in Figs. 12 and 13, respectively. The Pareto-front
obtained by random search (labeled B in both figures) is similar to the
Pareto-front obtained using exhaustive exploration (labeled A in both figures)
of the architecture-space, primarily due to a large number of inter-dependent
DNN parameters and their effects on the final accuracy. The exhaustive
exploration approach is able to successfully identify networks that trade-off
between output quality and hardware overhead, as illustrated by the figures.
For instance, we have successfully identified a DNN4 network that reduces the
hardware overhead by ~$30$MB for a quality loss of $0.5\%$ (labeled C in Fig.
13). The Precision, Recall, and F1-score for the Anomaly class is lower than
that of the Normal class. Therefore, a quality constraint can be imposed on
any of the four quality metrics based on the application requirement. The main
drawback of this approach is the time required for training and evaluating all
networks in the architecture-space of the DNN.

Figure 14: Exploration of the DNN1 Architecture-Space Using Genetic Algorithms
(Explored Designs are Highlighted).

Figure 15: Exploration of the DNN2 Architecture-Space Using Genetic Algorithms
(Explored Designs are Highlighted).

Figure 16: Exploration of the DNN3 Architecture-Space Using Genetic Algorithms
(Explored Designs are Highlighted).

Figure 17: Exploration of the DNN4 Architecture-Space Using Genetic Algorithms
(Explored Designs are Highlighted).

###  V-D Genetic Algorithm-Based Exploration and Analysis of Deep Neural
Networks from the DNN1, DNN2, DNN3, and DNN4 Architecture-Spaces

As illustrated in Fig. 11, the training time benefits obtained by using
genetic algorithms to search and select points from the architecture-space is
quite significant. We use the four genetic algorithms (see Section III)
instead of exhaustive exploration to search the architecture-space for all
five DNNs described at the start of Section V, using equal weights
($\alpha=0.5,~{}\beta=0.5$) to construct the cost function.

We illustrate a subset of the results obtained from these experiments in Figs.
14, 15, 16, and 17. Genetic Algorithms identify near-optimal designs, without
traversing the entire architecture-space of DNNs, thereby reducing the
exploration time. It is important to note that a design selected by one
algorithm might not be selected by another algorithm (for example, labels A
and B in Figs. 16 and 17, respectively). Therefore, a genetic algorithm should
be selected by the requirements of the application and the system designer.
This requires a preliminary analysis and evaluation of all genetic algorithms
under consideration. It is also interesting to note that NSGA-II and SPEA-2
select and train a small subset of the networks from the architecture-space as
compared to the Roulette Wheel and Tournament Search techniques.

Figure 18: Weighted Architecture Search of DNN3 for the Bundle Branch Class
for Various Values of $\alpha$ and $\beta$.

Figure 19: Weighted Architecture Search of DNN4 for the Ventricular Anomaly
Class for Various Values of $\alpha$ and $\beta$.

Figure 20: Weighted Architecture Search of DNN5 for the Ventricular
Fibrillation Class for Various Values of $\alpha$ and $\beta$.

###  V-E Proposed Weighted Architecture Search and Analysis of Networks from
DNN3, DNN4, and DNN5 Architecture-Spaces

Next, we explore the architecture-space of all DNNs using the proposed
weighted DNN architecture search approach illustrated in Algorithm 1. Although
we have explored the architecture-space of all DNNs using all the genetic
algorithms for various values of $\alpha$ and $\beta$, we illustrate a subset
of these results for DNN3, DNN4, and DNN5 architectures for three cases,
namely,  (i) focus on minimizing storage overhead ($\alpha=0.2$ and
$\beta=0.8$),  (ii) simultaneously optimize for quality and storage overhead
($\alpha=0.5$ and $\beta=0.5$), and  (iii) focus on maximizing quality
($\alpha=0.8$ and $\beta=0.2$),  in Fig. 18, 19, and 20.

The DNN architectures selected by the genetic algorithms clearly illustrate
the effects of the cost function on the architecture-space exploration. For
example, in Case-(i), a large number of points are selected closer to the low-
memory region, because that is the focus of exploration (for example, label A
in Fig. 19). Similarly, in Case-(iii), since the search focuses more on the
accuracy, a large number of points with high accuracy are selected, as
depicted by label B in Fig. 18. A large percentage of the DNN5 architectures
have $0\%$ output quality (label C in Fig. 20) due to a bias against
Ventricular Fibrillation in the number of samples present in the dataset.

###  V-F Results for Pruning and Quantization of Deep Neural Networks from
DNN2, DNN3, and DNN4 Architecture-Spaces

Finally, we evaluate the benefits of applying pruning and quantization
techniques on three different DNN architectures obtained from exploring the
architecture-spaces of DNN2, DNN3, and DNN4. The first DNN architecture (M1),
denoted by [$\alpha=1$], prioritizes the accuracy of the DNN with no
consideration of the hardware overhead of the network. The second network
architecture (M2), depicted by [$\alpha=0.5,~{}\beta=0.5$], is the trade-off
point obtained from the architecture-space of the DNN, which offers the best
output quality for minimal hardware overhead. The third DNN (M3), denoted by
[$\beta=1$], prioritizes the hardware overhead of the network over its
accuracy.

Figure 21: Accuracy and Hardware Overhead Analysis of Pruning Pareto-Optimal
DNN3. Fig. (a) illustrates various levels of pruning in model M1 ($\alpha=1$),
which has the maximum accuracy from the architecture-space; Fig. (b) analyzes
various levels of pruning in model M2 ($\alpha=0.5,~{}\beta=0.5$), which is
the best trade-off between output quality and hardware overhead; Fig. (c)
illustrates various levels of pruning in model M3 ($\beta=1$), which has the
lowest hardware overhead in the architecture-space. Figure 23: Accuracy and
Overhead Analysis of Pruning and Quantizing Pareto-Optimal DNN Architectures
Obtained from the DNN4 Architecture-Space. Figs. (a) to (i) compare the
original DNNs to networks that have been obtained after pruning X% of network
parameters and quantized, where X is increased in steps of $10$, starting from
$10\%$ up to $90\%$. Fig. (j) illustrates the pruned and quantized network for
$95\%$ parameters pruned. [M1: $\alpha=1$; M2: $\alpha=\beta=0.5$; M3:
$\beta=1$]. Figure 22: Accuracy and Hardware Overhead Analysis of Quantizing
Pareto-Optimal DNN2. The models M1 in SF1, M2 in SF2, and M3 in SF3 obtained
from the DNN2 architecture-space are quantized using $6,5,4,3,$ and $2$ bits.
[M1: $\alpha=1$; M2: $\alpha=\beta=0.5$; M3: $\beta=1$]

We illustrate the benefits of pruning various percentages of DNN parameters on
the above-discussed three different networks from the DNN3 architecture-space
using the technique presented in [37] (see Fig. 21). Similarly, we illustrate
the reduction in storage overhead for various levels of quantization for the
networks obtained from the DNN2 architecture-space (see Fig. 22). Finally, we
illustrate the benefits of combining pruning and quantization (discussed in
[34] for image classification) on the networks obtained from the architecture-
space of DNN4 in Fig. 23.

Figure 24: ROC Graphs of Similar DNN4 Architectures. MPNQ depicts a network
with M% DNN parameters pruned and quantized using N-bits. [$\alpha=1$],
[$\alpha=\beta=0.5$], and [$\beta=1$] depict models M1, M2, and M3,
respectively, which have been discussed in the previous sub-section. The ideal
curve illustrates the scenario where all classifications are correct with zero
false-positives.

Pruning, as a standalone technique, can be quite effective in reducing the
hardware overhead by ~$40\%$ while improving the accuracy by up to $0.15\%$ as
illustrated by labels A and D in Fig. 21. The labels B, C, and E denote DNN
architectures that have an output quality similar to the original network
while minimizing the overhead by more than $50\%$. Networks of model M1 can
endure the process of pruning to a significantly larger extent, without loss
of accuracy, as compared to models M2 and M3, primarily due to the higher
number of non-essential parameters in the network. Quantization, is similarly
useful in reducing the hardware overhead of the network by ~$5\times$ compared
to the original DNNs for an output quality loss of $<0.1\%$, as illustrated by
labels A, B, and C in Fig. 22. A combination of pruning and quantization can
be used to achieve a $53\times$ reduction in hardware overhead of the DNN for
a quality loss of ~$0.2\%$, as illustrated by A in Fig. 23. Similarly, the
best networks obtained from models M2 and M3 are depicted by labels B and C,
similar to the output quality of the network A.

###  V-G Receiver Operating Characteristics Analysis of the Pruned and
Quantized DNNs

Receiver operating characteristics (ROC) graphs are useful in visualizing and
evaluating various types of classifiers. They have been widely used in
evaluating the quality of DNNs and ML-based classifiers since the $1990$s
[70]. In this work, we use ROC graphs to analyze the effectiveness of the
DNNs, obtained after pruning and quantization, that have similar output
quality and hardware overhead. For this purpose, we explore the following two
scenarios in each model of the DNN discussed in the previous section:

  1. (1)

Maximize the DNN’s accuracy with a constraint of $0.5$MB maximum hardware
overhead (Fig. 24(a));

  2. (2)

Minimize the hardware overhead of the DNN with a minimum output quality of
$96.7\%$ (Fig. 24(b)).

The results of this analysis are presented in Fig. 24. While model M1 has the
best ROC, M2 and M3 have similar, albeit slightly attenuated, characteristics,
when compared to M1. We can clearly observe that the operating characteristics
of the DNN obtained after pruning $90\%$ of model M1’s parameters and
quantizing them with $3/4$-bits have severely deteriorated. The pruned and
quantized networks obtained from M2 and M3, which have similar output quality
to the pruned and quantized network obtained from M1, exhibit better operating
characteristics, similar to the original uncompressed networks. Therefore,
these compressed DNNs are better suited for deployment in wearables.

To further re-emphasize the efficacy of our framework, we have included
additional results in the supplementary material.

##  VI Conclusion

To summarize and answer the questions raised in Section I:

[A1] We presented the BioNetExplorer framework, which can be used to
systematically explore the architecture-space of Deep Neural Networks for bio-
signal processing applications. Based on the required output classes, user’s
quality requirements, and hardware constraints of the target wearable device,
we perform a genetic algorithm-based exploration of the architecture-space to
identify (near-) optimal DNN architectures that can be deployed on the
wearable and benchmark it against an exhaustive search, while reducing
exploration time by $9\times$.

[A2] We proposed a weighted DNN architecture search algorithm with a modified
cost function to simultaneously search for networks that offer high accuracy
and minimum hardware overhead. Based on these explorations, we have obtained a
wide-range of (near-) optimal DNN architectures for all five cases that have
been explored in Section V.

[A3] We have investigated the applicability of model compression techniques
such as pruning and quantization in our framework to further reduce the
hardware overhead of the DNN with minimal loss in output quality. The
framework is successful in reducing the hardware overhead of the network by
$53\times$ for a quality loss of less than $0.2\%$. Furthermore, we have also
illustrated that DNNs obtained through simultaneous quality and hardware
overhead optimization that have been subsequently pruned and quantized exhibit
better operating characteristics, when compared to DNNs that have been
optimized for quality, making them more suitable for deployment in wearables.

Our framework can also be used to evaluate DNNs for various bio-signal
processing use-cases, besides anomaly detection in ECG signals, such as
seizure detection and prediction using EEG signals, detecting anxiety attacks
in users using a combination of ECG, blood pressure, and heartrate, hypoxia
using SpO2 measurements, etc. The BioNetExplorer framework and the explored
DNN architectures will be open-sourced and available online at
https://bionetexplorer.sourceforge.io to ensure reproducibility.

## Acknowledgement

This work was partially supported by Doctoral College Resilient Embedded
Systems which is run jointly by TU Wien’s Faculty of Informatics and FH-
Technikum Wien.

## References

  * [1] International Data Corporation. (2019) The Growth in Connected IoT Devices. [Online]. Available: https://www.idc.com/getdoc.jsp?containerId=prUS45213219
  * [2] Cisco. (2016) Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2015–2020. [Online]. Available: https://www.cisco.com/c/dam/m/en_in/innovation/enterprise/assets/mobile-white-paper-c11-520862.pdf
  * [3] W. H. Organization _et al._ , “Coronavirus disease 2019 (covid-19): situation report, 72,” 2020.
  * [4] Reuters. (2020) Germany Launches Smartwatch App to Monitor Coronavirus Spread. [Online]. Available: https://reut.rs/3b92buG
  * [5] A. M. Nia, M. Mozaffari-Kermani, S. Sur-Kolay, A. Raghunathan, and N. K. Jha, “Energy-efficient long-term continuous personal health monitoring,” _IEEE Transactions on Multi-Scale Computing Systems_ , vol. 1, no. 2, pp. 85–98, 2015.
  * [6] C. Stergiou, K. E. Psannis, B.-G. Kim, and B. Gupta, “Secure integration of iot and cloud computing,” _Future Generation Computer Systems_ , vol. 78, pp. 964–975, 2018.
  * [7] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen _et al._ , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in _International conference on machine learning_ , 2016, pp. 173–182.
  * [8] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey _et al._ , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _arXiv preprint arXiv:1609.08144_ , 2016.
  * [9] W. Li, X. Zhu, and S. Gong, “Harmonious attention network for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2285–2294.
  * [10] G. Huang, S. Liu, L. Van der Maaten, and K. Q. Weinberger, “Condensenet: An efficient densenet using learned group convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2752–2761.
  * [11] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, “Convolutional sequence to sequence learning,” _arXiv preprint arXiv:1705.03122_ , 2017\.
  * [12] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge computing: The communication perspective,” _IEEE Communications Surveys & Tutorials_, vol. 19, no. 4, pp. 2322–2358, 2017.
  * [13] F. Zhou and Y. Chai, “Near-sensor and in-sensor computing,” _Nature Electronics_ , vol. 3, no. 11, pp. 664–671, 2020.
  * [14] Apple. (2019) Apple Watch Series 5. [Online]. Available: https://www.apple.com/apple-watch-series-5/
  * [15] Samsung. (2019) Galaxy Watch. [Online]. Available: https://www.samsung.com/global/galaxy/galaxy-watch/
  * [16] Fitbit. (2019) Fitness Tracker. [Online]. Available: https://www.fitbit.com/
  * [17] S. Seneviratne, Y. Hu, T. Nguyen, G. Lan, S. Khalifa, K. Thilakarathna, M. Hassan, and A. Seneviratne, “A survey of wearable devices and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 19, no. 4, pp. 2573–2620, 2017.
  * [18] M. P. Turakhia, M. Desai, H. Hedlin, A. Rajmane, N. Talati, T. Ferris, S. Desai, D. Nag, M. Patel, P. Kowey _et al._ , “Rationale and design of a large-scale, app-based study to identify cardiac arrhythmias using a smartwatch: The apple heart study,” _American heart journal_ , vol. 207, pp. 66–75, 2019.
  * [19] Y. Fu and J. Guo, “Blood cholesterol monitoring with smartphone as miniaturized electrochemical analyzer for cardiovascular disease prevention,” _IEEE transactions on biomedical circuits and systems_ , vol. 12, no. 4, pp. 784–790, 2018.
  * [20] D. Sopic, A. Aminifar, A. Aminifar, and D. Atienza, “Real-time event-driven classification technique for early detection and prevention of myocardial infarction on wearable systems,” _IEEE transactions on biomedical circuits and systems_ , vol. 12, no. 5, pp. 982–992, 2018.
  * [21] A. Amirshahi and M. Hashemi, “Ecg classification algorithm based on stdp and r-stdp neural networks for real-time monitoring on ultra low-power personal wearable devices,” _IEEE transactions on biomedical circuits and systems_ , vol. 13, no. 6, pp. 1483–1493, 2019.
  * [22] J. Sun, H. Xiong, X. Liu, Y. Zhang, X. Nie, and R. H. Deng, “Lightweight and privacy-aware fine-grained access control for iot-oriented smart health,” _IEEE Internet of Things Journal_ , 2020.
  * [23] W. Liu, P. Popovski, Y. Li, and B. Vucetic, “Wireless networked control systems with coding-free data transmission for industrial iot,” _IEEE Internet of Things Journal_ , vol. 7, no. 3, pp. 1788–1801, 2019.
  * [24] T. Elsken, J. H. Metzen, F. Hutter _et al._ , “Neural architecture search: A survey.” _J. Mach. Learn. Res._ , vol. 20, no. 55, pp. 1–21, 2019.
  * [25] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” _Advances in neural information processing systems_ , vol. 25, pp. 2951–2959, 2012.
  * [26] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _arXiv preprint arXiv:1611.01578_ , 2016.
  * [27] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 19–34.
  * [28] Y. Liu _et al._ , “Towards flops-constrained face recognition,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [29] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. IEEE, 2009, pp. 248–255.
  * [30] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, “Mnasnet: Platform-aware neural architecture search for mobile,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2820–2828.
  * [31] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 734–10 742.
  * [32] X. Dai, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, Y. Jia _et al._ , “Chamnet: Towards efficient network design through platform-aware model adaptation,” in _Proceedings of the IEEE Conference on computer vision and pattern recognition_ , 2019, pp. 11 398–11 407.
  * [33] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1314–1324.
  * [34] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [35] S. Anwar, K. Hwang, and W. Sung, “Structured pruning of deep convolutional neural networks,” _ACM Journal on Emerging Technologies in Computing Systems (JETC)_ , vol. 13, no. 3, pp. 1–18, 2017.
  * [36] J.-H. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5058–5066.
  * [37] A. Marchisio, M. A. Hanif, M. Martina, and M. Shafique, “Prunet: Class-blind pruning method for deep neural networks,” in _2018 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2018, pp. 1–8.
  * [38] J. Lin, Y. Rao, J. Lu, and J. Zhou, “Runtime neural pruning,” in _Advances in neural information processing systems_ , 2017, pp. 2181–2191.
  * [39] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “Haq: Hardware-aware automated quantization with mixed precision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 8612–8620.
  * [40] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2704–2713.
  * [41] C. Zhu, S. Han, H. Mao, and W. J. Dally, “Trained ternary quantization,” _arXiv preprint arXiv:1612.01064_ , 2016.
  * [42] D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 365–382.
  * [43] S. Jain, S. Venkataramani, V. Srinivasan, J. Choi, P. Chuang, and L. Chang, “Compensated-dnn: energy efficient low-precision deep neural networks by compensating quantization errors,” in _2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)_. IEEE, 2018, pp. 1–6.
  * [44] A. Kumar, S. Goyal, and M. Varma, “Resource-efficient machine learning in 2 kb ram for the internet of things,” in _International Conference on Machine Learning_ , 2017, pp. 1935–1944.
  * [45] A. Gural and B. Murmann, “Memory-optimal direct convolutions for maximizing classification accuracy in embedded applications.” in _ICML_ , 2019, pp. 2515–2524.
  * [46] Y. Xia, H. Zhang, L. Xu, Z. Gao, H. Zhang, H. Liu, and S. Li, “An automatic cardiac arrhythmia classification system with wearable electrocardiogram,” _IEEE Access_ , vol. 6, pp. 16 529–16 538, 2018.
  * [47] Ö. Yıldırım, P. Pławiak, R.-S. Tan, and U. R. Acharya, “Arrhythmia detection using deep convolutional neural network with long duration ecg signals,” _Computers in biology and medicine_ , vol. 102, pp. 411–420, 2018.
  * [48] A. Y. Hannun, P. Rajpurkar, M. Haghpanahi, G. H. Tison, C. Bourn, M. P. Turakhia, and A. Y. Ng, “Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network,” _Nature medicine_ , vol. 25, no. 1, p. 65, 2019.
  * [49] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [50] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _European conference on computer vision_. Springer, 2016, pp. 630–645.
  * [51] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [52] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [53] K. Sastry, D. Goldberg, and G. Kendall, “Genetic algorithms,” in _Search methodologies_. Springer, 2005, pp. 97–125.
  * [54] F. Glover and M. Laguna, “Tabu search,” in _Handbook of combinatorial optimization_. Springer, 1998, pp. 2093–2229.
  * [55] P. J. Van Laarhoven and E. H. Aarts, “Simulated annealing,” in _Simulated annealing: Theory and applications_. Springer, 1987, pp. 7–15.
  * [56] M. Dorigo, M. Birattari, and T. Stutzle, “Ant colony optimization,” _IEEE computational intelligence magazine_ , vol. 1, no. 4, pp. 28–39, 2006\.
  * [57] C. Reeves and J. E. Rowe, _Genetic algorithms: principles and perspectives: a guide to GA theory_. Springer Science & Business Media, 2002, vol. 20.
  * [58] G. Zames, N. Ajlouni, N. Ajlouni, N. Ajlouni, J. Holland, W. Hills, and D. Goldberg, “Genetic algorithms in search, optimization and machine learning,” _Information Technology Journal_ , vol. 3, no. 1, pp. 301–302, 1981.
  * [59] B. L. Miller, D. E. Goldberg _et al._ , “Genetic algorithms, tournament selection, and the effects of noise,” _Complex systems_ , vol. 9, no. 3, pp. 193–212, 1995.
  * [60] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,” _IEEE transactions on evolutionary computation_ , vol. 6, no. 2, pp. 182–197, 2002.
  * [61] E. Zitzler, M. Laumanns, and L. Thiele, “Spea2: Improving the strength pareto evolutionary algorithm,” _TIK-report_ , vol. 103, 2001.
  * [62] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” _Advances in neural information processing systems_ , vol. 28, pp. 1135–1143, 2015.
  * [63] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1026–1034.
  * [64] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [65] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley, “Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,” _circulation_ , vol. 101, no. 23, pp. e215–e220, 2000.
  * [66] G. B. Moody and R. G. Mark, “The impact of the mit-bih arrhythmia database,” _IEEE Engineering in Medicine and Biology Magazine_ , vol. 20, no. 3, pp. 45–50, 2001.
  * [67] F. Nolle, F. Badura, J. Catlett, R. Bowser, and M. Sketch, “Crei-gard, a new concept in computerized arrhythmia monitoring systems,” _Computers in Cardiology_ , vol. 13, pp. 515–518, 1986.
  * [68] F.-M. De Rainville, F.-A. Fortin, M.-A. Gardner, M. Parizeau, and C. Gagné, “Deap: A python framework for evolutionary algorithms,” in _Proceedings of the 14th annual conference companion on Genetic and evolutionary computation_ , 2012, pp. 85–92.
  * [69] L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 367–377.
  * [70] K. A. Spackman, “Signal detection theory: Valuable tools for evaluating inductive learning,” in _Proceedings of the sixth international workshop on Machine learning_. Elsevier, 1989, pp. 160–163.

|  Bharath Srinivas Prabakaran (S’19) is a PhD Student at Doctoral College -
Resilient Embedded Systems, Faculty of Informatics, TU Wien, Austria. He is
working in the Computer Architecture and Robust Energy-Efficient Technologies
(CARE-Tech.) research group, which is led by Prof. Muhammad Shafique. He
graduated with a Bachelor of Engineering in Electrical and Electronics and a
Master of Science in Biological Sciences from the Birla Institute of
Technology and Science (BITS), Pilani, India in 2017. He was as a visiting
researcher at TU Dresden, Germany for a span of 1 year from 2016, where he
completed his master thesis. His research interests include fault-tolerant
computing, wearable architectures, healthcare systems, energy-efficient
technologies, and embedded machine learning.
---|---
|  Asima Akhtar received her Bachelor of Science degree in Electronic
Engineering from University of Engineering and Technology (UET)-Taxila,
Pakistan, in 2016. She is currently completing her Master’s in Electrical
Engineering from School of Electrical Engineering and Computer Science,
National University of Science and Technology (NUST), Pakistan. Her main
research interests are Embedded Machine Learning, Neural Architecture Search
Applications, and Smart Healthcare Systems. She was also a recipient of the
prestigious DAC 2020 Young Student Fellow Program.
---|---
|  Semeen Rehman is currently with the Faculty of Electrical Engineering,
Technische Universität Wien (TU Wien) as a tenure-track Assistant Professor.
In October 2020, she received her habilitation in the area of Embedded Systems
from TU Wien. Before that, she was a Postdoctoral Researcher with Technische
Universität Dresden (TU Dresden) and Karlsruhe Institute of Technology (KIT),
Germany, since 2015. In July 2015, she received her Ph.D. from Karlsruhe
Institute of Technology (KIT), Germany. She has co-authored one book, multiple
book chapters, and more than 50 publications in premier journals and
conferences. Her main research interests include dependable systems, cross-
layer design for error resiliency with a focus on run-time adaptations,
emerging computing paradigms, such as approximate computing, hardware
security, energy-efficient computing, embedded systems, MPSoCs, Internet of
Things, and Cyber Physical Systems. She has received the CODES+ISSS 2011 and
2015 Best Paper Awards, DATE 2017 Best Paper Award Nomination, several HiPEAC
Paper Awards, Richard Newton Young Student Fellow Award at DAC 2015, and
Research Student Award at KIT, in 2012. She has served as the TPC track chair
for the ISVLSI 2020 conference and on the TPC of multiple premier conferences
on design automation and embedded systems (such as DAC, DATE, CASES, ASP-DAC).
She has (co-)chaired sessions at the DATE 2019, 2018, and 2017 conferences.
---|---
|  Osman Hasan received his BEng (Hons) degree from the University of
Engineering and Technology, Peshawar Pakistan in 1997, and the MEng and PhD
degrees from Concordia University, Montreal, Quebec, Canada in 2001 and 2008,
respectively. Before his PhD, he worked as an ASIC Design Engineer from 2001
to 2004 at LSI Logic. He worked as a postdoctoral fellow at the Hardware
Verification Group (HVG) of Concordia University for one year until August
2009. Currently, he is an Associate Professor and the Dean of the School of
Electrical Engineering and Computer Science of National University of Science
and Technology (NUST), Islamabad, Pakistan. He is the founder and director of
System Analysis and Verification (SAVe) Lab at NUST, which mainly focuses on
the design and formal verification of energy, embedded and e-health related
systems. He has received several awards and distinctions, including the
Pakistan’s Higher Education Commission’s Best University Teacher (2010) and
Best Young Researcher Award (2011) and the President’s gold medal for the best
teacher of the University from NUST in 2015. Dr. Hasan is a senior member of
IEEE, member of the ACM, Association for Automated Reasoning (AAR) and the
Pakistan Engineering Council.
---|---
|  Muhammad Shafique (M’11 - SM’16) received the Ph.D. degree in computer
science from the Karlsruhe Institute of Technology (KIT), Germany, in 2011.
Afterwards, he established and led a highly recognized research group at KIT
for several years as well as conducted impactful collaborative R&D activities
across the globe. In Oct.2016, he joined the Institute of Computer Engineering
at the Faculty of Informatics, Technische Universität Wien (TU Wien), Vienna,
Austria as a Full Professor of Computer Architecture and Robust, Energy-
Efficient Technologies. Since Sep.2020, he is with the Division of
Engineering, New York University Abu Dhabi (NYU-AD), United Arab Emirates, and
is a Global Network faculty at the NYU Tandon School of Engineering, USA. His
research interests are in design automation and system level design for brain-
inspired computing, AI & machine learning hardware, wearable healthcare
devices and systems, autonomous systems, energy-efficient systems, robust
computing, hardware security, emerging technologies, FPGAs, MPSoCs, and
embedded systems. His research has a special focus on cross-layer analysis,
modeling, design, and optimization of computing and memory systems. The
researched technologies and tools are deployed in application use cases from
Internet-of-Things (IoT), smart Cyber-Physical Systems (CPS), and ICT for
Development (ICT4D) domains. Dr. Shafique has given several Keynotes, Invited
Talks, and Tutorials, as well as organized many special sessions at premier
venues. He has served as the PC Chair, General Chair, Track Chair, and PC
member for several prestigious IEEE/ACM conferences. Dr. Shafique holds one
U.S. patent has (co-)authored 6 Books, 10+ Book Chapters, and over 300 papers
in premier journals and conferences. He received the 2015 ACM/SIGDA
Outstanding New Faculty Award, AI 2000 Chip Technology Most Influential
Scholar Award in 2020, six gold medals, and several best paper awards and
nominations at prestigious conferences.
---|---