# A Learning Framework for Bandwidth-Efficient Distributed Inference in
Wireless IoT

Mostafa Hussien1,2, Kim Khoa Nguyen2, and Mohamed Cheriet2 1 Resilient
Machine-learning Institute (ReMI)
2 École de technologie supérieure (ÉTS), Univeristy of Québec
Montréal, QC, Canada

###### Abstract

In wireless Internet of things (IoT), the sensors usually have limited
bandwidth and power resources. Therefore, in a distributed setup, each sensor
should compress and quantize the sensed observations before transmitting them
to a fusion center (FC) where a global decision is inferred. Most of the
existing compression techniques and entropy quantizers consider only the
reconstruction fidelity as a metric, which means they decouple the compression
from the sensing goal. In this work, we argue that data compression mechanisms
and entropy quantizers should be co-designed with the sensing goal,
specifically for machine-consumed data. To this end, we propose a novel deep
learning-based framework for compressing and quantizing the observations of
correlated sensors. Instead of maximizing the reconstruction fidelity, our
objective is to compress the sensor observations in a way that maximizes the
accuracy of the inferred decision (i.e., sensing goal) at the FC. Unlike prior
work, we do not impose any assumptions about the observations distribution
which emphasizes the wide applicability of our framework. We also propose a
novel loss function that keeps the model focused on learning complementary
features at each sensor. The results show the superior performance of our
framework compared to other benchmark models.

###### Index Terms:

deep learning, wireless sensor networks, distributed inference, data
compression.

##  I Introduction

Many wireless Internet of things (IoT) applications employ a distributed
inference mechanism e.g., radar systems, surveillance video, or multi-sensory
human activity recognition. In the later system for example, a human wears
multiple, spatially-distributed, sensors (e.g., gyroscope and accelerometer).
A decision about the human activity (e.g., walking, running, etc.) is inferred
from the received sensor signals. Fig. 1 shows this general scenario. In such
a scenario, if each sensor considered only its local observations for
inferring decision, the error probability would be much higher compared to the
scenario in which a global decision is taken based on aggregated sensor data
[1].

Figure 1: Multiple correlated sensors transmitting compressed quantized form,
z, of the sensed data to a FC which applies a decision rule based the
aggregated data.

To tackle this problem, a distributed setup may be employed in which the
sensed data (a.k.a, environment observations) are sent to a central node,
called fusion center (FC). The FC infer a global decision based on the data
aggregated from all sensors. However, the sensors usually have limited power
and bandwidth resources. For example, each sensor may assigned a fixed data
rate of, $R$ bps. Therefore, each sensor should compress and quantize its
observation to fit the assigned bit rate before transmission. The FC, then,
performs a specific inference task (i.e., the sensing goal). Given that the
compression and quantization of raw observations introduce some information
loss, the FC takes the decision based on partial information. This may result
in a reduced decision accuracy at the FC [1]. Optimally processing the
observations at each sensor can minimize the loss in the decision accuracy
[2]. For conditionally-independent sensor observations, an optimal decision
can be easily reached using Bayesian inference theory [3]. However, the
conditional-independence assumption does not hold for many real-life problems.
In prior work [4, 5], the observations distribution is assumed to be known at
the FC. In this case, the goal is to design an optimal decision rule that
maximizes the likelihood of the correct decisions. Unfortunately, in many
practical applications, this distribution is not known in advance which
increases the problem complexity. In this case, data-driven solutions can be
introduced as robust and feasible alternatives.

Although different works in the literature propose compression and
quantization techniques for sensor data, their goal was mainly obtaining a
high-fidelity reconstruction at the FC [6]. This seems relevant for human-
consumed data such as images and videos. However, for machine-consumed data,
adopting the reconstruction fidelity as a metric is doubtful. Indeed, the
accuracy of the automated decisions taken based on the received data is the
most crucial factor.

In this work, we tackle the problem of compressing and quantizing correlated-
sensor observations for distributed inference tasks. Our main objective is
maximizing the accuracy of the inferred decision rather than minimizing the
reconstruction loss. While most of the literature work assume sensors
independence for mathematical tractability, we address the more challenging
scenario of correlated sensors. We argue that this correlation can be
exploited to obtain higher compression ratios without considerable loss in the
decision accuracy. These higher compression ratios can be obtained by
transmitting the unique features of each sensor, and avoid transmitting
redundant features which are likely be transmitted by other nodes in the
network. In other words, we can formulate our research question as: can we
distributively screen redundancies in sensor observations to transmit only
informative data without imposing any assumptions on the observations
distribution?

To answer this question, we exploit the current revolution in statistical
learning techniques, especially deep learning. We propose a novel deep
learning framework for compressing and quantizing the observations at each
sensor. Ina addition, the framework is jointly trained with the decision rule
at the FC in an end-to-end fashion to maximize the accuracy of the inferred
decision. End-to-end learning refers to training a possibly complex learning
system by applying gradient-based learning to the system as a whole [7]. We
also propose a new loss function for the proposed framework that helps the
sensors learn decision-aware representation for the observations. Furthermore,
we propose a training algorithm to efficiently train the proposed framework.
Extensive results show the robustness and superiority of our proposed
framework compared with different benchmark models.

##### Contribution

This paper presents a novel deep learning-based compression framework for
correlated-sensors data. Discrete representation’s autoencoders are adopted at
each sensor to generate the compressed quantized form of the observations. At
the FC, a multi-layer perceptron (MLP) architecture is adopted to jointly
learn the decision rule with the sensor encoders. The main contribution of
this work comes in three folds:

  1. 1.

Extending autoencoders to learn a compressed and quantized representation for
correlated-sensor observations. This learned representation conveys the
complementary features at each sensor observation which help maximizing the
likelihood of the correct decision at the FC while satisfying a communication
constraint. This representation is jointly learned with the decision-rule at
the FC in an end-to-end fashion to maximize the decision accuracy.

  2. 2.

Proposing a novel loss function that encourages the model to learn the unique
features at each sensor. The function learns the soft probabilities of a
baseline model trained to predict $P(Y|X)$. Moreover, we present a training
algorithm that efficiently works in a wide range of applications.

  3. 3.

Eliminating the conditional-independence assumption between sensor
observations which has been widely adopted for mathematical tractability.
Beyond most of the literature work which, for the sake of simplicity, assume
only binary hypothesis problems, we consider the more complicated scenario of
multi-hypothesis problems.

The rest of this paper is organized as follows: Section II formulates the
problem. In Section III, we describe the various elements of the proposed
framework. The discussion and the experimental results are given in Section
IV. Section V briefly summarizes the relevant work in the literature, while
Section VI concludes our work.

##  II Problem Statement

Notation: Through this text, we refer to random variables by capital letters
(e.g. $X$). Small letters refer to one realization of a random variable (e.g.,
$x$). Superscripts denote the sensor number. For example, $x^{i}$ denotes the
observation at sensor $i$. The observations are referred to by $X$ while $Y$
refers to the random variable of the labels (i.e., the target decisions at the
FC). The parameters of the encoder at the $i^{th}$ sensor is referred to as
$\phi_{i}$. The parameters of the decision rule at the FC is referred to by
$\omega$. The $\log(\cdot)$ function uses a base of 2. Table I summarizes the
used symbols and notations.

TABLE I: The notation used through the text.

Symbols |  Description
---|---
$x^{i}$ |  The current observation captured by sensor $i$.
$z^{i}$ |  The compressed and quantized representation for the current observation at sensor $i$.
$y$ |  The true label (or decision) to be predicted at the FC.
$\hat{y}$ |  The predicted label at the FC.
$\phi_{i}$ |  The encoder parameters at sensor $i$.
$\theta$ |  The parameters of the decision function (i.e., decision rule) at the FC.
$\omega$ |  The parameters of the decision function given the raw-observations.
$S$ |  The total number of sensors.
$d$ |  The dimension of the raw observations.
$n$ |  The dimension of the compressed and quantized observations.
$R$ |  The bandwidth (in bps) assigned to each sensor.
$\chi$ |  The observation space, $\mathbb{R}^{d}$.
$Z$ |  The latent space, $\\{0,1\\}^{n}$.
$f_{\phi_{i}}$ |  The encoder function at the $i^{th}$ sensor given by a neural network parameterized by parameters $\phi_{i}$.
$g_{\theta}$ |  The decision rule at the FC given by a neural network parameterized by parameters $\theta$.
$\mathbb{S}^{n}$ |  An n-dimensional vector where each element belongs to the set $\mathbb{S}$.

Suppose $Y$ is a discrete random variable, representing a hypothesis about an
environment. The variable takes values in: $y\in\left\\{1,2,\dots,C\right\\}$
where $C$ is the number of possible hypothesis or classes. Our goal is to form
an estimate, $\hat{Y}$, of the true hypothesis, based on a set of observations
collected from a set of $S$ sensors. Accordingly, for each $t=1,\dots,S$, let
$x^{t}$ represents the observation at node $t$, where $x^{t}\in\mathbb{R}^{d}$
in some space $\chi$ known as the observation space. The set of all
observations correspond to an $S$-dimensional random vector
$X=(x^{1},x^{2},\dotsc,x^{S})\in\chi^{S}$ drawn from the conditional
distribution $P(X|Y)$.

Our objective is to reach an optimal estimate $\hat{Y}$ for the true labels
$Y$ at the FC. If the FC has access to the observations distribution,
$P(X|Y)$, then an optimal decision rule can be easily formulated. For example,
with binary hypothesis, an optimal decision rule can be reached by means of
likelihood ratio test: $P(X|Y=1)/P(X|Y=-1)$. However, in real-world problems,
the FC does not know the distribution of the observation a priori, and it has
access to only summarized forms of the original observations, $z^{t}$, for all
values of $t$. More specifically, we assume that each sensor, $t$, is
restricted to a given bandwidth of, $R$, bps. Therefore, each sensor is
allowed to transmit an n-dimensional message, $z^{t}\in\\{0,1\\}^{n}$, taking
values in some space $Z$, such that $n\leq R$. The conversion from the
observation space, $\chi$, to $Z$-space is carried out by an encoder
$q:\chi\rightarrow Z$. The encoder, $q$, maps an input observation, x, in
$\chi$-space, to a codeword, $z$, in $Z$-space. This encoded observation, $z$,
will be sent to the FC. To compute the estimate $\hat{Y}$, the FC applies a
certain decision rule, $g_{\theta}$, on the aggregated received messages such
that $\hat{Y}=g_{\theta}(z^{1},z^{2},\dots,z^{S})$. It is known from the rate-
distortion theory that the rate, $R$, and the distortion at the receiver (in
terms of reconstruction loss) are inversely proportional. Therefore, larger
rate, $R$, implies better reconstruction fidelity at the receiver end.
However, in our problem we are not concerned about the reconstruction fidelity
as our main objective. Rather, we are more interested in maximizing the
accuracy of the inferred decisions.

Inherently, increasing the rate, $R$, will increase the information included
in a message, $z^{t}$, which increases the FC accuracy. In other words,
increasing the rate, $R$, increases the mutual information, $I$, between the
joint distributions $P(\hat{Y}|Z)$ and $P(\hat{Y}|X)$. However, for limited
bandwidth systems, increasing the bandwidth is not an available option and
each sensor should commit to the assigned bandwidth of $R$ bps. In this case,
for correlated sensor observations, the redundancy between the different
sensor observations can be exploited to obtain more efficient compression with
minimal loss in the decision accuracy at the FC. This can be expressed by the
optimization function given in (1).

$\displaystyle\overset{min}{\phi,\theta}$
$\displaystyle\frac{1}{N}\sum_{j=1}^{N}-\log(g_{\theta}(z_{j})=y_{j})$ (1)
s.t. $\displaystyle
z_{j}=(f_{\phi_{1}}(x_{1}),f_{\phi_{2}}(x_{2}),\dots,f_{\phi_{S}}(x_{S})),$
$\displaystyle f_{\phi_{i}}\in\\{0,1\\}^{n}\;\;\forall i\in\\{1,2,\dots,S\\},$
$\displaystyle n\leq R$

where $N$ is the total number of points in a test set, $g_{\theta}$ is the
decision function at the FC parameterized by parameters $\theta$,
$f_{\phi_{i}}$ is the encoder function at the $i^{th}$ sensor parameterized by
$\phi_{i}$, and $R$ is the bandwidth assigned for each sensor. The same
objective can be formulated in terms of the Kullback–Leibler divergence
between the two conditional distributions of the decision given the raw
observations and the compressed messages as given in (2) and (4).

$\displaystyle\overset{\min}{\omega,\theta,\phi_{i}}$ $\displaystyle
KL\left(P(\hat{Y}|X)\left|\right|P(\hat{Y}|Z)\right)$ (2) s.t. $\displaystyle
P(\hat{Y}|X)=f_{\omega}\left(x_{1},x_{2},\dots,x_{S}\right),$ $\displaystyle
P(\hat{Y}|Z)=g_{\theta}\left(f_{\phi_{1}}(x_{1}),f_{\phi_{2}}(x_{2}),\dots,f_{\phi_{S}}(x_{S})\right),$
$\displaystyle f_{\phi_{i}}\in\\{0,1\\}^{n}\;\;\forall i\in\\{1,2,\dots,S\\},$
$\displaystyle n\leq R$

where $\omega$ is the parameters of a benchmark model (larger neural network
model that trained to classify the original observations without compression).
But the KL-divergence is given by:

$\small KL(P||Q)=\sum_{i}P(i)\log(\frac{P(i)}{Q(i)})$ (3)

Substituting the $KL$ term in (2) by (3), we get Eq. (4).

$\displaystyle\overset{\min}{\omega,\theta,\phi_{i}}$
$\displaystyle\sum_{i}P(\hat{Y}_{i}|X_{i})\log(\frac{P(\hat{Y}_{i}|X_{i})}{P(\hat{Y}_{i}|Z_{i})})$
(4) s.t. $\displaystyle
P(\hat{Y}|X)=f_{\omega}\left(x_{1},x_{2},\dots,x_{S}\right),$ $\displaystyle
P(\hat{Y}|Z)=g_{\theta}\left(f_{\phi_{1}}(x_{1}),f_{\phi_{2}}(x_{2}),\dots,f_{\phi_{S}}(x_{S})\right),$
$\displaystyle f_{\phi_{i}}\in\\{0,1\\}^{n}\;\;\forall i\in\\{1,2,\dots,S\\},$
$\displaystyle n\leq R$

Two main points should be considered. Firstly, the message space
$\left\\{0,1\right\\}$ is significantly smaller than the observation space
$\mathbb{R}$. Secondly, the required dimension for the message, $n$, is
substantially smaller than that of the raw observation, $d$ (i.e., $n\ll d$).
Therefore, the problem can be thought of as finding, for each sensor, $t$, an
optimal encoder and quntizer $q$: $q(x^{t})=z^{t}$ that maximizes the mutual
information between the two distributions $P(Y|X)$ and $P(\hat{Y}|Z)$ under a
certain rate $R$.

##  III Proposed Framework

Figure 2: Diagram demonstrating the system model. In the left, we see the
sensor observations going through the discrete encoders to obtain the
compressed quantized form of the observations. Then these messages are sent to
the FC which passes the aggregated message to the neural network architecture
to get a hypothesis estimation.

###  III-A Autoencoders

Figure 3: The proposed framework for deep distributed inference in wireless
sensor networks.

One of the powerful deep-learning architectures that achieved state-of-the-art
results in different contexts is autoencoder (AE). AE is a neural-network
architecture consisting of two models namely, encoder and decoder. The encoder
model is sometimes referred to as the inference model while the decoder model
is referred to as the generative model. The encoder maps an I-dimensional
input to an O-dimensional codeword, where $O\ll I$. The decoder then
reconstructs the input from this, compressed, codeword. This codeword is
usually referred to as latent representation and it belongs to a space called
latent space. This process is performed in an end-to-end fashion which implies
that the encoder learns to compress the data in a way that help the decoder in
the reconstruction process. If the codeword is quantized (binary or multi-
level), then the architecture is referred to as discrete representation’s
autoencoder. For further details on autoencoder architecture, we refer to [8].

According to the aforementioned problem formulation, our objective is to
jointly learn an optimal encoder and quantizer at each sensor, $q^{t}$:
$q^{t}(x_{i})=z_{i}$, and an optimal decision rule at the FC
$g_{\theta}(z^{1},z^{2},\dots,z^{S})$. To this end, we adopt a discrete-
representation autoencoder at each sensor node to compress and quantize sensor
observations, see Fig. 2. Its worth to differentiate between compression and
quantization in this context. By compression, we mean the mapping from a
higher-dimensional to a lower-dimensional space,
$f:\mathbb{S}^{d}\rightarrow\mathbb{S}^{n}$, where $n\ll d$ and $\mathbb{S}$
is a certain set. On the other hand, quantization is mapping the values of
individual dimensions from a set $\mathbb{S}_{1}$ to a set $\mathbb{S}_{2}$,
where the cardinality of $\mathbb{S}_{1}$ is smaller than that of
$\mathbb{S}_{2}$, (i.e., $|\mathbb{S}_{1}|<|\mathbb{S}_{2}|$).

Each sensor transmits the output of its encoder model to the FC. The output of
the encoder model at sensor $i$ is given by: $f_{\phi_{i}}(\cdot)$. At the FC,
an MLP neural network parameterized by parameters, $\theta$, is used to
approximate the optimal decision rule, see Fig. 3. The decision rule at the FC
is given by:

$g_{\theta}([f_{\phi_{1}}(x^{1}),f_{\phi_{2}}(x^{2}),\dots,f_{\phi_{S}}(x^{S})]).$
(5)

where $x^{i}$ is the current observation at the $i^{th}$ sensor.

###  III-B Implementation Details

The encoder architecture at each sensor is an MLP of three fully-connected
layers with ReLU activations. In the output layer of the encoder, a QSigmoid
activation is used [9]. In the FC, we used six fully connected layers with
ReLU activations in the hidden layers and Softmax activation in the output
layer. The model weights are initialized using He initializer [10]. The models
are trained using Adam optimizer [11], with (0.01) learning-rate and optimized
to minimize our proposed loss function given in Eq. (8). Due to the adopted
end-to-end training, the encoders will learn to encode the unique information
at each sensor that help the FC to infer the correct decisions. Furthermore,
the FC model optimizes its weights to maximize the likelihood of the correct
decision given the encoded observations. So, we can interpret the optimization
of the classifier weights at the FC as learning an optimized threshold
function for the decision rule.

###  III-C Training Procedure

Input: Dataset $D$, consisting of $N$ tuples of observations acquisted from
$S$ sensors and the corresponding label $y$.

Output: Model parameters, $\theta$, and $\phi_{i}$ for $i\in\\{1,2,\dots,S\\}$

  * -

At each sensor, $s_{i}$, train an autoencoder to reconstruct its input using
observations in D;

* -

Train an inference model, $I_{1}$, to approximate the conditional distribution
$p(\hat{Y}|X)$;

* -

Freeze the weights of $I_{1}$, known as $\omega$;

* -

Train an inference model, $I_{2}$, (jointly with the encoders weights,
$\phi_{i}$ for $i\in\\{1,2,\dots,S\\}$) to approximate the conditional
distribution $p(\hat{Y}|Z)$;

* -

Return the learned parameters of $I_{2}$ (i.e., $\omega$), along with
$\phi_{i}$ for $i\in\\{1,2,\dots,S\\}$.

Algorithm 1 The training procedure for the proposed framework, $S$, sensors.

The training of the proposed framework comes in three phases. In the first
phase, we train an autoencoder at each sensor. The autoencoders are trained
for input reconstruction from a compressed codewords by minimizing l2-norm
function given in Eq. (6).

$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}{\left\|x_{i}-\hat{x}_{i}\right\|}^{2}$
(6)

In the second phase, we, independently, train an inference model, $I_{1}$,
that takes as input the raw observations, X = [$x_{1},x_{2},\dots,x_{S}$], and
outputs the corresponding decision. Note that the inputs to this model are the
raw observations without compression or quantization. The model is trained to
optimize the classical Crossentropy function (7).

$f(y,\hat{y})=-\sum_{i=1}^{C}y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i}),\vspace{-0.1cm}$
(7)

where $y_{i}$ is the target label vector of the $i^{th}$ data point and $C$ is
the number of classes. The output of the model $I_{1}$ approximates the
conditional distribution $P(\hat{Y}|X)$. The set of parameters in the model
$I_{1}$, denoted by $\omega$, is then frozen and its will be used only for
computing the loss value of the inference model at the FC. This model
represent the baseline model that we aim to approximate after the compression
and the quantization take place. We elaborate more on this part in subsection
III-D.

In the third and last phase, we use the encoder model (of the autoencoders
trained in the first phase) at each sensor to compress the observation
captured by this sensor. The output of the encoder at sensor $i$ is denoted by
$z_{i}$. The parameters of an encoder model of the autoencoder at sensor, $i$,
denoted by $\phi_{i}$. Accordingly, $z^{i}=f_{\phi_{i}}(x)$. The output of all
the encoders are concatenated and fed to an inference model, $I_{2}$, with
parameters $\theta$ to predict the output. In this case, the output of $I_{2}$
approximates the conditional distribution $P(\hat{Y}|Z)$. Its worth noting
that the weights of $I_{2}$, $\theta$, are trained jointly with the encoders
weights, $\phi_{i}$, at each sensor. This means that the training in the last
phase is done in an end-to-end fashion between the encoders weights and the
parameters of decision rule at the FC. Algorithm 1 summarizes the training
procedure.

Randomly select two random points $x_{1}$, $x_{2}$;

Encode each data point using the encoder function, $f_{\phi}$:

$z_{1},z_{2}=f_{\phi}(x_{1}),f_{\phi}(x_{2})$;

$h=z_{1}\oplus z_{2}$;

i=0;

while  _$i <len(h)$_ do

if  _h[i] equals 1_ then

Flip the bit at $z_{1}$[i];

i = i + 1;

end if

$\bar{x_{1i}}=f_{\theta}(z_{1})$; where $f_{\theta}$ is the decoder function.

Plot $\bar{x_{1i}}$;

end while

Algorithm 2 The procedure for evaluating the semantics of the latent
codewords.

###  III-D Proposed Loss Function

In the first and second phases of the training, we optimize the MSE and
Crossentropy loss functions, respectively. However, in the third phase of the
training in which we jointly train the sensors encoders and the inference-
model at the FC, minimizing the traditional Crossentropy is found to be
limited in solving the problem being addressed. Recall from previous section
that the objective of the proposed framework is to make the encoders benefit
from the redundancies (between the sensor observations) to obtain high
compression ratios without harming the decision accuracy (i.e., the sensing
goal). This implies that encoders should learn to encode the complementary
features of its observation. To this end, we propose a novel loss function
(8).

$\mathcal{L}(Y,\hat{Y})=CE(Y,\hat{Y})+KL(P(\hat{Y}|X)||P(\hat{Y}|Z))$ (8)

The proposed function helps the model to learn a joint conditional
distribution for the decision given the compressed observations,
$P(\hat{Y}|Z)$, which is as similar as possible to the joint conditional
distribution for the label given the uncompressed observations,
$P(\hat{Y}|X$). This term reduces the loss in the decision accuracy due to the
compression of the sensor observations. Given a limited budget of data bits to
encode the observations, we argue the proposed function encourages the
encoders to encode only the relevant features that help in maximizing the
likelihood of the correct decision at the FC.

Note that we handcrafted a model for each dataset to achieve the highest
possible accuracy. The models have been selected according to the proposed
loss function, Eq. (8), such that it emphasizes on learning complementary
features at each sensor. To this end, the second term in Eq. (8) adds a
regularization term based on the KL-divergence between the conditional
probability distribution of the decision given the full observation
$P(\hat{Y}|X))$, and the conditional probability distribution of the decision
given the compressed and quantized version of the observations
$P(\hat{Y}|Z))$. Moreover, our model jointly learns a quantizer function
(entropy encoder) along with the source encoder. Jointly learning the encoders
with the decision rule encourages the model to learn only the complementary
features at each sensor. The proposed models work well with each problem
without overwhelming the framework with complex architectures such as AlexNet,
ResNet, GoogleNet, etc [12]. The power of these complex models is required
mainly for high-dimensional observation space, such as surveillance cameras.
In this case, the hidden (deep) convolutional layers can extract spatial
features in the observations in an efficient way. However, in the problems
with lower-dimensional observation space such as our case, handcrafted models
will outperform these models in terms of accuracy and complexity. This
conclusion is compatible with the findings reported in [13].

###  III-E Dataset Preparation

The proposed framework is general and widely applicable in different problems.
For the framework to be employed in a certain distributed inference task, a
dataset should be prepared for training. A typical dataset consists of $N$
data points along with the associated labels $\\{x_{i},y_{i}\\}_{i=1}^{N}$.
Each data point, $x_{i}$, represents the concatenation of simultaneous
readings from $S$ sensors such that
$x_{i}=[x^{1}_{i},x^{2}_{i},\dots,x^{S}_{i}]$. The label
$y_{i}\in\\{1,2,\dots,C\\}$ is the one-hot encoded vector of the target
hypothesis (i.e., class) associated with these sensor readings. It is worth to
note that these readings are assumed to be perfectly synchronized and each
data point represents the readings at the same time step.

##  IV Results and Discussion

We show the results of testing the proposed framework using different
datasets. Each dataset represents different environment setting and generating
distribution. This section is organized to three subsections. The accuracy of
distributed detection problem is presented in subsection IV-A. The semantics
of the latent representations is presented in subsection IV-B. The last
subsection explores the efficiency of the codewords generated at each sensor
in input reconstruction.

###  IV-A Distributed Inference Accuracy

####  IV-A1 Comparative Evaluation

TABLE II: The classification accuracy of the proposed framework under
different compression ratios compared with different work from the literature
on WARD dataset.

Method | CR=2 | CR=4 | CR=8
---|---|---|---
Cheng et al. (ASRCM) [14] | 94% | 88% | 83%
Cheng et al. (NN) [14] | 82% | 78% | 75%
Zhang et al. [15] | 87% | 83% | 80%
Our Framework | 99.7% | 97.4% | 95.6%

TABLE III: The classification accuracy of the proposed framework at
compression ratio (CR=2) compared with different work from the literature on
WARD dataset.

Method | Detection Accuracy
---|---
Zhu et al. [3] | 99.00%
Yang et al. [16] | 93.60%
Huynh [17] | 96.97%
He et al. + PCA [18] | 76.31%
He et al. + LDA [18] | 40.30%
He et al. + GDA [18] | 99.20%
Guo (Majority voting) [19] | 94.96%
Guo (Maximum) [19] | 96.20%
Guo (WLOP) [19] | 98.02%
Guo (WLOGP) [19] | 98.78%
Sheng et al. [20] | 95.90%
Oniga and Jozef [21] | 98.10 %
Our Framework |  99.7%

To evaluate the effectiveness of the proposed framework, we used a publicly
available dataset called Wearable Action Recognition Database (WARD) presented
in [16]. The obtained performance is compared against three other literature
works applied to the same dataset. The dataset is designed for human activity
recognition from sensors data. This dataset is collected from five sensor-
boards attached to different points in the human body. Each sensor board has a
tri-axial accelerometer and a bio-axial gyroscope with three and two
dimensional outputs respectively. Each human operator performs 13 different
actions which represent the labels (classes) predicted by the classifier at
the FC.

Figure 4: Comparison of model accuracy under different compression ratios.

Table II and Fig. 4 show a comparison between the performance of the proposed
framework and other works in literature under different compression ratios. We
can see from the table that the performance of our framework outperforms other
works under all compression ratios. We can see that our framework preserves
high accuracy even under high compression ratios. For example, increasing the
compression ratio from 2 to 8 decreased the accuracy only 4.1% from 99.7% to
95.6%. This is a small margin compared with 11% loss in Cheng et al. (ASRCM)
[14], and 7% in Cheng et al. (NN) [14] and Zhang et al. [15].

Table III shows the classification accuracy of the proposed framework compared
with the accuracy of other literature works. The table reports results for Zhu
et al. [3], Yang et al. [16], Huynh [17], He et al. [18], Guo et al. [19],
Oniga et al. [21], and Sheng et al. [20]. It is clear from the table that the
proposed framework achieves state-of-the-art accuracy compared with other work
in the literature. In addition, the proposed framework involves the minimal
required bit rate, $R$, from the sensor nodes to the FC, which highly
contributes to power saving and prolonging sensors lifetime. These results can
be attributed to the fact that we learn complementary features between
correlated sensors that highly contribute to improving the decision accuracy
rather than learning local features for each sensor. This learning behaviour
is motivated by the proposed loss function. Moreover, our framework jointly
learns a quantizer function $q:\chi\rightarrow Z$ with the encoder function
which minimizes the end-to-end error and improves the sensing task accuracy.
Note that the work in [3] explores the correlation between the sensor
observations to disable the transmission on the sensors that did not capture
new relevant features and thus save the consumed bandwidth. Comparatively, in
our work, we exploit this correlation to transmit only the relevant
complementary features. Consequently, we contribute in two directions, namely,
saving the consumed bandwidth and, at the same time, improving the decision
accuracy.

####  IV-A2 Artificial Problem

Figure 5: The decision accuracy of the proposed framework with four datasets
under three different compression ratios. The baseline model represents the
case in which we fuse the raw-observations to the FC without compression.

We tested the proposed framework with four datasets, which are: 1) MNIST [22],
2) Fashion-MNIST [23], 3) Street View Houses (SVH) [24], 4) CIFAR-10 [25].
These datasets represent different environments and generating distributions.
For each dataset, we used different Compression Ratios, $CR$. $CR$ is defined
as the ratio between the uncompressed dimension and compressed dimension [26].
It is worth to note that the compression ratios of the literature work
consider only compression by dimensionality reduction (i.e., any input or
output dimension $\in\mathbb{R}$). Based on that, the input and output space
remains the same. Unlike prior methods, we go beyond to counts for the
quantization (since an input dimension is $\in\mathbb{R}$ while an output
dimension is quantized $\in\\{0,1\\}$). In these experiments, we simulate two
sensors ($s_{1}$,$s_{2}$) sending their data to a FC. Assume the observations
at sensor $s_{1}$ belongs to a class $C_{i}$ and at sensor $s_{2}$ belongs to
a class $C_{j}$. The decision rule at the FC can be defined as:

$\psi(z^{1},z^{2})=\left\\{\begin{matrix}i&ifi=j\\\ -1&ifi\neq
j\end{matrix}\right.$ (9)

In other words, the decision will be the class label if the two observations
belong to the same label, and -1 otherwise. Since each dataset consists of
images belonging to one out of 10 total classes, we expect the classifier to
have 11 classes.

In order to make fair comparison between the framework accuracies with
different compression ratios, we used the same classifier capacity (in terms
of number of layers, the nodes in each layer, the activation functions used,
etc.) for each dataset. We compared the obtained results with the baseline
model accuracy. The baseline model is defined as the accuracy of a neural
network classifier taking as input the raw observations without compression or
quantization, $x^{t}$. In this case, the FC has the complete vector of sensed
data, which represents the optimal case in terms of the data availability at
the FC.

Fig. 5 shows the obtained results in each case. We can notice that the
framework performance approaches the baseline with the lowest compression
ratio, $CR=2$ in the table. A small loss in the accuracy is reported with
higher $CR$ (i.e., $CR=4$ and 8). However, the obtained accuracy is still high
even with the highest $CR$. For example, we obtained 95.3% of the baseline
with $CR=8$ in MNIST dataset. Which means compressing the observations to just
12.5% of its original dimension with quantization, results in 4.7% reduction
in accuracy.

Figure 6: A heatmap representation for the confusion matrix of an MNIST
classifier with 98-dimension latent code corresponding to compression ratio of
eight. The class label F here represent class label -1.

In the reconstruction of the training dataset, we randomly shuffle the
datasets in each sensor. Consequently, most of the observations combinations
fall in the class of -1 which means the two observations are not in the same
class. This produces an imbalanced class distributions. Due to this imbalance,
we report the confusion matrix of the framework classifier for MNIST dataset
and 98-dimension latent code. Fig. 6 shows that the model is capable of taking
the right decision with high accuracy even with imbalanced data.

The key idea of compressing correlated sensors data is extracting
complementary information from correlated observations and ignoring any
redundancies. Our proposed loss function (Eq 8) achieved this goal by
incorporating a KL divergence term to the loss function between the soft
labels generated by a baseline model (e.g., a large model trained on raw
observations to predict $P(Y|X)$) and the decision function at the FC
($P(Y|Z)$). To minimize this term, we encode only the complementary features
from each sensor which the help of the FC that behaves as the baseline model.
As described in Algorithm. 1, we jointly train the encoder models at each
sensor with the decision function at the FC in an end-to-end fashion. This
end-to-end training makes the encoders jointly learn these features with the
decision function as they receive penalization based on the distance between
the predicted distribution and that of the baseline model.

Figure 7: Interpolation between two points in the latent space. We choose a
start and end points, then we gradually flip a bit each time along the
different bits between the two vectors. The starting point shown in the top-
left corner, the end point in the bottom-right.

###  IV-B Semantics of the Latent Representation

Figure 8: Reconstruction of MNIST data images using a 98-dimensional discrete
latent code. The top row shows the original input, while the reconstruction is
shown in the bottom row.

In autoencoders-based architectures for dimensionality reduction, a special
interest is paid to the robustness of the learned codewords in the latent
space [27]. To evaluate the robustness of such codewords, we interpolate
between different points in the latent space and observe, qualitatively, the
gradual changes in the reconstructed data. This widely used experiment
verifies that the model: (a) has injected enough redundancies into the
codewords and consequently the model is capable for reconstructing the input
even in the presence of errors in the codeword, (b) has learned relevant
features of the underlying structure of the data.

We randomly select two test points to represent the start and end points. Each
step, we flip a bit in the latent codeword, fed the new obtained codeword to
the decoder model and observe the gradual changes in the reconstruction.
Algorithm 2, describes this experiment in more details. Fig. 7 shows the
gradual transition in the digit shape with the gradual bit flipping. We can
observe that decrementing the hamming distance between the start and end
points, each bit-flip, slowly alters the characteristic features of digit
until the digit reaches the end point.

###  IV-C Rate/Computation Tradeoff

Figure 9: Tradeoff between model accuracy (achieved at different compression
ratios) and the consequent increase of computation requirement.

Although observation compression reduces the consumed bandwidth for
transmission, this reduction comes with the cost of computation and accuracy.
The required computation resources (measured by the floating-point operations
(FLOPS)) increases according to the model complexity (measured by the number
of weights). Moreover, increasing the model complexity leads to improved
compression, and consequently improved decision accuracy at the FC. Therefore,
a design decision should compromise between the model complexity on the one
hand, and the consumed computations and FC accuracy on the other hand.
However, the training phase can be done off-line (before the deployment of the
sensors), and only the inference will take place during the operation which
requires only one forward pass (a very small number of FLOPS) to predict the
encoded messages. Fig. 9 shows this trade-off trend between the computation
requirement (measured by FLOPS) and the model accuracy. In this figure, we can
see that increasing the model accuracy requires adopting smaller compression
ratios which implies higher data transmission. On the other hand, a smaller
compression ratio requires transmitting more data and requires more
computational resources at each sensor. The optimization of compression ratios
is out of the scope of this paper, and will be explored in our future work.

###  IV-D Applicability

Figure 10: The decision accuracy for a wireless link adaptation problem under
different compression ratios. The results confirm the general applicability of
the proposed system to problem from different domains.

Our proposed framework along with the proposed loss function and training
procedure shown in Algorithm. 1 can work with any type of parallel distributed
detection network. This type of settings has various applications in wireless
IoT. Although minor customizations are required to fit each specific problem,
the framework still widely applicable to various problems from various
domains. In this paper, we reported the experimental results on various types
of sensors and applications (e.g., image classification, human activity
recognition, etc.). Specifically, we experimented 5 different datasets (MNSIT,
Fashion MNIST, SVHN, CIFAR-10, WARD) representing three different types of
sensors (cameras, gyroscope, and accelerometer). To verify the generality of
our framework, we tested the proposed framework in a completely different
domain, i.e., wireless link adaptation using three datasets combined in a
global dataset [28]. In this scenario, the sensors are the antennas at each
mobile node, the observations are the channel state information (CSI) captured
at each mobile, and the environment that the sensors monitoring is the
wireless channel [29]. The sensors send their observations to a FC to take a
global decision. The base station (BS) acts as a FC in this case, and the
decision is the selected modulation and coding scheme. The results shown in
Fig. 10 show minor loss in the adaptation decision at the FC with the increase
in the adopted compression ratio. For example, when compressing the original
raw observations (i.e., CSI in this case), the accuracy only drops from 94.25%
to 93.7%. This means only 0.55% loss in accuracy is achieved while saving 75%
of the original bandwidth. The obtained results confirm the general
applicability of our proposed method in different domains and problems.

###  IV-E Results of Input Reconstruction

To further evaluate the robustness of the learned features, we evaluated the
proposed model in input reconstruction task. In this experiment, MNIST and
Fashion-MNIST datasets are used for evaluation. We used a $CR$ equals to eight
which corresponds to a latent code of 98-bit. Fig. 8 shows the result of the
input reconstruction.

##  V Related Work

In this section, we review prior work that is related to various aspects of
this paper.

A similar work has been proposed for specific problems. For example, a line of
work has been proposed for the problem of human activity recognition [30, 19,
17, 18]. In this problem, the hypotheses are the different human actions,
while the data comes from multiple sensors fixed on the actor body (e.g.,
gyroscope, accelerometer, etc.). This work focuses on compressing the sensors
data without hurting the recognition accuracy. For example, in [30], the
authors aimed to achieve a high action-classification accuracy with the
minimum bandwidth consumption. At each sensor, the decision is taken based on
its local information. The FC then takes a global decision using a majority-
voting mechanism. Although they obtained a good results, this approach ignores
any complementary information captured by other sensors. Another work has been
proposed for the problem of earthquakes detection from wireless IoT sensors
network [31]. They presented a distributed approach for rapid detection of
earthquakes using cell phone accelerometers, consumer USB devices, and cloud
computing based sensor fusion. The approach proposed in [31] learns a
threshold for each sensor involved in the network in a way that maximizes the
performance of the anomaly detection algorithm employed at the FC.
Experimental results showed that the proposed approach successfully
distinguished between seismic motion from accelerations due to normal daily
manipulation.

The work in [32] studied the problem of binary hypothesis testing with two
observers, where the collected observations are assumed to be statistically
correlated. To reach a decision, one of three solutions can be adopted. The
first is a centralized solution in which the observations collected by both
observers are sent to the FC. A global decision is taken at the FC based on
the received sensor observations. The main concern of this solution is the
huge bandwidth incurred in observation transmission. The second solution makes
each observer rely on its own locally collected observation. Then, each node
exchanges its locally obtained decision with other sensors to reach a global
decision. The main limitation of this solution is that each sensor depends
only on its local information and ignores any complementary information
captured by other sensors. In the last solution, each observer formulates the
problem as a sequential hypothesis-testing problem. The authors in [33]
proposed a framework for exploiting the correlation between observations to
reduce the mean square error of the distributed estimation. Specifically, each
node predicts its next observation and transmits the quantized prediction
errors (innovations) to the FC instead of the quantized observations.

In the context of task-aware compression, a similar problem has been addressed
in [34, 35, 36]. For example, the authors in [34] used a reinforcement agent
at each sensor node to compress the observations before feeding them to the
FC. The reward function at each agent considers its commitment to the assigned
bandwidth. Although they achieved a good performance, there is a probability
that the agent does not meet the bandwidth constraints after deployment. While
in [35], the authors proposed Starafish, an image compression framework that
outperforms JPEG by by up to 3X in terms of bandwidth consumption and up to
2.5X in power consumption. The authors in [35] used an AutoML technique to
search for tiny ML models that can work on power AIoT accelerators.

We can summarize the limitations of the literature work, which we addressed in
our work, as: 1) the conditional-independence assumption of the sensor
observations does not usually hold; 2) the conditional-independence assumption
ignores the potential opportunity to benefit from complementary features
captured by different sensors; 3) the compression algorithms are designed
independently from the sensing goal; 4) the limited power of analytical-based
techniques in dealing with large number of possible decisions and correlated
sensors.

##  VI Conclusion

In this paper, we proposed a deep learning framework for compressing
correlated sensor observations in distributed inference problems. The proposed
framework employ discrete representation autoencoders to encode the
observations at each sensor. A novel loss function is proposed to improve the
accuracy of the framework. A multi layer perceptron architecture has been used
at the FC to jointly-learn the decision rule. The proposed framework addresses
the hard to tackle problem of correlated sensor observations and does not
assume any prior knowledge about the distribution of these observations. The
performance of the model has been extensively verified using different
datasets and proved to provide significant performance improvement.

## References

  * [1] S. Salehkalaibar and et al., “On Hypothesis Testing Against Conditional Independence with Multiple Decision Centers,” _IEEE Tran. on Communications_ , vol. 66, no. 6, pp. 2409–2420, 2018.
  * [2] Y. Abdi and T. Ristaniemi, “The Max-Product Algorithm Viewed as Linear Data-Fusion: A Distributed Detection Scenario,” _IEEE Tran. on Wireless Communications_ , vol. 19, no. 11, pp. 7585–7597, 2020.
  * [3] P. Zhu and et al., “Cost Aware Inference for IoT Devices,” in _International Conference on Artificial Intelligence and Statistics_ , 2019\.
  * [4] J.-F. Chamberland and V. V. Veeravalli, “Wireless Sensors in Distributed Detection Applications,” _IEEE Signal Processing Magazine_ , vol. 24, no. 3, pp. 16–25, 2007.
  * [5] W. P. Tay and et al., “Bayesian Detection in Bounded Height Tree Networks,” _IEEE Tran. on Signal Processing_ , vol. 57, no. 10, pp. 4042–4051, 2009\.
  * [6] M. e. a. Hussien, “Fault-Tolerant 1-bit Representation for Distributed Inference Tasks in Wireless IoT,” in _International Conference on Network and Service Management (CNSM)_. IEEE, 2021, pp. 427–431.
  * [7] T. Glasmachers, “Limits of End-to-End Learning,” in _Asian Conference on Machine Learning (ACML)_. PMLR, 2017, pp. 17–32.
  * [8] A. Majumdar, “Blind Denoising Autoencoder,” _IEEE Tran. on Neural Networks and Learning Systems_ , vol. 30, no. 1, pp. 312–317, 2018.
  * [9] B. Moons and et al., “Minimum Energy Quantized Neural Networks,” in _Asilomar Conference on Signals, Systems, and Computers_. IEEE, 2017.
  * [10] K. He and et al., “Delving Deep into Rectifiers: Surpassing Human-level Performance on ImageNet Classification,” in _International Conference on Computer Vision (ICCV)_. IEEE, 2015\.
  * [11] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [12] Y. LeCun and et al., “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [13] J. Suto and S. Oniga, “Efficiency Investigation from Shallow to Deep Neural Network Techniques in Human Activity Recognition,” _Cognitive Systems Research_ , vol. 54, pp. 37–49, 2019.
  * [14] L. Cheng and et al., “Accelerated Sparse Representation for Human Activity Recognition,” in _International Conference on Information Reuse and Integration_. IEEE, 2017.
  * [15] M. Zhang and A. A. Sawchuk, “Human daily activity recognition with sparse representation using wearable sensors,” _Journal of Biomedical and Health Informatics_ , vol. 17, no. 3, pp. 553–560, 2013.
  * [16] A. Y. Yang and et al., “Distributed Recognition of Human Actions using Wearable Motion Sensor Networks,” _Journal of Ambient Intelligence and Smart Environments_ , vol. 1, no. 2, pp. 103–115, 2009.
  * [17] D. T. G. Huynh, “Human Activity Recognition with Wearable Sensors,” Ph.D. dissertation, Technische Universität, 2008.
  * [18] W. He and et al., “Recognition of Human Activities with Wearable Sensors,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2012, no. 1, p. 108, 2012.
  * [19] Y. Guo and et al., “Human Activity Recognition by Fusing Multiple Sensor Nodes in the Wearable Sensor Systems,” _Journal of Mechanics in Medicine and Biology_ , vol. 12, no. 05, p. 1250084, 2012.
  * [20] M. Sheng and et al., “Short-time Activity Recognition with Wearable Sensors using Convolutional Neural Network,” in _ACM SIGGRAPH Conference on Virtual Reality Continuum and its Applications in Industry_ , 2016, pp. 413–416.
  * [21] S. Oniga and S. Jozsef, “Optimal Recognition Method of Human Activities using Artificial Neural Networks,” _Measurement Science Review_ , vol. 15, no. 6, p. 323, 2015.
  * [22] Y. LeCun. (1998) The mnist database of handwritten digits. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [23] H. Xiao and et al., “Fashion-MNIST: A Novel Image Dataset For Benchmarking Machine Learning Algorithms,” _arXiv preprint arXiv:1708.07747_ , 2017.
  * [24] Y. Netzer and et al., “Reading Digits in Natural Images with Unsupervised Feature Learning,” 2011.
  * [25] A. Krizhevsky and et al., “Learning Multiple Layers of Features From Tiny Images,” 2009.
  * [26] K. Sayood, _Introduction to Data Compression_. Morgan Kaufmann, 2017.
  * [27] K. Choi and et al., “Neural Joint Source-channel Coding,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [28] M. Hussien and et al., “Towards More Reliable Deep Learning-based Link Adaptation for WiFi 6,” in _International Conference on Communications (ICC)_. IEEE, 2021, pp. 1–6.
  * [29] ——, “PRVNet: Variational Autoencoders for Massive MIMO CSI Feedback,” _arXiv preprint arXiv:2011.04178_ , 2020.
  * [30] A. Y. Yang and et al., “Distributed Segmentation and Classification of Human Actions using a Wearable Motion Sensor Network,” in _Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2008.
  * [31] M. Faulkner and et al., “The next big one: Detecting earthquakes and other rare events from community-based sensors,” in _ACM/IEEE International Conference on Information Processing in Sensor Networks_. IEEE, 2011.
  * [32] A. Raghavan and J. S. Baras, “Binary Hypothesis Testing By Two Collaborating Observers: A Fresh Look,” in _Mediterranean Conference on Control and Automation (MED)_. IEEE, 2019.
  * [33] T. Bouchoucha and et al., “Distributed Estimation Based on Observations Prediction in Wireless Sensor Networks,” _Signal Processing Letters_ , vol. 22, no. 10, pp. 1530–1533, 2015.
  * [34] S. P. Chinchali and et al., “Neural Networks Meet Physical Networks: Distributed Inference Between Edge Devices and the Cloud,” in _ACM Workshop on Hot Topics in Networks_ , 2018.
  * [35] P. Hu and et al., “Starfish: Resilient Image Compression for AIoT Cameras,” in _Conference on Embedded Networked Sensor Systems_ , 2020, pp. 395–408.
  * [36] H. Amer, “Image/Video Compression: Human and Computer Vision Perspectives,” Ph.D. dissertation, University of Waterloo, Canada, 2020.