# Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge

May Malka, Erez Farhan, Hai Morgenstern, and Nir Shlezinger
Parts of this work were presented in the IEEE International Conference on
Acoustics, Speech, and Signal Processing (ICASSP) 2021 as the paper [1]. This
work was supported in part by the BGU data science center. M. Malka and N.
Shlezinger are with the School of ECE, Ben-Gurion University of the Negev,
Beer-Sheva, Israel (e-mail<EMAIL_ADDRESS>nirshl@bgu.ac.il). E.
Farhan is with Facebook (e-mail: erezfarhan@gmail.com). H. Morgenstern is with
Ramon.Space, Israel (e-mail: hai.morgenstern@ramon.space).

###### Abstract

The success of deep neural networks (DNNs) is heavily dependent on
computational resources. While DNNs are often employed on cloud servers, there
is a growing need to operate DNNs on edge devices. Edge devices are typically
limited in their computational resources, yet, often multiple edge devices are
deployed in the same environment and can reliably communicate with each other.
In this work we propose to facilitate the application of DNNs on the edge by
allowing multiple users to collaborate during inference to improve their
accuracy. Our mechanism, coined edge ensembles, is based on having diverse
predictors at each device, which form an ensemble of models during inference.
To mitigate the communication overhead, the users share quantized features,
and we propose a method for aggregating multiple decisions into a single
inference rule. We analyze the latency induced by edge ensembles, showing that
its performance improvement comes at the cost of a minor additional delay
under common assumptions on the communication network. Our experiments
demonstrate that collaborative inference via edge ensembles equipped with
compact DNNs substantially improves the accuracy over having each user infer
locally, and can outperform using a single centralized DNN larger than all the
networks in the ensemble together.

##  I Introduction

Deep learning systems have demonstrated unprecedented success in various
applications, including computer vision, natural language processing, and
speech [2]. Such systems employ deep neural networks that consist of highly-
parameterized models, trained using massive volumes of data. Consequently,
deep learning is traditionally the domain of large scale computer servers,
which have the computational resources and the ability to aggregate the data
required to store, train, and apply DNNs.

While DNNs are traditionally trained and maintained on powerful computer
servers, the interface with the real world is typically carried out using edge
devices. The data used for inference, including images, text messages, and
signal recordings, is obtained on the user’s side via devices such as
smartphones, tablets, sensors, and autonomous vehicles. This motivates the
implementation of DNNs on edge devices [3]. Applying DNNs as a form of mobile
edge computing allows to infer on the same device where the data is collected,
rather than having the samples sent to a centralized cloud server. Such DNN-
aided edge devices can operate at various connectivity conditions with reduced
latency, as well as alleviate privacy issues and facilitate the
personalization of artifical intelligence (AI) systems [4]. As a result,
recent years have witnessed growing interest in implementing deep learning and
deploying DNNs on edge device [3, 4, 5, 6, 7, 8, 9].

One of the main challenges associated with implementing trained DNNs on edge
devices stems from their limited computational resources [10]. A leading
strategy to tackle this challenge is to make DNNs more compact. This is
achieved by compressing existing highly-parameterized DNNs via pruning [11,
12] and quantization [13, 14] schemes, or alternatively, by designing DNN-
aided systems to utilize compact networks by incorporating statistical model-
based domain knowledge [15, 16, 17]. Such strategies focus on a single edge
user, and thus do not exploit the fact that while each device is limited in
its hardware, multiple users can confidently and securely collaborate to
benefit from their joint computational resources.

An alternative approach which accounts for the ability of edge devices to
collaborate is based on partitioning and dividing a highly-parameterized DNN
among multiple devices. These can be a group of edge devices which can
communicate with one another, or the edge user and its server or access point.
The partitioning of a multi-layered DNN among multiple users to jointly form
the large network during inference is typically referred to as computation
offloading [18, 3, 4], while the division of a DNN between an edge device and
its edge server is commonly coined collaborative intelligence [19, 20]. In
both cases, each participating device only maintains a subset of the layers of
the DNN, and communicates its output features, which are possibly compressed
to reduce the overhead [20, 21], to the specific device which maintains the
subsequent layers. The main drawback of this approach is that each user cannot
infer on its own, and the complete set of devices among which the DNN is
divided must be present. This results in high dependence on connectivity and
possibly increased latency.

The dynamic and mobile nature of edge devices and their ability to communicate
with each other in a device-to-device fashion or via mobile ad-hoc networks
[22] indicates that DNN-aided edge devices should be able to both infer
locally (for limited connectivity scenario) as well as benefit from
collaboration among multiple users. Nonetheless, the above mentioned existing
approaches either rely solely on local inference (via compact DNNs), or rely
on constant reliable communications with a predefined set of edge devices (in
computation offloading) or an edge server (for collaborative intelligence).
This motivates designing protocols and architectures for DNN-aided edge
devices that are adaptive to different conditions in the communication
network. Such protocols should allow each user to infer on its own while
supporting collaboration between a varying number of different neighboring
devices when communications is attainable in a purely decentralized manner.

In this work we propose a framework for utilizing DNNs on edge devices by
leveraging device-to-device communications to allow multiple users to form a
deep ensemble. Deep ensembles are scalable DNN architectures comprised of a
multitude of diverse deep predictors, which carry out inference by aggregating
the individual predictions [23]. Deep ensembles were empirically shown to
achieve high accuracy and generalization performance in a manner that improves
with the number of individual DNNs [24, 25, 26, 27, 28, 29]. Here, we show
that the concept of deep ensembles gives rise to a collaborative inference
method for AI-empowered edge devices, referred to as edge ensembles, which
allows users to benefit from the presence of communication links with other
devices to improve accuracy while being operable in limited connectivity
environments.

We present a decentralized collaborative inference protocol which allows each
user to form a deep ensemble with its neighboring devices during inference. To
be able to benefit from collaboration, we propose to partition the DNN
architecture into a shared encoder and user-specific decoder, such that
diversity among the models, which is key to successful collaboration, is
achieved. To reduce the communication overhead during inference and its
associated latency, we employ a trainable quantizer applied to the encoded
features, resulting in the users exchanging a limited number of bits during
inference stage. Such setups harness the capabilities of each user to infer
locally, while allowing the users to operate in the absence of connectivity.
Our proposed collaborative inference method can be applied to a variable
number of devices, while benefiting from the presence of neighbouring users
with low latency by sharing only quantized features.

We propose two methods for collaborative inference with quantized features. In
the first approach all participating users process the quantized features,
thus requiring each device to maintain a single deep model; the second
technique requires only the neighbouring devices to process the quantized
features and has the aggregating user process the unquantized sample. The
latter approach utilizes a dedicated quantization-aware aggregation mechanism,
which allows to achieve improved accuracy, while requiring the users to
maintain two deep models. We analyze the delay associated with edge ensembles,
showing that the parallel nature of deep ensembles in which all DNNs predict
simultaneously, results at an improved accuracy with a minor additional delay
compared with having single devices inferring on their own.

Our experimental study considers edge devices utilizing the MobilenetV2 deep
classifier [30], applied to images taken from the CIFAR-10, CIFAR-100, and
ImageWoof datasets. We carry out extensive numerical evaluations which
demonstrate that the proposed decentralized low-latency collaborative
inference protocol can notably improve the accuracy compared to inferring
locally, while mitigating the dependence of edge devices on reliable
communications with a centralized server. Furthermore, we show that in some
scenarios, edge ensembles, in which each user has a limited amount of weights,
can achieve improved accuracy over a single centralized DNN with more
parameters than those of all the devices together.

The rest of this paper is organized as follows; in Section II, we detail the
problem of DNN-based edge inference and review some basics in deep ensembles.
Section III presents the edge ensembles mechanism and analyzes its latency.
Experimental results are stated in Section IV. Finally, Section V provides
concluding remarks.

Throughout the paper, we use boldface lower-case letters for vectors, e.g.,
${{\boldsymbol{x}}}$; the $i$th element of ${{\boldsymbol{x}}}$ is written as
$[{{\boldsymbol{x}}}]_{i}$. calligraphic letters, such as $\mathcal{X}$, are
used for sets, and $\mathcal{R}$ is the set of real numbers. The $\ell_{2}$
norm is denoted by $\|\cdot\|_{2}$, while $\Pr(\cdot)$ is the probability
function.

##  II System Model

In this section we present the system model, under which our proposed edge
ensembles protocol is derived in Section III. We begin by formulating the
problem of DNN-based edge inference in Subsection II-A. Then, we discuss the
notion of collaborative inference and its associated requirements in
Subsection II-B. We conclude by reviewing some preliminaries in deep ensembles
and in deep vector quantization in Subsections II-C and II-D, respectively.

###  II-A Problem Formulation

We consider a set of $K$ AI-empowered mobile edge devices. Each device can
communicate with its neighbor devices over dedicated rate-limited
communication links. At a given time $t$, user
$i_{t}\in\\{1,\ldots,K\\}\triangleq\mathcal{K}$ receives an input data sample
${{\boldsymbol{x}}}_{i_{t}}$, such as an image or an audio recording, and
wishes to carry out inference, e.g., classification. The common practice for
applying DNN-aided inference to data acquired by edge devices requires the
user to send the input data to a centralized server that maintains a pre-
trained highly parameterized DNN. The server carries out inference and conveys
its result back to the user. The growing demands for low latency on-device
inference in different connectivity conditions, as well as privacy
considerations limiting the sharing of data samples, necessitates the users to
be able to infer reliably without relying on a central server.

In mobile edge computing, the improved computational capabilities of modern
edge devices are exploited to allow each user to maintain its local deep
model. In particular, the user of index $i\in\mathcal{K}$ has access to a pre-
trained model comprised of $M$ parameters, represented by the vector
${\boldsymbol{\theta}}_{i}\in\mathcal{R}^{M}$, such that the model mapping is
dented by $f_{{\boldsymbol{\theta}}_{i}}(\cdot)$. The users can communicate
with each other, e.g., over a wireless communication channel. Due to the
dynamic nature of the devices, the communication links may not be stable and
have a limited throughput. The quality of the link is represented by its
instantaneous capacity, where we use the random variable (RV) $C_{i,j}^{t}$
for the capacity of the link between the users $i,j\in\mathcal{K}$ at time
instance $t$. If the link between the users $i,j\in\mathcal{K}$ is unavailable
or broken at time instance $t$, then its capacity is zero. To formulate the
links behavior, we define the set of RVs $\\{L_{i,j}^{t}\\}$ representing the
status of the communication links between the users $i$ and $j$ at time
instance $t$, i.e., $L_{i,j}^{t}=0$ implies that $C_{i,j}^{t}=0$ while
$C_{i,j}^{t}>0$ when $L_{i,j}^{t}=1$. We assume reciprocal channels, such that
$L_{i,j}^{t}=L_{j,i}^{t}$, and set $L_{i,i}^{t}\equiv 1$. Such setups
represent, for example, AI-empowered vehicles which may operate in areas
without connectivity to some centralized cloud server while being able to
communicate in a device-to-device manner. An illustration of the resulting
setup is depicted in Fig. 1.

Figure 1: DNN-based edge inference illustration.

Our goal is to characterize a mechanism and the corresponding DNN architecture
for facilitating inference based on the sample ${{\boldsymbol{x}}}_{i_{t}}$,
focusing on adaptive schemes which do not require connectivity with a central
server. We use ${{\boldsymbol{y}}}_{i_{t}}$ and
$\hat{{\boldsymbol{y}}}_{i_{t}}$ to denote the label value associated with
${{\boldsymbol{x}}}_{i_{t}}$ and its estimate produced by the set of AI-
empowered mobile devices, respectively, while $\mathcal{L}_{e}(\cdot;\cdot)$
is the evaluation loss. Using these notations, we aim to design a
collaborative inference procedure along with the incorporation of variable
number of dedicated trained models, which allow us to successfully recover
$\hat{{\boldsymbol{y}}}_{i_{t}}$ such that at each time instance $t$, the user
of index $i_{t}$ recovers $\hat{{\boldsymbol{y}}}_{i_{t}}$ based on the loss
measure
$\mathcal{L}_{e}({{\boldsymbol{y}}}_{i_{t}};\hat{{\boldsymbol{y}}}_{i_{t}})$.

###  II-B Collaborative Inference Strategies

A straight-forward design is to train a single DNN with $M$ parameters and
have each device use it as its local model. Recalling that
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{i}}(\cdot)$ is the mapping carried
out by the DNN model ${\boldsymbol{\theta}}_{i}$, the estimate produced is
given by
$\hat{{\boldsymbol{y}}}_{i_{t}}={\boldsymbol{f}}_{{\boldsymbol{\theta}}_{i_{t}}}({{\boldsymbol{x}}}_{i_{t}})$.
This approach allows each user to infer without requiring any form of
connectivity with the remaining devices, and avoids privacy issues as the
users do not need to send their data to a centralized server for inference.
The drawback of this approach is that while modern edge devices have improved
computational capabilities, their compute resources are still limited compared
with powerful centralized servers, and thus large networks typically need to
be pruned or quantized to be applied on edge devices [11, 12, 13, 14]. As a
result, $M$ is typically small compared with the amount of parameters used in
conventional centralized DNNs, implying that the inference accuracy may be
degraded.

An alternative strategy which exploits the ability of the users to collaborate
during inference is based on partitioning a pre-trained highly-parameterized
DNN among multiple users. Here, the computation of
$\hat{{\boldsymbol{y}}}_{i_{t}}$ is divided among all the $K$ users, such that
the resulting estimate corresponds to the output of a highly-parameterized DNN
with $K\cdot M$ weights. For instance, ${\boldsymbol{\theta}}_{i}$ can be set
to be the $i$th layer of a DNN with $K$ layers, and thus setting
$\hat{{\boldsymbol{y}}}_{i_{t}}=\big{(}{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{K}}\circ\cdots\circ{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{1}}\big{)}({{\boldsymbol{x}}}_{i_{t}})$
implements highly-parameterized DNN-based inference of potentially improved
performance [3, Sec. IV]. Nonetheless, this strategy, referred to as
sequential offloading, involves increased delay, due to the need to operate in
a sequential manner, and requires connectivity among all users. For instance,
when the edge devices are AI-aided vehicles, this implies that the same group
of vehicles must be located in the same physical area whenever one has to
carry out inference. In fact, such forms of collaboration are typically
proposed for settings where inference is to be carried out by a central server
rather than on the edge device, which only handles part of the processing
locally before sending the features, namely, the output of an internal layer
of a deep model, to the server (see, e.g., [31, 32, 33]). The main gains in
collaboration in such centralized settings, which differ from inference on the
edge as formulated in Subsection II-A, follow from the fact that conveying
processed features as opposed to sending the sample
${{\boldsymbol{x}}}_{i_{t}}$ to the server facilitates compression over rate
limited links and alleviates some privacy concerns, in addition to balancing
the computational load [19].

The relatively intuitive approaches discussed above reveal some key guidelines
that should be accounted form in tackling the problem formulated in Subsection
II-A:

  1. G1

Each model ${\boldsymbol{\theta}}_{i}$ must correspond to a dedicated DNN
capable of inferring on its own in order to allow operation without any
connectivity.

  2. G2

These individual DNNs should be diverse, e.g., if
${\boldsymbol{\theta}}_{i}={\boldsymbol{\theta}}_{j}$ for some $i\neq j$ then
users $i$ and $j$ have no added value in collaborating. Thus diversity is key
to benefiting from collaboration among devices during inference.

  3. G3

Collaboration should be accompanied with inference-oriented protocols based on
the exchange of compressed features. In such way we can facilitate
collaboration over rate-limited networks compared to exchanging raw samples.

Guidelines G1-G2 motivate utilizing architectures based on deep ensembles
[23], briefly reviewed in the following subsection, while G3 indicates the
potential of incorporating layers with quantized features [34], discussed in
Subsection II-D.

###  II-C Deep Ensembles

Ensemble methods utilize multiple models whose outputs are combined to achieve
improved performance [35]. Deep ensembles utilize DNNs as the individual
models. Here, during inference, the input sample is processed by each of these
DNNs in parallel, and their outputs are aggregated into a single prediction
[23]. Various techniques are proposed in the literature for ensemble
aggregation, depending on the overall task, including averaging for regression
and classifiers with soft outputs, as well as majority vote for hard-decision
classifiers [23]. Since these aggregation methods can be applied with
different numbers of models, deep ensembles are inherently scalable [24],
which is desirable for our edge-based collaborative settings.

Deep ensembles require the individual models to be diverse, namely, while each
predictor is designed for the same task, their individual mapping is
different. Various measures have been proposed for quantifying such diversity
[36, 37], and different learning algorithms have been suggested for training
diverse models. The common approaches for achieving diversity have each model
trained with different data, either in parallel (via bagging) or sequentially
(via boosting) [38]. Yet, one can also train diverse models by formulating a
joint regularized objective as in [27, 26, 39], or even by simply using
different randomized initializations [29]. When diversity holds, ensemble
methods are known to achieve improved accuracy over that of the individual
models, in a manner which increases with the number of models [23].

###  II-D Deep Models with Quantized Features

DNN-based compression mechanisms have gained increased popularity over the
last few years [40]. These methods compress the input into a discrete form,
mainly for the task of reconstructing it [41, 13, 42, 43], but also for the
purpose of information extraction [44, 45]. These kind of models are usually
comprised of trainable encoder which encodes the input data into a continuous
latent space followed by a dedicated discretization layer that compresses the
features into a discrete latent representation, i.e., a codebook. The discrete
codeword is processed by a trainable decoder that performs the desirable task
on the compressed output of the encoder. The discretization layer can perform
scalar quantization, which is applied element-wise [44, 41], as well as a
joint mapping of multiple elements into single discrete representation via
vector quantization [13, 34].

The main challenge of applying a discretization layer in a DNN is the
inability to differentiate through continuous-to-discrete mappings, whose
gradients are inherently nullified almost anywhere. This limits the ability to
simultaneously train the encoder and the decoder with conventional first order
methods, e.g., stochastic gradient descent (SGD) and its variants. Common
approaches to tackle this differentiability issue include replacement of the
quantization mapping during training with soft [13, 42, 46] or stochastic [41,
43] approximations.

VQ-VAE Model: A popular architecture for quantizing DNN features in a
trainable and bit-efficient manner, which is adopted in the sequel, is the
vector quantized variational autoencoder (VQ-VAE) model, proposed in [34] for
discrete representation learning. The VQ-VAE architecture is comprised of an
encoder-decoder structure with a latent discretization layer implementing
vector quantization, and is accompanied with a dedicated training mechanism
that allows to jointly train the encoder and the decoder, along with learning
of the vector quantization mapping. Let us denote the DNN parameters of the
encoder and the decoder as ${\boldsymbol{\theta}}_{E}$ and
${\boldsymbol{\theta}}_{D}$, respectively. For an input data
${{\boldsymbol{x}}}$, e.g., an image, the continuous output is written as
${{\boldsymbol{x}}}_{e}={\boldsymbol{f}}_{{\boldsymbol{\theta}}_{E}}\left({{\boldsymbol{x}}}\right)$.
The VQ-VAE model holds a discrete codebook ($\mathcal{Q}$), which is an array
of vectors of size $d$, where the cardinality of the codebook is
$P=|\acs{cb}|$. The output of the encoder, ${{\boldsymbol{x}}}_{e}$, is
divided into $m$ vectors of size $d\times 1$, denoted
${{\boldsymbol{x}}}_{e}=[{{\boldsymbol{x}}}_{e}^{1},\ldots,{{\boldsymbol{x}}}_{e}^{m}]$.
The parameter $d$ determines the number of samples that are jointly quantized,
where for $d=1$ the discretization layer is specialized into a trainable
scalar quantizer as in [44]. Then, each vector is mapped into its nearest
codeword in $\mathcal{Q}$ in the $\ell_{2}$ norm sense, i.e.,

${{\boldsymbol{z}}}^{i}=\mathop{\arg\min}\limits_{{\boldsymbol{e}}\in\mathcal{Q}}\left\lVert{{\boldsymbol{x}}}_{e}^{i}-{\boldsymbol{e}}\right\rVert_{2}^{2}.$
(1)

These discrete vectors are stacked to form
${{\boldsymbol{z}}}=[{{\boldsymbol{z}}}^{1},\ldots,{{\boldsymbol{z}}}^{m}]$,
and we write ${{\boldsymbol{z}}}=f_{\mathcal{Q}}({{\boldsymbol{x}}}_{e})$. The
discrete representation ${{\boldsymbol{z}}}$ is provided as input to the
decoder, i.e.,
$\hat{{\boldsymbol{y}}}={\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D}}\left({{\boldsymbol{z}}}\right)$.
The overall number of bits used for representing the features is thus
$B=m\cdot\log_{2}P$.

VQ-VAE Training: Training VQ-VAE with first-order methods such as SGD is
impossible since the argmin operation is non-differentiable. In order to
tackle this, approximation of the gradient is made by copying it from the
decoder input ${{\boldsymbol{z}}}$ into the encoder output
${{\boldsymbol{x}}}_{e}$, i.e., the presence of the codebook is bypassed when
computing the gradients via backpropagation. This complete process depicted in
Fig. 2, where the codebook bypassing is presented in the dashed line.

Figure 2: Vector quantization layer of VQ-VAE.

While bypassing the vector quantizer allows to approximate the gradient of the
training loss, denoted
$\mathcal{L}({{\boldsymbol{y}}};\hat{{\boldsymbol{y}}})$, with respect to
${\boldsymbol{\theta}}_{E}$, it does not support the learning of the codebook
$\mathcal{Q}$ in light of the overall task. To tackle this, it was proposed in
[34] to include two additional terms in the loss function which account for
learning $\mathcal{Q}$. The first term is the VQ-loss given by
$\left\lVert{\rm
sg}\left({{\boldsymbol{x}}}_{e}\right)-{{\boldsymbol{z}}}\right\rVert$, which
is responsible to move the codebook vectors towards the encoder outputs, where
${\rm sg}$ stands for stop-gradient operator, preventing the gradients from
propagating through ${{\boldsymbol{x}}}_{e}$. The second term is the
commitment loss given by
$\beta\left\lVert{{\boldsymbol{x}}}_{e}-sg\left({{\boldsymbol{z}}}\right)\right\rVert$,
which is intended to bring the encoder outputs closer to the codebook vector.
The hyperparameter $\beta$ determines the weight put in for the second term.
The overall training loss of the model is written as

$\displaystyle\mathcal{L}_{tot}({{\boldsymbol{y}}};\hat{{\boldsymbol{y}}})=$
$\displaystyle\mathcal{L}({{\boldsymbol{y}}};\hat{{\boldsymbol{y}}})+\left\lVert{\rm
sg}\left({{\boldsymbol{x}}}_{e}\right)-{{\boldsymbol{z}}}\right\rVert_{2}^{2}+\beta\left\lVert{{\boldsymbol{x}}}_{e}-{\rm
sg}\left({{\boldsymbol{z}}}\right)\right\rVert_{2}^{2}.$ (2)

Since the codebook size $P$, which is the number of vectors in $\mathcal{Q}$,
is a design hyperparameter, we can reduce or enlarge the codebook size, and
consequently control the number of bits required for representing the codebook
vectors indices. The variability of the codebook size and the codebook-vector
size is beneficial for keeping the communication efficient when the discrete
representation is conveyed over rate-limited channels, as well as preserving
the accuracy of an unquantized model, as we consider in the proposed method
for collaborative inference over rate-limited channels detailed in the
following section.

##  III Edge Ensembles

Here, we present the proposed strategy of edge ensembles for collaborative
inference. Unlike offloading-based schemes, our approach is scalable in the
number of participating users and allows local inference, while exploiting the
presence of collaborative devices to improve accuracy, and supporting low-
delay collaboration over rate-limited channels. We begin by presenting the
proposed inference mechanism and its underlying rationale in Subsection III-A,
then we formulate our collaborative inference protocol in Subsection III-B,
after which we characterize its associated delay and provide a discussion in
Subsections III-C-III-D, respectively.

###  III-A High-Level Rationale

The core idea of edge ensembles is to provide each user the ability to carry
out inference on its own, while allowing to benefit from collaboration with
neighboring devices by forming together an ensemble of DNNs. This is achieved
by letting the set of local models represent a diverse deep ensemble. Under
such a setup, each individual user can utilize its local model for inference,
achieving possibly limited accuracy due to the fact that its model is
comprised of a compact network with a relatively small number of parameters.
Nonetheless, multiple users which are capable of communicating with each other
can now collaborate in inference as a form of AI-based wisdom of crowds [47].
In particular, the scalable nature of ensemble aggregation functions, that can
operate with varying number of participating predictors, allows edge ensembles
to operate in dynamic and possibly unstable communication conditions, without
being dependent on the participation of any specific device in the inference
procedure.

In order to support low-latency inference over rate-limited channels, we adopt
DNN architectures which incorporate quantized features. We set the models to
be divided into encoder-decoder structure with quantized latent features,
where the encoder ${\boldsymbol{\theta}}_{E}$ is shared among all the devices
with its output layer being a trainable quantizer with codebook $\mathcal{Q}$.
However, the decoders differ between the edge device. The latter guarantees
diversity in inference, exploited by the ensemble model, while the usage of a
shared encoder with quantized features balances the number of bits the
inferring device sends to each neighbor device during inference. The bit rate
can be set depending on the achievable rate in the existing links, and to
support the required delay in collaboration.

###  III-B Edge Ensemble Inference Protocol

To formulate the inference procedure of edge ensembles, let us recall that
$L_{i,j}^{t}$ denotes whether the link between the users $i$ and $j$ at time
instance $t$ is active. We define $\mathcal{S}_{i}^{t}$ as the set of users
which user $i$ can reliably communicate with at time $t$, i.e.,

$\mathcal{S}_{i}^{t}\triangleq\\{j\in\mathcal{K}|L_{i,j}^{t}\neq 0\\}.$ (3)

By definition $i\in\mathcal{S}_{i}^{t}$, and
$|\mathcal{S}_{i}^{t}|\in\\{1,\ldots,K\\}$.

The inference stage begins at a given time instance $t$, when each user has a
trained deep model available locally. These deep models are comprised of a
shared encoder and codebook followed by a user-specific decoder. We denote the
shared parameters of the encoder as ${\boldsymbol{\theta}}_{E}$ and the shared
codebook of the quantizer (see Subsection II-D) as $\mathcal{Q}$. The
parameters of the decoders, which are unique and diverse for each edge device,
are denoted by $\\{{\boldsymbol{\theta}}_{D,j}\\}$. As explained in Subsection
II-A, at time instance $t$ the user of index $i_{t}$ observes a data sample
${{\boldsymbol{x}}}_{i_{t}}$, to be used for inference. To assist with its
neighbor devices for improving performance, the $i_{t}$th user encodes and
quantizes ${{\boldsymbol{x}}}_{i_{t}}$ for the purpose of sharing the input
while maintaining low latency requirements, resulting in a conveyed quantized
version of ${{\boldsymbol{x}}}_{i_{t}}$, written as

${{\boldsymbol{z}}}_{i_{t}}={\boldsymbol{f}}_{\mathcal{Q}}\big{(}{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{E}}\big{(}{{\boldsymbol{x}}}_{i_{t}}\big{)}).$
(4)

The user of index $i_{t}$ broadcasts the quantized features,
${{\boldsymbol{z}}}_{i_{t}}$, making it available to its active neighbours,
i.e., all users in the set $\mathcal{S}_{i_{t}}^{t}$. Note that when the
encoder and quantizer are set to the identity mapping, i.e.,
${{\boldsymbol{z}}}_{i_{t}}={{\boldsymbol{x}}}_{i_{t}}$, sharing the quantized
features boils down to sharing the observed sample
${{\boldsymbol{x}}}_{i_{t}}$.

Next, each user in $\mathcal{S}_{i}^{t}$ applies its local decoding model to
${{\boldsymbol{z}}}_{i_{t}}$, and conveys the resulting
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}$
back to user $i_{t}$, which aggregates them into the predicted
$\hat{{\boldsymbol{y}}}_{i_{t}}$. In particular, for classification tasks with
$N$ different labels, the output of each individual network
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{x}}}_{i_{t}}\big{)}$
is an $N\times 1$ vector whose entries are an estimate of the conditional
distribution of each label given ${{\boldsymbol{z}}}_{i_{t}}$, and the
predicted $\hat{{\boldsymbol{y}}}_{i_{t}}$ can be obtained as the label which
maximizes the averaged conditional distribution [27], i.e.,

$\mathop{\arg\max}\limits_{n=1,\ldots,N}\frac{1}{|\mathcal{S}_{i}^{t}|}\sum_{j\in\mathcal{S}_{i}^{t}}\left[{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}\right]_{n}.$
(5)

The resulting inference procedure is summarized as Algorithm 1 and illustrated
in Fig. 3(a).

Figure 3: Illustration of collaborative inference protocol with shared
quantized features, with $K=4$ users, of which one user is not reachable. The
quantized features can either be used for inference by all users $(a)$, or
solely by the neighbouring devices, while the inferring user processes the
input sample $(b)$.

Input: Data sample ${{\boldsymbol{x}}}_{i_{t}}$ observed at time $t$ at user
$i_{t}$.

1 User $i_{t}$ encodes and quantizes ${{\boldsymbol{x}}}_{i_{t}}$ into
${{\boldsymbol{z}}}_{i_{t}}$ via (4);

2 User $i_{t}$ broadcasts ${{\boldsymbol{z}}}_{i_{t}}$ to its neighboring
devices;

3 for  _each $j\in\mathcal{S}_{i_{t}}^{t}$ _ do

4 User $j$ computes
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}$;

5 User $j$ sends the result to user $i_{t}$;

6

7 end for

8User $i_{t}$ aggregates
$\\{{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}\\}$
into ${\boldsymbol{y}}_{i_{t}}$ via, e.g., (5);

Output: Prediction $\hat{{\boldsymbol{y}}}_{i_{t}}$.

Algorithm 1 Collaborative Edge Ensembles Inference with Local Quantized
Features

Algorithm 1 formulates the collaborative inference rule in which all
participating users, i.e., the inferring user $i_{t}$ and its neighbours,
apply their local decoders to the quantized features
${{\boldsymbol{z}}}_{i_{t}}$. However, while quantizing the features conveyed
to the neighbor devices is essential to cope with rate-limited communications,
the inferring user is not restricted to produce its local estimate based on
the quantized features, and can use the actual sample
${{\boldsymbol{x}}}_{i_{t}}$ to which it has access. In order to support such
non-quantized local inference, each user should maintain two different
decoders, ${\boldsymbol{\theta}}_{D,j}$ for inference with the non-quantized
version of a locally observed input, and ${\boldsymbol{\theta}}_{D_{q},j}$ for
the quantized version of a sample acquired by a neighbour device. In this
case, local inference at the $i_{t}$th user is carried out by applying
${\boldsymbol{\theta}}_{D,i_{t}}$ (trained to be applied directly to input
samples) to ${{\boldsymbol{x}}}_{i_{t}}$, while the neighbouring devices,
which have access only to the quantized ${{\boldsymbol{z}}}_{i_{t}}$, use
${\boldsymbol{\theta}}_{D_{q},j}$, as illustrated in Fig. 3(b).

The non-quantized version of the input, which is available to the inferring
user, contains information that may be distorted during quantization. Hence,
naive aggregating via, e.g., taking the mean of the estimated probabilities as
in (5), can result in a predication that is less accurate compared with
inferring locally. We thus propose a weighted aggregation scheme which
balances the significance to the results of the decisions made based on the
quantized features by the neighbouring devices, and that made based on the raw
sample by the inferring users. To that aim, we replace (5) with the following
weighted aggregation

$\mathop{\arg\max}\limits_{n=1,\ldots,N}\frac{1}{|\mathcal{S}_{i}^{t}|}\left(\left(\sum_{\begin{subarray}{c}j\in\mathcal{S}_{i}^{t}\\\
{j\neq
i}\end{subarray}}\alpha_{j}\left[{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D_{q},j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}\right]_{n}\right)+\alpha_{i}\left[{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,i}}\big{(}{{\boldsymbol{x}}}_{i_{t}}\big{)}\right]_{n}\right).$
(6)

In (6), the coefficients $\\{\alpha_{j}\\}$ are weighting parameters taking
values in $[0,1)$, which are set to balance the contribution of the
neighbouring devices and that of the local model. These coefficients could be
set based on the associated quantization resolution, since the more coarse the
quantization is, the lesser the confidence in the decisions based on the
quantized features is expected to be. In our experimental study in Section IV
we used the setting of $\alpha_{j}$ which is based on the number of
participating users and the validation accuracy of each user, i.e., users with
improved validation accuracy are assigned with higher values of $\alpha_{j}$,
see detailed description in Subsection IV-D. The resulting inference
procedure, summarized in Algorithm 2, allows to assign more confidence to the
non-quantized model while still being helped by neighboring devices and
benefiting from collaboration.

Input: Data sample ${{\boldsymbol{x}}}_{i_{t}}$ observed at time $t$ at user
$i_{t}$; aggregation weights $\\{\alpha_{j}\\}$

1 User $i_{t}$ encodes and quantizes ${{\boldsymbol{x}}}_{i_{t}}$ into
${{\boldsymbol{z}}}_{i_{t}}$ via (4);

2 User $i_{t}$ broadcasts ${{\boldsymbol{z}}}_{i_{t}}$ to its neighboring
devices;

3 User $i_{t}$ computes
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,i_{i}}}\big{(}{{\boldsymbol{x}}}_{i_{t}}\big{)}$;

4 for  _each $j\in\mathcal{S}_{i_{t}}^{t}/i_{t}$ _ do

5 User $j$ computes
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D_{q},j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}$;

6 User $j$ sends the result to user $i_{t}$;

7

8 end for

9User $i_{t}$ aggregates
$\\{{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D_{q},j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}\\}$
and
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,i_{i}}}\big{(}{{\boldsymbol{x}}}_{i_{t}}\big{)}$
into ${\boldsymbol{y}}_{i_{t}}$ via, e.g., (6);

Output: Prediction $\hat{{\boldsymbol{y}}}_{i_{t}}$.

Algorithm 2 Collaborative Edge Ensembles Inference without Local Quantized
Features

###  III-C Inference Latency Analysis

Latency is one of the main motivations for utilizing AI-empowered edge devices
over conventional cloud server-based DNNs. Our proposed collaborative
inference algorithm entails some excessive delay compared with having the
users infer only locally, due to the communication overhead and system
heterogeneity in the devices. In this section we provide an analysis of the
statistical properties of the overall inference latency of Algorithm 1 (which
is similar to that of Algorithm 2, as both include the same communications
overhead).

To that aim, we first introduce the following symbols: Let $\\{T_{i,j}^{t}\\}$
be the communication delays between a pair of distinct users at the beginning
of the inference procedure at time $t$. In particular, $T_{i,j}^{t}$ is the
sum of the duration needed for communicating a sample ${{\boldsymbol{z}}}_{i}$
from user $i$ to user $j$ at given time $t$, denoted as $\tau_{i,j}$; the
computation time required by user $j$ to apply its local decoding model,
denoted as $\tau_{P_{j}}$; and the time required for conveying
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i}\big{)}$
back to user $i$ , denoted $\tau_{j,i}$. Therefore, we can write the total
delay time in the communication between user $i$ and $j$ as

$T_{i,j}^{t}=\tau_{i,j}+\tau_{P_{j}}+\tau_{j,i}.$ (7)

The transmitted features vector ${{\boldsymbol{z}}}_{i_{t}}$ is comprised of
$B_{1}$ bits, which is the number of bits sent by user $i$ to user $j$ at time
instance $t$. While the channel conditions can vary in time, we consider the
conventional model of block-fading temporal variations for wireless channels,
where the channels remain fixed during the entire inference stage. Hence, the
transmitting delay $\tau_{i,j}$ of an active link can be written as
$\tau_{i,j}=\frac{B_{1}}{C_{i,j}^{t}}$. Similarly, letting $B_{2}$ be the
number of bits needed to represent the local inference output at device $j$,
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,j}}\big{(}{{\boldsymbol{z}}}_{i_{t}}\big{)}$,
we can write $\tau_{j,i}=\frac{B_{2}}{C_{j,i}^{t}}$. Since the computation
delays $\tau_{P,j}$ are dictated by the hardware of each device, which is
static, they are assumed here to be deterministic quantities invariant in
time. Consequently, we can write the delay term (7) as

$T_{i,j}^{t}=L_{i,j}^{t}\left(\frac{B_{1}}{C_{i,j}^{t}}+\frac{B_{2}}{C_{j,i}^{t}}+\tau_{P,j}\right).$
(8)

The communication delay in (8) accounts for the possibility of links to be
active or non-active, assuming that the inferring user knows with which
devices it can communicate. For active links we receive the same term as in
(7), while for non-active links there is no latency. We assume that the
channels are reciprocal, such that $C_{i,j}^{t}=C_{j,i}^{t}$, and focus on the
inference being a classification task with $N$ labels for simplicity. In such
cases, the response of each neighbouring device, i.e., Step 1 of Algorithm 1,
conveys solely an $N\times 1$ probability vector, which is typically of a much
lower dimension compared with the exchanged features
${{\boldsymbol{z}}}_{i_{t}}$ in Step 1, and thus $B_{1}\gg B_{2}$. In such
case, the overall delay $T_{i,j}^{t}$ in (8) is reduced into

$T_{i,j}^{t}=L_{i,j}^{t}\left(\frac{B_{1}}{C_{i,j}^{t}}+\tau_{P,j}\right).$
(9)

By using these notations, we can define the overall delay induced by the
collaborative inference procedure that starts at time $t$ in Algorithm 1 as
the RV $\Delta^{t}$, given by

$\Delta^{t}\triangleq\max\limits_{j\in\mathcal{K}}\left(T_{i,j}^{t}\right)=\max\limits_{j\in\mathcal{K}}\left(L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P,j}\right)\right).$
(10)

The formulation of the inference delay in (10) allows us to characterize its
distribution for i.i.d. links, as stated in the following theorem:

######  Theorem 1.

When the communication links RVs $\\{L_{i,j}^{t}\\}_{i\neq j}$ are i.i.d. with
$\Pr(L_{i,j}^{t}=1)\equiv p$, and the channel capacities
$\\{C_{i,j}^{t}\\}_{i\neq j}$ are positive and i.i.d. RVs with conditional
cumulative distribution function given by
$F_{C}(\epsilon)\triangleq\Pr(C_{i,j}^{t}<\epsilon|L_{i,j}^{t}=1)$, then for
any $\epsilon>0$, the overall delay RV $\Delta^{t}$ satisfies

$\Pr\left(\Delta^{t}<\epsilon\right)=\begin{cases}0&\epsilon\leq\tau_{P_{i_{t}}},\\\
\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau_{P_{j}}}\end{subarray}}\left(1-p\cdot
F_{C}\left(\frac{B}{\epsilon-\tau_{P_{j}}}\right)\right)\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon\leq\tau_{P_{j}}}\end{subarray}}\left(1-p\right)&\epsilon>\tau_{P_{i_{t}}}.\end{cases}$
(11)

Proof: See Appendix -A.

Theorem 1 reveals some of the statistical properties of the overhead induced
by collaborative inference via Algorithm 1. For instance, consider the case in
which all devices have the same local processing delay, i.e.,
$\tau_{P,j}\equiv\tau$, and denote the smallest capacity among all the active
neighbouring users as
$C_{\min}=\min_{j}\left\\{C_{i,j}^{t}|L_{i,j}^{t}=1\right\\}$. Hence, for a
fixed number of bits $B$ needed to represent ${{\boldsymbol{z}}}_{i_{t}}$, the
communication delays are upper bounded by a finite
$T_{\max}=\frac{B}{C_{\min}}+\tau$. We note that for each $\epsilon\geq
T_{\max}$, it holds that $\Pr\left(\Delta^{t}<\epsilon\right)$ equals 1 and
$\Pr\left(\Delta^{t}<\epsilon\right)<1$ for each $\epsilon<T_{max}$. Here, it
immediately follows that the delay $\Delta^{t}$ is not larger than
$\frac{B}{C_{\min}}+\tau$, which indicates that the resulting delay is smaller
than that obtained using edge computing based on sequential offloading with
[4]. This follows from the fact that the computations of each of the DNNs in a
deep ensemble is carried out in parallel, without requiring any of the users
to wait for another device to finish its local computation before it begins.
Nonetheless, Theorem 1 also indicates that the delay is likely to increase as
the number of users grow, as stated in the following corollary:

######  Corollary 1.

When the computation time is identical among all the devices, i.e.,
$\tau_{P,j}\equiv\tau$ for each $j\in\mathcal{K}$, the communication delays
are upper bounded by $T_{\max}$, and $p>0$, then for each $\epsilon<T_{\max}$
it holds that

$\lim_{K\rightarrow\infty}\Pr\left(\Delta^{t}<\epsilon\right)=0.$ (12)

Proof: See Appendix -B.

Corollary 1 implies that the aforementioned upper bound on the overall latency
becomes tight as the number of users grow under identical computation times.
Still, this upper bound is typically smaller than the overall delay induced
when utilizing sequential offloading, which involves more computation and
communication iterations for carrying out inference. Specifically, since the
duration of inferring locally, which involves the smallest latency, is
$\tau_{P,i}$, Corollary 1 suggests that the only additional latency in that
case is the maximal one induced by the single communication round in Steps 1
and 1 of Algorithm 1, i.e., $T_{\max}$.

###  III-D Discussion

The proposed edge ensembles framework facilitates the operation of deep
learning on edge devices by accounting for their unique properties and
requirements. In particular, Algorithms 1-2 are tailored to account for the
limited hardware and computation capabilities of such devices, combined with
their mobile dynamic nature and ability to collaborate in a decentralized
manner. In fact, edge ensembles treat AI-empowered mobile devices as a crowd
of diverse intelligent individuals: each agent is capable of inferring on its
own, and can thus operate solely without any connectivity. However, a group of
users can infer more reliably in a collaborative manner as a form of
(artificial) wisdom of crowds [47]. Specifically, this collaboration boils
down to an equivalent deep ensemble, which is an established concept in the
deep learning literature [23] known to improve accuracy and robustness.

Collaboration based on edge ensembles is fundamentally different from
computation offloading mechanisms, which facilitate complex DNN-based
computations over a set of edge devices by partitioning a DNN and providing
each user with a different partition [4]. Specifically, computation offloading
requires all the users to participate in order to carry out inference, and
thus requires constant connectivity, while typically inducing increased
latency when the DNN partitions must be applied in a sequential manner, e.g.,
when each partition is a set of layers in a feed-forward network [3, Sec. IV].
Edge ensembles do not rely on connectivity, as each user can infer on its own,
and collaboration is applicable with a varying number of users. As Algorithms
1 and 2 are designed to form a deep ensemble, the more diverse users
participate, the more accurate the prediction is expected to be [35]. This
comes at the probable cost of increased latency, as noted in Subsection III-C.
The accuracy can be thus further improved by allowing more users to
participate, e.g., by extending Algorithms 1-2 to support multi-hop
communications, as done in the context of distributed quantization in [48],
again at the cost of increased latency.

Collaboration naturally gives rise to challenges that originate from the fact
that multiple users participate in inference. These include privacy concerns,
due to the exchange of acquired samples; security considerations, as a
malicious users can affect prediction; and communication bottlenecks affecting
the exchange of information among the participating users. The communication
delay of our proposed collaborative inference approach is reduced by
compressing the samples, thus conveying smaller volumes of data. We propose to
carry out such compression by implementing vector quantization to the shared
samples. The contribution of the quantization step is that the number of bits
selected to represent the shared sample is controlled, resulting in overseeing
the latency of the communication between the users, which depends on the
capacities of the channels. Moreover, quantizing the shared features is likely
to contribute to privacy conservation by the fact that the raw samples are not
shared, but rather the quantized version of its extracted features [49].
Security considerations can potentially be alleviated by incorporating
Byzantine robust aggregation methods [50]. Nonetheless, we leave the analysis
of the privacy enhancement capabilities of our approach and its combination
with secure aggregation for future work. While we opt a learned quantization
mechanism based on the VQ-VAE architecture in our experimental study in
Section IV, edge ensembles can also be employed with alternative compression
schemes for the exchanged features, such as the usage of universal vector
quantizers [51] or of trainable scalar quantizers [44].

Designing separate DNN-aided edge devices to form a deep ensemble is a
consideration that has to be accounted for in the training of the models. For
once, the individual DNNs have to be different from one another in order to
benefit from collaboration [37]. This can be achieved by either training the
DNNs jointly while boosting diversity, e.g., by using a regularized objective
as in [27, 39] or different randomized initialization [29]. Alternatively,
diverse local models can be trained by modifying distributed learning
algorithms to result in different individual networks [52]. In our numerical
study reported in Section IV we use both a centralized training approach where
the local models are trained with the same data and diversity is achieved by
different initializations, as well as a distributed-oriented training approach
in which each model is trained using shared and user-specific data. Both
techniques are shown to yield diverse models allowing the users to benefit
from forming an edge ensemble.

##  IV Experimental Study

In this section we numerically evaluate the proposed collaborative inference
framework. We begin by evaluating the latency analysis in Subsection IV-A,
after which we examine the accuracy of edge ensembles collaborating via
Algorithms 1 and 2. For the latter, we study the gains of collaboration over
centralized inference in Subsection IV-B, and then we evaluate Algorithms 1
and 2 in Subsections IV-C and IV-D, respectively.

For all our experiments we adopt the MobilenetV2 DNN architecture [30], being
a deep model with multiple layers that is designed to be used on edge devices.
We focus on image classification tasks using the CIFAR-10, CIFAR-100, and
Imagewoof datasets111The source code used in our experiments is available at
https://github.com/MayMalka10/Ensembles-On-The-Edge. We examine the
performance with $K\in\\{1\dots 16\\}$ devices in which $\\{L_{i,j}^{t}\\}$
are i.i.d. with connectivity probability $p\in[0,1]$.

###  IV-A Latency Evaluation

We begin by numerically evaluating the latency analysis derived in Subsection
III-C in order to visualize the effects of different parameters on the overall
latency. Here, the local computation time $\tau_{P,j}$ is identical among all
the devices, and is set to $700$ milliseconds, being identified in [53] as the
MobilenetV2 inference time on various light-weight devices. We evaluate the
latency distribution characterized in Theorem 1 when the capacity RVs
$C_{i,j}^{t}$ obey an i.i.d. Rayleigh distribution. The connectivity
probability $p$ takes values in $\\{0.2,0.8\\}$; the number of bits to
communicate $B_{1}$ examined is either $32$ or $128$ bits; and the number of
users participating in the collaborative inference is $K=4$ and $K=64$.

Figure 4: $\Pr(\Delta^{t}<\epsilon)$ versus the latency in milliseconds.

The resulting latency distributions are depicted in Fig. 4. We first observe
in Fig. 4 that, as expected, the probability that the delay is smaller than
$\epsilon$ tends to zero as the required latency is $\epsilon<\tau_{P,j}$, and
that the probability starts to increase as $\epsilon$ grows. We can observe
the influence of an accretion in the number of bits or in the number of
participating users as a more moderate rise in $\Pr(\Delta^{t}<\epsilon)$. We
also see a trade-off behavior in the case of $p=0.2$ between ($128$ bits, $4$
users) and ($32$ bits, $64$ users). This indicates that in cases of limited
connectivity environments, increasing the number of participating users can
bridge the gap of lower number of bits.

###  IV-B Collaborative Inference vs. Centralized Inference

We proceed to examine the benefits in performance of collaboration among edge
devices using compact models compared with using a centralized larger model.
Since we do not consider here communications and latency considerations, we do
not process the input ${\boldsymbol{x}}_{i_{t}}$ prior to its sharing with the
collaborating devices. Namely, we utilize Algorithm 1 with an encoder and
quantizer mappings set to the identity function. Our centralized model is a
MobilenetV2 DNN with width factor one, which has $M=2.23\cdot 10^{6}$
trainable parameters and is trained from random initialization (no pre-
training) with CIFAR-10 dataset. For the edge devices, we train a set of
diverse MobilenetV2 DNNs with width factors of $\frac{1}{4},\frac{1}{3}$, and
$\frac{1}{2}$, for which the number of parameters is $M=\\{2.51,3.96,7\\}\cdot
10^{5}$, respectively. The edge networks are trained using the same CIFAR-10
dataset and training algorithm, but with different random weight
initialization in order to achieve diverse individual networks [29]. These
individual edge devices then collaborate during inference via Algorithm 1
where the aggregation is computed via (5) with
${\boldsymbol{z}}_{i_{t}}={\boldsymbol{x}}_{i_{t}}$.

(a) Accuracy versus connectivity probability.

(b) Accuracy versus overall parameters.

Figure 5: Accuracy comparison of edge ensembles with different width factors
compared with centralized inference with a large network of width factor $1$.

For this setup, we evaluate how the accuracy of edge ensembles grows with the
connectivity probability $p$ for a network with $K=16$ users. The resulting
accuracy versus connectivity probability for 16 devices compared to the
baseline centralized full MobilnetV2 (abbreviated MobilenetV2_1.0) is depicted
in Fig. 5(a). Observing Fig. 5(a), we note that collaboration among edge
devices substantially improves accuracy, while allowing each device to infer
solely. For example, having each device utilize a DNN with width factor
$\frac{1}{2}$, which has roughly $30\%$ the number of weights as
MobilenetV2_1.0, allows each user to achieve inference accuracy of $75\%$
(i.e., for $p=0$), compared to $79\%$ achieved by the full model. However,
having multiple devices collaborate via Algorithm 1 improves the accuracy to
over $80\%$ when the users can communicate with probability as low as $20\%$.
It is also observed in Fig. 5(a) that even when each device has a model as
compact as comprising of $6\%$ of the overall number of parameters, which is
the case for width factor of $\frac{1}{4}$, the resulting accuracy still
improves when $p$ grows, namely, by collaborating via Algorithm 1, though the
collaborating users are unable to outperform the centralized full model,
reaching an accuracy of less than $77\%$ for $p=1$, i.e., full device
participation.

Next, we evaluate how the accuracy scales with respect to the overall number
of parameters used for inference, which dictates the capacity of the DNN. The
results, depicted in Fig. 5(b), illustrate that collaborative edge ensembles
can outperform a centralized model with more parameters than that used by all
the participating users together. In particular, we observe that three edge
devices equipped with diverse trained MobilnetV2 DNN with width factor
$\frac{1}{2}$, whose overall number of weights is $2\cdot 10^{6}$, outperform
the centralized MobilenetV2_1.0, which has $10^{5}$ additional parameters.
These results indicate that collaborative inference of edge ensembles does not
only facilitate inference of DNN-based hardware-limited edge devices, but in
some scenarios can also allow to achieve improved accuracy with less
parameters compared to deep centralized models.

###  IV-C Collaborative Inference Protocol via Algorithm 1

In this part, we examine the performance of the proposed collaborative
inference protocol when the users communicate over rate-limited channels, and
thus share quantized features. We focus here on the case where all users
processes quantized features as proposed in Algorithm 1. We consider image
classification with the CIFAR-100 dataset. The encoder, which is the first
five residual blocks of MobilenetV2 (with the standard width factor one), is
shared among all participating user alongside the quantizer, that is the
trainable codebook of vectors. The decoders are the remaining residual blocks
of MobilenetV2, deployed on each edge user, where diversity is achieved in two
ways: random initialization and bagging. For the latter we train each DNN with
different subsets of the training data, where $34000$ out of the $50000$
training samples of CIFAR-100 are available to all users, and $1000$ unique
samples are used by each one of the 1$K=16$ users. This represents settings
where training involves small sets of private data available at the users
side, as in federated learning, as well as some shared non-private data. The
size $d$ of each vector in the quantizer codebook is the number of the encoder
output channels, i.e., the encoder output is a tensor of size ${\rm
height}\times{\rm width}\times d$, and is divided into ${\rm height}\cdot{\rm
width}$ vectors of size $d\times 1$. We test the accuracy achieved by
collaborating edge users when quantizing using $\log_{2}P\in\\{1,4,12\\}$
bits, where we achieve different bit settings by dividing each $d\times 1$
sub-vector into up to $8$ parts that are quantized separately. We compare the
resulting accuracy to that of a single user inferring without quantization.

(a) Accuracy versus number of users.

(b) Accuracy versus number of bits.

Figure 6: Ensemble of edge devices via algorithm 1 \- random initialization

(a) Accuracy versus number of users.

(b) Accuracy versus number of bits.

Figure 7: Ensemble of edge devices via Algorithm 1 \- bagging

The accuracy achieved versus number of users and versus number of bits when
diversity is achieved via random initialization is depicted in Fig. 6. From
Fig. 6(a) we can see the substantial gains of collaborating with different
edge devices by sharing solely quantized features. For instance, a single
model with non-quantized features achieves $\sim 74\%$ accuracy, while the
collaboration of $16$ users with quantized features surpasses an accuracy of
$80\%$, namely, a gain of over $6\%$ in accuracy with minor communication
overhead. Moreover, we see that as the number of users grow, the higher the
accuracy is, though at a cost of saturated behavior due to limitation in
diversity. A similar behavior is observed in Fig. 7 where diversity is
achieved via bagging. In particular, in Fig. 6(b) we observe gains in
performance for $\log_{2}P\geq 10$ bits when only two users are collaborating,
while in Fig. 7(a) we observe that for non-crude quantization, the difference
between collaboration among $K=10$ users and $K=16$ users is minor, which is
likely due to the difficulty in yielding sufficiently diverse models using the
considered division of data.

###  IV-D Collaborative Inference Protocol via Algorithm 2

We conclude our experimental study by evaluating Algorithm 2, where the
aggregating user applies a sepeate decoder DNN to the input sample directly.
The aggregation accounts for the fact that predictions made by the neighboring
users based on quantized features are likely to be less accurate. In
particular, we set the aggregation coefficients $\\{\alpha_{j}\\}$ in (6)
using the validation accuracy values $\\{V_{j}\\}$ computed during training.
Setting $j=1$ to be the index of the inferring user, $V_{1}$ is the accuracy
achieved with the non-quantized decoder, i.e.,
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D,1}}\big{(}\cdot\big{)}$, while
$V_{j}$ for $j>1$ is obtained based on the quantized features, namely, with
the mapping
${\boldsymbol{f}}_{{\boldsymbol{\theta}}_{D_{q},j}}\big{(}{\boldsymbol{f}}_{\mathcal{Q}}\big{(}{\boldsymbol{f}}_{{\boldsymbol{\theta}}_{E}}\big{(}\cdot\big{)}\big{)}\big{)}$.

In particular, we set the aggregation coefficients in two steps. First,
according to each model validation accuracy, $V_{j}$, we compute relative
accuracy of each device as
$\hat{V}_{j}=\frac{V_{j}^{\rho}}{\sum_{i=1}^{K}V_{i}^{\rho}}$ where $\rho$ is
a sharpening constant which we empirically set to $8$. Then, the total
aggregation is performed with weighting coefficients

$\alpha_{j}=\frac{1}{V_{1}+\frac{1}{\sqrt{K}}\sum_{i=2}^{K}\hat{V}_{i}}\cdot\begin{cases}V_{1}&j=1,\\\
\frac{\hat{V}_{j}}{\sqrt{K}}&j>1.\end{cases}$ (13)

Here, all models are trained to be diverse with the same data via random
initialization, which was shown in the previous subsection to yield similar
diversity to that achieved via data division. We evaluate Algorithm 2 for
image classification using CIFAR-100 dataset as well as Imagewoof, which is a
subset of 10 dog breed classes from the Imagenet dataset. The results achived
with CIFAR-100 and with Imagewoof are reported in Figs. 8 and 9, respectively.

(a) Accuracy versus number of users.

(b) Accuracy versus number of bits.

Figure 8: Ensemble of Edge devices via Algorithm 2 \- CIFAR100

(a) Accuracy versus number of users.

(b) Accuracy versus number of bits.

Figure 9: Ensemble of Edge devices via Algorithm 2 \- Imagewoof

In Fig. 8(a) we can observe the same behavior as seen in Fig. 6(a) where an
ensemble accuracy of slightly more than $80\%$ is reached. Since the first
model processes unquantized features, Algorithm 2 achieves accurate
predictions while balancing the contribution of decisions made by neighbouring
devices based on crudely quantized features. In Fig. 6(a), 4-bit quantization
reaches an accuracy of single unquantized model for more than $K=10$
participating users, where in Fig. 8(a) only two models are needed to
outperform single unquantized model. The same behavior is reported in Fig.
8(b) alongside Fig. 6(b). These results demonstrate that Algorithm 2 allows to
benefit from collaboration and outperform single model even for crude
quantization, such as 4-bit quantization, without harming the performance of
the single unquantized model. This is most notable when observing that in Fig.
6(a), $K=16$ users collaborating with 4-bit quantization via Algorithm 1 were
able to perform similarly as a single unquantized model, where in Fig. 8(a),
$K=16$ users improve by $3\%$ the accuracy of single model. This shows the
strength of Algorithm 2 in limited connectivity environments over Algorithm 1.

While each sample CIFAR-100 is an image of solely $(32\times 32)$ pixels, in
Imagewoof each sample is comprised of $(256\times 256)$ pixels, and thus
compression becomes even more critical to reduce communication overhead and
delay. Nonetheless, the high dimensionality of this data set indicates that
one can potentially apply crude quantization while preserving the ability to
reliably infer via Algorithm 2, as we observe in Fig. 9. In particular, as
observed in Fig. 9(a), the accuracy of 1-bit quantization reaches $87.4\%$
accuracy for large number of participating models, which is almost as that of
a single unquantized model which achieves $87.5\%$ accuracy. Noting that with
the lower-dimensional CIFAR-100, using 1-bit quantizers lead to bad
predictions comparing with an unquantized single model, we conclude that the
benefits of the proposed edge ensembles mechanism in allowing decentralized
collaborative inference with low latency grow more notable for realistic high-
dimensional data.

##  V Conclusions

In this paper we proposed edge ensembles, which is a DNN-based mobile edge
collaborative inference scheme that is based on the joint forming of deep
ensembles by multiple users. Our proposed edge ensemble mechanism allows AI-
empowered edge devices to operate at limited connectivity setups, as well as
achieve improved accuracy by collaboration with neighboring devices. We have
characterized the inference latency of this strategy, showing that it results
in a minor overhead compared to local inference. Our numerical tests
demonstrate that such a collaboration allows a group of users equipped with
compact DNNs to achieve improved accuracy by collaboration for a variety of
data sets, Our results indicate the potential of the proposed strategy in
facilitating accurate DNN-aided inference on hardware-limited edge devices.

###  -A Proof of Theorem 1

Since $L_{i_{t},i_{t}}=1$ and $T_{i_{t},i_{t}}=0$ with probability one, it
follows from (10) that $\Delta^{t}$ is not smaller than $\tau_{P_{i_{t}}}$
with probability one, i.e., $\Pr(\Delta^{t}<\epsilon)=0$ for
$\epsilon\leq\tau_{P_{i_{t}}}$. For $\epsilon>\tau_{P_{i_{t}}}$ it holds that

$\displaystyle\Pr(\Delta^{t}<\epsilon)=$
$\displaystyle\Pr\left(\max\limits_{j\in\mathcal{K}}\left(L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P_{j}}\right)\right)<\epsilon\right)$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}$
$\displaystyle\Pr\left(\max\limits_{j\neq
i_{t}}\left(L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P_{j}}\right)\right)<\epsilon\right)$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}$ $\displaystyle\prod_{j\neq
i_{t}}\Pr\left(L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P_{j}}\right)<\epsilon\right)$
$\displaystyle\stackrel{{\scriptstyle(c)}}{{=}}$ $\displaystyle\prod_{j\neq
i_{t}}(1-p)+p\Pr\left(\frac{B}{C_{i,j}^{t}}+\tau_{P_{j}}<\epsilon|L_{i_{t},j}=1\right)$
$\displaystyle\stackrel{{\scriptstyle(d)}}{{=}}$
$\displaystyle\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau_{P_{j}}}\end{subarray}}\left((1-p)+p\Pr\left(C_{i,j}^{t}>\frac{B}{\epsilon-\tau_{P_{j}}}|L_{i_{t},j}=1\right)\right)\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon\leq\tau_{P_{j}}}\end{subarray}}\left(1-p\right)$
$\displaystyle\stackrel{{\scriptstyle(e)}}{{=}}$
$\displaystyle\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau_{P_{j}}}\end{subarray}}\left((1-p)+p\left(1-F_{C}\left(\frac{B}{\epsilon-\tau_{P_{j}}}\right)\right)\right)\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon\leq\tau_{P_{j}}}\end{subarray}}\left(1-p\right)$
$\displaystyle\stackrel{{\scriptstyle(f)}}{{=}}$
$\displaystyle\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau_{P_{j}}}\end{subarray}}\left(1-pF_{C}\left(\frac{B}{\epsilon-\tau_{P_{j}}}\right)\right)\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon\leq\tau_{P_{j}}}\end{subarray}}\left(1-p\right)$ (-A.1)

Here, $(a)$ follows since
$L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P,j}\right)=\tau_{P_{j}}<\epsilon$
with probability one; $(b)$ holds since the RVs
$L_{i,j}^{t}\left(\frac{B}{C_{i,j}^{t}}+\tau_{P,j}\right)_{j\neq i_{t}}$ are
mutually independent; and $(c)$ stems from the low of total probability and
the fact that $L_{i_{t},j}$ takes the values of zero and one with
probabilities $1-p$ and $p$, respectively; In $(d)$ we split the equation into
two terms, where for $\epsilon<\tau_{P_{j}}$ the probability term equals zero;
$(e)$ is obtained by substituting $F_{C}(\epsilon)$ instead of
$\Pr(C_{i,j}^{t}<\epsilon|L_{i,j}^{t}=1)$; and $(f)$ follows since capacity is
non-negative, therefore $F_{C}(\mu)=0$ for $\mu\leq 0$, and particularly for
$\mu=\frac{B}{\epsilon-\tau_{P_{j}}}\leq 0$. Equation (-A.1) coincides with
(11), proving the theorem. ∎

###  -B Proof of Corollary 1

For $\tau_{P,j}\equiv\tau\leq\epsilon$ , it holds that

$\displaystyle\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau}\end{subarray}}\left(1-pF_{C}\left(\frac{B}{\epsilon-\tau_{P,j}}\right)\right)\\!=\\!\left(1-pF_{C}\left(\frac{B}{\epsilon-\tau}\right)\right)^{K\\!-\\!1}.$
(-B.1)

Now, for every fixed $\epsilon$ such that $\epsilon<T_{\max}$ it holds that
$F_{C}(\frac{B}{\epsilon\\!-\\!\tau})$ is strictly smaller than one, and thus
for every $p>0$ we have that
$\left(1-pF_{C}\left(\frac{B}{\epsilon-\tau}\right)\right)<1$ and thus (-B.1)
tends to zero as $K$ grows arbitrarily large. For
$\tau_{P,j}\equiv\tau>\epsilon$ , it holds that

$\displaystyle\prod_{\begin{subarray}{c}j\neq{i}^{t}\\\
{\epsilon>\tau}\end{subarray}}\left(1-p\right)=\left(1-p\right)^{K\\!-\\!1}.$
(-B.2)

Since $p>0$ we get that $\left(1-p\right)<1$ and thus (-B.2) also tends to
zero as $K$ grows arbitrarily large. Consequently, the product of (-B.1) and
(-B.2) approaches zero as $K\rightarrow\infty$, proving the corollary. ∎

## References

  * [1] N. Shlezinger, E. Farhan, H. Morgenstern, and Y. C. Eldar, “Collaborative inference via ensembles on the edge,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 8478–8482.
  * [2] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, p. 436, 2015.
  * [3] J. Chen and X. Ran, “Deep learning with edge computing: A review,” _Proc. IEEE_ , vol. 107, no. 8, pp. 1655–1674, 2019.
  * [4] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge computing: The communication perspective,” _IEEE Commun. Surveys Tuts._ , vol. 19, no. 4, pp. 2322–2358, 2017.
  * [5] Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge intelligence: Paving the last mile of artificial intelligence with edge computing,” _Proc. IEEE_ , vol. 107, no. 8, pp. 1738–1762, 2019.
  * [6] C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” _IEEE Commun. Surveys Tuts._ , vol. 21, no. 3, pp. 2224–2287, 2019.
  * [7] H. B. McMahan, E. Moore, D. Ramage, and S. Hampson, “Communication-efficient learning of deep networks from decentralized data,” _arXiv preprint arXiv:1602.05629_ , 2016.
  * [8] T. Gafni, N. Shlezinger, K. Cohen, Y. C. Eldar, and H. V. Poor, “Federated learning: A signal processing perspective,” _IEEE Signal Process. Mag._ , vol. 39, no. 3, pp. 14–41, 2022.
  * [9] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” _IEEE Signal Process. Mag._ , vol. 37, no. 3, pp. 50–60, 2020.
  * [10] X. Xu, Y. Ding, S. X. Hu, M. Niemier, J. Cong, Y. Hu, and Y. Shi, “Scaling for edge inference of deep neural networks,” _Nature Electronics_ , vol. 1, no. 4, pp. 216–222, 2018.
  * [11] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and acceleration for deep neural networks,” _arXiv preprint arXiv:1710.09282_ , 2017.
  * [12] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1389–1397.
  * [13] E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini, and L. V. Gool, “Soft-to-hard vector quantization for end-to-end learning compressible representations,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 1141–1151.
  * [14] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2704–2713.
  * [15] N. Shlezinger, N. Farsad, Y. C. Eldar, and A. J. Goldsmith, “Learned factor graphs for inference from stationary time sequences,” _IEEE Trans. Signal Process._ , vol. 70, pp. 366–380, 2022.
  * [16] N. Farsad, N. Shlezinger, A. J. Goldsmith, and Y. C. Eldar, “Data-driven symbol detection via model-based machine learning,” _arXiv preprint arXiv:2002.07806_ , 2020.
  * [17] N. Shlezinger, J. Whang, Y. C. Eldar, and A. G. Dimakis, “Model-based deep learning,” _arXiv preprint arXiv:2012.08405_ , 2020.
  * [18] L. Lin, X. Liao, H. Jin, and P. Li, “Computation offloading toward edge computing,” _Proc. IEEE_ , vol. 107, no. 8, pp. 1584–1607, 2019.
  * [19] I. V. Bajić, W. Lin, and Y. Tian, “Collaborative intelligence: Challenges and opportunities,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 8493–8497.
  * [20] R. A. Cohen, H. Choi, and I. V. Bajić, “Lightweight compression of intermediate neural network features for collaborative intelligence,” _IEEE Open Journal of Circuits and Systems_ , vol. 2, pp. 350–362, 2021.
  * [21] M. Merluzzi, A. Martino, F. Costanzo, P. Di Lorenzo, and S. Barbarossa, “Dynamic ensemble inference at the edge,” in _IEEE Global Communications Conference (GLOBECOM)_ , 2021.
  * [22] J. L. Burbank, P. F. Chimento, B. K. Haberman, and W. T. Kasch, “Key challenges of military tactical networking and the elusive promise of MANET technology,” _IEEE Commun. Mag._ , vol. 44, no. 11, pp. 39–45, 2006.
  * [23] O. Sagi and L. Rokach, “Ensemble learning: A survey,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 8, no. 4, p. e1249, 2018.
  * [24] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in neural information processing systems_ , 2017, pp. 6402–6413.
  * [25] H. Chen and A. Shrivastava, “Group ensemble: Learning an ensemble of convnets in a single convnet,” _arXiv preprint arXiv:2007.00649_ , 2020.
  * [26] H. Li, J. Y.-H. Ng, and P. Natsev, “Ensemblenet: End-to-end optimization of multi-headed models,” _arXiv preprint arXiv:1905.09979_ , 2019.
  * [27] B. Brazowski and E. Schneidman, “Collective learning by ensembles of altruistic diversifying neural networks,” _arXiv preprint arXiv:2006.11671_ , 2020.
  * [28] T. Raviv, N. Raviv, and Y. Be’ery, “Data-driven ensembles for deep and hard-decision hybrid decoding,” _arXiv preprint arXiv:2001.06247_ , 2020\.
  * [29] S. Fort, H. Hu, and B. Lakshminarayanan, “Deep ensembles: A loss landscape perspective,” _arXiv preprint arXiv:1912.02757_ , 2019.
  * [30] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [31] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” _ACM SIGARCH Computer Architecture News_ , vol. 45, no. 1, pp. 615–629, 2017.
  * [32] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “JointDNN: an efficient training and inference engine for intelligent mobile cloud computing services,” _IEEE Trans. Mobile Comput._ , vol. 20, no. 2, pp. 565–576, 2021.
  * [33] Z. Chen, K. Fan, S. Wang, L. Duan, W. Lin, and A. C. Kot, “Toward intelligent sensing: Intermediate deep feature compression,” _IEEE Trans. Image Process._ , vol. 29, pp. 2230–2243, 2019.
  * [34] A. Van Den Oord and O. Vinyals, “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [35] Z.-H. Zhou, _Ensemble methods: foundations and algorithms_. CRC press, 2012.
  * [36] Y. Liu and X. Yao, “Ensemble learning via negative correlation,” _Neural networks_ , vol. 12, no. 10, pp. 1399–1404, 1999.
  * [37] L. I. Kuncheva and C. J. Whitaker, “Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,” _Machine learning_ , vol. 51, no. 2, pp. 181–207, 2003.
  * [38] P. Bühlmann, “Bagging, boosting and ensemble methods,” in _Handbook of computational statistics_. Springer, 2012, pp. 985–1022.
  * [39] C. Shui, A. S. Mozafari, J. Marek, I. Hedhli, and C. Gagné, “Diversity regularization in deep ensembles,” _arXiv preprint arXiv:1802.07881_ , 2018\.
  * [40] S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, and S. Wang, “Image and video compression with neural networks: A review,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 30, no. 6, pp. 1683–1698, 2019.
  * [41] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” _arXiv preprint arXiv:1611.01704_ , 2016.
  * [42] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool, “Conditional probability models for deep image compression,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4394–4402.
  * [43] M. B. Mashhadi, Q. Yang, and D. Gündüz, “Distributed deep convolutional compression for massive MIMO csi feedback,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 4, pp. 2621–2633, 2020.
  * [44] N. Shlezinger and Y. C. Eldar, “Deep task-based quantization,” _Entropy_ , vol. 23, no. 1, p. 104, 2021.
  * [45] R. Torfason, F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool, “Towards image understanding from deep compression without decoding,” _arXiv preprint arXiv:1803.06131_ , 2018.
  * [46] N. Shlezinger, R. J. G. van Sloun, I. A. M. Hujiben, G. Tsintsadze, and Y. C. Eldar, “Learning task-based analog-to-digital conversion for MIMO receivers,” in _Proc. IEEE ICASSP_ , 2020.
  * [47] J. Surowiecki, _The wisdom of crowds_. Anchor, 2005.
  * [48] A. Cohen, N. Shlezinger, S. Salamatian, Y. C. Eldar, and M. Médard, “Serial quantization for sparse time sequences,” _IEEE Trans. Signal Process._ , vol. 69, pp. 3299–3314, 2021.
  * [49] N. Lang and N. Shlezinger, “Joint privacy enhancement and quantization in federated learning,” in _Proc. IEEE ISIT_ , 2022.
  * [50] A. Ghosh, J. Hong, D. Yin, and K. Ramchandran, “Robust federated learning in a heterogeneous environment,” _arXiv preprint arXiv:1906.06629_ , 2019.
  * [51] N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “UVeQFed: Universal vector quantization for federated learning,” _IEEE Trans. Signal Process._ , vol. 69, pp. 500–514, 2020.
  * [52] N. Shlezinger, S. Rini, and Y. C. Eldar, “The communication-aware clustered federated learning problem,” in _Proc. IEEE ISIT_ , 2020.
  * [53] C. Luo, X. He, J. Zhan, L. Wang, W. Gao, and J. Dai, “Comparison and benchmarking of AI models and frameworks on mobile devices,” _arXiv preprint arXiv:2005.05085_ , 2020.

  *[DNNs]: deep neural network
  *[DNN]: deep neural network
  *[AI]: artifical intelligence
  *[RV]: random variable
  *[RVs]: random variable
  *[SGD]: stochastic gradient descent
  *[VQ-VAE]: vector quantized variational autoencoder
  *[$\mathcal{Q}$]: codebook