LBS

    location based services
FD

    federated distillation
FL

    federated learning
RSSI

    radio signal strength indicator
IoT

    Internet of Things
ML

    machine learning
GPS

    global positioning system
LOS

    line of sight
NLOS

    non line of sight
GNSS

    global navigation satellites system
WSN

    wireless sensor network
CSI

    channel state information
UWB

    ultra-wide bandwidth
MLP

    multi-layer perceptron
ToA

    time of arrival
TDoA

    time difference of arrival
AoA

    angle of arrival
RFID

    radio frequency identification
MSE

    mean squared error
MAE

    mean absolute error
AP

    access point
RP

    reference position
CD

    co-distillation
KD

    knowledge distillation
NTK

    neural tangent kernel
DNN

    deep neural network
IPS

    indoor positioning paper
I.I.D

    independently and identically distributed
N.I.I.D

    non independently and identically distributed
GAN

    generative adversarial network
RMSE

    root mean squared error
SoA

    state-of-the-art
XR

    extended reality
LoRaWAN

    long range wide area network
MLP

    multi-layer perceptron

# Federated Distillation based Indoor Localization for IoT Networks

Yaya Etiabi  School of Computer Science, Mohammed VI Polytechnic University,
Benguerir, Morocco  Marwa Chafii  Engineering Division, New York University
(NYU) Abu Dhabi, UAE  NYU WIRELESS, NYU Tandon School of Engineering,
Brooklyn, NY
Email: {yaya.etiabi<EMAIL_ADDRESS><EMAIL_ADDRESS>El Mehdi
Amhoud  School of Computer Science, Mohammed VI Polytechnic University,
Benguerir, Morocco

###### Abstract

Federated distillation (FD) paradigm has been recently proposed as a promising
alternative to federated learning (FL), especially in wireless sensor networks
with limited communication resources. However, all state-of-the-art FD
algorithms are designed for classification tasks only and less attention has
been given to regression tasks. In this work, we propose an FD framework that
properly operates on regression learning problems. Afterwards, we present a
use-case implementation by proposing an indoor localization system that shows
a good trade-off communication load vs. accuracy compared to FL-based indoor
localization. With our proposed framework, we reduce the number of transmitted
bits by up to 98%. We analyze the energy efficiency regarding the number of
communication rounds it takes for both FD and FL systems to achieve the same
localization accuracy. The results revealed that FD comes with greater energy
efficiency by reasonably saving transmission energy at the expense of
computation energy. This is a substantial advantage for battery-powered IoT
systems with very limited bandwidth. Moreover, we show that the proposed
framework is much more scalable than FL, thus more likely to cope with the
expansion of wireless networks.

###### Index Terms:

Federated distillation, Localization, RSSI Fingerprinting, Internet of Things
(IoT), Wireless networks

##  I Introduction

Location based services play an important role in several applications by
providing targeted information to individuals or entities based on their
geographic location in real or near-real time, typically through wireless
communication networks. These applications include navigation, individual
tracking, emergency services, asset tracking, logistics planning, workforce
management, location-based advertising and social networking.

In recent decades, location based services (LBS) have grown considerably and
are now more than ever at the core of the digital revolution we are
witnessing. Their market is expected to reach $\$318.64$ billion in $2030$ [1]
due to the increase of demand from different sectors such as agriculture,
defense, transportation, energy, healthcare, etc. This growth is the direct
result of the evolvement of the underlying technologies centered around
wireless sensor networks.

Today with the advent of 5G [2], and the prospect of 6G [3], the number of
connected devices will grow at an unprecedented rate, resulting in the massive
deployment of Internet of Things (IoT) infrastructures [4]. Moreover, 6G comes
with new applications such as multisensory extended reality (XR) applications,
connected robotics and autonomous systems [5]. It is also expected to offer
sensing and localization as new services [6]. All these transformational
applications drive the need for accurate localization systems which require
lots of resources due to the massive deployment of IoT devices. Yet in the
literature, in addition to the range-free techniques such as centroid method
[7] and distance vector hop (DV-Hop) technique [8], typical ranging techniques
based on channel state information (CSI)[9] , angle of arrival (AoA) [10],
time of arrival (ToA) [11], time difference of arrival (TDoA) [12], and radio
signal strength indicator (RSSI) [13] using various wireless technologies such
as radio frequency identification (RFID) [14], ultra-wide bandwidth (UWB)
[15], WiFi [16], LoRaWAN [17], and Bluetooth [18] have been proposed for
indoor positioning. All of these techniques present a number of issues,
including low precision, high computational complexity, and unreliability due
to wireless channel impairments such as multipath effects caused by non line
of sight (NLOS) propagation in indoor environments, while most positioning
devices lack sufficient computing power. Moreover, these techniques require
the building of empirical models that capture all the channel effects
including noises, multipath fading, channel variations due to the indoor
movements, leading to a very high time complexity and a high dependence to
channel conditions.

Consequently, due to the difficulty to derive robust models that capture these
indoor channel impairments, researchers turn towards data-based localization
using machine learning (ML) which does not require empirical models but rather
uses constructed datasets that capture all variations in the indoor
environments. Indeed, ML is used to tackle the aforementioned limitations of
traditional methods as enlightened in the works in [19, 20, 21, 22, 23, 24].
In these works, we can see that ML is a very promising and game-changing
technology for IoT localization in the sense that it provides robust and
scalable localization systems with improved accuracy and relatively low
complexity[25].

Nonetheless, ML based localization systems require important data exchange
between IoT devices and the central server. To proceed the later, ML operation
with a respect of users privacy, federated learning, has been introduced.

FL has been firstly introduced by Google researchers in [26]. It is a ML
paradigm that enables end-devices to collaboratively learn a shared prediction
model while keeping all the training data on-device. Such approach decouples
the ability to do ML from the need to store the data in the cloud. Research
works in [27, 28, 29, 30] show the prominence of using such algorithm. Indeed,
in [31], authors have proposed a federated learning (FL) based localization
framework coined FedLoc, which richly addresses the privacy concern in
collaboration among numerous IoT devices.

However, in an IoT network where a massive number of sensors is deployed, FL
is not enough since when implemented over capacity-constrained communication
links, communication cost and latency may severely limit the performance of
FL. Therefore, federated distillation (FD) has been proposed in [32] to deal
with these issues. In fact, FD is a compelling distributed learning solution
that only exchanges the model outputs whose dimensions are commonly much
smaller than the model size. In the literature, FD has been used in several
applications implying only classification tasks such as handwritten digits
recognition [33] and image classification [34]. In this work, we consider a FD
based regression problem that predicts the location of an IoT device given its
RSSI measurements in that location. Unlike the state-of-the-art FD approaches,
our localization problem is a regression one, hence the need to develop a FD
framework for regression problems.

The motivation of this paper lays in three points: (i) Secure crowdsourcing
and decoupling the need to store the RSSI fingerprints data in a fusion center
from the ability to train the deep neural network (DNN) model. Indeed, RSSI
fingerprints are crowdsourced by a pool of IoT devices all over the network.
Instead of sending the crowd-sourced data to the fusion center, each device
keeps its data locally for privacy preservation and leverage it to train a
local model. The global model is then obtained through the federation of the
different local models. This is the fundamental of FL; (ii) Reducing as much
as possible the communication load involved in the federated training of the
localization model. In fact, the IoT devices namely FL clients are assumed to
have limited bandwidth leading to a constrained model parameters exchange.
Hence the introduction of FD for only output exchange which is much less
bandwidth consuming. Note that communication is more challenging for IoT
devices. Therefore, reducing the number of transmitted bits may lead to
greater energy efficiency of FD system for the same performance goal as FL;
(iii) Constructing a FD framework for regression problems for a better
implementation of the localization model which is a regression problem by
design. In fact, state-of-the-art FD algorithms do not deal with the
regression problems.

The main contributions of this work are summarized as follows:

  * •

We develop a FD framework for regression tasks since the previously proposed
FD algorithms are dealing with classification problems and no attention has
been given to the regression ones. We validate our FD framework through
performance evaluation using different publicly available experimental
datasets.

  * •

We propose an IoT localization system based on our proposed FD framework. We
prove that this localization framework works for indoor systems as well as for
outdoor systems. Also to the best of our knowledge, this work is the first to
tackle localization problem under a FD framework which considerably reduces
the communication complexity over bandwidth-constrained wireless networks.

The reminder of this paper is organized as follows. Section II presents
related works. In Section III, we describe our proposed FD framework followed
by its performance evaluation in Section IV. Finally, Section V concludes our
work and provides future research directions.

##  II Related Work

In this section, we present the works related to the following: RSSI
fingerprinting based localization, FL based IoT localization frameworks and
federated distillation.

###  II-A RSSI fingerprinting based localization

Being the key pillar of LBS, localization in IoT networks is a field of study
that has started very early with the first wireless sensor networks. At its
early stage, global navigation satellites system (GNSS) such as the global
positioning system (GPS) were widely used to perform sensor nodes localization
with some appropriate algorithms [35]. With the increase of sensor network
density and the expansion of IoT devices’ deployment mainly in indoor
environments, GPS solutions become too expensive and suffer from indoor
environment’s impairments. To address this issue, many developed localization
algorithms do not use GPS directly, but employ it as an assistance in some
cases [36, 37] while more efforts are made on the excavation of the IoT
network itself by exploiting wireless signal characteristics including CSI,
AoA, ToA, TDoA, RSSI. The most commonly used signal, due to its lowest cost
and high availability, is the RSSI which captures the power-distance
relationship of a signal propagation environment. Indeed, fingerprinting based
methods firstly construct a RSSI database by extracting the representative and
distinguishable parameters, namely fingerprint, from the received signals of
IoT devices at different locations, then use the constructed database to
predict the location corresponding to a new RSSI recording based on its
similarity with the recorded fingerprints.

However RSSI measurements are challenged by channel impairments such as the
multipath fading caused by the NLOS signals leading to high complexity of
analytical solutions. These problems are accentuated by the growth of IoT
networks which involves a systematic adaptation for localization schemes in
order to maintain the mainstream of LBS. Therefore, ML has been introduced to
maintain a good trade-off between accuracy and complexity as shown in [38, 39,
40] where many deep learning frameworks have been proposed for RSSI
fingerprinting based localization.

###  II-B FL based IoT localization frameworks

Although RSSI based fingerprinting has gathered researchers’ attention in the
past decade and has achieved a great breakthrough, it suffers from an inherent
problem: The localization accuracy degrades abruptly over time due to the very
dynamic environment and unstable wireless devices leading to a high
calibration effort for fingerprint collection.
Thus, by using conventional ML, we end up with a huge fingerprint database in
the server with sometimes privacy issues. Therefore, employing FL allows
keeping users private data locally, thus ensuring the privacy preservation and
reducing the dataset size while enabling an adequate localization performance.
Consequently, researchers are turning towards the design of localization
systems based on federated learning models. Indeed, the work in [41], is the
first to propose a federated learning framework called FLoc, an RSSI
fingerprinting based indoor localization scenario to solve the security
problem in fingerprint database updating localization. Similarly in [42, 43,
31], federated learning is driving the design of localization systems. Yet,
besides the great advantage of privacy preservation and training computation
load reduction that FL brings in the cited works, it fails to meet the
bandwidth and energy constraints of the real world IoT systems since the
communication load and energy consumption involved in the federated training
is not often suitable for IoT devices with very limited capabilities. Hence,
in this work, we overcome these limitations of FL by utilizing federated
distillation(FD). The first advantage of the FD lies indeed, in the
flexibility it brings to devices to train tailored model architectures. This
feature is beneficial for networks with heterogeneous constraints.
Additionally, FD comes with augmented robustness in the sens that adversarial
or malicious clients can not directly influence the parameterization of the
model during training. However, the most substantial benefits of FD resides in
its great communication efficiency compared to FL.

###  II-C Federated distillation

Federated distillation combines the ideas of knowledge distillation and
federated learning. Knowledge distillation [44] is a student-teacher approach
in which, a student model is helped in its training by a pretrained teacher
model in order to transfer the knowledge of the teacher to the student and
thus speed up the learning process. It is worth noting that both the student
and the teacher have access to the same dataset. A distributed version of this
approach namely co-distillation has been proposed in [45], where instead of
having a pretrained teacher, each student considers the aggregated knowledge
of the remaining students as its teacher, leading to an online teacher model
training alongside students models. FD can be seen as a derivative of co-
distillation as shown in [46] with the difference that each FD student
possesses its own data as in FL. It is proved therein through asymptotical
analysis leveraging the kernel method in [47] that the FD can achieve even
better performance than knowledge distillation specifically if in the latter,
the teacher model is not well pretrained. Therefore, recent works [32, 48, 49,
50, 51] introduces FD to tackle the limitations of FL in terms of bandwidth
requirements and energy constraints.

However, to date, all the proposed FD frameworks are targeting only
classification tasks and no attention has been given to regression tasks.
Since several real-world ML target variables are continuous and need to be
treated as regression problems, we first propose a federated distillation
framework for regression. Then, we leverage it to build an indoor localization
system.

##  III Proposed Framework

###  III-A Localization system model

In traditional settings, RSSI data from multiple anchor nodes are collected
through a crowdsourcing system and merged in a server for centralized training
of the DNN based localization model. In this work, we substitute this by a
decentralized approach where collected RSSI data are no longer sent to the
server. Instead, a pool of IoT devices, namely FD clients is selected to
crowdsource RSSI data all over the network. Then, each selected device train a
local model with its crowdsourced data. In FD training setting, FD clients
share their knowledge by exchanging their models’ outputs so that at the end
of the training, the local models predict nearly the same positions given new
RSSI measurements. Thus, for a node to be localized in this network, it
records RSSI from in-range anchors and sends them to the nearest FD client
which outputs the corresponding position by running its model. Note that the
predicted position remains essentially the same as if the prediction was done
by any other FD client. Indeed, the models are trained in a manner that
provides them a global knowledge of the overall network. The FD system
architecture and the whole FD training process are described in the next
sections.

###  III-B Federated distillation system architecture

The overall framework contains three main components namely the parameter
server called coordinator also known as teacher, the clients or workers also
known as students, and the communication system.

####  III-B1 Server

The server (coordinator or teacher) is the entity that supervises the whole FD
training process. It starts by defining all the common parameters such as the
total number of communication rounds, the number of clients to be selected for
each communication round, the wireless communication settings. The server is
also in charge of defining the global DNN model. The necessary information to
reproduce locally the global model is then broadcasted to the clients by the
help of the communication system.

####  III-B2 Clients

The clients (workers or students) are in charge of training their respective
local models using their private datasets. During the training, workers
periodically upload their local average estimations per each target segment to
the parameter server which will aggregate them to obtain a global estimation
per each corresponding segment. The global estimations from the server are
then used by the clients to update their respective loss functions for the
next local training phase as shown in Fig. 1(a).

####  III-B3 Communications system

The role of the communication system as its name suggests is to deal with the
wireless communication techniques used to communicate the model and the target
estimations from the server to clients and vice-versa. This module is further
discussed in Section III-F.

(a) Illustration of segment-based knowledge sharing

(b) Target variable segmentation

Figure 1: Illustration of FD based Regression training process. Figure 2:
3-Layer Multi-Layer Perceptron Architecture.

###  III-C Problem formulation

The objective of our proposed framework is to cooperatively train a DNN using
a communication-efficient federated learning scheme namely federated
distillation, in a network of IoT devices. For the sake of convergence
analysis in section III-E, we consider a 3-layer neural network comprising an
input layer, a hidden layer and output layer with respectively $N_{i}$,
$N_{h}$, and $N_{o}$ neurons as shown in Fig. 2. Given an input vector
$X_{i}$, the prediction of the target $y_{i}$ is given by
$\hat{y}_{i}=F_{\Theta}(X_{i})$, where $F_{\Theta}$ is the function
representing our DNN and $\Theta=[W\;\alpha]$(see Fig. 2) is the set of the
DNN’s weights. Then, for a standalone training, the goal is to minimize the
loss function given by:

$\mathcal{L}(y,\hat{y})=\sum_{i}\mathcal{L}(y_{i},\hat{y}_{i}),$ (1)

where

$\hat{y}_{i}=F_{\Theta}(X_{i})=\frac{1}{\sqrt{N_{h}}}\sum_{n=1}^{N_{h}}\alpha_{n}\ell_{n}(X_{i}),$
(2)

with $\ell_{n}(X_{i})=\sigma(W_{n}^{T}X_{i})$ is the logit representing the
output of the $n^{th}$ neuron of the last hidden layer, $\sigma(\cdot)$ a non-
linear activation function, and $W_{n}$ the weights vector of the $n^{th}$
neuron. For the localization framework, the input $X_{i}$ is a vector
containing the RSSI measurements from all the access points in the network and
the target variable $y_{i}$ is the 2-D coordinates of the location where these
RSSIs have been recorded. To simplify the notations we will consider
$F_{\Theta}(X_{i})=F(X_{i})$ in the next sections.

In vanilla distillation process so called knowledge distillation, each student
independently learns the knowledge of the teacher by adding a regularization
term regarding the gap between its prediction and the teacher’s. Ideally,
similar to the distillation in classification context in [45], the
distillation for a learning task can be formulated as:

$\mathcal{L}(y,\hat{y})=\underbrace{\sum_{i}\mathcal{L}\left(y_{i},\hat{y}_{i}\right)}_{\text{loss
}}+\lambda\underbrace{\sum_{i}\sum_{n=1}^{N_{h}}\mathcal{L}\left(\ell_{n}(X_{i}),L_{n}(X_{i})\right)}_{\text{distillation
regularizer }},$ (3)

where $L_{n}(X_{i})$ is the pretrained teacher $n^{th}$ output logit given an
input $X_{i}$ and $\lambda$ is the regularization coefficient, assuming that
the teacher and the student have access to the same dataset.

On the other hand, co-distillation (CD) is a distributed version of this
system where workers have to distill knowledge from each other. In fact, the
main principle of CD is to consider a set of predictions from several
individual models as the teacher’s knowledge, which is frequently more correct
than the individual predictions. In this configuration, each student sees the
ensemble of remaining students as its teacher. Consequently, (3) is
transformed into:

$\mathcal{L}(y,\hat{y})=\sum_{k}\sum_{i}\left[\mathcal{L}\left(y_{i},\hat{y}_{i}\right)+\lambda\sum_{n}\mathcal{L}\left(L_{n}^{k}(X_{i}),\ell_{n}^{k}(X_{i})\right)\right],$
(4)

where $L_{n}^{k}(X_{i})=\frac{1}{K-1}\sum_{k\prime\neq
k}\ell_{n}^{k\prime}(X_{i}),\forall k$ with $K$ the number of students. Note
that this loss function encompasses all the individual losses for students
since this distributed student-teacher configuration creates an
interdependence in the distillation process. However, in a federated learning
setting, both KD and CD are not realistic since the data privacy and
heterogeneity are the cornerstone of federated learning systems. In the
considered IoT network, IoT devices are assumed to possess only their own
datasets. Consequently, similarly to the technique used to train
classification models with FD where the samples are grouped by labels at each
device and the mean prediction of each label group is shared with the network,
we develop a FD scheme for regression problems. Indeed, for regression
problems, since we do not have classes as labels, the output variable being
continuous, we will end up exchanging a vector with equal size to the dataset,
which is not only in contradiction with our objective of reducing the number
of transmitted bits but also unpractical regarding the non independently and
identically distributed aspect of the local datasets with respect to the
output variables. Thus, in order to get closer to the technique used for FD in
classification context, under the mild assumption on the boundaries of the
target values, the target $y$ is divided into $\mathbf{\mathcal{S}}$ segments
as shown in Fig. 1(b) so that the average predictions per segment are sent to
the server for knowledge sharing (Fig. 1(a)). Each segment $s$ is defined by a
half-opened interval $\mathcal{I}_{s}$ defined as follows:

$\mathcal{I}_{s}=\left\\{\begin{matrix}\left]-\infty,y^{1}\right[&if&s=1\\\
\\\ \left[y^{s-1},y^{s}\right[&if&1<s<\mathcal{S}\\\ \\\
\left[y^{s-1},+\infty\right[&if&s=\mathcal{S}\\\ \\\ \end{matrix}\right.,$ (5)
$\text{where }y^{s}=y_{min}+s\times\epsilon\text{ with
}\epsilon=\frac{y_{max}-y_{min}}{\mathcal{S}}$

######  Remark 1.

The number of segments $\mathcal{S}$ is a hyper-parameter depending on the
resolution $\epsilon$ set for the learning task, i.e, regarding a regression
problem, the resolution $\epsilon$ can be the maximum tolerable error in
estimation. Then $\mathcal{S}$ is given by

$\mathcal{S}=\frac{y_{max}-y_{min}}{\epsilon}.$

######  Remark 2.

Since the data is distributed across devices, computing the min and the max
requires the server to know the boundaries of each local dataset, which may
violate the privacy of the federated system. As such, one approach to get this
done securely is through a multiparty computation scheme between devices.
However, for the sake of computation and communication efficiency, the server
can set them arbitrary or request devices to upload their local min and max
using cryptographic tools with order preserving encryption to preserve
devices’ data privacy.

For a client $k$ in the network, we define the following settings.

$\left\\{\begin{matrix}\begin{aligned} \mathcal{I}_{s}^{k}&=\\{i\\}\text{ such
that }y_{i}\in\mathcal{I}_{s}\\\ &\text{ and
}(X_{i},y_{i})\in\mathcal{D}_{k},\text{ with
}\left|\mathcal{I}_{s}^{k}\right|=N_{s}^{k}\\\
\bar{\ell}_{n,s}^{k}&=\frac{1}{N_{s}^{k}}\sum_{i\in\mathcal{I}_{s}^{k}}\ell_{n,s}^{k}(X_{i})\\\
L_{n,s}^{k}&=\frac{1}{K-1}\sum_{k\prime\neq
k}\bar{\ell}_{n,s}^{k\prime}\end{aligned}\end{matrix}\right.,$ (6)

where $\mathcal{I}_{s}^{k}$ is the partition of $k^{th}$ client’s dataset
belonging to the $s^{th}$ segment. $\ell^{k}_{n,s}(x)$ is the $n^{th}$ output
logit of client $k$ when the input $x$ belongs to segment $s$,
$\bar{\ell}_{n,s}^{k}$ and $L_{n,s}^{k}$ are respectively the local and global
average of the $n^{th}$ output logit over the samples in segment $s$.

The above formulations leads to the new global loss function of the federated
system given by:

$\displaystyle\mathcal{L}(y,\hat{y})=$
$\displaystyle\sum_{k}\sum_{s}\left(\sum_{i\in\mathcal{I}_{s}^{k}}\mathcal{L}\left(y_{i},\hat{y}_{i}\right)+\right.$
(7)
$\displaystyle\left.\lambda\sum_{i\in\mathcal{I}_{s}^{k}}\sum_{n}\mathcal{L}\left(L_{n,s}^{k},\ell_{n,s}^{k}(X_{i})\right)\right).$

###  III-D FD for regression algorithm

Considering only the computation aspect of our framework, the overall process
of solving the problem in (7) is described by Algorithm 1 which comprises both
the client side pseudo-code and the server’s.

In fact, in this configuration, in contrast to knowledge distillation, we do
not have a pretrained teacher model. So both the student and the teacher are
learning during the process especially since the teacher knowledge for a given
student is actually the aggregated knowledge of the remaining students.

As a consequence, at the initial communication round, it is not possible to
obtain the regularization term of the loss function since no prediction has
been done yet. That is why, as enlightened in “line 1 - line 3” of Algorithm
1, we start by training the local models for a few steps. Until then, the
distillation process can start (“line 6 - line 36”).

The overall process in Algorithm 1 can be summarized in the following steps:

  1. 1.

Each client trains its local model using its private dataset and stores
locally the average estimations per segment.

  2. 2.

Each client periodically uploads its local average estimations per segment to
the server.

  3. 3.

The server computes the global average estimations per segment by averaging
all the local average estimations per segment sent by all the clients.

  4. 4.

Each client downloads the global average estimations per segment and updates
its loss function according to (7).

  5. 5.

Each client repeats all the above steps until convergence.

Inputs : $\\{\mathcal{D}_{k}\\}$ ;

/* Client’s Datasets */

1

Inputs : $\mathcal{L}(\cdot)$ ;

/* Loss function */

2

3

Outputs : $\\{\mathcal{W}_{k}\\}$ ;

/* Clients trained models parameters */

4

5 foreach  _Client $k\in\\{1,2,\dots,K\\}$_ do

6 Local training of parameters $\\{\mathcal{W}_{k}\\}$ using
$\\{\mathcal{D}_{k}\\}$

7 end foreach

8$r\leftarrow 1$ ;

9 get $\\{L^{k}_{s,r}\\}$ ;

10 while  _$r <max\\_com\\_rounds$_ do

11 ClientLocalTraining
_$(\mathcal{D}_{k},\mathcal{W}_{k},\\{L^{k}_{s,r}\\})$_

12

13 foreach  _Client $k\in\\{1,2,\dots,K\\}$_ do

/* In parallel */

14 for  _$e\leftarrow 1$ to $\\#Epochs$_ do

15 foreach  _batch $b\in\mathcal{D}_{k}$_ do

16 foreach  _$X_{i,b},y_{i,b}\in b$_ do

17 $\begin{aligned}
&\mathcal{W}_{k}\leftarrow\mathcal{W}_{k}-\eta\nabla\\{\mathcal{L}\left(\hat{y}_{i,b},y_{i,b}\right)+\\\
&\lambda\cdot\mathcal{L}\left(L_{s(y_{i,b}),r}^{k},\ell_{s(y_{i,b}),r}^{k}(X_{i,b})\right)\\}\end{aligned}$;

18 $\begin{aligned}
\bar{\ell}_{s(y_{i,b}),r}^{k}\leftarrow&\bar{\ell}_{s(y_{i,b}),r}^{k}+\\\
&\ell_{s(y_{i,b}),r}^{k}(X_{i,b})\;\end{aligned}$;

19

20 $N_{s(y_{i,b}),r}^{k}\leftarrow N_{s(y_{i,b}),r}^{k}+1$;

21

22 end foreach

23

24 end foreach

25

26 end for

27 for  _$s\leftarrow 1$ to $S$_ do

28 $\bar{\ell}^{k}_{s,r}\leftarrow\bar{\ell}^{k}_{s,r}/N_{s,r}^{k}$;

29

30 end for

31

32 end foreach

33 return _$\\{\bar{\ell}^{k}_{s,r}\\}$_ ;

34

35

36 ServerknowledgeDistillation  _$(\\{\bar{\ell}_{s,r}^{k}\\})$_

37 for  _$s\leftarrow 1$ to $S$_ do

38 foreach  _Client $k\in\\{1,2,\dots,K\\}$_ do

39 $L^{k}_{s,r}\leftarrow L^{k}_{s,r}+\bar{\ell}^{k}_{s,r}$;

40

41 end foreach

42

43 end for

44 foreach  _Client $k\in\\{1,2,\dots,K\\}$_ do

45 for  _$s\leftarrow 1$ to $S$_ do

46 $L^{k}_{s,r+1}\leftarrow\frac{K\cdot
L^{k}_{s,r}-\bar{\ell}^{k}_{s,r}}{K-1}$;

47

48 end for

49

50 end foreach

51 return _$\\{L^{k}_{s,r+1}\\}$_ ;

52

53

54 $r\leftarrow r+1$ ;

55

56 end while

Algorithm 1 Federated Distillation for Regression

###  III-E Convergence analysis

After applying our segmentation technique and considering the mean square
error (MSE) as the loss function, i.e
$\mathcal{L}(y,\hat{y})=\mathbf{MSE}(y,\hat{y})$ in (7), we globally try to
solve the following optimization problem:

$\displaystyle\min_{\mathcal{W}_{1},\mathcal{W}_{2},\dots,\mathcal{W}_{K}}$
$\displaystyle\sum_{k}\sum_{s}\left[\sum_{i\in\mathcal{I}_{s}^{k}}\left(y_{i}-\hat{y}_{i}\right)^{2}+\right.$
(8)
$\displaystyle\left.\lambda\sum_{i\in\mathcal{I}_{s}^{k}}\sum_{n}\left(L_{n,s}^{k}-\ell_{n,s}^{k}(X_{i})\right)^{2}\right].$

The convergence of the latter optimization problem can be established through
the asymptotical analysis of the two foundational building blocks of federated
distillation namely knowledge distillation (KD) [44] and Co-Distillation
(CD)[45] by exploiting the theory of neural tangent kernel (NTK)[47].

####  III-E1 Convergence analysis of Knowledge distillation

The problem of KD in this configuration is cast into the minimization of the
objective function defined as follows:

$\displaystyle J(W_{k})=$
$\displaystyle\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\left(y_{i}-\hat{y}_{i}\right)^{2}+$
(9)
$\displaystyle\lambda\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\sum_{n=1}^{N_{h}}\left(\ell_{n}\left(X_{i}\right)-L_{n}\left(X_{i}\right)\right)^{2},$

where $W_{k}$ contains the weights of the the model of a client $k$. We assume
that the activation function $\sigma(\cdot)$ in the hidden layers is Lipschitz
continuous and differentiable. The first derivative $\sigma^{\prime}(\cdot)$
is also assumed to be Lipschitz continuous. Using the gradient descent
algorithm with an infinitesimal step-size to solve the problem in (3) in the
NTK settings [47] results in the following dynamics of the weights:

$\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}=\sum_{s}\mathbf{\Phi}_{n}^{s}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\hat{\mathbf{y}^{s}}\right)+\lambda\left(\mathbf{L}_{n}^{s}-\boldsymbol{\ell}_{n}^{s}\right)\right]$
(10)

###### Proof.

In the kernel regime[47], with an infinitesimal step size $\eta$, the gradient
descent iterations depicted by
$W_{k}^{n}\left(\tau+1\right)=W_{k}^{n}\left(\tau\right)-\eta\nabla
J\left(W_{k}^{n}\left(\tau\right),\mathcal{D}\right),\forall n=1,2,...,N_{h}$
can be translated into continuous time domain variation given by:

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$
$\displaystyle=-\frac{\mathrm{d}}{\mathrm{d}\mathbf{W}_{k}^{n}}\mathbf{J}\left(W_{k},\mathcal{D}_{k}\right)$
$\displaystyle=-\frac{\mathrm{d}}{\mathrm{d}\mathbf{W}_{k}^{n}}\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\left(\mathbf{y}_{i}-\frac{1}{\sqrt{N_{h}}}\sum_{n}\alpha_{n}\sigma\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\right)^{2}$
$\displaystyle+\lambda\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\sum_{n}\left(\mathbf{L}_{n}(X_{i})-\sigma\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\right)^{2}$
$\displaystyle=\sum_{s}2\sum_{i\in\mathcal{I}_{s}^{k}}X_{i}\cdot\sigma^{\prime}\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\left(\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}_{i}-\hat{\mathbf{y}}_{i}\right)\right)$
$\displaystyle+\lambda\sum_{s}2\sum_{i\in\mathcal{I}_{s}^{k}}X_{i}\cdot\sigma^{\prime}\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\left(\mathbf{L}_{n}(X_{i})-\ell_{n}\left(X_{i}\right)\right)$

Let $\mathbf{\Phi}_{n}^{s}$ be the line vector containing the values
$X_{i}\cdot\sigma^{\prime}\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)$,
$\mathbf{y}^{s}$ the vector of labels $y_{i}$ such that
$i\in\mathcal{I}_{s}^{k}$. Similarly, $\boldsymbol{\ell}_{n}^{s}$ and
$\mathbf{L}_{n}^{s}$ are the vectors containing the $n^{th}$ predicted logits
of all the samples with labels in segment $s$ (i.e
$y_{i}/i\in\mathcal{I}_{s}^{k}$) for the student and the teacher respectively.
Then, it follows that:

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$
$\displaystyle=\sum_{s}\mathbf{\Phi}_{n}^{s}\left(\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\mathbf{\hat{y}}^{s}\right)\right)$
$\displaystyle+\lambda\sum_{s}\mathbf{\Phi}_{n}^{s}\left(\mathbf{L}_{n}^{s}-\boldsymbol{\ell}_{n}^{s}\right)$
$\displaystyle=\sum_{s}\mathbf{\Phi}_{n}^{s}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\hat{\mathbf{y}^{s}}\right)+\lambda\left(\mathbf{L}_{n}^{s}-\boldsymbol{\ell}_{n}^{s}\right)\right]$

which is the results in (10). ∎

Note that (10) reflects the modelling by a continuous-time differential
equation of the convergence of a trajectory of the discrete algorithm to a
smooth curve. Due to the difficulty to analyze the dynamics of the weights,
(10) is translated in dynamics of the logits as:

$\frac{\mathrm{d}}{\mathrm{d}\tau}\boldsymbol{\ell}_{k}^{n}=\sum_{s}\left(\mathbf{\Phi}_{n}^{s}\right)^{T}\mathbf{\Phi}_{n}^{s}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\hat{\mathbf{y}^{s}}\right)+\lambda\left(\mathbf{L}_{n,k}^{s}-\boldsymbol{\ell}_{n,k}^{s}\right)\right].$
(11)

Indeed,

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\tau}\boldsymbol{\ell}_{k}^{n}(X_{i})$
$\displaystyle=\frac{\mathrm{d}}{\mathrm{d}\mathbf{W}_{k}^{n}}\boldsymbol{\ell}_{k}^{n}(X_{i})\cdot\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$
$\displaystyle=\frac{\mathrm{d}}{\mathrm{d}\mathbf{W}_{k}^{n}}\sigma\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\cdot\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$
$\displaystyle=X_{i}\cdot\sigma^{\prime}\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\cdot\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$

Therefore,

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}\tau}\boldsymbol{\ell}_{k}^{n}$
$\displaystyle=\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\frac{\mathrm{d}}{\mathrm{d}\tau}\boldsymbol{\ell}_{k}^{n}(X_{i})$
$\displaystyle=\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}X_{i}\cdot\sigma^{\prime}\left(\left(\mathbf{W}_{k}^{n}\right)^{T}X_{i}\right)\cdot\frac{\mathrm{d}}{\mathrm{d}\tau}\mathbf{W}_{k}^{n}$
$\displaystyle=\sum_{s}\sum_{i\in\mathcal{I}_{s}^{k}}\left(\Phi_{n}^{s}(X_{i})\cdot\sum_{s^{\prime}}\mathbf{\Phi}_{n}^{s^{\prime}}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s^{\prime}}-\hat{\mathbf{y}^{s^{\prime}}}\right)\right.\right.$
$\displaystyle\left.\left.+\lambda\left(\mathbf{L}_{n}^{s^{\prime}}-\boldsymbol{\ell}_{n}^{s^{\prime}}\right)\right]\right)$
$\displaystyle=\sum_{s}(\Phi_{n}^{s})^{T}\Phi_{n}^{s}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\hat{\mathbf{y}^{s}}\right)+\lambda\left(\mathbf{L}_{n}^{s}-\boldsymbol{\ell}_{n}^{s}\right)\right].$

Let $\mathbf{\Psi}_{n}^{s}=(\mathbf{\Phi}_{n}^{s})^{T}\mathbf{\Phi}_{n}^{s}$.
Then, (11) becomes:

$\frac{\mathrm{d}}{\mathrm{d}\tau}\boldsymbol{\ell}_{k}^{n}=\sum_{s}\mathbf{\Psi}_{n}^{s}\left[\frac{\alpha_{n}}{\sqrt{N_{h}}}\left(\mathbf{y}^{s}-\hat{\mathbf{y}^{s}}\right)+\lambda\left(\mathbf{L}_{n,k}^{s}-\boldsymbol{\ell}_{n,k}^{s}\right)\right]$
(12)

######  Remark 3.

The matrix $\mathbf{\Psi}$ is referred to as a neural tangent kernel (NTK)
[47] whose theory establishes that the random initialization of the weights
matrix and the over-parameterization hypothesis induce together a kernel
regime such that $\mathbf{\Psi}(\tau)\sim\mathbf{\Psi}(0),\forall\tau>0$. This
remark emanates from the fact that in over-parameterization setting, each
weight vector along the trajectory of the gradient descent algorithm remains
constant over time and maintains a very tight relationship to its initial
value.

######  Lemma 1.

The student network output converges asymptotically as follows:

$\lim_{\tau\rightarrow\infty}\mathbf{F^{s}}(\tau)=\mathbf{F}^{s}_{\infty}=\frac{1}{\alpha+\lambda}\left(\alpha\mathbf{y}^{s}+\lambda\sum_{n=1}^{N_{h}}\frac{\alpha_{n}\mathbf{L}_{n}^{s}}{\sqrt{N_{h}}}\right),\forall
s$ (13)

where $\alpha=\sum_{n=1}^{N_{h}}\frac{\alpha_{n}}{N_{h}}\alpha_{n}^{2}$.

###### Proof.

The theory of linear systems with a finite order [52] has established that a
linear dynamics of finite order can roughly represent the gradient descent
behavior on the over-parameterized neural network, and the evolution of
$\mathbf{F^{s}}(\tau)$ can be written as:

$\mathbf{F}^{s}(\tau)=\mathbf{F}^{s}_{\infty}+\sum_{j=1}^{d}\phi^{s}_{j}e^{-z^{s}_{j}\tau},\forall
s$ (14)

with $d$ representing the order of the linear system.
$\\{\phi^{s}_{j}\\}_{j=1}^{d}$ are complex-value vectors determined by the
specifications of the dynamics, whereas $\\{z^{s}_{j}\\}_{j=1}^{d}$ are the
poles of the linear system. Note that the non-zero complex-values
$\\{z^{s}_{j}\\}_{j=1}^{d}$, $z^{s}_{j}\neq 0$ correspond the singular points
of the Laplace transform of $\mathbf{F^{s}}(\tau)$ except the zero component
$z^{s}_{j}=0$ which corresponds to the constant $\mathbf{F}^{s}_{\infty}$.

Thanks to the assumptions on the eigenvalues of the matrices $\mathbf{\Psi}$
at initialization (i.e, under mild assumptions on eigenvalues of
$\mathbf{\Psi}(0)$) in the kernel regime [47], [52] has shown that with
bounded inputs, bounded weights, all existing poles are positive-valued,
yielding
$\lim_{\tau\rightarrow\infty}\mathbf{F^{s}}(\tau)=\mathbf{F}^{s}_{\infty}$.
Moreover, the analysis in infinite width regime of the neural network
($N_{h}\rightarrow\infty$) with the set assumptions demonstrates the
calculation of $\mathbf{F}^{s}_{\infty}$ and produces the result in (13). ∎

######  Remark 4.

The student’s prediction error with respect to the teacher’s can be estimated
by:

$\mathbf{e}^{s}=\left\|\mathbf{F}^{s}_{\infty}-\mathbf{y}^{s}\right\|_{2}=\frac{\lambda}{\alpha+\lambda}\left\|\mathbf{y}^{s}-\sum_{n}\frac{\alpha_{n}\mathbf{L}^{s}_{n}}{\sqrt{N_{h}}}\right\|_{2},\forall
s.$ (15)

This result implies that the learning accuracy of the student is highly
correlated to the quality of the teacher in representing labels since for an
imperfect teacher, the error $\mathbf{e}^{s}$ monotonically increases with
$\lambda$.

####  III-E2 Federated distillation

Federated distillation extends knowledge distillation to co-distillation where
each worker possesses its own data and see the other workers as its teacher,
leading to an online knowledge distillation. Overall, the objective is the
minimization of the global loss function defined in (8). Individually, each
student minimizes its local loss function $J(\mathbf{W}_{k})$ using knowledge
distillation where the teacher knowledge is the ensemble prediction of the
remaining students.

######  Lemma 2.

Based on the results of lemma 1, and the interactions between students, our
optimization problem in (8) converges asymptotically as:

$\lim_{r\rightarrow\infty}\mathbf{F}_{k}(r)=\mathbf{y},\forall
k\in\\{1,2,\ldots,K\\},$ (16)

where $r$ designates the $r^{th}$ communication round.

###### Proof.

The first iteration of knowledge distillation is performed after
initialization and a first run of the gradient descent by all students.
Consequently, the first updates i.e, $\\{\mathbf{F}^{k}(0)\\}_{k=1}^{K}$ are
shared. Then each student $k$ locally runs gradient descent and, by lemma 1,
its model converges to:

$\mathbf{F}^{k}(1)=\frac{1}{\alpha+\lambda}\left(\alpha\mathbf{y}+\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}^{k^{\prime}}(0)\right).$ (17)

Therefore, at the $r^{th}$ communication round, the output of the model of the
student $k$ converges to:

$\displaystyle\mathbf{F}_{k}(r)$
$\displaystyle=\frac{1}{\alpha+\lambda}\left[\alpha\mathbf{y}+\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}_{k^{\prime}}(r-1)\right]$ (18)
$\displaystyle=\frac{1}{\alpha+\lambda}\left[\alpha\mathbf{y}+\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\left(\frac{1}{\alpha+\lambda}\left[\alpha\mathbf{y}+\frac{\lambda}{K-1}\sum_{k^{"}\neq
k^{\prime}}^{K}\mathbf{F}_{k^{"}}(r-2)\right]\right)\right]$ (19)
$\displaystyle=\frac{1}{\alpha+\lambda}\left[\alpha\mathbf{y}+\frac{\lambda}{(K-1)(\alpha+\lambda)}\left(\alpha(K-1)\mathbf{y}+\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\sum_{k^{"}\neq k^{\prime}}^{K}\mathbf{F}_{k^{"}}(r-2)\right)\right]$
(20)
$\displaystyle=\frac{1}{\alpha+\lambda}\left[\alpha\mathbf{y}+\frac{\lambda}{(K-1)(\alpha+\lambda)}\left(\alpha(K-1)\mathbf{y}+\lambda\mathbf{F}_{k}(r-2)+\frac{\lambda(K-2)}{K-1}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}_{k^{\prime}}(r-2)\right)\right]$ (21)

On the other hand, (18) can be rewritten as:

$(\alpha+\lambda)\mathbf{F}_{k}(r)-\alpha\mathbf{y}=\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}_{k^{\prime}}(r-1).$ (22)

Then, let

$u_{r+1}=\frac{\lambda}{K-1}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}_{k^{\prime}}(r)=(\alpha+\lambda)\mathbf{F}_{k}(r+1)-\alpha\mathbf{y}.$
(23)

By introducing $u_{r+1}$ in (21), we can construct a linear non-homogeneous
recurrence relation given by:

$\displaystyle\mathbf{u}_{r+1}$
$\displaystyle=\frac{(K-2)\lambda}{(K-1)(\alpha+\lambda)}\mathbf{u}_{r}+\frac{\lambda^{2}}{(K-1)(\alpha+\lambda)^{2}}\mathbf{u}_{r-1}$
(24)
$\displaystyle+\frac{\lambda^{2}\alpha\mathbf{y}}{(K-1)(\alpha+\lambda)^{2}}+\frac{\lambda\alpha\mathbf{y}}{(\alpha+\lambda)}.$

The resolution of such recurrence relation as presented in [53], provides a
close-form solution defined as:

$\mathbf{u}_{r}=\lambda\mathbf{y}+\gamma\left(\frac{\lambda}{\alpha+\lambda}\right)^{r}+\rho\left(-\frac{\lambda}{(K-1)(\alpha+\lambda)}\right)^{r}$
(25)

with
$\gamma=\frac{\lambda}{K}\sum_{k=1}^{K}\mathbf{F}^{k}(0)-\lambda\mathbf{y}$
and
$\rho=\frac{\lambda}{K(K-1)}\sum_{k^{\prime}\neq
k}^{K}\mathbf{F}^{k^{\prime}}(0)-\frac{\lambda}{K}\mathbf{F}^{k}(0)$.
Combining (22) and (23), we obtain the following:

$\mathbf{F}^{k}(r)=\frac{1}{\alpha+\lambda}(\alpha\mathbf{y}+\mathbf{u}_{r}).$
(26)

It follows that:

$\lim_{r\rightarrow\infty}\mathbf{F}^{k}(r)=\frac{1}{\alpha+\lambda}(\alpha\mathbf{y}+\lim_{r\rightarrow\infty}\mathbf{u}_{r})=\mathbf{y},$
(27)

due to the fact that
$\lim_{r\rightarrow\infty}\mathbf{u}_{r}=\lambda\mathbf{y}$, for $K\geq 2$.
Indeed for $K\geq 2$, $\left|-\frac{\lambda}{(K-1)(\alpha+\lambda)}\right|<1$
and $\left|\frac{\lambda}{\alpha+\lambda}\right|<1$. ∎

Extensive numerical evaluation using simulation as well as experimental data
in Section IV confirm the convergence property of our FD system.

###  III-F Communication System

####  III-F1 Energy consumption

In a FL system as well as a FD system, regarding the IoT network resources
limitations, the overall system efficiency heavily relies on the communication
system since the energy consumption $\mathcal{E}$ of every single device
participating in the training process is expressed as:

$\mathcal{E}=\left(\mathcal{E}_{\mathrm{C}}+\mathcal{E}_{\mathrm{T}}\right)\times\mathcal{N}_{\mathrm{T}},$
(28)

where $\mathcal{E}_{C}$ is the computation energy of each device during the
local training at each training step or communication round.
$\mathcal{E}_{T}$, the transmission energy, is the quantity of energy used by
each device to transmit its parameters to the server at each communication
round, and $\mathcal{N}_{\mathrm{T}}$ is the total number of training steps
till convergence.

From (28), we can see that the total energy consumption depends on three major
components namely: $(i)$ the ML model parameters size, $(ii)$ the local
computation and $(iii)$ the number of communication rounds.

Yet our proposed FD framework by design solves the problem $(i)$. In fact,
instead of exchanging model parameters, FD devices only transmit their
predictions to the server which greatly reduces the communication cost.
However, this reduction causes a slight accentuating of the problem $(ii)$
because of the regularization term in the devices updated loss functions.
Lastly, the problem $(iii)$ is data dependant. Therefore, for this work,
assuming the same number of communication rounds for both FL and FD, we only
focus on the transmission energy $\mathcal{E}_{T}$ which is the dominant part
of the system overall energy consumption.

To clearly illustrate the communication behaviour of our framework, we
consider a simple multiple access communication system. In fact, on the
uplink, FD clients share a Gaussian multiple-access channel whose equation is
given by:

$\mathbf{y}_{r}=\sum_{c=1}^{C}\mathbf{h}_{r}^{c}\mathbf{x}_{r}^{c}+\mathbf{z}_{r},$
(29)

where $\mathbf{x}_{r}^{c}$ is the signal to be transmitted by client $c$ at
the $r^{th}$ communication round, $\mathbf{h}_{r}^{c}$ the corresponding
channel response and $\mathbf{z}_{r}$ an additive independently and
identically distributed (I.I.D) Gaussian noise with
$\mathbf{z}_{r}\sim\mathcal{N}(0,1)$. For the sake of simplicity, the downlink
communication is assumed to be noiseless so that we can focus on the more
challenging shared uplink. Thus, for FD, at each iteration, each client
transmits its local average estimations per segment over a wireless shared
uplink channel with the access point connected to the server. Assuming that
the target variables have dimension $N_{o}$ each divided into $\mathcal{S}$
segments, the output vector to be transmitted is given by:

$\mathbf{V}=\begin{bmatrix}\mathbf{\bar{f}}_{1}^{c}\\\
\mathbf{\bar{f}}_{2}^{c}\\\ \vdots\\\ \mathbf{\bar{f}}_{s}^{c}\\\ \vdots\\\
\mathbf{\bar{f}_{\mathcal{S}}}^{c}\end{bmatrix}\ \ \ \text{with}\ \ \
\mathbf{\bar{f}}_{s}^{c}=\begin{bmatrix}\mathbf{\bar{f}}_{s}^{c}(1)\\\
\mathbf{\bar{f}}_{s}^{c}(2)\\\ \vdots\\\
\mathbf{\bar{f}}_{s}^{c}(L)\end{bmatrix}^{T}.$ (30)

Thus, unlike FL where each device transmits $W\times R$ bits to the server for
each communication round, in FD the total number of transmitted bits per
device per communication round is given by:

$\mathcal{N}_{bits}=\mathcal{S}\times N_{o}\times R,$ (31)

where $R$ is the bits resolution. The total number of parameters of our
considered neural network is defined as:

$W=(N_{i}\times N_{h})+N_{h}+(N_{h}\times N_{o})+N_{o}.$ (32)

So obviously, it follows that $\mathcal{S}\times N_{o}\ll W$ which makes our
model much more communication-efficient than FL. This lightweightness of our
system overcomes the bandwidth and energy limitations since each device
involved in the training process is subject to the power constraint
$\mathrm{E}\left[\left\|\mathrm{x}_{r}^{c}\right\|_{2}^{2}\right]/T\leq\mathcal{P}$
[54], where $T$ is the number of channels and $\mathcal{P}$ is the maximum
transmit power. Indeed, considering a conventional digital implementation as
in [55], the channel’s uplink capacity is equally shared by the devices
leading to bandwidth constraints limiting the number of bits $B$ that can be
transmitted by each device. In fact, considering the Shannon’s capacity [56],
the maximum number of bits per transmission for each device is given by:

${B}_{max}^{r,c}=\frac{T}{C}\log_{2}(1+C\left|\mathbf{h}_{r}^{c}\right|^{2}\mathcal{P}),\text{
so that }{B}\leq{B}_{max}^{r,c}.$ (33)

To sum up, this constraint does not affect too much the FD system since the
condition $\mathcal{N}_{bits}<B_{max}^{r,c}$ is very likely to always hold as
demonstrated by the results in Section IV.

##  IV Performance Evaluation

###  IV-A Indoor Localization with Federated Distillation

####  IV-A1 System model and data collection

Our goal is to construct a DNN based indoor positioning system (IPS) to
predict the location of an IoT device given its RSSI measurements in that
location. Therefore, the dataset is an ensemble of measurements of RSSI from
different access points at different locations and at different time to
capture the channel variations. To do so, we consider an indoor IoT network in
an area of interest of size $l\times w\ m^{2}$ containing $M$ APs randomly
distributed over the area. For the data crowdsourcing, we consider a pool of
$C$ FD clients. Each client will collect RSSI data from the environment to
train a local model. The environment is characterized by its path loss
exponent $\beta$ and its sigma-shadowing $\sigma$ representing the channel
attenuation.
In this localization system we exploit the RSSI values gathered from different
APs in the network. Consequently, for the data collection we consider $N$
reference positions such that at each RP we capture $M$ RSSI values from the
$M$ APs, taking into account the out-of-range APs. At each captured RSSI, the
corresponding RP is added as a label. Moreover, at each RP, the operation is
repeated for $k$ time intervals in order to capture the variations and
different impairments experienced by the wireless channel.

####  IV-A2 Simulated RSSI data

The RSSI measurements are strongly time-space varying and depend on the path
loss model for the corresponding environment. In this work, similar to work in
[57], we use the log-distance path loss model formulated as follows:

$PL=PL_{0}+10\beta\log_{10}{\frac{d}{d_{0}}}+X_{\sigma},$ (34)

where $\beta$ is the path loss exponent, $d_{0}$ is the reference distance,
$X_{\sigma}$ represents the log-normal shadowing with standard deviation
$\sigma$ in dB. $PL_{0}$ is the path loss at the reference distance $d_{0}$
and given by the free space propagation model as:

${PL_{0}}=10\log_{10}\left(\left({4\pi
d_{0}\cdot\frac{f}{c}}\right)^{2}\right),$ (35)

with $f$ the frequency of the signal and $c$ the speed of light.

Then the RSSI is computed as follows:

$\displaystyle RSSI$ $\displaystyle=P_{Rx}=P_{Tx}-PL.$ (36)

Consequently, the RSSI value captured from the $m^{th}$ AP at the $n^{th}$ RP
at time $t$ can be expressed as:

$\displaystyle RSSI_{n,t}^{m}$
$\displaystyle=P_{Tx}-\left[20\log_{10}\left(\frac{4\pi
d_{0}}{c}\right)+20\log_{10}\left(f\right)+\right.$ (37)
$\displaystyle\left.10\beta\log_{10}\left(\frac{d_{n}^{m}}{d_{0}}\right)+X_{\sigma,{n,m}}^{t}\right].$

###  IV-B Simulation Results

(a) Convergence of student models

(b) Performance comparison

(c) Estimation errors of FD model

Figure 3: Regression based FD framework evaluation on simulation data.

####  IV-B1 Set-up

For the simulation, we consider a Wi-Fi powered IoT network of $l\times
w=20\times 20m^{2}$ with $M=10$ APs where the data are collected from $N=100$
RPs. The environment variables $\beta$ and $\sigma$ are set to $3.23$ and $2$
respectively according to the experimental measurements conducted in [58]. The
number of repetitions is set to $T=10$. In order to make the dataset
reproducible, the RSSI values are generated using (37) with a random seed set
to 200.
The objective being the training of DNN localization models using federated
distillation, we set the number of clients (workers or students) $C$ to $5$.
For the DNN models, we consider a multi-layer perceptron (MLP) characterized
by the configuration given in TABLE I.

TABLE I: Parameters of the MLP model  Parameter |  Description |  Value
---|---|---
Optimizer |  DNN Model Optimizer |  Adam [59]
$\mu$ |  Learning rate |  0.0001
$\beta_{1},\beta_{2}$ |  Exponential decay rates |  $0.1,0.99$
$N_{i}$ |  Input layer units |  10
$N_{h}$ |  Hidden layer units |  $1\times 1000$
$N_{o}$ |  Output layer units |  2
$\sigma_{h}(\cdot)$ |  Hidden Activation function |  ReLu
$\sigma_{o}(\cdot)$ |  Output Activation function |  Linear
$\mathcal{C}$ |  Number of students (clients) |  5
$b$ |  Batch size |  32
$\mathcal{R}$ |  Communication rounds for FL and FD |  100
$\lambda$ |  Regularization term |  0.1

####  IV-B2 Results

The models are trained by the $C$ students and the FD server using Algorithm
1. The evaluation is done using the mean absolute error (MAE) metric defined
by:

$\displaystyle MAE$
$\displaystyle=\mathbb{E}\left[\mathbf{y}-\mathbf{\hat{y}}\right]=\frac{1}{N}\sum_{i=1}^{N}\left\|\mathbf{y}_{i}-\mathbf{\hat{y}}_{i}\right\|_{1}$
(38)
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\frac{\left|x_{i}-\hat{x}_{i}\right|+\left|y_{i}-\hat{y}_{i}\right|}{2},$

where $\mathbf{y}_{i}=[x_{i}\ y_{i}]^{T}$ and
$\mathbf{\hat{y}}_{i}=[\hat{x}_{i}\ \hat{y}_{i}]^{T}$ are respectively the
true and estimated locations corresponding to the $i^{th}$ RSSI recording in
the validation dataset. The results are presented in Fig. 3. In fact, we can
see from Fig. 3(a) that all the students’ models converge to the same MAE
value, meaning that the federated students are effectively learning from each
other in order to consolidate their models. To analyse the performance of our
framework, we consider different learning scenarios namely federated learning
(FL), centralized learning (CL) which is the traditional training approach,
and standalone learning (SL) where the student train independently its model.
As such, in Fig. 3(b), a single data point per reference position is chosen to
feed the trained models and the predictions are compared to the ground-truth
labels. We can see that the FD model improves the localization accuracy
compared to the SL since the students share their knowledge with each other.
Nonetheless, it remains less accurate than FL and CL due to the nature of its
operating mode. It is important to note that this accuracy decreases is the
price of a huge communication gain which is the principal goal of this work.
Indeed, in terms of communication complexity as shown in TABLE II, with a bits
resolution $R=32$, the number of segments $\mathcal{S}=10$ and the output
dimension $N_{o}=2$ using $C=5$ students, the FD model is far better than the
FL. In fact, with FD, only $5\times 640=3200$ bits (31) are transmitted at
each round in comparison to $5\times 416064=2080320$ bits (32) for FL, leading
to a FD-to-FL ratio of $0.15\%$. Consequently, FD can save up to $99.85\%$ of
the transmission energy $\mathcal{E}_{T}$ used in FL while remaining only
$1.6\times$ less accurate than FL.

Finally, we can see from Fig. 3(c) how accurate is our localization framework
by observing the estimation errors plotted therein. The network overall root
mean squared error (RMSE) is computed for each model and the results are
presented in TABLE II. These results attest that our framework is suitable for
localization in indoor IoT network since it presents relatively good accuracy
regarding the geometry of the network and preserves network resources.

###  IV-C Experimental validation and benchmark

(a) Convergence of FD models

(b) Models comparison

(c) Comparison for MAE = 10m

Figure 4: Regression based FD framework evaluation on UJIIndoorLoc
experimental data.

In this part, we implement our framework using experimental localization data
as well as other regression tasks data. First of all, we perform indoor
localization using UJIIndoorLoc WiFi fingerprinting dataset [60] collected on
a campus area of approximately $390m\times 270m$ containing $21048$ entries.
The results are presented in Fig. 4. Fig. 4(a) shows the convergence of all
the MAE of all students to the same MAE value which confirms the convergence
of the proposed FD algorithm. In Fig.4(b), our proposed FD model is
benchmarked with the newly introduced FL model in [43] in the context of
indoor localization using the same dataset in [60]. Note that we used the same
DNN model presented in [43] on all the datasets. In order to highlight the
relevance of our FD framework, we keep showing the results of the standalone
model (independent learning) as well as the centralized system whenever
possible. It comes out that the FD learning slightly improves the estimation
accuracy compared to its standalone (baseline) model, while remaining less
accurate than the FL approach.

The benefit of FD over FL in terms of energy efficiency is seen in Fig.4(c)
where the number of communication rounds and the transmitted bits of FD are
compared with those of FL for a fixed accuracy (MAE = 10m). In fact, even
though it takes several rounds, FD transmits fewer bits to achieve the same
accuracy as FL. Due to the fact that the computation energy is lower in
comparison to the transmission energy, the FD still consumes significantly
less energy when compared to the FL. This is a substantial advantage for IoT
systems with very limited bandwidths. Therefore, for the same localization
accuracy, the proposed FD framework comes with a greater energy efficiency by
a reasonable saving of transmission energy. Simply put, in addition to the
slight increase of the accuracy of FD regarding its baseline model, the main
advantage of this framework over FL relies in its communication efficiency as
shown in TABLE III. Therein, it can be seen that FD can save more than $98\%$
of the transmitted bits per each communication round. Consequently, the
transmission energy $\mathcal{E}_{T}$ is minimized.

Moreover, we perform outdoor localization in an urban long range wide area
network (LoRaWAN) using the dataset reported in [61] and diamond prices
prediction using the dataset in [62]. We also present the results of the
baseline models used in the original works where the datasets are presented.
The results are summarized in TABLE III for the communication complexity and
TABLE IV for the accuracy. Note that for fair comparison between different
systems, the same number of epochs are used for standalone training as well as
for centralized training considering the number of local epochs per round and
the number of communication rounds in the federated settings. It worth noting
that for UJIIndoorLoc and LoRa datasets, the target variables have dimension 2
(2-D localization) while diamond dataset’s is 1-D (the diamond’s price). The
results assert that our framework can be used for any localization system in
wireless networks and by extent, for any regression task since it has also
shown great performance with the diamond pricing task which is a typical
regression problem.

TABLE II: Network global RMSE Model |  RMSE(m) |  bits/round
---|---|---
FD Model |  0.35 |  3200
Standalone Model |  0.38 |  -
FL Model |  0.21 |  2080320
Central Model |  0.20 |  -
TABLE III: Comparison of bits/round on different datasets Model |  UJIIndoorLoc[60] |  Urban LoRa[61] |  Diamonds Pricing[62]
---|---|---|---
FD Model |  3200 |  3200 |  1600
FL Model |  1739520 |  267520 |  147360
FD-to-FL ratio |  0.18% |  1.19% |  1.08%
TABLE IV: Comparison of RMSE (m) on different datasets Model |  UJIIndoorLoc[60] (MAE) |  Urban LoRa[61] |  Diamonds Pricing[62]
---|---|---|---
FD Model |  19.75 (9.91) |  483.42 |  178.46
Standalone Model |  20.67 (10.80) |  484.48 |  206.76
FL Model |  11.02 (5.76) |  438.44 |  46.99
Central Model |  10.81 (5.66) |  440.35 |  40.51
Baseline Model |  7.98 (-) |  398.40 |  462.62

###  IV-D Scaling Capability

In this part, we discuss the scalability potential of our framework by varying
the number of students involved in the learning process. The accuracy in terms
of RMSE and the number of transmitted bits $\mathcal{N}_{bits}$ are
highlighted, and the results are presented in Fig. 5.
Fig. 5(a) shows that as the number of students increases, the system accuracy
keeps improving for both FL and FD with more effect on the FL system. This
observation is straightforward since in FD, students contribute to the
learning by only sharing their output in contrast to FL where all the models’
parameters are shared in the network. On the other hand, Fig. 5(b) presents
the communication loads in terms of the total number of transmitted bits by
all students to the server, showing the prominence of using FD. We observe
that this results confirms (31) and effectively the FD is more bandwidth
conservative than FL. The FD/FL ratios of these metrics are shown in Fig. 5(c)
where we can see that in FD we transmit only $1.46\%$ of the number of bits
transmitted in FL, making FD $\sim 98,54\%$ more efficient than FL in terms of
communication while remaining only $\sim 1\times\text{ to }2.7\times$ less
accurate.

(a) Estimation error

(b) Transmitted bits per round

(c) FD to FL metrics ratio

Figure 5: Regression based FD scalabity analysis, UJIIndoorLoc dataset[60]

(a) Segmentation strategies

(b) Strategy 1

(c) Strategy 2

Figure 6: Impact of segmentation on data distribution, UJIIndoorLoc
dataset[60] Figure 7: Impact of segmentation on system accuracy Figure 8:
Impact of the distillation regularizer on the FD system accuracy

###  IV-E The impact of segmentation and data distribution

In this part, we analyse the impact of the segmentation on the system
performance. Indeed, we consider two segmentation strategies as shown in Fig.
6(a). In the first strategy in Fig. 6(b), a uniform split which is so far used
in this work and was explained in Fig. 1(b), we divide the output target into
equal size segments referred to as intervals. So depending on the data
distribution, some intervals may be empty or at least very less populated
because the target value is far from being uniformly distributed.
Consequently, the private datasets of clients will only contain a subset of
segments playing the role of labels here. This results in missing label
problem which needs to be properly mitigated depending on the dataset.

Alternatively, we design a second segmentation strategy in Fig. 6(c), a
density-based split in which each segment/interval possesses the same number
of samples, meaning that their sizes are necessarily different. This will
consequently affect the accuracy of the models as shown in Fig. 7, so that at
the end, the choice of a strategy relies on the data distribution. Moreover,
intuitively, increasing the number of segments should improve the models
accuracy but actually as shown in Fig. 7, it can in contrast degrades the
model performance if the distribution impairments induced by the segmentation
is not properly mitigated.

###  IV-F Impact of the distillation regularizer

The FD students learn from their teachers by distilling the teachers’
knowledge weighted by a regularizer. This regurlaizer is an hyperparameter of
the FD system that plays an import role in how much the students learn from
their teachers. Consequently, the choice of the regurlarizer impacts directly
the students models accuracy. In Fig 8, we show the effect of the regularizer
$\lambda$ in the FD based indoor localization with the UJIIndoorLoc dataset.
We can observe that for this application, the smaller the regularizer the
better the accuracy. Thus, a good choice of $\lambda$ for this application has
to meet the condition $\lambda<1$. This result is in accordance with the
theoretical analysis conducted in Section III-E, where it has been shown that
the student prediction error increase with the value of $\lambda$ in the
presence of imperfect teacher model.

##  V Conclusion

In this work, we have presented a novel federated distillation framework meant
for regression problems and leverage it to design an indoor IoT localization
system. Our solution has shown remarkable results in terms of bandwidth and
energy saving compared to FL based localization systems. The strength of our
framework relies in its novelty and its communication efficiency since it can
reduce the communication load by $\sim 98\%$ compared to FL framework. In
addition to its communication efficiency, our proposed framework presents a
great scalability capability making it suitable for large applications in IoT
networks. However, despite its great performance, this framework can be
further improved by tackling its limitations such as the impairments induced
by the segmentation and devices heterogeneity in terms of computation and
communication capabilities.

## References

  * [1] A. M. Research, “Location-based services market to reach $318.64 billion in 2030,” _GlobeNewswire News Room_ , November 2020, accessed: 2022-02-15.
  * [2] S. Li, L. D. Xu, and S. Zhao, “5G internet of things: A survey,” _Journal of Industrial Information Integration_ , vol. 10, pp. 1–9, 2018\.
  * [3] W. Jiang, B. Han, M. A. Habibi, and H. D. Schotten, “The road towards 6G: A comprehensive survey,” _IEEE Open Journal of the Communications Society_ , vol. 2, pp. 334–366, 2021.
  * [4] M. Jouhari, E.-M. Amhoud, N. Saeed, and M.-S. Alouini, “A survey on scalable LoRaWAN for Massive IoT: Recent advances, potentials, and challenges,” _ArXiv_ , vol. abs/2202.11082, 2022.
  * [5] W. Saad, M. Bennis, and M. Chen, “A vision of 6G wireless systems: Applications, trends, technologies, and open research problems,” _IEEE Network_ , vol. 34, no. 3, pp. 134–142, 2020.
  * [6] C. De Lima, D. Belot, R. Berkvens, A. Bourdoux, D. Dardari, M. Guillaud, M. Isomursu, E.-S. Lohan, Y. Miao, A. N. Barreto, M. R. K. Aziz, J. Saloranta, T. Sanguanpuak, H. Sarieddeen, G. Seco-Granados, J. Suutala, T. Svensson, M. Valkama, B. Van Liempd, and H. Wymeersch, “Convergent communication, sensing and localization in 6G systems: An overview of technologies, opportunities and challenges,” _IEEE Access_ , vol. 9, pp. 26 902–26 925, 2021.
  * [7] L. Ru and L. Zhang, “A weighted centroid localization algorithm for wireless sensor networks based on weight correction,” in _9th International Conference on Advanced Infocomm Technology (ICAIT)_ , 2017, pp. 165–169.
  * [8] P. Wang, F. Xue, H. Li, Z. Cui, L. Xie, and J. Chen, “A multi-objective DV-Hop localization algorithm based on NSGA-II in internet of things,” _Mathematics_ , vol. 7, 2019.
  * [9] X. Tong, Y. Wan, Q. Li, X. Tian, and X. Wang, “CSI fingerprinting localization with low human efforts,” _IEEE/ACM Transactions on Networking_ , vol. 29, no. 1, pp. 372–385, 2020.
  * [10] S. Wang, X. Jiang, and H. Wymeersch, “Cooperative localization in wireless sensor networks with AOA measurements,” _IEEE Transactions on Wireless Communications_ , pp. 1–1, 2022.
  * [11] S. Wu, S. Zhang, and D. Huang, “A TOA-based localization algorithm with simultaneous NLOS mitigation and synchronization error elimination,” _IEEE Sensors Letters_ , vol. 3, no. 3, pp. 1–4, 2019.
  * [12] B. Jin, X. Xu, and T. Zhang, “Robust time-difference-of-arrival (TDOA) localization using weighted least squares with cone tangent plane constraint,” _Sensors_ , vol. 18, no. 3, p. 778, 2018.
  * [13] S. Yiu, M. Dashti, H. Claussen, and F. Perez-Cruz, “Wireless RSSI fingerprinting localization,” _Signal Processing_ , vol. 131, pp. 235–244, 2017.
  * [14] Y. Zeng, X. Chen, R. Li, and H.-Z. Tan, “UHF RFID indoor positioning system with phase interference model based on double tag array,” _IEEE Access_ , vol. 7, pp. 76 768–76 778, 2019.
  * [15] D. Ni, O. A. Postolache, C. Mi, M. Zhong, and Y. Wang, “UWB indoor positioning application based on kalman filter and 3D TOA localization algorithm,” in _11th International Symposium on Advanced Topics in Electrical Engineering (ATEE)_ , 2019, pp. 1–6.
  * [16] R. Joseph and S. B. Sasi, “Indoor positioning using WiFi fingerprint,” in _International Conference on Circuits and Systems in Digital Enterprise Technology (ICCSDET)_. IEEE, 2018, pp. 1–3.
  * [17] Y. Etiabi, M. Jouhari, and E.-M. Amhoud, “Spreading factor and RSSI for localization in LoRa networks: A deep reinforcement learning approach,” _ArXiv_ , vol. abs/2205.11428, 2022.
  * [18] A. Satan and Z. Toth, “Development of bluetooth based indoor positioning application,” in _IEEE international conference on future IoT technologies (Future IoT)_. IEEE, 2018, pp. 1–6.
  * [19] A. Nessa, B. Adhikari, F. Hussain, and X. N. Fernando, “A survey of machine learning for indoor positioning,” _IEEE Access_ , vol. 8, pp. 214 945–214 965, 2020.
  * [20] C. Jiang, J. Shen, S. Chen, Y. Chen, D. Liu, and Y. Bo, “UWB NLOS/LOS classification using deep learning method,” _IEEE Communications Letters_ , vol. 24, pp. 2226–2230, 10 2020.
  * [21] M. Anjum, M. A. Khan, S. A. Hassan, A. Mahmood, H. K. Qureshi, and M. Gidlund, “RSSI fingerprinting-based localization using machine learning in lora networks,” _IEEE Internet of Things Magazine_ , vol. 3, no. 4, pp. 53–59, 2020.
  * [22] G. Cerar, A. Švigelj, M. Mohorčič, C. Fortuna, and T. Javornik, “Improving CSI-based massive MIMO indoor positioning using convolutional neural network,” in _2021 Joint European Conference on Networks and Communications 6G Summit (EuCNC/6G Summit)_ , 2021, pp. 276–281.
  * [23] M. Arnold, J. Hoydis, and S. ten Brink, “Novel massive MIMO channel sounding data applied to deep learning-based indoor positioning,” in _12th International ITG Conference on Systems, Communications and Coding (SCC 2019)_ , 2019.
  * [24] P. Ferrand, A. Decurninge, and M. Guillaud, “DNN-based localization from channel estimates: Feature design and experimental results,” in _GLOBECOM 2020 - 2020 IEEE Global Communications Conference_ , 2020, pp. 1–6.
  * [25] K. S. Kim, S. Lee, and K. Huang, “A scalable deep neural network architecture for multi-building and multi-floor indoor localization based on Wi-Fi fingerprinting,” _Big Data Analytics_ , vol. 3, no. 1, pp. 1–17, 2018.
  * [26] H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” _ArXiv_ , vol. abs/1602.05629, 2016\.
  * [27] N. H. Tran, W. Bao, A. Zomaya, M. N. H. Nguyen, and C. S. Hong, “Federated learning over wireless networks: Optimization model design and analysis,” _IEEE INFOCOM 2019 - IEEE Conference on Computer Communications_ , pp. 1387–1395, 2019.
  * [28] C. T. Dinh, N. H. Tran, M. N. H. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, “Federated learning over wireless networks: Convergence analysis and resource allocation,” _IEEE/ACM Trans. Netw._ , vol. 29, no. 1, p. 398–409, feb 2021.
  * [29] J. Zhou, S. Zhang, Q. Lu, W. W. Dai, M. Chen, X. Liu, S. Pirttikangas, Y. Shi, W. Zhang, and E. Herrera‐Viedma, “A survey on federated learning and its applications for accelerating industrial internet of things,” _ArXiv_ , vol. abs/2104.10501, 2021.
  * [30] M. Bennis, “Federated learning and control at the wireless network edge,” _GetMobile: Mobile Comp. and Comm._ , vol. 24, no. 3, p. 9–13, jan 2021\.
  * [31] F. Yin, Z. Lin, Y. Xu, Q. Kong, D. Li, S. Theodoridis, and Shuguang, “FedLoc: Federated learning framework for data-driven cooperative localization and location data processing,” _IEEE Open Journal of Signal Processing_ , vol. 1, pp. 187–215, 2020.
  * [32] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under Non-IID private data,” _ArXiv_ , vol. abs/1811.11479, 2018.
  * [33] D. Li and J. Wang, “FedMD: Heterogenous federated learning via model distillation,” _ArXiv_ , vol. abs/1910.03581, 2019.
  * [34] F. Sattler, A. Marban, R. Rischke, and W. Samek, “CFD: Communication-efficient federated distillation via soft-label quantization and delta coding,” _IEEE Transactions on Network Science and Engineering_ , pp. 1–1, 2021.
  * [35] B. Cheng, R. Du, B. Yang, W. Yu, C. Chen, and X. Guan, “An accurate GPS-based localization in wireless sensor networks: A GM-WLS method,” in _40th International Conference on Parallel Processing Workshops_ , 2011, pp. 33–41.
  * [36] Y. Etiabi, E. M. Amhoud, and E. Sabir, “A distributed and collaborative localization algorithm for internet of things environments,” _Proceedings of the 18th International Conference on Advances in Mobile Computing & Multimedia_, p. 114–118, 2020.
  * [37] Y. Etiabi, E. Amhoud, and E. Sabir, “Huber estimator and statistical bootstrap based light-weight localization for IoT systems,” _7th International Symposium on Ubiquitous Networking, UNet 2021_ , vol. 12845, pp. 79–92, 2020.
  * [38] W. Njima, R. Zayani, M. Terre, and R. Bouallegue, “Deep CNN for indoor localization in IoT-sensor systems,” _Sensors (Switzerland)_ , vol. 19, 7 2019.
  * [39] I. B. F. de Almeida, M. Chafii, A. Nimr, and G. Fettweis, “Blind transmitter localization in wireless sensor networks: A deep learning approach,” _IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , pp. 1241–1247, 2021.
  * [40] L. Gogolak, S. Pletl, and D. D. Kukolj, “Indoor fingerprint localization in WSN environment based on neural network,” _2011 IEEE 9th International Symposium on Intelligent Systems and Informatics_ , pp. 293–296, 2011.
  * [41] Y. Liu, H. Li, J. Xiao, and H. Jin, “FLoc: Fingerprint-based indoor localization system under a federated learning updating framework,” _15th International Conference on Mobile Ad-Hoc and Sensor Networks, MSN_ , pp. 113–118, 12 2019.
  * [42] W. Li, C. Zhang, and Y. Tanaka, “Pseudo label-driven federated learning-based decentralized indoor localization via mobile crowdsourcing,” _IEEE Sensors Journal_ , vol. 20, pp. 11 556–11 565, 10 2020.
  * [43] B. S. Ciftler, A. Albaseer, N. Lasla, and M. M. Abdallah, “Federated learning for localization: A privacy-preserving crowdsourcing method,” _ArXiv_ , vol. abs/2001.01911, 2020.
  * [44] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _NIPS Deep Learning and Representation Learning Workshop_ , 2015\. [Online]. Available: http://arxiv.org/abs/1503.02531
  * [45] R. Anil, G. Pereyra, A. Passos, R. Ormándi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,” _ArXiv_ , vol. abs/1804.03235, 2018.
  * [46] H. Seo, J. Park, S. Oh, M. Bennis, and S.-L. Kim, “Federated knowledge distillation,” _ArXiv_ , vol. abs/2011.02367, 2020.
  * [47] A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , p. 8580–8589, 2018.
  * [48] F. Sattler, A. Marban, R. Rischke, and W. Samek, “Communication-efficient federated distillation,” _arXiv preprint_ , vol. arXiv:2012.00632, 2020.
  * [49] S. Oh, J. Park, E. Jeong, H. Kim, M. Bennis, and S.-L. Kim, “Mix2FLD: Downlink federated learning after uplink federated distillation with two-way mixup,” _IEEE Communications Letters_ , vol. 24, no. 10, pp. 2211–2215, 2020\.
  * [50] M. Takamoto, Y. Morishita, and H. Imaoka, “An efficient method of training small models for regression problems with knowledge distillation,” _IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_ , pp. 67–72, 2020.
  * [51] S. Itahara, T. Nishio, Y. Koda, M. Morikura, and K. Yamamoto, “Distillation-based semi-supervised federated learning for communication-efficient collaborative training with Non-IID private data,” _IEEE Transactions on Mobile Computing_ , 2021.
  * [52] A. Rahbar, A. Panahi, C. Bhattacharyya, D. P. Dubhashi, and M. H. Chehreghani, “On the unreasonable effectiveness of knowledge distillation: Analysis in the kernel regime,” _ArXiv_ , vol. abs/2003.13438, 2020.
  * [53] K. H. Rosen, _Discrete mathematics and its applications_ , 7th ed. McGraw-Hill, 2012.
  * [54] J.-H. Ahn, O. Simeone, and J. Kang, “Wireless federated distillation for distributed edge learning with heterogeneous data,” _IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , pp. 1–6, 2019.
  * [55] J.-H. Ahn, O. Simeone, and J. Kang, “Cooperative learning via federated distillation over fading channels,” _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 8856–8860, 2020.
  * [56] T. M. Cover and J. A. Thomas, _Elements of Information Theory_. John Wiley & Sons, Ltd, April 2005, ch. 9, pp. 261–299.
  * [57] W. Njima, M. Chafii, A. Chorti, R. M. Shubair, and H. V. Poor, “Indoor localization using data augmentation via selective generative adversarial networks,” _IEEE Access_ , vol. 9, pp. 98 337–98 347, 2021.
  * [58] W. Njima, M. Chafii, A. Nimr, and G. Fettweis, “Deep learning based data recovery for localization,” _IEEE Access_ , vol. 8, pp. 175 741–175 752, 2020.
  * [59] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, May 7-9_ , 2015.
  * [60] J. Torres-Sospedra, R. Montoliu, A. M. Usó, J. P. Avariento, T. J. Arnau, M. Benedito-Bordonau, and J. Huerta, “UJIIndoorLoc: A new multi-building and multi-floor database for wlan fingerprint-based indoor localization problems,” _International Conference on Indoor Positioning and Indoor Navigation (IPIN)_ , pp. 261–270, 2014.
  * [61] M. Aernouts, R. Berkvens, K. Van Vlaenderen, and M. Weyn, “Sigfox and LoRaWAN datasets for fingerprint localization in large urban and rural areas,” _MDPI Data_ , vol. 3, no. 2, p. 13, 2018.
  * [62] S. Agrawal, “Analyze diamonds by their cut, color, clarity, price, and other attributes,” https://www.kaggle.com/shivam2503/diamonds, accessed: 2022-01-30.

  *[LBS]: location based services
  *[IoT]: Internet of Things
  *[XR]: extended reality
  *[CSI]: channel state information
  *[AoA]: angle of arrival
  *[ToA]: time of arrival
  *[TDoA]: time difference of arrival
  *[RSSI]: radio signal strength indicator
  *[RFID]: radio frequency identification
  *[UWB]: ultra-wide bandwidth
  *[NLOS]: non line of sight
  *[ML]: machine learning
  *[FL]: federated learning
  *[FD]: federated distillation
  *[DNN]: deep neural network
  *[GNSS]: global navigation satellites system
  *[GPS]: global positioning system
  *[CD]: co-distillation
  *[I.I.D]: independently and identically distributed
  *[APs]: access point
  *[RP]: reference position
  *[MLP]: multi-layer perceptron
  *[RMSE]: root mean squared error
  *[LoRaWAN]: long range wide area network