# A Quantization-based Technique for Privacy Preserving Distributed Learning

Maurizio Colombo<EMAIL_ADDRESS>Rasool Asal<EMAIL_ADDRESS>Ernesto Damiani<EMAIL_ADDRESS>Lamees Mahmoud AlQassem
<EMAIL_ADDRESS>Al Anoud Almemari<EMAIL_ADDRESS>Yousof
Alhammadi<EMAIL_ADDRESS>
###### Abstract

The massive deployment of Machine Learning (ML) models raises serious concerns
about data protection. Privacy-enhancing technologies (PETs) offer a promising
first step, but hard challenges persist in achieving confidentiality and
differential privacy in distributed learning. In this paper, we describe a
novel, regulation-compliant data protection technique for the distributed
training of ML models, applicable throughout the ML life cycle regardless of
the underlying ML architecture. Designed from the data owner’s perspective,
our method protects both training data and ML model parameters by employing a
protocol based on a quantized multi-hash data representation _Hash-Comb_
combined with randomization. The hyper-parameters of our scheme can be shared
using standard Secure Multi-Party computation protocols. Our experimental
results demonstrate the robustness and accuracy-preserving properties of our
approach.

###### keywords:

Random Quantization , Hashing , Confidentiality , Differential Privacy

[inst1]organization=EBTIC, Khalifa University,addressline=P.O. Box: 127788,
city=Abu Dhabi, country=UAE [inst2]organization=College of Engineering and IT,
University of Dubai, addressline=Academic City, Emirates Road, city=Dubai,
country=UAE [inst3]organization=C2PS, Khalifa University,addressline=P.O. Box:
127788, city=Abu Dhabi, country=UAE [inst4]organization=Computer Science
Department, Univ. degli Studi di Milano,addressline=via Celoria 18,
city=Milan, country=Italy [inst5]organization=Department of Computer Science,
Khalifa University,addressline=P.O. Box: 127788, city=Abu Dhabi, country=UAE

##  1 Introduction

The ongoing massive deployment of Machine Learning (ML) models involves
collection, storage and exchange of huge amounts of training data, which
raised serious concerns about data protection and attracted the attention of
regulatory bodies worldwide [1]. To name but one, the European General Data
Protection Regulation (GDPR) imposes obligations onto organizations anywhere,
so long as they target or collect data related to people in the EU. The
regulation mentions the use of _Privacy-enhancing technologies_ (PETs) to
protect personal data [2]. In the last few years, many attempts have been made
[3] to develop specific PETs to achieve confidentiality and privacy in
distributed training of Machine Learning (ML) models.

Nevertheless, there are still many privacy and confidentiality threats
affecting processing and communication of training data and ML models’
parameters [4]. While methodologies for modeling, detecting and alleviating
such threats are still being developed [5], organizations are required to
satisfy privacy regulations when carrying out distributed learning, especially
when personal data are involved [6].

In this paper, we describe a novel data protection technique designed from the
viewpoint of the data owners who hold training data sets, ML model parameters,
or both, and want to protect them independently of the specific ML
architectures and training algorithms. These data owners need a way to make
sure that the PET choice and configuration they use are both effective and
regulation-compliant. To tackle this challenge, we combine randomization of
quantization as proposed in [7] with _Hash-Comb_ , our own quantized multi-
hash data representation [8] to alleviate privacy and confidentiality threats
to distributed Machine Learning (ML) while guaranteeing regulatory compliance.
The major contribution of this paper is threefold:

  * 1.

An entirely novel, efficient and easy-to-implement method to achieve the
desired level of differential privacy of ML models’ parameters and data
exchanged in distributed training by adding randomized quantization noise.

  * 2.

A distributed protocol based on secret sharing, integrating our method with
existing frameworks for federated learning.

  * 3.

Integrated support for parameter and data confidentiality, both in-transit and
at-rest, using only standard hashing function that comply with regulations.

Our experimentation on multiple datasets shows that our method provides faster
training convergence and better accuracy of the final ML models compared to
benchmark techniques.

###  1.1 Background and Problem Statement

To introduce the main privacy and confidentiality threats to ML models’
training, let us consider a typical ML task: classifying the items of a set
$DS$ into classes of interest belonging to a finite set $C=(C_{1},...,C_{n})$
[9].

As we do not have access to the entire domain $DS$, we use a _sample_
$S\subset DS$ of the domain and tabulate a partial classification function
$f():S\rightarrow C$, obtaining a subset of $DS\times C$, which, with a slight
abuse of notation, we call $f$. Each entry $e$ of $f$ has the form
$\\{s,C_{i}\\}$ where $s\in S$ and $C_{i}\in C$. We can then use $f$ as a
_training set_ to train a supervised ML model that computes an estimator
function $F:DS\rightarrow C$, hopefully coinciding with $f$ at (most of) the
points belonging to $S$.

The model $F$ can be either deployed into production to classify elements of
$DS$ as they become available (the so-called _inference_ step); alternatively,
one can repeat the process, collecting $k$ samples $S_{1},...,S_{k}$ in $DS$
and training the corresponding ML models $F_{1},...F_{k}$. These estimators
can then be merged to obtain a _federated_ global estimator $F_{G}$.

This procedure is subject to two potential threats: a _threat to privacy_
regarding the entries of the training set $f$, if $f$ can be reverse-
engineered from $F$ (or $F_{G}$), and a _threat to confidentiality_ regarding
the parameters of the ML models $F_{i},i=1,..,k$ when they are communicated to
the central server in charge of merging them into $F_{G}$. The privacy
violation will happen if, by observing the outcome of $F$, an attacker can
reconstruct one or more entries of $f$. To alleviate the privacy threat, we
can pursue a _non-disclosure_ goal: given any entry $e\in f$, any observer of
the execution of $F$ should be able to infer the same information about $f$ as
by observing any other model $F^{\prime}$ obtained using the training set
$f-{e}+{r}$, where $r$ is a random entry.

Two decades ago, Cynthia Dwork proposed the notion of _differential privacy_
(DP) [10] to provide a quantitative measure of the degree of satisfaction of
the above goal after any data transformation. Other seminal papers [11], [12],
[13] have shown that it is possible to add noise to data sets and alleviate
the threat to privacy while preserving some measure of accuracy of any ML
model trained on them.

###  1.2 Differential Privacy

Formally, a ML model $F$ trained on $f$ guarantees a level $\epsilon$ of
_differential privacy_ (DP) if, for all possible training sets $f^{\prime}$
differing in a single value from $f$, for all outputs $C_{i}\in C$ and for all
$x\in DS$ we have:

$(1-\epsilon)\leq{{Pr(F(x)\in C_{i})}\over{Pr(F^{\prime}(x)\in
C_{i})}}\leq(1+\epsilon)$ (1)

where $F^{\prime}$ is (the same) model trained over $f^{\prime}$.

A popular technique for achieving the desired level of differential privacy is
_non-interactive_ DP [14], which consists in adding random noise to an
existing training $f$ to obtain an _adjacent_ training set $f^{\prime}$. A
convenient probability density function (PDF) for such noise is the Laplace
distribution [15]:

$p(z)=e^{-{|z|\over{\sigma}}}=e^{-|z|\epsilon}$ (2)

This distribution is “concentrated around the truth”: after generating an
adjacent data set by noise addition, the probability that an entry $e$ in it
will fall $z$ units from the true value drops off exponentially with $z$, with
a progression given by the $\epsilon$ parameter. In other words, a lower
$\epsilon$ increases the probability of values far from the truth, achieving
stronger privacy. Besides Laplace, other distributions have been used for DP,
including the Gaussian [16], the geometric ] [12], and the exponential one
[17]. Due to the noise, DP introduces some uncertainty: an ML model trained of
a training set $f^{\prime}$ adjacent to $f$ will no longer compute $F$ but
another function $F^{\prime}$, potentially less accurate in production. The
need for balancing data privacy and accuracy brought to a more general
definition of _interactive_ differential privacy (NIDP) [18], also known as
($\epsilon,\delta$)-DP. Again, ($\epsilon,\delta$)-DP is based on the concept
of adjacent data sets, i.e. data sets that differ in just one entry. It can be
formally defined as follows [19]:

######  Definition 1.1.

Let $S$ and $S^{\prime}$ be two adjacent data sets. Then, a randomized
function $F:\chi\rightarrow C$ satisfies ($\epsilon,\delta$)-DP if for any
$S,S^{\prime}\in\chi$ and any subset of outputs ($O\subseteq C$):

$Pr(F(S)\in O)\leq e^{\epsilon}Pr(F(S^{\prime})\in O)+\delta$ (3)

where:

  * 1.

$Pr(F(S)\in O)$ is the probability of model $F$ to give an output in $O$ for
an input in $S$.

  * 2.

$Pr(F(S^{\prime})\in O)$: probability of model $F$ to give an output in $O$
for input in $S^{\prime}$.

  * 3.

$\epsilon$ is a non-negative parameter, also known as _privacy_ budget, that
quantifies the level of privacy protection.

  * 4.

$\delta$ is a non-negative parameter that represents the _failure probability_
of not holding the privacy guarantee. The probability of privacy breach is
bounded by $\delta$.

Values of $\delta$ and $\epsilon$ can be adjusted according to requirements
and even modified interactively [18]. In 2017, Ilya Mironov introduced the
notion of _Rényi differential privacy_ (RDP) [20]. RDP requires the
_divergence_ between the distributions of the original and the noisy samples
$S$ and $S^{\prime}$ to be properly bounded. We provide a formal definition of
Rényi divergence in Section 3.1. A key property of RDP is that a mechanism
which provides RDP also provides classic DP.

The remainder of this article is organized as follows: In Section 2, we review
related work on quantization-based privacy in distributed learning. In Section
3, we present our own approach. In Section 4, we show how standard hashing can
be used to turn the quantized representation into our multi-hash encoding,
_Hash-Comb_ [8]. In Section 5, we outline how the hyper-parameters of our
representation scheme can be securely shared using _Multi-Party Computation_
(MPC). In Section 6, we discuss centralized and federated training of ML
models on randomized _Hash-Comb_ data. In the context of Federated Learning,
we show that our method preserves the confidentiality of local models’
parameters while allowing for the computation of an accurate global model
$F_{G}$. In Section 7, we carry out an experimental comparison between our
approach and other techniques for achieving DP by noise addition. Finally, in
Section 8, we draw our conclusions and highlight some future challenges and
application scenarios.

##  2 Related Work

Much research has targeted communication efficiency and privacy preservation
in distributed learning [21, 22, 23, 24, 7]. Already early research recognized
that privacy can be improved by quantization of model updates, particularly
gradients [23, 21]. Pioneering work [23] presented a differentially private
distributed stochastic gradient descent (DP-SGD) algorithm where quantization
is randomized via a binomial mechanism, serving as a discrete counterpart to
the Laplacian or Gaussian mechanisms outlined in Section 1.2. Zong et al. [21]
proposed _Universal Vector Quantization for Federated Learning (FL_) with
Local Differential Privacy (LDP). In this approach, FL clients achieve LDP by
introducing Gaussian noise to their model updates. Then, the noisy updates are
quantized and encoded into digital code-words to be transmitted to the server
for aggregation.

This ”quantization-then-noise” family of approaches showed some limitations in
accuracy and efficiency, leading to biased estimation due to modular clipping.
The alternative method is swapping the order of the operation, adding noise to
quantized parameters. For instance, Yan et al. [22] add binomial noise to
uniformly quantized gradients. Their approach attains the desired level of
differential privacy with only a slight reduction in communication efficiency.

Recent research is based on the notion that quantization alone may achieve the
desired privacy level. Several research efforts, such as [7, 24], successfully
leveraged quantization to simultaneously optimize communication efficiency and
preserve privacy in distributed learning. These works demonstrate that the
perturbation introduced by the compression process can be translated into
quantifiable noise. The work closest to ours [7], achieves Rényi differential
privacy (RDP) through a two-tiered randomization process. Initially, it
randomly selects a subset of feasible quantization levels, followed by a
randomized rounding procedure utilizing these discrete levels. A more recent
work [24] introduces Gau-LRQ (Gaussian Layered Randomized Quantization),
leveraging layered multi-shift couplers to compress gradients while shaping
the error towards a Gaussian distribution, thereby ensuring client-level
differential privacy (CLDP). Gau-LRQ is implemented within the distributed
stochastic gradient descent (SGD) framework and is evaluated on various
datasets, including MNIST, CIFAR-10, and CIFAR-100. The results demonstrate
better performance compared to the approach proposed in [22] and to a baseline
that uses Gaussian DP only, without quantization.

All methods reviewed above enhance privacy and communication efficiency in the
context of gradient-based large-scale distributed learning. We follow a
different approach, pursuing a random quantization technique applicable to any
value, be it training data or model updates. Our approach can achieve
differential privacy in a general setting without imposing constraints on the
model parameters or integrating other privacy techniques. Our approach also
achieves efficient transmission due to the fixed message size exchanged
between the server and client. A summary of our technique’s features and a
comparison to the ones of other approaches are presented in Table 1.

Table 1: Quantization-based Approaches Comparison

Ref | Random Quantization | Extra Privacy technique | DP-Noise Addition | differential privacy guarantee | Application
---|---|---|---|---|---
[23] | $\times$ | $\checkmark$ | Gaussian Noise | $\checkmark$ | gradient-based large-scale distributed learning
[21] | $\times$ | $\checkmark$ | Gaussian Noise | $\checkmark$ | gradient-based FL model
[22] | $\times$ | $\checkmark$ | Binomial noise | $\checkmark$ | gradient-based large-scale distributed learning
[7] | $\checkmark$ | $\times$ | not needed | $\checkmark$ | General
[24] | $\checkmark$ | $\times$ | not needed | $\checkmark$ | gradient-based large-scale distributed learning
Our Work | $\checkmark$ | $\times$ | not needed | $\checkmark$ | General

##  3 Achieving Differential Privacy via Quantization

In this section, for the sake of simplicity, we will assume that each data
point in $DS$ is a scalar $x$; our discussion is readily extendable to any
vector data space ${\bf x}=x_{1},...,x_{n}$. The quantization of samples in
each sampling domain $S$ involves three steps:

  * 1.

Negotiating the overall quantization range and the number $L$ of possible
quantizations.

  * 2.

Performing a _randomized selection_ of a number $l<L$ of the quantizations to
apply to the sampled $x$ values in $S$.

  * 3.

Computing the quantized image of the data points in the sample $S$. Each point
in $S$ corresponds to $l$ quantized values.

We describe each step in detail below.

  1. 1.

Range Enlargement This step establishes the range of values in the quantized
image of $S$. We subtract a parameter $\Delta$ from $x_{min}$, the smallest
$x\in S$ and adding $\Delta$ to $x_{max}$, the largest $x\in S$. The
enlargement relative size is expressed by the ratio $x_{max}\over{\Delta}$.
The enlargement of the value range is necessary to achieve privacy: if we used
for the quantized image of $S$ the same range of $S$, the quantization outputs
for $x_{max}$ and $x_{min}$ would always end up in the top and bottom channels
of each quantization, leaking information about $x$. Within this augmented
range we establish $L$ quantization levels, respectively involving
$m_{1},..,m_{i},..,m_{L}$ evenly spaced quantization channels
$(C(0),C(1),\textperiodcentered\textperiodcentered\textperiodcentered,C(m_{i}-1))$.

  2. 2.

Sub-sampling of quantizations This step randomly sub-samples $l$ of the
possible $L$ quantization levels, including each level of quantization in our
encoding with a _selection probability_ $p>0$. The value $p=0.5$ corresponds
to standard fair coin randomization. As we will see in Section 4, different
values of $p$ can be used to achieve the desired randomization (i.e., the
target value of $l$) with a limited number of extractions, neutralizing
selection bias [25].

  3. 3.

Quantized image computation In this step, we compute the multi-value quantized
image of $S$. Each point $x\in S$ generates a list of values, corresponding to
the channel where $x$ falls in each of the $l$ sub-sampled quantizations. At
each level, an approximation of the value of $x$ can be extracted by
estimating $x$ as the mid-point of the quantization channel it fell in.
Section4 includes a worked-out example of this computation.

###  3.1 Rényi Differential Privacy

We now discuss how our scheme achieves Rényi differential privacy. We start
with defining _divergence_. Informally, divergence is a binary function which
establishes the separation from one probability distribution to another. The
simplest divergence is squared Euclidean distance. Other classic divergences
like _Kullback–Leibler_ (KL) divergence[26] can be seen as generalizations of
it.
Here, we are interested in the _Rényi divergence_ of positive order
$\alpha\neq 1$ between a discrete probability distribution
$P=(p_{1},...,p_{n})$ and another distribution $Q=(q_{1},...,q_{n})$. It can
be written as follows [27]:

$D_{\alpha}(P||Q)={{1}\over{\alpha-1}}ln\sum_{i=1}^{n}p_{i}^{\alpha}q_{i}^{1-\alpha}$
(4)

where, for $\alpha>1$, we read $q^{1-\alpha}$ as $q^{{\alpha}-{1}}$ and adopt
the conventions that $0/0=1$ and $x/0=\infty$ for $x>0$.

We recall the relationship between the Rényi divergence with $\alpha=\infty$
and $\epsilon$-differential privacy (Section 1.2, Eq.1). Any randomized data
representation mechanism is _differentially private_ if and only if its
distribution over any two adjacent data sets $S$ and $S^{\prime}$ satisfies
the following equation:

$D_{\infty}(f(S)||f(S^{\prime}))\leq\epsilon$ (5)

To state this property for the distributions of any pair of adjacent samples
in the multi-quantization procedure of Section 4, we consider a point
$x_{j}\in S$ and a replacement value $x^{\prime}_{j}$ (so that $S$ and
$S^{\prime}=S-\\{x_{j}\\}+\\{x^{\prime}_{j}\\}$ are adjacent). Let us call $P$
and $Q$ the probability distributions of the (multi-level) quantized
representations of $S$ and $S^{\prime}$ across the same random selection of
the quantization levels. We consider the finest grained quantization in our
representation, and map each data point $x_{i}$ ($i=1,...,n$) to a quantized
value $t_{h}$ for $h=1,..c$ where $c$ is the number of quantization bins. The
$c$-dimensional distribution of values $t_{h}$ is itself a quantization of the
$n$-dimensional discrete probability distribution of the data values, with a
given number $c$ of supporting points [28], coinciding with the bins’
midpoints. Let $p_{h}$ ($q_{h}$) be the number of occurrences of each value
$t_{h}$ divided by the size $\lvert S\rvert$ of $S$ (or $S^{\prime}$). In
other words $p_{h}={{\\#(t_{h})}\over{|S|}}$. As $S$ and $S^{\prime}$ differ
only in $x_{j}$ being replaced by $x^{\prime}_{j}$, $P$ and $Q$ contain the
same quantization, so $p_{h}=q_{h}$ for $h<>r,s$ where the $r$ and $s$ are the
(possibly coinciding) quantization bins where $x_{j}$ and $x^{\prime}_{j}$
have fallen. We get:

$D_{\alpha}(P||Q)={{1}\over{\alpha-1}}ln(\sum_{h=1}^{k}p_{h}^{\alpha}q_{i}^{\alpha-1})=$
(6)
${{1}\over{\alpha-1}}ln(\sum_{h<>r,s}^{n}p_{h}^{2\alpha-1}+p_{r}^{\alpha}q_{s}^{\alpha-1})$
(7)

where $p_{r}$ and $q_{s}$ are the frequencies (i.e., the normalized
occurrences) of the quantization bins where $x_{j}$ and $x^{\prime}_{j}$ have
fallen. We can now cap the value of our Equation 7 as follows:

${{1}\over{\alpha-1}}ln(n\ p_{max}^{2\alpha-1})=$ (8)
${{1}\over{\alpha-1}}(ln(n)+(2\alpha-1)ln(p_{max}))=$ (9)
${{1}\over{\alpha-1}}ln(n)+{{2\alpha-1}\over{\alpha-1}}ln\ (p_{max})$ (10)

where $p_{max}$ is the largest value among all $p_{h}$. For
$\alpha\rightarrow\infty$ we get:

$D_{\infty}(P||Q)\leq\epsilon=2\ ln(p_{max})$ (11)

Equation 11 shows that, for any Hash-Comb, the Rényi divergence between the
probability distributions of two adjacent samples’ quantizations is bounded by
a constant, which can be tuned by adjusting the channel width of the finest-
grained quantization. Therefore, our data representation is Rényi-
differentially private and also differentially private in the classic sense
[7]. Our Hash-Comb representation provides a small set of hyper-parameters
whose setting can quantify the leakage of private information within the
framework of Rényi Differential Privacy[29]. Such hyper-parameters can be
chosen at each training run. Literature studies [30] show that the reference
RDP range is from $4$ to $7$. Using a biased coin with tunable selection
probability $q$ and ${x_{max}\over{\Delta}}=2$, we can achieve any value in
this reference range of RDP by using a relatively small number ($l=4$ to
$l=8$) of quantization levels.

##  4 Multi-Hash Representation

In this Section we compute our multi-quantization encoding for the parameters
of a ML model [8]. Our method consists of hashing the quantized values,
turning each multi-quantized value into a multi-hash representation. We use
regulation-compliant, small-footprint hashes suitable for efficient
communication.

Encoding is performed at each round of the FL protocol as detailed in
Algorithm 1. At each round, the local model parameters are encoded before
being submitted to the central unit for averaging. The Hash-Comb algorithm
uses _k_ quantizers whose quantization level $1\leq l\leq k$ is defined by
$2^{l}$ channels. Once established the channel where it falls at each level,
the encoding value w is rounded to the value representing the midpoint of the
channel itself, providing an approximation up to $S(1/2)^{l+1}$, where S is
the size of our initial data space.

Finally, the encoding for w is obtained with the concatenation of $k$ hashes,
each representing a specific branch of the binary graph in which the root is
$S$ and each node in the path is the hashed value of the channel containing w
at every quantization level. The sequence of (hashed) channels captures
different granularity of distance information from $S$. The last channel
represents the finest-grained approximation for w in our encoding, and
represents the output of the decoding process.

At this point, we need to identify the most suitable value for _k_. Proceeding
empirically, we found that encoding using $8$ quantization levels does not
affect the outcome of model training, and in some cases even results in
enhanced performance (Section 7). This ”magic number” of eight quantizers is
also consistent with literature results (Section 3.1) about the quantizers
required to achieve RDP. Another requisite to address is the one regarding
sub-sampling (Section 3), which dictates a theoretical maximum $L$ of
quantization levels. Once more, we proceeded empirically by assuming a
conservative scenario with an enlarged range $[-1,+1]$ and $L=16$. In this
case any decoded w would have an approximation error up to $\pm
1.52587890625e-05$, which can be considered negligible for the training of the
model. Adding more levels to the algorithm would only exponentially increase
the computational workload without additional benefits.

To sub-sample, we assign different levels of quantization to each parameter w
while maintaining an average $\overline{k}$ equals to $8$, as discussed above.
The randomization factor in selecting the level is modelled as tossing a coin
for $16$ times. The last toss in which Head occurs indicates the quantization
level for the current w, as shown in Figure 1. A _fair coin_ where each
outcome has probability $p=0.5$ is unsuitable to our scheme. It results in
$\overline{k}=15$, as easily demonstrable through the formula (12), which does
not guarantee the desired privacy. To balance the average quantization across
all weights and provide the necessary noise we use a biased coin with
$Pr(H)=p$ and $Pr(T)=(1-p)$ guaranteeing $\overline{k}=8$. The solution
$p=0.087826$ was found by solving the linear system (13) shown in Figure 2.

Algorithm 1 The algorithm computing Hash-Combs on each node at each
communication round

$N\geq 1,R\geq 1$

$k_{i}\in\mathbb{Z},\quad 1\leq k_{i}\leq L$

procedure HashCombAlg($N,R$) $\triangleright$ number of nodes and rounds

$r\leftarrow 0$

while $r\not=R$ do

$H\leftarrow\\{\\}$ $\triangleright$ initialize the communications

for all $i\in N$ do $\triangleright$ this is performed locally on each node
$i$

<compute $W^{i}$> $\triangleright$ $W$ result of $n$-cycles of optimization
algorithm

for all $w^{i}_{j}\in W^{i}$ do

$k_{j}\leftarrow$ <randomly select $k$>

$h_{j}=HC_{k_{j}}(w^{i}_{j})$ $\triangleright$ HC with a quantization layer
$k$

$H^{i}\leftarrow$ add($h_{j}$)

end for

$H\leftarrow$ add($H^{i}$) $\triangleright$ add the locally encoded weights to
the queue

end for

return $H$$\triangleright$ all encoded parameters sent for FedAVG

$r=r+1$

end while

end procedure

Figure 1: Hash-Comb levels as numbers of consecutive tosses of a biased coin.

${\overline{k}}=\sum_{i=0}^{L}(L-i)\cdot Pr(k_{L-i})$ (12)
${\sum_{i=0}^{L}(L-i)\cdot p(1-p)^{i}=8,\quad 0\leq p\leq 1}$ (13)

Figure 2: Solution for p when $\overline{k}=8$.

##  5 The Negotiation Protocol

In our scheme all participants must securely share the initial parameters of
the quantization procedure, namely the $l<L$ selected quantization levels to
be used and the enlarged ranges $[c_{min},c_{max}]$, with
$c_{min}=(x_{min}-\Delta$) and $c_{max}=(x_{max}+\Delta$). Many different
_Multi-Party Computation_ (MPC) protocols can be used for achieving common
knowledge of these parameters without sharing the local feature values or
ranges [31]. Below, we describe a secure sharing protocol under the reasonable
assumption of a majority of honest participants.

###  5.1 Secret Sharing

Our protocol utilises the classic technique of _Shamir secret sharing_ [32] to
share a secret $s$ amongst $n$ parties, so that any subset of $t+1$ or more of
the parties can reconstruct the secret, yet no subset of $t$ or fewer parties
can learn anything about the secret. Shamir’s secret sharing scheme relies on
the fact that for any for $t+1$ points on the two dimensional plane
$(x_{1},y_{1}),...,(x_{t+1},y_{t+1})$ with unique $x_{i}$, there exists a
unique polynomial $q(x)$ of degree at most $t$ such that $q(x_{i})=y_{i}$ for
every $i$, and that it is possible to efficiently reconstruct the polynomial
$q(x)$, or any specific point on it111All computations in this section are in
a finite field $Z_{p}$, for any prime number $p>n$. In order to share a secret
$s$, for example its local $x_{min}$, the participant chooses a random
polynomial $q(x)$ of degree at most $t$ under the constraint that $q(0)=s$.

In practice, each participant sets $a_{0}=s$, chooses random coefficients
$a_{1},...,a_{t}\in Z_{p}$,and sets $q(x)=\sum_{i=0}a_{i}\textperiodcentered
x^{i}$. Then, for every $i=1,...,n$ the participant provides the $i$-th party
with the share $y_{i}=q(i)$. Note that $p>n$, so different shares can be given
to each party. Reconstruction by a subset of any $t$ parties works by
interpolating the polynomial to derive $q(x)$ and then compute $s=q(0)$.

Although $t+1$ parties can completely recover $s$, any subset of $t$ or fewer
parties cannot learn anything about $s$, as they do not have enough points on
the polynomial. Since the polynomial is random, all polynomials are equally
likely, and all values of $s\in Z_{p}$ are equally likely.

###  5.2 The MPC protocol

We are now ready to outline the steps of our protocol.

  1. 1.

Coordinator election. The first conceptual step in our protocol is selecting
the coordinating node who will be in charge of deciding the enlargement
parameter $\Delta$. In practice, this step can be merged with the next one, as
the node in charge of announcing $\Delta$ can simply be the one who shared the
largest local $x_{max}$.

  2. 2.

Local range sharing: In this step, each party $i$ shares its local
$x_{min_{i}}$ and $x_{max_{i}}$ with the other parties, using Shamir’s secret
sharing.

  3. 3.

Quantization set-up The coordinator decides the number of quantization levels
$L$ and carries out the random extraction of the $l<L$ quantization levels to
be used in the training.

  4. 4.

Hyper-parameter sharing The coordinator announces the $x_{min}$, $x_{max}$,
$\Delta$ parameters and the $l$ extracted quantization levels to the other
parties, using Shamir’s secret sharing.

The protocol must be executed before distributed training can begin. It is
secure for semi-honest adversaries, as long as less than $n/2$ parties are
corrupted[33]. This is because the only values seen by the parties during the
computation are secret shares (that reveal nothing about the values they
hide). To achieve security in the presence of malicious adversaries who will
deviate from the protocol’s specification, it is necessary to rely on specific
methods to prevent cheating [34].

###  5.3 Brute Force attack analysis

Secure sharing of the algorithm parameters is necessary but not sufficient to
guarantee privacy through our Hash-Comb encoding. A malicious observer could
still crack our algorithm using a _brute-force_ approach to guess the range
$[x_{min},x_{max}]$, hash the values and match the results with the target to
validate the guess. It should be noted that the following considerations are
intended for normalized values of the training datasets as well as of the
learning model parameters.

Because the time complexity of brute force is $O(n\cdot m)$, where $n$
characters in a string of length $m$ requires $n*m$ tries, the smallest is the
range of values and the more susceptible it is to brute-force attack. The
authors in [35] explain that a 128-bit security is well beyond the exhaustive
search power of current technology. We can consider a secret on 128-bit
computationally secure as to simply iterate through all the possible values
requires $2^{128}$ flips on a conventional processor
222https://en.wikipedia.org/wiki/Brute-force_attack, the Von Neumann-Landauer
Limit can be applied to estimate the energy required as $\approx 10^{18}$
joules, which is equivalent to consuming 30 gigawatts of power for one year.
This is equal to $30\cdot 10^{9}W\cdot 365\cdot 24\cdot 3600s=9.46\cdot
10^{17}J$ or $262.7$ $TWh$ which is approximately 0.1% of the yearly world
energy production.

Being $h_{i}=f_{hash}([x_{min_{i}};x_{max_{i}}])$ the hashed value of the
channel $i$ having range $[x_{min_{i}},x_{max_{i}}]$ and assuming the use of
double-precision floating-point representation (according to the IEEE 754
standard), where a bit is dedicated for the sign, 11 bit for the exponent and
52 bits for the mantissa, the combination of the limits perfectly fits as a
shared secret of 128 bits required for the encoding. Unfortunately, due to the
nature of the scalars in our scenarios, which are either ML models’ parameters
(weights) or normalized features values, the potential attacker could make
strong assumptions in guessing those ranges.

For example, the attacker can assume model parameters having
$x_{min}\in[-1,-0.5]$ and $x_{max}\in[0.5,1]$, therefore the merging can be
represented using $2^{52}*2^{52}=2^{104}$ values. More conservative ranges
such as $[-1,-0.25]$ and $[0.25,1]$ only increase the possible representations
to $2^{106}$, both assumptions significantly reduce the number of tries for
the attacker. In the case of normalized values, the task for the attacker is
further simplified by the fact that $x_{min}$ and $x_{max}$ are fixed and
corresponding to the normalization range itself, and the only effort would be
the guess of the enlargement value $\Delta$.

In conclusion, we use an additional number of bits to encode $y$ concatenated
to the other shared secret to achieve the target of 128-bits to calculate
$h_{i}$ as $f_{hash}([x_{min_{i}};y;x_{max_{i}}])$ and to achieve a schema
that is brute-force proof.

##  6 Training Models with Hash-Combs

We start our experimental validation by training a monolithic ML model using
multi-quantized training data.

###  6.1 Datasets

For our experiments, we used three different datasets. The first dataset is
part of the UCI Machine Learning Repository [36]. Data consists of information
from 4601 email messages, originally used in a study to identify junk email,
or “spam”. For all 4601 email messages, the ground truth (”email” or ”spam”)
is available, along with the relative frequencies of 57 of the most commonly
occurring words and punctuation marks in the email message. Figure 3 shows
some of those features. The second dataset, the recent IoT-23 dataset [37],
has been created simulating IoT network traffic generated by popular IoT
devices, including a Philips HUE smart LED lamp, an Amazon Echo home
intelligent personal assistant and a Somfy smart door lock. This dataset
includes 23 traffic scenarios, 20 of which are malicious traffic from infected
device and 3 are normal traffic. Such flows are labeled ”benign” or ”malign”,
with an additional label specifying the attack type. For our experiment, we
sampled the data from benign traffic and from malign traffic belonging to two
types of attack, PartOfAHorizontalPortScan and Okiru. Finally, the
Cardiovascular Disease
333https://www.kaggle.com/datasets/sulianova/cardiovascular-disease-
dataset/data dataset consisting of 70000 records of patients data, 11 features
divided as factual information, results of medical examination and information
given by the patient. The target class represents the risk of a cardiovascular
problem.

###  6.2 ML Model

The ML model used for this experiments is a simple Multi-Layer Perceptron
(MLP), implemented in Java 17. Our implementation can be configured to work as
monolithic model as well as in federated learning with multiple instances of
MLP communicating with a central server for sending/receiving updated weights
and implementing FedAvg. The model optimization technique is Stochastic
Gradient Descent (SGD) and _ReLu_ is used as activation function. The
architecture of our MLP counts 5 hidden layers, configured with
$50,25,20,25,50$ neurons respectively. The learning rate is $\eta=0.05$, while
the number of epochs for the training depends on the requirements of the model
to learn the specific dataset. The hardware is an Intel Core i5-9300H CPU
2.40GHz, 2400 Mhz, 4 Cores/8 Logical Cores. The same configuration for the
model and hardware specification were maintained throughout all experiments.

####  6.2.1 Monolithic Training

The results in Table 2 conveys the performance of the monolithic training over
the four datasets after normalization. The accuracy of the results was
verified by comparison with the MLPClassifier444https://scikit-
learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html
availbable in Sklearn, a Python library commonly used in similar research
projects which has provided perfectly matching performance, therefore proving
the accurateness of our implementation. The purpose of this test is to set a
benchmark for the experiments in the following section. In fact, the
comparison of the monolithic training with the results of the training paired
with privacy preserving methods exemplify the performance improvement or decay
of the latter when applied to FL.

Figure 3: Some of the most relevant features in SPAM dataset Table 2: Training
and Testing on a monolithic model

Training Testing Dataset Target label Support Epochs Accuracy F1-score Spam
Spam 1813/3732 25K 0.8801 0.8804 IoT23-Okiru Malicious 9890/20286 10K 0.9911
0.9911 IoT23-HPS Malicious 7478/18624 40K 0.8280 0.8224 Coronary Cardio
10465/21004 12K 0.6231 0.6231

##  7 Experiments

We experimented our Hash-Comb representation applied to ML weights, encoded
and communicated by local models to the central unit in a FL protocol. The
experiments in Section 7.1 aim at comparing different quantization levels of
Hash-Comb in order to find the optimal number of quantizers and thus validate
the theory as explained in Section 4, while in Section 7.2 the focus is on
comparing our approach to the most recent competing methods so as to
demonstrate effectiveness through performance juxtaposing.

Regarding the metric, we adopted the $F1$ score which computes an average of
precision and recall. The $F1$ score is proven to be a valid performance
metric when trying to solve binary classification problems as in our
experiments.

Table 3: Testing results for Experiment 1

F1-Score Epochs $NoHC$ $HC_{4}$ $HC_{6}$ $HC_{8}$ $HC_{10}$ Spam 6K*4 0.8856
0.7019 0.9084 0.9002 0.8969 IoT23-Okiru 2.5K*4 0.9938 0.9870 0.9935 0.9941
0.9941 IoT23-HPS 12K*4 0.8226 0.6255 0.7577 0.8256 0.8224 Coronary 3K*4 0.6217
0.5960 0.6157 0.6229 0.6223

Figure 4: Training and Testing (last step) from Experiment 1

###  7.1 Federated Learning with Hash-Combed parameters

We replicated rhe monolithic training in Table 2 in a federated environment by
deploying 4 instances of MLP communicating with the central unit for
averaging. At each round, each local model is trained with exactly 25% of
randomly selected samples from the original set, the weights are sent to the
central unit once a training step is completed. The purpose of the following 2
experiments is to verify the impact of Hash-Comb on the overall training and
also to capture its performance by varying the number of quantizers.

When the weights are sent in clear ($NoHC$), the standard FedAvg Algorithm
(14) is applied on the central unit. While using Hash-Comb ($HC_{L}$), the
quantization described in Section 4 is applied based on the initial range
$S=[x_{min},x_{max}]$ and a sequence of $L$ hashes is generated for each $w$
at every round. The encoding that we use for the experiments is defined as
$h_{L}(w)=f_{hash}(ch_{i})$ where $i\in[1,2^{L}]$, it represents the hashing
function applied to $ch_{i}$, the $i^{th}$ channel at level $L$ containing
$w$. This encoding is used in (15), implemented in the central unit to compute
the encoded average of the received hashed values.

####  7.1.1 Experiment 1: Bounded communication rounds

The first experiment’s aftermath is outlined in Table 3. It appears
immediately evident a slight improvement resulting from the mere exploitation
of the FL protocol ($NoHC$) when compared with the monolithic training, an
outcome that has been widely reported in the literature.

Most importantly, the use of Hash-Comb distinctly reveals a significant
improvement with $l\geq 6$ quantization levels, in particular when $l=8$ the
model enhances its performances across all datasets. Furthermore, the diagrams
in Figure 4 suggest a major over-fitting due to the excessive rounding of the
weights when $l$ is lower than 8. In conclusion, these findings confirms $l=8$
as a magic number for quantization as previously theorized in Section 4.

${w_{t+1}\leftarrow\sum_{n=1}^{N}{w^{n}_{t}\over N}}$ (14) $\quad
w_{t+1}\leftarrow\sum_{n=0}^{N}{c_{i}h_{i}\over
N}\begin{cases}c_{i}=0&\text{if n=0}\\\ c_{i}=c_{i}+1&\text{if $n>0$ and
$h_{L}(w^{n}_{t})=h_{i}$}\\\ \end{cases}$ (15)

####  7.1.2 Experiment 2: Incremental number of communication rounds

The second set of experiments (see Table 4) has the objective of evaluating
the use of Hash-Comb by increasing the frequency of communication rounds and,
consequently, the number of FedAvg iterations over the same total number of
epochs as before.

Each score within the diagrams in Figure 5 is obtained by validating the
general model at each round of communication, confirming the observations
previously pointed out: the model using Hash-Comb with $L\geq 8$ achieves
better results in the performance and sometimes is a faster learner when
compared with non-quantized weights. Some minor instability is observed within
the Coronary dataset, probably due to insufficient data quality. In this case,
averaging and rounding of the weights may cause fluctuating performance,
noticeably for low $l$ values, though it eventually converges at the end of
the training. This last observation suggests a possible correlation between
the value $l$ and some measure of training data quality [38].

Table 4: Testing results for Experiment 2

F1-Score Epochs $NoHC$ $HC_{6}$ $HC_{8}$ $HC_{10}$ Spam 1K*40 0.9082 (40)
0.9148 (36) 0.9228 (40) 0.9212 (36) IoT23-Okiru 1K*14 0.9941 (10) 0.9935 (10)
0.9944 (10) 0.9941 (8) IoT23-HPS 1K*60 0.8224 (40) 0.8164 (55) 0.8221 (50)
0.8299 (35) Coronary 1K*26 0.6235 (20) 0.6104 (26) 0.6282 (26) 0.6279 (26)

Figure 5: Model validation score at each FedAvg iteration from Experiment 2

###  7.2 Experiment 3: Comparison with classic DP

In this Section our method is compared to a competing approach [39]
functionally close to ours which is an ideal benchmark for our proposal. In
fact, while the majority of related works focus on FedSGD the method in the
above mentioned work adds standard differential privacy (DP) to the FedAvg
algorithm. The authors claim that to achieve the desired
$(\epsilon,\delta)$-DP, the required noise level is:

${\sigma_{t,k}^{2}={2(\triangle
f_{t,k})^{2}ln(1.25q/\delta)\over\epsilon^{2}}}$ (16) ${\triangle f\triangleq
max_{D,D^{\prime}}||f(D)-f(D^{\prime})||}$ (17)

The $\triangle f$ is so-called global sensitivity of function $f$ and being
$D$ and $D^{\prime}$ a neighboring datasets on the node k, then:

${\triangle f_{t+1,k}=max||w^{k}_{t+1}(D_{k})-w^{k}_{t+1}(D_{k}^{\prime})||}$
(18)

The equation in (18) represents the global sensitivity which is calculated as
$\triangle f_{t+1,k}=4QG\eta_{t+1}$, where $Q$ is the number of SGD updates
within each communication round, $q$ is the ratio between Q and the rows in
the dataset and $G$ is the clipped local gradient on node $k$, whereas the
learning rate is $\eta$. Based on the experiments mentioned in [39] and also
on some preliminary tests we conducted on a training set with added Gaussian
noise, the desirable variance is $\sigma^{2}\approx 0.08$. In fact, a lower
value does not guarantee the required privacy while a higher volume of noise
significantly reduces the usability of the weights, resulting in a compromised
learning outcome.

In our experiment we aim at providing high protection level of
$(2,10^{-3})$-DP while retaining an acceptable usability of the weights. After
plugging the above values into Formula (16), setting $G=2$ and a constant
$\eta=0.05$, the desired $\sigma^{2}$ is obtained with q = ${SGD\over N_{k}}$
$\approx 0.008$, establishing a suitable number of gradient updates per
communication round given the dataset size $N$.

The derived $(\epsilon,\delta)$-DP is applied to the training of our datasets
for a direct comparison and the results are overlapped to ours and showed in
Figure 6. From the analysis of the graphs it is immediately clear that despite
a lower volume of noise, compensated by the high frequency of communications,
the achieving of a desired DP level leads to a significant degradation of the
model, both in terms of performance as well as learning convergence.

Finally, the results in Table 5 show the performance gaining of our Hash-Comb
($HC_{8}$), featuring an average improvement of $0.6\%$ in the F1-Score when
compared to standard training and of $1.92\%$ when compared to
$(\epsilon,\delta)$-DP.

Table 5: Results of comparison between Hash-Comb and DP |  | F1-Score
---|---|---
| Epochs | $NoHC$ | ($\epsilon,\delta$)-DP | $HC_{8}$
Spam | 40K | 0.9082 (40) | 0.8777 (36) | 0.9228 (40)
IoT23-Okiru | 14K | 0.9941 (10) | 0.9721 (8) | 0.9944 (10)
IoT23-HPS | 60K | 0.8224 (40) | 0.8264 (60) | 0.8221 (50)
Coronary | 26K | 0.6235 (20) | 0.6236 (26) | 0.6282 (26)
Figure 6: Comparison between Hash-Comb and DP

##  8 Conclusion and Outlook

While data privacy is often mentioned as the motivation for Federated Learning
schemes, many popular techniques for federated ML models training cannot yet
achieve certifiable privacy and confidentiality. Also, they are often clumsy
to implement. We described a novel procedure that smoothly combines randomized
quantization to achieve differential privacy and multi-hashing (using
standard, regulation-compliant hashes) to provide privacy, confidentiality and
regulatory compliance. We empirically studied the performance of our technique
and demonstrated that, compared to classic Differential Privacy approaches, it
delivers an improved privacy-accuracy trade-off and a much smaller footprint.

## Acknowledgement

We would like to express our sincere appreciation to ASPIRE for their
financial support (Award Number AARE21-366).

## References

  * Pugliese et al. [2021] R. Pugliese, S. Regondi, R. Marini,  Machine learning-based approach: Global trends, research directions, and regulatory standpoints,  Data Science and Management 4 (2021) 19–29.
  * El Mestari et al. [2024] S. Z. El Mestari, G. Lenzini, H. Demirci,  Preserving data privacy in machine learning systems,  Computers & Security 137 (2024) 103605.
  * Jegorova et al. [2022] M. Jegorova, C. Kaul, C. Mayor, A. Q. O’Neil, A. Weir, R. Murray-Smith, S. A. Tsaftaris,  Survey: Leakage and privacy at inference time,  IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
  * Caroline et al. [2020] B. Caroline, B. Christian, B. Stephan, B. Luis, D. Giuseppe, E. Damiani, H. Sven, L. Caroline, M. Jochen, D. C. Nguyen, et al., Artificial intelligence cybersecurity challenges; threat landscape for artificial intelligence, 2020.
  * Mauri and Damiani [2021] L. Mauri, E. Damiani,  Stride-ai: An approach to identifying vulnerabilities of machine learning assets,  in: 2021 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, 2021, pp. 147–154.
  * Caroline et al. [2021] B. Caroline, B. Christian, B. Stephan, B. Luis, D. Giuseppe, E. Damiani, H. Sven, L. Caroline, M. Jochen, D. C. Nguyen, et al., Securing machine learning algorithms, 2021.
  * Youn et al. [2023] Y. Youn, Z. Hu, J. Ziani, J. Abernethy, Randomized quantization is all you need for differential privacy in federated learning, 2023\. arXiv:2306.11913.
  * Almahmoud et al. [2022] A. Almahmoud, E. Damiani, H. Otrok,  Hash-comb: A hierarchical distance-preserving multi-hash data representation for collaborative analytics,  IEEE Access 10 (2022) 34393–34403.
  * Cimato and Damiani [2018] S. Cimato, E. Damiani,  Some ideas on privacy-aware data analytics in the internet-of-everything,  From Database to Cyber Security: Essays Dedicated to Sushil Jajodia on the Occasion of His 70th Birthday (2018) 113–124.
  * Dwork [2006] C. Dwork,  Differential privacy,  in: International colloquium on automata, languages, and programming, Springer, 2006, pp. 1–12.
  * Dwork [2008] C. Dwork,  Differential privacy: A survey of results,  in: International conference on theory and applications of models of computation, Springer, 2008, pp. 1–19.
  * Ghosh et al. [2009] A. Ghosh, T. Roughgarden, M. Sundararajan,  Universally utility-maximizing privacy mechanisms,  in: Proceedings of the forty-first annual ACM symposium on Theory of computing, 2009, pp. 351–360.
  * Chen et al. [2011] R. Chen, N. Mohammed, B. C. Fung, B. C. Desai, L. Xiong,  Publishing set-valued data via differential privacy,  Proceedings of the VLDB Endowment 4 (2011) 1087–1098.
  * Leoni [2012] D. Leoni,  Non-interactive differential privacy: a survey,  in: Proceedings of the First International Workshop on Open Data, 2012, pp. 40–52.
  * Dwork et al. [2006] C. Dwork, F. McSherry, K. Nissim, A. Smith,  Calibrating noise to sensitivity in private data analysis,  in: Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, Springer, 2006, pp. 265–284.
  * Dong et al. [2022] J. Dong, A. Roth, W. J. Su,  Gaussian differential privacy,  Journal of the Royal Statistical Society Series B: Statistical Methodology 84 (2022) 3–37.
  * McSherry and Talwar [2007] F. McSherry, K. Talwar,  Mechanism design via differential privacy,  in: 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), 2007, pp. 94–103. doi:10.1109/FOCS.2007.66.
  * Lyu [2022] X. Lyu,  Composition theorems for interactive differential privacy,  Advances in Neural Information Processing Systems 35 (2022) 9700–9712.
  * Dwork et al. [2014] C. Dwork, A. Roth, et al.,  The algorithmic foundations of differential privacy,  Foundations and Trends® in Theoretical Computer Science 9 (2014) 211–407.
  * Mironov [2017] I. Mironov,  Rényi differential privacy,  in: 2017 IEEE 30th computer security foundations symposium (CSF), IEEE, 2017, pp. 263–275.
  * Zong et al. [2021] H. Zong, Q. Wang, X. Liu, Y. Li, Y. Shao,  Communication reducing quantization for federated learning with local differential privacy mechanism,  in: 2021 IEEE/CIC International Conference on Communications in China (ICCC), 2021, pp. 75–80. doi:10.1109/ICCC52777.2021.9580315.
  * Yan et al. [2023] G. Yan, T. Li, K. Wu, L. Song,  Killing two birds with one stone: Quantization achieves privacy in distributed learning,  arXiv preprint arXiv:2304.13545 (2023).
  * Agarwal et al. [2018] N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, B. McMahan,  cpsgd: Communication-efficient and differentially-private distributed sgd,  in: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (Eds.), NIPS’18: Proceedings of the 32nd International Conference on Neural Information Processing Systems, Curran Associates, Inc., 2018, p. 7575–7586.
  * Yan et al. [2023] G. Yan, T. Li, T. Lan, K. Wu, L. Song,  Layered randomized quantization for communication-efficient and privacy-preserving distributed learning,  arXiv preprint arXiv:2312.07060 (2023).
  * Frane [1998] J. W. Frane,  A method of biased coin randomization, its implementation, and its validation,  Drug Information Journal: DIJ/Drug Information Association 32 (1998) 423–432.
  * Agrawal et al. [2019] R. Agrawal, Y.-H. Chen, T. Horel, S. Vadhan,  Unifying computational entropies via kullback–leibler divergence,  in: Annual International Cryptology Conference, Springer, 2019, pp. 831–858.
  * Van Erven and Harremos [2014] T. Van Erven, P. Harremos,  Rényi divergence and kullback-leibler divergence,  IEEE Transactions on Information Theory 60 (2014) 3797–3820.
  * Roychowdhury and Roychowdhury [2022] L. Roychowdhury, M. K. Roychowdhury, Quantization for a probability distribution generated by an infinite iterated function system, 2022. arXiv:1603.00731.
  * Liu and Talwar [2019] J. Liu, K. Talwar,  Private selection from private candidates,  in: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, 2019, pp. 298–309.
  * Papernot and Steinke [2021] N. Papernot, T. Steinke,  Hyperparameter tuning with renyi differential privacy,  arXiv preprint arXiv:2110.03620 (2021).
  * Lindell [2020] Y. Lindell, Secure multiparty computation (mpc), Cryptology ePrint Archive, Paper 2020/300, 2020\. URL: https://eprint.iacr.org/2020/300. doi:10.1145/3387108, https://eprint.iacr.org/2020/300.
  * Shamir [1979] A. Shamir,  How to share a secret,  Communications of the ACM 22 (1979) 612–613.
  * Veugen et al. [2015] T. Veugen, F. Blom, S. J. de Hoogh, Z. Erkin,  Secure comparison protocols in the semi-honest model,  IEEE Journal of Selected Topics in Signal Processing 9 (2015) 1217–1228.
  * Ishai et al. [2009] Y. Ishai, M. Prabhakaran, A. Sahai,  Secure arithmetic computation with no honest majority,  in: Theory of Cryptography: 6th Theory of Cryptography Conference, TCC 2009, San Francisco, CA, USA, March 15-17, 2009. Proceedings 6, Springer, 2009, pp. 294–314.
  * Tezcan [2022] C. Tezcan,  Key lengths revisited: Gpu-based brute force cryptanalysis of des, 3des, and present,  Journal of Systems Architecture 124 (2022) 102402. URL: https://www.sciencedirect.com/science/article/pii/S1383762122000066. doi:https://doi.org/10.1016/j.sysarc.2022.102402.
  * Hopkins et al. [1999] M. Hopkins, E. Reeber, G. Forman, J. Suermondt, Spambase, UCI Machine Learning Repository, 1999. DOI: https://doi.org/10.24432/C53G6X.
  * Sebastian Garcia [2020] . M. J. E. Sebastian Garcia, Agustin Parmisano, Iot-23: A labeled dataset with malicious and benign iot network traffic (version 1.0.0) [data set]. zenodo. http://doi.org/10.5281/zenodo.4743746, 2020.
  * Mauri and Damiani [2021] L. Mauri, E. Damiani,  Estimating degradation of machine learning data assets,  ACM Journal of Data and Information Quality (JDIQ) 14 (2021) 1–15.
  * Li et al. [2020] Y. Li, T.-H. Chang, C.-Y. Chi,  Secure federated averaging algorithm with differential privacy,  in: 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), 2020, pp. 1–6. doi:10.1109/MLSP49062.2020.9231531.


Machine Learning for QoS Prediction in Vehicular Communication: Challenges and Solution Approaches

This work was supported by the Federal Ministry of Education and Research (BMBF) of the Federal Republic of Germany as part of the AI4Mobile project (16KIS1170K). The authors alone are responsible for the content of the paper.

1]Alexandros Palaios
2]Christian L. Vielhaus
3]Daniel F. Külzer
1]Cara Watermann
4]Rodrigo Hernangomez
5]Sanket Partani
1]Philipp Geuer
6]Anton Krause
5]Raja Sattiraju
4]Martin Kasparick
6]Gerhard Fettweis
2]Frank H. P. Fitzek
5]Hans D. Schotten
4,7]Slawomir Stańczak

[1]Ericsson Research, Germany, {alex.palaios, cara.watermann<EMAIL_ADDRESS>[2]Deutsche Telekom Chair, Technische Universität Dresden, Germany, {christian.vielhaus<EMAIL_ADDRESS>[3]BMW Group, Germany<EMAIL_ADDRESS>[4]Fraunhofer Heinrich Hertz Institute,  Germany, {rodrigo.hernangomez, martin.kasparick<EMAIL_ADDRESS>[5]Technische Universität Kaiserslautern, Germany, {partani, sattiraju<EMAIL_ADDRESS>[6]Vodafone Chair, Technische Universität Dresden, Germany, {anton.krause<EMAIL_ADDRESS>[7]Network Information Theory Group,Technische Universität Berlin, Germany

As cellular networks evolve towards the 6g, ml is seen as a key enabling technology to improve the capabilities of the network. Machine learning provides a methodology for predictive systems, which, in turn, can make networks become proactive. This proactive behavior of the network can be leveraged to sustain, for example, a specific qos requirement. With pqos, a wide variety of new use cases, both safety- and entertainment-related, are emerging, especially in the automotive sector. Therefore, in this work, we consider maximum throughput prediction enhancing, for example, streaming or high-definition mapping applications. We discuss the entire ml workflow highlighting less regarded aspects such as the detailed sampling procedures, the in-depth analysis of the dataset characteristics, the effects of splits in the provided results, and the data availability. Reliable ml models need to face a lot of challenges during their lifecycle. We highlight how confidence can be built on ml technologies by better understanding the underlying characteristics of the collected data. We discuss feature engineering and the effects of different splits for the training processes, showcasing that random splits might overestimate performance by more than twofold. Moreover, we investigate diverse sets of input features, where network information proved to be most effective, cutting the error by half. Part of our contribution is the validation of multiple ml models within diverse scenarios. We also use xai to show that ml can learn underlying principles of wireless networks without being explicitly programmed. Our data is collected from a deployed network that was under full control of the measurement team and covered different vehicular scenarios and radio environments.

Intelligent transportation systems, machine learning, quality of service, throughput prediction, vehicular communication, wireless networks.

§ INTRODUCTION

ml has a strong potential to overcome challenges arising in vehicular communication and networking, as presented in <cit.>. Examples of such challenges are resource allocation <cit.> and qos prediction <cit.>. The qos prediction, in turn, is an enabler for a variety of use cases in the domain of vehicular communication, such as autonomous driving, platooning, cooperative maneuvering, tele-operated driving, and smart navigation. Some of these are presented in <cit.>.
Predictions can not only enhance the existing use cases and support new ones, but also be a stepping stone for a network to become proactive. Such predictions can be an integral part of a network that proactively reacts to sustain, for example, a specific kpi target, like the maximum achievable throughput under high network load.

In this work, we focus on the prediction of qos as a way of looking into the typical workflow for building and applying ml algorithms. We start with data acquisition, moving on to data analysis and statistical characterization of the data, proceeding to feature engineering and different split strategies, and concluding by testing the performance of the model.
Predicting qos is a relatively complex task since it depends on several time-varying factors. It is especially complex in the case of vehicular communication where the radio conditions might change very drastically in a short amount of time <cit.>.

Capturing datasets that reveal the complex inter-dependencies between network operations, radio environment and terminal behavior is a challenging task in itself. The captured data might often lack the required quality for ml-based prediction, rendering it inappropriate. There are multiple factors that can reduce the quality of a collected dataset for training ml models, with a few examples being dataset imbalance, coverage of radio regions with bounded dynamics or limited network states gathered. Moreover, acquiring different types of data for a prediction task comes with varying acquisition costs that are typically not discussed extensively.
From the theoretical perspective of statistical learning, a training dataset should often be iid and drawn from the same probability distribution as the test dataset <cit.>.
Radio terminals frequently experience drastic shifts in the data distribution <cit.> that usually degrade the performance of ml models <cit.>.

Meaningful comparisons between prior art proposals, though desirable, can be hard. There are multiple reasons for this, for example, the pre-processing steps are not well reported, or the used performance metrics differ.
Additionally, literature in the area of applying ML for wireless communications often reports the performance of the models without giving an in-depth analysis of the dataset’s underlying characteristics. This raises the question of how robust such models are to changes in different environments, where the dataset’s statistical properties might be different.

Our results emerge from the data analysis of a dedicated measurement campaign with the goal of capturing several characteristics of the radio environment and enabling an in-depth discussion of ml workflows.
The code for this analysis is based on customary functions from usual Python libraries for scientific computing
(e.g., numpy, pandas, scikit-learn...).
Our contributions include the following:

* We study the prediction of maximum throughput prediction enhancing, for example, streaming or high-definition mapping applications.

* We discuss the entire machine learning workflow highlighting less regarded, but important aspects, such as the detailed sampling procedures, the in-depth analysis of the dataset characteristics, and the effects of splits on performance. For example, we test against the data stationarity assumption and discuss methods to handle those for commercial networks. Many proposals in the literature avoid discussing such theoretical violations that many ML models require.

* We put the characteristics of wireless environments in the foreground explaining what might hinder and benefit the adoption of ML approaches for new-generation wireless networks.

* We further showcase why the lack of this type of information, missing often from the literature, can generate overconfidence in proposed ML solutions.

* We investigate the effects of diverse sets of input features specifically those that are not typically available in the prior art.

* We showcase that consumer devices and low sampling speeds can be enablers for the adoption of ML in commercial networks.

* We also use explainable AI to demonstrate that machine learning can learn underlying principles of wireless networks without being explicitly programmed.

* Our data is collected from a deployed network that avoids limitations of theoretical studies or simulation-based results.

The remaining of the paper is structured as follows:
Section <ref> presents related work in the area of pqos, and the measurement campaign is described in Section <ref>. In Section <ref>,
we discuss various radio environment properties in relation to ml workflows, such as vehicle speed and stationarity. We continue the analysis on ml workflows in Section <ref> with a careful look into train/test data splitting and feature engineering. In Section <ref>, we apply a set of ml algorithms to predict the maximum achievable throughput for diverse prediction models, communication directions, feature sets, splitting strategies, and prediction horizons. In Section <ref> we look at the topic of xai <cit.> showcasing what the models have learned in the absence of explicitly programmed rule sets. Finally, we draw conclusions in Section <ref>.

§ RELATED WORK

The application of ml for qos prediction in vehicular networks has drawn attention recently. In the following, we present an overview of related works.
Measurement campaigns that focus on the feasibility of tod are described in  <cit.>. The authors of  <cit.> also include a sensitivity analysis in their work and examine the impact of factors such as speed and distance to a serving cell.
When applying gathered data from measurements to qos prediction in the current literature, a sampling interval in the order of seconds is often used (e.g., <cit.>). This does not reflect the fast-changing wireless channel associated with high-mobility vehicular communication scenarios <cit.>, and there is an open question as to which degree sampling schemes need to be adjusted based on the vehicle speed. Some works already have compared measurement applications running on slow-sampling gnet with minipc analysis <cit.> and propagation model tuning <cit.>.
As vehicular communication is characterized by high mobility of the terminal, it is an open question whether gnet's low-cost transceiver chains might render collected data unsuitable for applying ml.

Moreover, it is known that the radio environment has dynamics that can drastically change its statistical properties over a distance of a few meters <cit.>. Such changes in statistical properties of the data stream, where the ml model needs to reformulate its decision boundaries, are being discussed in the literature as concept drift <cit.>. Detection and analysis of concept drift using state-of-the-art algorithms have been explained in <cit.>. Extensive research is conducted on the management of concept drift, some of which are: using an ensemble learning approach <cit.>, concept drift-aware federated averaging <cit.>, or model updating mechanisms <cit.>.

In <cit.> and <cit.> the authors present a measurement campaign and evaluate several ml models regarding their performance for dll throughput prediction. One limitation is that the authors measure in a public network and thus cannot consider features such as the cell load. The authors of <cit.> consider the movement of the user and build a two-staged ml model to predict the tcp throughput. In the first step, the movement pattern is identified. Based on this, a lstm, trained with corresponding data, is selected for the throughput prediction. Some preliminary results for ul and dll throughput prediction using Random Forest and employing the dataset used for this paper are presented in <cit.>. qos prediction in a 5G non-standalone network is examined in <cit.>. Besides throughput, ml is also used to predict latency <cit.> and handovers <cit.>. Recently, several datasets intended for ml-based studies have been made publicly available <cit.> that typically do not provide cell-related information. Moreover, those works focus on introducing the datasets instead of applying ml methods.

An inherent limitation of ml algorithms, especially as they get more complex, is that their decision-making is difficult to understand, rendering them into black boxes. Recently, the field of xai <cit.> has gained traction, to ensure ML models perform as desired, not only in test environments but also in target applications. With xai, models can be judged in the context of human domain knowledge, e.g. by investigating the learned feature importance.

We extend prior art by providing detailed insights into the aspects mentioned above. Our dedicated measurement campaign, on a fully controlled private network, allowed us to perform a deep analysis and enable an in-depth discussion. We could control which devices could connect to the network, we could adjust the total network interference, and generate diverse traffic dynamics. We consider different radio environments, device types, sampling frequencies, measurement scenarios, and prediction horizons. We present the complete workflow towards building a pqos model, predicting the maximum achievable throughput. We discuss several design decisions, such as the train-test split, various feature groups, and different models, and show the influence on the achieved performance. In the end, we discuss the issue of trustworthiness.

The map of the areas where the measurements took place. The A9 highway is highlighted in red, while the rural and side street areas are marked in green (not all rural streets are plotted to reduce clutter). The blue area is the suburban city of Feucht.

§ MEASUREMENT CAMPAIGN

Within the project AI4Mobile, an extensive measurement campaign was performed in the 5G-ConnectedMobility test field[https://www.ericsson.com/en/cases/2020/5g-connectedmobility]. In total, more than 3000 km were driven by four vehicles for one week to perform measurements from the lte network. A detailed description of the measurement campaign is available in <cit.>. Since we had control over the network infrastructure, we were able to measure and collect data from all parts of the network. We were also able to define custom scenarios and set parameters accordingly.
Subsequently, we briefly describe the setup and highlight some scenarios.

§.§ Measurement Setup

During the measurement campaign, four vehicles were equipped with several ue. In each vehicle, an identical minipc was used for measurements with high granularity. The minipc is a Linux-based PC including a cellular modem connected to a 2x2 MIMO car antenna, which is placed on the roof of the respective car. Additionally, a GPS receiver was placed on the roof and connected to the minipc, providing accurate location and time synchronization via a pps signal. To capture the data from the cellular modem, we used the application MobileInsight <cit.>.

Each vehicle was also equipped with several gnet phones as pure data generators and measurement devices with a low sampling frequency for a comparison to the minipc's measurements.
The gnet uses the application G-Net Track Pro <cit.> to capture radio measurements via the Android API. All devices used the application Iperf <cit.> to exchange data with a local server connected to the network. In addition to the ue (gnet and minipc), we also collected data from the base stations and the core network, which were also time-synchronized. An overview of all captured data is given in Table <ref>.

The network is deployed in the south of Germany, close to Nuremberg. It consists of 10 fdd-lte cells with 10 MHz of bandwidth at a carrier frequency of 700 MHz, with typical radio tower panel antennas (horizontal half-power beam width of 65$^{\circ}$). Since it is a test network, our devices were the only devices connected to the network. In Fig. <ref>, the area of the test field is shown, including the position of the distinct cells. While the cells were positioned to provide excellent connectivity along the highway and in the suburban city of Feucht, part of the rural area next to the highway was also covered, which allowed us to capture several radio environments.

§.§ The Conducted Studies

The purpose of the measurements was to thoroughly examine numerous parameters through a variety of real-time scenarios in our captured dataset. This involved analyzing different driving patterns at different speeds, stationary measurements, recording different radio environments, and generating varied data traffic to create contrasting load scenarios. Furthermore, we utilized various protocols and generated traffic in both the ul and dll to the server and to other vehicles. Additional information on the parameters captured in the measurement campaign can be found in <cit.>.

Simplified view of the used measurement methodology. All clocks are synchronized and all generated data is marked accordingly and captured at separate entities. Figure adapted from <cit.>.

§.§ Data Preprocessing

For this study, only udp measurements from the dataset were considered since the protocol properties of tcp and udp differ. Therefore the E2E throughput could not be limited by the transport layer.
We used the data, where the devices were requesting maximum throughput.

Moreover, we filtered out stationary measurements, i.e., when vehicles were parked for some time. The stationary measurements contain similar radio properties for the majority of the samples, which could result in overly optimistic prediction results.

§ PROPERTIES OF THE RADIO ENVIRONMENT

In this section, we study properties of the radio environment that might influence ml algorithms. Radio environments have very diverse characteristics that depend on multiple factors. Some of these factors are environment-based, for example, radio propagation can be drastically different in a rural area and in a crowded city center <cit.>, as the type and density of buildings influence the radio propagation. Another factor is, for example, the specific network deployment (e.g. antenna heights, frequency layers and the number of cells in an area). All these factors contribute to very unique properties in the collected data as UEs move. We first discuss the data collection procedures and how these might affect the quality of the collected dataset. We continue looking at the statistical properties of the radio environment. As many ml algorithms rely on multiple statistical assumptions <cit.>, we try to highlight some of those, as they might need to be better understood by the research community that applies ml on wireless networks.

§.§ Sampling the Radio Environment

High sampling intervals of the radio environment come at a cost, either in terms of acquiring more capable hardware, or in terms of higher power consumption or in terms of signaling, as more data need to be transferred. We specifically study the effects of different sampling intervals on the characteristics of the collected dataset. We also study the effects of the vehicular speed in the characteristics of the collected dataset to see if higher velocities call for faster sampling intervals, for sustaining specific quality characteristics in the collected dataset.
We focus on lte signal values for the analysis, i.e., rsrp and rssi, since these are most affected by the vehicle's speed and influence the target throughput, in turn, as we show in Section <ref> and in Section <ref>

rsrp (minipc/10 ms)

rssi (minipc/10 ms)

rsrp (gnet/1 s)

rssi (gnet/1 ms)

Relation between speed and mean absolute error (MAE) for different sampling intervals (shaded areas indicate the interquartile range).

To quantify the sensitivity of the collected measurements on these two parameters, we need to introduce the de at the $k$-th time instance as

\begin{equation}
\label{eq::DE}
RE(k) = x_{o}(k) - x_{rs}(k) \; , \; 0 \le k < K \; ,
\end{equation}

where $x_{o}(k)$ represents the original signal of length $K$ at a sampling interval of 10ms for the minipc and 1s for the gnet. The corresponding resampled value for a lower sampling interval is $x_{rs}(k)$. To determine $x_{rs}(k)$, the following procedure is applied: first, the original dataset is downsampled by averaging to a target rate to obtain $x_{ds}\left(k'\right)$:

\begin{equation}
\label{eq::ds}
x_{ds}\left(k'\right)=\frac{1}{M}\sum_{m=0}^{M-1}{x_o\left(m+k'M\right)} \; , \; 0 \le k' < K/M \; ,
\end{equation}

where $M$ accounts for the downsampling ratio for a certain pair of original and target sampling interval, e.g., $M=10$ for a target rate of 10s when considering the gnet's original sampling interval of 1s. Secondly, the downsampled dataset are then upsampled by the forward filling method, which can be expressed by $M$-integer division on the original index $k$:

\begin{equation}
\label{eq::us}
x_{rs}\left(k\right)=x_{ds}\left(\left\lfloor k/M \right\rfloor\right)=x_{ds}\left(\frac{k-(k\mod{M})}{M}\right)\;.
\end{equation}

Finally, Equations (<ref>), (<ref>) and (<ref>) can be combined into

\begin{equation}
RE(k) = \frac{1}{M}\sum_{m=0}^{M-1}{x_o\left(k\right)-x_o\left(m+k-(k\mod{M})\right)}
\;.
\end{equation}

For each sample at the $k$-th time instance, the vehicle speed $v(k)$ is also known, which allows to explore the relation between the mobility of the terminal and the resulting de. Therefore, the speed range from 0 to 140 is split in intervals of 5. For each interval with a lower speed boundary $v'$, the mean absolute de $\overline{RE}_{v'}$ is calculated according to Eq. (<ref>).

\begin{equation}
\label{eq::mean_DE}
\overline{RE}_{v'} = \frac{1}{K} \sum_{v' \leq v(k) < v'+5\,km/h} |RE(k)|
\end{equation}

The results are shown for rsrp and rssi and measurements from the minipc and gnet in Fig. <ref>. The shaded areas indicate the central half of the distribution of the errors.

Focusing on the effect of different sampling intervals for a minipc dataset (Figs. <ref> and <ref>), one can see that the de increases with a larger sampling interval as expected. Interestingly, this increase seems to be below 2dB for a sampling interval of 1s and between 2 and 3dB for a sampling interval of 10s, which might still be acceptable for certain ml applications.

The mobility of the terminal seems to have some effect on the quality of the collected dataset. When the terminal is stationary, the de is close to 0. Both rsrp and rssi are characterized by a steep rise of the error once the terminal starts to move at an approximate speed of 10. Above that speed, the error remains stable. Of particular interest is a sampling interval of 1s, as it shows approximately the error introduced by employing a gnet instead of a minipc, which collects data at a sampling interval of 10ms. The mean de for rsrp and rssi are well below 2 dB, which suggests that also gnet samples may enable an accurate qos prediction. Only the sampling interval of 10s shows larger variations across different speeds. Here we notice that the variance of the error is drastically increased for the lower sampling interval.

We also executed the same experiments with a dataset from the less granular gnet. We received similar results for the minipc dataset as shown in Fig.  <ref> and <ref>, i.e., a rise of the error in the beginning, followed by a saturation of the de. We empathize that a direct comparison of the resulting de between the two different devices should be avoided, as the original sampling intervals are larger and thus the number of samples that are considered for each downsampling interval is smaller. This, combined with the simpler transceiver chain of the gnet, is a potential explanation for why these results are less smooth.

The results indicate that the vehicle speed does not have a noticeable impact on the absolute error for different sampling intervals, except for vehicles moving at very low speeds. The observed effects can possibly be attributed to the Doppler effect. However, no channel measurements were carried out during the measurement campaign and thus the causes cannot be conclusively assessed on the basis of our data set. The relatively constant error for slow sampling at different speeds can further simplify the sampling procedures without the need to introduce adaptive sampling schemes. Also, lower sampling speeds can still provide useful input to ml models as the reconstruction error remains low. This type of result is necessary for optimizing data collection procedures for future ml applications.
In the following sections, we use downsampled data from the minipc with a sampling interval of 1s that also enables a comparison with the gnet.
The resulting dataset sizes for DL and UL are 18846 and 19535 samples, respectively.

§.§ Data Stationarity

Many theoretical results in ml literature are based on the assumption that the
available dataset is iid <cit.>.
This assumption can be violated in several ways, for instance, if
i) samples are correlated in time (i.e., they are not independent), or
ii) the underlying time series is non-stationary (i.e., not identically distributed).

There are both parametric and non-parametric hypothesis tests available in the literature to study time stationarity.
Parametric tests are often more powerful, but one needs to assure that the assumptions they
require (i.e., a specific well-known underlying distribution) are met by the tested data.
As such, we have opted to use the parametric
adf test <cit.>, which searches a unit root in the tested time series
as its null hypothesis,
while the alternative hypothesis is stationarity. In other words, a low $p$-value
(for our analysis we have considered $p<0.05$) indicates strong evidence for stationarity
in the data.
As a parametric test, adf requires a suitable choice of the maximum lag
of the assumed autoregressive model.
Since the lag order is unknown for our data, we estimate it according to the aic <cit.>.

We have run the adf test for all phy features from all measurements (c.f. <Ref>, first row) in an accumulated manner.
That is, for every analyzed time series of size $T$, we construct $T$ sub-series each one consisting of all
samples from 0 to $t\in\left\lbrace 0,1,\ldots,T-1\right\rbrace$.
We then run the adf test against every sub-series and concatenate the output
$p$-values into a vector of size $T$. Fig. <ref>
shows the resulting accumulated adf test
as a color code on top of some time series.

Low-variance scenario

High-variance scenario

Scenario shift

Stationarity analysis of the data using the adf test.

The expected behavior is that the accumulated adf test tends to $p\ll0.05$ as the
size of the evaluated time sub-series increases. The number of samples needed for this depends
on some statistics, such as the variance (c.f. Fig. <ref> & <ref>),
but it rarely exceeds 15 minutes for our data.

This time stationarity assumption is violated for measurements with an appreciable scenario shift, e.g., those where vehicles
remain idle for some time and drive away afterward (Fig. <ref>).
This shows that stationarity cannot always be assumed, even though it might hold true
for some datasets under a long-enough measurement time. For the measurement scenarios, we opted for a duration of 40 to 60 minutes while driving to alleviate the problem of non-stationarities.

In summary, we acknowledge the presence of non-stationary portions in our data and its possible negative impact
on the performance  of ml algorithms.

§.§ Radio Environment Correlations

Additionally, we look at an important characteristic of the radio environment, which is the correlation in time and space <cit.>. In Fig. <ref>, we show an example of the autocorrelation function of the sinr for three distinct radio environments. We see that the radio environment tends to be highly correlated in time for all three distinct radio environments. Some higher correlations are noted for the rural environment, with the highway having the weakest of the three. This is probably explained by the faster movement of the vehicles.

Autocorrelation of the SINR for excerpts of different DL measurement runs.

In <cit.>, the authors discuss that if correlations are not handled properly, there is a higher risk of reporting overly optimistic results. The reason is that the ml model does not learn the underlying relations between input and output clearly, and instead learns the existing dependencies of a dataset, which in our case, are the reported correlations.