# Covert Channel Attack to Federated Learning Systems

Gabriele Costa1, Fabio Pinelli1, Simone Soderi1, Gabriele Tolomei2
1IMT School for Advanced Studies, Lucca<EMAIL_ADDRESS>
2Sapienza University of Rome<EMAIL_ADDRESS>
###### Abstract

Federated learning (FL) goes beyond traditional, centralized machine learning
by distributing model training among a large collection of edge clients. These
clients cooperatively train a global, e.g., cloud-hosted, model without
disclosing their local, private training data. The global model is then shared
among all the participants which use it for local predictions.

In this paper, we put forward a novel attacker model aiming at turning FL
systems into covert channels to implement a stealth communication
infrastructure. The main intuition is that, during federated training, a
malicious sender can poison the global model by submitting purposely crafted
examples. Although the effect of the model poisoning is negligible to other
participants, and does not alter the overall model performance, it can be
observed by a malicious receiver and used to transmit a single bit.

###### Index Terms:

federated learning, adversarial attacks, covert channel.

##  I Introduction

_Federated learning_ (FL) [1, 2] recently emerged as the leading technology
for implementing distributed, large scale and efficient machine learning (ML)
infrastructures. The main idea is that multiple clients connect to the FL
system, and collaboratively train a shared, global model. A frequent setting
in FL is to have a centralized, e.g., cloud-hosted, server and many edge
clients. Clients and server iterate the execution of FL rounds, which consist
of the following steps.

  1. 1.

The server sends the current, global model to every client and selects some of
them for the next training round.

  2. 2.

Each selected client trains its local model using its own private data
starting from the received global model, and sends the resulting local model
back to the server.

  3. 3.

The server applies an _aggregation function_ to the local models received by
clients and updates the global model.

FL allows clients to concurrently train a shared global model, without
disclosing private training data. Hence, FL provides great advantages in terms
of both scalability and privacy. Since this process smoothly integrates with
ubiquitous, distributed infrastructures, it has been widely applied to IoT
[3], Fog computing [4], autonomous vehicles [5], smartphones [6], and wearable
devices [7]. As a result, nowadays billions of devices are connected to one or
more FL systems.

The growing adoption of FL also raises security concerns, in particular, about
the confidentiality, integrity, and availability of FL systems. Indeed,
several authors considered attack scenarios targeting FL systems. For
instance, in the last years some works investigated _data poisoning_ attacks,
where an adversary pollutes the training set with maliciously crafted examples
[8]. A similar scenario is that of _model poisoning_ , in which the attacker
directly attempts to tamper with the global model parameters [9]. The aim of
both these types of attacks is usually to detriment the prediction accuracy of
the learned model. Also, a large body of work deals with privacy leakage that
may expose the local data of some clients [10].

In this paper, we discuss a new attack scenario: we consider an adversary
implementing a _covert channel_ [11] over an FL system. Covert channels allow
an attacker to establish an illicit communication between two agents (e.g.,
two devices) that should stay isolated. In theory, since no trust relationship
exists among the clients, FL should not be intended to support the creation of
covert channels. In practice, being shared among the participants, the global
model can be turned into a communication channel. More specifically, two FL
clients, i.e., a sender and a receiver, can agree on an aimed poisoning
strategy that allows them to transfer one bit.

We start by describing our attacker model and the covert channel
implementation strategy. Our attacker only requires limited capabilities and,
thus, it appears very realistic. As a matter of fact, communications are
established by poisoning the training set of a single client, i.e., the
sender. Poisoned examples are crafted by modifying benign input examples
through a simple, effective and efficient heuristics.

We explain the implementation strategy behind this attack by considering an FL
system, where clients collaboratively train a global model to recognize
handwritten digits of the popular MNIST dataset [12]. In our FL system,
private training data is represented by a subset of the MNIST dataset that is
randomly assigned to each client. Such an implementation is often given as a
template of a generic FL system in official tutorials.111For example, see
https://www.tensorflow.org/federated/tutorials/federated_learning_for_image_classification.

The main contributions of this paper are listed below.

  * •

A novel attacker model for FL-based covert channel.

  * •

An implementation strategy working on any FL system.

  * •

A discussion of the main channel features.

The rest of the paper is organized as follows. Section II describes the main
background concepts used in this work. In Section III, we revise the
literature about FL security and application level covert channels. Section IV
introduces our attacker model, and Section V details the implementation of the
covert channel. In Section VI, we describe the properties of our covert
channel. Finally, Section VII concludes the paper.

##  II Background

In this section, we provide the reader with the essential context needed to
understand the subject of this work.

###  II-A Machine & Federated Learning

We consider the supervised learning task as the reference example of a typical
ML problem. The goal of supervised learning is to estimate a function that
maps an input to an output, based on a sample of observed input-output pairs,
called _examples_ , which is usually referred to as _training set_.

More formally, let $\mathcal{D}=\\{(\boldsymbol{x}_{i},y_{i})\\}_{i=1}^{n}$ be
a training set of $n$ examples. Each
$\boldsymbol{x}_{i}\in\mathcal{X}\subseteq\mathbb{R}^{d}$ is a $d$-dimensional
vector of _features_ representing the $i$-th input and $y_{i}\in\mathcal{Y}$
is its corresponding output value. Here, we focus on _classification_ problems
where $\mathcal{Y}=\\{1,\ldots,\ell\\}$ and each $y_{i}$ is known as the
_class label_ (as opposed to regression problems where
$\mathcal{Y}\subseteq\mathbb{R}$).

Supervised learning assumes the existence of an unknown target function
$g:\mathcal{X}\mapsto\mathcal{Y}$ that maps any feature vector to its
corresponding output. The goal is therefore to estimate a function $m^{*}$,
namely a parametric model, that best approximates $g$ on $\mathcal{D}$. More
specifically, the optimal parametric model $m^{*}$ is the one that minimizes
the value of a loss function $\mathcal{L}$, which measures the cost of
replacing the true $g$ with $m^{*}$ on the training set. In other words,
learning $m^{*}$ reduces to the following optimization problem, also known as
empirical risk minimization (ERM) [13].

$m^{*}=\text{argmin}_{m}\mathcal{L}(m,\mathcal{D})$ (1)

Depending on the supervised learning task, different loss functions can be
adopted. For example, cross-entropy is a commonly used loss function for
classification [14], whilst mean squared error is typically employed in
regression settings [15].

The standard framework above assumes that the actual training procedure, i.e.,
the optimizer used to solve (1), runs on a centralized location where the
whole dataset $\mathcal{D}$ is stored. In the case of FL, instead, the
learning process is distributed among several clients that collaboratively
train a shared, global model with a centralized server acting as orchestrator.
Thus, the FL framework consists of a centralized server $S$ and a set of
distributed, federated clients $\mathcal{C}$, such that $|\mathcal{C}|=n_{c}$.
Each client $c\in\mathcal{C}$ has access to its own private training set
$\mathcal{D}_{c}$, namely the set of its local labeled examples.

The generic $t$-th round of FL comprises the following steps.

  1. 1.

$S$ sends the current, global model $m^{(t)}$ to every client and selects a
subset ${\mathcal{C}}^{(t)}\subseteq\mathcal{C}$, such that
$1\leq|{\mathcal{C}}^{(t)}|\leq n_{c}$.

  2. 2.

Each selected client $c\in{\mathcal{C}}^{(t)}$ trains its local model
$m_{c}^{(t)}$ by optimizing the same objective of (1) on its own private data
$\mathcal{D}_{c}$, starting from $m^{(t)}$; the resulting $m_{c}^{(t)}$ is
thus sent back to $S$.

  3. 3.

$S$ computes $m^{(t+1)}=\phi(\\{m_{c}^{(t)}~{}|~{}c\in\mathcal{C}^{(t)}\\})$
as the updated global model, where $\phi$ is an _aggregation function_ (e.g.,
FedAvg [2] or one of its variants [16]).

At the beginning, $m^{(0)}$ may be randomly initialized. Then, FL rounds as
the one described above are iteratively executed until convergence of the
global model, i.e., until $t=T$ such that $m^{(T)}=m^{*}$. In practice,
though, many FL models are continuously trained due to the highly dynamic
nature of the infrastructure (e.g., new clients joining or leaving the system
and fresh local data generated over time).

To simplify the notation, in the following we refer to $m$ as the global model
and to $m_{c}$ as the local model of client $c$. Furthermore, we call
$m(\boldsymbol{x})=\hat{y}\in\mathcal{Y}$ (resp.,
$m_{c}(\boldsymbol{x})=\hat{y}_{c}\in\mathcal{Y}$) the global (resp., local)
model prediction on input $\boldsymbol{x}$.

###  II-B Channels and communication quality

Figure 1: Binary memoryless covert channel.

In this paper, we are interested in _binary memoryless channels_ (BMC) [17].
Briefly, an BMC has discrete input $b$ and output $b^{\prime}$ such that
$b,b^{\prime}\in\\{0,1\\}$. We describe the relationship between channel’s
input and output through the conditional probability
$P(\mathbf{b}|\mathbf{b})$, where $\mathbf{b}=\\{0,1\\}$. Figure 1 depicts a
generic BMC in which $P(1|0)=p_{1}$ and $P(0|1)=p_{2}$ are the probabilities
for input/output bit inversion errors. Consequently, their complements give
the probability of receiving the correct bit, e.g., $P(0|0)=1-p_{1}$. For a
BMC, the _channel capacity_ $C$ is the maximum communication rate that sender
and receiver can reach over the channel. Following [17], $C$ is computed as
follows.

$C=\max_{P}\mathbb{I}(x;y)=\sum\limits_{\begin{subarray}{c}x\in\mathbf{b}\\\
y\in\mathbf{b}\end{subarray}}P(x)P(y|x)\log\Bigg{(}\frac{P(y|x)}{P(y)}\Bigg{)},$
(2)

where $\mathbb{I}(x;y)$ is the mutual information between the input $x$ and
output $y$. From (2), we obtain the channel capacity as

$\displaystyle C=$
$\displaystyle\>1+\sum_{i=1,2}\frac{1}{2}(p_{i}+(1-p_{i})\log_{2}(1-p_{i}))=$
$\displaystyle=$
$\displaystyle\>1-\sum_{i=1,2}\frac{H(p_{i})}{2}=1-\frac{H(p_{1})}{2}-\frac{H(p_{2})}{2},$
(3)

where $H$ is the binary Shannon entropy function.

Communication quality is typically measured in terms of _bit error rate_ (BER)
and _signal-to-noise ratio_ (SNR). In general, BER is obtained as the number
of bit inversions, e.g., due to channel noise, over the total number of
transmitted bits. Instead, SNR is defined as the ratio of signal power to the
noise power, i.e., $s^{2}/n^{2}$, where $s$ is the channel signal and $n$ is
the noise. In general, when signal and noise are modeled by means of two
random variables, called $S$ and $N$ (respectively), we have that
$s^{2}=E[S^{2}]$ and $n^{2}=E[N^{2}]$, where $E[\cdot]$ denotes the expected
value. Moreover, when $N$ has zero mean, i.e., $E[N]=0$, $n^{2}$ reduces to
$\sigma^{2}_{n}=Var(N)=E[N^{2}]-E[N]^{2}$, that is $E[N^{2}]$ is equal to the
variance of $N$.

##  III Related work

Our proposal consists of an application level covert channel implemented
through an adversarial attack to an FL system. Although some authors reviewed
the security challenges of FL, e.g., see [18] for an extensive discussion, we
are not aware of papers describing similar, covert channel scenarios. Thus, in
the following, we revise some related work about _adversarial attacks to ML_ ,
and _application level covert channels_ which are closer to our proposal.

###  III-A Adversarial attacks to ML

There exists a large body of work investigating the security of both
traditional, i.e., centralized, and federated ML against so-called adversarial
attacks [19, 20, 21, 22, 23, 24, 25, 26, 27]. Adversarial attacks can have
different targets. For instance, model inversion attacks [28, 29], membership
inference attacks [30, 31, 32], and property inference attacks [33, 34] aim to
violate the confidentiality of user’s private training/test data. Also, it has
been reported that an attacker can compromise the confidentiality/intellectual
property of a model provider by stealing its model parameters and
hyperparameters [35, 36, 37].

From this perspective, our attacker model belongs to adversarial attacks that
tamper with the integrity and the performance of a predictive model [38].
These attacks are classified according to the stage(s) of the ML pipeline in
which they occur. In particular, attacks can happen at training time only,
both at training and at test time, or at test time only. Those are called
_poisoning_ , _backdoor_ , and evasion attacks, respectively.

From another viewpoint, depending on the attacker’s goal, adversarial attacks
can be further classified as untargeted (or random) [39, 40, 41, 42, 43, 8] or
targeted [44, 45, 46, 47]. The former aims to reduce the overall accuracy of
the learned model at inference time, regardless of what specific testing
examples get incorrectly classified. The latter forces the learned model to
output attacker-desired labels for certain testing examples, e.g., predicting
spam messages as non-spam, while not altering the output for other examples.
Since targeted attacks have to do with a specific goal, they usually require
rather strong attacker’s capabilities.

In the following we provide a detailed overview of the most prominent
adversarial attacks.

#### Poisoning Attacks

To compromise the performance of a predictive model, poisoning attacks can
target two components of the training stage, i.e., the training dataset and
the learning process. The former are known as data poisoning attacks [44, 20,
40, 48, 8]. The latter are referred to as model poisoning attacks [49, 50].

Data Poisoning. These attacks pollute the training dataset by injecting it
with new malicious examples or by corrupting existing ones. There are two main
types of data poisoning attacks, called _(i)_ _clean-label_ [45] and _(ii)_
_dirty-label_ [51], respectively. In clean-label poisoning attacks, the
adversary has no control over the labeling process. The attacker simply
injects a small number of slightly perturbed examples (whose labels remain
correct) into the training set of the victim. These attacks have been proven
effective only when the attacker has complete knowledge of the victim’s model,
i.e., under _white-box_ assumption [23]. Such a knowledge is needed to craft
the malicious examples [52]. More recently, clean-label poisoning attacks for
unknown, i.e., _black-box_ [53], deep image classifiers have been also
explored [54].

In dirty-label poisoning the adversary can introduce a number of training
instances to be misclassified with a specific label in a targeted way. An
example of dirty-label poisoning is the label-flipping attack [40, 55]. Here,
the labels of honest training examples of one class are flipped to another
class, while the features of the data are kept unchanged. For instance, a
malicious agent can poison the training set of a handwritten digit recognition
system by flipping all 1s to 7s.

In the FL setting, it is common to assume that the attacker controls some
clients and their training sets. Thus, dirty-label attacks have been more
often considered. In [26], the authors study targeted label flipping attacks
on FL. They find that poisons injected late in the training process are
significantly more effective than those injected early. Other proposals adopt
a bi-level optimization approach for poisoning multi-task FL [56] and GAN-
generated poisons [57].

Model Poisoning. Differently from data poisoning attacks, these attacks
threaten the learning process directly, e.g., by changing some model
parameters. Model poisoning is generally perceived as difficult to implement
in centralized ML systems as it requires the adversary to access the target
model, i.e., assuming either _grey-box_ or _white-box_ knowledge. On the other
hand, model poisoning becomes rather feasible in the case of FL, where a
malicious client has direct influence over the jointly-trained global model
via its local parameters updates [58, 49, 50]. As with any poisoning attack,
the adversary’s goal is to cause wrong predictions of the FL model. However,
she/he aims to force classification errors at inference time and without
modifying test examples. In this respect, it is opposed to backdoor and
evasion attacks (which are discussed below). In model poisoning, the
misclassification results from the adversarial corruption of the training
process, which can be achieved either by _gradient_ or learning rule
manipulation. In gradient manipulation, the adversary poisons local model
gradients (or model updates), which are then sent to the central server for
aggregation, thereby jeopardizing the global model performance [59]. In
learning rule manipulation, instead, the attacker corrupts the actual training
logic. In some cases, these model poisoning attacks have proven more effective
than data poisoning, and an attacker can compromise the global model even when
controlling a single client. For instance, in [49] the authors successfully
achieve a stealthy targeted model poisoning attack by adding a penalty term to
the objective function to minimize the distance between malicious and benign
weight update distributions.

#### Backdoor Attacks

Backdoor attacks – also known as trojan attacks – exploit the adversary’s
capability of having (limited) access to input examples also at test time
[27]. Hence, backdoor attacks exceed poisoning attacks, since the adversary
can manipulate both training and test inputs. Therefore, backdoor attacks are
often considered more disruptive toward the victim model.

Like for standard poisoning attacks, we can distinguish between backdoor
attacks affecting the data or the model. The former are referred to as
backdoor data poisoning and consist of adding attacker-chosen examples to the
training set. The attacker examples contain a particular trigger [60, 61],
i.e., a distinguished feature that activates the backdoor. The model learned
on such poisoned training set will embed a backdoor, which the attacker
exploits at test time by submitting examples that contain the same trigger.
Instead, backdoor model poisoning requires a stronger threat model, where the
attacker can get direct access to the learning system and change the model’s
internals (i.e., parameters and architecture) to embed a backdoor [62, 63].

Interestingly enough, backdoor attacks have been proven ineffective under FL
settings [64]. The main obstacle is that aggregation involves many clients
and, assuming that the attacker only controls a minority, the effect of the
adversarial updates is weakened. To overcome this limitation, the authors of
[64] consider a model replacement approach, where the attacker scales up a
malicious model update to increase its effect on the aggregation function.

In [65], the authors propose distributed backdoor attacks, which better
exploit the decentralized nature of FL. Specifically, they decompose the
backdoor pattern for the global model into multiple distributed small
patterns, and inject them into training sets used by up to 40% adversarial
participants at each federated round. Although more effective than global
backdoor trigger injection, this approach comes at the price of controlling a
significant subset of the FL clients.

#### Evasion Attacks

Adversarial attacks that occur only at test time are called evasion attacks.
Here, the goal of the adversary is still to fool an ML model yet without
tampering it at training time. In fact, evasion attacks make use of so-called
adversarial examples [38, 66], i.e., carefully-crafted (minimal) perturbations
of test instances that cause prediction errors (either targeted or untargeted)
when input to a legitimately trained model.

Evasion attacks may look similar to backdoor poisoning attacks [67]. However,
the key difference between the two is that evasion attacks exploit the
decision boundaries learned by an uncorrupted model to construct adversarial
examples that are misclassified by the model. In contrast, backdoor attacks
intentionally shift these decision boundaries as a result of a jeopardized
training process, so that certain examples get eventually misclassified [27].

Several works have explored evasion attacks in the context of computer vision
[68], where adversarial examples are obtained by adding random noise to test
images. Even though such images look legitimate to a human, they are wrongly
classified by the image recognition system. Also, more recent works
investigate the applicability of evasion attacks to malware classification
[69].

In the FL setting, the global model maintained by the server suffers from the
same evasion attacks as in the conventional ML setting when the target model
is deployed as a service. Moreover, at each FL training round the global model
sent to the federated clients is exposed as a white-box to any malicious
participant. Thus, FL requires extra efforts to defend against white-box
evasion attacks [24].

In this work, we consider an attack scenario partially related to the one
presented in [64]. As a matter of fact, our covert channel is implemented by
means of maliciously crafted examples that carry a trigger as in backdoor
attacks. In particular, our adversary $(i)$ crafts malicious examples carrying
a trigger (see Section V-A), and $(ii)$ poisons the federated model to
transmit one bit through each trigger (see Section V-B).

###  III-B Application level covert channels

In a general sense, a covert channel is any communication channel that is not
intended for information transfer [11]. Although we are not aware of FL-based
covert channels, some authors already investigated the implementation of
covert channels at application level. Most authors considered _encapsulation_
of hidden communications in application-layer network protocols. Being the
main application level protocol, HTTP is the primary target for covert channel
implementations, e.g., see [70]. Nevertheless, the entire TCP/IP ecosystem can
be at risk, and we refer the interest reader to [71] for a survey. More
recently, also web applications were proposed for the implementation of covert
channels. For instance, in [72] the author considers social networks such as
Facebook and Twitter. However, since this kind of covert channels rely on
already existing communications between devices, in practice, they usually do
not break any sandbox policy.222On the contrary, they are very relevant, for
instance, when considering inter-process channels as in [73]. Also, as the
authors of [74] point out, most of these covert channels can be detected and
some effective countermeasures exist, e.g., packet inspection can be used to
detect illegal traffic. Possibly for these reasons, application level covert
channels appear to be less frequently considered in the literature.

Loosely speaking, also our proposal relies on a sort of encapsulation
mechanism. However, here we do not wrap information inside protocol messages.
Rather, we embed information inside FL models, which prevents standard
detection techniques based on traffic inspection.

##  IV Attacker Model

In this section, we present our attacker model. The goal of the attacker is to
establish a covert channel between two clients, namely _Sender_ and _Receiver_
, of an FL infrastructure. In terms of capabilities, our adversary resembles
that of [64]. Here, we assume that both Sender and Receiver are controlled by
the attacker. For instance, think of Sender as a malware-compromised device
and Receiver as the malware command and control server. Although they are
compromised, we do not assume that the attacker can tamper with the standard
FL infrastructure behavior, i.e., Sender and Receiver follow the FL client
protocol. More precisely, Sender is only allowed to poison its local dataset
and Receiver can only classify examples using its own local model. In
particular, we stress that here we assume the attacker to operate in _black-
box_ mode, i.e., we assume that Sender and Receiver can neither inspect nor
tamper with their local models. It is worth noticing that these assumptions
are very general and they apply to most FL systems.

$m=\phi(\\{m_{s},m_{1},m_{2}\\})$SenderClient 1Client 2FL Servercovert
channelReceiverClient 3Client 4Attacker$m_{s}$$m_{1}$$m_{2}$$m$$m$$m$ crafted
example crafted
example $b$$b$ Figure 2: Overview of the attacker model.

The overall attacker model is schematically depicted in Figure 2. At each
federated round, the FL server randomly selects a subset of clients. Selected
clients work as expected, i.e., they use their own private datasets to train
their local models starting from the last global model received by the server
(see Client 1 and Client 2 in Figure 2). Then, FL clients upload their newly
trained local models to the server, which aggregates them into an updated
global model $m$ through the aggregation function $\phi$. At the end of each
federated round, _all_ the clients receive a copy of $m$. When selected,
Sender (top left of Figure 2) poisons its local model $m_{s}$ by training it
with some malicious, attacker-provided examples. Its goal is to transmit a bit
$b$ by inducing a perturbation of the global model that Receiver can test. On
the other hand, Receiver (top right of Figure 2) uses $m$ to classify test
examples, e.g., the same malicious examples used by Sender. According to the
classification outcome, Receiver deduces whether 0 or 1 was sent. Implementing
such a covert channel is non-trivial and also depends on the underlying FL
system. We discuss the implementation details in the next section.

##  V Covert channel implementation

ReceiverFL
ServerSender$m^{(1)}$$\vdots$$m^{(f)}$$(f,\widetilde{\boldsymbol{x}},h,l)$$m^{(f+1)}$$\vdots$$m^{(2f)}$selected$m_{b}$write
$b$read $b$calibrationtransmission Figure 3: Communication protocol phases.

In this section, we detail the implementation strategy for creating the
previously described covert channel. Without loss of generality, every
implementation is based on the abstract protocol schematically depicted in
Figure 3.

A transmission starts with a _calibration_ phase during which Receiver
observes the global model updates $m^{(1)},\ldots,m^{(f)}$ (for $f$ FL
rounds). Eventually, Receiver computes _channel parameters_ (see Section V-A)
to be shared with Sender. For instance, these parameters can be provided
through a secondary channel or hard-coded in Sender before its deployment.
Then, Sender and Receiver synchronize on transmission frames of size $f$ to
send a bit $b$. During each frame, when selected, Sender trains its local
model according to the channel parameters and the bit to be transmitted. In
the meanwhile, Receiver monitors the global model updates and, at the end of
the frame, it tests the received bit.

Below, we discuss the implementation of both the calibration and transmission
phases.

###  V-A Calibration of channel parameters

The first parameter to be determined is the size of the transmission frame
$f$, i.e., the number of FL rounds necessary to transmit a single bit.
Assuming that the attacker ignores the internals of the FL server, i.e.,
current number of clients ($n_{c}$) and client selection probability
($p_{c}$), Receiver must estimate $f$. Clearly, a small $f$ decreases the
probability for Sender to be selected (which is necessary for the transmission
to take place), while large $f$ reduces the transmission rate. Under the
assumption that clients are uniformly selected at random, Receiver takes
advantage of its selection notifications to estimate $f$. For instance, it can
set $f$ as the number of rounds that it takes to be selected $T$ times (for a
constant $T$).

Another channel parameter to be determined is the crafted example
$\widetilde{\boldsymbol{x}}$ carrying the trigger that Sender will use to
poison its local training set. To generate $\widetilde{\boldsymbol{x}}$,
Receiver applies a _linear transformation function_ 333Although we do not
explicitly prove it, the reader can easily check that all the transformation
functions presented in the following are linear, since they reduce to finite
sums of matrices. $\psi$ to a subset of $k$ randomly selected examples from
its local training set. Intuitively,
$\widetilde{\boldsymbol{x}}=\psi(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\alpha)$
means that, starting from $k>0$ examples
$\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k}$, $\psi$ returns a new example
$\widetilde{\boldsymbol{x}}$, where $\alpha\in[0,1]$ is a parameter that
controls the transformation.

To provide an intuition of this process, we put forward an example taken from
the MNIST dataset. Consider $\boldsymbol{x}$ to be the representation of a
generic MNIST input image, i.e., a $28\\!\times\\!28$ matrix of pixels flatten
into a $784$-dimensional vector. We define $\psi_{e}(\boldsymbol{x},\alpha)$
as the function erasing, from left to right, a fraction $\alpha$ of a single
example image $\boldsymbol{x}$ (i.e., $k=1$). For instance, when $\alpha=0.3$,
$\psi_{e}$ sets to 0 the $235$ values associated with the leftmost pixels of
the target image vector. More formally,
$\psi_{e}(\boldsymbol{x},\alpha)=\boldsymbol{x}-\boldsymbol{x}^{(\alpha)}$
where $\boldsymbol{x}^{(\alpha)}=[x^{1},\ldots,x^{\lfloor\alpha\cdot
784\rfloor},0,\ldots,0]$ and $x^{i}$ is the $i$-th element of
$\boldsymbol{x}$. The behavior of $\psi_{e}$ is shown in Figure 4, where we
highlighted the erased portion of the original example.

|  |
---|---|---
Figure 4: Linear transformation $\psi_{e}$ of an example with $\alpha=0.3$
(middle) and $\alpha=0.5$ (rightmost).

Intuitively, the attacker can select a function $\psi$ which is based on some
_semantic_ property of the classification domain. For instance, in the
previous example $\psi_{e}$ encodes the simple fact that the right half of an
handwritten 8 looks like a 3. Other similar functions can be found, e.g., in
[75].

Receiver repeatedly applies $\psi$ to the selected examples until it
identifies an _edge_ example $\widetilde{\boldsymbol{x}}$. Formally, given
Receiver’s local model $m_{r}$ and an arbitrarily small $\varepsilon>0$, we
look for
$\widetilde{\boldsymbol{x}}=\psi(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\bar{\alpha})$
such that
$m_{r}(\psi(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\bar{\alpha}))=h$ and
$m_{r}(\psi(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\bar{\alpha}+\varepsilon))=l$,
where $h,l\in\mathcal{Y}$ and $l\neq h$. Interestingly, given
$\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k}$, Receiver can efficiently
compute $\bar{\alpha}$ through the binary search procedure sketched in
Algorithm 1.

Input: $\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k}$; $\varepsilon>0$

$\alpha_{\textnormal{high}}$ := H;

$\alpha_{\textnormal{low}}$ := L;

repeat

$\boldsymbol{x}_{\textnormal{high}}$ :=
$\psi$($\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\alpha_{\textnormal{high}}$);

$\hat{y}_{\textnormal{high}}$ := $m_{r}(\boldsymbol{x}_{\textnormal{high}})$;

$\boldsymbol{x}_{\textnormal{low}}$ :=
$\psi$($\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\alpha_{\textnormal{low}}$);

$\hat{y}_{\textnormal{low}}$ := $m_{r}(\boldsymbol{x}_{\textnormal{low}})$;

if  _$\hat{y}_{\textnormal{high}}=\hat{y}_{\textnormal{low}}$_ then

Failure: cannot be equal

end if

$\alpha_{\textnormal{mid}}$ :=
$(\alpha_{\textnormal{low}}+\alpha_{\textnormal{high}})/2$;

$\boldsymbol{x}_{\textnormal{mid}}$ :=
$\psi$($\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{k},\alpha_{\textnormal{mid}}$);

$\hat{y}_{\textnormal{mid}}$ := $m_{r}(\boldsymbol{x}_{\textnormal{mid}})$;

if  _$\hat{y}_{\textnormal{high}}\neq\hat{y}_{\textnormal{mid}}$_ then

$\alpha_{\textnormal{low}}$ := $\alpha_{\textnormal{mid}}$;

else

$\alpha_{\textnormal{high}}$ := $\alpha_{\textnormal{mid}}$;

end if

until  _$\alpha_{\textnormal{low}}-\alpha_{\textnormal{high}} <\varepsilon$_;

Output: $\widetilde{\boldsymbol{x}}:=\boldsymbol{x}_{\textnormal{high}}$,
labels $h:=\hat{y}_{\textnormal{high}}$ and $l:=\hat{y}_{\textnormal{low}}$

Algorithm 1 Edge example binary search algorithm.

The search procedure starts from the predefined interval [H, L].444In our
experiments we use H = $0$ and L = $1/2$. At each iteration, two examples,
i.e., $\boldsymbol{x}_{\textnormal{high}}$ and
$\boldsymbol{x}_{\textnormal{low}}$, are generated and classified with
Receiver’s model $m_{r}$, so obtaining $\hat{y}_{\textnormal{high}}$ and
$\hat{y}_{\textnormal{low}}$, respectively. If
$\hat{y}_{\textnormal{high}}=\hat{y}_{\textnormal{low}}$, the algorithm
terminates with a failure and no edge example is returned. Otherwise, the
current search interval is split in half by computing
$\alpha_{\textnormal{mid}}$, $\boldsymbol{x}_{\textnormal{mid}}$, and
$\hat{y}_{\textnormal{mid}}$. Then, if
$\hat{y}_{\textnormal{high}}\neq\hat{y}_{\textnormal{mid}}$, the algorithm
iterates on the first half of the current interval. Otherwise, if
$\hat{y}_{\textnormal{low}}\neq\hat{y}_{\textnormal{mid}}$, the search
procedure continues on the second half. The loop terminates when the interval
width goes under a threshold $\varepsilon$, which represents the granularity
of a single position in the feature vectors of the example, e.g., a single
pixel in the case of Figure 4. Eventually, the algorithm returns the edge
example $\boldsymbol{x}_{\textnormal{high}}$, as well as the two class labels
$h$ and $l$ associated with the last, smallest interval.

It is worth noticing that, since it does not explore the entire feature space,
Algorithm 1 might fail to generate an edge examples on some inputs. In
general, the effectiveness of this heuristics depends on the choice of $\psi$.

At the end of the calibration phase, channel parameters generated by Receiver
amount to the tuple $(f,\boldsymbol{\widetilde{x}},h,l)$.

###  V-B Bit transmission

The transmission of one bit is based on the variations, during $f$ FL rounds,
of $m(\widetilde{\boldsymbol{x}})$ between $h$ and $l$. In particular, by
poisoning its local model, Sender drives $m(\widetilde{\boldsymbol{x}})$ to
assume the desired value, while Receiver monitors it to read the transmitted
bit. Transmissions are organized in consecutive frames of size $f$. Sender and
Receiver are synchronized through an FL round counter $r\in\\{1,\ldots,f\\}$.
Sender follows the procedure given in Algorithm 2.

Input: $f$, $\boldsymbol{\widetilde{x}}$, $h$, $l$

repeat

if  _$r=1$_ then

$b$ := nextBit();

$v$ := $m(\boldsymbol{\widetilde{x}})$;

$v_{\lnot}$ := if  _$v=h$_ then $l$ else $h$;

end if

if  _selected by server for training_ then

$v_{r}$ := $m(\boldsymbol{\widetilde{x}})$;

switch  _b_ do

case  _0_ do

if  _$v_{r}\neq v$ _ then train($m_{s}$, $\boldsymbol{\widetilde{x}}$, $v$);

end case

case  _1_ do

if  _$v_{r}\neq v_{\lnot}$ _ then train($m_{s}$, $\boldsymbol{\widetilde{x}}$,
$v_{\lnot}$);

end case

end switch

upload($m_{s}$);

end if

$r$ := $(r\%f)+1$ ;

until  _transmission completed_ ;

Algorithm 2 Sender transmission algorithm.

At the beginning of each transmission frame, i.e., when $r=1$, Sender sets the
next bit to be sent $b$, and uses the last received global model $m_{r}=m$ to
classify $\boldsymbol{\widetilde{x}}$, thus obtaining $v\in\\{h,l\\}$. Also,
Sender sets $v_{\lnot}\in\\{h,l\\}$ so that $v\neq v_{\lnot}$. At each round
$r$, if Sender is selected by the FL server, it classifies
$\boldsymbol{\widetilde{x}}$ with the global model $m$, so obtaining $v_{r}$.
Then, depending on the bit to be sent, Sender trains its local model $m_{s}$
according to Table I.

TABLE I: Sender’s model poisoning cases. $\boldsymbol{b=0}$ | $\boldsymbol{b=1}$
---|---
$\boldsymbol{v_{r}=v}$ | $\boldsymbol{v_{r}=v_{\lnot}}$ | $\boldsymbol{v_{r}=v}$ | $\boldsymbol{v_{r}=v_{\lnot}}$
do nothing |  train($m_{s},\widetilde{\boldsymbol{x}},v$) |  train($m_{s},\widetilde{\boldsymbol{x}},v_{\lnot}$) | do nothing

Intuitively, the purpose of the operation above is to keep the label assigned
to $\widetilde{\boldsymbol{x}}$ by the global model when sending 0, and to
flip it when sending 1. This channel implementation amounts to a Differential
Manchester encoding [17]. Eventually, Sender uploads its local model to the FL
server.

Input: $f$, $\boldsymbol{\widetilde{x}}$, $h$, $l$

repeat

if  _$r=1$_ then

$v_{1}$ := $m(\boldsymbol{\widetilde{x}})$;

else if  _$r=f$_ then

$v_{f}$ := $m(\boldsymbol{\widetilde{x}})$ ;

$b$ := if  _$v_{1}=v_{f}$_ then 0 else 1;

received($b$);

end if

$r$ := $(r\%f)+1$ ;

until  _transmission completed_ ;

Algorithm 3 Receiver bit test algorithm.

Concurrently, Receiver executes Algorithm 3. Receiver uses the global model
$m$ to classify $\boldsymbol{\widetilde{x}}$ both when $r=1$ and $r=f$, so
obtaining $v_{1}$ and $v_{f}$, respectively. Finally, Receiver reads 0 if
$v_{1}=v_{f}$ and 1 otherwise.

$-10$$0$$10$$20$$30$Score$3$$8$Label12345678910$0$$1$Time (Frame)Bit83 Figure
5: Transmission of bits 0101001110 over a covert channel.

In Figure 5, we show as an example the transmission of 10 bits for the MNIST
scenario discussed above, using $\widetilde{\boldsymbol{x}}$ of Figure 4,
$h=8$, and $l=3$. The top diagram shows the internal scores assigned by the
global model to the edge example $\boldsymbol{\widetilde{x}}$ during the
transmission. By classifying $\boldsymbol{\widetilde{x}}$, Receiver observes
the alternation of labels 8 and 3 (center). Then, Receiver interprets it as
the sequence of bits 0101001110.

##  VI Channel description

For each edge example, our implementation provides a digital, broadcast
channel supporting _half duplex_ 555Intuitively, Sender and Receiver cannot
transmit at the same time, but they can alternate their roles. transmissions.
In principle, an attacker can even create several, parallel channels of this
type to enlarge the communication bandwidth. However, parallel channels may
generate interference. Below, we detail the features of a single channel.

In terms of capacity, we treat the covert channel as a standard BMC (see,
Section II-B). We just notice that the binary input of the channel is $m_{b}$,
with $b\in\\{0,1\\}$. Intuitively, inputs $m_{b}$ represent Sender’s local
models $m_{s}$ uploaded to the FL server. More precisely, we use $m_{b}$ to
distinguish between the two model poisoning cases used to transmit $b$ (see
Table I).

In terms of communication quality, we consider BER and SNR, as discussed in
Section II-B. While BER is straightforward, defining SNR in our context
requires more attention. In general, SNR is computed by periodically sampling
the channel signal $s$ and noise $n$ at the end of each transmission frame
$t$. Here, defining $s$ and $n$ is non-trivial, since our channel does not
rely on a physical medium. Indeed, a covert channel consists of two prediction
labels, namely $h$ and $l$, and a perturbed example
$\boldsymbol{\widetilde{x}}$. For each label $i\in\\{h,l\\}$, at time $t$ we
can measure the prediction score assigned by the global model to $i$ when
classifying $\boldsymbol{\widetilde{x}}$ at each frame’s end. Also, assuming
that prediction scores range within the interval $[Z,-Z]$,666$Z$ can be
esteemed as the maximum score observed during a transmission. we normalize the
prediction scores by linearly scaling them in $[1,-1]$. Thus, for each label
$i$ the label signal $z_{i}(t)$ amounts to the normalized score described
above. We define the overall received signal $z(t)=z_{h}(t)-z_{l}(t)$, i.e.,
as the differential, normalized signal.

Since a direct measure of the transmitted signal $s(t)$ cannot be computed, we
approximate it to the differential signal that switches its intensity between
$1$ and $-1$. Intuitively, this is equivalent to stating that Sender attempts
to transmit 0 by setting $z(t)\cdot z(t+1)=1$ and 1 by setting $z(t)\cdot
z(t+1)=-1$.777Notice that this assumption is more restrictive w.r.t. the
actual implementation, since Receiver only requires $z(t)\cdot z(t+1)<0$ and
$z(t)\cdot z(t+1)>0$ to read 0 and 1, respectively. Thus, for each frame $t$,
we set

$\displaystyle s(t)$ $\displaystyle=\>\frac{1-2b}{s(t-1)},$ (4)

where $b$ is the bit transmitted during frame $t$, and, by construction,
$s(0)=1$. Then, we define the noise at time $t$ as $n(t)=z(t)-s(t)$.

The overall intuition behind our definitions of $z(t)$, $s(t)$ and $n(t)$ is
given in Figure 6. There, red and blue dashed lines denote $z_{h}$ and $z_{l}$
signals, i.e., the normalized version of the scores of Figure 5. Instead, the
purple line denotes $z(t)$ and the gray line denotes $s(t)$. Also, the
vertical arrow shows the value of $n(t)$ at $t=3$. From this and (4), we
define SNR of the covert channel as

$SNR=\frac{z(t)^{2}}{\sigma_{n}^{2}},$ (5)

where $\sigma_{n}$ is the standard deviation of the _normalized_ noise
$\bar{n}(t)$ defined as
$\bar{n}(t)=\tfrac{n(t)}{\max_{t^{\prime}}|n(t^{\prime})|}$.

$-10$$0$$10$$20$$30$12345678910$-1$$0$$1$Time (FL
rounds)Intensity$n(3)$83$z_{h}(t)$$z_{l}(t)$$z(t)$$s(t)$ Figure 6: Signals and
noise for the FL covert channel of Figure 5.

##  VII Conclusion

In this paper, we introduced a new covert channel leveraging on FL systems.
Our attack allows a malicious agent to establish stealth communications
between FL clients that should rather stay isolated. This attack scenario
paves the way for several exploits that we plan to investigate in the future.
Among them, we are particularly interested in attacks breaking the air gap.
These attacks appear realistic since many FL clients feed the training process
with sensor-generated examples. Other future directions include $(i)$
measuring the performance of the covert channel, $(ii)$ evaluating whether the
channel can be detected, and $(iii)$ investigating possible countermeasures.

## References

  * [1] B. McMahan and D. Ramage, “Federated Learning: Collaborative Machine Learning without Centralized Training Data,” https://ai.googleblog.com/2017/04/federated-learning-collaborative.html, 2017, accessed: 2017-04-06.
  * [2] B. McMahan, E. Moore, D. Ramage, S. Hampson, and A. y. B. Arcas, “Communication-efficient learning of deep networks from decentralized data,” _AISTATS_ , pp. 1273–1282, 2017.
  * [3] Q. Wu, K. He, and X. Chen, “Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework,” _IEEE Open Journal of the Computer Society_ , vol. 1, pp. 35–44, 2020. [Online]. Available: https://doi.acm.org/10.1109/OJCS.2020.2993259
  * [4] C. Zhou, A. Fu, S. Yu, W. Yang, H. Wang, and Y. Zhang, “Privacy-Preserving Federated Learning in Fog Computing,” _IEEE Internet of Things Journal_ , vol. 7, no. 11, pp. 10 782–10 793, 2020. [Online]. Available: https://doi.acm.org/10.1109/JIOT.2020.2987958
  * [5] S. R. Pokhrel and J. Choi, “A Decentralized Federated Learning Approach for Connected Autonomous Vehicles,” in _2020 IEEE Wireless Communications and Networking Conference Workshops (WCNCW)_ , 2020, pp. 1–6. [Online]. Available: https://doi.acm.org/10.1109/WCNCW48565.2020.9124733
  * [6] T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong, D. Ramage, and F. Beaufays, “Applied Federated Learning: Improving Google Keyboard Query Suggestions,” _CoRR_ , vol. abs/1812.02903, 2018. [Online]. Available: http://arxiv.org/abs/1812.02903
  * [7] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao, “FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare,” _IEEE Intelligent Systems_ , vol. 35, no. 4, pp. 83–93, 2020. [Online]. Available: https://doi.acm.org/10.1109/MIS.2020.2988604
  * [8] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li, “Manipulating machine learning: Poisoning attacks and countermeasures for regression learning,” in _2018 IEEE Symposium on Security and Privacy (SP)_. IEEE Computer Society, 2018, pp. 19–35. [Online]. Available: https://doi.org/10.1109/SP.2018.00057
  * [9] A. N. Bhagoji, S. Chakraborty, P. Mittal, and S. Calo, “Analyzing Federated Learning through an Adversarial Lens,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, 09–15 Jun 2019, pp. 634–643.
  * [10] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting Unintended Feature Leakage in Collaborative Learning,” in _2019 IEEE Symposium on Security and Privacy (SP)_ , 2019, pp. 691–706. [Online]. Available: https://doi.acm.org/10.1109/SP.2019.00029
  * [11] B. W. Lampson, “A note on the confinement problem,” _Communications of the ACM_ , vol. 16, no. 10, pp. 613–615, Oct. 1973. [Online]. Available: https://doi.acm.org/10.1145/362375.362389
  * [12] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [13] V. Vapnik, “Principles of risk minimization for learning theory,” in _Proceedings of the 4th International Conference on Neural Information Processing Systems_ , ser. NIPS’91. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1991, p. 831–838.
  * [14] K. P. Murphy, _Machine Learning: A Probabilistic Perspective_. The MIT Press, 2012, ISBN: 0262018020.
  * [15] T. Hastie, R. Tibshirani, and J. Friedman, _The Elements of Statistical Learning_ , ser. Springer Series in Statistics. New York, NY, USA: Springer New York Inc., 2001, ISBN: 0387848576\.
  * [16] Y. Lu and L. Fan, “An efficient and robust aggregation algorithm for learning federated cnn,” in _Proceedings of the 2020 3rd International Conference on Signal Processing and Machine Learning_ , ser. SPML 2020. New York, NY, USA: Association for Computing Machinery, 2020, pp. 1–7. [Online]. Available: https://doi.org/10.1145/3432291.3432303
  * [17] P. Massoud Salehi and J. Proakis, _Digital Communications_. McGraw-Hill Education, 2007, ISBN: 9780072957167.
  * [18] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, “Advances and Open Problems in Federated Learning,” _Foundations and Trends in Machine Learning_ , vol. 14, no. 1, 2021. [Online]. Available: http://dx.doi.org/10.1561/2200000083
  * [19] N. N. Dalvi, P. M. Domingos, Mausam, S. K. Sanghai, and D. Verma, “Adversarial classification,” in _SIGKDD Conference on Knowledge Discovery and Data Mining_. ACM, 2004, pp. 99–108. [Online]. Available: https://doi.org/10.1145/1014052.1014066
  * [20] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar, “Adversarial machine learning,” in _Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence_ , ser. AISec ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 43–58. [Online]. Available: https://doi.org/10.1145/2046684.2046692
  * [21] B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of adversarial machine learning,” _Pattern Recognition_ , vol. 84, pp. 317 – 331, 2018\. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320318302565
  * [22] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in _2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019_. IEEE, 2019, pp. 739–753. [Online]. Available: https://doi.org/10.1109/SP.2019.00065
  * [23] N. Pitropakis, E. Panaousis, T. Giannetsos, E. Anastasiadis, and G. Loukas, “A taxonomy and survey of attacks against machine learning,” _Computer Science Review_ , vol. 34, p. 100199, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013718303289
  * [24] L. Lyu, H. Yu, and Q. Yang, “Threats to Federated Learning: A Survey,” _arXiv e-prints_ , Mar. 2020. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2020arXiv200302133L
  * [25] M. S. Jere, T. Farnan, and F. Koushanfar, “A taxonomy of attacks on federated learning,” _IEEE Security Privacy_ , pp. 0–0, 2020. [Online]. Available: https://doi.acm.org/10.1109/MSEC.2020.3039941
  * [26] V. Tolpegin, S. Truex, M. E. Gursoy, and L. Liu, “Data poisoning attacks against federated learning systems,” 2020. [Online]. Available: https://arxiv.org/abs/2007.08432
  * [27] M. Goldblum, D. Tsipras, C. Xie, X. Chen, A. Schwarzschild, D. Song, A. Madry, B. Li, and T. Goldstein, “Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses,” _arXiv e-prints_ , p. arXiv:2012.10544, Dec. 2020.
  * [28] M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, “Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing,” in _23rd USENIX Security Symposium (USENIX Security 14)_. San Diego, CA: USENIX Association, Aug. 2014, pp. 17–32. [Online]. Available: https://www.usenix.org/conference/usenixsecurity14/technical-sessions/presentation/fredrikson_matthew
  * [29] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 1322–1333. [Online]. Available: https://doi.org/10.1145/2810103.2813677
  * [30] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _2017 IEEE Symposium on Security and Privacy (SP)_ , 2017, pp. 3–18.
  * [31] A. Pyrgelis, C. Troncoso, and E. D. Cristofaro, “Knock knock, who’s there? membership inference on aggregate location data,” in _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA_. The Internet Society, 2018.
  * [32] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _2019 IEEE Symposium on Security and Privacy (SP)_ , 2019, pp. 691–706. [Online]. Available: https://doi.acm.org/10.1109/SP.2019.00029
  * [33] G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici, “Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers,” _International Journal of Security and Networks_ , vol. 10, no. 3, p. 137–150, Sep. 2015. [Online]. Available: https://doi.org/10.1504/IJSN.2015.071829
  * [34] K. Ganju, Q. Wang, W. Yang, C. A. Gunter, and N. Borisov, “Property inference attacks on fully connected neural networks using permutation invariant representations,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 619–633. [Online]. Available: https://doi.org/10.1145/3243734.3243834
  * [35] B. Liang, M. Su, W. You, W. Shi, and G. Yang, “Cracking classifiers for evasion: A case study on the google’s phishing pages filter,” in _Proceedings of the 25th International Conference on World Wide Web_ , ser. WWW ’16. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2016, p. 345–356. [Online]. Available: https://doi.org/10.1145/2872427.2883060
  * [36] F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis,” in _Proceedings of the 25th USENIX Conference on Security Symposium_ , ser. SEC’16. USA: USENIX Association, 2016, p. 601–618.
  * [37] B. Wang and N. Z. Gong, “Stealing hyperparameters in machine learning,” in _2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings, 21-23 May 2018, San Francisco, California, USA_. IEEE Computer Society, 2018, pp. 36–52. [Online]. Available: https://doi.org/10.1109/SP.2018.00038
  * [38] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar, “Can machine learning be secure?” in _Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security_ , ser. ASIACCS ’06. New York, NY, USA: Association for Computing Machinery, 2006, p. 16–25. [Online]. Available: https://doi.org/10.1145/1128817.1128824
  * [39] B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h. Lau, S. Rao, N. Taft, and J. D. Tygar, “Antidote: Understanding and defending against poisoning of anomaly detectors,” in _Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement_ , ser. IMC ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 1–14. [Online]. Available: https://doi.org/10.1145/1644893.1644895
  * [40] B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” in _Proceedings of the 29th International Coference on International Conference on Machine Learning_ , ser. ICML’12. Madison, WI, USA: Omnipress, 2012, p. 1467–1474.
  * [41] H. Xiao, B. Biggio, G. Brown, G. Fumera, C. Eckert, and F. Roli, “Is feature selection secure against training data poisoning?” in _Proceedings of the 32nd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 1689–1698. [Online]. Available: http://proceedings.mlr.press/v37/xiao15.html
  * [42] B. Li, Y. Wang, A. Singh, and Y. Vorobeychik, “Data poisoning attacks on factorization-based collaborative filtering,” in _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 1893–1901.
  * [43] G. Yang, N. Z. Gong, and Y. Cai, “Fake co-visitation injection attacks to recommender systems,” in _24th Annual Network and Distributed System Security Symposium, NDSS 2017, San Diego, California, USA, February 26 - March 1, 2017_. The Internet Society, 2017\. [Online]. Available: https://www.ndss-symposium.org/ndss2017/ndss-2017-programme/fake-co-visitation-injection-attacks-recommender-systems/
  * [44] B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. P. Rubinstein, U. Saini, C. Sutton, J. D. Tygar, and K. Xia, “Exploiting machine learning to subvert your spam filter,” in _Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats_ , ser. LEET’08. USA: USENIX Association, 2008.
  * [45] A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, and T. Goldstein, “Poison frogs! targeted clean-label poisoning attacks on neural networks,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 6106–6116.
  * [46] T. Gu, K. Liu, B. Dolan-Gavitt, and S. Garg, “Badnets: Evaluating backdooring attacks on deep neural networks,” _IEEE Access_ , vol. 7, pp. 47 230–47 244, 2019. [Online]. Available: https://doi.org/10.1109/ACCESS.2019.2909068
  * [47] H. Aghakhani, T. Eisenhofer, L. Schönherr, D. Kolossa, T. Holz, C. Kruegel, and G. Vigna, “VENOMAVE: clean-label poisoning against speech recognition,” _CoRR_ , vol. abs/2010.10682, 2020. [Online]. Available: https://arxiv.org/abs/2010.10682
  * [48] M. Fang, G. Yang, N. Z. Gong, and J. Liu, “Poisoning attacks to graph-based recommender systems,” in _Proceedings of the 34th Annual Computer Security Applications Conference_ , ser. ACSAC ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 381–392. [Online]. Available: https://doi.org/10.1145/3274694.3274706
  * [49] A. N. Bhagoji, S. Chakraborty, P. Mittal, and S. Calo, “Analyzing federated learning through an adversarial lens,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 634–643. [Online]. Available: http://proceedings.mlr.press/v97/bhagoji19a.html
  * [50] M. Fang, X. Cao, J. Jia, and N. Z. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in _29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020_ , S. Capkun and F. Roesner, Eds. USENIX Association, 2020, pp. 1605–1622. [Online]. Available: https://www.usenix.org/conference/usenixsecurity20/presentation/fang
  * [51] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” _CoRR_ , vol. abs/1708.06733, 2017. [Online]. Available: http://arxiv.org/abs/1708.06733
  * [52] O. Suciu, R. Mărginean, Y. Kaya, H. Daumé, and T. Dumitraş, “When does machine learning fail? generalized transferability for evasion and poisoning attacks,” in _Proceedings of the 27th USENIX Conference on Security Symposium_ , ser. SEC’18. USA: USENIX Association, 2018, p. 1299–1316.
  * [53] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security_ , ser. ASIA CCS ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 506–519. [Online]. Available: https://doi.org/10.1145/3052973.3053009
  * [54] C. Zhu, W. R. Huang, H. Li, G. Taylor, C. Studer, and T. Goldstein, “Transferable clean-label poisoning attacks on deep neural nets,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 7614–7623. [Online]. Available: http://proceedings.mlr.press/v97/zhu19a.html
  * [55] C. Fung, C. J. M. Yoon, and I. Beschastnikh, “Mitigating Sybils in Federated Learning Poisoning,” _arXiv e-prints_ , Aug. 2018. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2018arXiv180804866F
  * [56] G. Sun, Y. Cong, J. Dong, Q. Wang, and J. Liu, “Data poisoning attacks on federated machine learning,” _ArXiv_ , vol. abs/2004.10020, 2020. [Online]. Available: https://arxiv.org/abs/2004.10020
  * [57] J. Zhang, J. Chen, D. Wu, B. Chen, and S. Yu, “Poisoning attack in federated learning using generative adversarial nets,” in _18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications / 13th IEEE International Conference On Big Data Science And Engineering, TrustCom/BigDataSE 2019, Rotorua, New Zealand, August 5-8, 2019_. IEEE, 2019, pp. 374–380. [Online]. Available: https://doi.org/10.1109/TrustCom/BigDataSE.2019.00057
  * [58] S. Mahloujifar, M. Mahmoody, and A. Mohammed, “Universal multi-party poisoning attacks,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 4274–4283. [Online]. Available: http://proceedings.mlr.press/v97/mahloujifar19a.html
  * [59] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 118–128.
  * [60] X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” _CoRR_ , vol. abs/1712.05526, 2017\. [Online]. Available: http://arxiv.org/abs/1712.05526
  * [61] Y. Liu, S. Ma, Y. Aafer, W. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018_. The Internet Society, 2018\. [Online]. Available: http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2018/02/ndss2018_03A-5_Liu_paper.pdf
  * [62] J. Dumford and W. J. Scheirer, “Backdooring convolutional neural networks via targeted weight perturbations,” _CoRR_ , vol. abs/1812.03128, 2018. [Online]. Available: http://arxiv.org/abs/1812.03128
  * [63] Y. Ji, X. Zhang, S. Ji, X. Luo, and T. Wang, “Model-reuse attacks on deep learning systems,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 349–363. [Online]. Available: https://doi.org/10.1145/3243734.3243757
  * [64] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How To Backdoor Federated Learning,” in _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, S. Chiappa and R. Calandra, Eds., vol. 108. Online: PMLR, 26–28 Aug 2020, pp. 2938–2948.
  * [65] C. Xie, K. Huang, P.-Y. Chen, and B. Li, “Dba: Distributed backdoor attacks against federated learning,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rkgyS0VFvr
  * [66] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2014. [Online]. Available: http://arxiv.org/abs/1312.6199
  * [67] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” in _Machine Learning and Knowledge Discovery in Databases_ , H. Blockeel, K. Kersting, S. Nijssen, and F. Železný, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 387–402.
  * [68] N. Carlini and D. A. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017_. IEEE Computer Society, 2017, pp. 39–57. [Online]. Available: https://doi.org/10.1109/SP.2017.49
  * [69] O. Suciu, S. E. Coull, and J. Johns, “Exploring adversarial examples in malware detection,” in _2019 IEEE Security and Privacy Workshops (SPW)_ , 2019, pp. 8–14.
  * [70] M. Bauer, “New Covert Channels in HTTP: Adding Unwitting Web Browsers to Anonymity Sets,” in _Proceedings of the 2003 ACM Workshop on Privacy in the Electronic Society_ , ser. WPES ’03. New York, NY, USA: Association for Computing Machinery, 2003, p. 72–78. [Online]. Available: https://doi.org/10.1145/1005140.1005152
  * [71] A. Mileva and B. Panajotov, “Covert channels in TCP/IP protocol stack - Extended version -,” _Central European Journal of Computer Science_ , vol. 4, pp. 45–66, 06 2014. [Online]. Available: https://doi.org/10.2478/s13537-014-0205-6
  * [72] J. Selvi, “Covert channels over social networks,” SANS Institute, Tech. Rep., June 2012, https://www.sans.org/reading-room/whitepapers/engineering/paper/33960.
  * [73] S. Chandra, Z. Lin, A. Kundu, and L. Khan, “Towards a systematic study of the covert channel attacks in smartphones,” in _International Conference on Security and Privacy in Communication Networks_ , J. Tian, J. Jing, and M. Srivatsa, Eds. Cham: Springer International Publishing, 2015, pp. 427–435.
  * [74] S. Zander, G. Armitage, and P. Branch, “Covert channels and countermeasures in computer network protocols,” _IEEE Communications Surveys and Tutorials_ , vol. 9, pp. 44–57, 09 2007. [Online]. Available: https://doi.org/10.1109/COMST.2007.4317620
  * [75] D. C. Castro, J. Tan, B. Kainz, E. Konukoglu, and B. Glocker, “Morpho-MNIST: Quantitative assessment and diagnostics for representation learning,” _Journal of Machine Learning Research_ , vol. 20, no. 178, 2019.