100 tcb@breakable

# Privacy and Robustness in Federated Learning: Attacks and Defenses

Lingjuan Lyu∗, Han Yu∗, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang
Yang∗, , and Philip S. Yu Lingjuan Lyu is with Sony AI. E-mail:
<EMAIL_ADDRESS>Yu and Jun Zhao are with the School of Computer
Science and Engineering, Nanyang Technological University, Singapore. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Ma is with the School of
Information Technology, Deakin University, Geelong, Australia. E-mail:
<EMAIL_ADDRESS>Chen is with College of Computer Science,
Zhejiang University, China<EMAIL_ADDRESS>Sun is with
Department of Computer Science and Engineering, Lehigh University. E-mail:
<EMAIL_ADDRESS>Yang is with Department of AI, WeBank, Shenzhen,
China, and Department of Computer Science and Engineering, Hong Kong
University of Science and Technology. E-mail<EMAIL_ADDRESS>S. Yu is with
Information Technology, University of Illinois at Chicago. E-mail:
<EMAIL_ADDRESS>authors.

###### Abstract

As data are increasingly being stored in different silos and societies
becoming more aware of data privacy issues, the traditional centralized
training of artificial intelligence (AI) models are facing efficiency and
privacy challenges. Recently, federated learning (FL) has emerged as an
alternative solution and continue to thrive in this new reality. Existing FL
protocol designs have been shown to be vulnerable to adversaries within or
outside of the system, compromising data privacy and system robustness.
Besides training powerful global models, it is of paramount importance to
design FL systems that have privacy guarantees and are resistant to different
types of adversaries. In this paper, we conduct a comprehensive survey on
privacy and robustness in federated learning over the past 5 years. Through a
concise introduction to the concept of FL, and a unique taxonomy covering: 1)
threat models; 2) privacy attacks and defenses; 3) poisoning attacks and
defenses, we provide an accessible review of this important topic. We
highlight the intuitions, key techniques as well as fundamental assumptions
adopted by various attacks and defenses. Finally, we discuss promising future
research directions towards robust and privacy-preserving FL, and their
interplays with multidisciplinary goals of FL.

###### Index Terms:

Federated Learning, Privacy, Robustness, Attacks, Defenses

##  I Introduction

As computing devices become increasingly ubiquitous, people generate huge
amounts of data through their day-to-day usage. Collecting such data into
centralized storage facilities is costly and time consuming. Traditional
centralized machine learning (ML) approaches cannot support such ubiquitous
deployments and applications due to infrastructure shortcomings such as
limited communication bandwidth, intermittent network connectivity, and strict
delay constraints [1]. Another critical concern is data privacy and user
confidentiality as the usage data usually contain sensitive information [2].
Sensitive data such as facial images, location-based services, or health
information can be used for targeted social advertising and recommendation,
posing immediate or potential privacy risks. Hence, private data should not be
directly shared without any privacy protection. As societies become
increasingly aware of privacy preservation, legal restrictions such as the
General Data Protection Regulation (GDPR) are emerging, which makes data
aggregation practices less feasible [3].

In this scenario, federated learning (FL) (also well known as collaborative
learning), which distributes model training to the devices from which data
originate, emerged as a promising alternative ML paradigm [4]. FL enables a
multitude of participants to construct a joint ML model without exposing their
private training data [4, 5]. It can also handle unbalanced and non-
independent and identically distributed (non-I.I.D.) data, which naturally
arise in the real world [6]. In recent years, FL has benefited a wide range of
applications such as next word prediction [6, 7], visual object detection for
safety [8], entity resolution [9], recommendation [10, 11, 12], industrial IoT
[13], and graph-based analysis [14, 15, 16], etc.

###  I-A Categorization of Federated Learning based on Distribution

Based on the distribution of data features and data samples among
participants, federated learning can be generally classified as horizontally
federated learning (HFL), vertically federated learning (VFL) and federated
transfer learning (FTL) [17].

TABLE I: Taxonomy for horizontal federated learning (HFL).

HFL |  Number of Participants |  Training Participation |  Technical Capability
---|---|---|---
H2B |  small |  frequent |  high
H2C |  large |  not frequent |  low

Under HFL, datasets owned by each participant share similar features but
concern different users [18]. For example, several hospitals may each store
similar types of data (e.g., demographic, clinical, and genomics) about
different patients. If they decide to build a machine learning model together
using FL, we refer to such a scenario as HFL. In this paper, we further
classify HFL into HFL to businesses (H2B), and HFL to consumers (H2C). A
comparison between H2B and H2C is listed in Table I. The main difference lies
in the number of participants, FL training participation level and technical
capability, which can influence how adversaries attempt to compromise the FL
system. Under H2B, there are typically a handful of participants. They can be
frequently selected during FL training. The participants tend to possess
significant computational power and sophisticated technical capabilities [3].
Under H2C, there can be thousands or even millions of potential participants.
In each round of training, only a subset of them are selected. As their
datasets tend to be small, the chance of a participant being selected
repeatedly for FL training is low. They generally possess limited
computational power and low technical capabilities. An example of H2C is
Google’s GBoard application [7].

VFL is applicable to the cases in which participants have large overlaps in
the sample space but differ in the feature space, i.e., different participants
hold different attributes of the same records [19]. VFL mainly targets
business participants. Thus, the characteristics of VFL participants are
similar to those of H2B participants.

Nowadays, FTL is attracting increasing attention in industries such as
finance, medicine and healthcare. FTL deals with scenarios in which FL
participants have little overlap in both the sample space and the feature
space [17, 3]. In this case, transfer learning [20] techniques can be applied
to provide solutions for the entire sample and feature space under a
federation. FTL enables complementary knowledge to be transferred across
domains in a data federation, thereby enabling a target-domain party to build
flexible and effective models by leveraging rich labels from a source domain
[21].

Figure 1: A typical FL training process, in which both the (potentially
malicious) FL server/aggregator and malicious participants may pose threats to
the FL system.

###  I-B Categorization of Federated Learning based on Architectures

FL with Homogeneous Architectures: Sharing gradients is typically limited only
to homogeneous FL architectures, i.e., the same model is shared with all
participants. Participants aim to collaboratively learn a more accurate model.
Specifically, the model parameters $w$ of the model are often obtained via
solving the following optimization problem:
$\min_{w}{\textstyle\sum}_{i=1}^{n}F(w,D_{i})$, where $F(w,D_{i})$ is the
objective function for the local training dataset on the $i$-th participant
and characterizes how well the parameters $w$ model the local training dataset
$D_{i}$ on the $i$-th participant. Different classifiers (e.g., logistic
regression, deep neural networks) use different objective functions. In FL,
each participant maintains a local model for its local training dataset. The
server maintains a global model via aggregating local models from $n$
participants. Specifically, FL with homogeneous architectures performs the
steps in Fig. 1. FL with homogeneous architectures generally comes in two
forms [6]: (1) FedSGD, in which each participant sends every SGD update to the
server; (2) FedAvg, in which participants locally batch multiple iterations of
SGD before sending updates to the server, which is more communication
efficient. These methods are all based on the mean aggregation rule that takes
the average of the local model parameters as the global model. However, the
global model mean value can be arbitrarily manipulated even if just one
participant is compromised [22, 23].

FL with Heterogeneous Architectures: The most recent efforts extended FL to
collaboratively train models with heterogeneous architectures [24, 25].
Conventional federated model training which directly averages model weights is
only possible if all local models have the same model structure. Naturally, it
limits collaboration among data owners with heterogeneous model architectures.
Sharing model prediction instead of model parameters or updates removes this
obstacle and eliminates the risk of white-box inference attacks in
conventional federated learning [26, 25]. Unlike the existing federated
learning algorithms, _Federated Model Distillation_ (FedMD) does not force a
single global model onto local models. Instead, it is conducted in a succinct,
black-box and model agnostic manner. Each local model is updated separately,
participants share the knowledge of their local models via their predictions
on an unlabeled public set [25]. Another obvious benefit of sharing logits is
the reduced communication costs, without significantly affecting utility [25].

In summary, all the above sharing methods did not provide defense against
privacy and poisoning attacks – two main source of threats to FL.

###  I-C Threats to FL

FL offers a privacy-aware paradigm of model training which does not require
data sharing and allows participants to join and leave a federation freely.
Nevertheless, recent works have demonstrated that FL may not always provide
sufficient privacy and robustness guarantees. Existing FL protocol designs are
vulnerable to: (1) a malicious server who aims to infer sensitive information
from individual updates over time, tamper with the training process or control
the view of the participants on the global parameters; (2) any adversarial
participant who can infer other participants’ sensitive information, tamper
the global parameter aggregation or poison the global model.

In terms of privacy leakage, communicating gradients throughout the training
process can reveal sensitive information [27, 28], even cause deep leakage
[29], either to a third party or the central server [7, 30]. For instance, as
mentioned in [31], even a small portion of gradients can reveal a fair amount
of sensitive information about the local data. Recent works further show that,
by simply observing the gradients, a malicious attacker can successfully steal
the training data [29, 32].

In terms of robustness, FL systems are vulnerable to both data poisoning [33,
34] and model poisoning attacks [35, 36, 37, 38]. Malicious participants can
attack the convergence of the global model or implant backdoor triggers into
the global model by deliberately altering their local data (data poisoning) or
their gradients uploads (model poisoning). More broadly, poisoning attacks can
be categorized into (1) untargeted attack such as Byzantine attack where the
adversary aims to destroy the convergence and performance of the global model
[39, 22]; and (2) targeted attack such as backdoor attack where the adversary
aims to implant a backdoor trigger into the global model so as to trick the
model to constantly predict an adversarial class on a subtask while keeping
good performance on the main task [35, 36, 34].

These privacy and robustness attacks pose significant threats to FL. In
centralized learning, the server is responsible for all the participants’
privacy and model robustness. However, in FL, any participant can attack the
server and spy on other participants, which sometimes even without involving
the server. Therefore, it is important to understand the principles behind
these privacy and robustness attacks. The properties of the representative
privacy and robustness attacks in server-based FL are summarized in Table II.

TABLE II: A summary of attacks against server-based FL.

Attack Type | Attack Target | Attacker Role | FL Scenario | Attack Complexity
---|---|---|---|---
Model | Training Data | Participant | Server | H2B | H2C | Attack Iteration | Auxiliary Knowledge Required
|  |  |  |  |  | One Round | Multiple Rounds |
Robustness | Untargeted attack | YES | NO | YES | NO | YES | NO | YES | YES | YES
Targeted attack | YES | NO | YES | NO | YES | NO | YES | YES | YES
Privacy | Infer Class Representatives | NO | YES | YES | YES | YES | NO | NO | YES | YES
Infer Membership | NO | YES | YES | YES | YES | NO | NO | YES | YES
Infer Properties | NO | YES | YES | YES | YES | NO | NO | YES | YES
Infer Training Inputs and Labels | NO | YES | NO | YES | YES | NO | YES | YES | NO

###  I-D Secure FL

Attacks on FL come from either the privacy perspective when a malicious
participant or the central server attempts to infer the private information of
a victim participant, or the robustness perspective when a malicious
participant aims to compromise the global model.

To secure FL against privacy attacks, existing privacy-preserving
methodologies in centralized machine learning have been tried to be adopted
into FL, including homomorphic encryption (HE), secure multiparty computation
(SMC), and differential privacy (DP). However, HE and SMC may not be
applicable to large-scale FL, as they incur substantial communication and
computation overhead. In aggregation-based tasks, DP requires the aggregated
value to contain random noise up to a certain magnitude to ensure
$(\epsilon,\delta)$-DP, thus is also not ideal for FL. The noise addition
required by DP is also hard to execute in FL. In an ideal scenario where the
server (aggregator) is trusted, the server can add the noise to the aggregated
gradients [7]. However, in many real-world scenarios, the participants may not
trust the central server nor each other. In this case, the participants would
compete with each other, and all want to ensure Local Differential Privacy
(LDP) by adding as much noise as possible to their local gradients [30, 40,
41]. This tends to accumulate significant error at the server side.
_Distributed Differential Privacy_ (DDP) [30, 40, 41] can mitigate this
problem to some extent when at least a certain fraction of the participants
are honest and do not conduct such malicious competition.

Defending FL against various robustness attacks (e.g., untargeted Byzantine
attack, targeted backdoor attack) is an extremely challenging task. This is
due to two main reasons. First, the defense can only be executed at the server
side where only local gradients are available. This invalids many backdoor
defense methods developed in the centralized machine learning, for example,
denoising (preprocessing) methods [42, 43, 44, 45, 46], backdoor
sample/trigger detection methods [47, 48, 49, 50, 51, 52], robust data
augmentations [53], finetuning methods [53], neural attention distillation
(NAD) based method [54], and more recent anti-backdoor learning method based
on a sophisticated learning process [55]. Second, the defense method has to be
robust to both data poisoning and model poisoning attacks. Most existing
robustness defenses are gradient aggregation methods mainly developed for
defending against the untargeted Byzantine attackers, such as Krum/Multi-Krum
[22], AGGREGATHOR [56], Byzantine Gradient Descent (BGD) [57], Median-based
Gradient Descent [23], Trimmed-mean-based Gradient Descent [23] and SIGNSGD
[39]. These defense methods have never been tested on the targeted backdoor
attacks [35, 36, 38, 33, 34]. Dedicated defense methods against both data
poisoning and model poisoning attacks have been investigated, such as norm
clipping [38], geometric median based Robust Federated Aggregation (RFA) [58]
and robust learning rate [59]. For the collusion of Sybil attacks,
contribution similarity [37] can be leveraged as a strategy for defense.

TABLE III: A list of abbreviations used in this survey.

AI |  Artificial Intelligence
---|---
ML |  Machine Learning
FL |  Federated Learning
GDPR |  General Data Protection Regulation
I.I.D. |  Independent and Identically Distributed
IoT |  Internet of Things
HFL |  Horizontally Federated Learning
VFL |  Vertically Federated Learning
FTL |  Federated Transfer Learning
H2B |  HFL to businesses
H2C |  HFL to consumers
SGD |  Stochastic Gradient Descent
SMC |  Secure Multiparty Computation
DP |  Differential Privacy
CDP |  Centralized Differential Privacy
LDP |  Local Differential Privacy
DDP |  Distributed Differential Privacy
HE |  Homomorphic Encryption
RFA |  Robust Federated Aggregation
GAN |  Generative Adversarial Network
MIA |  Membership Inference Attack
AT |  Adversarial Training
FAT |  Federated Adversarial Training
API |  Application Programming Interface

Figure 2: Survey organization.

###  I-E Motivation of this Survey and Our Contribution

Existing surveys on FL are mostly focused on the system or protocol design
[17, 60, 61]. A notable number of research works have been conducted on
privacy and robustness. Although these works attempt to discover the
vulnerabilities of FL and aim to enhance the privacy and system robustness of
FL, there are very few efforts for categorizing them in a systemic manner, and
privacy and robustness threats to FL have not been systematically explored. To
fill in this gap, in this paper, we have conducted an extensive survey on the
recent advances in privacy and robustness threats to FL and their defenses. In
particular, we focus on two specific threats initiated by insiders in FL
systems: 1) privacy attacks that attempt to infer the victim participants’
private information; 2) poisoning attacks that attempt to prevent the learning
of a global model, or implant triggers to control the behaviour of the global
model. This article mainly surveys the literature over the past 5 years on
privacy and robustness in federated learning, it can be a notable inclusion to
the existing literature, helping the community better understand the state-of-
the-art privacy and robustness progress in FL. The limitations and the
promising use cases of the existing works in literature, and open directions
for future research are also offered to identify the research gaps to address
the challenges of privacy and robustness in FL. The major contributions of
this survey include:

  * •

This survey presents a comprehensive categorization of FL, and summarized
threats and the corresponding protections for FL in a systematic manner.

  * •

Existing privacy and robustness attacks and defenses are well explored to help
readers better understand the assumptions, principles, reasons and differences
of the current progress in the domain of FL privacy and robustness.

  * •

The conflicts between privacy and robustness, and among multiple design goals
are identified; the gaps between the current works and the real scenarios in
FL are summarized.

  * •

Future research directions will assist community to rethink and improve their
current designs towards robust and privacy-preserving FL of real practicality
and impact. Meanwhile, it is suggested to integrate multidisciplinary goals in
the system design of FL.

###  I-F Survey Organization

The rest of the survey is organized as follows. Before going into an in-depth
discussion on privacy and robustness in FL, in Section II, we first summarize
the threat models from a general perspective and discuss the customized threat
models for privacy and robustness respectively. Section III presents a
comprehensive review of the privacy attacks in FL, particularly targeting the
sensitive information (class representative, membership, properties, training
inputs and labels) in HFL with homogeneous architectures. To address the
corresponding privacy attacks, Section IV lists the most representative
privacy-preserving techniques and current practices that have applied these
techniques in FL. Section V shows the detailed poisoning attacks that aim to
compromise system robustness, including the untargeted and targeted poisoning
attacks, followed by their countermeasures in Section VI. From the lessons
learned in this survey paper, the research gaps towards realizing trustworthy
FL, along with directions for future research are provided in Section VII.
Finally, concluding remarks are drawn in Section VIII.

For better readability, we give a diagram in Fig. 2 showing the different
aspects covered in the survey. The list of abbreviations used in this survey
is provided in Table III. Throughout this survey, we will interchangeably use
participants/clients/users to represent the participants in FL.

##  II Threat Models

Before reviewing attacks on FL, we first present a summary of the threat
models. Generally, threat models in FL can be categorized into two types: (1)
Insider v.s. Outsider; (2) Training Phase v.s. Inference Phase. These threat
models apply to both the privacy and robustness. Additionally, privacy and
robustness have their own threat models.

###  II-A Insider v.s. Outsider

Attacks can be carried out by insiders and outsiders. Insider attacks include
those launched by the FL server and the participants in the FL system.
Outsider attacks include those launched by the eavesdroppers on the
communication channel between participants and the FL server, and by users of
the final FL model when it is deployed as a service.

Insider attacks are generally more dangerous than outsider attacks, as it
strictly enhances the capability of the adversary. Thus, our discussion of
attacks against FL will focus primarily on the insider attacks.

###  II-B Training Phase v.s. Inference Phase

Training Phase. Attacks conducted during the training phase attempt to learn,
influence, or corrupt the FL model itself [62]. During the training phase, the
attacker can run data poisoning attacks to compromise the integrity of the
training dataset [37, 35, 63, 64, 65, 66], or model poisoning attacks to
compromise the integrity of the learning process [36, 67]. The attacker can
also launch a range of inference attacks on an individual participant’s update
or on the aggregated update from all participants during training phase [28].

Inference Phase. Attacks conducted during the inference phase are called
evasion or exploratory attacks [68]. They generally do not alter the targeted
model, instead, either trick it to produce wrong predictions
(targeted/untargeted) or collect evidence about the model characteristics,
causing privacy and robustness problems. The effectiveness of such attacks are
largely determined by the information that is available to the adversary about
the model. Inference phase attacks can be classified into white-box attacks
(i.e., with full access to the FL model) and black-box attacks (i.e., only
able to query the FL model). In FL, the global model maintained by the server
suffers from the same evasion attacks as in the conventional ML setting when
the target model is deployed as a service. While black-box attacks may be more
natural to consider in the centralized settings, the model broadcast step in
FL makes the global model a white-box to any malicious participant. Thus, FL
requires extra efforts to defend against white-box evasion attacks [61].

###  II-C Privacy: Semi-honest v.s. Malicious

Semi-honest Setting. Adversaries are considered passive or honest-but-curious.
They try to learn the private states of other participants without deviating
from the FL protocol. The adversaries can only observe the received
information, i.e., parameters of the global model.

Malicious Setting. An active or malicious adversary tries to learn the private
states of honest participants, and deviates arbitrarily from the FL protocol
by modifying, re-playing, or removing messages. This setting allows the
adversary to conduct particularly devastating attacks.

###  II-D Robustness: Untargeted v.s. Targeted

  1. 1.

Untargeted attack: The untargeted poisoning attack aims to arbitrarily
compromise the integrity of the target model. Byzantine attack is one type of
the untargeted poisoning attacks that uploads arbitrarily malicious gradients
to the server so as to cause the failure of the global model [69, 22, 56, 39,
70].

  2. 2.

Targeted attack: The targeted poisoning attack induces the model to output the
target label specified by the adversary for particular testing examples, while
the testing error for other testing examples is unaffected [35, 36, 37, 38].

##  III Privacy Attacks

Although FL prevents the participants from directly sharing their private
data, a series of works have demonstrated that exchanging gradients in FL can
also leak sensitive information about the participants’ private data to either
passive or active attackers [71, 72, 28, 29, 73]. For example, gradients or
two consecutive snapshots of the FL model parameters can leak unintended
features of the participants’ training data to the adversarial participants,
as deep learning models tend to recognize and remember more features of the
data than needed for the main learning task [74]. Fig. 3 illustrates the set
of information an adversary can infer from the gradients (i.e.,
$\Delta\bm{w}_{1}^{t}$), or equivalently the difference of two successive
snapshots of the model parameters (i.e., $\bm{w}^{t+1}-\bm{w}^{t}$).

Figure 3: A demo of privacy leakage in FL. Attacker can infer various private
information about the victim participant from the received gradients or the
snapshot of the FL model parameters.

The reason why gradients can cause privacy leakage is that the gradients are
derived from the participants’ private training data, and a learning model can
be considered as a representation of the high-level statistics of the dataset
it was trained on [75]. In deep learning models, gradients of a given layer
are computed based on the layer’s features and the error from the layer after
(i.e., backpropagation). In the case of sequential fully-connected layers, the
gradients of the weights are the inner products of the current layer’s
features and the error from the layer after. Similarly, for a convolutional
layer, the gradients of the weights are convolutions of the layer’s features
and the error from the layer after [28]. Consequently, observations of
gradients can be used to infer a significant amount of private information,
such as class representatives, membership and properties of a subset of the
training data. Even worse, an attacker can infer labels from the shared
gradients and recover the original training samples without any prior
knowledge about the training data [29]. Next, we detail the potential privacy
leakage of FL according to the type of the sensitive information that the
attacker is targeting at.

###  III-A Inferring Class Representatives

Hitaj et al. [76] firstly devised an active inference attack called
_Generative Adversarial Networks_ (GAN) attack against deep FL models. In this
attack, a malicious participant can intentionally compromise any other
participant. The GAN attack exploits the real-time nature of the FL learning
process which allows the adversarial participant to train a GAN to generate
prototypical samples of the targeted private training data. The generated
samples appear to come from the same distribution as the training data. Hence,
GAN attack is not targeted to reconstruct the exact training inputs, but only
the class representatives. It should be noted that GAN attack assumes the
entire training corpus for a given class comes from a single participant,
which means the GAN-constructed representatives are similar to the training
data only when all class members are similar. This resembles model inversion
attacks in the centralized ML settings [77]. Note that these assumptions may
be less practical in FL. Since GAN attack requires a substantial amount of
computational resources to train the GAN model, it is less suitable for H2C
scenarios.

###  III-B Inferring Membership

Given an exact data point, membership inference attacks (MIA) aim to determine
if it was used to train the model [78]. For example, an attacker can infer
whether a specific patient profile was used to train a classifier associated
with a certain disease. FL opens new possibilities for such attacks. In FL,
the adversary can infer if a particular sample belongs to the private training
data of a particular participant (if the target update is from a single
participant) or any participant (if the target update is the aggregate). For
example, during FL model training, the non-zero gradients of the embedding
layer of a deep natural language processing model trained on text data can
reveal which words are in the training batches of the honest participants
[28].

Attackers in a FL system can conduct both active and passive membership
inference attacks [73, 28]. In the passive case, the attacker observes the
updated model parameters and performs inference without modifying the learning
process. In the active case, the attacker can tamper with the FL model
training protocol and perform a more powerful attack against other
participants. For instance, the attacker may share malicious updates and trick
the FL model to expose more information about other participants’ local data.
One such attack is the gradient ascent attack [73], where the attacker runs
gradient ascent on a target data sample and observes whether its increased
loss can be drastically reduced in the next communication round, if so, the
sample is very likely to be in the training set. This attack can be applied on
a batch of target data samples all at the same time [73].

###  III-C Inferring Properties

An adversary can launch both passive and active property inference attacks to
infer certain properties of other participants’ training data [28]. Property
inference attacks assume that the adversary has auxiliary training data that
are correctly labelled with the target property. A passive adversary can only
observe or eavesdrop the gradients and perform inference by training a binary
property classifier. An active adversary can exploit multi-task learning to
trick the FL model into learning a better separation between data with and
without the target property so as to extract more information. An adversarial
participant can also infer when a property appears or disappears in the
training data (e.g., identifying when a person first appears in the photos
used to train a gender classifier). The assumption of auxiliary training data
in property inference attacks may limit its applicability in H2C.

###  III-D Inferring Training Inputs and Labels

One recent work called _Deep Leakage from Gradient_ (DLG) proposes an
optimization algorithm to extract both the training inputs and the labels
[29]. This attack is much stronger than previous approaches. It can accurately
recover the raw images and texts used to train a deep learning model. In a
follow-up work [32], an analytical approach called _Improved Deep Leakage from
Gradient_ (iDLG) was proposed to extract labels based on the shared gradients
and an exploration of the correlation between the labels and the signs of the
gradients. iDLG can be applied to attack any differentiable models trained
with cross-entropy loss and one-hot labels, which is a typical setting for
classification tasks.

In summary, inference attacks generally assume that the adversaries possess
sophisticated technical capabilities and unlimited computational resources.
Moreover, most attacks assume that the adversarial participants can be
selected (to update the global model) in many rounds of the FL training
process. In FL, these assumptions are generally not practical in H2C
scenarios, but more likely happen in H2B scenarios. These inference attacks
highlight the need for gradient protection in FL, possibly through various
privacy-preserving mechanisms [3] detailed in Section IV.

##  IV Defenses against Privacy Attacks

TABLE IV: Privacy-preserving Techniques for FL. Privacy-preserving Techniques | Existing Works
---|---
Homomorphic Encryption |  [79, 80]
DP | CDP |  [81, 7]
LDP |  [82, 83, 84, 27, 85, 86, 87]
DDP+Cryptography |  [30, 40, 88]
Secure Multiparty Computation |  [89, 5]

While privacy preservation has been extensively studied in the machine
learning community, privacy preservation in federated learning can be more
challenging due to the sporadic access to power and network connectivity,
statistical heterogeneity in the data, etc. Existing works in privacy-
preserving federated learning are mostly developed based on the well-known
privacy-preserving techniques, including: (1) _homomorphic encryption_ (HE),
such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan
cryptosystems [92]; (2) _Secure Multiparty Computation_ (SMC), such as garbled
circuits [93] and secret sharing [94]; and (3) _differential privacy_ (DP)
[95, 96]. A concise summary of privacy-preserving techniques is listed in
Table IV.

###  IV-A Privacy Preservation through Homomorphic Encryption

A homomorphic encryption scheme allows arithmetic operations to be directly
performed on ciphertexts, which is equivalent to a specific linear algebraic
manipulation of the plaintext. Existing homomorphic encryption techniques can
be categorized into: 1) fully homomorphic encryption, 2) somewhat homomorphic
encryption, and 3) partially homomorphic encryption. Fully homomorphic
encryption can support arbitrary computation on ciphertexts, but is less
efficient [92]. On the other hand, somewhat homomorphic encryption and
partially homomorphic encryption are more efficient but are specified by a
limited number of operations [97, 98, 91, 90]. Partially homomorphic
encryption schemes are more widely used in practice, including RSA [98], El
Gamal [91], Paillier [90], etc. The homomorphic properties can be described
as:

$E_{pk}(m_{1}+m_{2})=c_{1}\oplus c_{2}\\\ $ $E_{pk}(a\cdot m_{1})=a\otimes
c_{1}$

where $a$ is a constant, $m_{1}$, $m_{2}$ are the plaintexts that need to be
encrypted, $c_{1}$, $c_{2}$ are the ciphertext of $m_{1}$, $m_{2}$
respectively.

Homomorphic encryption is widely used and is especially useful for securing
the learning process by computing on encrypted data. However, doing arithmetic
on the encrypted numbers comes at a cost of memory and processing time. For
example, with Paillier encryption scheme, the encryption of an encoded
floating-point number (whether single or double precision) is $2m$ bits long,
where $m$ is typically at least 1024 and the addition of two encrypted numbers
is 2$\sim$3 orders of magnitude slower than the unencrypted equivalent [9].
Moreover, polynomial approximations need to be made to evaluate non-linear
functions in machine learning algorithms, resulting in a trade-off between
utility and privacy [79, 80]. For example, to protect individual gradients,
Aono et al. [31] used additively homomorphic encryption to preserve the
privacy of gradients and enhance the security of the distributed learning
system. However, their protocol not only incurs large communication and
computational overhead, but also results in utility loss. Furthermore, it is
not able to withstand collusion between the server and multiple participants.
Hardy et al. [9] applied federated logistic regression on vertically
partitioned data encrypted with an additively homomorphic scheme to secure
against an honest-but-curious adversary. Overall, all these works incur extra
communication and computational overheads, which limit their applications in
H2C scenarios.

###  IV-B Privacy-Preservation through SMC

_Secure Multiparty Computation_ (SMC) [93] enables different participants with
private inputs to perform a joint computation on their inputs without
revealing them to each other. Mohassel et al. [89] proposed SecureML which
conducts privacy-preserving learning via SMC, where data owners need to
process, encrypt and/or secret-share their data among two non-colluding
servers in the initial setup phase. SecureML allows data owners to train
various models on their joint data without revealing any information beyond
the outcome. However, this comes at a cost of high computation and
communication overhead, which may hamper participants’ interest to
collaborate. Bonawitz et al. [5] proposed a secure, communication-efficient,
and failure-robust protocol based on SMC for secure aggregation of individual
gradients. It ensures that the only information about the individual users the
server learns is what can be inferred from the aggregated results. The
security of their protocol is maintained under both the honest-but-curious and
malicious settings, even when the server and a subset of users act maliciously
– colluding and deviating arbitrarily from the protocol. That is, no party
learns anything more than the sum of the inputs of a subset of honest users of
a large size [5].

In general, SMC techniques ensure a high level of privacy and accuracy, at the
expense of high computation and communication overhead, thereby doing a
disservice to attracting participation. Another main challenge facing SMC-
based schemes is the requirement for simultaneous coordination of all
participants during the entire training process. Such a multi-party
interaction model may not be desirable in practical settings, especially under
the commonly considered participant-server architecture in FL settings.
Besides, SMC-based protocols can enable multiple participants to
collaboratively compute an agreed-upon function without leaking input
information from any participant except for what can be inferred from the
outcomes of the computation [99, 100]. That said, SMC cannot fully guarantee
protection from information leakage, which requires additional differential
privacy techniques to be incorporated into the multi-party protocol to address
such concerns [101, 102, 103, 104].

In summary, homomorphic encryption or SMC-based approaches may not be
applicable to large-scale FL scenarios as they incur substantial additional
communication and computation costs. Moreover, encryption based techniques
need to be carefully designed and implemented for each operation in the target
learning algorithm [105, 106]. Lastly, all the cryptography based protocols
prevent anyone from auditing participants’ updates to the joint model, which
leaves spaces for the malicious participants to attack. For example, malicious
participants can introduce stealthy backdoor functionality into the global
model without being detected [36].

###  IV-C Privacy-Preservation through Differential Privacy

Differential privacy (DP) was originally designed for the single database
scenario, where for every query made, a database server answers the query in a
privacy-preserving manner with tailored randomization [95]. In comparison with
encryption based approaches, differential privacy trades off privacy and
accuracy by perturbing the data in a way that (i) is computationally
efficient, (ii) does not allow an attacker to recover the original data, and
(iii) does not severely affect the utility.

The concept of differential privacy is that the effect of the presence or the
absence of a single record on the output likelihood is bounded by a small
factor $\epsilon$. As defined in Definition IV.1,
$(\epsilon,\delta)$-approximate differential privacy [96] relaxes pure
$\epsilon$-differential privacy by a $\delta$ additive term, which means the
unlikely responses need not satisfy the pure differential privacy criterion.

###### Definition IV.1

$(\epsilon,\delta)$-differential privacy [96]. For scalars $\epsilon>0$ and
$0\leq\delta<1$, mechanism $\mathcal{M}$ is said to preserve (approximate)
$(\epsilon,\delta)$-differential privacy if for all adjacent datasets
$D,D^{\prime}\in\mathcal{D}^{n}$ and measurable
$S\in\mathrm{range}(\mathcal{M})$,

$\displaystyle\Pr\\{\mathcal{M}(D)\in S\\}$ $\displaystyle\leq$
$\displaystyle\exp(\epsilon)\cdot\Pr\\{\mathcal{M}(D^{\prime})\in
S\\}+\delta\enspace.$

To avoid the worst-case scenario of always violating privacy of a $\delta$
fraction, the standard recommendation is to choose $\delta\ll 1/|D|$, where
$|D|$ is the size of the database. This strategy forecloses possibility of one
particularly devastating outcome, but other forms of information leakage
remain.

The privacy community generally categorizes DP into the following three
categories as per different trust assumptions and noise sources: centralized
DP (CDP), local DP (LDP) and distributed DP (DDP). A comprehensive comparison
among CDP, LDP and DDP is listed in Table V.

TABLE V: Comparative analysis among CDP, LDP and DDP.

DP type | Trusted aggregator? | Who should add noise? | Privacy Guarantee | Error Bound
---|---|---|---|---
CDP [81, 7] | Yes | aggregator | aggregated value | $O(\frac{1}{\epsilon})$
LDP [27, 107] | No | user | locally released value | $O(\frac{\sqrt{n}}{\epsilon})$
DDP [30, 41] | No | user | aggregated value | $O(\frac{1}{\epsilon})$

Centralized Differential Privacy (CDP). CDP was originally designed for the
centralized scenario where a trusted database server, who is entitled to see
all participants’ data in the clear, wishes to _answer queries or publish
statistics_ in a privacy-preserving manner by randomizing query results [95,
108, 109]. When CDP meets FL, CDP assumes a trusted aggregator, who is
responsible for adding noise to the aggregated local gradients to ensure
record-level privacy of the whole data of all participants [7, 81]. However,
CDP is geared to tackle thousands of users for training to converge and
achieve an acceptable trade-off between privacy and accuracy [7], resulting in
a convergence problem with a small number of participants [110]. Moreover, CDP
can achieve acceptable accuracy only with a large number of participants, thus
not applicable to H2B with relatively a small number of participants.

Meanwhile, the assumption of a trusted server in CDP is ill-suited in many
applications as it constitutes a single point of failure for data breaches,
and saddles the trusted curator with legal and ethical obligations to keep the
user data secure. When the aggregator is untrusted which is often the case in
distributed scenarios, Local Differential Privacy (LDP) [111] or _Distributed
Differential Privacy_ (DDP) are needed [112, 102] to protect privacy of
individuals.

Local Differential Privacy (LDP). Local differential privacy (LDP) [111]
offers stronger privacy guarantee, data owners perturb their private
information to satisfy DP locally before reporting it to an untrusted data
curator [85, 113, 86]. A comprehensive survey of LDP can be referred to [114].
A formal definition of LDP is given in Definition IV.2.

###### Definition IV.2

($(\epsilon,\delta)$-Local Differential Privacy). A randomized algorithm
$\mathcal{M}$ satisfies $(\epsilon,\delta)$-local differential privacy
($(\epsilon,\delta)$-LDP) if and only if for any input $v$ and $v^{\prime}$,
we have

$\displaystyle\Pr\\{\mathcal{M}(v)=o\\}$ $\displaystyle\leq$
$\displaystyle\exp(\epsilon)\cdot\Pr\\{\mathcal{M}(v^{\prime})=o\\}+\delta\enspace$

for $\forall o\in Range(\mathcal{M})$, where $Range(\mathcal{M})$ denotes the
set of all possible outputs of the algorithm $\mathcal{M}$. Furthermore
$\mathcal{M}$ is said to preserve (pure) $\epsilon$-LDP if the condition holds
for $\delta=0$.

Although the randomized response [115] and its variants [116] have been widely
used to provide LDP when individuals disclose their personal information. We
remark that all the randomization mechanisms used for CDP, such as Laplace
mechanism and Gaussian mechanism [96], can be individually used by each
participant to ensure LDP in isolation. However, in distributed scenario,
without the help of cryptographic techniques, each participant has to add
enough calibrated noise to ensure LDP. The attractive privacy properties of
LDP thus come with a huge utility degradation, especially with billions of
individuals. Under the CDP model, the aggregator releases the aggregated value
with an expected additive error of at most $\Theta(1/\epsilon)$ to ensure
$\epsilon$-DP (e.g., using the Laplace mechanism [96]). In contrast, under the
LDP model, at least $\Omega(\sqrt{n}/\epsilon)$ additive error in expectation
must be incurred by any $\epsilon$-DP mechanism for the same task [117, 111].
This gap is essential for eliminating the trust in the centralized server, and
cannot be removed by algorithmic improvement [118].

Several prior works have attempted to apply LDP to FL. For example, Shokri et
al. [119] firstly applied LDP to distributed/federated learning, in which each
participant individually adds noise to its gradients before releasing to the
server, thus ensuring local DP. However, their privacy bounds are given per-
parameter, the large number of parameters prevents their method from providing
a meaningful privacy guarantee [108]. Other approaches that also considered to
apply LDP to FL can only support shallow models such as logistic regression
and only focus on simple tasks and datasets [82, 83, 84]. Bhowmick et al. [27]
presented a viable approach to large-scale local private model training, and
introduced a relaxed version of LDP by limiting the power of potential
adversaries. Due to the high variance of their mechanism, it requires more
than 200 communication rounds and incurs much higher privacy cost, i.e., MNIST
($\epsilon=500$) and CIFAR-10 ($\epsilon=5000$). Note that the $\epsilon$
required in [27] is relatively large, as they considered only privacy
protection against _reconstruction attacks_ instead of membership attacks, and
they pointed out that local privacy as traditionally employed may prove too
stringent for practical use, especially in modern high-dimensional statistical
and machine learning problems. Their obtained results suggested that using LDP
mechanisms with _large_ $\epsilon$ may still provide decent protection against
reconstruction. Li et al. [107] proposed locally differentially-private
algorithms in the context of meta-learning, which might be applicable to FL
with personalization. However, it only provides provable learning guarantees
in convex settings. Truex et al. [86] applied Condensed Local Differential
Privacy ($\alpha$-CLDP) into FL. However, $\alpha$-CLDP required privacy
budget $\epsilon=\alpha\cdot 2c\cdot 10^{p}$ (e.g. $\alpha=1,c=1,p=10$), which
results in a weak privacy guarantee. Another contemporary work called LDP-FL
[85] achieves better performance on both effectiveness and efficiency than
[86] with a special communication design for deep learning approaches.

In addition to the privacy leakage in FL with homogeneous architectures, FL
with heterogeneous architectures suffers from the similar privacy issues. In
FL with homogeneous architectures, the predictions from local models are also
sensitive and may leak private information [108, 75, 41, 87]. Currently, there
is no theoretic guarantee that sharing prediction is private and secure [75,
87]. To address this issue, one naive approach is adding the locally
differentially private random noise to the predictions like previous works.
Although the privacy concern is mitigated with random noise perturbation, it
brings a new problem with a substantial trade-off between privacy budget and
model utility. Sun et al. [87] filled in this gap by proposing a novel
framework called FEDMD-NFDP, which integrated a novel Noise-Free Differential
Privacy (NFDP) mechanism into federated model distillation. The LDP guarantee
of NFDP roots in local data sampling process, which explicitly eliminates
noise addition and privacy cost explosion issues in previous works.

(a) FL without privacy.

(b) Centralized DP: FL with a trusted server.

(c) Local DP: FL without a trusted server.

(d) Distributed DP with SMC: FL without a trusted server.

Figure 4: An illustration of one round of federated learning without privacy
and with different DP mechanisms. $M$ denotes a DP mechanism used to privatize
the data. In centralized DP (b), the central server is trusted. In local DP
(c), the central server is not trusted, gradients are perturbed to ensure
local DP before forwarding to the central server. In distributed DP (d), the
central server is also not trusted, gradients are perturbed via DP mechanism
$M$ and encrypted via encryption operation $E$ to ensure privacy before
forwarding to the central server, who needs to finally decrypt ($D$) the
aggregated ciphertext.

Distributed Differential Privacy (DDP). _Distributed Differential Privacy_
(DDP) bridges the gap between LDP and CDP, while ensuring the privacy of each
individual by combining with cryptographic protocols [101, 102, 103, 104, 30].
Therefore, DDP avoids placing trust in any server, and offers better utility
than LDP. Theoretically, DDP offers the same utility as CDP, as the total
amount of noise is the same.

The notion of DDP reflects the fact that the required noise in the target
statistic is sourced from multiple participants [112]. Approaches to DDP that
implement an overall additive noise mechanism by summing the same mechanism
run at each participant (typically with less noise) necessitate mechanisms
with stable distributions—to guarantee proper calibration of known end-to-end
response distribution—and cryptography for hiding all but the final result
from participants [102, 112, 103, 101, 30, 40, 88]. Stable distributions
include Gaussian distribution, Binomial distribution [41], etc, i.e., sum of
Gaussian random variables still follow a Gaussian distribution, and sum of
Binomial random variables still follow a Binomial distribution. DDP utilizes
this nice stability to permit each participant to randomise its local
statistic to a lesser degree than would LDP. However, in DDP, only the sum of
the individually released statistics is $(\epsilon,\delta)$-differentially
private but not the individually released statistic, i.e.,
${\textstyle\sum}{\mathchoice{\mbox{\boldmath$\displaystyle
r_{i}$}}{\mbox{\boldmath$\textstyle r_{i}$}}{\mbox{\boldmath$\scriptstyle
r_{i}$}}{\mbox{\boldmath$\scriptscriptstyle r_{i}$}}}$ is sufficient for this
level of differential privacy, but individual noise $\textstyle r_{i}$ alone
is not sufficient, thus $\mathchoice{\mbox{\boldmath$\displaystyle
x_{i}$}}{\mbox{\boldmath$\textstyle x_{i}$}}{\mbox{\boldmath$\scriptstyle
x_{i}$}}{\mbox{\boldmath$\scriptscriptstyle
x_{i}$}}+\mathchoice{\mbox{\boldmath$\displaystyle
r_{i}$}}{\mbox{\boldmath$\textstyle r_{i}$}}{\mbox{\boldmath$\scriptstyle
r_{i}$}}{\mbox{\boldmath$\scriptscriptstyle r_{i}$}}$ cannot be released
directly. Here $\textstyle x_{i}$ and $\textstyle r_{i}$ indicate the
individual plaintext and noise respectively. Therefore, DDP necessitates the
help of SMC to maintain utility and ensure aggregator obliviousness, as
evidenced in [101, 102, 103, 104, 30, 40, 88].

An illustration of one round of FL without privacy and with different DP
mechanisms is given in Fig. 4. Another parallel line of work for privacy-
preserving distributed learning is to transfer the knowledge of the ensemble
of multiple models to a student model [120, 108, 109, 121]. For example, Hamm
et al. [120] first created labeled data from auxiliary unlabeled data, and
then used the labeled auxiliary data to find an empirical risk minimizer,
finally released a differentially private classifier using output perturbation
[122]. Similarly, Papernot et al. [108, 109] proposed _Private Aggregation of
Teacher Ensembles_ (PATE) to first train an ensemble of teachers on disjoint
subsets of private data, then perturb the knowledge of the ensemble of
teachers by adding noise to the aggregated teacher votes before transferring
the knowledge to a student. Finally, a student model is trained on the
aggregate output of the ensemble, such that the student learns to accurately
mimic the ensemble. PATE requires a lot of participants to achieve reasonable
accuracy, and each participant needs to have enough data to train an accurate
model, which might not hold in FL system, where the data distribution of
participants might be highly unbalanced, making this approach unsuitable to FL
system.

##  V Poisoning Attacks

Different from privacy attacks that are targeting at data privacy, poisoning
attacks aim to compromise the system robustness. Depending on the attacker’s
objective, poisoning attacks can be broadly classified into two categories: 1)
untargeted poisoning attacks [123, 124, 125, 70, 67]; and 2) targeted
poisoning attacks [126, 127, 35, 36, 128, 129, 130, 131].

Note that the untargeted and targeted poisoning attacks during the training
phase can be mounted on both the data and the model. Fig. 5 shows that the
poisoned updates can be sourced from two poisoning attacks: (1) data poisoning
attack during local data collection; and (2) model poisoning attack during
local model training process. At a high level, both poisoning attacks attempt
to modify the behavior of the target model in some undesirable way. However,
due to the model sharing nature of FL with homogeneous architectures, data
poisoning attacks are generally less effective than model poisoning attacks
[35, 36, 37, 38]. In fact, model poisoning subsumes data poisoning in FL
settings, as data poisoning attacks eventually change a subset of updates sent
to the model at any given iteration. This is functionally identical to a
centralized poisoning attack in which a subset of the training data is
poisoned.

Figure 5: Data v.s. model poisoning attacks in FL.

###  V-A Untargeted Attacks

Untargeted poisoning attacks aim to arbitrarily compromise the integrity of
the target model. Byzantine attack is one type of _untargeted_ poisoning
attacks that uploads arbitrarily malicious gradients to the server so as to
cause the failure of the global model [69, 22, 56, 39, 70]. A formal
definition of Byzantine attack is given in Definition V.1.

###### Definition V.1

[Byzantine attack] [22, 23]. An honest participant uploads
$\Delta\bm{w}_{i}\coloneqq\nabla F_{i}(\bm{w}_{i})$ while a dishonest
participant can upload arbitrary values.

$\displaystyle\Delta\bm{w}_{i}=\begin{cases}*,&\mbox{if $i$-th participant is
Byzantine},\\\ \nabla F_{i}(\bm{w}_{i}),&\mbox{otherwise,}\end{cases}$ (1)

where “$*$” represents arbitrary values, $F_{i}$ represents participant $i$’s
local model objective function.

Blanchard et al. [22] showed that the aggregation of FL can be completely
controlled by a single Byzantine participant if there is no defense in the FL.
In particular, suppose there are $n-1$ benign participants and a Byzantine
participant, the server aggregates the gradients by
$\Delta\bm{w}^{\prime}=\frac{1}{n}\sum_{i=1}^{n}\Delta\bm{w}_{i}$, where
$\Delta\bm{w}^{\prime}$ is the aggregated gradient. Assume the $n$-th
participant is Byzantine, it can always make the aggregated gradient become
any vector $\bm{u}$ by uploading the following gradient:

$\Delta\bm{w}_{n}=n\bm{u}-\sum_{i=1}^{n-1}\Delta\bm{w}_{i}.$ (2)

Such a simple attack exposes the vulnerability of FL against Byzantine attack.

Chen et al. [132] discussed Byzantine attacks in Adam-based FL and proposed a
camouflage attack that can camouflage the model updates and launch effective
attacks. Their proposed attack also works on other well-known optimizers such
as AdaGrad and RMSProp. Baruch et al. [133] showed that the core part of
gradient descent algorithms is the direction of the descent. Specifically, for
gradient descent algorithms, to guarantee the descent of the loss, the inner
product between the ground-truth gradient and the robust aggregated gradient
must be non-negative:

$\begin{split}\langle\Delta\bm{w},\Delta\bm{w}^{\prime}\rangle\geq
0,\end{split}$ (3)

where $\langle\cdot,\cdot\rangle$ is the inner product operation,
$\Delta\bm{w}$ is the optimal gradient, and $\Delta\bm{w}^{\prime}=$
aggregate$(\Delta\bm{w}_{1},\cdots,\Delta\bm{w}_{n})$ is the aggregated
gradient with aggregate$(\cdot)$ being arbitrary aggregation function. To make
the aggregation fail, they proposed an “inner product manipulation attack”
that can make the inner product between the ground-truth gradient and the
robust aggregated gradient negative. To do this, each Byzantine participant
uploaded the negative of the average benign gradients. They showed the
proposed attack can successfully bypass Coordinate-wise Median [23] and Krum
[22]. Xie et al. [70] claimed that by consistently applying small changes to
many parameters, a Byzantine participant can perturb the model’s convergence.
First, they used the local data of Byzantine participants to estimate the mean
and standard deviation of the distribution. Then, they analyzed the range in
which changes to the parameters will not be detected by the defense, and upon
choosing the maxima of this range the convergence is averted.

###  V-B Targeted Attacks

In targeted poisoning attacks, the learnt model outputs the target label
specified by the adversary for particular testing examples, e.g., predicting
spams as non-spams, and predicting attacker-desired labels for testing
examples with a particular Trojan trigger (backdoor/trojan attacks). However,
the testing error for other testing examples is unaffected. Generally,
targeted attacks is more difficult to conduct than untargeted attacks as the
attacker has a specific goal to achieve.

One common example of targeted poisoning attack is the label-flipping attack
[124, 37]. The labels of honest training examples of one class are flipped to
another class while the features of the data are kept unchanged. For example,
the malicious participants in the system can poison their dataset by flipping
all 1s into 7s. A successful attack produces a model that is unable to
correctly classify 1s and incorrectly predicts them to be 7s.

Another realistic targeted poisoning attack is backdoor poisoning attack, in
which an adversary can modify individual features or small regions of the
original training dataset to implant a backdoor trigger into the model. The
model will behave normally on clean data, yet will constantly predict a target
class whenever the trigger (e.g., a stamp on an image) appears. For instance,
a backdoor attack can cause the FL model to reach 100% accuracy on the
backdoor task, e.g., to control an image classifier to assign an attacker-
chosen label to images with certain features in an image-classification task,
or a next-word predictor completes certain sentences with an attacker-chosen
word in a word-prediction task [35].

Backdoor attacks can be further divided into two categories: dirty-label
attacks [129, 128, 53, 134] and clean-label attacks [135, 136, 130, 137, 53,
66]. Clean-label attacks assume that the adversary cannot change the label of
any training data as there is a process by which data are certified as
belonging to the correct class and the poisoning of data samples has to be
imperceptible. In contrast, in dirty-label poisoning, the adversary can
introduce a number of data samples that are expected to be misclassified by
the model with the desired target label into the training data. Clean-label
attacks are arguably stealthier as they do not change the labels.

The targeted poisoning attack in FL can be carried out by any FL participant
or via collusion on either the data or the gradients. Bhagoji et al. [36]
demonstrated a single, non-colluding malicious participant can cause the model
to misclassify a set of chosen inputs with high confidence. Bagdasaryan et al.
[35] pointed out that the poisoned updates can be generated by training the
local model on backdoored local training data, and even a single-shot attack
may be enough to inject a backdoor into the global model. Xie et al. [34]
demonstrated that a global trigger pattern can be decomposed into separate
local patterns and embedded into the training set of colluding adversarial
participants respectively. The impact on the FL model depends on the extent to
which the backdoor participants engage in the attacks, and the amount of
training data being poisoned. A recent work shows that poisoning edge-case
(low probability) training samples are more effective [33].

Lastly, we remark that most of the previous research on poisoning attacks
focus on Byzantine or backdoor attackers. A system that allows participants to
join and leave is susceptible to Sybil attacks [138], in which an attacker
gains influence by joining a system to inject $c$ fake participants into the
FL system or compromise $c$ benign participants [37]. Sybil attacks can be
launched in both the untargeted and targeted manner. For example, targeted
poisoning can be conducted by sybil clones, who contribute updates towards a
specific poisoning objective [37]. Concretely, [37] considered two types of
targeted attacks by sybil clones: label-flipping attacks and backdoor attacks.

TABLE VI: The state-of-the-art defenses against federated learning poisoning.
$n$ is number of participants. Note that some defenses have no theoretic
breaking point.

Defense | Technique | IID Data | Non-IID Data | Breaking Point | Targeted Poisoning | Untargeted Poisoning
---|---|---|---|---|---|---
AUROR [139] | Clustering | $\checkmark$ | $\times$ | NA | $\times$ | $\checkmark$
Krum/Multi-Krum [22] | Euclidean distance | $\checkmark$ | $\times$ | $(n-2)/2n$ | $\times$ | $\checkmark$
Coordinate-wise Median [23] | Coordinate-wise median | $\checkmark$ | $\times$ | 1/2 | $\times$ | $\checkmark$
Bulyan [140] | Krum + trimmed median | $\checkmark$ | $\times$ | $(n-3)/4n$ | $\times$ | $\checkmark$
RFA [58] | Geometric median | $\checkmark$ | $\times$ | NA | $\times$ | $\checkmark$
FoolsGold [37] | Contribution similarity | $\checkmark$ | $\checkmark$ | NA | $\checkmark$ | $\times$
Sun et al. [38] | Norm-bounding and DP | $\checkmark$ | $\checkmark$ | NA | $\checkmark$ |  $\checkmark$ [141]
Wu et al. [142] | Pruning | $\checkmark$ | $\checkmark$ | NA | $\checkmark$ | $\times$
CRFL [143] | Clipping and smoothing | $\checkmark$ | $\checkmark$ | NA | $\checkmark$ | $\times$

##  VI Defenses against Poisoning Attacks

Robustness to poisoning attacks is a desirable property in FL. To address
poisoning attacks, many robust aggregation schemes are proposed in the
literature. Known defenses to poisoning attacks in a centralized setting, such
as robust losses [144] and anomaly detection [123], assume control of the
participants or explicit observation of the training data. Neither of these
assumptions are applicable to FL in which the server only observes model
parameters/updates sent as part of the iterative ML algorithm [37]. We
summarize the robustness-focused FL defenses against untargeted and targeted
attacks as follows.

###  VI-A Defenses against Untargeted Attacks

For Byzantine-resilient aggregation, an algorithm is Byzantine fault tolerant
[22] if its convergence is robust even when a large portion of participants
are adversarial. Below, we list several representative attempts that try to
defend against the untargeted Byzantine attacks.

Shen et al. [139] introduced a statistical mechanism called AUROR to detect
the malicious users while generating an accurate model. AUROR is based on the
observation that indicative features (most important model features) from the
majority of honest users will exhibit a similar distribution while those from
malicious users will exhibit an anomalous distribution. It then uses k-means
to cluster participants’ updates across training rounds and discards the
outliers, i.e., contributions from small clusters that exceed a threshold
distance are removed. The accuracy of a model trained using AUROR drops by
only 3% even when 30% of all the users are adversarial.

Blanchard et al. proposed Krum [22], in which, the top $f$ contributions to
the model that are furthest from the mean participant contribution are removed
from the aggregation. Krum uses the Euclidean distance to determine which
gradient contributions should be removed, and can theoretically withstand
poisoning attacks of up to 33% adversaries in the participant pool, i.e.,
given $n$ agents of which $f$ are Byzantine, Krum requires that $n\geq 2f+3$.
At any time step $t$, updates $\\{\delta_{1}^{t},\cdots,\delta_{n}^{t}\\}$ are
received at the server. For each $\delta_{i}^{t}$, the $n-f-2$ closest (in
terms of $L_{2}$ norm) other updates are chosen to form a set $C_{i}$, and
their distances are added up to give a score
$S(\delta_{i}^{t})={\textstyle\sum}_{\delta\in
C_{i}}\|\delta_{i}^{t}-\delta\|$. Krum then chooses
$\delta_{krum}=\delta_{i}^{t}$ with the lowest score to update the global
parameter $w_{i}^{t+1}=w_{i}^{t}+\delta_{krum}$. Krum is resistant to attacks
by omniscient adversaries – aware of a good estimate of the gradient – who
send the opposite vector multiplied by a large factor. It is also resistant to
attacks by adversaries who send random vectors drawn from a Gaussian
distribution (the larger the variance of the distribution, the stronger the
attack). Multi-Krum is a variant of Krum, which intuitively interpolates
between Krum and averaging, thereby combining the resilience properties of
Krum with the convergence speed of averaging. Essentially, Krum filters
outliers based on the entire update vector, but does not filter coordinate-
wise outliers.

To address this issue, Yin et al. [23] proposed two robust distributed
gradient descent algorithms, one based on coordinate-wise median, and the
other based on the coordinate-wise trimmed mean. Unfortunately, median-based
rules can incur a prohibitive computational overhead in large-scale settings
[145]. Mhamdi et al. [140] proposed a meta-aggregation rule called Bulyan, a
two-step meta-aggregation algorithm based on the Krum and trimmed median,
which filters malicious updates followed by computing the trimmed median of
the remaining updates. Median and geometric-median based robust aggregation
rules are also extensively explored in [146, 147]. Pillutla et al. [58]
proposed a robust aggregation approach called RFA by replacing the weighted
arithmetic mean with an approximate geometric median, so as to reduce the
impact of the contaminated updates. Unfortunately, RFA can only handle a few
types of poisoning attackers, but not applicable to Byzantine attacks.

In spite of their robustness guarantees, recent inspections revealed that
previous Byzantine-robust FL mechanisms are also quite brittle and can be
easily circumvented. Bhagoji et al. [36] showed that targeted model poisoning
of deep neural networks is effective even against the Byzantine-robust
aggregation rules such as Krum and coordinate-wise median. [133, 70] showed
that while the Byzantine-robust aggregation rules may ensure that the
influence of the Byzantine workers in any single round is limited, the
attackers can couple their attacks across the rounds, moving weights
significantly away from the desired direction and thus achieve the goal of
lowering the model quality. Xu et al. [148] demonstrated that Multi-Krum is
not robust against the untargeted poisoning. This is because Multi-Krum is
based on the distance between each gradient vector and the mean vector, while
the mean vector is not robust against untargeted poisoning. Fang et al. [67]
showed that aggregation rules (e.g., Krum [22], Bulyan [140], trimmed mean
[23], coordinate-wise median [23], and other median-based aggregators [57])
that were claimed to be robust against Byzantine failures are not effective in
practice against optimized local model poisoning attacks that carefully craft
local models on the compromised participants such that the aggregated global
model deviates the most towards the inverse of the direction along which the
global model would change when there are no attacks. All these highlight the
need for more effective defenses against Byzantine attackers in FL.

Other works investigate Byzantine robustness from different lens. Chen et al.
[145] presented DRACO, a framework for robust distributed training via
algorithmic redundancy. DRACO is robust to arbitrarily malicious computing
nodes, while being orders of magnitude faster than state-of-the-art robust
distributed systems. However, DRACO assumes that each participant can access
other participants’ data, limiting its practicability in FL. Su et al. [72]
proposed to robustly aggregate the gradients computed by the Byzantine
participants based on the filtering procedure proposed by Steinhardt et al.
[149]. Bernstein et al. [39] proposed SIGNSGD, which is combined with majority
vote to enable participants to upload element-wise signs of their gradients to
defend against three types of half “blind” Byzantine adversaries: (i)
adversaries that arbitrarily rescale their stochastic gradient estimate; (ii)
adversaries that randomise the sign of each coordinate of the stochastic
gradient; (iii) adversaries that invert their stochastic gradient estimate.

###  VI-B Defenses against Targeted Attacks

Existing defenses against the targeted backdoor attacks can be categorized
into two types: detection methods and erasing methods [150]. Detection methods
exploit activation statistics or model properties to determine whether a model
is backdoored [151, 152], or whether a training/test example is a backdoor
example [47].

There are a number of detection algorithms that are designed to detect which
inputs contain backdoor, and which parts of the model (its activation
functions specifically) are responsible for triggering the adversarial
behavior of the model, in order to remove the backdoor [48, 128, 153, 42, 47].
These algorithms rely on the statistical difference between the latent
representations of backdoor-enabled and clean (benign) inputs in the poisoned
model. These backdoor detection algorithms can however be bypassed by
maximizing the latent indistinguishability of backdoor-enabled adversarial
inputs and clean inputs [154].

While detection can help identify potential risks, the backdoored model still
needs to be purified since the potential impact of backdoor triggers remains
uncleared in the backdoored models. The erasing methods take a step further
and aim to purify the adverse impacts on models caused by the backdoor
triggers. The current state-of-the-art erasing methods are Mode Connectivity
Repair (MCR) [155] and Neural Attention Distillation (NAD) [54]. MCR mitigates
the backdoors by selecting a robust model in the path of loss landscape, while
NAD leverages knowledge distillation to erase triggers. Other previous
methods, including finetuning, denoising, and fine-pruning [153], have been
shown to be insufficient against the latest attacks [156, 53]. Another more
recent work called Anti-Backdoor Learning (ABL) [55] aims to train clean
models given backdoor-poisoned data. They frame the overall learning process
as a dual-task of learning the clean and the backdoor portions of data. From
this view, they identify two inherent characteristics of backdoor attacks as
their weaknesses: 1) the models learn backdoored data much faster than
learning with clean data, and the stronger the attack the faster the model
converges on backdoored data; 2) the backdoor task is tied to a specific class
(the backdoor target class). Based on these two weaknesses, ABL introduces a
two-stage gradient ascent mechanism for standard training to 1) help isolate
backdoor examples at an early training stage, and 2) break the correlation
between backdoor examples and the target class at a later training stage.
Extensive experiments on multiple benchmark datasets against 10 state-of-the-
art attacks empirically show that ABL can automatically prevent backdoor
attacks during training, without degrading the main performance.

Despite the promising backdoor defense results in the centralized setting, it
is still unclear whether these defenses can be smoothly adapted to FL setting,
especially in the non-iid setting. For backdoor defense in FL, Sun et al. [38]
showed that clipping the norm of model updates and adding Gaussian noise can
mitigate backdoor attacks that are based on the model replacement paradigm.
Andreina et al. [157] incorporated an additional validation phase in each
round of FL to detect backdoor. However, None of these provides certified
robustness guarantees. Certified robustness for FL against backdoor attacks
remain largely unexplored. Xie et al. [143] provided the first general
framework called _Certifiably Robust Federated Learning_ (CRFL), to train
certifiably robust FL models against backdoors.

To defend against the targeted poisoning attack by sybil clones, Fung et al.
[37] exploited the characteristic behavior that sybils are more similar to
each other than the similarity observed amongst the honest clients, and
proposed FoolsGold: a new defense scheme against FL Sybil attacks by adapting
the learning rate of participants based on contribution similarity. Note that
FoolsGold does not bound the expected number of attackers by assuming that
attackers can spawn a large number of Sybils, rendering assumptions about
proportions of honest participants unrealistic [22]. Additionally, FoolsGold
requires no auxiliary information beyond the learning process, and makes fewer
assumptions about participants and their data. The robustness of FoolsGold
holds for different distributions of participant data, varying poisoning
targets, and various Sybil strategies, and can be applied successfully on both
FedSGD and FedAvg.

We list the most representative defenses against poisoning attacks against FL
in Table VI. Some robust aggregation algorithms have breaking points, i.e.,
the fraction of malicious participants, robustness guarantees cannot be
provided if the fraction of malicious participants is larger than the breaking
point.

Remark. Note that both the untargeted and targeted poisoning attacks are less
effective in settings with infrequent participation like H2C [35]. Moreover,
under practical production FL environments, Shejwalkar et al. [141] have
showed that FL, even without any defenses, is highly robust in practice. For
production cross-device FL (H2C), which contains thousands to billions of
clients, poisoning attacks have no impact on existing robust FL algorithms
even with impractically high percentages of compromised clients. For
production cross-silo FL (H2B), which contains up to hundred clients, data
poisoning attacks are completely ineffective; model poisoning attacks are
unlikely to play a major risk when the clients involved are bound by contract
and their software stacks professionally maintained (e.g., in banks,
hospitals). Some exceptional cross-silo scenarios are most likely with a
strong incentive (e.g., financial) causing multiple parties to be willing to
risk breach of contract by colluding or for one party to hack thereby risking
criminal liability. Therefore, we conclude that these poisoning attacks are
more likely to happen in some exceptional H2B scenarios.

##  VII Discussions and Promising Directions

There are still potential vulnerabilities which need to be addressed in order
to improve the privacy and robustness of FL systems. Moreover, there are
multiple design goals that are equally important with privacy and robustness,
thus need to be considered simultaneously in FL. In this section, we outline
research directions which we believe are promising.

Curse of Dimensionality: Large models, with high dimensional parameter
vectors, are particularly susceptible to privacy and security attacks [158].
Most FL algorithms require overwriting the local model parameters with the
global model. This makes them susceptible to poisoning attacks, as the
adversary can make small but damaging changes in the high-dimensional models
without being detected. Almost all of the well-designed Byzantine-robust
aggregators [22, 23, 58] still suffer from the curse of dimensionality.
Specifically, the estimation error scales up with the size of the model in a
square-root manner. Thus, sharing model parameters may not be a strong design
choice in FL, it opens all the internal state of the model to inference
attacks, and maximizes the model’s malleability by poisoning attacks. The
large number of hyperparameters might also adversely affect communication and
accuracy, though the follow up work [159] tried adaptive gradient clipping
strategies to help alleviate this issue. To address these fundamental
shortcomings of FL, it is worthwhile to explore whether sharing gradients is
essential. Instead, sharing less sensitive information (e.g.,SIGNSGD [160]) or
only sharing model predictions [158, 25, 87] in a black-box manner may result
in more robust privacy protection in FL.

Rethinking Current Privacy Attacks: There are several inherent weaknesses in
current attacks that may limit their applicability in FL [161]. For example,
GAN attack assumes that the entire training corpus for a given class comes
from a single participant, and only in the special case where all class
members are similar, GAN-constructed representatives are similar to the
training data [76]. These assumptions may be less practical in FL. For DLG
[29] and iDLG [32], both works: (1) adopt a second-order optimization method
called L-BFGS, which is more computationally expensive compared with first-
order optimization methods; (2) are only applicable to gradients computed on
mini-batches of data, i.e., at most B=8 in DLG, and B=1 in iDLG, which is not
the real case for FL, in which gradient is normally shared after at least 1
epoch of local training; (3) used untrained model, neglecting gradients over
multiple communication rounds. Attacking FL system in a more efficient manner
and under more practical settings remains largely unexplored.

Rethinking Current Defenses: FL with secure aggregation for the purpose of
privacy is more susceptible to poisoning attacks as the individual updates
cannot be inspected. Similarly, it is still unclear if adversarial training,
one state-of-the-art defense approach against adversarial attacks in
conventional ML [162, 163, 164], can be adapted to FL, as adversarial training
was developed primarily for IID data and remains unclear for its performance
in non-IID settings. Moreover, adversarial training is computationally
expensive and may hurt the performance [165], which may not be feasible for
H2C scenario. In terms of differential privacy (DP) based methods [7, 166, 84,
167, 168, 169], record-level DP bounds the success of membership inference,
but does not prevent property inference applied to a group of training records
[28]. Participant-level DP, on the other hand, is geared to work with
thousands of users for training to converge and achieving an acceptable trade-
off between privacy and accuracy [7]. The FL model fails to converge with a
small number of participants, making it unsuitable for H2B scenarios.
Furthermore, DP may hurt the accuracy of the learned model [170], which may
not be appealing to the industry. Further work is needed to investigate if
participant-level DP can protect FL systems with fewer participants.

Optimizing Defense Mechanism Deployment: When deploying defense mechanisms to
check if any adversary is attacking the FL system, the FL server will need
additional computational cost. In addition, different types of defense
mechanisms may have different effectiveness against different attacks, and
incur different costs. It is important to study how to optimize the timing of
deploying the defense mechanisms or the announcement of deterrence measures.
Game theoretic research holds promise in addressing this challenge.

Test-phase Privacy in FL: This survey mainly focuses on the training phase
attacks and defenses in FL, considering the more attack possibilities opened
by the distributed training property of FL systems. In fact, FL is also
vulnerable to both privacy and robustness attacks during test/inference phase
by the users of the final FL model when it is deployed as a service.

In terms of the privacy vulnerability, the trained global model may reveal
sensitive information from model predictions when deployed as a service,
causing privacy leakage. In such a setting, an adversary does not have direct
access to the model parameters, but may be able to view input-output pairs.
Previous researches have shown a series of privacy leakage given only black-
box access to the trained models, such as (1) model stealing attacks in which
model parameters can be reconstructed by an adversary who only has access to
an inference/prediction API based on those parameters [171, 172, 173, 174];
(2) membership inference attacks which aim to determine if a particular record
was used to train the model [78]. FL models face the similar dilemma during
model deployment for testing purpose. The development of effective defenses
against privacy leakage during model deployment calls for further
investigations.

Test-phase Robustness in FL: In terms of the robustness vulnerability, recent
studies [175, 176, 177] have shown that FL is also vulnerable to well-crafted
adversarial examples. During inference time, the attackers can add a very
small perturbation to the test data, making the test data almost
indistinguishable from natural data and yet classified incorrectly by the
global model. For federated robustness against adversarial examples, [175,
176] proposed to apply adversarial training (AT) to FL, i.e, federated
adversarial training (FAT), in order to achieve adversarial robustness in FL.
[175] noticed that conducting AT on all participants leads to divergence of
the model. To solve this problem, they conducted AT on only a proportion of
participants for better convergence. Another recent work [176] considered
hardware heterogeneity in FL, i.e., only limited users can afford AT. Hence,
they conduct AT on only a proportion of participants that have powerful
computation resources while conducting standard training on the rest of the
participants. [177] investigated the impact of communication rounds in FAT and
proposed a dynamic adversarial training. The training of all the above FAT
works are unstable, which potentially hurts the convergence and performance.
Moreover, adversarial training typically requires significant computation and
longer time to converge, and it is unclear how it performs in non-IID
settings. How to speed up adversarial training in FL and the investigation of
its applicability in non-IID settings may be required in the future. Overall,
there exist difficulties in applying adversarial training to the federated
setting. This motivates future works to explore more effective approaches to
maintain both natural accuracy and robust accuracy in FL.

In addition to the adversarial examples, recent works [172, 173] have
validated that the API services (the victim/target model) can be easily stolen
and are vulnerable to adversarial example transferability attack. It would be
interesting to explore whether the collaboratively built global model in FL is
also facing the similar problem, and how to effectively claim the ownership of
the trained model [178].

Overall, it would be of much importance towards realizing trustworthy FL by
defending against both training-phase and test-phase attacks.

Relationship with GDPR: GDPR 111https://gdpr-info.eu defines 6-core principles
as rational guidelines for service providers to manage personal data,
including: (1) Lawfulness, fairness and transparency; (2) Purpose limitation;
(3) Data minimisation; (4) Accuracy; (5) Storage limitation; (6) Integrity and
confidentiality (security). GDPR also requires Data Controllers to provide the
following rights for Data Subjects if capable (The GDPR Articles 12–23): (1)
Right to be informed, (2) Right of access, (3) Right to rectification, (4)
Right to erasure (Right to be forgotten), (5) Right to restrict processing,
(6) Right to data portability, (7) Right to object, and (8) Rights in relation
to automated decision making and profiling. Although FL has emerged as a
prospective solution that facilitates distributed collaborative learning
without disclosing original training data, unfortunately, FL is not naturally
compliant with the GDPR [179], as pointed out by a recent survey [179] that
has dedicated to surveying the relationship between FL and GDPR requirements.
For example, secure aggregation mechanism in FL amplifies the lack of
transparency and fairness in FL systems, thus fails to fully comply with the
GDPR requirements of fairness and transparency; malicious participants in FL
may conduct either data or model poisoning attack for an unauthorised purpose,
local ML model parameters obtained from participants is no longer minimal for
the original purpose. These possible attacks, which lead to non-compliance
with the GDPR, should be addressed. Henceforth, it is worthwhile to explore
approaches to empower FL-based systems to follow the GDPR regulatory
guidelines, thus fully comply with the GDPR.

Threats and Protections of VFL and FTL: This survey mainly focuses on the
threats to HFL, there are some recent exploratory efforts on threats and
protections of VFL and FTL.

For VFL, Secureboost [180] considered user privacy and data confidentiality in
VFL, and presented an approach to train a high-quality tree boosting model
collaboratively. A recent work called FederBoost [181] pointed out that
Secureboost is expensive since it requires cryptoraphic computation and
communication for each possible split, thus they proposed a vertical
FederBoost which does not require any cryptographic operation. Another recent
work [182] uncovered the risk of _catastrophic data leakage in vertical
federated learning_ (CAFE) through a novel algorithm that can perform large-
batch data leakage with high data recovery quality and theoretical guarantees.
They empirically demonstrated that CAFE can recover large-scale private data
from the shared aggregated gradients in VFL settings, overcoming the batch
limitation problem in current data leakage attacks.

For FTL, [183] proposed an end-to-end privacy-preserving multi-party learning
approach with two variants based on homomorphic encryption and secret sharing
techniques, respectively, to build a heterogeneous federated transfer learning
(HFTL) framework. [21] adopted two secure approaches, namely, homomorphic
encryption (HE) and secret sharing for preserving privacy. The HE approach is
simple but computationally expensive. By contrast, the secret sharing approach
offers the following advantages: (i) there is no accuracy loss, (ii)
computation is much faster than HE approach. The major drawback of the secret
sharing approach is that one has to offline generate and store many triplets
before online computation.

Overall, there is still a large space for VFL and FTL. It is worth further
investigation as for whether existing threats in HFL are all valid in VFL and
FTL, or if there are new threats and countermeasures in VFL and FTL.

Vulnerabilities to Free-riding Participants: In FL systems, there may exist
free-riders, who aim to benefit from the global model, but do not want to
contribute any useful information, thus compromising collaborative fairness
[167, 168, 184]. The main incentives for free-riders include: (1) the
participant dose not have any data to train the local model; (2) the
participant is too concerned about its privacy thus chooses to release fake
updates; (3) the participant does not want to consume or does not have any
local computation power to train the local model. In the current FL paradigm
[6], all participants receive the same federated model at the end of the
collaborative training, regardless of their individual contributions. This
makes the paradigm vulnerable to free-riding participants [167, 168, 185, 148,
186]. How to prevent free-riding remains an open challenge.

More Possibilities in FL with Heterogeneous Architectures: Most privacy and
robustness researches are focused on FL with homogeneous architectures. It
remains unclear whether existing attacks, privacy-preserving techniques and
defense mechanisms can be adapted to FL with heterogeneous architectures. It
is valuable future work to explore similar types of attacks and defenses in
heterogeneous FL.

Decentralized FL: Decentralized FL is an emerging research area, where there
is no single central server in the system [7, 3, 167, 168]. Decentralized FL
is potentially more useful for H2B scenarios where the business participants
do not trust any third party. In this paradigm, each participant could be
elected as a server in a round robin manner. The recent emerging swarm
learning [187] can be deemed as a decentralized FL framework, which unites
edge computing, blockchain-based peer-to-peer networking and coordination
while maintaining confidentiality without the need for a central coordinator.
It is interesting to investigate whether existing threats to server-based FL
still apply in decentralized FL.

Efficient FL with Single Round Communication: In addition to privacy and
robustness, communication cost is another major concern that may hinder the
practical deployment of FL. One-shot FL has recently emerged as a promising
approach for communication efficiency. It allows the central server to learn a
model in a single communication round. Despite the low communication cost,
existing one-shot FL methods are mostly impractical or face inherent
limitations, e.g., a public dataset is required, participants’ models are
homogeneous, additional data/model information needs to be uploaded,
unsatisfactory performance [188, 189, 190]. A recent work proposed a more
practical data-free approach named FedSyn for one-shot FL framework with
heterogeneity [191]. FedSyn is the first method that can be practically
applied to various real-world applications due to the following benefits: (1)
FedSyn requires no additional information (except the model parameters) to be
transferred between participants and the server; (2) FedSyn does not require
any auxiliary dataset for training; (3) FedSyn is the first to consider both
model and statistical heterogeneities in FL, i.e., the participants’ data are
non-iid and different participants may have different model architectures.
Other alternative one-shot FL approaches with practical assumptions are
worthwhile to explore, considering the alluring communication efficiency and
less privacy and robustness attack surfaces exposed in one-shot FL.

Achieving Multiple Objectives Simultaneously: There are no existing works that
can satisfy multiple goals simultaneously: (1) fast algorithmic convergence;
(2) good generalisation performance; (3) communication efficiency; (4) fault
tolerance; (5) privacy preservation; and (6) robustness to targeted,
untargeted poisoning attacks, and free-riders. Previous works have attempted
to solve multiple objectives at the same time. For example, [168, 167]
addressed collaborative fairness and privacy simultaneously; [148, 192]
proposed a _Robust and Fair Federated Learning_ (RFFL) framework to address
both collaborative fairness and Byzantine robustness. However, it is important
to highlight that there is an inherent conflict between privacy and
robustness: defending against robustness attacks usually require complete
control of the training process or access to the training data [193, 139, 22,
23, 37, 194], which goes against the privacy requirements of FL. Although
using encryption or DP-based techniques can provide provably privacy
preservation, they are not robust to poisoning attacks and may produce models
with undesirably poor privacy-utility trade-offs. Agarwal et al. [30] combined
differential privacy with model compression techniques to reduce communication
cost and obtain privacy benefits simultaneously. It remains largely unexplored
and there exist large gaps as for how to simultaneously achieve all the above
six objectives.

##  VIII Conclusions

Although federated learning is still in its infancy, it will continue to
thrive and will be an active and important research area in the foreseeable
future. As FL evolves, so will the privacy and robustness threats to FL. It is
of vital importance to provide a broad overview of current attacks and
defenses on FL so that future FL system designers are well aware of the
potential vulnerabilities in the current designs, and help them clear
roadblocks towards the real-world deployment of FL. This survey serves as a
concise and accessible overview of this topic, and it would greatly help our
understanding of the privacy and robustness attack and defense landscape in
FL. Global collaboration on FL is emerging through a number of workshops in
leading AI conferences222http://www.federated-learning.org/. The ultimate goal
of developing a general purpose FL defense mechanism that can be robust
against various attacks without degrading model performance will require
interdisciplinary effort from the wider research community.

## References

  * [1] H. Li, K. Ota, and M. Dong, “Learning iot in edge: Deep learning for the internet of things with edge computing,” _IEEE Network_ , vol. 32, no. 1, pp. 96–101, 2018.
  * [2] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in _CCS_ , 2016, pp. 308–318.
  * [3] Q. Yang, Y. Liu, Y. Cheng, Y. Kang, T. Chen, and H. Yu, _Federated Learning_. Morgan & Claypool Publishers, 2019.
  * [4] H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” _CoRR, arXiv:1602.05629_ , 2016.
  * [5] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _CCS_ , 2017, pp. 1175–1191.
  * [6] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_ , 2017, pp. 1273–1282.
  * [7] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in _ICLR_ , 2018.
  * [8] Y. Liu, A. Huang, Y. Luo, H. Huang, Y. Liu, Y. Chen, L. Feng, T. Chen, H. Yu, and Q. Yang, “Fedvision: An online visual object detection platform powered by federated learning,” in _IAAI_ , 2020.
  * [9] S. Hardy, W. Henecka, H. Ivey-Law, R. Nock, G. Patrini, G. Smith, and B. Thorne, “Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption,” _CoRR, arXiv:1711.10677_ , 2017.
  * [10] C. Wu, F. Wu, R. Liu, L. Lyu, Y. Huang, and X. Xie, “Fedkd: Communication efficient federated learning via knowledge distillation,” _arXiv preprint arXiv:2108.13323_ , 2021.
  * [11] C. Wu, F. Wu, L. Lyu, T. Di, Y. Huang, and X. Xie, “Fedctr: Federated native ad ctr prediction with multi-platform user behavior data,” _arXiv preprint arXiv:2007.12135_ , 2020.
  * [12] J. Cui, C. Chen, L. Lyu, C. Yang, and W. Li, “Exploiting data sparsity in secure cross-platform social recommendation,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [13] J. Li, L. Lyu, X. Liu, X. Zhang, and X. Lv, “Fleam: A federated learning empowered architecture to mitigate ddos in industrial iot,” _IEEE Transactions on Industrial Informatics_ , 2021.
  * [14] C. Wu, F. Wu, Y. Cao, L. Lyu, Y. Huang, and X. Xie, “Fedgnn: Federated graph neural network for privacy-preserving recommendation,” _arXiv preprint arXiv:2102.04925_ , 2021.
  * [15] J. Zhou, C. Chen, L. Zheng, X. Zheng, B. Wu, L. Lyu, Z. Liu, and L. Wang, “Privacy-preserving graph neural network for node classification,” _arXiv e-prints_ , pp. arXiv–2005, 2020.
  * [16] X. Ni, X. Xu, L. Lyu, C. Meng, and W. Wang, “A vertical federated learning framework for graph convolutional network,” _arXiv preprint arXiv:2106.11593_ , 2021.
  * [17] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 10, no. 2, pp. 1–19, 2019.
  * [18] M. Kantarcioglu and C. Clifton, “Privacy-preserving distributed mining of association rules on horizontally partitioned data,” _IEEE Transactions on Knowledge & Data Engineering_, no. 9, pp. 1026–1037, 2004.
  * [19] J. Vaidya and C. Clifton, “Privacy preserving association rule mining in vertically partitioned data,” in _KDD_ , 2002, pp. 639–644.
  * [20] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2009.
  * [21] Y. Liu, Y. Kang, C. Xing, T. Chen, and Q. Yang, “A secure federated transfer learning framework,” _IEEE Intelligent Systems_ , vol. 35, no. 4, pp. 70–82, 2020.
  * [22] P. Blanchard, R. Guerraoui, J. Stainer _et al._ , “Machine learning with adversaries: Byzantine tolerant gradient descent,” in _NeurIPS_ , 2017, pp. 119–129.
  * [23] D. Yin, Y. Chen, K. Ramchandran, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” _CoRR, arXiv:1803.01498_ , 2018.
  * [24] D. Gao, Y. Liu, A. Huang, C. Ju, H. Yu, and Q. Yang, “Privacy-preserving heterogeneous federated transfer learning,” in _IEEE BigData_ , 2019.
  * [25] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” _arXiv preprint arXiv:1910.03581_ , 2019.
  * [26] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” _arXiv preprint arXiv:1811.11479_ , 2018.
  * [27] A. Bhowmick, J. Duchi, J. Freudiger, G. Kapoor, and R. Rogers, “Protection against reconstruction and its applications in private federated learning,” _CoRR, arXiv:1812.00984_ , 2018.
  * [28] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _SP_ , 2019, pp. 691–706.
  * [29] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in _NeurIPS_ , 2019, pp. 14 747–14 756.
  * [30] N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan, “cpsgd: Communication-efficient and differentially-private distributed sgd,” in _NeurIPS_ , 2018, pp. 7564–7575.
  * [31] Y. Aono, T. Hayashi, L. Wang, S. Moriai _et al._ , “Privacy-preserving deep learning via additively homomorphic encryption,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 5, pp. 1333–1345, 2018.
  * [32] B. Zhao, K. R. Mopuri, and H. Bilen, “idlg: Improved deep leakage from gradients,” _CoRR, arXiv:2001.02610_ , 2020.
  * [33] H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, and D. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” _NeurIPS_ , 2020.
  * [34] C. Xie, K. Huang, P. Chen, and B. Li, “DBA: distributed backdoor attacks against federated learning,” in _8th International Conference on Learning Representations_ , 2020.
  * [35] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” _CoRR, arXiv:1807.00459_ , 2018.
  * [36] A. N. Bhagoji, S. Chakraborty, P. Mittal, and S. Calo, “Analyzing federated learning through an adversarial lens,” _CoRR, arXiv:1811.12470_ , 2018.
  * [37] C. Fung, C. J. Yoon, and I. Beschastnikh, “The limitations of federated learning in sybil settings,” in _23rd International Symposium on Research in Attacks, Intrusions and Defenses ( $\\{$RAID$\\}$ 2020)_, 2020, pp. 301–316.
  * [38] Z. Sun, P. Kairouz, A. T. Suresh, and H. B. McMahan, “Can you really backdoor federated learning?” _arXiv preprint arXiv:1911.07963_ , 2019.
  * [39] J. Bernstein, J. Zhao, K. Azizzadenesheli, and A. Anandkumar, “signSGD with majority vote is communication efficient and byzantine fault tolerant,” in _In Seventh International Conference on Learning Representations (ICLR)_ , 2019.
  * [40] S. Truex, N. Baracaldo, A. Anwar, T. Steinke, H. Ludwig, and R. Zhang, “A hybrid approach to privacy-preserving federated learning,” _CoRR, arXiv:1812.03224_ , 2018.
  * [41] L. Lyu, Y. W. Law, K. S. Ng, S. Xue, J. Zhao, M. Yang, and L. Liu, “Distributed privacy-preserving prediction,” in _International Conference on Systems, Man, and Cybernetics_ , 2020.
  * [42] Y. Liu, Y. Xie, and A. Srivastava, “Neural trojans,” in _ICCD_ , 2017, pp. 45–48.
  * [43] B. G. Doan, E. Abbasnejad, and D. C. Ranasinghe, “Februus: Input purification defense against trojan attacks on deep neural network systems,” in _arXiv: 1908.03369_ , 2019.
  * [44] S. Udeshi, S. Peng, G. Woo, L. Loh, L. Rawshan, and S. Chattopadhyay, “Model agnostic defence against backdoor attacks in machine learning,” _arXiv preprint arXiv:1908.02203_ , 2019.
  * [45] M. Villarreal-Vasquez and B. Bhargava, “Confoc: Content-focus protection against trojan attacks on neural networks,” _arXiv preprint arXiv:2007.00711_ , 2020.
  * [46] Y. Li, T. Zhai, B. Wu, Y. Jiang, Z. Li, and S. Xia, “Rethinking the trigger of backdoor attack,” _arXiv preprint arXiv:2004.04692_ , 2020.
  * [47] B. Tran, J. Li, and A. Madry, “Spectral signatures in backdoor attacks,” in _NeurIPS_ , 2018, pp. 8000–8010.
  * [48] B. Chen, W. Carvalho, N. Baracaldo, H. Ludwig, B. Edwards, T. Lee, I. Molloy, and B. Srivastava, “Detecting backdoor attacks on deep neural networks by activation clustering,” _CoRR, arXiv:1811.03728_ , 2018.
  * [49] D. Tang, X. Wang, H. Tang, and K. Zhang, “Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection,” _USENIX_ , 2021.
  * [50] E. Soremekun, S. Udeshi, S. Chattopadhyay, and A. Zeller, “Exposing backdoors in robust machine learning models,” _arXiv preprint arXiv:2003.00865_ , 2020\.
  * [51] A. Chan and Y.-S. Ong, “Poison as a cure: Detecting & neutralizing variable-sized backdoor attacks in deep neural networks,” _arXiv preprint arXiv:1911.08040_ , 2019.
  * [52] E. Chou, F. Tramer, and G. Pellegrino, “Sentinet: Detecting localized universal attack against deep learning systems,” _IEEE SPW_ , 2020.
  * [53] Y. Liu, X. Ma, J. Bailey, and F. Lu, “Reflection backdoor: A natural backdoor attack on deep neural networks,” in _ECCV_. Springer, 2020, pp. 182–199.
  * [54] Y. Li, X. Lyu, N. Koren, L. Lyu, B. Li, and X. Ma, “Neural attention distillation: Erasing backdoor triggers from deep neural networks,” _arXiv preprint arXiv:2101.05930_ , 2021.
  * [55] ——, “Anti-backdoor learning: Training clean models on poisoned data,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [56] G. Damaskinos, E. M. El Mhamdi, R. Guerraoui, A. H. A. Guirguis, and S. L. A. Rouault, “Aggregathor: Byzantine machine learning via robust gradient aggregation,” in _SysML_ , 2019.
  * [57] Y. Chen, L. Su, and J. Xu, “Distributed statistical machine learning in adversarial settings: Byzantine gradient descent,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 1, no. 2, p. 44, 2017\.
  * [58] K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” _arXiv preprint arXiv:1912.13445_ , 2019.
  * [59] M. S. Ozdayi, M. Kantarcioglu, and Y. R. Gel, “Defending against backdoors in federated learning with robust learning rate,” _arXiv preprint arXiv:2007.03767_ , 2020.
  * [60] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” _CoRR, arXiv:1908.07873_ , 2019\.
  * [61] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _CoRR, arXiv:1912.04977_ , 2019\.
  * [62] B. Biggio, B. Nelson, and P. Laskov, “Support vector machines under adversarial label noise,” in _ACML_ , 2011, pp. 97–112.
  * [63] C. Miao, Q. Li, H. Xiao, W. Jiang, M. Huai, and L. Su, “Towards data poisoning attacks in crowd sensing systems,” in _Proceedings of the Eighteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing_ , 2018, pp. 111–120.
  * [64] C. Miao, Q. Li, L. Su, M. Huai, W. Jiang, and J. Gao, “Attack under disguise: An intelligent data poisoning attack mechanism in crowdsourcing,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 13–22.
  * [65] H. Zhang, T. Zheng, J. Gao, C. Miao, L. Su, Y. Li, and K. Ren, “Data poisoning attack against knowledge graph embedding,” _arXiv preprint arXiv:1904.12052_ , 2019.
  * [66] G. Sun, Y. Cong, J. Dong, Q. Wang, L. Lyu, and J. Liu, “Data poisoning attacks on federated machine learning,” _IEEE Internet of Things Journal_ , 2021\.
  * [67] M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in _29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20)_, 2020, pp. 1605–1622.
  * [68] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar, “Can machine learning be secure?” in _ICCS_ , 2006, pp. 16–25.
  * [69] L. Lamport, R. Shostak, and M. Pease, “The byzantine generals problem,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 4, no. 3, pp. 382–401, 1982.
  * [70] C. Xie, O. Koyejo, and I. Gupta, “Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation,” in _UAI_. PMLR, 2020, pp. 261–270.
  * [71] L. T. Phong, Y. Aono, T. Hayashi, L. Wang, and S. Moriai, “Privacy-preserving deep learning via additively homomorphic encryption,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 5, pp. 1333–1345, 2018.
  * [72] L. Su and J. Xu, “Securing distributed machine learning in high dimensions,” _CoRR, arXiv:1804.10140_ , 2018.
  * [73] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in _SP_ , 2019, pp. 739–753.
  * [74] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in _ICLR_ , 2017.
  * [75] L. Lyu, “Privacy-preserving machine learning and data aggregation for internet of things,” Ph.D. dissertation, The University of Melbourne, 2018.
  * [76] B. Hitaj, G. Ateniese, and F. Pérez-Cruz, “Deep models under the gan: information leakage from collaborative deep learning,” in _CSS_ , 2017, pp. 603–618.
  * [77] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _CCS_ , 2015, pp. 1322–1333.
  * [78] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _SP_ , 2017, pp. 3–18.
  * [79] Y. Aono, T. Hayashi, L. Trieu Phong, and L. Wang, “Scalable and secure logistic regression via homomorphic encryption,” in _Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy_ , 2016, pp. 142–144.
  * [80] M. Kim, Y. Song, S. Wang, Y. Xia, and X. Jiang, “Secure logistic regression based on homomorphic encryption: Design and evaluation,” _JMIR medical informatics_ , vol. 6, no. 2, p. e19, 2018.
  * [81] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” _CoRR, arXiv:1712.07557_ , 2017.
  * [82] T. T. Nguyên, X. Xiao, Y. Yang, S. C. Hui, H. Shin, and J. Shin, “Collecting and analyzing data from smart device users with local differential privacy,” _arXiv preprint arXiv:1606.05053_ , 2016.
  * [83] N. Wang, X. Xiao, Y. Yang, J. Zhao, S. C. Hui, H. Shin, J. Shin, and G. Yu, “Collecting and analyzing multidimensional data with local differential privacy,” in _2019 IEEE 35th International Conference on Data Engineering (ICDE)_. IEEE, 2019, pp. 638–649.
  * [84] Y. Zhao, J. Zhao, M. Yang, T. Wang, N. Wang, L. Lyu, D. Niyato, and K. Y. Lam, “Local differential privacy based federated learning for internet of things,” _arXiv preprint arXiv:2004.08856_ , 2020.
  * [85] L. Sun, J. Qian, X. Chen, and P. S. Yu, “Ldp-fl: Practical private aggregation in federated learning with local differential privacy,” _arXiv preprint arXiv:2007.15789_ , 2020.
  * [86] S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “Ldp-fed: federated learning with local differential privacy,” in _Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking_ , 2020, pp. 61–66.
  * [87] L. Sun and L. Lyu, “Federated model distillation with noise-free differential privacy,” _arXiv preprint arXiv:2009.05537_ , 2020.
  * [88] L. Lyu, “Lightweight crypto-assisted distributed differential privacy for privacy-preserving distributed learning,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [89] P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-preserving machine learning,” in _SP_ , 2017, pp. 19–38.
  * [90] P. Paillier _et al._ , “Public-key cryptosystems based on composite degree residuosity classes,” in _Eurocrypt_ , vol. 99, 1999, pp. 223–238.
  * [91] T. ElGamal, “A public key cryptosystem and a signature scheme based on discrete logarithms,” _IEEE Transactions on Information Theory_ , vol. 31, no. 4, pp. 469–472, 1985.
  * [92] C. Gentry, “Fully homomorphic encryption using ideal lattices,” in _STOC_ , 2009, pp. 169–178.
  * [93] A. C. Yao, “Protocols for secure computations,” in _SFCS_ , 1982, pp. 160–164.
  * [94] D. Demmler, T. Schneider, and M. Zohner, “Aby-a framework for efficient mixed-protocol secure two-party computation.” in _NDSS_ , 2015.
  * [95] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in _Theory of cryptography conference_ , 2006, pp. 265–284.
  * [96] C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3–4, pp. 211–407, 2014.
  * [97] I. Damgård, V. Pastro, N. Smart, and S. Zakarias, “Multiparty computation from somewhat homomorphic encryption,” in _Annual Cryptology Conference_ , 2012, pp. 643–662.
  * [98] R. L. Rivest, A. Shamir, and L. Adleman, “A method for obtaining digital signatures and public-key cryptosystems,” _Communications of the ACM_ , vol. 21, no. 2, pp. 120–126, 1978.
  * [99] S. Goryczka and L. Xiong, “A comprehensive comparison of multiparty secure additions with differential privacy,” _IEEE transactions on dependable and secure computing_ , vol. 14, no. 5, pp. 463–477, 2015.
  * [100] M. S. Riazi, C. Weinert, O. Tkachenko, E. M. Songhori, T. Schneider, and F. Koushanfar, “Chameleon: A hybrid secure computation framework for machine learning applications,” in _CCS_ , 2018, pp. 707–721.
  * [101] V. Rastogi and S. Nath, “Differentially private aggregation of distributed time-series with transformation and encryption,” in _Proceedings of the 2010 ACM SIGMOD International Conference on Management of data_ , 2010, pp. 735–746.
  * [102] E. Shi, H. Chan, E. Rieffel, R. Chow, and D. Song, “Privacy-preserving aggregation of time-series data,” in _Annual Network & Distributed System Security Symposium (NDSS)_, 2011.
  * [103] G. Ács and C. Castelluccia, “I have a dream!(differentially private smart metering).” in _Information hiding_ , vol. 6958, 2011, pp. 118–132.
  * [104] L. Lyu, K. Nandakumar, B. Rubinstein, J. Jin, J. Bedo, and M. Palaniswami, “PPFA: Privacy preserving fog-enabled aggregation in smart grid,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 8, pp. 3733–3744, 2018.
  * [105] V. Chen, V. Pastro, and M. Raykova, “Secure computation for machine learning with spdz,” _CoRR, arXiv:1901.00329_ , 2019.
  * [106] P. Mohassel and P. Rindal, “Aby 3: a mixed protocol framework for machine learning,” in _CCS_ , 2018, pp. 35–52.
  * [107] J. Li, M. Khodak, S. Caldas, and A. Talwalkar, “Differentially private meta-learning,” _CoRR, arXiv:1909.05830_ , 2019.
  * [108] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar, “Semi-supervised knowledge transfer for deep learning from private training data,” in _ICLR_ , 2017.
  * [109] N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson, “Scalable private learning with pate,” _CoRR, arXiv:1802.08908_ , 2018.
  * [110] L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,” _arXiv preprint arXiv:2003.02133_ , 2020.
  * [111] J. C. Duchi, M. I. Jordan, and M. J. Wainwright, “Local privacy and statistical minimax rates,” in _Proceedings of the 54th IEEE Annual Symposium on Foundations of Computer Science_ , 2013, pp. 429–438.
  * [112] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in _Annual International Conference on the Theory and Applications of Cryptographic Techniques_ , 2006, pp. 486–503.
  * [113] L. Lyu, X. He, and Y. Li, “Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairness,” _arXiv preprint arXiv:2010.01285_ , 2020.
  * [114] M. Yang, L. Lyu, J. Zhao, T. Zhu, and K.-Y. Lam, “Local differential privacy and its applications: A comprehensive survey,” _arXiv preprint arXiv:2008.03686_ , 2020.
  * [115] S. L. Warner, “Randomized response: A survey technique for eliminating evasive answer bias,” _Journal of the American Statistical Association_ , vol. 60, no. 309, pp. 63–69, 1965.
  * [116] Ú. Erlingsson, V. Pihur, and A. Korolova, “Rappor: Randomized aggregatable privacy-preserving ordinal response,” in _CCS_ , 2014, pp. 1054–1067.
  * [117] T. H. Chan, E. Shi, and D. Song, “Optimal lower bound for differentially private multi-party aggregation,” in _European Symposium on Algorithms_ , 2012, pp. 277–288.
  * [118] T. Chan, K.-M. Chung, B. M. Maggs, and E. Shi, “Foundations of differentially oblivious algorithms,” in _Proceedings of the 30th Annual ACM-SIAM Symposium on Discrete Algorithms_ , 2019, pp. 2448–2467.
  * [119] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in _CCS_ , 2015, pp. 1310–1321.
  * [120] J. Hamm, Y. Cao, and M. Belkin, “Learning privately from multiparty data,” in _International Conference on Machine Learning_ , 2016, pp. 555–563.
  * [121] L. Lyu and C.-H. Chen, “Differentially private knowledge distillation for mobile analytics,” in _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2020, pp. 1809–1812.
  * [122] K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private empirical risk minimization,” _Journal of Machine Learning Research_ , vol. 12, no. Mar, pp. 1069–1109, 2011.
  * [123] B. I. P. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S. Lau, S. Rao, N. Taft, and J. D. Tygar, “ANTIDOTE: understanding and defending against poisoning of anomaly detectors,” in _Proceedings of the 9th ACM SIGCOMM Internet Measurement Conference_. ACM, 2009, pp. 1–14.
  * [124] B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” _CoRR, arXiv:1206.6389_ , 2012.
  * [125] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li, “Manipulating machine learning: Poisoning attacks and countermeasures for regression learning,” in _SP_ , 2018, pp. 19–35.
  * [126] B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. Rubinstein, U. Saini, C. A. Sutton, J. D. Tygar, and K. Xia, “Exploiting machine learning to subvert your spam filter.” _LEET_ , vol. 8, pp. 1–9, 2008.
  * [127] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar, “Adversarial machine learning,” in _Proceedings of the 4th ACM workshop on Security and Artificial Intelligence_ , 2011, pp. 43–58.
  * [128] X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” _CoRR, arXiv:1712.05526_ , 2017.
  * [129] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” _CoRR, arXiv:1708.06733_ , 2017\.
  * [130] A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, and T. Goldstein, “Poison frogs! targeted clean-label poisoning attacks on neural networks,” in _NeurIPS_ , 2018, pp. 6103–6113.
  * [131] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in _NDSS_ , 2018.
  * [132] C. Chen, J. Zhang, A. K. Tung, M. Kankanhalli, and G. Chen, “Robust federated recommendation system,” _arXiv preprint arXiv:2006.08259_ , 2020.
  * [133] G. Baruch, M. Baruch, and Y. Goldberg, “A little is enough: Circumventing defenses for distributed learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 8632–8642.
  * [134] T. A. Nguyen and A. Tran, “Input-aware dynamic backdoor attack,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 3454–3464, 2020.
  * [135] L. Muñoz-González, B. Biggio, A. Demontis, A. Paudice, V. Wongrassamee, E. C. Lupu, and F. Roli, “Towards poisoning of deep learning algorithms with back-gradient optimization,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 27–38.
  * [136] P. W. Koh and P. Liang, “Understanding black-box predictions via influence functions,” in _ICML_ , 2017, pp. 1885–1894.
  * [137] S. Zhao, X. Ma, X. Zheng, J. Bailey, J. Chen, and Y.-G. Jiang, “Clean-label backdoor attacks on video recognition models,” in _CVPR_ , 2020, pp. 14 443–14 452.
  * [138] J. R. Douceur, “The sybil attack,” in _International Workshop on Peer-to-Peer Systems_ , 2002, pp. 251–260.
  * [139] S. Shen, S. Tople, and P. Saxena, “Auror: defending against poisoning attacks in collaborative deep learning systems,” in _Proceedings of the 32nd Annual Conference on Computer Security Applications_. ACM, 2016, pp. 508–519.
  * [140] E. M. E. Mhamdi, R. Guerraoui, and S. Rouault, “The hidden vulnerability of distributed learning in byzantium,” _CoRR, arXiv:1802.07927_ , 2018.
  * [141] V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage, “Back to the drawing board: A critical evaluation of poisoning attacks on federated learning,” _arXiv preprint arXiv:2108.10241_ , 2021.
  * [142] C. Wu, X. Yang, S. Zhu, and P. Mitra, “Mitigating backdoor attacks in federated learning,” _arXiv preprint arXiv:2011.01767_ , 2020.
  * [143] C. Xie, M. Chen, P.-Y. Chen, and B. Li, “Crfl: Certifiably robust federated learning against backdoor attacks,” _arXiv preprint arXiv:2106.08283_ , 2021\.
  * [144] B. Han, I. W. Tsang, and L. Chen, “On the convergence of a family of robust losses for stochastic gradient descent,” in _Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD_ , 2016, pp. 665–680.
  * [145] L. Chen, H. Wang, Z. Charles, and D. Papailiopoulos, “Draco: Byzantine-resilient distributed training via redundant gradients,” _CoRR, arXiv:1803.09877_ , 2018.
  * [146] C. Xie, O. Koyejo, and I. Gupta, “Generalized byzantine-tolerant sgd,” _CoRR, arXiv:1802.10116_ , 2018.
  * [147] D. Alistarh, Z. Allen-Zhu, and J. Li, “Byzantine stochastic gradient descent,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 4613–4623.
  * [148] X. Xu and L. Lyu, “Towards building a robust and fair federated learning system,” _arXiv preprint arXiv:2011.10464_ , 2020.
  * [149] J. Steinhardt, M. Charikar, and G. Valiant, “Resilience: A criterion for learning in the presence of arbitrary outliers,” _arXiv preprint arXiv:1703.04940_ , 2017.
  * [150] Y. Li, B. Wu, Y. Jiang, Z. Li, and S.-T. Xia, “Backdoor learning: A survey,” _arXiv preprint arXiv:2007.08745_ , 2020.
  * [151] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 707–723.
  * [152] H. Chen, C. Fu, J. Zhao, and F. Koushanfar, “Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks.” in _IJCAI_ , 2019, pp. 4658–4664.
  * [153] K. Liu, B. Dolan-Gavitt, and S. Garg, “Fine-pruning: Defending against backdooring attacks on deep neural networks,” in _International Symposium on Research in Attacks, Intrusions, and Defenses_ , 2018, pp. 273–294.
  * [154] T. J. L. Tan and R. Shokri, “Bypassing backdoor detection algorithms in deep learning,” _CoRR, arXiv:1905.13409_ , 2019.
  * [155] P. Zhao, P.-Y. Chen, P. Das, K. N. Ramamurthy, and X. Lin, “Bridging mode connectivity in loss landscapes and adversarial robustness,” _arXiv preprint arXiv:2005.00060_ , 2020.
  * [156] Y. Yao, H. Li, H. Zheng, and B. Y. Zhao, “Latent backdoor attacks on deep neural networks,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , 2019, pp. 2041–2055.
  * [157] S. Andreina, G. A. Marson, H. Möllering, and G. Karame, “Baffle: Backdoor detection via feedback-based federated learning,” in _2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 2021, pp. 852–863.
  * [158] H. Chang, V. Shejwalkar, R. Shokri, and A. Houmansadr, “Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer,” _CoRR, arXiv:1912.11279_ , 2019.
  * [159] O. Thakkar, G. Andrew, and H. B. McMahan, “Differentially private learning with adaptive clipping,” _CoRR, arXiv:1905.03871_ , 2019.
  * [160] J. Bernstein, J. Zhao, K. Azizzadenesheli, and A. Anandkumar, “signsgd with majority vote is communication efficient and fault tolerant,” _CoRR, arXiv:1810.05291_ , 2018.
  * [161] L. Lyu and C. Chen, “A novel attribute reconstruction attack in federated learning,” _arXiv preprint arXiv:2108.06910_ , 2021.
  * [162] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _ICLR_ , 2018.
  * [163] Y. Wang, X. Ma, J. Bailey, J. Yi, B. Zhou, and Q. Gu, “On the convergence and robustness of adversarial training.” in _ICML_ , vol. 1, 2019, p. 2.
  * [164] Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu, “Improving adversarial robustness requires revisiting misclassified examples,” in _ICLR_ , 2019\.
  * [165] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in _ICLR_ , 2019.
  * [166] L. Lyu, J. C. Bezdek, X. He, and J. Jin, “Fog-embedded deep learning for the internet of things,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 7, pp. 4206–4215, 2019.
  * [167] L. Lyu, J. Yu, K. Nandakumar, Y. Li, X. Ma, J. Jin, H. Yu, and K. S. Ng, “Towards fair and privacy-preserving federated deep models,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 31, no. 11, pp. 2524–2541, 2020.
  * [168] L. Lyu, Y. Li, K. Nandakumar, J. Yu, and X. Ma, “How to democratise and protect ai: Fair and differentially private decentralised deep learning,” _IEEE Transactions on Dependable and Secure Computing_ , 2020.
  * [169] L. Lyu, J. C. Bezdek, J. Jin, and Y. Yang, “Foreseen: Towards differentially private deep inference for intelligent internet of things,” _IEEE Journal on Selected Areas in Communications_ , 2020.
  * [170] X. Pan, M. Zhang, S. Ji, and M. Yang, “Privacy risks of general-purpose language models,” in _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2020, pp. 1314–1331.
  * [171] F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis.” in _USENIX Security Symposium_ , 2016, pp. 601–618.
  * [172] X. He, L. Lyu, L. Sun, and Q. Xu, “Model extraction and adversarial transferability, your bert is vulnerable!” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 2006–2012.
  * [173] Q. Xu, X. He, L. Lyu, L. Qu, and G. Haffari, “Beyond model extraction: Imitation attack for black-box nlp apis,” _arXiv preprint arXiv:2108.13873_ , 2021.
  * [174] C. Chen, X. He, L. Lyu, and F. Wu, “Killing one bird with two stone: Stealing model and inferring attribute from bert-based apis,” _arXiv preprint arXiv:2105.10909_ , 2021.
  * [175] G. Zizzo, A. Rawat, M. Sinn, and B. Buesser, “Fat: Federated adversarial training,” _arXiv preprint arXiv:2012.01791_ , 2020.
  * [176] J. Hong, H. Wang, Z. Wang, and J. Zhou, “Federated robustness propagation: Sharing adversarial robustness in federated learning,” _arXiv preprint arXiv:2106.10196_ , 2021.
  * [177] D. Shah, P. Dube, S. Chakraborty, and A. Verma, “Adversarial training in communication constrained federated learning,” _arXiv preprint arXiv:2103.01319_ , 2021.
  * [178] X. He, Q. Xu, L. Lyu, F. Wu, and C. Wang, “Protecting intellectual property of language generation apis with lexical watermark,” in _AAAI_ , 2022.
  * [179] N. Truong, K. Sun, S. Wang, F. Guitton, and Y. Guo, “Privacy preservation in federated learning: An insightful survey from the gdpr perspective,” _Computers & Security_, vol. 110, p. 102402, 2021.
  * [180] K. Cheng, T. Fan, Y. Jin, Y. Liu, T. Chen, D. Papadopoulos, and Q. Yang, “Secureboost: A lossless federated learning framework,” _IEEE Intelligent Systems_ , 2021.
  * [181] Z. Tian, R. Zhang, X. Hou, J. Liu, and K. Ren, “Federboost: Private federated learning for gbdt,” _arXiv preprint arXiv:2011.02796_ , 2020.
  * [182] X. Jin, P.-Y. Chen, C.-Y. Hsu, C.-M. Yu, and T. Chen, “Catastrophic data leakage in vertical federated learning,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [183] D. Gao, Y. Liu, A. Huang, C. Ju, H. Yu, and Q. Yang, “Privacy-preserving heterogeneous federated transfer learning,” in _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 2019, pp. 2552–2559.
  * [184] X. Xinyi, L. Lyu, X. Ma, C. Miao, C.-S. Foo, and B. K. H. Low, “Gradient driven rewards to guarantee fairness in collaborative machine learning,” in _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021\.
  * [185] X. X. Lingjuan Lyu and Q. Wang, “Collaborative fairness in federated learning,” _https://arxiv.org/abs/2008.12161v1_ , 2020.
  * [186] Q. Yang, L. Fan, and H. Yu, Eds., _Federated Learning: Privacy and Incentive_. Springer International Publishing, 2020.
  * [187] S. Warnat-Herresthal, H. Schultze, K. L. Shastry, S. Manamohan, S. Mukherjee, V. Garg, R. Sarveswara, K. Händler, P. Pickkers, N. A. Aziz _et al._ , “Swarm learning for decentralized and confidential clinical machine learning,” _Nature_ , vol. 594, no. 7862, pp. 265–270, 2021.
  * [188] N. Guha, A. Talwalkar, and V. Smith, “One-shot federated learning,” _arXiv preprint arXiv:1902.11175_ , 2019.
  * [189] Q. Li, B. He, and D. Song, “Practical one-shot federated learning for cross-silo setting,” _arXiv preprint arXiv:2010.01017_ , 2020.
  * [190] D. K. Dennis, T. Li, and V. Smith, “Heterogeneity for the win: One-shot federated clustering,” 2021.
  * [191] J. Zhang, C. Chen, B. Li, L. Lyu, S. Wu, J. Xu, S. Ding, and C. Wu, “A practical data-free approach to one-shot federated learning with heterogeneity,” _arXiv preprint arXiv:2112.12371_ , 2021.