# Threats and Defenses in Federated Learning Life Cycle: A Comprehensive
Survey and Challenges

Yanli Li, Zhongliang Guo, Nan Yang, Huaming Chen, Dong Yuan, and Weiping Ding
Yanli Li is with the School of Artificial Intelligence and Computer Science,
Nantong University, Nantong 226019, China, and also with the School of
Electrical and Computer Engineering, The University of Sydney, Sydney, NSW
2006, Australia (e-mail: yanli.li@sydney.edu.au).Zhongliang Guo is with the
University of St Andrews, St Andrews, Fife, KY16 9SX, UK (e-mail: zg34@st-
andrews.ac.uk).Nan Yang, Huaming Chen, and Dong Yuan are with the School of
Electrical and Computer Engineering, The University of Sydney Sydney, NSW
2006, Australia (e-mail: {n.yang, huaming.chen,
dong.yuan}@sydney.edu.au,Weiping Ding is with the School of Artificial
Intelligence and Computer Science, Nantong University, Nantong 226019, China
(e-mail: dwp9988@163.com).*Corresponding author.

###### Abstract

Federated Learning (FL) offers innovative solutions for privacy-preserving
collaborative machine learning (ML). Despite its promising potential, FL is
vulnerable to various attacks due to its distributed nature, affecting the
entire life cycle of FL services. These threats can harm the model’s utility
or compromise participants’ privacy, either directly or indirectly. In
response, numerous defense frameworks have been proposed, demonstrating
effectiveness in specific settings and scenarios. To provide a clear
understanding of the current research landscape, this paper reviews the most
representative and state-of-the-art threats and defense frameworks throughout
the FL service life cycle. We start by identifying FL threats that harm
utility and privacy, including those with potential or direct impacts. Then,
we dive into the defense frameworks, analyze the relationship between threats
and defenses, and compare the trade-offs among different defense strategies.
Finally, we summarize current research bottlenecks and offer insights into
future research directions to conclude this survey. We hope this survey sheds
light on trustworthy FL research and contributes to the FL community.

Index Terms— Federated Learning (FL), Adversarial Machine Learning (AML),
Privacy, Robustness.

##  I INTRODUCTION

Nowadays, smart devices have become integral to people’s daily lives. While
engaging with these applications or smart services, users generate tens of
billions of data points per second. This data, regarded as a valuable resource
for fueling data-driven models, but remains underutilized due to the
centralized nature of traditional training paradigms [1, 2]. In particular,
centralized data collection necessitates stable network connections and high-
speed broadband, conditions that often conflict with the realities of edge
devices. On the other hand, transferring data from the user’s domain raises
significant privacy concerns, particularly when dealing with sensitive
financial, medical, and social media information. As phrased by authors in
[3], “data remains isolated within individual owners’ hands, creating data
silos.”

To mitigate the problem of data silos, federated learning (FL) was introduced
by Google in 2017 [4]. This new machine learning (ML) paradigm allows
participants to collaborate in a loosely federated manner while retaining a
degree of “autonomy” [3]. Under the FL paradigm, participants locally train
their model and only upload model updates for aggregation. Due to its privacy-
preserving characteristics, FL has attracted significant attention from both
academia and industry, leading to the deployment of numerous real-world
applications. Notable examples include Google’s smart keyboard Gboard [5],
Apple’s intelligent voice assistant Siri [6], and WeBank’s finance risk
prediction service for reinsurance [7]. A typical FL service life cycle [3, 4,
1] begins with a service provider publishing a task and selecting
participants. Each selected client then receives the global model, trains it
locally, uploads the gradients, and participates in the aggregation process.
Once the global model achieves the desired performance, the server distributes
rewards and releases the application to the public.

Despite its promise, FL is vulnerable to various attacks due to its
distributed nature [8]. Based on the capacity of adversaries, these threats
can completely [9, 10, 11] or partially [12, 13, 14] harm the utility of
model, break the privacy of participants [15, 16, 17], and could happen at any
phase during the FL service life cycle. In response, numerous defense
frameworks [18, 19, 20] have been proposed and shown effectiveness across
different settings and scenarios. Given the extensive security research, the
FL community urgently needs comprehensive reviews to provide a clear
understanding of the current research landscape.

Although some reviews exist [21, 22, 23], their scope is generally limited to
threats leading to direct, serious harm and around the local training phase,
failing to provide a comprehensive picture of the research field. To mitigate
the research gap, we present this review, which aims to comprehensively
investigate all threats throughout the FL life cycle, along with their
corresponding defenses. Unlike existing reviews, our study considers both
direct and potential threats and expands the scope to encompass the entire FL
life cycle.

Figure 1: Organization of the survey.

In summary, the main contributions of this survey are as follows:

  * •

Providing the entire life cycle of FL services, from the learning task publish
to service delivery; identifying each learning phase’s vulnerabilities, the
potential threats’ impact, and actors.

  * •

Identifying and comparing the utility-harmed threats in the FL environment,
providing their attack and poisoning strategies, ranging from those with the
potential to direct impact.

  * •

Identifying and comparing the privacy-harmed threats in the FL environment,
ranging from the application to the network layer.

  * •

Classifying the existing defense frameworks, summarizing their effectiveness
when defending against various utility-harmed or privacy-harmed threats, and
discussing their assumptions and trade-offs.

  * •

Discussing the existing bottlenecks of trustworthy FL, identifying the
emerging requirements, and providing insights about future trends.

The rest of this paper is organized as follows: We provide the background
information of FL in Section II, including its life cycle, vulnerabilities,
and categories. Sections III and IV review the most representative and state-
of-the-art threats targeting the utility and privacy of FL systems. In
Sections V and VI, we summarize the defense frameworks for addressing FL
privacy concerns and enhancing FL robustness, compare their effectiveness, and
discuss their trade-offs. Section VII offers insights into future research
trends, and Section VIII concludes the survey. For better readability, we
present a diagram in Figure 1, illustrating the organization of the survey.

Figure 2: Federated learning life cycle.

##  II Federated Learning Background

In this section, we introduce the FL service life cycle, different types of
FL, and FL threat models to provide the survey’s background.

###  II-A Federated Learning Life cycle

0\. Learning task bidding and participants selection: The life cycle of FL
begins when service providers publish a learning task within FL communities or
mention it in an application’s user terms. Interested clients submit bids to
participate based on the specified requirements and promised rewards. These
bids include commitments of their computing resources, bandwidth, network
capabilities, and training data. Service providers then review these
commitments and select the participants for the learning task.

1\. Global model broadcasting: The server, which can be the service provider
or another third-party computing center, broadcasts the current global model
to all selected participants. During the initialization of the learning task
(the first learning round), the server also specifies the conditions for
concluding both local and global learning iterations.

2\. Participants’ local training: Once the current global model is received,
each participant trains the model locally using their private data. This
training process continues for several rounds until preset conditions, such as
a specified number of learning rounds or a desired model accuracy, are met.
Subsequently, the participants send their local model updates back to the
server, completing the local training phase.

3\. Aggregation and global model update: The server aggregates the collected
local model updates using a predefined aggregation rule. The resulting outcome
serves as the gradient to update the global model. If the end condition has
not been reached, the updated global model is then broadcast to all
participants for the next round of training.

4\. Incentive distribution: When the global model achieves the expected
performance, the server stops broadcasting, and the training task is
finalized. The service provider then evaluates each participant’s contribution
and distributes incentives accordingly, which could be either profit or non-
profit.

5\. Model publish, and user access: Finally, the service provider publishes
the FL model or FL-based service online. Users can then access and utilize the
application for various purposes, such as querying, making predictions, and
more.

Figure 2 illustrates the life cycle of FL service.

###  II-B Federated learning Categories

The FL can be categorized based on the participants’ data distribution or the
system architecture [24].

Based on the distribution of data features and data samples, FL can be
classified as horizontal federated learning (HFL), vertical federated learning
(VFL), and federated transfer learning (FTL). Specifically, HFL is applicable
to cases in which participants own different data samples but the same feature
spaces. For example, in rare disease research, hospitals may have distinct
patients but their records own highly similar features. Contrary to HFL, the
VFL participants overlap in sample spaces but differ in feature spaces. For
example, banks and telecom operators in a city may keep records of the same
citizens, but these records contain different features. FTL integrates the
concepts of FL and transfer learning, where transfer learning leverages
knowledge acquired from one domain to enhance performance in another domain.
FTL addresses the challenge of limited overlapping samples and features by
training a model on a large public dataset. Figure 3 illustrates the clients’
data distribution of HFL, VFL and FTL.

On the other hand, FL can be divided into centralized federated learning (CFL)
and decentralized federated learning (DFL) based on the system architecture.
Under CFL, a server organizes the whole learning process, including models
broadcast, collection and aggregation. In contrast, DFL does not involve the
central server. Following a preset rule, participants can directly exchange
their models and automatically perform aggregation. Without further
notification, we use FL to represent CFL in this survey. Figure 4 illustrates
the system architectures of CFL and DFL.

Figure 3: Illustration of the clients’ data distribution of HFL, VFL, and FTL. Figure 4: Illustration of CFL and DFL system architectures. TABLE I: Illustration of the vulnerabilities throughout the FL life cycle and the threat’s impacts. FL Life Cycle | Client Selection | Model Broadcast | Local Training | Aggregation | Incentive | Users Access
---|---|---|---|---|---|---
Actor | Server, Potential clients | Server | Clients | Sever | Server, Clients | Users
Interacted Surface | Cloud, Ground | Cloud, Network | Ground, Network | Cloud | Cloud, Ground | Ground
Utility | Poisoning |  |  |  |  |  |
Promise Breaking |  |  |  |  |  |
Sybil |  |  |  |  |  |
Communication |  |  |  |  |  |
Evasion |  |  |  |  |  |
Privacy | Inference | $\blacksquare$ | $\blacksquare$ | $\square$ | $\square$ | $\blacksquare$ | $\square$
Inversion | $\blacksquare$ | $\blacksquare$ | $\blacksquare$ |  $\square$ | $\blacksquare$ | $\blacksquare$
Eavesdropping | $\blacksquare$ | $\square$ | $\square$ | $\blacksquare$ | $\blacksquare$ | $\blacksquare$
$\blacksquare$ Non-exist,  Exist and Potential impact,  $\square$ Exist and
Direct impact

###  II-C Threat Models in Federated Learning Life Cycle

In FL, each participant has a certain degree of autonomy and can independently
conduct local training. While the federated architecture potentially enhances
participants’ privacy, it also introduces new attack surfaces. Beyond the
training phase attacks that directly degrade FL model accuracy and compromise
client privacy, dishonest behavior in other phases of the FL life cycle can
also negatively impact the model. For instance, dishonest self-reporting
during participant selection can lead to insufficient computing or data
resources during training, resulting in lower-than-expected model performance.
In light of these scenarios, we propose a broader definition of threats and
attackers in this survey:

Threats (i.e., attacks) are the misbehavior that may directly or potentially
harm FL models’ performance (i.e., utility) or privacy.

Attackers are the actors who intentionally launch threats.

Vulnerabilities in FL: Today, numerous studies have demonstrated that attacks
can occur at any stage throughout the entire FL life cycle. Initially, a
dishonest client may exaggerate its capabilities to secure a spot in the
selection process and join the learning task. Under the local training, the
malicious clients may use modified training data to train the model, or
directly perturb the trained model (or gradients) to degrade, even fully break
the model performance. During server-client communication, the information
transmitted can be replaced or tampered with by a ”middleman.” Once the
training is complete, the dishonest participants can forge their consumption
to defraud high returns. When the FL-based application is published, an
attacker can use a crafted data item to mislead the application to generate
unexpected outcomes. Although the server is assumed to be trustworthy in most
cases, the server is potentially curious or hijacked during the training
process. In those cases, the server may act maliciously and infer sensitive
information from individual updates collected. We summarize and provide
detailed discussions of the utility-harmed and privacy-harmed threats in
Section III and IV. Table I illustrates the vulnerabilities throughout the FL
life cycle and the threat’s impacts.

Attacker’s Knowledge and Capacity: Attacker’s knowledge represents how much
information of the FL system can be inspected by the adversary. From the
perspective of model parameters, the attacker’s knowledge can be categorized
as black-box, grey-box, or white-box [25, 22]. These three settings correspond
to scenarios where the adversary has no access to the FL model, has the
ability to inspect the model of a particular state (possibly a historical
version), and can inspect the FL model parameters of the current learning
round, respectively. Considering participants receive the global model to
perform local training each iteration, the grey box is the most general
setting in adversarial FL contents. As some FL algorithms introduce the
byzantine-robust aggregation rule as a defense mechanism, the attacker’s
knowledge is also considered as to whether the aggregation rule is exposed and
reviewed by adversaries.

The attacker’s capacity in the FL context generally refers to the level of
collusion among adversaries. Malicious clients are often assumed to constitute
a certain percentage of the overall participants, and these attackers can
collude to agree on the same attack strategy to maximize attack efficiency. In
a non-collusion setting, the adversaries are considered to have different
attacking objectives or strategies, resulting in a weaker attack effect. We
note that some recent works [26] classifies the attacker’s capacity as passive
or active, depending on whether the adversary can directly interfere with the
global model or simply inspect it. However, we believe that the terms passive
and active do not accurately reflect the capacity of the attackers but rather
the characteristics of different phases in the FL life cycle. Specifically,
participants can arbitrarily modify their local model during the training
phase, while they can only access or visit the FL application during the pre-
and post-training phases. Thus, in this survey, we do not discuss the passive
or active nature of adversaries from their perspective. Instead, we emphasize
the interaction methods of various parties to the FL system at different
phases in the FL life cycle.

In some attack strategies, attackers may need to perform large-scale grid
searches or complex computations to generate the desired crafted parameters.
Under these scenarios, the attacker’s capacity extends to computing resources,
which could be at the average level of the overall clients or potentially
unlimited.

Figure 5: Demonstration and comparison of different FL utility-harmed attacks.

##  III Utility-harmed Attacks

Compared to data-centralized collection learning schemes, FL provides a
different way of training models collaboratively [4, 3]. Instead of relying on
a visible and centralized collection dataset, FL trains the model across a
potentially vast array of unreliable devices, each equipped with private,
uninspectable datasets. Furthermore, the local training process also
introduces a new attack surface and brings opportunities for potential attacks
[27, 11, 28, 29]. Specifically, as data remains on the devices and only model
updates are shared, adversaries could attempt to exploit this setup to
compromise the model’s utility or performance. We demonstrate and compare
different FL utility-harmed attacks in Figure 5.

###  III-A Poisoning Attacks

As one of the most representative adversarial attacks, poisoning attacks
attempt to degrade the model performance or totally destroy the global model.
Depending on the stage at which the poisoning occurs, poisoning attacks can be
further categorized into model poisoning attacks and data poisoning attacks.

####  III-A1 Model poisoning Attacks

Model poisoning attacks [30] refer to adversaries directly manipulating
reports (local model updates) submitted to the service provider. One of the
most straightforward strategies for executing model attacks involves
introducing a fixed perturbation or substituting the benign gradient
parameters with fixed values. For instance, Reverse Attack [9], and Random
Attack [10] generate the malicious gradient by reversing and randomly
replacing the original gradients. Under the Partial Drop Attack [10], the
adversary replaces a preset percentage of the benign gradient parameters as 0;
the variants of Partial Drop Attack enlarge the attack effectiveness by using
other malicious values if the benign gradients naturally carry a large amount
of 0. Although these attack strategies can effectively degrade the global
model performance and decrease the testing accuracy, the malicious gradients
are unavoidably away from the benign gradients, which can be identified and
discarded by the byzantine resilient FL methods.

To enhance the attacking performance and avoid being identified by the
defender, some existing attacks [27, 28, 11] design more dynamic attack
strategies. For instance, Little is Enough Attack [31] first normally trains
the model through the data of the clients controlled. Then, the adversary
statistics the mean value and standard deviation (Std) of the gradients owned.
Based on the statistics, the adversary subsequently generates the crafted
gradient by adding a scaled Std on the mean value. The Fall of Empires Attack
[32] follows a similar strategy, which generates the malicious gradient by
timing the mean value with a preset scalier. To maintain the stealth of model
poisoning attack, research [27] proposes an optimization-based model poisoning
attack, which only manipulates a small fraction of the local model updates
instead of all gradient parameters. Under the study [27], adversary training
is organized into two tasks: the main task and the adversarial task. In the
main task, the adversary trains the model in the usual manner. Conversely, in
the adversarial task, the adversary embeds adversarial features into the
neural network’s redundant space to enhance the attack’s persistence.

Recently, Local Model Poisoning Attack [11] proposes a more flexible attacking
algorithm that adopts the attack strategy for different defense frameworks.
Based on the information owned (partial or full knowledge), this attack first
infers the optimal global model updates and subsequently formulates the
optimization problem that aims to reverse the global model update the most
along the infers. This work [11] applies the attack in Krum [33], Trimmed-mean
[34], and Median [34] frameworks and consequently generates the Krum Attack
and Trim Attack. The study [28] targets the Krum framework with an emphasis on
evasion detection, specifically through covert model poisoning (CMP). Research
[28] first formulates the model poisoning as an optimization problem, aiming
to minimize the Euclidean distance between the manipulated model and a
designated one, all within the constraints set by Krum. Based on the solution
to the optimization problem, [28] subsequently formulates CMP algorithms to
counter the Krum framework. To improve the practicability, a low-complexity
CMP algorithm is introduced and achieves the optimization complexity
reduction. We summarize various model poisoning strategies in Table II.

TABLE II: Illustration of various model poisoning strategies.

Nomination
---
$G^{\prime}$ Crafted gradients |  $\dot{G}$ Gradients of benign clients
$\ddot{G}$ Malicious gradients |  $G$ Gradients of all clients
$M$ Support matrix |  $P$ Preset attack possibility
$z,\gamma$ Attack scalar |  $\mu$ Mean of gradients
$\sigma$ Standard deviation |  $\nabla$ Additive perturbation

Attacks | Poisoning Strategy
---|---
Reverse [9] | $G^{\prime}=-\ddot{G}$
Mimic [35] | $G^{\prime}=\ddot{G_{i}}$
Partial Drop [10] | $G^{\prime}=\ddot{G}\cdot M,M\sim Bernoulli(P)$
Random [10] | $G^{\prime}=\ddot{G}\cdot M,M\sim Random(P)$
LIE [31] |  $G^{\prime}=\mu-z\sigma$ or $\mu+z\sigma$
FOE [32] |  $G^{\prime}=-z\mu$ or $z\mu$
LMP [11] partial |  $G^{\prime}\in$ $(\mu+3\sigma,\mu+4\sigma)$ or $(\mu-4\sigma,\mu-3\sigma)$
LMP [11] full |  $G^{\prime}\in$ $(G_{max},z\cdot G_{max})$ or $(z\cdot G_{min},G_{min})$
Min-Max [36] | $G^{\prime}=f_{avg}(\dot{G})+\gamma\nabla,$
| $\mathop{\arg\max}\limits_{\gamma}max||G-\dot{G_{i}}||_{2}\leq
max||\dot{G_{i}}-\dot{G_{j}}||_{2}$
Min-Sum [36] | $G^{\prime}=f_{avg}(\dot{G})+\gamma\nabla,$
|
$\mathop{\arg\max}\limits_{\gamma}\sum||G-\dot{G_{i}}||_{2}^{2}\leq\sum||\dot{G_{i}}-\dot{G_{j}}||_{2}^{2}$

####  III-A2 Data poisoning Attacks

Different from model poisoning attacks that directly introduce the
perturbation to the gradient, data poisoning attacks generate crafted
gradients by training models based on poisoned data. Label Flipping Attack is
one of the most representative data poisoning attacks. In Label Flipping
Attacks, the adversary mismatches the training labels and training data to
generate the malicious gradient, aiming to degrade the attacked class learning
performance of the joint model. Study [29] considers the support vector
machine (SVM) algorithm and aims to find a label-flipping combination under a
given budget so that a classifier trained on such data will have maximal
classification error. Research [37] introduces the Label Flipping Attack in
the deep learning scenarios and evaluates the key conditions for successfully
performing attacks, including sufficient adversarial participants and
continuous participation. To enhance the flexibility of the attacks, study
[38] proposes the On-Off Label Flipping strategy. Under the On-Off Label
Flipping strategy, the adversary first acts as a benign client for a period of
time and builds a positive expectation in the defense system to strengthen the
impact of the following updates. Then, the adversary betrays the system and
performs an attack for a subsequent period of time. As the betrayal behavior
lowers the reputation, the adversary may toggle back to acting as the benign
client again after the One-Off attack and recover its reputation.

Label Flipping Attacks also show significant effectiveness in real-world
applications. For instance, Study [19] implements this attack within a malware
detection task for Android platforms, employing a Silhouette Clustering-based
Label Flipping Attack (SCLFA). This method involves attackers calculating the
silhouette scores of data samples and targeting those with negative values for
generating polluted data through label flipping. Similarly, research [39]
illustrates the potency of Label Flipping Attacks in compromising spam
filtering systems. By executing entropy-based flipping attacks, their study
managed to elevate the false negative rate of Naive Bayes classifiers in the
presence of label noise without impairing the classification of legitimate
emails. Furthermore, study [40] explores the impact of Label Flipping Attacks
on hardware Trojan detection systems, demonstrating how model performance
could be substantially degraded through a stochastic hill-climbing search-
based flipping strategy, incurring minimal costs for label manipulation. These
instances underscore the critical need for robust defenses against Label
Flipping Attacks in diverse application areas.

####  III-A3 Backdoor Attacks

Backdoor Attack is a special type of poisoning attack, which aims to lead the
global model to misbehave only on a selected minority of examples while
maintaining good overall accuracy on all other examples [12, 13]. Recent
studies indicate that the Backdoor Attack can be executed either by poisoning
the clean training data or by introducing crafty perturbations to benign
gradients. In other words, the Backdoor Attack can be categorized as either a
model poisoning attack or a data poisoning attack based on the poisoning
strategy.

When performing Backdoor Attacks through the data poisoning strategy, the
adversary should first decide on a trigger (backdoor) and consequently assign
a selected label for the trigger carrier to form the poisoning data. This
backdoor could be semantic or artificial. For semantic backdoor, study [14]
leverages the samples with uncommon attributes as the backdoor, such as the
cars with unconventional colors (e.g., green), scenes containing a unique
object (e.g., a striped pattern), or trigger sentences in word prediction
problems that conclude with an attacker-selected target word. Similarly,
research [41] proposes an attack strategy to introduce a backdoor in the FL-
based IoT intrusion detection system. In this scenario, the adversary targets
packet sequences originating from specific malware-driven malicious traffic.
Different from the semantic backdoor that replies on existing share
properties, the artificial backdoor manually poisons benign samples by
introducing triggers. Study [1] demonstrates the adversary can manually add
pattern “L” at the images’ corner to activate the backdoor, where the pattern
“L” does not naturally exist in the benign samples. A recent study [42]
extends the scope of backdoor attacks to real-world settings. The research
reveals that beyond digital triggers, physical triggers such as sunglasses,
tattoos, and earrings can also serve as triggers for organizing backdoor
attacks.

Within the Vanilla FL framework, the most straightforward method to execute a
backdoor attack using a model poisoning strategy is scaling the malicious
updates containing backdoor information to dominate updates from benign
clients. Study [42] first employs the replace method, in which the attacker
seeks to substitute the new global model with a poisoned one by a wisely
chosen factor. While the scaling-replacement method proves effective in
averaging aggregation, straightforward scaling appears to be naive to success
under clipping and restricting defenses. To enhance the stealth of model
poisoning attacks, research [43] proposes a projected gradient descent (PGD)
attack strategy, which projects the crafted model on a small ball centered
around the global model from the previous iteration.

###  III-B Promise breaking

During the FL life cycle, clients may pledge to provide their training data or
computing resources to gain participation opportunities or claim their
contributions to receive incentives. However, malicious clients can exploit
the FL system to gain advantages, such as access to the joint model and
incentives, without fulfilling their promises [44, 45]. Although these
attackers do not deliberately poison the model, promise-breaking behaviors can
lead to insufficient computing resources and training data, as well as unfair
incentive distribution. This, in turn, may degrade learning performance and
diminish participant motivation.

Research [45] introduces a Free-riding Attack framework based on the Vanilla
FL and shows that the adversary can perform a Free-riding Attack in each
learning iteration. In particular, the Free-riding attacker does not perform
local training but adds noise to the constructed parameter updates and applies
Stochastic Gradient Descent (SGD) to maximize similarity with updates from
other benign clients. Research [44] proposes a novel Free-riding Attack in
which the model is trained locally using a small dataset masqueraded as a
large dataset to obtain more incentive rewards. Study [46] introduces a Free-
riding attack strategy, namely Random Weights Attack. Under this attack, the
adversary creates a gradient update matrix with dimensions matching the
received global model by randomly sampling each parameter from a uniform
distribution within a pre-designed range.

Studies [47, 48, 49] investigate dishonest reporting during intensive
distribution. To cheat the reward system, malicious participants may falsely
report their resource consumption or claim to have trained the joint model
with high-quality data while actually using low-quality data, such as spam.

Table III compares different promise-breaking threats.

TABLE III: Illustration of different promise-breaking attacks. Promise Breaking | Phase | Objective
---|---|---
Free-riding [45, 44, 46] | Local training | Model acquisition
Dishonest reporting | Client selection | Model acquisition
[47, 48, 49] | Incentive distribution | Incentive / Reward

###  III-C Sybil Attacks

The Sybil Attack refers to a single attacker (or a small amount attackers)
joining the system with multiple colluding identities to enhance the stealth
and effectiveness of an adversarial attack. Given that some traditional FL
permits selected participants to freely join and exit during training tasks,
it becomes inherently susceptible to Sybil Attacks.

The research [50] first introduces Sybil Attacks within the FL context. It
demonstrates Sybil Attacks can significantly enhance the impact of other
attacks (such as the Label Flipping Attack) by misleading the global model
into classifying handwritten digits “1” as “7” through only two sybils. Study
[51] builds upon the findings of [50], delving deeper into the vulnerability
of FL to Sybil Attacks, as well as exploring associated attack strategies and
objectives. This work [51] categorizes and names the Denial-of-Service (DoS)
attacks instigated by Sybil Attacks as Training Inflation and demonstrates
Sybil Attacks can be performed from various FL dimensions. From the data
distribution perspective, Sybils can use identical datasets, different
datasets, or synthetic data to train local models and generate crafted updates
[52]. On the other hand, to dominate the benign clients and amplify their
influence on the system, Sybils might coordinate amongst themselves before
producing the subsequent series of model updates, achieving different levels
of coordination.

###  III-D Communication Layer Attacks

FL continuously updates the joint model, which relies on communications
between the server and multiple participants. However, the large volume of
data exchange and frequent communication drives a significant communication
bottleneck and brings new attack interfaces for adversaries.

The man-in-the-middle (MiTM) attacks [53], as one of the most representative
network layer attacks, has been introduced in FL by recent studies [2, 54]. As
FL leverages the single server to aggregate model updates and organize the
learning process, performing MiTM between the server and participants can
effectively block the server and lead to a single point of failure. Research
[55] explores attacking the FL communication bottleneck by decreasing the
bandwidth, delaying the response, and increasing the struggle possibility.
Through performing attacks on the communication of FL, [55] shows the model
convergence and performance can be degraded significantly.

##  IV Privacy-harmed Attacks

Figure 6: Demonstration and comparison of different FL privacy-harmed attacks.

###  IV-A Evasion Attacks

In evasion attacks, the adversary deliberately adds slight malicious
perturbations to input samples during the inference phase, causing the
classifier to misclassify the sample with a high probability. One of the most
representative works of the evasion attack has been introduced by [56], which
is a so-called adversarial example. By adding unnoticeable pixel perturbation
on a panda image, [56] successfully changes the classification of GoogLeNet
[57] from panda to gibbon with 99.3% confidence.

Based on the information owned by the adversary, the evasion attacks could be
further divided into white-box and black-box attacks. In the white-box
setting, the adversary has full knowledge of the learning algorithm and model
parameters and can craft adversarial examples based on the information. For
instance, studies [58, 59] generate adversarial examples by maximizing the
loss function, subject to a norm constraint, using constrained optimization
techniques like projected gradient ascent. On the other hand, the black-box
setting refers to the attackers having no knowledge about the algorithm and
parameter information of the FL system and generating adversarial samples
through the interaction process or query access with the system. Considering
the black-box setting, [60] proposes a zeroth order optimization (ZOO) based
attack to estimate the target model updates and generate adversarial examples
consequently. Similarly, Boundary Attack [61] has been designed in the same
scenario. When performing Boundary Attack [61], the adversary first identifies
the boundary separating adversarial and non-adversarial samples, then subtly
introduces perturbations along this boundary to produce adversarial examples.

A recent work [62] introduces an Evasion Attack in the Vertical FL (VFL)
scenarios and proposes the Adversarial Dominating Inputs Attack. Unlike the
aforementioned adversarial samples that domain the whole feature space, the
Adversarial Dominating Inputs Attack [62] can dominate the inputs of other
clients by controlling a fraction of the feature inputs and lead to the
misclassification of specific inputs. In the meanwhile, the Adversarial
Dominating Inputs Attack can also result in reduced and unfair incentive
rewards for impacted participants due to their decreasing contributions.

While FL is designed to protect the privacy of participants, recent studies
have shown that privacy-harmed attacks still exist in FL systems. These attack
objectives include identifying members (MIA) or properties (PIA) of victim
training data, generating representative or (RIA) original samples (Inversion
Attacks) of victim training data, and eavesdropping on transmissions between
the victim and the server. We demonstrate and compare different FL privacy-
harmed attacks in Figure 6.

###  IV-B Inference Attacks

During the training process of FL, the model will inadvertently learn the
latent information of private data. Therefore, an honest but curious server or
any other user can launch an inference attack to recover the original training
data without any prior knowledge. Inference Attacks include model extraction
attacks, attribute inference attacks, and membership inference attacks.

####  IV-B1 Membership Inference Attack (MIA)

Similar to MIAs in traditional machine learning scenarios, malicious clients
or users can execute MIAs in FL applications to infer whether a data sample
belongs to its training dataset. Study [15] focuses on black-box within
Machine Learning as a Service (MLaaS) settings. By accessing multiple
generated shadow models, the study achieves high accuracy in inferring
membership. ML-Leaks [70] reduces the cost of launching attacks and
demonstrates the effectiveness of MIA using only a single (even without)
shadow model. Study [16] considers the white-box settings and proposes the
gradient ascent MIA strategy to magnify the presence of data points in others’
training sets. It demonstrates that all algorithms trained using Stochastic
Gradient Descent (SGD) are susceptible to privacy vulnerabilities from MIAs.
The study [71] further extends MIAs to NLP learning tasks and effectively
infers whether a specific text segment belongs to the training dataset.

####  IV-B2 Property Inference Attacks (PIA)

Property inference attacks aim to infer private features or characteristics
(e.g., value, distribution, etc.) of the training data without direct exposure
to the data. Although some surveys [24, 72] categorize threats aimed at
reconstructing training data as PIAs, we consider PIAs to primarily focus on
inferring properties of the training data, rather than the data itself.
Therefore, in this survey, we differentiate threats aimed at data
reconstruction from PIAs and classify them as Inversion Attacks.

Attackers in the study [73] use auxiliary data and trained classifiers to
determine if observed model updates are generated from a dataset that includes
the target property. Considering fully connected neural networks (FCNNs) are
invariant under the permutation of nodes when represented using matrices,
study [74] introduces PIAs in white-box settings and shows good inference
performance in shallow-FCNNs. The study [75] further extends PIAs to logistic
regression (LR), long short-term memory networks (LSTM), and graph
convolutional networks (GCN) across multiple domains. It demonstrates that
even in black-box settings, PIAs can still achieve accuracies exceeding 60%.

####  IV-B3 Representative Inference Attacks (RIA)

Representative Inference Attacks indicate the malicious generate dummy outcome
that is representative of the victim’s training data [76, 77]. The study [76]
first introduces record-level RIAs in collaborative learning scenarios and
demonstrates that GAN-based attacks can effectively compromise against
Distributed Selective SGD (DSSGD [78]). However, subsequent research [77]
reveals its performance degradation in FL settings due to the averaging of
updates and introduces the mGAN-AI framework. This framework enhances GANs
with a multitask discriminator capable of simultaneously assessing the
category, reality, and client identity of input samples. Compared to [76],
mGAN-AI enables the generator to recover representative private data for
specific users.

###  IV-C Inversion Attacks

Although FL allows clients to only upload gradients during model training, it
has been found that some properties of the training data are implicitly
carried by these gradients, which could be revealed through inverse attacks.
DLG [64] first introduces gradient inversion in FL computer vision areas.
Specifically, the DLG generates random dummy input samples $x^{\prime}$ and
labels $y^{\prime}$ and corresponding dummy gradients $G^{\prime}$.
$x^{\prime}$ and $y^{\prime}$ are Iteratively optimized through L-BFGS to
minimize the difference between dummy and benign gradient. Formally, the
objective function of DLG shown as follows:

$\mathop{\arg\min}\limits_{x^{\prime},y^{\prime}}||G^{\prime}-G||^{2}$ (1)

Considering the high-dimensional direction of gradients carried important
information, study [65] introduces cosine similarity in the loss function to
find images that lead to a similar change in model prediction as the ground
truth instead of finding images with a gradient that best fits the observed
gradient. The objective function is shown as follows, where TV represents the
Total Variance regularization.

$\mathop{\arg\min}\limits_{x^{\prime}}(1-\frac{G^{\prime}\cdot
G}{||G^{\prime}||\cdot||G||}+\alpha\cdot TV(x^{\prime}))$ (2)

Although DLG [64], IG[65] show effectiveness on single image and small batch
size training scenarios, their inversion performance degrades when facing
large batch size ($>8$). To further improve the attack effectiveness, GGL [68]
leverages a generative model trained on public datasets as the learned natural
image prior to ensure the reconstructed image quality. Additionally,
GradInversion (GI) [67] integrates Fidelity and Group Consistency
regularization into the objective function and effectively increases the
inversion batch size to 48. These are designed to ensure that the generated
images closely resemble real images and to penalize any dummy images that
deviate from a “consensus” image, respectively. GradViT [69] extends the GI
approach to Vision Transformer (ViT) models, demonstrating that ViTs are
significantly more vulnerable to inversion attacks than the previously studied
CNNs due to the attention mechanism.

We collect the images from study [79, 67] and compare the ImageNet batch
gradient inversion outcomes under different inversion attacks in Figure
LABEL:compare2.

###  IV-D Eavesdropping

Network eavesdropping involves an attacker intercepting and capturing data
packets transmitted over a network using their own devices, with the intent to
analyze the contents of these packets to get private information [80]. Since
model training under the FL scheme requires regular communication between the
server and clients, FL is naturally vulnerable to eavesdropping attacks. This
vulnerability is particularly pronounced when the communication between
participants and the server is in plaintext or employs a weak encryption
method [23, 81]. Since the messages transmitted are usually models or
gradients, the attacker may further leverage inference or inversion attacks to
decipher the eavesdropped messages to understand the private information [82,
83, 84]. Study [85] considers the eavesdropping attacks in unmanned aerial
vehicles (UAVs) scenarios, demonstrating that eavesdroppers can deduce the raw
data from the shared parameters and compromise participants’ privacy.

##  V Defence Against Utility-Harmed Attacks

As the diversity and complexity of adversarial attacks against FL increase,
new defenses are being developed to counteract their malicious impacts. These
defense frameworks generally follow the setting that the server is honest and
can play the defender role. As some defense methods are effective against
multiple types of attack categories, we survey these FL defense frameworks
from different technologies instead of grouping them according to the attack
category defended. Table IV provides the overview of these methods.

TABLE IV: Illustration and comparison of different defense methods for
utility-harmed attacks.
Defense Methods |  Trusted Server |  Defender | Preformed Phase |  Technique |  Remark
---|---|---|---|---|---
Data Sanitization |  $\checkmark$ |  Client, Server | Before training |  Remove suspected data |  Suffering from privacy issue, applicable to attacks related to poisoned data.
Anomaly Detection |  $\checkmark$ |  Server | Before aggregation |  Filter suspected identity |  Suffering from privacy and non-IID issues.
Adversarial Training |  $\checkmark$ |  Client | During training |  Robustness enhancement |  Could degrade the model performance.
Model Compress |  $\checkmark$ |  N/A | N/A |  Attack surface reduction |  Also mitigates the communication bottleneck but may degrade the model performance.
Byzantine-Resilient Aggregation |  $\checkmark$ |  Server | During aggregation |  Mitigate impacts carried by gradient outliers |  Performance could be limited by assumptions and knowledge of adversary.
Blockchain FL |  $\times$ |  Client | N/A |  Fully decentralized mechanism |  Can address the single point of failure issue but cannot inherently defend against attacks.

###  V-A Data Sanitization

Data sanitization [86, 87, 88] has been introduced to remove malicious and
suspicious data before the training process and consequently mitigate crafted-
data-related threats, including data poisoning and evasion attacks. The
research [86] introduces a novel training pipeline that incorporates a data
sanitization phase. During each iteration, this phase creates multiple models,
termed micro-models, based on subsets of the training data. These micro-models
are used to produce provisional labels for each training input and
consequently combined in a voting scheme to determine which parts of the
training data may be potentially attacked. To filter out noise-labeled data,
FedDiv [89] maintains a global filter. This filter is trained, exchanged, and
aggregated similarly to the global model and supports local data filtration.
From the data shuffling perspective, study [88] proposes a byzantine-resilient
matrix-vector (MV) multiplication and further proposes an algorithm based on
the data encoding process and error correction over real numbers to combat
various adversarial attacks.

Although data sanitization can filter out some potential attacked data and
keep training data clean, it relies on directly accessing the local data of
clients, which may raise privacy issues. A recent work [90] indicates that
data sanitization can only work in cross-silo FL with strong authentication
and be performed locally by the client itself. However, this framework [90]
assumes the trustworthiness of the clients, which may not be applicable to the
adversarial FL environment.

###  V-B Anomaly Detection

To perform threats, attackers unconsciously exhibit different patterns from
benign clients, including abnormal behaviors or data patterns. To this end,
anomaly detection has been proposed to identify suspicious through statistical
or behavior analysis technology [91, 92, 93]. We follow the study [24] and
divide the anomaly detection of FL in client and data from two perspectives.

A) Client Anomaly Detection
Under FL, Client Anomaly Detection could be performed based on the information
from the communication (IP address, User ID, Time Stamp [94]) or application
layer, namely client model updates. By employing a pre-trained auto-encoder
model, study [91] introduces Auto-encoder-Based Anomaly Detection to detect
abnormal model weight updates from the clients and consequently erase the
negative impact carried by the potential attackers. Inspired by spectral
anomaly detection that distinguishes abnormal data instances by capturing the
normal data features, research [95] designs a novel robust FL framework. The
spectral anomaly detection model is trained through a centralized training
process and applied in the following learning iteration. Since the detection
threshold is adjusted dynamically in each communication round, it becomes
challenging for the adversary to understand the framework, ensuring they can
be effectively excluded from the aggregation. Recent research [92] presents a
visualization scheme for anomaly detection, introducing VADAF. Specifically,
VADAF captures the intricate dynamics of the FL training process, aiding in
investigating various client issues and assessing the severity of anomalous
clients.

B) Data Anomaly Detection
Anomalous data, potentially introduced during the data sampling process, can
adversely affect the model’s training performance. Consequently, data anomaly
detection has been introduced to identify outliers in the dataset or values
that are far from the normal data feature values. Research [96] proposes a
two-phase iterative adversarial detection method to identify software samples
in the malicious application detection system that have been subjected to
poisoning attacks. Research [97] proposes an anomaly detection method for time
series datasets based on recursive auto-encoding, which reduces the impact of
overfitting to anomalies. To further improve robust and efficient anomaly
detection in unsupervised time series scenarios, a variational recurrent
encoder model [98] can separate anomalies from normal data without relying on
anomaly labels. However, this operation requires direct access to clients’
data or transferring trust to participants, leading to limitations similar to
those in Data Sanitization.

###  V-C Adversarial Training

During the model training process, a minor and well-designed perturbation can
be manually introduced to enhance the model’s robustness, namely Adversarial
Training.

Research [99] first introduces ensemble adversarial training in centralized
learning settings, which augments training data through transferring
perturbations from other pre-trained models. Studies [100, 93, 101] further
extend the adversarial training in decentralized settings. Specifically, study
[100] applies adversarial training in the FL environment to reduce model drift
and accelerate model convergence. Nevertheless, as the perturbation
introduction inevitably affects the classification accuracy, the performance
carried by adversarial training may not be stable against complex black-box
attacks. On the other hand, adversarial training relies on a massive volume of
training datasets, which increases the cost of computation resources on the
local devices. Especially in cross-device FL environments with multi-
participants, lightweight clients can not afford the high costs of adversarial
training. Research [93] proposes a novel learning scheme that propagates
adversarial robustness from high-resource users, who can undertake adversarial
learning, to low-resource users throughout the FL process. Research [101]
investigates the effectiveness of defencing Evasion Attacks through
adversarial learning. Specifically, Gaussian noise is used to smooth the
training data by including adversarial data in the training dataset.

###  V-D Model Compression

The large model size enables the DL model to achieve a good learning
performance but also introduces both communication challenges and a broadened
attack surface in the FL environment. To reduce communication overhead and
parameter redundancy, model compression technology has been widely
investigated by recent studies [102, 103, 104].

As one of the model compression technologies, knowledge distillation has been
incorporated into transfer FL in research [103], which introduces a general
framework named FedMD. FedMD allows clients with different computational
resources to design different network structures, protecting data privacy and
enhancing the performance of local models. While FedMD improves the local
model performance, the participants are required to devote a portion of data
privacy to form a shared dataset. Following FedMD [103], research [104]
leverages unlabeled data from local model outputs to achieve model fusion,
which further accelerates model convergence and mitigates data privacy
leakage.

TABLE V: Illustration of the techniques, requirements, and limitations of
various byzantine-robust aggregation methods.
Aggregation | Algorithm | Technique | Requirements & Limitations
---|---|---|---
| (Multi) Krum[33] | Euclidean distance | Knowledge of adversary amount assumption, non-IID issue
| Trimmed-Mean [34] | Mean value | Knowledge of adversary amount assumption, non-IID issue
| Median [34] | Median value | Non-IID issue
| Bulyan [105] | Euclidean distance+Mean value | Knowledge of adversary amount assumption, non-IID issue
Statistical Feature | RFA [106] | Geometric median | Applicable to a few poisoning attacks
-based | Sniper [107] | Maximum clique problem | Applicable to attacks bring large perturbation
| AFA [108] | Cosine similarity+Adaptive mean | Benign clients’ gradients should be close, non-IID issue
| LICM-SGD [109] | Lipschitz constant | Non-IID issue
| FoolsGold [50] | Cosine similarity | Applicable to sybil attacks
| MeaMed [110] | Mean+Median | Non-IID issue, Curse of dimensionality
| Byrd-SAGA [111] | SAGA+Geometric median | Non-IID issue
| AVBA [112] | Client model accuracy | Applicable to model poisoning attacks
| FL Trust [20] | Cosine similarity+Clipping | Clean&IID evaluation dataset, vulnerable to specific attacks
| Sageflow [113] | Loss value | Clean&IID evaluation dataset, vulnerable to specific attacks
| Contribution-wise [52] | Cosine sim/Loss value+Clipping | Clean&IID evaluation dataset
Performance-based | Zeno [114] | Loss value (decrease) | A sampling method for collecting evaluation dataset
| DGDA [115] | Loss value variant | Applicable to attacks bring large perturbation
| SCL [116] | Client cross validation | Extra communication and computation cost
| PDGAN [117] | GAN generates evaluation data | Extra training iterations
| HSCS FL [118] | Client model accuracy on class | Applicable to label flipping attacks
| Byzantine ClusterFL [119] | Cosine similarity | Malicious clients should be grouped as the largest group
| Mini FL [94] | Non-IID feature+K means | Non-IID feature should be reliable and explicit
| RFL in HE [120] | Trimmed-K means | Applicable to attacks bring large perturbation
Clustering-based | Robust-DPFL [121] | Mean value+K means | Mainly applicable to Attack-DPFL [121] attacks
| Fair attack detection [122] | Feature+Gaussian mixture model | Clients have to sacrifice a part of privacy
| ZeKoC [123] | Euclidean distance+K means | Evaluation dataset, Applicable to model poisoning attacks
| DAFLA [37] | Principal component analysis | Non-IID issue, Applicable to label flipping attacks
Training function | RSA [124] | Regularization-based loss function | Assumptions of general stochastic gradient-based methods
optimization-Based | HLMA [125] | Huber loss variant | (extreme) Non-IID issue
RLR [18] | Learning rate adjustment | Applicable to backdoor attacks

Pruning is another model compression technique that can eliminate redundant or
poisoned model neurons. Research [126] introduces a new pruning technique
through trimming and fine-tuning the model returned by attackers to defend
against backdoor poisoning attacks on the training set and communication
bottleneck attacks. Recent research [127] proposes a PruneFL method with
adaptive and distributed parameter pruning in FL settings. Through employing
PruneFL, the model can achieve a similar learning performance while reducing
computational costs and shortening the learning time. Inspired by the coding
theory [128], [129] proposes the DRACO framework for robust distributed
training. In the DRACO [129] framework, each client evaluates redundant
gradients, which the parameter server subsequently employs to counteract the
impact of adversarial updates. Similarly, DETOX [130] has also been introduced
based on redundancy to provide a more robust gradient aggregation. DETOX
algorithm [130] includes two steps: a filtering step to reduce the effect of
byzantine nodes by using limited redundancy and a hierarchical aggregation
step that could be used in tandem with other existing robust aggregation
methods. CrowdGuard [131] further enhances the pruning scheme by introducing
client cross-validation. Specifically, suspicious client models are marked as
malicious and pruned during aggregation to prevent backdoor attacks.

###  V-E Byzantine-Resilient Aggregation

Linear combination rules, including averaging, have been demonstrated to lack
resilience against byzantine attacks [33]. In particular, a single malicious
worker can corrupt the global model and even prevent global model convergence.
To defend against various attacks, byzantine-resilient aggregation methods
have been proposed to replace the averaging aggregation and enhance FL
robustness.

####  V-E1 Statistical Feature-Based Robust Aggregation

When introducing perturbation in the model updates, malicious updates are
unavoidable to deviate more from other normal updates in FL. Based on this
insight, some research designs statistical-based aggregators on the server
side to aggregate the most honest clients’ gradients and keep the global model
away from potential malicious updates. For instance, Krum (multi-Krum) [33]
regards the client (could be a small number of clients) that is closest to all
participants as the most honest and uses its (them) gradient as the
aggregation outcome. Specifically, Krum first calculates the L2 (Euclidean)
distance between the gradient to others and consequently ranks the distance.
The gradient achieving the shortest distance value will act as the global
gradient and update the global model. Trimmed mean [34] and Median [34]
enhance the FL robustness by selecting the mean and median value of each
gradient parameter; these coordinates are subsequently organized as the
aggregation outcome. To avoid the aggregation outcome being impacted by the
extreme value, Trimmed mean first discards a preset percentage of extremum.
Bulyan [105] combines the Krum and Trimmed-mean algorithms. Bulyan first
selects a number of gradients as candidates by repeatedly performing Krum. and
removes them from the candidate pool. Then, Bulyan performs Trimmed-mean among
the candidates and generates the outcome to update the global model.

However, these previously mentioned methods operate under the strong
assumption that the defender is aware of the adversary’s volume and
distribution, an assumption that may be challenging to apply in practical
scenarios. Research [107] addresses this limitation by proposing the Sniper
scheme. Sniper [107] does not require prior knowledge of the number of
malicious users and identifies benign local model updates by solving the
Maximum Clique Problem (MCP). Similarly, study [132] relies on the Markov
model with median and cosine similarly to discard the suspicious local model
updates in each iteration and does not require the knowledge of the adversary
either.

####  V-E2 Local Model Performance-Based Robust Aggregation

To evaluate the honesty of clients’ model updates and limit the participation
of suspicion, some recent studies introduce an evaluation dataset to decide
the aggregation weight of each client. For instance, FL Trust [20] proposes a
novel solution by keeping a server-version model. Under FL Trust, the server
first collects a small and clean dataset as the training dataset. Then, the
server normally keeps and trains the model (called the server-version model)
in each iteration. Once the clients’ gradients are received, the server
calculates the cosine similarity between them and the server-version gradient;
the gradient achieving high cosine similarity will subsequently gain a high
weight in aggregation. On the other hand, FL Trust normalizes the magnitudes
of clients’ gradients through the server-version model update to defend
against scaling attacks. Similarly, Sageflow [113] evaluates the entropy of
each gradient received through the evaluation dataset and subsequently divides
the weight to different clients based on the entropy generated. Sageflow also
considers the volume of training data in each client; the client owning a
large data size can gain a large weight in the aggregation and vice versa.
Research [114] proposes a score-based ranking mechanism, namely Zeno. Zeno
suspects clients that are potentially defective, which is tolerant to an
arbitrary number of attackers with at least one benign client. Based on Zeno,
Zeno++ [133] has been introduced to further mitigate the communication
bottleneck, allowing for asynchronous communication. Research [115] generates
the gradient of noise based on a small clean dataset. The noise gradient is
subsequently compared with the clients’ model updates to identify and filter
out suspicious gradients.

Although the existing byzantine-robust aggregation can achieve robustness
under particular assumptions like independent and identically distributed
(IID) distribution of overall training data, the knowledge of clients’ data
distribution, or a set of clean public data, these assumptions can not be
guaranteed in the practical FL scenarios, which usually results in degrading
defense performance and leaves opportunities for state-of-the-art attacks to
poison the model.

####  V-E3 Clustering-Based Robust Aggregation

In the FL context, clients may have different preferences, making it
challenging for a single model to fit all clients’ data-generating
distributions at the same time. Based on this insight, Clustered Federated
Learning (ClusterFL) [134] has been introduced, leveraging the geometric
properties of the FL loss surface to group the client population into clusters
with jointly trainable data distributions. The study [119] further extends
ClusterFL in the adversarial environment. Under standard ClusterFL [134],
cosine similarity is used to determine the cluster of each client, the client
distribution ensures that the cosine similarity is maximized between clients
within the same cluster and minimized between different clusters. Byzantine-
robust ClusterFL [119] employs the same method to divide clients but considers
the largest cluster to be benign while treating all other clusters as
malicious. As a result, the smaller clusters are discarded during aggregation.
The study [94] considers the non-independent and identically distributed (non-
IID) nature of FL as the primary factor contributing to a large attack
surface, and proposes Mini-FL to achieve attack surface reduction.
Specifically, Mini-FL identifies time, geography, and user features as the
main sources of non-IID data and sets the grouping rules according to the
characteristics of these features. In each iteration, the received gradients
are assigned to different groups and robustly aggregated, respectively. The
aggregation outcomes from each group are then further combined into a global
gradient, which supports the model update. Besides non-IID, study [121] finds
that the Differential Privacy (DP) introduced after local training also
expands the overall attack surface and enables adversarial to perform attacks.
To this end, Robust-DPFL [121] has been introduced to distinguish poisoned
gradients from clean ones. Since malicious gradient elements are naturally
larger than benign ones, Robust-DPFL first averages the element values and
then uses k-means clustering to divide the received gradients into clean (with
smaller values) and poisoned (with larger values) categories.

####  V-E4 Training Function Optimization-Based Robust Aggregation

Different from the aforementioned aggregation rules that filter or limit the
suspicious based on their gradient, training function optimization-based
robust aggregation leverages the DL loss function optimization to achieve
byzantine resilience. Compared with other robust aggregations, training
function optimization-based robust aggregation is still in the early research
stage. Research [124] introduces regularization in the loss function to
prevent the local model from deviating excessively from the global model
during training. Instead of relying on a fixed clipping norm, study [135]
proposes a method with dynamic clip value at the specific quantile estimated
online. Recent work [125] generalizes the Huber loss definition and leverages
the optimal Huber loss to organize the loss function. Although this approach
provides a theoretical guarantee under IID and moderately heterogeneous client
environments, the guarantee may not hold in extreme non-IID settings.

###  V-F Other Emerging Defense Frameworks

Although most byzantine-resilient FL methods [3] follow the setting that the
server (service provider) acts as the defender role, some works consider
defending the poisoning attacks from the client’s perspective. A recent work
[136] proposes a novel algorithm to enhance FL robustness from the
participants’ side, namely White Blood Cell for Federated Learning (FL-WBS).
This study [136] first introduces the Attack Effect on Parameter (AEP) to
evaluate the model poisoning attacks’ impact on global model parameters and
subsequently introduces the FL-WBS framework to prevent AEP from being hidden
in the kernels of Hessian matrices on benign devices. SVeriFL[137] leverages
BLS signature and homomorphic encryption technology to verify the integrity of
the parameters uploaded by the participants and the correctness of the results
returned by the server.

Owing the anonymity and distributed characteristics, enhancing FL robustness
through blockchain has become an appealing research trend [138, 139]. Research
[140] considers mitigating the one-point failure (server be controlled by
malicious) of FL by introducing blockchain technology in the current FL
algorithm. Specifically, research [140] proposes BlockFL, which replaces the
server with a blockchain to relieve the server security issue and achieve
decentralization. In each iteration, the clients normally train the model and
then broadcast the gradients to the miners. The miners cross-verify the
gradients and generate the candidate block until a preset block size is
achieved. Once the block is published, the clients download the block and
locally compute the global model update and consequently utilize the outcome
to update the model. Study [141] proposes a reputation-based FLchain system,
namely BAFL, to achieve asynchronous FL strategy and maintain FL robustness.
Under BAFL [141], the training time, training data size, gradients
correlation, and global update cheating times are generated by the entropy
weight method and are recorded in the blockchain to achieve long-term trust.

Some work [142, 117, 143, 144] consider introducing the defense framework from
traditional centralized ML scenarios to FL settings. For instance, PDGAN [117]
generates testing datasets through GAN technology to defend against Data
Poisoning Attacks. Research [145] proposes Moving Target Defense to increase
attacking cost and complexity by changing a system to reduce or move the
attack surface available for exploitation by the adversary. However, most of
these studies [145, 143] may show effectiveness in Horizontal FL (HFL)
settings but face new challenges when deploying in Vertical FL (VFL)
scenarios.

##  VI Defense Against Privacy-harmed Attacks

Unlike utility-oriented defense frameworks that generally assume a trusted
central server, defenses against privacy-oriented attacks can be implemented
by either the server or the clients, depending on different settings. Today,
various defense frameworks have been proposed to mitigate privacy concerns,
ranging from cryptography-based methods to fully decentralized architectures.
In this section, we survey these defense solutions, introduce their
techniques, and compare their advantages and limitations.

###  VI-A Differential Privacy

Differential Privacy (DP) has been proposed as a privacy protection data-
sharing method. By introducing artificial noise, DP guarantees statistical
queries on a database without disclosing information specific to individuals
[146]. In the FL system, DP provides a privacy guarantee that clients’ data
and information cannot be recovered during the training and user access
phases.

####  VI-A1 Centralized Differential Privacy (CDP)

When the server is trusted, noise can be added centrally to the aggregated
local gradients (known as CDP) to ensure record-level privacy for the entire
dataset of all participants. Formally, the definition of CDP is presented as
follows:

Definition 1 (($\epsilon$,$\delta$)-CDP [146]): The algorithm $\mathcal{A}$
satisfies ($\epsilon$,$\delta$)-DP, if any pair of adjacent datasets $D$,
$D^{\prime}\in\mathcal{D}^{n}$ and outputs $S\in$range$(\mathcal{A})$ we have:

$Pr[\mathcal{A}(D)\in S]\leq e^{\epsilon}\cdot Pr[\mathcal{A}(D^{\prime})\in
S]+\delta$

Here, $\epsilon>0$ controls the level of privacy guarantee in the worst case,
and the factor $\delta\geq 0$ represents the failure probability that the
privacy guarantee does not hold. In practice, the value of $\delta$ should be
negligible, such that $\delta\ll 1/||D||$.

For the first time, the study [147] introduces CDP in FL to conceal clients’
contributions during training. Subsequently, DP-FedAvg and DP-FedSGD [148]
extend CDP-FL to LSTM language models, achieving results that are
quantitatively and qualitatively similar to those of un-noised learning
algorithms. To reduce the impact of the DP mechanism, DP-FedSAM [149]
leverages Sharpness-Aware Minimization (SAM) optimization to search for areas
of low loss in the loss landscape and improve the model’s generalization
ability. By sparsifying local updates and excluding the softmax layer from
uploading, studies [150] and [151] also achieve a reduction in the volume of
injected noise and enhance the performance of CDP-FL. Considering that both
data features and label spaces could be regarded as private in VFL scenarios,
HDP-VFL [152] introduces Gaussian noise on both the server and client sides to
ensure the data confidentiality of VFL participants.

While CDP-FL demonstrates effectiveness in privacy protection, it faces
limitations from two main perspectives [153]: (i) CDP-FL algorithms generally
rely on a large number of participants to achieve convergence, learning
scenarios with a small number of participants may result in significant
performance degradation. (ii) The assumption of a trustworthy server may not
hold in many real-world settings.

####  VI-A2 Local Differential Privacy (LDP)

Different from CDP, LDP introduces the DP noise from the participant side to
guarantee data privacy. As the gradient left from the participant surface is
noised, LDP does not assume a trusted server and potential defenses against
eavesdropping. Formally, the definition of LDP is presented as follows:

Definition 2 (($\epsilon$,$\delta$)-LDP [154]): The algorithm $\mathcal{A}$
satisfies ($\epsilon$,$\delta$)-DP, if any input $d$, $d$ and $\forall
t\in$range$(\mathcal{A})$ we have:

$Pr[\mathcal{A}(d)=t]\leq e^{\epsilon}\cdot
Pr[\mathcal{A}(d^{\prime})=t]+\delta$

Early LDP-FL works [78, 155] required clients to locally introduce noise to
model parameters before releasing them to the server. However, with the
explosive growth of model size and dimensions, the privacy budget allocated to
each dimension decreases, preventing these methods from providing an expected
privacy guarantee. On one hand, some studies propose new mean estimation
techniques for high-dimensional data to enhance the performance of LDP-FL,
including Piecewise Mechanism (PM) [152, 156] and Hybrid Mechanism (HM) [152].
On the other hand, studies that leverage dimension sampling have also shown
effectiveness in mitigating the curse of dimensionality in LDP-FL [157]. For
instance, PPeFL [158] introduces a filtering and screening method through an
exponential mechanism (EM). Based on the contribution of weight parameters to
the neural network, PPeFL filters out the better parameters using EM, thereby
addressing the parameter dimension dependence problem to some extent when LDP
is applied to complex ML tasks.

Because of the different perturbation introduced strategies, CDP and LPD show
different robustness when facing the same threats. Study [159] compares these
two methods and finds that: (i) Both LDP and CDP protect against white-box MIA
and do not necessarily incur a high cost on utility. (ii) LDP does not work
against PIA. Although CDP can defend against such attacks, it suffers from a
significant model utility degradation. We follow the study and compare the
representative DL-FLs in Table VI. Meanwhile, study [157] provides a detailed
table of DP-FLs from other perspectives.

TABLE VI: Illustration and comparison of different DP-FL methods. DP Category | Algorithm | FL settings | Non-IID | Perturbation | Model | Learning Task Dataset
---|---|---|---|---|---|---
CDP | DP-FL[160] | HFL |  | Gaussian | CNN | MNIST, Cifar-10, Iris
UDP-FL[161] | HFL |  | Gaussian | SVM, MLP, CNN | IPUMS-US, MNIST, Cifar-10
Federated f-DP[162] | HFL |  | Gaussian | CNN | MNIST
FEDMD-NFDP[163] | TFL |  | Sampling | MLP | MNIST, Cifar-10, -100 (variants)
DynamicPFL[164] | HFL |  | Gaussian | CNN | Cifar-10, FEMNIST, SVHN
| 2DF-IDS[162] | HFL (DFL) |  | Gaussian | DNN | Edge-IIoTset
| VFL-Chain[165] | VFL (DFL) |  | Laplace | N/A | MNIST, BelgiumTS
LDP | DP-FedSAM[166] | HFL |  | Gaussian | CNN, ResNet-18 | EMNIST, Cifar-10, -100
| FL2DP[167] | HFL |  | Exponential | CNN | MNIST, Cifar-10
| ProxyFL[168] | HFL (DFL) |  | Gaussian | LeNet5, MLP, CNN | Fashion-, MNIST, Cifar-10
Hybrid | iDP-FL[169] | HFL |  | Gaussian | MLP | Fashion-, MNIST, Cifar-10
PPFL[170] | HFL |  | Gaussian, Laplace | MLP | MNIST
Not applicable,  Applicable to limited settings,  Applicable

###  VI-B Decentralized Learning Scheme

####  VI-B1 Blockchain

Since curious central servers are one of the main sources of privacy-oriented
attacks, recent studies have considered leveraging blockchain as a
decentralized alternative to replace the central server [171].

The blockchain-based FL (BCFL) [140, 172, 173, 174] generally follow the
learning process shown as the Figure 13. Specifically, the clients locally
train the models and upload gradients to associated miners. Miner broadcasts
received gradients and records in the candidate block after cross-
verification. Once a preset condition is met, each miner runs a cryptographic
proof, and the first successor propagates its block. Finally, all participants
download the block from the associated miners and update the global model
accordingly. As the whole learning process does not require centralized
collected gradients, blockchain-based FLs are potentially robust against
inversion attacks.

To enhance scalability, studies [175, 176, 17] introduce smart contracts,
which serve as executable code embodying mutual agreements between two or more
parties in blockchain-based FL systems. For instance, the study [175] utilizes
active smart contracts to automatically monitor, organize, and facilitate
secure data transfer, validating the system’s efficacy. Similarly, BAFFLE
[176] uses smart contracts to decompose global parameters into different
blocks, employing strategies to perform model aggregation and update tasks. To
mitigate the storage bottleneck of blockchain, PriModChain [17] introduces the
Inter Planetary File System (IPFS) to blockchain-based FL. Under PriModChain,
IPFS provides extensive decentralized storage, while the blockchain stores
only the target addresses of the IPFS.

Besides enhancing privacy, blockchain also benefits the FL system from an
incentive perspective. The study [177] proposes a completely decentralized
framework using blockchain technology to reward members with high-performance
or high-quality data, encouraging voluntary contributions and potentially
enhancing model performance.

Figure 13: The learning process of blockchain-based FL.

####  VI-B2 Decentralized Federated Learning (DFL)

To mitigate the privacy leakage risk of FL, another research pipeline is that
transfer the centralized FL to decentralized (shown as Figure 4). The study
[178] first proposes a DFL scheme, where clients directly update their beliefs
by aggregating information from their one-hop neighbors to learn a model that
best fits the observations. Study [168] further introduces the KD in DFL and
proposes a ProxyFL scheme. Under ProxyFL, each client owns both a student
model and a teacher model, which are responsible for learning knowledge and
model exchange, respectively. Since the student model is not shared during the
training process, ProxyFL provides property (model as property) and privacy
protection for the model architecture and trained parameters.

While DFL offers a novel solution for addressing FL privacy concerns, existing
algorithms generally assume that clients have reliable and stable networks and
cannot tolerate stragglers, which hinders their deployment in the real world.

###  VI-C Homomorphic Encryption

Homomorphic encryption (HE) is a form of encryption that enables operations to
be performed on encrypted data, and guarantees the encrypted computations
results identical to the output of operations performed on the unencrypted
data. Formally, HE satisfies the following equation [72]:

$Dec[Enc(g_{1})\odot Enc(g_{2})]=g_{1}\odot g_{2}$ (3)

Here, $Enc(\cdot)$ and $Dec[\cdot]$ represent encryption and decryption,
respectively, and $g$ represents the plaintext, which is typically the
gradient in the FL context. Based on the limitations of the operations $\odot$
(addition $\oplus$ or multiplication $\otimes$), HE could be categorized into:
Fully HE (FHE), Somewhat HE (SWHE), and Partially HE (PHE). We compare the
three different types of HE and summarize their characteristics in Table VII.

TABLE VII: Illustration of homomorphic encryption methods and applied HE-FL
schemes.

Homomorphic Encryption | Operation times | $\oplus$ | $\otimes$ | Computation cost | Summary
---|---|---|---|---|---
FHE [8, 179, 180] | Unlimited | ✓ | ✓ | High |  High privacy protection, but the high computation cost limits it be applied in the real world applications.
SWHE [181, 182] | Limited | ✓ | ✓ | Medium | Operations may introduce noise, applicable to specific scenarios.
PHE (AHE) [183, 184, 185] | Unlimited | ✓ | $\times$ | Medium to Low |  Simple in design but limited in functionality, applicable across various scenarios.
PHE (MHE) [186] | Unlimited | $\times$ | ✓

subtableComparison of the different types of homomorphic encryption.

Schemes | Operation | Method |  Technique & Remark
---|---|---|---
Liu et al.[187] | Paillier | PHE |  Leveraging the additive properties of PHE to protect models from exposure to the fusion server.
BatchCrypt[182] | Paillier | PHE |  Batch process multiple gradients into a large integer and encrypt it with HE.
FLashe [188] | Paillier | PHE |  Using symmetric keys to accelerate HE encryption and decryption.
Li et al. [8] | Paillier | PHE |  Implement a dual-server model, leveraging a proxy re-encryption scheme with additive homomorphic properties.
Kumar et al. [179] | CKKS | FHE |  Federated Cryptography Defense, implementing FHE through row-based cryptographic transformation.
Poseidon [180] | Paillier | FHE |  Lattice-based HE enables Single Instruction, Multiple Data (SIMD) operations on encrypted data
Rieyan et al. [189] | Paillier | PHE |  Storage and analysis of medical image privacy via data fabric architecture.
Han et al. [190] | Paillier | PHE |  Adaptive batch encryption considers both model privacy and heterogeneity issues.
Hijazi et al. [191] | CKKS | FHE |  Addressing privacy issues in IoT-enabled smart cities by combining FHE and FL.
Tian et al. [192] | FV | PHE |  Lattice-based distributed threshold AHE uses secure aggregation protocols to defend against semi-trusted adversaries.

subtableComparison of different HE-FL schemes.

Study [183] introduces Additive Homomorphic Encryption (AHE) in the FL systems
to protect the privacy of all participants. Specifically, all clients agree on
a key pair, with the public key used for encrypting gradients before transfer
and the private key used for decrypting the received gradients from the
server. Similarly, study [184] follows the AHE scheme but leverages PPDM [193]
for encrypting to mitigate the efficiency bottleneck. Although these works
provide privacy protection against a curious server, they face several main
limitations. First, the proposed schemes are not robust against collusion
between the server and even a single participant. Second, the encryption
process incurs a large amount of computational overhead, which cannot be
afforded by most lightweight edge devices.

To address the collusion issues, HE-based multi-key aggregation protocol
(MKAgg) [8] has been proposed. The protocol is built on a dual-server model
and uses proxy re-encryption technology to convert ciphertexts of different
public keys into ciphertexts under a public key. From a multiple encryption
perspective, PEFL [194] embeds the homomorphic ciphertext of private gradients
into augmented learning with errors (A-LWE) term to defend against attacks
performed through collusion. By employing multiparty lattice-based
cryptography, POSEIDON [180] further increases the tolerance to collusion
among up to $N-1$ parties.

To reduce the formidable overhead associated with encrypting all gradient
parameters, MASKCRYPT [195] has been proposed. Instead of encrypting entire
model updates, MASKCRYPT applies an encryption mask to select a small portion
of the updates for encryption. Compared to encrypting the entire model
updates, MASKCRYPT achieves a 4.15× reduction in communication overhead. A
recent study, Federated Cryptography Defense [179], introduces KD in HE to
enhance robustness against both evasion black-box data poisoning and model
inversion attacks. Instead of gradient space, FCD transfers the entire
learning process into an encrypted data space and leverages a Kullback-Leibler
(KL) divergence-based distillation loss to support the KD process.

The huge communication and encryption overhead are long-term bottlenecks for
HE-FL schemes. Today, many studies have proposed methods to reduce these
encryption and communication costs and to mitigate this research gap. We
summarize these studies in Table VII.

###  VI-D Secure Multi-Party Computation (SMPC)

SMPC was first proposed by Yao [196] to address the problem known as the ”two
millionaires problem,” where two millionaires are interested in knowing which
of them is richer without revealing their actual wealth. In the FL context,
SMPC provides potential solutions to the problem of securely computing an
agreed-upon function without a trusted server, ensuring that each participant
learns only the final computation result without gaining any additional
information about others.

To achieve SMPC, the study [197] proposed secure aggregated FL (namely PPML)
based on a Secret Sharing and Key Agreement Protocol. Under PPML, clients’
gradients are divided into multiple encrypted shares and then aggregated once
a preset number of shares are collected. Since the proposed PPML does not
require the gradients of all clients, it can potentially benefit FL by
tolerating stragglers. The sharing ($SS.share(\cdot)$)and reconstruction
($SS.recon(\cdot)$) processes of secret sharing shown as following, where $S$
represents the secret, $u$ is the client that belongs to all participants $U$
and $t$ is the threshold for reconstructing $S$:

$SS.share(s,t,U)\rightarrow S_{u},u\in U$ (4) $SS.recon\\{(s_{u}),u\in
U,t\\}\rightarrow S$ (5)

MLGuard [198] further enhances the robustness of secret sharing-based FL to
poisoning attacks. In each iteration, clients split their local model updates
into two parts using secret sharing and send them to two servers separately.
After excluding suspicious gradients with low cosine similarity, each server
aggregates the shares received and then collaborates with the other server to
generate the global model. Considering resource-constrained edge devices, the
study [199] proposes an efficient and secure SMPC-FL algorithm. This algorithm
uses compressed sensing technology to compress transmitted data and introduces
additive secret sharing (ASS) and functional encryption (FE) to construct a
new inner product functional encryption for executing the SMPC-FL algorithm
(FL-IPFE).

While multiple SMPC-FL studies have demonstrated effectiveness in privacy
protection, most of these works incur significant computation and
communication costs. Effectively deploying SMPC in network- / device-
heterogeneous FL scenarios remains an open research question.

###  VI-E Trusted Execution Environment (TEE)

TEE has been proposed to protect the security of data and computing in
untrusted remote environments. By creating a secure area through the isolation
of certain software and hardware resources, TEE ensures that the
confidentiality, integrity, and measurability of programs and data running
within it are protected from external interference.

Under FL architectures, TEE allows local models and the aggregation process to
be loaded in a trusted and private environment (e.g., Intel SGX [200]), which
mitigates the information leakage and potentially defend against gradient
crafting (i.e., model poisoning attacks) and training skipping (i.e., free-
riding attacks), mitigating the information leakage [170, 201]. However,
besides suffering from side-channel attacks, current secure enclaves are also
limited in memory and provide access only to CPU resources, which conflicts
with the GPU computing requirements needed for large ML models and complex
learning tasks. Thus, it remains an open question how to effectively deploy FL
across secure enclaves, edge computing resources, and client devices.

##  VII Challenges and Future Directions

Based on the threats and defenses discussed above, we summarize the
effectiveness of various defense methods against different FL threats in Table
VIII. It can be observed that most existing defense solutions are effective
against limited threats and fail to provide comprehensive protection for both
the utility and privacy of FL. In this section, we remark on these threats and
defenses, discuss their opportunities and bottlenecks, and explore future
research trends for the trustworthy FL community.

TABLE VIII: Illustration of the effectiveness of various defense methods against different FL threats. Defends Threats |  Poisoning |  Promise Break |  Sybil |  Communi |  Evasion |  Inference |  Inversion | Eavesdrop
---|---|---|---|---|---|---|---|---
Data Sanitization |  |  |  |  |  |  |  |
Anomaly Detection |  |  |  |  |  |  |  |
Adversarial Training |  |  |  |  |  |  |  |
Byzantine-R Aggregation |  |  |  |  |  |  |  |
Model Compression |  |  |  |  |  |  |  |
Trusted Execution E |  |  |  |  |  |  |  |
Differential Privacy |  |  |  |  |  |  |  |
Homomorphic Encryption |  |  |  |  |  |  |  |
Blockchain |  |  |  |  |  |  |  |
Decentralized FL |  |  |  |  |  |  |  |
Secure Multi-parity Comp |  |  |  |  |  |  |  |
represent the method that can effectively, partially, or can not defend
against the target attacks. Based on different capacities of the adversary,
the defend methods could show different performances; thus, we use two marks
to represent a range of its robustness.

###  VII-A Large FL Attack Surface Reduction

To effectively degrade model performance and avoid detection, malicious
clients must introduce sufficient perturbation to their model updates while
ensuring that the crafted gradients remain similar to benign ones [11].
Essentially, the difference between benign gradients can be considered a
surface within which attackers can craft their gradients without being
detected. A large attack surface provides opportunities for attacks and poses
challenges for defenders [94]. In FL, this large attack surface primarily
stems from two factors: (i) the high dimensionality of the model and (ii) the
non-IID data distribution among participants. Therefore, reducing the FL
attack surface by addressing these two aspects could be a promising direction
for future research.

###  VII-B New Perspective for Attacks

FL involves multiple participants collaborating to train a single joint model,
making fairness across participants an increasingly important focus in recent
studies [202]. These fairness enhancement solutions aim to guarantee similar
representation across different data distributions, ensure that different
participants have an equal probability of being selected, and provide
reasonable rewards. However, most of these solutions assume the honesty and
trustworthiness of participants, which may not hold in real-world scenarios,
thereby creating new opportunities for attacks. Given that fairness-oriented
approaches are still in their infancy, it is worthwhile to explore their
vulnerabilities and develop corresponding defense solutions.

###  VII-C Attackers vs Disadvantaged Clients

Due to the non-IID nature, FL participants are usually heterogeneous in terms
of computing capacity, data quality, data distribution, and bandwidth. For
instance, a UAV deployed in a high-altitude area may capture more unique
photos compared to a UAV in urban environments. Despite acting honestly, it
may be mistakenly identified as malicious if its gradients exhibit low
evaluation performance or deviate significantly from others. This
misidentification occurs because current Byzantine-resilient aggregation
methods identify malicious clients based on the performance or statistical
values of their gradients. Notably, unusual data distributions often carry
valuable information for model training. Therefore, it is important to study
how to distinguish disadvantaged clients from attackers and ensure their
participation.

###  VII-D Federated Unlearning

Current FL defense methods emphasize analyzing clients’ behavior or gradient
performance in the current learning iteration to evaluate their honesty.
However, many attacks can achieve their objectives (e.g., introducing a
backdoor) over several learning rounds. In such cases, excluding a suspicious
client from only the current learning iteration may still allow the introduced
perturbations to remain in the joint model. Federated unlearning [203] has
emerged as a new technology, originally designed to eliminate the
contributions of a client once they leave the collaboration. By introducing
federated unlearning, defenses are expected to resolve the issue of residual
poisoning in the model.

###  VII-E Research on VFL, FTL, and DFL

Existing research on trustworthy FL mainly focuses on the HFL setting.
However, since VFL, FTL, and HFL have different shared feature spaces, defense
methods for HFL may not be directly applicable to VFL and FTL systems, and
their effectiveness remains unclear. Furthermore, due to their different label
spaces, FTL and TFL also present specific attack opportunities (e.g., overall
feature space exposure and sample ID leakage). Overall, there remains a
significant gap in research for VFL and FTL.

Furthermore, as an emerging learning paradigm, the research on DFL is still in
its infancy. Without a centralized node, traditional byzantine-robust
aggregation methods are potentially inapplicable. Enhancing the robustness of
DFL requires further investigation.

###  VII-F Emerging Privacy Requirements

Traditional privacy requirements in FL focus on the information at the
training data level (e.g., original data samples) or client level (e.g.,
participation). However, with the evolution of machine learning, current
models have become much larger, requiring extensive costs for training
(consider the server distributing incentives to clients). Additionally, model
architectures are often well-tuned to achieve high learning performance.
Consequently, well-trained models and elaborate-tuned model architectures have
become new objectives for privacy protection [168], which servers may be
reluctant to share during training. Therefore, it is essential to explore new
privacy requirements and design new FL algorithms that fulfill these emerging
requirements.

###  VII-G Both Privacy- and Security- Enhanced FL algorithms

Although various FL algorithms have been proposed, most of these methods focus
on single issues and fail to address both privacy and security concerns
simultaneously. Unfortunately, directly combining different techniques may not
result in a cumulative effect and even introduce new attack opportunities. For
instance, DP has been shown to potentially enlarge the attack surface [94],
which may weaken the effectiveness of statistical-feature-based Byzantine-
robust aggregation methods. This area remains largely unexplored, and there
are significant gaps in achieving FL algorithms that enhance both privacy and
security.

##  VIII Conclusion

FL is an innovative ML paradigm that allows multiple clients to
collaboratively train a model without sharing their raw data. Due to its
inherent privacy-preserving nature, FL has the potential to safeguard data
privacy, effectively transforming data from ”visible and usable” to ”invisible
yet usable.” This capability extends the applicability of data-driven ML
models to real-world scenarios. In this paper, we comprehensively survey the
most significant and cutting-edge threats and defense mechanisms throughout
the entire FL lifecycle. Our detailed discussion and comparison enhance the
understanding of the attack and defense dynamics in FL. In the section on
future research trends, we highlight areas within FL that necessitate thorough
research and exploration. This survey paper is an invaluable resource for
researchers, designers, and engineers engaged in FL security. It offers
insights into the current challenges and future directions for research in
this evolving field.

## References

  * [1] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: vision, hype and reality for data privacy and protection,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 1, pp. 3347–3366, 2021.
  * [2] H. Ye, L. Liang, and G. Y. Li, “Decentralized federated learning with unreliable communications,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 3, pp. 487–500, 2022.
  * [3] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021.
  * [4] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics (AISTATS)_. PMLR, 2017, pp. 1273–1282.
  * [5] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” _arXiv preprint arXiv:1811.03604_ , 2018.
  * [6] Apple, “Designing for privacy (video and slide deck), wwdc,” https://developer.apple.com/videos/play/wwdc2019/708, 2019.
  * [7] WeBank, “Webank and swiss re signed cooperation, mou,” https://finance.yahoo.com/news/webank-swiss-signed-cooperation-mou-112300218.html, 2019.
  * [8] Y. Li, J. Lai, R. Zhang, and M. Sun, “Secure and efficient multi-key aggregation for federated learning,” _Information Sciences_ , vol. 654, p. 119830, 2024.
  * [9] G. Damaskinos, E.-M. El-Mhamdi, R. Guerraoui, A. Guirguis, and S. Rouault, “Aggregathor: Byzantine machine learning via robust gradient aggregation,” in _Machine Learning and Systems (MLSys)_. ACM, 2019, pp. 81–106.
  * [10] E.-M. El-Mhamdi, R. Guerraoui, A. Guirguis, L.-N. Hoang, and S. Rouault, “Genuinely distributed byzantine machine learning,” _Distributed Computing_ , vol. 35, pp. 305–331, 2022.
  * [11] M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in _USENIX Security Symposium (USENIX Security)_. USENIX Association, 2020, pp. 1605–1622.
  * [12] X. Gong, Y. Chen, Q. Wang, and W. Kong, “Backdoor attacks and defenses in federated learning: State-of-the-art, taxonomy, and future directions,” _IEEE Wireless Communications_ , vol. 30, no. 2, pp. 114–121, 2023.
  * [13] Y. Li, Y. Jiang, Z. Li, and S.-T. Xia, “Backdoor learning: A survey,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 1, no. 1, pp. 1–18, 2022.
  * [14] A. Baghbani, T. Choudhury, S. Costa, and J. Reiner, “Application of artificial intelligence in geotechnical engineering: A state-of-the-art review,” _Earth-Science Reviews_ , vol. 228, no. 1, pp. 1–26, 2022.
  * [15] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _2017 IEEE symposium on security and privacy (SP)_. IEEE, 2017, pp. 3–18.
  * [16] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in _2019 IEEE symposium on security and privacy (SP)_. IEEE, 2019, pp. 739–753.
  * [17] S. Zhang and J. Zhu, “Privacy protection federated learning framework based on blockchain and committee consensus in iot devices,” in _2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)_. IEEE, 2023, pp. 627–636.
  * [18] M. S. Ozdayi, M. Kantarcioglu, and Y. R. Gel, “Defending against backdoors in federated learning with robust learning rate,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 10, 2021, pp. 9268–9276.
  * [19] R. Taheri, R. Javidan, M. Shojafar, Z. Pooranian, A. Miri, and M. Conti, “On defending against label flipping attacks on malware detection systems,” _Neural Computing and Applications_ , vol. 32, pp. 14 781–14 800, 2020.
  * [20] X. Cao, M. Fang, J. Liu, and N. Z. Gong, “Fltrust: Byzantine-robust federated learning via trust bootstrapping,” in _ISOC Network and Distributed System Security Symposium (NDSS)_. ISOC, 2021, pp. 1–18.
  * [21] G. Xia, J. Chen, C. Yu, and J. Ma, “Poisoning attacks in federated learning: A survey,” _IEEE Access_ , vol. 11, pp. 10 708–10 722, 2023.
  * [22] J. Wen, Z. Zhang, Y. Lan, Z. Cui, J. Cai, and W. Zhang, “A survey on federated learning: challenges and applications,” _International Journal of Machine Learning and Cybernetics_ , vol. 14, no. 2, pp. 513–535, 2023.
  * [23] V. Mothukuri, R. M. Parizi, S. Pouriyeh, Y. Huang, A. Dehghantanha, and G. Srivastava, “A survey on security and privacy of federated learning,” _Future Generation Computer Systems_ , vol. 115, pp. 619–640, 2021.
  * [24] Y. Gao, X. Chen, Y. Zhang, W. Wang, H. Deng, P. Duan, and P. Chen, “A survey of attack and defense techniques for federated learning systems,” _Chinese Journal Of Computes_ , pp. 1–25, 2023.
  * [25] Z. Guo, W. Li, Y. Qian, O. Arandjelovic, and L. Fang, “A white-box false positive adversarial attack method on contrastive loss based offline handwritten signature verification models,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2024, pp. 901–909.
  * [26] K. N. Kumar, C. K. Mohan, and L. R. Cenkeramaddi, “The impact of adversarial attacks on federated learning: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [27] X. Zhou, M. Xu, Y. Wu, and N. Zheng, “Deep model poisoning attack on federated learning,” _Future Internet_ , vol. 13, no. 3, pp. 1–14, 2021.
  * [28] K. Wei, J. Li, M. Ding, C. Ma, Y.-S. Jeon, and H. Vincent Poor, “Covert model poisoning against federated learning: Algorithm design and optimization,” _IEEE Transactions on Dependable and Secure Computing_ , pp. 1–14, 2023.
  * [29] H. Xiao, H. Xiao, and C. Eckert, “Adversarial label flips attack on support vector machines,” in _European Conference on Artificial Intelligence (ECAI)_. IOS Press, 2012, pp. 870–875.
  * [30] X. Li, N. Wang, S. Yuan, and Z. Guan, “Fedimp: Parameter importance-based model poisoning attack against federated learning system,” _Computers & Security_, p. 103936, 2024.
  * [31] G. Baruch, M. Baruch, and Y. Goldberg, “A little is enough: Circumventing defenses for distributed learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2019, pp. 1–11.
  * [32] C. Xie, O. Koyejo, and I. Gupta, “Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation,” in _Uncertainty in Artificial Intelligence (UAI)_. PMLR, 2020, pp. 261–270.
  * [33] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2017, pp. 1–11.
  * [34] D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in _International Conference on Machine Learning (ICML)_. PMLR, 2018, pp. 5650–5659.
  * [35] S. P. Karimireddy, L. He, and M. Jaggi, “Byzantine-robust learning on heterogeneous datasets via bucketing,” in _International Conference on Learning Representations (ICLR)_ , 2022.
  * [36] V. Shejwalkar and A. Houmansadr, “Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning,” in _NDSS_ , 2021.
  * [37] V. Tolpegin, S. Truex, M. E. Gursoy, and L. Liu, “Data poisoning attacks against federated learning systems,” in _European Symposium on Research in Computer Security (ESORICS)_. Springer, 2020, pp. 480–501.
  * [38] C. Lewis, V. Varadharajan, and N. Noman, “Attacks against federated learning defense systems and their mitigation,” _Journal of Machine Learning Research_ , vol. 24, no. 30, pp. 1–50, 2023.
  * [39] H. Zhang, N. Cheng, Y. Zhang, and Z. Li, “Label flipping attacks against naive bayes on spam filtering systems,” _Applied Intelligence_ , vol. 51, pp. 4503–4514, 2021.
  * [40] R. Sharma, G. Sharma, and M. Pattanaik, “A catboost based approach to detect label flipping poisoning attack in hardware trojan detection systems,” _Journal of Electronic Testing_ , vol. 38, no. 6, pp. 667–682, 2022.
  * [41] T. D. Nguyen, P. Rieger, M. Miettinen, and A.-R. Sadeghi, “Poisoning attacks on federated learning-based iot intrusion detection system,” in _Workshop on Decentralized IoT Systems and Security (DISS)_. ISOC, 2020, pp. 1–7.
  * [42] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in _International Conference on Artificial Intelligence and Statistics (AISTATS)_. PMLR, 2020, pp. 2938–2948.
  * [43] H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, and D. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2020, pp. 16 070–16 084.
  * [44] W. Wan, J. Lu, S. Hu, L. Y. Zhang, and X. Pei, “Shielding federated learning: A new attack approach and its defense,” in _IEEE Wireless Communications and Networking Conference (WCNC)_. IEEE, 2021, pp. 1–7.
  * [45] Y. Fraboni, R. Vidal, and M. Lorenzi, “Free-rider attacks on model aggregation in federated learning,” in _International Conference on Artificial Intelligence and Statistics (AISTATS)_. PMLR, 2021, pp. 1846–1854.
  * [46] B. Pejó and G. Biczók, “Quality inference in federated learning with secure aggregation,” _IEEE Transactions on Big Data_ , vol. 9, no. 5, pp. 1430–1438, 2023.
  * [47] Y. Liu and J. Wei, “Incentives for federated learning: a hypothesis elicitation approach,” _arXiv preprint arXiv:2007.10596_ , 2020.
  * [48] R. Zeng, C. Zeng, X. Wang, B. Li, and X. Chu, “A comprehensive survey of incentive mechanism for federated learning,” _arXiv preprint arXiv:2106.15406_ , 2021.
  * [49] J. Weng, J. Weng, J. Zhang, M. Li, Y. Zhang, and W. Luo, “Deepchain: Auditable and privacy-preserving deep learning with blockchain-based incentive,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 18, no. 5, pp. 2438–2455, 2019.
  * [50] T. Tuor, S. Wang, B. J. Ko, C. Liu, and K. K. Leung, “Overcoming noisy and irrelevant data in federated learning,” in _International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 5020–5027.
  * [51] X. Cao, Z. Zhang, J. Jia, and N. Z. Gong, “Flcert: Provably secure federated learning against poisoning attacks,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 3691–3705, 2022.
  * [52] Y. Li, W. Ding, H. Chen, W. Bao, and D. Yuan, “Contribution-wise byzantine-robust aggregation for class-balanced federated learning,” _Information Sciences_ , vol. 667, p. 120475, 2024.
  * [53] S. Agrawal, M. L. Das, and J. Lopez, “Detection of node capture attack in wireless sensor networks,” _IEEE Systems Journal_ , vol. 13, no. 1, pp. 238–247, 2018.
  * [54] A. Vangala, A. K. Das, A. Mitra, S. K. Das, and Y. Park, “Blockchain-enabled authenticated key agreement scheme for mobile vehicles-assisted precision agricultural iot networks,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 904–919, 2023.
  * [55] X. Yao, C. Huang, and L. Sun, “Two-stream federated learning: Reduce the communication costs,” in _IEEE Visual Communications and Image Processing (VCIP)_. IEEE, 2018, pp. 1–4.
  * [56] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _International Conference on Learning Representations (ICLR)_. IMLS, 2015, pp. 1–11.
  * [57] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2015, pp. 1–9.
  * [58] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations (ICLR)_. IMLS, 2018, pp. 1–23.
  * [59] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” in _International Conference on Learning Representations (ICLR)_. IMLS, 2017, pp. 1–17.
  * [60] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _ACM Workshop on Artificial Intelligence and Security (AISec)_. ACM, 2017, pp. 15–26.
  * [61] W. Brendel, J. Rauber, and M. Bethge, “Decision-based adversarial attacks: Reliable attacks against black-box machine learning models,” in _International Conference on Learning Representations (ICLR)_. IMLS, 2018, pp. 1–17.
  * [62] Q. Pang, Y. Yuan, S. Wang, and W. Zheng, “Adi: Adversarial dominating inputs in vertical federated learning systems,” in _2023 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2023, pp. 1875–1892.
  * [63] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [64] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [65] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients-how easy is it to break privacy in federated learning?” _Advances in neural information processing systems_ , vol. 33, pp. 16 937–16 947, 2020.
  * [66] H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz, “Dreaming to distill: Data-free knowledge transfer via deepinversion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8715–8724.
  * [67] H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov, “See through gradients: Image batch recovery via gradinversion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16 337–16 346.
  * [68] Z. Li, J. Zhang, L. Liu, and J. Liu, “Auditing privacy defenses in federated learning via generative gradient leakage,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 10 132–10 142.
  * [69] A. Hatamizadeh, H. Yin, H. R. Roth, W. Li, J. Kautz, D. Xu, and P. Molchanov, “Gradvit: Gradient inversion of vision transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 10 021–10 030.
  * [70] A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes, “Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models,” in _Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS)_ , 2019.
  * [71] Z. Li and Y. Zhang, “Membership leakage in label-only exposures,” in _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021, pp. 880–895.
  * [72] Y. Gu and Y. Bai, “Survey on security and privacy of federated learning models,” _Ruan Jian Xue Bao/Journal of Software_ , vol. 34, no. 6, pp. 2833–2864, 2023.
  * [73] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _2019 IEEE symposium on security and privacy (SP)_. IEEE, 2019, pp. 691–706.
  * [74] K. Ganju, Q. Wang, W. Yang, C. A. Gunter, and N. Borisov, “Property inference attacks on fully connected neural networks using permutation invariant representations,” in _Proceedings of the 2018 ACM SIGSAC conference on computer and communications security_ , 2018, pp. 619–633.
  * [75] W. Zhang, S. Tople, and O. Ohrimenko, “Leakage of dataset properties in Multi-Party machine learning,” in _30th USENIX Security Symposium (USENIX Security 21)_. USENIX Association, Aug. 2021, pp. 2687–2704.
  * [76] B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under the gan: information leakage from collaborative deep learning,” in _Proceedings of the 2017 ACM SIGSAC conference on computer and communications security_ , 2017, pp. 603–618.
  * [77] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in _IEEE INFOCOM 2019-IEEE conference on computer communications_. IEEE, 2019, pp. 2512–2520.
  * [78] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , 2015, pp. 1310–1321.
  * [79] Y. Sun, Y. Yan, J. Cui, G. Xiong, and J. Liu, “Review of deep gradient inversion attacks and defenses in federated learning,” _Journal of Electronics & Information Technology_, vol. 46, no. 2, pp. 429–442, 2024.
  * [80] Y. Zou, X. Wang, and W. Shen, “Eavesdropping attack in collaborative wireless networks: Security protocols and intercept behavior,” in _Proceedings of the 2013 IEEE 17th international conference on computer supported cooperative work in design (CSCWD)_. IEEE, 2013, pp. 704–709.
  * [81] T. Wang, N. Huang, Y. Wu, and T. Q. Quek, “Energy-efficient wireless federated learning: A secrecy oriented design via sequential artificial jamming,” _IEEE Transactions on Vehicular Technology_ , vol. 72, no. 5, pp. 6412–6427, 2023.
  * [82] C. Ma, J. Li, M. Ding, H. H. Yang, F. Shu, T. Q. Quek, and H. V. Poor, “On safeguarding privacy and security in the framework of federated learning,” _IEEE network_ , vol. 34, no. 4, pp. 242–248, 2020.
  * [83] Y.-A. Xie, J. Kang, D. Niyato, N. T. T. Van, N. C. Luong, Z. Liu, and H. Yu, “Securing federated learning: A covert communication-based approach,” _IEEE Network_ , vol. 37, no. 1, pp. 118–124, 2023.
  * [84] C. Xu and G. Neglia, “What else is leaked when eavesdropping federated learning?” in _CCS workshop Privacy Preserving Machine Learning (PPML)_ , 2021.
  * [85] X. Hou, J. Wang, C. Jiang, X. Zhang, Y. Ren, and M. Debbah, “Uav-enabled covert federated learning,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 10, pp. 6793–6809, 2023.
  * [86] G. F. Cretu, A. Stavrou, M. E. Locasto, S. J. Stolfo, and A. D. Keromytis, “Casting out demons: Sanitizing training data for anomaly sensors,” in _IEEE Symposium on Security and Privacy (SP)_. IEEE, 2008, pp. 81–95.
  * [87] Q. Li, Z. Liu, Q. Li, and K. Xu, “martfl: Enabling utility-driven data marketplace with a robust and verifiable federated learning architecture,” in _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , 2023, pp. 1496–1510.
  * [88] D. Data, L. Song, and S. N. Diggavi, “Data encoding for byzantine-resilient distributed optimization,” _IEEE Transactions on Information Theory_ , vol. 67, no. 2, pp. 1117–1140, 2021.
  * [89] J. Li, G. Li, H. Cheng, Z. Liao, and Y. Yu, “Feddiv: Collaborative noise filtering for federated learning with noisy labels,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 4, 2024, pp. 3118–3126.
  * [90] P. W. Koh, J. Steinhardt, and P. Liang, “Stronger data poisoning attacks break data sanitization defenses,” _Machine Learning_ , vol. 111, no. 1, pp. 1–47, 2022.
  * [91] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, M. Hammoudeh, H. Karimipour, and G. Srivastava, “Block hunter: Federated learning for cyber threat hunting in blockchain-based iiot networks,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 11, pp. 8356–8366, 2022.
  * [92] L. Meng, Y. Wei, R. Pan, S. Zhou, J. Zhang, and W. Chen, “Vadaf: Visualization for abnormal client detection and analysis in federated learning,” _ACM Transactions on Interactive Intelligent Systems_ , vol. 11, no. 4, pp. 1–23, 2021.
  * [93] J. Hong, H. Wang, Z. Wang, and J. Zhou, “Federated robustness propagation: Sharing adversarial robustness in heterogeneous federated learning,” in _AAAI Conference on Artificial Intelligence (AAAI)_. AAAI, 2023, pp. 7893–7901.
  * [94] Y. Li, D. Yuan, A. S. Sani, and W. Bao, “Enhancing federated learning robustness in adversarial environment through clustering non-iid features,” _Computers & Security_, vol. 132, no. 1, pp. 1–13, 2023.
  * [95] R. Taheri, M. Shojafar, M. Alazab, and R. Tafazolli, “Fed-iiot: A robust federated malware detection architecture in industrial iot,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 12, pp. 8442–8452, 2020.
  * [96] S. Chen, M. Xue, L. Fan, S. Hao, L. Xu, H. Zhu, and B. Li, “Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach,” _Computers & Security_, vol. 73, pp. 326–344, 2018.
  * [97] T. Kieu, B. Yang, C. Guo, and C. S. Jensen, “Outlier detection for time series with recurrent autoencoder ensembles.” in _International Joint Conference on Artificial Intelligence (IJCAI)_. Morgan Kaufmann, 2019, pp. 2725–2732.
  * [98] T. Kieu, B. Yang, C. Guo, R.-G. Cirstea, Y. Zhao, Y. Song, and C. S. Jensen, “Anomaly detection in time series with robust variational quasi-recurrent autoencoders,” in _IEEE International Conference on Data Engineering (ICDE)_. IEEE, 2022, pp. 1342–1354.
  * [99] F. Tramèr, D. Boneh, A. Kurakin, I. Goodfellow, N. Papernot, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _International Conference on Learning Representations (ICLR)_. IMLS, 2018, pp. 1–20.
  * [100] J. Li, T. Zhu, W. Ren, and K.-K. Raymond, “Improve individual fairness in federated learning via adversarial training,” _Computers & Security_, pp. 1–10, 2023.
  * [101] C. Chen, B. Kailkhura, R. Goldhahn, and Y. Zhou, “Certifiably-robust federated adversarial learning via randomized smoothing,” in _IEEE International Conference on Mobile Ad Hoc and Smart Systems (MASS)_. IEEE, 2021, pp. 173–179.
  * [102] M. Phuong and C. Lampert, “Towards understanding knowledge distillation,” in _International Conference on Machine Learning (ICML)_. PMLR, 2019, pp. 5142–5151.
  * [103] E. Sannara, F. Portet, P. Lalanda, and V. German, “A federated learning aggregation algorithm for pervasive computing: Evaluation and comparison,” in _IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2021, pp. 1–10.
  * [104] T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2020, pp. 2351–2363.
  * [105] R. Guerraoui, S. Rouault _et al._ , “The hidden vulnerability of distributed learning in byzantium,” in _International Conference on Machine Learning (ICML)_. PMLR, 2018, pp. 3521–3530.
  * [106] K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 1142–1154, 2022.
  * [107] D. Cao, S. Chang, Z. Lin, G. Liu, and D. Sun, “Understanding distributed poisoning attack in federated learning,” in _IEEE International Conference on Parallel and Distributed Systems (ICPADS)_. IEEE, 2019, pp. 233–239.
  * [108] L. Muñoz-González, K. T. Co, and E. C. Lupu, “Byzantine-robust federated machine learning through adaptive model averaging,” _arXiv preprint arXiv:1909.05125_ , 2019.
  * [109] H. Yang, X. Zhang, M. Fang, and J. Liu, “Byzantine-resilient stochastic gradient descent for distributed learning: A lipschitz-inspired coordinate-wise median approach,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 5832–5837.
  * [110] C. Xie, O. Koyejo, and I. Gupta, “Generalized byzantine-tolerant sgd,” _arXiv preprint arXiv:1802.10116_ , 2018.
  * [111] Z. Wu, Q. Ling, T. Chen, and G. B. Giannakis, “Federated variance-reduced stochastic gradient descent with robustness to byzantine attacks,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 4583–4596, 2020.
  * [112] Y. Wang, T. Zhu, W. Chang, S. Shen, and W. Ren, “Model poisoning defense on federated learning: A validation based approach,” in _International Conference on Network and System Security_. Springer, 2020, pp. 207–223.
  * [113] J. Park, D.-J. Han, M. Choi, and J. Moon, “Sageflow: Robust federated learning against both stragglers and adversaries,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2021, pp. 840–851.
  * [114] C. Xie, S. Koyejo, and I. Gupta, “Zeno: Distributed stochastic gradient descent with suspicion-based fault-tolerance,” in _International Conference on Machine Learning (ICML)_. PMLR, 2019, pp. 6893–6901.
  * [115] X. Cao and L. Lai, “Distributed gradient descent algorithm robust to an arbitrary number of byzantine attackers,” _IEEE Transactions on Signal Processing_ , vol. 67, no. 22, pp. 5850–5864, 2019.
  * [116] L. Zhao, S. Hu, Q. Wang, J. Jiang, C. Shen, X. Luo, and P. Hu, “Shielding collaborative learning: Mitigating poisoning attacks through client-side detection,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 18, no. 5, pp. 2029–2041, 2020.
  * [117] Y. Zhao, J. Chen, J. Zhang, D. Wu, J. Teng, and S. Yu, “Pdgan: A novel poisoning defense method in federated learning using generative adversarial network,” in _International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP)_. Springer, 2020, pp. 595–609.
  * [118] Y. Li, H. Chen, W. Bao, Z. Xu, and D. Yuan, “Honest score client selection scheme: Preventing federated learning label flipping attacks in non-iid scenarios,” _arXiv preprint arXiv:2311.05826_ , 2023.
  * [119] F. Sattler, K.-R. Müller, T. Wiegand, and W. Samek, “On the byzantine robustness of clustered federated learning,” in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8861–8865.
  * [120] A. Ghosh, J. Hong, D. Yin, and K. Ramchandran, “Robust federated learning in a heterogeneous environment,” in _ICML (Workshop)_ , 2019.
  * [121] T. Qi, H. Wang, and Y. Huang, “Towards the robustness of differentially private federated learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 18, 2024, pp. 19 911–19 919.
  * [122] A. K. Singh, A. Blanco-Justicia, and J. Domingo-Ferrer, “Fair detection of poisoning attacks in federated learning on non-iid data,” _Data Mining and Knowledge Discovery_ , vol. 37, no. 5, pp. 1998–2023, 2023.
  * [123] Z. Chen, P. Tian, W. Liao, and W. Yu, “Zero knowledge clustering based adversarial mitigation in heterogeneous federated learning,” _IEEE Transactions on Network Science and Engineering_ , vol. 8, no. 2, pp. 1070–1083, 2020.
  * [124] L. Li, W. Xu, T. Chen, G. B. Giannakis, and Q. Ling, “Rsa: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets,” in _AAAI Conference on Artificial Intelligence (AAAI)_. AAAI, 2019, pp. 1544–1551.
  * [125] P. Zhao, F. Yu, and Z. Wan, “A huber loss minimization approach to byzantine robust federated learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 19, 2024, pp. 21 806–21 814.
  * [126] K. Liu, B. Dolan-Gavitt, and S. Garg, “Fine-pruning: Defending against backdooring attacks on deep neural networks,” in _International Symposium on Research in Attacks, Intrusions, and Defenses (RAID)_. Springer, 2018, pp. 273–294.
  * [127] Y. Jiang, S. Wang, V. Valls, B. J. Ko, W.-H. Lee, K. K. Leung, and L. Tassiulas, “Model pruning enables efficient federated learning on edge devices,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 1, no. 1, pp. 1–13, 2022.
  * [128] J. H. Van Lint, _Introduction to coding theory_. Springer Science & Business Media, 1998.
  * [129] L. Chen, H. Wang, Z. Charles, and D. Papailiopoulos, “Draco: Byzantine-resilient distributed training via redundant gradients,” in _International Conference on Machine Learning (ICML)_. PMLR, 2018, pp. 903–912.
  * [130] S. Rajput, H. Wang, Z. Charles, and D. Papailiopoulos, “Detox: A redundancy-based framework for faster and more robust gradient aggregation,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2019, pp. 1–11.
  * [131] P. Rieger, T. Krauß, M. Miettinen, A. Dmitrienko, and A.-R. Sadeghi, “Crowdguard: Federated backdoor detection in federated learning,” in _Network and Distributed System Security (NDSS) Symposium_ , 2024, pp. 1–18.
  * [132] J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” _Pattern Recognition_ , vol. 45, no. 1, pp. 521–530, 2012.
  * [133] C. Xie, S. Koyejo, and I. Gupta, “Zeno++: Robust fully asynchronous sgd,” in _International Conference on Machine Learning (ICML)_. PMLR, 2020, pp. 10 495–10 503.
  * [134] F. Sattler, K.-R. Müller, and W. Samek, “Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints,” _IEEE transactions on neural networks and learning systems_ , vol. 32, no. 8, pp. 3710–3722, 2020.
  * [135] G. Andrew, O. Thakkar, B. McMahan, and S. Ramaswamy, “Differentially private learning with adaptive clipping,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2021, pp. 17 455–17 466.
  * [136] J. Sun, A. Li, L. DiValentin, A. Hassanzadeh, Y. Chen, and H. Li, “Fl-wbc: Enhancing robustness against model poisoning attacks in federated learning from a client perspective,” in _Advances in Neural Information Processing Systems (NeurIPS)_. MIT Press, 2021, pp. 12 613–12 624.
  * [137] H. Gao, N. He, and T. Gao, “Sverifl: Successive verifiable federated learning with privacy-preserving,” _Information Sciences_ , vol. 622, pp. 98–114, 2023.
  * [138] D. C. Nguyen, M. Ding, Q.-V. Pham, P. N. Pathirana, L. B. Le, A. Seneviratne, J. Li, D. Niyato, and H. V. Poor, “Federated learning meets blockchain in edge computing: Opportunities and challenges,” _IEEE Internet of Things Journal_ , vol. 8, no. 16, pp. 12 806–12 825, 2021.
  * [139] Y. Qu, M. P. Uddin, C. Gan, Y. Xiang, L. Gao, and J. Yearwood, “Blockchain-enabled federated learning: A survey,” _ACM Computing Surveys_ , vol. 55, no. 4, pp. 1–35, 2022.
  * [140] H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Blockchained on-device federated learning,” _IEEE Communications Letters_ , vol. 24, no. 6, pp. 1279–1283, 2019.
  * [141] L. Feng, Y. Zhao, S. Guo, X. Qiu, W. Li, and P. Yu, “Bafl: A blockchain-based asynchronous federated learning framework,” _IEEE Transactions on Computers_ , vol. 71, no. 5, pp. 1092–1103, 2022.
  * [142] B. Liang, H. Li, M. Su, X. Li, W. Shi, and X. Wang, “Detecting adversarial image examples in deep neural networks with adaptive noise reduction,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 18, no. 1, pp. 72–85, 2018.
  * [143] A. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients,” in _AAAI Conference on Artificial Intelligence (AAAI)_. AAAI, 2018, pp. 1–10.
  * [144] C. Xie, Y. Long, P.-Y. Chen, Q. Li, S. Koyejo, and B. Li, “Unraveling the connections between privacy and certified robustness in federated learning against poisoning attacks,” in _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , 2023, pp. 1511–1525.
  * [145] R. Zhuang, S. A. DeLoach, and X. Ou, “Towards a theory of moving target defense,” in _ACM Workshop on Moving Target Defense (MTD)_. ACM, 2014, pp. 31–40.
  * [146] C. Dwork, A. Roth _et al._ , “The algorithmic foundations of differential privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3–4, pp. 211–407, 2014.
  * [147] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” _arXiv preprint arXiv:1712.07557_ , 2017.
  * [148] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in _International Conference on Learning Representations_ , 2018.
  * [149] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” _arXiv preprint arXiv:2010.01412_ , 2020.
  * [150] A. Cheng, P. Wang, X. S. Zhang, and J. Cheng, “Differentially private federated learning with local regularization and sparsification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 122–10 131.
  * [151] Z. Xu, M. Collins, Y. Wang, L. Panait, S. Oh, S. Augenstein, T. Liu, F. Schroff, and H. B. McMahan, “Learning to generate image embeddings with user-level differential privacy,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7969–7980.
  * [152] C. Wang, J. Liang, M. Huang, B. Bai, K. Bai, and H. Li, “Hybrid differentially private federated learning on vertically partitioned data,” _arXiv preprint arXiv:2009.02763_ , 2020.
  * [153] L. Lyu, H. Yu, X. Ma, C. Chen, L. Sun, J. Zhao, Q. Yang, and P. S. Yu, “Privacy and robustness in federated learning: Attacks and defenses,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–21, 2022.
  * [154] J. C. Duchi, M. I. Jordan, and M. J. Wainwright, “Local privacy and statistical minimax rates,” in _2013 IEEE 54th annual symposium on foundations of computer science_. IEEE, 2013, pp. 429–438.
  * [155] S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “Ldp-fed: Federated learning with local differential privacy,” in _Proceedings of the third ACM international workshop on edge systems, analytics and networking_ , 2020, pp. 61–66.
  * [156] Y. Zhao, J. Zhao, M. Yang, T. Wang, N. Wang, L. Lyu, D. Niyato, and K.-Y. Lam, “Local differential privacy-based federated learning for internet of things,” _IEEE Internet of Things Journal_ , vol. 8, no. 11, pp. 8836–8853, 2020.