# Privacy and Fairness in Federated Learning: on the Perspective of Trade-off

Huiqiang Chen, Tianqing Zhu*, Tao Zhang  huiqiang.chen,
<EMAIL_ADDRESS><EMAIL_ADDRESS>University of
Technology SydneyPO Box 123 BroadwaySydneyNSWAustralia2007 ,  Wanlei Zhou
<EMAIL_ADDRESS>City University of MacauAvenida Padre Tomás Pereira
TaipaMacauChina and  Philip S. Yu<EMAIL_ADDRESS>University of Illinois at
Chicago851 S. Morgan St., Rm 1138 SEO, Chicago, IL 60607ChicagoUS

###### Abstract.

Federated learning (FL) has been a hot topic in recent years. Ever since it
was introduced, researchers have endeavored to devise FL systems that protect
privacy or ensure fair results, with most research focusing on one or the
other. As two crucial ethical notions, the interactions between privacy and
fairness are comparatively less studied. However, since privacy and fairness
compete, considering each in isolation will inevitably come at the cost of the
other. To provide a broad view of these two critical topics, we presented a
detailed literature review of privacy and fairness issues, highlighting unique
challenges posed by FL and solutions in federated settings. We further
systematically surveyed different interactions between privacy and fairness,
trying to reveal how privacy and fairness could affect each other and point
out new research directions in fair and private FL.

Federated learning, data privacy, model fairness

††ccs: General and reference Surveys and overviews††ccs: Computing
methodologies Machine learning

##  1\. Introduction

Machine learning has changed our lives and will undoubtedly bring us more
excitement. However, its success is closely tied to the availability of large-
scale training data, and as new learning models keep emerging, the demand for
more data persists relentlessly. One worrisome issue with collecting massive
amounts of data is the risks that present to privacy. FL (McMahan et al.,
2017) has emerged as an attractive learning paradigm to meet privacy
requirements.

Unlike traditional centralized machine learning, FL trains models in a
distributed and parallel way such that different clients collectively train a
model with their training data. This technique offers two enormous benefits.
First, it saves companies the costly process of collecting large-scale data
because the clients provide their local data. Second, it preserves the
client’s privacy by keeping data locally. With such benefits, it is no
surprise that the industry has already leaped to put FL into practice, such as
Gboard (Hard et al., 2018).

###  1.1. Privacy and Fairness in FL

Great achievements have been made with FL. However, the paradigm of FL still
suffers from ethical issues surrounding data use. One of those ethical issues
is privacy. Although raw data never leave the device, the uploaded
gradients/parameters still carry local information. Therefore, a trained model
could hold the client’s data distribution. Consequently, an adversary can
infer information about what data are included in the training set (Hitaj et
al., 2017; Zhang et al., 2020e) or, even worse, reconstruct the training data
(Geiping et al., 2020). As such, the research community is endeavoring to
identify all potential privacy risks by launching different privacy attacks
(Nasr et al., 2019). Accordingly, defenses for all these attacks are also
being proposed to secure private data (Wei et al., 2020a). This wargaming
between attack and defense is leading us to more private FL environments.

Another ethical issue in FL is fairness, which refers to reducing the model’s
bias towards disadvantaged groups, such as ethnic minorities, women, or the
aged. Fairness in FL is defined at two different levels. The first pertains to
algorithmic fairness (Mehrabi et al., 2021; Chouldechova and Roth, 2018),
where model output should not skew towards disadvantaged groups defined by
some sensitive attributes. The second pertains to client fairness (Li et al.,
2019b; Mohri et al., 2019). In vanilla FL (McMahan et al., 2017), models
trained on the larger dataset are given higher importance during aggregation.
Hence, the global model will be optimized to capture the data distributions of
clients with a larger dataset. Therefore the model performance will vary
significantly among clients, which imposes unfairness at the client level.

Privacy and fairness are two crucial ethical notions, and violating either of
them is unacceptable. However, to date, the research community has primarily
considered these two issues separately, yet they are inextricably entwined.
For example, it is well known that privacy comes at the cost of accuracy. What
is surprising is that the cost is not consistent across all groups as
expected, where disadvantaged groups often suffer more of an accuracy decrease
than the other groups due to data scarcity (Bagdasaryan et al., 2019; Kuppam
et al., 2020). In other words, ensuring privacy can exacerbate the inequities
between groups. Fairness, in turn, may negatively affect privacy. For example,
to ensure a classification model is fair, the server usually needs to know the
underlying distribution of the training dataset to eliminate bias existing in
either the training data (Feldman et al., 2015; Kamiran and Calders, 2012,
2010) or the model (Zafar et al., 2017; Goh et al., 2016; Zemel et al., 2013).
This means the client will share more data with the server, increasing the
privacy risk.

Therefore, in addition to reviewing privacy and fairness issues in FL, another
motivation of this survey is to explore the possible interactions between
privacy and fairness and to discuss the relationships between fairness and
privacy. It is worth noting that the issue of client fairness adds an extra
layer of complexity to the federated setting. To the best of our knowledge,
this survey is the first attempt to examine the relationships between privacy
and fairness in the federated setting.

Table 1. Comparison to Related Surveys on Privacy or Fairness in FL Reference | Privacy-preserving | Fairness-aware |  Interactions between privacy and fairness
---|---|---|---
| Privacy
---
attack
Defense |  | Algorithmic
---
fairness
| Client
---
fairness
(Yang et al., 2019b) | ✓ | ✓ |  |  |
(Yin et al., 2021b) | ✓ | ✓ |  |  |
(Lyu et al., 2020) | ✓ | ✓ |  |  |
(Kairouz et al., 2019) | ✓ | ✓ | ✓ | ✓ |
Our work | ✓ | ✓ | ✓ | ✓ | ✓

###  1.2. Main Contribution

This survey provides a broad view of privacy and fairness issues in FL. We
first illustrated that FL is not as private as it claimed to be. Adversarial
clients and server have several new attack vectors at their disposal, and we
outlined several techniques for preserving privacy against these attacks.
Turning to fairness, we explained the two lines of fairness notions adopted in
FL and the corresponding debiasing strategies. Lastly, we discussed
interactions between privacy and fairness. Our contributions are as follows:

  * •

This is the first survey that provides a comprehensive overview of privacy,
fairness, and the interactions between the two.

  * •

We present a detailed survey of privacy attacks and defenses in FL, discuss
how these privacy attacks could damage privacy in FL and highlight the
assumptions and principle methods of these attack strategies.

  * •

Following a rigorous enumeration of the sources of bias in the FL pipeline, we
discuss the fairness notions adopted in FL and summarize fairness-aware FL
approaches.

  * •

We point out several future research directions toward training private and
fair FL models.

##  2\. Background Knowledge

###  2.1. Definition of FL

The goal of FL is to train a global model in a distributed way. The objective
function is formulated as:

(1)
$\mathop{\min}\limits_{w}f\left(w\right)=\sum\limits_{k=1}^{m}{{p_{k}}{F_{k}}\left(w\right)}$

where $m$ is the number of clients, $p_{k}>0$ is the aggregating weight of
client $k$, satisfying $\sum_{k}p_{k}=1$. $F_{k}(w)$ is the empirical risk on
client $k$’s data. In a trivial setting, $p_{k}$ is the ratio of local samples
to total samples of all clients. The process of FL consists of two stages: the
training and inference stages. Three actors are involved: 1) clients, each of
which has a local dataset and will use it to contribute to the global model’s
training by uploading local gradients/parameters, noting that each client’s
dataset may vary from the others; 2) a server that coordinates the learning
process; and 3) users, who will use the final well-trained model.

In each iteration of FL, selected clients download the global model and
perform learning algorithms locally. They communicate their updates to the
server for aggregation and model updating. This interaction between clients
and the server repeats until the model converges. At the inference stage, a
well-trained model is deployed to users, where users can infer the model via
black-box access. This step is no different from a traditional data center
approach.

###  2.2. Privacy Disclosure in FL

Recent research verified the privacy risk of FL. Adversaries can glean the
participants’ training data. For example, Zhu et al. (Zhu et al., 2019) fully
reconstructed training data from the victim client through the uploaded
gradients as shown in Fig.1(a). In the course of the FL pipeline, several
attack interfaces exist for an adversary, who could be the server or a client
in FL. The attack could occur during the training or inference stage, within
or outside the FL. The attack targets include membership, property, class
representative, and the raw data (Yin et al., 2021b).

(a) Overview of reconstruction attack in FL. An adversary iterative optimizes
generated sample by matching gradients $\nabla W$.

(b) Attack results on MNIST, CIFAR-100, SVHN and LFW. The adversary is able to
reconstruct the private images of other clients truthfully (reproduced from
(Zhu et al., 2019)).

Figure 1. Reconstruction attack in FL.

####  2.2.1. Membership Inference Attacks

Membership Inference Attacks (MIA) aim to identify whether a given sample was
used to train the target model. These types of attacks can pose privacy risks
to individuals. For example, confirming a patient’s clinical record was used
to train a model associated with a particular disease would reveal that
patient’s health condition. MIA was initially investigated by Shokri et al.
(Shokri et al., 2017). The attack models are essentially binary classifiers.
Given an instance $\boldsymbol{X}$ and a target model $F_{t}$, the goal of the
MIA model is to identify whether or not $\boldsymbol{X}$ is contained within
the training dataset $D$ of the target model $F_{t}$.

####  2.2.2. Property Inference Attacks

Property inference attacks aim to recover some property of the training set,
which may be irrelevant to the main tasks. Such as the property of ”wearing
glasses” against a gender classifier or the composition of the training
dataset. This kind of attack also leads to privacy issues. With proper prior
knowledge, the adversary can infer the presence of a specific sample in the
training set.

####  2.2.3. Model Inversion Attacks

Model inversion attacks aim to recover class-specific features or construct
class representatives by accessing the target model and other possible
auxiliary information. The recovered data is a representing sample (usually a
synthetic sample) that only reflects some aspects of the training data and is
not a member of the training set.

####  2.2.4. Reconstruction Attacks

Reconstruction attacks (Dinur and Nissim, 2003) aim to reconstruct a
probabilistic version of samples in the training set. Success in a
reconstruction attack is measured by comparing the reconstruction with the
original data. If the two are similar, the attack has been successful.
Fig.1(b) (Zhu et al., 2019) shows an example. Unlike the model inversion
attacks, the recovered data here is almost the same as the training dataset at
the pixel level and belongs to the training dataset.

###  2.3. Privacy-preserving Techniques

In the realm of FL, plenty of studies have shown how to break the basic
privacy assurances, such as determining a client’s membership in the training
set (Nasr et al., 2019), ascertaining the class representations of the
client’s training data (Hitaj et al., 2017; Wang et al., 2019a; Song et al.,
2020), and, the worst case of all, procuring the raw training data (Geiping et
al., 2020). Several privacy-preserving techniques can help stop these privacy
leakages, including cryptographic techniques and the perturbation approach.

####  2.3.1. Cryptographic Approach

Secure computation is a cryptographic technique in which functions are
evaluated based on a set of distributed inputs without revealing additional
information, e.g., the parties’ inputs or intermediate results. Secure multi-
party computation, homomorphic encryption, and secret sharing are the most
common choices for a secure computing platform.

Multi-party computation (Yao, 1982) was first introduced to secure the private
inputs of multiple participants while they jointly compute an agreed-upon
model or function. Formally, $n$ participants $p_{1},p_{2},…$, and $p_{n}$ can
collaboratively compute $y=f\left(x_{1},…,x_{n}\right)$, where $x_{i}$ is a
secret input that belongs to participants $p_{i}$, This form of secure
computing offers both correctness and privacy. After all, no participant
learns anything about the others’ data other than the final result.

Homomorphic encryption (Yi et al., 2014) allows certain mathematical
operations, such as addition and multiplication, to be performed directly on
ciphertexts. These can then be used as the basis for more complex arbitrary
functions.

####  2.3.2. Perturbation Approach

With privacy concerns in mind, one needs to be cautious of how much
information about a participating client is revealed during training. The
perturbation approach arises as a natural way of preventing information leaks.
By injecting the proper amount of artificial noise into the original data, the
statistical information calculated from the perturbed data will be
statistically indistinguishable from the original data.

There are three types of widely used perturbation techniques: differential
privacy (DP), additive perturbation, and multiplicative perturbation. DP
proposed by Dwork (Dwork et al., 2006), is the gold standard. The intuition
behind DP is to mask the contribution of any individual user by a sufficient
level of uncertainty. A randomized mechanism $\mathcal{M}$ is said to be
$\left(\epsilon,\delta\right)$-differentially private if, for any pair of
neighboring datasets $\mathcal{D}$ and $\mathcal{D^{\prime}}$, and for every
set of $S\subseteq Range\left(\mathcal{M}\right)$, if $\mathcal{M}$ satisfies:

(2) ${\Pr\left[{\mathcal{M}\left(\mathcal{D}\right)\in
S}\right]\leq\exp\left(\varepsilon\right)\cdot\Pr\left[{\mathcal{M}\left(\mathcal{D^{\prime}}\right)\in
S}\right]+\delta}$

The parameter $\epsilon$ is defined as privacy budget, which measures how
alike the two adjacent datasets $\mathcal{D}$ and $\mathcal{D^{\prime}}$ are
to each other. A smaller $\epsilon$ indicates a stronger privacy guarantee. If
$\delta=0$, then the randomized mechanism $\mathcal{M}$ is degraded into
$\epsilon$\- DP.

###  2.4. Fairness in FL

Algorithms are widely used to assist in making recommendations, assessing loan
applications, etc. Several studies have identified the unfairness in different
algorithmic scenarios (Mehrabi et al., 2021). One example is the hate speech
detector designed to rate the toxicity score of the given phrases to help
companies like Twitter recognize harmful speech. The detector relies on a tool
called Perspective, which is trained on labeled data. The detector behaves
differently towards phrases written in African American English in a racially
biased way, as Fig. 2 shows (Sap et al., 2019).

Figure 2. Racial disparities in classifier predictions on tweets written in
African-American English and in Standard American English (reproduced from
(Sap et al., 2019))

####  2.4.1. Bias in FL

Fairness can be eroded by bias. According to Olteanu et al. (Olteanu et al.,
2019), bias can slip into data flows from generation to collection to
processing (Olteanu et al., 2019). The distributed learning paradigm of FL
brings new and unique challenges to our efforts to build fair models. One
challenge is that the independent and identical distribution (i.i.d.)
assumption no longer holds (Kairouz et al., 2019). In FL, bias can also be
introduced by either the client or the server.

  * •

Client-introduced bias. Clients can introduce bias in several ways. First,
bias exists in a client’s local data – prejudice, underestimation, negative
legacies, etc. (Kamishima et al., 2012). This bias is then integrated into the
global model through client and server interactions. Second, bias can strike
when clients are dropped out of the federated setting due to device shutdowns
or communication limitations. In these cases, the global model will find it
hard to fit the clients’ data properly. The massively distributed data also
incurs bias. In this case, the client does not have enough data to capture an
underlying distribution, and moreover, the underlying distributions of
different clients are probably not the same. (Zhao et al., 2018a; Briggs et
al., 2020; Li et al., 2021)

  * •

Server-introduced bias. The server can also add bias. As the coordinator of
the learning scheme, the server will sample clients in each round to train a
global model with their local data (Nagalapatti and Narayanam, 2021). However,
the sampling process is prone to producing bias if it is not done with careful
consideration. First, in terms of efficiency, only a fraction of clients are
selected in each round (McMahan et al., 2017). Yet the data distribution of
only a few selected clients forms an inadequate representation of the actual
population distribution. Second, the sampling may be skewed toward certain
clients. For instance, to speed up convergence, the server prefers clients
that meet specific criteria (Nishio and Yonetani, 2019; Cho et al., 2020).

####  2.4.2. Fairness Notions in FL

To date, researchers have proposed several definitions of fairness, see, e.g.,
(Verma and Rubin, 2018; Mehrabi et al., 2021). These definitions vary from
scenario to scenario, and it is unlikely that there will ever be one
particular definition of fairness that fits all circumstances. Table 2 lists
common algorithmic fairness notions adopted in centralized machine learning.
At a high level, two families of definitions exist the individual notion and
statistical notion (Chouldechova and Roth, 2018).

  * •

Individual notion. The individual notions ensure fairness between specific
pairs of individuals: ”Give similar predictions to similar individuals.”
(Dwork et al., 2012). Formally, for a set of samples $V$, a distance metric is
defined as $d:V\times V\to R$ to measure the similarity. A function
$\mathcal{M}:V\to\Delta A$ maps the samples $V$ to the probability
distributions over outcomes, and another distance $D$ metric measures the
distance between the distributions of outputs. Fairness is achieved if and
only if $D\left(\mathcal{M}\left(x\right),\mathcal{M}\left(y\right)\right)\leq
d\left(x,y\right)$. This family of definitions provides a meaningful
guarantee. However, they are at the cost of making significant assumptions,
some of which are non-trivial problems in fairness.

  * •

Statistical notion. The statistical notions provide fairness assurance at a
statistical level. For the protected demographic groups $G$ (such as racial
minorities), some statistical measures are required to be equal across all of
these groups. These statistical measures include positive classification rates
(Kamishima et al., 2011; Calders and Verwer, 2010; Dwork et al., 2012), false
positive and false negative rates (Hardt et al., 2016; Kleinberg et al.,
2017), and positive predictive value (Zafar et al., 2017; Chouldechova, 2017).
Detailed enumeration can be found in (Berk et al., 2021a; Verma and Rubin,
2018). This family of definitions requires no assumption over data and can be
easily verified. However, statistical notions are insufficient as a fairness
constraint, which does not give meaningful guarantees to individuals or
structured subgroups of the protected demographic groups. Jiang et al. (Jiang
et al., 2022) generalized the demographic parity (Kusner et al., 2018) to
continuous sensitive attribute.

Table 2. Definitions of Algorithmic Fairness Notions Fairness notion | Definition | Explanation
---|---|---
| Individual Fairness
---
(Dwork et al., 2012)
$D(M(x),M(y))\leq d(x,y)$ |  | Similar samples receive
---
similar treatment
| Eqal Opportunity
---
(Hardt et al., 2016)
$\Pr[\hat{Y}=1|A=0,Y=1]=\Pr[\hat{Y}=1|A=1,Y=1]$ |  | Equal true positive rates
---
for protected/unprotected
groups
| Equal Accuracy
---
(Berk et al., 2021b)
$\Pr[\hat{Y}=Y|A=0]=\Pr[\hat{Y}=Y|A=1]$ |  | Equal prediction accuracy
---
for protected/unprotected
groups
| Equalized Odds
---
(Hardt et al., 2016)
| $\Pr[\hat{Y}=1|A=1,Y=y]=\Pr[\hat{Y}=1|A=0,Y=y],$
---
$y\in\\{0,1\\}$
| Equal positive rates for
---
protected/unprotected
groups
| Treatment Equality
---
(Berk et al., 2021b)
| Equal false negatives and fase positives for protected/unprotected groups
---
| Demographic Parity
---
(Kusner et al., 2018)
$\Pr[\hat{Y}|A=0]=\Pr[\hat{Y}|A=1]$ |  | Outcome is independent
---
of the protected attribute

Apart from algorithmic fairness, which is measured on sensitive attributes,
fairness in FL can also be made from a client’s view since clients are
naturally grouped by attributes like geographic location, gender and income
(Du et al., 2021). At a client level, fairness can be evaluated by different
metrics.

###### Definition 0 (Good-intent fairness (Mohri et al., 2019)).

The training procedure does not overfit a model to any device at the expense
of other clients in FL.

This metric improves the worst-case performance. Li et al. (Li et al., 2019b)
took a further step. They tried to ensure a fair FL model for all clients by
producing a more uniform model performance across all clients. Fairness is
defined as the uniformity of the accuracy distribution across clients in FL.

###### Definition 0 (Accuracy parity (Li et al., 2019b)).

Consider two trained models, $f(w)$ and $f(\tilde{w})$. The model that
provides the most uniform performance across all clients will also provide the
fairest solution to the FL objective in Eq. (1).

###  2.5. Interactions between Privacy and Fairness

Both fairness and privacy are important ethical notions in machine learning
and have been extensively studied. However, the majority of current studies in
the research community consider fairness and privacy separately. However, the
interactions between privacy and fairness are bilateral.

  * •

Privacy degrades fairness. Several works have observed inconsistent reductions
in accuracy caused by private mechanisms on classification (Farrand et al.,
2020) and generative tasks (Ganev et al., 2022). It turns out that privacy
mechanisms affect the underrepresented group more than other groups.

  * •

Fairness increases privacy risk. Fairness comes at the cost of privacy. To
ensure fairness, a model is trained to perform equally on data from different
groups, even though the underrepresented group didn’t have enough data in the
training set, which incurs overfit and increases the privacy risk (Chang and
Shokri, 2021).

##  3\. Privacy in FL

With the advent of FL, many claims that user data are now secure. However,
even sharing a small fraction of gradients (Shokri and Shmatikov, 2015; Aono
et al., 2017) with a server would raise privacy concerns. In FL, there are
several unexplored types of privacy attacks: membership inference attacks,
property inference attacks, model inversion attacks, and reconstruction
attacks. This section will outline these attacks before moving on to
mitigation techniques and discussions.

###  3.1. Membership Inference Attacks

The adversary’s goal in FL is to determine whether a given sample belongs to a
single client’s private training data or of any participants (Yang et al.,
2020b). MIAs take occur in different ways in FL.

####  3.1.1. White-box and Black-box MIAs

Based on the access granted to the adversary, MIAs can be divided into black-
box and white-box (Nasr et al., 2019). In the black-box setting, the adversary
can only obtain a prediction vector computed by the target model while the
internal parameters remain secret. MIAs in this setting exploit the
statistical differences between a model’s predictions on its training set
versus unseen data (Shokri et al., 2017). Truex et al. (Truex et al., 2019b)
described a systematic approach to constructing a black-box MIA model and the
general formulation of each component in the attack model.

However, since the global model is shared with all participants for local
training, it is often assumed that an adversary has white-box access in FL.
The white-box access renders much more information to the adversary, such as
the internal parameters of each layer. This enables the adversary to calculate
the outputs of each layer. Nasr et al. (Nasr et al., 2019) designed a deep
learning attack model that separately processes the gradients extracted from
different layers of the target model and combines this information to compute
the membership probability of a target data point.

####  3.1.2. Training and Inference MIAs

MIAs can be launched during the training stage or once the model is complete
in the inference stage in FL.

In the training stage, the adversary could be the server or any client
participating in the training. Both characters have white-box access to the
global model and can easily save the snapshots of the global model at each
iteration during training. In this way, the adversary obtains multiple
versions of the target model over time and acquires the updated information to
infer private data (Phuong et al., 2019). In addition to passively collecting
the updated information, the adversary may further modify the information to
allure the victim clients to reveal more information.

In the inference phase, the FL model is well-trained and fixed. The adversary
can only perform an inference attack passively. In this case, MIA in FL
resembles that in a centralized setting. The attack’s success largely depends
on the information that is revealed to the adversary. Melis et al. (Melis et
al., 2019) investigated privacy leaks concerning membership during the
inference phase and showed that positions of words in a batch could be
revealed from a deep learning model.

####  3.1.3. Active and Passive MIAs

The adversary can conduct MIAs against the FL model actively or passively. For
instance, the server can either adaptively modify the aggregate parameters or
honestly calculate the global model and passively conduct MIAs (Zhang et al.,
2020f).

Melis et al. (Melis et al., 2019) designed MIAs against models operating on
non-numerical data (e.g., natural-language text). An embedding layer is
equipped for the target model, transforming the inputs into a lower-
dimensional vector representation. The adversary passively saves a snapshot of
the joint model parameters ${w_{t}}$. The difference between the consecutive
snapshots $\Delta{w_{t}}={w_{t}}-{w_{t-1}}={\Sigma_{k}}\Delta w_{t}^{k}$
reveals the aggregated updates from all participants and hence reveals the
membership.

Nasr et al. (Nasr et al., 2019) performed active MIAs on FL models by
reversing the stochastic gradient descent algorithm and extracting membership
information. If the target data point belongs to the training dataset, the
attacker’s modifications will be nullified since the target model will descend
the model’s gradient for training samples. However, if the target data sample
is not used during the training, the target model will not respond to the
attacker’s modification. Thus, membership can be deduced. In (Hitaj et al.,
2017), a malicious client actively mislabels the training sample to fool the
victim into releasing private information.

####  3.1.4. Insider and Outsider MIAs

FL involves two types of actors who can access model information: internal
actors (participating clients and the server) and external actors (model
consumers and eavesdroppers). Therefore, FL systems must withstand potential
adversaries within and outside the protocol.

The inner adversary could be a client or a server. Clients are picked at
random to participate in a training round. When training with hundreds or
millions of clients, malicious clients are highly likely involved, who will
attempt to deduce the sensitive information of others (Hitaj et al., 2017).
The real-time nature of FL added to the inner attacker’s strength. For
example, Zhang et al. (Zhang et al., 2020f) trained a GAN as a malicious
client during training to infer the data of other clients in FL. A malicious
central server poses a greater threat than a malicious client. Because it can
manipulate the global model supplied to victims and obtain more information.
Nasr et al. (Nasr et al., 2019) launched MIAs from both the client and server
sides and witnessed a higher inference accuracy as a curious central server
than as a malicious client.

In addition to internal threats, FL also faces potential attacks from
adversaries outside the system. Once the FL training is finished and the model
is deployed to users, these users may conduct both black- and white-box
attacks depending on their access.

####  3.1.5. Discussion

The attacks mentioned above demonstrate the vulnerability of FL to privacy
attacks, and these privacy risks stem from two assumptions made within the FL
protocols: 1) The server is trustworthy. FL gives the server access to each
participant’s updates in the form of gradients or model parameters containing
clients’ private information. The server can even purposefully send a modified
model to steal information. 2) Clients are honest. A malicious client can
collect several copies of the global model from the rounds it participates in.
In this way, inference phase attacks on data privacy are also plausible during
the learning phase. Additionally, adversarial clients may influence and shift
the bounds of the model during development rather than just abusing the
boundaries of a model’s service while it is in production.

###  3.2. Property Inference Attacks

The adversary in property inference attacks attempts to infer the specific
property of the subset of the training dataset. The target property may be
irrelevant to the classification task (e.g., ”wearing glasses” in a gender
classification task) and do not characterize the whole class. The attack is
made at the population level as opposed to a single sample in MIA. In terms of
when the attack is launched, property inference attacks can be classified as
static or dynamic attacks. The static attack is applied after the training
phase has concluded and the target training set is fixed. The dynamic attack
typically occurs during the training phase in FL. In this instance, the
training set is changing dynamically.

####  3.2.1. Static Attacks

The research of property inference attacks dates back to (Ateniese et al.,
2013). Ateniese et al. performed a property inference attack against Hidden
Markov Models and Support Vector Machine based on a meta-classifier. A set of
shadow classifiers were trained on a dataset similar to the target model
except for the target property. The meta-classifier is trained with shadow
classifiers as the input to find the classifiers trained on the dataset with
the target property. Ganju et al. (Ganju et al., 2018) extended this attack to
a fully connected neural network case. They shared a similar idea, using the
gradient of shadow classifiers to train a meta-classifier. Different from
(Ateniese et al., 2013), their research focuses on improving the attack
efficiency by taking permutation invariance into account.

####  3.2.2. Dynamic Attacks

In every communication round of FL, clients are selected at random. This means
the training data is dynamically changing, which weakens the property
inference attack because the target property appears unpredictable, thereby
diminishing the distinguishability of model updates (Wang et al., 2022). Wang
et al. (Wang et al., 2019b) explored property inference attacks within the FL
framework. Inspired by the relationship between the changing of neuron weights
in the output layer and the sample label, the authors proposed three attacks
as an eavesdropper to infer the labels’ quantity composition proportion.
Recently, Wang et al. (Wang et al., 2022) presented a poisoning-assisted
property inference attack in FL from the client’s viewpoint, aiming at
inferring if and when a sensitive property emerges. The authors built their
attacks around the realization that regular model updates reflect the shift in
data distribution and, in particular. A binary classifier is trained to make
predictions based on these periodic model updates. A property-specific
poisoning attack is proposed to distort the decision boundary of the shared
model on target attribute data. Thus, model updates have a better discerning
ability to infer target property.

####  3.2.3. Discussion

The MIAs and reconstruction attacks represent two ends of a spectrum of
privacy invasion. Property inference attacks locate in the middle and seek to
determine if the attackers’ target property is present in the training
samples. This type of attack is more complex than MIA since the target
property doesn’t always match the attributes that characterize the classes of
the FL model. Nonetheless, a property inference attack poses a greater threat
than MIA. Using the real-time nature of FL, the adversary can even infer when
the target property appears.

###  3.3. Model Inversion Attacks

Fredrikson et al. (Fredrikson et al., 2014) initiated model inversion attacks
on tabular data. A subsequent work (Fredrikson et al., 2015) extended it to
the image data. The attack is formulated as an optimization problem to
synthesize the input for a given label:
$y:{\max_{x}}\log{T_{y}}\left(x\right)$, where ${T_{y}}\left(x\right)$ is the
probability of the model $T$ outputs label $y$ for input $x$. The access could
be black-box (Fredrikson et al., 2015) or white-box (Zhang et al., 2020a; Chen
et al., 2020a).

####  3.3.1. Black-box Attacks

In the black-box setting, the attacker can only make prediction queries to the
model. Fredrikson et al. (Fredrikson et al., 2015) built attack algorithms
following the maximum a posterior principle. Their attack recovered a
recognizable image of a person given only API access to a facial recognition
system and a specific name of a target person. Yang et al. (Yang et al.,
2019a) engineered an inversion model a perform the inversion attacks. The
adversary composed an auxiliary set assumed generic enough to retain
meaningful information to regularize the ill-posed inversion problem (Salem et
al., 2020). However, the target model is usually assumed to be simple
networks, and the generalization to complex models is not trivial. The
inversion problem of a neural network is non-convex, and the optimization
suffers minimal local problems, which leads to poor attack performance.

####  3.3.2. White-box Attacks

n the white-box setting, the attacker has complete knowledge of the model.
Zhang et al. (Zhang et al., 2020a) sketched a generative model to learn an
informative prior from the public dataset. This prior is then used to regulate
the inversion problem. Benefiting from this, the authors revealed private
training data of DNNs with high fidelity. Chen et al. (Chen et al., 2020a)
boosted (Zhang et al., 2020a)’s methods. They leveraged the target model to
label a public dataset, and a GAN model was trained to distinguish not only
real and synthesized samples but also labels. They also modeled the private
data distribution to reconstruct representative data points better. The
success of model inversion attack benefits from an informative prior.

####  3.3.3. Discussion

MIAs can be performed with either black-box access or white-box access. When
given black-box access, the attack’s success heavily relies on the auxiliary
dataset, which is assumed to share the same generic features as the private
target dataset. Furthermore, the target models in this category are usually
simple due to limited access. In the white-box case, the target models extend
to DNNs. Most attacks implement GAN to synthesize samples to mimic the private
samples regarding the soft labels. This kind of attack is less common compared
with reconstruction attacks in FL.

###  3.4. Reconstruction Attacks

Unlike MIAs, reconstruction attacks attempt to retrieve training data and pose
a much more severe threat to privacy. As demonstrated by Aono et al. (Aono et
al., 2017), the gradient of the weights is proportional to that of the bias in
the first layer of the model, and their ratio approximates the training input.
Geiping et al. (Geiping et al., 2020) demonstrated that it is possible to
faithfully reconstruct images at high resolution given knowledge of the
parameter gradients. Such a privacy break is possible even for deep neural
networks. Huang et al. (Huang et al., 2021) evaluated existing reconstruction
attacks and defenses. Gupta et al. (Gupta et al., 2022) extended this attack
to text data and successfully reconstructed single sentences with high
fidelity for large batch sizes. To date, we know of two kinds of
reconstruction attacks, namely optimization-based attacks (Opt-based) and
closed-form attacks.

Table 3. Comparison between two reconstruction attack categories |  | Theoretical
---
guarantee
Convergence |  | Running
---
time
| Recovered
---
image
Applicability | Insight
Opt-based | No | Local optimal | Slow | With artifacts | No limitation | No
Closed-form | Yes | / | Fast | Original | Limited | Yes

####  3.4.1. Optimization-based Attack

Raw data can be reconstructed from gradients by solving an optimization
problem. Given a machine learning model $f(w)$ and the gradient
$g=\frac{1}{b}\sum\nolimits_{j=1}^{b}{{\nabla_{w}}{L_{w}}\left({x_{j}^{*},y_{j}^{*}}\right)}$
computed on a private batch $\left(x^{*},y^{*}\right)\in\mathbb{R}^{b\times
d}\times\mathbb{R}^{b}$ with bath size $b$. The adversary tries to reconstruct
$x\in\mathbb{R}^{b\times d}$ as an approximation of the true data ${x^{*}}$ by
solving the following optimization problem:

(3)
$\arg\mathop{\min}\limits_{x}{\mathcal{L}_{grad}}\left(x;w,g\right)+\alpha{\mathcal{R}_{aux}}\left(x\right)$

The first part of Eq. 3 pushes the recovered gradients towards the true
gradients $g$, hence deducing a better approximation. The regularization term
${\mathcal{R}_{aux}}\left(x\right)$ is used to incorporate the prior knowledge
to further improve reconstruction.

Zhu and Han (Zhu et al., 2019) proposed DLG attack with $l_{2}$ distance as
the reconstruction loss $\mathcal{L}_{grad}$. DLG starts with randomly
generated dummy samples and then iteratively optimizes the dummy samples and
labels until they converge. Finally, DLG achieves pixel-wise recovery accuracy
for image classification and token-wise recovery accuracy for a masked
language model. Zhao et al. (Zhao et al., 2020) improved Zhu and Han’s work
(Zhu et al., 2019) on convergence speed and reconstruction fidelity by
leveraging the relationship between the ground-truth labels and the signs of
the gradients.

The optimization problem Eq. 3 is often under-determined (Jeon et al., 2021),
as the information in the gradients $g$ is usually insufficient to recover the
training data ${x^{*}}$. Even when the gradient size is substantially bigger
than the input data dimension. As demonstrated by Zhu and Blaschko (Zhu and
Blaschko, 2020), when the learning model is huge, there may be a pair of
separate data sharing the same gradient.

In response to this, one may introduce prior knowledge as a regularization
term to narrow the search space, making it more consistent with the underlying
distribution of training data. Yin et al. (Yin et al., 2021a) utilized the
local batch norm as the regularization term since adjacent pixels in natural
photographs are likely to have comparable values. They achieved precise
recovery of the high-resolution images on complex datasets, deep networks, and
large batch sizes. Hatamizadeh et al. (Hatamizadeh et al., 2022) extended (Yin
et al., 2021a)’s approach to vision transformers and discovered that, because
of the attention mechanism, vision transformers are substantially more
sensitive than previously researched CNNs. In the image domain, total variance
is another choice (Wang et al., 2019a; Geiping et al., 2020).

Selecting a proper loss function can also contribute to attack efficiency. By
combining mean square error and Wasserstein distance (Arjovsky et al., 2017),
Ren et al. (Ren et al., 2022) achieved a better reconstruction result than
(Zhu et al., 2019; Zhao et al., 2020) in terms of batch size and
reconstruction fidelity. Geiping et al. (Geiping et al., 2020) adopted cosine
distances to better capture the observation that the angle between two data
points quantifies the change in prediction. With this method, they rebuilt a
single high-resolution image and a series of low-resolution photos with a
maximum batch size of 100. Jeon et al. (Jeon et al., 2021) systematically
investigated ways to best utilize gradients, including using them to extract
prior information. Wei et al. (Wei et al., 2020b) conducted a thorough
analysis of how different hyper-parameter setups and settings for attack
algorithms influence the effectiveness of the attack and its cost. In an
effort to investigate the worst-case attack and evaluate the effectiveness of
defense methods for reconstruction attacks, Balunovic et al. (Balunović et
al., 2021) formulated the gradient leakage problem in a Bayesian framework and
analyzed the condition for a Bayes optimal adversary.

####  3.4.2. Closed-form Attack

In an attempt to provide a theoretical understanding of how and when gradients
lead to the remarkable recovery of original data, several studies investigated
the possibility of recovering the input of a learnable affine function from
gradients (Aono et al., 2017; Qian and Hansen, 2020; Fan et al., 2020). Aono
et al. (Aono et al., 2017) initiated the closed-form attack based on
gradients. In certain circumstances, an honest-but-curious server could
directly calculate individual data from the gradients uploaded by clients
(Aono et al., 2017; Zhu and Blaschko, 2020).

Consider a fully connected layer $Wx+b=z$ with $l=l(f(x),y)$ as the loss
function, where $x$ and $z$ are the input and output vectors, respectively.
Private data $x$ can be derived from $l$’s gradients w.r.t $W$ and $b$, i.e.,:
${x^{T}}=\frac{{\partial l}}{{\partial W}}\oslash\frac{{\partial l}}{{\partial
b}}$. Here $\oslash$ denotes entry-wise division. The assumption of a fully
connected layer holds for the last prediction layers in many popular
architectures. As a result, the prediction modules’ input, which is the output
of the preceding layers, can be rebuilt. These outputs typically include some
information about the training data, making them vulnerable to attackers. In
this light, the ability to recover ground truth label information from the
gradients of the final fully-connected layer, as stated in Zhao et al. (Zhao
et al., 2020), is very intriguing.

Despite its inspiring nature, Aono et al.’s work (Aono et al., 2017) has some
limitations. First, it does not apply to convolutional neural networks due to
a mismatch in dimensions. Second, it cannot deal with batch inputs. For the
batch input $\left\\{x_{j}\right\\}_{j=1}^{b}$, all derivatives are summed
over the batch dimension $b$ and the recovered $\bar{x}$ is merely
proportional to the average of batch inputs
$\sum\nolimits_{j=1}^{b}{{x_{j}}}$.

To fix the dimension mismatch issue, Zhu and Blaschko (Zhu and Blaschko, 2020)
converted the convolutional layer into a fully connected layer using circulant
matrix representation (Golub and Van Loan, 2013) of the convolutional kernel
(Fan et al., 2020). The gradients of each layer were interpreted as the
gradient constraints. Finally, they recursively reconstructed the layer-wise
input. However, their implementation can only recover low-resolution images in
settings where the batch size equals 1. For the batch input, the algorithm
returns a linear combination of the training data.

To address these difficulties with the batch size, Fowl et al. (Fowl et al.,
2021) suggested making minor but malicious changes to the global model to
reconstruct the client’s data from a batch of gradient updates. The key idea
is to separate the batch data by some quantity $h$, such as image brightness.
To this end, an imprint module is added to the global model, which acts as a
filter that separates a batch of samples based on quantity $h$. Qian and
Hansen (Qian and Hansen, 2020) found bias term in the output layer is the key
to the success of reconstruction. A fully-connected neural network requires
just one node in one hidden layer for single-input reconstruction. In
contrast, mini-batch reconstruction requires that the hidden units exceed the
input size. Pan et al. (Pan et al., 2020) conducted an analytic investigation
of the security boundary of the reconstruction attacks. Given a batch input,
the secure/insecure boundary of the reconstruction attack was characterized by
the number of Exclusively Activated Neurons (ExANs), where the more ExANs, the
more likely the attack’s success.

Table 4. Comparison of reconstruction attacks in FL Method | Objective function |  Maximal batch size |  Opt-based/ Closed-form |  Theoretical guarantee |  Additional information |
---|---|---|---|---|---|---
$\mathcal{L}_{grad}$ | $\mathcal{R}_{aux}$ |
iDLG (Zhao et al., 2020) |  $l_{2}$ distance | / | 8 | Opt-based | No | No |
DLG (Zhu et al., 2019) |  $l_{2}$ distance | / | 8 | Opt-based | Yes | No |
| Inverting
---
gradients(Geiping et al., 2020)
| Cosine
---
similarity
Total variance | 100 | Opt-based | Yes |  | Local updates;
---
BN statistcs
(Wei et al., 2020b) |  $l_{2}$ distance |  | Label-based
---
regularizer
8 | Opt-based | Yes | No |
SAPAG (Wang et al., 2020a) |  | Gaussian
---
kerneal
based function
/ | 8 | Opt-based | No | No |
R-GAP (Zhu and Blaschko, 2020) |  | Recursive
---
gradients
/ | 5 | Closed-form | No | No |
| Theory-
---
oriented (Pan et al., 2020)
$l_{2}$ distance |  |  $l_{1}$ distance of
---
feature map
32 | Closed-form | Yes |  | Exclusive
---
activated
neurons
| GradInversion
---
(Yin et al., 2021a)
$l_{2}$ distance |  | Group
---
consistency
48 | Opt-based | No | BN statistcs |
CAFÉ (Jin et al., 2021) |  $l_{2}$ distance | Total variance | 100 | Opt-based | Yes | Batch indices |
GIAS&GIM (Jeon et al., 2021) |  | Negative
---
cosine
|  $l_{2}$ distance in
---
latent space
4 | Opt-based | No | No |
| Imprint
---
module (Fowl et al., 2021)
| One-shot
---
mechanism
/ | 16384 | Closed-form | Yes | CDF |
GradViT (Hatamizadeh et al., 2022) |  $l_{2}$ distance |  | Image prior;
---
Auxiliary
Regularization
64 | Opt-based | No |  | Auxiliary
---
networks

####  3.4.3. Discussion

The closed-form attacks outperform optimization-based attacks in several
aspects. First, the closed-form attacks provide a theoretical guarantee of
convergence. In contrast optimization-based attacks suffer from the local
optimum problem since a non-convex optimization may not always converge to a
correct solution (Geiping et al., 2020). Further, optimization-based attacks
are sensitive to initialization (Zhu et al., 2019), whereas closed-form
attacks do not. Second, the deterministic algorithms run by closed-form
attacks are faster than optimization-based attacks. Third, closed-form attacks
recover the data more accurately, while optimization-based methods, like
GradInversion (Yin et al., 2021a), recover data with artifacts. Jin et al.
(Jin et al., 2021) made a good comparison between different reconstruction
attacks in FL. Table 4 summarizes their finding and includes some additional
results from this study.

###  3.5. Privacy-preserving Techniques

Privacy-preserving machine learning approaches can be roughly classified as
cryptographic approaches and perturbation approaches. Cryptographic approaches
enable computation over encrypted data and provide rigorous privacy guarantees
in the training process. However, they come at a high computational cost
compared to the non-encryption alternatives (Xu et al., 2021a). This
computation overhead limits their application in some learning scenarios,
particularly in deep neural networks with huge amounts of parameters. As a
result, most state-of-the-art privacy-preserving methods are perturbation-
based. The perturbation can be accomplished by adding artifact noise into the
dataset, such as DP mechanism (Geyer et al., 2017; Wei et al., 2020a). Or by
representing the raw dataset with a surrogate dataset (Wu et al., 2021;
Scheliga et al., 2022) or abstracting the dataset via sketch techniques (Li et
al., 2019a; Haddadpour et al., 2020).

####  3.5.1. Cryptographic Approaches

Secure multi-party computation is a sub-field of cryptography that executes
calculations on data dispersed among multiple parties in such a way that the
computation results are only revealed to the participants (Yao, 1982). It can
take the form of homomorphic encryption (HE) or secret sharing.

As one of the defacto privacy-preserving solutions, homomorphic encryption
(HE) provides perfect privacy protection in the face of a malicious server. It
allows clients to encrypt their updates in such a way that the server may
directly aggregate ciphertexts without divulging anything about the plain text
underneath. The downside is that encryption followed by decryption will
inevitably impose both a computation and a communications overhead. Phong et
al. (Phong et al., 2018) used additively homomorphic encryption to ensure no
information was leaked to a malicious server. The encrypted aggregation was
formulated as follows:

(4) $\bf{E}(W_{global}):=\bf{E}(W_{global})+\bf{E}(-\alpha\cdot G_{local})$

where E is a homomorphic encryption operator that supports addition over
ciphertexts, $\bf{G_{local}}$ is the aggregated gradient. The decryption key
is public to the clients and private to the server, and thus, the client’s
information is secured. Due to the additively homomorphic property of E, each
client is still able to receive the correct updated model $W_{global}$ via
decryption.

(5) $\bf{E}(W_{global})+\bf{E}(-\alpha\cdot
G_{local})=\bf{E}(W_{global}-\alpha\cdot G_{local})$

However, the amount of data transferred between clients and the server is
inflated by two orders of magnitude over the vanilla setting (Phong et al.,
2018). To reduce the communication load, Zhang et al. (Zhang et al., 2019a)
chose a distributed selective stochastic gradient descent (DSSGD) method in
the local training phase to achieve distributed encryption and reduce the
computation costs. Zhang et al. (Zhang et al., 2020c) presented a BatchCrypt
as a simple batch encryption technique. Clients first quantize their local
gradients and then encode a batch of quantized updates into a long integer. As
a result, the communication overhead is reduced by up to 101 times. Jiang et
al. (Jiang et al., 2021) further reduced communication overhead by sending
only a sparse subset of local states to the server.

Another drawback is that all participants share the same private key for
decryption since homomorphic operations require all values to be encrypted
with the same public kewhich degrades privacy protection in the face of
malicious clients. To counter this problem, Park and Lim (Park and Lim, 2022)
sketched a privacy-preserving FL scheme based on a distributed homomorphic
cryptosystem that allows clients to have their own unique private key for the
homomorphic encryption scheme.

Secret sharing (Shamir, 1979) is another kind of cryptographic technique. It
splits a secret data $\mathcal{D}$ into $n$ pieces such that the secret
$\mathcal{D}$ can be easily reconstructed with at least $k$ pieces. However,
any set containing less than $k$ piece reveals no information about
$\mathcal{D}$. It enables the server to aggregate at least a certain number of
clients’ updates without disclosing any individual client’s contribution.
Bonawitz et al. (Bonawitz et al., 2017; Mondal et al., 2022) proposed a secure
aggregation method for FL based on $t$-out-of-$n$ secret sharing. The key idea
is to mask the raw data in a symmetric way. Thus, when aggregated by the
server, the introduced noise will be nullified. Liu et al. (Liu et al., 2020)
incorporated secure sharing into their federated transfer learning framework
to protect privacy. Based on an investigation of how secure aggregation
parameters influence communication efficiency, Bonawitz et al. (Bonawitz et
al., 2019) used quantization to build a communication-efficient secure
aggregation scheme. So et al. (So et al., 2021) designed Turbo-Aggregate that
leverages additive secret sharing and Lagrange coding to reduce the secure
aggregation overhead. Shao et al. (Shao et al., 2022) shared a similar ideal,
utilizing Lagrange coding to secretly share private datasets among clients.

Even though FL based on a homomorphic encryption scheme can prevent privacy
leaks during training, it remains vulnerable to attacks in the inference
stage. The trained model embodies the distribution of training data to a
certain extent and the privacy risk still exists. Model inversion attack gives
such an example. Given the white-box access to a trained model, Zhang et al.
(Zhang et al., 2020a) successfully discovered the sensitive features x
associated with a specific label y.

####  3.5.2. Perturbation Methods

Due to its theoretical guarantee of privacy protection and its low
computational and communication complexity, the DP technique (Dwork, 2008) has
emerged as the most popular choice for privacy protection among a variety of
options. In differential privacy, a proper amount of noise is added to the raw
data (Gupta and Raskar, 2018), the model (Liu and Meng, 2020; Geyer et al.,
2017), the output (Chaudhuri et al., 2011), or the gradients (Song et al.,
2013) to protect privacy.

Geyer et al. (Geyer et al., 2017) applied a differentially private mechanism
to the FL scenario, where they approximated the averaging operations with a
randomized mechanism that provided client-level privacy. Truex et al. (Truex
et al., 2019a) presented an alternative approach that draws on both DP and
secure multi-party computation. The clients and the server communicate through
a secure channel. Upon receiving a request from the server, the clients upload
their answers following the principles of DP. Xu et al. (Xu et al., 2019a)
took a different approach. They approximated the objective function of a
regression problem via polynomial representation and then added Laplace noise
to the polynomial coefficients to protect privacy. Khalili et al. (Khalili et
al., 2021) exploited an exponential mechanism (McSherry and Talwar, 2007) to
privately select applicants based on the qualification scores predicted by a
pre-trained model.

One concern with perturbation techniques is the trade-off between privacy,
accuracy and convergence. A significant noise perfectly protects privacy at
the cost of accuracy and convergence. Conversely, a weak noise is futile to
privacy attacks (Hitaj et al., 2017). Wei et al. (Wei et al., 2020a) conducted
a theoretical analysis of the convergence behavior of FL with DP and
identified the trade-off between convergence performance and privacy
protection levels.

Many studies have been published on ways to deal with this trade-off (Fan et
al., 2020; Zhang and Wang, 2021; Shokri and Shmatikov, 2015). Shokri and
Shmatikov (Shokri and Shmatikov, 2015) suggested randomly selecting and
sharing a small fraction of gradient elements (those with large magnitudes) to
reduce privacy loss. Fan et al. (Fan et al., 2020) leveraged element-wise
adaptive gradient perturbations to defeat reconstruction attacks and maintain
high model accuracy. In a similar manner, Wei and Liu (Wei and Liu, 2021) used
dynamic privacy parameters. Introducing noise with a greater variance at the
beginning of training and progressively decreasing the amount of noise and
variance as training progresses.

Huang et al. (Huang et al., 2020b) proposed InstaHide, a combination of
cryptographic and perturbation approaches to provide rigorous privacy
protection at the cost of minor effects on accuracy. InstaHide encrypts the
raw image by mixing it with multiple random images from a large public
dataset. After that, it randomly flips the signs of the pixels before using it
to train the model.

Yang created et al. (Yang et al., 2021) NISS to avoid the trade-off between
accuracy and privacy by permitting clients to collaborate on reducing the
total amount of injected noise. In particular, each client’s noise is
neutralized and distributed to other clients. Theoretically, if all clients
are trustworthy, the locally introduced noise can be perfectly offset by the
server’s aggregation, completely avoiding the privacy accuracy trade-off. A
similar idea can be found in Yang et al. (Yang et al., 2020a).

####  3.5.3. Trusted Execution Environment

Some researchers use Trusted Execution Contexts (TEEs) like Intel SGX and ARM
TrustZone to secure ML training in untrusted environments (Tramer and Boneh,
2018; Mo et al., 2020; Gu et al., 2019). With hardware and software
safeguards, TEEs secure critical code from other programs. Compared with
purely cryptography methods, TEEs provide much better performance since it
only requires extra operations to create the trusted environment and
communicate between trusted and untrusted components. Gu et al. (Gu et al.,
2019) partitioned DNN models and solely encased the first layers in an SGX-
powered TEE to protect input information. Hynes et al. (Hynes et al., 2018)
investigated speeding up the training using Graphics Processing Units (GPU).
Tramer et al. (Tramer and Boneh, 2018) shared the same concept and offered
effective privacy-preserving neural network inference utilizing trusted
hardware that delegated matrix multiplication to an untrusted GPU. However,
this work does not translate well to FL due to the possible adversary server
and limited computation power of the client device. To remedy this, Mo et al.
(Mo et al., 2020) advocated using the TEE of client devices in tandem with
model partitioning to defend against MIA. The model is divided into two
halves, and the final layers are calculated within TEE. Kato et al. (Kato et
al., 2022) proposed to combine DP with TEE in FL in the presence of an
untrusted server. The models are aggregated within the TEE of the server’s
device. (Chen et al., 2020b)

####  3.5.4. Discussion

Table 5 summarized and compared the existing defense techniques. Cryptographic
approaches preserve privacy to a great extent while suffering from
computational complexity and are less feasible. The perturbation approaches
trade-off privacy for model performance. Several inspiring works demonstrate
that it may be possible to avoid that trade-off through either client
collaborations to neutralize locally added noise on the server side or by
using a surrogate dataset to protect the raw data without adding noise. The
cryptographic approaches only ensure that no information will leak during
training. They do not protect privacy during the inference stage. In contrast,
the perturbation approaches (e.g., DP) protect privacy in both the training
and inference stages. One may combine cryptographic and perturbation
approaches to obtain better privacy protection throughout the machine learning
pipeline.

Table 5. Privacy-preserving methods in FL

Attack |  | Defense
---
method
Rationale | Advantage | Disadvantage
RA |  | HE
---
(Aono et al., 2017; Shin et al., 2021; Jiang et al., 2021)
| Gradients are encrypted
---
Accurate |  | 1\. Vulnerable if there are
---
multiple colluding entities;
2\. Ineffective at inference
| Secret sharing
---
(Bonawitz et al., 2017; Mondal et al., 2022; Shao et al., 2022)
| Hiding information about
---
clients’ individual update,
except for their sum
| 1\. Accurate; 2. Robust to
---
users dropping out
| Ineffective at inference
---
| Variational
---
bottleneck
(Scheliga et al., 2022)
| Using surrogate gradient
---
to protect privacy.
| Keep training process
---
and performance intact
| Limit to optimization-based
---
attack
| Gradient
---
compression
(Sun et al., 2020; Zhu et al., 2019; Li et al., 2022)
| Compressing gradients to
---
prevent reconstruct private
data by matching gradients
| 1\. Easy to implement;
---
2\. Reduce communication
Requires considerable noise, degrades model performance, and increases
convergence time
RA, MIA, and PIA |  | DP
---
(Wei et al., 2020b; Chen et al., 2020c; Ma et al., 2021)
| Hiding private information
---
by injecting noise to the
raw data, model, or output
| 1\. Easy to implement;
---
2\. Long-term protection
| TEEs
---
(Mo et al., 2020; Kato et al., 2022)
| Isolating part of networks
---
from the untrusted
environments
Reduce computation | Limited memory space
RA: Reconstruction attack; MIA: Membership inference attack; PIA: Property
inference attack.

Table 6. Comparison of privacy attacks in FL Attack | Ref. | Access |  | Attack
---
interface
Assumptions | Key technique
Membership Inference Attack | (Shokri et al., 2017) | BB |  | Confidence
---
vector
| Knowledge about population
---
data
| Inferring from the discrepancies of
---
predictions on training set versus
unseen data
(Nasr et al., 2019) | WB |  | Model
---
parameters
| A significant portion of the
---
training data
| Reversing the SGD algorithm
---
| (Leino and Fredrikson, 2020) | WB |  | Model
---
parameters
| A proxy dataset sampled from
---
the ground-truth distribution
| Inferring from parameter differences
---
between the target and proxy model
Property Inference Attack | (Ateniese et al., 2013) | WB |  | Model
---
parameters
| 1.Knowledge about training
---
data structure; 2. Access to the
ground-truth distribution
| A meta-classifier to infer properties
---
from multiple shadow classifier
parameters
| (Wang et al., 2022) | WB |  | Model
---
updates
| Adversary can manipulate
---
more than one device in FL
| Inferring other clients’ data from the
---
periodic model updates
| (Wang et al., 2019b) | WB |  | Model
---
updates
| 1\. Client’s average label count;
---
2\. Number of samples per label
| Inferring from the layer neuron
---
weight changes
Model Inversion Attack | (Fredrikson et al., 2015) | BB/WB |  | Confidence
---
vectors /
Model
parameters
| 1\. Side information; 2. Simple
---
networks
| Optimizing the input to maximize
---
confidence vectors subject to the
classification matches the target
(Yang et al., 2019a) | BB |  | Confidence
---
vectors
| 1\. Auxiliary dataset retains
---
meaningful prior information;
2\. Simple networks
| An inverse model approximates the
---
mapping between predictions
and images
(Zhang et al., 2020a) | WB |  | Feature
---
extractor
| An auxiliary dataset retains
---
meaningful prior information
| 1\. Distilling prior knowledge from
---
an auxiliary dataset via GAN;
2\. Optimizing the generate image to
maximize likelihood
Reconstruction Attack | (Wang et al., 2019a) | WB |  | Client
---
updates
| 1\. Shallow target model;
---
2\. Low-resolution image
| A GAN with multi-task discriminator
---
to enhance fidelity and identify client
(Geiping et al., 2020) | WB |  | Client
---
updates
| 1\. Honest-but-curious server in
---
FL; 2. Small batches
| Optimizing the image to get a similar
---
change in model prediction as the
target images
(Zhu et al., 2019) | WB | Gradients |  | 1\. Small image size; 2. Single
---
batches; 3. Target model is
twice differentiable
| Jointly optimizing inputs and labels
---
to match gradients
(Hatamizadeh et al., 2022) | WB | Gradients |  | Auxiliary networks provide
---
image prior
| Optimizing inputs to match the target
---
gradients with image prior and
auxiliary regularizer
(Yin et al., 2021a) | WB | BN layers |  | 1\. BN layers in target model;
---
2\. No repeating labels in a
batch.
| Optimizing the input to match the
---
statistics of BN layers of the target
model
BB: black-box; WB: white-box

###  3.6. Discussion of privacy attacks and defenses in FL

This section reviews existing privacy attacks and defense approaches in FL.
Table 6 summarized the existing privacy attacks in FL. From the attacker’s
perspective, FL differentiates from the centralized counterparts in sever
aspects: 1) The active attacker in FL. Due to the collaboration between
clients and the server, an adversary could actively attack for victim’s
private data. For example, the attacker may maliciously reverse the gradients
(Nasr et al., 2019) or mislabel the training sample (Hitaj et al., 2017) to
neutralize the benign clients’ efforts and fool them into revealing more
information about their private data. Hence making them more venerable
compared to centralized machine learning. 2) The real-time nature of FL
strengthens the attacker’s ability. During the training process, the adversary
could adaptive change their strategy to infer the victim’s private data. As a
result, the adversary can even infer a specific clients’ data (Wang et al.,
2019a) when the target features appear in FL (Wang et al., 2019b, 2022), which
is way more severe than centralized setting. 3) Gradients are shared between
clients and server. Unlike the centralized counterpart, where the adversary
could at most access the white-box access to the target model. In FL,
gradients are repeatedly shared between clients and the server. Which enables
gradient-based privacy attacks. As shown by (Zhu et al., 2019; Yin et al.,
2021a), the malicious server can reconstruct clients’ training data at the
pixel level by minimizing the distance to the target gradients.

From the defender’s perspective, protecting privacy in FL is also different
from that in the centralized scenario. 1) Malicious could be the server or any
client. FL allows clients to keep private data local. A central server is
designed to orchestrate the training process. Which complicated privacy
protection. The adversary could be the server (Wang et al., 2019a; Ren et al.,
2022; Geiping et al., 2020) or the client (Hitaj et al., 2017). The malicious
adversary is able to infer the target client’s privacy passively or actively.
For example, sending the modified global model to the target client to probe
private data (Fowl et al., 2021). This brings challenges to defending against
potential privacy attacks. DP is a prevailing choice, but it degrades
performance. The cryptographic approaches, like HE, and MPC, retain both
privacy and performance at the cost of computation overhead, which is a more
severe issue in FL since most clients’ devices are limited in computation
power. 2) Training and inference stage privacy attacks. Different from
centralized machine learning, where the major privacy leakage happens at the
inference stage, i.e., malicious users probe private training data by
inferring the target model. In FL, the attacks could happen during or after
the training. This requires the defenders to be aware of both possibilities.
The cryptographic approaches proved provable privacy protection during the
training stage. However, fail at the inference stage since training
distribution is embedded in the trained model’s parameters. The perturbation
approaches, e.g., DP, provides long-term protection and covers both the
training and inference stage. One can hide sensitive information from
adversaries by adding appropriate noise to the training data.

##  4\. Fairness in FL

Fairness, as discussed in the centralized setting, is mainly defined at either
the group level (Kusner et al., 2018; Hardt et al., 2016) or the individual
level (Dwork et al., 2012). In the FL scenario, fairness has a broader
definition. Beyond the long-established algorithmic fairness (Kusner et al.,
2018; Hardt et al., 2016), client-level fairness (Yu et al., 2020; Li et al.,
2019b; Mohri et al., 2019; Martinez et al., 2020) arises as a new challenge in
FL.

###  4.1. Algorithmic Fairness

Algorithmic fairness is commonly used to describe the discrepancies in
algorithm decisions made across distinct groups as defined by a sensitive
attribute. FL often involves a deep neural network with redundant parameters
and is pruned to overfit the privileged groups. Various debiasing methods have
been devised for different applications, including machine learning (Zafar et
al., 2017; Zhang et al., 2019b; Berk et al., 2017; Bolukbasi et al., 2016;
Brunet et al., 2019), representation learning (Louizos et al., 2016; Zhang et
al., 2018a), and natural language processing (Zhao et al., 2018b; May et al.,
2019; Bordia and Bowman, 2019; Font and Costa-jussà, 2019). These methods vary
in detail but share similar principles. Following the data flow, debiasing
methods can be grouped into pre-processing, in-processing and post-processing
categories, which address the discriminate issues at three distinct stages of
the data’s handling (d’Alessandro et al., 2017).

####  4.1.1. Pre-processing

Pre-processing tries to remove the underlying discrimination from the data
typically by 1) altering the values of the sensitive attributes/class labels;
2) mapping the training data to a new space where the sensitive attributes and
class labels are no longer relevant (Feldman et al., 2015; Kamiran and
Calders, 2012, 2010); or 3) reweighting the samples in the training dataset to
compensate for skewed treatment (Kamiran and Calders, 2012).

Intuitively, by training a classifier on discrimination-free data, it is
likely that the resulting predictions will be discrimination-free. Inspired by
this idea, Kamiran and Calders (Kamiran and Calders, 2012) proposed three
types of pre-processing solutions to learn a fair classification, messaging,
reweighing and sampling. Feldman et al. (Feldman et al., 2015) investigated
the problem of identifying and removing disparate impacts in the data. Xu et
al. (Xu et al., 2018) proposed FairGAN, which generates fair data from the
original training data and uses the generated data to train the model. Abay et
al. (Abay et al., 2020) proposed two reweighting methods for the FL setting
(Kamiran and Calders, 2012), local reweighing and global reweighing with DP.

Notably, these pre-processing techniques require access to the training data,
which violates the privacy principles of FL. As a result, these types of
techniques can only be deployed locally on each client. However, in the
presence of data heterogeneous among clients, local debiasing cannot provide
fair performance for an entire population (Cui et al., 2021).

####  4.1.2. In-processing

In-processing modifies traditional learning algorithms to address
discrimination (Zafar et al., 2017; Goh et al., 2016; Kamishima et al., 2012;
Berk et al., 2017; Zhang et al., 2018b; Zemel et al., 2013). Such as adding a
regularization term to the loss function. Berk et al. (Berk et al., 2017), for
example, incorporated a family of fairness regularizers into the objective
function for regression problems. These regularizers span the range from
notions of group fairness to individual fairness. They also create a trade-off
between accuracy and fairness.

Another in-processing option is imposing constraints. Zhang et al. (Zhang et
al., 2018b) used a GAN to constrain the bias in a model trained on biased
data. During training, the scheme simultaneously tries to maximize the
accuracy of the predictor while minimizing the ability of the adversary to
predict the protected variable. In FL, G’alvez et al. (G’alvez et al., 2021)
studied the notion of group fairness as an optimization problem with fairness
constraints. Papadaki et al. (Papadaki et al., 2021) formulated a min-max
optimization problem to investigate group fairness in scenarios where
population data were distributed across clients. Ezzeldin et al. (Ezzeldin et
al., 2021) replaced the aggregation protocol FedAvg with FairFed, which
adaptively updates the aggregating weights in each round to improve group
fairness. Clients whose local measurements match the global fairness measure
are given preferential treatment. Khedr et al. (Khedr and Shoukry, 2022) add a
regularizer term to minimize the average loss in fairness across all training
data.

####  4.1.3. Post-processing

Post-processing addresses discrimination issues after the model is trained and
doesn’t need to change the training process. The general methodology of post-
processing algorithms is to take a subset of samples and change their
predicted labels to meet a group fairness requirement (Calders and Verwer,
2010; Hardt et al., 2016; Bolukbasi et al., 2016; Pleiss et al., 2017; Canetti
et al., 2019; Lohia et al., 2019).

Hardt et al. (Hardt et al., 2016) proposed a post-processing technique to
construct a non-discriminating predictor $\tilde{Y}$ from a learned
discriminatory binary predictor $\hat{Y}$. Only access to the prediction
$\hat{Y}$, the protected attribute $A$ and target label $Y$ in the data are
required, while details of the mapping of features $X$ to prediction $\hat{Y}$
are not needed. Canetti et al. (Canetti et al., 2019) and Pleiss et al.
(Pleiss et al., 2017) shared the key characteristics as Hardt et al.’s (Hardt
et al., 2016) work. Lohia et al. (Luo et al., 2021) designed a post-processing
method to increase both individual and group fairness. Salvador et al.
(Salvador et al., 2021) introduced a conditional calibration method for fair
face verification. Their method clusters images into different sets and
assigns distinct thresholds to different sets.

####  4.1.4. Discussion

Three different kinds of debiasing methods are at hand in centralized machine
learning. However, solutions in the centralized setting cannot be applied
directly in the FL scenario due to limitations with the training data. More
specifically, in federated settings, the clients usually have limited amounts
of data. Hence, a single client can’t accurately represent the true
distribution over all clients. Consequently, debiasing data before training is
not an option. Another limitation is that direct access to local data is
prohibited on the server side. Nevertheless, canny researchers have found
inspiration from and workarounds to these issues. Gálvez et al. (G’alvez et
al., 2021), for example, bypassed this access restriction by using statistics
to guide the model’s training instead of the raw data.

###  4.2. Client Fairness

Client fairness in FL is another different fairness notion than algorithmic
notions. Ideally, the models produced from FL should capture clients’ data
distributions and generalize well when deployed on the client side. However,
data distribution usually varies among clients. As a result, the global model
has inconsistent performance on different clients’ dataset. At the client
level, a FL protocol is considered to be fair if the performance fluctuates
within a limited range, i.e., the variance in the model’s performance across
clients falls under a predefined threshold. To this end, two lines of research
exist to mitigate fairness issues in FL. These are the single model approach
and the personalized models approach.

####  4.2.1. Single Model Approach

The single model approach trains a single global model for all clients as a
standard FL scheme. Here, the focus is on solving any statistical
heterogeneity during the training phase rather than smoothing the distribution
difference.

  * •

Data augmentation is a straightforward solution to statistical heterogeneity.
It increases data diversity on the client side. Several researchers have
studied ways to enhance the statistical homogeneity of local data in FL (Zhao
et al., 2018a; Jeong et al., 2018; Hao et al., 2021). Zhao et al. (Zhao et
al., 2018a) suggested a data share scheme, which creates a globally-shared
dataset that is balanced by class. The experiment shows a 30% improvement on
accuracy with only 5% globally shared data. Jeong et al. (Jeong et al., 2018)
proposed FAug. Clients first collectively train a GAN model, which is then
distributed to clients to augment their local data towards yielding an i.i.d
dataset.

  * •

Client Selection is another strategy that focuses on sampling data from a
homogeneous distribution. Wang et al. (Wang et al., 2020c) proposed a control
framework to actively select the best subset of clients in each training
round. In Yang et al.’s (Yang et al., 2020b) method, the local data
distribution is estimated first by comparing local updated gradients and
gradients inferred from a balanced proxy dataset. The client selection
algorithm based on a combinatorial multi-armed bandit was designed to minimize
the effect of class imbalances.

  * •

Agnostic approach trains a robust model against a possible unknown testing
distribution. Mohri et al. (Mohri et al., 2019) modeled testing distributions
as an unknown mixture of all $m$ clients’ data. The global model is optimized
for all possible target distributions. This makes the global model more robust
to an unknown testing distribution. Du et al. (Du et al., 2021) introduced a
fairness constraint into Mohri et al.’s method (Mohri et al., 2019) and
proposed AgnosticFair, a fairness-aware FL framework. Their method can provide
both _Good-intent fairness_ and _demographic parity_

  * •

Reweighting tries to train a fair model by assigning suitable aggregating
weights $p_{k}$ in Eq. 1 to clients. Inspired by $\alpha$-fairness notions
(Lan et al., 2010; Mo and Walrand, 2000), Li et al. (Li et al., 2019b)
sketched $q$-Fair FL ($q$-FFL) to foster fairer accuracy distribution across
all clients by up-weighing clients with lower performance during aggregation.
Huang et al. (Huang et al., 2020a) shared a similar idea where, for each round
of aggregation, clients with lower accuracy or less training participant times
are assigned higher aggregation weights.

####  4.2.2. Personalized Models Approach

Instead of smoothing the statistical heterogeneity, in personalized FL,
multiple distinct models are trained for clients with different data
distributions. A global model is first trained collaboratively and then
personalized to clients using private data. In this way, clients can benefit
from other clients’ data and solve the issue of statistical heterogeneity.
Mansour et al. (Mansour et al., 2020) designed and analyzed three approaches
to learning personalized models to learn personalized models. Kulkarni et al.
(Kulkarni et al., 2020) conducted a brief overview of personalized FL. Chen et
al. (Chen et al., 2022) provided a comprehensive benchmark of various
personalized FL methods. Tan et al. (Tan et al., 2021) systematically reviewed
this topic and classified personalized FL techniques in terms of data-based
and model-based approaches. Here, we summarize their conclusions.

  * •

Multi-task learning treats building models for each client as different tasks.
Smith et al. (Smith et al., 2017) pioneered this approach and explored
personalized FL via a multi-task learning framework. (Agarwal et al., 2020;
Liang et al., 2020) followed this principle. Dinh et al. (Dinh et al., 2021)
proposed FedU, which incorporates a Laplacian regularization term into the
optimization problem to leverage relationships between clients.

  * •

Model interpolation trains local and global models simultaneously, where the
global model is used for its generalization ability, and the local model is
used to improve local performance. Hanzely and Richtárik (Hanzely and
Richtárik, 2021) formulated an optimization problem that learns a mixture of
the global and local models. The local model is trained solely on each
client’s private data. Softly-enforced similarity from multi-task learning is
borrowed to discourage the local model from departing too much from the mean
model. Deng et al. (Deng et al., 2020) and Mansour et al. (Mansour et al.,
2020) adopt a similar formulation to determine the optimal interpolation of
the local and global models. In Zhang et al.’s (Zhang et al., 2020d) work,
clients are given access to multiple models uploaded by other clients to
evaluate how much they will benefit from these models. An optimal combination
is then used as a personal update. Lin et al. (Lin et al., 2022) investigated
the trade-offs between local and global models.

  * •

Parameter decoupling learns local parameters as an independent task performed
locally. The local model is designed to assist in personalizing the global
model to local distributions. Liang et al. (Liang et al., 2020) devised the
local-global federated averaging algorithm, which jointly learns compact local
representations for each client and a global model across all devices. Chen
and Chao (Chen and Chao, 2021) decomposed a FL model as a generic predictor,
which is trained globally, along with a personalized predictor that is trained
locally. The personalized predictor is formulated as a lightweight, adaptive
module on top of the generic predictor.

  * •

Transfer learning is a practical training paradigm that leverages knowledge
from a source domain to help train a model in a target domain. The performance
of transfer learning depends on the similarity between the two domains.
Federated transfer learning was first introduced by Liu et al. (Liu et al.,
2020). Since clients in the same federation usually share the same domain, a
FL scheme would make a suitable partner for transfer learning. Li and Wang (Li
and Wang, 2019) subsequently proposed FedMD, which combines transfer learning
and knowledge distillation. Each client performs transfer learning by training
a model to converge on a public dataset and subsequently fine-tune it on local
data.

  * •

Clustering arranges clients into different groups and trains a specific model
for each group. Ghosh et al. (Ghosh et al., 2020) iteratively determines the
membership of each client to a cluster and optimizes each of the cluster
models via gradient descent in a distributed setting. Sattler et al. (Sattler
et al., 2020) clusters clients according to the cosine similarity between the
clients’ gradient updates. This allows clients with a similar distribution to
profit from one another while minimizing detrimental interference from others.
In Briggs et al.’s (Briggs et al., 2020) method, a clustering step is
periodically inserted into the training process to cluster clients based on
their local updates. The clusters are then trained individually and in
parallel on specialized models. Mansour et al. (Mansour et al., 2020) proposed
hypothesis-based clustering, partitioning clients into $q$ clusters and
finding the best hypothesis for each cluster.

  * •

Regularization prevents overfitting when training models and has been used in
several studies to remedy the weight divergence problem in FL settings. Li et
al. (Li et al., 2020) introduced a proximal term that considers the
differences between global and local models to limit the effect of local
updates. Yao et al. (Yao and Sun, 2020) considered parameter importance in the
regularised local loss function by using elastic weight consolidation
(Kirkpatrick et al., 2017). In addition, a regularization term is introduced
to penalize the deviation of the local model from the global model.

  * •

Meta-learning aims to leverage prior experience with other tasks to facilitate
the learning process. The resulting models are highly-adaptable to new
heterogeneous tasks (Nichol et al., 2018; Finn et al., 2017). Fallah et al.
(Fallah et al., 2020) studied a personalized variant of FedAvg based on model-
agnostic meta-learning formulation. The proposed Per-FedAvg algorithm looks
for an initial model that performs well after one step of the local gradient
update on each client’s data. Others have interpreted FedAvg as a meta-
learning algorithm, breaking it into two stages of training and fine-tuning to
optimize personalized performance and model convergence (Jiang et al., 2019;
Khodak et al., 2019).

Table 7. Summary of Fairness-aware FL Reference |  | Single
---
Model
| Personalized
---
Model
| Algorithmic
---
Fairness
| Client
---
Fairness
Method
(Zhao et al., 2018a; Jeong et al., 2018; Hao et al., 2021) | ✓ |  |  | ✓ | Data Augmentation
(Wang et al., 2020c; Duan et al., 2021; Yang et al., 2020b) | ✓ |  |  | ✓ | Client Selection
(Papadaki et al., 2021) | ✓ |  | ✓ |  | Agnostic approach
(Du et al., 2021) | ✓ |  | ✓ | ✓ | Agnostic approach
(Mohri et al., 2019; Hu et al., 2020) | ✓ |  |  | ✓ | Agnostic approach
(Ezzeldin et al., 2021) | ✓ |  | ✓ |  | Reweight
(Li et al., 2019b; Huang et al., 2020a) | ✓ |  |  | ✓ | Reweight
(Kanaparthy et al., 2022; G’alvez et al., 2021; Yue et al., 2021) | ✓ |  | ✓ |  | Regularization
(Mansour et al., 2020; Ghosh et al., 2020; Sattler et al., 2020; Briggs et al., 2020) |  | ✓ |  | ✓ | Cluster
(Mansour et al., 2020; Hanzely and Richtárik, 2021; Deng et al., 2020; Zhang et al., 2020d) |  | ✓ |  | ✓ | Model interpolation
(Smith et al., 2017; Agarwal et al., 2020; Liang et al., 2020; Dinh et al., 2021) |  | ✓ |  | ✓ | Multi-task learning
(Liang et al., 2020; Chen and Chao, 2021) |  | ✓ |  | ✓ | Parameter decoupling
(Liu et al., 2020; Li and Wang, 2019) |  | ✓ |  | ✓ | Transfer learning
(Li et al., 2020; Shoham et al., 2019; Yao and Sun, 2020) |  | ✓ |  | ✓ | Regularization
(Fallah et al., 2020; Khodak et al., 2019; Jiang et al., 2019; Singhal et al., 2021) |  | ✓ |  | ✓ | Meta-learning

####  4.2.3. Discussion

In addition to algorithmic fairness, client fairness is another concern in the
FL community. Table 7 enumerated various works on these two topics. Regarding
client fairness, the single model approach focuses on smoothing data
heterogeneity, where it is easy to implement and can be added to the general
FL paradigm since it only needs modest modification. On the downside, the
single model approach is less effective than personalized approaches in terms
of capturing local data distribution and may be insufficient when the data
distributions vary significantly between clients. Additionally, the single-
model approach does not allow clients to customize their models.

###  4.3. Discussion of fairness in FL

There are two definitions of fairness in FL, client fairness and algorithmic
fairness. Algorithmic fairness has been extensively studied in centralized
machine learning. These algorithms presuppose centralized access to data,
however, one virtue of FL is data never leaves the device. This means neither
the server nor any client gains centralized access to the training data.
Therefore, generalizing the fair learning algorithms to FL is not trivial. On
the one hand, data is stored locally in FL. The server cannot directly access
the local data of clients. Hence, server-side debiasing is not a viable
solution. On the other hand, debiasing on the client side is ineffective due
to the inadequate data, which can hardly represent the global data
distribution (McMahan et al., 2017). There is no guarantee that model debiased
with local data will generalize to the global distribution. The non-i.i.d data
distributions further complicated this problem (Kairouz et al., 2019).

Client fairness is tailored to FL and stems from the non-i.i.d data. Each
client sampled the training data from a distinct distribution. In this case,
the vanilla FL protocol, FedAvg, fails to train a model to fits clients’ data
distribution. Various methods have been proposed to alleviate this. From the
data aspect, (Zhao et al., 2018a; Jeong et al., 2018; Hao et al., 2021)
proposed to augment client data to yield an i.i.d dataset. (Wang et al.,
2020c; Yang et al., 2020b; Duan et al., 2021) proposed to select participant
clients to form a more homogeneous distribution. However, their methods did
not consider the possible algorithmic fairness issues and may introduce bias
to the model by choosing specific clients at a higher probability than others.
From the model perspective, training different models for different clients
seems a natural solution to the non-i.i.d. challenge. The core idea is to
train global data collaboratively and then personalize it to local data
distribution.

##  5\. Interactions between Privacy and Fairness

As shown in Fig 3, privacy and fairness are intertwined. On the one hand,
fairness comes at the cost of privacy. A fair model is trained to perform
equally on data from different groups, which incurs overfit problems and
consequently increases the privacy risk (Chang and Shokri, 2021). On the other
hand, privacy also harms fairness. Several works (Bagdasaryan et al., 2019;
Kuppam et al., 2020; Tran et al., 2021b; Sanyal et al., 2022) inconsistent
reductions in accuracy caused by private mechanisms on classification (Farrand
et al., 2020) and generative tasks (Ganev et al., 2022). Due to the tension
between fairness and privacy, researchers often need to make trade-offs
between these two notions. The trade-off could be increasing privacy
protection at the expense of fairness, i.e., adopting relaxed fairness notions
instead of exact notions or the opposite way. Table 8 gives various trade-offs
between privacy and fairness. On the basis of the adopted privacy/fairness
notions, the trade-offs can be divided into two categories. The first type
sacrifices fairness for solid privacy protection, i.e. $epsilon$-DP. The
second form prioritizes fairness over privacy and employs relaxed DP in order
to accommodate exact fairness.

Figure 3. Privacy, fairness, and accuracy trade-offs in deep learning: 1) privacy comes at the cost of accuracy; 2) fairness comes at the cost of accuracy; and 3) privacy interacts with fairness (Zhang et al., 2021) Table 8. Private and Fair Learning Reference |  Privacy notion |  Fairness notion | Techniques to achieve |  Trade-off type
---|---|---|---|---
Privacy | Fairness
(Cummings et al., 2019) |  $\epsilon$-DP |  $\alpha$-Discrimination |  | Exponential
---
mechanism
| Minimize
---
discrimination scores
I
(Xu et al., 2019b) |  $\epsilon$-DP |  | Decision boundary
---
fairness
| Functional
---
mechanism
Fairness constraints | I
(Mozannar et al., 2020) |  $\epsilon$-DP |  $\alpha$-Equal opportunity | Local DP |  | Post-processing
---
I
(Lamy et al., 2019) |  $\epsilon$-DP |  | Equal odds &
---
Demographic parity
| Class conditional
---
noise
Fairness constraints | I
(Ding et al., 2020) |  |  $\epsilon$-DP &
---
$(\epsilon,\delta)$-DP
| Decision boundary
---
fairness
| Functional
---
mechanism
Fairness constraints | I
(Jagielski et al., 2019) |  $(\epsilon,\delta)$-DP |  $\alpha$-Equal opportunity |  | Exponential
---
mechanism
& Laplace noise
Fairness constraints | /
(Lowy et al., 2023) |  $(\epsilon,\delta)$-DP |  | Equal odds &
---
Demographic parity
DP-SGDA | ERMI regularizer | II
(Esipova et al., 2022) |  $(\epsilon,\delta)$-DP | Excessive risk gap | DPSGD-Global-Adapt | Gradient correction | II
(Tran et al., 2021a) |  |  $(\alpha,\epsilon_{p})$-
---
Rényi DP
| Equal odds,
---
Accuracy parity
& Demographic parity
DP-SGD | Fairness constraints | II
(Kilbertus et al., 2018) | / | Equal accuracy | MPC | Fairness constraints | II
(Gupta et al., 2018) | / | Equal opportunity | Proxy attribute | Post-processing | II
(Wang et al., 2020b) | / | Demographic parity | Noisy attribute | Fairness constraints | II
(Awasthi et al., 2020) | / | Equal odds | Noisy attribute | Post-processing | II
I: Trade fairness for privacy. Relaxing fairness notions to achieve purely DP
II: Trade privacy for fairness. Adopting relaxed DP notion to accommodate
exact fairness

###  5.1. Privacy Degrades Fairness

Currently, there are two approaches to privately train a fair model: the
cryptography approach and the DP approach. In the first line of works, Veale
et al. (Veale and Binns, 2017) suggested storing protected characteristics in
a trusted third party to protect privacy. The third-party will perform
discrimination discovery and incorporate fairness constraints into model-
building. Kilbertus et al. (Kilbertus et al., 2018) relaxed the assumption of
a trusted third party and proposed to train a fair model with encrypted
sensitive attributes (Mozannar et al., 2020). The cryptography approach is
guaranteed to provide comparable performance as the non-private alternative at
the cost of communication overhead. The trade-offs between privacy and
efficiency are the main concern in this category.

However, as argued by (Jagielski et al., 2019), the cryptographic approach
does not guarantee privacy at the inference stage. It only ensures the
training data remain private during training and cannot prevent the adversary
from inferring training samples from the neural network parameters (Zhao et
al., 2020; Song et al., 2017). On the contrary, DP guarantees a fair model
will not leak anything beyond what could be carried out from ”population
level” correlations. As such, the majority of works focus on learning fair,
and DP model (Jagielski et al., 2019; Cummings et al., 2019; Bagdasaryan et
al., 2019; Xu et al., 2019b). Thus, this subsection focus on DP as the
privacy-preserving techniques.

####  5.1.1. Empirical Findings

The impact of privacy on fairness was initially observed in empirical studies.
Bagdasaryan et al. (Bagdasaryan et al., 2019) first observed that the
reduction in accuracy caused by deep DP models negatively impacts
underrepresented subgroups disproportionately. DP-SGD strengthens the model’s
”bias” toward the most prominent features of the distribution that is being
learned. Kuppam et al. (Kuppam et al., 2020) reached a similar conclusion when
they examined the effects of DP on fairness in three real-world tasks
involving sensitive public data. When the noise added by a private algorithm
is negligible in relation to the underlying statistics, the costs of adopting
a private technique may be minor. When stronger privacy is implemented or when
a task entails a small population, significant disparities may emerge. Farrand
et al. (Farrand et al., 2020) demonstrated that even minor differences and
weak privacy protections could result in disparate outcomes. Ganev et al.
(Ganev et al., 2022) shifted the emphasis to generative models and tabular
synthetic data. Three DP generative models PrivBayes (Zhang et al., 2017), DP-
WGAN (Alzantot and Srivastava, [n.d.]), and PATE-GAN (Jordon et al., 2019),
were involved. They witnessed a disparate effect on the accuracy of
classifiers trained on synthetic data generated by all generative models. The
losses are greater and/or more dispersed for underrepresented groups. Uniyal
et al. (Uniyal et al., 2021) compared DP-SGD and Private Aggregation of
Teacher Ensembles (PATE) (Papernot et al., 2016), an alternative DP mechanism
for discreetly training a deep neural network, in terms of fairness. They
discovered that PATE has a disparate effect, but it is considerably less
severe than DP-SGD.

####  5.1.2. Theoretical Explanations

Several works have attempted to determine the mechanism underlying the well-
known relationship between privacy and unfairness. Bagdasaryan et al.
(Bagdasaryan et al., 2019) associated the impact with the gradient clipping
operation in DP-SGD. During training, the model generates larger gradients for
samples from underrepresented subgroups; consequently, clipping slows their
learning rate. Therefore, the model learns less from the underrepresented
subgroups, and its performance on those subgroups is negatively impacted more.
Tran et al. (Tran et al., 2021a) conducted an in-depth study into this
phenomenon with output perturbation (Chaudhuri et al., 2011) and DP-SGD as the
private mechanism. By measuring fairness with excessive risk gap, Tran et al.
proved that output perturbation mechanisms incur unfairness when the local
curvatures of the loss functions of different groups differ substantially. For
DP-SGD, Tran et al. found that the clipping bound, the norm of inputs, and the
group’s distance to the decision boundary collectively contributed to the
unfairness raised by DP-SGD. Esipova et al. (Esipova et al., 2022) examined
the same issue from the gradient perspective. They proved that the gradient
misalignment caused by DP-SGD is the main reason for unfairness. If the
clipping operation disproportionately and sufficiently increases the direction
error for group $a$ relative to group $b$, then group $a$ incurs larger
excessive risk due to gradient misalignment.

####  5.1.3. Mitigation Strategies

Diverse methods have been proposed to mitigate the effect of private
mechanisms on fairness. Xu et al. (Xu et al., 2021b) proposed DP-SGD-F, a
variant of DP-SGD that reduces the divergent impact on different populations.
By adaptively designating clipping bounds for each group, DP-SGD-F achieves a
level of privacy proportional to each group’s utility-privacy trade-off. For
the group whose clipping bias is greater (due to large gradients), a larger
clipping bound is adopted to mitigate for their greater privacy cost. Tran et
al. (Tran et al., 2021a) formulated a regularized optimization problem that
minimizes empirical loss while satisfying two additional constraints. The
first constraint equalizes the averaged non-private and private gradients,
while the second constraint penalizes the difference between the local
curvatures of distinct groups’ loss functions. Esipova et al. (Esipova et al.,
2022) modified DP-SGD and developed DP-SGD-Global-Adapt to preserve gradient
direction. It is assumed that a hyperparameter $Z$ is the upper bound for most
gradients. Gradients less than $Z$ are uniformly scaled, whereas gradients
greater than $Z$ are trimmed to $Z$.

###  5.2. Fairness Increases Privacy Risk

Fairness in turn presents challenges for privacy mechanisms. Chang and Shokri
(Chang and Shokri, 2021) observed an elevated privacy risk for underprivileged
subgroups in a fair model. In order to ensure fairness, the model must perform
equally well for all subgroups. However, limited data availability for
underprivileged subgroups can lead to overfitting of the training data for
unprivileged subgroups in a fair model, thereby increasing the privacy risk.
Previous works on fairness-aware machine learning often assume that the
sensitive features are reliable and accessible. This assumption unavoidably
introduces privacy risks. However, achieving precise notions of fairness, such
as demographic parity, becomes unattainable without access to sensitive
attributes, specifically the membership information of sensitive groups. To
address this issue, several techniques have been proposed to safeguard the
privacy of sensitive attributes during the training of fair models.

####  5.2.1. Training Fair Models with Noisy Representation

Researchers in this field train approximately fair models using noisy
sensitive attributes to protect privacy. Gupta et al. (Gupta et al., 2018)
substituted protected groups with proxy groups. To achieve fairness, the proxy
groups need to align with the true positive group and even overlap with the
ground-truth groups. Thus, the fairness guarantee comes at the cost of
privacy. Several studies have explored fairness with imperfect group
information. (Kallus et al., 2020; Wang et al., 2020b; Lamy et al., 2019;
Awasthi et al., 2020). Lamy et al. (Lamy et al., 2019) introduced a mutual
contaminated model to simulate a noisy distribution with corrupted attributes.
Under this framework, they demonstrated that the fairness constraint on the
clean distribution is equivalent to a scaled fairness constraint on the noisy
distribution. To protect the privacy of sensitive attributes, they added class
conditional noise to release the noisy dataset. Awasthi et al. (Awasthi et
al., 2020) addressed the challenging problem of training a fair model with
perturbed sensitive attribute values, where each attribute is independently
flipped to its complementary value with probability $\gamma$. They identified
conditions on the perturbation under which the classifier, denoted as
$\hat{Y}$, obtained by Hardt et al.’s method (Hardt et al., 2016), is fairer
than the vanilla classifier, denoted as $\tilde{Y}$, trained on accurate
attributes. They further provided a formal guarantee of effectiveness under
the necessary conditions. Wang et al. (Wang et al., 2020b) trained a fair
binary classifier based on a noisy label $\hat{G}\in\\{1,...,\hat{m}\\}$,
i.e., $\hat{G}$ could be ”country of residence” as a noisy representation of
the true group labels $G=$ ”language spoken at home”.

####  5.2.2. Training Fair Models with DP

Works in this area protect privacy by adding noise to the private
characteristic (Tran et al., 2021a). The trade-off between privacy and
fairness depends on the amount of noise added, with no noise and excessive
noise representing the two extremes. In the case of no noise, the model’s
performance remains unaffected but could lead to information breaches.
Conversely, high levels of noise are effective in preserving privacy but can
compromise the model’s utility. Tran et al. (Tran et al., 2021a) proposed a
constrained optimization problem to address both private and fair learning
tasks. Their framework ensures $(\alpha,\epsilon_{p})$-Rényi DP (Mironov,
2017) for the sensitive attributes by solving the constrained problem with DP-
SGD. Jagielski et al. (Jagielski et al., 2019) extended Agarwal et al.’s
approach (Agarwal et al., 2018) by incorporating privacy considerations. They
formulated a two-player zero-sum game, played between a ”learner” and an
”auditor,” to derive a fair classifier. Laplacian noise (Dwork et al., 2006)
and the exponential mechanism (McSherry and Talwar, 2007) were utilized
separately for the ”learner” and the ”auditor”. As a result, the learned model
satisfies $(\epsilon,\delta)$-DP and achieves equalized odds.

###  5.3. Fair and Private FL

In centralized machine learning, one entails centralized access to training
data (either the true data or noisy data). However, this is invalid in FL,
where neither the server nor clients have access to others’ data. Therefore,
one cannot simply apply centralized fair learning algorithms in FL tasks. This
raises a question: How can we promote algorithmic fairness in FL without
accessing clients’ data in FL? Several studies made progress in response to
this challenge.

Using a surrogate model to preserve privacy. Padala et al. (Padala et al.,
2021) tried to satisfy both $(\epsilon,\delta)$-local DP and demographic
fairness through a fair and private FL framework. To circumvent the access
restriction, they decomposed the learning into two phases. First, each client
learns a fair and accurate model on a local dataset, where the fairness
constraint acts as a regularization term in the loss function. Then, every
client trains a surrogate model to match the fair predictions from the first
model with a DP guarantee. Finally, only the surrogate model is communicated
to the server.

Privacy through secure aggregation. Zhang et al. (Zhang et al., 2020b)
investigated classification problems in FL through multiple goal optimization
problems with privacy constraints. The objective is to minimize the accuracy
loss and the discrimination risk. To this end, a team Markov game was designed
to select participating clients at each communication round. In each round,
clients decide whether or not to participate based on the global model’s
state, which is characterized by bias level and accuracy. Further, a secure
aggregation protocol is designed to estimate the global model’s status based
on polynomial interpolation (Karnin et al., 1983) for privacy concerns. Under
this protocol, the server is able to calculate the discrimination status
without accessing the local data.

Achieve fairness based on statistics. Gálvez et al. (G’alvez et al., 2021)
formulated a constrained optimization problem that is solved by the
differential multiplier. Local statistics are provided to the server for
debiasing the global model. To further protect privacy, client updates are
clipped and perturbed by Gaussian noise before being sent to the server.
Finally, their solution is able to provide the approximate group fairness
notion over multiple attributes and $(\epsilon,\delta)$-DP.

Fairness through agnostic learning. Shifts in distribution is one source of
bias in FL. The global model is trained on the data of all clients (source
distribution), but each client’s local data distribution (target distribution)
may differ. When deployed to the client, unfavorable outcomes occur. Du et al.
(Du et al., 2021) proposed treating the client data distribution in an
agnostic way. An adversary generates any possible unknown local data
distribution to maximize the loss, while the learner aims to optimize the
accuracy and fairness.

Calculate fairness violations locally. Chu et al. (Chu et al., 2021)
formulated a constraint optimization problem to learn a fair and private model
in FL. Each client locally calculates fairness violations to avoid impinging
on the data privacy of any client. Chu et al. (Chu et al., 2021) further
optimized this method by aggregating fairness constraints to better estimate
the true fairness violation for all data.

Although some fair FL algorithms do not directly access the training data (Du
et al., 2021; Chu et al., 2021), faithfully sharing the model/gradients in FL
could incur privacy leakage risks. The privacy breach could happen during the
training or inference stage. The attack could be carried out by either the
server or the clients (Yin et al., 2021b). For example, an honest-but-curious
server can lunch a reconstruction attack (Zhu et al., 2019) to recover the
private data from the gradients uploaded by the victim client. However, the
main challenge to training a fair model in FL is restricted data access, e.g.,
data never leaving local devices, which is an under-investigated topic in FL
literature. In the case of the adversary clients/server in FL, some privacy-
preserving techniques, such as DP, can be combined with the aforementioned
fair FL approaches to prevent privacy leakage.

###  5.4. Discussion of Privacy and Fairness Interactions

The complex interactions between privacy and fairness have been thoroughly
examined and documented in various studies. These investigations highlight the
intricate trade-offs and challenges that arise when attempting to
simultaneously address both privacy and fairness objectives (Cummings et al.,
2019).

The impact of privacy and fairness on each other is indeed bilateral. In one
scenario, privacy measures can degrade fairness. For instance, in widely-used
privacy mechanisms like DP-SGD, to protect privacy, the algorithm clips and
adds noise to the gradients. However, due to the scarcity of data for certain
groups, these modifications can disproportionately affect underrepresented
groups, exacerbating unfairness. Therefore, the implementation of DP can
inadvertently worsen existing unfairness by disproportionately impacting
certain groups.

In another case, fairness can increase privacy risks. To achieve fairness, it
may be necessary to collect additional demographic information about users,
even if it is irrelevant to the task at hand. This data collection is aimed at
guiding modifications to the model, such as addressing inconsistent responses
or removing discrimination in statistical models (Voigt and Von dem Bussche,
2017; Goodman, 2016; Žliobaitė and Custers, 2016). However, the collection of
such sensitive information raises privacy concerns, as it expands the scope of
data being collected and potentially increases the risk of privacy breaches.

In the context of Federated Learning (FL), the cooperative game between
clients and the server adds complexity to the privacy and fairness challenges.
FL introduces new privacy attack surfaces, as discussed in Section 3, where
potential malicious participants can actively or passively infer the private
data of other clients. Consequently, securing private information in FL
requires even stronger privacy protection measures compared to the centralized
setting. Merely protecting group membership is insufficient to address the
privacy risks in FL. Furthermore, the non-i.i.d. (non-independent and
identically distributed) nature of FL poses another challenge. In a typical FL
system, clients’ data are sampled from different distributions, leading to
data heterogeneity. A model that achieves fairness within the local
distribution of each client is not guaranteed to perform unbiasedly on a
global scale. The non-i.i.d. issue also introduces potential fairness concerns
at the client level, as the performance of the model can vary significantly
among clients. It is crucial to address this variation and ensure fairness
across all participating clients in FL. The challenge lies in training a fair
model in FL without violating the data access restrictions imposed by each
client. Finding methods to mitigate the fairness issues arising from the
non-i.i.d. nature of the data while respecting the privacy and data access
constraints in FL remains a challenging task.

##  6\. Open Research Directions

The research community has made fruitful progress in privacy and fairness in
FL. However, throughout this survey, we found this field still faces several
challenges that need to be solved.

  * •

Trade-offs between Privacy and Fairness. The interaction between privacy and
fairness is an under-studied topic. Existing works have focused on exploring
the two notions in isolation, either focused on privacy-preserving machine
learning (Xu et al., 2021a) or on paradigms that respect fairness (Shi et al.,
2021). However, as demonstrated by several studies (Chang and Shokri, 2021;
Bagdasaryan et al., 2019; Kuppam et al., 2020), privacy and fairness may
compete with each other. In the realm of FL, challenges and opportunities
coexist. On the one hand, restricted information and non-i.i.d distribution
complicate the problem settings. On the other hand, the flexibility of the FL
paradigm may enable more possible solutions. For instance, the personalized
model (Kulkarni et al., 2020; Tan et al., 2021; Zhu et al., 2021) has been
widely used in FL to address statistical challenges by assigning clients
personalized models. We may combine privacy and personalized models to achieve
a better trade-off between privacy, fairness, and utility. Thus, we believe it
is worth examining the trade-offs between privacy and fairness in FL.

  * •

The Compatibility of Fairness and DP. We believe it would be worth
investigating techniques that simultaneously accommodate fairness and DP. As
pointed out in Dwork et al.’s (Dwork et al., 2012) work, given a carefully
designed distance metric, it is possible to achieve individual fairness
through $\epsilon$-DP. Two characteristics of FL make individual fairness a
superior choice over group fairness: 1) Data distribution in FL may vary
significantly between clients, and individual fairness is more suitable in
such cases. Since it is defined at the sample level, thus, it generates better
than group notions when addressing new samples which may be distinct from
those in the training set; 2) The restricted access to information in FL lends
itself more to individual fairness because individual fairness relies on a
Lipschitz continual prediction model and does not require access to
demographic data. This perfectly fits the FL setting.

  * •

How can one satisfy fairness at both the algorithm and client levels in FL?
The majority of studies on fairness in FL focus on promoting fairness at the
client level. However, client-level fairness does not necessarily imply
algorithmic fairness. Consider a scenario where multiple companies (clients)
collaborate to train a credit card approval model. Consumer demographic
compositions vary between each company. Although a federated model trained
subject to client-level fairness constraints might handle the different
companies fairly, the model could still be biased towards sensitive attributes
(such as race or educational background). This raises a question: How can one
satisfy fairness at both the algorithm and the client levels while preserving
privacy in FL?.

##  7\. Conclusion

In this article, we conducted a detailed survey of data privacy and model
fairness issues in FL. Uniquely, we also documented the interactions between
privacy and fairness from the perspective of trade-offs. In terms of privacy
in FL, we first reviewed privacy attacks in FL. Then, we presented three kinds
of privacy-preserving techniques. Regarding fairness, we first analyzed the
possible sources of bias and how bias can be introduced on both the client and
server sides. Following a review of the notions of fairness adopted in machine
learning and those originating from FL, a discussion of the various fairness-
aware FL algorithms is presented. The last part of the survey focused on the
interactions between privacy and fairness. We identified three relations in
the general context and further listed possible solutions to achieve both fair
and private FL.

###### Acknowledgements.

This paper is supported by the Australian Research Council Discovery
DP200100946 and DP230100246, and NSF under grants III-1763325,
III-1909323,III-2106758, and SaTC-1930941.

## References

  * (1)
  * Abay et al. (2020) Annie Abay, Yi Zhou, Nathalie Baracaldo, Shashank Rajamoni, Ebube Chuba, and Heiko Ludwig. 2020\.  Mitigating bias in federated learning.  _arXiv preprint arXiv:2012.02447_ (2020).
  * Agarwal et al. (2018) Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018\.  A reductions approach to fair classification. In _International Conference on Machine Learning_. PMLR, 60–69.
  * Agarwal et al. (2020) Alekh Agarwal, John Langford, and Chen-Yu Wei. 2020\.  Federated residual learning.  _arXiv preprint arXiv:2003.12880_ (2020).
  * Alzantot and Srivastava ([n.d.]) Moustafa Alzantot and Mani Srivastava. [n.d.].  Differential Privacy Synthetic Data Generation using WGANs, 2019.  _URL https://github. com/nesl/nist_differential_privacy_ synthetic_data_challenge_ ([n. d.]).
  * Aono et al. (2017) Yoshinori Aono, Takuya Hayashi, Lihua Wang, Shiho Moriai, et al. 2017\.  Privacy-preserving deep learning: Revisited and enhanced. In _International Conference on Applications and Techniques in Information Security_. Springer, 100–110.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017\.  Wasserstein generative adversarial networks. In _International conference on machine learning_. PMLR, 214–223.
  * Ateniese et al. (2013) Giuseppe Ateniese, Giovanni Felici, Luigi V. Mancini, Angelo Spognardi, Antonio Villani, and Domenico Vitali. 2013.  Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers.  https://doi.org/10.48550/ARXIV.1306.4447
  * Awasthi et al. (2020) Pranjal Awasthi, Matthäus Kleindessner, and Jamie Morgenstern. 2020.  Equalized odds postprocessing under imperfect group information. In _International conference on artificial intelligence and statistics_. PMLR, 1770–1780.
  * Bagdasaryan et al. (2019) Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. 2019\.  Differential privacy has disparate impact on model accuracy.  _Advances in Neural Information Processing Systems_ 32 (2019), 15479–15488.
  * Balunović et al. (2021) Mislav Balunović, Dimitar I Dimitrov, Robin Staab, and Martin Vechev. 2021.  Bayesian framework for gradient leakage.  _arXiv preprint arXiv:2111.04706_ (2021).
  * Berk et al. (2017) Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017\.  A convex framework for fair regression.  _arXiv preprint arXiv:1706.02409_ (2017).
  * Berk et al. (2021a) Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021a.  Fairness in Criminal Justice Risk Assessments: The State of the Art.  _Sociological Methods & Research_ 50, 1 (2021), 3–44.  https://doi.org/10.1177/0049124118782533
  * Berk et al. (2021b) Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021b.  Fairness in criminal justice risk assessments: The state of the art.  _Sociological Methods & Research_ 50, 1 (2021), 3–44.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ 29 (2016), 4349–4357.
  * Bonawitz et al. (2017) Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017.  Practical secure aggregation for privacy-preserving machine learning. In _proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_. 1175–1191.
  * Bonawitz et al. (2019) Keith Bonawitz, Fariborz Salehi, Jakub Konečnỳ, Brendan McMahan, and Marco Gruteser. 2019\.  Federated learning with autotuned communication-efficient secure aggregation. In _2019 53rd Asilomar Conference on Signals, Systems, and Computers_. IEEE, 1222–1226.
  * Bordia and Bowman (2019) Shikha Bordia and Samuel R Bowman. 2019.  Identifying and reducing gender bias in word-level language models.  _arXiv preprint arXiv:1904.03035_ (2019).
  * Briggs et al. (2020) Christopher Briggs, Zhong Fan, and Péter András. 2020\.  Federated learning with hierarchical clustering of local updates to improve training on non-IID data.  _2020 International Joint Conference on Neural Networks (IJCNN)_ (2020), 1–9.
  * Brunet et al. (2019) Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019\.  Understanding the origins of bias in word embeddings. In _International Conference on Machine Learning_. PMLR, 803–811.
  * Calders and Verwer (2010) Toon Calders and Sicco Verwer. 2010.  Three naive bayes approaches for discrimination-free classification.  _Data mining and knowledge discovery_ 21, 2 (2010), 277–292.
  * Canetti et al. (2019) Ran Canetti, Aloni Cohen, Nishanth Dikkala, Govind Ramnarayan, Sarah Scheffler, and Adam Smith. 2019\.  From Soft Classifiers to Hard Decisions: How fair can we be?  arXiv:1810.02003 [cs.LG]
  * Chang and Shokri (2021) Hongyan Chang and Reza Shokri. 2021.  On the privacy risks of algorithmic fairness. In _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 292–303.
  * Chaudhuri et al. (2011) Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. 2011\.  Differentially private empirical risk minimization.  _Journal of Machine Learning Research_ 12, 3 (2011).
  * Chen et al. (2022) Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. 2022.  pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning.  _arXiv preprint arXiv:2206.03655_ (2022).
  * Chen and Chao (2021) Hong-You Chen and Wei-Lun Chao. 2021.  On Bridging Generic and Personalized Federated Learning.  _arXiv preprint arXiv:2107.00778_ (2021).
  * Chen et al. (2020c) Junjie Chen, Wendy Hui Wang, and Xinghua Shi. 2020c.  Differential privacy protection against membership inference attack on machine learning for genomic data. In _BIOCOMPUTING 2021: Proceedings of the Pacific Symposium_. World Scientific, 26–37.
  * Chen et al. (2020a) Si Chen, Ruoxi Jia, and Guo-Jun Qi. 2020a.  Improved Techniques for Model Inversion Attacks.  _arXiv preprint arXiv:2010.04092_ (2020).
  * Chen et al. (2020b) Yu Chen, Fang Luo, Tong Li, Tao Xiang, Zheli Liu, and Jin Li. 2020b.  A training-integrity privacy-preserving federated learning scheme with trusted execution environment.  _Information Sciences_ 522 (2020), 69–79.
  * Cho et al. (2020) Yae Jee Cho, Jianyu Wang, and Gauri Joshi. 2020.  Client selection in federated learning: Convergence analysis and power-of-choice selection strategies.  _arXiv preprint arXiv:2010.01243_ (2020).
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5, 2 (2017), 153–163.
  * Chouldechova and Roth (2018) Alexandra Chouldechova and Aaron Roth. 2018.  The frontiers of fairness in machine learning.  _arXiv preprint arXiv:1810.08810_ (2018).
  * Chu et al. (2021) Lingyang Chu, Lanjun Wang, Yanjie Dong, Jian Pei, Zirui Zhou, and Yong Zhang. 2021\.  FedFair: Training Fair Models In Cross-Silo Federated Learning.  _ArXiv_ abs/2109.05662 (2021).
  * Cui et al. (2021) Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. 2021.  Addressing algorithmic disparity and performance inconsistency in federated learning.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Cummings et al. (2019) Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern. 2019.  On the compatibility of privacy and fairness. In _Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization_. 309–315.
  * d’Alessandro et al. (2017) Brian d’Alessandro, Cathy O’Neil, and Tom LaGatta. 2017\.  Conscientious classification: A data scientist’s guide to discrimination-aware classification.  _Big data_ 5, 2 (2017), 120–134.
  * Deng et al. (2020) Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. 2020.  Adaptive personalized federated learning.  _arXiv preprint arXiv:2003.13461_ (2020).
  * Ding et al. (2020) Jiahao Ding, Xinyue Zhang, Xiaohuan Li, Junyi Wang, Rong Yu, and Miao Pan. 2020\.  Differentially private and fair classification via calibrated functional mechanism. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 622–629.
  * Dinh et al. (2021) Canh T Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. 2021.  FedU: A Unified Framework for Federated Multi-Task Learning with Laplacian Regularization.  _arXiv preprint arXiv:2102.07148_ (2021).
  * Dinur and Nissim (2003) Irit Dinur and Kobbi Nissim. 2003.  Revealing information while preserving privacy. In _Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems_. 202–210.
  * Du et al. (2021) Wei Du, Depeng Xu, Xintao Wu, and Hanghang Tong. 2021\.  Fairness-aware Agnostic Federated Learning. In _Proceedings of the 2021 SIAM International Conference on Data Mining (SDM)_. SIAM, 181–189.
  * Duan et al. (2021) M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang. 2021.  Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems.  _IEEE Transactions on Parallel & Distributed Systems_ 32, 01 (2021), 59–71.  https://doi.org/10.1109/TPDS.2020.3009406
  * Dwork (2008) Cynthia Dwork. 2008\.  Differential privacy: A survey of results. In _International conference on theory and applications of models of computation_. Springer, 1–19.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through awareness. In _Proceedings of the 3rd innovations in theoretical computer science conference_. 214–226.
  * Dwork et al. (2006) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006.  Calibrating Noise to Sensitivity in Private Data Analysis. In _Theory of Cryptography_ , Shai Halevi and Tal Rabin (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 265–284.
  * Esipova et al. (2022) Maria S Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse C Cresswell. 2022.  Disparate Impact in Differential Privacy from Gradient Misalignment.  _arXiv preprint arXiv:2206.07737_ (2022).
  * Ezzeldin et al. (2021) Yahya H Ezzeldin, Shen Yan, Chaoyang He, Emilio Ferrara, and Salman Avestimehr. 2021.  Fairfed: Enabling group fairness in federated learning.  _arXiv preprint arXiv:2110.00857_ (2021).
  * Fallah et al. (2020) Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020\.  Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. In _Advances in Neural Information Processing Systems_ , Vol. 33. 3557–3568.
  * Fan et al. (2020) Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, Chang Liu, Chee Seng Chan, and Qiang Yang. 2020.  Rethinking privacy preserving deep learning: How to evaluate and thwart privacy attacks.  In _Federated Learning_. Springer, 32–50.
  * Farrand et al. (2020) Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. 2020.  Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy. In _Proceedings of the 2020 workshop on privacy-preserving machine learning in practice_. 15–19.
  * Feldman et al. (2015) Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015\.  Certifying and removing disparate impact. In _proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_. 259–268.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks. In _International Conference on Machine Learning_. PMLR, 1126–1135.
  * Font and Costa-jussà (2019) Joel Escudé Font and Marta R Costa-jussà. 2019.  Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques. In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_. 147–154.
  * Fowl et al. (2021) Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. 2021.  Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models.  _arXiv preprint arXiv:2110.13057_ (2021).
  * Fredrikson et al. (2015) Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015\.  Model inversion attacks that exploit confidence information and basic countermeasures. In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_. 1322–1333.
  * Fredrikson et al. (2014) Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. 2014\.  Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing. In _23rd USENIX Security Symposium (USENIX Security 14)_. USENIX Association, San Diego, CA, 17–32.  https://www.usenix.org/conference/usenixsecurity14/technical-sessions/presentation/fredrikson_matthew
  * G’alvez et al. (2021) Borja Rodr’iguez G’alvez, Filip Granqvist, Rogier C. van Dalen, and Matthew Stephen Seigel. 2021\.  Enforcing fairness in private federated learning via the modified method of differential multipliers.  _ArXiv_ abs/2109.08604 (2021).