# Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation
Metrics

Nicolò Romandini , Alessio Mora , Carlo Mazzocca , Rebecca Montanari , Paolo
Bellavista Nicolò Romandini, Alessio Mora, Carlo Mazzocca, Rebecca Montanari,
and Paolo Bellavista are with the Department of Computer Science and
Engineering (DISI), University of Bologna, Bologna, Italy (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>paolo.bellavista@unibo.it). 0000-0002-2820-5978
0000-0001-8161-1070 0000-0001-8949-2221 0000-0002-3687-0361
0000-0003-0992-7948

###### Abstract

Federated Learning (FL) enables collaborative training of a Machine Learning
(ML) model across multiple parties, facilitating the preservation of users’
and institutions’ privacy by keeping data stored locally. Instead of
centralizing raw data, FL exchanges locally refined model parameters to build
a global model incrementally. While FL is more compliant with emerging
regulations such as the European General Data Protection Regulation (GDPR),
ensuring the right to be forgotten in this context — allowing FL participants
to remove their data contributions from the learned model — remains unclear.
In addition, it is recognized that malicious clients may inject backdoors into
the global model through updates, e.g. to generate mispredictions on specially
crafted data examples.

Consequently, there is the need for mechanisms that can guarantee individuals
the possibility to remove their data and erase malicious contributions even
after aggregation, without compromising the already acquired good knowledge.
This highlights the necessity for novel Federated Unlearning (FU) algorithms,
which can efficiently remove specific clients’ contributions without full
model retraining. This survey provides background concepts, empirical
evidence, and practical guidelines to design/implement efficient FU schemes.
Our study includes a detailed analysis of the metrics for evaluating
unlearning in FL and presents an in-depth literature review categorizing
state-of-the-art FU contributions under a novel taxonomy. Finally, we outline
the most relevant and still open technical challenges, by identifying the most
promising research directions in the field.

###### Index Terms:

Federated Learning, Federated Unlearning, Machine Unlearning, Privacy, Right
to be Forgotten, Unlearning Metrics

##  I Introduction

To achieve high-performance levels suitable for real-world applications,
Machine Learning (ML) and, in particular, Deep Learning (DL) models need to
access large amounts of data. In numerous scenarios, users willingly furnish
their data to service providers in return for enhanced service quality and
improved user experience. Practical examples include the training of natural
language processing models on datasets derived from Amazon reviews [1] or the
development of micro-video recommendation systems leveraging user data from
platforms such as TikTok and Instagram [2]. Data analytics hold the promise of
delivering unprecedented value to companies and organizations, e.g., as
pointed out by its key role in influencing election campaigns in the so-called
Facebook-Cambridge Analytica scandal [3].

Therefore, the growing outsourcing of user information has also led to an
increasing interest in privacy preservation measures. In this direction,
recent years have witnessed a worldwide development and establishment of
privacy-protection regulations, aiming to address the growing sensibility of
individuals to the treatment of their personal information by third parties.
The European Union (EU) and California state have been among the pioneers in
proposing such regulations, respectively with the General Data Protection
Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Among the
enforced guidelines about the collection, storage, and processing of personal
data, both GDPR and CCPA include the on-demand removal of personal data that
were previously voluntarily disclosed by the users. For example, the right of
erasure, also known as the right to be forgotten, is one of the cardinal
articles in the EU GDPR [4] and states that the data subject shall have the
right to obtain from the controller the erasure of personal data concerning
him or her without undue delay […] where the data subject withdraws consent on
which the processing is based […].. Similarly, CCPA [5] includes the right to
delete, which empowers business users with the right to request that
businesses delete the personal information they collected from them and tell
their service providers to do the same.

In the context of ML, the right to have data removed can be realized through
Machine Unlearning (MU) [6]. Data owners may want to withdraw their
contributions to a trained model due to privacy or security concerns [7]. MU
involves the post-processing of a trained model, by selectively eliminating
the influence of specific training samples. At first glance, the intuitive
approach to unlearn targeted data consists of retraining the model from
scratch utilizing the remaining data. However, this seemingly straightforward
method poses significant technical challenges in terms of load and
performance: retraining from scratch is not only computationally expensive but
also becomes impractical when faced with frequent unlearning requests.

Furthermore, the emergence of increasingly privacy-conscious data owners and
the adoption of data regulations have influenced the perception of traditional
centralized ML/DL, where the training corpus frequently contains user data,
e.g., habits or preferences. In this context, Federated Learning (FL) is
considered a promising solution, laying the foundation for privacy-preserving
ML. In FL, data remains distributed among various sources or data owners, and
a global model is iteratively built by aggregating the contributions of
participants (often referred to as clients), e.g., model updates or gradients.
Consequently, clients exchange ephemeral information, locally computed on
their private data and only valid for the current state of the global ML
model.

Nevertheless, even under the FL paradigm, how to guarantee the right to be
forgotten is not clear. Indeed, the collaboratively learned model inevitably
embeds the knowledge extracted from individual user data, potentially posing a
risk of leaking sensitive information [8] and vulnerability to membership
attacks [9], i.e., the ability to infer whether a certain data sample was part
of the training set. These considerations raise the following question: how
should the FL system react when a client exercises its right to be forgotten
at runtime?

As for MU, the most trivial solution to erase the contribution from a specific
set of data examples (e.g., the client’s private data) is to exclude them from
the training set in the first place. However, retraining a model from scratch
with a sanitized dataset in a federated setting is often unfeasible. This
could be due to the unavailability of clients that have already participated
or the excessive cost in terms of required time, computation, and energy.
Therefore, there is the need for effective Federated Unlearning (FU) methods,
directly applicable to already trained or in-learning models, without
discarding the good knowledge acquired after the unlearning point.

TABLE I: Comparison of surveys in the field.

Year Reference Objective Formalization Original Taxonomy Comprehensive
Literature Review Experimental Insights Comprehensive Metrics Analysis In-
depth Security & Privacy Challenges & Future Directions 2023 Yang and Zhao
[10] ✓ ✓ - - - - ✓ 2023 Wang et al. [11] - ✓ - - - ✓ - 2023 Liu et al. [12] -
✓ ✓ - - - ✓ Now Ours ✓ ✓ ✓ ✓ ✓ - ✓

###  I-A Related Surveys

The interest in FU has been rapidly increasing. This trend is supported by the
numerous research papers that have recently introduced novel FU algorithms.
Most published works primarily focus on MU [6, 10, 13, 14], while FU is only
covered from a general perspective with a limited amount of references.

By delving into distinct aspects of FU, a few other surveys have been proposed
very recently. Wang et al. [11] review some studies on FU by mainly
concentrating on the security and privacy dimensions of FU and by paying
special attention to membership inference attacks and corresponding
countermeasures. Liu et al. [12] present a comprehensive survey that
highlights challenges, methods, and promising research directions in the
field. The authors classify FU algorithms based on who initiates the
unlearning and what needs to be forgotten. Additionally, their survey
discusses optimization strategies for general FL. Similarly, Yang et al. [10]
direct their attention toward challenges and future research directions, but
by adopting a high-level perspective and without providing in-depth technical
details. The authors categorize works according to the objectives of FU, with
limited practical and implementation insights about the surveyed solutions. To
provide a clear and concise comparison, Table I compares this paper with the
other few surveys already conducted in the general field of MU.

###  I-B Contributions

This paper offers a comprehensive overview of FU tailored to both technical
and non-technical audiences. In contrast to other surveys, our work goes
beyond theoretical discussions, by providing readers also with practical
design/implementation insights that contribute to clarifying all facets of FU.

Specifically, to be highly accessible, we first present the fundamental
concepts and principles of FU, encompassing objectives, motivations, and
challenges. Then, we describe the primary emerging guidelines for the design
of efficient FU algorithms, by including a comprehensive analysis of the
metrics to assess unlearning - a critical concern within federated scenarios.
Notably, we complement our reported analysis with a series of experiments that
offer tangible insights into the necessity of FU. Moreover, we conduct an
extensive review of the existing FU literature, categorizing papers through a
novel taxonomy based on their objectives and the metrics employed. Finally, we
identify open problems and discuss potential future research directions. We
believe that this survey serves as a valuable resource for researchers seeking
a clear understanding of FU and the recent related research advancements and
as a practical guide for efficiently designing and implementing novel FU
solutions. The paper contributions can be summarized as follows:

  * •

We introduce the background concepts and the motivations for FU algorithms. We
formalize the setting and possible objectives of FU methods, and what
differentiates them from classical MU.

  * •

We perform a set of experiments to empirically demonstrate that an FL global
model memorizes contributions of individual clients’ local data. Our findings
show that FL global models perform significantly better on seen client data
and that such knowledge fades away very slowly even if the specific clients do
not participate in the next rounds.

  * •

We provide comprehensive guidelines for the efficient design and
implementation of FU algorithms. This includes the requirements that the FU
algorithms must meet and a detailed analysis of the metrics adopted for
evaluation.

  * •

We conduct a thorough review of the existing FU literature, by presenting the
most relevant and updated contributions to the field, with a technical focus
on the rationale behind their design choices. For ease of understanding, we
classify the considered work by using a novel taxonomy presented in Section V.

  * •

We identify and discuss the most relevant and open technical challenges in the
field. This also outlines the most promising future research directions, by
providing a roadmap for ongoing and upcoming investigations.

###  I-C Organization

The remainder of this paper is organized as follows. Section II introduces the
needed background about MU and FL. In Section III, we formalize the objectives
of FU, we provide qualitative experimental evidence that motivates the
research in the field, and we clarify the technical motivations why FU is more
challenging than regular MU. In Section IV, we offer a detailed analysis of
the requirements that FU algorithms have to meet and the associated assessment
metrics. Furthermore, Section V presents an in-depth exploration of all the
surveyed solutions, by highlighting the rationale behind their algorithm
design and implementation choices. Section VI and VII respectively summarize
the key lessons learned and the most promising directions for future work in
FU, while Section VIII concludes the manuscript.

TABLE II: Notation. Symbol | Description
---|---
$u$ | Target client that requests the unlearning process
$k$ | A client in the federation
$K$ | Amount of clients selected per round
$n$ | Total amount of clients’ data examples
$c$ | A specific class of the classification task
$n_{k}$ | Amount of client $k$’s data samples
$n^{c}_{k}$ | Amount of client $k$’s data samples labeled with $c$
$t$ | A round of FL training
$D$ | A general dataset
$x$ | Data sample
$y$ | Data sample’s label
$L$ | Number of total classes in classification task
$D_{u}$ | Client $u$’s private dataset
$S_{u}$ | Subset of client $u$’s private dataset
$w$ | The weights or parameters of an ML/DL model
$w_{t}$ | Global model at a certain round $t$
$w^{u}_{t}$ | Unlearned global model at round $t$
$w^{r}_{t}$ | Retrained global model at round $t$
$\mathcal{U}(\cdot)$ | An unlearning function or algorithm
$C$ | Set of training data belonging to the same class $c$
$g(\cdot)$ | A readout function
$|\cdot|$ | Cardinality of a dataset
$T$ | A time measurement

Figure 1: Illustrative organization of the paper.

##  II Background

###  II-A Machine Unlearning

Considering a set of model weights $w$ trained on a dataset $D$, the purpose
of MU is to remove the influence of a specific $D$’s subset on $w$.

Let $D_{u}$ be the subset of $D$ to be forgotten, and let $D_{r}$ be its
complement, i.e. $D_{u}$:$=D_{r}^{\complement}$ $=$ $D/D_{r}$. We will refer
to $D_{u}$ as the forgetting dataset and to $D_{r}$ as the retain dataset. Let
$w_{r}$ be a set of model weights trained only on $D_{r}$, usually referred to
as the retrained model. The goal of an MU algorithm is to efficiently obtain
$w_{u}$, a sanitized version of $w$, which is indistinguishable or
approximately indistinguishable from $w_{r}$ [15, 6]. Two sets of model
weights are indistinguishable if an attacker with access to $w_{u}$ cannot
design a function $g(\cdot)$ to extract information specific to $D_{u}$.
Considering, for example, a Kullback-Leibler (KL) divergence as a measure for
similarity among distributions (distributions of weights in this case), the
unlearned model weights $w_{u}$ and the retrained model weights $w_{r}$ are
formally indistinguishable if their KL divergence is zero [15]:

$KL(g(w_{u}),g(w_{r}))=0$ (1)

with $w_{u}=\mathcal{U}(w)$ and $\mathcal{U}(\cdot)$ representing an
unlearning algorithm. Similarly, the unlearned model weights $w_{u}$ and the
retrained model weights $w_{r}$ are approximately (or
$\epsilon$-indistinguishable) if their KL divergence is lower than a small
threshold value $\epsilon$,

$KL(g(w_{u}),g(w_{r}))<\epsilon.$ (2)

As anticipated, the naive solution to unlearn the contribution of $D_{u}$
would consist of retraining the model from scratch only on $D_{r}$. However,
this method can be costly in terms of time, computation, and energy
consumption, especially when thinking of large models. Hence, MU mechanisms
should be more efficient than the retraining strategy. Furthermore, in some
scenarios, the entire corpus of $D_{r}$ may not be directly accessible or may
not be available anymore after the training of the original model, i.e., as we
will see in the following sections, for FL settings.

It is worth noting that MU mechanisms are not only needed to meet regulations
about individuals’ privacy, such as GDPR and CCPA, but also represent a
security tool for the provider of the ML service. Training data may include
natural or malicious outliers, which can be detrimental to the model
performances or can introduce potential security threats (e.g., model
backdoor). Under this perspective, MU algorithms represent a mechanism to
selectively remove the influence of such a subset of threatening data without
discarding the useful knowledge already acquired.

###  II-B Federated Learning

Considering $K$ learners (also referred to as clients) participating in the
collaborative training, with each client $k$ possessing a local dataset
$D_{k}$, FL aims to minimize a global objective function, $f(w)$, that can be
expressed as follows:

$\min_{w}f(w):=\sum_{k=1}^{K}\frac{n_{k}}{n}F_{k}(w),$ (3)

where $f(w)$ is a weighted average of local objective functions $F_{k}$, such
as cross-entropy loss for a supervised classification task, across all the
datasets of clients. $w$ represents the parameters of the global model,
$n_{k}$ represents the number of training examples contained in local datasets
$D_{k}=\\{x_{i},y_{i}\\}^{n_{k}}_{i}$ with $x$ and $y$ being the data point
and the related label. $n$ is the total number of data points in the
federation of clients.

Federated Averaging [16], also known as FedAvg, serves as the foundational
algorithm for FL and can be described as a synchronous protocol that adopts a
client-server paradigm [17]. FedAvg proceeds in rounds and heuristically tries
to solve the optimization problem in Eq. 3. During each round, the server
distributes the current global model to a random selection of available
clients among the entire pool of learners, which fine-tune the received model
parameters on their local data. Then, the activated clients communicate a
model update to the central server (i.e., the difference between the locally
fine-tuned and the received model parameters). Once the server has gathered
such model updates, a weighted-average aggregation is performed, and the
resultant averaged update is applied to the current global model. The initial
formulation of FedAvg applies the resultant aggregated update by just summing
it to the server model. However, aggregated updates can be seen as pseudo-
gradients, and can be applied by using an optimizer of choice, i.e., different
from regular Stochastic Gradient Descent (SGD). Under this perspective,
vanilla FedAvg employs SGD with a learning rate set to 1 as server-side
optimizer [18].

While FedAvg is considered the baseline for FL, peer-to-peer and/or
asynchronous variants exist. In peer-to-peer FL (or fully decentralized),
clients directly communicate with each other without the need for a central
coordinator, while in asynchronous FL the server can aggregate the
contributions of clients as soon as it receives them (without waiting for the
slower learners) [17].

It is worth noting that, while sharing model updates in place of raw data
makes it harder for attackers to extract information about clients’ training
data, FL does not ensure complete privacy protection. Indeed, it has been
proved that disclosing model updates or gradients can expose clients to
information leakage (e.g., [8, 19, 20]).

###  II-C Catastrophic Forgetting and Differential Privacy

In this subsection, we introduce two concepts, i.e. catastrophic forgetting
and Differential Privacy, that are similar to unlearning algorithms in their
effects while being substantially different in principle.

Catastrophic forgetting. In the field of Continual Learning (CL) [21], a key
challenge is catastrophic forgetting. This occurs when a neural network, while
adapting to new tasks, tends to disrupt the representations learned to solve
the previous downstream tasks [22]. Recent studies have demonstrated that
catastrophic forgetting also happens in FL training due to the round-by-round
shift of data distributions. The pools of selected participants can change at
each round, potentially including clients with skewed data distribution with
respect to the global data distribution. This can lead the global model to
specialize mainly on the specific per-round data distribution of activated
clients while struggling to generalize overall [23, 24, 25].

Catastrophic forgetting may seem a natural method for unlearning, i.e., the
global model forgets the specific contributions of a client’s data as soon as
that client is not included in FL rounds anymore. However, it has been
demonstrated [26] (see also Subsection III-B) that even though a participant
leaves the training, the global model still retains contributions learned from
its data. Consequently, relying on catastrophic forgetting cannot be
considered a reliable strategy. Moreover, when unlearning methods are applied
to an ML model, there is the risk that catastrophic forgetting is induced by
the unlearning, i.e. the unlearning algorithm also removes to-retain knowledge
as a side effect.

Differential Privacy. Within the context of ML, it is of paramount importance
that the public release of model parameters does not leak too much sensitive
information about the original training data, even when the adversary can have
access to auxiliary information. Differential Privacy (DP) [27, 28] tries to
solve this apparently paradoxical requirement of extracting useful information
from a population while guaranteeing to learn nothing about an individual. To
this end, DP promises that the inclusion, or exclusion, of an individual’s
data in a private database does not essentially affect the outcome of any
analysis conducted on that database. Therefore, an individual should be almost
indifferent between disclosing her data and not111If the inclusion (or
exclusion) of an individual’s data did not affect the outcome at all, it would
mean that the individual’s data is useless.. In a nutshell, DP is a
mathematically rigorous formalization of this promise, capable of quantifying
the qualitative indications of too much, essentially, and almost thanks to a
privacy measure, a parameter $\delta$, referred to as privacy budget. When
applied to ML/DL models, DP injects $\epsilon$-calibrated noise, e.g. drawn
from a Gaussian distribution, to perturb model updates or gradients, at the
cost of possible performance degradation (e.g., [29, 30, 31]), with $\epsilon$
being the parameter that tunes the noise magnitude, the smaller the $\epsilon$
the stronger the noise. In FL processes, DP should be employed from the
beginning, hence it cannot be seamlessly integrated into ongoing training;
furthermore, DP is not helpful when there is the need to remove harmful
contributions from malicious clients. From these observations, FU can be
considered an orthogonal tool with respect to DP.

Figure 2: Federated unlearning overview. When the client U requests to be
forgotten, the FL server erases its contributions from the global model
$w_{t}$, generating a new version $w_{t}^{\bar{u}}$.

##  III Federated Unlearning

Suppose that after $t$ rounds of FL, a given client $u$ submits an unlearning
request for a subset $S_{u}$ of its private data $D_{u}$ and an unlearning
algorithm $\mathcal{U}$ is executed to fulfill the request. An unlearning
algorithm $\mathcal{U}$ is applied to the global model $w_{t}$, which has been
trained including client $u$ in the pool of clients, generating an unlearned
global model $w^{\bar{u}}_{t}$, i.e. $w^{\bar{u}}_{t}=\mathcal{U}(w_{t})$.
$\mathcal{U}(\cdot)$ can be defined as a function that ensures that
$w^{\bar{u}}_{t}$ exhibits performances indistinguishable or approximately
indistinguishable from a global model $w^{r}_{t}$ trained without the
contribution of $S_{u}$.

FU can have different objectives (sample unlearning, class unlearning, and
client unlearning), as we detail in Subsection III-A. In Subsection III-B we
qualitatively and experimentally motivate the need for unlearning even though
FL does not require direct access to raw data. In fact, the weights of the
global model being trained round by round may leak information about clients’
data. As we describe in Subsection III-C, the architectural design of FL
introduces a more challenging setting than regular MU, due to the
decentralization and inscrutability of data, and the iterative and stochastic
nature of the FL process. In Subsection IV-A we outline the requirements that
FU algorithms should meet, while in the last Subsection IV-B we outline and
describe the metrics being used in the related literature to assess that an FU
mechanism fulfills the previously introduced requirements.

###  III-A Objectives

Figure 3: Visualization of FU objectives. We consider four clients in the
federation, i.e., clients A, B, C, and D, and four total classes for the data
held by clients, i.e., L1, L2, L3, and L4. The figure reports the label that
corresponds to a specific sample in the local dataset of clients (e.g., data
sample 1 in client A’s dataset belongs to class L1).

As depicted in Fig. 3, FU aims to achieve one or more of the following
unlearning objectives [32]:

Sample Unlearning. Sample unlearning aims to remove the contribution of
specific data samples from the trained model. In this case, the client that
requests the unlearning would like to erase the contribution of a subset of
its data, i.e. $S_{u}\subset D_{u}$.

Class Unlearning. Class unlearning aims to remove the contribution of all the
data samples that belong to a certain class $c$ across clients, i.e. to remove
the contribution of $C=\bigcup_{k\in K}S_{k}$ with
$S_{k}=\\{x_{i},c\\}^{n^{c}_{k}}_{i}$, with $n^{c}_{k}$ the number of local
samples at client $k$ labeled with class $c$. Ideally, the unlearned model, if
provided with samples belonging to a removed class, should produce an outcome
at random between the remaining classes.

Client Unlearning. Client unlearning is peculiar to federated settings, and it
relates to the right to be forgotten of clients. In this case, the client that
requests the unlearning would like to erase the contribution of its entire
local dataset, i.e. $S_{u}=D_{u}$. Client Unlearning can be seen as applying
Sample Unlearning on all the samples that a specific client held at the time
of local training. Similarly, Client Unlearning can also be a special case of
class unlearning when a single client $u$ is the only one holding all the
samples for class $c$ across the federation, i.e.,
$C=S_{u}=\\{x_{i},c\\}^{n_{u}}_{i}$ and $S_{u}=D_{u}$.

Figure 4: The global model’s accuracy on test data (solid line) versus the
global model’s accuracy on client $u$’s train data (dashed line) across many
FL rounds. Grey bars indicate that client $u$ has participated in that round.
The upper row of charts considers IID data distribution among clients. The
bottom row of charts considers non-IID data distribution among clients,
simulated via distribution-based labels skew (with concentration parameter
$\alpha$=0.1). The last column of charts depicts the global model’s test
accuracy with or without client $u$ in the federation across rounds.

(a) IID

(b) Non-IID

Figure 5: The KL divergence between the global model’s output probability on
client $u$’s train data and a uniformly distributed output probability (as
proposed in [26]), across many FL rounds. The red line refers to regular
training (client $u$ is included in the federation only for the first 50
rounds). Red bars indicate that client $u$ has participated in that round. The
gray line refers to retraining (sanitized federation) where client $u$ never
participates.

###  III-B Why Is Unlearning Needed if Data Are Not Shared?

To intuitively understand the impact of a client’s data on the global model we
performed a set of experiments, whose results are shown in Table III.
Specifically, we compared the performance in accuracy of a global model
trained including or excluding a specific client $u$ in the federation. While
the test accuracy is very similar among such two versions of the global model,
the global model exhibits significantly higher accuracy on client $u$’s train
data when the federation includes client $u$. This means that the global model
trained with client $u$ among the federation may leak information about client
$u$’s private data and that may be susceptible to, at least, membership
attacks222As we will deepen in Sec. IV-B, membership attacks tries to
determine whether one or more specific samples have been used for training..
It is worth noting that the same trend emerges with homogeneous and
heterogeneous data distributions among clients.

Table III reports the accuracy of a global model $w$ trained on a federation
of participants including or excluding a specific client $u$. Three datasets
for image classification were considered (CIFAR-100, Birds and Aircraft), and
a visual transformer architecture was used (MiT-B0 [33]), starting from a
pretrained checkpoint and performing FL for 100 rounds. The test column refers
to the global model’s accuracy on a test set that clients never see during
training. The train column refers to the global model’s accuracy on the train
set of client $u$. We tracked the accuracy of the global model on test and
client $u$’s train data across 100 rounds when the federation includes client
$u$, visually highlighting whether client $u$ participates in a round. As
evident from Figure 4, when client $u$ is included in the training round, the
global model spikes up to a higher level of accuracy on client $u$’s train
set. Also, it is worth noting that if client $u$ does not participate for a
while, the global model tends to become less accurate on that train data, but
still above the test performances. This phenomenon has also been noted in
[26].

Moreover, to understand whether natural forgetting could be a viable option
for unlearning, i.e. just excluding the client that requested its removal from
the federation, we performed a set of further experiments.We want to
empirically determine whether the contribution of that client naturally fades
away after one or a few rounds, taking advantage of catastrophic forgetting
(see also Sec. II-C). We simulated a large-scale federation with 500 clients
training for 100 rounds (4% participation rate), with client $u$ leaving at
round 50. As depicted in Fig. 5a (homogeneous data among clients), after
client $u$ leaves and for the rest of the FL process, the global model
steadily keeps generating more meaningful predictions than the retrained
baseline on client $u$’s train data for all the tracked rounds. Furthermore,
the global model still outputs more accurate predictions (with lower loss as
well) on client $u$’s train data compared to a sanitized federation from the
beginning (see also the complete metrics in Fig. 8). Contrarily, when data are
heterogeneous (reported in Fig. 5b), catastrophic forgetting seems to help
natural unlearning: after a few rounds, there is no sensible difference
between the tracked metrics for the two versions of the global model.

Overall, the empirical evidence shows that, even with a large-scale federation
of clients holding small local datasets, the influence of a client’s
contribution on the global model may remain noticeable for long at least when
data are homogeneously distributed.

Further details about the experimental settings, employed metrics, and
reference to our code repository are provided in Appendix A.

TABLE III: Accuracy (%) of a global model trained with or without (w/o) a specific client in the federation. |  |  $w$ trained w/o client $u$ |  $w$ trained with client $u$
---|---|---|---
| Dataset | Test | Train | Test | Train
IID | CIFAR-100 | 79.37 | 76.80 | 78.72 | 87.00
Aircraft | 68.53 | 66.00 | 68.83 | 90.00
Birds | 69.4 | 68.00 | 70.66 | 99.50
non-IID | CIFAR-100 | 70.90 | 67.34 | 70.98 | 94.60
Aircraft | 54.64 | 58.82 | 58.36 | 88.24
Birds | 59.13 | 57.3 | 57.37 | 83.78

###  III-C Challenges in Federated Unlearning

Concerning centralized MU, FL settings introduce even more challenging
constraints. The iterative nature of collaborative learning, the stochastic
selection of per-round participating clients, and the decentralization and
inscrutability of local data represent additional degrees of complexity that
have to be accounted for in FU algorithms.

Iterative Learning Process. FL algorithms proceed in iterative rounds, and the
global model $w_{t}$ aggregated at round $t$ is the starting state for round
$t+1$. Suppose that at round $t_{i}$ a client $u$, which has been included in
a round $t_{u}$ ($t_{u}<t_{i}$), requires its contribution deletion (e.g.,
client unlearning). The naive solution to client unlearning would be to
recover the global model at round $t_{u}$ and to exclude client $u$’s updates
from the aggregation of round $t_{u}$. However, the iterative nature of FL
algorithms implies that all the global aggregations subsequent to round
$t_{u}$ should be invalidated.

Non-deterministic Training Process. The selection of clients that participate
in an FL round is subject to various dynamics. For example, clients that are
eligible for a certain round may not be available in the future, due to
connection or battery issues, or due to unwillingness to participate. This
peculiarity of FL makes the possibility of exactly reproducing the rounds
after $t_{u}$ unrealistic. At the same time, discarding all the knowledge
acquired between the aggregation at round $t_{u}$ and the aggregation at round
$t_{i}$ (i.e., the most recent round), that possibly may be distant in time
with $t_{u}<<t_{i}$, would waste significant learning efforts of the
federation. In this sense, methods that can embed that knowledge are most
appealing.

Inscrutable Model Updates and Finite Memory. In principle, the FL aggregator
or server should not or could not be able to inspect the model updates of
single clients (e.g., because the updates could be aggregated via Multi-Party
Computation [34, 35]). This would make unfeasible the naive solution of
rolling back to the global aggregation of round $t_{u}$ because it requires
access to single updates of clients. At the other end of the spectrum, the
central FL aggregator should store the entire history of aggregated models and
collected updates of each round, which would translate into unfeasible storage
capacity.

Inscrutable Local Datasets. While in centralized ML all the data used for
training are supposed to be directly accessible and potentially be used for
MU. Contrary, in FL contexts, datasets remain private by design. This prevents
the application of solutions from plain MU that require direct access to the
data to forget and/or to the data to retain. For example, the recent work in
[36], targeting centralized unlearning, selectively induces noisy knowledge on
the data to forget while reinforcing good knowledge on the retain set
respectively employing an incompetent or competent teacher in a Knowledge-
Distillation framework. This kind of mechanism cannot be directly adapted to
FL due to the inscrutability of local datasets of clients, which cannot be
exported or accessed by other parties.

Data Heterogeneity. In centralized ML the data is supposed to be identically
and independently distributed (IID) since data are centralized in the same
location and can be shuffled and arbitrarily partitioned among machines in
case of distributed training. On the contrary, one of the defining
characteristics of the FL setting is represented by the presence of data
heterogeneity among clients, i.e. clients hold data that are likely to be non-
IID (see Fig. 6 in Appendix for an example of non-IID data among clients).
While this may hamper the convergence of the global model [37, 38], this trait
of FL settings could be convenient for an FU algorithm, since it may be easier
to erase the learned representations peculiar to a skewed subset of data (see
also the illustrative results in Sec. III-B). However, since data are not
directly accessible, it may be not straightforward to assess the level of data
heterogeneity in advance.

###  III-D Entity Performing Unlearning

The unlearning phase can be performed by one or more parties in the
federation, with relative benefits and drawbacks [11]:

  * •

Server. The server has more computation power and storage capacity than
clients. It could store the historical global models as well as keep track of
historical individual clients’ updates. In this way, the server could remove
the contribution of a specific client, when requested to. However, to perform
such a removal, the server should be able to associate one or more specific
historical updates with the identity of a specific client, which poses
additional privacy concerns.

  * •

Target Client. The target client, i.e. client $u$, has direct access to the
data to be forgotten. It can locally perform and assess the unlearning
success. However, this poses further security concerns when considering
possibly malicious clients, who may request a local unlearning to inject a
backdoor in the model or to hamper the training on purpose, taking advantage
of their role in the unlearning phase. Furthermore, regular MU methods often
require both retain and forgetting data, e.g., to alternate forgetting and
retaining phases so that the model can selectively unlearn. The technical
issue in only-local FU is to be able to retain good knowledge without having
access to other clients’ data.

  * •

Remaining Clients: The remaining clients, i.e. the federation without client
$u$, hold the retain data and are typically used to recover the performance of
an unlearned model.

Given the above benefits and drawbacks, mixed solutions may be also possible,
e.g., target client $u$ and the server may cooperate during the unlearning
phase.

##  IV Design and Implementation Guidelines for Efficient FU Algorithms

Here, we present the primary requirements that an FU algorithm should meet to
be efficiently adopted and integrated into practical scenarios. In addition,
we delve into the in-depth exploration of the metrics that have been proposed
to assess the extent to which data has been effectively unlearned from the
global model.

###  IV-A Requirements of Federated Unlearning Algorithms

  1. 1.

Improved Efficiency wrt. the Retraining Strategy. An unlearning algorithm
should be more efficient than the naive, possibly unfeasible, retraining
strategy, e.g. re-utilizing the knowledge that would have been discarded by a
global model rollback

  2. 2.

Retained or Recovered Performances. Either just after the unlearning process
(retained performances) or after a set of new FL rounds (recovered
performance), the unlearned model should attain similar performance to the
original model. In particular, it should exhibit the following properties.

     * •

Comparable performances on test data wrt. the retrained model. The performance
(e.g., loss and accuracy) of the unlearned model on a test set should be
comparable to the performance of a model trained only on a sanitized dataset.
Since the unlearned global model needs a certain amount of rounds to recover
the performance of the original model.

     * •

Comparable performances on test data wrt. the original model. The performance
(e.g., loss and accuracy) of the unlearned model on a test set should be
comparable to the model’s performance before the unlearning algorithm is
applied.

  3. 3.

Effective Forgetting.

     * •

Comparable performances on forgetting data wrt. the retrained model. Ideally,
the prediction of an unlearned model on forgetting data should be
indistinguishable from the prediction of a model trained only on the retained
dataset. Referring to Table III, after an ideal unlearning phase, the global
model trained including client $u$ should not significantly outperform the
retrained global model on client $u$’s train data.

     * •

Significantly different performances on forgetting data wrt. the original
model. If the unlearning process is successful, the unlearned model and the
original model should respond significantly differently when presented with
forgetting data. Accordingly, to the specific metric or indicator, the
unlearned model can be expected to improve or reduce the specific metric value
on forgetting data (e.g., reducing a backdoor attack success rate). In
general, the absolute value of the delta among the unlearned model performance
and original model performance metric on forgetting data should be maximized.

  4. 4.

Same Privacy Levels as Regular FL. FL’s main characteristic is the privacy by
design that comes with leaving data distributed on the data owner’s premises.
An FU algorithm should not violate this fundamental requirement.

###  IV-B Metrics to Assess Unlearning

TABLE IV: Summary of the main metrics in FU literature.

Category ID Metric Range Exp. Value Sample Class Client Improved Efficiency
IE.1 Speed-up ratio $(0,+\infty)$ $\uparrow,w^{r}$ ✓ ✓ ✓ IE.2 Communication
cost reduction ratio $(0,+\infty)$ $\uparrow,w^{r}$ ✓ ✓ ✓ Recovered
Performance RP.1 SAPE [0, 1] $\downarrow,w^{r}$ ✓ - ✓ RP.2 Performance on test
dataset $\approx,w^{r},w^{i}$ ✓ - ✓ Accuracy [0, 1] Loss $[0,+\infty)$ RP.3
Performance on retained training dataset $\approx,w^{r},w^{i}$ ✓ - ✓ Accuracy
[0, 1] Loss $[0,+\infty)$ RP.4 Performance on retained or unlearned classes
(test dataset) $\approx,w^{r},w^{i}$ - ✓ - Accuracy [0, 1] Loss $[0,+\infty)$
RP.5 ECE [39] [0, 1] $\approx,w^{r},w^{i}$ - - ✓ Forgetting Verification FV.1
Performance on client $u$’s data $(\approx,w^{r}),(\downarrow/\uparrow,w^{i})$
✓ - ✓ Accuracy [0, 1] Loss $[0,+\infty)$ FV.2 Accuracy on removed class(es)
(test dataset) [0, 1] $\downarrow$ - ✓ - FV.3 KL divergence (model’s output
vs. uniform distrib.) $[0,+\infty)$ $\downarrow$ ✓ - ✓ FV.4 KL divergence
(per-class accuracy distributions) $[0,+\infty)$ $\downarrow$ - ✓ - FV.5 KL
divergence between retrained and unlearned weights $[0,+\infty)$ 0 - - ✓ FV.6
$\ell_{2}$-norm between retrained and unlearned weights $[0,+\infty)$
$\downarrow$ ✓ - ✓ FV.7 FR (Liu et al. [40]) [0, 1] $\uparrow$ ✓ - ✓ FV.8 IF
[41] on forgetting data $(-\infty,+\infty)$ 0 - - ✓ FV.9 RE [42] $[0,+\infty)$
$\downarrow$ ✓ - - FV.10 AD [42] $[0,+\infty)$ $(\downarrow,w^{r})$,
$(\uparrow,w^{i})$ ✓ - ✓ FV.11 BASR ([43, 44, 45]) [0, 1] $(\approx,w^{r})$,
$(\downarrow,w^{i})$ ✓ - ✓ FV.12 MIA (Shokri et al. [9]) [0, 1]
$(\approx,w^{r})$, $(\downarrow,w^{i})$ ✓ - ✓ FV.13 MIA (Golatkar et al. [46])
[0, 1] $(\approx,w^{r})$, $(\downarrow,w^{i})$ ✓ - ✓ FV.14 MIA (Yeom et al.
[47]) [0, 1] $(\approx,w^{r})$, $(\downarrow,w^{i})$ ✓ - ✓ FV.15 MIB (Hu et
al. [48]) [0, 1] $(\approx,w^{r})$, $(\downarrow,w^{i})$ ✓ - ✓

The requirements described in Subsection IV-A call for metrics that can assess
the efficacy of unlearning algorithms. In this subsection, we gather and
introduce the metrics used in literature to evaluate FU methods and we
classify them according to the scope of their assessment. Table IV summarizes
the main characteristics of the considered metrics. All the reported metrics
are supposed to be applied to the unlearned model $w^{\bar{u}}$. The first
column indicates the scope of the metric. The ID column associates an
identifier to the metrics that will be useful in the next sections. Range
column reports the limit values of the metric. The Expected Value column
indicates if higher values or smaller values are better. If the Expected Value
column contains tuples $(\cdot,\cdot)$ in the form of
$(\\{\uparrow,\downarrow,\approx\\},w^{*})$, it indicates if higher values,
smaller values or values similar to a baseline $w^{*}$ (e.g., the retrained
model’s performances) are better. $w^{i}$ represents the global model before
unlearning, $w^{r}$ represents the retrained model. Accuracy and success rate
are expressed as probability and can be equivalently expressed in percentage
(i.e., in a range [0%, 100%]). The three rightmost columns report the
unlearning objectives for which the metric is most appropriate.

Improved Efficiency Metrics and Indicators. This class of metrics relates to
the Requirement (1) of the previous section and usually compares, as a primary
dimension, the time needed by the proposed unlearning algorithm to recover
performance similar to a baseline, e.g. the original model or the retrained
model. For example, the ratio among $T_{w^{*}}/T_{w_{\bar{u}}}$, with
$T_{w^{*}}$ being the time needed by a model trained with a specific baseline,
e.g. $w^{r}$, and with $T_{w^{\bar{u}}}$ being the time needed by the
unlearned model to attain a certain test performance, can be used as an
indicator of speed-up. However, recovery time is not the only measure used in
the FU literature. In [49], Halimi et al. consider the required cumulative
communication cost in client-to-server model updates to achieve similar
performance (e.g., in test accuracy) to assess the efficiency of the unlearned
method compared to the retrained model, with the lower the better. This
comparison could be also expressed as a ratio $Bytes_{w^{*}}/Bytes_{w_{u}}$
(higher is better), with $Bytes$ representing the size (e.g., in MB) of the
cumulative client-to-server updates.

In [50], Su et al. also focus on minimizing the number of clients that
participate in the recover process.

Retained or Recovered Performance Metrics. This class of metrics relates to
the Requirement (2) of the previous section. The straightforward metrics that
can be employed to asses whether the unlearned model achieves performances
similar to a baseline model are the test performances (e.g., loss, accuracy).
Another potential metric to consider is the Expected Calibration Error (ECE)
[39]. It is used to evaluate the calibration of probabilistic models, focusing
on the alignment between predicted probabilities and actual observed
frequencies. It involves dividing predictions into probability bins, comparing
average predicted probabilities with observed accuracy in each bin, and
calculating the average absolute differences. ECE provides a measure of how
well a model’s predicted probabilities reflect the true likelihood of events,
indicating the model’s reliability and confidence calibration. In [51], Liu et
al. propose to use a compact indicator for comparing the test performances of
the unlearned model to a reference model. Such a metric is the Symmetric
Absolute Percentage Error (SAPE), computed as

$\frac{|Acc({w^{u}},D_{test})-Acc({w^{*}},D_{test})|}{|Acc({w^{u}},D_{test})|+|Acc({w^{*}},D_{test})|},$
(4)

with the baseline model indicated with $w^{*}$ (e.g., the retrained model or
the original model) and $Acc({\cdot},D_{test})$ being the accuracy on test
data. SAPE spans in a range $\in[0,1]$, and the lower the value, the closer
the unlearned model is to match the baseline model’s test performance. To
assess Class Unlearning, the test performance should be measured and compared
only on the subset of test data belonging to the retained classes [52].

Forgetting Verification Metrics and Indicators. This class of metrics relates
to the Requirement (3) of the previous section. Comparing the performance
(e.g., loss and accuracy) of the unlearned model to a baseline model (e.g.,
the retrained model) on client $u$’s train data can represent a preliminary
indicator to asses selective forgetting. For example, the retrained model and
the unlearned model should exhibit similar performance on client $u$’s train
data. However, from a real-world, practical perspective, this can be only
evaluated locally by clients, since their data are private.

Another metric used more often to evaluate FU algorithms is the success rate
of backdoor attacks (BASR) [43, 44, 45]. In a nutshell, the attacker client
crafts malicious inputs to instill a backdoor into the model, meaning that
when the model encounters a similar input, it will misclassify it. For
example, to inject a backdoor pattern in a visual classification task, the
malicious client modifies some pixels in the original images and associates
wrong labels to those inputs. The success rate of the attack is the
probability that the model predicts induced wrong class for any malicious
inputs. From the metric perspective, backdoor attacks should induce the global
model to mispredict specific inputs that embed the backdoor pattern, while the
global model’s performance should remain unaffected by regular inputs. The
unlearning algorithm removes the contributions in the global model of such
harmful backdoored training data. Hence, a successfully unlearned global model
should reduce the success rate of backdoor attacks when triggered by
backdoored samples but should perform well on the test dataset (i.e.,
similarly to the retrained model or the original model).

Inspired by [43], Hu et al. [48] proposed a novel membership inference
technique namely Membership Inference Backdoor (MIB). In MIB, a data owner
marks their samples before releasing them. If the marked samples have been
used by an unauthorized ML model, a backdoor will be injected into the model.
To prove that their data was used for training, data owners can demonstrate
the presence of the backdoor in the model. In the context of FU, MIB can be
leveraged to evaluate unlearning methods by introducing backdoor triggers to
the erased data samples and validating the unlearned model’s susceptibility to
backdoor attacks. As backdoored samples are hard to remove, the efficacy of
unlearning these data will be even higher when removing regular data.

Another alternative attack for assessing the effectiveness of unlearning
methods is the Membership Inference Attack (MIA) [9, 47, 46]. MIA tries to
guess whether a specific sample was included in the training set. Then, the
success of MIA can be measured via standard accuracy, precision, and recall
metrics, with the unlearned model and the retrained model exhibiting similar
sensitivity to MIA. The rationale when employing MIA in assessing the quality
of unlearning is that a model has forgotten only if an attacker cannot guess
with reliable confidence the membership of the forgetting data. The pioneering
MIA proposed by Shokri et al. [9] firstly trains the so-called attack model
that, once trained, will take as input the predictions of a model on client
$u$’s train data and will output a membership probability, building on the
assumption that if a model is very confident in his predictions then the input
example is likely to be used for training. It is worth noting that membership
attacks like the one proposed by Shokri et al. [9] require that the attacker
knows the type and architecture of the ML model and has access to some data
from the same underlying distribution. In [46], Golatkar et al. propose a
black-box MIA that trains the attacker models on the entropy of the model’s
output probabilities. During the training phase, the attacker model learns to
predict membership by seeing samples from the training data as members and
samples from the test set as non-members. In [47], Yoem et al. propose an MIA
that requires knowing the average training loss of the model as well as having
access to its output logits. Then, a data sample is then inferred as a member
if the training loss of the model on the input data point is smaller than the
known average training loss.

To assess client unlearning within a visual classification task, Halimi et al.
[49] introduce in the federation a subset of clients that hold flipped images.
A successful unlearning algorithm should reduce the accuracy on the flipped
samples while retaining good accuracy on regular samples333When the flip-based
evaluation method presented in [49] is used, no data augmentation is applied
during training..

In [40], Liu et al. introduce the Forgetting Rate (FR), a measure to evaluate
the performance of unlearning. To calculate FR, it is necessary to use a
membership oracle that can discern whether a given sample is categorized as
known or not. Given a trained ML model and an arbitrary sample $x$, an ideal
membership oracle outputs true if $x$ belongs to the training set; otherwise,
outputs false. FR is an indicator to measure how many samples are changed from
the memorized set to the unknown set after unlearning, and it is calculated as
the transformation rate between known and unknown states, i.e.:

$FR=\frac{BT}{BT+BF}\times\frac{AF}{AT+AF},$ (5)

denoting as BT (the number of eliminated training samples that are predicted
to be true by the oracle before unlearning), BF (the number of eliminated
training samples that are predicted to be false by the oracle before
unlearning), AF (the number of eliminated training samples that are predicted
to be false by the oracle after unlearning), AT (the number of eliminated
training samples that are predicted to be true by the oracle after
unlearning).

Reconstruction Error (RE) and Activation Distance (AD) are two metrics
utilized in [42]. The RE metric evaluates the model’s ability to reconstruct
data subjected to unlearning, with a lower score indicating enhanced
performance. Simultaneously, the AD metric evaluates the indistinguishability
between model outputs by measuring the average distance between the model’s
predictions before and after unlearning, employing the $2$-distance metric on
forgetting data, the higher the better. A similar metric is used in [53] and
referred to as prediction difference.

In [26], Gao et al. propose to employ metrics on specific subsets of client
$u$’s forgetting data, called markers, on which calculating more reliable
indicators about unlearning performance. Gao et al., most notably, consider as
markers the forgettable samples, i.e. samples with the highest loss variance
across several rounds, and erroneous samples, i.e. the high loss samples. To
assess unlearning, Gao et al. mainly track four metrics: loss, accuracy,
influence function (IF) [41], which estimates the impact of a training sample
on model prediction, and a metric based on KL divergence. The latter is
computed as follows:

$KL(f_{w}(S_{u}),p).$ (6)

The formulation above measures the distributional distance between a model’s
output on forgetting data $f_{w}(S_{u})$ and a uniformly distributed output
probability $p$, with $p=(\frac{1}{L},...,\frac{1}{L})$, $L$ the number of
total classes, and $KL$ the KL divergence. If the KL divergence is low or even
0, the model is not able to produce any meaningful predictions on forgetting
data, i.e. the model has forgotten such data. The KL divergence can be also
used to measure the distance between weight distributions, i.e. to assess the
disparity between the weight distribution obtained through the unlearning
process and that obtained through retraining [54], with lower values expected.

While the metrics detailed above are mainly employed to assess the forgetting
in client or sample unlearning methods, other metrics tailored to evaluate
class unlearning exist. For example, suppose a test set is divided into two
subsets, i.e., the set of samples labeled with the unlearned class and the set
of all the other samples. The unlearned model’s accuracy on the former set
should ideally be 0, whereas the accuracy on the latter set should be similar
to the retrained model [55, 52]. However, it should be ensured that the
unlearned model does not just misclassify the removed class with another. A
possible metric to assess that the unlearned model spans the removed class’s
prediction across the remaining classes is presented in [52]. Wang et al. [52]
measures the KL divergence among the distribution of the unlearned model’s
per-class accuracy and the distribution of retrained model’s per-class
accuracy. The closer to 0 the distance, the more similar the distributions
are.

##  V Literature Review of Federated Unlearning Methods

This section offers a comprehensive review of the existing literature on FU
methods. We categorize the referenced works based on the unlearning
objectives, techniques, and metrics employed for evaluation. The key features
of the reviewed papers are summarized in Table V. Each referenced paper is
linked with an unlearning objective determined by the respective authors
proposing the method. The Where column indicates the entities involved in the
unlearning process, while the Metrics column reports the identifiers of the
metrics (refer to Table IV) employed for evaluating the effectiveness of the
unlearning scheme. Additionally, the Proxy Data column denotes whether
supplementary data, typically required by the server for recovery, are
necessary. Finally, we highlight if the authors have made their solution
accessible to the community or if they plan to do so upon acceptance.

TABLE V: Summary of FU referenced works.

Objective Technique Ref. Where Metrics Proxy Data Open Source Client Re-
calibration of Historical Updates [53] Remaining Clients & Server IE.1, RP.2,
FV.1, FV.12 - ✓ [56] Remaining Clients & Server RP.2, FV.11 - - [57] Remaining
Clients & Server IE.2, RP.2, FV.11 - - [26] Target Client & Server IE.1, IE.2,
FV.1, FV.3, FV.8 - - [58] Server IE.1, IE.2, RP.2, FV.1 Yes - Knowledge
Distillation [59] Server FV.1, FV.11 Yes - [60] Target Client & Neighborhood
RP.2 Yes - [55] All Clients & Server IE.1, RP.3, FV.1, FV.11 - - Gradient
Modification [49] All Clients & Server IE.1, IE.2, RP.3, FV.11, FV.12, FV.14 -
- [61] All Clients & Server RP.2, FV.11 - - [62] Target Client RP.2, FV.11 - ✓
Clustering [50] Clients in the Cluster & Server IE.1, RP.2 - ✓ Bayesian FL
[54] Target Client FV.5 - - [63] Target Client RP.3, FV.1 - - [64] Target
Client RP.2, RP.3, RP.5, FV.1 - - [65] Target Client IE.1, RP.2, RP.3, FV.1,
FV.5, FV.6, FV.15 - ✓ Differential Privacy [66] Server IE.1, RP.2, FV.12 - -
Other Approaches [67] Server IE.1, RP.2, RP.3 - - [68] Target Client & Server
IE.1, RP.2 - ✓ Class Pruning [52] Target Client IE.1, RP.3, FV.1, FV.2, FV.4,
FV.12 - - Knowledge Distillation [55] All Clients & Server IE.1, RP.3, FV.1,
FV.11 - - Sample Gradient Modification [51] All Clients IE.1, RP.1, RP.2 - ✓
[69] Target Clients & Remaining Clients IE.1, IE.2, RP.3, RP.4, FV.1, FV.13 -
✓ [70] Target Client & Server IE.1 Yes - [71] Target Client RP.2, RP.3, FV.1,
FV.13 Yes - [40] Target Client RP.2, RP.3, FV.7 - - [72] Target Client RP.2,
FV.12 - - Quantization [73] Target Client & Server IE1, RP.1, FV.12 - ✓ [74]
All Clients & Server IE.1, RP.2, RP.3, FV.1 - - Reinforcement Learning [42]
Target Client RP.2, FV.9, FV.10 - - Other Approaches [75] All Clients & Server
IE.1, IE.2, RP.2, FV.1 - ✓ [68] Target Client & Server IE.1, RP.2 - ✓

###  V-A Client Unlearning

Most of the methods proposed for client unlearning can be seen as fast
retraining strategies, which design specific ways to retain as much knowledge
as possible from the ongoing training while selectively overwriting the
influence of forgetting data.

Re-calibration of Historical Updates. FedEraser [53] is one of the first
methods that has been proposed to perform client unlearning, which is achieved
through a retraining that speeds up the recovering phase by leveraging
historical parameter updates stored on the server side. Suppose that the
current round is $t_{i}$ (when unlearning is requested) and that client $u$
participated in round $t_{u}$. FedEraser conducts a calibration phase only
including the retained clients to iteratively sanitize the updates that have
been produced after client $u$ participated instead of discarding them.
FedEraser performs $t_{i}-t_{u}$ calibration rounds, and each round includes
local training of calibrated updates and server-side aggregation of such
updates to produce the sanitized global model to be broadcast at the next re-
calibration round, until all the past updates have been recovered. In [56],
Yuan et al. introduce FRU, an unlearning algorithm designed for federated
recommendation systems based on FedEraser [53]. To minimize storage space
required, FRU suggests a user-item mixed semi-hard negative sampling component
to decrease the number of item embedding updates and an importance-based
update selection component to retain only crucial model updates.

Cao et al. [57] propose FedRecover, a method designed to recover a poisoned
model using historical information. FedRecover utilizes retained updates from
clients to estimate the update produced during retrain-from scratch, employing
the Cauchy mean theorem and the L-BFGS algorithm [76] for an efficient
approximation of the integrated Hessian matrix. To enhance the model’s
performance, it implements various strategies for fine-tuning the recovered
model. These strategies include warm-up, periodic correction, abnormality
fixing, and final tuning. In the warm-up phase, clients calculate exact
updates and send them to the server to create buffers used by the L-BFGS
algorithm in constructing Hessian matrices. These buffers are periodically
updated during the periodic correction phase, where the server requests
clients to send exact updates. These updates are also used to correct the
recovered global model. When an estimated model update for a client is
substantial, the server asks the client to send the exact update to mitigate
the impact of potentially inaccurately estimated large model updates,
constituting the abnormality fixing phase. Finally, in the last phase of final
tuning, clients send exact updates to fine-tune the recovered global model, a
step proven to enhance the model’s performance.

Gao et al. [26] propose a unified framework to perform unlearning and
verification, named VeriFi. It is the first work that grants users the right
to verify (RTV), in which participants could actively check the unlearning
effect. VeriFi consists of three parts: (1) an unlearning module, (2) a
verification module, and (3) an unlearning-verification mechanism, that chains
(1) and (2) into an integrated pipeline. Although the unlearning module can be
any unlearning method, the authors also propose a more efficient one-step
unlearning approach scale-to-unlearn (S2U). The unlearning process is
triggered only when a participant leaves during the later stages of the FL
process when the model has already stabilized. S2U scales up the contribution
of the clients staying in the federation and scales down the contribution of
the leaving client. This adjustment is designed to align the global model more
closely with the local models of those who remain while distancing it from the
leaving one. The verification process is performed locally by the leaving
client. It consists of two steps: marking and checking. The marking step
involves utilizing marked data, known as markers, to fine-tune the local
model. Subsequently, the checking step verifies the extent to which the global
model has unlearned the markers.

FU serves as a viable tool for eliminating contributions from malicious
clients. Guo et al. [58] introduce FAST, a protocol designed to remove the
influence of byzantine participants on the global model. The server retains
all the clients’ updates, using this information to directly adjust the model
parameters and derive the unlearning model. Specifically, the unlearning model
is obtained by subtracting the contributions of malicious clients in each
training round from the final global model. For every iteration, the server
compares the accuracy of the current unlearning model with that of the
previous one. If the current unlearning model consistently outperforms its
predecessor, it indicates that the historical influence of the malicious
client persists, prompting the continuation of the step to delete
contributions in the next round. Conversely, if the accuracy consistently
surpasses that of the previous unlearning model, the unlearning operation
proceeds until the maximum number of attempts to eliminate malicious clients’
contributions is reached. However, their method may eventually lead to a loss
in the predictive power for other data that are not required to be forgotten.
To overcome this concern, the server maintains an additional small benchmark
dataset for supplementary training of the unlearning model, thereby enhancing
the overall accuracy.

Knowledge Distillation. In [59], Wu et al. propose a mechanism for FU based on
Knowledge Distillation (KD), following a recent trend that sees many KD-based
mechanisms designed to improve the FL process [77]. The solution in [59]
leverages KD to quickly recover the global model’s performance after the
removal of the unlearning client’s historical model updates. The unlearning
algorithm proceeds in two server-side steps: (1) The historical updates of the
unlearned client are removed from the current global model to produce a
sanitized model; (2) The previous global model works as the teacher and the
sanitized model works as the student. The sanitized global model mimics the
outputs of the previous global model on proxy unlabeled data to recover the
drop in performance after step (1). In [59], clients do not directly have to
participate in the unlearning phase. However, as a drawback, the solution
proposed by Wu et al. requires (unlabeled) data on the server side, which may
not be easily available for specific tasks. Furthermore, the server should
maintain a full history of the updates collected from clients, as well as the
server should be able to identify clients’ updates to then respond to
unlearning requests.

Zaho et al. [55] present a knowledge erasure strategy called Momentum
Degradation (MoDe). The proposed method decouples the FU process from
training, making it applicable to any model architecture that has completed FL
training without altering the training procedure. Unlearning consists of two
phases: knowledge erasure and memory guidance. In the first phase, the pre-
trained model parameters are adjusted to reduce discriminability for target
data, aligning them with retrained model parameters. The server initializes a
degradation model, sends it to clients for FL training, and uses its weights
as the guiding direction for updating the pre-trained model. Following
momentum degradation, the pre-trained model experiences reduced
discriminability for all data points. The memory guidance phase restores
performance by sending the pre-trained model to all clients and the
degradation model to the target client. The target client uses the degradation
model’s output on its local dataset as pseudo labels, guiding the local
training of the pre-trained model. This ensures the unlearned model’s output
closely resembles the retrained model’s output. All clients contribute to
aggregating the pre-trained model. The knowledge erasure and memory guidance
steps are iterated multiple times.

Ye et al. [60] present a decentralized unlearning framework called HDUS,
employing distilled (smaller) models to build erasable ensembles for each
client. HDUS operates in a fully decentralized learning scenario, where
clients exchange information without the need for a central server. In
addition to its local main dataset and model, each client owns a distilled
model and reference dataset shared among all clients. The client’s objective
is to align its distilled model as closely as possible with the main model,
optimizing the former using a specific loss function and the reference
dataset. Distilled models are exchanged with neighboring clients. During the
inference phase, the client generates an ensemble model output by weighting
the output of its main model and the outputs of all neighbors’ distilled
models. If client $u$ requests unlearning of its data, its neighbors simply
exclude client $u$’s distilled model from the ensemble.

Gradient Modification. In [49], Halimi et al. introduce a method to perform
unlearning directly on the target client. The method employs projected
gradient ascent to maximize the loss from a reference model, defined as the
average of the models of the remaining clients. A $\ell_{2}$-norm sphere with
$\delta$ radius around the reference model that constrains the unbounded loss.
To improve the performance of the unlearned model, the remaining clients run
additional rounds of FL, achieving good performance in a few rounds.
Similarly, Li et al. [61] propose a subspace-based FU method (SFU) that
leverages gradient ascent, constrained by the orthogonal space of input
gradient spaces formed by other clients, to eliminate the target client’s
contribution. SFU does not require storing intermediate updates on the clients
or the server. The process includes three steps: (1) each client, except the
target one, creates a representation matrix using a part of its local data and
sends it to the server. A representation matrix for each layer in a neural
network is formed by combining outputs from the forward pass of random samples
through the network, with each column representing the output for a different
sample. The privacy of the representation matrix is protected through a DP
algorithm. Random factors are added to each client’s layer representation to
prevent the leakage of individual client data information. This addition does
not affect the subspace search process due to the orthogonal nature of the
matrix. (2) The target client executes some round of gradient ascent and sends
the updated gradient to the server. (3) The server then performs Singular
Value Decomposition (SVD) [78] on the set of representation matrices to get
the set of input gradient subspace. It gets the orthogonal projection of the
target gradient for this subspace and removes the contribution of the target
client by updating the global model.

In [62] Alam et al. shift the perspective and explore FU to eliminate
backdoors for the benefit of adversaries. Once attackers have accomplished
their intended objectives or suspect the possibility of detection, they may
wish to eliminate the backdoors they previously injected and erase any trace
of their presence. The proposed method leverages gradient ascent on the target
client to forget its contribution. However, to face challenges like
catastrophic forgetting, the generation of deviating models during the
unlearning process, and the varying importance of individual parameters, the
authors propose two strategies: memory preservation and dynamic penalization.
The first one implies utilizing the benign dataset of the target client to
prevent catastrophic forgetting. The second strategy implements a dynamic
penalty mechanism designed to penalize diversion from the global model.
Moreover, weights are also incorporated into the penalty term, assigning a
unique weight to each parameter based on its significance. Introducing non-
uniform penalties for all parameters may yield superior results. The loss
function is modified following the strategies depicted before.

Clustering. KNOT [50] performs client clustering to speed-up retraining in
asynchronous FL. Indeed, when unlearning is requested, retraining is conducted
exclusively within client $u$’s cluster while the remaining clients are
unaffected.

Bayesian FL. Bayesian FL combines Bayesian methods with FL, introducing a
distribution of models to capture uncertainty instead of a fixed model. Gong
et al. [54] introduce the first FU method in a decentralized network within a
Bayesian framework. It is based on exponential family parametrization and
leverages local gossip-driven communication. When the unlearning client is
scheduled, it removes its local variational parameter from the current global
one, and the result is forwarded to the next client. Gong et al. [63] also
propose a non-parametric federated Bayesian unlearning method, referred to as
Forget-Stein Variational Gradient Descent (Forget-SVGD). Forget-SVGD is an
extension of SVGD [79], a particle-based approximate Bayesian inference method
utilizing gradient-based deterministic updates, and its federated variant
called Distributed SVGD (DSVGD) [80]. After the conclusion of the FL process,
one or more clients may request to forget their data. Unlearning a dataset
from the variational posterior is formulated as the minimization of the
unlearning free energy. This involves identifying a distribution that closely
aligns with the current variational posterior while simultaneously maximizing
the average training loss for the datasets designated for forgetting. An
optimized version for rate-constrained channels is presented in [64]. It
applies quantization and sparsification across multiple particles.

BFU [65] is another bayesian FU method that incorporates parameter self-
sharing. The proposed scheme introduces an unlearning rate that balances the
trade-off between forgetting the erased data and remembering the original
global model. To limit potential performance degradation, data erasure and
maintaining learning accuracy are considered two different objectives.

Differential Privacy. In Sec. II-C, we presented DP as an orthogonal tool for
unlearning methods. In [66], Zhang et al. propose FedRecovery, an algorithm
that embeds DP in their unlearning mechanism. Indeed, when client $u$’s
requests unlearning, firstly the server removes its influence by eliminating
all its gradient residuals, which are generated from clients’ historical
submissions. Then, the server injects Gaussian noise into the resulting model
to make such an unlearned model statistically indistinguishable from the
retrained one. The noise is calibrated according to the upper bound of the
distance between the two models, which is estimated through the smoothness of
client $u$’s loss functions. By leveraging DP, an observer cannot assess
whether a model has been trained by $K$ or $K-1$ clients.

Other Approaches. RevFRF [67] is a framework designed for federated Random
Forests (RFs) that supports secure participant revocation. The authors delve
into the revocation process from two distinctive perspectives: (1) ensuring
the removal of targeted data from the RF, and (2) preempting the potential
collusion of removed clients with the server to illicitly continue utilizing
the outdated RF. To achieve the first level of revocation, the RevFRF
traverses the random decision trees (RDTs) within the trained RF and prunes
all splits contributed by the participant seeking to be forgotten.
Subsequently, the RF undergoes a reconstruction, wherein only the remaining
RDTs are considered. Due to their isolation from each other, the rebuilding of
one RDT does not affect the others. It is crucial to note, however, that this
approach is not easily extendable to other ML models, and the worst-case
scenario coincides with the reconstruction of the model from scratch. The
second level of revocation involves executing additional computations to
refresh the revoked splits with random values, rendering them inaccessible to
any potentially colluding server.

In [68], Pan et al. introduce a novel FU algorithm to perform federated
K-means++. In a nutshell, clients maintain centroid vectors computed on their
local data. Those vectors are then transmitted to the server, which uses them
to obtain a centroid set. If a client wishes its data to be forgotten, it
resets its vector of centroids to zero, prompting the server to re-execute
clustering on the remaining clients’ vectors.

###  V-B Class Unlearning

In Convolutional Neural Network (CNN) models, the visualization of feature
maps activated by different channels reveals their varying contributions to
distinct image categories in image classification. Wang et al. [52] leverage
this feature to introduce FU by selectively pruning channels with high
discrimination on the target category to forget. Class discrimination is
determined using the Term Frequency Inverse Document Frequency (TF-IDF), a
statistical metric originally designed to determine the relevance of a word to
a document within a collection of documents. In this context, each channel’s
output can be seen as a word, the feature maps representing different
categories as documents, and TF-IDF evaluates the relevance between channels
and categories. Once the discriminative channels have been pruned, a fine-
tuning operation is applied to restore the model’s performance.

The MoDe method proposed by Zaho et al. [55] is also presented as a solution
for category removal, without modifying the algorithm introduced in the
previous subsection.

###  V-C Sample Unlearning

Given that sample unlearning can be viewed as a more specialized instance of
client unlearning, these methods inherently adopt strategies inspired by the
same principles. We identify similar categories of methods.

Gradient Modification. Liu et al. [51] propose a rapid retraining strategy
that leverages a low-cost Hessian approximation method and applies it to the
local training of clients, after forgetting data deletion. Specifically, at
the core of the mechanism, a diagonal empirical Fisher Information Matrix
(FIM) is used, with the update rule of the unlearned model employing first and
second-order moments for Hessian momentum, similar to optimizer like Adam
[81]. However, this method may imply a significant reduction in performance
when dealing with complex models.

Dhasade et al. [69] introduce QuickDrop, an FU method that leverages Dataset
Distillation (DD) to perform unlearning and recovery. DD is a technique to
condense a large training dataset into a compact and smaller synthetic dataset
[82]. However, the production of good-quality synthetic data requires many
local update steps. The clients reuse the gradient updates computed during FL
training to reduce the computational overhead. When an unlearning request
comes, each client performs Stochastic Gradient Ascent (SGA) on their local
distilled dataset. While, in the recovery phase, the network then executes
recovery rounds, during which they also use the distilled data that are not
extracted from the samples to forget. It is worth noting that the use of
distilled data slightly decreases the performance. To avoid decreasing
performance, in the recovery phase, the authors include a few original samples
in the distilled datasets that nullify potential performance drops.

FedFilter [70] is an edge caching scheme that uses FU to remove invalid and
outdated data from the global model. The unlearning is initiated by the server
that selects the content to be forgotten and generates a reverse gradient.
Moreover, to maximize the unlearning while minimizing its impact on the model
accuracy, the parameters are iterated through SGD. The reverse gradient is
applied to the local model to erase the selected content locally. Finally, for
the basic layer parameters, FU is achieved by aggregating and broadcasting
model parameters.

Jin et al. [71] present the Forgettable Federated Linear Learning (2F2L)
framework. 2F2L introduces a federated linear learning strategy that leverages
a linear approximation of a Deep Neural Network (DNN) given by the first-order
Taylor expansion. The first initial value for the weights to perform linear
approximation is built using the server’s dataset. With a reasonable amount of
data on the server, this value is believed to be close to the optimal model
weight based on the entire dataset. Having a linear approximation, it is
possible to perform data removal performing gradient ascent using the
quadratic forgetting loss function [46]. However, to achieve removal it is
necessary to compute Hessian matrix inversion, which is a complex task. For
this reason, 2F2L leverages a numerical approximation based on the server’s
data.

Forsaken [40] is a memorization elimination algorithm that employs dummy
gradients to stimulate the neurons of the ML model and erase the knowledge of
specific data. Client $u$ possesses a trainable generator that produces
gradients. During each unlearning epoch, the generator is optimized based on
an objective function aiming to reduce the distance between the current
confidence vector and the confidence vector of a perfectly forgotten sample
rules. Then, the client produces a new dummy gradient and sends it to the
server.

In [72], Xia et al. propose $FedME^{2}$, a FU framework for Digital Twins for
Mobile Networks (DTMN). $FedME^{2}$ comprises two modules: MEval and MErase.
The first one aims to build a memory evaluation model to detect whether the
data to be forgotten is remembered and the degree of memory. This information
is used by the MErase module to guide the privacy erasure process. On the
other hand, the MErase module optimizes the local model leveraging the model
loss, the MEval loss, and a penalty term, which constrains similarity between
the global model before and after data erasure. The server then re-aggregates
the local model from the client and completes the construction of the global
model without privacy information.

Quantization. Xiong et al. [73] introduce Exact-Fun, a quantized FU algorithm
designed to eliminate the influence of a subset of data from a target client
within a global model. In instances where a client requests the removal of a
specific portion of their data, the algorithm calculates a local model using
the client’s trained model up to that iteration. If the resulting quantized
model matches the previously global model, the deletion of that data subset
has no impact on the overall model. However, if quantization leads to a
mismatch, retraining becomes necessary to effectively eliminate the influence
of the unlearned data.

In [74], Che et al. extend their prior work on Prompt Certified Machine
Unlearning (PCMU) [83] to operate effectively in a federated setting. PCMU
employs randomized gradient smoothing and quantization to concurrently execute
training and unlearning operations, enhancing overall efficiency. The use of
random smoothing on gradients implies that the MU model, directly trained on
the entire dataset, shares identical gradients (and parameters) with the model
retrained solely on the remaining data. This occurs within a specific
certified radius, concerning the gradient change before and after data
removals, and a certified budget for data removals. To implement PCMU in a
federated setting, the authors propose a method that involves creating MU
models on clients using the PCMU algorithm. These models are then reformulated
as output functions of a Nemytskii operator, inducing a Frechet differentiable
smooth manifold. This manifold possesses a global Lipschitz constant, that
bounds the difference between two local MU models. At the server, a global
gradient is computed by averaging gradients from all clients, resulting in the
creation of a global MU model. The global Lipschitz property ensures that this
model closely aligns with each local MU model on the clients within a distance
determined by the Lipschitz constant. Consequently, the global MU model can
maintain the certified radius and budget of data removals from the local MU
models to a certain degree.

Reinforcement Learning. In [42], Shaik et al. present FRAMU, an attention-
based MU framework that leverages federated reinforcement learning. FRAMU can
perform unlearning of outdated, irrelevant, and private data in both single-
modality and multi-modality scenarios. It adopts a federated reinforcement
learning architecture, employing local agents for real-time data collection
and model updates, a central server for aggregation and global model updates
using the FedAvg algorithm, and an attention layer to dynamically weigh the
relevance of each data point in learning and unlearning. This layer acts as a
specialized approximator, enhancing individual agents’ learning capabilities
by assigning attention scores to data points, indicating their relevance in
local learning, and continually refining the model through interactions with
the environment and feedback. If these scores fall below specific
predetermined thresholds, the data is considered outdated or irrelevant and is
consequently removed from the model.

Other Approaches. FL can be also employed with alternative data representation
such as a Knowledge Graph (KG), a structured knowledge base of real-world
entities and their relations in terms of triplets. KGs have also been adopted
to advance representation learning techniques, i.e., KG embeddings combine
entities and their relations into a unified semantic space [84]. FL can be
integrated with KG embeddings, giving birth to a novel paradigm namely
Federated KG embedding learning [85]. In this direction, Zhu et al. [75]
propose FedLU, an FL framework for heterogeneous KG embedding. FedLU uses an
unlearning method to erase specific knowledge from local embeddings. On the
one hand, local knowledge from clients is erased through an interference phase
based on hard and soft confusions, which reflect the distance between scores
of the triplet in the forgetting set and its negative samples, i.e., the set
of triplets whose intersection with the local KG is empty. On the other hand,
to limit the decrease in model performance caused by the interference is
recovered via knowledge distillation providing the local client KG without the
forgetting set.

The framework introduced in [68] is also applicable when a client wishes to
remove some of their data. If the data to be removed is not a centroid, no
action is needed. However, if it is a centroid, the client needs to eliminate
that centroid and select a new one by sampling from its local data using a
specific distribution. The updated centroid vector is then sent to the server,
which re-executes clustering on the new vector set, producing a new set of
server centroids.

##  VI Distilled Lessons Learned

In this section, we summarize the main lessons learned from the reported in-
depth review of state-of-the-art FU research.

  1. 1.

Motivation: FU schemes are proposed as valuable mechanisms to preserve privacy
and/or enhance security. From the privacy perspective, FU can guarantee the
right to be forgotten. On the security side, it is a technique for mitigating
the impact of malicious clients by eliminating their contributions, i.e.,
poisoned data and backdoors.

  2. 2.

Empirical Evidences: The contribution of a specific client $u$ has a tangible
influence on the global model, which outputs significantly more accurate
predictions when presented with client $u$’s samples if client $u$ has been
included in the pool of clients. Also, the participation of client $u$
translates to evident spikes of enhanced accuracy (and reduced loss) on client
$u$’s train data for that round. Natural forgetting, i.e. just excluding
client $u$ from the federation in the hope of catastrophic forgetting to
rapidly remove its contribution, is not a viable option. Indeed, empirical
evidences show that the impact of a client participation fades away very
slowly when data is homogeneous, and remains concretely noticeable for a large
number of subsequent rounds. It is worth noting that in the particular case of
a large-scale federation with low participation rate of clients and non-IID
local datasets, natural forgetting can help the unlearning process.

  3. 3.

Requirements and Metrics: As we have detailed from the requirements in Sec.
IV-A, an unlearning algorithm should be more efficient than a retraining
alternative, and the (fine-tuned) unlearned global model should achieve
performance of generalization comparable to the original model. At the same
time, with effective unlearning, the unlearned global model should not respond
suspiciously confidently when presented with forgetting data. Then, how to
verify those requirements?

While for improved efficiency and recovered performance there is quite a
consensus on the metrics to be used (respectively speed-up ratio and test
accuracy are most often employed), the works in literature are less aligned
for forgetting verification. MIA and backdoor success rates emerged as the
most used assessment metrics, while often being arranged differently work by
work.

  4. 4.

Methods: The majority of the presented solutions center around the complete
removal of a specific client’s contributions, i.e., client unlearning.
However, the algorithms designed for a particular FU objective may have
broader applicability. Specifically, client unlearning coincides with sample
unlearning when the samples marked for forgetting are exclusively provided by
a singular client and encompass all of their data. Furthermore, the concept of
client unlearning can be viewed as a specialized instance of class unlearning,
applicable when only a single client contributes to the category targeted for
erasure. Similarly, these considerations can be extended to sample unlearning,
wherein the subset marked for forgetting corresponds to a class to be
unlearned.

##  VII Open Problems and Future Directions

As outlined in previous sections, FU is a relatively novel concept, attracted
growing interest over recent years. However, its rapid evolution has also led
to the rise of new issues that necessitate careful consideration. This section
delves into a comprehensive discussion of the most interesting challenges and
underlines future research directions demanding further exploration.

###  VII-A Standardization

The evaluations performed on the proposed FU schemes underscore a notable
deficiency in standardization, particularly concerning the choice of datasets
for experiments and the metrics employed to evaluate the effectiveness of
unlearning algorithms. The review of referenced papers highlights that the
datasets and metrics adopted are often different. This lack of uniformity
poses a challenge to conducting fair assessments and comparisons across
different FU mechanisms. For instance, researchers may manipulate chosen
datasets differently and utilize a diverse array of metrics, introducing
variability that hinders meaningful comparisons.

To address this issue, it becomes crucial to establish a common ground by
providing all researchers with the same ready-to-use datasets. These datasets
should be sourced from a specific, agreed-upon data repository and pre-
processed uniformly. Similarly, adopting standardized metrics for the
evaluation process is essential. These measures can help ensure consistency
and facilitate a more meaningful comparison among distinct FU proposals.

However, the lack of any standardized datasets and metrics in MU and
consequently in FU demonstrate that a significant amount of work needs to be
done in this direction. While some efforts have been made in the context of FL
(e.g. LEAF [86], TensorFlow Federated datasets [87] and Flower datasets [88,
89]), extending such standardization to MU and FU is an area that requires
substantial attention and development to foster more reliable and comparable
research outcomes.

###  VII-B Incentive to Avoid Unlearning

Existing literature on FU mainly focuses on novel schemes to guarantee the
user’s right to be forgotten and/or remove malicious contributions. Reviewing
the referenced papers, we observed that incentivizing clients to be engaged
and continue contributing is still an under-explored field since the design of
efficient incentive mechanisms is a complex task. The performance of the
unlearned model depends on the specific client who requests to leave. For the
server, it is difficult to design incentives for heterogeneous individuals
with multidimensional private information like training costs. Furthermore,
the server must distinguish between high-quality leaving users and low-quality
ones. It is worth noting that the use of incentives may also encourage
strategic users to intentionally request revocation to be rewarded.

In this direction, only Ding et al. [90] propose an incentive mechanism for FL
and FU. Their method is based on a four-stage game that captures the
interaction and information updates during the learning and unlearning
process. The users’ multi-dimensional private information is summarized as
one-dimensional guiding the incentive design.

###  VII-C Multiple Unlearning Request

Although in real-world scenarios clients may continually request multiple,
sequential unlearning requests, almost all the existing FU methods do not take
this condition into account [69].

Given that FU aims to tackle a highly practical challenge, it becomes
necessary to design and evaluate unlearning mechanisms in practical scenarios.
In this regard, it is essential to examine the system’s efficacy under the
specific circumstance of handling multiple unlearning requests simultaneously.
Therefore, FU schemes should satisfy the requirements of unlearning multiple
classes, or the data of multiple clients using a single unlearning and
recovery stage (if needed).

###  VII-D Feature Unlearning

The analysis of the literature showcases that unlearning solutions focus
predominantly on horizontal FU and do not work in a vertical setting. Feature
unlearning is unique to vertical FL (VFL) and aims to erase data features
owned by a client $u$. This approach differs significantly from traditional
unlearning methods, which typically focus on eliminating specific data
samples. Consequently, techniques developed for other unlearning scenarios may
not be directly applicable. Notably, client unlearning can be viewed as a
specialized case of feature unlearning where a client $u$ possesses only the
features targeted for unlearning.

Despite the potential of feature unlearning is still an open challenge, as
supported by the very limited body of research. Deng et al. [91] contribute to
addressing this gap by presenting a method based on logistic regression for
vertical FU. In the context of VFL, participants collaboratively share a
common set of data samples while maintaining distinct sets of features [92].
The proposed unlearning algorithm subtracts the updates from the target
client, contributing to the refinement of the global model. To enhance both
efficiency and accuracy, constraints are imposed on intermediate updates,
thereby minimizing the sum of intermediate parameters. This optimization
strategy results in a reduced impact on the sum of intermediate parameters for
the remaining clients when the target client’s contributions are removed. To
generate the unlearned model, the server must retain the intermediate
parameters submitted by each client during the overall training. When a target
client requests unlearning of their data, the server retrieves the
intermediate updates from that client in the last round, negates these
updates, and distributes them to the remaining clients. Subsequently, each
client utilizes these negated updates to adjust their local model parameters

###  VII-E Security and Privacy

To unlearn data from a model, the server or clients must perform some
operations that lead to a novel version of the global model. The reviewed FU
methods are based on the assumption that the server and clients trust each
other and data involved in the training are correct [58]. However, it is
trivial to achieve this level of trust, especially in a highly distributed
environment such as FL where participants are typically unknown [93].
Additionally, many of the existing unlearning schemes are directly executed by
clients and we cannot control whether they behave honestly or not. For
example, in [52], the TF-IDF used to determine which channel must be pruned to
unlearn a class is directly computed by clients. Therefore, a malicious
participant may intentionally provide a wrong value, causing the removal of
another category. Therefore, we need to build trustworthy FU mechanisms that
can address these issues.

Furthermore, most of the proposed FU schemes are based on gradient information
from the target client or all clients. As proven in different works [94, 95,
96], a malicious server can leverage the client’s model and the global model
to reconstruct the original client’s data. Therefore, several privacy-
preserving techniques, such as DP (see Section II-C), Homomorphic Encryption
(HE) [97], and Multiparty Computation (MPC) [98], have been proposed to
aggregate client’s updates while preserving their privacy. These privacy
concerns demand FU methods that enable unlearning without leaking private
information.

###  VII-F Data Heterogeneity Among Clients

Data heterogeneity among clients naturally arises in typical FL settings. As
shown in III-B, in specific configurations characterized by low client
participation rates and substantial heterogeneity among clients, achieving
unlearning of client data may be easier. However, the influence of data
heterogeneity on FU algorithms is often under-explored. Frequently, empirical
evaluations of proposed algorithms assume heterogeneous data distribution
among clients as well as small-scale federations.

Moreover, the efficacy of proposed unlearning methods within FedAvg variants
designed to address non-IID data [38, 99] is still an unexplored dimension.
Understanding how this specific class of algorithm either facilitates or
hinders the unlearning process can provide valuable insights into the
robustness and adaptability of unlearning techniques in different FL
environments.

##  VIII Conclusions

Privacy-preserving techniques have gained increasing attention in recent
years. In this perspective, recent regulations have stated that individuals
should have full control over their information, including its dynamic
removal. This condition poses a serious challenge for ML, given that models
are trained by employing data contributed by users. The right to eliminate the
influence of specific training samples has given rise to the novel concept of
MU, known as FU in federated scenarios.

This paper presents a comprehensive survey on FU, with a particular focus on
technical design/implementation guidelines and practical usage scenarios.
Firstly, we introduce the background concepts needed to understand the FU
research area, by including the formalization of FU objectives and the
clarifications of the key distinctions from traditional MU. To provide
concrete insights into FU and its implications, we conduct a series of
experiments by using various datasets. Subsequently, we offer valuable
guidelines for designing and implementing FU algorithms, encompassing an in-
depth analysis of the most employed FU evaluation metrics. Moreover, the paper
offers a technically detailed review of the existing literature, by
categorizing the proposed solutions according to a novel taxonomy based on FU
objectives and metrics employed. Finally, we discuss open challenges and
identify the most promising directions for future research work in the field.

## References

  * [1] J. McAuley and J. Leskovec, “Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text,” in _Proceedings of the 7th ACM Conference on Recommender Systems_ , ser. RecSys ’13. New York, NY, USA: Association for Computing Machinery, 2013, p. 165–172.
  * [2] Z. Yi, X. Wang, I. Ounis, and C. Macdonald, “Multi-modal graph contrastive learning for micro-video recommendation,” in _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , ser. SIGIR ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1807–1811.
  * [3] Wikipedia, “Facebook–cambridge analytica data scandal,” URL https://en.wikipedia.org/wiki/Facebook-Cambridge_Analytica_data_scandal, accessed on October 2023.
  * [4] E. Union, “Complete guide to general data protection regulation compliance,” URL https://gdpr.eu/, accessed on October 2023.
  * [5] S. of California Department of Justice, “California consumer privacy act (ccpa),” URL https://oag.ca.gov/privacy/ccpa, accessed on October 2023.
  * [6] H. Xu, T. Zhu, L. Zhang, W. Zhou, and P. S. Yu, “Machine unlearning: A survey,” _ACM Computing Surveys_ , vol. 56, no. 1, pp. 1–36, 2023.
  * [7] Y. Cao and J. Yang, “Towards Making Systems Forget with Machine Unlearning,” in _2015 IEEE Symposium on Security and Privacy_ , 2015, pp. 463–480.
  * [8] L. Zhu, Z. Liu, and S. Han, “Deep Leakage from Gradients,” in _Proc. of Conference on Neural Information Processing Systems_ , 2019, pp. 14 747–14 756.
  * [9] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2017, pp. 3–18.
  * [10] J. Yang and Y. Zhao, “A Survey of Federated Unlearning: A Taxonomy, Challenges and Future Directions,” _arXiv preprint arXiv:2310.19218_ , 2023.
  * [11] F. Wang, B. Li, and B. Li, “Federated unlearning and its privacy threats,” _IEEE Network_ , pp. 1–7, 2023.
  * [12] Z. Liu, Y. Jiang, J. Shen, M. Peng, K.-Y. Lam, and X. Yuan, “A survey on federated unlearning: Challenges, methods, and future directions,” _arXiv preprint arXiv:2310.20448_ , 2023.
  * [13] H. Zhang, T. Nakamura, T. Isohara, and K. Sakurai, “A review on machine unlearning,” _SN Computer Science_ , vol. 4, no. 4, p. 337, 2023.
  * [14] Y. Qu, X. Yuan, M. Ding, W. Ni, T. Rakotoarivelo, and D. Smith, “Learn to Unlearn: A Survey on Machine Unlearning,” _arXiv preprint arXiv:2305.07512_ , 2023.
  * [15] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9304–9312.
  * [16] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [17] P. Bellavista, L. Foschini, and A. Mora, “Decentralised Learning in Federated Deployment Environments: A System-Level Survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 1, pp. 1–38, 2021.
  * [18] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” _arXiv preprint arXiv:2003.00295_ , 2020.
  * [19] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 691–706.
  * [20] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks,” _arXiv preprint arXiv:1812.00910_ , 2018.
  * [21] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” _Information fusion_ , vol. 58, pp. 52–68, 2020.
  * [22] R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, “Measuring catastrophic forgetting in neural networks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [23] G. Legate, L. Caccia, and E. Belilovsky, “Re-weighted softmax cross-entropy to control forgetting in federated learning,” _arXiv preprint arXiv:2304.05260_ , 2023.
  * [24] D. Caldarola, B. Caputo, and M. Ciccone, “Improving Generalization in Federated Learning by Seeking Flat Minima,” in _Proc. of European Computer Vision Conference_. Springer, 2022, pp. 654–672.
  * [25] G. Lee, M. Jeong, Y. Shin, S. Bae, and S.-Y. Yun, “Preservation of the global knowledge by not-true distillation in federated learning,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [26] X. Gao, X. Ma, J. Wang, Y. Sun, B. Li, S. Ji, P. Cheng, and J. Chen, “Verifi: Towards verifiable federated unlearning,” _arXiv preprint arXiv:2205.12709_ , 2022.
  * [27] C. Dwork, “Differential privacy,” _Encyclopedia of Cryptography and Security_ , pp. 338–340, 2011.
  * [28] C. Dwork, A. Roth _et al._ , “The algorithmic foundations of differential privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3–4, pp. 211–407, 2014.
  * [29] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private language models without losing accuracy,” _arXiv preprint arXiv:1710.06963_ , 2017.
  * [30] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” _arXiv preprint arXiv:1712.07557_ , 2017.
  * [31] E. Bagdasaryan, O. Poursaeed, and V. Shmatikov, “Differential privacy has disparate impact on model accuracy,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 15 453–15 462.
  * [32] L. Wu, S. Guo, J. Wang, Z. Hong, J. Zhang, and Y. Ding, “Federated unlearning: Guarantee the right of clients to forget,” _IEEE Network_ , vol. 36, no. 5, pp. 129–135, 2022.
  * [33] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 077–12 090, 2021.
  * [34] S. Truex, N. Baracaldo, A. Anwar, T. Steinke, H. Ludwig, R. Zhang, and Y. Zhou, “A Hybrid Approach to Privacy-preserving Federated Learning,” in _Proc. of the ACM Workshop on Artificial Intelligence and Security_ , 2019, pp. 1–11.
  * [35] C. Zhang, S. Ekanut, L. Zhen, and Z. Li, “Augmented multi-party computation against gradient leakage in federated learning,” _IEEE Transactions on Big Data_ , 2022.
  * [36] V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli, “Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 6, 2023, pp. 7210–7217.
  * [37] T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” _arXiv preprint arXiv:1909.06335_ , 2019.
  * [38] Q. Li, Y. Diao, Q. Chen, and B. He, “Federated Learning on Non-iid Data Silos: An Experimental Study,” in _Proc. of IEEE International Conference on Data Engineering (ICDE)_. IEEE, 2022, pp. 965–978.
  * [39] M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 29, no. 1, 2015.
  * [40] Y. Liu, Z. Ma, X. Liu, and J. Ma, “Learn to forget: User-level memorization elimination in federated learning,” _arXiv preprint arXiv:2003.10933_ , vol. 1, 2020.
  * [41] P. W. Koh and P. Liang, “Understanding black-box predictions via influence functions,” in _International conference on machine learning_. PMLR, 2017, pp. 1885–1894.
  * [42] T. Shaik, X. Tao, L. Li, H. Xie, T. Cai, X. Zhu, and Q. Li, “FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning,” _arXiv preprint arXiv:2309.10283_ , 2023.
  * [43] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” _arXiv preprint arXiv:1708.06733_ , 2017.
  * [44] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in _International conference on artificial intelligence and statistics_. PMLR, 2020, pp. 2938–2948.
  * [45] M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to $\\{$Byzantine-Robust$\\}$ federated learning,” in _29th USENIX security symposium (USENIX Security 20)_ , 2020, pp. 1605–1622.
  * [46] A. Golatkar, A. Achille, A. Ravichandran, M. Polito, and S. Soatto, “Mixed-privacy forgetting in deep networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 792–801.
  * [47] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in _2018 IEEE 31st computer security foundations symposium (CSF)_. IEEE, 2018, pp. 268–282.
  * [48] H. Hu, Z. Salcic, G. Dobbie, J. Chen, L. Sun, and X. Zhang, “Membership inference via backdooring,” in _The 31st International Joint Conference on Artificial Intelligence (IJCAI-22)_ , 2022.
  * [49] A. Halimi, S. Kadhe, A. Rawat, and N. Baracaldo, “Federated unlearning: How to efficiently erase a client in fl?” _arXiv preprint arXiv:2207.05521_ , 2022.
  * [50] N. Su and B. Li, “Asynchronous federated unlearning,” in _IEEE INFOCOM 2023 - IEEE Conference on Computer Communications_ , 2023, pp. 1–10.
  * [51] Y. Liu, L. Xu, X. Yuan, C. Wang, and B. Li, “The right to be forgotten in federated learning: An efficient realization with rapid retraining,” in _IEEE INFOCOM 2022-IEEE Conference on Computer Communications_. IEEE, 2022, pp. 1749–1758.
  * [52] J. Wang, S. Guo, X. Xie, and H. Qi, “Federated Unlearning via Class-Discriminative Pruning,” in _Proceedings of the ACM Web Conference 2022_ , ser. WWW ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 622–632.
  * [53] G. Liu, X. Ma, Y. Yang, C. Wang, and J. Liu, “Federaser: Enabling efficient client-level data removal from federated learning models,” in _2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS)_ , 2021, pp. 1–10.
  * [54] J. Gong, O. Simeone, and J. Kang, “Bayesian variational federated learning and unlearning in decentralized networks,” in _2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2021, pp. 216–220.
  * [55] Y. Zhao, P. Wang, H. Qi, J. Huang, Z. Wei, and Q. Zhang, “Federated unlearning with momentum degradation,” _IEEE Internet of Things Journal_ , pp. 1–1, 2023.
  * [56] W. Yuan, H. Yin, F. Wu, S. Zhang, T. He, and H. Wang, “Federated unlearning for on-device recommendation,” in _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_ , 2023, pp. 393–401.
  * [57] X. Cao, J. Jia, Z. Zhang, and N. Z. Gong, “Fedrecover: Recovering from poisoning attacks in federated learning using historical information,” in _2023 IEEE Symposium on Security and Privacy (SP)_ , 2023, pp. 1366–1383.
  * [58] X. Guo, P. Wang, S. Qiu, W. Song, Q. Zhang, X. Wei, and D. Zhou, “FAST: Adopting Federated Unlearning to Eliminating Malicious Terminals at Server Side,” _IEEE Transactions on Network Science and Engineering_ , pp. 1–14, 2023.
  * [59] C. Wu, S. Zhu, and P. Mitra, “Federated unlearning with knowledge distillation,” _arXiv preprint arXiv:2201.09441_ , 2022.
  * [60] G. Ye, Q. V. H. Nguyen, and H. Yin, “Heterogeneous Decentralized Machine Unlearning with Seed Model Distillation,” _arXiv preprint arXiv:2308.13269_ , 2023.
  * [61] G. Li, L. Shen, Y. Sun, Y. Hu, H. Hu, and D. Tao, “Subspace based federated unlearning,” _arXiv preprint arXiv:2302.12448_ , 2023.
  * [62] M. Alam, H. Lamri, and M. Maniatakos, “Get rid of your trail: Remotely erasing backdoors in federated learning,” _arXiv preprint arXiv:2304.10638_ , 2023.
  * [63] J. Gong, J. Kang, O. Simeone, and R. Kassab, “Forget-svgd: Particle-based bayesian federated unlearning,” in _2022 IEEE Data Science and Learning Workshop (DSLW)_ , 2022, pp. 1–6.
  * [64] J. Gong, O. Simeone, and J. Kang, “Compressed particle-based federated bayesian learning and unlearning,” _IEEE Communications Letters_ , vol. 27, no. 2, pp. 556–560, 2023.
  * [65] W. Wang, Z. Tian, C. Zhang, A. Liu, and S. Yu, “BFU: Bayesian Federated Unlearning with Parameter Self-Sharing,” in _Proceedings of the 2023 ACM Asia Conference on Computer and Communications Security_ , ser. ASIA CCS ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 567–578.
  * [66] L. Zhang, T. Zhu, H. Zhang, P. Xiong, and W. Zhou, “FedRecovery: Differentially Private Machine Unlearning for Federated Learning Frameworks,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 4732–4746, 2023.
  * [67] Y. Liu, Z. Ma, Y. Yang, X. Liu, J. Ma, and K. Ren, “RevFRF: Enabling Cross-Domain Random Forest Training With Revocable Federated Learning,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 19, no. 6, pp. 3671–3685, 2022.
  * [68] C. Pan, J. Sima, S. Prakash, V. Rana, and O. Milenkovic, “Machine unlearning of federated clusters,” _arXiv preprint arXiv:2210.16424_ , 2022.
  * [69] A. Dhasade, Y. Ding, S. Guo, A.-m. Kermarrec, M. De Vos, and L. Wu, “QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillation,” _arXiv preprint arXiv:2311.15603_ , 2023.
  * [70] P. Wang, Z. Yan, M. S. Obaidat, Z. Yuan, L. Yang, J. Zhang, Z. Wei, and Q. Zhang, “Edge Caching with Federated Unlearning for Low-latency V2X Communications,” _IEEE Communications Magazine_ , pp. 1–7, 2023.
  * [71] R. Jin, M. Chen, Q. Zhang, and X. Li, “Forgettable Federated Linear Learning with Certified Data Removal,” _arXiv preprint arXiv:2306.02216_ , 2023.
  * [72] H. Xia, S. Xu, J. Pei, R. Zhang, Z. Yu, W. Zou, L. Wang, and C. Liu, “FedME2: Memory Evaluation & Erase Promoting Federated Unlearning in DTMN,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 11, pp. 3573–3588, 2023.
  * [73] Z. Xiong, W. Li, Y. Li, and Z. Cai, “Exact-Fun: An Exact and Efficient Federated Unlearning Approach.”
  * [74] T. Che, Y. Zhou, Z. Zhang, L. Lyu, J. Liu, D. Yan, D. Dou, and J. Huan, “Fast federated machine unlearning with nonlinear functional theory,” in _Proceedings of the 40th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23–29 Jul 2023, pp. 4241–4268.
  * [75] X. Zhu, G. Li, and W. Hu, “Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning,” in _Proceedings of the ACM Web Conference 2023_ , ser. WWW ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 2444–2454.
  * [76] J. Nocedal, “Updating quasi-newton matrices with limited storage,” _Mathematics of Computation_ , vol. 35, no. 151, pp. 773–782, 1980.
  * [77] A. Mora, I. Tenison, P. Bellavista, and I. Rish, “Knowledge distillation for federated learning: a practical guide,” _arXiv preprint arXiv:2211.04742_ , 2022.
  * [78] A. Hoecker and V. Kartvelishvili, “Svd approach to data unfolding,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 372, no. 3, pp. 469–481, 1996.
  * [79] Q. Liu and D. Wang, “Stein variational gradient descent: A general purpose bayesian inference algorithm,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [80] R. Kassab and O. Simeone, “Federated generalized bayesian learning via distributed stein variational gradient descent,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 2180–2192, 2022.
  * [81] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [82] J. Geng, Z. Chen, Y. Wang, H. Woisetschlaeger, S. Schimmler, R. Mayer, Z. Zhao, and C. Rong, “A Survey on Dataset Distillation: Approaches, Applications and Future Directions,” _arXiv preprint arXiv:2305.01975_ , 2023.
  * [83] Z. Zhang, Y. Zhou, X. Zhao, T. Che, and L. Lyu, “Prompt certified machine unlearning with randomized gradient smoothing and quantization,” in _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 13 433–13 455. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/5771d9f214b75be6ff20f63bba315644-Paper-Conference.pdf
  * [84] S. Ji, S. Pan, E. Cambria, P. Marttinen, and P. S. Yu, “A Survey on Knowledge Graphs: Representation, Acquisition, and Applications,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 2, pp. 494–514, 2022.
  * [85] H. Peng, H. Li, Y. Song, V. Zheng, and J. Li, “Differentially Private Federated Knowledge Graphs Embedding,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, ser. CIKM ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 1416–1425. [Online]. Available: https://doi.org/10.1145/3459637.3482252
  * [86] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” _arXiv preprint arXiv:1812.01097_ , 2018.
  * [87] TensorFlow, “Tensorflow federated,” https://flower.dev/docs/datasets/d_2, 2023.
  * [88] D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusmão _et al._ , “Flower: A friendly federated learning research framework,” _arXiv preprint arXiv:2007.14390_ , 2020.
  * [89] Flower, “Flower datasets,” https://www.tensorflow.org/federated, 2023.
  * [90] N. Ding, Z. Sun, E. Wei, and R. Berry, “Incentive mechanism design for federated learning and unlearning,” in _Proceedings of the Twenty-Fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing_ , ser. MobiHoc ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 11–20.
  * [91] Z. Deng, Z. Han, C. Ma, M. Ding, L. Yuan, C. Ge, and Z. Liu, “Vertical federated unlearning on the logistic regression model,” _Electronics_ , vol. 12, no. 14, 2023.
  * [92] S. Feng and H. Yu, “Multi-participant multi-class vertical federated learning,” _arXiv preprint arXiv:2001.11154_ , 2020.
  * [93] C. Mazzocca, N. Romandini, M. Mendula, R. Montanari, and P. Bellavista, “TruFLaaS: Trustworthy Federated Learning as a Service,” _IEEE Internet of Things Journal_ , vol. 10, no. 24, pp. 21 266–21 281, 2023.
  * [94] K. Singhal, H. Sidahmed, Z. Garrett, S. Wu, J. Rush, and S. Prakash, “Federated Reconstruction: Partially Local Federated Learning,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 11 220–11 232.
  * [95] C. Chen, L. Lyu, H. Yu, and G. Chen, “Practical Attribute Reconstruction Attack Against Federated Learning,” _IEEE Transactions on Big Data_ , pp. 1–1, 2022.
  * [96] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting Gradients - How easy is it to break privacy in federated learning?” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 16 937–16 947.
  * [97] A. Acar, H. Aksu, A. S. Uluagac, and M. Conti, “A Survey on Homomorphic Encryption Schemes: Theory and Implementation,” _ACM Comput. Surv._ , vol. 51, no. 4, jul 2018.
  * [98] D. Byrd and A. Polychroniadou, “Differentially Private Secure Multi-Party Computation for Federated Learning in Financial Applications,” in _Proceedings of the First ACM International Conference on AI in Finance_ , ser. ICAIF ’20. New York, NY, USA: Association for Computing Machinery, 2021.
  * [99] A. Mora, D. Fantini, and P. Bellavista, “Federated Learning Algorithms with Heterogeneous Data Distributions: An Empirical Evaluation,” in _Proc. of IEEE/ACM Symposium on Edge Computing (SEC)_. IEEE, 2022, pp. 336–341.
  * [100] A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009.
  * [101] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-UCSD Birds 200,” California Institute of Technology, Tech. Rep. CNS-TR-2010-001, 2010.
  * [102] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” Tech. Rep., 2013.
  * [103] E. Hu, Y. Tang, A. Kyrillidis, and C. Jermaine, “Federated learning over images: Vertical decompositions and pre-trained backbones are difficult to beat,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 19 385–19 396.

|  Nicolò Romandini graduated from the University of Bologna, Italy, where he
received an M.Sc. degree in computer science engineering, in 2021. He is
currently a Ph.D. student at the Department of Computer Science and
Engineering at the University of Bologna. His research focuses mainly on
blockchain, cybersecurity and machine learning, and how to integrate them into
IoT domains.
---|---
|  Alessio Mora received his Ph.D. in Computer Science and Engineering, at the
University of Bologna, Bologna, Italy. He is currently a post-doctoral
researcher at the University of Bologna. His research interests include
Decentralized Learning, with a particular focus on Federated Learning, Deep
Learning, and Edge Intelligence.
---|---
|  Carlo Mazzocca received his M.Sc. and B.Sc. degrees in Computer Engineering
in 2018 and 2020, respectively, both from the University of Naples Federico
II, Italy. He is currently a Ph.D. student in Computer Science and Engineering
at the University of Bologna, Bologna, Italy. His research interests mainly
include digital identity, security mechanisms based on distributed ledger
technologies, and authentication and authorization solutions for the cloud-to-
thing continuum.
---|---
|  Rebecca Montanari full professor at the University of Bologna since 2020
carries out her research in the area of information security and the
design/development of middleware solutions for the provision of services in
mobile and IoT systems. Her research is currently focused on blockchain
technologies to support various supply chains, including agrifood,
manufacturing and fashion, and on security systems for Industry 4.0.
---|---
|  Paolo Bellavista received the Ph.D. degree in computer science engineering
from the University of Bologna, Italy, in 2001. He is currently a Full
Professor with the University of Bologna. His research interests include
middleware for mobile computing, QoS management in the cloud continuum,
infrastructures for big data processing in industrial environments, and
performance optimization in wide-scale and latency-sensitive deployment
environments. He serves on the Editorial Boards of IEEE COMMUNICATIONS SURVEYS
AND TUTORIALS, IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, IEEE
TRANSACTIONS ON SERVICES COMPUTING, ACM CSUR, ACM TIOT, and PMC (Elsevier). He
is the Scientific Coordinator of the H2020 IoTwins Project (https:
www.iotwins.eu).
---|---

##  Appendix A Experimental Setup

Figure 6: The label distribution across clients, i.e. the per-label number of
local examples in each client’s dataset.

This section describes the experimental setting for the results reported in
Table III, Fig. 4, Fig. 7, Fig. 5, and Fig. 8. We implemented the simulations
with Flower [88] and TensorFlow. Our code is publicly available at
www.github.com/alessiomora/unlearning_fl.

###  A-A Model Architecture and Hyperparameters

We used a visual transformer, i.e. MiT-B0 [33], with approximately 3.6M
parameters, initialized from a pre-trained model checkpoint trained on
ImageNet-1k (69.27% accuracy on test data). We adapted the one-layer
classification head to the specific task, initializing such a layer from
scratch. We employed the AdamW optimizer with a client learning rate of 3e-4,
with a round-wise exponential decay of 0.998, 5 local epochs, batch size of
32, and weight decay regularization of 1e-3.

###  A-B Datasets

We performed the experiments on three datasets: CIFAR-100 [100], Caltech-2011
(birds) [101], and FGVC-Aircraft (aircraft) [102]. We provided two settings
for the data partitioning among clients of each dataset: identically and
independently distributed (IID) data, i.e. each client holds approximately the
same number of per-class examples, and non-IID simulated via distribution-
based label skew following the method from [37]. For the latter setting, we
tuned the label skew using a concentration parameter of $\alpha=0.1$ to rule
the Dirichlet distribution, with Fig. 6 showing the label distribution across
clients. The same data example is not repeated among multiple clients.

CIFAR-100. CIFAR-100 consists of 60,000 examples of 32x32 color images –
50,000 for training and 10,000 for testing – belonging to 100 classes. To
match the transformer models’ input size, we resized the images to a
resolution of 224x224 pixels; we also preprocessed the training images with
random crop and horizontal flip layers. For the experiments reported in Fig.
4, in Fig. 7 and in Table III, we partitioned the training set to simulate 100
clients in the federation; we set 100 as the number of clients so that, in the
IID setting, each client can have at least five per-class (unique) examples.
At each round, 10 clients out of 100 were randomly selected to participate.
For the experiments on natural forgetting, reported in Fig. 5 and 8, we
partitioned the training set to simulate 500 clients in the federation, with
20 clients selected per round.

Caltech-2011. Caltech 2011 (birds) consists of 11,788 examples of color images
– 5,994 for training and 5,794 for testing – belonging to 200 classes. To
match the input size of the transformer models we resized the images to a
resolution of 224x224 pixels; we also preprocessed the training images with
random crop and horizontal flip layers, similarly to the work in [103]. We
partitioned the training set to simulate 29 clients in the federation; we set
29 as the number of clients so that, in the IID setting, each client can have
at least one per-class (unique) example. At each round, 5 clients out of 29
were randomly selected to participate.

TABLE VI: Dataset-specific configurations. Dataset | Classes | Train examples | Tot. Clients | Active Clients
---|---|---|---|---
Cifar-100 | 100 | 50,000 | 100 - 500 | 10 - 20
Birds | 200 | 5,994 | 29 | 5
Aircraft | 100 | 6,667 | 65 | 7

FGVC-Aircraft. The FGVC-Aircraft dataset contains 10,000 images of aircraft –
6,667 for training and validation and 3,333 for testing. The aircraft labels
are organized in a four-level hierarchy, i.e., model, variant, family, and
manufacturer (from finer to coarser). We consider a classification at the
variant level (e.g. Boeing 737-700). A variant collapses all the models that
are visually indistinguishable into one class. The dataset contains 100 images
for each of the 100 different aircraft model variants. We removed the
copyright banner from the images by cutting off the bottom 20 pixels in
height. To match the input size of the transformer models we resized the
images to a resolution of 224x224 pixels; we also preprocessed the training
images with random crop and horizontal flip layers, similarly to the work in
[103]. We partitioned the training set to simulate 65 clients in the
federation; we set 65 as the number of clients so that, in the IID setting,
each client can have at least one per-class (unique) example. At each round, 7
clients out of 65 were randomly selected to participate.

Table VI summarizes the dataset-specific configurations.

###  A-C Metrics

We tracked three main metrics: accuracy, loss, and KL divergence. Accuracy and
loss are measured either on test data or on client $u$’s train data. The third
metric calculates the KL divergence between the output distribution of the
model on client $u$’s data and a uniformly distributed output probability.
This metric should work as a proxy for the meaningfulness of model predictions
(the greater the divergence the more meaningful the predictions).

Figure 7: The global model’s loss on test data (solid line) versus the global
model’s loss on client $u$’s train data (dashed line) across many FL rounds.
Grey bars indicate that client $u$ has participated in that round. The upper
row of charts considers IID data distribution among clients. The bottom row of
charts considers non-IID data distribution among clients, simulated via
distribution-based labels skew (with concentration parameter $\alpha$=0.1).

Figure 8: KL divergence, loss and accuracy tracked for 100 rounds. 500 clients
(4% participation rate). For the red line, client $u$ participates only during
the first 50 rounds. For the gray line, client $u$ never participates.