# Differential Privacy: What is all the noise about?

Roxana Danger111regulaition.com

###### Abstract

Differential Privacy (DP) is a formal definition of privacy that provides
rigorous guarantees against risks of privacy breaches during data processing.
It makes no assumptions about the knowledge or computational power of
adversaries, and provides an interpretable, quantifiable and composable
formalism. DP has been actively researched during the last 15 years, but it is
still hard to master for many Machine Learning (ML)) practitioners. This paper
aims to provide an overview of the most important ideas, concepts and uses of
DP in ML, with special focus on its intersection with Federated Learning (FL).

##  1 Introduction

Privacy was first defined by Alan F. Westin as “the claim of individuals,
groups, or institutions to determine for themselves when, how, and to what
extent information about them is communicated to others.” [1]. The rise of
databases, internet, social networks, location technology, digital searching
and advertising, along with all their tracking mechanisms, has made digital
privacy a huge concern, to the point where governments and international
institutions have dictated laws and regulations to provide minimal privacy
guarantees for individuals.

Researchers have tackled the problem using techniques as encryption,
anonymity, obfuscation, diversity, delta-presence, and strict security
measures to reduce access to the data, but all of these have proven to be
insufficient from security or usability perspectives [2, 3]. Differential
privacy (DP) was born in this context to bring a new way of measuring and
guaranteeing privacy. DP is a conceptual framework with a rigorous
mathematical foundation that can be applied to any type of algorithm, and it
sits at the intersection of computer science, statistics, economics, law and
ethics. It has gained an enormous support by the research community, and some
companies and governments are beginning to incorporate it into their own data
pipelines [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15].

Unlike previous privacy definitions, DP is not based on assumptions about a
privacy attacker (whose characterization is a very difficult problem in
itself), but rather on the notion on the concept of _privacy leakage_ [16]
(also referred as privacy loss). Data processors using DP determine the level
of privacy leakage users could tolerate or is ethically acceptable, and
applied DP mechanisms to guarantee these limits are not exceeded.

Many different interrelated concepts are associated with DP: privacy loss,
mechanisms of DP, local and centralized DP, uses of DP in statistics, Machine
Learning and Federated Learning. This paper aims to provide an overview of the
most important ideas, concepts and uses of DP in ML, with special focus on its
intersection with Federated Learning, as well as to provide a practical
guideline for applying DP in a concrete problem.

Privacy leakage is usually denoted by $\epsilon$; it is a provable limit to
the inference that can be derived about an individual private, sensitive,
information from observing the outcome of a DP-mechanism. This formalism is
often called $\epsilon$-DP and its strong mathematical properties makes any
$\epsilon$-differentially private mechanism a trustful one [17]. The original
definitions of $\epsilon$-DP and privacy loss as well as the properties
associated to them were introduced by Cynthia Dwork and her colleagues [16]
and they are discussed in Section 2.

The main idea behind DP is that by adding certain calibrated noise to the
outcome of a non-DP algorithm (called also mechanism in this context), it
converts to an equivalent DP one. The noise is calibrated based on the maximum
privacy loss guarantee while maximizing the utility of the DP results. The
noise is sampled from statistical distributions like Laplace, gaussian,
geometrical- thus producing Laplace, gaussian, geometrical DP-mechanisms.
These basic mechanisms are described in Section 3. More complex mechanisms
have been proposed to fulfil requirements of privacy in multiple scenarios. A
brilliant taxonomy of all these variants with formal definitions, axioms and
theorems that relate to them can be found at: [18], but in Section 4 the
interested reader can peruse a brief summary of them.

When solving a ML task (i.e. classification, regression, clustering), DP can
be applied at any phase of the ML pipeline (preprocessing, parameter
optimization, loss computation, etc.) using the mechanisms previously
described. Particularly important are the extensions of ML optimization
algorithm SGD and the ensemble techniques to convert them into DP mechanisms,
given way to DP-SDG [19] and the ensemble-DP framework [20], which have led to
PATE [21], a well recognised DP framework in ML. Section 5.1 is devoted to
describe these algorithms.

A second path to obtain privacy in ML is through the use of synthetic data
instead of real data. Generative Adversarial Networks (GAN) that have proved
to be powerful tool for generating synthetic data. So, Section 5.2 introduces
some works that combines DP with GAN models.

Strong guarantees for privacy can be obtained by using the above methods.
However, the original private data are still saved and maintained in a server,
and the users have to trust the data holder about their data. In some
scenarios, private data is not stored in a central database and the user can
add noise to the data before allowing any learning task to be performed on it.
This setting is known as local differential privacy (LDP) and the most common
use cases are discussed in Section 6.

Finally, another way of reducing the problems associated with data privacy in
ML (and at the same time improve the generalization power of models) is using
Federated Learning (FL). Federated Learning, in contrast with traditional
centralised ML, trains an algorithm across multiple decentralized servers
holding local data, without exchanging or downloading the data locally. The
core element of FL is that data never leave their original location, therefore
privacy is somehow guaranteed. Other types of attacks in the FL setting are
still possible and multiple counter-attack solutions have been proposed, as
described in Section 7.

In Section 8 we summarise the themes discussed in this study and give some
guidelines for applying DP in practice. Finally, Section 9 contains the
conclusions of the paper.

##  2  $\epsilon$-differential privacy definition

Differential privacy is defined on the concept of adjacent datasets. Two
datasets are adjacent if one of them can be created by removing (or changing)
a single example from the other dataset. DP is formalized as follows [16]:

######  Definition 1 ($\epsilon$-DP)

A randomized mechanism of function
$\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$ with domain $\mathcal{D}$ and
range $\mathcal{R}$ satisfies $\epsilon$-differential privacy if, for any two
adjacent inputs $d,d^{\prime}\in\mathcal{D}$ and for any subset of outputs
$\mathcal{X}\subseteq\mathcal{R}$, it holds that:

$Pr[\mathcal{M}(d)\in X]\leq e^{\epsilon}Pr[\mathcal{M}(d^{\prime})\in X]$

.

$Pr[\mathcal{M}(d)\in X]$ is the probability that when the mechanism
$\mathcal{M}$ runs in the dataset $d$ its output is in the set $X$. So, this
equation says that the change in likelihood of the outcome of $\mathcal{M}$ by
using $d$ or $d^{\prime}$ is bounded by $e^{\epsilon}$. If the likelihood of
obtaining a response is as much tripled when any row of the dataset is removed
or added, $\frac{Pr[\mathcal{M}(d)\in X]}{Pr[\mathcal{M}(d^{\prime})\in
X]}=3$, we are in presence of a $1.1$-DP mechanism, as $e^{\epsilon}=3$ when
$\epsilon=1.1$

The definition of adjacent datasets depend on the application, but it allows
the estimation of a value of the privacy loss, $\epsilon$, when a mechanism is
applied to the adjacent datasets, that is, how much an observer can learn
about a single example by applying the mechanism to a dataset. In fact, using
the Bayes’s Rule and the above definition, it can establish the lower and
bounds of the posterior probability, after applying a $\epsilon$-DP mechanism
and it can be used as an indication of the knowledge gain an observer of $M$
can obtain. Specifically, given a prior probability $p$ representing the
uncertainty of the current knowledge $K$ about an example, the uncertainty
about the example, after applying the mechanism $\mathcal{M}$, i.e. the
posterior probability, $P(\mathcal{M}(d)\in X|K)$, can be bounded as:

$\frac{p}{e^{\epsilon}+(1-e^{\epsilon})*p}\leq Pr(\mathcal{M}(d)\in
X|K)\leq\frac{e^{\epsilon}*p}{1+(e^{\epsilon}-1)*p}$

Figure 1: (a)

Figure 2: (b)

Figure 3: (a) Upper and lower bounds of posterior probability based on epsilon
and a prior probability; (b)Upper and lower bounds breakdown at different
epsilon values.

Figure 3 shows these bounds graphically. The greater $\epsilon$, the greater
the posterior probability, therefore, the greater the privacy loss. Therefore,
choosing smaller $\epsilon$ guarantees a more privacy-preserving (with less
privacy loss or leakage) mechanism.

The next questions are: given a non-DP mechanism, how can it be made
differentially private? And which value of $\epsilon$ should be chosen?

The answer to the first question is to add uncertainty - a probabilistic
noise- to the outcome of the mechanism. This noise addition provides the
necessary randomness to protect sensitive user data. It was proved in [16],
that a mechanism $\mathcal{M}$ can be made $\epsilon$-differentially private
by adding an independent Laplace noise vector $v$:

$Pr[v]=Lap(x|\mu=0,b)=\frac{1}{2b}e^{\frac{||v||_{1}}{b}}\varpropto
e^{\frac{||v||_{1}}{b}}$

where: $||.||_{1}$ is the $L_{1}$ norm, $b=S(\mathcal{M})/\epsilon$ is defined
in terms of the sensitivity of the mechanism $\mathcal{M}$, $S(\mathcal{M})$,
defined by:

$S(\mathcal{M})=max_{d,d^{\prime}}||\mathcal{M}(d)-\mathcal{M}(d^{\prime})||_{1}$

Informally, the sensibility of a mechanism expresses the maximum expected
change of its response considering any pair of adjacent datasets.

A ‘good’ $\epsilon$ value should be low and at the same time maintain a
certain level of utility of the answers provided by the DP mechanism. In other
words, it should still produce responses near to the true values. Its
estimation depends on the application, and it is a challenging open question
to establish the right values for problems in different scenarios [22].

This definition assumes that privacy is only lost due to the observability of
the mechanism. A natural extension is assuming that there are also some
uncontrolled events that can affect privacy. This notion of uncontrolled risk
is captured in the definition of $(\epsilon,\delta)$-DP as follows [23]:

######  Definition 2 (($\epsilon,\delta$)-DP)

A randomized mechanism of function
$\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$ with domain $\mathcal{D}$ and
range $\mathcal{R}$ satisfies $(\epsilon,\delta)$-differential privacy if any
two adjacent inputs $d,d^{\prime}\in\mathcal{D}$ and, for any subset of
outputs $\mathcal{X}\subseteq\mathcal{R}$, it holds that:

$Pr[M(d)\in X]\leq e^{\epsilon}Pr[M(d^{\prime})\in X]+\delta$

.

In this case, $\delta$ is a variable representing the probability that DP is
broken by factors external to the DP mechanism. As for $\epsilon$, we would
like to set $\delta$ to a very low value. To avoid the worst-case scenario of
always violating privacy of a $\delta$ fraction of the dataset, the standard
recommendation is to choose $\delta\ll 1/N$ or even $\delta=negl(1/N)$, where
$N$ is the number of contributors. This strategy avoids the possibility of one
particularly devastating outcome, but other forms of information leakage may
remain [24].

###  2.1 Features of differential privacy

In addition to permit attack modelling, the above definition of DP allows to
quantify privacy loss and satisfy multiple mathematical features:

Composability

: Differential privacy composability states that the joint distribution of
differentially private mechanisms are also differentially private [25]:

  * •

Sequential: If there are $n$ independent mechanisms:
$\mathcal{M}_{1},\dots,\mathcal{M}_{n}$, whose privacy guarantees are
$\epsilon_{1},\dots,\epsilon_{n}$-differential privacy, respectively, then any
function $g(\mathcal{M}_{1},\dots,\mathcal{M}_{n})$ is
$\sum\limits_{i=1}^{n}\epsilon_{i}$-differentially private.

  * •

Parallel: If the previous mechanisms are computed on disjoint subsets of the
database then the function $g$ would be
$(\max\limits_{i}\epsilon_{i})$-differentially private.

This means that we can combine DP-mechanisms and still estimate the privacy
loss of the combined mechanisms.

Robustness to post-processing

: Any deterministic or randomized function $f$ defined over the image of a
$(\epsilon,\delta)$-differentially private mechanism $\mathcal{M}$ is also
differentially private, that is $f({\mathcal{M}})$ is also a
$(\epsilon,\delta)$-differentially private mechanism.

This means that we can use the output of a DP mechanism in any other process
and the final result will not reveal more than what was already revealed by
the DP-mechanism.

Group privacy

: According to [26], if we would like to protect databases differing in $c$
rows (that is, avoiding that an attacker with auxiliary information can deduce
if the dataset contains the information of $c$ participants), we only need to
bound the privacy value as $\epsilon c$ instead of $\epsilon$, in this case,
for $d_{1}$, $d_{2}$ differing on $c$ items:

$Pr[M(d_{1})\in X]\leq e^{\epsilon c}Pr[M(d_{2})\in X]+\delta$

So, every group of $c$ items is $\epsilon$-differentially private protected
and each item is $\frac{\epsilon}{c}$-differentially private protected.

These features combined mean DP is a comprehensive framework for privacy whose
tools allows: 1) measurement of the privacy guarantee that a system provides
and 2) understanding of its interaction with other DP and non-DP mechanisms.
Intense research has been directed towards defining other types of DP
mechanisms and/or describing variants that consider other elements besides
$epsilon$, to define more realistic or useful DP-mechanisms.

##  3 Differential privacy mechanisms

We have previously described Laplace mechanisms, but there are other
probability distribution functions that have been proved useful as
differential privacy mechanisms, such as:

  * •

Gaussian mechanism: Very similar to Laplace mechanism, but it adds Gaussian
noise instead of Laplacian noise, and it needs parameter $\delta>0$ (which is
an acceptable assumption). Although the general accuracy of gaussian mechanism
is lower than the Laplacian, it has gained great attention, for example in ML
applications, since L1 or L2 norm are both acceptable sensibility functions,
and the noise it produces at the point of the true value is lower than in the
Laplace case, which improves the accuracy of the mechanism if the sensibility
is low [26, 27]

  * •

Geometric mechanism: This mechanism draws noise from the double-geometric
distribution, which is the discrete version of the Laplace distribution. It
only protects count queries, but the specialization improves performance over
the more general Laplace mechanism [28]

  * •

Uniform mechanism: The uniform noise distribution is near optimal and it can
be seen as a special case of the bounded Laplace mechanism in a regime with
$\epsilon=0$. [29, 30]

  * •

Binary mechanism: Given a binary input value, the mechanism randomly decides
to flip the binary value or not, in order to satisfy differential privacy
[31].

  * •

Exponential mechanism: Achieves differential privacy on numerical and
categorical inputs, by randomly choosing an output value for a given input
value, with greater probability assigned to the values that are closer to the
input, as measured by a given utility function [32].

  * •

Bingham mechanism: Uses Bingham distribution for estimating the first
eigenvector of a covariance matrix [33].

A large survey of applications of DP in multiple industry setup scenarios can
be seen at [34]. and notable implementations of the above mechanisms (and the
variations explained in Section 4) are available at:

  * •

https://github.com/IBM/differential-privacy-library

  * •

https://github.com/google/differential-privacy

  * •

https://github.com/yuxiangw/autodp, and

  * •

https://github.com/opendp/smartnoise-core

  * •

https://github.com/tensorflow/privacy

DP is not hard to use, and the data holder can maintain the original data but
share analysis over these data with a certain privacy guarantee, that is
controlled according to the DP parameters. However, the DP results have a
reduced utility with respect to using a non DP procedure. Selecting the right
parameters to compensate the trade-off between privacy and accuracy is a tough
problem. Finally, due to the dimensionality curse, the more data needs to be
freed in a DP fashion, the more difficult this trade-off become. Therefore, a
lot of research has been done to define mechanisms or scenarios whose
conditions allow to identify tighter privacy loss bounds.

##  4 Other variants of Differential privacy

There are more than 200 variants and extensions of DP definitions in the
literature, each of them focused on different scenarios, datatypes and
attacker models. A brilliant taxonomy of these variants with formal
definitions, axioms and theorems that relate to them can be found at: [18]. In
this work the DP definitions are partitioned into the following seven
categories depending on which features of the original definition were
modified:

Quantification of privacy loss

: Whilst the original definition of privacy loss (that considers only the
parameter $\epsilon$) models the risk of the worst-case scenario, additions
such as considering a small probability error $\delta$ are more natural and
offer better composition properties. Particular attention has been given to
averaging the privacy loss (using other measures of divergence obtained by the
privacy mechanism, such as Kullback-Leibler or the Renyi-divergence [24]
222Renyi-DP strong interpretation along with the well-suited composition of
heterogeneous mechanisms have made Renyi-DP one of the most popular frameworks
for privacy. See https://www.johndcook.com/blog/2018/11/21/renyi-differential-
privacy/ for a concise description of the Renyi-DP.). It is also worth
mentioning the case of controlling the privacy loss of the tail of the
distributions, in which both average and rare cases are private-preserved.

Neighbourhood definition

: these variants are based on changing the definition of sensitivity, e.g. by
increasing or reducing the number of rows that two adjacent datasets can
differ on, adding a policy that specifies which records are sensitive, or
applying the definition to a particular datatype.

Variance of the privacy loss

: in DP $\epsilon$ is applied uniformly, but in practice some users require
more protection than others. Some definitions allow the privacy loss to vary
across users either explicitly (using an user acceptance level) or implicitly
(by averaging the risk or leaving some users at risk).

Background knowledge

: normally, it is assumed the attacker to have full knowledge of the dataset,
and their only uncertainty is if a user belongs to the dataset or not. Instead
of using adjacent datasets, these types of DP consider the existence of
certain probability distributions on the data that is used to simulate the
attacker’s knowledge and restrict the privacy guarantees. This is an active
area of research, but not currently used in practice as there is no guarantee
about the limited attacker background knowledge.

Change in formalism

: other statistical formalisms have been proposed, for example using
hypothesis testing where DP is interpreted as the probability of an attacker
to know that a particular outcome comes from a dataset or its neighbourhood
dataset. This type of formalism have evolved on the f-differential privacy,
that guarantees the composition of any mechanisms and provides an exact
formula for composition, therefore gaining a lot of practical attraction in
the SOTA DP-libraries.

Relativization of knowledge gain

: beyond the probabilistic information bounded by DP, information can be
leaked in other ways. These other types of information leakage can be modelled
using auxiliary functions or algorithms.

Computational power

: these variants consider that the computational power an adversary has can
lead to a different privacy guarantee level. Limited computational power is a
reasonable assumption, but for a large class of queries or algorithms, it
cannot provide significant benefits over classical DP in a typical client-
server setup, thus, existing works are focused on federated settings [35].

Some works extend the concept of DP based on a mix of the above categories or
scenarios. They are also discussed in [18].

##  5 DP techniques in ML

Thanks to the post-processing feature of DP, we can apply DP techniques at
various stages of the ML process and still achieve a differentially private
final model. DP can be applied to: 1) the datasets during data preprocessing,
2) optimization algorithm whilst training the model parameters, 3) the result
of the loss function just before updating parameters or 4) the final trained
parameters. Surveys about DP applied to classical ML algorithms (such as Naive
Bayes, Linear Regression, classification, Support Vector Machines,
dimensionality reduction and time series) can be found in [36], and [37].

In this section we will focus on general methods that can be used in a broader
list of ML algorithms instead of a particular one. In Section 5.1, the
extensions of ML optimization algorithm stochastic gradient descent (SGD) and
the ensemble techniques to convert them into DP mechanisms, given way to DP-
SGD and the ensemble-DP framework, which have led to Private Aggregation of
Teacher Ensembles (PATE), a well recognised DP framework in ML. Then, in
Section 5.2 we introduce a second path to obtain privacy in ML, through the
use of synthetic data instead of real data and its solution by using
Generative Adversarial Networks (GAN) and combining it with the PATE algorithm
(PATE-GAN).

###  5.1 Ensemble-DP and DP-SGD

One of the first works towards this direction appears in [20]. In this paper,
the authors propose an ensemble-DP framework. It works by first applying a
strong privacy-preserving mechanism to the dataset and then using an ensemble
classifier on top of the perturbed data. It introduces an association between
data utility and increasing the number of weak decision tree learners, where a
combined adjustment of the privacy parameters and an increase in the number of
weak learners in the ensemble compensated for the increase in classification
error after application of DP. Multiple DP-ML applications rely on this
framework, due to its simplicity.

Since most ML algorithms optimize an objective function that minimizes the
learning error using stochastic gradient descendent (SGD), and given that SGD
is inherently a random mechanism, creating a differentially private SGD is a
good foundation for the task. DP-SGD was formulated in [19] and we have
reproduced in Algorithm 1. As with SGD, the algorithm begins randomly
assigning parameters $\theta$ of the model and then iteratively updating them
according to the loss function $\mathcal{L}(\theta)$. The difference is that
in the DP-SGD, at each iteration the gradient is computed for a random subset
of examples, then clipped with the L2 norm of the gradients and a gaussian
noise is added to protect privacy. The number of iterations, $T$, satisfies
$T\gg N/B$, where $N$ is the number of examples and $B$ is the mini-batch
size, to guarantee each sample is used multiple times during the training, .

1

Input:

  * •

Training data: $\mathcal{D}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{N}\\}$

  * •

Loss function:
$\mathcal{L}[\theta]=\frac{1}{N}\sum_{i}\mathcal{L}(\theta,\mathbf{x}_{i})$

  * •

learning rate; $\eta_{t}$

  * •

noise scale: $\sigma$

  * •

batch size: $B$

  * •

gradient norm bound: $C$

Output: Final parameters: $\theta_{T}$

2 Initialize $\theta_{0}$ randomly

3for  _$t=1,\dots,T$_ do

4

5 Randomly sample a batch $\mathcal{D}_{t}\subset\mathcal{D}$ with sampling
probability $B/N$

6 Compute the gradient
$\mathbf{g}_{t}[\mathbf{x}_{i}]=\nabla_{\theta}\mathcal{L}[\theta_{t-1},\mathbf{x}_{i}]$
for each $\mathbf{x}_{i}\in\mathcal{D}_{t}$

7 Clip the gradients:
$\mathbf{g}_{t}[\mathbf{x}_{i}]\leftarrow\frac{\mathbf{g}_{t}[\mathbf{x}_{i}]}{\max\left[1,\frac{\|\mathbf{g}_{t}(\mathbf{x}_{i})\|}{C}\right]}$
(1)

8 Add noise
$\xi_{i,t}\sim\mbox{Norm}_{\xi_{i,t}}[\mathbf{0},\sigma^{2}C^{2}\mathbf{I}]$
$\mathbf{g}_{t}[\mathbf{x}_{i}]\leftarrow\mathbf{g}_{t}[\mathbf{x}_{i}]+\xi_{i,t}$
(2)

9 Average the clipped, noisy gradients
$\bar{\mathbf{g}}_{t}=\frac{1}{B}\sum_{x_{i}\in{D}_{t}}\mathbf{g}_{t}[\mathbf{x}_{i}]$
(3)

10 Take a step in the gradient direction
$\theta_{t}\leftarrow\theta_{t-1}-\eta_{t}\bar{\mathbf{g}}_{t}$ (4)

11

The overall privacy cost $(\epsilon,\delta)$ can be computed by using a
privacy accounting method

Algorithm 1 DP-SGD: Differentially-private stochastic gradient descent

Notice that the algorithm is expressed in function of a noise scale, and the
value of $\epsilon$ is computed by using the moment accountant method that was
described in the same paper to keep a better track of the privacy loss.
Besides, for a fixed level of differential privacy, there is a connection
between the amount of noise $\sigma$ and the amount of clipping $C$. A larger
value of $C$ does less clipping, but requires more noise to retain the same
degree of privacy (compensating the effect of sensitive values). Specifically,
it was proved that the Algorithm 1 is $(\epsilon,\delta)$-DP when using
$\sigma=\Omega(q\frac{\sqrt{Tlog(1/\delta)log(T/\delta)}}{\epsilon})$, being
$q=B/N$, sampling probability.

An important outcome of this paper is that in their experiments, DP-SGD
improved generalization performance and that the gap between training and
testing accuracy tended to be smaller, particularly for small privacy budgets.

Another paper, [21], mixes the ideas in the ensemble-DP framework and DP-SG in
a proposal known as the PATE algorithm. The PATE framework basically allows a
public model to learn by noisily ensembling the predictions of multiple
models. In this setup it is assumed that there is a private dataset
$\mathcal{D}$, a private model $PrM$ and a public unlabelled dataset
$\mathcal{UD}$. The private dataset is split in multiple parts to produce
multiple teacher models $f_{i}$. An ensemble model, called aggregate teacher
$f$, is defined as noisy voting of the teacher models and then, used to
produce a private labelled dataset $\mathcal{D^{\prime}}$ from the public
unlabelled one. $\mathcal{D^{\prime}}$ is then used to train a public model
that is called the student model. Notice that 1) the labels produced by the
aggregate teacher are never revealed, nor are the parameters of the teacher
models; 2) aggregate model as well as the labels are differentially private
thanks to the added noise, but the privacy budget of the model is limited and
possibly expended during the predictions; 3) the produced differentially
private dataset can now be used to generate any student model that is also
differentially private by the post-processing property of DP. A general
overview of the whole data process in PATE Algorithm is described in Figure4.
Details are provided in Algorithm2. PATE333Code available at:
https://github.com/tensorflow/privacy/tree/master/research/pate_2018. and the
following improvements in [38] have shown that this framework produces higher
test set accuracy with lower privacy budgets.

Figure 4: General architecture and data processing of PATE algorithm

1

Input:

  * •

Training data: $\mathcal{D}$

  * •

Number of splits: $N$

  * •

Private model: $PrM$

  * •

Public model: $PuM$

  * •

Public unlabelled data: $\mathcal{UD}$

Output: Final parameters: $\theta_{T}$ of $PuM$

2 Split randomly the dataset $\mathcal{D}$ in $N$ parts,
$\mathcal{D}=\\{\mathcal{D}_{1},\dots,\mathcal{D}_{N}\\}$

3for  _$i=1,\dots,N$_ do

4 Generate $f_{i}$ by training $PrM$ with the subset $\mathcal{D}_{i}$

5

6Generate the aggregate teacher model, a noisy ensemble voting model:
$f[x]=\underset{j}{\operatorname{argmax}}\left[\sum_{i}[f_{i}[x]=j]+\xi\right]$,
with $\xi\sim Lap[\epsilon^{-}1]$

7Produce $\mathcal{D^{\prime}}$ a private labelled dataset by using $f$ to
label $\mathcal{UD}$

8Use $\mathcal{D^{\prime}}$ to train $PuM$. Let $\theta_{T}$ be the trained
parameters of $PuM$

Algorithm 2 PATE: Private Aggregation of Teacher Ensembles

This algorithm showed another direction in private ML; instead of creating
differentially private models, we could produce differentially private
datasets that can be used to train differentially private models.

Yet another direction in private ML is model creation based trained with
synthetic data instead of using real data. Synthetic data generation systems
aim at mirroring the statistical distributions of the original data without
revealing information of any particular data instance. But, generating private
synthetic data is known to be hard in the worst case
https://eccc.weizmann.ac.il/report/2010/017/ and therefore it is necessary to
use techniques such as Generative Adversarial Network (GAN), a type of NN
designed to simulate (or recreate) noisy examples that follow the patterns of
an input dataset. This is the topic of the following section.

###  5.2 DP GAN models: DP-GAN and PATE-GAN

The general architecture of a GAN model is shown in Figure 5. It is composed
of two Neural Networks (NN): the generator and the discriminator. The
generator model $G(z,\zeta$) is a network with parameters $\zeta$ that
generates samples similar to real examples $x$ from a random vector $z$; the
discriminator model is a network with parameters $\phi$ that seeks to classify
samples (those artificial generated by the generator model and the real ones
$x$) as real or fake. The two models are trained together using SGD: the
discriminator uses the loss function in Eq. 5 to get better at discriminating
real and fake samples, and the generator uses the loss function in Eq 6 to
learn to produce examples able to fool the discriminator. Once this two-player
game has converged (when the discriminator model obtains a probability close
to 0.5 for each fake/real class), samples can be generated by $G(Z,\theta)$
from random vectors $z$.

Figure 5: General architecture of a GAN

$\max_{\phi}\left[\mathbb{E}_{x}[\log D(x,\phi)]-\mathbb{E}_{z}[\log
D(G(z,\theta),\phi)]\right]$ (5)

$\min_{\theta}\left[-\mathbb{E}_{z}\left[\log
D(G(z,\theta),\phi)\right]\right],$ (6)

DP-GAN [39] applies DP-SGD method during the GAN training. The performance of
DP-GAN is relatively poor in practice, but performance has been recently
improving (see e.g. [40], [41]).

An extension of PATE for training GAN models which is known as PATE-GAN444See
https://github.com/BorealisAI/private-data-generation for an open-source
toolbox that implements PATE-GAN and several other methods for private data
generation was proposed in [42] and is described in Algorithm 3. The
algorithms privately learn to discriminate fake from real labels using both
private and generated data from the public generator. Then, teacher
discriminators are ensembled to obtain the aggregate teacher; in turn, the
aggregate teacher will use part of its private budget to predict the labels of
a new public generated dataset. Consequently, the public generated dataset and
the new private labelled dataset are merged and shuffled to be used to train
the public discriminator and generator. So, the whole PATE-GAN algorithm works
by alternatively generating datasets (with the generator model) and use them
to a) train the aggregate teachers and b) update the discriminator by mixing
them with privately labelled datasets.

1

Input:

  * •

Training data: $D\cup D_{\mathcal{G}}$ where $D_{\mathcal{G}}$ is a dataset
generated by using the public generator model $\mathcal{M}_{\mathcal{G}}$

  * •

Number of splits: $N$

  * •

Private model: $\mathcal{M}_{pr}$

  * •

Public GAN model with discriminatory model $\mathcal{M}_{\mathcal{D}}$; and
generator model $\mathcal{M}_{\mathcal{G}}$

Output: Final parameters: $\theta$ of $\mathcal{M}_{\mathcal{G}}$ and $\phi$
of $\mathcal{M}_{\mathcal{D}}$

2 Split randomly the dataset $\mathcal{D}\cup\mathcal{D_{G}}$ in $N$ parts,
$\\{\mathcal{D}_{1},\dots,\mathcal{D}_{N}\\}$

3for  _$i=1,\dots,N$_ do

4 Generate $f_{i}$ by training $\mathcal{M}_{pr}$ with the subset
$\mathcal{D}_{i}$

5

6Generate the aggregate teacher model, a noisy ensemble voting model:
$f[x]=\underset{j}{\operatorname{argmax}}\left[\sum_{i}[f_{i}[x]=j]+\xi\right]$,
with $\xi\sim Lap[\epsilon^{-}1]$

7Produce a public dataset, $\mathcal{D^{\prime}}$, using the generator model
$M_{\mathcal{G}}$

8Produce $\mathcal{D^{\prime\prime}}$, a private generated dataset by using
$f$ to label $\mathcal{D^{\prime}}$

9Use $\mathcal{D^{\prime}}\cup\mathcal{D^{\prime\prime}}$ to train
$\mathcal{M}_{\mathcal{D}}$ and $\mathcal{M}_{\mathcal{G}}$

10if  _train has not ended 555if $\mathcal{M}_{\mathcal{D}}$ and
$\mathcal{M}_{\mathcal{G}}$ have converged or a maximum number of steps have
been reached._ then

11 Generate another $D_{\mathcal{G}}$

12 Go Step 1

13

14Let $\theta$ and $\phi$ be the parameters of public GAN model.

Algorithm 3 PATE-GAN

Figure 6: General architecture and data processing of PATE-GAN algorithm

To evaluate the algorithm, the authors used the PATE-GAN algorithm to generate
a dataset, and used this synthetic dataset to train a model, and assesses the
quality if the PATE-GAN model using a real test dataset. PATE-GAN was
significantly better than DP-GAN given the same privacy budget (although it
was worse than training directly on the real data for some datasets), and only
slightly worse than using the same method with a non-differentially private
GAN. Therefore, it seems that the performance gap is caused by the GAN model
itself and not from the DP mechanism. This constitutes an encouraging result
in the area of DP and synthetic data generation.

Whilst the majority of this work has been applied to image processing with
high success [43], [44], [45]666See here: https://github.com/nashory/gans-
awesome-applications for a curated list of GAN applications less work has been
done in Natural Language Processing (NLP).

However, recent advances in this area are encouraging. Most of the work, [46,
47, 48, 49, 50, 51] follows the proposal from [46] of mixing GANs with
reinforcement learning, where the discriminator is trained to distinguish real
from generated text samples and to provide rewards for the generator, which in
turn is optimised via policy gradient. Rewards can contain also information
about novelty and diversification of the texts such as in [50]. The
instability of the reinforcement learning on discrete data have been addressed
in [51] (by training the generator using maximum likelihood estimators and
changing the discriminator rewards more those samples near the word
distribution of real texts) and their system shown a significant improvement
on with respect to the previous works.

A new paradigm in [52] challenges the generation of long and diverse texts by
the use of Variational Modelling. Their proposal follows the mechanisms of
natural writing: first, planing a script and then filling it up with sentences
that are generated upon the resulting script and to consider the previously
generated context. This proposal not only outperforms state-of-the-art
baselines for long texts but also provides an interesting efficient approach,
”decomposing long text generation into dependent sentence generation sub-
tasks“.

##  6 Local differential privacy

One problem with differential privacy is that even if the statistics obtained
over datasets are noised by a trusted server to protect users details, the
real values are still saved in the datasets themselves, and therefore users
have to trust the database maintainer to keep their privacy [53].

A stronger privacy guarantee can be offered for individual users in the local
differential privacy (LDP) setting where: 1) there is no need of a centralized
authority and 2) each user can encode and perturb their inputs before
transmitting them to the unstrusted server. The challenge with this setting is
related to managing the overall variance of the perturbed input data the
server receives, that in fact depends on the number of participants. So, most
of the work on LDP has being applied to counting problems (training models in
a LDP fashion is being addressed as Federated Learning with untrusted server
as described in Section 7):

Frequency Oracles

: Estimation of locally private frequency, i.e. given a domain $\mathcal{D}$,
a frequency oracle protocol estimates the frequency of an element
$d\in\mathcal{D}$. For example, we could collect how many females are in a
dataset without revealing the gender of a specific user. An abstract frequency
oracle protocol and multiple specific implementations on top of it are
described in [54], which also contains several improvements in accuracy over
well-known protocols as RAPPOR from Google.

Heavy hitting identification

: In the case of large domains, instead of estimating the frequency of all of
the elements $d\in\mathcal{D}$, we could estimate the frequency of the common
domain elements (the heavy hitters). Efficient algorithms, such as TreeHist
and Bitstogram are described in [55].

Itemset mining

: Instead of computing frequency of single-valued inputs, this problem deals
with the itemset mining in a local differential private settings. Specific
algorithms can be found in [56] and [57].

Private spatial data collection

: Estimate the spatial distribution of users without revealing/saving the
specific location of each user at a given time. A framework for this problem
has been proposed in [58].

A few libraries implementing these types of LDP problems are available at:
https://github.com/vvv214/LDP_Protocols (all of the above methods),
https://github.com/Samuel-Maddock/pure-LDP (for frequency oracle and heavy
hitting) and https://github.com/sisaman/LDP-mechanisms/blob/main/mechanisms.py
(for general types of mechanisms).

How to combine LDP with the classic centralized DP as well as the application
of ML on the top of LDP (and Deep Learning in particular) are still open
questions, however a few steps towards these directions have been provided,
especially in a federated context [59].

##  7 Privacy in Federated Machine Learning

Traditional ML uses the centralized approach of sending a full dataset to the
datacenter or machine where the learning will take place. In the Federated
Learning (FL) setup, model owners send their models to the data holders, so
the data never leaves the data holder premises, and a global model is trained
from the parameters learned locally. This setup is more secure (and some
authors classify FL as a privacy-preserving methodology), but still there are
several types of attacks that can compromise data privacy (see section 7.1 for
more information about them).

There are three main flavours of FL:

Centralized FL

: Where there is a central server that orchestrate the different steps of the
algorithm, coordinate all the participating nodes during the learning process
and generates a global model. This setup can lead to a single point of failure
in the aggregator side. To minimize privacy concerns local models are only
sent to a trustable aggregator, in a framework known as Trustable FL (TFL).

Decentralized FL

: Where the nodes are able to coordinate themselves to obtain the global
model. This setup prevents single point failures but orchestration is more
difficult and performance may be affected by the network topology. Rollout and
blockchain-based federated learning frameworks are examples of decentralised
FL.

Heterogeneous FL

: where there are no assumptions about the independence and distribution of
the data, types of devices, collaborative schemes and models interacting in
the learning, but all these resources are optimized to produce a single global
model. Please, refer to [60] extended explanation and examples.

In the following section we describe the most important types of attacks to FL
systems and in sections 7.2 and 7.3, we will describe practical examples of
privacy solutions in the centralized and decentralized FL using DP777Other
privacy-preserving techniques described in the literature such as Secure
Multiparty Computation (SMPC) and Homomorphic Encryption (HE) are out of the
scope of this review.

###  7.1 Attacks in Federated Learning

(Mainly taken from [61])

Attackers in FL aim to identify the underlining training data or trigger a
miss-classification on the final model. Many of them have been identified and
coined [61, 62, 63] as:

Model Inversion

: The aim is to reconstruct the training data. A potential attacker with
access to target labels and model query, can query the final trained model to
obtain the classification scores and reconstruct the rest of the data.

Membership Inference

: It uses similar strategy as the Model Inverse attack to identify if some
data were part of the training dataset.

Model Encoding

: The attacker with access to the model tries to identify training data. In
white-box models, the attacker uses the explainability/memorization power of
the model to reveal training data; in black-box models, the attacker overfits
the model to leak part of the target labels.

Model stealing

: Malicious participants could create a second model that mimics the decisions
of the original model, and therefore avoid fee payment to the original ML
experts or even sell the model to third parties.

Model poisoning

: Malicious participants submit a local model that triggers specific results
given specific inputs, for example modifying a classifier that assigns an
attacker-chosen label to certain features of the data.

Data poisoning

: Malicious participants introduces “bad” training data, making the model
accuracy drop.

Adversarial examples

: The attacker does not require access to the training procedure, it tries to
trick the model in order to falsely classify a prediction. A potential
attacking is a malware that evades the detection of a ML intrusion detection
system.

The three most important techniques for preventing these attacks are: 1)
applying differential privacy to the parameters of the local models, so they
do not leak any personal data; 2) using Secure Multiparty Computation (SMPC),
in which gradients and parameters are cryptographically split amongst the
participants of the computation and all the data processing is done in a
decentralised manner; 3) Homomorphic Encryption (HE), in which the ML training
is done over encrypted data.

Whilst DP is a non-expensive technique, it produces a decrease in accuracy;
SMPC does not protect during testing phase of ML; and HE is a promising method
but with a very high computational cost which is not tolerable in real-world
situations.

Many other defence methods are available in the literature (see [61] for a
comprehensive list of them), but unlike DP, SNPC and HE, they are tailored to
specific types of attacks and therefore less useful in a practical and more
broad setups.

###  7.2 Privacy in Centralized FL using DP

Most works in centralized FL with DP are focused on obfuscating gradient
variations on each of the clients. Here we discussed three alternatives
described in the literature:

Random subsampling and distorsion

: In [64] the authors propose a DP mechanism in which DP is applied on the
aggregator side by using a randomised mechanism and distortion of the updates.
The random subsampling consists of creating rounds that involve only a random
subset of clients updating the model and submitting the updated gradient;
distortion is done by using a Gaussian mechanism that clips the updates and
add noise, both depending on the sensitivity of the updates. Their results
show that at early communication rounds a small number of clients can
contribute, but in later rounds the number of users involved needs to increase
to gain accuracy. In setup with reduced number of users Gaussian mechanism
cannot be applied (due to trade off between noise-utility) and also, random
subsampling may not obfuscate the information about the involved clients in
each round.

Random masks and adaptative quantization

: Parameter quantization has been widely used for DNN model compression for
both training and testing [65]. In [66] it is used to produce local
differential privacy in the sense that each client (in this case, a sensor)
transmit DP parameters. The DP-parameters can be obtained combining random
masks with adaptative quantization to reduce the number of parameters sent to
the aggregators, and improve the privacy leakage from the gradients. Random
mask mechanism simply generates a sparse matrix from the gradient matrix, by
replacing portion of random set of the gradients by zero. Adaptative
quantization compresses each weight matrix in the DNN, $W$, based on one-bit
adaptive quantization, that is $W_{i,j}\sim Poisson(p_{i,j})$,
$p_{i,j}=\frac{W_{i,j}-min_{i,j}W}{max_{i,j}W-min_{i,j}W}$. This paper does
not relates these heuristics with DP concepts, but it is very difficult to
reconstruct a $W$ matrix from the quantized one and it can be considered as an
alternative to DP mechanisms that at the same time solve communication
problems for large models. Their results over a LSTM network are encouraging
as a very similar accuracy is obtained using or not adaptative quantization.

Smashed data and gradient sign

: In [67] the training is divided in two phases. In a first phase, devices
learn compact but informative representations of the raw data locally, that
they call smashed data. In the second phase, the shared global model uses
these smashed data as input, rather than the raw data. Parameters are not sent
to the server, instead, a 1-bit sign of their gradient is sent. The server
then aggregates the signs and send them back to the clients. The global model
is aggregated locally on the server based on the global loss and the local
loss sent from the clients.

Privacy in Federated Learning can be further boosted by using LDP during
training as done in [68]888 https://github.com/lowya/Locally-Differentially-
PrivateFederated-Learning where the authors have trained a Logistic regression
model in a Federated fashion noising the local models.

###  7.3 Privacy in Decentralized FL using blockchain

Recently, private decentralized FL has been addressed using blockchain
technology. Two of the more interesting approaches are described in [69] and
[61]:

  * •

BlockFLow is described in [69] as an accountable federated learning system
that is fully decentralized and privacy-preserving. It incorporates
differential privacy, introduces a novel auditing mechanism for model
contribution, and uses Ethereum smart contracts to incentivize good behaviour
and keep all of the interactions during training. Their system does not
require a centralized test dataset, sharing of datasets directly between the
agents, or trusted auditors; it is fully decentralized and resilient up to a
50% collusion attack in a malicious trust model. “Each agent must run their
own instance of the BlockFLow client, which handles all aspects of an agent’s
participation in a federated learning experiment. For every federated learning
round, each BlockFLow client trains a local model, applies differential
privacy by adding Laplacian noise to the model, shares its model with the
other clients in the experiment by using IPFS, retrieves and evaluates other
clients’ models, reports the evaluation scores to the BlockFLow smart
contract, and retrieves the overall scores and averages the clients’ models.”.
Contribution scoring procedure described in the paper penalizes contributions
with malicious models, promotes the submission of strong contributor models
honesty during evaluation. The major drawback of this solution is the overhead
time due to all of the cryptographic operations involved during the training
process (when a data/model is retrieved or saved to IPFS or during the
evaluation of other agent models).

  * •

In [61] the authors use the DIDComm messaging implementation available in
Hyperledger Aries, a toolkit designed “for initiatives and solutions focused
on creating, transmitting and storing verifiable digital
credentials”999https://www.hyperledger.org/use/aries to create a secure
communication channel to execute FL. Hyperledger Aries blockchain maintains
all the peer’s DIDs and DID documents101010https://www.w3.org/TR/did-core/ and
emits credentials that can be used to execute the services described in the
DID documents. In their proposal, ML practitioners, Regulators and
organizations (their use case is around Medical Health Records and therefore
organizations are mainly formed by NHS Trust agents) are connected to a
Hyperledger Aries network and when a FL process is requested, their protocol
allows all the peers to exchange and verify DIDs and establish trusted
connections. Once all trusted connections are established, a roll up FL
algorithm can be executed.

##  8 Discussion

The two more popular DP ML algorithms (DP-SGD and PATE) were discussed: the
first one re-structures SGD to add noise to the gradients conveniently during
training so model parameters do not leak any private information; the second
completes a public unlabeled dataset with differentially private labels
produced from a noised ensemble model and uses this dataset, in turn, to
create a DP model.

Ideas of the PATE algorithm resembles synthetic data generation, and in fact,
it is one of the approaches to avoid leaking private data during data
processing. However, statistical approaches for synthetic data generation have
proven to be very difficult and GAN models can be used to produce better
results. Merging the ideas of these two algorithms, PATE-GAN is intended to
produce synthetic data by integrating a PATE-kind discriminator to generate DP
data which in turns can be use to produce a DP GAN model.

Local Differential Privacy can be used in scenarios where the original
sensitive data do not need and/or can not be centrally stored and a DP version
of these data can be used instead. LDP applications are very difficult to
configure and optimise due to the disparities of the noise added at the
different locations to preserve individualities. So, applications heavily
relying on counters are typical use cases of LDP, but LDP has been also used
in federated contexts to avoid sharing the real gradients or parameters with
other parties in both centralized and decentralized frameworks.

Solutions in decentralised frameworks are mainly focused on the integration of
DP with a blockchain technology. This symbiosis implies a high latency during
processing; however, it brings, amongst other advantages, a high level of
privacy guarantee as the operational traceability in blockchain prevents any
attack without revealing the attacker identity and therefore prevents the
attack happening in the first place.

Figure 7: Guideline for applying DP during in data processing.

Figure 7 offers a guideline for applying DP in practice. The first step is to
determine which data and data processes are susceptible to having privacy
issues. The data practitioner should consider that in the local vs centralised
setup, the original, sensitive data should not be centrally saved, and that
the noise applied to the data at each location could be different and
therefore affect the utility of the overall processing. As DP could be applied
at different stages of data processing, the data used in a DP-mechanism will
be DP-protected as long as 1) only the outcome of a DP process is revealed
along with the DP parameters and 2) downstream processes uses the outcome of
this DP process without accessing the original source.

The next step is to chose a maximum allowed $\delta$ (risk probability due to
uncontrolled causes). Data processing is next decomposed so that a DP-
mechanism can be applied to each component. The DP-mechanisms used should be
chosen according to their suitability for the task. Then, the final composed
value of $epsilon$ should be computed and checked against the privacy
agreements with the users. Finally, the selected DP-mechanisms can be applied,
and the results of the processing returned along with the DP-parameters
$(\epsilon,\delta)$.

To enhance privacy the full data processing described in Figure 7 could be
embedded in a blockchain application and all privacy parameters along with
metrics to encourage collaboration and honest behaviour could be,
consequently, saved on a blockchain.

##  9 Conclusions

This paper has summarised the most important concepts about Differential
Privacy. First, a deep understanding of the DP definition and their variants
was presented. We highlighted the importance of variants as Reny-DP and f-DP,
specially for ML applications, and listed the libraries where available
implementations can be found.

The relation of DP with synthetic data generation, in particular with GAN
methods, as well as the difference between local and centralised DP were all
described here and relevant works were detailed.

Finally, we offer the reader a practical guideline for applying DP.

##  10 Acknowledgements

The author would like to thank Dave Lewis, Clinton Swan and Rahul Sharma whose
useful review and comments helped to significantly improve this paper’s
presentation.

## References

  * [1] Alan Westin.  Privacy and freedom, 1967.
  * [2] Priyank Jain, Manasi Gyanchandani, and Nilay Khare.  Big data privacy: a technological perspective and review.  Journal of Big Data, 2016.
  * [3] Oleksandr Tomashchuk, Dimitri Van Landuyt, Daniel Pletea, Kim Wuyts, and Wouter Joosen.  A data utility-driven benchmark for de-identification methods.  In Stefanos Gritzalis, Edgar R. Weippl, Sokratis K. Katsikas, Gabriele Anderst-Kotsis, A Min Tjoa, and Ismail Khalil, editors, Trust, Privacy and Security in Digital Business, pages 63–77, Cham, 2019. Springer International Publishing.
  * [4] M. B. Hawes.  Implementing differential privacy: Seven lessons from the 2020 united states census, 2020.
  * [5] Sarah Bird.  Introducing the new differential privacy platform from microsoft and harvard’s opendp, 2020.
  * [6] J. Greig.  Google hopes to protect users with open source differential privacy library., 2019.
  * [7] Enabling developers and organizations to use differential privacy, 2019.
  * [8] Covid-19 community mobility reports, 2020.
  * [9] N. M. Johnson, J. P. Near, and D. S. Xiaodong.  Practical differential privacy for SQL queries using elastic sensitivity.  CoRR, abs/1706.09479, 2017.
  * [10] Facebook outlines new differential privacy framework to protect user information in shared datasets, 2020.
  * [11] A. Dow, A. Herdağdelen, P. Mohassel, A. Pompe, and B. State.  Protecting privacy in facebook mobility data during the covid-19 response.
  * [12] T. Diethe.  Preserving privacy in analyses of textual data.
  * [13] O. Feyisetan, B. Balle, T. Drake, and T. Diethe.  Privacy- and utility-preserving textual analysis via calibrated multivariate perturbations.
  * [14] et al. Rogers, R.  Linkedin’s audience engagements api: A privacy preserving data analytics system at scale.
  * [15] V. Pihur, A. Korolova, F. Liu, S. Sankuratripati, M. Yung, D. Huang, and R. Zeng.  Differentially-private “draw and discard” machine learning, 2018.
  * [16] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.  Calibrating noise to sensitivity in private data analysis.  In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, pages 265–284, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  * [17] Daniel Reusche and Nicolás Della Penna.  Towards prior-free approximately truthful one-shot auction learning via differential privacy, 2021.
  * [18] Damien Desfontaines and Balázs Pejó.  Sok: Differential privacies - a taxonomy of differential privacy variants and extensions, 2020.
  * [19] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * [20] Kato Mivule, Claude Turner, and Soo-Yeon Ji.  Towards a differential privacy and utility preserving machine learning classifier.  Procedia Computer Science, 12:176–181, 2012.
  * [21] Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar.  Semi-supervised knowledge transfer for deep learning from private training data, 2017.
  * [22] Aaron Roth Cynthia Dwork.  The Algorithmic Foundations of Differential Privacy.  2014\.
  * [23] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor.  Our data, ourselves: Privacy via distributed noise generation.  In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 486–503. Springer, 2006.
  * [24] Ilya Mironov.  Rényi differential privacy.  2017 IEEE 30th Computer Security Foundations Symposium (CSF), Aug 2017.
  * [25] Frank McSherry.  Privacy integrated queries.  In Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data (SIGMOD). Association for Computing Machinery, Inc., June 2009.  For more information, visit the project page: http://research.microsoft.com/PINQ.
  * [26] Cynthia Dwork.  Differential privacy.  In 33rd International Colloquium on Automata, Languages and Programming, part II (ICALP 2006), volume 4052 of Lecture Notes in Computer Science, pages 1–12. Springer Verlag, July 2006.
  * [27] Naoise Holohan and Stefano Braghin.  Secure random sampling in differential privacy, 2021.
  * [28] Arpita Ghosh, Tim Roughgarden, and Mukund Sundararajan.  Universally utility-maximizing privacy mechanisms.  In Proceedings of the Forty-First Annual ACM Symposium on Theory of Computing, STOC ’09, page 351–360, New York, NY, USA, 2009. Association for Computing Machinery.
  * [29] Quan Geng and Pramod Viswanath.  Optimal noise adding mechanisms for approximate differential privacy, 2013\.
  * [30] Quan Geng, Wei Ding, Ruiqi Guo, and Sanjiv Kumar.  Privacy and utility tradeoff in approximate differential privacy, 2019\.
  * [31] Naoise Holohan, Douglas J. Leith, and Oliver Mason.  Optimal differentially private mechanisms for randomised response.  IEEE Transactions on Information Forensics and Security, 12(11):2726–2735, Nov 2017.
  * [32] Frank McSherry and Kunal Talwar.  Mechanism design via differential privacy.  In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), pages 94–103, 2007.
  * [33] John T. Kent, Asaad M. Ganeiber, and Kanti V. Mardia.  A new unified approach for the simulation of a wide class of directional distributions.  Journal of Computational and Graphical Statistics, 27(2):291–301, 2018.
  * [34] Muneeb Ul Hassan, Mubashir Husain Rehmani, and Jinjun Chen.  Differential privacy techniques for cyber physical systems: A survey.  CoRR, abs/1812.02282, 2018.
  * [35] Johes Bater, Xi He, William Ehrich, Ashwin Machanavajjhala, and Jennie Rogers.  Shrinkwrap: Efficient sql query processing in differentially private data federations.  Proc. VLDB Endow., 12(3):307–320, nov 2018.
  * [36] Zhanglong Ji, Zachary C Lipton, and Charles Elkan.  Differential privacy and machine learning: a survey and review.  arXiv preprint arXiv:1412.7584, 2014.
  * [37] Anand D Sarwate and Kamalika Chaudhuri.  Signal processing and machine learning with differential privacy: Algorithms and challenges for continuous data.  IEEE signal processing magazine, 30(5):86–94, 2013.
  * [38] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson.  Scalable private learning with pate, 2018.
  * [39] Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou.  Differentially private generative adversarial network.  CoRR, abs/1802.06739, 2018.
  * [40] Lorenzo Frigerio, Anderson Santana de Oliveira, Laurent Gomez, and Patrick Duverger.  Differentially private generative adversarial networks for time series, continuous, and discrete open data, 2019.
  * [41] Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten.  Dp-cgan: Differentially private synthetic data and label generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  * [42] Jinsung Yoon, James Jordon, and Mihaela van der Schaar.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  In International Conference on Learning Representations, 2019.
  * [43] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.  cite arxiv:1511.06434Comment: Under review as a conference paper at ICLR 2016.
  * [44] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  CoRR, abs/1710.10196, 2017.
  * [45] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  CoRR, abs/1809.11096, 2018.
  * [46] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  CoRR, abs/1609.05473, 2016.
  * [47] Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio.  Maximum-likelihood augmented discrete generative adversarial networks.  CoRR, abs/1702.07983, 2017.
  * [48] William Fedus, Ian Goodfellow, and Andrew Dai.  Maskgan: Better text generation via filling in the ___.  2018\.
  * [49] Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang.  Towards diverse text generation with inverse reinforcement learning.  CoRR, abs/1804.11258, 2018.
  * [50] Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun.  Diversity-promoting GAN: A cross-entropy based generative adversarial network for diversified text generation.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3940–3949, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * [51] Pei Ke, Fei Huang, Minlie Huang, and Xiaoyan Zhu.  Araml: A stable adversarial training framework for text generation.  arXiv preprint arXiv:1908.07195, 2019.
  * [52] Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu.  Long and diverse text generation with planning-based hierarchical variational model.  arXiv preprint arXiv:1908.06605, 2019.
  * [53] Björn Bebensee.  Local differential privacy: a tutorial.  CoRR, abs/1907.11908, 2019.
  * [54] Tianhao Wang, Jeremiah Blocki, Ninghui Li, and Somesh Jha.  Locally differentially private protocols for frequency estimation.  In Proceedings of the 26th USENIX Conference on Security Symposium, SEC’17, page 729–745, USA, 2017. USENIX Association.
  * [55] Raef Bassily, Kobbi Nissim, Uri Stemmer, and Abhradeep Thakurta.  Practical locally private heavy hitters.  CoRR, abs/1707.04982, 2017.
  * [56] Zhan Qin, Yin Yang, Ting Yu, Issa Khalil, Xiaokui Xiao, and Kui Ren.  Heavy hitter estimation over set-valued data with local differential privacy.  In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, page 192–203, New York, NY, USA, 2016. Association for Computing Machinery.
  * [57] T. Wang, N. Li, and S. Jha.  Locally differentially private frequent itemset mining.  In 2018 IEEE Symposium on Security and Privacy (SP), pages 127–143, Los Alamitos, CA, USA, may 2018. IEEE Computer Society.
  * [58] Rui Chen, Haoran Li, A. K. Qin, Shiva Prasad Kasiviswanathan, and Hongxia Jin.  Private spatial data aggregation in the local setting.  In 2016 IEEE 32nd International Conference on Data Engineering (ICDE), pages 289–300, 2016.
  * [59] Mohammad Naseri, Jamie Hayes, and Emiliano De Cristofaro.  Toward robustness and privacy in federated learning: Experimenting with local and central differential privacy.  CoRR, abs/2009.03561, 2020.
  * [60] Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen.  Heterogeneous federated learning.  CoRR, abs/2008.06767, 2020.
  * [61] Pavlos Papadopoulos, Will Abramson, Adam J. Hall, Nikolaos Pitropakis, and William J. Buchanan.  Privacy and trust redefined in federated machine learning.  CoRR, abs/2103.15753, 2021.
  * [62] M. S. Jere, T. Farnan, and F. Koushanfar.  A taxonomy of attacks on federated learning.  IEEE Security and Privacy, 19(02):20–28, mar 2021.
  * [63] Nuria Rodr’iguez-Barroso, Daniel Jim’enez L’opez, M. Victoria Luz’on, Francisco Herrera, and Eugenio Martínez-Cámara.  Survey on federated learning threats: concepts, taxonomy on attacks and defences, experimental study and challenges.  2022\.
  * [64] Robin C. Geyer, Tassilo Klein, and Moin Nabi.  Differentially private federated learning: A client level perspective.  CoRR, abs/1712.07557, 2017.
  * [65] Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard.  Adaptive quantization for deep neural network.  CoRR, abs/1712.01048, 2017.
  * [66] Milan Biswal, Abu Saleh Md Tayeen, and Satyajayant Misra.  AMI-FML: A privacy-preserving federated machine learning framework for AMI.  CoRR, abs/2109.05666, 2021.
  * [67] Zichen Ma, Zihan Lu, Yu Lu, Wenye Li, Jinfeng Yi, and Shuguang Cui.  Federated two-stage learning with sign-based voting, 2021.
  * [68] Andrew Lowy and Meisam Razaviyayn.  Locally differentially private federated learning: Efficient algorithms with tight risk bounds.  CoRR, abs/2106.09779, 2021.
  * [69] Vaikkunth Mugunthan, Ravi Rahman, and Lalana Kagal.  Blockflow: An accountable and privacy-preserving solution for federated learning.  CoRR, abs/2007.03856, 2020.