# A Quantitative Metric for Privacy Leakage in Federated Learning

###### Abstract

In the federated learning system, parameter gradients are shared among
participants and the central modulator, while the original data never leave
their protected source domain. However, the gradient itself might carry enough
information for precise inference of the original data. By reporting their
parameter gradients to the central server, client datasets are exposed to
inference attacks from adversaries. In this paper, we propose a quantitative
metric based on mutual information for clients to evaluate the potential risk
of information leakage in their gradients. Mutual information has received
increasing attention in the machine learning and data mining community over
the past few years. However, existing mutual information estimation methods
cannot handle high-dimensional variables. In this paper, we propose a novel
method to approximate the mutual information between the high-dimensional
gradients and batched input data. Experimental results show that the proposed
metric reliably reflect the extent of information leakage in federated
learning. In addition, using the proposed metric, we investigate the
influential factors of risk level. It is proven that, the risk of information
leakage is related to the status of the task model, as well as the inherent
data distribution.

Index Terms—  Security and Privacy, Federated Learning, Information Theory,
Security Metric.

##  1 Introduction

In the contemporary AI industry, there is an ever-rising quest for organized
data. In most industries, though, large amounts of data exist in isolated
devices and institutes, being wasted away under the restriction of security or
privacy regulations. Federated learning (FL) [1, 2, 3] is devised to activate
the isolated data sources. Distinguished from the centralized machine
learning, in an FL system, data do not leave their protected source locations
[4]. Instead, the model parameter gradients are reported to a central
modulator for global model aggregation.

The FL framework is a promising ideology. Yet, when it comes to application,
many data holders still lack incentives to participate in the FL process [5,
6]. One of the major concerns lies within the verifiability information
security. It has been proved that by observing parameter gradients, an
adversary can make precise inference on the raw input data [7, 8]. Although
provable encryption schemes, such as homomorphic encryption [9, 10], secure
multi-party computation [11, 12, 13] and secret sharing, has been proposed to
guarantee information security, their implementation and operation are to
costly for practical applications. In terms of data obfuscation techniques,
such as differential privacy [14, 15], engineers need to empirically balance
the privacy level and federated model performance. From a data holder’s
perspective, the level of security provided by such designs is too arbitrary
to be convincing. Therefore, a quantifiable and universal metric is essential
to promote incentives for data contribution in the FL systems.

Some may advocate the added noise level in the differential privacy scheme as
an indicator of security degree. But there is no proof of a quantifiable
relationship between the noise level and the information leakage risk. It
still remains an open problem to systematically define the sufficient noise
level for a differentially private model [16].

A practical metric for information leakage risk should satisfy the following
properties:

  * •

Scale invariance \- the quantitative value should have the same meanings under
different circumstances.

  * •

Interpretability \- the metric should be inline with provable information
bounds.

  * •

Model-agnostic \- the model itself can be distributed as a blackbox to
clients.

In the information theory, mutual information (MI) is a measure of the common
information between two random variables. It provides a theoretically
provable, universal and quantifiable metric for the amount of information
leakage on one variable given the other. In the machine learning society,
studies have been dedicated to estimate the MI between observable variables
(model parameters, gradients, etc.) and the original data. However, existing
methods were mostly based on discrete variables, or made risky assumptions
about the probability density functions [17, 18].

Fig. 1: Structure of an FL system with risk pre-alarm.

For the FL framework, we aim to measure the risk of information leakage before
reporting the calculated model parameters to the central server. Consequently,
a client can make informed decision on whether it is safe to upload its model
parameters. Therefore, the corresponding MI input / output variables are
batched raw data and the computed gradients, respectively. Both variables
easily have hundreds of dimensions, well beyond the scope of discussion in
previous works. In this paper, we propose a novel hierarchical mutual
information estimation method, H-MINE, for high-dimensional MI approximation.
The proposed method is then applied to estimate the risk of information
leakage in an FL client under various experimental settings. The main
contributions of this paper are as follows:

  * •

Propose a novel hierarchical model, H-MINE, for robust and efficient high-
dimensional MI estimation.

  * •

Apply H-MINE in the quantification of information leakage risk in FL systems.

  * •

Verify the credibility of H-MINE through comparison of inference attack
results.

  * •

Analyse the inherent influential factors of information security in FL
systems.

##  2 Proposed Method

###  2.1 Risk Pre-Alarm in FL Systems

In the federated stochastic gradient descent (FedSGD) algorithm, clients and
the central server communicates iteratively to jointly optimize the global
task model $f_{\mathbf{\theta}}$ [1]. In a communication round $t$, a client
$i$ obtains the current global model parameters $\mathbf{\theta}^{t}$ from the
central server. Client $i$ calculates the parameter gradient
$G_{i}^{t}=\nabla_{\mathbf{\theta}}f_{\mathbf{\theta}}^{t}(X_{B}^{i})$ with a
batch sampled from local dataset $\mathcal{D}_{i}$. That is, the batched data
$X_{B}^{i}=\\{x_{1}^{i},x_{2}^{i},...,x_{B}^{i}\\}\subseteq\mathcal{D}_{i}$,
where $B$ is the batch size. Client gradients $G_{i}^{t}$, $i=1,...,N$, are
sent to the central server. On the central server, $G_{i}^{t}$’s are
aggregated to update the global model, such that
$\mathbf{\theta}^{t+1}\leftarrow\mathbf{\theta}^{t}-\eta\sum_{i}G_{i}^{t}$,
where $\eta$ is the learning rate.

In this paper, we assume a modest security environment, where all participants
are honest-but-curious. Both the clients and central server would not try to
poison the learning process, but may probe the underlying raw data when they
have access to other participants’ public information $G_{i}^{t}$. Therefore,
clients are susceptible to inference attacks from the central server or other
intercepting adversaries. We propose to implement an information leakage risk
estimator on the client side, so that a client can be alarmed of the potential
risk before publishing its gradient information (see Fig.1). In this paper,
the information leakage risk is quantified by the mutual information between
the batched raw data $X_{B}^{i}$ and the gradient $G_{i}^{t}$, i.e.,
$I(X_{B}^{i};G_{i}^{t})$.

###  2.2 The Neural Estimator for Mutual Information

Without loss of generality, the datapoints in a client dataset are assumed to
be independently and identically distributed (IID), i.e.,
$x_{j}^{i}\sim\mathcal{X}^{i}$, $j=1,...,|\mathcal{D}_{i}|$. It follows that
$G_{i}^{t}(X_{B}^{i})\sim\mathcal{G}_{i}^{t}$.

As discussed in previous sections, MI estimation is non-trivial, especially
for high-dimensional random variables. Belghazi et al. proposed to solve this
problem with a neural network [17]. In fact, the MI between two random
variables $I(X;G)$ is equivalent to the Kullback-Leibler Divergence
$D_{\text{KL}}[\mathbb{P}_{XG}||\mathbb{P}_{X}\otimes\mathbb{P}_{G}]$. The
Donsker-Varadhan representation [19] of the KL-Divergence gives a lower bound
on $I(X_{B};G^{t})$:

$\displaystyle I(X_{B};G^{t})$
$\displaystyle=D_{KL}[\mathbb{P}_{X_{B}G^{t}}||\mathbb{P}_{X_{B}}\otimes\mathbb{P}_{G^{t}}]$
$\displaystyle\geq\sup_{T\in\mathcal{T}}\mathbb{E}_{\mathbb{P}_{X_{B}G^{t}}}[T]-\log(\mathbb{E}_{\mathbb{P}_{X_{B}}\otimes\mathbb{P}_{G^{t}}}[e^{T}]),$

where $\mathcal{T}$ can be any class of functions
$T:(\mathcal{X}^{i},\mathcal{G}_{i}^{t})\rightarrow\mathbb{R}$ satisfying the
integrability constraints of the Donsker-Varadhan theorem. As proposed by
Belghazi et al, choosing a neural network as $\mathcal{T}$ transforms the MI
estimation problem into a network optimization one. This transformation
exploits the flexibility of neural networks in approximating arbitrarily
complex functions, but also benefits from the well-developed tools for network
optimization. The structure of the statistic network $T_{\phi}$ can be
designed to accommodate different data types. Its parameters $\phi$ are
optimized with iterative sampling from the joint and marginal distributions of
the $X_{B}$ and $G^{t}$.

###  2.3 Hierarchical Mutual Information Neural Estimation (H-MINE)

Fig. 2: H-MINE: the hierarchical statistic network.

Numerous architectures of the statistic network have been experimented in the
literature. They performed reasonably well as a constraining factor in tasks
like generative adversarial networks [20], representation learning [21], and
so on. However, when we look into the accuracy of the estimated MI itself, the
previously proposed statistic networks were not so successful. When the
variable dimensions increased, conventional statistic networks failed to
converge or converged at insignificant values. In this section, a novel
statistic network structure is proposed that handles the high-dimensional
input and output variables in the FL clients.

The structure of the proposed statistic network is illustrated in Fig.2.
Particularly, we would like to make use of the fact that the data points
$x\in\mathcal{D}_{i}$ are IID. That is to say, the input variable in our
statistic network, $X_{B}$, can be divided into $B$ independent random
variables. In our proposed statistic network, $x_{m}$ of batch $X_{B}$ and the
gradient $G^{t}$ are grouped into a block $O_{m}=\\{x_{m},G^{t}\\}$,
$m=1,...,B$. Since $O_{m}$’s are identically distributed, a shared sub-
network, called BlockModel, is used to map them into $B$ embedding vectors
$h_{m}$. The embedding vectors are then concatenated as input to the
consequent MixModel, which outputs a single scalar in the final layer. This
statistic network structure is termed hierarchical MINE (H-MINE) for brevity.

As opposed to increasing the depth of the statistic network, H-MINE utilizes a
shared sub-network to extract information from IID components. The
hierarchical structure of the proposed statistic network effectively reduces
the dimensionality of the input variables. The number of parameters in the
input layer is also reduced by $B$ times, compared to a naive fully connected
layer. The thorough procedure for MI approximation using H-MINE is elaborated
in Algorithm 1.

Inputs: Task model batch size $B$, sample size $S$, task model status
$\mathbf{\theta}^{t}$.

Initialize: H-MINE parameters $\phi$.

1 while  _not converged_ do

2 for  _$k=1,...,S$_ do

3 Generate batch samples $X_{B},\hat{X}_{B}$:

4 $X_{B}\leftarrow[x_{1},...,x_{B}]$,
$\hat{X}_{B}\leftarrow[\hat{x}_{1},...,\hat{x}_{B}]$

5

6 Calculate corresponding gradients:

7 $G=\nabla_{\mathbf{\theta}}f_{\mathbf{\theta}^{t}}(X_{B})$,
$\hat{G}=\nabla_{\mathbf{\theta}}f_{\mathbf{\theta}^{t}}(\hat{X}_{B})$

8 for  _$j=1,...,B$_ do

9 $h_{j}=\textbf{BlockModel}_{\phi}(x_{j},G)$

10 $\hat{h}_{j}=\textbf{BlockModel}_{\phi}(x_{j},\hat{G})$

11

12 end for

13 $H=\\{h_{1},h_{2},...,h_{B}\\}$,
$\hat{H}=\\{\hat{h}_{1},\hat{h}_{2},...,\hat{h}_{B}\\}$

14 $v_{k}=T_{\phi}(X_{B},G)=\textbf{MixModel}_{\phi}(H,G)$,

15
$\hat{v}_{k}=T_{\phi}(X_{B},\hat{G})=\textbf{MixModel}_{\phi}(\hat{H},\hat{G})$

16

17 end for

18 Evaluate the lower-bound:

19
$V(\phi)=\frac{1}{S}\sum_{k=1}^{S}v_{k}-\log(\frac{1}{S}\sum_{k=1}^{S}e^{\hat{v}_{k}})$

20 Update H-MINE parameters:

21 $\phi\leftarrow\phi+\nabla_{\phi}V(\phi)$ ;

22

23 end while

Algorithm 1 H-MINE Algorithm.

##  3 Results

###  3.1 Experiment Settings

In this section, we evaluate the performance of our proposed methods with the
Adult dataset [22]. The task of the Adult dataset is to predict whether an
individual’s annual income exceeds $50K. It contains 14 private attributes
about each individual, including education level, age, gender, occupation, and
so on. The dataset contains 32,560 samples (7,841 positive and 24,719
negative). Missing values are replaced with the medians in the dataset. A
logistic regression is used as the task classification model.

In H-MINE, the BlockModel is a multi-layer perception (MLP), which contains 3
fully connected layers with 200, 200, 5 neurons, respectively. The MixModel is
a two-layer MLP with 500, 1 neurons, respectively. We use Adam [23] optimizer
to train H-MINE and learning rate $\alpha_{\phi}=5\times 10^{-5}$.

As a baseline of correlation metrics, the sum of all elements of covariance
matrix $C(X_{B},G^{t})=\text{sum}\left(\text{cov}(X_{B},G^{t})\right)$ is also
included in the subsequent figures.

###  3.2 Convergence Analysis

Fig. 3: Comparison of convergence speed of H-MINE and a conventional statistic
network. Gradients are evaluated at $epoch=1$. Batch sizes are 1 and 3
respectively.

In our proposed method, the MI is approximated via optimization of a statistic
network. The convergence curves of the proposed statistic network, H-MINE, are
presented in Fig.3. The curves of a conventional statistic network, comprising
of a three layer MLP [17], are depicted for comparison. In these experiments,
the number of dimensions in the input variable multiplies as the batch size
$B$ increases. The conventional statistic network diverges in all attempted
configurations, for its inability to handle high-dimensional variables. In
contrast, H-MINE conveges steadily even with increased batch size.

###  3.3 Validation Via Inference Attack

In this section, we use the DeepLeakage model to validate the accuracy of our
estimated mutual information $I(X_{B};G^{t})$. In theories, smaller mutual
information between the two variables leads to increased difficulty in the
inference attack. Therefore, in the following experiments, we compare the
inference error of the recovered data with our estimated $I(X_{B};G^{t})$ in
various circumstances, and verify their correlation.

Fig. 4: Validation with DeepLeakage. Gradients are evaluated at $epoch=3$. For
the added noise experiment, $B=3$.

The DeepLeakage model in our experiments is a three-layer MLP that contains
100, 100, and 2 neurons, respectively. For each configuration, the DeepLeakage
is performed 5 times with random initialization, producing inference results
$\hat{X}_{B}^{(k)}$, $k=1,...,5$. The inference error of DeepLeakage is
defined as $\epsilon=\text{var}(\hat{X}_{B}^{(k)})$.

Zhu et al. suggested two means to defend against inference attacks, namely,
increasing the batch size, and adding noise to the gradients. Fig.4
demonstrates the change of $I(X_{B};G^{t})$ and $\epsilon$ verses batch size
(left) and added noise on the gradients (right). As it shows, the estimated MI
decreases as the batch size and noise level increase. At the same time, the
inference error increases quadratically. The results are inline with our
hypothesis, but also consistent with the results in [7]. In other words, our
estimated MI can faithfully reflect the risk of inference attacks.

###  3.4 Inherent Factors Affecting Information Leakage

In this section, we analyze the factors affecting the risk of information
leakage that are inherent in the data or the training process themselves.

(a) Leakage risk at different timesteps, batch size $B=3$.

(b) Leakage risk with unbalanced datasets, $epoch=3$, $B=3$.

Fig. 5: Inherent factor analysis.

Firstly, FedSGD is an iterative process. Clients calculate their gradients
over the evolving parameter values. At different timesteps, the gradients may
present different levels of information about the batched data, due to its
interaction with the current model status. Fig.5(a) presents $I(X_{B};G^{t})$
verses the iteration step $t$. The MI is the largest at the first epoch, while
it gradually decreases over the optimization process. At initial steps, every
data point is new to the model. Thus the gradients possess strong information
about how the model misses to fit the observed data. When the model starts to
gain a reasonable form, the amount of miss-fit shrinks, therefore revealing
less about the observed data. As the task model converges, the MI between the
gradients and the raw data also stablizes at a certain level.

Secondly, it is speculated that the data distribution itself also affects the
risk of information leakage [24]. In this experiment, we simulate a client
dataset with different ratios of positive and negative entries, and
investigate the corresponding variation of $I(X_{B};G^{t})$. As shown in Fig
5(b), the more unbalanced the dataset is, the more information about the
batched data is preserved in the gradients. The results verify that, gradients
on unbalanced data distributions are more vulnerable to inference attacks.

##  4 Conclusions

In this paper, we proposed a novel security metric for FL clients based on
high-dimensional mutual information estimation. The proposed algorithm H-MINE
can effectively and faithfully approximate the mutual information between
model gradients and the original batched data. Therefore, it provides a
quantitative metric for potential risk alarms on the client side. Analysis on
the inherent factors of information leakage risk suggest data holders to be
cautious with initial training steps and unbalanced data distributions.

##  5 ACKNOWLEDGEMENTS

This paper is supported by National Key Research and Development Program of
China under grant No. 2018YFB1003500, No. 2018YFB0204400 and No.
2017YFB1401202. Corresponding author is Jianzong Wang from Ping An Technology
(Shenzhen) Co., Ltd.

## References

  * [1] Jakub Konecnỳ, H Brendan McMahan, Daniel Ramage, and Peter Richtárik,  “Federated optimization: Distributed machine learning for on-device intelligence,”  2016\.
  * [2] Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon,  “Federated learning: Strategies for improving communication efficiency,”  arXiv preprint arXiv:1610.05492, 2016.
  * [3] Anxun He, Jianzong Wang, Zhangcheng Huang, and Jing Xiao,  “Fedsmart: An auto updating federated learning optimization mechanism,”  in APWeb and WAIM Joint International Conference on Web and Big Data. Springer, 2020, pp. 716–724.
  * [4] Lingwei Kong, Hengtao Tao, Jianzong Wang, and et al,  “Network coding for federated learning systems,”  in ICONIP. Springer, 2020.
  * [5] Xinghua Zhu, Jianzong Wang, Zhenhou Hong, and Jing Xiao,  “Empirical studies of institutional federated learning for natural language processing,”  in Findings of EMNLP. ACL, 2020.
  * [6] Xinghua Zhu, Jianzong Wang, Zhenhou Hong, Tian Xia, and Jing Xiao,  “Federated learning of unsegmented chinese text recognition model,”  in 2019 IEEE 31st ICTAI. IEEE, 2019, pp. 1341–1345.
  * [7] Ligeng Zhu, Zhijian Liu, and Song Han,  “Deep leakage from gradients,”  in Advances in Neural Information Processing Systems, 2019, pp. 14774–14784.
  * [8] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller,  “Inverting gradients–how easy is it to break privacy in federated learning?,”  arXiv preprint arXiv:2003.14053, 2020.
  * [9] Ronald L Rivest, Len Adleman, Michael L Dertouzos, et al.,  “On data banks and privacy homomorphisms,”  Foundations of secure computation, vol. 4, no. 11, pp. 169–180, 1978.
  * [10] Abbas Acar, Hidayet Aksu, A Selcuk Uluagac, and Mauro Conti,  “A survey on homomorphic encryption schemes: Theory and implementation,”  ACM Computing Surveys (CSUR), vol. 51, no. 4, pp. 1–35, 2018.
  * [11] Adi Shamir,  “How to share a secret,”  Communications of the ACM, vol. 22, no. 11, pp. 612–613, 1979.
  * [12] Oded Goldreich,  “Secure multi-party computation,”  Manuscript. Preliminary version, vol. 78, 1998.
  * [13] Payman Mohassel and Peter Rindal,  “Aby3: A mixed protocol framework for machine learning,”  in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, 2018, pp. 35–52.
  * [14] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith,  “Calibrating noise to sensitivity in private data analysis,”  in Theory of cryptography conference. Springer, 2006, pp. 265–284.
  * [15] Cynthia Dwork, Aaron Roth, et al.,  “The algorithmic foundations of differential privacy,”  Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3-4, pp. 211–407, 2014.
  * [16] Jaewoo Lee and Chris Clifton,  “How much is enough? Choosing $\epsilon$ for differential privacy,”  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 7001 LNCS, pp. 325–340, 2011.
  * [17] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm,  “Mine: mutual information neural estimation,”  arXiv preprint arXiv:1801.04062, 2018.
  * [18] Morteza Noshad, Yu Zeng, and Alfred O Hero,  “Scalable mutual information estimation using dependence graphs,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 2962–2966.
  * [19] M. D. Donsker and S. R.S. Varadhan,  “Asymptotic evaluation of certain markov process expectations for large time—iii,”  Communications on Pure and Applied Mathematics, vol. 29, no. 4, pp. 389–461, July 1976.
  * [20] Thang Doan, Joao Monteiro, Isabela Albuquerque, Bogdan Mazoure, Audrey Durand, Joelle Pineau, and R Devon Hjelm,  “On-line adaptative curriculum learning for gans,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 3470–3477.
  * [21] Liangjian Wen, Yiji Zhou, Lirong He, Mingyuan Zhou, and Zenglin Xu,  “Mutual information gradient estimation for representation learning,”  arXiv preprint arXiv:2005.01123, 2020.
  * [22] Dheeru Dua and Casey Graff,  “UCI machine learning repository,” 2017.
  * [23] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [24] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown,  “Federated visual classification with real-world data distribution,”  arXiv preprint arXiv:2003.08082, 2020.