# Mace: A flexible framework for
membership privacy estimation in generative models

Yixi Xu<EMAIL_ADDRESS>
Microsoft Sumit Mukherjee*<EMAIL_ADDRESS>
Insitro Xiyang Liu<EMAIL_ADDRESS>
University of Washington Shruti Tople<EMAIL_ADDRESS>
Microsoft Research Rahul Dodhia<EMAIL_ADDRESS>
Microsoft Juan Lavista Ferres<EMAIL_ADDRESS>
Microsoft 00footnotetext: Equal contribution. equal contribution

###### Abstract

Generative machine learning models are being increasingly viewed as a way to
share sensitive data between institutions. While there has been work on
developing differentially private generative modeling approaches, these
approaches generally lead to sub-par sample quality, limiting their use in
real world applications. Another line of work has focused on developing
generative models which lead to higher quality samples but currently lack any
formal privacy guarantees. In this work, we propose the first formal framework
for membership privacy estimation in generative models. We formulate the
membership privacy risk as a statistical divergence between training samples
and hold-out samples, and propose sample-based methods to estimate this
divergence. Compared to previous works, our framework makes more realistic and
flexible assumptions. First, we offer a generalizable metric as an alternative
to the accuracy metric (Yeom et al., 2018; Hayes et al., 2019) especially for
imbalanced datasets. Second, we loosen the assumption of having full access to
the underlying distribution from previous studies (Yeom et al., 2018;
Jayaraman et al., 2020), and propose sample-based estimations with theoretical
guarantees. Third, along with the population-level membership privacy risk
estimation via the optimal membership advantage, we offer the individual-level
estimation via the individual privacy risk. Fourth, our framework allows
adversaries to access the trained model via a customized query, while prior
works require specific attributes (Hayes et al., 2019; Chen et al., 2019;
Hilprecht et al., 2019).

##  1 Introduction

The past decade has seen much progress in machine learning, largely due to the
rapid growth in the number of large-scale datasets. However, concerns about
the privacy of individuals being represented in datasets have led to a variety
of regulations that made it increasingly difficult to share sensitive data
across institutions especially in healthcare (Voigt & Von dem Bussche, 2017).
Recent progress in the area of generative machine learning has made it
possible to share synthetic data, which reflects the statistical properties of
the original datasets (Georges-Filteau & Cirillo, 2020). Such synthetic data
has been shown to allow for the development of downstream machine learning
models with limited loss of performance compared to the original data (Rajotte
et al., 2021). This has led to synthetic data sharing being increasingly
viewed as a potentially privacy preserving alternative to sharing the raw data
(Tom et al., 2020) between institutions.

Despite the appeal of using synthetic data generated by generative models as
an alternative to traditional data sharing, recent work has shown common
generative modeling approaches are often vulnerable to a variety of privacy
attacks (Hilprecht et al., 2019; Hayes et al., 2019; Chen et al., 2019). This
led to the development of differentially private generative modeling
approaches which allow for the generation of synthetic data while providing
strong formal privacy guarantees (Xie et al., 2018; Jordon et al., 2018).
However, in the case of high dimensional datasets (such as images), such
approaches have been shown to produce synthetic samples of very poor quality
for any reasonable level of guaranteed privacy (Xie et al., 2018; Mukherjee et
al., 2019). For example, Mukherjee et al. (2019) generated 50,000 synthetic
CIFAR10 samples using the differentially private GAN (Xie et al., 2018) with a
large privacy budget $\epsilon=100$. Then, a classifier was trained on the
synthetic data, however the accuracy was below 20% when validated on the test
set. It was initially assumed that the poor performance was the result of
loose privacy accounting (leading to an overestimation of $\varepsilon$), but
this assumption is recently challenged by a work indicating that moments
accountant-based approaches lead to tight estimates of $\varepsilon$ (Nasr et
al., 2021). Thus greatly reducing the possibility of developing a
differentially private generative modeling approach that could generate
samples good enough to train a strong ML model in practical high-dimensional
data settings.

More recent work has focused on developing novel generative learning methods
that have been empirically shown to be protected against certain types of
privacy attacks, such as membership inference attacks (Mukherjee et al., 2019;
Chen et al., 2021). Despite these advancements in empirically improving the
privacy of generative models in a few settings, there is currently no approach
to provide formal privacy certificates for such models. This in turn has
limited the usability of such models in real-world applications where the
complete lack of formal certificates would pose a problem with regulators. A
promising line of related work has been in using membership inference attacks
to audit the privacy of trained machine learning models (primarily
discriminative models) with theoretical justifications (Yeom et al., 2018;
Jayaraman et al., 2020). More specifically, these works estimate the
membership privacy risk of a model against a specific adversary. As a result,
it would be computationally expensive to estimate the maximum risk when there
is a large group of adversaries, or even impossible given an infinite set of
adversaries. As a comparison, MACE is able to estimate the maximum privacy
risk via the Bayes optimal classifier. Another limitation is that these
frameworks assume a full access to the underlying data distribution. For
example, Yeom et al. (2018) used a dataset including 4819 patients who were
prescribed warfarin, collected by the International Warfarin Pharmacogenetics
Consortium to demonstrate their methodology. However, the method could hardly
generalize if the population of interest includes all the patients who were
prescribed warfarin instead of the specific 4819 patients. While the above
case is more common in practice, it is usually not feasible to get full access
to this kind of sensitive data. Thus, existing methods (Yeom et al., 2018;
Jayaraman et al., 2020) are not applicable, and this leaves a gap between
theory and practice. To overcome this restriction, MACE allows for not only a
full access but also a limited access to the data via a simple random sample.
Furthermore, MACE is able to provide consistent estimators of membership
privacy risks at both individual and population level. Last but not least,
much of this line of work has focused on auditing differentially private
discriminative models. As a comparison, we focus on trained generative models,
which may or may not be differentially private.

To motivate our paper, let us look at a real-world application scenario. A
clinical research institution wants to publicly release a medical imaging
dataset to enable machine learning model development using the data. However,
due to concerns about the personal health information (PHI) in the dataset,
they look into synthetic data generation. Having identified a viable
generative modeling approach, the institution is faced with a few questions
prior to data/model release: i) should they release the synthetic dataset or
the trained generative model?, ii) if they just release synthetic data, does
it matter how many synthetic samples they release?, iii) how vulnerable is the
synthetic data or the trained model to membership inference attacks.
Currently, there is no answer to these questions, unless making a strong
assumption of the adversaries i.e. only considering a few specific heuristic
membership inference attacks (Hayes et al., 2019; Hilprecht et al., 2019),
which is not realistic in practice.

In this paper, we begin to answer these questions through the development of a
flexible statistical framework to measure the membership privacy risk in
generative models (MACE: Membership privACy Estimation). Our framework is
built on the formulation of the membership privacy risk (given a query access)
as a statistical divergence between the distribution of training-set and non-
training-set samples. We show the utility of our framework using many SOTA
queries from the literature and some new ones against common computer vision
as well as medical imaging datasets.

Our primary contributions are as follows:

  * •

We develop a framework to estimate the maximum membership privacy risk against
adversaries that have query access to the model. Our framework can not only
estimate membership privacy risks that are defined as the accuracy of the
membership inference attack as in (Yeom et al., 2018), but also those from a
more general risk class (Koyejo et al., 2014). This gives the users
flexibility to measure the ability of a membership inference attack to
distinguish members from non-members from different angles, especially when
the training set is a small part of the total available dataset. In addition,
MACE is capable of estimating the membership privacy risk given any scalar or
vector valued attributes from a learned model, while prior works (Hayes et
al., 2019; Chen et al., 2019; Hilprecht et al., 2019) restrict to a set of
specific attributes.

  * •

Our framework is able to measure the individual-level membership privacy risk.
This measures the risk of each individual sample against specific modes of
membership inference attacks, allowing users to identify those high risk
individuals and decide whether to exclude high risk samples and re-train their
model.

  * •

We loosen the assumption of having full access to the training and non-
training set from previous studies (Yeom et al., 2018; Jayaraman et al.,
2020), and extend to the case where only a simple random sample is feasible.
Furthermore, we derive consistent estimators for both the maximum membership
privacy risk and the individual-level membership privacy risk with theoretical
justifications.

  * •

We demonstrate the usability of MACE by experiments that analyze the
membership privacy risks with regards to various query types and generative
model architectures on three real-world datasets via the membership advantage
and the individual privacy risk under both the accuracy-based and generalized
metrics.

##  2 Background

In this section, we present a brief background on query functions, membership
inference attacks, attack experiments, and the Bayes optimal classifier. Then
we briefly discuss the limitations of current membership inference approaches.
We assume readers already possess a general understanding of generative models
and Differential Privacy, but provide a short background section in the
Appendix for the sake of completeness.

###  2.1 Notation

We introduce notations that will be used in the rest of the paper.

  * •

Let $z=(x,y)\in{\cal Z}={\cal X}\times{\cal Y}$ be a data point from a data
distribution ${\cal D}$. Note that $y$ is some extra information for
generative models such as conditional GANs. In other words, $z=x\in{\cal X}$
for normal generative models.

  * •

We assume an adversary ${\cal A}$ would have query access to a model via a
query function $Q_{S}(\cdot):{\cal X}\times{\cal Y}\to\mathbb{R}^{q}$, where
$q$ is an integer. Examples of adversaries and query functions are given in
Sections 2.3 and 2.2, respectively.

  * •

Let $S\sim{\cal D}^{n}$ be an ordered list of $n$ points. It is referred to as
training set, sampled from ${\cal D}$. We will assume the training set $S$ to
be fixed in this paper.

  * •

$z\sim S$ denotes uniformly sampling from a training set $S$. Also,
$z\sim{\cal D}\backslash S$ denotes uniformly sampling from the data
distribution not including the training set $S$, which is referred to as
sampling from a hold-out set.

  * •

For a set of samples $\\{z_{1},z_{2},\cdots,z_{N}\\}$, we define their
associated membership labels as $\\{m_{1},m_{2},\cdots,m_{N}\\}$, where
$m_{i}=1$ if $z_{i}$ is in the training set and $m_{i}=-1$ otherwise for
$i=1,\cdots,N$.

  * •

For a given condition $\cal C$, let $\mathbb{I}({\cal C})=1$ if the condition
$\cal C$ holds, otherwise 0.

###  2.2 Query functions

In this subsection, we introduce some representative query functions
$Q_{S}(z)$ considered in or motivated by prior work. Following (Chen et al.,
2019; Hilprecht et al., 2019; Hayes et al., 2019), we divide our attack
settings based on the accessibility of model components: (1) access only to
generated synthetic data; (2) access to models.

####  2.2.1 Accessible synthetic datasets

In the common practice of synthetic data releasing, researchers or data
providers may consider releasing only generated datasets or just the
generator. However, prior works (Chen et al., 2019; Hilprecht et al., 2019)
have shown releasing generator/synthetic datasets can cause privacy leakage.
Specifically, for the case where a generative model is released, Cheng et al.
(Chen et al., 2019) consider the following query function
$Q_{S}(z)=\min_{w}L_{2}(z,G(w))$, where $G$ is the generator released to the
public. Alternatively, Hilprecht et al. (Hilprecht et al., 2019) first
generate a large synthetic dataset $g_{1},g_{2},\cdots,g_{n}$ using the
generator and then use the following query function:

$\displaystyle Q_{S}(z)=-\frac{1}{n}\sum_{i=1}^{n}\mathbb{I}({g_{i}\in
U_{\varepsilon}(z)})\log d\left(g_{i},z\right),$ (1)

where $d$ is some distance metric and $U_{\varepsilon}$ is $\varepsilon$-ball
defined on distance metric $d$.

Similar to these approaches, we propose a query function (defined below). For
a sample $z$ , we consider using the nearest neighbor distance to synthetic
datasets as the query function:

$\displaystyle Q_{S}(z)=\min_{j\in[n]}d(z,g_{j})\;,$ (2)

where $d$ is a distance metric.

####  2.2.2 Accessible models

In this setting, we assume the adversary has query access to the model (the
discriminator and the generator in the case of GANs). Such a situation
commonly arises when researchers open source model parameters, or share model
parameters insecurely.

For generative models, especially GANs, the most successful attack known
(Hayes et al., 2019)assumes adversaries to access the model via the following
query:

$\displaystyle Q_{S}(z)=D(z)\;,$ (3)

where $D(z)$ is the output of the discriminator corresponding to input sample
$z$. Intuitively, if a sample is in the training set, the discriminator would
be more likely to output high values. While the adversary could solely access
the discriminator via the above query, we introduce a query below allowing
accessing both the generator and the discriminator.

$\displaystyle Q_{S}(z)=(D(z),\min_{j\in[n]}d(z,g_{j}))\;.$ (4)

This attack is a combination of attacks described in equations 3 and 2.

While the discriminator score has been shown to be a very effective query for
generative models with one discriminator, many recent privacy preserving
generative modeling approaches often have multiple discriminators (Jordon et
al., 2018; Mukherjee et al., 2019), with each discriminator being exposed to a
part of the training dataset. Our previous query will not be useful in such
situations. Here, we consider the recent work privGAN((Mukherjee et al., 2019)
and present two queries (one single dimensional and one multi-dimensional).
The single dimensional query used in (Mukherjee et al., 2019) is as follows:

$\displaystyle Q_{S}(z)=\max_{i}D_{i}(z)\;,$ (5)

where $D(z)$ is the output of the discriminator $i$ corresponding to input
sample $z$. We propose a new multi-dimensional query which is stated as
follows:

$\displaystyle Q_{S}(z)=(D_{1}(z),\cdots,D_{r}(z))\;,$ (6)

For the purposes of demonstration, in this paper we use $r=2$.

###  2.3 Membership inference attack adversaries

The goal of a membership inference attack (MIA) (Li et al., 2013; Shokri et
al., 2017; Truex et al., 2018; Long et al., 2017), is to infer whether a
sample $z$ is a part of the training set $S$. In our paper, we assume an MIA
access the model via a query function $Q_{S}$. Thus, an MIA adversary is
equivalent as a classifier given $Q_{S}(z)$ as the input to predict whether
$z$ belongs to $S$.

In this paper, we focus on generative machine learning models (such as GANs).
The study of MIAs against generative models is a relatively new research area.
Hayes et al. (2019) first demonstrated MIAs against GANs. They propose: i) a
black-box adversary that trains a shadow GAN model using the released
synthetic data, ii) a white-box adversary that uses a threshold on the
discriminator score of a released GAN model. Hilprecht et al. (2019)
demonstrates a black-box MIA adversary that uses only the generator of the
GANs (or synthetic samples) and operates by thresholding the L2 distance
between the query sample and the closest synthetic sample. These existing
works focus on the construction of a strong binary classifier as the MIA
adversary, given different query functions. The details of these query
functions have been discussed in Section 2.2.

###  2.4 The attack experiment

The membership privacy risk that arises from a query to the model is usually
evaluated through a membership privacy experiment (Yeom et al., 2018;
Jayaraman et al., 2020). The experiment assumes we have sampled a training set
$S$ with size $|S|=n$ from the data distribution ${\cal D}$. Then a learning
algorithm is trained on $S$, and an adversary would have access to the trained
model through a query function $Q_{S}:{\cal Z}\rightarrow{\cal Q}$. To be
specific, An adversary is provided with the query output of a randomly sampled
point $z$ from either $S$ (with probability $p$) or ${\cal D}\backslash S$
(with probability $1-p$). The adversary would then get a utility $1$ if it
guesses the membership correctly or an utility $-1$ otherwise.

###  2.5 The Bayes optimal classifier

Since membership identification is essentially a binary classification task, a
membership adversary can then be seen simply as a binary classification model.
Indeed, many existing papers on membership inference explicitly train binary
classifiers for the purpose of membership inference (Shokri et al., 2017). The
performance of such classifiers is often also used to empirically measure the
membership privacy risk of different models (Mukherjee et al., 2019). As the
binary classifiers used in such papers are heuristically chosen, there is no
guarantee that a better classification model does not exist for the task. The
classifier that minimizes the expected error rate (maximizes accuracy) is
called the Bayes optimal classifier (Devroye et al., 2013). Given, samples
$(x,y)\in\mathcal{R}^{d}\times\\{-1,1\\}$, the Bayes optimal classifier is:

$C^{Bayes}(x)=\operatorname*{arg\,max}_{r\in\\{-1,1\\}}{\mathbb{P}}(Y=r|X=x).$
(7)

Note: the Bayes optimal classifier can only be approximated in practical
scenarios, since we rely on estimates of ${\mathbb{P}}(Y=r|X=x)$.

While the Bayes optimal classifier in Equation 7 was originally designed to
maximize the classification accuracy, Koyejo et al. (2014) has extended it to
a family of the generalized metrics, defined as follows:

$\ell({\cal A},{\cal
P})=\frac{a_{0}+a_{11}\mathrm{TP}+a_{10}\mathrm{FP}+a_{01}\mathrm{FN}+a_{00}\mathrm{TN}}{b_{0}+b_{11}\mathrm{TP}+b_{10}\mathrm{FP}+b_{01}\mathrm{FN}+b_{00}\mathrm{TN}},\;$
(8)

where ${\cal A}$ is the adversary, ${\cal P}$ is the distribution, $a_{0}$,
$b_{0}$, $a_{ij}$ and $b_{ij}$ are pre-defined scalars for $i=0,1$ and
$j=0,1$, $(X,Y)\sim{\cal P}$, and $TP$, $FP$, $FN$, $TN$ stand for
${\mathbb{P}}({\cal A}(X)=1,Y=1)$, ${\mathbb{P}}({\cal A}(X)=1,Y=-1)$,
${\mathbb{P}}({\cal A}(X)=-1,Y=1)$ and ${\mathbb{P}}({\cal A}(X)=-1,Y=-1)$
respectively. The generalized metric can be used to represent several commonly
used metrics such as accuracy, PPV, TPR, TNR, WA etc. (see Appendix). It is
then demonstrated that the Bayes optimal classifier for this family of
generalized metrics takes the forms:

${\rm sgn}(\eta(x)-t_{\ell})\;\mathrm{or}\;{\rm sgn}(t_{\ell}-\eta(x)),$ (9)

where $\eta(x):={\mathbb{P}}(Y=1|{\cal A}(X)={\cal A}(x))$ and
$t_{\ell}\in(0,1)$ is a constant depending on the metric $\ell$, when the
marginal distribution of $X$ is absolutely continuous with respect to the
dominating measure on ${\cal X}$ (Koyejo et al., 2014). For the rest of this
paper, we assume the above condition always holds. It is worth noting that the
Bayes optimal classifier for the generalized metric can only be approximated
in practical scenarios, due to the lack of closed-form expressions of
$\eta(x)$ and $t_{\ell}$.

###  2.6 Limitations of current membership inference approaches

There are several limitations in the existing literature on membership
inference. First, most papers (Hayes et al., 2019; Hilprecht et al., 2019)
focus on developing novel heuristic membership inference attacks, which are
often limited in scope and can hardly be extended to another query. This is
particularly problematic as much of these heuristic approaches cannot readily
generalize to the generative modeling setting. Second, the current formal
membership privacy estimation frameworks Yeom et al. (2018); Jayaraman et al.
(2020) require a full access to the underlying data distribution, while this
is not always possible in practice. Third, no paper has yet provided a
rigorous approach to estimate the membership privacy risk at the individual
level. Fourth, for most of the current membership inference methods (Hayes et
al., 2019; Hilprecht et al., 2019) probability $p$ of Experiment 1 is usually
set as $0.5$ to form a balanced binary classification problem. However, in
practice, $p$ is usually much smaller than $0.5$, as pointed out in prior work
(Jayaraman et al., 2020; Rezaei & Liu, 2020). In this work we seek to address
all these issues.

##  3 The Membership Privacy Risk Quantification

In this section, we first introduce the membership advantage of a given
adversary. Then, we define the optimal membership advantage as the maximum
membership advantage. It is the maximum expected membership privacy risk of
any adversary for the whole population. Furthermore, we present the optimal
membership inference adversary. Finally, the individual privacy risk would be
proposed to estimate the membership privacy risk at the individual level. The
first subsection focuses on the accuracy-based metric, and the second
subsection extends to the generalized metrics. We first propose an experiment
formalizing membership inference attacks, adapted from Yeom et al. (2018).

######  Experiment 1.

Let ${\cal D}$ be the data distribution on ${\cal Z}$. We first have a fixed
training set $S\sim{\cal D}^{n}$ with size $|S|=n$ and have a trained model.
An adversary ${\cal A}:{\cal Q}\rightarrow\\{-1,1\\}$ would access the trained
model via the query function $Q_{S}:{\cal Z}\rightarrow{\cal Q}$. The
membership experiment proceeds as follows:

  1. 1.

Randomly sample $m\in\\{-1,1\\}$ such that $m=1$ with probability $p$.

  2. 2.

If $m=1$, then uniformly sample $z\sim S$; otherwise sample $z\sim{\cal
D}\backslash S$ uniformly.

Let ${\cal E}({\cal D},S,Q_{S})$ denote the distribution of $(z,m)$ in
Experiment 1.

###  3.1 For the accuracy-based metric

To define the optimal membership advantage, we first introduce the membership
advantage of an adversary ${\cal A}$ given the query $Q_{S}$, as given in
Definition 4 of Yeom et al. (2018).

######  Definition 1.

The membership advantage of $Q_{S}$ by an adversary ${\cal A}$ is defined as

${\rm Adv}_{p}(Q_{S},{\cal A})=2{\mathbb{P}}\left({\cal
A}(Q_{S}(z))=m\right)-1,$ (10)

where the probability is taken over the random sample $(z,m)$ drawn from
${\cal E}({\cal D},S,Q_{S})$.

If the adversary ${\cal A}$ is random guessing, ${\rm Adv}_{p}(Q_{S},{\cal
A})=0$. If the adversary always gets the membership right, then ${\rm
Adv}_{p}(Q_{S},{\cal A})=1$. Further, note that ${\mathbb{P}}\left({\cal
A}(Q_{S}(z))=m\right)$ is the expected accuracy of the adversary’s
predictions. Hence this definition of membership advantage is directly related
to accuracy. After introducing the membership advantage, we define the optimal
membership advantage as the maximum membership advantage of all possible
adversaries.

######  Definition 2.

The optimal membership advantage is defined as

${\rm Adv}_{p}(Q_{S})=\max_{\cal A}{\rm Adv}_{p}(Q_{S},{\cal A}).$ (11)

The following lemma obtains the optimal adversary for the accuracy-based
metric.

######  Lemma 1.

Given the query $Q_{S}$, the data distribution ${\cal D}$, the training set
$S$ and the prior probability $p$, the Bayes optimal classifier ${\cal A}^{*}$
maximizing membership advantage is given by

$\displaystyle{\cal A}^{*}(Q_{S}(z))={\rm
sgn}\left({\mathbb{P}}\left(m=1|Q_{S}(z)\right)-\frac{1}{2}\right)\;,$ (12)

where the probability is taken over the random sample $(z,m)$ drawn from
${\cal E}({\cal D},S,Q_{S})$.

Furthermore, the optimal membership advantage can be re-written as

$\displaystyle{\rm Adv}_{p}(Q_{S})=\mathbb{E}_{z}\left[|f_{p}(z)|\right]\;,$
(13)

where

$f_{p}(z)=\frac{{\mathbb{P}}(Q_{S}(z)|m=1)p-{\mathbb{P}}(Q_{S}(z)|m=-1)(1-p)}{{\mathbb{P}}(Q_{S}(z)|m=1)p+{\mathbb{P}}(Q_{S}(z)|m=-1)(1-p)}.$
(14)

###### Proof.

See the complete proof in Appendix D.1 ∎

Now we introduce the individual privacy risk at a sample $z_{0}$. It is
proportional to the accuracy of the Bayesian optimal classifier ${\cal A}^{*}$
conditioning on $Q_{S}(z)=Q_{S}(z_{0})$.

######  Definition 3.

The individual privacy risk of a sample $z_{0}$ for a query $Q_{S}$ under the
accuracy-based metric is defined as:

${\rm AdvI}_{p}(Q_{S},z_{0})=2\mathbb{P}[{\cal
A}^{*}(Q_{S}(z))=m|Q_{S}(z)=Q_{S}(z_{0})]-1,$

where the probabilities are taken over the random sample $(z,m)$ drawn from
${\cal E}({\cal D},S,Q_{S})$.

we then provide several convenient properties of the individual privacy risk
${\rm AdvI}_{p}(Q_{S},z_{0})$ under the accuracy-based metric at a sample
$z_{0}$. First, we show that it could be rewritten as $|f_{p}(z_{0})|$, where
$f_{p}$ is defined in Equation 14. This can be used in the following sections
to derive a consistent estimator and confidence interval. Secondly, we connect
the individual privacy risk to the optimal membership advantage. Thirdly, we
show that ${\rm AdvI}_{p}(Q_{S},z_{0})$ is proportional to the highest
accuracy conditioning on $Q_{S}(z_{0})$. Fourthly, we establish the connection
between the optimal membership advantage, the individual privacy risk and
Differential Privacy.

######  Theorem 1.

Let $Q_{S}$ be the query function and $z_{0}$ a fixed sample $\in{\cal Z}$.

  1. 1.

The individual privacy risk at $z_{0}$ can be re-written as

${\rm AdvI}_{p}(Q_{S},z_{0})=|f_{p}(z_{0})|$

  2. 2.

The optimal membership advantage is the expectation of the individual privacy
risk given the sample $z$ as a random variable.

${\rm Adv}_{p}(Q_{S})=\mathbb{E}_{z}[{\rm AdvI}_{p}(Q_{S},z)]$ (15)

  3. 3.

The Bayesian optimal classifier ${\cal A}^{*}$ is optimal at the individual-
level, as

${\rm AdvI}_{p}(Q_{S},z_{0})=2\max_{{\cal A}}\mathbb{P}[{\cal
A}(Q_{S}(z))=m|Q_{S}(z)=Q_{S}(z_{0})]-1$

  4. 4.

If a training algorithm is $\varepsilon$-differentially private, then for any
choice of $z_{0}$, we have both the optimal membership advantage and the
individual privacy risk bounded by a constant determined by $\varepsilon$:

$\displaystyle{\rm Adv}_{p}(Q_{S}),{\rm
AdvI}_{p}(Q_{S},z_{0})\leq\max\left\\{\left|\tanh\left(\frac{\varepsilon+\lambda}{2}\right)\right|,\left|\tanh\left(\frac{-\varepsilon+\lambda}{2}\right)\right|\right\\},$

where $\lambda:=\log{\left(\frac{p}{1-p}\right)}$. When $p=0.5$ and
$\varepsilon=1$, we have ${\rm Adv}_{p}(Q_{S}),,{\rm
AdvI}_{p}(Q_{S},z_{0})\leq 0.462$ for any $z_{0}\in{\cal Z}$.

###### Proof.

See the complete proof in Appendix D.3 ∎

###  3.2 For the generalized metrics

As previously mentioned, membership privacy leakage is a highly imbalanced
problem (Jayaraman et al., 2020; Rezaei & Liu, 2020). For example, the
training set in a medical dataset may consist of data from the patients
admitted to a clinical study with a particular health condition and the
distribution ${\cal D}$ may represent data from all patients (in the world).
Notably, previous works (Hayes et al., 2019; Hilprecht et al., 2019; Mukherjee
et al., 2019) used metrics such as accuracy, precision and recall to measure
the privacy risks even for the highly imbalanced setting where $p=0.1$ (Hayes
et al., 2019). Although these attacks result in high accuracy, precision or
recall during the privacy evaluation stage, it has been shown to suffer from a
high false positive rate(Rezaei & Liu, 2020), thus is less useful in practice.

To overcome these issues, prior work Jayaraman & Evans (2019); Jayaraman et
al. (2020) proposes to use the positive predictive value (PPV), which is
defined as the ratio of true members predicted among all the positive
membership predictions made by an adversary ${\cal A}$. Here, we allow users
even more flexibility by adopting the generalized metric defined in Equation
8. Through Experiment 1, we define the following metric to measure the
membership privacy risk under the generalized metric.

######  Definition 4.

The membership advantage of an adversary that has access to a trained model
via a query $Q_{S}$ ${\cal A}$ under the generalized metric $\ell$ is defined
as

${\rm Adv}_{\ell,p}(Q_{S},{\cal A})=\ell({\cal A}(Q_{S}(\cdot)),{\cal E}({\cal
D},S,Q_{S})),$

where ${\cal E}({\cal D},S,Q_{S})$ is the distribution generated by Experiment
1.

After introducing the membership advantage under the generalized metric $l$,
we define the optimal membership advantage as the maximum membership
advantage.

######  Definition 5.

The optimal membership advantage is defined as

${\rm Adv}_{\ell,p}(Q_{S})=\max_{\cal A}{\rm Adv}_{\ell,p}(Q_{S},{\cal A}).$

Similar to the accuracy-based metric, the optimal adversary for the
generalized metric is the Bayes optimal classifier. While we mention that the
exact function depends on the dataset and the metric, in some cases there
exists a closed form solution for it.

######  Lemma 2.

Given the query $Q_{S}$, the data distribution ${\cal D}$, the training set
$S$ and the prior probability $p$, if $b_{11}=b_{01}$ and $b_{10}=b_{00}$,
then the Bayes optimal classifier ${\cal A}^{*}$ maximizing membership
advantage under the generalized metric $l$ asymptotically is given by

$\displaystyle{\cal A}^{*}(Q_{S}(z))={\rm
sgn}\left({\mathbb{P}}\left(m=1|Q_{S}(z)\right)-t_{l}\right)\;,$ (16)

where the probability is taken over the sample pair $(z,m)$ drawn from ${\cal
E}({\cal D},S,Q_{S})$ and
$t_{l}=\frac{a_{00}-a_{10}}{a_{11}-a_{10}-a_{01}+a_{00}}$.

###### Proof.

See the complete proof in Appendix D.2 ∎

As a sanity check, for accuracy, we have $t_{ACC}=\frac{1}{2}$. A wide range
of performance measure under class imbalance settings can be seen as instances
of the family of metrics under Lemma 2. For example, AM measure (Menon et al.,
2013) defined as $\text{AM}:=1/2\;\left(\text{TPR}+\text{TNR}\right)$ has
optimal threshold $t_{AM}=p$. As another example, TPR or recall has the
optimal threshold $t_{TPR}=0$, which means in practice, we can always predict
as positive to get the highest recall.

Similar to the case of the accuracy based metric, we define the individual
privacy risk for the generalized metric as follows:

######  Definition 6.

The individual privacy risk of a sample $z_{0}$ for a query $Q_{S}$ under
generalized metric $\ell$ is defined as

${\rm AdvI}_{l,p}(Q_{S},z_{0})=\ell({\cal A}^{*}(Q_{S}(\cdot)),{\cal E}({\cal
D},S,Q_{S}|Q_{S}(z_{0})),$

where ${\cal A}^{*}$ is the Bayes Optimal Adversary under the generalized
metris and ${\cal E}({\cal D},S,Q_{S}|Q_{S}(z_{0})$ is the distribution
generated by Experiment 1 given $Q_{S}(z)=Q_{S}(z_{0})$.

######  Remark 1.

While we focus on a single query function in this section, the result can be
easily extended to a set of queries
$\\{Q_{S}^{1},Q_{S}^{2},\cdot,Q_{S}^{q}\\}$ by computing the maximum risk of
all the queries $Q_{S}^{i}$, $i=1,\cdots,q$.

##  4 Estimation of the Individual Privacy Risk

In the ideal world, we have a full access to the data distribution ${\cal D}$.
As a result, we could get the exact numbers to describe the individual privacy
risk and the optimal membership advantage by Definition 2\- 6. However, this
could hardly be true in practice. For example, it is almost impossible to get
access to the health records of all the patients around the world, while a
simple random sample could be feasible. Previous studies (Yeom et al., 2018;
Jayaraman et al., 2020) dealt with this by assuming the existence of a subset
to fully represent a distribution. For instance, Jayaraman et al. (2020) used
a Texas hospital data set consisting of 67,000 patient records as the data
distribution to validate their framework. However, their framework could
hardly be adapted to the case when the 67,000 patient records are only part of
the population of interest.

To conquer this limitation, we extend to the situation when one can only
access the training set $S$ and the data distribution ${\cal D}$ via a simple
random sample. Moreover, we will provide a practical and principled approach
to estimate the individual privacy risk for both the accuracy-based and the
generalized metrics, In order to achieve this, we first propose an experiment.
Note that this experiment would also be used to estimate the optimal
membership advantage in the following section.

######  Experiment 2.

Let ${\cal D}$ be the data distribution on ${\cal Z}$. We first have a
training set $S\sim{\cal D}^{n}$ with size $|S|=n$ and have a trained model.
An adversary would access the trained model via the query function
$Q_{S}:{\cal Z}\rightarrow{\cal Q}$. The ratio $p$ is the prior probability.
The sampling process proceeds as follows:

  1. 1.

Uniformly sample $N_{1}=Np$ points $x_{1},\cdots,x_{N_{1}}$ from $S$. Assign
$z_{i}=x_{i}$ and $m_{i}=1$, for $i=1,\cdots,N_{1}$.

  2. 2.

Uniformly sample $N_{2}=(1-p)N$ points $y_{1},\cdots,y_{N_{2}}$ from ${\cal
D}\backslash S$. Assign $z_{i}=y_{i-N_{1}}$ and $m_{i}=-1$, for
$i=N_{1}+1,\cdots,N_{2}$.

  3. 3.

Create a consolidated set of samples $\\{(z_{i},m_{i})\\}_{i=1}^{N}$, where
$m_{i}=-1$ if $i\leq N_{1}$ and $m_{i}=0$ if $i\geq N_{1}$.

We assume that $Np$ and $N(1-p)$ are always integers for simplicity in this
paper. But keep in mind that all the algorithms, lemmas and theorems could be
easily extended to the case when this assumption does not hold.

###  4.1 For the accuracy-based metric

By Theorem 1, we have ${\rm AdvI}_{p}(Q_{S},z)=|f_{p}(z)|$. Thus, it is
sufficient to estimate $f_{p}(z)$ to estimate ${\rm AdvI}_{p}(Q_{S},z)$. In
the following sub-sections, we describe the construction of consistent
estimators for $f_{p}(z)$.For simplicity, denote
$P:={\mathbb{P}}(Q_{S}(z)|m=1)$ and $Q:={\mathbb{P}}(Q_{S}(z)|m=-1)$. Notice
that the support is the domain of the query ${\cal Q}$. Now the problem
reduces to the estimation of the statistical difference between $P$ and $Q$
defined by Equation 14.

####  4.1.1 Discrete queries

When $Q_{S}(z)\in{\cal Q}$ is discrete, for a particular output $j\in{\cal
Q}$, let $p_{j}={\mathbb{P}}(Q_{S}(z)=j|m=1)$ and
$q_{j}={\mathbb{P}}(Q_{S}(z)=j|m=-1)$. Frequency-based plug-in methods have
been used empirically in similar settings(Mukherjee et al., 2019; Yaghini et
al., 2019). We simply collect samples from Experiment 2 and plug-in the
fraction to estimate $p_{j}$ and $q_{j}$. Then we account for the estimation
error of this process by using a Clopper-Pearson confidence interval (Clopper
& Pearson, 1934). We find the $\frac{\delta}{2}$-Clopper Pearson lower bound
for $p_{j}$ denoted as $\hat{p}_{j,\rm{lower}}$ and the
$\frac{\delta}{2}$-Clopper Pearson upper bound for $p_{j}$ denoted as
$\hat{p}_{j,\rm{upper}}$. Finally we have the following theorem.

######  Theorem 2.

Let $\\{X_{1},\cdots,X_{N_{1}},Y_{1},\cdots,Y_{N_{2}}\\}$ randomly sampled by
Experiment 2. If $Q_{S}(z)=j$, then

  1. 1.

A consistent estimator of $f_{p}(z)$ is given as follows:

$\frac{\hat{p}_{j}p-\hat{q}_{j}(1-p)}{\hat{p}_{j}p+\hat{q}_{j}(1-p)},$

where
$\hat{p}_{j}=\frac{1}{N_{1}}\sum_{i=1}^{N_{1}}\mathbb{I}(Q_{S}(X_{i})=j)$, and
$\hat{q}_{j}=\frac{1}{N_{2}}\sum_{i=1}^{N_{2}}\mathbb{I}(Q_{S}(Y_{i})=j)$.

  2. 2.

The $(1-\delta)$-confidence interval $C_{1-\delta}(z)$ of $f_{p}(z)$ is

$\displaystyle\left[\frac{\hat{p}_{j,\rm{lower}}p-\hat{q}_{j,\rm{upper}}(1-p)}{\hat{p}_{j,\rm{lower}}p+\hat{q}_{j,\rm{upper}}(1-p)},\frac{\hat{p}_{j,\rm{upper}}p-\hat{q}_{j,\rm{lower}}(1-p)}{\hat{p}_{j,\rm{upper}}p+\hat{q}_{j,\rm{lower}}(1-p)}\right]\;.$

###### Proof.

See the complete proof in Appendix D.4 ∎

####  4.1.2 Continuous queries

Consider when $Q_{S}$ is a continuous query. Let
$p(x)={\mathbb{P}}(Q_{S}(z)=x|m=1)$ and $q(x)={\mathbb{P}}(Q_{S}(z)=x|m=-1)$.
We first use Kernel Density Estimators (KDE) for both $p(x)$ and $q(x)$.

Recall that for samples $v_{1},v_{2},\cdots,v_{N}\in{\mathbb{R}}^{d}$ from an
unknown distribution $P$ defined by a probability density function $p$, a KDE
with bandwidth $h$ and kernel function $K$ is given by

$\displaystyle\frac{1}{Nh^{d}}\sum_{i=1}^{N}K\left(\frac{v-v_{i}}{h}\right)\;.$
(17)

And we have a plug-in confidence interval for KDE (Chen, 2017) (see Lemma 3).
Using this, we have a consistent estimator for $f_{p}(z)$ with a confidence
interval to estimate the uncertainty.

######  Theorem 3.

Let $Q_{S}$ be a continuous query and $z\in{\cal Z}$. Let
$\\{X_{1},\cdots,X_{N_{1}},Y_{1},\cdots,Y_{N_{2}}\\}$ randomly sampled from
Experiment 2.

  1. 1.

A consistent estimator of $f_{p}(z)$ is given as follows:

$\frac{\widehat{p}_{N}(Q_{S}(z))p-\widehat{q}_{N}(Q_{S}(z))(1-p)}{\widehat{p}_{N}(Q_{S}(z))p+\widehat{q}_{N}(Q_{S}(z))(1-p)},$

where
$\widehat{p}_{N}(Q_{S}(z))=\frac{1}{N_{1}h^{d}}\sum_{i=1}^{N_{1}}K\left(\frac{Q_{S}(z)-Q_{S}(X_{i})}{h}\right)$,
and
$\widehat{q}_{N}(Q_{S}(z))=\frac{1}{N_{2}h^{d}}\sum_{i=1}^{N_{2}}K\left(\frac{Q_{S}(z)-Q_{S}(Y_{i})}{h}\right)$.

  2. 2.

The $(1-\delta)$-confidence interval $C_{1-\delta}(z)$ of $f_{p}(z)$ is

$\displaystyle\left[\frac{\widehat{p}_{\rm lower}(Q_{S}(z))p-\widehat{q}_{\rm
upper}(Q_{S}(z))(1-p)}{\widehat{p}_{\rm lower}(Q_{S}(z))p+\widehat{q}_{\rm
upper}(Q_{S}(z))(1-p)},\frac{\widehat{p}_{\rm
upper}(Q_{S}(z))p-\widehat{q}_{\rm lower}(Q_{S}(z))(1-p)}{\widehat{p}_{\rm
upper}(Q_{S}(z))p+\widehat{q}_{\rm lower}(Q_{S}(z))(1-p)}\right],$

where $[\widehat{p}_{\rm lower},\widehat{p}_{\rm upper}],[\widehat{q}_{\rm
lower},\widehat{q}_{\rm upper}]$ are $(1-\delta/2)$ confidence intervals of
$p(Q_{S}(z))$ and $q(Q_{S}(z))$ respectively. Both confidence intervals are
derived by Lemma 3.

###### Proof.

See the complete proof in Appendix D.5 ∎

###  4.2 For the generalized metrics

Similar to the case of the accuracy-based metric, we propose a consistent
estimator to estimate the individual privacy risk for the generalized metric.
Though there is not always a closed-form expression for the individual privacy
risk for the generalized metric, we are able to provide an explicit formula
given a few conditions. The following theorem provides a way to calculate the
confidence interval for the generalized metric given a known $t_{\ell}$.

######  Theorem 4.

Let $c_{1}=(a_{0}+a_{01})$, $c_{2}=(a_{0}+a_{00})$, $c_{3}=(a_{11}-a_{01})$,
$c_{4}=(a_{10}-a_{00})$, $d_{1}=(b_{0}+b_{01})$, $d_{2}=(b_{0}+b_{00})$,
$d_{3}=(b_{11}-b_{01})$, $d_{4}=(b_{10}-b_{00})$, and $z_{0}$ be a sample
$\in{\cal Z}$. Under the following conditions:

  1. 1.

The Bayes optimal classifier ${\cal A}^{*}$ for the generalized metric $l$ can
be written in the form of ${\rm sgn}(\eta(x)-t_{\ell})$, and $t_{l}$ is a
known constant;

  2. 2.

${\mathbb{P}}(m=1|Q_{S}(z)=Q_{S}(z_{0}))\neq t_{l}$;

  3. 3.

$pd_{1}{\mathbb{P}}(Q_{S}(z_{0})|m=1)+d_{2}(1-p){\mathbb{P}}(Q_{S}(z_{0})|m=-1)+pd_{3}{\mathbb{P}}(Q_{S}(z_{0})|m=1)\mathbb{I}({\mathbb{P}}(m=1|Q_{s}(z_{0}))>t_{l})+d_{4}(1-p){\mathbb{P}}(Q_{S}(z_{0}|m=-1)\mathbb{I}({\mathbb{P}}(m=1|Q_{s}(z_{0}))>t_{l})\neq
0$;

we have

  1. 1.

A consistent estimator of $AdvI_{l,p}(Q_{S},z_{0})$ can be given as follows:

$\frac{c_{1}\hat{p}p+c_{2}\hat{q}(1-p)+c_{3}\hat{p}p\mathbb{I}((1-t_{l})p\hat{p}>t_{l}(1-p)\hat{q})+c_{4}\hat{q}(1-p)\mathbb{I}((1-t_{l})p\hat{p}>t_{l}(1-p)\hat{q})}{d_{1}\hat{p}p+d_{2}\hat{q}(1-p)+d_{3}\hat{p}p\mathbb{I}((1-t_{l})p\hat{p}>t_{l}(1-p)\hat{q})+d_{4}\hat{q}(1-p)\mathbb{I}((1-t_{l})p\hat{p}>t_{l}(1-p)\hat{q})}.$
(18)

If ${\cal Q}$ is discrete and $Q_{S}(z_{0})=j$, then $\hat{p}=\hat{p}_{j}$ and
$\hat{q}=\hat{q}_{j}$, as defined in Theorem 2. If $Q_{S}$ is a continuous
query, then $\hat{p}=\hat{p}_{N}(Q_{S}(z_{0}))$ and
$\hat{q}=\hat{q}_{N}(Q_{S}(z_{0}))$, as defined in Theorem 3

  2. 2.

The $(1-\delta)$-confidence interval of $AdvI_{l,p}(Q_{S},z_{0})$ follows by
plugging the $(1-\delta/2)$ confidence intervals of $p$ and $q$ –
$[\hat{p}_{lower},\hat{p}_{upper}]$and $[\hat{q}_{lower},[\hat{q}_{upper}]$
into Equation 18. In the case of discrete queries,
$[\hat{p}_{\rm{lower}},\hat{p}_{\rm{upper}}]=[\hat{p}_{\rm{j,lower}},\hat{p}_{\rm{j,upper}}]$,
where $Q_{S}(z_{0})=j$, , as defined in Theorem 2. In the case of continuous
queries, $[\hat{p}_{\rm{lower}},\hat{p}_{\rm{upper}}]=[\widehat{p}_{\rm
lower}(Q_{S}(z_{0})),\widehat{p}_{\rm upper}(Q_{S}(z_{0}))]$, as defined in
Theorem 3. $\hat{q}_{lower}$ and $\hat{q}_{upper}$ are defined in a similar
manner.

###### Proof.

See the complete proof in Appendix D.6. ∎

A wide range of performance measures satisfy Condition 1. For example, AM
measure (Menon et al., 2013) has the optimal threshold $t_{\rm AM}=p$, and
$TPR$ has the optimal threshold $t_{\rm TPR}=0$. Condition 2 can be waived if
$c_{3}=c_{4}=d_{3}=d_{4}=0$. Metrics such as accuracy, recall and specificity
meet the above condition. Condition 3 assumes a nonzero expected value of the
denominator in Equation 18.As for the confidence interval in 2, there could
exist a closed-form expression if for example, ${\rm AdvI}(Q_{S},z_{0})$ is a
monotonic function in terms of
${\mathbb{P}}(Q_{S}(z_{0}))|m=1)/{\mathbb{P}}(Q_{S}(z_{0}))|m=-1)$. Otherwise,
an optimization problem would need to be solved, as outlined in 2.

##  5 Estimation of the Optimal Membership Advantage

In the above section, we proposed to estimate individual privacy risk for both
the accuracy-based and generalized metrics with theoretical justifications. In
this section, we further develop methods to estimate the optimal membership
advantage ${\rm Adv}_{p}(Q_{S})$.

###  5.1 For the accuracy-based metric

We first propose a consistent estimator for the optimal membership advantage
given a discrete query.

######  Theorem 5.

Let $X_{1},\cdots,X_{N_{1}}$ be i.i.d. random variables drawn from $S$ and
$Y_{1},\cdots,Y_{N_{2}}$ be i.i.d. random variables drawn from ${\cal
D}\backslash S$ in Experiment 2. Assume that ${\cal Q}$ is a finite set.
Define

$\displaystyle W_{N}=W(X_{1},\cdots,X_{N_{1}},Y_{1},\cdots,Y_{N_{2}})$
$\displaystyle=\sum\limits_{j\in{\cal
Q}}|\frac{p}{N_{1}}\sum\limits_{i=1}^{N_{1}}\mathbb{I}(Q_{S}(X_{i})=j)-\frac{1-p}{N_{2}}\sum\limits_{i=1}^{N_{2}}\mathbb{I}(Q_{S}(Y_{i})=j)|\;.$

Then (1) $W_{N}$ is a consistent estimator of the optimal membership advantage
${\rm Adv}_{p}(Q_{S})$; (2)
$\mathbb{P}(|W_{N}-\mathbb{E}W_{N}|\geq\sqrt{\frac{2}{N}\log(\frac{2}{\delta})})\leq\delta$.

###### Proof.

See the complete proof in Appendix D.7. ∎

We then propose a consistent estimator for the optimal membership advantage
given a continuous query.

######  Theorem 6.

Let $X_{1},\cdots,X_{N_{1}}$ be i.i.d. random variables drawn from $S$, and
$Y_{1},\cdots,Y_{N_{2}}$ be i.i.d. random variables drawn from ${\cal
D}\backslash S$ in Experiment 2, where $N_{1}=Np$ and $N_{2}=N(1-p)$. Assume
that ${\cal Q}=\mathbb{R}^{d}$ and $h\rightarrow 0$, as $N\rightarrow\infty$.
Define

$\displaystyle U_{N}$ $\displaystyle=$ $\displaystyle
U(X_{1},\cdots,X_{N_{1}},Y_{1},\cdots,Y_{N_{2}})$ $\displaystyle=$
$\displaystyle\int|\frac{p}{N_{1}h^{d}}\sum\limits_{i=1}^{N_{1}}K(\frac{x-X_{i}}{h})-\frac{1-p}{N_{2}h^{d}}\sum\limits_{i=1}^{N_{2}}K(\frac{x-Y_{i}}{h})|dx.$

Then (1) $U_{N}$ is a consistent estimator of the optimal membership advantage
${\rm Adv}_{p}(Q_{S})$; (2)
$\mathbb{P}(|U_{N}-\mathbb{E}U_{N}|\geq\sqrt{\frac{2}{N}\log(\frac{2}{\delta})})\leq\delta$.

###### Proof.

See the complete proof in Appendix D.8. ∎

The practical estimation of the optimal membership advantage for the accuracy-
based metric is described in Algorithm 1 using Monte Carlo integration.

0: number of samples $N$, prior of membership $p$, training set $S$, query
function $Q_{S}$, confidence level $\delta$

0: the optimal membership advantage estimate $\widehat{{\rm Adv}}_{p}(Q_{S})$

1: Perform Experiment 2 and draw one set of samples $\\{z_{i}\\}_{i=1}^{N}$
with membership $\\{m_{i}\\}_{i=1}^{N}$ respectively.

2: if $Q_{S}$ is discrete then

3: Using the tuples $\\{(z_{i},m_{i})\\}_{i=1}^{N}$ calculate $W_{N}$

4: $\widehat{{\rm Adv}}_{p}(Q_{S})\leftarrow W_{N}$

5: else

6: Using the tuples $\\{(z_{i},m_{i})\\}_{i=1}^{N}$ calculate $U_{N}$

7: $\widehat{{\rm Adv}}_{p}(Q_{S})\leftarrow U_{N}$

8: end if

Algorithm 1 Practical estimation of the optimal membership advantage
$\widehat{{\rm Adv}}_{p}(Q_{S})$

###  5.2 For the generalized metrics

For the generalized metrics, the optimal membership advantage ${\rm
Adv}_{\ell,p}(Q_{S})$ is estimated as the mean individual privacy risk as
described in Algorithm 2. In this subsection, we assume that the optimal
adversary can be written as ${\rm sgn}(\eta(x)-t_{\ell})$. It is
straightforward to adapt the following algorithm to the case where the optimal
adversary takes another form ${\rm sgn}(t_{\ell}-\eta(x))$ in Equation 9.

In order to estimate the individual privacy risk, we provide two different
methods depending on whether the threshold $t_{\ell}$ in Equation 9 is known.
When $t_{l}$ is available, we can simply apply Theorem 4. When $t_{l}$ is not
known, we adapted Algorithm 1 from Koyejo et al. (2014) to estimate the
individual privacy risk. As shown in Algorithm 2, we first split the samples
into three partitions, and use the first partition to obtain the estimator
$\widehat{\eta}(Q_{S}(z))$ for $\eta(Q_{S}(z))$. Similarly as in Theorem 4, if
$Q_{S}(z)$ is discrete, we use the plug-in estimator directly, if $Q_{S}(z)$
is continuous, we apply the KDE estimator. Then we use the second partition to
find the optimal $t_{\ell}$ that maximizes the generalized metric. Finally, we
average over the third partition, and obtain an empirical optimal membership
advantage.

0: number of samples $N$, prior of membership $p$, training set $S$, query
function $Q_{S}$, generalized metric $\ell$

0: the empirical membership privacy ${\rm Adv}_{\ell,p}(Q_{S})$ estimate

Perform Experiment 2 and draw three sets of samples
$\\{z_{i}^{(1)}\\}_{i=1}^{N}$, $\\{z_{i}^{(2)}\\}_{i=1}^{N}$ and
$\\{z_{i}^{(3)}\\}_{i=1}^{N}$ with membership $\\{m_{i}^{(1)}\\}_{i=1}^{N}$,
$\\{m_{i}^{(2)}\\}_{i=1}^{N}$ and $\\{m_{i}^{(3)}\\}_{i=1}^{N}$ respectively.

2: Estimate $\eta(Q_{S}(z))$ by samples from $\\{z_{i}^{(1)}\\}_{i=1}^{N}$.

$t_{\ell}\leftarrow\operatorname*{arg\,max}_{x\in(0,1)}\frac{1}{N}\sum_{i=1}^{N}\ell({\rm
sgn}(\widehat{\eta}(Q_{S}(z_{i}^{(2)}))-x),m_{i}^{(2)})$

4: Let ${\cal A}^{*}(Q_{S}(z))={\rm sgn}(\widehat{\eta}(Q_{S}(z))-t_{\ell})$.

Evaluate $\widehat{{\rm
Adv}}_{\ell,p}(Q_{S})\leftarrow\frac{1}{N}\sum_{i=1}^{N}\ell({\cal
A}^{*},y_{i}^{(3)})$.

Algorithm 2 Practical estimation of the optimal membership advantage ${\rm
Adv}_{\ell,p}(Q_{S})$ under a generalized metric $\ell$

##  6 Experiments

In this section, we demonstrate how to use MACE by performing practical
membership privacy estimation on trained generative models.

###  6.1 Setup

The different GAN architectures used were WGAN-GP (Gulrajani et al., 2017) (on
CIFAR-10 & MNIST), JS-GAN (Goodfellow et al., 2014) (on skin-cancer MNIST) and
privGAN (Mukherjee et al., 2019) (on MNIST). A JS-GAN is the original GAN
formulation which uses a Jensen-Shannon divergence based loss. A WGAN-GP is an
improved GAN formulation with a Wasserstein distance based loss with a
gradient penalty. A privGAN is a GAN formulation that utilizes multiple
generator-discriminator pairs and has been empirically shown to provide
membership privacy.

Our experiments would be based on the following three real-world datasets:
MNIST, CIFAR-10 and skin cancer MNIST. MNIST contains gray scale images of
handwritten digits with 70000 digits from 0 to 9. CIFAR-10 comprises of 10
classes of 32 x32 RGB colored images with 60000 images in the whole dataset.
Both of them are commonly used in the GAN literature. Additionally, to
demonstrate a real world use case in healthcare, we use the skin cancer MNIST
dataset (Tschandl et al., 2018) which comprises of 10,000 $64\times 64$ RGB
images of skin lesions (both cancerous and benign).

Following the common practice of membership inference attacks on generative
models (Hayes et al., 2019; Mukherjee et al., 2019; Chen et al., 2019), we
choose a random $10\%$ subset of the entire dataset as a training set to show
overfitting. In sub-section 6.3, $10\%$ of the training images were corrupted
by placing a white box at the center of images (with no changes to the non-
training set images). For all the experiments, we set the confidence level
$\delta=0.05$. To create discrete queries, we bin the continuous interval into
$100^{d}$ bins (where $d$ is the dimension of the query).

Figure 1: a) Comparison of discrete vs continuous queries against the WGAN-GP
on the CIFAR-10 dataset. b) Comparison of queries against the generator and
the discriminator on the skin-cancer MNIST dataset (for JS-GAN). In all cases
$\delta$ is set to $0.05$.

###  6.2 Estimation of the optimal membership advantage

####  6.2.1 For the accuracy-based metric

We first demonstrate the utility of our estimators of the optimal membership
advantage under the accuracy-based metric with regard to different query
types. In Figure 1a we show the applicability of our discrete and continuous
estimators using the discriminator score as the query. Additionally, we
compare the estimated performance of the optimal adversaries against a
heuristic adversary that uses the same query (Hayes et al., 2019). We find
that for both discrete and continuous queries, the estimated membership
advantage is higher for the optimal adversary compared to the heuristic
adversary. We note that the continuous query is seen to yield somewhat poorer
performance than the discrete query, most likely due to sub-optimal selection
of the KDE bandwidth. Optimal choice of the binning/KDE hyperparameters are
beyond the scope of this paper but we direct readers to existing papers on
this topic (Chen, 2015; Knuth, 2019). In Figure 1b we show the utility of our
estimators on queries against accessible models and accessible datasets using
a query of each type. We use the discriminator score as an example of queries
on an accessible model and the query described in Equation 2 as an example of
queries on an accessible synthetic dataset. Additionally, we compare our
optimal membership advantage with the membership advantage of SOTA heuristic
adversaries that use the similar queries as in (Hayes et al., 2019; Hilprecht
et al., 2019). As widely reported (Hayes et al., 2019; Mukherjee et al.,
2019), we find among our experiments that the optimal membership advantage is
a lot smaller when adversaries gain access to the datasets compared to when
they get direct access to the model. Furthermore, our optimal membership
advantage estimates are higher than the SOTA heuristic adversaries in both
settings. This validates Theorem 5 and 6, and demonstrates that our estimators
are good estimators for the optimal membership advantage that would bound all
the membership advantages including those due to the SOTA heuristic
adversaries.

Figure 2: Comparison of single dimensional vs multi-dimensional queries
against (a) privGAN on the MNIST dataset and (b) WGAN-GP on the skin-cancer
MNIST dataset.

Next, we demonstrate the applicability of our estimators to multi-dimensional
queries. In Figure 2a we compare the estimated membership advantage for a
single and a multi-dimensional queries against privGAN (Mukherjee et al.,
2019). We see that the estimated membership advantage with the multi-
dimensional query is much higher than that of the 1-d query used in the
privGAN paper. This indicates that while the privGAN is less likely to suffer
from overfitting than the JS-GAN, releasing multiple discriminators could
potentially increase it’s privacy risk. In Figure 2b, we compare the estimated
membership advantage for two single and one multi-dimensional queries against
the JS-GAN. The multi-dimensional query is indeed a hybrid query that is the
combination of the two 1-d queries (Equation 4). Intuitively, a hybrid query
should impose a higher privacy risk than an individual query. Our experiment
has validated this assumption and shown that the hybrid query has a higher
estimated membership advantage than each individual 1-d query.

Figure 3: The membership advantages of the optimal adversary vs. the SOTA
adversary against WGAN-GP under different metrics on the MNIST and CIFAR-10
datasets. For the sake of consistency, the membership advantage estimation for
both metrics is done using the method used for the generalized metric.

####  6.2.2 Generalized metric

To demonstrate how to apply MACE under the generalized metrics, we
qualitatively compare the membership advantages under AM and the accuracy-
based metric. We set $p=0.1$ here to construct an imbalanced dataset. In
Figure 3, under the accuracy-based metric, we compare the optimal membership
advantage (the membership advantage of the optimal adversary) with the
membership advantage of the heuristic adversary defined in (Hayes et al.,
2019) using the discriminator score as the query. We find the heuristic
adversary has comparable performance to the optimal adversary under the
accuracy-based metric. We then repeat the same experiment but evaluating the
membership privacy risk in terms of AM, as shown in Figure 3. In this case,
the performance of the optimal adversary is consistently better than the
heuristic adversary. This suggests that heuristic adversaries designed with a
particular metric in mind may not be as effective when performance is measured
in terms of a different metric. As a result, it would underestimate the true
membership privacy risks using these pre-selected adversaries that were
designed to optimize for another metric. This further highlights the utility
of being able to use different metrics in our privacy framework.

Figure 4: Application of individual privacy risk estimation. a) Example
corrupted and uncorrupted images from skin-cancer MNIST, along with their
estimated individual privacy risk. b) Comparison of average individual privacy
risk between corrupted and uncorrupted images.

###  6.3 Estimation of individual privacy risks

Having demonstrated the utility of our estimators for estimating the optimal
membership advantage, here we demonstrate the utility of our estimators for
the individual privacy risk. It is well known that samples from minority sub-
groups can often be more vulnerable to membership inference attacks (Yaghini
et al., 2019). Thus it is necessary to reveal the membership privacy at the
individual level to reflect the true risks faced by the minority group. For
the purposes of a simple demonstration, we constructed a toy dataset where
$10\%$ of the images were corrupted (described in section 6.1). We then
trained a JS-GAN on this dataset and estimated the individual privacy risk of
samples against the discriminator score query using the individual privacy
risk estimator described in section 4.1.1. In Figure 4, we show (both
qualitatively and quantitatively) that the corrupted images on average have a
higher individual privacy risk than the uncorrupted images. This shows that
outlier samples or certain sub-groups can be more vulnerable than the rest of
the population, and the optimal membership advantage alone may not capture
this information. It is worth noting here that there are particularly severe
ramifications of higher privacy leakage risks of minorities in healthcare
settings, specially if such information points to the disease status. Data and
model owners in such cases can consider retraining their generative model by
omitting such sub-groups of samples.

##  7 Conclusion and Remarks

We developed the first formal framework that provides a certificate for the
membership privacy risk of a trained generative model posed by adversaries
having query access to the model at both the individual and population level.
While our theory works regardless of the query dimension, we do not have a
practical way to generate a meaningful certificate for the high-dimensional
case. This would be a focus of our future work. Our framework works for a
large family of metrics, allowing users flexible ways to measure the
membership risk. Through experiments on multiple datasets, queries, model
types and metrics, we show the practical applicability of our framework in
measuring the optimal membership advantage as well as the individual privacy
risk. Finally, to wrap up the paper, we re-visit our fictional example from
the introduction to explain how MACE can help such data/model owners:

  * •

By comparing the optimal membership advantage against a given trained model of
queries that access the model via different practical queries, the model
owners can determine the relative risk of releasing: i) the complete trained
model, ii) parts of the trained model, iii) the synthetic dataset.

  * •

For model owners who are only interested in releasing synthetic data, MACE can
help identify how much data can be released for a desired level of membership
advantage (see Appendix for example).

  * •

For datasets containing sensitive groups of samples, MACE allows to estimate
the group-level membership privacy risk through the estimation of the
individual privacy risk and identify the most vulnerable minorities.

While we focus on generative models in this paper, our framework can be
applied to discriminative models as well. An example application of our
framework to a discrminative model is shown in the Appendix. While we focus on
the theoretical aspects of membership privacy estimation in this paper, future
work could look at using MACE to design new queries which can lead to stronger
MIAs and a better understanding of membership privacy risks of different
generative models. Another important direction is to derive a consistent
estimator of the optimal membership advantage for the generalized metric by
adapting Koyejo et al. (2014). It would also be interesting to extend our work
to high-dimensional queries such as certain layers of the generator and
discriminator.

## References

  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan.  _arXiv preprint arXiv:1701.07875_ , 2017.
  * Berthelot et al. (2017) David Berthelot, Thomas Schumm, and Luke Metz.  Began: Boundary equilibrium generative adversarial networks.  _arXiv preprint arXiv:1703.10717_ , 2017.
  * Chen et al. (2019) Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz.  Gan-leaks: A taxonomy of membership inference attacks against gans.  _arXiv preprint arXiv:1909.03935_ , 2019.
  * Chen et al. (2021) Junjie Chen, Wendy Hui Wang, Hongchang Gao, and Xinghua Shi.  Par-gan: Improving the generalization of generative adversarial networks against membership inference attacks.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pp. 127–137, 2021.
  * Chen (2015) Su Chen.  Optimal bandwidth selection for kernel density functionals estimation.  _Journal of Probability and Statistics_ , 2015, 2015.
  * Chen (2017) Yen-Chi Chen.  A tutorial on kernel density estimation and recent advances.  _Biostatistics & Epidemiology_, 1(1):161–187, 2017.
  * Clopper & Pearson (1934) Charles J Clopper and Egon S Pearson.  The use of confidence or fiducial limits illustrated in the case of the binomial.  _Biometrika_ , 26(4):404–413, 1934.
  * Devroye & Gyorfi (1985) L. Devroye and L. Gyorfi.  _Nonparametric Density Estimation: The L1 View_.  Wiley Interscience Series in Discrete Mathematics. Wiley, 1985.  ISBN 9780471816461.  URL https://books.google.com/books?id=ZVALbrjGpCoC.
  * Devroye et al. (2013) Luc Devroye, László Györfi, and Gábor Lugosi.  _A probabilistic theory of pattern recognition_ , volume 31.  Springer Science & Business Media, 2013.
  * Elkan (2001) Charles Elkan.  The foundations of cost-sensitive learning.  In _International joint conference on artificial intelligence_ , volume 17, pp. 973–978. Lawrence Erlbaum Associates Ltd, 2001.
  * Georges-Filteau & Cirillo (2020) Jeremy Georges-Filteau and Elisa Cirillo.  Synthetic observational health data with gans: from slow adoption to a boom in medical research and ultimately digital twins?  _arXiv preprint arXiv:2005.13510_ , 2020.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In _Advances in neural information processing systems_ , pp. 5767–5777, 2017.
  * Hayes et al. (2019) Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro.  Logan: Membership inference attacks against generative models.  _Proceedings on Privacy Enhancing Technologies_ , 2019(1):133–152, 2019.
  * Hilprecht et al. (2019) Benjamin Hilprecht, Martin Härterich, and Daniel Bernau.  Monte carlo and reconstruction membership inference attacks against generative models.  _Proceedings on Privacy Enhancing Technologies_ , 2019(4):232–249, 2019.
  * Jayaraman & Evans (2019) Bargav Jayaraman and David Evans.  Evaluating differentially private machine learning in practice.  In _28th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 19)_, pp. 1895–1912, 2019.
  * Jayaraman et al. (2020) Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu.  Revisiting membership inference under realistic assumptions.  _arXiv preprint arXiv:2005.10881_ , 2020.
  * Jordon et al. (2018) James Jordon, Jinsung Yoon, and Mihaela van der Schaar.  Pate-gan: Generating synthetic data with differential privacy guarantees.  In _International Conference on Learning Representations_ , 2018.
  * Knuth (2019) Kevin H Knuth.  Optimal data-based binning for histograms and histogram-based probability density models.  _Digital Signal Processing_ , 95:102581, 2019.
  * Koyejo et al. (2014) Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon.  Consistent binary classification with generalized performance metrics.  In _Advances in Neural Information Processing Systems_ , pp. 2744–2752, 2014.
  * Li et al. (2013) Ninghui Li, Wahbeh Qardaji, Dong Su, Yi Wu, and Weining Yang.  Membership privacy: a unifying framework for privacy definitions.  In _Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security_, pp. 889–900, 2013.
  * Long et al. (2017) Yunhui Long, Vincent Bindschaedler, and Carl A Gunter.  Towards measuring membership privacy.  _arXiv preprint arXiv:1712.09136_ , 2017.
  * Menon et al. (2013) Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla.  On the statistical consistency of algorithms for binary classification under class imbalance.  In _International Conference on Machine Learning_ , pp. 603–611, 2013.
  * Mirza & Osindero (2014) Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Mukherjee et al. (2019) Sumit Mukherjee, Yixi Xu, Anusua Trivedi, and Juan Lavista Ferres.  Protecting gans against privacy attacks by preventing overfitting.  _arXiv preprint arXiv:2001.00071_ , 2019.
  * Nasr et al. (2021) Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini.  Adversary instantiation: Lower bounds for differentially private machine learning.  _arXiv preprint arXiv:2101.04535_ , 2021.
  * Rajotte et al. (2021) Jean-Francois Rajotte, Sumit Mukherjee, Caleb Robinson, Anthony Ortiz, Christopher West, Juan M. Lavista Ferres, and Raymond T. Ng.  Reducing bias and increasing utility by federated generative modeling of medical images using a centralized adversary.  GoodIT ’21, pp. 79–84, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450384780.
  * Rezaei & Liu (2020) Shahbaz Rezaei and Xin Liu.  Towards the infeasibility of membership inference on deep models.  _arXiv preprint arXiv:2005.13702_ , 2020.
  * Sablayrolles et al. (2019) Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou.  White-box vs black-box: Bayes optimal strategies for membership inference.  In _International Conference on Machine Learning_ , pp. 5558–5567, 2019.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pp. 3–18. IEEE, 2017.
  * Tom et al. (2020) Elysse Tom, Pearse A Keane, Marian Blazes, Louis R Pasquale, Michael F Chiang, Aaron Y Lee, Cecilia S Lee, and AAO Artificial Intelligence Task Force.  Protecting data privacy in the age of ai-enabled ophthalmology.  _Translational Vision Science & Technology_, 9(2):36–36, 2020.
  * Truex et al. (2018) Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei.  Towards demystifying membership inference attacks.  _arXiv preprint arXiv:1807.09173_ , 2018.
  * Tschandl et al. (2018) P Tschandl, C Rosendahl, and H Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. scientific data 5, 180161 (aug 2018), 2018.
  * Voigt & Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche.  The eu general data protection regulation (gdpr).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , 10:3152676, 2017.
  * Xie et al. (2018) Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou.  Differentially private generative adversarial network.  _arXiv preprint arXiv:1802.06739_ , 2018.
  * Yaghini et al. (2019) Mohammad Yaghini, Bogdan Kulynych, and Carmela Troncoso.  Disparate vulnerability: On the unfairness of privacy attacks against machine learning.  _arXiv preprint arXiv:1906.00389_ , 2019.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _2018 IEEE 31st Computer Security Foundations Symposium (CSF)_ , pp. 268–282. IEEE, 2018.

##  Appendix A Background on Generative Adversarial Networks

Generative Adversarial Networks are the most common class of generative
models. The original GAN algorithm (Goodfellow et al., 2014) learns a
distribution of a dataset by adversarially training two modules, namely, a
generator and a discriminator. The goal of the generator $G(w)$ is to learn a
transformation that would convert a random vector $w$ to a realistic data
sample. The goal of the discriminator module $D$ is to reliably distinguish
synthetic samples (generated by the generator) from real samples. The
mathematical formulation of this problem is as follows:

$\displaystyle\min_{G}\max_{D}$ $\displaystyle\mathbb{E}_{x\sim p_{r}(x)}[\log
D(x)]+$ $\displaystyle\mathbb{E}_{x\sim p_{G}(x)}[\log(1-D(x)]\;.$

Here, $P_{r}$ is the real data distribution, and $P_{G}$ is the distribution
of $G(w)$. There have been many GAN variants proposed since (Arjovsky et al.,
2017; Mirza & Osindero, 2014; Berthelot et al., 2017). In this work, we
examine our framework on the original GAN and some of its variations.

##  Appendix B Examples of common metrics that can be derived from the
generalized metric

$\displaystyle\mathrm{ACC}=\frac{{\rm TP}+{\rm TN}}{{\rm TP}+{\rm FP}+{\rm
TN}+{\rm FN}}$ $\displaystyle\mathrm{PPV}\text{ or
}\mathrm{Precision}=\frac{{\rm TP}}{{\rm TP}+{\rm FP}}$
$\displaystyle\mathrm{TPR}\text{ or
}\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
$\displaystyle\mathrm{TNR}=\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$
$\displaystyle\mathrm{WA}=\frac{w_{1}\mathrm{TP}+w_{2}\mathrm{TN}}{w_{1}\mathrm{TP}+w_{2}\mathrm{TN}+w_{3}\mathrm{FP}+w_{4}\mathrm{FN}}\;.$

##  Appendix C Background on differential privacy

The definition of $\varepsilon$-differential privacy is given as follows:

######  Definition 7 ($\varepsilon$-Differential Privacy).

We say a randomized algorithm $M$ is $\varepsilon$-differentially private if
for any pair of neighbouring databases $S$ and $S^{\prime}$ that differ by one
record and any output event $E$, we have

$\displaystyle{\mathbb{P}}(M(S)\in
E)\;\;\leq\;\;e^{\varepsilon}{\mathbb{P}}(M(S^{\prime})\in E)\;.$ (19)

##  Appendix D Detailed Proofs

###  D.1 Proof for Lemma 1

###### Proof.

As we can see from Definition 10, the membership advantage is defined as
$2{\rm Accuracy}({\cal A})-1$. This means the Bayes classifier is given by
${\rm
sgn}\left({\mathbb{P}}\left(m=1|Q_{S}(z)\right)-\frac{1}{2}\right)$(Sablayrolles
et al., 2019). We get the optimal membership advantage by plugging in this
Bayes classifier. ∎

###  D.2 Proof for Lemma 2

###### Proof.

When $b_{11}=b_{01}$ and $b_{10}=b_{00}$, our loss
$\ell=\frac{a_{0}+a_{11}\mathrm{TP}+a_{10}\mathrm{FP}+a_{01}\mathrm{FN}+a_{00}\mathrm{TN}}{b_{0}+b_{11}(\mathrm{TP}+\mathrm{FN})+b_{00}(\mathrm{TN}+\mathrm{FP})}=\frac{a_{0}+a_{11}\mathrm{TP}+a_{10}\mathrm{FP}+a_{01}\mathrm{FN}+a_{00}\mathrm{TN}}{b_{0}+b_{11}p+b_{00}(1-p)}$.
It becomes a linear combination of TP, TN, FP and FN, which is also called
cost sensitive classification as defined in (Elkan, 2001). As outlined in
(Elkan, 2001), the optimal classifier would predict $1$ when

$\displaystyle\eta(Q_{S}(z))a_{01}+(1-\eta(Q_{S}(z)))a_{00}\geq\eta(Q_{S}(z))a_{11}+(1-\eta(Q_{S}(z)))a_{10}\;.$

Rearranging the terms completes the proof. ∎

###  D.3 Proof for Theorem 1

###### Proof.

By Equation 16 and Definition 3, we have ${\rm
AdvI}_{p}(Q_{S},z_{0})=|\mathbb{P}(m=1|Q_{S}(z_{0}))-\mathbb{P}(m=-1|Q_{S}(z_{0}))|=|f_{p}(z_{0})|$,
and 1 follows.

2 follows from Equation 15 and Equation 13.

Then, we prove 3. If ${\cal A}$ equals 1 at $Q_{S}(z_{0})$, $2\mathbb{P}[{\cal
A}(Q_{S}(z))=m|Q_{S}(z)=Q_{S}(z_{0})]-1=2\mathbb{P}[m=1|Q_{S}(z_{0})]-1=\mathbb{P}[m=1|Q_{S}(z_{0})]-\mathbb{P}[m=-1|Q_{S}(z_{0})]$.If
${\cal A}$ equals -1 at $Q_{S}(z_{0})$, $2\mathbb{P}[{\cal
A}(Q_{S}(z))=m|Q_{S}(z)=Q_{S}(z_{0})]-1=2\mathbb{P}[m=-1|Q_{S}(z_{0})]-1=\mathbb{P}[m=-1|Q_{S}(z_{0})]-\mathbb{P}[m=1|Q_{S}(z_{0})]$.
Thus, the maximum equals to
$|\mathbb{P}(m=1|Q_{S}(z_{0}))-\mathbb{P}(m=-1|Q_{S}(z_{0}))|={\rm
AdvI}_{p}(Q_{S},z_{0})$.

Finally we show 4. By the post-processing property, the
$\varepsilon$-differential privacy indicates that the query output satisfies
for any record z, we have

$\displaystyle\left|\log\frac{{\mathbb{P}}(Q_{S}(z)|m=1)}{{\mathbb{P}}(Q_{S}(z)|m=-1)}\right|\leq\varepsilon$
(20)

Then 4 directly follows from 1, 2 and the fact that

$\displaystyle\frac{x-1}{x+1}=\tanh(\frac{1}{2}\ln(x)).$ (21)

∎

###  D.4 Proof of Theorem 2

###### Proof.

By the law of large numbers, we have $\hat{p}_{j}\xrightarrow{p}p_{j}$ and
$\hat{q}_{j}\xrightarrow{p}q_{j}$. Then 1 follows from Slutsky’s theorem and
and $pp(Q_{S}(z)=j)+(1-p)q(Q_{S}(z)=j)>0$. To prove 2, we first derive the
$(1-\delta/2)$ confidence intervals of $p_{j}$ and $q_{j}$ by Clopper &
Pearson (1934). Then we could divide the nominator and the denominator by
$q_{j}$, and 2 follows from the fact that $\frac{x-1}{x+1}$ is a monotonically
increasing function for $x>0$. ∎

###  D.5 Proof of Theorem 3

###### Proof.

By Chen (2017), we have $\hat{p}_{N}(Q_{S}(z))\xrightarrow{p}p(Q_{S}(z))$ and
$\hat{q}_{N}(Q_{S}(z))\xrightarrow{p}q(Q_{S}(z))$. Then 1 follows from
Slutsky’s theorem and $pp(Q_{S}(z))+(1-p)q(Q_{S}(z))>0$. To prove 2, we first
derive the confidence intervals of $p(Q_{S}(z))$ and $q(Q_{S}(z))$ by Chen
(2017). Then we could divide the nominator and the denominator by
$q(Q_{S}(z))$, and 2 follows from the fact that $\frac{x-1}{x+1}$ is a
monotonically increasing function for $x>0$. ∎

###  D.6 Proof of Theorem 4

###### Proof.

Note that the individual privacy risk at a sample $z_{0}$ is written as

$AdvI_{l,p}(Q_{S},z_{0})=\frac{a_{0}+a_{11}\mathrm{TP(Q_{S}(z_{0}))}+a_{10}\mathrm{FP(Q_{S}(z_{0}))}+a_{01}\mathrm{FN(Q_{S}(z_{0}))}+a_{00}\mathrm{TN(Q_{S}(z_{0}))}}{b_{0}+b_{11}\mathrm{TP}+b_{10}\mathrm{FP(Q_{S}(z_{0}))}+b_{01}\mathrm{FN(Q_{S}(z_{0}))}+b_{00}\mathrm{TN(Q_{S}(z_{0}))}},\;$
(22)

where $a_{0}$, $b_{0}$, $a_{ij}$ and $b_{ij}$ are pre-defined scalars for
$i=0,1$ and $j=0,1$ as in Equation 8. $\mathrm{TP}(Q_{S}(z_{0}))$,
$\mathrm{FP}(Q_{S}(z_{0}))$, $\mathrm{FN}(Q_{S}(z_{0}))$ and
$\mathrm{TN}(Q_{S}(z_{0}))$ are the conditional versions of $\mathrm{TP}$,
$\mathrm{FP}$, $\mathrm{FN}$ and $\mathrm{TN}$ on $Q_{S}(z)=Q_{S}(z_{0})$. The
four terms can be written as ${\mathbb{P}}({\cal
A}^{*}(Q_{S}(z))=1,m=1|Q_{S}(z_{0}))$, ${\mathbb{P}}({\cal
A}^{*}(Q_{S}(z))=1,m=-1|Q_{S}(z_{0}))$, ${\mathbb{P}}({\cal
A}^{*}(Q_{S}(z))=-1,m=1|Q_{S}(z_{0}))$ and ${\mathbb{P}}({\cal
A}^{*}(Q_{S}(z))=-1,m=-1|Q_{S}(z_{0}))$, respectively.

By $\mathrm{FP}(Q_{S}(z_{0}))=P(m=-1|Q_{S}(z_{0}))-\mathrm{TN}(Q_{S}(z_{0}))$
and $\mathrm{FN}(Q_{S}(z_{0}))=P(m=1|Q_{S}(z_{0}))-\mathrm{TP}(Q_{S}(z_{0}))$,
Equation 22 can be re-written as

$\frac{a_{0}+a_{10}{\mathbb{P}}(m=-1|Q_{S}(z_{0}))+a_{01}{\mathbb{P}}(m=1|Q_{S}(z_{0}))+(a_{11}-a_{01})\mathrm{TP}(Q_{S}(z_{0}))+(a_{00}-a_{10})\mathrm{TN}(Q_{S}(z_{0}))}{b_{0}+b_{10}{\mathbb{P}}(m=-1|Q_{S}(z_{0}))+b_{01}{\mathbb{P}}(m=1|Q_{S}(z_{0}))+(b_{11}-b_{01})\mathrm{TP}(Q_{S}(z_{0}))+(b_{00}-b_{10})\mathrm{TN}(Q_{S}(z_{0}))}.$

We then re-write
$\mathrm{TP}(Q_{S}(z_{0}))={\mathbb{P}}(m=1|Q_{S}(z_{0}))\mathbb{I}({\mathbb{P}}(m=1|Q_{S}(z_{0}))>t_{l})$
and
$\mathrm{TN}(Q_{S}(z_{0}))={\mathbb{P}}(m=-1|Q_{S}(z_{0}))\mathbb{I}({\mathbb{P}}(m=1|Q_{S}(z_{0}))\leq
t_{l})$ by plug-in Equation 16. Then, by
${\mathbb{P}}(m=1|Q_{S}(z_{0}))={\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=1)p/({\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=1)p+{\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=-1)(1-p))$
and
${\mathbb{P}}(m=-1|Q_{S}(z_{0}))={\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=-1)(1-p)/({\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=1)p+{\mathbb{P}}(Q_{S}(z)=Q_{S}(z_{0})|m=-1)(1-p))$,
and we have $AdvI_{l,p}(Q_{S},z_{0})$ equal to

${\frac{pc_{1}p(Q_{S}(z_{0}))+c_{2}(1-p)q(Q_{S}(z_{0}))+pc_{3}p(Q_{S}(z_{0}))I(Q_{s}(z_{0}))+c_{4}(1-p)q(Q_{S}(z_{0})I(Q_{s}(z_{0}))}{pd_{1}p(Q_{S}(z_{0}))+d_{2}(1-p)q(Q_{S}(z_{0}))+pd_{3}p(Q_{S}(z_{0}))I(Q_{s}(z_{0}))+d_{4}(1-p)q(Q_{S}(z_{0})I(Q_{s}(z_{0}))}},$

where $p(Q_{S}(z_{0}))={\mathbb{P}}(Q_{S}(z_{0})|m=1)$,
$q(Q_{S}(z_{0}))={\mathbb{P}}(Q_{S}(z_{0})|m=-1)$, and
$I(Q_{s}(z_{0}))=\mathbb{I}((1-t_{l})pp(Q_{S}(z_{0}))>t_{l}(1-p)q(Q_{S}(z_{0})))$.
Note that, we have $\hat{p}(Q_{S}(z_{0}))\xrightarrow{p}p(Q_{S}(z_{0}))$,
$\hat{q}(Q_{S}(z_{0}))\xrightarrow{p}q(Q_{S}(z_{0}))$ by the proof of Theorems
2 and 3.
$\mathbb{I}((1-t_{l})p\hat{p}>t_{l}(1-p)\hat{q})\xrightarrow{p}I(Q_{S}(z_{0}))$
follows by further applying Slutsky’s theorem, the continuous mapping theorem
and the condition ${\mathbb{P}}(m=1|Q_{S}(z)=Q_{s}(z_{0}))\neq t_{l}$.
Finally, 1 follows by Slutsky’s theorem and the third condition.

Finally, we observe that given $(1-\delta/2)$-confidence intervals for
${\mathbb{P}}(Q_{S}(z)|m=1)$, ${\mathbb{P}}(Q_{S}(z)|m=-1)$, and the
independence of ${\mathbb{P}}(Q_{S}(z)|m=1)$ and
${\mathbb{P}}(Q_{S}(z)|m=-1)$, the joint $(1-\delta)$-confidence interval of
${\mathbb{P}}(Q_{S}(z)|m=1)$ and ${\mathbb{P}}(Q_{S}(z)|m=-1)$ is simply the
union of the two (using union bound), and 2 follows. ∎

###  D.7 Proof of Theorem 5

###### Proof.

First note that

$\displaystyle{\cal A}^{*}(Q_{S})=\mathbb{E}_{Z}|f_{p}(Z)|$
$\displaystyle=\mathbb{E}_{Z}\left|\frac{\mathbb{P}(Q_{S}(Z)|M=1)p-\mathbb{P}(Q_{S}(Z)|M=-1)(1-p)}{\mathbb{P}(Q_{S}(Z))}\right|$
$\displaystyle=\sum\limits_{j\in{\cal
Q}}\left|\mathbb{P}(Q_{S}(Z)=j|M=1)p-\mathbb{P}(Q_{S}(Z)=j|M=-1)(1-p)\right|$

It is sufficient to show that $\forall j\in{\cal Q}$,

$\displaystyle\begin{split}&\left|\frac{p}{N_{1}}\sum\limits_{i=1}^{N_{1}}\mathbb{I}(Q_{S}(X_{i})=j)-\frac{1-p}{N_{2}}\sum\limits_{i=1}^{N_{2}}\mathbb{I}(Q_{S}(Y_{i})=j)\right|\xrightarrow{p}\left|\mathbb{P}(Q_{S}(Z)=j|M=1)p-\mathbb{P}(Q_{S}(Z)=j|M=-1)(1-p)\right|,\end{split}$
(23)

and then (1) follows by Slutsky’s Theorem.

To prove Equation (23), we first show that

$\displaystyle\frac{1}{N_{1}}\sum\limits_{i=1}^{N_{1}}\mathbb{I}(Q_{S}(X_{i})=j)\xrightarrow{p}\mathbb{P}(Q_{S}(Z)=j|M=1),$
$\displaystyle\frac{1}{N_{2}}\sum\limits_{i=1}^{N_{2}}\mathbb{I}(Q_{S}(Y_{i})=j)\xrightarrow{p}\mathbb{P}(Q_{S}(Z)=j|M=-1)$

by the law of large numbers. Then Equation (23) follows by applying the
continuous mapping theorem.

Next, we prove (2). It is easy to verify that $\forall
i\in\\{1,\cdots,N_{1}\\}$, $\forall
x_{1},\cdots,x_{i},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}},x_{i}^{{}^{\prime}}\in{\cal
Z}$, we always have

$\displaystyle|W(x_{1},\cdots,x_{i},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}})-W(x_{1},\cdots,x_{i}^{{}^{\prime}},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}})|\leq\frac{2p}{N_{1}}=\frac{2}{N}.$

Similarly, for $\forall i\in\\{1,\cdots,N_{2}\\}$ and

$\forall
x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i},\cdots,y_{N_{2}},y_{i}^{{}^{\prime}}\in{\cal
Z}$, we always have

$\displaystyle|W(x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i},\cdots,y_{N_{2}})-W(x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i}^{{}^{\prime}},\cdots,y_{N_{2}})|\leq\frac{2(1-p)}{N_{2}}=\frac{2}{N}.$

Then by McDiarmid’s inequality, $\mathbb{P}(|W_{N}-\mathbb{E}W_{N}|\geq t)\leq
2\exp\left(-\frac{N}{2}t^{2}\right)$. Let
$\delta=2\exp\left(-\frac{N}{2}t^{2}\right)$, and we have (2). ∎

###  D.8 Proof of Theorem 6

###### Proof.

(1) We first prove the consistency. Note that

$\displaystyle{\cal A}^{*}(Q_{S})$ $\displaystyle=\mathbb{E}_{Z}|f_{p}(Z)|$
$\displaystyle=\mathbb{E}_{Z}\left|\frac{\mathbb{P}(Q_{S}(Z)|M=1)p-\mathbb{P}(Q_{S}(Z)|M=-1)(1-p)}{\mathbb{P}(Q_{S}(Z))}\right|$
$\displaystyle=\int\left|p(x)p-q(x)(1-p)\right|dx,$

where $p(x)={\mathbb{P}}(Q_{S}(z)=x|m=1)$ and
$q(x)={\mathbb{P}}(Q_{S}(z)=x|m=-1)$. Rewrite $U_{N}$ as

$\displaystyle U_{N}$
$\displaystyle=\int|\frac{p}{N_{1}h^{d}}\sum\limits_{i=1}^{N_{1}}K(\frac{x-X_{i}}{h})-\frac{1-p}{N_{2}h^{d}}\sum\limits_{i=1}^{N_{2}}K(\frac{x-Y_{i}}{h})|dx$
$\displaystyle=\int\left|p\left(\frac{1}{N_{1}h^{d}}\sum\limits_{i=1}^{N_{1}}K(\frac{x-X_{i}}{h})-p(x)\right)+(pp(x)-(1-p)q(x))+(1-p)\left(q(x)-\frac{1}{N_{2}h^{d}}\sum\limits_{i=1}^{N_{2}}K(\frac{x-Y_{i}}{h}\right)\right|dx.$

Then

$\displaystyle|U_{N}-{\cal A}^{*}(Q_{S})|$
$\displaystyle\leq\int\left|p\left(\frac{1}{N_{1}h^{d}}\sum\limits_{i=1}^{N_{1}}K(\frac{x-X_{i}}{h})-p(x)\right)+(1-p)\left(q(x)-\frac{1}{N_{2}h^{d}}\sum\limits_{i=1}^{N_{2}}K(\frac{x-Y_{i}}{h}\right)\right|dx$
$\displaystyle\leq
p\int\left|\left(\frac{1}{N_{1}h^{d}}\sum\limits_{i=1}^{N_{1}}K(\frac{x-X_{i}}{h})-p(x)\right)\right|dx+(1-p)\int\left|\left(q(x)-\frac{1}{N_{2}h^{d}}\sum\limits_{i=1}^{N_{2}}K(\frac{x-Y_{i}}{h}\right)\right|dx$
$\displaystyle\xrightarrow{p}0,$

where the last step follows from Theorem 7 and Slutsky’s theorem.

(2) It is easy to verify that $\forall i\in\\{1,\cdots,N_{1}\\}$,
$\forall
x_{1},\cdots,x_{i},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}},x_{i}^{{}^{\prime}}\in{\cal
Z}$, we always have

$\displaystyle|U(x_{1},\cdots,x_{i},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}})-U(x_{1},\cdots,x_{i}^{{}^{\prime}},\cdots,x_{N_{1}},y_{1},\cdots,y_{N_{2}})|\leq\frac{2p}{N_{1}}=\frac{2}{N}.$

Similarly, for $\forall i\in\\{1,\cdots,N_{2}\\}$ and

$\forall
x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i},\cdots,y_{N_{2}},y_{i}^{{}^{\prime}}\in{\cal
Z}$, we always have

$\displaystyle|U(x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i},\cdots,y_{N_{2}})-U(x_{1},\cdots,x_{N_{1}},y_{1},\cdots,y_{i}^{{}^{\prime}},\cdots,y_{N_{2}})|\leq\frac{2(1-p)}{N_{2}}=\frac{2}{N}.$

Then by McDiarmid’s inequality, $\mathbb{P}(|U_{N}-\mathbb{E}U_{N}|\geq t)\leq
2\exp\left(-\frac{N}{2}t^{2}\right)$. Let
$\delta=2\exp\left(-\frac{N}{2}t^{2}\right)$, and we have (2). ∎

##  Appendix E Model architectures and hyper–parameters

Here we outline the different layers used in the model architectures for
different datasets. The last layers of discriminators for WGAN experiments do
not have sigmoid activation functions. The hyper-parameters are chosen the
same same as (Goodfellow et al., 2014; Gulrajani et al., 2017).

###  E.1 MNIST

####  E.1.1 Generator layers

  * •

Dense(units$=512$, input size$=100$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=512$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=1024$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=784$, activation = ’tanh’)

####  E.1.2 Discriminator layers

  * •

Dense(units$=2048$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=512$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=256$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=1$, activation = ’sigmoid’)

###  E.2 CIFAR–10

####  E.2.1 Generator layers

  * •

Dense(units=$2\times 2\times 512$)

  * •

Reshape(target shape$=(2,2,512)$)

  * •

Conv2DTranspose(filters$=128$, kernel size$=4$, strides$=1$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=64$, kernel size$=4$, strides$=2$, padding$=1$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=32$, kernel size$=4$, strides$=2$, padding$=1$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=3$, kernel size$=4$, strides$=2$,padding$=1$,
activation = ’tanh’)

####  E.2.2 Discriminator layers

  * •

Conv2D(filters$=64$, kernel size$=5$, strides$=2$)

  * •

Conv2D(filters$=128$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Conv2D(filters$=128$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Conv2D(filters$=256$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=1$, activation = ’sigmoid’)

###  E.3 Skin-cancer MNIST

####  E.3.1 Generator layers

  * •

Dense(units=$4\times 4\times 512$)

  * •

Reshape(target shape$=(4,4,512)$)

  * •

Conv2DTranspose(filters$=256$, kernel size$=5$, strides$=2$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=128$, kernel size$=5$, strides$=2$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=64$, kernel size$=5$, strides$=2$)

  * •

ReLU()

  * •

Conv2DTranspose(filters$=3$, kernel size$=5$, strides$=2$,activation = ’tanh’)

####  E.3.2 Discriminator layers

  * •

Conv2D(filters$=64$, kernel size$=5$, strides$=2$)

  * •

Conv2D(filters$=128$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Conv2D(filters$=128$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Conv2D(filters$=256$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=1$, activation = ’sigmoid’)

####  E.3.3 Toy binary classifier layers

  * •

Conv2D(filters$=512$, kernel size$=5$, strides$=2$)

  * •

LeakyReLU($\alpha=0.2$)

  * •

Dense(units$=2$, activation = ’relu’)

  * •

Dense(units$=1$, activation = ’sigmoid’)

##  Appendix F Auxiliary lemmas and theorems

######  Theorem 7.

((Devroye & Gyorfi, 1985)) Let $p_{N}$ be an automatic kernel estimate with
arbitary density $K$, as defined in Equation (17). If
$h+(nh^{d})^{-1}\rightarrow 0$ completely (almost surely, in probability),
then $\int|p_{N}-p|\rightarrow 0$ (almost surely, in probability), for all
density $p$ on $\mathbb{R}^{d}$.

######  Lemma 3 ((Chen, 2017)).

With probability $(1-\delta/2)$, we have

$\displaystyle p(x)\in[\widehat{p}_{\rm lower}(x),\;\;\widehat{p}_{\rm
upper}(x)]\;,$ (24)

where

$\displaystyle\widehat{p}_{\rm
lower}(x)=\widehat{p}_{N}(x)-z_{1-\delta/4}\sqrt{\frac{\mu_{K}\cdot\widehat{p}_{N}(x)}{Nh^{d}}}\;,$
$\displaystyle\widehat{p}_{\rm
upper}(x)=\widehat{p}_{N}(x)+z_{1-\delta/4}\sqrt{\frac{\mu_{K}\cdot\widehat{p}_{N}(x)}{Nh^{d}}}\;.$

and $\mu_{K}:=\int K^{2}(x)dx$, $z_{1-\delta/4}$ is the $(1-\delta/4)$
quantile of a standard normal distribution.

##  Appendix G The optimal membership advantage as a function of synthetic
dataset size

To explore the effect the size of synthetic datasets on the membership
advantage, we first trained a JS-GAN on the skin-cancer MNIST dataset. The JS-
GAN was trained for 2000 epochs and several synthetic datasets of sizes
varying from $10$ to $10^{5}$ samples was generated. The adversary used was
the one described in Equation 2. It can be seen in Figure 5 that as the
synthetic dataset size increases, so does the optimal membership advantage.

Figure 5: The optimal membership advantage vs. the synthetic dataset size.

##  Appendix H Estimation of optimal membership advantage for discriminative
models

To demonstrate that our estimators of optimal membership advantage are also
applicable to discriminative models, we trained a simple binary classifier
(architecture described in section E.3.3) on the skin-cancer MNIST dataset
(same experimental setting as the generative model). We chose two queries: i)
a black-box query - the final output of the classifier, ii) a white-box query
- the output of the penultimate layer of the classifier. We discretized the
outputs and used our discrete estimator to estimate the optimal membership
advantage in each case, as seen in Figure 6. As expected, the white-box query
has a higher optimal membership advantage than the black-box query.

Figure 6: Estimation of the optimal membership advantage with white-box and
black-box queries against a binary classification model on skin-cancer MNIST.