AI

    Artificial Intelligence
DP

    Differential Privacy
GDPR

    General Data Protection Regulation
MIA

    Membership Inference Attack
SOTA

    state-of-the-art
FPR

    False Positive Rate
TPR

    True Positive Rate
FNR

    False Negative Rate
AUC

    area under the receiver-operator curve
MCC

    Matthews’ Correlation Coefficient
ROC

    receiver-operator characteristic
MRI

    magnetic resonance imaging
CT

    computed tomography
ReRo

    reconstruction robustness
SSIM

    structural similarity
PET

    privacy-enhancing technology
DRA

    data reconstruction attack
DP-SGD

    Differentially Private Stochastic Gradient Descent

# Reconciling AI Performance and Data Reconstruction Resilience for Medical
Imaging

Alexander Ziller  Technical University of Munich  Tamara T. Mueller  Technical
University of Munich  Simon Stieger  Technical University of Munich  Leonhard
Feiner  Technical University of Munich  Johannes Brandt  Technical University
of Munich  Rickmer Braren  Technical University of Munich  Daniel Rueckert
Technical University of Munich  Imperial College London  Georgios Kaissis
Technical University of Munich  Imperial College London  Helmholtz Munich

###### Abstract

Artificial Intelligence (AI) models are vulnerable to information leakage of
their training data, which can be highly sensitive, for example in medical
imaging. Privacy Enhancing Technologies (PETs), such as Differential Privacy
(DP), aim to circumvent these susceptibilities. DP is the strongest possible
protection for training models while bounding the risks of inferring the
inclusion of training samples or reconstructing the original data. DP achieves
this by setting a quantifiable privacy budget. Although a lower budget
decreases the risk of information leakage, it typically also reduces the
performance of such models. This imposes a trade-off between robust
performance and stringent privacy. Additionally, the interpretation of a
privacy budget remains abstract and challenging to contextualize. In this
study, we contrast the performance of AI models at various privacy budgets
against both, theoretical risk bounds and empirical success of reconstruction
attacks. We show that using very large privacy budgets can render
reconstruction attacks impossible, while drops in performance are negligible.
We thus conclude that not using DP –at all– is negligent when applying AI
models to sensitive data. We deem those results to lie a foundation for
further debates on striking a balance between privacy risks and model
performance.

##  1 Main

The rapid rise of Artificial Intelligence (AI) applications in medicine
promises to transform healthcare, offering improvements ranging from specific
applications, such as more precise pathology detection, outcome prediction, to
the promise of general medical AI [1, 2, 3, 4, 5]. However, recent results
highlight a significant vulnerability: AI models may disclose details of their
training data. This can happen either inadvertently or be forced through
attacks by malicious third parties, also called adversaries. Among the most
critical attacks are data reconstruction attacks, where the adversary attempts
to extract training data from the model or its gradients [6, 7, 8, 9, 10, 11,
12, 13]. Such attacks harbour distinct risks. On one hand, a successful data
reconstruction attack severely undermines the trust of patients whose data is
exposed. This not only jeopardises the relationship between medical
practitioners and patients, but likely also diminishes the willingness of
patients to make their health data for the training of AI models or for other
research purposes available. This is problematic since the success of AI
models in medicine is dependent on the availability of large and diverse real-
world patient datasets. On the other hand, a successful attack can also
constitute a breach of patient data privacy regulations.

While privacy laws vary globally, the protection of health data is generally
considered of high importance. For example, the European Union’s General Data
Protection Regulation (GDPR) declares the protection of personal data as a
fundamental right. Notably, these laws deem the removal of personal
identifiers (e.g. name or date of birth) –de-identification– sufficient
protection. However, it has been demonstrated on several occasions that
commonly used de-identification techniques such as anonymisation,
pseudonymisation, or $k$-anonymity are vulnerable to re-identification attacks
[14, 15, 16]. This also holds true in the case of medical imaging data. For
example, the facial contours of a patient can be obtained from a reconstructed
magnetic resonance imaging (MRI) scan even if their name has been removed from
the record, thus enabling their re-identification from publicly available
photographs [17]. Figuratively, this is analogous to considering passport
photos without additional information not as personal data. Arguably, this
highlights the tension between what is considered private in a legal sense and
what individuals consider acceptable in terms of informational self-
determination. We thus contend that AI systems which process sensitive data
should not only rely on de-identification techniques but also implement
privacy-enhancing technologies, i.e. technologies which furnish an objective
or formal guarantee of privacy protection.

Figure 1: Comparison of a worst-case and a realistic threat model. (A)
Adversaries can have various capabilities depending on the setting (B) The
combination of the adversary’s capabilities defines the threat model. In a
worst-case analysis, they have all capabilities. However, access to the
database is a pessimistic, practically irrelevant scenario. (C) The necessary
privacy protection depends on the threat model. In a worst-case threat model,
the adversary only needs to match the model and gradient to an image in the
database. In a practically more relevant scenario, the image must be
reconstructed from the model and gradient. Here, much less privacy protection
is necessary (D) The more stringent the privacy protection is chosen, the
higher the impacts on the model performance are. Thus, if a realistic threat
model is considered appropriate, models can perform better.

###  1.1 Differential Privacy as the Strongest Possible Privacy Preservation

Among the privacy-enhancing technologies, Differential Privacy (DP) [18] is
considered the optimal protection for training AI models while moderating the
privacy risk faced by participating patients. This is because it has several
appealing properties: DP is the only technique providing mathematical
guarantees for an upper bound of the success of reconstructing data samples
[19, 20]. Also, it satisfies requirements imposed by regulations like the GDPR
concerning re-identification [15, 21]. Moreover, the privacy guarantees of DP
systems cannot be degraded through the use of side information or through
post-processing by additional computation (two notable vulnerabilities of
traditional de-identification schemes). Last but not least, DP satisfies
composability, i.e., the notion that privacy guarantees degrade predictably
when multiple DP algorithms are executed on the same dataset. This enables the
concept of a privacy budget, which makes the cumulative re-identification risk
quantifiable and which can be set depending on policy or preference. We note
that this ability to moderate risks stemming from AI applications is
particularly beneficial, as this ability is also mandated by recent legal
frameworks such as the European AI act [22]. These properties are leading to
its increasing adoption in industry and government applications [23, 24].

Despite these benefits, the effective and efficient implementation of DP in
large-scale AI systems also presents a series of challenges. DP has been
criticised for the fact that the choice of an appropriate privacy budget is
delicate. Higher privacy budgets correspond to lower levels of privacy
protection and thus increase the risk of successful attacks, while lower
privacy budgets limit the information available for training. By that, it
introduces a new challenge, namely a trade-off between privacy protection and
model performance, i.e. diagnostic accuracy for a given use-case. Furthermore,
this trade-off also depends on the specific input data and learning task,
which can vary significantly between scenarios. Arguably, concerns about
reduced model performance are a probable reason why, despite its many
benefits, DP is not yet widely implemented in medical AI workflows. After all,
finding a trade-off between diagnostic accuracy and privacy represents a
complex technical and ethical dilemma. This dilemma is also pictured at its
extreme as DP is underlaid by a worst-case set of assumptions. These
assumptions, also called a threat model, include an adversary who is able to
deeply manipulate and interfere with the dataset, the training process, model
architecture, and (hyper-)parameters, and has access to all parameters of the
DP algorithm (mechanism). Moreover, the canonical DP adversary is not assumed
to execute a data reconstruction attack but a much simpler type of attack,
namely a membership inference attack which attempts to determine whether a
specific individual’s data (which is available to the adversary) was included
in the training dataset or not. Since there are only two possible outcomes of
such an attack (member or non-member), membership inference must only reveal a
single bit of information, compared to a data reconstruction attack, which
must successfully reveal a much larger record (e.g. an image comprising
several million bits of information). Evidently, although worst-case
assumptions are prudent for the theoretical modelling of adversaries, the DP
threat model is unlikely to ever be encountered in practice. Moreover, the
aforementioned membership inference attack in which the adversary has access
to a target record and tries to determine whether it was used for training a
specific model is arguably of very low practical relevance. Instead, data
reconstruction attacks are likely perceived as a significantly more relevant
privacy threat by patients. Moreover, realistic adversaries in the medical
setting (where data is strongly guarded) can likely be assumed to not have
access to the training data (as they would have little incentive to attack a
model otherwise).

In this paper, we investigate the question of whether the aforementioned
typical DP threat model might be too pessimistic for practical use-cases and,
by that, impose unnecessary privacy/performance trade-offs. To investigate
this hypothesis, our work studies the privacy/performance characteristics of
AI models trained on large-scale medical imaging datasets under more realistic
threat models which still allow for strong privacy protection but represent a
step down from the worst-case assumptions made by DP. Our main finding is
that, even in complex medical imaging tasks, it is possible to train AI models
with excellent diagnostic performance while still defending against data
reconstruction attacks and thus patient re-identification. We achieve this by
training models under privacy budgets which would be considered too large to
offer any protection against the threats considered under the worst-case DP
threat model. This provides evidence that there is a recommendation to by
default train AI models with DP protection. Therefore, although more
restrictive privacy budgets than the ones used in our study remain relevant
for use-cases in which protection against membership inference is explicitly
required, our findings highlight the existence of an additional option: When
high model performance is required but cannot be achieved without
relinquishing membership inference protection, our findings offer a compromise
whereby an important and relevant class of attacks can be defended against
while fulfilling the requirement for high diagnostic accuracy.

###  1.2 Risk Quantification of AI Models

| Worst-case | Relaxed | Realistic
---|---|---|---
Model Architecture & Weight | yes | yes | yes
Hyperparameter | yes | yes | yes
Dataset Access | yes | partially | no
Perfect Reconstruction Algorithm | n.n. | yes | no
Risk Analysis | Theoretical | Theoretical | Empirical
Table 1: Overview of the capabilities of an adversary in the threat models
analysed in this study.

As alluded to in the previous section, DP allows for a quantifiable reduction
in the risk of privacy attacks associated with the training of AI models.
Before introducing our main results, we provide a more concrete explanation of
how the risks of re-identification through a data reconstruction attack are
moderated by DP. Concretely, we avail ourselves of the framework of
reconstruction robustness (ReRo), which will allow us to formulate an upper
bound on the success rate of data reconstruction attacks against AI models
trained with DP under the specific threat models discussed below.

ReRo was introduced by [19]. It is a guarantee pertaining to an algorithm
which processes sensitive data, e.g. an AI model trained with DP. Intuitively,
if at most a proportion $0\geq\gamma\leq 1$ of the total samples used to train
the model can be successfully reconstructed by an adversary with a
reconstruction error lower than $\eta\geq 0$, then the model satisfies
$(\eta,\gamma)$-ReRo. Recent works have proven that all models trained with DP
automatically satisfy ReRo and that for certain settings, it is possible to
directly quantify the upper bound for $\gamma$ [19, 25, 20]. In other words,
DP automatically provides strong and quantifiable protection against data
reconstruction attacks.

In the results section below, we will study the ReRo guarantees of models
trained with DP under three distinct sets of assumptions about the
capabilities of the adversary, i.e. three distinct threat models:

  1. 1.

The worst-case threat model: This corresponds to the adversary usually
considered in DP, namely one who has unbounded computational abilities, can
deeply manipulate the model’s (hyper-)parameters and has access to the target
image itself, which they can use to attack the model. Evidently, this threat
model is not realistic (as an adversary who has access to the target point
would not need to attack the model), but is used to provide guarantees when
all bets are off, i.e. in a so-called privacy auditing scenario when one is
interested in the absolute worst-case behaviour of a system.

  2. 2.

The relaxed threat model [26]: This threat model is still quite pessimistic,
as it still assumes unbounded computational ability and access to model
(hyper-)parameters. However, this adversary only has restricted access to the
dataset, notably, they cannot use the target image itself to attack the model.
Although it renders this threat model more appropriate for scenarios where the
dataset can be safely assumed to be kept secure, e.g. in a hospital’s
database, it still makes assumptions, which are not encountered in any
practical scenario. Most importantly, the adversary has a black-box
reconstruction algorithm, which yields either a perfect reconstruction or
fails, and the only decision the adversary has to make is whether the
reconstruction was indeed the target data. The term relaxed stems from
security research, where a relaxation denotes a weakening of a security
assumption.

  3. 3.

The realistic threat model: The final threat model considers an adversary with
unbounded computational ability and the power to manipulate model
(hyper-)parameters but only very limited access to information about the
dataset. For example, the adversary can know the dimensions of the images to
be reconstructed but not any of their contents. We note that even this threat
model is relatively pessimistic, as it assumes an active adversary who is
trusted and therefore the actions are not reviewed by other participants. Such
adversaries could manipulate the model to their advantage in order to
reconstruct training data. In many cases, this could be detected simply by
inspecting the model architecture. Nonetheless, we use this threat model as it
is conceivable that such adversaries can exist in, e.g. federated learning
settings in untrustworthy consortia.

Of note, it is possible to provide theoretical bounds on the reconstruction
attack success rate in both the worst-case and the relaxed threat models using
the techniques presented in [25, 20]. For the realistic threat model, we
assess the attack success rate empirically. A concise overview of the
aforementioned threat models is provided in Table 1.

In summary, while conservative (i.e. worst-case or relaxed) threat models are
important tools in security research because they allow one to derive closed-
form bounds on the attack success rate of very powerful adversaries, such
threat models are in most reasonable scenarios too pessimistic. The key
contribution of our work is to investigate the realistic risks posed by a type
of adversary who is still very powerful but can be reasonably assumed to exist
in real-world medical AI model training use cases. In the next section, we
will show that perfectly defending against such adversaries is possible while
maintaining a competitive diagnostic model performance with that of a model
trained without any privacy protection.

##  2 Results

###  2.1 Setup

####  2.1.1 Datasets

We begin by introducing our rationale for choosing specific datasets for our
experiments. We identified four characteristics of medical imaging datasets,
which reoccur frequently: (1) Datasets are often small compared to non-medical
datasets. For example, most medical AI algorithms, which are currently
approved by the US Food and Drug Administration (FDA), are trained on less
than $1\,000$ data samples [27]. (2) Diagnoses occur with very different
frequencies, leading to often imbalanced datasets skewed toward more common
diagnoses. In segmentation tasks, this may happen due to different spatial
extensions of objects. (3) While natural images are all captured with standard
cameras as RGB images, medical images are from multi-modal imaging devices
such as computed tomography (CT), MRI, or ultrasound.

Dataset | Task | Small | Imbalanced | Multi-modal
---|---|---|---|---
RadImageNet | Classification |  | ✓ | ✓
HAM10000 | Classification | ✓ | ✓ |
MSD Liver | Segmentation | ✓ | ✓ |
Table 2: Overview of characteristics of our datasets.

In this study, we aim to give a broad discussion of settings in medical AI.
Hence, we have chosen three datasets, which encompass the above-discussed
scenarios (c.f. Table 2).

  1. 1.

The RadImageNet dataset [28] contains over $1.3$ million 2D images with CT,
MRI, and ultrasound scans representing three imaging modalities with 165
classification targets, which are highly imbalanced.

  2. 2.

The HAM10000 dataset [29] is a collection of $10\,000$ skin lesion RGB images
spread across seven categories. We intentionally amplified the class imbalance
to a strong but not untypical $80:20$ class ratio by merging classes based on
the need for immediate treatment (see Section 4.1).

  3. 3.

Lastly, we use the MSD Liver dataset [30, 31], a demanding image-to-image task
involving just $131$ CT scans annotated at voxel level. Given the small number
of available training samples as well as a segmentation task (i.e., per-pixel
classification) with tumours only encompassing a tiny fraction of each scan,
it represents a very challenging medically relevant task.

To the best of our knowledge, no prior work shows the performance of AI models
trained under formal privacy guarantees on such a comprehensive and large
dataset as RadImageNet or a 3D image-to-image task as MSD Liver represents.

Privacy Budget | Noise | Test MCC | Reconstruction Risk
---|---|---|---
$\varepsilon\textrm{ at }\delta=8.0\cdot 10^{-7}$ | $\sigma$ |  Mean $\pm$ StD. | Worst-Case | Relaxed | Realistic
RadImageNet
$1$ | $0.67$ | $64.95\%\pm 0.13\%$ | $0.00\%$ | $0.00\%$ | $0\%$
$8$ | $0.34$ | $68.75\%\pm 0.13\%$ | $0.04\%$ | $0.01\%$ | $0\%$
$32$ | $0.267$ | $69.99\%\pm 0.25\%$ | $13.18\%$ | $3.96\%$ | $0\%$
$10^{12}$ | $0.054$ | $70.83\%\pm 0.19\%$ | $100\%$ | $100\%$ | $0\%$
Non-private | $0$ | $71.83\%\pm 1.86\%$ | $100\%$ | $100\%$ | $100\%$
HAM10000
$1$ | $0.92$ | $15.60\%\pm 4.13\%$ | $0.03\%$ | $0.01\%$ | $0\%$
$8$ | $0.47$ | $37.48\%\pm 3.45\%$ | $1.22\%$ | $0.04\%$ | $0\%$
$20$ | $0.40$ | $42.83\%\pm 2.37\%$ | $22.30\%$ | $0.78\%$ | $0\%$
$10^{9}$ | $0.02$ | $51.98\%\pm 2.52\%$ | $100\%$ | $100\%$ | $0\%$
Non-private | $0$ | $51.66\%\pm 1.38\%$ | $100\%$ | $100\%$ | $100\%$
MSD Liver
|  | Dice Score Liver | Dice Score Tumour | Reconstruction Risk
|  |  Mean $\pm$ StD. |  Mean $\pm$ StD. | Worst-Case | Relaxed | Realistic
$1$ | $9.97$ | $42.84\%\pm 1.83\%$ | $0.96\%\pm 0.37\%$ | $1.66\%$ | $0.97\%$ | $0\%$
$8$ | $1.66$ | $74.71\%\pm 3.14\%$ | $3.01\%\pm 0.96\%$ | $17.96\%$ | $3.68\%$ | $0\%$
$20$ | $0.96$ | $79.06\%\pm 2.17\%$ | $5.55\%\pm 0.72\%$ | $74.24\%$ | $27.37\%$ | $0\%$
$10^{9}$ | $0.0054$ | $91.20\%\pm 0.23\%$ | $29.73\%\pm 2.89\%$ | $100\%$ | $100\%$ | $0\%$
Non-private | $0$ | $91.58\%\pm 0.41\%$ | $28.38\%\pm 2.29\%$ | $100\%$ | $100\%$ | $100\%$
Table 3: Comparison of performance to privacy risk over multiple datasets and
privacy budgets. Test MCC denotes Matthew’s Correlation Coefficient on the
test dataset. For all performance metrics, we give the mean $\pm$ the standard
deviation (StD.) over five runs with different random seeds. Reconstruction
risk denotes the upper bounds for the risk of a successful reconstruction
attack of a worst-case and minimally relaxed adversary, as well as the
empirical success of one of the strongest realistic attacks. An image is
considered successfully reconstructed if the structural similarity (SSIM) to
any reconstruction is higher than $80\%$. Note that the noise multiplier
$\sigma$ is given for the empirical attack scenario where an adversary
manipulated hyperparameters in their favour. Noise multipliers for performance
analysis are generally higher.

Figure 2: Theoretical reconstruction bounds for a worst-case and slightly
relaxed adversary. From left to right: RadImageNet, HAM10000, MSD Liver. We
see that the mathematical upper bound for a reconstruction risk of a minimally
relaxed threat model (orange) is already substantially lower compared to a
worst-case setting (purple). Figure 3: Reconstruction threat analysis for
three datasets. Each row shows one dataset. From top to bottom: RadImageNet,
HAM10000, MSD Liver. Left column: Cumulative number of images, which have in
an empirical reconstruction a structural similarity (SSIM) difference lower
than the value on the x-axis. Note that it is the SSIM reconstruction error
and thus perfect at 0 and worst at 1. Exemplarily, we see that on the MSD
dataset at a reconstruction error of $10\%$ all non-private (green) images,
$39\%$ at $\varepsilon=10^{15}$ (pink) and none at more restrictive privacy
guarantees can be reconstructed On the right: The top five images with the
best reconstruction score and their corresponding best reconstruction at
various privacy budgets.

####  2.1.2 Evaluation

Our evaluation focuses on how various privacy risks on multiple datasets
correlate to the algorithm’s performance. First, we show the correlation of
the AI performance on our datasets to privacy budgets. Second, we illustrate
the implications of a certain privacy budget in a risk profile, summarising
the reconstruction risk under different threat models. We recall that a threat
model corresponds to the set of assumptions over the attacker, where we give
the theoretical bounds for a worst-case and a slightly relaxed adversary. Both
are more pessimistic than any real-world scenario. Thus, we add a third threat
model representing the worst realistic case.

To measure the performance of the models on classification tasks, we use
Matthews’ Correlation Coefficient (MCC) [32]. Opposed to more frequently used
metrics such as accuracy or $F_{1}$-score, it incorporates the entire
confusion matrix and, by that, is extremely robust against any class imbalance
[33]. It is also better interpretable as for random predictions it is $0$ and
for perfect predictions $1$, whereas the accuracy depends on the class
distribution. For the segmentation task, we measure the class-wise Dice score
of the 3D volumes and report the average over all volumes for the liver and
tumour, as they are the targets of interest in our task. A perfect prediction
yields a $100\%$ Dice score.

In Table 3 we list for all datasets over increasing privacy budgets the best
possible AI model performance and the corresponding reconstruction risk. The
risk is three-tiered: (1) The upper bound of a worst-case adversary. This is
the absolute maximum that the risk can be under this setting and can not be
increased by post-processing or side information. (2) The upper bound of a
minimally relaxed adversary as introduced by [20]. (3) The reconstruction
success of the real-world adversary. We argue that for practical use cases,
protection against such an attacker suffices. By listing all three, we provide
an overview of how the risk varies in constant settings by changing
assumptions about the adversary.

###  2.2 Performance Trade-Offs Under Varying Privacy Levels

####  2.2.1 Impacts on Model Performance is Substantial for Small Datasets

At first, we analyse the impact of a very restrictive (small) privacy budget
of $\varepsilon=1$ on the AI predictive performance of our datasets (see Table
3). Across the board, we see that at these budgets, the impacts on the model
performance are strong. Concretely, we find that on RadImageNet, a standard
non-private AI model reaches $71.83\%$ on average, while trained at such
restrictive privacy guarantee we find an average MCC of $64.95\%$, which is
still $90\%$ of the non-private MCC score. The gap turns much larger on the
HAM10000 dataset, where the model performance, when trained with a very low
privacy budget of $\varepsilon=1$ is closely above the chance level at an MCC
of $15.60\%$. Similarly, on the MSD Liver dataset at restrictive privacy
budgets, the average Dice score for the liver goes down to $42.84\%$ (Non-
private: $91.58\%$) and negligible $0.96\%$ for the tumour. This exemplifies
the challenges of furnishing strong privacy protection when training AI models
on small or difficult datasets.

####  2.2.2 Prediction Quality under Medium Budgets Depends on Dataset

Next, we consider medium privacy budgets that range in our study from
$\varepsilon=8$ to $\varepsilon=32$, which are a typical choice [34, 35] in
literature. As $\varepsilon$ is an exponential parameter ($e^{\varepsilon}$),
larger values correspond to exponentially increased privacy guarantees. For
this reason, some argue that the guarantees provided by such medium budgets
are meaningless [18, 36].

At these privacy budgets, although the performance substantially increases,
compared to the extremely restrictive privacy budget, on none of the datasets,
the private AI models match the non-private performance. On the RadImageNet
dataset, the achieved result closely approaches the non-private baseline.
Specifically, with a privacy budget of $\varepsilon=32$, the MCC score is
$69.99\%$ compared to $71.83\%$ of the corresponding non-private performance.
Also for the HAM10000 dataset performance is now strongly improved at
$42.83\%$ MCC score, yet is still a decrease of $9\%$ compared to the non-
private result. Lastly, in the MSD Liver dataset, the liver as a larger organ
can now be learned up to a reasonable dice score of $79.06\%$ at
$\varepsilon=20$. However, it is also still far from the non-private
performance. Especially concerning is the prediction quality of the tumour,
which is a much smaller and more complex structure. This leads to a poor
segmentation quality and only achieves, on average, a dice score of $5.55\%$,
which is unsuitable for a real-world application. Again, we note that
performance trade-offs impact especially smaller and imbalanced datasets.

####  2.2.3 Performance Trade-Offs Vanish Under Large Privacy Budgets

For very large privacy budgets, we observe that the gap between private and
non-private performance disappears. We recall that HAM10000 and MSD Liver as
small datasets are extremely challenging under restrictive DP conditions. When
increasing the privacy budget to $\varepsilon=10^{9}$ no statistically
significant difference to the non-private model can be detected anymore
(p-values: HAM10000: $0.36$, MSD Liver dataset p-value liver: $0.10$, tumour:
$0.29$, Student’s t-test). Only on RadImageNet, although the non-private model
is still statistically significantly superior (p-value: 0.001), the private
model at an $\varepsilon=10^{12}$ achieves $99\%$ of the non-private baseline
performance.

It is unsurprising that increasing the privacy budget lifts the implications
on the model performance. Hence, the question that must be asked is what level
of privacy is necessary for a specific setting. This cannot be answered
generally and must be carefully considered for each use case. Important for
these considerations is which risks are associated with a certain privacy
budget. We analyse the reconstruction risk for our three datasets and three
threat models discussed previously.

###  2.3 Low Worst-Case Reconstruction Bounds Require Small Privacy Budgets

Although, for most use cases too pessimistic, worst-case analyses have the
advantage that we have a formal guarantee, which gives an absolute upper bound
on the risk of this scenario. When analysing the theoretical worst-case
(highest) success of reconstruction attackers, we find that for the large
RadImageNet dataset for budgets $\varepsilon\leq 8$, the risk is smaller than
$0.05\%$. However, already at an $\varepsilon=32$ almost $15\%$ of the samples
can theoretically be recovered. Here, the smaller datasets are again at higher
risk. While at $\varepsilon=1$ the risks remain low, it strongly increases at
$\varepsilon=8$ for HAM10000 ($0.03\%$ to $1.22\%$) and MSD Liver ($1.66\%$ to
$17.96\%$). At $\varepsilon=20$ theoretically, up to $74.24\%$ of all data
samples of the MSD Liver dataset can be reconstructed.

However, even minimally relaxing the assumptions for such an adversary to a
threat model decreases the risk associated with these privacy budgets
drastically. We recall that under this threat model, the only thing which is
relaxed compared to the worst-case is that the attacker does not know the
sample, which is reconstructed beforehand. Yet, for theoretical analysis,
there is still the assumption that the reconstruction algorithm is either
perfect or fails and the risk which is then calculated is the maximum rate
where the attacker correctly decides if the reconstruction they obtained was
indeed the dataset sample in question. This threat model is still too
pessimistic for any real-world use case and the analysis is mostly for
theoretical purposes. Still, such a practically irrelevant minimal relaxation
already gives a much more favourable risk profile, especially for medium
privacy budgets. Exemplarily, the risk associated with $\varepsilon=20$ goes
down from over $20\%$ to less than $1\%$ for the HAM10000 dataset. Similarly,
the risk for the MSD dataset at $\varepsilon=8$ decreases from $18\%$ to
$4\%$. A visualisation of the risk difference in worst-case and relaxed threat
models can be found in Figure 2.

###  2.4 Empirical Protection is Provided Even at Large Privacy Budgets

The previously discussed theoretical analyses show rapidly growing risks
associated with small and medium privacy budgets. However, as discussed
before, we argue that these analyses are too strict for any realistic use-
case. Hence, we ask what is the worst case of any practical scenario. We find
this as a federated learning setup, where a central server coordinates the
learning on the data of distributed clients, which follow each training
command sent by the server. This implies that the server can freely choose any
network architecture and hyperparameters. Note that any client who performs a
simple check would notice such a malicious server. For such cases, attacks
have been shown in literature, which analytically can recover the model input
perfectly [8, 9]. We employ these attacks as empirical risk assessments. For
our analysis to measure the reconstruction success, we use the structural
similarity score, which is a standard metric for image similarity [37].

In contrast to the aforementioned theoretical risk bounds, we find that for
practical attacks even so far considered meaningless privacy budgets
$\varepsilon>10^{9}$ can provide effective protection against successful
reconstruction. In the left column of Figure 3, we visually demonstrate for
several privacy budgets the step curve of how many dataset images are below an
increasing structural similarity (SSIM) error. It can be thought of as the
cumulative density function of the distribution of reconstruction errors. We
observe that for all datasets without the addition of DP constraints, nearly
all images can be reconstructed perfectly. As soon as some privacy guarantee
is introduced, even very generous budgets at an $\varepsilon$ of one billion
provide empirical protection against the reconstruction of data samples.
Furthermore, confirming previous works [8, 38], our threat model is still
extremely powerful. A server without the control of hyperparameters but still
over the model architecture already imposes a substantially lower
reconstruction risk. If the server does not set the batch size to one but is
set to the real training batch size, for example, on the RadImagenet dataset
even in the non-private case we could only reconstruct less than $5\%$ of all
images at a batch size of $3328$. We note that such large privacy budgets we
found to still protect are so far universally shunned by the privacy community
as meaningless. In fact, it is remarkable as $\varepsilon$ is an exponential
parameter, implying that $\varepsilon=10^{9}$ is $e^{10^{9}-1}$ times less
private than $\varepsilon=1$. This number is almost inconceivably large and,
for example, much larger than the number of atoms in the universe. Yet, we
show that even this pinch of privacy has drastic effects if practical
scenarios instead of worst-case analyses are considered. Complemented by the
above-discussed finding that performance trade-offs melt away in these
minimally restrictive realms, this signifies a potential compromise between
protection and real-world usability.

##  3 Discussion

In this study, we explore the relationship between privacy risks and AI
performance in the context of sensitive applications, such as medical imaging.
Currently, practitioners are confronted with a trade-off triangle where AI
performance, privacy protection, and computational efficiency are conflicting,
and no solution has so far been found to facilitate all of these goals.
Previous work showed that pretraining on a massive dataset of four billion
images could allow models to transfer to private datasets [39]. However, in
practice this is typically not feasible due to limited access to such large
datasets or the computational resources to train such a model. Furthermore,
such data scales only exist for natural 2D images but not yet for 3D images,
which are typical in medical imaging. Therefore, often the choice remains for
practitioners to prioritise privacy and sacrifice performance or to put
entrusted sensitive data at risk of being leaked. Currently, there is no clear
method to balance these two objectives, leaving practitioners without
guidance. To make informed decisions on how to trade off these competing
interests, a broad discourse involving ethicists, lawmakers, and the general
population is crucial. A prerequisite of this dialogue is understanding the
risks associated with specific privacy budgets and the potential trade-offs in
AI performance. Our comprehensive study across three representative medical
imaging datasets with different characteristics lays the foundation for this
conversation. We find that a significant category of risks can factually be
averted without this trade-off. In fact, privacy-performance trade-offs have
so far always been based on worst-case assumptions, which do not overlap with
realistic training settings. In most real-world scenarios, an attacker will
not already have access to the data, as they could simply use the dataset to
obtain the information in that case. Furthermore, because of its maximal
simplicity usually considered inference membership attack of an already
available data point on an AI model gives the attacker only minimal additional
information. We postulate that it is more critical to prevent the
reconstruction of sensitive data in the worst real-world setting. We show that
for derisking the threat of reconstruction attacks in realistic settings large
privacy budgets suffice. Even more, we find that the trade-off between privacy
risks and model performance vanishes when using such large but protective
privacy budgets. This compromises privacy risks, model performance and
computational efficiency and could allow a widespread breakthrough for the use
of DP in medical settings.

From previous studies [40, 41, 19, 25], we already know that privacy-enhancing
techniques protect AI models in sensitive contexts from reconstruction
attacks. We found that even considerably relaxed privacy budgets can offer
substantial protection against data reconstruction attacks in realistic
settings. In such scenarios, the AI performance loss is negligible. While we
note that our results are based on an empirical analysis, it is apparent that
DP training with minimal guarantees still provides better protection than non-
private AI training. Considering this finding that we can have some protection
without strong decreases in model performance, it seems negligent to train AI
models without any form of formal privacy guarantee. We note, that for all
intents an purposes our empirical attack can be considered sufficient for all
current attack scenarios. However, as we have no theoretical lower bound, the
possibility exists that future attacks could achieve success closer to the
upper bound. Because of this, we explicitly warn readers to take our results
as a carte blanche to use arbitrarily high privacy budgets. The truth lies in
the middle: If the alternative is to not use any privacy at all, rather use DP
with a very high budget.

On this note, we remark that the effectiveness of the DP protection against
attacks is a factor of the noise multiplier at a constant clip norm, batch
size, duration of the training and the training set size. However, as shown,
these noise multipliers are small enough to be converted to large privacy
budgets and facilitate high-performing AI models. We also observed that the AI
performance loss introduced by DP tends to be smaller on larger datasets, due
to less injected noise per sample and more information to achieve a certain
privacy budget at consistent hyperparameters. Yet, many medical datasets are
inherently small and as seen above strong privacy protection can prevent
successful learning of high-quality tumour segmentations. This can have
negative consequences for the applicability of such networks in clinical
practice. For models to be effectively trained on such challenging datasets,
when pretraining is not possible for reasons of data availability or
computational resources our techniques come to a limit and there might be a
need to either accept elevated privacy risks or obtain access to more data.
The solution to both problems might go hand in hand as with more robust and
unbreakable mathematical guarantees safeguarding data privacy, we anticipate
that patients may be more inclined to share their data, thereby allowing
large-scale medical AI training. If such a scenario occurs the privacy-
performance trade-offs presented might even be more favourable than our
findings indicate. This would be complemented by a workflow, where multiple
privacy-enhancing technologies are employed to enable various aspects to
privacy. For example, a system using Federated Learning to assert the data
governance remains at the original hospital, Secure Aggregation to conceal
contributions from different sites, and Differential Privacy to limit the
private information of single patients demonstrated in previous works [40]
would provide a holistic workflow.

We note that our choice of datasets and architectures is motivated by medical
imaging settings. In those settings, typically computational resources are
limited and data is scarce. In fact, we are convinced that the widespread use
of such methods will only ensue once they can be used by the majority of
practitioners who typically are without access to large computing clusters.
Hence, we carefully designed our study to cover typical and representative
medical problems to provide a holistic analysis with trade-offs in
computational resources. Under these considerations, we limited ourselves to a
few model architectures, which are known to be trained efficiently (ResNet,
DenseNet, UNet) and datasets, which represent a broad range of typical
problems.

An additional technical limitation stems from the fact that the authors of the
RadImagenet dataset [28] mention, that some patients contributed multiple
images. However, we have no information about image-to-patient correspondence.
As we calculate the privacy guarantees over the dataset per image, the per-
patient privacy guarantee depends on the number of images one patient
contributed and might be significantly lower.

In conclusion, we show that even the use of very loose privacy guarantees
still provides substantially better protection than standard AI training,
while achieving comparable performance. This can be the compromise between
provable risk management and performance trade-offs, which previously
prevented the breakthrough of DP. Further research should be directed towards
analysing various threat models such that not only worst-case scenarios are
considered. Only by illuminating the risks for multiple scenarios the basis
for a wide discussion among ethicists, policymakers, patients and other
stakeholders is provided, on how to trade-off privacy and performance as
fundamental goals of AI in sensitive applications.

##  4 Methods

In this section we report all details necessary for our experiments on
training models in a differentially private way on our datasets as well as the
procedures to analyse risk profiles. Furthermore, we describe the rationale
for several choices in our study design and describe hyperparameters necessary
for reproducibility.

###  4.1 Data

In Section 2.1.1 we described characteristics of typical medical datasets. We
note, that these characteristics partially amplify the negative performance
impact by the constraints introduced by DP. Broadly speaking, at a constant
clipping norm the amount of introduced noise during the DP-process determines
the negative impact on the AI performance. At any privacy budget the injected
noise increases if more training steps are performed or if a higher sampling
rate, i.e., the ratio between batch size and dataset size, is used. However,
the batch size is typically irrespective of the dataset size, which implies
that smaller datasets typically have higher sampling rates. Furthermore, they
often require more training epochs, i.e., the amount of times the entire
dataset was (on average) presented to the network. As a consequence, the
amount of noise that is injected when training on small datasets compared to
larger ones is increased and higher performance penalties are expected.
Furthermore, DP bounds the magnitude a single sample can change the training.
This is important for training with imbalanced datasets with underrepresented
classes, which often suffer an additional performance loss [42].

For detailed descriptions of the datasets we refer to the original
publications [28, 29, 30, 31]. In the following, we describe modifications we
performed and the effects on the data distribution.

For the HAM10000 dataset [29] we merged classes into whether there is
indication for immediate treatment, which is still a medically important
distinction. By this we convert the multi-class classification problem into a
highly imbalanced binary classification problem. We categorised here as
follows:

Treatment indication
---
immediate | not immediate
actinic keratoses and intraepithelial carcinomas | melanocytic nevi
basal cell carcinomas | benign keratinocytic lesions
melanomas | dermatofibromas
| vascular lesions

In total, this dataset has $10\,015$ images, of which $1\,954$ are labelled
for immediate treatment and $8\,061$ are not.

###  4.2 Model Training

All of our experiments were performed using an NAdam optimiser, which is
extremely robust to learning rate changes allowing us to keep a consistent
learning rate of $2\mathrm{e}-3$. Input data was always normalised with the
mean and standard deviation of all images in the training set. For each
dataset, we perform a hyperparameter search, where we evaluate for one privacy
level ($\varepsilon=8$) and the non-private training the optimal setting for
architecture, batch size, loss weighting, and augmentation. In the non-private
case, we perform an early stopping strategy to determine the number of epochs.
In the private case, this is not possible as the number of epochs directly
influences the amount of added noise. However, previous works showed that
longer training almost always yields better results [34]. Yet to limit
training time we also search for the point of saturation. Also for reasons of
computational complexity, we assume that the optimal settings for these
parameters transfer to all other privacy regimes. Furthermore, we limit the
choice of architectures to a ResNet-9 with ScaleNorm and a WideResNet40-4,
which have in previous literature been proven to be especially suited for
differentially private training [43, 34]. In the segmentation case, we limit
ourselves to a standard U-Net [44, 45], where we optimise the number of
channels on the bottleneck. We then evaluate for each privacy setting
separately the optimal clipping norm. Again for reasons of computational
complexity, we evaluate this after one epoch and assume it transfers to longer
trainings. Finally, we train for each setting five models with different
random seeds and report the mean and standard deviation of the respective
performance metric.

All our models are trained from scratch, i.e., we have not pretrained on any
other dataset. This is because there is no good choice of a dataset for
pretraining. ImageNet, which for most computer vision tasks is the standard,
is not very effective for medical imaging tasks [28]. Furthermore, pretraining
on medical databases is unacceptable, as it risks leaking the information from
the pretraining data, which could be just as private [46, 47].

We used the Opacus [48] library for accounting the privacy loss. In
particular, we used an RDP accountant, as it provides numerically the most
stable implementation. We used an extension of the objax library [49] as
implementation for the DP-SGD algorithm.

We will open-source the program code used for this study upon acceptance of
the manuscript.

####  4.2.1 RadImagenet

As described in section 4.2, we analysed the architecture, number of epochs,
batch size, loss and multiplicity for the non-private and one private setting
($\varepsilon=8$). For the non-private case, we found a WideResNet40-4 using
an unweighted loss function, a batch size of 16, and random vertical ($p=0.2$)
and horizontal flips ($p=0.1$) as augmentation to yield the best results. To
determine the number of epochs, we used an early stopping strategy with a
patience of 5 epochs and $0.1\%$ improvement threshold. For the private case,
a ResNet-9 trained for 50 epochs, using an unweighted loss function, using an
augmentation multiplicity of 4 again with random vertical ($p=0.2$) and
horizontal ($p=0.2$) flips with a batch size of 3328 yielded best results. The
clipping norm was tuned for each budget separately and was set as follows:

$\varepsilon$ | $1$ | $8$ | $32$ | $1\mathrm{e}12$
---|---|---|---|---
Clip Norm | $6.46$ | $5.66$ | $5$ | $3.75$

####  4.2.2 HAM10000

For the modified HAM10000 dataset we found the ResNet9 to perform best in
private and non-private settings. In the non-private case, we trained with a
weighted loss function at a batch size of $32$ using random vertical flips
($p=0.5$) as augmentation. We trained using an early stopping strategy using a
patience of $50$ epochs at a minimal improvement threshold of $0.1\%$. For the
private case, we used an unweighted loss function at a batch size of 2048 and
trained for 100 epochs. We used the same augmentations as in the non-private
case for a privacy level of $\varepsilon=10^{9}$, for all others, we did not
use augmentations. Clipping norms are as follows:

$\varepsilon$ | $1$ | $8$ | $20$ | $1\mathrm{e}9$
---|---|---|---|---
Clip Norm | $18$ | $8.5$ | $9.5$ | $9$

####  4.2.3 MSD Liver

For the MSD Liver dataset, we found for both private and non-private cases a
U-Net with $16$ channels and no augmentations to perform best. In the non-
private case we used a weighted loss function (Background: $0.1$, Liver:
$0.4$, Tumour: $0.5$) and trained at a batch size of two. Again, we employed
an early stopping strategy with a patience of 50 epochs and a minimal
improvement threshold of $0.1\%$. In the private case, we trained at a batch
size of one for $500$ epochs. For privacy budgets $\varepsilon\leq 20$ we used
an unweighted loss function, for higher privacy budgets we used the same
weighting as in the non-private case.

$\varepsilon$ | $1$ | $8$ | $20$ | $1\mathrm{e}9$
---|---|---|---|---
Clip Norm | $0.0004$ | $0.046$ | $0.0015$ | $0.33$

###  4.3 Reconstruction Risk Analysis

In our empirical reconstruction attacks there is no clear way to evaluate
whether a specific sample was reconstructed. For each input batch consisting
of $N$ samples, we receive $M$ reconstructions. We evaluate this by
calculating the pairwise distance between all data samples and reconstructions
and assigning each input the reconstruction with the lowest distance. However,
this approach loses meaning in the case of images, which have no structure but
are entirely dark. This is the case for the RadImagenet dataset, where we put
a constraint, that only data samples are considered, which contain more than
10% non-zero pixels.

We evaluate the practical reconstruction success by using a principle
demonstrated in previous literature [8, 9] adapted to our use case. The
network architecture is slightly modified by prepending two linear layers in
front of the actual network architecture. The first takes all input image
pixels as input and projects them to an intermediate representation of $N$
bins. In our experiments, we set $N=10$. This intermediate representation is
afterwards projected again to the number of all pixels and resized to the
original image shape. To each of the outputs, the mean of the intermediate
representations is added. Afterwards, it can be processed as usual by the
remaining neural network. Because our adversary is assumed to have control
over all hyperparameters they can set the batch size to one and by that
enforce that no reconstruction of two images overlap. If now a gradient is
calculated over the network, which is non-zero for the weights $W_{i}$ and
biases $b$ of the first linear layer the input $x$ can be analytically
recovered by
$x=\nabla_{W_{i}}\mathcal{L}\oslash\frac{\partial\mathcal{L}}{\partial b}$,
where $\oslash$ is the elementwise division. We note that for this attack it
is irrelevant what network architecture comes after this imprint block. We
used implementations provided by [50].

###  4.4 Choice of privacy budgets.

For our experiments on the utility trade-off we chose several privacy budgets.
We note that this choice was arbitrary. For all experiments, we used a
$\delta=8\cdot 10^{-7}$. For all settings, we evaluated $\varepsilon=1$ and
$\varepsilon=8$, which are standard values in literature [43, 34, 35].
Furthermore, we calculate the theoretical reconstruction bound of the worst
case and relaxed threat models. As the already included privacy budgets at
$\varepsilon=1$ and $\varepsilon=8$ already showcase very low reconstruction
bounds, we add one more privacy level for all datasets, where a significant
amount of samples is already at risk of being reconstructed. In addition, we
report a privacy budget $\varepsilon=10^{3N},N\in\mathbb{N}$, where the
characteristic reconstruction robustness curve is still similar to random
noise.

## Competing interest

The authors declare no competing interest.

## Data Availability Statement

All datasets used in this study are published and publicly available. Access
to RadImageNet [28] must be requested at https://www.radimagenet.com/. The
HAM10000 dataset [29] is available at https://doi.org/10.7910/DVN/DBW86T. The
MSD Liver dataset [30, 31] is available at http://medicaldecathlon.com/.

## Code Availability Statement

Our program code will be publicly released upon acceptance of this manuscript.
Prior use can be requested via inquiry to the authors.

## Environmental impact

Lastly, we would like to give a rough estimate of the climate impact of this
study. We assume the average German power mix which as of 2021 according to
the German Federal Environment Agency corresponds to 420g CO2/kWh. Only the
final RadImagenet trainings (no hyperparameter optimisation) ran on 8 NVIDIA
A40, where we assume a power consumption of 250W on average, each for almost 4
days, 5 privacy levels, and 5 repetitions. Hence, this amounts to around
960kWh and thus more than 400kg of CO2 equivalents. This almost equals a
return flight from Munich to London. Hence, we tried to limit our
hyperparameter searches to the necessary. In total, we assume that this study
produced at least 2 tons of CO2 equivalents.

## References

  * Lång et al. [2023] Kristina Lång, Viktoria Josefsson, Anna-Maria Larsson, Stefan Larsson, Charlotte Högberg, Hanna Sartor, Solveig Hofvind, Ingvar Andersson, and Aldana Rosso.  Artificial intelligence-supported screen reading versus standard double reading in the mammography screening with artificial intelligence trial (masai): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study.  _The Lancet Oncology_ , 24(8):936–944, 2023.
  * Wang et al. [2023] Guangyu Wang, Xiaohong Liu, Kai Wang, Yuanxu Gao, Gen Li, Daniel T Baptista-Hon, Xiaohong Helena Yang, Kanmin Xue, Wa Hou Tai, Zeyu Jiang, et al.  Deep-learning-enabled protein–protein interaction analysis for prediction of sars-cov-2 infectivity and variant evolution.  _Nature Medicine_ , pages 1–12, 2023.
  * Al-Zaiti et al. [2023] Salah S Al-Zaiti, Christian Martin-Gill, Jessica K Zègre-Hemsey, Zeineb Bouzid, Ziad Faramand, Mohammad O Alrawashdeh, Richard E Gregg, Stephanie Helman, Nathan T Riek, Karina Kraevsky-Phillips, et al.  Machine learning for ecg diagnosis and risk stratification of occlusion myocardial infarction.  _Nature Medicine_ , pages 1–10, 2023.
  * Singhal et al. [2023] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  _Nature_ , pages 1–9, 2023.
  * Jiang et al. [2023] Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, et al.  Health system-scale language models are all-purpose prediction engines.  _Nature_ , pages 1–6, 2023.
  * Geiping et al. [2020] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller.  Inverting gradients-how easy is it to break privacy in federated learning?  _Advances in Neural Information Processing Systems_ , 33:16937–16947, 2020.
  * Yin et al. [2021] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov.  See through gradients: Image batch recovery via gradinversion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16337–16346, 2021.
  * Fowl et al. [2022] Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, and Tom Goldstein.  Robbing the fed: Directly obtaining private data in federated learning with modified models.  _Tenth International Conference on Learning Representations_ , 2022.
  * Boenisch et al. [2023] Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot.  When the curious abandon honesty: Federated learning is not private.  In _2023 IEEE 8th European Symposium on Security and Privacy (EuroS &P)_, pages 175–199. IEEE, 2023.
  * Wang et al. [2021] Kuan-Chieh Wang, Yan Fu, Ke Li, Ashish Khisti, Richard Zemel, and Alireza Makhzani.  Variational model inversion attacks.  _Advances in Neural Information Processing Systems_ , 34:9706–9719, 2021.
  * Haim et al. [2022] Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani.  Reconstructing training data from trained neural networks.  _Advances in Neural Information Processing Systems_ , 35:22911–22924, 2022.
  * Carlini et al. [2023] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  _arXiv preprint arXiv:2301.13188_ , 2023.
  * Buzaglo et al. [2023] Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin, and Michal Irani.  Deconstructing data reconstruction: Multiclass, weight decay and general losses.  _arXiv preprint arXiv:2307.01827_ , 2023.
  * Narayanan and Shmatikov [2008] Arvind Narayanan and Vitaly Shmatikov.  Robust de-anonymization of large sparse datasets.  In _2008 IEEE Symposium on Security and Privacy (sp 2008)_ , pages 111–125. IEEE, 2008.
  * Cohen and Nissim [2020] Aloni Cohen and Kobbi Nissim.  Towards formalizing the gdpr’s notion of singling out.  _Proceedings of the National Academy of Sciences_ , 117(15):8344–8352, 2020.
  * Cohen [2022] Aloni Cohen.  Attacks on deidentification’s defenses.  In _31st USENIX Security Symposium (USENIX Security 22)_ , pages 1469–1486, 2022.
  * Schwarz et al. [2019] Christopher G Schwarz, Walter K Kremers, Terry M Therneau, Richard R Sharp, Jeffrey L Gunter, Prashanthi Vemuri, Arvin Arani, Anthony J Spychalla, Kejal Kantarci, David S Knopman, et al.  Identification of anonymous mri research participants with face-recognition software.  _New England Journal of Medicine_ , 381(17):1684–1686, 2019.
  * Dwork et al. [2014] Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  _Found. Trends Theor. Comput. Sci._ , 9(3-4):211–407, 2014.
  * Balle et al. [2022] Borja Balle, Giovanni Cherubin, and Jamie Hayes.  Reconstructing training data with informed adversaries.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 1138–1156. IEEE, 2022.
  * Kaissis et al. [2023a] Georgios Kaissis, Jamie Hayes, Alexander Ziller, and Daniel Rueckert.  Bounding data reconstruction attacks with the hypothesis testing interpretation of differential privacy.  _arXiv preprint arXiv:2307.03928_ , 2023a.
  * Nissim [2021] Kobbi Nissim.  Privacy: From database reconstruction to legal theorems.  In _Proceedings of the 40th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems_ , pages 33–41, 2021.
  * Parliament and of the Council [2021] European Parliament and of the Council.  Regulation laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts, 2021.  URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206.  Document 52021PC0206.
  * Foote et al. [2019] Andrew David Foote, Ashwin Machanavajjhala, and Kevin McKinney.  Releasing earnings distributions using differential privacy: disclosure avoidance system for post-secondary employment outcomes (pseo).  _Journal of Privacy and Confidentiality_ , 9(2), 2019.
  * Aktay et al. [2020] Ahmet Aktay, Shailesh Bavadekar, Gwen Cossoul, John Davis, Damien Desfontaines, Alex Fabrikant, Evgeniy Gabrilovich, Krishna Gadepalli, Bryant Gipson, Miguel Guevara, et al.  Google covid-19 community mobility reports: anonymization process description (version 1.1).  _arXiv preprint arXiv:2004.04145_ , 2020.
  * Hayes et al. [2023] Jamie Hayes, Saeed Mahloujifar, and Borja Balle.  Bounding training data reconstruction in dp-sgd.  _arXiv preprint arXiv:2302.07225_ , 2023.
  * Kaissis et al. [2023b] Georgios Kaissis, Alexander Ziller, Stefan Kolek, Anneliese Riess, and Daniel Rueckert.  Optimal privacy guarantees for a relaxed threat model: Addressing sub-optimal adversaries in differentially private machine learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.
  * [27] U.S. Food and Drug Administration.  Artificial intelligence and machine learning (ai/ml)-enabled medical devices.  https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices.  URL https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices.
  * Mei et al. [0] Xueyan Mei, Zelong Liu, Philip M. Robson, Brett Marinelli, Mingqian Huang, Amish Doshi, Adam Jacobi, Chendi Cao, Katherine E. Link, Thomas Yang, Ying Wang, Hayit Greenspan, Timothy Deyer, Zahi A. Fayad, and Yang Yang.  Radimagenet: An open radiologic deep learning research dataset for effective transfer learning.  _Radiology: Artificial Intelligence_ , 0(ja):e210315, 0.  doi: 10.1148/ryai.210315.  URL https://doi.org/10.1148/ryai.210315.
  * Tschandl et al. [2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific data_ , 5(1):1–9, 2018.
  * Simpson et al. [2019] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms.  _arXiv preprint arXiv:1902.09063_ , 2019.
  * Antonelli et al. [2022] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  _Nature communications_ , 13(1):4128, 2022.
  * Matthews [1975] Brian W Matthews.  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  _Biochimica et Biophysica Acta (BBA)-Protein Structure_ , 405(2):442–451, 1975.
  * Chicco and Jurman [2020] Davide Chicco and Giuseppe Jurman.  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  _BMC genomics_ , 21:1–13, 2020.
  * De et al. [2022] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle.  Unlocking high-accuracy differentially private image classification through scale.  _arXiv preprint arXiv:2204.13650_ , 2022.
  * Sander et al. [2022] Tom Sander, Pierre Stock, and Alexandre Sablayrolles.  Tan without a burn: Scaling laws of dp-sgd.  _arXiv preprint arXiv:2210.03403_ , 2022.
  * Stock et al. [2022] Pierre Stock, Igor Shilov, Ilya Mironov, and Alexandre Sablayrolles.  Defending against reconstruction attacks with r$\backslash$’enyi differential privacy.  _arXiv preprint arXiv:2202.07623_ , 2022.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Usynin et al. [2022] Dmitrii Usynin, Daniel Rueckert, Jonathan Passerat-Palmbach, and Georgios Kaissis.  Zen and the art of model adaptation: Low-utility-cost attack mitigations in collaborative machine learning.  _Proc. Priv. Enhancing Technol._ , 2022(1):274–290, 2022.
  * Berrada et al. [2023] Leonard Berrada, Soham De, Judy Hanwen Shen, Jamie Hayes, Robert Stanforth, David Stutz, Pushmeet Kohli, Samuel L Smith, and Borja Balle.  Unlocking accuracy and fairness in differentially private image classification.  _arXiv preprint arXiv:2308.10888_ , 2023.
  * Kaissis et al. [2021] Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  _Nature Machine Intelligence_ , 3(6):473–484, 2021.
  * Ziegler et al. [2022] Joceline Ziegler, Bjarne Pfitzner, Heinrich Schulz, Axel Saalbach, and Bert Arnrich.  Defending against reconstruction attacks through differentially private federated learning for classification of heterogeneous chest x-ray data.  _Sensors_ , 22(14):5195, 2022.
  * Bagdasaryan et al. [2019] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov.  Differential privacy has disparate impact on model accuracy.  _Advances in neural information processing systems_ , 32, 2019.
  * Klause et al. [2022] Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis.  Differentially private training of residual networks with scale normalisation.  _Theory and Practice of Differential Privacy Workshop, ICML_ , 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Çiçek et al. [2016] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19_ , pages 424–432. Springer, 2016.
  * Abascal et al. [2023] John Abascal, Stanley Wu, Alina Oprea, and Jonathan Ullman.  Tmi! finetuned models leak private information from their pretraining data.  _arXiv preprint arXiv:2306.01181_ , 2023.
  * Tramèr et al. [2022] Florian Tramèr, Gautam Kamath, and Nicholas Carlini.  Considerations for differentially private learning with large-scale public pretraining.  _arXiv preprint arXiv:2212.06470_ , 2022.
  * [48] Opacus.  Opacus PyTorch library.  Available from opacus.ai.
  * Objax Developers [2020] Objax Developers.  Objax, 2020.  URL https://github.com/google/objax.
  * Jonas Geiping [2023] Yuxin Wen Jonas Geiping, Liam Fowl.  Breaching.  https://github.com/JonasGeiping/breaching, April 2023.

  *[AI]: Artificial Intelligence
  *[GDPR]: General Data Protection Regulation
  *[MRI]: magnetic resonance imaging
  *[DP]: Differential Privacy
  *[ReRo]: reconstruction robustness
  *[CT]: computed tomography
  *[SSIM]: structural similarity
  *[MCC]: Matthews’ Correlation Coefficient