# DPAdapter: Improving Differentially Private Deep Learning through
Noise Tolerance Pre-training

Zihao Wang1 , Rui Zhu111footnotemark: 1 , Dongruo Zhou1, Zhikun Zhang3, John
Mitchell2,
Haixu Tang1, and XiaoFeng Wang1
1Indiana University Bloomington 2Stanford University
3CISPA Helmholtz Center for Information Security The first two authors
contributed equally to this work.

###### Abstract

Recent developments have underscored the critical role of differential privacy
(DP) in safeguarding individual data for training machine learning models.
However, integrating DP oftentimes incurs significant model performance
degradation due to the perturbation introduced into the training process,
presenting a formidable challenge in the differentially private machine
learning (DPML) field. To this end, several mitigative efforts have been
proposed, typically revolving around formulating new DPML algorithms or
relaxing DP definitions to harmonize with distinct contexts. In spite of these
initiatives, the diminishment induced by DP on models, particularly large-
scale models, remains substantial and thus, necessitates an innovative
solution that adeptly circumnavigates the consequential impairment of model
utility.

In response, we introduce DPAdapter, a pioneering technique designed to
amplify the model performance of DPML algorithms by enhancing parameter
robustness. The fundamental intuition behind this strategy is that models with
robust parameters are inherently more resistant to the noise introduced by DP,
thereby retaining better performance despite the perturbations. DPAdapter
modifies and enhances the sharpness-aware minimization (SAM) technique,
utilizing a two-batch strategy to provide a more accurate perturbation
estimate and an efficient gradient descent, thereby improving parameter
robustness against noise. Notably, DPAdapter can act as a plug-and-play
component and be combined with existing DPML algorithms to further improve
their performance. Our experiments show that DPAdapter vastly enhances state-
of-the-art DPML algorithms, increasing average accuracy from 72.92% to 77.09%
with a privacy budget of $\epsilon=4$.

##  1 Introduction

Recent years have witnessed an exponential growth in applications of deep
neural networks (DNNs) to various domains [29, 51, 14]. However, DNN models
trained with standard pipelines can be attacked by an adversary that seeks to
reveal the data on which the model was trained. For example, Carlini et al.
[8] show that adversaries can generate and detect text sequences from the
training set of a large transformer language model, while Balle et al. [5]
demonstrate that powerful adversaries can reconstruct images from the training
set of a classifier. Alongside other results [39, 7, 37], these studies
indicate that models trained on sensitive datasets present a significant
privacy risk. Differential privacy (DP) [21] has been the golden standard for
effective control of the risks of exposing training examples, and has already
been adopted by various machine learning tasks [62, 43, 36, 15] for
protection. A differentially private algorithm is a randomized algorithm
providing a formal guarantee that any single example in the training set only
has a limited impact on the output distributions of the algorithm. The privacy
guarantee, denoted ($\epsilon$, $\delta$)-DP, is defined by two parameters
($\epsilon$, $\delta$), which we refer to as the privacy budget. The smaller
these two parameters are, the closer the output distributions between the
training sets that differ by a single example, and therefore the more
difficult it becomes for an adversary to infer whether any single data point
is included during training.

Challenges in Private Deep Learning. Differentially private stochastic
gradient descent (DP-SGD) [1] stands out as a prevalent DPML technique. DP-SGD
modifies the conventional mini-batch gradient calculation used in SGD by
incorporating a privatized version, whereby the gradient of each training
sample is clipped to a maximum norm. Subsequently, Gaussian noise,
proportional to the clipping norm, is introduced to the sum of the clipped
gradients, masking the influence of any individual example on the sum.
However, the incorporation of the DP noise often comes with a notable
degradation in the performance of trained models [1, 54]. In response, efforts
to mitigate the adverse impact of DP on model utility have been made, either
through formulating new algorithms [47, 54, 63, 71] or by relaxing the
definition of DP to align with particular contexts [40, 20, 24]. Nonetheless,
the detriment imposed by DP noise on differentially private models remains
significant.

Our Solution. To mitigate the adverse impact of DP on model utility, prior
research has underscored that transfer learning [46] from public data markedly
enhances the model performance of DPML algorithms [34]. In our research, we
went a step further. Specifically, we observed that the deep learning training
process is highly sensitive to noise, with even a small amount of noise
leading to a substantial impact on performance. However, this issue can be
mitigated by the model pre-trained to minimize parameter sensitivity to noise,
which we call parameter robustness, so as to control DP’s performance impact.
Therefore, we designed a new technique that pre-trains a model with its
parameters robust to noise to enhance the performance of the downstream model
fine-turned with DPML on private data.

A technique that could serve this purpose is sharpness-aware minimization
(SAM), which augments parameter robustness [23, 59, 70] for the purpose of
enhancing a model’s generality. This technique, however, turns out to be less
effective in controlling the performance impact of DPML, since it is not
designed for maximizing parameter robustness. Concretely, SAM adds the worst-
case perturbation to parameters before computing gradients and later removes
the perturbation after each round of parameter updates. For this purpose, it
calculates both the perturbation and gradients on a small batch of training
samples. A problem is that although a small batch could be enough for
improving a model’s generality, it is inadequate for making an accurate
estimate of the worst-case perturbation, thereby rendering the model
parameters less robust than they could be. To address this issue, we enhanced
SAM for our purpose with a new technique called DPAdapter, which utilizes two
batches of training instances, a large one for a more accurate estimate of the
perturbation and a small one for effective gradient descent to ensure
convergence. We further theoretically analyzed why this approach improves
parameter robustness against noise, thereby reducing DPML’s performance impact
on the downstream model fine-tuning.

Empiricaly Results. We implemented DPAdapter and evaluated its performance
using a model pre-trained on CIFAR-100, which was then fine-tuned for
classification tasks on CIFAR-10, SVHN, and STL-10 datasets. In these
experiments, we utilized DPAdapter as the pre-training method and the
CIFAR-100 dataset as the public dataset for the pre-training. The downstream
tasks involve fine-tuning the pre-trained model with DPML. Our experimental
results consistently show a stable improvement in the accuracy achieved by the
downstream tasks, compared with those fine-tuned from the models without
robustness enhancement. For instance, when the privacy budget is set to
$\epsilon=4$ and DPML configured to DP-SGD (a popular setting), DPAdapter
elevates the average accuracy across three downstream tasks to 77.09%. By
comparison, when utilizing the pre-trained models without DPAdapter, the
accuracy is 72.92%, over 4% below that of our approach. Note that all existing
DPML enhancements could only achieve a performance gain no more than 3% [58]
when the privacy budget is set to $\epsilon=4$. Also, many of them cannot work
together, since they all focus on the downstream fine-tuning step. Our
approach, however, is designed for more generic protection, independent of
tasks, and compatible with these existing solutions, given its focus on the
pre-training step.

Contributions. Our key contributions are outlined below:

$\bullet$ New technique. We developed DPAdapter, a new technique for enhancing
parameter robustness, which leads to a general solution significantly
outperforming yet compatible with existing techniques for controlling DPML’s
negative impacts.

$\bullet$ Theoretical understandings. We theoretically analyzed our solution
to justify its effectiveness, unveiling intrinsic relations among parameter
robustness, transferability, and DPML’s performance impacts. Our analysis
leads to new insights about how a pre-trained model can be designed to
maximize the benefits of DPML.

$\bullet$ Extensive empirical studies. We conducted comprehensive empirical
studies of DPAdapter on various downstream tasks and DPML algorithms under
different privacy budgets. Our results show that our approach effectively
enhances the performance of the downstream tasks fine-tuned by DPML.

Roadmap. The rest of the paper is organized as follows. Section 2 presents the
background of our research; Section 3 presents our key observations; Section 4
elaborates our method and theoretic analysis; Section 5 reports the evaluation
of our techniques; Section 6 discusses the limitations of our study and
potential future work; Section 7 reviews the related prior studies and Section
8 concludes the paper.

##  2 Background

###  2.1 Transfer Learning

In the deep learning landscape, a particularly effective approach to model
training involves a two-step process: pre-training and fine-tuning. This
methodology, central to transfer learning [46, 60], allows practitioners to
leverage the power of large-scale data and subsequently specialize a model for
domain-specific tasks.

Pre-training serves as the first phase of transfer learning, often leveraging
unsupervised or self-supervised learning to train a model on a large, general
dataset [16, 9, 28, 11, 12]. The objective is to extract useful, high-level
features from the data without necessarily focusing on a specific task. The
pre-trained model acts as a feature extractor and captures generalized
knowledge from the initial dataset. This generalization facilitates transfer
learning, where the knowledge gained from one task can be transferred to a
different but related task.

The second phase, fine-tuning, builds upon the pre-trained model to specialize
it for a specific task. In this phase, the model is trained on a smaller,
labeled dataset that is directly related to the target task. The key
innovation here is the transfer of knowledge from the pre-training phase;
instead of initializing the neural network with random weights, the model
starts with the weights obtained during pre-training. Because the model has
already learned relevant features, it typically requires less data and fewer
epochs to fine-tune effectively. Fine-tuning essentially adapts the
generalized learning from the pre-training phase to a specific task,
demonstrating the power of transfer learning to yield high performance with
comparatively less data and computational effort.

###  2.2 Differentially Private Machine Learning

Differential privacy [21] (DP) serves as a mathematical structure intended to
quantify the assurance of privacy within data analysis, providing insights
into the preservation of privacy when an individual’s data becomes part of a
publicly analyzed dataset. This concept typically involves a privacy budget,
represented by $\epsilon$, where a larger value signifies augmented privacy
protection, and an optional failure probability, $\delta$, defines the
permissible deviation from impeccable privacy. Formally, we can define
differential privacy as:

######  Definition 2.1 (($\epsilon$, $\delta$)-DP).

Given two neighboring datasets $D$ and $D^{\prime}$ differing by one record, a
mechanism $\mathcal{M}$ satisfies ($\epsilon$, $\delta$)-differential privacy
if

$Pr[\mathcal{M}(D)\in S]\leq e^{\epsilon}\cdot Pr[\mathcal{M}(D^{\prime})\in
S]+\delta,$

where $\epsilon$ is the privacy budget, and $\delta$ is the failure
probability.

Gaussian Mechanism. There are several approaches for designing mechanisms that
satisfy $(\epsilon,\delta)$-differential privacy, among which the Gaussian
mechanism is the most widely used one. It computes a function $f$ on the
dataset $D$ in a differentially private manner, by adding to $f(D)$ a random
noise. The magnitude of the noise depends on $\Delta_{f}$, the _global
sensitivity_ or the $\ell_{2}$ sensitivity of $f$. Such a mechanism
$\mathcal{M}$ is given below:

$\begin{array}[]{crl}&\mathcal{M}(D)&=f(D)+\mathcal{N}\left(0,\Delta_{f}^{2}\sigma^{2}\mathbf{I}\right)\end{array}$

where $\Delta_{f}=\max\limits_{(D,D^{\prime}):D\simeq
D^{\prime}}||f(D)-f(D^{\prime})||_{2}$,
$\mathcal{N}(0,\Delta_{f}^{2}\sigma^{2}\mathbf{I})$ denotes a multi-
dimensional random variable sampled from the normal distribution with mean $0$
and standard deviation $\Delta_{f}\sigma$, and
$\sigma=\sqrt{2\ln\frac{1.25}{\delta}}/\epsilon$.

DP-SGD. The integration of differential privacy into deep learning aims to
build models that can learn from data without compromising the privacy of
individuals within the dataset. Differentially private stochastic gradient
descent (DP-SGD) [1] is the most widely used algorithm to enforce DP guarantee
for the deep learning models. It adapts the standard SGD algorithm by
introducing a few privacy-preserving modifications: gradient clipping and
noise addition.

The gradient clipping operation aims to limit the sensitivity of each
gradient, ensuring that a single data point does not unduly influence the
model’s learning process. Following gradient clipping, Gaussian noise is added
to the clipped gradients before they are used to update the model parameters.
This noise ensures that the exact values of the gradients—which could reveal
sensitive information—are masked.

Privacy Budget Composition. On a broader spectrum, the overall privacy budget
calculation of DP-SGD is established by demonstrating the privacy budget of
each iteration under certain $(\epsilon,\delta)$ values, followed by applying
amplification by subsampling and composition throughout the iterations.

###  2.3 Adversarial Robustness and Model Parameter Robustness

Adversarial Training (AT). It is widely regarded as the most effective defense
method against adversarial attacks [4, 25, 53, 48, 6]. AT essentially employs
a “fight fire with fire” approach. It introduces adversarial examples into the
training set, which are slight, carefully calculated modifications of the
original inputs. These modifications are designed to mislead the model into
making erroneous predictions. By learning from these adversarial examples, the
model improves its ability to correctly classify such inputs in the future,
thereby enhancing its overall resilience and robustness against adversarial
attacks.

Parameter Robustness. It typically refers to the sensitivity of a model to
small perturbations in its parameters. When discussing parameter robustness,
we are interested in the model output (e.g., predictions on testing data)
changes when its parameters are subjected to some level of noise.

Specifically, given a model $f$ parameterized by $\theta$ and a test sample
$x$, the robustness of the model with respect to its parameters can be defined
as:

$\rho(f)=\max_{\bm{x},\theta,\Delta}|f_{\theta+\Delta}(\bm{x})-f_{\theta}(\bm{x})|/\|\Delta\|_{2}$

where $\rho(f)$ represents the maximum change in the model’s output. $\Delta$
is the noise added to the model parameters $\theta$. $\|\cdot\|$ denotes a
norm, for instance, the L2 norm.

This definition provides a measure of the potential variation in the model’s
output when its parameters undergo perturbation due to noise. A diminutive
value of $\rho$ is indicative of pronounced parameter robustness.
Subsequently, we aim to enhance the parameter robustness of $f$, denoted by
the objective of reducing the value of $\rho(f)$.

Robust Accuracy. We resort to robust accuracy as an evaluation metric for
parameter robustness in the classification paradigm; Given a specific model
$f_{\theta}$, we introduce Gaussian noise, represented as $N(0,0.1)$,
uniformly across parameters spanning all layers. The resultant perturbed model
is represented as $f_{\hat{\theta}}$. The model’s performance, when faced with
this perturbation, is then assessed. Consequently, the robust accuracy is
computed over a dataset $D$ and is delineated as:

$\text{Robust Accuracy}=\frac{1}{|D|}\sum_{x\in D}I(f_{\hat{\theta}}(x)=y)$

where $I$ is the indicator function that returns 1 when the condition inside
is true and 0 otherwise. A model exhibiting a higher robust accuracy indicates
stronger parameter robustness, corresponding to a lower value of $\rho$.

The optimization of parameter robustness can be achieved through a technique
known as sharpness-aware minimization (SAM) [23, 59, 70]. This method
typically involves computing the worst-case perturbation on the parameters.
The optimization then takes place on the model parameters subjected to this
worst-case perturbation. The intention is to ensure that the model retains a
significant portion of its performance, even when subjected to the worst-case
perturbation. However, this technique is not designed to maximize parameter
robustness but to achieve better generality. Consequently, it proves to be
less effective in mitigating the performance impact of DPML (Section 5.2).

##  3 Problem Formulation and Key Observations

###  3.1 Problem Formulation

Threat Model. We consider a scenario where an attacker applies privacy attacks
[8, 5, 39, 7, 37] to unveil the training data of a model. To defend against
the attacks, the defender endeavors to build the model using differential
privacy, which provably protects sensitive data while preserving its original
utility. Additionally, the defender possesses a public dataset related to the
private data, aligning with the standard assumption of transfer learning that
the source and the target datasets follow similar but not identical
distributions.

Problem Definition. Our aim is to build a pre-trained model that can be used
for improving both the accuracy and privacy protection of the models trained
for downstream tasks. Specifically, to build a pre-trained model that serves
as a beneficial initialization for DP fine-tuning, we seek to answer two key
questions: 1) what properties the pre-trained model should exhibit for the
benefit of downstream models? and consequently 2) how the pre-training of the
model should be executed to acquire these properties?

###  3.2 Motivation and Key Observations

Motivation. Our motivation to enhance the performance of the DPML algorithm
using pre-trained models is rooted in our understanding of adversarial
robustness training. The traditional adversarial training (AT) incrementally
conditions the model to resist perturbations of its input. However, in the
DPML algorithm, the noise is predominantly introduced into the gradients, or
equivalently, into the model’s parameters. Hence, we ask if there is any
technique similar to AT that may condition the model to resist perturbations
of its parameters. Obviously, if this can be achieved, the impact of the noise
introduced by the DPML algorithm on the model performance could be minimized.
It turns out, such an AT method has been proposed previously, referred to as
the sharpness aware minimization (SAM), which confers noise resilience to
parameters Section 2.3. Based on the premise, we introduce two working
hypotheses.

We first consider a specialized transfer learning scenario where the training
datasets for the upstream and downstream tasks are i.i.d. (independent and
identically distributed); for instance, each of these training datasets
represent a random sub-sample of an overarching dataset. Under this scenario,
we propose the first hypothesis:

Hypothesis 1. Assuming the datasets $A$ and $B$ are randomly sampled from the
same distribution, if a pre-trained model built from the dataset $A$ possesses
enhanced parameter robustness, the model fine-tuned on the dataset $B$ using
the DPML algorithm under a fixed privacy budget will yield better performance.

Next, we extend our hypothesis into a more general transfer learning scenario,
where the upstream and downstream datasets are not identical but related, for
example, when the upstream training data comes from ImageNet and the
downstream data is sourced from CIFAR-10. Under this scenario, we have our
second hypothesis.

Hypothesis 2. Assuming the datasets $A$ and $B$ are sampled from two different
but similar distributions, respectively, if a pre-trained model built from the
dataset $A$ exhibits strong parameter robustness, the model fine-tuned on the
dataset $B$ using the DPML algorithm under a fixed privacy budget will yield
better performance.

Rationales. Hypothesis 1 is quite intuitive. Drawing parallels from AT — which
can counteract the effects of adversarial examples by introducing noise into
the input — we anticipate that SAM can similarly mitigate the implications of
the noise added to parameters by the DPML algorithm.

Hypothesis 2, on the other hand, is inspired by prior work [49]. It has been
observed that pre-trained models, when subjected to AT, can impart some degree
of their acquired adversarial robustness to downstream models via transfer
learning. This led us to a deeper contemplation: Can the parameter robustness
garnered through SAM training also be transferred to downstream tasks through
transfer learning? If this transfer of parameter robustness is indeed
feasible, then, based on Hypothesis 1, it could potentially alleviate the
performance degradation in downstream training using the DPML algorithm.

Empirical Validation. To evaluate our hypotheses, we utilized state-of-the-art
Adversarial Model Perturbation (AMP) methodologies. Our models were exposed to
an array of AMP intensities, producing pre-trained models that span a spectrum
of parameter robustness. We then measured their parameter robustness by
assessing the model accuracy on testing data after perturbation, and analyzed
their performance in conjunction with DP-SGD.

Figure 1: Impact of perturbation magnitude on AMP.

Specifically, to validate Hypothesis 1, we examined a scenario where both
upstream and downstream training data were sourced from the CIFAR-100 dataset,
with the upstream data constituting 90% of the CIFAR-100 training dataset and
the downstream data constituting the remaining 10%. As depicted in Figure 1,
the x-axis signifies the parameter $\gamma$ in AMP, which modulates the
intensity of parameter robustness. By adjusting $\gamma$, we derived pre-
trained models of various levels of robustness (indicated by the red line).
This robustness peaks with increasing $\gamma$ and then starts to wane.
Correspondingly, the green line, which illustrates the accuracy of the model’s
performance after downstream optimization using DP-SGD, displays an initial
rise followed by a decline, mirroring the trajectory of parameter robustness.
This result suggests that if the upstream and downstream training datasets
both follow an i.i.d. distribution, the parameter robustness of upstream model
is directly correlated with the performance of the downstream fine-tuned
model.

For Hypothesis 2, we used the upstream training data comprising the same 90%
of the CIFAR-100 dataset, while the downstream training data was entirely from
the CIFAR-10 dataset. Still referencing Figure 1, the blue line indicates the
performance (accuracy) of the downstream model after DP-SGD optimization.
Consistent with the results above, the blue line showcases an initial increase
in performance of the downstream model followed by a significant drop, which
is highly consistent with the pattern of the parameter robustness. This result
suggests that in a transfer learning scenario, the parameters robustness of an
upstream pre-trained model can influence the performance of the downstream
model trained using the DPML algorithm. Additionally, we observed a gradual
decline in the accuracy of the upstream model with an increase in its
parameter robustness, indicating a trade-off between the parameter robustness
and the accuracy of the model. If the accuracy of the upstream model drops too
significantly, it can detrimentally impact the performance of the downstream
model. Taking these observation into account, we outline our design objectives
below, aiming to illustrate our intent of devising a cost-effective strategy
to build a pre-trained model that strikes a balance between two goals, i.e.,
to increase the parameter robustness while maintaining the high accuracy of
the pre-trained model.

Figure 2: The relationship between the parameter robustness of the pretrained
models and that of the models fine-tuned on the downstream tasks.

We further explore the transferability of parameter robustness in the context
of transfer learning. In specific, we used CIFAR-100 as the upstream training
dataset, and employed the CIFAR-10, SVHN, and STL-10 datasets, respectively
for downstream training. We first trained the model on the upstream data with
five different intensities of Adversarial Model Perturbation (AMP)
characterized by the values $\gamma=0,0.1,0.2,0.5,$ and $1.0$. This procedure
yielded five sets of pre-trained models, each with distinct robust accuracies.
Subsequently, we performed transfer learning from each of these five distinct
pre-trained models on the downstream data. Upon the completion of training,
the robust accuracy of models on the downstream dataset was computed. Figure 2
shows the high between the robust accuracies of the pre-trained models
(x-axis) and the robust accuracy of downstream models (y-axis). In general, it
can be observed that the pre-trained models with higher robust accuracy tend
to maintain their enhanced robust accuracy during the transfer learning for
the downstream tasks. This observation underscores the inherent
transferability of parameter robustness in a transfer learning context.

Design Goals. Based on the results from the toy examples as illustrated above,
our objective is to build a pre-trained model that achieves two primary goals:
robustness and effectiveness, which we elaborate below.

$\bullet$ Robustness. As observed earlier, we aim for our pre-trained model to
exhibit strong parameter robustness (for instance, a smaller value of $\rho$).
This ensures that when the downstream model employs the DPML algorithm, the
influence on its performance is minimal.

$\bullet$ Generality. While striving for pronounced parameter robustness in
the pre-trained model, it is imperative that the model’s inherent performance
remains commendable. Should we solely emphasize parameter robustness, the
model may exhibit an extremely low value of $\rho$—indicating high
robustness—but low accuracy. For instance, $f(x;\theta+\Delta)=f(x;\theta)\neq
y$. As illustrated in Figure 1, when $\gamma$ exceeds 1, the pre-trained
model’s performance may deteriorate considerably. As a result, the DPML
algorithm employed for downstream training might not inherit sufficient
effective features from the pre-trained model, leading to a diminished
outcome. Hence, it is equally pivotal for the pre-trained model to retain high
performance as compared to enhance its parameter robustness.

##  4 DPAdapter

In this section, we present the design and implementation of the proposed
DPAdapter, together with a theoretical analysis of its effectiveness.

In essence, the design of DPAdapter is rooted in the two observations
mentioned earlier. Our objective is for the pre-trained model to achieve
higher parameter robustness while maintaining its performance. Given the
observation that parameter robustness can be transferred when the model is
passed downstream, the fine-tuning DPML algorithms can operate under the same
privacy budget. This, in turn, minimizes the impact on performance.

###  4.1 Design Challenge

We observe that while AMP can enhance parameter robustness, leading to an
improvement in the DP-SGD performance of downstream fine-tuned models, the
extent of this robustness enhancement is quite limited. This, in turn, results
in only a marginal performance boost. As we illustrated in Section 5.4 using
AMP as an example, the effect of existing model parameter adversarial training
on enhancing the parameter robustness of pre-trained models is not
significant. Consequently, the impact on the performance of downstream DPML
algorithms is also marginal. This is primarily because the existing methods
aimed at improving parameter robustness were primarily designed to enhance
model generalization. Since they were not explicitly designed for robustness
enhancement, the gains in parameter robustness tend to be limited. While these
methods can indeed be adjusted to prioritize robustness by increasing the
perturbation magnitude at the expense of generalization, such a trade-off is
often costly. As depicted in Figure 1, a high perturbation magnitude not only
fails to bring much-added robustness but also makes the model challenging to
converge. Given these challenges, we introduce DPAdapter, a novel approach to
help pre-trained models achieve higher parameter robustness and minimize the
adverse impact on performance when downstream models employ DPML algorithms.

Figure 3: Impact of different batch sizes on AMP.

###  4.2 Design Intuition

As highlighted in the “Challenge” section above, traditional methods of
enhancing parameter robustness often result in a significant trade-off between
parameter robustness and model performance. We’ve observed that these
conventional methods of improving parameter robustness primarily leverage
schemes akin to SGD for adversarial training. During a single optimization
iteration, the batch used to compute the perturbation and the batch used to
calculate the gradient direction for optimization are the same. However, the
rationale behind this approach has not been extensively discussed. In fact,
taking the AMP as an example, as depicted in Figure 3, when a large batch size
is chosen, the model’s training becomes sub-optimal due to the utilization of
this large batch. On the contrary, if the batch size is too small, computing
the perturbation based on such limited data leads to inaccurate perturbations,
subsequently compromising the robustness of the training. Note that a recent
study suggests employing a linear scaling rule to adjust the learning rate
based on the modified batch size [27]; specifically, the learning rate should
be multiplied by $k$ when using a batch size of $kN$. Consequently, in this
experiment, we also adhere to this setting to approximate the optimal learning
rate for each batch size.

Based on these observations, we decided to decouple the batches used for
computing perturbations and those for calculating parameter gradients during
the optimization process. For computing perturbations, we chose a batch size
larger than what is typically required for regular training. The reasoning is
that perturbation computations don’t necessarily benefit from being
stochastic. Ideally, the entire dataset would be used to compute the
perturbation. However, due to computational resource constraints, using the
complete dataset for this purpose is challenging. Hence, we opt for a
compromise by selecting a relatively larger batch size for the perturbation
computations. In contrast, for batches used to compute model optimization
gradients, we chose sizes that are conducive to model optimization, ensuring
we avoid sizes that are excessively large or small that could impair model
performance.

Figure 4: Overview of the DPAdapter approach. In the first step, known as
standard training, we use the training data to conduct standard training on
the model. This is analogous to the preheating process common in typical
Adversarial Training (AT), where the model’s accuracy is first brought up to a
standard level, resulting in the model $f_{\theta}$. Next, in step 2, We
selected a batch data $\mathcal{B}_{1}$ (big batch size) from the training
data, and utilize it to compute the worst-case model perturbation by Equation
3 (①), producing the model $f_{\theta+\Delta}$. In step 3, we use another
batch of data $\mathcal{B}_{2}$ to do sgd on the model $f_{\theta+\Delta}$
(②), resulting in the model $f_{\mathbf{w}+\Delta+\alpha}$. In step 4, we
reverse the perturbation made in step 2 within the model
$f_{\theta+\Delta+\alpha}$ (③), obtaining the model $f_{\theta+\alpha}$. Steps
2 to 4 are performed iteratively (④), and upon reaching the desired number of
iterations, we return the target model (⑤). This culminating model is tailored
to facilitate subsequent fine-tuning processes, enabling models built upon it
to more efficaciously deploy the DPML optimization algorithm(⑥). This ensures
not only the protection of the privacy of the fine-tuned dataset but also a
notable enhancement in DPML performance.

###  4.3 Methodology Overview

Figure 4 illustrates the overall workflow of DPAdapter. To efficiently achieve
the parameter robustness of pre-trained models, DPAdapter employs a Min-Max
optimization, iterating over four steps of model updating as presented below.

Step 1: Warming-up. The pre-trained model initiates with a warming-up phase
using standard training techniques to ensure the accuracy (ACC) meets the
targeted benchmark. We refer to the model post-warming-up as
$f_{\mathbf{\theta}}$.

Step 2: Worst-case Model Perturbation. We select an enlarged batch of training
data, denoted as $\mathcal{B}_{1}$. We then compute the direction of the
gradient that maximizes the loss of this batch under the current model
parameters. Subsequently, we perturb the model parameters along this direction
with the intention of impairing the model’s predictive capability on this
large batch of data when subjected to such perturbation. We denote the
perturbation as $\Delta$, and the model after perturbation as
$f_{\mathbf{\theta}+\Delta}$.

Step 3: Update on the Perturbed Model. To enhance the model’s resilience to
perturbations in its parameters (parameter robustness), we optimize the
perturbed model with the goal of maintaining good performance even when the
model is subjected to such perturbations. We select a standard-sized batch of
training data, represented as $\mathcal{B}_{2}$. We then perform regular
training using stochastic gradient descent (SGD) on $\mathcal{B}_{2}$. The
updated model is denoted as $f_{\mathbf{\theta}+\Delta+\alpha}$, where
$\alpha$ represents the weight update resulting from the SGD.

Step 4: Revert Perturbation. In Step 3, we have already enhanced the model’s
resilience to parameter perturbations. As a result, there’s no longer a need
to retain the noise applied to the parameters in Step 2, as doing so would
only degrade our model’s accuracy. Therefore, in this step, we revert the
update of the model weight made in Step 2: the final model is represented as
$f_{\mathbf{\theta}+\Delta+\alpha-\Delta}=f_{\mathbf{\theta}+\alpha}$.

Our workflow operates iteratively. Upon completion of Step 4, the process
loops back to Step 1. In each iteration, distinct batches of training data are
randomly selected for both $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$. The
algorithm takes the total number of iterations as an input, ensuring that
training concludes after the specified number of iterations.

Remark. It is important to highlight the distinction between our approach and
conventional techniques aimed at enhancing a model’s generality by improving
parameter robustness, such as AMP [70]. Specifically, AMP optimizes using the
same $\mathcal{B}_{2}$ and $\mathcal{B}_{1}$.

The key innovation in our method lies in the strategic batch selection during
two different optimization processes: the worst-case perturbation computation
and the SGD gradient computation. By ensuring that the batch size for
$\mathcal{B}_{1}$ is relatively large, we obtain a more precise and general
perturbation. In contrast, a standard batch size is used for $\mathcal{B}_{2}$
to facilitate optimal training with SGD. Such nuanced improvements enable our
pretrained model, DPAdapter, to achieve heightened parameter robustness,
resulting in significant enhancements when downstream DPSGD is applied.

###  4.4 Design Details

We now delve into a detailed presentation of the DPAdapter algorithm, focusing
primarily on Steps 2 and 3. These steps are comparatively more complex: while
Step 1 involves standard training and Step 4 simply involves a subtraction to
remove noise from the parameters. Notably, it’s Steps 2 and 3 that distinctly
set DPAdapter apart from other SAM algorithms.

First, like any typical SAM algorithm, we define a norm ball ${\rm\bf
B}(\bm{\mu};\gamma)$, as a region around a given point $\bm{\mu}$ in the
parameter space $\Theta$ (i.e., $\bm{\mu}\in\Theta$) with the radius of
$\gamma$ ($\gamma\geq 0$):

${\rm\bf
B}(\bm{\mu};\gamma):=\\{\bm{\theta}\in\Theta:\|\bm{\theta}-\bm{\mu}\|\leq\gamma\\}$
(1)

Recall that a typical SAM loss is designed upon the norm ball ${\rm\bf
B}(\theta;\gamma)$:

$\mathcal{L}_{\mathrm{SAM}}(\bm{\theta}):=\max_{\Delta\in{\rm\bf
B}(\bm{0};\gamma)}\frac{1}{|\mathcal{D}|}\sum_{(\bm{x},\bm{y})\in\mathcal{D}}\ell(\bm{x},\bm{y};\bm{\theta}+\Delta)$
(2)

where $\mathcal{D}$ represent the training dataset. The training loss, denoted
by $\ell$, varies based on the task at hand: for classification tasks, it is
typically the cross-entropy, while for regression tasks, it is often the least
squares. The perturbation $\Delta$ is the worst-case model perturbation

Worst-case Model Perturbation. In this section, we detail the computation of
$\Delta$ used to obtain $f_{\theta+\Delta}$ in Step 2. The worst-case model
perturbation, $\Delta$, is strategically designed to induce the model
parameters $\theta$ to achieve the fastest reduction in loss on the dataset:

$\Delta_{\mathcal{B}_{1}}=\operatorname*{arg\,max}_{\Delta\in{\rm\bf
B}(\bm{\theta};\gamma)}\frac{1}{|\mathcal{B}_{1}|}\sum_{(\bm{x},\bm{y})\in\mathcal{B}_{1}}\ell(\bm{x},\bm{y};\bm{\theta}+\Delta)$
(3)

Recall that in DPAdapter, $\mathcal{B}_{1}$ denote the batch of data use as to
compute the worst-case model perturbation. $\gamma$ denotes the radius of the
norm ball and acts as a hyperparameter that governs the degree of parameter
robustness training. The robustness of the model’s parameters is directly
influenced by this radius: a larger $\gamma$ implies that the model can
tolerate more noise in the parameter space, as determined by DPAdapter.
Consequently, a greater $\gamma$ results in a model with enhanced parameter
robustness.

Update Perturbed Model. In this section, we detail the computation of $\alpha$
used to obtain $f_{\theta+\Delta+\alpha}$ in Step 3. Since in practice, the
optimization is typically carried out using mini-batches. In this case, the
$\mathcal{L}_{\mathrm{DPAdapter}}$ can be approximated using a mini-batch
$\mathcal{B}_{2}$:

$\displaystyle\mathcal{L}_{\mathrm{~{}DPAdapter{}}}(\bm{\theta})$
$\displaystyle\approx\max_{\Delta_{\mathcal{B}_{1}}\in{\rm\bf
B}(\bm{0};\gamma)}\frac{1}{|\mathcal{B}_{2}|}\sum_{(\bm{x},\bm{y})\in\mathcal{B}_{2}}\ell(\bm{x},\bm{y};\bm{\theta}+\Delta_{\mathcal{B}_{1}})$
$\displaystyle:=\mathcal{J}_{\mathrm{~{}DPAdapter{}},\mathcal{B}_{1},\mathcal{B}_{2}}(\bm{\theta})$
(4)

Hence, at this point, the model’s update is given by

$\alpha=\eta_{2}\nabla\mathcal{J}_{\mathrm{~{}DPAdapter{}},\mathcal{B}_{1},\mathcal{B}_{2}}(\bm{\theta})$
(5)

where $\eta_{2}$ represents the optimization step size. It is crucial to
highlight the primary distinction between DPAdapter and other SAM algorithms,
which lies in Equation 3 and Section 4.4. In other SAM algorithms, such as
those cited in [70, 59, 23], the batch used to compute the worst-case
perturbation, denoted as $\mathcal{B}_{1}$, is the same as the batch utilized
for computing the update. We argue that by separately selecting
$\mathcal{B}_{1}$ and $\mathcal{B}_{2}$, the trained model can achieve a
balance between parameter robustness and overall model performance in Section
5.4.

The specifics of the DPAdapter algorithm can be found in Algorithm 1.
Importantly, in alignment with other SAM algorithms such as [70, 59, 23], the
finalized model utilizes the parameters denoted as
$\bm{\theta}^{\ast}_{\mathrm{DPAdapter}}$. These parameters are employed for
model inference (or prediction) without the need for additional perturbations.

As mentioned earlier, the training process of DPAdapter is divided into two
primary stages: The perturbation computation and the update computation.

$\bullet$ Perturbation Computation. Line 5 to Line 9 in Algorithm 1. This
phase is responsible for calculating the worst-case perturbation,
$\Delta_{\mathcal{B}_{1}}$, based on the prevailing conditions.

$\bullet$ Update Computation. Line 10 to Line 12 in Algorithm 1. In this
phase, the $\Delta_{\mathcal{B}_{1}}$ derived from the previous perturbation
computation step is used to determine the parameter optimization gradient,
$\nabla\mathcal{J}_{\mathrm{~{}DPAdapter{}},\mathcal{B}_{1},\mathcal{B}_{2}}$.
Following this, the model parameters are updated.

Training can be executed for a specified number of epochs $K$, or it can
continue until the parameter loss stabilizes, which indicates the completion
of the training process.

0: Training set $\mathcal{D}=\\{(\bm{x},\bm{y})\\}$, perturbation batch size
$m_{1}$, update batch size $m_{2}$, loss function $\ell$, perturbation
computation learning rate $\eta_{1}$, update learning rate $\eta_{2}$, norm
ball radius $\gamma$, number of update iteration $K$.  1: while $k<K$ do 2:
Draw $\mathcal{B}_{1}=\\{(\bm{x}_{i},\bm{y}_{i})\\}_{i=1}^{m_{1}}$ from
training set $\mathcal{D}$ 3: Draw
$\mathcal{B}_{2}=\\{(\bm{x}_{i},\bm{y}_{i})\\}_{i=1}^{m_{2}}$ from training
set $\mathcal{D}$ 4: Set perturbation:
$\Delta_{\mathcal{B}_{1}}\leftarrow\bm{0}$ 5: Calculate the perturbation:
$\qquad\Delta_{\mathcal{B}_{1}}\leftarrow\eta_{1}\operatorname*{arg\,max}_{\Delta\in{\rm\bf
B}(\bm{\theta_{k}};\gamma)}\frac{1}{|m_{1}|}\sum_{(\bm{x},\bm{y})\in\mathcal{B}_{1}}\ell(\bm{x},\bm{y};\bm{\theta_{k}}+\Delta)$
6: if $\|\Delta_{\mathcal{B}_{1}}\|_{2}>\gamma$ then 7: Normalize
perturbation:
$\Delta_{\mathcal{B}_{1}}\leftarrow\gamma\Delta_{\mathcal{B}_{1}}/\|\Delta_{\mathcal{B}_{1}}\|_{2}$
8: end if 9: Add the perturbation to parameters:
$\qquad\bm{\theta}_{k}\leftarrow\bm{\theta}_{k}+\Delta_{\mathcal{B}_{1}}$ 10:
Compute update gradient:
$\qquad\nabla\mathcal{J}_{\mathrm{~{}DPAdapter{}},{\mathcal{B}_{1}},{\mathcal{B}_{2}}}\leftarrow\sum_{i=1}^{|\mathcal{B}_{2}|}\nabla_{\bm{\theta}}\ell(\bm{x}_{i},\bm{y}_{i};\bm{\theta}_{k})/{m_{2}}$
11: Update on the perturbed model :
$\qquad\bm{\theta}_{k+1}\leftarrow\bm{\theta}_{k}-\eta_{2}\nabla\mathcal{J}_{\mathrm{~{}DPAdapter{}},\mathcal{B}_{1},\mathcal{B}_{2}}$
12: Revert perturbation:
$\bm{\theta}_{k+1}\leftarrow\bm{\theta}_{k+1}-\Delta_{\mathcal{B}_{1}}$ 13:
Increment iteration: $k\leftarrow k+1$ 14: end while

Algorithm 1 DPAdapter Training

###  4.5 Theoretical Analyses

In this section, we aim to provide a theoretical explanation of why DPAdapter
achieves improvements in the DPML algorithm. Specifically, our theoretical
examination revolves around two key points:

  1. 1.

Understanding the rationale behind the separate selection of $\mathcal{B}_{1}$
and $\mathcal{B}_{2}$, particularly emphasizing why the batch size of
$\mathcal{B}_{2}$ should be large.

  2. 2.

Elucidating how enhancing the robustness of pretrained parameters can ensure
superior performance in downstream training, given the same privacy budget.

The first point is theoretically proven in Theorem 1, while the second is
validated in Theorem 2.

To give a basic idea, through Theorem 1, we aim to theoretically demonstrate
that, upon selecting an appropriate ${\mathcal{B}_{1}}$ conducive to
optimization and holding it constant, we can bound the difference between the
actual loss and the maximum lower bound (Infimum) of the loss. Essentially, as
the magnitude of ${\mathcal{B}_{2}}$ increases, the upper bound on the
expected convergence rate of the model obtained through DPAdapter also
increases.

In theorem 2, we prove that given a model training by DP-SGD, the expected
difference between the model’s loss and the maximum lower bound of the
achievable loss (termed as DP-SGD’s performance) is positively proportional to
$\rho$. Specifically, a lower value of $\rho$ (indicating stronger parameter
robustness) in the model initial stage, results in a lower loss when optimized
using DP-SGD.

Next, we delineate the setup for our theoretical framework. We first analyze
the convergence behavior of our proposed algorithm. We show that by using a
different update batch (where $|\mathcal{B}_{1}|\neq|\mathcal{B}_{2}|$). We
summarize our algorithm as follows. Let
$L_{i}(\bm{\theta}):=\ell(f_{\bm{\theta}}(\bm{x}_{i}),\bm{y}_{i})$. For any
batch of the data $\mathcal{S}\subseteq\mathcal{D}$, let $L_{\mathcal{S}}$ be
the average of $L_{i},i\in\mathcal{S}$. Then our algorithm is as follows.
Starting from the initial parameter $\bm{\theta}_{0}$, for each iteration $t$,
we have

$\displaystyle\bm{\theta}_{t+1}=\bm{\theta}_{t}-\eta_{2}\cdot\nabla
L_{\mathcal{B}_{2}}(\bm{\theta}_{t}+\Delta_{t}),\
\Delta_{t}:=\eta_{1}\cdot\nabla L_{\mathcal{B}_{1}}(\bm{\theta}_{t}).$ (6)

Equation 6 is essentially the same as Algorithm 1 in Section 4.3 without the
perturbation normalization step (line 6-8 in Algorithm 1). Meanwhile, we need
the following assumptions.

######  Assumption 1.

$f$ is $\beta$-smooth w.r.t. $\bm{\theta}$, i.e., $\|\nabla
f_{\bm{\theta}}(\bm{x})-\nabla
f_{\bm{\theta}^{\prime}}(\bm{x})\|_{2}\leq\beta\|\bm{\theta}-\bm{\theta}^{\prime}\|_{2}$.

######  Assumption 2.

$\ell$ is $\beta_{1}$-Lipschitz continuous w.r.t. $\bm{x}$ i.e.,
$|\ell(\bm{x},\bm{y})-\ell(\bm{x}^{\prime},\bm{y})|\leq\beta_{1}|\bm{x}-\bm{x}^{\prime}|$.

######  Assumption 3.

$\ell$ is $\beta_{2}$-smooth w.r.t. $\bm{x}$, i.e.,
$|\nabla_{\bm{x}}\ell(\bm{x},\bm{y})-\nabla_{\bm{x}}\ell(\bm{x}^{\prime},\bm{y})|\leq\beta_{2}|\bm{x}-\bm{x}^{\prime}|$.

######  Assumption 4.

$\nabla L_{i}$ is $\hat{\sigma}^{2}$-variance bounded gradient, i.e.,
$\mathbb{E}_{i}\|\nabla L_{i}(\bm{\theta})-\nabla
L_{\mathcal{D}}(\bm{\theta})\|_{2}^{2}\leq\hat{\sigma}^{2}$. Here
$\mathbb{E}_{i}$ denotes the expectation over $i\in 1,\dots,n$.

######  Assumption 5.

$L_{\mathcal{D}}$ satisfies the $\mu$-PL-condition, i.e., $\|\nabla
L_{\mathcal{D}}(\bm{\theta})\|_{2}^{2}\geq
1/\mu\cdot(L_{\mathcal{D}}(\bm{\theta})-\inf_{\hat{\bm{\theta}}}L_{\mathcal{D}}(\hat{\bm{\theta}}))$.

We also formally define the parameter robustness in Section 2.3 as follows:

$\rho(f)=\max_{\bm{x},\bm{\theta},\Delta}|f_{\theta+\Delta}(\bm{x})-f_{\theta}(\bm{x})|/\|\Delta\|_{2}.$

For simplicity, we use $\rho:=\rho(f)$ with a slight abuse of notation. Then
we have the following theorem.

######  Theorem 1 (Informal).

With proper selection of parameters, our algorithm enjoys the following
gradient norm bound:

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}(L_{\mathcal{D}}(\bm{\theta}_{t})-\inf_{\hat{\bm{\theta}}}L_{\mathcal{D}}(\hat{\bm{\theta}}))$
$\displaystyle\leq\mu\cdot\bigg{(}\frac{L_{\mathcal{D}}(\bm{\theta}_{0})}{T}+\frac{\hat{\sigma}^{2}}{16\hat{\beta}}(1/|\mathcal{B}_{2}|+1/|\mathcal{B}_{1}|)\bigg{)},$

where $\hat{\beta}=(\rho^{2}\beta_{2}+\beta\beta_{1})$.

The detailed theorem and proof are deferred to Appendix A.1. Theorem 1
suggests that selecting a large perturbation batch size $|\mathcal{B}_{1}|$
makes our algorithm have a better convergence rate, indicating better
performance while using a large batch of perturbations $|\mathcal{B}_{1}|$.

We have another analysis of the noise tolerance and the DP-SGD performance.
Then we have our theorem.

######  Theorem 2 (Informal).

Given $(\epsilon,\delta)$, with proper selection of parameters, let
$\bm{\theta}_{\text{out}}$ be the final output of DP-SGD, then DP-SGD is
$(\epsilon,\delta)$-DP and enjoys the following utility bound:

$\displaystyle\mathbb{E}(L_{\mathcal{D}}(\bm{\theta}_{\text{out}})-\inf_{\bm{\theta}}L_{\mathcal{D}}(\bm{\theta}))\leq
C\cdot\frac{\rho\sqrt{\rho^{2}\beta_{2}+\beta\beta_{1}}}{|\mathcal{D}|\epsilon},$

where
$C=c\cdot\mu\beta_{1}\sqrt{d\log(|\mathcal{D}|/\delta)\log(1/\delta)L_{\mathcal{D}}(\bm{\theta}_{0})}$,
$c$ is some positive constant, $d$ is the dimension of parameter
$\bm{\theta}$.

The detailed algorithm, theorem, and proof are deferred to Appendix A.2.
Theorem 2 suggests that the parameter robustness $\rho$ indeed affects the
final utility.

##  5 Evaluation

###  5.1 Experimental Setup

Datasets. We use the following five datasets in our evaluation.

$\bullet$ CIFAR-10 [33]: This dataset contains $50,000$ training images and
$10,000$ testing images. Each image has a size of $32\times 32\times 3$ and
belongs to one of $10$ classes.

$\bullet$ CIFAR-100 [33]: This dataset contains $50,000$ training images and
$10,000$ testing images. Each image has a size of $32\times 32\times 3$ and
belongs to one of $100$ classes.

$\bullet$ SVHN [45]: In this dataset, each image represents a digit from the
house numbers in Google Street View. The size of each image is $32\times
32\times 3$. Each image belongs to one of the $10$ digits. This dataset has
$73,257$ training images and $26,032$ testing images.

$\bullet$ STL-10 [13]: This dataset contains $5,000$ training images and
$8,000$ testing images. Each image has a size of $96\times 96\times 3$ and
belongs to one of $10$ classes.

In this paper, CIFAR-100 serves as a public dataset used for pre-training,
while CIFAR-10, SVHN, and STL-10 serves as the private data/tasks.

Pre-training Procedure. In our experiments, we utilize CIFAR-100 as our pre-
training dataset and employ DPAdapter to train a ResNet20 model [29] as the
pre-trained model. We compute the mean and standard deviation on the training
set to normalize the input images. We adopt cross-entropy as the loss function
$\ell$ and utilize the SGD optimizer with momentum, incorporating a step-wise
learning rate decay. Specifically, the model is trained for $K=1,000$ epochs,
with the outer learning rate $\eta_{2}$ initialized at $0.1$ and divided by
$10$ after $500$ and $750$ epochs. The momentum is set to $0.9$, and the
weight decay is set to $1\text{\times}{10}^{-4}$. The inner batch size $m_{1}$
is set to $5,000$, with the inner learning rate $\eta_{1}$ set to $1.0$; the
outer batch size $m_{2}$ is set to $50$. For perturbation magnitude, we adopt
$\gamma=2.0$ by default.

Fine-tuning Procedure. Given a pre-trained model, we utilize it to train
private downstream classifiers for the remaining three datasets: CIFAR-10,
SVHN, and STL-10. We employ the parameters of the pre-trained model as the
initial parameters of the downstream classifier, which is then trained on the
downstream dataset.

We use Rényi DP to accumulate the overall privacy budget and precompute the
required noise scale ($\sigma$ in DP-SGD) numerically [1, 44]. We maintain
$\delta=10^{-5}$ and utilize different privacy budgets: $\epsilon=\\{1,4\\}$.
The clipping threshold for all algorithms is fixed at 4, except when an
algorithm employs special clipping strategies. The cross-entropy loss function
and the DP-SGD optimizer with momentum are adopted when training a downstream
classifier. The model is fine-tuned for $100$ epochs, with the learning rate
initialized at $0.01$ and momentum set to $0.9$.

DPML Algorithms. According to the different optimization methods that the
downstream model could adopt, we consider three types of DP algorithms in
addition to vanilla DP-SGD:

$\bullet$ GEP [64]: Yu et al. observed that in vanilla DP-SGD, the amount of
noise increases with the model size and proposed a solution, GEP [64], to
reduce the gradient dimension before adding noise. GEP first calculates an
anchor subspace, which contains some gradients of public data, using the power
method. Subsequently, it projects the gradient of private data into the anchor
subspace, resulting in a low-dimensional gradient embedding and a small-norm
residual gradient. These two components are independently processed with the
DP mechanism and then combined to update the original weight.

$\bullet$ AdpAlloc [65]: It proposes a dynamic noise-adding mechanism,
eschewing the practice of maintaining a constant noise multiplier $\sigma$
throughout every training epoch in vanilla DP-SGD. Instead of utilizing a
static variance in the Gaussian mechanism, it replaces it with a function of
the epoch:
$\mathit{M}(d)=\mathit{f}(d)+\mathbf{N}(0,S_{\mathit{f}}^{2}\cdot\sigma_{t}^{2})$,
where the value of $\sigma_{t}$ is contingent upon the final privacy budget,
epoch, and schedule function. The schedule function delineates the adjustment
of the noise scale throughout training. Yu et al. proposed several pre-defined
schedules. For our evaluation, we select Exponential Decay, which demonstrated
the best average performance in [65]. The mathematical expression for
Exponential Decay is $\sigma_{t}=\sigma_{0}e^{-kt}$, where $k(k>0)$ represents
the decay rate and $\sigma_{0}$ denotes the initial noise scale.

$\bullet$ AdpClip [2]: An adaptive clipping threshold mechanism is utilized,
setting the clip threshold to a specified quantile of the update norm
distribution at each epoch. Formally, the clipping threshold $C_{t}$ in epoch
$t$ can be computed as
$C_{t}=C_{t-1}\cdot\exp\left(-\eta_{C}(\overline{b}-\gamma)\right)$, where
$\gamma\in\left[0,1\right]$ is a quantile to be matched,
$\overline{b}\triangleq\frac{1}{m}\sum_{i\in[m]}\mathbb{I}_{x_{i}\leq C}$
represents the empirical fraction of samples with a value at most $C$, and
$\eta_{C}$ is the learning rate with a default value of 0.2, as indicated in
[2]. To circumvent the issue of $\overline{b}$ revealing private information,
the Gaussian mechanism is applied to $\overline{b}$:
$\tilde{b}^{t}=\frac{1}{m}\left(\sum_{i\in\mathit{Q}^{t}}b_{i}^{t}+\mathit{N}\left(O,\sigma_{b}^{2}\right)\right)$.
This method expends a negligible privacy budget while closely tracking the
quantile. While AdpClip was initially designed for federated learning [42], it
can also be used in traditional centralized learning scenarios.

Table 1: Comparison of the performance of training from scratch, standard pre-training, vanilla SAM, and DPAdapter across various downstream tasks under different privacy budgets. DPML Algorithms |  Upstream Training Method | CIFAR10 | SVHN | STL10 | Average
---|---|---|---|---|---
$\epsilon=1$ | $\epsilon=4$ | $\epsilon=1$ | $\epsilon=4$ | $\epsilon=1$ | $\epsilon=4$ | $\epsilon=1$ | $\epsilon=4$
DP-SGD |  None (Scratch) [1] | 0.4288 | 0.5070 | 0.7194 | 0.8380 | 0.2831 | 0.3370 | 0.4771 | 0.5607
Standard Pre-training [34] | 0.5928 | 0.7210 | 0.7822 | 0.8970 | 0.3282 | 0.5695 | 0.5677 | 0.7292
Vanilla SAM (Ours) | 0.6216 | 0.7650 | 0.8042 | 0.9014 | 0.3625 | 0.6212 | 0.5961 | 0.7625
DPAdapter (Ours) | 0.6416 | 0.7746 | 0.8058 | 0.9018 | 0.3951 | 0.6364 | 0.6142 | 0.7709
AdpClip |  None (Scratch) [2] | 0.3738 | 0.5348 | 0.6258 | 0.8294 | 0.2270 | 0.3543 | 0.4089 | 0.5728
Standard Pre-training [34] | 0.4198 | 0.6780 | 0.6196 | 0.8914 | 0.2406 | 0.4890 | 0.4267 | 0.6861
Vanilla SAM (Ours) | 0.5962 | 0.7108 | 0.6672 | 0.8962 | 0.2539 | 0.5446 | 0.5058 | 0.7172
DPAdapter (Ours) | 0.6008 | 0.7186 | 0.6730 | 0.9012 | 0.3023 | 0.5676 | 0.5254 | 0.7291
GEP |  None (Scratch) [64] | 0.4008 | 0.4672 | 0.5892 | 0.8158 | 0.2360 | 0.3239 | 0.4087 | 0.5356
Standard Pre-training [34] | 0.6512 | 0.7456 | 0.8078 | 0.8500 | 0.3326 | 0.6002 | 0.5972 | 0.7319
Vanilla SAM (Ours) | 0.6808 | 0.7472 | 0.8132 | 0.8538 | 0.3739 | 0.6168 | 0.6226 | 0.7393
DPAdapter (Ours) | 0.6890 | 0.7692 | 0.8180 | 0.8686 | 0.4730 | 0.6462 | 0.6600 | 0.7613
AdpAlloc |  None (Scratch) [65] | 0.4370 | 0.5166 | 0.6248 | 0.7678 | 0.2923 | 0.3391 | 0.4514 | 0.5412
Standard Pre-training [34] | 0.4506 | 0.6982 | 0.6604 | 0.8938 | 0.2946 | 0.5104 | 0.4685 | 0.7008
Vanilla SAM (Ours) | 0.5296 | 0.7372 | 0.7652 | 0.8998 | 0.2933 | 0.5675 | 0.5294 | 0.7348
DPAdapter (Ours) | 0.5352 | 0.7406 | 0.7862 | 0.9008 | 0.2938 | 0.6111 | 0.5384 | 0.7508

###  5.2 Effectiveness of DPAdapter

In this section, we empirically validate the overall effectiveness of
DPAdapter and conduct ablation studies to illustrate the effectiveness of each
component.

Setup. We conduct experiments using four different DPML algorithms across
three distinct private downstream tasks (CIFAR-10, SVHN, and STL-10), under
four different pre-training settings: (i) From Scratch: utilizing randomly
initialized weight values (i.e., no pre-training); (ii) Standard Pre-training:
conducting pre-training on the CIFAR-100 dataset using standard training
procedures; (iii) Vanilla Sharpness-Aware Minimization (SAM): pre-training
with AMP [70], employing the optimal perturbation magnitude $\gamma=1.0$ and
the optimal batch size $m=100$; (iv) DPAdapter: pre-training with the proposed
DPAdapter, using hyper-parameters described in Section 5.1.

Observations. Table 1 illustrates the downstream accuracy for various
settings. In general, we observe that the proposed DPAdapter substantially
enhances the downstream accuracy in all settings. For instance, when the
privacy budget is set to $\epsilon=1$ and the DPML algorithm is configured to
vanilla DP-SGD, DPAdapter elevates the average accuracy across three
downstream datasets to 61.42%. In contrast, when utilizing models pre-trained
normally, the accuracy is only 56.95%, marking an improvement of over 4%. When
the DPML algorithm is AdpClip, DPAdapter boosts the accuracy from 42.67% to
52.54%, representing a near 10% improvement.

We note that, compared to training from scratch, a normally pre-trained model
consistently yields an improvement in downstream accuracy. This finding aligns
with previous work which noted that transfer learning from public data
significantly enhances the accuracy of DPML algorithms [34]. Further, we note
that employing vanilla SAM to provide parameter robustness can additionally
enhance downstream accuracy. For example, when the privacy budget is
designated as $\epsilon=1$ and the DPML algorithm is configured to AdpAlloc,
using a standard pre-trained model improves the average downstream accuracy by
1.71%. Meanwhile, applying vanilla SAM can further augment the average
downstream accuracy by 6.09% (compared with standard pre-training), achieving
an additional gap that is over three times larger than the previous gap
achieved by standard pre-training. This underscores the significance of
leveraging parameter robustness to enhance DPML algorithms.

Moreover, we observe that employing DPAdapter consistently achieves higher
downstream accuracy compared to using vanilla SAM. For instance, when the
privacy budget is set to $\epsilon=4$ and the DPML algorithm is configured to
GEP, applying vanilla SAM enhances the average downstream accuracy by 0.74%
compared with standard pre-training. Meanwhile, applying DPAdapter can further
increase the average downstream accuracy by 2.20% (compared with vanilla SAM).
When the privacy budget is $\epsilon=1$, applying vanilla SAM improves the
average downstream accuracy by 2.54%; DPAdapter, in contrast, achieves an
improvement of 6.28% compared with standard pre-training. This emphasizes the
importance of leveraging decoupled batches to further enhance parameter
robustness.

###  5.3 Impact of Perturbation Magnitude

Recall that DPAdapter fundamentally accumulates parameter robustness by
introducing adversarial perturbation during the training process, the
magnitude of which is determined by the $\gamma$ term. To comprehend how
DPAdapter influences the performance of downstream tasks, we assess DPAdapter
utilizing various $\gamma$ settings. In this experiment, we fix the privacy
budget at $\epsilon=1$ and the DPML algorithm is configured to vanilla DP-SGD.

Observations. Figure 5 illustrates the fluctuations in upstream performance,
upstream parameter robustness, and the performance of various downstream tasks
when applying DPAdapter with different $\gamma$ values, representing the
perturbation magnitude. We observe that the downstream accuracy trend largely
aligns with the trend of upstream parameter robustness, while demonstrating a
nearly inverse relationship with the level of upstream accuracy. This
observation suggests that parameter robustness is the principal factor through
which DPAdapter enhances downstream accuracy. Given that the trends of
upstream accuracy and downstream accuracy exhibit nearly inverse patterns with
changes in $\gamma$, we can essentially rule out the possibility that upstream
accuracy is the predominant contributor to downstream accuracy.

Note that when $\gamma$ is further increased to $5.0$, the upstream accuracy
experiences a slight increase while the performance across different
downstream tasks all decline. This could be attributed to the scenario where
the enhancement in parameter robustness becomes constrained and the decline in
upstream accuracy becomes significant. The advantages conferred by improved
parameter robustness are unable to offset the disadvantages brought about by
the reduction in upstream accuracy, since upstream accuracy also contributes
to downstream accuracy through enhanced generalization ability. This
underscores the vital necessity of maintaining a balance between parameter
robustness and generalization ability.

In this experiment, for the first time, we identify and establish a connection
between parameter robustness and the performance of DPML algorithms, marking
one of the key contributions of this paper.

Figure 5: Impact of perturbation magnitude on DPAdapter.

Figure 6: The results of DPAdapter and vanilla SAM under different
perturbation magnitudes. The results of the vanilla SAM are represented by
blue triangles, while the results of the proposed DPAdapter are depicted by
red diamonds. Results obtained using a vanilla pre-trained model are
illustrated by a purple rectangle. All the downstream models have the same
privacy budget $\epsilon=1$.

###  5.4 Comparison with Vanilla SAM

Setup. In this experiment, we consider four perturbation magnitudes: 0.1, 0.5,
1.0, and 2.0. The DPML algorithm is fixed at vanilla DP-SGD and the privacy
budget is fixed at $\epsilon=1$. The results of using the vanilla SAM are
denoted by blue triangles, while the results of the proposed DPAdapter are
denoted by red diamonds. The result of using a normally pre-trained model is
denoted as a purple rectangle.

Observations. Figure 6 illustrate the influence of perturbation magnitude on
both vanilla SAM and DPAdapter. The x-axis represents the robust accuracy,
which measures parameter robustness, while the y-axis indicates the accuracy,
reflecting model performance. We observe that models using standard pre-
training typically appear in the bottom-left region, demonstrating a weak
robustness and generalization trade-off, identified as crucial for DPML
algorithms in Section 5.3. Conversely, the results derived from the proposed
DPAdapter predominantly reside in the top-right region, showcasing an
enhancement over the vanilla SAM. This implies that DPAdapter can further
refine the robustness-generalization trade-off compared to vanilla SAM, which
may elucidate its consistent ability to amplify the performance of DPML
algorithms with the most substantial improvements. Additional results,
obtained when the privacy budget is set to $\epsilon=4$, are provided in
Appendix B.

##  6 Discussion

The current design of DPAdapter is specifically targeted towards supervised
learning and cannot be directly applied to unsupervised learning techniques
like contrastive learning [28, 11, 12]. Thus, the attacker in our scenario
needs to have a labeled dataset for pre-training, which might pose a challenge
in certain specialized fields. However, the central idea of leveraging
parameter robustness to enhance DPML algorithms is a broad concept and not
limited to supervised learning. Yet, the question of how to actively induce
accumulate parameter robustness in the context of contrastive learning remains
open and presents an exciting direction for future research.

Moreover, while our discussion and implementation center on a single-party
scenario, it is worth noting that our framework is adaptable to both single-
party and multi-party situations. In a multi-party context, the private data
originates from multiple sources. Consequently, private fine-tuning processes
can be facilitated through federated learning. In these instances, the DP-SGD
algorithm would need to be supplanted by the DP-FedAvg algorithm [42] to
integrate DP noise during aggregation. Although DPAdapter provides a promising
foundation, the adaptation, execution, and assessment of the methodology in
multi-party contexts necessitate further exploration and rigorous validation.
Future research endeavors will delve deeper into the complexities and nuances
of multi-party scenarios, evaluating the adaptability, scalability, and
efficiency of DPAdapter in these environments.

##  7 Related Work

Differential Privacy. Differential privacy (DP)[19, 22] is a widely-adopted,
rigorous mathematical framework that formalizes and measures privacy
guarantees, primarily using a parameter termed the privacy budget. It has been
employed in various data analysis tasks, such as synthetic dataset generation
[66, 69, 56, 18], marginal release [68], range query [17], and stream data
analysis [57]. Abadi et al. introduced vanilla DP-SGD [1] as the pioneering
general DP algorithm for deep learning. Subsequent research efforts have
focused on minimizing the impact of DP on model utility, either by devising
new algorithms [47, 54, 63, 71] or by relaxing DP definition to suit specific
contexts [40, 20, 24]. In contrast, our proposed method, DPAdapter, seeks to
minimize the impact of DP on model utility by providing a parameter-robust
pre-trained model as a service, an approach that has not been explored
previously.

Parameter Robustness. Numerous studies have demonstrated that enhancing
parameter robustness effectively reduces the generalization gap, thereby
improving model generality [70, 35]. Additional research indicates that
improvements in parameter robustness can also mitigate robust overfitting
during adversarial training, achieving heightened generality with respect to
model robustness against input perturbations [59, 61, 55, 52]. However, all of
these studies utilize the additional benefits conferred by parameter
robustness, i.e., generality, without directly leveraging the property of
parameter robustness itself. In contrast, the proposed DPAdapter is among the
first strategies that directly utilize parameter robustness to benefit private
learning. This introduces a fresh perspective on the potential applications of
the parameter robustness property.

Pre-trained Model as a Service. Adversarial training [26, 41] is recognized as
a standard method for developing empirically robust classifiers in supervised
learning. The core concept involves generating adversarial examples from
training instances during the training process and augmenting the training
data with these examples. Several studies [10, 31, 32, 50] have generalized
adversarial training for the pre-training of robust models in self-supervised
learning. Generally, the approach first generates adversarial examples that
result in significant loss, which are then used for model pre-training.
However, previous studies have focused solely on offering input robustness as
a service, neglecting to offer parameter robustness, which is the primary
contribution of this paper.

Attacks and Defenses in Transfer Learning. Research has also delved into
membership inference attacks within transfer learning scenarios. Liu et
al.[38] demonstrated a membership inference attack against pre-trained
encoders, elucidating the capability to infer whether an input was part of the
encoder’s training dataset. Concurrently, He et al.[30] introduced a
mitigation strategy capable of countering such threats. However, our work
pivots the focus towards the privacy risks associated with downstream tasks,
rather than those tied to pre-trained encoders. DPAdapter represents the first
attempt to offer a pre-training method aimed at enhancing defenses against
downstream attacks.

##  8 Conclusion

In this study, we unveiled DPAdapter, a groundbreaking technique engineered to
augment parameter robustness, thereby navigating the traditionally adversarial
relationship between Differential Privacy (DP) noise and model utility in Deep
Learning. By meticulously reallocating batch sizes for perturbation and
gradient calculations, DPAdapter refines Sharpness-Aware Minimization (SAM)
algorithms, delivering enhanced parameter robustness and, consequently,
mitigating the impact of DP noise. Our comprehensive evaluations across
several datasets substantiate DPAdapter’s capability to substantially bolster
the accuracy of DPML algorithms on various downstream tasks, thereby
highlighting its potential as a pivotal technique in future privacy-preserving
machine learning endeavors.

## References

  * [1] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308–318, 2016.
  * [2] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy.  Differentially private learning with adaptive clipping.  Advances in Neural Information Processing Systems, 34:17455–17466, 2021.
  * [3] Maksym Andriushchenko and Nicolas Flammarion.  Towards understanding sharpness-aware minimization.  In International Conference on Machine Learning, pages 639–668. PMLR, 2022.
  * [4] Anish Athalye, Nicholas Carlini, and David A. Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 274–283, 2018.
  * [5] Borja Balle, Giovanni Cherubin, and Jamie Hayes.  Reconstructing training data with informed adversaries.  In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pages 1138–1156, 2022.
  * [6] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III, pages 387–402, 2013.
  * [7] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr.  Membership inference attacks from first principles.  In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pages 1897–1914, 2022.
  * [8] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel.  Extracting training data from large language models.  In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pages 2633–2650, 2021.
  * [9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pages 1691–1703, 2020.
  * [10] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang.  Adversarial robustness: From self-supervised pre-training to fine-tuning.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 696–705, 2020.
  * [11] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  CoRR, abs/2003.04297, 2020.
  * [12] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9620–9629, 2021.
  * [13] Adam Coates, Andrew Y. Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, pages 215–223, 2011.
  * [14] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa.  Natural language processing (almost) from scratch.  J. Mach. Learn. Res., 12:2493–2537, 2011.
  * [15] Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle.  Unlocking high-accuracy differentially private image classification through scale.  CoRR, abs/2204.13650, 2022.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
  * [17] Linkang Du, Zhikun Zhang, Shaojie Bai, Changchang Liu, Shouling Ji, Peng Cheng, and Jiming Chen.  AHEAD: adaptive hierarchical decomposition for range query under local differential privacy.  In CCS ’21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 1266–1288, 2021.
  * [18] Yuntao Du, Yujia Hu, Zhikun Zhang, Ziquan Fang, Lu Chen, Baihua Zheng, and Yunjun Gao.  Ldptrace: Locally differentially private trajectory synthesis.  Proc. VLDB Endow., 16(8):1897–1909, 2023.
  * [19] Cynthia Dwork.  Differential privacy: A survey of results.  In Theory and Applications of Models of Computation, 5th International Conference, TAMC 2008, Xi’an, China, April 25-29, 2008. Proceedings, pages 1–19, 2008.
  * [20] Cynthia Dwork.  Differential privacy in new settings.  In Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2010, Austin, Texas, USA, January 17-19, 2010, pages 174–183, 2010.
  * [21] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith.  Calibrating noise to sensitivity in private data analysis.  In Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, pages 265–284, 2006.
  * [22] Cynthia Dwork and Aaron Roth.  The algorithmic foundations of differential privacy.  Found. Trends Theor. Comput. Sci., 9(3-4):211–407, 2014.
  * [23] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  arXiv preprint arXiv:2010.01412, 2020.
  * [24] Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang.  Deep learning with label differential privacy.  In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 27131–27145, 2021.
  * [25] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [26] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [27] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch SGD: training imagenet in 1 hour.  CoRR, abs/1706.02677, 2017.
  * [28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.  Momentum contrast for unsupervised visual representation learning.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726–9735, 2020.
  * [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778, 2016.
  * [30] Xinlei He and Yang Zhang.  Quantifying and mitigating privacy risks of contrastive learning.  In CCS ’21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 845–863, 2021.
  * [31] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang.  Robust pre-training by adversarial contrastive learning.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [32] Minseon Kim, Jihoon Tack, and Sung Ju Hwang.  Adversarial self-supervised contrastive learning.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [33] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Tech. report, University of Toronto, 2009.
  * [34] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta.  Toward training at imagenet scale with differential privacy.  CoRR, abs/2201.12328, 2022.
  * [35] Tao Li, Weihao Yan, Zehao Lei, Yingwen Wu, Kun Fang, Ming Yang, and Xiaolin Huang.  Efficient generalization improvement guided by random weight perturbation.  CoRR, abs/2211.11489, 2022.
  * [36] Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto.  Large language models can be strong differentially private learners.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
  * [37] Zheng Li and Yang Zhang.  Membership leakage in label-only exposures.  In CCS ’21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 880–895, 2021.
  * [38] Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong.  Encodermi: Membership inference against pre-trained encoders in contrastive learning.  In CCS ’21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 2081–2095, 2021.
  * [39] Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang.  Ml-doctor: Holistic risk assessment of inference attacks against machine learning models.  In 31st USENIX Security Symposium, USENIX Security 2022, Boston, MA, USA, August 10-12, 2022, pages 4525–4542, 2022.
  * [40] Ashwin Machanavajjhala, Daniel Kifer, John M. Abowd, Johannes Gehrke, and Lars Vilhuber.  Privacy: Theory meets practice on the map.  In Proceedings of the 24th International Conference on Data Engineering, ICDE 2008, April 7-12, 2008, Cancún, Mexico, pages 277–286, 2008.
  * [41] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
  * [42] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, pages 1273–1282, 2017.
  * [43] Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky.  Large scale transfer learning for differentially private image classification.  CoRR, abs/2205.02973, 2022.
  * [44] Ilya Mironov, Kunal Talwar, and Li Zhang.  Rényi differential privacy of the sampled gaussian mechanism.  CoRR, abs/1908.10530, 2019.
  * [45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In Advances in Neural Information Processing Systems (NIPS), pages 2843–2851, 2011.
  * [46] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Trans. Knowl. Data Eng., 22(10):1345–1359, 2010.
  * [47] Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar.  Semi-supervised knowledge transfer for deep learning from private training data.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
  * [48] Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In IEEE European Symposium on Security and Privacy, EuroS&P 2016, Saarbrücken, Germany, March 21-24, 2016, pages 372–387, 2016.
  * [49] Wenjie Qu, Jinyuan Jia, and Neil Zhenqiang Gong.  Reaas: Enabling adversarially robust downstream classifiers via robust encoder as a service.  arXiv preprint arXiv:2301.02905, 2023.
  * [50] Wenjie Qu, Jinyuan Jia, and Neil Zhenqiang Gong.  Reaas: Enabling adversarially robust downstream classifiers via robust encoder as a service.  In 30th Annual Network and Distributed System Security Symposium, NDSS 2023, San Diego, California, USA, February 27 - March 3, 2023, 2023.
  * [51] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91–99, 2015.
  * [52] David Stutz, Matthias Hein, and Bernt Schiele.  Relating adversarially robust generalization to flat minima.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 7787–7797, 2021.
  * [53] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [54] Florian Tramèr and Dan Boneh.  Differentially private learning needs better features (or much more data).  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
  * [55] Yu-Lin Tsai, Chia-Yi Hsu, Chia-Mu Yu, and Pin-Yu Chen.  Formalizing generalization and adversarial robustness of neural networks to weight perturbations.  In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 19692–19704, 2021.
  * [56] Haiming Wang, Zhikun Zhang, Tianhao Wang, Shibo He, Michael Backes, Jiming Chen, and Yang Zhang.  Privtrace: Differentially private trajectory synthesis by adaptive markov models.  In 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, 2023.
  * [57] Tianhao Wang, Joann Qiongna Chen, Zhikun Zhang, Dong Su, Yueqiang Cheng, Zhou Li, Ninghui Li, and Somesh Jha.  Continuous release of data streams under both centralized and local differential privacy.  In CCS ’21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 1237–1253, 2021.
  * [58] Chengkun Wei, Minghu Zhao, Zhikun Zhang, Min Chen, Wenlong Meng, Bo Liu, Yuan Fan, and Wenzhi Chen.  Dpmlbench: Holistic evaluation of differentially private machine learning.  CoRR, abs/2305.05900, 2023.
  * [59] Dongxian Wu, Shu-Tao Xia, and Yisen Wang.  Adversarial weight perturbation helps robust generalization.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [60] Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu.  An information-theoretic analysis for transfer learning: Error bounds and applications.  arXiv preprint arXiv:2207.05377, 2022.
  * [61] Chaojian Yu, Bo Han, Mingming Gong, Li Shen, Shiming Ge, Du Bo, and Tongliang Liu.  Robust weight perturbation for adversarial training.  In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 3688–3694, 2022.
  * [62] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang.  Differentially private fine-tuning of language models.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
  * [63] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu.  Do not let privacy overbill utility: Gradient embedding perturbation for private learning.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
  * [64] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu.  Do not let privacy overbill utility: Gradient embedding perturbation for private learning.  In International Conference on Learning Representations (ICLR), 2021.
  * [65] Lei Yu, Ling Liu, Calton Pu, Mehmet Emre Gursoy, and Stacey Truex.  Differentially private model publishing for deep learning.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 332–349. IEEE, 2019.
  * [66] Quan Yuan, Zhikun Zhang, Linkang Du, Min Chen, Peng Cheng, and Mingyang Sun.  Privgraph: Differentially private graph data publication by exploiting community information.  In 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, 2023.
  * [67] Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang.  Efficient private erm for smooth objectives.  In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3922–3928, 2017.
  * [68] Zhikun Zhang, Tianhao Wang, Ninghui Li, Shibo He, and Jiming Chen.  CALM: consistent adaptive local marginal for marginal release under local differential privacy.  In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018, pages 212–229, 2018.
  * [69] Zhikun Zhang, Tianhao Wang, Ninghui Li, Jean Honorio, Michael Backes, Shibo He, Jiming Chen, and Yang Zhang.  Privsyn: Differentially private data synthesis.  In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pages 929–946, 2021.
  * [70] Yaowei Zheng, Richong Zhang, and Yongyi Mao.  Regularizing neural networks via adversarial model perturbation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 8156–8165, 2021.
  * [71] Yuqing Zhu, Xiang Yu, Manmohan Chandraker, and Yu-Xiang Wang.  Private-knn: Practical differential privacy for computer vision.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 11851–11859, 2020.

## Appendix

##  Appendix A Theoretical Results

###  A.1 Proof of Theorem 1

We first restate Theorem Theorem 1 as follows.

######  Theorem 3 (Formal version of Theorem 1).

Denote $\hat{\beta}:=(\rho^{2}\beta_{2}+\beta\beta_{1})$. Select step sizes
$\eta_{1}=\eta_{2}=1/(4\hat{\beta})$. Then Algorithm 1 enjoys the following
utility bound:

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}(L_{\mathcal{D}}(\bm{\theta}_{t})-\inf_{\hat{\bm{\theta}}}L_{\mathcal{D}}(\hat{\bm{\theta}}))\leq\mu\cdot\bigg{(}\frac{L_{\mathcal{D}}(\bm{\theta}_{0})}{T}+\frac{\hat{\sigma}^{2}}{16\hat{\beta}}(1/|\mathcal{B}_{2}|+1/|\mathcal{B}_{1}|)\bigg{)}.$

To prove Theorem 3, we need the following lemma which estimates the Lipschitz
constant and smoothness constant of $L_{i}$.

######  Lemma A.1.

For each $i$, $L_{i}$ is $\beta_{1}\rho$-Lipschitz continuous and
$(\rho^{2}\beta_{2}+\beta\beta_{1})$-smooth.

###### Proof.

We have for all $\bm{\theta},\bm{\theta}^{\prime}$,

$\displaystyle|L_{i}(\bm{\theta})-L_{i}(\bm{\theta}^{\prime})|$
$\displaystyle=|\ell(f_{\bm{\theta}}(x_{i}),y_{i})-\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})|$
$\displaystyle\leq\beta_{1}|f_{\bm{\theta}}(x_{i})-f_{\bm{\theta}^{\prime}}(x_{i})|$
$\displaystyle\leq\beta_{1}\rho\|\bm{\theta}-\bm{\theta}^{\prime}\|_{2},$

and

$\displaystyle\|\nabla L_{i}(\bm{\theta})-\nabla
L_{i}(\bm{\theta}^{\prime})\|_{2}$
$\displaystyle=\|\nabla_{x}\ell(f_{\bm{\theta}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}}(x_{i})-\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}^{\prime}}(x_{i})\|_{2}$
$\displaystyle\leq\|\nabla_{x}\ell(f_{\bm{\theta}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}}(x_{i})-\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}}(x_{i})\|_{2}+\|\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}}(x_{i})-\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})\nabla
f_{\bm{\theta}^{\prime}}(x_{i})\|_{2}$ $\displaystyle\leq\|\nabla
f_{\bm{\theta}}(x_{i})\|_{2}|\nabla_{x}\ell(f_{\bm{\theta}}(x_{i}),y_{i})-\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})|+|\nabla_{x}\ell(f_{\bm{\theta}^{\prime}}(x_{i}),y_{i})|\|\nabla
f_{\bm{\theta}}(x_{i})-\nabla f_{\bm{\theta}^{\prime}}(x_{i})\|_{2}$
$\displaystyle\leq\rho^{2}\beta_{2}\|\bm{\theta}-\bm{\theta}^{\prime}\|_{2}+\beta_{1}\beta\|\bm{\theta}-\bm{\theta}^{\prime}\|_{2}.$

Therefore, $L_{i}$ is $\beta_{1}\rho$-Lipschitz continuous and
$(\rho^{2}\beta_{2}+\beta\beta_{1})$-smooth. ∎

We now begin to prove Theorem 3.

###### Proof of Theorem 3.

The proof is adapted from [3]. First, by Lemma A.1 we know that $L_{i}$ is
$\beta_{1}\rho$-Lipschitz continuous and
$(\rho^{2}\beta_{2}+\beta\beta_{1})$-smooth. By Lemma 13 in [3], we have

$\displaystyle\mathbb{E}\langle\nabla
L_{\mathcal{B}_{2}}(\bm{\theta}_{t}+\eta_{2}\cdot\nabla
L_{\mathcal{B}_{1}}(\bm{\theta}_{t})),\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\rangle\geq(1-\hat{\beta}\eta_{2})\|L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\frac{\hat{\beta}^{2}\eta_{2}^{2}\hat{\sigma}^{2}}{2|\mathcal{B}_{1}|}.$
(7)

By Lemma 14 in [3], denote
$\bm{\theta}_{t+1/2}:=\bm{\theta}_{t}+\eta_{2}\cdot\nabla
L_{\mathcal{B}_{1}}(\bm{\theta}_{t})$, then we have for all $\eta_{1}\leq
1/(2\hat{\beta})$ and $\eta_{2}\leq 1/(2\hat{\beta})$,

$\displaystyle\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t+1})$
$\displaystyle\leq\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t})+\eta_{1}^{2}\hat{\beta}\hat{\sigma}^{2}/|\mathcal{B}_{2}|-\eta_{1}^{2}\hat{\beta}\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\eta_{1}(1-2\eta_{1}\hat{\beta})\mathbb{E}\langle\nabla
L_{\mathcal{D}}(\bm{\theta}_{t+1/2}),\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\rangle$
$\displaystyle\quad+\eta_{1}^{2}\hat{\beta}\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t+1/2})-\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}$
$\displaystyle\leq\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t})+\eta_{1}^{2}\hat{\beta}\hat{\sigma}^{2}/|\mathcal{B}_{2}|-\eta_{1}^{2}\hat{\beta}\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\eta_{1}(1-2\eta_{1}\hat{\beta})\mathbb{E}\langle\nabla
L_{\mathcal{D}}(\bm{\theta}_{t+1/2}),\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\rangle$
$\displaystyle\quad+\eta_{1}^{2}\hat{\beta}\mathbb{E}\|\bm{\theta}_{t+1/2}-\bm{\theta}_{t}\|_{2}^{2}$
$\displaystyle=\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t})+\eta_{1}^{2}\hat{\beta}\hat{\sigma}^{2}/|\mathcal{B}_{2}|-\eta_{1}^{2}\hat{\beta}\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\eta_{1}(1-2\eta_{1}\hat{\beta})\mathbb{E}\langle\nabla
L_{\mathcal{D}}(\bm{\theta}_{t+1/2}),\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\rangle$
$\displaystyle\quad+\eta_{1}^{2}\hat{\beta}^{3}\eta_{2}^{2}\mathbb{E}\|\nabla
L_{\mathcal{B}_{1}}(\bm{\theta}_{t})\|_{2}^{2}$
$\displaystyle\leq\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t})+\eta_{1}^{2}\hat{\beta}\hat{\sigma}^{2}/|\mathcal{B}_{2}|-\eta_{1}^{2}\hat{\beta}(1-2\hat{\beta}^{2}\eta_{2}^{2})\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\eta_{1}(1-2\eta_{1}\hat{\beta})\mathbb{E}\langle\nabla
L_{\mathcal{D}}(\bm{\theta}_{t+1/2}),\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\rangle$
$\displaystyle\quad+2\eta_{1}^{2}\hat{\beta}^{3}\eta_{2}^{2}\hat{\sigma}^{2}/|\mathcal{B}_{1}|.$

Then substituting Equation 7 and setting $\eta_{1}=\eta_{2}=1/(4\hat{\beta})$,
we have

$\displaystyle\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t+1})-\mathbb{E}L_{\mathcal{D}}(\bm{\theta}_{t})$
$\displaystyle\leq\hat{\sigma}^{2}/(16|\mathcal{B}_{2}|\hat{\beta})-1/(16\hat{\beta})\cdot
7/8\cdot\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-1/(8\hat{\beta})\cdot(3/4\cdot\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}-\hat{\sigma}^{2}/(32|\mathcal{B}_{1}|))+\hat{\sigma}^{2}/(128\hat{\beta}|\mathcal{B}_{1}|)$
$\displaystyle\leq\hat{\sigma}^{2}/(64\hat{\beta}|\mathcal{B}_{1}|)+\hat{\sigma}^{2}/(16\hat{\beta}|\mathcal{B}_{2}|)-1/(8\hat{\beta})\cdot\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}.$

Therefore, we have

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}\leq\frac{L_{\mathcal{D}}(\bm{\theta}_{0})}{T}+\frac{\hat{\sigma}^{2}}{16\hat{\beta}}(1/|\mathcal{B}_{2}|+1/|\mathcal{B}_{1}|).$
(8)

Finally, by using the PL-condition we have $\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{t})\|_{2}^{2}\geq
1/\mu\cdot(L_{\mathcal{D}}(\bm{\theta}_{t})-\inf_{\bm{\theta}}L_{\mathcal{D}}(\bm{\theta}))$.
Substituting it into (8) concludes the proof. ∎

###  A.2 Proof of Theorem 2

0: Training set $\mathcal{D}=\\{(\bm{x},\bm{y})\\}$, loss function
$L_{i}:=\ell(f_{\bm{\theta}}(\bm{x}_{i}),\bm{y}_{i})$, privacy parameter
$(\epsilon,\delta)$, learning rate $\eta$.  1: Randomly draw
$R\sim\mathbb{P}$, where
$\displaystyle\mathbb{P}(R=k+1):=\frac{1}{|\mathcal{D}|^{2}},k=0,1,\dots,|\mathcal{D}|^{2}-1.$
2: for $t=0,\dots,R-1$ do 3: Uniformly randomly draw $i$ from
$1,\dots,|\mathcal{D}|$ 4: Draw $z_{t}\in\mathbb{R}^{d}$ from the following
Gaussian distribution:  $\displaystyle
z^{t}\sim\mathcal{N}(0,4\beta_{1}^{2}\rho^{2}\log(3|\mathcal{D}|/\delta)\log(2/\delta)\cdot\mathbf{I})$
(9) 5: Update $\bm{\theta}_{t+1}:=\bm{\theta}_{t}-\eta\cdot(\nabla
L_{i}(\bm{\theta}_{t})+z_{t})$ 6: end for 6:
$\bm{\theta}_{\text{out}}=\bm{\theta}_{R}$

Algorithm 2 Random round DP-SGD

We present the random round DP-SGD proposed in [67] as Algorithm 2. Compared
with the standard DP-SGD [1], the main difference is the random selected round
number $R$ here. The use of a random round number is only due to the need of
theoretical proof. Next, we state the formal version of Theorem Theorem 2 as
follows.

######  Theorem 4 (Formal version of Theorem 2).

Select the step size
$\eta=\min\\{1/(\rho^{2}\beta_{2}+\beta\beta_{1}),D_{f}/(\sigma|\mathcal{D}|)\\}$,
where

$\displaystyle
D_{f}:=\sqrt{2(L_{\mathcal{D}}(\bm{\theta}_{0})-\min_{\bm{\theta}}L_{\mathcal{D}}(\bm{\theta}))/(\rho^{2}\beta_{2}+\beta\beta_{1})},$
$\displaystyle\sigma:=2\beta_{1}\rho\sqrt{1+\frac{d\log(3|\mathcal{D}|/\delta)\log(2\delta)}{\epsilon^{2}}},$

then we have the following utility bound:

$\displaystyle\mathbb{E}(L_{\mathcal{D}}(\bm{\theta}_{\text{out}})-\inf_{\bm{\theta}}L_{\mathcal{D}}(\bm{\theta}))\leq
C\cdot\frac{\rho\sqrt{\rho^{2}\beta_{2}+\beta\beta_{1}}}{|\mathcal{D}|\epsilon},$

where
$C=c\cdot\mu\beta_{1}\sqrt{d\log(|\mathcal{D}|/\delta)\log(1/\delta)L_{\mathcal{D}}(\bm{\theta}_{0})}$,
$c$ is some positive constant, $d$ is the dimension of parameter
$\bm{\theta}$.

###### Proof of Theorem 4.

First, by Lemma A.1, we have $L_{i}$ is $\beta_{1}\rho$-Lipschitz continuous
and $(\rho^{2}\beta_{2}+\beta\beta_{1})$-smooth. Then according to Theorem 5
in [67], we have the following utility bound:

$\displaystyle\mathbb{E}\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{\text{out}})\|_{2}^{2}\leq
c\cdot\bigg{(}\frac{\beta_{1}\rho\sqrt{\rho^{2}\beta_{2}+\beta\beta_{1}}\sqrt{d\log(|\mathcal{D}|/\delta)\log(1/\delta)L_{\mathcal{D}}(\bm{\theta}_{0})}}{|\mathcal{D}|\epsilon}\bigg{)},$
(10)

where $c$ is some positive constant. Meanwhile, random round DP-SGD is
$(\epsilon,\delta)$-DP due to Theorem 4 in [67]. Finally, by using the PL-
condition we have $\|\nabla
L_{\mathcal{D}}(\bm{\theta}_{\text{out}})\|_{2}^{2}\geq
1/\mu\cdot(L_{\mathcal{D}}(\bm{\theta}_{\text{out}})-\inf_{\bm{\theta}}L_{\mathcal{D}}(\bm{\theta}))$.
Substituting it into Equation 10 concludes the proof. ∎

##  Appendix B Comparison with Vanilla SAM

In this experiment, we consider four perturbation magnitudes: 0.1, 0.5, 1.0,
and 2.0. The DPML algorithm is fixed at vanilla DP-SGD and the privacy budget
is fixed at $\epsilon=4$. The results of using the vanilla SAM are denoted by
blue triangles, while the results of the proposed DPAdapter are denoted by red
diamonds. The result of using a normally pre-trained model is denoted as a
purple rectangle.

Figure 7: The results of DPAdapter and vanilla SAM under different
perturbation magnitudes. The results of the vanilla SAM are represented by
blue triangles, while the results of the proposed DPAdapter are depicted by
red diamonds. Results obtained using a vanilla pre-trained model are
illustrated by a purple rectangle. All the downstream models have the same
privacy budget $\epsilon=4$.