# Privately Fine-Tuning Large Language Models with Differential Privacy

Rouzbeh Behnia∗ ∗Equally contributing authors (alphabetically ordered by the
last name.) School of Information Systems
and Management
University of South Florida
Sarasota, USA
<EMAIL_ADDRESS>Mohammadreza (Reza) Ebrahimi∗ School of Information Systems
and Management
University of South Florida
Tampa, USA
<EMAIL_ADDRESS>Jason Pacheco  Balaji Padmanabhan  Department of Computer
Science
University of Arizona
Tucson, USA
<EMAIL_ADDRESS>School of Information Systems
and Management
University of South Florida
Tampa, USA
<EMAIL_ADDRESS>
###### Abstract

Pre-trained Large Language Models (LLMs) are an integral part of modern AI
that have led to breakthrough performances in complex AI tasks. Major AI
companies with expensive infrastructures are able to develop and train these
large models with billions and millions of parameters from scratch. Third
parties, researchers, and practitioners are increasingly adopting these pre-
trained models and fine-tuning them on their private data to accomplish their
downstream AI tasks. However, it has been shown that an adversary can
extract/reconstruct the exact training samples from these LLMs, which can lead
to revealing personally identifiable information. The issue has raised deep
concerns about the privacy of LLMs. Differential privacy (DP) provides a
rigorous framework that allows adding noise in the process of training or
fine-tuning LLMs such that extracting the training data becomes infeasible
(i.e., with a cryptographically small success probability). While the
theoretical privacy guarantees offered in most extant studies assume learning
models from scratch through many training iterations in an asymptotic setting,
this assumption does not hold in fine-tuning scenarios in which the number of
training iterations is significantly smaller. To address the gap, we present
EW-Tune, a DP framework for fine-tuning LLMs based on Edgeworth accountant
with finite-sample privacy guarantees. Our results across four well-
established natural language understanding (NLU) tasks show that while EW-Tune
adds privacy guarantees to LLM fine-tuning process, it directly contributes to
decreasing the induced noise to up to 5.6% and improves the state-of-the-art
LLMs performance by up to 1.1% across all NLU tasks. We have open-sourced our
implementations for wide adoption and public testing purposes.

###### Index Terms:

Differential privacy, large language models, fine-tuning, Edgeworth accountant

##  I Introduction

Large language models (LLMs) have become an integral component of modern AI.
Deep learning architectures with billions of parameters are often designed
based on transformers, a building block first introduced by Google’s BERT [1].
LLMs provide breakthrough performance in complex AI tasks such as dialogue
systems [2] and text/automated story generation [3]. Being equipped with the
hardware infrastructure, major AI companies such as Open AI and Facebook
provide new LLMs trained on the public data from the Internet [4, 5]. Common
examples include, RoBERTa [4] and GPT [5]. RoBERTa’s training dataset includes
English Wikipedia and millions of online news crawled from the internet.
Similarly, GPT was trained on outbound links from Reddit.

AI researchers and practitioners often fine-tune these pre-trained models on
their downstream AI tasks using their own private data to accomplish
downstream tasks such as malware detection [6], text-to-image generation [7].
However, recently, it has been shown that these pre-trained models are
vulnerable to privacy attacks [8]. This problem is mainly due to the model’s
tendency to memorize training samples without overfitting, also known as the
”memorization issue” [9]. This issue could lead to three major types of
privacy attacks: membership inference, model inversion, and training data
extraction.

  * •

Membership inference [10]: determines whether a certain user’s data was
included in the training.

  * •

Model inversion [11]: approximate the reconstruction of the training data.

  * •

Training data extraction [8]: aims to exactly reveal the training samples
which makes this type of attack the most powerful one with the most adverse
consequences for the users.

While all three types of attacks can jeopardize the privacy of the users whose
information is in the training data, training data extraction directly targets
users’ personally identifiable information and can endanger users’ identity
via revealing important information such as their address, social security
number, phone number, etc. The fine-tuned LLMs used by third parties on their
private data will face the same privacy concerns. These privacy concerns
around the issue necessitate privacy-preserving approaches for fine-tuning
LLMs. Such an approach will allow third parties to privately fine-tune the
LLMs on their private data without any information leak about their private
training samples.

Differential Privacy (DP) is a promising approach to ensure the training data
privacy with theoretical guarantees [12]. DP provides a mathematically
rigorous framework with privacy guarantees that enables Stochastic Gradient
Descent (SGD), the cornerstone of learning in LLMs, in a private setting. In
such a setting, SGD can be applied as a randomized mechanism multiple times in
each iteration of the training. Most DP methods provide asymptotic guarantees.
For theoretical guarantees, the number of SGD applications (known as
compositions) is often assumed to be unlimited in most privacy studies. This
assumption leads to asymptotic guarantees in these studies (i.e., infinite
compositions of SGD in the limit). However, in LLM fine-tuning the number of
SGD iterations is not only limited but also quite small (i.e., in the order of
several thousand) [13].

In this study, through a DP lens, and thanks to the finite sample guarantee
achieved by Edgeworth expansion [14], we propose a novel LLM fine-tuning
framework, called EW-Tune, with finite-sample guarantees. EW-Tune operates
based on an effective DP accounting approach known as Edgeworth accountant,
proposed in [14]. Edgeworth accountant computes the amount of noise that is
required to be added to the gradients in SGD to guarantee a certain privacy
budget (see Section II-B). EW-Tune also leverages the latest efficient
reparametrization technique proposed in [15].

###  I-A Our contribution

While EW-Tune is a general framework, we showcase its performance by focusing
on its application to enhance the privacy of LLM during fine-tuning. Our
contribution to the LLM’s private fine-tuning is two-fold:

  * •

Our study serves as the first step towards fine-tuning LLMs in a
differentially private setting when the number of compositions (i.e., the
applications of differentially private SGD) is finite and limited to only
several thousand (less than 4,000 times in our experiments). Compared to the
existing methods that provide an asymptotic bound on the privacy budget,
through utilizing Edgeworth accountant, EW-Tune is able to provide a non-
asymptotic privacy bound by using Berry-Esseen bound derived from the
Edgeworth approximation. In the case of fine-tuning LLMs, given the finite
number of compositions, for the same privacy budget, EW-Tune induces less
noise to SGD compared to the state-of-the-art. This directly improves the
learning and the accuracy of the model.

  * •

It is known that while fine-tuning via DP enhances the model’s privacy, it can
negatively affect the model’s utility (i.e., performance) [12]. Our
experiments show that EW-Tune significantly contributes to the state of the
art by enhancing the privacy of LLMs while preserving their utility/accuracy
compared to multiple recent alternative methods across several important
downstream benchmark tasks including text classification, entailment
detection, and question answering. Overall, EW-Tune decreases the noise-
induced to SGD up to 5.6%. EW-Tune also enhances the state-of-the-art model’s
accuracy by up to 1.1%.

##  II Background and Related Work

We review three areas of the literature: (1) LLMs to identify the state-of-
the-art in language modeling and their fine-tuning. (2) Differentially private
deep learning as the overarching framework to rigorously guarantee the privacy
of fine-tuning LLMs. (3) Edgeworth accountant as an emerging accountant method
that provides fine-sample guarantees, which could be a useful tool for fine-
tuning LLMs.

###  II-A Large Language Models (LLMs)

Large language models are deep neural network architectures with billions of
parameters [16, 17, 18]. They often benefit from an encoder-decoder
architecture that generates high-quality representations from sequence data
(text, image, malware, genes, etc.). Most LLMs use specific types of layers
with self-attention mechanisms known as transformers to dynamically assign
weights to input elements based on their surrounding context [16].
Transformers enable LLMs to provide high-quality representations of the input
sequence. At a high level, LLMs can be categorized into two types: masked and
autoregressive.

Masked language models are trained to predict a masked token based on its
surroundings. Highly effective examples of masked language models include BERT
[1] and RoBERTa [16]. On the contrary, autoregressive language models learn to
predict the next token based on the previously generated ones, which makes
them suitable for text generation tasks [4, 19].

Due to their ability to produce high-quality representations from input,
masked language models are widely used in major downstream AI tasks including
text classification, question answering, semantic entailment detection, and
speech recognition.

Pre-trained LLMs are often fine-tuned on specific tasks and datasets, through
which the weights of the original model are updated to better tune for the
domain-specific data and task in hand.

###  II-B Differentially Private Deep Learning

Differential privacy [20], formally defined in Definition 1, computes a
privacy guarantee when the results of an algorithm, run on private data, are
made public. When applied to machine learning, a differentially private (DP)
mechanism allows for the public release of the model parameters while ensuring
the privacy of the original training data.

###### Definition 1

A randomized mechanism $M:\mathcal{X}\rightarrow\mathcal{Y}$ is
$(\epsilon,\delta)$-DP, if for all adjacent datasets
$X,X^{\prime}\in\mathcal{X}$, differing in a single element only, and all
$Y\subset\mathcal{Y}$, $\mathbb{P}(M(X)\in Y)\leq
e^{\epsilon}\mathbb{P}(M(X^{\prime})\in Y)+\delta$ holds.

In Definition 1, $(\epsilon,\delta)$ is often referred to as the privacy
budget. $\epsilon$ defines the distance between the two sides of the equation
and $\delta$ defines a failure probability. Differential privacy enjoys from
properties such as robustness to auxiliary information and composability. The
former guarantees privacy even with the emergence of new side-information to
the adversary and the latter allows for modular design of mechanisms.
Essentially, composability implies that if two mechanisms $M_{1}(\cdot)$ and
$M_{2}(\cdot)$ are DP, and $M(X)=(M_{1}(X),M_{2}(X))$, then $M$ is also
differentially private.

Differentially private Stochastic Gradient Descent (DP-SGD). The gold standard
to achieve differential privacy in deep learning is to update the neural
network parameters with _noisy_ gradients. This is achieved by a randomized
algorithm called DP-SGD [12] via the two following steps:

  * –

Gradient clipping: given a clipping norm $C$, the gradient of each sample $x$,
$\mathbf{g}(x)$, is clipped
$\mathbf{g}^{\prime}(x)\leftarrow\mathbf{g}(x)/\max(1,\frac{\|\mathbf{g}(x)\|_{2}}{C})$.

  * –

Gaussian mechanism: the clipped gradients are aggregated and then an isotropic
Gaussian noise from $\mathcal{N}(0,C^{2}\sigma^{2})$, with $\sigma$ as a noise
multiplier, will be added to the gradients.

The noise multiplier in DP-SGD is determined by the privacy budget
$(\epsilon,\delta)$, the number of training rounds $m$, and the sampling
probability $q=B/N$ for batch size $B$ and $N$ as the total number of samples.

In their seminal work, Abadi et al. [12] introduced a method called Moments
accountant (MA) for computing an upper bound for the privacy curve of
compositions of DP algorithms. The method was then used to track the privacy
loss of DP-SGD by computing the _privacy curve_ for each training iteration
with itself $m$ times, where $m$ is the total number of iterations. In [21],
the MA framework is instantiated with Renyi Differential Privacy (RDP).
However, these algorithms, while being efficient (runtime is independent of
$m$), provide an upper bound which is rather impractical.

The Gaussian Differential Privacy (GDP) framework [22, 23] also called $f$-DP
is devised based on the central limit theorem (CLT). The GDP framework offers
a nice characterization of differential privacy using hypothesis testing
interpretation [24]. GDP can only provide an _approximation_ for the privacy
curve and it was shown to underreport the true epsilon value in [25].

Using the notion of privacy loss random variables (PRV) [26], Meiser and
Mohammadi [27] introduced an algorithm called _privacy bucket_ for
approximately composing privacy curves. By employing the notion of PRV, one
can utilize the nice property of PRV to compute the composition of $m$
mechanisms $M=M_{1}\circ M_{2}\circ\cdots\circ M_{m}$ by simply summing their
corresponding PRVs $\mathcal{D}=\sum_{i=1}^{m}\mathcal{D}_{i}$. The
distribution of $\mathcal{D}$ can then be approximated by computing the
convolution of its underlying distributions
$\mathcal{D}_{1},\dots,\mathcal{D}_{m}$. Koskela et al. [28] used fast Fourier
transform (FFT) to compute the convolution efficiently. Following [28], Gopi
et al. [25] leveraged FFT to numerically compose trade-off functions. Their
accountant, called the PRV accountant addresses the underestimation of $f$-DP
and provides an upper-bound and lower-bound on the leakage of $\epsilon$.

###  II-C Edgeworth Accountant

As noted, at the heart of EW-Tune is Edgeworth accountant [14]. Edgeworth
accountant relies on $f$-DP [23], which as discussed above, offers a full
characterization of differential privacy by utilizing hypothesis testing
interpretation. Informally, differential privacy measures the hardness in
distinguishing any pair of (neighboring) datasets based on the information
obtained from a mechanism $M$. In [23], the authors formulated the notion of
indistinguishably as a hypothesis testing problem for two neighboring datasets
$S$ and $S^{\prime}$. Therefore, the hypotheses are formed as $H_{0}$: the
underlying dataset is $S$ and $H_{1}$: the underlying dataset is $S^{\prime}$,
where the output of $M$ is the basis for conducting a hypothesis testing
problem. Let $P$ and $Q$ denote the probability distributions of $M(S)$ and
$M(S^{\prime})$, respectively. Now, for a rejection rule $0\leq\o\leq 1$, and
hypotheses $H_{0}:P$ and $H_{1}:Q$, the trade-off function
$f=T(P,Q)(\alpha)=\inf\\{\beta_{\o}:\alpha_{\o}\leq\alpha\\}$ defines the
mapping from the Type-I error to Type-II error, where
$\alpha_{\o}=\mathbb{E}_{P}[\o]$ and $\beta_{\o}=1-\mathbb{E}_{Q}[\o]$. To
compute the composition of the trade-off functions of the form
$f=\bigotimes^{m}_{i=1}f_{i}$, let us realize the $i$-th composition by two
hypothesis $H_{0,i}=w_{i}\sim P_{i}$ and $H_{1,i}=w_{i}\sim Q_{i}$. Now, to
evaluate the trade-off function $f=\bigotimes^{m}_{i=1}f_{i}$, we distinguish
between two composite hypothesis $H_{0,i}=\mathbf{w}\sim P_{1}\times
P_{2}\times\cdots\times P_{m}$ and $H_{1,i}=\mathbf{w}\sim Q_{1}\times
Q_{2}\times\cdots\times Q_{m}$ for $\mathbf{w}=(w_{1},\dots,w_{m})$.

Edgeworth accountant [14] defines random variables called privacy-loss log-
likelihood ratios (PLLRs) to enable the lossless conversion of the $f$-DP
guarantee into a collection of $(\epsilon,\delta)$-DP guarantees. PLLRs are
defined as a Radon-Nikodym derivatives of the hypotheses above as
$X_{i}\equiv\log\frac{dQ_{i}(\zeta_{i})}{dP_{i}(\zeta_{i})}$ and
$Y_{i}\equiv\log\frac{dQ_{i}(\chi_{i})}{dP_{i}(\chi_{i})}$ for $\zeta\sim
P_{i}$ and $\chi\sim Q_{i}$. The authors in [14] showed the primal-dual
relationship between $f$-DP and a collection of
$(\epsilon,\delta(\epsilon))$-DP via
$\delta=1-F_{Y,m}(\epsilon)-\mathbf{e}^{\epsilon}(1-F_{X,m}(\epsilon))$ where
$F_{X,m}$ and $F_{Y,m}$ are the CDFs of $\sum_{i=1}^{m}X_{i}$ and
$\sum_{i=1}^{m}Y_{i}$, respectively. To compute PLLRs through composite
hypotheses, the Edgeworth accountant uses a family of PLLR sequences to
compose the tightest possible trade-off function that satisfies all
$f^{(\alpha)}$-DP.

Assuming that for each $\alpha$, one can find a series of PLLRs corresponding
to $f^{(\alpha)}$, we can compute a collection of
$(\epsilon,\delta^{(\alpha)}(\epsilon))$-DP guarantee111The authors in [14]
discuss how one can invert the equation to compute a $\epsilon$ for a given
$\delta$. Then one has to compute an approximate CDF as a random variable
$X=\sum_{i=1}^{m}X_{i}$ using Edgeworth expansion to output the Edgeworth
accountant approximates as $F_{X,m}$ and $F_{Y,m}$.

##  III Proposed Method

###  III-A Threat Model

Adversary capabilities and objectives. We consider an adversary $\mathcal{A}$
to have black-box access to the language model. In this work, following [8],
we assume that $\mathcal{A}$ does not have access to the model’s specific
weights and hidden states but is able to obtain next-word predictions and
compute the probability of arbitrary sequences for instance via access to
auto-complete models. The ultimate goal of the adversary is to extract (the
memorized) training data from the model. The severity of an attack is
increased if more examples could be extracted from the model.

Figure 1: Abstract View of the Proposed EW-Tune Framework

Adversary’s target and tasks. While EW-Tune is a general framework that can be
applied to enhance the privacy of any LLM during fine-tuning. For specificity,
we focus on one of the most highly-adopted masked language models in AI tasks,
a successor of Google’s BERT, named roBERTa [16]. roBERTa mainly owes its
popularity to its ability to learn the bidirectional representation of the
sentence. These high-quality representations that are not available in
autoregressive language models such as GPT, specifically contribute to
breakthrough results in common downstream natural language understanding (NLU)
tasks such as sentiment analysis and text categorization. Also, to show the
generalizability of EW-Tune, we will test its utility and privacy guarantees
across four important and complex NLU tasks (all included in the well-known
General Language Understanding Evaluation (GLUE) benchmark dataset [29]). Each
task is associated with a well-established dataset:

  * •

MNLI [30]: The Multi-Genre Natural Language Inference (MNLI) is a collection
of 433,000 sentence pairs that are annotated with semantic entailment
information [30]. The LLM’s task in this corpus is to identify the semantic
relationships between a given pair of sentences (entailment, contradiction, or
neutral relationship).

  * •

QNLI [29]: The Question-answering Naural Language Inference (QNLI) is a
natural language inference dataset collected from Wikipedia that consists of
110,400 question-paragraph pairs, where only one of the sentences in the
paragraph is the answer to the corresponding question. The LLM’s task is to
determine whether a sentence includes the answer to a given question.

  * •

QQP [29]: The Quora Question Pairs (QQP) dataset includes over 400,000
question pairs. Each question pair is annotated to indicate whether these
questions are semantically equivalent (i.e., paraphrase of each other). The
LLM’s task is to determine whether either of the questions is the paraphrase
of the other one.

  * •

SST-2 [31]: The Stanford Sentiment Treebank (SST-2) includes 68,800 sentences
from movie reviews and annotations of their sentiment. The LLM’s task is to
predict the sentiment (positive or negative) of a given sentence.

To operationalize defense against the above threat model, we propose EW-Tune,
a general framework for fine-tuning LLMs for different downstream tasks.
Figure 1 depicts the components of our proposed EW-Tune framework.

Algorithm 1 EW-Tune Framework

1:Input: Examples $\\{x_{i}\\}_{i=1}^{N}$, mechanisms $\\{M_{i}\\}_{i=1}^{m}$,
a weight matrices $\\{\mathbf{W}^{(l)}\\}^{H}_{l=1}$, warm-up steps $T_{w}$,
group size $B$, gradient clipping bound $C$, the failure probability $\delta$
and an initial privacy budget $\epsilon$ and a $k$-th order Edgeworth
expansion.

2:Given the sampling probability $q=B/N$, for a given $\delta$, for each
mechanism and all $\alpha$ encode the PLLRs
$[(X_{i}^{\alpha},Y_{i}^{\alpha})]$ and the cumulants up to order $k+2$

3:For each $\alpha$ compute the Edgeworth approximation and calculate
$\epsilon^{(\alpha)}(\delta)$ and supremum
$\sup_{\alpha}\epsilon^{(\alpha)}(\delta)$

4:Given an $\epsilon_{init}$ and an arbitrary initial $\sigma_{arb}$ (e.g.,
$\sigma=10$)

5:while ($\epsilon<\epsilon_{init}$ AND $r<\sigma$) do

6: Recompute $\epsilon$ (Steps 2-4) and refine and reduce $\sigma$ using an
initial reducing factor $r$ (e.g., $r=0.5$)

7:end while

8:Choose a gradient-carrier matrix $\\{\mathbf{W}^{(l)}\\}^{H}_{l=1}$
according to [15].

9:Sample a batch of examples with probability $q$.

10:Compute historical update to find gradient carrier and decompose and
compute the low-rank gradient carrier $\mathbf{L}$ and $\mathbf{R}$ via [15,
Algorithm 2]

11:Run reparametrized forward/backward process and compute individual
gradients
$\\{\partial_{i}\mathbf{L}_{t}^{(l)},\partial_{i}\mathbf{R}_{t}^{(l)}\\}_{l\in[H],i\in
S_{t}}$

12:Given $C$ and $\sigma$, clip and add noise to the individual gradients as
in Section II-B to get $\tilde{\partial}\mathbf{L}_{t}^{(l)}$ and
$\tilde{\partial}\mathbf{R}_{t}^{(l)}$

13:Construct
$\tilde{\partial}\mathbf{W}_{t}^{(l)}=(\tilde{\partial}\mathbf{L})\mathbf{R}+\mathbf{L}(\tilde{\partial}\mathbf{R})-\mathbf{L}\mathbf{L}^{T}(\tilde{\partial}\mathbf{L})\mathbf{R}$

14:Output:
$\langle\tilde{\partial}\mathbf{W}_{t}^{(l)},(\epsilon,\delta)\rangle$

TABLE I: Performance Comparison of EW-Tuneagainst state of the art DP methods
across four different NLU tasks conducted by roBERTa LLM

Method | MNLI | QNLI | QQP | SST-2 | MNLI | QNLI | QQP | SST-2
---|---|---|---|---|---|---|---|---
Accuracy | Noise Multiplier
RDP | 81.25% | 86.63% | 84.60% | 89.24% | 0.65 | 0.829 | 0.6575 | 0.921
PRV | 81.22% | 86.79% | 84.78% | 91.82% | 0.607 | 0.768 | 0.6135 | 0.8485
EW-Tune |  81.81% |  87.71% |  84.91% |  92.19% | 0.573 | 0.739 | 0.579 | 0.8215

  * •

Performance reported for $\delta=1e-6$ for MNLI, QNLI, and QQP; $\delta=1e-5$
for SST-2; batch size $=2000$

As seen in Figure 1, the pre-trained LLM is learned on a public dataset (e.g.,
internet) from scratch by a major AI company (e.g., Google and OpenAI) (shown
in the left side of Figure 1). Subsequently, the pre-trained model is used as
an input to EW-Tune to fine-tune with privacy guarantees expressed by privacy
parameters (i.e., privacy budget $\epsilon\in[0,\infty)$ and failure
probability $\delta\in[0,1]$) (as shown in the right side of Figure 1). The
privacy parameters $(\epsilon,\delta)$ are provided by the user/practitioner.
A smaller $\epsilon$ indicates better privacy preservation (and lower
utility/performance). $\delta$ denotes the probability that the training
examples are accidentally being leaked. In the context of LLM, a suitable
value for $\epsilon$ is between 5 to 8, and $\delta$ is recommended to take a
value in the order of the inverse of the size of the training samples [13].

After the user provides the privacy parameters, the Edgeworth accountant
algorithm [14] is utilized to (1) compute the number of compositions (i.e.,
applications of DP-SGD) for a given dataset. (2) compute the amount of noise
that guarantees the given privacy budget. Subsequently, any variation of DP-
SGD [12, 15], as described in Section II-B, can be used to fine-tune the LLM
on the private dataset based on the appropriate noise multiplier $\sigma$
obtained in the previous step. Due to its breakthrough performance, we
utilized a recent version of the DP-SGD algorithm that is based on a new
method called reparameterized gradient perturbation (RGP) [15]. In the
original DP-SGD [12], the noise introduced highly depends on the model
parameters, and the per-example gradient clipping results in very high memory
and computation overhead. RGP addresses the problems of DP-SGD by
reparameterizing each layer’s weight matrix $\mathbf{W}$ into two low ranks
gradient-carrier matrices $\mathbf{L}$ and $\mathbf{R}$ and a residual weight
matrix $\tilde{\mathbf{W}}$. Finally, all the transformer layers of the LLM
will be fine-tuned on the private data through DP-SGD. The output of the EW-
Tune framework is the fine-tuned LLM as shown in the right side of Figure 1.

Algorithm 1 presents a more detailed version of our framework (EW-Tune). The
algorithm starts by computing an $\epsilon$ for a given $\delta$ based on the
Edgeworth accountant explained in Section II-C (Steps 2-4) by first computing
the PLLRs and then approximating their CDF using Edgeworth expansion. Then we
compute $\epsilon^{(\alpha)}(\delta)$ and its supremum. Next, in Steps 5-8, we
compute our final $\epsilon$ and an appropriate noise multiplier $\sigma$ to
be used in our reparametrized gradient perturbation (i.e., noise addition as
in Section II-B). In Step 5, with each iteration of the loop we compute
$\sigma=\sigma-r$ and the initial tuning factor $r$ is reduced by a constant
factor (e.g., we use $r/10$ in our code). Finally, given the $\sigma$, the RGP
algorithm efficiently perturbs the updated parameters (Steps 9-14). For each
update with the weight matrix $\mathbf{W}$, the algorithm works in four main
steps. In the first step the gradient carrier matrices $\mathbf{L}$ and
$\mathbf{R}$ are generated via the decomposition method proposed in [15,
Algorithm 2]. The output of this step is the orthonormalize version (via Gram-
Schmidt orthonormalization process) of the gradient carrier matrices. Next,
the weight matrices are reparametrized to compute and store individual
gradients via the forward/backward process presented in [15, Section 2]. In
the third step, the gradients are clipped and made noisy similar to the DP-SGD
method presented in Section II-B. Lastly, in Step 14, the noisy aggregated
gradients of the carrier matrices
$\langle\tilde{\partial}\mathbf{W}_{t}^{(l)}$ are used to compute the
gradients of the original weights.

##  IV Experiments

We have open-sourced our optimized framework for wide adoption and testing
purposes at the following link.

https://github.com/star-ailab/LLM_Tune

###  IV-A Experiment Setup

EW-Tune was developed and run on a single NVIDIA GeForce RTX 3090 with 10,496
CUDA cores and 24 GB of internal memory. To ensure that the LLM loading and
fine-tuning can take place in the internal memory we selected the roBERTa.base
pre-trained model with 125 million parameters. This LLM can be accessed from
https://github.com/facebookresearch/fairseq/tree/main/examples/roberta.

####  IV-A1 Parameter settings

Consistent with [29], we set the train and test partition for each dataset:
(MNLI: 393,000 for training and 20,000 for testing; QNLI: 105,000 for training
and 5400 for testing; QQP: 364,000 for training and 391,000 for testing,
SST-2: 67,000 for training and 1800 for testing).

To facilitate comparison, we set the privacy parameters $\epsilon$ and
$\delta$ consistent with [13]. Accordingly, we set $\epsilon=8$, $\delta=1e-6$
for larger datasets (i.e., MNLI, QNLI, and QQP; each with several hundred
thousand of samples) and $\delta=1e-5$ for the smaller dataset (i.e., SST-2;
with tens of thousands of samples).

####  IV-A2 Benchmark Experiments

Following [14, 13] We evaluated the performance of the proposed EW-Tune
framework against two widely-used state-of-the-art DP alternatives: Renyi
Differential Privacy (RDP) [21, 12] and Privacy Loss Random Variables (PRV)
[25]. To rigorously evaluate EW-Tune, we conduct two sets of experiments
(Section IV-B).

In Experiment 1, we evaluate the accuracy of the LLM in solving the four
mentioned NLU tasks (MNLI, QNLI, QQP, SST-2) after fine-tuning with EW-Tune,
RDP and PRV. In Experiment 2, we evaluate the amount of noise induced by
alternative privacy accountant algorithms to that of EW-Tune for different
values of $\epsilon$.

###  IV-B Results

####  IV-B1 Experiment 1

Table I shows the results of comparing the accuracy and noise multiplier of
the proposed EW-Tune against RDP and PRV across four NLU datasets (MNLI, QNLI,
QQP, and SST-2) carried out by the roBERTa LLM. To report the performance, we
repeated each experiment 3 times and reported the average. The highest
accuracy in performing each task appears in bold font. As seen in Table I, EW-
Tune yields the highest performance (81.81% on MNLI, 87.71% on QNLI, 84.91% on
QQP, and 92.19% on SST-2) among its counterparts. At the heart of EW-Tune, the
Edgeworth accountant utilizes an accurate privacy computation method called
$f$-differential privacy ($f$-DP) along with Edgeworth approximation, in place
of CLT, that enjoys from a much better convergence rate. This allows EW-Tune
to achieve the privacy guarantees by applying less noise to the transformer
layers of LLMs during training. The noise multiplier (i.e., the standard
deviation of Gaussian noise distribution) is shown on the right side of Tabel
I. As shown in Tabel I, EW-Tune yields the lowest noise multiplier. More
specifically, as compared to the state-of-the-art, the noise multiplier is up
to 4%, 3.2%, 5.6%, and 3.2% lower (for $5\leq\epsilon\leq 8$) for MNLI, QNLI,
QQP, and SST-2, respectively. In Tabel I, the highest performance numbers and
lowest noise multipliers are indicated in boldface. We note that if instead of
utilizing DP-SGD with RGP [15], one were to instantiate these LLM’s with the
original DP-SGD [12], the smaller noise multiplier numbers in EW-Tune would
result in a much higher performance gap with our counterparts.

####  IV-B2 Experiment 2

Experiment 2 evaluates the amount of noise induced by EW-Tune and other
benchmark privacy accountant algorithms at $\delta=1e-6$ for MNLI, QNLI, QQP,
and $\delta=1e-5$ for SST-2. As noted, it is desirable to achieve the same
privacy budget ($\epsilon$) by applying less noise to the transformer layers
of the LLM during the fine-tuning. As shown in Figure 2, EW-Tune yields the
lowest amount of noise across the benchmark privacy accounting methods (i.e.,
RDP and PRV). As shown in Figure 2, when $\epsilon$ changes from 5 to 8, EW-
Tune induces the lowest amount of noise into the SGD process for all four NLU
tasks (MNLI, QNLI, QQP, and SST-2).

Figure 2: The Changes of Noise Multiplier based on different values of
$\epsilon$ across three Privacy Accounting Algorithms (RDP, PRV, and EW-Tune.)

Overall, The results of Experiments 1 and 2 on four complex NLU task shows
that EW-Tune is able to enhance the performance thorough applying less noise
to the SGD process, while achieving the same privacy budgets as its
counterpart algorithms. EW-Tune outperforms the other privacy accountant
methods for different values of privacy budget.

##  V Conclusion and Future Work

In this work we presented a new framework called EW-Tune, specifically
designed for fine-tuning LLMs. By utilizing the state-of-the-art privacy
accountant and gradient perturbation methods, EW-Tune is able to provide
finite-sample privacy guarantee by introducing less noise as compared to the
existing methods. EW-Tune introduces up to 6% less noise when privately
training large language models which contributes to up to 1.1% performance
improvement. This can contribute to addressing the gap in privacy and accuracy
trade-off in the realm of data privacy and AI.

An interesting future work would be to further study the relationship between
the introduced noise and training accuracy by focusing on the model’s total
number of parameters, dataset size, task objectives, and the number of
compositions.

## Acknowledgment

We would like to thank Hua Wang from the Statistics Department at University
of Pennsylvania for illuminating discussions on Edgeworth accountant and
helpful comments on its implementation.

## References

  * [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [2] D. Ham, J.-G. Lee, Y. Jang, and K.-E. Kim, “End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 583–592.
  * [3] L. Fang, T. Zeng, C. Liu, L. Bo, W. Dong, and C. Chen, “Transformer-based conditional variational autoencoder for controllable story generation,” _arXiv preprint arXiv:2101.00828_ , 2021.
  * [4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [5] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [6] J. L. Hu, M. Ebrahimi, and H. Chen, “Single-shot black-box adversarial attacks against malware detectors: A causal language model approach,” in _2021 IEEE International Conference on Intelligence and Security Informatics (ISI)_. IEEE, 2021, pp. 1–6.
  * [7] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [8] N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, “Extracting training data from large language models,” in _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_ , M. Bailey and R. Greenstadt, Eds. USENIX Association, 2021, pp. 2633–2650.
  * [9] N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in _28th USENIX Security Symposium (USENIX Security 19)_ , 2019, pp. 267–284.
  * [10] S. Hisamoto, M. Post, and K. Duh, “Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system?” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 49–63, 2020.
  * [11] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , 2015, pp. 1322–1333.
  * [12] M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016_ , E. R. Weippl, S. Katzenbeisser, C. Kruegel, A. C. Myers, and S. Halevi, Eds. ACM, 2016, pp. 308–318. [Online]. Available: https://doi.org/10.1145/2976749.2978318
  * [13] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz _et al._ , “Differentially private fine-tuning of language models,” _arXiv preprint arXiv:2110.06500_ , 2021\.
  * [14] H. Wang, S. Gao, H. Zhang, M. Shen, and W. J. Su, “Analytical composition of differential privacy via the edgeworth accountant,” _arXiv preprint arXiv:2206.04236_ , 2022.
  * [15] D. Yu, H. Zhang, W. Chen, J. Yin, and T.-Y. Liu, “Large scale private learning via low-rank reparametrization,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 208–12 218.
  * [16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [17] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [18] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer.” _J. Mach. Learn. Res._ , vol. 21, no. 140, pp. 1–67, 2020.
  * [19] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [20] C. Dwork, F. McSherry, K. Nissim, and A. D. Smith, “Calibrating noise to sensitivity in private data analysis,” in _Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings_ , ser. Lecture Notes in Computer Science, S. Halevi and T. Rabin, Eds., vol. 3876. Springer, 2006, pp. 265–284. [Online]. Available: https://doi.org/10.1007/11681878_14
  * [21] I. Mironov, “Renyi differential privacy,” _CoRR_ , vol. abs/1702.07476, 2017\. [Online]. Available: http://arxiv.org/abs/1702.07476
  * [22] Z. Bu, J. Dong, Q. Long, and W. J. Su, “Deep learning with gaussian differential privacy,” _CoRR_ , vol. abs/1911.11607, 2019. [Online]. Available: http://arxiv.org/abs/1911.11607
  * [23] J. Dong, A. Roth, and W. J. Su, “Gaussian differential privacy,” _CoRR_ , vol. abs/1905.02383, 2019. [Online]. Available: http://arxiv.org/abs/1905.02383
  * [24] P. Kairouz, S. Oh, and P. Viswanath, “The composition theorem for differential privacy,” in _International conference on machine learning_. PMLR, 2015, pp. 1376–1385.
  * [25] S. Gopi, Y. T. Lee, and L. Wutschitz, “Numerical composition of differential privacy,” in _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 11 631–11 642. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/6097d8f3714205740f30debe1166744e-Abstract.html
  * [26] C. Dwork and G. N. Rothblum, “Concentrated differential privacy,” _CoRR_ , vol. abs/1603.01887, 2016. [Online]. Available: http://arxiv.org/abs/1603.01887
  * [27] S. Meiser and E. Mohammadi, “Tight on budget?: Tight bounds for r-fold approximate differential privacy,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018_ , D. Lie, M. Mannan, M. Backes, and X. Wang, Eds. ACM, 2018, pp. 247–264. [Online]. Available: https://doi.org/10.1145/3243734.3243765
  * [28] A. Koskela, J. Jälkö, and A. Honkela, “Computing tight differential privacy guarantees using FFT,” in _The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]_ , ser. Proceedings of Machine Learning Research, S. Chiappa and R. Calandra, Eds., vol. 108\. PMLR, 2020, pp. 2560–2569. [Online]. Available: http://proceedings.mlr.press/v108/koskela20b.html
  * [29] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=rJ4km2R5t7
  * [30] A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” _arXiv preprint arXiv:1704.05426_ , 2018.
  * [31] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 1631–1642.


acoustics actuators addictions admsci adolescents aerospace agriculture
agriengineering agronomy ai algorithms allergies analytica animals antibiotics
antibodies antioxidants appliedchem applmech applmicrobiol applnano applsci
aquacj architecture arts asc asi atmosphere atoms audiolres automation axioms
bacteria batteries bdcc behavsci beverages biochem bioengineering biologics
biology biomechanics biomed biomedicines biomedinformatics biomimetics
biomolecules biophysica biosensors biotech birds bloods brainsci breath
buildings businesses cancers carbon cardiogenetics catalysts cells ceramics
challenges chemengineering chemistry chemosensors chemproc children cimb
civileng cleantechnol climate clinpract clockssleep cmd coasts coatings
colloids colorant compounds computation computers condensedmatter conservation
constrmater cosmetics covid crops cryptography crystals ctn curroncol cyber
dairy data dentistry dermato dermatopathology designs diabetology diagnostics
digital disabilities diseases diversity dna drones dynamics earth ebj
ecologies econometrics economies education ejihpe electricity electrochem
electronicmat electronics encyclopedia endocrines energies eng engproc entropy
environments environsciproc epidemiologia epigenomes est fermentation fibers
fire fishes fluids foods forecasting forensicsci forests foundations
fractalfract fuels futureinternet futurephys futuretransp galaxies games gases
gastroent gastrointestdisord gels genealogy genes geographies geohazards
geomatics geosciences geotechnics geriatrics hazardousmatters healthcare
hearts hemato heritage highthroughput histories horticulturae humanities
humans hydrogen hydrology hygiene idr ijerph ijfs ijgi ijms ijns ijtm ijtpp
immuno informatics information infrastructures inorganics insects instruments
inventions iot j jcdd jcm jcp jcs jdb jfb jfmk jimaging jintelligence jlpea
jmmp jmp jmse jne jnt jof joitmc jor journalmedia jox jpm jrfm jsan jtaer jzbg
land languages laws life liquids literature livers logistics lubricants
machines macromol magnetism magnetochemistry make marinedrugs materials
materproc mathematics mca medicina medicines medsci membranes merits
metabolites metals metrology micro microarrays microbiolres micromachines
microorganisms microplastics minerals mining modelling molbank molecules mps
mti nanoenergyadv nanomanufacturing nanomaterials ncrna network neuroglia
neurolint neurosci nitrogen notspecified nri nursrep nutraceuticals nutrients
obesities oceans ohbm onco optics oral organics osteology oxygen parasites
parasitologia particles pathogens pathophysiology pediatrrep pharmaceuticals
pharmaceutics pharmacy philosophies photochem photonics phycology physchem
physics physiolsci plants plasma pollutants polymers polysaccharides poultry
preprints proceedings processes prosthesis proteomes psych psychiatryint
publications quantumrep quaternary qubs radiation reactions recycling
regeneration religions remotesensing reports reprodmed resources rheumato
risks robotics ruminants safety sci scipharm sensors separations sexes signals
sinusitis skins smartcities sna societies socsci soilsystems solids sports
standards stats stresses surfaces surgeries suschem sustainability symmetry
systems taxonomy technologies telecom test textiles thermo tomography
tourismhosp toxics toxins transplantology transportation traumacare traumas
tropicalmed universe urbansci uro vaccines vehicles venereology vetsci
vibration viruses vision water wem wevj women world