#  PDSS: A Privacy-Preserving Framework for Step-by-Step Distillation of Large
Language Models

Tao Fan1, 2, Yan Kang2, Weijing Chen2, Hanlin Gu2, Yuanfeng Song2,
Lixin Fan2, Kai Chen1, Qiang Yang1,2
1 Hong Kong University of Science and Technology, China
2 WeBank, China
Correspondence<EMAIL_ADDRESS>
###### Abstract

In the context of real-world applications, leveraging large language models
(LLMs) for domain-specific tasks often faces two major challenges: domain-
specific knowledge privacy and constrained resources. To address these issues,
we propose PDSS, a privacy-preserving framework for step-by-step distillation
of LLMs. PDSS works on a server-client architecture, wherein client transmits
perturbed prompts to the server’s LLM for rationale generation. The generated
rationales are then decoded by the client and used to enrich the training of
task-specific small language model(SLM) within a multi-task learning paradigm.
PDSS introduces two privacy protection strategies: the Exponential Mechanism
Strategy and the Encoder-Decoder Strategy, balancing prompt privacy and
rationale usability. Experiments demonstrate the effectiveness of PDSS in
various text generation tasks, enabling the training of task-specific SLM with
enhanced performance while prioritizing data privacy protection.

PDSS: A Privacy-Preserving Framework for Step-by-Step Distillation of Large
Language Models

Tao Fan1, 2, Yan Kang2, Weijing Chen2, Hanlin Gu2, Yuanfeng Song2, Lixin Fan2,
Kai Chen1, Qiang Yang1,2 1 Hong Kong University of Science and Technology,
China 2 WeBank, China Correspondence<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models(LLMs), boasting billions of parameters and remarkable
text generation abilities, have risen as a revolutionary force in artificial
intelligence. Prominent models, such as GPT-4 OpenAI (2023), LLaMATouvron et
al. (2023), and QwenBai et al. (2023), have garnered the attention of
researchers and practitioners alike, demonstrating unparalleled proficiency
across numerous tasks. Nevertheless, the sheer size of these models presents
significant obstacles for real-world deployment, particularly in environments
with limited resources. Meanwhile, as LLMs gain escalating popularity and
widespread utilization, privacy concerns have moved to the forefront,
especially when it comes to user data and model inference. In contrast, Small
Language Models(SLMs) often exhibit superior computational efficiency and
faster convergence rates, rendering them perfectly suited for real-time
applications or resource-constrained environments. Nonetheless, SLMs also
possess certain drawbacks stemming from their performance limitations. The
question then arises: How can we effectively combine the predictive prowess of
LLMs with the nimbleness of SLMs, all while adhering to privacy requirements?

To address these challenges, we introduce PDSS, a privacy-preserving framework
for step-by-step distillation of LLMs. In our envisioned setup, there’s a
high-powered server capable of deploying an LLM, paired with a client
possessing more limited computational resources running SLM. The challenge
lies in maintaining the privacy of client data while leveraging the server’s
LLM to aid in training the client’s SLM for text generation tasks, thereby
elevating its performance. PDSS aims to bridge this gap, enabling secure and
efficient knowledge transfer between LLM and SLM, and ultimately enhancing the
capabilities of the SLM without compromising privacy.

As illustrated in Figure 1, within our framework, the process works as
follows. Initially, the client transmits perturbed prompts to the server’s
LLM, which are protected by the PDSS prompt encoder module, thus ensuring
privacy protection. Subsequently, the server’s LLM generates perturbed
rationales from these prompts through the Chain of Thought (COT) approach Wei
et al. (2022) and relays them back to the client. Upon receiving these
perturbed rationales, the client’s rationales decoder module reconstructs them
into their original, aligned form corresponding to the raw prompt. Ultimately,
the client incorporates these rationales as supplementary and enriching
information for training its Task-Specific SLM within a multi-task learning
paradigm Wei et al. (2022); Hsieh et al. (2023); Zhang and Yang (2021). These
rationales justify the predicted labels and serve as insightful guidance for
training smaller and domain-specific models.

Within the PDSS framework, to achieve a balance between preserving the privacy
of user prompts and enhancing the usability of rationales, we introduce two
privacy protection strategies incorporated into the the prompt encoder module
and the rationales decoder module: the Exponential Mechanism Strategy and the
Encoder-Decoder Strategy. In the Exponential Mechanism Strategy, we utilize an
exponential mechanism to obfuscate the prompts Tong et al. (2023), followed by
decoding the perturbed rationales through In-Context Learning (ICL) Dong et
al. (2022). In the Encoder-Decoder strategy, we utilize an Encoder-Decoder SLM
specifically designed to encode raw prompts into perturbed prompts and
subsequently decode perturbed rationales back into their original form. To
effectively train this unified Encoder-Decoder SLM, we utilize a multi-task
learning paradigm Zhang and Yang (2021), encompassing both the encoding and
decoding training processes.

Our contributions are summarized as follows:

  * •

Privacy-Preserving Framework for LLM Distillation. We propose PDSS, a novel
framework that facilitates secure and efficient knowledge transfer from LLM to
SLM in resource-constrained environments while adhering to privacy
requirements. PDSS addresses the challenges posed by the massive size of LLMs
for real-world deployment and the privacy concerns surrounding user data. By
utilizing perturbed prompts and rationales, PDSS ensures data privacy while
leveraging the predictive prowess of LLMs to enhance the performance of SLMs.

  * •

Innovative Privacy Protection Strategies. Within PDSS, we introduce two
privacy protection strategies: the Exponential Mechanism Strategy and the
Encoder-Decoder Strategy. The former utilizes an exponential mechanism to
obfuscate user prompts, while the latter employs a specialized Encoder-Decoder
SLM to encode and decode perturbed prompts and rationales. These strategies
effectively balance user privacy and the usability of rationales, allowing for
secure and enhanced training of the client’s SLM without compromising on
privacy concerns.

  * •

Empirical Evaluation and Enhanced Performance of Task-Specific SLM. Through
experiments on various text generation tasks, PDSS demonstrates the
effectiveness of its framework in training task-specific SLM with enhanced
performance. By harnessing the rationales generated by the server-side LLM,
PDSS provides valuable task-specific knowledge to the SLM, enabling them to
achieve significant improvements with the support of the LLM while
prioritizing data privacy protections.

Figure 1:  Overview of our proposed PDSS workflow.  Figure 2: Privacy-
Preserving Rationals Generation Example.

##  2 Related Work

###  2.1 Chain of Thought in Large Language Models

The Chain of Thought(COT) approach has recently garnered significant attention
in the realm of LLMs, thanks primarily to its remarkable ability to enhance
the reasoning capabilities of these models. This innovative concept was first
introduced by Wei et al. (2022). Their research demonstrated that by prompting
LLMs to produce a sequence of intermediary reasoning steps(rationales), the
models’ performance in handling intricate reasoning tasks could be notably
boosted. This groundbreaking study opened the door for further explorations
into COT. Since the introduction of COT, several studies have delved into its
extensions and variations. For example, Kojima et al. (2022) proposed the use
of zero-shot COT, where the model is prompted to generate reasoning
steps(rationales) without relying on prior examples. COT has also been applied
to various domains, including arithmetic reasoningCobbe et al. (2021),
commonsense reasoningKlein and Nabi (2020).

Nonetheless, despite the impressive feats achieved by LLMs, the adoption of
LLMs in domain-specific applications with constrained resources poses a
significant challengeFan et al. (2023) Kang et al. (2023). Recent studies by
Hsieh et al. (2023) Ho et al. (2022) Li et al. (2023), have capitalized on the
generated rationales as a form of insightful supervision to train smaller and
domain-specific models. However, previous studies have not addressed the
domain-specific data privacy issue that arises when LLMs and domain-specific
smaller models are deployed across different parties. In our work, we endeavor
to address this significant challenge.

###  2.2 Privacy Preserving LLM Inference

With the escalating popularity and widespread utilization of LLMs, privacy
concerns have taken center stage, particularly regarding user data and model
inference. Previous research efforts aimed at preserving privacy during LLM
inference have predominantly focused on several key techniques, including
differential privacy(DP) Dwork (2006), fully homomorphic encryption(FHE)
Gentry (2009), and secure multi-party computation(MPC) Yao (1986) protocols.

Numerous studies have delved into the intricacies of LLM inference leveraging
DP techniques. Notably, methods like SANTEXT+ Yue et al. (2021), CUSTEXT+ Chen
et al. (2022), TextObfuscator Zhou et al. (2023) and InferDPT Tong et al.
(2023) have harnessed differential privacy to sequentially replace sensitive
words in the text with semantically similar alternatives from a predefined
word adjacency list.

FHE and MPC techniques have also garnered attention as viable methods for
ensuring privacy during LLM inference. For instance, CipherGPT Hou et al.
(2023) proposes a secure matrix multiplication and a novel protocol for
securely computing GELU within transformer architecture using FHE and MPC
protocols to facilitate secure two-party GPT inference. Likewise, Puma Dong et
al. (2023) has adopted FHE and MPC in its transformer architecture for secure
third-party LLM inference. While FHE and MPC can be utilized for privacy-
preserving text generation tasks, their practical applications remain limited
primarily due to significant computational and communication overheads.

The advancements in privacy-preserving techniques, such as differential
privacy, FHE, and MPC, offer promising solutions to mitigate privacy risks
associated with LLM inference. However, balancing privacy and efficiency
remains a challenge that requires further exploration and refinement.

##  3 The Proposed PDSS Framework

###  3.1 Overview

In this section, we introduce PDSS, an innovative privacy-preserving framework
specifically designed for distilling step-by-step LLMs. The PDSS framework can
enhance the performance of SLMs while maintaining privacy, leveraging the
capabilities of LLM. We illustrate the PDSS in Figure 1 and describe the
associated training algorithm in Algorithm 1. The workflow of PDSS is outlined
as follows:

  1. 1.

In the client, Prompt Encoder Module perturbs these prompts before sending
them to the server-side LLM.

  2. 2.

In the server, the server-side LLM generates perturbed rationales based on
these perturbed prompts and sends them back to the client.

  3. 3.

In the client, Rationales Decoder Module decodes the perturbed rationales.

  4. 4.

In the client, Task-Specific SLM Training Module employs both the original
label data and the filter rationales data for multi-task learning.

###  3.2 Prompt Encoder Module

In the prompt encoder module, as illustrated in Figure 3, we propose two
privacy protection strategies:

  1. 1.

Exponential Mechanism Encoder Strategy. In the first strategy, we utilize an
exponential mechanism McSherry and Talwar (2007)Tong et al. (2023), which
satisfies the criteria for the $\epsilon-DP$. This strategy works by replacing
each token in the prompt with a semantically similar one sampled from either a
predetermined adjacency list or a randomly generated adjacency list, based on
exponential mechanism.

The Definition of Exponential Mechanism Tong et al. (2023). For a given
scoring function $u:X\times Y\to R$, a randomized mechanism $M(X,u,Y)$ is
$\epsilon-DP$ compliant if it satisfies:

$\displaystyle P_{r}[y|x]\propto exp(\frac{\epsilon\cdot
u(x,y)}{2\bigtriangleup u})$ (1)

where the sensitivity $\bigtriangleup u$ is defined as:

$\displaystyle\bigtriangleup u=\max_{x,x^{{}^{\prime}}\in X,y\in
Y}|u(x,y)-u(x^{{}^{\prime}},y)|$ (2)

  2. 2.

Encoder-Decoder Encoder Strategy. The tokens within a prompt differ
significantly in terms of their importance and degree of privacy. Applying a
uniform privacy budget $\epsilon$ across all tokens may not lead to the most
optimal solution. To further optimize the privacy-utility balance, we propose
an Encoder-Decoder strategy. This strategy is built upon the first exponential
mechanism. In the Encoder-Decoder strategy, we utilize an Encoder-Decoder SLM
specifically designed to encode raw prompts into perturbed prompts and
subsequently decode perturbed rationales back into their original form. This
strategy involves two training process: encoding training process and decoding
training process. In this section, we mainly focus on encoding training
process, as illustrated in Figure 3.

Initially, an encoding training process is required for the Encoder-Decoder
SLM. Formally, let’s denote a public dataset as
$P=\left\\{(p_{i},p^{\epsilon}_{i}))\right\\}^{N}_{i=1}$, where $p_{i}$
represents raw private prompt, $p^{\epsilon}_{i}$ represents perturbed prompt
generated using the first exponential mechanism with a privacy budget of
$\epsilon$. In the encoding training process, we train the Encoder-Decoder
SLM: $g_{\phi}(p_{i})\to p^{\epsilon}_{i}$. The details of encoding training
process is illustrated in Algorithm 1.

The Encoder objective can be formulated as follows:

$\displaystyle\mathcal{L}_{\text{Encoder}}(\phi;\mathcal{P})=\mathbb{E}_{(p,p^{\epsilon})\sim\mathcal{P}}\ell_{\text{CE}}(g_{\phi}(p),p^{\epsilon})$
(3)

where $\ell_{\text{CE}}$ is the cross-entropy loss.

As illustrated in Figure 2, we can observe an exemplary comparison between the
original input and its perturbed input in Step 1 and Step 2. This perturbed
prompt serves as the new, privacy-enhanced input for further processing.

By incorporating this perturbation mechanism, we ensure that the privacy of
the original prompt is preserved. This approach not only satisfies the privacy
requirements but also enables effective data utilization for downstream tasks,
striking a balance between privacy and utility.

Figure 3: Prompt Encoder Module.

###  3.3 Generating Perturbed Rationales from LLM

When the server-side LLM receives the perturbed prompt, we leverage the Chain-
of-Thought (CoT) prompting technique introduced by Wei et al. (2022) to
generate rationales from the LLM using this perturbed prompt. These generated
rationales, which are also perturbed, are then transmitted to the client. For
instance, as illustrated in Figure 2, given a perturbed prompt in the Step 2,
the LLM generates perturbed rationales in the Step 3.

###  3.4 Rationales Decoder Module

Once the client receives the perturbed rationales from the server-side LLM, it
must initiate a "decoder" process within the rationales decoder module to
decode the rationales. In rationales decoder module, as illustrated in Figure
4, we also propose two strategies correspond to the two protection strategy of
the prompt encoder module:

  1. 1.

Exponential Mechanism Decoder Strategy. In the first decoding strategy, which
corresponds to Exponential Mechanism Encoder strategy. Here, we utilize In-
Context Learning(ICL) Dong et al. (2022) Tong et al. (2023) with the SLM to
decode the perturbed rationales. we can input a sample
$x_{i}=(p,p^{p},r^{p})_{i}$ into the SLM to prompt the generation of
rationales, where $p$ represents raw private prompt, $p^{p}$ represents
perturbed prompt and $r^{p}$ represents perturbed rationales generated from
LLM. $(p^{p},r^{p})_{i}$ can be viewed as an example for SLM in ICL. This
allows the SLM to generate rationales $r_{i}$ that are aligned with the
original, unperturbed prompt.

  2. 2.

Encoder-Decoder Decoder Strategy. In the second decoding strategy, which
corresponds to Encoder-Decoder Encoder strategy. The rationales decoder module
also use the same the Encoder-Decoder SLM with Section 3.2.

Initially, a decoding training process is required for the Encoder-Decoder
SLM. Formally, let’s denote a public dataset as
$R=\left\\{(x_{i},r_{i}))\right\\}^{N}_{i=1}$, where $x_{i}$ represents an
input, where $x_{i}=(p,p^{p},r^{p})_{i}$ , $p$ represents raw private prompt,
$p^{p}$ represents perturbed prompt generated from Encoder-Decoder SLM,
$r^{p}$ represents perturbed rationales generated from LLM. $r_{i}$ represents
the raw rationale of raw prompt $p$ generated from LLM. In the decoding
training process, we train the Encoder-Decoder SLM: $g_{\phi}(x_{i})\to
r_{i}$. The details of decoding training process is illustrated in Algorithm
1.

The Decoder objective can be formulated as follows:

$\displaystyle\mathcal{L}_{\text{Decoder}}(\phi;\mathcal{R})=\mathbb{E}_{(x,r)\sim\mathcal{R}}\ell_{\text{CE}}(g_{\phi}(x),r)$
(4)

where $\mathcal{L}_{\text{Decoder}}$ is the rational decoder loss, and
$\ell_{\text{CE}}$ is the cross-entropy loss.

Subsequently, once the decoding training process of Encoder-Decoder SLM is
finished, we can input a sample $x_{i}=(p,p^{p},r^{p})_{i}$ into the SLM,
where $r^{p}$ represents perturbed rationales generated from LLM. This allows
the SLM to generate rationales $r_{i}$ that are aligned with the original,
unperturbed prompt.

We approach the training of the Encoder-Decoder SLM as a multi-task learning
problem encompassing both the encoding and decoding training processes. The
multi-task learning objective can be formulated as follows:

$\displaystyle\mathcal{L}_{1}=\alpha\mathcal{L}_{\text{Encoder}}+(1-\alpha)\mathcal{L}_{\text{Decoder}}$
(5)

where $\alpha$ is the hyperparameters that control the weight of encoder and
decoder loss.

As illustrated in Figure 2, we can observe an exemplary comparison between the
perturbed rationales from LLM and its decoded rationales from SLM in Step 3
and Step 4. It’s worth noting that although the SLM has the ability to
generate aligned rationales independently, the quality often falls short due
to its limited capabilities. By leveraging the perturbed rationales, we
effectively transfer the powerful capabilities of the server-side LLM to
enhance the Encoder-Decoder SLM, thereby improving the overall quality of the
generated rationales.

Figure 4: Rationales Decoder Module. Algorithm 1 PDSS

1:

2:$T$: total number of rounds;

3:$\mathcal{P}$: encoding training datasets;

4:$\mathcal{R}$: decoding training datasets;

5:$\mathcal{D}$: task-Spec training datasets;

6:$\eta_{\phi}$: learning rate of Encoder-Decoder SLM;

7:$\eta_{\omega}$: learning rate of Task-Specific SLM.

8:$g_{\phi}$, $f_{\omega}$.

9:$\triangleright$ Multi-Task Training for Encoder-Decoder SLM based on Public
Datasets $\mathcal{P}$ and $\mathcal{R}$.

10:for each epoch $t\in[T]$ do

11: $\phi^{t+1}\leftarrow\phi^{t}-\eta_{\phi}\bigtriangledown\mathcal{L}_{1}$.

12:end for

13:$\triangleright$ Generated $p^{p}$ using the updated Encoder.

14:$p^{p}=SLM_{Encoder}(p)$.

15:$\triangleright$ Generated perturbed rationales from LLM on the server.

16:$r^{p}=LLM(p^{p})$.

17:$\triangleright$ Decoded perturbed rationales using the updated Encoder-
Decoder SLM.

18:$r=SLM_{Decoder}(r^{p})$.

19:$\triangleright$ Multi-Task Training for Task-Specific SLM based on
Datasets $\mathcal{D}$.

20:for each epoch $t\in[T]$ do

21:
$\omega^{t+1}\leftarrow\omega^{t}-\eta_{\omega}\bigtriangledown\mathcal{L}_{2}$.

22:end for

###  3.5 Task-Specific SLM Training Module

In our work, we undertake the training of the client’s Task-Specific SLM
tailored for text generation tasks. Initially, we elaborate on the prevalent
framework for learning task-specific models. Leveraging this established
framework, we enhance it by integrating rationales produced from the
rationales decoder module into the training process. Formally, let’s denote a
dataset as $D=\left\\{(x_{i},(y_{i},r_{i}))\right\\}^{N}_{i=1}$, where $x_{i}$
represents an input, $y_{i}$ represents the associated expected output label,
and $r_{i}$ is the corresponding desired rationale.

We conceptualize learning with rationales as a multi-task learning problem, as
illustrated in Figure 5. Specifically, we train the model
$f_{\omega}(x_{i})\to(y_{i},r_{i})$ to accomplish not just the prediction of
task labels but also the generation of the corresponding rationales based on
textual inputs. This multi-task training ensures that our model not only
produces accurate predictions but also provides insightful justifications for
its decisions. By doing so, we enhance the transparency and explainability of
the model. The multi-task learning objective can be formulated as follows:

$\displaystyle\mathcal{L}_{2}=\beta\mathcal{L}_{\text{Label}}+(1-\beta)\mathcal{L}_{\text{Rationale}}$
(6)

where $\mathcal{L}_{\text{Label}}$ is the label prediction loss:

$\displaystyle\mathcal{L}_{\text{Label}}(\omega;\mathcal{D})=\mathbb{E}_{(x,y)\sim\mathcal{D}}\ell_{\text{CE}}(f_{\omega}(x),y)$
(7)

and $\mathcal{L}_{\text{Rationale}}$ is the rationale generation loss:

$\displaystyle\mathcal{L}_{\text{Rationale}}(\omega;\mathcal{D})=\mathbb{E}_{(x,r)\sim\mathcal{D}}\ell_{\text{CE}}(f_{\omega}(x),r)$
(8)

where $\ell_{\text{CE}}$ is the cross-entropy loss, $f_{\omega}(.)$ is the
Task-Specific SLM model, and $\beta$ is the hyperparameters that control the
weight of label prediction loss and rationale generation loss.

Figure 5: Task-Specific SLM Training Module.

##  4 Experiments

###  4.1 Setup

We have established a scenario to evaluate the performance of the PDSS
framework across a range of text generation tasks. This setup involves a
client-server architecture, where the client holds two downstream SLMs :an
Encoder-Decoder SLM, which specializes in encoder-decoder functionalities and
a Task-Specific SLM, tailored for specific tasks. On the server-side, we host
a LLM for more general and powerful text generation capabilities.
Specifically, we have chosen Qwen-14BBai et al. (2023) as LLM, while both SLMs
are Qwen-0.5BBai et al. (2023). Table 1 outlines the detailed configurations
of both the LLM and the SLMs.

Setting | Server | Client | Client
---|---|---|---
Model Type | LLM | Encoder-Decoder SLM | Task-Specific SLM
Model Name | Qwen-14B | Qwen-0.5B | Qwen-0.5B
Parameters(Billion) | 14 | 0.5 | 0.5
Table 1:  LLM and SLMs Setting of PDSS.

Datasets and Evaluation Metrics. We conduct a comprehensive evaluation of PDSS
on 4 QA datasets. Specifically, we include CommonsenseQA(CQA) Talmor et al.
(2018), OpenBookQA(OBQA) Mihaylov et al. (2018), BoolQ Clark et al. (2019),
ARC-EClark et al. (2018). For these datasets, we primarily use Accuracy as the
evaluation metric.

Baselines. Since we incorporate two distinct strategies in the prompt encoder
module and rationales decoder module, we denote PDSS method with the
Exponential Mechanism Strategy as PDSS-EM and PDSS method with the Encoder-
Decoder Strategy as PDSS-ED. We conduct a comparative analysis to evaluate the
performance of our PDSS framework, which comprises both PDSS-EM and PDSS-ED.

These baselines included:

  * •

FewShot-LLM, which represents the few-shot capabilities of LLM on the server;

  * •

FewShot-SLM, which represents the few-shot performance of SLM on the client;

  * •

Standalone, where the client independently fine-tunes its local model using
its own private dataset;

  * •

DSSHsieh et al. (2023), where the client fine-tunes its local model by
distilling step-by-step LLM method without privacy-preserving.

###  4.2 Overall Performance Evaluation

In this section, we undertake a comprehensive analysis of the task performance
of PDSS. We assess both the PDSS-EM and PDSS-ED methods against other
baselines on Task-Specific SLM across various privacy budgets, denoted by
$\epsilon$.

The results, as presented in Table 2, clearly illustrate that both PDSS-EM and
PDSS-ED exhibit significantly better performance when compared to FewShot-SLM
and Standalone methods. With an increase in the privacy budget $\epsilon$,
both the performance of PDSS-EM and PDSS-ED have risen notably. Furthermore,
PDSS-ED demonstrates notably superior performance compared to PDSS-EM under
the same privacy budget $\epsilon$ . Specifically, under a privacy budget of
$\epsilon=3$, PDSS-EM surpasses the Standalone method by 3.4% and 17% in the
CQA and OBQA datasets, respectively, while PDSS-ED outperforms it by 5.2% and
22.4%. Similarly, when the privacy budget is increased to $\epsilon=10$, PDSS-
EM exceeds the Standalone approach by 6.3% and 21.6% within the CQA and OBQA
datasets, respectively, and PDSS-ED beats it by 7.2% and 28.6%. Remarkably,
across all datasets evaluated, when the privacy budget is set to
$\epsilon=10$, PDSS achieves comparable performance to DSS, highlighting its
efficacy and versatility in balancing privacy and utility.

Method | CQA | OBQA | BoolQ | ARC-E
---|---|---|---|---
FewShot-LLM | 80.9 | 82.8 | 85.2 | 80.3
FewShot-SLM | 25.7 | 28.6 | 59.7 | 40.7
Standalone | 55.7 | 43.4 | 78.4 | 50.3
DSS | 59.3 | 55.1 | 80.5 | 57.6
PDSS-EM($\epsilon=1$) | 57.7 | 49.2 | 80.1 | 52.3
PDSS-EM($\epsilon=3$) | 57.6 | 50.8 | 79 | 52.6
PDSS-EM($\epsilon=5$) | 58.8 | 53.2 | 80 | 55.3
PDSS-EM($\epsilon=10$) | 59.2 | 52.8 | 80.2 | 56.2
PDSS-ED($\epsilon=1$) | 58.2 | 50.8 | 80.3 | 56.4
PDSS-ED($\epsilon=3$) | 58.6 | 53.1 | 80.2 | 56.5
PDSS-ED($\epsilon=5$) | 58.3 | 53.4 | 80.4 | 56.3
PDSS-ED($\epsilon=10$) | 59.7 | 55.8 | 80.7 | 57.9
Table 2:  We compare the performance of Task-Specific SLM trained with PDSS-EM
and PDSS-ED across different privacy budgets $\epsilon$ against the Task-
Specific SLM trained using baseline methods.

###  4.3 Reducing Training Data Evaluation

In this section, we conduct an in-depth analysis to explore the influence of
training data size on model performance. We compare the PDSS method with the
Standalone approach, varying the amount of training data used. Table 3
provides a clear illustration of how PDSS(with $\epsilon=3$) consistently
outperforms the Standalone method.

Remarkably, PDSS achieves superior performance even with significantly fewer
training samples compared to Standalone. More specifically, when trained on
merely 75% of the complete CQA, OBQA, and BoolQ datasets, both PDSS-EM and
PDSS-ED surpasses the performance of Standalone fine-tuning that has been
trained on the entirety of these datasets. Likewise, by using only 50% of the
full ARC-E dataset, PDSS-EM exceeds the results achieved by Standalone fine-
tuning on the complete dataset. Furthermore, PDSS-ED exhibits significantly
better performance than PDSS-EM across various dataset sizes (ranging from 25%
to 100%). The results indicate that PDSS is capable of extracting more
valuable information from smaller datasets, making it a promising approach in
data-scarce environments.

Task | Method | 25% | 50% | 75% | 100%
---|---|---|---|---|---
CQA | PDSS-EM | 49 | 53.5 | 56.7 | 57.6
PDSS-ED | 54.2 | 54.6 | 56.1 | 58.6
Standalone | - | - | - | 55.7
OBQA | PDSS-EM | 34.8 | 42.2 | 45.6 | 50.8
PDSS-ED | 41.4 | 43.6 | 50.6 | 53.1
Standalone | - | - | - | 44.2
BoolQ | PDSS-EM | 63 | 74 | 78.7 | 79
PDSS-ED | 72.8 | 77.6 | 79.1 | 80.2
Standalone | - | - | - | 78.4
ARC-E | PDSS-EM | 45.3 | 52.2 | 53.1 | 53.8
PDSS-ED | 48 | 49.7 | 55.9 | 56.5
Standalone | - | - | - | 50.3
Table 3:  We compare the performance of Task-Specific SLM trained with PDSS-
EM($\epsilon=3$) and PDSS-ED($\epsilon=3$) against Standalone, across a range
of dataset sizes from 25% to 100%. The ’-’ indicates a method does not apply
to the corresponding dataset sizes.

###  4.4 Perturbed Rationales Evaluation

In this section, we focus on analyzing the quality of the perturbed
rationales($r^{p}$) generated from the perturbed prompt of LLM based on PDSS-
EM and PDSS-ED methods and compare them with the rationales($r$) generated
from raw prompt of the LLM. To evaluate the similarity between $r^{p}$ and
$r$, we use TokenRatio metric. A higher TokenRatio indicates a greater degree
of similarity between the perturbed and original rationales. For more details
about TokenRatio, please refer to Appendix C.

As shown in Table 4, with an increase in the privacy budget $\epsilon$ and a
corresponding decrease in perturbation, both the TokenRatio of PDSS-EM and
PDSS-ED have risen notably. Furthermore, in most of tasks, the TokenRatio of
PDSS-ED is higher than that of PDSS-EM in the same level of privacy budget
$\epsilon$. The experimental results confirm that the TokenRatio observed in
the perturbed rationales produced by both PDSS-EM and PDSS-ED, positively
correlate with the privacy budget $\epsilon$. This suggests that as the
privacy constraints are relaxed (higher $\epsilon$ values), the perturbed
rationales become more similar to the original rationales. This finding is
significant as it demonstrates the trade-off between privacy protection and
the utility of the generated rationales.

Method | CQA | OBQA | BoolQ | ARC-E
---|---|---|---|---
PDSS-EM($\epsilon=1$) | 19.8 | 26.2 | 26.6 | 24.6
PDSS-EM($\epsilon=3$) | 29.2 | 37.2 | 35.5 | 33.9
PDSS-EM($\epsilon=5$) | 48.8 | 59.6 | 55.2 | 53.9
PDSS-EM($\epsilon=10$) | 69.7 | 72 | 74.6 | 68.2
PDSS-ED($\epsilon=1$) | 26.7 | 33.1 | 29.7 | 31
PDSS-ED($\epsilon=3$) | 33.1 | 40.9 | 40.4 | 42.9
PDSS-ED($\epsilon=5$) | 49.6 | 61 | 57.5 | 63.5
PDSS-ED($\epsilon=10$) | 57.2 | 68.3 | 68 | 74.2
Table 4: We conduct a comparative analysis to assess the perturbed rationales
produced by PDSS-EM and PDSS-ED methods against the original, unperturbed
(raw) rationales that are directly generated from the raw prompt of the LLM.

###  4.5 Decoded Rationales Evaluation

In this section, we delve into the quality analysis of the decoded rationales
produced by the rationales decoder module based on PDSS-EM and PDSS-ED
methods. We compare these decoded rationales against those generated directly
from raw prompt of the LLM. We utilize the TokenRatio metric to assess their
similarities.

As shown in Table 5, in contrast to FewShot-SLM, it becomes apparent that the
decoded rationales’ quality based on PDSS-EM and PDSS-ED methods isn’t solely
reliant on the locally decoded SLM. The perturbed rationales crafted by the
LLM indeed fulfill their intended purpose. When juxtaposed with Table 4, it’s
clear that at comparable $\epsilon$ levels, the TokenRatio for the decoded
rationales surpass those of the perturbed rationales in the PDSS-EM and PDSS-
ED methods. This underscores the effectiveness of the rationales decoder
module in the PDSS-EM and PDSS-ED methods. Furthermore, with the increase of
the privacy budget $\epsilon$, the TokenRatio for the decoded rationales
generated by both PDSS-EM and PDSS-ED have increased significantly. This
suggests that as the privacy constraints are relaxed (higher $\epsilon$
values), the decoded rationales become more similar to the original
rationales. For more details about comparative analysis of perturbed
rationales and decoded rationales, please refer to Appendix D.

Method | CQA | OBQA | BoolQ | ARC-E
---|---|---|---|---
FewShot-SLM | 43.3 | 43.4 | 51.9 | 42.6
PDSS-EM($\epsilon=1$) | 38.3 | 37.1 | 38.4 | 41.5
PDSS-EM($\epsilon=3$) | 41.9 | 41.3 | 41.7 | 45.6
PDSS-EM($\epsilon=5$) | 53.1 | 54 | 55 | 58.3
PDSS-EM($\epsilon=10$) | 71.1 | 63 | 73.6 | 70.4
PDSS-ED($\epsilon=1$) | 57.2 | 53.4 | 45.2 | 57.5
PDSS-ED($\epsilon=3$) | 59 | 55.1 | 48 | 59.4
PDSS-ED($\epsilon=5$) | 59.8 | 59.5 | 55.7 | 65.5
PDSS-ED($\epsilon=10$) | 62 | 62.3 | 63.4 | 70.1
Table 5:  We conduct a comparative analysis to assess the decoded rationales
produced by PDSS-EM and PDSS-ED methods against the original, unperturbed
(raw) rationales that are directly generated from the raw prompt of the LLM.

##  5 Conclusions

We introduced PDSS, a privacy-preserving framework for LLM distillation,
addressing domain-specific knowledge privacy and resource constraints. PDSS
employs a server-client architecture with prompt encoding, rationale
generating, rationale decoding, and task-specific SLM training, bridging the
gap between LLM and SLM while maintaining data privacy. Experiments on various
text generation tasks demonstrate PDSS’s ability to enhance SLM performance
with LLM support while prioritizing data privacy.

## Limitations

Our current study faces limitations due to computational and storage
constraints, which hinder our ability to experiment with larger model sizes.
Additionally, our evaluation of PDSS has been restricted to the Qwen model
architecture, leaving the possibility that PDSS may need to be further
explored in other model architectures. We intend to tackle these issues in
future research endeavors.

## References

  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Chen et al. (2022) Huimin Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, and Jamie Cui. 2022.  A customized text sanitization mechanism with differential privacy.  _arXiv preprint arXiv:2207.01193_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _arXiv preprint arXiv:1905.10044_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey on in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Dong et al. (2023) Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Cheng. 2023.  Puma: Secure inference of llama-7b in five minutes.  _arXiv preprint arXiv:2307.12533_.
  * Dwork (2006) Cynthia Dwork. 2006.  Differential privacy.  In _International colloquium on automata, languages, and programming_ , pages 1–12. Springer.
  * Fan et al. (2023) Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023.  Fate-llm: A industrial grade federated learning framework for large language models.  _arXiv preprint arXiv:2310.10049_.
  * Gentry (2009) Craig Gentry. 2009.  _A fully homomorphic encryption scheme_.  Stanford university.
  * Ho et al. (2022) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_.
  * Hou et al. (2023) Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. 2023.  Ciphergpt: Secure two-party gpt inference.  _Cryptology ePrint Archive_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Kang et al. (2023) Yan Kang, Tao Fan, Hanlin Gu, Lixin Fan, and Qiang Yang. 2023.  Grounding foundation models through federated transfer learning: A general framework.  _arXiv preprint arXiv:2311.17431_.
  * Klein and Nabi (2020) Tassilo Klein and Moin Nabi. 2020.  Contrastive self-supervised learning for commonsense reasoning.  _arXiv preprint arXiv:2005.00669_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2023) Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023.  Symbolic chain-of-thought distillation: Small models can also" think" step-by-step.  _arXiv preprint arXiv:2306.14050_.
  * McSherry and Talwar (2007) Frank McSherry and Kunal Talwar. 2007.  Mechanism design via differential privacy.  In _48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07)_ , pages 94–103. IEEE.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Tong et al. (2023) Meng Tong, Kejiang Chen, Yuang Qi, Jie Zhang, Weiming Zhang, and Nenghai Yu. 2023.  Privinfer: Privacy-preserving inference for black-box large language model.  _arXiv preprint arXiv:2310.12214_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Yao (1986) Andrew Chi-Chih Yao. 1986.  How to generate and exchange secrets.  In _27th annual symposium on foundations of computer science (Sfcs 1986)_ , pages 162–167. IEEE.
  * Yue et al. (2021) Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman SM Chow. 2021.  Differential privacy for text analytics via natural text sanitization.  _arXiv preprint arXiv:2106.01221_.
  * Zhang and Yang (2021) Yu Zhang and Qiang Yang. 2021.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12):5586–5609.
  * Zhou et al. (2023) Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Yuran Wang, Yong Ding, Yibo Zhang, Qi Zhang, and Xuan-Jing Huang. 2023.  Textobfuscator: Making pre-trained language model a privacy protector via obfuscating word representations.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5459–5473.

##  Appendix A Rationales Generation through COT

We utilize the rationales data generated by server-side LLM through chain-of-
thought (CoT)Wei et al. (2022)Hsieh et al. (2023) technique to enhance the
performance of the client’s task-specific SLM. These rationales justify the
predicted labels and serve as insightful guidance for training smaller and
domain-specific models. Consider the following example: when asked “Question:A
beaver is know for building prowess, their supplies come from where? Answer
Choices: (a) british columbia (b) body of water (c) wooded area (d) pay debts
(e) zoo”. Utilizing the chain-of-thought (CoT) technique, the LLM can generate
intermediate rationales like, "The answer must be the place where beavers get
their supplies. Of the above choices, only wooded areas have the supplies that
beavers need.” Such rationales bridge the gap between the input and the final
answer, often encapsulating valuable task-related knowledge. This knowledge
would traditionally require extensive data for smaller and task-specific
models to acquire. Therefore, we harness these rationales as enriched training
material for small language models, employing a multi-task training paradigm
that encompasses both label prediction task and rationale prediction task.

##  Appendix B More on Experimental Details

###  B.1 Hyperparameter Settings

SLM Parameters. During the training process for both the Encoder-Decoder SLM
and the Task-Specific SLM, we specifically configured the parameters. We set
the batch size to 32 and employed the AdamW optimizer. The maximum number of
training steps ranged from 400 to 1500. Additionally, we assigned the values
of 0.5 to both $\alpha$ and $\beta$. Furthermore, the learning rates for
$\eta_{\phi}$ and $\eta_{\omega}$ were established at 5e-5.

###  B.2 Data Splitting

For the datasets CQA/OBQA/BoolQ//ARC-E/, all splits (training, validation, and
test) were downloaded from HuggingFace Lhoest et al. (2021). During the
training of the Encoder-Decoder SLM, we randomly divided the training data
into two equal parts. One part was designated as the public dataset, while the
other part was allocated as the private dataset for the client.

###  B.3 Dataset Licenses

For the datasets CQA/OBQA/BoolQ//ARC-E/ were downloaded from HuggingFaceLhoest
et al. (2021) and under Apache License, Version 2.0.

###  B.4 Machine Configuration

The experiments were conducted on machines equipped with 4 Nvidia V100 32G.

##  Appendix C The Definition of TokenRatio Metric

TokenRatio($r^{{}^{\prime}},r$). This metric calculates the unique words($u$)
in $r^{{}^{\prime}}$ and counts how many of these words are also present in
$r$, denoted as $i$. The TokenRatio is then calculated as $i$ divided by the
total number of unique words in $r^{{}^{\prime}}$ ($|u|$).

Figure 6: Comparative Analysis of Perturbed Rationales and Decoded Rationales.

##  Appendix D Comparative Analysis of Perturbed Rationales and Decoded
Rationales

As shown in Figure 6, we conduct a comparison of the quality between the
perturbed rationales and the decoded rationales, employing both the PDSS-EM
and PDSS-ED methods across various privacy budgets denoted by $\epsilon$. For
clarity, we designate the perturbed rationales generated using the PDSS-EM and
PDSS-ED methods as P-PDSS-EM and P-PDSS-ED, respectively. Similarly, the
decoded rationales derived from these methods are denoted as D-PDSS-EM and
D-PDSS-ED. It’s clear that at comparable $\epsilon$ levels, the TokenRatio for
decoded rationales consistently surpasses that of perturbed rationales in most
tasks, when utilizing the PDSS-EM and PDSS-ED methods. This finding
underscores the remarkable effectiveness of the rationales decoder module
within both the PDSS-EM and PDSS-ED frameworks.