# Evaluating Privacy Leakage in Split Learning

Xinchi Qiu,1 111This work is done while Xinchi Qiu was interning at Meta,
Email<EMAIL_ADDRESS>Ilias Leontiadis 2 Luca Melis 2 Alex Sablayrolles 2
Pierre Stock 2

###### Abstract

Privacy-Preserving machine learning (PPML) can help us train and deploy models
that utilize private information. In particular, on-device machine learning
allows us to avoid sharing raw data with a third-party server during
inference. On-device models are typically less accurate when compared to their
server counterparts due to the fact that (1) they typically only rely on a
small set of on-device features and (2) they need to be small enough to run
efficiently on end-user devices. Split Learning (SL) is a promising approach
that can overcome these limitations. In SL, a large machine learning model is
divided into two parts, with the bigger part residing on the server side and a
smaller part executing on-device, aiming to incorporate the private features.
However, end-to-end training of such models requires exchanging gradients at
the cut layer, which might encode private features or labels. In this paper,
we provide insights into potential privacy risks associated with SL.
Furthermore, we also investigate the effectiveness of various mitigation
strategies. Our results indicate that the gradients significantly improve the
attacker’s effectiveness in all tested datasets reaching almost perfect
reconstruction accuracy for some features. However, a small amount of
differential privacy (DP) can effectively mitigate this risk without causing
significant training degradation.

##  1 Introduction

On-device machine learning (ML) involves training and/or deploying models
directly on the device without relying on cloud-based computing. This approach
brings several benefits to the table, including increased privacy, reduced
latency, and access to fine-grained real-time data. Such models have been
deployed for a variety of machine learning tasks, such as smart keyboard
(AbdulRahman et al. 2020), personalized assistant services (Hao 2020),
computer vision (Liu et al. 2020), healthcare (Rieke et al. 2020), and ranking
(Hejazinia et al. 2022; Hartmann et al. 2019; Paulik et al. 2021).

At the same time, on the other hand, there are certain limitations that hinder
the wide adoption of on-device AI models. Firstly, the limited computational
and memory resources of client devices also restrict the size of the deployed
models. As a result, the learning capacity and accuracy can be significantly
worse than the equivalent server-based models. Secondly, end-user devices
might have limited access to large datasets or have the capacity to store and
process features that require large embedding tables.

While on-device AI helps us ensure privacy, a key observation is that not all
features might be sensitive, user-specific, or generated on-device. Examples
include e-commerce item embeddings in a recommendation system, word embeddings
of a large language model, ads-related features, etc. As such, training a
small model entirely on-device might not be the optimal policy.

One promising approach to overcome these limitations is Split Learning (SL)
(Gupta and Raskar 2018; Vepakomma, Balla, and Raskar 2020). Typically, a large
machine learning model is divided into two parts: the bigger part resides on
the server side (typically hosted by the model owner), and a smaller part can
be executed on-device (typically hosted by the end-users). Larger models can
then be collaboratively trained on both private (client) and non-private
(server) features while limiting the information exchange between the involved
parties.

During inference, the server initiates the forward pass utilizing all the
server-side features. Large embedding tables and model architectures can be
utilized at this stage. Only the activations of the _cut layer_ are then
shared with the end devices, typically a small vector. Each device continues
the execution on its own sub-model, combining it with its own private
features. Due to the limited capabilities, the client-side model is small,
often only utilizing numerical features or categorical features with small
cardinality. In this paper, we consider the most private scenario where the
label is also private (e.g., the label represents a user purchase or a
conversion after seeing an ad).

While split learning only considers two parties (e.g., ads publishers and
advertisers), Federated Split Learning (FSL) allows us to train such models
between a central party and millions of client devices (Thapa et al. 2022). In
both cases, training the server-side model requires exchanging the gradients
at the cut layer for each sample. Consequently, the returning cut-layer
gradients might encode information that can reveal either private features
and/or labels.

This paper provides insights into the potential risks of private data leakage
during split model training. To achieve this, we introduce an extensive attack
method that aims to reconstruct private information - features or labels - by
exploiting diverse information sources at the cut layer, such as the model
parameters, the activations, and gradients at the cut layer. Through our
study, we aim to highlight the potential privacy risks associated with split
learning by studying in which way features might be more sensitive to leakage.
Finally, a significant part of the paper is devoted to studying how different
strategies can help us mitigate these risks.

To sum up, in this work:

  * •

We are the first to use an extensive and custom clustering-based attack method
on SL. This method exploits diverse information sources, such as the model
parameters, activations, and gradients at the cut layer, to reconstruct
private information, including features or labels.

  * •

We study how different mitigation strategies, such as label and gradient
differential privacy (DP), can help us protect such on-device private
features.

  * •

Our results over three different datasets indicate that the gradients
significantly improve the effectiveness of the attack methods when compared to
the baselines. In all three datasets, an attacker is able to perfectly
reconstruct labels and most features. However, adding a small amount of noise
on the gradients at the cut layer (e.g., $\sigma=0.01$) effectively mitigates
this risk with a mere $0.01$ drop in the model’s AUC.

##  2 Background and Related Work

#### Split Learning (SL)

enables collaborative training of deep learning models among multiple parties
without the need to share raw data. While Federated Learning (McMahan et al.
2017) can be utilized for such models, it may not always be practical. For
instance, e-commerce and ad-ranking models often involve numerous sparse
(categorical) and numerical features, requiring large machine learning models
that can reach sizes of hundreds of gigabytes. Such models are typically too
large to be trained on mobile devices (Qiu et al. 2022a; Horvath et al. 2021),
whereas user-side features and labels (e.g., past purchases) might be too
sensitive to collect on the server side.

In split learning (SL), the overall model is horizontally divided into two
parts. The server handles the forward/backward passes of the first and larger
portion of the model while keeping the last few layers to be trained with
sensitive user-side data/features on the device. This division point in the
model architecture is referred to as the _cut layer_. The server uses server-
side features to perform a forward pass until the cut layer and then forward
the intermediate representations to the respective clients. At each client, a
smaller architecture processes the private features, which are then combined
with the server-side representations. An overarching architecture is employed
to make the final prediction. An example illustrating this process is depicted
on the left side of Figure 1. During back-propagation, gradients are
calculated from the last layer to the cut layer in a similar manner. The
corresponding gradients are then sent back to the server to complete the
server-side back-propagation. While no raw data are exchanged, these gradients
might still encode private information that can reveal either features and/or
the ground truth labels that are stored on the devices.

Figure 1: Illustration of SL (left) and our attack (right). During training,
the server performs a forward pass until the cut later and then sends the
intermediate representations to each client. This information and private
features are used on-device to resume the computation. During the backward
pass, the partial gradients returned might encode private client features or
labels. Our attack uses these gradients to reconstruct the private features.

#### Membership inference attacks

aim at inferring whether a given sample was part of a model’s training set.
Originally proposed in (Homer et al. 2008), it was popularized by the shadow
models approach of Shokri et al. (Shokri et al. 2017). Follow-up work has
shown that the membership signal can be derived from simple quantities, such
as a low loss (Salem et al. 2018; Sablayrolles et al. 2019). Recent research
has improved the performance of such attacks by comparing the loss to a
calibrating term (Watson et al. 2021) or computing statistics on the
distribution of the loss (Carlini et al. 2022).

#### Reconstruction attacks and attribute inference

aim at reconstructing points from the training set given access to a trained
model (Carlini et al. 2019), and to partial information about the sample in
the case of attribute inference (Fredrikson et al. 2014; Yeom et al. 2018).
Carlini et al. (Carlini et al. 2021) show that given access to a trained
language model, an attacker is able to reconstruct verbatim samples with high
precision (but low recall).

#### Attacks on Federated Learning.

Recovering private features from gradients has gained growing interest in the
privacy-preserving machine learning area. A popular method called Deep Leakage
from Gradients (DLG) (Zhu, Liu, and Han 2019) has been developed to extract
training data by using the shared model gradients. An improved version of DLG,
iDLG (Zhao, Mopuri, and Bilen 2020), resulted in a more reliable approach to
extracting accurate data and perfectly reconstructing the labels. However,
these methods lack generalization on model architecture and weight
distribution initialization (Wang et al. 2020). In (Geiping et al. 2020), an
analytical approach has been developed to derive the inputs before a fully
connected (FC) layer. (Fan et al. 2020) claimed that a convolutional layer can
always be converted to an FC layer, but the gradients of the original
convolutional layer are still different from the gradients of the converted FC
layer, which impedes the data reconstruction. In (Yin et al. 2021), the
authors developed GradInversion to recover images from noise based on given
gradients. All related work assumes access to the gradient of all the
_weights_ ; in this paper, we consider attacks given access only to gradients
of the _activations_ at the cut layer.

#### Differential Privacy (DP)

One of the methods to mitigate the effectiveness of these attacks is DP (Dwork
2006). In the paper, we experiment with both DP-SGD (Abadi et al. 2016a) and
Label DP (Ghazi et al. 2021a, b). Since SL requires the device and the server
to exchange activations and gradients, it can potentially leak private label
information (Hejazinia et al. 2022; Pasquini, Ateniese, and Bernaschi 2021).
Differential privacy(Abadi et al. 2016b) constitutes a strong standard for
privacy guarantees for algorithms on aggregate databases.

######  Definition 2.1.

A randomized mechanism $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$ with
domain $\mathcal{D}$ and range $\mathcal{R}$ satisfies
$(\varepsilon,\delta)$-differential privacy if for any two adjacent inputs
$d,d^{\prime}\in\mathcal{D}$ and for any subset of outputs
$S\subseteq\mathcal{R}$ it holds that:

$\displaystyle Pr[\mathcal{M}(d)\in S]\leq
e^{\varepsilon}Pr[\mathcal{M}(d^{\prime})\in S]+\delta.$

One of the most widely adopted methods to ensure DP is through DP-SGD (Abadi
et al. 2016b), via norm clipping and adding noise ($\sigma$) to gradients. In
our case, since only the gradients of the activations at the cut layer are
shared directly from the client to the server, we only consider clipping and
adding noise to this part of the whole gradient. In addition, the
$\varepsilon$ can be estimated through DP-accountant in various packages.
There is a trade-off between model performance and model security, and
carefully tuning the noise of DP is required to ensure that the trained model
is effectively protected while maintaining a reasonable model performance
(Kairouz et al. 2021).

On the other hand, Label DP is also based on a randomized response algorithm
to improve the robustness of the training model. It is implemented when only
labels need to be protected. It operates by randomly flipping the label based
on the flipping probability ($p$) during training. The corresponding privacy
budget $\varepsilon$ can be estimated through the formula:
$p=\frac{1}{e^{\varepsilon}+1}$.

##  3 Privacy and Split Learning

In this section, we present an extensive attack method for SL. We assume that
clients have client-side private features that they would not want to share
with any third party. Also, we consider that the ground-truth labels can also
be private (i.e., only known to the clients) to study label leakage too. We
consider the attack scenario where an honest-but-curious server follows the
regular SL protocol but intends to recover clients’ private data and the
ground-truth labels based on the gradients of the cut layer. We design a
custom and extensive attack method, termed _EXACT: Exhaustive Attack for Split
Learning_. We use tabular datasets throughout the paper and experiments and
will discuss the extension and future research later.

We consider a $C$ class classification problem defined over a server feature
space $\mathcal{X}_{server}$, a client feature space $\mathcal{X}_{client}$
and a label space $\mathcal{Y}=[C]$, where $[C]=\\{1,...,C\\}$. We define
$F_{server}$ to be the server-side function, such that
$F_{server}:\mathcal{X}_{server}\rightarrow\mathbb{R}^{d}$, which outputs the
server-side activations $a_{c}$. We also define the client-side
$F_{client}:\mathcal{X}_{client}\times\mathbb{R}^{d}\rightarrow\mathcal{S}$,
which maps the client feature space and the server’s output to the probability
simplex $\mathcal{S}$,
$\mathcal{S}=\\{\textbf{z}|\sum^{L}_{i=1}z_{i}=1,z_{i}\geq 0,\forall
i\in[C]\\}$. Both $F_{server}$ and $F_{client}$ are parameterized over the
hypothesis class $w=(w_{server},w_{client})$, which is the weight of the
neural network. $\mathcal{L}(\textbf{w})$ is the loss function, and we assume
the widely used cross-entropy loss.

In this way, the server’s output ($a_{c}$) is the activation transmitted from
the server to the client at the cut layer, and the weight of the cut layer is
$w_{c}$. Also, the gradients transmitted from the client to the server are the
gradient of the cut layer activations: $\partial\mathcal{L}/\partial a_{c}$.
On the other hand, the gradients of the cut layer weight
$\partial\mathcal{L}/\partial w_{c}$ stay on the client side to finish the
back-propagation and allow the weights (weights on the clients-side, including
the cut layer) to be updated, as shown on the left side of Figure 1.

#### Threat Model:

We assume a strong attacker with access to the client-side model parameters
during training. While we want to study cases where the attacker has fine-
grained information, the attack model can be relaxed in cases where the secure
aggregation is used, and only the final model parameters are known after the
training is finished. In our scenario, we assume that an honest-but-curious
server has knowledge of the server-side features, and the server-side and
client-side models, which is a realistic assumption given the distributed
setting of both SL and FSL. As a result, we assume that an adversary can
compute the server-side outputs $a_{c}$ for any server-side feature values,
and with the input of client-side feature, the adversary can then obtain the
corresponding cut-layer gradients ($\partial\mathcal{L}/\partial a_{c}$) from
client-side. As $\partial\mathcal{L}/\partial a_{c}$ depends on the private
features, the client-side architecture, and the output of the server $a_{c}$,
we want to use all available information to reconstruct the private features
(Figure 1).

###  3.1 Attack Method

assumes that the private features on the client side are either categorical or
can be binned/clustered into a finite number of categories. We then build a
list $L$ that contains all the possible configurations of client-side private
features and labels. For a given sample, the adversary can calculate $a_{c}$
and the gradient $\partial\mathcal{L}_{i}/\partial a_{c}$ for every possible
private configuration $L[i]$. We then iterate all the configurations and then
match the gradient $\partial\mathcal{L}_{i}/\partial a_{c}$ by choosing the
configuration $L[i]$ that minimizes the distance to the true gradient
$\partial\mathcal{L}/\partial a_{c}$ returned by the client. Here, we choose
to use the L2 distance as the distance metric to compare the gradients. The
details of the algorithm can be found in Algorithm 1.

Algorithm 1 _EXACT_ : Let $f_{1},...,f_{N}$ be the client-side private
features; $C$ be the number of classes for the task; $a_{c}$ be the server
output; $x(_{1},...,x_{N},l)_{i}$ be the private features and label of the
client i; $\frac{\partial\mathcal{L}}{\partial a_{c}}$ be the gradient of cut
layer activation that connected to the server; $F_{client}$ be the model on
the client side with $p_{clt}$ be the client output; $F_{server}$ be the model
on the server side.

procedure  _EXACT_

Get $\frac{\partial\mathcal{L}}{\partial a_{c}}$ $\triangleright$ Get the
transmitted gradient

Get server output $a_{c}$ $\triangleright$ Get the server’s output

$L=f_{1}\times f_{2}\times...\times f_{N}\times L$ $\triangleright$ list of
all combinations

for $i=1,...,|L|$ do

$L[i]=(x_{1},...,x_{N},l)_{i}$ $\triangleright$ iterate the ith combination

$p_{clt}=F_{client}(a_{c},(x_{1},...,x_{N},l)_{i})$ $\triangleright$ Forward-
prop on client model

$d_{i}=|\frac{\partial\mathcal{L}_{i}}{\partial
a_{c}}-\frac{\partial\mathcal{L}}{\partial a_{c}}|$ $\triangleright$ Compute
the dist. between reconstructed and original gradient

end for

$p=\operatorname{argmin}_{i}\\{d_{i}:i=1,...,|L|\\}$ $\triangleright$ Get the
index

return $L[p]$ $\triangleright$ Return the reconstructed features and label

end procedure

Noting that in case the search space is growing with the number of features or
categories to be attacked (i.e., many private features with thousands of
categorical values), a heuristic approach or smart search methods can be used
to speed up convergence to a given configuration. However, it is also worth
mentioning that for the attack method, our priory concern is the attack
performance for evaluation of the privacy leakage rather than speed. As a
reference, in the datasets used here, we could successfully reconstruct
features of a given sample within 16.8 seconds. Also, similar to many existing
attack methods, such as DLG (Zhu, Liu, and Han 2019), iDLG (Zhao, Mopuri, and
Bilen 2020), and GradInversion (Yin et al. 2021), our method _EXACT_
reconstructs the private data via gradient matching. Unlike the previous
methods, _EXACT_ does not rely on optimization steps, which often involve
second derivatives computations or carefully tuned regularization terms. By
iterating through all possible possibilities, _EXACT_ guarantee to reconstruct
the most relevant private features without having convergence issues in the
optimization steps.

##  4 Evaluation

###  4.1 Experimental Setup

We conducted comprehensive experiments on three different datasets, and the
details of implementation can be found below. We conducted training in both SL
and FSL ways. For FSL, we simulate the federated environment by randomly
allocating 16 samples for each client. All experiments are repeated three
times.

#### Datasets:

Experiments are conducted over three datasets: _Adult Income dataset_ (Kohavi
et al. 1996), _Bank Marketing dataset_ (Moro, Laureano, and Cortez 2011), and
_Taobao ad-display/click dataset_ (Li et al. 2021). The Adult Income dataset
is a classification dataset aiming to predict whether the income exceeds 50K a
year based on census data. It contains $48,842$ and $14$ columns. The Bank
Marketing dataset is related to the direct marketing campaigns of a Portuguese
banking institution. It contains $45,211$ rows and $18$ columns ordered by
date. We also conduct our experiment over a production scale ad-display/click
dataset of Taobao (Li et al. 2021). The dataset contains $26$ million
interactions (click/non-click when an Ad was shown) and $847$ thousand items
across an 8-day period. We use $90\%$ of the dataset as the training set and
leave $10\%$ as the testing set.

For the Adult Income and the Bank dataset, we randomly partition the features
into server features and private client features. For the Taobao dataset, we
keep the user-related features as private client features. It is worth noting
that since the datasets are not pre-partitioned, our methods can work with any
feature partitioned, which is shown separately in Section 4.4. We attack all
the private features, and the particular private features for each dataset can
be found in Table 32 and 4.

#### Model Architecture

We deploy the state-of-the-art model, DeepFM, as the classification model (Li
et al. 2021; Guo et al. 2017). We use a learning rate of $0.01$ with Adagrad
and binary cross entropy as the loss function. The default number of neurons
for the DNN layer for the DeepFM is chosen to be (256,128). For the attack, we
train the models in different scenarios in the SL and FSL fashion with the
training set and then attack the private client features using the testing set
that is not seen by the model before.

#### Baseline:

We also implement two baselines to compare the attack performance. The
baselines serve as guidance to show the extra information the gradient is
leaking compared to our prior ability to reconstruct these private features
using only server-side information. The first baseline is to reconstruct the
client’s features using the server’s features. The second baseline is to use
the server’s output $a_{c}$ to reconstruct the client’s features. For both
baselines, we use the K-nearest-neighbors algorithm (KNN), which is also a
non-parametric method like our method. Since our method already outputs the
configurations of the features that match the closest to the original
gradient, there is no extra benefit of comparing with existing parametric or
optimization-based methods.

#### DP:

We implement both Label DP and DP as explained in Section 2 as mitigation and
defense strategy for our attack. For Label DP, we implement flipping
probability $p$ of both $0.1(\epsilon=2.2)$ and $0.01(\epsilon=4.6)$. For DP,
we implemented DP-SGD using clip norm $C$ as half the gradient norm according
to (Andrew et al. 2021) and noise multiplier $0.01$.

###  4.2 Results

First of all, Table 1 shows the model performance for all three datasets in
various setups and scenarios. As all three datasets are unbalanced in terms of
classes, the table reports the AUC with or without DP noise. SL and FSL
columns report the unmitigated training without DP. We also consider Label DP,
DP, and a combination of Label DP and DP for the SL training. Both methods are
explained in Section 2. SL and FSL report almost the same performance, which
is reasonable, as both trainings are following per mini-batch steps. For Label
DP, the table reports two different flipping probabilities. As we expected,
with higher probability, the performance dropped for all datasets. For DP,
since the noise added to the gradient is small, the degradation of the
performance compared to the normal split training is also minimal. It is worth
noting that the model performance (AUC), as shown in Table 1 for Taobao, is
much lower than the other datasets, due to the fact that Taobao is a
production scale and more difficult dataset to train with.

Table 1: Results (AUC) of model for each dataset on test-set in different
scenarios.

Data | SL | FSL | Label DP | DP | Label DP & DP
---|---|---|---|---|---
|  |  | $p=0.1$ | $p=0.01$ | $\sigma=0.01$ | $p,\sigma=0.01$
Bank | 0.88±0.0024 | 0.88±0.0023 | 0.87±0.0039 | 0.88±0.0025 | 0.87±0.0003 | 0.87±0.0017
Adult | 0.89±0.0033 | 0.89±0.0030 | 0.89±0.0037 | 0.89±0.0036 | 0.89±0.0034 | 0.89±0.0024
Taobao | 0.66±0.0001 | 0.66±0.0001 | 0.62±0.0007 | 0.65±0.0002 | 0.65±0.0012 | 0.65±0.0013

Then, we demonstrate the attack performance in detail in Table 2, 3, and 4.
Since all features are not balanced, we choose to report the F1 score for our
attack performance instead of accuracy.

Table 2: Results (F1 scores) of the feature reconstruction attack on test-set,
compared with the baselines on the Adult Income dataset. The number of
categories for each feature is shown in the bracket next to each feature name.
For Label in Label DP, the accuracy is reported in the bracket

Features | SL | FSL | Label DP | DP | Comb. | Baseline
---|---|---|---|---|---|---
(Num) |  |  | $p=0.1$ | $p=0.01$ |  |  | features | output
Gender(2) | 0.9977 | 0.9990 | 0.9996 | 0.9996 | 0.3652 | 0.3430 | 0.7909 | 0.7729
Race(5) | 0.9878 | 0.9888 | 0.9777 | 0.9711 | 0.1119 | 0.0808 | 0.3344 | 0.2789
Relationship (6) | 0.9952 | 0.9957 | 0.9860 | 0.9974 | 0.0828 | 0.0756 | 0.1998 | 0.2917
Marital(7) | 0.9912 | 0.9526 | 0.9794 | 0.9903 | 0.1424 | 0.1241 | 0.1736 | 0.2570
Label(2) | 1 | 1 | 0.80(0.90) | 0.98(0.99) | 0.5497 | 0.5558 | 0.3850 | 0.5234

Table 3: Results (F1 scores) of the feature reconstruction attack on test-set,
compared with the baselines on the Bank Marketing dataset. The number of
categories for each feature is shown in the bracket next to each feature name.
For Label in Label DP, the accuracy is reported in the bracket

Features | SL | FSL | Label DP | DP | Comb. | Baseline
---|---|---|---|---|---|---
(Num) |  |  | $p=0.1$ | $p=0.01$ |  |  | features | output
Martial(3) | 0.9578 | 0.9712 | 0.9877 | 0.9800 | 0.2157 | 0.3037 | 0.3229 | 0.4343
Job(12) | 0.9490 | 0.9515 | 0.9780 | 0.9632 | 0.0182 | 0.0188 | 0.0966 | 0.1697
Education(4) | 0.9499 | 0.9622 | 0.9782 | 0.9795 | 0.1898 | 0.1280 | 0.2499 | 0.2845
Housing(2) | 0.9835 | 0.9808 | 0.9941 | 0.9911 | 0.5975 | 0.5670 | 0.7112 | 0.7584
Loan(2) | 0.9332 | 0.9418 | 0.9737 | 0.9621 | 0.2656 | 0.2666 | 0.0909 | 0.1310
Contact(3) | 0.9770 | 0.9716 | 0.9886 | 0.9841 | 0.2683 | 0.3859 | 0.5406 | 0.6102
Label(2) | 1 | 1 | 0.6893 (0.90) | 0.9587(0.99) | 0.3929 | 0.3682 | 0.3504 | 0.4275

Table 4: Results (F1 scores) of the feature reconstruction attack on test-set,
compared with the baselines on the Taobao dataset. The number of categories
for each feature is shown in the bracket next to each feature name. For Label
in Label DP, the accuracy is reported in the brackets. $0$ F1 scores indicate
that the attacker reconstructs all the label as $0$ in all cases.

Features | SL | FSL | Label DP | DP | Comb. | Baseline
---|---|---|---|---|---|---
(Num) |  |  | $p=0.1$ | $p=0.01$ |  |  | features | output
Age(7) | 0.8284 | 0.8135 | 0.8470 | 0.7852 | 0.0464 | 0.0001 | 0.2643 | 0.1660
P-value(3) | 0.8880 | 0.8499 | 0.9178 | 0.8607 | 0.0256 | 0.0256 | 0.4022 | 0.3280
Shopping(3) | 0.9034 | 0.8582 | 0.9321 | 0.8583 | 0.3065 | 0.3062 | 0.4503 | 0.3281
Occupation(2) | 0.8036 | 0.8562 | 0.8931 | 0.6523 | 0.1031 | 0.0975 | 0.1151 | 0.0228
Label(2) | 1 | 1 | 0.4832(0.90) | 0.9076(0.99) | 0 | 0 | 0.0066 | 0.0326

The first thing to notice is that with both SL and FSL training for all
datasets, the label can be reconstructed perfectly, which shows the importance
of applying techniques such as label DP.

Also, for both FL and FSL, the attack performance on the private features for
both adult income and bank marketing datasets are all above $0.95$, implying
accurate reconstructed performance for the attack in the unmitigated setting.
As for the production scale dataset Taobao, some performance dropped to be
around $0.80$ but still shows good attack performance, which might be due to
the fact that the model performance for Taobao is lower than the other two
datasets.

As for Label DP, the attack accuracy is reported next to the F1 score in all
three tables, which shows that the accuracy is exactly the same as the
flipping probability. The attack performance for private features for the
Label DP case is quite similar to the performance for the standard split
training in all three datasets, showing that the flipping label does not
provide enough protection for private features.

DP, on the other hand, has much more impact on the attack performance compared
to Label DP. As we can see from Table 2, 3 and 4, the F1 scores significantly
decrease for all private features and the Labels, with some of the F1 scores
near 0, even with very small noise. For example, the attack performance for
the Label for the Taobao dataset is 0 in Table 4 because the attack
reconstructed all the labels to be label $0$. It might be due to the fact that
the dataset is extremely unbalanced, with only $5\%$ of the total sample
labeled as positive. As for the Age features in Table 4, the F1 score is
extremely low since the attack accuracy is extremely low.

In addition, the tables also demonstrate the attack performance if we combine
both Label DP and DP. As we expected, the performance for the combination will
be more similar to the DP as DP has a significant impact on the attack
performance, and all results show similar performance for the combined setup
and the DP-only setup.

Lastly, for the baselines, the reconstructed performance for the baselines
varies depending on the dataset. The baselines provide guidance to show the
lower-bound information leakage from the server side, as we might see from the
tables that the attack performance for the normal SL and training with Label
DP outperforms the baselines for all datasets. However, there is no clear way
to distinguish between the DP case and the baselines.

###  4.3 Model Architecture

We also conduct experiments to show that our results are not dependent on the
particular model size, as shown in Table 5. We vary the DNN layers on the
client side, as the gradient of the activation at the cut layer only depends
on the model architecture on the client side, to see if model architecture can
impact the attack performance. As we can see from the table, all SL training
without mitigation demonstrates consistent performance across all private
features, and the attack performances for the label are perfect. Similarly,
for the DP performance, it all degrades to a similar level across all
different model architectures indicating the effectiveness of DP on the
attack.

Table 5: Results (F1 scores) of the feature reconstruction attack on test-set
on the Adult Income dataset with various architecture on the client-side
model.

Features | SL | DP
---|---|---
(Num) | (256,128) | (64,32) | (32,16) | (256,128) | (64,32) | (32,16)
Gender(2) | 0.9977 | 0.9979 | 0.9987 | 0.3652 | 0.3330 | 0.1631
Race(5) | 0.9878 | 0.9692 | 0.9830 | 0.1119 | 0.2010 | 0.0671
Relation(6) | 0.9952 | 0.9894 | 0.9896 | 0.0828 | 0.0831 | 0.0674
Marital(7) | 0.9912 | 0.9436 | 0.9789 | 0.1424 | 0.1233 | 0.1054
Label(2) | 1 | 1 | 1 | 0.5497 | 0.4855 | 0.4554

###  4.4 Different Features

First, we conduct experiments to show that our results are not dependent on
the selection of private features. The results are shown in Table 6. It
demonstrates that our method can reconstruct the private features with
unmitigated SL regardless of the partition of private features and the number
of categories for each feature. If we only attack 1 private feature (Gender
with 2 categories or Education with 16 categories), the reconstruction
performance can be $100\%$ regardless of the number of categories of the
features. It is worth noticing that the attack performance drop below $0.5$ if
we consider the extreme case when we attack all categorical features, but it
can still perfectly attack the ground truth label. Similar to the previous
results, the attack performance degrades significantly if we incorporate DP.
Also, if we attack all 7 features with DP, the label attack returns everything
as label $0$, which outputs the $0$ F1 score, meaning that the gradient of the
cut layer yields no useful information.

In addition, we also conduct experiments to investigate the attack
effectiveness if we include non-relevant private features on the client side.
‘cms group’ is the private feature that does not contribute to the model
performance. With or without ‘cms group’, testing AUC for the model are both
$0.66$, but as we can see from Table 7, adding the non-relevant features has
an impact on the attack performance, especially for the unmitigated SL case.

Table 6: Results (F1 scores) of the feature reconstruction attack on test-set
on the Adult Income dataset with various sets of private features. Feature
’Occu’ is short for ’Occupation’, ’Work’ for ’Workclass’, and ’Edu’ for
’Education’. For the F1 score, $1$ means that it is perfectly equal to $1$ and
$1.00$ means it is rounded up to $1.00$. ’-’ means that the feature is not
considered as a private feature, so it is not attacked.

Features | SL | DP | SL | DP | SL | DP | SL | DP | SL | DP | SL | DP | SL | DP | SL | DP | SL | DP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Gender(2) | 1 | 0.36 | 1 | 0.33 | 1.00 | 0.26 | 1.00 | 0.36 | 0.97 | 0.24 | 0.90 | 0.21 | 0.81 | 0.51 | - | - | - | -
Race(3) | - | - | 1 | 0.33 | 1 | 0.08 | 1.00 | 0.12 | 0.77 | 0.10 | 0.46 | 0.09 | 0.25 | 0.15 | - | - | - | -
Marital(7) | - | - | - | - | 1.00 | 0.14 | 1.00 | 0.09 | 0.74 | 0.12 | 0.50 | 0.09 | 0.25 | 0.07 | - | - | - | -
Relation(6) | - | - | - | - | - | - | 0.99 | 0.14 | 0.92 | 0.07 | 0.70 | 0.09 | 0.52 | 0.04 | - | - | - | -
Occu.(15) | - | - | - | - | - | - | - | - | 0.83 | 0.05 | 0.62 | 0.04 | 0.39 | 0.08 | - | - | 1.00 | 0.06
Work(9) | - | - | - | - | - | - | - | - | - | - | 0.52 | 0.05 | 0.26 | 0.10 | - | - | - | -
Edu.(16) | - | - | - | - | - | - | - | - | - | - | - | - | 0.32 | 0.03 | 1 | 0.08 | 0.99 | 0.04
Label(2) | 1 | 0.62 | 1 | 0.77 | 1 | 0.55 | 1 | 0.55 | 1 | 0.39 | 1 | 0.32 | 1 | 0 | 1 | 0.62 | 1 | 0.66

Table 7: Results (F1 scores) of the feature reconstruction attack on test-set
on the Taobao dataset with private features with or without ‘cms group’. $0$
F1 scores indicate that the attacker reconstructs all the labels as $0$ in all
cases.

Features | SL | DP | SL | DP
---|---|---|---|---
(Num) |  | $\sigma=0.01$ |  | $\sigma=0.01$
CMS Group (12) | 0.0584 | 0.0001 | - | -
Age(7) | 0.6884 | 0.0119 | 0.8284 | 0.0464
P-value(3) | 0.5792 | 0.0246 | 0.8880 | 0.0256
Shopping(3) | 0.6635 | 0.3076 | 0.9034 | 0.3065
Occupation(2) | 0.3890 | 0.1032 | 0.8562 | 0.1031
Label(2) | 1 | 0 | 1 | 0

###  4.5 Majority Vote

Furthermore, we also add a variation to our attack method. Instead of choosing
the feature that derives the closest distance between the reconstructed
gradient and the original gradient, like mentioned Algorithm 1, we return the
$k$ closest combination of reconstructed features and take the majority vote
for each feature as the final reconstructed feature. Table 8 shows the results
on the Adult dataset, with different values of $k$. The results show that, as
$k$ increases, the attack performance drop for both unmitigated SL training
and training with DP, which shows that the attack performance is highly
sensitive to the variation of gradient, and it has to match exactly to
generate good reconstruction performance.

Table 8: Results (F1 scores) of the feature reconstruction attack on test-set
on the Adult Income dataset with various k-value for the majority vote.

Features | SL |  DP
---|---|---
(Num) | $k=1$ | $k=5$ | $k=10$ | $k=1$ | $k=5$ | $k=10$
Gender(2) | 0.9977 | 0.8231 | 0.7760 | 0.3652 | 0.3269 | 0.2761
Race(5) | 0.9878 | 0.2359 | 0.0543 | 0.1119 | 0.0786 | 0.0626
Relationship(6) | 0.9952 | 0.5723 | 0.3777 | 0.0828 | 0.0759 | 0.0760
Marital(7) | 0.9912 | 0.4956 | 0.5153 | 0.1424 | 0.1348 | 0.1375
Label(2) | 1 | 1 | 0.9995 | 0.5479 | 0.5351 | 0.5029

##  5 Discussion

#### Feature reconstruction:

Our experiments indicate that the gradients required for SL from the client to
the server can be used to reconstruct private features and labels. Compared to
our baseline ability to predict the private features from the public features,
the gradients significantly improve the effectiveness of the attack method in
all three datasets we experimented with.

#### Effectiveness of DP noise:

We have studied adding a wide range of DP noise, and we observe that even a
small amount is enough to mitigate the reconstruction attack while allowing
the model to reach similar training performance. This is because we only need
to add noise on the cut-layer returned gradients. Note that in the Federated
Split Learning setting, global DP is also added on the client-side models too
before releasing outside the secure aggregator to ensure that the client-side
weights do not encode any private information.

#### Studying different architectures:

While we mostly focused on a typical (fully connected) DNN architecture on the
client side, in future work, we want to examine further how different
architectures (e.g., CNNs, RNNs, etc) can affect the ability to reconstruct
private features and the sensitivity to these DP mitigation strategies.

#### Speeding up the attack:

In our attack, we need to compare the returned gradient with every possible
gradient that can result from different configurations of private features.
Obviously, the wall clock time of the search depends largely on the search
space, and it will significantly increase if the number of categories to
search over is big. For example, for our dataset, on average, we require 20
milliseconds to finish one forward and backward pass. In this case, for the
Adult dataset, each reconstruction goes over 840 times, which amounts to 16.8
seconds for each sample reconstruction. There are multiple heuristic-based
grid-search techniques that can help us accelerate this search, including
subset searching, gradient descent, Bayesian search (Chang 2021), and so on.

#### Split learning and Federated Learning:

Our initial experiments indicate that these findings also hold in Federated
Split Learning (FSL), where there is more than one client with private
features participating in training. However, in our scenarios, we randomly
allocate samples to each client, which represents the Independent and
identically data distribution (IID) scenario. We plan to study how non-IID can
cause extra difficulties either in attacking FSL (Li et al. 2020; Karimireddy
et al. 2020; Qiu et al. 2022b).

#### Extending to other attacks:

In this work, we focused on reconstruction attacks on categorical features.
However, _EXACT_ can easily be extended to membership inference attacks if we
binned/clustered private features into a finite number of categories. In this
case, our method would be able to infer if the particular feature is from a
particular cluster.

##  6 Conclusion

In this paper, we study the potential leakage of private features and labels
during split model training. We introduce a custom and extensive feature
reconstruction method and apply it to various datasets. Our results indicate
that the exchanged gradients do encode private information, allowing the
adversary to perfectly reconstruct labels and reconstruct private features
with excellent performance. We then examine how mitigation strategies such as
DP-SGD and Label DP can be used to successfully mitigate these risks without
affecting the training quality. As in this work, we focused on tabular data,
and we would like to expand on other tasks, such as image, text, audio
processing, and other model architectures in future works.

## References

  * Abadi et al. (2016a) Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H. B.; Mironov, I.; Talwar, K.; and Zhang, L. 2016a.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , 308–318.
  * Abadi et al. (2016b) Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H. B.; Mironov, I.; Talwar, K.; and Zhang, L. 2016b.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , 308–318.
  * AbdulRahman et al. (2020) AbdulRahman, S.; Tout, H.; Ould-Slimane, H.; Mourad, A.; Talhi, C.; and Guizani, M. 2020.  A survey on federated learning: The journey from centralized to distributed on-site learning and beyond.  _IEEE Internet of Things Journal_ , 8(7): 5476–5497.
  * Andrew et al. (2021) Andrew, G.; Thakkar, O.; McMahan, B.; and Ramaswamy, S. 2021.  Differentially private learning with adaptive clipping.  _Advances in Neural Information Processing Systems_ , 34: 17455–17466.
  * Carlini et al. (2022) Carlini, N.; Chien, S.; Nasr, M.; Song, S.; Terzis, A.; and Tramer, F. 2022.  Membership inference attacks from first principles.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , 1897–1914. IEEE.
  * Carlini et al. (2019) Carlini, N.; Liu, C.; Erlingsson, Ú.; Kos, J.; and Song, D. 2019.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX Security Symposium (USENIX Security 19)_ , 267–284.
  * Carlini et al. (2021) Carlini, N.; Tramer, F.; Wallace, E.; Jagielski, M.; Herbert-Voss, A.; Lee, K.; Roberts, A.; Brown, T. B.; Song, D.; Erlingsson, U.; et al. 2021.  Extracting Training Data from Large Language Models.  In _USENIX Security Symposium_ , volume 6.
  * Chang (2021) Chang, D. T. 2021.  Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax.  arXiv:1912.05686.
  * Dwork (2006) Dwork, C. 2006.  Differential privacy.  In _Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33_ , 1–12. Springer.
  * Fan et al. (2020) Fan, L.; Ng, K. W.; Ju, C.; Zhang, T.; Liu, C.; Chan, C. S.; and Yang, Q. 2020.  Rethinking privacy preserving deep learning: How to evaluate and thwart privacy attacks.  _Federated Learning: Privacy and Incentive_ , 32–50.
  * Fredrikson et al. (2014) Fredrikson, M.; Lantz, E.; Jha, S.; Lin, S.; Page, D.; and Ristenpart, T. 2014.  Privacy in pharmacogenetics: An $\\{$End-to-End$\\}$ case study of personalized warfarin dosing.  In _23rd USENIX Security Symposium (USENIX Security 14)_ , 17–32.
  * Geiping et al. (2020) Geiping, J.; Bauermeister, H.; Dröge, H.; and Moeller, M. 2020.  Inverting gradients-how easy is it to break privacy in federated learning?  _Advances in Neural Information Processing Systems_ , 33: 16937–16947.
  * Ghazi et al. (2021a) Ghazi, B.; Golowich, N.; Kumar, R.; Manurangsi, P.; and Zhang, C. 2021a.  Deep learning with label differential privacy.  _Advances in neural information processing systems_ , 34: 27131–27145.
  * Ghazi et al. (2021b) Ghazi, B.; Golowich, N.; Kumar, R.; Manurangsi, P.; and Zhang, C. 2021b.  Deep learning with label differential privacy.  _Advances in neural information processing systems_ , 34: 27131–27145.
  * Guo et al. (2017) Guo, H.; Tang, R.; Ye, Y.; Li, Z.; and He, X. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_.
  * Gupta and Raskar (2018) Gupta, O.; and Raskar, R. 2018.  Distributed learning of deep neural network over multiple agents.  _Journal of Network and Computer Applications_ , 116: 1–8.
  * Hao (2020) Hao, K. 2020.  How Apple personalizes Siri without hoovering up your data.  _Technology Review_.
  * Hartmann et al. (2019) Hartmann, F.; Suh, S.; Komarzewski, A.; Smith, T. D.; and Segall, I. 2019.  Federated learning for ranking browser history suggestions.  _arXiv preprint arXiv:1911.11807_.
  * Hejazinia et al. (2022) Hejazinia, M.; Huba, D.; Leontiadis, I.; Maeng, K.; Malek, M.; Melis, L.; Mironov, I.; Nasr, M.; Wang, K.; and Wu, C.-J. 2022.  Fel: High capacity learning for recommendation and ranking via federated ensemble learning.  _arXiv preprint arXiv:2206.03852_.
  * Homer et al. (2008) Homer, N.; Szelinger, S.; Redman, M.; Duggan, D.; Tembe, W.; Muehling, J.; Pearson, J. V.; Stephan, D. A.; Nelson, S. F.; and Craig, D. W. 2008.  Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays.  _PLoS genetics_ , 4(8): e1000167.
  * Horvath et al. (2021) Horvath, S.; Laskaridis, S.; Almeida, M.; Leontiadis, I.; Venieris, S.; and Lane, N. 2021.  Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout.  _Advances in Neural Information Processing Systems_ , 34: 12876–12889.
  * Kairouz et al. (2021) Kairouz, P.; McMahan, H. B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A. N.; Bonawitz, K.; Charles, Z.; Cormode, G.; Cummings, R.; et al. 2021.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2): 1–210.
  * Karimireddy et al. (2020) Karimireddy, S. P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A. T. 2020.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.  In _Proceedings of the 37th International Conference on Machine Learning_. PMLR.
  * Kohavi et al. (1996) Kohavi, R.; et al. 1996.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In _Kdd_ , volume 96, 202–207.
  * Li et al. (2021) Li, L.; Hong, J.; Min, S.; and Xue, Y. 2021.  A Novel CTR Prediction Model Based On DeepFM For Taobao Data.  In _2021 IEEE International Conference on Artificial Intelligence and Industrial Design (AIID)_ , 184–187. IEEE.
  * Li et al. (2020) Li, T.; Sahu, A. K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V. 2020\.  Federated optimization in heterogeneous networks.  _Proceedings of Machine Learning and Systems_ , 2: 429–450.
  * Liu et al. (2020) Liu, Y.; Huang, A.; Luo, Y.; Huang, H.; Liu, Y.; Chen, Y.; Feng, L.; Chen, T.; Yu, H.; and Yang, Q. 2020.  Fedvision: An online visual object detection platform powered by federated learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 13172–13179.
  * McMahan et al. (2017) McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_. PMLR.
  * Moro, Laureano, and Cortez (2011) Moro, S.; Laureano, R.; and Cortez, P. 2011.  Using data mining for bank direct marketing: An application of the crisp-dm methodology.
  * Pasquini, Ateniese, and Bernaschi (2021) Pasquini, D.; Ateniese, G.; and Bernaschi, M. 2021.  Unleashing the tiger: Inference attacks on split learning.  In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2113–2129.
  * Paulik et al. (2021) Paulik, M.; Seigel, M.; Mason, H.; Telaar, D.; Kluivers, J.; van Dalen, R.; Lau, C. W.; Carlson, L.; Granqvist, F.; Vandevelde, C.; et al. 2021.  Federated evaluation and tuning for on-device personalization: System design & applications.  _arXiv preprint arXiv:2102.08503_.
  * Qiu et al. (2022a) Qiu, X.; Fernandez-Marques, J.; Gusmao, P. P.; Gao, Y.; Parcollet, T.; and Lane, N. D. 2022a.  ZeroFL: Efficient on-device training for federated learning with local sparsity.  _arXiv preprint arXiv:2208.02507_.
  * Qiu et al. (2022b) Qiu, X.; Fernandez-Marques, J.; Gusmao, P. P.; Gao, Y.; Parcollet, T.; and Lane, N. D. 2022b.  Zerofl: Efficient on-device training for federated learning with local sparsity.  _arXiv preprint arXiv:2208.02507_.
  * Rieke et al. (2020) Rieke, N.; Hancox, J.; Li, W.; Milletari, F.; Roth, H. R.; Albarqouni, S.; Bakas, S.; Galtier, M. N.; Landman, B. A.; Maier-Hein, K.; et al. 2020.  The future of digital health with federated learning.  _NPJ digital medicine_ , 3(1): 119.
  * Sablayrolles et al. (2019) Sablayrolles, A.; Douze, M.; Schmid, C.; Ollivier, Y.; and Jégou, H. 2019.  White-box vs black-box: Bayes optimal strategies for membership inference.  In _International Conference on Machine Learning_ , 5558–5567. PMLR.
  * Salem et al. (2018) Salem, A.; Zhang, Y.; Humbert, M.; Berrang, P.; Fritz, M.; and Backes, M. 2018.  Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models.  _arXiv preprint arXiv:1806.01246_.
  * Shokri et al. (2017) Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , 3–18. IEEE.
  * Thapa et al. (2022) Thapa, C.; Arachchige, P. C. M.; Camtepe, S.; and Sun, L. 2022.  Splitfed: When federated learning meets split learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 8485–8493.
  * Vepakomma, Balla, and Raskar (2020) Vepakomma, P.; Balla, J.; and Raskar, R. 2020.  Splintering with distributions: A stochastic decoy scheme for private computation.  _arXiv preprint arXiv:2007.02719_.
  * Wang et al. (2020) Wang, Y.; Deng, J.; Guo, D.; Wang, C.; Meng, X.; Liu, H.; Ding, C.; and Rajasekaran, S. 2020.  Sapag: A self-adaptive privacy attack from gradients.  _arXiv preprint arXiv:2009.06228_.
  * Watson et al. (2021) Watson, L.; Guo, C.; Cormode, G.; and Sablayrolles, A. 2021.  On the importance of difficulty calibration in membership inference attacks.  _arXiv preprint arXiv:2111.08440_.
  * Yeom et al. (2018) Yeom, S.; Giacomelli, I.; Fredrikson, M.; and Jha, S. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _2018 IEEE 31st computer security foundations symposium (CSF)_ , 268–282. IEEE.
  * Yin et al. (2021) Yin, H.; Mallya, A.; Vahdat, A.; Alvarez, J. M.; Kautz, J.; and Molchanov, P. 2021\.  See through gradients: Image batch recovery via gradinversion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16337–16346.
  * Zhao, Mopuri, and Bilen (2020) Zhao, B.; Mopuri, K. R.; and Bilen, H. 2020.  idlg: Improved deep leakage from gradients.  _arXiv preprint arXiv:2001.02610_.
  * Zhu, Liu, and Han (2019) Zhu, L.; Liu, Z.; and Han, S. 2019.  Deep leakage from gradients.  _Advances in neural information processing systems_ , 32.