# Model Extraction Attacks on Split Federated Learning

Jingtao Li1, Adnan Siraj Rakin1, Xing Chen1, Li Yang1, Zhezhi He2, Deliang
Fan1, Chaitali Chakrabarti1
1 School of Electrical Computer and Energy Engineering, Arizona State
University, Tempe, AZ
2 Department of Computer Science and Engineering, Shanghai Jiao Tong
University, Shanghai
1{jingtao1, asrakin, xchen382, lyang166, dfan<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Federated Learning (FL) is a popular collaborative learning scheme involving
multiple clients and a server. FL focuses on protecting clients’ data but
turns out to be highly vulnerable to Intellectual Property (IP) threats. Since
FL periodically collects and distributes the model parameters, a free-rider
can download the latest model and thus steal model IP. Split Federated
Learning (SFL), a recent variant of FL that supports training with resource-
constrained clients, splits the model into two, giving one part of the model
to clients (client-side model), and the remaining part to the server (server-
side model). Thus SFL prevents model leakage by design. Moreover, by blocking
prediction queries, it can be made resistant to advanced IP threats such as
traditional Model Extraction (ME) attacks. While SFL is better than FL in
terms of providing IP protection, it is still vulnerable. In this paper, we
expose the vulnerability of SFL and show how malicious clients can launch ME
attacks by querying the gradient information from the server side. We propose
five variants of ME attack which differs in the gradient usage as well as in
the data assumptions. We show that under practical cases, the proposed ME
attacks work exceptionally well for SFL. For instance, when the server-side
model has five layers, our proposed ME attack can achieve over 90% accuracy
with less than 2% accuracy degradation with VGG-11 on CIFAR-10.

##  1 Introduction

Federated Learning (FL) has become increasingly popular thanks to its ability
to protect users’ data and comply with General Data Protection Regulation
policy. In FedAvg [19], which is the most representative FL scheme, clients
locally update their model copies, send them to the server which then
aggregates the model parameters and sends the aggregated model back to the
clients. Such a setting only allows model parameters to be shared with the
server, and direct data sharing is avoided. However, we notice that FL is
vulnerable to Intellectual Property (IP) threat as a malicious client can
acquire the entire model for free (Fig. 1 (a)). Considering the cost of
hosting the central server and effort in co-ordinate the training, and the
danger in using stolen model to trigger adversarial attacks [7], the lack of
model IP protection in FL is a significant issue.

(a) IP threat in Federated Learning

(b) IP threat in Split Federated Learning

Figure 1: IP threats in Federated Learning. (a) FL suffers from direct model
leakage. (b) SFL prevents direct model leakage and is resistant to existing ME
attacks by blocking prediction query.

Split Federated Learning (SFL) scheme [27] is a variant of FL for training
with resource-constrained clients, where the neural network is split into a
client-side model and a server-side model. Each client only computes the
forward/backward propagation of the smaller client-side model while the
server, which has more compute resources, computes the forward/backward
propagation of the larger server-side model. SFL follows the same model
averaging routine as FL to synchronize the model. Unlike FL which passes on
the entire model over to the clients, SFL preserves the server-side model and
prevents the model from direct leakage. Moreover, according to our
investigation, SFL is resistant to existing Model Extraction (ME) attack [28,
12] where querying a publicly accessible prediction API is needed. If SFL
protocol does not allow prediction query access, all the prior ME attacks fail
to succeed, as illustrated in Fig. 1 (b).

We intend to answer the following questions: “Is SFL resistant to ME attacks?
If not, then how can it be made resistant?” To conduct the study, we test the
IP threat resistance of SFL by attacking it. However, existing ME attacks [3,
22, 29] cannot be performed since SFL presents an unique threat model where
the access to prediction results is blocked.

In this paper, we propose five novel ME attacks specially designed under SFL’s
unique threat model. They are listed as Craft-ME, GAN-ME, GM-ME, Train-ME, and
SoftTrain-ME. These ME attacks cover different gradient usage and data
assumptions. We also consider both train-from-scratch and fine-tuning SFL
applications since ME attacks behave differently in these two cases. We
benchmark the performance of five ME attacks on SFL for these two cases and
find that when the number of layers in the server-side model ($N$) is small,
ME attacks can succeed even without any data. However when $N$ is large, ME
attacks fail badly, but SFL schemes with a large $N$ are not practical because
fewer layers in client-side model could result in clients’ data being
compromised. Thus, SFL is not inherently resistant to ME attacks. To answer
the second question, we find that using L1 regularization on client-side model
can improve SFL schemes’ resistance to ME attacks. In summary, we make the
following contributions:

  * •

We are the first to show that SFL is not immune to ME attacks. We define a
practical threat model where prediction query is blocked and attacker has
white-box assumption on client-side model and gradient query access. We
propose five novel ME attacks under this threat model. We demonstrate how the
white-box assumption on client-side model and gradients information can be
used to extract the model. To the best of our knowledge, this is the first
work that performs a comprehensive study on ME attacks in SFL.

  * •

We study the effect of size of server-side model in SFL on the success of ME,
we find that when server-side model is small, our proposed ME attacks are
successful. For a 5-layer-in-server VGG-11 SFL model, even without original
training data, attacker can derive a surrogate model with around 85% accuracy
on CIFAR-10. But when client-side model is small, we find that the ME attacks
are hard to succeed. However, the small client-side model compromises clients’
data and hence is not a good design choice.

  * •

To make SFL resistant to ME attacks, we provide a potential ME defense based
on L1 regularization and show how it reduces the ME attack performance.

##  2 Related Work

###  2.1 Model-split Learning Schemes

The key idea for model-split learning schemes is to split the model so that
part of it is processed in the client and the rest is offloaded to the server.
This idea was first proposed in [13, 26, 17] for inference tasks and extended
by [9] for split learning, a collaborative multi-client neural network
training. However, the round-robin design in [9] need clients to learn
sequentially and thus required long training time.

###  2.2 Split Federated Learning

In SFL scheme, clients process their local models in parallel and perform
periodic synchronization as in FedAvg [19]. The detailed process of SFL is
shown in Algorithm 1, where we reference to the SFL-V2 scheme [27]. We define
$N$ as the number of layers in server-side model, as the key design parameter.

At the beginning of each epoch, server performs the synchronization of client-
side model and sends the updated version to all clients. Then, clients perform
forward propagation locally till layer $L-N$ (the last layer of client-side
model), sending the intermediate activation ${\bm{A}}_{i}$ to the server (line
8). Server accepts the activation and label ${\bm{y}}_{i}$ sent from clients,
and uses them to calculate the loss and initiates the backward process (line
9). The backward process (line 10) consists of several steps: server performs
backward propagation on the loss, updates server-side model and sends back
gradient $\nabla_{{\bm{A}}_{i}}\mathcal{L}$ to clients. Clients then continue
the backward propagation on their client-side model copies and perform model
updates accordingly.

Algorithm 1 Split Federated Learning

0: For $M$ clients, instantiate private training data
(${\mathbf{X}}_{i},{\mathbf{Y}}_{i}$) for $1,2,...,M$. Server-side model $S$
has $N$ layers and client-side model $C_{i}$ has $L-N$ layers.

1: initialize $C_{i},S$

2: for epoch $t\leftarrow 1$ to num_epochs do

3: $C^{*}=\frac{1}{M}\sum_{i=1}^{M}{C_{i}}$ {Model Synchronization}

4: $C_{i}\leftarrow C^{*}$ for all $i$

5: for step $s\leftarrow 1$ to num_batches do

6: for client $i\leftarrow 1$ to $M$ in Parallel do

7: data batch (${\bm{x}}_{i},{\bm{y}}_{i}$) $\leftarrow$
(${\mathbf{X}}_{i},{\mathbf{Y}}_{i}$)

8: ${\bm{A}}_{i}=C_{i}({\bm{W}}_{C_{i}};{\bm{x}}_{i})$ {Client forward; send
${\bm{A}}_{i}$ to Server}

9: end for

10: for client $i\leftarrow 1$ to $M$ in Sequential do

11: $\mathcal{L}=\mathcal{L}_{CE}(S({\bm{W}}_{S};{\bm{A}}_{i}),{\bm{y}}_{i})$
{Server forward}

12: $\nabla_{{\bm{A}}_{i}}\mathcal{L}\leftarrow$ back-propagation {Server
backward, send $\nabla_{{\bm{A}}_{i}}\mathcal{L}$ to Client}

13: Update ${\bm{W}}_{S}$;

14: end for

15: for client $i\leftarrow 1$ to $M$ in Parallel do

16: $\nabla_{{\bm{x}}_{i}}\mathcal{L}\leftarrow$ back-propagation {Client
backward}

17: Update ${\bm{W}}_{C_{i}}$;

18: end for

19: end for

20: end for

###  2.3 Model Extraction Attack

In SFL, the model is split and so IP threat due to directly downloading the
model is non-existent. However, there exists advanced IP threats due to ME
attacks. Such attacks are first demonstrated in [28], and the follow-up work
[12] shows that high fidelity and accurate model can be obtained with very few
model prediction queries.

A successful ME attack not only breaches the model IP, but also makes the
model more vulnerable to attacks. ME attack can support transferable
adversarial attacks [7], mainly targeted ones [18] against the victim model. A
high-fidelity surrogate model also be used to perform bit-flip attacks [24];
for instance, a few bit flips on model parameters can degrade ResNet-18 model
accuracy to below 1%.

###  2.4 Data protection in SFL

Similar to FL, SFL scheme protects clients’ data by not sending it directly to
the server. However, data protection in SFL can be compromised by attacks such
as MI attacks. In model-based MI attack [5], the attacker trains an inverted
version of client-side model and can directly reconstruct raw inputs from the
intermediate activation. Recent works [30, 16] provide practical ways to
mitigate MI attacks. However, they cannot achieve satisfactory mitigation when
the client-side model has very few number of layers (less than 3 in a VGG-11
model).

(a)

(b)

(c)

Figure 2: Case study VGG-11 CIFAR-10 model with different $N$: If prediction
API access is allowed, existing ME attacks are very successful on SFL which
suggests prediction API access should be blocked. (a) ME attacks using
CIFAR-100 as auxiliary dataset; (b) Data-free ME attack that demonstrates ME
attack on part of the model is much easier than ME attack on the entire model;
(c) Inconsistent gradient problem in training-from-scratch SFL. The y-axis
denotes the change in gradient (lower means more consistent) for the same
inputs in different epochs.

##  3 Threat Model

###  3.1 Attacker Assumptions

Objective. According to [12], there are three model extraction (ME) attack
objectives: i) functional equivalence, ii) high accuracy, and iii) high
fidelity. Since achieving functional equivalence is difficult in practical
applications most of the existing practical ME attacks focus on achieving high
accuracy and fidelity. To achieve the accuracy goal, the attacker aims to
obtain a model that maximizes the prediction correctness and to achieve the
fidelity goal, the attacker aims to derive a model with a similar decision
boundary as the victim model before launching adversarial attacks [1].

Data Assumption. We assume that attackers’ data assumption can fall into three
categories: (1) noise data, (2) natural auxiliary data or (3) limited amount
of training data. Having noise data represents cases where the attacker uses
randomly generated noise data. This case can happen when the attacker
participates as a “free-rider” without contributing any data, or when the
attacker does not have a similar enough dataset. The second case is motivated
by [29] where it is shown that it is better to use random noise rather than
use a drastically different dataset. In natural auxiliary data assumption, the
attacker has an auxiliary dataset that is similar but with different labels
from the victim’s training data. For example, CIFAR-100 is such an auxiliary
dataset for CIFAR-10. Furthermore, we assume a practical case where the
attacker has only a subset of original training data.

Capabilities. We assume the attacker participate in a multi-client SFL scheme
as outlined in Fig. 1. We assume the entire model has a total of $L$ layers
(or layer-like blocks, i.e. BasicBlock in ResNet) out of which the server
processes $N$ layers. The attacker holds white-box assumption on the client-
side model (consists of $L-N$ layers), that is, it knows the exact model
architecture and parameters for those layers. The attacker holds a grey-box
assumption on the $N$-layer server-side model, that is, it knows its
architecture and loss function while the model parameters are unknown. Also,
we assume server blocks the prediction queries thus neither logits nor
prediction labels are accessible by clients during training, but server allows
gradient queries to let client-side models be updated. Based on a client’s
activation ${\bm{A}}=C({\bm{x}})$ and its label ${\bm{y}}$, gradient
information $\nabla_{\bm{A}}\mathcal{L}$ is computed and sent back to clients.

###  3.2 Analysis

Reasons to block prediction APIs. Allowing predictions APIs makes SFL
vulnerable. This is particularly so since according to the white box
assumption, SFL gives away the client-side model and so the attacker only
needs to extract the server-side model to reveal the entire model. This
results in an easier problem setting than most traditional ME attacks’
assumption. Under this easier problem setting, existing ME attacks can be very
successful if prediction APIs are not blocked. Specifically, we investigate
CopyCat CNN [3], Knockoff-random [22] and data-free ME [29]. As shown in Fig.
2 (a), with auxiliary data (CIFAR-100) and enough query budget, both attacks
derive a surrogate model with very high accuracy even for a large $N$ setting.
Moreover, attacker with noise data can also succeed with data-free ME as shown
in Fig. 2 (a). When the query budget is equal to 2 million, the data-free ME
can extract the model with high accuracy even when $N$ is equal to 5. This
justifies the reason why SFL should block prediction APIs.

Ensuring consistency of gradient query. We find Gradient consistency plays an
important role for our proposed ME attacks. For fine-tuning applications [23],
attackers get consistent gradient information from gradient query, as server-
side model parameters are frozen or updated with a very small learning rate.
However, for a training-from-scratch usage, queries to SFL model obtain
inconsistent gradient information as the server-side model drastically changes
during training. As shown in Fig. 2 (c), for the same query input, the
gradient is drastically different in different epochs.

##  4 Proposed Model Extraction Attack

(a)

(b)

Figure 3: Demonstration of proposed ME attacks: (a) ME attacks without
training data. (b) ME attacks with training data. Table 1: Model Extraction
Attack Methods in SFL

Method | Data Assumption | Prediction Query | Gradient Usage | Client-side model Usage
---|---|---|---|---
Existing MEs | Varies | Required | None/Assistive | None
Naive Baseline | Limited Training | Not Required | None | None
Craft-ME | Noise Data | Not Required | Data Crafting | Initialization
GAN-ME | Noise Data | Not Required | Data Generator | Initialization
GM-ME | Natural Auxiliary | Not Required | Gradient Matching | Initialization
Train-ME | Limited Training | Not Required | None | Initialization
SoftTrain-ME | Limited Training | Not Required | Soft Label Crafting | Initialization

Previously, [20] demonstrated that using gradients to reveal one-layer linear
transformation is trivial. Given $f({\bm{x}})={\bm{W}}^{T}{\bm{x}}$, one can
directly infer ${\bm{W}}$ from a single gradient query given that
${\bm{W}}^{T}=\nabla_{\bm{x}}f({\bm{x}})$. However, using gradient only can go
no further than one layer. [20] shows that to recover a two-layer ReLU network
of the form
$f({\bm{x}})=\sum_{n=1}^{h}g({\bm{x}})_{i}{\bm{W}}_{i}{\bm{A}}^{T}_{i}{\bm{x}}$,
where $g({\bm{x}})=\mathbb{1}$ $\\{{\bm{A}}{\bm{x}}>0\\}$, ${\bm{A}}$ is of
$\mathbb{R}^{h\times d}$ and ${\bm{W}}$ is of $\mathbb{R}^{h}$, using input
gradient can recover the absolute value of normal vectors
$|{\bm{W}}_{i}{\bm{A}}_{i}|$ for $i\in[h]$. In order to get the sign
information of ${\bm{W}}_{i}{\bm{A}}_{i}$, prediction query is required which
is not supported by SFL’s threat model.

So in this paper, we investigate approximate ME attacks that differ in the
data assumptions, gradient usage and loss choices. We propose five novel ME
attacks as shown in Fig. 3. Each row corresponds to a ME attack from start to
finish. For example, Craft-ME first queries the gradients to craft a dataset
and uses the crafted dataset to train the surrogate server-side model from
scratch using cross-entropy loss. Despite the differences, the five proposed
attack methods all train a randomly initialized surrogate server-side model
from scratch. For comparison, we also include a naive baseline that ignore
white-box assumption and gradients by directly training the entire model from
scratch. We provide the detailed requirement of traditional ME attacks, naive
ME attacks and our proposed five ME attacks in Table 1.

###  4.1 ME Attacks without Training Data

First, we proposes three attacks under weaker data assumptions when the
attacker has noise data and natural auxiliary data.

Crafting model extraction (Craft-ME). Inspired by [10], where data-label pairs
(referred to as instances) with small-loss are shown to present useful
guidance for knowledge distillation, we propose a simple method to craft
small-loss instances using gradient queries and use them to train the
surrogate model. We initialize random input ${\bm{x}}_{r}$ for every class
label $c$, and use the gradient $\nabla_{{\bm{x}}_{r}}\mathcal{L}$ to update
${\bm{x}}_{r}$. For each input, updating is repeated for a number of steps. By
varying label $c$, a collection of small-loss instances is derived during SFL
training. Then, a surrogate model is trained from scratch on these small-loss
instances.

GAN-based model extraction (GAN-ME). Recent work [29] proposes a GAN-based
approach for data-free ME. The key idea is to use a generator $G$ to
continually feed fake inputs to the victim model $V$ and surrogate model $S$,
and use confidence score matching to let $S$ approach $V$. However, the
confidence score matching needs prediction query which is not allowed in our
case. Thus, we adapt the GAN-based method to gradient-query-only case and
propose a two-step method: First, a conditional-GAN (c-GAN) model $G(z|c)$ is
initialized. The attacker trains the generator during victim model’s training
by generating fake data ${\bm{x}}_{f}$ and label $c$ and performing gradient
queries to update $G$. After the training is done, generator $G$ is used to
supply small-loss instances (${\bm{x}}_{f},c$) to train the surrogate model
(the unknown part). We observe a serious mode collapse problem during the GAN
training. So we utilize the distance-aware training introduced in [32], to
encourage the c-GAN to generate more diverse small-loss instances. In the new
method, the training of the generator is not based on a min-max game, or on
traditional GAN training. Instead, it simply trains the generator toward
minimizing cross-entropy loss. While the generator $G$ fails to generate
natural-looking inputs even upon convergence, it generates abundant small-loss
instances for every label, and divergence loss helps it generate a variety of
outputs. During SFL training, the generator adjusts to the changing server-
side model.

Gradient matching model extraction (GM-ME). Gradient matching (GM) in ME
attack has been investigated in [12, 20] and is used in combination with
prediction query to improve the extraction performance. Since, in SFL,
prediction query is not allowed, we navigate this strict threat model’s
restriction by adopting gradient matching (GM) loss. For a given label
${\bm{y}}_{i}$, GM loss has the following form:

$\mathcal{L}_{GM}=|\nabla_{{\bm{x}}_{i}}\mathcal{L}(S(C({\bm{x}}_{i})),{\bm{y}}_{i})-\nabla_{{\bm{x}}_{i}}\mathcal{L}(V(C({\bm{x}}_{i})),{\bm{y}}_{i})|^{2}_{2}$
(1)

where, ${\bm{x}}_{i}$ denote inputs, $C$ denotes client-side model, $S$ and
$V$ denotes the surrogate model and victim model, respectively. For each
input, attacker would query gradients with different label ${\bm{y}}_{i}$ to
get as much information as possible. This attack performs extremely well for
small $N$ but degrades significantly for a larger $N$. Its performance also
depends on the domain similarity between the auxiliary dataset and the victim
dataset.

###  4.2 ME Attacks with Training Data

Next we describe the strongest data assumption case, i.e., the attacker has a
subset of training data,

Training-based model extraction (Train-ME). For attackers with a subset of the
training data, derivation of an accurate surrogate model can be done using
supervised learning (through minimizing the cross entropy loss on the
available data). We call this Train-ME, similar idea is also adopted in [6] to
extract the entire model of the other party. Train-ME only relies on the
white-box assumption of the client-side model, using it to initialize the
surrogate model and does not need to use the gradient query at all.
Surprisingly, it is one of the most effective ME attacks.

Table 2: ME attack performance on SFL on fine-tuning and training-from-scratch
applications. The victim is a VGG-11 model on CIFAR-10 with 91.89% validation
accuracy. For Train, SoftTrain and Naive baseline, for the fine-tuning
setting, data assumption is 1K training data (randomly sampled), and for the
train-from-scratch setting, the number of clients is 10 and each client has 5K
training data.

Metric | $N$ | Fine-tuning | Training-from-scratch
---|---|---|---
Craft | GAN | GM | Train | SoftTrain | Naive | Craft | GAN | GM | Train | SoftTrain | Naive
Accuracy (%) | 2 | 91.64 | 91.86 | 92.02 | 92.05 | 91.99 | 49.64 | 85.99 | 85.99 | 53.06 | 90.58 | 90.31 | 72.63
5 | 83.46 | 84.93 | 80.28 | 90.82 | 90.48 | 49.64 | 35.58 | 40.03 | 12.13 | 89.86 | 87.02 | 72.63
8 | 35.48 | 18.82 | 12.45 | 70.28 | 71.32 | 49.64 | 15.34 | 17.49 | 10.88 | 78.64 | 56.78 | 72.63
Fidelity (%) | 2 | 98.23 | 98.42 | 99.87 | 99.29 | 99.10 | 50.62 | 92.37 | 89.59 | 54.63 | 99.34 | 98.87 | 72.62
5 | 86.32 | 87.49 | 84.33 | 94.84 | 94.67 | 50.62 | 41.32 | 38.72 | 11.87 | 95.40 | 89.83 | 72.62
8 | 36.11 | 18.62 | 12.63 | 71.79 | 72.45 | 50.62 | 15.63 | 17.44 | 10.67 | 80.01 | 57.78 | 72.62

Gradient-based soft label training model extraction (Soft-train-ME). If
gradient query is allowed and a subset of training data is available, the
attacker can achieve better ME attack performance compared to Train-ME. To
utilize gradients, a naive idea is to combine the GM loss with cross-entropy
loss in Train-ME. However, our initial investigation shows they are not
compatible; the cross-entropy loss term usually dominates and the GM loss even
hurts the performance. An alternative approach is to use soft label. We build
upon the method in [8] which shows that gradient information of incorrect
labels is beneficial in knowledge distillation, and use it for surrogate model
training. Specifically, for each input ${\bm{x}}_{i}$, gradients of the ground
truth label as well as incorrect labels are collected $N_{C}$ times, where
$N_{C}$ is the number of classes. For an input ${\bm{x}}_{i}$ with true label
$c$, its soft label $q_{i}^{k}$ of $k$-th ($k\neq c$) label is computed as
follows:

$q_{i}^{k}=(1-\alpha)*\frac{cos({\bm{e}}^{k},{\bm{e}}^{c})}{\sum_{m=1,m\neq
c}^{N_{C}}(cos({\bm{e}}^{m},{\bm{e}}^{c})+1)}$ (2)

where, ${\bm{e}}^{k}$ denotes flattened gradients of label $k$, $q_{i}^{k}$
denotes soft label for the k-th label $k$ and $\alpha$ is a constant
($\alpha>0.5$). The derived (${\bm{x}}_{i},{\bm{q}}_{i}$) pair is then used in
the surrogate model training in addition to the true label $c$ (which is the
only difference from the Train-ME).

##  5 Model Extraction Performance

In this section, we demonstrate the performance of the proposed ME attacks and
the baseline attack on SFL schemes. All experiments are conducted on a single
RTX-3090 GPU. We use VGG-11 which has 11 layers as the model architecture. We
vary $N$ (the the number of layers in server-side model) from 2 to 8 to
generate different SFL schemes and evaluate them on CIFAR-10.

For the SFL model training, we set the total number of epochs to 200, and use
SGD optimizer with a learning rate of 0.05 and learning rate decay (multiply
by factor of 0.2 at epochs 60, 120 and 160). We assume all clients participate
in every epoch with an equal number of training steps. We set the number of
clients to 10 which corresponds to the cross-silo case. To perform ME attacks,
the attacker uses an SGD optimizer with a learning rate of 0.02 to train the
surrogate model and we report the best accuracy and fidelity. We evaluate
accuracy of the surrogate model on the validation dataset. We use the label
agreement as fidelity, defined as the percentage of samples that the surrogate
and victim models agree with over the entire validation dataset, as in [12].
We include details of the SFL setting detail in Sec. A.1.

###  5.1 ME Attack on SFL with Fine-tuning-based Training

We first perform the proposed ME attacks on fine-tuning SFL version with
consistent gradient query. Here we use a pre-trained model and set the number
of gradient queries to 100K. On a victim VGG-11 model on CIFAR-10 dataset,
whose original accuracy is 91.89%, performance of all five ME attacks are
shown in Table 2. For each of the ME attacks, we vary hyper-parameters and
report the one that achieves the best attack performance. When $N=2$, all five
ME attacks are successful and can achieve near-optimal accuracy and fidelity
performance. However, when $N$ is large, proposed ME attacks have worse attack
performance. For Craft, GAN and GM ME, the accuracy drops to around 80% when
$N$ is 5, and sharply drops to below 40% when $N$ is 8. And for Train and
SoftTrain ME, accuracy slightly degrades when $N$ is 5, and reduces to around
70% when $N$ is 8. These results show that ME attack performance strongly
correlates with $N$. ME attack performance reduces for larger $N$ as the
extraction problem becomes harder with more unknown parameters and more
complicated input feature space.

Figure 4: ME attacks without training data under limited query budget (1K and
10K).

Limited Query Budget. Gradient queries are very important for our proposed
attacks. We verify this by limit the qeury budget (original is 100K in Table
2). For Craft, GAN and GM MEs, we lower down the number of gradient queries to
1K and 10K. Results are shown in Fig. 4. Performance of all three ME attacks
reduces significantly for a large N. But most of them (except GAN-ME) still
achieves success when N is less than 4.

Summary. We summarize our findings based on results in Table 2 and extensive
evaluation in Sec. A.2. In the fine-tuning case, we see that Craft, GAN and GM
ME attacks are successful without training data. We observe the following
interesting characteristics: Craft-ME has a steady attack performance and can
succeed even with a tight gradient query budget. GAN-ME needs a large query
budget to train the c-GAN generator towards convergence but can achieve better
accuracy and fidelity than Craft-ME for $N\leq 5$. GM-ME requires an auxiliary
dataset that is similar to the training data and when CIFAR-100 is used to
attack CIFAR-10 model, GM-ME achieves almost perfect extraction for small $N$.
However, it has slightly worse performance if MNIST or SVHN are used as
auxiliary datasets. For large $N$, the surrogate model fails to converge on
the GM loss, and its extraction performance suffers from a sharp drop. For
attacks with training data such as Train-ME and  SoftTrain-ME, both accuracy
and fidelity are much higher than attacks without training data. When $N\geq
6$, SoftTrain-ME can achieve slightly better accuracy and fidelity than Train-
ME.

###  5.2 ME Attack on Training-from-scratch SFL

Next, we investigate the proposed ME attack performance in training-from-
scratch SFL case. A good attack-time-window for gradient-based ME attacks is
at the end of training when gradients do not vary as much and the model
converges. So for Craft, GAN and GM-ME, we launch the attack at epoch 160 to
get more consistent gradients. As the model is updated by multiple clients,
the percentage of malicious clients also affects the ME attack performance. We
found that with more malicious clients, the server-side model returns more
consistent gradients to the attacker. Attack performance for three attacks are
shown in Table 2 for 10-client SFL training-from-scratch case.

Summary. We summarize our findings based on results in Table 2 and extensive
evaluations in Sec. A.3. In the training-from-scratch case, for gradient-based
attacks without training data (Craft, GAN and GM MEs), we notice significant
attack performance drop compared to the consistent query case. However they
still can succeed in attacking a small server-side model ($N=5$) with around
86% accuracy. The same trend is shown in SoftTrain-ME. However SoftTrain’s
performance drops and that makes Train-ME the most effective attack for an
attacker with training data. We find that poison effect and inconsistent
gradients contribute to the sharp drop in ME attack performance. Since
gradient-based ME attacks require the attacker (as a participant) send noisy
inputs (Craft, GAN and GM MEs) or genuine inputs with incorrect labels
(SoftTrain-ME), the model accuracy suffers from a 2-3% degradation, resulting
in a less accurate target model. The inconsistent gradients reduces the effect
of gradient query since model parameter can change rapidly. For instance, in
Craft-ME, crafted inputs that have a small loss at an earlier epoch of the
training can have a large loss in the final model because of the update of
model parameters. Hence, inconsistent gradient information results in poor
accuracy in the surrogate model. Interestingly, compared to Craft-ME, GAN-ME
is more robust to inconsistent gradients as the generator can adjust itself to
the change of server-side model, resulting in a better attack performance when
$N=5$. However, when $N$ is larger, the generator does not converge well and
its performance drops drastically. Last but not the least, GM-ME completely
fails with inconsistent gradients, even for small $N$. This implies that the
GM loss is super sensitive to inconsistent gradients and is only effective in
consistent query cases.

(a)

(b)

Figure 5: (a) ME attack performance of VGG-11 on other datasets. (b) ME attack
performance of other architectures on CIFAR-10 dataset.

###  5.3 ME Attack on SFL on other datasets and architectures

First, we perform Train-ME attack with 1K training data on different datasets
including MNIST [15], FMNSIT [31], SVHN [21] and CIFAR-100 datasets [14],
using a VGG-11 model with $N$ set to 5. As shown in Fig. 5 (a), for all
datasets except CIFAR-100, ME attack achieves accuracy very close to the
original. But for CIFAR-100, the extracted accuracy is low that is $>$20%
below the original. Additionally, we also test Train-ME performance with 2%
and 20% ImageNet training data on Mobilenet-V2. As shown in Fig. 6 (a), ME
attacks are hard to succeed due to the complexity of ImageNet dataset [4],
resulting in a high accuracy gap of 10% when $N$ is set to 2. Second, we test
Train-ME attack on different architectures including Resnet-20, Resnet-32 [11]
and Mobilenet-V2 [25] on CIFAR-10 dataset (with necessary adaptations) For
Resnet and Mobilenet family, we assign last 4 layer-blocks and 1 FC layer to
server-side model. As shown in Fig. 5 (b), with the same proportion of layers
(5 out of 11) being assigned to server-side model, ME attack is much less
effective on Resnet-20 than on VGG-11. A comparison of the performance of
Resnet-32 and Mobilenetv2 with similar proportion of layers being assigned to
server-side (5 out of 17 and 20, respectively), ME on Resnet-32 is also much
worse than on MobilenetV2.

Summary. We find complex datasets such as CIFAR-100, ImageNet tend to be more
resistant to ME attacks. Also, some architectures such as Resnet-20 and
Mobilenet-V2 are more resistant than VGG-11.

##  6 Discussion

###  6.1 Tradeoff between IP and Data protection

Our evaluation showed that ME attack performance drops with increasing $N$
(the number of layers in server-side model). Thus, a simple idea to improve
resistance to ME attack is to use a larger $N$. However this implies that the
number of layers in client-side model would be smaller, thereby undermining
clients’ data. The tradeoff between IP protection and data protection is shown
in Fig. 6 (b). We use Mean Square Error (MSE) of reconstructed images by MI
attack as a metric to represent the degree of data protection in SFL, as in
[16]; its implementation detail is included in Sec. B.1. From Fig. 6 (b), we
can see that with larger $N$, the extracted accuracy decreases but MSE
decreases meaning MI attack is more successful and clients’ data protection is
compromised. Thus, IP threat mitigation cannot be simply done by increasing
$N$.

(a)

(b)

Figure 6: (a) ME attack performance of MobilenetV2 on ImageNet. (b) Tradeoff
between ME resistance and degree of data protection (MSE).

###  6.2 Potential Defenses

Next we demonstrate how simple regularization can be used as defensive methods
against ME attacks on SFL. The key idea is to restrict the useful information
in the client-side model that is leaked to the attack. This is done by
applying regularization techniques to restrict the client-side model’s feature
extraction capabilities. Specifically, on the client-side model, we apply L1
regularization with three different strength ($\lambda$ = 5e-5, 1e-4 and 2e-4)
to penalize its weight magnitude. As shown in Fig. 7 for Train-ME with 1K
data, this simple defense effectively improves the resistance to ME attack
though there is some accuracy degradation on the original model. For example,
accuracy degrades to 90.43% when $\lambda$ = 5e-5. Thus regularization can be
used to defend ME attacks. More details are provided in Sec. A.4.

Figure 7: L1 regularization as effective defense for ME attacks.

###  6.3 Ablation Study

ME attack with non-IID data. We consider the non-IID (independent and
identically distributed) case where the attacker only has data from $C$
classes of CIFAR-10. Results presented in Sec. A.5 show that ME attack
performance is still good for $C=5$ but degrades sharply when $C=2$.

Adversarial attack based on successful ME attack. The goal of ME attack is to
launch more successful adversarial attacks. We perform transfer adversarial
attacks using a surrogate model extracted by the strongest Train-ME attack. As
shown in Sec. A.6, SFL with proper $N$ achieves better resistance to
adversarial attacks.

ME attack without architecture information. In Sec. A.7, we investigate simple
variants (longer, shorter, wider, and thinner) of the original architecture as
the surrogate model architecture. We find that the performance of ME attacks
is similar for the different architectures – the exception is GM-ME which
fails for different surrogate architectures.

##  7 Conclusion

In this work, we show that SFL cannot guarantee model IP protection and is
vulnerable to ME attacks. We propose five novel ME attack methods and achieve
attack success under an unique threat model where gradient query is allowed
but prediction query is not allowed. By studying the effect of the model split
sizes on ME attack performance, we find that using a large number of layers in
server-side model can better protect IP, but compromises clients’ data and is
hence impractical. Finally, as a first step towards making SFL resistant to ME
attack, we show use of regularization as a potential defense mechanism.

## References

  * [1] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In Joint European conference on machine learning and knowledge discovery in databases, pages 387–402. Springer, 2013.
  * [2] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar.  Leaf: A benchmark for federated settings.  arXiv preprint arXiv:1812.01097, 2018.
  * [3] Jacson Rodrigues Correia-Silva, Rodrigo F Berriel, Claudine Badue, Alberto F de Souza, and Thiago Oliveira-Santos.  Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [5] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322–1333, 2015.
  * [6] Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang.  Label inference attacks against vertical federated learning.  In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA, 2022.
  * [7] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [8] Jindong Gu, Zhiliang Wu, and Volker Tresp.  Introspective learning by distilling knowledge from online self-explanation.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [9] Otkrist Gupta and Ramesh Raskar.  Distributed learning of deep neural network over multiple agents.  Journal of Network and Computer Applications, 116:1–8, 2018.
  * [10] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  Advances in neural information processing systems, 31, 2018.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [12] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot.  High accuracy and high fidelity extraction of neural networks.  In 29th USENIX Security Symposium (USENIX Security 20), pages 1345–1362, 2020.
  * [13] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang.  Neurosurgeon: Collaborative intelligence between the cloud and mobile edge.  ACM SIGARCH Computer Architecture News, 45(1):615–629, 2017.
  * [14] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [15] Yann LeCun.  The MNIST database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [16] Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti.  Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning, 2022.
  * [17] Peng Liu, Bozhao Qi, and Suman Banerjee.  Edgeeye: An edge service framework for real-time intelligent video analytics.  In Proceedings of the 1st international workshop on edge systems, analytics and networking, pages 1–6, 2018.
  * [18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [19] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  * [20] Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt.  Model reconstruction from model explanations.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 1–9, 2019.
  * [21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [22] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Knockoff nets: Stealing functionality of black-box models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4954–4963, 2019.
  * [23] Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye.  Federated split task-agnostic vision transformer for covid-19 cxr diagnosis.  Advances in Neural Information Processing Systems, 34, 2021.
  * [24] Adnan Siraj Rakin, Zhezhi He, and Deliang Fan.  Bit-flip attack: Crushing neural network with progressive bit search.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1211–1220, 2019.
  * [25] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [26] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung.  Distributed deep neural networks over the cloud, the edge and end devices.  In 2017 IEEE 37th international conference on distributed computing systems (ICDCS), pages 328–339. IEEE, 2017.
  * [27] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun.  Splitfed: When federated learning meets split learning.  arXiv preprint arXiv:2004.12088, 2020.
  * [28] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart.  Stealing machine learning models via prediction apis.  In 25th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 16), pages 601–618, 2016.
  * [29] Jean-Baptiste Truong, Pratyush Maini, Robert J Walls, and Nicolas Papernot.  Data-free model extraction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4771–4780, 2021.
  * [30] Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar.  Nopeek: Information leakage reduction to share activations in distributed deep learning.  In 2020 International Conference on Data Mining Workshops (ICDMW), pages 933–942. IEEE, 2020.
  * [31] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [32] Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee.  Diversity-sensitive conditional generative adversarial networks.  arXiv preprint arXiv:1901.09024, 2019.

Supplementary Material

##  Appendix A Details of Experimental Settings and Additional Results

###  A.1 Detailed Extraction Attack Setting

For the surrogate model training, we use SGD optimizer with a learning rate of
0.02 for 200 epochs. The learning rate is multiplied by a factor of 0.2 at
epochs 60, 120 and 160. For Craft-ME, we craft an equal number of small-loss
instances for each class. We use the Adam optimizer with a learning rate of
0.1 and set the total number of steps (iterations) to 20 or 50 to craft each
image. For GAN-ME, we use a conditional-GAN model as the generator, its
detailed architecture is given in Appendix A.10. For the generator training,
we use Adam optimizer with a learning rate of 1e-4 and apply the divergence-
aware regularization [32] with a factor of 50 to mitigate the mode collapse
problem. For GM-ME, we query with the entire training dataset (i.e. 50K) of
each auxiliary dataset (CIFAR-10, SVHN and MNIST). For Train-ME and SoftTrain-
ME, we apply standard data augmentation techniques including random rotating
and horizontal flipping, during the surrogate model training. For SoftTrain-
ME, we train the surrogate model with both the hard labels and gradient-based
soft labels with $\alpha$ parameter of 0.9. Data augmentation is disabled if
training uses soft labels.

###  A.2 Gradient-based Attack Performance with Consistent Gradient Access

We did extensive experiments for ME attacks in different settings with
consistent gradient query access and present the results here. These are in
addition to what was presented in Section 5.1. The query budget is set at 1K,
10K and 100K. Results for all five ME attacks with different settings are
shown separately in Fig. 8 (a), (b), (c), (d) and (e). The victim VGG-11 model
has 91.89% validation accuracy on CIFAR-10 dataset. For GM-ME, we CIFAR-100,
SVHN, and MNIST as the auxiliary dataset.

Conclusion. We observe all ME attacks are equally successful for small $N$.
Among different settings, Craft-ME performs better with 20 steps compared to
50 steps. This is possibly because for the same query budget, fewer steps
results in more images being crafted. GAN-ME performance is much better with a
larger query budget since a generator model needs more iterations of training
to converge. GM-ME’s performance heavily depends on the similarity of the
auxiliary dataset. Because the victim model is on CIFAR-10, it performs well
when CIFAR-100 is set as the auxiliary dataset while performing badly when
MNIST is used. Moreover, attacks with training data perform much better than
ME attacks without training data. Compared to Train-ME, SoftTrain-ME achieves
better accuracy and fidelity when $N\geq 6$.

(a)

(b)

(c)

(d)

(e)

Figure 8: Additional results for fine-tuning SFL case. Top row: ME attacks
without training data with different settings. (a) Craft-ME with different
number of crafting steps and query budgets. (b) GAN-ME with different query
budgets. (c) GM-ME with different auxiliary datasets and query budgets. Bottom
row: ME attacks with training data with different settings. (d) Train-ME with
1K/10K training data. (e) SoftTrain-ME with 1K/10K training data with
different query budgets.

###  A.3 Gradient-based Attack Performance with Inconsistent Gradient Access

Here, we provide results for ME attacks for training-from-scratch settings
with inconsistent gradient query access; a subset of these results was
presented in Section 5.2. We launch the attack by feeding malicious inputs at
late epochs, specifically, epochs 120 or 160, for the case when the number of
training epochs is 200. The attacker starts to collect gradients after the
attack is launched till the the end of training (epoch 200). We start the
gradient collection from later epochs since by then the model has achieved
near-optimal accuracy and hence is valuable as an attack target. Also the
model updating is slower because of application of learning rate decay to make
the gradients more consistent.

In multi-client SFL, the original 50K training data is divided equally to 5 or
10 benign clients, denoted as “5-client” and “10-client” case, respectively.
The attacker is an additional client without training data so a 5-client SFL
really has 6 clients (5 benign clients and 1 malicious client). All clients,
including the attacker, perform an equal number of queries in each epoch. The
performance of five ME attacks with inconsistent gradient queries are shown
separately in Fig. 9 (a), (b), (c), (d) and (e). Because of the poisoning
effect, the final model accuracy of the victim model is reduced by 2 $\sim$
3%. For GM-ME, we use CIFAR-100 as the auxiliary dataset, and we only use the
latest gradients to perform gradient matching instead of using all collected
gradients. We use “late50” to denote only gradients collected in 50 latest
training steps are used. This restriction greatly reduces the number of
gradients being available but makes them much more consistent.

Conclusion. Attacks without training data (Craft, GAN, GM MEs) work poorly
with inconsistent gradient queries. For Craft-ME, taking 20 steps also seems
to work better in both 5-client and 10-client cases. Collecting gradients
starting later at epoch 160 gets better performance than starting early at
epoch 120 because of more consistent gradients. The starting-later rule also
holds for GAN-ME and GM-ME, where we can see starting later achieves
consistently better ME attack performance. For GM-ME, it only gets meaningful
accuracy if only the latest gradients (within 10 training steps to the end of
training) are used, showing that it is extremely sensitive to gradient
consistency. For attacks with training data, we notice Train-ME attack
performance is not affected because it does not rely on gradients. However,
SoftTrain-ME performs much worse because of the poisoning effect and
inconsistent gradients.

(a)

(b)

(c)

(d)

(e)

Figure 9: Additional results for training-from-scratch SFL case. Top row: ME
attacks without training data with different settings. (a) Craft-ME with
different steps and starting epochs. (b) GAN-ME with different starting
epochs. (c) GM-ME uses the latest gradients with different restrictions.
Bottom row: ME attacks with training data with different settings. (d) Train-
ME with 10K/5K training data. (e) SoftTrain-ME with 10K/5K training data with
different starting epochs.

###  A.4 Accuracy Impact of Defensive Methods

We provide original accuracy, ME attack performance, as well as model
inversion attack performance in addition to what was presented in the main
paper in section 6. As shown in Table 3, L1 regularization works well for
$N=5$ where it reduces extraction performance a lot while slightly affecting
original accuracy, and at the same time also improves the resistance to model
inversion attack (better data protection).

Table 3: Detailed defensive performance of L1 regularization (L1Reg) of VGG-11
model on CIFAR-10. Extraction performance of Train-ME with 1K training data is
shown. Resistance to model inversion attack is shown by MSE.

Regularization | Strength | N=4 | N=5
---|---|---|---
Orig. Accu. | Accuracy | Fidelity | MSE | Orig. Accu. | Accuracy | Fidelity | MSE
None | 0.0 | 91.45 | 91.02 | 96.94 | 0.0217 | 91.71 | 90.23 | 94.73 | 0.0114
L1Reg | 5e-5 | 90.66 | 89.44 | 95.03 | 0.0274 | 90.43 | 87.45 | 91.10 | 0.0270
L1Reg | 1e-4 | 87.90 | 86.24 | 93.68 | 0.0280 | 88.37 | 82.18 | 86.01 | 0.0239
L1Reg | 2e-4 | 82.96 | 80.56 | 89.98 | 0.0262 | 85.00 | 76.45 | 80.78 | 0.0145

###  A.5 Extraction Performance under Non-IID data distribution

We demonstrate the ME attack performance in a Non-IID setting, where the
attacker only has access to training data from a subset of classes ($C$). The
new set of results corresponding to Train-ME attack are shown in Table 4. We
observe for $C$ smaller than 5, attacker performance degrades badly for both
CIFAR-10 and CIFAR-100 datasets.

Table 4: Under Non-IID data distribution, model extraction performance of
Train-ME attack on VGG-11 model on CIFAR-10 and CIFAR-100 dataset with
Original Accuracy of 91.89% and 68.64%, respectively.

Method | CIFAR-10 Accuracy | CIFAR-100 Accuracy
---|---|---
N=2 | N=5 | N=8 | N=2 | N=5 | N=8
C = 1 | 47.58 | 46.75 | 38.42 | 6.79 | 6.79 | 6.13
C = 2 | 82.45 | 79.30 | 58.69 | 13.18 | 13.36 | 11.25
C = 5 | 91.70 | 88.90 | 65.70 | 32.77 | 29.65 | 17.90

###  A.6 Adversarial Attack Performance

We demonstrate that with proper model IP protection, adversarial attacks can
be mitigated. We assume the attacker uses the strongest Train-ME attack (with
1K data) to obtain a high-fidelity surrogate model to perform transfer
adversarial attacks on the victim model with different IP protection strengths
(SFL with different $N$). We use FGSM [7], and targeted-PGD attack [18] to
perform the transfer adversarial attack. We set the $e$ for FGSM at 0.1, and
PGD-target at 0.002 for 50 iterations (the attacker randomly chooses the
original and target label). We report the average Attack Success Rate (ASR) -
the percentage of samples that are transferred successfully - to show the
attacking performance. The new set of results is shown in Table 5. We see that
both adversarial attacks achieve very high ASR for small $N$, where model IP
protection is weak. On a SFL scheme with large $N$, adversarial attack
performance degrades significantly using the surrogate model with less
fidelity.

Table 5: Adversarial Attack ASR performance based on the surrogate model
obtained using Train-ME attack, on VGG-11 model on CIFAR-10 with different $N$
setting.

Attack |  Number of Server-side Layer ($N$)
---|---
N=2 | N=3 | N=4 | N=5 | N=6 | N=7 | N=8
FGSM | 82.7 | 82.3 | 77.9 | 77.3 | 63.1 | 56.9 | 37.7
PGD-target | 100 | 100 | 99.8 | 100 | 99.5 | 73.4 | 34.2

###  A.7 Surrogate Architecture Performance

To investigate the impact on model extraction attacks caused by the surrogate
model’s architecture difference, we designed four variants of the true server-
side model, and used them as surrogate model architecture to perform model
extraction attacks. We fixed the settings to $N=5$ SFL and consistent gradient
query budget to 10K. The new set of results are shown in Table 6 for VGG-11
model on CIFAR-10 dataset. For most attacks, architecture does not make a huge
difference, and longer or wider surrogate architecture can achieve even better
accuracy and fidelity. The exception is GM-ME, which achieves much higher
extraction performance with the surrogate model having the same architecture.

Longer Architecture. Surrogate model has one extra fully connected layer
compared to the original true server-side model.

Shorter Architecture. Surrogate model has one less fully connected layer
compared to the original true server-side model.

Wider Architecture. Surrogate model has channel size that is 2 times of the
original true server-side model

Thinner Architecture. Surrogate model has channel size half of the original
channel size of the true server-side model.

Table 6: Extraction attack performance on surrogate models having slightly
different architectures from the true architecture of the server-side model.
$N$ is fixed at 5, gradients are consistent and the query budget is 10K.

Attacks | Accuracy (%) | Fidelity (%)
---|---|---
same | longer | shorter | wider | thinner | same | longer | shorter | wider | thinner
Craft-ME | 76.67 | 75.05 | 77.90 | 79.00 | 74.86 | 78.38 | 76.70 | 79.72 | 81.04 | 74.74
GAN-ME | 80.57 | 75.95 | 76.66 | 74.27 | 65.69 | 82.66 | 78.13 | 78.54 | 76.11 | 67.58
GM-ME | 65.77 | 11.41 | 18.04 | 14.77 | 14.42 | 69.60 | 11.22 | 18.61 | 14.90 | 14.35
Train-ME | 90.82 | 90.33 | 90.76 | 90.72 | 90.10 | 94.84 | 94.47 | 94.84 | 94.79 | 93.94
SoftTrain-ME | 90.57 | 90.43 | 90.66 | 90.62 | 90.10 | 94.76 | 94.62 | 94.84 | 94.59 | 94.13

Table 7: Model extraction performance of gradient-based ME attacks with
consistent gradient query (100K query budget) and inconsistent gradient query
for 10-client SFL on VGG-11 model CIFAR-100 dataset. Original Accuracy is
68.64%. We use 20 crafting steps for the Craft-ME for both cases. For the
inconsistent case, we launch ME attack at epoch 160, and use the “late10”
setting for GM-ME.

Case | Method | Accuracy (%) | Fidelity (%)
---|---|---|---
N=2 | N=3 | N=4 | N=5 | N=2 | N=3 | N=4 | N=5
Fine-tuning | Craft-ME | 66.44 | 64.68 | 35.37 | 15.4 | 86.97 | 81.37 | 40.35 | 16.7
GAN-ME | 56.54 | 46.53 | 13.11 | 6.69 | 69.91 | 55.56 | 14.86 | 7.11
GM-ME | 68.76 | 68.4 | 57.87 | 1.28 | 99.11 | 94.46 | 71.5 | 1.26
Train-from-scratch | Craft-ME | 11.53 | 8.49 | 2.61 | 2.41 | 13.67 | 10.15 | 2.71 | 2.45
GAN-ME | 49.4 | 41.9 | 22.1 | 10.75 | 60.04 | 49.29 | 25.46 | 12.55
GM-ME | 4.05 | 1.47 | 1.37 | 1.23 | 4.92 | 1.79 | 1.54 | 1.19

Table 8: Model extraction performance of gradient-based ME attacks with
consistent gradient query (100K query budget) on VGG-11 model FEMNIST dataset.
Original Accuracy is 74.62%. We use 50 crafting steps for the Craft-ME for
both cases.

Case | Method | Accuracy (%) | Fidelity (%)
---|---|---|---
N=2 | N=3 | N=4 | N=5 | N=2 | N=3 | N=4 | N=5
Fine-tuning | Craft-ME | 53.20 | 43.57 | 43.04 | 40.50 | 59.28 | 48.10 | 46.58 | 42.11
GAN-ME | 10.59 | 7.19 | 5.27 | 4.02 | 11.39 | 7.35 | 5.20 | 3.80
GM-ME | 56.67 | 22.53 | 9.87 | 3.78 | 69.70 | 25.14 | 10.10 | 3.68
Train-ME | 70.32 | 68.47 | 68.80 | 67.70 | 82.56 | 77.52 | 75.61 | 71.97
SoftTrain-ME | 75.70 | 74.93 | 74.42 | 74.46 | 83.87 | 81.24 | 77.39 | 76.30

Table 9: Model extraction performance of gradient-based ME attacks with
consistent gradient query (100K query budget) and inconsistent gradient query
for 10-client SFL on MobilenetV2 model CIFAR-10 dataset. Original Accuracy is
93.82%. We use 20 crafting steps for the Craft-ME for both cases. For the
inconsistent case, we launch ME attack at epoch 160, and use the “late10”
setting for GM-ME.

Case | Method | Accuracy (%) | Fidelity (%)
---|---|---|---
N=2 | N=3 | N=4 | N=5 | N=2 | N=3 | N=4 | N=5
Fine-tuning | Craft-ME | 92.29 | 76.74 | 72.74 | 61.08 | 96.04 | 77.86 | 73.24 | 61.46
GAN-ME | 92.67 | 79.17 | 68.92 | 57.61 | 96.46 | 80.35 | 69.7 | 58.25
GM-ME | 93.2 | 92.82 | 92.39 | 91.86 | 97.83 | 96.87 | 95.74 | 94.74
Train-from-scratch | Craft-ME | 78.55 | 63.04 | 61.77 | 58.76 | 80.8 | 64.87 | 63.35 | 60.07
GAN-ME | 77.08 | 38.2 | 35.08 | 32.49 | 79.43 | 38.87 | 35.6 | 33.11
GM-ME | 31.23 | 11.25 | 15.25 | 17.81 | 32.73 | 11.46 | 15.14 | 17.9

###  A.8 Other Empirical Results

In this section, we present more empirical results for ME attacks without
training data to show that our claims can generalize to other architecture and
datasets. The list of experiments are:

  * •

1\. ME attack performance (without training data only) of VGG-11 on CIFAR-100
(Table 7). An interesting observation is GAN-ME performs worse than Craft-ME
for consistent gradient cases for the increasing number of classes (100) makes
the generator even harder to converge. While for the inconsistent gradient
case, GAN-ME performs much better than Craft-ME because its generator can
adapt to the inconsistent gradients and Craft-ME cannot.

  * •

2\. ME attack Performance (without training data only) of Vgg11 on 5% subset
of FEMNIST dataset (62-class), following the same setting as leaf benchmark
[2]’ online document (Table 8). We observe a similar trend as in VGG-11 on
CIFAR-10 experiments.

  * •

3\. ME attack Performance (without training data only) of MobileNetV2 on
CIFAR-10 (Table 9). We observe a similar trend as in VGG-11 on CIFAR-10
experiments.

###  A.9 Time Cost Evaluation

We evaluate time cost of five attacks on VGG-11 CIFAR-10 model (fine-tuning
case). The time cost measurement is done on a PC with a R7-5800X CPU and a
single RTX-3090 GPU.

Table 10: Time costs of proposed five attacks of attacking VGG-11 on CIFAR-10
(N=8) in fine-tuning case.

Time Cost (s) | Craft | GAN | GM | Train | SoftTrain
---|---|---|---|---|---
Preparation | 317.8 | 44.5 | 30.7 | 4.7 | 18.9
Surrogate Training | 381.2 | 339.3 | 5523.1 | 313.4 | 949.5
Total | 699.0 | 383.8 | 5553.8 | 318.1 | 968.4

Tab. 10 provides time cost breakdown for two phases, namely, preparation phase
and training the surrogate model phase. The preparation phase includes
crafting inputs in Craft-ME, fitting conditional GAN in GAN-ME, and crafting
soft labels in SoftTrain-ME. From the results, we can see the Craft-ME needs
the most preparation time and GAN-ME ranks the second. Both require generating
crafted data and training the generator using collected gradients. For
training the surrogate model, GM-ME method requires the most time as solving
the gradient matching involves computation of second-order derivatives. Soft-
Train method also spends more time compared to Craft-, GAN- and Train-ME
because the soft-labels are used as the second objective.

In all the cases, the time cost of the proposed ME attacks is dominated by the
cost of training the surrogate model. This heavily depends on the network
topology, the number of iterations, and input size and vary from application
to application, making it difficult to provide a comprehensive time complexity
analysis.

###  A.10 Conditional-GAN Architecture

The detailed architecture of the conditional-GAN for GAN-ME attack is shown in
Fig. 10.

Figure 10: Architecture detail of the c-GAN in GAN-ME.

##  Appendix B Model Inversion Attack Implementation

###  B.1 Model Inversion Attack Setting

The degree of how well client’s data is protected in SFL is evaluated using
Mean Squared Error (MSE) between ground-truth images and reconstructed images
in Model Inversion Attack (MIA). For MIA, we follow the same model-based
attack methodology as in [30, 16]. The MIA flow is shown in Fig. 11. We assume
the honest-but-curious attacker (this time, the server) has access to the 10K
validation dataset of CIFAR-10. We use the L3 inversion model in [16] to
perform MIA, and use the trained L3 inversion model to reconstruct the raw
image from the intermediate activation sent by benign clients.

Figure 11: Details of model inversion attack using L3 inversion model and the
available validation dataset, done by an honest-but-curious server. (a) Train
the inversion model on the validation dataset. (b) Use the inversion model to
invert intermediate activation sent by clients.