# Training Latency Minimization for Model-Splitting Allowed Federated Edge
Learning

Yao Wen, Guopeng Zhang, Kezhi Wang, and Kun Yang  Yao Wen and Guopeng Zhang
are with the School of Computer Science and Technology, China University of
Mining and Technology, Xuzhou 221116, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Kezhi Wang is with the Department of Computer Science,
Brunel University London, Middlesex UB8 3PH, U.K. E-mail:
<EMAIL_ADDRESS>Kun Yang is with the School of Computer Science and
Electronic Engineering, University of Essex, Colchester CO4 3SQ, U.K. E-mail:
<EMAIL_ADDRESS>
###### Abstract

To alleviate the shortage of computing power faced by clients in training deep
neural networks (DNNs) using federated learning (FL), we leverage the edge
computing and split learning to propose a model-splitting allowed FL (SFL)
framework, with the aim to minimize the training latency without loss of test
accuracy. Under the synchronized global update setting, the latency to
complete a round of global training is determined by the maximum latency for
the clients to complete a local training session. Therefore, the training
latency minimization problem (TLMP) is modelled as a minimizing-maximum
problem. To solve this mixed integer nonlinear programming problem, we first
propose a regression method to fit the quantitative-relationship between the
cut-layer and other parameters of an AI-model, and thus, transform the TLMP
into a continuous problem. Considering that the two subproblems involved in
the TLMP, namely, the cut-layer selection problem for the clients and the
computing resource allocation problem for the parameter-server are relative
independence, an alternate-optimization-based algorithm with polynomial time
complexity is developed to obtain a high-quality solution to the TLMP.
Extensive experiments are performed on a popular DNN-model EfficientNetV2
using dataset MNIST, and the results verify the validity and improved
performance of the proposed SFL framework.

###### Index Terms:

Federated learning, split learning, edge computing, computing task offloading,
resource allocation.

##  I Introduction

The latest Artificial Intelligence (AI) products are powered by cutting-edge
machine learning (ML) technology, ranging from face detection [1] and speech
recognition [2] installed on mobile devices to virtual assistants deployed in
autonomous systems [3]. Large-scale data is essential for training high-
performance AI-models, e.g., decision trees, support vector machines (SVMs),
and deep neural networks (DNNs). However, centralized approach to training AI-
models requires clients to transfer privately-owned data to servers, which
poses a great threat to users’ privacy and security [4]. Google has proposed a
new ML paradigm, called federated learning (FL) [5], that allows multiple
clients to train an AI-model in a distributed manner, while keeping their data
local. The vanilla FL algorithm, called FedAvg is illustrated in Fig. 1 (the
left part). A parameter server (PS) first distributes an AI-model to be
trained to multiple selected clients. Then, each client trains the model
locally with the relevant data and upload the trained model to the PS for
aggregation. This process is iterated for several rounds and the aggregated
model achieves a certain test accuracy.

Figure 1: The frameworks of FedAvg and SFL.

However, FedAvg is more suitable for training lightweight AI-models, where the
communication cost is greater than the computational cost caused to clients
[5]. With the rapid popularization of high-speed mobile communications (such
as 5G) and the development of large-scale deep AI-models (such as DNNs), the
application context of FL has undergone fundamental changes:

  * •

The ultra-reliable and low-latency communication technologies of 5G make the
communication cost no longer the bottleneck for FL [6].

  * •

The computing power of mobile devices has not been improved significantly,
compared with the rapidly growing demand for training large-scale DNN-models
[7].

New techniques and methods are needed to cope with the new trends in the
development of AI-models.

Fortunately, the method of split learning (SL) [8] supported by edge computing
[8] provides a feasible solution to the above challenges. As illustrated in
Fig. 1 (the right part), the model-split allowed FL (SFL) first splits an AI-
model into two parts, which can be trained separately [4]. The part trained by
the client is called the client-side model, while the other part trained by
the PS is called the server-side model. The client initiates the forward
propagation (FP) on the client-side model to obtain the intermediate result,
called the smashed-data, and then, uploads it to the PS. The PS uses the
received smashed-data as the input to continue the FP on the server-side
model. Thereafter, the PS can initiate the backward propagation (BP) on the
server-side model. The obtained gradients, as the server-side intermediate
result, is transferred to the client, so the client can use it to complete the
BP on the client-side model.

Notably, by equipping the PS with a high-performance computing server, it can
then process multiple server-side models (offloaded by multiple clients) in
parallel, thus greatly reducing the computational burden of the clients.
However, unlike in FedAvg, the clients in SFL cannot independently perform the
FP and BP, resulting in the intermediate results being communicated hundreds
of times during training client-side models. The reduced computational burden
on clients may lead to the increased network data traffic. Although this fits
the trend of increased network bandwidth and larger-and-deeper AI-models, the
following issues must be addressed in order to improve the training efficiency
of SFL:

Challenge 1: Cut-layer selection of the clients. In the paradigm of SL, the
cut-layer for a client refers to the last layer of the client-side model [9],
as shown in Fig. 1. The client-selected cut-layer determines its communication
load (for transmitting the smashed data) and computational load (for training
the client-side model). However, there is no explicit relationship between the
cut-layer and other parameters of an AI-model, which brings to a client a huge
discrete solution space and extremely high computational complexity to choose
the optimal cut-layer.

Challenge 2: Computing resource allocation of the PS. In SFL, a PS can
simultaneously train multiple server-side models offloaded by the
participating clients. Due to the heterogeneity of clients in computing power
and datasets, the PS must optimize its allocation of the limited computing
resources to improve the training efficiency of SFL.

In this paper, we leverage edge computing and split learning to improve the
training efficiency of SFL. In particular, we aim to minimize the training
latency of the SFL using synchronized global model update (SGMU) [10] without
losing the test accuracy of the trained model. Under the SGMU setting, the
latency in completing a round of global training is determined by the maximum
latency for the participating clients to complete a session of local training.
Therefore, the training latency minimization problem (TLMP) is modelled as one
that minimizes the maximum latency for the clients to complete a local
training session. To address Challenge 1 inherent in the problem, we take a
popular AI-model, EfficientNetV2 [11], as an example and propose a regression
method to fit the quantitative relationship between the cut-layer of the
trained model and the resulting communication and computational loads for a
client. Then, the original mixed integer nonlinear programming (MINLP) problem
is transformed into a continuous one. Considering the relative independence of
cut-layer selection and computing resource allocation, this problem is further
decoupled into two subproblems, namely, the cut-layer selection problem of the
clients (to address Challenge 1) and the computing resource allocation problem
of the PS (to address Challenge 2). Finally, an alternate-optimization-based
algorithm is proposed to obtain a high-quality solution to the training
latency minimization problem. Extensive experiments are performed on
EfficientNetV2 using dataset MNIST [12], and the results verify the validity
of our proposed method.

In summary, the main technical contributions of this paper are as follows:

  1. 1.

The edge computing and split learning techniques are orchestrated to improve
the training efficiency of SFL. With the specific aim to minimize the training
latency of the SFL using SGMU without loss of test-accuracy, an MINLP problem,
called the TLMP, is formulated. The TLMP achieves the goal by constantly
removing the system bottleneck, i.e., the maximum latency for the
participating clients to complete a local training session.

  2. 2.

To address Challenge 1 inherent in the TLMP, a regression method is presented
to fit the quantitative relationship between the cut-layer of an AI-model and
the communication and computational loads generated to a client. In this way,
the original MINLP problem is transformed into a continuous one.

  3. 3.

By decoupling the continuous TLMP into two independent subproblems, namely,
the cut-layer selection problem of the clients and the computing resource
allocation problem of the PS, an alternate-optimization-based algorithm with
polynomial time complexity is designed to obtain a high-quality solution to
the original TLMP. Extensive experiments testify the effectiveness of the
proposed method.

The rest of this paper is organized as follows. Sec. II reviews the related
works. Sec. III describes the detailed training procedure of the SFL using
SGMU. Sec. IV first gives the parameterized expressions for the time
consumption of the SFL, and then, formulates the TLMP as a min-max problem.
The algorithm to solve the TLMP is developed in Sec. V. In Sec. VI, experiment
results are provided to verify the effectiveness of the proposed method.
Finally, the paper is summarized in Sec. VII.

##  II Related Works

Federated Learning: As an effective distributed machine learning method for
privacy protection, FL has been widely studied and applied in many fields.
Parameter Server architecture [13] is widely used to enable a large number of
computation nodes to train a shared model by aggregating locally-computed
updates. Federated optimization [14] is proposed in to improve communication
efficiency by minimizing communication rounds, while keeping training data
local and protecting data privacy. To make FL more practical, an effective
client selection algorithm is proposed in [15] to solve the straggler (clients
requiring longer training times) problem. Issues of fairness assurance between
participating clients were discussed in [16]. In [17], the authors
counterbalanced the bias introduced by non-IID data and accelerated the
convergence of model training. In [18], local training of clients constrained
by computing power is accelerated by training task offloading. In [19], the
authors studied the non-convex resource allocation problem of FL over wireless
networks.

Splitting Learning : Distributed training of large-scale AI-models requires
large amount of computing resources, but the limited computing resources of
clients become the bottleneck restricting the performance (e.g., training
latency, test accuracy, etc.) of FL. Thereby, some recent researches focus on
reducing the computational burden of clients. Split learning (SL) [20] is one
of the methods, which can divide an AI-model into multiple parts and trains
them separately in a certain order. A parallel SL method is proposed in [21]
to prevent overfitting due to differences in the training order and data size
of the segmented model parts. In [22], the authors use SL to assist the
collaborative computation of DNNs between mobile devices and cloud server, and
formulate the optimal computing-resource scheduling problem for the DNN layers
as the shortest path problem and integer linear programming. In [23], the
authors proposed a multi-split algorithm that can assign DNN submodels to each
computational node in a given network topology, and a computational graph
search problem was proposed to optimize the assignment. In [24], SL is used to
reduce the total energy cost of edge devices under time-varying wireless
channels. In [25], the coexistence of FL and SL is allowed in wireless
networks, and the convergence of hybrid split and federated learning (HSFL)
algorithm is analyzed with non-IID data distribution.

Collaborative training of AI-models: In a collaborative training framework
such as FL, the bottleneck is in the computing power and workload differences
between the client and the edge server. Therefore, some recent studies have
focused on the allocation of computing, storage and communication resources
for edge devices to improve the efficiency of collaborative training. To
eliminate the straggler clients in an FL task, a timeout threshold is set in
[26], after which the uploaded model will be discarded to reduce the overall
training latency. An effective method is proposed in [27] to dynamically and
virtually allocate the computing and communication resources of an edge server
to multiple clients, allowing the task offloading of the clients. The
challenge lies in estimating client training time, as the real time
consumption is only known after the actual run. To address this issue, the
data fitting method, i.e., optimal regression model, is proposed in [9] to
predict training time of clients. The authors in [16] modelled the estimation
of client’s local time consumption (consisting of transmitting and training
model) as a C2MAB problem. Then, they converted the online scheduling problem
into an offline problem with Lyapunov optimization and solved it using a
divide-and-conquer method. By applying deep reinforcement learning (DRL), a
capability-matched model offloading strategy is designed in [10] for
heterogeneous clients, aiming to minimize the training latency of an FL task.
The authors in [28] used DNN partitioning to minimize the FL training latency
under the constraints of device-specific participation rates, energy
consumption, and memory usage, regardless of the transmission of the smashed
data over a wireless network.

Summary: Inspired by the existing works, we leverage edge computing and SL to
improve the training efficiency of FL, a collaborative training paradigm.
Under the synchronized global model update (SGMU) setting, the two key
challenges described in Sec. I, namely, (1) cut-layer selection of the clients
and (2) computing resource allocation of the PS, must be dealt with jointly,
which is not yet involved in the related works.

##  III System Model

We consider a FL system consisting of a unique PS and a set $\mathcal{K}$ of
$K$ clients. The goal of the PS is to train an AI-model with the data owned by
the clients but without directly sharing the data. Next, we review FedAvg, the
vanilla FL algorithm, and then, describe SFL, the edge computing and split
learning supported FL method. The system parameters are summarized in the
following Table I.

TABLE I: Parameter Table Symbol |  Description
---|---
$\textbf{w}_{k}$, $\textbf{w}^{\text{C}}_{k}$, $\textbf{w}^{\text{S}}_{k}$ |  Local model of client $k$, client-side model of client $k$, server-side model of client $k$.
$\mathcal{D}_{k}$, $\mathcal{B}_{k}$ |  Local dataset of client $k$, mini-batch of client $k$.
$x_{k,n}$, $y_{k,n}$, $\hat{y}_{k,n}$ |  The $n^{\text{th}}$ sample of $\mathcal{D}_{k}$ or $\mathcal{B}_{k}$, ground-true label, prediction of the label.
$\mathcal{L}(\cdot)$, $\triangledown\mathcal{L}(\cdot)$ |  Sample-wise loss function, gradient of loss function.
$\eta$ |  Learning rate.
$I_{k}$ |  Number of epochs in a local training session of client $k$.
$w^{(l)}$ |  The parameters of the $l^{\text{th}}$ layer of model w.
$L$ |  The number of layers of an AI-model.
$l_{k}$, $l_{k}^{\text{min}}$ |  Cut-layer of client $k$, the minimum value of $l_{k}$ that client $k$ can take.
$\mathcal{S}_{k,n}$ |  Sample-wise smashed-data of client $k$.
$\mathcal{G}_{k,n}$ |  Sample-wise gradients of client $k$.
$\Lambda_{k}$ |  Size of $\mathcal{S}_{k,n}$ and $\mathcal{G}_{k,n}$.
$r_{k}$ |  Data rate between the PS and client $k$.
$f_{k}^{\text{C}}$ |  Computing power of client $k$ for training $\textbf{w}^{\text{C}}_{k}$.
$f_{k}^{\text{S}}$ |  Computing power of the PS allocated to client $k$ for training $\textbf{w}^{\text{S}}_{k}$.
$F^{\text{C}}_{k}$, $B^{\text{C}}_{k}$ |  Amount of computing resources required for client $k$ to perform FP/BP with one data-sample.
$F^{\text{S}}_{k}$, $B^{\text{S}}_{k}$ |  Amount of computing resources required for the PS to perform FP/BP with one data-sample.
$F_{k}^{\text{tot}}$ |  $F_{k}^{\text{tot}}=F^{\text{C}}_{k}+B^{\text{C}}_{k}$, which is the computational load of client $k$ for training the client-side model.
$\Gamma$ |  Amount of computing resources required to train an AI-model with one data-sample.
$F^{\text{max}}$ |  Amount of computing resources available for the PS.
$T_{k}$ |  Latency for client $k$ to complete a local training session.
$\mathcal{T}$ |  Latency for the $K$ clients to complete a round of global training.
L |  $\textbf{L}=(l_{1},\cdots,l_{K})$.
F |  $\textbf{F}=(f^{\text{S}}_{1},\cdots,f^{\text{S}}_{k},\cdots,f^{\text{S}}_{K})$.
$\tilde{T}_{k}$ |  Latency for client $k$ to complete a local training session using only the local computing power $f_{k}^{\text{C}}$.
$\tilde{k}$ |  New index of client $k$ according to $\tilde{T}_{k}$.
$\mathcal{K}_{\texttt{FedAvg}}$ |  Set of clients adopting FedAvg to train the local model.
$\mathcal{K}_{\text{{SFL}}}$ |  Set of clients adopting SFL to train the local model.
$\theta$ |  The index of the first client in $\mathcal{K}_{\text{{SFL}}}$.

###  III-A The Basic of FedAvg

In FedAvg [5], the PS first broadcasts the initial model w to the $K$ clients.
Then, each client $k$ ($\forall k\in\mathcal{K}$) trains the model,
represented by $\textbf{w}_{k}$, locally and independently by using its own
dataset
$\mathcal{D}_{k}=\\{\left(x_{k,n},y_{k,n}\right)|n={1,\cdots,n_{k}}\\}$, where
$n_{k}$ is the size of the dataset, and $x_{k,n}$ and $y_{k,n}$ are
respectively the sample and its ground-true label.

Local training at client k: Using the stochastic gradient descent (SGD), a
mini-batch $\mathcal{B}_{k}\subseteq\mathcal{D}_{k}$ can be randomly sampled
from $\mathcal{D}_{k}$ to train $\textbf{w}_{k}$ in each local training epoch.
With each $(x_{k,n},y_{k,n})\in\mathcal{B}_{k}$, client $k$ first performs the
FP to obtain $\hat{y}_{k,n}=\texttt{fp}(x_{k,n};\textbf{w}_{k})$, the
prediction of $y_{k,n}$ using the current model $\textbf{w}_{k}$. Then, client
$k$ performs the BP and calculates the gradient
$\triangledown\mathcal{L}(\hat{y}_{k,n},y_{k,n};\textbf{w}_{k})$ w.r.t
$\textbf{w}_{k}$, where $\mathcal{L}(\hat{y}_{k,n},y_{k,n};\textbf{w}_{k})$ is
the sample-wise loss function. The local model of client $k$ can be updated as

$\displaystyle\textbf{w}_{k}=\textbf{w}_{k}-\eta\triangledown\mathcal{L}(\hat{y}_{k,n},y_{k,n};\textbf{w}_{k}),\
\forall k\in\mathcal{K},$ (1)

where $\eta$ is the learning rate.

During a local training session, the above training epoch is iterated $I_{k}$
times. Thereafter, client $k$ uploads the trained model $\textbf{w}_{k}$ to
the PS for aggregation.

Global model update at the PS: Under the synchronized global model update
(SGMU) setting, the PS can update the global model w by using the following
eq. (2), only if all the local models of the $K$ clients are collected.

$\textbf{w}=\sum_{k\in\mathcal{K}}\frac{n_{k}}{\sum_{k\in\mathcal{K}}n_{k}}\textbf{w}_{k}.$
(2)

The global model update will be iterated several rounds until the learning
goal (e.g., a certain prediction accuracy on the test dataset or the maximum
number of global training rounds) is achieved.

###  III-B The Framework of SFL

Referring to [9], a layer of an AI-model is defined as the minimum divisible
unit of the model parameter, which can be either a separate layer (for
example, active, convolution, or fully connection layers) or a combination of
multiple consecutive layers. Then, an AI-model w can be partitioned into $L$
layers as

$\textbf{w}=w^{(1)}\uplus\cdots\uplus w^{(l)}\uplus\cdots\uplus w^{(L)},$ (3)

where $w^{(l)}$ represents the $l^{\text{th}}$ layer’s parameter of the model
and the operator $"\uplus"$ represents connecting any two consecutive layers.

Unlike FedAvg, which orders each client to train $\textbf{w}_{k}$
independently, SFL splits $\textbf{w}_{k}$ for each client $k$ into the
following two parts

$\textbf{w}_{k}=\textbf{w}^{\text{C}}_{k}\uplus\textbf{w}^{\text{S}}_{k},\
\forall k\in\mathcal{K},$ (4)

where $\textbf{w}^{\text{C}}_{k}=w^{(1)}\uplus\cdots\uplus w^{(l_{k})}$ and
$\textbf{w}^{\text{S}}_{k}=w^{(l_{k}+1)}\uplus\cdots\uplus w^{(L)}$ are
respectively called the client-side model and server-side model, and are
respectively submitted to client $k$ and the PS for training. The last layer
of the client-side model, $w^{(l_{k})}$, is termed as the cut-layer for client
$k$ [29].

Figure 2: The workflow of the SFL.

In Fig. 2, we show the workflow of SFL, which includes the following 5 phases:
1) Splitting local models; 2) Distributing local models; 3) Training local
models; 4) Collecting local models; and 5) Aggregating local models. The main
difference between SFL and FedAvg is in the $3^{\text{rd}}$ and
$5^{\text{th}}$ phases. The detail is given below.

Phase 3 (Training local models): Since $\textbf{w}_{k}$ is split into two
parts, namely $\textbf{w}^{\text{C}}_{k}$ and $\textbf{w}^{\text{S}}_{k}$, the
training of $\textbf{w}_{k}$ requires the collaboration between client $k$ and
the PS.

  * •

Forward Propagation:

    * (1)

Client-side FP: In each local training epoch, client $k$ starts the FP on the
client-side model $\textbf{w}^{\text{C}}_{k}$ using mini-batch
$\mathcal{B}_{k}$. The output at the cut-layer $w^{(l_{k})}$, called the
smashed-data [9], is given as

$\mathcal{S}_{k,n}=\texttt{fp}\big{(}x_{k,n};\textbf{w}^{\text{C}}_{k}\big{)},\
\forall k\in\mathcal{K}.$ (5)

    * (2)

Uploading smashed data: Client $k$ uploads the smashed data
$\mathcal{S}_{k,n}$ and the label $y_{k,n}$111Uploading labels to PS
undermines the data-privacy of clients. To address this issue, the authors in
[30] proposed a three-stage SL method to avoid the leakage. For reconstruction
attacks on training data, the leakage risk can also be reduced by using the
differential privacy [31][32]. This paper uses an approach similar to [9] that
allows a client to independently select the minimum cut-layer to implement its
security requirement. Please refer to eq. (17) and its description in Sec.
IV-B for detail. as the client-side intermediate results, to the PS for
further processing.

    * (3)

Server-side FP: Upon receiving $\mathcal{S}_{k,n}$ and $y_{k,n}$, the PS
continues the FP on the server-side model $\textbf{w}^{\text{S}}_{k}$, and
obtains the predicted value of $y_{k,n}$ as

$\hat{y}_{k,n}=\texttt{fp}\big{(}\mathcal{S}_{k,n};\textbf{w}^{\text{S}}_{k}\big{)},\
\forall k\in\mathcal{K}.$ (6)

  * •

Backward propagation:

    * (4)

Sever-side BP: With $\hat{y}_{k,n}$ and $y_{k,n}$, the PS instead of client
$k$ starts the BP. The server-side model $\textbf{w}^{\text{S}}_{k}$ is
updated as

$\textbf{w}^{\text{S}}_{k}=\textbf{w}^{\text{S}}_{k}-\eta\triangledown\mathcal{L}(\hat{y}_{k,n},y_{k,n};\textbf{w}^{\text{S}}_{k}),\
\forall k\in\mathcal{K}.$ (7)

    * (5)

Downloading gradients: Let $\mathcal{G}_{k,n}$ denote the gradients generated
at layer $w^{l_{k+1}}$, the first layer of $\textbf{w}^{\text{S}}_{k}$. The PS
should transfers $\mathcal{G}_{k,n}$ as the server-side intermediate results
to client $k$ to continue the BP.

    * (6)

Client-side BP: Upon receiving $\mathcal{G}_{k,n}$ from the PS, client $k$ can
complete the client-side BP and updates the client-side model
$\textbf{w}^{\text{C}}_{k}$ as

$\textbf{w}^{\text{C}}_{k}=\textbf{w}^{\text{C}}_{k}-\eta\triangledown\mathcal{L}(\mathcal{G}_{k,n};\textbf{w}^{\text{C}}_{k}),\
\forall k\in\mathcal{K}.$ (8)

Phase 5 (Aggregating local models): At the end of each local training session,
client $k$ uploads the trained client-side model $\textbf{w}^{\text{C}}_{k}$
to the PS. The PS can then combine $\textbf{w}^{\text{C}}_{k}$ with the
trained server-side model $\textbf{w}^{\text{S}}_{k}$ as
$\textbf{w}_{k}=\textbf{w}^{\text{C}}_{k}\uplus\textbf{w}^{\text{S}}_{k}$.
Under the SGMU setting, when obtaining all $\textbf{w}_{k}$ of the $K$
clients, the PS can aggregate them to update the global model w by using eq.
(2).

##  IV Problem Formulation

Although the SFL reduces the computational burden of clients for training
local models, the communication overhead inevitably increases because the
intermediate results $\mathcal{S}_{k,n}$ and $\mathcal{G}_{k,n}$ are
communicated multiple times between the PS and each client due to the
collaborative training.

In this section, we first quantify the latency of each phase of the SFL. Then,
the training latency minimization problem is proposed.

###  IV-A Latency of each phase

Referring to Sec. III-B, the latency of each phase of the SFL is analyzed as
follows.

Phase 1 (Splitting local models): The latency of this phase is caused by
performing Algorithm 1, namely, the joint cut-layer selection and computing
resource allocation algorithm proposed in Sec. V-E. Considering that the
algorithm has a polynomial time complexity and runs on the PS with sufficient
computing power, the latency is negligible.

Phase 2 (Distributing local models): Let $r_{k}$ denote the data rate between
the PS and client $k$ in the current round of global training 222In general,
the uplink bandwidth and downlink bandwidth are asymmetric. Since the
intermediate results generated by the clients are different, in the downlink,
PS can only distribute the intermediate results to the clients using
orthogonal unicast rather than broadcast. So we assume that the uplink and
downlink rate is the same for each client.. The latency for client $k$ to
download the client-side model $\textbf{w}^{\text{C}}_{k}$ from the PS is
given by

$\texttt{DM}_{k}=|\textbf{w}^{\text{C}}_{k}|/{r_{k}},\forall k\in\mathcal{K},$
(9)

where $|\textbf{w}^{\text{C}}_{k}|$ denotes the size (in bits) of
$\textbf{w}^{\text{C}}_{k}$.

Phase 3 (Training local models): As analyzed in Sec. III-B, the training of
the local model of any client $k$,
$\textbf{w}_{k}=\textbf{w}^{\text{C}}_{k}\uplus\textbf{w}^{\text{S}}_{k}$,
consists of 6 stages. The latency of each stage is given below.

  * (1)

Client-side FP. Let $F^{\text{C}}_{k}$ denote the number of float point
operations (Flops) required for client $k$ to perform the sample-wise FP on
$\textbf{w}^{\text{C}}_{k}$. Let $f_{k}^{\text{C}}$ (in Flops/s) denote the
available computing power for client $k$. Then, the latency to process the
total $|\mathcal{B}_{k}|$ samples is given by

$\texttt{FP}_{k}^{\text{C}}=F^{\text{C}}_{k}|\mathcal{B}_{k}|/{f_{k}^{\text{C}}},\
\forall k\in\mathcal{K}.$ (10)

  * (2)

Uploading smashed-data. Let $|\mathcal{S}_{k,n}|$ denoted the size (in bits)
of $\mathcal{S}_{k,n}$. The latency of transmitting the total
$|\mathcal{B}_{k}|$ pieces of smashed-data is given by333To start the BP, the
PS must have the label of each sample. The labels can be transferred by the
clients to the PS along with the smashed-data. Since the size of labels is
much smaller than the smashed-data, the transmission overhead is omitted here.

$\texttt{TS}_{k}=|\mathcal{S}_{k,n}||\mathcal{B}_{k}|/r_{k},\ \forall
k\in\mathcal{K}.$ (11)

  * (3)

Server-side FP. Let $F^{\text{S}}_{k}$ denote the number of Flops required for
the PS to perform the sample-wise FP on $\textbf{w}^{\text{S}}_{k}$. Let
$f_{k}^{\text{S}}$ (in Flops/s) denote the computing power allocated by the PS
to client $k$. Then, the latency of processing the total $|\mathcal{B}_{k}|$
samples is given by

$\texttt{FP}_{k}^{\text{S}}=F^{\text{S}}_{k}|\mathcal{B}_{k}|/f_{k}^{\text{S}},\
\forall k\in\mathcal{K}.$ (12)

  * (4)

Server-side BP. Let $B^{\text{S}}_{k}$ denote the number of Flops required for
the PS to perform the sample-wise BP on $\textbf{w}^{\text{S}}_{k}$. Then, the
latency of processing the total $|\mathcal{B}_{k}|$ samples is given by

$\texttt{BP}_{k}^{\text{S}}=B^{\text{S}}_{k}|\mathcal{B}_{k}|/f_{k}^{\text{S}},\
\forall k\in\mathcal{K}.$ (13)

  * (5)

Downloading gradient. Let $|\mathcal{G}_{k,n}|$ (in bits) denote the size of
$\mathcal{G}_{k,n}$. Then, the latency of downloading the total
$|\mathcal{B}_{k}|$ gradients is given by

$\displaystyle\texttt{TG}_{k}=|\mathcal{G}_{k,n}||\mathcal{B}_{k}|/{r_{k}},\forall
k\in\mathcal{K}.$ (14)

  * (6)

Client-side BP. Let $B^{\text{C}}_{k}$ denote the number of Flops required for
client $k$ to perform the sample-wise BP on $\textbf{w}^{\text{C}}_{k}$. The
time required to process the total $|\mathcal{B}_{k}|$ samples is given by

$\texttt{BP}_{k}^{\text{C}}=B^{\text{C}}_{k}|\mathcal{B}_{k}|/f_{k}^{\text{C}},\
\forall k\in\mathcal{K}.$ (15)

Phase 4 (Collecting client models): After $I_{k}$ local training epochs,
client $k$ uploads the updated client-side model $\textbf{w}^{\text{C}}_{k}$
to the PS for aggregation. The latency is given by

$\texttt{UM}_{k}=|\textbf{w}^{\text{C}}_{k}|/r_{k},\ \forall k\in\mathcal{K}.$
(16)

Phase 5 (Aggregating local models): Aggregating client models requires only
small computation effort. Since the PS has sufficient computing power, the
latency is negligible.

###  IV-B Overall time consumption of SFL

In general, the smashed-data and gradients generated by processing one data-
sample have the same size
$|\mathcal{S}_{k,n}|=|\mathcal{G}_{k,n}|=\Lambda_{k}$. The latency for client
$k$ to complete a local training session, i.e., $I_{k}$ local training epochs,
is given by

$\displaystyle T_{k}$ $\displaystyle=\texttt{DM}_{k}+\texttt{UM}_{k}+$
$\displaystyle\ \ \ \
I_{k}|\mathcal{B}_{k}|(\texttt{FP}_{k}^{\text{C}}+\texttt{TS}_{k}+\texttt{FP}_{k}^{\text{S}}+\texttt{BP}_{k}^{\text{S}}+\texttt{TG}_{k}+\texttt{BP}_{k}^{\text{C}})$
$\displaystyle=2\frac{|\textbf{w}^{\text{C}}_{k}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\left(\frac{F^{\text{C}}_{k}+B^{\text{C}}_{k}}{f_{k}^{\text{C}}}+\frac{F^{\text{S}}_{k}+B^{\text{S}}_{k}}{f_{k}^{\text{S}}}+2\frac{\Lambda_{k}}{r_{k}}\right),$
$\displaystyle\;\;\;\;\;\forall l_{k}\in\\{l_{k}^{\text{min}},\cdots,L\\},\
\forall k\in\mathcal{K}.$ (17)

where $l_{k}^{\text{min}}$ is the minimum value of $l_{k}$ that client $k$ can
take. The value of $l_{k}^{\text{min}}$ reflects the privacy requirement of
client $k$. The larger $l_{k}^{\text{min}}$, the less likely it is to derive
the raw information of the client from $\mathcal{S}_{k,n}$.

From eq. (17), we note that when the cut-layer is selected as $l_{k}=L$,
$\textbf{w}^{\text{C}}_{k}=\textbf{w}$ and
$\textbf{w}^{\text{S}}_{k}=\emptyset$. It means that client $k$ prefers to
train the entire model $\textbf{w}_{k}$ locally, as in FedAvg. In this case,
one can get

$\displaystyle F^{\text{S}}_{k}=0,\ B^{\text{S}}_{k}=0,\ \text{and}\
\Lambda_{k}=0,\ \text{if}\ l_{k}=L,\ \forall k\in\mathcal{K}.$ (18)

By substituting eq. (18) into eq. (17) , the latency for client $k$ to
complete a local training session in this case is given by

$\displaystyle
T_{k}=2\frac{|\textbf{w}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\frac{\Gamma}{f^{\text{C}}_{k}},\
\text{if}\ l_{k}=L,\ \forall k\in\mathcal{K},$ (19)

where $|\textbf{w}|$, $\Gamma$ are the size of model w (in bits) and total
amount of computing load for one data-sample of model w.

Combining eqs. (17) and (19), we can represent the training latency $T_{k}$ of
client $k$ in the following form.

$T_{k}=\begin{cases}\displaystyle{\frac{2|\textbf{w}^{\text{C}}_{k}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\left(\frac{F^{\text{C}}_{k}+B^{\text{C}}_{k}}{f_{k}^{\text{C}}}+\frac{F^{\text{S}}_{k}+B^{\text{S}}_{k}}{f_{k}^{\text{S}}}+2\frac{\Lambda_{k}}{r_{k}}\right)},\\\
\quad\quad\quad\quad\quad\quad\ \forall
l_{k}\in\\{l_{k}^{\text{min}},\cdots,L-1\\},\ \forall k\in\mathcal{K},\\\
\displaystyle{\frac{2|\textbf{w}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\frac{\Gamma}{f^{\text{C}}_{k}}},\
\text{if}\ l_{k}=L,\ \forall k\in\mathcal{K}.\\\ \end{cases}$ (20)

###  IV-C Training Latency Minimization Problem

From eq. (17), we know that

  * •

The latency for any client $k$ to train the model and communicate the
intermediate results depends on the selected cut-layer $l_{k}$;

  * •

While the PS is much more powerful than each of the clients, it needs to serve
many clients at the same time, so its available computing resources are
relatively limited.

Hence, the communication and computing resources of the clients and PS must be
jointly scheduled to adapt to the cut-layers selected for the clients, so that
the overall training latency of the SFL can be minimized.

Because we can only accurately predict the available computing resources and
data rate for the clients and PS for a short time in the future [10], our goal
is set to minimize $\mathcal{T}$, the latency to complete one round of global
training. As the SGMU is adopted, the overall latency $\mathcal{T}$ depends on
$\mathcal{T}=\max_{k\in\mathcal{K}}T_{k}$, the maximum latency for the $K$
clients to complete a local training session. Let
$\textbf{L}=(l_{1},\cdots,l_{K})$ and
$\textbf{F}=(f^{\text{S}}_{1},\cdots,f^{\text{S}}_{k},\cdots,f^{\text{S}}_{K})$.
The training latency minimization problem for the SFL is formulated as

$\displaystyle\min_{\textbf{L},\textbf{F}}\mathcal{T},\;\;\mathcal{T}=\max_{k\in\mathcal{K}}\
T_{k},$ (21) s.t. $\displaystyle l_{k}\in\\{l_{k}^{\text{min}},\cdots,\ L\\},\
\forall k\in\mathcal{K},$ (21.1)
$\displaystyle\sum^{K}_{k=1}f^{\text{S}}_{k}\leqslant F^{\text{max}},$ (21.2)
$\displaystyle|\textbf{w}^{\text{C}}_{k}|+|\textbf{w}^{\text{S}}_{k}|=|\textbf{w}|,\
\forall k\in\mathcal{K},$ (21.3) $\displaystyle
F^{\text{C}}_{k}+F^{\text{S}}_{k}+B^{\text{S}}_{k}+B^{\text{C}}_{k}=\Gamma,\
\forall k\in\mathcal{K},$ (21.4)

where constraint (21.1) is the value space of the cut-layer for client $k$,
constraint (21.2) limit the maximum amount of computing resources available
for the PS to $F^{\text{max}}$, constraint (21.3) comes from eq. (4), which
guarantees the model integrity after being split for any client $k$, and
constraint (21.4) comes from the following fact.

Given the AI-model to be trained, the amount of computing resources required
to train the model with one data-sample, represented as $\Gamma$, and the
amounts of computing resources required to perform the sample-wise BP and FP,
respectively represented as $F^{\text{tot}}$
($F^{\text{tot}}=F^{\text{C}}_{k}+F^{\text{S}}_{k}$) and $B^{\text{tot}}$
($B^{\text{tot}}=B^{\text{S}}_{k}+B^{\text{C}}_{k}$), are constants.
Therefore, we have the following equation, i.e., constraint (21.4).

$\Gamma=F^{\text{tot}}+B^{\text{tot}}=F^{\text{C}}_{k}+F^{\text{S}}_{k}+B^{\text{S}}_{k}+B^{\text{C}}_{k},\
\forall k\in\mathcal{K}.\\\ $ (22)

Since both continuous variable F and discrete variable L are involved, problem
(21) is a mixed integer nonlinear programming (MINLP) problem, which cannot be
solved directly by using conventional methods. In what follows, we develop
effective methods to address this problem.

##  V Solving the Problem

In problem (21), the size of the client-side model
($|\textbf{w}^{\text{C}}_{k}|$), the computing resources needed to train the
client-side model ($F_{k}^{\text{tot}}=F_{k}^{\text{C}}+B_{k}^{\text{C}}$),
and the amount of generated intermediate results ($\Lambda_{k}$) all depend on
the selected cut-layer $l_{k}$ for client $k$. However, as shown in the
following Figs. 3, 5, and 6, there is no explicit relationship between the
cut-layer and other parameters of an AI-model. Since a DNN usually has
hundreds of layers, this will lead to a unusually high time and space
complexity to find the optimal cut-layers for the clients. To the best of our
knowledge, there is currently no better way to address this issue. In this
paper, we propose a regression method to quantify the relationship between the
cut-layer and other parameters of a given AI-model. Based on that, a fast and
effective method is developed to find the high-quality solution of problem
(21).

It is worth noting that logistic regression and curve fitting methods have
been widely used to address such relationship fitting problems. For example,
in [33] and [34], the authors built practical non-linear energy harvesting
models by curve fitting for measurement data. Based on the fitted model,
effective resource allocation algorithms are developed for wireless
information and power transfer (SWIPT) systems.

###  V-A Fit The Relationship Between The Parameters of An AI-model

Different AI-models have different neural network structures. To the best of
our knowledge, there is currently no better way to obtain their relationship
expression. In this paper, we use the popular AI-model EfficientNetV2[11] as
an example and propose the regression method to fit the relationship between
the cut-layer and other parameters of an AI-model.

####  V-A1 Client-side model size against different cut-layers

As shown in Fig. 3, with increasing $l_{k}$, the increase of
$|\textbf{w}^{\text{C}}_{k}|$ is not obvious at the beginning, but the growth
rate increases sharply when $l_{k}>36$. Therefore, we set the relationship
between $l_{k}$ and $|\textbf{w}^{\text{C}}_{k}|$ as

$\displaystyle|\textbf{w}^{\text{C}}_{k}|=$ $\displaystyle\alpha(l_{k})^{2},\
\forall k\in\mathcal{K},$ (23)

where $\alpha\geq 0$ is the parameter to be fitted.

Figure 3: The quantitative-relationship between $l_{k}$ and
$|\textbf{w}^{\text{C}}_{k}|$.

####  V-A2 Training load of client against different cut-layers

The training of the client-side model includes the FP and BP. The
computational load of performing the FP can be obtained by using the python
package torchinfo [35], however, there is no direct way to know the
computational load of performing the BP. To address this issue, we trained
EfficientNetV2 1500 times using a dataset of size 32. The time consumed by FP
and BP in each training session is shown in Fig. 4.

Figure 4: The time consumption of FP and BP

From Fig. 4, we see that with the same computing power the time consumed by BP
is roughly several times the time consumed by FP. So we have

$B_{k}^{\text{tot}}\approx\kappa F_{k}^{\text{tot}},\ \forall
k\in\mathcal{K},$ (24)

where $\kappa\geqslant 1$. Fig. 5 shows the computational load of client $k$
for training the client-side model (that is,
$F_{k}^{\text{tot}}=F^{\text{C}}_{k}+B^{\text{C}}_{k}$) with different cut-
layers. There is an approximate linear relationship between them, which is
given by

$F_{k}^{\text{tot}}=F^{\text{C}}_{k}+B^{\text{C}}_{k}=\beta l_{k}(1+\kappa),\
\forall k\in\mathcal{K},$ (25)

where $\beta>0$ is the parameter to be fitted.

Figure 5: The quantitative-relationship between $l_{k}$ and
$F_{k}^{\text{tot}}$.

####  V-A3 Size of intermediate results against different cut-layers

The head of a DNN is always a convolutional neural network (CNN) to extract
features, thus reducing the computational burden of the subsequent feedforward
neural network (FNN). Hence, the size of the intermediate results decreases
rapidly in the CNN layers but changes slowly in the FNN layers, as shown in
Fig. 6. Accordingly, we set the relationship between $l_{k}$ and $\Lambda_{k}$
as

$\Lambda_{k}=\left|\mathcal{S}_{k,n}\right|=\left|\mathcal{G}_{k,n}\right|=\frac{\gamma_{1}}{l_{k}+\gamma_{2}},\
\forall k\in\mathcal{K},$ (26)

where $\gamma_{1}>0$ and $\gamma_{2}\geqslant 0$ are the parameters to be
fitted.

Figure 6: The quantitative-relationship between $l_{k}$ and $\Lambda_{k}$.

###  V-B Problem Reformulation

By substituting eqs. (23), (25), and (26) into eq. (20), the expression for
$T_{k}$ can be written as

$T_{k}=\begin{cases}2\frac{\alpha
l_{k}^{2}}{r_{k}}+I_{k}\left|\mathcal{B}_{k}\right|\left(\beta(1+\kappa)\left(\frac{l_{k}}{F_{k}^{\text{C}}}+\frac{L-l_{k}}{F_{k}^{\text{S}}}\right)+2\frac{\frac{\gamma_{1}}{l_{k}+\gamma_{2}}}{r_{k}}\right),\\\
\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{if}\
l_{k}^{\text{min}}\leqslant l_{k}<L,\ \forall k\in\mathcal{K},\ \\\
\displaystyle{\frac{2|\textbf{w}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\frac{\Gamma}{f^{\text{C}}_{k}}},\
\text{if}\ l_{k}=L,\ \forall k\in\mathcal{K}.\\\ \end{cases}$ (27)

Then, the MINLP problem (21) is transformed into the following continuous
problem.

$\displaystyle\min_{\textbf{L},\textbf{F}}\mathcal{T},\;\;\mathcal{T}=\max_{k\in\mathcal{K}}\
T_{k},$ (28) s.t. $\displaystyle l_{k}^{\text{min}}\leqslant l_{k}\leqslant
L,\ \forall k\in\mathcal{K},$ (28.1)
$\displaystyle\sum^{K}_{k=1}f^{\text{S}}_{k}\leqslant F^{\text{max}}.$ (28.2)

One can first find the solution of problem (28), and then, rounds it to an
integer. We note that if the computing resource allocation F for the $K$
clients were known, problem (28) can be decomposed into $K$ independent
subproblems, and the goal of the $k^{\text{th}}$ subproblem is to minimize the
latency for client $k$ to complete a local training session. These subproblems
can be solved in parallel without loss of optimality.

Based on the above findings, we can solve the cut-layer selection problem (to
find the optimal L) and the computing resource allocation problem (to find the
optimal F) alternately and iteratively. Finally, the solution to problem (28)
can be obtained.

###  V-C Solving The Cut-Layer Selection Problem

For any given F, problem (28) can be decomped into $K$ independent subproblems
as given below.

$\displaystyle\min_{l_{k}}$ $\displaystyle\;T_{k},\ \forall k\in\mathcal{K},$
(29) s.t. $\displaystyle l_{k}^{\text{min}}\leqslant l_{k}\leqslant L,\
\forall k\in\mathcal{K}.$ (29.1)

By solving problem (29), the optimal cut-layer of any client $k$ is obtained
in closed-form, as shown in the following lemma.

######  Lemma 1.

With $\alpha>0,\ \beta>0,\ \kappa>0,\ \gamma_{1}>0,\ and\ \gamma_{2}\geqslant
0$, the optimal cut-layer $l^{*}_{k}$ for any client $k$ is given by the
following rules.

$\displaystyle l^{*}_{k}=$
$\displaystyle\begin{cases}l_{k}^{\text{min}},&\frac{\partial T_{k}}{\partial
l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}>0,\\\ L,&\frac{\partial
T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}<0,\\\
\left\lfloor\underset{l_{k}}{\operatorname{arg}}\,\left(\frac{\partial
T_{k}}{\partial l_{k}}=0\right)\right\rfloor,&\frac{\partial T_{k}}{\partial
l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}\leqslant 0\ and\ \frac{\partial
T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}\geqslant 0.\end{cases}\;$ (30)

###### Proof.

Please refer to Appendix A. ∎

In the $3^{\text{rd}}$ case of eq. (30), the function $\frac{\partial
T_{k}}{\partial l_{k}}=0$ contains a cubic terms of $l_{k}$ and can be solved
by using the Cardano’s formula [36]. It is worth noting that the complexity of
finding the suboptimal cut-layers for the $K$ clients is only
$\mathcal{O}(K)$, thus avoiding an exhaustive search on the actual complex
relationships between the parameters of an AI-model with extremely high time
and space complexity.

###  V-D Computing Resource Allocation of the PS

Substituting the obtained cut-layers of the $K$ clients, namely,
$\textbf{L}^{*}=(l_{1}^{*},\cdots,l_{K}^{*})$, into eq. (27), one can get

$T_{k}=\begin{cases}2\frac{\alpha(l_{k}^{*})^{2}}{r_{k}}+I_{k}\left|\mathcal{B}_{k}\right|\left(\beta(1+\kappa)\left(\frac{l_{k}^{*}}{F_{k}^{\text{C}}}+\frac{L-l_{k}^{*}}{F_{k}^{\text{S}}}\right)+2\frac{\frac{\gamma_{1}}{l_{k}^{*}+\gamma_{2}}}{r_{k}}\right),\\\
\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{if}\
l_{k}^{\text{min}}\leqslant l_{k}^{*}<L,\ \forall k\in\mathcal{K},\ \\\
\displaystyle{\frac{2|\textbf{w}|}{r_{k}}+I_{k}|\mathcal{B}_{k}|\frac{\Gamma}{f^{\text{C}}_{k}}},\
\text{if}\ l_{k}^{*}=L,\ \forall k\in\mathcal{K}.\\\ \end{cases}$ (31)

Problem (28) can be simplified to

$\displaystyle\min_{\textbf{F}}\;\;$
$\displaystyle\mathcal{T}=\max_{k\in\mathcal{K}}\ T_{k},$ (32) s.t.
$\displaystyle\sum^{K}_{k=1}f^{\text{S}}_{k}\leqslant F^{\text{max}}.$ (32.1)

The difficulty in solving problem (32) is that the objective,
$\mathcal{T}=\max_{k\in\mathcal{K}}T_{k}$, to be optimized is a nonlinear
function w.r.t $T_{k}$, and $T_{k}$ is a piecewise function w.r.t $l_{k}^{*}$.
To solve this difficulty, we define $\tilde{T}_{k}$ as the latency for client
$k$ to complete a local training session using only the local computing power
$f_{k}^{\text{C}}$. We can sort the $K$ clients in ascending order according
to $\\{\tilde{T}_{k}|\forall k\in\mathcal{K}\\}$ as shown in eq. (33), where
the reordered index of client $k$ is represented as $\tilde{k}$,
$\forall\tilde{k}\in\mathcal{K}$.

$\displaystyle\overbrace{T_{\tilde{1}}\leqslant\cdots\leqslant
T_{\tilde{k}-1=\theta-1}}^{\texttt{FedAvg}}\leqslant\overbrace{T_{\tilde{k}=\theta}\leqslant\cdots\leqslant
T_{K}}^{\texttt{SFL}}$ (33)

Next, we derive the following two lemmas.

######  Lemma 2.

The $K$ clients participating in an FL task can be divided into two groups:
the clients in group
$\mathcal{K}_{\texttt{FedAvg}}=\\{\tilde{1},\cdots,\theta-1\\}$ adopt FedAvg
and train $\textbf{w}_{\tilde{k}}$ independently, while the clients in group
$\mathcal{K}_{\text{{SFL}}}=\\{\theta,\cdots,\tilde{K}\\}$ adopt SFL and train
$\textbf{w}_{\tilde{k}}$ in collaboration with the PS, where $\theta$ is the
index of the first client in $\mathcal{K}_{\text{{SFL}}}$.

###### Proof.

Please refer to Appendix B. ∎

######  Lemma 3.

The optimum of the objective of problem (32) is $\mathcal{T}=T_{\theta}$,
$\theta\in\tilde{\mathcal{K}}$.

###### Proof.

Please refer to Appendix C. ∎

By using Lemmas 2 and 3, problem (32) can be converted into the following
form.

$\displaystyle\min_{\theta,\ \textbf{F}}\;\;$ $\displaystyle T_{\theta},$ (34)
s.t.
$\displaystyle\mathcal{K}_{\texttt{FedAvg}}\cup\mathcal{K}_{\text{{SFL}}}=\mathcal{K},$
(34.1)
$\displaystyle\mathcal{K}_{\texttt{FedAvg}}\cap\mathcal{K}_{\text{{SFL}}}=\emptyset,$
(34.2) $\displaystyle T_{\tilde{k}}\leqslant T_{\theta},\
\forall\tilde{k}\in\mathcal{K}_{\texttt{FedAvg}},$ (34.3) $\displaystyle
T_{\tilde{k}}=T_{\theta},\ \forall\tilde{k}\in\mathcal{K}_{\text{{SFL}}}.$
(34.4)

By solving problem (34), the optimal amount of computing resources that the PS
allocates to any client $k$ is obtained in closed-form, as shown in the
following lemma.

######  Lemma 4.

The amount of computing resources allocated by the PS to each client
$\tilde{k}\in\mathcal{K}$ is given by the following formula.

$\displaystyle
f^{\text{S}}_{\tilde{k}}=\begin{cases}0,&\forall\tilde{k}\in\mathcal{K}_{\texttt{FedAvg}},\\\
\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{T_{\theta}-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{C}}_{\tilde{k}}+B^{\text{C}}_{\tilde{k}})}{f^{\text{C}}_{\tilde{k}}}-2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|\Lambda_{\tilde{k}}+|\textbf{w}^{\text{C}}_{\tilde{k}}|}{r_{{\tilde{k}}}}},&\forall\tilde{k}\in\mathcal{K}_{\texttt{SFL}}.\\\
\end{cases}$ (35)

###### Proof.

Please refer to Appendix D. ∎

According to Lemma 4, once $\mathcal{T}=T_{\theta}$ is known, the optimal
resource allocation of the PS, F, can be obtained. In order to solve
$T_{\theta}$, we substitute formula (35) into constraint (32.1) and convert
this inequality constraint into an equality constraint as

$\displaystyle\sum_{\tilde{k}\in\mathcal{K}_{\texttt{SFL}}}\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{T_{\theta}-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{C}}_{\tilde{k}}+B^{\text{C}}_{\tilde{k}})}{f^{\text{C}}_{\tilde{k}}}-2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|\Lambda_{\tilde{k}}+|\textbf{w}^{\text{C}}_{\tilde{k}}|}{r_{\tilde{k}}}}=F^{\text{max}}$
(36)

######  Lemma 5.

$T_{\theta}$ is strictly convex w.r.t $F^{\text{max}}$.

###### Proof.

Please refer to appendix E. ∎

Lemma 5 ensures that for a given $F^{\text{max}}$ an unique solution
$T_{\theta}$ exists for eq. (36), which can be solved by using the CVX tool
[37].

###  V-E Algorithm Implementation and the Convergence and Complexity Analysis

The overall algorithm to solve problem (21) is summarized in the following
Algorithm 1.

Algorithm 1 The overall algorithm to solve problem (21).

1: Use eq. (19) to obtain $\tilde{T}_{k}$, $\forall k\in\mathcal{K}$;

2: Sort the $K$ clients in ascending order according to
$\\{\tilde{T}_{k}|k\in\mathcal{K}\\}$, and update the index of client $k$ to
its serial number $\tilde{k}$;

3: Set the iteration number $i=0$,
$\textbf{F}^{i}=\\{\frac{F^{\text{max}}}{K},\cdots,\frac{F^{\text{max}}}{K}\\}$,
$\theta^{i}=1$, and $\mathcal{T}=T_{\theta^{i}}$;

4: repeat

5: With the given $\textbf{F}^{i}$, select cut-layers $\textbf{L}^{i}$ for the
clients using eq. (30);

6: Substitute $\textbf{L}^{i}$ and $\theta^{i}$ into eq. (34) and obtain
$T_{\theta^{i+1}}$ using the CVX tool [37];

7: With the obtained $T_{\theta^{i+1}}$, update $\textbf{F}^{i+1}$ using eq.
(35);

8:
$\theta^{i+1}=\Phi\left(f^{\text{S}}_{\tilde{1}},\cdots,f^{\text{S}}_{\tilde{K}}\right)$;

9: $i=i+1$;

10: until $\mathcal{T}=T_{\theta^{i}}$ converges, or the iteration number
reaches the maximum $i=i^{\text{max}}$;

11: return The computing allocation strategy F of the PS and the cut-layers
selected for the $K$ clients L.

In Step 8, function $\Phi(\cdot)$ counts the number of non-zero elements in a
vector. Since the algorithm contains iterative execution, the convergence is
proved below.

######  Lemma 6.

Algorithm 1 converges to a stable point.

###### Proof.

For a feasible solution $\left(\textbf{L},\textbf{F},\theta\right)$ of problem
(21), the resulting objective value can be obtained as
$\mathcal{T}(\textbf{L},\textbf{F},\theta)$.

In the $i^{\text{th}}$ iteration, the algorithm ensures that

$\displaystyle\mathcal{T}\left(\textbf{L}^{i},\textbf{F}^{i},\theta^{i}\right)$
$\displaystyle\geqslant\mathcal{T}(\textbf{L}^{i+1},\textbf{F}^{i},\theta^{i})$
$\displaystyle Step\ 5$ (37)
$\displaystyle\geqslant\mathcal{T}(\textbf{L}^{i+1},\textbf{F}^{i+1},\theta^{i})$
$\displaystyle Step\ 6$
$\displaystyle\geqslant\mathcal{T}(\textbf{L}^{i+1},\textbf{F}^{i+1},\theta^{i+1})$
$\displaystyle Step\ 7$ .

Therefore,
$\left\\{\mathcal{T}(\textbf{L}^{i},\textbf{F}^{i},\theta^{i})|i=1,\cdots,i^{\text{max}}\right\\}$,
the sequence of the objective values of problem (21), is non-increasing.
Furthermore, because $\mathcal{T}>0$, the alternate-optimization-based
algorithm can converge to a lower bound as the number of iterations increases.
∎

Finally, the computational complexity of Algorithm 1 is analyzed as follows.
The complexity of ranking $K$ clients in Step 2 is $\mathcal{O}(K^{2})$. The
complexity of performing cut-layer selection for the $K$ clients in Step 5 is
$\mathcal{O}(K)$. In Step 6, the optimal $T_{\theta}$ could be obtained by
using linear search which requires $\mathcal{O}(K^{3})$ to converge [38]. The
resource allocation of the PS in Step 7 requires $\mathcal{O}(K)$. As a
result, the computational complexity of Algorithm 1 is
$\mathcal{O}\left(K^{2}+i^{\text{max}}(K^{3}+2K)\right)$.

Although the alternate-optimization-based Algorithm 1 is with polynomial time
complexity, it generally converges at a sub-optimal solution to the original
MINLP problem, namely, problem (21), due to the regression method employed.
The high quality of the solution will be validated in the following
experiments.

##  VI Experiment Results

To verify the performance of the proposed SFL method, we experiment on the
EfficientNetV2 [11], a popular DNN-model with $L=59$ layers. The proposed SFL
method is also applicable to other AI-models with different neural network
structures. The EfficientNetV2 is trained to perform image classification task
on the dataset MNIST [12], which has 60,000 labeled samples (each sample
consists of 28$\times$28 pixels) in 10 classes with 6,000 images per class.
The samples of each class is evenly distributed across the clients involved in
an FL task. By using the data generation method [19], the number of samples of
each client ranges from 361 to 3,578, and the local dataset is split randomly
with 75% for training and 25% for testing. The SGD algorithm with the fixed
batch size $|\mathcal{B}_{k}|=32$ is adopted to train the local model
$\textbf{w}_{k}$ of any client $k$.

The available computing resources of the PS is set to $F^{\text{max}}=3,000$
GFlops, and a total of 30 clients serve as the candidates for the PS-initiated
FL tasks. The clients are heterogeneous and have different local computing
power $f_{k}^{\text{C}}$ or transmission rates $r_{k}$. The PS randomly
selects a fraction of the candidate clients to participate in each round of
global training.

###  VI-A Effectiveness of The Regression Method in Finding The Optimal Cut-
Layer

The regression method, proposed in Sec. V-A, is used to help a client quickly
find the approximately optimal cut-layer. To verify its validity, we set the
local computing power of client $k$ to $f_{k}^{\text{C}}=100$ GFlops, the edge
computing power of the PS allocated to client $k$ to $f_{k}^{\text{S}}=1,484$
GFlops, the transmission rate to $r_{k}=4$ Mb/s, and the local dataset size of
client $k$ to $|\mathcal{D}_{k}|=3,396$. The latency for client $k$ to
complete a local training session with different cut-layers is shown in Fig.
7. For each cut-layer, the latency of training the client-side model
$\textbf{w}^{\text{C}}_{k}$ and the server-side model
$\textbf{w}^{\text{S}}_{k}$ and transmitting the intermediate results
($\mathcal{S}_{k,n}$ and $\mathcal{G}_{k,n}$) are separately annotated in Fig.
7.

Figure 7: Training latency of SFL with different cut-layers.

As shown in Fig. 7, the SFL can considerably reduce the training latency of
client $k$, compared to the FedAvg. Specially, when the optimal cut-layer
$l_{k}=11$ is selected, up to 300 seconds can be saved. When $l_{k}<11$, the
training latency mainly results from transferring the intermediate results
between the client and PS. When $l_{k}>11$, the training latency mainly comes
from the training of the client-side model $\textbf{w}^{\text{C}}_{k}$. In all
cases, the latency for the PS to train the server-side model
$\textbf{w}^{\text{S}}_{k}$ and the latency for the client to download/upload
the client-side model $\textbf{w}^{\text{C}}_{k}$ are negligible for the
overall training latency.

From Fig. 7, we also note that the optimal cut-layer for the client found by
using the regression method is consistent with that obtained by an exhaustive
search on the space created by the actual quantitative relationship between
the parameters of EfficientNetV2. In this way, the high computational
complexity caused by searching for the optimal cut-layer of an AI-model is
avoided.

To quantify the prediction accuracy of the proposed regression method, we
adopt the determination coefficient $\mathcal{R}$ [9] as the evaluation
metric. $\mathcal{R}\triangleq
1-\frac{\sum^{L}_{l=1}(x-\hat{x}_{l})^{2}}{\sum^{L}_{l=1}(x_{l}-\tilde{x}_{l})^{2}}$,
where $x_{l}$, $\hat{x}_{l}$, and $\tilde{x}_{l}$ represent the true value,
the predicted value, and the historical mean value, respectively. The closer
the value of $\mathcal{R}$ is to 1, the better the regression performance is.
In the following Tab. II, we show the determination coefficient $\mathcal{R}$
of the fitted values of $|\textbf{w}^{\text{C}}_{k}|$ (see eq. (23)),
$F_{k}^{\text{tot}}$ (see eq. (25)) and $\Lambda_{k}$ (see eq. (26)).

TABLE II: The determination coefficient of the proposed regression method. Fitted Item |  |  $\mathcal{R}$
---|---|---
$|\textbf{w}^{\text{C}}_{k}|$ |  |  0.9482
$F_{k}^{\text{tot}}$ |  |  0.9659
$\Lambda_{k}$ |  |  0.9065

As shown in Tab. II, $0.9<\mathcal{R}<1$ for all the predicted items. This
indicates that the proposed regression method can accurately predict the
relationship between the selected cut-layer and other parameters of an AI-
model, which ensures that the proposed alternate-optimization-based Algorithm
1 can obtain high-quality solutions as analyzed in Sec. V-E.

###  VI-B Results for joint cut-layer selection and computing resource
allocation.

To validate the performance of the proposed joint cut-layer selection and
computing resource allocation algorithm, namely Algorithm 1, we select 10
clients from the 30 candidates, each with the same number of local training
epochs $I_{k}=20$. Fig. 8 compares the training latency for each client using
the proposed SFL to the vanilla FedAvg. For the SFL, the latency for each
client to train the client-side and server-side models and communicate the
intermediate results are also separately annotated in Fig. 8.

Figure 8: Training latency of different clients in the SFL.

As can be seen from Fig. 8, the PS did not allocate any resources to clients 1
to 6 in the SFL, so these clients used the FedAvg to train the local model,
while clients 7 to 10 shared the computing resources of the PS to reduce the
training latency. As a result, clients 7 to 10 have roughly the same training
latency (410 s), but the training latency of clients 1 to 6 is different and
each is lower than that of clients 7 to 10. This verifies that the proposed
SFL has the similar function as the water pouring algorithm [39], which can
allocate the computing resources of the PS to the resource-constraint clients
adaptively, thus reducing the difference in training latency of the clients.
In contrast, when all the clients adopt FedAvg, they can only train the local
model $\textbf{w}_{k}$ independently. As the overall training latency depends
on the longest training time of the clients, the latency of the FedAvg (980 s)
is much higher than that of the SFL (410 s).

Next, we demonstrate the convergence of Algorithm 1. As the number of
iterations increases, the convergence of the cut-layer selection and computing
resource allocation are shown in Fig. 9a and Fig. 9b, respectively, and the
resulting training latency is shown in Fig. 10. It can be seen that Algorithm
1 can converge to a fixed point after about 5 iterations. This verifies the
low computational complexity of the proposed algorithm.

(a) Convergence of cut-layer selection.

(b) Convergence of resource allocation.

Figure 9: Convergence of Algorithm 1 for the two subproblems. Figure 10:
Convergence of the training latency of SFL.

Finally, we show the effect of the amount of computing resources available for
the PS on the overall training latency of the SFL. For that purpose, we
increase the computing resources of the PS from $F^{\text{max}}=100$ GFlops to
$F^{\text{max}}=9,000$ GFlops. The variation of training latency of the SFL
(for one local training session) is shown in Fig. 11.

Figure 11: Training latency of SFL with different $F^{\text{max}}$.

As shown in Fig. 11, with the increasing computing resources available for the
PS, the training latency of the SFL gradually decreases. This is because more
and more resource-constrained clients can share the computing resources of the
PS, so that the system bottleneck (the longest training latency for the
clients to complete a local training session) is continuously eliminated, thus
reducing the overall training latency of the SFL.

It is also noted from Fig. 11 that the training latency of the SFL decreases
rapidly when the server resources increase from $F^{\text{max}}=100$ GFlops to
$F^{\text{max}}=2,000$ GFlops, while the latency decreases slowly when the
server resources continue to increase. This indicates that the PS can optimize
the amount of resource contributed to an FL task to improve the efficiency of
resource utilization. Improving the marginal benefits of server resources is
important when the PS simultaneously trains multiple AI-models or performs
multi-task learning. This is beyond the scope of this paper and will be left
for future studies.

###  VI-C Test Accuracy of The Trained Model

In this part, we show the test accuracy of the trained EfficientNetV2 after
multiple rounds of global training. In each round, 10 clients were randomly
selected from the 30 candidates to participate, each with the same number of
local training epochs $I_{k}=20$. As the number of global training rounds
increases, the time consumed by the SFL and the resulting test accuracy of the
trained model are shown in Fig. 12. Note that each experiment is run at least
3 times and the average results are reported.

Figure 12: Training latency of the SFL and the resulting test accuracy of
EfficientNetV2.

From Fig. 12, one can see that as the number of global training rounds
increases, the test accuracies of the models trained with the SFL and FedAvg
gradually increase and approach 0.90 after 60 rounds of global training.
Additionally, the accuracy convergence of the SFL is consistent with that of
the FedAvg. This verifies the effectiveness of the proposed SFL framework.
But, in order to achieve the same test accuracy, the SFL consumes much less
time than the FedAvg. This verifies that the proposed SFL can greatly improve
the training efficiency of FL without compromising the test accuracy.

##  VII Conclusions

In this paper, we leverage the edge computing and split learning techniques to
improve the training efficiency of SFL. Specially, we minimize the training
latency of the SGMU-based SFL without loss of the test-accuracy of the trained
model. However, the proposed problem is an MINLP problem which is challenging
to solve. So we first propose a regression method to transform it into a
continuous problem, and then, develop an alternate-optimization-based
algorithm with polynomial time complexity to solve it. The convergence of the
algorithm is proved and the high quality of the solution is experimentally
verified. The experiment results also show that the proposed SFL can train an
AI-model to have the same test accuracy with much less training-time compared
with the FedAvg. In addition, we also note that the PS can further improve the
marginal benefits of its computing resources, which is the key to solving
parallel-training of multiple AI-models as well as multi-task learning
problems. These topics are interesting and deserve further study in the
future.

## References

  * [1] C. Yan, Y. Zhang, Q. Zhang, Y. Yang, X. Jiang, Y. Yang, and B. Wang, “Privacy-preserving online automl for domain-specific face detection,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4124–4134, 2022.
  * [2] M. Soleymanpour, M. T. Johnson, R. Soleymanpour, and J. Berry, “Synthesizing dysarthric speech using multi-speaker tts for dysarthric speech recognition,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7382–7386.
  * [3] B. Sriman, S. A. Silviya, S. K. Shriram, S. R. Kumar, R. Shriya, and A. Sujitha, “Virtual assistant for automatic emotion monitoring using perceived stress scale (pss),” in _2022 4th International Conference on Inventive Research in Computing Applications (ICIRCA)_ , 2022, pp. 1529–1534.
  * [4] N. Yan, K. Wang, C. Pan, and K. K. Chai, “Private federated learning with misaligned power allocation via over-the-air computation,” _IEEE Communications Letters_ , vol. 26, no. 9, pp. 1994–1998, 2022.
  * [5] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _AISTATS_ , 2017.
  * [6] K. Shafique, B. A. Khawaja, F. Sabir, S. Qazi, and M. Mustaqim, “Internet of things (iot) for next-generation smart systems: A review of current challenges, future trends and prospects for emerging 5g-iot scenarios,” _IEEE Access_ , vol. 8, pp. 23 022–23 040, 2020.
  * [7] B.-S. Liang, “Ai computing in large-scale era: Pre-trillion-scale neural network models and exa-scale supercomputing,” in _2023 International VLSI Symposium on Technology, Systems and Applications (VLSI-TSA/VLSI-DAT)_ , 2023, pp. 1–3.
  * [8] O. Gupta and R. Raskar, “Distributed learning of deep neural network over multiple agents,” _Journal of Network and Computer Applications_ , vol. 116, pp. 1–8, 2018.
  * [9] H. Jiang, M. Liu, S. Sun, Y. Wang, and X. Guo, “Fedsyl: Computation-efficient federated synergy learning on heterogeneous iot devices,” in _2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)_ , 2022, pp. 1–10.
  * [10] D. Wu, R. Ullah, P. Harvey, P. Kilpatrick, I. Spence, and B. Varghese, “Fedadapt: Adaptive offloading for iot devices in federated learning,” _IEEE Internet of Things Journal_ , vol. 9, no. 21, pp. 20 889–20 901, 2022\.
  * [11] M. Tan and Q. Le, “Efficientnetv2: Smaller models and faster training,” in _International conference on machine learning_. PMLR, 2021, pp. 10 096–10 106.
  * [12] L. Deng, “The mnist database of handwritten digit images for machine learning research,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 141–142, 2012.
  * [13] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng, “Large scale distributed deep networks,” in _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1_ , ser. NIPS’12. Red Hook, NY, USA: Curran Associates Inc., 2012, p. 1223–1231.
  * [14] J. Konečný, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device intelligence,” _CoRR_ , vol. abs/1610.02527, 2016. [Online]. Available: http://arxiv.org/abs/1610.02527
  * [15] T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous resources in mobile edge,” in _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_ , 2019, pp. 1–7.
  * [16] T. Huang, W. Lin, W. Wu, L. He, K. Li, and A. Y. Zomaya, “An efficiency-boosting client selection scheme for federated learning with fairness guarantee,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 32, no. 7, pp. 1552–1564, 2021.
  * [17] H. Wang, Z. Kaplan, D. Niu, and B. Li, “Optimizing federated learning on non-iid data with reinforcement learning,” in _IEEE INFOCOM 2020 - IEEE Conference on Computer Communications_ , 2020, pp. 1698–1707.
  * [18] D. Wu, R. Ullah, P. Harvey, P. Kilpatrick, I. Spence, and B. Varghese, “Fedadapt: Adaptive offloading for iot devices in federated learning,” _IEEE INTERNET OF THINGS JOURNAL_ , vol. 9, no. 21, pp. 20 889–20 901, NOV 1 2022.
  * [19] C. T. Dinh, N. H. Tran, M. N. H. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, “Federated learning over wireless networks: Convergence analysis and resource allocation,” _IEEE/ACM Transactions on Networking_ , vol. 29, no. 1, pp. 398–409, 2021.
  * [20] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split learning for health: Distributed deep learning without sharing raw patient data,” _CoRR_ , vol. abs/1812.00564, 2018. [Online]. Available: http://arxiv.org/abs/1812.00564
  * [21] J. Jeon and J. Kim, “Privacy-sensitive parallel split learning,” in _2020 International Conference on Information Networking (ICOIN)_ , 2020, pp. 7–9.
  * [22] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “Jointdnn: An efficient training and inference engine for intelligent mobile cloud computing services,” _IEEE Transactions on Mobile Computing_ , vol. 20, no. 2, pp. 565–576, 2021.
  * [23] Y. Tian, Z. Zhang, Z. Yang, and Q. Yang, “Jmsnas: Joint model split and neural architecture search for learning over mobile edge networks,” in _2022 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2022, pp. 103–108.
  * [24] M. Krouka, A. Elgabli, C. B. Issaid, and M. Bennis, “Energy-efficient model compression and splitting for collaborative inference over time-varying channels,” in _2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2021, pp. 1173–1178.
  * [25] X. Liu, Y. Deng, and T. Mahmoodi, “Wireless distributed learning: A new hybrid split and federated learning approach,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 4, pp. 2650–2665, 2023.
  * [26] T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous resources in mobile edge,” _CoRR_ , vol. abs/1804.08333, 2018\. [Online]. Available: http://arxiv.org/abs/1804.08333
  * [27] C. Qiu, X. Wang, H. Yao, J. Du, F. R. Yu, and S. Guo, “Networking integrated cloud–edge–end in iot: A blockchain-assisted collective q-learning approach,” _IEEE Internet of Things Journal_ , vol. 8, no. 16, pp. 12 694–12 704, 2021.
  * [28] X. Deng, J. Li, C. Ma, K. Wei, L. Shi, M. Ding, and W. Chen, “Low-latency federated learning with dnn partition in distributed industrial iot networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 3, pp. 755–775, 2023.
  * [29] W. Wu, M. Li, K. Qu, C. Zhou, X. S. Shen, W. Zhuang, X. Li, and W. Shi, “Split learning over wireless networks: Parallel design and resource management,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, pp. 1051–1066, 2022.
  * [30] E. Erdoğan, A. Küpçü, and A. E. Çiçek, “Unsplit: Data-oblivious model inversion, model stealing, and label inference attacks against split learning,” in _Proceedings of the 21st Workshop on Privacy in the Electronic Society_ , ser. WPES’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 115–124. [Online]. Available: https://doi.org/10.1145/3559613.3563201
  * [31] J. Neera, X. Chen, N. Aslam, K. Wang, and Z. Shu, “Private and utility enhanced recommendations with local differential privacy and gaussian mixture model,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 4, pp. 4151–4163, 2023.
  * [32] M. Wu, G. Cheng, P. Li, R. Yu, Y. Wu, M. Pan, and R. Lu, “Split learning with differential privacy for integrated terrestrial and non-terrestrial networks,” _IEEE Wireless Communications_ , pp. 1–8, 2023.
  * [33] E. Boshkovska, D. W. K. Ng, N. Zlatanov, and R. Schober, “Practical non-linear energy harvesting model and resource allocation for swipt systems,” _IEEE Communications Letters_ , vol. 19, pp. 2082–2085, 2015.
  * [34] D. Alqahtani, Y. Chen, W. Feng, and M.-S. Alouini, “A new non-linear joint model for rf energy harvesters in wireless networks,” _IEEE Transactions on Green Communications and Networking_ , vol. 5, no. 2, pp. 895–907, 2021.
  * [35] K. Shibasaki, S. Fukuzaki, and M. Ikehara, “4k real time image to image translation network with transformers,” _IEEE Access_ , vol. 10, pp. 73 057–73 067, 2022.
  * [36] B. n. al-din abed, B. Z. Kamil, M. A. Hameed, and J. N. Abdullah, “Using cardano’s method for solving cubic equation in the cryptosystem to protect data security against cyber attack,” in _2020 2nd Annual International Conference on Information and Sciences (AiCIS)_ , 2020, pp. 127–131.
  * [37] M. Grant, S. Boyd, and Y. Ye, “Cvx: Matlab software for disciplined convex programming, version 2.0 beta,” 2013.
  * [38] S. P. Boyd and L. Vandenberghe, “Convex optimization,” _IEEE Transactions on Automatic Control_ , vol. 51, pp. 1859–1859, 2004.
  * [39] Q. Qi, A. Minturn, and Y. Yang, “An efficient water-filling algorithm for power allocation in ofdm-based cognitive radio systems,” in _2012 International Conference on Systems and Informatics (ICSAI2012)_ , 2012, pp. 2069–2073.

## Appendix A

Proof of Lemma 1: For any $l_{k}^{\text{min}}\leqslant l_{k}\leqslant L$, the
first- and second-order derivatives of $T_{k}$ w.r.t $l_{k}$ are obtained as

$\frac{\partial T_{k}}{\partial l_{k}}=4\frac{\alpha
l_{k}}{r_{k}}+I_{k}\left|\mathcal{B}_{k}\right|\left(\beta(1+\kappa)\left(\frac{1}{F_{k}^{\text{C}}}-\frac{1}{F_{k}^{\text{S}}}\right)-\frac{2\gamma_{1}}{r_{k}\left(l_{k}+\gamma_{2}\right)^{2}}\right),$
(38)

and

$\frac{\partial^{2}T_{k}}{\partial{l_{k}}^{2}}=4\frac{\alpha}{r_{k}}+\frac{4\gamma_{1}I_{k}\left|\mathcal{B}_{k}\right|}{r_{k}\left(l_{k}+\gamma_{2}\right)^{3}},$
(39)

respectively.

Because $\alpha>0$, $\gamma_{1}>0$, and $\gamma_{2}\geqslant 0$,
$\frac{\partial^{2}T_{k}}{\partial{l_{k}}^{2}}>0$. It means that
$\frac{\partial T_{k}}{\partial l_{k}}$ increases monotonically w.r.t $l_{k}$.
The optimal value of $l_{k}$, denoted by $l^{*}_{k}$, can be given by
considering the following three cases.

Case 1: $\frac{\partial T_{k}}{\partial
l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}>0$. If $\frac{\partial
T_{k}}{\partial l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}>0$, one can get
$\frac{\partial T_{k}}{\partial l_{k}}>0$ for any $l_{k}^{\text{min}}\leqslant
l_{k}\leqslant L$, because $\frac{\partial^{2}T_{k}}{\partial{l_{k}}^{2}}>0$.
Thus, $T_{k}$ increases monotonically w.r.t $l_{k}$ and reaches the minimum at
the left boundary of its domain. Therefore, $l^{*}_{k}=l_{k}^{\text{min}}$.

Case 2: $\frac{\partial T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}<0$. If
$\frac{\partial T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}<0$, we can get
$\frac{\partial T_{k}}{\partial l_{k}}<0$ for any $l_{k}^{\text{min}}\leqslant
l_{k}\leqslant L$, as $\frac{\partial^{2}T_{k}}{\partial{l_{k}}^{2}}>0$. Thus,
$T_{k}$ decreases monotonically w.r.t $l_{k}$ and reaches the minimum at the
right boundary of its domain $l^{*}_{k}=L$.

Case 3: $\frac{\partial T_{k}}{\partial
l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}\leqslant 0$ and $\frac{\partial
T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}\geqslant 0$. Because
$\frac{\partial^{2}T_{k}}{\partial{l_{k}}^{2}}>0$, $\frac{\partial
T_{k}}{\partial l_{k}}$ is monotonic and continuous w.r.t to $l_{k}$. When
$\frac{\partial T_{k}}{\partial
l_{k}}\big{|}_{l_{k}=l_{k}^{\text{min}}}\leqslant 0$ and $\frac{\partial
T_{k}}{\partial l_{k}}\big{|}_{l_{k}=L}\geqslant 0$, there exists a unique
solution $l_{k}$ for $\frac{\partial T_{k}}{\partial l_{k}}=0$ within
$l_{k}\in[l_{k}^{\text{min}},L]$. By rounding the resulting $l_{k}$ to an
integer, we can obtain
$l^{*}_{k}={\left\lfloor\underset{l_{k}}{\operatorname{arg}}\,\left(\frac{\partial
T_{k}}{\partial l_{k}}=0\right)\right\rfloor}.$

## Appendix B

Proof of Lemma 2: The heterogeneous clients have different local computing
power $f_{\tilde{k}}^{\text{C}}$, and the available computing resources of the
PS is limited to $F^{\text{max}}$. To solve the min-max problem (32), the PS
must allocate its computing resources to the clients with larger training
latency. Therefore, only part of the $K$ clients can share the computing
resources of the PS and can train $\textbf{w}_{k}$ using the SFL. This part of
clients form the group $\mathcal{K}_{\texttt{SFL}}$, and the left part of the
$K$ clients, that can only train $\textbf{w}_{k}$ independently using the
FedAvg, forms the group $\mathcal{K}_{\texttt{FedAvg}}$. The working mode is
like the water pouring method [37].

Next, we prove that the clients in group $\mathcal{K}_{\texttt{FedAvg}}$
consume shorter training time than those in group
$\mathcal{K}_{\text{{SFL}}}$. Assuming that this assertion holds true, we can
index the last client that adopts the FedAvg and the first client that adopts
the SFL as $\theta-1$ and $\theta$, respectively. According to eq. (33), we
have $T_{\theta-1}\leqslant T_{\theta}$. If not, the PS can take out a part of
its computing resources assigned to group $\mathcal{K}_{\text{{SFL}}}$ to
client $\theta-1$, until the training latency of all the clients in group
$\mathcal{K}_{\text{{SFL}}}\cup\theta-1$ are the same. Thus, the assertion is
proved.

## Appendix C

Proof of Lemma 3: From Lemma 2, we know that $T_{\theta-1}\leqslant
T_{\theta}$. This means that the clients in group
$\mathcal{K}_{\texttt{FedAvg}}$ consume shorter training times than those in
group $\mathcal{K}_{\text{{SFL}}}$.

Moreover, when the optimal solution of problem (32) is achieved, the PS
allocates its computing resources to the clients in group
$\mathcal{K}_{\text{{SFL}}}$, so that all the clients in group
$\mathcal{K}_{\text{{SFL}}}$ are with the same training time $T_{\theta}$.
Thus, the lemma is proved.

## Appendix D

Proof of Lemma 4: Any client $\tilde{k}$,
$\forall\tilde{k}\in\mathcal{K}_{\texttt{FedAvg}}$, trains
$\textbf{w}_{\tilde{k}}$ independently using the FedAvg. Therefore,
$f^{\text{S}}_{\tilde{k}}=0$,
$\forall\tilde{k}\in\mathcal{K}_{\texttt{FedAvg}}$.

For the clients in group $\mathcal{K}_{\texttt{SFL}}$, we take the first and
second-order derivatives of $T_{\tilde{k}}$ w.r.t $f^{\text{S}}_{\tilde{k}}$
using eq. (17) and have

$\frac{\partial T_{\tilde{k}}}{\partial
f^{\text{S}}_{\tilde{k}}}=-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{(f^{\text{S}}_{\tilde{k}})^{2}}<0,\
\forall\tilde{k}\in\mathcal{K}_{\texttt{SFL}},$ (40)

and

$\frac{\partial^{2}T_{\tilde{k}}}{\partial(f^{\text{S}}_{\tilde{k}})^{2}}=2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{(f^{\text{S}}_{\tilde{k}})^{3}}>0,\
\forall\tilde{k}\in\mathcal{K}_{\texttt{SFL}}.$ (41)

This indicates that $T_{\tilde{k}}$ is strictly convex w.r.t
$f^{\text{S}}_{\tilde{k}}$, and there exists a unique root for eq. (17).

By solving eq. (18), we can get the solution of $f^{\text{S}}_{\tilde{k}}$
($\forall\tilde{k}\in\mathcal{K}_{\texttt{SFL}}$) as shown in eq. (35)

## Appendix E

Proof of Lemma 5: Constraint (36) can be written as the following form.

$\displaystyle
F^{\text{max}}=H(T_{\theta})=\sum^{\theta}_{\tilde{k}=1}\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{T_{\theta}-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{C}}_{\tilde{k}}+B^{\text{C}}_{\tilde{k}})}{f^{\text{C}}_{\tilde{k}}}-2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|\Lambda_{\tilde{k}}+|\textbf{w}^{\text{C}}_{\tilde{k}}|}{r_{\tilde{k}}}}.$
(42)

The first and second-order derivatives of $H(T_{\theta})$ w.r.t to
$T_{\theta}$ are respectively given as

$\frac{\partial H}{\partial
T_{\theta}}=\sum^{\theta}_{\tilde{k}=1}\frac{-I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{\left(T_{\theta}-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{C}}_{\tilde{k}}+B^{\text{C}}_{\tilde{k}})}{f^{\text{C}}_{\tilde{k}}}-2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|\Lambda_{\tilde{k}}+|\textbf{w}^{\text{C}}_{\tilde{k}}|}{r_{{\tilde{k}}}}\right)^{2}}<0,$
(43)

and

$\frac{\partial^{2}H}{{\partial(T_{\theta})}^{2}}=\sum^{\theta}_{{\tilde{k}}=1}\frac{2I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{S}}_{\tilde{k}}+B^{\text{S}}_{\tilde{k}})}{\left(T_{\theta}-\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|(F^{\text{C}}_{\tilde{k}}+B^{\text{C}}_{\tilde{k}})}{f^{\text{C}}_{\tilde{k}}}-2\frac{I_{\tilde{k}}\left|\mathcal{B}_{\tilde{k}}\right|\Lambda_{\tilde{k}}+|\textbf{w}^{\text{C}}_{\tilde{k}}|}{r_{{\tilde{k}}}}\right)^{3}}>0.$
(44)

Therefore, $H(T_{\theta})$ has the inverse which is given by
$T_{\theta}=H^{-1}(F^{\text{max}})$.

The first- and second-order derivatives of $H^{-1}(F^{\text{max}})$ w.r.t
$F^{\text{max}}$ are given as

$\frac{\partial H^{-1}}{\partial F^{\text{max}}}=\frac{1}{\frac{\partial
H}{\partial T_{\theta}}}<0,\ \text{and}\
\frac{\partial^{2}H^{-1}}{{\partial(F^{\text{max}})}^{2}}=-\frac{\frac{\partial^{2}H}{{\partial(T_{\theta})}^{2}}}{{\left(\frac{\partial
H}{\partial T_{\theta}}\right)}^{3}}>0,$ (45)

respectively. This means that $T_{\theta}$ is convex w.r.t $F^{\text{max}}$.

|  Yao Wen received the bachelor’s degree from the Colloge of Information
Science and Technology, Nanjing Forestry University, Nanjing, China, in 2021.
He is currently working toward the M.E. degree with the School of Computer
Science and Technology, China University of Mining and Technology, Xuzhou,
China. His research interests include federated learning, edge computing, deep
neural network, split learning, and optimization theory.
---|---
|  Guopeng Zhang received the bachelor’s degree from the School of Computer
Science, Jiangsu Normal University, Xuzhou, China, in 2001, the master’s
degree from the School of Computer Science, South China Normal University,
Guangzhou, China, in 2005, and the Ph.D. degree from the School of
CommunicationEngineering, Xidian University, Xi’an, China, in 2009. He was
with ZTE Corporation Nanjing Branch for one year. In 2009, he joined the China
University of Minin and Technology, Xuzhou, China, where he is currently a
Professor with the School of Computer Science and Technology. He manages
research projects funded byvarious sources, such as the National Natural
Science Foundation of China. He has authored or coauthored more than 60
journal and conference papers. His main research interests include wireless
sensor networks, wireless personalarea networks, and their applications in the
Internet of Things.
---|---
|  Kezhi Wang received the Ph.D. degree in engineering from the University of
Warwick, U.K. He was with the University of Essex and Northumbria University,
U.K. Currently, he is a Senior Lecturer with the Department of Computer
Science, Brunel University London, U.K. His research interests include
wireless communications, mobile edge computing, and machine learning.
---|---
|  Kun Yang received his PhD from the Department of Electronic & Electrical
Engineering of University College London (UCL), UK. He is currently a Chair
Professor in the School of Computer Science & Electronic Engineering,
University of Essex, leading the Network Convergence Laboratory (NCL), UK. He
is also an affiliated professor at UESTC, China. Before joining in the
University of Essex at 2003, he worked at UCL on several European Union (EU)
research projects for several years. His main research interests include
wireless networks and communications, IoT networking, data and energy
integrated networks and mobile computing. He manages research projects funded
by various sources such as UK EPSRC, EU FP7/H2020 and industries. He has
published 400+ papers and filed 30 patents. He serves on the editorial boards
of both IEEE (e.g., IEEE TNSE, IEEE ComMag, IEEE WCL) and non-IEEE journals
(e.g., Deputy EiC of IET Smart Cities). He was an IEEE ComSoc Distinguished
Lecturer (2020-2021). He is a Member of Academia Europaea (MAE), a Fellow of
IEEE, a Fellow of IET and a Distinguished Member of ACM.
---|---