(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Tsinghua University, Shenzhen, China
22institutetext: State Key Lab of CAD&CG, Zhejiang University, Zhejiang, China
33institutetext: Tencent, Shenzhen, China 44institutetext: JD Explore Academy,
Beijing, China

# A General and Efficient Federated Split Learning with Pre-trained Image
Transformers for Heterogeneous Data

Yifan Shi  Equal Contribution. 1133 Yuhui Zhang  Equal Contribution. 2233
Ziyue Huang 33 Xiaofeng Yang 33 Li Shen 44 Wei Chen 22 Xueqian Wang 11

###### Abstract

Federated Split Learning (FSL) is a promising distributed learning paradigm in
practice, which gathers the strengths of both Federated Learning (FL) and
Split Learning (SL) paradigms, to ensure model privacy while diminishing the
resource overhead of each client, especially on large transformer models in a
resource-constrained environment, e.g., Internet of Things (IoT). However,
almost all works merely investigate the performance with simple neural network
models in FSL. Despite the minor efforts focusing on incorporating Vision
Transformers (ViT) as model architectures, they train ViT from scratch,
thereby leading to enormous training overhead in each device with limited
resources. Therefore, in this paper, we harness Pre-trained Image Transformers
(PITs) as the initial model, coined FES-PIT, to accelerate the training
process and improve model robustness. Furthermore, we propose FES-PTZO to
hinder the gradient inversion attack, especially having the capability
compatible with black-box scenarios, where the gradient information is
unavailable. Concretely, FES-PTZO approximates the server gradient by
utilizing a zeroth-order (ZO) optimization, which replaces the backward
propagation with just one forward process. Empirically, we are the first to
provide a systematic evaluation of FSL methods with PITs in real-world
datasets, different partial device participations, and heterogeneous data
splits. Our experiments verify the effectiveness of our algorithms.

##  1 Introduction

Federated learning (FL) [32, 24] allows distributed clients to collaboratively
train a shared model under the orchestration of the cloud without transmitting
local data in parallel. Meanwhile, Split Learning (SL) [9, 47] divides a whole
neural network into multiple subnetworks to protect the privacy between the
server and clients with limited computing resources (e.g., Internet of Things,
IoT). Nowadays, these two distributed learning paradigms gradually become
promising approaches in real-world applications, such as healthcare [35, 36,
18, 1], edge devices (e.g., mobile phones) in IoT [13, 49, 40], and wireless
network [34, 21, 4], where it is vital to protect data privacy.

To consider the limited resources of edge devices and training overhead in
terms of training time, computation, and communication costs when introducing
large language models (LLMs) into FL methods [27, 8, 50, 30, 7, 31], the
emerge of Federated Split Learning (FSL) as a new paradigm extensively appeals
to many researchers [41, 6, 15, 11, 33, 35, 1]. It provides model privacy and
reduces the training overhead, especially for each client in resource-
constrained environments by splitting the whole model into two portions:
multiple client-side models residing in local clients and a server-side model
stored in the server, meanwhile, adopting the parallel training paradigm
between the client and server similar as FL instead of sequential training in
SL.

However, similar to FL, FSL also suffers from the data heterogeneity challenge
in practical data distributions (e.g., non-IID), which can induce convergence
and model weight divergence [22, 25] as well as severe catastrophic forgetting
issues [10, 2] for FL methods. Recent works [37, 53] give insight that simply
replacing convolutional neural networks (CNN, e.g., ResNet model family) with
Vision Transformers (ViTs) can greatly improve the model performance in FL,
especially more robustness against heterogeneous data distributions due to
being capable of encoding long-range dependencies between input sequences [37,
17]. Furthermore, the model architecture of ViTs is a straightforward
configuration to decompose multiple portions, thereby being suitable for SL
settings. Thus, although prime researchers in FSL still investigate a simple
CNN architecture, several existing works in medical applications train ViTs
from scratch and generate model improvement. [35, 36, 1, 18].

Motivation. Due to immense resource requirements when training ViTs from
scratch in FSL, especially in a resource-constrained environment. Meanwhile,
to the best of our knowledge, there is no previous research systematically
analyzing the model performance under various heterogeneous data splits while
considering local computing power and model privacy such as gradient inversion
attacks [48, 16] or black-box scenarios [39, 28] where gradient information or
model parameters are inaccessible.

Contribution. In this paper, we aim to take the first investigation of
systematically evaluating the performance of Pre-trained Image Transformer
(PIT) in FSL scenarios. The main contributions are summarized as:

  * •

We incorporate the pre-trained Image Transformer (PIT) into PSL scenarios,
called FES-PIT. To further enhance gradient information protection and have
the capability compatible with black-box scenarios, we propose FES-PTZO, which
leverages zeroth-order (ZO) optimization into FES-PIT.

  * •

We are the first to evaluate FSL performance with multiple PIT models in terms
of model accuracy and convergence under various heterogeneous data
distributions in many real-world datasets.

  * •

For the systematical comparison and evaluation, we conduct comprehensive
experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets with various
PITs in terms of ViTs model family (ViT(S), ViT(T)) and distilled DeiTs model
family (DDeiT(S), DDeiT(T), DDeiT(B)) as well as DeiTs model family (DeiT(S),
DeiT(T), DeiT(B)) in non-IID settings with different data partitions, such as
Dirichlet settings with various $\alpha$ and pathological settings with
various limited classes in each client.

##  2 Related work

Federated Split Learning (FSL). Federated split learning (FSL) [46] combines
the strengths of two compelling distributed machine learning approaches: FL
and SL paradigms. It trains the global model in parallel among distributed
clients in FL and simultaneously splits the whole model as several sub-models
between clients and the server in SL. Specifically, Splitfed [45] is proposed
to eliminate the inherent drawbacks of both FL and SL, which is the first
training method in FSL. MiniBatch-SFL is presented in [15], which incorporates
MiniBatch SGD into FSL, where the clients train the client-side model in a FL
fashion while the server trains the server side and they also analyze the
convergence on the non-IID data. [33] utilize an auxiliary network to locally
update the client models while keeping only a single model on the server to
alleviate the issue of high communication overhead. [11] propose a new
direction to FL/SL based on updating the client/server-side models in parallel
via local-loss-based training specifically geared to SL. In addition, [41]
compare FL and SL settings under which each method outperforms the other in
terms of communication efficiency. Meanwhile, [6] provide empirical
comparisons of FL and SL in real-world Internet of Things (IoT) settings in
terms of learning performance and device implementation overhead.

Federated Split Vision Transformer. To maintain data privacy in medical
imaging scenarios while taking up less extensive computation resources and
bandwidth, recent works focus on the suitability of Transformers for
collaborative learning. For instance, FESTA [35] is the first method for
medical imaging classification tasks with a federated split vision
transformer. Then, FeSViBS [1] builds upon the FeSTA framework and introduces
a block sampling module, which leverages intermediate features extracted by
the Vision Transformer (ViT) at the server. [36] present a multi-task
distributed learning framework p-FESTA using ViT with random patch permutation
instead of using a CNN-based head as in FESTA. [18] enable decomposition
between the task-specific and common representations via an alternating
training strategy between clients and server.

FL with Zeroth-order Optimization. Zeroth-order (ZO) optimization [29] is a
subset of gradient-free optimization that emerges in many signal processing
and machine learning (ML) applications with only function evaluations rather
than requiring the gradient. In FL, [5] propose a derivative-free federated ZO
optimization (FedZO) algorithm featured by performing multiple local updates
based on stochastic gradient estimators in each communication round and
enabling partial device participation. [26] consider a local ZO algorithm
based on biased stochastic ZO update in a decentralized FL setting. And these
works [52, 44, 51] provide a detailed analysis of their convergence behavior
in distributed training.

The most related works to this paper are FESTA [35], FeSViBS [1], FedZO [5],
Splitfed [45], and MiniBatch-SFL [15]. However, these works still suffer from
enormous resource overhead due to training LLMs from scratch (i.e., FESTA and
FeSViBS) and inferior performance without leveraging the power of LLMs (i.e.,
Splitfed and MiniBatch-SFL), especially on heterogeneous data. Therefore, in
this paper, we try to divide the PIT model pre-trained by public data into
three parts. Specifically, we reside the PIT encoder on the server side, the
head and tail networks are allocated in each client. Finally, these modules
are updated simultaneously on the private data of each client. Furthermore, to
consider the model privacy and the ability compatible with black-box
scenarios, we utilize ZO optimization to approximate the gradient of the
server model. Finally, we are the first to systematically evaluate the
performance of our algorithms in terms of model accuracy and convergence under
various heterogeneous data distributions in many real-world datasets.

##  3 Methodology

In this section, we first define the problem setup for FSL with pre-trained
image Transformers. After that, we elaborate our algorithms: FES-PIT and FES-
PTZO.

###  3.1 Problem Setup

We consider a typical setting of FSL with $M$ clients, where each client $i$
has the data distribution $\mathcal{D}_{i}$. Let $w\in\mathbb{R}^{d}$
represent the parameters of a pre-trained transformer model and
$f_{i}(w;\xi_{c})$ is the local objective function associated with the
training data samples $\xi_{c}$. We decompose the whole transformer model as
three parts in terms of the head $w_{\mathcal{H}}^{i}$ and tail
$w_{\mathcal{T}}^{i}$ in local client $i\in\\{1,2,...,M\\}$ as well as
transformer encoder $w_{\mathcal{E}}$ in the server side, which means
$w=(w_{\mathcal{H}},w_{\mathcal{E}},w_{\mathcal{T}})$.

After that, a common objective of FSL is the following finite-sum stochastic
non-convex minimization problem:

$\min_{w}F(w)=\frac{1}{M}\sum_{i=1}^{M}\mathbb{E}_{\xi_{c}^{i,t}\sim\mathcal{D}_{i}}f_{i}(w;\xi_{c}).$
(1)

To be specific, the objective function of each client $i$ to optimize is

$\min_{w_{\mathcal{H}},w_{\mathcal{T}}}\mathbb{E}_{x_{c}^{i,t}\sim\mathcal{D}_{i}}l_{c}(y_{c}^{i,t},\mathcal{T}_{i}(w_{\mathcal{T}}^{i};\mathcal{E}(w_{\mathcal{E}};\mathcal{H}_{i}(w_{\mathcal{H}}^{i};\xi_{c}^{i,t})))).$
(2)

Where $\mathcal{H}_{i}$ and $\mathcal{T}_{i}$ represent the head network and
tail network in the local client, while the transformer encoder is viewed as
$\mathcal{E}$. The local training data is $\\{(\xi_{c}^{i,t},y_{c}^{i,t})\\}$
in the $t$-th communication round.

Meanwhile, for the transformer encoder update on the server side, the
optimization problem is

$\min_{w_{\mathcal{E}}}\frac{1}{M}\sum_{i=1}^{M}\mathbb{E}_{x_{c}^{i,t}\sim\mathcal{D}_{i}}l_{c}(y_{c}^{i,t},\mathcal{T}_{i}(w_{\mathcal{T}}^{i};\mathcal{E}(w_{\mathcal{E}};\mathcal{H}_{i}(w_{\mathcal{H}}^{i};\xi_{c}^{i,t})))).$
(3)

Therefore, for the whole model weights
$w=(w_{\mathcal{H}},w_{\mathcal{E}},w_{\mathcal{T}})$, where
$(w_{\mathcal{E}},w_{\mathcal{T}})=(\frac{1}{M}\sum_{i=1}^{M}w_{\mathcal{E}}^{i},\frac{1}{M}\sum_{i=1}^{M}w_{\mathcal{T}}^{i})$.
The objective function $f_{i}(w;\xi_{c})$ in the problem (1) can be
represented as
$l_{c}(y_{c}^{i,t},\mathcal{T}_{i}(w_{\mathcal{T}}^{i};\mathcal{E}(w_{\mathcal{E}};\mathcal{H}_{i}(w_{\mathcal{H}}^{i};\xi_{c}^{i,t}))))$.

The Challenge of FSL. In FSL, due to the whole model being decomposed into
several parts, the collaboration between these small models plays an essential
role in achieving better performance after the FL training process.

Input : Total number of clients $M$, sampling ratio of clients $q$, total
number of communication rounds $T$, learning rate $\eta$, and total number of
the local iterates are $K$. For gradient approximation, the perturbation scale
is $\epsilon$ and randomly sampled $z\in\mathbb{R}^{d}$ with
$z\sim\mathcal{N}(0,\mathbf{I}_{d})$.

Output : Generate the weights of local head and tail pairs
$(w_{\mathcal{H}}^{i},w_{\mathcal{T}}^{i})$ for each client $i$.

1 Initialization: Randomly initialize the local weights
$(w_{\mathcal{H}}^{i},w_{\mathcal{T}}^{i})$ of the head and tail pairs for
each client $i$. Meanwhile, the weights of the pre-trained body model
$w_{\mathcal{E}}^{0}$ keep frozen and start to update when the FL training
process begins.

2 for  _$t=0$ to $T-1$_ do

3 Sample a set of $m=qM$ clients at random without replacement, denoted by
$\mathcal{W}^{t}$.

4 for  _$k=1$ to $K$ _ do

5 for  _client $i=1$ to $m$ _in parallel_ _ do

6 $h_{c}^{k,t}(i)\leftarrow\textbf{ClientHead}(i,t)$. $\triangleright$ Extract
the intermediate feature from local data

7 $b_{s}^{k,t}(i)\leftarrow\mathcal{E}(h_{c}^{k,t}(i))$. $\triangleright$
Generate the smashed feature by using the server body model

8 $\mathcal{L}_{c}^{k,t}(i)\leftarrow\textbf{ClientTail}(i,b_{s}^{k,t}(i))$.
$\triangleright$ Finish the forward pass and get the prediction

9
$(w^{i,k+1}_{\mathcal{H}},w^{i,k+1}_{\mathcal{T}})\leftarrow\textbf{ClientUpdate}(i,\frac{\partial\mathcal{L}_{c}^{k,t}(i)}{\partial
h_{c}^{k,t}(i)})$. $\triangleright$ Start to backward propagation

10 if  _$k=K$_ then

11 Option I: (FES-PIT) Generate and store the server gradient by averaging
gradients: $g_{s}^{t}(i)=\frac{\partial\mathcal{L}_{c}^{K,t}(i)}{\partial
w_{\mathcal{E}}^{t}}$.

12 Option II: (FES-PTZO) Generate and store the server gradient approximation
by adopting ZO optimization:

13
$g_{s}^{t}(i)\doteq\frac{\mathcal{L}_{c}^{K,t}(i)(w_{\mathcal{E}}^{i,t}+\epsilon
z)-\mathcal{L}_{c}^{K,t}(i)(w_{\mathcal{E}}^{i,t}-\epsilon z)}{2\epsilon}z$.

14 end if

15

16 end for

17

18 end for

19 Update the weights of the server model for fine-tuning:
$w_{\mathcal{E}}^{i,t+1}\leftarrow
w_{\mathcal{E}}^{i,t}-\frac{\eta}{m}\sum_{i=1}^{m}g_{s}^{t}(i)$.

20 if  __t % fedround = 0__ then

21 Update the head/tail weights:
$(w_{\mathcal{H}}^{i},w_{\mathcal{T}}^{i})\leftarrow(\frac{1}{m}\sum_{i=1}^{m}w_{\mathcal{H}}^{i},\frac{1}{m}\sum_{i=1}^{m}w_{\mathcal{T}}^{i})$.

22 end if

23

24 end for

Algorithm 1 FES-PIT and FES-PTZO

1 /* Runs on client $i$ in $k$-th local iteration at $t$-th round */

2 ClientHead:

3 $\xi_{c}^{k,t}(i)\leftarrow$ Sample minibatch data from client $i$.

4 $h_{c}^{k,t}(i)\leftarrow\mathcal{H}_{i}(\xi_{c}^{k,t}(i))$.

5Return $h_{c}^{k,t}(i)$.

6ClientTail:

7 $y_{c}^{k,t}(i)\leftarrow$ The minibatch of the label from client $i$.

8 Compute loss value $\mathcal{L}_{c}^{k,t}(i)\leftarrow
l_{c}(y_{c}^{k,t}(i),\mathcal{T}_{i}(b_{s}^{k,t}(i)))$.

9Return $\mathcal{L}_{c}^{k,t}(i)$.

10ClientUpdate:

11 $w_{\mathcal{H}}^{i,k+1}\leftarrow
w_{\mathcal{H}}^{i,k}-\eta\frac{\partial\mathcal{L}_{c}^{k,t}(i)}{\partial
w_{\mathcal{H}}^{i,k}}$. $\triangleright$ Head back-propagation

12$w_{\mathcal{T}}^{i,k+1}\leftarrow
w_{\mathcal{T}}^{i,k}-\eta\frac{\partial\mathcal{L}_{c}^{k,t}(i)}{\partial
w_{\mathcal{T}}^{i,k}}$. $\triangleright$ Tail back-propagation

Return $(w^{i,k+1}_{\mathcal{H}},w^{i,k+1}_{\mathcal{T}})$.

Algorithm 2 Client-side Process

###  3.2 Algorithmic Details

In this subsection, we present our two algorithms: FES-PIT and FES-PTZO. Our
goal is to simultaneously optimize a server-side model (an encoder
$\mathcal{E}$ of a pre-trained image transformer) and multiple client-side
models in terms of head $\mathcal{H}_{i}$ and tail $\mathcal{T}_{i}$ networks
to make local clients $i\in\\{1,2,...,M\\}$ capable of training the large
transformer model under the constrained edge resources. Meanwhile, local
models suitable for different heterogeneous data can benefit from the pre-
trained image transformer, thereby accelerating the training process and
reducing the training overhead. Below, we elaborate on the detailed procedure
of our algorithms. The paradigm is introduced in Algorithm 1 and 2.

Pre-trained Image Transformers (PITs) on Public Data. Due to huge resource
requirements when training ViTs from scratch in FSL, especially in a resource-
constrained environment. Therefore, we utilize the public data with a similar
feature to private data and then train the PITs on the private data of each
client to reduce the training overhead in the FSL process.

Forward Propagation. During the forward propagation in both FES-PIT and FES-
PTZO algorithms, at the beginning of each communication round, each client
random initializes the local weights
$(w_{\mathcal{H}}^{i},w_{\mathcal{T}}^{i})$ of the head and tail pairs and the
encoder weights $w_{\mathcal{E}}$ at the server side retain the weights of the
pre-trained image transformer model. Then several clients are sampled to
participate in this round and perform their local iteration updates. For each
sampled client $i$, it calculates the intermediate feature $h_{c}^{k,t}(i)$
from its local minibatch data by using their local head model
$\mathcal{H}_{i}$, which means each client executes the forward propagation.
After that, all participating clients transmit their intermediate features to
the server side.

When the pre-trained transformer encoder obtains these local features on the
server side, the encoder performs forward propagation to generate the server-
side features $b^{k,t}_{s}(i)$ for corresponding client $i$ in parallel.
Finally, the server-side feature $b^{k,t}_{s}(i)$ is sent out to local client
$i$ simultaneously, and then the local tail network $\mathcal{T}_{i}$
calculates the final result $\mathcal{T}_{i}(b_{s}^{k,t}(i)))$ and loss value
$\mathcal{L}_{c}^{k,t}(i)$ according to the local data in parallel.

Backward Propagation. Different from previous works, we adopt a pre-trained
transformer encoder to accelerate the training process and only utilize the
loss value
$\\{\mathcal{L}_{c}^{K,t}(1),\mathcal{L}_{c}^{K,t}(2),...,\mathcal{L}_{c}^{K,t}(m)\\}$
of all participated clients $i\in\\{1,2,...,m\\}$ at the final local iteration
step $K$ in each communication round $t$ while having similar performance. To
massively reduce the communication overhead and local resources cost in terms
of computation and memory overheads, it is a significant improvement relative
to almost all existing FSL methods, such as classic methods: SplitFed,
minibatch-FSL, and FeSViBS.

To be specific, for FES-PIT, the target is to minimize the objective function
(2). So for client $i$, we can generate the gradients of the local head and
tail network, which is $\frac{\partial\mathcal{L}_{c}^{k,t}(i)}{\partial
w_{\mathcal{H}}^{i,k}}$ and $\frac{\partial\mathcal{L}_{c}^{k,t}(i)}{\partial
w_{\mathcal{T}}^{i,k}}$, respectively. After that, the gradients of these
local gradients are transmitted to the server. Subsequently, the transformer
encoder computes the averaged gradient:

$\small\frac{1}{M}\sum_{i=1}^{M}g_{s}^{t}(i)=\frac{1}{M}\sum_{i=1}^{M}\frac{\partial\mathcal{L}_{c}^{K,t}(i)}{\partial
w_{\mathcal{E}}^{t}}.$ (4)

Note the loss value $\mathcal{L}_{c}^{K,t}(i)$ is related to the local head
weights $w_{\mathcal{H}}^{i,K}$ and tail model weights $w_{\mathcal{T}}^{i,K}$
of client $i$, where $K$ is the final local iteration step. Then, it performs
one step of gradient update.

For FES-PTZO, the main focus is on the privacy of the model weights and
gradient information in practice, such as black-box scenarios, we adopt the
zero-order perturbation following the Simultaneous Perturbation Stochastic
Approximation (SPSA) [42, 43] to approximate the high-dimensional gradient
efficiently. For gradient approximation, let the perturbation scale is
$\epsilon\in(0,1)$ and randomly sampled is $z\in\mathbb{R}^{d}$ with
$z\sim\mathcal{N}(0,\mathbf{I}_{d})$. Thus, we obtain the server gradient by:

$g_{s}^{t}(i)\doteq\frac{\mathcal{L}_{c}^{K,t}(i)(w_{\mathcal{E}}^{i,t}+\epsilon
z)-\mathcal{L}_{c}^{K,t}(i)(w_{\mathcal{E}}^{i,t}-\epsilon z)}{2\epsilon}z.$
(5)

Note that the primary backward propagation procedure is identical to that in
FES-PIT. The sole difference is how to estimate the transformer encoder
gradient on the server side due to some important concerns, such as model
privacy.

Client-side Model Aggregation. For both algorithms, after several
communication rounds, the client-side head and tail parts need to be sent out
to the server for aggregating and updating the model weights as the same as
the general FL methods. In practice, we find that this parameter is a
significant factor in deciding the training process, thereby affecting the
final model performance.

##  4 Experiments

###  4.1 Experimental Setup

Dataset and Data Partition. We evaluate our approaches on CIFAR-10, CIFAR-100
[19], and Tiny-ImageNet [20] datasets with Dirichlet and Pathological data
partition. All detailed experiments on Tiny-ImageNet are placed in Appendix
0.B due to the limited space. For evaluating performance in various non-IID
scenarios, we partition the training and testing data according to the
Dirichlet distribution Dir($\alpha$) and Pathological data partition Path-c.
Note that the number of sampling classes is represented as “c”. For instance,
we sample 2 and 5 classes from a total of 10 classes on CIFAR-10, and 5 and 10
classes from a total of 100 classes on CIFAR-100 respectively for each client.
For the Dir($\alpha$) partition, we set $\alpha=\\{0.1,0.3\\}$ for all
datasets, where the smaller the $\alpha$ is, the more heterogeneous the
setting is. For the Path-c partition, we set $c=\\{2,5,10\\}$ in different
datasets, where the fewer classes $c$ each client has, the more heterogeneous
the setting is.

Models and Baselines. We use various PITs in terms of ViTs model family
(ViT(S), ViT(T)) and distilled DeiTs model family (DDeiT(S), DDeiT(T),
DDeiT(B)) as well as DeiTs model family (DeiT(S), DeiT(T), DeiT(B)) for all
datasets in various non-IID settings with different data partitions to verify
the effectiveness of our algorithms. For the systematical comparison, we
compare the proposed approaches with: 1) SOTA FL baselines in terms of FedAvg
[32], Fed-RoD [3], and Ditto [23]; 2) SOTA FSL baseline: FESTA [35]. All
baselines using the ResNet model family [14] replace the batch normalization
with the group normalization to avoid unstable performance.

Furthermore, to evaluate the performance of our algorithms fairly compared
with baselines using CNN models, we find that DeiT(S) is similar to ResNet-50
in terms of parameter size (MB) and FLOPs (B) as well as ImageNet result
measured by Table 2 in [12]. The compared table is shown below. Thus, we use
DeiT(S) for all algorithms using transformers and ResNet-50 for all baselines
using CNN models in the same setting, respectively.

Table 1: Comparison between ResNet-50 and DeiT(S). Note that Top-1 (%) accuracy is the result on ImageNet and the throughput is measured on NVIDIA V100 GPU and Pytorch, with $224\times 224$ input size. Model | Paras (MB) | FLOPs (B) | Throughput (image/s) | Top-1 (%)
---|---|---|---|---
ResNet-50 | 25.6 | 4.1 | 1226 | 79.1
DeiT(S) | 22 | 4.6 | 940 | 79.8

Implementation Details. We perform 500 rounds with 100 clients on CIFAR-10,
CIFAR-100, and Tiny-ImageNet with different client sampling ratios. The batch
size is 128. We set SGD [38] as the base optimizer of baselines with a
learning rate of 0.1 and local momentum of $0.9$. We report the mean
performance with 3 different seeds. More details can be found in Appendix 0.A.

Table 2:  Test accuracy (%) on CIFAR-10 & 100 in both Dirichlet and
Pathological distribution settings.

Algorithm | CIFAR-10 | CIFAR-100
---|---|---
Dirichlet | Pathological | Dirichlet | Pathological
$\alpha$ = 0.1 |  $\alpha$ = 0.3 | c = 2 | c = 5 |  $\alpha$ = 0.1 |  $\alpha$ = 0.3 | c = 5 | c = 10
FedAvg | $86.17_{\pm.28}$ | $82.02_{\pm.20}$ | $86.99_{\pm.11}$ | $83.18_{\pm.27}$ | $59.35_{\pm.03}$ | $57.12_{\pm.06}$ | $71.29_{\pm.43}$ | $68.10_{\pm.48}$
Fed-RoD | $91.15_{\pm.12}$ | $87.68_{\pm.08}$ | $92.10_{\pm.04}$ | $89.81_{\pm.45}$ | $67.79_{\pm.05}$ | $60.54_{\pm.69}$ | $82.50_{\pm.45}$ | $75.59_{\pm.15}$
Ditto | $82.22_{\pm.10}$ | $75.51_{\pm.04}$ | $86.96_{\pm.40}$ | $77.59_{\pm.32}$ | $50.85_{\pm.54}$ | $50.65_{\pm.50}$ | $71.48_{\pm.45}$ | $62.77_{\pm.30}$
FESTA | $75.16_{\pm.53}$ | $65.73_{\pm.30}$ | $86.53_{\pm.55}$ | $75.61_{\pm.17}$ | $40.98_{\pm.10}$ | $78.40_{\pm.31}$ | $69.58_{\pm.15}$ | $71.08_{\pm.52}$
FES-PIT | $96.28_{\pm.13}$ | $\textbf{96.87}_{\pm.20}$ | $94.77_{\pm.40}$ | $\textbf{97.54}_{\pm.25}$ | $\textbf{85.55}_{\pm.46}$ | $\textbf{81.49}_{\pm.25}$ | $93.71_{\pm.75}$ | $93.10_{\pm.17}$
FES-PTZO | $\textbf{96.93}_{\pm.10}$ | $95.88_{\pm.12}$ | $\textbf{96.63}_{\pm.30}$ | $96.85_{\pm.21}$ | $83.05_{\pm.41}$ | $80.51_{\pm.13}$ | $\textbf{96.77}_{\pm.30}$ | $\textbf{94.02}_{\pm.25}$

(a) CIFAR-10

(b) CIFAR-100

Figure 1: The averaged testing accuracy on CIFAR-10 and CIFAR-100 dataset for
all compared methods.

###  4.2 Performance Evaluation

Comparison with the baselines. As shown in Table 2 and Figure 1, the proposed
FES-PIT and FES-PTZO outperform other baseline methods with the best stability
and better performance in both different datasets and different data
heterogeneity scenarios. Specifically, on the CIFAR-10 dataset, FES-PIT and
FES-PTZO achieve 96.87% and 95.88% on the Directlet-0.3 setting, 9.19% and
8.20% ahead of the best-compared method Fed-RoD. On the CIFAR-100 dataset,
FES-PTZO achieves at least 14.27% and 19.43% improvement from the other
baselines on the Pathological-5 and Pathological-10 settings. We attribute
that FES-PIT and FES-PTZO have very superior performance and are considerably
suitable for the FSL setting based on the performance advantage of PITs. It is
also observed that FES-PTZO has better performance while protecting the model
privacy on the server side, demonstrating that ZO optimization is capable of
applying in distributed learning, such as FSL.

Evaluation on various non-IID scenarios. We discuss two types of data
heterogeneity: Dirichlet distribution and Pathological distribution under
various pre-trained transformer models on the CIFAR-10 dataset in Table 3, and
prove the effectiveness and robustness of the proposed methods. In Dirichlet
distribution, since the local training can’t cater for all classes inside
clients, the accuracy decreases with the level of heterogeneity decreasing. It
is clearly seen that our algorithms: FES-PIT and FES-PTZO can achieve stronger
stability for several heterogeneous settings under various pre-trained
transformer models. Pathological distribution defines limited classes for each
client which is a higher level of heterogeneity. FES-PIT and FES-PTZO still
have better performance on CIFAR-10 with only 2 categories per client.

Table 3:  Test accuracy (%) on various Dirichlet and Pathological distribution settings with different PIT models. Algorithm | Non-IID | Model
---|---|---
Distilled-DeiT | DeiT | ViT
Tiny | Small | Base | Tiny | Small | Base | Tiny | Small
Fes-PIT | Dir-0.1 | $81.45_{\pm.46}$ | $97.17_{\pm.06}$ | $97.89_{\pm.19}$ | $81.24_{\pm.20}$ | $96.29_{\pm.14}$ | $96.29_{\pm.14}$ | $80.36_{\pm.41}$ | $94.64_{\pm.45}$
Dir-0.3 | $95.28_{\pm.14}$ | $96.72_{\pm.18}$ | $97.40_{\pm.20}$ | $96.02_{\pm.16}$ | $96.97_{\pm.17}$ | $96.97_{\pm.17}$ | $74.82_{\pm.21}$ | $97.80_{\pm.30}$
Pat-2 | $90.39_{\pm.72}$ | $98.20_{\pm.40}$ | $98.86_{\pm.20}$ | $91.19_{\pm.92}$ | $91.60_{\pm.79}$ | $91.60_{\pm.79}$ | $90.80_{\pm.38}$ | $92.38_{\pm.34}$
Pat-5 | $95.87_{\pm.38}$ | $97.50_{\pm.25}$ | $98.15_{\pm.13}$ | $96.91_{\pm.51}$ | $97.54_{\pm.25}$ | $97.54_{\pm.25}$ | $81.69_{\pm.77}$ | $97.73_{\pm.35}$
Fes-PTZO | Dir-0.1 | $95.72_{\pm.10}$ | $96.99_{\pm.09}$ | $98.30_{\pm.08}$ | $79.32_{\pm.24}$ | $97.05_{\pm.21}$ | $97.05_{\pm.21}$ | $78.36_{\pm.30}$ | $78.98_{\pm.18}$
Dir-0.3 | $93.64_{\pm.42}$ | $95.96_{\pm.24}$ | $97.52_{\pm.40}$ | $71.88_{\pm.41}$ | $95.88_{\pm.17}$ | $95.88_{\pm.17}$ | $70.34_{\pm.41}$ | $71.66_{\pm.47}$
Pat-2 | $90.10_{\pm.60}$ | $98.42_{\pm.35}$ | $99.23_{\pm.22}$ | $89.72_{\pm.49}$ | $90.45_{\pm.15}$ | $90.45_{\pm.15}$ | $89.38_{\pm.28}$ | $90.28_{\pm.15}$
Pat-5 | $95.68_{\pm.71}$ | $97.17_{\pm.14}$ | $98.37_{\pm.34}$ | $76.62_{\pm.12}$ | $96.85_{\pm.26}$ | $96.85_{\pm.26}$ | $75.51_{\pm.15}$ | $76.77_{\pm.28}$

###  4.3 Ablation Study

In this subsection, we verify the influence of the main hyper-parameters in
Fed-PIT. All the ablation studies are conducted on the CIFAR-10 dataset.

(a) Effect of the local epochs.

(b) Effect of the clients’ size.

(c) Effect of fedround.

Figure 2:  Effect of the main hyper-parameters: local iteration steps $K$, the
number of participated clients $M$, and the local model aggregation interval:
fedround.

Local iteration steps $K$. In Figure 2(a), we illustrate the effect of local
epochs in different communication rounds $T$ ($T=\\{20,40,60,80,100\\}$) on
the CIFAR-10. To investigate the acceleration on $T$ by adopting a larger
local iteration steps $K$, we fix the total batchsize and change local
training epochs. With fixed communication round $T$, fewer local epochs of our
method have inferior model performance. With the increase of local epochs, the
accuracy is more superior. It is clear that under various communication rounds
$T$, the testing accuracy achieves the best value when the local epoch is set
to 5. Thus, in our paper, we set the local epoch to 5 for all algorithms.

Number of participated clients $M$. We compare the model performance between
different numbers of client participation on the CIFAR-10 with the data
distribution of Dir-0.3 in Figure 2(b). Compared with the larger participated
clients {150, 200}, the smaller participated clients {50, 100} achieve better
training convergence and testing accuracy due to the number of local training
data increasing.

Local model aggregation interval: fedround. In Figure 2(c), we illustrate the
effect of local model aggregation interval: fedround. It is obvious that the
convergence speed is fastest when fedround is set to 20 and the test accuracy
achieves the best performance when fedround is set to 50. To consider the
convergence acceleration during the model aggregation, we set fedround to 20
in this paper.

##  5 Conclusion

In this paper, we focus on the model privacy and the resource overhead issues
as well as performance improvement when training LLMs in a resource-
constrained environment, e.g., IoT. We are the first to investigate this area
and systematically confirm the performance of training PIT models with the way
of both distributed training and model splitting. For the systematical
comparison and evaluation, we conduct comprehensive experiments on CIFAR-10,
CIFAR-100, and Tiny-ImageNet datasets with various PITs in terms of ViTs model
family (ViT(S), ViT(T)) and distilled DeiTs model family (DDeiT(S), DDeiT(T),
DDeiT(B)) as well as DeiTs model family (DeiT(S), DeiT(T), DeiT(B))in non-IID
settings with different data partitions. We hope this work can provide some
insights and give a feasible direction to further study harnessing the
computational power of LLMs in the FSL community.

## References

  * [1] Almalik, F., Alkhunaizi, N., Almakky, I., Nandakumar, K.: Fesvibs: Federated split learning of vision transformer with block sampling. arXiv preprint arXiv:2306.14638 (2023)
  * [2] Chang, K., Balachandar, N., Lam, C., Yi, D., Brown, J., Beers, A., Rosen, B., Rubin, D.L., Kalpathy-Cramer, J.: Distributed deep learning networks among institutions for medical imaging. Journal of the American Medical Informatics Association 25(8), 945–954 (2018)
  * [3] Chen, H.Y., Chao, W.L.: On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00778 (2021)
  * [4] Deng, X., Li, J., Ma, C., Wei, K., Shi, L., Ding, M., Chen, W., Poor, H.V.: Blockchain assisted federated learning over wireless channels: Dynamic resource allocation and client scheduling. IEEE Transactions on Wireless Communications (2022)
  * [5] Fang, W., Yu, Z., Jiang, Y., Shi, Y., Jones, C.N., Zhou, Y.: Communication-efficient stochastic zeroth-order optimization for federated learning. IEEE Transactions on Signal Processing 70, 5058–5073 (2022)
  * [6] Gao, Y., Kim, M., Abuadbba, S., Kim, Y., Thapa, C., Kim, K., Camtepe, S.A., Kim, H., Nepal, S.: End-to-end evaluation of federated learning and split learning for internet of things. arXiv preprint arXiv:2003.13376 (2020)
  * [7] Guo, T., Guo, S., Wang, J.: pfedprompt: Learning personalized prompt for vision-language models in federated learning. In: Proceedings of the ACM Web Conference 2023. pp. 1364–1374 (2023)
  * [8] Guo, T., Guo, S., Wang, J., Tang, X., Xu, W.: Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing (2023)
  * [9] Gupta, O., Raskar, R.: Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications 116, 1–8 (2018)
  * [10] Gupta, S., Singh, P., Chang, K., Qu, L., Aggarwal, M., Arun, N., Vaswani, A., Raghavan, S., Agarwal, V., Gidwani, M., et al.: Addressing catastrophic forgetting for medical domain expansion. arXiv preprint arXiv:2103.13511 (2021)
  * [11] Han, D.J., Bhatti, H.I., Lee, J., Moon, J.: Accelerating federated learning with split learning on locally generated losses. In: ICML 2021 workshop on federated learning for user privacy and data confidentiality. ICML Board (2021)
  * [12] Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al.: A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence 45(1), 87–110 (2022)
  * [13] Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., Ramage, D.: Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 (2018)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] Huang, C., Tian, G., Tang, M.: When minibatch sgd meets splitfed learning: Convergence analysis and performance evaluation. arXiv preprint arXiv:2308.11953 (2023)
  * [16] Huang, Y., Gupta, S., Song, Z., Li, K., Arora, S.: Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems 34, 7232–7241 (2021)
  * [17] Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.: Transformers in vision: A survey. ACM computing surveys (CSUR) 54(10s), 1–41 (2022)
  * [18] Kim, B., Kim, J., Ye, J.C.: Task-agnostic vision transformer for distributed learning of image processing. IEEE Transactions on Image Processing 32, 203–218 (2022)
  * [19] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [20] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), 3 (2015)
  * [21] Li, J., Shao, Y., Wei, K., Ding, M., Ma, C., Shi, L., Han, Z., Poor, H.V.: Blockchain assisted decentralized federated learning (blade-fl): Performance analysis and resource allocation. IEEE Transactions on Parallel and Distributed Systems 33(10), 2401–2415 (2021)
  * [22] Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10713–10722 (2021)
  * [23] Li, T., Hu, S., Beirami, A., Smith, V.: Ditto: Fair and robust federated learning through personalization. In: International Conference on Machine Learning. pp. 6357–6368. PMLR (2021)
  * [24] Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine pp. 50–60 (2020)
  * [25] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, 429–450 (2020)
  * [26] Li, Z., Chen, L.: Communication-efficient decentralized zeroth-order method on heterogeneous data. In: 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP). pp. 1–6. IEEE (2021)
  * [27] Lin, B.Y., He, C., Zeng, Z., Wang, H., Huang, Y., Dupuy, C., Gupta, R., Soltanolkotabi, M., Ren, X., Avestimehr, S.: Fednlp: Benchmarking federated learning methods for natural language processing tasks. arXiv preprint arXiv:2104.08815 (2021)
  * [28] Lin, Z., Sun, Y., Shi, Y., Wang, X., Huang, L., Shen, L., Tao, D.: Efficient federated prompt tuning for black-box large pre-trained models. arXiv preprint arXiv:2310.03123 (2023)
  * [29] Liu, S., Chen, P.Y., Kailkhura, B., Zhang, G., Hero III, A.O., Varshney, P.K.: A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine 37(5), 43–54 (2020)
  * [30] Liu, X., Pang, T., Fan, C.: Federated prompting and chain-of-thought reasoning for improving llms answering. arXiv preprint arXiv:2304.13911 (2023)
  * [31] Lu, W., Hu, X., Wang, J., Xie, X.: Fedclip: Fast generalization and personalization for clip in federated learning. arXiv preprint arXiv:2302.13485 (2023)
  * [32] Mcmahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.Y.: Communication-efficient learning of deep networks from decentralized data pp. 1273–1282 (2017)
  * [33] Mu, Y., Shen, C.: Communication and storage efficient federated split learning. arXiv preprint arXiv:2302.05599 (2023)
  * [34] Niknam, S., Dhillon, H.S., Reed, J.H.: Federated learning for wireless communications: Motivation, opportunities, and challenges. IEEE Communications Magazine 58(6), 46–51 (2020)
  * [35] Park, S., Kim, G., Kim, J., Kim, B., Ye, J.C.: Federated split vision transformer for covid-19 cxr diagnosis using task-agnostic training. arXiv preprint arXiv:2111.01338 (2021)
  * [36] Park, S., Ye, J.C.: Multi-task distributed learning using vision transformer with random patch permutation. IEEE Transactions on Medical Imaging (2022)
  * [37] Qu, L., Zhou, Y., Liang, P.P., Xia, Y., Wang, F., Adeli, E., Fei-Fei, L., Rubin, D.: Rethinking architecture design for tackling data heterogeneity in federated learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10061–10071 (2022)
  * [38] Robbins, H., Monro, S.: A stochastic approximation method. Annals of Mathematical Statistics 22(3), 400–407 (1951)
  * [39] Roumeliotis, K.I., Tselikas, N.D.: Chatgpt and open-ai models: A preliminary review. Future Internet 15(6), 192 (2023)
  * [40] Shi, Y., Wei, K., Shen, L., Li, J., Wang, X., Yuan, B., Guo, S.: Efficient federated learning with enhanced privacy via lottery ticket pruning in edge computing. arXiv preprint arXiv:2305.01387 (2023)
  * [41] Singh, A., Vepakomma, P., Gupta, O., Raskar, R.: Detailed comparison of communication efficiency of split learning and federated learning. arXiv preprint arXiv:1909.09145 (2019)
  * [42] Spall, J.C.: Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control 37(3), 332–341 (1992)
  * [43] Spall, J.C.: A one-measurement form of simultaneous perturbation stochastic approximation. Automatica 33(1), 109–112 (1997)
  * [44] Tang, Y., Zhang, J., Li, N.: Distributed zero-order algorithms for nonconvex multiagent optimization. IEEE Transactions on Control of Network Systems 8(1), 269–281 (2020)
  * [45] Thapa, C., Arachchige, P.C.M., Camtepe, S., Sun, L.: Splitfed: When federated learning meets split learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 8485–8493 (2022)
  * [46] Thapa, C., Chamikara, M.A.P., Camtepe, S.A.: Advancements of federated learning towards privacy preservation: from federated learning to split learning. Federated Learning Systems: Towards Next-Generation AI pp. 79–109 (2021)
  * [47] Vepakomma, P., Gupta, O., Swedish, T., Raskar, R.: Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564 (2018)
  * [48] Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., Qi, H.: Beyond inferring class representatives: User-level privacy leakage from federated learning. In: IEEE INFOCOM 2019-IEEE conference on computer communications. pp. 2512–2520. IEEE (2019)
  * [49] Wei, K., Li, J., Ding, M., Ma, C., Su, H., Zhang, B., Poor, H.V.: User-level privacy-preserving federated learning: Analysis and performance optimization. IEEE Transactions on Mobile Computing 21(9), 3388–3401 (2021)
  * [50] Yu, S., Muñoz, J.P., Jannesari, A.: Federated foundation models: Privacy-preserving and collaborative learning for large models. arXiv preprint arXiv:2305.11414 (2023)
  * [51] Zhang, S., Dong, Y., Xie, D., Yao, L., Bailey, C.P., Fu, S.: Convergence analysis of nonconvex distributed stochastic zeroth-order coordinate method. In: 2021 60th IEEE Conference on Decision and Control (CDC). pp. 1180–1185. IEEE (2021)
  * [52] Zhang, S., Shen, T., Sun, H., Dong, Y., Xie, D., Zhang, H.: Zeroth-order stochastic coordinate methods for decentralized non-convex optimization. arXiv preprint arXiv:2204.04743 (2022)
  * [53] Zuo, X., Zhang, Q., Han, R.: An empirical analysis of vision transformer and cnn in resource-constrained federated learning. In: Proceedings of the 2022 5th International Conference on Machine Learning and Machine Intelligence. pp. 8–13 (2022)

Supplementary Material for “ A General and Efficient Federated Split Learning
with Pre-trained Image Transformers for Heterogeneous Data ”

##  Appendix 0.A More Details in the Implementation.

###  0.A.1 Datasets and Data Partition

CIFAR-10/100 and Tiny-ImageNet are three basic datasets in machine learning.
As shown in Table 4, they are all colorful images with different classes and
different resolutions. We use two non-IID partition methods to split the
training data in our implementation. One is based on Dirichlet distribution on
the label ratios to ensure data heterogeneity among clients. The Dirichlet
distribution defines the local dataset to obey a Dirichlet distribution, where
a smaller $\alpha$ means higher heterogeneity. Another assigns each client a
limited number of categories, called Pathological distribution. Pathological
distribution defines the local dataset to obey a uniform distribution of
active categories $c$, where fewer categories mean higher heterogeneity. The
distribution of the test datasets is as the same as in training datasets. We
run 500 communication rounds for CIFAR-10, CIFAR-100, and 300 rounds for Tiny-
ImageNet.

Table 4: The details on three datasets. Dataset | Training Data | Test Data | Class | Size
---|---|---|---|---
CIFAR-10 | 50,000 | 10,000 | 10 | 3×32×32
CIFAR-100 | 50,000 | 10,000 | 100 | 3×32×32
Tiny-ImageNet | 100,000 | 10,000 | 200 | 3×64×64

###  0.A.2 More details about the Fed-PIT and Fed-PTZO

For Fed-PIT and Fed-PTZO, we train the model for 5 epochs per round as the
same as other baselines. Therefore, we conduct ablation experiments to select
the most appropriate parameters for each data distribution. We set fedround to
50. Model parameters are optimized by Adam with a learning rate decay of
0.0002. The weight perturbation ratio in Fed-PTZO is set to zero in the
experiments. Furthermore, in Fed-PTZO, the main hyper-parameters of ZO
optimization are set to $\epsilon=0.0001$, the learning rate of the server-
side model $lr_{server}=1e-6$ with the decaying rate 0.5 and the learning rate
of the client model $lr_{client}=2e-4$.

###  0.A.3 More Details about Baselines

FedAvg [32] is the most commonly discussed method in FL. It selects some
clients to perform local training on each dataset and then aggregates the
trained local models to update the global model. Actually, the local model in
FedAvg is also the comparable personalized model for each client.

Fed-RoD [3] explicitly decouples a model’s dual duties with two prediction
tasks: generic optimization and personalized optimization and utilizes a hyper
network to connect the generic model and the personalized model. Each client
first updates the generic model with balanced risk minimization then updates
the personalized model with empirical risk minimization.

Ditto [23] achieves personalization via a trade-off between the global model
and local objectives. It totally trains two models on the local datasets, one
for the global model (similarly aggregated as in FedAvg) with local empirical
risk, and one for the personal model (kept locally) with both empirical and
proximal terms towards the global model. We perform a grid search (0.1, 0.4,
0.75, 1) of the regularization parameters $\lambda$, then report the best 0.75
in our experiments.

FESTA [35] is the first method for medical imaging classification tasks with a
federated split vision transformer. We train the model for 5 epochs per round.
Therefore, we conduct ablation experiments to select the most appropriate
parameters for each data distribution. We set fedround to 50. Model parameters
are optimized by Adam with a learning rate decay of 0.0002. The weight
perturbation ratio is set to zero in the experiments.

##  Appendix 0.B More Details in the Experiment.

Figure 3: Tiny-ImageNet Figure 4: The averaged testing accuracy on Tiny-ImageNet dataset for all compared methods.  Table 5:  Test accuracy (%) on Tiny-ImageNet in both Dirichlet and Pathological distribution settings. Algorithm | Tiny-ImageNet
---|---
Dirichlet | Pathological
$\alpha$ = 0.1 |  $\alpha$ = 0.3 | c = 5 | c = 10
FedAvg | 14.13 $\pm$ 0.13 | 7.42 $\pm$ 0.21 | 60.97 $\pm$ 0.33 | 46.56 $\pm$ 0.39
Ditto | 23.71 $\pm$ 0.66 | 16.47 $\pm$ 0.14 | 49.84 $\pm$ 0.75 | 42.65 $\pm$ 0.15
FedRoD | 31.03 $\pm$ 0.55 | 21.25 $\pm$ 0.45 | 56.42 $\pm$ 0.47 | 50.01 $\pm$ 0.40
FESTA | 36.75 $\pm$ 0.73 | 27.10 $\pm$ 0.18 | 66.79 $\pm$ 0.50 | 65.45 $\pm$ 0.25
Fed-PIT |  89.35 $\pm$ 0.30 |  87.77 $\pm$ 0.08 | 96.77 $\pm$ 0.10 | 95.70 $\pm$ 0.50
Fed-PTZO | 87.10 $\pm$ 0.11 | 85.40 $\pm$ 0.50 |  96.83 $\pm$ 0.16 |  96.41 $\pm$ 0.33

Comparison with the baselines. As shown in Table 5 and Figure 4, the proposed
FES-PIT and FES-PTZO outperform other baseline methods with the best stability
and better performance in both different datasets and different data
heterogeneity scenarios. Specifically, FES-PIT and FES-PTZO achieve 87.77% and
85.40% on the Directlet-0.3 setting, which are very far ahead of the best-
compared method Fed-RoD. We attribute that FES-PIT and FES-PTZO have very
superior performance and are considerably suitable for the FSL setting based
on the performance advantage of PITs.

Evaluation on various non-IID scenarios. We discuss two types of data
heterogeneity: Dirichlet distribution and Pathological distribution under
various pre-trained transformer models in Table 5, and prove the effectiveness
and robustness of the proposed methods. In Dirichlet distribution, since the
local training can’t cater for all classes inside clients, the accuracy
decreases with the level of heterogeneity decreasing. It is clearly seen that
our algorithms: FES-PIT and FES-PTZO can achieve stronger stability for
several heterogeneous settings under various pre-trained transformer models.


# Improving on the Markov-Switching Regression Model by the Use of an Adaptive
Moving Average

Piotr Pomorski, Denise Gorse
Department of Computer Science
University College London
London
{P.Pomorski<EMAIL_ADDRESS>

###### Abstract

Regime detection is vital for the effective operation of trading and
investment strategies. However, the most popular means of doing this, the two-
state Markov-switching regression model (MSR), is not an optimal solution, as
two volatility states do not fully capture the complexity of the market. Past
attempts to extend this model to a multi-state MSR have proved unstable,
potentially expensive in terms of trading costs, and can only divide the
market into states with varying levels of volatility, which is not the only
aspect of market dynamics relevant to trading. We demonstrate it is possible
and valuable to instead segment the market into more than two states not on
the basis of volatility alone, but on a combined basis of volatility and
trend, by combining the two-state MSR with an adaptive moving average. A
realistic trading framework is used to demonstrate that using two selected
states from the four thus generated leads to better trading performance than
traditional benchmarks, including the two-state MSR. In addition, the proposed
model could serve as a label generator for machine learning tasks used in
predicting financial regimes ex ante.

_K_ eywords Regime switching $\cdot$ Technical analysis $\cdot$ Markov model
$\cdot$ Trading

##  1 Introduction

Financial markets are characterised by periods of evolving, low-volatility
growth separated with a disruptive, high-volatility contractions. These
regimes can be distinguished by significant changes in asset returns,
variances, and correlations, laying a groundwork for accurate detection
techniques to be exploited by portfolio managers. One such detection technique
is the _Markov-switching regression model_ (MSR), introduced by Goldfeld &
Quandt in 1973 [1], and later extended by Hamilton in 1989 [2] and Krolzig in
1997 [3], which has since become one of the most popular statistical methods
to distinguish regime shifts in economics and finance. However, the original
two-state MSR model is not ideal, as two volatility states are insufficient to
describe the complexities of market dynamics, and attempts to use a three- or
higher state MSR can suffer from instability due to overly frequent regime
switches [4], which in addition adversely impact portfolio returns by
increasing the costs of trading. Furthermore, volatility by itself is not an
infallible indicator of up- or down-trending markets. This paper improves on
the current state of the art by combining the two-state MSR with the use of
_Kaufman’s adaptive moving average_ (KAMA) [5], a method that is both stable
(avoiding spurious regime shift detections) and accurate in locating the times
of onset of regime switches. This combination generates four detected regimes
based not on volatility alone, allowing, ultimately, a focus on those two
market states with the greatest trading value, namely low-variance bullish and
high-variance bearish, with trading results that will demonstrate the utility
of the method.

##  2 Background and Related Work

###  2.1 Two-State Markov-Switching Regression (MSR) Model

The model to be described here is also known as the _two-state Markov-
switching dynamic regression model_ [3] and will be used throughout this paper
as a fundamental part of the proposed regime-switching model, as well as (in
its unenhanced version) one of the models used for performance comparison.

The key to the MSR model’s widespread use is its ability to incorporate
observed characteristics of asset returns, such as asymmetries,
autocorrelation, and volatility clustering [6]. This gives a way to separate
the underlying process $y_{t}$ into $S_{t}$ states (’regimes’) at times
$t\in\\{0,1,…,T\\}$, where $S_{t}\in\\{0,1,…,k\\}$. Assuming the process
$y_{t}$ refers to daily log returns $\ln r_{t}$, the model can be specified as

$\ln r_{t}=\mu_{S_{t}}+\ln r_{t-1}\beta_{S_{t}}+\sigma_{S_{t}}\epsilon_{t},\ \
\epsilon_{t}\sim N(0,1),$ (1)

where $\mu_{S_{t}}$ is a state-dependent intercept, $\beta_{S_{t}}$ is a
state-dependent coefficient of lagged log returns, and $\sigma_{S_{t}}$ is a
state-dependent volatility [3]. With $S_{t}\in\\{0,1\\}$, the governing
dynamics of the underlying regime $S_{t}$ are considered to follow a time-
homogeneous Markov chain with fixed transition probabilities $p,q\in[0,1]$,

$\begin{pmatrix}p&{1-p}\\\ {1-q}&q\end{pmatrix}$

where

$p=Pr(S_{t}=0|S_{t-1}=0),\ \ q=Pr(S_{t}=1|S_{t-1}=1).$ (2)

If $(\delta,1-\delta)$, with $\delta=Pr(S_{t}=0)\in[0,1]$, is the initial
distribution of the Markov chain, then the model is completely specified by
the vector of parameters
$\theta=(p,q,\mu_{0},\mu_{1},\beta_{0},\beta_{1},\sigma_{0},\sigma_{1},\delta)$
[4], and the estimation of $\theta$ is performed using Gibbs sampling, a type
of Markov chain Monte Carlo algorithm described in [7].

If $\hat{\theta}_{t}$ denotes the estimate of $\theta_{t}$ , then for the two
volatility states $i\in\\{0,1\\}$ the probabilities at time
$t\in\\{0,1,…,T\\}$ are given by

$p_{t}^{i}=Pr(S_{t}=i\ |\ \ln r_{t};\hat{\theta}_{t}),$ (3)

in which $S_{t}=0$ and $S_{t}=0$ are the low and high volatility periods,
respectively. $p_{t}^{i}=Pr(S_{t}=i\ |\ \ln r_{t};\hat{\theta}_{t})$ estimates
the probability of a low ($i=0$) or high ($i=1$) volatility regime at time
$t$; the higher the $p_{t}^{i}$, the more likely the process is persistent
[4].

###  2.2 Kaufman’s Adaptive Moving Average (KAMA)

The popularity of moving averages in trading strategies has prompted the
creation of many different forms of these trend-following indicators. Some of
these, such as simple moving averages, are better at smoothing the underlying
price, and some, such as the zero-lag moving average, are better at minimising
the lag, though these objectives cannot be achieved simultaneously. An
adaptive moving average is, however, a specific class of moving average that
attempts to do so, accounting for both trend and volatility factors to balance
the smooth-lag issue [8]. (Note: From this point on, this paper will use the
terms ’adaptive moving average’, ’Kaufman’s adaptive moving average’, and the
abbreviation ’KAMA’ interchangeably.)

The calculation of KAMA begins with determining an _efficiency ratio_ (ER).
The ER helps KAMA identify and adapt to ever-changing asset conditions by
measuring the relative speed of price movement from one period to another.
Assuming a daily frequency over time $t\in\\{0,1,…,T\\}$ , the ER is defined
by

$ER_{t}=\frac{M_{t}}{V_{t}},$ (4)

where

$M_{t}=P_{t}-P_{t-n},\ \ V_{t}=\sum_{i=1}^{n}|P_{t}-P_{t-1}|,$ (5)

and the momentum, $M_{t}$, is the change in closing price $P$ over a period of
length $n$ ($n$-period) and the volatility, $V_{t}$, is the sum of the
absolute value of daily closing price changes during an $n$-period. The
efficiency ratio is constrained to $0\leq ER\leq 1$, such that a significant
price change as a proportion of low volatility will bring ER closer to 1,
whereas a minor price change as a proportion of high volatility will yield an
ER closer to 0. In other words, values close to 1 indicate a clearly defined
trend in the price, whereas values nearing 0 indicate a consolidating and
directionless market.

The ER is embedded into KAMA’s full formula as a part of its scaled smoothing
coefficient $C$. Considering again the daily frequency over time
$t\in\\{0,1,…,T\\}$, the adaptive moving average is computed as

$KAMA_{t}=KAMA_{t-1}+C_{t}(P_{t}-KAMA_{t-1}),$ (6)

where the scaled smoothing coefficient $C$ over time $t$ is expressed as

$C_{t}=[ER_{t}(k_{s}-k_{l})+k_{s}]^{2}.$ (7)

$k_{s}$ and $k_{l}$ in the above are smoothing constants relevant to a pair of
different (short-term and long-term) simple moving averages over $n$ period,
calculated as

$k_{s}=\frac{2}{n_{s}+1},\ \ k_{l}=\frac{2}{n_{l}+1},$ (8)

where $n_{s}$ and $n_{l}$ refer to shorter and longer time windows,
respectively [5].

By combining the trending aspect of moving averages and the volatility factor
of the ER, KAMA is able to both identify the global price trend and to
accurately locate larger turning points. Thus, the potential trader would not
need to so frequently switch position by buying and selling over local price
swings [5]. MSR models also seek to determine the current state of the process
with highest probability, in order to avoid switching states too often.
However, the main difference is that KAMA is primarily used to capture the
global trend, bullish or bearish, whereas an MSR model detects disruptions in
the variance of the underlying series.

###  2.3 Related Work

Since they were first introduced by [1], MSR models have been discussed
broadly in economic literature. Their performance has been analysed on
macroeconomic variables, such as GDP [2] and inflation [9], and they have been
additionally applied to financial time series, such as equities [5] and
foreign exchange [8].

The original, two-state MSR model is, however, not optimal, for reasons
discussed in the Introduction. However, as also discussed previously,
extension to three-state MSR models comes at a cost, since they are deemed
unstable due to their tendency to shift regimes too frequently [4]. There are
also concerns about potential over-fitting, as beyond the two-state model it
is necessary to pre-estimate the optimal number of states, applying each
version of the MSR model to each asset in order to find the optimal number of
states [10], and hoping that the same optimal number of regimes will continue
in the future. Finally, these multi-state MSR models do not solve the issue of
overly frequent switches—on the contrary, there is a high potential of
increasing them.

However, there is another potential solution to finding a better partition of
the market dynamics [11], which is to overlay the two-state model with moving
averages, which method allows partitioning without frequent state switches,
resulting in a four-state model in line with classic Wyckoff theory [12]. But
although the model of [11] has been found to add value in equity strategies,
it is constrained by the data required to calculate a necessary technical
indicator, which requires period-high and -low prices. This limits application
of the model to assets with a recorded history of high and low prices, even
when their closing prices have been available for a much longer time. The work
of this paper has taken inspiration from that of [11], replacing the custom
Keltner Channels [13] in that model with KAMA, which does not require the
recording of high and low prices; as will be seen in the Results section, the
new model is both highly effective and applicable to a wide range of financial
assets.

##  3 Data

Setting up the proposed and benchmark models requires daily closing price data
only. There are 56 assets in total, divided into four classes: equities (24
datasets), exchange rates (FX) (13 datasets), commodities (12 datasets), and
fixed income (seven datasets). The choice of these assets is based on the
extent of their use in the financial industry (for instance, major equity
markets for stock indices, G10 countries for FX), to ensure there will be
sufficient liquidity to minimise the risk of market manipulation. In addition
to the above-mentioned assets, a cash index is also used in the trading
strategy detailed later in the Methodology section. The rationale behind the
use here of a cash index stems from its low volatility and ever-rising trend.
With the rare exception of periods of negative interest rates, such as after
the financial crisis of 2008-2009, it is nearly impossible to lose whilst
betting on cash in the longer term, which makes it a useful ’money-parking’
tool. The start dates for the datasets vary, though are usually in the
1980s-1990s. The end dates are 26/03/2021 for equities, foreign exchange,
commodities, and the cash index, and 08/01/2021 for fixed income. For each
asset, that 85 % of the data closest to the start date is used for
optimisation of the KAMA model, using procedures to be described below, while
that 15 % of the data closest to the end date is used for out of sample
testing.

##  4 Methodology

This section is constructed as follows: first, the process of combining the
two-state MSR model with KAMA is described. Second, the optimisation of the
KAMA component within the combined model is outlined. Third, benchmark models
used for performance comparison are briefly discussed, and finally the trading
strategy, which assesses the tested models’ abilities to separate regimes, is
described.

###  4.1 Combining the Two-State MSR Model with KAMA

The proposed regime-switching model is initiated with a two-state MSR model to
detect high and low variance periods for each selected asset. Considering
states $S_{t}\in\\{0,1\\}$, and the 50 % level as a cut-off point for smoothed
probabilities in the MSR model, this initial phase of model construction
results in the separation:

  * •

Low variance regimes, defined as ones where the filtered probability of state
$S_{t}=0$ is higher than 50 %.

  * •

High variance regimes, defined as ones where the filtered probability of state
$S_{t}=1$ is higher than 50 %.

KAMA then works as an overlay to divide these low and high volatility periods
into bullish and bearish regimes. However, its practical application also
requires a mechanism to generate trading decisions, and for this purpose
Kaufman’s adaptive moving average is embedded within a construction called the
filter which generates a signal to enter or exit positions of interest. Over a
$t$-day period, the filter $f$ is computed as $f_{t}=\gamma\sigma(KAMA_{t})$,
where

$\sigma(KAMA_{t})=\sqrt{({\sum_{t=1}^{n}x_{t}^{2}}-\frac{{\sum_{t=1}^{n}x_{t}}}{n})}\
,$ (9)

is the standard deviation of the change in KAMA over $n$ days, where $n\leq t$
, $x_{t}=KAMA_{t}-KAMA_{t-1}$, and the parameter $\gamma$ is as discussed in
the Background section.

Calculating both KAMA and the filter allows the construction of a strategy for
trading in both bullish and bearish regimes:

  * •

Bullish, hence buy, when KAMA advances above its low over a prior period of
$n$ days by a value greater than the filter.

  * •

Bearish, hence sell, when KAMA descends below its low over a prior period of
$n$ days by a value greater than the filter

Combining the above with the MSR model’s results, the proposed regime-
switching model, which we will refer to as the _KAMA+MSR_ model, separates the
underlying price series into four regimes:

  * •

Low variance and bullish when the filtered probability of state $S_{t}=0$ is
higher than 50 %, and KAMA rises above its low over a prior period of $n$ days
by a value greater than the filter.

  * •

Low variance and bearish when the filtered probability of state $S_{t}=0$ is
higher than 50 %, and KAMA falls below its low over a prior period of $n$ days
by a value greater than the filter.

  * •

High variance and bullish when the filtered probability of state $S_{t}=1$ is
higher than 50 %, and KAMA rises above its low over a prior period of $n$ days
by a value greater than the filter.

  * •

High variance and bearish when the filtered probability of state $S_{t}=1$ is
higher than 50 %, and KAMA falls below its low over a prior period of $n$ days
by a value greater than the filter.

However, while all four regimes are of academic interest, not all are equally
valuable for trading. The second and third are likely to be less useful
because in the first of these cases the potential for profit is low, while in
the second the risk is overly high. In preliminary experiments this was
confirmed by the low Sharpe ratios obtainable during these periods. Thus, in
the trading results to be reported in Section 5, active trading is carried out
only in the first and fourth of the above-listed regimes.

###  4.2 Optimisation of the KAMA Component of the Model

The MSR component of the KAMA+MSR model does not require optimisation, as it
is parameter-free. In contrast, the KAMA model requires the optimisation, over
the training period that comprises the initial 85 % of the data, of
$\theta_{h}=(n,n_{s},n_{l},\gamma)$, that vector of parameters discussed in
the Background section, where $n$ is a moving window for the efficiency ratio
ER and the filter $f$, $n_{s}$ is a moving window for a short-term smoothing
constant $k_{s}$, $n_{l}$ is a moving window for a long-term smoothing
constant $k_{l}$, and $\gamma$ is the control parameter in the filter $f$
term.

Beginning with the initial vector of parameters $\theta_{h}$, the algorithm
constructs the proposed regime-switching model by overlaying the calculated
KAMA on a two-state Markov-switching dynamic regression. Based on the outcome
of this initial combination, the model splits each asset price into multiple
periods (’segments’) which are classified as being in one of the four regimes.
Subsequently, the price slope (bullish vs. bearish) and log returns volatility
(low vs. high variance) of each segment is computed, so that the optimised
model’s accuracy can be inspected. The parameter values are then adjusted to
optimise performance during the training period.

The test of meaningful separation of regimes is done using a method based on
K-Means clustering [14], with a two-step walk-forward cross-validation and a
custom scoring function. In order to implement cross-validation, the regime
slope and volatility data is split into non-shuffled training and validation
sets, whereby training is 50 % and 75 % of the training dataset in the first
and second steps, respectively, and the consecutive 25 % of the training data
are used for validation in both steps. The clustering ability of K-Means is
employed on the sliced training sets to investigate whether the already
refined regimes differ by slopes and volatilities. Given that this will be a
four-state model, K-Means here assumes four clusters; the clustering algorithm
additionally uses ten random initialisations of centroids, the same random
seed for each model, and 300 maximum iterations per initialisation. The
K-Means method then predicts clusters for the validation sets, and the
generated predictions are contrasted with the model’s initially chosen regime
labels.

To achieve this, the KAMA parameter optimisation algorithm uses a custom
function called the _misclassification score_. This scoring function groups
clusters and regime labels into a $4\times 4$ matrix in order to analyse the
dominant number of detected segments (periods of time that fall into a given
class) within each row and column, as in Table 1, and perform appropriate
actions.

Table 1: Example grouping matrix; matrix entries denote the number of detected segments Cluster |  | Low variance /
---
bullish
| Low variance /
---
bearish
| High variance /
---
bullish
| High variance /
---
bearish
Cluster 1 | 1 | 0 | 0 | 0
Cluster 2 | 0 | 5 | 1 | 9
Cluster 3 | 0 | 0 | 2 | 0
Cluster 4 | 3 | 0 | 3 | 0

The misclassification score initially loops over rows to subtract the dominant
number of segments from the row sum. Thus, in the example, in Clusters 1 and
3, since there is only one regime detected, the score is 0, which indicates
lack of misclassification within this cluster. However, in Cluster 2, there
are 6 segments (out of 15) whose underlying slope and volatility
characteristics do not resemble the dominant label, high variance and bearish,
so the score is 6. In Cluster 4, there are two dominant labels, each
mismatched (with respect to the other) with a score of 3, and we would in a
case like this assign an overall score for the cluster of 3. The number of
misclassifications per cluster is then summed up to indicate how many segments
have been inaccurately labelled by the optimised model. The total number is
finally divided by the length of the predicted dataset to form a ratio that
can be compared between optimisation trials. Ideally, the final
misclassification score should equal 0, representing a perfect alignment
between the K-Means discovered clusters and the proposed regime-switching
model, though this degree of alignment is unlikely in practice and hence the
optimisation algorithm is run 50 times to find the best-available vector of
parameters $\theta_{h}$ for the training period, which are saved for use in
the test period.

###  4.3 Benchmark Models

The most obvious candidate as a benchmark is the two-state MSR model, being a
component of the proposed KAMA+MSR model; comparison of MSR with the proposed
model will reveal the value of the KAMA component. Initial experimentation
determined that the model of [11], developed for equities, was not effective
over the full range of asset classes considered in this work. In addition,
this model requires knowledge of price highs and lows, and this data was not
available for the full desired training period, that stretched back into the
1980s-1990s. This model was thus set aside. Additionally, while a three-state
MSR model [15] was considered, it was found to be unstable (as discussed
previously, multi-state MSR models typically have this associated difficulty)
but to be effective when its three states were reduced to two, by removing
that state which had medium variance, and it was in this form included in the
set of benchmark models. It should be noted that the two chosen benchmark
models, like the MSR component of the proposed model, are parameter-free and
hence do not need a training period parameter optimisation.

###  4.4 Implementing a Trading Strategy for Performance Testing

The trading strategy requires an allocation of weights (adding to 100 %)
between the asset and cash. These are optimised over the training period
(earliest 85 % of the data, for each asset), by comparing the effectiveness of
1000 random weight allocations. Both annualised returns and the adjusted
Sharpe ratio (ASR) [16]

$ASR_{t}=MAR_{t}/\sigma^{\frac{MAR_{t}}{abs(MAR_{t})}}$ (10)

are calculated, on a per-segment basis. The adjusted Sharpe ratio is a
valuable measure because it penalises negative variance especially strongly,
while treating positive returns in the same manner as the standard Sharpe
ratio. $MAR_{t}$ in the above denotes mean excess annualised return over $t$
days, $\sigma$ is the annualised volatility, and $abs(MAR_{t})$ denotes mean
excess annualised return over $t$ days calculated from only positive excess
returns. In this work, mean excess annualised returns equal mean annualised
returns, as for simplification the risk-free rate is set to 0.

Inevitably, the asset returns are diminished by two-way transaction costs,
i.e., ones incurred during both buying and selling. Table 2 lists the costs
assumed in this work, as given in [17] (these values being considered in the
industry to be conservative and useful estimates of costs):

Table 2: Two-way (buy, sell) trading costs for each asset class Asset class |  | Brokerage
---
commissions (%)
| Bid-ask
---
spread (%)
| Market
---
impact (%)
| Total
---
cost (%)
Equities | 0.14 | 0.13 | 0.53 | 0.8
Currencies | 0 | 0.13 | 0 | 0.13
Commodities | 0.14 | 0.13 | 0 | 0.27
Fixed income (ETFs) | 0.14 | 0.13 | 0.53 | 0.8

The differing costs between the classes of assets stem from the sizes of these
markets and the potential influence a trader may have on them. For instance,
it is difficult to have any significant impact on foreign exchange, unless the
trader is an intervening central bank. Additionally, since in this research
raw currencies are traded, there are no brokerage commissions, and the only
cost remaining is the bid-ask spread. In the case of commodities, the trader
must account for brokerage fees, as well as the bid-ask spread on futures
contracts. However, only on rare occasions would the trade be large enough to
significantly move the entire market. For equity indices and fixed income
ETFs, the full cost is taken into consideration, since on top of commissions
and the bid-ask spread there is a potential market impact.

The KAMA+MS-DR model is finally compared to the benchmark models, in relation
to both annualised returns and ASR, via a ’winning score ratio’, $WS$,
calculated as

$WS=\frac{(Z_{winner}-Z_{runner-up})}{Z_{winner}}$ (11)

where $Z_{winner}$ refers to the model with the highest annualised returns or
adjusted Sharpe ratio, and $Z_{runner-up}$ refers to the second-best method.
By computing the ratio in the above equation, it is possible not only to
indicate the winner, but also by how much better it is relative to its closest
opponent. After the winning score ratio has been obtained for both weighted
annualised returns and the adjusted Sharpe ratio the two resulting ratios are
combined as an average to reveal the model with the most effective risk-reward
approach to detecting financial regimes. This combination is done as an
equally weighted average; depending on an investor’s preference either returns
or Sharpe ratio could be more highly weighted, but a ratio of 1:1 is the most
reasonable assumption in the absence of such a stated preference.

##  5 Results

The results of using the KAMA+MSR and two benchmark regime detection models
within the out-of-sample test periods (last 15 % of the data, for each asset
considered) are presented in Table 3. It is clear that the proposed model
performs strongly, in each of the two scoring metrics, for equities and fixed
income. In the case of equities, the notably higher adjusted Sharpe ratio for
KAMA+MSR points at a good balance between returns and volatility, which
suggests a solid blend between the use of trend in predicting regime switches
within KAMA and the use of variance for this purpose in Markov-switching
regression. It is also worth pointing out that the proposed model achieves
such results regardless of the equity region. Whether it is the USA, Europe,
or Emerging Markets, the KAMA+MSR model seems to be resistant to both global
and local shocks (though more indices, in particular from the Emerging Markets
region, could strengthen this argument). Results for commodities and foreign
exchange are more mixed, as can be seen from the table. However, after
combining the two winning scores the KAMA+MSR model can be seen to have the
overall best performance of those models tested, with an especially strong
result, again, for equities and fixed income.

Table 3: Out-of-sample trading performance results, in which _MSR 2S_ refers
to the two-state MSR model, _MSR 3S— >2S_ to the three-state MSR model
converted to two-state form, and _KAMA+MSR_ to the regime detection model
proposed in this work Average Winning Score Ratio: Highest Adjusted Sharpe
Ratio
---
| MSR 2S | MSR 3S—>2S | KAMA+MSR
Equities | 0.03 | 0.03 | 0.33
Commodities | 0.25 | 0.24 | 0.13
FX | 0.09 | 0.29 | 0.15
Fixed Income | 0.06 | 0.01 | 0.22
Average Winning Score Ratio: Highest Weighted Annual Returns
| MSR 2S | MSR 3S—>2S | KAMA+MSR
Equities | 0.08 | 0.03 | 0.10
Commodities | 0.02 | 0.12 | 0.27
FX | 0.22 | 0.02 | 0.19
Fixed Income | 0.25 | 0.00 | 0.18
Combined Average Winning Score Ratio
| MSR 2S | MSR 3S—>2S | KAMA+MSR
Equities | 0.11 | 0.06 | 0.43
Commodities | 0.27 | 0.35 | 0.40
FX | 0.31 | 0.31 | 0.35
Fixed Income | 0.31 | 0.01 | 0.40

##  6 Discussion

This paper presented an enhancement of the two-state Markov-switching
regression model (MSR) by the addition of Kaufman’s adaptive moving average
(KAMA), resulting in a new model, which we term the KAMA+MSR model. Unlike
multi-state MSR models, which seek only to subdivide variance (e.g., for a
three-state MSR, into low, medium, and high variance), and which have proven
both unstable and to result in large trading costs, KAMA+MSR seeks to instead
identify regimes as combinations of variance and market trend (bull or bear),
with an accurate timing of onset of each state, and without overly-frequent
regime switches. The proposed model initially discovers four states, of which
two combinations (low variance bullish and high variance bearish) were used
actively in the trading strategy, with the remaining two (low variance bearish
and high variance bullish) being periods in which it was deemed wisest to
refrain from action. Due to the use of both variance and trend measures to
detect regime switches in the KAMA+MSR model it proved able to outperform, on
average, the comparison models in each asset class considered. However, we
note that such an outcome could in principle be due to the strategy itself
(asset vs. cash), and hence intend to implement alternative strategies in
future work.

There are three further ways in which the work of this paper could be
extended. First, and most importantly, though the proposed model overall
outperformed the benchmark models, this outperformance was less pronounced for
foreign exchange and commodities, and it is possible that an alternative to
Markov switching could do better in combination with KAMA in these cases.
Second, as already mentioned, the current trading strategy discards two of the
four discovered regimes as being likely to be unprofitable; however, more
complex trading strategies, for example using derivatives-based hedging, could
potentially make use of these regimes. Finally, and most importantly for
future work, the KAMA+MSR model’s ability to smoothly and accurate distinguish
regimes will allow it to be used as a regime label generator for machine
learning tasks in which the aim is to predict such regimes ex ante.

## References

  * [1] S.M. Goldfeld and R.E. Quandt.  A Markov model for switching regressions.  Journal of Econometrics, 1(1):3–15, 2013.
  * [2] J.D. Hamilton.  A new approach to the economic analysis of nonstationary time series and the business cycle.  Econometrica: Journal of the Econometric Society, pages 357–384, 1989.
  * [3] H.M. Krolzig.  Markov-switching Vector Autoregressions: Modelling, Statistical Inference, and Application to Business Cycle Analysis.  Springer, New York, USA, 1997.
  * [4] J. Hauptmann, A. Hoppenkamps, A. Min, F. Ramsauer, and R. Zagst.  Forecasting market turbulence using regime-switching models.  Financial Markets and Portfolio Management, 28(2):139–164, 2014.
  * [5] P.J. Kaufman.  Smarter Trading: Improving Performance in Changing Markets.  McGraw-Hill, New York, USA, 1995.
  * [6] A. Timmerman.  Moments of Markov switching models.  Journal of Econometrics, 96(1):75–111, 2000.
  * [7] C.J. Kim and C. R. Nelson.  State Space Models with Regime Switching: Classical and Gibbs-sampling Approaches with Applications.  MIT Press, Cambridge, USA, 1998.
  * [8] O. Jeanne and P. Masson.  Currency crises, sunspots and Markov-switching regimes.  Journal of International Economics, 50(2):327–350, 2000.
  * [9] C.J. Kim.  Unobserved-component time series models with Markov-switching heteroscedasticity: Changes in regime and the link between inflation rates and inflation uncertainty.  Journal of Business & Economic Statistics, 11(3):341–349, 1993.
  * [10] O. Jeanne and P. Masson.  Hidden Markov models with t components. Increased persistence and other aspects.  Quantitative Finance, 11(3):459–475, 2011.
  * [11] S. Srivastava and R. Bhattacharya.  Evaluating the building blocks of a dynamically adaptive systematic trading strategy.  arXiv preprint arXiv:1812.02527, 2018.
  * [12] R.D. Wyckoff.  The Richard . Wyckoff Method of Trading in Stocks.  Wyckoff Associates, Inc., New York, USA, 1937.
  * [13] L. Ruiz-Franco, M. Jiménez-Gómez, and E. Lambis-Alandete.  Trading strategy on the future mini S&P 500.  International Journal of Applied Engineering Research, 13(13):11018–11024, 2018.
  * [14] D. Arthur and S. Vassilvitskii.  k-means++: The advantages of careful seeding.  Stanford, USA, 2006 Jun 7.
  * [15] M.D. Boldin.  A check on the robustness of Hamilton’s Markov switching model approach to the economic analysis of the business cycle.  Studies in Nonlinear Dynamics & Econometrics, 1(1), 1996\.
  * [16] C.L. Israelsen.  A refinement to the Sharpe ratio and information ratio.  Journal of Asset Management, 5(6):423–427, 2005.
  * [17] R. Edelen, R. Evans, and G. Kadlec.  Shedding light on “invisible” costs: Trading costs and mutual fund performance.  Financial Analysts Journal, 69(1):33–44, 2013.