# Visual Transformer Meets CutMix for Improved Accuracy,
Communication Efficiency, and Data Privacy in Split Learning

Sihun Baek1 Jihong Park2 Praneeth Vepakomma3
Ramesh Raskar3 Mehdi Bennis4 and Seong-Lyun Kim1 1Yonsei University, Seoul,
South Korea
2Deakin University, Geelong, Victoria, Australia
3MIT Media Lab, Cambridge, Massachusetts, USA
4University of Oulu, Oulu, Finland
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This article seeks for a distributed learning solution for the visual
transformer (ViT) architectures. Compared to convolutional neural network
(CNN) architectures, ViTs often have larger model sizes, and are
computationally expensive, making federated learning (FL) ill-suited. Split
learning (SL) can detour this problem by splitting a model and communicating
the hidden representations at the split-layer, also known as smashed data.
Notwithstanding, the smashed data of ViT are as large as and as similar as the
input data, negating the communication efficiency of SL while violating data
privacy. To resolve these issues, we propose a new form of CutSmashed data by
randomly punching and compressing the original smashed data. Leveraging this,
we develop a novel SL framework for ViT, coined CutMixSL, communicating
CutSmashed data. CutMixSL not only reduces communication costs and privacy
leakage, but also inherently involves the CutMix data augmentation, improving
accuracy and scalability. Simulations corroborate that CutMixSL outperforms
baselines such as parallelized SL and SplitFed that integrates FL with SL.

##  1 Introduction

Transformer architectures have revolutionized various application domains in
deep learning, ranging from natural language processing (NLP) Vaswani et al.
(2017) to speech recognition Karita et al. (2019). Fueled by this success,
recently there has been another paradigm shift in computer vision (CV) where
the _visual transformer (ViT)_ architecture has broken the performance record
set by the _de facto_ standard convolutional neural network (CNN)
architectures Dosovitskiy et al. (2020a). The core idea of ViT is to divide
each input image sample into multiple _patches_ as in the tokens in NLP, and
to process the patches in parallel using the _attention_ mechanism of
Transformer. This process is in contrast to that of CNN which processes only
neighboring pixels in sequence using the convolutional operations.

While the existing studies focus mostly on centralized ViT operations Han et
al. (2022), in this article we delve into the problem of distributed learning
with ViTs that are often computationally expensive, and require more training
samples than CNNs Khan et al. (2021); Han et al. (2022). In the recent
literature of distributed learning, federated learning (FL) is one promising
solution that enables training a global model across edge devices such as
phones, cameras, and the Internet of Things (IoT) devices Li et al. (2020).
The key idea is periodically averaging model parameters across edge devices or
clients through a parameter server, without directly exchanging private data.
However, model averaging requires every client to store and communicate the
entire model, so is ill-suited for ViT due to its large model size.

Alternatively, each client can store only a fraction of the entire model, and
offload the remaining segment onto the server under a model-split architecture
Gupta and Raskar (2018). _Split learning (SL)_ follows this model-split
parallelism, under which in the forward propagation (FP), each client uploads
the hidden layer activations at the split-layer, also known as _smashed data_
, followed by downloading gradients in the back propagation (BP) Vepakomma et
al. (2018). Unfortunately, as opposed to CNN’s smashed data distorted by
convolution operations, ViT’s smashed data wihtout convolution look similar to
their input data Yuan et al. (2021), which may leak private information on raw
data to the server. Furthermore, ViT often lacks pooling layers, so the
smashed data sizes are as large as the input data, negating the communication
efficiency of SL.

(a) Baseline 1: Parllel SL.

(b) Baseline 2: SFL.

(c) Proposed: CutMixSL.

Figure 1: Schematic illustrations of: (a) parallel split learning (SL), (b)
SplitFed (SFL), and (c) CutMixSL.

To resolve the aforementioned issues, inspired from the _CutMix_ data
augmentation technique Yun et al. (2019), we propose a new type of _CutSmashed
data_ , and thereby develop a novel split learning framework for ViTs, coined
_CutMixSL_. In CutMixSL, each client constructs the CutSmashed data by
randomly masking the patches of the original smashed data. For instance, Fig.
1 illustrates two clients locally constructing the CutSmashed data by applying
the mutually exclusive masks $010110$ and $101001$ to their original smashed
data, respectively. Here, to guarantee the mutual exclusiveness and the
subsequent operations, a common pseudo random sequence generator is shared by
all clients and the server, without sharing raw data. Then, the mixer at the
server adds these uploaded CutSmashed data, resulting in _CutMix data_ (i.e.,
with the mask $111111$) that FP in the server, followed by BP from the server
to clients.

The key benefits of CutMixSL are summarized as follows.

  * •

First, the clients in CutMixSL can only upload non-zero masked CutSmashed
data, _reducing privacy leakage_.

  * •

Second, CutMix data plays its original role as data augmentation complementing
ViT’s lack of useful inductive bias as opposed to CNNs, thereby _improving
accuracy_.

  * •

Lastly, CutMixSL is free from the standard paralell SL’s imbalance problem
between the server-side and client-side model updates Oh et al. (2022); Pal et
al. (2021), _achieving scalability_ in temrs of the accuracy increasing with
the number of clients.

  * •

Simulations under the CIFAR-10 classification task show that CutMixSL reduces
the privacy leakage (measured by reconstruction mean-squared errors) by around
8$\times$, decreases uplink communication payload sizes by $20-50\%$, and
improves accuracy by up to $18.5$% while achieving scalability up to (at
least) $10$ clients.

##  2 Related works

#### FL, SL, and Hybrids

FL and SL are two popular distributed machine learning techniques preserving
data privacy by avoiding raw data exchanges Konečnỳ et al. (2015); Gupta and
Raskar (2018). These methods have their own advantages and disadvantages. FL
is scalable in the sense that the accuracy increases with the number of
clients Konečnỳ et al. (2015). However, FL requires exchanging and averaging
model parameters, so cannot handle large models under limited memory,
computing, and communication resources Konečnỳ et al. (2016). On the other
hand, SL is applicable to large models by means of its spliting the entire
model into segments, e.g., two segments stored at the server and at each
client, respectively Gupta and Raskar (2018); Vepakomma et al. (2018). To
enjoy the advantages of both methods, SplitFed (SFL) integrates FL into SL by
averaging only clients’ model segments after BP Thapa et al. (2020); Gao et
al. (2021), at the cost of increasing communication overhead. Alternatively, a
similar effect can be achieved by averaging or mixing the hidden activations
during FP Xiao et al. (2021) and Oh et al. (2022). Our proposed method focuses
primarily on the activation mixing, and the segmentation averaging is
supplementary.

#### Transformer and ViT

Transformer is the first architecture driven entirely by the attention
mechanism Vaswani et al. (2017), making remarkable success in NLP as evidenced
by BERT Devlin et al. (2018) and GPT-3 Brown et al. (2020). Motivated by this,
Transformer has expended its applications to various domains, among which ViT
is the recent success made in CV Dosovitskiy et al. (2020b). In CV, there are
other architectures that partly utilize the concepts of Transformer Carion et
al. (2020); Zheng et al. (2021). As opposed to them, ViT is a Trasformer-
native architecture, which is viable by splitting and converting an input
image into the input sequence of image patches as analogous to the input
sequence of language tokens in NLP. Departing from standalone operations,
distributed Transformer has been studied in the context of FL and SL for
various applications. Transformer with FL has been investigated for a Text-to-
Speech (TTS) task in Hong et al. (2021). Transformer with SL has been
developed for a COVID-19 diagnosis in Park et al. (2021). ViT with FL has also
been studied under heterogeneous data in Qu et al. (2021). Inspired from these
preceding studies, for the first time, we investigate ViT with SL as well as
its parallel, communication-efficient, and scalable architecture.

##  3 Token-based Split Learning for ViT

In this paper, a novel split learning based on patches where transformers
operate is proposed to solve the issue of heavy communication payload and data
privacy leakage problem. We describe components of the proposed learning
algorithm in an order of training sequence. The overall procedure of the
proposed is visualized as Figure 1(c).

###  3.1 CutSmashed data for Communication Efficiency and Privacy Enhancement

To increase a communication efficiency, and to resolve the data privacy
leakage problem, we pay attention to the way how transformers address data.
Transformers divide data into multiple separate tokens and process the entire
range of sequence in parallel. In this sense, we instinctively expect that
cutting off a certain amount of patches of smashed data at random could reduce
communication cost and strengthen the data privacy since adversaries are not
able to regenerate the untransmitted region at the expense of performance.
This partially uploaded smashed data is labelled as CutSmashed Data.
CutSmashed data conceal its information by random removal of patches. While SL
requires to upload the entire smashed data of each client to the server, our
proposed does not need to upload the whole smashed data from each client.

Let there exist $n$ clients with a set of $\mathbb{C}=\\{1,2,\cdots,n\\}$ and
a single server. $(\mathbf{x}_{i},\mathbf{y}_{i})$ denotes a batch of raw
data-label tuples from $i$-th local dataset, $\mathbb{D}_{i}$. A neural
network model is denoted with weights $\mathbf{w}_{i}$, which is splitted into
two segments such that
$\mathbf{w}_{i}=[\mathbf{w}_{c,i},\mathbf{w}_{s}]^{\textrm{T}}$ for
$i\in\mathbb{C}$. We define $f$ as a representation for mapping from the input
data to the output. The smashed data is expressed as
$\mathbf{s}_{i}:=f_{\mathbf{w}_{c,i}}(\mathbf{x}_{i})$. Here, transformers
divide the raw data into $M$ number of patches, and each patch is transformed
into an embedding vector during FP. A smashed data is denoted as
$\mathbf{s}=[e_{1},...,e_{M}]\in\mathcal{R}^{M\times d_{m}}$, where $e_{i}$ is
the $i$-th embedding vector and $d_{m}$ is the dimension of the patch
embedding.

Before uploading CutSmashed data, a piece of information which patch
embeddings would be transmitted are shared between a server and clients. This
prior information is called as a pseudo random sequence, and is denoted as
$\mathbb{B}$. Then, CutSmashed data is expressed as
$\mathbf{s}^{\prime}_{i}=\mathbb{B}\odot\mathbf{s}_{i}$, where
$\mathbb{B}=[m_{j}]_{M\times 1}$, and $m_{j}\in\\{0,1\\}$ where 0 indicates
not to transmit, and 1 to transmit. $\odot$ denotes element-wise
multiplication operation. Its communication cost can be ignored, since
$\mathbb{B}$ is treated as a binary number which could be converted to one
integer-type data having a negligible communication payload. Figure 4(b) shows
examples of CutSmashed data whose black regions indicate the cut off regions.
The pseudo random mask over the smashed could be created regardless of the
depth of the cut layer since it is determined based on the number of patches
of the smashed data.

Figure 2 shows the performance of SL and SFL training with CutSmashed data
instead of smashed data in regard to the average size of CutSmashed data to
identify the effect of the size of masking. The cases when the positions to be
cut off are fixed for all rounds, and randomly selected at each iteration are
compared. In the both cases, the performance decreases more with a larger size
of the cutoff. The model cannot generalized well since the masked regions
discard meaningful features of original data distribution. Interestingly, when
the mask positions changed randomly, the performance is improved up to a
certain extent of the cut off size. This implies that randomly generated
CutSmashed data with a moderate size rather helps to prevent the model from
overfitting to the local dataset acting as regularization like Cutout DeVries
and Taylor (2017). While the original Cutout putting one square region of mask
to input, ours generate multiple masks to random locations.

In terms of the leakage of data privacy, CutSmashed data is preferable than
smashed data, since partial elements are concealed to the server. We elaborate
on the data privacy leakage in the setting of a reconstruction attack where
the attacker is willing to reconstruct an original data from uploaded smashed
data. Hard to be restored by reconstruction attack implies raw data has strong
privacy. In Section 4 evaluates the privacy leakage of CutSmashed data
compared with the other techniques to be described in the next section.

Figure 2: Performance of SL with CutSmashed data w.r.t. the average size of
CutSmashed data. Figure 3: Operation of CNN and ViT.

###  3.2 Smashed Patch CutMix: Inter-client Mixup

The vision transformer is difficult to be generalized well in situations where
there is insufficient data held by clients due to its low inductive bias
Baxter (2000). Inductive bias is a set of assumptions added as prior
information to solve unknown machine learning problems. The weak inductive
bias causes reliance on data augmentation and model regularization to gather
sufficient training data Steiner et al. (2021). Due to the data privacy, data
augmentation between data held by different clients is limited in a
distributed learning, making a severer problem for a transformer in a
distributed learning. To overcome a limited inter-client data augmentation,
and reduce the risk of the performance drop by uploading CutSmashed data, the
blank parts of CutSmashed could be filled with patches of different clients’
CutSmashed data.

A self-attention mechanism evaluates which patches they should pay more
attention to. Combining patches from different smashed data corresponds to a
new attention between data from different clients. We demonstrate this
assumption by putting CutSmashed data together from different clients during
FP like putting the puzzle together. Since the process resembles CutMix Yun et
al. (2019) in that some region of data are cut and pasted into different data
in the form of multiple patches with the same sizes, this operation is
expressed as Smashed Patch CutMix, and its mixed smashed data as CutMix data.
The central idea of Smashed Patch CutMix is to create a new smashed data to
supplement the performance loss due to partial uploading with filling blanks
of CutSmashed data with each other. As shown in Figure 1(c), a virtual entity
conducting the mix operation, a mixer, is conceptualized to compare with SL
and SFL. A mixer could be a 3rd party, or could be divided and belong to the
server and the clients. Since overlapped patches lose its clear
representation, non-overlapping mix between different CutSmashed data should
be guaranteed, and paired pseudo random sequences are generated for Smashed
Patch CutMix.

We call $k$-way CutMix to represent that each $k$ number of smashed data are
mixed to generate one CutMix data. Mixing groups,
$\mathbb{G}=\\{\mathbf{g}_{1},..,\mathbf{g}_{l}\\}$, are generated randomly
from $\mathbb{C}$ at each iteration. Then, the number of patches allocated to
each CutSmashed data (i.e. mixing ratio), $\\{a_{1},..,a_{k}\\}$, is
determined by sampling from a Dirichlet distribution with the dispersion
factor $\alpha>0$ Bishop et al. (2007). Here, the number of trials of the
distribution is set as the number $M$ of path tokens so that
$\sum_{i=1}^{k}a_{i}=M$, and a probability vector is drawn from a Dirichlet
distribution. Afterwards, patches at which positions of each smashed data are
determined by the given mixing ratio. Here, the selected positions for patches
of each client are not superposed with each other. Pseudo random sequence for
$i$-th smashed data in a mixing group is denoted as $\mathbb{B}_{a_{i}}$, and
$\mathbb{B}_{a_{i}}$ is determined uniformly and randomly.
$\sum_{j=1}^{M}m_{a_{i},j}=a_{i}$ where
$\mathbb{B}_{a_{i}}=[m_{a_{i},j}]_{M\times 1}$ for $i\in[k]$.
$\sum_{i=1}^{k}\mathbb{B}_{a_{i}}=\mathbf{1}_{M\times 1}$, where $\mathbf{1}$
is a vector of all ones.

Each client uploads CutSmashed data according to the received pseudo random
sequence generated and shared by a mixer. The mixer conducts Smashed Patch
CutMix, and CutMix data and its corresponding label by the 2-way Smashed Patch
CutMix is expressed as:

$\displaystyle\tilde{\mathbf{s}}_{\\{i,j\\}}=\mathbf{s}^{\prime}_{i}+\mathbf{s}^{\prime}_{j}=\mathbb{B}_{a_{i}}\odot{\mathbf{s}_{i}}+\mathbb{B}_{a_{j}}\odot{\mathbf{s}_{j}}$
(1)
$\displaystyle\tilde{\mathbf{y}}_{\\{i,j\\}}=\frac{a_{i}}{M}\cdot{\mathbf{y}_{i}}+\frac{a_{j}}{M}\cdot{\mathbf{y}_{j}},$
(2)

where $\odot$ denotes the element-wise product.

Algorithm 1 Operation of Mixer

function Sequence Generation

Sample {$a_{1},..,a_{k}$} $\sim$ Dir($\alpha$) for $k$-way CutMix

Generate $\\{\mathbb{B}_{a_{1}},..,\mathbb{B}_{a_{k}}\\}$ uniformly at random

return paired pseudo random sequences

end function

for $e\leftarrow 1$ to $E$ do

Generate a set of mixing groups, $\mathbb{G}$ from $\mathbb{C}$

Sequence Generation

Send $\\{\mathbb{B}_{a_{1}},..,\mathbb{B}_{a_{k}}\\}$ to clients based on
$\mathbb{G}$

Receive
$\\{(\mathbf{s}^{\prime}_{1},\mathbf{y}_{1}),..,(\mathbf{s}^{\prime}_{n},\mathbf{y}_{n})\\}$
from $\mathbb{C}$

for $\mathbf{g}\in\mathbb{G}$ do

$(\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}})\leftarrow(\sum_{j\in\mathbf{g}}\mathbf{s}^{\prime}_{j},\sum_{j\in\mathbf{g}}\frac{a_{j}}{M}\mathbf{y}_{j})$

Upload $\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}}$ to
the server

end for

end for

The proposed is simple to implement with negligible extra communication cost
for sharing pseudo random sequences. Intermixing CutSmashed data transmitted
from each client’s side enhances the server side’s generalization capability
acting as a data augmentation at feature space. Additionally, the impact of
shuffling the order of the sequence on the performance is analyzed in detail
in Appendix C.

(a) Raw images.

(b) Smashed data.

(c) Reconstruction from smashed data.

Figure 4: Examples of data with different operations at different levels.

###  3.3 Weight Update based on CutMix data

At the server, the upper model segment $\mathbf{w}_{s}$ propagates
$\tilde{\mathbf{s}}_{\\{i,j\\}}$ uploaded by $i$-th and $j$-th client, and
generates softmax output $f_{\mathbf{w}_{s}}(\mathbf{s}_{\\{i,j\\}})$. Then,
the loss $\tilde{L}_{\\{i,j\\}}$ generated by the server model with CutMix
data can be expressed as:

$\displaystyle\tilde{L}_{\\{i,j\\}}=\frac{1}{b}\sum\text{CE}\\!\left(f_{\mathbf{w}_{s}}(\tilde{\mathbf{s}}_{\\{i,j\\}}),\tilde{\mathbf{y}}_{\\{i,j\\}}\right),$
(3)

where $b$ is a batch size. The weights of the server and the clients are
updated by BP as follows:

$\displaystyle\begin{bmatrix}\mathbf{w}_{s}\\\
\mathbf{w}_{c,i\in\mathbb{C}}\end{bmatrix}\leftarrow\begin{bmatrix}\mathbf{w}_{s}\\\
\mathbf{w}_{c,i\in\mathbb{C}}\end{bmatrix}-\eta\begin{bmatrix}\sum_{\\{i,j\\}\in{\mathbb{G}}}{(\nabla_{\mathbf{w}_{s}}\tilde{L}_{\\{i,j\\}})}\\\
\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{\\{i,j\\}}\end{bmatrix}$ (4)

where $\mathbb{G}$ is a set of groups of clients whose smashed data are mixed,
$\eta$ is a learning rate, and $\nabla_{\mathbf{w}_{s}}\tilde{L}_{\\{i,j\\}}$
and $\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{\\{i,j\\}}$ are the derivatives of
the error with respect to $\mathbf{w}_{s}$ and $\mathbf{w}_{c,i}$,
respectively. The server sends the gradients of $\tilde{L}_{i,j}$ with respect
to the uploaded smashed data $\mathbf{s}^{\prime}_{i}$ and
$\mathbf{s}^{\prime}_{j}$, $\nabla_{\mathbf{s}_{i}}\tilde{L}_{i,j}$ and
$\nabla_{\mathbf{s}_{j}}\tilde{L}_{i,j}$, to the corresponding clients, and
the clients calculates $\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{\\{i,j\\}}$, which
is given as:

$\displaystyle\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{i,j}=\frac{\partial\tilde{L}_{\\{i,j\\}}}{\partial\mathbf{s}^{\prime}_{i}}\frac{\partial\mathbf{s}^{\prime}_{i}}{\partial\mathbf{w}_{c,i}}.$
(5)

The method above aims to resolve a server-clients update imbalance problem,
hindering the scalability of parallel split learning Oh et al. (2022). The
more clients, it incurs the more imbalanced updates between the server and the
clients. There are $n$ times of update on the upper model segment, whereas
each client updates its model once. Meanwhile, with our proposed method, the
server executes its server model’s update only once on a unit mixed smashed
data, as shown in Figure 1(c). Therefore, when all clients update their
parameters with CutMix data, $n$ times updates are in the server model reduce
to $\frac{n}{k}$ times updates for $k$-way CutMix. The CutMixSL ($k$ times) in
Table 2 is the case when gradients from one CutMix data flow to only one
client and the server updates $n$ times in total like SL. Compared with
CutMixSL in Table 2 where the server updates $\frac{n}{k}$ times, it shows
that the reduced update of the server by CutMix data has a positive impact on
a performance gain.

##  4 Evaluation

We consider the CIFAR-10 (60k samples of 32x32 pixel color images with 10
classes) classification task with up to 10 clients, where each client stores
randomly selected 5k samples and is associated with a common server. The
standard ViT architecture is visualized in Figure 5a. In our setting, we
follow the ViT-Tiny architecture Touvron et al. (2020) including 6 Transformer
blocks, each of which consists of multiple layers for operating the attention
mechanism. The ViT-Tiny is split into two segments, and the cut-layer is the
embedding layer, i.e., the embedding layer is stored at each client while the
rest is offloaded to the server. Given the CIFAR-10 dataset, each sample is
divided into 4x4 patches that are flattened into a sequence of 16 tokens. The
embedding layer concatenates these patch tokens and the class token (i.e., the
sample’s class), followed by adding positional embedding to identify the patch
positions within the sample. To train this paralle split ViT, we use the adamW
optimizer, and the learning is 0.001 with the cosine annealing. The batch size
is 128, and the total number of epochs is 600 with 5 warmup epochs.

#### Impact of Mixing Methods

To analyze the effectiveness of Smashed Patch CutMix on the transformer,
different types of mix operations on both transformer-based models and a CNN-
based model are evaluated. ViT-Tiny for a pure transformer, PiT-Tiny Heo et
al. (2021) for a pooling-based transformer, and VGG-16 Simonyan and Zisserman
(2014) for CNN are used whose simplified model architectures are described in
Figure 5. Many ViT models with hierarchical representations are proposed such
as Swin Transfromer Liu et al. (2021), and T2T-ViT Yuan et al. (2021)
including PiT. VGG is a representative model of CNN composed of convolutional
layers, pooling layers and a classifier, and the cut-layer is after two
convolutional and one pooling layers.

Figure 5: Simplified model architectures of ViT, PiT, and VGG.

Method | Models
---|---
ViT-Tiny | PiT-Tiny | VGG-16
Standalone | 49.14 | 47.77 | 54.97
Parallel SL | 57.05 | 52.28 | 62.62
Cutout | 65.03 | 60.87 | 67.06
Mixup | 71.02 | 65.92 | 74.43
Patch CutMix | 75.55 | 73.19 | 72.23
Shuffled CutMix | 72.78 | 57.59 | 33.50

Table 1: Performance w.r.t. mixing methods.

Table 1 shows the performances of Cutout (CutSmashed data), Mixup (Smashed
data + Mixup), Smashed Patch CutMix (CutSmashed data + CutMix), and Shuffled
CutMix (CutSmashed data + CutMix + Shuffling). Smashed Patch CutMix has 18.5%
and 20.9% performance gain on ViT and PiT compared to parallel SL. It is
higher than Mixup for ViT and PiT (4% and 8%, respectively), and 2% lower than
Mixup for VGG-16. Masking such as CutMix is better at preserving local
features than interpolation such as mixup without distorting data distribution
Harris et al. (2020). Transformers process data as divided separate patches,
and even though randomly chosen patches to be cut and pasted destroy a global
structure of image, transformers do not lose its inference ability because of
parallel processing for an entire range of sequence by attention mechanism. In
this sense, masking smashed data is expected to be more suitable than
interpolation to transformers. On the other hand, CNN uses sliding
convolutional filters and one of assumptions CNN poses is a locality of pixel
dependencies Krizhevsky et al. (2012). Neighboring pixels which tend to be
correlated, get uncorrelated by masking due to junctions of different clients’
patches. Notwithstanding, the result shows that the impact of masking is not
big enough to degrade the performance of CNN. We guess that the unchanged
position of patches from the original location keeping its architecture of the
data results in a positive impact to the performance gain.

(a) Mixing ratio.

Type | Mixing Ratio
---|---
Uniform | Dir($\alpha=6$)
2-way | 70.93 | 75.55
3-way | 68.59 | 70.86
4-way | 66.96 | 66.67
5-way | 67.01 | 67.04

(a) Mixing group size.

Figure 6: Top-1 Accuracy w.r.t. (a) mixing ratio $\alpha$ and (2) group size.

#### Impact of Mixing Ratio & Group Size

In Figure 6(a), the performance of CutMixSL with respect to $\alpha$ of
Dirichlet-multinomial distribution is evaluated to identify a sensitivity to
the mixing ratio. The higher $\alpha$, the higher the probability of even
mixing ratio. The result shows that the higher probability of evenly mix gains
more accuracy than the lower one. The top-1 accuracy when
$\alpha\rightarrow\infty$, which is the number of sampled patches is fixed to
even, is lower than the case for the lower $\alpha$. It implies that having an
uneven mixture with a moderate probability is preferable for a higher
performance. Table 2(a) evaluated a performance of the proposed in regard to
the size of mixing group with a uniform distributions and the dirichlet
distribution with $\alpha=6$. It is expected that the performance would be
improved with the larger mixing group size from the standpoint of update
imbalance problem, since the server updates less. Nevertheless, 2-way CutMix
has a best performance gain implying that overly mixing with different classes
causes the server cannot learn distinct features of each class.

#### Scalability

In Table 2, while all evaluated methods are scalable up to at least 10
clients, each has a different performance gain at scale. The brackets in Table
2 represents the distribution that a mixing ratio is sampled from. Parallel SL
has a weak scalable performance gain (4%) due to the update imbalance problem,
whereas CutMixSL has a higher scalable performance gain than parallel SL
(12.9%) and even SFL (12.7%), and an top-1 accuracy of CutMixSL is also higher
than SFL. CutMixSFL, which is CutMixSL with FedAvg on the lower models,
achieves 13.4% performance gain at scale and about 80% top-1 accuracy with ten
clients.

Method |  $\\#$ of clients
---|---
2 | 4 | 6 | 8 | 10
Parallel SL |  52.91 |  55.55 |  56.81 |  56.55 |  57.05
SplitFed |  53.96 |  58.39 |  63.07 |  65.59 |  66.70
CutMixSL ($k$ times, uniform) |  58.06 |  65.11 |  65.76 |  68.97 |  67.66
CutMixSL (uniform) |  61.66 |  66.80 |  67.69 |  69.86 |  70.93
CutMixSL ($\alpha$=6) |  62.62 |  69.47 |  71.80 |  73.67 |  75.55
CutMixSFL (uniform) |  65.82 |  72.31 |  74.31 |  78.88 |  80.14
CutMixSFL ($\alpha$=6) |  67.58 |  73.52 |  76.85 |  79.48 |  80.97

Table 2: Scalability. Top-1 accuracy w.r.t. the # of clients.

#### Communication Efficiency.

Compared to Parallel SL requiring entire smashed data, one of the advantages
of the proposed technique is a large reduction of upload payload size with an
enhanced accuracy. Figure 7 shows an upload payload size per communication
round and the performance of CutMixSL and its derivatives. The black line
shows an accuracy of SL with CutSmashed data regarding $\lambda$, the ratio of
the size of CutSmashed data to the one of the original smashed data described
in Figure 2. CutMixSL shows the highest performance and upload payload
reduction up to 20 - 50%. The more mixing group size increases, the higher
communication cost reduction can be achieved with a smaller performance gain.
In our settings, one unmatched client of $3$-way sends smashed data, and two
unmatched clients in 4-way CutMix conduct 2-way CutMix.

Figure 7: Upload payload size per communication round of CutMixSL and its
derivatives.

#### Privacy Leakage

Data privacy leakage increases with the mutual information between the
CutSmashed data and its raw data. Due to the unknown data distributions, the
mutual information is often approximated by the error when reconstructing the
input data Vepakomma et al. (2020). In this respect, following Wang et al.
(2021), we train an autoencoder whose input is the CutSmashed data and the
output is the original raw data. The trained autoencoder’s loss, given as the
mean squared error (MSE), can thereby treated as the amount of privacy leakage
to the server.

Table 3 shows the privacy leakage of 2-way CutMixSL and its variants on a test
set when the autoencoder trains 10% of a train set, and when it trains 100% of
a train set. As intuitively expected, training with CutSmashed data reduces
privacy leakage by 15x compared to the one with smashed data. For the mix
operations, Smashed Patch CutMix has higher reconstruction loss than Mixup,
and CutMixSL reduces data privacy by around 8 times compared to the baseline.
This gap gets highly amplified when the order of its sequences is shuffled
(Shuffled CutMix), implying that masking data and shuffling are two principal
factors for privacy enhancement. Figure 4(c) shows some examples of
reconstructed images by the reconstruction attack for a qualitative
comparison.

Type | Train Dataset(10%) | Train Dataset(100%)
---|---|---
Smashed data | 0.0091 | 0.0056
CutSmashed data | 0.0920 | 0.0829
Mixup | 0.0402 | 0.0351
Patch CutMix | 0.0458 | 0.0434
Shuffled CutMix | 0.1233 | 0.1250

Table 3: Privacy leakage measured by the reconstruction loss (MSE).

##  5 Conclusion

In this work, CutSmashed data is introduced to resolve data privacy leakage
and to improve communication efficiency of split learning motivated by a
process of a vision transformer handling data as a sequence of patches.
Furthermore, CutMixSL is proposed with Smashed patch CutMix, a smashed data
augmentation, to deploy a transformer-based model for split learning. We
analyzed the design elements’ impact on the proposed operation, and confirmed
that the it has advantages on a improved performance, communication
efficiency, and data privacy compared to Parallel SL and SFL.

## Acknowledgments

This work was partly supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (No.2021-0-00347, 6G Post-MAC (POsitioning- & Spectrum-aware
intelligenT MAC for Computing & Communication Convergence)), and Institute of
Information & communications Technology Planning & Evaluation (IITP) grant
funded by the Korea government(MSIT) (1711134613, 2021-0-00270, Development of
5G MEC framework to improve food factory productivity, automate and optimize
flexible packaging).

## References

  * Baxter [2000] Jonathan Baxter.  A model of inductive bias learning.  Journal of artificial intelligence research, 12:149–198, 2000.
  * Bishop et al. [2007] Yvonne M Bishop, Stephen E Fienberg, and Paul W Holland.  Discrete multivariate analysis: theory and practice.  Springer Science & Business Media, 2007.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * Dosovitskiy et al. [2020a] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * Dosovitskiy et al. [2020b] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  CoRR, abs/2010.11929, 2020.
  * Gao et al. [2021] Yansong Gao, Minki Kim, Chandra Thapa, Sharif Abuadbba, Zhi Zhang, Seyit Camtepe, Hyoungshick Kim, and Surya Nepal.  Evaluation and optimization of distributed machine learning techniques for internet of things.  IEEE Transactions on Computers, 2021.
  * Gupta and Raskar [2018] Otkrist Gupta and Ramesh Raskar.  Distributed learning of deep neural network over multiple agents.  Journal of Network and Computer Applications, 116:1–8, 2018.
  * Han et al. [2022] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on vision transformer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * Harris et al. [2020] Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, Adam Prügel-Bennett, and Jonathon S. Hare.  Understanding and enhancing mixed sample data augmentation.  CoRR, abs/2002.12047, 2020.
  * Heo et al. [2021] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  CoRR, abs/2103.16302, 2021.
  * Hong et al. [2021] Zhenhou Hong, Jianzong Wang, Xiaoyang Qu, Jie Liu, Chendong Zhao, and Jing Xiao.  Federated learning with dynamic transformer for text to speech.  arXiv preprint arXiv:2107.08795, 2021.
  * Karita et al. [2019] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al.  A comparative study on transformer vs rnn in speech applications.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 449–456. IEEE, 2019.
  * Khan et al. [2021] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  ACM Computing Surveys (CSUR), 2021.
  * Konečnỳ et al. [2015] Jakub Konečnỳ, Brendan McMahan, and Daniel Ramage.  Federated optimization: Distributed optimization beyond the datacenter.  arXiv preprint arXiv:1511.03575, 2015.
  * Konečnỳ et al. [2016] Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon.  Federated learning: Strategies for improving communication efficiency.  arXiv preprint arXiv:1610.05492, 2016.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * Li et al. [2020] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith.  Federated learning: Challenges, methods, and future directions.  IEEE Signal Processing Magazine, 37(3):50–60, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * Naseer et al. [2021] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Intriguing properties of vision transformers.  Advances in Neural Information Processing Systems, 34, 2021.
  * Oh et al. [2022] Seungeun Oh, Jihong Park, Praneeth Vepakomma, Sihun Baek, Ramesh Raskar, Mehdi Bennis, and Seong-Lyun Kim.  Locfedmix-sl: Localize, federate, and mix for improved scalability, convergence, and latency in split learning.  In Proceedings of the ACM Web Conference 2022, pages 3347–3357, 2022.
  * Pal et al. [2021] Shraman Pal, Mansi Uniyal, Jihong Park, Praneeth Vepakomma, Ramesh Raskar, Mehdi Bennis, Moongu Jeon, and Jinho Choi.  Server-side local gradient averaging and learning rate acceleration for scalable split learning.  arXiv preprint arXiv:2112.05929, 2021.
  * Park et al. [2021] Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye.  Federated split task-agnostic vision transformer for covid-19 cxr diagnosis.  Advances in Neural Information Processing Systems, 34, 2021.
  * Qu et al. [2021] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Li Fei-Fei, Ehsan Adeli, and Daniel Rubin.  Rethinking architecture design for tackling data heterogeneity in federated learning.  arXiv preprint arXiv:2106.06047, 2021.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * Steiner et al. [2021] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  CoRR, abs/2106.10270, 2021.
  * Thapa et al. [2020] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit Camtepe.  Splitfed: When federated learning meets split learning.  CoRR, abs/2004.12088, 2020.
  * Touvron et al. [2020] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  CoRR, abs/2012.12877, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Vepakomma et al. [2018] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar.  Split learning for health: Distributed deep learning without sharing raw patient data.  arXiv preprint arXiv:1812.00564, 2018.
  * Vepakomma et al. [2020] Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar.  Nopeek: Information leakage reduction to share activations in distributed deep learning.  In 2020 International Conference on Data Mining Workshops (ICDMW), pages 933–942. IEEE, 2020.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In International Conference on Machine Learning, pages 6438–6447. PMLR, 2019.
  * Wang et al. [2021] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang.  Revisiting locally supervised learning: An alternative to end-to-end training.  arXiv preprint arXiv:2101.10832, 2021.
  * Xiao et al. [2021] Danyang Xiao, Chengang Yang, and Weigang Wu.  Mixing activations and labels in distributed training for split learning.  IEEE Transactions on Parallel and Distributed Systems, 2021.
  * Yuan et al. [2021] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558–567, 2021.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881–6890, 2021.

##  Appendix A Pseudo Algorithm of CutMixSL

Algorithm 2 CutMixSL

function Sequence Generation

Sample {$a_{1},..,a_{k}$} $\sim$ Dir($\alpha$) for $k$-way CutMix

Generate $\\{\mathbb{B}_{a_{1}},..,\mathbb{B}_{a_{k}}\\}$ uniformly at random

return paired pseudo random sequences

end function

for $e\leftarrow 1$ to $E$ do

/*Runs on mixer*/

Generate a set of mixing groups, $\mathbb{G}$ from $\mathbb{C}$

Sequence Generation

Send $\\{\mathbb{B}_{a_{1}},..,\mathbb{B}_{a_{k}}\\}$ to clients based on
$\mathbb{G}$

Receive
$\\{(\mathbf{s}^{\prime}_{1},\mathbf{y}_{1}),..,(\mathbf{s}^{\prime}_{n},\mathbf{y}_{n})\\}$
from $\mathbb{C}$

for $\mathbf{g}\in\mathbb{G}$ do

$(\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}})\leftarrow(\sum_{j\in\mathbf{g}}\mathbf{s}^{\prime}_{j},\sum_{j\in\mathbf{g}}\frac{a_{j}}{M}\mathbf{y}_{j})$

Upload $\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}}$ to
the server

end for

/*Runs on clients*/

for each client $i\in\mathbb{C}$ in parallel do

Receive $\mathbb{B}_{i}$ from mixer

$\mathbf{s}_{i}\leftarrow
f_{\mathbf{w}_{c,i}}(\mathbf{x}_{i});\mathbf{s}^{\prime}_{i}\leftarrow\mathbb{B}_{i}\odot\mathbf{s}_{i}$

Upload $(\mathbf{s}^{\prime}_{i},\mathbf{y}_{i})$ to mixer

Receive $\nabla_{\mathbf{s}_{i}}{\tilde{L}_{\mathbf{g}}}$; Calculate gradient
$\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{\mathbf{g}}$

Weight Update
$\mathbf{w}_{c,i}\leftarrow\mathbf{w}_{c,i}-\eta\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{\mathbf{g}}$

end for

/*Runs on server*/

for $\mathbf{g}\in\mathbb{G}$ do

Receive $(\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}})$
from mixer

$\tilde{L}_{\mathbf{g}}\leftarrow\frac{1}{b}\sum_{\\{\tilde{\mathbf{s}}_{\mathbf{g}},\tilde{\mathbf{y}}_{\mathbf{g}}\\}}\text{CE}\\!\left(f_{\mathbf{w}_{s}}(\tilde{\mathbf{s}}_{\mathbf{g}}),\tilde{\mathbf{y}}_{\mathbf{g}}\right)$

Calculate gradient $\nabla_{\mathbf{w}_{s}}\tilde{L}_{\mathbf{g}}$

Weight Update
$\mathbf{w}_{s}\leftarrow\mathbf{w}_{s}-\eta\nabla_{\mathbf{w}_{s}}\tilde{L}_{\mathbf{g}}$

Download $\nabla_{\mathbf{s}_{i}}{\tilde{L}_{\mathbf{g}}}$ to client
$i\in\mathbf{g}$

end for

end for

##  Appendix B Mixing Methods

Data augmentation can generate new samples through mixing different samples,
and there are generally two types of inter-sample data augmentation: masking
and interpolation. The typical techniques of masking are CutMix Yun et al.
[2019] and Cutout DeVries and Taylor [2017], and the one of interpolation is
Mixup Zhang et al. [2017].

Mixup is an entire interpolation of two given raw data, $x_{i}$ and $x_{j}$
with a certain ratio $\lambda$, and is expressed as follows:

$\begin{gathered}\mathbf{x}_{mixup}=\lambda{\mathbf{x}_{i}}+(1-\lambda){\mathbf{x}_{j}}.\end{gathered}$
(6)

In Verma et al. [2019], Manifold Mixup has been proposed, which is Mixup at
the feature space of the deep neural network (DNN), and is expressed as
follows:

$\begin{gathered}\mathbf{s}_{mixup}=\lambda{\mathbf{s}_{i}}+(1-\lambda)\mathbf{s}_{j}.\end{gathered}$
(7)

The corresponding label for the generated data by Mixup and Manifold Mixup is
expressed as follows:

$\begin{gathered}\mathbf{y}_{mixup}=\lambda{\mathbf{y}_{i}}+(1-\lambda){\mathbf{y}_{j}}.\end{gathered}$
(8)

As a masking technique, Cutout DeVries and Taylor [2017] masks out square
regions of input, and is expressed as follows:

$\begin{gathered}\mathbf{x}_{cutout}=\mathbf{M}\odot{\mathbf{x}},\end{gathered}$
(9)

where $\mathbf{M}\in\\{0,1\\}^{W\times H}$ is a binary mask indicating which
pixels are to be dropped out. $\mathbf{M}$ fills $0$ inside the bounding box
coordinates, $\mathbf{B}=(r_{x},r_{y},r_{w},r_{h})$, indicating the cropping
region of the image.

As one step further, CutMix drops a unit square region of random size, and
fills in the blanks with a different raw image, and is expressed as following:

$\begin{gathered}\mathbf{x}_{cutmix}=\mathbf{M}\odot{\mathbf{x}_{i}}+(1-\mathbf{M}){\mathbf{x}_{j}}\\\
\mathbf{y}_{cutmix}=\lambda{\mathbf{y}_{i}}+(1-\lambda){\mathbf{y}_{j}},\end{gathered}$
(10)

where $\lambda=\frac{r_{w}r_{h}}{WH}$, is a mixing ratio.

Smashed Patch CutMix, which is proposed in this paper, is operated in the
feature space and is expressed as follows:

$\begin{gathered}\mathbf{s}_{cutmix}=\mathbf{M}\odot{\mathbf{s}_{i}}+(1-\mathbf{M}){\mathbf{s}_{j}}\\\
\mathbf{y}_{cutmix}=\lambda{\mathbf{y}_{i}}+(1-\lambda){\mathbf{y}_{j}}\end{gathered}$
(11)

The proposed masks a random number of patches with other client’s patches,
which is similar to the CutMix except for the number of patches to be
replaced; ours use multiple fixed-size patches.

Shuffled CutMix conducts shuffling after mixing smashed data by Smashed patch
CutMix, and is expressed as follows:

$\begin{gathered}\mathbf{s}_{shuffle}=\mathbf{S}(\mathbf{s}_{cutmix}),\end{gathered}$
(12)

where $\mathbf{S}$ is a shuffle operation on
$\textbf{s}_{cutmix}=[e_{1},...,e_{M}]\in\mathcal{R}^{M\times d_{m}}$. $e_{i}$
is the $i$-th embedding vector, $d_{m}$ is the size of a vector of a patch,
and $\mathbf{S}$ shuffles the sequence of the embedding vectors of
$\mathbf{s}_{cutmix}$. For CNN, activations can be reshaped to a 2D dimension
like activations in a vision transformer by dividing it with a given patch
size, and aligning them in parallel.

##  Appendix C Add-On Experiments

#### Shuffling

Transformers process data as a long-range sequence in parallel by attention
mechanism, and are not heavily affected by sequence order. This high
permutation invariance also applies to ViTs Naseer et al. [2021], and could
bring benefit on reducing privacy leakage by reconstruction attacks in SL
since shuffling patches can destroy images’ overall structure as shown in
Figure 4(a). While all patches retain their own positions during mixing by
Smashed Patch CutMix, it can be extended to a shuffled version, Shuffled
CutMix, in short, for a further privacy enhancement.

The last row of Table 1 shows that the influence of shuffling an order of
patches of CutMix data. ViT keeps its performance in spite of shuffling with a
slight performance loss by its capability of high permutation invariance.
However, PiT has a pooling operation that blurs shuffled patches and lose
their own distinct features, and this tendency of a degenerated performance is
more serious for CNN due to a consequence of convolutional filters in addition
to pooling. Nevertheless, a considerable profit by shuffling is an improvement
of reconstruction mitigation as shown in Table 3. It reduces the privacy
leakage by around 22 $\times$ compared to the the baseline (SL with smashed
data) with the ViT model.

#### FedAvg

FedAvg on clients can be utilized to enhance the performance gain and to be
tolerant to data distribution shifts. When data are non-IID to clients, each
client contains one or two dominant classes’ data, training its one-sided
classes predominantly. A mix operation like Smashed Patch CutMix can alleviate
unbalanced training in cooperation with FedAvg. In Figure 8(b), dirichlet
distribution with concentration parameter, $\mu$, 0.1 is used to formulate the
non-IID case. The result indicates that the degenerated performance of SL and
SFL by non-IID condition can be less worsened by Smashed Patch CutMix and
FedAvg.

During BP through CutMix data, the flow of gradients to each client can be
calculated from the perspective of the server and and the perspective of
clients. Pal et al. [2021] uses local gradient averaging for broadcasting the
gradients to clients. Likewise, the gradients from CutMix data can flow as
unicast and broadcast.

$\displaystyle\nabla_{\mathbf{w}_{c,i}}\tilde{L}_{i,j}\approx\begin{cases}\frac{\partial\tilde{L}_{\\{i,j\\}}}{\partial\mathbf{s}^{\prime}_{i}}\frac{\partial\mathbf{s}^{\prime}_{i}}{\partial\mathbf{w}_{c,i}};&\textrm{for
unicast}\\\
\frac{\partial\tilde{L}_{\\{i,j\\}}}{\partial\mathbf{s}_{\\{i,j\\}}}\frac{\partial\tilde{\mathbf{s}}_{\\{i,j\\}}}{\partial\mathbf{w}_{c,i}};&\textrm{for
broadcast}\end{cases}.$ (13)

The unicast case is that the gradients is calculated in the perspective of
clients, and the server knows paired pseudo random sequences indicating which
patches each client uploaded. The server sends individual gradients,
$\nabla_{\tilde{s}_{i}}\tilde{L}_{i,j}$ and
$\nabla_{\tilde{s}_{j}}\tilde{L}_{i,j}$, to $i$-th and $j$-th clients
respectively according to the corresponding portion of CutMix data by unicast.
For the broadcast case, the server is assumed not to know the pseudo random
sequences, thereby the gradient with respect to the whole region of CutMix
data is sent during BP. It is named as CutMix Gradient, and it broadcasts
combined gradients, $\nabla_{\tilde{s}_{\\{i,j\\}}}\tilde{L}_{i,j}$, to the
clients whose CutMix data are generated from. CutMix Gradient could be used
when the server does not know the pseudo random sequences.

Although the difference of the performances is negligible in the IID
condition, the performance of CutMixSFL with CutMix gradients is 6% higher
than the one with the gradients with respect to CutSmashed data in the non-IID
condition. It could be interpreted that CutMix Gradients by the mixed smashed
data is more effective in that each model trains data of scarce classes
indirectly through CutMix gradient with the help of FedAvg.

(a) IID.

(b) non-IID($\mu=0.1$).

Figure 8: Performance w.r.t. IID and non-IID data distribtuion.

#### Noise Injection

Many privacy-preserving data mining techniques involve noise injection, such
as differential privacy, to randomly distort and mask data reducing the
distance correlation between an output of a mechanism and a raw input data. In
Figure 9, the utility of mixing operations are evaluated when additive white
gaussian noises are integrated to the smashed data and its corresponding label
for data privacy. It shows a comparison of utility of Smashed Patch CutMix,
Smashed Mixup, and the baseline (Parallel SL) according to the scale,
$\sigma_{x}$ and $\sigma_{y}$ of the noise distribution on the raw data and
its label, respectively. The higher $\sigma$ preserves stronger privacy at the
larger cost of utility. The result indicates the proposed has a higher utility
compared to the others, even though the gap is decreased with a higher
$\sigma$. One interesting founding is that two cases except for the proposed
show utility gains when a small amount of noise is injected in Figure 9. It
could be explained that two methods get a positive effect through
regularization making the model less certain of its predictions, while the
proposed does not so since it already gets an enough regularization and data
augmentation effect.

Figure 9: Utility according to the scale of gaussian noise added to the
smashed data and its label.

##  Appendix D Additional Visualization of images

Additional examples of mix operations on raw images, smashed data and its
corresponding reconstruction images are shown in the next page. All the images
shown in Figure 10 are based on mixtures of two images (2-way).

(a) Raw images.

(b) Smashed data.

(c) Reconstruction from smashed data.

Figure 10: Additional examples of data with different operations at raw
images, smashed data, and reconstructed images.