11institutetext: Tampere University, Finland 22institutetext: RISE Research
Institutes of Sweden
22email: {tanveer.khan, khoa.nguyen<EMAIL_ADDRESS>
# A More Secure Split: Enhancing the Security of Privacy-Preserving Split
Learning††thanks: This work was funded by the Technology Innovation Institute
(TII) for the project ARROWSMITH and from Horizon Europe for HARPOCRATES
(101069535).

Tanveer Khan 11 0000-0001-7296-2178 Khoa Nguyen 11 0000-0001-5089-5250 Antonis
Michalas 1122 0000-0002-0189-3520

###### Abstract

Split learning (SL) is a new collaborative learning technique that allows
participants, e.g. a client and a server, to train machine learning models
without the client sharing raw data. In this setting, the client initially
applies its part of the machine learning model on the raw data to generate
Activation Maps (AMs) and then sends them to the server to continue the
training process. Previous works in the field demonstrated that reconstructing
AMs could result in privacy leakage of client data. In addition to that,
existing mitigation techniques that overcome the privacy leakage of SL prove
to be significantly worse in terms of accuracy. In this paper, we improve upon
previous works by constructing a protocol based on U-shaped SL that can
operate on homomorphically encrypted data. More precisely, in our approach,
the client applies homomorphic encryption on the AMs before sending them to
the server, thus protecting user privacy. This is an important improvement
that reduces privacy leakage in comparison to other SL-based works. Finally,
our results show that, with the optimum set of parameters, training with HE
data in the U-shaped SL setting only reduces accuracy by 2.65% compared to
training on plaintext. In addition, raw training data privacy is preserved.

###### Keywords:

Activation Maps Homomorphic Encryption Machine Learning Privacy Split Learning

##  1 Introduction

Nowadays, machine learning (ML) methods are widely used in many applications
due to their predictive and generative power. However, this raises serious
concerns regarding user data privacy, leading to the need for privacy-
preserving machine learning (PPML) solutions [9]. Split Learning (SL) and
Federated Learning (FL) are two PPML methods that rely on training ML models
on decentralized data sources [19]. In FL [22], every client runs a copy of
the entire model on its data. The server receives updated weights from each
client and aggregates them. The SL [7] model divides the Neural Network (NN)
into two parts: the client-side and the server-side. SL is used for training
NN among multiple data sources, while mitigating the need to directly share
raw labeled data with collaboration parties. The advantages of SL are
multifold:  (i) it allows multiple parties to collaboratively train a NN,
(ii) it allows users to train ML models without sharing their raw data with a
server running part of a NN model, thus preserving user privacy,  (iii) it
protects both the client and the server from revealing their parts of the
model, and  (iv) it reduces the client’s computational overhead by not running
the entire model [20].

Though SL offers an extra layer of privacy protection by definition, there are
no works exploring how it is combined with popular techniques that promise to
preserve user privacy (e.g. encryption). In [1], the authors studied whether
SL can handle sensitive time-series data and demonstrated that SL alone is
insufficient when performing privacy-preserving training for 1-dimensional
(1D) CNN models. More precisely, the authors showed raw data can be
reconstructed from the AMs of the intermediate split layer. The authors also
employed two mitigation techniques, adding hidden layers and applying
differential privacy to reduce privacy leakage. However, based on the results,
none of these techniques can effectively reduce privacy leakage from all
channels of the SL activation. Furthermore, both these techniques result in
significantly reducing the joint model’s accuracy.

In this paper, we focus on training an ML model in a privacy-preserving
manner, where a client and a server collaborate to train the model. More
specifically, we construct a model that uses Homomorphic Encryption (HE) [4]
to mitigate privacy leakage in SL. In our model, the client first encrypts the
AMs and then sends the Encrypted Activation Maps (EAMs) to the server. The
EAMs do not reveal anything about the raw data (i.e. it is not possible to
reconstruct the original raw data from the EAM).

Contributions: The main contributions of this paper are:

  1. bx1.

We designed a simplified version of the 1D CNN model presented in [1] and we
are using it to classify the ECG signals [16] in both local and SL settings.
More specifically, we construct a U-shaped split 1D CNN model and experiment
using plaintext AMs sent from the client to the server. Through the U-shaped
1D CNN model, clients do not need to share the input training samples and the
ground truth labels with the server – this is an important improvement that
reduces privacy leakage compared to [1].

  2. bx2.

We constructed the HE version of the U-shaped SL technique. In the encrypted
U-shaped SL model, the client encrypts the AM using HE and sends it to the
server. The advantage of HE encrypted U-shaped SL over the plaintext U-shaped
SL is that server performs computation over EAMs.

  3. bx3.

To assess the applicability of our framework, we performed experiments on two
heartbeat datasets: the MIT-DB [16] and the PTB-XL [21], with PTB-XL currently
being the largest open-source electrocardiography dataset to our knowledge.
For the MIT-DB dataset, we experimented with AMs of two lengths (256 and 512)
for both plaintext and homomorphically EAMs and we have measured the model’s
performance by considering training duration test accuracy, and communication
cost. We performed similar experiments with PTB-XL dataset, however, only with
AMs of length 256.

  4. bx4.

Moreover, our framework takes advantage of batch encryption, an optimization
technique for memory and computation, to improve computing performance over
encrypted data. We conducted experiments with and without batch encryption and
compared results.

  5. bx5.

We designed a detailed protocol to prove our construction’s and provide proof
of its security level under the malicious threat model.

Organization: The rest of the paper is organized as follows111Due to space
constraints, the necessary background information about 1D CNN, HE and SL are
in Appendix 0.A.: In section 2, we present important published works in the
area of SL. The architectures of the proposed models are presented in section
3. The design and implementation of split 1D CNN training protocols are
described in section 4, formal protocol construction in section 5, protocol
security in section 6, extensive experimental results in section 7, and
conclude the paper in section 8.

##  2 Related Work

One of the primary reasons researchers seek novel techniques is to bridge the
large gap between existing privacy solutions and the actual practical
deployment of NNs. PPML consists of cryptographic approaches such as HE and
Multiparty Computation (MPC), differential privacy as well as distributed ML
approaches such as FL, and SL [3]. Google AI Blog introduced FL, where users
(e.g. mobile devices) collaboratively train a model [15] under a central
server’s orchestration (e.g. service provider) without sharing their data.
However, in FL, sharing user model weights with server can lead to sensitive
information leaks [8].

SL approach [7] is a promising approach in terms of client raw data
protection, however, SL provides data privacy on the grounds that only
intermediate AMs are shared between the parties. Different studies showed the
possibility of privacy leakage in SL. In [19], the authors analyzed the
privacy leakage of SL and found a considerable leakage from the split layer in
the 2D CNN model. Furthermore, the authors mentioned that it is possible to
reduce the distance correlation (a measure of dependence) between the split
layer and raw data by slightly scaling the weights of all layers before the
split. This scaling works well in models with a large number of hidden layers
before the split.

The work of Abuadbba et al. [1] is the first study exploring whether SL can
deal with time-series data. The authors proved that only SL cannot preserve
the privacy of the data, and employed two techniques to resolve this privacy
problem. However, both suffer from a loss of model accuracy, with the use of
differential privacy degrading the classification accuracy significantly from
98.9% to 50%.

##  3 Architecture

In this section, we first describe the non-split version or local model of the
1D CNN used to classify the ECG signal. Then, we discuss the process of
splitting this local model into a U-shaped split model. Furthermore, we also
describe the involved parties (a client and a server) in the training process
of the split model, focusing on their roles and the parameters assigned to
them throughout the training process. Notations for all parameters and their
descriptions is in Table 1.

Table 1: Parameters and Description in the Algorithms

# | ML | Description | HE | Description
---|---|---|---|---
| Parameters |  | Parameters |
1 | D | Dataset | $\mathcal{P}$ | Polynomial modulus
2 |  $\mathbf{x}$, $\mathbf{y}$ | Input data samples and ground-truth labels | $\mathcal{C}$ | Coefficient modulus
3 | n, N | Batch size and number of batches to be trained | $\Delta$ | Scaling factor
4 | $\boldsymbol{w}^{i},\boldsymbol{b}^{i}$ |  Weights and biases in layer i | CKKS | Encryption scheme
5 | $f^{i}$ |  Linear or convolution operation of layer i | $\mathsf{pk}$ | Public key
6 | $g^{i}$ |  Activation function of layer i | $\mathsf{sk}$ | Secret key
7 | $\mathbf{a}^{i}$ |  Output activation maps of $g^{i}$ | $\mathsf{HE.Enc}$ | Homomorphic encryption
8 | $\mathbf{z}^{i}$ |  Output tensor of $f^{i}$ | $\mathsf{HE.Dec}$ | Homomorphic decryption
9 | $\eta$ | Learning rate | $\mathsf{HE.Eval}$ | Homomorphic evaluation
10 | $\boldsymbol{\Phi}$ | Model’s weights | $\mathsf{ctx_{pri}}$ | Private context
11 | $E$ | Number of training epochs | $\mathsf{ctx_{pub}}$ | Public context
12 |  $\mathcal{L}$, $J$ | Loss function and error | $\bar{\mathbf{a}}^{i}$ | Encrypted activation maps
13 | $O$ | Optimizer | $\bar{\mathbf{z}}^{i}$ | Encrypted tensor

###  3.1 1D CNN Local Model Architecture

We first implement and successfully reproduce the local model results [1].
This model contains two Conv1D layers and two FC layers. The optimal test
accuracy that this model achieves is 98.9%. We implement a simplified version
where the model has one less FC layer compared to the model from [1]. Our
local model consists of all the layer of Figure 3 without any split between
the client and the server. As can be seen in Figure 3, we limit our model to
two Conv1D layers and one linear layer as we aim to reduce computational costs
when HE is applied on AMs in the model’s split version. Reducing the number of
FC layers leads to a drop in the accuracy of the model. The best test accuracy
we obtained after training our local model for 10 epochs with a n of 4 is
92.84%. Although reducing the number of layers affects the model’s accuracy,
it is not within our goals to demonstrate how successful our ML model is for
this task; instead, our focus is to construct a split model where training and
evaluation on encrypted data are comparable to training and evaluation on
plaintext data. We also apply the simplified 1D CNN on the PTB-XL dataset,
with a small modification due to the difference in the number of input
channels compared to the dataset from [16]. The training result on the PTB-XL
dataset after 10 epochs with a n of 4 is 74.01%, with the best test accuracy
of 67.36%. In section 7, we detail results for the non-split version and
compare them with split version.

The training process of the local 1D CNN can be described as following:
Suppose we have a heartbeat data sample $\mathbf{x}\in\mathbb{R}^{c}$, where
$c$ is the number of input features or the number of time steps. $\mathbf{x}$
belongs to one out of $m$ ground-truth classes. Each data sample $\mathbf{x}$
has a corresponding encoded label vector $\mathbf{y}\in\mathbb{R}^{m}$ that
represents its ground-truth class. We can write the 1D CNN as a function
$f_{\boldsymbol{\Phi}}$, where $\boldsymbol{\Phi}$ is a set of adjustable
parameters as denoted in Table 1. $\Phi$ is first initialized to small random
values in the range $[-1,1]$. Our aim is to find the best set of parameters to
map $\mathbf{x}$ to a predicted output vector
$\mathbf{\hat{y}}\in\mathbb{R}^{m}$, where $\mathbf{\hat{y}}$ is as close as
possible to $\mathbf{y}$. $\mathbf{\hat{y}}$ can be a vector of $m$
probabilities, and we pick $\mathbf{x}$ to belong to the class with the
highest probability. To find the closest value of $\mathbf{\hat{y}}$ with
respect to $\mathbf{y}$, we try to minimize a loss function
$\mathcal{L}({\mathbf{\hat{y}}},\mathbf{y})$. Training the 1D CNN is an
iterative process to find the best $\boldsymbol{\Phi}$ to minimize the loss
function. This process consists of two sub-processes called “forward
propagation” and “backward propagation”. More specifically, forward
propagation moves from the input $\mathbf{x}$ throughout the network, reaching
the output layer and produces the predicted output $\mathbf{\hat{y}}$.
Conversely, backward propagation moves from the network’s output layer back to
the input layer to calculate the gradients of the loss function $\mathcal{L}$
w.r.t the weights $\boldsymbol{\Phi}$ of the network. These weights are then
updated according to the gradients. The process of calculating the predicted
output, the loss function, the gradients and then updating the weights is
called “training”. We train the NN with thousands of samples of $\mathbf{x}$’s
and corresponding $\mathbf{y}$’s, through many iterations of forward and
backward propagation. We do not train the network on each single data example,
but use a number of them at a time (defined by the n). The total number of
training batches is $N=\frac{|D|}{n}$, where $|D|$ is the size of the dataset.
Once the NN goes through all the training batches, it has completed one
training epoch. This process repeats for $E$ epochs in total.

###  3.2 U-shaped Split 1D CNN Model

In this section, we first present the constructed U-shaped split model. We
then report in more detail the roles and access rights of the actors who are
involved in the training protocols of the split 1D CNN on both plaintexts and
EAMs. The SL protocol consists of two parties: the client and server. We split
the local 1D CNN into multiple parts, where each party trains its part(s) and
communicates with others to complete the overall training procedure. More
specifically, we construct the U-shaped split 1D CNN in such a way that the
first few as well as the last layer are on the client-side, while the
remaining layers are on the server-side, as demonstrated in Figure 3. The
client and server collaborate to train the split model by sharing the AMs and
gradients. On the client-side, there are two Conv1D, two Max Pooling, two
Leaky ReLU layers, and a Softmax layer. On server-side, there is only one
linear layer. As mentioned earlier, the reason for having only one linear
layer on the server-side is due to computational constraints when training on
encrypted data.

###  3.3 Actors in the Split Learning Model

As mentioned earlier, in our SL setting, we have two involved parties: the
client and the server. Each party plays a specific role and has access to
certain parameters. More specifically, their roles and accesses are described
as

  * •

Client: In the plaintext version, the client holds two Conv1D layers and can
access their weights and biases in plaintext. In the HE encrypted version, the
client generates the HE context and has access to all context parameters
($\mathcal{P}$, $\mathcal{C}$, $\Delta$, $\mathsf{pk}$ and $\mathsf{sk}$).
Note that for both training on plaintext and EAMs, the raw data examples
$\mathbf{x}$’s and their corresponding labels $\mathbf{y}$’s reside on the
client side and are never sent to the server during the training process.

  * •

Server: In our model, the computation performed on the server-side is limited
to only one linear layer. Hence, the server can exclusively access the weights
and biases of this linear layer. The server also has access to the HE
parameters except for the secret key $\mathsf{sk}$. The hyperparameters shared
between the client and the server are $\eta$, $n$, $N$, $E$.

##  4 Split Model Training Protocols

We first present the protocol for training the U-shaped split 1D CNN on
plaintext AMs, followed by training the U-shaped split 1D CNN on EAMs.

###  4.1 Plaintext Activation Maps

We have used algorithm 1 and algorithm 2 to train the U-shaped split 1D CNN
reported in subsection 3.2. First, the client and server start the socket
initialization process and synchronize the hyperparameters $\eta,n,N,E$. They
also initialize the weights of their layers according to $\Phi$.

During forward propagation , the client forward-propagates the input
$\mathbf{x}$ until $l^{th}$ layer and sends $\mathbf{a}^{(l)}$ to the server.
The server continues to forward propagate and sends the output
$\mathbf{a}^{(L)}$ to the client. Next, the client applies the Softmax
function on $\mathbf{a}^{(L)}$ to get $\mathbf{\hat{y}}$ and calculates the
error $J=\mathcal{L}(\mathbf{\hat{y}},\mathbf{y})$.

The client starts the backward propagation by calculating and sending the
gradient of the error w.r.t $\mathbf{a}^{(L)}$, i.e. $\frac{\partial
J}{\partial\mathbf{a}^{(L)}}$, to the server. The server continues the
backward propagation, calculates $\frac{\partial J}{\partial\mathbf{a}^{(l)}}$
and sends $\frac{\partial J}{\partial\mathbf{a}^{(l)}}$ to the client. After
receiving the gradients $\frac{\partial J}{\partial\mathbf{a}^{(l)}}$ from the
server, the backward propagation continues to the first hidden layer on the
client-side. Note that the exchange of information between client and server
in these algorithms takes place in plaintext. As can be seen in algorithm 1,
the client sends the AMs $\mathbf{a}^{(l)}$ to the server in plaintext and
receives the output of the linear layer $\mathbf{a}^{(L)}$ from the server in
plaintext. The same applies on the server side: receiving $\mathbf{a}^{(l)}$
and sending $\mathbf{a}^{(L)}$ in the plaintext as can be seen in algorithm 2.
Sharif et al. [1] showed that the exchange of plaintext AMs between client and
server using SL reveals important information regarding the client’s raw data.
Later, in subsection 7.4 we show in detail how passing the forward AMs from
client to server in the plaintext will result in information leakage. To
mitigate this privacy leakage, we propose the protocol, where the client
encrypts AMs before sending them to the server (see subsection 4.2).

###  4.2 Encrypted Activation Maps

The protocol for training the U-shaped 1D CNN with a homomorphically EAM
consists of four phases: initialization, forward propagation, classification,
and backward propagation. The initialization phase only takes place once at
the beginning of the procedure, whereas the other phases continue until the
model iterates through all epochs. Each phase is explained in detail below.

#### Initialization

This phase consists of socket initialization, context generation, and random
weight loading. The client establishes a socket connection to the server and
synchronizes the four hyperparameters $\eta,\ n,\ N,E$ with the server, shown
in algorithm 3 and algorithm 4. These parameters must be synchronized on both
sides to be trained in the same way. Also, the weights on the client and
server are initialized with the same set of corresponding weights in the local
model to accurately assess and compare the influence of SL on performance. On
both, client and server, $\boldsymbol{w}^{(i)}$ are initialized using
corresponding parts of $\Phi$. $\mathbf{a}^{(i)}$, $\mathbf{z}^{(i)}$, and the
gradients are initially set to zero. In this phase, the context generated is
an object that holds $\mathsf{pk}$ and $\mathsf{sk}$ of the HE scheme as well
as $\mathcal{P}$, $\mathcal{C}$ and $\Delta$.  Initialization: $s\leftarrow$
socket initialized with port and address; s.connect $\eta,n,N,E\leftarrow
s.synchronize()$ $\\{\boldsymbol{w}^{(i)},\boldsymbol{b}^{(i)}\\}_{\forall
i\in\\{0..l\\}}\ \leftarrow\ initialize\ using\ \Phi$
$\\{\mathbf{z}^{(i)}\\}_{\forall
i\in\\{0..l\\}},\\{\mathbf{a}^{(i)}\\}_{\forall
i\in\\{0..l\\}}\leftarrow\emptyset\ $ $\left\\{\frac{\partial
J}{\partial\mathbf{z}^{(i)}}\right\\}_{\forall
i\in\\{0..l\\}},\left\\{\frac{\partial
J}{\partial\mathbf{a}^{(i)}}\right\\}_{\forall
i\in\\{0..l\\}}\leftarrow\emptyset\ $ for  _$\displaystyle e\ \in\ E$_ do for
_$\displaystyle\text{each}\ \text{batch}\ (\mathbf{x},\ \mathbf{y})\
\text{from}\ D\ $_ do $\displaystyle\mathbf{Forward\ propagation:}$
$\displaystyle\ \ \ \ O.zero\\_grad()$ $\displaystyle\ \ \ \ \mathbf{a}^{0}\ \
\leftarrow\mathbf{x}$ for  _$i\leftarrow 1$ to $l$_ do $\displaystyle\ \ \ \
\mathbf{for}\ i\ \leftarrow\ 1\ \mathbf{to}\ l\ \mathbf{do}$ $\displaystyle\ \
\ \ \ \ \ \ \mathbf{z}^{(i)}\ \leftarrow\
f^{(i)}\left(\mathbf{a}^{(i-1)}\right)$ $\displaystyle\ \ \ \ \ \ \ \
\mathbf{a}^{(i)}\ \leftarrow\ g^{(i)}\left(\mathbf{z}^{(i)}\right)$ end for
$\displaystyle\ \ \ \ s.send\ (\mathbf{a}^{(l)})$ $\displaystyle\ \ \ \
s.receive\ (\mathbf{a}^{(L)})$ $\displaystyle\ \ \ \ \hat{y}\ \leftarrow\
Softmax\left(\mathbf{a}^{(L)}\right)$ $\displaystyle\ \ \ \
J\leftarrow\mathcal{L}(\hat{\mathbf{y}},\mathbf{y})$
$\displaystyle\mathbf{Backward\ propagation:}$ $\displaystyle\ \ \ \
\text{Compute}\left\\{\frac{\partial J}{\partial\hat{\mathbf{y}}}\ \&\
\frac{\partial J}{\partial\mathbf{a}^{(L)}}\right\\}$ $\displaystyle\ \ \ \
s.send\ \left(\frac{\partial J}{\partial\mathbf{a}^{(L)}}\right)$
$\displaystyle\ \ \ \ s.receive\ \left(\frac{\partial
J}{\partial\mathbf{a}^{(l)}}\right)$ for  _$i\leftarrow 1$ to $l$_ do
$\text{Compute}\ \left\\{\frac{\partial J}{\partial\boldsymbol{w}^{(i)}},\
\frac{\partial J}{\partial\boldsymbol{b}^{(i)}}\right\\}$ $\displaystyle\ \ \
\ \ \ \ \ \text{Update}\ \boldsymbol{w}^{(i)},\ \boldsymbol{b}^{(i)}$ end for
end for end for Algorithm 1 Client Side Initialization: $s\leftarrow$ socket
initialized ; s.connect $\eta,n,N,E\leftarrow s.synchronize()$
$\\{\boldsymbol{w}^{(i)},\boldsymbol{b}^{(i)}\\}_{\forall i\in\\{0..l\\}}\
\leftarrow\ initialize\ using\ \Phi$ $\displaystyle\ \ \ \
\\{\mathbf{z}^{(i)}\\}_{\forall i\in\\{l+1..L\\}}\leftarrow\emptyset\ $
$\displaystyle\ \ \ \ \left\\{\frac{\partial
J}{\partial\mathbf{z}^{(i)}}\right\\}_{\forall
i\in\\{l+1..L\\}}\leftarrow\emptyset\ $ for  _$\displaystyle e\ \in\ E$_ do
for  _$\displaystyle i\leftarrow 1\ \mathbf{to}\ N\ $_ do
$\displaystyle\mathbf{Forward\ propagation:}$ $\displaystyle\ \ \ \
O.zero\\_grad()$ $\displaystyle\ \ \ \ s.receive\ (\mathbf{a}^{(l)})\ \ $
$\displaystyle\ \ \ \ \mathbf{a}^{(L)}\ \leftarrow\
f^{(i)}\left(\mathbf{a}^{(l)}\right)$ $\displaystyle\ \ \ \
s.send\left(\mathbf{a}^{(L)}\right)$ $\displaystyle\mathbf{Backward\
propagation:}$ $\displaystyle\ \ \ \ s.receive\ \left(\frac{\partial
J}{\partial\mathbf{a}^{(L)}}\right)$ $\displaystyle\ \ \ \ \text{Compute}\
\left\\{\frac{\partial J}{\partial\boldsymbol{w}^{(L)}},\ \frac{\partial
J}{\partial\boldsymbol{b}^{(L)}}\right\\}$ $\displaystyle\ \ \ \
\text{Update}\ \boldsymbol{w}^{(L)},\ \boldsymbol{b}^{(L)}$ $\displaystyle\ \
\ \ \text{Compute}\ \frac{\partial J}{\partial\mathbf{a}^{(l)}}$
$\displaystyle\ \ \ \ s.send\left(\frac{\partial
J}{\partial\mathbf{a}^{(l)}}\right)$ end for end for Algorithm 2 Server Side

Further information on the HE parameters and how to choose the best-suited
parameters can be found in the TenSEAL’s benchmarks tutorial. As shown in
algorithm 3 and algorithm 4, the context is either $\mathsf{ctx_{pub}}$ or
$\mathsf{ctx_{pri}}$ depending on whether it holds the secret key
$\mathsf{sk}$. Both the $\mathsf{ctx_{pub}}$ and $\mathsf{ctx_{pri}}$ have the
same parameters, though $\mathsf{ctx_{pri}}$ holds a $\mathsf{sk}$ and
$\mathsf{ctx_{pub}}$ does not. The server does not have access to the
$\mathsf{sk}$ as the client only shares the $\mathsf{ctx_{pub}}$ with the
server.

#### Forward propagation

In the forward propagation the client first zeroes out the gradients for the
batch of data $(\mathbf{x},\mathbf{y})$. He then begins calculating the
$\mathbf{a}^{(l)}$ AMs from $\mathbf{x}$, as can be seen in algorithm 3 where
each $f^{(i)}$ is a Conv1D layer.

The Conv1D layer can be described as following: given a 1D input signal that
contains $C$ channels, where each channel $\mathbf{x}_{(i)}$ is a 1D array
($i\in\\{1,\ldots,C\\}$), a Conv1D layer produces an output that contains
$C^{\prime}$ channels. The $j^{th}$ output channel $\mathbf{y}_{(j)}$, where
$j\in\\{1,\ldots,C^{\prime}\\}$, can be described as

$\scriptsize\mathbf{y}_{(j)}=\boldsymbol{b}_{(j)}+\sum_{i=1}^{C}\boldsymbol{w}_{(i)}\star\mathbf{x}_{(i)},$
(1)

where $\boldsymbol{w}_{(i)},i\in\\{1,\ldots,C\\}$ are the weights,
$\boldsymbol{b}_{(j)}$ are the biases of the Conv1D layer, and $\star$ is the
1D cross-correlation operation. The $\star$ operation can be described as

$\scriptsize\mathbf{z}(i)=(\boldsymbol{w}\star\mathbf{x})(i)=\sum_{j=0}^{m-1}\boldsymbol{w}(j)\cdot\mathbf{x}(i+j),$
(2)

where $\mathbf{z}(i)$ denotes the $i^{th}$ element of $\mathbf{z}$, and size
of the 1D weighted kernel is $m$.

In algorithm 3, $g^{(i)}$ can be seen as the combination of Max Pooling and
Leaky ReLU functions. The final output AMs of the $l^{th}$ layer from the
client is $\mathbf{a}^{(l)}$. The client then homomorphically encrypts
$\mathbf{a}^{(l)}$ and sends the EAMs $\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern 1.5mu$ to the
server. In algorithm 4, the server receives $\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern 1.5mu$ and then
performs forward propagation, which is a linear layer evaluated on HE
encrypted data $\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern 1.5mu$ as

$\scriptsize\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)}\mkern-1.5mu}\mkern 1.5mu=\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern
1.5mu\boldsymbol{w}^{(L)}+\boldsymbol{b}^{(L)}.$ (3)

Upon reception, the client decrypts $\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)}\mkern-1.5mu}\mkern 1.5mu$ to get
$\mathbf{a}^{(L)}$, performs Softmax on $\mathbf{a}^{(L)}$ to produce the
predicted output $\mathbf{\hat{y}}$ and calculate the loss $J$, as can be seen
in algorithm 3. Having finished the forward propagation we may move on to the
backward propagation part of the protocol.

#### Backward propagation

After calculating the loss $J$, the client starts the backward propagation by
initially computing $\frac{\partial J}{\partial\hat{\mathbf{y}}}$ and then
$\frac{\partial J}{\partial\mathbf{a}^{(L)}}$ and $\frac{\partial
J}{\partial\boldsymbol{w}^{(L)}}$ using the chain rule (algorithm 3).
Specifically, the client calculates: ∂J∂a(L) &= ∂J∂^y ∂^y∂a(L), ∂J∂w(L) =
∂J∂a(L) ∂a(L)∂w(L)

Following, the client sends $\frac{\partial J}{\partial\mathbf{a}^{(L)}}$ and
$\frac{\partial J}{\partial\boldsymbol{w}^{(L)}}$ to the server. Upon
reception, the server computes $\frac{\partial J}{\partial\boldsymbol{b}}$ by
simply doing $\frac{\partial J}{\partial\boldsymbol{b}}=\frac{\partial
J}{\partial\mathbf{a}^{(L)}}$, based on equation eq:serverHELinear. The server
then updates weights and biases of linear layer according to equation
equ:serverUpdateWB.

w^(L) = w^(L) - η∂J∂w(L), & b^(L) = b^(L) - η∂J∂b(L)

Next, the server calculates

$\scriptsize\frac{\partial J}{\partial\mathbf{a}^{(l)}}=\frac{\partial
J}{\partial\mathbf{a}^{(L)}}\frac{\partial\mathbf{a}^{(L)}}{\partial\mathbf{a}^{(l)}},$
(4)

and sends $\frac{\partial J}{\partial\mathbf{a}^{(l)}}$ to the client. After
receiving $\frac{\partial J}{\partial\mathbf{a}^{(l)}}$, the client calculates
the gradients of $J$ w.r.t the weights and biases of the Conv1D layer using
the chain-rule, which can generally be described as: ∂J∂w(i-1) &=
∂J∂w(i)∂w(i)∂w(i-1), ∂J∂b(i-1) = ∂J∂b(i)∂b(i)∂b(i-1)

Finally, after calculating the gradients $\frac{\partial
J}{\partial\boldsymbol{w}^{(i)}},\ \frac{\partial
J}{\partial\boldsymbol{b}^{(i)}}$, the client updates $\boldsymbol{w}^{(i)}$
and $\boldsymbol{b}^{(i)}$ using the Adam optimization algorithm [12].

##  5 Formal Protocol Construction

In this section, we formalize the communication between the client and the
server. To this end, we design a protocol that is divided in two phases (Setup
and Running) and relies on the following five building blocks:

Context Initialization:

$\displaystyle\ \ \ \ \ \mathsf{ctx_{pri}},\ \leftarrow\ \mathcal{P},\
\mathcal{C},\ \Delta,\ \mathsf{pk},\ \mathsf{sk}$

$\displaystyle\ \ \ \ \ \mathsf{ctx_{pub}},\ \leftarrow\ \mathcal{P},\
\mathcal{C},\ \Delta,\ \mathsf{pk}$

$\displaystyle\ \ \ \ \ s.send(\mathsf{ctx_{pub}})$

for  _$\displaystyle e\ \text{in}\ E$_ do

for  _$\displaystyle\text{each}\ \text{batch}\ (\mathbf{x},\ \mathbf{y})\
\text{from}\ \mathbf{D}\ $_ do

$\displaystyle\mathbf{Forward\ propagation:}$

$\displaystyle\ \ \ \ O.zero\\_grad()$

$\displaystyle\ \ \ \ \mathbf{a}^{0}\ \ \leftarrow\mathbf{x}$

for  _$i\ \leftarrow\ 1\ \mathbf{to}\ l$_ do

$\displaystyle\ \ \ \ \ \ \ \ \mathbf{z}^{(i)}\ \leftarrow\
f^{(i)}\left(\mathbf{a}^{(i-1)}\right)$

$\displaystyle\ \ \ \ \ \ \ \ \mathbf{a}^{i}\ \leftarrow\
g^{(i)}\left(\mathbf{z}^{(i)}\right)$

end for

$\displaystyle\ \ \ \ \mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern 1.5mu\
\leftarrow\ \mathsf{HE.Enc}\left(\mathsf{pk},\mathbf{a}^{(l)}\right)$

$\displaystyle\ \ \ \ s.send\ \mkern
1.5mu\overline{\mkern-1.5mu(\mathbf{a}^{(l)})\mkern-1.5mu}\mkern 1.5mu$

$\displaystyle\ \ \ \ s.receive\ (\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)})\mkern-1.5mu}\mkern 1.5mu$

$\displaystyle\ \ \ \ \mathbf{a}^{(L)}\ \leftarrow\
\mathsf{HE.Dec}\left(\mathsf{sk},\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)}\mkern-1.5mu}\mkern 1.5mu\right)$

$\displaystyle\ \ \ \ \hat{\mathbf{y}}\ \leftarrow\
Softmax\left(\mathbf{a}^{(L)}\right)$

$\displaystyle\ \ \ \
\mathbf{J}\leftarrow\mathcal{L}(\hat{\mathbf{y}},\mathbf{y})$

$\displaystyle\mathbf{Backward\ propagation:}$

$\displaystyle\ \ \ \ \text{Compute}\left\\{\frac{\partial
J}{\partial\hat{\mathbf{y}}}\&\frac{\partial
J}{\partial\mathbf{a}^{(L)}}\&\frac{\partial
J}{\partial\boldsymbol{w}^{(L)}}\right\\}$

$\displaystyle\ \ \ \ s.send\left(\frac{\partial
J}{\partial\mathbf{a}^{(L)}}\&\frac{\partial
J}{\partial\boldsymbol{w}^{(L)}}\right)$

$\displaystyle\ \ \ \ s.receive\left(\frac{\partial
J}{\partial\mathbf{a}^{(l)}}\right)$

for  _$i\leftarrow l\ \text{downto}\ 1$_ do

$\displaystyle\ \ \ \ \ \ \ \ \text{Compute}\left\\{\frac{\partial
J}{\partial\boldsymbol{w}^{(i)}},\ \frac{\partial
J}{\partial\boldsymbol{b}^{(i)}}\right\\}$

$\displaystyle\ \ \ \ \ \ \ \ \text{Update}\ \boldsymbol{w}^{(i)},\
\boldsymbol{b}^{(i)}$

end for

end for

end for

Algorithm 3 Client Side

Context Initialization:

$\displaystyle\ \ \ \ \ s.receive(\mathsf{ctx_{pub}})$

for  _$\displaystyle\mathbf{e}\ \text{in}\ \mathbf{E}$_ do

for  _$\displaystyle i\leftarrow 1\ \mathbf{to}\ N\ $ _ do

$\displaystyle\mathbf{Forward\ propagation:}$

$\displaystyle\ \ \ \ O.zero\\_grad()$

$\displaystyle\ \ \ \ s.receive\ \mkern
1.5mu\overline{\mkern-1.5mu(\mathbf{a}^{(l)})\mkern-1.5mu}\mkern 1.5mu$

$\displaystyle\ \ \ \ \mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)}\mkern-1.5mu}\mkern 1.5mu\
\leftarrow\ \mathsf{HE.Eval}\left(f^{(i)}\left(\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(l)}\mkern-1.5mu}\mkern
1.5mu\right)\right)$

$\displaystyle\ \ \ \ s.send\left(\mkern
1.5mu\overline{\mkern-1.5mu\mathbf{a}^{(L)}\mkern-1.5mu}\mkern 1.5mu\right)$

$\displaystyle\mathbf{Backward\ propagation:}$

$\displaystyle\ \ \ \ s.receive\left\\{\frac{\partial
J}{\partial\mathbf{a}^{(L)}}\&\frac{\partial
J}{\partial\boldsymbol{w}^{(L)}}\right\\}$

$\displaystyle\ \ \ \ \text{Compute}\ \frac{\partial
J}{\partial\boldsymbol{b}^{(L)}}$

$\displaystyle\ \ \ \ \text{Update}\ \boldsymbol{w}^{(L)},\
\boldsymbol{b}^{(L)}$

$\displaystyle\ \ \ \ \text{Compute}\ \frac{\partial
J}{\partial\mathbf{a}^{(l)}}$

$\displaystyle\ \ \ \ s.send\left(\frac{\partial
J}{\partial\mathbf{a}^{(l)}}\right)$

end for

end for

Algorithm 4 Server Side

  * •

A CCA2 secure public-key encryption scheme
${\mathsf{PKE}=\left(\mathsf{Gen,Enc,Dec}\right)}$;

  * •

An EUF-CMA secure signature scheme ${\mathsf{Sign}=(\mathsf{\sigma,ver})}$;

  * •

A Leveled Homomorphic Encryption scheme
${\mathsf{HE}=(\mathsf{KeyGen,Enc,Eval,Dec})}$;

  * •

A first and second pre-image resistant hash function $H$;

  * •

A synchronized clock between the Client and the Server.

#### Setup Phase

During this phase each entity generates a public/private key pair
$\mathsf{(pk,sk)}$ for the CCA2-secure public-key encryption scheme
$\mathsf{PKE}$ and a sign/verification key pair $\mathsf{(\sigma,ver)}$ for
the EUF-CMA-secure signature scheme $\mathsf{Sign}$. Furthermore, the client
runs $\mathsf{HE.KeyGen}$ to generate the public, private and evaluation key
of the HE scheme. Below we provide a list of all generated keys:

  * •

$\mathsf{(pk_{C},sk_{C})}$ \- public/private key pair for the Client;

  * •

$\mathsf{(\sigma_{C},ver_{C})}$ \- sign/verification key pair for the Client;

  * •

$\mathsf{(pk_{S},sk_{S})}$ \- public/private key pair for the Server;

  * •

$\mathsf{(\sigma_{S},ver_{S})}$ \- sign/verification key pair for the Server;

  * •

$\mathsf{(pk_{HE},sk_{HE},evk_{HE})}$ \- public, private and evaluation keys
generated by Client.

#### Running Phase

After successfully executing the Setup phase, the Client initiates the
protocol’s running phase by sending

$m_{1}=\langle
t_{1},\mathsf{PKE.Enc}(\mathsf{pk_{S}},\mathsf{evk_{HE}}),\mathsf{HE.Enc}(\mathsf{pk_{HE}},\mathrm{AMap}),\sigma_{C}(H_{1})\rangle$
to the server, where $t_{1}$ is a timestamp, $H_{1}$ is a hash such that:

$H_{1}=H(\mathsf{evk_{HE}}\|\mathsf{HE.Enc}(\mathsf{pk_{HE}},\mathrm{AMap}))$,
$\mathrm{AMap}$ is the AM and $\sigma_{C}(\cdot)$ denotes the cryptographic
signature of $C$. Upon reception, the Server verifies the freshness of the
message by looking at the timestmap $t_{1}$ and the signature of the sender.
If any of the verifications fail, the Server outputs $\perp$ and aborts the
protocol. Otherwise, it first decrypts the evaluation key $\mathsf{evk}_{HE}$
using its private key $\mathsf{sk_{S}}$ and subsequently uses the homomorphic
evaluation key $\mathsf{evk_{HE}}$ to operate on $\mathrm{Amap}$. The result
of these operations is a homomorphically encrypted output
$\mathsf{HE.Enc(pk_{HE},out)}$, which is sent back to the Client via:
$m_{2}=\langle t_{2},\mathsf{HE.Enc(pk_{HE},out)},\sigma_{S}(H_{2})\rangle$,
where $H(2)=H(t_{2}\|\mathsf{HE.Enc(pk_{HE},out)})$. Upon reception, the
Client first verifies the freshness of the message and the server’s signature.
Should the verification fail, the Client outputs $\perp$ and aborts the
protocol. Otherwise, the Client first recovers the encrypted output by running
$\mathsf{HE.Dec}(\mathsf{sk}_{\mathsf{HE}},\mathsf{HE.Enc(pk_{\mathsf{HE}},out)})\rightarrow\mathsf{out}$.
This output will be used by the Server to compute the initial gradients
$\mathsf{grad}$ for the ML model. Having computed $\mathsf{grad}$ the Client
forwards them to the Server via: $m_{3}=\langle
t_{3},\mathsf{PKE.Enc}(\mathsf{pk}_{C},\mathsf{grad}),\sigma_{S}(H_{3})\rangle$,
where $H_{3}=H(t_{3}\|\mathsf{grad})$. Upon receiving $m_{3}$, the Server
first verifies the freshness and the signature of the message. Should the
verification fail, the Server outputs $\perp$ and aborts the protocol.
Otherwise, it recovers the gradients by running
$\mathsf{PKE.Dec(sk_{S},PKE.Enc(pk_{S},grad)})\rightarrow\mathsf{grad}$. Based
on $\mathsf{grad}$, the Server can update the parameters of the ML model (i.e.
bias and weights), a process resulting to updated gradients
$\mathsf{grad^{\prime}}$. Finally, the Server outsources
$\mathsf{grad}^{\prime}$ to the Client via: $m_{4}=\langle
t_{4},\mathsf{PKE.Enc(pk_{C},grad^{\prime}}),\sigma_{C}(H_{4})\rangle$, where
$H_{4}=H(t_{4}\|\mathsf{grad}^{\prime}))$. Upon reception, Server verifies
freshness and message signature. Should the verification fail, Client outputs
$\perp$ and aborts the protocol. Otherwise it decrypts the updated gradients
by running
$\mathsf{PKE.Dec(sk_{C},\mathsf{PKE.Enc(pk_{C},grad^{\prime}}))}\rightarrow\mathsf{grad^{\prime}}$.
The running phase of our protocol is illustrated in Figure 3.

##  6 Protocol Security

We prove the security of our protocol in presence of a probabilistic
polynomial time (PPT) adversary $\mathcal{ADV}$. We assume that
$\mathcal{ADV}$ has the following capabilities:

  * •

$\mathcal{ADV}$ overhears the communication between the Client and the Server;

  * •

$\mathcal{ADV}$ is allowed to tamper with any message she sees, either by
changing the contents of the message, or by replacing it with another one.

In our threat model, we assume that $\mathcal{ADV}$ does not block the
communication between the Client and the Server.

More formally, we will prove the following proposition:

######  Proposition 1 (Protocol Soundness)

Let $\mathsf{PKE}$ be a CCA2-secure public-key encryption scheme and
$\mathsf{Sign}$ an EUF-CMA-secure signature scheme. Moreover, let
$\mathcal{ADV}$ be a PPT adversary. Then $\mathcal{ADV}$:

  1. 1.

Can not infer any information from the exchanged messages except from the time
of sending;

  2. 2.

Can not tamper with the content of any message in a way that goes unnoticed.

###### Proof

We examine each assumptions separately, and we will prove that they both hold
with overwhelming probability.

  1. A1:

Our first assumption is that $\mathcal{ADV}$ can not infer any information
from exchanged messages. Assuming that $\mathcal{ADV}$ does not collude with
neither Client nor Server, the only way to infer information about messages is
to successfully decrypt messages encrypted under $\mathsf{PKE}$ or
$\mathsf{HE}$. However, assuming security of both of those schemes, this can
only happen with negligible probability in the security parameters $\lambda$
and $\kappa$ of $\mathsf{PKE}$ and $\mathsf{HE}$ respectively. Hence, if we
denote advantage of $\mathcal{ADV}$ in decrypting exchanged ciphertexts by
$\epsilon$, we get: $\epsilon=negl(\lambda)+negl(\kappa)$. So finite sum of
negligible functions is still negligible, $\mathcal{ADV}$ can decrypt messages
with negligible probability and, our assumption holds with overwhelming
probability.

  2. A2:

$\mathcal{ADV}$ can try tampering with the exchanged messages in two possible
ways:

     * •

Generate and send her own messages in place of the actual messages;

     * •

Replay old messages.

Generating her own valid ciphertexts is trivial as every ciphertext is
encrypted under a public key. Moreover, $\mathcal{ADV}$ would also need to
forge a valid signature of the sender should she wish to create a malicious
message that is indistinguishable from a real one. However, given the EUF-CMA
security of the signature scheme $\mathsf{Sign}$, this can only happen with
negligible probability in the security parameter $\mu$ of $\mathsf{Sign}$.

Thus, the only alternative for $\mathcal{ADV}$ is to replay and old message
that was transmitted at some time in the past $t^{\prime}$. However, since,
each exchange message contains the current timestamp both in the first
component of the message and in the signed hash, $\mathcal{ADV}$ would once
again need to forge a valid signature of the sender since, otherwise, the
verification of the signature would pass, though the verification of the
timestamp would fail. Hence, if we denote by $\epsilon_{2}$ the advantage of
$\mathcal{ADV}$ in forging a valid signature, we conclude that the overall
advantage of $\mathcal{ADV}$ in tampering with the content of any message in
an indistinguishable way is:

$\scriptsize\epsilon_{2}=negl(\mu)+negl(\mu)=negl(\mu)$ (5)

Hence, our second assumption holds with overwhelming probability.

##  7 Performance Analysis

In this work, we evaluate our method on two ECG datasets: the MIT-BIH dataset
[16] and the PTB-XL dataset [21]. 222Due to limited space, the figures from
this section are moved to Appendix 0.B.

###  7.1 Experimental Setup

All models are trained on a machine with Ubuntu 20.04 LTS, processor Intel
Core i7-8700 CPU at 3.20GHz, 32Gb RAM, GPU GeForce GTX 1070 Ti with 8Gb of
memory. We write our program in the Python version 3.9.7. The NNs are
constructed using the PyTorch library version 1.8.1+cu102. For HE algorithms,
we employ the TenSeal library version 0.3.10.

In terms of hyperparameters, we train all networks with 10 epochs, a
$\eta=0.001$ learning rate, and a $n=4$ training n. For split NN with HE AMs,
we use the Adam optimizer for client model and mini-batch Gradient Descent for
server model. We use GPU for networks trained on plaintext. For U-shaped SL on
HE AMs, we train the client model on GPU, and server model on CPU.

###  7.2 Evaluation

In this section, we report the experimental results in terms of accuracy,
training duration and communication throughput. We measure the accuracy of the
three NN on the plaintext test set after the training processes are completed.

#### Networks with Different Activation Map Sizes:

The 1D CNN models used on both MIT-BIH and PTB-XL datasets have two Conv1D
layers and one linear layer. The AMs are the output of the last Conv1D layer.

For the MIT-BIH dataset, we experiment with two sizes of AMs:
$[\text{\textit{n}},512]$ (as in [1]) and $[\text{\textit{n}},256]$ [10]. We
get the AMs of size $[\textit{n},256]$ by reducing the number of output
channels in the second Conv1D layer by half (from 16 output channels to 8
output channels). We denote the 1D CNN model with an AM sized
$[\textit{n},256]$ as $M_{1}$, and the model with an AM sized
$[\textit{n},512]$ as $M_{2}$.

For the PTB-XL dataset, we change the number of the input channels for the
first Conv1D layer to 12, since the input data are 12-lead ECG signals [11].
Besides, we only experiment with 8 output channels for the second Conv1D
layer. We denote this network by $M_{3}$. Using $M_{3}$, the output AM size is
$[\textit{n},2000]$.

#### Training Locally:

The result when training the model $M_{1}$ locally on the MIT-BIH plaintext
dataset is shown in Figure 6. The NN learns quickly and is able to decrease
the loss drastically from epoch 1 to 5. After that, from epoch 6-10, the loss
begins to plateau. After training for 10 epochs, we test the trained NN on the
test dataset and get 88.06% accuracy. Training the model locally on plaintext
takes 4.8 seconds for each epoch on average.

Figure 6 shows the results when training the model $M_{2}$ on plaintext MIT-
BIH. After 10 epochs, the model achieves the best training accuracy of 91.66%.
the trained model results in 92.84% prediction accuracy. Each training epoch
takes 4.8 seconds on average-the same as model $M_{1}$. Even though the two
models differ in the output sizes of the AMs, they are relatively small
models: the number of parameters is 2061 for $M_{1}$, and 3989 for $M_{2}$. As
both models can fit in the GPU memory, the local training duration becomes
similar for both models.

Training $M_{3}$ locally on the plaintext PTB-XL dataset results in Figure 6
achieves a training accuracy of 72.65% after 10 epochs with a test accuracy is
67.68%. This low accuracy is due to small NN with 12013 trainable parameters
and limited training epochs (each takes 10.56 seconds on average).

###  7.3 U-shaped Split Learning using Plaintext Activation Maps

Our experiments, show that training the U-shaped split model on plaintext
(reported in section 3.2) produces the same results in terms of accuracy
compared to local training for both models $M_{1}$, $M_{2}$ and $M_{3}$. This
result is similar to the findings of [1]. Even though the authors of [1] only
used the vanilla split model, they also found that compared to training
locally, accuracy was not reduced.

We will now discuss the training time and communication overhead of the
U-shaped split models and compare them to their local versions. For the split
version of $M_{1}$, each training epoch takes 8.56 seconds on average, hence
43.9% longer than local training. Training the split version of $M_{2}$ takes
8.67 seconds per epoch on average, which is 44.6% longer compared to the 4.8
seconds of local training. The split version of $M_{3}$ on the PTB-XL dataset
takes 15.55 seconds per epoch to train and is 47.25% slower than the local
version. The U-shaped split models take longer to train due to the
communication between the client and the server. The communication cost for
one epoch of training split $M_{1}$ and $M_{2}$ are 33.06 Mb and 60.12 Mb,
respectively. $M_{2}$ incurs almost twice as much communication overhead
compared to $M_{1}$ due to the bigger size of the AMs. For $M_{3}$,
communication overhead is on average 316.9 Mb per epoch, which is much bigger
than $M_{1}$ and $M_{2}$ due to bigger AMs sent from client during training.

These figures show that the similarities are not as strong compared to the
MIT-BIH AMs. This is due to the fact that $M_{3}$ is a small network trained
on a limited number of epochs (72.65% training accuracy after 10 epochs),
therefore, the convolution layers are not yet able to produce highly similar
AMs compared to the original signals. Still the figures indicate similar
patterns between the AMs and the original PTB-XL signals. Nonetheless, to
reach any conclusions, we first need to experiment with better NNs that can
produce highly accurate predictions on both the train and test splits of the
PTB-XL dataset, then quantify similarities between AMs and input signals of
these networks.

###  7.4 Visual Invertibility

In SL, certain AMs sent from client to the server to continue the training
process show high similarity with the client’s input data, as demonstrated in
Figure 8 for models trained on the MIT-BIH dataset. The figure indicates that,
compared to the raw input data from the client (the first row of Figure 8),
some AMs (as plotted in the second and third row of Figure 8) have exceedingly
similar patterns. This phenomenon clearly compromises the privacy of the
client’s raw data. The authors of [1] quantify the privacy leakage by
measuring the correlations between the AMs and the raw input signal by using
two metrics: distance correlation and Dynamic Time Warping. This approach
allows them to measure whether their solutions mitigate privacy leakage work.
Since our work uses HE, said metrics are unnecessary as the AMs are encrypted.
Similar to MIT-BIH, we visualize the output AMs produced by $M_{3}$ to access
their visual similarity compared to original signals. Due to space
constraints, we visualize results only for normal class (see Figure 8) instead
of five different classes of heartbeat in PTB-XL dataset.

###  7.5 U-shaped Split 1D CNN with Encrypted Activation Maps

We train the split NNs $M_{1}$ and $M_{2}$ on the MIT-BIH dataset using EAMs
according to 4.2. To encrypt the AMs from the client before sending them to
the server, we experiment with five different sets of HE parameters for both
models $M_{1}$ and $M_{2}$. Furthermore, we also employ the batch encryption
(BE) feature of the CKKS encryption scheme. BE allows us to encrypt a $N\times
N$ matrix into $N$ ciphertexts, with each column encrypted as a ciphertext for
memory and computation optimization [2]. We experiment with training the NNs
with and without BE. Additionally, we perform experiments using different
combinations of HE parameters. Table 2 shows the results in terms of training
time, testing accuracy, and communication overhead for the NNs with different
configurations on the MIT-BIH dataset. For the U-shaped SL version on the
plaintext, we captured all communication between client and server. For
training split models on EAMs, we approximate the communication overhead for
one training epoch by getting the average communication of training on first
ten batches of data, multiply that with total number of training batches.

Results differ between training $M_{1}$ and $M_{2}$ with different sets of HE
parameters. For the $M_{1}$ model, the best test accuracy was 85.41%, when
using the set of HE parameters with $\mathcal{P}=4096$,
$\mathcal{C}=[40,20,20]$, $\Delta=2^{21}$ (denoted $s_{1}$), and without BE.
The accuracy drop was $2.65\%$ compared to plaintext training.

Table 2: Training and testing results on the MIT-BIH dataset

Network | Type of Network | HE Parameters | Training duration per epoch (s) | Test accuracy (%) | Communication per epoch (Tb)
---|---|---|---|---|---
BE | $\mathcal{P}$ | $\mathcal{C}$ | $\Delta$
$M_{1}$ | Local |  | 4.80 | 88.06 | 0
Split (plaintext) |  | 8.56 | 88.06 | 33.06e-6
Split (HE) | False | 8192 | [60,40,40,60] | $2^{40}$ | 50 318 | 85.31 | 37.84
8192 | [40,21,21,40] | $2^{21}$ | 48 946 | 80.63 | 22.42
4096 | [40,20,20] | $2^{21}$ | 14 946 | 85.41 | 4.49
4096 | [40,20,40] | $2^{20}$ | 18 129 | 80.78 | 4.57
2048 | [18,18,18] | $2^{16}$ | 5 018 | 22.65 | 0.58
True | 8192 | [60,40,40,60] | $2^{40}$ | 33 310 | 81.22 | 4.77
8192 | [40,21,21,40] | $2^{21}$ | 31 311 | 84.36 | 2.81
4096 | [40,20,20] | $2^{21}$ | 11 507 | 79.00 | 0.67
4096 | [40,20,40] | $2^{20}$ | 11 656 | 80.79 | 0.69
2048 | [18,18,18] | $2^{16}$ | 3 869 | 70.12 | 0.16
$M_{2}$ | Local |  | 4.80 | 92.84 | 0
Split (plaintext) |  | 8.67 | 92.84 | 60.12e-6
Split (HE) | False | 8192 | [60,40,40,60] | $2^{40}$ | 118 518 | 81.40 | 238.71
8192 | [40,21,21,40] | $2^{21}$ | n/a | n/a | n/a
4096 | [40,20,20] | $2^{21}$ | 31 711 | 81.38 | 12.86
4096 | [40,20,40] | $2^{20}$ | 31 791 | 80.12 | 14.60
2048 | [18,18,18] | $2^{16}$ | 12 087 | 22.65 | 1.786
True | 8192 | [60,40,40,60] | $2^{40}$ | 79 637 | 81.46 | 17.57
8192 | [40,21,21,40] | $2^{21}$ | 56 356 | 84.46 | 9.25
4096 | [40,20,20] | $2^{21}$ | 20 790 | 84.69 | 1.82
4096 | [40,20,40] | $2^{20}$ | 20 521 | 81.65 | 1.82
2048 | [18,18,18] | $2^{16}$ | 8 113 | 73.82 | 0.36

However, with BE, $s_{1}$ produced only $79\%$ accuracy. Compared to the
bigger sets of parameters with $\mathcal{P}=8192$, $s_{1}$ achieves higher
accuracy while requiring much lower training time and communication overhead.
The result when using the first set of parameters with $\mathcal{P}=8192$ is
close ($85.31\%$), but with a much longer training time (3.67 times longer)
and communication overhead (8.43 times higher). We observe that in some cases,
training with BE results in better testing accuracy, while in some other, it
leads to accuracy reduction. On the other hand, training on EAMs with BE is
23-35% faster. The amount of communication overhead is also significantly
reduced (up to 13 times). Interestingly, using the HE parameters with
$\mathcal{P}=2048$ and without BE drastically reduces accuracy to $22.65\%$,
however, with BE, it only reduces accuracy to $70.12\%$. Hence, the effect of
BE in NN training needs further study.

Although $M_{2}$ achieves better accuracy than $M_{1}$ on plaintext data, it
does not provide better results in encrypted version. The best accuracy of
$M_{2}$ on encrypted version is $84.69\%$ using $s_{1}$ and with BE. Compared
to the configuration that achieves best results for $M_{1}$ at 85.41%, best
configuration for $M_{2}$ takes 1.39 times more to train but incurs less
communication overhead (about 2.5 times less). This is because the best
configuration for $M_{2}$ occurs when BE is used, and best configuration for
$M_{1}$ occurs when BE is not used. In general, training $M_{2}$ with EAMs
takes 2-3 times longer to train and 3-6 times more communication overhead
compared to $M_{1}$ with the same HE configuration.

Table 3: Training and testing results on the PTB-XL dataset

Network | Type of Network | HE Parameters | Training duration per epoch (s) | Test accuracy (%) | Communication per epoch (Tb)
---|---|---|---|---|---
BE | $\mathcal{P}$ | $\mathcal{C}$ | $\Delta$
$M_{3}$ | Local |  | 10.56 | 67.68 | 0
Split (plaintext) |  | 15.55 | 67.68 | 316.9e-6
Split (HE) | True | 8192 | [40,21,21,40] | $2^{21}$ | 72 534 | 65.42 | 115.64
4096 | [40,20,20] | $2^{21}$ | 24 061 | 64.22 | 18.20
4096 | [40,20,40] | $2^{20}$ | 22 570 | 65.23 | 18.77
| 2048 | [18,18,18] | $2^{16}$ | 7 605 | 65.33 | 1.93

The results of training different settings of $M_{3}$ on the PTB-XL dataset
are reported in Table 3. We only train the split version of $M_{3}$ with an
EAM using BE. The HE set of parameters $\mathcal{P}=8192$,
$\mathcal{C}=[40,21,21,40]$, $\Delta=2^{21}$ achieves the best test accuracy
at 65.42%. This result is only 2.26% lower than the result obtained by the
plaintext version. However, this set of parameters incurs the most
communication overhead (115.64 Tb per epoch) and takes the longest to train
(72 534 seconds per epoch). Overall, test accuracies achieved by different HE
parameters are quite close to each other, with 65.42% being the lowest.
Interestingly, the smallest set of HE parameters
$\mathcal{P}=2048,\mathcal{C}=[18,18,18],\Delta=2^{16}$ achieves second-best
accuracy at 65.33%, requiring about 1/10 of the training duration and 1/100 of
the communication overhead compared to $\mathcal{P}=8192$. The two sets of
parameters with $\mathcal{P}=4096$ produce quite similar results, roughly
taking same amount of time and communication overhead to train.

Through our experiments, we see that training on EAMs can produce very
optimistic results, with accuracy dropping by 2-3% for the best sets of HE
parameters. Furthermore, training using BE can significantly reduce the amount
of training time and communication overhead needed, while producing comparable
results when it come to training without BE. The set of parameters with
$\mathcal{P}=8192$ always achieve the highest test accuracy, though incurring
the highest communication overhead and the longest training time. The set of
parameters with $\mathcal{P}=4096$ can offer a good trade-off as they can
produce on-par accuracy with $\mathcal{P}=8192$, while requiring significantly
less communication and training time. Experimental results show that with the
smallest set of HE parameters $\mathcal{P}=2048$, $\mathcal{C}=[18,18,18]$,
$\Delta=2^{16}$, the least amount of communication and training time is
required. In addition, this only works well when used together with BE. When
training the network $M_{3}$ on the PTB-XL dataset, this set of parameters
produces even better test accuracy compared to $\mathcal{P}=4096$. However,
this result may be because the network $M_{3}$ is small. The test accuracy on
the plaintext version is $67.68\%$, hence the noises produced by the HE
algorithm do not yet have a significant role in reducing the model’s accuracy.

###### Remark 1

As can be seen in Table 2, the accuracy of the same algorithm varies greatly
under different CKKS parameters. The parameter selection in CKKS is not
evident as a set of the parameter may result in efficient computation for one
application but also in poor performance for another application. In addition,
CKKS uses approximate arithmetic rather than exact arithmetic, in the sense
that once computation is finished the result may slightly differ compared to
that of a direct computation [5]. Hence, it is still open to research whether
for specific applications, a closed form of relation for the set of parameters
can be used to measure the accuracy. However, training both models $M_{1}$ and
$M_{2}$ with a different set of parameters, we observe:

  * •

Training model with BE yields better outcomes than without BE. This pattern
can be seen in $M_{2}$ and also in $M_{1}$ with two exceptions
($\mathcal{P}=8192,\mathcal{C}=[60,40,40,60],\Delta=2^{40}$) and
($\mathcal{P}=4096,\mathcal{C}=[40,20,20],\Delta=2^{21}$).

  * •

Training without BE, higher contexts yield better results than lower. This
pattern can be seen in $M_{2}$ and also in $M_{1}$ with two exceptions
$(\mathcal{P}=4096,\mathcal{C}=[40,20,20],\Delta=2^{21})$ and
$(\mathcal{P}=4096,\mathcal{C}=[40,20,40],\Delta=2^{20})$.

  * •

Also in both models $M_{1}$ and $M_{2}$, BE is suitable for lower context
$(\mathcal{P}=2048,\mathcal{C}=[18,18,18],\Delta=2^{16})$.

Open Science and Reproducible Research: To support open science and
reproducible research, and provide researchers with opportunity to use, test,
and extend our work, source code used for the evaluations is publicly
available333https://github.com/khoaguin/HESplitNet.

##  8 Conclusion

In this paper, we focused on training ML models in a privacy-preserving way.
We used the concept of SL in combination with HE and constructed protocols
allowing a client to train a model in collaboration with a server without
sharing valuable information about the raw data. To the best of our knowledge,
this is the first work that uses SL on encrypted data. Our experiments show
that our approach has achieved high accuracy, especially when compared with
less secure approaches that combine SL with differential privacy. The
limitation of our work is having only one client in the protocol. While
extending the protocol to multiple clients is an important task, it requires
us to rely on a multi-key HE scheme, which is beyond scope of this work and
remains to be addressed in future works.

## References

  * [1] Abuadbba, S., Kim, K., Kim, M., Thapa, C., Camtepe, S.A., Gao, Y., Kim, H., Nepal, S.: Can we use split learning on 1d cnn models for privacy preserving training? In: Proceedings of the 15th ACM Asia Conference on Computer and Communications Security. pp. 305–318 (2020)
  * [2] Benaissa, A., Retiat, B., Cebere, B., Belfedhal, A.E.: Tenseal: A library for encrypted tensor operations using homomorphic encryption. In: Workshop on Distributed and Private Machine Learning. ICLR (2021)
  * [3] Cabrero-Holgueras, J., Pastrana, S.: Sok: Privacy-preserving computation techniques for deep learning. Proceedings on Privacy Enhancing Technologies 2021(4), 139–162 (2021)
  * [4] Cheon, J.H., Kim, A., Kim, M., Song, Y.: Homomorphic encryption for arithmetic of approximate numbers. In: International Conference on the Theory and Application of Cryptology and Information Security. pp. 409–437. Springer (2017)
  * [5] Clet, P.E., Stan, O., Zuber, M.: Bfv, ckks, tfhe: Which one is the best for a secure neural network evaluation in the cloud? In: International Conference on Applied Cryptography and Network Security. pp. 279–300. Springer (2021)
  * [6] Gentry, C.: Fully homomorphic encryption using ideal lattices. In: Proceedings of the forty-first annual ACM symposium on Theory of computing. pp. 169–178 (2009)
  * [7] Gupta, O., Raskar, R.: Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications 116, 1–8 (2018)
  * [8] Hitaj, B., Ateniese, G., Perez-Cruz, F.: Deep models under the gan: information leakage from collaborative deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. pp. 603–618 (2017)
  * [9] Khan, T., Bakas, A., Michalas, A.: Blind faith: Privacy-preserving machine learning using function approximation. In: 2021 IEEE Symposium on Computers and Communications (ISCC). pp. 1–7. IEEE (2021)
  * [10] Khan, T., Nguyen, K., Michalas, A.: Split ways: Privacy-preserving training of encrypted data using split learning. In: Fletcher, G., Kantere, V. (eds.) Proceedings of the Workshops of the EDBT/ICDT 2023 Joint Conference, Ioannina, Greece, March, 28, 2023. CEUR Workshop Proceedings, vol. 3379. CEUR-WS.org (2023), https://ceur-ws.org/Vol-3379/HeDAI_2023_paper402.pdf
  * [11] Khan, T., Nguyen, K., Michalas, A., Bakas, A.: Love or hate? share or split? privacy-preserving training using split learning and homomorphic encryption. In: The 20th Annual International Conference on Privacy, Security & Trust (PST2023) 21-23 August, 2023, Copenhagen, Denmark (2023)
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR). ICLR (2015)
  * [13] Li, D., Zhang, J., Zhang, Q., Wei, X.: Classification of ecg signals based on 1d convolution neural network. In: 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom). pp. 1–6. IEEE (2017)
  * [14] Maas, A.L., Hannun, A.Y., Ng, A.Y., et al.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 30th International Conference on Machine Learning,. vol. 28, p. 3. Citeseer (2013)
  * [15] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)
  * [16] Moody, G.B., Mark, R.G.: The impact of the mit-bih arrhythmia database. IEEE Engineering in Medicine and Biology Magazine 20(3), 45–50 (2001)
  * [17] Paillier, P.: Public-key cryptosystems based on composite degree residuosity classes. In: International conference on the theory and applications of cryptographic techniques. pp. 223–238. Springer (1999)
  * [18] Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. Artificial Neural Networks - ICANN 2010 - 20th International Conference pp. 92–101 (01 2010). https://doi.org/10.1007/978-3-642-15825-4_10
  * [19] Vepakomma, P., Gupta, O., Dubey, A., Raskar, R.: Reducing leakage in distributed deep learning for sensitive health data. In: AI for Social Good Workshop. ICLR (2019)
  * [20] Vepakomma, P., Gupta, O., Swedish, T., Raskar, R.: Split learning for health: Distributed deep learning without sharing raw patient data. In: AI for social good workshop. ICLR (2019)
  * [21] Wagner, P., Strodthoff, N., Bousseljot, R.D., Kreiseler, D., Lunze, F.I., Samek, W., Schaeffter, T.: Ptb-xl, a large publicly available electrocardiography dataset. Scientific data 7(1), 1–15 (2020)
  * [22] Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T., Yu, H.: Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 13(3), 1–207 (2019)

##  Appendix 0.A Preliminaries

###  0.A.1 Convolutional Neural Network

In this work, we employ a 1D CNN [13, 1] as a feature detector and classifier
for two ECG heartbeat datasets, namely MIT-BIH [16] and PTB-XL [21]. The
employed 1D CNN has the following stacked layers:

  * •

Conv1D: Is used to swipe a kernel of adjustable weights over a 1D input
signal. The Conv1D outputs the AMs capturing feature information from said
input signal. Figure 3 visualizes the Conv1D operation and shows the
difference between a Conv1D and a Conv2D layer.

  * •

Leaky ReLU [14]: It is a non-linear function that can be described as
$f(x)=x$, if ${x\geq 0}$, and $f(x)=\alpha x$ if $x<0$, where $\alpha$ is a
small number, such as 0.01.

  * •

Max Pooling: In CNN, Max Pooling compresses the input, focusing on important
elements and allowing slight input changes with minimal impact on the pooled
version [18].

  * •

Fully Connected (FC): The FC layer has one output unit connected to all input
units, unlike the convolution layer, where one output unit in an AM only
connects to a small area in the input signal.

  * •

Softmax: A Softmax activation function takes a vector of $k$ real numbers,
e.g. $\mathbf{z}=(z_{1},\ldots,z_{k})\mathbb{R}^{k}$, and outputs a
probability distribution consisting of $k$ probabilities. Each probability in
the output vector is calculated as follows:

$\sigma(\mathbf{z})_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{k}e^{z_{j}}}.$ (6)

We use the 1D CNN as a supervised learning method, where both the input data
and corresponding labels are needed to train our network.

###  0.A.2 Homomorphic Encryption

HE is an emerging cryptographic technique for computations on encrypted data.
HE schemes are divided into three main categories according to their
functionality: Partial HE [17], leveled (or somewhat) HE [4], and fully HE
[6]. Each scheme has its own benefits and disadvantages. In this work, we use
the CKKS Leveled HE scheme [4]. CKKS allows users to do additions and a
limited number of multiplications on vectors of complex values (and hence,
real values too). Prior to the encryption, CKKS encodes a message
$\mathbf{z}\in\mathbb{C}^{N/2}$ to a ring of polynomials over the integers
$\mathbb{Z}[X]/\left(X^{N}+1\right)$. Working with polynomials over rings of
integers is a good trade-off between security and efficiency compared to
standard computations on vectors. During encoding, the vector $\mathbf{z}$ is
multiplied by a scaling factor $\Delta$ to keep a level of precision. The
encoded message is encrypted, and the resulted ciphertext is an element
$c\in\left(\mathbb{Z}_{q}[X]/\left(X^{N}+1\right)\right)^{2}$. Ciphertexts can
then be added or multiplied together. An issue arises during multiplication,
is that the term $\Delta^{2}$ appears in the ciphertext result. To address
this, CKKS deploys a rescaling operation to keep the scaling factor $\Delta$
constant. The inverse procedure needs to be followed for the decryption; that
is, the ciphertext will be decrypted first, and then the encoded message will
be decoded and multiplied by $1/\Delta$ to recover
$\mathbf{z}^{\prime}\in\mathbb{C}^{N/2}$. The number of allowed
multiplications is predefined by a list of prime numbers. To build this list,
the authors first choose $\left(p_{1},\dots,p_{L},q_{0}\right)$ primes, where
each $p_{\ell}\approx\Delta$ and $q_{0}>\Delta$. Finally, they set
$q_{L}=\prod_{1}^{L}p_{l}\cdot q_{0}$, where $q_{L}=q$ – the order of
$\mathbb{Z}_{q}$ – and the list is $(q_{L},\dots,q_{0})$. After each
multiplication, an element is deleted from the list. However, according to
[4], the security of CCKS is based on the ratio $N/q$. Hence, to maintain the
same level of security as we increase $q$, we also need to increase $N$-the
degree of the polynomials and hence, computational costs. Summing up, the most
important parameters of the CKKS scheme are:

  1. bx1.

Polynomial Modulus $\mathcal{P}$: Naturally, this parameter has a direct
impact on the scheme’s efficiency and security. According to [4], this value
needs to be a power of two. Common values include 2048, 4096, 8192, 16384 and
32768.

  2. bx2.

Coefficient Modulus $\mathcal{C}$: A list of primes that define current
scheme’s level. After each multiplication a different prime is used as
coefficient modulus. Hence, no more multiplications are allowed when all
primes are used.

  3. bx3.

Scaling Factor $\Delta$: This is a constant positive number multiplied by the
plaintext message during encoding to maintain a certain level of precision.

1D Convolutional1D ConvolutionalLeaky ReluLeaky ReluMax PoolingMax
PoolingClient-sideServer-sideSoftmaxOutputTraining Input DataFully Connected

Figure 1: U-shaped SL

1D Conv layer2D Conv layerA pixel in an image1D Kernel FeatureDetectorA value
in the ECG signal vector2D Kernel FeatureDetector

Figure 2: Conv1D vs Conv2D

AClient [9]BServer A$m_{1}=\langle
t_{1},\mathsf{PKE.Enc}(\mathsf{pk_{S}},\mathsf{evk_{HE}}),\mathsf{HE.Enc}(\mathsf{pk_{HE}},\mathrm{AMap}),\sigma_{C}(H_{1})\rangle$B

B$\mathsf{HE.Eval}(\mathsf{pk}_{\mathsf{HE}},\mathrm{AMap})$$\mathsf{HE.Enc(pk_{HE},out)}$
B$m_{2}=\langle t_{2},\mathsf{HE.Enc(pk_{HE},out)},\sigma_{S}(H_{2})\rangle$A

AUse $\mathsf{out}$ to compute gradients$\mathsf{grad}$ A$m_{3}=\langle
t_{3},\mathsf{PKE.Enc}(\mathsf{pk}_{C},\mathsf{grad}),\sigma_{S}(H_{3})\rangle$B
BUpdate parameters$\mathsf{grad^{\prime}}$ B$m_{4}=\langle
t_{4},\mathsf{PKE.Enc(pk_{C},grad^{\prime}}),\sigma_{C}(H_{4})\rangle$A

Figure 3: Running Phase

##  Appendix 0.B Datasets

#### MIT-BIH:

We use the pre-processed dataset from [1], which is based on the MIT-BIH
arrhythmia database [16]. The processed dataset contains 26,490 samples of
heartbeat that belong to 5 different types: N (normal beat), left (L) and
right (R) bundle branch block, atrial (A) and ventricular (V) premature
contraction. To train our network, the dataset is split into a train and test
split as matrices of size $[13245,1,128]$, meaning that each contain 13,245
ECG samples and, each sample has one channel and 128 timesteps [1].

#### PTB-XL:

According to [21], PTB-XL is the largest open-source ECG dataset since 2020.
The dataset contains 12-lead ECG-waveforms from 21837 records of 18885
patients. Compared to PTB-XL, MIT-BIH only contains 2-lead ECG-waveforms
obtained from 47 patients. Each waveform from PTB-XL has a duration of 10
seconds. Two sampling rates are used to collect the data: 100 Hz and 500 Hz.
In our experiment, we employ the 100 Hz waveforms. Each 12-lead ECG waveform
is associated with one or several classes out of five classes: normal (NORM),
conduction disturbance (CD), myocardial infarction (MI), hypertrophy (HYP),
and ST/T change (STTC). For waveforms that belong to multiple classes, we
choose only the first one and remove the others for simplicity. The dataset is
then split into a 90%-10% train-test ratio. In total, we have a training split
of size $[19267,12,1000]$, with 19,267 ECG waveform samples, of 12 channels
(or leads) and 1,000 timesteps each. The test split’s size is
$[2163,12,1000]$.

Figure 4:  MIT-BIH $[\textit{n},256]$

Figure 5: MIT-BIH $[\textit{n},512]$

Figure 6: PTB-XL $[\textit{n},512]$

Figure 7: Top: client input data. Middle: output channels $M_{1}$. Bottom:
output channels $M_{2}$.

Figure 8: Visual invertibility of the model $M_{3}$ on the PTB-XL dataset.
Left: input data (NORM class). Right: corresponding activation maps.