# I can’t see it but I can Fine-tune it: On Encrypted Fine-tuning of
Transformers using Fully Homomorphic Encryption 111Accepted for the
presentation at PPAI@The 38th Annual AAAI Conference on Artificial
Intelligence 2024

Prajwal Panzade 1, Daniel Takabi 2, Zhipeng Cai 1

###### Abstract

In today’s machine learning landscape, fine-tuning pre-trained transformer
models has emerged as an essential technique, particularly in scenarios where
access to task-aligned training data is limited. However, challenges surface
when data sharing encounters obstacles due to stringent privacy regulations or
user apprehension regarding personal information disclosure. Earlier works
based on secure multiparty computation (SMC) and fully homomorphic encryption
(FHE) for privacy-preserving machine learning (PPML) focused more on privacy-
preserving inference than privacy-preserving training. In response, we
introduce BlindTuner, a privacy-preserving fine-tuning system that enables
transformer training exclusively on homomorphically encrypted data for image
classification. Our extensive experimentation validates BlindTuner’s
effectiveness by demonstrating comparable accuracy to non-encrypted models.
Notably, our findings highlight a substantial speed enhancement of 1.5$\times$
to 600$\times$ over previous work in this domain.

## Introduction

Transformers have become the state-of-the-art neural architecture in fields
like natural language processing and computer vision, owing to their reliance
on attention mechanisms and large-scale pre-training (Vaswani et al., 2017;
Devlin et al., 2018; Touvron et al., 2021). With the growth of cloud
computing, many machine learning as a service (MLaaS) platforms now provide
pre-trained transformer fine-tuning capabilities. However, handling sensitive
data on these platforms necessitates adherence to privacy laws like GDPR
(General Data Protection Regulation) and CCPA (California Consumer Privacy
Act) (Ribeiro et al., 2015). In a standard MLaaS system, the client own the
data while ML computation occurs on the cloud (Liu et al., 2021). When data
contains private information like healthcare or banking records, significant
privacy risks arise. Consider a hospital using an MLaaS cancer prediction
service: while patients may consent to data access, externally sharing
sensitive information for modeling can enable breaches and unauthorized
access, violating privacy. Moreover, if patients revoke their consent, model
development becomes difficult.

Many approaches to privacy-preserving machine learning using secure multiparty
computation exist, including SecureML (Mohassel et al., 2018), SecureNN (Wagh
et al., 2019), and DeepSecure (Rouhani et al., 2018). However, these
techniques require substantial client-server communication (Wagh et al.,
2019). For lower communication, fully homomorphic encryption techniques like
CryptoNets (Gilad-Bachrach et al., 2016), CryptoDL (Hesamifard et al., 2018),
and ML Confidential (ML Confidential, 2021) are preferred, though more
research has focused on private inference over training (Reagen et al., 2021;
Gilad-Bachrach et al., 2016). Recent work has proposed encrypted neural
network training using FHE (Nandakumar et al., 2019). Moreover, privacy-
preserving transfer learning methodologies are developed in Glyph (Lou et al.,
2020) and HETAL (Lee et al., 2023). However, the existing methodologies used
for privacy-preserving transfer learning can be further refined, specifically
aiming to improve accuracy and reduce training duration. This paper focuses on
addressing these refinements.

The contributions of this paper are as follows:

  * •

We introduce BlindTuner, an optimized system designed for training transformer
models using FHE-encrypted data, facilitating privacy-preserving fine-tuning
within cloud environments.

  * •

In an effort to develop privacy-preserving strategies for model training, our
investigation delves into the complex relationship between FHE and data-
efficient image transformers (DEiT).

  * •

By conducting thorough experimentation and rigorous evaluation on a wide range
of datasets (including MNIST, CIFAR-10, DermaMNIST, and face mask detection),
we propose a methodology that guarantees accessibility without precluding the
need for specialized knowledge in cryptography. We’ll provide an open-source
implementation of BlindTuner at https://github.com/prajwalpanzade.

  * •

Significantly, our implementation exhibits superior performance compared to
Glyph (NeurIPS 2020), progressing by 300 times in the classification of MNIST
digits and 600 times in the classification of DermaMNIST images.

## Background Knowledge

### Fully Homomorphic Encryption

Fully Homomorphic Encryption (FHE) stands as an advanced cryptographic
technique, pioneered by Gentry (Gentry 2009), enabling computations to be
executed on encrypted data without necessitating decryption. In essence, it
permits a third party to perform computations on encrypted data without
exposure to the data itself or the resultant calculations. The implications of
FHE are profound for privacy and security across diverse applications, notably
in cloud computing and data outsourcing scenarios.

FHE encompasses several variations, including CKKS (Cheon-Kim-Kim-Song) (Cheon
et al. 2017), Brakerski-Gentry-Vaikuntanathan (BGV) (Brakerski, Gentry, and
Vaikuntanathan 2014), and the Brakerski/Fan-Vercauteren scheme (BFV). Among
these, CKKS gains popularity owing to its adeptness in handling real numbers.
Analogous to public key encryption (PKE), the CKKS scheme encompasses
encryption, decryption, and key generation algorithms. However, distinct from
PKE, CKKS integrates homomorphic addition and multiplication functionalities,
enabling addition or multiplication of two ciphertexts.

A succinct overview of these algorithms is as follows:

  * •

KeyGen(1λ): Generates a public key (pk), secret key (sk), and an evaluation
key (evk).

  * •

Encpk(m): Encrypts a message (m $\in$ R) using pk, producing ciphertext c,
where R represents a set of real numbers.

  * •

Decsk(c): Outputs m for a given ciphertext c using sk.

  * •

Add(c1,c2): Outputs element-wise addition Enc(m1+m2) when provided with c1 and
c$2$.

  * •

Multevk(c1,c2): For a ciphertext pair (c1,c2) and an evk, generates element-
wise multiplication Enc(m1*m2). Notably, both addition and multiplication
yield ciphertexts, necessitating sk for decryption. ML computations rely on
multivariate polynomials, and the CKKS scheme supports bootstrapping,
facilitating the computation of multivariate polynomials of arbitrary degrees
(Cheon et al. 2018). The state-of-the-art encrypted matrix multiplication
proposed by (Lee et al. 2023) uses the basic FHE operations to optimize matrix
multiplication algorithms of the form ABT and ATB. Also, they proposed an
optimized softmax approximation algorithm for ML computation. Both of these
encrypted matrix multiplications and softmax approximations are the basis of
our approach.

For additional details regarding the CKKS scheme, we suggest referring to
(Cheon et al. 2017) and (Cheon et al. 2018) for comprehensive insights and in-
depth discussions on this encryption technique.

### Data-Efficient Image Transformers

Vision transformers (ViT) (Dosovitskiy et al. 2020) have emerged as a
promising architecture for image classification tasks. However, historically,
ViT models have necessitated extensive pre-training on large datasets to
achieve competitive performance, unlike convolutional neural networks (CNNs)
(LeCun et al. 1998). Touvron et al. introduced Data-efficient Image
Transformers (DeiT) (Touvron et al. 2021), demonstrating that transformers can
match or surpass state-of-the-art CNNs when trained exclusively on ImageNet.
The authors employed various modifications to the training approach,
incorporating extensive data augmentation techniques such as RandAugment,
CutMix, and repeated augmentation. Additionally, they introduced a novel
distillation procedure utilizing a distillation token that engages with other
embeddings through self-attention. Without distillation, DeiT achieves a top-1
accuracy of 81.8% on ImageNet, surpassing prior ViT models at 77.9%. Employing
distillation from a RegNet teacher, DeiT achieves an accuracy of 85.2%,
exceeding CNNs like EfficientNet (Touvron et al. 2021). Transfer learning to
other datasets also reveals competitive performance, underscoring the
generalization capability of vision transformers. Overall, DEiT signifies a
substantial leap in establishing transformers as a credible alternative to
CNNs in computer vision tasks. Notably, DEiT enables the training of high-
performing transformer models without relying on extensive datasets.

### Transfer Learning

Transfer Learning (TL) stands as a machine learning technique leveraging
knowledge acquired from solving one problem to address a related but distinct
problem (Pan and Yang 2009). Specifically, TL involves retraining a model
initially trained on a large, general dataset using a second, smaller dataset
(Weiss, Khoshgoftaar, and Wang 2016). The rationale behind TL lies in the
notion that lower levels of neural networks identify fundamental, universal
features such as edges, which hold relevance across various tasks.
Consequently, pre-trained models serve as valuable starting points,
necessitating less data to learn task-specific features. In computer vision,
TL extensively employs large models like VGG (Simonyan and Zisserman 2014),
ResNet (He et al. 2016), and EfficientNet (Tan and Le 2019), pre-trained on
ImageNet (Deng et al. 2009), and subsequently fine-tuned for specialized
domains like medical or aerial imaging. By capitalizing on pre-trained
features, TL achieves high accuracy using moderately sized datasets. The
process of fine-tuning a pre-trained model proves notably faster and more
data-efficient compared to training a model from scratch. Fine-tuning remains
a predominant methodology responsible for breakthrough advancements,
particularly in applications constrained by limited training data.

## The Proposed Methodhology

### Threat Model and overview

Our system, BlindTuner, depicted in Figure 1, involves two primary entities:
the client and the cloud. The client, as the data owner, transmits FHE-
encrypted data to the cloud, which is responsible for conducting machine
learning computations in an MLaaS environment. The cloud executes the ML
computations, generating an encrypted trained ML model, thus safeguarding the
client’s data privacy. Operating under an honest but curious security model
(Goldreich 2004), BlindTuner ensures compliance with the protocol while both
cloud and client actively seek maximum information. Leveraging the fine-tuning
of pre-trained models, BlindTuner assumes a shared pre-trained model for
feature extraction between the client and the cloud. Subsequent subsections
delve into the intricacies of computations carried out by both entities.

Figure 1: Overview of the proposed end-to-end BlindTuner

### BlindTuner

Algorithm 1 presents the comprehensive sequence of actions executed by both
the client and the cloud within the BlindTuner system.

Input: Training data and labels $(X_{\text{train}},Y_{\text{train}})$,
Validation data and labels $(X_{\text{val}},Y_{\text{val}})$

Output: Trained model $W_{\text{pt}}$

1

2Step 1: Feature Extraction;

3 The client requests the cloud for the DEiT model tailored for image
classification tasks;

4 Using the DEiT model, the client extracts features from
$(X_{\text{train}},Y_{\text{train}})$ and $(X_{\text{val}},Y_{\text{val}})$;

5

6Step 2: Data Encryption;

7 The client encrypts $(X_{\text{train}},Y_{\text{train}})$ as
$D_{\text{ct\\_train}}=\\{(X_{\text{ct\\_train}},Y_{\text{ct\\_train}})\\}$
and $(X_{\text{val}},Y_{\text{val}})$ as
$(X_{\text{ct\\_val}},Y_{\text{ct\\_val}})$ using the FHE public key and
transmits to the cloud;

8

9Step 3: Training Loop;

10 for  _each epoch_ do

11 a) Parameters update with NAG;

12 for  _each $(X_{\text{ct}},Y_{\text{ct}})$ in $D_{\text{ct\\_train}}$_ do

13 The cloud updates parameters using NAG:;

14
$W_{\text{ct}}^{t+1}=V_{\text{ct}}^{t}-(\alpha/N)(P_{\text{ct}}-Y_{\text{ct}})^{T}X_{\text{ct}}$;

15
$V_{\text{ct}}^{t+1}=(1-\gamma_{t})W_{\text{ct}}^{t+1}+\gamma_{t}W_{\text{ct}}^{t}$;

16

17 where $\alpha$ = learning rate, $N$= batch size,
$P_{\text{ct}}=\text{ASoftmax}(X_{\text{ct}}(V_{\text{ct}}^{t})^{\intercal})$
with ASoftmax, and $\\{\gamma_{t}\\}_{t\geq 0}$ as proposed in (Lee et al.
2023);

18 b) Compute and Send Logits;

19 The cloud computes logits
$\ell_{\text{ct\\_val}}=X_{\text{ct\\_val}}(W_{\text{ct}})^{T}$ and transmits
them to the client;

20

21 c) Decryption and Validation Loss Computation;

22 The client decrypts logits and computes the validation loss $L_{CE}$ with
$\ell_{\text{pt\\_val}}$ and $Y_{\text{pt\\_val}}$;

23

24

25Step 4: Finalization;

26 The cloud transmits $W_{\text{ct}}$ to the client, which decrypts it using
the FHE private key to obtain the plain model $W_{\text{pt}}$;

27

Algorithm 1 The BlindTuner System

#### Client

The client, as the data owner, seeks privacy-preserving model training
services from the cloud to offload computational resources. Initially, both
the client and the cloud agree upon using a DEiT pre-trained model for fine-
tuning. The client utilizes DEiT to extract features from their dataset
intended for model training. Post feature extraction, the client preprocesses
and encrypts the features using CKKS-based FHE, employing its public key,
resulting in encrypted features. These encrypted features are subsequently
transmitted to the cloud for further processing.

#### Cloud

Upon receiving the encrypted features from the client, the cloud employs these
encrypted features for fine-tuning the ML model. The fine-tuning process
integrates Nesterov’s accelerated gradient (NAG) (Nesterov 1983) and encrypted
matrix multiplication, as proposed in (Lee et al. 2023), for softmax
approximation. NAG is a renowned method facilitating faster convergence in
homomorphic encryption-based ML computations (Crockett 2020). Since all
computations occur on encrypted data, no data is exposed to the cloud. Post
fine-tuning, the trained ML model is prepared for inference. During inference,
only encrypted features are necessary, and the cloud forwards the output layer
results back to the client. The client subsequently decrypts the results using
their private key.

### Security Analysis

Here, we evaluate the security of BlindTuner. During fine-tuning, the cloud
exclusively handles encrypted client features, ensuring no data exposure, even
in an adversarial context. Regarding potential attacks on FHE cryptosystems,
these systems are regarded as quantum-secure, safeguarding original plaintext
data and computed outcomes against unauthorized alterations, even in
compromised infrastructure scenarios (Creeger 2022).

## Experimental Results

### Datasets

To showcase our experimental evaluations in image classification, we utilized
four well-known datasets.

#### MNIST

The MNIST dataset (LeCun et al. 1998) comprises 28x28 pixel grayscale images
of handwritten digits ranging from 0 to 9, often used as a standard benchmark
in computer vision research.

#### CIFAR-10

CIFAR-10 (Krizhevsky, Hinton et al. 2009) consists of 32x32 color images
categorized into ten classes, commonly employed for image classification
tasks.

#### DermaMNIST from MedMNIST

The DermaMNIST dataset (Tschandl, Rosendahl, and Kittler 2018) is a subset of
MedMNIST (Yang et al. 2023), featuring grayscale images portraying various
skin conditions segregated into seven classes.

#### Face Mask Detection

The face mask detection dataset (Larxel 2020) includes images of individuals
both with and without face masks and masks worn incorrectly, presenting a
3-class classification problem.

### Environment Setup

Our primary computational tool for FHE operations is the HEaaN library (Cheon
et al. 2017), which supports bootstrapping (Cheon et al. 2018). We utilize the
GPU library version of HEaaN obtained from the developers of
HEaaN222https://heaan.it/. For feature extraction using pre-trained
transformers on the client side, we rely on PyTorch, TorchVision, NumPy, and
transformers libraries. Our experiments are conducted on a workstation
equipped with an Intel Xeon Gold 6230 R processor, operating at a clock speed
of 2.10 GHz, and 755 GB of usable RAM, NVIDIA Tesla V100 32 GB GPU, running
the Ubuntu operating system. Specific software versions used in our
experiments are detailed in Table 1.

Table 1: Software specifications Software | Version
---|---
Ubuntu | 20.04.4
Python | 3.8.5
Transformers | 4.33.1
Numpy | 1.22.2
PyTorch | 2.0.1
Torch Vision | 0.15.2
HEaaN | 0.2.0

### Experiments

#### The Client

The client utilizes a DEiT pre-trained model to extract features from the
specified datasets as detailed in the previous section. Four benchmark
datasets—MNIST, CIFAR-10, Face Mask Detection, and DermaMNIST—are employed for
rigorous evaluation. We leverage the deit-base-distilled-patch16-224 version
of the DEiT model by Facebook research, accessible through the Transformers
library. Post feature extraction, the client partitions the dataset into
train, validation, and test sets. Before transmitting the training and
validation sets to the cloud, they are encoded and encrypted using the ML
submodule of the HEaaN library. The FHE context setting utilized is FGb,
featuring a cyclomatic ring dimension of 216, ensuring a 128-bit security
level, as described in (Cheon, Son, and Yhee 2021). Keys are generated
individually for each dataset experiment, generated at the experiment’s outset
and maintained throughout. Consistency in context settings is ensured across
all experiments. Feature extraction time in the plain domain is not factored
in, as it’s considered an offline process and doesn’t impact the encrypted
fine-tuning demonstration.

#### The Cloud

Upon receiving the encrypted training and validation sets, the cloud sets up
the ML model for encrypted training using the ML submodule of the HEaaN
library. Hyperparameter tuning begins with a batch size of 2048, a learning
rate of 1, and 10 epochs. After multiple hyperparameter modifications, optimal
configurations are determined, detailed in Table 3. Experimental results,
provided in Table 2, showcase the outcomes of experiments on various datasets.
Enc training time denotes the duration for encrypted model training, while Enc
accuracy signifies the test accuracy of the encrypted model. Similarly, Unenc
accuracy and Unenc time represent test accuracy and computation time for the
unencrypted model. Identical hyperparameters are used for both domains to
facilitate comparison. Results (Table 2) highlight the performance of
BlindTuner for both encrypted and unencrypted models, with slight disparities
observed in encrypted model performance compared to their unencrypted
counterparts. This demonstrates the effectiveness of approximation arithmetic
proposed in (Cheon et al. 2017) and (Lee et al. 2023) for accurate ML model
training, despite the cryptographic FHE computations’ time-intensive nature.
Nonetheless, the benefits of preserving user data privacy without revealing
information to the cloud outweigh the computational time. Moreover, BlindTuner
exhibits competitive performance compared to prior privacy-preserving fine-
tuning

(a) MNIST

(b) MNIST

(c) CIFAR-10

(d) CIFAR-10

(e) Face Mask Detection

(f) Face Mask Detection

(g) DermaMNIST

(h) DermaMNIST

Figure 2: BlindTuner performance on various datasets

on GPUs. A comparative analysis with earlier works is discussed in the
subsequent subsection.

### Comparison

Table 4 presents a comprehensive comparison of the training latency between
our proposed approach, BlindTuner, and

Table 2: Performance of BlindTuner Dataset | Enc training time (min) | Enc Accuracy | Unenc accuracy | Unenc time (s) | #Epochs
---|---|---|---|---|---
MNIST | 37.8390 | 97.75% | 98.02% | 44.74 | 5
CIFAR-10 | 31.3342 | 95.20% | 95.20% | 61.26 | 5
Face mask detection | 3.9086 | 97.18 % | 97.06% | 9.90 | 10
DermaMNIST | 16.2993 | 76.11% | 76.16% | 30.25 | 12
Table 3: Training parameters Dataset | #Epochs | Learning rate | Batch size
---|---|---|---
MNIST | 5 | 0.1 | 1024
CIFAR-10 | 5 | 0.1 | 1024
Face mask detection | 10 | 0.1 | 512
DermaMNIST | 12 | 0.01 | 512

two existing methodologies, Glyph (Lou et al. 2020) and HETAL (Lee et al.
2023). In the interest of a fair assessment, we conduct an evaluation of HETAL
on our workstation, while relying on the reported results for Glyph due to its
similar architecture to ours.

Analyzing the MNIST dataset results, BlindTuner achieves a commendable test
accuracy of 97.75%, nearly on par with Glyph’s 98.6%. However, BlindTuner
significantly reduces training time to a mere 37.839 minutes, a drastic
improvement compared to Glyph’s extensive 8-day requirement. This renders
BlindTuner over 300 times faster than Glyph. Compared to HETAL, BlindTuner
exhibits superior performance by enhancing accuracy from 96.73% to 97.75%,
concurrently reducing training time from 57.3715 to 37.839 minutes,
demonstrating a 1.5 times increase in speed while improving accuracy.

For the CIFAR-10 dataset, BlindTuner achieves a competitive test accuracy of
95.20% in 31.3342 minutes, remaining competitive with HETAL, which achieves a
slightly higher accuracy of 96.57% but necessitates a longer training duration
of 51.905 minutes. Overall, BlindTuner showcases robust performance in
CIFAR-10 relative to HETAL, achieving comparable accuracy with faster
training.

On the Face Mask Detection dataset, BlindTuner achieves an impressive accuracy
of 97.18% in just 3.9086 minutes, surpassing HETAL by 2.4 times in training
efficiency.

Examining the DermaMNIST dataset, Glyph reports an accuracy of 73.2% with a
substantial training period of 7 days. In contrast, BlindTuner maintains
competitive accuracy of 76.11% with a remarkably reduced training time of
16.2993 minutes, emphasizing BlindTuner’s efficiency by being over 600 times
faster than Glyph and presenting quicker computation than HETAL. This
disparity underscores the effectiveness and adaptability of BlindTuner across
diverse data for image classification.

### ViT Vs DEiT

The introduction of the Vision Transformer (ViT) model in 2020 marked a
departure from conventional image classification architectures, relying solely
on attention mechanisms while omitting convolutional neural networks
(Dosovitskiy et al. 2020). ViT showcased commendable performance when pre-
trained on expansive datasets, yet its training process proved computationally
demanding. To address this challenge, the Data-Efficient Image Transformer
(DEiT) emerged in 2021, specifically designed to enhance ViT’s efficiency
(Touvron et al. 2021). DEiT incorporates novel techniques like distillation
and progressive resizing, strategically curbing ViT’s data demands.
Consequently, DEiT enables more resource-efficient model pre-training.
Notably, DEiT models pre-trained on datasets, like ImageNet, achieve
performance levels equivalent to ViT models when fine-tuned for tasks like
object detection and segmentation (Touvron et al. 2021). Furthermore, fine-
tuning DEiT models requires significantly fewer computational resources and
training samples compared to their ViT counterparts. This efficiency renders
DEiT an advantageous choice for fine-tuning various computer vision tasks,
especially in scenarios constrained by limited training data. Given these
advantages, we adopt DEiT in our implementation. Through our experiments, we
observe that encrypted training using DEiT outperforms ViTs across all
datasets, showcasing superior accuracy improvements and reduced training
times.

Table 4: The comparison of BlindTuner with respect to overall training latency Methodology | Dataset | Accuracy | Time
---|---|---|---
Glyph (Lou et al. 2020) | MNIST | 98.6% | 8 days
CIFAR-10 | Not reported | Not reported
Face Mask Detection | Not reported | Not reported
DermaMNIST | 73.2% | 7 days
HETAL (Lee et al. 2023) | MNIST | 96.73% | 57.3715 minutes
CIFAR-10 | 96.57% | 51.905 minutes
Face Mask Detection | 95.46% | 9.4453 minutes
DermaMNIST | 76.06% | 18.9498 1minutes
BlindTuner (Ours) | MNIST | 97.75% | 37.839 minutes
CIFAR-10 | 95.20% | 31.3342 minutes
Face Mask Detection | 97.18% | 3.9086 minutes
DermaMNIST | 76.11 % | 16.2993 minutes

## Related Work

Fully Homomorphic Encryption (FHE) (Gentry 2009), widely recognized as a holy
grail of cryptography, has garnered considerable attention in privacy-
preserving machine learning research. Graepel et al. introduced an innovative
approach to binary classification confidentiality in their work, ML
Confidential (Graepel, Lauter, and Naehrig 2012), rooted in polynomial
approximations and leveraging FHE. Simultaneously, Dowlin et al. explored
inference and learning on encrypted data in CryptoNets (Gilad-Bachrach et al.
2016), based on FHE principles and modifications to activation functions,
influencing advancements in privacy-preserving cloud-based neural networks.

Additionally, foundational work by Takabi et al. (Takabi, Hesamifard, and
Ghasemi 2016) in ensuring privacy in multiparty machine learning using FHE
laid the groundwork for the evolution of privacy-preserving Deep Learning (DL)
models. Hesamifard et al. (Hesamifard et al. 2018) introduced a technique for
approximating activation functions using low-degree polynomials, applied to
Convolutional Neural Networks (CNNs). Alongside FHE and polynomial
approximations, the Faster CryptoNets framework (Chou et al. 2018)
demonstrated the efficacy of pruning and quantization in harnessing sparse
representations within the cryptosystem, enhancing DL model inference
performance.

In large-scale real-time data analysis, Han et al. (Han et al. 2019) presented
a logistic regression model grounded in FHE, showing promising results.
Notably, Nandakumar et al. (Nandakumar et al. 2019) devised an FHE-based
approach for training neural networks on homomorphically encrypted data,
expanding the realm of privacy-preserving machine learning. The HomoPAI
framework (Li et al. 2020), proposed by Li et al., offers a secure
collaborative Machine Learning (ML) service for processing data from diverse
owners. Enriched with parallelism through the Message Passing Interface (MPI)
framework, it achieves performance enhancements and time-saving attributes.
Lou et al. introduced an approach for transfer learning using fast fully
homomorphic encryption over the torus and BGV cryptosystems in (Lou et al.
2020). Recently, Lee et al. (Lee et al. 2023) introduced privacy-preserving
transfer learning using FHE, achieving faster performance while utilizing GPUs
for computation.

## Conclusion

In this work, we introduce BlindTuner, a privacy-preserving fine-tuning system
that leverages fully homomorphic encryption in conjunction with data-efficient
image transformers. Our empirical findings underscore that BlindTuner
proficiently trains transformer models, maintaining parity in accuracy with
unencrypted models. Furthermore, our experimental observations substantiate
the superior performance of BlindTuner when compared to prior approaches. In
our future work, we will focus on extending our methodology to encompass more
intricate computer vision datasets and large-scale transformer models.

## References

  * Brakerski, Gentry, and Vaikuntanathan (2014) Brakerski, Z.; Gentry, C.; and Vaikuntanathan, V. 2014.  (Leveled) fully homomorphic encryption without bootstrapping.  _ACM Transactions on Computation Theory (TOCT)_ , 6(3): 1–36.
  * Cheon et al. (2018) Cheon, J. H.; Han, K.; Kim, A.; Kim, M.; and Song, Y. 2018.  Bootstrapping for approximate homomorphic encryption.  In _Advances in Cryptology–EUROCRYPT 2018: 37th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Tel Aviv, Israel, April 29-May 3, 2018 Proceedings, Part I 37_ , 360–384. Springer.
  * Cheon et al. (2017) Cheon, J. H.; Kim, A.; Kim, M.; and Song, Y. 2017.  Homomorphic encryption for arithmetic of approximate numbers.  In _Advances in Cryptology–ASIACRYPT 2017: 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, December 3-7, 2017, Proceedings, Part I 23_ , 409–437. Springer.
  * Cheon, Son, and Yhee (2021) Cheon, J. H.; Son, Y.; and Yhee, D. 2021.  Practical FHE parameters against lattice attacks.  _Cryptology ePrint Archive_.
  * Chou et al. (2018) Chou, E.; Beal, J.; Levy, D.; Yeung, S.; Haque, A.; and Fei-Fei, L. 2018.  Faster cryptonets: Leveraging sparsity for real-world encrypted inference.  _arXiv preprint arXiv:1811.09953_.
  * Creeger (2022) Creeger, M. 2022.  The Rise of Fully Homomorphic Encryption: Often called the Holy Grail of cryptography, commercial FHE is near.  _Queue_ , 20(4): 39–60.
  * Crockett (2020) Crockett, E. 2020.  A low-depth homomorphic circuit for logistic regression model training.  _Cryptology ePrint Archive_.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Gentry (2009) Gentry, C. 2009.  _A fully homomorphic encryption scheme_.  Stanford university.
  * Gilad-Bachrach et al. (2016) Gilad-Bachrach, R.; Dowlin, N.; Laine, K.; Lauter, K.; Naehrig, M.; and Wernsing, J. 2016.  Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy.  In _International conference on machine learning_ , 201–210. PMLR.
  * Goldreich (2004) Goldreich, O. 2004.  _Foundations of Cryptography, Volume 2_.  Cambridge university press Cambridge.
  * Graepel, Lauter, and Naehrig (2012) Graepel, T.; Lauter, K.; and Naehrig, M. 2012.  ML confidential: Machine learning on encrypted data.  In _International Conference on Information Security and Cryptology_ , 1–21. Springer.
  * Han et al. (2019) Han, K.; Hong, S.; Cheon, J. H.; and Park, D. 2019.  Logistic regression on homomorphic encrypted data at scale.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 9466–9471.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hesamifard et al. (2018) Hesamifard, E.; Takabi, H.; Ghasemi, M.; and Wright, R. N. 2018.  Privacy-preserving machine learning as a service.  _Proc. Priv. Enhancing Technol._ , 2018(3): 123–142.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Larxel (2020) Larxel. 2020.  Face Mask Detection Dataset.
  * LeCun et al. (1998) LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11): 2278–2324.
  * Lee et al. (2023) Lee, S.; Lee, G.; Kim, J. W.; Shin, J.; and Lee, M.-K. 2023.  HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption.
  * Li et al. (2020) Li, Q.; Huang, Z.; Lu, W.-j.; Hong, C.; Qu, H.; He, H.; and Zhang, W. 2020.  HomoPAI: A Secure Collaborative Machine Learning Platform based on Homomorphic Encryption.  In _2020 IEEE 36th International Conference on Data Engineering (ICDE)_ , 1713–1717. IEEE.
  * Lou et al. (2020) Lou, Q.; Feng, B.; Charles Fox, G.; and Jiang, L. 2020.  Glyph: Fast and accurately training deep neural networks on encrypted data.  _Advances in neural information processing systems_ , 33: 9193–9202.
  * Nandakumar et al. (2019) Nandakumar, K.; Ratha, N.; Pankanti, S.; and Halevi, S. 2019.  Towards deep neural network training on encrypted data.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 0–0.
  * Nesterov (1983) Nesterov, Y. E. 1983.  A method of solving a convex programming problem with convergence rate O$\backslash$bigl(k^2$\backslash$bigr).  In _Doklady Akademii Nauk_ , volume 269, 543–547. Russian Academy of Sciences.
  * Pan and Yang (2009) Pan, S. J.; and Yang, Q. 2009.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10): 1345–1359.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Takabi, Hesamifard, and Ghasemi (2016) Takabi, H.; Hesamifard, E.; and Ghasemi, M. 2016.  Privacy preserving multi-party machine learning with homomorphic encryption.  In _29th Annual Conference on Neural Information Processing Systems (NIPS)_.
  * Tan and Le (2019) Tan, M.; and Le, Q. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , 6105–6114. PMLR.
  * Touvron et al. (2021) Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and Jégou, H. 2021.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , 10347–10357. PMLR.
  * Tschandl, Rosendahl, and Kittler (2018) Tschandl, P.; Rosendahl, C.; and Kittler, H. 2018.  The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific data_ , 5(1): 1–9.
  * Weiss, Khoshgoftaar, and Wang (2016) Weiss, K.; Khoshgoftaar, T. M.; and Wang, D. 2016.  A survey of transfer learning.  _Journal of Big data_ , 3(1): 1–40.
  * Yang et al. (2023) Yang, J.; Shi, R.; Wei, D.; Liu, Z.; Zhao, L.; Ke, B.; Pfister, H.; and Ni, B. 2023.  MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification.  _Scientific Data_ , 10(1): 41.