# FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Zhaopeng Peng1 Xiaoliang Fan1,111Corresponding Author Yufan Chen1 Zheng Wang1
Shirui Pan2 Chenglu Wen1 Ruisheng Zhang3 Cheng Wang1
1Fujian Key Laboratory of Sensing and Computing for Smart Cities, School of
Informatics, Xiamen University
2School of Information and Communication Technology, Griffith University
3School of Information Science and Engineering, Lanzhou University
<EMAIL_ADDRESS><EMAIL_ADDRESS>{23020231154174,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Adapting Foundation Models (FMs) for downstream tasks through Federated
Learning (FL) emerges a promising strategy for protecting data privacy and
valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in
FL, however, leading to suboptimal performance due to insufficient tuning and
inevitable error accumulations of gradients. In this paper, we propose
Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation
in downstream tasks through FL by two key modules. First, the sub-FM
construction module employs a layer-wise compression approach, facilitating
comprehensive FM fine-tuning across all layers by emphasizing those crucial
neurons. Second, the sub-FM alignment module conducts a two-step
distillations—layer-level and neuron-level—before and during FL fine-tuning
respectively, to reduce error of gradient by accurately aligning sub-FM with
FM under theoretical guarantees. Experimental results on seven commonly used
datasets (i.e., four text and three vision) demonstrate the superiority of
FedPFT. Ours code is at https://github.com/pzp-dzd/FedPFT.

##  1 Introduction

(a) Two types of proxy sub-FM construction

(b) The problem of accumulating gradients errors

Figure 1: A motivating example of two challenges in FM fine-tuning using proxy
sub-model. (a) Existing methods constructing sub-FMs via layer-drop
compression discard intermediate layers in FM, causing mismatched and
insufficient fine-tuning, while FedPFT conducting layer-wise compression
ensures comprehensive fine-tuning of FM; and (b) as FL fine-tuning progresses,
the discrepancy between the updates made by sub-FMs and FMs grows, leading to
a deviation from the ideal update direction, while FedPFT aims to mitigate
this gap by accurately aligning sub-FMs and FMs.

In recent years, various transformer-based Foundation Models (FMs) Bommasani
et al. (2021) such as BERT Kenton and Toutanova (2019), GPT Radford et al. ,
LLaMA Touvron et al. (2023), and ViT Dosovitskiy et al. (2020) have attained
state-of-the-art performance across a diverse range of natural language
processing (NLP) and computer vision (CV) tasks, yet also face both data
privacy and FM copyright concerns. For instance, a FM trained on medical data
might inadvertently memorize sensitive patient information, and companies that
own closed-source FMs may choose not to share FMs with the public. Federated
Learning (FL) McMahan et al. (2017) offers a privacy-preserving approach for
collaborative fine-tuning of FMs among multiple participants. This approach is
increasingly promising for FM fine-tuning applications, ensuring the
adaptation of downstream tasks without directly sharing client data and server
FM.

Recent methods Xiao et al. (2023); Marchisio et al. (2023) mainly aim to fine-
tune FMs without using the full model, which leverage layer-drop techniques
Sajjad et al. (2023) to compress a FM and derive a sub-FM, enabling
approximate fine-tuning of the original FM. However, these methods still pose
two significant challenges that adversely reduce the performance of fine-tuned
FMs. On one hand, they failed to fine-tune FMs sufficiently as a result of
discarding those intermediate layers of FMs, consequently leading to the
performance degradation of fine-tuned FMs. As shown in Fig.1(a), layer-drop
methods fail to update intermediate layers of the FM during fine-tuning, due
to the mismatch between the FM and the constructed sub-FM. On the other hand,
there is a potential defect for the accumulation of gradient errors of FMs due
to the lack of alignment between sub-FMs and FMs during FL fine-tuning,
subsequently leading to further performance degradation. Fig.1(b) shows that,
due to the absence of alignment, existing methods might accumulate significant
gradients update errors between the FM and its constructed sub-FM during the
FL fine-tuning process.

To address the above two challenges, we propose a framework called Federated
Proxy Fine-Tuning (FedPFT) to enhance the adaptation of FMs for downstream
tasks, while neither server FMs nor client data are directly shared. First, we
design the sub-FM construction module, which performs layer compression on FMs
to obtain sub-FMs by measuring neurons saliency of Feed-Forward Network (FFN)
in transformer, facilitating comprehensive fine-tuning of FMs by emphasizing
those crucial neurons. Second, we design the sub-FM alignment module, which
conducts a two-step distillations—layer-level and neuron-level—before and
during FL fine-tuning respectively, ensuring the accurate alignment between
sub-FMs and FMs with a theoretical guarantee. Extensive experiments on three
FMs (i.e., BERT-base, RoBERTa-base, and ViT-base) and seven commonly used
datasets (i.e., SST-2, QNLI, MNLI, QQP, CIFAR-10, CIFAR-100, Flowers)
demonstrate that FedPFT outperforms existing baselines that fine-tune FMs
without using the full model.

Our contributions can be summarized as follows:

  * •

We introduced FedPFT, a novel federated fine-tuning of FM method that
establishes a sub-FM as a local proxy. FedPFT effectively improves fine-tuning
performance while maintaining the critical constraint that neither the server
FM nor the client data is directly shared.

  * •

We propose the first module for constructing sub-FMs through layer-wise
compression. This technique maintains layer correspondence across sub-FMs and
FMs, ensuring the comprehensive fine-tuning of FM layers while also
considering the alleviation of training overhead.

  * •

We proposed the second module to align sub-FMs with FMs via a two-step
distillation—layer-level and neuron-level—before and during FL fine-tuning
respectively. Additionally, we offer theoretical insights into the
significance of distillation for fine-tuning using sub-model.

  * •

We conducted extensive experiments on three FMs and seven commonly used
datasets. Results demonstrate that FedPFT consistently outperforms existing
baselines.

##  2 Related Works

###  2.1 FM Fine-tuning through FL

Traditional centralized fine-tuning faces privacy concerns due to data
sharing. Recent works Chen et al. (2023a); Yu et al. (2023); Zhuang et al.
(2023) introduce the concepts of Federated Foundation Models, to alleviate
privacy concerns. Fan et al. (2023); Kuang et al. (2023) propose various Fed-
LLM platforms to support federated training of LLMs. Xu et al. (2023) fine-
tune FM via FL on mobile devices. Chen et al. (2023b) apply FM to federated
multi-task learning. Chen et al. (2023c) save the communication cost during FL
training through block-level parameters dropout. Wang et al. (2023a) reduce
the communication and computation cost by training different layers of BERT in
each round. Zhang et al. (2023) apply parameter-efficient fine-tuning (PEFT)
methods to federated fine-tuning of FMs for privacy defense. However, most of
aforementioned methods rely on sharing the server FM. This limitation may pose
risks of FM copyright leakage and impose a substantial computational burden on
clients.

###  2.2 FM Fine-tuning without using the full model

Early PEFT methods, including Lora Hu et al. (2021), Adapter-Tuning Houlsby et
al. (2019), and Prefix-Tuning Li and Liang (2021), focus on efficient fine-
tuning of complete FMs by reducing the number of tunable parameters. Despite
these efforts, the gradient computation for tunable parameters still needs
backpropagation through the entire FM Sung et al. (2022). Recently, Offsite-
Tuning Xiao et al. (2023) is proposed to achieve fine-tuning without the full
model. In this approach, the FM owner sends a light-weight adapter and an
emulator constructed through layer-drop and knowledge distillation Hinton et
al. (2015) to clients. Clients then fine-tune the adapter on their data with
support from the emulator. The refined adapter is subsequently returned and
incorporated into the full model for the fine-tuning process. Similarly, mini-
model adaptation Marchisio et al. (2023) constructs a shallow mini-model from
a fraction of the FM’s parameters. However, those methods either discard
significant amount of intermediate layers in FM or face the problem of
gradient error accumulation, resulting in sub-optimal fine-tuning performance.
Different from conventional methods, we construct sub-FMs based on layer-wise
compression and mitigate gradient error accumulation by a two-step
distillations.

##  3 FedPFT

Figure 2: The overall framework of FedPFT that enhances FMs adaptation in
downstream tasks through FL by two key modules: (1) Sub-FM Construction Module
constructs sub-FM by layer-wise compression to facilitate comprehensive FM
fine-tuning; and (2) Sub-FM Alignment Module aligns sub-FM by two-step
distillation to ensure accurate alignment between sub-FM and FM with a
theoretical guarantee.

###  3.1 Preliminary

####  3.1.1 Federated Learning

Given $N$ parties $P_{i}(i=1,....,N)$, each party holds data $D_{i}$. Let
$L(\cdot,\cdot)$ be the loss function. FL aims to train a machine learning
model $\Theta$ using the dataset $D=\cup D_{i}(i=1,...,N)$ under the
coordination of a server $S$, while the raw data of all parties are not
directly shared, which is formally described as

$\displaystyle\Theta=\mathop{\arg\min}\limits_{\Theta}{\sum_{i=1}^{N}\frac{\lvert
D_{i}\rvert}{\lvert D\rvert}L(\Theta,D_{i})}.$ (1)

####  3.1.2 Foundation Model Fine-tuning

Given a foundation model $\Theta=\\{W_{1},W_{2},...,W_{n}\\}$ and a downstream
task dataset $D$, the fine-tuning aims to obtain a new model
$\Theta^{*}=\\{W_{1}^{*},W_{2}^{*},...,W_{n}^{*}\\}$, it is

$\displaystyle\Theta^{*}$ $\displaystyle=\Theta+\Delta\Theta,$ (2)
$\displaystyle\Delta\Theta$
$\displaystyle=\mathop{\arg\min}\limits_{\Delta\Theta}{L(\Theta+\Delta\Theta,D)}.$

###  3.2 Problem Definition

For FM fine-tuning using proxy sub-model, we first construct a sub-model
$\Theta^{{}^{\prime}}=\\{W_{1},W_{2},...,W_{k},W_{k+1}^{{}^{\prime}},...,W_{n}^{{}^{\prime}}\\}$
with fewer parameters for $\Theta$ to act as a proxy. Second, fine-tune the
proxy sub-model $\Theta^{{}^{\prime}}$ using the dataset $D$. Finally
synchronize the updated gradients on $\Theta^{{}^{\prime}}$ to $\Theta$.
Specifically, we construct $\Theta^{{}^{\prime}}$ by compressing $\Theta$, and
retain a portion of the parameter matrix in $\Theta$ during the compression
process. This compression process is formally described as follows:

$\displaystyle\Theta^{{}^{\prime}}=\Theta_{1}\cup C(\Theta_{2}),$ (3)

where $\Theta_{1}\cup\Theta_{2}=\Theta$, and $C(\cdot)$ denotes the
compression method. During the fine-tuning of $\Theta^{{}^{\prime}}$, we
update only $\Theta_{1}$ and synchronize the updated gradient on $\Theta_{1}$
into $\Theta$ after fine-tuning to obtain $\Theta^{*}$ ’s approximation of
$\Theta^{*^{\prime}}$, which is formally described as

$\displaystyle\Theta^{*^{\prime}}$
$\displaystyle=(\Theta_{1}+\Delta\Theta_{1}^{{}^{\prime}})\cup\Theta_{2},$ (4)
$\displaystyle\Delta\Theta_{1}^{{}^{\prime}}$
$\displaystyle=\mathop{\arg\min}\limits_{\Delta\Theta_{1}^{{}^{\prime}}}{L((\Theta_{1}+\Delta\Theta_{1}^{{}^{\prime}})\cup
C(\Theta_{2}),D)}.$

###  3.3 Method Overview

The overall framework of ours FedPFT is shown in Fig.2. We first derive a
proxy sub-FM for the server FM, then collaboratively fine-tune the sub-FM
through FL, and finally synchronise the updates on the sub-FM to the FM by
plugging-in. FedPFT enhances downstream tasks adaptation of FMs through FL by
two key module: (1) Sub-FM Construction Module that constructs sub-FMs by
performing layer-wise compression on FMs based on neuron saliency; and (2)
Sub-FM Alignment Module that reduces the difference between FMs and sub-FMs by
layer-level and neuron-level knowledge distillation before and during FL fine-
tuning, respectively. We will introduce those two modules in details as
follows.

###  3.4 Sub-FM Construction Module based on Layer-wise Compression

Transformer-based FM typically consist of three parts: an embedding layer, a
task head, and a sequence of transformer layers. Since the size of FM is
dominated by all transformer layers, we perform compression for each
transformer layer.

Each transformer layer contains two sub-layers: Multi-Head Attention (MHA) and
Feed-Forward Network (FFN), each of which applies residual connection and
followed by layer normalization. The output of MHA is

$\displaystyle{\rm MHA}(x)$ $\displaystyle={\rm Concat}({\rm
Attn}_{0}(x),...,{\rm Attn}_{h}(x))W^{O},$ (5) $\displaystyle{\rm Attn}(x)$
$\displaystyle={\rm
softmax}(\frac{xW^{Q}({xW^{K}})^{T}}{\sqrt{d_{k}}})xW^{V},$

where $W^{Q}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{K}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{V}\in\mathbb{R}^{d_{model}\times d_{k}}$ and
$W^{O}\in\mathbb{R}^{d_{model}\times d_{model}}$ are the weight matrices of
query, key, value, and output in MHA, respectively. $h$ is the number of
attention heads, $d_{k}$ and $d_{model}$ are the dimensions of key and FM,
respectively, and $d_{model}=d_{k}\times h$. The parameters number of MHA is
about $4d_{model}^{2}$.

The output of FFN is

$\displaystyle{\rm FFN}(x)$ $\displaystyle={\rm
gelu}(xW_{1}+b_{1})W_{2}+b_{2},$ (6)

where $W_{1}\in\mathbb{R}^{d_{model}\times d_{ff}}$ and
$W_{2}\in\mathbb{R}^{d_{ff}\times d_{model}}$ are the weight matrices of two
linear layers in FFN, respectively, $b_{1}\in\mathbb{R}^{d_{ff}}$ and
$b_{2}\in\mathbb{R}^{d_{model}}$ are the bias, $d_{ff}$ is the dimensions of
FFN and is usually set to $4\times d_{model}$. The parameters number of FFA is
about $8d_{model}^{2}$. Obviously, it is that most of the parameters in
transformer layer are contained in FFN.

Hence, we opt to compress the FFN rather than the MHA of each layer for sub-FM
construction. This minimizes the parameters number of sub-FM while ensuring a
consistent set of trainable parameters (i.e. MHA) between the FM and its sub-
FM at each layer. We accomplish layer-wise compression by systematically
removing neurons with low saliency in the FFN of each layer, employing a fixed
ratio.

First, by further transforming (6), we can represent the output of FFN as the
sum of $d_{ff}$ neurons outputs:

$\displaystyle{\rm FFN}(x)=\sum_{i=1}^{d_{ff}}{({\rm
gelu}(xu_{i}+b_{1i})w_{i}})+b_{2},$ (7)

where $w_{i}\in\mathbb{R}^{d_{model}}$ is the $i$th column vector in $W_{2}$,
$u_{i}\in\mathbb{R}^{d_{model}}$ is the $i$th row vector in $W_{1}$, $b_{1i}$
is the $i$th item in $b_{1}$.

Second, based on (7) and magnitude-based pruning method Wen et al. (2016), we
use the L2-norm of all connect weights of neuron to measure its saliency, that
is:

$\displaystyle{\rm
Saliency}(i)=\sqrt{\sum_{j=1}^{d_{model}}(w_{ij}^{2}+u_{ij}^{2})},$ (8)

where $i$ is the index of neurons in FFN.

Finally, we construct a sub-FM serving as a proxy for the FM, accomplished by
systematically eliminating neurons with low saliency in each layer at a fixed
ratio.

###  3.5 Sub-FM Alignment Module based on Two-step Knowledge Distillation

In accordance with the description of FM fine-tuning using proxy sub-model in
3.2, it is evident that the FM fine-tuning is entirely contingent on the
gradient descent of its sub-FM. This fine-tuning methodology prompts a
fundamental question: How can we ensure the convergence of FM to the optimal
value with the assistance of its sub-FM?

######  Theorem 1.

Suppose both the function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ and its
approximation $f^{\prime}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ are convex and
differentiable, and their gradient are Lipschitz continuous with constant
$L_{1}\textgreater 0$ and $L_{2}\textgreater 0$, respectively, i.e. we have
that$\|\triangledown f(x)-\triangledown f(y)\|_{2}\leq L_{1}\|x-y\|_{2}$ and
$\|\triangledown f^{\prime}(x)-\triangledown f^{\prime}(y)\|_{2}\leq
L_{2}\|x-y\|_{2}$ for any $x,y$. Then if we run gradient descent for $k$
iterations on $f^{\prime}$ with a fixed step size $\eta\leq\frac{1}{L_{1}}$
and synchronize the gradient to $f$, let $\triangledown
f^{\prime}-\triangledown f=\delta$, when satisfying

$\displaystyle\|\delta\|_{2}^{2}$
$\displaystyle\textless\frac{1}{2}\|\triangledown f\|_{2}^{2},$ (9)
$\displaystyle\eta\sum_{i=1}^{k}\|\delta^{(i)}\|_{2}^{2}$
$\displaystyle\leq\sum_{i=1}^{k}\langle\delta^{(i)},x^{(i)}-x^{*}\rangle,$
(10)

it will yield a solution $f^{(k)}$ which satisfies

$\displaystyle f(x^{(k)})-f(x^{*})\leq\frac{\|x^{(0)}-x^{*}\|_{2}^{2}}{2\eta
k},$ (11)

where $f(x^{*})$ is the optimal value.

###### Proof.

See Appendix.A ∎

Intuitively, Theorem.1 indicates that when (9) and (10) are satisfied,
gradient descent of FM with the help of sub-FM is guaranteed to converge and
converges with rate $O(\frac{1}{k})$. It is evident that both conditions (9)
and (10) are constraints on the difference between the actual and ideal update
gradients of FM, and thus how to minimize the difference of the update
gradients becomes a problem to be solved in the next step.

######  Theorem 2.

For a transformer, let the number of attention head be 1, and ignoring its
nonlinear function and residual connection, its output can be expressed as
$y=xW^{Q}(xW_{K})^{T}xW^{V}W^{O}W_{1}W_{2}$, let $A=W^{Q}(W^{K})^{T}$,
$B=W^{V}W^{O}$, $C=W_{1}W_{2}$, then $y=xAx^{T}xBC$, and the output of its
corresponding sub-layer after compressing FFN layer is expressed as
$y^{\prime}=xAx^{T}xBC^{\prime}$, assuming that the gradient of loss function
$loss=f(y)$ is Lipschitz continuous with constant $L_{3}\textgreater 0$ and
$\|C-C^{\prime}\|_{2}^{2}\leq\epsilon_{1}$,
$\|y-y^{\prime}\|_{2}^{2}\leq\epsilon_{2}$, there exists the constant
$K_{1}\textgreater 0$ and $K_{2}\textgreater 0$ such that

$\displaystyle\|\frac{\partial loss^{\prime}}{\partial A}-\frac{\partial
loss}{\partial A}\|_{2}^{2}$ $\displaystyle\leq
K_{1}\epsilon_{1}+K_{2}\epsilon_{2},$ (12) $\displaystyle\|\frac{\partial
loss^{\prime}}{\partial B}-\frac{\partial loss}{\partial B}\|_{2}^{2}$
$\displaystyle\leq K_{1}\epsilon_{1}+K_{2}\epsilon_{2},$

###### Proof.

See Appendix.B ∎

From Theorem.2, it is evident that shrinking the error of gradients can be
achieved by narrowing the difference in output and weights between the sub-FM
and FM.

Based on the above analysis, we grasp the importance of narrowing the
difference between sub-FM and FM via knowledge distillation to boost the
performance of FM that fine-tuned using sub-FM. Therefore, we propose a method
to align sub-FM using layer-level and neuron-level distillations in two
phases, before and during FL fine-tuning, respectively. These two distillation
methods are shown in the Fig.3.

Figure 3: An example of two distillation processes

####  3.5.1 Layer-level distillation before FL fine-tuning

Given that our sub-FMs are constructed based layer-wise compression, where
each layer retains a set of tunable parameters (i.e., MHA), we leverage the
outputs from all layers to compute the layer-level distillation loss.

Furthermore, based on Theorem.2, we enhance the aforementioned distillation
loss by introducing a regularization term. The purpose of this regularization
term is to quantify the disparity between the weights of FFN and sub-FFN in
each layer, to further facilitate a thorough knowledge transfer during fine-
tuning by refining the alignment process. Thus, the final distillation loss is
denoted as:

$\displaystyle L_{KD}=$
$\displaystyle\frac{1}{LM_{KD}}\sum_{i=1}^{L}((\sum_{j=1}^{M_{KD}}\|O_{j}^{(i)}-O_{j}^{{}^{\prime}(i)}\|_{2}^{2})$
$\displaystyle+\mu\|W_{1}^{(i)}W_{2}^{(i)}-W_{1}^{{}^{\prime}(i)}W_{2}^{{}^{\prime}(i)}\|_{2}^{2}),$
(13)

where $L$ is the number of layers, $M_{KD}$ is the size of distill dataset,
$O_{j}^{(i)}$ is the output of the $j$th sample in the $i$th layer,
$W_{1}^{(i)}$ and $W_{2}^{(i)}$ are the two weight matrices of the $i$th FFN,
$\mu$ is the regularization coefficient.

####  3.5.2 Neuron-level distillation during FL fine-tuning

In addition, the absence of alignment between FM and its constructed sub-FM
during FL fine-tuning may cause the actual gradient update direction of the FM
to gradually deviate from its ideal direction. This deviation can
substantially reduce the performance of the fine-tuned FM. To mitigate the
problem, we re-align the sub-FM with the FM after FL aggregation in certain
rounds.

However, since the datasets for distillation and the datasets for local fine-
tuning are typically collected from different domains, excessively aligning
sub-FMs through distillation may hinder the adaptation of sub-FMs to
downstream tasks. Inspired by Mallya and Lazebnik (2018), during the alignment
process in FL fine-tuning, we opt to update only a subset of neurons with low
saliency in local fine-tuning to prevent the risk of sub-FM forgetting
knowledge of local data.

Moreover, since all FFNs of sub-FM are not updated during FL fine-tuning, the
effectiveness of magnitude-based neuron saliency measurement methods
diminishes. To address this, we opt to select neurons for updating during
alignment based on the Average Percentage of Zero activations (APoZ) of
outputs on the downstream task dataset Hu et al. (2016). The $APoZ_{k}^{(i)}$
of the $k$th neuron in $i$th layer is defined as:

$\displaystyle{APoZ}_{k}^{(i)}=\frac{1}{M_{DT}S}\sum_{j=1}^{M_{DT}}\sum_{l=1}^{S}\mathbb{I}(O_{jkl}^{(i)}=0),$
(14)

where $M_{DT}$ is the size of the downstream task dataset, $S$ is the sequence
length of the $j$th sample, $O_{jkl}^{(i)}$ is the output of the $l$th token
of the $j$th sample at the $k$th neuron in $i$th layer, $\mathbb{I}(\cdot)$ is
the indicator function.

We calculate the APoZ for each neuron on the client using the local dataset
before each round that requires alignment, and subsequently select the neurons
that need to be updated during alignment based on their APoZ values.

Model | Setting | Method | SST-2 | QNLI | MNLI-(m/mm) | QQP |  Transformers Params
---|---|---|---|---|---|---|---
BERT |  Fine-tuning with full model | FedPETuning | 91.6 | 88.4 | 80.7/81.6 | 87.8 | 81M
Fine-tuning without full model | FedOT | 90.4 | 83.9 | 74.9/74.9 | 81.6 | 47M
FedPFT | 91.6 | 87.5 | 78.6/79.0 | 86.3 | 47M
RoBERTa |  Fine-tuning with full model | FedPETuning | 93.6 | 90.8 | 86.1/85.5 | 88.3 | 81M
Fine-tuning without full model | FedOT | 92.8 | 85.3 | 80.6/81.4 | 84.6 | 47M
FedPFT | 93.1 | 88.7 | 83.4/83.2 | 87.1 | 47M
Table 1: Overall experimental results on BERT and RoBERTa. The evaluation
metric is accuracy. For MNLI, ’m’ and ’mm’ denote the matched and mismatched
results, respectively. The best result for the same setting is marked in bold.

###  3.6 Cost Analysis

We perform a theoretical analysis of the computational and communication cost
of FedPFT based on BERT, and other models such as RoBERTa and ViT are similar.
Following the settings in Wang et al. (2023a), we assume that all FL clients
have the same training settings and exclude external differences such as local
dataset size and hardware environment.

####  3.6.1 Computational Cost

Given a BERT model, let $V$ be the vocabulary size, $S$ be the sequence
length, $L$ be the number of layers, and $c_{f}$,$c_{b}$ be the number of
forward propagation and backward propagation respectively. Based on the
analysis in 3.4, the computational cost of a BERT model is
$O(d_{model}(V+S)+L(4Sd_{model}^{2}+S^{2}d_{model})+2LSd_{model}d_{ff}+LSd_{model})$
where the four terms denote the cost of embedding, MHA, FFN and Add&Norm,
respectively.

Based on the above information, the overall time complexity of the full model
is computed as follows. First, the time complexity of embedding is
$O(d_{model}(V+S))$. Second, due to $d_{ff}=4d_{model}$, the forward
propagation time complexity is about
$O(c_{f}L(12Sd_{model}^{2}+S^{2}d_{model}))$. Identically, the backward
propagation time complexity is $O(c_{b}L(12Sd_{model}^{2}+S^{2}d_{model}))$.
Finally, the overall time cost is
$O(d_{model}(V+S)+L(c_{f}+c_{b})(12Sd_{model}^{2}+S^{2}d_{model}))$ and we
have $d_{model}(V+S)\ll L(c_{f}+c_{b})(12Sd_{model}^{2}+S^{2}d_{model})$ and
$S\textless d_{model}$.

In FedPFT, because $d^{\prime}_{ff}=d_{model}$ for sub-FM, the time complexity
during local training is $O(L(c_{f}+c_{b})(6Sd_{model}^{2}+S^{2}d_{model}))$.
Thus, compared with full model, FedPFT could reduce almost half the
computational cost of all clients.

####  3.6.2 Communication Cost

Following Vaswani et al. (2017), the space complexity of a BERT model is
$O(d_{model}(V+S)+12Ld_{model}^{2}+2Ld_{model})$.

In FedPFT, if PEFT methods is not used, all model parameters need to be
transmitted in each iteration, thus the communication cost will be in the
complexity of $O(d_{model}(V+S)+6Ld_{model}^{2}+2Ld_{model})$, again shrinking
nearly half of the cost. If PEFT method is used, take Lora as an example, let
$t$ be the interval between two alignments during FL fine-tuning, $q$ be the
proportion of neurons that need to be updated during alignment, $r$ be the
rank of Lora, and Lora is only added to MHA, then the communication cost will
be in the complexity of $O(8Lrd_{model}+\frac{2}{t}qLd_{model}^{2})$. Since
$\frac{1}{t}qd_{model}$ is usually smaller than $r$, the complexity is about
$O(Lrd_{model})$, which does not impose a significant computational cost.

##  4 Experiments

###  4.1 Experimental Setting

####  4.1.1 Models and Datasets

Our evaluations span a variety of FMs, including two NLP FMs (i.e., BERT-base
Kenton and Toutanova (2019), RoBERTa-base Liu et al. (2019)) and one CV FM
(i.e., ViT-base Dosovitskiy et al. (2020)). Three FMs share consistent hyper-
parameters for the number of layers $L$, attention heads $h$, hidden dimension
$d_{model}$, and FFN dimension $d_{ff}$, all set at $L=12$, $h=12$,
$d_{model}=768$, and $d_{ff}=3072$. Our NLP FM evaluations encompass four text
datasets: SST-2 Socher et al. (2013), QNLI Socher et al. (2013), MNLI Williams
et al. (2017), and QQP222https://quoradata.quora.com/First-Quora-Dataset-
Release-Question-Pairs. The CV FM is evaluated on three image datasets:
CIFAR-10 Krizhevsky et al. (2009), CIFAR-100 Krizhevsky et al. (2009) and
Flowers Nilsback and Zisserman (2008). Specifically, SST-2 is the text
sentiment analysis task, QNLI and MNLI are the natural language inference
tasks, QQP is the text matching task, while CIFAR-10, CIFAR-100 and Flowers
focus on image recognition tasks. In addition, we employ the Bookcorpus Zhu et
al. (2015) and Wikipedia datasets for distillation of NLP sub-FMs, and the
ImageNet-1k Russakovsky et al. (2015) for the distillation of CV sub-FM,
respectively. Detailed dataset descriptions can be found in Appendix.C.

####  4.1.2 Baselines

We compare FedPFT with two methods, including: (1) FedPETuning Zhang et al.
(2023) that performs parameter-efficient fine-tuning (PEFT) with full model
through FL; and (2) FedOT that performs PEFT without full model through FL. It
is worth noting that FedOT is the FL implementation of Offsite-Tuning Xiao et
al. (2023) with multiple clients.

####  4.1.3 Hyper-parameters and Implementation

In the FL scenario, we set up 100 clients with 500 total communication rounds
and employ the Dirichlet data partition method Hsu et al. (2019) to construct
different label-skew data heterogeneity scenarios. In each communication
round, we randomly select 10 clients for local fine-tuning, using a linear
decay of the global learning rate over rounds and AdamW as the local fine-
tuning optimizer. We use FedAvg McMahan et al. (2017) for global model
aggregation. For FedOT, following Xiao et al. (2023), we use 2 layers at the
bottom and 2 layers at the top as $Adapter$, and compress the intermediate 8
layers into 3 layers as $Emulator$. For FedPFT, we construct sub-FMs by
performing layer-wise compression on the intermediate 10 layers. For fair
comparison, we keep the number of trainable parameters the same for all three
methods. The evaluation metric is the accuracy on the given validation set.
All pre-trained models used in experiments are obtained from Hugging
Face333https://huggingface.co/. The FL scenarios were implemented using FLGo
Wang et al. (2023b), and all experiments are conducted in PyTorch 2.1 and
NVIDIA 3090 GPUs. Other detailed hyper-parameters can be seen in Appendix.C.

###  4.2 Overall Comparisons

Model | Method | CIFAR-10 |  Transformers Params
---|---|---|---
ViT | FedPETuning | 98.2 | 81M
FedOT | 95.5 | 47M
FedPFT | 97.2 | 47M
Table 2: Experimental results on ViT. The evaluation metric is accuracy. Model | Method | SST-2 | QNLI |  Transformers Params
---|---|---|---|---
Dir-1.0 | Dir-5.0 | Dir-10.0 | Dir-1.0 | Dir-5.0 | Dir-10.0
BERT | FedPETuning | 90.6 | 90.9 | 91.1 | 87.3 | 87.9 | 88.1 | 81M
FedOT | 89.1 | 89.9 | 90.1 | 82.4 | 82.9 | 83.1 | 47M
FedPFT | 89.7 | 91.1 | 91.6 | 86.1 | 86.9 | 87.2 | 47M
RoBERTa | FedPETuning | 93.0 | 93.3 | 93.5 | 90.2 | 90.6 | 90.8 | 81M
FedOT | 91.7 | 92.1 | 92.8 | 84.1 | 84.6 | 85.1 | 47M
FedPFT | 92.1 | 92.7 | 93.1 | 87.2 | 87.6 | 88.1 | 47M
Table 3: Non-IID experimental results on BERT and RoBERTa. The evaluation
metric is accuracy.

We first evaluate all three methods in Independent Identically Distributed
(I.I.D) data distribution scenarios. Table.1 compares our FedPFT with two
baselines for fine-tuning BERT and RoBERTa on four text datasets. We observe
that: 1) the performance of all FMs fine-tuned by our FedPFT surpasses that
achieved by FedOT and closely approaches the performance achieved by
FedPETuning that fine-tuning using full model; and 2) FedOT exhibit a
substantial performance gap compared to FedPETuning. This observed discrepancy
from FedOT might be attributed to the absence of training the $Emulator$ and
the accumulation of gradient errors during fine-tuning.

We further validate the effectiveness of FedPFT for fine-tuning CV FM. Table.2
presents the comparison results on the ViT model with CIFAR-10 dataset, and
FedPFT still outperforms FedOT and achieves competitive performance closer to
FedPETuning. The results on other two vision datasets (i.e., CIFAR-100,
Flowers) are shown in Appendix.D.

###  4.3 Impact of Data Heterogeneity

We then evaluate the performance of FMs fine-tuned by three different methods
in Non-Independent Identically Distributed (Non-I.I.D) data distribution
scenarios. For the datasets used in the data heterogeneity experiments, we
unequally partition the dataset into 100 clients following the label
distribution $Y_{i}\backsim Dir(\alpha p)$, where $i$ is the client id, $p$ is
the global label distribution, $\alpha$ is the degree of Non-I.I.D and a
smaller $\alpha$ generates a high label distribution shift. We construct three
different label-skewing scenarios by adjusting the value of $\alpha$: Dir-1.0,
Dir-5.0, and Dir-10.0. Table.3 shows the comparison of three methods for fine-
tuning BERT and RoBERTa in different data-heterogeneous scenarios. We observe
that: 1) the performance of all methods declines as the degree of Non-I.I.D
increases; 2) our FedPFT still outperforms FedOT and achieves competitive
performance closer to FedPETuning. Visualisation of label distributions and
results of Non-I.I.D experiments on vision datasets are shown in the
Appendix.E.

###  4.4 Ablation Study

Method | Model
---|---
BERT | RoBERTa
FedOT | 83.9 | 85.3
FedPFT_N | 83.0 | 78.9
FedPFT_B | 86.9 | 86.7
FedPFT_D | 84.3 | 85.5
FedPFT(ours) | 87.5 | 88.7
Table 4: Ablation study of FedPFT on QNLI

To evaluate the efficacy of individual components within FedPFT, we design the
following variants for conducting ablation study:

  * •

FedPFT_N which does not perform alignment by knowledge distillation;

  * •

FedPFT_B which perform sub-FM alignment by knowledge distillation only before
FL fine-tuning;

  * •

FedPFT_D which perform sub-FM alignment by knowledge distillation only during
FL fine-tuning;

Moreover, to validate the effectiveness of the sub-FM construction module of
FedPFT, we list the experimental results of FedOT for comparison with
FedPFT_B, as both perform sub-FM alignment by knowledge distillation only
before FL fine-tuning. Results are shown in Table.4 and we observe that: 1)
both FedPFT_N which does not align the sub-FMs entirely and FedPFT_D which
lacks the alignment before FL fine-tuning, exhibit notably poor performance.
This is consistent with our theoretical analysis (see Theorem.1), indicating
that a significant disparity between the gradients of sub-FM and FM can impede
the convergence of fine-tuning methods using proxy sub-model; 2) FedPFT_B
outperforms FedOT, showing the effectiveness of the sub-FM construction module
in FedPFT; and 3) FedPFT outperforms FedPFT_B, emphasizing the necessity of
sub-FM alignment during FL fine-tuning. Results on other text and vision
datasets are presented in Appendix.F.

###  4.5 Parameter Study

Hyper-Parameter | Model
---|---
BERT | RoBERTa
Alignment interval $t$ | 5 | 87.3 | 88.4
10 | 87.5 | 88.7
20 | 87.1 | 87.1
Updated neurons proportion $p$ | 0.3 | 87.4 | 87.8
0.5 | 87.5 | 88.7
0.8 | 87.1 | 87.9
Table 5: Parameter study of FedPFT on QNLI

In addition, we conduct a study to investigate the impact of two hyper-
parameters in the sub-FM alignment module: the interval $t$ between two
alignments during FL fine-tuning and the proportion $p$ of neurons that need
to be updated for each alignment. The effects of two hyper-parameters on the
QNLI dataset are presented in Table.5, suggesting that both $t$ and $p$ should
be chosen moderately. Regarding $t$, longer alignment intervals lead to the
accumulation of gradient errors, while shorter intervals can prohibit the
adaptation of downstream tasks. Concerning $p$, it is crucial to strike a
balance between updating an adequate proportion of neurons and avoiding
excessive disruption to local fine-tuning.

##  5 Conclusion

This paper introduces FedPFT, a federated fine-tuning framework designed for
Foundation Models (FMs). FedPFT addresses critical challenges related to
insufficient FM fine-tuning and the accumulation of gradient errors by
employing layer-wise compression for sub-FM construction and aligning sub-FM
through a two-step distillation process, respectively. This novel framework
achieves optimal downstream task adaptation of FM, resulting in an effective
fine-tuned FM with superior performance, all without direct sharing of either
server FM or client data. Experimental results across seven datasets showcase
the effectiveness of FedPFT. In the future, we aim to extend the application
of FedPFT to larger-scale FMs for tackling more complex downstream tasks.

##  6 Acknowledgment

The research was supported by Natural Science Foundation of China (62272403).

## References

  * Birhane and Prabhu [2021] Abeba Birhane and Vinay Uday Prabhu.  Large image datasets: A pyrrhic win for computer vision?  In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536–1546. IEEE, 2021.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * Chen et al. [2023a] Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Yin, and Xiaolin Zheng.  Federated large language model: A position paper.  arXiv preprint arXiv:2307.08925, 2023.
  * Chen et al. [2023b] Yiqiang Chen, Teng Zhang, Xinlong Jiang, Qian Chen, Chenlong Gao, and Wuliang Huang.  Fedbone: Towards large-scale federated multi-task learning.  arXiv preprint arXiv:2306.17465, 2023.
  * Chen et al. [2023c] Yuanyuan Chen, Zichen Chen, Pengcheng Wu, and Han Yu.  Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning.  In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 3541–3549. International Joint Conferences on Artificial Intelligence Organization, 8 2023.  Main Track.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * Fan et al. [2023] Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang.  Fate-llm: A industrial grade federated learning framework for large language models.  arXiv preprint arXiv:2310.10049, 2023.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * Hsu et al. [2019] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown.  Measuring the effects of non-identical data distribution for federated visual classification.  arXiv preprint arXiv:1909.06335, 2019.
  * Hu et al. [2016] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang.  Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.  arXiv preprint arXiv:1607.03250, 2016.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kuang et al. [2023] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou.  Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning.  arXiv preprint arXiv:2309.00363, 2023.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * Mallya and Lazebnik [2018] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * Marchisio et al. [2023] Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe.  Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 5474–5490, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008.
  * [22] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * Sajjad et al. [2023] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov.  On the effect of dropping layers of pre-trained transformer models.  Computer Speech & Language, 77:101429, 2023.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013.
  * Sung et al. [2022] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  Lst: Ladder side-tuning for parameter and memory efficient transfer learning.  Advances in Neural Information Processing Systems, 35:12991–13005, 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * Wang et al. [2023a] Xin’ao Wang, Huan Li, Ke Chen, and Lidan Shou.  Fedbfpt: An efficient federated learning framework for bert further pre-training.  In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 4344–4352. International Joint Conferences on Artificial Intelligence Organization, 8 2023.  Main Track.
  * Wang et al. [2023b] Zheng Wang, Xiaoliang Fan, Zhaopeng Peng, Xueheng Li, Ziqi Yang, Mingkuan Feng, Zhicheng Yang, Xiao Liu, and Cheng Wang.  Flgo: A fully customizable federated learning platform.  arXiv preprint arXiv:2306.12079, 2023.
  * Wen et al. [2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li.  Learning structured sparsity in deep neural networks.  Advances in neural information processing systems, 29, 2016.
  * Williams et al. [2017] Adina Williams, Nikita Nangia, and Samuel R Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  arXiv preprint arXiv:1704.05426, 2017.
  * Xiao et al. [2023] Guangxuan Xiao, Ji Lin, and Song Han.  Offsite-tuning: Transfer learning without full model.  arXiv preprint arXiv:2302.04870, 2023.
  * Xu et al. [2023] Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang.  Federated fine-tuning of billion-sized language models across mobile devices.  arXiv preprint arXiv:2308.13894, 2023.
  * Yu et al. [2023] Sixing Yu, J Pablo Muñoz, and Ali Jannesari.  Federated foundation models: Privacy-preserving and collaborative learning for large models.  arXiv preprint arXiv:2305.11414, 2023.
  * Zhang et al. [2023] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu.  FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 9963–9977, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015.
  * Zhuang et al. [2023] Weiming Zhuang, Chen Chen, and Lingjuan Lyu.  When foundation model meets federated learning: Motivations, challenges, and future directions.  arXiv preprint arXiv:2306.15546, 2023.

##  Appendix A Proof of Theorem 1

###### Proof.

Our assumption that $\triangledown f$ is Lipschitz continuous with constant
$L_{1}$ implies that $\triangledown^{2}f(x)\preceq L_{1}I$, or equivalently
that $\triangledown^{2}f(x)-L_{1}I$ is a negative semidefinite matrix. Using
this fact, we can perform a quadratic expansion of $f$ around $f(x)$ and
obtain the following inequality:

$\displaystyle f(y)$ $\displaystyle\leq f(x)+\triangledown
f(x)^{T}(y-x)+\frac{1}{2}\triangledown^{2}f(x)\|y-x\|_{2}^{2}$ (15)
$\displaystyle\leq f(x)+\triangledown
f(x)^{T}(y-x)+\frac{L_{1}}{2}\|y-x\|_{2}^{2}$

Since we run gradient descent on $f^{\prime}$ and synchronize the gradient to
$f$ and $\triangledown f^{\prime}-\triangledown f=\delta$, let’s plug in the
gradient descent update by letting $y=x^{+}=x-\eta\triangledown
f^{\prime}(x)=x-\eta\triangledown f(x)-\eta\delta$. We then get:

$\displaystyle f(x^{+})$ $\displaystyle\leq f(x)+\triangledown
f(x)^{T}(x^{+}-x)+\frac{L_{1}}{2}\|x^{+}-x\|_{2}^{2}$ (16) $\displaystyle
f(x^{+})$ $\displaystyle\leq f(x)+\triangledown f(x)^{T}(-\eta\triangledown
f(x)-\eta\delta)$ $\displaystyle+\frac{L_{1}}{2}\|\eta\triangledown
f(x)+\eta\delta\|_{2}^{2}$ $\displaystyle f(x^{+})$ $\displaystyle\leq
f(x)-\eta(\|\triangledown f(x)\|_{2}^{2}+\triangledown f(x)^{T}\delta)$
$\displaystyle+\frac{L_{1}\eta^{2}}{2}\|\triangledown f(x)+\delta\|_{2}^{2}$
$\displaystyle f(x^{+})$ $\displaystyle\leq f(x)-\eta(\|\triangledown
f(x)\|_{2}^{2}+\triangledown f(x)^{T}\delta)$
$\displaystyle+\frac{L_{1}\eta^{2}}{2}(\|\triangledown
f(x)\|_{2}^{2}+\|\delta\|_{2}^{2})$ $\displaystyle f(x^{+})$
$\displaystyle\leq f(x)-(1-\frac{L_{1}\eta}{2})\eta\|\triangledown
f(x)\|_{2}^{2}-\eta\triangledown f(x)^{T}\delta$
$\displaystyle+\frac{L_{1}\eta^{2}}{2}\|\delta\|_{2}^{2}$

Using $\eta\leq\frac{1}{L_{1}}$, we know that
$-(1-\frac{L_{1}\eta}{2})\leq-\frac{1}{2}$ and
$\frac{L_{1}\eta^{2}}{2}\leq\frac{\eta}{2}$. Plugging them into (16), we can
get the following:

$\displaystyle f(x^{+})$ $\displaystyle\leq f(x)-\frac{\eta}{2}\|\triangledown
f(x)\|_{2}^{2}-\eta\triangledown
f(x)^{T}\delta+\frac{\eta}{2}\|\delta\|_{2}^{2}$ (17)
$\displaystyle=f(x)-\frac{\eta}{2}(\|\triangledown
f(x)+\delta\|_{2}^{2}-2\|\delta\|_{2}^{2})$
$\displaystyle=f(x)-\frac{\eta}{2}(\|\triangledown
f^{\prime}(x)\|_{2}^{2}-2\|\delta\|_{2}^{2})$

If we have

$\displaystyle\|\delta\|_{2}^{2}$
$\displaystyle\textless\frac{1}{2}\|\triangledown f^{\prime}\|_{2}^{2},$ (18)

then

$\displaystyle f(x^{+})$ $\displaystyle\leq f(x).$ (19)

This implies that when (18) is satisfied, the value of $f$ strictly decreases
with each iteration of gradient decent of $f^{\prime}$ until it reaches the
optimal value $f(x)=f(x^{*})$.

Next, we can bound $f(x^{+})$, the objective value at the next iteration, in
terms of $f(x^{*})$, the optimal objective value. Since $f$ is convex, we can
write:

$\displaystyle f(x^{*})\geq f(x)+\triangledown f(x)^{T}(x^{*}-x),$ (20)
$\displaystyle f(x)\leq f(x^{*})+\triangledown f(x)^{T}(x-x^{*}),$ (21)

plugging them into (17), we obtain:

$\displaystyle f(x^{+})\leq f(x^{*})$ $\displaystyle+\triangledown
f(x)^{T}(x-x^{*})$ (22) $\displaystyle-\frac{\eta}{2}(\|\triangledown
f^{\prime}(x)\|_{2}^{2}-2\|\delta\|_{2}^{2})$ $\displaystyle
f(x^{+})-f(x^{*})$ $\displaystyle\leq(\triangledown
f^{\prime}(x)-\delta)^{T}(x-x^{*})$
$\displaystyle-\frac{\eta}{2}(\|\triangledown
f^{\prime}(x)\|_{2}^{2}-2\|\delta\|_{2}^{2})$ $\displaystyle
f(x^{+})-f(x^{*})$ $\displaystyle\leq\frac{1}{2\eta}(2\eta\triangledown
f^{\prime}(x)^{T}(x-x^{*})-\eta^{2}\|\triangledown f^{\prime}(x)\|_{2}^{2})$
$\displaystyle-\delta^{T}(x-x^{*})+\eta\|\delta\|_{2}^{2}$ $\displaystyle
f(x^{+})-f(x^{*})$ $\displaystyle\leq\frac{1}{2\eta}(2\eta\triangledown
f^{\prime}(x)^{T}(x-x^{*})-\eta^{2}\|\triangledown f^{\prime}(x)\|_{2}^{2}$
$\displaystyle-\|x-x^{*}\|_{2}^{2})+\frac{1}{2\eta}\|x-x^{*}\|_{2}^{2}$
$\displaystyle-\delta^{T}(x-x^{*})+\eta\|\delta\|_{2}^{2}$ $\displaystyle
f(x^{+})-f(x^{*})$
$\displaystyle\leq-\frac{1}{2\eta}\|x-x^{*}-\eta\triangledown
f^{\prime}(x)\|_{2}^{2}+\frac{1}{2\eta}\|x-x^{*}\|_{2}^{2}$
$\displaystyle-\delta^{T}(x-x^{*})+\eta\|\delta\|_{2}^{2}$ $\displaystyle
f(x^{+})-f(x^{*})$
$\displaystyle\leq\frac{1}{2\eta}(\|x-x^{*}\|_{2}^{2}-\|x^{+}-x^{*}\|_{2}^{2})$
$\displaystyle-\delta^{T}(x-x^{*})+\eta\|\delta\|_{2}^{2}$

This inequality holds for $x^{+}$ on every update iteration. Summing over
iterations, we get:

$\displaystyle\sum_{i=1}^{k}f(x^{(i)})-f(x^{*})$
$\displaystyle\leq\sum_{i=1}^{k}\frac{1}{2\eta}(\|x^{(i-1)}-x^{*}\|_{2}^{2}-\|x^{(i)}-x^{*}\|_{2}^{2})$
(23) $\displaystyle-\delta^{(i)T}(x^{(i)}-x^{*})+\eta\|\delta^{(i)}\|_{2}^{2}$
$\displaystyle\sum_{i=1}^{k}f(x^{(i)})-f(x^{*})$
$\displaystyle\leq\frac{1}{2\eta}(\|x^{(0)}-x^{*}\|_{2}^{2}-\|x^{(k)}-x^{*}\|_{2}^{2})$
$\displaystyle+\sum_{i=1}^{k}\eta\|\delta^{(i)}\|_{2}^{2}-\delta^{(i)T}(x^{(i)}-x^{*})$
$\displaystyle\sum_{i=1}^{k}f(x^{(i)})-f(x^{*})$
$\displaystyle\leq\frac{1}{2\eta}\|x^{(0)}-x^{*}\|_{2}^{2}$
$\displaystyle+\sum_{i=1}^{k}\eta\|\delta^{(i)}\|_{2}^{2}-\delta^{(i)T}(x^{(i)}-x^{*})$

Then, using the fact that $f$ decreasing on every iteration, we can conclude
that:

$\displaystyle f(x^{(k)})-f(x^{*})$
$\displaystyle\leq\frac{1}{k}\sum_{i=1}^{k}f(x^{(i)})-f(x^{*})$ (24)
$\displaystyle f(x^{(k)})-f(x^{*})$ $\displaystyle\leq\frac{1}{2\eta
k}\|x^{(0)}-x^{*}\|_{2}^{2}$
$\displaystyle+\frac{1}{k}\sum_{i=1}^{k}\eta\|\delta^{(i)}\|_{2}^{2}-\delta^{(i)T}(x^{(i)}-x^{*})$

If we have

$\displaystyle\eta\sum_{i=1}^{k}\|\delta^{(i)}\|_{2}^{2}$
$\displaystyle\leq\sum_{i=1}^{k}\langle\delta^{(i)},x^{(i)}-x^{*}\rangle,$
(25)

then

$\displaystyle f(x^{(k)})-f(x^{*})\leq\frac{\|x^{(0)}-x^{*}\|_{2}^{2}}{2\eta
k}$ (26)

This implies that when (25) is satisfied, gradient descent of $f$ with the
help of $f^{\prime}$ is guaranteed to converge and converges with rate
$O(\frac{1}{k})$. ∎

##  Appendix B Proof of Theorem 2

###### Proof.

Based on the premise, we have:

$\displaystyle loss$ $\displaystyle=f(y),\quad y=xAx^{T}xBC;$ (27)
$\displaystyle loss^{\prime}$ $\displaystyle=f(y^{\prime}),\quad
y^{\prime}=xAx^{T}xBC^{\prime}.$

According to the chain rule, we can obtain:

$\displaystyle\frac{\partial loss}{\partial A}$
$\displaystyle=x^{T}\frac{\partial loss}{\partial y}C^{T}B^{T}x^{T}x,$ (28)
$\displaystyle\frac{\partial loss^{\prime}}{\partial A}$
$\displaystyle=x^{T}\frac{\partial loss^{\prime}}{\partial
y^{\prime}}C^{\prime T}B^{T}x^{T}x,$

then

$\displaystyle\|\frac{\partial loss^{\prime}}{\partial A}-\frac{\partial
loss}{\partial A}\|_{2}^{2}$ $\displaystyle=\|x^{T}(\frac{\partial
loss}{\partial y}C^{T}-\frac{\partial loss^{\prime}}{\partial
y^{\prime}}C^{\prime T})B^{T}x^{T}x\|_{2}^{2}$ (29)
$\displaystyle\|\frac{\partial loss^{\prime}}{\partial A}-\frac{\partial
loss}{\partial A}\|_{2}^{2}$
$\displaystyle\leq\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|\frac{\partial
loss}{\partial y}C^{T}-\frac{\partial loss^{\prime}}{\partial
y^{\prime}}C^{\prime T}\|_{2}^{2}$ $\displaystyle\|\frac{\partial
loss^{\prime}}{\partial A}-\frac{\partial loss}{\partial A}\|_{2}^{2}$
$\displaystyle\leq\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|\frac{\partial
loss}{\partial y}C^{T}-\frac{\partial loss^{\prime}}{\partial
y^{\prime}}C^{T}$ $\displaystyle+\frac{\partial loss^{\prime}}{\partial
y^{\prime}}C^{T}-\frac{\partial loss^{\prime}}{\partial y^{\prime}}C^{\prime
T}\|_{2}^{2}$ $\displaystyle\|\frac{\partial loss^{\prime}}{\partial
A}-\frac{\partial loss}{\partial A}\|_{2}^{2}$
$\displaystyle\leq\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|C\|_{2}^{2}\cdot\|\frac{\partial
loss}{\partial y}-\frac{\partial loss^{\prime}}{\partial
y^{\prime}}\|_{2}^{2}$
$\displaystyle+\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|\frac{\partial
loss^{\prime}}{\partial y^{\prime}}\|_{2}^{2}\cdot\|C-C^{\prime}\|_{2}^{2}.$

Additionally, our assumption that the gradient of loss function $loss=f(y)$ is
Lipschitz continuous with constant $L_{3}\textgreater 0$ implies that

$\displaystyle\|\frac{\partial loss}{\partial y}-\frac{\partial
loss^{\prime}}{\partial y^{\prime}}\|_{2}^{2}\leq
L_{3}\|y-y^{\prime}\|_{2}^{2}.$ (30)

Plugging this into (29), we have

$\displaystyle\|\frac{\partial loss^{\prime}}{\partial A}-\frac{\partial
loss}{\partial A}\|_{2}^{2}$ $\displaystyle\leq
L_{3}\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|C\|_{2}^{2}\cdot\|y-y^{\prime}\|_{2}^{2}$
(31) $\displaystyle+\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|\frac{\partial
loss^{\prime}}{\partial y^{\prime}}\|_{2}^{2}\cdot\|C-C^{\prime}\|_{2}^{2}.$

let

$\displaystyle K_{1}$
$\displaystyle=L_{3}\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|C\|_{2}^{2}$ (32)
$\displaystyle K_{2}$
$\displaystyle=\|x\|_{2}^{6}\cdot\|B\|_{2}^{2}\cdot\|\frac{\partial
loss^{\prime}}{\partial y^{\prime}}\|_{2}^{2}$

then we have

$\displaystyle\|\frac{\partial loss^{\prime}}{\partial A}-\frac{\partial
loss}{\partial A}\|_{2}^{2}$ $\displaystyle\leq
K_{1}\epsilon_{1}+K_{2}\epsilon_{2}$ (33)

Similarly, it can be derived that there exists the constant $K_{3}\textgreater
0$ and $K_{4}\textgreater 0$ such that:

$\displaystyle\|\frac{\partial loss^{\prime}}{\partial B}-\frac{\partial
loss}{\partial B}\|_{2}^{2}$ $\displaystyle\leq
K_{3}\epsilon_{1}+K_{4}\epsilon_{2}.$ (34)

Let $K_{1}=\max(K_{1},K_{3})$, $K_{2}=\max(K_{2},K_{4})$, then Theorem 2 is
proved. ∎

##  Appendix C Experimental Details

###  C.1 Datasets

####  C.1.1 SST-2

The Stanford Sentiment Treebank Socher et al. [2013] is a binary single-
sentence classification task consisting of sentences extracted from movie
reviews with human annotations of their sentiment. It is consists of a
training set of 67350 examples, a development set of 873 examples, and a test
set of 1821 examples.

####  C.1.2 QNLI

Question Natural Language Inference is a version of the Stanford Question
Answering Dataset which has been converted to a binary classification task
Wang et al. [2018]. It is consists of a training set of 104743 examples, a
development set of 5463 examples, and a test set of 5461 examples.

####  C.1.3 MNLI

Multi-Genre Natural Language Inference is a large-scale, crowdsourced
entailment classification task Williams et al. [2017]. It is consists of a
training set of 392702 examples, a matched development set of 9815 examples, a
mismatched development set of 9832 examples, a matched test set of 9796
examples, and a mismatched test set of 9847 examples.

####  C.1.4 QQP

Quora Question Pairs is a binary classification task where the goal is to
determine if two questions asked on Quora are semantically equivalent
444https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs. It
is consists of a training set of 363870 examples, a development set of 40431
examples, and a test set of 390965 examples.

####  C.1.5 CIFAR-10 and CIFAR-100

The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images
dataset Birhane and Prabhu [2021]. The CIFAR-10 dataset consists of 60000
32x32 colour images in 10 classes, with 6000 images per class, and there are
50000 training images and 10000 test images. The CIFAR-100 dataset is just
like the CIFAR-10, except it has 100 classes containing 600 images each, and
there are 500 training images and 100 testing images per class. The 100
classes in the CIFAR-100 are grouped into 20 superclasses. Each image comes
with a ”fine” label (the class to which it belongs) and a ”coarse” label (the
superclass to which it belongs).

####  C.1.6 Flowers

Oxford 102 Flower is an image classification dataset consisting of 102 flower
categories. The flowers were chosen to be flowers commonly occurring in the
United Kingdom. Each class consists of between 40 and 258 images. It is
consists of a training set of 6149 examples, a development set of 1020
examples, and a test set of 1020 examples.

###  C.2 Hyper-parameters

The number of distillation epochs for alignment before FL training for NLP
sub-FMs and CV sub-FM is 5, 50, respectively. For distillation of NLP FMs, the
learning rate is $6e-4$, the batch zise is 2048, and the weight decay is 0.01.
For distillation of CV FMs, the learning rate is $1e-3$, the batch size is
4096, and the weight decay is 0.1. All distillation use the linear learning
rate decay with warm up ratio of 0.06 and the gradient clipping with max grad
norm of 1. For client local fine-tuning, the fine-tuning method is Lora Hu et
al. [2021], the fine-tuning epoch is 1. For text datasets, we perform a grid
search to find the optimal parameters, with batch size$\in\\{8,16,32\\}$ and
learning rate$\in\\{4e-4,8e-4,1e-3\\}$. For CIFAR-10 and CIFAR-100 datasets,
we perform a grid search to find the optimal parameters, with batch
size$\in\\{64,128,256\\}$ and learning rate$\in\\{3e-3,8e-3,1e-2\\}$. For
Flowers dataset, we perform a grid search to find the optimal parameters, with
batch size$\in\\{16,32,64\\}$ and learning rate$\in\\{3e-3,8e-3,1e-2\\}$. For
FedPFT, alignment is performed every 10 rounds during FL training, with the
number of alignment epochs for NLP and CV being 0.02 and 0.2, respectively,
and the proportion of neurons that need to be updated during alignment being
0.5. The ratio of eliminated neurons in FFN is fixed at 0.75.

##  Appendix D Experimental results on I.I.D scenario

The I.I.D experiment results on CIFAR-100 and Flowers datasets are show in
Table.6. It is shown that FedPFT still outperforms FedOT and achieves
competitive performance closer to FedPETuning.

##  Appendix E Experimental results on Non-I.I.D scenario

(a) SST-2 Dir-1.0

(b) SST-2 Dir-5.0

(c) SST-2 Dir-10.0

(d) QNLI Dir-1.0

(e) QNLI Dir-5.0

(f) QNLI Dir-10.0

(g) CIFAR-10 Dir-1.0

(h) CIFAR-10 Dir-5.0

(i) CIFAR-10 Dir-10.0

Figure 4: Visualisations of the label distributions of SST-2, QNLI, and
CIFAR-10

Visualisations of the label distributions of the three datasets, SST-2, QNLI
and CIFAR-10, under different Non-I.I.D scenarios are shown in Fig.4. The
results of Non-I.I.D experiments on CIFAR-10 are shown in Table.7. Similarly,
We observe that: 1) the performance of all methods declines as the degree of
Non-I.I.D increases; 2) our FedPFT still outperforms FedOT and achieves
competitive performance closer to FedPETuning.

Method | CIFAR-100 | Flowers |  Transformers Params
---|---|---|---
FedPETuning | 89.3 | 99.7 | 81M
FedOT | 81.1 | 82.1 | 47M
FedPFT | 85.5 | 98.1 | 47M
Table 6: Other I.I.D experimental results on ViT. A higher value indicates better accuracy. Method | CIFAR-10 |  Transformers Params
---|---|---
Dir-1.0 | Dir-5.0 | Dir-10.0
FedPETuning | 97.8 | 98.0 | 98.2 | 81M
FedOT | 95.1 | 95.4 | 95.5 | 47M
FedPFT | 96.5 | 96.7 | 97.1 | 47M
Table 7: Non-IID experimental results on CIFAR-10. A higher value indicates
better accuracy.

##  Appendix F Ablation study on other datasets

The experiment results of ablation study on SST-2 and CIFAR-10 are shown in
Table.8. The experimental results exemplify the respective importance of the
sub-FM Construction Module and the Sub-FM Alignment Module in FedPFT.

Dataset | SST-2 | CIFAR-10
---|---|---
Model | BERT | RoBERTa | ViT
FedOT | 90.4 | 92.8 | 95.5
FedPFT_N | 84.2 | 89.1 | 95.1
FedPFT_B | 90.8 | 92.9 | 96.9
FedPFT_D | 88.4 | 89.7 | 96.0
FedPFT(ours) | 91.6 | 93.1 | 97.2
Table 8: Ablation study of FedPFT on SST-2 and CIFAR-10.

##  Appendix G Extension experimenal results

We extended the experiment in three parts: parameter study on CIFAR-10 (shown
in Table.9), a new evaluation metric on QQP (shown in Table.10), and
convergence analysis of FedPFT (shown in Table.11).

Hyper-Params | $t$ | $p$
---|---|---
5 | 10 | 20 | 0.1 | 0.5 | 0.9
Accuracy | 96.9 | 97.2 | 97.1 | 96.9 | 97.2 | 96.6
Table 9: Parameter study of FedPFT on CIFAR-10. Method | BERT | RoBERTa
---|---|---
F1 | rounds | F1 | rounds
FedPETuning | 83.5 | 235 | 84.7 | 116
FedOT | 77.9 | 287 | 77.7 | 80
FedPFT | 81.5 | 274 | 82.6 | 98
Table 10: F1 score of three methods on QQP. ’rounds’ is total communication rounds of required for the model to converge. Method | SST-2 | QNLI | QQP | MNLI
---|---|---|---|---
FedPETuning | 18 | 6 | 4 | 11
FedOT | 34 | 128 | 287 | 184
FedPFT | 21 | 31 | 24 | 23
Table 11: Communication rounds required to achieve the same performance by all
methods. We list the communication rounds required for all methods to achieve
the same performance as FedOT.


# Analytical and numerical study of
backreacting one-dimensional holographic superconductors
in the presence of Born-Infeld electrodynamics

Mahya Mohammadi  Physics Department and Biruni Observatory, Shiraz University,
Shiraz 71454, Iran  Ahmad Sheykhi<EMAIL_ADDRESS>Physics Department
and Biruni Observatory, Shiraz University, Shiraz 71454, Iran  Research
Institute for Astronomy and Astrophysics of Maragha (RIAAM), P. O. Box:
55134-441, Maragha, Iran  Mahdi Kord Zangeneh<EMAIL_ADDRESS>Physics
Department, Faculty of Science, Shahid Chamran University of Ahvaz, Ahvaz
61357-43135, Iran  Center for Research on Laser and Plasma, Shahid Chamran
University of Ahvaz, Ahvaz, Iran

###### Abstract

We analytically as well as numerically study the effects of Born-Infeld
nonlinear electrodynamics on the properties of $(1+1)$-dimensional $s$-wave
holographic superconductors. We relax the probe limit and further assume the
scalar and gauge fields affect on the background spacetime. We thus explore
the effects of backreaction on the condensation of the scalar hair. For the
analytical method, we employ the Sturm-Liouville eigenvalue problem and for
the numerical method, we employ the shooting method. We show that these
methods are powerful enough to analyze the critical temperature and phase
transition of the one dimensional holographic superconductor. We find out that
increasing the backreaction as well as nonlinearity makes the condensation
harder to form. In addition, this one-dimensional holographic superconductor
faces with second order phase transition and their critical exponent has the
mean field value $\beta={1}/{2}$.

##  I Introduction

The most well-known theory for describing the mechanism behind
superconductivity from microscopic perspective is the BCS theory proposed by
Bardeen, Cooper and Schrieffer. According to BCS theory, the condensation of
Cooper pairs into a boson-like state, at low temperature, is responsible for
infinite conductivity in solid state system BCS (1957). However, when the
temperature increases, the Cooper pairs decouples and thus the BCS theory is
unable to explain the mechanism of superconductivity for high temperature
superconductors BCS (1957). The correspondence between gravity in an Anti de-
Sitter (AdS) spacetime and a Conformal Field Theory (CFT) living on the
boundary of the spacetime provides a powerful tool for calculating correlation
functions in a strongly interacting field theory using a dual classical
gravity description Maldacena. (1998). According to the AdS/CFT duality
proposal an $n$-dimensional conformal field theory on the boundary is
equivalent to gravity theory in $(n+1)$-dimensional AdS bulk Maldacena.
(1998); Gubser et al. (1998); Witten et al. (1998); H08 ; Horowitz et al.
(2008); Ren. (2010). The dictionary of AdS/CFT duality implies that each
quantity in the bulk has a dual on the boundary. For example, energy-momentum
tensor $T_{\mu\nu}$ on the boundary corresponds to the bulk metric
$g_{\mu\nu}$Gubser et al. (1998); Witten et al. (1998). Based on this duality,
Hartnoll et al. proposed a model for holographic superconductor in $2008$ H08
. Their motivation was to shed light on the problem of high temperature
superconductors. According to the holographic superconductors, we need a hairy
black hole in gravity side to describe a superconductor on its boundary.
During the past decade, the investigation on the holographic superconductor
has got a lot of attentions (see e.g. Horowitz et al. (2008); Ren. (2010);
Hartnoll (2009); Herzog. (2009); Horowitz (2011); Gubser. (2009); HHH. (2008);
Jing ,Chena (210); Salahi et al. (2016); Sheykhi et al. (2016); cai (2015); Ge
(2010); SHsh (17); CAI (11); SHSH (16); shSh(16) ; Doa ; Afsoon ; cai (10,
14); yao (2013); n3 ; n4 ; n5 ; n6 ; Gan1 ).

On the other hand, BTZ (Bandos-Teitelboim-Zanelli) black holes, the well-known
solutions of general relativity in $(2+1)$-dimensional spacetime, provide a
simplified model to investigate some conceptual issues in black hole
thermodynamics, quantum gravity, string theory, gauge theory and AdS/CFT
correspondance Car1 ; Ash ; Sar ; Wit1 ; Car2 . It has been shown that the
quasinormal modes in this spacetime coincide with the poles of the correlation
function in the dual CFT. This gives quantitative evidence for AdS/CFT Bir .
In addition, BTZ black holes play a crucial role for improving our perception
of gravitational interaction in low dimensional spacetimes Wit2 . These kind
solutions have been studied from different point of views rin1 ; rin2 ; rin3 ;
rin4 .

Holographic superconductors dual to asymptotic BTZ black holes have been
explored widely (see e.g. Wang ; chaturvedi ; Li (2012); momeni ; peng17 ;
lashkari ; hua ; yanyan ; yan ; alkac ; 50-1 ; bina ). In order to construct
the $(1+1)$-dimensional holographic superconductors one should employ the
$AdS_{3}/CFT_{2}$ correspondence. In chaturvedi , the $(1+1)$-dimensional
holographic superconductors were explored in the probe limit and its
distinctive features in both normal and superconducting phases were
investigated. Employing the variational method of the Sturm-Liouville
eigenvalue problem, the one-dimensional holographic superconductors have been
analytically studied in Li (2012); momeni ; peng17 . It is also interesting to
study the $(1+1)$-dimensional holographic superconductor away from the probe
limit by considering the backreaction. In Wang , the effects of backreaction
have been studied for $s$-wave linearly charged one-dimensional holographic
superconductors.

Holographic superconductors have also been studied extensively in the presence
of nonlinear electrodynamics (see e.g. n4 ; Sheykhi et al. (2016); Salahi et
al. (2016); SHsh (17); SHSH (16); shSh(16) ; n3 ; n5 ; n6 ). The most famous
nonlinear electrodynamic is Born-Infeld electrodynamic. This model was
presented for the first time to solve the problem of divergence of electrical
field at the position of point particle 25 ; 26 ; 27 ; 28 ; 29 . It was later
showed that this model could be reproduced by string theory. In the present
work, we would like to extend the investigation on the $(1+1)$-dimensional
holographic superconductors by taking into account the nonlinear Born-Infeld
(BI) electrodynamics, as our gauge field. As well, we will study the effects
of backreaction on our holographic superconductors. We perform our
investigation both analytically and numerically and shall compare the result
of two methods. Our analytical approach is based on the Sturm-Liouville
variational method. In latter study, we find the relation between critical
temperature and chemical potential. Moreover, in order to study our
holographic superconductors numerically, we use the shooting method. We show
that analytical results are in good agreement with numerical ones which
implies that the Sturm-Liouvile variation method is still powerful enough for
studying the $(1+1)$-dimensional holographic superconductor.

The structure of our paper is as follows. In section II, the basic field
equations of one-dimensional holographic superconductors with backreaction in
the presence of BI nonlinear electrodynamics is introduced. In section III, we
describe the procedure of analytical study of one dimensional holographic
superconductor based on Sturm-Liouvile method and obtain the relation between
critical temperature and chemical potential. In section IV, the numerical
approach toward the study of our holographic superconductors will be
presented. Finally, we summarize our results in section V.

##  II Basic field equations

The action of three dimensional AdS gravity in the presence of a gauge and a
scalar field is given by

$\displaystyle S$ $\displaystyle=$ $\displaystyle\frac{1}{2\kappa^{2}}\int
d^{3}x\sqrt{-g}\left(R+\frac{2}{l^{2}}\right)$ $\displaystyle+\int
d^{3}x\sqrt{-g}\left[\mathcal{L}(\mathcal{F})-|\nabla\psi-
iqA\psi|^{2}-m^{2}|\psi|^{2}\right],$

where $m$ and $q$ shows the mass and the charge of scalar field,
$\kappa^{2}={8\pi G_{3}}$ and $G_{3}$ is $3$-dimensional Newton gravitation
constant. Also, $g$, $R$ and $l$ are the metric determinant, Ricci scalar and
AdS radius, respectively. In (LABEL:act), $\mathcal{F}=F_{\mu\nu}F^{\mu\nu}$
where $F_{\mu\nu}=\nabla_{[\mu}A_{\nu]}$ is the electrodynamics field tensor
and $A_{\mu}$ is the vector potential. $\mathcal{L}(\mathcal{F})$ stands for
the Lagrangian density of BI nonlinear electrodynamics defined as

$\mathcal{L}(\mathcal{F})=\frac{1}{b}\left(1-\sqrt{1+\frac{b\mathcal{F}}{2}}\right),$
(2)

where $b$ is the nonlinear parameter. When $b\rightarrow 0$, $\mathcal{L}$
reduces to $-F_{\mu\nu}F^{\mu\nu}/{4}$ which is the standard Maxwell
Lagrangian H08 . Variation of the above action with respect to scalar field
$\psi$, gauge field $A_{\mu}$ and the metric $g_{\mu\nu}$ yield the following
equations of motion

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\left(\nabla_{\mu}-i{q}A_{\mu}\right)\left(\nabla^{\mu}-i{q}A^{\mu}\right)\psi-m^{2}\psi\,,$
(3) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\nabla^{\mu}\left(4\mathcal{L}_{\mathcal{F}}F_{\mu\nu}\right)$
(4)
$\displaystyle-i{q}\left[-\psi^{\ast}(\nabla_{\nu}-i{q}A_{\nu})\psi+\psi(\nabla_{\nu}+i{q}A_{\nu})\psi^{\ast}\right]\,,$
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{1}{2\kappa^{2}}\left[R_{\mu\nu}-g_{\mu\nu}\left(\frac{R}{2}+\frac{1}{l^{2}}\right)\right]+2F_{ac}F_{b}{}^{c}\mathcal{L}_{\mathcal{F}}$
$\displaystyle-\frac{g_{\mu\nu}}{2}\left[\mathcal{L}(\mathcal{F})-m^{2}|\psi|^{2}-{|\nabla\psi-i{q}A\psi|^{2}}\right]$
$\displaystyle-\frac{1}{2}\left[(\nabla_{\mu}\psi-i{q}A_{\mu}\psi)(\nabla_{\nu}\psi^{\ast}+i{q}A_{\nu}\psi^{\ast})+\mu\leftrightarrow\nu\right],$

where $\mathcal{L}_{\mathcal{F}}=\partial\mathcal{L}/\partial{\mathcal{F}}$.

Since, we would like to consider the effect of the backreaction on the
holographic superconductor, we take a metric ansatz as follows Wang

${ds}^{2}=-f(r)e^{-\chi(r)}{dt}^{2}+\frac{{dr}^{2}}{f(r)}+\frac{r^{2}}{l^{2}}{dx}^{2}.$
(6)

The Hawking temperature of the three dimensional black hole on the outer
horizon $r_{+}$ (where $r_{+}$ is the horizon radius obtained as the largest
root of $f(r_{+})=0$), may be obtained through the use of the general
definition of surface gravity SheyKaz

$\displaystyle T$ $\displaystyle=$
$\displaystyle\frac{\kappa_{sg}}{2\pi}=\frac{1}{2\pi}\sqrt{-\frac{1}{2}(\nabla_{\mu}\chi_{\nu})(\nabla^{\mu}\chi^{\nu})},$
(7)

where $\kappa_{sg}$ is the surface gravity and $\chi=\partial/\partial t$ is
the null killing vector of the horizon. Taking $\chi^{\nu}=(-1,0,0)$, we have
$\chi_{\nu}=(f(r_{+})e^{-\chi(r_{+})},0,0)$ and hence on the horizon where
$f(r_{+})=0$, we find
$(\nabla_{\mu}\chi_{\nu})(\nabla^{\mu}\chi^{\nu})=-\frac{1}{2}\left[f^{\prime}(r_{+})\right]^{2}e^{-\chi(r_{+})}$.
Thus, the temperature is obtained as

$T=\frac{e^{-\chi(r_{+})/2}f^{{}^{\prime}}(r_{+})}{4\pi}.$ (8)

We also choose the scalar and the gauge fields as H08

$A_{\mu}=(\phi(r),0,0),\ \ \ \psi=\psi(r).$ (9)

Substituting (6) and (9) into the field equations (3)-(LABEL:Eein), we arrive
at

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\psi^{\prime\prime}+\psi^{\prime}\left[\frac{1}{r}+\frac{f^{\prime}}{f}-\frac{\chi^{\prime}}{2}\right]+\psi\left[\frac{q^{2}\phi^{2}{\mathrm{e}}^{\chi}}{f^{2}}-\frac{m^{2}}{f}\right],$
(10) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\phi^{\prime\prime}+\phi^{\prime}\left[-\frac{be^{\chi}\phi^{\prime
2}}{r}+\frac{\chi^{\prime}}{2}+\frac{1}{r}\right]-\frac{2q^{2}\psi^{2}\phi}{f}\left[1-be^{\chi}\phi^{\prime
2}\right]^{3/2},$ $\displaystyle 0$ $\displaystyle=$ $\displaystyle
f^{\prime}-\frac{2r}{l^{2}}$
$\displaystyle+2\kappa^{2}r\left[\frac{q^{2}e^{\chi}\psi^{2}\phi^{2}}{f}+f\psi^{\prime
2}+m^{2}\psi^{2}-\frac{1}{b}+\frac{1}{b\sqrt{1-be^{\chi}\phi^{\prime
2}}}\right],$ $\displaystyle 0$ $\displaystyle=$
$\displaystyle\chi^{\prime}+4\kappa^{2}r\left[\frac{q^{2}\phi^{2}\psi^{2}{\mathrm{e}}^{\chi}}{f^{2}}+\psi^{\prime
2}\right],$ (13)

where the prime denotes derivative with respect to $r$. Note that in the
presence of nonlinear BI electrodynamics the Eqs. (10) and (13) do not change
compared to the linear Maxwell case. In the limiting case where $b\rightarrow
0$ the equations of motion (LABEL:phir) and (LABEL:fr) turn to the
corresponding equations of one dimensional holographic superconductor with
Maxwell field Wang . The field equations (10)-(13) enjoy the symmetries

$\displaystyle q\rightarrow q/a,\text{ \ \ \ \ }\phi\rightarrow a\phi,\text{ \
\ \ \ }\psi\rightarrow a\psi,$ $\displaystyle\kappa\rightarrow\kappa/a,\text{
\ \ \ \ }b\rightarrow b/a^{2},$ (14) $\displaystyle l\rightarrow al,\text{ \ \
\ \ }r\rightarrow ar,\text{ \ \ \ \ }q\rightarrow q/a,$ $\displaystyle
m\rightarrow m/a,\text{ \ \ \ \ }b\rightarrow a^{2}b.$ (15)

Hereafter, we set $q$ and $l$ equal to unity by virtue of these symmetries.
The behavior of our model functions for large $r$ (near the boundary) read

$\displaystyle\chi\rightarrow 0,\ \ \ f(r)\sim r^{2},$
$\displaystyle\phi\sim\rho+\mu\ln(r),\ \ \
\psi\sim\psi_{-}r^{-\Delta_{-}}+\psi_{+}r^{-\Delta_{+}},$ (16)

where $\mu$ and $\rho$ are the chemical potential and the charge density of
the field theory on the boundary and $\Delta_{\pm}=1\pm\sqrt{1+m^{2}}$ which
implies $m^{2}\geq-1$. Actually, $\chi$ could be a constant near the boundary
but by using the symmetry of field equation $\mathrm{e}^{\chi}\rightarrow
a^{2}\mathrm{e}^{\chi},$ $\phi\rightarrow\phi/a$, it could be set to zero
there. From holographic superconductors point of view, either $\psi_{+}$ or
$\psi_{-}$ can be dual to the expectation value of condensation operator (or
order operator) $\left\langle O\right\rangle$ while the other is dual to its
source. We give $\psi_{-}$ the role of source and $\psi_{+}$ the role of
expectation value of the order parameter $\left\langle O_{+}\right\rangle$ in
this work. Since we seek for study the effects of $b$ and $\kappa$ on our
holographic superconductors and different values of the scalar field mass do
not influence this behavior qualitatively, we consider $m^{2}=0$ in this work.
With this choice, we have $\Delta_{+}=2$, $\Delta_{-}=0$ and thus

$\psi\sim\psi_{-}+\frac{\psi_{+}}{r^{2}},$ (17)

near the boundary. We set $\psi_{-}=0$ at the boundary and consider $\psi_{+}$
as the dual of order parameter $\left\langle O_{+}\right\rangle$. It is
remarkable to note that the asymptotic solution for $\psi$ given in Eq. (17)
do not depend on the type of electrodynamics and thus for the Maxwell case in
three dimensions the solution is the same as in Eq. (17). While the solution
depends on the spacetime dimensions. This is due to the fact that equation for
the $\psi$ given in (10) is independent on the type of electrodynamics but
depends on the spacetime dimensions and the mass parameter $m$ Wang ; Doa ;
ghor .

The next step is to solve the coupled nonlinear field equations (10)-(13)
simultaneously and obtain the behavior of model functions. Then, we could
figure out the behavior of different parameters of holographic superconductor
specially the order parameter $\left\langle O_{+}\right\rangle$ and the
critical temperature by using these functions. In this work, we use both
analytical and numerical methods for studying the holographic superconductor.
For analytical study, we perform Sturm-Liouville method while for numerical
study, we use shooting method.

##  III Sturm-Liouville method

Table 1: Analytical results of ${T_{c}}/{\mu}$ for different values of backreaction and nonlinearity parameters. | $b=0$ | $b=0.04$ | $b=0.08$
---|---|---|---
| Analytical | Numerical | Analytical | Numerical | Analytical | Numerical
$\kappa^{2}=0$ | $0.0429$ | $0.0460$ | $0.0360$ | $0.0410$ | $0.0275$ | $0.0362$
$\kappa^{2}=0.05$ | $0.0399$ | $0.0369$ | $0.0337$ | $0.0326$ | $0.0260$ | $0.0286$
$\kappa^{2}=0.1$ | $0.0381$ | $0.0295$ | $0.0311$ | $0.0260$ | $0.0218$ | $0.0227$
$\kappa^{2}=0.15$ | $0.0352$ | $0.0236$ | $0.0280$ | $0.0207$ | $0.0174$ | $0.0180$
$\kappa^{2}=0.2$ | $0.0313$ | $0.0189$ | $0.0242$ | $0.0165$ | $0.0136$ | $0.0143$
$\kappa^{2}=0.25$ | $0.0264$ | $0.0151$ | $0.0195$ | $0.0131$ | $0.0089$ | $0.0114$

In this section, we employ the Sturm-Liouville eigenvalue problem to
investigate analytically the phase transition of one dimensional $s$-wave
holographic superconductor in the presence of BI nonlinear electrodynamics. In
addition, we calculate the relation between the critical temperature $T_{c}$,
and chemical potential $\mu$, near the horizon. Furthermore, we study the
effect of backreaction and BI nonlinear electrodynamics on the critical
temperature. For future convenience, we define a new variable $z=r_{+}/r$
$(\in\left[0,1\right])$. With this new coordinate, the field equations
(10)-(13) could be rewritten as

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\psi^{\prime\prime}+\psi^{\prime}\left[\frac{f^{\prime}}{f}-\frac{\chi^{\prime}}{2}+\frac{1}{z}\right]+\psi\left[\frac{r_{+}^{2}e^{\chi}\phi^{2}}{z^{4}f^{2}}-\frac{m^{2}r_{+}^{2}}{z^{4}f}\right],$
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\phi^{\prime\prime}+\phi^{\prime}\left[\frac{bz^{3}e^{\chi}\phi^{\prime
2}}{r_{+}^{2}}+\frac{\chi^{\prime}}{2}+\frac{1}{z}\right]-\frac{2r_{+}^{2}\psi^{2}\phi}{z^{4}f}\Upsilon^{\frac{3}{2}},$
(19) $\displaystyle 0$ $\displaystyle=$ $\displaystyle
f^{\prime}+\frac{2r_{+}^{2}}{z^{3}}+\frac{2r_{+}^{2}\kappa^{2}}{z^{3}}$
$\displaystyle\times\left[\frac{1}{b}\left(1-\Upsilon^{-\frac{1}{2}}\right)-\frac{z^{4}f\psi^{\prime
2}}{r_{+}^{2}}-\frac{e^{\chi}\psi^{2}\phi^{2}}{f}-m^{2}\psi^{2}\right],$
$\displaystyle 0$ $\displaystyle=$
$\displaystyle\chi^{\prime}-4\kappa^{2}\left[\frac{r_{+}^{2}e^{\chi}\psi^{2}\phi^{2}}{z^{3}f^{2}}+z\psi^{\prime
2}\right],$ (21)

where $\Upsilon=1-bz^{4}e^{\chi}\phi^{\prime 2}/r_{+}^{2}$ and now the prime
indicates the derivative with respect to $z$. Since in the vicinity of
critical temperature the order parameter is small, we can consider it as an
expansion parameter

$\epsilon\equiv\left\langle O_{i}\right\rangle,$

where $i=+$ or $-$. We focus on solutions for small values of condensation
parameter $\epsilon$, therefore we can expand the model functions as

$\displaystyle\psi\approx\epsilon\psi_{1}+\epsilon^{3}\psi_{3}+\epsilon^{5}\psi_{5}+\cdots,$
$\displaystyle\phi\approx\phi_{0}+\epsilon^{2}\phi_{2}+\epsilon^{4}\phi_{4}+\cdots,$
$\displaystyle f\approx f_{0}+\epsilon^{2}f_{2}+\epsilon^{4}f_{4}+\cdots,$
$\displaystyle\chi\approx\epsilon^{2}\chi_{2}+\epsilon^{4}\chi_{4}+\cdots,$

where $\epsilon\ll 1$ near the critical temperature. Moreover, by considering
$\delta\mu_{2}>0$, the chemical potential can be expressed as:

$\mu=\mu_{0}+\epsilon^{2}\delta\mu_{2}+...\rightarrow\epsilon\thickapprox\Bigg{(}\frac{\mu-\mu_{0}}{\delta\mu_{2}}\Bigg{)}^{1/2}.$

During phase transition, $\mu_{c}=\mu_{0}$, thus the order parameter vanishes.
Meanwhile, the critical exponent $\beta=\frac{1}{2}$ is in a good agreement
with mean field theory result.

At zeroth order of $\epsilon$, the gauge field equation of motion (19) reduces
to

$\phi^{\prime\prime}+\frac{\phi^{\prime}}{z}+\frac{bz^{3}\phi^{\prime
3}}{r_{+}^{2}}=0,$ (22)

which could be rewritten as a first order Bernoulli differential equation by
taking $\phi^{\prime}$ as a new function TL (2000). Therefore, one receives

$\phi^{\prime}=\frac{\lambda r_{+}}{z\sqrt{b\lambda^{2}z^{2}+1}},$ (23)

for small values of $b$ where we define $\lambda=\mu/r_{+}$ and fix the
integration constants by looking at the behavior of $\phi$ near the boundary
given in (16). Integrating (23) and using the fact that $\phi(z=1)=0$111It is
necessary so that the norm of gauge potential is finite at horizon., we can
obtain

$\displaystyle\phi_{0}(z)$ $\displaystyle=$
$\displaystyle\int_{1}^{z}\frac{\lambda
r_{+}}{z}\left(1-\frac{1}{2}b\lambda^{2}z^{2}\right)\,dz$ (24)
$\displaystyle=$ $\displaystyle\lambda
r_{+}\log(z)-\frac{1}{4}b\lambda^{3}r_{+}\left(z^{2}-1\right).$

When $b=0$ the above equation reduces to one of Li (2012). Note that at the
zeroth order with respect to $\epsilon$, $\psi_{0}=\chi_{0}=0$. Substituting
(23) in the (LABEL:fz), the equation for $f$ at the zeroth order with respect
to $\epsilon$ has the following form

$\displaystyle f_{0}(z)=r_{+}^{2}g(z),$ $\displaystyle\text{\
}g(z)=\frac{1}{z^{2}}-1+\frac{1}{8}b\kappa^{2}\lambda^{4}\left(1-z^{2}\right)+\kappa^{2}\lambda^{2}\log(z).$
(25)

(a) b=0 (b) b=0.04

(c) b=0.08

Figure 1: The behavior of condensation parameter as a function of temperature
for different values of backreaction.

(a) $\kappa^{2}$=0 (b) $\kappa^{2}$=0.10

(c) $\kappa^{2}$=0.20

Figure 2: The behavior of condensation parameter as a function of temperature
for different values of nonlinearity parameter $b$.

The asymptotic behavior of the scalar field $\psi$ was given in (16). Near the
boundary, we define the function $F(z)$ so that

$\psi(z)=\frac{\left\langle O_{i}\right\rangle
z^{\triangle_{i}}}{\sqrt{2}r_{+}^{\triangle_{i}}}F(z).$ (26)

Inserting Eq. (26) in Eq. (LABEL:psiz) yields

$\displaystyle
F^{\prime\prime}(z)+F^{\prime}(z)\left[\frac{g^{\prime}(z)}{g(z)}+\frac{2\Delta}{z}+\frac{1}{z}\right]$
$\displaystyle+$ $\displaystyle F(z)\left[\frac{\Delta
g^{\prime}(z)}{zg(z)}-\frac{m^{2}}{z^{4}g(z)}+\frac{\Delta^{2}}{z^{2}}\right]$
$\displaystyle-$
$\displaystyle\frac{F(z)}{2z^{4}g(z)^{2}}\left[\lambda^{2}\log(z)\left(b\lambda^{2}r_{+}\left(z^{2}-1\right)-2\log(z)\right)\right]=0.$

We can rewrite this equation in the Sturm-Liouville form as

$\left[T(z)F^{\prime}(z)\right]^{\prime}+P(z)T(z)F(z)+\lambda^{2}Q(z)T(z)F(z)=0,$
(28)

where the functions $T$, $P$, $Q$ are defined as

$T(z)=z^{2\Delta+1}\left[\frac{1}{z^{2}}-1+\frac{1}{8}b\kappa^{2}\lambda^{4}\left(1-z^{2}\right)+\kappa^{2}\lambda^{2}\log(z)\right].$
(29)
$P(z)=\frac{\Delta}{z}\left(\frac{g^{\prime}(z)}{g(z)}+\frac{\Delta}{z}\right)-\frac{m^{2}}{z^{4}g(z)},$
(30)
$Q(z)=-\frac{\log(z)\left(b\lambda^{2}r_{+}\left(z^{2}-1\right)-2\log(z)\right)}{2z^{4}g(z)^{2}}.$
(31)

We can consider the trial function $F(z)=1-\alpha z^{2}$ which satisfies the
required boundary conditions $F(0)=1$ and $F^{{}^{\prime}}(0)=0$. Then, the
eigenvalue problem could be solved for (28) by minimizing the expression

$\lambda^{2}=\frac{\int_{0}^{1}T\left(F^{\prime
2}-PF^{2}\right)dz}{\int_{0}^{1}TQF^{2}dz},$ (32)

with respect to $\alpha$. For backreacting parameter, we could use the
iteration method and define LPJW2015

$\kappa_{n}=n\Delta\kappa,\ \ \ n=0,1,2,\cdots,$ (33)

where $\Delta\kappa=\kappa_{n+1}-\kappa_{n}$. Here, we take
$\Delta\kappa=0.05$. Since we are interested in finding the effects of
nonlinearity on backreaction up to the order $\kappa^{2}$, we have

$\kappa^{2}\lambda^{2}={\kappa_{n}}^{2}\lambda^{2}={\kappa_{n}}^{2}(\lambda^{2}|_{\kappa_{n-1}})+O[(\Delta\kappa)^{4}],$
(34)

where we take $\kappa_{-1}=0$ and $\lambda^{2}|_{\kappa_{-1}}=0$. We shall
also retain the linear terms with respect to nonlinearity parameter $b$ and
therefore,

$b\lambda^{2}=b\left(\lambda^{2}|_{b=0}\right)+\mathcal{O}(b^{2}).$ (35)

Then, the minimum eigenvalue of Eq. (32) can be obtained. At the critical
point, temperature is defined as (see Eq. (8) and note that at zeroth order
with respect to $\epsilon$, $\chi$ is zero.)

$T_{c}=\frac{f^{\prime}\left(r_{+c}\right)}{4\pi}.$ (36)

Using Eqs. (LABEL:fr) and (24), we receive

$f^{\prime}\left(r_{+c}\right)=2r_{+c}+\frac{2\kappa^{2}r_{+c}}{b}\left[1-\frac{1}{\sqrt{1-b\phi^{\prime}\left(r_{+c}\right){}^{2}}}\right],$
(37)

and thus

$\displaystyle T_{c}$ $\displaystyle=$
$\displaystyle\frac{1}{4\pi}(\frac{\mu}{\lambda})[2-\kappa_{n}^{2}(\lambda^{2}|_{\kappa_{n-1}})$
(38)
$\displaystyle-\frac{3}{4}b\kappa_{n}^{2}(\lambda^{4}|_{\kappa_{n-1},b=0})+b\kappa_{n}^{2}(\lambda^{4}|_{\kappa_{n-1},b=0})].$

As an example, if $b=\kappa^{2}=0$ we have

$\lambda^{2}=\dfrac{\frac{2}{3}\alpha^{2}-\frac{4}{3}\alpha+1}{-\frac{251\alpha^{2}}{864}+\frac{9\alpha}{16}+\frac{\alpha^{2}\zeta(3)}{4}-\frac{\alpha\zeta(3)}{2}+\frac{\zeta(3)}{4}-\frac{1}{4}}.$

Inserting $\alpha=0.759$, $\lambda_{min}^{2}=13.76$ and $T_{c}=0.429\mu$. The
latter result perfectly agrees with ones in Li (2012).

The values of $T_{c}/\mu$ for different backreaction and nonlinearity
parameters are listed in 1. As it shows, the effect of increasing the
backreaction parameter $\kappa$ for a fixed value of nonlinearity parameter
$b$ follows the same trend as raising $b$ for a fixed value of $\kappa$. In
both cases, the critical temperature $T_{c}$ diminishes by growing the
backreaction or nonlinearity parameters. It shows that the presence of
backreaction and Born-Infeld nonlinear electrodynamics make the scalar hair
harder to form. In next section, we will re-study the problem numerically
using the shooting method.

##  IV Shooting method

In this section, we will study our holographic superconductor numerically. In
order to do this, we use the shooting method Hartnoll (2009). In this method,
the boundary values is found by setting appropriate initial conditions. So,
for doing this, we need to know the behavior of equations of motion both at
horizon and boundary. Using Taylor expansion at horizon around $z=1$, we get

$\displaystyle f(z)=f_{1}\left(1-z\right)+f_{2}\left(1-z\right){}^{2}+\cdots,$
(39)
$\displaystyle\phi(z)=\phi_{1}\left(1-z\right)+\phi_{2}\left(1-z\right){}^{2}+\cdots,$
(40)
$\displaystyle\psi(z)=\psi_{0}+\psi_{1}\left(1-z\right)+\psi_{2}\left(1-z\right){}^{2}+\cdots,$
(41)
$\displaystyle\chi(z)=\chi_{0}+\chi_{1}\left(1-z\right)+\chi_{2}\left(1-z\right){}^{2}+\cdots.$
(42)

Note that $\phi=0$ at horizon, otherwise it will be ill-defined there. In our
procedure, we find all coefficients in terms of $\phi_{1}$, $\psi_{0}$ and
$\chi_{0}$ by using equations of motion. Varying them at the horizon, we try
to get $\psi_{-}=\chi=0$ at the boundary. So, the values of $\psi_{+}$ and
$\mu$ are achieved. In addition, we will set $r_{+}=1$ by virtue of the
equations of motion’s symmetry

$r\rightarrow ar,\text{ \ \ \ \ }f\rightarrow a^{2}f,\text{ \ \ \ \
}\phi\rightarrow a\phi.$

Performing numerical solution, we can find the values of $T_{c}/\mu$ for
different backreaction and nonlinearity parameters. In order to compare the
latter results with analytical ones, we listed both of them next to each other
in table 1. It is obvious that there is a reasonable agreement between the
results of both methods. Moreover, in table 1, the results of Wang  for $b=0$
has been recovered for different values of backreaction parameter. As one
could see in this table, increasing the backreaction parameter for a fixed
value of $b$, decreases the critical temperature. This means that the larger
values of backreaction parameter makes the condensation harder to form.
Similarly, for a fixed value of $\kappa$, increasing the nonlinearity of
electrodynamic model makes scalar hair harder to form because it diminishes
the critical temperature.

Figs. 1 and 2 give information about the effect of backreaction and nonlinear
electrodynamic on condensation. All curves follow a same trend. As
$b\rightarrow 0$, we regain the results of Maxwell case presented in Wang . As
figures show, the condensation gap increases by making backreaction and
nonlinearity parameters larger while the other one is fixed. So, it can be
understood that it is harder to form a superconductor. This is in agreement
with the results obtained from the behavior of critical temperature before.

##  V Summary and discussion

In this work, by using the Sturm-Liouville eigenvalue problem, we analytically
investigated the properties of $(1+1)$-dimensional holographic superconductor
developed in BTZ black hole background in the presence of BI nonlinear
electrodynamics. We have relaxed the probe limit and further assumed that the
gauge and scalar fields do backreact on the background metric. We determined
the critical temperature for different values of backreaction and nonlinear
parameters. We have also continued our study by using the numerical shooting
method and confirmed that the analytical results are in agreement with the
numerical approach. We observed that the formation of the scalar hair is
harder in the presence of BI nonlinear electrodynamics as well as backreaction
and it becomes harder and harder to form by increasing the strength of either
the nonlinear and backreaction parameters.

Finally, it would be of interest to extend this procedure for other nonlinear
electrodynamics like Power-Maxwell and logarithmic cases and investigate the
effects of nonlinear electrodynamics on the critical temperature and
condensation operator of one dimensional holographic superconductors. These
issues are now under investigations and the results will be appeared
elsewhere.

###### Acknowledgements.

We thank referee for constructive comments which helped us improve our paper
significantly. We also thank Shiraz University Research Council. The work of
AS has been supported financially by Research Institute for Astronomy and
Astrophysics of Maragha (RIAAM), Iran. MKZ would like to thank Shahid Chamran
University of Ahvaz, Iran.

## References

  * BCS (1957) J. Bardeen, L. N. Cooper, J. R. Schrieer, Phys. Rev. 108, 1175 (1957).
  * Maldacena. (1998) J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231 (1998) [hep-th/9711200v3 ]
  * Gubser et al. (1998) S. S. Gubser, I. R. Klebanov and A. M. Polyakov, Phys. Lett. B 428, 105 (1998) [hep-th/9802109].
  * Witten et al. (1998) E. Witten, Adv. Theor. Math. Phys. 2, 253 (1998) [hep-th/9802150].
  * (5) S. A. Hartnoll, C. P. Herzog and G. T. Horowitz, Phys. Rev. Lett. 101, 031601 (2008) [arXiv:0803.3295].
  * Horowitz et al. (2008) G. T. Horowitz and M. M. Roberts, Phys. Rev. D 78, 126008 (2008).
  * Ren. (2010) J. Ren, JHEP. 1011, 055 (2010) [arXiv:1008.3904].
  * Hartnoll (2009) S. A. Hartnoll, Class. Quantum Grav. 26, 224002 (2009) [arXiv:0903.3246].
  * Herzog. (2009) C. P. Herzog, J. Phys. A 42, 343001 (2009) [arXiv:0904.1975].
  * Horowitz (2011) G. T. Horowitz, Lect. Notes Phys. 828, 313 (2011) [arXiv:1002.1722].
  * Gubser. (2009) S. S. Gubser, C. P. Herzog, S. S. Pufu and T. Tesileanu, Phys. Rev. Lett. 103, 141601 (2009) [arXiv:0907.3510].
  * HHH. (2008) S. A. Hartnoll, C. P. Herzog and G. T. Horowitz, JHEP 0812, 015 (2008) [arXiv:0810.1563].
  * Jing ,Chena (210) J. Jing, S. Chen, Phys. Lett. B 686, 68 (2010) [arXiv:1001.4227].
  * cai (2015) R. G. Cai, L. Li, Li-Fang Li, Run-Qiu Yang, Sci China Phys. Mech. Astron. 58, 060401 (2015) [arXiv:1502.00437].
  * Ge (2010) X. H. Ge, B. Wang, S. F. Wu, and G. H. Yang, JHEP 1008, 108 (2010) [arXiv:1002.4901];
X. H. Ge, S. F. Tu, B. Wang, JHEP 09, 088 (2012) [arXiv:1209.4272];
X. M. Kuang, E. Papantonopoulos, G. Siopsis, B. Wang, Phys. Rev. D 88, 086008
(2013) [arXiv:1303.2575];
Q. Pan, J. Jing, B. Wang, JHEP 11, 088 (2011) [arXiv:1105.6153].

  * (16) M. Kord Zangeneh, Y. C. Ong, B. Wang, Phys. Lett. B 771, 235 (2017) [arXiv:1704.00557].
  * CAI (11) R. G. Cai, H. F Li, H.Q. Zhang, Phys. Rev. D 83, 126007 (2011).
  * cai (10) R. G. Cai, Z.Y. Nie, H.Q. Zhang, Phys. Rev. D 82, 066007 (2010).
  * cai (14) R. G. Cai, L. Li, L. F. Li, JHEP 1401, 032 (2014) [arXiv:1309.4877].
  * yao (2013) W. Yao, J. Jing, JHEP 1305, 101 (2013) [arXiv:1306.0064].
  * (21) Z. Zhao, Q. Pan, S. Chen and J. Jing, Nucl. Phys. B 871, 98 (2013) [arXiv:1212.6693].
  * (22) Y. Liu, Y. Gong and B. Wang, JHEP 1602, 116 (2016) [arXiv:1505.03603].
  * (23) S. Gangopadhyay, D. Roychowdhury, JHEP 05, 002 (2012) [arXiv:1201.6520];
S. Gangopadhyay and D. Roychowdhury, JHEP 05, 156 (2012) [arXiv:1204.0673].

  * Sheykhi et al. (2016) A. Sheykhi, H. R. Salahi, A. Montakhab, JHEP 1604, 058 (2016) [arXiv:1603.00075].
  * Salahi et al. (2016) H. R. Salahi, A. Sheykhi, A. Montakhab, Eur. Phys. J. C 76, 575 (2016) [arXiv:1608.05025].
  * SHsh (17) A. Sheykhi, F. Shaker, International Journal of Modern Physics D 26, 1750050 (2017) [arXiv:1606.04364];
A.Sheykhi, A. Ghazanfari, A. Dehyadegari, Eur. Phys. J. C 78, 159 (2018).

  * SHSH (16) A. Sheykhi, F. Shaker, Canadian J. of Phys. 94, 1372 (2016) [arXiv:1601.05817].
  * (28) A. Sheykhi, F. Shaker, Phys Lett. B 754, 281 (2016) [arXiv:1601.04035].
  * (29) A. Sheykhi, D. Hashemi Asl, A. Dehyadegari, Phys. Lett. B 781, 139 (2018) [arXiv:1803.05724].
  * (30) A. Sheykhi, A. Ghazanfari, A. Dehyadegari, Eur. Phys. J. C 78, 159 (2018) [arXiv:1712.04331].
  * (31) M. Kord Zangeneh, S. S. Hashemi, A. Dehyadegari, A. Sheykhi and B. Wang, [arXiv:1710.10162].
  * (32) S. I. Kruglov, arXiv:1801.06905.
  * (33) S. Carlip, Class. Quant. Grav. 12, 2853 (1995) [gr-qc/9506079].
  * (34) A. Ashtekar, J. Wisniewski and O. Dreyer, Adv. Theor. Math. Phys. 6, 507 (2002) [gr-qc/0206024].
  * (35) T. Sarkar, G. Sengupta and B. Nath Tiwari, JHEP 0611, 015 (2006) [hep-th/0606084].
  * (36) E. Witten, Adv. Theor. Math. Phys. 2, 505 (1998) [hep-th/9803131].
  * (37) S. Carlip, Class. Quant. Grav. 22, R85 (2005) [gr-qc/0503022].
  * (38) D. Birmingham, I. Sachs and S.N. Solodukhin, Phys. Rev. Lett. 88, 151301 (2002) [hep-th/0112055].
  * (39) E. Witten, arXiv:0706.3359.
  * (40) G. Panotopoulos and A. Rincon, Phys. Rev. D 97, 085014 (2018) [arXiv:1804.04684].
  * (41) A. Rincon and G. Panotopoulos, Phys. Rev. D 97, 024027 (2018) [arXiv:1801.03248].
  * (42) G. Panotopoulos and A. Rincon, Int. J. Mod. Phys. D 27, 1850034 (2018) [arXiv:1711.04146 ].
  * (43) A. Rincon, E. Contreras, P. Bargueño, B. Koch, G. Panotopoulos and A. Hernández-Arboleda, Eur. Phys. J. C 77, 494 (2017) [arXiv:1704.04845].
  * (44) P. Chaturvedi, G. Sengupta, Phys. Rev. D 90, 046002 (2014) [arXiv:1310.5128].
  * Li (2012) R. Li, Mod. Phys. Lett. A. 27, 1250001 (2012).
  * (46) D. Momeni, M. Raza, M. R. Setare, R. Myrzakulov, Int. J. Theor. Phys. 52, 2773 (2013) [arXiv:1305.5163].
  * (47) Y. Peng, G. liu, J. Mod. Phys. A 32 (2017).
  * (48) Y. Liu, Q. Pan and B. Wang, Phys. Lett. B 702, 94 (2011) [arXiv:1106.4353].
  * (49) N. Lashkari, JHEP 1111, 104 (2011) [arXiv:1011.3520].
  * (50) H. B. Zeng, arXiv:1204.5325.
  * (51) Y. Bu, Phys. Rev. D 86, 106005 (2012) [arXiv:1205.1614].
  * (52) Y. Peng, arXiv:1604.06990.
  * (53) G. Alkac, S. Chakrabortty, P. Chaturvedi, Phys. Rev. D 96, 086001 (2017) [arXiv:1610.08757].
  * (54) M. Kord Zangeneh, Y. C. Ong and B. Wang, Phys. Lett. B 771, 235 (2017) [arXiv:1704.00557].
  * (55) B. Binaei Ghotbabadi, M. Kord Zangeneh and A. Sheykhi, Eur. Phys. J. C 78, 381 (2018) [arXiv:1804.05442].
  * (56) M. Born and L. Infeld, Proc. R. Soc. A 144, 425 (1934).
  * (57) B. Hoffmann, Phys. Rev. 47, 877 (1935).
  * (58) W. Heisenberg and H. Euler, Phys. 98, 714 (1936) [arXiv:0605038].
  * (59) H. P. de Oliveira, Class. Quant. Grav. 11, 1469 (1994).
  * (60) G. W. Gibbons and D. A. Rasheed, Nucl. Phys. B 454, 185 (1995) [hep-th/9506035].
  * (61) A. Sheykhi, A. Kazemi, Phys. Rev. D 90 (2014) 044028\.
  * (62) D. Ghorai and S. Gangopadhyay, Eur. Phys. J. C 76, 146 (2016) [arXiv:1511.02444].
  * TL (2000) T. L. Chow, Mathematical Methods for physicists-A Concise Introduction, Cambridge University Press (2000).
  * (64) C. Lai, Q. Pan, J. Jing, and Y. Wang, Phys. Lett. B 749, 437 (2015) [arXiv:1508.05926].