# Grounding Foundation Models through Federated Transfer Learning: A General
Framework

Yan Kang<EMAIL_ADDRESS>WeBankChina ,  Tao Fan<EMAIL_ADDRESS>WeBank
and Hong Kong University of Science and TechnologyChina ,  Hanlin Gu
WeBankChina ,  Xiaojin Zhang  Huazhong University of Science and
TechnologyChina ,  Lixin Fan  WeBankChina and  Qiang Yang  WeBank and Hong
Kong University of Science and TechnologyChina

(2018)

###### Abstract.

Foundation Models (FMs) such as GPT-4 encoded with vast knowledge and powerful
emergent abilities have achieved remarkable success in various natural
language processing and computer vision tasks. Grounding FMs by adapting them
to domain-specific tasks or augmenting them with domain-specific knowledge
enables us to exploit the full potential of FMs. However, grounding FMs faces
several challenges, stemming primarily from constrained computing resources,
data privacy, model heterogeneity, and model ownership. Federated Transfer
Learning (FTL), the combination of federated learning and transfer learning,
provides promising solutions to address these challenges. Recently, the need
for grounding FMs leveraging FTL, coined FTL-FM, has arisen strongly in both
academia and industry. Motivated by the strong growth in FTL-FM research and
the potential impact of FTL-FM on industrial applications, we propose an FTL-
FM framework that formulates problems of grounding FMs in the federated
learning setting, construct a detailed taxonomy based on the FTL-FM framework
to categorize state-of-the-art FTL-FM works, and comprehensively overview FTL-
FM works based on the proposed taxonomy. We also establish correspondence
between FTL-FM and conventional phases of adapting FM so that FM practitioners
can align their research works with FTL-FM. In addition, we overview advanced
efficiency-improving and privacy-preserving techniques because efficiency and
privacy are critical concerns in FTL-FM. Last, we discuss opportunities and
future research directions of FTL-FM.

Federated Learning, Transfer Learning, Foundation Model, Privacy

††copyright: acmcopyright††journalyear: 2018††journal: JACM††journalvolume:
37††journalnumber: 4††article: 111††publicationmonth: 8

##  1\. Introduction

In recent years, the field of artificial intelligence (AI) has undergone a
significant transformation with the advent of Foundation Models (FMs)
(Bommasani et al., 2021). Advanced FMs, such as GPT-4 (OpenAI, 2023), PaLM
(Chowdhery et al., 2022) and LLaMA (Touvron et al., 2023) boasting billions of
parameters, have drawn considerable attention due to their remarkable
performance in various AI tasks ranging from natural language processing,
content generation to more complex tasks such as planning and reasoning.

While FMs have shown great success in various general natural language
processing (OpenAI, 2023; Liu et al., 2023c; Kamalloo et al., 2023) and
computer vision tasks (Kirillov et al., 2023; Rombach et al., 2022), FMs have
limitations that prevent them from being adopted to domain-specific
applications: (1) FMs typically are pre-trained on publicly available datasets
and lack industry-level and domain-specific knowledge, implying the deficiency
of applying FMs to applications of various industrialized domains. (2) FMs are
trained only up to a certain point in time, and thus, their knowledge is often
outdated, thereby making it challenging to apply FMs to scenarios that require
real-time updating of knowledge, e.g., in Newspaper publications. To make FMs
really excel in addressing challenging domain-specific problems, we need to
ground FMs by integrating them with industry-level and domain-specific
knowledge. Nevertheless, some powerful FMs are closed-source (e.g., GPT-4 and
PaLM). Valuable domain knowledge is often distributed across different
enterprises, organizations, or edge devices, stored in their local data
warehouses, or encoded in private machine learning models. Consequently,
grounding FMs requires transferring knowledge between FM providers and domain
knowledge holders. We refer to the former as the ”server” and the latter as
the ”client,” as depicted in Figure 1. In addition, domain knowledge often
encompasses sensitive and confidential information, and companies or edge
devices may have limited computational resources. These circumstances present
challenges for grounding FMs:

Figure 1. High-level illustration of knowledge transferring between a server
hosting foundation model and a client with domain knowledge (in the form of
domain models and data).

  * •

Constrained Resources. FMs comprise a colossal amount of parameters and thus
require massive amounts of computational resources to train, which prevents
them from being deployed in companies or edge devices with constrained storage
and computing resources.

  * •

Data Privacy. Clients with constrained computing resources (e.g., small-sized
companies and edge devices) can send their data to the FM server (e.g.,
OpenAI) to fine-tune the FM for them. However, sharing private data with the
FM server may compromise data privacy.

  * •

Model Ownership. The FM server can send the full FM model to clients for
further fine-tuning. However, directly sharing private FMs threatens the
ownership of the FMs.

  * •

Model Heterogeneity. The models held by the server and clients often have
heterogeneous architectures and sizes. Thus, advanced knowledge transfer
techniques are required to co-train models of all parties than the traditional
FedAvg algorithm (McMahan et al., 2017).

Federated Transfer Learning (FTL) provides promising solutions to address
these obstacles for grounding FMs. We refer to the concept of grounding FMs
through FTL techniques as FTL-FM. Specifically, FTL-FM provides servers of FMs
and clients of DMs with privacy and security defense tools (Li et al., 2023c),
federated learning algorithms (Yang et al., 2019; McMahan et al., 2017), and
transfer learning approaches (Pan and Yang, 2010) to collaboratively adapt FMs
to domain-specific models or augment FMs with domain-specific knowledge. In
recent years, a considerable amount of methods across the spectrum of FTL-FM
have been proposed to ground FMs. Subsequently, several survey and vision
papers were proposed to overview these works, summarize application scenarios,
and discuss potential challenges and solutions (Gou et al., 2021; Ling et al.,
2023; Zhu et al., 2023a; Zhuang et al., 2023; Chen et al., 2023). However,
these works either focus on specific techniques of customizing FMs without
considering distributed scenarios of knowledge transfer (Gou et al., 2021;
Ling et al., 2023; Zhu et al., 2023a) or provide high-level overview of
federated LLM scenarios, challenges, and potential solutions (Zhuang et al.,
2023; Chen et al., 2023) without formulating FTL-FM problems and
systematically reviewing relevant FTL-FM works. To fill these gaps, we propose
an FTL-FM framework that formulates the problems of grounding FMs in the
federated learning setting. Based on the FTL-FM framework, we systematically
review state-of-the-art FTL-FM works. The main contributions of this paper
include:

  * •

A general FTL-FM framework of grounding FMs. We first establish high-level
correspondence between FTL-FM and conventional phases of adapting FM. We then
propose an FTL-FM framework that formulates three settings of grounding FMs in
federated learning (see Definition 3.1). A broad range of state-of-the-art
FTL-FM works can be reduced to these FTL-FM formulations, which serve as the
basis for our proposed taxonomy of FTL-FM works.

  * •

A detailed taxonomy to categorize state-of-the-art FTL-FM works. We construct
the taxonomy (see Table 2) based on the three FTL-FM settings (see Definition
3.1). In each setting, the taxonomy is further formed based on specific
research issues, including what knowledge to transfer, how to transfer the
knowledge, what information to protect, and how to protect the information.
The FTL-FM framework, together with this taxonomy, can serve as guidance for
future FTL-FM works.

  * •

A systematic overview of state-of-the-art FTL-FM works. We systematically
review state-of-the-art FTL-FM works based on the proposed taxonomy. We also
discuss the relationship between reviewed FTL-FM works and the formulations
defined in the FTL-FM framework. In addition, we overview advanced efficiency-
improving and privacy-preserving methods since efficiency and privacy are
critical concerns of FTL-FM. In this paper, our main focus is to review FTL-FM
works that encompass large language models (LLM) with sizes no smaller than
$\text{BERT}_{\text{base}}$. Nevertheless, we also take into account
representative FTL-FM works that are dedicated to other types of models, such
as large vision and speech models.

  * •

A discussion on future research directions. Based on the comprehensive
investigation of application scenarios, knowledge transfer methods, and
privacy-preserving techniques adopted in existing FTL-FM works, we provide a
detailed discussion of the open opportunities and future directions of FTL-FM
research.

The rest of the paper is organized as follows. Section 2 overviews related
works. Section 3 defines a general framework of FTL-FM and Section 4 presents
a taxonomy to categorize state-of-the-art FTL-FM works. Moving forward,
Section 5, Section 6, and Section 7 comprehensively review FTL-FM works of
three settings, respectively. Subsequently, Section 9 and Section 10 examine
efficiency-improving and privacy-preserving methods, respectively. Section 11
discusses opportunities and future directions. Finally, Section 12 presents
the concluding remarks for this work. The outline of core sections of this
work is pictorially illustrated in Figure 2.

Figure 2.  Outline of core sections of this work.

##  2\. Related Work

In this section, we briefly review the existing survey and vision papers
related to grounding foundation models (FMs). These papers either focused on
techniques of compressing and customizing FMs without considering distributed
scenarios of augmenting and adapting FMs (Gou et al., 2021; Ling et al., 2023;
Zhu et al., 2023a), or broadly reviewed federated foundation models without
formulating FTL-FM problems and comprehensively reviewing relevant FTL-FM
works (Zhuang et al., 2023; Chen et al., 2023).

Gou et al. (2021) comprehensively reviewed knowledge distillation techniques,
which aim to reduce a large teacher model to a small student model without
jeopardizing model performance substantially, facilitating the deployment of
distilled but powerful deep models to resource-constrained entities, e.g.,
mobile phones and edge devices. More specifically, they reviewed knowledge
distillation techniques from a wide range of perspectives, including knowledge
categories, training schemes, teacher-student architecture, distillation
algorithms, performance comparison, and applications. Zhu et al. (2023a)
comprehensively reviewed model compression techniques tailored to large
language models (LLMs). They proposed a taxonomy of model compression methods
for LLMs, including pruning, knowledge distillation quantization, and low-rank
factorization. In addition, They exhaustively examined metrics and benchmarks
for evaluating compressed LLMs. While LLMs achieved remarkable performance
across various natural language processing and understanding tasks, they may
hallucinate because, among other reasons, they lack domain-specific knowledge.
Therefore, domain specification has become a pivotal research area for
customizing LLMs. Ling et al. (2023) proposed a systematic review,
categorization, and taxonomy of LLM domain specification techniques, including
external augmentation, prompt crafting, and model fine-tuning.

General-purpose FMs typically lack domain knowledge. Enhancing FMs with
industry-level or domain-specific knowledge requires a large amount of domain
data that are often dispersed among multiple private entities (e.g.,
corporations). Chen et al. (2023) proposed a concept of federated large
language models (LLMs) and presented a framework for training LLMs in a
federated manner. Technically, they focused on the federated learning
scenarios in which each client owns a local LLM, and all clients
collaboratively pre-train, fine-tune, or prompt engineer their LLMs. Zhuang et
al. (2023) provided an overview of the combination of FMs and FL. In addition
to discussing the motivation, challenges, and opportunities of federated FMs,
they reviewed the motivations, challenges, opportunities, and future
directions of leveraging FMs to facilitate FL.

##  3\. FTL-FM Framework

In this section, we propose a general framework of grounding foundation models
(FMs) through federated transfer learning (FTL), which we term FTL-FM. Before
delving into the formal definition of FTL-FM, we first establish the
correspondence between phases of FTL-FM and those of adapting FMs and between
FTL-FM and trustworthy federated learning.

Figure 3. The correspondence between phases of FTL-FM and those of adapting
FMs. We illustrate this correspondence with one server and one client.
However, multiple clients can be involved in FTL-FM.

Adapting an FM typically comprises three phases: pre-training, fine-tuning,
and prompting. Pre-training aims to train an FM from scratch based on broad
ranges of large-scale datasets so that the FM can be adaptable to a wide range
of downstream tasks. Fine-tuning aims to train a pre-trained FM with specific
goals. It can be instruction tuning that enables an FM to follow human’s
instructions for solving specific tasks (Ouyang et al., 2022), alignment
tuning that aligns an FM’s behaviors with human intentions and values (Ji et
al., 2023), or domain-specific tuning that augments the FM with domain-
specific or industry-level knowledge. Prompting is to query an FM with prompts
for solving various tasks.

FTL-FM aims to adapt knowledge of the FM to enhance the client’s domain model
(DM) or transfer knowledge from the client to augment the FM through federated
transfer learning. FTL-FM also involves three phases: pre-training, federated
transfer learning, and inference. Herein, we establish correspondence between
the phases of FTL-FM and those of adapting an FM, as illustrated in Figure 3.
This correspondence may help FM practitioners align their research works with
FTL-FM.

  * •

Pre-training: The server and the client may pre-train their local models
(i.e., FM and DM) using their respective local data. Note that the client’s DM
can also be an FM.

  * •

Federated Transfer Learning (FTL): FTL involves two directions. (1) The server
and the client collaboratively fine-tune the server’s FM based on the client’s
domain-specific knowledge. (2) The server and the client collaboratively
transfer and adapt general knowledge of the server’s FM to fine-tune the
client’s DM. While conducting FTL, the server and the client may fine-tune
their local models using their respective local data.

###### Remark 1.

While conducting FTL, the server and the client may train their local models
from scratch using their respective local data instead of fine-tuning pre-
trained models. For example, when the client does not have enough data or the
client’s model is unsuitable for pre-training, the client can train its local
model from scratch together with FTL.

###### Remark 2.

The two directions combined form a scenario where knowledge is transferred to
optimize the server’s FM and the client’s DM simultaneously. This scenario is
rarely explored in literature.

  * •

Inference: Upon the completion of federated transfer learning, end users can
query the FM of the server or the DM of the client to accomplish their
specific tasks. Note that the client may need to collaborate with the server
to complete the inference in some applications.

Figure 4. The correspondence between FTL-FM and trustworthy federated
learning.

FTL-FM adheres to the principles of trustworthy federated learning (TFL) (Kang
et al., 2023), as illustrated in Figure 4. To gain the trust of various
stakeholders, such as FL participants, users, and regulators, an FTL-FM
approach must simultaneously fulfill multiple objectives and optimize their
trade-offs. Among these objectives, optimizing utility, efficiency, and
privacy are the most critical ones to TFL.

###  3.1. The Definition of FTL-FM Framework

We formally define the framework of grounding foundation models through
federated transfer learning (FTL-FM) as follows.

Figure 5. Illustration of the three settings formulated in the FTL-FM
framework (see Definition 3.1). The objective of setting ① is transferring and
adapting knowledge of the server’s foundation model (FM) to clients’ domain
models (DMs), the objective of setting ② is leveraging clients’ domain-
specific knowledge to enhance the server’s FM, and the objective of setting ③
is co-optimizing both the server’s FM and clients’ DMs.

###### Definition 3.1 (FTL-FM).

Given a server with a dataset $\mathcal{D}^{S}$ and a foundation model (FM)
$\mathcal{W}^{S}_{\text{FM}}$ as well as $K$ clients that each client $k$ has
a private dataset $\mathcal{D}^{C_{k}}$ and a domain model (DM)
$\mathcal{W}^{C_{k}}$. $\mathcal{D}^{S}$ and $\mathcal{W}^{S}$ can be either
public or private. FTL-FM can be categorized into three settings, where each
setting aims to achieve a federated transfer learning objective through the
collaboration between the server and clients (illustrated in Figure 5):

  * •

Setting  1: Optimize clients’ DMs $\mathcal{W}^{C_{k}},k\in\\{1,...,K\\}$ by
leveraging knowledge of $\mathcal{D}^{S}$ and $\mathcal{W}^{S}_{\text{FM}}$.
This setting can involve two stages, which are formulated as follows. Note
that stage 1 is not a prerequisite for stage 2.

Stage 1: The server fine-tunes or compresses $\mathcal{W}^{S}_{\text{FM}}$
based on its data $\mathcal{D}^{S}$ while minimizing the distance between
distribution $\mathcal{P}^{S}$ of $\mathcal{D}^{S}$ and distribution
$\mathcal{P}^{C}$ of clients’ data and mitigating the private leakage
$\epsilon_{p}$ under a privacy constraint $\xi$:

(1)
$\begin{split}&\min_{\mathcal{W}^{S}_{\text{FM}}}\ell^{S}(\mathcal{W}^{S}_{\text{FM}})+\lambda\mathcal{M}(\mathcal{P}^{S},\mathcal{P}^{C})+\eta\epsilon_{p},\\\
&\text{ subject to}\quad\epsilon_{p}\leq\xi,\\\ &\text{ where
}\,\,\ell^{S}(\mathcal{W}^{S}_{\text{FM}})=\mathbb{E}_{d\sim\mathcal{D}^{S}}[\ell^{S}(\mathcal{W}^{S}_{\text{FM}};d)].\end{split}$

in which $\ell^{S}(\mathcal{W}^{S}_{\text{FM}})$ is the expected loss over
dataset $\mathcal{D}^{S}$ for fine-tuning or compressing
$\mathcal{W}^{S}_{\text{FM}}$; $\mathcal{M}$ is a distribution distance
metric; $\lambda$ and $\eta$ are preferences toward the distribution distance
and private leakage, respectively. The resulting model
$\widetilde{\mathcal{W}}^{S}_{\text{FM}}$ fine-tuned or compressed by Eq.(1)
is then used to facilitate clients to train their DMs through federated
learning or local training.

Stage 2: Transfer and adapt the knowledge of $\mathcal{W}^{S}_{\text{FM}}$ (or
$\widetilde{\mathcal{W}}^{S}_{\text{FM}}$ from stage 1) to help train clients’
DMs for their specific tasks while mitigating the private leakage
$\epsilon_{p}$ under a privacy constraint $\xi$.

(2)
$\begin{split}&\min_{\mathcal{W}^{C_{1}},\dots,\mathcal{W}^{C_{K}}}\mathcal{L}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}},\dots,\mathcal{W}^{C_{K}})=\sum^{K}_{k=1}p^{C_{k}}\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})+\eta\epsilon_{p},\\\
&\text{ subject to}\quad\epsilon_{p}\leq\xi,\\\ &\text{ where
}\,\,\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})=\mathbb{E}_{d\sim\mathcal{D}^{C_{k}}}[\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)],k\in\\{1,...,K\\}\end{split}$

in which $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})$ is
the expected loss over the dataset $\mathcal{D}^{C_{k}}$, aiming to transfer
and adapt the general knowledge and abilities of $\mathcal{W}^{S}_{\text{FM}}$
to client $k$’s DM $\mathcal{W}^{C_{k}}$ for a domain-specific task;
$p^{C_{k}}$ and $\eta$ are preferences toward the loss of client $C_{k}$ and
privacy leakage, respectively.

  * •

Setting  2: Optimize the server’s FM $\mathcal{W}^{S}_{\text{FM}}$ by
leveraging clients’ domain-specific knowledge of $\mathcal{D}^{C_{k}}$ and
$\mathcal{W}^{C_{k}}$$,k\in\\{1,...,K\\}$ while mitigating the private leakage
$\epsilon_{p}$ under a privacy constraint $\xi$. The main objective is
formulated as follows:

(3)
$\begin{split}&\min_{\mathcal{W}^{S}_{\text{FM}}}\mathcal{L}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}},\dots,\mathcal{W}^{C_{K}})=\sum^{K}_{k=1}p^{C_{k}}\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})+p^{S}\ell^{S}(\mathcal{W}^{S}_{\text{FM}})+\eta\epsilon_{p},\\\
&\text{ subject to}\quad\epsilon_{p}\leq\xi,\\\ &\text{ where
}\,\,\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})=\mathbb{E}_{d\sim\mathcal{D}^{C_{k}}}[\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)],k\in\\{1,...,K\\}\text{
and optionally}\\\
&\quad\quad\quad\,\ell^{S}(\mathcal{W}^{S}_{\text{FM}})=\mathbb{E}_{d\sim\mathcal{D}^{S}}[\ell^{S}(\mathcal{W}^{S}_{\text{FM}};d)]\end{split}$

in which $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})$ is
the expected loss over the dataset $\mathcal{D}^{C_{k}}$, aiming to transfer
domain-specific knowledge of $\mathcal{W}^{C_{k}}$ to enhance
$\mathcal{W}^{S}_{\text{FM}}$; $\ell^{S}(\mathcal{W}^{S}_{\text{FM}})$ is the
loss over dataset $\mathcal{D}^{S}$, aiming to train
$\mathcal{W}^{S}_{\text{FM}}$ for the server’s task; $p^{S}$, $p^{C_{k}}$ and
$\eta$ are preferences toward the loss of the server, loss of client $C_{k}$
and privacy leakage, respectively.

  * •

Setting  3: Co-optimize server’s FM $\mathcal{W}^{S}$ and clients’ DMs
$\mathcal{W}^{C_{k}},k\in\\{1,...,K\\}$ by leveraging the knowledge of each
other while mitigating the private leakage $\epsilon_{p}$ under a privacy
constraint $\xi$. The main objective is formulated as follows:

(4)
$\begin{split}&\min_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}},\dots,\mathcal{W}^{C_{K}}}\mathcal{L}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}},\dots,\mathcal{W}^{C_{K}})=\sum^{K}_{k=1}p^{C_{k}}\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})+p^{S}\ell^{S}(\mathcal{W}^{S}_{\text{FM}})+\eta\epsilon_{p},\\\
&\quad\text{subject to}\quad\epsilon_{p}\leq\xi,\\\ &\quad\text{where
}\,\,\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})=\mathbb{E}_{d\sim\mathcal{D}^{C_{k}}}[\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)],k\in\\{1,...,K\\}\text{
and optionally}\\\
&\quad\quad\quad\quad\,\ell^{S}(\mathcal{W}^{S}_{\text{FM}})=\mathbb{E}_{d\sim\mathcal{D}^{S}}[\ell^{S}(\mathcal{W}^{S}_{\text{FM}};d)]\end{split}$

in which $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}})$ is
the expected loss over the dataset $\mathcal{D}^{C_{k}}$, aiming to transfer
knowledge between server and clients to optimize $\mathcal{W}^{S}_{\text{FM}}$
and $\mathcal{W}^{C_{k}}$; $\ell^{S}(\mathcal{W}^{S}_{\text{FM}})$ is the loss
over dataset $\mathcal{D}^{S}$, aiming to train $\mathcal{W}^{S}_{\text{FM}}$
for the server’s task; $p^{S}$, $p^{C_{k}}$ and $\eta$ are preferences toward
the loss of the server, loss of client $C_{k}$ and privacy leakage,
respectively.

During the federated transfer learning, the following requirements should be
satisfied: (1) The private data and models of the server and clients should
not be passed to each other in plain text; (2) the intermediate training and
inference results passed among FL parties should be protected; (3) the
efficiency of training and inference should be guaranteed.

###### Remark 3.

Objectives of settings ① and ② aim to optimize clients’ DMs
$\mathcal{W}^{C_{k}},k=1,\dots,K$ and the server’s FM
$\mathcal{W}^{S}_{\text{FM}}$, respectively. Toward this end, other models may
be trained as part of the optimization process. For instance, Objective ① may
entail training $\mathcal{W}^{S}_{\text{FM}}$, while objective ② often involve
training $\mathcal{W}^{C_{k}},k=1,\dots,K$. For the sake of clarity, these
factors are not included in the formulation.

###### Remark 4.

A special case of the objective formulated in Eq.(4) is that the server
provides no model and acts as an aggregation function. The DMs of clients
$1,...,K$ are initialized with FMs. Thus, the co-optimizing FMs and DMs is
reduced to horizontal federated learning (McMahan et al., 2017; Tan et al.,
2022) where clients collaboratively train a global FM. However, since FMs are
expensive to train, clients typically train a global proxy model or a global
Parameter-Efficient Fine-Tuning (PEFT) module $\mathcal{W}^{G}_{\text{a}}$
(e.g., adapter and LoRA) by minimizing their local loss
$\ell^{C_{k}}(\mathcal{W}^{G}_{\text{a}},\mathcal{W}^{C_{k}}_{\text{FM}})$
while keeping their local FMs $\mathcal{W}^{C_{k}}_{\text{FM}}$ frozen.

(5)
$\begin{split}&\min_{\mathcal{W}^{G}_{\text{a}}}\mathcal{L}(\mathcal{W}^{G}_{\text{a}},\mathcal{W}^{C_{1}}_{\text{FM}},\dots,\mathcal{W}^{C_{K}}_{\text{FM}})=\sum^{K}_{k=1}p^{C_{k}}\ell^{C_{k}}(\mathcal{W}^{G}_{\text{a}},\mathcal{W}^{C_{k}}_{\text{FM}})+\eta\epsilon_{p},\\\
&\text{subject to}\quad\epsilon_{p}\leq\xi,\\\ &\text{where
}\,\,\ell^{C_{k}}(\mathcal{W}^{G}_{\text{a}},\mathcal{W}^{C_{k}}_{\text{FM}})=\mathbb{E}_{d\sim\mathcal{D}^{C_{k}}}[\ell^{C_{k}}(\mathcal{W}^{G}_{\text{a}},\mathcal{W}^{C_{k}}_{\text{FM}};d)],k\in\\{1,...,K\\}\end{split}$

###### Remark 5.

While formulations (i.e., Eqs.(2),(3), (4), and (5)) are derived from
horizontal federated learning, they can be generalized to vertical federated
learning (VFL) (Yang et al., 2019; Liu et al., 2022) by replacing the weighted
average aggregation function with aggregation approaches dedicated to VFL
scenarios.

Definition 3.1 defines a general FTL-FM framework that formulates three
settings of grounding FMs through FTL. These formulations reconcile a broad
range of machine learning tasks, knowledge transfer approaches, and privacy
leakage measures. Next, we provide detailed formulations based on existing
works to illustrate each aspect.

###  3.2. Formulations of Machine Learning Tasks

Federated transfer learning can be applied to address any machine learning
tasks. Herein, we provide auto-regressive Language Modeling (LM) (Radford et
al., 2018), classification, and regression, which are widely adopted in real-
world applications, as examples.

Auto-regressive LM:
$\displaystyle\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};\mathcal{D}^{C_{k}})=-\sum_{i=1}^{N}\sum_{j=1}^{|x|}\text{log}(\mathbb{P}(x_{i,j}|x_{i,<j})$
$\displaystyle\text{ where
}\mathbb{P}(x_{i,j}|x_{i,<j})=h_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}}}(x_{i}),$
Classification:
$\displaystyle\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};\mathcal{D}^{C_{k}})=-\sum_{i=1}^{N}\sum_{j=1}^{M}y_{i,j}\text{log}(p_{i,j})\text{
where }p_{i}=h_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}}}(x_{i}),$
Regression:
$\displaystyle\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};\mathcal{D}^{C_{k}})=\sum_{i=1}^{N}(y_{i}-p_{i})^{2}\text{
where }p_{i}=h_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}}}(x_{i}).$

where $M$ is the number of classes,
$\mathcal{D}^{C_{k}}=\\{x_{i},y_{i}\\}_{i=1}^{N}$ is the dataset of client
$k$, a sentence $x_{i}=\\{x_{i,0},x_{i,1},...,x_{i,|x_{i}|-1}\\}$, and $y_{i}$
is the ground truth label;
$h_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}}}(\cdot)$ parameterized by
$\mathcal{W}^{S}_{\text{FM}}$ and $\mathcal{W}^{C_{k}}$ denotes the general
federated transfer learning procedure between the server and client $k$, and
$p_{i}$ is the prediction of
$h_{\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}}}$ given $x_{i}$.

###  3.3. Formulations of Knowledge Transfer Approaches.

Various knowledge transfer approaches have been proposed to achieve each of
the three objectives defined in the FTL-FM framework (see Definition 3.1). We
overview and formulate representative approaches. These formulations are
summarized in Table 1. To better illustrate these formulations, we denote the
FM hosted by the server as $f_{\mathcal{W}^{S}_{\text{FM}}}(\cdot)$
parameterized by $\mathcal{W}^{S}_{\text{FM}}$, the DM owned by client $k$ as
$g_{\mathcal{W}^{C_{k}}}(\cdot)$ parameterized by $\mathcal{W}^{C_{k}}$, the
data owned by the server as $\mathcal{D}^{S}=\\{x^{S},y^{S}\\}$, and the data
owned by client $k$ as $\mathcal{D}^{k}=\\{x^{C_{k}},y^{C_{k}}\\}$.

Table 1. An overview of formulations of representative federated knowledge transfer approaches. Objective |  Formulation of Specific Federated Transfer Learning Loss $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};\\{x^{C_{k}},y^{C_{k}}\\})$ |  Knowledge Transfer Approach |  Representative Work
---|---|---|---
Optimize Domain Models $\mathcal{W}^{C_{k}}$ $k=1,\dots,K$ |  $\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}})\text{ where }$ $\mathcal{W}^{C_{k}}=\operatorname*{arg\,min}_{\mathcal{W}^{S}_{\text{SM}}}\ell^{S}_{\text{KD}}(f^{\prime}_{\mathcal{W}_{\text{SM}}^{S}}(x^{S}),f_{\mathcal{W}^{S}_{\text{FM}}}(x^{S}))$ \+ $\lambda\mathcal{M}(\mathcal{P}^{S},\mathcal{P}^{C})$ | domain adaptation | (Wang et al., 2023b; Hou et al., 2023b)
$\ell_{\text{TA}}^{C_{k}}(f_{\mathcal{W}^{S}_{\text{FM}}}([x^{C_{k}},g_{\mathcal{W}^{C_{k}}}(x^{C_{k}})]),y^{C_{k}})$ |  federated prompt tuning |  (Sun et al., 2022b; Diao et al., 2023; Li et al., 2023b) (Lin et al., 2023; Sun et al., 2023)
$\ell_{\text{TA}}^{C_{k}}(h_{\mathcal{W}^{\diamond}_{\text{HE}}}\circ f_{\mathcal{W}^{S}_{\text{BA}}}\circ g_{\mathcal{W}^{C_{k}}_{\text{BO}}}(x^{C_{k}}),y^{C_{k}})$ where $h_{\mathcal{W}^{\diamond}_{\text{HE}}},\diamond\in\\{S,C_{k}\\}$, $f_{\mathcal{W}^{S}_{\text{BA}}}$, and $g_{\mathcal{W}^{C_{k}}_{\text{BO}}}$ are the head, backbone, and bottom models split from the FM $f_{\mathcal{W}^{S}_{\text{FM}}}$ |  federated split learning |  (Zhou et al., 2023; Shen et al., 2023; Li et al., 2023c) (Xu et al., 2023c; Tian et al., 2022)
$\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}})+\ell_{\text{KD}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(z^{C_{k}});z^{S})$ $\text{ where }z^{S}=f_{\mathcal{W}^{S}_{\text{FM}}}(z^{C_{k}})$ |  federated knowledge distillation | (He et al., 2020)
Optimize Foundation Model $\mathcal{W}^{S}_{\text{FM}}$ |  $\ell_{\text{TA}}^{C_{k}}(\phi_{\mathcal{W}^{S}_{a}}\circ\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}\circ\psi_{\mathcal{W}^{S}_{b}}(x^{C_{k}}),y^{C_{k}})$ where $\phi_{\mathcal{W}^{S}_{a}}$ and $\psi_{\mathcal{W}^{S}_{b}}$ are adapters selected from $f_{\mathcal{W}^{S}_{\text{FM}}}$, and $\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}$ is emulator compressed from $f_{\mathcal{W}^{S}_{\text{FM}}}$ |  federated offsite tuning |  (Xiao et al., 2023; Fan et al., 2023; WeBank, [n. d.]) (Chua et al., 2023; Khalid et al., 2023)
$\ell_{\text{KD}}^{S}(f_{\mathcal{W}^{S}_{\text{FM}}}(x^{S}),z^{C_{k}})\text{ where }z^{C_{k}}=g_{\mathcal{W}^{C_{k}}}(x^{S})$ |  federated knowledge distillation |  (Yu et al., 2023b)
Optimize $\mathcal{W}^{S}_{\text{FM}}$ and $\mathcal{W}^{C_{k}}$, $k=1,\dots,K$ |  $\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}})\quad\quad\quad\quad\quad\quad\quad\quad$ $\quad\quad\quad\quad+\ell_{\text{FT}}(f_{\mathcal{W}_{\text{FM}}^{S}}(x^{a}),y^{a})+\ell_{\text{FT}}(g_{\mathcal{W}^{C_{k}}}(x^{b}),y^{b})$ where $\\{x^{a},y^{a}\\}$ and $\\{x^{b},y^{b}\\}$ are fine-tuning data and their specific forms are algorithm-dependent. |  federated co-optimizing |  (Fan et al., 2023; Deng et al., 2023)
$\ell_{\text{TA}}^{C_{k}}(\phi_{\mathcal{W}^{G}_{a}}\oplus g_{\mathcal{W}^{C_{k}}_{\text{FM}}}(x^{C_{k}}),y^{C_{k}})$ where $\phi_{\mathcal{W}^{G}_{a}}$ is a global PEFT module, and $g_{\mathcal{W}^{C_{k}}_{\text{FM}}}$ is the local FM and is frozen during training |  federated parameter- efficient fine-tuning |  (Zhang et al., 2023d; Zhao et al., 2023b) (Cai et al., 2023b; Zhang et al., 2023c)

To achieve the objective of setting ①, a straightforward way is through domain
adaptation: the server compresses the FM into a relatively smaller model that
is adapted to the client’s task and sends the distilled model to the
downstream client for further fine-tuning (Mireshghallah et al., 2022; Yu et
al., 2022; Wang et al., 2023b). We formulate this approach as follows.

(6) domain adaptation:
$\displaystyle\min_{\mathcal{W}^{C_{k}}}\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}}),$
$\displaystyle\text{ where
}\mathcal{W}^{C_{k}}=\operatorname*{arg\,min}_{\mathcal{W}^{S}_{\text{SM}}}\ell^{S}_{\text{KD}}(f^{\prime}_{\mathcal{W}_{\text{SM}}^{S}}(x^{S}),f_{\mathcal{W}^{S}_{\text{FM}}}(x^{S}))+\lambda\mathcal{M}(\mathcal{P}^{S},\mathcal{P}^{C}).$

where $\ell^{S}_{\text{KD}}$ is the knowledge distillation loss for distilling
knowledge from the teacher model $f_{\mathcal{W}^{S}_{\text{FM}}}$ to the
student model $f^{\prime}_{\mathcal{W}_{\text{SM}}^{S}}$, which is used to
initialize downstream client $k$’s domain model $g_{\mathcal{W}^{C_{k}}}$;
$\ell_{\text{TA}}^{C_{k}}$ is the task loss for training
$g_{\mathcal{W}^{C_{k}}}$.

As a case in point, Wang et al. (2023b) proposed an approach that involves
sampling data from a public dataset that closely resembles the distribution of
private data owned by clients. These selected samples are utilized to distill
an FM into an on-device language model. Then, clients employ these distilled
models as initialization for subsequent federated training. Hou et al. (2023b)
proposed a similar approach, but they fine-tune an FM for clients instead of
distillation.

In addition to model transfer, the server can adapt the general knowledge of
the FM to the clients’ domain models by transferring data-level and
representation-level knowledge. Federated prompt tuning (Lin et al., 2023; Sun
et al., 2023), federated split learning (Li et al., 2023c; Tian et al., 2022)
and federated knowledge distillation (He et al., 2020) are three
representative approaches. We formulate their objectives of transferring
knowledge from the server’s FM to the client $k$’s DM as follows:

(7) federated prompt tuning:
$\displaystyle\min_{\mathcal{W}^{C_{k}}}\ell_{\text{TA}}^{C_{k}}(f_{\mathcal{W}^{S}_{\text{FM}}}([x^{C_{k}},g_{\mathcal{W}^{C_{k}}}(x^{C_{k}})]),y^{C_{k}}),$
(8) federated split learning:
$\displaystyle\min_{\mathcal{W}^{C_{k}}_{\text{HE}},\mathcal{W}^{C_{k}}_{\text{BO}}}\ell_{\text{TA}}^{C_{k}}(h_{\mathcal{W}^{\diamond}_{\text{HE}}}\circ
f_{\mathcal{W}^{S}_{\text{BA}}}\circ
g_{\mathcal{W}^{C_{k}}_{\text{BO}}}(x^{C_{k}}),y^{C_{k}}),$
$\displaystyle\text{ where }\diamond\in\\{S,C_{k}\\},$ (9) federated knowledge
distillation:
$\displaystyle\min_{\mathcal{W}^{C_{k}}}\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}})+\ell_{\text{KD}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(z^{C_{k}}),z^{S})$
$\displaystyle\text{ where }z^{S}=f_{\mathcal{W}^{S}_{\text{FM}}}(z^{C_{k}}).$

where $h_{\mathcal{W}^{\diamond}_{\text{HE}}}$,
$f_{\mathcal{W}^{S}_{\text{BA}}}$, and $g_{\mathcal{W}^{C_{k}}_{\text{BO}}}$
are the head, backbone, and bottom models split from the FM
$f_{\mathcal{W}^{S}_{\text{FM}}}$, respectively; [·, ·] stands for
concatenation; $f\circ g$ is the function composition such that the outputs of
the function $g$ are fed into the function $f$; $\ell_{\text{TA}}^{C_{k}}$ is
the task loss for training client $k$’s DM using local data and the knowledge
from the FM; $\ell_{\text{KD}}^{C_{k}}$ is the knowledge distillation loss for
distilling knowledge of FM to client $k$’s DM based on representations
$z^{C_{k}}$ that are shared between the server and client $k$.

As demonstrated in studies (Sun et al., 2022b; Diao et al., 2023; Li et al.,
2023b; Lin et al., 2023; Sun et al., 2023), in federated prompt tuning, each
client $k$ trains a local prompt generator $g_{\mathcal{W}^{C_{k}}}$ based on
its own local data and the responses made by $f_{\mathcal{W}^{S}_{\text{FM}}}$
following Eq.(7). As illustrated by (Zhou et al., 2023; Shen et al., 2023; Li
et al., 2023c; Xu et al., 2023c; Tian et al., 2022), in federated split
learning, the FM $f_{\mathcal{W}^{S}_{\text{FM}}}$ typically is split into a
head model $h_{\mathcal{W}^{\diamond}_{\text{HE}}},\diamond\in\\{S,C_{k}\\}$,
a backbone model $f_{\mathcal{W}^{S}_{\text{BA}}}$, and a bottom model
$g_{\mathcal{W}^{C_{k}}_{\text{BO}}}$. The backbone model is deployed on the
server side, the bottom model is deployed on the client side, while the head
model can be deployed on either side. The server and clients collaborate to
train the dispersed FM following Eq.(8). The core idea of federated knowledge
distillation, as illustrated by (He et al., 2020), involves the training of
each client’s model $g_{\mathcal{W}^{C_{k}}}$ based on local datasets
$\mathcal{D}^{C_{k}}$ and the representations $z^{S}$ generated by the FM
$f_{\mathcal{W}^{S}_{\text{FM}}}$ following Eq.(9).

To achieve the objective of setting ②, federated offsite tuning (Xiao et al.,
2023; Fan et al., 2023; WeBank, [n. d.]; Chua et al., 2023; Khalid et al.,
2023) and federated knowledge distillation (Yu et al., 2023b) are two
representative approaches. We formulate their objectives as follows:

(10) federated offsite tuning:
$\displaystyle\min_{\mathcal{W}^{S}_{a},\mathcal{W}^{S}_{b}}\ell_{\text{TA}}^{C_{k}}(\phi_{\mathcal{W}^{S}_{a}}\circ\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}\circ\psi_{\mathcal{W}^{S}_{b}}(x^{C_{k}}),y^{C_{k}}),$
(11) federated knowledge distillation:
$\displaystyle\min_{\mathcal{W}^{S}_{\text{FM}}}\ell_{\text{KD}}^{S}(f_{\mathcal{W}^{S}_{\text{FM}}}(x^{S}),z^{C_{k}})\text{
where }z^{C_{k}}=g_{\mathcal{W}^{C_{k}}}(x^{S}).$

where $\phi_{\mathcal{W}^{S}_{a}}$ and $\psi_{\mathcal{W}^{S}_{b}}$ are the
two adapters selected from $f_{\mathcal{W}^{S}_{\text{FM}}}$, and
$\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}$ is emulator compressed
from $f_{\mathcal{W}^{S}_{\text{FM}}}$; $\ell_{\text{TA}}^{C_{k}}$ is the task
loss for training $\phi_{\mathcal{W}^{S}_{a}}$ and
$\psi_{\mathcal{W}^{S}_{b}}$ with the help of emulator
$\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}$; $x^{S}$ is the data
shared by the server and clients, and it serves as the base for knowledge
distillation, $\ell_{\text{KD}}^{S}$ is the knowledge distillation loss for
fine-tuning $f_{\mathcal{W}^{S}_{\text{FM}}}$ using representations
$z^{C_{k}}$ generated by client $k$’s local DM $g_{\mathcal{W}^{C_{k}}}$ based
on $x^{S}$.

In federated offsite tuning (Fan et al., 2023; WeBank, [n. d.]; Chua et al.,
2023), the server first sends $\phi_{\mathcal{W}^{S}_{a}}$,
$\psi_{\mathcal{W}^{S}_{b}}$, and
$\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}$ to all clients before
training. Then, each client trains the adaptors $\phi_{\mathcal{W}^{S}_{a}}$
and $\psi_{\mathcal{W}^{S}_{b}}$ with the help of the emulator
$\tilde{f}_{\widetilde{\mathcal{W}}^{S}_{\text{FM}}}$ based on its local data
following Eq.(10). Next, all clients send their adaptors back to the server,
which in turn aggregates clients’ adaptors and plugs the aggregated adaptors
back to the FM. In federated knowledge distillation (Yu et al., 2023b), the FM
$f_{\mathcal{W}^{S}_{\text{FM}}}$ at the server is fine-tuned by the
representations generated by each client’s DM $g_{\mathcal{W}^{C_{k}}}$
following Eq.(11). In both approaches, $f_{\mathcal{W}^{S}_{\text{FM}}}$ can
be optionally fine-tuned by the server’s local data if available.

To achieve the objective of setting ③, the server’s FM and clients’ DMs are
co-optimized through sharing knowledge of each other. This co-optimizing
objective can be formulated as follows.

(12) $\displaystyle\textit{federated co-
optimizing:}\quad\min_{\mathcal{W}^{C_{k}},\mathcal{W}^{S}_{\text{FM}}}$
$\displaystyle\ell_{\text{TA}}^{C_{k}}(g_{\mathcal{W}^{C_{k}}}(x^{C_{k}}),y^{C_{k}})$
$\displaystyle+\ell_{\text{FT}}(f_{\mathcal{W}_{\text{FM}}^{S}}(x^{a}),y^{a})+\ell_{\text{FT}}(g_{\mathcal{W}^{C_{k}}}(x^{b}),y^{b}).$

where $\ell_{\text{TA}}^{C_{k}}$ is the task loss for training client $k$’s DM
$g_{\mathcal{W}^{C_{k}}}$ based on local data; $\ell_{\text{FT}}$ is the loss
for fine-tuning $f_{\mathcal{W}^{S}_{\text{FM}}}$ and
$g_{\mathcal{W}^{C_{k}}}$ based on knoweldge in the form of
$\\{x^{a},y^{a}\\}$ and $\\{x^{b},y^{b}\\}$, respectively. The specific forms
of $\\{x^{a},y^{a}\\}$ and $\\{x^{b},y^{b}\\}$ are algorithm-dependent.

The federated co-optimizing can be implemented in various forms, Fan et al.
(2023) proposed FedCoLLM (Fan et al., 2023), a federated mutual knowledge
distillation approach that enables $f_{\mathcal{W}^{S}_{\text{FM}}}$ and
$g_{\mathcal{W}^{C_{k}}}$ to learn from each other through knowledge
distillation. In FedCoLLM, $x^{a}=x^{b}=x^{S}$, $y^{a}=g_{W^{C_{k}}}(x^{S})$,
and $y^{b}=f_{\mathcal{W}^{S}_{\text{FM}}}(x^{S})$ where $x^{S}$ is public
data. Deng et al. (2023) proposed CrossLM that accomplishes the co-
optimization based on a synthetic dataset generated by the server’s FM
$f_{\mathcal{W}^{S}_{\text{FM}}}$. In CrossLM, $y^{a}=y^{b}=y^{S}$ and
$x^{a}=x^{b}=f_{\mathcal{W}^{S}_{\text{FM}}}(\mathcal{F}(y^{S}))$ where
$y^{S}$ is the given labels and $\mathcal{F}(y^{S})$ is the corresponding
label-descriptive prompts.

A special case of objective ③, as formulated in Eq.(5), is that clients’ DMs
are initialized with FMs and they are trained with Parameter-Efficient Fine-
Tuning (PEFT) techniques. We formulate the objective of each client $k$ as
follows.

(13) federated PEFT:
$\displaystyle\min_{\mathcal{W}^{G}_{a}}\ell_{\text{TA}}^{C_{k}}(\phi_{\mathcal{W}^{G}_{a}}\oplus
g_{\mathcal{W}^{C_{k}}_{\text{FM}}}(x^{C_{k}}),y^{C_{k}}).$

where $g_{\mathcal{W}^{C_{k}}_{\text{FM}}}$ is the local FM of client $k$,
$\phi_{\mathcal{W}^{G}_{a}}$ is the global PEFT module (e.g., prompt model,
adaptor, and LoRA) parameterized by $\mathcal{W}^{G}_{a}$, and $\oplus$
denotes an operator that composes $g_{\mathcal{W}^{C_{k}}_{\text{FM}}}$ with
PEFT module $\phi_{\mathcal{W}^{G}_{a}}$; $\ell_{\text{TA}}^{C_{k}}$ is the
task loss for optimizing the global PEFT module $\phi_{\mathcal{W}^{G}_{a}}$
based on client $k$’s local data $x^{C_{k}}$ and $y^{C_{k}}$ while keeping
$g_{\mathcal{W}^{C_{k}}_{\text{FM}}}$ frozen.

The research works, FedPETuning (Zhang et al., 2023d), FedPrompt (Zhao et al.,
2023b), FedAdapter (Cai et al., 2023b), and FedIT (Zhang et al., 2023c) adhere
to the formulation of Eq.(13). FedPrompt, FedAdapter, and FedIT specifically
investigate the impact of prompt tuning, adapter, and LoRA under federated
learning, respectively. On the other hand, FedPETuning delves into exploring
four PEFT techniques, namely adapter tuning, prefix tuning, LoRA, and BitFit,
while operating under the FedAvg protocol.

###  3.4. Formulations of Privacy Leakage

The privacy leakage $\epsilon_{p}$ measures the privacy loss incurred by a
computation (e.g., deep learning) when certain privacy protection mechanisms
are applied. It can be defined in various forms. Herein, we introduce
Differential privacy leakage (Abadi et al., 2016; Sommer et al., 2018) and
Bayesian privacy leakage (Zhang et al., 2022a).

Differential privacy leakage. Let $\epsilon_{p}^{C_{k}}$ represent the privacy
leakage of client $k$. Given a randomization mechanism $\mathcal{M}$,
neighboring databases $\mathcal{D}^{C_{k}},\hat{\mathcal{D}}^{C_{k}}$ of
client $k$, the privacy leakage of the federated learning is defined as
follows.

(14) $\epsilon_{p}=\frac{1}{K}\sum_{k=1}^{K}\epsilon_{p}^{C_{k}},\text{ where
}\epsilon_{p}^{C_{k}}(o)=\text{log}\frac{Pr[\mathcal{M}(\mathcal{D}^{C_{k}})=o]}{Pr[\mathcal{M}(\hat{\mathcal{D}}^{C_{k}})=o]}$

where $\epsilon_{p}^{C_{k}}(o)$ is the privacy leakage of outcome $o$ at
client $k$.

The $\epsilon_{p}$ in formulation Eq.(14) quantifies the privacy loss that an
individual faces when its data is included in a computation and establishes
the correspondence between the maximum amount of privacy leakage and the level
of privacy protection. A smaller value of $\epsilon_{p}$ implies a stronger
privacy guarantee. The exact relationship between the privacy leakage and the
level of privacy protection may vary depending on the specific differentially
private mechanism used. Different mechanisms have different privacy guarantees
and may incur privacy leakage differently. The widely accepted differentially
private mechanism in deep learning is DP-SGD (Abadi et al., 2016).

Although DP-SGD is a commonly used protection mechanism, it has limitations.
Firstly, DP-SGD and its variations (Kairouz et al., 2021; Shi et al., 2022a)
generally do not consider the prior distributions of privacy data, thereby
being prior-independent. Consequently, they are inadequate for modeling recent
Bayesian privacy attacks, for example, DLG (Zhu and Han, 2020). Secondly,
differential privacy protects data privacy by introducing random noise during
computation, and it fails to consider other protection mechanisms, such as
sparsification. To mitigate these drawbacks, Zhang et al. (2022a) introduced
Bayesian privacy leakage to federated learning.

Bayesian privacy leakage. Let $\epsilon_{p}^{C_{k}}$ represent the privacy
leakage of client $k$. The privacy leakage of the federated learning is
defined as follows.

(15) $\epsilon_{p}=\frac{1}{K}\sum_{k=1}^{K}\epsilon_{p}^{C_{k}},\text{ where
}\epsilon_{p}^{C_{k}}=\sqrt{{\text{JS}}(\mathcal{P}_{A}^{C_{k}}||\mathcal{P}_{O}^{C_{k}})}$

where $\mathcal{P}_{A}^{C_{k}}$ and $\mathcal{P}_{O}^{C_{k}}$ represent the
attacker’s belief distribution about private data $\mathcal{D}^{C_{k}}$ of
client $k$ upon observing the protected information and without observing any
information, respectively, and $\text{JS}(\cdot||\cdot)$ denotes Jensen-
Shannon divergence between two distributions (Zhang et al., 2022a).

Bayesian privacy leakage measures the variation between the adversary’s prior
and posterior beliefs on private data. Based on Bayesian privacy leakage,
Zhang et al. (2022a) provided lower bounds of $\epsilon_{p}$ for three
privacy-preserving mechanisms, including Randomization, Sparsification, and
Homomorphic Encryption.

##  4\. FTL-FM Taxonomy

In FTL-FM, we consider the following five research issues: (1) What to
transfer; (2) How to transfer; (3) What to protect; (4) How to protect; (5)
How to attack. Based on the three FTL-FM settings and the five research
issues, we construct a taxonomy to categorize state-of-the-art FTL-FM works,
as shown in Table 2.

  * •

”What to Transfer (WT)” asks what kind of knowledge is transferred between
participating parties of federated learning. Motivated by (Pan and Yang,
2010), we consider three kinds of knowledge that can be transferred between
parties: data-level knowledge, representation-level knowledge, and model-level
knowledge. Specifically, data-level knowledge refers to synthetic or original
data from a dataset as well as prompts manually or automatically generated.
representation-level knowledge refers to intermediate training results such as
representations, logits, and gradients. model-level knowledge refers to model
parameters.

  * •

”How to Transfer (HT)” asks about the specific transfer method leveraged by an
FTL-LLM work. We provide five categories of transferring methods: data
transfer for transferring data-level knowledge; split learning and
representation transfer for transferring representation-level knowledge; homo-
model transfer and hetero-model transfer for transferring model-level
knowledge. From Section 5 to Section 7, we will discuss the specific transfer
method used by each FTL-FM work.

  * •

”What to Protect (WP)” asks what information is protected during knowledge
transfer. We consider four kinds of information that can be protected during
knowledge transfer: data ($\mathcal{D}^{S}$) and model ($\mathcal{W}^{S}$) of
the server as well as data ($\mathcal{D^{C}}$) and model ($\mathcal{W^{C}}$)
of a client. Particularly, we want to protect data privacy as well as model
ownership and performance.

  * •

”How to Protect (HP)” asks about the specific protection method (e.g.,
differential privacy and secure aggregation) used to protect the target
information.

  * •

”How to Attack (HA)” asks the way to attack the protected information. For
conciseness, we present semi-honest and malicious attacks in the taxonomy for
categorization. Semi-honest attackers follow federated learning training and
inference protocols but may try to infer the private data of participating
parties based on observed information, while malicious attackers may update
intermediate training results or model architecture maliciously to extract
private information or jeopardize the model performance of participating
parties.

The taxonomy (see Table 2) elucidates settings and technical details of state-
of-the-art FTL-FM works. Note that Table 2 presents only FTL-FM works that
consider all the research issues discussed aforementioned and focus on the
training phase111We are aware that many FTL-FM works are dedicated to
knowledge transfer techniques without explicitly protecting data privacy,
model ownership, or defending against backdoor attacks. We will discuss these
works in detail from Section 5 to Section 7.

From Table 2, we can see that most of the FTL-FM works focus on the setting of
transferring and adapting knowledge of the FM to DMs (i.e., the first
setting), and these works involve a wide range of knowledge transfer
techniques, including data transfer, split learning, homo-model transfer, and
hetero-model transfer. This is expected because adapting or customizing FMs to
downstream domain-specific tasks is a pivotal research area. The setting of
augmenting FMs with domain-specific knowledge (i.e., the second setting) is
rarely studied, involving only offsite-tuning (Xiao et al., 2023) and its
federated learning counterparts (Fan et al., 2023). The third setting focuses
on investigating model transfer techniques to co-optimize FMs and DMs.

From the perspective of data and model protection, most of the FTL-FM works
(superscripted by $*$) focus on protecting data privacy under semi-honest
attacks. Backdoor (superscripted by $\ddagger$), including both attack and
defense, is also an active research area. While the model ownership protection
(superscripted by $\dagger$) is under-explored.

Next, we will review FTL-FM works presented in Table 2 and other relevant
works that fall within the spectrum of FTL-FM in Sections 5, 6, and 7,
respectively. These methods can be reduced to the formulations defined under
the FTL-FM framework (see Definition 3.1). We will indicate their
corresponding relationships in the discussion.

HA |  WP HT |  Data Transfer |  Split Learning |  Repr Transfer |  Homo- Model Transfer |  Hetero- Model Transfer |  Data Transfer |  Split Learning |  Repr Transfer |  Homo- Model Transfer |  Hetero- Model Transfer |  Data Transfer |  Split Learning |  Repr Transfer |  Homo- Model Transfer |  Hetero- Model Transfer
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Semi-Honest |  $\mathcal{D}^{S}$ ($\mathcal{W}^{S}$) |  PPDG (Carranza et al., 2023; Kurakin et al., 2023)∗ (Mattern et al., 2022; Yue et al., 2023)∗ |  |  |  DP (Behnia et al., 2022; Bu et al., 2022; Shi et al., 2022b)∗, MP (Youssef et al., 2023)∗ |  DP (Yu et al., 2022; Mireshghallah et al., 2022)∗ |  |  |  |  |  CP (Xiao et al., 2023) (Fan et al., 2023; Chua et al., 2023; Khalid et al., 2023)† |  |  |  |  |
$\mathcal{D}^{C}$ ($\mathcal{W}^{C}$) |  |  DP (Li et al., 2023c; Shen et al., 2023)∗, WPE (Xu et al., 2023c)∗, PTB (Zhou et al., 2023)∗ |  |  PPDS (Hou et al., 2023b; Yu et al., 2023a)∗ |  DP (Wang et al., 2023b)∗ |  |  |  |  |  SA (Fan et al., 2023)∗ |  |  |  |  DP (Azam et al., 2023; Balunovic et al., 2022)∗, FE (Gupta et al., 2022)∗ HE (Jin et al., 2023)∗ |  DP (Xu et al., 2023a; Zhao et al., 2023b)∗, PEFT (Zhang et al., 2023d)∗, SA (Fan et al., 2023)∗
Malicious |  $\mathcal{D}^{S}$ ($\mathcal{W}^{S}$) |  |  |  |  |  |  |  |  |  |  |  |  |  |  NC, DP (Zhang et al., 2022b; Yoo and Kwak, 2022)‡ |
$\mathcal{D}^{C}$ ($\mathcal{W}^{C}$) |  |  |  |  FP (Zhang et al., 2021; Shen et al., 2021)‡, OWD (Chen et al., 2022b; Mei et al., [n. d.])‡, MDP (Xi et al., 2023)‡ |  OWD (Cai et al., 2022)‡ |  |  |  |  |  |  |  |  |  DP (Chu et al., 2023)∗ (Fowl et al., 2023; Rashid et al., 2023)∗ SCR (Rashid et al., 2023)∗ |

Hetero-Model Transfer means the model transferred to other parties has a
different architecture from the original one; If multiple clients are
involved, their model architectures are different.
---

$*$ signifies works defending against data privacy attacks. |  $\dagger$ signifies works defending against model stealing attacks. |  $\ddagger$ signifies works defending against backdoor attacks.
---|---|---

Item to be protected: |  $\mathcal{D}^{S}$: data of server |  $\mathcal{D}^{C}$: data of client |  $\mathcal{W}^{S}$: model of server |  $\mathcal{W}^{C}$: model of client
---|---|---|---|---

Protection Methods: |  |  |
---|---|---|---
DP: Differential Privacy | PPDG: Privacy-Preserving Data Generation | MP: Model Trained via Proxy Data | MDP: Masking-Differential Prompting
SA: Secure Aggregation | PPDS: Privacy-Preserving Data Selection | PEFT: Parameter-Efficient Fine-Tuning | HE: Homomorphic Encryption
NC: Norm Clipping | FE: Freezing Embeddings | PTB: Perturbation | CP: Compression
FP: Fine Pruning | WPE: Weight Permutation Equivalence | OWD: Outlier Word Detection | SCR: Scrubbing

Table 2. Taxonomy of existing FTL-FM works. This taxonomy only presents works
focusing on the training phase.

##  5\. FTL-FM: Foundation Models for Enhancing Domain Models

In this section, we overview FTL-FM methods that fall into setting ① (see
Definition 3.1), the objective of which is to transfer and adapt the knowledge
of foundation models (FMs) to downstream clients’ domain models (DMs) for
exploiting the power of FMs. To this end, the literature proposed a variety of
methods involving data-level knowledge transfer, representation-level
knowledge transfer, and model-level knowledge transfer.

###  5.1. FM adapted to DMs through data-level knowledge transfer

The literature has explored mainly two categories of methods that adapt the
knowledge of FMs to specific domains through data-level knowledge transfer:
(1) An FM server transfers data generated by general-purpose FMs or FMs
augmented with industry-level knowledge (e.g., LawGPT (Nguyen, 2023), FinGPT
(Yang et al., 2023), BloombergGPT (Wu et al., 2023a), and Med-PaLM2 (Singhal
et al., 2023)) to clients of specific domains; (2) a downstream client prompts
tuning a closed-source FM (e.g., ChatGPT) by learning a local prompt generator
based on domain-specific data and predictions generated by the FM. We
summarize data-level knowledge transfer methods that fall into setting ① in
Table 3222While we consider an FTL-FM approach should incorporate certain
protection mechanisms to protect data or models, we still review highly
relevant works that prioritize the development of knowledge transfer methods
without considering data or model protection and include them in the Table to
ensure thoroughness. (same for Section 6 and Section 7). These methods leave
”blank” in the ”What To Protect” and ”How To Protect” columns. and elaborate
on these methods in this subsection.

Table 3. Summary of data-level knowledge transfer methods of setting ①. $\bigcirc$ denotes that a reference work intends to protect data or model against privacy attacks using certain protection presented in the ”How To Protect” columns. DP: Differential Privacy; $S_{\text{FM}}$: FM server; $S_{\text{Fed}}$: federated server; $X$, $Y$, $\hat{Y}$, and $P$ are input, ground truth response, prediction, and prompt, respectively. Fr.: From. Transfer Method Category |  Reference |  What To Protect |  Exchanged Information | How To Protect
---|---|---|---|---
Server(s) |  Client(s) |  Server(s) To Client(s) |  Client(s) To Server(s) | Server |  Client
$\mathcal{D}^{S}$ | $\mathcal{W}^{S}_{\text{FM}}$ | $\mathcal{D}^{C}$ | $\mathcal{W}^{C}$
Data Transfer | Yue et al. (2023) | $\bigcirc$ |  |  |  |  Data generated via DP-tuned $\mathcal{W}^{S}_{\text{FM}}$ |  | DP |
Mattern et al. (2022) |  |  |  |  |
Kurakin et al. (2023) |  |  |  |  |
Carranza et al. (2023) |  |  |  |  |
BBT (Sun et al., 2022b, a) |  |  |  |  |  $\hat{Y}$ from $\mathcal{W}^{S}_{\text{FM}}$ |  $X$ \+ soft $P$ |  |
BlackVIP (Oh et al., 2023), |  |  |  |  |  |
BDPL (Diao et al., 2023) |  |  |  |  |  $\hat{Y}$ from $\mathcal{W}^{S}_{\text{FM}}$ |  $X$ \+ hard $P$ |  |
DSP (Li et al., 2023b) |  |  |  |  |  Reward from $\mathcal{W}^{S}_{\text{FM}}$ |  $X$ \+ hard $P$ |  |
Fed-BBPT (Lin et al., 2023) |  |  |  |  |  |  Fr. $S_{\text{FM}}$: |  $\hat{Y}$ of $\mathcal{W}^{S}_{\text{FM}}$
---|---
Fr. $S_{\text{Fed}}$: |  Global model
|  To $S_{\text{FM}}$: |  $X$ \+ soft $P$
---|---
To $S_{\text{Fed}}$: |  $\mathcal{W}^{C}$
|
FedBPT (Sun et al., 2023) |  |  |  |  |  |  Fr. $S_{\text{FM}}$: |  $\hat{Y}$ of $\mathcal{W}^{S}_{\text{FM}}$
---|---
Fr. $S_{\text{Fed}}$: |  Global distr. of $P$
|  To $S_{\text{FM}}$: |  $X$ \+ soft $P$
---|---
To $S_{\text{Fed}}$: |  Local distr. of $P$
|
ZooPFL (Lu et al., 2023) |  |  |  |  |  |  Fr. $S_{\text{FM}}$: |  $\hat{Y}$ of $\mathcal{W}^{S}_{\text{FM}}$
---|---
Fr. $S_{\text{Fed}}$: |  Global encoder
|  To $S_{\text{FM}}$: |  Perturbed $X$
---|---
To $S_{\text{Fed}}$: |  Local encoder
|
Figure 6. Illustration of adapting knowledge of the server’s foundation model
(FM) to clients’ domain models (DMs) through data-level knowledge transfer.
(a) illustrates that the knowledge of FM is transferred to downstream clients
via privacy-preserving synthetic data sharing. (b) illustrates that a client
can adapt knowledge of the FM to its local task via federated prompt learning.

In some proprietary domains, such as the audit department in a bank, data
resources are typically limited for deep learning tasks. One potential
solution to address data scarcity is leveraging FMs pre-trained on a large
corpus of domain-specific data to generate synthetic training and testing data
(e.g., LawGPT (Nguyen, 2023)). However, domain-specific data (e.g., financial
and legal data) typically contain sensitive and private information. Thus,
directly releasing FMs trained on these data presents potential privacy issues
because sensitive information encoded in FMs can possibly be recovered by
adversaries. In order to share synthetic data generated by FMs without
compromising the privacy of training data, privacy-preserving data generation
methods have been proposed (Carranza et al., 2023; Yue et al., 2023; Mattern
et al., 2022; Kurakin et al., 2023). Essentially, these methods first fine-
tune a pretrained public FM using differential privacy (DP) on private domain
data and then exploit the DP-tuned FM to generate new data, which can be
distributed to downstream clients to augment their training data (illustrated
in Figure 6(a)). These methods correspond to stage 1 of objective ① formulated
in Eq.(1) but with an additional step of generating data using fine-tuned FMs.

Alternatively, clients can adapt knowledge of the FM hosted by an FM server to
their specific domains through federated prompt tuning. A typical setting of
federated prompt tuning involves a client that learns a local model to
generate prompts that can guide the FM toward the desired responses
(illustrated in Figure 6(b)). For example, BBT (Sun et al., 2022b), BBTv2 (Sun
et al., 2022a), and BlackVIP (Oh et al., 2023) proposed that a client can
customize a remote FM (e.g., ${\text{RoBERTa}}_{\text{large}}$ (Liu et al.,
2019), ${\text{BART}}_{\text{large}}$ (Lewis et al., 2020), and ViT-B (Radford
et al., 2021)) by optimizing a domain-specific prompt generator based on its
local data together with the predictions (e.g., logits) generated by the
remote FM. BBT and BBTv2 employ a gradient-free method called CMA-ES
(Covariance Matrix Adaptation Evolution Strategy) (Hansen, 2016) to optimize
the client’s local prompt generator, whereas BlackVIP leverages SPSA
(Simultaneous Perturbation Stochastic Approximation) (Spall, 1992) to
efficiently approximates the high-dimensional gradients, which then are used
to optimize the prompt generator. BBT, BBTv2, and BlackVIP require the FM API
to take continuous (i.e., soft) prompts as input, and thus, they cannot be
applied to FM APIs that only accept discrete inputs. To eliminate this
constraint, BDPL (Diao et al., 2023) and DSP (Li et al., 2023b) were proposed
to learn a client side’s discrete (i.e., hard) prompt generator (DPG) for
customizing FM’s responses. BDPL adopts a variance-reduced policy gradient
algorithm (Zhou et al., 2021) to optimize the DPG, while DSP optimizes the DPG
through supervised fine-tuning and reinforcement learning. BBT, BBTv2,
BlackVIP, BDPL, and DSP involve only one client, and thus, they correspond to
stage 2 of the first objective formulated in Eq.(2) with $K=1,p^{C_{1}}=1$,
$\eta=0$, the domain model $\mathcal{W}^{C_{1}}$ is the client’s local prompt
generator and the specific form of the loss
$\ell^{C_{1}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}};d)$ is
formulated in Eq.(7).

Fed-BBPT (Lin et al., 2023) and FedBPT (Sun et al., 2023) extend the federated
prompt tuning with only one client to multiple clients, the workflow of which
is illustrated in Figure 7. More specifically, Fed-BBPT and FedBPT involve two
servers: an FM server, denoted as $S_{\text{FM}}$, and a federated server,
denoted as $S_{\text{Fed}}$. The former hosts an FM to be prompted by clients,
while the latter is for federated tuning clients’ local prompt generators. In
Fed-BBPT, each client leverages the Simultaneous Perturbation Stochastic
Approximation (Spall, 1997) to train its prompt generators based on local data
and predictions generated by the FM, and the federated server aggregates
clients’ local prompt generators to form the global prompt generator. FedBPT
differs from Fed-BBPT in that each client in FedBPT exploits CMA-ES to
optimize the distribution of prompts generated by its prompt generator, and
the server derives the global prompt generator by aggregating local
distributions of clients’ prompts. Generally, Fed-BBPT and FedBPT adhere to
the objective formulated in Eq.(2) with $K>1$, $\eta=0$, the domain model
$\mathcal{W}^{C_{k}}$ of each client $k$ is a prompt generator and the
specific loss of federated prompt tuning
$\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ between the
FM server and a client $k$ is formulated in Eq.(7).

Figure 7. Illustration of adapting knowledge of the server’s foundation model
(FM) to clients’ domain models (DMs) through data-level knowledge transfer.
This figure illustrates that multiple clients adapt the knowledge of the FM
hosted by an FM server to their local tasks via federated prompt tuning, and
these clients also leverage the FedAvg algorithm or its variants with the help
of the federated server to train their local domain models.

While these federated prompt tuning works can adapt knowledge of FMs to
clients’ domain-specific tasks, they put the privacy of clients’ local data at
high risk since they send prompts and input text directly to the FM server
without any privacy protection. For applications where (training and
inference) data contains sensitive and private information of users, privacy-
preserving techniques such as DP-rewrite (Igamberdiev et al., 2022) should be
applied to data transmitted between parties.

Instead of leveraging prompt tuning, ZooPFL (Lu et al., 2023) adapts the
knowledge of FM to downstream clients’ tasks by training a client-specific
embedding and a semantic re-mapping module for each client. More specifically,
each client appends the client-specific embedding to the embedding generated
by a pre-trained auto-encoder, which aims to make the client’s local input
consistent with the FM. Clients also collaboratively fine-tune encoders of
their auto-encoders through FedAVG (McMahan et al., 2017) to share knowledge.
The semantic re-mapping module aligns the logits output from the FM with the
semantic space of clients’ local tasks. ZooPFL adheres to the objective
formulated in Eq. (2) with $K>1$, $\eta=0$. The domain model
$\mathcal{W}^{Ck}$ of each client $k$ in ZooPFL is the client-specific
embedding and semantic re-mapping module.

###  5.2. FM adapted to DMs through representation-level knowledge transfer

Representation-level knowledge transfer (RLKT) refers to transferring and
adapting knowledge of FMs to domain-specific models through passing
representation-level knowledge, such as intermediate forward outputs and
backward gradients, between FL participating parties. In literature, split
learning (Liu et al., 2022) and representation transfer are two main
categories of RLKT explored in the setting ① of FTL-FM. We summarize the two
categories of RLKT methods in Table 4 and elaborate on these methods in this
subsection.

Table 4. Summary of representation-level knowledge transfer methods of setting ①. $\bigcirc$ denotes that a reference work intends to protect data or models against privacy attacks using certain protection presented in the ”How To Protect” columns. DP: Differential Privacy; PTB: Perturbation; WPE: Weight Permutation Equivalence. Transfer Method Category |  Reference |  What To Protect |  Exchanged Information | How To Protect
---|---|---|---|---
Server(s) |  Client(s) |  Server(s) To Client(s) |  Client(s) To Server(s) | Server |  Client
$\mathcal{D}^{S}$ | $\mathcal{W}^{S}_{\text{FM}}$ | $\mathcal{D}^{C}$ | $\mathcal{W}^{C}$
Split Learning | TextObfuscator (Zhou et al., 2023) |  |  | $\bigcirc$ |  | Backward gradient |  Forward repr $+Y$ |  | PTB
SAP (Shen et al., 2023) |  |  | $\bigcirc$ |  |  Prediction |  Forward repr, Backward gradient |  | PTB
RAPT (Li et al., 2023c) |  |  |  $\bigcirc$ |  |  Prediction Backward gradient |  Forward Repr, Backward gradient |  |  DP
Shuffled Transformer (Xu et al., 2023c) |  |  | $\bigcirc$ |  |  Forward Repr, Backward gradient |  Forward Repr, Backward gradient |  | WPE
PrivateLoRA (Wang et al., 2023a) |  |  |  |  |  Forward Repr, Backward gradient |  Forward Repr, Backward gradient |  |
FedBERT (Tian et al., 2022) |  |  |  |  |  Forward Repr, Backward gradient |  Forward Repr, Backward gradient |  |
Repr Transfer |  FedGKT (He et al., 2020) |  |  |  |  |  Global logit |  Repr + Logit |  |

In a two-party federated split learning setting involving a client and a
server, the client typically owns a private dataset and a small portion of an
FM, while the server owns the larger portion. The objective of FM-based split
learning is to adapt the knowledge of FM to the client’s specific domain by
fine-tuning the distributed FM using the client’s local data. The FM is often
split into an FM backbone and multiple other smaller parts. The FM backbone is
the most computationally expensive part and is typically deployed on a server
with sufficient computing resources, while other parts can be deployed on
either the client or server. The training procedure of distributed FM follows
the conventional split learning procedure using stochastic gradient descent,
but each portion of FM is trained by a different party (i.e., client and
server). Figure 8 illustrates three two-party federated split learning
settings studied in the literature.

Figure 8. Illustration of adapting knowledge of the server’s foundation model
(FM) to the client’s domain model (DM) through federated split learning that
involves one server and one client. The two-party federated split learning
generally has three scenarios: (a) The FM is split into an FM backbone model
and an FM bottom model. The FM backbone is deployed on the server, while the
FM bottom model is deployed on the client; (b) The FM is split into an FM
backbone model, an FM bottom model, and an FM head model. The FM backbone is
deployed on the server, while the FM head and bottom models are deployed on
the client; (c) The FM is split into an FM backbone model, an FM bottom model,
multiple FM middle models, and an FM head model. The FM backbone is deployed
on the server, while the FM head, middle, and bottom models are deployed on
the client.

TextObfuscator (Zhou et al., 2023), SAP (Shen et al., 2023), RAPT (Li et al.,
2023c), and Shuffled Transformer (Xu et al., 2023c) investigated the first two
settings of two-party federated split learning, as illustrated in Figure 8(a)
and Figure 8(b). In TextObfuscator (Zhou et al., 2023), a
${\text{RoBERTa}}_{\text{base}}$ model (Liu et al., 2019) is divided into a
large server model and a smaller client model. The client and server
collaboratively train the two models following the split learning protocol. To
protect privacy, the client perturbs the representations to be shared with the
server with semantically similar alternatives based on clustering. However,
the client in TextObfuscator sends labels directly to the server to compute
the loss, thereby leaking the private label information. SAP (Shen et al.,
2023) takes a similar setting as TextObfuscator. The difference is that, in
SAP, the client model is a frozen embedding model, and the server sends
predictions to the client to compute the task loss instead of the other way
around. In addition, SAP adopts the LoRA method to fine-tune the server model.
Figure 8(a) provides a high-level overview of the workflow of TextObfuscator
and SAP. RAPT (Li et al., 2023c) deploys the head and bottom of an FM (i.e.,
$\text{BERT}_{\text{base}}$ (Devlin et al., 2019) or $\text{T5}_{\text{base}}$
(Raffel et al., 2020)) on the client side while the FM backbone on the server
side, and thus the client in RAPT retains both input texts and labels at
local. Besides, RAPT leverages $d_{\chi}$-privacy (Chatzikokolakis et al.,
2013) to protect the client’s private data. Shuffled Transformer investigates
the federated split learning setting similar to RAPT, but they focus on
designing transformer-based networks (e.g., ViT) that are weight permutation
equivalent (WPE). Such WPE networks can prevent the server from inferring the
private data of the client without compromising model performance. Figure 8(b)
overviews the workflow of RAPT and Shuffled Transformer. TextObfuscator, SAP,
RAPT, and Shuffled Transformer (Xu et al., 2023c) adhere to the objective
formulated in Eq.(2) with $K=1,p^{C_{1}}=1$, and $\eta>0$. Their domain models
$\mathcal{W}^{C_{1}}$ are composed of an FM bottom and possibly an FM head.
The specific loss of federated split learning
$\ell^{C_{1}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{1}};d)$ between the
server and the client is formulated in Eq.(8).

PrivateLoRA (Wang et al., 2023a) explored the third setting of federated split
learning, as illustrated in Figure 8(c). Inspired by LoRA, PrivateLoRA
represents the update $\Delta W$ on each linear projection of query, key, or
value in self-attention with three sequential low-rank matrices $\Delta
W=A\times M\times B$. Consequently, there are $\\{A\\}_{i=1}^{K}$,
$\\{B\\}_{i=1}^{K}$, and $\\{M\\}_{i=1}^{K}$ for $K$ target linear
projections. PrivateLoRA deploys $\\{A\\}_{i=1}^{K}$ and $\\{B\\}_{i=1}^{K}$
on the server, whereas $\\{M\\}_{i=1}^{K}$ on the client in addition to
deploying the FM backbone on the server and FM head and bottom models on the
client. The $k$th FM middle model in Figure 8(c) corresponds to $M_{k}$.
During training, $\\{A\\}_{i=1}^{K}$ $\\{B\\}_{i=1}^{K}$, and the FM backbone
are frozen, while the $\\{M\\}_{i=1}^{K}$, FM head, and FM bottom models are
optimized. The benefits of PrivateLoRA lie in enhancing the performance of the
distributed FM on the client’s local task while significantly diminishing
communication overheads.

Figure 9. Illustration of adapting knowledge of the server’s foundation model
(FM) to clients’ domain models (DMs) through representation-level knowledge
transfer. (a) illustrates that the knowledge of FM is adapted to clients’ DMs
through federated split learning that involves multiple clients, while (b)
through federated representation transfer.

FedBERT (Tian et al., 2022) extends the federated split learning with one
client to multiple clients, the workflow of which is illustrated in Figure
9(a). In FedBERT, a pre-specified FM (e.g., ${\text{roberta}}_{\text{base}}$
and GPT2 (Radford et al., 2019)) is split into three parts: FM backbone, FM
head, and FM bottom. The FM backbone is deployed on the server, while the FM
head and FM bottom are deployed on each client. Clients in FedBERT collaborate
with the server to train the FM backbone and their corresponding FM bottoms
and heads following conventional split learning protocol (Liu et al., 2022).
FedBERT adheres to the objective formulation Eq.(2) with $K>1$, and $\eta=0$.
The domain model $\mathcal{W}^{C_{k}}$ of each client $k$ is composed of an FM
head and an FM bottom, and the specific loss of federated split learning
$\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ between the
server and a client $k$ is formulated in Eq.(8).

Federated split learning enables clients to adapt the knowledge of an FM to
their domain-specific tasks while consuming only a small amount of computing
resources (i.e., a client only needs to train a small portion of an FM).
However, clients have to collaborate with the server to make inferences, which
may not apply to scenarios where efficient online inference is critical.

FedGKT (He et al., 2020) offers an alternative representation-level transfer
approach that adapts the server’s large model to clients’ DMs. Although FedGKT
used ResNet-110 as the server’s model, the same wisdom can apply to FMs.
According to FedGKT, each DM and the FM are split into a top part and a bottom
part. Upon receiving global logits, each client trains its DM based on its
local data and global logits. Then, each client generates local
representations and logits using its DM and sends them to the server. The
server, in turn, trains its foundation top model using received local
representations and logits. Next, the server sends global logits generated
from the foundation top model to clients for the next round of training. The
training process of FedGKT is illustrated in Figure 9(b). In general, FedGKT
adheres to the objective formulated in Eq.(2) with $K>1$,
$\sum^{K}_{k=1}p^{C_{k}}=1$, $p^{S}>0$, and $\eta=0$. The specific loss
$\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ of training
client $k$’s domain model $\mathcal{W}^{C_{k}}$ using local data and logits
transferred from server’s $\mathcal{W}^{S}_{\text{FM}}$ is formulated in
Eq.(9).

###  5.3. FM adapted to DMs through model-level knowledge transfer

Transferring knowledge from FMs in the form of models to downstream clients is
an active research direction for grounding FMs. Model transferring methods can
be reduced to the formulation of Eq.(1). We group model transferring methods
into two categories. The first category is homo-model transfer, which first
minimizes loss $\ell^{S}(\mathcal{W}^{S}_{\text{FM}})$ by fine-tuning or pre-
training the FM $\mathcal{W}^{S}_{\text{FM}}$ and then transfers the resulting
model $\mathcal{\widetilde{W}}^{S}_{\text{FM}}$ to downstream clients.
$\mathcal{\widetilde{W}}^{S}_{\text{FM}}$ has the same architecture as the
original FM $\mathcal{W}^{S}_{\text{FM}}$ (illustrated in Figure 10(a)). The
second category is hetero-model transfer, which first minimizes loss
$\ell^{S}(\mathcal{W}^{S}_{\text{FM}})$ by compressing the FM
$\mathcal{W}^{S}_{\text{FM}}$ and then transfer the resulting model
$\mathcal{\widetilde{W}}^{S}_{\text{SM}}$ to downstream clients.
$\mathcal{\widetilde{W}}^{S}_{\text{SM}}$ has a different architecture as the
original FM $\mathcal{W}^{S}_{\text{FM}}$ (illustrated in Figure 10(b)). A
model transferring method involves domain adaptation if $\lambda>0$ and
privacy protection if $\eta>0$. We summarize the two categories of model-level
knowledge transfer methods in Table 5 and illustrate them in Figure 10(a) and
Figure 10(b), respectively. In the following, we will elaborate on these
methods of the two categories.

Table 5. Summary of model-level knowledge transfer methods of setting ①. $\bigcirc$ ($\bigcirc^{\ddagger}$) denotes that a reference work intends to protect data or model against privacy (backdoor) attacks using certain protection presented in the ”How To Protect” columns. DP: Differential Privacy; MP: Model Trained via Proxy Data; PPDS: Privacy-Preserving Data Selection; OWD: Outlier Word Detection; MDP: Masking-Differential Prompting; FP: Fine Pruning. BD: Backdoored; FT: Fine-Tuned; PT: Pre-Trained; $X$, $Y$, and $P$ are input, ground truth response, and prompt, respectively. Transfer Method Category |  Reference |  What To Protect |  Exchanged Information | How To Protect
---|---|---|---|---
Server(s) |  Client(s) |  Server(s) To Client(s) |  Client(s) To Server(s) | Server |  Client
$\mathcal{D}^{S}$ | $\mathcal{W}^{S}_{\text{FM}}$ | $\mathcal{D}^{C}$ | $\mathcal{W}^{C}$
Homo- Model Transfer |  FreD (Hou et al., 2023b) |  |  | $\bigcirc$ |  |  FT $\mathcal{W}^{S}_{\text{FM}}$ | Data statistics |  | PPDS
SPT (Yu et al., 2023a) |  |  | $\bigcirc$ |  |  FT $\mathcal{W}^{S}_{\text{FM}}$ | Domain classier |  | PPDS
EW-Tune (Behnia et al., 2022), | $\bigcirc$ |  |  |  |  FT $\mathcal{W}^{S}_{\text{FM}}$ |  | DP |
JFT (Shi et al., 2022b) |  |  |  |  |
DP-BiTFiT (Bu et al., 2022) | $\bigcirc$ |  |  |  |  $\mathcal{W}^{S}_{\text{FM}}$ with FT bias |  | DP |
Youssef et al. (Youssef et al., 2023) | $\bigcirc$ |  |  |  |  Partially PT $\mathcal{W}^{S}_{\text{FM}}$ |  | MP |
NeuBA (Zhang et al., 2021), |  |  |  | $\bigcirc^{\ddagger}$ | BD $\mathcal{W}^{S}_{\text{FM}}$ |  |  | FP
POR (Shen et al., 2021) |  |  |  |  |
BadPre (Chen et al., 2022b) |  |  |  | $\bigcirc^{\ddagger}$ |  BD $\mathcal{W}^{S}_{\text{FM}}$ |  |  | OWD
Notable (Mei et al., [n. d.]) |  |  |  |  $\bigcirc^{\ddagger}$ | BD $\mathcal{W}^{S}_{\text{FM}}$ |  |  |  OWD
UOR (Du et al., 2023) |  |  |  |  $\bigcirc^{\ddagger}$ | BD $\mathcal{W}^{S}_{\text{FM}}$ |  |  |
|  MDP (Xi et al., 2023) |  |  |  |  $\bigcirc^{\ddagger}$ | BD $\mathcal{W}^{S}_{\text{FM}}$ |  |  | MDP
Hetero- Model Transfer |  DP-Tune (Yu et al., 2022) | $\bigcirc$ |  |  |  |  Adapter of $\mathcal{W}^{S}_{\text{FM}}$ |  | DP |
DP-KD (Mireshghallah et al., 2022) | $\bigcirc$ |  |  |  |  Distilled $\mathcal{W}^{S}_{\text{FM}}$ |  | DP |
DPIMP (Mireshghallah et al., 2022) | $\bigcirc$ |  |  |  |  Compressed $\mathcal{W}^{S}_{\text{FM}}$ |  | DP |
Wang et al. (2023b) |  |  | $\bigcirc$ |  |  Distilled $\mathcal{W}^{S}_{\text{FM}}$ | $\mathcal{W}^{C}$ |  | DP
Yuan et al. (2023) |  |  |  |  |  PEFT-tuned $\mathcal{W}^{S}_{\text{FM}}$ |  |  |
GPT-FL (Zhang et al., 2023b) |  |  |  |  |  Model trained on data generated by $\mathcal{W}^{S}_{\text{FM}}$ |  |  |
Feng et al. (Feng et al., 2023) |  |  |  |  |  |  |
BD-FMFL (Li et al., 2023e) |  |  |  |  |  BD global model |  |  |
BadPrompt (Cai et al., 2022) |  |  |  |  $\bigcirc^{\ddagger}$ | BD prompt model |  $X+Y+P$ |  |  FP
|  PPT (Du et al., 2022b) |  |  |  |  $\bigcirc^{\ddagger}$ | BD prompt model |  $X+Y+P$ |  |
Figure 10. Illustration of transferring knowledge of the server’s foundation
model (FM) to clients through model transfer. (a) illustrates the knowledge of
FM is transferred to downstream clients through homo-model transfer, while (b)
through hetero-model transfer.

For homo-model transfer without adaptation, JFT (Shi et al., 2022b) proposed a
selective differential privacy approach to improve the utility of DP-tuned FMs
(i.e., Roberta-base and GPT2-small), which can be released to downstream
clients. More specifically, JFT first fine-tunes an FM with the redacted
version of a private dataset and then fine-tunes the FM again with the
original private data using an efficient DP-SGD (Li et al., 2022c). DP-BiTFiT
(Bu et al., 2022) aims to improve the efficiency of fine-tining LLMs with DP-
SGD by training only bias terms in an FM. Youssef et al. (2023) proposed a
privacy-preserving approach to share an FM (i.e., $\text{BERT}_{\text{base}}$)
pre-trained on a private dataset. More specifically, they proposed to protect
the privacy of the private data by sharing only parts of the FM’s pretrained
parameters with the rest parameters randomly initialized and releasing a proxy
of the private data for downstream clients to fine-tune the FM for
complementary knowledge transfer.

For transferring homo-models adaptive to downstream clients’ data, Yu et al.
(Yu et al., 2023a) proposed a selective pre-training method (SPT) to pre-train
an FM (e.g., $\text{BERT}_{\text{base}}$) based on training data selected by a
domain classifier, which is differential-privately trained on the downstream
client’s private data. Consequently, the client can perform better when
initialized with the FM pretrained on adaptively selected data than on
randomly selected data. FreD (Hou et al., 2023b) was proposed to select the
data used to fine-tune an FM (i.e., DistilGPT-2) for initializing models of
clients. The data selection is performed based on the statistical information
of clients’ private data that is collected leveraging a DP-based federated
learning algorithm. In addition to adaptation by data selection, Yuan et al.
(2023) proposed to build a one-size-fits-all mobile foundation model termed M4
for diversified mobile AI tasks. Specifically, M4 is built on pre-trained and
off-the-shelf FMs (e.g., LLaMA-7B) and exposes three learnable parts (e.g.,
LoRA) to be fine-tuned for downstream mobile AI tasks. Clients have the same
task, say image captioning, can deploy the M4 model with learnable parameters
trained on image captioning datasets.

For hetero-model transfer without adaptation, DP-Tune (Yu et al., 2022) was
proposed to differential-privately fine-tune an additional small set of
parameters on top of pre-trained FMs (e.g., GPT-2-XL) using a private dataset
so that the fine-tuned new parameters can preserve the privacy of the private
dataset when releasing to downstream tasks. Mireshghallah et al. (2022)
proposed privacy-preserving knowledge distillation (DP-KD) and pruning (DPIMP)
methods based on DP-SGD to compress a pre-trained FM (e.g.,
$\text{BERT}_{\text{base}}$) to a smaller model, which can meet the memory and
latency requirements of specific downstream applications.

For transferring hetero-models adaptive to downstream tasks, Wang et al.
(2023b) proposed an approach that leverages public data and pre-trained FM
(i.e., LaMDA (Cohen et al., 2022)) to enhance downstream cross-device
federated learning. More specifically, they first leverage their proposed
distribution matching algorithm to sample public data closely resembling the
distribution of FL clients’ private data, and then they use the sampled data
to distill a pre-trained FM (i.e., LaMDA) into an on-device language model,
which serves as an initialization for FL clients. GPT-FL (Zhang et al., 2023b)
leverages FMs (i.e., Stable Diffusion (Rombach et al., 2022), SpeechT5 (Ao et
al., 2022), and AudioLDM (Liu et al., 2023a)) to generate diversified
synthetic data to train a downstream model, which is then distributed to FL
clients for model initialization.

###  5.4. Privacy and Backdoor Attacks

In the setting ①, the server is the potential adversary, who can launch
privacy attacks with the intention of inferring clients’ private information.
The server can also carry out backdoor attacks to compromise the utility of
clients’ domain models.

Privacy attacks focus on inferring private data from knowledge at the
representation level. More specifically, TextObfuscator (Zhou et al., 2023),
RAPT (Li et al., 2023c) and Xu et al. (2023c) investigated model inversion
attacks in federated split learning scenarios, in which the server aims to
reconstruct the client’s private data based on the representations output from
the client’s local model. These studies proposed representation perturbation,
$d_{\chi}$-privacy, and weight permutation equivalence, respectively, to
thwart model inversion attacks. The rationale behind these defenses is to
reduce the correlation between private data and the representations shared
with the server. As a result, the chance for the adversary to recover private
data can be reduced.

The backdoor attacks are typically conducted through model transfer. In the
case of homo-model transfer, the server injects backdoors into clean
foundation models to be released. This line of works, including BadPre (Chen
et al., 2022b), POR (Shen et al., 2021), Notable (Mei et al., [n. d.]), NeuBA
(Zhang et al., 2021), UOR (Du et al., 2023), and MDP (Xi et al., 2023),
follows a similar pipeline: first, a poisoning dataset is prepared; next, a
clean foundation model is fine-tuned using a combination of clean and
poisoning data; finally, the backdoored foundation model is distributed to
downstream clients. During inference, the server can activate the backdoor
embedded in the client’s local model by making specific queries. NeuBA and POR
leverage Fine-Pruning (FP) (Liu et al., 2018), while BadPre and Notable
utilize Outlier Word Detection (OWD) (Qi et al., [n. d.]) to defend against
the backdoor attack, showing a trade-off between the backdoor’ effectiveness
and the model’s accuracy. MDP, short for Masking-Differential Prompting, is a
backdoor defense proposed by Xi et al. (2023). It identifies poisoned samples
by comparing the representations of given samples under varying masking. The
samples with significant variations are poisoned.

Regarding hetero-model transfer, Li et al. (2023e) proposed BD-FMFL, a
backdoor attack following the framework proposed in GPT-FL (Zhang et al.,
2023b). BD-FMFL works as follows: the attacker inserts backdoors into a
Language Model with Large Memory (LLM) by leveraging in-context learning.
Subsequently, the server utilizes the backdoored LLM to generate prompts,
which are then used to synthesize data using other FMs. The server proceeds to
train a global model based on the synthetic data, thereby transferring the
backdoor to the global model. Once accomplished, the backdoored global model
is fine-tuned using private datasets provided by clients within the standard
Federated Learning (FL) framework. Another avenue of research, explored by
BadPrompt (Cai et al., 2022) and PPT (Du et al., 2022b), focuses on prompt-
based backdoor attacks, in which the attacker injects backdoors into the
prompt model of an FM and subsequently releases the poisoned prompt model to
the public. Upon a client downloading and utilizing the compromised prompt
model for downstream tasks, the server gains the ability to activate the
backdoor. BadPrompt suggests that methods such as Fine-Pruning (FP) and
knowledge distillation could potentially defend against prompt-based backdoor
attacks.

##  6\. FTL-FM: Domain Knowledge for Augmenting Foundation Models

In this section, we overview FTL-FM methods that fall into setting ② (see
Definition 3.1), the objective of which is to augment the FM with industry-
level and domain-specific knowledge (e.g., healthcare, medicine, finance,
banking, and law) transferred from clients.

Table 6. Summary of FTL-FM works of setting ②. $\bigcirc$ ($\bigcirc^{\dagger}$) denotes that a reference work intends to protect data or model against privacy (backdoor) attacks using certain protection methods presented in the ”How To Protect” columns. CP: Compression; SA: Secure Aggregation; repr: Representation. Transfer Method Category |  Reference |  What To Protect |  Exchanged Information | How To Protect
---|---|---|---|---
Client(s) |  Server(s) |  Client(s) To Server(s) |  Server(s) To Client(s) | Client |  Server
$\mathcal{D}^{C}$ | $\mathcal{W}^{C}$ | $\mathcal{D}^{S}$ | $\mathcal{W}^{S}_{\text{FM}}$
Repr Transfer | CreamFL (Yu et al., 2023b) |  |  |  |  | Local repr |  Global repr generated by $\mathcal{W}^{S}_{\text{FM}}$ |  |
Hetero- Model Transfer |  Offsite-Tuning (Xiao et al., 2023) |  |  |  |  $\bigcirc$† | Adapter |  Emulator of $\mathcal{W}^{S}_{\text{FM}}$ |  | CP
FedOST (WeBank, [n. d.]; Fan et al., 2023) | $\bigcirc$ |  |  | $\bigcirc$† | Adapter |  Emulator of $\mathcal{W}^{S}_{\text{FM}}$, Initial adapter | SA | CP
FedOT (Kuang et al., 2023a) |  |  |  | $\bigcirc$† | Adapter |  Emulator of $\mathcal{W}^{S}_{\text{FM}}$, Initial adapter |  | CP
FedPEAT (Chua et al., 2023) |  |  |  | $\bigcirc$† | Adapter |  Emulator of $\mathcal{W}^{S}_{\text{FM}}$, Global adapter |  | CP
CEFHRI (Khalid et al., 2023) |  |  |  |

The key to augmenting FMs to be more knowledgeable, accurate, and reliable
lies in fine-tuning these FMs using high-quality domain-specific datasets.
These high-quality datasets may contain private and sensitive information and
are often scattered among isolated entities (devices and companies). FTL-FM
methods enable the server of FM and clients with domain knowledge to augment
the FM collaboratively while preserving the privacy of clients’ knowledge. In
literature, representation-level knowledge transfer and model-level knowledge
transfer are the main research directions explored to augment an FM hosted by
a server with knowledge transferred from domain-specific clients. We summarize
FTL-FM works that fall into this setting in Table 6, and in the following
subsections, we will elaborate on these works.

Figure 11. Illustration of transferring domain-specific knowledge of clients
to the server’s foundation model (FM) through (a) representation transfer and
(b) model transfer.

###  6.1. Domain knowledge transferred to FM through representation-level
knowledge transfer

The main goal of fine-tuning FMs with clients’ domain-specific knowledge
through federated learning is to obtain FMs that can generalize to (multiple
or many) domain-specific tasks with high accuracy and reliability.

Toward this end, CreamFL (Yu et al., 2023b) was proposed to learn an FM (i.e.,
$\text{BERT}_{\text{base}}$) by transferring representations from diverse
clients that own uni- and multi-modal data. In CreamFL, each DM and the FM are
split into top and bottom parts. Upon receiving global representations, each
client first trains its DM based on its local data and global representations.
Then, each client generates local representations using its domain bottom
model based on a public dataset and sends local representations to the server.
The server, in turn, aggregates local representations sent from clients with
global representations and trains the foundation bottom model through
knowledge distillation based on aggregated representations. Next, the server
sends global representations generated from the FM to clients for the next
round of training. The workflow of CreamFL is illustrated in Figure 11(a).
CreamFL generally adheres to the objective formulated in Eq.(3) with $K>1$,
$\sum^{K}_{k=1}p^{C_{k}}=1$, $p^{S}>0$, and $\eta=0$. The specific form of the
loss $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ for
transferring representations generated by $\mathcal{W}^{C_{k}}$ to train
$\mathcal{W}^{S}_{\text{FM}}$ is formulated in Eq.(11).

###  6.2. Domain knowledge transferred to FM through model-level knowledge
transfer

While representation transfer mitigates privacy leakage by retaining private
data locally, it still suffers from privacy risks because it sends sample-wise
representations of private data to the server. Therefore, the risks of leaking
data privacy by revealing these representations need to be carefully
investigated. An alternative way to transfer clients’ domain knowledge to the
server’s FM is through model transfer.

Offsite-Tuning (Xiao et al., 2023) is the representative work that leverages
model transfer combined with parameter-efficient fine-tuning to transfer the
knowledge of a downstream client to the FMs (i.e., BLOOM (Scao et al., 2022)
and OPT (Zhang et al., 2022c)) owned by a server while protecting both the
ownership of the server’s FM and the privacy of the client’s private data. In
Offsite-Tuning, the server first selects a small subset of its FM as the
adapter and compresses its FM into an emulator, and then it sends the adapter
and the emulator to the downstream client. The client, in turn, fine-tunes the
adapter on its private data with the assistance of the emulator. The fine-
tuned adapter is then returned and plugged into the FM to create an adapted
FM. As the server and the client do not share their full FM and private data,
respectively, Offsite-Tuning has the potential to protect both the privacy of
the client’s data and ownership of the server’s model. Offsite-Tuning (Xiao et
al., 2023) corresponds to the objective formulated in Eq.(3) with
$K=1,p^{C_{1}}=1,p^{S}=0,$ and $\eta=0$. The client has no DM but trains two
adaptors sent from the server with the help of an emulator compressed from
$\mathcal{W}^{S}_{\text{FM}}$. The specific form of the Offsite-Tuning loss is
formulated in Eq.(10).

While effective, Offsite-Tuning exhibits two limitations: (1) Offsite-Tuning
does not provide protection methods to explicitly protect data privacy, and
thus, the client’s private data can still be reconstructed by an adversary
when observing the adapter fine-tuned using the private data (Morris et al.,
2023); (2) Offsite-Tuning involves only one client.

To alleviate these limitations, Fan et al. (Fan et al., 2023) proposed a
federated learning version of Offsite-Tuning named FedOST. In FedOST, the
server first distributes the emulator and two adapters of its FM to all
clients at the beginning of FL. Then, each client fine-tunes the two adapters
on its private data with the assistance of the emulator. Next, all clients
send their adapters to the server, which aggregates received adapters through
secure aggregation (Bonawitz et al., 2016) and plugs the aggregated adapter
into the FM. FedOST protects the privacy of clients’ private data and can
build a fine-tuned FM with enhanced generalization capability. With similar
motivations as FedOST, Kuang et al. integrated Offsite-Tuning into
FederatedScope-LLM (Kuang et al., 2023b) and termed it FedOT. FedPEAT (Chua et
al., 2023) generalized the Offsite-Tuning approach to federated learning with
multiple communication rounds and proposed an adaptive control mechanism to
facilitate the adoption of FedPEAT in a dynamic environment. CEFHRI (Khalid et
al., 2023) adopted Offsite-Tuning to the human-robot interaction domain.

Similar to Offsite-Tuning, FedOST, FedOT, and FedPEAT adhere to the objective
formulated in Eq.(3) but with $K>1$, $\sum^{K}_{k=1}p^{C_{k}}=1$, and
$p^{S}=0$. FedOST has $\eta>0$ since it explicitly leverages secure
aggregation to protect data privacy. In general, the workflow of the three
methods can be illustrated in Figure 11(b).

###  6.3. Privacy and Backdoor Attacks

During the process of transferring domain-specific knowledge from clients to
the server, the server can potentially infer the private data of clients from
its observed information, while a client can poison the server’s FM by
injecting backdoors into the information transmitted to the server.
Nonetheless, privacy and backdoor attacks are rarely explored in this setting.
In this subsection, we discuss potential threats to data privacy and model
utility as well as possible countermeasures based on existing works.

In the Offsite-Tuning (Xiao et al., 2023), the server can potentially
reconstruct the client’s private data through the model inversion attack or
infer sensitive attributes through the attribute inference attack based on the
adapters sent from the client. DP-SGD (Abadi et al., 2016) and its variants
(Kairouz et al., 2021) are straightforward protection mechanisms against these
attacks. When more than two clients are involved in the offsite tuning, secure
aggregation can be employed to mitigate the privacy vulnerability of clients’
private data, as studied in FedOST (WeBank, [n. d.]).

In CreamFL (Wang et al., 2022a), the server can possibly infer the private
data of clients by investigating representations sent from clients. Takahashi
et al. (2023) undertook a preliminary investigation into the privacy
vulnerabilities of federated model distillation approaches that transfer
knowledge through logits (i.e., unnormalized probabilities of an instance
belonging to a certain class). They proposed a Paired-Logits Inversion (PLI)
attack to infer clients’ private data based on logits from a server model and
those from the client model. While PLI relies on logits to carry out the
attack, its rationale is worth verifying in inferring private data from
representations. To mitigate the chance for the adversary to infer private
data from logits or representations, a promising research direction is to
reduce the dependence between the private data and the logits or
representations shared with the adversary while maintaining the utility. Along
this direction, obfuscation mechanisms, such as Adaptive Obfuscation (Gu et
al., 2023) and Mutual Information Regularization (Zou et al., 2023), are worth
exploring.

On the other hand, the server is subject to backdoor attacks mounted by
clients. For example, the client in the Offsite-Tuning (Xiao et al., 2023) can
possibly poison the server’s FM by encoding backdoors into the adapters that
will be plugged into the server’s FM. The client in CreamFL (Wang et al.,
2022a) can embed backdoor triggers into the representations, which in turn
transfer the triggers to the server’s FM through fine-tuning. The backdoor
attacks through adapters and representations are rarely explored in literature
(Nguyen et al., 2024), thereby deserving investigation.

Table 7. Summary of FTL-FM works of setting ③. $\bigcirc$ ($\bigcirc^{\ddagger}$) denotes that a reference work intends to protect data or model against privacy (backdoor) attacks using certain protection methods presented in the ”How To Protect” columns. HE: Homomorphic Encryption; FE: Freeze Embedding; DP: Differential Privacy; NC: Norm Clipping; PEFT: Parameter-Efficient Fine-Tuning; SA: Secure Aggregation; SCR:Scrubbing. Transfer Method Category |  Reference |  What To Protect |  Exchanged Information | How To Protect
---|---|---|---|---
Client(s) |  Server(s) |  Client(s) To Server(s) |  Server(s) To Client(s) | Client |  Server
$\mathcal{D}^{C}$ | $\mathcal{W}^{C}$ | $\mathcal{D}^{S}$ | $\mathcal{W}^{S}_{\text{FM}}$
Homo- Model Transfer |  FedHE (Jin et al., 2023) |  $\bigcirc$ |  |  |  |  Partially HE- encrypted $\mathcal{W}^{C}$ |  Global model |  HE |
FL4ASR (Azam et al., 2023) |  $\bigcirc$ |  |  |  |  DP-trained $\mathcal{W}^{C}$ |  Global model |  DP |
FILM (Gupta et al., 2022) | $\bigcirc$ |  |  |  |  Gradients of $\mathcal{W}^{C}$ | Global model | FE |
LAMP (Balunovic et al., 2022) | $\bigcirc$ |  |  |  |  Gradients of $\mathcal{W}^{C}$ | Global model | DP |
Decepticons(Fowl et al., 2023), Panning(Chu et al., 2023) |  $\bigcirc$ |  |  |  |  Gradients of $\mathcal{W}^{C}$ |  Maliciously updated global model |  DP |
FLTrojan (Rashid et al., 2023) | $\bigcirc$ |  |  |  |  Maliciously updated $\mathcal{W}^{C}$ and benign $\mathcal{W}^{C}$ |  Global model |  DP & SCR |
Neurotoxin (Zhang et al., 2022b) |  |  |  | $\bigcirc^{\ddagger}$ |  Backdoored $\mathcal{W}^{C}$ | Global model |  | NC & DP
RE-GE (Yoo and Kwak, 2022) |  |  |  | $\bigcirc^{\ddagger}$ |  Backdoored $\mathcal{W}^{C}$ | Global model |  | NC & DP
FedCoLLM (Fan et al., 2023) |  $\bigcirc$ |  |  |  |  $\mathcal{W}^{C}$ |  Global model enhanced by $\mathcal{W}^{S}_{\text{FM}}$ |  SA |
CrossLM (Deng et al., 2023) |  |  |  |  |  $\mathcal{W}^{C}$ |  $\mathcal{W}^{C}$ enhanced by $\mathcal{W}^{S}_{\text{FM}}$ |  |
|  FedLLM on Edge (Woisetschläger et al., 2023a) |  |  |  |  |  $\mathcal{W}^{C}$ |  Global model |  |
Hetero- Model Transfer |  PEU+LoRA (Xu et al., 2023a) | $\bigcirc$ |  |  |  | LoRA model | Global LoRA model | DP |
DP-LoRA (Liu et al., 2023d) | $\bigcirc$ |  |  |  | LoRA model | Global LoRA model | DP |
FedPETuning (Zhang et al., 2023d) | $\bigcirc$ |  |  |  | PEFT model | Global PEFT model | PEFT |
FedPrompt (Zhao et al., 2023b) | $\bigcirc$ |  |  |  | Prompt model | Global prompt model | DP |
FedPPT (Zhao et al., 2023b) |  |  |  |  $\bigcirc^{\ddagger}$ |  Backdoored prompt model |  Global prompt model |  |
FedSplitBERT(Li et al., 2022b) |  |  |  |  |  Portion of $\mathcal{W}^{C}$ |  Global model |  |
FedDAT (Chen et al., 2024) |  |  |  |  | Dual-adapter |  Global dual-adapter |  |
FedIT (Zhang et al., 2023c) |  |  |  |  | LoRA model | Global LoRA model |  |
FedPepTAO (Che et al., 2023) |  |  |  |  | Prompt model | Global prompt model |  |
FedLPFM (Wu et al., 2023b) |  |  |  |  | Proxy model | Global proxy model |  |
CLIP2FL (Shi et al., 2024) |  |  |  |  |  CLIP-enhanced $\mathcal{W}^{C}$ and gradients of local classifier |  Global model |  |
HePCo (Halbe et al., 2023) |  |  |  |  |  Prompt model and classifier |  Global prompt model and classifier |  |
FedET (Liu et al., 2023b) |  |  |  |  |  Lightweight module group |  Global lightweight module group |  |
FeS (Cai et al., 2023a) |  |  |  |  | Bias model |  Global bias model |  |
FedAdapter (Cai et al., 2023b) |  |  |  |  | Adapter | Global adapter |  |
|  FwdLLM (Xu et al., 2023b) |  |  |  |  | LoRA model | Global LoRA model |  |
| HLoRA (Cho et al., 2023) |  |  |  |  |  Heterogeneous LoRA model | Global LoRA model |  |

##  7\. FTL-FM: Co-optimize Foundation Models and Domain Models

The FTL-FM settings we discussed in Section 5 and Section 6 focus on adapting
general knowledge of the FM to optimize DMs, and transferring domain-specific
knowledge of DMs to optimize FM, respectively. In this section, we explore the
setting ③, which endeavors to mutually optimize FMs and DMs hosted by
different parties (i.e., the server and clients). We summarize FTL-FM works
that fall into this setting in Table 7, and in the following subsections, we
will elaborate on these works.

###  7.1. Co-optimize FMs and DMs through model-level knowledge transfer

Optimizing both FMs and DMs helps establish a positive cycle to evolve FMs and
DMs (DAMO, [n. d.]) continuously. For example, an FM server can first deliver
the general knowledge and abilities of its FM to clients’ DMs, which are then
trained on domain-specific data for downstream applications. Clients’ DMs, in
turn, transfer industry-specific knowledge to enhance the server’s FMs. This
loop can go beyond one training task and continues over time. Nevertheless,
mutually enhancing FMs of the server and DMs of clients is rarely exploited in
FL literature. A more commonly studied scenario involves clients
collaboratively fine-tuning their domain models that are initialized with FMs.
In this section, we consider both two scenarios:

  1. (1)

The server owns an FM and clients own DMs. Clients’ DMs can optionally be
initialized with FMs. The objective of this scenario is to optimize the
server’s FM as well as the clients’ DMs. Consequently, the server ends up with
an FM augmented with domain-specific knowledge, and each client with a DM
enhanced with general knowledge of the FM.

  2. (2)

The server serves as an aggregation function while clients’ domain models are
initialized with or assisted by FMs. The objective of this scenario is to
fine-tune clients’ DMs through federated learning. Each client ends up with a
DM enhanced with the domain knowledge of all clients.

Figure 12. Illustration of co-optimizing server’s FM and clients’ FM through
transferring model-level knowledge among participating parties (i.e., server
and client). (a) illustrates the federated co-optimization of clients’ full
DMs and the server’s full FM. (b) illustrates federated parameter-efficient
co-optimization of clients’ DMs and the server’s FM when the clients
initialize their DMs with certain off-the-shelf FMs.

For the first scenario, FATE-LLM (Fan et al., 2023) proposed a FedCoLLM
(Federated Co-tuning LLM) framework (Fan et al., 2023) to co-optimize the
server’s FM and clients’ DMs simultaneously through mutual distillation.
Figure 12(a) illustrates the workflow of FedCoLLM. In FedCoLLM, clients’ DMs
are initialized with an off-the-shelf FM (i.e., LLaMA-7B (Touvron et al.,
2023)). In each communication round, clients fine-tune their DMs using local
data and then send them to the server with secure aggregation (Bonawitz et
al., 2016). The server, in turn, distills the knowledge mutually between its
hosted FM (i.e., LLaMa-65B) and the global DM aggregated from clients’ local
DMs to co-optimize both models. Subsequently, the server dispatches the global
DM to all clients for further training.  Deng et al. (2023) proposed CrossLM
that optimizes the server’s FM and clients’ DMs through synthetic data that
are generated by the server’s FM and validated by clients’ DMs, as illustrated
in Figure 12(b). Specifically, in each communication round, clients train
their DMs using local data and then send them to the server asynchronously.
The server, in turn, generates synthetic data and leverages received DMs to
validate the quality of synthetic data. Subsequently, the server utilizes
validation feedback from DMs and the validated synthetic data to fine-tune the
server’s FM. Next, the server uses the validated synthetic data to fine-tune
each client’s DM and dispatches the fine-tuned DMs to corresponding clients
for further training. This concerted effort is aimed at co-optimizing both the
FM and DMs. FedCoLLM and CrossLM adhere to the objective formulated in Eq.(4)
with $K>1$, $\sum^{K}_{k=1}p^{C_{k}}=1$, and $p^{S}=0$. FedCoLLM has $\eta>0$
whereas CrossLM has $\eta=0$. The specific form of the loss
$\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ for co-
optimizing the server’s $\mathcal{W}^{S}_{\text{FM}}$ and a client $k$’s
$\mathcal{W}^{C_{k}}$ is formulated in Eq.(12).

In addition to the model transfer method proposed by FedCoLLM, the
representation transfer method proposed by CreamFL (Yu et al., 2023b)
(illustrated in Figure 11(a)) also has the potential to achieve co-
optimization since representations can convey general knowledge and contextual
information of a model and thus can help improve the generalization capability
of both FMs and DMs.

Figure 13. Federated parameter-efficient-finetuning of clients’ domain-
specific FMs. Clients’ DMs are initialized with FMs and then learn from each
other through federated fine-tuning of their PEFT modules (e.g., adapter,
prompt model, and LoRA model). The server hosts no model and acts as an
aggregation function.

For the second scenario, FedNLP (Lin et al., 2022) provided a benchmarking
framework for evaluating federated learning algorithms (e.g., FedAvg, FedProx,
and FedOPT) on common NLP tasks with FMs (e.g., BART). Woisetschläger et al.
(2023b) provided a micro-level hardware benchmark for measuring Energy,
Computational, and Communication efficiency when fine-tuning the FLAN-T5 model
family, ranging from 80M to 3B parameters, using FL. Fine-tuning or training
clients’ full FMs is computationally expensive, primarily because of the large
sizes of FMs (Cai et al., 2023b). To alleviate this issue, various Parameter-
Efficient Fine-Tuning (PEFT) methods have been proposed. As illustrated in
Figure 13, these PEFT-based FL methods typically involve clients whose DMs are
initialized with pre-trained FMs. During training, each client only uploads a
PEFT module to the server for aggregation. The PEFT module can be an adapter,
LoRA model, or prompt model that contributes to a small portion of the
parameters of the original FM, thereby saving computational and communication
overheads. These PEFT-based FL methods correspond to the objective formulated
in Eq.(5) with $K>1$ and $\sum^{K}_{k=1}p^{C_{k}}=1$. The specific form of the
loss $\ell^{C_{k}}(\mathcal{W}^{S}_{\text{FM}},\mathcal{W}^{C_{k}};d)$ for
fine-tuning client $k$’s $\mathcal{W}^{C_{k}}$ with PEFT methods is formulated
in Eq.(13).

As for now, few works following Eq.(5) explicitly consider protecting data
privacy (i.e., $\eta>0$). FedPETuning (Zhang et al., 2023d), PEU+LoRA (Xu et
al., 2023a), DP-LoRA (Liu et al., 2023d), and FedPrompt (Zhao et al., 2023b)
are representative ones. FedPETuning leverages four PEFT techniques, namely
adapter tuning, prefix tuning, LoRA (Hu et al., 2022), and BitFit (Ben Zaken
et al., 2022), to fine-tune clients’ DMs (i.e.,
$\text{Roberta}_{\text{base}}$) and demonstrated that federated learning
combined with LoRA achieved the best privacy-preserving results. DP-LoRA and
PEU+LoRA apply LoRA with DP to fine-tune clients’ DMs, aiming to achieve
better privacy-utility-resource trade-offs than baselines. PEU+LoRA
additionally uses Partial Embedding Updates (PEU) to reduce the noise required
to protect privacy. Zhao et al. (2023b) proposed FedPrompt that leverages
federated learning combined with soft prompt tuning to fine-tune clients’ DMs
(e.g., T5 (Raffel et al., 2020)), aiming to reduce communication cost while
maintaining model accuracy. FedPrompt adopts local DP to protect the privacy
of clients’ data, demonstrating there is a trade-off between privacy and model
accuracy.

Most methods following Eq.(5) focus on improving computation and communication
efficiency (i.e., $\eta=0$). Chen et al. (2024) proposed FedDAT, in which each
client fine-tunes its local multi-modal model (e.g., VAuLT (Chochlakis et al.,
2022)) by leveraging a Dual-Adapter Teacher (DAT) module to achieve better
convergence rate than using conventional PEFT methods. Li et al. (2022b)
proposed FedSplitBERT, which splits the BERT encoder into a local part and a
global part and shares only a quantized version of the global part with the
server for aggregation, aiming to reduce the communication cost with minimal
performance loss. FedIT (Zhang et al., 2023c) was proposed to leverage
federated learning combined with LoRA to efficiently utilize clients’ diverse
instructions stored on local devices to improve the performance of global FM
(e.g., Vicuna (Chiang et al., 2023)). Che et al. (2023) proposed a parameter-
efficient prompt tuning approach with adaptive optimization, named FedPepTAO,
to fine-tune clients’ large language models (e.g., LLaMA-7B) in a non-IID
setting. Halbe et al. (2023) proposed HePCo to address a federated class-
incremental learning (FCIL) problem. In HePCo, each client is deployed with a
ViT model, and each client shares only a prompt model and a classifier (on top
of the ViT) with the server to save communication costs. To tackle
catastrophic forgetting brought by continual learning and data heterogeneity
across clients, the server trains a generator based on current and previous
tasks and distills the knowledge from the generator to the global prompt model
and classifier at each communication round.  FedET (Liu et al., 2023b) also
aims to address a FCIL problem. In FedET, each client leverages an enhancer
group, a set of lightweight modules, to transfer knowledge to and from the
server. Both clients and the server adopt a shared pre-trained Transformer to
ensure high precision on various tasks. To address catastrophic forgetting
caused by new classes introduced by new tasks and non-i.i.d. class
distribution across clients, FedET proposed an enhancer distillation method to
modify the imbalance between old and new knowledge and repair the non-i.i.d.
problem. Wu et al. (2023b) proposed Fed-LPFM, in which each client uses its
local data and a set of private FMs to train a proxy model, and clients’ proxy
models are shared with and aggregated by the server for federated training. At
inference, only the proxy models are used. As a result, the inference latency
can be considerably reduced, and the data distribution gap between clients can
be mitigated. Shi et al. (2024) proposed CLIP2FL that leverages an off-the-
shelf CLIP model to assist federated learning on heterogeneous and long-tailed
data. In the CLIP2FL, the local model of every client consists of a feature
extractor and a classifier. During each communication round, each client not
only trains its local models using local data but also distills knowledge from
a ViT-B/32 model into its local model. The server, on the other side,
generates federated features by aggregating classifier gradients and uses
these federated features to fine-tune a global classifier for all clients.

Besides the aforementioned works, a potentially more demanding and intriguing
avenue of research involves empowering edge devices (e.g., smartphones,
wearables, and Internet of Things devices) with the knowledge and capabilities
of FMs, thereby greatly augmenting the intelligence of systems facilitated by
these devices. Take, for instance, the case of smart home systems (King et
al., 2023) enhanced by FMs, which can elevate the quality of life for
individuals, or the case of surveillance systems (Yao et al., 2019) fortified
by FMs, leading to heightened efficiency and accuracy in detecting moving
objects such as people and traffic. To achieve these visions, the primary
challenge lies in effectively and efficiently adapting FMs to edge devices
that have limited memory and computing resources. Federated learning presents
a promising solution to this challenge, and several studies have investigated
this direction.

FedAdapter (Cai et al., 2023b) was proposed to improve the computational and
communication efficiency of fine-tuning local FMs (e.g., BERT) resided in
devices through PEFT (i.e., adapter). To achieve this, it dynamically adjusts
the depth and width of adapters during training. FeS (Cai et al., 2023a) was
proposed to address the problem of data scarcity in mobile NLP applications by
utilizing pseudo-labeling. It employs a curriculum pacing mechanism to
gradually speed up the pseudo-labeling speed, a representational filtering
mechanism to pseudo-label samples that are most beneficial to learn from, and
a co-planing mechanism to control the layer depth and capacity to be trained.
Furthermore, FeS fine-tunes only bias parameters of FMs (e.g., RoBERTa-large)
while keeping the other parameters frozen to improve training efficiency
further. FwdLLM (Xu et al., 2023b) aims to enhance the efficiency of PEFT
techniques. It observed that PEFT techniques mainly alleviate network
bottlenecks but do not significantly improve training efficiency as they
require backpropagation across the entire model to obtain gradients. To
address this challenge, FwdLLM employs a backward propagation (BP)-free method
to estimate gradients during the forward pass, thereby saving substantial
memory and training time. To tackle system heterogeneity, Cho et al. (2023)
proposed a heterogeneous LoRA (HLoRA) approach that can apply different rank
LoRA modules to devices with heterogeneous and constrained computing resources
by leveraging zero-padding and truncation for the aggregation. Woisetschläger
et al. (2023a) conducted an in-depth study on hardware performance
optimization for federated fine-tuning large language models at the edge. More
specifically, It investigated the state-of-the-art deep learning accelerators
as well as optimization levers for FL workloads on embedded hardware.

###  7.2. Privacy and Backdoor Attacks

During the federated learning training or fine-tuning process, clients are
subject to data reconstruction attacks initiated by the semi-honest or
malicious server. Gupta et al. (2022) and Balunovic et al. (2022) proposed
FILM and LAMP attacks that enable a semi-honest server to reconstruct a
client’s input text from the submitted gradients of the client’s FM, and they
proposed FWD (freezing word embeddings) and DP-SGD, respectively, to thwart
the proposed text reconstruction attacks. Fowl et al. (2023) and Chu et al.
(2023) proposed Decepticons and Panning attacks enabling the malicious server
to recover a client’s input text by sending a maliciously modified model to
clients to capture private information. The two studies suggested leveraging
DP-SGD to defend against the proposed malicious attacks. FLTrojan (Rashid et
al., 2023) explored scenarios in which a malicious client can leak the
privacy-sensitive data of some other clients in FL. Specifically, a malicious
client first identifies the training rounds in which the targeted victim is
involved and subsequently maximizes the victim model’s memorization of
privacy-sensitive data by manipulating selective weights that are responsible
for such memorization.

To study efficient privacy protection mechanisms against privacy attacks
mounted by the server, FedHE (Jin et al., 2023) proposed an efficient
Homomorphic encryption-based approach for FL that selectively encrypts the
most privacy-sensitive model parameters to reduce both computation and
communication overheads while providing customizable privacy preservation.
FL4ASR (Azam et al., 2023) investigated the crucial optimization factors,
including optimizers, training from a seed model pre-trained centrally or
starting from scratch, cohort size, and data heterogeneity, in the context of
FL with differential privacy applied to large automatic speech recognition
models.

On the other hand, the server may suffer from backdoor attacks mounted by
malicious clients. Zhang et al. (2022b) proposed Neurotoxin, a backdoor attack
that is performed by injecting backdoors into coordinates that benign clients
do not frequently update. Yoo and Kwak (2022) proposed a model poisoning
attack conducted by injecting triggers into the embedding of rare words and
utilizing gradient ensembling to enhance the poisoning capability. Both two
studies leveraged DP and Norm-Clipping to defend against the backdoor attack.

##  8\. Foundation Model Inference

After the completion of federated transfer learning, the server or the client
may expose its fine-tuned foundation model (FM) through an FM service API for
the public to make inferences. We refer to the entity that provides the FM
service as the FM provider and the entity that consumes the FM service as the
user. In this section, we delve into the privacy threats and potential
countermeasures of the FM inference phase, during which the user initiates
queries to the FM provider that, in turn, sends back the FM’s responses. It is
important to note that both the FM provider and the user can potentially act
as adversaries during this inference process.

Figure 14. The foundation model inference setting, in which the FM provider is
the adversary who aims to infer the user’s private data from the prompts
submitted by the user.

When the FM provider is the adversary, its goal is to infer private
information of the user by investigating the prompts sent by the user with the
assistance of the FM. The FM provider can infer privacy information from the
prompt of a single query (illustrated in Figure 14(a)) (Tong et al., 2023), or
it can steer a seemingly benign conversation with the user to lure the user
into providing prompts that can be exploited to learn private and sensitive
information 14(b)) (Tong et al., 2023).

In literature, most of the privacy-preserving methods that aim to protect the
user’s data privacy during inference focus on designing efficient Secure
Multi-Party Computation (SMPC) methods (Li et al., 2023d; Dong et al., 2023;
Liu and Liu, 2023; Zheng et al., 2023; Hou et al., 2023a; Hao et al., 2022;
Ding et al., 2023). Given that SMPC comes at a significant cost in terms of
computational resources and communication, these methods primarily focus on
optimizing the efficiency of FM architectures, predominantly Transformer-based
neural networks, or SMPC protocols. We will comprehensively review these SMPC-
based methods in Section 10. In a recent study, Tong et al. (2023) proposed
InferDPT, which leverages DP-guaranteed text perturbation on the user’s
prompts to prevent the adversary FM provider from inferring the user’s private
data. Specifically, in the InferDPT framework, the user first randomly
perturbs tokens of its prompt with semantically similar alternatives.
Subsequently, the user sends the perturbed prompt to an LLM (i.e., GPT-4),
which in turn generates a corresponding response and sends the response back
to the user. Finally, the user employs a local pre-trained language model,
such as Vicuna-7B, to generate the final response based on the original prompt
and the response received from the LLM. Staab et al. (2023) investigated
leveraging text anonymization to protect the user’s data privacy, and it
empirically demonstrated that anonymization is insufficient for protecting the
user’s data privacy.

Figure 15. The foundation model inference setting, in which the user is the
adversary who aims to design prompts and leverage these prompts to infer the
private information of other people.

When the user is the adversary, his/her goal is to extract the private
information of some targeted people from the FM (illustrated in Figure 15(a))
(Carlini et al., 2021) or from the private data sources used to augment the FM
(illustrated in Figure 15(b)) through privacy-invasive prompts (Panda et al.,
2023).

For defending against privacy-invasive prompts, Panda et al. (2023) proposed a
DP-ICL approach that allows the LLM to generate differentially private
responses through a noisy consensus among an ensemble of responses. Staab et
al. (2023) suggested that model alignment can be used to thwart privacy-
invasive prompting. Nonetheless, they also demonstrated that current LLMs are
not aligned against privacy-invasive prompts because much of the current
alignment research focuses primarily on preventing the LLM from generating
harmful and offensive content. Thus, leveraging alignment to safeguard the FM
provider’s privacy during inference presents a promising avenue for further
research.

##  9\. Improving Efficiency

Foundation models have become the fundamental infrastructure that drives AI
applications across various industries. Nevertheless, adapting large-scale FMs
to domain-specific AI applications incurs prohibitive costs. To address this
challenge, a series of research works have been focusing on designing and
developing efficient fine-tuning and knowledge transfer methods to adapt FMs
to downstream tasks or augment FMs with domain-specific knowledge. In this
section, we review parameter-efficient fine-tuning and efficient knowledge
transfer methods, which play an important role in grounding FMs.

###  9.1. Parameter-Efficient Fine-Tuning

The straightforward way to adapt general-purpose foundation models (FMs) to
downstream tasks is to fine-tune all the model parameters (full fine-tuning).
However, This is prohibitively expensive for many domain-specific tasks. To
mitigate this issue, Parameter-Efficient Fine-Tuning (PEFT) methods (Houlsby
et al., 2019)(He et al., 2021) are proposed to efficiently adapt FMs to
specific domains or tasks. To this end, PEFT methods fine-tune only a small
amount of model parameters while freezing the rest model parameters.

Various PEFT methods have been proposed. Specifically, adapter tuning (Houlsby
et al., 2019) inserts small modules called adapters to each layer of an FM and
only adapters are trained during the fine-tuning stage. Inspired by the
success of prompting methods that control FM through textual prompts, prompt
tuning (Lester et al., 2021) and prefix tuning (Li and Liang, 2021) prepend
additional $k$ tunable prefix tokens to the input or hidden layers and only
train these tunable tokens on domain-specific tasks. LoRA (Hu et al., 2022)
freezes the FM weights and injects trainable low-rank decomposition matrices
into each layer of the Transformer. In general, PEFT has several advantages
over full fine-tuning, including:

  * •

Improved efficiency: PEFT, by only updating a small number of parameters, can
significantly reduce the training time and computational resources required to
achieve the same model performance as full fine-tuning.

  * •

Better generalization: PEFT can generalize better to new tasks or domains
because PEFT retains pre-trained knowledge and prevents the fine-tuning
process from drastically changing the model’s initial representations.

  * •

Better performance: PEFT can often achieve similar or even better performance
than full model fine-tuning because PEFT helps avoid overfitting.

While PEFT techniques can alleviate network bottlenecks to some extent, they
fail to substantially enhance training efficiency as they still necessitate
backpropagation across the entire model to acquire gradients. Recognizing this
limitation, recent endeavors have emerged to tackle this predicament. For
instance, Xu et al. (2023b) proposed a groundbreaking approach that eliminates
the need for backward propagation, thereby saving both time and space for
training. Similarly, Sung et al. (2022) introduced a ladder side-tuning method
that circumvents the requirement of backpropagation throughout the entire
model by confining it solely to the side network and ladder.

###  9.2. Efficient Knowledge Transfer

Although PEFT significantly boosts the computational cost of training FMs, it
typically requires a vast amount of storage to deploy FMs that typically have
a colossal size of parameters, which prevents FMs from being deployed in
resource-constrained environments, such as mobile phones. Knowledge transfer
methods such as distillation and pruning have emerged as promising solutions
to alleviate this challenge. However, traditional knowledge transfer
techniques (Gou et al., 2021) alone may not be sufficient to accurately
satisfy the flexible requirements of downstream clients in the era of FMs. For
example, downstream clients are likely equipped with different computing
powers and storage environments, and thus, they can afford quite different
resources to transfer the knowledge from the upstream FM server. Advanced
knowledge distillation has become an intensive research area to alleviate the
computation and storage burdens of FM clients.

Figure 16. The high-level overview of different categories of knowledge
transfer methods.

In this section, we review these advanced knowledge transfer methods, and we
categorize them into progressive transfer, adaptive transfer, and emergent-
ability-based transfer based on how they transfer the knowledge from teacher
models to student models. Unlike traditional knowledge transfer methods that
typically perform a unidirectional knowledge transfer from teacher models
hosted by a server to student models for downstream clients’ tasks (see Figure
16(a)), progressive transfer, adaptive transfer, and emergent-ability-based
transfer typically leverage feedback from clients or student models to improve
the efficiency and effectiveness of the knowledge transfer.

####  9.2.1. Progressive Knowledge Transfer

Progressive Knowledge Transfer (PKT) methods transfer the knowledge of a
teacher model (e.g., an FM) into a student model (e.g., a DM) in an iterative
manner. More specifically, Instead of transferring all the knowledge from a
teacher model to a student model at one training stage, PKT breaks the
transferring procedure down into multiple stages. Each stage involves training
a student model to mimic the behavior of the teacher model at a specific level
of model complexity, as illustrated in Figure 16(b). The advantage of PKT is
that it allows for more efficient and effective knowledge transfer from the
server’s teacher model to the downstream client’s student model while
satisfying the flexible resource constraints of clients compared to
traditional knowledge distillation methods.