# A Survey on Heterogeneous
Federated Learning

Dashan Gao, Xin Yao, and Qiang Yang Dashan Gao is with the Department of
Computer Science and Engineering, HKUST, Hong Kong, China and the Department
of Computer Science and Engineering, Southern University of Science and
Technology, Shenzhen, China, e-mail<EMAIL_ADDRESS>Yao is with the
Department of Computer Science and Engineering, Southern University of Science
and Technology, Shenzhen, China, e-mail<EMAIL_ADDRESS>Yang is with
the Department of Computer Science and Engineering, HKUST, Hong Kong, China,
e-mail<EMAIL_ADDRESS>
###### Abstract

Federated learning (FL) has been proposed to protect data privacy and
virtually assemble the isolated data silos by cooperatively training models
among organizations without breaching privacy and security. However, FL faces
heterogeneity from various aspects, including data space, statistical, and
system heterogeneity. For example, collaborative organizations without
conflict of interest often come from different areas and have heterogeneous
data from different feature spaces. Participants may also want to train
heterogeneous personalized local models due to non-IID and imbalanced data
distribution and various resource-constrained devices. Therefore,
heterogeneous FL is proposed to address the problem of heterogeneity in FL. In
this survey, we comprehensively investigate the domain of heterogeneous FL in
terms of data space, statistical, system, and model heterogeneity. We first
give an overview of FL, including its definition and categorization. Then, We
propose a precise taxonomy of heterogeneous FL settings for each type of
heterogeneity according to the problem setting and learning objective. We also
investigate the transfer learning methodologies to tackle the heterogeneity in
FL. We further present the applications of heterogeneous FL. Finally, we
highlight the challenges and opportunities and envision promising future
research directions toward new framework design and trustworthy approaches.

###### Index Terms:

Heterogeneous Federated Learning, Vertical Federated Learning

##  I Introduction

In the last decade, machine learning (ML) has achieved tremendous success and
is widely applied to numerous areas, thanks to the surge of big data. Despite
the great success of ML, the broad application of ML is heavily restricted by
the lack of large-scale datasets. Moreover, the data collection and labeling
can be expensive, making it infeasible for a single organization to collect
enough data. In other cases, although companies may already have sufficient
data regarding the number of recordings. They may want to enrich the feature
space of their data by incorporating the auxiliary features distributed among
other institutions. However, sharing data between institutions or aggregating
data from user devices leads to the risk of privacy leakage.

In recent years, concerns about data privacy protection have been emerging due
to continuous privacy violations. In response, information privacy policies
and laws such as GDPR [1], and CCPA [2] stipulate on sharing of data between
companies to protect personal data from being abused. Moreover, data also has
enormous commercial values for companies. It is infeasible to transmit data
out of privacy, security, and business concerns. Therefore, it is challenging
to efficiently leverage data distributed in multiple institutions for ML.

Federated learning (FL) was proposed to train ML models based on data
distributed among clients [3, 4]. In FL, intermediate results such as
gradients are exchanged and aggregated to update models without raw data
transmission. Many efforts have been devoted to designing federated learning
algorithms from various aspects, including privacy preservation, robustness,
efficiency, security, scalability, and performance [5, 6, 4, 7, 8].

However, FL faces a problem of heterogeneity. Heterogeneity in FL comes from
various aspects, including imbalanced data distribution, heterogeneous feature
spaces, unstable network connectivity, and limited device resources. We
categorize the heterogeneity in FL into three types: data space, statistical,
and system heterogeneity. All types of heterogeneity have attracted tremendous
research interest in recent years. To tackle the heterogeneity as mentioned
above, various approaches that leverage model heterogeneity are also widely
explored, as heterogeneous models can help train tailored model architectures
according to the feature space and device resources of parties. Some surveys
have focused on statistical heterogeneous FL [9, 10] and system heterogeneous
FL [11, 12]. Some works investigate the system design of FL [13] However,
there is still a lack of comprehensive survey on all the above-mentioned
heterogeneity in FL.

To tackle statistical heterogeneity, some works train a robust global model
against the non-IID and unbalanced distributions in different participants
[14, 15, 16, 7]. Another approach is to train personalized local models for
all parties by formulating the FL as multi-task learning problems [17, 18,
19]. Similarly, to tackle system heterogeneity, some approaches propose to
efficiently build a single global model via client selection [20, 21, 11],
asynchronous aggregation [22, 23], and model compression [24, 25]. When
different resource-constraint devices want to deploy a personalized local
model, it is possible to train heterogeneous local models by model splitting
and clustered training [26, 27, 28].

Although most studies focus on statistical and system heterogeneity, the data
spaces of different participants may differ as well. In many real-world
applications, the collaborating participants are organizations (e.g., banks or
hospitals) from different industries without conflict of interest. Therefore,
the collected data of each participant are from heterogeneous feature spaces.
For example, in advertisement scenarios, an advertiser holds users’ browsing
history and clicking records, while a businesses hold the purchase history. To
leverage data from both feature spaces, the two parties must conduct data
space heterogeneous FL to build a model on the joint feature space. Such
heterogeneity in data space brings new challenges to FL in various aspects.

According to data space distribution, data space heterogeneous FL can be
categorized into vertical federated learning (VFL) and heterogeneous federated
transfer learning (Hetero-FTL). When data are partitioned by features, VFL
aims to train a model on the aligned instances. However, participants may
share limited or no aligned instances and some common features in many cases.
Such overlap on feature space or instance space provides opportunities to
transfer knowledge among parties. Fortunately, federated transfer learning is
proposed to adopt transfer learning to share knowledge among the participants
[29]. According to the shared data space among participants, we categorize
Hetero-FTL into three cases: feature-sharing, instance-sharing, and label-
sharing. Compared to VFL, Hetero-FTL relaxes the requirement on instance
alignment and improves the model availability. We find that the data space
heterogeneous FL, especially the Hetero-FTL is still under-explored and is a
promising research direction that attracts increasing research interests.

We investigate the approaches to tackle different types of heterogeneity in
FL. There are mainly five approaches to tackle heterogeneity: knowledge
distillation, data augmentation, parameter sharing, domain adaptation, and
matrix factorization. For each approach, we discuss and review how to use them
to address each type of heterogeneity.

We also investigate the applications of heterogeneous FL in different areas,
seeing it being actively promoted for industrial applications. Furthermore, we
conclude the major challenges and promising future directions in heterogeneous
FL algorithm designing.

Overall, our contributions are threefold:

  1. 1.

We make the first survey on the existing works about data space, statistical,
system, and model heterogeneity. We propose a taxonomy for each type of
heterogeneity based on the problem setting and learning approach.

  2. 2.

We identify five transfer learning strategies to address data space,
statistical, and system heterogeneity in FL and review the literature from
data-based, architecture-based, and model-based perspectives.

  3. 3.

We envision promising future research directions toward new framework design
and trustworthy approaches toward building heterogeneous FL systems.

Organization. The remainder of the survey is organized as shown in Figure 1.
Section II gives an overview of FL and introduces the heterogeneity, trade-
off, and categorization of FL. Section III proposes the notion of data space
heterogeneous FL and provide a categorization method according to the data
distribution of the heterogeneous FL. Section IV introduces the statistical
heterogeneity. Section V discusses the system heterogeneity. Section VI
introduces how to train heterogeneous models in FL and how model heterogeneity
can help to tackle data space, statistical, and system heterogeneity. Section
VII discusses how to use different transfer learning approaches to tackle
different types of heterogeneity in FL. Section VIII presents the applications
of heterogeneous FL. Section IX discusses the challenges in heterogeneous FL.
Finally, Section X draws the conclusion. Table I lists the key acronyms and
abbreviations used throughout the paper.

Figure 1:  Organization of this survey.  Acronyms | Definitions
---|---
FL | Federated Learning
HFL | Horizontal Federated Learning
VFL | Vertical Federated Learning
Hetero | Heterogeneous
Homo | Homogeneous
TL | Transfer Learning
FTL | Federated Transfer Learning
ML | Machine Learning
HE | Homomorphic Encryption
DP | Differential Privacy
MPC | Secure Multiparty Computation
TABLE I: List of Key Acronyms

##  II An Overview of Federated Learning

In this section, we introduce the basic idea of FL. We first give the
definition of FL. We categorize the FL according to the data distribution
among parties. From the view of data distribution, we follow the
categorization from [29].

###  II-A Definition of Federated Learning

We provide a formal definition of FL with the reference of [29, 13, 30]. We
assume there are $N$ parties denoted as $\\{P_{1},P_{2},...,P_{N}\\}$ with
their private isolated datasets
$\\{\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{N}\\}$, respectively.
There can be three components in a complete dataset: the feature space
$\mathcal{X}$, the label space $\mathcal{Y}$ and the ID space $\mathcal{I}$.
The feature space $\mathcal{X}$ and ID space $\mathcal{I}$ of the parties may
not be identical.

Different FL problems differ in the distribution of data among parties. We
denote a party $P_{i}$ with a complete dataset
$\mathcal{D}_{i}=\\{X_{i},Y_{i},I_{i}\\}$ as an active party, and a party
$P_{j}$ with data $\mathcal{D}_{j}=\\{X_{j},I_{j}\\}$ as a passive party. An
active party can train a model locally to predict the label. In contrast, a
passive party can only provide auxiliary features to other parties via ID
alignment and cannot train a valid model on its own. In the canonical FL
problem, a.k.a. HFL, all parties are active parties and hold homogeneous
labeled data where $\mathcal{X}_{i}=\mathcal{X}_{j}$ and
$\mathcal{Y}_{i}=\mathcal{Y}_{j}$ for $\forall i,j\in[1,N]$. In heterogeneous
FL, different parties may hold heterogeneous data. For example, in vertical
FL, $\mathcal{X}_{i}\cap\mathcal{X}_{j}=\emptyset$ for $\forall i,j\in[1,N]$,
and $\mathcal{Y}_{k}\neq\emptyset,\mathcal{Y}_{i}=\emptyset$ for $\forall
i\in[1,...,k-1,k+1,...,N]$. We demonstrate the detailed categorization of FL
problems in Section II-D.

For the local setting, a.k.a. distributed on-site learning[31], each active
party $P_{i}$ uses its own local data $\mathcal{D}_{i}$ to train a machine
learning model $\mathcal{M}_{i}$. The predictive accuracy of $\mathcal{M}_{i}$
is denoted as $\mathcal{V}_{i}$. For the centralized setting, all parties
aggregate and align their data together to get
$\mathcal{D}_{agg}=\bigcup\limits_{i=1}^{N}\mathcal{D}_{i}$. A model
$\mathcal{M}_{cen}$ is trained over the aggregated data $\mathcal{D}_{agg}$ in
a centralized fashion. In contrast, FL allows the parties to train a model
$\mathcal{M}_{fed}$ collaboratively on the whole dataset $\mathcal{D}_{agg}$,
while each party $P_{i}$ does not disclose its raw data $\mathcal{D}_{i}$ to
any other party. The predictive accuracy of $\mathcal{M}_{agg}$ and
$\mathcal{M}_{fed}$ are denoted as $\mathcal{V}_{agg}$ and
$\mathcal{V}_{fed}$, respectively. Then, an FL system should satisfy the
following properties:

  1. 1.

For a valid FL system, for local models trained by active parties $P_{act}$,
the performance of the federated model should be better than the performance
of any local model, i.e.

$\mathcal{V}_{fed}\geq\mathcal{V}_{i},\;\forall i\in P_{act}.$

  2. 2.

The difference between the performance of federated model $\mathcal{M}_{fed}$
and the centralized model $\mathcal{M}_{agg}$ should be bounded by $\delta>0$,
i.e.

$|\mathcal{V}_{agg}-\mathcal{V}_{fed}|\leq\delta,\;\exists\delta>0.$

###  II-B Heterogeneity in Federated Learning

In FL systems, heterogeneity is one of the most significant challenges and is
studied from multiple aspects, including 1) data space heterogeneity, 2)
statistical heterogeneity, and 3) system heterogeneity.

  * •

Data Space Heterogeneity: The data space, including feature space
$\mathcal{X}$, label space $\mathcal{Y}$ and ID space $\mathcal{I}$ can differ
among the parties in FL due to data availability, domain difference, and
device heterogeneity issues. Parties with different feature spaces or label
spaces can not train a shared model like traditional HFL. Parties who share
common instance IDs can leverage auxiliary features from other parties in
training and inference processes. In another case, parties without common
instance IDs can explore to transfer knowledge from other feature spaces to
boost the model performance in the training process. Heterogeneous FL
techniques investigated in this survey aim to tackle data space heterogeneity
via vertical and heterogeneous federated transfer learning. (Section III).

  * •

Statistical Heterogeneity: Data collected from the same data space can be non-
IID (independent and identically) distributed in the FL system. For example,
autonomous vehicles run in environments varying from rural areas to urban
areas. This leads to the marginal distributions $P(\mathcal{X})$ of collected
data differ. Moreover, different drivers in the same environments show various
driving style preferences, leading to varying conditional distributions
$P(\mathcal{Y}|\mathcal{X})$ of human driving behaviors. In addition, class
imbalance and data size imbalance across parties may vary significantly as
well. (Section IV).

  * •

System Heterogeneity: In FL systems, heterogeneity that happens at the system
level can also exert a significant influence on the FL training process.
Devices may differ in hardware (CPU, GPU, memory), network connectivity (WiFi,
4G, 5G), and power supply (battery mode), which leads to various computation,
communication, and storage capabilities. (Section V).

###  II-C Utility-Privacy-Efficiency Trade-off

The design of FL algorithms faces three major objectives: model performance
(utility), data privacy, and system efficiency. Improving one objective
usually leads to the decline of other objectives. Multiple factors can affect
such a trade-off. Here we identify six major factors in FL that affect the
trade-off: model complexity (e.g., linear model, decision tree, neural
networks), cryptographic privacy (e.g., HE, MPC), and obfuscation-based
privacy (DP), sharing labels, asynchronization, and coordinator. We summarize
the impact of each factor on the utility-privacy-efficiency trade-off in Table
II. Generally speaking, model performance can be improved by increasing model
complexity. Privacy can be improved by cryptographic or obfuscation-based
approaches. Efficiency can be improved via sharing labels, asynchronization,
and introducing a coordinator. We elaborate on each factor as follows.

Factor | Utility | Privacy | Efficiency
---|---|---|---
model complexity | $\bm{\uparrow}$ | $\downarrow$ | $\downarrow$
cryptographic privacy | - | $\bm{\uparrow}$ | $\downarrow$
obfuscation-based privacy | $\downarrow$ | $\bm{\uparrow}$ | $\downarrow$
asynchronization | $\downarrow$ | $\downarrow$ | $\bm{\uparrow}$
coordinator | - | $\downarrow$ | $\bm{\uparrow}$
sharing label | - | $\downarrow$ | $\bm{\uparrow}$
TABLE II: Impact of different factors on utility-privacy-efficiency trade-off.
$\uparrow$, -, and $\downarrow$ denote increase, no change, and decrease,
respectively.

####  II-C1 Utility.

As the model complexity increases from linear models to collaborative
filtering models, decision trees, and neural networks. The gradients and model
parameters contain more information about private training data. Therefore,
the risk of privacy leakage increases. Moreover, more complex models can
achieve higher utility with more significant computation and communication
costs.

####  II-C2 Privacy.

Cryptographic approaches protect data privacy by encrypting data and
intermediate results during FL. As no numerical error is introduced,
cryptographic approaches only trade efficiency for privacy. As one exception,
CKKS scheme [32] allows approximated decryption with little numerical error,
and it achieves much more efficient homomorphic operations on encrypted data
than previous fully HE schemes. Obfuscation-based approaches such as DP
protects the membership privacy of data by adding noise. The added noise not
only decreases model performance but also requires more iterations for
convergence. Therefore, they trade utility and efficiency for data privacy.

Figure 2:  Two strategies to improve FL algorithms in the trade-off space: 1)
Reduce system redundancy. 2) Trade one objective with little decline for
another with significant benefit. Smaller values are preferred. Each red dot
indicates an FL algorithm.

####  II-C3 Efficiency.

Some works explore asynchronous updates to improve training efficiency.
However, the plain-text asynchronous update may lead to utility decline and
privacy leakage to the server if the gradients are not protected by
cryptographic approaches. Some attempts [33] on asynchronization identified
and reduced system redundancy to improve efficiency without sacrificing model
performance and data privacy. A few works on VFL [34, 35] assume the share
labels of the active party are shared among all passive parties. Such an
assumption trades data privacy for training efficiency. Introducing a central
coordinator for model aggregation can avoid high peer-to-peer communication
costs when there are multiple participants in FL. However, the coordinator
might infer private information from participants. Therefore, introducing a
coordinator trades privacy for efficiency.

From the view of multi-objective optimization, the design of FL algorithms is
to search the Pareto front in the utility-privacy-efficiency space. Although
some research claims that there is no free lunch for the three objectives in
FL [36] and optimizing one object leads to decline of other objectives, we can
still optimize the objectives in two aspects. 1) First, we can identify the
redundancy in the FL algorithms and reduce the redundant operations and
communication delays as in [33]. Therefore, a newly proposed solution can
dominate previous FL algorithms in the utility-privacy-efficiency space. 2)
Second, when the Pareto front is highly convex, we can find the keen points by
proposing non-dominated approaches to trade one objective with a little
decline for another with significant benefit, thus achieve higher overall
benefit. For example, the CKKS scheme [32] achieves higher computational
efficiency with little or negligible utility decline via approximated
decryption. Figure 2 takes utility and privacy as an example to demonstrate
the two strategies of FL algorithm design. Each red dot indicates an FL
algorithm.

###  II-D A Categorization of Federated Learning

We categorize the typical scenarios of FL according to the data space
heterogeneity. According to the feature space in each party, FL can be divided
into Data Space-Homogeneous FL and Data Space-Heterogeneous FL. In this
section, we demonstrate the details of each category. We demonstrate the
categorization of FL in Figure 3

Figure 3: Categorization of FL in terms of data space heterogeneity.

####  II-D1 Data Space Homogeneous Federated Learning

In data space-homogeneous FL, the data spaces $\mathcal{D}_{i}$ in each party
$P_{i}$ are homogeneous in terms of both feature space $\mathcal{X}$ and label
space $\mathcal{Y}$. Homogeneous FL is widely applied in cross-device
scenarios. The devices (e.g., IoT devices and mobile phones) independently
collect private data sharing the same feature space and label space. Since it
was proposed, FL is meant to study the case where multiple clients holding
homogeneous data train a single global model collaboratively under specific
privacy constraints. Therefore, most existing researches focus on homogeneous
FL. Research on homogeneous FL can be categorized into two cases: horizontal
federated learning and homogeneous federated transfer learning. The major
difference is that, in HFL, there is one learning objective, and parties
collaboratively train a global model on all data. However, in homogeneous
federated transfer learning, each party optimizes its objective function and
builds its model. Data distribution in homogeneous FL can be summarized as
follows:

$\mathcal{X}_{i}=\mathcal{X}_{j},\mathcal{Y}_{i}=\mathcal{Y}_{j},\mathcal{I}_{i}\neq\mathcal{I}_{j},\;\;\forall
i,j\in[1,N],i\neq j.$

We discuss the two cases in the following. Interested readers may refer to
[30, 37] for more details on data space homogeneous FL.

a) Horizontal federated learning (HFL). HFL handles multi-client data that
share the same set of features but are different in samples and is also
referred to as federated optimization. The pioneering works [3, 4, 38] in FL
mainly focus on this setting, where participants collaboratively train a
global model based on their data with the help of a coordinator for model
aggregation. The goal of HFL is to train a high-quality centralized global
model. The objective can be formulated as:

$\min_{w\in\mathbb{R}^{d}}\left\\{f(w):=\frac{1}{n}\sum_{i=1}^{N}f_{i}(w)\right\\},$

where $w$ is the parameters of the federated model, and
$f_{i}:\mathbb{R}^{d}\to\mathbb{R}$ is the training objective function that
maps the model parameter set $w\in\mathbb{R}^{d}$ to a real valued training
loss with respect to the training data $\mathcal{D}_{i}$ of party $P_{i}$.

b) Homogeneous federated transfer learning (Homo-FTL). Different from HFL,
which learns a single model for all parties even if the data is non-IID across
different parties, homogeneous federated transfer learning aims to transfer
knowledge from the other parties to some or each party. Homogeneous federated
transfer learning optimizes multiple objectives to achieve the best
performance for each client with non-IID data. Therefore, homogeneous
federated transfer learning is highly related to multi-task learning,
personalization, and meta-learning. A variety of multi-model approaches are
proposed to tackle the non-IID data problem. We formulate the homogeneous
federated transfer learning as:

$\min_{W\in\mathbb{R}^{d\times
N}}\left\\{\mathcal{G}(W):=\sum_{i=1}^{N}f_{i}(w_{i})\right\\},$

where $W=[w_{1},...,w_{N}]$ is a $d\times N$-dimensional matrix that collects
$w_{1},...,w_{N}$ as its columns.

####  II-D2 Data Space Heterogeneous Federated Learning

Here we briefly introduce the data distribution, categorization, and major
application scenarios of heterogeneous FL to compare with homogeneous FL. In
Section III, we will discuss heterogeneous FL in depth. The data spaces
$\mathcal{D}_{i}$ held by parties $P_{i}$ are non-identical in heterogeneous
FL. Heterogeneous FL is common in cross-silo settings where organizations
collaborate to leverage the heterogeneous features held by other parties for
accurate prediction. Unlike cross-device settings where numerous devices, each
with limited samples, collaborate to boost data size, parties in cross-silo
settings generally hold a large dataset but are from different domains.
Therefore, tackling the lack of features is of the key importance in cross-
silo settings. Heterogeneous FL provides a promising approach to leveraging
different features in different parties. According to learning objectives and
the distribution characteristics of the data spaces between parties,
heterogeneous FL can be categorized into Vertical Federated Learning and
Heterogeneous Federated Transfer Learning.

a) Vertical federated learning (VFL). Vertical FL enables different parties
holding multi-view data partitioned by feature to collaboratively train
machine learning models without exposing one’s own data [29]. It addresses the
data silos and privacy problems together. Data distribution in vertical FL can
be summarized
as:$\mathcal{X}_{i}\cup\mathcal{X}_{j}=\emptyset,\;\mathcal{Y}_{1}\neq\emptyset,\;\mathcal{Y}_{k}=\emptyset,\;\mathcal{I}_{i}=\mathcal{I}_{j},\;\forall
i\neq j\in[1,N],\;\forall k\in[2,N].$ Here, party $P_{1}$ is the active party
that owns label space, and the other parties $P_{2},...,P_{N}$ are passive
parties that only own feature spaces. We denote that, in some cases, the
active party may only hold the label space $\mathcal{Y}_{1}$ and has no
feature $\mathcal{X}_{1}=\emptyset$.

The active party first sends the user ID to all parties in the inference
process. Then each party inputs their own features to the model and
collaboratively predicts the result. It is worth noting that if the user ID
does not exist in all parties, the model inference can not be directly
conducted without imputation. Therefore, the viability of the vertical
federated model dramatically decreases as the number of parties increases due
to the lack of sufficient linked training data and the hardness of alignment.

b) Heterogeneous federated transfer learning (Hetero-FTL). Instead of
collecting different features from separate parties to train a more accurate
model, heterogeneous federated transfer learning aims to transfer knowledge
from the heterogeneous data spaces in other parties to boost the performance
of the local model. Therefore, heterogeneous federated transfer learning
releases the ID alignment requirement in vertical FL. When the inference data
is not aligned between parties, the parties in heterogeneous FL can leverage
the trained local model for inference. Since the model is trained by
transferring knowledge from other parties, the model can achieve higher
performance than that trained solely on local data. In heterogeneous federated
transfer learning, data can be linked in three ways. 1) parties have an
overlap of ID space but no intersection of feature spaces. 2) parties have an
intersection of feature space but no overlap of ID spaces. 3) parties have
neither feature space overlap nor ID space overlap. In section III, we will
discuss heterogeneous FL in depth.

####  II-D3 Summary

We summarize the main objective of each category of FL in Table III. There has
been many surveys investigating papers that tackle non-IID data in FL, which
is statistical heterogeneity [9, 13, 39, 30]. To our best knowledge, there is
no survey investigate on data space heterogeneous FL, especially data space
heterogeneous federated transfer learning. To mitigate this gap, we deliver a
comprehensive investigation on data space heterogeneity in Section III. We
summarize the transfer learning techniques used to tackle data space
heterogeneity in Section VII.

Categorization | Data | Objective
---|---|---
Data Space Homogeneous FL |  HFL |  IID/ Non-IID, homo-feature | One single global model
Homo-FTL | Non-IID, homo-feature | Personalized local models for all parties
Data Space Heterogeneous FL | VFL | hetero-feature | One single global model
Hetero-FTL | hetero-feature | Personalized local models for all parties
TABLE III: Comparison of different FL settings.

###  II-E Security Definition and Privacy Threats

####  II-E1 Security Definition

In the cryptographic study, based on the extent to deviate from the protocol,
the adversaries in FL can be categorized into two types: semi-honest (a.k.a
honest-but-curious) adversaries and malicious adversaries [40]. The
definitions of security levels corresponding to each type of adversary are
listed below:

  * •

_Semi-honest security_ : The adversaries strictly follow the prespecified
protocol without deviation. However, they will collect all received
intermediate data and try to derive knowledge from it. Protocols achieving
semi-honest security prevent inadvertent information leakage between parties.

  * •

_Malicious Security_ : The adversaries can arbitrarily deviate from the
protocol in their attempt to cheat. The adversaries can only cause the honest
parties to abort. Furthermore, if the honest parties obtain output, they are
guaranteed that it is correct. Their privacy is always preserved.

####  II-E2 Privacy Threats

The idea of information privacy is to have control over the collection and
processing of one’s information. In ML tasks, the participants usually take up
three different roles: 1) as the input party, e.g., the data owner, 2) as the
computation party (e.g., the model builder and inference service provider),
and 3) as the result party (e.g., the model querier and user) [41].

Attacks on ML may happen at any stage, including data publishing, model
training, and model inference. Adversaries can attack each stage to breach
data privacy and infer private knowledge such as model parameters, raw data,
and membership. We introduce three types of attacks on data privacy: model-
inversion attack, data-reconstruction attack, and membership-inference attack.
We will discuss each attack in the following.

  * •

_Model-inversion attack_ can happen in the model training stage and model
inference stage. The adversary may attempt to reconstruct the model parameters
from intermediate gradients or model predictions or train a mimic model for
malevolent purposes.

  * •

_Data-reconstruction attack_ can happen in the data publishing stage and model
training stage. The computation party aims to reconstruct the data providers’
raw data or learn more information about the data providers than what the
model builders intend to reveal.

  * •

_Membership-inference attack_ can happen in the data publishing stage, model
training stage, and model inference stage. The goal is to know whether an
instance is inside the training set of the model. The adversary is assumed to
have black-box access or white-box access to the model and an instance.

###  II-F Privacy Preservation Techniques

There are four types of privacy preservation approaches widely used for
privacy preserving machine learning, namely 1) secure multi-party computation
(MPC), 2) homomorphic encryption (HE), 3) differential privacy (DP), and 4)
trusted execution environment (TEE). We elaborate these techniques in Appendix
A.

##  III Data Space Heterogeneity

In data space heterogeneous FL, the feature spaces $\mathcal{X}_{i}$ held by
parties $P_{i}$ are non-identical. Such a setting is meaningful and practical
in various cross-silo industrial applications, such as recommendation,
advertisement, fraud detection, and credit extension. Unlike HFL, it allows
non-competitive participants from different areas to cooperate with their
unique features. For example, a loan company with demographic features, an
e-commercial company with customers’ purchase history, and an online video
company with users’ movie browsing history can cooperate and leverage the
heterogeneous features via vertical FL to boost the performance of credit
extension business of the loan company. According to the learning objectives
and the distribution characteristics of the data spaces between parties, we
categorize heterogeneous FL into Vertical Federated Learning (VFL) and
Heterogeneous Federated Transfer Learning (Hetero-FTL), as demonstrated in
Figure II-D. We will elaborate on them in the following.

###  III-A Vertical Federated Learning

VFL enables multiple parties with multi-view data partitioned by feature to
collaboratively train machine learning models without violating data privacy
and confidentiality [29]. It addresses the data silos and privacy problems
together. The collaborating parties own data of the same set of
users/instances but with disjoint features, and only one party holds the
labels. Taking the collaboration between a book recommender and an auxiliary
data provider as an example, the intersection of their ID spaces is large. The
book recommender records the consumer’s rating, browsing history, and item
features, including book titles, authors, descriptions, and publishers. At the
same time, an auxiliary data provider owns the demographic information such as
age, phone number, location, and gender. Their feature spaces are very
different. Incorporating auxiliary user information helps the book recommender
predict more precise recommendations. Therefore, the book recommender can
collaborate with the auxiliary data provider to conduct VFL to train a model
on the combination of the feature spaces of the two parties. Figure 4
demonstrates the data distribution in VFL.

Figure 4: Data distribution in VFL

######  Definition III.1.

(Vertical Federated Learning). Given $N$ parties and each party $P_{i}$
contains a set of features and labels of some instances, i.e.,
$\mathcal{D}_{i}=\\{\bm{X}_{i},\bm{I}_{i},\bm{Y}_{i}\\}$. $P_{i}$ has features
$\bm{X}_{i}\in\mathcal{X}_{i}$, instances $\bm{I}_{i}\in\mathcal{I}_{i}$, and
labels $\bm{Y}_{i}\in\mathcal{Y}_{i}$. $P_{1}$ is an active party that holds
labels. The data distribution is:

$\displaystyle\mathcal{X}_{i}\cup\mathcal{X}_{j}=\emptyset,\;\mathcal{Y}_{1}\neq\emptyset,\;\mathcal{Y}_{k}=\emptyset,\;\mathcal{I}_{i}=\mathcal{I}_{j},$
$\displaystyle\forall i\neq j\in[1,N],\;\forall k\in[2,N].$

VFL aims to train a model without revealing private data of each party:

$\operatorname*{arg\,min}_{\bm{W}\in\mathbb{R}^{d\times
N}}\;L\left(\bm{Y}_{1},f_{\bm{W}}\left(\bm{X}_{1},z(\\{\bm{X}_{i=2}^{N}\\})\right)\right),$

where $\bm{W}=[\bm{w}_{1},...,\bm{w}_{N}]$ is a $d\times N$-dimensional matrix
that collects sub-models $\bm{w}_{1},...,\bm{w}_{N}$ of all parties as its
columns, $f_{\bm{W}}(\cdot,\cdot)$ is the prediction model parameterized by
$\bm{W}$ is the training objective function, $z(\cdot)$ stands for the data
processing function that exchanges intermediate values in training process.
And $L(\cdot,\cdot)$ is a loss function that maps the model parameter set
$\bm{W}$ to a real valued training loss with respect to the training data
$\\{\bm{X}_{i}\\}_{i=1}^{N}$ of all parties $\\{P_{i}\\}_{i=1}^{N}$.

We denote that, in some cases, the active party $P_{1}$ may only hold the
label space $\mathcal{Y}_{1}$ and has no feature $\mathcal{X}_{1}=\emptyset$.
In practice, parties need to find the instance IDs that co-occur in all
parties as training instances. Then, the shared instances are linked to form
the training instances of each party. Exchanging instance IDs with other
parties can leak membership information, which is not acceptable in VFL.
Privacy-preserving instance alignment allows different parties to find shared
instances. Meanwhile, it protects the membership privacy of records or users.
We will introduce instance alignment in Section III-A1.

The active party first sends the user ID to all parties in the inference
process. Then each party inputs its features to the model and collaboratively
predicts the result. It is worth noting that if the user ID does not exist in
all parties, the model inference can not be directly conducted without
imputation. Therefore, the availability of the vertical federated model
dramatically decreases as the number of parties increases due to the lack of
sufficient aligned training data.

####  III-A1 Private Instance Alignment

Recall that instance alignment is essential in the data preprocessing phase of
vertical federated model training. If the instance IDs of training data are
exchanged for instance alignment, one party can easily learn all membership
information from the other, leading to severe privacy violations. Delegating
the instance alignment to an external party is also risky and infeasible in
reality. Privacy-preserving instance alignment techniques, such as Private Set
Interaction (PSI) [42, 43], Privacy-Preserving Record Linkage (PPRL) [44, 45],
and Private Entity Resolution (PER) [46] can be applied to tackle this issue.
In PSI, separate parties compare encrypted versions of their sets to compute
the intersection. Neither party reveals anything to the others except for the
elements in the intersection. For example, Bellare et al. [47] introduce a
keyed-hash message authentication code to encrypt the IDs for comparison. PSI
requires exact matching of identifiers among parties and is suitable for
identifiers like phone numbers, national identification numbers, or social
security numbers. Nevertheless, instances can be linked by combining features
with errors, such as name, birthday, and location. PPRL and PER allow fuzzy
matching that tolerates errors in the identifiers. Schnell et al. [48] adopt
Bloom filters to perform fuzzy private record linkage on identifiers by
computing string similarities. Angelou et al. [49] implement a cross-platform,
open-source library for asymmetric PSI and PSI-cardinality, which combines
DDH-based protocols with compression based on Bloom filters to reduce
communication in preprocessing. Harty et al. [50] utilize Cryptographic
Longterm Key (CLK) to compute encrypted identifiers and use Dice coefficient
to match instances. After alignment, neither party knows the aligned instances
except for an encrypted mask. The training process with the encrypted ID mask
is costly as all un-aligned instances should be used for training. Since the
instance alignment process is independent of the training process,
participants can still dynamically align new data and add the aligned new data
for training in the training process.

####  III-A2 Existing Studies

In this section, we categorize the model-building approaches based on their
base models. Then, we also discuss the privacy leakage in VFL. Finally, we
demonstrate and compare the existing studies in VFL.

a) Linear Models. The study of training a regression model in privacy-
preserving distributed learning over vertically partitioned data can trace
back to Sanil [51], where the authors use secret sharing in a secure linear
regression model to ensure privacy in their approach. Liu et al. [52] apply
stochastic block coordinate descent to VFL. Before exchanging the intermediate
results, each party updates its local parameters for multiple rounds. Harty et
al. [50] leverage entity resolution and Paillier’s homomorphic encryption to
conduct vertical logistic regression between two parties with privacy against
semi-honest adversaries. Feng et al. [53] propose MMVFL, where each party
locally predicts labels based on its own features. It also studies the feature
selection problem in the vertical federated linear model.

Traditional VFL requires synchronous computation, which is inefficient. To
tackle this problem, Zhang et al. [35] integrate with backward updating
mechanism and bi-level asynchronous parallel architecture to train a linear
model. The convergence rates of the proposed algorithms are theoretically
analyzed under strongly convex and non-convex conditions. The security against
a semi-honest adversary is proved as well. Xu et al. [54] proposed FedV for
secure gradient computation for linear models, logistic regression, and
support vector machines. It introduces an aggregator and uses functional
encryption schemes to reduce communication among parties. Li et al. [55]
introduce hybrid differential privacy to protect membership differential
privacy for conducting vertical federated logistic regression. Xu et al. [56]
also applied differential privacy to vertical federated linear regression. Gu
et al. [57] study the asynchronous multi-party vertical federated linear model
by proposing a tree-structured communication protocol. They also provide the
convergence rate of the proposed AFSGD-VP approach. Chen et al. [58] and Gao
et al. [59] apply factorization machine with second-order feature interaction
into vertical federated recommender systems.

b) Neural Networks. Hu et al. [60] consider a multi-view learning setting,
where all parties hold the labels. Instead of exchanging the intermediate
results, it aggregates each participating party’s plain-text local prediction
results. The intermediate local prediction results for each training or
testing sample of each local party are sent to a server for aggregation. To
protect private local features against malicious servers or clients, DP is
applied by perturbing updated local predictions. Vepakomma et al. [61] study
the VFL on neural networks by proposing SplitNN, where a neural network is
divided into two parts. Each party trains several layers and shares the output
with the server in every iteration. A framework for multi-headed SplitNN is
implemented [62]. Chen et al. [63] study vertical asynchronous FL with
differential privacy based on neural networks. However, the labels of the
active party are transmitted to an honest server in [62, 63]. Zhou et al. [64]
investigate graph neural networks for privacy-preserving node classification.
Recently, Liu et al. [65] introduce the stochastic block coordinate descent
algorithm to VFL to achieve communication efficient training process. The
proposed FedBCD can prevent an adversary from inferring exact raw training
data and has superior convergence than standard VFL algorithms.

c) Decision Trees. Recently, many studies used decision trees to design
algorithms for VFL. This is because tree-based models are feasible to be
adapted to VFL by nature. Du et al. [66] first propose a decision tree for
vertically partitioned data by using a scalar product protocol to compute
information gained at a node. The labels are shared with all parties for
training. Wang et al. [34] also assume the labels in the training data could
be shared with all parties, which is counterfactual in most real-world cases.
They propose a secure decision tree construction trained in plain text by
applying PSI. Li et al. [67] leverage locality-sensitive hashing into vertical
federated GBDT to protect the privacy of each party with relaxed privacy
constraints. Additive homomorphic encryption is adapted for privacy protection
in several frameworks based on GBDT [68, 69, 33]. The work in [70] studies
privacy-preserving federated vertical decision trees by proposing Pivot. Pivot
provides protection against an honest-but-curious adversary without relying on
any trusted third party based on threshold Paillier encryption and secret
sharing. It identifies two privacy leakages of the plain-text trained model
and proposes to use differential privacy and message authentication code to
enhance the protocol. Pivot can also be extended to ensemble models such as
random forest and gradient boost decision trees. Liu et al. [71] propose
Federated Forest based on the CART tree and bagging. Each party stores the
split information with respect to its own features. The proposed approach
achieves comparable performance to the centralized version. Fu et al. [33]
designed an efficient vertical federated GBDT algorithm named VF2Boost, by
proposing a concurrent training protocol and customized cryptographic
operations to speed up homomorphic encryption. Federated gradient boosting
models sequentially build a boosting model with the base learner, which can be
communication inefficient and time-consuming. Therefore, Han et al. [72]
propose to integrate boosting and bagging to build multiple decision trees in
parallel. Chen et al. [73] explore the interpretability in the inference phase
for decision tree ensembles in VFL by canceling decision paths and adapting an
efficient, secure inference method.

| Utility | Privacy | Efficiency
---|---|---|---
VFL Studies | model type | privacy mechanism | share labels | asynchronous | aggregator
Sanil et al. [51] | LM | MPC | × | × | ×
Zhang et al. [35] | LM | MPC | ✓ | ✓ | ✓
Harty et al. [50] | LM | HE | × | × | ✓
Xu et al. [54] | LM | HE | × | × | ✓
Li et al. [55] | LM | DP | × | × | ✓
Xu et al. [56] | LM | DP | × | × | ✓
Liu et al. [52] | LM | PT | × | ✓ | ×
Gu et al. [57] | LM | PT | × | ✓ | ×
Feng et al. [53] | LM | PT | × | × | ✓
Chen et al. [58] | FM | MPC | × | × | ✓
Gao et al. [59] | FM | HE, MPC | × | × | ×
Liu et al. [65] | NN, LM | PT | × | ✓ | ×
Hu et al. [60] | NN, LM | DP | × | × | ✓
Vepakomma et al. [61] | NN | PT | × | ✓ | ×
Romanini et al. [62] | NN | PT | × | ✓ | ×
Chen et al. [63] | NN | DP | × | ✓ | ✓
Du et al. [66] | DT | PT | × | × | ✓
Wang et al. [34] | DT | PT | ✓ | × | ✓
Cheng et al. [68] | DT | PT | × | × | ×
Feng et al. [69] | DT | PT | × | × | ×
Wu et al. [70] | DT | HE, MPC | × | × | ×
Li et al. [67] | DT | LSH | × | × | ✓
Liu et al. [71] | DT | HE | × | × | ✓
Fu et al. [33] | DT | HE | × | ✓ | ✓
TABLE IV: Comparison among existing works of VFL. LM denotes Linear Models. FM
denotes Factorization Machines. NN denotes Neural Networks. DT denotes
Decision Trees. HE denotes Homomorphic Encryption. MPC denotes Secure
Multiparty Computation. DP denotes Differential Privacy. PT denotes Plaintext.
LSH denotes locality-sensitive hashing.

d) Privacy Leakage in VFL. The privacy leakage in VFL is studied [74, 75, 59].
Gao et al. [59] demonstrate that in a plain-text vertical federated
factorization machine, participants can infer model parameters as well as raw
training data according to intermediate results. The privacy threats in
vertical matrix factorization are investigated as well [75, 76]. Weng et al.
[74] design reverse sum attack and reverse multiplication attack to steal the
private training data in linear models without affecting the model accuracy.

Several works explored the label inference attack in VFL [77, 78]. Fu et al.
[78] proposed three label inference attack methods: passive model completion,
active model completion, and direct label inference attack. In model
completion attacks, the attacker requires extra auxiliary labeled samples or
sample-level gradients for fine-tuning its local model. Liu et al. [77]
consider a more challenging scenario where the communication between
participants in VFL is black-boxed (e.g., encrypted via HE). In the proposed
gradient inversion label inference attack and defense, the attacker only has
access to the gradients of the final local model.

####  III-A3 Summary

We summarize and compare the reviewed studies in VFL in Table IV.

###  III-B Heterogeneous Federated Transfer Learning

Although VFL boosts model performance by aligning the features from isolated
parties, it is hard to collect sufficient aligned data for training in the
real world. On the other hand, VFL can not conduct inference on un-aligned
test data. Therefore, it is desirable to leverage the numerous un-aligned
heterogeneous data for FL and inference on un-aligned test data.

Heterogeneous federated transfer learning (Hetero-FTL) aims to transfer
knowledge from the heterogeneous feature spaces of other parties to boost the
performance of the local model. Hetero-FTL relaxes the ID alignment
requirement in VFL, and transfers the knowledge of the un-aligned training
data to other parties to improve the performance of their local models. When
the test data are not aligned among all parties, the parties holding features
can leverage the local model trained in Hetero-FTL for inference. Since the
model is trained by transferring knowledge from other parties, the model can
achieve higher performance than that trained solely on local data.

In Hetero-FTL, we identify three settings according to the way data are
associated:

  * •

Instance-sharing Hetero-FTL. Parties share an overlap of ID space for a part
of instances, while there is no intersection of their feature spaces. In
contrast to VFL, parties can utilize the un-aligned local data to train local
models here. Moreover, the models learned in instance-sharing Hetero-FTL can
conduct inference on un-aligned test data with solely local features.

  * •

Feature-sharing Hetero-FTL. Parties have no overlap of ID spaces, while there
is an intersection of their feature spaces. The shared feature space is
leveraged to transfer knowledge between heterogeneous feature spaces of
different parties.

  * •

Label-sharing Hetero-FTL. Parties may have an overlap of neither feature space
nor ID space. However, the label space in their tasks is the same. Therefore,
the parties can each train a model to map their heterogeneous feature spaces
into a common latent space. Then, they collaboratively build a model to
project the common latent space to the label space via HFL.

In each setting, the association between parties is limited and weak. It
brings new challenges to transferring knowledge among privacy-sensitive
parties in heterogeneous FL. We identify four transfer learning methods
adopted by existing Hetero-FTL research, including representation learning,
knowledge distillation, collaborative filtering, and auxiliary public dataset.
We will elaborate on each setting in the following. For each setting, we
categorize and review the existing works according to the transfer learning
methods.

####  III-B1 Instance-sharing Hetero-FTL

In instance-sharing Hetero-FTL, there are some aligned data samples between
parties, making it possible to transfer knowledge across parties. However,
different parties have no shared feature. The data distribution in instance-
sharing Hetero-FTL is demonstrated in Figure 5.

(a) single-active party

(b) multi-active party

Figure 5: Data distribution in instance-sharing Hetero-FTL

######  Definition III.2.

(Instance-sharing Hetero-FTL). Given $N$ parties and each party $P_{i}$
contains a set of features and labels of some instances, i.e.,
$\mathcal{D}_{i}=\\{\bm{X}_{i},\bm{I}_{i},\bm{Y}_{i}\\}$. $P_{i}$ has features
$\bm{X}_{i}\in\mathcal{X}_{i}$, instances $\bm{I}_{i}\in\mathcal{I}_{i}$, and
labels $\bm{Y}_{i}\in\mathcal{Y}_{i}$. There are $k$ active parties $\exists
k\in P_{act}=\\{k|k\in[1,N]\\}\neq\emptyset$, where $P_{act}$ is a non-empty
set that contains the index of active parties. The data distribution is:

$\displaystyle\mathcal{X}_{i}\cap\mathcal{X}_{j}=\emptyset,\;\mathcal{I}_{i}\cap\mathcal{I}_{j}\neq\emptyset,\mathcal{Y}_{k}\neq\emptyset,\;$
$\displaystyle\forall i\neq j\in[1,N],\;\exists k\in
P_{act}=\\{k|k\in[1,N]\\}\neq\emptyset.$

Instance-sharing Hetero-FTL aims to train a model for each active party
without revealing private data of each party:

$\displaystyle\operatorname*{arg\,min}_{\bm{W}\in\bm{R}^{d\times N}}\sum_{i\in
P_{act}}\lambda_{i}L\Bigl{(}\bm{Y}_{i},f_{\bm{W}}\bigl{(}$ $\displaystyle
I_{\cap},\bm{X}_{i},z(\\{\bm{X}_{j}\\}_{j\in[1,N]\setminus\\{i\\}})$
$\displaystyle|\\{\mathcal{D}_{j}\\}_{j\in[1,N]\setminus\\{i\\}}\bigr{)}\Bigr{)},$

where $\lambda_{i}$ is the weight for balancing the performance on each party,
$I_{\cap}$ is the index of overlapped instances.
$\bm{W}=[\bm{w}_{1},...,\bm{w}_{N}]$ contains local models
$\\{\bm{w}_{i}\\}_{i\in[1,N]}$ corresponding to each party
$\\{P_{i}\\}_{i\in[1,N]}$.

After federated transfer learning, the trained model $f_{\bm{W}}(\cdot)$
predicts both aligned and un-aligned target data. For aligned instances, model
$f_{\bm{W}}(\cdot)$ takes features from both active and passive parties as
input for prediction, which is similar to VFL. However, for un-aligned
instances, model $f_{\bm{W}}(\cdot)$ leverages local model $\bm{w}_{i}$ to
predict on local features $\bm{X}_{i}$. In contrast, VFL can only predict on
aligned test data, which greatly restricts its application to real-world
problems where most test data are unaligned. Moreover, the prediction
performance of $f_{\bm{W}}(\cdot)$ over local un-aligned data outperforms the
naive model trained locally, as instance-sharing Hetero-FTL transfers
knowledge from other parties to local models $\bm{w}_{i}$ in active party
$P_{i}$.

The instance-sharing Hetero-FTL problems differ according to the number of
active parties. Specifically, there are two representative cases in instance-
sharing Hetero-FTL:

  * •

For the single-active party case, we have
$\mathcal{Y}_{1}\neq\emptyset,\mathcal{Y}_{k}=\emptyset,\;\;\forall
k\in[2,N]$. All $N$ parties collaboratively train parameters to improve the
performance on $\mathcal{D}_{1}$. Figure 5(a) demonstrate the single-active
party case.

  * •

For the multi-active party case, we have $\mathcal{Y}_{i}\neq\emptyset,\forall
i\in[1,N]$. Each active party has labeled data and the objective is to train a
model that improves performance for each active party by federated knowledge
transfer. Figure 5(b) demonstrates the multi-active party case.

a) Application examples. Faced with the fact that most instances can hardly be
aligned across multiple parties in the real world, it is desirable to
collaboratively train ML models with limited aligned data and no common
features. Instance-sharing Hetero-FTL is a promising relief from the stringent
restriction on data alignment in VFL. As illustrated in Figure 6, a
recommender A wants to build a recommendation system with the help of an
auxiliary data provider B. The recommender A holds the user’s rating, browsing
history, and item features such as price, category, color, etc. At the same
time, the auxiliary data provider B holds the demographic information,
including occupation, education, gender, etc. The recommender and the data
provider have a part of users in common. These common users can be aligned by
privacy-preserving instance alignment III-A1.

The recommender aims to boost the performance of the recommender system by
leveraging the data from the auxiliary data provider in a privacy-preserving
way. Since only limited aligned users are in training data, VFL cannot achieve
high accuracy and cannot be applied to un-aligned test data. The instance-
sharing Hetero-FTL helps build a better recommender system by making full use
of all data.

Figure 6: Overview of a two-party instance-sharing Hetero-FTL problem. In this
example, party A has a labeled dataset, and party B has some auxiliary
features. Some of the data co-occur in both parties’ datasets and can be
aligned. The two parties aim to build a recommender system collaboratively
with data from both parties. However, they cannot exchange their private data
due to privacy and security concerns.

b) Existing studies. Various transfer learning approaches have been explored
to tackle instance-sharing Hetero-FTL problems, including parameter sharing
and knowledge distillation. We elaborate on these approaches in Section VII.

####  III-B2 Feature-sharing Hetero-FTL

In feature-sharing Hetero-FTL, different parties have some features in common.
However, the ID spaces are disjoint. Thus the data samples cannot be aligned
across parties. We show the data distribution in feature-sharing Hetero-FTL in
Figure 7. To transfer knowledge to the active parties, the shared common
features can be used as a pivot to bridge knowledge from different domains.

######  Definition III.3.

(Feature-sharing Hetero-FTL). Given $N$ parties and each party $P_{i}$
contains a set of features and labels of some instances, i.e.,
$\mathcal{D}_{i}=\\{\bm{X}_{i},\bm{I}_{i},\bm{Y}_{i}\\}$. $P_{i}$ has features
$\bm{X}_{i}\in\mathcal{X}_{i}$, instances $\bm{I}_{i}\in\mathcal{I}_{i}$, and
labels $\bm{Y}_{i}\in\mathcal{Y}_{i}$. There are $k$ active parties $\exists
k\in P_{act}=\\{k|k\in[1,N]\\}\neq\emptyset$, where $P_{act}$ is a non-empty
set that contains the index of active parties. The data distribution is:

$\displaystyle\mathcal{X}_{i}\cap\mathcal{X}_{j}\neq\emptyset,\;\mathcal{I}_{i}\cap\mathcal{I}_{j}=\emptyset,\mathcal{Y}_{k}\neq\emptyset,\;$
$\displaystyle\forall i\neq j\in[1,N],\;\exists k\in
P_{act}=\\{k|k\in[1,N]\\}\neq\emptyset.$

Feature-sharing Hetero-FTL aims to train a model for each active party without
revealing private data of each party:

$\displaystyle\operatorname*{arg\,min}_{\bm{W}\in\bm{R}^{d\times N}}\sum_{i\in
P_{act}}\lambda_{i}L\Bigl{(}\bm{Y}_{i},f_{\bm{W}}\bigl{(}$ $\displaystyle
X_{\cap},\bm{X}_{i},z(\\{\bm{X}_{j}\\}_{j\in[1,N]\setminus\\{i\\}})$
$\displaystyle|\\{\mathcal{D}_{j}\\}_{j\in[1,N]\setminus\\{i\\}}\bigr{)}\Bigr{)},$

where $X_{\cap}$ is the index of overlapped features, $\lambda_{i}$ is the
weight for balancing the performance on each party, and
$\bm{W}=[\bm{w}_{1},...,\bm{w}_{N}]$ contains local models
$\\{\bm{w}_{i}\\}_{i\in[1,N]}$ corresponding to each party
$\\{P_{i}\\}_{i\in[1,N]}$.

Just as instance-sharing Hetero-FTL is an extension of vertical FL by
leveraging un-aligned private data for model training, feature-sharing Hetero-
FTL can also be considered as an extension of HFL by taking advantage of
private local features for model training. Before federated training, all
parties first find co-existing features. During model training, each party
locally trains a compact model with all local features as input. In FL,
different parties aggregate the model parameters of the common features and
compute an aggregated model. Therefore, the knowledge is transferred among
parties via the aggregation of the parameters of common features.

Figure 7: Data distribution in feature-sharing Hetero-FTL

a) Application examples. The federated recommendation system is a typical
application scenario of feature-sharing Hetero-FTL. As shown in Figure 8, a
transnational e-commercial company aims to build a recommendation system with
user-item interaction data collected from two countries. It is infeasible to
exchange raw data among parties subject to legal constraints. The datasets
from the two countries share the same set of products and different users, as
shown in Figure 8(a). The feature space shown in Figure 8(b) demonstrates that
each data record contains a user feature and an item feature. Since the two
parties share the same item features, the model weights of item features can
be aggregated when training a federated recommendation system.

Similarly, in another scenario, two recommenders may share the same set of
users but own different items. For example, a movie recommender and a book
recommender in the same country collaboratively build recommendation systems
with common users. The user features are shared and thus can be used to
transfer knowledge between recommenders. Some existing work [79] categorize
the first user-sharing federated recommendation as HFL and the second item-
sharing federated recommendation as VFL. However, Gao et al. [80] denote that
both federated recommendation scenarios are the same in feature space and are
typical feature-sharing Hetero-FTL problems.

(a) Data distribution

(b) Feature space

Figure 8: A typical scenario of feature-sharing Hetero-FTL. A transnational
e-commerce company builds a federated recommender system with data from two
countries. In this example, the company sells the same set of products (items)
in two countries with different users. When using the matrix factorization
algorithm for FL, the two parties share the same item features and different
user features.

b) Existing studies. There are mainly three types of transfer learning
approaches used to tackle feature-sharing Hetero-FTL problems, including data
augmentation [81], parameter sharing [82], and collaborative filtering [76,
83, 58, 59, 84]. Linear models and collaborative filtering approaches are
widely explored to tackle the feature-sharing Hetero-FTL problem. Most
practical algorithms are designed and implemented for federated
recommendation, which is the major application scenario in the feature-sharing
Hetero-FTL problem. However, deep models and three-based models are still
under-explored in feature-sharing Hetero-FTL. We elaborate on these approaches
in Section VII.

####  III-B3 Label-sharing Hetero-FTL

Label-sharing Hetero-FTL considers a more challenging scenario in terms of
data distribution, where all parties have (almost) neither co-occurring
instances/IDs nor co-existing features, and they only share the same label
space $\mathcal{Y}$. We demonstrate the data distribution in label-sharing
Hetero-FTL in Figure 9. To transfer knowledge between parties, the shared
label space could be used as a pivot to bridge different domains during
collaborative training.

######  Definition III.4.

(Label-sharing Hetero-FTL). Given $N$ parties and each party $P_{i}$ contains
a set of features and labels of some instances, i.e.,
$\mathcal{D}_{i}=\\{\bm{X}_{i},\bm{I}_{i},\bm{Y}_{i}\\}$. $P_{i}$ has features
$\bm{X}_{i}\in\mathcal{X}_{i}$, instances $\bm{I}_{i}\in\mathcal{I}_{i}$, and
labels $\bm{Y}_{i}\in\mathcal{Y}_{i}$. The data distribution of label-sharing
Hetero-FTL is:

$\mathcal{X}_{i}\cap\mathcal{X}_{j}=\emptyset,\;\mathcal{I}_{i}\cap\mathcal{I}_{j}=\emptyset,\mathcal{Y}_{i}=\mathcal{Y}_{j},\;\forall
i\neq j\in[1,N],$

Label-sharing Hetero-FTL aims to train a model for each active party without
revealing private data of each party:

$\operatorname*{arg\,min}_{\bm{W}\in\bm{R}^{d\times N}}\sum_{i\in
P_{act}}\lambda_{i}L\left(\bm{Y}_{i},f_{\bm{W}}\left(\bm{X}_{i}|z(\\{\mathcal{D}_{j}\\}_{j\in[1,N]\setminus\\{i\\})}\right)\right),$

where $\lambda_{i}$ is the weight for balancing the performance on each party,
and $\bm{W}=[\bm{w}_{1},...,\bm{w}_{N}]$ contains local models
$\\{\bm{w}_{i}\\}_{i\in[1,N]}$ corresponding to each party
$\\{P_{i}\\}_{i\in[1,N]}$.

Figure 9: Data distribution in label-sharing Hetero-FTL

a) Application examples. Electroencephalographic (EEG) signals are collected
from multiple electrodes on the head and are privacy-sensitive. As shown in
Figure 10, two companies with different devices that collect EEG signals from
different positions want to collaboratively train emotion prediction models
without privacy violations. Their features are heterogeneous in terms of both
channel number and electrode position, and there is no common user to align
the data. Therefore, the two companies can leverage label-sharing Hetero-FTL
to map their heterogeneous features into a common latent subspace according to
label distribution; then, they can train a shared emotion prediction model.

Figure 10: An example of label-sharing Hetero-FTL. Two EEG devices collect
signals from different positions on head and aim to do the same emotion
prediction task.

b) Existing studies. There are mainly two types of transfer learning
approaches used to tackle the label-sharing Hetero-FTL issue: 1) parameter
sharing, 2) knowledge distillation. We denote that these strategies can be
combined to boost performance [85, 86]. Although there are only a few works
explicitly study label-sharing Hetero-FTL, some recent studies on system
heterogeneity in FL could be adapted to label-sharing Hetero-FTL problems, as
they propose to train a heterogeneous local model architecture for each party
[87, 88, 85, 86]. We elaborate these approaches in Section VII.

####  III-B4 Summary

We summarize the utility-privacy-efficiency trade-off of the reviewed studies
in data space heterogeneous federated transfer learning (Hetero-FTL) in Table
V. We also list some related works proposed for Non-FL settings that can be
modified to tackle data space heterogeneity. It can be observed that each
aspect of Hetero-FTL is still under-explored. Most existing studies emphasize
model performance (utility) and efficiency more than privacy, as they prefer
to train plain-text neural networks without a rigorous privacy guarantee. Many
of them were initially proposed to tackle system heterogeneity. Therefore, it
is desirable to explore tailored Hetero-FTL algorithms with stringent privacy
protection.

|  |  | Utility | Privacy | Efficiency
---|---|---|---|---|---
Transfer Method | Setting | Existing Study |  model type |  privacy mechanism |  share labels |  asynchronous |  aggregator
Data Augmentation | FS |  Gao et al. [81] | LM | HE,MPC | × | × | ×
IS |  Kang et al. [89] | NN | PT | ✓ | × | ×
Parameter Sharing |  |  Liu et al. [90] | NN | HE,MPC | × | × | ×
IS |  Sharma et al. [91] | NN | MPC | × | × | ×
|  He et al. [92] | NN | PT | × | × | ✓
|  * Hou et al. [93] | LM | PT | - | - | -
|  Ju et al. [94] | NN | PT | × | ✓ | ×
LS |  Gao et al. [95] | NN | PT | × | ✓ | ×
|  Liang et al. [85] | NN | PT | × | ✓ | ✓
FS |  * Zhu et al. [82] | LM | PT | - | - | -
* Ye et al. [96] | LM | PT | - | - | -
Knowledge Distillation |  |  Ren et al. [97] | NN | PT | × | × | ✓
|  Wang et al. [98] | NN | PT | × | × | ✓
|  Li et a. [99] | NN | PT | × | × | ✓
IS |  * Zhang et al. [100] | NN | PT | - | - | -
|  * Peng et al. [101] | NN | PT | - | - | -
|  * Tian et al. [102] | NN | PT | - | - | -
|  * Guo et al. [103] | NN | PT | - | - | -
Matrix Factorization | IS,FS |  Chai et al. [84] | MF | RM,MPC | ✓ | × | ✓
|  Chai et al. [76] | MF | HE | × | ✓ | ✓
|  Ammad et al. [83] | MF | PT | × | ✓ | ×
FS |  Chen et al. [58] | MF | MPC,DP | ✓ | ✓ | ×
|  HegedHus et al. [104] | MF | PT | × | × | ✓
|  Duriakova et al. [105] | MF | PT | × | × | ✓
TABLE V: Comparison of existing works for data space heterogeneous FL. IS
denotes Instance-Sharing. FS denotes Feature-Sharing. LS denotes Label-
Sharing. * denotes the approach is proposed for non-FL setting and can be
modified to tackle this issue. LM denotes Linear Models. NN denotes Neural
Networks. HE denotes Homomorphic Encryption. MPC denotes Secure Multiparty
Computation. DP denotes Differential Privacy. RM denotes Random Matrix. PT
denotes Plain-text. - denotes not applicable.

Table VI summarizes the transfer learning strategies adopted by Hetero-FTL
researches. We identify four transfer learning strategies to address critical
Hetero-FTL challenges, including data augmentation, parameter sharing,
knowledge distillation, and matrix factorization. We find representation
learning and knowledge distillation are popular strategies to transfer
knowledge among parties and are widely used in all types of Hetero-FTL
problems. For neural network models, each party builds a local model to map
their feature space into a common latent subspace. For linear models, learning
a linear mapping can map the feature representation between parties or to a
common subspace via concept factorization. Knowledge distillation is still
under-explored, especially in feature-sharing Hetero-FTL. Some of the listed
KD approaches were proposed for system-heterogeneous FL and are not tailored
to Hetero-FTL. Therefore, it is desirable to develop KD-based approaches to
tackle Hetero-FTL problems. Collaborative filtering approaches such as matrix
factorization and factorization machines are also explored for federated
recommendation applications, as the federated recommendation is a feature-
sharing Hetero-FTL problem by nature. We will introduce the studies using
these transfer learning approaches to tackle data space heterogeneity in
Section VII.

Perspective | Transfer method | Instance-sharing | Feature-sharing | Label-sharing
---|---|---|---|---
Data-based | Data Augmentation |  Kang et al. [89] |  Gao et al. [81] | -
Architecture-based |  Parameter Sharing |  Liu et al. [90] Sharma et al. [91] |  - |  Liang et al. [85] Gao et al. [95] Ju et al. [94]
Knowledge Distillation |  Li et al. [99] Wang et al. [98] Ren et al. [97] |  - |  -
Model-based |  Matrix Factorization |  Chai et al. [84] |  Chai et al. [76] Ammad et al. [83] Chen et al. [58] HegedHus et al. [104] Duriakova et al. [105] |  -

TABLE VI: Summary of transfer learning strategies adopted by Hetero-FTL
research. Red boxes denote under-explored promising directions.

##  IV Statistical Heterogeneity in HFL

In this section, we investigate statistical heterogeneity in data space
homogeneous (a.k.a. horizontal) FL, where all parties have data from the same
feature space. Data collected from the same data space can be imbalanced and
non-IID (independent and identically) distributed in the FL system. For
example, autonomous vehicles run in rural and urban environments. Such
variance leads to the marginal distributions $P(\mathcal{X})$ of collected
data differ. Moreover, different drivers in the same environments show various
driving style preferences, leading to varying conditional distributions
$P(\mathcal{Y}|\mathcal{X})$ of human driving behaviors. In addition, class
imbalance and data size imbalance across parties may also vary significantly.

The non-IID data among parties leads to poor global model convergence and a
lack of solution personalization. These issues deteriorate the performance of
the global model on each participant and could also disincentives the
participants from participating in the FL. To tackle the non-IID issue,
transfer learning approaches have attracted tremendous research interest in
recent years. There are mainly two strategies to tackle statistical
heterogeneity in FL. 1) Train a global model with good generalizability, then
each party fine-tunes the pre-trained global model on its local dataset. 2)
Directly train personalized models for all parties.

In statistical heterogeneous FL, the data spaces $\mathcal{D}_{i}$ in each
party $P_{i}$ are the same in terms of both feature space $\mathcal{X}$ and
label space $\mathcal{Y}$. However, the marginal distribution or conditional
distribution may differ among different parties. Since it was proposed, FL is
meant to study the case where multiple clients holding homogeneous data aim to
train a single global model collaboratively under specific privacy
constraints. Therefore, most existing researches focus on data space
homogeneous FL. Research on data space homogeneous FL can be categorized into
two cases: horizontal FL (HFL) and homogeneous federated transfer learning
(Homo-FTL). The major difference is that, in HFL, there is one learning
objective, and parties collaboratively train a global model on all data. HFL
tackles the statistical heterogeneity by training a single global model.
However, in Homo-FTL, each party optimizes its objective function and builds
its model, and the statistical heterogeneity is tackled by training
personalized models for all parties.

We discuss the two cases in the following. Interested readers may refer to
[30, 37] for more details on homogeneous FL.

###  IV-A Single Global Model

HFL handles multi-client data that share the same set of features but are
different in samples and is also referred to as federated optimization. The
pioneering works [3, 4, 38] in FL mainly focus on this setting, where
participants collaboratively train a global model based on their data with the
help of a coordinator for model aggregation. The goal of HFL is to train a
high-quality centralized global model. The objective can be formulated as:

$\min_{w\in\mathbb{R}^{d}}\left\\{f(w):=\frac{1}{n}\sum_{i=1}^{N}f_{i}(w)\right\\},$

where $w$ is the parameters of the federated model, and
$f_{i}:\mathbb{R}^{d}\to\mathbb{R}$ is the training objective function that
maps the model parameter set $w\in\mathbb{R}^{d}$ to a real valued training
loss with respect to the training data $\mathcal{D}_{i}$ of party $P_{i}$.

To improve communication efficiency, Reisizadeh et al. [106] propose to use
periodic averaging and quantization to reduce data transmission. Wang et al.
[107] propose CMFL to preclude the irrelevant updated by clients. Konecny et
al. [108] propose structured updates and sketched updates, which learn updates
from a restricted space and compress the learned updates, respectively. For
privacy preservation, differential privacy has been intensively studied in FL
[109, 110, 111] to protect client-side privacy. Homomorphic encryption and
secure multiparty computation are also adopted for secure model training and
aggregation [112, 113, 114]. Moreover, data in HFL can be 1) non-IID, 2)
unbalanced, and 3) massively distributed, which eliminates the model
performance. The non-IID issue is common as each client corresponds to a
particular user, a location, or a time window. For example, in the virtual
keyboard application, users who live in the UK and the USA speak English
differently; thus, the conditional probability varies. To tackle the non-IID
problem, many approaches have been proposed to mitigate the clients [115]. For
example, Jeong et al. [116] proposed federated distillation and federated
augmentation for communication efficient FL on non-IID data. A generative
model is trained to augment each party’s local data towards yielding an IID
dataset. Lin et. al [117] introduced ensemble distillation for robust model
aggregation on non-IID data.

We elaborate different approaches to train a singel global model on Non-IID
data in Secion VII.

###  IV-B Personalized Local Models

Different from HFL, which learns a single model for all parties even if the
data is non-IID across different parties, Homo-FTL aims to transfer knowledge
from the other parties to some or each party. Homo-FTL optimizes multiple
objectives to achieve the best performance for each client with non-IID data.
Therefore, Homo-FTL is highly related to multi-task learning, personalization,
and meta-learning. A variety of multi-model approaches are proposed to tackle
the non-IID data problem. We formulate the Homo-FTL as:

$\min_{W\in\mathbb{R}^{d\times
N}}\left\\{\mathcal{G}(W):=\sum_{i=1}^{N}f_{i}(w_{i})\right\\},$

where $W=[w_{1},...,w_{N}]$ is a $d\times N$-dimensional matrix that collects
$w_{1},...,w_{N}$ as its columns.

Recently, personalized FL has attracted great attention. Fallah et al. [118]
propose a model-agnostic meta-learning approach to train an initial shared
model for each party to fine-tune with its local data. Huang et al. [119]
employ federated attentive message passing to facilitate similar clients to
collaborate more and enable each client to own a local, personalized model. In
some cases, different groups of parties have their own objectives, and
federated clustering approaches can estimate the cluster identities of the
parties and train models for the party clusters [120, 121].

We elaborate different approaches to train personalized local models on Non-
IID data in Section VII.

###  IV-C Summary

As shown in Table VII, we summarize the utility-privacy-efficiency trade-off
factors of the reviewed works on system heterogeneous federated transfer
learning. It can be observed that existing studies highly focus on improving
model performance by training plain-text neural networks and fails to protect
data privacy by leveraging privacy-preserving techniques.

Objective |  Transfer Method |  | Utility | Privacy | Efficiency
---|---|---|---|---|---
Existing Study |  model type |  privacy mechanism |  asynchronous |  aggregator
Single global model |  Data Augmentation |  Zhao et al. [122] | NN | PT | × | ✓
Fu et al. [123] | NN | PT | × | ✓
Duan et al. [124] | NN | PT | ✓ | ✓
Jeong et al. [116] | NN | PT | × | ✓
Domain Adaptation |  Peng et al. [125] | NN | PT | × | ✓
Sun et al. [126] | NN | PT | × | ✓
Hanzely et al. [127] | NN | PT | × | ✓
Deng et al. [128] | NN | PT | ✓ | ✓
Smith et al. [129] | NN | PT | × | ✓
Huang et al. [119] | NN | PT | × | ✓
Shoham et al. [130] | NN | PT | ✓ | ✓
Personalized local models |  Parameter Sharing |  Singhal et al. [131] | NN | PT | × | ✓
Bui et al. [132] | NN | PT | × | ✓
Liang et al. [85] | NN | PT | ✓ | ✓
Arivazhagan et al. [133] | NN | PT | × | ✓
Knowledge Distillation |  Li et al. [134] | NN | PT | × | ×
Chang et al. [135] | NN | PT | × | ✓
Zhang et al. [136] | NN | PT | × | ✓
Gong et al. [137] | NN | PT | ✓ | ×
Li et al. [138] | NN | PT | × | ✓
Yang et al. [139] | NN | PT | ✓ | ✓
Matsuda et al. [140] | NN | PT | ✓ | ✓
Li et al. [141] | NN | PT | × | ✓
Zhu et al. [142] | NN | PT | × | ✓
Afonin et al. [143] | NN | PT | × | ✓
Zhu et al. [144] | NN | PT | × | ✓
Matrix Factorization |  Chai et al. [84] |  MF |  RM |  × |  ✓
TABLE VII: Comparison of existing works for statistical heterogeneous FL. NN
denotes Neural Networks. MF denotes matrix factorization. RM denotes Random
Matrix. PT denotes Plain-text.

##  V System Heterogeneity

In FL systems, heterogeneity that happens at the system level can also exert a
significant influence on the FL training process. Devices may differ in
hardware (CPU, GPU, memory), network connectivity (WiFi, 4G, 5G), and power
supply (battery mode), which leads to various computation, communication, and
storage capabilities. Such heterogeneity of resource-constrained devices
brings multiple challenges to FL [12]:

  * •

Unreliable network connectivity. Network connectivity widely varies from one
client to another. Most often, clients are under slow, limited, expensive, and
unavailable connections, which significantly reduce the number of available
ones at once. In addition, among the available clients, many might not be able
to participate in each learning round due to their different computation
capabilities.

  * •

Low round efficiency. The faster devices have to wait for those straggler
devices in each communication round for aggregation.

  * •

Low resource utilization. The competent devices may not be utilized thoroughly
by the classical FL algorithms.

To tackle the above challenges, there are mainly two types of strategies in
system heterogeneous FL:

  * •

One is to train a global model with the best model performance under the
various resource constraints. Many works formulate the problem as resource
allocation problems or multi-objective optimization problems [145, 146]. Many
studies have been focusing on resource allocation to make the optimal
decisions about the chosen clients, learning hyper-parameters, quantity and
length of communication rounds, and aggregation strategies.

  * •

The other is to train a tailored local model for each party with highest local
model performance while optimizing the resource consumption. Many
contributions has been made to train heterogeneous model architectures [28] or
quantization [147] for different edge devices.

In this section, we investigate the approaches to mitigate system
heterogeneity, including client selection, adaptive aggregation, joint
optimization, and model compression.

###  V-A Single Global Model

####  V-A1 Client Selection

Instead of allowing all clients to contribute equally in federated training,
many client selection algorithms are proposed to select robust and powerful
clients to improve the training efficiency.

FedCS is proposed to mitigate the device heterogeneity [20]. In FedCS, a
server first gather information from a set of randomly selected clients for
network states and computing resources. Then, the server selects the maximum
possible number of clients to update models within a prespecified time
interval. To select the maximum possible number of clients, a greedy algorithm
is proposed, where the clients taking the least time for local training are
selected interatively. Yoshida et al. [21] further propose Hybrid-FL to handle
non-IID data distributions among clients. During the client selection phase,
the devices are selected to form an approximately IID dataset. Yang et al.
[148] apply FL to Google keyboard query suggestions where smartphones are only
available for training when they are in charging mode, connected to WiFi, and
screened off. Chen et al. [11] propose a heuristic greedy node selection
strategy that iteratively selects heterogeneous IoT nodes to participate in
global learning aggregation based on their local computing and communication
resources. Zhu et al. [149] limit the number of devices training at the same
time in the AFL network. A limit-size cache with a weighted averaging
mechanism is introduced onto the server to reduce the impact of model
staleness. Hao et al. [150] designed a prioritized node selecting function
according to the computing power and accuracy change of local models on each
node. Other unselected nodes continue the iterations locally at the same time.
Similarly, Imteaj et al. [151] proposed to use a trust score to each node
based on its activities.

####  V-A2 Asynchronous Aggregation

The efficiency of classical FL algorithms is susceptible to the straggling
devices, as the server waits for all devices to complete local update before
aggregation. Therefore, asynchronization is highlighted by many recent studies
to boost the scalability and efficiency. In asynchronous FL, a client can
update the global model whenever it sends a local update to the server.

Sprague et al. [22] point out that training FedAvg in an asynchronous manner
is robust to clients joining halfway during training. However, when the
training data is non-IID among devices, the convergence rate of the
asynchronous approach is slower than FedAvg. The FedAsync algorithm is
proposed to improve convergence [152]. FedAsync adaptively weight newly
received client updates according to staleness. An outdated update from a
straggler is weighted less, as it should be received in previous rounds.

The asynchronous FL algorithms can be harmed by the straggler devices, which
update low quality models. Semi-asynchronous algorithms are proposed [12].
Semi-asynchronous approaches combines synchronous and asynchronous FL. Hao et
al. [150] a priority function is introduced to accurately select nodes with
large amounts of data or high computation power. Meanwhile, local models on
unselected nodes will be cached for a specific number of iterations before
being submitted to the aggregation server. Besides, a restriction on the local
training round number is set to prevent specific nodes from being unselected
for a lengthy period of time, leading the global model to overfit certain
nodes. The effectiveness of the scheme is evidenced by experiments conducted
on iid and non-iid datasets.

Wu et al. [23] propose SAFA, a semi-asynchronous FL protocol, to address the
problems in FL such as low round efficiency and poor convergence rate in
extreme conditions. To boost efficiency and improve the quality of the global
model, SAFA designs a cache-based lag-tolerant mechanism for model
distribution, client selection and global aggregation. SAFA classifies clients
into three types: up-to-date, deprecated, and tolerable. The first two types
of clients update models in a synchronous manner, and the last type of client
update models in an asynchronous manner. Thereby, SAFA achieves a tradeoff
between faster convergence and lower communication cost.

####  V-A3 Model Compression

Model compression techniques are explored to reduce the communication cost.
Low-rank matrix factorization with heterogeneous model sizes is also explored
to train heterogeneous models for devices with different hardware resources.
FedHM [24] is a heterogeneous federated model compression framework that
distributes the heterogeneous low-rank models to clients and then aggregating
them into a full-rank model. In FedHM, each party low-rank factorize the
parameter matrix of the neural network with different sizes, according to its
hardware resources. In each communication round, the server receives the
factorized models and transforms the factorized low-rank models back to full-
rank models. Then, the server conducts aggregation using weighted averaging.

Caldas et al. [25] propose to minimize the communication cost by reducing the
size of the models transmitted between the server and the clients. First, the
server generates a smaller model with fewer parameters via federated dropout
technique. Then, the server performs lossy compression on the resulting model
and sends to the clients. Then, the clients decompress and train the local
models. Finally, the local models are lossily compressed and sent to the
server, for decompression and aggregation.

###  V-B Personalized Local Models

####  V-B1 Parameter Sharing

HeteroFL [28] proposes to train heterogeneous neural networks with various
width by averaging the overlapped sub-matrices of all parameter matrices of
all parameter matrices across various models. The number of parameters in
smaller models are reduced in width by cropping each parameter matrix of the
largest model into sub-matrix with various reduction radio. With this
construction, HeteroFL adaptively allocates subsets of global model parameters
to each party according to its corresponding capabilities. To perform
aggregation, each parameter is averaged from the parties whose local models
contain that parameter. However, this approach is not beneficial for retaining
large models’ knowledge in smaller models because the width pruning operation
breaks the original model structure. Moreover, even the same sub-matrix may
have different behaviors in the small and large models due to the model
structure differences, which may lead to a sub-optimal performance due to the
mismatch of feature spaces. In addition, simply sharing parts of parameters
across different models cannot effectively transfer useful knowledge encoded
by strong models to other weaker models. To tackle the above issue, Liu et al.
[153] further propose InclusiveFL, in which the shallow bottom model layers in
the largest model are shared with other smaller local models, to eliminate the
mismatch between small and large local models.

Split-Mix [26] proposes to train multiple models of different sizes and
adversarial robustness levels tailored to the budget of each device, to
address the under-training issue in large models. During training, strong
parties sample more base models for local training and weak parties sample
less base models. In each communication round, the server aggregates each base
model using the updates from the corresponding parties. In inference phase,
the trained local models can be aggregated on-demand according to inference
requirements. Munir et al. [27] proposed FedPrune to achieve inclusive FL. It
prunes the global model for weak parties based on their capabilities.

FedMask [154] is proposed to achieve communication and computation efficient
FL. In FedMask, each party learns a heterogeneous and structured sparse neural
network. To achieve this, each party learns a sparse binary mask while
freezing the parameters of local model. Only the binary masks are communicated
between devices and the server. The model aggregation is conducted by
averaging on the unmasked parameters for each local model. The authors of
FedMask also propose a similar approach named Hermes [155] that trains
personalized and sparse neural networks by applying the structured pruning.
Only the updates of these sub-networks are transmitted. To aggregate models,
the server performs the average on only overlapped parameters across each sub-
network. To transfer knowledge among heterogeneous models, the lower layers
are retained and aggregated and the higher layers are pruned by channel. Lu et
al. [156] propose to use model mapping to fuse heterogeneous models in FL.

####  V-B2 Clustered Training

Hierarchical knowledge transfer are widely explored in heterogeneous networks
where the devices are group according to location [157], communication delay
[158], or capacity [153]. Therefore, the knowledge is first transferred inside
a group via homogeneous FL, then the group-wise models are aggregated for
higher-level knowledge transfer. Abad et al. [157] studied a heterogeneous
cellular network where the mobile devices are grouped by small base stations
in different cells. The mobile devices are trained in a hierarchical way.
Specifically, the devices in each cell first conduct FL by base stations.
Then, the models trained within each cell are periodically aggregated via a
macro base station to mutually transfer knowledge among the base stations.
Similarly, Chai et al. [158] proposed a tire-based client selection approach
that groups clients according to communication delay. The devices with similar
training speed collaboratively conduct FL to train a model. The model trained
by each group are then aggregated by a server. Considering the devices in
internet of things (IoT) networks are composed of resource-limited devices and
powerful devices, these approaches can be adopted to IoT applications.

###  V-C Summary

We summarize the trade-off factors of the existing works on system
heterogeneous FL in Table VIII. It can be observed that most existing studies
in system heterogeneous FL focus on improving model performance by training
plain-text neural networks and merely leverage privacy-preserving techniques
such as HE and DP for privacy protection.

Objective |  Transfer Method |  | Utility | Privacy | Efficiency
---|---|---|---|---|---
Existing Study |  model type |  privacy mechanism |  asynchronous |  aggregator
Single global model |  Client Selection |  Nishio et al. [20] | NN | PT | × | ✓
Yoshida et al. [21] | NN | PT | ✓ | ✓
Yang et al. [148] | NN | PT | ✓ | ✓
Chen et al. [11] | NN | PT | × | ✓
Zhu et al. [149] | NN | PT | ✓ | ✓
Similarly, Imteaj et al. [151] | NN | PT | ✓ | ✓
Asynchronous Aggregation |  Sprague et ap. [22] | NN | PT | ✓ | ✓
Xu et al. [12] | NN | PT | ✓ | ✓
Hao et al. [150] | NN | PT | ✓ | ✓
Wu et al. [23] | NN | PT | ✓ | ✓
Model Compression |  Yao et al. [24] | NN | PT | × | ✓
Caldas et al. [25] | NN | PT | × | ✓
Domain Adaptation |  Smith et al. [129] | NN | PT | × | ✓
Marfoq et al. [159] | NN | PT | × | ✓
Personalized local models |  Parameter Sharing |  Diao et al. [28] | NN | PT | × | ✓
Liu et al. [153] | NN | PT | × | ✓
Hong et al. [26] | NN | PT | ✓ | ✓
Munir et al. [27] | NN | PT | × | ✓
Li et al. [154] | NN | PT | ✓ | ✓
Li et al. [155] | NN | PT | × | ✓
Lu et al. [156] | NN | PT | × | ✓
Knowledge Distillation |  Zhang et al. [160] | NN | PT | × | ✓
Liu et al. [153] | NN | PT | ✓ | ✓
Ozkara et al. [147] | NN | PT | × | ✓
He et al. [86] | NN | PT | × | ✓
Bistritz et al. [161] | NN | PT | × | ✓
Matrix Factorization |  Yao et al. [24] |  NN |  PT |  × |  ✓
TABLE VIII: Comparison of existing works for system heterogeneous FL. LM
denotes Linear Models. FM denotes Factorization Machines. NN denotes Neural
Networks. DT denotes Decision Trees. HE denotes Homomorphic Encryption. MPC
denotes Secure Multiparty Computation. DP denotes Differential Privacy. PT
denotes Plain-text.

##  VI Model Heterogeneity

Different parties may hold heterogeneous features and various device resources
in FL. Therefore, it is desirable to train heterogeneous models for
participants accordingly. However, averaging-based model aggregation
algorithms [3, 5] are not valid in such settings due to the infeasibility of
parameter matching among local models.

The requirement of model heterogeneity in FL comes from three aspects: data
space, statistical, and system heterogeneity. 1) For data space heterogeneous
FL where the feature spaces differ among different parties, each party must
train a heterogeneous local model taking data from different feature spaces as
input for label prediction. For example, in healthcare applications, a
hospital has medical image data such as CT and MRI, and another has tabular
medical record data. They may cooperate to do VFL by training a CNN model for
medical images and a decision tree model for tabular data. 2) For statistical
heterogeneous FL, parties can train heterogeneous local models to tackle non-
IID and imbalanced data. 3) For system-heterogeneous FL, the participating
devices or edges may have different constraints regarding performance,
computation, and communication. The participants desire to tailor their own
model architectures. For example, we can train human activity recognition
models on smartphones and wearable devices. Smartphones have powerful
computation resources. However, wearable devices collect more accurate
movement data but have fewer computation resources. Therefore, we can train
larger models for smartphones and lightweight models for wearable devices. In
summary, training heterogeneous local and global models is a critical
technique to tackle the three heterogeneity issues in FL.

There are many approaches proposed to train heterogeneous models. When
different parties have heterogeneous features such as VFL, it is natural to
build a specific model for each feature space. The different models can be
integrated into one model or aggregated for prediction. As a representative
type of model compression, knowledge distillation [162, 163] adopts a
"teacher-student" learning paradigm to learn a small model from a large
teacher model effectively. Knowledge distillation relaxes the stringent
requirement of homogeneous local models as it takes the logits as
representations for knowledge transfer, providing an approach to building FL
systems with heterogeneous model architectures.

In this section, we will introduce some approaches to training heterogeneous
model architectures and discuss how to apply the heterogeneous model to tackle
each type of heterogeneity in FL.

###  VI-A For Data Space Heterogeneity

####  VI-A1 Representation Learning

In data space heterogeneous FL, parameter sharing-based approaches are widely
studied to learn latent representations from different feature spaces based on
aligned samples [90, 89, 99, 97]. The representation learning approaches build
a party-specific model for each party. First, each party trains a local model
on their local features to extract latent representations. Then, a top global
model is trained based on the feature representations extracted by each local
model for prediction.

Liu et al. [90] proposed a secure federated transfer learning (SFTL) approach
to tackle the instance-sharing Hetero-FTL problem. SFTL first maps the feature
spaces from two parties to a common latent space and then trains a classifier
for prediction. To fully use the local information, the feature
representations extracted by local models can be concatenated as the input of
the task-specific top model. However, it requires the samples have full
features for training and inference, which may not be feasible for real-world
multi-party VFL applications. Moreover, the proposed SFTL can only transfer
knowledge from one source party to one target party. The knowledge from both
parties can not be transferred mutually. The number of participants can
neither scale up.

FedSVD [84] is a federate singular vector decomposition (SVD) approach that
can conduct SVD for horizontally and vertically partitioned data among
multiple parties. FedSVD employs a random orthogonal linear transformation to
map the private data into a random linear space. Then, the factorization
server does standard SVD computation to decompose the masked data. Finally,
the decomposed matrices are sent back to all clients to remove the random
transformation. The decomposed matrices contain a shared matrix and a private
matrix. For VFL, the shared matrix contains latent representations for
heterogeneous features of each sample. While the private matrix is the
personalized local model that maps heterogeneous feature spaces to the common
latent space.

####  VI-A2 Tree-based Models

Decision tree models are widely used to tackle data space heterogeneity in
VFL. The decision trees trained among all the parties are naturally
heterogeneous in architecture. Liu et al. [71] propose a Federated Forest
based on bagging and the CART tree. Concerning its own features, each party
preserves the partitioned information. Performance-wise, the proposed approach
is on par with the centralized version. In order to accelerate homomorphic
encryption, Fu et al. [33] propose a concurrent training protocol and tailored
cryptographic operations, which resulted in the creation of the effective
vertical federated GBDT technique known as VF2Boost. Federated gradient
boosting approaches generate a boosting model concurrently with the base
learner, which can be laborious and communication-inefficient. As a result,
Han et al. [72] suggest combining boosting and bagging to create many decision
trees concurrently. By canceling decision routes and modifying a practical,
secure inference approach, Chen et al. [73] investigate the interpretability
of decision tree ensembles in VFL during the inference phase.

####  VI-A3 Data Augmentation

Kang et al. [89] propose FedCVT to tackle the two-party instance-sharing
Hetero-FTL via representation learning. The main idea is to predict the
missing features of the other party, given the local features as input. Two
parties train a feature prediction model based on their aligned samples and
mutually predict the missing feature space and the pseudo-labels. Then, FedCVT
picks the pseudo-labeled samples with high probability as training data.
FedCVT trains three models on the augmented samples—two local classifiers for
local prediction on local features and one federated classifier for federated
prediction. Gao et al. [81] tackles the feature-sharing Hetero-FTL problem by
using feature mapping to convert heterogeneous feature spaces to homogeneous
feature space.

####  VI-A4 Knowledge Distillation

Knowledge distillation enables knowledge transfer across heterogeneous feature
spaces from different parties, assuming there exist some aligned instances.
Participants can hold heterogeneous local models in terms of architecture and
parameters in federated knowledge distillation frameworks.

Ren et al. [97] proposed to transfer knowledge from privileged features in
passive parties to the active party via knowledge distillation. Furthermore,
an oblivious transfer is leveraged to protect data ID privacy during training
and inference. Paillier homomorphic encryption is used to train federated
models in a privacy-preserving manner. Oblivious Transfer (OT) is used to
align samples without privacy leakage.

Wang et al. [98] propose to transfer knowledge via representation distillation
to tackle the multi-party instance-sharing Hetero-FTL problem. Specifically,
they adopt FedSVD for representation learning. Then, they train an auto-
encoder based on representation distillation to transfer knowledge from the
full feature space of aligned samples to the feature space of the active
party. Given the extracted task-independent feature representations, the
active party can train a task-specific model with the concatenated
representations as input. However, this approach can only transfer knowledge
in a pair-wise manner. When the number of the passive party grows, the
computation complexity grows linearly.

###  VI-B For Statistical Heterogeneity

Zhu et al. [144] proposed a data-free federated distillation algorithm by
training a feature extractor and a feature representation generator. This
proposed approach employs a feature extractor to project data into a latent
space. Lin et al. [117] proposed to use ensemble distillation for robust model
fusion on heterogeneous local models. To transfer knowledge, an unlabeled
dataset or a generator is used to sample data for all participants to compute
logits and distill knowledge. Lan et al. [142] propose a multi-branch design
that all local models share a common feature extractor, and each has a unique
unshared branch. A gate is used for the weighted aggregation of the logits of
all the branches. Although this approach is proposed for centralized learning,
it can be modified to fit model heterogeneous FL problems.

Li et al. [138] propose a peer-to-peer decentralized, FL approach via mutual
knowledge distillation. Specifically, in each communication round, a group of
parties is selected to conduct FL, and then the trained models are sent to the
second group of parties. The parties in the second group compute two soft
logits computed by their local models and the received models. These two
logits are aggregated as teacher logits to train both the local and the
received models. Yang et al. [139] propose personalized federated mutual
learning (PFML) to use the non-IID characteristics to generate tailored models
for each party. Specifically, the PFML approach integrates mutual learning
into the local update process in each party to improve the performance of both
the global model and the personalized local models. The convergence rate is
also accelerated by mutual distillation. FedH2L [141] is proposed to address
statistical and model heterogeneity via mutual distillation simultaneously. It
exchanges only posteriors on a shared seed set between participants in a
decentralized manner.

###  VI-C For System Heterogeneity

####  VI-C1 Representation Learning

HeteroFL [28] proposes to train heterogeneous neural networks with various
widths by averaging the overlapped sub-matrices of all parameter matrices
across various models. The number of parameters in smaller models is reduced
in width by cropping each parameter matrix of the largest model into a sub-
matrix with various reduction radios.

FedMask [154] is proposed to achieve communication and computation efficient
FL. In FedMask, each party learns a heterogeneous and structured sparse neural
network. To achieve this, each party learns a sparse binary mask while
freezing the parameters of the local model. Only the binary masks are
communicated between devices and the server. The model aggregation is
conducted by averaging the unmasked parameters for each local model. The
authors of FedMask also propose a similar approach named Hermes [155] that
trains personalized and sparse neural networks by applying structured pruning.
Only the updates of these sub-networks are transmitted. To aggregate models,
the server performs the average on only overlapped parameters across each sub-
network. The lower layers are retained and aggregated to transfer knowledge
among heterogeneous models, and the higher layers are pruned by channel. Lu et
al. [156] propose to use model mapping to fuse heterogeneous models in FL.

####  VI-C2 Knowledge Distillation

QuPeD, which Ozkara et al. [147] present, allows clients to train tailored
compressed models with various quantization settings and model dimensions.
QuPeD takes into account that clients with different resources have different
requirements for inference efficiency and test accuracy. The proposed
technique first trains quantized models using a relaxed optimization problem
that also optimizes quantization values. Then, by transferring information
from the global model to quantized local models, knowledge distillation is
used to train compressed local models.

A client-inclusive FL approach called InclusiveFL [153] is presented. It gives
clients with varied computing capacity access to models of various sizes.
First, because the lower layers are crucial for maintaining the information in
downstream tasks, the shallow bottom model layers in the biggest model are
shared with other smaller local models in order to reduce the mismatch between
small and large local models. The parameters of common bottom levels are then
updated using a layer-wise heterogeneous aggregation approach. In order to
better transfer information from large models with powerful customers to small
models with weak clients, a momentum knowledge distillation approach is
finally presented. Specifically, to transfer knowledge from a large model to
small models, the top encoder layer in small models is trained by momentum
distillation to imitate the behavior of the top encoder layers in a larger
model.

##  VII Apply Transfer Learning to Tackling Statistical, Data Space, System
and Model Heterogeneity

Federated learning faces heterogeneity in terms of data space, statistics, and
system. In this section, we discuss different transfer learning approaches
that can be adopted to address the different heterogeneity in FL.

###  VII-A Knowledge distillation

This section provides a comprehensive survey of knowledge distillation and
investigates how knowledge distillation can be used to tackle the model
heterogeneity issue in FL. We first introduce the definition of knowledge
distillation. Then, we review existing works that leverage knowledge
distillation to tackle the model heterogeneity problem in FL. Interested
readers may refer to [164] for a more detailed introduction.

####  VII-A1 Distillation Approaches

Knowledge distillation was initially proposed to compress an ensemble of
neural networks [162]. Three forms of knowledge can be leveraged to transfer
knowledge from the teacher model to the student model: response-based
knowledge, feature-based knowledge, and relation-based knowledge. Both
response-based and feature-based knowledge leverage the output of specific
layers to transfer knowledge from teacher to student. Relation-based knowledge
takes the instance-wise or layer-wise distance as knowledge to train the
student model. We elaborate these three forms of knowledge in Appendix B.

Knowledge distillation for heterogeneous FL falls into the group-based online
distillation scheme, which is an end-to-end training scheme. The idea of
group-based online distillation is to train multiple student local models
simultaneously by learning from ground-truth labels and distilling from their
aggregated soft targets, a specific form of aggregation of intermediate peer
predictions. In federated online distillation, the teacher model is the
aggregation of the local models, and each local model is a student model [103,
100, 142]. Therefore, the teacher model is updated simultaneously with the
student models, and the whole knowledge distillation framework is trained in
one phase. The online distillation approaches proposed for non-federated
settings are promising to be adopted to tackle heterogeneous FL problems.

Data-free knowledge distillation is also explored in heterogeneous FL to relax
the requirement of a public dataset for knowledge transfer. Nayak et al. [165]
first proposed zero-shot knowledge distillation by synthesizing the data
impressions from the teacher model and using them as surrogates for knowledge
distillation. Micaelli and Storkey [166] achieved data-free distillation by
training an adversarial generator to search for samples on which the student
poorly matches the teacher and then using them to teach the student model.

####  VII-A2 Existing Studies

a) Data space heterogeneity. Knowledge distillation can be used to tackle
instance-sharing Hetero-FTL problems, where the participants have different
feature spaces and some aligned instance IDs. In such settings, the knowledge
can be transferred by the logits of the aligned instances output by the
heterogeneous local models.

Only a few recently proposed works use knowledge distillation to tackle the
instance-sharing Hetero-FTL problem. To enable federated inference on non-
overlapping data in VFL, Ren et al. [97] proposed to distill knowledge from
privileged features that are only available in training phase in passive
parties to the active party. The active party trains a local model using the
VFL model trained on aligned samples as the teacher model. Thereby, the
collaboratively trained local model can predict un-aligned samples with higher
accuracy than the locally trained model. Furthermore, Paillier’s HE is
leveraged to achieve privacy-preserving model training. Oblivious transfer is
adopted to protect sample ID privacy in training and inference phases. Wang et
al. [98] representation distillation to propose a task-independent knowledge
transfer approach. First, all parties collaboratively learn latent
representations for aligned samples via FedSVD [84]. FedSVD maps heterogeneous
features from all parties to a latent space. Secondly, the active party trains
an auto-encoder on the feature representations, to distill knowledge from
aligned samples to un-aligned samples. Finally, the active party conducts
task-specific training and inference on the enriched representations and local
features. However, this approach transfers knowledge from each passive party
in a pair-wise manner, and the dimension of input features of the task-
specific model grows linearly as the passive party number grows.

Li et al. [99] propose SplitKD to first train a teacher VFL model on aligned
samples, then teach a student local model for the active party via knowledge
distillation. The teacher VFL model consists of a top model for classification
and personalized bottom models for each party. In the distillation phase, the
active party’s local model is initialized by the pre-trained bottom model and
finetuned by distilling knowledge from the VFL model.

Online mutual knowledge distillation methods [103, 100] simplify the training
process into one stage, where the soft predictions of each model are
aggregated into one teacher prediction. The teacher prediction updates all
local models together with the ground truth of private labeled data. Taking a
two-party instance-sharing Hetero-FTL problem as an example, two parties first
locally train heterogeneous student models with their private data in
different feature spaces. Then, they can use mutual knowledge distillation to
update two local models over the aligned instances collaboratively. In mutual
knowledge distillation [100], two student models learn from each other by
taking the logits from the other model as soft targets.

Guo et al. [103] consider a centralized setting with homogeneous data, treat
all the DNNs as student models, and collaboratively update them in a single
stage. Although this approach is proposed for centralized learning, it can be
modified to tackle data space heterogeneity. Since each party trains a compact
local model, the active party can conduct inference by ensemble aggregation
with an arbitrary number of aligned parties. The active party can use its
local model for inference to predict unaligned test data. On the other hand,
when the test data is aligned across several parties, all parties with aligned
features collaboratively predict with the ensemble of local models. Zhu et al.
[144] propose a data-free federated distillation algorithm that employs a
feature extractor to project data into a latent space. Although being proposed
for HFL, this approach can be adopted to tackle system or data space
heterogeneity by training heterogeneous feature extractors for different
parties. Similarly, He et al. [86] consider a centralized setting and propose
to train multiple feature extractors and a classifier via knowledge
distillation. The local feature extractor models are unique for each
heterogeneous feature space and map input data to a common latent subspace.
The local feature representations and logits are transmitted to a central
server to train a global classifier. The predictions of the global teacher
classifier are then sent back to each party for knowledge transfer.

Cross-modal distillation is another line of related studies in data space
heterogeneous FL. Although existing works are proposed for centralized
learning, these works can inspire us to propose approaches for instance-
sharing Hetero-FTL. Peng et al. [101] address the visual-textual cross-modal
learning problem by introducing a life-long knowledge distillation approach. A
hierarchical recurrent network is constructed to leverage information from
both semantic and attention levels through adaptive network expansion. Tian et
al. [102] propose to use a contrastive loss to transfer pair-wise information
across different modalities. Existing cross-modal distillation methods rely on
aligned or paired instances between two modalities [167, 102, 168]. Cross-
modal distillation is restricted by the lack of aligned samples across
different modalities.

b) Statistical heterogeneity. In general, there are four main types for
distillation-based FL architectures: 1) distillation of knowledge to each
client to learn stronger personalized models; 2) distillation of knowledge to
the server to learn stronger server models, 3) bidirectional distillation to
both the clients and the FL server; and 4) distillation amongst clients.

FedMD [134] is a distillation-based FL framework that leverages knowledge
distillation to enable parties to train unique local models on their private
local data. The main idea is to transfer knowledge by aggregating logits of
local models computed on a public dataset shared by the parties. For every
communication round, each party trains its model based on the teacher logits
computed on the public dataset as well as the logits computed by its local
model on its private dataset. Thereby, each party learns a personalized model
by transferring knowledge from other parties. Chen et al. [169] propose online
distillation with diverse peers by incorporating an attention-based mechanism
to generate a distinct set of weights for each local model. Each party assigns
individual weights to all the local models during aggregation to derive their
own target distributions. The first stage conducts group-based distillation
among all local models for knowledge transfer with diverse target
distributions. Then, the second distillation stage transfers the knowledge
from all local models to a global model.

Chang et al. [135] propose a two-stage robust knowledge distillation approach
to tackle heterogeneous distributions. In the first cold-start stage, each
party trains a local model on private data. In the second stage, the clients
transfer knowledge by sharing the logits of public data. The server aggregates
the logits via the robust mean estimation algorithm RobustFilter [170]. Then
the clients update local models using private data and soft-labeled public
data. Zhang et al. [136] introduce a knowledge coefficient matrix to train
personalized local models via knowledge distillation. The proposed KT-pFL
updates the personalized soft logits of each party by a linear combination of
all local soft logits using a knowledge coefficient matrix, which can
adaptively enhance knowledge transfer among parties with similar data
distribution. The knowledge coefficient matrix and the model are alternatively
updated in each round via gradient descent. Gong et al. [137] propose a one-
shot offline knowledge distillation approach named FedKD, using unlabeled
public data. A quantized and noisy ensemble of local predictions from fully
trained local models is proposed for privacy guarantees without sacrificing
accuracy.

Lan et al. [142] propose a multi-branch design that all local models share a
common feature extractor, and each has a unique unshared branch. A gate is
used for the weighted aggregation of the logits of all the branches. Although
this approach is proposed for centralized learning, it can be modified to fit
model heterogeneous FL problems. Afonin and Karimireddy [143] propose
federated kernel ridge regression to achieve model-agnostic FL. FedGKT [86] is
also proposed to improve model personalization performance via group knowledge
transfer. It leverages alternating minimization to train small local models
and a large global model via bidirectional knowledge distillation.

Lin et al. [117] proposed to use ensemble distillation for robust model fusion
on heterogeneous local models. To transfer knowledge, an unlabeled dataset or
a generator is used to sample data for all participants to compute logits and
distill knowledge. It assumes a setting in which the edge clients require
different model architectures due to diverse computational capabilities. The
server constructs p distinct prototype models, each representing clients with
identical model architectures (e.g., ResNet and MobileNet). For each
communication round, FedAvg is first performed among clients from the same
prototype group to initialize a student model. Cross-architecture learning is
then performed via ensemble distillation, in which the client (teacher) model
parameters are evaluated on an unlabelled public dataset to generate logit
outputs that are used to train each student model in the server.

To eliminate the dependence on a public dataset in FL, Zhu et al. [144]
proposed data-free federated knowledge distillation by training a feature
representation generator. The server trains the generator to aggregate the
distribution of class representations from all participants. Then, the
generator is sent to participants whose knowledge is distilled to local
models. This proposed approach employs a feature extractor to project data
into a latent space. Due to system heterogeneity or data space heterogeneity,
the feature extractor can be heterogeneous for each party. However, it remains
a challenge to generate high-quality and diverse data for data-free knowledge
distillation to improve the model generalizability.

Mutual Distillation. Deep Mutual Learning [100] has been explored to train
personalized local models to tackle statistical heterogeneity in a
decentralized manner. It allows two models to learn from each other from their
predicted logits during the training process. Recently, many works have
explored mutual knowledge distillation in FL to train heterogeneous local
models to address the non-IID issue [138, 139, 140, 141]. The key idea of
mutual federated distillation is to take the local model of another party or
group as a teacher model and use the predicted posteriors to each their own
local model. The local models are exchanged among parties for mutual knowledge
transfer. It enables each party to train a heterogeneous model in a
decentralized manner.

Li et al. [138] propose a peer-to-peer decentralized, FL approach via mutual
knowledge distillation. Specifically, in each communication round, a group of
parties is selected to conduct FL, and then the trained models are sent to the
second group of parties. The parties in the second group compute two soft
logits by their local models and the received models. These two logits are
aggregated as teacher logits to train both the local and the received models.

Yang et al. [139] propose personalized federated mutual learning (PFML) to use
the non-IID characteristics to generate tailored models for each party.
Specifically, the PFML approach integrates mutual learning into the local
update process in each party to improve the performance of both the global
model and the personalized local models. The convergence rate is also
accelerated by mutual distillation.

Koji et al. [140] propose FL via Model exchange (FedMe), which personalizes
models with automatic model architecture tuning during the learning process.
In FedMe, clients exchange their models for model architecture tuning and
model training. First, to optimize the model architectures for local data,
clients tune their own personalized models by comparing them to exchanged
models and picking the one that yields the best performance. Second, clients
train both personalized and exchanged models by using deep mutual learning
[100], despite different model architectures across the clients. Similarly,
FedH2L [141] is proposed to address statistical and model heterogeneity via
mutual distillation simultaneously. It exchanges only posteriors on a shared
seed set between participants in a decentralized manner.

To summarize, mutual knowledge distillation makes FL communication efficient
and model agnostic and can effectively transfer knowledge among heterogeneous
data silos.

c) System heterogeneity. Parameter-sharing approaches [8] require all local
models have the same model architecture. However, in a realistic business
setting, participants may desire to design their unique model architecture due
to system heterogeneity reasons such as computation or communication capacity.
This kind of model heterogeneity is caused by system heterogeneity and poses
new challenges to traditional FL.

FedZKT [160] is a zero-shot knowledge transfer algorithm that allows devices
to determine the on-device models based on their local resources
independently. The server receives heterogeneous local models and
adversarially trains generator with the by aggregating the soft predictions of
the local models. The distilled central knowledge is then sent back as the
corresponding on-device model parameters, which can be easily absorbed on the
device side. The generator is trained to maximize the discrepancy between the
global model prediction and the prediction of the ensemble of local models.
FedZKT uses a bidirectional knowledge transfer. To train the global model, the
global model is trained using the averaged soft targets of all local models.
Then, the global model is used as the teacher model to transfer knowledge to
each local model.

Chan and Ngai [88] introduce knowledge distillation to train heterogeneous
models in the clients. The logits of each category are transmitted and
aggregated in the server. Then, the clients match the average logits with the
same label to the instances. Therefore, the knowledge is transferred among
parties via class-level logit aggregation.

He et al. [86] propose group knowledge transfer as a way to enhance the
performance of model customization for resource-constrained edge devices.
Through a bidirectional distillation method, it trains tiny edge models on
edge and a big server model on the server using alternating minimization. The
big server model employs the KL-divergence loss to reduce the discrepancy
between the ground truth and soft labels predicted by the local models after
receiving extracted features from the local models as inputs. The server model
then incorporates the information passed from the local models. Similarly,
each local model uses its own private dataset and the projected soft labels
downloaded from the server to calculate the KL-divergence loss. This
streamlines the dissemination of information from the server model to the
local models.

Bistritz et al. [161] propose a distillation distributed algorithm for on-
device learning and is architecture independent. It assumes that all edge
devices are connected to just a small number of nearby devices in an IoT edge
FL environment. Only devices that are linked can exchange data. The learning
technique is semi-supervised; local training is done on private data, while
federated training is done on a publicly available dataset that is not
labeled. Each client broadcasts its soft decisions to its neighbors throughout
each communication cycle while simultaneously receiving their broadcasts. Each
client then uses a consensus technique to update its soft decisions depending
on those of its neighbors. The client’s model weights are then updated using
the revised soft choices by regularizing its local loss. This procedure
facilitates model learning via knowledge transfer amongst neighboring clients
in a network.

To learn tailored model architectures for different devices, knowledge
distillation is widely adopted for knowledge transfer among heterogeneous
models. Ozkara et al. [147] propose quantized personalization via distillation
(QuPeD) that clients with diverse resources have various requirements on
inference efficiency and test accuracy and allow clients to learn personalized
compressed models with diverse quantization parameters and model dimensions.
The proposed algorithm first trains quantized models through a relaxed
optimization problem, where quantization values are also optimized. Then,
knowledge distillation is introduced to train compressed local models by
transferring knowledge from the global model to quantized local models.

Recently, a client-inclusive FL method InclusiveFL [153] is proposed. It
assigns models of different sizes to clients with different computation
resources. First, to eliminate the mismatch between small and large local
models, the shallow bottom model layers in the largest model are shared with
other smaller local models, as the lower layers are most critical for
retaining the knowledge in downstream tasks. Then, a layer-wise heterogeneous
aggregation method is proposed to update the parameters of shared bottom
layers. Finally, a momentum knowledge distillation method is proposed to
better transfer knowledge in big models on powerful clients to the small
models on weak clients. Specifically, to transfer knowledge from a large model
to small models, the top encoder layer in small models is trained by momentum
distillation to imitate the behavior of the top encoder layers in a larger
model.

Both QuPeD and InclusiveFL adopt a hierarchical FL architecture. During FL,
the devices are first grouped according to capacity. Devices in each group
conduct homogeneous FL to train a model. Then, models with different sizes are
aggregated for knowledge transfer.

###  VII-B Data Augmentation

####  VII-B1 Data space heterogeneity

In feature-sharing Hetero-FTL problems, participants have some common features
but no aligned instance IDs, as shown in Figure 7. Gao et al. [81] first
studied the problem of privacy-preserving feature-sharing Hetero-FTL and
proposed to use feature mapping to convert heterogeneous feature spaces to
homogeneous feature spaces. The data are augmented by predicting its missing
features. Participants first learn instance weights over the shared feature
space for domain adaptation. For heterogeneous mutual transfer learning, each
party trains a linear model locally to map the shared features to the local
ones. Then, the other party fulfills the missing feature space via secure
computation. Once the whole feature space is fulfilled, the feature-sharing
Hetero-FTL problem is converted into a HFL problem. However, this approach
only applies to linear models.

To address the instance-sharing Hetero-FTL problem, Kang et al. [89] propose
FedCVT to estimate the representations of missing features via semi-supervised
learning. Two parties jointly predict the missing features and pseudo-labels
by training a feature prediction model on the aligned samples. FedCVT then
selects the highly likely pseudo-labeled samples as training data. FedCVT uses
the imputed (augmented) samples to jointly train three models, including one
federated classifier for federated prediction and two local classifiers for
local prediction on local features.

Yitao et al. [171] propose FedMC to align similar unaligned data via data
collaboration. FedMC first learns a latent feature space on aligned and
labeled samples. Then, it measures the distance between each pair of unaligned
samples from the active party and passive party in this latent space. Then,
FedMC aligns two samples that are similar in the latent space in a pair and
adds the aligned samples to the training set. Finally, FedMC trains a joint
federated model on the expanded training set.

####  VII-B2 Statistical Heterogeneity

A data sharing technique that gives each client a modest quantity of global
data that is balanced by classes, was proposed by Zhao et al. [122]. Their
research demonstrates that even a tiny amount of data can increase
considerable accuracy. The FedHome approach [123] trains a generative
convolutional auto-encoder model. Following the FL process, each client
further customizes a locally enhanced class-balanced dataset. This dataset is
created by applying a synthetic data generation on the encoder network’s low-
dimensional features using local data. A self-balancing FL framework named
Astraea [124] uses Z-score-based data augmentation and down-sampling of local
data to address the class imbalance. The server needs statistical data on the