# Federated PCA on Grassmann Manifold for IoT Anomaly Detection

Tung-Anh Nguyen, Long Tan Le, Tuan Dung Nguyen,
Wei Bao, Suranga Seneviratne,
Choong Seon Hong, Nguyen H. Tran T.-A. Nguyen, L. T. Le, N. H. Tran, W. Bao,
and S. Seneviratne are with the School of Computer Science, The University of
Sydney, Sydney, NSW 2006, Australia (email<EMAIL_ADDRESS>{long.le, nguyen.tran, wei.bao, suranga.seneviratne}@sydney.edu.au). T. D.
Nguyen is currently with the Department of Computer and Information Science,
University of Pennsylvania, USA (email: joshtn@seas.upenn.edu). C. S. Hong is
with the Department of Computer Science and Engineering, School of Computing,
Kyung Hee University, Yongin-si 17104, Republic of Korea (email:
cshong@khu.ac.kr).

###### Abstract

With the proliferation of the Internet of Things (IoT) and the rising
interconnectedness of devices, network security faces significant challenges,
especially from anomalous activities. While traditional machine learning-based
intrusion detection systems (ML-IDS) effectively employ supervised learning
methods, they possess limitations such as the requirement for labeled data and
challenges with high dimensionality. Recent unsupervised ML-IDS approaches
such as AutoEncoders and Generative Adversarial Networks (GAN) offer
alternative solutions but pose challenges in deployment onto resource-
constrained IoT devices and in interpretability. To address these concerns,
this paper proposes a novel federated unsupervised anomaly detection framework
– FedPCA – that leverages Principal Component Analysis (PCA) and the
Alternating Directions Method Multipliers (ADMM) to learn common
representations of distributed non-i.i.d. datasets. Building on the FedPCA
framework, we propose two algorithms, FedPE in Euclidean space and FedPG on
Grassmann manifolds. Our approach enables real-time threat detection and
mitigation at the device level, enhancing network resilience while ensuring
privacy. Moreover, the proposed algorithms are accompanied by theoretical
convergence rates even under a sub-sampling scheme, a novel result.
Experimental results on the UNSW-NB15 and TON-IoT datasets show that our
proposed methods offer performance in anomaly detection comparable to non-
linear baselines, while providing significant improvements in communication
and memory efficiency, underscoring their potential for securing IoT networks.

###### Index Terms:

Federated Learning, Internet of Things, Consensus Optimization, Grassmann
Manifolds, Anomaly Detection.

##  I Introduction

The ubiquity of Internet of Things (IoT) technologies has led to a
transformative shift where intelligent devices exchange information and enact
decisions via the internet, requiring minimal human intervention. This has
significantly enhanced the quality of life and found applications across
numerous sectors such as smart city infrastructures [1], energy systems [2],
or climate systems [3]. However, as the IoT ecosystem expands and becomes
increasingly complex, ensuring robust network security emerges as a paramount
task, especially considering the inherent security vulnerabilities of many IoT
devices due to their limited computational resources [4].

In this intricate landscape, machine learning-based intrusion detection
systems (ML-IDS) form a crucial line of defense. These systems often utilize
supervised learning methods that range from traditional techniques to more
sophisticated deep learning approaches [5]. While effective in various
scenarios, they come with several limitations. For instance, they necessitate
access to a considerable volume of _labeled data_ , which can be time-
consuming and costly to generate. Additionally, they often struggle with
_high-dimensional data_ and are limited in their ability to detect novel
anomalies not represented in the training data. In contrast, recent
unsupervised ML-IDS methods, such as AutoEncoders [6] and GAN-based models [7,
8], provide anomaly detection by learning representation of normal behavior
without relying on labeled data. However, these techniques often involve
training deep neural network models, which can be computationally demanding
and lack clear interpretability [9] – factors that pose significant challenges
for resource-limited IoT devices.

Traditionally, most ML-IDS lean on centralized deployment which is becoming
less feasible in IoT networks due to privacy concerns and resource
limitations. Hence, there’s a heightened demand for distributed security
frameworks where IoT devices can have self-monitoring capabilities that allow
them to identify potential threats in their network traffic while contributing
their insights to the broader system. Such a distributed approach would
considerably enhance the resilience of IoT networks, enabling real-time
anomaly detection and mitigation in heterogeneous environments. In light of
this, Federated Learning (FL) has stepped in as a promising paradigm that
allows IoT devices to collaboratively learn a shared model while retaining
their data locally, thereby harmonizing privacy preservation and efficient
resource utilization. _Despite its potential, the exploration of federated
unsupervised IDS under the heterogeneity of IoT traffic remains relatively
nascent_ , indicating the need for further research in this area.

In response to these challenges, we introduce a novel federated unsupervised
anomaly detection framework that leverages the representation power of PCA
[10] and the decomposability of ADMM [11], namely FedPCA [12]. Specifically,
we aim to harness PCA’s capabilities to learn a common representation of
distributed non-i.i.d datasets via consensus optimization that captures the
hidden states within the data. These hidden states typically reflect complex
correlations and underlying patterns within the data, which are instrumental
in effective anomaly detection. In order to solve FedPCA, we design two
distinct algorithms including FedPCA in Euclidean space (FedPE) and FedPCA on
Grassmann manifolds (FedPG). Each algorithm possesses unique properties
offering potential advantages for FL. While FedPE operates within a Euclidean
space affording both simplicity and interpretability, FedPG projects the data
onto Grassmann manifolds providing an efficient way to handle the high-
dimensional, complex, and potentially exhibit non-linear structures of network
data. We then investigate their convergence characteristics and provide
convergence guarantees under standard assumptions. Our proposed FL framework
offers notable advantages: it preserves privacy while providing efficient
anomaly detection in resource-constrained environments.

Within this work, each IoT device functions as an anomaly detection system,
playing an active role in fortifying the system’s security. The devices
continuously monitor their communication patterns, employing anomaly detection
models to identify abnormal behaviors that could signify a potential threat.
By deploying ML-IDS at the IoT device level, we establish a distributed
network of vigilant sentinels, each capable of real-time threat
identification. It enables IoT devices to partake in the learning process
without sharing sensitive local data—a characteristic of paramount importance
in an IoT setting where user data privacy is highly valued. This setup
substantially enhances the resilience and security of the entire IoT network,
and requires a shorter detection time compared to its unsupervised
counterparts. The main contributions of this work can be summarized as
follows:

  * •

We propose an unsupervised federated PCA framework for efficient host-based
IoT anomaly detection, which is formulated by a consensus optimization problem
for privacy-preserving and communication-efficient.

  * •

We introduce a new algorithm designed for FedPE and FedPG based on ADMM-based
procedures with client sub-sampling schemes aiming to enhance robustness and
mitigate communication bottlenecks.

  * •

We provide the theoretical convergence analysis for the proposed framework,
showing that it converges to a stationary point where the consensus constraint
is satisfied.

  * •

Through experiments on the UNSW-NB15 [13] and TON-IoT network [14] datasets,
we demonstrate that FedPE and FedPG not only show competitive performance
against non-linear methods in anomaly detection but also boast significant
improvements in communication and memory efficiency. The extensive evaluation
showcases the robustness of our proposed framework to non-i.i.d. data
distribution, marking it well-suited for IoT networks.

The rest of the paper is organized as follows. Section II outlines the
relevant background and previous works that closely relate to our topics of
interest. The problem formulation and the proposed framework are introduced in
Section III with a detailed explanation of algorithm designs. In Section IV,
we present theoretical convergence analyses for the proposed algorithms.
Numerical results and conclusion are conducted in Section V and Section VI,
respectively.

##  II Related Works

###  II-A Federated Learning

Federated Learning (FL) represents a cutting-edge distributed learning
approach that allows for the execution and subsequent updating of machine
learning models across thousands of participating devices, while
simultaneously safeguarding against data leaks [15]. In FL, models are
deployed and trained locally on individual clients’ devices using local data.
These locally trained models are then transmitted back to the central server
for aggregation, such as weight averaging [16]. Thanks to the attributes of
efficient communication and privacy protection, FL proves highly advantageous
for IoT networks and applications. Various versions of FL have been developed
after the birth of the standard FedAvg [16], significantly augmenting the
intelligence of IoT systems across a multitude of applications [17].

The primary challenges of FL can be broadly classified into two categories:
communication bottlenecks, resulting from the need to exchange large model’s
parameters between parties, and data heterogeneity, arising from the diversity
of the data held by FL clients. This motivated some prior works to design more
communication-efficient strategies such as local updates [18, 19, 20], model
compression [21], and adaptive communication [22]. Meanwhile, data
heterogeneity concerned with clients’ non-i.i.d. data has also been well-
studied [23, 24].

###  II-B Optimization on Grassmann Manifolds

Grassmann manifold optimization has recently gained traction as an advanced
mathematical tool in the fields of machine learning. This approach leverages
the geometric properties of Grassmann manifolds, a space representing linear
subspaces, to facilitate the analysis and understanding of high-dimensional
data, thereby enabling efficient optimization and enhanced modeling in various
machine learning and data analysis tasks [25]. In centralized learning, the
optimization over Grassmann manifolds has been employed to enhance manifold
learning and data classification, with applications extending to computer
vision and speech recognition [26, 27, 28]. In decentralized learning
contexts, particularly in FL, Grassmann learning has not fully explored yet,
making it a promising avenue for continued research and development.

###  II-C IoT Anomaly Detection

Owing to their inherent vulnerabilities, IoT systems require robust measures
against malicious activities [29]. This need has propelled anomaly-based
Intrusion Detection Systems (IDS) into the limelight, bolstered by the radical
advancements in machine learning. Early research on anomaly-based IDS within
IoT networks has been heavily centered around methodologies where data is
consolidated onto a central server [30, 31, 32, 33, 34, 35], raising concerns
related to privacy and cyber intrusions.

The recent trend has veered towards the use of FL frameworks for network
anomaly detection. For example, Nguyen et al.[36] developed a federated
intrusion detection system using Recurrent Neural Network models that were
trained on local gateways and then amalgamated into a single model. Similarly,
Wang et al.[37] demonstrated how a feed-forward artificial neural network can
augment detection performance. Mothukuri et al.[38] employed federated
training on Gated Recurrent Units (GRUs) models, preserving data on local IoT
devices by only sharing the learned weights with the central server. However,
accurately detecting unknown anomalous activities has proven challenging,
given the inherent imbalance between normal and abnormal traffic in IoT
networks[4].

###  II-D Principal Component Analysis

Over recent years, PCA [39] has become a popular unsupervised learning model
in centralized machine learning. From an IoT security standpoint, PCA-based
anomaly detection has emerged as an efficient technique, with PCA deployed to
bifurcate a dataset into benign and malicious subspaces [40]. For instance,
Brauckhoff et al. [41] suggested the use of the Karhunen-Loeve expansion in
light of the data’s temporal correlation, albeit limited to data from a single
router. Kernel PCA has been put forward to examine features considering both
variance and mean [42]. However, these methods, while promising, are
constrained by their reliance on data from a single device, rendering them
inadequate for IDS in settings where resources are limited and privacy is
paramount.

Distributed PCA has gained traction in the machine learning community [43].
For example, Ge et al.[44] proposed a minimax optimization framework for
generating a distributed sparse PCA estimator while maintaining privacy.
Grammenos et al.[45] suggested an asynchronous FL method to implement PCA in a
memory-limited environment, introducing an algorithm that calculates local
model updates using a streaming process and adaptively estimates its principal
components.

###  II-E ADMM-based Learning

The Alternating Direction Method of Multipliers (ADMM) has made significant
advancements in both theory and application over recent decades [11]. ADMM
acts as a powerful framework that combines the strengths of dual decomposition
and the method of multipliers to solve optimization problems. The core idea
behind this framework is to break down a problem into simpler subproblems that
can be solved in parallel, and then combine the solutions of these subproblems
to arrive at a global solution. The mathematical and convergence
characteristics of ADMM have been well-studied in the literature [46, 47, 48,
49].

Owing to its decompositional nature, ADMM has found significant applications
in large-scale and distributed optimization tasks, especially in machine
learning [50, 51]. In FL, several ADMM-based decentralized methods have been
developed to enhance communication efficiency [52, 53]. However, _employing
this framework for distributed PCA is still uncharted_.

In this study, we employ an alternative strategy to develop an effective
unsupervised method for network anomaly detection in IoT systems. We propose
an unsupervised federated PCA framework grounded in ADMM-based learning,
addressing privacy concerns while efficiently detecting anomalies in both
recognized and unprecedented attacks.

##  III Problem Formulation

Figure 1: Overview of a host-based IoT anomaly detection system utilizing
Federated PCA.

###  III-A Federated Principal Component Analysis

We consider an IoT network comprised of a central global coordinator $CO$ and
a set of $N$ IoT devices, each denoted by a unique identifier $ID_{i}$ holding
a local data set $X_{i}$, where $i\in{1,2,...,N}$. This network interacts
within a client-server topology, as depicted in Fig. 1. Our framework employs
a host-based IDS strategy, where each IoT device operates as a sentinel within
the network, examining its network traffic and identifying potential security
threats. This distributed approach significantly increases the network’s
resilience by facilitating real-time threat detection and mitigation.
Furthermore, since the learning is local to each device, it allows for a
reduction in communication overhead and strengthens privacy by keeping the
data on the device.

#### Principal Component Analysis Revisiting

We first revisit the basics of PCA to formally define our method. PCA projects
a given dataset $X$ onto principal components ordered by the amount of
variance they capture in the data [54]. Specifically, it seeks an optimal
rank-$k$ ($k<d$) matrix $\tilde{X}$ that minimizes the reconstruction error
$\lVert X-\tilde{X}\rVert_{F}$, where $||\cdot||_{F}$ denotes the Frobenius
norm [55]. Typically, $\tilde{X}$ is derived based on the singular value
decomposition (SVD) of $X$, i.e., $X=U\Sigma V^{\top}$, where
$U\in\mathbb{R}^{d\times d}$ is an orthonormal matrix containing the left
singular vectors of $X$, $\Sigma=\text{diag}(\sigma_{1},\ldots,\sigma_{d})$
contains the singular values
($\sigma_{1}\geq\sigma_{2}\geq\ldots\geq\sigma_{d}$), and
$V\in\mathbb{R}^{D\times D}$ contains the right singular vectors of $X$. Let
$U^{(k)}\in\mathbb{R}^{d\times k}$ be the first $k$ columns of $U$ and
$\Sigma_{k}=\text{diag}(\sigma_{1},\ldots,\sigma_{k})$. We can rewrite the
solution for $\tilde{X}$ as

$\tilde{X}=U^{(k)}U^{(k)\top}X,$ (1)

which can be interpreted as the projection of data matrix $X$ onto the column
space of $U^{(k)}$. The PCA problem can thus be formulated as an optimization
problem as follows:

$\displaystyle\min_{U\in\mathbb{R}^{d\times k}}$
$\displaystyle\lVert(I-UU^{\top})X\rVert_{F}^{2}$ (2)
$\displaystyle\mathrm{s.t.}$ $\displaystyle U^{\top}U=I.$

The essence of this problem is to locate an orthogonal matrix $U$ that
minimizes the reconstruction error of $X$ when projected onto the column space
of $U$, resulting in a matrix $Z=U^{\top}X\in\mathbb{R}^{k\times D}$ provides
a lower-dimensional representation of $X$.

In the context of developing ML-IDS for IoT networks, solving this PCA problem
typically requires IoT devices to transmit their local data to a centralized
server. While this approach offers a streamlined method for anomaly detection,
it is hampered by two main obstacles. First, data transferring within the
resource-limited and bandwidth-constrained landscape of IoT can lead to
significant communication bottlenecks, which in turn might escalate
operational costs, amplify latency, and induce network congestion. Second, the
act of centralizing such data amplifies privacy concerns as it risks exposing
sensitive information from personal environments, posing serious threats to
user confidentiality. Hence, an alternative would be to seek distributed
methods that address these issues while still effectively detecting anomalies.

#### Federated PCA via ADMM Consensus Optimization

To address these concerns, we propose a federated approach for solving PCA on
distributed datasets, namely FedPCA, which aims to collectively learn a
_common representation_ from all local datasets, $X_{1},X_{2},...,X_{N}$,
without the $CO$ directly accessing these datasets. We first introduce a set
of local variables $U_{1},\ldots,U_{N}$, where $U_{i}$ is the local PCA matrix
learned by client $i$. Then, we formulate a consensus optimization problem as
follows:

$\displaystyle\min_{U_{1},\ldots,U_{N}}$
$\displaystyle{\sum\limits_{i=1}^{N}\lVert(I-U_{i}U_{i}^{\top})X_{i}\rVert_{F}^{2}}$
(3) $\displaystyle\mathrm{s.t.}$ $\displaystyle U_{i}=Z,\;\forall
i=1,\ldots,N.$ $\displaystyle U_{i}^{\top}U_{i}=I,\;\forall i=1,\ldots,N,$

where the first linear constraint enforces the “consensus” to a _common low-
rank matrix_ $Z$ [56] for all local $U_{i}$. The second constraint ensures
that the matrix $U_{i}$ is orthonormal for every client keeping the
distributed problem consistent with the classical centralized PCA. However,
addressing the non-linearity introduced by this orthonormality constraint is a
challenging task due to the non-convex nature of the problem. Here, we employ
a surrogate technique, as introduced in [46], to handle the above non-linear
constraint as follows.

$h_{i}(U_{i})=\max{\left\\{0,U_{i}^{\top}U_{i}-I\right\\}}^{2},$ (4)

where the operators $\max$ and squared are component-wise. By reformulating
the problem to the $h_{i}(U_{i})$ function, we gently steer the optimization
towards matrices $U_{i}$ that approach orthonormality, making the landscape
more tractable for iterative frameworks. We then have the problem (3)
equivalent to the following:

$\displaystyle\min_{U_{1},\ldots,U_{N}}$
$\displaystyle{\sum\limits_{i=1}^{N}\lVert(I-U_{i}U_{i}^{\top})X_{i}\rVert_{F}^{2}}$
(5) $\displaystyle\mathrm{s.t.}$ $\displaystyle U_{i}=Z,\;\forall
i=1,\ldots,N$ $\displaystyle h_{i}(U_{i})\leq 0,\;\forall i=1,\ldots,N.$

Given the nature of problem (5) as a consensus optimization with linear
constraints, we opt to utilize ADMM [11] for its resolution. The essence of
this iterative framework is the alternating updates of primal and dual
variables to achieve two main objectives: the $U_{i}$ across clients converge
towards a common $Z$ and the objective function is iteratively minimized.
Applying this to our problem, we first construct the augmented Lagrangian
function associated with the problem (5) as follows.

$\begin{split}&{\mathcal{L}}(\\{U_{i}\\},Z,\\{Y_{i}\\},\\{T_{i}\\})=\sum\limits_{i=1}^{N}f_{i}(U_{i})+\sum\limits_{i=1}^{N}\left\langle
Y_{i},U_{i}-Z\right\rangle_{F}\\\ &+\sum\limits_{i=1}^{N}\left\langle
T_{i},h_{i}(U_{i})\right\rangle_{F}+\frac{\rho}{2}\sum\limits_{i=1}^{N}\lVert
U_{i}-Z\rVert_{F}^{2}+\frac{\rho}{2}\sum\limits_{i=1}^{N}\lVert
h_{i}(U_{i})\rVert_{F}^{2},\end{split}$ (6)

where $f_{i}(U_{i})=\lVert(I-U_{i}U_{i}^{\top})X_{i}\rVert_{F}^{2}$ is the
objective function, $\left\langle\cdot,\cdot\right\rangle_{F}$ is the
Frobenius inner product between two matrices.

This augmented Lagrangian makes the optimization problem more amenable to
iterative solution techniques while ensuring that constraints are respected.
The terms with Lagrange multipliers (or dual variables) –
$\sum\nolimits_{i=1}^{N}\left\langle Y_{i},U_{i}-Z\right\rangle_{F}$ and
$\sum\nolimits_{i=1}^{N}\left\langle T_{i},h_{i}(U_{i})\right\rangle_{F}$ –
quantify the influence of the constraints on achieving a common PCA
representation across distributed datasets. They ensure that each local PCA
matrix $U_{i}$ converges to a shared matrix $Z$ and adheres to the non-linear
constraints dictated by $h_{i}(U_{i})$. On the other hand, the penalty terms –
$(\rho/2)\sum\nolimits_{i=1}^{N}\lVert U_{i}-Z\rVert_{F}^{2}$ and
$(\rho/{2})\sum\nolimits_{i=1}^{N}\lVert h_{i}(U_{i})\rVert_{F}^{2}$ – are
governed by the penalty parameter $\rho$ to penalize any deviation of the
local PCA matrices $U_{i}$ from the global consensus matrix $Z$ or from
satisfying the orthogonality conditions $h_{i}(U_{i})$.

Since, the Frobenius inner product for matrices is analogous to the standard
dot products for vectors, we can obtain the gradients of the augmented
Lagrangian (6) w.r.t. dual variables $Y_{i}$ and $T_{i}$ as follows.

$\nabla_{Y_{i}}\mathcal{L}_{\rho}(U,Z,Y,T)=U_{i}-Z$
$\nabla_{T_{i}}\mathcal{L}_{\rho}(U,Z,Y,T)=h_{i}(U_{i})$

Algorithm 1 FedPCA in Euclidean Space (FedPE)

1: Randomly initialize $Z^{0}$ and $U_{i}^{0},\quad\forall i=1,\dots,N$

2: for $k=0,\ldots,T-1$ do

3: Sample subset clients $\mathcal{S}^{k}$

4: for each client $i\in\mathcal{S}^{k}$ in parallel do

5:
$U_{i}^{k+1}=\underset{U_{i}}{\operatorname{\text{arg\,min}}}\Bigl{\\{}f_{i}(U_{i})+\left\langle
Y_{i}^{k},U_{i}-Z^{k}\right\rangle_{F}+\left\langle
T_{i}^{k},h_{i}(U_{i})\right\rangle_{F}+\frac{\rho}{2}\lVert
U_{i}-Z^{k}\rVert_{F}^{2}+\frac{\rho}{2}\lVert
h_{i}(U_{i})\rVert_{F}^{2}\Bigr{\\}}$

6: end for

7: $CO$ updates
$Z^{k+1}=\frac{1}{|\mathcal{S}^{k}|}\sum_{i\in\mathcal{S}^{k}}{}(U_{i}^{k+1}+\frac{1}{\rho}Y_{i}^{k}$)

8: $CO$ broadcasts $Z^{k+1}$ to all clients

9: for each client $i\in\mathcal{S}^{k}$ in parallel do

10: $Y_{i}^{k+1}=Y_{i}^{k}+\rho\Bigl{(}U_{i}^{k+1}-Z^{k+1}\Bigr{)}$

11: $T_{i}^{k+1}=T_{i}^{k}+\rho\,h_{i}(U_{i}^{k+1})$

12: end for

13: end for

To solve the proposed problem, we employ an ADMM-based procedure in which a
subset of clients, denoted as $\mathcal{S}^{k}$, performs iterative updates
using a step size $\rho$ during the $k$-th communication round as follows.

####  III-A1 Primal Update (Local Update)

$\displaystyle U_{i}^{k+1}$
$\displaystyle=\underset{U_{i}}{\text{argmin}}\Bigl{\\{}f_{i}(U_{i})+\left\langle
Y_{i}^{k},U_{i}-Z^{k}\right\rangle_{F}+\left\langle
T_{i}^{k},h_{i}(U_{i})\right\rangle_{F}$
$\displaystyle\quad\quad\quad\quad\quad+\frac{\rho}{2}\lVert
U_{i}-Z^{k}\rVert_{F}^{2}+\frac{\rho}{2}\lVert
h_{i}(U_{i})\rVert_{F}^{2}\Bigr{\\}}$ (7)

####  III-A2 Consensus Update (Global Update)

$Z^{k+1}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\frac{1}{|\mathcal{S}_{k}|}\sum_{i\in\mathcal{S}_{k}}}\Bigl{(}U_{i}^{k+1}+\frac{1}{\rho}Y_{i}^{k}\Bigr{)}$
(8)

####  III-A3 Dual Update (Local Updates)

$Y_{i}^{k+1}=Y_{i}^{k}+\rho\Bigl{(}U_{i}^{k+1}-Z^{k+1}\Bigr{)}$ (9)
$T_{i}^{k+1}=T_{i}^{k}+\rho\,h_{i}(U_{i}^{k+1}).$ (10)

When $|\mathcal{S}_{k}|=N$, substituting $Z^{k+1}$ to $Y_{i}^{k+1}$ results in
$\frac{1}{N}\sum_{i=1}^{N}Y_{i}^{k+1}=0$. This which means $Z$ update can be
rewritten as[56]:

$\displaystyle Z^{k+1}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}U_{i}^{k+1}.$
(11)

These steps are repeated iteratively until convergence, at which point the
local PCA matrix $U_{i}$ and the consensus variable $Z$ agree, resulting in a
common matrix learned from the distributed data. This ADMM-based framework
collaboratively drives towards a consensus PCA solution while ensuring that
the unique characteristics of each dataset are preserved.

The above procedure is denoted as FedPCA in Euclidean space, abbreviated as
FedPE. The full procedure is detailed in Alg. 1. Specially, in each round, a
distinct subset of clients $\mathcal{S}^{k}$ is selected to perform the update
(Alg. 1, line 3). Each selected client is then involved in solving the primal
problem (7) on Euclidean space locally using gradient-based methods (Alg. 1,
line 5). Once finishing training, selected clients send their local PCA matrix
to the $CO$ for updating the common PCA matrix $Z$ (Alg. 1, line 7). Finally,
each selected client i update its dual variable $Y_{i}$ and $T_{i}$ based on
the latest $U_{i}$ and $Z$ (Alg. 1, lines 10-11). One main difference between
FedPE in this work and the one presented in [12] is that we employ a client
subsampling strategy during the federated training process. This offers
significant benefits, including reduced communication overhead, improved
scalability for large-scale IoT networks, and enhanced resilience to
stragglers or non-responsive devices.

Figure 2: Illustration of movement on a Grassmann manifold, represented by a
sphere. Given a point $U_{i}^{k}$ on Grassmann manifold $G$ and a vector
$\Delta$ on the tangent space denoted by $\mathcal{T}_{U_{i}}G$ at
$U_{i}^{k}$, a point $U_{i}^{k+1}$ is identified by exponential mapping
$\exp_{U_{i}^{k}}(\Delta)$. In our context, the manifold $G$ contains all
matrix $U_{i}$ satisfy the condition $U_{i}^{\top}U_{i}=I$.

###  III-B FedPCA using ADMM on Grassmann Manifold

Although FedPE effectively addresses the proposed problem, we further extend
its capacity by formulating FedPCA on Grassmann manifolds, denoted as FedPG,
to accelerate convergence. Notably, the orthonormality constraint utilized in
(3) signifies that the parameters of (7) can be projected onto a Grassmann
manifolds [25].

#### Orthonormality meets the Grassmannian

The Grassmann manifold, denoted as $G(n,d)$, represents a geometric space that
encapsulates sets of $d$-dimensional subspaces within an overaching
$n$-dimensional real or complex Euclidean space, subject to $n\geq d>0$ [25].
The distinct of this manifold lies in its representation capacity: Any point
on $G(n,d)$ can be identified by an $n\times d$ orthogonal matrix $U$ whose
column spans the corresponding subspace to ensure orthonormality
$U^{\top}U=I_{d}$. Hence, $G(n,d)$ is defined as follows.

$G(n,d)=\\{\text{span}(U):U\in\mathbb{R}^{n\times d},U^{\top}U=I_{d}\\}$ (12)

Fig. 2 shows an example of the Grassmann manifold as the surface of a sphere.
Here, the tangent space $\mathcal{T}_{U}G$ at $U$ captures the set of all
directions in which the subspace can ”move” but still stay very close to the
manifold. By capturing the space of all orthonormal subspaces, the Grassmann
manifold offers an elegant framework for addressing optimization problems with
orthonormality constraints (e.g., problem (3)) since these constraints are
automatically maintained when operations are performed within this manifold.

#### FedPCA via Grassmann Manifold Optimization

By framing problem (3) within the Grassmann manifold, the augmented Lagrangian
(6) evolves as follows.

$\begin{split}&{\mathcal{L}}(\\{U_{i}\\},Z,\\{Y_{i}\\})\\\
&=\sum\limits_{i=1}^{N}f_{i}(U_{i})+\sum\limits_{i=1}^{N}Y_{i}^{\top}(U_{i}-Z)+\frac{\rho}{2}\sum\limits_{i=1}^{N}\lVert
U_{i}-Z\rVert^{2}_{F}.\end{split}$ (13)

Here, the terms associated with $h_{i}(U_{i})$ are eliminated because the
inherent orthonormality constraints are satisfied.

Denoting the local update for each client $i$ at the $k$-th iteration by
following:

$F_{i}^{k}(U_{i}):=f_{i}(U_{i})+Y_{i}^{k\top}(U_{i}-Z^{k})+\frac{\rho}{2}\lVert
U_{i}-Z^{k}\rVert_{F}^{2}$ (14)

We then recast the update rules for $U_{i}$ and $Z$ on the Grassmann manifold
as follows.

$U_{i}^{k+1}=\text{arg\,min}_{U_{i}}\Bigl{\\{}F_{i}^{k}(U_{i})\Bigr{\\}}\quad$
(15)

Algorithm 2 FedPCA on Grassmann Manifold (FedPG)

1: Randomly initialize $Z^{0}$ and $U_{i}^{0},\quad\forall i=1,\dots,N$

2: for $k=1,\ldots,T$ do

3: Sample subset clients $\mathcal{S}^{k}$

4: for each client $i\in\mathcal{S}^{k}$ in parallel do

5: $U^{k,0}_{i}=U^{k}_{i}$

6: for $c=1,\ldots,C$ do

7:
$\nabla_{U}F_{i}^{k}(U_{i}^{k,c})=(I_{d}-U_{i}^{k,c}U_{i}^{k,c\top})F_{i}^{k}(U_{i}^{k,c})$

8: $U_{i}^{k,c+1}=R(U_{i}^{k,c}-\eta\nabla_{U}F_{i}^{k}(U_{i}^{k,c}))$

9: end for

10: $U_{i}^{k+1}=U_{i}^{k,C}$

11: end for

12: $CO$ updates
$Z^{k+1}=\frac{1}{|\mathcal{S}^{k}|}\sum_{i\in\mathcal{S}^{k}}{}(U_{i}^{k+1}+\frac{1}{\rho}Y_{i}^{k})$

13: $CO$ broadcasts $Z^{k+1}$ to all clients

14: for each client $i\in\mathcal{S}^{k}$ in parallel do

15: $Y_{i}^{k+1}=Y_{i}^{k}+\rho\Bigl{(}U_{i}^{k+1}-Z^{k+1}\Bigr{)}$

16: end for

17: end for

As detailed in Alg. 2 (lines 5-8), projected gradient descent on the Grassmann
manifold is applied in every local iteration in clients to find the update of
$U_{i}^{k+1}$ in (15). Considering the definition of the Grassmann manifold in
(12), we enforce the second constraint $U_{i}^{T}U_{i}=I$ by using the
projected gradient descent on the Grassmann manifold (Alg. 2, line 8) as
follows

$\displaystyle U^{k+1}_{i}=R(U^{k}_{i}-\eta\nabla_{U}F_{i}^{k}(U^{k}_{i})),$
(16)

Here, $\eta$ is the step size and $\nabla_{U}F_{i}^{k}(U^{k}_{i})$ is the
gradient of $F_{i}^{k}(U^{k}_{i})$ at the point $U_{i}^{k}$, and the
retraction function $R(\cdot)$ is the projection operation performed by QR
decomposition [25]. The projection process for $U^{k}_{i}$ and $U^{k+1}_{i}$
is graphically illustrated in Fig. 2. We compute the updates by projecting the
Euclidean gradient onto the tangent space of the manifold using orthogonal
projection $F_{i}^{k}(U^{k}_{i})\rightarrow\nabla_{U}F_{i}^{k}(U^{k}_{i})$
(Alg. 2, line 7) as follows.

$\nabla_{U}F_{i}^{k}(U^{k}_{i})=(I_{d}-U_{i}^{k}U_{i}^{k\top})F_{i}^{k}(U^{k}_{i}).$
(17)

Note that Equations (16) and (17) are repeatedly updated in $C$ local rounds
to solve Equation (15) (Alg. 2, lines 6-9).

The use of Grassmannian gradients in FedPG enables more precise and effective
steps toward the optimal solution, thus accelerating convergence. Furthermore,
PCA problems often include orthogonality constraints, which are complex to
handle in traditional Euclidean spaces and slow the convergence rate. FedPG
elegantly deals with these constraints by projecting the problem onto the
Grassmann manifold, where orthogonality constraints are naturally satisfied.
By doing so, it eliminates the need for extra steps to manage these
constraints, reducing computational overhead and facilitating faster
convergence.

Moreover, FedPG ensures an exact solution due to its unique feature of
projection onto the Grassmann manifold. This process maintains the inherent
orthogonality conditions that PCA requires, effectively preserving the
constraints of the original problem. Therefore, the solutions derived from
FedPG are precise and exact as they are always in the feasible region. On the
other hand, FedPE’s solutions are approximations. While FedPE works
effectively in solving the problem, it doesn’t inherently uphold the
orthogonality conditions. The resulting solutions, therefore, might be outside
the feasible region, and additional steps are necessary to bring the solution
back into the feasible space. This could lead to approximation errors, and
therefore the solutions are not always exact.

##  IV FedPCA: Convergence Analysis

In this section, we present a theoretical analysis to understand the
convergence properties of FedPG and FedPE within the FedPCA framework. To
begin with, we consider an FL environment with $N$ clients, indexed by
$i=1,2,\ldots,N$, and introduce the following notations for the analyses:

  1. 1.

$\mathcal{S}^{k}\subset\\{1,2,\ldots,N\\}$: denotes the subset of clients
selected in the $k$-th iteration.

  2. 2.

$k(i)$: denotes the latest iteration at which the client $i$ has its model
updated.

The orthonormality constraint in (3) is a critical component for our proposed
method. While FedPE handles this by introducing a surrogate linear constraint
in (4), FedPG relaxes this by leveraging projected gradient descent on
Grassmann manifolds. Hence, to ensure the generality of our framework, we
employ the augmented Lagrangian (LABEL:eqn:_lagrange) and rewrite it to adapt
for the convergence analysis of both algorithms as follows.

$\begin{split}\mathcal{L}(\\{U_{i}\\},Z,\\{Y_{i}\\})&=\sum\limits_{i=1}^{N}f_{i}(U_{i})+\sum\limits_{i=1}^{N}Y_{i}^{\top}(U_{i}-Z)\\\
&\qquad\qquad\qquad+\frac{\rho_{i}}{2}\sum\limits_{i=1}^{N}\lVert
U_{i}-Z\rVert^{2}_{F}\\\ \mathrm{s.t.}\quad U_{i}^{\top}U_{i}&=I.\end{split}$
(18)

In addition, we modify the global update (8) by following:

$Z^{k+1}=\frac{1}{|S_{k}|}\sum_{i\in
S_{k}}(U_{i}^{k+1}+\frac{1}{\rho_{i}}Y_{i}^{k})$ (19)

Accordingly, the local updates for $U_{i}^{k+1}$ in (7) and $Y_{i}^{k+1}$ in
(9) at a selected client $i$ will be changed as listed below.

$\displaystyle U_{i}^{k+1}$
$\displaystyle=\underset{U_{i}}{\operatorname{\text{arg\,min}}}\Bigl{\\{}f_{i}(U_{i})+\left\langle
Y_{i}^{k},U_{i}-Z^{k}\right\rangle_{F}$
$\displaystyle\qquad\qquad\qquad\qquad+\frac{\rho_{i}}{2}\sum\limits_{i=1}^{N}\lVert
U_{i}-Z^{k}\rVert^{2}_{F}\Bigr{\\}}$ (20) $\displaystyle\mathrm{s.t.}\quad
U_{i}^{\top}U_{i}=I$ $\displaystyle Y_{i}^{k+1}$
$\displaystyle=Y_{i}^{k}+\rho_{i}\Bigl{(}U_{i}^{k+1}-Z^{k+1}\Bigr{)}$ (21)

It is worth noting that this modification does not affect the performance of
FedPE and FedPG since the update results will be the same after one iteration.

Assumption. We first make the following assumptions:

  * •

A1: For each client’s objective function $f_{i},\forall i=1,\ldots,N$ there
exists a constant $L_{i}\geq 0$ such that for any two points $U_{i},Z_{i}$,
the gradient of $f_{i}$ satisfies the Lipschitz condition:

$\left\|\nabla
f_{i}\left(U_{i}\right)-\nabla{f_{i}}\left(Z_{i}\right)\right\|_{F}\leq
L_{i}\left\|U_{i}-Z_{i}\right\|_{F},\forall i$

  * •

A2: For all $i$, the penalty parameter $\rho_{i}$ is chosen large enough such
that the $U_{i}$ sub-problem (20) is strongly convex with modulus
$\mu_{i}\left(\rho_{i}\right)$.

  * •

A3: ${\mathcal{L}}(x)$ is bounded from below, i.e.,

$\underline{{\mathcal{L}}}\vcentcolon=\min_{X}{\mathcal{L}}(X)>-\infty$

######  Remark IV.1.

Assumptions A1 and A3 are standard in analyzing convergence properties for
ADMM. Under Assumption A2, as $\rho_{i}$ increases, the subproblem (20) will
be eventually strongly convex with respect to $U_{i}$. The associated strong
convexity modulus $\mu_{i}\left(\rho_{i}\right)$ is a monotonic increasing
function of $\rho_{i}$.

To establish a convergence guarantee of FedPE and FedPG, we aim to show the
following. Start with the variables $(\\{U_{i}^{k}\\},Z^{k},\\{Y_{i}^{k}\\})$.
After $T$ rounds (i.e., after all clients have participated at least once
since iteration $k$), the new variables
$(\\{U_{i}^{k+T}\\},Z^{k+T},\\{Y_{i}^{k+T}\\})$ lead to decrease in the global
augmented Lagrangian ${\mathcal{L}}$. Then we show these variables will
converge to a local stationary point of ${\mathcal{L}}$. Notably, our analysis
employs matrix variables, in contrast to other ADMM-based methods using vector
variables [46, 47, 48, 49]

###  IV-A Bound on the Successive Difference of Dual Variables

We establish an upper bound for the successive difference of the dual
variables $Y_{i}$ in terms of the primal variables $U_{i}$.

######  Lemma IV.2.

Suppose Assumptions A1 hold, then we have

$\left\|Y_{i}^{k+1}-Y_{i}^{k}\right\|_{F}^{2}\leq
L_{i}^{2}\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}^{2},\forall i,\forall k$
(22)

Proof. In the case of $i\not\in S^{k}$, (22) is true because both sides of
(22) are equal to zero. When $i\in S^{k}$, we have below equation from optimal
condition for (20).

$\nabla{f_{i}}\left(U_{i}^{k+1}\right)+Y_{i}^{k}+\rho_{i}(U_{i}^{k+1}-Z^{k+1})=0$
(23)

Substituting (23) to the dual variable update (21), the following equation is
derived.

$\nabla{f_{i}}\left(U_{i}^{k+1}\right)=-Y_{i}^{k+1}$ (24)

Combining with Assumption $A1$, and noting that for any given $i$, $U_{i}$ and
$Y_{i}$ are updated in the same iteration, we obtain for all
$i\in{S^{k}}/\\{0\\}$ :

$\begin{split}\left\|Y_{i}^{k+1}-Y_{i}^{k}\right\|_{F}&=\left\|\nabla{f_{i}}\left(U_{i}^{k+1}\right)-\nabla{f_{i}}\left(U_{i}^{k(i)}\right)\right\|_{F}\\\
&=\left\|\nabla{f_{i}}\left(U_{i}^{k+1}\right)-\nabla{f_{i}}\left(U_{i}^{k(i)}\right)\right\|_{F}\\\
&\leq
L_{i}\left\|U_{i}^{k+1}-U_{i}^{k(i)}\right\|_{F}=L_{i}\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}\end{split}$

###  IV-B Bound for the Augmented Lagrangian

Next, we show that the augmented Lagrangian (18) can be decreased sufficiently
and bounded below.

######  Lemma IV.3.

Suppose Assumption A1, A2 and A3 hold, then the augmented Lagrangian is
bounded as follows.

$\begin{split}&\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k+1}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)\\\
\leq&\sum_{i\in
S^{k}}\left(\frac{L_{i}^{2}}{\rho_{i}}-\frac{\mu_{i}}{2}\right)\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}^{2}-\frac{\gamma}{2}\left\|Z^{k+1}-Z^{k}\right\|^{2}\end{split}$

Proof. For each client $i\in S^{k}$, we can split the successive difference of
the augmented Lagrangian by following.

$\displaystyle\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k+1}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)$
$\displaystyle=\underbrace{\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k+1}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k}\right\\}\right)}_{A}$
$\displaystyle+\underbrace{\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)}_{B}$
$\displaystyle+\underbrace{\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)}_{C}$
(25)

Bounding Term A. The first term in (25) can be bounded as follows.

$\displaystyle\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k+1}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k}\right\\}\right)$
$\displaystyle=\sum_{i=1}^{N}\left\langle
Y_{i}^{k+1}-Y_{i}^{k},U_{i}^{k+1}-Z^{k+1}\right\rangle_{F}$ (26)
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\sum_{i\in
S^{k}}\frac{1}{\rho_{i}}\left\|Y_{i}^{k+1}-Y_{i}^{k}\right\|_{F}^{2}$

where (a) is achieved by using (21).

Bounding Term B. The second term in (25) can be bounded as follows.

$\displaystyle\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},Y^{k}\right)-\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)$
$\displaystyle\stackrel{{\scriptstyle(A2)}}{{\leq}}\sum_{i=1}^{N}\Big{(}\left\langle\nabla_{U_{i}}\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},Y^{k}\right),U_{i}^{k+1}-U_{i}^{k}\right\rangle$
$\displaystyle\hskip
140.00021pt-\frac{\mu_{i}}{2}\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}^{2}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}-\sum_{i\in
S^{+}}\frac{\mu_{i}}{2}\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}^{2}.$ (27)

where (b) is trivially true in the case of $i\not\in S^{k}$ as
$U_{i}^{k+1}=U_{i}^{k}$. In the case of $i\in S^{k}$, the
$\nabla_{U_{i}}\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\\{Y_{i}^{k}\\}\right)=0$
because of optimal condition of (20).

Bounding Term C. The third term in (25) can be bounded as follows.

$\displaystyle\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k},\\{Y_{i}^{k}\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\\{Y_{i}^{k}\\}\right)$
$\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}\sum_{i=1}^{N}\Big{(}\left\langle\nabla_{Z}\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k+1},\\{Y_{i}^{k}\\}\right),Z^{k+1}-Z^{k}\right\rangle$
$\displaystyle\hskip
140.00021pt-\frac{\gamma}{2}\left\|Z^{k+1}-Z^{k}\right\|_{F}^{2}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(d)}}{{=}}\sum_{i\in
S^{k}}-\frac{\gamma}{2}\left\|Z^{k+1}-Z^{k}\right\|_{F}^{2}.$ (28)

where in (c) we used the fact that
$\mathcal{L}\left(\left\\{U_{i}\right\\},Z,\\{Y_{i}\\}\right)$ is strongly
convex w.r.t. $Z$.

For (d), it is trivially true if $i\not\in S^{k}$ as $U_{i}^{k+1}=U_{i}^{k}$.
When $i\in S^{k}$, the
$\nabla_{Z}\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k+1},\\{Y_{i}^{k}\\}\right)=0$
because of the optimal condition of (20).

Sufficient decrease. Combining the inequalities Equation 26, Equation 27 and
Equation 28 together, we have

$\displaystyle{A+B+C}$
$\displaystyle=\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\left\\{Y_{i}^{k+1}\right\\}\right)-\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\left\\{Y_{i}^{k}\right\\}\right)$
(29) $\displaystyle\leq\sum_{i\in
S^{k}}-\frac{\gamma}{2}\left\|Z^{k+1}-Z^{k}\right\|_{F}^{2}+\left(\frac{L_{i}^{2}}{\rho_{i}}-\frac{\mu_{i}}{2}\right)\left\|U_{i}^{k+1}-U_{i}^{k}\right\|_{F}^{2}$

This implies that the value of the augmented Lagrangian will always decrease
if the following condition is satisfied:

$\rho_{i}\mu_{i}\geq 2L_{i}^{2}$ (30)

Lower bound for the augmented Lagrangian. We continue showing that
$\mathcal{L}\left(\left\\{U_{i}^{k}\right\\},Z^{k},\\{Y_{i}^{k}\\}\right)$
convergence, i.e., $\mathcal{L}$ is bounded below.

$\begin{split}&\mathcal{L}\left(\left\\{U_{i}^{k+1}\right\\},Z^{k+1},\\{Y_{i}^{k+1}\\}\right)\\\
&=\sum_{i=1}^{N}\Big{(}f_{i}\left(U_{i}^{k+1}\right)+\left\langle
Y_{i}^{k+1},U_{i}^{k+1}-Z^{k+1}\right\rangle\\\ &\hskip
140.00021pt+\frac{\rho_{i}}{2}\left\|U_{i}^{k+1}-Z^{k+1}\right\|_{F}^{2}\Big{)}\\\
&\stackrel{{\scriptstyle(e)}}{{=}}\sum_{i=1}^{N}\Big{(}f_{i}\left(U_{i}^{k+1}\right)+\left\langle\nabla
f\left(U_{i}^{k+1}\right),Z^{k+1}-U_{i}^{k+1}\right\rangle\\\ &\hskip
140.00021pt+\frac{\rho_{i}}{2}\left\|U_{i}^{k+1}-Z^{k+1}\right\|_{F}^{2}\Big{)}\\\
&\stackrel{{\scriptstyle(f)}}{{\geq}}\sum_{i=1}^{N}f_{i}\left(Z^{k+1}\right)=\underline{{\mathcal{L}}}\left(Z^{k+1}\right)\end{split}$

where (e) can be explained as following:

(e.1) If $i\in S^{k}$, based on (24):

$\left\langle
Y_{i}^{k+1},U^{k+1}-Z^{k+1}\right\rangle=\left\langle\nabla_{f}\left(U_{i}^{k+1}\right),Z^{k+1}-U^{k+1}\right\rangle$

(e.2) If $i\notin S^{k}$, $Y_{i}^{k+1}=Y_{i}^{k}=Y_{i}^{k(i)}$,
$U_{i}^{k+1}=U_{i}^{k}=U^{k(i)}$

$\begin{split}&\left\langle Y_{i}^{k+1},U_{i}^{k+1}-Z^{k+1}\right\rangle\\\
&=\left\langle\nabla
f\left(U_{i}^{k(i)}\right),Z^{k+1}-U_{i}^{k(i)}\right\rangle\\\
&=\left\langle\nabla
f\left(U_{i}^{k+1}\right),Z^{k+1}-U_{i}^{k+1}\right\rangle\end{split}$

From combining (e.1) and (e.2), we deduce that (e) holds true. Consequently,
due to the Lipschitz continuity of the gradient of $f_{i}$, (f) is also true.

######  Remark IV.4.

Lemma IV.3 shows that the augmented Lagrangian decreases by a sufficient
amount after each iteration. In practice, however, the penalty parameter
$\rho$ is difficult to estimate. Hence, for our experiments, we fine-tune
$\rho$ from a predefined range of values to select the best one.

###  IV-C Convergence of the augmented Lagrangian

Here, we combine Lemma (22) and (IV.3) to establish the following theorem.

######  Theorem IV.5.

Suppose the following are true:

  * •

After $T$ iterations, all clients have participated in training at least once.

  * •

The global loss function $\mathcal{L}(Z^{k},\\{U_{i}^{k}\\},\\{Y_{i}^{k}\\})$
is bounded below by a finite quantity $\underline{{\mathcal{L}}}$.

  * •

The hyperparameter $\rho_{i}$ for the augmented Lagrangian is chosen such that
$\rho_{i}\mu_{i}\geq 2L_{i}^{2}$.

Then the augmented Lagrangian ${\mathcal{L}}$ will monotonically decrease and
is convergent to a quantity of at least $\underline{{\mathcal{L}}}$. Further,
for all $i=1,\dots,N$ we have $\lim_{k\rightarrow\infty}\lVert
Z^{k+T}-U_{i}^{k+T}\rVert=0$.

Proof. Theorem IV.5 implies that the augmented Lagrangian decreases by a non-
negative amount after every $T$ global iterations. Given the fact that every
client will be updated at least once in the interval $[k,k+T]$, we conclude
that the limit
$\lim_{k\rightarrow\infty}\mathcal{L}(Z^{k+T},\\{U_{i}^{k+T}\\},\\{Y_{i}^{k+T}\\})$
exists and is at least $\underline{{\mathcal{L}}}$.

Now we prove the second statement. From Equation 29 and the fact that
${\mathcal{L}}$ converges, we conclude that as $k\rightarrow\infty$,

$\displaystyle\lVert Z^{k+T}-Z^{k}\rVert\rightarrow 0,\quad\lVert
U_{i}^{k+T}-U_{i}^{k}\rVert\rightarrow 0.$

Combining this with Equation 22, we have $\lVert
Y_{i}^{k+1}-Y_{i}^{k}\rVert\rightarrow 0$. Based on the definition of
$Y_{i}^{t+1}$, this implies that

$\lVert U_{i}^{k+1}-Z^{k+1}\rVert\rightarrow 0$

######  Remark IV.6.

Theorem IV.5 establishes that the sequence of primal and dual variables
updated after each $T$ global iterations of Algorithm 1 and Algorithm 2
converges. We further have $\lVert U_{i}^{k+1}-Z^{k+1}\rVert\rightarrow 0$,
i.e., the consensus constraint is satisfied.

###  IV-D Convergence to stationary point

In the below theorem we present another guarantee that the limit of the
sequence $(Z^{k},\\{U_{i}^{k}\\},\\{Y_{i}^{k}\\})$ is the a stationary
solution for Problem Equation 5.

######  Theorem IV.7.

Suppose the assumptions in Theorem IV.5 hold. Then the limit point
$(Z^{*},\\{U_{i}^{*}\\},\\{Y_{i}^{*}\\})$ of the sequence
$(Z^{k},\\{U_{i}^{k}\\},\\{Y_{i}^{k}\\})$ is a stationary solution to Problem
Equation 5. That is, for all $i$,

$\displaystyle\nabla f_{i}(Z^{*})+Y_{i}^{*}=0\text{ and }Z^{*}=U_{i}^{*}.$

Proof. See the proof to [49, Theorem 2.4]. Here we make some remarks. First,
because FedPE and FedPG allow for client sampling, the proof relies on an
assumption that all clients will participate after some fixed number of rounds
(in particular, $T$ rounds). Second, since local objectives are smooth
(possibly non-convex), the primal variables $U_{i}$ are updated using first-
order methods like SGD or projected GD. Finally, the proof aims to show that
after every $T$ rounds, we obtain a sufficient decrease in the augmented
Lagrangian; coupled with the fact that the augmented Lagrangian is bounded
below, the sequence of variables converges to a stationary point of
${\mathcal{L}}$.

###  IV-E Convergence rate of the augmented Lagrangian

We first define gradient of the Augmented Lagrangian as:

$\displaystyle\hat{\nabla}\mathcal{L}\left(\left\\{U_{i}\right\\},Z,Y\right):=\left[\begin{array}[]{c}\nabla_{U_{1}}\mathcal{L}\left(\left\\{U_{i}\right\\},Z,Y\right)\\\
\nabla_{U_{2}}\mathcal{L}\left(\left\\{U_{i}\right\\},Z,Y\right)\\\ \vdots\\\
\nabla_{U_{N}}\mathcal{L}\left(\left\\{U_{i}\right\\},Z,Y\right)\end{array}\right]$
(35)

We aim to use the below quantity to establish the progress of the proposed
methods:

$\begin{split}&P(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i}):=\sum_{i=1}^{N}||U^{k}_{i}-Z^{k}||^{2}\\\
&+||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||^{2}\\\
\end{split}$

It can be verified that if $P(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})\rightarrow 0$,
then a stationary solution is obtained, which $U_{i}$ converges to $Z$ and
gradient for Augmented Lagrangian of each client $i$ is equal to zero.

######  Theorem IV.8.

Suppose that Assumptions A.1, A.2, A.3 and Theorem IV.5 are satisfied. Let
$T(\epsilon)$ denote an iteration index in which the following inequality is
achieved:

$T(\epsilon):=\min(\\{k|P(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})\\}\leq\epsilon,k\geq
0)$

for some $\epsilon>0$. Then there exists some constant $\beta>0$ such that

$T(\epsilon)\leq\frac{\beta(\mathcal{L}(\\{U^{1}_{i}\\},Z^{1},Y^{1}_{i})-\underline{\mathcal{L}})}{\epsilon}$

where $\underline{\mathcal{L}}$ is defined by Assumption A.3.

Proof. Based on the optimal condition of $U_{i}$ (7):

$\nabla f_{i}(U^{k+1}_{i})+Y^{k}_{i}+\rho_{i}(U^{k+1}_{i}-Z^{k+1})=0$ (36)

By definition of $\mathcal{L}$, we can have:

$\begin{split}&||\nabla_{U_{i}}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||=||\nabla
f_{i}(U^{k}_{i})+Y^{k}_{i}+\rho_{i}(U^{k}_{i}-Z^{k})||\\\
&\stackrel{{\scriptstyle(g)}}{{=}}||\nabla
f_{i}(U^{k}_{i})+Y^{k}_{i}+\rho_{i}(U^{k}_{i}-Z^{k})\\\ &-(\nabla
f_{i}(U^{k+1}_{i})+Y^{k}_{i}+\rho_{i}(U^{k+1}_{i}-Z^{k+1}))||\\\ &\leq||\nabla
f_{i}(U^{k}_{i})-\nabla
f_{i}(U^{k+1}_{i})||+\rho_{i}||U^{k+1}_{i}-U^{k}_{i}||\\\
&+\rho_{i}||Z^{k+1}-Z^{k}||\\\
&\stackrel{{\scriptstyle(h)}}{{\leq}}L_{i}||U^{k+1}_{i}-U^{k}_{i}||+\rho_{i}||U^{k+1}_{i}-U^{k}_{i}||+\rho_{i}||Z^{k+1}-Z^{k}||\\\
&=(L_{i}+\rho_{i})||U^{k+1}_{i}-U^{k}_{i}||+\rho_{i}||Z^{k+1}-Z^{k}||\end{split}$
(37)

Where (g) is obtained by optimal condition of $U_{i}$ in (36). To derive (h),
the first term is bounded by assumption $A.1$ and the second term is bounded
by Lemma 22. Based on (37), we have the following:

$\begin{split}&||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||\leq\sum_{i=1}^{N}(L_{i}+\rho_{i})||U^{k+1}_{i}-U^{k}_{i}||\\\
&+\sum_{i=1}^{N}\rho_{i}||Z^{k+1}-Z^{k}||\end{split}$ (38)

By taking
$\sigma_{1}=\max\\{\sum_{i=1}^{N}\rho_{i},L_{1}+\rho_{1},...,L_{N}+\rho_{N}\\}$,
it is apparent that $\sigma_{1}>0$. We can have following:

$||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||\leq\sigma_{1}(||Z^{k+1}-Z^{k}||+\sum_{i=1}^{N}||U^{k+1}_{i}-U^{k}_{i}||)$
(39)

From (9), we have

$\begin{split}&\sum_{i=1}^{N}||U^{k+1}_{i}-Z^{k+1}||=\sum_{i=1}^{N}\frac{1}{\rho_{i}}||Y^{k+1}_{i}-Y^{k}_{i}||\\\
&\leq\sum_{i=1}^{N}\frac{L_{i}}{\rho_{i}}||U^{k+1}_{i}-U^{k}_{i}||\end{split}$
(40)

(39) and (40) imply for some $\sigma_{3}\geq 0$:

$\begin{split}&\sum_{i=1}^{N}||U^{k}_{i}-Z^{k}||^{2}+||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||^{2}\\\
&\leq\sigma_{3}(||Z^{k+1}-Z^{k}||^{2}+\sum_{i=1}^{N}||U^{k+1}_{i}-U^{k}_{i}||^{2})\\\
\end{split}$ (41)

It is noteworthy that we have changed from $||U^{k+1}_{i}-Z^{k+1}||^{2}$ to
$||U^{k}_{i}-Z^{k}||^{2}$. This is because we have proved that
$||U^{k+1}_{i}-Z^{k+1}||\rightarrow 0$, which means the gap between these two
terms $||U^{k+1}_{i}-Z^{k+1}||^{2}$ and $||U^{k}_{i}-Z^{k}||^{2}$ are finite.
Therefore, we always can find a finite number $\sigma_{3}\geq 0$ to satisfy
$||U^{k}_{i}-Z^{k}||^{2}\leq\sum_{i=1}^{N}\sigma_{3}||U^{k+1}_{i}-U^{k}_{i}||^{2}$.
From lemma IV.3, there exists a constant
$\sigma_{2}=min\\{\frac{L^{2}_{i}}{\rho_{i}}-\frac{\mu_{i}}{2},\mu\\}_{i=1}^{N}$
such that:

$\begin{split}&\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})-\mathcal{L}(\\{U^{k+1}_{i}\\},Z^{k+1},Y^{k+1}_{i})\\\
&\geq\sigma_{2}(||Z^{k+1}-Z^{k}||^{2}+\sum_{i=1}^{N}||U^{k+1}_{i}-U^{k}_{i}||^{2})\end{split}$
(42)

Combining (LABEL:r3:convergence_rate_5) and (42) we have:

$\begin{split}&\sum_{i=1}^{N}||U^{k}_{i}-Z^{k}||^{2}+||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||^{2}\\\
&\leq\frac{\sigma_{3}}{\sigma_{2}}(\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})-\mathcal{L}(\\{U^{k+1}_{i}\\},Z^{k+1},Y^{k+1}_{i}))\end{split}$
(43)

Summing both sides of the above inequality over $k=1,...,T$. We have the
following:

$\begin{split}&\sum_{k=1}^{T}(\sum_{i=1}^{N}||U^{k}_{i}-Z^{k}||^{2}+||\hat{\nabla}\mathcal{L}(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})||^{2})\\\
&\leq\frac{\sigma_{3}}{\sigma_{2}}(\mathcal{L}(\\{U^{1}_{i}\\},Z^{1},Y^{1}_{i})-\mathcal{L}(\\{U^{T+1}_{i}\\},Z^{T+1},Y^{T+1}_{i}))\\\
&\leq\frac{\sigma_{3}}{\sigma_{2}}(\mathcal{L}(\\{U^{1}_{i}\\},Z^{1},Y^{1}_{i})-\underline{\mathcal{L}})\end{split}$
(44)

The last inequality uses the fact that
$L(\\{U^{T+1}_{i}\\},Z^{T+1},Y^{T+1}_{i})$ is decreased and lower bounded by
$\underline{\mathcal{L}}$ by $A.3$. By putting
$T(\epsilon):=\min\\{k|P(\\{U^{k}_{i}\\},Z^{k},Y^{k}_{i})\leq\epsilon,k\geq
0\\}$, we can rewrite (44) as follows.

$T(\epsilon)\epsilon\leq\frac{\sigma_{3}}{\sigma_{2}}(\mathcal{L}(\\{U^{1}_{i}\\},Z^{1},Y^{1}_{i})-\underline{\mathcal{L}})$
(45)

or

$T(\epsilon)\leq\frac{\sigma_{3}}{\sigma_{2}}\frac{(\mathcal{L}(\\{U^{1}_{i}\\},Z^{1},Y^{1}_{i})-\underline{\mathcal{L}})}{\epsilon}$
(46)

This means that for a given $\epsilon$, the proposed algorithm converges with
rate of $O(\frac{1}{T(\epsilon)})$. We note that the use of $\sum_{i}^{N}$ in
the above proof does not mean that we require participation from all clients.
This is used to find an upper bound. Let
$\beta=\frac{\sigma_{3}}{\sigma_{2}}$, theorem IV.8 is achieved.

###  IV-F Convergence rates of the proposed methods

######  Theorem IV.9.

Let $T(\epsilon)$ be defined as in Theorem IV.8. Suppose the local update (7)
in FedPE is performed using gradient descent with $C$ iterations. Then, the
convergence rates for FedPE and FedPG are
$O\left(\frac{1}{CT(\epsilon)}\right)$ and
$O\left(\frac{1}{C^{2}T(\epsilon)}\right)$, respectively.

Proof. Based on Theorem IV.8, the general convergence rates of
$O(\frac{1}{T(\epsilon)})$ are proved for both FedPE and FedPG. While the
learning rate of solving problem (15) can be accelerated by using gradient
descent on Grassmann manifold with convergence rate of $O(1/C^{2})$ [57,
Theorem 1], convergence rate for gradient descent on Euclidean space to solve
(7) is $O(1/C)$ for strongly convex functions [58]. Therefore, the convergence
rates of FedPE and FedPG can be estimated as $O(\frac{1}{CT(\epsilon)})$ and
$O(\frac{1}{C^{2}T(\epsilon)})$ respectively.

##  V Experimental Results

In this section, we first present the datasets, model settings and measures
for performance assessment. Subsequently, to illustrate the efficacy of the
proposed methods, we contrast FedPE and FedPG with various benchmark
strategies within the context of an IDS deployment in IoT networks.

###  V-A Experimental settings

(a)

(b)

Figure 3: Class distribution of test sets for UNSW-NB15 and TON-IoT

In our research, all experiments are conducted in a coding environment powered
by an AMD Ryzen 3970X Processor with 64 cores and 256GB of RAM. Additionally,
four NVIDIA GeForce RTX 3090 GPUs are employed to accelerate the training
process. For software, we rely on Python 3 as our primary language,
supplemented by advanced machine learning and optimization frameworks such as
PyTorch [59] and Scikit-Learn [60]. Our code can be found at:
https://github.com/dual-grp/FedPCA_AnomalyDetection.

####  V-A1 Datasets and Metrics

In this study, we employ the UNSW-NB15 [13] and TON-IoT Network [14] datasets.
The former, a benchmark in network intrusion detection system (NIDS) research,
encompasses a myriad of network traffic scenarios, with various attack types
interlaced with normal activities commonly found in real-world network
(internet) environments. The latter, TON-IoT, includes heterogeneous data
sourced from Telemetry datasets of IoT and IIoT sensors, amassed through
parallel processing to capture a spectrum of normal and cyber-attack events.
Following preprocessing, from the UNSW-NB15 dataset, we utilized a training
set of 56,000 normal samples and a test set comprising 20,000 normal and
45,332 abnormal samples, all with 39 numerical features. From the TON-IoT
dataset, our training set consists of 114,956 normal samples, while the test
set includes 10,000 normal samples and 56,557 abnormal samples, all with 49
numerical features. The class distribution of the test sets for both datasets
is illustrated in Fig.3(a) and Fig.3(b) respectively. These carefully selected
and prepared datasets serve as a solid foundation for the evaluation of our
proposed approach.

In the setting of a host-based ML-IDS deployment, each IoT device functions as
a localized ML-IDS, using its unique data to develop an anomaly detection
model. This approach reflects the inherent diversity of traffic distribution
across disparate clients, and to mimic this realistically, the comprehensive
training set is equally subdivided into 100 individual client datasets, each
representing non-identically distributed (non-i.i.d.) local data. This
subdivision is achieved by grouping records based on the number of connections
that contain the same service (ct_srv_src) for UNSW-NB15 and the range of
source ports (srcports) for TON-IoT.

####  V-A2 Baselines

In order to highlight the efficacy of FedPE and FedPG in IoT network anomaly
detection, we conduct a comprehensive comparative study against a variety of
unsupervised techniques including:

  1. a)

Self-learning PCA: a stand-alone method where each client individually learns
its PCA matrix using only its local dataset. By identifying the principal
patterns within this dataset, anomalies are detected when there’s a
substantial discrepancy between the original data and its PCA-reconstructed
version.

  2. b)

AutoEncoder (AE): an unsupervised learning method constructed of an encoder
and a decoder in the form of neural networks. AE facilitates anomaly detection
by compressing input data during encoding and then attempting to reconstruct
it. Anomalies are identified when there’s a notable difference between the
input data and its reconstructed version, suggesting the input doesn’t conform
to typical patterns within the dataset [6].

  3. c)

Bidirectional Generative Adversarial Network (BiGAN): a generative model
composed of a generator, a discriminator and an encoder. During BiGAN’s
adversarial training, the encoder maps real data into latent representations.
The generator then tries to recreate data from these representations, aiming
to closely match the original input. Concurrently, the discriminator
distinguishes between pairs of real data and their latent encodings, and the
output from the generator with its corresponding latent representations. When
trained solely on normal data, BiGAN can detect anomalies by noting high
reconstruction errors, which indicate deviations from the learned
representations of normal data [7, 8].

We subsequently employ the de-facto FedAvg[16] to ensure consistency in the
training of AE and BiGAN within federated settings. In line with this, we
establish FedAE and FedBiGAN – which utilize two-layer neural networks for
their components, and FedAE-2 and FedBiGAN-2 – which incorporate four-layer
neural networks for more complex non-linear structures. For Self-learning PCA,
we train a distinct PCA matrix tailored for each client’s dataset and then
assess the average performance across all clients on detecting anomalies on
the test set. The selection of these baseline models, widely acclaimed in the
field and each offering distinct methodologies for anomaly detection, ensures
a robust evaluation of our methods against diverse unsupervised paradigms.

####  V-A3 Performance Metrics

In this work, we are primarily concerned with identifying anomalies in network
traffic. Hence, the main metrics employed are:

  1. 1.

Accuracy: calculates the ratio of correctly identified both normal and
anomalies to the total instances in the dataset, offering a general
perspective on the model performance.

  2. 2.

Precision: represents the ratio of true anomalies to all detections labeled as
anomalies, assessing the exactness of the identified anomalies.

  3. 3.

Recall: quantifies the proportion of actual anomalies that are correctly
flagged by the detection system, assessing the ability of a model to capture
all potential threats.

  4. 4.

F1-Score: calculates the harmonic mean of precision and recall, underscoring
the balance between the detection of true anomalies and the risk of false
alarms.

  5. 5.

False Negative Rate (FNR): Represents the fraction of actual anomalies that
are incorrectly flagged as normal.

  6. 6.

Area Under the Receiver Operating Characteristic curve (AUC-ROC): A graphical
representation plotting True Positive Rate (TPR) against False Positive Rate
(FPR) across varying threshold settings, furnishing a comprehensive
performance overview.

  7. 7.

Precision-Recall (PR) curves: Serving as a visual instrument to gauge the
relationship between precision and recall, this becomes vital, particularly in
imbalanced datasets where anomalies are few.

####  V-A4 Training Details

To provide a practical test for our proposed methods, we train our federated
PCA models on each client’s dataset, without inter-client data sharing. The
comprehensive testing set, inclusive of unknown attacks, is applied uniformly
to all methods. Preceding training, all features undergo normalization through
a z-score function, utilizing the standard deviation and mean values
calculated from the corresponding training set. The following experiments have
hyperparameters fine-tuned via a grid search to ensure optimal test
performance. Furthermore, each communication round includes a randomly
selected subset of clients, constituting 10% of the total count, replicating
real-world scenarios where not all devices might be available for
communication simultaneously.

###  V-B Experimental Results

####  V-B1 Efficiency of Federated PCA on Grassmann Manifold

(a)

(b)

(c)

(d)

Figure 4: (a,b) Effects of $\rho$ on the performance of FedPG and (c,d)
Training loss over global iterations

(a)

(b)

(c)

(d)

Figure 5: Effects of rank-$k$ on the performance of FedPE and FedPG

(a)

(b)

(c)

(d)

Figure 6: Test accuracy and training time over global iterations of FedPE and
FedPG on UNSW-NB15 (a,b) and TON-IoT (c,d)

In our analysis, we scrutinize the convergence efficacy of our Federated PCA
algorithms — FedPE (Federated PCA over Euclidean Space) and FedPG (Federated
PCA over Grassmann Manifold) — as depicted in Fig.4, Fig. 6 and Fig. 5.
Notably, for the same number of local communication rounds ($LE=10$), FedPG
displays a significantly faster convergence compared to FedPE. It’s worth
noting that FedPE can achieve quicker convergence by augmenting the value of
$LE$, although this also extends the training duration. These findings
underscore FedPG’s ability to deliver robust detection performance with
decreased training time, an attribute highly sought after in practical IDS
development contexts. Fig.4(a), 4(b) illustrate the precise performance of
FedPG across various thresholds. In our study, we uniformly set $\rho=1.0$
across all methods and investigate their detection performance by various
metrics.

The impact of low-rank approximation is presented in Fig. 5(a) and 5(b).
Surprisingly, increasing the rank-k does not necessarily lead to an
improvement in the performance of both FedPG and FedPE, as observed in both
datasets. This suggests that our PCA-based anomaly detection algorithms can
maintain a low memory footprint and operate efficiently with low computational
complexity, making them ideal for resource-constrained environments. Despite
FedPG and FedPE providing better approximations for a given dataset with
larger rank-k, they also tend to precisely recover data points lying in high-
dimensional space. However, high-dimensional data samples often act as
outliers, resembling attacks at times. This creates a dilemma between
achieving optimal performance and accurate approximation, as seen in the loss-
accuracy contrast presented in Fig. 5(c), 5(d), and Fig. 5(a), 5(b). Abnormal
detection using PCA relies on measuring the distance between data points and
principal components, making it challenging to differentiate between high-
dimensional data points and malicious data.

To examine the effectiveness of FedPG against FedPE in terms of computational
complexity, we investigate the training time of both algorithms on the same
machine. While Fig. 6(a) and 6(c) demonstrate that FedPE surpasses FedPG in
terms of the total number of communication rounds, our investigation reveals
that FedPG is more effective in managing computational costs, as depicted in
Fig. 6(b) and 6(d). It is worth noting that FedPG sacrifices a small amount of
computational resources in exchange for significantly higher performance.

####  V-B2 Effects of number clients on performance

(a)

(b)

Figure 7: Effects of number of clients on training time and performance of
FedPE and FedPG

We evaluate the performance of FedPE and FedPG on different scales of IoT
systems by varying the number of clients. The results illustrated in Fig. 7
reveal a noticeable trade-off between training time and accuracy for these
algorithms.

For both datasets, FedPG consistently demonstrated a significantly shorter
training time as compared to FedPE, regardless of the number of clients. For
example, on the TON-IoT dataset, the training time of FedPG was less than a
sixth of the training time of FedPE for the same number of clients. In
contrast, the accuracy of FedPG remained relatively constant and independent
of the number of clients in both datasets. FedPE, on the other hand,
demonstrated an improvement in accuracy as the number of clients increased.
However, even with this improvement, it was unable to reach the accuracy of
FedPG on the TON-IoT dataset. For the UNSW-NB15 dataset, both algorithms
performed comparably in terms of accuracy, with FedPE even slightly surpassing
FedPG in fewer clients, but then slightly underperforming in more clients.

These results suggest that FedPG could offer substantial time savings over
FedPE without compromising detection accuracy, particularly for large numbers
of clients. However, the appropriate choice of algorithm may also depend on
the specific requirements and constraints of the task and system at hand, such
as the importance of speed versus detection accuracy and the number of clients
in the system.

(a)

(b)

(c)

(d)

Figure 8: (a,b) Receiver Operating Characteristic (ROC) curve and (c,d)
Precision-Recall curve

(a)

(b)

(c)

(d)

Figure 9: Visualization of UNSW-NB15 traffic after reconstruction using
different methods TABLE I: Performance of network anomaly detection tasks on
UNSW-NB15 and TON-IoT datasets.

Dataset | UNSW-NB15 | TON-IoT
---|---|---
Method | Acc | Pre | Recall | F1-Score | FNR | Acc | Pre | Recall | F1-Score | FNR
Self-learning PCA | 63.80 | 83.19 | 59.94 | 69.68 | 40.06 | 51.81 | 86.79 | 51.07 | 64.30 | 48.93
FedAE | 80.88 | 80.66 | 95.27 | 87.36 | 4.73 | 87.28 | 93.52 | 91.36 | 92.43 | 8.64
FedAE-2 | 82.80 | 83.45 | 93.80 | 88.32 | 6.20 | 89.09 | 94.68 | 92.34 | 93.50 | 7.66
FedBiGAN | 81.21 | 83.04 | 91.63 | 87.12 | 8.36 | 88.53 | 93.98 | 92.42 | 93.19 | 7.57
FedBiGAN-2 | 81.59 | 81.98 | 94.16 | 87.65 | 5.84 | 89.08 | 95.05 | 91.93 | 93.47 | 8.07
FedPE (ours) | 82.15 | 81.06 | 95.14 | 87.53 | 6.25 | 87.06 | 89.82 | 94.94 | 92.31 | 6.13
FedPG (ours) | 81.95 | 82.82 | 93.36 | 87.77 | 6.63 | 88.94 | 92.79 | 94.32 | 93.55 | 5.68

####  V-B3 Performance of Network Anomaly Detection Tasks

We evaluate the performance of FedPG alongside other baselines in the context
of network anomaly detection on both the UNSW-NB15 and TON-IoT datasets. For
all methods, we establish optimal thresholds to delineate normal from
anomalous behavior using the AUC-ROC curve corresponding to their
reconstruction errors. This approach aims to strike a balance by maximizing
TPR and minimizing FPR, ensuring robustness and precision in anomaly
detection. Such an evidence-driven thresholding strategy underscores the
reliability of the reconstruction error as a metric for detecting anomalies in
network traffic.

As indicated in Table I, Self-learning PCA, while offering a foundational
unsupervised technique, lags behind in performance when compared to other
methods. This outcome aligns with expectations, given the diverse and highly
heterogeneous nature of clients’ data. The performance of FedPG and FedPE is
comparable and slightly better than FedAE and FedBiGAN. Additionally, they
demonstrate a competitive performance against more sophisticated non-linear
structures such as FedAE-2 and FedBiGAN-2. It should be noted that while FedPG
may not outperform all other techniques across all metrics, it does exhibit a
commendable balance of measures.

On the UNSW-NB15 dataset, FedPG not only shows a high accuracy (81.95%) but
also balances this with a relatively low false negative rate (6.63%),
indicating a robust ability to correctly classify normal behavior while
minimizing the chance of missing true anomalies. Similarly, on TON-IoT, FedPG
holds its own by showcasing the highest recall (94.32%), reflecting its
superior ability to correctly identify anomalous events. Moreover, it has the
lowest false negative rate (5.68%) among the evaluated methods, which
underscores its performance stability. Furthermore, the linear nature of FedPG
simplifies model interpretability, often leading to more computationally and
communication-efficient solutions while still being able to capture
significant patterns in the data for network anomaly detection. Meanwhile,
although FedAE, FedAE-2, FedBiGAN, and FedBiGAN-2 can potentially capture more
complex relationships in the data, they tend to require more computational and
memory resources, harder to interpret and may not always result in better
performance. This suggests that the balanced performance of our linear method
FedPG is an attractive alternative for real-world IDS where simplicity,
efficiency, and interpretability are crucial.

The superior performance of FedPG is further validated by the area under the
Receiver Operating Characteristic (AUC-ROC) and Precision-Recall curves,
depicted in Fig. 8. On both the UNSW-NB15 and TON-IoT datasets, FedPG
demonstrates higher AUC-ROC values of 0.82, surpassing other baselines (Fig.
8(a), 8(b)), indicating its enhanced ability to maintain high true positive
rates while reducing false positive rates. This performance superiority is
corroborated by the Precision-Recall curves, with FedPG achieving an Average
Precision (AP) of 0.89 and 0.96 on UNSW-NB15 and TON-IoT datasets respectively
(Fig. 8(c), 8(d)), comparable with the competing methods. These results
collectively affirm the robustness of FedPG in achieving high precision at
various levels of recall, underscoring its potential as an effective solution
for network anomaly detection.

To delve deeper into the efficacy of the suggested technique, we employed 3D
scatter plots to illustrate the transformation of UNSW-NB15 traffic logs as
per the techniques described, as depicted in Fig.9. For the purpose of
plotting, we chose three features111Network features in Fig.9: duration
(’dur’), source TCP base sequence number (’stcpb’), source interpacket arrival
time (’sinpkt’), following which we reconstructed the original logs from the
latent representation discerned by Self-Learning PCA, FedAE, and FedPG. It’s
evident that FedPG essentially repositions original logs from an established
coordinate framework into an alternate one, proficient in distinguishing
between regular logs and anomalies. This affirms that when FedPG redevelops
the original logs from the primary components, it results in a minimal
reconstruction discrepancy for regular data but a substantial error.

####  V-B4 Communication Efficiency

Figure 10: The size of model exchanged in each communication round. Figure 11:
The memory consumption and inference time of all methods.

In the realm of IoT anomaly detection, the transmission size of models during
communication rounds plays a pivotal role, directly influencing bandwidth and
overall efficiency. Our study reveals a significant communication advantage of
FedPG when juxtaposed against FedAE, FedBiGAN, FedAE-2, and FedBiGAN-2.
Specifically, when considering the UNSW-NB15 datasets, FedPG boasts a
communication efficiency that’s 6.7 and 10 times superior to FedAE and
FedBiGAN respectively. While FedAE-2 and FedBiGAN-2 may have a slight edge
performance-wise, their demand for bandwidth is significantly higher,
necessitating model sizes 10 and 15 times that of FedPG. A similar efficiency
is observed for the TON-IoT dataset, where FedPG outperforms by factors of up
to 20 times compared to other methods. These results hold considerable
significance in the context of FL, where the reduction of bandwidth
utilization is a paramount concern. This evidence underscores the ability of
FedPG to strike an effective balance between reliable anomaly detection and
optimal resource usage, thus bolstering its suitability for real-world NIDS
applications.

####  V-B5  Memory and Time Complexity

We analyze all models through the lens of GPU memory consumption (measured in
a percentage) and wall-clock time for inference (measured in a microsecond -
$\mu s$), a perceptible trade-off emerges between these two metrics. As
depicted in Fig. 11, FedPG and FedPE are relatively memory-efficient,
consuming around 9.03% and 8.01% of GPU memory during training, respectively.
In contrast, the more sophisticated models, FedAE, FedAE-2, FedBiGAN, and
FedBiGAN-2, are more resource-demanding, utilizing larger amounts of memory.
Despite the smaller memory footprint, FedPG and FedPE do not compromise on the
inference speed, clocking an impressive inference wall-clock time of
approximately 7$\mu s$. This slightly outperforms the inference time taken by
other non-linear methods. Thus, the superior balance struck by FedPG and FedPE
between GPU memory usage and wall-clock time solidifies their efficiency,
making them well-suited for resource-restricted environments.

##  VI Conclusion

In this study, we analyze the convergence of the federated PCA framework on
the Grassmann manifold, thereby enhancing efficient unsupervised anomaly
detection in IoT networks with realistic datasets. The proposed framework
empowers IoT devices to collaboratively create a profile of normal network
traffic while abstaining from the exchange of local data. Specifically, we
devise FedPE, an innovative federated algorithm that approximates PCA
utilizing the principle of ADMM and is accompanied by theoretical guarantees.
We further conceptualize the federated PCA via Grassmannian optimization and
propose another algorithm, FedPG. This algorithm is geared towards ensuring a
swift training process and early anomaly detection, even within computing
environments that are resource-constrained. Through our experimental
evaluations, we establish the adaptability of FedPE and FedPG to the IoT
environment and demonstrate their superior performance over conventional
approaches when tested on the UNSW-NB15 and TON-IoT datasets.

## References

  * [1] A. Zanella, N. Bui, A. Castellani, L. Vangelista, and M. Zorzi, “Internet of things for smart cities,” _IEEE Internet of Things Journal_ , vol. 1, no. 1, pp. 22–32, 2014.
  * [2] X. Yang, P. Zhao, X. Zhang, J. Lin, and W. Yu, “Toward a gaussian-mixture model-based detection scheme against data integrity attacks in the smart grid,” _IEEE Internet of Things Journal_ , vol. 4, no. 1, pp. 147–161, 2016\.
  * [3] Z. Afroz, G. Shafiullah, T. Urmee, and G. Higgins, “Modeling techniques used in building hvac control systems: A review,” _Renewable and sustainable energy reviews_ , vol. 83, pp. 64–84, 2018.
  * [4] A. A. Cook, G. Mısırlı, and Z. Fan, “Anomaly detection for iot time-series data: A survey,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6481–6494, 2019.
  * [5] D. Chou and M. Jiang, “A survey on data-driven network intrusion detection,” _ACM Comput. Surv._ , vol. 54, no. 9, oct 2021.
  * [6] C. Ieracitano, A. Adeel, F. C. Morabito, and A. Hussain, “A novel statistical analysis and autoencoder driven intelligent intrusion detection approach,” _Neurocomputing_ , vol. 387, pp. 51–62, 2020.
  * [7] H. Zenati, C.-S. Foo, B. Lecouat, G. Manek, and V. R. Chandrasekhar, “Efficient gan-based anomaly detection,” _ArXiv_ , vol. abs/1802.06222, 2018\.
  * [8] H. Zenati, M. Romain, C. Foo, B. Lecouat, and V. Chandrasekhar, “Adversarially learned anomaly detection,” in _2018 IEEE International Conference on Data Mining (ICDM)_. Los Alamitos, CA, USA: IEEE Computer Society, nov 2018, pp. 727–736.
  * [9] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Muller, “A Unifying Review of Deep and Shallow Anomaly Detection,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 756–795, May 2021.
  * [10] B. Schölkopf, J. Platt, and T. Hofmann, _In-Network PCA and Anomaly Detection_ , 2007, pp. 617–624.
  * [11] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein _et al._ , “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Foundations and Trends® in Machine learning_ , vol. 3, no. 1, pp. 1–122, 2011\.
  * [12] T.-A. Nguyen, J. He, L. T. Le, W. Bao, and N. H. Tran, “Federated pca on grassmann manifold for anomaly detection in iot networks,” in _IEEE INFOCOM 2023 - IEEE Conference on Computer Communications_ , 2023, pp. 1–10.
  * [13] N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in _2015 Military Communications and Information Systems Conference (MilCIS)_ , 2015, pp. 1–6.
  * [14] T. M. Booij, I. Chiscop, E. Meeuwissen, N. Moustafa, and F. T. H. d. Hartog, “Ton_iot: The role of heterogeneity and the need for standardization of features and attack types in iot network intrusion data sets,” _IEEE Internet of Things Journal_ , vol. 9, no. 1, pp. 485–496, 2022.
  * [15] J. Konecný, H. B. McMahan, and D. Ramage, “Federated optimization: Distributed optimization beyond the datacenter,” _ArXiv_ , vol. abs/1511.03575, 2015.
  * [16] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282.
  * [17] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, and H. V. Poor, “Federated learning for internet of things: A comprehensive survey,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 3, pp. 1622–1658, 2021.
  * [18] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated Learning: Strategies for Improving Communication Efficiency,” _arXiv:1610.05492 [cs]_ , Oct. 2017.
  * [19] A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan, “Distributed mean estimation with limited communication,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 3329–3337.
  * [20] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,” in _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, S. Chiappa and R. Calandra, Eds., vol. 108. PMLR, 26–28 Aug 2020, pp. 2021–2031.
  * [21] F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-i.i.d. data,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 31, no. 9, pp. 3400–3413, 2020.
  * [22] Y. Wang, L. Lin, and J. Chen, “Communication-efficient adaptive federated learning,” in _Proceedings of the 39th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 17–23 Jul 2022, pp. 22 802–22 838.
  * [23] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated Learning with Non-IID Data,” _arXiv:1806.00582 [cs, stat]_ , Jun. 2018.
  * [24] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the Convergence of FedAvg on Non-IID Data,” _arXiv:1907.02189_ , Jun. 2020.
  * [25] J. Zhang, G. Zhu, R. W. Heath Jr, and K. Huang, “Grassmannian learning: Embedding geometry awareness in shallow and deep learning,” _arXiv:1808.02229_ , 2018.
  * [26] Z. Huang, J. Wu, and L. Van Gool, “Building deep networks on grassmann manifolds,” in _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence_ , 2018.
  * [27] B. Wang, Y. Hu, J. Gao, Y. Sun, H. Chen, M. Ali, and B. Yin, “Locality preserving projections for grassmann manifold,” in _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)_ , Aug. 2017, pp. 2893–2900.
  * [28] Z. Huang, R. Wang, S. Shan, and X. Chen, “Projection metric learning on grassmann manifold with application to video based face recognition,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015, pp. 140–149.
  * [29] N. Neshenko, E. Bou-Harb, J. Crichigno, G. Kaddoum, and N. Ghani, “Demystifying iot security: An exhaustive survey on iot vulnerabilities and a first empirical look on internet-scale iot exploitations,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 3, 2019.
  * [30] Y. Meidan, M. Bohadana, Y. Mathov, Y. Mirsky, A. Shabtai, D. Breitenbacher, and Y. Elovici, “N-baiot—network-based detection of iot botnet attacks using deep autoencoders,” _IEEE Pervasive Computing_ , vol. 17, no. 3, pp. 12–22, 2018.
  * [31] E. Anthi, L. Williams, M. Słowińska, G. Theodorakopoulos, and P. Burnap, “A supervised intrusion detection system for smart home iot devices,” _IEEE Internet of Things Journal_ , vol. 6, no. 5, 2019.
  * [32] H. H. Pajouh, R. Javidan, R. Khayami, A. Dehghantanha, and K.-K. R. Choo, “A two-layer dimension reduction and two-tier classification model for anomaly-based intrusion detection in iot backbone networks,” _IEEE Transactions on Emerging Topics in Computing_ , vol. 7, no. 2, 2019.
  * [33] M. Zolanvari, M. A. Teixeira, L. Gupta, K. M. Khan, and R. Jain, “Machine learning-based network vulnerability analysis of industrial internet of things,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6822–6834, 2019.
  * [34] N. Wang, Y. Chen, Y. Hu, W. Lou, and Y. T. Hou, “Manda: On adversarial example detection for network intrusion detection system,” in _IEEE INFOCOM 2021 - IEEE Conference on Computer Communications_ , 2021, pp. 1–10.
  * [35] H. Qiu, T. Dong, T. Zhang, J. Lu, G. Memmi, and M. Qiu, “Adversarial attacks against network intrusion detection in iot systems,” _IEEE Internet of Things Journal_ , vol. 8, no. 13, pp. 10 327–10 335, 2021.
  * [36] T. D. Nguyen, S. Marchal, M. Miettinen, H. Fereidooni, N. Asokan, and A.-R. Sadeghi, “DÏot: A federated self-learning anomaly detection system for iot,” in _IEEE International Conference on Distributed Computing Systems (ICDCS)_ , 2019, pp. 756–767.
  * [37] N. Wang, Y. Chen, Y. Hu, W. Lou, and Y. T. Hou, “Feco: Boosting intrusion detection capability in iot networks via contrastive learning,” in _IEEE INFOCOM 2022 - IEEE Conference on Computer Communications_ , 2022, pp. 1409–1418.
  * [38] V. Mothukuri, P. Khare, R. M. Parizi, and et al., “Federated-learning-based anomaly detection for iot security attacks,” _IEEE Internet of Things Journal_ , vol. 9, no. 4, pp. 2545–2554, 2021.
  * [39] I. Jolliffe, _Principal Component Analysis_. John Wiley & Sons, Ltd, 2014. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06472
  * [40] A. Lakhina, M. Crovella, and C. Diot, “Diagnosing network-wide traffic anomalies,” _ACM SIGCOMM computer communication review_ , vol. 34, no. 4, pp. 219–230, 2004.
  * [41] D. Brauckhoff, K. Salamatian, and M. May, “Applying pca for traffic anomaly detection: Problems and solutions,” in _IEEE INFOCOM 2009_ , 2009, pp. 2866–2870.
  * [42] J. Chen, S. Yin, S. Cai, L. Zhao, and S. Wang, “L-kpca: an efficient feature extraction method for network intrusion detection,” in _17th International Conference on Mobility, Sensing and Networking (MSN)_ , 2021, pp. 683–684.
  * [43] S. X. Wu, H.-T. Wai, L. Li, and A. Scaglione, “A review of distributed algorithms for principal component analysis,” _Proceedings of the IEEE_ , vol. 106, no. 8, pp. 1321–1340, 2018.
  * [44] J. Ge, Z. Wang, M. Wang, and H. Liu, “Minimax-optimal privacy-preserving sparse pca in distributed systems,” in _Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics_ , vol. 84. PMLR, 09–11 Apr 2018, pp. 1589–1598.
  * [45] A. Grammenos, R. Mendoza Smith, J. Crowcroft, and C. Mascolo, “Federated principal component analysis,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6453–6464, 2020.
  * [46] J. Giesen and S. Laue, “Distributed convex optimization with many convex constraints,” _arXiv preprint arXiv:1610.02967_ , 2016.
  * [47] M. Hong, Z.-Q. Luo, and M. Razaviyayn, “Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 3836–3840.
  * [48] T.-H. Chang, W.-C. Liao, M. Hong, and X. Wang, “Asynchronous distributed admm for large-scale optimization—part ii: Linear convergence analysis and numerical performance,” _IEEE Transactions on Signal Processing_ , vol. 64, no. 12, pp. 3131–3144, 2016.
  * [49] M. Hong, Z.-Q. Luo, and M. Razaviyayn, “Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems,” _SIAM Journal on Optimization_ , vol. 26, no. 1, pp. 337–364, 2016.
  * [50] X. Zhang, M. M. Khalili, and M. Liu, “Improving the privacy and accuracy of ADMM-based distributed algorithms,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 5796–5805.
  * [51] A. Elgabli, J. Park, S. Ahmed, and M. Bennis, “L-fgadmm: Layer-wise federated group admm for communication efficient decentralized deep learning,” _2020 IEEE Wireless Communications and Networking Conference (WCNC)_ , pp. 1–6, 2019.
  * [52] S. Zhou and G. Y. Li, “Communication-efficient admm-based federated learning,” _CoRR_ , vol. abs/2110.15318, 2021.
  * [53] S. Yue, J. Ren, J. Xin, S. Lin, and J. Zhang, “Inexact-admm based federated meta-learning for fast and continual edge learning,” in _Proceedings of the Twenty-Second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing_ , ser. MobiHoc ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 91–100.
  * [54] I. T. Jolliffe, _Principal component analysis for special types of data_. Springer, 2002.
  * [55] M. Udell, C. Horn, R. Zadeh, and S. Boyd, “Generalized low rank models,” _Foundations and Trends® in Machine Learning_ , vol. 9, no. 1, pp. 1–118, 2016.
  * [56] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Foundations and Trends® in Machine Learning_ , vol. 3, no. 1, pp. 1–122, 2011.
  * [57] L. Lin, B. Saparbayeva, M. M. Zhang, and D. B. Dunson, “Accelerated algorithms for convex and non-convex optimization on manifolds,” _arXiv preprint arXiv:2010.08908_ , 2020.
  * [58] S. Bubeck _et al._ , “Convex optimization: Algorithms and complexity,” _Foundations and Trends® in Machine Learning_ , vol. 8, no. 3-4, pp. 231–357, 2015.
  * [59] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in _NIPS 2017 Workshop on Autodiff_ , 2017. [Online]. Available: https://openreview.net/forum?id=BJJsrmfCZ
  * [60] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _Journal of machine learning research_ , vol. 12, no. Oct, pp. 2825–2830, 2011.

##  VII Biography Section

|  Tung-Anh Nguyen received his B.E. degree in Electronics and
Telecommunications from Hanoi University of Science and Technology (HUST),
Vietnam, in 2016. He received his MSC degree in Computer and Information
Science from Korea University, Korea, in 2019. Currently, he is pursuing a PhD
degree in Computer Science at DUAL group, Faculty of Engineering, the
University of Sydney (USyd), NSW, Australia. His research interests include
Distributed Machine Learning, Federated Learning, Blockchain and Optimization
for Distributed Systems.
---|---
|  Long Tan Le received B.E. degree with honours in Computer Engineering from
Ho Chi Minh City University of Technology (HCMUT) in 2018. He was a researcher
at the Faculty of Computer Science and Engineering, HCMUT from 2018 to 2021.
Currently, he is pursuing a Ph.D degree in Computer Science at the Faculty of
Engineering, the University of Sydney (USyd). His research interests include
Machine Learning, Federated Learning, and their applications for next-
generation networking.
---|---
|  Tuan Dung Nguyen received his B.S. from the University of Melbourne and
M.Phil. from the Australian National University, both in computer science. He
is currently pursuing a Ph.D. in computer and information science at the
Uniersity of Pennsylvania, where he is a part of the Computational Social
Science Lab. His research interests span machine learning, convex
optimization, distributed computing and social network analysis.
---|---
|  Wei Bao (Member, IEEE) received the BE degree in communications engineering
from the Beijing University of Posts and Telecommunications, Beijing, China,
in 2009, the MASc degree in electrical and computer engineering from the
University of British Columbia, Vancouver, Canada, in 2011, and the PhD degree
in electrical and computer engineering from the University of Toronto,
Toronto, Canada, in 2016. He is currently a senior lecturer with the School of
Computer Science, University of Sydney, Sydney, Australia. His research covers
the area of network science, with particular emphasis on Internet of things,
mobile computing, edge computing, and distributed learning. He received the
Best Paper Awards in ACM International Conference on Modeling, Analysis and
Simulation of Wireless and Mobile Systems (MSWiM) in 2013 and 2019 and IEEE
International Symposium on Network Computing and Applications (NCA) in 2016.
---|---
|  Suranga Seneviratne (Senior Member, IEEE) received the bachelor’s degree
from University of Moratuwa, Sri Lanka, in 2005, and the Ph.D. degree from
University of New South Wales, Australia, in 2015. He is a Lecturer of
Security with the School of Computer Science, The University of Sydney. His
current research interests include privacy and security in mobile systems, AI
applications in security, and behavior biometrics. Before moving into
research, he worked nearly six years in the telecommunications industry in
core network planning and operations
---|---
|  Choong Seon Hong (S’95-M’97-SM’11) received the B.S. and M.S. degrees in
electronic engineering from Kyung Hee University, Seoul, South Korea, in 1983
and 1985, respectively, and the Ph.D. degree from Keio University, Tokyo,
Japan, in 1997. In 1988, he joined KT, Gyeonggi-do, South Korea, where he was
involved in broadband networks as a member of the Technical Staff. Since 1993,
he has been with Keio University. He was with the Telecommunications Network
Laboratory, KT, as a Senior Member of Technical Staff and as the Director of
the Networking Research Team until 1999. Since 1999, he has been a Professor
with the Department of Computer Science and Engineering, Kyung Hee University.
His research interests include future Internet, intelligent edge computing,
network management, and network security. Dr. Hong is a member of ACM, IEICE,
IPSJ, KIISE, KICS, KIPS, and OSIA. He was an Associate Editor of the IEEE TNSM
and the IEEE JSAC.
---|---
|  Nguyen H. Tran (S’10-M’11-SM’18) received BS and PhD degrees (with best PhD
thesis award), from HCMC University of Technology and Kyung Hee University, in
electrical and computer engineering, in 2005 and 2011, respectively. Dr Tran
is an Associate Professor at the School of Computer Science, The University of
Sydney. He was an Assistant Professor with Department of Computer Science and
Engineering, Kyung Hee University, from 2012 to 2017. His research group has
special interests in Distributed compUting, optimizAtion, and machine Learning
(DUAL group). He received several best paper awards, including IEEE ICC 2016
and ACM MSWiM 2019. He receives the Korea NRF Funding for Basic Science and
Research 2016-2023, ARC Discovery Project 2020-2023, and SOAR award 2022-2023.
He has been on the Editorial board of several journals such as IEEE
Transactions on Green Communications and Networking (2016-2020), IEEE Journal
of Selected Areas in Communications 2020, and IEEE Transactions on Machine
Learning in Communications Networking (2022-).
---|---