# Distributed Estimation by Two Agents with Different Feature Spaces

Aneesh Raghavan and Karl Henrik Johansson *Research supported by the Swedish
Research Council (VR), Swedish Foundation for Strategic Research (SSF), and
the Knut and Alice Wallenberg Foundation. The authors are with the Division of
Decision and Control Systems, Royal Institute of Technology, KTH, Stockholm.
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We consider the problem of estimation of a function by a system consisting of
two agents and a fusion center. The two agents collect data comprising of
samples of an independent variable and the corresponding value of a dependent
variable. The objective of the system is to collaboratively estimate the
function without any exchange of data among the members of the system. To this
end, we propose the following framework. The agents are given a set of
features using which they construct suitable function spaces to formulate and
solve the estimation problems locally. The estimated functions are uploaded to
a fusion space where an optimization problem is solved to fuse the estimates
(also known as meta-learning) to obtain the system estimate of the mapping.
The fused function is then downloaded by the agents to gather knowledge about
the other agents estimate of the function. With respect to the framework, we
present the following: a systematic construction of fusion space given the
features of the agents; the derivation of an uploading operator for the agents
to upload their estimated functions to a fusion space; the derivation of a
downloading operator for the fused function to be downloaded. Through an
example on least squares regression, we illustrate the distributed estimation
architecture that has been developed.

##  I Introduction

###  I-A Motivation

In statistical inference problems, observations about a given phenomenon can
be obtained using different kind or multiple sensors. The observations are
often collated to obtain a single data set. Each sensor output is considered
as a modality associated with the data set. When the data set is used to
estimate a mapping from a set of independent variables to a set of dependent
variables, the resulting algorithms are known as multimodal learning
algorithms.[1] provides an overview on to multimodal data fusion focusing on
why it is needed and how it can be achieved. [2] is a survey paper on deep
multimodal learning covering many aspects including comparison with
conventional multimodal learning, fusion structures, and applications. Multi-
modal learning has found applications in many areas including human activity
recognition [3] [4], autonomous driving [5] [6], and, health monitoring [7].

Kernel methods have played a central role in inference problems. For classical
literature on the application of kernel methods to estimation, we refer to
[8], [9], and, the references there in. In [10], [11], and, [12],
identification of discrete time and continuous time systems using these
methods have been reviewed and studied. In more recent times, multimodal
kernel learning methods have been studied and algorithms referred to as
“Multiple Kernel Learning” [13], have been developed and applied to problems
in object recognition [14], disease detection [15], etc. Kernel methods for
deep learning has been studied in [16]. Understanding deep learning through
comparison with kernel based learning has been done in [17]. Hence, kernel
methods which have been extensively used in classical inference problems, have
evolved, and are relevant in contemporary inference problems as well.

Multimodal learning problems are usually studied through data collation, i.e.,
it is a centralized learning approach. The objective of this paper is to take
a step towards achieving multi-modal learning through a distributed approach.
One such scheme that has been studied in the literature in the context of
IoTs, etc., is vertical federated learning, [18], [19], when the number of
agents is large. However, distributed schemes with fewer number of agents and
emphasis on the learning space itself has not received much attention. We note
that in our previous work, [20], we considered a distributed regression
problem with noisy data by two agents and a fusion center with the agents
learning in the same space.

For motivation, consider the following example. There are two agents (robots)
in a given region trying to understand their environment. Each agent is
restricted to survey a part of the space that they live in. Each agent is
aware of the predominant features of the environment in the region that it
surveys. By collecting data, given the features, the agent is able to estimate
certain aspects of its environment, though that may not be the complete
picture. By fusing the knowledge that they have learnt they could arrive at a
complete knowledge of their environment. Given the fused knowledge, they
should be able to interpret that in their own knowledge space, i.e., they
should be able to express it using their own feature vectors to reflect upon
the other agents perception of the environment. It could enhance their
perception of the environment.

###  I-B Problem Considered

Though we do not formally define data, information, knowledge in the context
of inference problems, we differentiate between them as following. A set of
measurable outcomes associated with an observable phenomenon or an experiment
is referred to as data. Structured data, that is, data which could used to
infer models or hidden patterns is referred to as information. The inferred
models or patterns are referred to as knowledge. In the context of the
experiment, the set of all possible models or patterns is referred to as the
knowledge space (KS). As information is received sequentially, the knowledge
about the observed phenomenon evolves in the KS. In the simplest setting, we
can consider the estimation of a mapping from an independent variable (input)
to a dependent variable (output). Information would correspond to pairs of
input, output measurements. Knowledge corresponds to the function from the
input to the output, while the function space where the inference problem is
studied is interpreted as a knowledge space. Other examples of knowledge
spaces include set of probability measures on a Borel $\sigma$ algebra,
probability measures on a orthoposet [21].

The problem considered is as follows. We consider two agents and a fusion
center. Each agent receives information comprising of samples of an
independent variable and corresponding values of the dependent variable. The
underlying phenomenon generating information for both the agents is the same.
Given the information, the objective of the agents and the fusion center is to
collaboratively estimate the mapping from the independent variable to the
dependent variable without exchanging any information between the agents or
the agents and the fusion center. Along with the samples, each agent is
provided a set of features predominant in the information received by them.

###  I-C Contributions

We propose the following architecture, refer Figure 1: Given local
information, each agent estimates the mapping from the input to the output by
solving a least squares regression problem in its local KS. The functions
estimated by both agents are uploaded to the fusion center. At the fusion
center, a fusion problem as an optimization problem is formulated and solved
to fuse the functions estimated by the agents. The fused function is
considered as the function estimated by the system. The fused function is
downloaded on to the local KSs by the agents and is considered as the final
estimates of the agents.

Figure 1: Schematic for Distributed Estimation Architecture

With respect to the above frame work we prove the following: Given the
features, we present the construction of the individual KSs of the agents as a
reproducing kernel Hilbert space (RKHS). The fusion space is defined as the
set of functions obtained through linear combination of functions in the the
local KSs. We prove that the fusion space is also an RKHS whose kernel is the
sum of the kernels of the agents. As a corollary, we obtain that the uploading
operator used by the agents to upload functions from the local KS to the
fusion space is a linear bounded operator. All functions in the fusion center
might not be decipherable in the local KSs, a download operator is needed to
suitably transform the fused function which can be interpreted in the local
KS. We present a detailed construction of such an download operator and prove
that it is linear and bounded. To illustrate the distributed estimation
scheme, we present a numerical example.

###  I-D Novelty and Interpretation of Proposed Solution

The novelty of the proposed solution is that distributed learning can be
achieved with different agents learning in different function spaces. The
framework is applicable to other learning problems including classification
and density estimation. The function spaces can be chosen based on the
predominant features in the data collected by them. We focus on the
construction of the learning spaces and how functions can be transferred from
one space to another and less on the estimation procedure itself, i.e., the
focus of the paper is on framework and the structure for distributed learning,
less on the learning algorithm itself. To emphasize the same, the example
presented here demonstrates how a least squares estimation problem over
polynomials and exponentials can be solved in distributed way; one agent
focusing on polynomials and other focusing on exponentials.

In kernel methods, the kernels used for learning are usually assumed to be
known. Depending on the data collected by an agent, there could be a
predominant feature in the data. This could dictate the kernel used by the
agent for estimation. By using different kernels the agents are tuning their
estimation procedure to the specific data collected by them. Further by using
different kernels they could be exhausting the set of kernels that could be
used for the given data set, if one is not able to make a clear choice
upfront. Thus, from a practical standpoint the system could benefit when the
agents chose different kernels.

There is an additional “benefit” of privacy as, the proposed framework
promotes local processing of data, thus keeping the data private to the agent.
An interpretation for distributed learning is “robust” learning. Though this
to be formally proven, different agents estimating the function using
different kernels ensures that the estimation scheme is robust to noise in
data (as same noise need not be observed by the agents) and to the kernels
chosen. If all the agents where to chose the same kernel, there could be a
potential bias in the estimation procedure. By choosing different kernels, it
is ensured that different features of the data is given importance by
different agents.

The paper is organized as follows. It is different from the order in which the
contributions were discussed. In Section II, we discuss the construction of
the individual KSs, the fusion space, the derivation of the uploading and
downloading operator. In Section III, we discuss the regression problem for
the agents, its solution, and, the fusion problem in fusion space. In Section
IV, we present a numerical example demonstrating the distributed estimation
methodology. In Section V, we summarize the key aspects of this paper and
focus on future work. Notation: we use superscript for the agent, subscript
for samples and summation indicies. We represent vectors obtained by
concatenating smaller vectors in boldface. For a function $f\in V$, $V$ vector
space, we use the notation $f$ when it is treated as a vector and the notation
$f(\cdot)$ when it is treated as a function. The projection onto a subspace
$\mathcal{M}$ of a Hilbert space $H$ is denoted by $\Pi_{\mathcal{M}}$.

##  II Construction of Knowledge Spaces

In this section, we discuss the construction of KSs for the individual agents
and the fusion space. Let $\mathcal{X}\subset\mathbb{R}^{d}$. Let
$(H,\langle\cdot,\cdot\rangle_{H})$ be a Hilbert space of functions,
$f:\mathcal{X}\to\mathbb{R}$. Let
$K:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ be a function, $n\in\mathbb{N}$,
$\\{x_{1},\ldots,x_{n}\\}\subset\mathcal{X}$, and,
$\mathbf{K}:=(K(x_{i},x_{j}))_{ij}$ be Gram (kernel) matrix of $K$ with
respect to $x_{1},\ldots,x_{n}$.

######  Definition 1.

The function $K(\cdot,\cdot)$ is said to be a positive definite kernel if the
gram matrix generated by the function is a positive definite matrix for all
$n$ and $\\{x_{1},\ldots,x_{n}\\}\subset\mathcal{X}$.

######  Definition 2.

$(H,\langle\cdot,\cdot\rangle_{H})$ is said to be a reproducing kernel Hilbert
space (RKHS) with a positive definite kernel $K$, if,

  * •

$K(\cdot,x)\in H,\;\forall x\in\mathcal{X},$

  * •

the reproducing property is satisfied

$\displaystyle f(y)=\langle f(\cdot),K(\cdot,y)\rangle_{H},f\in
H,y\in\mathcal{X}.$

###  II-A Construction of Individual Knowledge Spaces for the Agents

The set of features for agent $i$ is the set of functions,
$\\{\varphi^{i}_{j}(\cdot)\\}_{j\in\mathcal{I}^{i}}$, where
$\varphi^{i}_{j}:\mathcal{X}\to\mathbb{R}$ and $|\mathcal{I}^{i}|<\infty$. The
knowledge space for agent $i$ is the finite dimensional vector space, $H^{i}$,
defined as:

$\displaystyle
H^{i}=\\{f:f(\cdot)=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(\cdot),\\{\alpha_{j}\\}_{j\in\mathcal{I}^{i}}\subset\mathbb{R}\\}.$

The null vector for the space $H^{i}$ is the function $\theta^{i}(\cdot)$
defined as $\theta^{i}(x)=0,\forall x\in\mathcal{X}$. For agent $i$, the
features are assumed to be linearly independent, i.e.,
$\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}=\theta^{i}$ if and only
if $\alpha_{j}=0,\;\forall j$. The function space, $H^{i}$, is equipped with
the inner product, $\langle\cdot,\cdot\rangle_{H^{i}}:H^{i}\times
H^{i}\to\mathbb{R}$, defined as follows. For,
$f(\cdot)=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(\cdot),g(\cdot)=\sum_{j\in\mathcal{I}^{i}}\beta_{j}\varphi^{i}_{j}(\cdot)$,

$\displaystyle\langle
f(\cdot),g(\cdot)\rangle_{H^{i}}=\langle\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(\cdot),\sum_{j\in\mathcal{I}^{i}}\beta_{j}\varphi^{i}_{j}(\cdot)\rangle_{H^{i}}:=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\beta_{j}.$

It can be verified that the above definition of inner product on $H^{i}$
satisfies the axioms of a inner product on a vector space. The norm induced by
the inner product is
$||f||^{2}_{H^{i}}=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}^{2}$. The kernel
$K^{i}:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ is defined as,

$\displaystyle
K^{i}(x,y)=\sum_{j\in\mathcal{I}^{i}}\varphi^{i}_{j}(x)\varphi^{i}_{j}(y).$

Let $\\{x_{1},\ldots,x_{n}\\}\subset\mathcal{X}$. Let
$\mathbf{K^{i}}:=(K^{i}(x_{k},x_{l}))_{kl}$, be the Gram matrix of
$K^{i}(\cdot,\cdot)$ with respect to $\\{x_{1},\ldots,x_{n}\\}$. Then for any
$\boldsymbol{\alpha}\in\mathbb{R}^{n}$,
$\boldsymbol{\alpha}^{T}\mathbf{K^{i}}\boldsymbol{\alpha}=$

$\displaystyle\sum^{n}_{k=1}\sum^{n}_{l=1}\alpha_{k}\alpha_{l}K^{i}(x_{k},x_{l})=\sum^{n}_{k=1}\sum^{n}_{l=1}\alpha_{k}\alpha_{l}\sum_{j\in\mathcal{I}^{i}}\varphi^{i}_{j}(x_{k})\varphi^{i}_{j}(x_{l})$
$\displaystyle=\sum_{j\in\mathcal{I}^{i}}\Big{(}\sum^{n}_{k=1}\alpha_{k}\varphi^{i}_{j}(x_{k})\Big{)}\Big{(}\sum^{n}_{l=1}\alpha_{l}\varphi^{i}_{j}(x_{l})\Big{)}=||f||^{2}_{H^{i}}\geq
0,$

where
$f(\cdot)=\sum_{j\in\mathcal{I}^{i}}\Big{(}\sum^{n}_{k=1}\alpha_{k}\varphi^{i}_{j}(x_{k})\Big{)}\varphi^{i}_{j}(\cdot)$.
The Gram matrix of $K^{i}(\cdot,\cdot)$ is positive definite for any
$\\{x_{1},\ldots,x_{n}\\}\subset\mathcal{X}$, for all $n\in\mathbb{N}$. Thus,
$K^{i}(\cdot,\cdot)$ is a positive definite kernel (From Definition 1, also
refer [22]). We note that,
$K^{i}(\cdot,y)=\sum_{j\in\mathcal{I}^{i}}\varphi^{i}_{j}(y)\varphi^{i}_{j}(\cdot)\in
H^{i}$, with $\alpha^{i}_{j}=\varphi^{i}_{j}(y),\;\forall
j\in\mathcal{I}^{i}$. For $f\in H^{i}$,
$f(\cdot)=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(\cdot)$,

$\displaystyle\langle f(\cdot),K^{i}(\cdot,y)\rangle_{H^{i}}$
$\displaystyle=\langle\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(\cdot),\sum_{j\in\mathcal{I}^{i}}\varphi^{i}_{j}(y)\varphi^{i}_{j}(\cdot)\rangle_{H^{i}}$
$\displaystyle=\sum_{j\in\mathcal{I}^{i}}\alpha_{j}\varphi^{i}_{j}(y)=f(y).$

The reproducing property is satisfied by $f\in H^{i}$ with kernel
$K^{i}(\cdot,\cdot)$. From Definition 2, it follows that $H^{i}$ is a RKHS
with kernel $K^{i}(\cdot,\cdot)$.

###  II-B Construction of the Fusion Space

The motivation for the construction of a fusion space is to build a function
space where algebraic operations can be simultaneously performed on functions
living in the local KSs of the agents. Hence, at the very least it should
include functions from the local KSs. As a possible candidate, one could
consider $H^{1}\cup H^{2}$. The properties of the original KSs could be lost
as the resulting space need not be vector space or a RKHS. The space
$\big{(}H^{1}\oplus H^{2}\Big{)}/\sim$ seems to be another reasonable
candidate, however a set of equivalent classes of functions cannot be a RKHS.
From the assumption, the vectors in
$\\{\varphi^{1}_{j}(\cdot)\\}_{j\in\mathcal{I}^{1}}$ are linearly independent,
and so are the vectors in
$\\{\varphi^{2}_{j}(\cdot)\\}_{j\in\mathcal{I}^{2}}$. However, it is not
necessary that the vectors in
$\\{\varphi^{1}_{j}(\cdot)\\}_{j\in\mathcal{I}^{1}}\cup\\{\varphi^{2}_{j}(\cdot)\\}_{j\in\mathcal{I}^{2}}$
are linearly independent. Since $K^{1}$ and $K^{2}$ are positive definite
kernels, $K=K^{1}+K^{2}$ is also a positive definite kernel. This suggests
that the fusion space could be constructed as the RKHS corresponding to the
kernel $K(\cdot,\cdot)$ by considering the linear combination of all functions
in $H^{1}$ and $H^{2}$.

######  Definition 3.

The fusion space, $H$, is defined as $H=\\{f:f=f^{1}+f^{2},f^{1}\in
H^{1},f^{2}\in H^{2}\\}$.

In the following theorem, we characterize the fusion space as a RKHS after
associating a suitable inner product and find an expression for the norm
induced by the inner product.

######  Theorem II.1.

If $K^{i}(\cdot,\cdot)$ is the reproducing kernel of Hilbert space $H^{i}$,
with norm $||\cdot||_{H^{i}}$, then $K(x,y)=K^{1}(x,y)+K^{2}(x,y)$ is the
reproducing kernel of the space $H=\\{f:f=f^{1}+f^{2}|f^{i}\in H^{i}\\}$ with
the norm:

$\displaystyle||f||^{2}_{H}=\underset{\begin{subarray}{c}f^{1}+f^{2}=f,\\\
f^{i}\in
H^{i}\end{subarray}}{\min}\;\;||f^{1}||^{2}_{H^{1}}+||f^{2}||^{2}_{H^{2}}.$

###### Proof.

Let $H_{\prod}=H^{1}\times H^{2}$ denote the product space, with inner product
$\langle(f^{1},f^{2}),(g^{1},g^{2})\rangle_{H_{\prod}}=\langle
f^{1},g^{1}\rangle_{H^{1}}+\langle f^{2},g^{2}\rangle_{H^{2}}$. Let
$H=\\{f:f=f^{1}+f^{2},f^{i}\in H^{i},i=1,2\\}$. Clearly, $H$ is a vector space
whose null vector we denote by $\theta$. Let $L:H_{\Pi}\to H$ be a operator
defined as $L((f^{1},f^{2}))=f^{1}+f^{2}$. $L$ is a linear operator and its
null space, $\mathcal{N}(L)=\\{(f^{1},f^{2})\in
H_{\prod}:f^{1}+f^{2}=\theta\\}$ is a closed subspace as it is finite
dimensional. The basis vectors for $H_{\prod}$ are given by
$\\{\varphi^{1}_{j}\times\theta^{2}\\}_{j\in\mathcal{I}^{1}}\cup\\{\theta^{1}\times\varphi^{2}_{j}\\}_{j\in\mathcal{I}^{2}}$.
Any vector in $H_{\prod}$, can be expressed as
$(f^{1},f^{2})=(\sum_{j\in\mathcal{I}^{1}}\alpha^{1}_{j}\varphi^{1}_{j},\sum_{j\in\mathcal{I}^{2}}\alpha^{2}_{j}\varphi^{2}_{j})$
and thus $H_{\prod}$ is isomorphic to
$\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}}$. The null space
$\mathcal{N}(L)$ is isomorphic to the subspace,

$\displaystyle\mathcal{N}=\Big{\\{}\Big{(}\boldsymbol{\alpha^{1}},\boldsymbol{\alpha^{2}}$
$\displaystyle\Big{)}\in\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}}:\sum_{j\in\mathcal{I}^{1}}\alpha^{1}_{j}\varphi^{1}_{j}+\sum_{j\in\mathcal{I}^{2}}\alpha^{2}_{j}\varphi^{2}_{j}=\theta,$
$\displaystyle\boldsymbol{\alpha^{1}}=\Big{(}\alpha^{1}_{1},\ldots\alpha^{1}_{\mathcal{I}^{1}}\Big{)},\boldsymbol{\alpha^{2}}=\Big{(}\alpha^{2}_{1},\ldots\alpha^{2}_{\mathcal{I}^{2}}\Big{)}\Big{\\}}.$

Since $\mathcal{N}(L)$ is a closed subspace, there exists a unique closed
subspace $\mathcal{M}$ such that $H_{\prod}=\mathcal{M}\oplus\mathcal{N}(L)$.
The mapping $L_{\mathcal{M}}=L\circ\Pi_{\mathcal{M}}$ (operator $L$ restricted
to subspace $\mathcal{M}$) is bijection from $\mathcal{M}$ to $H$. For any
function $f\in H$, let $L^{-1}_{\mathcal{M}}(f)=(L^{1}(f),L^{2}(f))$, i.e.,
$(L^{1}(f),L^{2}(f))$ is the unique tuple of functions in $\mathcal{M}$ such
that $L^{1}(f)\in H^{1},\;L^{2}(f)\in H^{2}$ and $L^{1}(f)+L^{2}(f)=f$. We now
define the inner product on $H$ as follows:

$\displaystyle\langle f,g\rangle_{H}=\langle
L^{1}(f),L^{1}(g)\rangle_{H^{1}}+\langle L^{2}(f),L^{2}(g)\rangle_{H^{2}}.$

Since $K^{i}(\cdot,y)\in H^{i}$, it follows that
$K(\cdot,y)=K^{1}(\cdot,y)+K^{2}(\cdot,y)\in H$. We claim that
$L^{i}(K(\cdot,y))=K^{i}(\cdot,y)$. Indeed, since
$K(\cdot,y)=\sum_{j\in\mathcal{I}^{1}}\varphi^{1}_{j}(y)\varphi^{1}_{j}(\cdot)+\sum_{j\in\mathcal{I}^{2}}\varphi^{2}_{j}(y)\varphi^{2}_{j}(\cdot)$,
to prove the claim it suffices to prove that the vector
$\Big{(}\varphi^{1}_{1}(y),\ldots,\varphi^{1}_{\mathcal{I}^{1}}(y),\varphi^{2}_{1}(y),\ldots,\varphi^{2}_{\mathcal{I}^{2}}(y)\Big{)}$
is orthogonal to $\mathcal{N}$ for all $y\in\mathcal{X}$. That is,
$\forall\;\Big{(}\boldsymbol{\alpha^{1}},\boldsymbol{\alpha^{2}}\Big{)}\in\mathcal{N}$,

$\displaystyle\sum_{j\in\mathcal{I}^{1}}\alpha^{1}_{j}\varphi^{1}_{j}(y)+\sum_{j\in\mathcal{I}^{2}}\alpha^{2}_{j}\varphi^{2}_{j}(y)=0,$

which is true from the definition of $\mathcal{N}$. Thus,

$\displaystyle\langle f(\cdot),K(\cdot,y)\rangle_{H}$ $\displaystyle=\langle
L^{1}(f)(\cdot),L^{1}(K(\cdot,y))\rangle_{H^{1}}+\langle L^{2}(f)(\cdot),$
$\displaystyle L^{2}(K(\cdot,y))\rangle_{H^{2}}$ $\displaystyle=\langle
L^{1}(f)(\cdot),K^{1}(\cdot,y))\rangle_{H^{1}}+\langle L^{2}(f)(\cdot),$
$\displaystyle K^{2}(\cdot,y))\rangle_{H^{2}}$
$\displaystyle=L^{1}(f)(y)+L^{2}(f)(y)=f(y),$

satisfying the reproducing property. From Definition 2, it follows that
$(H,\langle\cdot,\cdot\rangle_{H})$ is a RKHS with kernel $K(\cdot,\cdot)$.
For $f\in H$, let $(f^{1},f^{2})\in H_{\prod}$ be such that
$f=f^{1}+f^{2}=L^{1}(f)+L^{2}(f)$. We note that
$(L^{1}(f),L^{2}(f))=\Pi_{\mathcal{M}}((f^{1},f^{2}))$. Computing norm of $f$,

$\displaystyle||f||^{2}_{H}=\langle
f,f\rangle_{H}=||L^{1}(f)||^{2}_{H^{1}}+||L^{2}(f)||^{2}_{H^{2}}$
$\displaystyle||f^{1}||^{2}_{H^{1}}+||f^{2}||^{2}_{H^{2}}=||(f^{1},f^{2})||^{2}_{H_{\prod}}=\underbrace{||\Pi_{\mathcal{M}}((f^{1},f^{2}))}+$
$\displaystyle\Pi_{\mathcal{N}(L)}((f^{1},f^{2}))||^{2}_{H_{\prod}}.\hskip
42.67912pt=||(L^{1}(f),L^{2}(f))||^{2}_{H_{\prod}}$

Thus, for $f^{1},f^{2}$ such that $f=f^{1}+f^{2}$, the minimum of
$||f^{1}||^{2}_{H^{1}}+||f^{2}||^{2}_{H^{2}}$ is achieved when
$\Pi_{\mathcal{N}(L)}((f^{1},f^{2}))||^{2}_{H_{\prod}}=\theta$, i.e.,
$f^{i}=L^{i}(f)$, and, is equal to $||f||^{2}_{H}$. ∎

Given $f\in H^{1}$, we let $f^{1}=f$ and $f^{2}=0$. From the Theorem II.1, we
conclude that $||f||_{H}\leq||f||_{H^{1}}$. Similarly
$||f||_{H}\leq||f||_{H^{2}},f\in H^{2}$.

######  Corollary II.2.

The uploading operator from agent $i$’s knowledge space, $H^{i}$, to the
fusion space $H$, $\hat{L}^{i}:H^{i}\to H$, is $\hat{L}(f)=f$.
$\hat{L}^{i}(\cdot)$, is linear and is bounded,
$||\hat{L}^{i}||=\sup\\{||f||_{H}:f\in H^{i},||f||_{H^{i}}=1\\}\leq 1$.

###  II-C Retrieval of Individual Knowledge Spaces from Fusion Space

Given a function in the fusion space, it is not necessary that it belongs to
both the KSs. Hence, it is mandatory to transform it to a form where it can be
expressed using the features of the local KS. The objective of this subsection
is to find an operation which would transform the function onto the individual
KSs. Given a function in the fusion space, one possibility is to express the
function as linear combination of
$\\{\varphi^{1}_{j}(\cdot)\\}_{j\in\mathcal{I}^{1}}\cup\\{\varphi^{2}_{j}(\cdot)\\}_{j\in\mathcal{I}^{2}}$
and then simply set the coefficients of the functions which do not belong to a
local KS to zero. The resulting function need not belong to the local KS,
i.e., need not satisfy the reproducing property of the KS. To ensure the same,
in the following derivation every function in either of the local KS can be
retrieved from a function in the fusion space. The operation corresponding to
one of the KS when applied to a function in the fusion space would only result
in a function in the KS.

######  Lemma II.3.

For every $f\in H$, there exits $\\{y_{k,f}\\}^{n}_{k=1}$ and
$\\{\beta_{k,f}\\}^{n}_{k=1}$ such that,

$\displaystyle
f(\cdot)=\sum_{i=1,2}\sum_{j\in\mathcal{I}^{i}}\sum^{n}_{k=1}\beta_{k,f}\varphi^{i}_{j}(y_{k,f})\varphi^{i}_{j}(\cdot)=\sum^{n}_{k=1}\beta_{k,f}K(\cdot,y_{k,f}).$

###### Proof.

Given $f\in H$, we characterize $L^{1}(f)$ and $L^{2}(f)$. Let,

$\displaystyle\Phi(y)=[\varphi^{1}_{1}(y),\ldots,\varphi^{1}_{\mathcal{I}^{1}}(y),\varphi^{2}_{1}(y),\ldots,\varphi^{2}_{\mathcal{I}^{1}}(y)]\in\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}},\;$

$y\in\mathcal{X}$. Let $\hat{\mathcal{M}}$ be the span of
$\\{\Phi(y)\\}_{y\in\mathcal{X}}$, i.e.,
$\hat{\mathcal{M}}=\\{\boldsymbol{\gamma}\in\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}}:\boldsymbol{\gamma}=\sum^{n}_{k=1}\beta_{k}\Phi(y_{k}),n\in\mathbb{N},\\{\beta_{k}\\}^{n}_{k=1}\subset\mathbb{R}\\}$,
and $\bar{\mathcal{M}}$ be such that
$\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}}=\bar{\mathcal{M}}\oplus\mathcal{N}$.
We claim that $\bar{\mathcal{M}}=\hat{\mathcal{M}}$. Indeed, from the
definition of $\mathcal{N}$, it follows that for every
$\boldsymbol{\alpha}\in\mathcal{N}$,
$\boldsymbol{\alpha}^{T}\Phi(y)=0,\;\forall y\in\mathcal{X}$. Thus, if
$\boldsymbol{\gamma}\in\hat{\mathcal{M}}$, then
$\boldsymbol{\gamma}^{T}\boldsymbol{\alpha}=0,\;\forall\boldsymbol{\alpha}\in\mathcal{N}$,
i.e., $\boldsymbol{\gamma}\in\bar{\mathcal{M}}$. Suppose,
$\boldsymbol{\bar{\gamma}}\in\bar{\mathcal{M}}$ is equal to
$\boldsymbol{\gamma}+\boldsymbol{v}$, where
$\boldsymbol{\gamma}\in\hat{\mathcal{M}}$ and
$\boldsymbol{v}\in\mathbb{R}^{\mathcal{I}^{1}+\mathcal{I}^{2}},\boldsymbol{v}\neq\theta,\boldsymbol{v}\notin\hat{\mathcal{M}}$.
Then,
$\boldsymbol{\bar{\gamma}}^{T}\boldsymbol{\alpha}=0,\;\boldsymbol{\alpha}\in\mathcal{N}$
which implies
$\boldsymbol{v}^{T}\boldsymbol{\alpha}=0,\;\boldsymbol{\alpha}\in\mathcal{N}$.
This is a contradiction as $\boldsymbol{\alpha}\in\mathcal{N}$ if and only if
$\boldsymbol{\alpha}^{T}\Phi(y)=0,\;\forall y\in\mathcal{X}$. The condition
$\boldsymbol{v}^{T}\boldsymbol{\alpha}=0,\;\boldsymbol{\alpha}\in\mathcal{N}$
introduces an additional constraint which is not satisfied, except when
$\boldsymbol{v}=\theta$. Thus,
$\boldsymbol{\bar{\gamma}}=\boldsymbol{\gamma}\in\hat{\mathcal{M}}$. Since
$\hat{\mathcal{M}}=\bar{\mathcal{M}}$ is isomorphic to $\mathcal{M}$, every
vector in $\mathcal{M}$ is given by

$\displaystyle\Big{(}\sum_{j\in\mathcal{I}^{1}}\gamma^{1}_{j}\big{(}\varphi^{1}_{j}\times\theta^{2}\big{)}+\sum_{j\in\mathcal{I}^{2}}\gamma^{2}_{j}\big{(}\theta^{1}\times\varphi^{2}_{j}\big{)}\Big{)}=\Big{(}\sum_{j\in\mathcal{I}^{1}}\gamma^{1}_{j}\varphi^{1}_{j},$
$\displaystyle\sum_{j\in\mathcal{I}^{2}}\gamma^{2}_{j}\varphi^{2}_{j}\Big{)}=\Big{(}\sum_{j\in\mathcal{I}^{1}}\sum^{n}_{k=1}\beta_{k}\varphi^{1}_{j}(y_{k})\varphi^{1}_{j},\sum_{j\in\mathcal{I}^{2}}\sum^{n}_{k=1}\beta_{k}\varphi^{2}_{j}(y_{k})\varphi^{2}_{j}\Big{)},$

where
$\boldsymbol{\gamma}=[\gamma^{1}_{1},\ldots,\gamma^{1}_{\mathcal{I}^{1}},\gamma^{2}_{1},\ldots,\gamma^{2}_{\mathcal{I}^{2}}]\in\bar{\mathcal{M}}$
and $\gamma^{i}_{j}=\sum^{n}_{k=1}\beta_{k}\varphi^{i}_{j}(y_{k})$. Thus, for
any $f\in H$,
$(L^{1}(f),L^{2}(f))=\Big{(}\sum^{n}_{k=1}\beta_{k,f}K^{1}(\cdot,y_{k,f}),\sum^{n}_{k=1}\beta_{k,f}K^{2}(\cdot,y_{k,f})\Big{)}$.
This implies that,
$f=L^{1}(f)+L^{2}(f)=\sum^{n}_{k=1}\beta_{k,f}K(\cdot,y_{k,f})$. ∎

######  Lemma II.4.

Given the RKHS, $(H,\langle\cdot,\cdot\rangle_{H})$, with kernel
$K(\cdot,\cdot)$ and the kernels $K^{i}(\cdot,\cdot),\;i=1,2$, such that
$K(x,y)=K^{1}(x,y)+K^{2}(x,y)$, we define operators, $\bar{L}^{i}:H\to H$, as

$\displaystyle\bar{L}^{i}(f)(x)=\langle
f(\cdot),K^{i}(\cdot,x)\rangle_{H},\text{ for},i=1,2.$

Then, $\bar{L}^{i}$ is linear, symmetric, positive and bounded,
$||\bar{L}^{i}||\leq 1$.

###### Proof.

From the linearity of the inner product with respect to the first argument, it
follows that $\bar{L}^{i}$ is linear. From equation set (1), we note that
$\bar{L}^{i}$ is positive, i.e., $\langle\bar{L}^{i}(f),f\rangle_{H}\geq
0,\;\forall f\in H$. From equation set(2), we note that $\bar{L}^{i}$ is
symmetric, i.e., $\langle\bar{L}^{i}(f),g\rangle_{H}=\langle
f,\bar{L}^{i}(g)\rangle_{H},\;\forall f,g\in H$. Since $\bar{L}^{1}$ and
$\bar{L}^{2}$ are symmetric, positive, and their sum is the identity operator
($\bar{L}^{1}+\bar{L}^{2}=\mathbb{I}$), $0\leq||\bar{L}^{i}||\leq 1$.

$\displaystyle f=$
$\displaystyle\sum^{n}_{j=1}\beta_{j}K(\cdot,y_{j}),\bar{L}^{i}(f)(x)=\langle\sum^{n}_{j=1}\beta_{j}K(\cdot,y_{j}),K^{i}(\cdot,x)\rangle_{H}=\sum^{n}_{j=1}\beta_{j}\langle
K^{i}(\cdot,x),K(\cdot,y_{j})\rangle_{H}=\sum^{n}_{j=1}\beta_{j}K^{i}(y_{j},x)$
$\displaystyle=$
$\displaystyle\sum^{n}_{j=1}\beta_{j}K^{i}(x,y_{j}).\;\langle\bar{L}^{i}(f),f\rangle_{H}=\langle\sum^{n}_{j=1}\beta_{j}K^{i}(\cdot,y_{j}),\sum^{n}_{k=1}\beta_{k}K(\cdot,y_{k})\rangle_{H}=\sum^{n}_{j=1}\sum^{n}_{k=1}\beta_{j}\beta_{k}K^{i}(y_{k},y_{j})=||\bar{L}^{i}(f)||^{2}_{H^{i}}\geq
0.$ (1) $\displaystyle g=$
$\displaystyle\sum^{m}_{l=1}\delta_{l}K(\cdot,y_{l}),\langle\bar{L}^{i}(f),g\rangle_{H}=\langle\sum^{n}_{j=1}\beta_{j}K^{i}(\cdot,y_{j}),\sum^{m}_{l=1}\delta_{l}K(\cdot,y_{l})\rangle_{H}=\sum^{n}_{j=1}\sum^{m}_{l=1}\beta_{j}\delta_{l}K^{i}(y_{l},y_{j}).\langle
f,\bar{L}^{i}(g)\rangle_{H}=\langle\sum^{n}_{j=1}\beta_{j}K(\cdot,y_{j}),$
$\displaystyle\sum^{m}_{l=1}$
$\displaystyle\delta_{l}K^{i}(\cdot,y_{l})\rangle_{H}=\sum^{n}_{j=1}\sum^{m}_{l=1}\beta_{j}\delta_{l}\langle
K^{i}(\cdot,y_{l}),K(\cdot,y_{j})\rangle_{H}=\sum^{n}_{j=1}\sum^{m}_{l=1}\beta_{j}\delta_{l}K^{i}(y_{j},y_{l})=\sum^{n}_{j=1}\sum^{m}_{l=1}\beta_{j}\delta_{l}K^{i}(y_{l},y_{j})=\langle\bar{L}^{i}(f),g\rangle_{H}.$
(2)

∎

######  Remark 1.

Invoking Lemma II.3 and the definition of inner product on $H$, the inner
product in the definition of operator $\bar{L}^{i}$ could be evaluated as
follows,

$\displaystyle\bar{L}^{i}(f)(y)=\langle
f(\cdot),K^{i}(\cdot,y)\rangle_{H}=\langle
L^{1}(f),L^{1}(K^{i}(\cdot,y))\rangle_{H^{1}}+$ $\displaystyle\langle
L^{2}(f),L^{2}(K^{i}(\cdot,y))\rangle_{H^{2}}=\langle\sum^{n}_{k=1}\beta_{k,f}K^{1}(\cdot,y_{k,f}),\sum^{m}_{l=1}\beta_{l,K^{i}(\cdot,y)}$
$\displaystyle
K^{1}(\cdot,y_{l,K^{i}(\cdot,y)})\rangle_{H^{1}}+\langle\sum^{n}_{k=1}\beta_{k,f}K^{2}(\cdot,y_{k,f}),\sum^{m}_{l=1}\beta_{l,K^{i}(\cdot,y)}$
$\displaystyle
K^{2}(\cdot,y_{l,K^{i}(\cdot,y)})\rangle_{H^{2}}=\sum^{n}_{k=1}\sum^{m}_{l=1}\beta_{k,f}\beta_{l,K^{i}(\cdot,y)}K^{1}(y_{l,K^{i}(\cdot,y)},$
$\displaystyle
y_{k,f})+\sum^{n}_{k=1}\sum^{m}_{l=1}\beta_{k,f}\beta_{l,K^{i}(\cdot,y)}K^{2}(y_{l,K^{i}(\cdot,y)},y_{k,f}).$

Unlike equation set (1), the above evaluation does not result in a “function”
form for $\bar{L}^{i}(f)$. This is because in the above evaluation, the
functions get mapped to their versions in the individual KSs and the inner
product is evaluated there. To obtain a function form as in equation set (1),
it is necessary to evaluate the inner product in the fusion space.

######  Theorem II.5.

Let $L:H\to H$ be a symmetric, positive, bounded operator. There exists a
unique square root of operator $L$, $\sqrt{L}:H\to H$, i.e.,
$\sqrt{L}(\sqrt{L}(f))=L(f)\forall f\in H$. $\sqrt{L}(\cdot)$ is linear,
bounded, symmetric.

###### Proof.

Since $L$ is symmetric, from the spectral theorem, it follows that (i) there
exists an orthonormal basis of $H$, $\\{\varphi_{j}\\}_{j\in\mathcal{I}}$,
which are eigenvectors of $L$ ; (ii) the eigenvalues of $L$,
$\\{\lambda_{j}\\}^{\mathcal{I}_{\lambda}}_{j=1}$, are real. Since $L$ is
positive, $\lambda_{j}\geq 0,1\leq j\leq\mathcal{I}_{\lambda}$. Let
$f=\sum^{\mathcal{I}}_{j=1}\alpha_{j}\varphi_{j},\\{\alpha_{j}\\}\subset\mathbb{R}$.
$L(f)=L(\sum^{\mathcal{I}}_{j=1}\alpha_{j}\varphi_{j})=\sum^{\mathcal{I}}_{j=1}\alpha_{j}\lambda_{j}\varphi_{j}$.
From the construction of $H$, it follows that
$\mathcal{I}\leq\mathcal{I}^{1}+\mathcal{I}^{2}$. Since some of the
eigenvalues of $L$ could repeated, $\mathcal{I}_{\lambda}\leq\mathcal{I}$. The
operator $\sqrt{L}$ is defined as
$\sqrt{L}(\varphi_{j})=\sqrt{\lambda_{j}}\varphi_{j}$, and, imposing linearity
$\sqrt{L}(f)=\sum^{\mathcal{I}}_{j=1}\alpha_{j}\sqrt{\lambda_{j}}\varphi_{j}$.

$\displaystyle\sqrt{L}(\sqrt{L}(f))=\sqrt{L}(\sum^{\mathcal{I}}_{j=1}\alpha_{j}\sqrt{\lambda_{j}}\varphi_{j})=\sum^{\mathcal{I}}_{j=1}\alpha_{j}\lambda_{j}\varphi_{j}=L(f)$
$\displaystyle\langle\sqrt{L}(f),f\rangle_{H}=\langle\sum^{\mathcal{I}}_{j=1}\alpha_{j}\sqrt{\lambda_{j}}\varphi_{j},\sum^{\mathcal{I}}_{l=1}\alpha_{l}\varphi_{l}\rangle_{H}=\sum^{\mathcal{I}}_{j=1}\alpha^{2}_{j}\sqrt{\lambda_{j}}\geq
0$
$\displaystyle\langle\sqrt{L}(f),g\rangle_{H}=\langle\sum^{\mathcal{I}}_{j=1}\alpha_{j}\sqrt{\lambda_{j}}\varphi_{j},\sum^{\mathcal{I}}_{l=1}\beta_{l}\varphi_{l}\rangle_{H}=\sum^{\mathcal{I}}_{j=1}\alpha_{j}\beta_{j}\sqrt{\lambda_{j}}$
$\displaystyle=\langle\sum^{\mathcal{I}}_{j=1}\alpha_{j}\varphi_{j},\sum^{\mathcal{I}}_{l=1}\beta_{l}\sqrt{\lambda_{l}}\varphi_{l}\rangle_{H}=\langle
f,\sqrt{L}(g)\rangle_{H}.$

Suppose $\tilde{L}$ is a positive semi-definite operator which is another
square root for $L$. To prove uniqueness, it suffices to prove that
$\tilde{L}(\phi_{j})=\sqrt{L}(\phi_{j})\forall\;j$, where
$\\{\phi_{j}\\}_{j\in\mathcal{I}}$ is a basis for $H$. By the spectral
theorem, $\tilde{L}$ posses a set of orthonormal
eigenvectors,$\\{\phi_{j}\\}_{j\in\mathcal{I}}$, which form a basis for $H$.
$L(\phi_{j})=\tilde{L}\big{(}\tilde{L}(\phi_{j})\big{)}=\tilde{L}(\bar{\lambda}_{j}\phi_{j})=\bar{\lambda}^{2}_{j}\phi_{j}$.
Thus, $\\{\phi_{j}\\}_{j\in\mathcal{I}}$ are eigenvectors for $L$. By
definition of $\sqrt{L}$, $\\{\phi_{j}\\}_{j\in\mathcal{I}}$ are eigenvectors
for $\sqrt{L}$ as well. As the eigenvectors $\\{\phi_{j}\\}_{j\in\mathcal{I}}$
could be ordered differently than $\\{\varphi_{j}\\}_{j\in\mathcal{I}}$, we
denote the corresponding eigenvalues by $\hat{\lambda}_{j}$. Thus,
$\sqrt{L}\big{(}\sqrt{L}(\phi_{j})\big{)}=\hat{\lambda}^{2}_{j}\phi_{j}=L(\phi_{j})=\bar{\lambda}^{2}_{j}\phi_{j}$.
Since the operators are positive semidefinite,
$\hat{\lambda}_{j}=\bar{\lambda}_{j}$. Hence,
$\tilde{L}(\phi_{j})=\sqrt{L}(\phi_{j}),\forall j$, i.e. $\tilde{L}=\sqrt{L}$.
∎

The above theorem, Theorem II.5, is well known in linear algebra and we
mention the proof as the construction of the square root operator is essential
for the proof of the theorem below.

######  Theorem II.6.

The linear space $\bar{H}^{i}=\\{g:g=\sqrt{\bar{L}^{i}}(f),f\in H\\}$ is a
RKHS with kernel $K^{i}$. $\sqrt{\bar{L}^{i}}(\cdot)$ establishes an isometric
isomorphism between $\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}^{\perp}$ and
$\bar{H}^{i}$, and the norm, $||f||_{\bar{H}^{i}}=||g||_{H}$, where
$f=\sqrt{\bar{L}^{i}}g,g\in\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}^{\perp}$.
Thus, the individual knowledge spaces can be retrieved from the fusion space.

###### Proof.

Since $\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}$ is closed subspace of $H$,
$H=\mathcal{M}^{i}\oplus\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}$, where
$\mathcal{M}^{i}=\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}^{\perp}$.
$\sqrt{\bar{L}^{i}}()$ maps one-one from $\mathcal{M}^{i}$ to $\bar{H}^{i}$.
Hence, $\mathcal{M}^{i}\subset\bar{H}^{i}$. Let
$f\in\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}$ and $g\in H$. Then, $\langle
f,\sqrt{\bar{L}^{i}}(g)\rangle_{H}=\langle\sqrt{\bar{L}^{i}}(f),g\rangle_{H}=0$,
i.e., $\sqrt{\bar{L}^{i}}(g)\perp f,\;\forall
f\in\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}$, $\forall g\in H$. Hence,
$\bar{H}^{i}\subset\mathcal{N}\big{(}\sqrt{\bar{L}^{i}}\big{)}^{\perp}=\mathcal{M}^{i}$.
Therefore, $M^{i}=\bar{H}^{i}$. The inner product on $\bar{H}^{i}$ is defined
as,

$\displaystyle\langle
f,g\rangle_{\bar{H}^{i}}=\langle\sqrt{\bar{L}^{i}}(\bar{f}),\sqrt{\bar{L}^{i}}(\bar{g})\rangle_{\bar{H}^{i}}=\langle\Pi_{\mathcal{M}^{i}}(\bar{f}),\Pi_{\mathcal{M}^{i}}(\bar{g})\rangle_{H},$

where $f=\sqrt{\bar{L}^{i}}(\bar{f}),\bar{f}\in H$ and
$g=\sqrt{\bar{L}^{i}}(\bar{g}),\bar{g}\in H$. For $f(\cdot)=K(\cdot,y)$,
$\bar{L}^{i}(f)(\cdot)=K^{i}(\cdot,y)$. Hence,
$K^{i}(\cdot,y)\in\mathcal{R}(\bar{L}^{i})\subseteq\bar{H}^{i},\;\forall
y\in\mathcal{X}$. Let $\\{\psi^{i}_{j}\\}$ be the eigenvectors of
$\bar{L}^{i}$ with corresponding eigenvalues, $\\{\lambda^{i}_{j}\\}$. Then,
from the construction of $\sqrt{\bar{L}^{i}}$ in Theorem II.5,
$\\{\psi^{i}_{j}\\}$ are the eigenvectors of $\sqrt{\bar{L}^{i}}$ with
corresponding eigenvalues, $\\{\sqrt{\lambda^{i}_{j}}\\}$. For any
eigenvector, $\psi^{i}_{j}$, with $\lambda^{i}_{j}\neq 0$,

$\displaystyle\langle\psi^{i}_{j}(\cdot),K^{i}(\cdot,y)\rangle_{\bar{H}^{i}}=$
$\displaystyle\langle\sqrt{\bar{L}^{i}}\Big{(}\frac{1}{\sqrt{\lambda^{i}_{j}}}\psi^{i}_{j}(\cdot)\Big{)},\sqrt{\bar{L}^{i}}\big{(}\sqrt{\bar{L}^{i}}(K(\cdot,y))\big{)}\rangle_{\bar{H}^{i}}$
$\displaystyle\overset{(a)}{=}\langle\frac{1}{\sqrt{\lambda^{i}_{j}}}\psi^{i}_{j}(\cdot),\Pi_{\mathcal{M}^{i}}(\sqrt{\bar{L}^{i}}(K(\cdot,y)))\rangle_{H},\overset{(b)}{=}\langle\frac{1}{\sqrt{\lambda^{i}_{j}}}\psi^{i}_{j}(\cdot),$
$\displaystyle\sqrt{\bar{L}^{i}}(K(\cdot,y))\rangle_{H}\overset{(c)}{=}\langle\sqrt{\bar{L}^{i}}(\frac{1}{\sqrt{\lambda^{i}_{j}}}\psi^{i}_{j}(\cdot)),K(\cdot,y)\rangle_{H}$
$\displaystyle=\langle\psi^{i}_{j}(\cdot),K(\cdot,y)\rangle_{H}=\psi^{i}_{j}(y),$

Thus, the reproducing property is satisfied by $\psi^{i}_{j}(\cdot)$. Since
every function in $\mathcal{M}^{i}=\bar{H}^{i}$ can expressed as unique linear
combination of $\\{\psi^{i}_{j}\\}$, and by the linearity of inner product it
follows that $\langle
f(\cdot),K^{i}(\cdot,y)\rangle_{\bar{H}^{i}}=f(y),\;\forall f\in\bar{H^{i}}$.
The reasoning for the equalities are as follows, $(a)$ by the definition of
the inner product and $\psi^{i}_{j}\in\mathcal{M}^{i}$ as $\lambda^{i}_{j}\neq
0$, $(b)\;$
$\sqrt{\bar{L}^{i}}(K^{i}(\cdot,y))\in\bar{H}^{i}=\mathcal{M}^{i}$, and $(c)$
symmetry of $\sqrt{\bar{L}^{i}}$. From Definition 2, it follows that
$\bar{H}^{i}$ is a RKHS with kernel $K^{i}(\cdot,\cdot)$. ∎

######  Corollary II.7.

The downloading operator from the fusion space $H$ to agent $i$’s knowledge
space, $H^{i}$, is $\sqrt{\bar{L}^{i}}\circ\Pi_{\mathcal{M}^{i}}$. The
downloading operator is linear and bounded.

##  III Regression and Fusion Problem

The knowledge spaces constructed in the previous section can used to formulate
many inference problems including regression, classification, etc. In this
section, we consider the least squares regression problem and the fusion
problem as an optimization problem.

###  III-A Regression at the Agents

Given (input-output) information pairs,
$\\{(x^{i}_{j},y^{i}_{j})\\}^{m}_{j=1}$, to agent $i$, the objective of the
agent is to estimate a mapping from input to output. The estimation problem
formulated as least squares regression problem is

$\displaystyle\underset{f\in
H^{i}}{\min}\sum^{m}_{j=1}(y^{i}_{j}-f(x^{i}_{j}))^{2}+\varrho^{i}||f||^{2}_{H^{i}}.$

Let, $\mathbf{K^{i}}=(K^{i}(x^{i}_{j},x^{i}_{k}))_{jk}=(\langle
K^{i}(\cdot,x^{i}_{k}),K^{i}(\cdot,x^{i}_{j})\rangle_{H^{i}})$, be the Gram
matrix corresponding to agent $i$ as defined in the beginning of Section II.
It is well known from the representer theorem (refer [22]), that the solution
for the above problem is given by
$f^{i}(\cdot)=\sum^{m}_{l=1}\alpha^{i}_{l}K^{i}(\cdot,x^{i}_{l})$ where
$\boldsymbol{\alpha^{i}}=(\mathbf{K^{i}}^{T}\mathbf{K^{i}}+\varrho^{i}\mathbf{K^{i}})^{-1}\mathbf{K^{i}}^{T}\mathbf{y^{i}}$
and $\mathbf{y^{i}}=(y^{i}_{1},\ldots,y^{i}_{m})$.

###  III-B Function Fusion Problem

The functions estimated by the agents are transmitted to the fusion center
where the following fusion problem is considered. As presented in [20], we
consider $\\{\mathfrak{b}=\\{b_{k}\\}_{k\geq 1}\subset H\\}$ which span $H$ to
define a dissimilarity measure between $f^{1}$ and $f^{2}$ as:

$\displaystyle d_{\mathfrak{b}}(f,g)=\sum_{k}\langle
f-g,b_{k}\rangle^{2}_{H}.$

The fusion problem is to find a linear combination of $f^{1}$ and $f^{2}$,
$f^{*}$, such that the dissimilarity between $f^{1},f^{*}$ and $f^{2},f^{*}$
is minimized. The fusion problem as an optimization problem is

$\displaystyle\;\underset{a,b\in\mathbb{R}}{\min}d_{\mathfrak{b}}(af^{1}+bf^{2},f^{1})+d_{\mathfrak{b}}(af^{1}+bf^{2},f^{2})+$
$\displaystyle\varrho||af^{1}+bf^{2}||^{2}_{H}.$

The fused function is considered as the function estimated by the system. It
is downloaded by the agents to compare (in the sense of norm) against their
own estimates.

##  IV Example

In this section, we demonstrate the application of the theory developed in the
previous sections. We consider the estimation of a real valued cubic
polynomial with real valued inputs. The coefficients of the polynomial where
chosen at random. Agent 1 was provided with $20$ samples of input-output data,
where the input was restricted to the set $[-5,5]$. The inputs were uniformly
spaced on the interval $[-5,5]$ and the corresponding outputs were obtained by
providing the inputs to the true function. The features considered by Agent 1
were, $\varphi^{1}_{1}(x)=1,\varphi^{1}_{2}(x)=x$, and
$\varphi^{1}_{2}(x)=x^{2}$, which implies that its kernel was
$K^{1}(x,y)=1+xy+x^{2}y^{2}$. Agent 2 was also provided with $20$ samples of
input-output data where the input data was uniformly spaced and restricted to
$[-10,-5]\cup[5,10]$, while the features considered by it where
$\varphi^{2}_{1}(x)=x^{2}$ and $\varphi^{2}_{2}(x)=x^{3}$. Hence,
$K^{2}(x,y)=x^{2}y^{2}+x^{3}y^{3}$.

With this set up, the regression problems (subsection III-A) were solved by
the agents. These functions were uploaded to the fusion center. Function
uploaded by Agent 1 and Agent 2 are plotted in Figure 2 and Figure 3
respectively. At the fusion center, the set
$\mathfrak{b}=\\{K(\cdot,\bar{x}_{j})\\}^{40}_{j=1}$ was considered, where
$\\{\bar{x}_{j}\\}^{40}_{j=1}$ were randomly sampled from $[-10,10]$. The
function fusion problem (subsection III-B) was solved. The fused function is
plotted in Figure 4.

To download the fused function onto the individual KSs, we demonstrate the
procedure outlined in subsection II-C. Suppose we choose the set of basis
vectors for the space $H$ as
$\varphi_{1}(x)=1,\varphi_{2}(x)=x,\varphi_{3}(x)=\sqrt{2}x^{2},\varphi_{4}(x)=x^{3}$,
then the kernel generated is
$K(x,y)=1+xy+2x^{2}y^{2}+x^{3}y^{3}=K^{1}(x,y)+K^{2}(x,y)$. With these basis
vectors, the coefficients for $K^{1}(\cdot,y)$ are
$[1,y,\frac{y^{2}}{\sqrt{2}},0]$, and for $K^{2}(\cdot,y)$ are
$[0,0,\frac{y^{2}}{\sqrt{2}},y^{3}]$. Thus, the matrix representation for
$\bar{L}^{1}$ is obtained as follows:

$\displaystyle\bar{L}^{1}(\varphi_{1})(y)=\langle\varphi_{1}(\cdot),K^{1}(\cdot,y)\rangle_{H}=\langle[1,0,0,0],[1,y,\frac{y^{2}}{\sqrt{2}},0]\rangle_{\mathbb{R}^{4}}$
$\displaystyle=1.\;\bar{L}^{1}(\varphi_{2})(y)=\langle[0,1,0,0],[1,y,\frac{y^{2}}{\sqrt{2}},0]\rangle_{\mathbb{R}^{4}}=y.\;\bar{L}^{1}(\varphi_{3})(y)$
$\displaystyle=\langle[0,1,0,0],[1,y,\frac{y^{2}}{\sqrt{2}},0]\rangle_{\mathbb{R}^{4}}=\frac{y^{2}}{\sqrt{2}}.\;\bar{L}^{1}(\varphi_{4})(y)=0.$
$\displaystyle L^{1}_{M}=\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\
0&0&\frac{1}{2}&0\\\
0&0&0&0\end{bmatrix},\;\sqrt{L^{1}_{M}}=\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\
0&0&\frac{1}{\sqrt{2}}&0\\\ 0&0&0&0\end{bmatrix},$ $\displaystyle
L^{2}_{M}=\begin{bmatrix}0&0&0&0\\\ 0&0&0&0\\\ 0&0&\frac{1}{2}&0\\\
0&0&0&1\end{bmatrix},\;\sqrt{L^{2}_{M}}=\begin{bmatrix}0&0&0&0\\\ 0&0&0&0\\\
0&0&\frac{1}{\sqrt{2}}&0\\\ 0&0&0&1\end{bmatrix}.$

Figure 2: True function, function uploaded by Agent 1, function downloaded by
Agent 1. Figure 3: True function, function uploaded by Agent 2, function
downloaded by Agent 2. Figure 4: True function, fused function, function
obtained through centralized estimation at fusion center.

The coefficients of the fused function with respect to basis chosen for $H$
were obtained. The coefficients for downloaded functions were obtained through
the operation $\boldsymbol{\alpha}^{i}=\sqrt{L^{i}_{M}}\boldsymbol{\alpha}$
where $\boldsymbol{\alpha}$ is the vector of coefficients corresponding to the
fused function. The downloaded functions correspond to
$\sum^{4}_{j=1}\alpha^{i}_{j}\varphi_{j}$, where
$\boldsymbol{\alpha}^{i}=[\alpha^{i}_{1},\alpha^{i}_{2},\alpha^{i}_{3},\alpha^{i}_{4}]$.
The function downloaded by Agent 1 and Agent 2 are plotted in Figures 2 and 3
respectively.

We observe that both the uploaded and downloaded function for Agent 1 do not
estimate the true function well as it is missing the $\varphi_{4}(x)=x^{3}$
feature. Agent $2$ is able to “better” estimate the function however the
impact of missing data points ( between $[-5,5]$ ) and missing features is
visible. However, we note that the agents are able to exchange knowledge as
the downloaded functions are “significantly” better than the uploaded
functions. To compare the performance of the fusion procedure, we compare it
against the centralized estimation method. In this method, the data collected
by both agents is sent to the fusion center, where a regression problem (refer
subsection III-A) is solved using the collective data. The function estimated
using this method is plotted in Figure 4.

We observe that the fused function is the best estimate among all estimates
considered so far. The centralized scheme is where the data collected by both
the agents is simultaneously processed in the fusion space. In this particular
example, it is possible that the joint processing of the data using both the
kernels led to “poor” computations as compared to local estimation using
individual kernels and averaging of the local estimates in the fusion space.
Further theoretical analysis is needed to compare the fusion method and the
centralized estimation method.

##  V Conclusion and Future Work

We considered the problem of function estimation by two agents given local
data and different set of features. We presented the construction of suitable
spaces for the estimation problem and fusion problems to be studied. We
derived operators to transform functions across spaces coherently. A
distributed estimation scheme without exchange of data was presented to solve
the problem considered. As future work, we are interested in: (i) developing a
sequential collaborative learning scheme involving the agents and the fusion
center; (ii) studying the consistency properties of such a scheme in the local
KS and the fusion space; (iii) quantifying the transfer of knowledge from one
agent to another.

## References

  * [1] D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” _Proceedings of the IEEE_ , vol. 103, no. 9, pp. 1449–1477, 2015.
  * [2] D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” _IEEE signal processing magazine_ , vol. 34, no. 6, pp. 96–108, 2017.
  * [3] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, “Berkeley mhad: A comprehensive multimodal human action database,” in _2013 IEEE workshop on applications of computer vision (WACV)_. IEEE, 2013, pp. 53–60.
  * [4] C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in _2015 IEEE International conference on image processing (ICIP)_. IEEE, 2015, pp. 168–172.
  * [5] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [6] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 year, 1000 km: The oxford robotcar dataset,” _The International Journal of Robotics Research_ , vol. 36, no. 1, pp. 3–15, 2017.
  * [7] O. Banos, C. Villalonga, R. Garcia, A. Saez, M. Damas, J. A. Holgado-Terriza, S. Lee, H. Pomares, and I. Rojas, “Design, implementation and validation of a novel open framework for agile development of mobile health applications,” _Biomedical engineering online_ , vol. 14, no. 2, pp. 1–20, 2015.
  * [8] A. Berlinet and C. Thomas-Agnan, _Reproducing kernel Hilbert spaces in probability and statistics_. Springer Science & Business Media, 2011.
  * [9] I. Steinwart and A. Christmann, _Support vector machines_. Springer Science & Business Media, 2008.
  * [10] G. Pillonetto, F. Dinuzzo, T. Chen, G. De Nicolao, and L. Ljung, “Kernel methods in system identification, machine learning and function estimation: A survey,” _Automatica_ , vol. 50, no. 3, pp. 657–682, 2014.
  * [11] T. Chen, M. S. Andersen, L. Ljung, A. Chiuso, and G. Pillonetto, “System identification via sparse multiple kernel-based regularization using sequential convex optimization techniques,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 11, pp. 2933–2945, 2014.
  * [12] M. Espinoza, J. A. Suykens, and B. De Moor, “Kernel based partially linear models and nonlinear identification,” _IEEE Transactions on Automatic Control_ , vol. 50, no. 10, pp. 1602–1606, 2005.
  * [13] M. Gönen and E. Alpaydın, “Multiple kernel learning algorithms,” _The Journal of Machine Learning Research_ , vol. 12, pp. 2211–2268, 2011\.
  * [14] S. S. Bucak, R. Jin, and A. K. Jain, “Multiple kernel learning for visual object recognition: A review,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 7, pp. 1354–1369, 2013.
  * [15] M. Donini, J. M. Monteiro, M. Pontil, J. Shawe-Taylor, and J. Mourao-Miranda, “A multimodal multiple kernel learning approach to alzheimer’s disease detection,” in _2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2016, pp. 1–6.
  * [16] Y. Cho and L. Saul, “Kernel methods for deep learning,” _Advances in neural information processing systems_ , vol. 22, 2009.
  * [17] M. Belkin, S. Ma, and S. Mandal, “To understand deep learning we need to understand kernel learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 541–549.
  * [18] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021\.
  * [19] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021\.
  * [20] A. Raghavan and K. H. Johansson, “Distributed regression by two agents from noisy data,” in _2023 European Control Conference (ECC)_. IEEE, 2023, pp. 1–6.
  * [21] J. S. Baras and A. Raghavan, “Order effects of measurements in multi-agent hypothesis testing,” _Communications in Information and Systems_ , vol. 21, no. 2, pp. 173–208, 2021.
  * [22] T. Hofmann, B. Schölkopf, and A. J. Smola, “Kernel methods in machine learning,” _The Annals of Statistics_ , vol. 36, no. 3, pp. 1171–1220, 2008\.


# CATFL: Certificateless Authentication-based Trustworthy Federated Learning
for 6G Semantic Communications

Gaolei Li 1, Yuanyuan Zhao2, and Yi Li 3
1School of Electronic Information and Electrical Engineering, Shanghai Jiao
Tong University, China  2School of Information Science and Technology,
Hangzhou Normal University, China
3 College of Information, Mechanical and Electrical Engineering, Shanghai
Normal University, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Federated learning (FL) provides an emerging approach for collaboratively
training semantic encoder/decoder models of semantic communication systems,
without private user data leaving the devices. Most existing studies on
trustworthy FL aim to eliminate data poisoning threats that are produced by
malicious clients, but in many cases, eliminating model poisoning attacks
brought by fake servers is also an important objective. In this paper, a
certificateless authentication-based trustworthy federated learning (CATFL)
framework is proposed, which mutually authenticates the identity of clients
and server. In CATFL, each client verifies the server’s signature information
before accepting the delivered global model to ensure that the global model is
not delivered by false servers. On the contrary, the server also verifies the
server’s signature information before accepting the delivered model updates to
ensure that they are submitted by authorized clients. Compared to PKI-based
methods, the CATFL can avoid too high certificate management overheads.
Meanwhile, the anonymity of clients shields data poisoning attacks, while
real-name registration may suffer from user-specific privacy leakage risks.
Therefore, a pseudonym generation strategy is also presented in CATFL to
achieve a trade-off between identity traceability and user anonymity, which is
essential to conditionally prevent from user-specific privacy leakage.
Theoretical security analysis and evaluation results validate the superiority
of CATFL.

###### Index Terms:

6G semantic communication, Federated learning, Certificateless authentication,
Pseudonym generation, Privacy-enhancing.

##  I Introduction

The 6G communication goes beyond the mobile internet to embrace omnipresent
Internet of Everything applications and artificial intelligence (AI) services.
This shall evolve the wireless communication networks from “connected things”
to “connected intelligence”. The 6G communications will enable the
interconnections among lots of intelligent agents within a hyper-connected
cyber-physical system [1]. However, 6G communications face with many
challenges generated by constrained network resources, high energy
consumption, and new attack surfaces (e.g., privacy leakage, data poisoning
attacks), which continues to bring significant hindrance to their world-wide
realization and deployment. Semantic communication is a brand-new
communication paradigm, which meets the interaction requirements of
intelligent agents in the 6G era [2, 3, 4]. Meanwhile, federated Learning (FL)
provides a promising collaborative mode to train joint learning models for
connected intelligence at the network edge [5]. Compared to the centralized
learning framework, the FL allows mobile/edge devices to collaboratively
extract the distributed knowledge from the data that does not leave the
devices [6, 7]. The characteristics of local training and joint learning of FL
is helpful to protect the privacy of information senders and receivers in 6G
semantic communications [6].

Figure 1: Security threats of federated learning in 6G semantic
communications. Before participating in the joint training of semantic
encoder/decoder, the client needs to obtain a certificate issued by the
certification center, and only the client with the legal certificate can join
the FL model aggregation. Meanwhile, before receiving the encoded semantic
information sent by the sender, the receiver also needs to verify if the
sender’s identity is valid.

However, FL still exposes many security threats. Firstly, poisoning attacks
seriously threatens its usability [8, 9]. On one hand, since the users’
original data and the local training process is not open for the FL server,
malicious FL clients may submit poisoned parameters to induce encoder/decoder
errors at the model testing phase. Notably, a poisoned parameter may be
diffused across all FL clients by model aggregation and distribution, thereby
rendering the final global learning model to be backdoored. For a purpose-
based terrorist to misguide other autonomous vehicles at the stop sign, it is
possible to move forward by submitting maliciously-crafted inputs, leading to
a serious traffic accident [10]. Therefore, any clients and parameters they
submit in the FL training process need to be authenticated and traced. On the
other hand, FL is also vulnerable to model poisoning attacks because it is
extremely hard to guarantee that the FL server is trustworthy and robust
especially when deployed in 6G edge computing scenarios. In [11], the attacker
ambitiously substitutes the aggregated global model with a malicious model to
strengthen the poisoning effect. That’s why the FL server also needs to be
authenticated by each client to reject false global model. Secondly, possible
privacy leakage from FL-based semantic communication systems also should be
concerned. On one hand, the semi-honest FL server can reconstruct sensitive
training samples (e.g. facial images, financial records, and medical images)
of the targeted FL clients through the shared gradients or parameters [12,
13]. On the other hand, since 6G semantic communication transmits compressed
semantic information between senders and receivers, the training data can be
reconstructed from the semantic representation vectors if the attacker
establishes an eavesdrop mechanism on the semantic communication channels.
Therefore, it is still necessary to propose an additional privacy-enhancing
mechanism against such privacy leakage risks in FL-based semantic
communication systems.

Motivated by the aforementioned issues, a trustworthy FL should consider the
following fundamental issues: 1) how to guarantee that users’ privacy is not
leaked during the model training process and 2) how to guarantee the model
robustness to malicious manipulations. Therefore, resisting poisoning attacks
while protecting users’ privacy has became an meaningful and urgent demand.
Cryptography-based methods (e.g., homomorphic encryption and secret sharing)
are essential to guarantee the confidentiality of shared gradients or
parameters between FL clients. However, cryptography-based methods are always
invalid when deployed on devices with limited computational and communication
resources. For example, too high certificate management overheads reduce the
practicality of PKI-based schemes in 6G semantic communication systems.

In this paper, we propose a Certificateless Authentication-based Trustworthy
Federated Learning (CATFL) framework for 6G semantic communications, which can
efficiently defend against poisoning attacks without leaking users’ privacy.
In CATFL, the FL server has two types of private key: 1) partial private key
and 2) full private key, which are respectively generated by the trusted key
generator center (KGC) and itself. Therefore, even an attacker who colludes
with one of the CATFL server and KGC, it can not get the full secret key to
impersonate, preventing the poisoning attacker from substituting the global
learning model with a maliciously-modified model. Meanwhile, the CATFL is able
to guarantee the trustworthiness of the local client’s gradients because of
providing mutual authentication for each client. We also designate a pseudonym
generation strategy to hide each client’s real identity. This strategy allows
to trace the original real identity of each CATFL client by trusted third-
parties, enabling to identify malicious mobile/edge devices. In a summary, the
main contributions of our work are listed as follows:

  * •

We propose a Certificateless Authentication-based Trustworthy Federated
Learning (CATFL) framework as the underlying technology of 6G semantic
communications to provide higher security. Each elements in CATFL has two
types of private key , which are independently generated by the KGC and
itself. We prove that the CATFL can resist two types of security threats: 1)
poisoning attacks (including server-side and client-side); 2) privacy leakage
(including gradient leakage and semantic representation leakage).

  * •

A pseudonym generation strategy is presented to achieve a trade-off between
user anonymity and identity traceability in CATFL. Hence, a powerful deterrent
against malicious threats to FL-based 6G semantic communication systems is
achieved.

  * •

We provide a comprehensive theoretic proof for the security and
trustworthiness of proposed CATFL by comparison to existing PKI-based methods.
It demonstrates that the CATFL is more applicable for practical 6G semantic
communication systems.

##  II Related Work

We comprehensively overview the security challenges of FL in 6G semantic
communications, mainly caused by poisoned attacks and privacy leakage. And
then, the disadvantages of existing secure authentication methods for
trustworthy FL in 6G semantic communications is discussed in detail.

###  II-A Security Challenges of FL in 6G Semantic Communications

Different from existing advanced channel encoding/decoding and modulation
techniques, semantic communication attempts to extract the “meanings” of sent
information at the transmitter using artificial intelligence and then
transmits these “meanings” to the receivers. With the assistance of a shared
knowledge base (KB), the receiver can accurately catch the received “meanings”
[14]. In 6G semantic communication systems, semantic encoder/decoder could be
jointly trained using the FL framework for ensuring low loss rate of shared
semantic knowledge and transmission accuracy. However, many security risks of
FL have not yet been identified in details, lacking of effect countermeasures.

####  II-A1 Poisoning Attacks

Many practical poisoning attacks on FL have been proposed, which can be
divided into two aspects: 1) data poisoning attacks and 2) model poisoning
attacks. Usually, the data poisoning attack uses crafted data to maliciously
substitute normal samples secretly. To generate crafted data, Zhang et al.
[15] introduced generative adversarial networks to inversely reconstruct
training data from the given models, which enables a high attack efficiency.
By model replacement and adjusting the scaling factor, Eugene et al. [11]
enhanced the persistence of backdoor in FL-based systems.

####  II-A2 Privacy Leakage Threats

Nowadays, although the FL is designated to enhance the privacy-preserving
ability, privacy leakage threats have plagued the development of FL
technologies. On one hand, the untrusted parameter server may reconstruct the
training data using an adversarial algorithm proposed in [16] from the
gradients submitted by each mobile/edge device. This is mainly because the
updated model parameters in the training process may memory sensitive
information and leak them to malicious adversaries [17]. On the other hand,
the attacker can reconstruct the training data by exploiting some query-
feedback information on the targeted learning model, i.e., membership
inference and model inversion [18]. The common feature of this kind of privacy
leakage threat is to generate dummy samples that can approximate the real
gradients, predictions, and weight parameters submitted by local FL clients.

###  II-B Secure Authentication for Trustworthy Federated Learning

Secure authentication is widely deployed as an emerging technique of
exchanging model parameters and certification to provide security and
trustworthiness for FL-based system against privacy leakage threats and
poisoning attacks. Authors in [19] proposed a verifiable federated learning
(VerifyNet) framework, enabling each client to verify whether the FL server
works correctly. In VerfyNet, the adversary can not deceive users by forging
the identity proof unless the formulated NP-hard problem is resolved. Salvia
et al. [20] provided an implementation case of secure authentication on the
open-source FL platform named as “Flower”. Since there are two types of
malicious behaviors: i) unauthorized manipulations to the submitted model
updates; ii) sending the same model update to the FL server with multiple
times, authors in [21] introduced data signatures to prevent from malicious
behavior i). Meanwhile, this work also proposes to exploit blind signatures to
sign local model updates only once to avoid malicious behavior of ii).
However, all existing methods use the public key infrastructure (PKI)-based
authentication, which is a certificate-based mechanism so that the certificate
management cost and communication overheads must be affordable. Therein, many
unmanned mobile/edge devices in semantic communication systems can not be
equipped with such complex PKI-based authentication mechanisms. To this end,
we present a novel, fundamental, efficient certificateless authentication
countermeasures for deploying FL in 6G semantic communication systems. To the
best of our knowledge, this is the first work to introduce certificateless
authentication into the FL-based 6G semantic communication field.

##  III Proposed CATFL framework

In this section, we will introduce the proposed CATFL framework for 6G
semantic communications in detail, including the preliminaries, security
requirements, adversarial model and overview of proposed CATFL.

###  III-A Preliminaries

####  III-A1 Certificateless Cryptography

Certificateless Cryptography is first proposed by Al-Riyami et al. [22] to
deal with the key escrow limitation of the traditional PKI-based cryptography.
In Certificateless Cryptography, a trusted third party named as KGC is
responsible to generate a partial private key (PSK) for the users. The user
obtains the full private key by combining the PSK with a secret value, which
is unknown to the KGC. Under this seeting, the KGC can not achieve the user’s
private keys. For FL-based 6G semantic communication systems, intelligent
agents are often deployed on mobile/edge devices that has limited computation
and communication resources. To reduce authentication overheads and enhance
the FL trustworthiness, using Certificateless Cryptography to establish a
novel 6G communication architecture has became an urgent and important demand.
In this article, we will discuss its feasibility in detail (refer to Section
III.D).

####  III-A2 Elliptic Curve Cryptography

In Elliptic Curve Cryptography (ECC), there are two types of components, 1)
elliptic curve, 2) pre-defined operation rules. Given a base point on the
elliptic curve, it is very hard to calculate the discrete logarithm value of a
random elliptic curve element. To save computation resouces, we will use ECC
to implement the certificateless authentication mechanism in CATFL, in which
the elliptic curve is formulated as the following equation:

$\mathbb{E}=\\{(x,y)|y^{2}=x^{3}+ax+b,4a^{3}+27b^{2}=0\\}$ (1)

Where $\mathbb{E}$ presents an elliptic curve and $G$ is a generator with the
order $r$. Given a base point $Q=kG$ and $k\in Z_{r}$, it is very hard and
almost unpractical to find an integer $k$ in the polynomial time. In FL-based
6G semantic communication systems, we can apply the ECC to implement the
certficateless authentication mechanism among the server and clients as well
as the sender and the receiver.

###  III-B Security Requirements

To bring out the motivation of our work, we summarize the security
requirements of FL in 6G semantic communication systems as follows.

  * •

Message authentication: The receiver of semantic information needs to verify
the semantic encodes’ validity. Any tamper on the semantic encodes shall be
easily detected.

  * •

Non-repudiation: In CATFL, all authenticated messages could not be repudiated,
that means no CATFL entities can deny a valid signature.

  * •

Anonymity: Each participator needs to generate a pseudonym and its real
identity has to be hidden during the communication process.

  * •

Un-linkability: No CATFL entities can link multiple different messages to the
same user.

  * •

Resistant against attacks: The CATFL framework also needs to prevent from some
typical attacks including data modification attacks and relay attacks.

  * •

Conditional traceability: Only the trusted third-party that is named as KGC
can know the real identity of each participator in CATFL.

###  III-C Adversarial Model

According to listed security requirements of CATFL for 6G semantic
communications, we present two types of adversaries, which are denoted as
follows: 1) $\mathcal{A}_{1}$, and 2) $\mathcal{A}_{2}$. Therein,
$\mathcal{A}_{1}$ has the ability of changing the public key of each
participator using a selected constant value. However, it fails to obtain the
master private key of KGC. Different from $\mathcal{A}_{1}$, $\mathcal{A}_{2}$
can get the master private key of KGC, but it fails to modify public keys of
any participator.

If a valid signature $Verify(P_{pub}$, $m^{*}$, $AID^{*}$, $\Theta^{*})=1)$ is
forged by an adversary $\mathcal{A}$, it can be consider the $\mathcal{A}$
($\mathcal{A}_{1}$ or $\mathcal{A}_{2}$) has launched a successful attack. The
proposed CATFL framework can be validated to be secure if the attack success
rate for any attackers is negligible.

###  III-D Overview of Proposed CATFL

The proposed CATFL framework consists of four main entities, i.e., TRA, KGC,
CS and User, which connect with each other over wireless communication
channels. The proposed CATFL framework has two different communication levels:
1) upper level, and 2) lower level. The upper level contains the
communications between the TRA and KGC, as well as Server-to-User (S2U)
communications via a secret channel, while the lower level consists of User-
to-User (U2U) semantic communications via a public channel. The system
entities of CATFL is shown in Fig. 2 and explained below in detail.

  * •

Tracing Authority (TRA): The TRA in CATFL is a trusted entity with enough
resources. Functions of this entity contain pseudonym generation for
participants (FL server and clients) and the corresponding tracing strategy
configuration (if needed).

  * •

Key Generation Center (KGC): The KGC in CATFL is an independent and trusted
third party, which is responsible to distribute all the private and public
keys of each participator in the 6G communication systems.

  * •

Cloud Server (CS): The CS in CATFL is responsible to receive all model updates
submitted by FL clients and aggregate them (usually by averaging) to achieve
an optimized global model.

  * •

Users: All message senders and receivers (also named as clients) in 6G
communications, collaboratively train a uniform encoder-decoder model using
the federal paradigm. Each CATFL entity trains the model with several epochs
over the private local data on each mobile/edge device, and then uploads the
signed model updates to CS.

Figure 2: The workflow of proposed CATFL. 1) System setup, 2) Identity
anonymization, 3) Get the PSK, 4) Extract the USK, 5) CS configuration, 6)
Signature/verification, 7) Aggregation/distribution, 8) Secure semantic
transmission.

PKI-based methods usually involve the certificate management issues, in which
a troublesome certificate revocation list should be maintained. Meanwhile, ID-
based schemes may suffer from key escrow problems. We utilize certificateless
cryptography to overcome these disadvantages. The concrete key generation
steps in CATFL and its corresponding authentication procedures are described
in detail as follows. The CATFL framework consists of eight steps in total,
which are 1) System setup, 2) Identity anonymization, 3) Get PSK, 4) Extract
USK, 5) CS configuration, 6) Signature/verification, 7)
Aggregation/distribution, 8) Secure semantic transmission. Fig. 2 illustrates
the workflow of proposed CATFL framework.

####  III-D1 System setup

Firstly, this algorithm specifies a security parameter $\kappa$ as the input,
and then it generates a cyclic additive group $(\mathbb{G},q,P)$ and four hash
functions
$H_{0}:\mathbb{G}\times\mathbb{G}\times\\{0,1\\}^{n}\rightarrow\\{0,1\\}^{n}$,
$H_{1}:\mathbb{G}\times\mathbb{G}\times\mathbb{G}\rightarrow Z_{q}^{*}$,
$H_{2}:\\{0,1\\}^{n}\times\mathbb{G}\times\mathbb{G}\times\mathbb{G}\times\\{0,1\\}^{n}\rightarrow
Z_{q}^{*}$ and
$H_{3}:\\{0,1\\}^{n}\times\mathbb{G}\times\mathbb{G}\times\\{0,1\\}^{n}\rightarrow
Z_{q}^{*}$, then sends $\\{\mathbb{G},q,P,H_{0},H_{1},H_{2},H_{3}\\}$ to TRA
and KGC, respectively. After receiving those information, the TRA and KGC in
CATFL will initialize system parameters according to the following three
steps.

  * •

TRA in CATFL randomly selects $\alpha\in Z_{q}^{*}$, $T_{pub}=\alpha P$.

  * •

The KGC in CATFL randomly selects a value $\beta\in Z_{q}^{*}$, and configures
$P_{pub}=\beta P$.

  * •

The information $params=\\{\mathbb{G},q,$ $P,H_{0},H_{1},H_{2},H_{3},$
$T_{pub},$ $P_{pub}\\}$ is published.

####  III-D2 Identity anonymization

The TRA invokes this step to initialize anonymous identities of $CS$ and each
user $U_{i}$ in CATFL, the corresponding identity information is denoted as
$RID_{n}$.

  * •

$CS$ or $U_{i}$ in CATFL randomly selects $r_{i}\in Z^{*}_{q}$.

  * •

$CS$ or $U_{i}$ in CATFL computes $AID_{i,1}=r_{i}\times P$ and sends
$RID_{i},AID_{i,1}$ to the TRA where $i=1,2,...,n$ through a secure wireless
channel.

  * •

The TRA checks the validity of $RID_{i}$. If not, the TRA will reject this
request; Otherwise, it holds the system timestamp $T_{i}$ and calculates
$AID_{i,2}=RID_{i}\oplus H_{0}(\alpha AID_{i,1},T_{pub},T_{i})$. The pseudonym
of each CATFL entity is generated as $AID_{i}$ =
$\\{AID_{i,1},AID_{i,2},T_{i}\\}$.

  * •

The TRA stores the computed $AID_{i}$ and transmits them to the other entities
in CATFL.

####  III-D3 Get the PSK

The KGC in CATFL generates and sends back the PSK to the requester secretly.
After that, the requester will generate its full private key and corresponding
public key for the proposed certificateless authentication mechanism. In
detail, the requester $CS$ or $U_{i}$ sends $AID_{i}$ to the KGC. The KGC
computes the PSK of each requester and then retrieves the $AID_{i}$ from the
real identity list. If the $AID_{i}$ exists in the base, the KGC needs to
execute the following operations:

  * •

Randomly produces a value $k_{i}\in Z^{*}_{q}$ as the system input.

  * •

Calculates $U_{i}=k_{i}P$, generates $\theta_{i}=H_{1}(AID_{i},U_{i},$
$P_{pub})$ and then produces $\lambda_{i}=(k_{i}+\theta_{i}\beta)(mod\;q)$.

  * •

Publishes the $\\{\lambda_{i},U_{i}\\}$ to $CS$ or $U_{i}$ secretly.

####  III-D4 Extract the USK

When $CS$ or $U_{i}$ receives $\\{\lambda_{i},U_{i}\\}$, the participator will
produce its key pair according to the following operations:

  * •

Calculate $\theta_{i}^{*}$ = $H_{1}(AID_{i},U_{i},P_{pub})$.

  * •

Check whether the equation $\lambda_{i}P=U_{i}+$ $\theta_{i}^{*}P_{pub}$ is
right. If not, close the current session; Otherwise, go to the next step.

  * •

Computes $X_{i}=\mu_{i}P$ and configures the public key as
$PK_{i}=\\{X_{i},U_{i}\\}$. Subsequently, these public keys will be shared
with other CATFL entities.

In CATFL, a batch of $AID_{i}$ and PSK $\\{\lambda_{i},U_{i}\\}$ will be pre-
loaded into the mobile/edge devices and store safely. Each user can utilize a
unique $AID_{i}$ and a PSK $\\{\lambda_{i},U_{i}\\}$ to validate the
identities of other entities. If the user uses up every $AID_{i}$, it can
build a new connection with the TRA again and replenish a stock of $AID_{i}$
and $\\{\lambda_{i},U_{i}\\}$ using a secure communication channel.

####  III-D5 CS configuration

The CS in CATFL will serve as a parameter server to provide model aggregation
services. First, the server needs to initialize the weights of the global
model and required model parameters (e.g., the total number of FL rounds, the
total number of FL clients, and the participation rate of FL clients in each
training round). It then activates all selected FL clients and broadcasts the
initialized global model for local training.

####  III-D6 Signature/verification

The signature process is invoked by any $CS$ or $U_{i}$ to compute
message/signature pairs, $CS$ or $U_{i}$ will execute the following
operations:

  * •

Randomly chooses $a_{i}\in Z^{*}_{q}$, and then generate the message signature
by computing $A_{i}=a_{i}P$, $h_{1,i}$
$=H_{2}(m_{i},AID_{i},PK_{i},A_{i},P_{pub},t_{i})$,
$h_{2,i}=H_{3}(m_{i},AID_{i},$ $PK_{i},A_{i},P_{pub},h_{1,i})$. Therein,
$t_{i}$ denotes the timestamp of the whole system.

  * •

Calculates $\eta_{i}=a_{i}-h_{1,i}\mu_{i}-h_{2,i}\lambda_{i}(modq)$.

  * •

Configures the message signature as $\Theta_{i}=\\{\eta_{i},A_{i}\\}$ and
broadcasts $(m_{i},AID_{i},\theta_{i},PK_{i},\Theta_{i},t_{i})$ to other
relational $CS$ or $U_{i}$.

The identity verification process is also invoked by any $CS$ or $U_{i}$,
which aims to verify the validity of each CATFL entity. If yes, the receiver
in CATFL can accept the semantic information and perform further actions. The
identity verification process is shown below:

  * •

Verifies the parameters $T_{i}$ and $t_{i}$. If both of them are not fresh,
the received semantic information will be discarded.

  * •

Checks if $\Theta_{i}$ is equal to $H_{1}(AID_{i},U_{i},P_{pub})$. If not,
discards this traffic status; Otherwise, continues to execute further
operations.

  * •

Computes the hash values one by one $h_{1,i}^{*}$ = $H_{2}(m_{i},$
$AID_{i},PK_{i},$ $A_{i},P_{pub},t_{i})$, and $h_{2,i}^{*}$ =
$H_{3}(m_{i},AID_{i},$ $PK_{i},A_{i},P_{pub},h^{*}_{1,i})$ and $A_{i}^{*}$ =
$\eta_{i}P+$ $h_{1,i}^{*}X_{i}+h^{*}_{2,i}$ $U_{i}+(h^{*}_{2,i}$
$\theta_{i})P_{pub}$ respectively.

  * •

Validates if the value of $A_{i}$ is equal to $A^{*}_{i}$. If not, this
semantic information will be discarded; Otherwise, the traffic status can be
acceptable.

Since the CATFL is derived from [23], the Proof of Correction can refer to
that article.

####  III-D7 Aggregation/distribution

The CATFL server first aggregates the model updates sent by each FL client and
then sends back the aggregated global model to the FL clients for the next
training round.

##  IV Security Analysis and Evaluation

The CATFL can be proved to meet the aforementioned security requirements.
Specifically, to prove the security of proposed CATFL, we design two types of
games: 1) playing between a challenger $\mathcal{C}$ and the adversaries
$\mathcal{A}_{1}$, 2) playing between a challenger $\mathcal{C}$ and the
adversaries $\mathcal{A}_{2}$.

The detailed security analysis is shown as follows:

  1. 1.

Message Authentication/Integrity: According to the features of ECC, the CATFL
is proven secure under two types of adversarial models so that the FL model
updates and transmitted semantics can not be forged.

  2. 2.

Anonymity: The anonymous identity is used to hide the real identity of CATFL
entities because the attacker can not compute $\alpha AID_{i,1}$.

  3. 3.

Non-Repudiation: The TRA could trace the real identity $RID_{i}$ of each CATFL
entity through the pseudonym identity $AID_{i}$. Therefore, in the proposed
CATFL, no entity can deny the validity of the received signature.

  4. 4.

Conditional Traceability: The real identity $RID_{i}$ can be reconstructed
only by the TRA via its master secret key $\alpha$. If poisoned model updates
are submitted by a malicious client, the TRA can trace this client.

  5. 5.

Server impersonation attack: To persistent the backdoor in FL encoder/decoder
model of 6G semantic communications, the attacker should generate a
message/signature pair
$\\{m_{i},AID_{i},\theta_{i},PK_{i},\Theta_{i},t_{i}\\}$, satisfying the
equations:

$\theta_{i}=H_{1}(AID_{i},U_{i},P_{pub})$ (2)
$A_{i}=\eta_{i}P+h_{1,i}X_{i}+h_{2,i}(U_{i}+\theta_{i}P_{pub})$ (3)

However, $\mathcal{PPT}$ attackers cannot generate the valid message/signature
pair because none of them can solve the puzzle in ECC. To this end, the CATFL
can successfully resist the server impersonation attack.

  6. 6.

Un-Linkability: In the identity anonymization phase, the TRA randomly picks
$r_{i}$ to generate a pseudonym for the requester. In addition, the requester
randomly selects $a_{i}$ to generate the message signature. It is not possible
for $\mathcal{PPT}$ attackers to connect any two pseudonym identities or two
signatures to a specific user.

  7. 7.

Modification attack: If a $\mathcal{PPT}$ attackers modified the
message/signature pair $\\{m_{i},AID_{i},\theta_{i},PK_{i},\Theta_{i}\\}$, the
modification could be discovered by checking the equations
$\theta_{i}=H_{1}(AID_{i},U_{i},P_{pub})$ and $A_{i}=\eta_{i}P+h_{1,i}X_{i}$
$+h_{2,i}(U_{i}+\theta_{i}P_{pub})$. Therefore, our CATFL framework can resist
the modification attack.

To bring out the efficiency of proposed CATFL, we compare the communication
latency with existing methods. Firstly, since the CATFL contains two stages:
1) training stage and 2) testing stage, the communication latency for both two
stages are computed, respectively. The signature latency for each training
round is denoted as $T_{sign}$, and the corresponding verification latency is
presented as $T_{veri}$. Thus, for $N$ training rounds, the communication cost
of CATFL is $N*(T_{sign}+T_{veri})$. For $M$ messages exchanged between the
sender and the receiver, the communication cost is $M*(T_{sign}+T_{veri})$. As
an instance, Fig. 3 shows the comparison of communication costs against
certificate-based authentication. Another impact factor for the communication
cost is the number of CATFL entities $K=2*P+1$, where $P$ is the number of
sender/receiver pairs. Besides, since the FL-based system should obey the
synchronous principle, it exists a waiting latency that obeys the Poisson
distribution: $\Delta T_{ca}\sim\pi(\lambda)$.

Figure 3: The comparison of communication costs against certificate-based
authentication.

##  V Conclusion

In this paper, we propose a Certificateless Authentication-based Trustworthy
Federated Learning (CATFL) for 6G Semantic Communications scheme. With the
certificateless authentication technique, the CATFL entities have two types of
private key (i.e., partial and full), which are generated independently by the
trusted authority and itself. Therefore, even an attacker who colludes with
the KGC cannot obtain the participant’s full secret key. On the basis of this,
the proposed CATFL can prevent the semi-honest servers from inferring the
users’ private data. The security analysis and evaluation shows the proposed
CATFL has higher security and lower communication cost against poisoning
attacks and privacy leakage threats. In the future, we also can introduce more
emerging techniques (e.g., blockchain) to construct a secure and efficient 6G
semantic communication systems.

## Acknowledgment

This research work is funded by Shanghai Sailing Program under Grant No.
21YF1421700, Defence Industrial Technology Development Program Grant No.
JCKY2020604B004, and National Nature Science Foundation of China under Grant
No. 62202303 and U20B2048.

## References

  * [1] Z. M. Fadlullah, B. Mao, and N. Kato, “Balancing qos and security in the edge: Existing practices, challenges, and 6g opportunities with machine learning,” _IEEE Communications Surveys & Tutorials_, vol. 24, no. 4, pp. 2419–2448, 2022.
  * [2] W. Yang, H. Du, and et al., “Semantic communications for future internet: Fundamentals, applications, and challenges,” _IEEE Communications Surveys & Tutorials_, pp. 1–1, 2022.
  * [3] Z. Yang, M. Chen, G. Li, Y. Yang, and Z. Zhang, “Secure semantic communications: Fundamentals and challenges,” _arXiv:2301.01421v1_ , pp. 1–7, 2023.
  * [4] Y. Wang, M. Chen, T. Luo, W. Saad, D. Niyato, H. V. Poor, and S. Cui, “Performance optimization for semantic communications: An attention-based reinforcement learning approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 9, pp. 2598–2613, 2022.
  * [5] H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” _CoRR_ , 2016. [Online]. Available: http://arxiv.org/abs/1602.05629
  * [6] H. Tong, Z. Yang, and e. a. Wang, “Federated learning based audio semantic communication over wireless networks,” in _IEEE GLOBECOM_ , 2021, pp. 1–6.
  * [7] B. Ghimire and D. B. Rawat, “Recent advances on federated learning for cybersecurity and cybersecurity for federated learning for internet of things,” _IEEE Internet of Things Journal_ , vol. 9, no. 11, pp. 8229–8249, 2022.
  * [8] G. Li, J. Wu, S. Li, W. Yang, and C. Li, “Multitentacle federated learning over software-defined industrial internet of things against adaptive poisoning attacks,” _IEEE Transactions on Industrial Informatics_ , vol. 19, no. 2, pp. 1260–1269, 2023.
  * [9] Z. Yan, G. Li, Y. TIan, J. Wu, S. Li, M. Chen, and H. V. Poor, “Dehib: Deep hidden backdoor attack on semi-supervised learning via adversarial perturbation,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 12, pp. 10 585–10 593, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/17266
  * [10] B. Ghimire and D. B. Rawat, “Secure, privacy preserving, and verifiable federating learning using blockchain for internet of vehicles,” _IEEE Consumer Electronics Magazine_ , vol. 11, no. 6, pp. 67–74, 2022.
  * [11] E. Bagdasaryan, A. Veit, and et al., “How to backdoor federated learning.” _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , pp. 2938–2948, 2020.
  * [12] W. Wei and L. Liu, “Gradient leakage attack resilient deep learning,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 303–316, 2022.
  * [13] W. Wei, L. Liu, Y. Wut, G. Su, and A. Iyengar, “Gradient-leakage resilient federated learning,” in _2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)_ , 2021, pp. 797–807.
  * [14] X. Luo, H.-H. Chen, and Q. Guo, “Semantic communications: Overview, open issues, and future research directions,” _IEEE Wireless Communications_ , vol. 29, no. 1, pp. 210–219, 2022.
  * [15] J. Zhang, B. Chen, and et al., “Poisongan: Generative poisoning attacks against federated learning in edge computing systems,” _IEEE Internet of Things Journal_ , vol. 8, no. 5, pp. 3310–3322, 2021.
  * [16] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” _NeurIPS_ , pp. 14 774–14 784, 2019.
  * [17] A. Bhowmick, J. Duchi, and et al., “Protection against reconstruction and its applications in private federated learning,” 2019. [Online]. Available: https://arxiv.org/pdf/1812.00984
  * [18] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _IEEE Symposium on Security and Privacy (SP)_ , 2019, pp. 691–706.
  * [19] G. Xu, H. Li, S. Liu, K. Yang, and X. Lin, “Verifynet: Secure and verifiable federated learning,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 911–926, 2020.
  * [20] K. H. Li, P. P. B. de Gusmão, D. J. Beutel, and N. D. Lane, “Secure aggregation for federated learning in flower,” _Proceedings of the ACM International Workshop on Distributed Machine Learning_ , 2021.
  * [21] F. Karakoç, L. Karaçay, and et al., “A security-friendly privacy solution for federated learning,” in _AI6G@WCCI_ , 2022.
  * [22] S. S. Al-Riyami and K. G. Paterson, “Certificateless public key cryptography,” 2003. [Online]. Available: https://eprint.iacr.org/2003/126
  * [23] X. Zhou, M. Luo, and et al., “Efficient certificateless conditional privacy-preserving authentication for vanets,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 7, pp. 7863–7875, 2022.