# FedMT: Federated Learning with Mixed-type Labels

Qiong Zhang
Department of Statistics
The University of British Columbia
Institute of Statistics and Big Data
Renmin University of China
<EMAIL_ADDRESS>
Work done while transitioning to Renmin university of China. Aline Talhouk
Department of Obstetrics & Gynecology
The University of British Columbia
<EMAIL_ADDRESS>
Gang Niu
RIKEN
<EMAIL_ADDRESS>Xiaoxiao Li
Department of Electrical and Computer Engineering
The University of British Columbia
<EMAIL_ADDRESS>
Corresponding to: Xiaoxiao Li (xiaoxiao.li@ece.ubc.ca).

###### Abstract

In _federated learning_ (FL), classifiers (e.g., deep networks) are trained on
datasets from multiple centers without exchanging data across them, and thus
improves _sample efficiency_. In the classical setting of FL, the _same
labeling criterion_ is usually employed across all centers being involved in
training. This constraint greatly limits the applicability of FL. For example,
standards used for _disease diagnosis_ are more likely to be different across
clinical centers, which mismatches the classical FL setting. In this paper, we
consider an important yet under-explored setting of FL, namely FL with _mixed-
type labels_ where _different labeling criteria_ can be employed by various
centers, leading to inter-center label space differences and challenging
existing FL methods designed for the classical setting. To effectively and
efficiently train models with mixed-type labels, we propose a theory-guided
and model-agnostic approach that can make use of the underlying correspondence
between those label spaces and can be easily combined with various FL methods
such as FedAvg. We present _convergence analysis_ based on over-parameterized
ReLU networks. We show that the proposed method can achieve linear convergence
in label projection, and demonstrate the impact of the parameters of our new
setting on the convergence rate. The proposed method is evaluated and the
theoretical findings are validated on benchmark and medical datasets.

##  1 Introduction

Federated learning (FL) enables centers to jointly learn a model while keeping
data at each center. It avoids the centralization of data which is restricted
by regulations such as CCPA (Legislature,, 2018), HIPAA (Act,, 1996), and GDPR
(Voigt et al.,, 2018) and has gained popularity in various applications. The
widely used FL methods, such as FedAvg (McMahan et al.,, 2017), FedAdam (Reddi
et al.,, 2020), and others use iterative optimization algorithms to achieve
jointly model training across centers. At each round, local center performs
stochastic gradient descent (SGD) for several steps then centers communicate
their current model weight to a central server to be aggregated.

When training a classifier in the classical FL setting, the datasets across
all centers are annotated with the same labeling criterion. However, in real
applications such as healthcare, standards for disease diagnosis may be
different across clinical centers due to varying levels of expertise or
technology available at different sites. For example, when diagnosing ADHD
with brain imaging, the labels are usually acquired over a long period of
behavior studies. Different centers may follow different diagnosis and
statistical manuals (McKeown et al.,, 2015) and it is difficult to ask centers
to relabel data using a unified criterion as some behavior studies cannot be
repeated. This leads to different label spaces across centers. In addition,
the center with the most complex labeling criterion, whose label space is
desired for future prediction, typically only has limited labeled samples due
to labeling difficulty or cost. In this paper, we aim to answer the following
important question:

With limited samples from the desired label space, how to leverage the
commonly used FL pipeline (e.g., FedAvg) and data from other centers in
different label spaces to jointly learn an FL model in the desired label
space, without additional feature exchanging and data relabeling?

Problem Setting: We study an FL problem for a given classification task. Each
center has one labeling criterion, and the criteria across centers can be
different. Samples do not overlap across centers. As shown in Fig. 1, first,
label spaces are not necessarily nested. One class from the desired space may
overlaps with different classes in another space and vice versa (e.g., ,
disease diagnoses often exhibit imperfect agreement). Second, following the
motivated healthcare example, we assume _limited_ amount of labeled data
($<5\%$) in the _desired label space_ is available 111Due to labeling
difficulties, such labels can also be noisy. Hence, we also explore this
property in our work.. Moreover, for the ease of experiment design, we
consider the case where these data are stored in one ‘specialized center,’ and
this center can be treated as the server to coordinate FL but still perform
local model updating like the other clients, i.e., the centers with the other
labeling criteria. All the centers jointly train an FL model following the
standard FL training protocol as shown in Fig. 1 (b).

Prior methods for dealing with different labels spaces include personalized FL
(Collins et al.,, 2021), but they fail to leverage the correspondence across
different label spaces. Transfer learning (Yang et al.,, 2019) which pretrains
a model on one space and finetunes the pretrained model on other spaces can be
an alternative solution in FL, but sub-optimal pretraining may lead to
negative transfer (Chen et al.,, 2019). Therefore, to address the limitation
of above methods, we want to a) simultaneously leverage different types of
labels and their correspondence and b) learn FL model end-to-end. To the best
of our knowledge, other possible centralized methods meet our needs a) and b)
are restricted to coarse to fine label spaces that have hierarchical
structures (Touvron et al.,, 2021; Chen et al., 2021a, ), which does not hold
for the general problem of our interest, or require pulling all data features
together for similarity comparison using more sophisticated training
strategies (Hu et al.,, 2022). These methods cannot be simply extended to
widely used FL methods (e.g., FedAvg) and require feature sharing across
centers which increases privacy risks.

To address the above limitations, we propose a method called FedMT, which is a
versatile strategy that can be easily combined with various FL pipelines such
as FedAvg. Specifically, we use models with the same architecture whose output
dimension is the number of classes in the desired label space across all
centers. To use client data from the other label space for supervision, we
align two spaces either with label (probability) projection that projects the
label (class scores) to the other space. We further show that our methods has
the bonus to handle label noise.

Contributions: Our contributions are three folds. Methodologically, we propose
a novel FL method, FedMT, which is a computationally efficient and versatile
solution. Theoretically, we present the convergence of FedMT in FL with over-
parameterized ReLU neural networks, and explore the impact of amount of data
from desired label space and different noise levels; Empirically, we
demonstrate the superior results in this challenging setting over prior art
with extensive experiments on benchmark and medical datasets.

Figure 1: Illustration of the problem setting and our proposed FedMT method.
(a) We consider different label spaces (i.e., desired label space
${\mathcal{Y}}$ with $K$ classes and the other space $\widetilde{\mathcal{Y}}$
with $J$ classes) where classes may overlap, such as $Y^{1}$ and
$\widetilde{Y}^{2}$. Annotation using the desired label criterion is usually
harder and more expensive to obtain, thus less such labeled samples are
available. (b) We use fixed label space correspondence matrix ${\bm{Q}}$ to
associate label spaces $\widetilde{\mathcal{Y}}$ with ${\mathcal{Y}}$ and
noise correction matrix ${\bm{T}}$ to correct label noises in ${\mathcal{Y}}$
(if any). We correct predictions by multiplying classifiers’ probability
output $f$ by projection matrices locally (FedMT (P)) or correct labels by
multiplying by the inverse of projection matrices to sample observed labels
(FedMT (L)) under FedAvg framework.

##  2 Related Work

Federated learning FL is emerging as a learning paradigm for distributed
clients that bypasses data sharing to train local models collaboratively. To
aggregate model parameters, FedAvg (McMahan et al.,, 2017) is the most widely
used approach in FL. Variants of FedAvg have been proposed to improve
optimization (Reddi et al.,, 2020; Rothchild et al.,, 2020) and for non-iid
data (Li et al.,, 2021; Li et al., 2020a, ; Karimireddy et al.,, 2020).
Recently FL methods have been studied in semi-supervised learning (Jeong et
al.,, 2021; Bdair et al.,, 2021), weakly supervised learning (Lu et al.,,
2021), and positive label only learning (Yu et al.,, 2020). Theoretical
studies of FL have not yet directly addressed neural network algorithms nor
accounted for the influence of individual data samples (Karimireddy et al.,,
2020; Li et al.,, 2019; Khaled et al.,, 2020). Recently, a theoretical
framework for FL on fully supervised neural network regression using FedAvg
was proposed in Huang et al., (2021). To the best of our knowledge, this is
the first work to investigate FL’s theory on neural network classification
under mixed-type labeled data.

Neural Tangent Kernel (NTK) NTK is an essential tool to study the learnability
of neural networks. NTK was first studied by Jacot et al., (2018) showing the
equivalence between training an infinitely wide neural network with gradient
descent and kernel regression. NTK theory has been extended to convolutional
neural networks (CNNs) (Arora et al.,, 2019), graph neural networks (GNNs) (Du
et al.,, 2019; Jiang et al.,, 2019), recurrent neural networks (RNNs)
(Alemohammad et al.,, 2020). NTK for FL was recently studied by Huang et al.,
(2021), in the supervised FL regression problem using mean squared error loss.

##  3 Methods

###  3.1 Preliminaries: Classical FL

We start by reviewing the classical FL approach, FedAvg, with full supervision
(McMahan et al.,, 2017) for classification with the same labeling criterion.
We consider a $K$-class classification problem with the feature space
${\mathcal{X}}$ and the label space ${\mathcal{Y}}=[K]$. Let
${\bm{x}}\in{\mathcal{X}}$ and $y\in{\mathcal{Y}}$ be the input and output
random variables following an underlying joint distribution with density
$p({\bm{x}},y)$. Let ${\bm{f}}:{\mathcal{X}}\to{\mathbb{R}}^{K}$ be a
$K$-class classifier such that $\sigma(f_{k}({\bm{x}}))=p(y=k|{\bm{x}})$,
where $f_{k}({\bm{x}})$ is the $k$-th element of ${\bm{f}}({\bm{x}})$ and
$\sigma(f_{k})=\exp(f_{k})/\sum_{k^{\prime}=1}^{K}\exp(f_{k^{\prime}})$ is the
softmax function. Then the predicted label can be obtained via
$y_{\rm{pred}}=\mathop{\operatorname*{arg\,max}}_{k\in[K]}f_{k}({\bm{x}})$.

In the classical FL setup, each client $c\in[C]$ has access to a labeled
training set
${\mathcal{D}}_{c}=\\{({\bm{x}}^{c}_{i},y^{c}_{i})\\}_{i=1}^{N_{c}}$ of size
$N_{c}$ and learns its local model ${\bm{f}}^{c}$ by minimizing the following
empirical risk:

$\displaystyle\widehat{R}_{c}^{\rm{a}}({\bm{f}}^{c};{\mathcal{D}}_{c})=\frac{1}{N_{c}}\sum\nolimits_{i=1}^{N_{c}}\ell({\bm{f}}^{c}({\bm{x}}^{c}_{i}),y^{c}_{i}),$
(1)

where $\ell$ is the _cross-entropy loss_ , i.e.,
$\ell_{\text{CE}}({\bm{f}}^{c}({\bm{x}}),y)=-\sum\nolimits_{k=1}^{K}\mathbbm{1}(y=k)\log
f_{k}^{c}({\bm{x}})=-\log f_{y}^{c}({\bm{x}})$ and $\mathbbm{1}(\cdot)$ is the
indicator function. The goal of classical FL is that $C$ clients
collaboratively train a global classification model ${\bm{f}}$ that
generalizes well with respect to $p({\bm{x}},y)$, without sharing their local
data ${\mathcal{D}}_{c}$. The problem can be formalized as minimizing the
aggregated risk:
$R({\bm{f}})=\frac{1}{C}\sum\nolimits_{c=1}^{C}\widehat{R}_{c}^{\rm{a}}({\bm{f}};{\mathcal{D}}_{c}).$
FedAvg employs a server to coordinate the iterative distributed training
through model parameter averaging as described in Algorithm 1.

###  3.2 Problem Formulation

In contrast to the classical supervised FL classification, where all the
centers share the same label space, we consider the case where the label
spaces across centers may be different. For the ease of presentation, we
assume there are two label spaces, namely
${\mathcal{Y}}=\\{Y^{k}\\}_{k=1}^{K}$ and
$\widetilde{\mathcal{Y}}=\\{\widetilde{Y}^{j}\\}_{j=1}^{J}$ where $K>J$.
222Our method can be easily extended to multiple label spaces. For the
‘specialized center’ with the desired label space, we denote it as a server
and its dataset as
${\mathcal{D}}^{s}=\\{({\bm{x}}_{i}^{s},y_{i}^{s}):i\in[nK]\\}$, where
$y_{i}^{s}\in{\mathcal{Y}}$. We also have in total $N$ labeled data with a
different criterion that are stored across $C$ clients. For $c\in[C]$, we
denote by $S_{c}$ the indices of data from the other label space
${\mathcal{D}}^{c}=\\{({\bm{x}}_{i}^{c},y_{i}^{c}):i\in S_{c}\\}$ on $c$-th
client, where $y_{i}^{c}\in\widetilde{\mathcal{Y}}$. Let $N_{c}=|S_{c}|$ and
we have $N=\sum_{c\in[C]}N_{c}$. Our objective is to train a global classifier
in the desired label space ${\bm{f}}:{\mathcal{X}}\mapsto{\mathcal{Y}}$ using
data in the system from different label spaces.

###  3.3 Proposed Method

Motivation Learning under different label spaces can be formulated as a
corrupted label learning problem (Van Rooyen and Williamson,, 2017; Patrini et
al.,, 2017), where label space with less classes $\widetilde{\mathcal{Y}}$ is
treated as the corrupted observation of a true underlying label space
${\mathcal{Y}}$ with more classes, i.e., our desired label space. Our method
is based on loss correction (Van Rooyen and Williamson,, 2017; Patrini et
al.,, 2017), which is a well-established corrupted label learning approach in
the centralized domain with statistically consistency guarantees.

Let ${\mathbb{T}}:{\mathcal{Y}}\mapsto\widetilde{\mathcal{Y}}$ be a linear
transformation, whose pseudo inverse is ${\mathbb{T}}^{-1}$. Under proper
assumptions (see Theorem 3.1) and a given function
${\bm{f}}:\mathcal{X}\mapsto\mathcal{Y}$, Patrini et al., (2017) showed two
ways to perform loss correction that ${\mathbb{T}}^{-1}$ can pull back from
functions of corrupted labels to functions of true labels and ${\mathbb{T}}$
can transfer functions of true labels to those of corrupted labels.
Mathematically, this can be written as:

######  Theorem 3.1 (Loss correction (informal) (Patrini et al.,, 2017; Van
Rooyen and Williamson,, 2017)).

Given a non-singular linear mapping matrix ${\mathbb{T}}$ and a proper loss
$\ell$, one can achieve the same minimizer of the original loss under the true
label distribution:

$\displaystyle\operatorname*{arg\,min}_{\bm{f}}{\mathbb{E}}_{{\bm{x}},{\bm{y}}\in{\mathcal{Y}}}\ell({\bm{y}},{\bm{f}})=\operatorname*{arg\,min}_{\bm{f}}{\mathbb{E}}_{{\bm{x}},{\bm{y}}\in\widetilde{\mathcal{Y}}}\ell({\bm{y}},{\mathbb{T}}{\bm{f}})\quad\text{and}\quad{\mathbb{E}}_{{\bm{x}},{\bm{y}}\in{\mathcal{Y}}}\ell({\bm{y}},{\bm{f}})={\mathbb{E}}_{{\bm{x}},{\bm{y}}\in\widetilde{\mathcal{Y}}}{\mathbb{T}}^{-1}\ell({\bm{y}},{\bm{f}}).$

In addition, loss correction methods are model agnostic and can be generalized
to many loss functions, therefore provide a versatile framework for learning
from corrupted labels. Since our goal is to jointly learn a classifier
${\bm{f}}:{\mathcal{X}}\to{\mathbb{R}}^{K}$ for clients and server, the
overall loss can be written as:

$R_{\text{overall}}({\bm{f}})=\frac{1}{C+1}\left\\{\widehat{R}_{s}({\bm{f}})+\sum_{c=1}^{C}\widehat{R}_{c}({\bm{f}};{\mathcal{D}}^{c})\right\\},$
(2)

where $\widehat{R}_{s}({\bm{f}})$ and $\widehat{R}_{c}({\bm{f}})$ are
respectively the empirical risk based on the server’s data and the clients’
data from a different label space. To minimize the overall loss in (2), we
need to align the predictions or losses of both types of labels to the
underlying true label space. Considering the specific form of cross-entropy
(1), we adapt the findings of Patrini et al., (2017); Van Rooyen and
Williamson, (2017) to our FL setting. For all the clients in FL $l\in[C]$, we
propose leveraging the following two kinds of projections to both server and
clients as:

$\displaystyle\text{{{Probability projection}}:
}\widehat{R}_{l}({\bm{f}};{\mathcal{D}}^{l})$
$\displaystyle=-\frac{1}{N_{l}}\sum_{i\in
S_{l}}\sum_{j=1}^{J}\mathbbm{1}(y_{i}=\widetilde{Y}^{j})\log\left\\{\sum_{k=1}^{K}{\mathbb{T}}_{jk}^{l}\sigma(f_{k}({\bm{x}}_{i}))\right\\},$
(3) $\displaystyle\text{{{Label projection}}:
}\widehat{R}_{l}({\bm{f}};{\mathcal{D}}^{l})$
$\displaystyle=-\frac{1}{N_{l}}\sum_{i\in
S_{l}}\sum_{k=1}^{K}\left\\{\sum_{j=1}^{J}{\mathbb{T}}_{kj}^{-1}\mathbbm{1}(y_{i}=\widetilde{Y}^{j})\right\\}\log\sigma(f_{k}({\bm{x}}_{i})).$
(4)

For clients, we denote ${\mathbb{T}}={\bm{Q}}\in[0,1]^{J\times K}$, whose
$j$-th row denotes the mixing weights of classes in the desired label space
for $j$-th class in the other label space. As both probability projection and
label projection are performed locally, the overall loss (2) can be optimized
using general FL strategies, such as FedAvg (McMahan et al.,, 2017) used in
this work, or other variants like Li et al., 2020a ; Li et al., (2021);
Karimireddy et al., (2020). Note our method can be easily extended to more
than one centers in the desired label space as shown in Appendix D.5.

Extension to Noisy Labels Another advantage of our method is that it has the
flexibility to extend to the case where there are noisy observations, that is
the label is mislabeled to a wrong one in the same label space. Let us explain
with the case where the server has noisy labels. Specifically, on the server,
we let ${\mathbb{T}}={\bm{T}}\in[0,1]^{K\times K}$ be a matrix whose
$(i,j)$-th element denotes $p(\widetilde{y}^{s}=Y^{j}|y^{s}=Y^{i})$ where
$\widetilde{y}^{s}$ is the observed noisy label and $y^{s}$ is the true label.

The detailed FedMT algorithm is described in Algorithm 2,
${\bm{T}}\neq\mathbf{I}^{K\times K}$ under noisy label case and
${\bm{T}}=\mathbf{I}^{K\times K}$ for noise-free setting. In this algorithm,
it is worth noting that FedMT is easy to implement as it only slightly
modifies FedAvg as highlighted in blue.

Algorithm 1 FL using FedAvg (McMahan et al.,, 2017)

Server Input: initial ${\bm{f}}$, aggregation step-size $\eta_{\rm agg}$, and
global communication round $R$
Client Input: local model ${\bm{f}}^{c}$, local dataset ${\mathcal{D}}_{c}$,
SGD step-size $\eta_{\rm sgd}$, and local updating iterations $t$ (for
$c\in[C]$)

1:For $r=1\to R$ rounds, we run A on each client and B iteratively .

2:procedure A. ModelUpdate($r$)

3: ${\bm{f}}^{c}\leftarrow{\bm{f}}$ $\triangleright$ Receive updated model
from proc. B

4: for $\tau=1\to t$ do

5:
${\bm{f}}^{c}\\!\leftarrow\\!{\bm{f}}^{c}\\!-\\!\eta_{\mathrm{sgd}}\\!\cdot\\!\nabla\widehat{R}_{c}^{\rm{a}}({\bm{f}}^{c};{\mathcal{D}}_{c})$
$\triangleright$ Model updates via SGD

6: send ${\bm{f}}^{c}-{\bm{f}}$ to proc. B

7:procedure B. ModelAgg($r$)

8: receive model updates ${\bm{f}}^{c}-{\bm{f}}$ from proc. A

9:
${\bm{f}}\leftarrow{\bm{f}}-\eta_{\mathrm{agg}}\cdot\sum\nolimits_{c=1}^{C}({\bm{f}}^{c}-{\bm{f}})$

10: broadcast ${\bm{f}}$ to proc. A

Algorithm 2 FL using FedMT (Ours)

Server Input: inputs of Alg 1, server model ${\bm{f}}^{s}$, small noisy
dataset ${\mathcal{D}}_{s}=\\{{\bm{x}}^{s},{\bm{y}}^{s}\\}$ where
${\bm{y}}^{s}\in{\mathcal{Y}}$, projection matrix $T$
Client Input: inputs of Alg 1 but
${\mathcal{D}}_{c}=\\{{\bm{x}}^{c},{\bm{y}}^{c}\\}$ where
${\bm{y}}^{c}\in\widetilde{\mathcal{Y}}$, and projection matrices ${\bm{Q}}$

1:For $r=1\to R$ rounds, we run A on each client and B iteratively .

2:procedure A.ModelUpdate($r$)

3: ${\bm{f}}^{c}\leftarrow{\bm{f}}$ $\triangleright$ Receive updated model
from proc. B

4: for $\tau=1\to t$ do

5: if probability projection then

6:
${\bm{f}}^{c}\leftarrow{\bm{f}}^{c}-\eta_{\mathrm{sgd}}\cdot\nabla\widehat{R}_{c}({\color[rgb]{0,0,1}{\bm{Q}}}{\bm{f}}^{c};{\bm{y}}^{c})$

7:
${\bm{f}}^{s}\leftarrow{\bm{f}}^{s}-\eta_{\mathrm{sgd}}\cdot\nabla\widehat{R}_{s}({\color[rgb]{0,0,1}{\bm{T}}}{\bm{f}}^{s};{\bm{y}}^{s})$

8: else if label projection then

9:
${\bm{f}}^{c}\leftarrow{\bm{f}}^{c}-\eta_{\mathrm{sgd}}\cdot\nabla\widehat{R}_{c}({\bm{f}}^{c};{\color[rgb]{0,0,1}{\bm{Q}}^{-1}}{\bm{y}}^{c})$

10:
${\bm{f}}^{s}\leftarrow{\bm{f}}^{s}-\eta_{\mathrm{sgd}}\cdot\nabla\widehat{R}_{s}({\bm{f}}^{s};{\color[rgb]{0,0,1}{\bm{T}}^{-1}}{\bm{y}}^{s})$

11: send ${\bm{f}}^{l}-{\bm{f}}$ to proc. B for $l\in\\{[C],s\\}$

12:procedure B. ModelAgg($r$)

13: receive model updates from proc. A

14:
${\bm{f}}\leftarrow{\bm{f}}-\eta_{\mathrm{agg}}\cdot\sum\nolimits_{l\in\\{[C],s\\}}({\bm{f}}^{l}-{\bm{f}})$

15: broadcast ${\bm{f}}$ to proc. A

###  3.4 Theoretical Analysis

Although label projection-based loss correction methods have shown to have
good properties under centralized setting (see Theorem 3.1 Patrini et al.,
(2017); Van Rooyen and Williamson, (2017)), their theoretical convergence
under FL have not been explored. Hence, we focus on the theoretical analysis
of label projection in (4), given it has been shown to have better theoretical
guarantees than (3) in a previous study (see Theorem 3.1 (Patrini et al.,,
2017)). Further, we are interested in investigating the impact of various
parameters on theoretical convergence. In this section, we establish a novel
theoretical analysis of FedMT using NTK (Arora et al.,, 2019; Lee et al.,,
2019; Huang et al.,, 2021).

#### NTK setup

Our theoretical results are studied under an over-parameterized one-hidden
layer neural network (NN). Let ${\bm{f}}:{\mathcal{X}}\to{\mathbb{R}}^{K}$ be
the output of the NN whose $k$th element is

$f_{k}({\bm{u}},{\bm{x}})=\frac{1}{\sqrt{M}}\sum_{m=1}^{M}a_{km}\phi({\bm{u}}_{m}^{\top}{\bm{x}})$

where $\phi(z)=\max\\{z,0\\}$ is the ReLU activation function and
${\bm{u}}=[{\bm{u}}_{1},{\bm{u}}_{2},\ldots,{\bm{u}}_{M}]\in{\mathbb{R}}^{d\times
M}$.

######  Definition 3.2 (Initialization).

We initialize ${\bm{u}}\in\mathbb{R}^{d\times M}$ and $a_{km}$ as follows. For
each $m\in[M]$, ${\bm{u}}_{m}$ is sampled from ${\cal N}(0,I)$. For each
$k\in[K]$ and $m\in[M]$, $a_{km}$ is sampled from $\\{-1,+1\\}$ uniformly at
random and is not trainable.

In the $r$-th global round, the server broadcasts the global model weight
${\bm{u}}_{m}(r)$ to every client. Each client $c$ then starts from
${\bm{u}}_{m,c}(0,r)={\bm{u}}(r)$ and takes $t$ local gradient steps

${\bm{u}}_{m,c}(\tau+1,r)={\bm{u}}_{m,c}(\tau,r)-\eta_{\mathrm{sgd}}\frac{\partial\widehat{R}_{c}({\bm{u}}_{m,c}(\tau,r))}{\partial{\bm{u}}_{m}}$

where ${\bm{u}}_{m,c}(\tau,r)$ is the value of ${\bm{u}}_{m}$ on $c$th client
at step $\tau$ in $r$-th global round. Then the client sends
$\Delta{\bm{u}}_{m,c}(r)={\bm{u}}_{m,c}(t,r)-{\bm{u}}_{m,c}(0,r)$ to server
and server computes a new ${\bm{u}}_{m,c}(r+1)$ based on the average of all
$\Delta{\bm{u}}_{m,c}(r)$ via

$\displaystyle{\bm{u}}_{m}(r+1)={\bm{u}}_{m}(r)+\eta_{\mathrm{agg}}\cdot\sum_{c\in[C]}\Delta{\bm{u}}_{m,c}(r)/C.$

#### NTK analysis results

Let ${\bm{g}}({\bm{u}},{\bm{x}})=\sigma({\bm{f}}({\bm{u}},{\bm{x}}))$ and
${\bm{y}}_{i}^{c}$ be the multi-hot vector representation of $y_{i}^{c}$ and
${\bm{y}}_{i}^{s}$ be the one-hot vector representation of $y_{i}^{s}$ in
${\mathbb{R}}^{K}$. Let ${\bm{Q}}$ be the linear mapping from ${\mathcal{Y}}$
to $\widetilde{\mathcal{Y}}$ and ${\bm{T}}$ be the same as before. Following
Neural Tangent Kernel (NTK) analysis (Lee et al.,, 2019; Arora et al.,, 2019),
we consider the mean squared error loss333which can be easily extended to the
label projection loss respectively on the client and server

$\begin{split}&\widehat{R}_{c}({\bm{u}})=\frac{1}{N_{c}}\sum_{i\in
S_{c}}\sum_{j=1}^{J}\sum_{k=1}^{K}Q_{jk}^{-1}(y_{ik}^{c}-g_{k}({\bm{u}},{\bm{x}}_{i}^{c}))^{2},\\\
&\widehat{R}_{s}({\bm{u}})=\frac{1}{nK}\sum_{i=1}^{nK}\sum_{k^{\prime}=1}^{K}\sum_{k=1}^{K}T_{k^{\prime}k}^{-1}(y_{ik}^{s}-g_{k}({\bm{u}},{\bm{x}}_{i}^{s}))^{2}.\\\
\end{split}$ (5)

######  Lemma 3.3.

Let the notation be the same as before, the NTK kernel ${\bm{G}}(r)$ based on
our proposed novel loss (LABEL:eq:lam_loss) is a block matrix with $K$ row
partitions and $K$ column partitions. The block matrix in the $l$-th row and
$m$-th column has the following form

$\displaystyle{\bm{G}}^{l,m}(r)=\begin{pmatrix}{\mathcal{G}}_{1,1}^{l,m}(r)&{\mathcal{G}}_{1,2}^{l,m}(r)&\ldots&{\mathcal{G}}_{1,C}^{l,m}(r)&{\mathcal{G}}_{1,s}^{l,m}(r)\\\
{\mathcal{G}}_{2,1}^{l,m}(r)&{\mathcal{G}}_{2,2}^{l,m}(r)&\ldots&{\mathcal{G}}_{2,C}^{l,m}(r)&{\mathcal{G}}_{2,s}^{l,m}(r)\\\
\vdots&\vdots&\ddots&\vdots&\vdots\\\
{\mathcal{G}}_{C,1}^{l,m}(r)&{\mathcal{G}}_{C,2}^{l,m}(r)&\ldots&{\mathcal{G}}_{C,C}^{l,m}(r)&{\mathcal{G}}_{C,s}^{l,m}(r)\\\
{\mathcal{G}}_{s,1}^{l,m}(r)&{\mathcal{G}}_{s,2}^{l,m}(r)&\ldots&{\mathcal{G}}_{s,C}^{l,m}(r)&{\mathcal{G}}_{s,s}^{l,m}(r)\\\
\end{pmatrix}$ (6)

for $l\in[K]$ and $m\in[K]$ where each sub-block matrix has the following
form:

$\displaystyle{\mathcal{G}}^{l,m}_{c,j}(t)$
$\displaystyle=\left\\{\sum_{j}Q_{jm}^{-1}\right\\}\nabla_{{\bm{u}}}g_{l}({\bm{u}}(t),{\mathcal{D}}^{c})\nabla_{{\bm{u}}}^{\top}g_{m}({\bm{u}}(t),{\mathcal{D}}^{j}),~{}\forall
j\in\\{[C],s\\},c\in[C]$ (7) $\displaystyle{\mathcal{G}}^{l,m}_{s,s}(t)$
$\displaystyle=\left\\{\sum_{k}T_{km}^{-1}\right\\}\nabla_{{\bm{u}}}g_{l}({\bm{u}}(t),{\mathcal{D}}^{s})\nabla_{{\bm{u}}}^{\top}g_{m}({\bm{u}}(t),{\mathcal{D}}^{s}).$

We show detailed derivation of Lemma 3.3. Note that as $J<K$,
${\bm{Q}}\in{\mathbb{R}}^{J\times K}$ is not inevitable. In practice, we
compute its pseudo inverse instead. Although looking at ${\bm{Q}}$ alone, the
plausible property of Theorem 3.1 may not hold. We claim that by optimizing
$\widehat{R}_{c}({\bm{f}})$ together with $\widehat{R}_{s}({\bm{f}})$, we
benefit from the gradient of labels in the desired label space, as shown in
the expression of $\mathcal{G}_{c,s}^{i,j}(r)$ in Eq. (7).

Following Huang et al., (2021), we further conclude the convergence of the our
proposed loss in (LABEL:eq:lam_loss).

######  Theorem 3.4 (Convergence).

Let $M=\Omega(\lambda^{-4}(N+nK)^{4}\log((N+nK)/\delta))$, we iid initialize
${\bm{u}}_{m}(0)$, $a_{km}$ as Definition 3.2. Let
$\lambda=\lambda_{\min}({\bm{G}}(0))$ denote the smallest eigenvalue of
${\bm{G}}(0)$. Let $\kappa$ denote the condition number of ${\bm{G}}(0)$. For
$C$ clients, for any $\epsilon$, let

$\displaystyle
R=O\left(\frac{C}{\lambda\eta_{\mathrm{sgd}}\eta_{\mathrm{agg}}t}\cdot\log(1/\epsilon)\right),$

$\eta_{\mathrm{sgd}}=O\left(\lambda/\kappa L(N+nK)^{2}\right)$, and
$\eta_{\mathrm{agg}}=O(1)$, the above algorithm satisfies

${\mathcal{L}}({\bm{u}}(r))\leq\\\
\left(1-\frac{\eta_{\mathrm{agg}}\eta_{\mathrm{sgd}}\lambda
t}{2C}\right)^{r}{\mathcal{L}}({\bm{u}}(0)).$

with probability at least $1-\delta$.

The details of the proof are deferred to Appendix B. Note that a smaller value
of $\eta_{\mathrm{agg}}\eta_{\mathrm{sgd}}\lambda t/2C$, e.g., a smaller
eigenvalue $\lambda$ and a larger number of clients, leads to slower
convergence. It is clear that $\lambda$ depends on both ${\bm{Q}}$ and
${\bm{T}}$, therefore it is important to see its connection with ${\bm{Q}}$
and ${\bm{T}}$, we discuss the algorithm convergence under the following two
special cases.

######  Corollary 3.5 (Convergence under various label granularity
differences).

Let $k_{1},k_{2},\ldots,k_{J}$ be $J$ non-negative integers such that
$K=\sum_{j=1}^{J}k_{j}$. If the linear transformation from desired label space
to the other label space is
${\bm{Q}}=\text{diag}(\mathbf{1}_{k_{1}}^{\top},\mathbf{1}_{k_{2}}^{\top},\ldots,\mathbf{1}_{k_{J}}^{\top})\in[0,1]^{J\times
K}$ (i.e., the classes in the desired label space are subclasses of classes in
the other label space), then the smaller the value of $J$, the lower the
convergence rate.

Proof Sketch: Based on the form of ${\bm{G}}(r)$ in (7), it can be seen that
(Li et al.,, 2021)

$\lambda=\lambda_{\min}({\bm{G}}(0))\leq\min_{k,c,s}\\{\lambda_{\min}({\mathcal{G}}_{c,c}^{k,k}(0)),\lambda_{\min}({\mathcal{G}}_{s,s}^{k,k}(0))\\}\leq\lambda_{\min}({\mathcal{G}}_{c,c}^{k,k}(0))\lesssim\left\\{\min_{j}k_{j}^{-1}\right\\}\lambda_{0}$

where
$\lambda_{0}=\lambda_{\min}(\nabla_{{\bm{u}}}{\bm{g}}({\bm{u}}(0),{\mathcal{D}})\nabla_{{\bm{u}}}^{\top}{\bm{g}}({\bm{u}}(0),{\mathcal{D}}))$.
Since $\sum_{j}k_{j}=K$, the smaller the value of $J$, the smaller the value
of $\lambda$ and hence the algorithm converges slower. See detailed proof in
Appendix B.

######  Corollary 3.6 (Convergence under different noise levels).

We assume noisy labels are corrupted via

${\bm{T}}=\left(1-K/(K-1)\xi\right){\mathbb{I}}_{K}+\xi/(K-1)\mathbbm{1}_{K}\mathbbm{1}_{K}^{\top},$

where
$T_{ij}=p(\widetilde{y}^{s}=Y_{{\mathcal{F}}}^{j}|y^{s}=Y_{{\mathcal{F}}}^{i})$
with observed noisy label $\widetilde{y}^{s}$ and true $y^{s}$ is the true
label, and $\xi=1-T_{ii}\in(0,1)$ for $i\in[K]$ denotes the noisy level. In
this specified case, the convergence of FedMT with label projection on NTK is
does not depend on the noise level $\xi$.

Proof Sketch: By using matrix inversion lemma, we find
${\bm{T}}^{-1}=(K-1)/(K-1-K\xi){\mathbb{I}}_{K}-\xi/(K-1-K\xi)\mathbbm{1}_{K}\mathbbm{1}_{K}^{\top}$.
Hence $\sum_{k=1}^{K}T_{kk^{\prime}}=1$ for any $k^{\prime}\in[K]$. See
Appendix B for details.

##  4 Experiments

In this section, we demonstrate the effectiveness of local loss correction to
jointly train a model with data from different label spaces. Compared with
alternative training strategies and prior art, our proposed method, FedMT,
consistently achieves better classification accuracy in predicting labels in
the desired label space with limited amount of data from this space as
demonstrated on CIFAR100 (Krizhevsky et al.,, 2009) and a medical dataset.

###  4.1 Benchmark Experiments Setup

Dataset and setting We use the CIFAR100 (Krizhevsky et al.,, 2009) dataset to
mimic our proposed problem setting. We assume one center (as server to
coordinate FL training) has a small amount of data ($\leq 5\%$ of total sample
size) with sub-class annotations and other centers are labeled using super-
class annotations. The sub-class space is viewed as our desired label space.
Our objective is to train a classification model using FL to predict sub-class
labels with all the centers simultaneously.

The CIFAR100 dataset consists of 50K images, associated with $K=100$ sub-
classes that could be further grouped into $J=20$ super-classes. To study the
effect of the number of training samples in desired label space, we randomly
select $n$ observations from each of the $100$ sub-classes on the server and
the rest of the observations in the training set are split into $C=10$ subsets
completely at random, ensuring that each class is equally represented in each
subset. Each subset corresponds to a dataset stored on one client and we use
$N_{c}=4000$. We use the super-class as our label for all clients. Experiment
results with other values of $C$ and $N_{c}$ is given in Appendix D.

We use ResNet18 (He et al.,, 2016) as our classifier. We use SGD optimizer
Ruder, (2016) with a learning rate of $10^{-2}$, momentum $0.9$, and weight
decay $5\times 10^{-4}$. The learning rate is divided by $5$ at $20$, $30$,
and $40$ epochs. If not specified, our default setting for local update epochs
is $E=1$. Based on the superclass information in CIFAR100, we are able to
formulate the transition matrix ${\bm{Q}}$ for all clients. Its form and more
training details are available in Appendix C.1.

Baselines We compare the following learning approaches: 1) Single: only the
data from the desired label space is used to train the classifier; 2) FedMatch
(Jeong et al.,, 2021): we treat client samples as unlabeled data, perform
pseudo-labeling on the unlabeled sets, and augment samples in supervised loss
training. 3) FedRep (Collins et al.,, 2021): clients train a $J$-class
classifier and the server trains a $K$-class classifier that differs in the
last layer with different output dimension. These classifiers share the same
backbone parameters using FedAvg, the performance is tested using the
classifier on the server; 4) FedTrans (Chen et al., 2021b, ): all clients
trains a $J$-class classifier using FedAvg which is later fine-tuned on server
with a new $K$-class linear layer. We refer FedMT (P) as our proposed method
with probability projection loss; and refer FedMT (L) as our proposed method
with label projection loss.

###  4.2 Benchmark Experiments Results

Comparison under different amount of data on server We fix number of clients
$C$ at $10$, and set different number of samples per-class $n$ on the server
in the range of {5, 10, 15, 20, 25}. The advantage of FL is to enable access
to more data to improve model performance. As shown in Tab. 1, when $n$
increases (supervision information increases), the accuracy of all approaches
improves. Both FedMT (P & L) significantly outperform alternative methods when
supervision is limited $n<25$. We achieve similar but much more stable
performance compared with FedMatch when $n=25$. FedMatch adaptively augments
the training data with labels in the desired label space using heuristically
derived confident pseudo labels. Future work could combine FedMT with similar
approaches used in FedMatch to evaluate heuristic results in a theoretically
provable framework.

Table 1: Comparison of the accuracy for 100-class classification using our
methods and alternative methods on the CIFAR100 benchmark dataset with
different per sub-class number $n$ images on the server. We report mean (sd)
from three trial runs. The best method is highlighted in boldface.

$n$ | Single | FedMatch | FedRep | FedTrans |  Ours: FedMT (P & L)
---|---|---|---|---|---
5 | 8.56(0.33) | 13.20(0.14) | 16.89(0.74) | 21.31(1.27) |  21.68(0.67) | 20.56(0.42)
10 | 12.29(0.32) | 18.02(0.25) | 21.90(1.27) | 24.53(1.40) |  26.44(0.34) | 23.89(0.13)
15 | 14.34(0.15) | 23.44(0.08) | 23.39(0.67) | 26.21(0.95) |  28.25(0.34) | 25.74(0.59)
20 | 16.23(0.21) | 27.70(0.09) | 24.70(0.55) | 27.49(0.85) |  29.75(0.66) | 27.43(0.35)
25 | 17.73(0.20) | 30.71(0.59) | 26.53(0.29) | 28.16(1.30) | 30.63(0.06) | 28.53(0.52)

Analysis on convergence To validate our developed theoretical results in
Theorem 3.4, we investigate the convergence rate of loss (2) of FedMT with
label projection. In Fig. 2, we explore loss as a function of the number of
classes $J\in\\{5,10,20\\}$ with $C=10$ under noise-free setting, and the
convergence is consistent with Corollary 3.5, smaller $J$ leads to lower
convergence. The loss curves in Fig. 2 show FedMT convergence was achieved
faster when training with more clients for a fixed total number of samples.
This result is consistent with the theoretical result of Theorem 3.4. As shown
in Fig. 2, under the default setting, the convergence rates are similar at
different noise levels. The results demonstrate our method is reasonably
stable to noise, which is in line with Corollary 3.6.

Figure 2: Ablation studies on CIFAR100. (a) Effects of the number of classes
$J$ on convergence. (b) Effects of the number of clients on convergence. (c)
Effects of noise level $\xi$ on convergence. (d) Study of performance accuracy
with different noise levels on the server. All the convergence results are
shown on the first 50 rounds using FedMT (L) for better visualization.

Effect of varying noise levels Our method has the flexibility to adapt to the
case where the server data is noisy. We use the accuracy of $K$-class
prediction in the held out test set to compare our method with other
approaches in CIFAR100, the results are reported in Fig. 2. Given the
mislabeling rate in many real applications (e.g., healthcare) could be as high
as $40\%$ (Lettieri et al.,, 2005), we compare FedMT with other learning
approaches, by varying the noise level $\xi$ on the server from {0, 0.1, 0.2,
0.3, 0.4} and set $n=10$ on the server.444We assume the labels are sampled
from known, systemic, and instance-independent noisy transition matrix $T$,
whose diagonal values are $1-\xi$. Our method can be extended to estimate $T$
using anchor point-based methods (Scott,, 2015; Liu and Tao,, 2015) but
discussing the estimation method is not in the scope of this work.. From the
results, our proposed FedMT method significantly outperforms all alternative
methods both even when the noise level $\xi$ is up to $0.4$. Second, when
increasing noise levels, FedMT shows more stable performance compared to
baselines. Our method also shows stable performance if we have more than 1
client has observations from desired label space and the results is given in
Appendix 10.

###  4.3 Tremor Severity Prediction of Parkinson’s Disease

Dataset Our method is not limited to sub- and super-classes as in CIFAR100, to
illustrate the effectiveness of our method under the general overlapping class
case, we evaluate FedMT for the prediction of tremor severity for patients
with Parkinson’s disease (PD). Tremor is a typical movement disorder occurring
on the limbs of patients with PD (Qin et al.,, 2020). Surface electromyography
(sEMG) is widely used for movement disorder assessment by noninvasively
recording electrical signals on the skin. The patterns of sEMG signals can be
used to turn PD diagnosis into different subtypes based on the diagnostic
values. We use the synthetic sEMG dataset in Qin et al., (2020). In our
experiment, we consider $K=5$ on the server and $J=3$ on the client, according
to MDS-UPDRS19. Goetz et al., (2008) 555The labeling strategies are detailed
in Appendix C.2 In this dataset, the classes in two label spaces has overlap.
The form of ${\bm{Q}}$ is given in Appendix C.2.

Setup We randomly sample $9000$ observations from the sEMG dataset as client
training data. We hold out $1000$ observations for testing. We then split the
training set to $C=50$ clients completely at random, but ensuring that samples
are balanced within each class in both label spaces. Each client, therefore,
has roughly $60$ observations from one of the three classes. In addition to
the training set on clients, we also have $n$ observations per class on the
server. Similar to the benchmark experiment, we vary $n$ on the server.
Following Qin et al., (2019), we use $12$ summary statistics of sEMG as our
features: mean absolute value (MAV), mean square value (MSV), root mean square
(RMS), variance (VAR), standard deviation (STD), waveform length (WL),
Willison amplitude (WAMP), log detector (LOG), slope sign change (SSC), zero
crossing (ZC), mean spectral frequency (MSF) and median frequency (MF). The
mathematical definition of these features is given in Chowdhury et al.,
(2013). We then use a single hidden layer multiple layer perceptron (MLP) with
$128$ hidden units and ReLU activation function as our backbone. We use SGD
optimizer with learning rate as $10^{-3}$, and set batch size as 16. Local
update frequency is 1 epoch. We train the model for 100 communication rounds.
More training details are available in Appendix C.2. We use the same
performance measure and the same alternative approaches as those described in
Section 4.1 for comparison.

Table 2: Comparison of the accuracy for 5-class classification using our
methods and alternative methods on the sEMG dataset with various $n$ and
$C=50$ under noise-free setting. We report mean (sd) from three trial runs.
The best method is highlighted in boldface.

$n$ | Single | FedMatch | FedRep | FedTrans |  Ours: FedMT (P & L)
---|---|---|---|---|---
1 | 27.30(0.88) | 28.20(0.67) | 39.34(1.12) | 42.31(2.12) |  66.89(0.91) | 63.67(0.28)
2 | 30.83(1.30) | 31.90(1.45) | 40.49(1.16) | 46.45(0.63) |  67.00(0.38) | 64.22(0.28)
3 | 34.66(1.09) | 32.90(1.74) | 41.29(1.16) | 51.25(2.57) |  67.34(0.57) | 64.33(0.48)
4 | 39.11(1.48) | 33.53(2.12) | 43.24(0.68) | 51.81(1.99) |  67.67(0.64) | 64.44(0.35)
5 | 42.57(0.78) | 37.20(2.09) | 44.24(0.74) | 51.83(2.14) |  67.22(0.30) | 64.44(0.55)

Results Overall, our quantitative results in Tab. 2 show that FedMT with
probability projection performs better than probability projection on the sEMG
dataset and consistently outperforms alternative learning approaches. FedMT
improves mean accuracy by a non-negligible margin, demonstrating a synergistic
collaborative effect with clients’ data from the other label space. The
superiority of FedMT in this demanding medical data setting further
demonstrates the efficacy and robustness of our algorithm. To test of the
performance of FedMT at different degree of overlapping, we also conduct
experiments when $K=10$ with various noisy levels, the results is given in
Appendix E. Our method also outperforms other methods in this case.

##  5 Conclusion

In this work, we propose a new FL framework, FedMT, to address an important
yet under-explored mixed-type label setting. Theoretically, we provide the
convergence guarantee of FedMT with the extension of NTK. Through extensive
experiments on a benchmark and a medical dataset, we demonstrate FedMT can
outperform alternative methods. We also emphasize that since FedMT makes minor
modifications to local predictions or labels, it has much more flexibility to
integrate with other FL strategies beyond FedAvg. The performance of our
proposed method can be further improved by combining our provable method with
other heuristic-based weakly supervised learning approaches. As a plug-and-
play method, FedMT can be applied to non-IID setting by combining with
advanced FL schemes, including Li et al., 2020b ; Li et al., (2021);
Karimireddy et al., (2020).

## References

  * Act, (1996) Act, A. (1996).  Health insurance portability and accountability act of 1996.  Public law, 104:191.
  * Alemohammad et al., (2020) Alemohammad, S., Wang, Z., Balestriero, R., and Baraniuk, R. (2020).  The recurrent neural tangent kernel.  arXiv preprint arXiv:2006.10246.
  * Arora et al., (2019) Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. (2019).  On exact computation with an infinitely wide neural net.  Advances in Neural Information Processing Systems, 32.
  * Bdair et al., (2021) Bdair, T., Navab, N., and Albarqouni, S. (2021).  Fedperl: Semi-supervised peer learning for skin lesion classification.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 336–346. Springer.
  * (5) Chen, J., Niu, L., Liu, L., and Zhang, L. (2021a).  Weak-shot fine-grained classification via similarity transfer.  Advances in Neural Information Processing Systems, 34:7306–7318.
  * Chen et al., (2019) Chen, X., Wang, S., Fu, B., Long, M., and Wang, J. (2019).  Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning.  Advances in Neural Information Processing Systems, 32.
  * (7) Chen, Y., Liu, Z., Xu, H., Darrell, T., and Wang, X. (2021b).  Meta-baseline: Exploring simple meta-learning for few-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9062–9071.
  * Chowdhury et al., (2013) Chowdhury, R. H., Reaz, M. B., Ali, M. A. B. M., Bakar, A. A., Chellappan, K., and Chang, T. G. (2013).  Surface electromyography signal processing and classification techniques.  Sensors, 13(9):12431–12466.
  * Collins et al., (2021) Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. (2021).  Exploiting shared representations for personalized federated learning.  In International Conference on Machine Learning, pages 2089–2099. PMLR.
  * Du et al., (2019) Du, S. S., Hou, K., Salakhutdinov, R. R., Poczos, B., Wang, R., and Xu, K. (2019).  Graph neural tangent kernel: Fusing graph neural networks with graph kernels.  Advances in neural information processing systems, 32.
  * Goetz et al., (2008) Goetz, C. G., Tilley, B. C., Shaftman, S. R., Stebbins, G. T., Fahn, S., Martinez-Martin, P., Poewe, W., Sampaio, C., Stern, M. B., Dodel, R., et al. (2008).  Movement disorder society-sponsored revision of the unified parkinson’s disease rating scale (MDS-UPDRS): scale presentation and clinimetric testing results.  Movement disorders: official journal of the Movement Disorder Society, 23(15):2129–2170.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  * Hu et al., (2022) Hu, A., Sun, Z., and Li, Q. (2022).  Weakly supervised fine-grained recognition based on combined learning for small data and coarse label.  In Proceedings of the 2022 International Conference on Multimedia Retrieval, pages 194–201.
  * Huang et al., (2021) Huang, B., Li, X., Song, Z., and Yang, X. (2021).  FL-NTK: A neural tangent kernel-based framework for federated learning analysis.  In International Conference on Machine Learning, pages 4423–4434. PMLR.
  * Jacot et al., (2018) Jacot, A., Gabriel, F., and Hongler, C. (2018).  Neural tangent kernel: Convergence and generalization in neural networks.  Advances in neural information processing systems, 31.
  * Jeong et al., (2021) Jeong, W., Yoon, J., Yang, E., and Hwang, S. J. (2021).  Federated semi-supervised learning with inter-client consistency & disjoint learning.  In International Conference on Learning Representations.
  * Jiang et al., (2019) Jiang, Y., Konečnỳ, J., Rush, K., and Kannan, S. (2019).  Improving federated learning personalization via model agnostic meta learning.  arXiv preprint arXiv:1909.12488.
  * Kairouz et al., (2021) Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021).  Advances and open problems in federated learning.  Foundations and Trends® in Machine Learning, 14(1–2):1–210.
  * Karimireddy et al., (2020) Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020).  Scaffold: Stochastic controlled averaging for federated learning.  In International Conference on Machine Learning, pages 5132–5143. PMLR.
  * Khaled et al., (2020) Khaled, A., Mishchenko, K., and Richtárik, P. (2020).  Tighter theory for local sgd on identical and heterogeneous data.  In International Conference on Artificial Intelligence and Statistics, pages 4519–4529. PMLR.
  * Krizhevsky et al., (2009) Krizhevsky, A., Hinton, G., et al. (2009).  Learning multiple layers of features from tiny images.  Technical report, University of Toronto.
  * Lee et al., (2019) Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. (2019).  Wide neural networks of any depth evolve as linear models under gradient descent.  Advances in neural information processing systems, 32.
  * Legislature, (2018) Legislature, C. S. (2018).  California consumer privacy act (ccpa).  https://oag.ca.gov/privacy/ccpa.
  * Lettieri et al., (2005) Lettieri, C. J., Veerappan, G. R., Parker, J. M., Franks, T. J., Hayden, D., Travis, W. D., and Shorr, A. F. (2005).  Discordance between general and pulmonary pathologists in the diagnosis of interstitial lung disease.  Respiratory medicine, 99(11):1425–1430.
  * (25) Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020a).  Federated optimization in heterogeneous networks.  Proceedings of Machine Learning and Systems, 2:429–450.
  * (26) Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020b).  Federated optimization in heterogeneous networks.  Proceedings of Machine Learning and Systems, 2:429–450.
  * Li et al., (2019) Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. (2019).  On the convergence of fedavg on non-iid data.  arXiv preprint arXiv:1907.02189.
  * Li et al., (2021) Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. (2021).  FedBN: Federated learning on non-iid features via local batch normalization.  In International Conference on Learning Representations.
  * Liu and Tao, (2015) Liu, T. and Tao, D. (2015).  Classification with noisy labels by importance reweighting.  IEEE Transactions on pattern analysis and machine intelligence, 38(3):447–461.
  * Lu et al., (2021) Lu, N., Wang, Z., Li, X., Niu, G., Dou, Q., and Sugiyama, M. (2021).  Unsupervised federated learning is possible.  In International Conference on Learning Representations.
  * McKeown et al., (2015) McKeown, R. E., Holbrook, J. R., Danielson, M. L., Cuffe, S. P., Wolraich, M. L., and Visser, S. N. (2015).  The impact of case definition on attention-deficit/hyperactivity disorder prevalence estimates in community-based samples of school-aged children.  Journal of the American Academy of Child & Adolescent Psychiatry, 54(1):53–61.
  * McMahan et al., (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017).  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR.
  * Patrini et al., (2017) Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. (2017).  Making deep neural networks robust to label noise: A loss correction approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944–1952.
  * Qin et al., (2020) Qin, Z., Chen, J., Jiang, Z., Yu, X., Hu, C., Ma, Y., Miao, S., and Zhou, R. (2020).  Learning fine-grained estimation of physiological states from coarse-grained labels by distribution restoration.  Scientific Reports, 10(1):1–10.
  * Qin et al., (2019) Qin, Z., Jiang, Z., Chen, J., Hu, C., and Ma, Y. (2019).  sEMG-based tremor severity evaluation for Parkinson’s disease using a light-weight CNN.  IEEE Signal Processing Letters, 26(4):637–641.
  * Reddi et al., (2020) Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečnỳ, J., Kumar, S., and McMahan, H. B. (2020).  Adaptive federated optimization.  arXiv preprint arXiv:2003.00295.
  * Rothchild et al., (2020) Rothchild, D., Panda, A., Ullah, E., Ivkin, N., Stoica, I., Braverman, V., Gonzalez, J., and Arora, R. (2020).  Fetchsgd: Communication-efficient federated learning with sketching.  In International Conference on Machine Learning, pages 8253–8265. PMLR.
  * Ruder, (2016) Ruder, S. (2016).  An overview of gradient descent optimization algorithms.  arXiv preprint arXiv:1609.04747.
  * Scott, (2015) Scott, C. (2015).  A rate of convergence for mixture proportion estimation, with application to learning from noisy labels.  In Artificial Intelligence and Statistics, pages 838–846. PMLR.
  * Touvron et al., (2021) Touvron, H., Sablayrolles, A., Douze, M., Cord, M., and Jégou, H. (2021).  Grafit: Learning fine-grained image representations with coarse labels.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 874–884.
  * Van Rooyen and Williamson, (2017) Van Rooyen, B. and Williamson, R. C. (2017).  A theory of learning with corrupted labels.  J. Mach. Learn. Res., 18(1):8501–8550.
  * Voigt et al., (2018) Voigt, P. et al. (2018).  The EU general data protection regulation (GDPR).  Intersoft consulting.
  * Yang et al., (2019) Yang, Q., Liu, Y., Chen, T., and Tong, Y. (2019).  Federated machine learning: Concept and applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19.
  * Yu et al., (2020) Yu, F., Rawat, A. S., Menon, A., and Kumar, S. (2020).  Federated learning with only positive labels.  In International Conference on Machine Learning, pages 10946–10956. PMLR.

##  Appendix A Notation Table

Table 3: Important notations used in the paper.

Notations | Description
---|---
$d$ | input dimension
$n$ | number of per desired label space class samples on the server
$t$ | local updating iterations
$C$ | number of clients in the FL, we index client by $c\in[C]$
$J$ | number of classes in the other label space
$K$ | number of classes in the desired label space
${\bm{T}}$ | linear transformation of noisy labels on the server
${\bm{Q}}$ | linear transformation between two label spaces
$R$ | aggregation communication rounds
$N_{c}$ | size of samples on the $c$-th client
$S_{c}$ | the set of indices of observations on $c$-th client
${\mathcal{X}}$ | input space ${\mathcal{X}}\subset\mathbb{R}^{d}$
$\widetilde{\mathcal{Y}}$ | the other label space $\widetilde{\mathcal{Y}}=\\{\widetilde{Y}^{1},\widetilde{Y}^{2},\ldots,\widetilde{Y}^{J}\\}$
${\mathcal{Y}}$ | desired label space ${\mathcal{Y}}=\\{Y^{1},Y^{2},\ldots,Y^{K}\\}$
${\mathcal{D}}^{c}$ | dataset $\mathcal{D}_{c}=\\{({\bm{x}}_{i}^{c},y_{i}^{c}):i\in S_{c}\\}$ on $c$-th client
${\mathcal{D}}^{s}$ | dataset on server ${\mathcal{D}}^{s}=\\{({\bm{x}}_{i}^{s},y_{i}^{s}):i\in[nK]\\}$ where $y_{i}^{s}\in{\mathcal{Y}}_{\text{fine}}$
${\bm{x}}$ | inputs variable ${\bm{x}}\in\mathcal{X}$
${\bm{f}}(\cdot)$ | a classification model that assigns a score to each of the $K$ classes for a given input ${\bm{x}}$
$\ell(\cdot,\cdot)$ | loss function for the classification task
$\widehat{R}_{c}(\cdot)$ | empirical risk on $c$-th client
$\widehat{R}_{s}(\cdot)$ | empirical risk on the server
$\eta_{\text{sgd}}$ | local SGD step size
$\eta_{\text{agg}}$ | aggregation step size
$\xi$ | the noise level in ${\bm{T}}$

##  Appendix B Technical Details of the Neural Tangent Kernel Analysis

In this section, we present the technical details of the Neural Tangent Kernel
Analysis on the convergence of the algorithm of the label projection approach.
We first present the NTK analysis of a simple regression model with weighted
loss. We then show how the conclusion be extended to our classification case
under FL.

###  B.1 NTK with Weighted MSE Loss for Regression

Let us first consider an over-parameterized one-hidden layer neural network
(NN) for regression. Let $f:{\mathcal{X}}\to{\mathbb{R}}$ be the output of the
NN with the form

$f({\bm{u}},{\bm{x}})=\frac{1}{\sqrt{M}}\sum_{m=1}^{M}a_{m}\phi({\bm{u}}_{m}^{\top}{\bm{x}})$

where $\phi(z)=\max\\{z,0\\}$ is the ReLU activation function and
${\bm{u}}=[{\bm{u}}_{1},{\bm{u}}_{2},\ldots,{\bm{u}}_{M}]\in{\mathbb{R}}^{d\times
M}$. Let ${\mathcal{D}}=\\{({\bm{x}}_{i},y_{i}):i\in[n]\\}$ be a set of
training examples. We consider the learning $f$ under the following weighted
MSE loss

${\mathcal{L}}({\bm{u}})=\frac{1}{2}\sum_{i=1}^{n}w_{i}(f({\bm{u}},{\bm{x}}_{i})-y_{i})^{2}.$

Let $\eta$ be the learning rate of gradient descent algorithm, the evolution
of the parameters ${\bm{u}}$ and the $f$ via continuous time gradient descent
satisfies

$\begin{split}&\frac{df({\bm{u}}(t),{\bm{x}}_{i})}{dt}=\frac{\partial
f({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{d{\bm{u}}(t)}{dt},\\\
&\frac{d{\bm{u}}(t)}{dt}=-\eta\sum_{i=1}^{n}\frac{\partial
f({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}}\frac{\partial{\mathcal{L}}}{\partial
f({\bm{u}}(t),{\bm{x}}_{i})}\\\ \end{split}$ (8)

based on the chain rule.

Let
$f({\bm{u}},{\mathcal{D}})=(f({\bm{u}},{\bm{x}}_{1}),f({\bm{u}},{\bm{x}}_{2}),\ldots,f({\bm{u}},{\bm{x}}_{n}))^{T}$.
Given the weighted MSE loss, we have

$\frac{\partial{\mathcal{L}}}{\partial
f({\bm{u}},{\mathcal{D}})}=(w_{1}(f({\bm{u}},{\bm{x}}_{1})-y_{1}),w_{2}(f({\bm{u}},{\bm{x}}_{2})-y_{2}),\ldots,w_{n}(f({\bm{u}},{\bm{x}}_{n})-y_{n}))^{\top}$

and

$\frac{\partial
f({\bm{u}},{\mathcal{D}})}{\partial{\bm{u}}}=\left(\frac{\partial
f({\bm{u}},{\bm{x}}_{1})}{\partial{\bm{u}}^{\top}},\frac{\partial
f({\bm{u}},{\bm{x}}_{2})}{\partial{\bm{u}}^{\top}},\ldots,\frac{\partial
f({\bm{u}},{\bm{x}}_{n})}{\partial{\bm{u}}^{\top}}\right)^{\top}.$

Hence, we get

$\frac{df({\bm{u}},{\bm{x}}_{i})}{dt}=-\eta\sum_{j=1}^{n}w_{j}\frac{\partial
f({\bm{u}},{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{\partial
f({\bm{u}},{\bm{x}}_{j})}{\partial{\bm{u}}}\\{f({\bm{u}},{\bm{x}}_{j})-y_{j}\\}.$

Then the full evolution dynamic of learning $f$ is

$\frac{df({\bm{u}}(t),{\mathcal{D}})}{dt}=-\eta\bm{\Lambda}(t)(f({\bm{u}}(t),{\mathcal{D}})-{\bm{y}})$

where ${\bm{y}}=(y_{1},y_{2},\ldots,y_{n})^{\top}$, the Gram matrix
$\bm{\Lambda}(t)\in{\mathbb{R}}^{n\times n}$ is

$\bm{\Lambda}(t)=\nabla_{{\bm{u}}}f({\bm{u}}(t),{\mathcal{D}})W\nabla_{{\bm{u}}}^{\top}f({\bm{u}}(t),{\mathcal{D}}),$

and $W=\text{diag}(w_{1},w_{2},\ldots,w_{n})$.

###  B.2 NTK under Weighted MSE Loss for Classification

With the preparation in Section B.1, we now show the NTK kernel under the
weighted MSE loss for classification. We first consider the centralized
version and then extend the conclusions to the federated learning case.

Centralized Classification To make the presentation easier, we first introduce
some notations. Let ${\mathcal{D}}=\\{({\bm{x}}_{i},{\bm{y}}_{i}):i\in[n]\\}$
be a set of training examples where ${\bm{y}}_{i}$ is the one-hot
representation of the label. Let
${\bm{f}}:{\mathcal{X}}\mapsto{\mathbb{R}}^{K}$ be a $K$-class classifier and
${\bm{g}}({\bm{u}},{\bm{x}})=\sigma({\bm{f}}({\bm{u}},{\bm{x}}))$. In this
section, we show the evolution dynamic of learning ${\bm{g}}$ under the
following weighted MSE loss

${\mathcal{L}}({\bm{u}})=\sum_{i=1}^{n}{\mathcal{L}}_{i}({\bm{u}})=\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{J}\sum_{k=1}^{K}Q_{jk}^{-1}\\{y_{ik}-g_{k}({\bm{u}},{\bm{x}}_{i})\\}^{2}$

where ${\bm{Q}}^{-1}\in{\mathbb{R}}^{J\times K}$ is a known matrix and
$Q_{jk}^{-1}$ is the $(j,k)$-th element of ${\bm{Q}}^{-1}$.

Following (8), the evolution of the parameters ${\bm{u}}$ and the output
${\bm{g}}$ via continuous time gradient descent satisfies

$\begin{split}&\frac{dg_{k}({\bm{u}}(t),{\bm{x}}_{i})}{dt}=\frac{\partial
g_{k}({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{d{\bm{u}}(t)}{dt}\\\
&\frac{d{\bm{u}}(t)}{dt}=-\eta\sum_{l=1}^{n}\sum_{\kappa=1}^{K}\frac{\partial
g_{\kappa}({\bm{u}}(t),{\bm{x}}_{l})}{\partial{\bm{u}}}\frac{\partial{\mathcal{L}}_{l}({\bm{u}}(t))}{\partial
g_{\kappa}}\\\ \end{split}$ (9)

Under the weighted MSE loss, we have

$\frac{\partial{\mathcal{L}}_{l}({\bm{u}}(t))}{\partial
g_{\kappa}}=\sum_{j=1}^{J}Q_{j\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{l})-y_{l\kappa}\\}.$

Plugin the gradient to (9), we have

$\frac{dg_{k}({\bm{u}}(t),{\bm{x}}_{i})}{dt}=-\eta\sum_{l=1}^{n}\sum_{\kappa=1}^{K}\sum_{j=1}^{J}\frac{\partial
g_{k}({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{\kappa}({\bm{u}}(t),{\bm{x}}_{l})}{\partial{\bm{u}}}Q_{j\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{l})-y_{l\kappa}\\}$

Let
${\bm{g}}({\bm{u}},{\mathcal{D}})=\sigma({\bm{f}}({\bm{u}},{\mathcal{D}}))=\left(g_{1}({\bm{u}},{\bm{x}}_{1}),\ldots,g_{1}({\bm{u}},{\bm{x}}_{n}),\ldots,g_{K}({\bm{u}},{\bm{x}}_{1}),\ldots,g_{K}({\bm{u}},{\bm{x}}_{n})\right)^{\top}$,
and

${\bm{y}}=(y_{11},\ldots,y_{n1},\ldots,y_{1K},\ldots,y_{nK})^{\top}.$

Then in the matrix form, the full evolution dynamic of learning ${\bm{g}}$
becomes

$\frac{d{\bm{g}}({\bm{u}}(t),{\mathcal{D}})}{dt}=-\eta{\bm{H}}(t)({\bm{g}}({\bm{u}}(t),{\mathcal{D}})-{\bm{y}})$

where the Gram matrix ${\bm{H}}(t)\in{\mathbb{R}}^{Kn\times Kn}$ is a block
matrix with $K$ row partitions and $K$ column partitions. The
$(p_{1},p_{2})$-th element of the block matrix in the $l$-th row and $m$-th
column has the following form

$\begin{split}{\bm{H}}_{p_{1},p_{2}}^{l,m}(t)=\sum_{j=1}^{J}Q_{jm}^{-1}\frac{\partial
g_{l}({\bm{u}}(t),{\bm{x}}_{p_{1}})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{m}({\bm{u}}(t),{\bm{x}}_{p_{2}})}{\partial{\bm{u}}}.\end{split}$ (10)

Federated Classification We can now extend the study of the full evolution
dynamic of learning ${\bm{g}}$ under centralized version to federated
classification. For federated learning case, we have the following datasets:
the set of training examples
${\mathcal{D}}^{c}=\\{({\bm{x}}_{i}^{c},{\bm{y}}_{i}^{c}):i\in S_{c}\\}$ on
$c$-th client where ${\bm{y}}_{i}$ is the one-hot representation of the label
in dimension $K$ and the training data on the server
${\mathcal{D}}^{s}=\\{({\bm{x}}_{i}^{s},{\bm{y}}_{i}^{s}):i\in[nK]\\}$ where
${\bm{y}}_{i}^{s}$ is the one-hot representation of the label. Let
${\bm{f}}:{\mathcal{X}}\mapsto{\mathbb{R}}^{K}$ be a $K$-class classifier. We
consider the learning of ${\bm{f}}$ under the mean squared error loss

${\mathcal{L}}_{c}({\bm{u}})=\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{J}\sum_{k=1}^{K}Q_{jk}^{-1}\\{y_{ik}^{c}-g_{k}({\bm{u}},{\bm{x}}_{i}^{c})\\}^{2},~{}{\mathcal{L}}_{s}({\bm{u}})=\frac{1}{2}\sum_{i=1}^{nK}\sum_{k^{\prime}=1}^{K}\sum_{k=1}^{K}T_{k^{\prime}k}^{-1}\\{y_{ik}^{s}-g_{k}({\bm{u}},{\bm{x}}_{i}^{s})\\}^{2}.$

The overall loss is

${\mathcal{L}}({\bm{u}})={\mathcal{L}}_{s}({\bm{u}})+\sum_{c=1}^{C}{\mathcal{L}}_{c}({\bm{u}}).$
(11)

Same as the centralized case, let us consider the evolution of ${\bm{g}}$ via
continuous time gradient descent.

$\begin{split}&\frac{dg_{k}({\bm{u}}(t),{\bm{x}}_{i})}{dt}=\frac{\partial
g_{k}({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{d{\bm{u}}(t)}{dt}\\\
&\frac{d{\bm{u}}(t)}{dt}=-\eta\sum_{l}\sum_{\kappa=1}^{K}\frac{\partial
g_{\kappa}({\bm{u}}(t),{\bm{x}}_{l})}{\partial{\bm{u}}}\frac{\partial{\mathcal{L}}_{l}({\bm{u}}(t))}{\partial
g_{\kappa}}\end{split}$

The difference of our federated version from the centralized version is
$\partial{\mathcal{L}}/\partial g_{\kappa}$. Under (11), we have

$\frac{\partial{\mathcal{L}}_{l}({\bm{u}}(t))}{\partial
g_{\kappa}}=\begin{cases}\sum_{j=1}^{J}Q_{j\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{l})-y_{l\kappa}\\}&\text{if
}{\bm{x}}_{l}\text{ on }c\text{-th client},\\\
\sum_{k^{\prime}=1}^{K}T_{k^{\prime}\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{l})-y_{l\kappa}\\}&\text{if
}{\bm{x}}_{l}\text{ on the server}.\end{cases}$

Hence we have

$\begin{split}\frac{dg_{k}({\bm{u}}(t),{\bm{x}}_{i})}{dt}=&-\eta\sum_{i^{\prime}=1}^{nK}\sum_{\kappa=1}^{K}\frac{\partial
g_{k}({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{\kappa}({\bm{u}}(t),{\bm{x}}_{i^{\prime}}^{s})}{\partial{\bm{u}}}\sum_{k^{\prime}=1}^{K}T_{k^{\prime}\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{i^{\prime}}^{s})-y_{i^{\prime}\kappa}^{s}\\}\\\
&-\eta\sum_{c=1}^{C}\sum_{i^{\prime}\in
S_{c}}\sum_{\kappa=1}^{K}\frac{\partial
g_{k}({\bm{u}}(t),{\bm{x}}_{i})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{\kappa}({\bm{u}}(t),{\bm{x}}_{i^{\prime}}^{c})}{\partial{\bm{u}}}\sum_{j=1}^{J}Q_{j\kappa}^{-1}\\{g_{\kappa}({\bm{u}},{\bm{x}}_{i^{\prime}}^{c})-y_{i^{\prime}\kappa}^{c}\\}.\end{split}$

To write the evolution of ${\bm{g}}$ in matrix form, let us first introduce
some notations. Let

$g_{k}({\bm{u}},{\mathcal{D}})=(g_{k}({\bm{u}},{\bm{x}}_{1}^{1}),\ldots,g_{k}({\bm{u}},{\bm{x}}_{N_{1}}^{1}),\ldots,g_{k}({\bm{u}},{\bm{x}}_{1}^{C}),\ldots,g_{k}({\bm{u}},{\bm{x}}_{N_{C}}^{C}),g_{k}({\bm{u}},{\bm{x}}_{1}^{s}),\ldots,g_{k}({\bm{u}},{\bm{x}}_{nK}^{s}))^{\top}$

and
${\bm{g}}({\bm{u}},{\mathcal{D}})=\left(g_{1}^{\top}({\bm{u}},{\mathcal{D}}),\ldots,g_{K}^{\top}({\bm{u}},{\mathcal{D}})\right)^{\top}$.
Note the vector ${\bm{g}}({\bm{u}},{\mathcal{D}})$ is first grouped by
machines and then classes. Similarly, we also denote
${\bm{y}}^{k}=(y_{1k}^{1},\ldots,y_{N_{1},k}^{1},\ldots,y_{1k}^{C},\ldots,y_{N_{C},k}^{C},y_{1k}^{s},\ldots,y_{nK,k}^{s})$
and ${\bm{y}}=({\bm{y}}^{1},{\bm{y}}^{2},\ldots,{\bm{y}}^{K})$. Then in the
matrix form, the full evolution dynamic of learning ${\bm{g}}$ becomes

$\frac{d{\bm{g}}({\bm{u}}(t),{\mathcal{D}})}{dt}=-\eta{\bm{G}}(t)({\bm{g}}({\bm{u}}(t),{\mathcal{D}})-{\bm{y}})$

where the Gram matrix ${\bm{G}}(t)\in{\mathbb{R}}^{K(nK+N)\times K(nK+N)}$ is
a block matrix with $K$ row partitions and $K$ column partitions. The block
matrix in the $l$-th row and $m$-th column has the following form

$\displaystyle{\bm{G}}^{l,m}(t)=\begin{pmatrix}{\mathcal{G}}_{1,1}^{l,m}(t)&{\mathcal{G}}_{1,2}^{l,m}(t)&\ldots&{\mathcal{G}}_{1,C}^{l,m}(t)&{\mathcal{G}}_{1,s}^{l,m}(t)\\\
{\mathcal{G}}_{2,1}^{l,m}(t)&{\mathcal{G}}_{2,2}^{l,m}(t)&\ldots&{\mathcal{G}}_{2,C}^{l,m}(t)&{\mathcal{G}}_{2,s}^{l,m}(t)\\\
\vdots&\vdots&\ddots&\vdots&\vdots\\\
{\mathcal{G}}_{C,1}^{l,m}(t)&{\mathcal{G}}_{C,2}^{l,m}(t)&\ldots&{\mathcal{G}}_{C,C}^{l,m}(r)&{\mathcal{G}}_{C,s}^{l,m}(t)\\\
{\mathcal{G}}_{s,1}^{l,m}(t)&{\mathcal{G}}_{s,2}^{l,m}(t)&\ldots&{\mathcal{G}}_{s,C}^{l,m}(t)&{\mathcal{G}}_{s,s}^{l,m}(t)\\\
\end{pmatrix}$

for $l\in[K]$ and $m\in[K]$. The block matrix ${\bm{G}}^{l,m}(r)$ is also
consisted of block matrices. The block matrix
${\mathcal{G}}^{l,m}_{c_{1},c_{2}}(t)\in{\mathbb{R}}^{N_{c_{1}}\times
N_{c_{2}}}$, ${\mathcal{G}}^{l,m}_{s,c}(t)\in{\mathbb{R}}^{nK\times N_{c}}$,
and ${\mathcal{G}}^{l,m}_{s,s}(t)\in{\mathbb{R}}^{nK\times nK}$ for
$c_{1}\in[C]$, $c_{2}\in[C]$, and $c\in[C]$.

We now specify the elements in ${\mathcal{G}}^{l,m}_{c_{1},c_{2}}(t)$,
${\mathcal{G}}^{l,m}_{s,c}(t)$, and ${\mathcal{G}}^{l,m}_{s,s}(t)$
respectively. The derivation is the same as (10). We have

  * •

The $(p_{1},p_{2})$-th element of ${\mathcal{G}}^{l,m}_{c_{1},c_{2}}(t)$ has
the following form

$\sum_{j=1}^{J}Q_{jm}^{-1}\frac{\partial
g_{l}({\bm{u}},{\bm{x}}_{p_{1}}^{c_{1}})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{m}({\bm{u}},{\bm{x}}_{p_{2}}^{c_{2}})}{\partial{\bm{u}}}.$

  * •

The $(p_{1},p_{2})$-th element of ${\mathcal{G}}^{l,m}_{s,c}(t)$ has the
following form

$\sum_{j=1}^{J}Q_{jm}^{-1}\frac{\partial
g_{l}({\bm{u}},{\bm{x}}_{p_{1}}^{s})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{m}({\bm{u}},{\bm{x}}_{p_{2}}^{c})}{\partial{\bm{u}}}.$

  * •

The $(p_{1},p_{2})$-th element of ${\mathcal{G}}^{l,m}_{s,s}(t)$ has the
following form

$\sum_{k=1}^{K}T_{km}^{-1}\frac{\partial
g_{l}({\bm{u}},{\bm{x}}_{p_{1}}^{s})}{\partial{\bm{u}}^{\top}}\frac{\partial
g_{m}({\bm{u}},{\bm{x}}_{p_{2}}^{s})}{\partial{\bm{u}}}.$

###  B.3 Proof of Corollaries

In this section, we provide the proof for Corollary 3.5 and Corollary 3.6.

Proof for Corollary 3.6 The key for the proof is to find the form of
${\bm{T}}^{-1}$ when
${\bm{T}}=(1-K\xi/(K-1)){\mathbb{I}}_{K}+\xi/(K-1)\mathbbm{1}_{K}\mathbbm{1}_{K}^{\top}$.
To find $T^{-1}$, we make use of the Woodbury matrix identity

$({\bm{A}}+{\bm{U}}{\bm{C}}{\bm{V}})^{-1}={\bm{A}}^{-1}-{\bm{A}}^{-1}{\bm{U}}({\bm{C}}^{-1}+{\bm{V}}{\bm{A}}^{-1}{\bm{U}})^{-1}{\bm{V}}{\bm{A}}^{-1}$

where ${\bm{A}}$, ${\bm{U}}$, ${\bm{C}}$, and ${\bm{V}}$ are matrices of
proper sizes.

In our case, ${\bm{A}}=\\{1-K\xi/(K-1)\\}{\mathbb{I}}_{K}$,
${\bm{U}}=\mathbbm{1}_{K}$, ${\bm{C}}=\xi/(K-1)$, and
${\bm{V}}=\mathbbm{1}_{K}^{\top}$. By applying Woodbury matrix identity, we
get

${\bm{T}}^{-1}=(K-1)/(K-1-K\xi){\mathbb{I}}_{K}-\xi/(K-1-K\xi)\mathbbm{1}_{K}\mathbbm{1}_{K}^{\top}.$

Hence

$\sum_{k=1}^{K}T_{kk^{\prime}}=\left({\bm{T}}^{-1}\mathbbm{1}_{K}\right)_{k}=\left((K-1)/(K-1-K\xi)\mathbbm{1}_{K}-K\xi/(K-1-K\xi)\mathbbm{1}_{K}\right)_{k}=\left(\mathbbm{1}_{K}\right)_{k}=1$

which completes the proof.

Proof for Corollary 3.5 Let $k_{1},k_{2},\ldots,k_{J}$ be $J$ non-negative
integers such that $K=\sum_{j=1}^{J}k_{j}$. If the linear transformation from
desired label space to the other label space is
${\bm{Q}}=\text{diag}(\mathbf{1}_{k_{1}},\mathbf{1}_{k_{2}},\ldots,\mathbf{1}_{k_{J}})\in[0,1]^{K\times
J}$ (i.e., the classes in the desired label space are sub-classes of the class
in the other label space), then the smaller the value of $J$, the lower the
convergence rate.

We will use the following conclusion (Li et al.,, 2021) in the proof. For a
block positive definite matrix $\mathbf{A}=(\mathbf{A}_{ij})_{n\times n}$, we
must have

$\lambda_{\min}(\mathbf{A})\leq\min_{i}\lambda_{\min}(\mathbf{A}_{ii}).$

By recursively using this conclusion on ${\bm{G}}(t)$ given in (7). We have

$\lambda=\lambda_{\min}({\bm{G}}(0))\leq\min_{k,c,s}\\{\lambda_{\min}({\mathcal{G}}_{c,c}^{k,k}(0)),\lambda_{\min}({\mathcal{G}}_{s,s}^{k,k}(0))\\}\leq\min_{k,c}\lambda_{\min}({\mathcal{G}}_{c,c}^{k,k}(0)).$

Let
$\lambda_{0}^{c}=\lambda_{\min}(\nabla_{{\bm{u}}}g_{k}({\bm{u}}(0),{\mathcal{D}}^{c})\nabla_{{\bm{u}}}g_{k}^{\top}({\bm{u}}(0),{\mathcal{D}}^{c}))$
for $c\in[C]$. Under the special case where the two label spaces have
hierarchical structures, we have
$Q^{-1}=\text{diag}(k_{1}^{-1}\mathbbm{1}_{k_{1}}^{\top},k_{2}^{-1}\mathbbm{1}_{k_{2}}^{\top},\ldots,k_{J}^{-1}\mathbbm{1}_{k_{J}}^{\top})$

Since
${\mathcal{G}}_{c,c}^{k,k}(0)=\left\\{\sum_{j}Q_{jk}^{-1}\right\\}\nabla_{{\bm{u}}}g_{k}({\bm{u}}(0),{\mathcal{D}}^{c})\nabla_{{\bm{u}}}g_{k}^{\top}({\bm{u}}(0),{\mathcal{D}}^{c})$,
it is easy to see that

$\lambda_{\min}({\mathcal{G}}_{c,c}^{k,k}(0))\leq\left\\{\min_{j}k_{j}^{-1}\right\\}\lambda_{0}^{c}.$

where $\lambda_{0}^{c}$ is a constant, the smallest eigen value of
$\nabla_{{\bm{u}}}g_{k}({\bm{u}}(0),{\mathcal{D}}^{c})\nabla_{{\bm{u}}}g_{k}^{\top}({\bm{u}}(0),{\mathcal{D}}^{c})$
(i.e., the original kernel of FL-NTK Huang et al., (2021)), which does not
depend on $J$.

Hence

$\lambda\leq\min_{c}\left\\{\min_{j}k_{j}^{-1}\right\\}\lambda_{0}^{c}.$

Since $\sum_{j}k_{j}=K$, and in our setting we have $k_{1}\approx
k_{2}\approx\cdots\approx k_{J}$, the smaller the value of $J$, the smaller
the value of $\lambda$ and hence the algorithm converges slower.

##  Appendix C Experimental Details

We present our model architectures and training details of the benchmark and
real-world experiments in this section.

###  C.1 Model Architecture and Implementation Details on Benchmark

The details of the experiments on the benchmark CIFAR100 dataset are presented
in this section.

#### Model architecture

For our benchmark experiments on CIFAR100, we use ResNet18 as our backbone.
The network details are listed in Tab. 4.

Table 4: Model architecture of the benchmark experiment on CIFAR100. For
convolutional layer (Conv2D), we list parameters with sequence of input and
output dimension, kernel size, stride and padding. For max pooling layer
(MaxPool2D), we list kernel and stride. For fully connected layer (FC), we
list input and output dimension. For BatchNormalization layer (BN), we list
the channel dimension.

Layer | Details
---|---
1 | Conv2D(3, 64, 7, 2, 3), BN(64), ReLU
2 | Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
3 | Conv2D(64, 64, 3, 1, 1), BN(64)
4 | Conv2D(64, 64, 3, 1, 1), BN(64), ReLU
5 | Conv2D(64, 64, 3, 1, 1), BN(64)
6 | Conv2D(64, 128, 3, 2, 1), BN(128), ReLU
7 | Conv2D(128, 128, 3, 1, 1), BN(64)
8 | Conv2D(64, 128, 1, 2, 0), BN(128)
9 | Conv2D(128, 128, 3, 1, 1), BN(128), ReLU
10 | Conv2D(128, 128, 3, 1, 1), BN(64)
11 | Conv2D(128, 256, 3, 2, 1), BN(128), ReLU
12 | Conv2D(256, 256, 3, 1, 1), BN(64)
13 | Conv2D(128, 256, 1, 2, 0), BN(128)
14 | Conv2D(256, 256, 3, 1, 1), BN(128), ReLU
15 | Conv2D(256, 256, 3, 1, 1), BN(64)
16 | Conv2D(256, 512, 3, 2, 1), BN(512), ReLU
17 | Conv2D(512, 512, 3, 1, 1), BN(512)
18 | Conv2D(256, 512, 1, 2, 0), BN(512)
19 | Conv2D(512, 512, 3, 1, 1), BN(512), ReLU
20 | Conv2D(512, 512, 3, 1, 1), BN(512)
21 | AvgPool2D
22 | FC(512, 100)

#### Data split and label Generation

Here, we describe how to prepare the CIFAR100 training dataset into sub-class
but noisy data on the server and super-class datasets on the clients. We first
select $n$ samples from each of the $K=100$ sub-classes as the datasets on the
server. To generate noisy labeled data at noise level $\xi\in[0,1]$, we assume
that the noise matrix is known, symmetric, instance independent, and
reconstructable, represented as the matrix
${\bm{T}}_{\xi}=\\{1-K/(K-1)\xi\\}{\mathbb{I}}_{K}+\xi/(K-1)\mathbbm{1}_{K}\mathbbm{1}_{K}^{\top}$.
Then the observed labels are sampled from this given matrix by random
flipping, i.e., the label in one class is flipped to the label in another
class with probability $\xi/(K-1)$. The higher the value of $\xi$, the larger
the proportion of noise labels. Then for the rest of the observations in the
CIFAR100 training set, we randomly split then into $C$ pieces each with
$N_{c}$ observations and each client is assigned $N_{c}$ samples. Hence in
this case ${\bm{Q}}$ is a $200\times 100$ matrix, its $(j,i)$-th element is
$1$ if the $i$-th class is in $j$-th superclass and $0$ otherwise.

#### Training details

We implement all the methods by PyTorch and conducted all the experiments on
an NVIDIA Tesla V100 GPU. For our FedMT, we use SGD optimizer Ruder, (2016)
with a learning rate of $10^{-2}$, momentum $0.9$, and weight decay $5\times
10^{-4}$. Each model updates one epoch then aggregates with the others. The
total communication iterations is set to be 100. The learning rate is divided
by $5$ at $20$, $30$, and $40$ iterations. We set the batch size to $16$ and
the number of training examples on $c$-th client to $N_{c}=4000$ if not
specified otherwise. In Tab. 1, we consider various values for the per number
of observations $n$ on the server under the noise-free setting. We also
consider various values for the noisy level $\xi$ when $n=10$. For additional
results in the Appendix D, we specify the hyper-parameters only when they are
different.

We describe details of the baseline methods and corresponding hyper-parameters
as follows.

  * •

Single For this baseline method, we only use the limited data on the server to
train a $K$-class classifiers. Under the noisy label scenarios, we also apply
the label projection and the probability projection on the server and the
learning objectives is

$\widehat{R}({\bm{f}})=-\frac{1}{nK}\sum_{i=1}^{nK}\sum_{k=1}^{K}\mathbbm{1}(y_{i}^{s}=Y^{k})\log\left\\{\sum_{k^{\prime}=1}^{K}T_{kk^{\prime}}\sigma(f_{k^{\prime}}({\bm{x}}_{i}^{s}))\right\\},$

for probability projection and

$\widehat{R}({\bm{f}})=-\frac{1}{nK}\sum_{i=1}^{nK}\sum_{k=1}^{K}\left\\{\sum_{k^{\prime}=1}^{K}T_{kk^{\prime}}^{-1}\mathbbm{1}(y_{i}^{s}=Y^{k^{\prime}})\right\\}\log\sigma(f_{k}({\bm{x}}_{i}^{s})),$

for label projection. We use the same optimizer as our FedMT.

  * •

FedMatch We use the semi-supervised FedMatch proposed by (Lu et al.,, 2021).
We treat client samples as unlabeled data, perform pseudo-labeling on the
unlabeled sets under the supervision of labeled data on the server. The
pseudo-labels are updated heuristically during the training. The data
augmentation strategy is also used. For fair comparison, we do not use the
pretrained backbone during training. We set learning rate to $0.01$, batch
size to $16$, and the rest of the hyperparameters are the same as the default
setting in the implementation.

  * •

FedRep (Collins et al.,, 2021) For this baseline method, all clients train a
super-class classifier and the server trains a sub-class classifier that
differs in the last layer with different output dimension. The learning
objective is

$\widehat{R}({\bm{f}})=\frac{1}{C}\sum_{c=1}^{C+1}\left\\{\widehat{R}_{c}({\bm{f}};{\mathcal{D}}^{c})+\widehat{R}_{s}({\bm{f}};{\mathcal{D}}^{s})\right\\},$

where

$\widehat{R}_{c}({\bm{f}};{\mathcal{D}}^{c})=-\frac{1}{N_{c}}\sum_{i\in
S_{c}}\sum_{j=1}^{J}\mathbbm{1}(y_{i}=\widetilde{Y}^{j})\log\sigma(f_{j}({\bm{x}}_{i}))$

and $\widehat{R}_{s}({\bm{f}};{\mathcal{D}}^{s})$ is the same as the objective
function in the Single baseline approach. All classifiers on the server and on
the clients use the same ResNet18 backbone, but they have different last
layers. For the classifiers on clients, their last layer is FC(512,20); for
the classifier on the server, its last layer is FC(512,100). The last layers
of classifiers on clients are also not shared during the training. The rest of
the parameters of all classifiers are aggregated using FedAvg during training.
We use the same optimizer as our FedMT.666Note that the data labeling
settings, models, and evaluation methods implemented in our work is adjusted
to make fair comparison with other methods and different from those in Collins
et al., (2021).

  * •

FedTrans For this baseline method, we first pretrain a model using the client
data to train a $J$-class classifiers using FedAvg with probability projection
or label projection. This pretrained net is then fine-tuned on the server.
During the fine-tuning, we add a new linear layer with $K$ classes with random
initialization and backbone is initialized with the pretrained model. We use
the same optimizer as our FedMTduring the pretrain step and the model is fine-
tuned on the server for $100$ epochs with SGD optimizer and learning rate
$0.01$.

###  C.2 Model Architecture and Implementation Details on Medical Dataset

#### Model architecture

We use a simple MLP as the model for the medical data, whose architecture is
listed in Table 5.

Table 5: Model architecture of the experiment on medical dataset sEMG. For
fully connected layer (FC), we list input and output dimension. For
BatchNormalization layer (BN), we list the channel dimension.

Layer | Details
---|---
1 | FC(12,128), BN(128), ReLU
2 | FC(128, 10)

#### Data Preprocessing

The raw sEMG data is collected from Qin et al., (2020), with the
dimensionality of 2048, i.e., 2048 time points per sample,
${\bm{x}}=[x_{1},\dots,x_{2048}]$ . Following Qin et al., (2019), instead of
using the raw time series, we use the extracted features as described in
Section 4.The features twelve hand-crafted features, including mean absolute
value (MAV), mean square value (MSV), root mean square (RMS), variance (VAR),
standard deviation (STD), waveform length (WL), Willison amplitude (WAMP), log
detector (LOG), slope sign change (SSC), zero crossing (ZC), mean spectral
frequency (MSF) and median frequency (MF). Denote $P_{j}$ is the sEMG power
spectrum at frequency bin $j$, and $f_{j}$ is the frequency of the sEMG power
spectrum at frequency bin $j$. Mathematically, these features are defined in
Tab. 6

Table 6: Mathematical representation of widely used sEMG feature extraction
methods. The constant $\zeta$ is an user specified threshold and we set
$\zeta=1$.

Feature Extraction | Mathematical Equation
---|---
mean absolute value (MAV) | $\frac{1}{N}\sum_{n=1}^{N}|x_{n}|$
mean square value (MSV) | $\frac{1}{N}\sum_{n=1}^{N}x_{n}^{2}$
root mean square (RMS) | $\sqrt{\frac{1}{N}\sum_{n=1}^{N}x_{n}^{2}}$
variance (VAR) | $\frac{1}{N-1}\sum_{n=1}^{N}x_{n}^{2}$
standard deviation (STD) | $\sqrt{\frac{1}{N-1}\sum_{n=1}^{N}x_{n}^{2}}$
waveform length (WL) | $\sum_{n=1}^{N-1}|x_{n+1}-x_{n}|$
Willison amplitude (WAMP) |  $\sum_{n=1}^{N-1}f(|x_{n+1}-x_{n}|)$; $f(x)=\begin{cases}1&\text{if }x\geq\zeta\\\ 0&\text{otherwise}\end{cases}$
log detector (LOG) | $\exp{\left(\frac{1}{N}\sum_{n=1}^{N}\log|x_{n}|\right)}$
slope sign change (SSC) |  $\sum_{n=2}^{N-1}f[(x_{n}-x_{n-1})\times(x_{n}-x_{n+1})]$; $f(x)=\begin{cases}1&\text{if }x\geq\zeta\\\ 0&\text{otherwise}\end{cases}$
zero crossing (ZC) |  $\sum_{n=1}^{N-1}[\text{sgn}(x_{n}\times x_{n+1})\bigcap|x_{n}-x_{n+1}|\geq\zeta]$; $\text{sgn}(x)=\begin{cases}1&\text{if }x\geq\zeta\\\ 0&\text{otherwise}\end{cases}$
mean spectral frequency (MSF) | $\frac{\sum_{j=1}^{M}f_{j}P_{j}}{\sum_{j=1}^{M}P_{j}}$
median frequency (MF) |  $\frac{1}{2}\sum_{j=1}^{M}P_{j}$ ${\mathcal{X}}\subset\mathbb{R}^{d}$

We randomly sample 9000 samples as training data annotated with label from the
other label space, and 1000 samples as the held testing data. We set the
number of clients $C=50$, namely each client has $180$ samples. The data are
labeling based on the severity value from 0 to 5. We equally partition the
values into 3 parts with the same intervals (i.e., [0,5/3), [5/3,10/3), and
[10/3,5]) corresponding to $J=3$ labeled classes on the clients. Such class
partition strategy can simulate the overlapped case between two label spaces,
where samples in the same $Y^{k}$ class can belong to different
$\widetilde{Y}^{j}$ classes. From the remaining sEMG samples, we randomly
sample $n$ observations per $5$ class to represent the data on server. Under
the noisy label scenarios, the noisy labels are generated using the same
protocol as described in Section C.1.

#### Training details

We implement all the methods by PyTorch and conducted all the experiments on
an NVIDIA Tesla V100 GPU. All the methods are trained for 100 round and each
round, model update once before aggregation. We use SGD optimizer Ruder,
(2016) with a learning rate of $10^{-2}$, momentum $0.9$, and weight decay
$5\times 10^{-4}$. The learning rate is divided by $5$ at $20$, $30$, and $40$
epochs. We set the batch size to $16$. We also consider various values for the
noisy level $\xi$ from 0 to 0.4. All the experiments are repeated with
independent random seeds.

##  Appendix D More Experimental Results on CIFAR100

In this section, we present more experiment results on the benchmark CIFAR100
dataset. In Section D.1, we show the convergence of the loss function as a
function of batch sizes and the local rounds. In Section D.2, we show the
experiment results of our FedMT when different weighting schemes are used for
server and clients. In Section D.3, we show the experiment results as the
number of per client observation increases. In Section D.4, we study the
performance of our proposed method under label heterogeneous across clients.
In Section D.5, we show the performance of our method when limited data from
desired label space are further split into more pieces.

###  D.1 Effects of Local Rounds and Batch Size

To demonstrate the robustness of the proposed FedMT and validate our developed
theoretical results in Theorem 3.4, we investigate the effects of different
local rounds $t$ and the batch size $B$ on training losses (2) of FedMT with
label projection in Section 4.1. For all experiments, the learning rate is set
to be $0.0001$ and no learning rate scheduler is used during the training. In
Fig. 3(a), we explore loss as a function of the batch size
$B\in\\{16,64,128\\}$ with $C=10$ and without noise on super-class labels. The
larger the batch size, the faster the loss function converges. In Fig. 3(b),
we explore loss as a function of the local rounds $t\in\\{1,4,8\\}$ with
$C=10$ and without noise on super-class labels. The larger the local rounds,
the faster the convergence, which is consistent with the theoretical result of
Theorem 3.4. It is worth noting that all the training curves of different
variants of the parameters reach convergence.

(a) Loss curve for different batch sizes

(b) Loss curve for different local rounds

Figure 3: Effects of (a) the batch size $B$ and (b) the local rounds $t$ on
the rate of convergence on the loss function.

###  D.2 Benchmark Experiments with Weighted FedMT

In our main results, the loss on the each client and the loss on the server
are equally weighted with weight $1/(C+1)$. To see the performance of our
FedMT, we also investigate the case when these losses are weighted unequally
and report the classification accuracy of the corresponding trained model on
the held out test set in Tab. 7. For these experiment results, we consider
different number of $n\in\\{5,10,20,40,80\\}$ with $\xi=0.2$. The sample size
on $c$-th client $N_{c}=4000$. For the unequal weight case, the weight on the
server is $0.5$ regardless of the size of $n$ and the weight on each client is
$0.5/C$. As shown in the table, the test classification accuracy increases
with $n$ for both equally weighted and unequally weighted cases. The
performance of the unequally weighted case is not as good as the equally
weighted case.

Table 7: Comparison of the accuracy for 100-class classification using
different FL aggregation strategies on CIFAR100 benchmark dataset at different
number of per $K$ class training examples $n$ when noise level $\xi=0.2$. We
report mean (sd) from three trial runs.

Weights | Methods | 5 | 10 | 20 | 40 | 80
---|---|---|---|---|---|---
Equal | Our (P) | 19.70(0.30) | 24.26(0.85) | 27.48(0.48) | 30.56(0.61) | 34.85(0.12)
Our (L) | 18.10(0.41) | 21.26(0.32) | 24.37(0.26) | 28.28(0.18) | 32.15(0.86)
Unequal | Our (P) | 19.27(0.20) | 21.49(0.50) | 23.83(0.12) | 26.66(0.11) | 29.25(0.25)
Our (L) | 13.16(0.62) | 14.40(0.21) | 16.09(0.22) | 19.15(0.25) | 21.61(0.33)

###  D.3 Benchmark Experiments with More Clients

In Tab. 1, we check the performance of our proposed method for varying amount
of data on the server. In this section, we present the performance of our
proposed method under varying amount of data on the client when $n=10$ and
$\xi=0.2$ on the server. We also set the per client number of observations to
be $N_{c}=800$. We change the number of clients $C\in\\{10,20,30,40,50\\}$.
The larger the number of clients $C$, more data we have. During the
aggregation, we set the weights of the server to be $0.5$ and the rest of the
client to be $0.5/C$. In comparison, if the equal weight $1/(C+1)$ is used,
the super-class data on the server plays a less importance role as $C$
increases. We therefore use the unequally weighted strategy to avoid. The
classification result of the aggregated model on the held out test set is
given in Table 8. As shown in the table, the classification accuracy increases
as the number of clients increases.

Table 8: Our FedMT classification accuracy on benchmark CIFAR100 with
different number of clients.

Methods | 10 | 20 | 30 | 40 | 50
---|---|---|---|---|---
Our (P) | 13.61(0.16) | 15.17(0.29) | 15.28(0.11) | 15.92(0.44) | 16.32(0.27)

###  D.4 Benchmark Experiments under Non-IID Data Split

The heterogeneity of the training data across clients is a major challenge in
FL Kairouz et al., (2021). We therefore study the performance of our method
under two different FL settings: IID and non-IID. The overall label
distribution across clients is the same in the IID setting, whereas clients
have different label distributions in the non-IID setting. For the IID
setting, we split the sub-class label training data evenly to all clients
completely at random. For the non-IID setting, we split the dataset to clients
as follows:

  * •

the classes are divided into the majority and minority classes, where the
fraction of each minority class is less than 0.08 while the fraction of each
majority class is in the range [0.15, 0.25];

  * •

the training data for every client are sampled from 2 majority classes and
rest from minority classes;

We compare our method with personalized FL method FedRep both under noise-free
and label noise on the server.

Table 9: Comparison of the accuracy for 100-class classification using our
methods and FedRep on the CIFAR100 benchmark dataset under IID and non-IID
data split. We report mean (sd) from three trial runs.

$\xi$ | 0.0 | 0.1 | 0.2 | 0.3 | 0.4
---|---|---|---|---|---
IID
Our (P) | 26.44(0.34) | 25.88(0.67) | 24.26(0.85) | 22.25(0.41) | 21.04(0.33)
FedRep | 21.00(0.45) | 19.34(0.57) | 17.89(0.37) | 16.16(0.08) | 14.80(0.89)
Non-IID
Our (P) | 21.40(0.21) | 20.37(0.28) | 20.14(0.26) | 18.72(0.45) | 17.44(0.39)
FedRep | 20.71(0.58) | 18.45(0.69) | 17.11(0.35) | 14.73(0.25) | 13.74(0.13)

We make the following comparisons: 1) the performance of our method under IID
case and non-IID case. 2) the performance of our method under non-IID case and
FedRep under non-IID case. 3) the performance of our method under non-IID case
and other baselines under IID case. Since our aggregation is based on FedAvg,
it can be seen from the table that there is a 4-5% performance drop of our
proposed method when data on the clients are non-IID. However, it is worth
noticing that our proposed method under non-IID case works better than other
baselines under IID case. Moreover, our proposed method still outperforms
FedRep when both approaches are applied on non-IID data split. Therefore, the
above experiment results show the robustness of our proposed method compared
with other approaches.

###  D.5 Different Number of “Servers"

In our problem setting, we assume without loss of generality that there is one
‘specialized center’ with a small amount of data from desired label space and
we call this center as the server. Such a setting is motivated by the real
clinical application in Section 1 that “the centers with the most complex
labeling criteria…has much less labeled samples due to labeling difficulty or
cost". It is technically trivial to generalize our problem to the case with
several entities with data from desired label space by simplify modifying
Equation 11 as

$R_{\text{overall}}({\bm{f}})=\frac{1}{C+S}\left\\{\sum_{s=1}^{S}\widehat{R}_{s}({\bm{f}};\widetilde{\mathcal{D}}^{s})+\sum_{c=1}^{C}\widehat{R}_{c}({\bm{f}};\widetilde{\mathcal{D}}^{c})\right\\}$

where $\widetilde{\mathcal{D}}^{s}$ is the noisy labeled data on $s$-th server
and $S$ is the total number of such servers. To show the effectiveness of our
proposed method under this setting, we run the following experiments on
CIFAR100. We have $C=10$ clients with super-class clean data with each client
having $N_{c}=4000$ samples. We also have $n=20$ noisy sub-class data. We
consider the following 3 ways to store these $20\times 100$ images: 1) we have
one “server" with all noisy sub-class images. 2) we have $2$ “server"s with
each containing $n=10$ noisy sub-class images. 3) we have $4$ “server"s with
each containing $n=5$ noisy sub-class images. We change the level of noise
$\xi\in\\{0.1,0.2,0.3,0.4,0.5\\}$ and show the results for both FedMT(P) and
FedMT(L).

Table 10: Comparison of the accuracy for 100-class classification using
FedMT(P) on the CIFAR100 benchmark dataset with different split strategy of
the sub-class label at different noise level $\xi$. We report mean (sd) from
three trial runs.

Split | 0.1 | 0.2 | 0.3 | 0.4
---|---|---|---|---
Our (P)
1x20 | 28.26(0.88) | 27.48(0.48) | 26.36(0.84) | 24.36(1.47)
2x10 | 29.11(0.21) | 27.88(0.62) | 26.38(1.09) | 24.16(0.27)
4x5 | 28.13(0.36) | 27.26(0.37) | 24.64(0.63) | 23.37(0.36)
Our (L)
1x20 | 27.53(0.71) | 24.37(0.26) | 23.78(0.50) | 22.45(0.73)
2x10 | 25.64(0.52) | 24.08(0.29) | 22.29(0.71) | 20.67(0.43)
4x5 | 23.88(0.30) | 22.58(0.50) | 21.52(0.18) | 20.51(0.20)

It can be seen from the table that an increase in the number of “server"s does
not hurt the performance of FedMT(P) and there is a slight performance drop
for FedMT(L).

##  Appendix E More Experimental Results on Tremor Severity Prediction

For the experiment on tremor severity prediction in Section 4.3, we let $K=5$
and the ${\bm{Q}}$ matrix between two label spaces is

${\bm{Q}}=\begin{bmatrix}3/5&2/5&0&0&0\\\ 0&1/5&3/5&1/5&0\\\ 0&0&0&2/5&3/5\\\
\end{bmatrix}.$

There are 2 out of 5 classes in the desired space overlaps with one of the
class in the other space, each with degree of overlapping $1/5=20\%$. To
investigate the performance of various approaches under different degrees of
overlapping, we create the labels on the servers as below. We equally
partition the values into $10$ parts with the same intervals (i.e.,
$[0.5k,0.5k+0.5)$ for $k\in\\{0,2,\dots,9\\}$) and keep $J=3$ and the
corresponding labels the same as before. In this case, the ${\bm{Q}}$ matrix
between two label spaces becomes

${\bm{Q}}=\begin{bmatrix}3/10&3/10&3/10&1/10&0&0&0&0&0&0\\\
0&0&0&1/5&3/10&3/10&1/5&0&0&0\\\ 0&0&0&0&0&0&1/10&3/10&3/10&3/10\\\
\end{bmatrix}$

Under this case, there are $2$ out of $10$ classes in the desired space
overlaps with one of the class in the other space, each with degree of
overlapping is $1/10=10\%$. As a comparison and additional experiment results,
we compare the performance of all methods for different values of $K$ under
the case when there are noisy labels on the server. We let $n=3$ and the rest
of the settings the same as before. The experiment results are given in Tab.
11. It is shown in the table that our method can beat the rest of the methods
regardless of the noise level.

Table 11: Comparison of the accuracy using our methods and alternative methods
on the sEMG dataset with $n=3$ and $K=5$ that has smaller degree of class
overlapping and $K=10$ that has larger degree of class overlapping. We conduct
the experiment when $C=50$ at different noise level $\xi$. We report mean (sd)
from three trial runs. The best method is highlighted in boldface.

$\xi$ | Single | FedMatch | FedRep | FedTrans |  _Ours_ : FedMT (P&L)
---|---|---|---|---|---
larger degree of overlapping
0.0 | 22.35(2.31) | 13.04(1.79) | 29.23(2.31) | 28.75(1.21) | 28.97(0.32) | 32.59(1.81)
0.1 | 21.12(2.34) | 13.14(0.76) | 21.12(2.34) | 27.73(1.72) | 28.86(0.73) | 30.60(1.16)
0.2 | 18.75(1.68) | 12.51(2.19) | 18.75(1.68) | 25.95(0.99) | 27.63(1.05) | 30.26(1.69)
0.3 | 17.49(1.50) | 10.54(1.07) | 17.49(1.50) | 21.56(2.04) | 27.00(0.42) | 28.16(1.53)
0.4 | 14.25(1.17) | 11.64(1.17) | 14.25(1.17) | 18.00(2.15) | 27.03(1.48) | 28.96(1.39)
smaller degree of overlapping
0.0 | 34.39(2.86) | 34.07(1.32) | 41.12(1.34) | 51.25(2.57) | 67.34(0.56) | 64.33(0.48)
0.1 | 31.04(2.69) | 33.03(0.99) | 39.64(1.62) | 45.52(2.15) | 61.80(0.24) | 66.56(0.57)
0.2 | 30.77(1.89) | 32.37(1.21) | 37.66(1.80) | 41.07(1.45) | 61.34(0.30) | 66.00(0.57)
0.3 | 27.41(3.02) | 31.36(0.99) | 34.37(3.38) | 39.10(1.31) | 61.26(0.31) | 67.22(0.30)
0.4 | 24.66(2.59) | 27.56(1.26) | 27.09(2.03) | 37.01(3.15) | 66.44(0.41) | 63.74(0.27)

##  Appendix F Properties of Baselines for Comparison

For the empirical experiments in the paper, we compare the performance of our
proposed method with several baseline approaches, namely Single, FedRep,
FedMatch, and FedTrans. The details of these approaches and their
corresponding objective functions are given in Appendix C. As a summary, we
highlight their properties in Table 12 in terms of the type of labels they use
on the server and clients, their learning strategy, whether they are simple
extensions of FedAvg, and whether the theoretical guarantee on the convergence
of the algorithm has been established.

Table 12: The conceptual comparison of the baseline approaches with our
proposed method in terms of the type of labeled information used on server and
clients, the learning strategy, whether the approach is a simple extension of
FedAvg, and whether there is a theoretical guarantee on the convergence of the
algorithm.

| Server Label | Client Label | Strategy | Simple | Theory
---|---|---|---|---|---
Single | Desired | NA | supervised | Yes | Yes
FedMatch | Desired | Yes | semi-supervised | No | No
FedRep | Desired | Other | supervised | Yes | No
FedTrans | Desired | Other | supervised | Yes | No
Ours | Desired | Other | supervised | Yes | Yes

For the server label, NA means the server label is not used and Desired means
datasets are annotated with classes from the desired label space. For the
client label, NA means the client label is not used, Other means datasets
annotated with classes from the other label space are used, and Desired class
prior means only the information of the prior distribution of the desired
label space class labels is used. The second last column means whether the
proposed method is a simple extension of FedAvg and the last column means
whether the theoretical convergence of the method has been established under
FL. For methods that are simple extensions of FedAvg, the communication cost
is in general much cheaper than other more complicated approaches.