# Federated Multi-organ Segmentation with Inconsistent Labels

Xuanang Xu, Hannah H. Deng, Jaime Gateno, and Pingkun Yan Asterisk indicates
corresponding author.X. Xu and P. Yan* are with the Department of Biomedical
Engineering and the Center for Biotechnology and Interdisciplinary Studies at
Rensselaer Polytechnic Institute, Troy, NY 12180 USA<EMAIL_ADDRESS>yanp2@rpi.edu).H.H. Deng and J. Gateno are with the Department of Oral and
Maxillofacial Surgery at Houston Methodist Research Institute, Houston, TX
77030 USA<EMAIL_ADDRESS>jgateno@houstonmethodist.org).J. Gateno
is also with the Department of Surgery (Oral and Maxillofacial Surgery) at
Weill Medical College at Cornell University, New York, NY 10065 USA.This work
was supported in part by the National Science Foundation (NSF) under CAREER
award OAC 2046708 and the National Institutes of Health (NIH) under awards
R01DE022676 and R21EB028001.

###### Abstract

Federated learning is an emerging paradigm allowing large-scale decentralized
learning without sharing data across different data owners, which helps
address the concern of data privacy in medical image analysis. However, the
requirement for label consistency across clients by the existing methods
largely narrows its application scope. In practice, each clinical site may
only annotate certain organs of interest with partial or no overlap with other
sites. Incorporating such partially labeled data into a unified federation is
an unexplored problem with clinical significance and urgency. This work
tackles the challenge by using a novel federated multi-encoding U-Net (Fed-
MENU) method for multi-organ segmentation. In our method, a multi-encoding
U-Net (MENU-Net) is proposed to extract organ-specific features through
different encoding sub-networks. Each sub-network can be seen as an expert of
a specific organ and trained for that client. Moreover, to encourage the
organ-specific features extracted by different sub-networks to be informative
and distinctive, we regularize the training of the MENU-Net by designing an
auxiliary generic decoder (AGD). Extensive experiments on six public abdominal
CT datasets show that our Fed-MENU method can effectively obtain a federated
learning model using the partially labeled datasets with superior performance
to other models trained by either localized or centralized learning methods.
Source code is publicly available at https://github.com/DIAL-RPI/Fed-MENU.

###### Index Terms:

Federated learning, Deep learning, Medical image segmentation, Inconsistent
labels.

##  I Introduction

Convolutional neural network (CNN) based deep learning (DL) as a data-driven
methodology has demonstrated unparalleled performance in various segmentation
tasks, providing that the model can train on large-scale data with sufficient
diversity. To suffice the large appetite of CNNs, researchers often collect
data from multiple sources to jointly train a model for improved performance.
However, in the healthcare domain, such _centralized learning_ paradigm is
often impractical because the clinical data cannot be easily shared across
different institutions due to the regulations, such as Health Insurance
Portability and Accountability Act (HIPAA). To overcome the barrier of data
privacy and realize large-scale DL on isolated data, federated learning (FL)
[1], an emerging _decentralized learning_ paradigm, has been adopted in the
medical image analysis, solving various clinical problems such as prostate
segmentation [2, 3] and COVID-19 diagnosis [4, 5].

FL allows different data-owners to collaboratively train one global DL model
without sharing the data. The model training is completed by iterating over a
server node and several client nodes. Each client individually trains a copy
of the global model using their local data after the server updates the global
model by aggregating the locally trained models. By repeating this process,
the global model can effectively absorb the knowledge contained in the client
datasets without data sharing. As data privacy has become a critical issue
concerned by healthcare stakeholders, FL attracts a growing attention from
both the research and clinical communities in recent years.

Although the capability of FL for medical image analysis has been demonstrated
by the prior studies, it comes with limitations. A critical issue is about the
strict requirement for the label consistency. Specifically, in an FL system,
all the participating sites need to have identical regions-of-interest (ROIs)
annotated on their local data, so the FL model can be optimized following the
same objective across different clients. However, in practical scenarios,
different clinical sites often have different expertise and thus follow
different protocols for data annotation. This leads to inconsistent ROI labels
across different sites. The requirement for labeling consistency hinders the
FL methods from large-scale deployment in practice. Therefore, a more flexible
FL framework supporting the training with inconsistently labeled client data
is highly desired. From a technical point of view, this is an FL problem with
partial labels since each client merely has a partially labeled dataset with
respect to the whole set of ROIs in the federation. To the best of our
knowledge, this is a new problem that has not yet been explored before, but at
the same time is of great clinical significance and technical urgency to
tackle.

In this paper, we propose a novel method, called federated multi-encoding
U-Net (Fed-MENU), to address the above challenge. We then demonstrate its
performance on a representative task, _i.e._ , multiple abdominal organ
segmentation from computed tomography (CT) images. The underlying assumption
of our design is that, although the data from different clients show
disparities in terms of the labeled ROIs, they share the same or similar image
contents and thus can provide complementary information to facilitate the
learning of robust features in a unified FL system. Since the client datasets
are partially labeled with different ROIs, they can be seen as a set of
experts with different expertise. Each expert focuses on learning the features
within its expertise (labeled ROIs) while avoiding introducing the unreliable
or noisy information from the non-expertise (unlabeled ROIs). To achieve this
goal, we design a multi-encoding U-Net (MENU-Net) to decompose the multi-organ
feature learning task into a series of individual sub-tasks of organ-specific
feature learning. Each of them is bound with a sub-network in the MENU-Net.
During the local training stage of FL, the client node can only tune the sub-
network associated with the organ on which they have labels (expertise) while
keeping the other parts of the network unchanged. Due to this decomposed
feature learning strategy, the MENU-Net is encouraged to learn organ-specific
features via different sub-networks without interference from other label-
absent organs. Moreover, to further encourage the extracted organ-specific
features to be informative and distinctive from other organs, we design an
auxiliary generic decoder (AGD) to regularize the training of the MENU-Net. As
a result, the MENU-Net can extract discriminative features during its encoding
stage and thus facilitate the segmentation in the following decoding stage.

To demonstrate the performance of our Fed-MENU method, we conducted extensive
experiments using four public abdominal CT datasets, each of which is
annotated with a different set of abdominal organs. Our results show that,
without sharing the raw data, the proposed Fed-MENU method can effectively
utilize the isolated datasets with different partial labels to train a global
model for multi-organ segmentation. The performance of the trained model is
superior to the localized learning model trained by any single dataset and
also the centralized learning model trained by combining all the datasets. The
main contributions of this paper are three-fold.

  * •

We addressed a new problem in FL to enable collaboratively training of a
global model using isolated datasets with inconsistent labels.

  * •

We proposed a novel Fed-MENU method to deal with this challenging problem. In
our method, organ-specific features are individually extracted by different
sub-networks of the MENU-Net and further enhanced by the accompanied AGD.

  * •

We evaluated the performance of our Fed-MENU method for multi-organ
segmentation using six public abdominal CT datasets. Our experimental results
on both in-federation and out-of-federation testing sets demonstrated the
effectiveness and superior performance of our design.

The rest of this paper is organized as follows. Section II gives a brief
review of the previous literature related to this work. Section III
illustrates the details of the proposed method. Section IV presents the
experiments and results on four public datasets. Finally, we discuss the
limitation of this work and conclude it in Section V.

##  II Related works

Since our study involves both FL-based medical image analysis and DL-based
image segmentation with partially labeled data, we review the related works in
these two areas before presenting our proposed method.

###  II-A Background of federated learning

FL, first emerged in 2017 [1], is a decentralized learning paradigm designed
to address data privacy issue in deep learning. Unlike the conventional
centralized deep learning that requires all training data gathered on a server
node, FL allows distributed data owners to collaboratively train a model
without sharing their data. Federated averaging (FedAvg) [1] was acknowledged
as the benchmarking algorithm in FL, which defined the framework for the
following studies in this field. In FedAvg, the model is simultaneously
trained on several client devices and the clients transmit the trained model
(parameters) to the server to update/maintain the global model (parameters). A
major challenge in FL is how to deal with the client data that is not
independently and identically distributed (non-iid) or heterogeneous [6, 7, 8,
9, 10, 11, 12]. For example, Li _et al._ [7] derived FedProx from FedAvg by
regularizing the local models to be closer to the global model, in which way
the method showed more stable and accurate performance when learning over the
heterogeneous data. Acar _et al._ [8] proposed FedDyn by introducing a dynamic
regularization term to the local training objective of FedAvg. The resulting
model achieved not only better robustness against data heterogeneity but also
higher communication efficiency. Li _et al._ [9] proposed a simple yet
effective scheme, FedBN, to address the heterogeneity issue in feature space
by keeping the batch normalization layers locally updated. Except for the
above FL algorithms, data harmonization [13] can also be considered as a way
to address the heterogeneities of multi-center data in FL.

###  II-B Federated learning for medical image analysis

In recent years, FL has been widely applied in various medical image analysis
tasks to address the conflict between large-scale DL model training and
healthcare data privacy. For example, Dayan _et al._ [5] trained an FL model
for COVID-19 clinical outcome prediction using chest X-ray images collected
from 20 institutes across the globe. The experimental results showed that, by
utilizing multi-site data, the trained FL model achieved an average area under
the curve (AUC) around 0.920 for predicting the patients’ oxygen therapy
categories after 24- and 72-hour periods from initial admission to the
emergency department. This AUC value of the FL model is 16% higher than the
average AUC (0.795) of the 20 locally trained models with single-site data.
Xia _et al._ [14] derived an auto-FedAvg method from FedAvg algorithm [1] for
medical image segmentation using data that are not independently and
identically distributed (non-iid). Their method automatically learns the
aggregation weights of each client based on the client data distribution. Roth
_et al._ [2] combined FL with neural architecture search strategy to develop a
super-network with a self-learned structure for whole prostate segmentation
from multi-institute magnetic resonance imaging (MRI) data. Recently, Yang _et
al._ [4] extended FL to semi-supervised learning paradigm and applied it to
COVID region segmentation using chest CT data from three nations. Another work
by Park _et al._ [15] employed the Vision Transformer (ViT) architecture [16]
in an FL framework to diagnose the COVID-positive cases from chest X-ray
images.

The prior studies [17, 18, 5, 14, 2, 4, 19, 20, 12] have demonstrated the
feasibility and effectiveness of FL in solving the problem of large-scale DL
model training without data sharing in the healthcare community. However,
these successes are built upon a prerequisite that the participating sites
have consistently labeled data so that they can contribute to the same
training objective of one global model. However, such a strong requirement may
not be met in the real scenarios, where different clinical sites often follow
different protocols to annotate their local data. Although some prior efforts
like FedMix proposed by Wicaksana _et al._ [21] allowed mixed supervised FL
with different levels of label (_e.g._ , pixel-level mask, bounding-box-level
annotation, and image-level labels), the gap caused by the label inconsistency
(in terms of the target classes) remained unresolved, which largely narrows
the application scope of the FL-based methods for medical image analysis and
thus motivates our study in this work.

It is worth noting that Dong _et al._ [22, 23] worked on a similar problem of
federated learning-based X-ray image classification with partially labeled
data, which is related to our work and thus provided positive implications for
us during the revision of this paper. However, there are significant
differences between these two works. First of all, Dong _et al._ ’s work is
targeting at the image classification task while our work focused on the image
segmentation task, which is more general and common in terms of the scope of
clinical application. As a consequence, the methods proposed in these two
works are fundamentally different and thus cannot be directly compared.
Moreover, the method by Dong _et al._ was only evaluated in an in-federation
setting, _i.e._ , the testing data follows the distribution of the training
data from one of the clients, while our study also includes an out-of-
federation evaluation, _i.e._ , the testing data comes from an unseen domain
out of any one of the client datasets, which is more exhaustive and
challenging.

###  II-C Medical image segmentation with partial labels

Due to the high cost of data annotation, medical images are often partially
labeled with different ROIs or labels, even though they may share the same
imaging field. This created a barrier for the DL-based medical image
segmentation methods [24, 25] when the model is trained on the partially
labeled data. To solve this problem, Yan _et al._ [26] developed a universal
lesion detection algorithm trained by CT images labeled with different lesion
types. In their method, the missing labels were mined by exploiting clinical
prior knowledge and cross-dataset knowledge transfer. Recently, Petit _et al._
[27] proposed to conduct partially labeled DL training through an iterative
confidence relabeling method, in which a self-supervised scheme was employed
to iteratively relabel the missing organs by introducing pseudo labels into
the training set. For the partial label problem in segmentation tasks, Fang
and Yan [28] proposed a pyramid-input-pyramid-output feature abstraction
network (PIPO-FAN) for multi-organ segmentation, in which a target adaptive
loss is integrated with a unified training strategy to enable image
segmentation over multiple partially labeled datasets with a single model. Shi
_et al._ [29] designed two types of loss functions, namely marginal loss and
exclusion loss, to train a multi-organ segmentation network using a set of
partially labeled datasets. Furthermore, Zhang _et al._ [30] proposed an
ingenious approach, namely dynamic on-demand network (DoDNet), to achieve
multi-organ segmentation with partially labeled data. Unlike the conventional
deep neural networks with fixed parameters after training, the DoDNet can
dynamically generate model parameters to adapt to different organ segmentation
tasks. This innovative design largely improved the model efficiency and
flexibility.

Although the problem of training DL models using partially labeled data [26,
27, 28, 29, 30, 31] had been studied in the context of centralized learning,
it is still an unexplored area for the field of FL. In this paper, we present
the Fed-MENU method, which can learn from partially labeled data distributed
on different sites.

##  III Method

Figure 1: Scheme of the proposed Fed-MENU method. For better understanding,
the method is presented using an example scenario where three clinical sites
with partially labeled abdominal CT images collaboratively train an FL model
for multi-organ segmentation of the liver, kidney, and pancreas. The left
panel of the figure shows the federation structure. The right panel details
the local training procedure on Client node 3, where the pancreas is the only
labeled organ in this client dataset. For brevity, we present only the
connections between the third sub-encoder (colored in blue) and the shared
decoder, as well as the AGDs. In the final implementation, all sub-encoders
are connected to the shared decoder and AGDs at equivalent positions.

Fig. 1 gives an overview of the proposed Fed-MENU method. For a better
understanding, it is presented in a specific scenario where the federation
contains three client nodes and each node possesses a CT dataset partially
labeled with one of the abdominal organs (liver, kidney, and pancreas). We
then describe the FL framework adopted by our method in Section III-A. The
technical innovation of this work resides in the MENU-Net designed for organ-
specific feature extraction (Section III-B) and the AGD designed for organ-
specific feature enhancement (Section III-C). The training configurations and
other technical details of the proposed method are provided in Section III-D.

###  III-A Federated learning framework

Our method inherits the framework of the FedAvg [1] algorithm, which is a
commonly adopted benchmark in FL. It consists of one server node and $K$
client nodes. The server takes the responsibility of coordinating the
communication and computation across different clients, while the clients
focus on model training using their local data and devices. Given the
universal label set of $M$ organs, each of the client nodes has a local
dataset $D_{k}$ partially labeled with a subset of $N_{k}$($\leq$$M$) organs.
Our goal is to train a $M$-organ segmentation network $F(\cdot,\theta)$ using
the partially labeled datasets $\\{D_{k}\\}_{k=1}^{K}$, which reside in
different client nodes and cannot be shared or gathered for centralized
training.

The training process of the FL framework consists of $T$ rounds of
communication between the server and the client nodes. Specifically, in the
$t$-th communication round, each client node $k$ will first download the
parameters of the current segmentation network $F(\cdot,\theta^{t})$ on server
(denoted as global model), resulting in a local copy of $F(\cdot,\theta^{t})$
(denoted as local model). The client node then trains the local model using
its local dataset $D_{k}$ and device for $E$ epochs. After the local training,
the server collects the trained local models $F(\cdot,\theta_{k}^{t})$ from
all the $K$ client nodes and aggregates them into a new global model through a
parameter-wise averaging:

$\theta^{t+1}=\sum_{k}^{K}\frac{|D_{k}|}{\sum_{j}^{K}|D_{j}|}\cdot\theta_{k}^{t},$
(1)

where $|D_{k}|$ indicates the sample size of dataset $D_{k}$. The whole FL
procedure is accomplished by repeating the above process until the global
model $F(\cdot,\theta^{t})$ converges.

Because the local models are individually trained on the client nodes, the
server node only needs to transfer the model parameters instead of the raw
data from the clients. That helps the FL model obtain knowledge contained in
the isolated client datasets without violating the data privacy. However,
since the original FedAvg framework is designed for FL with fully labeled
data, there is a technical gap to bridge before we can deploy it to the case
of partially labeled data. Therefore, we propose the following MENU-Net with
AGD to meet the need.

###  III-B MENU-Net for organ-specific feature extraction

A straightforward way to train a multi-organ segmentation network using the
partial labels is to calculate the training loss merely using the labeled
organs while ignoring the unlabeled ones. However, in the case of FL with
partial labels, different organs are labeled in the client datasets, focusing
on different expertise for image segmentation. As a consequence, the
segmentation performance of the trained local models would be biased to the
labeled organs. Intuitively, a uniform aggregation of all local models (see
Eq. 1) may weaken the expert client models. A more reasonable way may be to
promote the strength of the local models on segmenting the labeled organs and
avoid the interference from their weaknesses with the unlabeled organs.

To achieve the above goal, we designed the MENU-Net to explicitly decompose
multi-organ feature extraction into several individual processes of organ-
specific feature extraction. As illustrated in the right panel of Fig. 1, the
MENU-Net consists of $M$ sub-encoders
$\\{g_{m}(\cdot\,|\,\theta_{g_{m}})\\}_{m=1}^{M}$ followed by a shared decoder
$f(\cdot\,|\,\theta_{f})$ as well as the skip connections between them. Fig. 1
shows an example with $M$=3. Each sub-encoder
$g_{m}(\cdot\,|\,\theta_{g_{m}})$ serves as an organ-specific feature
extractor for the $m$-th organ. An input image $x$ is fed to all the sub-
encoders to get the features of all the organs in parallel. The extracted
features are then concatenated into one stream and fed to the shared decoder
$f(\cdot\,|\,\theta_{f})$, which is used to interpret the organ-specific
features into the multi-organ segmentation masks. This process is formally
defined as

$\displaystyle F(x)$ $\displaystyle=f((g_{1}(x)\oplus g_{2}(x)\ldots\oplus
g_{M}(x)))$ $\displaystyle=f\left(\oplus_{m=1}^{M}g_{m}(x)\right),$ (2)

where $\oplus$ denotes channel-wise concatenation.

Given a client node with partial labels on the $m$-th organ, the local model
training only tunes the parameters in the sub-encoder
$g_{m}(\cdot\,|\,\theta_{g_{m}})$ and the shared decoder
$f(\cdot\,|\,\theta_{f})$, which can be expressed as:

$\underset{\theta_{f},\theta_{g_{m}}\subset\theta}{\arg\min}\
L_{sup}\left[F_{m}(x\,|\,\theta),\ \hat{y}_{m}\right],$ (3)

where $F_{m}(x)$ and $\hat{y}_{m}$ are the predicted and ground-truth
segmentation of the $m$-th organ, respectively. $L_{sup}$ denotes the
supervised loss measuring the similarity between the prediction and ground
truth, which can be in any form of pixel-wise loss functions (_e.g._ , Cross-
entropy loss and Dice loss [32]) or their combinations.

TABLE I: Detailed information of client datasets involved in this study. (L:
liver, K: kidney, P: pancreas, S: spleen, G: gallbladder, ✓: labeled organ,
$\times$: unlabeled organ, OoF: out-of-federation)

[b]  Datasets Image information Labeled organ Imaging phase # of CT images
(train/val./test) Slice size [in pixel] # of slice [per image] Spacing [$mm$]
Thickness [$mm$] L K P S G Client #1: Liver 131 (79/13/39) 512 74$\sim$987
0.56$\sim$1.00 0.70$\sim$5.00 $\checkmark$ $\times$ $\times$ $\times$ $\times$
Portal-venous phase Client #2: Kidney 210 (126/21/63) {512,796} 29$\sim$1059
0.44$\sim$1.04 0.50$\sim$5.00 $\times$ ✓ $\times$ $\times$ $\times$ Late
arterial contrast phase Client #3: Pancreas 281 (169/28/84) 512 37$\sim$751
0.61$\sim$0.98 0.70$\sim$7.50 $\times$ $\times$ ✓ $\times$ $\times$ Portal-
venous phase Client #4: Spleen 41 (24/5/12) 512 31$\sim$168 0.61$\sim$0.98
1.50$\sim$8.00 $\times$ $\times$ $\times$ ✓ $\times$ Portal-venous phase
Client #5: AMOS 200 (120/20/60) 60$\sim$768 64$\sim$512 0.45$\sim$3.00
0.82$\sim$5.00 ✓ ✓ ✓ ✓ ✓ See Appendix B.1 in [33] OoF client: BTCV 30 (0/0/30)
512 85$\sim$198 0.59$\sim$0.98 2.50$\sim$5.00 ✓ ✓ ✓ ✓ ✓ Portal-venous phase
Total 893(518/87/288) 60$\sim$796 29$\sim$1059 0.44$\sim$3.00 0.50$\sim$8.00 -
- - - - -

###  III-C AGD for organ-specific feature enhancement

Benefitting from the decomposed feature extraction in MENU-Net, organ-specific
knowledge from the labeled clients (experts) can be individually learned by
different sub-encoders with less interference from other unlabeled clients
(non-experts). However, since each local model can only see the images from
one client dataset, the organ-specific features learned by the sub-encoders
may also include some domain-specific information. This is unfavorable for the
subsequent shared decoder. Ideally, the sub-encoder should focus on the
structural information of the corresponding organ, which is invariant across
domains. In addition, the extracted organ-specific features should be
distinctive enough from that of other organs, making the subsequent shared
decoder easier to interpret them into different organs’ segmentation.

Motivated by the above idea, we design the AGD shared across different sub-
encoders to help regularize the training of our MENU-Net, aiming to enhance
the extracted organ-specific features. Specifically, as illustrated in the
mid-bottom of Fig. 1, given the organ-specific features $g_{m}(x)$ extracted
by an arbitrary sub-encoder $g_{m}(\cdot\,|\,\theta_{g_{m}})$, we feed them to
a set of AGDs to perform an organ-agnostic (binary) segmentation of the $m$-th
organ:

$G_{m}(x)=h(g_{m}(x)).$ (4)

Here, we denote the collection of all AGDs as $h(\cdot\,|\,\theta_{h})$
parameterized by $\theta_{h}$. The AGD has a lite structure consisting of
three convolutional layers. The first two convolutional layers contain
3$\times$3$\times$3 kernels followed by instance normalization [34] and
LeakyReLU activation [35]. The last convolutional layer has
1$\times$1$\times$1 kernels with two output channels followed by a softmax
activation. Due to the shallow structure of AGD, it has limited representation
ability and thus enforces the preceding sub-encoder layers to extract more
discriminative features to approach the organ segmentation. On the other hand,
since the AGD is working as a generic decoder for different organs when
combined with different sub-encoders, the sub-encoders are encouraged to
extract features distinctive enough from each other so that the following AGD
can interpret them into the corresponding organ’s segmentation without extra
information.

During the local training stage, the AGDs are tuned together with different
sub-encoders over all client nodes by minimizing the segmentation error
between the output $G_{m}(x)$ and the corresponding ground truth
$\hat{y}_{m}$. This process can be expressed as:

$\underset{\theta_{h},\theta_{g_{m}}}{\arg\min}\
L_{aux}\left[G_{m}(x\,|\,\theta_{h},\theta_{g_{m}}),\ \hat{y}_{m}\right],$ (5)

where $L_{aux}$ denotes the auxiliary loss quantifying the binary segmentation
error between the AGDs’ output and the corresponding ground truth. We
calculate $L_{aux}$ as a sum of the Cross-entropy loss and Dice loss [32].
Note that, since the AGDs are connected to the multi-scale levels of the sub-
encoder, they have multiple outputs in different scales. All these outputs
will be resampled to the original size of the ground-truth segmentation
$\hat{y}_{m}$ and counted in the auxiliary loss in Eq. 5. After the local
training, the tuned parameters of AGDs from all clients are aggregated through
the FedAvg algorithm as shown in Eq. 1, which is the same as that of the
shared decoder $f(\cdot\,|\,\theta_{f})$. The AGDs are used to regularize the
training of the proposed MENU-Net. During the inference stage, we only utilize
the trained MENU-Net to infer the multi-organ segmentation masks.

###  III-D Implementation details

The proposed method is implemented for 3D segmentation using PyTorch. Model
parameters in the segmentation network are initialized using the Xavier
algorithm [36] and optimized by an SGD optimizer. The learning rate is
initialized to be 0.01 and decayed throughout the training following a poly
learning rate policy [37] with a momentum factor of 0.99. We train the model
for $T$=400 rounds of communication ($E$=1 epoch of local training per round)
and evaluate its performance on the validation set every epoch using Dice
similarity coefficient (DSC) as the metric. The model achieving the highest
DSC on the validation set is selected as the final model to be evaluated on
the testing set. The training batch size is set to 4 on four NVIDIA A100 GPUs.
The CT images are resampled to a uniform resolution of
1.0$\times$1.0$\times$1.5$mm^{3}$ (pixel spacing of 1.0$\times$1.0$mm^{2}$ and
slice thickness of 1.5$mm$) before training. Image patches with a fixed pixel
size of 256$\times$256$\times$32 are randomly cropped from the resampled CT
images as the input of the segmentation network. The image intensities are
normalized from [-200.0, 400.0] Hounsfield units (HU) to [0.0, 1.0] for a good
soft-tissue contrast. Random translation ([-20,20]$mm$) and rotation
([-0.1,0.1]$rad$) are used to augment the training samples. In the inference
time, an unseen CT image is first resampled to the resolution of
1.0$\times$1.0$\times$1.5$mm^{3}$ and divided into a series of patches, which
are then fed to the trained segmentation network. The predicted patch-wise
segmentation will be assembled to get the final segmentation mask, which is
then resampled back to their original CT resolution for quantitative
evaluation. Unless otherwise noted, all the competing methods and ablation
models in our experiments are trained and evaluated using the same
configuration as our method. For better reproducibility, the source code is
released at https://github.com/DIAL-RPI/Fed-MENU.

##  IV Experiments

###  IV-A Datasets

We conducted experiments using six public abdominal CT image datasets,
including 1) liver tumor segmentation challenge (LiTS) dataset
[38]111https://competitions.codalab.org/competitions/17094, 2) kidney tumor
segmentation challenge (KiTS) dataset [39, 40]222https://kits19.grand-
challenge.org/home, 3) pancreas and 4) spleen segmentation datasets in medical
segmentation decathlon challenge [41, 42]333http://medicaldecathlon.com (Task
#7 and #9, respectively), 5) multi-modality abdominal multi-organ segmentation
challenge (AMOS) dataset [33]444https://amos22.grand-challenge.org, and 6)
multi-atlas labeling beyond the cranial vault challenge (BTCV) dataset
[43]555https://www.synapse.org/#!Synapse:syn3193805/wiki/89480. For brevity,
we denote the six datasets as the liver, kidney, pancreas, spleen, AMOS, and
BTCV, respectively, in the rest of the paper. Table I shows the detailed
information of these datasets.

Liver dataset contains 131 images, whose sizes range between
[74$\sim$987]$\times$[512]$\times$[512] (in $D$$\times$$H$$\times$$W$ pixels).
The in-plane spacing of these CT slices varies from 0.56$mm$ to 1.00$mm$ and
the slice thickness varies from 0.70$mm$ to 5.00$mm$. Each CT image has a
pixel-wise annotation of liver and tumor segmentation stored in the same size
as the corresponding image. We treat the liver tumor regions as part of the
liver in our experiments.

Kidney dataset contains 210 images, whose sizes range between
[29$\sim$1059]$\times$[512]$\times${512,796} (in $D$$\times$$H$$\times$$W$
pixels). The in-plane spacing of CT slices varies from 0.44$mm$ to 1.04$mm$
and the slice thickness varies from 0.50$mm$ to 5.00$mm$. Each CT image has a
pixel-wise annotation of kidney and tumor segmentation stored in the same size
as the corresponding image. We treat the kidney tumor regions as part of the
kidney in our experiments.

Pancreas dataset contains 281 images, whose sizes range between
[37$\sim$751]$\times$[512]$\times$[512] (in $D$$\times$$H$$\times$$W$ pixels).
The in-plane spacing of these CT slices varies from 0.61$mm$ to 0.98$mm$ and
the slice thickness varies from 0.70$mm$ to 7.50$mm$. Each CT image has a
pixel-wise annotation of pancreas and tumor segmentation stored in the same
size as the corresponding image. We treat the pancreas tumor regions as part
of the pancreas in our experiments.

Spleen dataset contains 41 images, whose sizes range between
[31$\sim$168]$\times$[512]$\times$[512] (in $D$$\times$$H$$\times$$W$ pixels).
The in-plane spacing of these CT slices varies from 0.61$mm$ to 0.98$mm$ and
the slice thickness varies from 1.50$mm$ to 8.00$mm$. Each CT image has a
pixel-wise annotation of spleen segmentation stored in the same size as the
corresponding image.

AMOS dataset contains 200 images, whose sizes range between
[64$\sim$512]$\times$[60$\sim$768]$\times$[192$\sim$768] (in
$D$$\times$$H$$\times$$W$ pixels). The in-plane spacing of these CT slices
varies from 0.45$mm$ to 3.00$mm$ and the slice thickness varies from 0.82$mm$
to 5.00$mm$. Each CT image has a pixel-wise annotation of liver, kidney,
pancreas, spleen, and gallbladder segmentation stored in the same size as the
corresponding image.

BTCV dataset contains 30 images, whose sizes range between
[85$\sim$198]$\times$[512]$\times$[512] (in $D$$\times$$H$$\times$$W$ pixels).
The in-plane spacing of these CT slices varies from 0.59$mm$ to 0.98$mm$ and
the slice thickness varies from 2.50$mm$ to 5.00$mm$. Each CT image has a
pixel-wise annotation of liver, kidney, pancreas, spleen, and gallbladder
segmentation stored in the same size as the corresponding image.

For the liver, kidney, pancreas, spleen, and AMOS datasets, we randomly split
each of them into training/validation/testing sets with a fixed ratio of
60%:10%:30%, respectively. The experimental results on the five testing sets
are used for _in-federation_ evaluation, which indicates the model performance
when the testing data follows the same distribution as the training and
validation data. For the BTCV dataset, we reserve it as an _out-of-federation_
testing set, which is completely unseen to the model during training and
validation. The performance on the BTCV dataset gives a good indication of the
model’s generalization ability.

###  IV-B Metrics

We calculated the mean and standard deviation (SD) of Dice similarity
coefficient (DSC) and average symmetric surface distance (ASD) for each organ
to quantitatively evaluate the segmentation results yielded by different
methods. For each labeled organ in a certain client dataset, we first
calculate the mean value $Q_{c}^{k}$ over all cases to get the result for
organ $c$ in this client $k$. Then, the mean value of the $C_{k}$ labeled
organs $Q^{k}=1/C_{k}\sum_{c=1}^{C_{k}}Q_{c}^{k}$ is calculated as the
performance index of client $k$. Finally, the mean value of the $K$ clients
$Q=1/K\sum_{k=1}^{K}Q^{k}$ is considered as the global index indicating the
model accuracy. Instead of performing an overall case-wise averaging, this
strategy was to avoid the bias from unbalanced sample numbers among the
datasets, ensuring the global DSC and ASD calculated from each dataset with
different sizes play an equal role. We also conducted paired $t$-tests on the
above metrics to check the statistical significance between different groups
of results.

TABLE II: Quantitative performance evaluation of different methods when tested
on the in-federation data. The best results are marked in bold. The underlined
results indicate a statistically significant difference from our result
($p$$<$0.05).

[b]  Models (# of param.) DSC [Mean(SD) %] Client #1 Client #2 Client #3
Client #4 Client #5 Global Liver Kidney Pancreas Spleen Liver Kidney Pancreas
Spleen Gallbladder Localized \- Client #1 93.46(3.20) - - - - - - - - \-
Client #2 - 91.17(7.60) - - - - - - - \- Client #3 - - 76.82(13.01) - - - - -
- 88.50 \- Client #4 - - - 91.98(3.75) - - - - - \- Client #5 - - - -
95.93(2.00) 94.80(2.81) 78.91(11.06) 95.47(2.78) 80.29(19.40) Centralized \-
U-Net (97M) 95.22(2.68) 95.52(3.30) 80.50(11.82) 95.52(1.82) 96.69(1.28)
93.27(4.48) 82.25(9.37) 96.17(2.95) 84.76(17.77) 91.48 \- MENU-Net (78M)
95.27(2.81) 95.52(6.28) 81.48(9.09) 96.32(1.58) 96.99(1.00) 95.00(2.33)
83.88(8.67) 96.56(2.35) 85.20(17.98) 92.02 Federated \- U-Net (97M)
94.48(2.28) 94.54(6.18) 79.38(12.69) 93.75(4.71) 96.76(1.55) 92.40(4.56)
80.79(11.89) 95.72(2.97) 83.38(15.62) 90.39 \- MENU-Net (78M) 94.07(3.09)
95.94(3.75) 80.05(12.02) 94.65(2.98) 96.70(1.15) 93.74(2.23) 82.14(9.45)
96.34(2.09) 86.08(14.48) 91.14 [b]  Models (# of param.) ASD [Mean(SD) mm]
Client #1 Client #2 Client #3 Client #4 Client #5 Global Liver Kidney Pancreas
Spleen Liver Kidney Pancreas Spleen Gallbladder Localized \- Client #1
4.00(2.88) - - - - - - - - \- Client #2 - 2.40(4.26) - - - - - - - \- Client
#3 - - 3.01(3.82) - - - - - - 2.34 \- Client #4 - - - 1.11(0.62) - - - - - \-
Client #5 - - - - 0.95(1.08) 0.87(3.31) 1.48(1.03) 0.67(1.01) 1.99(3.33)
Centralized \- U-Net (97M) 1.95(1.63) 3.26(6.31) 2.04(2.80) 1.17(1.62)
1.02(1.82) 4.87(9.15) 1.30(1.26) 0.64(1.93) 1.52(3.26) 2.06 \- MENU-Net (78M)
1.86(2.07) 0.88(1.98) 1.85(1.49) 0.30(0.13) 0.61(0.47) 0.72(1.40) 1.19(1.28)
0.36(0.62) 1.24(2.22) 1.14 Federated \- U-Net (97M) 3.75(4.08) 3.11(6.11)
2.36(4.01) 1.54(2.72) 1.40(1.86) 2.82(6.02) 1.93(3.87) 0.45(0.60) 1.56(2.89)
2.48 \- MENU-Net (78M) 2.56(2.47) 0.79(1.03) 2.10(2.45) 0.54(0.42) 0.94(1.59)
0.80(0.89) 1.53(1.54) 0.41(0.52) 1.34(2.70) 1.40

###  IV-C Comparison with benchmarks

####  IV-C1 Benchmarks

We compared our method with three benchmarks to demonstrate its effectiveness.
The benchmarks include:

Localized learning mode: Four single-organ segmentation networks and a five-
organ segmentation network were individually trained using the liver, kidney,
pancreas, spleen, and AMOS datasets, respectively. This benchmark simulated
the scenario where the clinical sites cannot share their data with each other
and no techniques are adopted to deal with the data privacy issue during the
model training.

Centralized learning mode: A five-organ segmentation network was trained using
the combination of the liver, kidney, pancreas, spleen, and AMOS datasets.
This benchmark simulated an ideal scenario where the clinical sites can freely
share their data without any concern on the data privacy during the model
training.

Federated learning mode: A five-organ segmentation network was trained using
the FedAvg [1] algorithm with the liver, kidney, pancreas, spleen, and AMOS
datasets distributed on five client nodes. This benchmark simulated a
practical scenario where the clinical sites cannot share their data with each
other and a naive FL solution is adopted to deal with the data privacy issue
during the model training.

We adopted U-Net [44] as the segmentation network for the above models. To
ensure the performance gain comes from the design choice rather than the
additional network parameters, we extended the base channel number of the
U-Net from 32 to 96 to make it have a slightly larger number of parameters
than our method. In the leftmost column of Table II and Table III, we
specified the trainable parameters in each model. The training configuration
of all the methods followed the recommended settings by nnU-Net [37]. For the
U-Net in federated learning mode, we employed the marginal loss and exclusion
loss [29] as the training objective to adapt it to the partially labeled
client data. The core idea of the marginal/exclusion loss is to combine the
unlabeled organs with the background category and try to maximize/minimize the
overlap with the merged background/foreground ground truth, respectively,
which can be formulated as follows:

$\displaystyle L_{margin}$
$\displaystyle=L_{ce}(p^{\prime},1-\hat{y}^{\prime})+L_{dice}(p^{\prime},1-\hat{y}^{\prime})$
(6) $\displaystyle L_{exclusion}$
$\displaystyle=-L_{ce}(p^{\prime},\hat{y}^{\prime})-L_{dice}(p^{\prime},\hat{y}^{\prime})$
(7)

where $p^{\prime}$ and $\hat{y}^{\prime}$ denote the predicted segmentation of
the merged background category and ground-truth segmentation of the merged
foreground category, respectively. We utilized the sum of the Cross-entropy
loss and Dice loss [32] to quantify the overlapping regions, which is
consistent with the original nnU-Net [37] training settings. For the sake of
fairness, the same objective was also used as the supervised loss $L_{sup}$ in
our method.

TABLE III: Quantitative performance evaluation of different methods when
tested on the out-of-federation data. The best results are marked in bold. The
underlined results indicate a statistically significant difference from our
result ($p$$<$0.05).

[b]  Models (# of param.) DSC [Mean(SD) %] ASD [Mean(SD) mm] Liver Kidney
Pancreas Spleen Gallbladder Global Liver Kidney Pancreas Spleen Gallbladder
Global Localized \- Client #1 92.63(3.46) - - - - 3.70(4.93) - - - - \- Client
#2 - 85.07(16.53) - - - - 5.28(17.98) - - - \- Client #3 - - 73.78(11.47) - -
76.13 - - 6.02(7.19) - - 4.57 \- Client #4 - - - 85.01(15.57) - - - -
3.64(9.40) - \- Client #5 92.78(4.97) 89.14(16.38) 57.13(26.67) 88.18(15.80)
53.45(33.93) 3.15(5.41) 4.19(11.60) 6.73(7.94) 3.49(5.60) 5.29(9.17)
Centralized \- U-Net (97M) 95.35(4.07) 85.50(16.97) 77.44(11.05) 91.46(10.59)
68.02(32.17) 83.55 2.06(4.68) 9.09(15.33) 2.91(3.97) 2.49(5.23) 2.08(3.43)
3.73 \- MENU-Net (78M) 95.84(3.35) 88.70(15.30) 78.65(9.98) 92.57(8.89)
67.86(31.26) 84.72 1.85(4.57) 4.26(14.35) 2.48(4.15) 1.58(3.38) 3.79(13.25)
2.79 Federated \- U-Net (97M) 95.00(5.41) 88.03(15.76) 76.53(10.52)
92.37(12.44) 64.35(30.94) 83.26 2.76(6.66) 5.97(15.98) 2.59(2.71) 1.13(2.39)
1.70(1.87) 2.83 \- MENU-Net (78M) 95.21(4.11) 88.12(16.88) 77.92(10.91)
90.97(13.29) 69.45(31.61) 84.33 2.39(5.74) 4.56(14.78) 3.44(9.85) 1.38(2.64)
1.79(3.28) 2.71

####  IV-C2 In-federation performance

Table II shows the quantitative results of the in-federation evaluation. It
can be observed that, among the three learning modes, the centralized learning
models generally exhibited the upper-bound performance. The FL models
significantly outperformed the localized learning models, which justified the
feasibility of using different partially labeled datasets to collaboratively
train a multi-organ segmentation model through FL. When compared with the
baseline U-Net model, our method achieved higher global accuracy in both
centralized learning mode and FL mode, which demonstrated the superiority of
our designs. Although not all the improvement margins are statistically
significant, the standard deviation of our results is much lower than those of
the U-Net baseline model, which indicated our method had a more stable
performance on the multi-center data. We also observed that, while the region-
based DSC metric showed relatively small differences, our model performed much
better than the U-Net in terms of the distance-based ASD metric. This
phenomenon suggested that there are fewer false positives in our results. It
is worth noting that the trainable parameter number of our method is $\sim$78
million, which is 24% fewer than that of the U-Net ($\sim$97 million
parameters). This result suggests that the performance gain of our method was
coming from the design choice rather than the additional parameters. More
importantly, the smaller model size can also benefit the FL due to the lower
communication and computation costs.

####  IV-C3 Out-of-federation performance

Table III presents the quantitative results of the out-of-federation
evaluation. Similar to the in-federation scenario, the centralized learning
models showed superior accuracy over the FL and localized learning models.
However, there was a global performance degradation from the in-federation
results to the out-of-federation results due to the data distribution shift
from the training domain (the liver, kidney, pancreas, spleen, and AMOS
datasets) to an unseen testing domain (the BTCV dataset). Furthermore, we
observed that the performance gap between the centralized learning models and
the FL models decreased for the out-of-federation results, as compared to the
in-federation results. These results suggest that out-of-federation
segmentation is a more challenging task than in-federation segmentation, and
our method can effectively handle it with better performance over the baseline
U-Net model.

Figure 2: 2D Visualization of segmentation results yielded by the competing
methods. Each column shows a case from one client. For better comparison, the
ground-truth contours are also superimposed on the segmentation results as
yellow dashed lines.

####  IV-C4 Result visualization

Figs. 2 and 3 visualize the segmentation results yielded by different methods
in 2D (axial) and 3D views, respectively. Each column shows a case from one
client dataset. In the 2D views, the ground-truth contours are superimposed on
the segmentation results as yellow dashed lines for better comparison. In the
3D views, the segmentation results of the unlabeled organs are represented by
transparent meshes. It can be observed that our proposed method produced fewer
false positive results in either in-federation or out-of-federation results.

Figure 3: 3D Visualization of segmentation results by the competing methods.
Each column shows a case from one client. The segmentation of the unlabeled
organs is represented by transparent meshes. TABLE IV: Experimental results of
ablation studies on the proposed method. The best results are marked in bold.

[b]  Ablation models In-federation Out-of-federation DSC [%] ASD [mm] DSC [%]
ASD [mm] baseline 90.39 2.48 83.26 2.83 +MENU-Net 90.95 1.76 83.86 3.39 +NEMU-
Net+ALD 90.78 1.62 83.36 3.65 +NEMU-Net+AGD 91.14 1.40 84.33 2.71

Figure 4: Gradient-weighted class activation maps (Grad-CAMs) [45] generated
by the MENU-Net trained without AGD (left), with ALD (middle), and with AGD
(right). Yellow dashed lines indicate ground-truth contours. We use the
activation (output) of the third convolutional block in the sub-encoders to
calculate the Grad-CAMs.

###  IV-D Ablation study

In this section, we conducted ablation studies on the proposed method to
evaluate the effectiveness of the two key designs, including 1) the MENU-Net
for organ-specific feature extraction and 2) the AGD for organ-specific
feature enhancement. A U-Net [44] trained using the FedAvg algorithm [1] was
used as the baseline method, on which the MENU-Net and AGD components are
sequentially imposed. In addition, we also tried to replace the AGD with a
non-generic version, _i.e._ , the auxiliary decoders are locally trained on
the client nodes without parameter fusion (see Eq. 1) through the server node.
We denote it as _auxiliary localized decoder_ (ALD) in the following contents.
This ablation model is used to justify the necessity of parameter sharing in
learning organ-specific features.

The quantitative results of the ablation studies are listed in Table IV. The
observations are three-fold. 1) Compared with the baseline, the design of the
MENU-Net (Table IV, “+MENU-Net”) can effectively improve the global
segmentation indices except for the ASD metric in out-of-federation
evaluation. This result indicated a positive effect of the task decomposition
design (_i.e._ , the multi-encoder network architecture) in learning domain-
invariant features. 2) Based on the MENU-Net, the AGD (Table IV, “+MENU-
Net+AGD”) can further boost the segmentation accuracy for both in-federation
and out-of-federation evaluation. This result suggested that the auxiliary
supervisions on the intermediate layers can facilitate the learning of more
discriminative features, and thus, benefit the subsequent shared decoder for
accurate segmentation. 3) By replacing the AGD with the ALD, the accuracy of
our method (Table IV, “+MENU-Net+ALD”) dropped in both the in-federation
setting and the out-of-federation setting. This result demonstrated the
importance of parameter sharing in our AGD since the auxiliary decoder shared
across the clients enforced the preceding sub-encoders to extract
discriminative features for different organs.

TABLE V: Quantitative performance evaluation of the proposed method trained
with different communication frequencies $T$$\times$$E$. $T$ is the number of
communication rounds and $E$ is the number of local training epochs. The best
results are marked in bold.

[b]  Communication frequency $T$ $\times$ $E$ In-federation Out-of-federation
DSC [%] ASD [mm] DSC [%] ASD [mm] 25$\times$16 90.63 1.78 83.10 3.47
50$\times$8 90.64 1.92 83.45 2.95 100$\times$4 90.72 1.89 83.58 2.98
200$\times$2 90.95 1.57 83.81 3.27 400$\times$1 91.14 1.40 84.33 2.71

Fig. 4 visualized the gradient-weighted class activation maps (Grad-CAMs) [45]
produced by our MENU-Net (extracted from the third convolutional block in the
sub-encoders) when it is trained without AGD, with ALD, and with AGD,
respectively. The model trained with AGD generated activation maps with more
accurate shapes and fewer false positive regions than the other two models,
which indicated that the AGD can effectively enhance the organ-specific
features extracted by the sub-encoders with better explainability [46].

Figure 5: Average DSC of FL models trained with different communication
frequencies. The red dashed line indicates the average DSC achieved by
centralized learning model.

###  IV-E Effects of communication frequency

Communication frequency is a key factor affecting the performance of the FL
methods in practice. In the proposed method, the communication frequency is
jointly controlled by the number of communication rounds $T$ and the number of
local training epochs $E$. We successively trained our method with different
combinations of $T$$\times$$E$ to investigate the effects of communication
frequency. For the sake of fairness, the product of these two parameters are
fixed to 400, which means all models are optimized with the same number of
training batches (iterations).

The experimental results are shown in Table V and Fig. 5. It can be seen that
higher communication frequency generally brought better performance to the
trained model, which is in line with observations in other FL-based methods
[1, 14, 9, 4], and our method consistently outperformed the baseline FL U-Net
model. Considering our method is designed for the cross-silo FL [47] scenario,
where the clients (clinical sites) have stable internet connections with
sufficient bandwidth, we finally choose $T$=400 and $E$=1 in our method to
achieve higher accuracy.

TABLE VI: Quantitative performance evaluation of the proposed method when
combined with different federated learning strategies. The best results are
marked in bold.

[b]  Models In-federation Out-of-federation DSC [%] ASD [mm] DSC [%] ASD [mm]
FedAvg [1] 91.14 1.40 84.33 2.71 FedAvgM [6] 90.28 1.98 82.32 3.29 FedProx [7]
90.95 1.62 83.73 2.97 FedDyn [8] 91.33 1.56 84.60 3.23 FedSM [11] 91.15 1.41
84.38 2.70 FedNorm [10] 90.92 1.75 84.21 3.12 Fed-DoDNet [30] 91.09 1.75 83.82
2.74

###  IV-F Comparison with different FL strategies

Another important factor relating to the FL systems’ performance is the choice
of the learning strategy, which determines how the locally trained models are
optimized and aggregated to update the server model. In the prior experiments,
to focus on our own designs, we consistently used the most basic FL strategy,
_i.e._ , the FedAvg algorithm [1], for this choice. Our method is also
flexible to be combined with other FL strategies. To demonstrate this
property, we tried to deploy our method with six different FL strategies,
including FedAvg [1], FedAvgM [6], FedProx [7], FedDyn [8], FedSM [11], and
FedNorm [10]. The experimental results of these hybrid models are shown in
Table VI. It can be observed that, by combining with more advanced FL
strategies, the segmentation accuracy of our method can be further improved.
The highest DSC was achieved when our method was combined with the FedDyn [8]
algorithm. This experiment demonstrates the flexibility of our proposed
method. While the existing FL strategies mainly focus on resolving the problem
of image heterogeneities, our method is designed to deal with label
heterogeneities. Thus, the proposed method and FL strategies can complement
each other. We also compared our method with another multi-organ segmentation
method for partially labeled data, _i.e._ , DoDNet [30], which was originally
designed for centralized learning scenario and extended in this experiment by
combining with the FedAvg [1] algorithm (denoted as “Fed-DoDNet” in Table VI).
The segmentation accuracy of this competing model is slightly lower than our
model, which demonstrated the superiority of our method designs.

##  V Discussion and conclusion

In our experiments, the FL model was trained with five client datasets, each
containing varying numbers of images that were labeled with different organs.
This setup was designed to simulate real scenarios in clinical settings. Our
experimental results revealed that our proposed method is flexible and capable
of handling such complex scenarios. However, it is important to note that our
approach has a minimum criterion for the number of labeled organs required.
Specifically, each client dataset must be labeled with at least one of the
target organs. For more complicated scenarios involving unlabeled images, it
would require some additional semi-supervised or unsupervised learning
techniques, such as the teacher-student consistency learning strategy [48,
49], to deal with. Fortunately, our Fed-MENU framework is highly adaptable and
can seamlessly integrate these methods, making it possible to extend the
framework to these situations.

TABLE VII: Quantitative performance evaluation of the proposed method when the
lesion regions were treated as background/a part of the target organs. The
best results are marked in bold.

[b]  Models In-federation Out-of-federation DSC [%] ASD [mm] DSC [%] ASD [mm]
Lesion as background 89.76 1.65 84.00 2.81 Lesion as part of organ 91.14 1.40
84.33 2.71

Most of the CT images used in this study contained tumors, and we treated the
lesion regions as a part of the surrounding organ in the experiments. There
could be a potential influence of these abnormities on the performance of the
proposed method. To investigate this impact, we compared the performance of
our method when the lesion regions in the CT images were treated 1) as
background and 2) as part of the foreground organs, respectively. The
quantitative results yielded from these two settings are reported in Table
VII. It can be seen that, when the lesion regions were treated as the
background category, the resulting performance of the proposed method
degraded. The reason can be attributed to the fact that, although the lesion
regions exhibited different textures or intensities than the healthy tissues,
they may not substantially change the contour shape of the target organs.
Thus, it would be relatively easy to segment these regions as a part of the
foreground. Otherwise, if the lesions were treated as the background, it may
significantly change the structure of the target organs, which would
eventually affect the segmentation accuracy. Since the proposed method is
designed to deal with the inconsistently labeled data for FL, we can also
consider the tumor labels as a new class inconsistently distributed in
different client nodes and train our method to segment the tumors. This could
lead us to a potential application of disease diagnosis. Naturally, to be
adaptive to the task of diagnosis (classification), the output branch of the
proposed MENU-Net may be replaced with a global pooling layer followed by
fully-connected layers. As this study mainly focused on the application of
multi-organ segmentation, this direction will be explored in our future
research.

The experiments conducted in this study are done in a virtual environment, in
which we focus on the training problem of inconsistently labeled data. In
practical scenarios, there are some security issues that should be considered.
For example, encryptions are required to secure the communications between
server and client nodes. Unencrypted communication would raise a risk of
information leakage and interference from malicious attacks [50]. In addition,
challenges of data memorization [51] as well as adversarial inference [52]
also raise privacy concerns to the FL methods. It has been demonstrated that
an FL model can be maliciously manipulated to reconstruct the private training
data if no protective measures are taken [53]. To address this issue,
differential privacy (DP) [54] techniques can be used to enhance the FL
methods by introducing noise to their input data, computation results, or
optimization process [55]. Since the goal of this study is to develop an FL
framework to enable collaborative learning on inconsistently labeled data, we
did not draw extra efforts to the security issue. Fortunately, our method is
compatible to the above-mentioned techniques and thus can be combined with
them to achieve secure FL.

One potential limitation of the proposed method could be its scalability. In
this study, we designed our method for a five-organ segmentation task.
However, when more organs are desired in the segmentation task, more encoders
are required in the MENU-Net. As a consequence, the concatenation of the
output feature maps from the multi-encoders would increase model complexity
and GPU memory consumption. One solution to this issue could be using one
single shared encoder combined with conditional input to replace the multi-
encoder design. This would result in a variant of our method sharing the
DoDNet structure [30]. In TableVI, we reported the experimental results of
this variant model (denoted as “Fed-DoDNet”), which demonstrated the
feasibility of this “shared-encoder+conditional input” design.

In this paper, we revealed and defined a new problem of FL with partially
labeled data in the context of medical image segmentation, which is of great
clinical significance and technical urgency to solve. Subsequently, a novel
Fed-MENU method was proposed to tackle this challenging problem. Compared with
the conventional FL framework that worked on the fully-labeled data, our Fed-
MENU method had two key designs to adapt to the partially labeled client
datasets, 1) the MENU-Net for organ-specific feature extraction and 2) the AGD
for organ-specific feature enhancement. Extensive experiments were conducted
using six public abdominal CT image datasets. The experimental results
comprehensively demonstrated the feasibility and effectiveness of our method
in solving the partial label problem in the context of FL.

## References

  * [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [2] H. R. Roth, D. Yang, W. Li, A. Myronenko, W. Zhu, Z. Xu, X. Wang, and D. Xu, “Federated whole prostate segmentation in MRI with personalized neural architectures,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 357–366.
  * [3] K. V. Sarma, S. Harmon, T. Sanford, H. R. Roth, Z. Xu, J. Tetreault, D. Xu, M. G. Flores, A. G. Raman, R. Kulkarni _et al._ , “Federated learning improves site performance in multicenter deep learning without data sharing,” _Journal of the American Medical Informatics Association_ , vol. 28, no. 6, pp. 1259–1264, 2021.
  * [4] D. Yang, Z. Xu, W. Li, A. Myronenko, H. R. Roth, S. Harmon, S. Xu, B. Turkbey, E. Turkbey, X. Wang _et al._ , “Federated semi-supervised learning for COVID region segmentation in chest CT using multi-national data from china, italy, japan,” _Medical image analysis_ , vol. 70, p. 101992, 2021\.
  * [5] I. Dayan, H. R. Roth, A. Zhong, A. Harouni, A. Gentili, A. Z. Abidin, A. Liu, A. B. Costa, B. J. Wood, C.-S. Tsai _et al._ , “Federated learning for predicting clinical outcomes in patients with COVID-19,” _Nature medicine_ , vol. 27, no. 10, pp. 1735–1743, 2021.
  * [6] T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” _arXiv preprint arXiv:1909.06335_ , 2019.
  * [7] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” _Proceedings of Machine learning and systems_ , vol. 2, pp. 429–450, 2020.
  * [8] D. A. E. Acar, Y. Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” _arXiv preprint arXiv:2111.04263_ , 2021.
  * [9] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “FedBN: Federated learning on non-iid features via local batch normalization,” _arXiv preprint arXiv:2102.07623_ , 2021.
  * [10] T. Bernecker, A. Peters, C. L. Schlett, F. Bamberg, F. Theis, D. Rueckert, J. Weiß, and S. Albarqouni, “Fednorm: Modality-based normalization in federated learning for multi-modal liver segmentation,” _arXiv preprint arXiv:2205.11096_ , 2022.
  * [11] A. Xu, W. Li, P. Guo, D. Yang, H. R. Roth, A. Hatamizadeh, C. Zhao, D. Xu, H. Huang, and Z. Xu, “Closing the generalization gap of cross-silo federated medical image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 866–20 875.
  * [12] X. Xu, H. H. Deng, T. Chen, T. Kuang, J. C. Barber, D. Kim, J. Gateno, J. J. Xia, and P. Yan, “Federated cross learning for medical image segmentation,” in _Medical Imaging with Deep Learning_ , 2023.
  * [13] Y. Nan, J. Del Ser, S. Walsh, C. Schönlieb, M. Roberts, I. Selby, K. Howard, J. Owen, J. Neville, J. Guiot _et al._ , “Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions,” _Information Fusion_ , 2022.
  * [14] Y. Xia, D. Yang, W. Li, A. Myronenko, D. Xu, H. Obinata, H. Mori, P. An, S. Harmon, E. Turkbey _et al._ , “Auto-FedAvg: Learnable federated averaging for multi-institutional medical image segmentation,” _arXiv preprint arXiv:2104.10195_ , 2021.
  * [15] S. Park, G. Kim, J. Kim, B. Kim, and J. C. Ye, “Federated split task-agnostic vision transformer for covid-19 cxr diagnosis,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 24 617–24 630, 2021.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [17] H. R. Roth, K. Chang, P. Singh, N. Neumark, W. Li, V. Gupta, S. Gupta, L. Qu, A. Ihsani, B. C. Bizzo _et al._ , “Federated learning for breast density classification: A real-world implementation,” in _Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning_. Springer, 2020, pp. 181–191.
  * [18] Q. Liu, C. Chen, J. Qin, Q. Dou, and P.-A. Heng, “FedDG: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1013–1023.
  * [19] M. Jiang, H. Yang, C. Cheng, and Q. Dou, “IOP-FL: Inside-outside personalization for federated medical image segmentation,” _arXiv preprint arXiv:2204.08467_ , 2022.
  * [20] P. Guo, D. Yang, A. Hatamizadeh, A. Xu, Z. Xu, W. Li, C. Zhao, D. Xu, S. Harmon, E. Turkbey _et al._ , “Auto-FedRL: Federated hyperparameter optimization for multi-institutional medical image segmentation,” _arXiv preprint arXiv:2203.06338_ , 2022.
  * [21] J. Wicaksana, Z. Yan, D. Zhang, X. Huang, H. Wu, X. Yang, and K.-T. Cheng, “Fedmix: Mixed supervised federated learning for medical image segmentation,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [22] N. Dong, M. Kampffmeyer, and I. Voiculescu, “Learning underrepresented classes from decentralized partially labeled medical images,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII_. Springer, 2022, pp. 67–76.
  * [23] N. Dong, M. Kampffmeyer, I. Voiculescu, and E. Xing, “Federated partially supervised learning with limited decentralized medical images,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [24] F. Shi, W. Hu, J. Wu, M. Han, J. Wang, W. Zhang, Q. Zhou, J. Zhou, Y. Wei, Y. Shao _et al._ , “Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy,” _Nature Communications_ , vol. 13, no. 1, p. 6566, 2022.
  * [25] X. Ye, D. Guo, J. Ge, S. Yan, Y. Xin, Y. Song, Y. Yan, B.-s. Huang, T.-M. Hung, Z. Zhu _et al._ , “Comprehensive and clinically accurate head and neck cancer organs-at-risk delineation on a multi-institutional study,” _Nature Communications_ , vol. 13, no. 1, p. 6137, 2022.
  * [26] K. Yan, J. Cai, Y. Zheng, A. P. Harrison, D. Jin, Y. Tang, Y. Tang, L. Huang, J. Xiao, and L. Lu, “Learning from multiple datasets with heterogeneous and partial labels for universal lesion detection in ct,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 10, pp. 2759–2770, 2020.
  * [27] O. Petit, N. Thome, and L. Soler, “Iterative confidence relabeling with deep convnets for organ segmentation with partial labels,” _Computerized Medical Imaging and Graphics_ , vol. 91, p. 101938, 2021.
  * [28] X. Fang and P. Yan, “Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 11, pp. 3619–3629, 2020.
  * [29] G. Shi, L. Xiao, Y. Chen, and S. K. Zhou, “Marginal loss and exclusion loss for partially supervised multi-organ segmentation,” _Medical Image Analysis_ , vol. 70, p. 101979, 2021.
  * [30] J. Zhang, Y. Xie, Y. Xia, and C. Shen, “DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1195–1204.
  * [31] P. Liu, L. Xiao, and S. K. Zhou, “Incremental learning for multi-organ segmentation with partially labeled datasets,” _arXiv preprint arXiv:2103.04526_ , 2021.
  * [32] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 fourth international conference on 3D vision (3DV)_. IEEE, 2016, pp. 565–571.
  * [33] Y. Ji, H. Bai, G. Chongjian, J. Yang, Y. Zhu, R. Zhang, Z. Li, L. Zhanng, W. Ma, X. Wan _et al._ , “Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation,” in _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * [34] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [35] A. L. Maas, A. Y. Hannun, A. Y. Ng _et al._ , “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1. Citeseer, 2013, p. 3.
  * [36] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proc. Thirteenth Int. Conf. Artif. Intell. and Statist._ JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.
  * [37] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” _Nature methods_ , vol. 18, no. 2, pp. 203–211, 2021\.
  * [38] P. Bilic, P. F. Christ, E. Vorontsov, G. Chlebus, H. Chen, Q. Dou, C.-W. Fu, X. Han, P.-A. Heng, J. Hesser _et al._ , “The liver tumor segmentation benchmark (lits),” _arXiv preprint arXiv:1901.04056_ , 2019.
  * [39] N. Heller, N. Sathianathen, A. Kalapara, E. Walczak, K. Moore, H. Kaluzniak, J. Rosenberg, P. Blake, Z. Rengel, M. Oestreich _et al._ , “The KiTS19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes,” _arXiv preprint arXiv:1904.00445_ , 2019.
  * [40] N. Heller, F. Isensee, K. H. Maier-Hein, X. Hou, C. Xie, F. Li, Y. Nan, G. Mu, Z. Lin, M. Han _et al._ , “The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge,” _Medical Image Analysis_ , p. 101821, 2020.
  * [41] A. L. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. Van Ginneken, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze _et al._ , “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” _arXiv preprint arXiv:1902.09063_ , 2019.
  * [42] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers _et al._ , “The medical segmentation decathlon,” _Nature communications_ , vol. 13, no. 1, p. 4128, 2022.
  * [43] B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein, “MICCAI multi-atlas labeling beyond the cranial vault–workshop and challenge,” in _Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge_ , vol. 5, 2015, p. 12.
  * [44] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [45] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [46] G. Yang, Q. Ye, and J. Xia, “Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond,” _Information Fusion_ , vol. 77, pp. 29–52, 2022.
  * [47] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021\.
  * [48] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [49] X. Xu, T. Sanford, B. Turkbey, S. Xu, B. J. Wood, and P. Yan, “Shadow-consistent semi-supervised learning for prostate ultrasound segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 6, pp. 1331–1345, 2021.
  * [50] G. A. Kaissis, M. R. Makowski, D. Rückert, and R. F. Braren, “Secure, privacy-preserving and federated machine learning in medical imaging,” _Nature Machine Intelligence_ , vol. 2, no. 6, pp. 305–311, 2020.
  * [51] C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in _Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security_ , 2017, pp. 587–601.
  * [52] D. Usynin, A. Ziller, M. Makowski, R. Braren, D. Rueckert, B. Glocker, G. Kaissis, and J. Passerat-Palmbach, “Adversarial interference and its mitigations in privacy-preserving collaborative machine learning,” _Nature Machine Intelligence_ , vol. 3, no. 9, pp. 749–758, 2021.
  * [53] G. Kaissis, A. Ziller, J. Passerat-Palmbach, T. Ryffel, D. Usynin, A. Trask, I. Lima Jr, J. Mancuso, F. Jungmann, M.-M. Steinborn _et al._ , “End-to-end privacy preserving deep learning on multi-institutional medical imaging,” _Nature Machine Intelligence_ , vol. 3, no. 6, pp. 473–484, 2021\.
  * [54] C. Dwork, A. Roth _et al._ , “The algorithmic foundations of differential privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3–4, pp. 211–407, 2014.
  * [55] A. Rajkumar and S. Agarwal, “A differentially private stochastic gradient descent algorithm for multiparty classification,” in _Artificial Intelligence and Statistics_. PMLR, 2012, pp. 933–941.