11institutetext:  Intelligent Healthcare Unit, Baidu Inc., Beijing, China
11email<EMAIL_ADDRESS>22institutetext: State Key Laboratory of Ophthalmology,
Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key
Laboratory of Ophthalmology and Visual Science, Guangzhou, China
22email<EMAIL_ADDRESS>Institute of High
Performance Computing, Agency for Science, Technology and Research, Singapore

# Dataset and Evaluation algorithm design for GOALS Challenge

Huihui Fang 11 Fei Li 22 Huazhu Fu 33 Junde Wu 11 Xiulan Zhang Corresponding
authors.22 Yanwu Xu 1⋆1⋆

###### Abstract

Glaucoma causes irreversible vision loss due to damage to the optic nerve, and
there is no cure for glaucoma.OCT imaging modality is an essential technique
for assessing glaucomatous damage since it aids in quantifying fundus
structures. To promote the research of AI technology in the field of OCT-
assisted diagnosis of glaucoma, we held a Glaucoma OCT Analysis and Layer
Segmentation (GOALS) Challenge in conjunction with the International
Conference on Medical Image Computing and Computer Assisted Intervention
(MICCAI) 2022 to provide data and corresponding annotations for researchers
studying layer segmentation from OCT images and the classification of
glaucoma. This paper describes the released 300 circumpapillary OCT images,
the baselines of the two sub-tasks, and the evaluation methodology. The GOALS
Challenge is accessible at
https://aistudio.baidu.com/aistudio/competition/detail/230.

###### Keywords:

GOALS Challenge glaucoma classification OCT layer segmentation Circumpapillary
OCT.

##  1 Introduction

Glaucoma is a chronic neurodegenerative condition that is one of the leading
causes of irreversible blindness in the world. It is a multifactorial optic
neuropathy characterized by progressive neurodegeneration of retinal ganglion
cells (RGCs) and their axons, resulting in retinal nerve fiber layer (RNFL)
attenuation, a specific pattern of damage to the optic nerve head, and visual
field loss [1]. In 2020, about 80 million people have glaucoma worldwide [2],
and this number is projected to be 111.8 million in 2040 [3]. Optical
coherence tomography (OCT) is a powerful tool for diagnosing ocular diseases
because of its no radiation, non-invasive, high resolution, high detection
sensitivity and other characteristics [4, 5]. In contrast to color fundus
images, which can only provide information about the surface of the retina,
OCT images can provide cross-sectional information about fundus structures.
The retinal structures contain RNFL, ganglion cell-inner plexiform layer
(GCIPL), inner nuclear layer (INL), outer plexiform layer (OPL), outer nuclear
layer (ONL), external limiting membrane (ELM), inner photoreceptor segment,
inner/outer photoreceptor segment junction, outer photoreceptor segment,
retinal pigment epithelium (RPE) interdigitation, RPE/Bruch’s membrane
complex, as well as choroid layer [6, 7]. In the diagnosis of glaucoma, the
disease can be judged by observing changes in the thickness of the optic nerve
fiber layer, etc., which is easier to detect early glaucoma than by observing
fundus color images. The circumpapillary OCT image corresponds to a circular
scan located around the optic nerve head, where a wealth of information about
the different retinal layers can be found [8].

Currently, there are only a limited number of OCT datasets [9, 10] available
in public, to facilitate researchers to conduct research on OCT images, we
hold a GOALS Challenge in conjunction with MICCAI 2022, aiming to provide
circumpapillary OCT images for studying layer segmentation and glaucoma
classification. This paper mainly introduces the 300 circumpapillary OCT
images released in the GOALS challenge, and provides baselines for the two
sub-tasks (Layer segmentation, and Glaucoma classification). Meanwhile, the
evaluation methods are described in detail.

##  2 Dataset

The 300 circumpapillary OCT images are randomly selected from previous
glaucoma study cohorts collected over the past five years in Zhongshan
Ophthalmic Center, Sun Yat-sen University, Guangzhou, China. The images are
all acquired by using a TOPCON DRI Swept Source OCT [11]. The acquired images
are saved in BMP format with a resolution of $1024\times 247$ or in JPG format
with $1270\times 763$. In the GOALS Challenge, we store the images in PNG
format with a resolution of $1100\times 800$. The summary of the GOALS dataset
and its population demographic are shown in Table 1.

Table 1: Summary of the GOALS dataset and the demographic of the population. |  | Person | Eyes | Age | Gender (Female)
---|---|---|---|---|---
Total dataset | Total | 66 | 99 | 45.91±15.04 | 36.40%
Glaucoma | 13 | 22 | 44.59±12.77 | 30.80%
Training set | Total | 16 | 24 | 39.08±14.08 | 31.30%
Glaucoma | 4 | 7 | 40.86±9.23 | 25%
Preliminary set | Total | 19 | 30 | 44.8±16.32 | 42.11%
Glaucoma | 4 | 7 | 42.92±17.09 | 75%
Final set | Total | 31 | 45 | 50.05±14.48 | 35.50%
Glaucoma | 5 | 8 | 50±8.48 | 0%

The GOALS dataset provides the glaucoma labels and the segmentation masks of
RNFL, GCIPL and choriod layers in the circumpapillary OCT images. The glaucoma
labels are determined by the clinical records, which can reflect the findings
from a series of eye examinations. The annotations of the layer segmentation
are manually marked by the ophthalmologists of the iChallenge-GOALS study
group. The iChallenge-GOALS study group contains ten ophthalmologists from
different hospitals who have been working in the ocular field for 5 years or
more. The 300 images are randomly divided into 2 subsets, and repeating this
operation 5 times, we obtain 10 subsets of the data, where each image appears
in 5 different subsets. The 10 subsets are randomly assigned to the 10
ophthalmologists, that is, each image is labeled by 5 different
ophthalmologists. The results of the 5 initial labeling results are then
aggregated by a more senior ophthalmologist for fusion. Specifically, The
ophthalmologist need to delineate the upper and lower margin of the RNFL,
GCIPL and choroid regions, as shown in Fig. 1(A).After that, the senior
ophthalmologist analyze the 5 initial annotations of each image, remove the
annotations with large deviations, and average the remaining initial
annotations as the final annotation result for each image. We then assigne
different pixel values to pixels within the boundaries of RNFL, GCIPL and
choroid layer to obtain the final ground truths of the layer segmentation
(RNFL: 0, GCIPL:80, choroid:160, elsewhere:255, as shown in Fig. 1(B)).

Figure 1: Schematic diagram of the annotations for RNFL, GCIPL, and choroid
layer segmentation.(A) Annotations for the boundaries of the targets; (B)
Segmentation masks.

In GOALS Challenge, 300 images are divided into three partitions according to
the patient dimension, i.e. the images acquired from the same patient’s eyes
are guaranteed to be divided into the same partition. These three data
partitions correspond to the training set, the preliminary set, and the final
set. The data in the training set contains the original OCT images and their
glaucoma labels and layer segmentation masks, which are used for training
models. While the preliminary and final sets only contain the original OCT
images, which are used for testing models in preliminary and final rounds.

##  3 Baseline

We design a baseline model for each of the two challenge sub-tasks. As shown
in Figs. 2 and 3, we utilize a U-shape network with residual concept to
achieve the layer segmentation, and utilize a ResNet50 to perform the glaucoma
classification. The baseline codes are avaliable at
https://aistudio.baidu.com/aistudio/competition/detail/230/0/related-material.

Figure 2: A baseline framework for OCT layer segmentation. Figure 3: A
baseline framework for glaucoma classification.

We implement the baselines via PaddlePaddle. During training, we use an Adam
optimizer with learning rate = $10^{-3}$ in the layer segmentation task, as
well as with learning rate = $10^{-6}$ in the glaucoma classification task.
The training procedure consist of 3000 iterations and 1000 iterations for
layer segmentation and glaucoma classification with a Nvidia Tesla V100-SXM2
GPU, respectively. The batch sizes are 8 for both tasks.

##  4 Evaluation

In this section, we introduce the evaluation metrics for the two challenge
sub-tasks. For the layer segmentation task, a DICE coefficient and a mean
Euclidean distance (MED) are used to evaluate the predicted region and
boundary, respectively. For the glaucoma classification, the weighted
combination of sensitivity (Sen), specificity (Spe), accuracy (Acc), $F_{1}$
score, and area under the receiver operating characteristic curve (AUC) are
utilized to evaluate the predicted results.

###  4.1 Task 1: Layer Segmentation

To measure the accuracy of the predicted region, we use the frequently-used
DICE coefficient in the segmentation task:

$Dice=\frac{2|X\cap Y|}{|X|+|Y|}$ (1)

where, $X$ represents the segmented target pixel point set in the ground
truth; $Y$ represents the segmented pixel point set in the prediction result;
$|X\cap Y|$ represents the intersection between $X$ and $Y$; $|X|$ and $|Y|$
represent the number of the elements of $X$ and $Y$. The formula for
calculating the score corresponding to the DICE metric is

$Score_{Dice}=DICE\times 10$ (2)

In addition to evaluating the accuracy of the region segmentation, we also
evaluate the accuracy of the boundary of the segmentation results by using
Euclidean distance, due to the importance of the boundaries between the
structural layers in the OCT images. Specifically, we first traverse each
pixel on the predicted boundary, and calculate the Euclidean distance from
each pixel to the nearest pixel on the gold standard boundary. Then the sum of
the above Euclidean distances is averaged based on the number of pixels on the
predicted boundary:

$MED=\frac{1}{N}\sum_{i=1}^{N}\sqrt{(x_{i}-x_{i}^{0})^{2}+(y_{i}-y_{i}^{0})^{2}}$
(3)

where $N$ is the number of pixels on the predicted boundary, $(x_{i},y_{i})$
is the $i$th pixel on the predicted boundary, and $(x_{i}^{0},y_{i}^{0})$ is
the nearest pixel on the boundary of the ground truth to $(x_{i},y_{i})$. The
score corresponding to the MED metric is calculated by

$Score_{MED}=(MED+1)^{-0.3}$ (4)

Since the layer segmentation task contains the segmentation of the three
regions of RNFL, GCIPL and choroid, the DICE and MED metrics of these three
regions should be taken into account in the score calculation. In addition,
because the RNFL layer is more important for the diagnosis of glaucoma, we
assign higher weights to the scores obtained from RNFL segmentation:

$Score_{task1}=0.4\times Score_{RNFL}+0.3\times Score_{GCIPL}+0.3\times
Score_{choroid}$ (5) $\begin{split}Score_{region}=&0.5\times
Score_{DICE_{region}}+0.5\times Score_{MED_{region}},\\\
&region\in\\{RNFL,GCIPL,choroid\\}\end{split}$ (6)

###  4.2 Task 2: Glaucoma Classification

For glaucoma classification, we adopt five common metrics including Sen, Spe,
Acc, $F_{1}$, and AUC:

$Sen=\frac{TP}{TP+FN}$ (7) $Spe=\frac{TN}{TP+FP}$ (8)
$Acc=\frac{TP+TN}{TP+FN+TN+FP}$ (9) $F_{1}=\frac{2\times TP}{2\times
TP+FP+FN}$ (10)

where $TP$, $TN$, $FP$ and $FN$ represent the numbers of true positive, true
negative, false positive, and false negative detection of the glaucoma.
Sen,Spe and Acc can reflect the proportions of positive samples, negative
samples and all samples predicted correctly, respectively. $F_{1}$ provides a
overall metric of the model’s ability to detect comprehensively and
accurately. And AUC reflects the classification ability of the model when the
positive and negative samples are unbalanced. In our evaluation framework,
these metrics are implemented via scikit-learn package [12], which is an open
source machine learning toolkit base on Python. Since the GOALS dataset has a
balanced distribution of positive and negative samples, we assign the lowest
weight to the AUC metric in the score calculation.

$Score_{task2}=(0.1\times AUC+0.25\times Sen+0.25\times Spe+0.2\times
ACC+0.2\times F_{1})\times 10$ (11)

Based on the results of the baseline model, we find that positive and negative
samples in the GOALS dataset have obvious distinguishable image features, and
therefore score high in Task 2. Hence, for preliminary and final rounds, a
lower weight is assigned to Task 2 in the score calculation:

$\begin{split}Score_{round}=0.8\times Score_{task1}&+0.2\times
Score_{task2},\\\ &round\in\\{preliminary,final\\}\end{split}$ (12)

Since the preliminary leaderboard is visible to the players, one can adjust
the model parameters or strategies to get the best prediction on the
preliminary set. To avoid players’ results from getting overfitting results on
the preliminary set and getting high scores, we assign lower weights to the
preliminary score when counting the total challenge scores. Hence, the total
score is:

$Score=0.3\times Score_{preliminary}+0.7\times Score_{final}$ (13)

Based on the evaluation criteria, our baselines receive 7.2802 score on the
preliminary set and 7.2398 score on the final set. The results of each
specific evaluation index are shown in Table. 2.

Table 2: The evaluation results of the baseline model on different datasets. Dataset | Preliminary set | Final set
---|---|---
Score | 7.2802 | 7.2398
Layer Segmentation | RNFL_DICE | 0.8161 | 0.8433
RNFL_ED | 4.0597 | 4.151
GCIPL_DICE | 0.6295 | 0.6234
GCIPL_ED | 3.318 | 3.6011
choroid_DICE | 0.8193 | 0.8746
choroid_ED | 8.9155 | 9.8953
Glaucoma Classification | AUC | 0.9984 | 0.9927
F1 | 0.9346 | 0.8829
ACC | 0.93 | 0.8687
SEN | 1 | 1
SPE | 0.86 | 0.74

##  5 Conclusion

In this paper, we introduce the GOALS Challenge at MICCAI 2022. In the
challenge, we focus on OCT which is a powerful imaging technology for glaucoma
diagnostics. We design two challenge sub-tasks, including OCT layer
segmentation of RNFL, GCIPL and choroid, and glaucoma classification. The
dataset collection and labeling process, as well as the result evaluation
design are described in detail in the paper. GOALS Challenge dataset and
evaluation framework are publicly accessible through the AI Studio website at
https://aistudio.baidu.com/aistudio/competition/detail/230. Participants are
welcome to join the GOALS Challenge and submit their predicted results on the
website.

## References

  * [1] Mitra Sehi, Xinbo Zhang, David S Greenfield, YunSuk Chung, Gadi Wollstein, Brian A Francis, Joel S Schuman, Rohit Varma, David Huang, Advanced Imaging for Glaucoma Study Group, et al.  Retinal nerve fiber layer atrophy is associated with visual field loss over time in glaucoma suspect and glaucomatous eyes.  American journal of ophthalmology, 155(1):73–82, 2013.
  * [2] Glaucoma: Facts and figures.  https://www.brightfocus.org/glaucoma/article/glaucoma-facts-figures.
  * [3] Yih-Chung Tham, Xiang Li, Tien Y Wong, Harry A Quigley, Tin Aung, and Ching-Yu Cheng.  Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis.  Ophthalmology, 121(11):2081–2090, 2014.
  * [4] Olena Puzyeyeva, Wai Ching Lam, John G Flanagan, Michael H Brent, Robert G Devenyi, Mark S Mandelcorn, Tien Wong, and Christopher Hudson.  High-resolution optical coherence tomography retinal imaging: a case series illustrating potential and limitations.  Journal of ophthalmology, 2011, 2011.
  * [5] Zahid Yaqoob, Jigang Wu, and Changhuei Yang.  Spectral domain optical coherence tomography: a better oct imaging strategy.  Biotechniques, 39(6):S6–S13, 2005.
  * [6] G Mohandass, R Ananda Natarajan, and S Sendilvelan.  Retinal layer segmentation in pathological sd-oct images using boisterous obscure ratio approach and its limitation.  Biomedical and Pharmacology Journal, 10(3):1585–1591, 2017.
  * [7] Felipe A Medeiros, Linda M Zangwill, Luciana M Alencar, Christopher Bowd, Pamela A Sample, Remo Susanna, and Robert N Weinreb.  Detection of glaucoma progression with stratus oct retinal nerve fiber layer, optic nerve head, and macular thickness measurements.  Investigative ophthalmology & visual science, 50(12):5741–5748, 2009.
  * [8] Gabriel García, Rocío del Amor, Adrián Colomer, and Valery Naranjo.  Glaucoma detection from raw circumpapillary oct images using fully convolutional neural networks.  In 2020 IEEE international conference on image processing (ICIP), pages 2526–2530. IEEE, 2020.
  * [9] Reza Rasti, Hossein Rabbani, Alireza Mehridehnavi, and Fedra Hajizadeh.  Macular oct classification using a multi-scale convolutional neural network ensemble.  IEEE transactions on medical imaging, 37(4):1024–1034, 2017.
  * [10] Peyman Gholami, Priyanka Roy, Mohana Kuppuswamy Parthasarathy, and Vasudevan Lakshminarayanan.  Octid: Optical coherence tomography image database.  Computers & Electrical Engineering, 81:106532, 2020.
  * [11] Dri oct triton series.  https://topconhealthcare.eu/uploads/media/60cb7b98ea585/topcon-triton-brochure-rev5-27-05-21-e325-lores.pdf.
  * [12] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  the Journal of machine Learning research, 12:2825–2830, 2011.


:
1 2022 Machine Learning for Health (ML4H) 2022

# Universal EHR Federated Learning Framework

Junu Kim<EMAIL_ADDRESS>
Kyunghoon Hur<EMAIL_ADDRESS>
Seongjun Yang<EMAIL_ADDRESS>
Edward Choi<EMAIL_ADDRESS>
KAIST

###### Abstract

Federated learning (FL) is the most practical multi-source learning method for
electronic healthcare records (EHR). Despite its guarantee of privacy
protection, the wide application of FL is restricted by two large challenges:
the heterogeneous EHR systems, and the non-i.i.d. data characteristic. A
recent research proposed a framework that unifies heterogeneous EHRs, named
UniHPF. We attempt to address both the challenges simultaneously by combining
UniHPF and FL. Our study is the first approach to unify heterogeneous EHRs
into a single FL framework. This combination provides an average of 3.4%
performance gain compared to local learning. We believe that our framework is
practically applicable in the real-world FL.

###### keywords:

Electronic Healthcare Record, Federated Learning, Multi-Source Learning,
Centralized Learning, UniHPF

##  1 Introduction

fig:prob_def

Figure 1: The application of FL with EHR data is restricted by the two
problems: (1) EHR system heterogeneity, and (2) non-i.i.d. problem. Although
multiple prior researches attempt to resolve (2), there is no known solution
for (1). Our framework is the first attempt to handle the both problems
simultaneously by combining UniHPF and FL.

Electronic healthcare record (EHR) is a rich data source that records whole
hospital events for each patient. Using EHR in machine learning (ML) allows us
to make useful predictions for patients’ future health status (Choi et al.,
2016a, b). Since the predictions can affect the life of the patients, accurate
prediction is vital. Due to the nature of ML, using more data is helpful to
improve the model accuracy (Sordo and Zeng, 2005; Prusa et al., 2015).
Considering that each EHR is maintained by each hospital, the size of the
single-source data is limited. Therefore, employing multiple hospitals’ data
(multi-source learning) is required to improve the accuracy.

Unfortunately, this is not straightforward due to the privacy issue of EHR.
Since healthcare data contains personal information, exporting data outside
the hospital is highly restricted. Therefore, traditional centralized learning
(CL) has limited practicality, because it need to gather all data into a
central server (alg:cl).

In this situation, federated learning (FL) can be a solution since it does not
need to share data among clients (hospital). It only has to share model
weights that are trained only on each client data. The global server
aggregates the weights and sends the global model to each client in each
communication round (alg:fed). For this mechanism, FL is the most appropriate
method for achieving multi-source learning on EHR while protecting patient
privacy.

Despite its benefits, the application of FL is limited due to the
heterogeneity of EHR system. In each EHR system (i.e. client, hospital), the
medical codes and the database schema are typically not shared (fig:prob_def
(1)). Therefore, most of the previous studies conduct FL experiments only with
clients using a single system (Lee and Shin, 2020; Huang et al., 2019; Yang et
al., 2022). However, these approaches are not able to handle the system
heterogeneity of the real-world EHRs. Unifying all EHR systems into a standard
format (common data model, CDM) can resolve this limitation (Rajkomar et al.,
2018; Li et al., 2019b). However, it is not yet examined due to the cost- and
time-consuming nature.

On the other hand, UniHPF (Hur et al., 2022b) is a framework that can
effectively handle heterogeneous EHR systems in the cost- and time-efficient
manner. It replaces medical codes with text and linearizes different database
schemas to mutually compatible free text format (fig:prob_def). However, the
success of UniHPF was only shown in the CL setting, which has the
aforementioned practical limitations. In FL, as opposed to CL, we have to
consider the differences among the clients’ data distributions (non-i.i.d.
problem, fig:prob_def (2)) (Rieke et al., 2020; Li et al., 2022).

Therefore, we combine UniHPF with multiple FL methods to resolve the
non-i.i.d. problem and compare among the methods. Since the performance is
increased in FL compared to without multi-source learning, we successfully
resolve both the privacy problem and the non-i.i.d. problem. Our main
contributions can be summarized as follows:

  * •

We suggest a practically applicable EHR multi-source learning framework by
combining UniHPF and FL.

  * •

Our proposed framework demonstrated improved prediction performance compared
to local learning, and even occasionally showed similar performance to
centralized learning.

  * •

To the best of our knowledge, it is the first attempt to unify heterogeneous
time-series EHRs into a single FL framework.

fig:unihpf

Figure 2: Visualization of UniHPF (Hur et al., 2022b). UniHPF first makes the
text representation of each event by linearizing the schema and replacing the
medical codes to its descriptions. The text representations are encoded
independently by the event encoder, and aggregated by the event aggregator to
make a prediction. Since UniHPF treats EHRs as free text, this is capable of
handling heterogeneous EHR systems with a single model. Note that the sub-word
tokenizer and word embedding layer is omitted in this figure.

##  2 Background and Methods

###  2.1 Federated Learning

Federated learning is a kind of distributed learning that trains a model
without sharing data among clients (McMahan et al., 2017) (alg:fed). It
enables to train the model without the hazard of data leakage by aggregating
the parameters or gradients (Brisimi et al., 2018). An obstacle to applying FL
is that the data among clients is often not independent and identically
distributed (non-i.i.d.). This makes optimizing a global model challenging
(Rieke et al., 2020). We examine four well-known FL algorithms with UniHPF.

  * •

FedAvg (McMahan et al., 2017), a de facto algorithm of FL, simply averages the
local model weights. Since this method does not fully consider the non-i.i.d.
problem, various FL algorithms have been developed.

  * •

FedProx (Li et al., 2020) regularizes the local model with $L_{2}$ distance
between local and global parameters. It prevents the weights of the local
optimal points from taking the global point.

  * •

FedBN (Li et al., 2021) handle the feature heterogeneity among clients, by
excluding the batch normalization layers from the aggregation step.

  * •

FedPxN (Yang et al., 2022) combined the advantages of FedProx and FedBN, and
it is reported to show best performance for FL with EHRs.

###  2.2 UniHPF

As mentioned earlier, the EHR system heterogeneity is the biggest obstacle to
performing multi-source learning. To overcome this problem, unifying the input
format is required. Recently, UniHPF (Hur et al., 2022b) has successfully
addressed this problem without using domain knowledge and excessive
preprocessing (fig:unihpf).

The two key concepts of UniHPF are treating EHR as free text, and utilizing
the EHR hierarchy. A patient $\mathcal{P}$ in any EHR system is composed of
multiple medical events $m_{i}\in\mathcal{P}$, and each event has its type
$e_{i}$, such as “labevents” or “prescriptions”. The events are composed of
the corresponding features, which is composed of name and value
$(n_{i,j},v_{i,j})\in m_{i}$. Some of the values are in the form of the
medical codes $c$, and these differ among the EHR systems. Thus, UniHPF
replace the code $c$ with its text description $d$ (Hur et al., 2022a). For
example, the lab measurement code “50912” can be converted into “Glucose”.
UniHPF makes a free text representation $R_{i}$ of each event $m_{i}$ by
linearizing the schema as

$R_{i}=(e_{i}\oplus n_{i,1}\oplus v_{i,1}\oplus n_{i,2}\oplus
v_{i,2}\oplus\cdots)$

, where $\oplus$ is a concatenation operator. Note that UniHPF does not
perform the feature selection, which is time- and cost-consuming. Since these
text representations are mutually compatible among the heterogeneous EHR
systems, UniHPF is a suitable framework to perform FL. To make a prediction
$\hat{y}$, UniHPF uses sub-word tokenizer (Tok) and word embedding layer
(Emb), and encodes the text-represented events individually with the event
encoder (Enc).

$z_{i}=\text{Enc}(\text{Emb}(\text{Tok}(R_{i})))$

The encoded events are aggregated by the event aggregator (Agg).

$\hat{y}=\text{Agg}(z_{1},z_{2},\cdots)$

This helps the model to understand the patient-event level hierarchy of EHRs.
Since no medical domain knowledge is used in any of the above steps, UniHPF
can unify heterogeneous EHR systems efficiently.

fig:result

Figure 3: Test AUPRC of the local learning (LL), federated learning (FL), and
centralized learning (CL) experiments. Note that the graphs are ordered by the
client size from left top to right bottom. $\star$ mark indicates the p-value
of the Student’s t-test is lower than 0.05.

##  3 Experiments and Discussion

###  3.1 Datasets

We use three open-sourced EHR datasets: MIMIC-III (Johnson et al., 2016),
MIMIC-IV (Johnson et al., 2022), and Philips eICU (Pollard et al., 2018). The
first two are composed of data from a single hospital, and the last one is a
combination of data from multiple hospitals. MIMIC-III is recorded with two
heterogeneous EHR systems, so we split it into MIMIC-III-CV (CareVue) and
MIMIC-III-MV (Metavision) based on the systems. Since different hospitals have
different data distributions, we treat the 7 largest hospitals in eICU dataset
as independent clients. To summarize, we have a total of 10 clients from 4
different EHR systems and 10 different cohorts : MIMIC-IV, MIMIC-III-CV,
MIMIC-III-MV, and 7 hospitals in eICU. Note that the clients are heterogeneous
enough in terms of the demographic information and label distributions
(apd:stat). The data is split into train, valid, and test set with 8:1:1 ratio
in a stratified manner for each task.

###  3.2 Experimental Setting

Our cohorts include the patients over 18 years of age who stayed in intensive
care unit (ICU) longer than 24 hours. We only use the first 12 hours of the
first ICU stay from each hospital admission to make predictions. We follow the
settings of Hur et al. (2022b), except that we use GRU (Chung et al., 2014) as
the event encoder of UniHPF. All experimental resources and hyperparameters
are available on github111https://github.com/starmpcc/UniFL. We adopt 5
prediction tasks from McDermott et al. (2020).

  * •

Diagnosis (Dx): Predict all categorized diagnosis codes during the whole
hospital stay of a patient.

  * •

Length of Stay (LOS3, LOS7): Predict whether a patient would stay in ICU
longer than 3 or 7 days.

  * •

Mortality (Mort): Predict whether a patient would be alive or die within 60
hours.

  * •

Readmission (Readm): Predict whether a patient would readmit to ICU within the
same hospital admission.

To compare with multi-source learning, we examine the performance of Local
Learning (LL), which is training and evaluating with each client’s data alone.
We used Area Under Precision-Recall Curve (AUPRC) as the metric. We assume a
stable internet connection and full participation because the hospitals are
generally connected by LAN. All the experiments are repeated with five random
seeds with one NVIDIA A100 80G or two RTX A6000 48G gpus.

###  3.3 Experimental Result

The experimental results are shown in fig:result. The average performance for
each task and client are reported in apd:res. First, we evaluate whether our
framework successfully handles the EHR system heterogeneity. Second, we
compare the FL algorithms with respect to the non-i.i.d. problem.

Consistent with Hur et al. (2022b), UniHPF always gets an average of 10.4%
performance increase by using CL compared with LL. This means that UniHPF
properly overcomes the EHR system heterogeneity.

Overall, the FL shows an average of 3.4% performance increase compared to LL.
This implies that UniHPF is helpful in aggregating the clients’ data into a
single FL model, demonstrating its potential since this combined method is
practically applicable in real-world heterogeneous EHRs.

We compare the performance among the algorithms with respect to the non-i.i.d.
problem. Our results agree with Choudhury et al. (2019); Niu et al. (2020),
which showed CL is the upper bound of the FL performance. In the CL setting,
the clients’ data is pooled before start the training, which prevents the
non-i.i.d. problem. Therefore, CL has the best performance among the learning
methods. In contrast, FedAvg can be treated as an empirical lower bound among
the FL algorithms with some non-i.i.d. data (Li et al., 2019a; Hsu et al.,
2019). Although FedProx is an algorithm that handles the non-i.i.d. problem,
the performance is lower than FedAvg. The reason for this result seems that
the performance of FedProx heavily depends on the hyperparameter $\mu$. The
performance of FedBN and FedPxN are higher than FedAvg, and lower than CL.
This suggests that these algorithms do address the non-i.i.d. problem in EHRs
to some extent, but not completely.

Nevertheless, our framework shows its potential when the data distribution is
extremely heterogeneous. Contrary to  the other clients, eICU-73 and eICU-443
do not have drug infusion information. Even in these extreme cases, performing
FL with FedBN or FedPxN resulted in some performance increment compared to LL.

For the training time, FL requires an average of 1.9 times more communication
rounds than CL epochs until satisfying the same early stopping criterion.
Nevertheless, the performance of FL is inferior to CL. This result denotes
that the gradient update is relatively less accurate for each communication
round. We expect that this can be improved by developing a better EHR-specific
FL algorithm.

##  4 Conclusion

In this paper, we empirically show that the combination of UniHPF and FL
successfully resolves both the EHR system heterogeneity and the non-i.i.d.
problem simultaneously. The lower performance of FL compared to CL implies
that there is still a room for improvement with a new FL algorithm in EHR. We
leave the investigation of EHR-specific pretraining with FL as our future
work.

This work was supported by Institute of Information & Communications
Technology Planning & Evaluation (IITP) grant (No.2019-0-00075), Korea Medical
Device Development Fund grant (Project Number: 1711138160,
KMDF_PR_20200901_0097), and the Korea Health Industry Development Institute
(KHIDI) grant (No.HR21C0198), funded by the Korea government (MSIT, MOTIE,
MOHW, MFDS).

## References

  * Brisimi et al. (2018) Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi.  Federated learning of predictive models from federated electronic health records.  _International journal of medical informatics_ , 112:59–67, 2018.
  * Choi et al. (2016a) Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun.  Doctor ai: Predicting clinical events via recurrent neural networks.  In _Machine learning for healthcare conference_ , pages 301–318. PMLR, 2016a.
  * Choi et al. (2016b) Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  _Advances in neural information processing systems_ , 29, 2016b.
  * Choudhury et al. (2019) Olivia Choudhury, Yoonyoung Park, Theodoros Salonidis, Aris Gkoulalas-Divanis, Issa Sylla, et al.  Predicting adverse drug reactions on distributed health data using federated learning.  In _AMIA Annual symposium proceedings_ , volume 2019, page 313. American Medical Informatics Association, 2019.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Hsu et al. (2019) Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown.  Measuring the effects of non-identical data distribution for federated visual classification.  _arXiv preprint arXiv:1909.06335_ , 2019.
  * Huang et al. (2019) Li Huang, Andrew L Shea, Huining Qian, Aditya Masurkar, Hao Deng, and Dianbo Liu.  Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records.  _Journal of biomedical informatics_ , 99:103291, 2019.
  * Hur et al. (2022a) Kyunghoon Hur, Jiyoung Lee, Jungwoo Oh, Wesley Price, Younghak Kim, and Edward Choi.  Unifying heterogeneous electronic health records systems via text-based code embedding.  In _Conference on Health, Inference, and Learning_ , pages 183–203. PMLR, 2022a.
  * Hur et al. (2022b) Kyunghoon Hur, Jungwoo Oh, Junu Kim, Min Jae Lee, Eunbyeol Cho, Jiyoun Kim, Seong-Eun Moon, Young-Hak Kim, and Edward Choi.  Unihpf: Universal healthcare predictive framework with zero domain knowledge.  _arXiv preprint arXiv:2207.09858_ , 2022b.
  * Johnson et al. (2022) Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark.  Mimic-iv.  _version 2.0). PhysioNet. 2.0). https://doi.org/10.13026/7vcr-e114._ , 2022.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Lee and Shin (2020) Geun Hyeong Lee and Soo-Yong Shin.  Federated learning on clinical benchmark data: performance assessment.  _Journal of medical Internet research_ , 22(10):e20891, 2020.
  * Li et al. (2022) Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_ , pages 965–978. IEEE, 2022.
  * Li et al. (2020) Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  _Proceedings of Machine Learning and Systems_ , 2:429–450, 2020.
  * Li et al. (2019a) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.  On the convergence of fedavg on non-iid data.  _arXiv preprint arXiv:1907.02189_ , 2019a.
  * Li et al. (2021) Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  _arXiv preprint arXiv:2102.07623_ , 2021.
  * Li et al. (2019b) Ziyi Li, Kirk Roberts, Xiaoqian Jiang, and Qi Long.  Distributed learning from multiple ehr databases: contextual embedding models for medical events.  _Journal of biomedical informatics_ , 92:103138, 2019b.
  * McDermott et al. (2020) Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzyeh Ghassemi.  A comprehensive evaluation of multi-task learning and multi-task pre-training on ehr time-series data.  _arXiv preprint arXiv:2007.10185_ , 2020.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pages 1273–1282. PMLR, 2017.
  * Niu et al. (2020) Chaoyue Niu, Fan Wu, Shaojie Tang, Lifeng Hua, Rongfei Jia, Chengfei Lv, Zhihua Wu, and Guihai Chen.  Billion-scale federated learning on mobile clients: A submodel design with tunable privacy.  In _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking_ , pages 1–14, 2020.
  * Pollard et al. (2018) Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi.  The eicu collaborative research database, a freely available multi-center database for critical care research.  _Scientific data_ , 5(1):1–13, 2018.
  * Prusa et al. (2015) Joseph Prusa, Taghi M Khoshgoftaar, and Naeem Seliya.  The effect of dataset size on training tweet sentiment classifiers.  In _2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA)_ , pages 96–102. IEEE, 2015.
  * Rajkomar et al. (2018) Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al.  Scalable and accurate deep learning with electronic health records.  _NPJ digital medicine_ , 1(1):1–10, 2018.
  * Rieke et al. (2020) Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al.  The future of digital health with federated learning.  _NPJ digital medicine_ , 3(1):1–7, 2020.
  * Sordo and Zeng (2005) Margarita Sordo and Qing Zeng.  On sample size and classification accuracy: A performance comparison.  In _International Symposium on Biological and Medical Data Analysis_ , pages 193–201. Springer, 2005.
  * Yang et al. (2022) Seongjun Yang, Hyeonji Hwang, Daeyoung Kim, Radhika Dua, Jong-Yeup Kim, Eunho Yang, and Edward Choi.  Towards the practical utility of federated learning in the medical domain.  _arXiv preprint arXiv:2207.03075_ , 2022.

##  Appendix A Federated and Centralized Learning Algorithms

In federated learning (alg:fed), each client has an individual model weight
copy. The copies are initialized with the same weights and trained locally
with corresponding client’s data. After the local training, the weights of the
clients are gathered into the central server, aggregated, and synchronized.
Each FL algorithms have different corresponding Aggregate function.

Input: clients $C_{1},\cdots,C_{N}$, data $D_{1},\cdots,D_{N}$, total epochs
$T$, learning rate $\eta$

Output: model parameter $w_{T}$

main  __:

for  _$i=1$ to $N$_ do

Copy $D_{i}$ to server;

end for

initialize server model with $w$;

$\mathcal{D}$ = shuffle$(D_{1},\cdots,D_{N})$;

for  _$t=0$ to $T-1$_ do

for  _batch $b\leftarrow(x,y)$ of $\mathcal{D}$_ do

$w\leftarrow w-\eta\nabla\mathcal{L}(w;b)$;

end for

end for

return $w$

Algorithm 1 Centralized Learning

Input: clients $C_{1},\cdots,C_{N}$, data $D_{1},\cdots,D_{N}$, total
communication rounds $T$, local epochs $L$, learning rate $\eta$

Output: model parameter $w_{T}$

main  __:

initialize client models with $w_{0}$;

for  _$t=0$ to $T-1$_ do

for  _$i=1$ to $N$_ do

$w_{t,i}\leftarrow w_{t}$;

$w_{t,i}\leftarrow$LocalTrain $(w_{t,k},D_{i})$;

end for

$w_{t+1}=$Aggregate$(w_{1},\cdots,w_{N})$;

end for

return $w_{T}$

def  _LocalTrain(_w,D_)_:

for  _$l=0$ to $L-1$_ do

for  _batch $b\leftarrow(x,y)$ of $D_{k}$_ do

$w\leftarrow w-\eta\nabla\mathcal{L}(w;b)$;

end for

end for

return $w$;

Algorithm 2 Federated Learning

##  Appendix B Clients Statistics

Table 1: Cohort Statics and Label Distributions

MIMIC-IV MIMIC-III-MV MIMIC-III-CV eICU-264 eICU-420 eICU-338 eICU-73 eICU-243
eICU-458 eICU-443 Micro Avg. Macro Avg. Cohort Size 65594 21160 16831 3637
3153 2636 2612 2423 2368 2367 12278.10 No. of Unique codes 1908 1923 3226 347
320 367 384 306 284 281 1844.43 934.60 Average No. of events per sample 112.89
91.43 100.46 53.69 87.60 47.32 58.13 51.80 58.09 51.82 99.07 71.32 Demographic
Informations Mean Ages 63.28 75.18 73.90 62.92 63.73 61.84 63.54 63.77 61.60
55.06 66.58 64.48 Gender(%) M 55.81 56.31 56.74 51.61 57.78 55.73 54.98 55.57
54.18 57.02 55.92 55.57 F 43.69 43.69 43.26 48.39 42.22 44.27 45.02 44.43
45.82 42.98 44.08 44.38 Ethnicity(%) White 67.78 72.78 70.67 87.82 86.08 92.87
75.61 64.05 64.02 42.97 70.18 72.46 Black 10.84 10.25 8.74 7.26 4.19 1.52
13.51 31.82 29.10 52.68 11.61 16.99 Hispanic 3.82 4.07 2.80 0.33 0.03 1.29
7.66 0.00 0.00 1.10 3.35 2.11 Asian 2.96 2.72 2.10 0.82 1.49 0.27 1.30 0.99
1.27 0.38 0.00 1.43 Other 14.60 10.18 15.69 3.77 8.21 4.06 1.91 3.14 5.62 2.87
14.87 7.00 Label Ratio Dx(%) 1 4.73 4.81 4.99 3.78 3.26 0.48 4.49 3.40 3.10
3.85 3.00 3.69 2 3.99 4.25 4.16 2.56 1.75 1.54 2.59 1.98 1.23 4.84 1.30 2.89 3
10.36 10.87 12.16 5.55 12.40 8.87 12.25 11.75 5.04 7.09 3.43 9.63 4 6.77 6.55
6.24 2.47 8.97 1.67 3.33 3.23 1.55 1.25 2.38 4.20 5 7.73 6.60 5.16 2.49 5.35
1.90 2.67 2.11 1.98 2.55 2.81 3.85 6 6.18 5.76 4.27 9.21 5.83 6.12 4.82 5.98
6.63 8.69 2.34 6.35 7 11.15 11.92 15.35 23.57 11.55 22.44 18.75 24.62 23.34
19.63 4.38 18.23 8 6.95 7.52 9.24 17.27 9.91 18.52 12.81 13.47 15.40 15.92
2.96 12.70 9 7.25 7.43 7.46 6.94 5.95 5.99 4.19 3.88 4.96 3.92 3.26 5.80 10
6.98 7.33 7.53 5.29 6.93 6.72 9.58 7.56 12.60 4.51 3.30 7.50 11 0.08 0.05 0.07
0.06 0.03 0.05 0.09 0.04 0.12 0.07 0.04 0.06 12 1.53 1.72 1.88 0.55 0.88 1.14
0.46 0.42 0.56 0.24 0.77 0.94 13 4.34 4.33 2.96 0.63 0.58 0.51 0.46 0.36 0.40
0.54 2.20 1.51 14 0.58 0.56 0.57 0.00 0.02 0.00 0.06 0.03 0.00 0.14 0.31 0.20
15 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.01 0.00 16 5.78 6.52
8.12 14.89 10.85 20.57 18.70 14.50 18.32 23.59 3.03 14.18 17 6.67 5.55 3.81
3.58 5.58 2.53 2.20 4.98 3.83 2.55 3.71 4.13 18 8.91 8.25 6.04 1.40 10.25 1.36
2.73 1.93 1.25 1.02 5.23 4.31 LOS3(%) true 32.50 36.93 42.24 42.86 48.24 38.66
37.10 39.50 39.74 45.59 36.26 40.34 LOS7(%) true 11.08 12.51 15.37 12.48 16.97
11.87 11.06 10.94 15.58 17.66 12.44 13.55 Mort(%) true 1.68 2.70 2.84 2.12
3.55 2.35 0.54 1.32 3.04 2.83 2.11 2.30 Readm(%) true 7.94 5.81 5.72 8.83
11.48 8.35 17.99 12.30 8.49 10.31 7.75 9.72

##  Appendix C Experimental Result

Table 2: Average performance for each task and client. The numbers in the
parentheses mean the relative performance improvement compared to the local
learning (LL). Red and blue texts mean the negative and more than 10%
increments, respectively.

Local FedAvg FedProx FedBN FedPxN Centralized Dx 0.622 0.619 ($-$0.38%) 0.616
($-$0.87%) 0.663 (+6.67%) 0.641 (+3.08%) 0.714 (+14.81%) LOS3 0.603 0.603
(+0.12%) 0.609 (+1.07%) 0.609 (+1.12%) 0.613 (+1.81%) 0.623 (+3.54%) LOS7
0.266 0.295 (+11.33%) 0.299 (+12.67%) 0.302 (+13.88%) 0.297 (+11.94%) 0.312
(+17.51%) Mort 0.153 0.167 (+9.33%) 0.157 (+2.76%) 0.162 (+5.76%) 0.158
(+3.35%) 0.166 (+8.90%) Readm 0.117 0.118 (+1.62%) 0.116 ($-$0.34%) 0.116
($-$0.75%) 0.116 ($-$0.59%) 0.129 (+10.70%) MIMIC-IV 0.414 0.428 (+3.59%)
0.410 ($-$0.72%) 0.424 (+2.45%) 0.412 ($-$0.24%) 0.417 (+0.83%) MIMIC-III-MV
0.384 0.431 (+12.15%) 0.423 (+10.33%) 0.437 (+13.81%) 0.423 (+10.10%) 0.428
(+11.61%) MIMIC-III-CV 0.41 0.416 (+1.52%) 0.411 (+0.38%) 0.423 (+3.27%) 0.409
($-$0.24%) 0.427 (+4.23%) eICU-264 0.283 0.282 ($-$0.06%) 0.289 (+2.36%) 0.299
(+6.12%) 0.300 (+6.17%) 0.336 (+18.95%) eICU-420 0.457 0.445 ($-$2.58%) 0.445
($-$2.67%) 0.458 (+0.25%) 0.450 ($-$1.49%) 0.483 (+5.65%) eICU-338 0.269 0.289
(+7.89%) 0.286 (+6.60%) 0.304 (+13.14%) 0.300 (+11.67%) 0.322 (+19.86%)
eICU-73 0.31 0.327 (+5.77%) 0.326 (+5.46%) 0.334 (+8.13%) 0.327 (+5.72%) 0.380
(+23.00%) eICU-243 0.356 0.367 (+3.15%) 0.368 (+3.43%) 0.375 (+5.39%) 0.381
(+7.14%) 0.382 (+7.29%) eICU-458 0.343 0.330 ($-$3.59%) 0.349 (+1.96%) 0.349
(+1.84%) 0.350 (+2.26%) 0.359 (+4.92%) eICU-443 0.296 0.291 ($-$1.52%) 0.286
($-$3.12%) 0.300 (+1.76%) 0.298 (+0.96%) 0.355 (+20.33%) Average 0.352 0.361
(+2.54%) 0.359 (+2.19%) 0.370 (+5.29%) 0.365 (+3.76%) 0.389 (+10.57%)