Advances in machine learning (ML), and particularly deep learning (DL), have shown promise in addressing complex healthcare problems <cit.>. However, there are concerns about generalizability on data from sources that did not participate in model training, i.e., “out-of-sample” data <cit.>. Literature indicates that training robust and accurate models requires large amounts of data <cit.>, the diversity of which affects model generalizability to “out-of-sample” cases <cit.>. To address these concerns, models need to be trained on data originating from numerous sites representing diverse population samples. The current paradigm for such multi-site collaborations is “centralized learning” (CL), in which data from different sites are shared to a centralized location following inter-site agreements <cit.>. However, such data centralization is difficult to scale (and might not even be feasible), especially at a global scale, due to concerns <cit.> relating to privacy, data-ownership, intellectual property, technical challenges (e.g., network and storage limitations), as well as compliance with varying regulatory policies (e.g., Health Insurance Portability and Accountability Act (HIPAA) of the United States <cit.> and the General Data Protection Regulation (GDPR) of the European Union <cit.>). In contrast to this centralized paradigm, “federated learning” (FL) describes an approach where models are trained by only sharing model parameter updates from decentralized data (i.e., each site retains its data locally) <cit.>, without sacrificing performance when compared to CL-trained models <cit.>. Thus, FL can offer an alternative to CL, potentially creating a paradigm shift that alleviates the need for data sharing, and hence increase access to geographically-distinct collaborators, thereby increasing the size and diversity of data used to train ML models.

FL has tremendous potential in healthcare <cit.>, particularly towards addressing health disparities, under-served populations, and “rare” diseases <cit.>, by enabling ML models to gain knowledge from ample and diverse data that would otherwise not be available. With that in mind, here we focus on the “rare” disease of glioblastoma, and particularly on the detection of its extent using multi-parametric magnetic resonance imaging (mpMRI) scans <cit.>. While glioblastoma is the most common malignant primary brain tumor <cit.>, it is still classified as a “rare” disease, as its incidence rate (i.e., $~3$/$100,000$ people) is substantially lower than the rare disease definition rate (i.e., $<10$/$100,000$ people) <cit.>. This means that single sites cannot collect large and diverse datasets to train robust and generalizable ML models, and necessitates collaboration between geographically distinct sites. Despite extensive efforts to improve prognosis of glioblastoma patients with intense multimodal therapy, their median overall survival is only $14.6$ months after standard-of-care treatment, and $4$ months without treatment <cit.>. Although the subtyping of glioblastoma has been improved <cit.> and the standard-of-care treatment options have expanded during the last twenty years, there have been no substantial improvements in overall survival <cit.>. This reflects the major obstacle in treating these tumors that is their intrinsic heterogeneity <cit.>, and the need for analyses of larger and more diverse data towards better understanding the disease. In terms of radiologic appearance, glioblastomas comprise $3$ main sub-compartments, defined as i) the “enhancing tumor” (ET), representing the vascular blood-brain barrier breakdown within the tumor, ii) the “tumor core” (TC), which includes the ET and the necrotic (NCR) part, and represents the surgically relevant part of the tumor, and iii) the “whole tumor” (WT), which is defined by the union of the TC and the peritumoral edematous/infiltrated tissue (ED), and represents the complete tumor extent relevant to radiotherapy (Fig. <ref>.b). Detecting these sub-compartment boundaries therefore defines a multi-parametric multi-class learning problem <cit.>, and is a critical first step towards further quantifying and assessing this heterogeneous rare disease and ultimately influencing clinical decision-making.

Representation of the study's global scale, diversity, and complexity. a, The map of all sites involved in the development of FL consensus model. b, example of a glioblastoma mpMRI scan with corresponding reference annotations of the tumor sub-compartments. c-d, comparative performance evaluation of the final consensus model with the public initial model on the collaborators' local validation data (in c) and on the complete out-of-sample data (in d), per tumor sub-compartment. Note the box and whiskers inside each violin plot, represent the true min and max values. The top and bottom of each “box” depict the 3rd and 1st quartile of each measure. The white line and the red $\times$, within each box, indicate the median and mean values, respectively. The fact that these are not necessarily at the centre of each box indicates the skewness of the distribution over different cases. The “whiskers” drawn above and below each box depict the extremal observations still within $1.5$ times the interquartile range, above the 3rd or below the 1st quartile. e, number of contributed cases per collaborating site.

Co-authors in this study have previously introduced FL in healthcare in a simulated setting <cit.> and evaluated different FL approaches <cit.> for the same use case as the present study, i.e., detecting the boundaries of glioblastoma sub-compartments. Findings from these studies supported the superiority of the FL approach used in the present study (i.e., based on an aggregation server <cit.>), which had almost identical performance to CL. The present study describes the largest to-date global FL effort to develop an accurate and generalizable ML model for detecting glioblastoma sub-compartment boundaries, based on $25,256$ MRI scans (over 5TB) of $6,314$ glioblastoma patients from $71$ geographically distinct sites, across $6$ continents (Fig. <ref>.a). Notably, this describes the largest and most diverse dataset of glioblastoma patients ever considered in the literature. It was the use of FL that successfully enabled our ML model to gain knowledge from such an unprecedented dataset. The extended global footprint and the task complexity is what sets this study apart from current literature, since it dealt with a multi-parametric multi-class problem with reference standards that require expert clinicians following an involved manual annotation protocol, rather than recording a categorical entry from medical records <cit.>. Moreover, varying characteristics of the mpMRI data due to scanner hardware and acquisition protocol differences <cit.> were handled at each collaborating site via established harmonized preprocessing pipelines <cit.>.

The scientific contributions of this manuscript can be summarized by i) the insights garnered during this work that can pave the way for more successful FL studies of increased scale and task complexity, ii) making a potential impact for the treatment of the rare disease of glioblastoma, by eventually publicly releasing an optimized trained consensus model for use in resource-constrained clinical settings, and iii) demonstrating the effectiveness of FL at such scale and task complexity as a paradigm shift redefining multi-site collaborations, while alleviating the need for data sharing.

§ METHODS

§.§ Data

The data considered in this study described patient populations with adult-type diffuse glioma <cit.>, and specifically displaying the radiological features of glioblastoma, scanned with mpMRI to characterize the anatomical tissue structure <cit.>. Each case is specifically described by i) native T1-weighted (T1), ii) Gadolinium-enhanced T1-weighted (T1Gd), iii) T2-weighted (T2), and iv) T2-weighted-Fluid-Attenuated-Inversion-Recovery (T2-FLAIR) MRI scans. Cases with any of these sequences missing were not included in the study. Note that no inclusion/exclusion criterion applied relating to the type of acquisition (i.e., both 2D axial and 3D acquisitions were included, with a preference for 3D if available), or the exact type of sequence (e.g., MP-RAGE vs SPGR). The only exclusion criterion was for T1-FLAIR scans that were intentionally excluded to avoid mixing varying tissue appearance due to the type of sequence, across native T1-weighted scans.
The publicly available data from the International Brain Tumor Segmentation (BraTS) 2020 challenge <cit.>, was used to train the public initial model of this study. The BraTS challenge <cit.>, seeking methodological advancements in the domain of neuro-oncology, has been providing the community with i) the largest publicly-available and manually-curated mpMRI dataset of diffuse glioma patients (an example of which is illustrated in Fig. <ref>.b), and ii) a harmonized preprocessing pipeline <cit.> to handle differences in inter-site acquisition protocols. The public initial model was used to initialize the FL training, instead of a randomly generated initialization, as starting from a pretrained model leads to faster convergence <cit.>. The complete BraTS 2020 dataset originally included cases from sites that also participated in this study as independent collaborators. To avoid any potential data leakage, we reduced the size of the complete BraTS dataset by removing cases acquired by these specific sites, resulting in a dataset of $231$ cases from $16$ international sites, with varying contributing cases across sites (Fig. <ref>.e). The exact site IDs that construct the data of the public initial model are: 47, 51, 55, 57, 58, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, and 71. Subsequently, the resulted dataset was split at a $4:1$ ratio between cases for training ($n=185$) and validation ($n=46$).
The eligibility of collaborating sites to participate in the federation was determined based on data availability, and approval by their respective institutional review board. $55$ sites participated as independent collaborators in the study defining a dataset of $6,083$ cases. The MRI scanners used for data acquisition were from multiple vendors (i.e., Siemens, GE, Philips, Hitachi, Toshiba), with magnetic field strength ranging from 1T to 3T. The data from all $55$ collaborating sites followed a male:female ratio of $1.47:1$ with age ranging between $7$ and $94$ years.

From all $55$ collaborating sites, $49$ were chosen to be part of the training phase, and $6$ sites were categorized as “out-of-sample”, i.e., none of these were part of the training stage. These specific $6$ out-of-sample sites (Site IDs: 8, 11, 19, 20, 21, 43) were allocated based on their availability, i.e., they have indicated an expected delayed participation rendering them optimal for model generalizability validation. One of these $6$ out-of-sample sites (Site 11) contributed aggregated a priori data from a multi-site randomized clinical trial for newly diagnosed glioblastoma (ClinicalTrials.gov Identifier: NCT00884741, RTOG0825 <cit.>, ACRIN6686 <cit.>), with inherent diversity benefiting the intended generalizability validation purpose. The American College of Radiology (ACR - Site 11) serves as the custodian of this trial's imaging data on behalf of ECOG-ACRIN, who made the data available for this study. Following screening for availability of the $4$ required mpMRI scans with sufficient signal-to-noise ratio judged by visual observation, a subset of $362$ cases from the original trial data were included in this study. The out-of-sample data totaled $590$ cases intentionally held-out of the federation, with the intention of validating the consensus model in completely unseen cases. To facilitate further such generalizability evaluation without burdening the collaborating sites, a subset consisting of $332$ cases (including the multi-site clinical data provided by ACR) from this out-of-sample data was aggregated, to serve as the “centralized out-of-sample” dataset. Furthermore, the $49$ sites participating in the training phase define a collective dataset of $5,493$ cases. The exact $49$ site IDs are: 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 15, 16, 17, 18, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 45, 46, 48, 49, 50, 52, 53, 54, 56, 59, 60. These cases were automatically split at each site following a $4:1$ ratio between cases for training and local validation. During the federated training phase the data used for the public initial model were also included as a dataset from a separate node, such that the contribution of sites providing the publicly available data is not forgotten within the global consensus model. This results in the final consensus model being developed based on data from $71$ sites over a total dataset of $6,314$ cases.

§.§ Harmonized Data Preprocessing

Once each collaborating site identified their local data, they were asked to use the preprocessing functionality of the software platform we provided. This functionality follows the harmonized data preprocessing protocol defined by the BraTS challenge <cit.>,
to account for inter-site acquisition protocol variations, e.g., 3D vs 2D axial plane acquisitions.

File-type conversion / Patient De-identification: The respective mpMRI scans (i.e., T1, T1Gd, T2, T2-FLAIR) of every case are downloaded onto a local machine in the Digital Imaging and Communications in Medicine (DICOM) format <cit.>, and converted to the Neuroimaging Informatics Technology Initiative (NIfTI) file format <cit.> to ensure easier parsing of the volumetric scans during the computational process. The conversion of DICOM to NIfTI files has the benefit of eliminating all patient-identifiable metadata from the header portion of the DICOM format <cit.>.

Rigid Registration: Once the scans are converted to the NIfTI format, each volume is registered to a common anatomical space, namely the SRI24 atlas <cit.>, to ensure a cohesive data shape ($[240,240,155]$) and an isotropic voxel resolution ($1mm^3$), thereby facilitating in tandem analysis of the mpMRI scans. One of the most common types of MRI noise is based on the inhomogeneity of the magnetic field <cit.>. It has been previously <cit.> shown that the use of non-parametric, non-uniform intensity normalization to correct for these bias fields <cit.> obliterates the MRI signal relating to the regions of abnormal T2-FLAIR signal. Here, we have taken advantage of this adverse effect and used the bias field-corrected scans to generate a more optimal rigid registration solution across the mpMRI sequences. The bias field-corrected images are registered to the T1Gd image, and the T1Gd image is rigidly registered to the SRI24 atlas, resulting in $2$ sets of transformation matrices per MRI sequence. These matrices are then aggregated to a single matrix defining the transformation of each MRI sequence from their original space to the atlas. We then apply this single aggregated matrix to the NIfTI scans prior to the application of the bias field correction to maximize the fidelity of the finally registered images.

Brain Extraction: This process focuses on generating a brain mask to remove all non-brain tissue from the image (including neck, fat, eyeballs, and skull), to enable further computational analyses while avoiding any potential face reconstruction/recognition <cit.>. For this step we utilized the Brain Mask Generator (BrainMaGe) <cit.>, which has been explicitly developed to address brain scans in presence of diffuse glioma, and considers brain shape as a prior, hence being agnostic to the sequence/modality input.

Generation of automated baseline delineations of tumor sub-compartment boundaries: We provided the ability to the collaborating sites to generate automated delineations of the tumor sub-compartments from three popular methods from the BraTS challenge, using models trained using the challenge's training data: i) DeepMedic <cit.>, ii) DeepScan <cit.>, and iii) nnU-Net <cit.>. Along with segmentations from each method, label fusion strategies were also employed to provide a reasonable approximation to the reference labels that should be manually refined and approved by expert neuroradiologists to create the final reference labels. The label fusion approaches considered were i) standard voting <cit.>, ii) Simultaneous Truth And Performance Level Estimation (STAPLE) <cit.>, iii) majority voting <cit.>, and iv) Selective and Iterative Method for Performance Level Estimation (SIMPLE) <cit.>.

Manual Refinements towards Reference Standard Labels: It was communicated to all participating sites to leverage the annotations generated using the automated mechanism as a baseline on which manual refinements were needed by neuroradiology experts, following a consistently communicated annotation protocol. The reference annotations comprised the Gd-enhancing tumor (ET - label `4'), the peritumoral edematous/invaded tissue (ED - label `2'), and the necrotic tumor core (NCR - label `1'). ET is generally considered the most active portion of the tumor, described by areas with both visually avid, as well as faintly avid, enhancement on the T1Gd scan. NCR is the necrotic part of the tumor, the appearance of which is hypointense on the T1Gd scan. ED is the peritumoral edematous and infiltrated tissue, defined by the abnormal hyperintense signal envelope on the T2-FLAIR scans, which includes the infiltrative non-enhancing tumor, as well as vasogenic edema in the peritumoral region <cit.> (an illustration can be seen in Fig. <ref>.b).

Data splits: Once the data were preprocessed, training and validation cohorts were created randomly in a $4:1$ ratio, and the splits were preserved during the entire duration of the FL training to prevent data leakage. The performance of every model was compared against the local validation data cohort on every federated round.

§.§ Data loading and processing

We leveraged the data loading and processing pipeline from the Generally Nuanced Deep Learning Framework (GaNDLF) <cit.>, to enable experimentation with various data augmentation techniques. Immediately after data loading, we removed the all-zero axial, coronal, and sagittal planes from the image, and performed a z-score normalization of the non-zero image intensities <cit.>. Each tumor sub-compartment of the reference label is first split into an individual channel and then passed to the neural network for processing. We extracted a single random patch per mpMRI volume set during every federated round. The patch size was kept constant at $[128,128,128]$ to ensure that the trained model can fit to the memory of the baseline hardware requirement of each collaborator, i.e., a discrete graphics processing unit with a minimum of 11GB dedicated memory. For data augmentation, we added random noise augmentation ($\mu=0.0$, $\sigma=0.1$) with probability of $p=0.2$, random rotations ($90^\circ$ and $180^\circ$, with the axis of rotation being uniformly selected in each case from the set of coronal, sagittal, and axial planes) each with probability of $p=0.5$, and a random flip augmentation with a probability of $p=1.0$ with equal likelihood of flips across the sagittal, coronal, and axial planes.

§.§ The neural network architecture

The trained model to delineate the different tumor sub-compartments was based on the popular 3D U-Net with residual connections (3D-ResUNet) <cit.>. The network had $30$ base filters, with a learning rate of $lr=5\times10^{-5}$ optimized using the Adam optimizer <cit.>. For the loss function used in training, we used the generalized $DSC$ score <cit.> (represented mathematically in Eq. <ref>) on the absolute complement of each tumor sub-compartment independently. Such mirrored $DSC$ loss has been shown to capture variations in smaller regions better <cit.>. No penalties were used in the loss function, due to our use of `mirrored' $DSC$ loss <cit.>. The final layer of the model was a sigmoid layer, providing three channel outputs for each voxel in the input volume, one output channel per tumor sub-compartment. While the generalized $DSC$ score was calculated using a binarized version of the output (check sigmoid value against the threshold $0.5$) for the final prediction, we used the floating point $DSC$ <cit.> during the training process.

\begin{equation}
\label{eq:dice}
DSC = \frac{2\left\|RL \odot PM \right\|_1}{\left\|RL \right\|_1 + \left\|PM \right\|_1}
\end{equation}

where RL serves as the reference label, $PM$ is the predicted mask, $\odot$ is the Hadamard product <cit.> (i.e., component wise multiplication), and $\left\| x \right\|_1$ is the L1-norm <cit.>, i.e., sum of the absolute values of all components).

§.§ The Federation

The collaborative network of the present study spans $6$ continents (Fig. <ref>), with data from $71$ geographically distinct sites. The training process was initiated when each collaborator securely connected to a central aggregation server, which resided behind a firewall at the University of Pennsylvania. As soon as the secure connection was established, the public initial model was passed to the collaborating site. Using FL based on an aggregation server, collaborating sites then trained the same network architecture on their local data for one epoch, and shared model updates with the central aggregation server. The central aggregation server received model updates from all collaborators, combined them (by averaging model parameters) and sent the consensus model back to each collaborator to continue their local training. Each such iteration is called a “federated round”. After not observing any meaningful changes since round $42$, we stopped the training after a total of $73$ federated rounds. Additionally, we performed all operations on the aggregator on secure hardware (by leveraging trusted execution environments <cit.>), in order to increase the trust by all parties in the confidentiality of the model updates being computed and shared, as well as to increase the confidence in the integrity of the computations being performed <cit.>.

The federated training was initialized using a “public initial model” trained on $231$ cases from $16$ sites. This was done as opposed to a random initialization point to facilitate faster convergence of the model performance <cit.>. After the training process was complete, the “final consensus model” was obtained after model selection from all the global consensus models obtained for each federated round. Thus, the final consensus model was developed on $6,314$ cases from $71$ sites. To quantitatively evaluate the performance of the trained models, $20\%$ of the total cases contributed by each participating site were excluded from the model training process and were used as “local validation data”. To further evaluate the generalizability of the models in unseen data, $6$ sites were not involved in any of the training stages to represent an unseen “out-of-sample” data population of $590$ cases. To facilitate further evaluation without burdening the collaborating sites, a subset ($n=332$) of these cases was aggregated to serve as a “centralized out-of-sample” dataset. Model performance was quantitatively evaluated here using the Dice Similarity Coefficient ($DSC$), which assesses the spatial agreement between the model's prediction and the reference standard for each of the $3$ tumor sub-compartments (ET, TC, WT).

We followed a staged approach for the training of the global consensus model, starting from a preliminary smaller federation across a subset ($n=35$) of the participating sites to evaluate the complete process and resolve any initial network issues. Note that $16$ of these $35$ sites were used to train the public initial model, and used in the preliminary federation as an aggregated dataset. The exact $19$ site IDs that participated in the training phase of the preliminary federation, as independent sites are: 2, 3, 9, 14, 22, 23, 24, 27, 28, 29, 31, 33, 36, 37, 41, 46, 53, 54, and 59. The total data held by this smaller federation represented approximately $42\%$ ($n=2,471$) of the data used in the full federation. We also trained individual models (initialized using the public initial model) using centralized training at all sites holding $>200$ training cases, and performed a comparative evaluation of the consensus model with an ensemble of these “single institution models”. The per voxel sigmoid outputs of the ensemble were computed as the average of such outputs over the individual single institution models. As with all other models in this study, binary predictions were computed by comparing these sigmoid outputs to a threshold value of $0.5$. The single institution model ensemble utilized (via the data at the single institutions) approximately $33\%$ of the total data across the federation.

§.§ Model Run-time in
Clinical Environments

Clinical environments typically have constrained computational resources, such as the availability of specialized hardware (e.g., DL acceleration cards) and increased memory, which affect the run-time performance of DL inference workloads. Thus, taking into consideration the potential deployment of the final consensus model in such low-resource settings, we decided to proceed with a single 3D-ResUNet, rather than an ensemble of multiple models. This decision ensured a reduced computational burden, when compared with running multiple models, which is typically done in academic research projects <cit.>.

To further facilitate use in low-resource environments, we plan to publicly release run-time optimized <cit.> version of the final consensus model. For this model, graph level optimizations (i.e., operators fusion) were initially applied, followed by optimizations for low precision inference, i.e., converting the floating point single precision model to a fixed precision 8-bit integer model (a process known as “quantization” <cit.>). In particular, we used accuracy-aware quantization <cit.>, where model layers were iteratively scaled to a lower precision format. These optimizations yielded several run-time performance benefits, such as lower inference latency (a platform-dependent $4.48\times$ average speedup and $2.29\times$ reduced memory requirement when compared with the original consensus model), and higher throughput (equal to the $4.48\times$ speedup improvement since the batch size used is equal to $1$), while the trade-off was an insignificant ($p_{Average}<7\times10^{-5}$) drop in the average $DSC$.

§.§ Data availability

The datasets used in this study, from the $71$ participating sites, are not made publicly available as a collective data collection due to restrictions imposed by acquiring institutions. The public initial model data from $16$ sites are publicly available through the BraTS challenge <cit.>. The data from each of the $55$ collaborating sites were neither publicly available during the execution of the study, nor shared among collaborating sites or with the aggregator. They were instead used locally, within each of the acquiring sites, for the training and validation of the global consensus model at each federated round. The anatomical template used for co-registration during preprocessing is the SRI24 atlas <cit.> and is available from <https://www.nitrc.org/projects/sri24>.

§.§.§ Clinically-Deployable Consensus Models

To further encourage reproducibility of our study, and considering enhancing the potential impact for the study of the rare disease of glioblastoma, we publicly released the trained models of this study. We specifically released the final singlet and triplet consensus models, including the complete source-code used in the project. Taking into consideration the potential deployment of these models in clinical settings, we refrained from training an ensemble of models (as typically done in academic research projects <cit.>), due to the additional computational burden of running multiple models. Furthermore, to facilitate use in low-resource environments, we also provide a post-training run-time optimized <cit.> version of the final consensus model that obviates the need for any specialized hardware (such as DL acceleration cards), and performs insignificantly different from the final consensus model when evaluated against the centralized out-of-sample data.

§.§ Code availability

Motivated by findability, accessibility, interoperability, and reusability (FAIR) criteria in scientific research <cit.>, all the code used to design the Federated Tumor Segmentation (FeTS) platform for this study is available at <https://github.com/FETS-AI/Front-End>. The functionality related to preprocessing (i.e., DICOM to NIfTI conversion, population-based harmonized preprocessing, co-registration) and manual refinements of annotation is derived from the open-source Cancer Imaging Phenomics Toolkit (CaPTk, <https://github.com/CBICA/CaPTk>) <cit.>. The co-registration is performed using the Greedy framework <cit.>, available via CaPTk <cit.>, ITK-SNAP <cit.>, and the FeTS tool. The brain extraction <cit.> is done using the BrainMaGe method <cit.>, and is available in <https://github.com/CBICA/BrainMaGe> and via the Generally Nuanced Deep Learning Framework (GaNDLF) <cit.> at <https://github.com/CBICA/GaNDLF>. To generate automated annotations, DeepMedic's <cit.> integration with CaPTk was used, and we used the model weights and inference mechanism provided by the other algorithm developers (DeepScan <cit.> and nnU-Net <cit.> (<https://github.com/MIC-DKFZ/nnunet>)). DeepMedic's original implementation is available in <https://github.com/deepmedic/deepmedic>, whereas the one we used in this study can be found at <https://github.com/CBICA/deepmedic>. The fusion of the labels was done using the Label Fusion tool <cit.> available at <https://github.com/FETS-AI/LabelFusion>. The data loading pipeline and network architecture was developed using the GaNDLF framework <cit.> by using PyTorch <cit.>. The data augmentation was done via GaNDLF by leveraging TorchIO <cit.>. The FL backend developed for this project has been open-sourced as a separate software library, to encourage further research on FL <cit.> and is available at <https://github.com/intel/openfl>. The optimization of the consensus model inference workload was performed via OpenVINO <cit.> (<https://github.com/openvinotoolkit/openvino/tree/2021.4.1>), which is an open-source toolkit enabling acceleration of neural network models through various optimization techniques. The optimizations were evaluated on an Intel Core® i7-1185G7E CPU @ 2.80GHz with $2 \times 8$ GB DDR4 3200MHz memory on Ubuntu 18.04.6 OS and Linux kernel version 5.9.0-050900-generic.

§ RESULTS

§.§ Access to non-public data via FL significantly improved model performance

At the time of initialization of the federation, the public initial model was evaluated against the local validation data of all sites, resulting in an average (across all cases of all sites) $DSC$ per sub-compartment, of: $DSC_{ET}=0.63$, $DSC_{TC}=0.62$, $DSC_{WT}=0.75$. To summarize the model performance with a single collective score, we then calculate the average $DSC$ (across all $3$ tumor sub-compartments per case, and then across all cases of all sites) as equal to $0.66$.

Following model training across all sites, the final consensus model garnered significant performance improvements against the collaborators' local validation data of $27\%$ ($p_{ET}<1\times10^{-36}$), $33\%$ ($p_{TC}<1\times10^{-59}$), and $16\%$ ($p_{WT}<1\times10^{-21}$), for ET, TC, and WT, respectively (Fig. <ref>.c).

To further evaluate the potential generalizability improvements of the final consensus model on unseen data, we compared it with the public initial model against the complete out-of-sample data, and noted significant performance improvements of $15\%$ ($p_{ET}<1\times10^{-5}$), $27$% ($p_{TC}<1\times10^{-16}$), and $16\%$ ($p_{WT}<1\times10^{-7}$), for ET, TC, and WT, respectively (Fig. <ref>.d).

Notably, the only difference between the public initial model and the final consensus model, was that the latter gained knowledge during training from increased datasets contributed by the complete set of collaborators. The conclusion of these findings reinforces the importance of using large and diverse data for generalizable models to ultimately drive patient care.

§ DISCUSSION

In this study, we have described the largest real-world FL effort to-date utilizing data of $6,314$ glioblastoma patients from $71$ geographically unique sites spread across $6$ continents, to develop an accurate and generalizable ML model for detecting glioblastoma sub-compartment boundaries. Notably, this extensive global footprint of the collaborating sites in this study, also yield the largest dataset ever reported in the literature assessing this rare disease. It is the use of FL that successfully enabled i) access to such an unprecedented dataset of the most common and fatal adult brain tumor, and ii) meaningful ML training to ensure generalizability of models across out-of-sample data. Since glioblastoma boundary detection is critical for treatment planning and the requisite first step for further quantitative analyses, the models generated during this study have the potential to make far-reaching clinical impact.

The large and diverse data that FL enabled, led to the final consensus model garnering significant performance improvements over the public initial model against both the collaborators' local validation data and the complete out-of-sample data. The improved result is a clear indication of the benefit that can be afforded through access to more data. In comparison with the limited existing real-world FL studies <cit.>, our use case is larger in scale and substantially more complex, since it 1) addresses a multi-parametric multi-class problem, with reference standards that require expert collaborating clinicians to follow an involved manual annotation protocol, rather than simply recording a categorical entry from medical records, and 2) requires the data to be preprocessed in a harmonized manner to account for differences in MRI acquisition.

However, increasing the data size for model training without considerations relating to data quality, reference labels, and potential site bias (e.g., scanner acquisition protocols, demographics, or sociocultural considerations, such as more advanced presentation of disease at diagnosis in low-income regions <cit.>) might not always improve results. Literature also indicates a ML performance stagnation effect, where each added case contributes less to the model performance as the number of cases increase <cit.>. This is in line with our finding in the federated setting (Fig. <ref>), where performance improvements across the public initial model, the preliminary consensus model, and the final consensus model, were not directly/linearly related to the amount of data used for training. This happened even though the final consensus model was trained on over twice the number of cases (and included 2 of the largest contributing sites - Sites 1 & 4) when compared to the preliminary consensus model. Further noting that the preliminary federation model was already within the intra- and inter-rater variability range for this use case (20% and 28%, respectively) <cit.>, any further improvements for the full federation consensus model would be expected to be minimal <cit.>.

To further assess these considerations, we coordinated independent model training for the four largest collaborating sites (i.e., $>200$ cases) by starting from the same public initial model and using only their local training data. The ensemble of these four largest site local models did not show significant performance differences to the final consensus model for any tumor sub-compartment. Since each of these collaborators initiated their training from the public initial model, their independent models and their ensemble could have inherited some of the initial model's data diversity, which could justify the observed insignificant differences (Fig. <ref> and Fig. <ref>). Though these findings are an indication that the inclusion of more data alone may not lead to better performance, it is worth noting that these four largest sites
used for the independent model training represent comprehensive cancer centers (compared to hospitals in community settings) with affiliated sophisticated labs focusing on brain tumor research, and hence were familiar with the intricacies of computational analyses. Further considering the aforementioned ML performance stagnation effect, we note the need for generalizable solutions to quantify the contribution of collaborating sites to the final consensus model performance, such that future FL studies are able to formally assess both the quantity and the quality of the contributed data needed by the collaborating sites and decide on their potential inclusion on use-inspired studies.

As noted in our results, due to the lack of such generalizable solutions, we were only able to identify quality issues after the model training. Specifically, we hypothesize that although Site 48 had data quality issues, its effect on the consensus model performance was not significant due to its relatively small dataset ($n=46$) when compared to the other collaborating sites. The curves of Fig. <ref>.a indicate that the global consensus model continues to consistently gain knowledge from the federation as a whole during training, highlighting robustness to such data quality issues. It remains unknown however, how much better the consensus model would have performed if sites with problematic data were excluded or if these specific problematic data at Site 48 were excluded or corrected. These findings are aligned with literature observations (on the same use case) <cit.>, where a DL model <cit.> trained on $641$ glioblastoma cases from $8$ sites produced higher quality predictions on average than those created as reference standard labels by radiology expert operators. Quality was judged by $20$ board-certified neuroradiologists, in a blinded side-by-side comparison of $100$ sequestered unseen cases, and concluded that perfect or near-perfect reference labels may not be required to produce high quality prediction systems. In other words, DL models may learn to see past imperfect reference training labels. These findings provide impetus for further experimentation as they have implications for future FL studies. Future research is needed to automatically detect anomalies in the consensus model performance during training, particularly associated with contributions from individual sites.

There are a number of practical considerations that need to be taken into account to set up a multi-national real-world federation, starting with a substantial amount of coordination between each participating site. As this study is the first at this scale and task complexity, we have compiled a set of governance insights from our experience that can serve as considerations for future successful FL studies. These insights differ from previous literature that describe studies that were smaller in scale and involved simpler tasks <cit.>. By “governance” of the federation we refer both to the accurate definition of the problem statement (including reference labels and harmonization considerations accounting for inter-site variability), and the coordination with the collaborating sites for eligibility and compliance with the problem statement definition, as well as security and technical considerations.

The coordination began with engaging the security teams of collaborating sites, and providing them access to the source code of the platform developed to facilitate this study. These security discussions highlighted the benefit of the platform being open-source, making security code reviews easier. Resource gathering was then carried out by identifying technical leads and assessing computational resources at each site. With the technical leads, we then proceeded to test the complete workflow to further identify gaps in the requirements, such as network configurations and hardware requirements. We then proceeded with data curation and preprocessing,
and finally connected individual sites to the aggregation server to initiate their participation.

Following the precise definition of our problem statement <cit.>, ensuring strict compliance with the preprocessing and annotation protocol for the generation of reference standards was vital for the model to learn correct information during training. To this end, we instituted an extensively and comprehensively documented annotation protocol with visual example representations and common expected errors (as observed in the literature <cit.>) to all collaborators. We have further circulated an end-to-end platform developed to facilitate this federation, providing to each collaborating site all the necessary functionalities to i) uniformly curate their data and account for inter-site acquisition variability, ii) generate the reference standard labels, and iii) participate in the federated training process.
Finally, we held interactive sessions to complement the theoretical definition of the reference standards, and further guide collaborating sites.

In general, debugging issues with the inputted local data and annotations is more difficult during FL due to the level of coordination and/or privacy issues involved, since the data are always retained at the collaborating site. We gained substantial experience during this effort that went into further development of use-inspired but generalizable data sanity checking functionality in the tools we developed, towards facilitating further multi-site collaborations.

Upon conclusion of the study, sites participating in the model training process were given a survey to fill in regarding various aspects of their experience. According to the provided feedback, $96\%$ of the sites found the comprehensive documentation on preprocessing and data curation essential, and thought that lack of such documentation could have resulted in inconsistent annotations. Additionally, $92\%$ found the documentation relating to establishing secure connectivity to the aggregation server easy to follow and essential to expedite reviews by the related groups. Furthermore, $84\%$ of the sites appreciated the user-friendly interface of the provided tool and its associated complete functionality (beyond its FL backend), and indicated their intention to use it and recommend it for projects and data analysis pipelines beyond the scope of this study. To generate the reference standard labels for their local data, $86\%$ of the collaborating sites indicated that they used either FeTS (i.e., the tool developed for this study), CaPTk <cit.>, or ITK-SNAP <cit.>, whereas the remaining $14\%$ used either 3D-Slicer <cit.>, the BraTS toolkit <cit.>, or something else. In terms of hardware requirements at each site, $88\%$ used a dedicated workstation for their local workload, and the remaining $12\%$ used either a containerized form of the FeTS platform, or a virtual machine.

Although our study has the potential to become the baseline upon which future ML research studies will be done, there is no automated mechanism to assess inputted data quality from collaborators, which could result in models trained using sub-optimal data. Additionally, we used a single off-the-shelf neural network architecture for training, but it has been shown that model ensembles perform better for the task at hand <cit.>, and it remains to be explored how such strategy could be explored in a federated study. Moreover, the instantiation of the federation involved a significant amount of coordination between each site and considering the limited real-world FL studies at the time, there were no tools available to automate such coordination and orchestration. These involved i) getting interviewed by information security officers of collaborating sites, ii) ensuring that the harmonized preprocessing pipeline was used effectively, iii) clear communication of the annotation protocol, and iv) testing the network communication between the aggregator and each site. This amount of effort, if not aided by automated tools, will continue being a huge roadblock for FL studies, and dedicated coordination and orchestration resources are required to conduct this in a reproducible and scalable manner.

We have demonstrated the utility of an FL approach to develop an accurate and generalizable ML model for detecting glioblastoma sub-compartment boundaries, a finding which is of particular relevance for neurosurgical and radiotherapy planning in patients with this disease. This study is meant to be used as a blueprint for future FL studies that result in clinically deployable ML models. Building on this study, a continuous FL consortium would enable downstream quantitative analyses with implications for both routine practice and clinical trials, and most importantly, increase access to high-quality precision care worldwide. Furthermore, the lessons learned from this study with such a global footprint are invaluable and can be applied to a broad array of clinical scenarios with the potential for great impact to rare diseases and underrepresented populations.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

Radiological Society of North America

titlePattern analysis of dynamic susceptibility contrast-enhanced MR imaging demonstrates peritumoral tissue heterogeneity

Oxford University Press

titleImaging surrogates of infiltration obtained via multiparametric imaging pattern analysis predict subsequent location of recurrence of glioblastoma

journaltitleClinical Cancer Research
titleIn vivo detection of EGFRvIII in glioblastoma via perfusion magnetic resonance imaging signature consistent with deep peritumoral infiltration: the $\varphi$-index

Oxford University Press US

titleIn vivo evaluation of EGFRvIII mutation in primary glioblastoma patients via complex multiparametric MRI signature

journaltitleCancer cell
titleEpidermal growth factor receptor extracellular domain mutations in glioblastoma present opportunities for clinical imaging and therapeutic development

International Society for Optics

booktitleMedical Imaging 2018: Computer-Aided Diagnosis
titleDeriving stable multi-parametric MRI radiomic signatures in the presence of inter-scanner variations: survival prediction of glioblastoma via imaging pattern analysis and machine learning techniques

Oxford University Press US

titleNIMG-40. Non-invasive in vivo signature of idh1 mutational status in high grade glioma, from clinically-acquired multi-parametric magnetic resonance imaging, using multivariate machine learning

American Medical Association

journaltitleJAMA pediatrics
titleUse of fetal magnetic resonance image analysis and machine learning to predict the need for postnatal cerebrospinal fluid diversion in fetal ventriculomegaly

journaltitleMagnetic resonance imaging
titlePrecision diagnostics based on machine learning-derived imaging signatures

Wiley Online Library

journaltitleJournal of Magnetic Resonance Imaging
titleImaging signatures of glioblastoma molecular characteristics: a radiogenomics review

International Society for Optics

journaltitleJournal of Medical Imaging
titleOverall survival prediction in glioblastoma patients using structural magnetic resonance imaging (MRI): advanced radiomic features may compensate for lack of advanced MRI modalities

Wiley Online Library

titleHistopathology-validated machine learning radiographic biomarker for noninvasive discrimination between true progression and pseudo-progression in glioblastoma

Annual Reviews

journaltitleAnnual review of biomedical engineering
titleIntegrated biophysical modeling and image analysis: application to neuro-oncology

journaltitleComputerized medical imaging and graphics
titleAnalyzing magnetic resonance imaging data from glioma patients using deep learning

journaltitleMedical Image Analysis
titleThe reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study

Public Library of Science San Francisco, CA USA

journaltitlePLoS medicine
titleVariable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study

NIH Public Access

journaltitleThe New England journal of medicine
titlePredicting the future—big data, machine learning, and clinical medicine

journaltitlearXiv preprint arXiv:1801.00631
titleDeep learning: A critical appraisal

titleNeural networks and deep learning

journaltitleBrain imaging and behavior
titleThe ENIGMA Consortium: large-scale collaborative analyses of neuroimaging and genetic data

titleGlioma through the looking GLASS: molecular evolution of diffuse gliomas and the Glioma Longitudinal Analysis Consortium

Oxford University Press US

titleAI-based prognostic imaging biomarkers for precision neuro-oncology: the ReSPOND consortium

Oxford University Press US

titleiGLASS: imaging integration into the Glioma Longitudinal Analysis Consortium

Nature Publishing Group

journaltitleNPJ digital medicine
titleThe future of digital health with federated learning

Nature Publishing Group

journaltitleScientific reports
titleFederated learning in medicine: facilitating multi-institutional collaborations without sharing patient data

MEDICAL PUBLISHING GROUP-MASS MEDIC SOCIETY

journaltitleNew England Journal of Medicine
titleHIPAA regulations-a new era of medical-record privacy?

journaltitleA Practical Guide, 1st Ed., Cham: Springer International Publishing
titleThe eu general data protection regulation (gdpr)

booktitleArtificial intelligence and statistics
titleCommunication-efficient learning of deep networks from decentralized data

booktitleInternational MICCAI Brainlesion Workshop
titleMulti-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation

Nature Publishing Group

journaltitleNature medicine
titleFederated learning for predicting clinical outcomes in patients with COVID-19

Oxford University Press

journaltitleJournal of the American Medical Informatics Association
titleDistributed deep learning networks among institutions for medical imaging

booktitleProceedings of the Second Workshop on Distributed Infrastructures for Deep Learning
titleA performance evaluation of federated learning algorithms

Oxford University Press

journaltitleJournal of the American Medical Informatics Association
titleFederated learning improves site performance in multicenter deep learning without data sharing

booktitleClinical Image-Based Procedures, Distributed and Collaborative Learning, Artificial Intelligence for Combating COVID-19 and Secure and Privacy-Preserving Machine Learning
titleMulti-task Federated Learning for Heterogeneous Pancreas Segmentation

journaltitleMedical image analysis
titleFederated semi-supervised learning for COVID region segmentation in chest CT using multi-national data from China, Italy, Japan

Nature Publishing Group

journaltitleNature medicine
titleClinically applicable deep learning for diagnosis and referral in retinal disease

Nature Publishing Group

journaltitleNature medicine
titleCardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network

journaltitleMolecular genetics and metabolism
titleClinical research for rare disease: opportunities, challenges, and solutions

journaltitleChin Clin Oncol
titleAdvanced magnetic resonance imaging in glioblastoma: a review

titleThe somatic genomic landscape of glioblastoma

journaltitleCancer cell
titleIntegrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in PDGFRA, IDH1, EGFR, and NF1

National Acad Sciences

journaltitleProceedings of the National Academy of Sciences
titleIntratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics

Oxford University Press US

titleCBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2012–2016

Oxford Academic

titleThe 2021 WHO classification of tumors of the central nervous system: a summary

Am Soc Neuroradiology

journaltitleAmerican Journal of Neuroradiology
titleDeep transfer learning and radiomics feature prediction of survival of patients with high-grade gliomas

booktitleBrainLes 2015
titleGLISTRboost: combining multimodal MRI segmentation, registration, and biophysical tumor growth modeling with gradient boosting machines for glioma segmentation

journaltitleProceeding of the Multimodal Brain Tumor Image Segmentation Challenge
titleSegmentation of gliomas in multimodal magnetic resonance imaging volumes based on a hybrid generative-discriminative framework

booktitleInternational Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries
titleSegmentation of gliomas in pre-operative and post-operative multimodal magnetic resonance imaging volumes based on a hybrid generative-discriminative framework

journaltitleFrontiers in Computational Neuroscience
titleMulti-disease segmentation of gliomas and white matter hyperintensities in the BraTS data using a 3D convolutional neural network

journaltitleBiomedical signal processing and control
titleLongitudinal brain tumor segmentation prediction in MRI using feature and label fusion

booktitleDomain Adaptation and Representation Transfer, and Distributed and Collaborative Learning
titleFederated learning for breast density classification: A real-world implementation

journaltitleJournal of Clinical Neuroscience
titleMulti-institutional validation of a preoperative scoring system which predicts survival for patients with glioblastoma

American Society of Clinical Oncology

journaltitleJCO clinical cancer informatics
titleCancer imaging phenomics via CaPTk: multi-institutional prediction of progression-free survival and pattern of recurrence in glioblastoma

journaltitleIEEE transactions on medical imaging
titleThe multimodal brain tumor image segmentation benchmark (BRATS)

Nature Publishing Group

journaltitleScientific data
titleAdvancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features

journaltitlearXiv preprint arXiv:1811.02629
titleIdentifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge

journaltitlearXiv preprint arXiv:2107.02314
titleThe rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification

American Society of Clinical Oncology

titleRTOG 0825: Phase III double-blind placebo-controlled trial evaluating bevacizumab (Bev) in patients (Pts) with newly diagnosed glioblastoma (GBM).

Mass Medical Soc

journaltitleNew England Journal of Medicine
titleA randomized trial of bevacizumab for newly diagnosed glioblastoma

Oxford University Press US

titlePrognostic value of contrast enhancement and FLAIR for survival in newly diagnosed glioblastoma treated with and without bevacizumab: results from ACRIN 6686

Oxford University Press US

titleValue of dynamic contrast perfusion MRI to predict early response to bevacizumab in newly diagnosed glioblastoma: results from ACRIN 6686 multicenter trial

booktitleInternational Conference on Medical image computing and computer-assisted intervention
titleU-net: Convolutional networks for biomedical image segmentation

booktitleInternational conference on medical image computing and computer-assisted intervention
title3D U-Net: learning dense volumetric segmentation from sparse annotation

booktitleProceedings of the IEEE conference on computer vision and pattern recognition
titleDeep residual learning for image recognition

booktitleDeep Learning and Data Labeling for Medical Applications
titleThe importance of skip connections in biomedical image segmentation

booktitleInternational MICCAI Brainlesion Workshop
titleBrain tumor segmentation based on 3D residual U-Net

journaltitlearXiv preprint arXiv:1412.6980
titleAdam: A method for stochastic optimization

booktitleDeep learning in medical image analysis and multimodal learning for clinical decision support
titleGeneralised dice overlap as a deep learning loss function for highly unbalanced segmentations

journaltitleIEEE transactions on medical imaging
titleMorphometric analysis of white matter lesions in MR images: method and validation

Nature Publishing Group

journaltitleNature Methods
titlennU-Net: a self-configuring method for deep learning-based biomedical image segmentation

journaltitleNeural Computing and Applications
titleEfficient and robust deep learning with correntropy-induced loss function

booktitleInternational workshop on machine learning in medical imaging
titleTversky loss function for image segmentation using 3D fully convolutional deep networks

journaltitlearXiv preprint arXiv:1908.03679
titleDistance map loss penalty term for semantic segmentation

journaltitlearXiv preprint arXiv:1906.11031
titleContinuous dice coefficient: a method for evaluating probabilistic segmentations

booktitleProc. Symp. Appl. Math
titleThe hadamard product

Wiley Online Library

journaltitleJournal of the Royal Statistical Society: Series C (Applied Statistics)
titleL1 approximation and the analysis of data

Nature Publishing Group

journaltitleNature Machine Intelligence
titleSecure, privacy-preserving and federated machine learning in medical imaging

journaltitleIEEE Security & Privacy
titleThe untapped potential of trusted execution environments on mobile devices

journaltitleAdvances in neural information processing systems
titleTransfusion: Understanding transfer learning for medical imaging

journaltitleInternational Journal of Advanced Trends in Computer Science and Engineering
titleApplicability of various pre-trained deep convolutional neural networks for pneumonia classification based on X-Ray Images

journaltitleIntel White Paper
titleLower numerical precision deep learning inference and training

booktitleInternational conference on machine learning
titleFixed point quantization of deep convolutional networks

journaltitleIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
titleEnhanced precision analysis for accuracy-aware bit-width optimization using affine arithmetic

Nature Publishing Group

journaltitleScientific data
titleThe FAIR Guiding Principles for scientific data management and stewardship

International Society for Optics

journaltitleJournal of medical imaging
titleCancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome

booktitleInternational MICCAI Brainlesion Workshop
titleBrain cancer imaging phenomics toolkit (brain-CaPTk): an interactive platform for quantitative analysis of glioblastoma

booktitleInternational MICCAI Brainlesion Workshop
titleThe cancer imaging phenomics toolkit (captk): Technical overview

journaltitleAlzheimer's & Dementia: The Journal of the Alzheimer's Association
titleFast automatic segmentation of hippocampal subfields and medial temporal lobe subregions in 3 Tesla and 7 Tesla T2-weighted MRI

titleUser-Guided 3D Active Contour Segmentation of Anatomical Structures: Significantly Improved Efficiency and Reliability

titleBrain extraction from normal and pathological images: A joint PCA/image-reconstruction approach

titleBrain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training

journaltitlearXiv preprint arXiv:2103.01006
titleGandlf: A generally nuanced deep learning framework for scalable end-to-end clinical workflows in medical imaging

journaltitleMedical image analysis
titleEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation

booktitleInternational MICCAI Brainlesion Workshop
titleEnsembles of densely-connected CNNs with label-uncertainty for brain tumor segmentation

titleLabelFusion: Medical Image label fusion of segmentations

booktitleAdvances in neural information processing systems
titlePytorch: An imperative style, high-performance deep learning library

journaltitleComputer Methods and Programs in Biomedicine
titleTorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning
Medical image computing,Deep learning,Data augmentation,Preprocessing

journaltitlearXiv preprint arXiv:2105.06413
titleOpenFL: An open-source framework for Federated Learning

booktitleProceedings of the IEEE/CVF International Conference on Computer Vision Workshops
titleOpenVINO deep learning workbench: Comprehensive analysis and tuning of neural networks inference

[heading=subbibliography,title=Methods References]

§ ACKNOWLEDGMENTS

Research and main methodological developments reported in this publication were partly supported by the National Institutes of Health (NIH) under award numbers NIH/NCI:U01CA242871 (S.Bakas), NIH/NINDS:R01NS042645 (C.Davatzikos), NIH/NCI:U24CA189523 (C.Davatzikos), NIH/NCI:U24CA215109 (J.Saltz), NIH/NCI:U01CA248226 (P.Tiwari), NIH/NCI:P30CA51008 (Y.Gusev), NIH:R50CA211270 (M.Muzi), NIH/NCATS:UL1TR001433 (Y.Yuan), NIH/NIBIB:R21EB030209 (Y.Yuan), NIH/NCI:R37CA214955 (A.Rao), and NIH:R01CA233888 (A.L.Simpson). The authors would also like to acknowledge the following NIH funded awards for the multi-site clinical trial (NCT00884741, RTOG0825/ACRIN6686): U10CA21661, U10CA37422, U10CA180820, U10CA180794, U01CA176110, R01CA082500, CA079778, CA080098, CA180794, CA180820, CA180822, CA180868. Research reported in this publication was also partly supported by the National Science Foundation, under award numbers 2040532 (S.Baek), and 2040462 (B.Landman). Research reported in this publication was also supported by
i) a research grant from Varian Medical Systems (Palo Alto, CA USA) (Y.Yuan),
ii) the Ministry of Health of the Czech Republic (Grant Nr. NU21-08-00359) (M.Keřkovský and M.Kozubek),
iii) Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project-ID 404521405, SFB 1389, Work Package C02, and Priority Programme 2177 “Radiomics: Next Generation of Biomedical Imaging” (KI 2410/1-1 | MA 6340/18-1) (P.Vollmuth),
iv) DFG Project-ID B12, SFB 824 (B.Wiestler),
v) the Helmholtz Association (funding number ZT-I-OO1 4) (K.Maier-Hein),
vi) the Dutch Cancer Society (KWF project number EMCR 2015-7859) (S.R.van der Voort),
vii) the Chilean National Agency for Research and Development (ANID-Basal FB0008 (AC3E) and FB210017 (CENIA)) (P.Guevara),
viii) the Canada CIFAR AI Chairs Program (M.Vallières),
ix) Leeds Hospital Charity (Ref: 9RO1/1403) (S.Currie),
x) the Cancer Research UK funding for the Leeds Radiotherapy Research Centre of Excellence (RadNet) and the grant number C19942/A28832 (S.Currie),
xi) Medical Research Council (MRC) Doctoral Training Programme in Precision Medicine (Award Reference No. 2096671) (J.Bernal),
xii) The European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 757173) (B.Glocker),
xiii) The UKRI London Medical Imaging & Artificial Intelligence Centre for Value Based Healthcare (K.Kamnitsas),
xiv) Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Center for Medical Engineering (WT 203148/Z/16/Z) (T.C.Booth),
xv) American Cancer Society Research Scholar Grant RSG-16-005-01 (A.Rao),
xvi) the Department of Defense (DOD) Peer Reviewed Cancer Research Program (PRCRP) W81XWH-18-1-0404, Dana Foundation David Mahoney Neuroimaging Program, the V Foundation Translational Research Award, Johnson & Johnson WiSTEM2D Award (P.Tiwari),
xvii) RSNA Research & Education Foundation under grant number RR2011 (E.Calabrese),
xviii) the National Research Fund of Luxembourg (FNR) (grant number: C20/BM/14646004/GLASS-LUX/Niclou) (S.P.Niclou),
xix) EU Marie Curie FP7-PEOPLE-2012-ITN project TRANSACT (PITN-GA-2012-316679) and the Swiss National Science Foundation (project number 140958) (J.Slotboom), and
xx) CNPq 303808/2018-7 and FAPESP 2014/12236-1 (A.Xavier Falcão). The content of this publication is solely the responsibility of the authors and does not represent the official views of the NIH, the NSF, the RSNA R&E Foundation, or any of the additional funding bodies.

§ AUTHOR CONTRIBUTIONS

Study Conception: S.Pati, U.Baid, B.Edwards, M.Sheller, G.A.Reina, J.Martin, S.Bakas.

Development of software used in the study: S.Pati, B.Edwards, M.Sheller, S.Wang, G.A.Reina, P.Foley, A.Gruzdev, D.Karkada.

Data Acquisition: M.Bilello, S.Mohan, E.Calabrese, J.Rudie, J.Saini, R.Y.Huang, K.Chang, T.So, P.Heng, T.F.Cloughesy, C.Raymond, T.Oughourlian, A.Hagiwara, C.Wang, M.To, M.Keřkovský, T.Kopřivová, M.Dostál, V.Vybíhal, J.A.Maldjian, M.C.Pinho, D.Reddy, J.Holcomb, B.Wiestler, M.Metz, R.Jain, M.Lee, P.Tiwari, R.Verma, Y.Gusev, K.Bhuvaneshwar, C.Bencheqroun, A.Belouali, A.Abayazeed, A.Abbassy, S.Gamal, M.Qayati, M.Mekhaimar, M.Reyes, R.R.Colen, M.Ak, P.Vollmuth, G.Brugnara, F.Sahm, M.Bendszus, W.Wick, A.Mahajan, C.Balaña Quintero, J.Capellades, J.Puig, Y.Choi, M.Muzi, H.F.Shaykh, A.Herrera-Trujillo, W.Escobar, A.Abello, P.LaMontagne, B.Landman, K.Ramadass, K.Xu, S.Chotai, L.B.Chambless, A.Mistry, R.C.Thompson, J.Bapuraj, N.Wang, S.R.van der Voort, F.Incekara, M.M.J.Wijnenga, R.Gahrmann, J.W.Schouten, H.J.Dubbink, A.J.P.E.Vincent, M.J.van den Bent, H.I.Sair, C.K.Jones, A.Venkataraman, J.Garrett, M.Larson, B.Menze, T.Weiss, M.Weller, A.Bink, B.Pouymayou, Y.Yuan, S.Sharma, T.Tseng, B.C.A.Teixeira, F.Sprenger, S.P.Niclou, O.Keunen, L.V.M.Dixon, M.Williams, R.G.H.Beets-Tan, H.Franco-Maldonado, F.Loayza, J.Slotboom, P.Radojewski, R.Meier, R.Wiest, J.Trenkler, J.Pichler, G.Necker, S.Meckel, E.Torche, F.Vera, E.Lóópez, Y.Kim, H.Ismael, B.Allen, J.M.Buatti, J.Park, P.Zampakis, V.Panagiotopoulos, P.Tsiganos, E.Challiasos, D.M.Kardamakis, P.Prasanna, K.M.Mani, D.Payne, T.Kurc, L.Poisson, M.Vallières, D.Fortin, M.Lepage, F.Morón, J.Mandel, C.Badve, A.E.Sloan, J.S.Barnholtz-Sloan, K.Waite, G.Shukla, S.Liem, G.S.Alexandre, J.Lombardo, J.D.Palmer, A.E.Flanders, A.P.Dicker, G.Ogbole, M.Soneye, D.Oyekunle, O.Odafe-Oyibotha, B.Osobu, M.Shu'aibu, F.Dako, A.Dorcas, D.Murcia, R.Haas, J.Thompson, D.R.Ormond, S.Currie, K.Fatania, R.Frood, J.Mitchell, J.Farinhas, A.L.Simpson, J.J.Peoples, R.Hu, D.Cutler, F.Y.Moraes, A.Tran, M.Hamghalam, M.A.Boss, J.Gimpel, B.Bialecki, A.Chelliah.

Data Processing: C.Sako, S.Ghodasara, E.Calabrese, J.Rudie, M.Jadhav, U.Pandey, R.Y.Huang, M.Jiang, C.Chen, C.Raymond, S.Bhardwaj, C.Chong, M.Agzarian, M.Kozubek, F.Lux, J.Michálek, P.Matula, C.Bangalore Yogananda, D.Reddy, B.C.Wagner, I.Ezhov, M.Lee, Y.W.Lui, R.Verma, R.Bareja, I.Yadav, J.Chen, N.Kumar, K.Bhuvaneshwar, A.Sayah, C.Bencheqroun, K.Kolodziej, M.Hill, M.Reyes, L.Pei, M.Ak, A.Kotrotsou, P.Vollmuth, G.Brugnara, C.J.Preetha, M.Zenk, J.Puig, M.Muzi, H.F.Shaykh, A.Abello, J.Bernal, J.Gómez, P.LaMontagne, K.Ramadass, S.Chotai, N.Wang, M.Smits, S.R.van der Voort, A.Alafandi, F.Incekara, M.M.J.Wijnenga, G.Kapsas, R.Gahrmann, A.J.P.E.Vincent, P.J.French, S.Klein, H.I.Sair, C.K.Jones, J.Garrett, H.Li, F.Kofler, Y.Yuan, S.Adabi, A.Xavier Falcão, S.B.Martins, D.Menotti, D.R.Lucio, O.Keunen, A.Hau, K.Kamnitsas, L.Dixon, S.Benson, E.Pelaez, H.Franco-Maldonado, F.Loayza, S.Quevedo, R.McKinley, J.Trenkler, A.Haunschmidt, C.Mendoza, E.Ríos, J.Choi, S.Baek, J.Yun, P.Zampakis, V.Panagiotopoulos, P.Tsiganos, E.I.Zacharaki, C.Kalogeropoulou, P.Prasanna, S.Shreshtra, T.Kurc, B.Luo, N.Wen, M.Vallières, D.Fortin, F.Morón, C.Badve, V.Vadmal, G.Shukla, G.Ogbole, D.Oyekunle, F.Dako, D.Murcia, E.Fu, S.Currie, R.Frood, M.A.Vogelbaum, J.Mitchell, J.Farinhas, J.J.Peoples, M.Hamghalam, D.Kattil Veettil, K.Schmidt, B.Bialecki, S.Marella, T.C.Booth, A.Chelliah, M.Modat, C.Dragos, H.Shuaib.

Data Analysis & Interpretation: S.Pati, U.Baid, B.Edwards, M.Sheller, S.Bakas.

Site PI/Senior member (of each collaborating group): C.Davatzikos, J.Villanueva-Meyer, M.Ingalhalikar, R.Y.Huang, Q.Dou, B.M.Ellingson, M.To, M.Kozubek, J.A.Maldjian, B.Wiestler, R.Jain, P.Tiwari, Y.Gusev, A.Abayazeed, R.R.Colen, P.Vollmuth, A.Mahajan, C.Balaña Quintero, S.Lee, M.Muzi, H.F.Shaykh, M.Trujillo, D.Marcus, B.Landman, A.Rao, M.Smits, H.I.Sair, R.Jeraj, B.Menze, Y.Yuan, A.Xavier Falcão, S.P.Niclou, B.Glocker, J.Teuwen, E.Pelaez, R.Wiest, S.Meckel, P.Guevara, S.Baek, H.Kim, D.M.Kardamakis, J.Saltz, L.Poisson, M.Vallières, F.Morón, A.E.Sloan, A.E.Flanders, G.Ogbole, D.R.Ormond, S.Currie, J.Farinhas, A.L.Simpson, C.Apgar, T.C.Booth.

Writing the Original Manuscript: S.Pati, U.Baid, B.Edwards, M.Sheller, S.Bakas.

Review, Edit, & Approval of the Final Manuscript: All authors.

§ COMPETING INTEREST DECLARATION

The Intel affiliated authors (B.Edwards, M.Sheller, S.Wang, G.A.Reina, P.Foley, A.Gruzdev, D.Karkada, P.Shah, J.Martin) would like to disclose the following (potential) competing interest as Intel employees. Intel may develop proprietary software that is related in reputation to the OpenFL open source project highlighted in this work. In addition, the work demonstrates feasibility of federated learning for brain tumor boundary detection models. Intel may benefit by selling products to support an increase in demand for this use case.

Illustration of the different approaches to train DL models during multi-site collaborations. a, Institutional incremental learning (IIL). b, Cyclic Institutional incremental learning. c, Federated learning by independent aggregation. d, Centralized learning by aggregating data.

Illustration of the U-Net architecture with residual connections, plotted using PlotNeuralNet <https://github.com/HarisIqbal88/PlotNeuralNet>. One spatial dimension is left out for clarity.

Comparative performance evaluation of the final consensus model, per tumor sub-compartment and averaged across cases. Comparison with the public initial model with each single site model (for collaborators holding $>200$ samples), as well as with their ensemble. Significance of results (Wilcoxon signed-rank test): Site 1: $p_{Average}=2\times 10^{-7}$, $p_{ET}=4\times 10^{-8}$, $p_{TC}=6\times 10^{-8}$, $p_{WT}=2\times 10^{-9}$; Site 2: $p_{Average}=5\times 10^{-6}$, $p_{ET}=1\times 10^{-5}$, $p_{TC}=1\times 10^{-4}$, $p_{WT}=1\times 10^{-4}$, Site 3: $p_{Average}=3\times 10^{-7}$, $p_{ET}=2\times 10^{-6}$, $p_{TC}=1\times 10^{-6}$, $p_{WT}=6\times 10^{-9}$, Site 4: $p_{Average}=4\times 10^{-6}$, $p_{ET}=5\times 10^{-5}$, $p_{TC}=8\times 10^{-7}$, $p_{WT}=8\times 10^{-3}$.

Average $DSC$ over cases for various singlet and triplet models against the centralized out-of-sample data.

Model Type     Tumor Sub-Compartment

singlet_0     Average      0.743996

ET      0.683714

TC      0.713348

WT      0.834925

singlet_1     Average      0.745362

ET      0.690743

TC      0.714308

WT      0.831035

singlet_2     Average      0.742246

ET      0.680600

TC      0.711266

WT      0.834871

singlet_3     Average      0.745255

ET      0.697808

TC      0.710747

WT      0.827210

singlet_4     Average      0.742130

ET      0.687588

TC      0.711013

WT      0.827789

triplet_0     Average      0.749014

ET      0.697808

TC      0.714308

WT      0.834925

triplet_1     Average      0.750356

ET      0.697808

TC      0.718336

WT      0.834925

triplet_2     Average      0.748694

ET      0.697808

TC      0.713348

WT      0.834925

triplet_3     Average      0.748996

ET      0.697808

TC      0.714308

WT      0.834871

triplet_4     Average      0.746683

ET      0.697808

TC      0.714308

WT      0.827934

Average $DSC$ over cases for various singlet and triplet models against the centralized out-of-sample data

Results and improvements in $DSC$ for various configurations of the singlet and triplet models when compared with the public initial model. This table also includes the significance of the differences as calculated using Wilcoxon signed-ranked test.

singlet_0     singlet_1     singlet_2     singlet_3         singlet_4

triplet     metric

triplet_0     Average       5.9e-06          0.31       8.3e-12          0.21      1.8e-06

ET       3.2e-19       5.1e-06       3.7e-17            NA      2.3e-22

TC          0.35            NA       2.3e-06          0.37      7.6e-01

WT            NA          0.34         0.026          0.61      3.7e-04

triplet_1     Average       5.5e-11        0.0047       1.6e-16        0.0017      1.1e-13

ET       3.2e-19       5.1e-06       3.7e-17            NA      2.3e-22

TC         0.012         0.013       2.4e-09         0.027      1.6e-02

WT            NA          0.34         0.026          0.61      3.7e-04

triplet_2     Average       4.3e-19          0.97         7e-13          0.21      6.4e-06

ET       3.2e-19       5.1e-06       3.7e-17            NA      2.3e-22

TC            NA          0.35       2.6e-08          0.83      2.9e-01

WT            NA          0.34         0.026          0.61      3.7e-04

triplet_3     Average        0.0021          0.57       1.9e-11          0.87      4.6e-05

ET       3.2e-19       5.1e-06       3.7e-17            NA      2.3e-22

TC          0.35            NA       2.3e-06          0.37      7.6e-01

WT         0.026       0.00071            NA          0.24      8.8e-01

triplet_4     Average          0.53       6.5e-08          0.16        0.0094      7.7e-01

ET       3.2e-19       5.1e-06       3.7e-17            NA      2.3e-22

TC          0.35            NA       2.3e-06          0.37      7.6e-01

WT         4e-26       7.7e-21       3.7e-24       4.9e-10      1.3e-12

P values providing significance of the difference in mean $DSC$ scores across cases between each singlet and triplet model pairing on the centralized out-of-sample data. Significance is calculated using Wilcoxon signed-ranked test where sample pairs are correspondingly the singlet and triplet model score for a single case. Entries with `NA' correspond to scoring where the singlet model was the member of the triplet responsible for producing the particular tumor sub-compartment output, so that the scores between the singlet and the triplet model for that tumor sub-compartment were identical.

5c|singlet Index

Demographic information of all cases used in this study.

4c|[HTML]FFFFFFAGE: Male
4c|[HTML]FFFFFFAGE: Female
3c|[HTML]FFFFFFIDH status
-2*[HTML]FFFFFFSite ID
c]@c@PIM: 47, 51, 55,
57, 58, 61,
62, 63, 64,
65, 66, 67,
68, 69, 70, 71

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

Radiological Society of North America

titlePattern analysis of dynamic susceptibility contrast-enhanced MR imaging demonstrates peritumoral tissue heterogeneity

Oxford University Press

titleImaging surrogates of infiltration obtained via multiparametric imaging pattern analysis predict subsequent location of recurrence of glioblastoma

journaltitleClinical Cancer Research
titleIn vivo detection of EGFRvIII in glioblastoma via perfusion magnetic resonance imaging signature consistent with deep peritumoral infiltration: the $\varphi$-index

Oxford University Press US

titleIn vivo evaluation of EGFRvIII mutation in primary glioblastoma patients via complex multiparametric MRI signature

journaltitleCancer cell
titleEpidermal growth factor receptor extracellular domain mutations in glioblastoma present opportunities for clinical imaging and therapeutic development

International Society for Optics

booktitleMedical Imaging 2018: Computer-Aided Diagnosis
titleDeriving stable multi-parametric MRI radiomic signatures in the presence of inter-scanner variations: survival prediction of glioblastoma via imaging pattern analysis and machine learning techniques

Oxford University Press US

titleNIMG-40. Non-invasive in vivo signature of idh1 mutational status in high grade glioma, from clinically-acquired multi-parametric magnetic resonance imaging, using multivariate machine learning

American Medical Association

journaltitleJAMA pediatrics
titleUse of fetal magnetic resonance image analysis and machine learning to predict the need for postnatal cerebrospinal fluid diversion in fetal ventriculomegaly

journaltitleMagnetic resonance imaging
titlePrecision diagnostics based on machine learning-derived imaging signatures

Wiley Online Library

journaltitleJournal of Magnetic Resonance Imaging
titleImaging signatures of glioblastoma molecular characteristics: a radiogenomics review

International Society for Optics

journaltitleJournal of Medical Imaging
titleOverall survival prediction in glioblastoma patients using structural magnetic resonance imaging (MRI): advanced radiomic features may compensate for lack of advanced MRI modalities

Wiley Online Library

titleHistopathology-validated machine learning radiographic biomarker for noninvasive discrimination between true progression and pseudo-progression in glioblastoma

Annual Reviews

journaltitleAnnual review of biomedical engineering
titleIntegrated biophysical modeling and image analysis: application to neuro-oncology

journaltitleComputerized medical imaging and graphics
titleAnalyzing magnetic resonance imaging data from glioma patients using deep learning

journaltitleMedical Image Analysis
titleThe reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study

Public Library of Science San Francisco, CA USA

journaltitlePLoS medicine
titleVariable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study

NIH Public Access

journaltitleThe New England journal of medicine
titlePredicting the future—big data, machine learning, and clinical medicine

journaltitlearXiv preprint arXiv:1801.00631
titleDeep learning: A critical appraisal

titleNeural networks and deep learning

journaltitleBrain imaging and behavior
titleThe ENIGMA Consortium: large-scale collaborative analyses of neuroimaging and genetic data

titleGlioma through the looking GLASS: molecular evolution of diffuse gliomas and the Glioma Longitudinal Analysis Consortium

Oxford University Press US

titleAI-based prognostic imaging biomarkers for precision neuro-oncology: the ReSPOND consortium

Oxford University Press US

titleiGLASS: imaging integration into the Glioma Longitudinal Analysis Consortium

Nature Publishing Group

journaltitleNPJ digital medicine
titleThe future of digital health with federated learning

Nature Publishing Group

journaltitleScientific reports
titleFederated learning in medicine: facilitating multi-institutional collaborations without sharing patient data

MEDICAL PUBLISHING GROUP-MASS MEDIC SOCIETY

journaltitleNew England Journal of Medicine
titleHIPAA regulations-a new era of medical-record privacy?

journaltitleA Practical Guide, 1st Ed., Cham: Springer International Publishing
titleThe eu general data protection regulation (gdpr)

booktitleArtificial intelligence and statistics
titleCommunication-efficient learning of deep networks from decentralized data

booktitleInternational MICCAI Brainlesion Workshop
titleMulti-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation

Nature Publishing Group

journaltitleNature medicine
titleFederated learning for predicting clinical outcomes in patients with COVID-19

Oxford University Press

journaltitleJournal of the American Medical Informatics Association
titleDistributed deep learning networks among institutions for medical imaging

booktitleProceedings of the Second Workshop on Distributed Infrastructures for Deep Learning
titleA performance evaluation of federated learning algorithms

Oxford University Press

journaltitleJournal of the American Medical Informatics Association
titleFederated learning improves site performance in multicenter deep learning without data sharing

booktitleClinical Image-Based Procedures, Distributed and Collaborative Learning, Artificial Intelligence for Combating COVID-19 and Secure and Privacy-Preserving Machine Learning
titleMulti-task Federated Learning for Heterogeneous Pancreas Segmentation

journaltitleMedical image analysis
titleFederated semi-supervised learning for COVID region segmentation in chest CT using multi-national data from China, Italy, Japan

Nature Publishing Group

journaltitleNature medicine
titleClinically applicable deep learning for diagnosis and referral in retinal disease

Nature Publishing Group

journaltitleNature medicine
titleCardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network

journaltitleMolecular genetics and metabolism
titleClinical research for rare disease: opportunities, challenges, and solutions

journaltitleChin Clin Oncol
titleAdvanced magnetic resonance imaging in glioblastoma: a review

titleThe somatic genomic landscape of glioblastoma

journaltitleCancer cell
titleIntegrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in PDGFRA, IDH1, EGFR, and NF1

National Acad Sciences

journaltitleProceedings of the National Academy of Sciences
titleIntratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics

Oxford University Press US

titleCBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2012–2016

Oxford Academic

titleThe 2021 WHO classification of tumors of the central nervous system: a summary

Am Soc Neuroradiology

journaltitleAmerican Journal of Neuroradiology
titleDeep transfer learning and radiomics feature prediction of survival of patients with high-grade gliomas

booktitleBrainLes 2015
titleGLISTRboost: combining multimodal MRI segmentation, registration, and biophysical tumor growth modeling with gradient boosting machines for glioma segmentation

journaltitleProceeding of the Multimodal Brain Tumor Image Segmentation Challenge
titleSegmentation of gliomas in multimodal magnetic resonance imaging volumes based on a hybrid generative-discriminative framework

booktitleInternational Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries
titleSegmentation of gliomas in pre-operative and post-operative multimodal magnetic resonance imaging volumes based on a hybrid generative-discriminative framework

journaltitleFrontiers in Computational Neuroscience
titleMulti-disease segmentation of gliomas and white matter hyperintensities in the BraTS data using a 3D convolutional neural network

journaltitleBiomedical signal processing and control
titleLongitudinal brain tumor segmentation prediction in MRI using feature and label fusion

booktitleDomain Adaptation and Representation Transfer, and Distributed and Collaborative Learning
titleFederated learning for breast density classification: A real-world implementation

journaltitleJournal of Clinical Neuroscience
titleMulti-institutional validation of a preoperative scoring system which predicts survival for patients with glioblastoma

American Society of Clinical Oncology

journaltitleJCO clinical cancer informatics
titleCancer imaging phenomics via CaPTk: multi-institutional prediction of progression-free survival and pattern of recurrence in glioblastoma

journaltitleIEEE transactions on medical imaging
titleThe multimodal brain tumor image segmentation benchmark (BRATS)

Nature Publishing Group

journaltitleScientific data
titleAdvancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features

journaltitlearXiv preprint arXiv:1811.02629
titleIdentifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge

journaltitlearXiv preprint arXiv:2107.02314
titleThe rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification

American Society of Clinical Oncology

titleRTOG 0825: Phase III double-blind placebo-controlled trial evaluating bevacizumab (Bev) in patients (Pts) with newly diagnosed glioblastoma (GBM).

Mass Medical Soc

journaltitleNew England Journal of Medicine
titleA randomized trial of bevacizumab for newly diagnosed glioblastoma

Oxford University Press US

titlePrognostic value of contrast enhancement and FLAIR for survival in newly diagnosed glioblastoma treated with and without bevacizumab: results from ACRIN 6686

Oxford University Press US

titleValue of dynamic contrast perfusion MRI to predict early response to bevacizumab in newly diagnosed glioblastoma: results from ACRIN 6686 multicenter trial

booktitleInternational Conference on Medical image computing and computer-assisted intervention
titleU-net: Convolutional networks for biomedical image segmentation

booktitleInternational conference on medical image computing and computer-assisted intervention
title3D U-Net: learning dense volumetric segmentation from sparse annotation

booktitleProceedings of the IEEE conference on computer vision and pattern recognition
titleDeep residual learning for image recognition

booktitleDeep Learning and Data Labeling for Medical Applications
titleThe importance of skip connections in biomedical image segmentation

booktitleInternational MICCAI Brainlesion Workshop
titleBrain tumor segmentation based on 3D residual U-Net

journaltitlearXiv preprint arXiv:1412.6980
titleAdam: A method for stochastic optimization

booktitleDeep learning in medical image analysis and multimodal learning for clinical decision support
titleGeneralised dice overlap as a deep learning loss function for highly unbalanced segmentations

journaltitleIEEE transactions on medical imaging
titleMorphometric analysis of white matter lesions in MR images: method and validation

Nature Publishing Group

journaltitleNature Methods
titlennU-Net: a self-configuring method for deep learning-based biomedical image segmentation

journaltitleNeural Computing and Applications
titleEfficient and robust deep learning with correntropy-induced loss function

booktitleInternational workshop on machine learning in medical imaging
titleTversky loss function for image segmentation using 3D fully convolutional deep networks

journaltitlearXiv preprint arXiv:1908.03679
titleDistance map loss penalty term for semantic segmentation

journaltitlearXiv preprint arXiv:1906.11031
titleContinuous dice coefficient: a method for evaluating probabilistic segmentations

booktitleProc. Symp. Appl. Math
titleThe hadamard product

Wiley Online Library

journaltitleJournal of the Royal Statistical Society: Series C (Applied Statistics)
titleL1 approximation and the analysis of data

Nature Publishing Group

journaltitleNature Machine Intelligence
titleSecure, privacy-preserving and federated machine learning in medical imaging

journaltitleIEEE Security & Privacy
titleThe untapped potential of trusted execution environments on mobile devices

journaltitleAdvances in neural information processing systems
titleTransfusion: Understanding transfer learning for medical imaging

journaltitleInternational Journal of Advanced Trends in Computer Science and Engineering
titleApplicability of various pre-trained deep convolutional neural networks for pneumonia classification based on X-Ray Images

journaltitleIntel White Paper
titleLower numerical precision deep learning inference and training

booktitleInternational conference on machine learning
titleFixed point quantization of deep convolutional networks

journaltitleIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
titleEnhanced precision analysis for accuracy-aware bit-width optimization using affine arithmetic

Nature Publishing Group

journaltitleScientific data
titleThe FAIR Guiding Principles for scientific data management and stewardship

International Society for Optics

journaltitleJournal of medical imaging
titleCancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome

booktitleInternational MICCAI Brainlesion Workshop
titleBrain cancer imaging phenomics toolkit (brain-CaPTk): an interactive platform for quantitative analysis of glioblastoma

booktitleInternational MICCAI Brainlesion Workshop
titleThe cancer imaging phenomics toolkit (captk): Technical overview

journaltitleAlzheimer's & Dementia: The Journal of the Alzheimer's Association
titleFast automatic segmentation of hippocampal subfields and medial temporal lobe subregions in 3 Tesla and 7 Tesla T2-weighted MRI

titleUser-Guided 3D Active Contour Segmentation of Anatomical Structures: Significantly Improved Efficiency and Reliability

titleBrain extraction from normal and pathological images: A joint PCA/image-reconstruction approach

titleBrain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training

journaltitlearXiv preprint arXiv:2103.01006
titleGandlf: A generally nuanced deep learning framework for scalable end-to-end clinical workflows in medical imaging

journaltitleMedical image analysis
titleEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation

booktitleInternational MICCAI Brainlesion Workshop
titleEnsembles of densely-connected CNNs with label-uncertainty for brain tumor segmentation

titleLabelFusion: Medical Image label fusion of segmentations

booktitleAdvances in neural information processing systems
titlePytorch: An imperative style, high-performance deep learning library

journaltitleComputer Methods and Programs in Biomedicine
titleTorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning
Medical image computing,Deep learning,Data augmentation,Preprocessing

journaltitlearXiv preprint arXiv:2105.06413
titleOpenFL: An open-source framework for Federated Learning

booktitleProceedings of the IEEE/CVF International Conference on Computer Vision Workshops
titleOpenVINO deep learning workbench: Comprehensive analysis and tuning of neural networks inference