8787institutetext: Center for Biomedical Image Computing and Analytics
(CBICA), University of Pennsylvania, Philadelphia, PA, USA 22institutetext:
Department of Radiology, Perelman School of Medicine at the University of
Pennsylvania, Philadelphia, PA, USA 33institutetext: Department of Pathology
and Laboratory Medicine, Perelman School of Medicine at the University of
Pennsylvania, Philadelphia, PA, USA 44institutetext: Department of Radiology &
Biomedical Imaging, University of California San Francisco, CA, USA
55institutetext: Unity Health Toronto, University of Toronto, Toronto, ON
66institutetext: Center for Biomedical Informatics and Information Technology,
National Cancer Institute, National Institutes of Health 77institutetext:
Athinoula A. Martinos Center for Biomedical Imaging, MGH/Harvard Medical
School, MA, USA 88institutetext: Department of Radiology, Thomas Jefferson
University, Philadelphia, PA, USA 99institutetext: Universidade Federal de São
Paulo (UNIFESP), São Paulo, Brazil 1010institutetext: Diagnósticos da América
SA (DASA), São Paulo, Brazil 1111institutetext: Department of Informatics,
Technical University of Munich, Munich, Germany 1212institutetext: The Ohio
State University Wexner Medical Center, Columbus, OH, USA 1313institutetext:
Penn Statistics in Imaging and Visualization Center, Department of
Biostatistics, Epidemiology and Informatics, Perelman School of Medicine,
University of Pennsylvania, Philadelphia, PA, USA 1414institutetext: Sage
Bionetworks, Seattle, WA, USA 1515institutetext: Kaggle 1616institutetext:
Center for Intelligent Imaging, University of California San Francisco, CA,
USA 1717institutetext: Leidos Biomedical Research, Inc., Frederick National
Laboratory for Cancer Research, Frederick, MD 21701, USA 1818institutetext:
Department of Neuroradiology, Klinikum rechts der Isar, School of Medicine,
Technical University of Munich, München, Germany 1919institutetext: Department
of Diagnostic Radiology, University of Texas MD Anderson Cancer Center,
Houston, TX, USA 2020institutetext: Hillman Cancer Center, University of
Pittsburgh Medical Center, Pittsburgh, PA, 15232, USA 2121institutetext:
Neuroimaging Informatics and Analysis Center, Washington University in St.
Louis, St. Louis, MO, USA 2222institutetext: Department of Radiology,
Washington University in St. Louis, St. Louis, MO, USA 2323institutetext:
Department of Neurology, The University of Alabama at Birmingham, Birmingham,
AL, USA 2424institutetext: Institute for Surgical Technology and Biomechanics,
University of Bern, Bern, Switzerland 2525institutetext: Support Centre for
Advanced Neuroimaging Inselspital, Institute for Diagnostic and Interventional
Neuroradiology,Bern University Hospital, Bern, Switzerland 2626institutetext:
Center for MR-Research, University Children’s Hospital Zurich, Zurich,
Switzerland 2727institutetext: Institute of Diagnostic and Interventional
Radiology, Pediatric Radiology and Neuroradiology, University Medical Center
Rostock, Rostock, Germany 2828institutetext: Department of Radiodiagnosis,
Tata Memorial Hospital, Tata Memorial Centre, Homi Bhabha National Institute,
Mumbai, India 2929institutetext: Department of Quantitative Biomedicine,
University of Zurich, Zurich, Switzerland 3030institutetext: Mercy catholic
medical center, Darby, PA, USA 3131institutetext: Diagnóstico Avançado por
Imagem and Hospital Erasto Gaertner, Curtiba, Brazil 3232institutetext:
Department of Medical Imaging, Gold Coast University Hospital, Southport,
Australia 3333institutetext: Department of Radiology, University of Ottawa,
Ottawa, Canada 3434institutetext: Department of Radiology, Baylor College of
Medicine, Houston, TX, USA 3535institutetext: Department of Radiology,
Hennepin Healthcare, Minneapolis, MN, USA 3636institutetext: Uniformed
Services University, Bethesda, MD, USA 3737institutetext: Institute of
Neurology, University College London, London, United Kingdom
3838institutetext: Virtual Radiologic Professionals, LLC - Branson, Eden
Prairie, MN, USA 3939institutetext: University of Pittsburgh Medical Center,
Pittsburg, PA, USA 4040institutetext: Hahnemann University Hospital Drexel
University College of Medicine, PA, USA 4141institutetext: Department of
Radiology, Oregon Health & Science University, Portland, OR, USA
4242institutetext: Dr Jones and Partners Medical Imaging, South Australia
4343institutetext: Department of Neuroradiology. Hospital Italiano de Buenos
Aires, Buenos Aires, Argentina 4444institutetext: Mallinckrodt Institute of
Radiology, Washington University in St. Louis, MO, USA 4545institutetext:
Neuroradiology Department, Hospital Garcia de Orta EPE, Almada, Portugal
4646institutetext: Department of Radiology, Tufts MedicalCenter, Boston, MA,
USA. 4747institutetext: Centro Universitario Euro-Americana (UNIEURO),
Brasília, DF, Brazil 4848institutetext: Department of Radiology, Division of
Neuroradiology, University of Southern California, Keck School of Medicine,
Los Angeles, CA, USA. 4949institutetext: Radiology (Neuroradiology Section),
University of Louisville, Louisville, KY, USA 5050institutetext: Northwestern
University Feinberg School of Medicine, Chicago, IL, USA 5151institutetext:
Stanford Hospital and Clinics, Stanford University, Stanford, CA, USA
5252institutetext: Department of Radiology and Medical Imaging University of
Virginia Health System Charlottesville, VA, USA 5353institutetext: Department
of Radiology, Baylor College of Medicine, Houston, Tex, USA 5454institutetext:
NYU Langone Medical Center, New York, NY, USA 5555institutetext: Kaiser
Permanente, San Diego, CA, USA 5656institutetext: Instituto Nacional de
Ciencias Medicas y Nutricion, Maxico City, Maxico 5757institutetext: Northport
VA Medical Center Northport, NY, USA 5858institutetext: Ohio State University
Wexner Medical Center, Columbus, OH, USA 5959institutetext: University of
Virginia Medical Center, Charlottesville, VA, USA 6060institutetext:
Department of Radiology, Kingston General Hospital, Queen’s University,
Kingston, Canada 6161institutetext: Department of Radiology, University of
Minnesota Health,Minneapolis, MN, USA 6262institutetext: NorthShore University
HealthSystem, Chicago, IL, USA 6363institutetext: Neuroradiology and Imaging
Informatics Imaging Institute, Cleveland Clinic, Cleveland, OH, USA
6464institutetext: University of Utah Health Sciences Center, Salt Lake City,
UT, USA 6565institutetext: London Health Sciences Centre, London, Ontario,
Canada 6666institutetext: Dr Abdurrahman Yurtaslan Ankara Oncology Training
and Research Hospital, University of Health Sciences, Ankara, Turkey
6767institutetext: University of Texas Health San Antonio, TX, USA
6868institutetext: Department of Radiology, Beth Israel Deaconess Medical
Center, Harvard Medical School, Boston, MA, USA 6969institutetext:
Neuroradiology and Imaging Informatics Imaging Institute, Cleveland Clinic,
Cleveland, OH, USA 7070institutetext: Emory University, Atlanta, GA, USA
7171institutetext: Memorial Sloan Kettering Cancer Center, New York, NY, USA
7272institutetext: Yale University School of Medicine, Department of Radiology
& Biomedical Imaging, New Haven, CT, USA 7373institutetext: Mayo Clinic,
Jacksonville, FL, USA 7474institutetext: Dartmouth Hitchcock Medical Center,
NH, USA 7575institutetext: Oakland University William Beaumont School of
Medicine, Rochester, MI, USA 7676institutetext: Radiology Department at
Klinikum Hochrhein Waldshut-Tiengen, Germany 7777institutetext: Maastricht
University Hospital, Maastricht, The Netherlands 7878institutetext: Radiology
department,Centre Hospitalier de l’Universite de Montreal (CHUM), Centre de
Recherche du CHUM (CRCHUM) Montreal, Quebec, Canada 7979institutetext:
Department of Radiology, NewYork-Presbyterian Hospital, Weill Cornell Medical
College, New York, NY, USA 8080institutetext: University of Iowa Hospitals and
Clinics, Iowa City, IA, USA 8181institutetext: Department of Radiology
Stanford University, Stanford, CA, USA 8282institutetext: Department of
Radiology, Northwell Health, Zucker Hofstra School of Medicine at Northwell,
North Shore University Hospital, Hempstead, New York, NY, USA.
8383institutetext: Department of Medical Imaging, University of Toronto, ON,
Canada 8484institutetext: University of California Los Angeles, CA, USA
8585institutetext: University of Rochester Medical Center,Rochester, NY, USA
8686institutetext: Lahey Clinic, Burlington, MA, USA 8787institutetext: Temple
University Hospital, Philadelphia, PA, USA
† People involved in the organization of the challenge.
‡ People contributing data from their institutions.
§ People involved in annotation process.
¶ Equal senior authors.
* Corresponding author: 8787email<EMAIL_ADDRESS>
# The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and
Radiogenomic Classification

Ujjwal Baid 112233††‡‡ Satyam Ghodasara 22†† Suyash Mohan 1122†† Michel
Bilello 1122†† Evan Calabrese 44††‡‡§§ Errol Colak 55††‡‡ Keyvan Farahani 66††
Jayashree Kalpathy-Cramer 77†† Felipe C. Kitamura 991010†† Sarthak Pati
1122331111†† Luciano M. Prevedello 1212†† Jeffrey D. Rudie 44††‡‡§§ Chiharu
Sako 1122†† Russell T. Shinohara 111313†† Timothy Bergquist 1414†† Rong Chai
1414†† James Eddy 1414†† Julia Elliott 1515†† Walter Reade 1515†† Thomas
Schaffter 1414†† Thomas Yu 1414†† Jiaxin Zheng 1414†† Ahmed W. Moawad 3030§§
Luiz Otavio Coelho 3131§§ Olivia McDonnell 3232§§ Elka Miller 3333§§ Fanny E.
Morón 3434§§ Mark C. Oswood 3535§§ Robert Y. Shih 3636§§ Loizos Siakallis
3737§§ Yulia Bronstein 3838§§ James R. Mason 3939§§ Anthony F. Miller 4040§§
Gagandeep Choudhary 4141§§ Aanchal Agarwal 4242§§ Cristina H. Besada 4343§§
Jamal J. Derakhshan 4444§§ Mariana C. Diogo 4545§§ Daniel D. Do-Dai 4646§§
Luciano Farage 4747§§ John L. Go 4848§§ Mohiuddin Hadi 4949§§ Virginia B. Hill
5050§§ Michael Iv 5151§§ David Joyner 5252§§ Christie Lincoln 5353§§ Eyal
Lotan 5454§§ Asako Miyakoshi 5555§§ Mariana Sanchez-Montaño 5656§§ Jaya Nath
5757§§ Xuan V. Nguyen 5858§§ Manal Nicolas-Jilwan 5959§§ Johanna Ortiz Jimenez
6060§§ Kerem Ozturk 6161§§ Bojan D. Petrovic 6262§§ Chintan Shah 6363§§ Lubdha
M. Shah 6464§§ Manas Sharma 6565§§ Onur Simsek 6666§§ Achint K. Singh 6767§§
Salil Soman 6868§§ Volodymyr Statsevych 6969§§ Brent D. Weinberg 7070§§ Robert
J. Young 7171§§ Ichiro Ikuta 7272§§ Amit K. Agarwal 7373§§ Sword C. Cambron
7474§§ Richard Silbergleit 7575§§ Alexandru Dusoi 7676§§ Alida A. Postma
7777§§ Laurent Letourneau-Guillon 7878§§ Gloria J. Guzmán Pérez-Carrillo
4444§§ Atin Saha 7979§§ Neetu Soni 8080§§ Greg Zaharchuk 8181§§ Vahe M.
Zohrabian 8282§§ Yingming Chen 8383§§ Milos M. Cekic 8484§§ Akm Rahman 8585§§
Juan E. Small 8686§§ Varun Sethi 8787§§ Christos Davatzikos 1122‡‡ John Mongan
441616††‡‡ Christopher Hess 44‡‡ Soonmee Cha 44‡‡ Javier Villanueva-Meyer 44‡‡
John B. Freymann 1717‡‡ Justin S. Kirby 1717‡‡ Benedikt Wiestler 1818‡‡
Priscila Crivellaro 55‡‡ Rivka R. Colen 19192020‡‡ Aikaterini Kotrotsou 1919‡‡
Daniel Marcus 212122‡22‡ Mikhail Milchenko 21212222‡‡ Arash Nazeri 2222‡‡
Hassan Fathallah-Shaykh 2323‡‡ Roland Wiest 24242525‡‡ Andras Jakab 2626‡‡
Marc-André Weber 2727‡‡ Abhishek Mahajan 2828‡‡ Bjoern Menze 11112929††‡‡¶¶
Adam E. Flanders 88††¶¶ Spyridon Bakas 112233††‡‡¶¶**

###### Abstract

The BraTS 2021 challenge celebrates its $10^{th}$ anniversary and is jointly
organized by the Radiological Society of North America (RSNA), the American
Society of Neuroradiology (ASNR), and the Medical Image Computing and Computer
Assisted Interventions (MICCAI) society. Since its inception, BraTS has been
focusing on being a common benchmarking venue for brain glioma segmentation
algorithms, with well-curated multi-institutional multi-parametric Magnetic
Resonance Imaging (mpMRI) data. Gliomas are the most common primary
malignancies of the central nervous system, with varying degrees of
aggressiveness and prognosis. The RSNA-ASNR-MICCAI BraTS 2021 challenge
targets the evaluation of computational algorithms assessing the same tumor
compartmentalization, as well as the underlying tumor’s molecular
characterization, in pre-operative baseline mpMRI data from 2,040 patients.
Specifically, the two tasks that BraTS 2021 focuses on are: a) the
segmentation of the histologically distinct brain tumor sub-regions, and b)
the classification of the tumor’s O[6]-methylguanine-DNA methyltransferase
(MGMT) promoter methylation status. The performance evaluation of all
participating algorithms in BraTS 2021 will be conducted through the Sage
Bionetworks Synapse platform (Task 1) and Kaggle (Task 2), concluding in
distributing to the top ranked participants monetary awards of $60,000
collectively.

###### Keywords:

Glioma, glioblastoma, brain, tumor, BraTS, challenge, segmentation,
classification, MGMT, machine learning, radiomics

##  1 Introduction

Glioblastoma (GBM), and diffuse astrocytic glioma with molecular features of
GBM (WHO Grade 4 astrocytoma), are the most common and aggressive malignant
primary tumor of the central nervous system (CNS) in adults, with extreme
intrinsic heterogeneity in appearance, shape, and histology [1, 2, 3, 4, 5, 6,
7]. GBM patients have an average prognosis of 14 months, following standard of
care treatment (comprising surgical resection followed by radiotherapy and
chemotherapy), and 4 months left untreated [8]. Although various experimental
treatment options have been proposed during the past 20 years, there have not
been any substantial differences in patient prognosis.

Accurate identification of brain tumor sub-regions boundaries in MRI is of
profound importance in many clinical applications, such as surgical treatment
planning, image-guided interventions, monitoring tumor growth, and the
generation of radiotherapy maps. However, manual detection and tracing of
tumor sub-regions is tedious, time-consuming, and subjective. In a clinical
setup, this manual process is carried out by radiologists in a qualitative
visual manner, and hence becomes impractical when dealing with numerous
patients. This highlights the unmet need for automated deterministic
segmentation solutions that could contribute in expediting this process.

The release of the current revised World Health Organization (WHO)
classification of CNS tumors [9] highlighted the appreciation of integrated
diagnostics, and transitioned the clinical tumor diagnosis from a purely
morphologic-histopathologic classification to integrating molecular-
cytogenetic characteristics. O[6]-methylguanine-DNA methyltransferase (MGMT)
is a DNA repair enzyme that the methylation of its promoter in newly diagnosed
GBM has been identified as a favorable prognostic factor and a predictor of
chemotherapy response [10]. Thus, determination of MGMT promoter methylation
status in newly diagnosed GBM can influence treatment decision making.

The RSNA ASNR MICCAI Brain Tumor Segmentation (BraTS) 2021 challenge utilizes
multi-institutional multi-parametric Magnetic Resonance Imaging (mpMRI) scans,
to address both the automated tumor sub-region segmentation and the prediction
of one of the genetic characteristics of glioblastoma (MGMT promoter
methylation status) from pre-operative baseline MRI scans. Specifically, BraTS
2021 focuses on the evaluation of state-of-the-art methods for the accurate
segmentation of intrinsically heterogeneous brain glioma sub-regions and on
the evaluation of classification methods distinguishing between MGMT
methylated (MGMT+) and unmethylated (MGMT-) tumors. This manuscript describes
the characteristics of the data included in the BraTS 2021 challenge, along
with the annotation protocol followed to prepare the challenge data, an
elaborate description of the challenge’s tasks, and the performance evaluation
of all participating methods (in Section 2) and then discusses the limitations
and currently considered future directions (in Section 3).

##  2 Materials & Methods

###  2.1 Data

The BraTS dataset describes a retrospective collection of brain tumor mpMRI
scans acquired from multiple different institutions under standard clinical
conditions, but with different equipment and imaging protocols, resulting in a
vastly heterogeneous image quality reflecting diverse clinical practice across
different institutions. Inclusion criteria comprised pathologically confirmed
diagnosis and available MGMT promoter methylation status. These data have been
updated, since BraTS 2020 [11, 12, 13, 14, 15], increasing the total number of
cases from 660 to 2,000. Ground truth annotations of every tumor sub-region
for task 1 were approved by expert neuroradiologists, whereas the MGMT
methylation status was based on the laboratory assessment of the surgical
brain tumor specimen [16].

Following the paradigm of algorithmic evaluation in machine learning, the data
included in the BraTS 2021 challenge are divided in training, validation, and
testing datasets. The challenge participants are provided with the ground
truth labels only for the training data. The validation data are then provided
to the participants without any associated ground truth and the testing data
are kept hidden from the participants at all times.

Participants are not allowed to use additional public and/or private data
(from their own institutions) for extending the provided BraTS data, for the
training of the algorithm chosen to be ranked. Similarly, using models that
were pretrained on such datasets is not allowed. This is due to our intentions
to provide a fair comparison among the participating methods. However,
participants are allowed to use additional public and/or private data (from
their own institutions), only for scientific publication purposes and if they
explicitly mention this in their submitted manuscripts. Importantly,
participants that decide to proceed with this scientific analysis they must
also report results using only the BraTS’21 data to discuss potential result
differences.

####  2.1.1 Imaging Data Description

The mpMRI scans included in the BraTS 2021 challenge describe a) native (T1)
and b) post-contrast T1-weighted (T1Gd (Gadolinium)), c) T2-weighted (T2), and
d) T2 Fluid Attenuated Inversion Recovery (T2-FLAIR) volumes, acquired with
different protocols and various scanners from multiple institutions.

Standardized pre-processing has been applied to all the BraTS mpMRI scans.
Specifically, the applied pre-processing routines include conversion of the
DICOM files to the NIfTI file format [17], co-registration to the same
anatomical template (SRI24) [18], resampling to a uniform isotropic resolution
($1mm^{3}$), and finally skull-stripping. The pre-processing pipeline is
publicly available through the Cancer Imaging Phenomics Toolkit (CaPTk)
111https://cbica.github.io/CaPTk/ [19, 20, 21] and Federated Tumor
Segmentation (FeTS) tool 222https://github.com/FETS-AI/Front-End/. Conversion
to NIfTI strips the accompanying metadata from the DICOM images, and
essentially removes all Protected Health Information (PHI) from the DICOM
headers. Furthermore, skull-stripping mitigates potential facial
reconstruction/recognition of the patient [22, 23]. The specific approach we
have used for skull stripping is based on a novel DL approach that accounts
for the brain shape prior and is agnostic to the MRI sequence input [24].

Specifically for Task 1 (Tumor sub-region segmentation), all imaging volumes
have then been segmented using the STAPLE [25] fusion of previous top-ranked
BraTS algorithms, namely, nnU-Net [26], DeepScan [27], and DeepMedic [28].
These fused labels were then refined manually by volunteer neuroradiology
experts of varying rank and experience, following a consistently communicated
annotation protocol. The manually refined annotations were finally approved by
experienced board-certified attending neuro-radiologists, with more than 15
years of experience working with gliomas. The annotated tumor sub-regions are
based upon known observations visible to the trained radiologist (VASARI
features) and comprise the Gd-enhancing tumor (ET — label 4), the peritumoral
edematous/invaded tissue (ED — label 2), and the necrotic tumor core (NCR —
label 1). ET is the enhancing portion of the tumor, described by areas with
both visually avid, as well as faint, enhancement on T1Gd MRI. NCR is the
necrotic core of the tumor, the appearance of which is hypointense on T1Gd
MRI. ED is the peritumoral edematous and infiltrated tissue, defined by the
abnormal hyperintense signal envelope on the T2 FLAIR volumes, which includes
the infiltrative non enhancing tumor, as well as vasogenic edema in the
peritumoral region. The tumor sub-regions are shown in Fig. 1.

Figure 1: Glioma sub-regions considered in the RSNA-ASNR-MICCAI BraTS 2021
challenge. Image panels with the tumor sub-regions annotated in the different
mpMRI scans. The image panels A-C denote the regions considered for the
performance evaluation of the participating algorithms and specifically
highlight (from left to right): the enhancing tumor (ET - yellow) visible in a
T1Gd scan, surrounding the cystic/necrotic components of the core (panel A),
the tumor core (TC – magenta) and the whole tumor (WT - cyan) visible in the
corresponding T2 (panel B) and T2-FLAIR (panel C) scans, respectively. Panel
D, depicts the combined segmentations generating the final tumor sub-region
labels, as provided to the BraTS 2021 participants: enhancing core (yellow),
necrotic/cystic core (red), and edema/invasion (green).

For Task 2 (Radiogenomic Classification), all the imaging volumes were first
pre-processed as for Task 1, in order to obtain the skull-stripped volumes,
which are then converted from NIfTI to DICOM file format. To facilitate this
conversion, while ensuring that the original patient space is preserved, the
skull-stripped brain volumes in NIfTI format of each MRI sequence and their
corresponding original DICOM scans (in the patient space) are required. The
DICOM volume is read as an ITK image [29] and the skull-stripped volume is
rigidly registered to it, providing a transformation matrix that defines the
spatial mapping between the 2 volumes. This transformation matrix is applied
to the skull-stripped volume and to the corresponding segmentation labels, to
translate them both to the patient space. This process is taking place for
each modality separately. The transformed volumes are then passed through
CaPTk’s NIfTI to DICOM conversion engine to generate DICOM image volumes for
the skull-stripped images. Once all mpMRI sequences were converted back to the
DICOM file format, further de-identification took place based on a two-step
process. The first step used the RSNA CTP (Clinical Trials Processor)
Anonymizer 333http://mirc.rsna.org/download/Anonymizer-installer.jar with the
standard built-in script. Step two then consisted of whitelisting the DICOM
files from step 1. The whitelisting process removes all non-essential tags
from the DICOM header. This last process ensures there are no protected health
information (PHI) entries left in the DICOM header.

####  2.1.2 MGMT Promoter Methylation Data Description

The MGMT promoter methylation status data is defined as a binary label (0:
unmethylated, 1: methylated), and provided to the participants as a comma-
separated value (.csv) file with the corresponding pseudo-identifiers of the
mpMRI volumes (study-level label).

The MGMT promoter methylation status of the BraTS 2021 dataset was determined
at each of the host institutions based on various techniques, including
pyrosequencing, and next generation quantitative bisulfite sequencing of
promoter CpG sites. Sufficient tumor tissue collected at time of surgery was
required for both approaches. For the pyrosequencing approach, the genomic DNA
was initially extracted from 5lm tissue sections of formalin-fixed paraffin-
embedded (FFPE) tissue samples. DNA was further cleaned and purified. The DNA
concentration, protein to nucleic acid ratio, and DNA to RNA ratio for purity
were assessed by spectrophotometer. Approximately 500–1000ng total DNA was
subjected to bisulfite conversion using the EPiTect Bisulfite Kit. A total of
50–100 ng bisulfite-treated DNA was carried on for PCR using F-primer and
R-primer. Pyrosequencing methylation assay was then conducted using the
sequencing primer on the PyroMark Q96ID pyrosequencer. The Pyromark CpG MGMT
kit detected the average level of methylation on CpG 74–81 sites located in
the MGMT gene. A cytosine not followed by a guanine served as an internal
control for completion of bisulfite conversion. The percent methylation above
10% was interpreted as positive. A sample below 10% methylation was
interpreted as negative. For the latter approach, a total of 17 MGMT promoter
CpG sites were amplified by nested polymerase chain reaction (PCR) using a
bisulfite treated DNA template. Quantitative PCR was performed for each CpG
site to determine its methylation status. A result of 2% or more methylated
CpG sites in the MGMT promoter (out of 17 total sites) was considered a
positive result.

####  2.1.3 Comparison with Previous BraTS datasets

Table 1: Summary of distribution of BraTS Challenge data across training, validation and test cohorts since the inception of BraTS initiative. (TBA: To Be Announced) Year |  | Total
---
Data
| Training
---
Data
| Validation
---
Data
| Testing
---
Data
Tasks | Timepoint
2012 | 50 | 35 | NA | 15 | Segmentation | Pre-operative
2013 | 60 | 35 | NA | 25 | Segmentation | Pre-operative
2014 | 238 | 200 | NA | 38 | Segmentation | Longitudinal
2015 | 253 | 200 | NA | 53 |  | Segmentation
---
Disease progression
Longitudinal
2016 | 391 | 200 | NA | 191 |  | Segmentation
---
Disease progression
Longitudinal
2017 | 477 | 285 | 46 | 146 |  | Segmentation
---
Survival prediction
Pre-operative
2018 | 542 | 285 | 66 | 191 |  | Segmentation
---
Survival prediction
Pre-operative
2019 | 626 | 335 | 125 | 166 |  | Segmentation
---
Survival prediction
Pre-operative
2020 | 660 | 369 | 125 | 166 |  | Segmentation
---
Survival prediction
Pre-operative
| 2021
---
2040 | 1251 | 219 | 570 |  | Segmentation
---
MGMT classification
Pre-operative

The first BraTS challenge was organized in 2012 in conjunction with the MICCAI
conference, and was making available a total of 50 mpMRI glioma cases (Table
1). The BraTS’12-’13 dataset was manually annotated by clinical experts, and
the task at hand was the segmentation of the glioma sub-regions (ET, NCR, ED).
In BraTS’14-’16 the dataset provided to the participants included a large
contribution of data from The Cancer Imaging Archive (TCIA) [30], and
specifically from the TCGA-GBM [31] and the TCGA-LGG [32] collections. Both
pre- and post-operative scans were included from these collections, and the
ground truth segmentations were annotated by the fusion of previous algorithms
that ranked highly during BraTS’12 and ’13. During the BraTS’17 challenge all
the data were revised by board-certified neuroradiologists, who assessed the
complete TCIA collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and
categorized each scan as pre- or post-operative, and only the scans without
any prior instrumentation were included as a part of the BraTS challenge this
year onwards [14, 15, 12]. In BraTS’17-’20’ the challenge was extended to the
prediction of patient overall survival for the glioblastoma cases that
underwent gross-total resection. This year, the BraTS 2021 challenge continues
its focus on the segmentation of glioma sub-regions, with a substantially
larger dataset (2,000 glioma cases = 8,000 mpMRI scans), and extends to the
clinically relevant task of identifying the tumor’s MGMT promoter methylation
status (methylated/unmethylated). These additional exams were obtained as a
collection of the pre-operative cases of the TCIA public collections of TCGA-
GBM, TCGA-LGG, IvyGAP [33, 34], CPTAC-GBM [35, 36], and ACRIN-FMISO-Brain
(ACRIN 6684) [37, 38], as well as contributions from private institutional
collections. The name mapping between the previous and the current challenge,
as well as all the TCIA collections will be provided to further facilitate
research beyond the directly BraTS related tasks.

####  2.1.4  Tumor Annotation Protocol

We designed the following tumor annotation protocol, to ensure consistency in
the ground truth delineations across various annotators. For the tasks related
to BraTS, only structural mpMRI volumes were considered (T1, T1Gd, T2,
T2-FLAIR), all of them co-registered to a common anatomical template (SRI24
[18]) and resampled to 1mm3. The end to end pipeline is available for these
through CaPTk [19, 20, 21] and FeTS tools. We note that radiologic definition
of tumor boundaries, especially in such infiltrative tumors as gliomas, is a
well-known problem. In an attempt to offer a standardized approach to assess
and evaluate various tumor sub-regions, the BraTS initiative, after
consultation with internationally recognized expert neuroradiologists, defined
the various tumor sub-regions. However, we note that other criteria for
delineation could be set, resulting in slightly different tumor sub-regions.
For the BraTS 2021 challenge the regions considered are: i) the “enhancing
tumor” (ET), ii) the “tumor core” (TC) and iii) the complete tumor extent also
referred to as the “whole tumor” (WT). The ET is described by areas that show
hyper-intensity in T1Gd when compared to T1, but also when compared to
“healthy” white matter in T1Gd. The TC describes the bulk of the tumor, which
is what is typically considered for surgical excision. The TC entails the ET,
as well as the necrotic (NCR) parts of the tumor, the appearance of which is
typically hypo-intense in T1Gd when compared to T1. The WT describes the
complete extent of the disease, as it entails the TC and the peritumoral
edematous/invaded tissue (ED), which is typically depicted by the abnormal
hyper-intense signal in the T2-FLAIR volume.

The BraTS tumor sub-regions (visual features) are image-based and do not
reflect strict biologic entities. For example, the ET regions may be defined
as hyper-intense signal on T1Gd images. However, in high grade tumors, non-
necrotic, non-cystic regions are present that do not enhance and can be
separable from the surrounding vasogenic edema, representing non-enhancing
infiltrative tumor. Another issue is defining the tumor center in low grade
gliomas as it is difficult to differentiate tumor from vasogenic edema,
particularly in the absence of enhancement. In the previous BraTS challenges
annotators would start from the manual delineation of the abnormal signal in
the T2-weighted images, primarily defining the WT, then address the TC, and
finally the enhancing and non-enhancing/necrotic core, possibly using semi-
automatic tools.

To facilitate the annotation process for BraTS 2021, initial automated
segmentations were generated by fusing previously top-performing BraTS
methods. The specific methods fused were the DeepMedic [28], DeepScan [27],
and nnU-Net [26], all trained on the BraTS 2020 dataset [11, 12, 13]. The
STAPLE label fusion [25] was used to aggregate the segmentation produced by
each of the individual methods, and account for systematic errors generated by
each of them separately. All these segmentation methods and the exact pipeline
used to generate the fused automated segmentation has been made publicly
available through the Federated Tumor Segmentation (FeTS)
platform444https://www.med.upenn.edu/cbica/fets/ [39].

The volunteer neuroradiology expert annotators were provided with four mpMRI
scans along with the fused automated segmentation volume to initiate the
manual refinements. The ITK-SNAP [40] software was used for making these
refinements. Once the automated segmentations were refined by the annotators,
two senior attending board-certified neuroradiologists with more than 15 years
of experience each, reviewed the segmentations. Depending upon correctness,
these segmentations were either approved or returned to the individual
annotator for further refinements. This process was followed iteratively until
the approvers found the refined tumor sub-region segmentations acceptable for
public release and the challenge conduction.

####  2.1.5 Common errors of automated segmentations

Building upon observations during all previous BraTS instances, we note some
common errors in the automated segmentations. The most typical such errors
observed are:

  1. 1.

The choroid plexus and areas of T1 bright blood products (when they can be
discriminated by comparing with the pre contrast T1 images), have erroneously
been labelled as ED (Fig. 2(a)).

  2. 2.

Vessels within the peritumoral T2 FLAIR edematous area, have been marked as ET
(Fig. 2(b)).

  3. 3.

Vessels within the peritumoral T2 FLAIR edematous area, have been marked as ED
(Fig. 2(c)).

  4. 4.

Periventricular white matter hyperintensities being confused and segmented as
tumor/peritumoral regions (Fig. 2(d)).

(a) Choroid plexus erroneously marked as ED.

(b) Vessels in ED marked at ET.

(c) Vessels in ED

(d) Periventricular white matter hyperintensities. Figure taken from [41].

Figure 2: Common errors expected from the automatic segmentations.

###  2.2 Challenge Tasks

The BraTS 2021 challenge utilizes multi-institutional mpMRI scans, and focuses
on (Task 1) the evaluation of state-of-the-art methods for the segmentation of
intrinsically heterogeneous brain glioblastoma sub-regions in mpMRI scans.
Furthermore, to pinpoint the clinical relevance of this segmentation task,
BraTS 2021 also focuses on (Task 2) the evaluation of methods to predict the
MGMT promoter methylation status at the pre-operative baseline scans, via
integrative analyses of quantitative imaging phenomic features and machine
learning algorithms. Participants are free to choose whether they want to
focus only on one or both tasks.

####  2.2.1 Task 1: Brain Tumor Sub-region Segmentation

The participants are called to address this task by using the provided
clinically-acquired training data to develop their method and produce
segmentation labels of the glioma sub-regions. The sub-regions considered for
evaluation are the “enhancing tumor” (ET), the “tumor core” (TC), and the
“whole tumor” (WT). The provided segmentation labels have values of 1 for NCR,
2 for ED, 4 for ET, and 0 for everything else. For this task this year’s BraTS
challenge makes available a dataset of 8,000 MRI scans from 2,000 glioma
patients. These cases are distributed across training, validation, and testing
datasets following a machine learning paradigm.

####  2.2.2 Task 2: Radiogenomic Classification

Participants are provided with mpMRI data and the MGMT promoter methylation
status associated with each case. The methylated cases are marked as ‘1’ and
unmethylated as ‘0’ in the csv file which is provided with the data.
Researchers have proposed methods to predict the MGMT promoter methylation
status with appropriate imaging/radiomic features extraction, and analyse them
through machine learning algorithms. The participants do not need to be
limited to volumetric parameters, but can also consider intensity,
morphologic, histogram-based, and textural features, as well as spatial
information, and glioma diffusion properties extracted from glioma growth
models. Participants will be evaluated for the predicted MGMT status of the
subjects indicated in the accompanying spreadsheet.

###  2.3 Performance Evaluation

Participants are called to submit the results on the online evaluation
platform for the training and validation dataset. The test dataset will never
be shared with the participants and they will upload their proposed methods in
a containerized way for the final testing phase. To evaluate the
generalizability of the proposed methods, we will evaluate the performance on
the cohort which is not part of either training or validation cohort, also
termed as testing out of distribution cohort. The distribution for methylated
and unmethylated cases across the training, validation, testing cohort is
given in Table 1.

####  2.3.1 Task 1: Tumor Sub-region Segmentation

Consistent with the configuration of previous BraTS challenges, we intend to
use the “Dice similarity coefficient”, and the “Hausdorff distance (95%)” as
performance evaluation metrics. Expanding upon this evaluation scheme, we will
also provide the metrics of “Sensitivity” and “Specificity”, allowing to
determine potential over- or under-segmentations of the tumor sub-regions by
participating methods.

The ranking scheme followed during the BraTS 2017-2020 comprised the ranking
of each team relative to its competitors for each of the testing subjects, for
each evaluated region (i.e., ET, TC, WT), and for each measure (i.e., Dice and
Hausdorff). For example, in BraTS 2020, each team was ranked for 166 subjects,
for 3 regions, and for 2 metrics, which resulted in $166\times 3\times 2=996$
individual rankings. The final ranking score (FRS) for each team was then
calculated by firstly averaging across all these individual rankings for each
patient (i.e., Cumulative Rank), and then averaging these cumulative ranks
across all patients for each participating team. This ranking scheme has also
been adopted in other challenges with satisfactory results, such as the
Ischemic Stroke Lesion Segmentation challenge555http://www.isles-
challenge.org/ [42].

We then conducted further permutation testing, to determine statistical
significance of the relative rankings between each pair of teams. This
permutation testing would reflect differences in performance that exceeded
those that might be expected by chance. Specifically, for each team we started
with a list of observed subject-level Cumulative Ranks, i.e., the actual
ranking described above. For each pair of teams, we repeatedly randomly
permuted (i.e., for 100,000 times) the Cumulative Ranks for each subject. For
each permutation, we calculated the difference in the FRS between this pair of
teams. The proportion of times the difference in FRS calculated using randomly
permuted data exceeded the observed difference in FRS (i.e., using the actual
data) indicated the statistical significance of their relative rankings as a
p-value. These values were reported in an upper triangular matrix providing
insights of statistically significant differences across each pair of
participated teams.

Top ranked methods in the validation phase will be invited at MICCAI 2021 for
presentation of their methods and results. The final top three ranked
participating teams according to their evaluation against the testing data,
will be invited at RSNA 2021 for presentation and to receive their monetary
awards.

####  2.3.2 Task 2: Radiogenomic Classification

The methods submitted by the participating teams for task 2 will be evaluated
based on the area under the ROC curve (AUC), accuracy, FScore (Beta) and
Matthew’s Correlation Coefficient of the classification of the MGMT status as
methylated and unmethylated. The AUC is a metric that measures the overall
discriminatory capacity of a model for all possible thresholds and allows for
comparing the performance of the entries by each participant, even though it
has no straightforward clinical meaning and does not guarantee the model is
calibrated. The AUC will be used as the reference metric to rank the
participants in the leaderboard of task 2.

###  2.4 Participation Timeline

The challenge will commence with the release of the training dataset, which
will consist of imaging data and the corresponding ground-truth labels.
Participants can start designing and training their methods using this
training dataset.

The validation data will then be released within three weeks after the
training data is released. This will allow participants to obtain preliminary
results in unseen data and also report these in their submitted short MICCAI
LNCS papers, in addition to their cross-validated results on the training
data. The ground truth of the validation data will not be provided to the
participants, but multiple submissions to the online evaluation platforms will
be allowed. The top-ranked participating teams in the validation phase will be
invited to prepare their slides for a short oral presentation of their method
during the BraTS challenge at MICCAI 2021.

Finally, all participants will be evaluated and ranked on the same unseen
testing data, which will not be made available to the participants, after
uploading their containerized method in the evaluation platforms. The final
top-ranked participating teams will be announced at the 2021 RSNA Annual
Meeting. The top-ranked participating teams of both the tasks will receive
monetary prizes of total value of $60,000, sponsored by Intel, RSNA, and
NeoSoma Inc.

##  3 Discussion

In this paper we presented the design of the $10^{th}$ BraTS challenge,
jointly organised by the RSNA, ASNR, and MICCAI societies, and offering what
can possibly be considered the largest curated multi-label annotated dataset
of mpMRI scans for a single disease. Members of the RSNA and ASNR communities
had graciously volunteered to refine tumor sub-region annotations for all
2,000 cases included in the BraTS 2021 challenge, until satisfactory quality
for releasing the data. Considering the size of this year’s challenge and also
its potential continuation after the announcement of this year’s winners, the
testing data will be kept hidden at all times and their performance evaluation
will be based on the challenge evaluation platforms of Sage Bionetworks
Synapse (Task 1) and Kaggle (Task 2), concluding in distributing to the top
ranked participants monetary awards of $60,000 collectively. We hope that the
well-labelled multi-institutional data of BraTS 2021 will provide an optimized
community benchmark and a common dataset to the research community focusing on
computational neuro-oncology, even beyond the specific BraTS 2021 tasks.

Although we designed the BraTS 2021 challenge with utmost care there are still
some limitations that need further consideration. Firstly, the tumor feature
segmentations of each case are refined by a single annotator with an iterative
process with a group of approvers, until approval from the latter, and hence
the potential inter-rater agreement can not be assessed. Secondly, since the
provided MGMT promoter methylation status was determined based on varying
methods across the multiple institutions that contributed data, and each
institute follows its own methodology (e.g., pyrosequencing vs quantitative
PCR) and thresholds, only a binary classification of the methylation status
was made available to the participants instead of a continuous value. Lastly,
we note that some of the MRI datas included in the challenge harbor more
abnormalities than just gliomas. Since the focus of the challenge was on
gliomas all other abnormalities (such as white matter hyperintensities that
are typically secondary to small vessel ischemic disease) were not considered
in the annotation process. This was made particularly apparent from previous
efforts that attempted to perform a multi-disease segmentation [41].

With this multi-disease segmentation in mind, one of the main future
directions for the BraTS challenge would be to expand beyond its current focus
on glial tumors towards general brain abnormalities. Furthermore, the
extension from solely pre-operative baseline scans to post-operative scans,
and the inclusion of an additional label for the resection cavity would be a
very interesting and clinically appealing direction, as it would speak
directly to the assessment of treatment response and disease progression. To
ensure robustness and generalizability of the computational algorithms, ample
patient data from multiple sites, capturing diverse patient populations are
desired. A major hindrance for accessing these datasets is data siloing due to
tedious bureaucratic process, data ownership concerns, and legal
considerations reflected in patient privacy regulations, such as the American
HIPAA [43] and the European GDPR[44]. In future, we aim at moving from the
current centralised data approach to a federated approach, which would enable
researchers to access potentially unprecedented size of data and hence design
more robust and generalizable algorithms [39, 45, 46, 47].

## Acknowledgments

Success of any challenge in the medical domain depends upon the quality of
well annotated multi-institutional datasets. We are grateful to all the data
contributors, annotators and approvers for their time and efforts.

## Funding

Research reported in this publication was partly supported by the National
Cancer Institute (NCI) Informatics Technology for Cancer Research (ITCR)
program and the National Institute of Neurological Disorders and Stroke
(NINDS) of the National Institutes of Health (NIH), under award numbers
NCI:U01CA242871, NCI:U24CA189523, NINDS:R01NS042645, Contract No.
HHSN261200800001E, Ruth L. Kirschstein Institutional National Research Service
Award number T32 EB001631. Research reported in this publication was also
partly supported by the RSNA Research & Education Foundation grant number
RR2011, and by the ASNR Foundation Grant in Artificial Intelligence (JDR).
Sage Bionetworks support of challenge organization and infrastructure was
supported by the NCI ITCR program under award number U24CA248265. The content
of this publication is solely the responsibility of the authors and does not
represent the official views of the NIH or of the RSNA R&E Foundation, or the
views or policies of the Department of Health and Human Services, nor does
mention of trade names, commercial products, or organizations imply
endorsement by the U.S. Government.

## References

  * [1] D. N. Louis, D. W. Ellison, D. J. Brat, K. Aldape, D. Capper, C. Hawkins, W. Paulus, A. Perry, G. Reifenberger, D. Figarella-Branger, et al., “cimpact-now: a practical summary of diagnostic points from round 1 updates,” Brain Pathology, vol. 29, no. 4, pp. 469–472, 2019.
  * [2] D. N. Louis, P. Wesseling, W. Paulus, C. Giannini, T. T. Batchelor, J. G. Cairncross, D. Capper, D. Figarella-Branger, M. B. Lopes, W. Wick, et al., “cimpact-now update 1: not otherwise specified (nos) and not elsewhere classified (nec),” Acta neuropathologica, vol. 135, no. 3, pp. 481–484, 2018.
  * [3] D. N. Louis, C. Giannini, D. Capper, W. Paulus, D. Figarella-Branger, M. B. Lopes, T. T. Batchelor, J. G. Cairncross, M. van den Bent, W. Wick, et al., “cimpact-now update 2: diagnostic clarifications for diffuse midline glioma, h3 k27m-mutant and diffuse astrocytoma/anaplastic astrocytoma, idh-mutant,” Acta neuropathologica, vol. 135, no. 4, pp. 639–642, 2018.
  * [4] D. J. Brat, K. Aldape, H. Colman, E. C. Holland, D. N. Louis, R. B. Jenkins, B. Kleinschmidt-DeMasters, A. Perry, G. Reifenberger, R. Stupp, et al., “cimpact-now update 3: recommended diagnostic criteria for “diffuse astrocytic glioma, idh-wildtype, with molecular features of glioblastoma, who grade iv”,” Acta neuropathologica, vol. 136, no. 5, pp. 805–810, 2018\.
  * [5] D. W. Ellison, C. Hawkins, D. T. Jones, A. Onar-Thomas, S. M. Pfister, G. Reifenberger, and D. N. Louis, “cimpact-now update 4: diffuse gliomas characterized by myb, mybl1, or fgfr1 alterations or braf v600e mutation,” Acta neuropathologica, vol. 137, no. 4, pp. 683–687, 2019.
  * [6] D. J. Brat, K. Aldape, H. Colman, D. Figrarella-Branger, G. N. Fuller, C. Giannini, E. C. Holland, R. B. Jenkins, B. Kleinschmidt-DeMasters, T. Komori, et al., “cimpact-now update 5: recommended grading criteria and terminologies for idh-mutant astrocytomas,” Acta neuropathologica, vol. 139, no. 3, pp. 603–608, 2020.
  * [7] D. N. Louis, P. Wesseling, K. Aldape, D. J. Brat, D. Capper, I. A. Cree, C. Eberhart, D. Figarella-Branger, M. Fouladi, G. N. Fuller, et al., “cimpact-now update 6: new entity and diagnostic principle recommendations of the cimpact-utrecht meeting on future cns tumor classification and grading,” 2020.
  * [8] S. Bakas, G. Shukla, H. Akbari, G. Erus, A. Sotiras, S. Rathore, C. Sako, S. Min Ha, M. Rozycki, R. T. Shinohara, M. Bilello, and C. Davatzikos, “Overall survival prediction in glioblastoma patients using structural magnetic resonance imaging (mri): advanced radiomic features may compensate for lack of advanced mri modalities,” Journal of medical imaging (Bellingham, Wash.), vol. 7, p. 031505, May 2020.
  * [9] D. N. Louis, A. Perry, G. Reifenberger, A. Von Deimling, D. Figarella-Branger, W. K. Cavenee, H. Ohgaki, O. D. Wiestler, P. Kleihues, and D. W. Ellison, “The 2016 world health organization classification of tumors of the central nervous system: a summary,” Acta neuropathologica, vol. 131, no. 6, pp. 803–820, 2016.
  * [10] A. L. Rivera, C. E. Pelloski, M. R. Gilbert, H. Colman, C. De La Cruz, E. P. Sulman, B. N. Bekele, and K. D. Aldape, “MGMT promoter methylation is predictive of response to radiotherapy and prognostic in the absence of adjuvant alkylating chemotherapy for glioblastoma,” Neuro-Oncology, vol. 12, pp. 116–121, 12 2009.
  * [11] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., “The multimodal brain tumor image segmentation benchmark (brats),” IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  * [12] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific data, vol. 4, no. 1, pp. 1–13, 2017.
  * [13] S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki, et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” arXiv preprint arXiv:1811.02629, 2018.
  * [14] S. Bakas, H. Akbari, A. Sotiras, et al., “Segmentation labels for the pre-operative scans of the tcga-gbm collection.,” The cancer imaging archive, 2017.
  * [15] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, J. Freymann, K. Farahani, and C. Davatzikos, “Segmentation labels and radiomic features for the pre-operative scans of the tcga-lgg collection,” The cancer imaging archive, vol. 286, 2017.
  * [16] E. Calabrese, J. Villanueva-Meyer, J. Rudie, A. Rauschecker, U. Baid, S. Bakas, J. Mongan, C. Hess, and S. Cha, “The university of california san francisco preoperative diffuse glioma (ucsf-pdgm) mri dataset,” 2021.
  * [17] R. Cox, J. Ashburner, H. Breman, K. Fissell, C. Haselgrove, C. Holmes, J. Lancaster, D. Rex, S. Smith, J. Woodward, et al., “A (sort of) new image data format standard: Nifti-1: We 150,” Neuroimage, vol. 22, 2004\.
  * [18] T. Rohlfing, N. M. Zahr, E. V. Sullivan, and A. Pfefferbaum, “The sri24 multichannel atlas of normal adult human brain structure,” Human brain mapping, vol. 31, no. 5, pp. 798–819, 2010.
  * [19] C. Davatzikos, S. Rathore, S. Bakas, S. Pati, M. Bergman, R. Kalarot, P. Sridharan, A. Gastounioti, N. Jahani, E. Cohen, et al., “Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome,” Journal of medical imaging, vol. 5, no. 1, p. 011018, 2018.
  * [20] S. Rathore, S. Bakas, S. Pati, H. Akbari, R. Kalarot, P. Sridharan, M. Rozycki, M. Bergman, B. Tunc, R. Verma, et al., “Brain cancer imaging phenomics toolkit (brain-captk): an interactive platform for quantitative analysis of glioblastoma,” in International MICCAI Brainlesion Workshop, pp. 133–145, Springer, 2017.
  * [21] S. Pati, A. Singh, S. Rathore, A. Gastounioti, M. Bergman, P. Ngo, S. M. Ha, D. Bounias, J. Minock, G. Murphy, et al., “The cancer imaging phenomics toolkit (captk): Technical overview,” in International MICCAI Brainlesion Workshop, pp. 380–394, Springer, 2019.
  * [22] C. G. Schwarz, W. K. Kremers, T. M. Therneau, R. R. Sharp, J. L. Gunter, P. Vemuri, A. Arani, A. J. Spychalla, K. Kantarci, D. S. Knopman, R. C. Petersen, and C. R. Jack, “Identification of anonymous mri research participants with face-recognition software,” New England Journal of Medicine, vol. 381, no. 17, pp. 1684–1686, 2019.  PMID: 31644852.
  * [23] C. G. Schwarz, W. K. Kremers, T. M. Therneau, R. R. Sharp, J. L. Gunter, P. Vemuri, A. Arani, A. J. Spychalla, K. Kantarci, D. S. Knopman, R. C. Petersen, and C. R. Jack, “Identification from mri with face-recognition software,” New England Journal of Medicine, vol. 382, no. 5, pp. 489–490, 2020.  PMID: 31995706.
  * [24] S. Thakur, J. Doshi, S. Pati, S. Rathore, C. Sako, M. Bilello, S. M. Ha, G. Shukla, A. Flanders, A. Kotrotsou, et al., “Brain extraction on mri scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training,” NeuroImage, vol. 220, p. 117081, 2020.
  * [25] S. K. Warfield, K. H. Zou, and W. M. Wells, “Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation,” IEEE transactions on medical imaging, vol. 23, no. 7, pp. 903–921, 2004.
  * [26] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,” Nature Methods, pp. 1–9, 2020.
  * [27] R. McKinley, R. Meier, and R. Wiest, “Ensembles of densely-connected cnns with label-uncertainty for brain tumor segmentation,” in International MICCAI Brainlesion Workshop, pp. 456–465, Springer, 2018.
  * [28] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker, “Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation,” Medical image analysis, vol. 36, pp. 61–78, 2017.
  * [29] M. McCormick, X. Liu, L. Ibanez, J. Jomier, and C. Marion, “Itk: enabling reproducible research and open science,” Frontiers in Neuroinformatics, vol. 8, p. 13, 2014.
  * [30] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, et al., “The cancer imaging archive (tcia): maintaining and operating a public information repository,” Journal of digital imaging, vol. 26, no. 6, pp. 1045–1057, 2013.
  * [31] L. Scarpace, L. Mikkelsen, T. Cha, S. Rao, S. Tekchandani, S. Gutman, and D. Pierce, “Radiology data from the cancer genome atlas glioblastoma multiforme [tcga-gbm] collection,” The Cancer Imaging Archive, vol. 11, no. 4, p. 1, 2016.
  * [32] N. Pedano, A. E. Flanders, L. Scarpace, T. Mikkelsen, J. M. Eschbacher, B. Hermes, and Q. Ostrom, “Radiology data from the cancer genome atlas low grade glioma [tcga-lgg] collection,” Cancer Imaging Arch, vol. 2, 2016\.
  * [33] R. B. Puchalski, N. Shah, J. Miller, R. Dalley, S. R. Nomura, J.-G. Yoon, K. A. Smith, M. Lankerovich, D. Bertagnolli, K. Bickley, et al., “An anatomic transcriptional atlas of human glioblastoma,” Science, vol. 360, no. 6389, pp. 660–663, 2018.
  * [34] N. Shah, X. Feng, M. Lankerovich, R. B. Puchalski, and B. Keogh, “Data from ivy gap,” The Cancer Imaging Archive, vol. 10, p. K9, 2016.
  * [35] TCIA, “Radiology data from the clinical proteomic tumor analysis consortium glioblastoma multiforme [cptac-gbm] collection,” The Cancer Imaging Archive, vol. 10, p. K9, 2016.
  * [36] L.-B. Wang, A. Karpova, M. A. Gritsenko, J. E. Kyle, S. Cao, Y. Li, D. Rykunov, A. Colaprico, J. H. Rothstein, R. Hong, et al., “Proteogenomic and metabolomic characterization of human glioblastoma,” Cancer cell, vol. 39, no. 4, pp. 509–528, 2021.
  * [37] P. Kinahan, M. Muzi, B. Bialecki, and L. Coombs, “Data from acrin-fmiso-brain,” 2018.
  * [38] E. R. Gerstner, Z. Zhang, J. R. Fink, M. Muzi, L. Hanna, E. Greco, M. Prah, K. M. Schmainda, A. Mintz, L. Kostakoglu, et al., “Acrin 6684: assessment of tumor hypoxia in newly diagnosed glioblastoma using 18f-fmiso pet and mri,” Clinical Cancer Research, vol. 22, no. 20, pp. 5079–5086, 2016.
  * [39] M. J. Sheller, B. Edwards, G. A. Reina, J. Martin, S. Pati, A. Kotrotsou, M. Milchenko, W. Xu, D. Marcus, R. R. Colen, et al., “Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data,” Scientific reports, vol. 10, no. 1, pp. 1–12, 2020\.
  * [40] P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho, J. C. Gee, and G. Gerig, “User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability,” Neuroimage, vol. 31, no. 3, pp. 1116–1128, 2006.
  * [41] J. D. Rudie, D. A. Weiss, R. Saluja, A. M. Rauschecker, J. Wang, L. Sugrue, S. Bakas, and J. B. Colby, “Multi-disease segmentation of gliomas and white matter hyperintensities in the brats data using a 3d convolutional neural network,” Frontiers in Computational Neuroscience, vol. 13, p. 84, 2019\.
  * [42] O. Maier, B. H. Menze, J. von der Gablentz, L. Häni, M. P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen, et al., “Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri,” Medical image analysis, vol. 35, pp. 250–269, 2017.
  * [43] G. J. Annas, “Hipaa regulations — a new era of medical-record privacy?,” New England Journal of Medicine, vol. 348, no. 15, pp. 1486–1490, 2003\.  PMID: 12686707.
  * [44] P. Voigt and A. Von dem Bussche, “The eu general data protection regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, p. 3152676, 2017.
  * [45] N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B. A. Landman, K. Maier-Hein, et al., “The future of digital health with federated learning,” NPJ digital medicine, vol. 3, no. 1, pp. 1–7, 2020.
  * [46] S. Pati, U. Baid, M. Zenk, B. Edwards, M. Sheller, G. A. Reina, P. Foley, A. Gruzdev, J. Martin, S. Albarqouni, et al., “The federated tumor segmentation (fets) challenge,” arXiv preprint arXiv:2105.05874, 2021.
  * [47] S. Pati, S. P. Thakur, M. Bhalerao, U. Baid, C. Grenko, B. Edwards, M. Sheller, J. Agraz, B. Baheti, V. Bashyam, et al., “Gandlf: A generally nuanced deep learning framework for scalable end-to-end clinical workflows in medical imaging,” arXiv preprint arXiv:2103.01006, 2021.