YYYY:NNN Fathi Kazerooni, et al. 1 2024 m1/yyyy m2/yyyy Medical Imaging with
Deep Learning (MIDL) 2020 Marleen de Bruijne, Tal Arbel, Ismail Ben Ayed,
Hervé Lombaert BraTS-PEDs Results 2023Fathi Kazerooni, et al. 1 Center for
Data-Driven Discovery in Biomedicine (D3b), The Children’s Hospital of
Philadelphia, PA, USA, 2 Sheikh Zayed Institute for Pediatric Surgical
Innovation, Children’s National Hospital, Washington DC, USA, 3 Department of
Neurosurgery, Thomas Jefferson University Hospital, PA, USA, 4 Dana-Farber
Brigham Cancer Center & Boston Children’s Hospital, Boston, MA, USA, 5
Department of Paediatric Radiology, Royal Manchester Children’s Hospital,
Manchester University Hospitals NHS Foundation Trust, University of
Manchester, Manchester, UK, 6 University of Utah, UT, USA, 7 Cincinnati
Children’s Hospital Medical Center, OH, USA, 8 Department of Radiology, The
Children’s Hospital of Philadelphia, PA, USA, 9 Division of Neuroradiology,
Department of Radiology, Boston Children’s Hospital, Harvard Medical School,
Boston, MA, USA, 10 Department of Radiology, Nationwide Children’s Hospital,
OH, USA, 11 College of Arts and Sciences, University of Pennsylvania, PA, USA,
12 Department of Diagnostic and Interventional Radiology, All India Institute
of Medical Sciences, Rishikesh, India, 13 Department of Diagnostic &
Interventional Imaging, Neuroradiology section, The University of Texas Health
at Houston, Texas, USA, 14 Beth Israel Deaconess Medical Center, Harvard
Medical School, MA, USA, 15 UNIDAD DE PATOLOGIA CLINICA:Guadalajara, JALISCO,
Mexico, 16 Department of Radiology, Children’s Health Orange County, CA, USA,
17 Medical Artificial Intelligence (MAI) Lab, Crestview Radiology, Lagos,
Nigeria, 18 Sage Bionetworks, USA, 19 Montreal Neurological Institute (MNI),
McGill University, Montreal, QC, Canada, 20 Department of Radiology, Perelman
School of Medicine, University of Pennsylvania, Philadelphia, PA, USA, 21
MLCommons, USA, 22 Division of Computational Pathology, Department of
Pathology and Laboratory Medicine, Indiana University School of Medicine,
Indianapolis, IN, USA, 23 Department of Neurosurgery at the University of
Southern California, CA, USA, 24 Department of Radiology, Duke University
Medical Center, Durham, NC, USA, 25 Mayo Clinic, MN, USA, 26 Department of
Informatics, Technical University Munich, Germany, 27 National Cancer
Institute, National Institutes of Health, Bethesda, MD, USA, 28 Biomedical
Engineering Rutgers University, New Brunswick, NJ, USA, 29 Athinoula A
Martinos Center for Biomedical Imaging, Massachusetts General Hospital and
Harvard Medical School, Boston, MA, USA, 30 Yale University, New Haven, CT,
USA, 31 PrecisionFDA, U.S. Food and Drug Administration, Silver Spring, MD,
USA, 32 Helmholtz AI, Helmholtz Munich, Germany, 33 Department of Radiation
Oncology, Duke University Medical Center, Durham, NC, USA, 34 Department of
Applied Mathematics and Computer Science, Technical University of Denmark,
Denmark, 35 Athinoula A Martinos Center for Biomedical Imaging, Massachusetts
General Hospital, Boston, MA, USA, 36 Booz Allen Hamilton, McLean, VA, USA, 37
Biomedical Image Analysis & Machine Learning, Department of Quantitative
Biomedicine, University of Zurich, Switzerland, 38 Mercy Catholic Medical
Center, Darby, PA, USA, 39 University of San Diego, CA, USA, 40 Cornell
University, Ithaca, NY, USA, 41 Statistics in Imaging and Visualization
Center, Department of Biostatistics, Epidemiology, and Informatics, University
of Pennsylvania, Philadelphia, PA, USA, 42 Department of Neuroradiology,
Technical University of Munich, Munich, Germany, 43 Greensboro Radiology,
Greensboro, NC, USA, 44 Center for AI and Data Science for Integrated
Diagnostics (AI2D) & Center for Biomedical Image Computing and Analytics
(CBICA), University of Pennsylvania, Philadelphia, PA, USA, 45 Brain Tumor
Institute, Children’s National Hospital, Washington, DC, USA, 46 Brain Tumor
Center, Cincinnati Children’s Hospital, Cincinnati, OH, USA, 47 Center for
Cancer and Blood Disorders, Phoenix Children’s Hospital, Phoenix, AZ, US, 48
Center for Cancer and Blood Disorders, Children’s National Hospital,
Washington, DC, USA

# BraTS-PEDs: Results of the Multi-Consortium International Pediatric Brain
Tumor Segmentation Challenge 2023

Anahita Fathi Kazerooni1  Nastaran Khalili1  Xinyang Liu2  Debanjan Haldar3
Zhifan Jiang2  Anna Zapaishchykova4  Julija Pavaine5  Lubdha M. Shah6  Blaise
V. Jones7  Nakul Sheth8  Sanjay P. Prabhu9  Aaron S. McAllister10  Wenxin Tu11
Khanak K. Nandolia12  Andres F. Rodriguez13  Ibraheem Salman Shaikh14  Mariana
Sanchez-Montano15  Hollie Anne Lai16  Maruf Adewole17  Jake Albrecht18  Udunna
Anazodo19  Hannah Anderson20  Syed Muhammed Anwar2  Alejandro Aristizabal21
Sina Bagheri20  Ujjwal Baid22  Timothy Bergquist18  Austin J. Borja23  Evan
Calabrese24  Verena Chung18  Gian-Marco Conte25  James Eddy18  Ivan Ezhov26
Ariana M. Familiar1  Keyvan Farahani27  Deep Gandhi1  Anurag Gottipati1
Shuvanjan Haldar28  Juan Eugenio Iglesias29  Anastasia Janas30  Elaine
Elaine31  Alexandros Karargyris21  Hasan Kassem21  Neda Khalili1  Florian
Kofler32  Dominic LaBella33  Koen Van Leemput34  Hongwei B. Li35  Nazanin
Maleki30  Zeke Meier36  Bjoern Menze37  Ahmed W. Moawad38  Sarthak Pati21
Marie Piraud32  Tina Poussaint4  Zachary J. Reitman33  Jeffrey D. Rudie39
Rachit Saluja40  MIcah Sheller21  Russell Takeshi Shinohara41  Karthik
Viswanathan1  Chunhao Wang33  Benedikt Wiestler42  Walter F. Wiggins43
Christos Davatzikos44  Phillip B. Storm1  Miriam Bornhorst45  Roger Packer45
Trent Hummel46  Peter de Blank46  Lindsey Hoffman47  Mariam Aboian8  Ali
Nabavizadeh1  Jeffrey B. Ware1  Benjamin H. Kann4  Brian Rood48  Adam Resnick1
Spyridon Bakas22  Arastoo Vossough8  Marius George Linguraru2

###### Abstract

Pediatric central nervous system tumors are the leading cause of cancer-
related deaths in children. The five-year survival rate for high-grade glioma
in children is less than 20%. The development of new treatments is dependent
upon multi-institutional collaborative clinical trials requiring reproducible
and accurate centralized response assessment. We present the results of the
BraTS-PEDs 2023 challenge, the first Brain Tumor Segmentation (BraTS)
challenge focused on pediatric brain tumors. This challenge utilized data
acquired from multiple international consortia dedicated to pediatric neuro-
oncology and clinical trials. BraTS-PEDs 2023 aimed to evaluate volumetric
segmentation algorithms for pediatric brain gliomas from magnetic resonance
imaging using standardized quantitative performance evaluation metrics
employed across the BraTS 2023 challenges. The top-performing AI approaches
for pediatric tumor analysis included ensembles of nnU-Net and Swin UNETR,
Auto3DSeg, or nnU-Net with a self-supervised framework. The BraTS-PEDs 2023
challenge fostered collaboration between clinicians (neuro-oncologists,
neuroradiologists) and AI/imaging scientists, promoting faster data sharing
and the development of automated volumetric analysis techniques. These
advancements could significantly benefit clinical trials and improve the care
of children with brain tumors.

###### doi:

https://doi.org/10.59275/j.melba.2024-AAAA

###### keywords:

Brain Tumor Segmentation, Pediatric Brain Tumors, BraTS-PEDs, Benchmarking

††volume:

##  1 Introduction

Although rare, pediatric tumors of the central nervous system are the leading
cause of cancer-related death in children. While pediatric brain tumors may
share certain similarities with adult brain tumors, their imaging and clinical
presentations differ significantly. For example, both adult glioblastoma (GBM)
and pediatric diffuse midline glioma (DMG), including diffuse intrinsic
pontine glioma (DIPG), are high-grade glial tumors with short average overall
survival (Mackay et al., 2017; Jansen et al., 2015). The incidence of adult
GBM is 3 in 100,000 people, while DMG is approximately three times rarer.

GBMs are most commonly found in the frontal and/or temporal lobes and
diagnosed at an average age of 64 years (Thakkar et al., 2014). In contrast,
DMG is primarily located in the pons and is most often diagnosed in children
between 5 and 10 years of age. Enhancing tumor regions on post-gadolinium
T1-weighted magnetic resonance imaging (MRI) and radiologically defined
necrotic tissue regions are common imaging findings in GBM but are less common
or clear in DMG, particularly at initial diagnosis. Therefore, pediatric brain
tumors require dedicated imaging tools to aid in their characterization and
improve diagnosis and prognosis (Familiar et al., 2024).

Pediatric brain tumors vary in aggressiveness and prognosis, presenting with
heterogeneous histologic subregions such as peritumoral edematous/infiltrated
tissue, necrotic core, and enhancing tumor margin, which are reflected in
their radio-phenotypes on multi-parametric MRI (mpMRI) scans (Fathi Kazerooni
et al., 2023a). Some pediatric brain tumors, such as DIPG, are located in
surgically inaccessible areas, making them unresectable. Consequently, tumor
progression assessment primarily relies on changes in tumor size measured from
longitudinal MRI scans.

The current standard, recommended by the Response Assessment in Pediatric
Neuro-Oncology (RAPNO) cooperative working group (Erker et al., 2020; Cooney
et al., 2020), is based on two-dimensional (2D) linear measurements of the
tumor subregions, including enhancing tumor, non-enhancing tumor, cystic
components, and peritumoral edematous tissue (Familiar et al., 2024), in the
axial slice of the largest tumor extent. However, manual measurement has high
inter-operator variability and 2D measurements are inaccurate surrogates for
tumor volumes as they do not account for the irregularity of tumor shape and
appearance. Studies in adult brain tumors have demonstrated the superiority of
three-dimensional (3D) volumetric measurement of tumor extent in predicting
clinical endpoints over 2D methods (Elingson et al., 2022).

Tumor segmentation for volumetric analysis is crucial not only for surgical
and treatment planning but also for assessing treatment response and
longitudinal monitoring. The importance of volumetric tumor measurements, and
thus automated tumor segmentation, has been increasingly recognized in
pediatric brain tumor assessment (Jansen et al., 2015; Lazow et al., 2022).
However, there is a lack of available automated tools to facilitate the
segmentation of tumor subregions on mpMRI scans. A few notable automated tumor
segmentation methods have been proposed explicitly for pediatric brain tumors
(Fathi Kazerooni et al., 2023a; Artzi et al., 2020; Kharaji et al., 2024; Liu
et al., 2023; Mansoor et al., 2016; Tor-Diez et al., 2020; Peng et al., 2022;
Nalepa et al., 2022; Vossough et al., 2022). Most of these methods focus on
the segmentation of the entirety of abnormal signal on T2 fluid attenuated
inversion recovery (FLAIR) images (Artzi et al., 2020; Nalepa et al., 2022),
referred to as the whole tumor (WT) in BraTS terminology (Bakas et al., 2018).
However, due to the absence of benchmarking platforms, these proposed methods
have not been evaluated and compared using the same validation data or through
a controlled study design.

The BraTS challenges, organized in conjunction with the Medical Image
Computing and Computer Assisted Intervention (MICCAI, https://miccai.org)
conferences since 2012, have established a community benchmark dataset and
environment for adult brain glioma (Bakas et al., 2018; Menze et al., 2014;
Bakas et al., 2017; Baid et al., 2021). In 2023, the BraTS challenge expanded
to include a variety of tumor entities such as adult glioma, intracranial
meningioma (LaBella et al., 2023), brain metastases (Moawad et al., 2023),
Sub-Saharan African glioma, and pediatric tumors (Kazerooni et al., 2023b) for
the segmentation task. Additionally, new tasks were introduced, including
Synthesis (Global) - Missing MRI (Li et al., 2023), Synthesis (Local) –
Inpainting (Kofler et al., 2023), and Augmentation. The formation of the
pediatric challenge (BraTS-PEDs) followed the initiative at BraTS 2022, where
a small set of 60 pediatric DMGs were included only in the test phase of the
challenge. The findings from this evaluation encouraged the organization of a
larger and more diverse initiative in 2023, incorporating multi-consortia
pediatric data for training, validation, and testing of algorithms.

This manuscript presents the data, design, and outcomes of the BraTS-PEDs 2023
challenge, the first benchmarking initiative focused on pediatric brain tumor
segmentation. Through a comprehensive analysis of the performance of
participating algorithms, this manuscript aims to establish a standard
framework for evaluating pediatric brain tumor segmentation algorithms. The
goal is to facilitate clinical translation and extend the application of these
algorithms to response assessment in pediatric brain tumors.

##  2 Data and Methods

###  2.1 Cohort Description

The BraTS-PEDs 2023 dataset was supported by multiple consortia, including
Children’s Brain Tumor Network (CBTN, https://cbtn.org/) (Lilly et al., 2023),
International DIPG/DMG Registry (https://www.dipgregistry.org) and
COllaborative Network for NEuro-oncology Clinical Trials (CONNECT,
https://connectconsortium.org/), along with additional clinical centers,
including Boston’s Children Hospital and Yale University. All cases contained
pre- and post-gadolinium T1-weighted (labeled as T1 and T1CE, respectively),
T2-weighted (T2), and T2-weighted fluid attenuated inversion recovery
(T2-FLAIR) images. These conventional mpMRI sequences are commonly acquired as
part of standard clinical care for brain tumors. However, the image
acquisition protocols, and MRI equipment differ across different institutions,
resulting in heterogeneity in image quality and appearance in the multi-
consortium cohort. Inclusion criteria comprised of pediatric subjects with:
(1) histologically-approved high-grade glioma, i.e., astrocytoma and DMG,
including radiologically or histologically-proven DIPG; (2) availability of
all four structural mpMRI sequences on treatment-naive imaging sessions.
Exclusion criteria consisted of: (1) images of low quality or with artifacts
that would not allow for reliable tumor segmentation; and (2) infants younger
than one month of age. In the 2023 BraTS-PEDs, data for 167 patients was
included (CBTN, n = 113; Boston’s Children Hospital, n = 30; Yale University,
n = 24). Data from the International DIPG/DMB Registry was only processed but
not segmented for the 2023 challenge, but will be included in the 2024
challenge.

###  2.2 Imaging Data

For all patients, mpMRI scans were pre-processed using a standardized
approach, including conversion of the DICOM files to the NIfTI file format,
co-registration to the same anatomical template (i.e., SRI24) (Rohlfing et
al., 2010), and resampling to an isotropic resolution of 1 mm3. The pre-
processing pipeline is publicly available through the Cancer Imaging Phenomics
Toolkit (CaPTk) (Pati et al., 2020; Rathore et al., 2018; Davatzikos et al.,
2018) and Federated Tumor Segmentation (FeTS) tool (Pati et al., 2022)
(https://github.com/FETS-AI/Front-End/). The de-identification was performed
through removing protected health information from DICOM headers. De-facing
was performed via skull-stripping using a pediatric-specific skull-stripping
method (Fathi Kazerooni et al., 2023a) to prevent any potential facial
reconstruction/recognition of the patients. Figure 1A presents the data
preparation workflow.

The pre-processed images were segmented into tumor subregions using either of
two pediatric automated deep learning segmentation models (Fathi Kazerooni et
al., 2023a; Liu et al., 2023) before being manually corrected by pediatric
neuro-radiologists. The tumors were segmented into four subregions. These
subregions comprising of enhancing tumor (ET), non-enhancing tumor (NET),
cystic component (CC), and peritumoral edema (ED) regions, with definitions
provided in (Familiar et al., 2024), and summarized below:

  * •

ET is described by areas with enhancement (brightness) on T1 post-contrast
images as compared to T1 pre-contrast. In case of mild enhancement, checking
the signal intensity of normal brain structures can be helpful.

  * •

CC typically appears with hyperintense signal (very bright) on T2 and
hypointense signal (dark) on T1CE. The cystic portion should be within the
tumor, either centrally or peripherally (as compared to ED which is
peritumoral). The brightness of CC is defined as comparable or close to
cerebrospinal fluid (CSF).

  * •

NET is defined as any other abnormal signal intensity within the tumorous
region that cannot be defined as enhancing or cystic. For example, the
abnormal signal intensity on T1, T2-FLAIR, and T2 that is not enhancing on
T1CE should be considered as non-enhancing portion.

  * •

ED is defined by the abnormal hyperintense signal (very bright) on FLAIR
scans. ED is finger-like spreading that preserves underlying brain structure
and surrounds the tumor.

The American Society of Neuroradiology (www.asnr.org, ASNR) collaborated in
generating ground truth annotation for the majority of data in this challenge.
Four labels were generated (Figure 1) using the preliminary automated
segmentation and were then manually revised by ASNR volunteer neuroradiology
experts of varying rank and experience in accordance with the annotation
guidelines. The expert annotators were provided with the four mpMRI sequences
(T1, T1CE, T2, FLAIR) along with the fused automated segmentation volume to
initiate the manual refinements using the ITK-SNAP software (Yushkevich et
al., 2006). After segmentation corrections by the annotators, three attending
board-certified neuroradiologists reviewed the segmentations to either approve
or return to the individual annotators for further refinements. This process
was followed iteratively until the approvers found the refined tumor subregion
segmentations acceptable for public release and the challenge conduction. The
final segmentations were provided and used as ground truth for model training
and evaluation.

Figure 1: A schematic diagram of different steps in data preparation for
BraTS-PEDs 2023 data: (A) Data preparation process; (B) Tumor segmentation
approach: Left - original annotations including four subregions, Right – final
annotations provided to the teams for model training (the final annotations
and their label values were prepared in agreement with other BraTS 2023
challenges). The color maps in the images shown in Figure 1B illustrate the
label descriptions provided in the original annotations given to the
annotators (left), as compared to the final annotations (right) – which adhere
to the standards of other BraTS challenges – provided to the teams; (C)
Evaluation labels: for calculating the performance metrics, ET, TC
(combination of ET and NC), and WT (a combination of ET, NC, ED) were used.

Although four tumor subregions were annotated for all the pediatric data, to
follow the guidelines of the other BraTS 2023 challenges and to allow the
teams to potentially incorporate data from adult glioma challenge into their
model training, the teams were provided with only three segmentation labels,
i.e., non-enhancing component (NC – the union of non-enhancing tumor, cystic
component, and necrosis; label 1), ED (label 2), and ET (label 3), as shown in
Figure 1B. The color maps in the images shown in Figure 1B illustrate the
label descriptions provided in the original annotations given to the
annotators (Figure 1B; left), as compared to the final annotations (Figure 1B;
right) – which adhere to the standards of other BraTS challenges – provided to
the teams.

###  2.3 Challenge Setup

The BraTS-PEDs 2023 cohort was split into training (n = 98), validation (n =
44), and testing datasets (n = 24). Training and validation included a random
split between CBTN and Boston cases. All Yale cases were used for independent
testing. The training data shared with the teams comprised mpMRI scans and
ground truth labels for the training cohort, released in mid-May 2023. The
cohort shared for validation included only the mpMRI sequences without
associated ground truth and it was released in late June 2023. Notably, the
testing cohort was kept “unseen” to participants.

Teams were prohibited from training their algorithm on any additional public
and/or private data besides the provided BraTS-PEDs 2023 data or to using
models pretrained on other datasets. This restriction was imposed to allow for
a direct and fair comparison among the participating methods. However, teams
were allowed to use additional public and/or private data for publication of
their scientific papers, on the condition that they also provide a report of
their results on the data from BraTS-PEDs 2023 challenge alone and discuss
potential differences in the obtained results.

The methods submitted to the BraTS-PEDs 2023 challenge were evaluated based on
performances for the segmentation of ET, tumor core (TC – the combination of
ET and NC), and the entire tumorous region, or whole tumor (WT – the union of
ET, NC, and ED) (Figure 1C). The containerized submissions of the teams were
evaluated on the synapse.org platform supported by Medperf (Karargyris et al.,
2023). Teams were required to send the output of their methods to the
evaluation platform for the scoring to occur during the training and the
validation phases. At the end of the validation phase, teams were asked to
identify the method they would like to be evaluated by the organizers in the
final testing/ranking phase. The training, validation, and testing phases ran
between May-July 2023, July-August 2023, and August 2023, respectively. The
results and ranking of the challenge were presented at the MICCAI 2023
conference on October 10, 2023.

###  2.4 Benchmark Design: Evaluation and Ranking

The evaluation and ranking were based on a lesionwise ranking scheme in which
the ranking for each team was computed relative to other challenge competitors
for each of the subjects in the test set, for each tumor region, e.g., ET, TC,
and WT. We used two established performance evaluation metrics, i.e., Dice
similarity coefficient (Dice) and 95% Hausdorff distance (HD95) using the
Generally Nuanced Deep Learning Framework (GaNDLF; gandlf.org) (Pati et al.,
2023). At BraTS-PEDs 2023, each team was ranked for 24 test subjects, 3 tumor
regions, and 2 metrics, resulting in 144 individual rankings. Then for each
team, the final ranking score (FRS) was computed by producing a cumulative
rank as an average of all individual rankings for each patient, followed by
averaging the cumulative ranks across all testing subjects for each competing
team.

The lesion-wise Dice and HD95 were calculated by isolating and identifying
each disjoint lesion, through dilating the ground truth segmentation masks for
each tumor region, i.e., ET, TC, and WT, by 3 pixels in all directions (i.e.,
by a size of 3x3x3). Dice and HD95 scores were calculated for each component
or lesion individually, after penalizing all false positive and false negative
values with a score of 0. Subsequently, the mean value of Dice and HD95 scores
across all lesions were calculated for each subject. The lesions below a
cutoff value of 50 voxels were removed from any further evaluation. If the
models were not able to predict a region for a subject, the Dice score was
assigned a value of 0 and HD95 was set to 374 (the distance between corners in
the SRI atlas space), and if they correctly did not predict a non-existing
region, the Dice score and HD95 were given values of 1 and 0, respectively.
Lesion-wise Dice and HD95 scores for each case were calculated according to
equations 1 and 2. More information on the evaluation metrics along with the
codes can be found at https://github.com/rachitsaluja/BraTS-2023-Metrics.

$LesionwiseDiceScore=\frac{\Sigma_{i}^{L}Dice(l_{i})}{TP+FN+FP}$ (1)
$LesionwiseHD95=\frac{\Sigma_{i}^{L}HD95(l_{i})}{TP+FN+FP}$ (2)

Where L represents the number of ground truth lesions, TP denotes the number
of true positives, FN indicates the number of false negatives, and FP
signifies the number of false positives.

Finally, the statistical differences in performance, beyond those expected by
chance, between each pair of teams were determined using permutation testing.
This was achieved for each pair of teams, by randomly permuting the cumulative
ranks for each subject and calculating the difference in FRS between the pair
of teams in each permutation. Statistical significance of rankings was
determined using a p-value when the proportion of times the difference in FRS
using the permutated data exceeded the observed difference in FRS using the
actual data. These values are reported in an upper triangular matrix.

###  2.5 Data Availability

The training data, including ground truth segmentations, and the validation
data without ground truth segmentations can be accessed and downloaded from
the Synapse www.synapse.org/Synapse:syn51156910/wiki/627000. More information
about the BraTS-PEDs 2023 challenge is at
www.synapse.org/Synapse:syn51156910/wiki/622461.

##  3 Results

Table 1: Cumulative ranks across testing subjects and final ranking score
(FRS) for the competing teams in the BraTS-PEDs2023 challenge. The lower ranks
show better performance.

Teams | Method | Cumulative Ranks Across Subjects | FRS | Rank in the Challenge
---|---|---|---|---
CNMCPMI2023 | Ensemble of nnU-Net and Swin UNETR | 241.5 | 10.06 | \- 111Participating teams that included challenge organizers underwent evaluation without ranking to uphold fairness in our benchmarking initiative. This approach aligns with MICCAI guidelines, as well as the CBTN policy for hackathons.
NVAUTO | Auto3DSeg | 246 | 10.25 | 1
SherlockZyb | nnU-Net (self-supervised) | 286.5 | 11.93 | 2
Blackbean | Scalable and Transferable U-Net (STU-Net) | 290.5 | 12.1 | 3
BiomedMBZ | Ensemble of SegResnet and MedNext | 318 | 13.25 | 4
SchlaugLab | Ensemble of ONet and UNet | 395.5 | 16.47 | 5
VisionLabODU23 | Multiresolution Fractal Deep Neural Network (MFDNN) | 427 | 17.79 | 6
Isahajmistry | Modified nnU-Net | 483 | 20.12 | 7
UMNiverse | Temporal Cubic PatchGAN (TCuP-GAN) | 552 | 23 | 8

Table 2: Summary of the performance evaluation metrics for segmentation of enhancing tumor (ET) region for the participating teams. The results are shown in mean $\pm$ standard deviation (median) format. Teams | Dice Similarity Coefficient | 95% Hausdorff Distance | Sensitivity
---|---|---|---
CNMCPMI2023 | 0.65 ± 0.32 (0.74) | 43.89 ± 108.59 (3.67) | 0.70 ± 0.18 (0.74)
NVAUTO | 0.55 ± 0.37 (0.68) | 115.32 ± 144.03 (4.04) | 0.77 ± 0.20 (0.80)
SherlockZyb | 0.63 ± 0.32 (0.71) | 59.68 ± 116.70 (3.93) | 0.71 ± 0.27 (0.74)
Blackbean | 0.53 ± 0.34 (0.60) | 22.02 ± 52.12 (5.83) | 0.63 ± 0.31 (0.71)
BiomedMBZ | 0.55 ± 0.27 (0.61) | 45.32 ± 109.10 (5.69) | 0.80 ± 0.29 (0.92)
SchlaugLab | 0.55 ± 0.36 (0.65) | 65.23 ± 122.23 (4.30) | 0.55 ± 0.37 (0.59)
VisionLabODU23 | 0.17 ± 0.38 (0) | 311.67 ± 142.38 (374) | 0.17 ± 0.38 (0)
Isahajmistry | 0.48 ± 0.36 (0.49) | 128.12 ± 147.11 (10.85) | 0.66 ± 0.35 (0.82)
UMNiverse | 0.17 ± 0.38 (0) | 311.67 ± 142.38 (374) | 0.17 ± 0.38 (0)
Table 3: Summary of the performance evaluation metrics for segmentation of tumor core (TC) region for the participating teams. The results are shown in mean $\pm$ standard deviation (median) format. Teams | Dice Similarity Coefficient | 95% Hausdorff Distance | Sensitivity
---|---|---|---
CNMCPMI2023 | 0.81 ± 0.18 (0.85) | 21.82 ± 75.15 (5.26) | 0.76 ± 0.18 (0.81)
NVAUTO | 0.78 ± 0.19 (0.85) | 27.10 ± 72.11 (5.08) | 0.74 ± 0.14 (0.77)
SherlockZyb | 0.77 ± 0.21 (0.83) | 22.28 ± 75.08 (5.70) | 0.68 ± 0.21 (0.72)
Blackbean | 0.79 ± 0.14 (0.83) | 22.02 ± 52.15 (5.83) | 0.75 ± 0.11 (0.79)
BiomedMBZ | 0.77 ± 0.20 (0.82) | 30.36 ± 82.47 (7.21) | 0.73 ± 0.19 (0.92)
SchlaugLab | 0.70 ± 0.20 (0.75) | 31.61 ± 82.20 (7.00) | 0.61 ± 0.20 (0.65)
VisionLabODU23 | 0.71 ± 0.25 (0.81) | 38.40 ± 71.01 (6.5) | 0.70 ± 0.23 (0.77)
Isahajmistry | 0.32 ± 0.29 (0.30) | 130.25 ± 135.52 (41.59) | 0.28 ± 0.25 (0.21)
UMNiverse | 0.25 ± 0.22 (0.19) | 55.15 ± 107.21 (16.35) | 0.19 ± 0.16 (0.14)

Of the participating teams in the training phase of the challenge, 18 teams
with 159 submissions participated in the validation phase, with 147
submissions being scored. Nine of these teams completed all stages of the
challenge and were evaluated during the testing phase of the BraTS-PEDs 2023
challenge. Table 1 provides a summary of the cumulative ranks and FRS for each
team across all test subjects, including the teams’ rankings, and a high-level
description of the methods used by each team. The CNMCPMI2023 team achieved
the highest score among the teams, followed by NVAUTO, SherlockZyb, and
Blackbean (see Table 1 that illustrates the scores and methods used by all
teams that completed the challenge tasks).

Table 4: Summary of the performance evaluation metrics for segmentation of whole tumor (WT) region for the participating teams. The results are shown in mean $\pm$ standard deviation (median) format. Teams | Dice Similarity Coefficient | 95% Hausdorff Distance | Sensitivity
---|---|---|---
CNMCPMI2023 | 0.83 ± 0.18 (0.87) | 20.86 ± 75.31 (4.24) | 0.76 ± 0.18 (0.81)
NVAUTO | 0.84 ± 0.16 (0.87) | 18.05 ± 62.77 (4.30) | 0.80 ± 0.09 (0.82)
SherlockZyb | 0.83 ± 0.17 (0.87) | 6.11 ± 4.50 (4.79) | 0.75 ± 0.17 (0.80)
Blackbean | 0.81 ± 0.15 (0.86) | 23.56 ± 61.60 (4.95) | 0.78 ± 0.07 (0.80)
BiomedMBZ | 0.78 ± 0.20 (0.82) | 30.45 ± 82.46 (6.85) | 0.72 ± 0.19 (0.77)
SchlaugLab | 0.79 ± 0.18 (0.84) | 22.36 ± 75.20 (5.92) | 0.71 ± 0.18 (0.75)
VisionLabODU23 | 0.68 ± 0.26 (0.81) | 68.18 ± 69.56 (6.35) | 0.73 ± 0.21 (0.79)
Isahajmistry | 0.35 ± 0.23 (0.29) | 221.95 ± 99.53 (250.67) | 0.76 ± 0.20 (0.81)
UMNiverse | 0.60 ± 0.25 (0.70) | 72.62 ± 107.48 (10.69) | 0.61 ± 0.16 (0.59)
Figure 2: Pairwise p-values between the participating teams. Figure 3: Dice
and HD95 Scores for segmentation of enhancing tumor (ET) across participating
teams in the BraTS-PEDs 2023 Challenge. Figure 4: Dice and HD95 Scores for
segmentation of tumor core (TC) across participating teams in the BraTS-PEDs
2023 Challenge. Figure 5: Dice and HD95 Scores for segmentation of whole tumor
(WT) across participating teams in the BraTS-PEDs 2023 Challenge. Figure 6:
Examples of tumor segmentation for sample testing patients across the best
performing teams: (A-B) best cases; (C-D) worst cases.

Figure 2 illustrates the p-values calculated between pairs of teams to
evaluate statistical significance between the achieved performance.
Specifically, among the best performing teams, the results for CNMCPMI2023 and
NVAUTO (p = 0.45), NVAUTO and SherlockZyb (p = 0.1), and SherlockZyb and
Blackbean (p = 0.43) were not significantly different. This may be due to the
small sample size of the test cohort (n = 24) for this challenge.

Figures 3 to 5 display the Dice and HD95 scores for all test subjects for all
the participating teams. Tables 2 to 4 summarize the performance metrics—Dice,
HD95, and sensitivity—of each team. The CNMCPMI2023 $|$ NVAUTO $|$ SherlockZyb
$|$ Blackbean teams achieved Dice scores of 0.83 $|$ 0.84 $|$ 0.83 $|$ 0.81
for WT, 0.81 $|$ 0.78 $|$ 0.77 $|$ 0.79 for TC, and 0.65 $|$ 0.55 $|$ 0.63 $|$
0.53 for ET segmentation, and HD95 values of 20.86 $|$ 18.05 $|$ 6.11 $|$
23.56 for WT, 21.82 $|$ 27.10 $|$ 22.28 $|$ 22.02 for TC, and 43.89 $|$ 115.32
$|$ 59.68 $|$ 22.02 for ET segmentation. Interestingly, the sensitivity of the
NVAUTO algorithm for the segmentation of WT and ET was higher than that of the
other two teams, indicating a higher overall rate of true positive predictions
by this algorithm. However, this increased sensitivity came at the cost of
over-segmentation, resulting in more false positives and less accurate
boundary placement for ET segmentation.

The prediction results for ET demonstrate a greater variability and skewness
in the Dice score distribution, as indicated by the mean and median values in
Table 2 and the box plots in Figure 3. In contrast, the segmentation of TC and
WT (Figures 4-5) shows less variability and skewness. Given the small volume
or absence of the ET region in DMG tumors, the Dice score more heavily
penalizes segmentation errors in the ET region. TC and WT segmentations are
more consistent and accurate across all teams, with WT segmentation being the
most precise. This indicates that, on average, most methods reliably locate
the lesion within MRI scans, while distinguishing between different tumor
compartments is more challenging.

Figure 6 showcases examples of the performances of CNMCPMI2023, NVAUTO,
SherlockZyb, and Blackbean teams for tumor segmentation in four testing
subjects. Panels (A-B) indicate the best results for these three teams, with
WT Dice above 0.90. Panels (C-D) present the subjects with worst results
across the top performing teams. As indicated in these examples, there are
situations where one algorithm performs better than the remaining algorithms
in segmentation of tumorous regions. In Figure 6(C), CNMCPMI2023, SherlockZyb,
and Blackbean achieved Dice scores of 0.88, while NVAUTO achieved a low Dice
score (0.18) for WT segmentation. This low performance for NVAUTO was achieved
due to many false positive results for segmentation of the tumor for this
subject. On the other hand, in the example presented in Figure 6(D), NVAUTO
and Blackbean algorithms produced WT segmentation results with Dice scores of
0.60 and 0.70, respectively, while the other two algorithms, i.e., CNMCPMI2023
and SherlockZyb, did not generate any segmentation outputs (WT Dice = 0 or
0.05). This situation underscores the potential benefit of ensembling
different algorithms in achieving better overall performance.

##  4 Discussion

The results of the BraTS-PEDs 2023 challenge provide valuable insights into
different automated volumetric segmentation methods for pediatric brain
tumors. The comprehensive evaluation metrics and rigorous validation phase in
this challenge highlight the strengths and weaknesses of the various methods,
offering a clear benchmark for future developments.

###  4.1 Overview of the Challenge Results

In 2022, the BraTS Glioma challenge tested the top-performing
methods—developed for adult glioma segmentation—on a small set of unseen
pediatric brain tumor data. Results revealed that although models trained on
adult brain tumors achieved high Dice scores for segmenting the whole
pediatric tumor from mpMRI scans, their performance on tumor subregions (such
as ET or NC) was inaccurate. This confirmed our hypothesis that effective
pediatric segmentation techniques require training on a curated and annotated
pediatric brain tumor dataset. However, as with any machine learning and deep
learning endeavor, developing a model that generalizes well necessitates a
large, standardized dataset. To address this, the BraTS-PEDs 2023 initiative
has compiled the most extensive annotated collection of pediatric high-grade
gliomas, including astrocytomas and DMGs.

The top-performing methods at BraTS-PEDs 2023 included an ensemble method
combining nnU-Net and Swin UNETR, Auto3DSeg using the SegResNet algorithm, and
an extension of the nnU-Net model with self-supervised pretraining integrated
with adaptive region-specific loss. All of these methods are based on the
U-Net convolutional neural network architecture. The models achieved mean Dice
scores of 0.83-0.84 for WT segmentation and slightly lower yet respectable
Dice scores of 0.77-0.81 for TC segmentation. However, lower and more variable
performance was observed in the segmentation of the ET region, with mean Dice
scores of 0.65, 0.63 and 0.55 across the three top performing teams, which
emphasizes the unique challenges posed by pediatric brain tumors. The small
volume or absence of ET region in DMG likely contributes to the greater
variability and skewness in the Dice score distribution for this subregion.
This finding indicates that while most methods can reliably locate the WT
within MRI scans, accurately distinguishing between different tumor
compartments remains a significant challenge in pediatric brain tumors.

The examples in the manuscript demonstrate that no single algorithm
consistently outperforms others across all cases. The variability in
performance, especially in challenging scenarios, highlights the need for
continued research and development in this field. Specifically, ensembling the
top-performing models may achieve better overall performance for segmentation
of multi-institutional data by leveraging their individual strengths and
mitigating their weaknesses. For instance, a method with high sensitivity,
despite relatively lower Dice and higher HD95 scores, can be useful when it is
critical not to miss any positive regions (such as in ET segmentation). By
combining this method with others, it can be refined to reduce false positives
and improve boundary accuracy.

###  4.2 Limitations

There were several limitations at BraTS-PEDs 2023, including the small size of
the dataset and the limited types of pediatric brain tumor histologies. The
statistical analysis of performance between the top teams revealed no
significant differences, which may be attributed to the small sample size of
the test cohort (n = 24). This limitation suggests that future challenges
could benefit from larger test cohorts to better discern performance
differences and validate the robustness of the algorithms. In addition, to
align with the rest of the BraTS 2023 cluster of challenges, we focused our
evaluation on the segmentation of enhancing tumor, tumor core, and whole
tumor, which may limit application of these methods for response assessment in
pediatric brain tumors. Recognizing the distinct differences between adult and
pediatric brain tumors (Familiar et al., 2024), especially in components
critical for response assessment (Familiar et al., 2024; Cooney et al., 2020;
Erker et al., 2020), we will customize future evaluations to better address
pediatric tumor characteristics. Furthermore, the typically small size of ET
regions in DMG tumors prompted us to set a threshold of 50 voxels for the
detection of a true positive, which might have impacted the accuracy of ET
predictions. Moving forward, we intend to adopt a more data-driven approach to
setting this threshold in subsequent challenges.

###  4.3 Future Directions

Encouraged by the results of the BraTS-PEDs 2023 challenge, our next steps
include expanding the dataset to include subjects from additional institutions
and incorporating various histologies of high- and low-grade pediatric glioma.
This initiative will offer the research community access to a comprehensive
dataset of rare pediatric tumors with curated mpMRI and annotation, thereby
aiding in the development of advanced tools for computer-assisted treatment
planning and prognosis. Future data collection will also encompass post-
operative and post-treatment scans and clinical outcomes. The data is publicly
available at www.synapse.org/Synapse:syn51156910/wiki/627000.

##  5 Conclusion

The BraTS-PEDs 2023 challenge provided an open-access, curated, annotated
dataset of multi-sequence MRIs of pediatric brain tumors. The challenge also
provided the platform to evaluate methods for automated volumetric
segmentation of pediatric brain tumors, thereby supporting the development of
techniques that enhance decision support systems for assessing treatment
responses and predicting the outcomes of these rare conditions. This
manuscript presented the overview and results of the BraTS-PEDs 2023
challenge.

Success in any challenging medical domain depends upon the quality of well
annotated multi-institutional datasets. We are grateful to all the data
contributors, annotators, and approvers for their time and efforts. Our
profound thanks go to the Children’s Brain Tumor Network (CBTN), the
Collaborative Network for Neuro-oncology Clinical Trials (CONNECT), the
International DIPG/DMG Registry (IDIPGR), the American Society of
Neuroradiology (ASNR), and the Medical Image Computing and Computer Assisted
Intervention (MICCAI) Society for their invaluable support of this challenge.

Research reported in this publication was partly supported by the National
Institutes of Health (NIH) award: NCI/ITCR:U01CA242871 and NCI:UH3CA236536,
and by grant funding from the Pediatric Brain Tumor Foundation and DIPG/DMG
Research Funding Alliance (DDRFA). The content of this publication is solely
the responsibility of the authors and does not represent the official views of
the NIH.

For this retrospective study, informed consent had been obtained from all
subjects at their respective institutions or a waiver of informed consent was
approved by the local institutional review board. The protocol for releasing
the data was approved by the institutional review board of the data-
contributing institution. All patients were fully de-identified and stripped
of any patient identifiers before sharing with the organizing team. Further
de-identification was performed through renaming all images and the respective
ground truth segmentations, and defacing all MRI images through skull-
stripping before sharing with the participants.

None of the authors have any conflicts of interest related to this manuscript.

The training data, including ground truth segmentations, and the validation
data without ground truth segmentations can be accessed and downloaded from
the Synapse www.synapse.org/Synapse:syn51156910/wiki/627000. More information
about the BraTS-PEDs 2023 challenge is at
www.synapse.org/Synapse:syn51156910/wiki/622461.

## References

  * Artzi et al. (2020) M. Artzi et al.  Automatic segmentation, classification, and follow‐up of optic pathway gliomas using deep learning and fuzzy c‐means clustering based on mri.  _Medical Physics_ , 47(11):5693–5701, 2020.
  * Baid et al. (2021) U. Baid et al.  The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.  _arXiv:2107.02314_ , 2021.
  * Bakas et al. (2017) S. Bakas et al.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  _Scientific data_ , 4(1):1–13, 2017.
  * Bakas et al. (2018) S. Bakas et al.  Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge.  _arXiv:1811.02629_ , 2018.
  * Cooney et al. (2020) T.M. Cooney et al.  Response assessment in diffuse intrinsic pontine glioma: recommendations from the response assessment in pediatric neuro-oncology (rapno) working group.  _The Lancet Oncology_ , 21(6):e330–e336, 2020.
  * Davatzikos et al. (2018) C. Davatzikos et al.  Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome.  _Journal of medical imaging_ , 5(1):011018–011018, 2018.
  * Elingson et al. (2022) B.M. Elingson et al.  Volumetric measurements are preferred in the evaluation of mutant idh inhibition in non-enhancing diffuse gliomas: evidence from a phase i trial of ivosidenib.  _Neuro-Oncology_ , 24(5):770–778, 2022.
  * Erker et al. (2020) C. Erker et al.  Response assessment in paediatric high-grade glioma: recommendations from the response assessment in pediatric neuro-oncology (rapno) working group.  _The Lancet Oncology_ , 21(6):e317–e329, 2020.
  * Familiar et al. (2024) A.M. Familiar et al.  Towards consistency in pediatric brain tumor measurements: Challenges, solutions, and the role of ai-based segmentation.  _Neuro-Oncology_ , page noae093, 2024.
  * Fathi Kazerooni et al. (2023a) A. Fathi Kazerooni et al.  Automated tumor segmentation and brain tissue extraction from multiparametric mri of pediatric brain tumors: A multi-institutional study.  _Neuro-Oncology Advances_ , 5(1):vdad027, 2023a.
  * Jansen et al. (2015) M.H. Jansen et al.  Survival prediction model of children with diffuse intrinsic pontine glioma based on clinical and radiological criteria.  _Neuro-Oncology_ , 17(1):160–166, 2015.
  * Karargyris et al. (2023) A. Karargyris et al.  Federated benchmarking of medical artificial intelligence with medperf.  _Nat Mach Intell_ , 5:799–810, 2023.
  * Kazerooni et al. (2023b) A.F. Kazerooni et al.  The brain tumor segmentation (brats) challenge 2023: Focus on pediatrics (cbtn-connect-dipgr-asnr-miccai brats-peds).  _arXiv_ , 2023b.
  * Kharaji et al. (2024) M. Kharaji et al.  Brain tumor segmentation with advanced nnu-net: Pediatrics and adults tumors.  _Neuroscience Informatics_ , page 100156, 2024.
  * Kofler et al. (2023) F. Kofler et al.  The brain tumor segmentation (brats) challenge 2023: Local synthesis of healthy brain tissue via inpainting.  _arXiv:2305.08992_ , 2023.
  * LaBella et al. (2023) D. LaBella et al.  The asnr-miccai brain tumor segmentation (brats) challenge 2023: Intracranial meningioma.  _arXiv:2305.07642_ , 2023.
  * Lazow et al. (2022) M.A. Lazow et al.  Volumetric endpoints in diffuse intrinsic pontine glioma: comparison to cross-sectional measures and outcome correlations in the international dipg/dmg registry.  _Neuro-Oncology_ , 24(9):1598–1608, 2022.
  * Li et al. (2023) H.B. Li et al.  The brain tumor segmentation (brats) challenge 2023: Brain mr image synthesis for tumor segmentation (brasyn).  _arXiv_ , 2023.
  * Lilly et al. (2023) J.V. Lilly et al.  The children’s brain tumor network (cbtn)-accelerating research in pediatric central nervous system tumors through collaboration and open science.  _Neoplasia_ , 35:100846, 2023.
  * Liu et al. (2023) X. Liu et al.  From adult to pediatric: deep learning-based automatic segmentation of rare pediatric brain tumors.  In _Proceedings of SPIE Medical Imaging: Computer-Aided Diagnosis_ , volume 12465, page 1246505, 2023.
  * Mackay et al. (2017) A. Mackay et al.  Integrated molecular meta-analysis of 1,000 pediatric high-grade and diffuse intrinsic pontine glioma.  _Cancer cell_ , 32(4):520–537, 2017.
  * Mansoor et al. (2016) A. Mansoor et al.  Deep learning guided partitioned shape model for anterior visual pathway segmentation.  _IEEE transactions on medical imaging_ , 35(8):1856–1865, 2016.
  * Menze et al. (2014) B.H. Menze et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * Moawad et al. (2023) A.W. Moawad et al.  The brain tumor segmentation (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri.  _arXiv_ , 2023.
  * Nalepa et al. (2022) J. Nalepa et al.  Segmenting pediatric optic pathway gliomas from mri using deep learning.  _Computers in Biology and Medicine_ , 142:105237, 2022.
  * Pati et al. (2020) S. Pati et al.  The cancer imaging phenomics toolkit (captk): technical overview.  In _Proceedings of MICCAI Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries Workshop_ , 2020.
  * Pati et al. (2022) S. Pati et al.  Federated learning enables big data for rare cancer boundary detection.  _Nature communications_ , 13(1):7346, 2022.
  * Pati et al. (2023) S. Pati et al.  Gandlf: the generally nuanced deep learning framework for scalable end-to-end clinical workflows.  _Communications Engineering_ , 2:23, 2023.
  * Peng et al. (2022) J. Peng et al.  Deep learning-based automatic tumor burden assessment of pediatric high-grade gliomas, medulloblastomas, and other leptomeningeal seeding tumors.  _Neuro-Oncology_ , 24(2):289–299, 2022.
  * Rathore et al. (2018) S. Rathore et al.  Brain cancer imaging phenomics toolkit (brain-captk): an interactive platform for quantitative analysis of glioblastoma.  In _Proceedings of MICCAI Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries Workshop_ , 2018.
  * Rohlfing et al. (2010) T. Rohlfing et al.  The sri24 multichannel atlas of normal adult human brain structure.  _Human brain mapping_ , 31(5):798–819, 2010.
  * Thakkar et al. (2014) J.P. Thakkar et al.  Epidemiologic and molecular prognostic review of glioblastoma.  _Cancer Epidemiol Biomarkers Prev_ , 23(10):1985–1996, 2014.
  * Tor-Diez et al. (2020) C. Tor-Diez et al.  Unsupervised mri homogenization: application to pediatric anterior visual pathway segmentation.  In _Proceedings of MICCAI Machine Learning in Medical Imaging (MLMI) Workshop_ , volume 12436, pages 180–188, 2020.
  * Vossough et al. (2022) A. Vossough et al.  Training and comparison of nnu-net and deepmedic methods for autosegmentation of pediatric brain tumors.  _American Journal of Neuroradiology_ , 2022.
  * Yushkevich et al. (2006) P.A. Yushkevich et al.  User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability.  _Neuroimage_ , 31(3):1116–1128, 2006.