# Glioblastoma Tumor Segmentation using an Ensemble of Vision Transformers

Huafeng Liu1, Benjamin Dowdell1, Todd Engelder1, Zarah Pulmano1, Nicolas Osa1,
and Arko Barman2  1Department of Computer Science, Rice University  2Data to
Knowledge Lab & Department of Electrical and Computer Engineering, Rice
University
Email<EMAIL_ADDRESS>
###### Abstract

Glioblastoma is one of the most aggressive and deadliest types of brain
cancer, with low survival rates compared to other types of cancer. Analysis of
Magnetic Resonance Imaging (MRI) scans is one of the most effective methods
for the diagnosis and treatment of brain cancers such as glioblastoma.
Accurate tumor segmentation in MRI images is often required for treatment
planning and risk assessment of treatment methods. Here, we propose a novel
pipeline, Brain Radiology Aided by Intelligent Neural NETworks (BRAINNET),
which leverages MaskFormer, a vision transformer model, and generates robust
tumor segmentation maks. We use an ensemble of nine predictions from three
models separately trained on each of the three orthogonal 2D slice directions
(axial, sagittal, and coronal) of a 3D brain MRI volume. We train and test our
models on the publicly available UPenn-GBM dataset, consisting of 3D multi-
parametric MRI (mpMRI) scans from 611 subjects. Using Dice coefficient (DC)
and 95% Hausdorff distance (HD) for evaluation, our models achieved state-of-
the-art results in segmenting all three different tumor regions – tumor core
(DC = 0.894, HD = 2.308), whole tumor (DC = 0.891, HD = 3.552), and enhancing
tumor (DC = 0.812, HD = 1.608).

###### Index Terms:

glioblastoma, MaskFormer, tumor segmentation, vision transformer

##  I Introduction

###  I-A Background

Glioblastoma (GBM) is one of the most aggressive and deadliest types of brain
cancer, accounting for 49.1% of all primary malignant cancers, and GBM-
positive patients currently have a 6.8% five-year survival rate [1]. It can
present as a heterogeneous tumor with several regions, including the necrotic
tumor core (NCR), peritumoral edematous tissue (ED), and enhancing tumor
(ET)[2]. Treatment options include either neurosurgery, radiation therapy,
chemotherapy, immunotherapy, or targeted therapy[3]. Treatment planning and
risk assessment of treatment strategies greatly benefit from the segmentation
of the tumor from imaging, which is thus a crucial task.

Magnetic resonance imaging (MRI) is routinely used in the diagnosis and
treatment planning of GBM because it is non-invasive and can resolve subtle
contrasts within soft tissue. MRI exams can include multiple modalities,
depending on recording parameters and whether a contrast agent is used,
resulting in multiple 3-dimensional volumes for a single patient[2]. The
diagnosis and treatment planning of cancer using MRI requires a radiologist to
review and interpret a large volume of 2D slices per patient. The current
demand requires a radiologist to review a new 2D MRI image every 3 to 4
seconds continuously for 8 hours across 255 working days a year [4], also
resulting in radiologist fatigue, thus impacting diagnostic efficacy and
efficiency[5].

Artificial Intelligence (AI) assisted medical image analysis for diagnosis and
treatment has been widely reported in recent years[6, 7, 8]. In this study, we
introduce a neural network-based glioblastoma tumor segmentation model, _Brain
Radiology Aided by Intelligent Neural NETworks_ (_BRAINNET_), that leverages a
vision transformer model called MaskFormer[9], along with an ensemble strategy
using separate models trained on 2D axial, sagittal, and coronal slices of 3D
multi-parametric MRI (mpMRI) images of the brain. We also apply transfer
learning to simplify the training process by fine-tuning a pre-trained
MaskFormer model to segment the GBM tumor. BRAINNET can be applied to reduce
the time it takes for a radiologist to review MRI scans by reviewing the
segmentation masks from the model side-by-side with the MRI image.

###  I-B Literature review

Neural networks have improved both the efficiency and accuracy of medical
image segmentation substantially. Several novel approaches, such as UNet [10],
Region-based Convolutional Neural Network (R-CNN) [11], Mask R-CNN [12], and
state-of-the-art vision transformer models[13] have been applied for
segmentation tasks involving radiological images. For segmentation, MaskFormer
[9], a vision transformer model, unifies the tasks of detection and
segmentation in a single end-to-end trainable framework that can be used for
instance-level segmentation, demonstrating strong performance on various
instance segmentation benchmarks.

Starting in 2012, the International Brain Tumor Segmentation (BraTS) Challenge
has been an annual competition for developing state-of-the-art machine
learning models for brain tumor segmentation [14]. One of the earliest neural
network-based architectures applied to this dataset used a multi-scale 3D
Convolutional Neural Network (CNN) with a fully connected Conditional Random
Field to remove false positives[15]. Another architecture trained a 3D auto-
encoder with skip connections for brain tumor segmentation achieving state-of-
the-art in the BraTS 2018 challenge[16]. UNet-based architectures with
sophisticated strategies, such as cascading, region-based training, and data
augmentation, for brain tumor segmentation have also been proposed[17, 18, 19,
20]. Models have been developed for uncertainty estimation and survival
prediction based on segmentation along with the use of clinical data[18, 21].
In recent years, vision transformers have also been applied to obtain state-
of-the-art performance in GBM tumor segmentation using the BraTS dataset[13].

###  I-C Our Contributions

In our work, we develop and test BRAINNET, a pipeline that uses MaskFormer
models and an ensemble approach to accomplish automated segmentation of GBM
tumors. Our contributions include: (i) the development of an end-to-end
pipeline for GBM segmentation using Maskformer models for mpMRI images; (ii)
leveraging an ensemble strategy that combines the results from separate models
for axial, sagittal, and coronal slices; (iii) achieving state-of-the-art
performance with relatively low computational and memory requirements.

##  II Methods

###  II-A Maskformer

We adapt the state-of-the-art MaskFormer model [9], a vision transformer
model, for our proposed BRAINNET pipeline due to its simplicity, flexibility,
and performance on general 2D image segmentation tasks. MaskFormer has an
intuitive architecture with a flexible choice of the backbone (Figure 1). It
has three separate modules:

  * •

a pixel-level module that extracts image features, $\mathcal{F}$ using a
backbone network and subsequently a pixel decoder network for upsampling
$\mathcal{F}$ and extracting per-pixel embeddings,
$\mathcal{E}_{\text{pixel}}$;

  * •

a transformer module that feeds $\mathcal{F}$ into a transformer decoder to
generate per-segment embeddings, $\mathcal{Q}$; and

  * •

a segmentation module that generates $N$ class predictions along with $N$
corresponding mask embeddings, $\mathcal{E}_{\text{mask}}$ by feeding
$\mathcal{Q}$ through a multilayer perceptron (MLP). The final mask
predictions are generated by taking the dot product of
$\mathcal{E}_{\text{pixel}}$ with $\mathcal{E}_{\text{mask}}$ and using a
sigmoid activation.

Figure 1: MaskFormer model used in BRAINNET.

In our experiments, we used a pre-trained MaskFormer[9] with shifted windows
(SWIN) transformer as the backbone[22], which was trained on the ADE20K, a
scene segmentation benchmark dataset[23].

###  II-B Loss Function

For fine-tuning the pre-trained Maskformer model, we used a main mask
classification loss, $\mathcal{L}_{\text{mask-cls}}$, which is a linear
combination of a cross-entropy classification loss for each predicted
segmentation mask ($\mathcal{L}_{\text{CE}}$), a focal loss
($\mathcal{L}_{\text{focal}}$), and a dice loss ($\mathcal{L}_{\text{dice}}$),
as shown below,

$\displaystyle\mathcal{L}_{\text{mask-
cls}}=\lambda_{\text{CE}}\mathcal{L}_{\text{CE}}+\lambda_{\text{focal}}\mathcal{L}_{\text{focal}}+\lambda_{\text{dice}}\mathcal{L}_{\text{dice}}$
(1)

The cross-entropy loss is used for learning the mask class while the focal
loss and dice loss are used for learning the mask itself. The weights,
$\lambda_{\text{CE}}$, $\lambda_{\text{focal}}$, and $\lambda_{\text{dice}}$
were empirically tuned using the validation set. In our experiments, we
empirically found that $\lambda_{\text{focal}}=20$, $\lambda_{\text{dice}}=20$
and $\lambda_{\text{CE}}=1$ worked best for the dataset.

###  II-C Ensemble method

Using an ensemble of models is a common strategy for achieving better
performance in artificial intelligence and has also been used to improve the
accuracy of models for the brain tumor segmentation task [16]. We use an
ensemble method as a way to combine the results of multiple MaskFormer models
trained to segment GBM tumors using 2D axial, sagittal, and coronal slices in
a 3D MRI volume (Figure 5 (a-c)). We show in our results that an ensemble of
models trained on all three orthogonal slice planes achieves superior
performance than the traditional use of a single model trained only on 2D
axial slices. In addition, for each slice direction, we independently train 3
different models with different strategies for a total of 9 models in our
ensemble.

At inference time, each model predicts segmentation masks in 2D slices in the
same slicing direction as the training data. Next, we map the predictions for
the 2D slices back to the 3D volume for all models. As a result, we obtain 9
sets of prediction volumes. We generate an ensemble segmentation mask by
aggregating the predicted masks from all the nine models by majority voting
for each voxel.

##  III Dataset, Experiments and Results

###  III-A Dataset

We used UPenn-GBM, an openly available suite of preprocessed mpMRI scans
curated by The University of Pennsylvania Health System hosted by the Cancer
Imaging Archive[24, 25, 26]. The dataset consists of 3D MRI data for 611
unique subjects, each including four types of MRI imaging sequences: T1
(native T1-weighted), T1-GD (post-contrast T1), T2 (native T2-weighted) and
FLAIR (T2 fluid-attenuated inversion recovery). Preprocessing includes skull-
stripping and co-registration. Notably, the BraTS challenge dataset is a
subset of this dataset and contains 173 samples[24].

The dataset also contains segmentation masks labelling the various tumor
regions and sub-regions: NCR (necrotic tumor core), ED (peritumoral edematous
tissue), ET (enhancing tumor), and Else (everything else, i.e., healthy brain
tissue). Of the 611 subjects, 147 contain segmentation masks annotated by
experts which we use as ground truth labels during training [24].
Additionally, all 611 samples contain automatically generated segmentation
masks using an ensemble of three previous BraTS challenge top-scoring models
[15, 19, 21].

We preprocess the raw 3D MRI volumes into 2D RGB images using the following
steps:

  1. (i)

Data reduction: Remove zero slices across all volumes and obtain an image of
size 163x193x146, i.e., 51.44% of the original size.

  2. (ii)

Data normalization: Scale each channel for each subject independently to be in
[0, 1].

  3. (iii)

3D to 2D conversion: Convert the 3D volumes into 2D axial, sagittal, and
coronal slices and keep track of their 3D volume positions.

  4. (iv)

Data mapping: Select the FLAIR, T1, and T1-GD channels only and map them to
RGB channels. Note that T2 is ignored because of the reported strong
correlation between T1 and T2 channels for tumor segmentation[24].

An example of a preprocessed image slice is shown in Figure 2.

Figure 2: An example of 2D RGB image after pre-processing the raw data.

###  III-B Experiments

We made a 60:20:20 split for training, validation, and test data volumes
respectively. The pre-trained MaskFormer model was finetuned using the
training split and empirical design choices were made using the validation
split. The following additional preprocessing steps were used: (i) normalize
the images using the same mean and standard deviation as that used during the
pre-training; (ii) resize the height and width of the input image. During the
finetuning, we initialized the model with pre-trained model weights and
allowed updates on all model parameters. We chose to update all model
parameters rather than only the last several layers due to the different
characteristics of the mpMRI data compared to the ADE20K dataset on which it
was pretrained. We used the largest batch size possible with 15GB GPU memory.
The actual batch size we used depended on the resolution of the input image
and the size of the model (described below). The optimizer we used for
training the model is the Adam optimizer.

To evaluate the model performance, we used the two main metrics in the BraTS
challenge, the Dice Coefficient ($DC$) and 95% Hausdorff distance
($HD_{95}$)[27]. Dice coefficient [28] quantifies the similarity between a
ground truth mask, $X$, and a predicted mask, $Y$, by calculating their
overlap as,

$DC=2\frac{|X\cap Y|}{|X|\cup|Y|}$ (2)

95% Hausdorff distance [29, 30] captures the spatial dissimilarity between two
sets by calculating the maximum distance between any point in a reference set,
$X$ to its closest point in the predicted set, $Y$, as,

$HD_{95}(X,Y)=P_{95\%}\left\\{\sup_{x\in X}\inf_{y\in Y}\;d(x,y),\;\sup_{y\in
Y}\inf_{x\in X}\;d(y,x)\right\\}$ (3)

where $P_{95\%}$ indicates the $95$-th percentile. Calculating the 95th
percentile instead of the maximum value eliminates the impact of possible
outliers.

We tested multiple different training schedules and empirically found the
following two settings that performed the best: a constant learning rate of
$10^{-5}$ and train for $20$ epochs; and, a cosine annealing learning rate
scheduler[31] to vary learning rates throughout the training, starting with an
initial value of $10^{-4}$ and train for $15$ epochs.

We implemented the following data augmentation strategies:

  * •

Weak data augmentation[32]: Jitter in HSV with probability $1$, horizontal
flipping with probability $0.5$, and random cropping with a minimum $0.8$ of
the image area with probability $0.8$.

  * •

Strong data augmentation: Contrast (gamma) jitter with probability $0.5$,
horizontal flipping with probability $0.3$, and random cropping with a minimum
$0.8$ of the image area with probability $0.2$.

In our experiments, the small Maskformer model (63M parameters) outperformed
both the base model (102M parameters) and the tiny model (42M parameters). We
also experimented with the input image resolution (original resolution,
$2\times$ original resolution, and $512\times 512$ resolution).

Finally, an ensemble method with the $3$ best combinations of model size,
input resolution, data augmentation, and learning rate scheduler was used for
the 2D axial, sagittal, and coronal slices individually for a total of $9$
models:

  * •

model 1: small model; $2\times$ input resolution; strong data augmentation;
fixed learning rate

  * •

model 2: small model; $2\times$ input resolution; weak data augmentation;
cosine annealing learning rate schedule

  * •

model 3: small model; $512\times 512$ input resolution; weak data
augmentation; cosine annealing learning rate schedule

###  III-C Results

(a) Dice coefficients

(b) 95% Hausdorff distance

Figure 3: BRAINNET Model performance as compared with the performance of 3D
autoencoder[16] and nnU-Net[19]. The blue boxes show the distribution of 9
individual models. Results are shown for tumor core (TC), whole tumor (WT),
and enhancing tumor (ET) segmentations.

We trained BRAINNET to predict NCR, ED, and ET tumor segmentation masks.
However, for the purposes of consistency while comparing with the state-of-
the-art models, we report quantitative metrics (Dice coefficient, $DC$, and
95% Hausdorff distance, $HD_{95}$) for the following:

  * •

Tumor core (TC): Combination of NCR and ET, i.e., $TC=NCR\;\cup\;ET$

  * •

Whole tumor (WT): Combination of NCR, ET, and ED, i.e.,
$WT=NCR\;\cup\;ET\;\cup\;ED$

  * •

Enhancing tumor (ET)

We observed that the use of an ensemble resulted in a considerably better
performance than each of the individual models(Figure 4), resulting in higher
$DC$ and lower $HD_{95}$. We also compared the performance of our BRAINNET
model with two state-of-the-art models that were trained on the same dataset –
3D autoencoder[16] and nnU-Net[19]. Comparing using $DC$, BRAINNET performed
at par with these models and surpassed both these methods for TC segmentation.
Using $HD_{95}$, BRAINNET outperformed both the models in predicting TC and ET
while for WT, it performed at par. These results are also shown in Table I.

TABLE I: Performance comparison of BRAINNET with 3D autoencoder[16] and nnU-Net[19]. (The best performance for each tumor category using each performance metric is shown in bold.) Model | $DC$ | $HD_{95}$
---|---|---
TC | WT | ET | TC | WT | ET
3D Autoencoder | 0.815 | 0.884 | 0.766 | 4.809 | 4.516 | 3.926
nnU-Net | 0.878 | 0.928 | 0.845 | 7.623 | 3.47 | 20.73
BRAINNET | 0.894 | 0.891 | 0.812 | 2.308 | 3.552 | 1.608

An example of BRAINNET ensemble model predictions alongside the ground truth
for all 3 tumor segmentation categories along with the corresponding sagittal
MRI slice is shown in Figure 4. We observed that false positive and false
negative tumor predictions for each model in each slice direction are
ultimately eliminated to a large extent by using the ensemble approach (an
example is shown in Figure 5). For simplicity, Figure 5 only shows the
prediction of one model in each slice direction.

Figure 4: Examples of model predictions for three tumor classes (NCR, ED and
ET). Figure 5: Example of forming the prediction ensemble using individual
model predictions: (a-c) MRI data slicing directions, (d) an example of input
MRI slice in axial direction, (e-g) comparison of model predictions at the
example slice with the ground truth annotation, (h) comparison of ensemble
prediction with ground truth annotation. Note how the false positive and false
negative predictions in the individual slice directions are eliminated in the
ensemble prediction.

##  IV Discussion and Conclusion

Automated brain tumor segmentation is targeted to help alleviate the workload
of radiologists, who would have to now review segmentation masks instead of
following the time-intensive process of creating the masks themselves. In this
paper, we have introduced BRAINNET, a pipeline for segmenting different GBM
tumor regions from mpMRI images using an ensemble of fine-tuned MaskFormer
models that combines predictions of nine models, three along each 2D
orthogonal slice direction (axial, sagittal, and coronal). Experiments showed
that BRAINNET achieves state-of-the-art performance in predicting all the
tumor regions using only 15GB GPU memory. The model was trained and tested on
the publicly available UPenn-GBM dataset, from which the BraTS challenge uses
a subset. Our model achieved performance that is comparable with or surpasses
other state-of-the-art models when evaluated using the Dice coefficient and
95% Hausdorff distance. In the future, we will further investigate the
possibility of improving performance using more sophisticated ensemble methods
that leverage the raw voxel-wise probability outputs of the individual
segmentation models instead of the segmentation masks.

The source code for BRAINNET is publicly available at
https://github.com/RiceD2KLab/BRAINNET.

##  V Compliance with Ethical Standards

This study was retrospectively conducted using human subject data that have
been made openly available by The University of Pennsylvania Health System and
hosted by the Cancer Imaging Archive [24, 25, 26]. Ethical approval was not
required for this study as confirmed by the open access license (Creative
Commons Attribution 3.0 Unported License).

##  VI Conflicts of Interest

The authors have no relevant financial or non-financial interests to disclose.
No funding was received for this study.

## References

  * [1] Quinn T Ostrom, Gino Cioffi, Kristin Waite, et al.,  “CBTRUS Statistical Report: Primary Brain and Other Central Nervous System Tumors Diagnosed in the United States in 2014–2018,”  Neuro-Oncology, vol. 23, no. Supplement_3, pp. iii1–iii105, 10 2021.
  * [2] Spyridon Bakas, Mauricio Reyes, András Jakab, et al.,  “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,”  CVPR, 2018.
  * [3] Aaron C. Tan, David M. Ashley, Giselle Y. López, et al.,  “Management of glioblastoma: State of the art and future directions,”  CA: A Cancer Journal for Clinicians, vol. 70, no. 4, pp. 299–312, 2020.
  * [4] Robert J. McDonald, Kara M. Schwartz, Laurence J. Eckel, et al.,  “The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload,”  Academic Radiology, vol. 22, no. 9, pp. 1191–1198, 2015.
  * [5] Henry Zhan, Kevin Schartz, Matthew E. Zygmont, et al.,  “The impact of fatigue on complex ct case interpretation by radiology residents,”  Academic Radiology, vol. 28, no. 3, pp. 424–432, 2021.
  * [6] Yogesh Kumar, Surbhi Gupta, Ruchi Singla, and Yu-Chen Hu,  “A systematic review of artificial intelligence techniques in cancer prediction and diagnosis,”  Archives of Computational Methods in Engineering, vol. 29, pp. 2043–2070, 2022.
  * [7] D. Selvathi and A. Poornila,  “Breast cancer detection in mammogram images using deep learning technique,”  J Sci Res, vol. 25, no. 2, pp. 417 – 426, 2017.
  * [8] C. Zhang and H. Guo,  “Smart software can diagnose prostate cancer as well as pathologist,”  ScienceDaily, 2018.
  * [9] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov,  “Per-pixel classification is not all you need for semantic segmentation,”  Computer Vision and Pattern Recognition, 2021.
  * [10] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  Computer Vision and Pattern Recognition, 2015.
  * [11] Emre Ölmez, Volkan Akdoğan, Murat Korkmaz, and Orhan Er,  “Automatic segmentation of meniscus in multispectral MRI using regions with convolutional neural network (R-CNN),”  J. Digit. Imaging, vol. 33, no. 4, pp. 916–929, Aug. 2020.
  * [12] Sanskriti Singh,  “A novel mask r-cnn model to segment heterogeneous brain tumors through image subtraction,”  Image and Video Processing, 2022.
  * [13] Wenxuan Wang, Chen Chen, Meng Ding, et al.,  “Transbts: Multimodal brain tumor segmentation using transformer,”  Computer Vision and Pattern Recognition, 2021.
  * [14] Bjoern H Menze, Andras Jakab, Stefan Bauer, et al.,  “The multimodal brain tumor image segmentation benchmark (BRATS),”  IEEE Trans. Med. Imaging, vol. 34, no. 10, pp. 1993–2024, Oct. 2015\.
  * [15] Konstantinos Kamnitsas, Christian Ledig, Virginia F. J. Newcombe, et al.,  “Efficient multi-scale 3d cnn with fully connected cfr for accurate brain lesion segmentation,”  Med Image Anal, pp. 61–78, 2017.
  * [16] Andriy Myronennko,  “3d mri brain tumor segmentation using autoencoder regularization,”  Computer Vision and Pattern Recognition, 2018.
  * [17] Xue Feng, Nicholas Tusison, Sohil Patel, and Craig Meyer,  “Brain tumor segmentation using an ensemble of 3d u-net and overall survival prediction using radiomic features,”  Front. Comput. Neurosci., 2020.
  * [18] Zeyu Jiang, Changxing Ding, Minfeng Liu, and Dacheng Tao,  “Two-stage cascaded u-net: 1st place solution to BraTS challenge 2019 segmentation task,”  In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2019. Lecture Notes in Computer Science(), pp. 231–241, 2020,  Springer, Cham.
  * [19] Fabian Isensee, Paul F. Jager, Peter M. Full, et al.,  “nnu-net for brain tumor segmentation,”  In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science(), 2021,  Springer, Cham.
  * [20] Huan Minh Luu and Sung-Hong Park,  “Extending nn-unet for brain tumor segmentation,”  in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Alessandro Crimi and Spyridon Bakas, Eds., Cham, 2022, pp. 173–186, Springer International Publishing.
  * [21] Richard McKinley, Michael Rebsamen, Katrin Datwyler, et al.,  “Uncertainty-driven refinement of tumor-core segmentation using 3d-to-2d networks with label uncertainty,”  In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science(), 2021,  Springer, Cham.
  * [22] Ze Liu, Yutong Lin, Yue Cao, et al.,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  Computer Vision and Pattern Recognition, 2021.
  * [23] Bolei Zhou, Hang Zhao, Xavier Puig, et al.,  “Scene parsing through ade20k dataset,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5122–5130.
  * [24] Spyridon Bakas, Chiharu Sako, Hamed Akbari, et al.,  “The university of pennsylvania glioblastoma (upenn-gbm) cohort: advanced mri, clinical, genomics, & radiomics,”  Sci. Data, vol. 9, no. 453, 2022.
  * [25] Spyridon Bakas, Chiharu Sako, Hamed Akbari, et al.,  “Multi-parametric magnetic resonance imaging (mpmri) scans for de novo glioblastoma (gbm) patients from the university of pennsylvania health system (upenn-gbm),” 2021.
  * [26] Kenneth Clark, Bruce Vendt, Kirk Smith, et al.,  “The cancer imaging archive (tcia): Maintaining and operating a public information repository,”  Journal of Digital Imaging, vol. 26, no. 6, pp. 1045–1057, 2013\.
  * [27] Spyridon Bakas,  “Rsna-asnr-miccai brain tumor segmentation (brats) challenge,” http://braintumorsegmentation.org/, 2021.
  * [28] Ali A Taha and Allan Hanbury,  “Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool,”  BMC Med Imaging, vol. 15, pp. 29, 8 2015.
  * [29] Annika Reinke, Minu D. Tizabi, Carole H. Sudre, et al.,  “Common limitations of image processing metrics: A picture story,” 2022\.
  * [30] Abdel Aziz Taha and Allan Hanbury,  “An efficient algorithm for calculating the exact hausdorff distance,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 11, pp. 2153–2163, Nov 2015.
  * [31] Ilya Loshchilov and Frank Hutter,  “SGDR: stochastic gradient descent with restarts,”  CoRR, vol. abs/1608.03983, 2016.
  * [32] Fabian Isensee, Paul F. Jager, Peter M. Full, et al.,  “nnu-net for brain tumor segmentation,”  in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Alessandro Crimi and Spyridon Bakas, Eds., 2021, p. 123,  Springer, Cham.