Preoperative brain tumor imaging: models and software for segmentation and
standardized reporting

David Bouget1,*, André Pedersen1,2,3, Asgeir Store Jakola4,5, Vasileios
Kavouridis6, Kyrre Eeg Emblem7, Roelant S. Eijgelaar8,9, Ivar Kommers8,9,
Hilko Ardon10, Frederik Barkhof11,12, Lorenzo Bello13, Mitchel S. Berger14,
Marco Conti Nibali13, Julia Furtner15, Shawn Hervey-Jumper14, Albert J.S.
Idema16, Barbara Kiesel17, Alfred Kloet18, Emmanuel Mandonnet19, Domenique
M.J. Müller8,9, Pierre A. Robe20, Marco Rossi13, Tommaso Sciortino13, Wimar
Van den Brink21, Michiel Wagemakers22, Georg Widhalm17, Marnix G. Witte23,
Aeilko H. Zwinderman24, Philip C. De Witt Hamer8,9, Ole Solheim6,25, Ingerid
Reinertsen1,26

1 Department of Health Research, SINTEF Digital, NO-7465 Trondheim, Norway
2 Department of Clinical and Molecular Medicine, Norwegian University of
Science and Technology, NO-7491 Trondheim, Norway
3 Clinic of Surgery, St. Olavs hospital, Trondheim University Hospital,
NO-7030 Trondheim, Norway
4 Department of Neurosurgery, Sahlgrenska University Hospital, 41345
Gothenburg, Sweden
5 Department of Clinical Neuroscience, Institute of Neuroscience and
Physiology, Sahlgrenska Academy, University of Gothenburg, 40350 Gothenburg,
Sweden
6 Department of Neurosurgery, St. Olavs hospital, Trondheim University
Hospital, NO-7030 Trondheim, Norway
7 Department of Physics and Computational Radiology, Division of Radiology and
Nuclear Medicine, Oslo University Hospital, 0450 Oslo, Norway
8 Department of Neurosurgery, Amsterdam University Medical Centers, Vrije
Universiteit, 1081 HV Amsterdam, The Netherlands
9 Cancer Center Amsterdam, Brain Tumor Center, Amsterdam University Medical
Centers, 1081 HV Amsterdam, The Netherlands
10 Department of Neurosurgery, Twee Steden Hospital, 5042 AD Tilburg, The
Netherlands
11 Department of Radiology and Nuclear Medicine, Amsterdam University Medical
Centers, Vrije Universiteit, 1081 HV Amsterdam, The Netherlands
12 Institutes of Neurology and Healthcare Engineering, University College
London, London WC1E 6BT, UK
13 Neurosurgical Oncology Unit, Department of Oncology and Hemato-Oncology,
Humanitas Research Hospital, Università Degli Studi di Milano, 20122 Milano,
Italy
14 Department of Neurological Surgery, University of California San Francisco,
San Francisco, CA 94143, USA
15 Department of Biomedical Imaging and Image-Guided Therapy, Medical
University Vienna, 1090 Wien, Austria
16 Department of Neurosurgery, Northwest Clinics, 1815 JD Alkmaar, The
Netherlands
17 Department of Neurosurgery, Medical University Vienna, 1090 Wien, Austria
18 Department of Neurosurgery, Haaglanden Medical Center, 2515 VA The Hague,
The Netherlands
19 Department of Neurological Surgery, Hôpital Lariboisière, 75010 Paris,
France
20 Department of Neurology and Neurosurgery, University Medical Center
Utrecht, 3584 CX Utrecht, The Netherlands
21 Department of Neurosurgery, Isala, 8025 AB Zwolle, The Netherlands
22 Department of Neurosurgery, University Medical Center Groningen, University
of Groningen, 9713 GZ Groningen, The Netherlands
23 Department of Radiation Oncology, The Netherlands Cancer Institute, 1066 CX
Amsterdam, The Netherlands
24 Department of Clinical Epidemiology and Biostatistics, Amsterdam University
Medical Centers, University of Amsterdam, 1105 AZ Amsterdam, The Netherlands
25 Department of Neuromedicine and Movement Science, Norwegian University of
Science and Technology, NO-7491 Trondheim, Norway
26 Department of Circulation and Medical Imaging, Norwegian University of
Science and Technology, NO-7491 Trondheim, Norway

*<EMAIL_ADDRESS>
###### Abstract

For patients suffering from brain tumor, prognosis estimation and treatment
decisions are made by a multidisciplinary team based on a set of preoperative
MR scans. Currently, the lack of standardized and automatic methods for tumor
detection and generation of clinical reports, incorporating a wide range of
tumor characteristics, represents a major hurdle. In this study, we
investigate the most occurring brain tumor types: glioblastomas, lower grade
gliomas, meningiomas, and metastases, through four cohorts of up to $4\,000$
patients. Tumor segmentation models were trained using the AGU-Net
architecture with different preprocessing steps and protocols. Segmentation
performances were assessed in-depth using a wide-range of voxel and patient-
wise metrics covering volume, distance, and probabilistic aspects. Finally,
two software solutions have been developed, enabling an easy use of the
trained models and standardized generation of clinical reports: Raidionics and
Raidionics-Slicer. Segmentation performances were quite homogeneous across the
four different brain tumor types, with an average true positive Dice ranging
between 80% and 90%, patient-wise recall between 88% and 98%, and patient-wise
precision around 95%. In conjunction to Dice, the identified most relevant
other metrics were the relative absolute volume difference, the variation of
information, and the Hausdorff, Mahalanobis, and object average symmetric
surface distances. With our Raidionics software, running on a desktop computer
with CPU support, tumor segmentation can be performed in $16$ to $54$ seconds
depending on the dimensions of the MRI volume. For the generation of a
standardized clinical report, including the tumor segmentation and features
computation, $5$ to $15$ minutes are necessary. All trained models have been
made open-access together with the source code for both software solutions and
validation metrics computation. In the future, a method to convert results
from a set of metrics into a final single score would be highly desirable for
easier ranking across trained models. In addition, an automatic classification
of the brain tumor type would be necessary to replace manual user input.
Finally, the inclusion of post-operative segmentation in both software
solutions will be key for generating complete post-operative standardized
clinical reports.

_K_ eywords 3D segmentation, Deep learning, RADS, MRI, Glioma, Meningioma,
Metastasis, Open-source software

##  1 Introduction

Prognosis in patients with brain tumors is heterogeneous with survival rates
varying from weeks to several years depending on the tumor grade and type, and
for which most patients will experience progressive neurological and cognitive
deficit [1]. Brain tumors can be classified as either primary or secondary. In
the former, tumors originate from the brain itself or its supporting tissues
whereas in the latter cancer cells have spread from tumors located elsewhere
in the body to reach the brain (i.e., brain metastasis). According to the
World Health Organization classification of tumors [2], primary brain tumors
are graded by histopathological and genetic analyses and can be regrouped in
$100$ different subtypes with frequent to relatively rare occurrences. Amongst
the most frequent subtypes, tumors arising from the brain’s supportive cell
population(i.e., glial tissue) are referred to as gliomas. The more aggressive
entities are labelled as high-grade gliomas (HGGs) and are graded between $3$
and $4$, while the less aggressive entities are referred to as diffuse lower
grade gliomas (LGGs) and are graded between $2$ and $3$. Tumors arising from
the meninges, which form the external membranous covering the brain, are
referred to as meningiomas. Aside from the aforementioned large categories,
other and less frequent tumor types exist (e.g., in pituitary, sellar, or
pineal regions). Each tumor category has a distinct biology, prognosis, and
treatment [3, 4]. The most common primary malignant brain tumor type in adults
is high-grade glioma which remains among the most difficult cancers to treat
with a limited 5-year overall survival [5].

For patients affected by brain tumors, prognosis estimation and treatment
decisions are made by a multidisciplinary team (including neurosurgeons,
oncologists, and radiologists), and based on a set of preoperative MR scans. A
high accuracy in the preoperative diagnostics phase is of utmost importance
for patient outcomes. Judgements concerning the complexity or radicality of
surgery, or the risks of postoperative complications hinge on data gleaned
from MR scans. Additionally, tumor-specific characteristics such as volume and
location, or cortical structures profile can to a large degree be collected
[6]. Retrospectively, such measurements can be gathered from the analysis of
surgical cohorts, multicenter trials, or registries in order to devise patient
outcome prediction models [7, 8, 9]. Reliable measurements and reporting of
tumor characteristics are therefore instrumental in patient care. Standard
reporting and data systems (RADSs) have been established for several solid
tumors such as prostate cancer [10] and lung cancer [11]. Very few attempts
have been made for brain cancer in general [12] or glioblastomas [13]. The
main goal of RADSs is to provide rules for imaging techniques, terminology of
reports, definitions of tumor features, and treatment response to reduce
practice variation and obtain reproducible tumor classification. A broad
implementation can facilitate collaborations and stimulate evaluation for
development and improvement of RADSs.

Currently, the lack of standardized and automatic methods for tumor detection
in brain MR scans represents a major hurdle towards the generation of clinical
reports incorporating a wide range of tumor characteristics. Manual tumor
delineation or assessment by radiologists is time-consuming and subject to
intra and inter-rater variations that are difficult to characterize [14] and
therefore rarely done in clinical practice. As a result, informative tumor
features (e.g., location or volume) are often estimated from the images solely
based on crude measuring techniques (e.g., eyeballing) [15].

###  1.1 Related work

From the fast-growing development in the field of deep learning, convolutional
neural networks have demonstrated impressive performance in various
segmentation tasks and benchmark challenges, with the added-value of being
fully automatic and deterministic [16]. Regarding brain tumor segmentation,
performances have specifically been assessed on the Brain Tumor Segmentation
challenge (BraTS) dataset [17, 18]. Occurring every year since 2012, the
challenge focuses on gliomas (i.e., HGGs and LGGs) and has reached a notable
cohort size with a total of $2\,040$ patients included in the $2021$ edition,
and multiple MR sequences included for each patient (i.e., T1c, T1w, T2,
FLAIR). Segmentation performance has been assessed using the Dice similarity
coefficient and the $95$th percentile Hausdorff distance (HD95) as metrics
[19]. The current state-of-the-art is an extension of the nnU-Net architecture
[20] with an asymmetrical number of filters between the encoding and decoding
paths, substitution of all batch normalization layers by group normalization,
and addition of axial attention [21]. An average Dice score of $85$% together
with a $17.70$ mm HD95 were obtained for the enhancing tumor segmentation task
in glioblastomas. The segmentation of other brain tumor types has been
sparsely investigated in the literature in comparison, possibly due to a lack
of open-access annotated data, as illustrated by recent reviews or studies
investigating brain tumor segmentation in general [22, 23]. Grovik et al. used
a multicentric and multi-sequence dataset of $165$ metastatic patients to
train a segmentation model with the DeepLabV3 architecture [24, 25]. The best
segmentation results were around $79$% Dice score with $3.6$ false positive
detections per patient on average. Other prior studies have focused on using
variations of the DeepMedic architecture [26], using contrast-enhanced
T1-weighted MRI volumes as input, to train their segmentation models [27, 28].
Datasets were of a similar magnitude with around $200$ patients. However, in
both cases the test sets were limited to up to $20$ patients, making it
difficult to assess the generalization ability of the trained models in the
absence of cross-validation studies. Obtained average Dice scores over the
contrast-enhancing tumor were approximating $75$%, with almost $8$ false
positive detections per patient. From a recent review on the use of machine
learning applied to different meningioma-related tasks using MRI scans [29],
more than $30$ previous studies have investigated automatic diagnosis or
grading but only a handful focused on the segmentation task. In addition,
datasets’ magnitude used for segmentation purposes has been consistently
smaller than for the other tasks, with barely up to $126$ patients in the
reported studies. Laukamp et al. reported the best Dice scores using well-
known 3D neural network architectures such as DeepMedic and BioMedIA, though
at the expense of heavy preprocessing techniques the likes of atlas
registration [30, 31]. In a previous study, we achieved equally promising
performance using an attention-based U-Net architecture, reaching an average
Dice score of up to $88$% on contrast-enhanced T1-weighted MRI volumes [32].
In addition, the cross-validation studies performed over up to $600$ patients
with a wide range of tumor sizes, coming from the hospital and the outpatient
clinic, exhibited a proper ability to generalize from the trained models.

To summarize, with the exception of the BraTS challenge, there is a dearth of
high-quality MRI datasets for brain tumor segmentation. Furthermore, open-
access pretrained models and inference code are scarce and can be cumbersome
to operate, hence hindering the generation of private datasets for brain tumor
segmentation tasks. On the other hand, open-source tools are being developed
to assist in image labeling and generation of AI models for clinical
evaluation, such as MONAI label [33]. Yet, they do not integrate nor provide
access to the latest and highest performing brain tumor segmentation models
from the literature. From a validation standpoint, the focus has been on
reporting Dice scores and often Hausdorff distances, while many other
meaningful and possibly more relevant metrics exist and could be investigated
to better highlight strengths and weaknesses of the different segmentation
methods [34, 35].

The literature on RADSs for brain tumors is equally scarce with only few
attempts for preoperative glioblastoma surgery [13] or post-treatment
investigation [36]. In the former, automatic segmentation and computation of
relevant tumor features was provided, and an excellent agreement has been
shown between characteristics computed over the manual and automatic
segmentations. In the latter, the interpretation of the post-treatment MR
scans was provided using a structured set of rules, but deprived of any
automatic tumor segmentation or image analysis support.

###  1.2 Contributions

While research is exceedingly ahead for glioma segmentation under the aegis of
the BraTS challenge community, the segmentation of meningiomas and metastases
is trailing behind. In addition, validation studies in the literature have too
often been dominated by Dice score reporting and a broader inspection is
essential to ensure clinical relevance. Finally, the outcome of this research
is often not readily available, especially for the intended end-users who are
clinicians without programming experience. As such, the contributions of our
study are: (i) the training of robust segmentation models for glioblastomas,
lower grade gliomas, meningiomas, and metastases assessed using a panel of
more than $20$ different metrics to better highlight performance, (ii) the
development of two software solutions enabling easy use of the trained models
and tumor features computation: Raidionics and Raidionics-Slicer, and (iii)
open-access models and source code for the software and validation metrics
computation.

##  2 Data

For this study, four different datasets have been assembled, one for each main
tumor type considered: glioblastoma, lower grade glioma, meningioma, and
metastasis. The tumor type was assessed at time of surgery, when applicable,
following the currently applicable guidelines (i.e., either WHO 2007 or WHO
2016). Tumors were manually segmented in 3D by trained raters using as support
either a region growing algorithm [37] or a grow cut algorithm [38], and
subsequent manual editing. Trained raters were supervised by neuroradiologists
and neurosurgeons. On contrast-enhanced T1-weighted scans, the tumor was
defined as gadolinium-enhancing tissue including non-enhancing enclosed
necrosis or cysts. On FLAIR scans, the tumor was defined as the hyperintense
region. The four datasets are introduced in-depth in the subsequent sections.
An overall summary of the data available is reported in Table 1, and some
visual examples are provided in Fig. 1.

Table 1: Overview of the datasets gathered for the four brain tumor types considered. Only one MRI sequence is available for each patient, and T1c corresponds to Gd-enhanced T1-weighted MR scans. Tumor type | Sequence type | # patients | # sources | Volume average (ml) | Volume range (ml)
---|---|---|---|---|---
Glioblastoma | T1c | $2134$ | $15$ | $34.37\pm 28.83$ | $[0.01,243.39]$
Lower grade glioma | FLAIR | $659$ | $4$ | $51.71\pm 78.60$ | $[0.14,478.83]$
Meningioma | T1c | $719$ | $2$ | $19.40\pm 28.62$ | $[0.07,209.38]$
Metastasis | T1c | $396$ | $2$ | $17.53\pm 17.97$ | $[0.01,114.77]$
Figure 1: Examples of brain tumors from the raw MRI volumes collected in this
study. Each row illustrates a tumor type: glioblastoma, lower grade glioma,
meningioma, metastasis (from top to bottom). The manual annotation contours
are overlaid in red.

###  2.1 Glioblastomas

The glioblastoma dataset is made of a total of $2\,134$ Gd-enhanced
T1-weighted MRI volumes originating from fourteen different hospitals, and one
public challenge.

The first $1\,841$ patients have been collected from fourteen different
hospitals worldwide: $38$ patients from the Northwest Clinics, Alkmaar,
Netherlands (ALK); $97$ patients from the Amsterdam University Medical
Centers, location VU medical center, Netherlands (AMS); $86$ patients from the
University Medical Center Groningen, Netherlands (GRO); $103$ patients from
the Medical Center Haaglanden, the Hague, Netherlands (HAG); $75$ patients
from the Humanitas Research Hospital, Milano, Italy (MIL); $74$ patients from
the Hôpital Lariboisière, Paris, France (PAR); $134$ patients from the
University of California San Francisco Medical Center, U.S. (SFR); $49$
patients from the Medical Center Slotervaart, Amsterdam, Netherlands (SLO);
$153$ patients from the St Elisabeth Hospital, Tilburg, Netherlands (TIL);
$171$ patients from the University Medical Center Utrecht, Netherlands (UTR);
$83$ patients from the Medical University Vienna, Austria (VIE); $72$ patients
from the Isala hospital, Zwolle, Netherlands (ZWO); $456$ patients from the
St. Olavs hospital, Trondheim University Hospital, Norway (STO); and $249$
patients from the Sahlgrenska University Hospital, Gothenburg, Sweden. An in-
depth description of most cohorts can be found in a recent study [13]. The
remaining $293$ patients correspond to the training set of the BraTS challenge
(edition $2020$), but have already undergone preprocessing transformations
such as skull-stripping.

Overall, MRI volume dimensions are covering
$[159;896]\times[86;896]\times[17;512]$ voxels, and the voxel size ranges
$[0.26;1.25]\times[0.26;2.00]\times[0.47;7.50]$ mm3. An average MRI volume is
$[303\times 323\times 193]$ pixels with a spacing of $[0.86\times 0.84\times
1.24]$ mm3.

###  2.2 Lower grade gliomas

The lower grade glioma dataset is made of a total of $659$ FLAIR MRI volumes,
with mostly grade 2 diffuse gliomas, coming from four different hospitals:
$330$ patients from the Brigham and Womens Hospital, Boston, USA; $165$
patients from the St. Olavs hospital, Trondheim University Hospital, Norway;
$154$ patients from the Sahlgrenska University Hospital, Gothenburg, Sweden;
and $10$ from the University Hospital of North Norway, Norway.

Overall, MRI volume dimensions are covering
$[192;576]\times[240;640]\times[16;400]$ voxels, and the voxel size ranges
$[0.34;1.17]\times[0.34;1.17]\times[0.50;8.0]$ mm3. An average MRI volume is
$[349\times 363\times 85]$ pixels with a spacing of $[0.72\times 0.72\times
4.21]$ mm3.

###  2.3 Meningiomas

The meningioma dataset is made of $719$ Gd-enhanced T1-weighted MRI volumes,
mostly built around a dataset previously introduced [39], showcasing patients
either followed at the outpatient clinic or recommended for surgery at the St.
Olavs hospital, Trondheim University Hospital, Norway.

Overall, MRI volume dimensions are covering
$[192;512]\times[224;512]\times[11;290]$ voxels, and the voxel size ranges
$[0.41;1.05]\times[0.41;1.05]\times[0.60;7.00]$ mm3. An average MRI volume is
$[343\times 350\times 147]$ pixels with a spacing of $[0.78\times 0.78\times
1.67]$ mm3.

###  2.4 Metastases

The metastasis dataset is made of a total of $396$ Gd-enhanced T1-weighted MRI
volumes, collected from two different hospitals: $329$ patients from the St.
Olavs hospital, Trondheim University Hospital, Norway; and $67$ patients from
Oslo University Hospital, Oslo, Norway.

Overall, MRI volume dimensions are covering
$[128;560]\times[114;560]\times[19;561]$ voxels, and the voxel size ranges
$[0.43;1.33]\times[0.43;1.80]\times[0.45;7.0]$ mm3. An average MRI volume is
$[301\times 370\times 289]$ pixels with a spacing of $[0.85\times 0.76\times
1.08]$ mm3.

##  3 Methods

First, the process for automatic brain tumor segmentation including data
preprocessing, neural network architecture, and training design is introduced
in Section 3.1. Second, the tumor characteristics extraction process, using
the generated tumor segmentation as input, is summarized in Section 3.2.
Finally, a description of the two developed software solutions for performing
segmentation and standardized reporting is given in Section 3.3

###  3.1 Tumor segmentation

The architecture selected to train segmentation models for each brain tumor
type is AGU-Net, which has shown to perform well on glioblastoma and
meningioma segmentation [40, 32]. In the following, the different training
blocks are presented with some inner variations specified by roman numbers
inside brackets. A global overview is provided in Table 2 summarizing used
variants.

Table 2: Summary of the model training strategy followed for each tumor type.

Tumor type Preprocessing Strategy Protocol glioblastoma (ii) skull-stripping
(i) from-scratch (i) leave-one-out lower grade glioma (i) tight clipping (i)
from-scratch (ii) 5-fold Meningioma (i) tight clipping (i) from-scratch (ii)
5-fold Metastasis (ii) skull-stripping (ii) transfer-learning (ii) 5-fold

##### Architecture:

Single-stage approach leveraging multi-scale input and deep supervision to
preserve details, coupled to a single attention module. The loss function used
was the class-averaged Dice loss, excluding the background. The final
architecture was as described in the original article with $5$ levels and
$[16,32,128,256,256]$ as convolution blocks.

##### Preprocessing:

The following preprocessing steps were used:

  1. 1.

resampling to an isotropic spacing of $1\,\text{mm}^{3}$ using spline
interpolation of order 1 from NiBabel 111https://github.com/nipy/nibabel.

  2. 2.

(i) tight clipping around the patient’s head, excluding the void background,
or (ii) skull-stripping using a custom brain segmentation model.

  3. 3.

volume resizing to $128\times 128\times 144\,\text{voxels}$ using spline
interpolation of order 1.

  4. 4.

intensity normalization to the range $[0,1]$.

##### Training strategy:

Models were trained using the Adam optimizer over a batch size of $32$ samples
with accumulated gradients (actual batch size $2$), stopped after $30$
consecutive epochs without validation loss improvement, following either: (i)
training from scratch with $1e^{-3}$ initial learning rate, or transfer
learning with an initial learning rate of $1e^{-4}$ fine-tuning over the best
glioblastoma model.

For the data augmentation strategy, the following transforms were applied to
each input sample with a probability of $50$%: horizontal and vertical
flipping, random rotation in the range $[-20^{\circ},20^{\circ}]$, and
translation up to 10% of the axis dimension.

##### Training protocol:

Given the magnitude difference within our four datasets, two different
protocols were considered: (i) a three-way split at the hospital level whereby
MRI volumes from one hospital constituted the validation fold; MRI volumes
from a second hospital constituted the test fold; and the remaining MRI
volumes constituted the training fold. As such, each hospital was used in turn
as the test set in order to properly assess the ability of the different
models to generalize. Or (ii) a 5-fold cross-validation with random two-way
split over MRI volumes whereby four folds are used in turn as training set and
the remaining one as validation set, without the existence of a proper
separate test set.

###  3.2 Preoperative clinical reporting

For the generation of standardized preoperative clinical reports in a
reproducible fashion, the computation of tumor characteristics was performed
after alignment to a standard reference space. As described in-depth in our
previous study [13], the reference space was constituted by the symmetric
Montreal Neurological Institute ICBM2009a atlas (MNI) [41]. The atlas space
not possessing any brain average as FLAIR sequence, the T1 atlas file was used
for all tumor types.

For each tumor type, the collection of features includes: volume, laterality,
multifocality, cortical structure location profile, and subcortical structure
location profile. Specifically tailored for glioblastomas, resectability
features are therefore not available for the other brain tumor types.

###  3.3 Proposed software

In order to make our models and tumor features easily available to the
community, we have developed two software solutions. The first one is a stand-
alone software called Raidionics, and the second one is a plugin to 3D Slicer
given its predominant and widespread use in the field [42]. Both solutions
provide access to a similar back-end including inference and processing code.
However, the GUI and intended user interactions differ. The trained models are
stored in a separate online location and are downloaded on the user’s computer
at runtime. Models can be improved over time and a change will be
automatically detected, resulting in the replacement of outdated models on the
user’s machine.

Figure 2: Illustration of the Raidionics software after generating the
standardized report for a patient suffering from a glioblastoma. The left side
presents the tumor characteristics belonging to the report, whereas the right
side offers a simplistic viewer.

####  3.3.1 Stand-alone solution: Raidionics

The software proposes two modes: (i) single-use where only one patient is to
be processed and results can be visually assessed in the 2D viewer, and (ii)
batch-mode whereby a collection of patients can be processed sequentially
without any viewing possibility. In each mode, the option is left to the user
to solely perform tumor segmentation, or to compute the whole set of tumor
characteristics and generate the standardized report. For each patient, the
software expects an MRI scan as input (i.e., Gd-enhanced T1-weighted or FLAIR
sequence) and the tumor type must be manually selected. Additionally, a pre-
existing tumor segmentation mask can be provided to bypass the automatic
segmentation, if collecting the tumor characteristics is the main interest and
manual annotations have been performed beforehand. The total set of processed
files saved on disk includes the standardized reports, brain and tumor
segmentation masks in both patient and MNI space, cortical and subcortical
structures masks in both patient and MNI space, and the registration files to
navigate from patient to MNI space. To complement the reporting and give the
possibility for follow-up statistical studies, the complete set of computed
features is also provided in comma separated value format (i.e., .csv).

The software has been developed in Python 3.6.9, using PySide2 v5.15.2 for the
graphical user interface, and only uses the Central Processing Unit (CPU) for
the various computations. The software has been tested and is compatible with
Windows ($\geq$ 10), macOS ($\geq$ Catalina 10.15), and Ubuntu Linux ($\geq$
18.04). An illustration of the software is provided in Fig. 2. Cross-platform
installers and source code are freely available at
https://github.com/dbouget/Raidionics.

####  3.3.2 3D Slicer plugin: Raidionics-Slicer

The 3D Slicer plugin has been developed using the DeepInfer plugin as baseline
[43], and is mostly intended for tumor segmentation purposes. Through a
slider, the possibility is provided to manually alter the probability
threshold cutoff in order to refine the proposed binary mask. Further manual
editing can be performed thereafter using the existing 3D Slicer
functionalities. The back-end processing code has been bundled into a Docker
image for convenience, and therefore administrator rights are required for the
end-user to perform the installation locally. The same inputs, behaviour, and
outputs can be expected as for the stand-alone software.

The GitHub repository for the 3D Slicer plugin can be found at
https://github.com/dbouget/Raidionics-Slicer, and an illustration is provided
in Fig. 3.

Figure 3: Illustration of the Raidionics-Slicer plugin after generating the
standardized report for a patient suffering from a glioblastoma.

##  4 Validation studies

In the validation studies, only the automatic segmentation performances are
assessed. The clinical validity and relevance of the extracted tumor features
has been addressed thoroughly in a previous study [13]. To better grasp the
different aspects of the segmentation performance, a wider set of metrics is
studied as described in Section 4.1. For the voxel-wise segmentation task,
only two classes are considered as the whole tumor extent (including contrast-
enhancing regions, cysts, and necrosis) is the target: non-tumor tissue or
tumor tissue. In that sense, a positive voxel is a voxel exhibiting tumor
tissue, whereas a negative voxel is a voxel exhibiting background or normal
tissue.

###  4.1 Metrics

Following a review on metrics for evaluating 3D medical image segmentation
[35], a broad spectrum of $25$ metrics was selected, computed either voxel-
wise or instance-wise, and grouped according to the following categories:
overlap-based, volume-based, information theory-based, probabilistic, and
spatial distance-based.

##### Voxel-wise:

For quantifying semantic segmentation performance, we have selected the
following metrics computed directly and indiscriminately over all voxels of a
given patient MRI volume:

  1. 1.

Overlap-based: (i) True Positive Rate (TPR), also called recall or
sensitivity, is the probability that an actual positive voxel will test
positive; (ii) True Negative Rate (TNR), also called specificity, is the
probability that an actual negative voxel will test negative; (iii) False
Positive Rate (FPR), is the probability that a false alarm will be raised
(i.e., a negative voxel will test positive); (iv) False Negative Rate (FNR),
also called missed rate, is the probability that a true positive voxel will
test negative; (v) Positive Predictive Value (PPV), also referred to as
precision, is the ratio of truly positive voxels over all voxels which tested
positive; (vi) Dice score (Dice), also called the overlap index and gauging
the similarity of two samples, is the most commonly used metric in validating
medical volume segmentation [44]; (vii) Dice True Positive score (Dice-TP) is
similar to the Dice score, but is only computed over the true positive
predictions (i.e., when the model found the tumor); (viii) Intersection Over
Union (IoU), also called the Jaccard index, measures the volume similarity as
the size of the intersection divided by the size of the union of two samples
[45]; (ix) Global Consistency Error (GCE), defined as the error measure
averaged over all voxels [46].

  2. 2.

Volume-based: (i) Volumetric Similarity (VS), as the absolute volume
difference divided by the sum of the compared volumes [47]; (ii) Relative
Absolute Volume Difference (RAVD), as the relative absolute volume difference
between the joint binary objects in the two images. This is a percentage value
in the range $[-1.0,\infty)$ for which a $0$ denotes an ideal score.

  3. 3.

Information theory-based: (i) Normalized Mutual Information (MI),
normalization of the mutual information score to scale the results between $0$
(no mutual information) and $1$ (perfect correlation) [48]; (ii) Variation Of
Information (VOI), measuring the amount of information lost or gained when
changing from one variable to the other, in this case to compare clustering
partitions [49].

  4. 4.

Probabilistic: (i) Cohen’s Kappa Score (CKS), measuring the agreement between
two samples [50]. The metric ranges between $-1.0$ and $1.0$ whereby the
maximum value means complete agreement, and zero or lower means chance
agreement; (ii) Area Under the Curve (AUC), first presented as the measure of
accuracy in the diagnostic radiology [51], further adjusted for the validation
of machine learning algorithms; (iii) Volume Correlation (VC), as the linear
correlation in binary object volume, measured through the Pearson product-
moment correlation coefficient where the coefficient ranges $[-1.,1.]$; (iv)
Matthews Correlation Coefficient (MCC), as a measure of the quality of binary
and multiclass classifications, taking into account true and false positives
and negatives and generally regarded as a balanced measure [52]. The metric
ranges between $-1.0$ and $1.0$ whereby $1.0$ represents a perfect prediction,
$0.0$ an average random prediction, and $-1.0$ an inverse prediction; (v)
Probabilistic Distance (PBD), as a measure of the distance between fuzzy
segmentations [53].

  5. 5.

Spatial-distance-based: (i) $95$th percentile Hausdorff distance (HD$95$),
measuring the boundary delineation quality (i.e., contours). The $95\%$
version is used to make measurements more robust to small outliers [54]; (ii)
the Mahalanobis distance (MHD), measuring the correlation of all points and
calculated according to the variant described for the validation of image
segmentation [55]; (iii) Average Symmetric Surface Distance (ASSD), as the
average symmetric surface distance between the binary objects in two images.

##### Instance-wise:

For quantifying instance detection performance, we chose the following
metrics, reported in a patient-wise fashion (PW) or in an object-wise fashion
(OW). In the latter, and in case of multifocal tumors, each focus is
considered as a separate tumor. The detection threshold has been set to $0.1$%
Dice to determine whether an automatic segmentation is eligible to be
considered as a true detection or a false positive.

  1. 1.

Overlap-based: (i) Recall, as the ratio in % of tumors properly identified;
(ii) Precision, as the ratio in % of tumors incorrectly detected; (iii)
F1-score (F1), measuring information retrieval as a trade-off between the
recall and precision [56]; (iv) False Positives Per Patient (FPPP), as the
average number of incorrect detections per patient.

  2. 2.

Probabilistic: (i) Adjusted Rand Index (ARI), as a similarity measure between
two clusters by considering all pairs of samples and counting pairs that are
assigned in the same or different clusters between the model prediction and
the ground truth [57]. The metric ranges from $-1.0$ to $1.0$, whereby random
segmentation has an ARI close to $0.0$ and $1.0$ stands for perfect match.

  3. 3.

Spatial-distance-based: (i) Object Average Symmetric Surface Distance (OASSD),
as the average symmetric surface distance (ASSD) between the binary objects in
two volumes.

###  4.2 Measurements

Pooled estimates, computed from each fold’s results, are reported for each
measurement [58]. Overall, measurements are reported as mean and standard
deviation (indicated by $\pm$) in the tables.

##### Voxel-wise:

For semantic segmentation performance, the Dice score is computed between the
ground truth volume and a binary representation of the probability map
generated by a trained model. The binary representation is computed for ten
different equally-spaced probability thresholds (PT) in the range $]0,1]$.

##### Instance-wise:

For instance detection performance, a connected components approach coupled to
a pairing strategy was employed to associate ground truth and detected tumor
parts. A minimum size threshold of $50$ voxels was set and objects below that
limit were discarded. A detection was deemed true positive for any Dice score
strictly higher than $0$%.

###  4.3 Experiments

To validate the trained models, the following set of experiments was
conducted:

  1. (i)

Overall performance study: k-fold cross-validation studies for the different
tumor types for assessing segmentation performance. For easy interpretation,
only Dice scores together with patient-wise and object-wise recall, precision,
and F1-score values are reported.

  2. (ii)

Metrics analysis: in-depth performance comparison using the additional
metrics, and confusion matrix computation between the metrics to identify
redundancy in their use.

  3. (iii)

Representative models selection: identification of one final segmentation
model for each tumor type, which will be made available for use in our
software solutions.

  4. (iv)

Speed study: computation of the pure inference speed and the total elapsed
time required to generate predictions for a new patient, obtained with CPU
support and reported in seconds. The operations required to prepare the data
to be sent through the network, to initialize the environment, to load the
trained model, and to reconstruct the probability map in the referential space
of the original volume are accounted for. The experiment was repeated ten
consecutive times over the same MRI volume for each model, using a
representative sample of each dataset in terms of dimension and spacing.

##  5 Results

###  5.1 Implementation details

Results were obtained using a computer with the following specifications:
Intel Core Processor (Broadwell, no TSX, IBRS) CPU with $16$ cores, $64$GB of
RAM, Tesla V100S ($32$GB) dedicated GPU, and a regular hard-drive. Training
and inference processes were implemented in Python 3.6 using TensorFlow
v1.13.1, and the data augmentation was performed using the Imgaug Python
library [59]. The metrics were for the most part computed manually using the
equations described in the supplementary material, or alternatively using the
sklearn v0.24.2 [60] and medpy v0.4.0 [61] Python libraries. The source code
used for computing the metrics and performing the validation studies is made
publicly available at
https://github.com/dbouget/validation_metrics_computation.

###  5.2 Overall performance study

Figure 4: Volume-wise (equally binned) Dice performance as boxplots for each
of the four tumor types. Table 3: Segmentation performance summary for each
tumor type.

Voxel-wise Patient-wise Object-wise Tumor type Dice Dice-TP F1-score Recall
Precision F1-score Recall Precision Glioblastoma $85.69\pm 16.97$ $87.36\pm
12.17$ $97.40\pm 01.01$ $98.08\pm 01.29$ $96.76\pm 01.43$ $89.61\pm 04.11$
$85.78\pm 07.95$ $94.19\pm 02.71$ LGG $75.39\pm 25.95$ $81.24\pm 16.01$
$93.60\pm 01.74$ $92.86\pm 03.19$ $94.42\pm 01.07$ $81.58\pm 02.25$ $75.58\pm
02.41$ $88.70\pm 03.16$ Meningioma $75.00\pm 30.52$ $84.81\pm 15.07$ $90.67\pm
01.42$ $88.46\pm 02.12$ $93.25\pm 04.76$ $83.85\pm 03.60$ $80.93\pm 04.34$
$87.77\pm 08.30$ Metastasis $87.73\pm 18.94$ $90.02\pm 12.80$ $97.54\pm 00.76$
$97.46\pm 01.38$ $97.63\pm 00.77$ $88.71\pm 01.34$ $82.80\pm 02.38$ $95.60\pm
01.45$

An overall summary of brain tumor segmentation performance for all four tumor
subtypes is presented in Table 3. Meningiomas and lower grade gliomas appear
more difficult to segment given average Dice scores of $75$%, compared to
average Dice scores of $85$% for glioblastomas and metastases. A similar
trend, yet with a slightly smaller gap, can be noted for the Dice-TP scores
ranging between $81$% and $90$% with a standard deviation around $15$%,
indicating the quality and relative stability of the trained models. From a
patient-wise perspective, those results demonstrate the difficulty of
achieving good recall while keeping the precision steadily above $95$%. Even
though a direct comparison to the literature is impossible since different
datasets have been used, obtained performance is on-par if not better than
previously reported performances where Dice scores have been ranging from
$75$% to $85$%.

Regarding the lower grade glioma tumor subtype, the diffuse nature of the
tumors and less pronounced gradients over image intensities are possible
explanations for the lower segmentation performance. For the meningioma
category, the reason for the lower Dice-score and recall values can be
attributed to the larger number of small tumors ($<2$ ml) compared to other
subtypes. In addition, outliers have been identified in this dataset whereby a
small extent of the tumors were either partly enhancing because of
calcification, or non-enhancing due to intraosseous growth. For all tumor
types, Dice-score distributions are reported against tumor volumes in Fig. 4
for ten equally-sized bins. For meningiomas, four bins are necessary to group
tumors with a volume up to $4$ ml while only one bin is necessary for the
glioblastomas, indicating a volume distribution imbalance between the two
types. The diamond-shaped points outside the boxes represent cases where the
segmentation model did not perform well (cf. Figures S1, S2, S3, and S4, Sup.
Mat.).

Figure 5: Examples of segmentation performances. One row illustrates one tumor
type: glioblastoma, lower grade glioma, meningioma, metastasis (from top to
bottom), and each column depicts a different patient. The manual delineation
is shown in red, the automatic segmentation in blue, and the patient-wise Dice
score in white.

While tumor volumes and outlier MR scans are reasons for the discrepancy in
Dice and recall values across the board, precision is rather unaffected and
more stable. The nature of the convolutional neural network architecture and
training strategy used can explain those results. By leveraging volumes
covering the full brain, global relationships can be learned by the trained
model hence reducing the confusion between tumor regions and other contrast-
enhancing structures such as blood vessels. Given GPU memory limitation, the
preprocessed MR scans have undergone a significant downsampling, and as such
small tumors are reduced to very few voxels, impacting mainly recall
performance.

Finally, an average decrease of $\sim 10$% can be noticed between patient-wise
and object-wise detection metrics, whereby satellite tumors are on average an
order of magnitude smaller than the main tumor, and are hence more prone to be
omitted or poorly segmented by our models. Segmentation performance is
illustrated in Fig. 5. Each row corresponds to one tumor type and each column
depict a different patient.

###  5.3 Metrics analysis

Side-by-side voxel-wise performances regarding the overlap-based metrics are
reported in Table 4 and Table 5. Unsurprisingly, given the good precision
performance and the absence of patients without a tumor, both TNR and its
opposite FPR scores are almost perfect for all tumor types. Similarly, the TPR
and its opposite FNR metrics are scoring similarly to Dice. Within each tumor
category, the overlap-based metrics are following the same trend whereby a
higher average Dice score would correlate with a higher score for any other
metrics and vice versa (e.g.,IoU). An exception can be made regarding the
behaviour of the GCE metric, scoring on average higher for glioblastomas than
for meningiomas and as such not following the same pattern as Dice. Upon
careful visual inspection, the GCE metric seems to be extremely sensitive to
outliers, either coming from the image quality or manual ground truth
correctness (cf. top row in Figures S1-S4, Sup. Mat.). Given the non-
normalized state of the GCE metric, and its absence of any upper bound, an
extremely poor agreement between manual ground truth and automatic
segmentation will result in a score orders of magnitude higher than its
average expression over a given dataset. Regarding the two volume-based
metrics, featured rightmost in the second table, an antagonistic pattern
towards Dice can be observed. The VS metric has the same cross-type trend as
Dice with similar yet slightly greater scores. On the other hand, while the
RAVD metric scores best over the metastasis group similar to Dice, its worst
average value is obtained for the glioblastoma group, hence potentially
exhibiting the same frailty towards outliers as for the GCE metric.

Table 4: Voxel-wise overlap-based metrics performance summary for each tumor
type.

Tumor type TPR TNR FPR FNR PPV Glioblastoma $87.88\pm 17.64$ $99.96\pm 00.06$
$00.04\pm 00.06$ $12.12\pm 17.64$ $87.35\pm 13.29$ LGG $77.91\pm 27.89$
$99.90\pm 00.16$ $00.09\pm 00.16$ $22.08\pm 27.89$ $82.16\pm 17.01$ Meningioma
$77.44\pm 32.48$ $99.97\pm 00.04$ $00.02\pm 00.04$ $22.56\pm 32.48$ $84.77\pm
15.69$ Metastasis $88.45\pm 20.82$ $99.98\pm 00.03$ $00.01\pm 00.03$ $11.54\pm
20.82$ $89.43\pm 16.78$

Table 5: Voxel-wise performance summary for each tumor type for overlap-based
and volume-based metrics.

Overlap-based Volume-based Tumor type Dice Dice-TP IoU GCE (1e4) VS RAVD
Glioblastoma $85.69\pm 16.97$ $87.36\pm 12.17$ $77.59\pm 17.99$ $12.34\pm
12.57$ $90.43\pm 16.94$ $13.98\pm 171.2$ LGG $75.39\pm 25.95$ $81.24\pm 16.01$
$65.72\pm 25.32$ $34.15\pm 46.34$ $82.20\pm 26.44$ $07.88\pm 60.14$ Meningioma
$75.00\pm 30.52$ $84.81\pm 15.07$ $67.13\pm 29.39$ $09.04\pm 17.53$ $80.21\pm
31.08$ $07.87\pm 61.31$ Metastasis $87.73\pm 18.94$ $90.02\pm 12.80$ $81.56\pm
20.42$ $04.55\pm 07.62$ $91.37\pm 18.61$ $02.11\pm 55.35$

Next off, voxel-wise performance for information theory-based and
probabilistic metric are regrouped in Table 6. The MI and VOI metrics, both
based on information theory, are exhibiting an inverse behaviour in line with
observations about the relationship between Dice and GCE metrics. The
normalized mutual information metric ranges from $0.668$ to $0.829$ for Dice
scores between $75$% and $87$%, showcasing stability but also correlation. On
the contrary, the VOI metric expresses a behaviour concurrent to GCE whereby
worst performance is obtained for the lower grade gliomas and then
glioblastomas categories, while it performs best over metastases where Dice
also scores the highest. Alike the aforementioned metric groups exhibiting
inner discrepancies, three of the five probabilistic metrics follow a similar
trend scoring high alongside Dice, with an average gap of $0.1$ corresponding
to a $\sim 10$% Dice score difference. Meanwhile, the PBD metric has a
behaviour of its own scoring an order of magnitude worse for the meningioma
category than for the three other subtypes. The metric is not normalized and
an extremely poor agreement between the manual ground truth and automatic
segmentation would result in an extremely large score, similar to the GCE
metric, hence reporting the median score in addition might be of interest (cf.
second row in Figures S1-S4, Sup. Mat.).

Table 6: Voxel-wise performance summary for each tumor type for information
theory-based and probabilistic metrics.

Information theory-based Probabilistic Tumor type MI VOI CKS AUC VC MCC PBD
Glioblastoma $0.787\pm 0.168$ $0.011\pm 0.009$ $0.856\pm 0.169$ $0.939\pm
0.088$ $0.978\pm 0.089$ $0.875\pm 0.122$ $0.840\pm 24.02$ LGG $0.668\pm 0.246$
$0.026\pm 0.030$ $0.753\pm 0.259$ $0.889\pm 0.139$ $0.961\pm 0.119$ $0.812\pm
0.167$ $0.573\pm 04.82$ Meningioma $0.691\pm 0.291$ $0.008\pm 0.013$ $0.749\pm
0.305$ $0.887\pm 0.162$ $0.954\pm 0.149$ $0.841\pm 0.171$ $5.358\pm 103.4$
Metastasis $0.829\pm 0.191$ $0.004\pm 0.006$ $0.877\pm 0.189$ $0.942\pm 0.104$
$0.978\pm 0.100$ $0.901\pm 0.127$ $0.152\pm 0.623$

Finally, the voxel-wise distance-based metrics are reported in Table 7.
Similar cross-type trends can also be noted whereby the best HD95 of $4.97$ mm
is obtained for the glioblastoma category and the worst HD95 of $10$ mm for
meningiomas, heavily correlated to Dice performance. Our average HD95 results
appear lower than previously reported results in the literature, however a
strong statement can hardly be made as the tumors featured can vary highly in
terms of volume and number of satellites which might reflect greatly on
metrics’ average scores. The other two spatial distance-based metrics display
a similar behaviour to HD95, whereby tumor types can be ranked as follows
based on best to worse performance: glioblastoma, metastasis, lower grade
glioma, and meningioma.

Table 7: Voxel-wise performance summary for each tumor type for spatial
distance-based metrics.

Tumor type HD95 MHD ASSD Glioblastoma $04.97\pm 09.06$ $00.41\pm 03.69$
$01.46\pm 03.22$ LGG $08.37\pm 13.31$ $00.53\pm 03.27$ $02.19\pm 05.06$
Meningioma $10.11\pm 21.82$ $00.72\pm 03.57$ $02.77\pm 07.91$ Metastasis
$07.54\pm 20.61$ $00.54\pm 04.56$ $01.73\pm 05.89$

Regarding instance-wise metrics, grouped in Table 8, the close OASSD average
values between glioblastomas and meningiomas represents the most surprising
outcome given the $5$% difference in F1-score. Unsurprisingly, the lower grade
glioma category achieves the highest average OASSD with $2.6$ mm together with
the lowest F1-score. As one might expect, the amount of FPPP correlates
greatly with the average precision values obtained. Ultimately, the ARI metric
generates scores extremely similar to voxel-wise Dice and correlates highly
with the F1-score whereby the glioblastoma and metastasis categories obtain
almost $0.1$ more than for the meningioma and lower grade glioma subtypes.

Table 8: Instance-wise performance for each tumor type.

Tumor type F1-score Recall Precision FPPP ARI OASSD Glioblastoma $89.61\pm
04.11$ $85.78\pm 07.95$ $94.19\pm 02.71$ $0.078\pm 0.037$ $0.856\pm 0.169$
$01.45\pm 02.82$ LGG $81.58\pm 02.25$ $75.57\pm 02.40$ $88.67\pm 03.16$
$0.129\pm 0.041$ $0.751\pm 0.259$ $02.60\pm 06.10$ Meningioma $83.85\pm 03.60$
$80.93\pm 04.34$ $87.77\pm 08.30$ $0.151\pm 0.128$ $0.749\pm 0.305$ $01.62\pm
04.09$ Metastasis $88.71\pm 01.34$ $82.79\pm 02.38$ $95.60\pm 01.45$ $0.061\pm
0.020$ $0.877\pm 0.189$ $0.672\pm 0.869$

For completeness, the correlation between the different metrics computed in
this study has been assessed, and the results over the glioblastoma category
are shown in Table 9 (cf. other correlation matrices in Tables S2, S4, S6, and
S8, Sup. Mat.). Some metrics have been excluded given inherent correlation
from their computation, such as FPR and FNR being the opposite of TNR and TPR.
Similarly, metrics having computation in a voxel-wise, patient-wise, or
instance-wise fashion were not considered in the matrix (i.e., recall,
precision, and F1-score). Overall, the conclusions identified by analyzing the
raw average results are further confirmed whereby a majority of voxel-wise
metrics correlate with one another and thus do not bring any additional
information to Dice. However, relevant insight can be obtained from the RAVD
and GCE/VOI metrics given their low correlation to Dice and their higher
sensitivity towards outliers, enabling to quantify the ability to generalize
of the model or potentially the quality of the data and manual ground truth
(cf. third row in Figures S1-S4, Sup. Mat.). The correlation between HD95 and
MHD appears also quite low for spatial distance-based metrics, indicating a
potential usefulness. Finally, in the instance-wise category, the OASSD is a
stand-alone metric offering to properly assess model performance over the
detection of satellite tumors. To conclude, a final pool of metrics to
consider for benchmarking purposes and capturing all aspects of the
segmentation performances are: Dice, RAVD, VOI, HD95, MHD, and OASSD. Given
the task, reporting patient-wise and instance-wise recall, precision, and
F1-score is always of interest because of an innate comprehension of their
meaning, easy to interpret for clinicians or other end-users.

Table 9: Metrics correlation matrix for glioblastoma segmentation. The color
intensity of each cell represents the strength of the correlation, where blue
denotes direct correlation and red denotes inverse correlation.

Overlap Volume Information theory Probabilistic Spatial distance Instance-wise
Dice TPR TNR PPV IoU GCE VS RAVD MI VOI CKS AUC VC MCC PBD HD95 MHD ASSD ARI
OASSD Dice blue!1001.0 blue!700.7 blue!280.29 blue!610.62 blue!980.98
red!22-0.22 blue!940.94 red!34-0.35 blue!980.99 red!23-0.23 blue!991.0
blue!700.71 blue!780.78 blue!991.0 red!34-0.34 red!54-0.55 red!43-0.43
red!70-0.71 blue!991.0 red!30-0.3 TPR blue!700.7 blue!1001.0 red!16-0.17
red!6-0.07 blue!700.71 red!8-0.08 blue!620.62 blue!90.1 blue!700.7 red!8-0.08
blue!700.7 blue!991.0 blue!510.51 blue!700.71 red!26-0.26 red!37-0.38
red!33-0.34 red!47-0.47 blue!700.7 red!20-0.2 TNR blue!280.29 red!16-0.17
blue!1001.0 blue!570.58 blue!280.28 red!75-0.76 blue!280.29 red!36-0.36
blue!330.33 red!75-0.76 blue!280.29 red!16-0.17 blue!220.23 blue!280.29
red!4-0.04 red!16-0.16 red!4-0.04 red!27-0.27 blue!290.29 red!22-0.22 PPV
blue!610.62 red!6-0.07 blue!570.58 blue!1001.0 blue!640.64 red!24-0.24
blue!540.55 red!49-0.49 blue!640.64 red!24-0.25 blue!610.62 red!6-0.07
blue!470.47 blue!620.63 red!15-0.16 red!37-0.38 red!21-0.21 red!47-0.47
blue!610.62 red!21-0.22 IoU blue!980.98 blue!700.71 blue!280.28 blue!640.64
blue!1001.0 red!23-0.24 blue!890.9 red!28-0.29 blue!990.99 red!23-0.24
blue!980.98 blue!700.71 blue!710.71 blue!980.99 red!27-0.28 red!55-0.55
red!37-0.37 red!69-0.7 blue!980.98 red!30-0.31 GCE red!22-0.22 red!8-0.08
red!75-0.76 red!24-0.24 red!23-0.24 blue!1001.0 red!19-0.19 blue!120.13
red!30-0.3 blue!991.0 red!22-0.23 red!8-0.09 red!13-0.14 red!22-0.23
blue!20.02 blue!170.18 blue!20.03 blue!280.29 red!22-0.23 blue!270.28 VS
blue!940.94 blue!620.62 blue!280.29 blue!540.55 blue!890.9 red!19-0.19
blue!1001.0 red!37-0.37 blue!900.9 red!20-0.2 blue!940.94 blue!620.62
blue!750.76 blue!920.92 red!35-0.36 red!48-0.48 red!43-0.43 red!65-0.65
blue!940.94 red!26-0.26 RAVD red!34-0.35 blue!90.1 red!36-0.36 red!49-0.49
red!28-0.29 blue!120.13 red!37-0.37 blue!1001.0 red!31-0.31 blue!140.15
red!34-0.35 blue!90.1 red!38-0.39 red!34-0.34 blue!170.18 blue!190.19
blue!140.14 blue!280.28 red!34-0.35 blue!140.15 MI blue!980.99 blue!700.7
blue!330.33 blue!640.64 blue!990.99 red!30-0.3 blue!900.9 red!31-0.31
blue!1001.0 red!30-0.31 blue!980.99 blue!700.7 blue!730.74 blue!990.99
red!30-0.31 red!56-0.56 red!39-0.4 red!71-0.71 blue!980.99 red!32-0.32 VOI
red!23-0.23 red!8-0.08 red!75-0.76 red!24-0.25 red!23-0.24 blue!991.0
red!20-0.2 blue!140.15 red!30-0.31 blue!1001.0 red!23-0.23 red!8-0.08
red!15-0.15 red!23-0.24 blue!20.03 blue!180.18 blue!30.03 blue!290.3
red!23-0.24 blue!280.28 CKS blue!991.0 blue!700.7 blue!280.29 blue!610.62
blue!980.98 red!22-0.23 blue!940.94 red!34-0.35 blue!980.99 red!23-0.23
blue!1001.0 blue!700.71 blue!780.78 blue!991.0 red!34-0.34 red!54-0.55
red!43-0.43 red!70-0.71 blue!991.0 red!30-0.3 AUC blue!700.71 blue!991.0
red!16-0.17 red!6-0.07 blue!700.71 red!8-0.09 blue!620.62 blue!90.1 blue!700.7
red!8-0.08 blue!700.71 blue!1001.0 blue!510.51 blue!710.71 red!26-0.27
red!38-0.38 red!33-0.34 red!47-0.47 blue!700.71 red!20-0.2 VC blue!780.78
blue!510.51 blue!220.23 blue!470.47 blue!710.71 red!13-0.14 blue!750.76
red!38-0.39 blue!730.74 red!15-0.15 blue!780.78 blue!510.51 blue!1001.0
blue!770.78 red!49-0.49 red!51-0.51 red!58-0.58 red!71-0.71 blue!780.78
red!22-0.22 MCC blue!991.0 blue!700.71 blue!280.29 blue!620.63 blue!980.99
red!22-0.23 blue!920.92 red!34-0.34 blue!990.99 red!23-0.24 blue!991.0
blue!710.71 blue!770.78 blue!1001.0 red!35-0.36 red!55-0.55 red!44-0.44
red!71-0.71 blue!991.0 red!30-0.31 PBD red!34-0.34 red!26-0.26 red!4-0.04
red!15-0.16 red!27-0.28 blue!20.02 red!35-0.36 blue!170.18 red!30-0.31
blue!20.03 red!34-0.34 red!26-0.27 red!49-0.49 red!35-0.36 blue!1001.0
blue!150.16 blue!970.97 blue!280.29 red!34-0.34 blue!50.05 HD95 red!54-0.55
red!37-0.38 red!16-0.16 red!37-0.38 red!55-0.55 blue!170.18 red!48-0.48
blue!190.19 red!56-0.56 blue!180.18 red!54-0.55 red!38-0.38 red!51-0.51
red!55-0.55 blue!150.16 blue!1001.0 blue!250.25 blue!890.89 red!55-0.55
blue!130.14 MHD red!43-0.43 red!33-0.34 red!4-0.04 red!21-0.21 red!37-0.37
blue!20.03 red!43-0.43 blue!140.14 red!39-0.4 blue!30.03 red!43-0.43
red!33-0.34 red!58-0.58 red!44-0.44 blue!970.97 blue!250.25 blue!1001.0
blue!390.4 red!43-0.43 blue!50.06 ASSD red!70-0.71 red!47-0.47 red!27-0.27
red!47-0.47 red!69-0.7 blue!280.29 red!65-0.65 blue!280.28 red!71-0.71
blue!290.3 red!70-0.71 red!47-0.47 red!71-0.71 red!71-0.71 blue!280.29
blue!890.89 blue!390.4 blue!1001.0 red!71-0.71 blue!190.2 ARI blue!991.0
blue!700.7 blue!290.29 blue!610.62 blue!980.98 red!22-0.23 blue!940.94
red!34-0.35 blue!980.99 red!23-0.24 blue!991.0 blue!700.71 blue!780.78
blue!991.0 red!34-0.34 red!55-0.55 red!43-0.43 red!71-0.71 blue!1001.0
red!30-0.3 OASSD red!30-0.3 red!20-0.2 red!22-0.22 red!21-0.22 red!30-0.31
blue!270.28 red!26-0.26 blue!140.15 red!32-0.32 blue!280.28 red!30-0.3
red!20-0.2 red!22-0.22 red!30-0.31 blue!50.05 blue!130.14 blue!50.06
blue!190.2 red!30-0.3 blue!1001.0

###  5.4 Representative models selection

Only one model can be provided in the software solutions for each tumor type,
and the best model selection was done empirically according to the following
criterion: size of the validation or test set, average Dice score and patient-
wise F1-score performances. The exhaustive list of chosen models is the
following: the model trained for fold $0$ was selected for the glioblastomas,
the model trained for fold $3$ was selected for the lower grade gliomas, for
the meningiomas the model trained for fold $2$ was selected, and finally for
the metastases the model trained for fold $2$ was selected.

###  5.5 Speed study

A comparison in processing speed regarding pure tumor segmentation and
complete generation of standardized reports is provided in Table 10, when
using the Raidionics software with CPU support. The high-end computer is the
computer used for training the models, whereas the mid-end computer is a
Windows laptop with an Intel Core Processor ($i7@2.20$GHz), and $16$GB of RAM.

For the smallest MRI volumes on average, $17$ seconds are needed to perform
tumor segmentation whereas $4.5$ minutes are required to generate the complete
standardized report with the high-end computer. Unsurprisingly, the larger the
MRI volume the more time required to perform the different processing
operations (cf. Section S3, Sup. Mat.). For the largest MRI volumes overall,
$59$ seconds are needed to perform tumor segmentation whereas $15$ minutes are
required to generate the complete standardized report. When using the mid-end
laptop, overall runtime is increased by $1.5$ times for the different MRI
volume sizes. On average, $9$ minutes are necessary to generate the
standardized report for MRI volumes of reasonable quality.

Table 10: Segmentation (Segm.) and standardized reporting (SR) execution
speeds for each tumor subtype, using our Raidionics software.

High-end computer (Desktop) Mid-end computer (Laptop) Dimensions (voxels)
Segm. (s) SR (m) Segm. (s) SR (m) LGG $394\times 394\times 80$ $16.69\pm
0.426$ $04.50\pm 0.09$ $28.69\pm 0.577$ $07.32\pm 0.07$ Meningioma $256\times
256\times 170$ $17.21\pm 0.425$ $05.48\pm 0.12$ $31.41\pm 0.862$ $09.09\pm
0.32$ Glioblastoma $320\times 320\times 220$ $21.99\pm 0.177$ $05.89\pm 0.03$
$33.65\pm 1.429$ $09.06\pm 0.24$ Metastasis $560\times 560\times 561$
$59.06\pm 1.454$ $15.35\pm 0.41$ $98.54\pm 2.171$ $24.06\pm 0.93$

##  6 Discussion

In this study, we have investigated the segmentation of a range of common main
brain tumor types in 3D preoperative MR scans using a variant of the Attention
U-Net architecture. We have conducted experiments to assess the performances
of each trained model using close to $30$ metrics, and developed two software
solutions for end-users to freely benefit from our segmentation models and
standardized clinical reports. The main contributions are the high
performances of the models, on-par with performances reported in the
literature for the glioblastomas, with illustrated robustness and ability to
generalize thanks to the multiple and widespread data sources. In addition,
the two proposed open-access and open-source software solutions include our
best models, together with a RADS for computing tumor characteristics. This is
the first open RADS solution which supports all major brain tumor types. The
software is user-friendly, requiring only a few clicks and no programming to
use, making it easily accessible for clinicians. The overall limitations are
those already known for deep learning approaches whereby a higher amount of
patients or data sources would improve the ability to generalize, boost
segmentation performances, and increase the immunity toward rare tumor
expressions. The employed architecture also struggles with smaller tumors
given the large downsampling to feed the entire 3D MR scan in the network,
hence the need for a better design combining local and global features either
through multiple steps or ensembling.

The architecture and training strategy used in this study were identical to
our previously published work considering that the intent was not to directly
make advances on the segmentation task. Nevertheless, the stability and
robustness to train efficient models had been documented, alongside
performance comparison to another well-known architecture (e.g., nnU-Net
[20]), thus not precluding its use to train models for other brain tumor
types. Aside from evident outliers in the datasets, where either tumors with
partial or missing contrast uptake or suboptimal manual annotations were
identified, the major pitfall from using the AGU-Net architecture lies in its
struggle to segment equally satisfactorily small tumor pieces with a volume
below $2$ ml. Overall, the glioblastoma model is expected to be the most
robust and able to generalize since patient data from $15$ different sources
was used. For other models trained on data from much fewer hospitals, with an
expected limited variability in MR scan quality, their robustness is likely to
be inferior. While larger datasets is often correlated with improved
segmentation performance, the metastasis model is the best performing with the
lowest amount of patients included. The relative easiness of the task from a
clear demarcation of the tumor from surrounding normal tissue in contrast-
enhanced T1-weighted volumes, and the potentially low variance in tumor
characteristics with patient data coming from two hospitals only, can explain
the results. Additionally, the metastasis model has been trained by transfer-
learning using as input the second best performing glioblastoma model where
the most data was used, which may have been the compelling factor. Lower-grade
gliomas represent the most difficult type to manually segment since tumors are
diffuse and infiltrating with an average volume in FLAIR sequences a lot
higher than in T1 sequences for the other tumor types, and as such overall
worse performances were expected.

The in-depth assessment of a larger pool of metrics allowed us to identify
redundancy and uniqueness, and proved that the Dice score is overall quite
robust and indicative of expected performance. However, the sole use of Dice
score cannot cover all aspects of a model performance, and spatial distance-
based metrics (e.g., HD95 and MHD) are suggested to be used in conjunction as
providing values uncorrelated to Dice. In addition, some metrics were
identified to be more sensitive to outliers and are as such powerful to either
assess the ability to generalize of a model across data acquired on different
scanners from multiple sources, or quickly identify potential issues in a
large body of data. Finally, and depending on the nature of the patients
included in one’s study and amount of satellite tumors, specific object-wise
metrics are imperative to use (e.g., OASSD). Only a combination of various
metrics computed either voxel-wise, patient-wise, or instance-wise can give
the full picture of a model’s performance. Unfortunately, interpreting and
comparing sets of metrics can prove challenging and as such further
investigations regarding their merging into a unique informative and coherent
score are fundamental (e.g., Roza [62]). Furthermore, an inadequacy lies in
the nature of the different metrics whereby some can be computed across all
segmentations generated by a trained model, whereas others are exclusively
eligible on true positive cases, i.e., when the model has correctly segmented
some extent of the tumor. For models generating perfect patient-wise recall,
all metrics will be eligible for every segmentation. However, in this field of
research and as of today, no trained model can fulfill this requirement due to
the substantially large inter-patient variability. Ideally, the identification
of relevant metrics, bringing unique information for interpreting the results,
should not be confined to the validation studies. More metrics should be
considered to be a part of the loss function computation during training of
neural network architectures. Attempts have been made towards using the
Hausdorff distance as loss function, but a direct minimization is challenging
from an optimization viewpoint. For example, approximation of Hausdorff
distance based on distance transforms, on morphological operations, or with
circular and spherical kernels showed potential for medical image segmentation
[63]. In general, a careful mix between losses (e.g., Dice, cross-entropy, and
HD95) is challenging to achieve and adaptive strategies might be required to
avoid reaching a local minimum where overall segmentation performance may
suffer [64].

As a current trend in the community, inference code and trained segmentation
models are often at best available on GitHub repositories. As a consequence,
only engineers, or people with some extent of knowledge in machine learning
and programming, can benefit from such research advances. Besides, the
research focus is heavily angled towards gliomas, due to the BraTS challenge
influence, whereby segmentation models are expected to yield superior
performance than for meningiomas and metastases. By developing and giving free
and unrestricted access to our two proposed software solutions, we hope to
facilitate more research on all brain tumor types. Willing research institutes
have the opportunity to generate private annotated datasets at a faster pace
than through fully manual labour by exploiting our trained models. Having made
all source code available on GitHub, as customarily done, we made the effort
to further make stand-alone solutions with easy-to-use GUIs. Hopefully,
clinicians and other non-programming end-users should feel more comfortable
manipulating such tools, available across the three major operating systems
and necessitating only a computer with average hardware specifications. For
the generation of standardized clinical reports, the computation of tumor
characteristics relies heavily on the quality of the automatic segmentation,
occasional mishaps are expected as models are not perfect and can omit the
tumor. Therefore, manual inputs will be required sporadically to correct the
tumor segmentation. Over time, new and better models will be generated and
made available seamlessly into the two software through regular updates.

In the future, an approach incorporating a set of metrics and converting them
into one final score would be highly desirable (e.g., Roza). Not only would it
help to automatically select the best model from a k-fold validation study
from one unique score, but a proper assessment and ranking across multiple
methods would be enabled. With all preoperative brain tumor types available
for segmentation and reporting in our software, a key missing component is the
automatic tumor type classification to supplement manual user input.
Concurrently, the variety and amount of tumor characteristics to compute
should be extended, considering more type-specific features similar to the
resection index for glioblastomas. Alternatively, bringing a similar focus on
post-operative segmentation of residual tumor is of great interest to both
assess the quality of the surgery and refine the estimated patient outcome.
The generation of a complete post-operative standardized clinical report would
also be permitted with new features such as the extent of resection.
Otherwise, intensifying the gathering of patient data from more widespread
hospital centers and a larger array of MRI scanners is always of importance.
The inclusion of more than one MR sequence per patient as segmentation input
has the potential to boost overall performance, but at the same time might
reduce models’ potency as not always routinely available across all centers
worldwide.

##  7 Conclusion

Efficient and robust segmentation models have been trained on pre-operative MR
scans for the four main brain tumor types: glioblastoma, lower grade glioma,
meningioma, and metastasis. In-depth performance assessment allowed to
identify the most relevant metrics from a large panel, computed either voxel-
wise, patient-wise, or instance-wise. Trained models and standardized
reporting have been made publicly available and packaged into a stand-alone
software and a 3D Slicer plugin to enable effortless widespread use.

### Disclosures

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.
Informed consent was obtained from all individual participants included in the
study.

### Acknowledgments

Data were processed in digital labs at HUNT Cloud, Norwegian University of
Science and Technology, Trondheim, Norway.

### Author Contributions

Funding acquisition, I.R., O.S., P.C.D.W.H., K.E.E., and A.S.J.; Data
curation, A.S.J., K.E.E., V.K., I.K., D.B., H.A., F.B., L.B., M.S.B., M.C.N.,
J.F., S.H.-J., A.J.S.I., B.K., A.K., E.M., D.M.J.M., P.A.R., M.R., T.S.,
W.A.v.d.B., M.W., G.W., O.S. and P.C.D.W.H.; Conceptualization, D.B., A.P.,
I.R., O.S. and P.C.D.W.H.; Methodology, D.B.; Software, D.B. and A.P.;
Validation, D.B. and A.P.; Visualization, D.B.; Supervision, I.R., O.S. and
P.C.D.W.H.; Project administration, I.R., O.S., and P.C.D.W.H.;
Writing—original draft, D.B., A.P., I.R., O.S., A.S.J., K.E.E., and
P.C.D.W.H.; Writing—review and editing, H.A., F.B., L.B., M.S.B., M.C.N.,
J.F., S.H.-J., A.J.S.I., B.K., A.K., E.M., D.M.J.M., P.A.R., M.R., T.S.,
W.A.v.d.B., M.W., G.W., M.G.W. and A.H.Z.

### Funding

This work was funded by the Norwegian National Advisory Unit for Ultrasound
and Image-Guided Therapy (usigt.org); South-Eastern Norway Regional Health
Authority; Contract grant numbers: 2016102 and 2013069; Contract grant
sponsor: Research Council of Norway; Contract grant number: 261984; Contract
grant sponsor: Norwegian Cancer Society; Contract grant numbers: 6817564 and
3434180; Contract grant sponsor: European Research Council under the European
Union’s Horizon 2020 Program; Contract grant number: 758657-ImPRESS; an
unrestricted grant of Stichting Hanarth fonds, “Machine learning for better
neurosurgical decisions in patients with glioblastoma”; a grant for public-
private partnerships (Amsterdam UMC PPP-grant) sponsored by the Dutch
government (Ministry of Economic Affairs) through the Rijksdienst voor
Ondernemend Nederland (RVO) and Topsector Life Sciences and Health (LSH),
“Picturing predictions for patients with brain tumors”; a grant from the
Innovative Medical Devices Initiative program, project number
10-10400-96-14003; The Netherlands Organisation for Scientific Research (NWO),
2020.027; a grant from the Dutch Cancer Society, VU2014-7113; the Anita
Veldman foundation, CCA2018-2-17.

## References

  * [1] Julia Day, David C Gillespie, Alasdair G Rooney, Helen J Bulbeck, Karolis Zienius, Florien Boele, and Robin Grant.  Neurocognitive deficits and neurocognitive rehabilitation in adult brain tumors.  Current treatment options in neurology, 18(5):1–16, 2016.
  * [2] David N Louis, Arie Perry, Pieter Wesseling, Daniel J Brat, Ian A Cree, Dominique Figarella-Branger, Cynthia Hawkins, HK Ng, Stefan M Pfister, Guido Reifenberger, et al.  The 2021 who classification of tumors of the central nervous system: a summary.  Neuro-oncology, 23(8):1231–1251, 2021.
  * [3] Lisa M DeAngelis.  Brain tumors.  New England journal of medicine, 344(2):114–123, 2001.
  * [4] James L Fisher, Judith A Schwartzbaum, Margaret Wrensch, and Joseph L Wiemels.  Epidemiology of brain tumors.  Neurologic clinics, 25(4):867–890, 2007.
  * [5] Sarah Lapointe, Arie Perry, and Nicholas A Butowski.  Primary brain tumours in adults.  The Lancet, 392(10145):432–446, 2018.
  * [6] Philipp Kickingereder, Sina Burth, Antje Wick, Michael Götz, Oliver Eidel, Heinz-Peter Schlemmer, Klaus H Maier-Hein, Wolfgang Wick, Martin Bendszus, Alexander Radbruch, et al.  Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models.  Radiology, 280(3):880–889, 2016.
  * [7] Raymond Sawaya, Maarouf Hammoud, Derek Schoppa, Kenneth R Hess, Shu Z Wu, Wei-Ming Shi, and David M WiIdrick.  Neurosurgical outcomes in a modern series of 400 craniotomies for treatment of parenchymal tumors.  Neurosurgery, 42(5):1044–1055, 1998.
  * [8] Tiit Mathiesen, Inti Peredo, and Stefan Lönn.  Two-year survival of low-grade and high-grade glioma patients using data from the swedish cancer registry.  Acta neurochirurgica, 153(3):467–471, 2011.
  * [9] Pascal O Zinn, Rivka R Colen, Ekkehard M Kasper, and Jan-Karl Burkhardt.  Extent of resection and radiotherapy in gbm: A 1973 to 2007 surveillance, epidemiology and end results analysis of 21,783 patients.  International journal of oncology, 42(3):929–934, 2013.
  * [10] Jeffrey C Weinreb, Jelle O Barentsz, Peter L Choyke, Francois Cornud, Masoom A Haider, Katarzyna J Macura, Daniel Margolis, Mitchell D Schnall, Faina Shtern, Clare M Tempany, et al.  Pi-rads prostate imaging–reporting and data system: 2015, version 2.  European urology, 69(1):16–40, 2016.
  * [11] Spencer C Dyer, Brian J Bartholmai, and Chi Wan Koo.  Implications of the updated lung ct screening reporting and data system (lung-rads version 1.1) for lung cancer screening.  Journal of Thoracic Disease, 12(11):6966, 2020.
  * [12] Benjamin M Ellingson, Martin Bendszus, Jerrold Boxerman, Daniel Barboriak, Bradley J Erickson, Marion Smits, Sarah J Nelson, Elizabeth Gerstner, Brian Alexander, Gregory Goldmacher, et al.  Consensus recommendations for a standardized brain tumor imaging protocol in clinical trials.  Neuro-oncology, 17(9):1188–1198, 2015.
  * [13] Ivar Kommers, David Bouget, André Pedersen, Roelant S Eijgelaar, Hilko Ardon, Frederik Barkhof, Lorenzo Bello, Mitchel S Berger, Marco Conti Nibali, Julia Furtner, et al.  Glioblastoma surgery imaging—reporting and data system: Standardized reporting of tumor volume, location, and resectability based on automated segmentations.  Cancers, 13(12):2854, 2021.
  * [14] Elisabetta Binaghi, Valentina Pedoia, and Sergio Balbi.  Collection and fuzzy estimation of truth labels in glial tumour segmentation studies.  Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 4(3-4):214–228, 2016.
  * [15] Erik Magnus Berntsen, Anne Line Stensjøen, Maren Staurset Langlo, Solveig Quam Simonsen, Pål Christensen, Viggo Andreas Moholdt, and Ole Solheim.  Volumetric segmentation of glioblastoma progression compared to bidimensional products and clinical radiological reports.  Acta Neurochirurgica, 162(2):379–387, 2020.
  * [16] Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  IEEE transactions on pattern analysis and machine intelligence, 2021\.
  * [17] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [18] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  Scientific data, 4(1):1–13, 2017.
  * [19] Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al.  The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.  arXiv preprint arXiv:2107.02314, 2021.
  * [20] Fabian Isensee, Jens Petersen, Andre Klein, David Zimmerer, Paul F Jaeger, Simon Kohl, Jakob Wasserthal, Gregor Koehler, Tobias Norajitra, Sebastian Wirkert, et al.  nnu-net: Self-adapting framework for u-net-based medical image segmentation.  arXiv preprint arXiv:1809.10486, 2018.
  * [21] Huan Minh Luu and Sung-Hong Park.  Extending nn-unet for brain tumor segmentation.  arXiv preprint arXiv:2112.04653, 2021.
  * [22] Arti Tiwari, Shilpa Srivastava, and Millie Pant.  Brain tumor segmentation and classification from magnetic resonance images: Review of selected methods from 2014 to 2019.  Pattern Recognition Letters, 131:244–260, 2020.
  * [23] Sérgio Pereira, Adriano Pinto, Victor Alves, and Carlos A Silva.  Brain tumor segmentation using convolutional neural networks in mri images.  IEEE transactions on medical imaging, 35(5):1240–1251, 2016.
  * [24] Endre Grøvik, Darvin Yi, Michael Iv, Elizabeth Tong, Daniel Rubin, and Greg Zaharchuk.  Deep learning enables automatic detection and segmentation of brain metastases on multisequence mri.  Journal of Magnetic Resonance Imaging, 51(1):175–182, 2020.
  * [25] Endre Grøvik, Darvin Yi, Michael Iv, Elizabeth Tong, Line Brennhaug Nilsen, Anna Latysheva, Cathrine Saxhaug, Kari Dolven Jacobsen, Åslaug Helland, Kyrre Eeg Emblem, et al.  Handling missing mri sequences in deep learning segmentation of brain metastases: a multicenter study.  NPJ digital medicine, 4(1):1–7, 2021.
  * [26] Konstantinos Kamnitsas, Enzo Ferrante, Sarah Parisot, Christian Ledig, Aditya V Nori, Antonio Criminisi, Daniel Rueckert, and Ben Glocker.  Deepmedic for brain tumor segmentation.  In International workshop on Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries, pages 138–149. Springer, 2016\.
  * [27] Yan Liu, Strahinja Stojadinovic, Brian Hrycushko, Zabi Wardak, Steven Lau, Weiguo Lu, Yulong Yan, Steve B Jiang, Xin Zhen, Robert Timmerman, et al.  A deep convolutional neural network-based automatic delineation strategy for multiple brain metastases stereotactic radiosurgery.  PloS one, 12(10):e0185844, 2017.
  * [28] Odelin Charron, Alex Lallement, Delphine Jarnet, Vincent Noblet, Jean-Baptiste Clavier, and Philippe Meyer.  Automatic detection and segmentation of brain metastases on multimodal mr images with a deep convolutional neural network.  Computers in biology and medicine, 95:43–54, 2018.
  * [29] Eleftherios Neromyliotis, Theodosis Kalamatianos, Athanasios Paschalis, Spyridon Komaitis, Konstantinos N Fountas, Eftychia Z Kapsalaki, George Stranjalis, and Ioannis Tsougos.  Machine learning in meningioma mri: past to present. a narrative review.  Journal of Magnetic Resonance Imaging, 55(1):48–60, 2022.
  * [30] Kai Roman Laukamp, Frank Thiele, Georgy Shakirin, David Zopfs, Andrea Faymonville, Marco Timmer, David Maintz, Michael Perkuhn, and Jan Borggrefe.  Fully automated detection and segmentation of meningiomas using deep learning on routine multiparametric mri.  European radiology, 29(1):124–132, 2019.
  * [31] Kai Roman Laukamp, Lenhard Pennig, Frank Thiele, Robert Reimer, Lukas Görtz, Georgy Shakirin, David Zopfs, Marco Timmer, Michael Perkuhn, and Jan Borggrefe.  Automated meningioma segmentation in multiparametric mri.  Clinical Neuroradiology, pages 1–10, 2020.
  * [32] David Bouget, André Pedersen, Sayied Abdol Mohieb Hosainey, Ole Solheim, and Ingerid Reinertsen.  Meningioma segmentation in t1-weighted mri leveraging global context and attention mechanisms.  arXiv preprint arXiv:2101.07715, 2021.
  * [33] The MONAI Consortium.  Project monai, December 2020.
  * [34] Annika Reinke, Matthias Eisenmann, Minu D Tizabi, Carole H Sudre, Tim Rädsch, Michela Antonelli, Tal Arbel, Spyridon Bakas, M Jorge Cardoso, Veronika Cheplygina, et al.  Common limitations of image processing metrics: A picture story.  arXiv preprint arXiv:2104.05642, 2021.
  * [35] Abdel Aziz Taha and Allan Hanbury.  Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool.  BMC medical imaging, 15(1):29, 2015.
  * [36] Brent D Weinberg, Ashwani Gore, Hui-Kuo G Shu, Jeffrey J Olson, Richard Duszak, Alfredo D Voloschin, and Michael J Hoch.  Management-based structured reporting of posttreatment glioma response with the brain tumor reporting and data system.  Journal of the American College of Radiology, 15(5):767–771, 2018\.
  * [37] T Huber, G Alber, S Bette, T Boeckh-Behrens, J Gempt, F Ringel, E Alberts, C Zimmer, and JS Bauer.  Reliability of semi-automated segmentations in glioblastoma.  Clinical neuroradiology, 27(2):153–161, 2017.
  * [38] Vladimir Vezhnevets and Vadim Konouchine.  Growcut: Interactive multi-label nd image segmentation by cellular automata.  In proc. of Graphicon, volume 1, pages 150–156. Citeseer, 2005\.
  * [39] David Bouget, André Pedersen, Sayied Abdol Mohieb Hosainey, Johanna Vanel, Ole Solheim, and Ingerid Reinertsen.  Fast meningioma segmentation in t1-weighted magnetic resonance imaging volumes using a lightweight 3d deep learning architecture.  Journal of Medical Imaging, 8(2):024002, 2021.
  * [40] David Bouget, Roelant S Eijgelaar, André Pedersen, Ivar Kommers, Hilko Ardon, Frederik Barkhof, Lorenzo Bello, Mitchel S Berger, Marco Conti Nibali, Julia Furtner, et al.  Glioblastoma surgery imaging–reporting and data system: Validation and performance of the automated segmentation task.  Cancers, 13(18):4674, 2021.
  * [41] Vladimir S Fonov, Alan C Evans, Robert C McKinstry, C Robert Almli, and DL Collins.  Unbiased nonlinear average age-appropriate brain templates from birth to adulthood.  NeuroImage, (47):S102, 2009.
  * [42] Andriy Fedorov, Reinhard Beichel, Jayashree Kalpathy-Cramer, Julien Finet, Jean-Christophe Fillion-Robin, Sonia Pujol, Christian Bauer, Dominique Jennings, Fiona Fennessy, Milan Sonka, et al.  3d slicer as an image computing platform for the quantitative imaging network.  Magnetic resonance imaging, 30(9):1323–1341, 2012.
  * [43] Alireza Mehrtash, Mehran Pesteie, Jorden Hetherington, Peter A Behringer, Tina Kapur, William M Wells III, Robert Rohling, Andriy Fedorov, and Purang Abolmaesumi.  Deepinfer: Open-source deep learning deployment toolkit for image-guided therapy.  In Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling, volume 10135, pages 410–416. SPIE, 2017.
  * [44] Lee R Dice.  Measures of the amount of ecologic association between species.  Ecology, 26(3):297–302, 1945.
  * [45] Paul Jaccard.  The distribution of the flora in the alpine zone. 1.  New phytologist, 11(2):37–50, 1912.
  * [46] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [47] Rubén Cárdenes, Rodrigo de Luis-Garcia, and Meritxell Bach-Cuadra.  A multidimensional segmentation evaluation for medical image data.  Computer methods and programs in biomedicine, 96(2):108–124, 2009\.
  * [48] Daniel B Russakoff, Carlo Tomasi, Torsten Rohlfing, and Calvin R Maurer.  Image similarity using mutual information of regions.  In European Conference on Computer Vision, pages 596–607. Springer, 2004.
  * [49] Marina Meilă.  Comparing clusterings by the variation of information.  In Learning theory and kernel machines, pages 173–187. Springer, 2003.
  * [50] Jacob Cohen.  A coefficient of agreement for nominal scales.  Educational and psychological measurement, 20(1):37–46, 1960.
  * [51] Andrew P Bradley.  The use of the area under the roc curve in the evaluation of machine learning algorithms.  Pattern recognition, 30(7):1145–1159, 1997.
  * [52] Pierre Baldi, Søren Brunak, Yves Chauvin, Claus AF Andersen, and Henrik Nielsen.  Assessing the accuracy of prediction algorithms for classification: an overview.  Bioinformatics, 16(5):412–424, 2000.
  * [53] Guido Gerig, Matthieu Jomier, and Miranda Chakos.  Valmet: A new validation tool for assessing and improving 3d object segmentation.  In International conference on medical image computing and computer-assisted intervention, pages 516–523. Springer, 2001.
  * [54] Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge.  Comparing images using the hausdorff distance.  IEEE Transactions on pattern analysis and machine intelligence, 15(9):850–863, 1993.
  * [55] Goeffrey J McLachlan.  Mahalanobis distance.  Resonance, 4(6):20–26, 1999.
  * [56] Nancy Chinchor and Beth M Sundheim.  Muc-5 evaluation metrics.  In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993, 1993.
  * [57] Lawrence Hubert and Phipps Arabie.  Comparing partitions.  Journal of classification, 2(1):193–218, 1985.
  * [58] Peter R Killeen.  An alternative to null-hypothesis significance tests.  Psychological science, 16(5):345–353, 2005.
  * [59] Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft, Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfeiffer, Ben Cook, Ismael Fernández, François-Michel De Rainville, Chi-Hung Weng, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al.  imgaug.  https://github.com/aleju/imgaug, 2020.  Online; accessed 01-Feb-2020.
  * [60] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [61] Oskar Maier, Alex Rothberg, Pradeep Reddy Raamana, Rémi Bèges, Fabian Isensee, Michael Ahern, mamrehn, VincentXWD, and Jay Joshi.  loli/medpy: Medpy 0.4.0, February 2019.
  * [62] Mesut Melek and Negin Melek.  Roza: a new and comprehensive metric for evaluating classification systems.  Computer Methods in Biomechanics and Biomedical Engineering, pages 1–13, 2021.
  * [63] Davood Karimi and Septimiu E Salcudean.  Reducing the hausdorff distance in medical image segmentation with convolutional neural networks.  IEEE Transactions on medical imaging, 39(2):499–513, 2019.
  * [64] A Ali Heydari, Craig A Thompson, and Asif Mehmood.  Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions.  arXiv preprint arXiv:1912.12355, 2019.