11institutetext: Department of Electrical and Systems Engineering, Washington
University in St. Louis, St. Louis, MO 63130, USA 22institutetext:
Mallinckrodt Institute of Radiology, Washington University School of Medicine,
St. Louis, MO 63110, USA 33institutetext: Department of Computer Science and
Engineering, Washington University in St. Louis, St. Louis, MO 63130, MO, USA
44institutetext: Department of Radiation Oncology, Division of Radiation
Oncology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe
Boulevard, Houston, TX 77027, USA 55institutetext: Institute for Informatics,
Washington University School of Medicine, St. Louis, MO 63110, USA
✉ Corresponding author: 55email<EMAIL_ADDRESS>
# Integrative Imaging Informatics for Cancer Research: Workflow Automation for
Neuro-oncology (I3CR-WANO)

Satrajit Chakrabarty 11 Syed Amaan Abidi 22 Mina Mousa 22 Mahati Mokkarala 22
Isabelle Hren 33 Divya Yadav 44 Matthew Kelsey 22 Pamela LaMontagne 22 John
Wood 44 Michael Adams 44 Yuzhuo Su 44 Sherry Thorpe 44 Caroline Chung 44
Aristeidis Sotiras 2255 Daniel S. Marcus 22

###### Abstract

Efforts to utilize growing volumes of clinical imaging data to generate tumor
evaluations continue to require significant manual data wrangling owing to the
data heterogeneity. Here, we propose an artificial intelligence-based solution
for the aggregation and processing of multisequence neuro-oncology MRI data to
extract quantitative tumor measurements. Our end-to-end framework i)
classifies MRI sequences using an ensemble classifier, ii) preprocesses the
data in a reproducible manner, iii) delineates tumor tissue subtypes using
convolutional neural networks, and iv) extracts diverse radiomic features.
Moreover, it is robust to missing sequences and adopts an expert-in-the-loop
approach, where the segmentation results may be manually refined by
radiologists. Following the implementation of the framework in Docker
containers, it was applied to two retrospective glioma datasets collected from
the Washington University School of Medicine (WUSM; n = 384) and the M.D.
Anderson Cancer Center (MDA; n = 30) comprising preoperative MRI scans from
patients with pathologically confirmed gliomas. The scan-type classifier
yielded an accuracy of over 99%, correctly identifying sequences from 380/384
and 30/30 sessions from the WUSM and MDA datasets, respectively. Segmentation
performance was quantified using the Dice Similarity Coefficient between the
predicted and expert-refined tumor masks. Mean Dice scores were 0.882 (±0.244)
and 0.977 (±0.04) for whole tumor segmentation for WUSM and MDA, respectively.
This streamlined framework automatically curated, processed, and segmented raw
MRI data of patients with varying grades of gliomas, enabling the curation of
large-scale neuro-oncology datasets and demonstrating a high potential for
integration as an assistive tool in clinical practice.

###### Keywords:

data curation deep learning natural language processing scan-type
classification segmentation glioma neuro-oncology DICOM.

##  1 Introduction

Advancements in in vivo neuroimaging have resulted in large volumes of
clinically acquired, high-dimensional MRI datasets. Quantitative analysis of
multimodal imaging data in clinical settings is essential for assessing tumor
burden and treatment response in an objective and noninvasive fashion. The
scientific community has sought to capitalize on recent advancements of
artificial intelligence (AI)-assisted approaches in neuro-oncology [1] to
develop AI-driven software packages [2, 3, 4] for automating neuro-oncology
workflows.

However, the distillation of high-dimensional and multimodal imaging
information into meaningful quantitative information remains an ongoing
pursuit [5]. A major challenge is that AI-driven tools typically require
manually curated, preprocessed datasets comprising scans of specific
sequence(s). Manual curation and preparation of imaging data are time-
consuming and error-prone for multiple reasons, including non-standardized
naming conventions of scans across different manufacturers or acquisition
protocols [6], same series descriptions of scans irrespective of acquisition
type, and missing metadata [7]. This challenge has been magnified by the
growing number of diverse and complementary image acquisition protocols that
make parsing these datasets using automated methods or heuristics extremely
difficult. Nonetheless, limited efforts have been invested in automated tools
for scan-type classification and data curation [8, 9]. Importantly, these have
not been integrated into existing software tools. These practical challenges
have limited the widespread adoption of existing software packages for
heterogeneous clinical imaging data.

To centralize these efforts and expedite the translation of state-of-the-art
AI models from research to clinical practice, we have developed an end-to-end
AI-driven framework called Integrative Imaging Informatics for Cancer
Research: Workflow Automation for Neuro-oncology (I3CR-WANO), which classifies
MRI sequences using an ensemble of natural language processing (NLP) and
convolutional neural network (CNN) models, preprocesses the data in a
reproducible manner, and segments tumor tissue subtypes using CNNs, enabling
the extraction of radiomic features. Additionally, I3CR-WANO is robust to
missing sequences and adopts an expert-in-the-loop approach, where the
segmentation results may be manually refined by radiologists. In this study,
we implemented this framework for low- and high-grade gliomas using pre-
contrast T1-weighted, (T1WI), post-contrast T1WI (Gd-T1WI), T2-weighted
(T2WI), and fluid-attenuated inversion recovery (FLAIR) sequences. All
framework components were packaged as Docker containers to ensure
reproducibility across platforms and to facilitate dissemination and
deployment. For greater flexibility, the core components of the framework were
implemented as independent modules instead of as a single monolithic
structure.

##  2 Materials and methods

###  2.1 End-to-end framework

I3CR-WANO (Figure 1A) consists of the following stages: I) image curation and
preprocessing, II) segmentation, III) expert refinement of the tumor mask and
segmentation evaluation, and IV) post-processing and visualization of the
segmentation mask.

Figure 1: (A) End-to-end processing framework of I3CR-WANO consisting of the
following stages: I) data curation and image pre-processing, II) segmentation,
III) expert refinement of tumor mask and segmentation evaluation, and IV)
post-processing of the tumor mask. (B) Outputs of framework at different
steps: unprocessed pre- and post-contrast T1-weighted (T1WI, Gd-T1WI),
T2-weighted (T2WI), and FLAIR sequences (step 1), same sequences after pre-
processing (step 3), tumor segmentation predicted by convolutional neural
network (step 4), and tumor segmentation after refinement (step 5). White
arrows in tumor segmentation mask image point to errors, which were corrected
during refinement.

####  2.1.1 Curation and pre-processing

The first step in the framework takes MRI data in the Digital Imaging and
Communications in Medicine (DICOM) format as input and identifies the
sequences that can be used in downstream segmentation and feature extraction
tasks. The scan-type classifier adopts a cascade architecture. The first stage
of the cascade is based on an NLP classifier (Classifier1) [10], whereas the
second stage is based on a CNN classifier (Classifier2) [9]. For both
classifiers, previously published pretrained models were used [9, 10].
Classifier1 uses information regarding the DICOM series description (i.e., tag
[0008, 103E]) and the number of instances per series to classify every scan
into one of two classes: segmentable or non-segmentable. Subsequently,
Classifier2 performs a more granular classification of these potentially
segmentable scans into T1WI, Gd-T1WI, T2WI, FLAIR, and non-segmentable
classes. Additionally, the orientation of each scan (i.e., axial, coronal,
sagittal) is determined by leveraging the DICOM MR acquisition type (tag
[0018,0023]) and image orientation (patient) (tag [0020,0037]) information.

In the event of multiple occurrences of T1WI, Gd-T1WI, T2WI, or FLAIR
sequences, we prioritize i) axial scans over sagittal and coronal scans and
ii) scans with a higher number of instances. Based on the absence of certain
DICOM tags or sequence types, a particular scan or an entire session is
excluded from the subsequent processing (Supplementary S1.1). In all other
cases, the curation process results in a maximum of four scans (one each for
T1WI, Gd-T1WI, T2WI, and FLAIR sequences) that are used for downstream
processing (Figure 1B, “scans after step 1”).

I3CR-WANO comprises the following preprocessing functionalities: registration,
N4-bias correction, skull-stripping, and intensity normalization. During
registration, for every session, the scan with the highest number of instances
among those identified in the previous step is selected as the target scan for
co-registration. Then, all other scans are rigidly co-registered to the target
scan, followed by affine registration to a common anatomical atlas [11]. The
transformation matrix from patient-space to atlas-space (patient2atlasmat) is
stored and later used as described in Section 2.1.4. For all registration
steps, we used FMRIB’s Linear Image Registration Tool (FLIRT) [12, 13]. Next,
we perform N4 bias field correction and skull-stripping of the registered
sequences using the Robust Brain Extraction (ROBEX) [14] tool (Figure 1B,
“scans after step 3”). This is followed by an image intensity normalization
step, where intensities within the brain are normalized to zero mean and unit
variance after excluding intensities below the 5th and above the 95th
percentile.

####  2.1.2 Segmentation

In this step, the preprocessed scans are segmented using pretrained
segmentation models (Supplementary S1.2) to produce a multiclass tumor
segmentation mask (Figure 1B, “automatic segmentation”). The mask comprises
the edema (ED), non-enhancing/necrotic tumor core (NC), and enhancing tumor
(ET) classes. Additional outputs include the tumor core (TC) class, which is
created by combining the ET and NC classes, and the whole tumor (WT) class,
which is constructed by combining all classes.

To render our framework robust to missing sequences, we have trained
segmentation models on different combinatorial subsets of sequences (e.g.,
Gd-T1WI+T2WI, Gd-T1WI+FLAIR, only Gd-T1WI, etc.). Depending on the available
sequences, the segmentation module produces a multi-class segmentation mask
comprising NC, ET, ED classes (if at least a Gd-T1WI is available), a binary
WT segmentation mask (in the absence of Gd-T1WI but presence of T2WI and/or
FLAIR), or no mask (in the absence of Gd-T1WI, T2WI, and FLAIR).

####  2.1.3 Expert refinement and segmentation evaluation

Automated segmentation models are prone to errors [15] like occasional
erroneous labelling of vessels within the peritumoral edematous area,
periventricular white matter hyperintensities, choroid plexus, and areas of
Gd-T1WI bright blood products. To address this, we have adopted an expert-in-
the-loop approach, where the predicted segmentation mask is optionally sent to
radiologists for refinement (Figure 1B, “Expert-refined segmentation”).
Subsequently, the refined mask is compared with the predicted mask to evaluate
the performance of the segmentation model.

####  2.1.4 Post-processing

Once the segmentation mask is created, the mask is warped back to the patient
space by inverting the patient2atlasmat transformation matrix generated in the
registration step and applying it to the tumor mask using nearest-neighbor
interpolation. Subsequently, this mask is converted to a DICOM Segmentation
image object format using the itkimage2segimage command from the dcmqi [16]
tool. The mask can also be used for downstream processing such as the
extraction of quantitative features from the tumor mask. This is supported by
our framework using the PyRadiomics [17] tool, which can calculate first-order
statistics, 3D shape features, and texture features for every combination of
tumor class and input sequence (Supplementary S1.3).

###  2.2 Framework implementation and distribution

The source code, detailed documentation, pre-trained AI models as well as a
live demonstration of I3CR-WANO are made publicly available for non-commercial
use at https://github.com/satrajitgithub/NRG_AI_NeuroOnco_preproc and
https://github.com/satrajitgithub/NRG_AI_NeuroOnco_segment. In addition, to
ensure portability and easy deployment, the framework has been packaged into
Docker images available from DockerHub
(https://hub.docker.com/r/satrajit2012/nrg_ai_neuroonco_preproc and
https://hub.docker.com/r/satrajit2012/nrg_ai_neuroonco_segment). The
segmentation Docker image has been implemented using the NVIDIA GPU CLOUD
runtime to leverage NVIDIA Graphics processing units during test-time
inference. Besides Docker usage through the command-line interface, we also
provide a visual interface of the framework through its integration with the
open-source Extensible Neuroimaging Archive Toolkit (XNAT) informatics
platform [18]. XNAT equips the user with advanced functionalities, such as
launching a Docker on a batch of sessions (batch mode) or launching a sequence
of Docker commands (command orchestration). Additionally, through its
integrated Open Health Imaging Foundation (OHIF) Viewer [19], XNAT provides
the user with a powerful set of image visualization and annotation tools,
including the ability to natively refine tumor annotations, perform
measurements, and save those contours and segmentation objects back into XNAT.
Documentation for command-line Docker usage as well as XNAT usage are
available at
https://github.com/satrajitgithub/NRG_AI_NeuroOnco_preproc/tree/master/documentation.

###  2.3 Application on glioma datasets

I3CR-WANO was validated on two independent clinical datasets (Supplementary
Table S1) acquired from the retrospective health records of the Washington
University School of Medicine (WUSM; n = 384; median age 56 years, range 44 –
66 years, 154 females, 230 males) and the M.D. Anderson Cancer Center (MDA; n
= 30; median age 58 years, range 44 – 65 years, 15 females, 15 males). Data
collected from WUSM and MDA were obtained with Institutional Review Board
(IRB) approval and met the criteria for the general waiver of consent and
waiver of Health Insurance Portability and Accountability Act (HIPAA)
authorization. For both datasets, the only inclusion criterion was
pathologically confirmed glioma (grade II-IV) from preoperative patients with
no prior resection. To ensure the broad applicability of the framework to
heterogeneous clinical data, no exclusions were made based on the image
acquisition parameters, image quality, or glioma grade. The segmentation
models used in the framework were pre-trained on the Brain Tumor Segmentation
Challenge (BraTS) 2021 [15] dataset, publicly available from the Synapse
platform (https://www.synapse.org/#!Synapse:syn27046444/wiki/616992).

###  2.4 Statistical analyses

The performance of the scan-type classifier was quantified using the overall
accuracy, F1 score for each class, and confusion matrix showing the error
distribution across different classes. Failures during preprocessing were
identified through visual inspection. Segmentation performance was assessed
using the Dice Similarity Coefficient (DSC) metric for the WT, TC, and ET
classes. For the BraTS 2021 dataset, the DSC was calculated between the
predicted segmentations and the provided expert-annotated ground truths. For
both the WUSM and MDA datasets, the predicted tumor segmentations from the
framework were refined by experts, and these expert-refined tumor masks were
used as surrogate ground truths. Differences in DSC between groups with and
without certain sequences were calculated using the Welch’s t-test. For all
statistical tests, the threshold for statistical significance was set at
$P<.05$.

##  3 Results

Figure 2: (A), (C) Confusion matrix and F1-scores demonstrating the
performance of the scan-type classifier on the WUSM and MDA datasets,
respectively. (B), (D) Dispersion of Dice Similarity Coefficient values,
stratified by tumor grade, showing the agreement in terms of overlap between
the tumor segmentation masks predicted by convolutional neural network and the
masks after refinement by a radiologist for WUSM and MDA datasets,
respectively. NS = non-segmentable, WT = whole tumor, TC = tumor core, ET =
enhancing tumor, WUSM = Washington University School of Medicine, MDA = M.D.
Anderson Cancer Center.

###  3.1 Scan-type classification and pre-processing

The scan-type classifier yielded high overall accuracy (99.61%, 8835/8870
scans) across the five classes (Figure 2A). Precision values were above 0.99
for all classes except T1WI. The precision for T1WI (0.97, 557/572 scans)
showed a minor drop because 12 non-segmentable and 3 FLAIR scans were
misclassified as T1WI. Of the 12 non-segmentable scans, 10 were determined to
be non-brain MR scans (i.e., spinal). In terms of recall, the classifier
yielded values above 0.99 for all classes except FLAIR, which had a slightly
lower recall of 0.98, because six scans were misclassified. Of the six FLAIR
scans, three were misclassified as T1WI. These were all sagittal scans with a
very low axial resolution. Overall, of the 384 WUSM sessions, the scan-type
classifier identified all possible segmentable sequences for 380 sessions. For
these 380 sessions, there were no failures in terms of preprocessing. Hence,
they were used for the subsequent segmentation and radiomic feature
extraction.

For the MDA dataset, the classifier yielded a very high overall accuracy
(99.84%, 643/644 scans) across five classes, with only a single sagittal FLAIR
scan with low axial resolution being misclassified as T1WI. Overall, of the 30
MDA sessions, the scan-type classifier could identify all possible segmentable
sequences for all sessions, and there were no failures in terms of
preprocessing. Hence, all the sessions were used for subsequent segmentation
and radiomic feature extraction.

###  3.2 Segmentation

The segmentation results varied depending on the input sequences, with an
overall deterioration in performance with increasing number of missing
sequences, especially for ET in the absence of Gd-T1WI (Supplementary S2.1).
On the WUSM data, the segmentation models yielded high mean DSC for WT
(0.882±0.244), TC (0.75±0.334), and ET (0.91±0.24). For all tumor classes, the
mean DSCs for WHO grade IV (WT:0.901, TC:0.869, ET:0.879) were higher than
those for WHO grade II (WT:0.87, TC:0.82) and grade III (WT:0.852, TC:0.778,
ET:0.832) (Figure 2B).

On the MDA data, a high mean DSC was obtained for WT (0.977±0.04), TC
(0.984±0.028), and ET (0.899±0.097). Overall, the model generalized well for
both the external datasets.

##  4 Discussion

In this paper, we proposed I3CR-WANO, an AI-driven framework for curation,
pre-processing, segmentation, and radiomic feature extraction in neuro-
oncology MR studies. Through its end-to-end operation, the framework
transforms unstructured DICOM MR data into quantitative 3D measurements of
tumors, which can be directly used for predicting treatment response and
overall survival. I3CR-WANO was validated in 414 patient cases acquired from
two different clinical sites, with good overall performance in all facets of
processing. The different AI models used for scan-type classification and
segmentation generalized well on unseen data. The source code, dockers, and
all pre-trained models of this study have been made publicly available.

In recent years, AI-assisted tools such as niftynet [2], DeepNeuro [3], and
the cancer imaging phenomics toolkit (CaPTk) [4] have been proposed for
automating neuro-oncology workflows in clinical practice. These tools
typically depend on carefully curated data sets. However, they did not address
the problem of scan-type classification or data curation within their
operations. Instead, they rely on manual interaction, which is often the most
time-consuming step in an AI workflow [20]. In contrast, I3CR-WANO provides a
more holistic solution, from data curation to radiomic feature extraction, and
completely obviates the need for any intermediate manual interaction. Thus, it
greatly facilitates the generation of datasets required for the development
and validation of models supporting quantitative tumor measurements.
Additionally, the framework’s modular structure includes the necessary
commonalities in upstream pipelining that allow cascading with a wide array of
downstream applications (e.g., the curation and pre-processing modules are not
application-specific).

The proposed framework can streamline clinical workflows and support decision
making by automating tumor segmentation and characterization. In this emerging
era of precision diagnostics, the quantitative volumetric tumor measurements
extracted from this framework can drive personalized treatment planning and
response assessment (e.g., Response Assessment in Neuro-Oncology [RANO]
criteria [21]). The generated segmentation masks can be used to track tumors
longitudinally and quantitatively assess their growth. In a research setting,
it can significantly reduce the latency of data curation, thus expediting
model prototyping and facilitating the creation of standardized large-scale
neuro-oncology datasets for multi-institutional collaborations [15, 22] that
attempt to establish public benchmarks for various aspects of quantitative
tumor analysis.

This study has certain limitations that merit discussion. First, the CNN-based
scan-type classifier is currently pre-trained only on axial scans and has a
minor performance drop, particularly on coronal or sagittal FLAIR scans, with
a very low off-plane resolution. Second, segmentation models are currently
trained on preoperative glioma cases and cannot be used on postoperative
images. However, the current curation and pre-processing modules of the
framework are applicable to any MRI study, irrespective of the pathology or
treatment status. Moreover, owing to the modular nature of the framework, both
limitations can be addressed by using more advanced containerized models that
can be simply used as drop-in replacements for the current models. This
flexibility can also enable the extension of this framework to multiple tumor
types by integrating tumor classification models [23] and cascading it with
segmentation, radiomics, and quantitative report-generation utilities tailored
for specific tumor types.

In conclusion, we developed I3CR-WANO, an AI-driven framework that transforms
raw MRI DICOM data of patients with high- and low-grade gliomas to
quantitative tumor measurements through systematic data curation, processing,
tumor segmentation, and radiomic feature extraction, without the requirement
of any manual intervention. This work can streamline clinical workflows and
support clinical decision-making by automating tumor segmentation and
characterization as well as help in curating large-scale neuro-oncology
datasets.

## References

  * [1] Rudie Jeffrey D, Rauschecker Andreas M, Bryan R Nick, Davatzikos Christos, Mohan Suyash. Emerging applications of artificial intelligence in neuro-oncology Radiology.  2019;290:607.
  * [2] Gibson Eli, Li Wenqi, Sudre Carole, et al. NiftyNet: a deep-learning platform for medical imaging Computer methods and programs in biomedicine.  2018;158:113–122.
  * [3] Beers Andrew, Brown James, Chang Ken, et al. DeepNeuro: an open-source deep learning toolbox for neuroimaging Neuroinformatics.  2021;19:127–140.
  * [4] Davatzikos Christos, Rathore Saima, Bakas Spyridon, et al. Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome Journal of medical imaging.  2018;5:011018.
  * [5] Chung Caroline, Jaffray David A. Cancer Needs a Robust” Metadata Supply Chain” to Realize the Promise of Artificial Intelligence. Cancer Research.  2021;81:5810–5812.
  * [6] Ooijen Peter, others . Quality and curation of medical images and data in Artificial intelligence in medical imaging:247–255Springer 2019.
  * [7] Hirsch Jeffrey D, Siegel Eliot L, Balasubramanian Sridhar, Wang Kenneth C. We built this house; it’s time to move in: Leveraging existing DICOM structure to more completely utilize readily available detailed contrast administration information Journal of digital imaging.  2015;28:407–411.
  * [8] Remedios Samuel, Pham Dzung L, Butman John A, Roy Snehashis. Classifying magnetic resonance image modalities with convolutional neural networks in Medical Imaging 2018: Computer-Aided Diagnosis;10575:558–563SPIE 2018.
  * [9] Voort Sebastian R, Smits Marion, Klein Stefan. DeepDicomSort: an automatic sorting algorithm for brain magnetic resonance imaging data Neuroinformatics.  2021;19:159–184.
  * [10] Chakrabarty Satrajit, LaMontagne Pamela, Marcus Daniel S, Milchenko Mikhail. Preprocessing of clinical neuro-oncology MRI studies for big data applications in Medical Imaging 2020: Imaging Informatics for Healthcare, Research, and Applications;11318:69–76SPIE 2020.
  * [11] Rohlfing Torsten, Zahr Natalie M, Sullivan Edith V, Pfefferbaum Adolf. The SRI24 multichannel atlas of normal adult human brain structure Human brain mapping.  2010;31:798–819.
  * [12] Jenkinson Mark, Bannister Peter, Brady Michael, Smith Stephen. Improved optimization for the robust and accurate linear registration and motion correction of brain images Neuroimage.  2002;17:825–841.
  * [13] Jenkinson Mark, Smith Stephen. A global optimisation method for robust affine registration of brain images Medical image analysis.  2001;5:143–156.
  * [14] Iglesias Juan Eugenio, Liu Cheng-Yi, Thompson Paul M, Tu Zhuowen. Robust brain extraction across datasets and comparison with publicly available methods IEEE transactions on medical imaging.  2011;30:1617–1634.
  * [15] Baid Ujjwal, Ghodasara Satyam, Mohan Suyash, et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification arXiv preprint arXiv:2107.02314.  2021\.
  * [16] Herz Christian, Fillion-Robin Jean-Christophe, Onken Michael, et al. DCMQI: an open source library for standardized communication of quantitative image analysis results using DICOM Cancer research.  2017;77:e87–e90.
  * [17] Van Griethuysen Joost JM, Fedorov Andriy, Parmar Chintan, et al. Computational radiomics system to decode the radiographic phenotype Cancer research.  2017;77:e104–e107.
  * [18] Marcus Daniel S, Olsen Timothy R, Ramaratnam Mohana, Buckner Randy L. The extensible neuroimaging archive toolkit Neuroinformatics.  2007;5:11–33.
  * [19] Doran Simon J, Al Sa’d Mohammad, Petts James A, et al. Integrating the OHIF Viewer into XNAT: Achievements, Challenges and Prospects for Quantitative Imaging Studies Tomography.  2022;8.
  * [20] Montagnon Emmanuel, Cerny Milena, Cadrin-Chênevert Alexandre, et al. Deep learning workflow in radiology: a primer Insights into imaging.  2020;11:1–15.
  * [21] Bent Martin J, Wefel Jeffrey S, Schiff David, et al. Response assessment in neuro-oncology (a report of the RANO group): assessment of outcome in trials of diffuse low-grade gliomas The lancet oncology.  2011;12:583–593.
  * [22] Baheti Bhakti, Waldmannstetter Diana, Chakrabarty Satrajit, et al. The brain tumor sequence registration challenge: Establishing correspondence between pre-operative and follow-up mri scans of diffuse glioma patients arXiv preprint arXiv:2112.06979.  2021\.
  * [23] Chakrabarty Satrajit, Sotiras Aristeidis, Milchenko Mikhail, LaMontagne Pamela, Hileman Michael, Marcus Daniel. MRI-based identification and classification of major intracranial tumor types by using a 3D convolutional neural network: A retrospective multi-institutional analysis Radiology: Artificial Intelligence.  2021;3.
  * [24] Isensee Fabian, Kickingereder Philipp, Wick Wolfgang, Bendszus Martin, Maier-Hein Klaus H. Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge in International MICCAI Brainlesion Workshop:287–297Springer 2017.
  * [25] Menze Bjoern H, Jakab Andras, Bauer Stefan, et al. The multimodal brain tumor image segmentation benchmark (BRATS) IEEE transactions on medical imaging.  2014;34:1993–2024.
  * [26] Bakas Spyridon, Akbari Hamed, Sotiras Aristeidis, et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features Scientific data.  2017;4:1–13.
  * [27] Bakas Spyridon, Reyes Mauricio, Jakab Andras, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge arXiv preprint arXiv:1811.02629.  2018\.
  * [28] Bergstra James, Bengio Yoshua. Random search for hyper-parameter optimization. Journal of machine learning research.  2012;13.
  * [29] Kingma Diederik P, Ba Jimmy. Adam: A method for stochastic optimization arXiv preprint arXiv:1412.6980.  2014\.

##  Supplementary Data

##  S1 Supplementary Methods

###  S1.1 Conditions for excluding scan or session from processing

In the following conditions, a scan or an entire session is excluded from
subsequent processing:

  * •

For the scan-type classification step (specifically for Classifier1), the
series description tag is compulsory. So, in its absence, a particular scan is
excluded from all subsequent processing.

  * •

Scans that do not contain the string ‘ORIGINAL/PRIMARY’ in DICOM Image Type
attribute (i.e., DICOM tag (0008,0008)) are excluded.

  * •

Scans that have the DICOM Angio Flag attribute (i.e., DICOM tag (0018,0025))
set to ‘Y’ (i.e., image is Angio), are excluded.

  * •

In the absence of Gd-T1WI, T2WI, and FLAIR sequences for a particular session,
the entire session is marked as not segmentable and excluded from subsequent
processing.

###  S1.2 Pre-training of segmentation models

For this purpose, we adopted a 3D CNN architecture [24] and trained the model
to produce multi-class tumor segmentation maps consisting of vasogenic edema,
necrotic/non-enhancing core, and enhancing core. For development of the
segmentation models, the BraTS 2021 [15, 25, 26, 27] dataset was split into
cross-validation (n = 1251) and testing (n = 219) cohorts. For hyperparameter
tuning, we performed a random search [28] using five-fold cross-validation on
the cross-validation data by using 80% of the data for training (n =1000) and
20% for validation (n = 251). The hyperparameters that yielded the best cross-
validation results were then selected as the “best” hyperparameters. Next, the
model was trained on 100% of this data (n = 1251) using the “best” set of
hyperparameters and subsequently used for prediction on the hold-out test data
(n = 219). Scans from the BraTS 2021 dataset were already registered to the
SRI24 anatomical atlas [11], resampled to 1-mm3 isotropic resolution and
skull-stripped. Additionally, for every scan, image intensities within the
brain were normalized to zero mean and unit variance after excluding
intensities below the 5th and above the 95th percentile. Subsequently, the
image volumes were resized to a dimension of 128x128x128 before being fed into
the network. We used Adam [29] optimizer with multi-class dice loss function.
The training batch size was set to 5. The network was trained for 300 epochs
with an early-stopping callback, which stopped the training after 50 epochs if
the validation loss was not improving. The training was started with an
initial learning rate of 5 x 10-4, which was reduced by a factor of 2 every
time the validation loss became stagnant for 10 epochs. To prevent the network
from overfitting, the data were augmented using mirroring along the vertical
axis with a probability of 0.5.

###  S1.3 Radiomic feature extraction

Radiomics features were extracted using the PyRadiomics 3.0.1
(https://github.com/Radiomics/pyradiomics) [17] tool. This method produces a
set of quantitative features for various combinations of input image and
segmentation, as specified by the user. Based on our proposed framework, input
images can include at most four scans belonging to the four segmentable
sequences i.e., T1WI, Gd-T1WI, T2WI and FLAIR. As per our segmentation module
discussed in Section 2.1.2, the input segmentations can include at most five
different tumor classes viz. ED, NC, ET, TC, and WT. The default set of
radiomics features were extracted including 3D shape features (n = 14), first
order features (n = 18), and texture features (n = 75). Shape features are
independent of image contrast and were therefore only extracted once per
segmentation per session (5 segmentations × 14 features = 70 shape features
per session). First order and texture features were extracted from each
combination of image and segmentation (4 image contrasts × 5 segmentations ×
93 first order and texture features = 1,860 features per patient). Thus, the
complete radiomics feature-set included 1,930 individual image features (70
shape features + 1860 first order and texture features) per session. For all
input images, intensity-normalized scans were used as described in Section
2.1.1. The parameter values for all feature extraction were set as default for
ease of use and reproducibility.

##  S2 Supplementary Results

###  S2.1 Segmentation results on BraTS 2021 dataset

For the BraTS 2021 dataset (Supplementary Figure S1, Supplementary Table S2),
in presence of Gd-T1WI, the mean DSC values were high for all tumor
segmentation classes viz. WT (0.90±0.03), TC (0.88±0.01), and ET (0.81±0.01).
In absence of Gd-T1WI, the mean DSC for ET (0.27 drop, $P<.001$) and TC (0.15
drop, $P<.001$) dropped significantly. Hence, in absence of Gd-T1WI, the
deployed models on WUSM and MDA were used only for WT segmentation.

Figure S1: Dispersion of Dice Similarity Coefficient (DSC) values, stratified by presence of sequences, showing the agreement in terms of overlap between the tumor segmentation masks predicted by convolutional neural network and the expert-annotated masks. WT = whole tumor, TC = tumor core, ET = enhancing tumor. Table S1: Demographic and clinical information of patient data acquired from Washington University School of Medicine (WUSM) and M.D. Anderson Cancer Center (MDA). | WUSM (n = 384) | MDA (n = 30)
---|---|---
Age | 56 (44 - 66) | 58 (44 - 65)
Sex |  |
Female | 154 (40.1%) | 15 (50.0%)
Male | 230 (59.9%) | 15 (50.0%)
WHO Grade |  |
G2 | 65 (16.93%) | 5 (16.67%)
G3 | 63 (16.41%) | 3 (10.0%)
G4 | 256 (66.67%) | 22 (73.33%)
Table S2: Segmentation performance stratified by presence of sequence on the BraTS 2021 dataset in terms of Dice Similarity Co-efficient (DSC) for Whole tumor, Tumor core and Enhancing tumor classes. The table shows the DSC for all possible configurations of sequences being either absent ($\circ$) or present ($\bullet$), in order of FLAIR ($F$), T1WI ($T_{1}$), Gd-T1WI ($T_{1}c$), T2WI ($T_{2}$). Sequences | Tumor class
---|---
$\;F$ | $\,T_{1}$ | $T_{1}c$ | $\,T_{2}$ |  Whole Tumor |  Tumor Core |  Enhancing Tumor
$\bullet$ | $\circ$ | $\circ$ | $\circ$ |  0.914 (0.091) |  0.733 (0.221) |  0.531 (0.22)
$\circ$ | $\bullet$ | $\circ$ | $\circ$ |  0.819 (0.132) |  0.713 (0.212) |  0.501 (0.221)
$\circ$ | $\circ$ | $\bullet$ | $\circ$ |  0.811 (0.158) |  0.88 (0.173) |  0.805 (0.219)
$\circ$ | $\circ$ | $\circ$ | $\bullet$ |  0.879 (0.114) |  0.719 (0.232) |  0.525 (0.231)
$\bullet$ | $\bullet$ | $\circ$ | $\circ$ |  0.916 (0.085) |  0.757 (0.21) |  0.567 (0.229)
$\circ$ | $\bullet$ | $\bullet$ | $\circ$ |  0.893 (0.103) |  0.868 (0.218) |  0.799 (0.242)
$\circ$ | $\bullet$ | $\circ$ | $\bullet$ |  0.885 (0.115) |  0.73 (0.235) |  0.546 (0.23)
$\bullet$ | $\circ$ | $\bullet$ | $\circ$ |  0.926 (0.063) |  0.886 (0.175) |  0.818 (0.216)
$\circ$ | $\circ$ | $\bullet$ | $\bullet$ |  0.886 (0.111) |  0.875 (0.195) |  0.804 (0.232)
$\bullet$ | $\circ$ | $\circ$ | $\bullet$ |  0.924 (0.062) |  0.751 (0.209) |  0.563 (0.233)
$\bullet$ | $\bullet$ | $\bullet$ | $\circ$ |  0.924 (0.066) |  0.892 (0.175) |  0.825 (0.217)
$\circ$ | $\bullet$ | $\bullet$ | $\bullet$ |  0.892 (0.121) |  0.869 (0.215) |  0.803 (0.247)
$\bullet$ | $\bullet$ | $\circ$ | $\bullet$ |  0.916 (0.1) |  0.74 (0.217) |  0.554 (0.234)
$\bullet$ | $\circ$ | $\bullet$ | $\bullet$ |  0.926 (0.071) |  0.882 (0.181) |  0.813 (0.223)
$\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ |  0.923 (0.07) |  0.89 (0.169) |  0.814 (0.224)