LaBella and Calabrese 1 2024 Medical Imaging with Deep Learning (MIDL) 2020
Marleen de Bruijne, Tal Arbel, Ismail Ben Ayed, Hervé Lombaert 2024 BraTS-MEN-
RT ChallengeLaBella and Calabrese 1 Department of Radiation Oncology, Duke
University Medical Center, Durham, NC, USA
2 Department of Radiation Oncology, SUNY Upstate Medical University, Syracuse,
NY, USA
3 Center for Intelligent Imaging (ci2), Department of Radiology and Biomedical
Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
4 Department of Neurosurgery, King’s College Hospital, London, United Kingdom
5 School of Biomedical Engineering and Imaging Sciences, King’s College
London, London, United Kingdom
6 Guy’s and St Thomas’ NHS Foundation Trust
7 Department of Radiation Oncology, University of Washington, Seattle, WA, USA
8 Department of Radiology, Northwestern University, Evanston, IL, USA
9 Department of Radiation Oncology, Northwestern University, Evanston, IL, USA
10 Department of Radiation Medicine and Applied Sciences, University of
California San Diego, La Jolla, CA, USA
11 Department of Radiology, University of California San Diego, La Jolla, CA,
USA
12 Department of Bioengineering, University of California San Diego, La Jolla,
CA, USA
13 Department of Neurological Surgery, Indiana University School of Medicine,
Indianapolis, IN, USA
14 Department of Radiation Oncology, Indiana University, Indianapolis, IN, USA
15 Children’s National Hospital, Washington DC, USA
16 George Washington University, Washington DC, USA
17 Mayo Clinic, Rochester, MN, USA
18 CMH Lahore Medical College, Lahore, Pakistan
19 Duke University Medical Center, School of Medicine, Durham, NC, USA
20 Sage Bionetworks, USA
21 University of Zürich, Zürich, Switzerland
22 Artificial Intelligence Lab, Department of Radiology, Mayo Clinic,
Rochester, MN, USA
23 MLCommons
24 Factored AI
25 Center For Federated Learning in Medicine, Indiana University,
Indianapolis, IN, USA
26 Division of Computational Pathology, Department of Pathology and Laboratory
Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
27 Medical Working Group, MLCommons, San Fransisco, CA, USA
28 Intel
29 Duke University, Durham, NC, USA
30 Cornell University, Ithaca, NY, USA
31 Helmholtz AI, Helmholtz Munich, Germany
32 Department of Informatics, Technical University Munich, Germany
33 TranslaTUM - Central Institute for Translational Cancer Research, Technical
University of Munich, Germany
34 Department of Diagnostic and Interventional Neuroradiology, School of
Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany
35 Institute of Neuroscience and Medicine (INM-4), Research Center Juelich,
Juelich, Germany
36 Department of Nuclear Medicine, University Hospital RWTH Aachen, Aachen,
Germany
37 Department of Medical Image Computing, German Cancer Research Center
(DKFZ), Heidelberg, Germany
38 Department of Neuroradiology, University Hospital Bonn, Bonn Germany
39 Department of Radiology and Nuclear Medicine, Université Laval, Québec,
Québec, Canada
40 Medical Artificial Intelligence (MAI) Lab, Crestview Radiology, Lagos,
Nigeria
41 Athinoula A Martinos Center for Biomedical Imaging, Massachusetts General
Hospital, Boston, MA, USA
42 Department of Neuroradiology, Technical University of Munich, Munich,
Germany
43 University of Zurich, Switzerland
44 Children’s Hospital of Philadelphia, University of Pennsylvania,
Philadelphia, PA, USA
45 Center for AI and Data Science for Integrated Diagnostics (AI2D) and Center
for Biomedical Image Computing and Analytics (CBICA), University of
Pennsylvania, Philadelphia, PA, USA
46 University of Missouri, Columbia, MO, USA
47 Montreal Neurological Institute (MNI), McGill University, Montreal, QC,
Canada
48 Mercy Catholic Medical Center, Darby, PA, USA
49 Biomedical Image Analysis and Machine Learning, Department of Quantitative
Biomedicine, University of Zurich, Switzerland
50 Center for Federated Learning in Medicine, Indiana University,
Indianapolis, IN, USA
51 Department of Radiology, Mayo Clinic, Rochester, MN, USA
52 Neosoma Inc. Stanford Medicine, Stanford, CA, USA
53 Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA
54 Department of Surgical Sciences, Section of Neuroradiology, Uppsala
University, Uppsala, Sweden
55 Department of Radiology, University of California San Diego, CA, USA.
56 Department of Radiology and Imaging Sciences, Indiana University School of
Medicine, Indianapolis, IN, USA
57 Department of Biostatistics and Health Data Science, Indiana University
School of Medicine, Indianapolis, IN, USA
58 Duke University Medical Center, Department of Radiology, Durham, NC, USA

# Brain Tumor Segmentation (BraTS) Challenge 2024: Meningioma Radiotherapy
Planning Automated Segmentation

Dominic LaBella10000-0003-1713-9538 Katherine Schumacher2 Michael Mix2 Kevin
Leu3 Shan McBurney-Lin3 Pierre Nedelec3 Javier Villanueva-Meyer3 Jonathan
Shapey4 Tom Vercauteren5 Kazumi Chia6 Omar Al-Salihi6 Justin Leu7 Lia Halasz7
Yury Velichko8 Chunhao Wang1 John Kirkpatrick1 Scott Floyd1 Zachary J.
Reitman1 Trey Mullikin1 Ulas Bagci9 Sean Sachdev8 Jona A. Hattangadi-Gluth10
Tyler M. Seibert10,11,12 Nikdokht Farid11 Connor Puett10 Matthew W. Pease13
Kevin Shiue14 Syed Muhammad Anwar15,16 Shahriar Faghani17 Peter Taylor2
Muhammad Ammar Haider18 Pranav Warman19 Jake Albrecht20 András Jakab21 Mana
Moassefi22 Verena Chung20 Alejandro Aristizabal23,24 Alexandros Karargyris23
Hasan Kassem23 Sarthak Pati25,26,27 Micah Sheller28,23 Aaron Coley29 Christina
Huang1 Siddharth Ghanta29 Alex Schneider29 Conrad Sharp29 Rachit Saluja30
Florian Kofler31,32,33,34 Philipp Lohmann35,36 Phillipp Vollmuth37,38 Louis
Gagnon39 Maruf Adewole40 Hongwei Bran Li41,42,43 Anahita Fathi Kazerooni44,45
Nourel Hoda Tahon46 Udunna Anazodo47 Ahmed W. Moawad48 Bjoern Menze49,42
Marius George Linguraru15,16 Mariam Aboian44 Benedikt Wiestler42 Ujjwal
Baid26,50 Gian-Marco Conte51 Andreas M. Rauschecker3 Ayman Nada46 Aly H.
Abayazeed52 Raymond Huang53 Maria Correia de Verdier54,55 Jeffrey D. Rudie55,3
Spyridon Bakas26,56,13,570000-0001-8734-6482 Evan Calabrese58,3

###### Abstract

The 2024 Brain Tumor Segmentation Meningioma Radiotherapy (BraTS-MEN-RT)
challenge aims to advance automated segmentation algorithms using the largest
known multi-institutional dataset of radiotherapy planning brain MRIs with
expert-annotated target labels for patients with intact or post-operative
meningioma that underwent either conventional external beam radiotherapy or
stereotactic radiosurgery. Each case includes a defaced 3D post-contrast
T1-weighted radiotherapy planning MRI in its native acquisition space,
accompanied by a single-label “target volume” representing the gross tumor
volume (GTV) and any at-risk post-operative site. Target volume annotations
adhere to established radiotherapy planning protocols, ensuring consistency
across cases and institutions. For pre-operative meningiomas, the target
volume encompasses the entire GTV and associated nodular dural tail, while for
post-operative cases, it includes at-risk resection cavity margins as
determined by the treating institution. Case annotations were reviewed and
approved by expert neuroradiologists and radiation oncologists. Participating
teams will develop, containerize, and evaluate automated segmentation models
using this comprehensive dataset. Model performance will be assessed using the
lesion-wise Dice Similarity Coefficient and the 95% Hausdorff distance. The
top-performing teams will be recognized at the Medical Image Computing and
Computer Assisted Intervention Conference in October 2024. BraTS-MEN-RT is
expected to significantly advance automated radiotherapy planning by enabling
precise tumor segmentation and facilitating tailored treatment, ultimately
improving patient outcomes.

###### keywords:

Meningioma, BraTS, Machine Learning, Segmentation, BraTS-Meningioma, Image
Analysis Challenge, Artificial Intelligence, AI, Radiation Oncology,
Radiotherapy, Stereotactic Radiosurgery, Gamma Knife

##  1 Introduction and Related Works

Meningioma is the most common primary intracranial tumor and comprises 40.8%
of all CNS tumors and 55.4% of all non-malignant CNS tumors (Ogasawara et al.,
2021; Huntoon et al., 2020; Ostrom et al., 2023). The vast majority, 99.1%,
are non-malignant and can be followed with serial magnetic resonance imaging
(MRI) if asymptomatic (Ostrom et al., 2023; Goldbrunner et al., 2021; Yano and
Kuratsu, 2006). WHO grade I comprise 80% of meningioma and standard of care
for tumors requiring treatment is maximal surgical resection if surgically
accessible. Higher grade meningiomas (World Health Organization [WHO] grades
II and III), if left untreated, are associated with higher morbidity and
mortality rates and often recur despite optimal management (Zhao et al., 2020;
Simpson, 1957; Louis et al., 2021; Saraf et al., 2011). For WHO grade I
meningioma, if gross tumor resection is achieved, then surveillance imaging
alone is often appropriate. In cases of subtotal resection or non-resectable
tumors, observation or radiotherapy with external beam radiotherapy (EBRT) or
stereotactic radiosurgery (SRS) can be considered (Kent et al., 2022; Pinzi et
al., 2023; Rogers et al., 2017, 2020; Weber et al., 2018). The Simpson grading
system for meningioma resection classifies resection from a grade of 0 – 5 and
helps guide indication for adjuvant radiotherapy (Simpson, 1957; Simon and
Gousias, 2024). Essentially all grade III and many grade II meningiomas will
be treated with radiotherapy, either as a primary treatment modality, as an
adjunct in the immediate post-operative setting, and/or in the setting of
meningioma recurrence (Aghi et al., 2009; Hug et al., 2000; Boskos et al.,
2009). For WHO grade II and III meningioma, radiotherapy has shown improvement
in progression free survival and overall survival (Kent et al., 2022; Kessel
et al., 2017).

Accurate segmentation of the pre-operative meningioma gross tumor volume (GTV)
and clinical target volume (CTV) is essential for radiotherapy planning. The
phase II EORTC 22042-026042 study on adjuvant postoperative high-dose
radiotherapy for atypical and malignant meningioma defines the GTV as visible
tumor which is the region of enhancement on post-operative brain MRI (Weber et
al., 2018). They define the clinical target volume CTV1 as the GTV and/or sub
clinical microscopic tumor (may include the pre-operative tumor bed,
peritumoral edema, hyperostotic changes if any, and dural enhancement or
thickening as seen in the CT/MRI at diagnosis) plus a 3D 10 mm margin (Weber
et al., 2018). The CTV2 was defined as the GTV and/or sub clinical microscopic
tumor plus a 3D 5 mm margin (Weber et al., 2018). The phase II RTOG 0539 study
of observation for low-risk meningioma and radiotherapy for intermediate and
high-risk meningioma defines GTV as the tumor bed and residual enhancement,
including nodular dural tail enhancement, but not small linear dural tail
enhancement (Rogers et al., 2017, 2020).

Unfortunately, GTV and CTV segmentation is complex, time-consuming, and
requires considerable expertise. Automated tumor segmentation on brain MRI has
matured into a clinically viable tool that can provide objective assessments
of tumor volume and can assist in surgical planning, radiotherapy, and
treatment response assessment. To date, published data on reliable automated
methods for meningioma GTV segmentation is limited. Most tumor segmentation
studies, including all prior BraTS challenges, have focused exclusively on
pre-operative tumors after pre-processing to a $1mm^{3}$ isotropic resampled
image space, which limits clinical utility (LaBella et al., 2023, 2024a; Bakas
et al., 2017; Menze et al., 2014; Kazerooni et al., 2023; Moawad et al., 2023;
Adewole et al., 2023). Segmenting post-operative tumors is a considerably more
complex challenge but is also considerably more clinically relevant. Recent
studies have reported on post-operative automated segmentation models for
glioma, but none known to date have focused on post-operative meningioma
segmentation (Ermiş et al., 2020; Bianconi et al., 2023). The BraTS 2023
Meningioma (2023 BraTS-MEN) challenge focused on pre-operative meningioma
cases and utilized multi-sequence co-registered brain MRI studies to segment
regions of interest including the whole tumor (WT), tumor core (TC), and
enhancing tumor (ET) (LaBella et al., 2023). The TC consisted of all enhancing
and non-enhancing tumor. The WT consisted of the TC and all surrounding non-
enhancing T2/FLAIR hyperintensity (SNFH). However, in radiotherapy, the SNFH
does not play a common role in the delineation of target volumes for
meningioma.

Furthermore, previous BraTS challenges utilized skull-stripping, whereas the
2024 Brain Tumor Segmentation Meningioma Radiotherapy (BraTS-MEN-RT) challenge
will preserve extracranial structures and instead use automated defacing
algorithms to preserve patient anonymity (Bischoff-Grethe et al., 2007). By
including extracranial tissue, using a more generous defacing anonymization
algorithm, the automated segmentation models will be more relevant and
deployable into clinical workflows. Finally, target labels will consist of a
single tumor region (the GTV) in the native acquisition space. To reduce
barriers to participation, we will provide software to automatically convert
challenge data to a canonical reference space like previous BraTS challenges,
as well as to convert resulting labels back to patient native space for
subsequent evaluation.

The purpose of the BraTS-MEN-RT challenge is to create a community benchmark
for automated segmentation of meningioma GTV based on pre-radiotherapy
planning brain MRI exams. This task, if successful, will provide an important
tool for the objective delineation of meningioma GTV, which will be the first
BraTS challenge that is immediately relevant for radiotherapy planning.
Participating teams’ automated segmentation models will be open-source,
providing both industry partners and clinical researchers the opportunity to
utilize and build upon the radiotherapy planning automated segmentation models
for meningioma.

##  2 Methods

###  2.1 Data Description

Each case within the BraTS-MEN-RT challenge consists exclusively of
radiotherapy planning brain MRI scans in either the pre-operative or post-
operative setting. All brain MRI studies included tumors in the field of view
that were radiographically or pathologically consistent with meningioma. Brain
MRI studies consisted of a single series (3D postcontrast T1-weighted imaging,
most commonly spoiled gradient echo or similar (T1c)) in native acquisition
space, which mimics the data available for most radiotherapy planning
scenarios. This has evolved from the 4 multi-sequence MRI scans that were co-
registered to a canonical atlas space, SRI24, with $1mm^{3}$ isotropic
resampling that were utilized in each of the BraTS 2023 automated segmentation
challenges (LaBella et al., 2023; Adewole et al., 2023; Kazerooni et al.,
2023; Moawad et al., 2023; Rohlfing et al., 2010).

###  2.2 Defining Meningioma Target Volume

For the purposes of the BraTS-MEN-RT challenge, there will be a single target
volume label. The target volume annotation protocol will differ depending on
whether the radiotherapy planning scan was obtained in the pre-operative or
post-operative setting.

If the meningioma radiotherapy course was planned in the pre-operative
setting, then the target volume label will comprise of the portion of the
tumor visible on the T1c brain MRI (Figure 1).

Figure 1: Image panels depicting a case that utilizes a radiation planning
Gamma Knife headframe. Panels A, B, and C depict an intact meningioma (red) in
Meckel’s cave on T1c radiation planning axial, sagittal, and coronal images,
respectively. Note that this challenge’s defacing technique preserves this
meningioma as compared to the BraTS 2023 Meningioma Segmentation challenge’s
skull-stripping pre-processing technique which would have excluded this case.
Panel D shows the SRS localizer box fiducials attached to a standard Gamma
Knife headframe.

If the meningioma radiotherapy course was planned in the post-operative
setting, then the target will comprise of the post-op resection bed and any
residual ET on the T1c brain MRI (Figures 2 and 3).

Figure 2: Panels A, B, and C depict an extra-axial meningioma overlying the
left frontal lobe in T1c radiotherapy planning axial, sagittal, and coronal
views respectively. Note that these images contain “zipper” artifact as
demonstrated by the streaking horizontal white lines seen in panels B and C,
which may be caused by any combination of radiofrequency interference,
inadequate shielding, or hardware issues. Panel D shows the SRS localizer box
fiducials attached to a standard Gamma Knife headframe. Figure 3: Panels A, B,
and C showing a postoperative left parietal meningioma target volume (red) in
the axial, sagittal, and coronal planes, respectively, as delineated by the
treating institution.

These label definitions are clinically useful in radiotherapy planning and
were agreed upon by a coalition of BraTS organizers consisting of board-
certified radiation oncologists and board-certified fellowship trained
neuroradiologists. The target volume labels were agreed upon after review of
the EORTC 22042-026042 and RTOG 0589 annotation protocols (Rogers et al.,
2017, 2020; Weber et al., 2018).

All visible intracranial meningiomas are included in the GTV label, even if
they were not treated in the real world clinical scenario (Figure 4). The
rationale for labeling all meningioma is to allow the treating radiation
oncology team the opportunity to utilize automated segmentations for any and
all meningioma within the patient’s respective brain MRI. This approach also
ensures that automated segmentation algorithm training will not be adversely
affected by non-segmented non-target meningiomas.

Figure 4: Panels A, B, and C depict an anterior left falx meningioma (red) on
axial, sagittal, and coronal images respectively. Panel B demonstrates an area
of hypointense edema between two separate anterior falcine meningiomas. Note
that the hypointense edema region is not labeled since this is not typically
treated in meningioma radiotherapy. Panel C shows a coronal slice
demonstrating the distinct intensity difference between the zero T1c
intensity, true black, defaced region (blue arrow) compared to the low T1c
intensity, gray, region outside the patient’s head (green arrow).

###  2.3 Participating Sites

Approximately 700 radiotherapy planning T1c brain MRI scans were contributed
from 6 academic medical centers across the United States (Table 1) at the time
of release of the training set in late May 2024. Additional academic centers
have the opportunity to contribute additional data prior to the release of
validation set images in June-July 2024 and internal evaluation of test set
cases in August 2024. Cases were identified based on any pre-operative or
post-operative meningioma that had undergone radiotherapy with any
radiotherapy technique. Radiotherapy techniques could vary between EBRT or
SRS, and utilized either photon, Cobalt-60, or protons as the radiation
source. For patients that underwent SRS with GKRS, the stereotactic localizer
fiducials are visible within the brain MRI as seen in Figures 1 and 2. The
stereotactic localizer fiducials correspond to the real-world Gamma Knife
headframe like the unmodified figure depicted by Wiant et al as shown in
Figure 5 (Wiant and Bourland, 2009; Gallagher et al., 2008).

Figure 5: A Leksell Model G GK headframe fixed to a skull phantom. This figure
shows the CT fiducial box that is used to orient the image reference frame in
LGP.

Case collection methods were chosen by each participating site independently
to promote contribution to the challenge dataset and data contributors were
not required to disclose data collection methods or MRI protocol information.
Like prior BraTS challenges, imaging parameters including field strength,
echo/repetition time, and image resolution, varied considerably between and
within institutions (LaBella et al., 2023). Participating sites had the option
of submitting their own GTV labels for review for potential inclusion in the
BraTS-MEN-RT challenge. However, all site-submitted GTV labels underwent
rigorous evaluation by the BraTS expert annotators to ensure consistency with
the challenge annotation protocol. If the site-submitted GTV labels did not
conform to the challenge annotation protocol, then they underwent manual
revision until conformity was met as described in section 2.6.

Table 1: This table presents the total number of cases provided by each institution at the time of training data release in late May 2024. Note that the training data was released with ground truth labels. The included institutions are University of California San Francisco (UCSF), State University of New York Upstate Medical University (SUNY), University of Washington (UW), University of Missouri (MISS), Duke University (Duke), and University of San Diego (UCSD). | Total Count
---|---
Overall |  800
UCSF |  240
SUNY |  220
UW |  150
MISS |  90
DUKE |  60
UCSD |  30

###  2.4 Image Data Preprocessing

All radiotherapy planning images underwent pre-processing. This included
conversion from DICOM and DICOM-RT to Neuroimaging Informatics Technology
Initiative (NIfTI) image file format using dcmrtstruct2nii followed by
automated defacing using the Analysis of Functional Neuroimages toolbox (AFNI)
as seen in Figure 6 (Cox, 1996; Cox and Hyde, 1997; Phil et al., 2023).

This decision was made due to the superior performance of increased inclusion
of meningioma tumors within the respective pre-processed brain MRI using the
AFNI defacing method compared to other defacing algorithms (Cox, 1996; Cox and
Hyde, 1997; Theyers et al., 2021).

Figure 6: Example of a brain MRI before and after automated defacing.

All radiotherapy structures provided within the DICOM-RT structure sets were
evaluated, and only the treating institutions’ GTV structure for each
respective case was included in the dataset. The institutions’ GTV structure
was set as the starting target label before manual revision. Manual quality
control was performed on all pre-processed images to ensure adequate image
preparation without total exclusion of meningioma. If there was only partial
inclusion of meningioma within the face anonymized brain MRI as seen in Figure
7, then that respective case was still included within the dataset after
manual revision of the face anonymized brain MRI mask to ensure total
meningioma volume inclusion. If the AFNI defacing algorithm removed meningioma
from the field-of-view in its entirety, then the case was removed from the
challenge dataset. All manual quality control was performed using ITKSnap, a
free open-source, multi-platform interactive software application used to
navigate and manually segment structures in 3D and 4D biomedical images
(Yushkevich et al., 2006).

Figure 7: Example of an axial MRI image-label pair slice where the treating
institution’s meningioma GTV extended outside of the defaced image.

###  2.5 Automated Pre-segmentation

For cases that did not have a GTV provided by the treating institution, a pre-
segmentation algorithm was performed on the respective brain MRI. A deep
convolutional neural network-based automated segmentation model, nnUnet, was
used for automated GTV pre-segmentation (Isensee et al., 2021). The initial
model was trained on the 1424 brain MRI from the BraTS 2023 Intracranial
Meningioma Segmentation Challenge (2023 BraTS-MEN) (LaBella et al., 2023,
2024a). However, the cases in the 2023 BraTS-MEN challenge consisted of
treatment naive skull-stripped multi-sequence images that were co-registered
to the SRI24 atlas with isotropic $1mm^{3}$ resampling, which limits
generalization to post-operative and post-treatment tumors in native
acquisition space and with face removal (Pati et al., 2022). The multi-
sequence images included T1-weighted (T1), T2-weighted (T2), T2-FLAIR (FLAIR),
and T1c. The 2023 BraTS-MEN cases had multi-compartment labels consisting of
an ET, TC, and surrounding non-enhancing T2/FLAIR hyperintensity. Therefore,
in order to most accurately reflect the image and labels used in the BraTS-
MEN-RT challenge, only the TC region of interest and the T1c skull-stripped
image were used for training of the pre-segmentation algorithm. The TC region
of interest consisted of the addition of the ET label (blue) and non-enhancing
TC label (red) as shown in Figure 8, which is an unmodified figure from
LaBella et al (LaBella et al., 2023, 2024a). The pre-segmentation model was
applied to all of the cases without institution GTV labels, which comprised
only about 10% of the overall case data. The automated pre-segmentation
algorithm was retrained on an iterative basis as additional BraTS-MEN-RT cases
were reviewed. The purpose of retraining the model on an iterative basis was
to improve its ability to recognize meningioma within the brain MRI.

Figure 8: Meningioma sub-compartments considered in the BraTS Pre-operative
Meningioma Dataset. Image panels A-C denote the different tumor sub-
compartments included in manual annotations; (A) ET (blue) visible on a
T1-weighted post-contrast image; (B) the non-enhancing TC (red) visible on a
T1-weighted post-contrast image; (C) the surrounding FLAIR hyperintensity
(green) visible on a T2/FLAIR-weighted image; (D) combined segmentations
generating the final tumor sub-compartment labels provided in the BraTS Pre-
operative Meningioma Dataset.

###  2.6 Manual Corrections

For each meningioma case, after either automated pre-segmentation or
processing of the provided institution’s image-label pair, manual review and
correction by a senior radiation oncology resident (D.L.) was performed per
the annotation protocol outlined in section 2.2. Common corrective changes
included segmenting additional meningioma targets within the image range,
smoothing out label edges on adjacent axial slices to most accurately reflect
the meningiomas, and correcting for any misregistration between the image-
label pair. These common errors are demonstrated in Figure 9. After initial
manual review and correction, each case was further reviewed by a fellowship
trained neuroradiologist “approver” (E.C.) before inclusion in the challenge
dataset. Manual review and corrections were performed using ITKSnap
(Yushkevich et al., 2006).

Figure 9: Axial, sagittal, and coronal brain MRI of a patient with multiple
meningioma demonstrating the difference between the provided institution’s GTV
as seen in panels A1, B1, and C1 compared to the manually revised target label
as seen in panels A2, B2, and C2. Note that corrections were made regarding
inclusion of additional meningioma, correction of label edges, and inter-axial
slice label smoothening.

##  3 Discussion

###  3.1 Potential Benefits of the Challenge

The BraTS 2024 Meningioma challenge leverages a unique and extensive dataset,
offering significant advancements in the automated segmentation of meningiomas
on radiotherapy planning brain MRI. By focusing on a single T1c MRI sequence
in its native resolution, BraTS-MEN-RT addresses the need for clinically
practical and easily deployable models. This approach contrasts with previous
BraTS challenges that often required multiple MRI sequences and pre-processing
steps including co-registration to an atlas space with isotropic resampling
and extensive skull-stripping (LaBella et al., 2023; Bakas et al., 2017; Menze
et al., 2014; Moawad et al., 2023; Kazerooni et al., 2023; Adewole et al.,
2023). After completion of the challenge, participating teams’ automated
segmentation models will be open source, providing both industry partners and
clinical researchers the opportunity to utilize and build upon the
radiotherapy planning automated segmentation models for meningioma.

###  3.2  Clinical Relevance

In radiation oncology, automated segmentation models of meningioma GTV can
immediately accelerate the generation of radiotherapy treatment plans.
Automated segmentation models provide consistent and objective tumor volume
delineations, which are essential for developing precise and effective
radiotherapy plans. By reducing the variability and potential errors
associated with manual segmentation, automated segmentation tools can enhance
the overall quality and reproducibility of radiotherapy treatments.

Automated segmentation lays the groundwork for developing predictive models
that can non-invasively identify meningioma grade, subtype, and
aggressiveness. These models have the potential to serve as tools for
assessing tumor progression and response to therapies, thereby facilitating
personalized treatment plans. The 2023 BraTS-MEN challenge utilized multi-
sequence multi-compartment labels which provide even more diagnostic
radiographic data regarding the meningioma cases. Future research can build on
these models to develop tools that predict the risk of recurrence and guide
follow-up care.

###  3.3 Recommendation to Challenge Participants

Participants in the BraTS-MEN-RT challenge are encouraged to use additional
public meningioma image datasets, such as the 1424 pre-operative intact
meningioma cases from the 2023 BraTS-MEN challenge (LaBella et al., 2023,
2024a). In order to best match the case data in the BraTS-MEN-RT challenge,
only the T1c sequence and a single target label should be included. The 2023
BraTS-MEN challenge’s TC region of interest best represents the BraTS-MEN-RT
target label. The TC region of interest comprises the enhancing tumor (blue)
and the non-enhancing tumor core (red) as seen in Figure 8 (LaBella et al.,
2023, 2024a). Note that the 2023 BraTS-MEN cases underwent pre-processing
including skull-stripping and $1mm^{3}$ isotropic resampling to the SRI24
atlas space, which may introduce model development difficulties due to the
different image spaces between challenges’ images (Pati et al., 2022). For
participants intending to use this data, it is recommended to perform pre-
processing and post-processing of the BraTS-MEN-RT data similar to the
$1mm^{3}$ isotropic resampling in the SRI24 atlas space using the FeTS toolkit
and then back to native resolution may assist with model development (Pati et
al., 2022). Participants must properly cite all additional datasets used in
their challenge submissions to ensure proper acknowledgment and
reproducibility and to be eligible for awards at the challenge conclusion.

###  3.4 Limitations of the Challenge

Despite the significant advancements, BraTS-MEN-RT faces several limitations
that must be acknowledged:

####  3.4.1 Single Modality Focus

The reliance on a single T1c imaging modality may not capture the full
heterogeneity of meningiomas and the surrounding tissue. Multimodal imaging
approaches, which integrate data from different MRI sequences or other imaging
techniques such as CT or PET, could provide more comprehensive insights into
tumor characteristics and help with target volume delineation (Huang et al.,
2019). CT imaging best demonstrates bony changes associated with meningioma
including hyperostosis, osteolysis, and pneumosinus dilatans (Huang et al.,
2019). PET labeled with somatostatin receptor II ligands showed an increased
sensitivity for detection and delineation of meningioma when compared to T1c
brain MRI, especially near the skull base, falcine area, orbit, and sagittal
and cavernous sinus due to transosseous growth (Huang et al., 2019).

The 2023 BraTS-MEN challenge had a focus on diagnostics and interval changes
in tumor region of interest volumes and therefore utilized a multi-sequence
brain MRI and multi-label dataset. Incorporating multiple imaging modalities
introduces additional complexity in data processing and model training, and
the RTOG and EORTC only require T1c image sequences for radiotherapy planning,
and therefore we only utilized T1c images (Rogers et al., 2017, 2020; Weber et
al., 2018).

####  3.4.2 Variability in MRI Acquisition

Differences in MRI acquisition protocols across participating institutions can
affect the consistency and generalizability of the automated segmentation
models. Variations in scanner types, imaging parameters, and patient head
immobilization may introduce biases that are difficult to account for, even
with rigorous data pre-processing and standardization efforts. Ensuring model
robustness across different imaging conditions remains a significant
challenge. However, by including patients that undergo EBRT and SRS, we offer
a more heterogeneous dataset that can help create more generalizable automated
tumor segmentation models.

####  3.4.3 Clinical Utility

While automated segmentation models hold great promise, translating these
tools into clinical practice involves several hurdles. Clinicians may require
additional training to use these tools effectively, and the models must be
thoroughly validated in diverse clinical settings to ensure their reliability
and accuracy. Additionally, integration with existing clinical workflows and
electronic health record systems needs careful consideration to maximize their
utility. Many existing commercial radiotherapy planning applications already
use automated contouring for a variety of normal organ at risk structures and
a limited number of tumor target volume structures. We anticipate that the
open-source release of the BraTS-MEN-RT automated segmentation models
developed by each of the participating teams will provide both academic
researchers and industry partners the opportunity to create robust and
generalizable models for their radiotherapy planning applications.

###  3.5 Goals for Future Challenges

As we look beyond the BraTS-MEN-RT challenge, several exciting opportunities
for future research and challenges emerge. These future challenges should aim
to encompass a broader range of tumor types for radiotherapy planning and to
incorporate more comprehensive data, including different imaging modalities
like CT simulation imaging and PET. Incorporating additional imaging
modalities such as CT and PET can provide more detailed information about
tumor characteristics and surrounding anatomy. CT imaging can help in
identifying bony changes and delineating tumors near bony structures, while
Gallium-68 labeled somatostatin analogs used in PET imaging can enhance the
detection and delineation of tumors with high metabolic activity or expression
of SSTR ligands (Huang et al., 2019). This multimodal approach can lead to
more accurate and robust segmentation models.

Examples of other brain tumor types that commonly undergo radiotherapy include
gliomas, acoustic neuromas, vestibular schwannoma, and pediatric tumors (Stupp
et al., 2005; Sheehan et al., 2013; Hargrave et al., 2006; Yang et al., 2011).
Future studies should focus on building large multi-institutional expert
annotated radiotherapy planning image datasets for each brain tumor type to
facilitate the development of robust automated segmentation models.

Future challenges should ensure that both GTVs and CTVs are included and
labeled according to radiation therapy annotation protocol consensus
guidelines for the respective tumor types. This comprehensive automated
segmentation will allow for the development of models that are more clinically
relevant and useful in real world radiotherapy planning.

##  4 Conclusion

The BraTS-MEN-RT challenge provides the largest known dataset of expert
annotated meningioma radiotherapy planning brain MRIs and aims to push the
boundaries of automated segmentation of meningioma for radiotherapy planning.

This challenge emphasizes the clinical application and relevance of automated
segmentation algorithms by utilizing a single T1c brain MRI sequence at its
native resolution with one target label. This deliberate choice aims to
simplify the integration of participants’ models into clinical workflows,
enhancing their accessibility and practicality for real-world application.

The potential clinical impact of the BraTS-MEN-RT challenge is substantial.
Automated segmentation tools have the capacity to significantly reduce the
time and expertise required for manual contouring, which is a critical and
time-consuming step in radiotherapy planning (Ye et al., 2022). By providing
consistent and objective tumor delineations, these tools can enhance the
quality and reproducibility of treatment plans, leading to more precise and
effective patient care.

Research reported in this publication was partly supported by the National
Institutes of Health (NIH) under award numbers: NCI/ITCR U24CA279629, and
NCI/ITCR U01CA242871. The content of this publication is solely the
responsibility of the authors and does not represent the official views of the
NIH.

The work follows appropriate ethical standards in conducting research and
writing the manuscript, following all applicable laws and regulations
regarding treatment of animals and human subjects. All participating sites had
institutional review board (IRB) approval. A waiver for informed consent was
provided by each institution’s respective IRB.

TMS reports honoraria from Varian Medical Systems, WebMD, GE Healthcare, and
Janssen; he has an equity interest in CorTechs Labs, Inc. and serves on its
Scientific Advisory Board; he receives research funding from GE Healthcare
through the University of California San Diego.

The BraTS-MEN-RT data will be publicly available on Synapse as the challenge
commences in late May 2024 - October 2024 (LaBella et al., 2024b).

## References

  * Ogasawara et al. (2021) Christian Ogasawara, Brandon D Philbrick, and D Cory Adamson.  Meningioma: a review of epidemiology, pathology, diagnosis, treatment, and future directions.  _Biomedicines_ , 9(3):319, 2021.
  * Huntoon et al. (2020) Kristin Huntoon, Angus Martin Shaw Toland, and Sonika Dahiya.  Meningioma: a review of clinicopathological and molecular aspects.  _Frontiers in Oncology_ , 10:579599, 2020.
  * Ostrom et al. (2023) Quinn T Ostrom, Mackenzie Price, Corey Neff, Gino Cioffi, Kristin A Waite, Carol Kruchko, and Jill S Barnholtz-Sloan.  Cbtrus statistical report: primary brain and other central nervous system tumors diagnosed in the united states in 2016—2020.  _Neuro-oncology_ , 25(Supplement_4):iv1–iv99, 2023.
  * Goldbrunner et al. (2021) Roland Goldbrunner, Pantelis Stavrinou, Michael D Jenkinson, Felix Sahm, Christian Mawrin, Damien C Weber, Matthias Preusser, Giuseppe Minniti, Morten Lund-Johansen, Florence Lefranc, et al.  Eano guideline on the diagnosis and management of meningiomas.  _Neuro-oncology_ , 23(11):1821–1834, 2021.
  * Yano and Kuratsu (2006) Shigetoshi Yano and Jun-ichi Kuratsu.  Indications for surgery in patients with asymptomatic meningiomas based on an extensive experience.  _Journal of neurosurgery_ , 105(4):538–543, 2006.
  * Zhao et al. (2020) Lianhua Zhao, Wei Zhao, Yanwei Hou, Cuixia Wen, Jing Wang, Pei Wu, and Zaiyu Guo.  An overview of managements in meningiomas.  _Frontiers in oncology_ , 10:1523, 2020.
  * Simpson (1957) Donald Simpson.  The recurrence of intracranial meningiomas after surgical treatment.  _Journal of neurology, neurosurgery, and psychiatry_ , 20(1):22, 1957.
  * Louis et al. (2021) David N Louis, Arie Perry, Pieter Wesseling, Daniel J Brat, Ian A Cree, Dominique Figarella-Branger, Cynthia Hawkins, HK Ng, Stefan M Pfister, Guido Reifenberger, et al.  The 2021 who classification of tumors of the central nervous system: a summary.  _Neuro-oncology_ , 23(8):1231–1251, 2021.
  * Saraf et al. (2011) Santosh Saraf, Bridget J McCarthy, and J Lee Villano.  Update on meningiomas.  _The oncologist_ , 16(11):1604–1613, 2011.
  * Kent et al. (2022) Collin L Kent, Yvonne M Mowery, Olayode Babatunde, Ato O Wright, Ian Barak, Frances McSherry, James E Herndon II, Allan H Friedman, Ali Zomorodi, Katherine Peters, et al.  Long-term outcomes for patients with atypical or malignant meningiomas treated with or without radiation therapy: A 25-year retrospective analysis of a single-institution experience.  _Advances in Radiation Oncology_ , 7(3):100878, 2022.
  * Pinzi et al. (2023) Valentina Pinzi, Marcello Marchetti, Anna Viola, Irene Tramacere, Irene Cane, Cecilia Iezzoni, and Laura Fariselli.  Hypofractionated radiosurgery for large or in critical-site intracranial meningioma: results of a phase 2 prospective study.  _International Journal of Radiation Oncology* Biology* Physics_ , 115(1):153–163, 2023.
  * Rogers et al. (2017) Leland Rogers, Peixin Zhang, Michael A Vogelbaum, Arie Perry, Lynn S Ashby, Jignesh M Modi, Anthony M Alleman, James Galvin, David Brachman, Joseph M Jenrette, et al.  Intermediate-risk meningioma: initial outcomes from nrg oncology rtog 0539.  _Journal of neurosurgery_ , 129(1):35–47, 2017.
  * Rogers et al. (2020) C Leland Rogers, Minhee Won, Michael A Vogelbaum, Arie Perry, Lynn S Ashby, Jignesh M Modi, Anthony M Alleman, James Galvin, Shannon E Fogh, Emad Youssef, et al.  High-risk meningioma: initial outcomes from nrg oncology/rtog 0539.  _International Journal of Radiation Oncology* Biology* Physics_ , 106(4):790–799, 2020.
  * Weber et al. (2018) Damien C Weber, Carmen Ares, Salvador Villa, Saskia M Peerdeman, Laurette Renard, Brigitta G Baumert, Anna Lucas, Theo Veninga, Alessia Pica, Sarah Jefferies, et al.  Adjuvant postoperative high-dose radiotherapy for atypical and malignant meningioma: a phase-ii parallel non-randomized and observation study (eortc 22042-26042).  _Radiotherapy and oncology_ , 128(2):260–265, 2018.
  * Simon and Gousias (2024) Matthias Simon and Konstantinos Gousias.  Grading meningioma resections: the simpson classification and beyond.  _Acta Neurochirurgica_ , 166(1):28, 2024.
  * Aghi et al. (2009) Manish K Aghi, Bob S Carter, Garth R Cosgrove, Robert G Ojemann, Sepideh Amin-Hanjani, Robert L Martuza, William T Curry Jr, Fred G Barker, et al.  Long-term recurrence rates of atypical meningiomas after gross total resection with or without postoperative adjuvant radiation.  _Neurosurgery_ , 64(1):56–60, 2009.
  * Hug et al. (2000) Eugen B Hug, Alexander DeVries, Allan F Thornton, John E Munzenrider, Francisco S Pardo, E Tessa Hedley-Whyte, Marc R Bussiere, and Robert Ojemann.  Management of atypical and malignant meningiomas: role of high-dose, 3d-conformal radiation therapy.  _Journal of neuro-oncology_ , 48:151–160, 2000.
  * Boskos et al. (2009) Christos Boskos, Loic Feuvret, Georges Noel, Jean-Louis Habrand, Pascal Pommier, Claire Alapetite, Hamid Mammar, Regis Ferrand, Gilbert Boisserie, and Jean-Jacques Mazeron.  Combined proton and photon conformal radiotherapy for intracranial atypical and malignant meningioma.  _International Journal of Radiation Oncology* Biology* Physics_ , 75(2):399–406, 2009.
  * Kessel et al. (2017) Kerstin A Kessel, Josefine Hesse, Christoph Straube, Claus Zimmer, Friederike Schmidt-Graf, Jürgen Schlegel, Bernhard Meyer, and Stephanie E Combs.  Modification and optimization of an established prognostic score after re-irradiation of recurrent glioma.  _PLoS One_ , 12(7):e0180457, 2017.
  * LaBella et al. (2023) Dominic LaBella, Maruf Adewole, Michelle Alonso-Basanta, Talissa Altes, Syed Muhammad Anwar, Ujjwal Baid, Timothy Bergquist, Radhika Bhalerao, Sully Chen, Verena Chung, et al.  The asnr-miccai brain tumor segmentation (brats) challenge 2023: Intracranial meningioma.  _arXiv preprint arXiv:2305.07642_ , 2023.
  * LaBella et al. (2024a) Dominic LaBella, Omaditya Khanna, Shan McBurney-Lin, Ryan Mclean, Pierre Nedelec, Arif S. Rashid, Nourel hoda Tahon, Talissa Altes, Ujjwal Baid, Radhika Bhalerao, Yaseen Dhemesh, Scott Floyd, Devon Godfrey, Fathi Hilal, Anastasia Janas, Anahita Kazerooni, Collin Kent, John Kirkpatrick, Florian Kofler, Kevin Leu, Nazanin Maleki, Bjoern Menze, Maxence Pajot, Zachary J. Reitman, Jeffrey D. Rudie, Rachit Saluja, Yury Velichko, Chunhao Wang, Pranav I. Warman, Nico Sollmann, David Diffley, Khanak K. Nandolia, Daniel I Warren, Ali Hussain, John Pascal Fehringer, Yulia Bronstein, Lisa Deptula, Evan G. Stein, Mahsa Taherzadeh, Eduardo Portela de Oliveira, Aoife Haughey, Marinos Kontzialis, Luca Saba, Benjamin Turner, Melanie M. T. Brüßeler, Shehbaz Ansari, Athanasios Gkampenis, David Maximilian Weiss, Aya Mansour, Islam H. Shawali, Nikolay Yordanov, Joel M. Stein, Roula Hourani, Mohammed Yahya Moshebah, Ahmed Magdy Abouelatta, Tanvir Rizvi, Klara Willms, Dann C. Martin, Abdullah Okar, Gennaro D’Anna, Ahmed Taha, Yasaman Sharifi, Shahriar Faghani, Dominic Kite, Marco Pinho, Muhammad Ammar Haider, Michelle Alonso-Basanta, Javier Villanueva-Meyer, Andreas M. Rauschecker, Ayman Nada, Mariam Aboian, Adam Flanders, Spyridon Bakas, and Evan Calabrese.  A multi-institutional meningioma mri dataset for automated multi-sequence image segmentation.  _Scientific Data_ , 11:496, 2024a.  10.1038/s41597-024-03350-9.  URL https://doi.org/10.1038/s41597-024-03350-9.
  * Bakas et al. (2017) Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  _Scientific data_ , 4(1):1–13, 2017.
  * Menze et al. (2014) Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * Kazerooni et al. (2023) Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, et al.  The brain tumor segmentation (brats) challenge 2023: Focus on pediatrics (cbtn-connect-dipgr-asnr-miccai brats-peds).  _ArXiv_ , 2023.
  * Moawad et al. (2023) Ahmed W Moawad, Anastasia Janas, Ujjwal Baid, Divya Ramakrishnan, Leon Jekel, Kiril Krantchev, Harrison Moy, Rachit Saluja, Klara Osenberg, Klara Wilms, et al.  The brain tumor segmentation (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri.  _ArXiv_ , 2023.
  * Adewole et al. (2023) Maruf Adewole, Jeffrey D Rudie, Anu Gbdamosi, Oluyemisi Toyobo, Confidence Raymond, Dong Zhang, Olubukola Omidiji, Rachel Akinola, Mohammad Abba Suwaid, Adaobi Emegoakor, et al.  The brain tumor segmentation (brats) challenge 2023: Glioma segmentation in sub-saharan africa patient population (brats-africa).  _ArXiv_ , 2023.
  * Ermiş et al. (2020) Ekin Ermiş, Alain Jungo, Robert Poel, Marcela Blatti-Moreno, Raphael Meier, Urspeter Knecht, Daniel M Aebersold, Michael K Fix, Peter Manser, Mauricio Reyes, et al.  Fully automated brain resection cavity delineation for radiation target volume definition in glioblastoma patients using deep learning.  _Radiation oncology_ , 15:1–10, 2020.
  * Bianconi et al. (2023) Andrea Bianconi, Luca Francesco Rossi, Marta Bonada, Pietro Zeppa, Elsa Nico, Raffaele De Marco, Paola Lacroce, Fabio Cofano, Francesco Bruno, Giovanni Morana, et al.  Deep learning-based algorithm for postoperative glioblastoma mri segmentation: a promising new tool for tumor burden assessment.  _Brain Informatics_ , 10(1):26, 2023.
  * Bischoff-Grethe et al. (2007) Amanda Bischoff-Grethe, I Burak Ozyurt, Evelina Busa, Brian T Quinn, Christine Fennema-Notestine, Camellia P Clark, Shaunna Morris, Mark W Bondi, Terry L Jernigan, Anders M Dale, et al.  A technique for the deidentification of structural brain mr images.  _Human brain mapping_ , 28(9):892–903, 2007.
  * Rohlfing et al. (2010) Torsten Rohlfing, Natalie M Zahr, Edith V Sullivan, and Adolf Pfefferbaum.  The sri24 multichannel atlas of normal adult human brain structure.  _Human brain mapping_ , 31(5):798–819, 2010.
  * Wiant and Bourland (2009) D Wiant and JD Bourland.  Simulated gamma knife™ head frame placement for radiosurgical pre-planning.  _Technology in cancer research & treatment_, 8(4):265–270, 2009.
  * Gallagher et al. (2008) Thomas A Gallagher, Alexander J Nemeth, and Lotfi Hacein-Bey.  An introduction to the fourier transform: relationship to mri.  _American journal of roentgenology_ , 190(5):1396–1405, 2008.
  * Cox (1996) Robert W Cox.  Afni: software for analysis and visualization of functional magnetic resonance neuroimages.  _Computers and Biomedical research_ , 29(3):162–173, 1996.
  * Cox and Hyde (1997) Robert W Cox and James S Hyde.  Software tools for analysis and visualization of fmri data.  _NMR in Biomedicine: An International Journal Devoted to the Development and Application of Magnetic Resonance In Vivo_ , 10(4-5):171–178, 1997.
  * Phil et al. (2023) Thomas Phil, Thomas Albrecht, Skylar Gay, and Mathis Ersted Rasmussen.  Sikerdebaard/dcmrtstruct2nii: dcmrtstruct2nii v5 (version v5), 2023.  URL https://doi.org/10.5281/zenodo.4037864.
  * Theyers et al. (2021) Athena E Theyers, Mojdeh Zamyadi, Mark O’Reilly, Robert Bartha, Sean Symons, Glenda M MacQueen, Stefanie Hassel, Jason P Lerch, Evdokia Anagnostou, Raymond W Lam, et al.  Multisite comparison of mri defacing software across multiple cohorts.  _Frontiers in psychiatry_ , 12:617997, 2021.
  * Yushkevich et al. (2006) Paul A Yushkevich, Joseph Piven, Heather Cody Hazlett, Rachel Gimpel Smith, Sean Ho, James C Gee, and Guido Gerig.  User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability.  _Neuroimage_ , 31(3):1116–1128, 2006.
  * Isensee et al. (2021) Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  _Nature methods_ , 18(2):203–211, 2021.
  * Pati et al. (2022) Sarthak Pati, Ujjwal Baid, Brandon Edwards, Micah J Sheller, Patrick Foley, G Anthony Reina, Siddhesh Thakur, Chiharu Sako, Michel Bilello, Christos Davatzikos, et al.  The federated tumor segmentation (fets) tool: an open-source solution to further solid tumor research.  _Physics in Medicine & Biology_, 67(20):204002, 2022.
  * Huang et al. (2019) Raymond Y Huang, Wenya Linda Bi, Brent Griffith, Timothy J Kaufmann, Christian la Fougère, Nils Ole Schmidt, Jöerg C Tonn, Michael A Vogelbaum, Patrick Y Wen, Kenneth Aldape, et al.  Imaging and diagnostic advances for intracranial meningiomas.  _Neuro-oncology_ , 21(Supplement_1):i44–i61, 2019.
  * Stupp et al. (2005) Roger Stupp, Warren P Mason, Martin J Van Den Bent, Michael Weller, Barbara Fisher, Martin JB Taphoorn, Karl Belanger, Alba A Brandes, Christine Marosi, Ulrich Bogdahn, et al.  Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma.  _New England journal of medicine_ , 352(10):987–996, 2005.
  * Sheehan et al. (2013) Jason P Sheehan, Robert M Starke, David Mathieu, Byron Young, Penny K Sneed, Veronica L Chiang, John YK Lee, Hideyuki Kano, Kyung-Jae Park, Ajay Niranjan, et al.  Gamma knife radiosurgery for the management of nonfunctioning pituitary adenomas: a multicenter study.  _Journal of neurosurgery_ , 119(2):446–456, 2013.
  * Hargrave et al. (2006) Darren Hargrave, Ute Bartels, and Eric Bouffet.  Diffuse brainstem glioma in children: critical review of clinical trials.  _The lancet oncology_ , 7(3):241–248, 2006.
  * Yang et al. (2011) Huai-che Yang, Hideyuki Kano, Nasir Raza Awan, L Dade Lunsford, Ajay Niranjan, John C Flickinger, Josef Novotny, Jagdish P Bhatnagar, and Douglas Kondziolka.  Gamma knife radiosurgery for larger-volume vestibular schwannomas.  _Journal of neurosurgery_ , 114(3):801–807, 2011.
  * Ye et al. (2022) Xianghua Ye, Dazhou Guo, Jia Ge, Senxiang Yan, Yi Xin, Yuchen Song, Yongheng Yan, Bing-shen Huang, Tsung-Min Hung, Zhuotun Zhu, et al.  Comprehensive and clinically accurate head and neck cancer organs-at-risk delineation on a multi-institutional study.  _Nature communications_ , 13(1):6137, 2022.
  * LaBella et al. (2024b) Dominic LaBella, Evan Calabrese, et al.  Brats-men-rt challenge.  https://www.synapse.org/#!Synapse:syn53708249/wiki/627503, 2024b.  Accessed: 2024-05-24.


# Solving the HP model with Nested Monte Carlo Search

Milo Roucairol, Tristan Cazenave

###### Abstract

In this paper we present a new Monte Carlo Search (MCS) algorithm for finding
the ground state energy of proteins in the HP-model. We also compare it
briefly to other MCS algorithms not usually used on the HP-model and provide
an overview of the algorithms used on HP-model.
The algorithm presented in this paper does not beat state of the art
algorithms, see PERM (Hsu and Grassberger 2011), REMC (Thachuk, Shmygelska,
and Hoos 2007) or WLRE (Wüst and Landau 2012) for better results.

## Introduction

Monte Carlo search algorithms have proven to be powerful as game playing
agents, with recent successes like AlphaGo(Silver et al. 2016). These
algorithms have the advantage of only needing an evaluation function for the
final state of the space they explore.

Protein folding is crucial to our understanding of biology and designing
drugs, however, trying our algorithms directly on accurate models could be
counterproductive. In this paper, we use a new MCS algorithm to fold proteins
in a simplified lattice based model called the HP model.

First we will present the protein folding and the HP model, then the different
algorithms we used to explore the problem space and finally the results of our
experiments.

## The problem

### Protein folding

With recent developments in ARNm technology, it is now possible to incite
cells to produce a specific protein (Gros et al. 1961), like the spike protein
used in COVID-19 vaccines. Unfortunately, deducing the shape a protein will
take based given the amino acids sequence is not obvious nor trivial. That is
a reason protein folding is a very important problem in molecular biology and
medicine.

Proteins are chains of amino acids (primary structure), they can fold in many
different ways, the secondary structure is the shape the protein will take at
a local level (a coil for example), the tertiary structure is the global shape
of the protein with less discernible patterns, finally the quaternary
structure is how a protein can assemble with another. Here we are interested
in predicting the ground state energy folding (secondary and tertiary
structure) from the primary structure. Many forces drive the folding, which
prevents the creation of a very accurate simulator, the main driving force is
the hydrophobic one.

One can not approach protein folding without mentioning DeepMind’s AlphaFold
(Jumper et al. 2021). Placing first at the Critical Assessment of Techniques
for Protein Structure Prediction in 2018 and 2020, it is the best program for
protein structure prediction yet. AlphaFold uses machine learning on a large
protein database to train neural networks, in addition to physics based rules
in order to predict the folding of a protein.

AlphaFold is the greatest achievement to protein folding prediction in
decades, but the research is not over yet. AlphaFold accuracy can still be
perfected, and by using neural networks the explainability is low and the
model may not be able to predict structure of proteins never seen before. Our
objective here is to provide a better algorithm for Monte Carlo physics
simulation, which may be more explainable than AlphaFold, but is currently way
less accurate.

### HP model

The main idea behind the creation of the HP model is that the Hydrophobic-
Polar (HP) force is the main force driving the folding of a protein, thus it
is the only one used here.

The HP model is a very simplified lattice based model for protein folding, it
exists in 2D and 3D versions.

In the HP model, proteins are represented as a chain of H and P residues
(amino acids), the chain is then folded onto a grid, two residues can not
share the same positions. The residue contacts determine the energy of a
chain, usually, the reward for an H-H connection is -1, and 0 for H-P and P-P
contacts in a context of minimisation (since the ground energy state is the
state with the least potential energy). Other rewards can be used to obtain
different results or guide the search.

### State of the art on HP model

The HP model was introduced in 1985 by Ken Dill (Dill 1985), it has seen a
number of algorithms trying to solve it. All of the best performing algorithms
on the HP model are Monte Carlo based, policy learning using neural networks
or reinforcement learning like NRPA (Rosin 2011) led to poor results. In these
Monte Carlo algorithms, we can identify two types, the chain growth algorithms
and the replica exchange ones.

The chain growth methods add the residues one after the other, next to the
previous one, it is similar to a self avoiding walk and it is the method we
used in our own algorithm.

The replica exchange methods use pull moves, pulling the chain at one point by
rotating a residue around one of its neighbors, symmetrically rotating a part
of the chain or pulling from one side of the chain. This means the entirety of
the chain is present on the lattice at any given state of the research, and is
in a physically possible conformation, this method is used in simulated
annealing like Monte Carlo algorithms. To see a representation of these moves
check Chris Thchuk, Alena Shmygelska and Holger H Hoos REMC article (Thachuk,
Shmygelska, and Hoos 2007).

Here is a short review of the methods we encountered.

1) PERM : Initially used on Self Avoiding Walks (SAW), PERM is a chain growth
algorithm and was used on the HP model by Peter Grassberger in 1997
(Grassberger ). It stands for Pruned Enriched Rosenbluth Method, the idea is
to explore the possible chains uniformly with a bias on the immediate gain,
cutting (pruning) branches leading to too few choices and poor performances,
and cloning (enriching) branches that lead to great results. PERM has seen
many new versions until 2011 (Hsu and Grassberger 2011), mainly proposed by
its creator, Peter Grassberger. It still is one of the best algorithms
available but has been outperformed by pull-moves based more recent
algorithms.

2) REMC : Introduced in 2007 by Chris Thachuk, Alena Shmygelska and Holger H
Hoos (Thachuk, Shmygelska, and Hoos 2007), the Replica Exchange Monte Carlo
algorithm uses pull moves and simulated annealing. That algorithm keeps only a
certain number of replicas (it was determined the best number of replicas for
the 3D HP-model was 2), each with a given temperature. At each step the
algorithm mutates each replica with a Monte-Carlo Search using the pull moves,
the probabilities of keeping a mutation are decided by the score gain (energy
loss) of the mutated state and the temperature. Then, once each state is
produced through mutation, the replicas are then again swapped
probabilistically according to their score (energy) and temperatures.

3) Wang-Landau sampling : Introduced in 2012 on the HP model, the Wang-Landau
sampling (WLS) (Wüst and Landau 2012) method seems to be the new best
algorithm for solving the HP model. It is a replica-exchange (simulated
annealing) algorithm that uses the same pull moves as the REMC, but also uses
moves consisting in cutting and joining of the molecule (thus reallocating all
the residues according to their position), together they are named the Monte-
Carlo trial moves. With these moves, the WLS explores the conformation space
to estimate a histogram of the energies of these conformations. With this
histogram, the WLS can then direct the exchange of the replicas.

## Algorithm

### Biased Growth

In a similar way to the PERM algorithm (Hsu and Grassberger 2011), we try to
favor the immediate reward when building/folding the molecule. To do this, we
use biased playouts, the chances of selecting a move $m$ from $M$ the possible
moves with an immediate gain $G$ follows a softmax distribution with $b$ the
bias factor:

$\frac{exp(G[m]*b)}{sum([exp(G[i]*b)for\;i\;in\;M])}$.

$M_{c\mbox{-}state}$ denotes the legal moves available from the state
$current\mbox{-}state$.
$G_{M_{c\mbox{-}state}}$ denotes the immediate gains of each legal moves
available from the state $current\mbox{-}state$.

1:function playout($c\mbox{-}state$, $b$)

2: $ply\leftarrow 0$

3: $seq\leftarrow\\{\\}$

4: while $c\mbox{-}state$ is not terminal do

5: $gains\leftarrow G_{M_{c\mbox{-}state}}$

6: $move\leftarrow softMaxChoice(M_{c\mbox{-}state},gains*b)$

7: $current\mbox{-}state\leftarrow play(c\mbox{-}state,move)$

8: $seq[ply]\leftarrow move$

9: $ply+=1$

10: end while

11: return $score(current\mbox{-}state),seq$

12:end function

Algorithm 1 The biased growth playout algorithm.

### Nested Monte Carlo Search

1:function NMCS($c\mbox{-}state$, $l$, $b$)

2: if $l=0$ then return $playout(c\mbox{-}state,b)$

3: else

4: $best\mbox{-}score\leftarrow-\infty$

5: $best\mbox{-}sequence\leftarrow[]$

6: $ply\leftarrow 0$

7: while $c\mbox{-}state$ is not terminal do

8: for each $move$ in $M_{c\mbox{-}state}$ do

9: $n\mbox{-}st\leftarrow play(c\mbox{-}state,move)$

10: $(score,seq)\leftarrow$ NMCS($n\mbox{-}st,l-1,b$)

11: if $score\geq best\mbox{-}score$ then

12: $best\mbox{-}score\leftarrow score$

13: $best\mbox{-}sequence[ply..]\leftarrow move+seq$

14: end if

15: end for

16: $next\mbox{-}move\leftarrow best\mbox{-}sequence[ply]$

17: $ply\leftarrow ply+1$

18: $c\mbox{-}state\leftarrow play(c\mbox{-}state,next\mbox{-}move)$

19: end while

20: return ($best\mbox{-}score,best\mbox{-}sequence$)

21: end if

22:end function

Algorithm 2 The NMCS algorithm.

NMCS (Cazenave 2009) is a Monte Carlo Search algorithm that recursively calls
lower level NMCS on children states of the current state in order to decide
which move to play next, the lowest level of NMCS being a random playout,
selecting uniformly the move to execute among the possible moves. A heuristic
can be added to the playout move choices, and it is the case here with the
biased growth playouts.

Algorithm 2 gives the NMCS algorithm, $l$ is the nesting level and $b$ the
playout bias.

### Lazy Nested Monte Carlo Search

1:$tr\leftarrow[]$

2:function LNMCS($c\mbox{-}st$, $l$, $b$, $p$, $r$)

3: if $level=0$ then return $\textsc{playout}(c\mbox{-}st,b)$

4: else

5: $best\mbox{-}score\leftarrow-\infty$

6: $best\mbox{-}sq\leftarrow[]$

7: $ply\leftarrow 0$

8: while $c\mbox{-}st$ is not terminal do

9: for each $move$ in $M_{c\mbox{-}state}$ do

10: $n\mbox{-}st\leftarrow play(c\mbox{-}st,move)$

11: for $i$ in $0..p$ do

12: $(playoutSc,\\_)\leftarrow playout(n\mbox{-}st,b)$

13: $es\leftarrow es+playoutSc/p$

14: end for

15: if $tr.length()<c\mbox{-}st.nbplay+1$ then

16: $tr.push(0.0)$

17: end if

18: if $tr[c\mbox{-}state.nbplay]<es$ then

19: $tr[c\mbox{-}state.nbplay]=es$

20: end if

21: if $es<ratio*tr[c\mbox{-}st.nbplay]$ then

22: $(sc,sq)\leftarrow$LNMCS($c.1,0,b,p,r$)

23: else

24: $(sc,sq)\leftarrow$LNMCS($c.1,l-1,b,p,r$)

25: end if

26: if $sc\geq best\mbox{-}score$ then

27: $best\mbox{-}score\leftarrow sc$

28: $best\mbox{-}sq[ply..]\leftarrow move+sq$

29: end if

30: end for

31: $next\mbox{-}move\leftarrow best\mbox{-}sq[ply]$

32: $ply\leftarrow ply+1$

33: $c\mbox{-}st\leftarrow play(c\mbox{-}st,next\mbox{-}move)$

34: end while

35: return ($best\mbox{-}score,best\mbox{-}sq$)

36: end if

37:end function

Algorithm 3 The Lazy NMCS algorithm.

The lazy NMCS inherits its main features from the NMCS, but solves an obstacle
encountered on this problem. Solving the 3D HP model with the NMCS requires
using a level of 4 at least, however, it requires computing many 3 level NMCS,
already very costly, one for each possible move the level 4 NMCS can make. The
main idea behind the lazy NMCS is that there are moves that lead to low
potential states, to do so, we estimate the potential of a state by launching
a number of biased growth playouts and calculating the mean of their scores,
then we compare that score to a threshold (relative to the number of moves
already done) calculated from the previous estimations to decide if we want to
expand the search tree from this state, or prune it. To update the pruning
threshold, it is possible to use a mean, a median or a max from the previous
estimations, here we use the max as it gave the best results on these
problems.

In the following pseudocode in algorithm 3, $p$ is the number of playouts used
to evaluate a state and $r$ is the ratio to the threshold a state will be
pruned on. $l$ is the nesting level and $b$ is the playout bias.

From line 9 to line 14, the state is evaluated with the mean of $p$ playouts.

From line 13 to line 15, the threshold list is extended on the first entry of
a new molecule length, this step is not needed if the list is initialized with
the right size from the start for problems we know the maximum number of moves
that will be played.

From lines 16 to 18, the threshold is updated with the evaluation.

From lines 19 to 23, it is decided with the evaluation, the pruning ratio and
the corresponding threshold if the search will be costly or not.

As you can see there is only one FOR loop iterating over the moves in this
implementation of the LNMCS, it means the evaluation is incomplete when the
algorithm decides whether to prune the first branches or not. This is a minor
flaw in this version of the algorithm and it is easily fixed by ulterior
versions (along with other shortcomings). Nonetheless, the experiments were
made with this ”prototype” version of the algorithm.

## Results

### Lazy Nested Monte Carlo Search

We conducted experiments on the 10 molecules with 48 mers from the benchmark
we can find in Holger’s (Thachuk, Shmygelska, and Hoos 2007) and Hsu’s (Hsu
and Grassberger 2011) work.

ID | molecule | -E*
---|---|---
1 |  HPHHPPHHHHPHHHPPHHPPHPH HHPHPHHPPHHPPPHPPPPPPPPHH | 32
2 |  HHHHPHHPHHHHHPPHPPHHPPH PPPPPPHPPHPPPHPPHHPPHHHPH | 34
3 |  PHPHHPHHHHHHPPHPHPPHPHH PHPHPPPHPPHHPPHHPPHPHPPHP | 34
4 |  PHPHHPPHPHHHPPHHPHHPPPH HHHHPPHPHHPHPHPPPPHPPHPHP | 33
5 |  PPHPPPHPHHHHPPHHHHPHHPH HHPPHPHPHPPHPPPPPPHHPHHPH | 32
6 |  HHHPPPHHPHPHHPHHPHHPHPP PPPPPHPHPPHPPPHPPHHHHHHPH | 32
7 |  PHPPPPHPHHHPHPHHHHPHHPH HPPPHPHPPPHHHPPHHPPHHPPPH | 32
8 |  PHHPHHHPHHHHPPHHHPPPPPP HPHHPPHHPHPPPHHPHPHPHHPPP | 31
9 |  PHPHPPPPHPHPHPPHPHHHHHH PPHHHPHPPHPHHPPHPHHHPPPPH | 34
10 |  PHHPPPPPPHHPPPHHHPHPPHP HHPPHPPHPPHHPPHHHHHHHPPHH | 33
Table 1:  The benchmarks molecules and their speculated lowest energy state

To obtain our results, we used the lazy NMCS with a timeout of 150s, if the
algorithm has not found a conformation with the lowest known energy before the
end of the timeout then we restart the algorithm until that happens. In our
experiments, the playout biased growth gives an immediate gain of 1 to any
legal H-H connection, but also a penalty of -0.2 to the HP connections, this
was made in order to incite the biased growth to keep a maximum of H mers open
to future connections, we did not experiment on that variable.

Molecule 4 used a lazy NMCS with a threshold based on the mean of the
evaluation playouts with the following parameters:

level | 4
---|---
#eval playouts | 10
pruning ratio | 0.97
playout bias | 20

The other molecules used a lazy NMCS with a threshold based on the best
average from a batch of evaluation playouts with the following parameters:

level | 5
---|---
#eval playouts | 20
pruning ratio | 0.9
playout bias | 20

Different methods of evaluation and pruning can greatly change the performance
of the algorithm, and some methods can be ineffective on a set of molecules
while being capable on another set.

ID | mean time | interquartile
---|---|---
1 | 5.5 | 5
2 | 12.5 | 15.5
3 | 10 | 14
4 | 20 | 25
5 | 10 | 10
6 | +- 180 | -
7 | +- 60 | -
8 | 13.5 | 12
9 | +- 120 | -
10 | 7 | 9
Table 2:  Mean time and interquatile (in minutes) of LNMCS on the benchmark
molecules

These results are displayed in minutes and were obtained on a 3.50GHz Intel
core i5-6600K CPU.

Our LNMCS performed very poorly on molecules 6 and 7, we were not able to
gather enough data to compute the statistics. This was unexpected since only
molecules 4 and 9 are difficult for PERM (the state of the art chain growth
algorithm) to solve according to Grassberger and Hsu’s latest paper (Hsu and
Grassberger 2011), and molecule 4 posed less problems. However, the lazy NMCS
could attain the second best energy level very reliably in less than 150s in
half of the launches with both molecules.

LNMCS was also able to easily reach the second best level of energy on
molecule 9 but could only reach the optimal state every 2 hours approximately,
that result was expected since PERM encounters difficulties with that molecule
too.

### Other MCS algorithms

We tried to solve The HP model with a variety of different Monte Carlo
algorithms. We only applied these algorithms to the first molecule from the
benchmark (referred as ”the molecule” in this section), the results presented
in this section are only to give an idea of these algorithms’ performances and
do not necessarily reflect their potential on the HP model.

#### Nested Monte Carlo Search

The good performances of the NMCS (Cazenave 2009) compared to the other
algorithms presented in this section is what decided us to try to improve it
for this problem into the LNMCS: the NMCS was able to find the optimal value
on the molecule in less than 10mn.

In Figure 2 and Figure 1 we compare performances of the level 5 NMCS and the
level 5 LNMCS with a ratio of 0.9 on molecule 1, with both a playout bias of
20 over 20 runs with a 150s timeout.

$-32$$-31$$-30$$-29$$0$$5$$10$$15$EnergyN Figure 1:  Energy distribution with
the Lazy NMCS $-32$$-31$$-30$$-29$$0$$5$$10$$15$EnergyN Figure 2:  Energy
distribution with the NMCS

As you can see on these figures, the LNMCS provides a substantial performance
gain over the NMCS on this problem. Lowest energy conformations are way
sparser than the second lowest energy conformations (about an order of
magnitude or two), being able to reach them 4 times as often is a great
improvement.

#### Nested Rollout Policy Adaptation

The NRPA (Rosin 2011) is similar to the NMCS, the main difference is that the
NRPA learns a policy to decide which move play during playouts, that policy
discovery is interesting for many problems which are too complex to implement
a man made policy (like we did here). On many problems the NRPA outperforms
the NMCS, however in our case it was not able to do. The performance of the
NRPA and GNRPA (Cazenave 2020) are widely dependent on the move
representations, here it is the number corresponding to the number of residues
already placed and its direction and NRPA and GNRPA with a bias of 20 were not
able to reach the optimal energies (-28 or -29 for the GNRPA when -32 is the
best known). Other moves representations were tried, using the last few
previous moves instead of the number of residues already placed for example,
but none were able to provide better performances. Our inability to obtain
good results with NRPA does not mean it is impossible to solve this problem
with it.

#### Greedy Best First Search with playouts

The Greedy BFS (Doran and Michie 1966) is a simple search algorithm that uses
a ranked list of the nodes to open according to their scores given by an
evaluation function. Iteratively, the Greedy BFS opens the best node from the
list and launches the evaluation function on every child of this node to
insert them in the ranked list. Here we evaluate the children with their
results with one or multiple playouts. This method converges rapidly to a
”good enough” solution (a local minimum), -29 when the best known is -32 on
molecule 1, but then improves very little, it is due to a large number of good
scoring states that do not lead to an optimal solution, making the search too
exhaustive. Pruning the search tree could improve the results of this method.

#### Upper Confidence bounds applied to Trees

UCT (Kocsis and Szepesvári 2006) is used on game playing and usually provides
good results. It iteratively starts from the initial state and launches
playouts, based on the results of each playout, the value used to determine
which moves should be taken is updated. UCT does not work well on the HP-model
without biased growth (achieving about the same scores as a single biased
playout, -18), with biased growth it achieves scores around -28/-29 on
molecule 1, like the other non NMCS algorithms discussed here.

## Conclusion

While the Lazy NMCS algorithm likely does not outperform the state of the art
algorithms, it has the advantage to be easier to implement and may be
applicable to more problems. It is also shown to be an improvement on the NMCS
algorithm for this specific problem. In future works, we aim to apply it to
other problems and find ways to improve its performance.

## References

  * Cazenave (2009) Cazenave, T. 2009.  Nested Monte-Carlo Search.  In Boutilier, C., ed., _IJCAI_ , 456–461.
  * Cazenave (2020) Cazenave, T. 2020.  Generalized Nested Rollout Policy Adaptation.  In _Monte Carlo Search at IJCAI_.
  * Dill (1985) Dill, K. A. 1985.  Theory for the folding and stability of globular proteins.  _Biochemistry_ , 24(6): 1501–1509.
  * Doran and Michie (1966) Doran, J. E.; and Michie, D. 1966.  Experiments with the graph traverser program.  _Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences_ , 294(1437): 235–259.
  * (5) Grassberger, P. ????  Pruned-enriched Rosenbluth method: Simulations of polymers of chain length up to 1 000 000.
  * Gros et al. (1961) Gros, F.; Gilbert, W.; Hiatt, H. H.; Attardi, G.; Spahr, P. F.; and Watson, J. D. 1961.  Molecular and Biological Characterization of Messenger RNA.  _Cold Spring Harbor Symposia on Quantitative Biology_ , 26: 111–132.  Publisher: Cold Spring Harbor Laboratory Press.
  * Hsu and Grassberger (2011) Hsu, H.-P.; and Grassberger, P. 2011.  A review of Monte Carlo simulations of polymers with PERM.  _Journal of Statistical Physics_ , 144(3): 597–637.
  * Jumper et al. (2021) Jumper, J.; Evans, R.; Pritzel, A.; Green, T.; Figurnov, M.; Ronneberger, O.; Tunyasuvunakool, K.; Bates, R.; Žídek, A.; Potapenko, A.; Bridgland, A.; Meyer, C.; Kohl, S. A. A.; Ballard, A. J.; Cowie, A.; Romera-Paredes, B.; Nikolov, S.; Jain, R.; Adler, J.; Back, T.; Petersen, S.; Reiman, D.; Clancy, E.; Zielinski, M.; Steinegger, M.; Pacholska, M.; Berghammer, T.; Bodenstein, S.; Silver, D.; Vinyals, O.; Senior, A. W.; Kavukcuoglu, K.; Kohli, P.; and Hassabis, D. 2021.  Highly accurate protein structure prediction with AlphaFold.  _Nature_ , 596(7873): 583–589.
  * Kocsis and Szepesvári (2006) Kocsis, L.; and Szepesvári, C. 2006.  Bandit Based Monte-Carlo Planning.  In _Machine Learning: ECML 2006_ , volume 4212, 282–293. Berlin, Heidelberg: Springer Berlin Heidelberg.  ISBN 978-3-540-45375-8 978-3-540-46056-5.  Series Title: Lecture Notes in Computer Science.
  * Rosin (2011) Rosin, C. D. 2011.  Nested Rollout Policy Adaptation for Monte Carlo Tree Search.  In _IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence_ , 649–654.
  * Silver et al. (2016) Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; van den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham, J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T.; Leach, M.; Kavukcuoglu, K.; Graepel, T.; and Hassabis, D. 2016\.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529: 484–489.
  * Thachuk, Shmygelska, and Hoos (2007) Thachuk, C.; Shmygelska, A.; and Hoos, H. H. 2007.  A replica exchange Monte Carlo algorithm for protein folding in the HP model.  _BMC Bioinformatics_ , 8(1): 342.
  * Wüst and Landau (2012) Wüst, T.; and Landau, D. P. 2012.  Optimized Wang-Landau sampling of lattice polymers: Ground state search and folding thermodynamics of HP model proteins.  _The Journal of Chemical Physics_ , 137(6): 064903.