# MRSegmentator: Robust Multi-Modality Segmentation of 40 Classes in MRI and
CT Sequences

Hartmut Häntze * Department of Radiology, Charité - Universitätsmedizin Berlin
corporate member of Freie Universität Berlin and Humboldt Universität zu
Berlin, Hindenburgdamm 30, 12203 Berlin, Germany  Diagnostic Image Analysis
Group, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA
Nijmegen, the Netherlands  Lina Xu * Department of Radiology, Charité -
Universitätsmedizin Berlin corporate member of Freie Universität Berlin and
Humboldt Universität zu Berlin, Hindenburgdamm 30, 12203 Berlin, Germany
Felix J. Dorfner  Department of Radiology, Charité - Universitätsmedizin
Berlin corporate member of Freie Universität Berlin and Humboldt Universität
zu Berlin, Hindenburgdamm 30, 12203 Berlin, Germany  Athinoula A. Martinos
Center for Biomedical Imaging, Massachusetts General Hospital and Harvard
Medical School, 149 Thirteenth St, Charlestown, MA 02129, USA  Leonhard Donle
Department of Radiology, Charité - Universitätsmedizin Berlin corporate member
of Freie Universität Berlin and Humboldt Universität zu Berlin, Hindenburgdamm
30, 12203 Berlin, Germany  Felix Busch  Department of Radiology, Klinikum
rechts der Isar, Technical University Munich, Ismaninger Str. 22, 81675
Munich, Germany  Markus R Makowski  Department of Radiology, Klinikum rechts
der Isar, Technical University Munich, Ismaninger Str. 22, 81675 Munich,
Germany  Avan Kader  Department of Radiology, Klinikum rechts der Isar,
Technical University Munich, Ismaninger Str. 22, 81675 Munich, Germany  Hugo
Aerts  Artificial Intelligence in Medicine (AIM) Program, Mass General
Brigham, Harvard Medical School, 221 Longwood Avenue, Boston, MA 02115, USA
Departments of Radiation Oncology and Radiology, Dana-Farber Cancer Institute
and Brigham and Women’s Hospital, 221 Longwood Avenue, Boston, MA 02115, USA
Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University,
Minderbroedersberg 4-6, 6211 LK Maastricht, Netherlands  Daniel Truhn
Department of Diagnostic and Interventional Radiology, University Hospital
Aachen, Pauwelsstr. 30, 52074 Aachen, Germany  Mathias Prokop  Diagnostic
Image Analysis Group, Radboud University Medical Center, Geert Grooteplein
Zuid 10, 6525 GA Nijmegen, the Netherlands  Bram van Ginneken  Diagnostic
Image Analysis Group, Radboud University Medical Center, Geert Grooteplein
Zuid 10, 6525 GA Nijmegen, the Netherlands  Alessa Hering  Diagnostic Image
Analysis Group, Radboud University Medical Center, Geert Grooteplein Zuid 10,
6525 GA Nijmegen, the Netherlands  Lisa C. Adams $\dagger$ Department of
Radiology, Klinikum rechts der Isar, Technical University Munich, Ismaninger
Str. 22, 81675 Munich, Germany  Keno K. Bressem $\dagger$$\ddagger$ Department
of Radiology, Klinikum rechts der Isar, Technical University Munich,
Ismaninger Str. 22, 81675 Munich, Germany  German Heart Center Munich,
Technical University Munich, Lazarethstr. 36, 80636 Munich, Germany

(10.05.2024)

###### Abstract

Purpose: To introduce a deep learning model capable of multi-organ
segmentation in MRI scans, offering a solution to the current limitations in
MRI analysis due to challenges in resolution, standardized intensity values,
and variability in sequences.

Materials and Methods: The model was trained on 1,200 manually annotated MRI
scans from the UK Biobank, 221 in-house MRI scans and 1228 CT scans,
leveraging cross-modality transfer learning from CT segmentation models. A
human-in-the-loop annotation workflow was employed to efficiently create high-
quality segmentations. The model’s performance was evaluated on NAKO and the
AMOS22 dataset containing 600 and 60 MRI examinations. Dice Similarity
Coefficient (DSC) and Hausdorff Distance (HD) was used to assess segmentation
accuracy. The model will be open sourced.

Results: The model showcased high accuracy in segmenting well-defined organs,
achieving Dice Similarity Coefficient (DSC) scores of 0.97 for the right and
left lungs, and 0.95 for the heart. It also demonstrated robustness in organs
like the liver (DSC: 0.96) and kidneys (DSC: 0.95 left, 0.95 right), which
present more variability. However, segmentation of smaller and complex
structures such as the portal and splenic veins (DSC: 0.54) and adrenal glands
(DSC: 0.65 left, 0.61 right) revealed the need for further model optimization.

Conclusion: The proposed model is a robust, tool for accurate segmentation of
40 anatomical structures in MRI and CT images. By leveraging cross-modality
learning and interactive annotation, the model achieves strong performance and
generalizability across diverse datasets, making it a valuable resource for
researchers and clinicians. It is open source and can be downloaded from
https://github.com/hhaentze/MRSegmentator.

*These authors contributed equally as first authors.
$\dagger$These authors contributed equally as last authors.
$\ddagger$Corresponding author<EMAIL_ADDRESS>

_Keywords_ MRI, Segmentation, Cross-Modality, Artificial Intelligence,
Radiology

##  1 Introduction

Artificial intelligence (AI)-based segmentation of medical images has numerous
applications, including the calculation of organ volumes, extraction of image-
based biomarkers, identification of regions of interest, and provision of
anatomical context to improve AI-based diagnosis and radiomics pipelines [1].
For instance, accurate localization of a tumor and determination of its
proximity to critical structures can inform treatment planning and prognosis.
Considerable progress has been made in developing open-source segmentation
models for CT images, most notably TotalSegmentator, which can robustly
segment 104 anatomical structures [2]. However, progress in MRI segmentation
has been slower due to several challenges. Compared to CT, MRI images have
lower resolution and lack standardized intensity values, such as Hounsfield
units, making it more difficult for deep learning algorithms to generalize to
new data. MRI also has much greater variability arising from the wide range of
possible sequences and higher prevalence of artifacts, anisotropic voxel
sizes, registration errors, and varying fields of view [3]. Despite these
challenges, MRI segmentation is highly valuable as the multiple available
sequences provide rich information that can potentially yield more imaging
biomarkers and improve diagnostic algorithms compared to CT. While numerous
MRI segmentation models exist, many focus only on the brain [4, 5]. Among
those that focus on the body, few are openly available, and most target only
one or a few organs, such as the kidneys, prostate, or spleen [6, 7, 8]. The
increasing availability of large biobanks with extensive MRI data further
underscores the need for robust multi-organ MRI segmentation. Detailed
segmentation of multiple organs and structures would enable more in-depth
research on imaging biomarkers and their associations with disease risk,
progression, and treatment response. Furthermore, the development of a
comprehensive, open-source multi-organ MRI segmentation model would
significantly accelerate research in this field by providing a standardized
tool for extracting quantitative data from MRI scans. Such a model could be
used to analyze large datasets, identify novel imaging biomarkers, and develop
more accurate diagnostic and prognostic algorithms. Additionally, the
integration of multi-organ segmentation with other AI techniques, such as
radiomics and deep learning-based classification, could lead to more powerful
and comprehensive AI-based tools for medical image analysis. To address this
need, we have developed MRSegmentator, a deep learning model capable of
accurately segmenting 40 different structures in the pelvic, abdominal, and
thoracic regions of MRI scans. Our approach leverages cross-modality transfer
learning from CT segmentation models, which has shown promise in previous
studies [9]. Additionally, we used a human-in-the-loop annotation workflow to
establish an efficient annotation pipeline. This approach allowed us to create
a model that robustly handles both MRI and CT scans and generalizes well to
external datasets. To accelerate research on MRI-based biomarkers and AI
diagnostic systems, we have made the MRSegmentator model, training data, and
code publicly available at https://github.com/hhaentze/MRSegmentator.

##  2 Materials & Methods

This study was conducted in accordance with the Declaration of Helsinki and
approved by the local ethics committee (EA4/062/20). Patient consent was
waived due to the retrospective nature of the study.

###  2.1 Datasets

We utilized four datasets in this study. For training, we used data from the
UK Biobank and an in-house dataset. For testing, data from the German National
Cohort (NAKO) and the AMOS22 challenge were used.

####  2.1.1 UK Biobank Dataset

The UK Biobank is a large-scale biomedical database containing genetic and
health information from half a million UK participants [10]. We accessed the
pancreatic MRI subset (data-field 20202, October 2023), which includes full-
body MRI scans of 66,050 participants. Each participant’s MRI data is divided
into six regions, from shoulders to knees, with each region covered by four
sequences acquired using the Dixon technique: in-phase (IN), opposed-phase
(OPP), fat-only (F), and water-only (W) (Image 2). This dataset is highly
standardized, with consistent image size, spacing, and participant positioning
within each region. We downloaded 1,200 examinations from 50 randomly selected
participants, which were then manually annotated by a radiologist (LX). Access
to the UK Biobank dataset is available for scientific research upon request at
https://www.ukbiobank.ac.uk/.

####  2.1.2 In-House Dataset

Our in-house dataset consists of abdominal MRI scans from 177 patients with
kidney tumors and cysts, yielding 221 scans. Tumor size is below 7 cm for 213
scans and goes up to 13 cm in the remaining 8 scans. This dataset has an
approximately equal distribution of T1, T2, and T1-weighted-fat-saturated
(T1fs) sequences. It is heterogeneous, as images have be acquired at multiple
different scanners with varying intensity distributions, spacings, and sizes.
A radiologist (LX) manually annotated 40 classes that were included in most
examinations.

####  2.1.3 TotalSegmentator Dataset

Including CT examinations to the training pipeline can improve segmentation
quality for MRI examinations as well [9]. For this purpose we use the
TotalSegmentator dataset, which consists of 1228 CT examinations with a wide
range of different pathologies, scanners, sequences and institutions [2]. From
the 117 segmented structures we select the subset that overlaps with
MRSegmentator. Vertebrae, left lung lobes and right lung lobes are combined
into joint classes respectively. The dataset can be downloaded from
https://zenodo.org/records/10047292.

####  2.1.4 NAKO Dataset

The German National Cohort (NAKO) is a population-based prospective study
investigating the causes for the development of major chronic diseases [11].
The study includes whole-body MRI scans of 30,000 participants from the
general population. We obtained a subset of 600 MRI scans for testing
purposes. The data consists of three T1-GRE modalities (OPP, W, F) for each
patient (Table 1), for which two board-certified radiologists manually refined
segmentations. The data was used for testing purposes only.

####  2.1.5 AMOS22 Dataset

The multi-modality abdominal multi-organ segmentation challenge (AMOS22) was
held at the MICCAI conference in 2022 [9]. The accessible training and
validation sections include 300 CT and 60 MRI images from multi-center, multi-
vendor, multi-modality, multi-phase, and multi-disease patients, each with
voxel-level annotations of 15 abdominal organs. The AMOS22 dataset was used
only for evaluation of our model. We excluded annotations for the bladder, due
to label errors in the MRI images (see Appendix A1), and prostate, as it was
not part of MRSegmentator’s classes, from our evaluation. The data was used
for testing purposes only, it can be accessed at
https://zenodo.org/records/7262581.

Table 1: Data Composition

Training and Validation Data In-House UKBB TotalSegmentator Nr. Participants
171 (m:121, f:56) 501 1228 (m: 716, f:510) Nr. Examinations 221(m:150, f:71)
12001 1228 (m: 716, f:510) Age [years] 37 - 83 (median=62) 40-691 15 - 98
(median=65) Scanner Types Siemens MAGNETOM Avanto 1.5T Siemens MAGNETOM Vida
3T 1.5 Tesla MRI 20 different models Sequences T1 (n=90) T2 (n=64) T1fs post
contrast (n=67) IN (n=300) OPP (n=300) W (n=300) F (n=300) CT (n=1228) Test
Data NAKO (MRI) AMOS (MRI) AMOS (CT) Nr. Participants 50 (m:25, f:25) 60 (m:
55, f: 45)2 300 (m: 314, f: 186)2 Nr. Examinations 600 (m:300, f:300) 60 (m:
55, f: 45)2 300 (m: 314, f: 186)2 Age [years] 26-69 (median=52.5) 22-85
(median=50)2 14-94 (median=54)2 Scanner Types 3 Tesla MR 3 different models 5
different models Sequences T1 GRE OPP (n=200) T1 GRE W (n=200) T1 GRE F
(n=200) MRI (n=60) CT (n=300)

  * 1

UK Biobank targets healthy individuals between the ages of 40 and 69 with a
target gender-ratio of 1:1.

  * 2

Age and sex of the participants of the AMOS22 dataset are given for all 600
images, of which 360 are publicly available with manual annotations.

###  2.2 Annotation Strategy

We employed a human-in-the-loop annotation approach to efficiently create
high-quality segmentations for our training data. This iterative process
involved the following steps:

  1. 1.

Pre-segmentation: We generated initial segmentations by running
TotalSegmentator [2] on the MRI scans, preceded by several preprocessing
steps, such as inversion or histogram equalization [12]. See Figure A2 for an
overview of the preprocessing steps. While some of these pre-segmentations
were accurate enough to require only minor corrections (e.g., kidneys),
TotalSegmentator performed poorly on several organ systems (e.g., muscles or
bones), necessitating the creation of these segmentations from scratch.

  2. 2.

Manual annotation: A radiologist (LX) reviewed and refined the pre-
segmentations using MONAI Label and 3D Slicer. Overall, 40 different labels
were created, which are detailed in Table 2. Due to the non-isotropic voxel
spacing of the axial slices in the UK Biobank data, we combined all vertebrae
into a single "spine" label. Furthermore, as MRI provides poor contrast in
air-filled organs, the lungs were not annotated on a lobe level.

  3. 3.

Model retraining: After each set of 50 annotated examinations, we trained a
segmentation model based on nnUNet using the updated segmentations [13]. The
results of this retrained model served as the basis for the next annotation
iteration, progressively improving the quality and efficiency of the manual
annotation process.

This human-in-the-loop annotation strategy allowed us to efficiently create a
large, high-quality dataset for training MRSegmentator. By iteratively
refining the segmentations and retraining the model, we progressively improved
the accuracy of the pre-segmentations, reducing the manual effort required in
subsequent annotation rounds. Once the annotation process was complete, we
trained an nnUNet with five-fold cross-validation on the fully annotated
images, resulting in the final model.

Table 2: Overview of different body regions and organs, supported by MRSegmentator Body Region | Organs
---|---
Chest | Heart
Left and right lungs
Esophagus
Gastrointestinal tract | Liver
Spleen
Pancreas
Gallbladder
Stomach
Duodenum
Small bowel (Jejunum and ileum)
Colon
Retroperitoneum | Left and right kidneys
Left and right adrenal glands
Urinary bladder
Musculoskeletal | Spine
Sacrum
Left and right hip
Left and right femurs
Left and right gluteus maximus muscles
Left and right gluteus medius muscles
Left and right gluteus minimus muscles
Left and right autochthonous muscles
Left and right iliopsoas muscles
Vessels | Aorta
Inferior vena cava
Portal/splenic vein
Left and right iliac arteries
Left and right iliac veins

###  2.3 Metrics

We evaluated the performance of MRSegmentator using the Dice Similarity
Coefficient (DSC) and the Hausdorff Distance (HD). The DSC quantifies the
overlap between the predicted segmentation and the ground truth annotation.
Its values range from 0 to 1, with 1 indicating perfect overlap between the
predicted segmentation and the ground truth. However, the DSC has some
limitations, especially for small tubular structures (e.g., adrenal glands,
blood vessels), where small segmentation differences can result in
significantly lower DSC values compared to those for large structures [14].
Additionally, the DSC does not distinguish between oversegmentation and
undersegmentation and does not capture differences in shape. The HD is a
boundary-based metric that measures the maximum distance between the predicted
segmentation and the ground truth annotation. Since the HD is sensitive to
outliers, such as single falsely annotated pixels, we used the 95th percentile
variant of the HD. The HD complements the DSC by capturing shape differences
and is less sensitive to small segmentation errors. By using both metrics, we
provide a comprehensive evaluation of MRSegmentator’s performance, accounting
for both overlap and shape differences between the predicted segmentations and
the ground truth annotations.

##  3 Results

Figure 1: Sample segmentation of MRSegmentator for axial (A), coronal (B) and
sagital (C) views of an MRI image. Figure 2: Class-wise dice similarity
coefficients of the NAKO dataset

###  3.1 Study Sample Characteristics

The UK Biobank dataset included 1,200 MRI scans from 50 participants between
40 and 69 years of age. The in-house dataset consisted of 221 MRI scans from
177 patients with kidney lesions, ranging in age from 37 to 83 years, with a
male-to-female ratio of 2.3:1. The characteristics of the in-house dataset are
summarized in Table 2. The TotalSegmentator dataset comprises 1,228 CT scans
from patients between 15 and 98 years old, with a male-to-female ratio of
1.4:1. The NAKO dataset included 600 MRI scans from 50 patients, with a median
age of 52.5 years and a male-to-female ratio of 1:1. The AMOS22 dataset
contained 60 MRI scans and 300 CT scans from multi-center, multi-vendor,
multi-modality, multi-phase, and multi-disease patients. The mean age for the
MRI and CT scans in the AMOS22 dataset was 48.7 and 53.6 years, respectively,
with male-to-female ratios of 1.2:1 for MRI and 1.7:1 for CT.

###  3.2 Segmentation Performance

MRSegmentator was evaluated on the NAKO and AMOS22 datasets. On the NAKO
dataset, the model achieved a mean Dice score of 0.85 ± 0.13 and a mean
Hausdorff distance of 6.55 ± 11.90 mm for MRI scans. For the AMOS22 dataset,
MRSegmentator yielded a mean Dice score of 0.79 ± 0.11 and a mean Hausdorff
distance of 8.29 ± 7.35 mm for MRI scans, demonstrating consistent performance
across different datasets. When applied to CT scans from the AMOS22 dataset,
the model achieved a mean Dice score of 0.84 ± 0.11 and a mean Hausdorff
distance of 7.86 ± 11.12 mm, showcasing its ability to generalize across
different imaging modalities. Detailed per-class results can be found in the
appendix (Table A1). The highest DSC on the NAKO data was achieved for the
left and right lungs, with a value of 0.97 each, and a HD of 3.36 and 3.80.
Conversely, the portal/splenic vein and the adrenal glands proved to be the
most challenging for the model, with a DSC of 0.54 and a HD of 21.86. On the
AMOS22 dataset, the highest DSC was achieved for the spleen with ad DSC of
0.95 and a HD of 2.84. Again, the lowest performance was achieved for the
adrenal glands with a DSC of 0.55 and a HD of 9.27. The greatest variability
between cases was seen for portal/splenic vein, adrenal glands and
gallbladder, where the standard deviation of the DSC ranges from 0.22 to 0.26.
Average segmentation quality for males (DSC=0.87 ± 0.03) is slightly better
than for females (DSC=0.85 ± 0.03) in the external NAKO data (p=0.03). This is
likely caused by the larger proportion of males in all training data sets. At
a categorical level, the greatest differences are measured for the pancreas
($\Delta$DSC=0.11) and left and right adrenal glands ($\Delta$DSC=0.11/0.10).
We could not prove this to be significant with Bonferroni correction but,
admittedly, the power of our non-corrected one-tailed t-test to detect a
medium effect size is comparatively low ($\beta$ = 0.46) [15]. The age of the
probands does positively correlate with the Dice score (r=0.32, p=0.02). This
could be due to the increased age in the training population, however, might
also be a statistical phenomenon caused by a few outliers (Figure A3).

Table 3: Segmentation Quality

Validation Data (5-fold Cross-Validation) T1 T2 T1fs IN OPP W F CT DSC 0.84
(0.14)* 0.77 (0.18) 0.85 (0.14) 0.76 (0.17) 0.79 (0.16) 0.82 (0.16) 0.78
(0.16) 0.91 (0.15) HD [mm] 5.05 (6.09) 6.91 (8.51) 4.63 (5.83) 8.12 (8.45)
7.34 (8.25) 7.07 (9.13) 7.71 (8.43) 7.82 (20.75) External NAKO Data T1 GRE OPP
T1 GRE W T1 GRE F DSC 0.86 (0.13) 0.84 (0.12) 0.86 (0.13) HD [mm] 6.48 (11.86)
5.56 (8.10) 7.60 (13.57) External AMOS Data1 MRI CT Average NAKO MRI2 DSC 0.79
(0.11) 0.84 (0.11) 0.81 (0.16) HD [mm] 8.29 (7.35) 7.86 (11.12) 4.75 (4.66)

  * *

Standard deviation is calculated class-wise and between samples

  * 1

We evaluate on 13 classes of the AMOS data (See Apendix A1)

  * 2

Average results of the NAKO data for the same 13 classes

###  3.3 Failure Cases and Performance on Pathological Images

While MRSegmentator demonstrated robust performance across various datasets
and modalities, some failure cases were observed. The model occasionally
struggled with small organs and vessels, such as adrenal glands and
porta/splenic vein, resulting in lower Dice scores for these structures.
Additionally, segmentation quality varied slightly between different MRI
sequence types, with the model performing best on T1 GRE opposed-phase (OPP)
sequences (DSC = 0.86 ± 0.13) and worst on Dixon in-phase (IN) sequences (Dice
= 0.76 ± 0.17). In few instances the models incorrectly mixes left and right,
particularly in the pelvic region. We implemented a postprocessing step, which
effectively resolves this for distinct structures but the problem persists for
more intricate ones such as blood vessels. Additionally, the model sometimes
misses or incorrectly adds structures at the edge of an examination, resulting
in a DSC of zero. In these cases few missed voxels can have a large impact on
the segmentation metrics. The performance of MRSegmentator on pathological
images was also evaluated. In the in-house dataset, which included MRI scans
of patients with kidney cancer, the model accurately segmented kidneys of all
8 patients with tumors greater than 7 cm (Figure 3). However, in some
instances it appears to oversegment, likely due to the heterogeneous
appearance and irregular borders of the tumors. Further investigation is
needed to assess the model’s performance on a wider range of pathological
conditions.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
Figure 3: The first row depicts manually annotated kidneys (red) and tumors
(blue). The second row shows kidney segmentations of MRSegmentator. Kidneys
were localized and segmented accurately even in the presence of large tumors.
The missing right kidney in sample four does correctly not result in a
segmentation mask.

##  4 Discussion

In this study, we developed MRSegmentator, a multi-modal, multi-structure
segmentation model capable of accurately segmenting 40 organs in MRI and CT
images, achieving strong performance across various datasets, sequence types,
and modalities. These findings highlight the robustness and generalizability
of MRSegmentator in handling diverse imaging data. We have made MRSegmentator
the model and code openly available. One of the key advantages of
MRSegmentator is its ability to learn from both MRI and CT data simultaneously
through cross-modality learning. Although incorporating CT scans into the
training pipeline did not significantly improve MRI segmentation performance
compared to an MRI-only model, we believe that this approach holds potential
to enhance the model’s generalizability to a greater variety of sequences.
Moreover, a model that supports multiple modalities can streamline workflows
and reduce overhead for researchers working with both MRI and CT data.
Compared to existing models, MRSegmentator demonstrates competitive
performance. For CT segmentation, our model achieves results comparable to
TotalSegmentator [2], which reported a mean Dice score of 0.94. Regarding MRI
segmentation, MRSegmentator performs on par with several organ-specific
models. For example, a recent study [16] reported a Dice score of 0.95 for
liver segmentation in MRI, equal to the DSC achieved by MRSegmentator.
Similarly, for the spleen, MRSegmentator comes close to [17] who reported a
Dice score of 0.96.

In a study about multi organ segmentation in the NAKO dataset Kart et al. [18]
segment liver, spleen, kidneys and pancreas with Dice scores of 0.98, 0.96,
0.98 and 0.89. Correspondingly, our model achieves comparable scores of 0.95,
0.89, 0.89 and 0.75 for NAKO MRI, despite it being an external dataset. On the
AMOS dataset, which was annotated by external radiologists, we achieve Dice
scores of 0.96, 0.95, 0.95 and 0.81. Granted, we cannot surpass the current
leader (stand 07.05.2024) of the AMOS challenge, who achieves a DSC of 0.90
for the 13 labels of the combined CT and MRI test set, while or model has an
average of 0.82. However, AMOS is an external dataset for the model and
results for it free of overfitting. Although the MRSegmentator does not always
surpass specific models on single organ segmentation, the strength of the
model is the rich variety of organs and robustness across different MR and CT
sequences. This simplicity in use likely will outweigh slightly inferior
performance compared to specialized models.

However, our study has limitations. First, the same radiologist annotated our
training and validation data using a "human-in-the-loop" approach, potentially
introducing annotation bias. For some structures, such as duodenum, it can be
challenging to define beginning and end, resulting in a low inter-observer
agreement. The inter-observer agreement of the pancreas, for example, can have
a Dice score as low as 0.85 [18]. Although the model’s strong performance on
the external AMOS22 dataset suggests that this bias may not significantly
impact generalizability for most structures, multiple annotators could have
mitigated this issue. Second, the inherent complexity of MRI images poses
challenges in distinguishing target structures from surrounding tissue,
particularly for small structures. While MRSegmentator produces segmentations
that visually are sometimes as accurate or even better than human annotations,
voxel-based metrics such as Dice and Hausdorff distance may not fully capture
the quality of these segmentations when they deviate from the predefined
ground truth. It is important to note that the acceptable level of accuracy
and the impact of under- or over-segmentation may vary depending on the
specific application. The Dice coefficient, while widely used, does not fully
represent these nuances [14]. Furthermore, the correction effort required to
refine the segmentations is another critical factor to consider. For example,
if an application requires ’perfect’ segmentation, a model that makes small
errors throughout the image (e.g., with a Dice score of 0.9) may require
several minutes of correction time. In contrast, a model that makes one large
error (e.g., with a Dice score of 0.85) may only require a few seconds of
correction, making it a more practical choice despite having a lower Dice
score. In conclusion, MRSegmentator represents an advance in multi-modal,
multi-structure segmentation, demonstrating strong performance and
generalizability across diverse datasets. By leveraging cross-modality
learning and interactive annotation, our model can accurately segment 40
classes in MRI and CT images, making it a valuable tool for researchers and
clinicians. The open-source availability of MRSegmentator will facilitate its
adoption and contribute to the advancement of automated segmentation
techniques in medical imaging research. As the field progresses, it will be
crucial to consider application-specific requirements and the trade-offs
between accuracy metrics and practical usability to develop models that best
serve the needs of researchers and clinicians.

## Acknowledgements

Much of the computation resources required for this research was performed on
computational hardware generously provided by the Charité HPC cluster
(https://www.charite.de/en/research/research_support_services/research_infrastructure/science_it/#c30646061).
This work was in large parts funded by the Wilhelm Sander Foundation. Funded
by the European Union. Views and opinions expressed are however those of the
author(s) only and do not necessarily reflect those of the European Union or
European Health and Digital Executive Agency (HADEA). Neither the European
Union nor the granting authority can be held responsible for them.

## References

  * Gillies et al. [2016] Robert J Gillies, Paul E Kinahan, and Hedvig Hricak.  Radiomics: images are more than pictures, they are data.  _Radiology_ , 278(2):563–577, 2016.  doi:10.1148/radiol.2015151169.
  * Wasserthal et al. [2023] Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  _Radiology: Artificial Intelligence_ , 5(5), 2023.  doi:10.1148/ryai.230024.
  * Litjens et al. [2017] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez.  A survey on deep learning in medical image analysis.  _Medical image analysis_ , 42:60–88, 2017.  doi:10.1016/j.media.2017.07.005.
  * Huo et al. [2019] Yuankai Huo, Zhoubing Xu, Yunxi Xiong, Katherine Aboud, Prasanna Parvathaneni, Shunxing Bao, Camilo Bermudez, Susan M Resnick, Laurie E Cutting, and Bennett A Landman.  3d whole brain segmentation using spatially localized atlas network tiles.  _NeuroImage_ , 194:105–119, 2019.  doi:10.1016/j.neuroimage.2019.03.041.
  * Myronenko [2019] Andriy Myronenko.  3d mri brain tumor segmentation using autoencoder regularization.  In _Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4_ , pages 311–320. Springer, 2019.  doi:10.1038/s41592-023-02151-z.
  * Adams et al. [2022] Lisa C Adams, Marcus R Makowski, Günther Engel, Maximilian Rattunde, Felix Busch, Patrick Asbach, Stefan M Niehues, Shankeeth Vinayahalingam, Bram van Ginneken, Geert Litjens, et al.  Prostate158-an expert-annotated 3t mri dataset and algorithm for prostate cancer detection.  _Computers in Biology and Medicine_ , 148:105817, 2022.  doi:10.1016/j.compbiomed.2022.105817.
  * Huo et al. [2017] Yuankai Huo, Jiaqi Liu, Zhoubing Xu, Robert L Harrigan, Albert Assad, Richard G Abramson, and Bennett A Landman.  Robust multicontrast mri spleen segmentation for splenomegaly using multi-atlas segmentation.  _IEEE Transactions on Biomedical Engineering_ , 65(2):336–343, 2017.  doi:10.1109/TBME.2017.2764752.
  * Langner et al. [2020] Taro Langner, Andreas Östling, Lukas Maldonis, Albin Karlsson, Daniel Olmo, Dag Lindgren, Andreas Wallin, Lowe Lundin, Robin Strand, Håkan Ahlström, et al.  Kidney segmentation in neck-to-knee body mri of 40,000 uk biobank participants.  _Scientific reports_ , 10(1):20963, 2020.  doi:10.1038/s41598-020-77981-4.
  * Ji et al. [2022] Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  _Advances in Neural Information Processing Systems_ , 35:36722–36732, 2022.  doi:10.48550/arXiv.2206.08023.
  * Littlejohns et al. [2020] Thomas J Littlejohns, Jo Holliday, Lorna M Gibson, Steve Garratt, Niels Oesingmann, Fidel Alfaro-Almagro, Jimmy D Bell, Chris Boultwood, Rory Collins, Megan C Conroy, et al.  The uk biobank imaging enhancement of 100,000 participants: rationale, data collection, management and future directions.  _Nature communications_ , 11(1):2624, 2020.  doi:10.1038/s41467-020-15948-9.
  * geschaeftsstelle@ nationale-kohorte. de [2014] German National Cohort (GNC) Consortium geschaeftsstelle@ nationale-kohorte. de.  The german national cohort: aims, study design and organization.  _European journal of epidemiology_ , 29(5):371–382, 2014.  doi:10.1007/s10654-014-9890-7.
  * Häntze et al. [2024] Hartmut Häntze, Lina Xu, Leonhard Donle, Felix J. Dorfner, Alessa Hering, Lisa C. Adams, and Keno K. Bressem.  Improve cross-modality segmentation by treating mri images as inverted ct scans.  _arXiv_ , 2024.  doi:10.48550/ARXIV.2405.03713.
  * Isensee et al. [2021] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  _Nature methods_ , 18(2):203–211, 2021.  doi:10.1038/s41592-020-01008-z.
  * Maier-Hein et al. [2024] Lena Maier-Hein, Annika Reinke, Patrick Godau, Minu D Tizabi, Florian Buettner, Evangelia Christodoulou, Ben Glocker, Fabian Isensee, Jens Kleesiek, Michal Kozubek, et al.  Metrics reloaded: recommendations for image analysis validation.  _Nature methods_ , pages 1–18, 2024.  doi:10.1038/s41592-023-02151-z.
  * Faul et al. [2007] Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner.  G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences.  _Behavior research methods_ , 39(2):175–191, 2007.  doi:10.3758/bf03193146.
  * Hossain et al. [2023] Md Sakib Abrar Hossain, Sidra Gul, Muhammad EH Chowdhury, Muhammad Salman Khan, Md Shaheenur Islam Sumon, Enamul Haque Bhuiyan, Amith Khandakar, Maqsud Hossain, Abdus Sadique, Israa Al-Hashimi, et al.  Deep learning framework for liver segmentation from t 1-weighted mri images.  _Sensors_ , 23(21):8890, 2023.  doi:10.3390/s23218890.
  * Sharbatdaran et al. [2022] Arman Sharbatdaran, Dominick Romano, Kurt Teichman, Hreedi Dev, Syed I Raza, Akshay Goel, Mina C Moghadam, Jon D Blumenfeld, James M Chevalier, Daniil Shimonov, et al.  Deep learning automation of kidney, liver, and spleen segmentation for organ volume measurements in autosomal dominant polycystic kidney disease.  _Tomography_ , 8(4):1804–1819, 2022.  doi:10.3390/tomography8040152.
  * Kart et al. [2021] Turkay Kart, Marc Fischer, Thomas Küstner, Tobias Hepp, Fabian Bamberg, Stefan Winzeck, Ben Glocker, Daniel Rueckert, and Sergios Gatidis.  Deep learning-based automated abdominal organ segmentation in the uk biobank and german national cohort magnetic resonance imaging studies.  _Investigative Radiology_ , 56(6):401–408, 2021.  doi:10.1097/RLI.0000000000000755.

##  5 Appendix

Figure A1: The AMOS22 dataset contains only two MRI examinations with bladder
annotations. Of these two annotations, one incorrectly combines the bladder
(yellow) with the uterus (beige). Consequently, we exclude the label "bladder"
from our evaluation of the AMOS22 dataset.

(a)

(b)

(c)

Figure A2: Segmentation quality of TotalSegmentator on a T1-weighted MRI image (a) compared to an inversed version (b) and a refined ground truth (c). Figure A3: The Dice similarity coefficient correlates positively with proband age (r=0.33, p=0.02). Table A1: Comparison of DSC and HD for NAKO, AMOS22 (MRI), and AMOS22 (CT). Note that not all 40 classes supported by MRSegmentator are present in the AMOS22 dataset. Structure | NAKO (MRI) | AMOS22 (MRI) | AMOS22 (CT)
---|---|---|---
| DSC | HD | DSC | HD | DSC | HD
Liver | 0.95 | 4.57 | 0.96 | 4.26 | 0.96 | 5.46
Heart | 0.95 | 3.36 | … | … | … | …
Lung left | 0.97 | 3.36 | … | … | … | …
Lung right | 0.97 | 3.80 | … | … | … | …
Esophagus | 0.84 | 2.71 | 0.64 | 13.48 | 0.80 | 8.43
Spleen | 0.89 | 4.79 | 0.95 | 2.84 | 0.95 | 3.92
Pancreas | 0.75 | 8.86 | 0.81 | 7.91 | 0.81 | 9.26
Gallbladder | 0.74 | 5.37 | 0.76 | 8.20 | 0.81 | 9.18
Stomach | 0.89 | 5.08 | 0.86 | 9.34 | 0.89 | 12.81
Duodenum | 0.76 | 7.81 | 0.58 | 18.44 | 0.68 | 14.38
Small bowel (Jejunum and ileum) | 0.75 | 14.08 | … | … | … | …
Colon | 0.81 | 8.61 | … | … | … | …
Kidney left | 0.88 | 4.28 | 0.95 | 3.08 | 0.94 | 3.51
Kidney right | 0.90 | 3.35 | 0.95 | 3.57 | 0.95 | 3.24
Adrenal left | 0.65 | 4.71 | 0.55 | 9.27 | 0.71 | 5.86
Adrenal right | 0.61 | 3.82 | 0.56 | 7.78 | 0.70 | 5.97
Urinary bladder | 0.95 | 3.01 | … | … | … | …
Spine | 0.89 | 2.99 | … | … | … | …
Sacrum | 0.87 | 3.61 | … | … | … | …
Hip left | 0.89 | 9.64 | … | … | … | …
Hip right | 0.89 | 11.95 | … | … | … | …
Femur left | 0.94 | 8.95 | … | … | … | …
Femur right | 0.93 | 5.59 | … | … | … | …
Gluteus maximus left | 0.85 | 7.62 | … | … | … | …
Gluteus maximus right | 0.91 | 10.88 | … | … | … | …
Gluteus medius left | 0.92 | 2.78 | … | … | … | …
Gluteus medius right | 0.90 | 2.91 | … | … | … | …
Gluteus minimus left | 0.88 | 2.73 | … | … | … | …
Gluteus minimus right | 0.89 | 2.35 | … | … | … | …
Autochthonous muscles Left | 0.94 | 3.38 | … | … | … | …
Autochthonous muscles Right | 0.94 | 3.20 | … | … | … | …
Iliopsoas muscles left | 0.92 | 6.88 | … | … | … | …
Iliopsoas muscles Right | 0.91 | 8.16 | … | … | … | …
Aorta | 0.90 | 2.58 | 0.90 | 13.26 | 0.93 | 13.51
Inferior vena cava | 0.81 | 3.85 | 0.82 | 6.33 | 0.84 | 6.63
Portal/splenic vein | 0.54 | 21.86 | … | … | … | …
Iliac arteries left | 0.77 | 18.67 | … | … | … | …
Iliac arteries right | 0.72 | 7.77 | … | … | … | …
Iliac veins left | 0.80 | 17.36 | … | … | … | …
Iliac veins right | 0.82 | 4.48 | … | … | … | …
Mean | 0.85 | 6.55 | 0.79 | 8.29 | 0.84 | 7.86