# Quality Sentinel: Estimating Label Quality and Errors in Medical
Segmentation Datasets

Yixiong Chen, Zongwei Zhou, Alan Yuille
Johns Hopkins University
Corresponding author: Zongwei Zhou<EMAIL_ADDRESS>
###### Abstract

An increasing number of public datasets have shown a transformative impact on
automated medical segmentation. However, these datasets are often with varying
label quality, ranging from manual expert annotations to AI-generated pseudo-
annotations. There is no systematic, reliable, and automatic quality control
(QC). To fill in this bridge, we introduce a regression model, Quality
Sentinel, to estimate label quality compared with manual annotations in
medical segmentation datasets. This regression model was trained on over 4
million image-label pairs created by us. Each pair presents a varying but
quantified label quality based on manual annotations, which enable us to
predict the label quality of any image-label pairs in the inference. Our
Quality Sentinel can predict the label quality of 142 body structures. The
predicted label quality quantified by Dice Similarity Coefficient (DSC) shares
a strong correlation with ground truth quality, with a positive correlation
coefficient ($r=0.902$). Quality Sentinel has found multiple impactful use
cases. (I) We evaluated label quality in publicly available datasets, where
quality highly varies across different datasets. Our analysis also uncovers
that male and younger subjects exhibit significantly higher quality. (II) We
identified and corrected poorly annotated labels, achieving 1/3 reduction in
annotation costs with optimal budgeting on TotalSegmentator. (III) We enhanced
AI training efficiency and performance by focusing on high-quality pseudo
labels, resulting in a 33%–88% performance boost over entropy-based methods,
with a cost of 31% time and 4.5% memory. The data and model are released.

##  1 Introduction

Recent advances in large-scale organ segmentation (Liu et al., 2023) have
shown significant potential for developing foundational models for general
medical segmentation tasks. The datasets (Wasserthal et al., 2023; Jaus et
al., 2023; Qu et al., 2024), which comprise thousands of 3D scans covering
numerous organs, facilitate the training of powerful and generalizable models.
However, the annotation process for most of these datasets involves AI-
assisted methodologies (Budd et al., 2021), resulting in a predominance of
model-generated pseudo labels. The inherent label noise can adversely affect
the scalability and performance of the datasets. The models trained on these
datasets may also lack reliability. Consequently, there is a critical need for
a quality control (QC) method to make the annotation quality accessible, so
that both the data quality and model performance can be improved.

The primary approach to controlling label quality involves radiologists’
manual inspection. However, this is subjective and becomes increasingly time-
consuming and impractical with the data growth, such as the AbdomenAtlas 1.1
(Li et al., 2023), which contains 3.7 million CT slices. Automatic methods for
estimating pseudo label quality typically rely on the entropy of model
predictions (Culotta and McCallum, 2005; Mahapatra et al., 2018) (i.e.,
uncertainty) and the inconsistency of prediction among multiple models (Li and
Guo, 2013; Gal et al., 2017) (i.e., diversity). However, these methods present
significant drawbacks. Firstly, a model’s feedback reflects its own confidence
rather than the actual label quality. Secondly, the metrics are class-
dependent (e.g., organs with larger volumes exhibit higher entropy), which
complicates the comparison of label quality across classes.

To address the challenges of accurately and efficiently measuring label
quality under a uniform standard across all classes, we have devised a
framework that trains models specifically for estimating the quality of labels
(masks) in organ segmentation tasks. This framework employs the DSC, a
comparison between the current label and the gold standard (ground truth), as
the predictive metric for label quality. We refer to the models trained within
this framework as Quality Sentinel. In previous literature on label evaluation
models (Huang et al., 2016; DeVries and Taylor, 2018; Galdran et al., 2018;
Zhang et al., 2021; Zhou et al., 2023), the common practice is using an image-
label pair as the input and predicting a quality metric. They are typically
trained on a single or small collection of organs, hindering their practical
use on modern multi-organ segmentation datasets. What sets Quality Sentinel
apart from existing methods is it incorporates large language models to
identify target organs, enhancing its performance in images featuring multiple
organs. Specifically, we utilize the pretrained Contrastive Language-Image
Pre-training (CLIP) (Radford et al., 2021) text encoder to embed the text
description of 142 organs. This serves as a conditional input, allowing the
model to recognize the organ it is evaluating. Furthermore, we introduce a
scale-invariant ranking loss to overcome the challenges associated with
training Quality Sentinel, a regressor, on a skewed DSC distribution. This not
only facilitates smoother training but also significantly boosts its ranking
accuracy. In summary, our work presents three major contributions to the field
of medical image segmentation and dataset evaluation.

  * •

First, we introduce Quality Sentinel, a novel medical segmentation evaluation
model designed to estimate label quality through DSC prediction. This model is
distinguished by its use of text embeddings as conditions for different
classes and a novel ranking loss mechanism, enabling effective multi-organ
segmentation evaluation (Sect. 3, Fig. 1).

  * •

Second, we apply Quality Sentinel to reveal a large label quality variability
among datasets. It also uncovers notable gender and age disparities in
annotation quality. Specifically, labels associated with female and older
patients exhibit lower quality, highlighting the need for increased diligence
in dataset construction to mitigate the annotation biases, especially for
these groups (Sect. 4.3, Tab. 2, Fig. 5).

  * •

Third, we demonstrate the efficacy of Quality Sentinel as a sample selector
for training segmentation models. It outperforms traditional diversity and
uncertainty-based methods. On the one hand, it enhances the data efficiency in
the human-in-the-loop process (active learning) for dataset development. On
the other hand, by selecting high-quality pseudo labels in semi-supervised
training, the segmentation models can be more reliable and powerful (Sect.
4.4, Fig. 6, Tab. 3).

##  2 Related Work

###  2.1 Multi-Organ Segmentation

With the rapid advances of deep segmentation models (Ronneberger et al., 2015;
Isensee et al., 2021; Oktay et al., 2018; Chen et al., 2021; Hatamizadeh et
al., 2022; Tang et al., 2022) and the emergence of large-scale multi-organ
segmentation datasets (Ji et al., 2022; Ma et al., 2021; Wasserthal et al.,
2023), the need for data processing, filtering, and corresponding model
training strategies has also grown. The existing dataset with the largest
sample size is AbdomenAtlas (Qu et al., 2024). It labels 32 organs and tumors
over 9,000 3D CT volumes with a human-in-the-loop technique. This method
accelerates the annotation process to a large extent (as reported, the dataset
was annotated within three weeks), but the high ratio of AI-generated pseudo
labels may also make the dataset potentially risky in low-quality annotations.
TotalSegmentator (Wasserthal et al., 2023), a large dataset with 1,000+
samples and 100+ classes, relies upon quality insurance via 3D renderings and
expert checks. The labor-intensive checking results in the final checking rate
of only 100 CT volumes. To this end, an efficient QC tool to ensure annotation
quality is in urgent need.

###  2.2 Medical Segmentation Quality Control

QC tools raise a flag when the segmentation model under analysis incurs a lack
of reliability or robustness (Galati et al., 2022). They can be categorized
into two folds: the pre-analysis and the post-analysis QC. Pre-analysis QC
(Lorch et al., 2017; Tarroni et al., 2018; Oksuz et al., 2019) performs on the
input of the segmentation model, it emulates expert criteria and detects low-
quality samples (e.g., incomplete or corrupted scans). By giving binary
feedback, it can discard those samples to improve the robustness of the
segmentation pipeline. Post-analysis QC (Alba et al., 2018; Robinson et al.,
2018; Lin et al., 2022) focuses on the assessment of the segmentation outputs
of a model. It usually infers common segmentation metrics (e.g., DSC,
Hausdorff Distance (HD), or uncertainty estimates), thus detecting a
malfunction of models. Quality Sentinel is a tool that can serve as both pre-
analysis and post-analysis. On the one hand, it helps to detect flaws in the
existing datasets by analysis of the label quality. On the other hand, it can
evaluate model outputs and find the weaknesses of the model, which can be
beneficial for further refinement. Moreover, existing QC techniques primarily
work on single organs (e.g., Cardiac MR (Lorch et al., 2017; Tarroni et al.,
2018; Oksuz et al., 2019; Alba et al., 2018; Robinson et al., 2018)) or a
small set of relevant organs (e.g., knee and calf muscle (Zaman et al.,
2023)). Quality Sentinel is an exploration of general-purpose QC tool that
works for a wide variety of organs simultaneously.

###  2.3 Language-driven Medical Image Analysis

Large language models (Devlin et al., 2019; Brown et al., 2020) widely succeed
in language processing and understanding. Visual language models (VLMs) (Wang
et al., 2022) such as CLIP have been explored to promote medical image
analysis tasks such as medical vision question answering (Eslami et al.,
2023), medical object recognition (Qin et al., 2022), and organ/tumor
segmentation (Liu et al., 2023; Ulrich et al., 2023). VLM could be useful in
these scenarios because the language encoder can generate intrinsic semantics
of the anatomical structures for tasks in the medical domain with carefully
designed medical prompts. Therefore, the vision module can be augmented with
knowledge from the language domain. For Quality Sentinel, the text embedding
acts as the condition for the vision representation of a target class, and
similar classes share similar knowledge from language.

##  3 Quality Sentinel

Figure 1: Overview of Quality Sentinel: Inputs are image-label pairs with
pseudo labels. It employs a vision encoder to estimate DSC relative to ground
truths. The Text-driven Condition Module encodes texts to augment the model
for organ identification, addressing the ambiguity of organ-label matching
(e.g., for $(I_{k},S_{k})$ at upper left, the mask covers two organs).
Training involves a compositional loss, combining optimal pair ranking and
MSE, to align predicted with actual DSC.

###  3.1 Formulation of the Quality Estimation Framework

The Quality Sentinel framework, depicted in Fig. 1, evaluates data batches of
$N$ samples, denoted as $\mathcal{D}=\\{I_{i},S_{i},G_{i}\\}_{i=1}^{N}$, where
$I$, $S$, and $G$ represent an image, a segmented mask (pseudo label), and a
ground truth label, respectively. The estimation model $\mathcal{Q}$,
parameterized by $\theta$, estimates the DSC
$\hat{h}_{i}=\mathcal{Q}(I_{i},S_{i};\theta)$, compared against the actual DSC
$h_{i}=\mathcal{E}(S_{i},G_{i})$. To align the predicted $\hat{h}$ with the
actual $h$, we introduce a compositional loss function:

$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{MSE}(\hat{h}_{i},h_{i})+\lambda\mathcal{L}_{Rank}(\hat{\textbf{h}},\textbf{h}),$
(1)

with $\lambda$ as a scaling constant. The mean square error (MSE) loss
$\mathcal{L}_{MSE}(\hat{h}_{i},h_{i})$ aims to match the model’s output with
the ground truth DSC. The novel optimal-pair ranking loss
$\mathcal{L}_{Rank}(\hat{\textbf{h}},\textbf{h})$ enhances the model’s ranking
capability by optimizing the order of predicted and actual DSC within the
batch $\mathcal{D}$.

For practical application, we process a 2D image-label pair as input to
$\mathcal{Q}$, averaging estimates from 10 uniformly sampled 2D slices along
the z-axis for 3D organ label quality estimation. This 2D-based method is
superior to 3D methods because slice-level prediction is more accurate and can
provide clinicians with specific information related to the single slices
(Fournel et al., 2021).

###  3.2 Text-driven Condition Module

Challenges arise with expanded or fragmented masks during the segmentation of
small or irregularly shaped organs, as seen in the upper left of Fig. 1 (a
mask covers two organs), causing ambiguity in organ-label matching. This
ambiguity compromised the evaluation model’s accuracy. To address this,
Quality Sentinel employs CLIP text embeddings of organ classes as conditions
to discern the target organ amidst multiple organs within an image. The text
embeddings also encourage the model to learn shared representations for
similar classes due to their similar conditions. In this way, similar classes
can learn similar evaluation criteria and promote each other’s performance.

As for the generation of text embeddings, it is crucial to choose an
appropriate prompt template (Qin et al., 2022) because it is the key to
eliciting knowledge from pre-trained VLMs. According to our ablation study in
Section 4.2, the best prompt is “[CLS]", the class name itself.

Quality Sentinel views text embeddings as class-conditional attention added
before the decision layers. For a given class $k$, its CLIP text embedding
$\varphi_{k}$ is concatenated with the image feature $f_{1}=E_{v}(x)$ produced
by a vision encoder (e.g., ResNet50 (He et al., 2016)). A multi-layer
perceptron (MLP) then computes attention weights
$[\omega_{1},\omega_{2}]=MLP([f_{1},\varphi_{k}])$, used by the regression
prediction head, consisting of two fully connected (FC) layers $g_{1}(\cdot)$
and $g_{2}(\cdot)$, to make the final prediction:

$\hat{h}=g_{2}(\omega_{2}\otimes g_{1}(\omega_{1}\otimes E_{v}(x))),$ (2)

where $\otimes$ denotes element-wise multiplication. Employing two FC layers
allows for a phased feature compression, enabling the attention mechanism to
more significantly influence prediction outcomes.

###  3.3 Optimal Pair Ranking Loss

Reliance on MSE loss alone is inadequate for training Quality Sentinel due to
its tendency to bias DSC predictions towards the most common score interval.
Considering the framework’s application in ranking sample quality and
selecting pseudo labels for revision, we introduce a scale-insensitive ranking
loss as an auxiliary training mechanism.

Inspired by the approach in (Yoo and Kweon, 2019), we compare sample pairs.
However, in the context of multi-organ segmentation, comparing dissimilar
organ classes (e.g., spleen vs. skull) is unproductive. Thus, we form $N/2$
pairs within a batch of size $N$, ensuring each pair consists of identical or
similar class samples by maximizing their total embedding similarity. The
pairing process involves three steps: 1) computing a cosine similarity matrix
$H\in\mathbb{R}^{N\times N}$; 2) converting $H$ into a cost matrix
$H^{\prime}=-H+\infty\cdot I_{N\times N}$ to prevent self-pairing; 3)
identifying optimal pairs through the Jonker-Volgenant algorithm for linear
sum assignment (Crouse, 2016).

For an optimal pair $\\{x^{p}=\\{x_{i},x_{j}\\}\\}$, the ranking loss is:

$\mathcal{L}_{Rank}(\hat{h}_{i},\hat{h}_{j},h_{i},h_{j})=max(0,(\hat{h}_{i}-\hat{h}_{j})\cdot(h_{j}-h_{i})+\xi),$
(3)

where $\xi$ is a predefined positive margin. This loss specifically penalizes
discrepancies in the directional agreement between the predicted and actual
DSC differences.

##  4 Experiments & Results

###  4.1 Experimental Settings

Stage 1 Dataset Construction. We fine-tuned the pretrained STUNet (Huang et
al., 2023) on the DAP Atlas (Jaus et al., 2023), a comprehensive dataset that
aggregates 142 categories from 14 datasets. Model checkpoints were saved at
specified epochs: ${10,20,30,40,50,100,200,300,400,500}$. From each
checkpoint, pseudo labels for 15 CT scans were generated, with 10 designated
for training and 5 for testing, creating a dataset of CT scans paired with
pseudo labels of varying quality and their corresponding ground truth DSC.
Because DAP Atlas is also an AI-assisted dataset, its ground truth masks may
not be accurate enough. Therefore, these imperfect ground truths cannot serve
as the training samples with $DSC=1.0$. The generated masks and their DSCs are
feasible to train the model because their variations are minimized when
averaged over the large dataset. To further decrease the inaccuracy of ground
truth DSC for training, we resample data to refine their quality.

Stage 2 Data Resampling. Utilizing the trained Quality Sentinel, we resampled
50 CT scans of the highest label quality from the DAP Atlas to create a new
dataset, ensuring a balanced sex distribution (25 male, 25 female) and
dividing them into 20 training and 5 testing samples each. This dataset,
containing approximately 4 million 2D image-label pairs, has improved training
label quality. The stage 2 resampling leads to more accurate ground truth DSC
and enhanced model performance, as evidenced in our ablation study in the
following section.

Training Details. We opted for ResNet50 as the vision encoder, adjusting its
initial layer to accommodate 2-channel inputs. CT HU values were clipped to
$[-200,200]$ and images were cropped and resized to $256\times 256$ around
pseudo label centers. These processed pseudo-label masks were then
concatenated with the CT images for training. The Quality Sentinel was trained
over 30 epochs using the Adam optimizer, with a learning rate of $10^{-3}$ and
batch size of 128. The loss function incorporated a scaling constant
$\lambda=1$. Our experiments were conducted on a PyTorch 2.1.0 framework,
utilizing an Intel Xeon Gold 5218R<EMAIL_ADDRESS>and 8 Nvidia Quadro RTX 8000
GPUs.

###  4.2 Evaluation of Quality Sentinel

Table 1: Ablation study for Quality Sentinel. The baseline is a ResNet50 with
MSE loss. All results are reported on the resampled testing set.

Condition |  Opt. Pair Loss |  Data Resampling |  LCC |  SROCC |  MAP@5 |  MAP@10
---|---|---|---|---|---|---
- | - | - | 0.797 | 0.775 | 0.417 | 0.481
One-hot | - | - | 0.817 | 0.795 | 0.427 | 0.501
CLIP: “A computerized tomography of a [CLS]." | - | - | 0.840 | 0.805 | 0.438 | 0.510
CLIP: “There is [CLS] in this computerized tomography." | - | - | 0.847 | 0.801 | 0.444 | 0.525
CLIP: “A photo of a [CLS]." | - | - | 0.852 | 0.808 | 0.445 | 0.533
CLIP: “[CLS]" | - | - | 0.853 | 0.808 | 0.449 | 0.531
- | ✓ | - | 0.817 | 0.778 | 0.438 | 0.535
CLIP: “[CLS]" | ✓ | - | 0.855 | 0.810 | 0.463 | 0.546
CLIP: “[CLS]" | ✓ | ✓ | 0.902 | 0.856 | 0.500 | 0.565

Ablation Study. To assess the effectiveness of our method, we performed an
ablation study concerning text embedding, optimal pair ranking loss, and the
data resampling strategy. We utilized three metrics for evaluation: the Linear
Correlation Coefficient (LCC), the Spearman Rank Order Correlation Coefficient
(SROCC), and the Mean Average Precision (MAP@k), where ‘k’ refers to the
samples with the lowest ground truth DSC. The MAP@k metric is an important
ranking metric in our evaluation as it focuses on the samples with the lowest
ground truth Dice Similarity Coefficient (DSC), which are cases that may
degrade the model training. For each class, the Average Precision at k (AP@k)
is first calculated, which involves selecting the k samples with the lowest
actual DSC scores and checking if these indices correspond to low predicted
scores as well. The precision is accumulated for hits within the top-k ranked
predictions, and the MAP@k is computed as the mean AP@k across all classes.

The ablation study is shown in Tab. 1. Incorporating each of the proposed
techniques significantly enhances model performance. First, adding class
conditions to the model is beneficial for model performance. Even with one-hot
embedding of classes, Quality Sentinel performs 2.0% better than the baseline
w.r.t. LCC. For different CLIP embeddings, we have tried four different prompt
templates. The results show that the best prompt is the simplest one, “[CLS]".
The text embeddings notably improve both regression (i.e., correlation
coefficients) and ranking performance (MAP@k). The reason that CLIP embedding
performs better than one-hot embedding is that it addresses the label
orthogonality problem by exploiting semantic relationships among organs (Liu
et al., 2023). A simpler prompt leads to greater similarity differences
between classes, thus eliciting better knowledge from the language. The
optimal pair ranking loss is particularly beneficial for ranking (the MAP@10
improved from 0.481 to 0.535 compared to baseline). This would be helpful when
using Quality Sentinel to rank sample quality and filter unsatisfying samples.
Finally, data resampling markedly boosts all evaluated metrics by enriching
the dataset both in volume and quality of ground truth labels.

Figure 2: Illustration of (a) the curves of model performance and data amount
and (b) the curves of performance deviation w.r.t. slice amount compared to
prediction with all slices.

Evaluation of Model Configurations. The training dataset of Quality Sentinel
is obtained from 40 original CT volumes of DAP Atlas. To investigate the
impact of training data volume on Quality Sentinel’s efficacy, we analyzed how
model performance correlates with the amount of data, represented in Fig. 2
(a). Results indicate performance plateaus at approximately 20-30 CT scans,
ensuring the adequacy of our training dataset’s size. In addition, the
inference of Quality Sentinel uniformly samples 10 2D slices from a 3D mask,
we illustrate the mean difference in DSC w.r.t. number of slices used for
inference in Fig. 2 (b). The difference between sampling strategy and
inference with all slices quickly decreases as the number of slices increases.
Specifically, when inferring with 10 slices, the mean difference for all
classes is only 0.021, which is acceptable for most practical scenarios.

Figure 3: Evaluation of model performance. The scatter plots are illustrated
between predicted DSC and ground truth DSC of Quality Sentinel testing set (a)
and external evaluation on BTCV with erosion mask degeneration (b) and
dilation degeneration (c).

Figure 4: Illustration of labels with lower (left) and higher (right)
predicted DSC in AbdomenAtlas dataset. Typical label errors detected include
reduced masks (the first two of the liver), missing masks (the first one of
the pancreas), and expanded masks (the first three of the gallbladder).

External Evaluation. Fig. 2 (a) showcases Quality Sentinel’s optimal
predictions on its testing set, revealing a strong correlation between the
model’s DSC predictions and actual values, underscoring the model’s predictive
reliability. However, generalization risks remain since the model was
developed on a single data source. We do external evaluations to validate its
performance on more data domains. First, we use BTCV (Landman et al., 2015), a
well-known manually annotated dataset with 13 classes, to show the model’s
performance. As shown in Fig. 2 (b) and (c), we use morphological operations
(i.e., erosion and dilation) to generate degraded masks and use Quality
Sentinel to infer the DSC, showcasing consistent results with the testing set.
Even though the LCC decreases to 0.850 and 0.775 in these two cases, this
experiment validates the generalization of Quality Sentinel on out-of-
distribution data and unseen mask degradation. We attribute this ability to
the large-scale training dataset comprising a wide variety of different
degradation modes. Second, we apply Quality Sentinel to the largest multi-
organ segmentation dataset, AbdomenAtlas (Qu et al., 2024), to visually
demonstrate the model’s ability to detect label errors. In Fig. 4, we show the
model prediction of DSCs on four representative abdominal organs. On the left
of the figure, imperfect labels with missing, overly expanded, and reduced
masks are detected. On the right of the figure, it can be seen that high-
quality masks are far smoother and more precise than imperfect masks.

###  4.3 Dataset Evaluation with Quality Sentinel

Table 2: Comparison of label quality evaluation for different datasets.

Dataset | # Samples | Annotators | Organ-wise Average Estimation | Overall Estimation
---|---|---|---|---
|  |  | aorta | gallbladder | kidney | liver | pancreas | postcava | spleen | stomach | Mean DSC | DSC$<0.8$
CHAOS (Valindria et al., 2018) | 20 | Human | - | - | - | 0.985 | - | - | - | - | 0.985 | 0.0%
TCIA Pancreas (Roth et al., 2015) | 42 | Human | - | 0.904 | 0.963 | 0.961 | 0.919 | - | 0.946 | 0.957 | 0.941 | 0.8%
BTCV (Landman et al., 2015) | 47 | Human | 0.943 | 0.875 | 0.961 | 0.981 | 0.928 | 0.921 | 0.962 | 0.969 | 0.944 | 1.7%
AbdomenCT-12organ (Ma et al., 2023) | 50 | Human | 0.965 | 0.932 | 0.963 | 0.965 | 0.961 | 0.960 | 0.964 | 0.961 | 0.959 | 0.0%
WORD (Luo et al., 2021) | 120 | Human | - | 0.874 | 0.968 | 0.981 | 0.946 | - | 0.958 | 0.970 | 0.952 | 1.9%
LiTS (Bilic et al., 2019) | 131 | Human | - | - | - | 0.963 | - | - | - | - | 0.963 | 0.0%
CT-ORG (Rister et al., 2020) | 140 | Human&AI | - | - | 0.911 | 0.948 | - | - | - | - | 0.924 | 0.5%
AMOS22 (Ji et al., 2022) | 200 | Human&AI | 0.954 | 0.874 | 0.956 | 0.971 | 0.928 | 0.927 | 0.953 | 0.942 | 0.941 | 2.1%
KiTS∗ (Heller et al., 2020) | 489 | Human | - | - | 0.867 | - | - | - | - | - | 0.867 | 11.3%
DAP Atlas (Jaus et al., 2023) | 533 | AI | 0.945 | 0.856 | 0.946 | 0.961 | 0.851 | 0.919 | 0.959 | 0.945 | 0.925 | 2.4%
AbdomenCT-1k (Ma et al., 2021) | 1000 | Human&AI | - | - | 0.951 | 0.963 | 0.949 | - | 0.956 | - | 0.955 | 0.2%
TotalSegmentator (Wasserthal et al., 2023) | 1204 | Human&AI | 0.927 | 0.794 | 0.914 | 0.941 | 0.853 | 0.880 | 0.933 | 0.915 | 0.901 | 8.0%

  * •

∗ Due to a different annotation protocol of the KiTS dataset, its kidney
annotation excludes two tumor types included in it and may account for a
relatively large volume, resulting in an underestimated DSC.

Label Quality Comparison. We conducted a comparative analysis of 12
segmentation datasets focusing on major abdominal organs using Quality
Sentinel, as detailed in Tab. 2. We observed that smaller datasets,
particularly those manually annotated such as AbdomenCT-12organ, a benchmark
dataset for the FLARE challenge, exhibit superior organ-wise label quality.
Conversely, in larger datasets like TotalSegmentator, where AI-generated
pseudo labels are more prevalent, there is a noticeable decline in the overall
DSC estimation. Human annotation agreement is usually higher than $0.8$ (Qu et
al., 2024). So if considering a quality estimation below $0.8$ as inadequate,
Quality Sentinel can be used to diagnose the dataset and find the label
errors. For instance, it reveals that the TotalSegmentator contains $8\%$
unsatisfying annotations for the abdomen organs and requires potential
refinement.

Figure 5: The annotation bias w.r.t. gender (left) and age (right) for
TotalSegmentator. We show several representative organs with higher label
quality for males than females and find a significant overall quality
difference for different genders. In addition, older patients generally have
worse label quality than younger patients.

Annotation Bias. Investigating the TotalSegmentator dataset, which includes
comprehensive metadata, revealed significant biases in label quality related
to gender and age, as illustrated in Fig. 5. Annotations for males and younger
individuals were generally of higher quality compared to females and older
individuals, with the discrepancies being statistically significant (gender
bias $p<10^{-3}$, age bias $p<0.05$). These biases may stem from the
complexity and severity of conditions in certain patients, which can obscure
organ visibility and complicate the annotation. To enhance the
generalizability of models for clinical application, future dataset
constructions should prioritize more meticulous annotations for these groups.

###  4.4 Data Efficient Training

In this section, we demonstrate that Quality Sentinel effectively
ranks/selects training samples, enhancing the efficiency of training
segmentation models. We assessed its performance against random selection,
Monte-Carlo Dropout (for diversity) (Gal and Ghahramani, 2016), and Entropy
(for uncertainty) (Joshi et al., 2009) on the TotalSegmentator dataset
(Wasserthal et al., 2023). We chose this dataset for its wide range of 104
classes and it is a fair out-of-domain benchmark for comparison. Utilizing a
pretrained SwinUNETR (Tang et al., 2022), we fine-tuned the model on 20 full-
body CT scans and generated pseudo labels for an additional 100 samples. We
then employed these methods to evaluate the quality of pseudo labels,
selecting the most suitable samples for further annotating/training, with DSC
and Normalized Surface Distance (NSD) serving as the key performance
indicators. In the following experiments, the only difference between groups
is their selected samples based on different methods.

Human-in-the-Loop (Active Learning). In the human-in-the-loop annotation of
datasets, we usually have a small annotation budget, and select a small number
of samples that the initial model cannot predict well. These identified
samples (namely most informative samples in active learning) are then
annotated with human annotators. In our experiment, each method selects $n$
samples based on the lowest pseudo-label quality and revises them with ground
truth labels. We combine these samples alongside the original 20 samples for
fine-tuning segmentation models. Results (Fig. 6) reveal that Quality Sentinel
consistently outperforms the other methods. With a labeling budget of 20
samples, it exceeds random selection by 0.006 and 0.011 in DSC and NSD
metrics, respectively. This is comparable to a random selection budget of 30
samples. It means when used for real annotation scenarios, Quality Sentinel
can help to reduce about 1/3 annotation cost.

Figure 6: Human-in-the-Loop (active learning) results of label quality ranking
methods on the TotalSegmentator. Error bars are reported with 5 repeated
experiments. Quality Sentinel helps to reduce annotation costs from 30 samples
to 20 samples compared to random selection.

Semi-supervised Learning. In this experiment, we perform a typical semi-
supervised learning strategy: self-training with pseudo labels. Each ranking
method selects the top 20 pseudo labels with the highest quality. We combine
these samples and the initial 20 ground truth-labeled samples to fine-tune The
segmentation models. If the selection is not reliable enough and low-quality
pseudo-labels are included for training, the model performance may not improve
or even be harmed. As Tab. 3 indicates, Quality Sentinel outperforms all
alternatives. Specifically, it increases the performance improvement by 33%
($0.67\%\rightarrow 0.89\%$) on DSC and 88% ($0.41\%\rightarrow 0.77\%$) on
NSD compared to the entropy-based method. Compared to random selection,
Quality Sentinel has statistically significant improvements ($p=0.036$ for DSC
and $p=0.0035$ for NSD). As for resource overhead, MC dropout and entropy-
based methods need to operate on the 3D raw outputs of the model with the
shape $[C,D,H,W]$ and float data type. Entropy needs to be computed with per-
voxel logits, and MC dropout requires the segmentation model to do inference
three times and calculate the standard deviation of each voxel’s probability.
In practice, they both require saving the model’s 3D outputs to disk and
reloading them to memory for further computation. Instead, Quality Sentinel
only needs the segmentation masks with the shape $[D,H,W]$ and the data type
of uint8. Considering their different compression formats (.npy and .nii.gz),
the disk usage can vary by 20,000 times. The 2D processing style and the
uniform sampling scheme of Quality Sentinel also largely reduce quality
estimation costs (6 times less time and 60 times less RAM than MC dropout).

Table 3: Semi-supervised learning results of label quality ranking methods on
the TotalSegmentator dataset. Standard deviations are reported with 5 repeated
experiments.

Method | # Scans Used | Metrics | Quality Estimation Cost
---|---|---|---
|  Labeled |  Unlabeled | DSC[%] | NSD[%] |  Time[h] |  RAM[GB] |  Disk[GB]
SwinUNETR | 20 | 0 | 82.55 ($\pm$ 0.31) | 45.76 ($\pm$ 0.28) | - | - | -
Random | 20 | 100 | 83.16 ($\pm$ 0.26) | 46.08 ($\pm$ 0.33) | - | - | -
MC dropout | 20 | 100 | 83.32 ($\pm$ 0.57) | 46.35 ($\pm$ 0.46) | 3.17 | 92.4 | 1032
Entropy | 20 | 100 | 83.21 ($\pm$ 0.37) | 46.17 ($\pm$ 0.43) | 1.61 | 33.1 | 344
Quality Sentinel | 20 | 100 |  83.44 ($\pm$ 0.35) |  46.53 ($\pm$ 0.38) | 0.50 | 1.5 | 0.05

##  5 Discussion and Limitations

Quality Sentinel learns generalizable evaluation capability on a large dataset
comprising 4M slices. Although it shows good performance on external
evaluation, it may not cover all segmentation scenarios. Its training data
come from the DAP Atlas dataset with 142 healthy organs. In the KiTS dataset,
kidney and kidney tumors are marked as two categories. The definition
discrepancy makes the model underestimate label quality (Tab. 2). Moreover,
Quality Sentinel is trained on a dataset with balanced gender. Due to the
annotation bias discussed before, the different annotation qualities of
different groups would possibly affect the ground truth DSC during training.
Finally, with Quality Sentinel, it is possible to achieve purposeful QC in
data sets. On the one hand, it can help improve the annotation and balance of
the data set, but on the other hand, there is also the risk of being exploited
to intentionally control the preferences of the data and model.

##  6 Conclusion

Our primary contribution is the development of Quality Sentinel, a label
quality estimation model trained on a vast and diverse set of image-label
pairs. This model enables a thorough evaluation of dataset quality and
identification of biases. It acts as a pivotal tool for sample selection, both
for guiding human-in-the-loop annotation and for model training. While our
empirical results show the practical value of the model on CT images, an
important future work is to extend Quality Sentinel to more modalities to be a
generic label evaluator for medical image segmentation. We envision that
Quality Sentinel will play a crucial role in quality control for expansive
datasets, thereby enhancing training efficiency and applicability in real-
world scenarios.

## References

  * Alba et al. [2018] Xenia Alba, Karim Lekadir, Marco Pereanez, Pau Medrano-Gracia, Alistair A Young, and Alejandro F Frangi.  Automatic initialization and quality control of large-scale cardiac mri segmentations.  _Medical image analysis_ , 43:129–141, 2018.
  * Bilic et al. [2019] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al.  The liver tumor segmentation benchmark (lits).  _arXiv preprint arXiv:1901.04056_ , 2019.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , 33:1877–1901, 2020.
  * Budd et al. [2021] Samuel Budd, Emma C Robinson, and Bernhard Kainz.  A survey on active learning and human-in-the-loop deep learning for medical image analysis.  _Medical Image Analysis_ , 71:102062, 2021.
  * Chen et al. [2021] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  _arXiv preprint arXiv:2102.04306_ , 2021.
  * Crouse [2016] David F Crouse.  On implementing 2d rectangular assignment algorithms.  _IEEE Transactions on Aerospace and Electronic Systems_ , 52(4):1679–1696, 2016.
  * Culotta and McCallum [2005] Aron Culotta and Andrew McCallum.  Reducing labeling effort for structured prediction tasks.  In _AAAI_ , volume 5, pages 746–751, 2005.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , pages 4171–4186, 2019.
  * DeVries and Taylor [2018] Terrance DeVries and Graham W Taylor.  Leveraging uncertainty estimates for predicting segmentation quality.  _arXiv preprint arXiv:1807.00502_ , 2018.
  * Eslami et al. [2023] Sedigheh Eslami, Christoph Meinel, and Gerard De Melo.  Pubmedclip: How much does clip benefit visual question answering in the medical domain?  In _EACL_ , pages 1181–1193, 2023.
  * Fournel et al. [2021] Joris Fournel, Axel Bartoli, David Bendahan, Maxime Guye, Monique Bernard, Elisa Rauseo, Mohammed Y Khanji, Steffen E Petersen, Alexis Jacquier, and Badih Ghattas.  Medical image segmentation automatic quality control: A multi-dimensional approach.  _Medical Image Analysis_ , 74:102213, 2021.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , pages 1050–1059. PMLR, 2016.
  * Gal et al. [2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _ICML_ , pages 1183–1192. PMLR, 2017.
  * Galati et al. [2022] Francesco Galati, Sébastien Ourselin, and Maria A Zuluaga.  From accuracy to reliability and robustness in cardiac magnetic resonance image segmentation: a review.  _Applied Sciences_ , 12(8):3936, 2022.
  * Galdran et al. [2018] Adrian Galdran, Pedro Costa, Alessandro Bria, Teresa Araújo, Ana Maria Mendonça, and Aurélio Campilho.  A no-reference quality metric for retinal vessel tree segmentation.  In _MICCAI_ , pages 82–90. Springer, 2018.
  * Hatamizadeh et al. [2022] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu.  Unetr: Transformers for 3d medical image segmentation.  In _WACV_ , pages 574–584, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * Heller et al. [2020] Nicholas Heller, Sean McSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Joel Rosenberg, et al.  An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging., 2020.
  * Huang et al. [2016] Chao Huang, Qingbo Wu, and Fanman Meng.  Qualitynet: Segmentation quality evaluation with deep convolutional networks.  In _VCIP_ , pages 1–4. IEEE, 2016.
  * Huang et al. [2023] Ziyan Huang, Haoyu Wang, Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He, Yun Gu, Lixu Gu, Shaoting Zhang, et al.  Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training.  _arXiv preprint arXiv:2304.06716_ , 2023.
  * Isensee et al. [2021] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  _Nature methods_ , 18(2):203–211, 2021.
  * Jaus et al. [2023] Alexander Jaus, Constantin Seibold, Kelsey Hermann, Alexandra Walter, Kristina Giske, Johannes Haubold, Jens Kleesiek, and Rainer Stiefelhagen.  Towards unifying anatomy segmentation: automated generation of a full-body ct dataset via knowledge aggregation and anatomical guidelines.  _arXiv preprint arXiv:2307.13375_ , 2023.
  * Ji et al. [2022] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  _arXiv preprint arXiv:2206.08023_ , 2022.
  * Joshi et al. [2009] Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos.  Multi-class active learning for image classification.  In _CVPR_ , pages 2372–2379. IEEE, 2009.
  * Landman et al. [2015] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein.  Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge.  In _Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge_ , volume 5, page 12, 2015.
  * Li et al. [2023] Wenxuan Li, Alan Yuille, and Zongwei Zhou.  How well do supervised models transfer to 3d image segmentation?  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Li and Guo [2013] Xin Li and Yuhong Guo.  Adaptive active learning for image classification.  In _CVPR_ , pages 859–866, 2013.
  * Lin et al. [2022] Qiao Lin, Xin Chen, Chao Chen, and Jonathan M Garibaldi.  A novel quality control algorithm for medical image segmentation based on fuzzy uncertainty.  _IEEE Transactions on Fuzzy Systems_ , 2022.
  * Liu et al. [2023] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou.  Clip-driven universal model for organ segmentation and tumor detection.  In _ICCV_ , pages 21152–21164, 2023.
  * Lorch et al. [2017] Benedikt Lorch, Ghislain Vaillant, Christian Baumgartner, Wenjia Bai, Daniel Rueckert, and Andreas Maier.  Automated detection of motion artefacts in mr imaging using decision forests.  _Journal of medical engineering_ , 2017, 2017.
  * Luo et al. [2021] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Tao Song, Xiaofan Zhang, Kang Li, Guotai Wang, and Shaoting Zhang.  Word: Revisiting organs segmentation in the whole abdominal region.  _arXiv preprint arXiv:2111.02403_ , 2021.
  * Ma et al. [2021] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al.  Abdomenct-1k: Is abdominal organ segmentation a solved problem.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Ma et al. [2023] Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Ma, Adamo Young, Cheng Zhu, Kangkang Meng, Xin Yang, Ziyan Huang, et al.  Unleashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge.  _arXiv preprint arXiv:2308.05862_ , 2023.
  * Mahapatra et al. [2018] Dwarikanath Mahapatra, Behzad Bozorgtabar, Jean-Philippe Thiran, and Mauricio Reyes.  Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 580–588. Springer, 2018.
  * Oksuz et al. [2019] Ilkay Oksuz, Bram Ruijsink, Esther Puyol-Antón, James R Clough, Gastao Cruz, Aurelien Bustin, Claudia Prieto, Rene Botnar, Daniel Rueckert, Julia A Schnabel, et al.  Automatic cnn-based detection of cardiac mr motion artefacts using k-space data augmentation and curriculum learning.  _Medical image analysis_ , 55:136–147, 2019.
  * Oktay et al. [2018] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  _arXiv preprint arXiv:1804.03999_ , 2018.
  * Qin et al. [2022] Ziyuan Qin, Huahui Yi, Qicheng Lao, and Kang Li.  Medical image understanding with pretrained vision language models: A comprehensive study.  _arXiv preprint arXiv:2209.15517_ , 2022.
  * Qu et al. [2024] Chongyu Qu, Tiezheng Zhang, Hualin Qiao, Yucheng Tang, Alan L Yuille, Zongwei Zhou, et al.  Abdomenatlas-8k: Annotating 8,000 ct volumes for multi-organ segmentation in three weeks.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763. PMLR, 2021.
  * Rister et al. [2020] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin.  Ct-org, a new dataset for multiple organ segmentation in computed tomography.  _Scientific Data_ , 7(1):1–9, 2020.
  * Robinson et al. [2018] Robert Robinson, Ozan Oktay, Wenjia Bai, Vanya V Valindria, Mihir M Sanghvi, Nay Aung, José M Paiva, Filip Zemrak, Kenneth Fung, Elena Lukaschuk, et al.  Real-time prediction of segmentation quality.  In _MICCAI_ , pages 578–585. Springer, 2018.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , pages 234–241. Springer, 2015.
  * Roth et al. [2015] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers.  Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation.  In _MICCAI_ , pages 556–564. Springer, 2015.
  * Tang et al. [2022] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In _CVPR_ , pages 20730–20740, 2022.
  * Tarroni et al. [2018] Giacomo Tarroni, Ozan Oktay, Wenjia Bai, Andreas Schuh, Hideaki Suzuki, Jonathan Passerat-Palmbach, Antonio De Marvao, Declan P O’Regan, Stuart Cook, Ben Glocker, et al.  Learning-based quality control for cardiac mr images.  _IEEE transactions on medical imaging_ , 38(5):1127–1138, 2018.
  * Ulrich et al. [2023] Constantin Ulrich, Fabian Isensee, Tassilo Wald, Maximilian Zenk, Michael Baumgartner, and Klaus H Maier-Hein.  Multitalent: A multi-dataset approach to medical image segmentation.  In _MICCAI_ , pages 648–658. Springer, 2023.
  * Valindria et al. [2018] Vanya V Valindria, Nick Pawlowski, Martin Rajchl, Ioannis Lavdas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker.  Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri.  In _WACV_ , pages 547–556. IEEE, 2018.
  * Wang et al. [2022] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.  Medclip: Contrastive learning from unpaired medical images and text.  In _EMNLP_ , 2022.
  * Wasserthal et al. [2023] Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  _Radiology: Artificial Intelligence_ , 5(5), 2023.
  * Yoo and Kweon [2019] Donggeun Yoo and In So Kweon.  Learning loss for active learning.  In _CVPR_ , pages 93–102, 2019.
  * Zaman et al. [2023] Fahim Ahmed Zaman, Lichun Zhang, Honghai Zhang, Milan Sonka, and Xiaodong Wu.  Segmentation quality assessment by automated detection of erroneous surface regions in medical images.  _Computers in biology and medicine_ , 164:107324, 2023.
  * Zhang et al. [2021] Zhenxi Zhang, Jie Li, Chunna Tian, Zhusi Zhong, Zhicheng Jiao, and Xinbo Gao.  Quality-driven deep active learning method for 3d brain mri segmentation.  _Neurocomputing_ , 446:106–117, 2021.
  * Zhou et al. [2023] Tianfei Zhou, Liulei Li, Gustav Bredell, Jianwu Li, Jan Unkelbach, and Ender Konukoglu.  Volumetric memory network for interactive medical image segmentation.  _Medical Image Analysis_ , 83:102599, 2023.