In contrast to semantic segmentation, instance segmentation problems
distinguish different instances of the same class (e.g., different lesions).
Similarly to object detection problems, the task is to detect individual
instances of the same class, but detection performance is measured by pixel-
level correspondences (as in semantic segmentation problems). Optionally,
instances can be applied to one of multiple classes. Validation metrics in
instance segmentation problems often combine common detection metrics with
segmentation metrics applied per instance. For instance, segmentation
problems, we consider different localization criteria, namely:

##### Localization criteria:

Boundary Intersection over Union (Boundary IoU):

[Cheng et al., 2021]: Fig. SN 3.73

Mask IoU:

[Jaccard, 1912]: Fig. SN 3.76

Intersection over Reference (IoR):

[Maška et al., 2014]: Fig SN 3.75

##### Additional counting metric:

If detection and segmentation performance should be assessed simultaneously in
a single score, the PQ metric can be utilized [Kirillov et al., 2019]: Fig. SN
3.51.

It should be noted that instance segmentation problems are often phrased as
semantic segmentation problems with an additional post-processing step, such
as connected component analysis [Rosenfeld and Pfaltz, 1966].

##  2\. Metric Pitfalls

This section presents common limitations of image processing metrics related
to [P1] an inadequate choice of problem category (Suppl. Note 2.1), [P2] poor
metric selection (Suppl. Note 2.2) and [P3] poor metric application (Suppl.
Note 2.3) in an illustrated manner.

To preserve visual clarity, the most important of the presented metric values
may be highlighted with color. Green metric values correspond to a ”good”
metric value (e.g. a high Sensitivity score), whereas red values correspond to
a ”bad” value (e.g. a low Sensitivity). Green check marks indicate desirable
behavior of metrics, red crosses indicate undesirable behavior. Please note
that a low metric value is not automatically a ”bad” score. A metric value
should always be put into perspective and compared to inter-rater variability.
For simplicity, we still use the terms ”good” and ”bad/poor” throughout the
section. Finally, our illustrations do not provide the concrete class
probabilities of the presented classifiers.

###  2.1. Pitfalls related to an inadequate choice of the problem category

Performance metrics are typically expected to reflect a domain-specific (e.g.,
clinical) validation goal. Previous research, however, suggests that this is
often not the case [Saha et al., 2021]. Before choosing validation metrics,
the correct problem category needs to be defined. In the following, we present
pitfalls related to metrics not being applied to the appropriate problem
category. These can either be associated with a wrong choice of the problem
category (here: Figs. 3 and SN 2.3; more examples are provided in [Reinke et
al., 2021a]) or the lack of a matching problem category (Fig. SN 2.4).

Assessing object detection performance at image level yields misleading
results

Extended Data Fig. SN 2.3. Image-level classification metrics such as the Area
under the Receiver Operating Characteristic Curve (AUROC) curve can be used to
validate object detection models by first aggregating predictions to one
image-level score (per class). This validation scheme discards the information
on the object matching (localization, number of objects etc.). This leads to
several problems: (a) The image-level ROC curve does not measure the
localization performance. Both Prediction 1 and 2 are considered as True
Positive (TP) due to their score being very high, although Prediction 2 does
not hit the annotated object. (b) The image-level ROC is invariant to the
number of annotated objects in an image. The curve does not discriminate
between a model detecting all positives (Prediction 1) and a model detecting
only one of the positives (Prediction 2), as long as the maximum score is the
same. (c) The image-level ROC is invariant to the number of detections in an
image. The curve does not discriminate between a model with many False
Positives (FP) (Prediction 2), and a model with just one FP (Prediction 1), as
long as the maximum score is the same. The class probabilities are represented
by confidence scores (Conf.).

Common metrics may not reflect the domain interest

Extended Data Fig. SN 2.4. In the absence of a matching problem category for
the problem at hand, it may not be possible to find a common metric that
ideally captures the domain interest. In this example, accuracy of the ratio
between two volumes is the property of interest (e.g., the percentage of blood
volume ejected in each cardiac cycle [Bamira and Picard, 2018]). Using
overlap-based segmentation metrics (here: Dice Similarity Coefficient (DSC))
to measure the volumetric ratio may be misleading. Predictions 1 and 2 result
in similar averaged DSC metric values although they result in a different
ratio between structure volumes, which is the parameter of interest.
$\varnothing$ refers to the average DSC values.

###  2.2. Pitfalls related to poor metric selection

Validation metrics typically assess a specific property of interest. Thus, a
metric designed for a particular purpose often cannot be used to appropriately
validate another property. This is due to both the limitations as well as the
mathematical properties of individual metrics, both of which are often
neglected. In this section, we present pitfalls related to poor metric
selection.

####  2.2.1. Pitfalls related to disregard of the domain interest

Several requirements for metric selection arise from the domain interest,
which may clash with particular metric limitations. In the following, we
present pitfalls related to disregard of the domain interest, stemming from
the following sources:

  * •

Importance of structure boundaries (Figs. 4a and SN 2.5)

  * •

Importance of structure volume (Fig. SN 2.6)

  * •

Importance of structure center(line) (Fig. SN 2.7)

  * •

Importance of confidence awareness (Fig. SN 2.8)

  * •

Importance of comparability across data sets (Figs. SN 2.9)

  * •

Unequal severity of class confusions (Figs. 4b and SN 2.10)

  * •

Importance of cost-benefit analysis (Fig. SN 2.11)

Volume-based metrics alone are inadequate for assessing performance

Extended Data Fig. SN 2.5. Effect of only focusing on object volume. Both
Predictions 1 and 2 result in the correct volume difference of 0, but do not
overlap the reference ( Dice Similarity Coefficient (DSC) and Intersection
over Union (IoU) of 0). Only the boundary-based measures ( Hausdorff Distance
(HD), Hausdorff Distance 95th Percentile (HD95), Average Symmetric Surface
Distance (ASSD), Mean Average Surface Distance (MASD), and Normalized Surface
Distance (NSD)) recognize the mislocalization. This pitfall is also relevant
for localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask
IoU ¿ 0, Point inside Mask/Box/Approx, and Intersection over Reference (IoR) .

Boundary-based metrics disregard holes in the segmentation

Extended Data Fig. SN 2.6. Boundary-based metrics commonly ignore the overlap
between structures and are thus insensitive to holes in structures. In the
examples, the Prediction respectively features a hole or spotted pattern
within the object. Boundary-based metrics (here: Normalized Surface Distance
(NSD)) do not recognize this problem, yielding (near) perfect metric scores of
1.0 and 0.9, whereas the volumetric difference reflects the fact that the
inner area is inadequately predicted. NSD was calculated for $\tau=2$. This
pitfall is also relevant for other boundary-based metrics such as Average
Symmetric Surface Distance (ASSD), Boundary Intersection over Union (Boundary
IoU), Hausdorff Distance (HD), Hausdorff Distance 95th Percentile (HD95), and
Mean Average Surface Distance (MASD), as well as localization criteria such as
Center Distance, Mask IoU ¿ 0, Point inside Mask/Box/Appeox, Boundary IoU,
Intersection over Reference (IoR), and Mask IoU.

Overlap-based metrics are unaware of object centers

Extended Data Fig. SN 2.7. The most common counting-based metrics are poor
proxies for the center point alignment. Here, Predictions 1 and 2 yield the
same Dice Similarity Coefficient (DSC) value although Prediction 1
approximates the location of the object much better. This pitfall is also
relevant for other boundary- and overlap-based metrics such as Average
Symmetric Surface Distance (ASSD), Boundary Intersection over Union (IoU),
Hausdorff Distance (HD), Hausdorff Distance 95th Percentile (HD95), IoU,
pixel-level Fβ Score, and Mean Average Surface Distance (MASD), and
localization criteria such as Box/Approx/Mask IoU, Mask IoU ¿ 0, Point inside
Mask/Box/Approx, Boundary IoU, and Intersection over Reference (IoR).

Common calibration metrics falsely imply perfect calibration

Extended Data Fig. SN 2.8. Effect of different definitions of calibration on
the Expected Calibration Error (ECE) when focusing on confidence or predicted
class scores (confidence awareness). For top-label calibration, only the
maximum values of the predicted class scores $g(X)$ are considered, while all
other values are neglected, resulting in a perfect calibration for this
example. Similarly, for class-wise calibration, the predicted class scores are
compared class-wise per value, also yielding a perfect score. Only canonical
calibration considers all components of the predicted class score vectors,
showing that the model is not perfectly calibrated [Gruber and Buettner, 2022,
Vaicenavicius et al., 2019]. A more detailed insight in different definitions
of calibration is given in [Maier-Hein et al., 2022]. It should be noted that
discrimination metrics generally do not assess calibration performance, i.e.,
perfect discrimination does not imply good calibration performance.

Comparison of metric scores across data sets may be misleading

Extended Data Fig. SN 2.9. Effect of prevalence dependency. An algorithm with
specific inherent properties (here: Sensitivity of 0.9 and Specificity of 0.8)
may perform completely differently on different data sets if the prevalences
differ (here: 50% (left) and 90% (right)) and prevalence-dependent metrics are
used for validation (here: Accuracy, Positive Predictive Value (PPV), Negative
Predictive Value (NPV), F1 Score, Matthews Correlation Coefficient (MCC),
Cohen’s Kappa $\kappa$). In contrast, prevalence-independent metrics (here:
Balanced Accuracy (BA), Youden’s Index J, Positive Likelihood Ratio (LR+), and
Expected Cost (EC)) can be used to compare validation results across different
data sets. Used abbreviations: True Positive (TP), False Negative (FN), False
Positive (FP) and True Negative (TN). This pitfall is also relevant for other
counting metrics such as Net Benefit (NB).

Overlap-based metrics prefer oversegmentation over undersegmentation

Extended Data Fig. SN 2.10. Effect of undersegmentation vs. oversegmentation.
The outlines of the predictions of two algorithms (Prediction 1/2) differ in
only a single layer of pixels (Prediction 1: undersegmentation – smaller
structure compared to reference, Prediction 2: oversegmentation – larger
structure compared to reference). This has no (or only a minor) effect on the
Hausdorff Distance (HD)/(95%), the Normalized Surface Distance (NSD), MASD,
and the Average Symmetric Surface Distance (ASSD), but yields a substantially
different Dice Similarity Coefficient (DSC) or Intersection over Union (IoU)
score [Taha and Hanbury, 2015, Yeghiazaryan and Voiculescu, 2018]. If
penalizing of either over- or undersegmentation is desired (unequal severity
of class confusions), other metrics such as the Fβ Score provide specific
penalties for either depending on the chosen hyperparameter $\beta$. This
pitfall is also relevant for other overlap-based metrics such as centerline
Dice Similarity Coefficient (clDice) and localization criteria such as
Box/Approx/Mask IoU, Boundary IoU, and Intersection over Reference (IoR).

Common metrics disregard cost-benefit analysis

Extended Data Fig. SN 2.11. Effect of neglecting a cost-benefit analysis. In a
cost-benefit analysis, clinicians are able to define a risk-specific exchange
rate that is used in the computation of the Net Benefit (NB) metric. Common
metrics such as Accuracy do not consider this analysis and would favor the
marker-based decision on biopsy, while NB indicates that biopsies of all
patients actually yield a better clinical outcome [Vickers et al., 2016]. This
pitfall is also relevant for other counting metrics such as Balanced Accuracy
(BA), Positive Likelihood Ratio (LR+), Matthews Correlation Coefficient (MCC),
Negative Predictive Value (NPV), Positive Predictive Value (PPV), Sensitivity,
and Specificity. For binary problems, the hyperparameter $\beta$ of the Fβ
Score can be used as a dynamic penalty for class confusions.

####  2.2.2. Pitfalls related to disregard of the properties of the target
structure

For problems that require capturing local properties (object detection,
semantic or instance segmentation), the properties of the target structures to
be localized and/or segmented may have severe implications for metric choice.
Pitfalls can be further subdivided into size-related and shape- and topology-
related pitfalls. In the following, we present pitfalls stemming from the
following sources:

Size-related pitfalls:

  * •

Small structure sizes (Extended Data Fig. 1a and Fig. SN 2.12)

  * •

High variability of structure sizes (Fig. SN 2.13)

Shape- and topology-related pitfalls

  * •

Complex structure shapes (Extended Data Fig. 1b and Fig. SN 2.14)

  * •

Occurrence of overlapping or touching structures (Fig. SN 2.15)

  * •

Occurrence of disconnected structures (Fig. SN 2.16)

Common localization criteria and overlap-based metrics are sensitive to
structure sizes

Extended Data Fig. SN 2.12. Comparison of Mask and Boundary Intersection over
Union (IoU) localization criteria in the case of particular importance of
structure boundaries. Overlapping pixels from the reference and prediction are
shown in light blue. The Mask IoU (second column) is less sensitive to
boundary errors for large objects. The Boundary IoU (third and fourth column)
especially considers contours, (1) yields smaller metric scores, thus
penalizing errors in the boundaries, and (2) is more invariant to structure
sizes, leading to very similar values for large and small structures (fourth
column) [Cheng et al., 2021]. This pitfall is also relevant for other overlap-
based metrics such as centerline Dice Similarity Coefficient (clDice), Dice
Similarity Coefficient (DSC), and pixel-level Fβ Score, as well as
localization criteria such as Box/Approx IoU and Intersection over Reference
(IoR).

Effect of high variability of structure sizes

Extended Data Fig. SN 2.13. Large structures completely dominate overlap-based
metrics in semantic segmentation problems. While Prediction 1 perfectly
segments all three small structures, the metric score (here: Dice Similarity
Coefficient (DSC)) is much worse compared to the score of Prediction 2, with
only one perfect prediction for the large structure. This is highlighted by
only computing the metric without the large structure. This pitfall is also
relevant for other overlap-based metrics such as centerline Dice Similarity
Coefficient (clDice), Dice Similarity Coefficient (DSC), and pixel-level Fβ
Score, as well as localization criteria such as Mask/Box/Approx Intersection
over Union (IoU) and Intersection over Reference (IoR).

Common metrics are unaware of object shapes

Extended Data Fig. SN 2.14. Effect of complex shapes. Common overlap-based
metrics such as the Dice Similarity Coefficient (DSC) are unaware of complex
structure shapes and treat Predictions 1 and 2 equally. The centerline Dice
Similarity Coefficient (clDice) uncovers that Prediction 1 misses the fine-
granular branches of the reference and favors Prediction 2, which focuses on
the object’s center line and better captures its fine branches. This pitfall
is also relevant for other overlap-based metrics such as Intersection over
Union (IoU) and pixel-level Fβ Score, and localization criteria such as
Box/Approx/Mask IoU, Center Distance, Mask IoU ¿ 0, Point inside
Mask/Box/Approx, and Intersection over Reference (IoR).

Common metrics do not account for hierarchical label structure

Extended Data Fig. SN 2.15. Effect of nested multi-label structures. The
requirement of Label 2 being inside Label 1 is violated by Prediction 2.
Nevertheless, Prediction 2 has a higher Dice Similarity Coefficient (DSC)
score compared to Prediction 1, which adheres to the requirement. This pitfall
is also relevant for other boundary- and overlap-based metrics such as Average
Symmetric Surface Distance (ASSD), Boundary Intersection over Union (IoU),
centerline Dice Similarity Coefficient (clDice), Hausdorff Distance (HD),
Hausdorff Distance 95th Percentile (HD95), IoU, pixel-level Fβ Score, Mean
Average Surface Distance (MASD), and Normalized Surface Distance (NSD).

Bounding boxes are inadequate for representing complex shapes and disconnected
structures

Extended Data Fig. SN 2.16. Bounding boxes are not well-suited for
representing disconnected shapes, in particular multi-component structures.
Predictions 1 and 2 both yield a True Positive (TP) detection, as the Box
Intersection over Union (IoU) is larger than the threshold 0.3. However,
Prediction 1 does not hit the real object at all.

####  2.2.3. Pitfalls related to disregard of the properties of the data set
and algorithm output

Properties of the data set such as class imbalances or high inter-rater
variability may directly affect metric values. Pitfalls can be further
subdivided into class-related and reference-related pitfalls. For reference-
based metrics, the algorithm output will be compared against the reference
annotation to compute a metric score. Thus, the content and format of the
prediction is of high relevance for metric choice. In the following, we
present pitfalls stemming from the following sources:

[P2.3] Disregard of the properties of the data set

  * •

High class imbalance (Figs. 5a and SN 2.17)

  * •

Small test set size (Figs. 5b and SN 2.18)

  * •

Imperfect reference standard (Figs. 5c and SN 2.19)

[P2.4] Disregard of the properties of the algorithm output

  * •

Possibility of empty prediction (Extended Data Fig. 2b and Fig. SN 2.20)

  * •

Possibility of overlapping predictions (Extended Data Fig. 2a and Fig. SN
2.21)

  * •

Lack of predicted class scores (Fig. SN 2.22)

Common metrics yield implausible results in the presence of class imbalance

Extended Data Fig. SN 2.17. Effect of class imbalance. Not every metric is
designed to reflect class imbalance [Chicco and Jurman, 2020]. In the case of
underrepresented classes, an unsuitable metric, such as Accuracy, yields a
high value even if the classifier performs very poorly for one of the classes
(here: Prediction 2). Multi-threshold metrics, such as the Area under the
Receiver Operating Characteristic Curve (AUROC) and the Average Precision
(AP), reveal the weakness, indicating that Prediction 2 is not better than
random guessing. For comparison, a no-skill classifier (random guessing) is
shown as a black dashed line. For the Precision-Recall (PR) curves, the
interpolation applied to compute the AP metric is shown as a dashed grey line.
Thresholds used for curve generation are provided as small numbers above the
curve. Further abbreviations: Positive Predictive Value (PPV), Negative
Predictive Value (NPV), Matthews Correlation Coefficient (MCC), Weighted
Cohen’s Kappa (WCK). This pitfall is also relevant for other counting metrics
such as Net Benefit (NB).

Common multi-threshold metrics are not well-suited for small sample sizes

Extended Data Fig. SN 2.18. Effect of calculating the Area under the Receiver
Operating Characteristic Curve (AUROC) for very small sample sizes. The AUROC
is very unstable for small sample sizes. Data sets 1 and 2 only contain six
samples each, for which only one predicted score differs between sets. Drawing
the Receiver Operating Characteristic (ROC) curve and calculating the AUROC
leads to a large difference in scores between both data sets. The 95%
Confidence Interval (CI) reveals that there is a large range of possible AUROC
values. CIs were calculated based on [DeLong et al., 1988]. This pitfall is
also relevant for other counting metrics such as Accuracy, Average Precision
(AP), Balanced Accuracy (BA), Expected Cost (EC), Fβ Score, Free-Response
Receiver Operating Characteristic (FROC) Score, Positive Likelihood Ratio
(LR+), Matthews Correlation Coefficient (MCC), Net Benefit (NB), Negative
Predictive Value (NPV), Positive Predictive Value (PPV), Sensitivity,
Specificity, and Weighted Cohen’s Kappa (WCK).

Common metrics do not account for inter-rater variability

Extended Data Fig. SN 2.19. Effect of inter-rater variability between two
annotators. Assessing the performance of Annotator 2 while using an reference
annotation created by Annotator 1 leads to a low Dice Similarity Coefficient
(DSC) score because inter-rater variability is not taken into account by
common overlap-based metrics. In contrast, the Normalized Surface Distance
(NSD), applied with a threshold of $\tau=1$, captures this variability. It
should be noted, however, that this effect occurs primarily in small
structures as overlap-based metrics tend to be robust to variations in the
object boundaries in large structures. Further abbreviations: Intersection
over Union (IoU), Hausdorff Distance (HD), Hausdorff Distance 95th Percentile
(HD95), Average Symmetric Surface Distance (ASSD), Mean Average Surface
Distance (MASD). This pitfall is also relevant for other boundary- and
overlap-based metrics Boundary IoU, centerline Dice Similarity Coefficient
(clDice), pixel-level Fβ Score and Mean Average Surface Distance (MASD) and
localization criteria such as Mask IoU ¿ 0, Point inside Mask, Boundary IoU,
IoR, and Mask IoU.

Empty reference or prediction leads to invalid scores

Extended Data Fig. SN 2.20. Effect of empty references or predictions when
applying common metrics per image (here for semantic segmentation). Empty
images lead to division by zero for many common metrics as the numbers of the
TPs, FP, FN turn zero. Used abbreviations: Average Symmetric Surface Distance
(ASSD), Dice Similarity Coefficient (DSC), Hausdorff Distance (HD), Hausdorff
Distance 95th Percentile (HD95), Intersection over Union (IoU), Mean Average
Surface Distance (MASD), Not a Number (NaN), Normalized Surface Distance
(NSD). This pitfall is also relevant for other boundary-based, overlap-based
and counting metrics such as Boundary IoU, centerline Dice Similarity
Coefficient (clDice), Fβ Score, Negative Predictive Value (NPV), Positive
Predictive Value (PPV), Sensitivity, and Specificity.

Common segmentation metrics are not well-suited for overlapping structures

Extended Data Fig. SN 2.21. Effect of overlapping predictions in segmentation
problems. In semantic segmentation problem (SemS; right), overlapping
predictions are merged into a single object, yielding a perfect metric score.
Phrasing the problem as an instance segmentation problem reveals that the dark
blue instance is not well-approximated at all. This issue is not revealed by
common metrics if only semantic segmentation is performed (here: Dice
Similarity Coefficient (DSC)). This pitfall is also relevant for other
boundary- and overlap-based metrics such as Average Symmetric Surface Distance
(ASSD), Boundary Intersection over Union (IoU), centerline Dice Similarity
Coefficient (clDice), pixel-level Fβ Score, Hausdorff Distance (HD), Hausdorff
Distance 95th Percentile (HD95), IoU, Mean Average Surface Distance (MASD),
and Normalized Surface Distance (NSD).

Selection of multi-threshold metrics in the absence of predicted class scores

Extended Data Fig. SN 2.22. Multi-threshold metrics should only be computed if
predicted class scores are available, although an increasing body of work
computes multi-threshold metrics such as AP in the absence of class scores
(e.g., [Bai and Urtasun, 2017, De Brabandere et al., 2017, Gao et al., 2019,
Hirsch et al., 2020, Kulikov and Lempitsky, 2020]). Otherwise, the strategy
chosen for compensating the lack of class scores (here reflected by
Implementations 1 and 2) leads to metric scores that are less well
interpretable than those of established counting metrics working on a fixed
confusion matrix (here: F${}_{\text{1}}$ Score). This pitfall is also relevant
for other multi-threshold metrics such as Area under the Receiver Operating
Characteristic Curve (AUROC) and Free-Response Receiver Operating
Characteristic (FROC) Score.

###  2.3. Pitfalls related to poor metric application

A data set typically contains several hundreds or thousands of images. When
analyzing, aggregating and combining metric values, a number of factors need
to be taken into account.

####  2.3.1. Pitfalls related to inadequate metric implementation

The implementation of metrics is, unfortunately, not standardized. While some
metrics are straightforward to implement, others require more advanced
techniques and offer a variety of implementation possibilities. Sources of
metric implementation pitfalls include:

  * •

Non-standardized definitions (Figs. 6a and SN 2.23)

  * •

Discretization issues (Fig. SN 2.24)

  * •

Sensitivity to hyperparameters (Fig. SN 2.25)

  * •

Metric-specific issues (Fig. SN 2.26)

Lack of standardization leads to variation in metric scores

Extended Data Fig. SN 2.23. Effect of defining different ranges for the False
Positives per Image (FPPI) (which are unbounded to the top) used to draw the
Free-Response Receiver Operating Characteristic (FROC) curve for the same
prediction (top). The resulting FROC Scores differ for different boundaries of
the x-axis used for the FPPI ([0, 1], [0, 2] and [0, 4]). Publications make
use of different ranges for the x-axis, complicating comparison between works.

Common metrics suffer from discretization issues

Extended Data Fig. SN 2.24. Effect of choosing different bins for calculating
the Expected Calibration Error (ECE) and Maximum Calibration Error (MCE).
Three different strategies are chosen for the binning of the interval [0, 1]
of the predicted class scores of the Prediction. The resulting metric scores
are substantially affected by the number of bins [Guo et al., 2017].

Choice of hyperparameters may have largely impact metric scores

Extended Data Fig. SN 2.25. Effect of the Intersection over Union (IoU)
threshold on the localization (here Box IoU). (a) When defining a True
Positive (TP) by a very loose IoU ¿ 0, the resulting localizations may be
deceived by very large predictions. (b) On the other hand, a strict IoU
criterion may be problematic when the bounding box does not approximate the
target structure shapes well. Although Predictions 1 and 2 are very similar
(differing in one pixel in one dimension), only Prediction 1 is a TP because
the number of bounding box pixels increases quadratically with the size of
diagonal narrow structures. Further abbreviation: False Positive (FP).

Per-class tuning of the decision threshold yields misleading results

Extended Data Fig. SN 2.26. Effect of the determination of a global threshold
for all classes based on a single class. In a data set of three classes and
nine images, the Area under the Receiver Operating Characteristic Curve
(AUROC) score is 1.0 for every class. In practice, however, a global decision
threshold needs to be set in multi-class problems, which typically renders
substantially worse results. Here, the optimal threshold for Class 1 yields
poor results for Classes 2 and 3 (see e.g., [Krause et al., 2018, De Fauw et
al., 2018]). Used abbreviations: Positive Predictive Value (PPV), Negative
Predictive Value (NPV), Matthews Correlation Coefficient (MCC), Cohen’s Kappa
$\kappa$, and Balanced Accuracy (BA).

####  2.3.2. Pitfalls related to inadequate metric aggregation

When aggregating metric values over multiple cases (data points), the method
of metric aggregation should be clearly defined and reported including details
for example on the aggregation operator (e.g., mean or median) and missing
value handling. In addition, special care should be taken when aggregating
across classes or different hierarchy levels. Pitfalls can be further
subdivided into class-related and data set-related pitfalls. In the following,
we present pitfalls stemming from the following sources:

Class-related pitfalls

  * •

Hierarchical label structure (Fig. SN 2.27)

  * •

Multi-class problem (Fig. SN 2.28)

Data set-related pitfalls

  * •

Non-independence of test cases (Figs. 6b and SN 2.29)

  * •

Risk of bias (Fig. SN 2.30)

  * •

Possibility of invalid prediction (Fig. SN 2.31)

Standard aggregation schemes disregard hierarchical class structures

Extended Data Fig. SN 2.27. Classes in categorical classification may be
hierarchically structured, for example in the form of multiple positive
classes and one negative class. The phrasing of the problem as binary vs.
multi-class hugely affects the validation result. Binary classification
(middle), differentiating triangles from circles, yields a good Accuracy,
while per-class validation yields a poor score because the two circle classes
cannot be distinguished well. Incorrect predictions are overlaid by a red
shape of the correct reference class.

Lack of per-class validation conceals important information

Extended Data Fig. SN 2.28. Effect of ignoring the presence of multiple
classes when aggregating metric values (here: using the mean). The overall
average of all Dice Similarity Coefficient (DSC) scores for the four images is
0.7. Averaging per class reveals a very low performance for Classes 2 and 3.
$\varnothing$ refers to the average DSC values.

Inter-class dependencies are concealed in standard aggregation schemes

Extended Data Fig. SN 2.29. Effect of interdependencies between classes. A
prediction may show a near-perfect Accuracy score of 0.94 for the dark blue
triangle as it frequently appears in conjunction with the orange square. By
calculating the Accuracy in the presence and absence of the orange square
class, it can be seen that the algorithm only works well in the presence of
the orange square class.

Lack of stratification conceal biases

Extended Data Fig. SN 2.30. Effect of disregarding relevant meta-information
(here: gender). When ignoring the available meta-information of the patient’s
gender per image, any metric (here: Accuracy) fails to reveal that the
algorithm performs much better for men compared to women. In this example,
correct predictions are marked by a green check mark, incorrect predictions by
a red cross.

Lack of missing data handling strategy yields misleading results

Extended Data Fig. SN 2.31. Effect of invalid predictions (missing values)
when aggregating metric values. In this example, ignoring missing values leads
to a substantially higher Dice Similarity Coefficient (DSC)) compared to
setting missing values to the worst possible value (here: 0).

####  2.3.3. Pitfalls related to inadequate ranking scheme

Rankings are often created to compare algorithm performances. In this context,
we present pitfalls stemming from the following sources:

  * •

Metric relationships (Fig. SN 2.32)

  * •

Ranking uncertainty (Fig. SN 2.33)

Related metrics may yield identical rankings

Extended Data Fig. SN 2.32. Effect of using mathematically closely related
metrics. The Dice Similarity Coefficient (DSC) and Intersection over Union
(IoU) typically lead to the same ranking, whereas metrics from different
families (here: Hausdorff Distance (HD)) may lead to substantially different
rankings [Taha et al., 2014, Taha and Hanbury, 2015]. Combining metrics that
are related will not provide additional information for a ranking, and having
multiple metrics measuring the same properties may overrule rankings of other
properties (here: HD).

Ranking tables do not reflect ranking uncertainty

Extended Data Fig. SN 2.33. Effect of ranking uncertainty. The results of two
benchmarking experiments with five algorithms $A1$-$A5$ differ substantially,
as shown by the boxplots of the metric values for every algorithm. While the
left situation introduces a clear ranking visible from the boxplots, the right
use case is not clear as performance is very similar across algorithms.
However, both situations lead to the same ranking [Maier-Hein et al., 2018,
Wiesenfarth et al., 2021]. Thus, solely providing ranking tables conceals
information on ranking uncertainty.

####  2.3.4. Pitfalls related to inadequate metric reporting

A thorough reporting of metric values and aggregates is important both in
terms of transparency and interpretability. However, several pitfalls are to
be avoided in this regard. Sources of metric reporting pitfalls include:

  * •

Non-determinism of algorithms (Fig. 6)

  * •

Uninformative visualization (Figs. 6c and SN 2.35)

The non-determinism of neural networks effects metric results

Extended Data Fig. SN 2.34. Effect of non-determinism of artificial
intelligence (AI) algorithms. An algorithm trained under identical conditions
may yield different results when changing seeds (left), but also with fixed
seeds (right). The latter may, for example, be caused by parallel processes,
order of threads, auto-selection of primitive operations, and other factors
[Pham et al., 2020]666See for example:
https://pytorch.org/docs/stable/notes/randomness.html. Fixing seeds does not
guarantee reproducibility even for the same hardware/software configuration as
many software libraries have a degree of randomness on their operations.

Common visualization schemes conceal relevant information

Extended Data Fig. SN 2.35. Effect of different visualization types. A single
boxplot (top left) does not provide sufficient information about the raw
metric value distribution (here: Dice Similarity Coefficient (DSC)). Using a
violin plot (top right) or adding the raw metric values as jittered dots on
top (bottom left) adds important information. In the case of non-independent
validation data, color/shape-coding helps reveal data clusters (bottom right).

####  2.3.5. Pitfalls related to inadequate interpretation of metric values

Interpreting metric scores and aggregates is an important step in algorithm
performance analysis. However, several pitfalls can arise from interpretation.
In the following, we present pitfalls related to:

  * •

Low resolution (Fig. SN 2.36)

  * •

Lack of upper/lower bounds (Fig. SN 2.37)

  * •

Insufficient domain relevance of metric score differences (Fig. SN 2.38)

Image resolution affects metric scores

Extended Data Fig. SN 2.36. Effect of different grid sizes. Differences in the
grid size (resolution) of an image highly influence the image and the
reference annotation (dark blue shape (reference) vs. pink outline (desired
circle shape)), with a prediction of the exact same shape leading to different
metric scores. Abbreviations: Dice Similarity Coefficient (DSC), Intersection
over Union (IoU), Hausdorff Distance (HD), Hausdorff Distance 95th Percentile
(HD95), Average Symmetric Surface Distance (ASSD), Mean Average Surface
Distance (MASD), Normalized Surface Distance (NSD).

Lower bounds of metrics may not be achievable in practice

Extended Data Fig. SN 2.37. Effect of theoretical bounds that may not be
achievable in practice. In this multi-class example, all samples were
predicted incorrectly. However, the theoretical lowest value for the Matthews
Correlation Coefficient (MCC) metric (-1) cannot be achieved in this
situation, rendering interpretation difficult.

Metric score differences leading to different rankings may be irrelevant

Extended Data Fig. SN 2.38. Effect of irrelevant metric score differences in
rankings. The difference of the metric score aggregates of algorithms A1 and
A2 is extremely low and not of biomedical relevance. However, the numerical
difference would assign them different ranks.

##  3\. Metric Profiles

This section presents profiles for the metrics deemed particularly relevant by
the Metrics Reloaded consortium [Maier-Hein et al., 2022]. For each metric,
the respective description, formula, and value range (upward arrow: higher
values better than lower values; downward arrow: lower values are better than
higher values) are provided, along with further important characteristics,
such as the used cardinalities of a confusion matrix, or potential prevalence
dependency. Finally, relevant pitfalls are highlighted. Many of the presented
metrics rely on the confusion matrix, which is illustrated in Fig. SN 3.39.

Extended Data Fig. SN 3.39. Schematic example of the confusion matrix for two
and for $C$ classes. For the latter case, we also present a weight or cost
matrix with weights $w_{ij}>0$ without loss of generality. For the binary
confusion matrix, we show an example illustrating the cardinalities for a
prediction of triangles and circles.

###  3.1. Discrimination metrics

####  3.1.1. Counting metrics

+

Extended Data Fig. SN 3.40. Metric profile of Accuracy. The upward arrow in
the value range indicates that higher values are better than lower values.
Abbreviations: False Negative (FN), False Positive (FP), Instance Segmentation
(InS), Object Detection (ObD), True Negative (TN), True Positive (TP).
Reference: Tharwat, 2020: [Tharwat, 2020]. Mentioned figures: Figs. 4b, 5a, SN
2.9, SN 2.11, SN 2.17.

Extended Data Fig. SN 3.41. Metric profile of Balanced Accuracy (BA). The
upward arrow in the value range indicates that higher values are better than
lower values. Abbreviations: False Negative (FN), False Positive (FP),
Instance Segmentation (InS), Object Detection (ObD), Positive Predictive Value
(PPV), True Negative (TN), True Positive (TP). References: Grandini et al.,
2020: [Grandini et al., 2020], Maier-Hein et al., 2022: [Maier-Hein et al.,
2022], Reinke et al., 2021: [Reinke et al., 2021a], Tharwat, 2020: [Tharwat,
2020]. Mentioned figures: Figs. 4b, 5a, SN 2.11. Extended Data Fig. SN 3.42.
Metric profile of centerline Dice Similarity Coefficient (clDice). The upward
arrow in the value range indicates that higher values are better than lower
values. Abbreviations: False Negative (FN), False Positive (FP), True Negative
(TN), True Positive (TP). Reference: Shit et al., 2021: [Shit et al., 2021].
Mentioned figures: Extended Data Fig. 1a, Figs. SN 2.10, SN 2.12, SN 2.13, SN
2.19, SN 2.20. Extended Data Fig. SN 3.43. Metric profile of Dice Similarity
Coefficient (DSC). The upward arrow in the value range indicates that higher
values are better than lower values. Abbreviations: False Negative (FN), False
Positive (FP), Positive Predictive Value (PPV), True Negative (TN), True
Positive (TP). References: Dice, 1945: [Dice, 1945], Reinke et al., 2021:
[Reinke et al., 2021a]. Mentioned figures: Figs. 4a, SN 2.7, SN 2.10, SN 2.12,
SN 2.13, SN 2.14, SN 2.19, SN 2.20, Extended Data Fig. 1a-b. Extended Data
Fig. SN 3.44. Metric profile of Expected Cost (EC). The downward arrow in the
value range indicates that lower values are better than higher values.
Abbreviations: False Negative (FN), False Positive (FP), True Negative (TN),
True Positive (TP). References: Bishop and Nasrabadi, 2006: [Bishop and
Nasrabadi, 2006], Ferrer 2022: [Ferrer, 2022], Hastie et al., 2009: [Hastie et
al., 2009], Maier-Hein et al., 2022: [Maier-Hein et al., 2022]. Extended Data
Fig. SN 3.45. Metric profile of Fβ Score.[Chinchor, 1992, Van Rijsbergen,
1979]. The upward arrow in the value range indicates that higher values are
better than lower values. Abbreviations: Dice Similarity Coefficient (DSC),
False Negative (FN), False Positive (FP), Positive Predictive Value (PPV),
True Negative (TN), True Positive (TP).References: Chinchor 1992: [Chinchor,
1992], Reinke et al., 2021: [Reinke et al., 2021a], Van Rijsbergen, 1979: [Van
Rijsbergen, 1979]. Mentioned figures: Figs. 4a, SN 2.7, SN 2.9, SN 2.12, SN
2.14, SN 2.17, SN 2.19, SN 2.20, Extended Data Figs. 1a-b and 2b. Extended
Data Fig. SN 3.46. Metric profile of False Positives per Image (FPPI). The
upward arrow in the value range indicates that higher values are better than
lower values. Abbreviations: False Negative (FN), False Positive (FP), Free-
Response Receiver Operating Characteristic (FROC), True Negative (TN), True
Positive (TP). References: Bandos et al., 2009: [Bandos et al., 2009], Reinke
et al., 2021: [Reinke et al., 2021a], Van Ginneken et al., 2010: [Van Ginneken
et al., 2010]. Mentioned figure: Fig. SN 2.23. Extended Data Fig. SN 3.47.
Metric profile of Intersection over Union (IoU). The upward arrow in the value
range indicates that higher values are better than lower values.
Abbreviations: False Negative (FN), False Positive (FP), Positive Predictive
Value (PPV), True Negative (TN), True Positive (TP). References: Jaccard,
1912: [Jaccard, 1912], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned
figures: Figs. 4a, SN 2.7, SN 2.10, SN 2.12, SN 2.13, SN 2.14, SN 2.19, SN
2.20, Extended Data Fig. 1a-b. Extended Data Fig. SN 3.48. Metric profile of
Matthews Correlation Coefficient (MCC). The upward arrow in the value range
indicates that higher values are better than lower values. Abbreviations:
Expected Cost (EC), False Negative (FN), False Positive (FP), Instance
Segmentation (InS), Object Detection (ObD), True Negative (TN), True Positive
(TP). References: Ferrer, 2022: [Ferrer, 2022], Matthews, 1975: [Matthews,
1975], Reinke et al., 2021: [Reinke et al., 2021a], Zhu, 2020: [Zhu, 2020].
Mentioned figures: Figs. 4b, SN 2.9, SN 2.11, SN 2.37. Extended Data Fig. SN
3.49. Metric profile of Net Benefit (NB). The upward arrow in the value range
indicates that higher values are better than lower values. Abbreviations:
Expected Cost (EC), False Negative (FN), False Positive (FP), True Negative
(TN), True Positive (TP). References: Ferrer, 2022: [Ferrer, 2022], Vickers
and Elkin, 2006: [Vickers and Elkin, 2006], Vickers et al., 2016: [Vickers et
al., 2016]. Mentioned figure: Fig. SN 2.22. Extended Data Fig. SN 3.50. Metric
profile of Negative Predictive Value (NPV). The upward arrow in the value
range indicates that higher values are better than lower values.
Abbreviations: False Negative (FN), False Positive (FP), Instance Segmentation
(InS), Object Detection (ObD), True Negative (TN), True Positive (TP).
References: Reinke et al., 2021: [Reinke et al., 2021a], Tharwat, 2020:
[Tharwat, 2020]. Mentioned figures: Figs. 4b, SN 2.9, SN 2.11, SN 2.17, SN
2.20, Extended Data Fig. 2b. Extended Data Fig. SN 3.51. Metric profile of
Panoptic Quality (PQ). The upward arrow in the value range indicates that
higher values are better than lower values. Abbreviations: Average Precision
(AP), False Negative (FN), False Positive (FP), Free-Response Receiver
Operating Characteristic (FROC), Intersection over Union (IoU), True Negative
(TN), True Positive (TP). References: Kirillov et al., 2019: [Kirillov et al.,
2019], Reinke et al., 2021: [Reinke et al., 2021a]. Extended Data Fig. SN
3.52. Metric profile of Positive Likelihood Ratio (LR+). The upward arrow in
the value range indicates that higher values are better than lower values.
Abbreviations: False Negative (FN), False Positive (FP), Instance Segmentation
(InS), Object Detection (ObD), Positive Predictive Value (PPV), True Negative
(TN), True Positive (TP). References: Attia, 2003: [Attia, 2003], Reinke et
al., 2021: [Reinke et al., 2021a]. Mentioned figures: Figs. 4b, 5a, SN 2.11,
SN 2.20, Extended Data Fig. 2b. Extended Data Fig. SN 3.53. Metric profile of
the Positive Predictive Value (PPV). The upward arrow in the value range
indicates that higher values are better than lower values. Abbreviations used
in the figure: False Negative (FN), False Positive (FP), Instance Segmentation
(InS), Object Detection (ObD), True Negative (TN), True Positive (TP).
References used in the figure: Reinke et al., 2021: [Reinke et al., 2021a],
Tharwat, 2020: [Tharwat, 2020]. Mentioned figures: Figs. 4b, SN 2.9, SN 2.11,
SN 2.17, SN 2.20, Extended Data Fig. 2b. Extended Data Fig. SN 3.54. Metric
profile of Sensitivity. The upward arrow in the value range indicates that
higher values are better than lower values. Abbreviations: False Negative
(FN), False Positive (FP), Object Detection (ObD), Positive Predictive Value
(PPV), True Negative (TN), True Positive (TP). References: Maier-Hein et al.,
2022: [Maier-Hein et al., 2022], Reinke et al., 2021: [Reinke et al., 2021a],
Tharwat, 2020: [Tharwat, 2020]. Mentioned figures: Figs. 4b, SN 2.11, SN 2.20,
Extended Data Fig. 2b. Extended Data Fig. SN 3.55. Metric profile of
Specificity. The upward arrow in the value range indicates that higher values
are better than lower values. Abbreviations: False Negative (FN), False
Positive (FP), True Negative (TN), True Positive (TP). References: Reinke et
al., 2021: [Reinke et al., 2021a], Tharwat, 2020: [Tharwat, 2020]. Mentioned
figures: Figs. 4b SN 2.11, SN 2.20, Extended Data Fig. 2b. Extended Data Fig.
SN 3.56. Metric profile of Weighted Cohen’s Kappa (WCK). The upward arrow in
the value range indicates that higher values are better than lower values.
Abbreviations: Balanced Accuracy (BA), Cohen’s Kappa (CK), Expected Cost (EC),
False Negative (FN), False Positive (FP), Instance Segmentation (InS), Object
Detection (ObD), True Negative (TN), True Positive (TP). References: Cohen,
1960: [Cohen, 1960], Delgado and Tibau, 2019: [Delgado and Tibau, 2019],
Ferrer, 2022: [Ferrer, 2022], Powers, 2012: [Powers, 2012], Reinke et al.,
2021: [Reinke et al., 2021a], Warrens, 2012: [Warrens, 2012]. Mentioned
figures: Figs. SN 2.9, SN 2.17, SN 2.37.

####  3.1.2. Multi-threshold metrics

Extended Data Fig. SN 3.57. Metric profile of Area under the Receiver
Operating Characteristic Curve (AUROC). The upward arrow in the value range
indicates that higher values are better than lower values. Abbreviations:
False Negative (FN), False Positive (FP), Instance Segmentation (InS), Object
Detection (ObD), Receiver Operating Characteristic (ROC), True Negative (TN),
True Positive (TP). References: Hanley and McNeil, 1982: [Hanley and McNeil,
1982], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures: Figs.
5a, SN 2.3, SN 2.18, SN 2.22.

Extended Data Fig. SN 3.58. Metric profile of Average Precision (AP). The
upward arrow in the value range indicates that higher values are better than
lower values. Abbreviations: Area under the Receiver Operating Characteristic
Curve (AUROC), False Negative (FN), False Positive (FP), Instance Segmentation
(InS), Object Detection (ObD), Precision-Recall (PR), True Negative (TN), True
Positive (TP). References: Everingham et al., 2015: [Everingham et al., 2010],
Lin et al., 2014: [Lin et al., 2014], Maier-Hein et al., 2022: [Maier-Hein et
al., 2022], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures:
Figs. 6a, SN 2.9, SN 2.22. Extended Data Fig. SN 3.59. Metric profile of Free-
Response Receiver Operating Characteristic (FROC). The upward arrow in the
value range indicates that higher values are better than lower values.
Abbreviations: False Negative (FN), False Positive (FP), False Positives per
Image (FPPI), True Negative (TN), True Positive (TP). References: Van Ginneken
et al., 2010: [Van Ginneken et al., 2010]. Mentioned figures: Figs. SN 2.22,
SN 2.23.

####  3.1.3. Distance-based metrics

Extended Data Fig. SN 3.60. Metric profile of Average Symmetric Surface
Distance (ASSD). The downward arrow in the value range indicates that lower
values are better than higher values. Abbreviation: Semantic Segmentation
(SemS). References: Reinke et al., 2021: [Reinke et al., 2021a], Yeghiazaryan,
Varduhi and Voiculescu, 2015: [Yeghiazaryan and Voiculescu, 2015]. Mentioned
figures: Figs. 5c, SN 2.6, SN 2.19, SN 2.20.

Extended Data Fig. SN 3.61. Metric profile of the Boundary Intersection over
Union (IoU). The upward arrow in the value range indicates that higher values
are better than lower values. References: Cheng et al., 2021: [Cheng et al.,
2021], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures: Figs.
5c, SN 2.6, SN 2.19, SN 2.20. Extended Data Fig. SN 3.62. Metric profile of
Hausdorff Distance (HD). The downward arrow in the value range indicates that
lower values are better than higher values. Abbreviation: Semantic
Segmentation (SemS). References : Huttenlocher, 1993: [Huttenlocher et al.,
1993], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures: Figs.
5c, SN 2.6, SN 2.19, SN 2.20. Extended Data Fig. SN 3.63. Metric profile of
Mean Average Surface Distance (MASD). The downward arrow in the value range
indicates that lower values are better than higher values. Abbreviation:
Semantic Segmentation (SemS). References: Beneš and Zitová, 2015: [Beneš and
Zitová, 2015], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures:
Figs. 5c, SN 2.6, SN 2.19, SN 2.20. Extended Data Fig. SN 3.64. Metric profile
of Normalized Surface Distance (NSD). The upward arrow in the value range
indicates that higher values are better than lower values. Abbreviation: Dice
Similarity Coefficient (DSC). References: Nikolov et al., 2021: [Nikolov et
al., 2021], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures:
Figs. SN 2.6, SN 2.20. Extended Data Fig. SN 3.65. Metric profile of Xth
Percentile of Hausdorff Distance (HD). The downward arrow in the value range
indicates that lower values are better than higher values. Abbreviations:
Hausdorff Distance (HD), Semantic Segmentation (SemS). References:
Huttenlocher, 1993: [Huttenlocher et al., 1993], Reinke et al., 2021: [Reinke
et al., 2021a]. Mentioned figures: Figs. SN 2.6, SN 2.19, SN 2.20.

###  3.2. Calibration metrics

Extended Data Fig. SN 3.66. Metric profile of Brier Score (BS). The downward
arrow in the value range indicates that lower values are better than higher
values. Abbreviation: Brier Skill Score (BSS). References: Gneiting and
Raftery, 2007: [Gneiting and Raftery, 2007], Reinke et al., 2021: [Reinke et
al., 2021a]. Mentioned figure: Fig. SN 2.9.

Extended Data Fig. SN 3.67. Metric profile of Class-Wise Calibration Error
(CWCE). The downward arrow in the value range indicates that lower values are
better than higher values. References: Kumar et al., 2019: [Kumar et al.,
2019], Kull et al., 2019: [Kull et al., 2019]. Mentioned figures: Figs. 5b, SN
2.8, SN 2.24. Extended Data Fig. SN 3.68. Metric profile of Expected
Calibration Error (ECE). The downward arrow in the value range indicates that
lower values are better than higher values. References: Maier-Hein et al.,
2022: [Maier-Hein et al., 2022], Naeini et al., 2015: [Naeini et al., 2015],
Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figures: Figs. 5b, SN
2.8, SN 2.24. Extended Data Fig. SN 3.69. Metric profile of Expected
Calibration Error Kernel Density Estimate (ECEKDE). The downward arrow in the
value range indicates that lower values are better than higher values.
Abbreviation: Expected Calibration Error (ECE). Reference used in the figure:
Popordanoska et al., 2022: [Popordanoska et al., 2022]. Extended Data Fig. SN
3.70. Metric profile of Kernel Calibration Error (KCE). References: Gruber and
Buettner, 2022: [Gruber and Buettner, 2022], Widmann et al., 2019: [Widmann et
al., 2019]. Extended Data Fig. SN 3.71. Metric profile of Negative Log
Likelihood (NLL). The downward arrow in the value range indicates that lower
values are better than higher values. References: Cybenko et al., 1998:
[Cybenko et al., 1998], Popordanoska et al., 2022: [Popordanoska et al.,
2022], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned figure: Fig. 5b.
Extended Data Fig. SN 3.72. Metric profile of Root Brier Score (RBS). The
downward arrow in the value range indicates that lower values are better than
higher values. Reference: Gruber and Buettner, 2022: [Gruber and Buettner,
2022].

###  3.3. Localization criteria

Extended Data Fig. SN 3.73. Metric profile of the Boundary Intersection over
Union (IoU) localization criterion. The upward arrow in the value range
indicates that higher values of Boundary IoU are better than lower values.
References: Cheng et al., 2021: [Cheng et al., 2021], Reinke et al., 2021:
[Reinke et al., 2021a]. Mentioned figures: Figs. SN 2.6, SN 2.20.

Extended Data Fig. SN 3.74. Metric profile of the Center Distance localization
criterion. The downward arrow in the value range indicates that lower values
of the Center Distance are better than higher values. References: Gurcan et
al., 2010: [Gurcan et al., 2010], Reinke et al., 2021: [Reinke et al., 2021a].
Extended Data Fig. SN 3.75. Metric profile of the Intersection over Reference
(IoR) localization criterion. The upward arrow in the value range indicates
that higher values of IoR are better than lower values. Abbreviations: False
Negative (FN), False Positive (FP), True Negative (TN), True Positive (TP).
References: Maška et al., 2014: [Maška et al., 2014], Reinke et al., 2021:
[Reinke et al., 2021a]. Mentioned figures: Figs. 4a, SN 2.7, SN 2.10, SN 2.12,
SN 2.13, SN 2.14, SN 2.20, SN 2.25, Extended Data Fig. 1b. Extended Data Fig.
SN 3.76. Metric profile of the Mask/Box/Approx Intersection over Union (IoU)
localization criterion. Abbreviations: False Negative (FN), False Positive
(FP), True Negative (TN), True Positive (TP). References: Jaccard, 1912:
[Jaccard, 1912], Reinke et al., 2021: [Reinke et al., 2021a]. Mentioned
figures: Figs. 4a, SN 2.7, SN 2.10, SN 2.12, SN 2.13,SN 2.14, SN 2.16, SN
2.19, SN 2.20, SN 2.25, Extended Data Fig. 1a-b. Extended Data Fig. SN 3.77.
Metric profile of the Mask Intersection over Union (IoU) ¿ 0 localization
criterion. Abbreviations: False Negative (FN), False Positive (FP), True
Negative (TN), True Positive (TP). References: Jaccard, 1912: [Jaccard, 1912],
Wack et al., 2012: [Wack et al., 2012]. Mentioned figure: Fig. SN 2.25.
Extended Data Fig. SN 3.78. Metric profile of Point inside
Mask/Box/Approximation. References: https://cada.grand-
challenge.org/Assessment/, Reinke et al., 2021: [Reinke et al., 2021a].

###  3.4. Assignment strategies

Extended Data Fig. SN 3.79. Cheat Sheet for the Greedy (by Score) Matching.
Reference used in the figure: Everingham et al., 2015: [Everingham et al.,
2015].

Extended Data Fig. SN 3.80. Cheat Sheet for the Greedy (by Localization
Criterion) Matching. Reference used in the figure: Maier-Hein et al., 2022:
[Maier-Hein et al., 2022]. Extended Data Fig. SN 3.81. Cheat Sheet for the
Optimal (Hungarian) Matching. References used in the figure: Kuhn et al.,
1955: [Kuhn, 1955], Maier-Hein et al., 2022: [Maier-Hein et al., 2022].
Extended Data Fig. SN 3.82. Cheat Sheet for the Matching via Overlap ¿ 0.5.
References used in the figure: Everingham et al., 2006: [Everingham et al.,
2006], Maier-Hein et al., 2022: [Maier-Hein et al., 2022].

## Acronyms

Acc

    Accuracy
AI

    artificial intelligence
AP

    Average Precision
ASSD

    Average Symmetric Surface Distance
AUC

    Area under the Curve
AUROC

    Area under the Receiver Operating Characteristic Curve
BA

    Balanced Accuracy
BIAS

    Biomedical Image Analysis ChallengeS
Boundary IoU

    Boundary Intersection over Union
Box IoU

    Box Intersection over Union
BM

    Bookmaker Informedness
BPMN

    Business Process Model and Notation
BS

    Brier Score
BSS

    Brier Skill Score
CI

    Confidence Interval
clDice

    centerline Dice Similarity Coefficient
COCO

    Common Objects in Context
CK

    Cohen’s Kappa
CWCE

    Class-Wise Calibration Error
CT

    computed tomography
DSC

    Dice Similarity Coefficient
EC

    Expected Cost
ECE

    Expected Calibration Error
ECEKDE

    Expected Calibration Error Kernel Density Estimate
ER

    Error Rate
FN

    False Negative
FP

    False Positive
FPPI

    False Positives per Image
FPR

    False Positive Rate
FROC

    Free-Response Receiver Operating Characteristic
HD

    Hausdorff Distance
HD95

    Hausdorff Distance 95th Percentile
ImLC

    Image-level Classification
InS

    Instance Segmentation
IoU

    Intersection over Union
IoR

    Intersection over Reference
J

    Youden’s Index
LR+

    Positive Likelihood Ratio
KCE

    Kernel Calibration Error
LS

    Logarithmic Score
mAP

    mean Average Precision
MASD

    Mean Average Surface Distance
MCC

    Matthews Correlation Coefficient
MCE

    Maximum Calibration Error
MCP

    Maximum Class Probability
MICCAI

    Medical Image Computing and Computer Assisted Interventions
ML

    machine learning
MONAI

    Medical Open Network for Artificial Intelligence
MRI

    Magnetic Resonance Imaging
NaN

    Not a Number
NB

    Net Benefit
NPV

    Negative Predictive Value
NLL

    Negative Log Likelihood
NSD

    Normalized Surface Distance
PPV

    Positive Predictive Value
ObD

    Object Detection
PQ

    Panoptic Quality
PHI

    Protected Health Information
PR

    Precision-Recall
RBS

    Root Brier Score
RI

    Rand Index
ROC

    Receiver Operating Characteristic
SemS

    Semantic Segmentation
TN

    True Negative
TNR

    True Negative Rate
TP

    True Positive
TPR

    True Positive Rate
VoI

    Variation of Information
WCK

    Weighted Cohen’s Kappa
WSI

    Whole Slide Imaging
Xth Percentile HD

     Xth Percentile Hausdorff Distance
  *[AI]: artificial intelligence
  *[clDice]: centerline Dice Similarity Coefficient
  *[MICCAI]: Medical Image Computing and Computer Assisted Interventions
  *[mAP]: mean Average Precision
  *[PPV]: Positive Predictive Value
  *[AP]: Average Precision
  *[IoU]: Intersection over Union
  *[DSC]: Dice Similarity Coefficient
  *[BA]: Balanced Accuracy
  *[FP]: False Positive
  *[HD]: Hausdorff Distance
  *[IoR]: Intersection over Reference
  *[EC]: Expected Cost
  *[LR+]: Positive Likelihood Ratio
  *[MCC]: Matthews Correlation Coefficient
  *[NB]: Net Benefit
  *[NPV]: Negative Predictive Value
  *[AUROC]: Area under the Receiver Operating Characteristic Curve
  *[WCK]: Weighted Cohen’s Kappa
  *[ECE]: Expected Calibration Error
  *[HD95]: Hausdorff Distance 95th Percentile
  *[TP]: True Positive
  *[FN]: False Negative
  *[TN]: True Negative
  *[COCO]: Common Objects in Context
  *[FROC]: Free-Response Receiver Operating Characteristic
  *[ASSD]: Average Symmetric Surface Distance
  *[PR]: Precision-Recall
  *[BIAS]: Biomedical Image Analysis ChallengeS
  *[MONAI]: Medical Open Network
for Artificial Intelligence

  *[Boundary IoU]: Boundary Intersection over Union
  *[MASD]: Mean Average Surface Distance
  *[NSD]: Normalized Surface Distance
  *[PQ]: Panoptic Quality
  *[TPs]: True Positive
  *[TPR]: True Positive Rate
  *[TNR]: True Negative Rate
  *[AUC]: Area under the Curve
  *[BS]: Brier Score
  *[CWCE]: Class-Wise Calibration Error
  *[ECEKDE]: Expected Calibration Error Kernel Density Estimate
  *[KCE]: Kernel Calibration Error
  *[NLL]: Negative Log Likelihood
  *[RBS]: Root Brier Score
  *[TNs]: True Negative
  *[FPPI]: False Positives per Image
  *[ROC]: Receiver Operating Characteristic
  *[CI]: Confidence Interval
  *[NaN]: Not a Number
  *[MCE]: Maximum Calibration Error
  *[InS]: Instance Segmentation
  *[ObD]: Object Detection
  *[CK]: Cohen’s Kappa
  *[SemS]: Semantic Segmentation
  *[BSS]: Brier Skill Score